mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 06:16:06 -10:00
ff7e7b4963
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted `claude_local` runs execute in a sandbox. The sandbox has a fixed memory limit (4 GiB in the observed sandboxes). > - In that sandbox, every process has the same `oom_score_adj`. When a Bash tool command uses most of the memory, the kernel kills the largest process. That can be the Claude Code process. > - When Claude Code is killed, `claude-agent-acp` reports "The connection to Claude was lost." and the whole run fails. Server logs show `Claude Agent process died: Claude Code process terminated by signal SIGKILL` for each failure. > - A failed tool command is recoverable. A dead agent is not. > - This pull request makes Claude's tool commands the first out-of-memory victims in managed sandboxes. > - The benefit is that a memory-heavy command fails with exit 137, and the agent continues the run. ## Linked Issues or Issue Description Refs #16366 (diagnostic PR for the same failure). Related: #14509 protects the host control plane from local agent processes. That PR does not cover a sandboxed agent and its own tool commands. **What happened?** Hosted `claude_local` runs fail with `acpx_turn_failed` and "The connection to Claude was lost." while one `execute` (Bash) tool call is pending. The pending commands are memory-heavy, for example `tsc --noEmit` with a 3.3 GB heap, a monorepo `pnpm install`, or a type check plus `vitest`. At the same second, the server log shows `Claude Agent process died: Claude Code process terminated by signal SIGKILL`. Over 10 days, all 12 such lines on one staging stack were `SIGKILL`, and they match the failed runs. The sandbox memory cgroup has `memory.max` 4 GiB and `memory.oom.group` 0, so the kernel kills one process. Every process has `oom_score_adj` 0, so the kernel picks the largest one. **Expected behavior** Under memory pressure, the kernel kills the tool command. Claude gets a failed command (exit 137) and continues the run. **Steps to reproduce** 1. Start a `claude_local` run on a sandbox target with a 4 GiB memory limit. 2. Ask the agent to run a memory-heavy command in a large repository, for example a full `tsc --noEmit` and a test run together. 3. The kernel kills the Claude Code process, and the run fails with "The connection to Claude was lost." **Agent adapter(s) involved** Claude Code (`claude_local`, ACP lane and CLI lane, managed sandbox targets). ## What Changed - `claude-config.ts`: add `CLAUDE_TOOL_SHELL_PREFIX_SCRIPT`. Claude Code calls a `CLAUDE_CODE_SHELL_PREFIX` with one argument, the full command string. The script raises its own `oom_score_adj` to 1000 (no privilege is necessary to raise it). Then it runs the command string in the shell that Claude Code selects: `CLAUDE_CODE_SHELL`, then `SHELL`, when either is an executable bash or zsh, else zsh or bash from `PATH` (bash first when `SHELL` names bash). - `buildRemoteClaudeConfigMaterializationCommand` can install the script (umask 077, write to `.partial`, `chmod 700`, `mv`). The install runs last in the existing materialization command, so it adds no sandbox round trip. It prints a marker only when the file is in place. - `materializeRemoteClaudeConfig` returns `toolShellPrefixInstalled`. It is true only when the command exits 0 and prints the marker. - ACP lane (`acp.ts`) and CLI lane (`execute.ts`): set `CLAUDE_CODE_SHELL_PREFIX` to the installed script only when the install is confirmed. A prefix path that does not exist would fail every Bash tool call, so a failed install keeps the previous behavior. An operator `CLAUDE_CODE_SHELL_PREFIX` in the adapter env is kept, and no script is installed. - `docs/adapters/claude-local.md`: add "Sandbox memory limits". ## Verification - `npx vitest run src/server/claude-config.test.ts` in `packages/adapters/claude-local`: 10 pass, 6 new. The new tests install the script with the real materialization command through `/bin/sh` and check: - the file content, mode 0700, the marker, and that no `.partial` file remains; - the command runs with `oom_score_adj` 1000 while the caller keeps its own score (Linux only); - shell selection: a valid bash `SHELL`, an invalid `CLAUDE_CODE_SHELL` that falls back to `SHELL`, and a `SHELL` of `/bin/sh` that falls back to bash from `PATH`; - quoting and the exit status of the command are kept; - an operator prefix wins, and no marker means no install. - `acp.test.ts`: the managed-config test now checks that `CLAUDE_CODE_SHELL_PREFIX` names an executable 0700 file next to the config dir. A new test checks that an operator prefix is kept and no script is installed. Both use the local sandbox runner, which runs the setup commands for real. - `tsc --noEmit -p packages/adapters/claude-local` is clean. - Two tests fail in my sandbox both with and without this change: `uses the same default model in ACP startup and session identity` and `reselects the full assignment and bootstrap guidance after a failed resume`. They also fail on clean `master` in the same environment, so I think they depend on that environment. CI covers them. - Real Claude Code check (Claude Code 2.1.286): with `CLAUDE_CODE_SHELL_PREFIX` set to the script, a Bash tool call printed `oom_score_adj` 1000 and ran in zsh (the selected shell), and its parent was the `claude` process at score 0. Without a prefix file that exists, every Bash tool call fails with exit 127. This is why the env is set only after a confirmed install. - Real OOM check in a 4 GiB Daytona sandbox (`memory.oom.group` 0). A stand-in "agent" process held 1.2 GB at score 0, and a tool command started 14 processes of 250 MB each: - without the prefix, the kernel killed the stand-in agent (SIGKILL, 137; `oom_kill` +1); - through the installed prefix, the stand-in agent stayed alive, and the kernel killed 5 tool processes (`oom_kill` +5). - Assumption that could invalidate the design: Claude Code passes one argument (the command string) to the prefix and honors the variable through `claude-agent-acp`. I checked the argument shape with a logging prefix, and `claude-agent-acp` 0.73.0 spawns the CLI with `{ ...process.env }`. ## Risks - Every Bash tool command and hook in a managed sandbox now runs through one more `sh` process and an `exec`. The cost is a few milliseconds for each command. - If a future Claude Code release changes how it calls `CLAUDE_CODE_SHELL_PREFIX`, Bash tool calls can fail. The tests above and the documented variable limit this risk. Removing the variable restores the old behavior. - The inner shell is not a login shell. Claude Code sources its own shell snapshot inside the command string, so the environment stays the same. If Claude Code runs without a snapshot, it uses a login shell. Only exported variables then pass through to the inner shell. - If the selected shell differs from the one that Claude Code chose (for example, a custom `CLAUDE_CODE_SHELL` that is not bash or zsh), the script uses the same fallback order as Claude Code. - This does not help when the Claude Code process itself uses most of the memory. The local (non-sandbox) lanes do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use and code execution. It read the repository and the Claude Code CLI behavior, ran the OOM experiment, edited the code, and ran the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (two tests fail on clean `master` in my sandbox too; see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Paperclip <[email protected]>