fix(restart-chaos): rebuild the Daytona plugin when its sources change (#19798)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness runs Daytona waves with a local control
plane that installs the bundled Daytona sandbox-provider plugin from
`packages/plugins/sandbox-providers/daytona`
> - That plugin is excluded from the pnpm workspace, so workspace builds
and `scripts/agent-preflight.mjs` never rebuild it
> - The harness built it only when `dist/` was missing. A long-lived
chaos worktree that moved across many master commits kept serving the
plugin it built on day one
> - As a result, plugin-side fixes such as #19696 (reclaim sandboxes
left by a restart during acquire) and #19754 (keep a runner sandbox
alive through the reconnect grace) were not under test in those waves,
while server and runner fixes were
> - This pull request keys the plugin build on a content hash of its
inputs and rebuilds when the hash changes
> - The benefit is that every Daytona wave tests the plugin of the
commit it claims to test

## Linked Issues or Issue Description

**Bug description**
`ensureDaytonaPlugin()` in `tests/restart-chaos/main.ts` returned early
whenever `dist/manifest.js`, `dist/worker.js` and the Daytona SDK
existed.

**Steps to reproduce**
Build the plugin on commit A, check out a later commit B that changes
`packages/plugins/sandbox-providers/daytona/src`, and run a `--env
daytona` wave. The wave installs the commit-A plugin. In one chaos
worktree, `dist/plugin.js` was 20 hours older than `src/plugin.ts` and
lacked `runnerReconnectGraceMs` from #19754.

**Expected behavior**
The harness rebuilds the plugin whenever its sources, lockfile or the
plugin SDK source changed since the last build.

Refs #19544

## What Changed

- `tests/restart-chaos/plugin-build.ts` (new): `pluginBuildInputHash`
hashes the plugin's `package.json`, `pnpm-lock.yaml`, `tsconfig.json`
and `src/**`, plus extra inputs: the plugin SDK `src`, `package.json`
and `tsconfig.json`, and the root `tsconfig.json` and
`tsconfig.base.json` that both packages extend. It excludes
`node_modules` and `dist`. `pluginBuildState` compares that hash with a
stamp in `dist/.restart-chaos-build-input.sha256` and reports a reason.
- `tests/restart-chaos/main.ts`: `ensureDaytonaPlugin` installs, links
the SDK and builds when the state is stale, then writes the stamp. It
records `daytona_plugin_build_state` and `daytona_plugin_built` in the
timeline.
- `tests/restart-chaos/plugin-build.test.ts` (new): an unstamped build
is stale; the stamped build is up to date; a changed source, lockfile or
SDK source is stale; changes to `node_modules` and `dist` are ignored;
missing outputs or dependencies are stale.

Other out-of-workspace packages, checked as requested:
- `pnpm-workspace.yaml` excludes only
`packages/plugins/sandbox-providers/**`.
- The restart chaos local mode (`--env local`) does not install a
sandbox provider.
- CI's runner-e2e "Prepare bundled Daytona plugin" step runs on a fresh
checkout per job, so it cannot see a stale `dist/`.
- The other sandbox providers are not used by these harnesses. The risk
is therefore limited to long-lived local worktrees running Daytona
waves, which this change covers.

## Verification

- `vitest run --config tests/restart-chaos/vitest.config.ts`: 121/121
pass, including the new staleness tests.
- `tsc -p tests/restart-chaos/tsconfig.json`: no errors in the changed
files.
- Manual: in the chaos worktree, `dist/plugin.js` (built 2026-11-06
11:16) lacked `runnerReconnectGraceMs`. After a rebuild at master
35d41f11bb it contains it.

## Risks

- This change touches the test harness only.
- The first wave after this change rebuilds the plugin once in each
worktree, because existing builds have no stamp. That takes a few
seconds.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, GitHub CLI). Default context window,
extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>

tests/restart-chaos/main.ts

@@ -7,7 +7,7 @@
  *   node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --replay <wave>/<task>
  */
 import { execFileSync, spawnSync } from "node:child_process";
-import { existsSync } from "node:fs";
+import { existsSync, writeFileSync } from "node:fs";
 import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
 import os from "node:os";
 import path from "node:path";
@@ -15,6 +15,7 @@ import { setTimeout as delay } from "node:timers/promises";
 import { ChaosApi, isSocketTypeOfServiceRace } from "./api.js";
 import { CREDENTIAL_FOR_ADAPTER, parseCommand, replayCommand, waveIdFor } from "./config.js";
 import { assertControllerMatchesImage, prepareDaytonaImageArtifacts } from "./daytona-image.js";
+import { PLUGIN_BUILD_STAMP, pluginBuildState } from "./plugin-build.js";
 import { DaytonaWaveSweeper, leaksForTask, unattributedLeaks, type DaytonaSweepResult } from "./daytona-sweep.js";
 import { knownSecretValues, writeEvidenceBundle } from "./evidence.js";
 import { gradeTask } from "./grader.js";
@@ -59,13 +60,27 @@ function run(command: string, args: string[], cwd: string, label: string) {
   if (result.status !== 0) throw new Error(`${label} failed (${command} ${args.join(" ")})`);
 }
 
-/** The bundled Daytona plugin is outside the root workspace; prepare it as CI does. */
-function ensureDaytonaPlugin() {
+/**
+ * The bundled Daytona plugin is outside the root workspace; prepare it as CI
+ * does, and rebuild it whenever its sources (or the plugin SDK's) changed since
+ * the last build, so a worktree that moved to a new master tests that master's
+ * plugin rather than an older build.
+ */
+function ensureDaytonaPlugin(timeline: Timeline) {
   const root = path.join(repositoryRoot, "packages/plugins/sandbox-providers/daytona");
-  if (["dist/manifest.js", "dist/worker.js", "node_modules/@daytonaio/sdk"].every(file => existsSync(path.join(root, file)))) return;
+  // Inputs outside the plugin directory: the linked plugin SDK and the tsconfig chain both packages extend.
+  const extraInputs = [
+    ...["src", "package.json", "tsconfig.json"].map(file => path.join(repositoryRoot, "packages/plugins/sdk", file)),
+    ...["tsconfig.json", "tsconfig.base.json"].map(file => path.join(repositoryRoot, file)),
+  ];
+  const state = pluginBuildState(root, extraInputs);
+  timeline.add("daytona_plugin_build_state", { reason: state.reason, inputHash: state.inputHash });
+  if (!state.needsBuild) return;
   run("pnpm", ["install", "--ignore-workspace", "--frozen-lockfile", "--ignore-scripts"], root, "Daytona plugin install");
   run(process.execPath, [path.join(repositoryRoot, "scripts/link-plugin-dev-sdk.mjs")], repositoryRoot, "Plugin SDK link");
   run("pnpm", ["--dir", root, "build"], repositoryRoot, "Daytona plugin build");
+  writeFileSync(path.join(root, PLUGIN_BUILD_STAMP), `${state.inputHash}\n`);
+  timeline.add("daytona_plugin_built", { inputHash: state.inputHash });
 }
 
 /**
@@ -80,7 +95,7 @@ async function prepareDaytona(plan: WavePlan, timeline: Timeline): Promise<Recor
   const { localContentId } = await assertControllerMatchesImage(artifacts, repositoryRoot);
   timeline.add("daytona_image_matched", { image, contentId: artifacts.contentId, localContentId, sourceRevision: artifacts.sourceRevision,
     runnerdSha256: artifacts.runnerdSha256, providerPackDigest: artifacts.providerPackDigest, platformManifestDigest: artifacts.platformManifestDigest });
-  ensureDaytonaPlugin();
+  ensureDaytonaPlugin(timeline);
   timeline.add("daytona_vcpu_plan", { cpuPerSandbox: plan.options.daytonaCpu, concurrency: plan.options.concurrency,
     taskVcpu: plan.options.concurrency * (plan.options.daytonaCpu ?? 0) });
   return { PAPERCLIP_RUNNER_REMOTE_BINARY_PATH: artifacts.runnerdPath, PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH: artifacts.providerPackPath };

tests/restart-chaos/plugin-build.test.ts

@@ -0,0 +1,77 @@
+import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { afterEach, describe, expect, it } from "vitest";
+import { PLUGIN_BUILD_STAMP, pluginBuildInputHash, pluginBuildState } from "./plugin-build.js";
+
+const roots: string[] = [];
+afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
+
+async function plugin(): Promise<{ root: string; sdk: string }> {
+  const base = await mkdtemp(path.join(os.tmpdir(), "chaos-plugin-"));
+  roots.push(base);
+  const root = path.join(base, "daytona");
+  const sdk = path.join(base, "sdk-src");
+  for (const [file, content] of Object.entries({
+    "package.json": "{\"name\":\"@paperclipai/plugin-daytona\"}", "pnpm-lock.yaml": "lock: 1\n", "tsconfig.json": "{}",
+    "src/plugin.ts": "export const v = 1;\n", "src/nested/worker.ts": "export {};\n",
+    "dist/manifest.js": "", "dist/worker.js": "", "node_modules/@daytonaio/sdk/package.json": "{}",
+    "node_modules/ignored/index.js": "ignored",
+  })) {
+    await mkdir(path.dirname(path.join(root, file)), { recursive: true });
+    await writeFile(path.join(root, file), content);
+  }
+  await mkdir(sdk, { recursive: true });
+  await writeFile(path.join(sdk, "index.ts"), "export const sdk = 1;\n");
+  return { root, sdk };
+}
+
+const stamp = (root: string, hash: string) => writeFile(path.join(root, PLUGIN_BUILD_STAMP), `${hash}\n`);
+
+describe("Daytona plugin build staleness", () => {
+  it("rebuilds an unstamped build, then reports up to date once stamped with its inputs", async () => {
+    const { root, sdk } = await plugin();
+    const first = pluginBuildState(root, [sdk]);
+    expect(first).toMatchObject({ needsBuild: true, needsInstall: false, reason: "build was not stamped with its inputs" });
+    await stamp(root, first.inputHash);
+    expect(pluginBuildState(root, [sdk])).toMatchObject({ needsBuild: false, reason: "up to date" });
+  });
+
+  it("detects a changed plugin source, lockfile or plugin SDK source after a build", async () => {
+    const { root, sdk } = await plugin();
+    await stamp(root, pluginBuildInputHash(root, [sdk]));
+    await writeFile(path.join(root, "src/plugin.ts"), "export const v = 2;\n");
+    expect(pluginBuildState(root, [sdk])).toMatchObject({ needsBuild: true, reason: "plugin sources changed since the last build" });
+
+    await stamp(root, pluginBuildInputHash(root, [sdk]));
+    await writeFile(path.join(root, "pnpm-lock.yaml"), "lock: 2\n");
+    expect(pluginBuildState(root, [sdk]).needsBuild).toBe(true);
+
+    await stamp(root, pluginBuildInputHash(root, [sdk]));
+    await writeFile(path.join(sdk, "index.ts"), "export const sdk = 2;\n");
+    expect(pluginBuildState(root, [sdk]).needsBuild).toBe(true);
+  });
+
+  it("detects a change to an inherited tsconfig passed as a single-file extra input", async () => {
+    const { root, sdk } = await plugin();
+    const baseConfig = path.join(path.dirname(root), "tsconfig.base.json");
+    await writeFile(baseConfig, "{\"compilerOptions\":{\"target\":\"ES2022\"}}");
+    await stamp(root, pluginBuildInputHash(root, [sdk, baseConfig]));
+    expect(pluginBuildState(root, [sdk, baseConfig]).needsBuild).toBe(false);
+    await writeFile(baseConfig, "{\"compilerOptions\":{\"target\":\"ES2024\"}}");
+    expect(pluginBuildState(root, [sdk, baseConfig])).toMatchObject({ needsBuild: true, reason: "plugin sources changed since the last build" });
+  });
+
+  it("ignores node_modules and build output, and requires dependencies and outputs", async () => {
+    const { root, sdk } = await plugin();
+    const hash = pluginBuildInputHash(root, [sdk]);
+    await writeFile(path.join(root, "node_modules/ignored/index.js"), "changed");
+    await writeFile(path.join(root, "dist/worker.js"), "changed");
+    expect(pluginBuildInputHash(root, [sdk])).toBe(hash);
+    await stamp(root, hash);
+    await rm(path.join(root, "dist/manifest.js"));
+    expect(pluginBuildState(root, [sdk])).toMatchObject({ needsBuild: true, reason: "build output missing" });
+    await rm(path.join(root, "node_modules/@daytonaio"), { recursive: true });
+    expect(pluginBuildState(root, [sdk])).toMatchObject({ needsInstall: true, reason: "dependencies missing" });
+  });
+});

tests/restart-chaos/plugin-build.ts

@@ -0,0 +1,63 @@
+/**
+ * The bundled Daytona plugin lives outside the pnpm workspace, so workspace
+ * builds and `agent-preflight` never rebuild it. A worktree that moves to a new
+ * master would otherwise keep serving the plugin built for an older commit,
+ * and plugin-side fixes would silently go untested.
+ *
+ * The build is keyed by a content hash of the plugin's inputs, recorded next
+ * to its output.
+ */
+import { createHash } from "node:crypto";
+import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
+import path from "node:path";
+
+export const PLUGIN_BUILD_STAMP = "dist/.restart-chaos-build-input.sha256";
+const INPUT_FILES = ["package.json", "pnpm-lock.yaml", "tsconfig.json"];
+const INPUT_DIRECTORIES = ["src"];
+
+function listFiles(root: string, relative: string): string[] {
+  const directory = path.join(root, relative);
+  if (!existsSync(directory)) return [];
+  return readdirSync(directory, { withFileTypes: true }).flatMap(entry => {
+    const child = path.posix.join(relative, entry.name);
+    if (entry.isDirectory()) return entry.name === "node_modules" ? [] : listFiles(root, child);
+    return entry.isFile() ? [child] : [];
+  });
+}
+
+/** sha256 over the plugin's build inputs: paths and contents, in sorted order. */
+export function pluginBuildInputHash(pluginRoot: string, extraInputs: readonly string[] = []): string {
+  const hash = createHash("sha256");
+  const files = [...INPUT_FILES.filter(file => existsSync(path.join(pluginRoot, file))),
+    ...INPUT_DIRECTORIES.flatMap(directory => listFiles(pluginRoot, directory))].sort();
+  for (const file of files) {
+    hash.update(`${file}\0`);
+    hash.update(readFileSync(path.join(pluginRoot, file)));
+    hash.update("\0");
+  }
+  // Inputs outside the plugin directory (for example the linked plugin SDK) are hashed as given.
+  for (const extra of [...extraInputs].sort()) {
+    hash.update(`extra:${extra}\0`);
+    for (const file of statSync(extra, { throwIfNoEntry: false })?.isDirectory() ? listFiles(extra, "").sort() : [""]) {
+      const absolute = file ? path.join(extra, file) : extra;
+      if (!existsSync(absolute)) continue;
+      hash.update(`${file}\0`);
+      hash.update(readFileSync(absolute));
+    }
+  }
+  return hash.digest("hex");
+}
+
+export interface PluginBuildState { needsInstall: boolean; needsBuild: boolean; inputHash: string; reason: string }
+
+/** Whether the installed and built plugin matches its current sources. */
+export function pluginBuildState(pluginRoot: string, extraInputs: readonly string[] = []): PluginBuildState {
+  const inputHash = pluginBuildInputHash(pluginRoot, extraInputs);
+  const needsInstall = !existsSync(path.join(pluginRoot, "node_modules/@daytonaio/sdk"));
+  const outputs = ["dist/manifest.js", "dist/worker.js"].every(file => existsSync(path.join(pluginRoot, file)));
+  const stamp = existsSync(path.join(pluginRoot, PLUGIN_BUILD_STAMP))
+    ? readFileSync(path.join(pluginRoot, PLUGIN_BUILD_STAMP), "utf8").trim() : null;
+  const reason = needsInstall ? "dependencies missing" : !outputs ? "build output missing"
+    : stamp === null ? "build was not stamped with its inputs" : stamp !== inputHash ? "plugin sources changed since the last build" : "up to date";
+  return { needsInstall, needsBuild: reason !== "up to date", inputHash, reason };
+}