test: overlap long recovery and maintenance waits (#18544)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - CI verifies recovery, execution authority, and background work
before changes merge.
> - Diff selection saves work, but a selected file still runs its cases
in sequence.
> - The three longest files in a recent full master run each took over
eight minutes.
> - Many of these cases must observe real lease expiry or idle periods.
> - This pull request gives independent cases separate fixtures so those
periods can overlap.
> - The benefit is a shorter test path with the same assertions and
observation windows.

## Linked Issues or Issue Description

Refs #18464.

The latest measured successful master [Cloud readiness
run](https://github.com/paperclipai/paperclip/actions/runs/41300933157)
spent 581 seconds in issue recovery, 569 seconds in native restart
recovery, and 512 seconds in maintenance server routes. File-level
selection cannot divide those long serial files. Shortening their lease
or idle windows would weaken the regressions they protect.

## What Changed

- Split issue recovery into four files with separate database and worker
fixtures. Keep all 18 cases.
- Split native restart coverage into recovery, lease renewal,
preparation, checkpoint cost, and disk-loss files. Keep all original 79
cases, including the existing container-qualified cases. The master
update also retains its newly added recovery cases.
- Close each native case's database clients after pending failure
reports drain. The shortened file must not rely on idle timeouts to stay
below PostgreSQL's connection cap.
- Run the native Runner lane with two isolated fork workers. Register
every new file in the lane and verify that the partition still includes
every suite.
- Give each maintenance case an owned database. Run at most two cases
together with test-local assertion contexts. Keep both real 90-second
quiet windows and all 17 cases.
- Preserve serial execution when any explicit shared
`PAPERCLIP_*_TEST_DATABASE_URL` is present. An admin URL that creates a
separate database per fixture can run concurrently.
- Refresh shard weights from the protected master run. Mark the initial
split-file weights as conservative estimates until master measures them.
- Anchor the attention-retention fake clock to PostgreSQL time. Its
fixed clock started trailing real database timestamps after 12:00 UTC on
November 1, causing an archive-deadline test to time out. Keep its
deadline assertions and timeout unchanged.
- Document the fixture and concurrency rules. Leave runner authorization
and workflows unchanged.

## Verification

- Final head `c9a4ff11b3ba81277341294c7721b34fb8705ede`: [full PR CI
passed](https://github.com/paperclipai/paperclip/actions/runs/41477848686),
including typecheck, build, all test shards, E2E, and the final
verification gate. The rollup has 55 successful checks, one neutral
review check, four expected skips, and no pending or failed checks.
Greptile reviewed this exact head at 5/5 with no actionable findings.
- November 2 master update: merge commit
`c9a4ff11b3ba81277341294c7721b34fb8705ede` resolves the native recovery
import conflict while retaining the new busy-row and remote continuity
cases from master. All 38 test-runner checks and the three new busy-row
cases passed locally.
- Retention verification after the master update: the first local full
run passed 52 cases and timed out at 15 seconds in the paged-backlog
case. That case passed in isolation. The unchanged master suite passed
all 53 cases (82.95 seconds), and a full PR rerun also passed all 53
(76.65 seconds). The initial timeout cause remains unproven; no timeout
or assertion was weakened.
- Previously verified head `fd288a8e8c88df54dcdca7b7e42366b019182782`:
[full PR CI
passed](https://github.com/paperclipai/paperclip/actions/runs/41312625439).
There are 55 successful checks, one neutral review check, four expected
skips, and no pending or failed checks. Greptile reviewed this head at
5/5 with no actionable findings.
- Issue recovery: 18/18 passed before and after the split on the same
local machine and PostgreSQL fixture path. Wall time changed from
**630.6 seconds to 319.7 seconds** (49% less). All case titles match.
- Maintenance also passed all 17 cases on AWS: structured file cost
dropped from **511.8 to 282.6 seconds** across the linked master run and
the first PR run. These are different CI runs, so the matched local
comparison below is the stronger isolated measurement.
- Maintenance: 17/17 passed before and after with the restored pinned
dependencies. Wall time changed from **488.1 seconds to 317.2 seconds**
(35% less). All 120 assertion lines remain. Both real 90-second windows
also passed a final targeted run.
- Native restart: 71 passed and eight existing container-only skips
before and after the split. Wall time changed from **632.1 seconds to
346.8 seconds** (45% less). All 79 cases and 254 assertion lines remain.
- `pnpm -r typecheck`: passed.
- `pnpm build`: passed.
- Extra TypeScript checking of the moved recovery and maintenance tests
passed. The standard server config excludes `src/__tests__`. A separate
check of attention-retention fixtures reports existing type errors; the
unchanged baseline has identical diagnostics, with none introduced by
the clock fix.
- `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/ci/test-selection*.test.mjs`: 61 passed.
- `pnpm test:run`: stopped in the second chat partition: 335 passed and
four failed (two GitHub retry-state assertions and two Teams timeouts),
after 339 passed in the first partition. These chat tests are unchanged.
All four pass in a focused run (19.1 seconds). The full-suite
ordering/load cause remains unproven. All three CI chat jobs passed. The
previously failing cases passed there in 1.3, 1.3, 2.4, and 2.2 seconds;
see [the chat job
log](https://github.com/paperclipai/paperclip/actions/runs/41309691650/job/124857580714).
The attention-retention timeout repeated in CI and in an isolated local
run. Its fake clock was fixed at November 1 noon, which had fallen
behind real PostgreSQL timestamps. Anchoring it to the database clock
passes all 53 cases; the failed case now takes 425 ms.
- The native CI lane passed all 162 runnable cases with eight existing
skips after client cleanup: **650.7 seconds on the earlier master run to
391.0 seconds on this PR**.
- CI found native connection exhaustion after the long waits moved out
of the main file. The same failure reproduced locally at a
100-connection cap. Per-case client cleanup fixes the shortened file at
that cap, and all five native files now pass together at that cap (71
passed, eight existing skips).
- Early attempts exposed a missed fixture reference and a missed
reporting setup hook during extraction; both were restored. The first CI
policy run also required moving the new native fixture under test
helpers; that boundary check now passes locally. A separate auxiliary
command picked up the host pnpm version and removed dependencies. The
original maintenance baseline reproduced the same missing-package
failures, so that failed attempt is excluded from timing comparisons. A
later unchanged-baseline run passed all 17 cases. The pinned install was
restored before the passing maintenance run.

The audit also covered heartbeat process recovery, deferred receipts,
chat, workspace recovery, attention, and connection management. Their
shared fake clocks/provider mocks or existing partitioning need
different changes. The 16,400-row attention case and the large
checkpoint stream remain intact because their size protects real limits.

## Risks

- Two isolated processes use more memory and database connections than
one. Concurrency is capped at two. Shared database overrides force
serial execution.
- Moving setup into helpers can change test lifecycle order. The split
suites keep the same case assertions and use independent process state;
focused runs and the full CI suite check the move.
- Split-file weights are estimates based on case timings from the linked
run plus repeated setup cost. The next protected-master refresh replaces
them. Weights never permit tests to be skipped.
- Local timing varies with host load. CI measurements after merge are
needed to establish the overall PR wall-time improvement.

## Model Used

OpenAI Codex, GPT-6, with reasoning, repository inspection, tool use,
and code execution. The exact backend revision and context-window size
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (changed suites and focused
reruns; the full local-run caveat is recorded above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <[email protected]>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-02 10:19:24 -06:00
1 parent baa4a2e64e
commit 412834a860
18 files changed
+2911 -2595

No files matched your search

+9 -3
View File
@@ -355,9 +355,15 @@ supported reruns still use them.
General server shards run at most two test files at a time in separate Vitest
forks. Each file retains its own home, config and process state, and its test
cases run serially. An explicit `PAPERCLIP_TEST_DATABASE_URL` keeps those shards
serial because the database is shared. Unsharded local tests, route/authz lanes,
and native Runner lanes retain their existing concurrency. Sharded chat jobs run
cases run serially. An explicit `PAPERCLIP_*_TEST_DATABASE_URL`, including
`PAPERCLIP_TEST_DATABASE_URL`, keeps those shards serial because the database
is shared. Native Runner suites also use two isolated file workers. Their long
lease, preparation, disk-loss and checkpoint tests have separate files so real
observation windows can overlap. Issue recovery suites split by producer and
recovery behavior for the same reason. Unsharded local tests and route/authz
lanes retain their existing file concurrency. Maintenance server cases overlap
two owned server/database fixtures within their file; an explicit shared
maintenance database keeps them serial. Sharded chat jobs run
two independent partitions, each with its own embedded PostgreSQL, home and
config; an explicit shared test database retains one partition per job.
The Runner transport lease-expiry and cold-admission regressions live in their
@@ -324,6 +324,8 @@ const usageDeliverySuitePath = "server/src/services/native-runtime/native-usage-
const dotRunnerSuitePath = "server/src/__tests__/dot-runner.test.ts";
const restartRecoverySuitePath =
"server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts";
const restartWaitSuites = ["lease-renewal", "preparation", "checkpoint-cost", "diskless"]
.map(name => `server/src/services/native-runtime/native-runner-restart-${name}.integration.test.ts`);
// Mirrors pr-trusted.yml (12 shards, called by pr.yml so GITHUB_WORKFLOW is
// "PR"): the chat suite runs in its dedicated lanes and the cargo-dependent
@@ -365,7 +367,7 @@ test("12 PR without-chat shards plus the dedicated chat and native-runner lanes
assert.ok(!files.includes(dotRunnerSuitePath));
assert.ok(!files.includes(restartRecoverySuitePath));
assert.ok(!files.includes(usageDeliverySuitePath));
assert.deepEqual([...files, chatSuitePath, nativeRunnerSuitePath, dotRunnerSuitePath, restartRecoverySuitePath, usageDeliverySuitePath, "server/src/services/native-runtime/plugin-runner-probe.test.ts"].sort(), full.selectedGeneralServerSuites.sort());
assert.deepEqual([...files, chatSuitePath, nativeRunnerSuitePath, dotRunnerSuitePath, restartRecoverySuitePath, ...restartWaitSuites, usageDeliverySuitePath, "server/src/services/native-runtime/plugin-runner-probe.test.ts"].sort(), full.selectedGeneralServerSuites.sort());
assert.equal(new Set(files).size, files.length);
const defaultRun = dryRunJson([], prEnv);
assert.ok(defaultRun.generalServerSuiteCount === full.generalServerSuiteCount);
@@ -414,6 +416,7 @@ test("the native-runner lane runs exactly the cargo-dependent suites", () => {
"server/src/services/native-runtime/native-codex-runner.integration.test.ts",
dotRunnerSuitePath,
restartRecoverySuitePath,
...restartWaitSuites,
usageDeliverySuitePath,
]);
});
@@ -724,3 +727,31 @@ test("a failed chat child fails the job after its sibling finishes", { skip: pro
rmSync(directory, { recursive: true, force: true });
}
});
test("native files overlap only when their database fixtures are isolated", { skip: process.platform === "win32" }, () => {
const directory = mkdtempSync(path.join(os.tmpdir(), "native-file-workers-"));
const receipt = path.join(directory, "args.json");
const fakePnpm = path.join(directory, "pnpm");
writeFileSync(fakePnpm, `#!/usr/bin/env node\nrequire('node:fs').writeFileSync(${JSON.stringify(receipt)}, JSON.stringify(process.argv.slice(2)));\n`);
chmodSync(fakePnpm, 0o755);
try {
for (const [databaseEnv, workers] of [
[{}, 2],
[{ PAPERCLIP_TEST_DATABASE_URL: "postgresql://shared/test" }, 1],
[{ PAPERCLIP_ISSUE_RECOVERY_TEST_DATABASE_URL: "postgresql://shared/recovery_test" }, 1],
[{ PAPERCLIP_TEST_POSTGRES_ADMIN_URL: "postgresql://owned/postgres" }, 2],
]) {
const env = workflowEnv({ PATH: `${directory}${path.delimiter}${process.env.PATH}`, PAPERCLIP_CI_PROFILE: "false" });
for (const key of Object.keys(env)) if (key.endsWith("_TEST_DATABASE_URL")) delete env[key];
Object.assign(env, databaseEnv);
const result = spawnSync(process.execPath, [script, "--mode", "general", "--group", "general-server-native-runner"], {
cwd: repoRoot, encoding: "utf8", env,
});
assert.equal(result.status, 0, result.stdout + result.stderr);
const args = JSON.parse(readFileSync(receipt, "utf8"));
assert.ok(args.includes(`--maxWorkers=${workers}`));
assert.ok(args.includes(workers === 2 ? "--fileParallelism" : "--no-file-parallelism"));
for (const file of [restartRecoverySuitePath, ...restartWaitSuites]) assert.ok(args.includes(file));
}
} finally { rmSync(directory, { recursive: true, force: true }); }
});
File diff suppressed because it is too large. Load diff
+19 -5
View File
@@ -72,6 +72,11 @@ const additionalSerializedServerTests = new Set([
"server/src/__tests__/redaction.test.ts",
"server/src/__tests__/routines-e2e.test.ts",
]);
// Suite-specific external URLs can also name a shared mutable database.
// Owned database factories use PAPERCLIP_TEST_POSTGRES_ADMIN_URL instead.
const hasSharedTestDatabase = Object.entries(process.env).some(([key, value]) =>
key.startsWith("PAPERCLIP_") && key.endsWith("_TEST_DATABASE_URL") && value?.trim(),
);
let invocationIndex = 0;
const serializedModeName = "serialized";
const generalModeName = "general";
@@ -92,6 +97,10 @@ const nativeRunnerSuites = [
"server/src/__tests__/dot-runner.test.ts",
// These process cases otherwise skip in server shards without Runner binaries.
"server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts",
"server/src/services/native-runtime/native-runner-restart-lease-renewal.integration.test.ts",
"server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts",
"server/src/services/native-runtime/native-runner-restart-checkpoint-cost.integration.test.ts",
"server/src/services/native-runtime/native-runner-restart-diskless.integration.test.ts",
"server/src/services/native-runtime/native-usage-delivery.integration.test.ts",
];
// In the PR workflow (pr.yml, the caller of pr-trusted.yml — reusable
@@ -481,7 +490,7 @@ async function runProjectGroup(projects, groupName, shardIndex = null, shardCoun
}
function chatShardWorkers(index, count) {
const workers = count > 1 && !process.env.PAPERCLIP_TEST_DATABASE_URL ? 2 : 1;
const workers = count > 1 && !hasSharedTestDatabase ? 2 : 1;
return Array.from({ length: workers }, (_, offset) => ({
index: (index ?? 0) * workers + offset,
count: (count ?? 1) * workers,
@@ -502,7 +511,12 @@ async function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCo
}
if (groupName === generalServerNativeRunnerGroupName) {
await runVitest(
["--project", "@paperclipai/server", ...serializedServerVitestArgs, ...nativeRunnerSuites],
["--project", "@paperclipai/server",
// Each file owns its database, home, and processes. Keep cases serial,
// but overlap long real lease windows in two isolated fork workers.
...(hasSharedTestDatabase ? serializedServerVitestArgs
: ["--pool=forks", "--isolate", "--fileParallelism", "--maxWorkers=2"]),
...nativeRunnerSuites],
"native runner vertical-slice suite",
);
return;
@@ -531,7 +545,7 @@ async function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCo
// Files have separate fork processes and isolated homes/configs.
// Keep each file's cases serial; overlap only two independent files.
// An explicit shared test database cannot be isolated this way.
...(process.env.PAPERCLIP_TEST_DATABASE_URL
...(hasSharedTestDatabase
? serializedServerVitestArgs
: ["--pool=forks", "--isolate", "--fileParallelism", "--maxWorkers=2"]),
...shardFiles,
@@ -645,7 +659,7 @@ if (selection && options.dryRun) {
writeFileSync(1, JSON.stringify({ selection }));
process.exit(0);
}
const defaultChatShards = !selection && options.group === null && options.mode !== serializedModeName && !process.env.PAPERCLIP_TEST_DATABASE_URL
const defaultChatShards = !selection && options.group === null && options.mode !== serializedModeName && !hasSharedTestDatabase
? Array.from({ length: 6 }, (_, index) => ({ index, count: 6 }))
: null;
if (options.dryRun) {
@@ -708,7 +722,7 @@ try {
if (selection) {
for (const [project, files] of Object.entries(selection)) {
const serverArgs = project !== "@paperclipai/server" ? []
: options.mode === serializedModeName || process.env.PAPERCLIP_TEST_DATABASE_URL
: options.mode === serializedModeName || hasSharedTestDatabase
? ["--pool=forks", "--isolate", ...serializedServerVitestArgs]
: ["--pool=forks", "--isolate", "--fileParallelism", "--maxWorkers=2"];
await runVitest(["--project", project, ...serverArgs, ...files],
+167 -172
View File
@@ -3,186 +3,181 @@
"unit": "ms",
"sources": [
{
"run": 41202111422,
"run": 41300933157,
"attempt": 1,
"sha": "c2971badde04b602fa9fe698015e1f3653b39e6b",
"url": "https://github.com/paperclipai/paperclip/actions/runs/41202111422"
},
{
"run": 41201965500,
"attempt": 1,
"sha": "d5766e40748ac5881c4b9879fad1d5a61d9b493d",
"url": "https://github.com/paperclipai/paperclip/actions/runs/41201965500"
"sha": "247552fe0719dcc92f7b42c50b658be6f624986d",
"url": "https://github.com/paperclipai/paperclip/actions/runs/41300933157"
}
],
"coverage": {
"measured": 163,
"withSetup": 0,
"inventory": 163
"measured": 164,
"withSetup": 164,
"inventory": 164
},
"durations": {
"server/src/__tests__/access-routes-hidden-floor.test.ts": 4202,
"server/src/__tests__/access-routes-permissions-upgrade.test.ts": 3661,
"server/src/__tests__/activity-routes.test.ts": 15018,
"server/src/__tests__/adapter-auth-signal-routes.test.ts": 14386,
"server/src/__tests__/adapter-model-refresh-routes.test.ts": 7225,
"server/src/__tests__/adapter-routes-authz.test.ts": 13466,
"server/src/__tests__/adapter-routes.test.ts": 3359,
"server/src/__tests__/agent-action-audit-routes.test.ts": 4488,
"server/src/__tests__/agent-adapter-validation-routes.test.ts": 77234,
"server/src/__tests__/agent-cross-tenant-authz-routes.test.ts": 4565,
"server/src/__tests__/agent-device-login-routes.test.ts": 4861,
"server/src/__tests__/agent-hire-auth-inheritance-routes.test.ts": 3045,
"server/src/__tests__/agent-hire-idempotency-routes.test.ts": 1271,
"server/src/__tests__/agent-instructions-routes.test.ts": 3836,
"server/src/__tests__/agent-live-run-routes.test.ts": 69136,
"server/src/__tests__/agent-permissions-routes.test.ts": 3946,
"server/src/__tests__/agent-runner-qualification-routes.test.ts": 1009,
"server/src/__tests__/agent-secrets-routes.test.ts": 1104,
"server/src/__tests__/agent-skills-routes.test.ts": 4859,
"server/src/__tests__/agent-test-environment-routes.test.ts": 6760,
"server/src/__tests__/ai-connection-router.test.ts": 19925,
"server/src/__tests__/announcements-routes.test.ts": 626,
"server/src/__tests__/approval-routes-idempotency.test.ts": 3352,
"server/src/__tests__/artifact-review-document-routes.test.ts": 7798,
"server/src/__tests__/assets.test.ts": 8543,
"server/src/__tests__/auth-routes.test.ts": 100,
"server/src/__tests__/auth-session-route.test.ts": 100,
"server/src/__tests__/authz-company-access.test.ts": 100,
"server/src/__tests__/access-routes-hidden-floor.test.ts": 4135,
"server/src/__tests__/access-routes-permissions-upgrade.test.ts": 5005,
"server/src/__tests__/activity-routes.test.ts": 15673,
"server/src/__tests__/adapter-auth-signal-routes.test.ts": 12341,
"server/src/__tests__/adapter-model-refresh-routes.test.ts": 6883,
"server/src/__tests__/adapter-routes-authz.test.ts": 13707,
"server/src/__tests__/adapter-routes.test.ts": 3323,
"server/src/__tests__/agent-action-audit-routes.test.ts": 6056,
"server/src/__tests__/agent-adapter-validation-routes.test.ts": 84643,
"server/src/__tests__/agent-cross-tenant-authz-routes.test.ts": 4315,
"server/src/__tests__/agent-device-login-routes.test.ts": 6491,
"server/src/__tests__/agent-hire-auth-inheritance-routes.test.ts": 7530,
"server/src/__tests__/agent-hire-idempotency-routes.test.ts": 5637,
"server/src/__tests__/agent-instructions-routes.test.ts": 4157,
"server/src/__tests__/agent-live-run-routes.test.ts": 66708,
"server/src/__tests__/agent-permissions-routes.test.ts": 4758,
"server/src/__tests__/agent-runner-qualification-routes.test.ts": 5232,
"server/src/__tests__/agent-secrets-routes.test.ts": 5078,
"server/src/__tests__/agent-skills-routes.test.ts": 5414,
"server/src/__tests__/agent-test-environment-routes.test.ts": 4719,
"server/src/__tests__/ai-connection-router.test.ts": 34133,
"server/src/__tests__/announcements-routes.test.ts": 4301,
"server/src/__tests__/approval-routes-idempotency.test.ts": 3592,
"server/src/__tests__/artifact-review-document-routes.test.ts": 7252,
"server/src/__tests__/assets.test.ts": 9197,
"server/src/__tests__/auth-routes.test.ts": 1477,
"server/src/__tests__/auth-session-route.test.ts": 3303,
"server/src/__tests__/authz-company-access.test.ts": 1265,
"server/src/__tests__/authz-existence-oracle-guard.test.ts": 100,
"server/src/__tests__/authz-secret-context.test.ts": 100,
"server/src/__tests__/board-chat-route-feature-flag.test.ts": 895,
"server/src/__tests__/bootstrap-claim-routes.test.ts": 100,
"server/src/__tests__/browser-use-route-scope.test.ts": 136,
"server/src/__tests__/built-in-agent-routes.test.ts": 6740,
"server/src/__tests__/cases-routes.test.ts": 4496,
"server/src/__tests__/cli-auth-routes.test.ts": 17331,
"server/src/__tests__/cloud-routes.test.ts": 100,
"server/src/__tests__/companies-route-cross-company-authz.test.ts": 11340,
"server/src/__tests__/companies-route-path-guard.test.ts": 100,
"server/src/__tests__/company-branding-route.test.ts": 4804,
"server/src/__tests__/company-import-transfer-routes.test.ts": 1428,
"server/src/__tests__/company-portability-routes.test.ts": 2672,
"server/src/__tests__/company-portability.test.ts": 1315,
"server/src/__tests__/company-search-extract-routes.test.ts": 115,
"server/src/__tests__/company-search-rate-limit-routes.test.ts": 100,
"server/src/__tests__/company-skill-policy-routes.test.ts": 626,
"server/src/__tests__/company-skills-import-authz-routes.test.ts": 1524,
"server/src/__tests__/company-skills-routes.test.ts": 3409,
"server/src/__tests__/company-user-directory-route.test.ts": 100,
"server/src/__tests__/cost-data-quality-routes.test.ts": 100,
"server/src/__tests__/costs-service.test.ts": 1070,
"server/src/__tests__/decision-queues-routes.test.ts": 2092,
"server/src/__tests__/document-annotation-routes.test.ts": 8418,
"server/src/__tests__/email-routes-provider-errors.test.ts": 100,
"server/src/__tests__/environment-custom-image-routes.test.ts": 100,
"server/src/__tests__/environment-instance-routes.test.ts": 100,
"server/src/__tests__/environment-routes.test.ts": 198,
"server/src/__tests__/environment-selection-route-guards.test.ts": 119,
"server/src/__tests__/execution-workspace-runtime-lease-route.test.ts": 2694,
"server/src/__tests__/execution-workspaces-routes.test.ts": 100,
"server/src/__tests__/express5-auth-wildcard.test.ts": 100,
"server/src/__tests__/external-object-routes.test.ts": 4096,
"server/src/__tests__/folders-routes.test.ts": 1358,
"server/src/__tests__/health-dev-server-token.test.ts": 104,
"server/src/__tests__/health.test.ts": 100,
"server/src/__tests__/heartbeat-dependency-scheduling.test.ts": 27885,
"server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts": 47816,
"server/src/__tests__/heartbeat-process-recovery.test.ts": 324638,
"server/src/__tests__/heartbeat-run-privacy-routes.test.ts": 4967,
"server/src/__tests__/inbox-agent-policy-routes.test.ts": 831,
"server/src/__tests__/inbox-archive-routes.test.ts": 2794,
"server/src/__tests__/instance-database-backups-routes.test.ts": 100,
"server/src/__tests__/instance-settings-routes.test.ts": 3161,
"server/src/__tests__/invite-accept-existing-member.test.ts": 100,
"server/src/__tests__/invite-accept-gateway-defaults.test.ts": 584,
"server/src/__tests__/invite-accept-replay.test.ts": 100,
"server/src/__tests__/invite-create-route.test.ts": 7648,
"server/src/__tests__/invite-expiry.test.ts": 100,
"server/src/__tests__/invite-join-manager.test.ts": 100,
"server/src/__tests__/invite-list-route.test.ts": 457,
"server/src/__tests__/invite-logo-route.test.ts": 100,
"server/src/__tests__/invite-onboarding-text.test.ts": 100,
"server/src/__tests__/invite-rate-limit-route.test.ts": 5390,
"server/src/__tests__/invite-summary-route.test.ts": 6262,
"server/src/__tests__/invite-test-resolution-route.test.ts": 4135,
"server/src/__tests__/invite-url-public-base-url.test.ts": 4075,
"server/src/__tests__/issue-access-grants-routes.test.ts": 4910,
"server/src/__tests__/issue-activity-events-routes.test.ts": 9271,
"server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts": 9524,
"server/src/__tests__/issue-assigned-backlog-contract-routes.test.ts": 4067,
"server/src/__tests__/issue-assignee-invokability-routes.test.ts": 125,
"server/src/__tests__/issue-attachment-routes.test.ts": 4226,
"server/src/__tests__/issue-blocker-diagnostics-routes.test.ts": 1268,
"server/src/__tests__/issue-closed-workspace-routes.test.ts": 4032,
"server/src/__tests__/issue-comment-attribution-audit-routes.test.ts": 4466,
"server/src/__tests__/issue-comment-cancel-routes.test.ts": 3762,
"server/src/__tests__/issue-comment-reopen-routes.test.ts": 501,
"server/src/__tests__/issue-create-deduplication-routes.test.ts": 1646,
"server/src/__tests__/issue-created-from-routes.test.ts": 828,
"server/src/__tests__/issue-creation-source-routes.test.ts": 1836,
"server/src/__tests__/issue-dependency-wakeups-routes.test.ts": 8947,
"server/src/__tests__/issue-document-restore-routes.test.ts": 7787,
"server/src/__tests__/issue-execution-policy-routes.test.ts": 3678,
"server/src/__tests__/issue-feedback-routes.test.ts": 3786,
"server/src/__tests__/issue-identifier-routes.test.ts": 622,
"server/src/__tests__/issue-list-assignee-filter-routes.test.ts": 6773,
"server/src/__tests__/issue-list-updatedsince-filter-routes.test.ts": 772,
"server/src/__tests__/issue-onboarding-first-task-routes.test.ts": 6124,
"server/src/__tests__/issue-queued-comments-routes.test.ts": 47964,
"server/src/__tests__/issue-scheduled-retry-routes.test.ts": 7804,
"server/src/__tests__/issue-stale-execution-lock-routes.test.ts": 2222,
"server/src/__tests__/issue-stalled-review-decision-routes.test.ts": 2218,
"server/src/__tests__/issue-status-version-route-service.test.ts": 881,
"server/src/__tests__/issue-subtree-diagnostics-routes.test.ts": 1299,
"server/src/__tests__/issue-summaries-routes.test.ts": 832,
"server/src/__tests__/issue-telemetry-routes.test.ts": 4919,
"server/src/__tests__/issue-thread-interaction-routes.test.ts": 14647,
"server/src/__tests__/issue-tree-control-routes.test.ts": 1004,
"server/src/__tests__/issue-update-comment-wakeup-routes.test.ts": 4204,
"server/src/__tests__/issue-wake-diagnostics-routes.test.ts": 1493,
"server/src/__tests__/issue-watchdogs-routes.test.ts": 6085,
"server/src/__tests__/issue-workspace-command-authz.test.ts": 4361,
"server/src/__tests__/authz-secret-context.test.ts": 1282,
"server/src/__tests__/board-chat-route-feature-flag.test.ts": 881,
"server/src/__tests__/bootstrap-claim-routes.test.ts": 3293,
"server/src/__tests__/browser-use-route-scope.test.ts": 3223,
"server/src/__tests__/built-in-agent-routes.test.ts": 6590,
"server/src/__tests__/cases-routes.test.ts": 26922,
"server/src/__tests__/cli-auth-routes.test.ts": 17761,
"server/src/__tests__/cloud-routes.test.ts": 1476,
"server/src/__tests__/companies-route-cross-company-authz.test.ts": 11710,
"server/src/__tests__/companies-route-path-guard.test.ts": 1523,
"server/src/__tests__/company-branding-route.test.ts": 4996,
"server/src/__tests__/company-import-transfer-routes.test.ts": 2992,
"server/src/__tests__/company-portability-routes.test.ts": 3065,
"server/src/__tests__/company-portability.test.ts": 9374,
"server/src/__tests__/company-search-extract-routes.test.ts": 4311,
"server/src/__tests__/company-search-rate-limit-routes.test.ts": 4322,
"server/src/__tests__/company-skill-policy-routes.test.ts": 3653,
"server/src/__tests__/company-skills-import-authz-routes.test.ts": 5508,
"server/src/__tests__/company-skills-routes.test.ts": 4067,
"server/src/__tests__/company-user-directory-route.test.ts": 3338,
"server/src/__tests__/cost-data-quality-routes.test.ts": 3245,
"server/src/__tests__/costs-service.test.ts": 4246,
"server/src/__tests__/decision-queues-routes.test.ts": 5259,
"server/src/__tests__/document-annotation-routes.test.ts": 8439,
"server/src/__tests__/email-routes-provider-errors.test.ts": 3201,
"server/src/__tests__/environment-custom-image-routes.test.ts": 4007,
"server/src/__tests__/environment-instance-routes.test.ts": 4041,
"server/src/__tests__/environment-routes.test.ts": 3697,
"server/src/__tests__/environment-selection-route-guards.test.ts": 4164,
"server/src/__tests__/execution-workspace-runtime-lease-route.test.ts": 4063,
"server/src/__tests__/execution-workspaces-routes.test.ts": 3474,
"server/src/__tests__/express5-auth-wildcard.test.ts": 135,
"server/src/__tests__/external-object-routes.test.ts": 3862,
"server/src/__tests__/folders-routes.test.ts": 1474,
"server/src/__tests__/health-dev-server-token.test.ts": 3152,
"server/src/__tests__/health.test.ts": 3543,
"server/src/__tests__/heartbeat-dependency-scheduling.test.ts": 31376,
"server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts": 49291,
"server/src/__tests__/heartbeat-process-recovery.test.ts": 362170,
"server/src/__tests__/heartbeat-run-privacy-routes.test.ts": 9796,
"server/src/__tests__/inbox-agent-policy-routes.test.ts": 5019,
"server/src/__tests__/inbox-archive-routes.test.ts": 7129,
"server/src/__tests__/instance-database-backups-routes.test.ts": 1474,
"server/src/__tests__/instance-settings-routes.test.ts": 3499,
"server/src/__tests__/invite-accept-existing-member.test.ts": 3537,
"server/src/__tests__/invite-accept-gateway-defaults.test.ts": 4795,
"server/src/__tests__/invite-accept-replay.test.ts": 4065,
"server/src/__tests__/invite-create-route.test.ts": 7645,
"server/src/__tests__/invite-expiry.test.ts": 4250,
"server/src/__tests__/invite-join-manager.test.ts": 4317,
"server/src/__tests__/invite-list-route.test.ts": 3634,
"server/src/__tests__/invite-logo-route.test.ts": 4177,
"server/src/__tests__/invite-onboarding-text.test.ts": 4032,
"server/src/__tests__/invite-rate-limit-route.test.ts": 6008,
"server/src/__tests__/invite-summary-route.test.ts": 6090,
"server/src/__tests__/invite-test-resolution-route.test.ts": 4352,
"server/src/__tests__/invite-url-public-base-url.test.ts": 4379,
"server/src/__tests__/issue-access-grants-routes.test.ts": 6138,
"server/src/__tests__/issue-activity-events-routes.test.ts": 10999,
"server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts": 14786,
"server/src/__tests__/issue-assigned-backlog-contract-routes.test.ts": 3922,
"server/src/__tests__/issue-assignee-invokability-routes.test.ts": 3837,
"server/src/__tests__/issue-attachment-routes.test.ts": 4351,
"server/src/__tests__/issue-blocker-diagnostics-routes.test.ts": 5648,
"server/src/__tests__/issue-closed-workspace-routes.test.ts": 4191,
"server/src/__tests__/issue-comment-attribution-audit-routes.test.ts": 5562,
"server/src/__tests__/issue-comment-cancel-routes.test.ts": 4176,
"server/src/__tests__/issue-comment-reopen-routes.test.ts": 4355,
"server/src/__tests__/issue-create-deduplication-routes.test.ts": 5953,
"server/src/__tests__/issue-created-from-routes.test.ts": 4860,
"server/src/__tests__/issue-creation-source-routes.test.ts": 6117,
"server/src/__tests__/issue-dependency-wakeups-routes.test.ts": 8961,
"server/src/__tests__/issue-document-restore-routes.test.ts": 7554,
"server/src/__tests__/issue-execution-policy-routes.test.ts": 4791,
"server/src/__tests__/issue-feedback-routes.test.ts": 3966,
"server/src/__tests__/issue-identifier-routes.test.ts": 4818,
"server/src/__tests__/issue-list-assignee-filter-routes.test.ts": 10945,
"server/src/__tests__/issue-list-updatedsince-filter-routes.test.ts": 5143,
"server/src/__tests__/issue-onboarding-first-task-routes.test.ts": 10593,
"server/src/__tests__/issue-queued-comments-routes.test.ts": 53688,
"server/src/__tests__/issue-scheduled-retry-routes.test.ts": 12618,
"server/src/__tests__/issue-stale-execution-lock-routes.test.ts": 6168,
"server/src/__tests__/issue-stalled-review-decision-routes.test.ts": 6305,
"server/src/__tests__/issue-status-version-route-service.test.ts": 5153,
"server/src/__tests__/issue-subtree-diagnostics-routes.test.ts": 5592,
"server/src/__tests__/issue-summaries-routes.test.ts": 5080,
"server/src/__tests__/issue-telemetry-routes.test.ts": 4631,
"server/src/__tests__/issue-thread-interaction-routes.test.ts": 15508,
"server/src/__tests__/issue-tree-control-routes.test.ts": 1520,
"server/src/__tests__/issue-update-comment-wakeup-routes.test.ts": 4994,
"server/src/__tests__/issue-wake-diagnostics-routes.test.ts": 5428,
"server/src/__tests__/issue-watchdogs-routes.test.ts": 10901,
"server/src/__tests__/issue-workspace-command-authz.test.ts": 4648,
"server/src/__tests__/issues-checkout-wakeup.test.ts": 100,
"server/src/__tests__/issues-goal-context-routes.test.ts": 100,
"server/src/__tests__/issues-service.test.ts": 14248,
"server/src/__tests__/llms-routes.test.ts": 1700,
"server/src/__tests__/low-trust-red-team-routes.test.ts": 21637,
"server/src/__tests__/managed-agent-profile-routes-authz.test.ts": 100,
"server/src/__tests__/multilingual-issues-routes.test.ts": 990,
"server/src/__tests__/muse-qualification-routes.test.ts": 100,
"server/src/__tests__/onboarding-seed-route.test.ts": 2184,
"server/src/__tests__/openapi-routes.test.ts": 808,
"server/src/__tests__/openclaw-invite-prompt-route.test.ts": 100,
"server/src/__tests__/opencode-local-adapter-environment.test.ts": 6051,
"server/src/__tests__/openrouter-attribution.test.ts": 100,
"server/src/__tests__/permissions-upgrade-boundary-routes.test.ts": 1292,
"server/src/__tests__/plugin-install-route-security.test.ts": 3548,
"server/src/__tests__/plugin-routes-authz.test.ts": 3421,
"server/src/__tests__/plugin-scoped-api-routes.test.ts": 2549,
"server/src/__tests__/project-goal-telemetry-routes.test.ts": 3418,
"server/src/__tests__/project-routes-env.test.ts": 4210,
"server/src/__tests__/project-workspace-managed-sandbox-routes.test.ts": 6883,
"server/src/__tests__/projects-list-archived-routes.test.ts": 1152,
"server/src/__tests__/redaction.test.ts": 100,
"server/src/__tests__/resource-memberships-routes.test.ts": 1629,
"server/src/__tests__/routine-document-annotation-routes.test.ts": 2027,
"server/src/__tests__/routines-e2e.test.ts": 17403,
"server/src/__tests__/routines-routes.test.ts": 6826,
"server/src/__tests__/secret-proposals-routes.test.ts": 9151,
"server/src/__tests__/secrets-routes-claude-oauth-service.test.ts": 1360,
"server/src/__tests__/secrets-routes.test.ts": 198,
"server/src/__tests__/sidebar-preferences-routes.test.ts": 3043,
"server/src/__tests__/subscription-routes.test.ts": 100,
"server/src/__tests__/summary-slot-routes.test.ts": 5832,
"server/src/__tests__/task-connector-read-routes.test.ts": 122,
"server/src/__tests__/task-title-routes.test.ts": 4602,
"server/src/__tests__/teams-catalog-routes.test.ts": 3849,
"server/src/__tests__/user-profile-routes.test.ts": 1209,
"server/src/__tests__/issues-goal-context-routes.test.ts": 4119,
"server/src/__tests__/issues-service.test.ts": 26999,
"server/src/__tests__/llms-routes.test.ts": 1838,
"server/src/__tests__/low-trust-red-team-routes.test.ts": 25961,
"server/src/__tests__/maintenance-server-routes.test.ts": 511763,
"server/src/__tests__/managed-agent-profile-routes-authz.test.ts": 3182,
"server/src/__tests__/multilingual-issues-routes.test.ts": 5137,
"server/src/__tests__/muse-qualification-routes.test.ts": 3072,
"server/src/__tests__/onboarding-seed-route.test.ts": 5499,
"server/src/__tests__/openapi-routes.test.ts": 2400,
"server/src/__tests__/openclaw-invite-prompt-route.test.ts": 3380,
"server/src/__tests__/opencode-local-adapter-environment.test.ts": 6840,
"server/src/__tests__/openrouter-attribution.test.ts": 778,
"server/src/__tests__/permissions-upgrade-boundary-routes.test.ts": 5309,
"server/src/__tests__/plugin-install-route-security.test.ts": 4283,
"server/src/__tests__/plugin-routes-authz.test.ts": 3539,
"server/src/__tests__/plugin-scoped-api-routes.test.ts": 3119,
"server/src/__tests__/project-goal-telemetry-routes.test.ts": 3626,
"server/src/__tests__/project-routes-env.test.ts": 4082,
"server/src/__tests__/project-workspace-managed-sandbox-routes.test.ts": 6431,
"server/src/__tests__/projects-list-archived-routes.test.ts": 5044,
"server/src/__tests__/redaction.test.ts": 136,
"server/src/__tests__/resource-memberships-routes.test.ts": 5339,
"server/src/__tests__/routine-document-annotation-routes.test.ts": 2206,
"server/src/__tests__/routines-e2e.test.ts": 18070,
"server/src/__tests__/routines-routes.test.ts": 6367,
"server/src/__tests__/secret-proposals-routes.test.ts": 16478,
"server/src/__tests__/secrets-routes-claude-oauth-service.test.ts": 4530,
"server/src/__tests__/secrets-routes.test.ts": 4522,
"server/src/__tests__/sidebar-preferences-routes.test.ts": 3352,
"server/src/__tests__/subscription-routes.test.ts": 3167,
"server/src/__tests__/summary-slot-routes.test.ts": 5656,
"server/src/__tests__/task-connector-read-routes.test.ts": 6058,
"server/src/__tests__/task-title-routes.test.ts": 13080,
"server/src/__tests__/teams-catalog-routes.test.ts": 4167,
"server/src/__tests__/user-profile-routes.test.ts": 2245,
"server/src/__tests__/workspace-command-authz.test.ts": 100,
"server/src/__tests__/workspace-preferences-routes.test.ts": 100,
"server/src/__tests__/workspace-restore-repair-route.test.ts": 1666,
"server/src/__tests__/workspace-runtime-routes-authz.test.ts": 3430,
"server/src/__tests__/workspace-runtime-service-authz.test.ts": 926
"server/src/__tests__/workspace-preferences-routes.test.ts": 1477,
"server/src/__tests__/workspace-restore-repair-route.test.ts": 6027,
"server/src/__tests__/workspace-runtime-routes-authz.test.ts": 3659,
"server/src/__tests__/workspace-runtime-service-authz.test.ts": 2230
}
}
@@ -58,8 +58,10 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000;
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "board", membershipRole: "owner", status: "active" });
await db.insert(agents).values({ id: agentId, companyId, name: "Origin", role: "engineer", status: "active" });
await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`)));
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true });
vi.setSystemTime(new Date("2026-11-01T12:00:00Z"));
// Ordinary issue creation uses database timestamp defaults. Start the fake
// clock there too; a fixed calendar date eventually trails those deadlines.
const [clock] = await db.execute<{ now: string }>(sql`select (extract(epoch from clock_timestamp()) * 1000)::text as now`);
vi.useFakeTimers({ now: Number(clock.now), toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true });
});
afterEach(async () => {
await coordinator?.stop(); coordinator = undefined;
@@ -0,0 +1,253 @@
import { recoveryPostgresSupport, startRecoveryPostgres } from "./recovery-postgres.js";
import { randomUUID } from "node:crypto";
import { existsSync } from "node:fs";
import { mkdtemp, rm } from "node:fs/promises";
import { createServer, type Server } from "node:http";
import { tmpdir } from "node:os";
import { resolve } from "node:path";
import { eq } from "drizzle-orm";
import { agents, companies, closeRegisteredClients, createDb, heartbeatRuns, issues, nativeRunFinalizations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
import { afterAll, afterEach, beforeAll, describe, it } from "vitest";
import { defaultCapabilityRunnerdBinary } from "../../vendor/paperclip-runner/index.js";
import { startEmbeddedPostgresTestDatabase } from "./embedded-postgres.js";
import { registerRunnerPrpAuthority, runnerPrpWebSocketInternals, setupRunnerPrpWebSocketServer } from "../../realtime/runner-prp-ws.js";
import { readProcessStartedAt } from "../../services/hot-restart.js";
import { prepareNativeHeartbeatRun } from "../../services/native-runtime/prepare-native-run.js";
import { createIsolatedTestHome } from "./isolated-test-home.js";
import { type NativeControllerIdentity } from "../../services/native-runtime/native-restart-recovery.js";
import { waitForPendingRunFailureReports } from "../../services/run-failure-report.js";
// An explicitly supplied disposable database also permits qualification on hosts
// where native Postgres cannot allocate semaphores. Never fall back silently.
export const describeEmbeddedPostgres: typeof describe.skip = recoveryPostgresSupport.supported ? describe : describe.skip;
export const fakeCodexAppServer = resolve(
import.meta.dirname,
"../../../../packages/paperclip-runner/runner/target/debug/fake-codex-app-server",
);
const binariesAvailable =
existsSync(defaultCapabilityRunnerdBinary()) && existsSync(fakeCodexAppServer);
export const realProcessIt: typeof it.skip = binariesAvailable ? it : it.skip;
async function closeServer(server: Server | null): Promise<void> {
if (!server) return;
server.closeAllConnections();
if (!server.listening) return;
await new Promise<void>((resolveClose) => server.close(() => resolveClose()));
}
export function processAlive(pid: number | null): boolean {
if (!pid) return false;
try {
process.kill(pid, 0);
return true;
} catch {
return false;
}
}
export async function waitForCondition(
description: string,
condition: () => boolean | Promise<boolean>,
timeoutMs = 10_000,
): Promise<void> {
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
if (await condition()) return;
await new Promise<void>((resolveWait) => setTimeout(resolveWait, 25));
}
throw new Error(`Timed out waiting for ${description}`);
}
export async function stopOwnedProcessGroup(
pid: number | null,
expectedStateRoot: string,
): Promise<void> {
if (!pid || !processAlive(pid)) return;
const command = await import("node:child_process").then(
({ execFileSync }) =>
execFileSync("ps", ["-p", String(pid), "-o", "command="], {
encoding: "utf8",
}).trim(),
);
if (!command.includes(expectedStateRoot)) {
throw new Error(`Refusing to signal process ${pid}; ownership changed`);
}
if (process.platform !== "win32") process.kill(-pid, "SIGKILL");
else process.kill(pid, "SIGKILL");
await waitForCondition(`owned process ${pid} to exit`, () => !processAlive(pid));
}
export function nativeRestartFixture() {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let runtimeRoot: string;
let paperclipHome: string;
let server: Server | null = null;
let apiUrl: string;
const originalPaperclipHome = process.env.PAPERCLIP_HOME;
let isolatedHome: ReturnType<typeof createIsolatedTestHome>;
const companyId = randomUUID();
const agentId = randomUUID();
let successor: NativeControllerIdentity;
beforeAll(async () => {
temporary = await startRecoveryPostgres("native-runner-restart-recovery-");
runtimeRoot = await mkdtemp(resolve(tmpdir(), "native-restart-runtime-"));
isolatedHome = createIsolatedTestHome();
paperclipHome = isolatedHome.home;
process.env.PAPERCLIP_HOME = paperclipHome;
const controllerStartedAt = await readProcessStartedAt(process.pid);
successor = {
bootId: randomUUID(),
pid: process.pid,
processStartedAt: controllerStartedAt
? new Date(controllerStartedAt)
: new Date(),
};
server = createServer();
await new Promise<void>((resolveListen) =>
server!.listen(0, "127.0.0.1", resolveListen),
);
const address = server.address();
if (!address || typeof address === "string") {
throw new Error("Expected restart recovery TCP listener");
}
apiUrl = `http://127.0.0.1:${address.port}`;
setupRunnerPrpWebSocketServer(server, { apiUrl });
const db = createDb(temporary.connectionString);
await db.insert(companies).values({
id: companyId,
name: "Native restart recovery",
issuePrefix: "NRR",
requireBoardApprovalForNewAgents: false,
});
await db.insert(agents).values({
id: agentId,
companyId,
name: "Native restart runner",
role: "engineer",
status: "active",
adapterType: "paperclip_runner",
adapterConfig: { provider: "codex" },
runtimeConfig: {},
permissions: {},
});
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterEach(async () => {
// Each case creates its own clients. With the long waits in other files,
// idle reaping no longer keeps their total below Postgres's connection cap.
await waitForPendingRunFailureReports();
if (temporary) await closeRegisteredClients(temporary.connectionString, { scope: "database" });
});
afterAll(async () => {
runnerPrpWebSocketInternals.resetForTests();
await closeServer(server);
await temporary?.cleanup();
if (runtimeRoot) await rm(runtimeRoot, { recursive: true, force: true });
isolatedHome?.cleanup();
if (originalPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME;
else process.env.PAPERCLIP_HOME = originalPaperclipHome;
});
async function seedRun(
label: string,
existingDb?: ReturnType<typeof createDb>,
) {
const db = existingDb ?? createDb(temporary.connectionString);
const issueId = randomUUID();
const runId = randomUUID();
await db.insert(issues).values({
id: issueId,
companyId,
identifier: `NRR-${label}`,
title: `Recover ${label}`,
status: "in_progress",
priority: "medium",
workMode: "standard",
assigneeAgentId: agentId,
});
const [run] = await db
.insert(heartbeatRuns)
.values({
id: runId,
companyId,
agentId,
status: "running",
runtimeMode: "native",
nativeIssueId: issueId,
invocationSource: "assignment",
triggerDetail: "system",
contextSnapshot: { issueId },
})
.returning();
if (!run) throw new Error("Failed to seed native restart run");
await db
.update(issues)
.set({ executionRunId: runId })
.where(eq(issues.id, issueId));
const native = await prepareNativeHeartbeatRun({
db,
run,
issue: {
id: issueId,
title: `Recover ${label}`,
description: null,
reviewPolicy: null,
},
environmentLeaseId: `lease-${label}`,
});
await db.insert(nativeRunFinalizations).values({
runId,
companyId,
issueId,
phase: "observed",
});
const [preparedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
return { db, issueId, runId, native, preparedProfile: preparedRun.runnerProfileJson };
}
function transportOptions(
fixture: Awaited<ReturnType<typeof seedRun>>,
stateDirectory: string,
) {
return {
runnerBinary: defaultCapabilityRunnerdBinary(),
codexCommand: fakeCodexAppServer,
codexArgs: [
"--state-file",
resolve(stateDirectory, "fake-codex-state.json"),
"--call-log",
resolve(stateDirectory, "fake-codex-calls.log"),
],
stateDirectory,
lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 },
prpIdentity: {
runnerInstanceId: fixture.native.runnerInstanceId,
environmentLeaseId: fixture.native.environmentLeaseId,
runId: fixture.runId,
normalizedSessionId: fixture.native.normalizedSessionId,
turnId: fixture.native.turnId,
itemId: fixture.native.itemId,
},
controlPlaneRegistration: (authority: Parameters<typeof registerRunnerPrpAuthority>[0]["authority"]) =>
registerRunnerPrpAuthority({
companyId,
runId: fixture.runId,
authority,
}),
};
}
return { seedRun, transportOptions, get current() { return { temporary, runtimeRoot, paperclipHome, apiUrl, companyId, agentId, successor }; } };
}
export function remotePreparedWorkspace() {
return { schema: "paperclip.native-workspace-sync/v3", state: "prepared",
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null,
workspaceId: randomUUID(), leaseId: randomUUID(), providerLeaseId: randomUUID(),
remoteCwd: "/workspace", resourceDisposition: "destroy" };
}
@@ -0,0 +1,50 @@
import { eq } from "drizzle-orm";
import { expect, it, vi } from "vitest";
import { heartbeatRuns, issues } from "@paperclipai/db";
import { issueService } from "../services/issues.js";
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
describeIssueRecovery("issue recovery production producers and assignments", () => {
const state = issueRecoveryFixture();
const { start, fixture, settled, quiet, finish, completed } = state;
it("dispatches an ordinary new assignment after commit and returns to quiet after completion", async () => {
const f = await fixture(); await start(); await quiet();
const issue = await issueService(state.db).create(f.companyId, { title: "Recover assignment", status: "todo", assigneeAgentId: f.agentId });
await completed(issue.id);
await finish(issue.id); await quiet();
}, 90_000);
it("fences nested assignment writes until outer commit and ignores rollback", async () => {
const f = await fixture(); await start(); await quiet();
let committedId = "";
await state.db.transaction(async outer => {
await outer.transaction(async nested => {
const issue = await issueService(state.db).create(f.companyId, { title: "Nested commit", status: "todo", assigneeAgentId: f.agentId }, nested);
committedId = issue.id;
});
await new Promise(resolve => setTimeout(resolve, 100));
expect(await state.db.select().from(heartbeatRuns)).toHaveLength(0);
});
await completed(committedId); await finish(committedId); await settled();
await expect(state.db.transaction(async outer => {
await outer.transaction(async nested => {
await issueService(state.db).create(f.companyId, { title: "Nested rollback", status: "todo", assigneeAgentId: f.agentId }, nested);
});
throw new Error("rollback control");
})).rejects.toThrow("rollback control");
await quiet();
expect(await state.db.select().from(issues).where(eq(issues.title, "Nested rollback"))).toHaveLength(0);
}, 90_000);
it("retains the first committed assignment during drain and executes on resume", async () => {
const f = await fixture(); await start(); await quiet(); state.draining = true;
const issue = await issueService(state.db).create(f.companyId, { title: "Drained assignment", status: "todo", assigneeAgentId: f.agentId });
await new Promise(resolve => setTimeout(resolve, 100));
const reads = vi.spyOn(state.db, "select");
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
try { await vi.advanceTimersByTimeAsync(90_000); expect(reads).not.toHaveBeenCalled(); }
finally { vi.useRealTimers(); reads.mockRestore(); }
expect(await state.db.select().from(heartbeatRuns)).toHaveLength(0);
state.draining = false; state.worker.wake(); state.dispatch.wake();
await completed(issue.id); await finish(issue.id); await quiet();
}, 90_000);
});
@@ -0,0 +1,69 @@
import { randomUUID } from "node:crypto";
import { and, eq } from "drizzle-orm";
import { expect, it, vi } from "vitest";
import { heartbeatRuns, issues, issueComments } from "@paperclipai/db";
import { instanceSettingsService } from "../services/instance-settings.js";
import { undeliveredConversationComments } from "../services/agent-conversations.js";
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js";
import { issueService } from "../services/issues.js";
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
describeIssueRecovery("issue recovery production producers and conversations", () => {
const state = issueRecoveryFixture();
const { quiet, completed, savedConversation } = state;
it("dispatches a saved conversation only from the committed Agent Chat enable transition", async () => {
const f = await savedConversation();
expect(await undeliveredConversationComments(state.db, f.companyId, f.chat.id)).toHaveLength(1);
await f.settings.updateExperimental({ enableAgentChat: true });
expect(state.coordinator.nextWakeAt()).not.toBeNull();
await vi.waitFor(async () => expect(await state.db.select().from(heartbeatRuns)
.where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1), { timeout: 40_000 });
const run = await completed(f.chat.id);
expect(f.replies).toEqual([run.id]); expect(f.failures).toEqual([]);
expect(await undeliveredConversationComments(state.db, f.companyId, f.chat.id)).toEqual([]);
expect((await state.db.select().from(issues).where(eq(issues.id, f.chat.id)))[0])
.toMatchObject({ conversationState: "waiting", executionRunId: null });
expect(await state.db.select().from(issueComments).where(and(eq(issueComments.createdByRunId, run.id),
eq(issueComments.authorAgentId, f.agentId)))).toHaveLength(1);
await quiet();
let hints = 0;
const stop = subscribeDeliveryWork(state.db, DELIVERY_QUEUES.strandedAssignments, () => { hints++; });
try { await f.settings.updateExperimental({ enableAgentChat: true }); expect(hints).toBe(0); await quiet(); }
finally { stop(); }
expect(f.replies).toEqual([run.id]);
});
it("keeps disabled saved conversations parked after rollback, unrelated flags and unchanged values", async () => {
const f = await savedConversation();
await expect(state.db.transaction(async tx => {
await instanceSettingsService(tx as unknown as typeof state.db).updateExperimental({ enableAgentChat: true });
throw new Error("chat flag rollback");
})).rejects.toThrow("chat flag rollback");
await quiet();
expect((await f.settings.getExperimental()).enableAgentChat).toBe(false);
let hints = 0;
const stop = subscribeDeliveryWork(state.db, DELIVERY_QUEUES.strandedAssignments, () => { hints++; });
try {
await f.settings.updateExperimental({ enableConferenceRoomChat: true });
await f.settings.updateExperimental({ enableAgentChat: false });
expect(hints).toBe(0); await quiet();
} finally { stop(); }
expect(f.replies).toEqual([]); expect(f.failures).toEqual([]);
expect(await state.db.select().from(heartbeatRuns)).toEqual([]);
expect(await undeliveredConversationComments(state.db, f.companyId, f.chat.id)).toHaveLength(1);
});
it("does not dispatch an accepted saved message after Agent Chat is disabled", async () => {
const f = await savedConversation(false);
await f.settings.updateExperimental({ enableAgentChat: true }); await quiet();
state.draining = true;
await issueService(state.db).addComment(f.chat.id, "Retain while disabled", { userId: "recovery-user" }, { clientRequestId: randomUUID() });
let hints = 0;
const stop = subscribeDeliveryWork(state.db, DELIVERY_QUEUES.strandedAssignments, () => { hints++; });
try { await f.settings.updateExperimental({ enableAgentChat: false }); expect(hints).toBe(0); }
finally { stop(); }
state.draining = false; await quiet();
expect(f.replies).toEqual([]); expect(f.failures).toEqual([]);
expect(await state.db.select().from(heartbeatRuns)).toEqual([]);
expect(await undeliveredConversationComments(state.db, f.companyId, f.chat.id)).toHaveLength(1);
});
});
@@ -1,325 +1,44 @@
import { createServer } from "node:http";
import { setImmediate as immediate } from "node:timers/promises";
import { idleWorkSnapshot } from "../services/task-admission.js";
import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { agents, authUsers, companies, companyMemberships, createDb, heartbeatRuns, issueRecoveryActions, issueRelations, issues, issueWatchdogs, issueComments } from "@paperclipai/db";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { expect, it, vi } from "vitest";
import { agents, heartbeatRuns, issueRecoveryActions, issueRelations, issues, issueWatchdogs } from "@paperclipai/db";
import { heartbeatService } from "../services/heartbeat.js";
import { instanceSettingsService } from "../services/instance-settings.js";
import { undeliveredConversationComments } from "../services/agent-conversations.js";
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js";
import { issueService } from "../services/issues.js";
import { issueRecoveryActionService } from "../services/issue-recovery-actions.js";
import { registerIssueRecoveryWork } from "../services/recovery/work.js";
import { notifyIssueRecovery } from "../services/issue-recovery-notifications.js";
import { taskWatchdogService } from "../services/task-watchdogs.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
vi.setConfig({ testTimeout: 90_000 });
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_ISSUE_RECOVERY_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
describe.skipIf(!support.supported)("issue recovery production producers and deadlines", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: ReturnType<typeof createDb>;
let heartbeat: ReturnType<typeof heartbeatService>;
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator>;
let worker: ReturnType<ReturnType<typeof heartbeatService>["registerIssueStateRecoveryWork"]>;
let dispatch: ReturnType<ReturnType<typeof heartbeatService>["registerRunDispatchWork"]>;
let cancellation: ReturnType<ReturnType<typeof heartbeatService>["registerScheduledRetryCancellationWork"]>;
let draining = false;
const tasks = new Map<string, { nextRunAt?: () => number | null; hasPending: () => Promise<boolean> }>();
const errors: unknown[] = [];
const passes = new Set<Promise<unknown>>();
const extraStops: Array<() => Promise<void>> = [];
const extraDeadlines: Array<() => number | null> = [];
beforeAll(async () => {
if (!external) database = await startEmbeddedPostgresTestDatabase("issue-recovery-work-");
db = createDb(external ?? database!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`)));
}, 90_000);
beforeEach(async () => {
await db.execute(sql`TRUNCATE companies CASCADE`);
heartbeat = heartbeatService(db); draining = false; errors.length = 0;
});
afterEach(async () => {
for (const stop of extraStops.splice(0)) await stop();
extraDeadlines.length = 0;
await worker?.stop(); await dispatch?.stop(); await cancellation?.stop(); await coordinator?.stop();
await heartbeat.drainActiveRunExecutions();
vi.useRealTimers(); vi.restoreAllMocks();
expect(errors).toEqual([]);
});
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); });
async function start() {
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true,
onError: error => errors.push(error) });
tasks.clear();
const register = coordinator.register.bind(coordinator);
coordinator.register = (queue, task) => {
tasks.set(queue, task);
return register(queue, { ...task, run: signal => {
const work = task.run(signal); passes.add(work);
void work.then(() => passes.delete(work), () => passes.delete(work));
return work;
} });
};
worker = heartbeat.registerIssueStateRecoveryWork(coordinator, () => !draining);
dispatch = heartbeat.registerRunDispatchWork(coordinator, () => !draining);
cancellation = heartbeat.registerScheduledRetryCancellationWork(coordinator, () => !draining);
await Promise.all([worker.ready, dispatch.ready, cancellation.ready]);
}
async function compete() {
const other = heartbeatService(db);
const competing = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const register = competing.register.bind(competing);
competing.register = (queue, task) => register(queue, { ...task, run: signal => {
const pass = task.run(signal); passes.add(pass);
void pass.then(() => passes.delete(pass), () => passes.delete(pass));
return pass;
} });
const work = other.registerIssueStateRecoveryWork(competing, () => true);
extraStops.push(() => competing.stop()); extraDeadlines.push(competing.nextWakeAt); await work.ready;
}
async function fixture() {
const companyId = randomUUID();
await db.insert(authUsers).values({ id: "recovery-user", name: "Recovery operator", email: "[email protected]", createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing();
await db.insert(companies).values({ id: companyId, name: "Recovery", issuePrefix: `R${companyId.slice(0, 7)}`,
requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: "recovery-user" });
await db.insert(companyMemberships).values({ companyId, principalId: "recovery-user", principalType: "user", status: "active", membershipRole: "owner" });
const [agent] = await db.insert(agents).values({ companyId, name: "Recovery process", role: "engineer", status: "idle",
adapterType: "process", adapterConfig: { command: process.execPath, args: ["-e", "console.log('recovery-proof')"], cwd: process.cwd() },
runtimeConfig: { heartbeat: { enabled: true, intervalSec: 0, wakeOnDemand: true } },
}).returning();
return { companyId, agentId: agent.id };
}
async function completingProvider(f: { companyId: string; agentId: string }, beforeComplete?: () => Promise<unknown>) {
const failures: unknown[] = [];
const listener = createServer(async (request, response) => {
try {
let body = "";
for await (const chunk of request) body += chunk;
const { runId } = JSON.parse(body) as { runId: string };
const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId),
eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.agentId, f.agentId)));
if (!run?.contextSnapshot?.issueId) throw new Error("Unknown controlled task run");
await beforeComplete?.();
await issueService(db).update(String(run.contextSnapshot.issueId), { status: "done" });
response.end("ok");
} catch (error) { failures.push(error); response.statusCode = 500; response.end("completion failed"); }
});
await new Promise<void>(resolve => listener.listen(0, "127.0.0.1", resolve));
extraStops.push(async () => { listener.closeAllConnections(); await new Promise<void>(resolve => listener.close(() => resolve())); });
const port = (listener.address() as { port: number }).port;
const script = `fetch("http://127.0.0.1:${port}/complete", {method:"POST", body:JSON.stringify({runId:process.env.PAPERCLIP_RUN_ID})}).then(async response=>{if(!response.ok)throw new Error(await response.text());console.log("recovery-proof")}).catch(error=>{console.error(error);process.exitCode=1})`;
await db.update(agents).set({ adapterConfig: { command: process.execPath, args: ["-e", script], cwd: process.cwd() } })
.where(eq(agents.id, f.agentId));
return failures;
}
async function settled() {
await vi.waitFor(async () => {
await Promise.allSettled([...passes]);
await heartbeat.drainActiveRunExecutions();
const wake = coordinator.nextWakeAt();
const detail = wake === null ? "" : JSON.stringify({ now: Date.now(), tasks: await Promise.all([...tasks].map(async ([queue, task]) => ({ queue, at: task.nextRunAt?.(), pending: await task.hasPending() }))),
runs: (await db.select().from(heartbeatRuns)).map(run => ({ id: run.id, status: run.status, error: run.errorCode, lease: run.controllerLeaseExpiresAt, reason: run.contextSnapshot?.wakeReason })) });
expect(wake, detail).toBeNull();
expect(extraDeadlines.map(next => next())).toEqual(extraDeadlines.map(() => null));
expect(idleWorkSnapshot().active).toBe(0);
}, { timeout: 40_000 });
}
async function quiet() {
await settled();
expect(await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.status} in ('queued', 'running')`)).toHaveLength(0);
const reads = vi.spyOn(db, "select"); const executes = vi.spyOn(db, "execute");
const selected = reads.mock.calls.length, executed = executes.mock.calls.length;
const statements: string[] = [], priorDebug = db.$client.options.debug;
db.$client.options.debug = (_connection, query) => { statements.push(query); };
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
try {
await vi.advanceTimersByTimeAsync(20 * 60_000);
expect(reads).toHaveBeenCalledTimes(selected); expect(executes).toHaveBeenCalledTimes(executed);
expect(statements).toEqual([]);
} finally { vi.useRealTimers(); db.$client.options.debug = priorDebug; reads.mockRestore(); executes.mockRestore(); }
}
async function finish(issueId: string) {
const retries = () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.status, "scheduled_retry"),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`));
const before = await retries();
await issueService(db).update(issueId, { status: "done" });
// Closing one blocker can legitimately start its dependent. Wait for this
// task's dispatch and execution; quiet() verifies the whole fixture later.
await vi.waitFor(async () => {
await Promise.allSettled([...passes]);
await heartbeat.drainActiveRunExecutions();
expect(await db.select().from(heartbeatRuns).where(and(
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,
sql`${heartbeatRuns.status} in ('queued', 'running')`,
))).toHaveLength(0);
}, { timeout: 15_000 });
const saved = [...new Map([...before, ...await retries()].map(run => [run.id, run])).values()];
if (saved.length > 0) {
const trace: unknown[] = [];
await vi.waitFor(async () => {
await Promise.allSettled([...passes]);
await heartbeat.drainActiveRunExecutions();
const remaining = await retries();
if (remaining.length > 0) trace.push({ now: Date.now(),
deadlines: [coordinator.nextWakeAt(), ...extraDeadlines.map(next => next())],
remaining: remaining.map(run => ({ id: run.id, at: run.scheduledRetryAt })) });
expect(remaining, JSON.stringify(trace)).toHaveLength(0);
}, { timeout: Math.max(...saved.map(run => run.scheduledRetryAt!.getTime())) - Date.now() + 30_000 });
for (const retry of saved) {
const [settledRetry] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, retry.id));
expect(settledRetry.status).toBe("cancelled");
}
}
}
async function completed(issueId: string) {
let runId: string | undefined;
await vi.waitFor(async () => {
const runs = await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`);
const run = runs.find(row => row.status === "running" || row.status === "succeeded");
expect(run).toBeDefined(); runId = run!.id;
}, { timeout: 15_000 });
await vi.waitFor(async () => {
await heartbeat.drainActiveRunExecutions();
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId!));
expect(current.status, JSON.stringify({ error: current.error, code: current.errorCode, stderr: current.stderrExcerpt })).toBe("succeeded");
}, { timeout: 15_000 });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId!));
expect(run.stdoutExcerpt).toContain("recovery-proof");
return run;
}
async function savedConversation(withMessage = true) {
const f = await fixture(), settings = instanceSettingsService(db);
await settings.updateExperimental({ enableAgentChat: true, enableConferenceRoomChat: false });
const replies: string[] = [], failures: unknown[] = [];
const listener = createServer(async (request, response) => {
try {
let body = "";
for await (const chunk of request) body += chunk;
const { runId } = JSON.parse(body) as { runId: string };
const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId),
eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.agentId, f.agentId)));
if (!run) throw new Error("Unknown controlled conversation run");
await issueService(db).addComment(String(run.contextSnapshot?.issueId), "Saved chat delivered", {
agentId: f.agentId, runId: run.id,
});
replies.push(run.id); response.end("ok");
} catch (error) { failures.push(error); response.statusCode = 500; response.end("controlled reply failed"); }
});
await new Promise<void>(resolve => listener.listen(0, "127.0.0.1", resolve));
extraStops.push(async () => { listener.closeAllConnections(); await new Promise<void>(resolve => listener.close(() => resolve())); });
const port = (listener.address() as { port: number }).port;
const script = `fetch("http://127.0.0.1:${port}/reply", {method:"POST", body:JSON.stringify({runId:process.env.PAPERCLIP_RUN_ID})}).then(async response=>{if(!response.ok)throw new Error(await response.text());console.log("recovery-proof")}).catch(error=>{console.error(error);process.exitCode=1})`;
await db.update(agents).set({ adapterConfig: { command: process.execPath, args: ["-e", script], cwd: process.cwd() } })
.where(eq(agents.id, f.agentId));
const chat = await issueService(db).create(f.companyId, { title: "Saved conversation", status: "in_review",
conversationAgentId: f.agentId, conversationUserId: "recovery-user", assigneeAgentId: f.agentId,
conversationState: "waiting" });
const message = withMessage ? await issueService(db).addComment(chat.id, "Keep this message", { userId: "recovery-user" }, { clientRequestId: randomUUID() }) : null;
await settings.updateExperimental({ enableAgentChat: false });
await start(); await quiet();
expect(replies).toEqual([]);
return { ...f, settings, chat, message, replies, failures };
}
it("dispatches a saved conversation only from the committed Agent Chat enable transition", async () => {
const f = await savedConversation();
expect(await undeliveredConversationComments(db, f.companyId, f.chat.id)).toHaveLength(1);
await f.settings.updateExperimental({ enableAgentChat: true });
expect(coordinator.nextWakeAt()).not.toBeNull();
await vi.waitFor(async () => expect(await db.select().from(heartbeatRuns)
.where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1), { timeout: 40_000 });
const run = await completed(f.chat.id);
expect(f.replies).toEqual([run.id]); expect(f.failures).toEqual([]);
expect(await undeliveredConversationComments(db, f.companyId, f.chat.id)).toEqual([]);
expect((await db.select().from(issues).where(eq(issues.id, f.chat.id)))[0])
.toMatchObject({ conversationState: "waiting", executionRunId: null });
expect(await db.select().from(issueComments).where(and(eq(issueComments.createdByRunId, run.id),
eq(issueComments.authorAgentId, f.agentId)))).toHaveLength(1);
await quiet();
let hints = 0;
const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.strandedAssignments, () => { hints++; });
try { await f.settings.updateExperimental({ enableAgentChat: true }); expect(hints).toBe(0); await quiet(); }
finally { stop(); }
expect(f.replies).toEqual([run.id]);
});
it("keeps disabled saved conversations parked after rollback, unrelated flags and unchanged values", async () => {
const f = await savedConversation();
await expect(db.transaction(async tx => {
await instanceSettingsService(tx as unknown as typeof db).updateExperimental({ enableAgentChat: true });
throw new Error("chat flag rollback");
})).rejects.toThrow("chat flag rollback");
await quiet();
expect((await f.settings.getExperimental()).enableAgentChat).toBe(false);
let hints = 0;
const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.strandedAssignments, () => { hints++; });
try {
await f.settings.updateExperimental({ enableConferenceRoomChat: true });
await f.settings.updateExperimental({ enableAgentChat: false });
expect(hints).toBe(0); await quiet();
} finally { stop(); }
expect(f.replies).toEqual([]); expect(f.failures).toEqual([]);
expect(await db.select().from(heartbeatRuns)).toEqual([]);
expect(await undeliveredConversationComments(db, f.companyId, f.chat.id)).toHaveLength(1);
});
it("does not dispatch an accepted saved message after Agent Chat is disabled", async () => {
const f = await savedConversation(false);
await f.settings.updateExperimental({ enableAgentChat: true }); await quiet();
draining = true;
await issueService(db).addComment(f.chat.id, "Retain while disabled", { userId: "recovery-user" }, { clientRequestId: randomUUID() });
let hints = 0;
const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.strandedAssignments, () => { hints++; });
try { await f.settings.updateExperimental({ enableAgentChat: false }); expect(hints).toBe(0); }
finally { stop(); }
draining = false; await quiet();
expect(f.replies).toEqual([]); expect(f.failures).toEqual([]);
expect(await db.select().from(heartbeatRuns)).toEqual([]);
expect(await undeliveredConversationComments(db, f.companyId, f.chat.id)).toHaveLength(1);
});
describeIssueRecovery("issue recovery production producers and deadlines", () => {
const state = issueRecoveryFixture();
const { start, compete, fixture, completingProvider, settled, quiet, finish, completed } = state;
it("keeps empty, configured idle, and terminal history quiet", async () => {
await fixture(); await start(); await quiet();
});
it("creates a blocking child through the ordinary root service transaction", async () => {
const f = await fixture();
const parent = await issueService(db).create(f.companyId, { title: "Parent", status: "todo" });
const child = await issueService(db).createChild(parent.id, { title: "Blocking child", status: "todo", blockParentUntilDone: true });
const parent = await issueService(state.db).create(f.companyId, { title: "Parent", status: "todo" });
const child = await issueService(state.db).createChild(parent.id, { title: "Blocking child", status: "todo", blockParentUntilDone: true });
expect(child.parentBlockerAdded).toBe(true);
expect(await db.select().from(issueRelations).where(and(eq(issueRelations.companyId, f.companyId),
expect(await state.db.select().from(issueRelations).where(and(eq(issueRelations.companyId, f.companyId),
eq(issueRelations.issueId, child.issue.id), eq(issueRelations.relatedIssueId, parent.id)))).toHaveLength(1);
});
it.each(["startup", "committed"])("restores blocker-only and completed-action debt with no assignment candidates (%s)", async mode => {
const f = await fixture();
if (mode === "committed") { await start(); await quiet(); }
const blocker = await issueService(db).create(f.companyId, { title: "Orphan blocker", status: "blocked", createdByAgentId: f.agentId });
const dependent = await issueService(db).create(f.companyId, { title: "Blocked dependent", status: "blocked", blockedByIssueIds: [blocker.id] });
const completedSource = await issueService(db).create(f.companyId, { title: "Completed source", status: "done" });
const action = await issueRecoveryActionService(db).upsertSourceScoped({ companyId: f.companyId, sourceIssueId: completedSource.id,
kind: "stranded_assigned_issue", ownerType: "agent", ownerAgentId: f.agentId, cause: "stranded_assigned_issue",
fingerprint: "source-terminal", nextAction: "Restore work", wakePolicy: { type: "wake_owner" }, evidence: {} });
if (mode === "startup") await start();
await vi.waitFor(async () => {
expect((await db.select().from(issues).where(eq(issues.id, blocker.id)))[0].assigneeAgentId).toBe(f.agentId);
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0].status).toBe("resolved");
}, { timeout: 40_000 });
await finish(blocker.id); await finish(dependent.id); await quiet();
});
it("bounds subsidiary scans for100 assignment candidates and coalesces repeated company writes", async () => {
const f = await fixture(), foreign = await fixture();
await db.update(agents).set({ lifecycleState: "preparing" }).where(eq(agents.id, f.agentId));
await db.insert(issues).values(Array.from({ length: 100 }, (_, index) => ({ companyId: f.companyId,
await state.db.update(agents).set({ lifecycleState: "preparing" }).where(eq(agents.id, f.agentId));
await state.db.insert(issues).values(Array.from({ length: 100 }, (_, index) => ({ companyId: f.companyId,
title: `Waiting setup ${index}`, status: "todo", assigneeAgentId: f.agentId })));
const [source] = await db.insert(issues).values({ companyId: f.companyId, title: "Done source", status: "done" }).returning();
const action = await issueRecoveryActionService(db).upsertSourceScoped({ companyId: f.companyId, sourceIssueId: source.id,
const [source] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Done source", status: "done" }).returning();
const action = await issueRecoveryActionService(state.db).upsertSourceScoped({ companyId: f.companyId, sourceIssueId: source.id,
kind: "stranded_assigned_issue", ownerType: "agent", ownerAgentId: f.agentId, cause: "stranded_assigned_issue",
fingerprint: "query-budget", nextAction: "Restore work", wakePolicy: { type: "wake_owner" }, evidence: {} });
const scans = { blockers: 0, actions: 0, loads: 0 };
const originalSelect = db.select.bind(db);
vi.spyOn(db, "select").mockImplementation(((fields?: Record<string, unknown>) => {
const originalSelect = state.db.select.bind(state.db);
vi.spyOn(state.db, "select").mockImplementation(((fields?: Record<string, unknown>) => {
const query = originalSelect(fields as never); const from = query.from.bind(query);
query.from = ((table: unknown) => {
if (table === issues && fields?.id === issues.id && fields.companyId === issues.companyId && Object.keys(fields).length === 2) scans.loads++;
@@ -328,15 +47,15 @@ describe.skipIf(!support.supported)("issue recovery production producers and dea
return from(table as never);
}) as typeof query.from;
return query;
}) as typeof db.select);
}) as typeof state.db.select);
await start(); await settled();
expect(scans.actions).toBe(1); expect(scans.blockers).toBe(0); expect(scans.loads).toBeLessThanOrEqual(8);
expect((await originalSelect().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0].status).toBe("resolved");
const [foreignSource] = await db.insert(issues).values({ companyId: foreign.companyId, title: "Foreign done", status: "done" }).returning();
const [foreignAction] = await db.insert(issueRecoveryActions).values({ companyId: foreign.companyId, sourceIssueId: foreignSource.id,
const [foreignSource] = await state.db.insert(issues).values({ companyId: foreign.companyId, title: "Foreign done", status: "done" }).returning();
const [foreignAction] = await state.db.insert(issueRecoveryActions).values({ companyId: foreign.companyId, sourceIssueId: foreignSource.id,
kind: "stranded_assigned_issue", ownerType: "board", cause: "stranded_assigned_issue", fingerprint: "foreign-query-budget", nextAction: "Inspect" }).returning();
const before = { ...scans };
for (let index = 0; index < 100; index++) await db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId }));
for (let index = 0; index < 100; index++) await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId }));
await new Promise(resolve => setTimeout(resolve, 100));
expect(scans.loads).toBe(before.loads);
await settled();
@@ -347,179 +66,106 @@ describe.skipIf(!support.supported)("issue recovery production producers and dea
});
it("limits an ordinary comment to its issue among one thousand open assignments", async () => {
const f = await fixture();
const rows = await db.insert(issues).values(Array.from({ length: 1000 }, (_, index) => ({
const rows = await state.db.insert(issues).values(Array.from({ length: 1000 }, (_, index) => ({
companyId: f.companyId, title: `Open work ${index}`, status: "todo", assigneeAgentId: f.agentId,
}))).returning({ id: issues.id });
const checked: string[] = [];
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const register = coordinator.register.bind(coordinator);
coordinator.register = (queue, task) => register(queue, { ...task, run: signal => {
const pass = task.run(signal); passes.add(pass);
void pass.then(() => passes.delete(pass), () => passes.delete(pass));
state.coordinator = createDeliveryWorkCoordinator({ owner: state.db, canRun: () => true, onError: error => state.errors.push(error) });
const register = state.coordinator.register.bind(state.coordinator);
state.coordinator.register = (queue, task) => register(queue, { ...task, run: signal => {
const pass = task.run(signal); state.passes.add(pass);
void pass.then(() => state.passes.delete(pass), () => state.passes.delete(pass));
return pass;
} });
// Observe the actual paged reconstruction boundary without executing one
// thousand adapters. The producer, transaction, coordinator and SQL are real;
// the other acceptance cases exercise the resulting recovery decisions.
worker = registerIssueRecoveryWork(db, coordinator, {
state.worker = registerIssueRecoveryWork(state.db, state.coordinator, {
stranded: async scope => { checked.push(scope.issueId); },
orphanBlockers: async () => {}, recoveryActions: async () => {},
dependencies: async () => ({ enqueueFailed: 0 }), locks: async () => {},
watchdogs: async () => ({ incomplete: 0 }), schedulingSuppressed: async () => false,
}, () => true);
await worker.ready;
await state.worker.ready;
async function settleWork() {
for (let step = 0; step < 120; step++) {
await vi.advanceTimersByTimeAsync(1000);
await Promise.allSettled([...passes]); await immediate();
if (coordinator.nextWakeAt() === null && idleWorkSnapshot().active === 0) return;
await Promise.allSettled([...state.passes]); await immediate();
if (state.coordinator.nextWakeAt() === null && idleWorkSnapshot().active === 0) return;
}
throw new Error("Recovery did not settle its committed work within two virtual minutes");
}
await settleWork();
expect(new Set(checked).size).toBe(1000);
checked.length = 0;
await issueService(db).addComment(rows[0].id, "New evidence on this task", { userId: "recovery-user" });
await issueService(state.db).addComment(rows[0].id, "New evidence on this task", { userId: "recovery-user" });
await settleWork();
expect(checked).toEqual([rows[0].id]);
// Company eligibility changes still reconstruct all assignments. Narrowing
// comment hints must not hide unrelated work after an operator resumes it.
checked.length = 0;
await db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId }));
await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId }));
await settleWork();
expect(new Set(checked).size).toBe(1000);
await quiet();
});
it("dispatches an ordinary new assignment after commit and returns to quiet after completion", async () => {
const f = await fixture(); await start(); await quiet();
const issue = await issueService(db).create(f.companyId, { title: "Recover assignment", status: "todo", assigneeAgentId: f.agentId });
await completed(issue.id);
await finish(issue.id); await quiet();
}, 90_000);
it("reacts to blocker resolution once and dispatches the durable dependency wake", async () => {
const f = await fixture();
const blocker = await issueService(db).create(f.companyId, { title: "Blocker", status: "todo" });
const dependent = await issueService(db).create(f.companyId, { title: "Dependent", status: "blocked", assigneeAgentId: f.agentId,
blockedByIssueIds: [blocker.id] });
const failures = await completingProvider(f);
await start(); await quiet();
await compete();
await issueService(db).update(blocker.id, { status: "done" });
const run = await completed(dependent.id);
expect(run.contextSnapshot?.wakeReason).toBe("issue_blockers_resolved");
expect(failures).toEqual([]);
await finish(dependent.id); await quiet();
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'issue_blockers_resolved'`))).toHaveLength(1);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
}, 90_000);
it("settles saved disposition debt on its original real-clock timers before measuring idle", async () => {
const f = await fixture();
const blocker = await issueService(db).create(f.companyId, { title: "Saved debt blocker", status: "todo" });
const issue = await issueService(db).create(f.companyId, { title: "Saved disposition debt", status: "blocked",
assigneeAgentId: f.agentId, blockedByIssueIds: [blocker.id] });
await start(); await compete();
await issueService(db).update(blocker.id, { status: "done" });
await completed(issue.id);
let retryId: string | undefined;
await vi.waitFor(async () => {
const [retry] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.status, "scheduled_retry"),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`));
expect(retry).toBeDefined(); retryId = retry!.id;
}, { timeout: 40_000 });
await finish(issue.id); await quiet();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, retryId!)))[0].status).toBe("cancelled");
}, 120_000);
it("uses ordinary watchdog configuration and subtree closure to dispatch a reusable review", async () => {
const f = await fixture();
const source = await issueService(db).create(f.companyId, { title: "Finished subtree", status: "done" });
const failures = await completingProvider(f, () => taskWatchdogService(db).disableForIssue(f.companyId, source.id));
const source = await issueService(state.db).create(f.companyId, { title: "Finished subtree", status: "done" });
const failures = await completingProvider(f, () => taskWatchdogService(state.db).disableForIssue(f.companyId, source.id));
await start(); await quiet();
await compete();
const watchdog = await taskWatchdogService(db).upsertForIssue(f.companyId, source.id, { agentId: f.agentId });
const watchdog = await taskWatchdogService(state.db).upsertForIssue(f.companyId, source.id, { agentId: f.agentId });
let reviewId: string | undefined;
await vi.waitFor(async () => {
const [row] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, watchdog.watchdog.id));
const [row] = await state.db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, watchdog.watchdog.id));
expect(row.watchdogIssueId).toBeTruthy(); reviewId = row.watchdogIssueId!;
}, { timeout: 15_000 });
await completed(reviewId!);
expect(failures).toEqual([]);
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
expect(await state.db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'task_watchdog_stopped_subtree'`))).toHaveLength(1);
await finish(reviewId!);
await taskWatchdogService(db).disableForIssue(f.companyId, source.id);
await taskWatchdogService(state.db).disableForIssue(f.companyId, source.id);
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet();
}, 90_000);
it("bounds ordinary writes to their company and reconstructs foreign stale debt after restart", async () => {
const selected = await fixture(); await start(); await quiet();
const foreign = await fixture();
const [run] = await db.insert(heartbeatRuns).values({ companyId: foreign.companyId, agentId: foreign.agentId,
const [run] = await state.db.insert(heartbeatRuns).values({ companyId: foreign.companyId, agentId: foreign.agentId,
status: "failed", finishedAt: new Date() }).returning();
const [stale] = await db.insert(issues).values({ companyId: foreign.companyId, title: "Foreign stale lock", status: "done",
const [stale] = await state.db.insert(issues).values({ companyId: foreign.companyId, title: "Foreign stale lock", status: "done",
checkoutRunId: run.id }).returning();
await issueService(db).create(selected.companyId, { title: "Selected change", status: "done" }); await quiet();
expect((await db.select().from(issues).where(eq(issues.id, stale.id)))[0].checkoutRunId).toBe(run.id);
await worker.stop(); await dispatch.stop(); await cancellation.stop(); await coordinator.stop();
heartbeat = heartbeatService(db); await start(); await quiet();
expect((await db.select().from(issues).where(eq(issues.id, stale.id)))[0].checkoutRunId).toBeNull();
await issueService(state.db).create(selected.companyId, { title: "Selected change", status: "done" }); await quiet();
expect((await state.db.select().from(issues).where(eq(issues.id, stale.id)))[0].checkoutRunId).toBe(run.id);
await state.worker.stop(); await state.dispatch.stop(); await state.cancellation.stop(); await state.coordinator.stop();
state.heartbeat = heartbeatService(state.db); await start(); await quiet();
expect((await state.db.select().from(issues).where(eq(issues.id, stale.id)))[0].checkoutRunId).toBeNull();
});
it("fences nested assignment writes until outer commit and ignores rollback", async () => {
const f = await fixture(); await start(); await quiet();
let committedId = "";
await db.transaction(async outer => {
await outer.transaction(async nested => {
const issue = await issueService(db).create(f.companyId, { title: "Nested commit", status: "todo", assigneeAgentId: f.agentId }, nested);
committedId = issue.id;
});
await new Promise(resolve => setTimeout(resolve, 100));
expect(await db.select().from(heartbeatRuns)).toHaveLength(0);
});
await completed(committedId); await finish(committedId); await settled();
await expect(db.transaction(async outer => {
await outer.transaction(async nested => {
await issueService(db).create(f.companyId, { title: "Nested rollback", status: "todo", assigneeAgentId: f.agentId }, nested);
});
throw new Error("rollback control");
})).rejects.toThrow("rollback control");
await quiet();
expect(await db.select().from(issues).where(eq(issues.title, "Nested rollback"))).toHaveLength(0);
}, 90_000);
it("retains the first committed assignment during drain and executes on resume", async () => {
const f = await fixture(); await start(); await quiet(); draining = true;
const issue = await issueService(db).create(f.companyId, { title: "Drained assignment", status: "todo", assigneeAgentId: f.agentId });
await new Promise(resolve => setTimeout(resolve, 100));
const reads = vi.spyOn(db, "select");
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
try { await vi.advanceTimersByTimeAsync(90_000); expect(reads).not.toHaveBeenCalled(); }
finally { vi.useRealTimers(); reads.mockRestore(); }
expect(await db.select().from(heartbeatRuns)).toHaveLength(0);
draining = false; worker.wake(); dispatch.wake();
await completed(issue.id); await finish(issue.id); await quiet();
}, 90_000);
it("retains a fresh subtree's exact first-run grace then disarms on disable", async () => {
const f = await fixture();
const source = await issueService(db).create(f.companyId, { title: "Fresh unassigned subtree", status: "todo" });
const source = await issueService(state.db).create(f.companyId, { title: "Fresh unassigned subtree", status: "todo" });
await start(); await quiet();
await taskWatchdogService(db).upsertForIssue(f.companyId, source.id, { agentId: f.agentId });
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBe(source.createdAt.getTime() + 15_000), { timeout: 5_000 });
expect(await db.select().from(issues).where(eq(issues.originKind, "task_watchdog"))).toHaveLength(0);
await taskWatchdogService(db).disableForIssue(f.companyId, source.id);
await taskWatchdogService(state.db).upsertForIssue(f.companyId, source.id, { agentId: f.agentId });
await vi.waitFor(() => expect(state.coordinator.nextWakeAt()).toBe(source.createdAt.getTime() + 15_000), { timeout: 5_000 });
expect(await state.db.select().from(issues).where(eq(issues.originKind, "task_watchdog"))).toHaveLength(0);
await taskWatchdogService(state.db).disableForIssue(f.companyId, source.id);
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet();
});
it("waits for a live foreign controller's exact stored lease before clearing a terminal lock", async () => {
const f = await fixture();
const source = await issueService(db).create(f.companyId, { title: "Remote owner", status: "done" });
const source = await issueService(state.db).create(f.companyId, { title: "Remote owner", status: "done" });
const lease = new Date(Date.now() + 5_000);
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running",
const [run] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running",
runtimeMode: "legacy", controllerBootId: randomUUID(), controllerLeaseExpiresAt: lease,
contextSnapshot: { issueId: source.id }, processPid: 2_000_000_000, startedAt: new Date() }).returning();
await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, source.id));
await state.db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, source.id));
await start();
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBe(lease.getTime()), { timeout: 3_000 });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("running");
await vi.waitFor(async () => expect((await db.select().from(issues).where(eq(issues.id, source.id)))[0].executionRunId).toBeNull(), { timeout: 12_000 });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("succeeded");
await vi.waitFor(() => expect(state.coordinator.nextWakeAt()).toBe(lease.getTime()), { timeout: 3_000 });
expect((await state.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("running");
await vi.waitFor(async () => expect((await state.db.select().from(issues).where(eq(issues.id, source.id)))[0].executionRunId).toBeNull(), { timeout: 12_000 });
expect((await state.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("succeeded");
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet();
}, 90_000);
});
@@ -0,0 +1,62 @@
import { and, eq, sql } from "drizzle-orm";
import { expect, it, vi } from "vitest";
import { heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db";
import { issueService } from "../services/issues.js";
import { issueRecoveryActionService } from "../services/issue-recovery-actions.js";
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
describeIssueRecovery("issue recovery production producers and dependencies", () => {
const state = issueRecoveryFixture();
const { start, compete, fixture, completingProvider, quiet, finish, completed } = state;
it.each(["startup", "committed"])("restores blocker-only and completed-action debt with no assignment candidates (%s)", async mode => {
const f = await fixture();
if (mode === "committed") { await start(); await quiet(); }
const blocker = await issueService(state.db).create(f.companyId, { title: "Orphan blocker", status: "blocked", createdByAgentId: f.agentId });
const dependent = await issueService(state.db).create(f.companyId, { title: "Blocked dependent", status: "blocked", blockedByIssueIds: [blocker.id] });
const completedSource = await issueService(state.db).create(f.companyId, { title: "Completed source", status: "done" });
const action = await issueRecoveryActionService(state.db).upsertSourceScoped({ companyId: f.companyId, sourceIssueId: completedSource.id,
kind: "stranded_assigned_issue", ownerType: "agent", ownerAgentId: f.agentId, cause: "stranded_assigned_issue",
fingerprint: "source-terminal", nextAction: "Restore work", wakePolicy: { type: "wake_owner" }, evidence: {} });
if (mode === "startup") await start();
await vi.waitFor(async () => {
expect((await state.db.select().from(issues).where(eq(issues.id, blocker.id)))[0].assigneeAgentId).toBe(f.agentId);
expect((await state.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0].status).toBe("resolved");
}, { timeout: 40_000 });
await finish(blocker.id); await finish(dependent.id); await quiet();
});
it("reacts to blocker resolution once and dispatches the durable dependency wake", async () => {
const f = await fixture();
const blocker = await issueService(state.db).create(f.companyId, { title: "Blocker", status: "todo" });
const dependent = await issueService(state.db).create(f.companyId, { title: "Dependent", status: "blocked", assigneeAgentId: f.agentId,
blockedByIssueIds: [blocker.id] });
const failures = await completingProvider(f);
await start(); await quiet();
await compete();
await issueService(state.db).update(blocker.id, { status: "done" });
const run = await completed(dependent.id);
expect(run.contextSnapshot?.wakeReason).toBe("issue_blockers_resolved");
expect(failures).toEqual([]);
await finish(dependent.id); await quiet();
expect(await state.db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'issue_blockers_resolved'`))).toHaveLength(1);
expect(await state.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
}, 90_000);
it("settles saved disposition debt on its original real-clock timers before measuring idle", async () => {
const f = await fixture();
const blocker = await issueService(state.db).create(f.companyId, { title: "Saved debt blocker", status: "todo" });
const issue = await issueService(state.db).create(f.companyId, { title: "Saved disposition debt", status: "blocked",
assigneeAgentId: f.agentId, blockedByIssueIds: [blocker.id] });
await start(); await compete();
await issueService(state.db).update(blocker.id, { status: "done" });
await completed(issue.id);
let retryId: string | undefined;
await vi.waitFor(async () => {
const [retry] = await state.db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.status, "scheduled_retry"),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`));
expect(retry).toBeDefined(); retryId = retry!.id;
}, { timeout: 40_000 });
await finish(issue.id); await quiet();
expect((await state.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, retryId!)))[0].status).toBe("cancelled");
}, 120_000);
});
@@ -0,0 +1,241 @@
import { createServer } from "node:http";
import { idleWorkSnapshot } from "../services/task-admission.js";
import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, vi } from "vitest";
import { agents, authUsers, companies, companyMemberships, createDb, heartbeatRuns } from "@paperclipai/db";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { heartbeatService } from "../services/heartbeat.js";
import { instanceSettingsService } from "../services/instance-settings.js";
import { issueService } from "../services/issues.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
vi.setConfig({ testTimeout: 90_000 });
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_ISSUE_RECOVERY_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
export const describeIssueRecovery: ReturnType<typeof describe.skipIf> = describe.skipIf(!support.supported);
export function issueRecoveryFixture() {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: ReturnType<typeof createDb>;
let heartbeat: ReturnType<typeof heartbeatService>;
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator>;
let worker: ReturnType<ReturnType<typeof heartbeatService>["registerIssueStateRecoveryWork"]>;
let dispatch: ReturnType<ReturnType<typeof heartbeatService>["registerRunDispatchWork"]>;
let cancellation: ReturnType<ReturnType<typeof heartbeatService>["registerScheduledRetryCancellationWork"]>;
let draining = false;
const tasks = new Map<string, { nextRunAt?: () => number | null; hasPending: () => Promise<boolean> }>();
const errors: unknown[] = [];
const passes = new Set<Promise<unknown>>();
const extraStops: Array<() => Promise<void>> = [];
const extraDeadlines: Array<() => number | null> = [];
beforeAll(async () => {
if (!external) database = await startEmbeddedPostgresTestDatabase("issue-recovery-work-");
db = createDb(external ?? database!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`)));
}, 90_000);
beforeEach(async () => {
await db.execute(sql`TRUNCATE companies CASCADE`);
heartbeat = heartbeatService(db); draining = false; errors.length = 0;
});
afterEach(async () => {
for (const stop of extraStops.splice(0)) await stop();
extraDeadlines.length = 0;
await worker?.stop(); await dispatch?.stop(); await cancellation?.stop(); await coordinator?.stop();
await heartbeat.drainActiveRunExecutions();
vi.useRealTimers(); vi.restoreAllMocks();
expect(errors).toEqual([]);
});
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); });
async function start() {
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true,
onError: error => errors.push(error) });
tasks.clear();
const register = coordinator.register.bind(coordinator);
coordinator.register = (queue, task) => {
tasks.set(queue, task);
return register(queue, { ...task, run: signal => {
const work = task.run(signal); passes.add(work);
void work.then(() => passes.delete(work), () => passes.delete(work));
return work;
} });
};
worker = heartbeat.registerIssueStateRecoveryWork(coordinator, () => !draining);
dispatch = heartbeat.registerRunDispatchWork(coordinator, () => !draining);
cancellation = heartbeat.registerScheduledRetryCancellationWork(coordinator, () => !draining);
await Promise.all([worker.ready, dispatch.ready, cancellation.ready]);
}
async function compete() {
const other = heartbeatService(db);
const competing = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const register = competing.register.bind(competing);
competing.register = (queue, task) => register(queue, { ...task, run: signal => {
const pass = task.run(signal); passes.add(pass);
void pass.then(() => passes.delete(pass), () => passes.delete(pass));
return pass;
} });
const work = other.registerIssueStateRecoveryWork(competing, () => true);
extraStops.push(() => competing.stop()); extraDeadlines.push(competing.nextWakeAt); await work.ready;
}
async function fixture() {
const companyId = randomUUID();
await db.insert(authUsers).values({ id: "recovery-user", name: "Recovery operator", email: "[email protected]", createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing();
await db.insert(companies).values({ id: companyId, name: "Recovery", issuePrefix: `R${companyId.slice(0, 7)}`,
requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: "recovery-user" });
await db.insert(companyMemberships).values({ companyId, principalId: "recovery-user", principalType: "user", status: "active", membershipRole: "owner" });
const [agent] = await db.insert(agents).values({ companyId, name: "Recovery process", role: "engineer", status: "idle",
adapterType: "process", adapterConfig: { command: process.execPath, args: ["-e", "console.log('recovery-proof')"], cwd: process.cwd() },
runtimeConfig: { heartbeat: { enabled: true, intervalSec: 0, wakeOnDemand: true } },
}).returning();
return { companyId, agentId: agent.id };
}
async function completingProvider(f: { companyId: string; agentId: string }, beforeComplete?: () => Promise<unknown>) {
const failures: unknown[] = [];
const listener = createServer(async (request, response) => {
try {
let body = "";
for await (const chunk of request) body += chunk;
const { runId } = JSON.parse(body) as { runId: string };
const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId),
eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.agentId, f.agentId)));
if (!run?.contextSnapshot?.issueId) throw new Error("Unknown controlled task run");
await beforeComplete?.();
await issueService(db).update(String(run.contextSnapshot.issueId), { status: "done" });
response.end("ok");
} catch (error) { failures.push(error); response.statusCode = 500; response.end("completion failed"); }
});
await new Promise<void>(resolve => listener.listen(0, "127.0.0.1", resolve));
extraStops.push(async () => { listener.closeAllConnections(); await new Promise<void>(resolve => listener.close(() => resolve())); });
const port = (listener.address() as { port: number }).port;
const script = `fetch("http://127.0.0.1:${port}/complete", {method:"POST", body:JSON.stringify({runId:process.env.PAPERCLIP_RUN_ID})}).then(async response=>{if(!response.ok)throw new Error(await response.text());console.log("recovery-proof")}).catch(error=>{console.error(error);process.exitCode=1})`;
await db.update(agents).set({ adapterConfig: { command: process.execPath, args: ["-e", script], cwd: process.cwd() } })
.where(eq(agents.id, f.agentId));
return failures;
}
async function settled() {
await vi.waitFor(async () => {
await Promise.allSettled([...passes]);
await heartbeat.drainActiveRunExecutions();
const wake = coordinator.nextWakeAt();
const detail = wake === null ? "" : JSON.stringify({ now: Date.now(), tasks: await Promise.all([...tasks].map(async ([queue, task]) => ({ queue, at: task.nextRunAt?.(), pending: await task.hasPending() }))),
runs: (await db.select().from(heartbeatRuns)).map(run => ({ id: run.id, status: run.status, error: run.errorCode, lease: run.controllerLeaseExpiresAt, reason: run.contextSnapshot?.wakeReason })) });
expect(wake, detail).toBeNull();
expect(extraDeadlines.map(next => next())).toEqual(extraDeadlines.map(() => null));
expect(idleWorkSnapshot().active).toBe(0);
}, { timeout: 40_000 });
}
async function quiet() {
await settled();
expect(await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.status} in ('queued', 'running')`)).toHaveLength(0);
const reads = vi.spyOn(db, "select"); const executes = vi.spyOn(db, "execute");
const selected = reads.mock.calls.length, executed = executes.mock.calls.length;
const statements: string[] = [], priorDebug = db.$client.options.debug;
db.$client.options.debug = (_connection, query) => { statements.push(query); };
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
try {
await vi.advanceTimersByTimeAsync(20 * 60_000);
expect(reads).toHaveBeenCalledTimes(selected); expect(executes).toHaveBeenCalledTimes(executed);
expect(statements).toEqual([]);
} finally { vi.useRealTimers(); db.$client.options.debug = priorDebug; reads.mockRestore(); executes.mockRestore(); }
}
async function finish(issueId: string) {
const retries = () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.status, "scheduled_retry"),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`));
const before = await retries();
await issueService(db).update(issueId, { status: "done" });
// Closing one blocker can legitimately start its dependent. Wait for this
// task's dispatch and execution; quiet() verifies the whole fixture later.
await vi.waitFor(async () => {
await Promise.allSettled([...passes]);
await heartbeat.drainActiveRunExecutions();
expect(await db.select().from(heartbeatRuns).where(and(
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`,
sql`${heartbeatRuns.status} in ('queued', 'running')`,
))).toHaveLength(0);
}, { timeout: 15_000 });
const saved = [...new Map([...before, ...await retries()].map(run => [run.id, run])).values()];
if (saved.length > 0) {
const trace: unknown[] = [];
await vi.waitFor(async () => {
await Promise.allSettled([...passes]);
await heartbeat.drainActiveRunExecutions();
const remaining = await retries();
if (remaining.length > 0) trace.push({ now: Date.now(),
deadlines: [coordinator.nextWakeAt(), ...extraDeadlines.map(next => next())],
remaining: remaining.map(run => ({ id: run.id, at: run.scheduledRetryAt })) });
expect(remaining, JSON.stringify(trace)).toHaveLength(0);
}, { timeout: Math.max(...saved.map(run => run.scheduledRetryAt!.getTime())) - Date.now() + 30_000 });
for (const retry of saved) {
const [settledRetry] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, retry.id));
expect(settledRetry.status).toBe("cancelled");
}
}
}
async function completed(issueId: string) {
let runId: string | undefined;
await vi.waitFor(async () => {
const runs = await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`);
const run = runs.find(row => row.status === "running" || row.status === "succeeded");
expect(run).toBeDefined(); runId = run!.id;
}, { timeout: 15_000 });
await vi.waitFor(async () => {
await heartbeat.drainActiveRunExecutions();
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId!));
expect(current.status, JSON.stringify({ error: current.error, code: current.errorCode, stderr: current.stderrExcerpt })).toBe("succeeded");
}, { timeout: 15_000 });
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId!));
expect(run.stdoutExcerpt).toContain("recovery-proof");
return run;
}
async function savedConversation(withMessage = true) {
const f = await fixture(), settings = instanceSettingsService(db);
await settings.updateExperimental({ enableAgentChat: true, enableConferenceRoomChat: false });
const replies: string[] = [], failures: unknown[] = [];
const listener = createServer(async (request, response) => {
try {
let body = "";
for await (const chunk of request) body += chunk;
const { runId } = JSON.parse(body) as { runId: string };
const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId),
eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.agentId, f.agentId)));
if (!run) throw new Error("Unknown controlled conversation run");
await issueService(db).addComment(String(run.contextSnapshot?.issueId), "Saved chat delivered", {
agentId: f.agentId, runId: run.id,
});
replies.push(run.id); response.end("ok");
} catch (error) { failures.push(error); response.statusCode = 500; response.end("controlled reply failed"); }
});
await new Promise<void>(resolve => listener.listen(0, "127.0.0.1", resolve));
extraStops.push(async () => { listener.closeAllConnections(); await new Promise<void>(resolve => listener.close(() => resolve())); });
const port = (listener.address() as { port: number }).port;
const script = `fetch("http://127.0.0.1:${port}/reply", {method:"POST", body:JSON.stringify({runId:process.env.PAPERCLIP_RUN_ID})}).then(async response=>{if(!response.ok)throw new Error(await response.text());console.log("recovery-proof")}).catch(error=>{console.error(error);process.exitCode=1})`;
await db.update(agents).set({ adapterConfig: { command: process.execPath, args: ["-e", script], cwd: process.cwd() } })
.where(eq(agents.id, f.agentId));
const chat = await issueService(db).create(f.companyId, { title: "Saved conversation", status: "in_review",
conversationAgentId: f.agentId, conversationUserId: "recovery-user", assigneeAgentId: f.agentId,
conversationState: "waiting" });
const message = withMessage ? await issueService(db).addComment(chat.id, "Keep this message", { userId: "recovery-user" }, { clientRequestId: randomUUID() }) : null;
await settings.updateExperimental({ enableAgentChat: false });
await start(); await quiet();
expect(replies).toEqual([]);
return { ...f, settings, chat, message, replies, failures };
}
return {
start, compete, fixture, completingProvider, settled, quiet, finish, completed, savedConversation,
get db() { return db; },
get heartbeat() { return heartbeat; },
set heartbeat(value: typeof heartbeat) { heartbeat = value; },
get coordinator() { return coordinator; },
set coordinator(value: typeof coordinator) { coordinator = value; },
get worker() { return worker; },
set worker(value: typeof worker) { worker = value; },
get dispatch() { return dispatch; },
get cancellation() { return cancellation; },
get draining() { return draining; },
set draining(value: typeof draining) { draining = value; },
get errors() { return errors; },
get passes() { return passes; },
};
}
@@ -6,7 +6,7 @@ import { agents, agentApiKeys, agentSessionGoalActions, agentTaskSessions, authU
import { access, mkdir, readdir, utimes, writeFile } from "node:fs/promises";
import path from "node:path";
import { setTimeout as delay } from "node:timers/promises";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { beforeAll, describe, it, vi } from "vitest";
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { maintenanceDatabaseSupport } from "./helpers/maintenance/environment.js";
@@ -14,19 +14,31 @@ import { CLOUD_RUNTIME_IDENTITY_AUDIENCE, CLOUD_RUNTIME_IDENTITY_ISSUER, CLOUD_R
import { prepareMaintenanceRuntime } from "./helpers/maintenance/runtime-build.js";
import { startMaintenanceServer, withMaintenanceProvider } from "./helpers/maintenance/server-harness.js";
let databaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_MAINTENANCE_BOOT_TEST_DATABASE_URL");
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
const databaseSupport = await maintenanceDatabaseSupport(databaseUrl);
const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_MAINTENANCE_BOOT_TEST_DATABASE_URL");
const databaseSupport = await maintenanceDatabaseSupport(externalDatabaseUrl);
const describeDatabase = databaseSupport.supported ? describe : describe.skip;
if (!databaseSupport.supported) console.warn(`Skipping maintenance PostgreSQL tests: ${databaseSupport.reason ?? "unsupported environment"}`);
async function clearDatabase() {
if (!temporary && !new URL(databaseUrl!).pathname.endsWith("_test")) throw new Error("Maintenance boot requires an owned disposable _test database");
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
try {
const tables = await fixture.$client<{ tablename: string }[]>`select tablename from pg_tables where schemaname='public'`;
if (tables.length) await fixture.$client.unsafe(`truncate ${tables.map(table => `"${table.tablename.replaceAll('"', '""')}"`).join(",")} cascade`);
} finally { await fixture.$client.end(); }
}
// Each case already owns its server process, home, proxy, ports and provider.
// Give it an owned database too, so the real observation windows can overlap.
// An explicitly shared external database retains the original serial behavior.
vi.setConfig({ maxConcurrency: externalDatabaseUrl ? 1 : 2 });
const maintenanceTest = it.extend<{ databaseUrl: string }>({
databaseUrl: async ({}, use) => {
const temporary = externalDatabaseUrl ? undefined : await startEmbeddedPostgresTestDatabase("maintenance-server-");
const databaseUrl = externalDatabaseUrl ?? temporary!.connectionString;
try {
if (externalDatabaseUrl) {
if (!new URL(databaseUrl).pathname.endsWith("_test")) throw new Error("Maintenance boot requires an owned disposable _test database");
const fixture = createDb(databaseUrl, { maxConnections: 1 });
try {
const tables = await fixture.$client<{ tablename: string }[]>`select tablename from pg_tables where schemaname='public'`;
if (tables.length) await fixture.$client.unsafe(`truncate ${tables.map(table => `"${table.tablename.replaceAll('"', '""')}"`).join(",")} cascade`);
} finally { await fixture.$client.end(); }
}
await use(databaseUrl);
} finally { await temporary?.cleanup(); }
},
});
async function settleClock(server: Awaited<ReturnType<typeof startMaintenanceServer>>) {
for (let step = 0; step < 60; step++) {
if (server.proxy.pending === 0 && (await server.snapshot()).idle.active === 0) return server.settled(100);
@@ -48,12 +60,9 @@ function receipt(queries: Array<{ protocol: string; sql: string; fingerprint: st
describeDatabase("assembled maintenance server production wiring", () => {
beforeAll(async () => {
await prepareMaintenanceRuntime();
if (!databaseUrl) { temporary = await startEmbeddedPostgresTestDatabase("maintenance-server-"); databaseUrl = temporary.connectionString; }
}, 150_000);
afterAll(async () => { await temporary?.cleanup(); });
it.each(["empty", "configured terminal history"] as const)("keeps enabled %s quiet for 90 real seconds and two qualified hourly windows", async state => {
await clearDatabase();
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
maintenanceTest.concurrent.for(["empty", "configured terminal history"] as const)("keeps enabled %s quiet for 90 real seconds and two qualified hourly windows", { timeout: 240_000 }, async (state, { databaseUrl, expect }) => {
const fixture = createDb(databaseUrl, { maxConnections: 1 });
if (state !== "empty") {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID();
await fixture.insert(companies).values({ id: companyId, name: "Retained idle history", issuePrefix: "MIH" });
@@ -62,7 +71,7 @@ describeDatabase("assembled maintenance server production wiring", () => {
await fixture.insert(issues).values({ id: issueId, companyId, title: "Retained terminal task", status: "done", assigneeAgentId: agentId });
await fixture.insert(heartbeatRuns).values({ companyId, agentId, issueId, status: "succeeded", finishedAt: new Date(), contextSnapshot: { issueId } });
}
const server = await startMaintenanceServer(databaseUrl!, { clock: true });
const server = await startMaintenanceServer(databaseUrl, { clock: true });
try {
if (state !== "empty") await server.api("PATCH", "/instance/settings/experimental", { enableExternalObjects: true, enableStatusCards: true });
await settleClock(server); await server.freeze();
@@ -107,12 +116,11 @@ describeDatabase("assembled maintenance server production wiring", () => {
boundary: "actual startServer, real elapsed 90s plus qualified child clock; pool connection housekeeping reported separately; no active provider/native work, Cloud enrollment, installed plugins, telemetry exporter, announcements or database backup; no host or database suspension claim",
}, null, 2));
} finally { await server.stop(); await fixture.$client.end(); }
}, 240_000);
});
it("releases an ordinary held task Retry into one real process execution", async () => {
await clearDatabase();
const server = await startMaintenanceServer(databaseUrl!, { clock: true });
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
maintenanceTest.concurrent("releases an ordinary held task Retry into one real process execution", async ({ databaseUrl, expect }) => {
const server = await startMaintenanceServer(databaseUrl, { clock: true });
const fixture = createDb(databaseUrl, { maxConnections: 1 });
try {
await settleClock(server);
const company = await server.api<{ id: string }>("POST", "/companies", { name: "Receipt ingress" });
@@ -160,10 +168,9 @@ describeDatabase("assembled maintenance server production wiring", () => {
} finally { await server.stop(); await fixture.$client.end(); }
}, 150_000);
it("reconstructs residual work from a normal scheduler-admission change after suppression", async () => {
await clearDatabase();
const server = await startMaintenanceServer(databaseUrl!, { clock: true, suppressed: true });
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
maintenanceTest.concurrent("reconstructs residual work from a normal scheduler-admission change after suppression", async ({ databaseUrl, expect }) => {
const server = await startMaintenanceServer(databaseUrl, { clock: true, suppressed: true });
const fixture = createDb(databaseUrl, { maxConnections: 1 });
try {
await server.settled();
const company = await server.api<{ id: string }>("POST", "/companies", { name: "Admission reconstruction" });
@@ -193,9 +200,8 @@ describeDatabase("assembled maintenance server production wiring", () => {
} finally { await server.stop(); await fixture.$client.end(); }
}, 150_000);
it("isolates a real goal-startup write failure and still registers residual and attention owners", async () => {
await clearDatabase();
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
maintenanceTest.concurrent("isolates a real goal-startup write failure and still registers residual and attention owners", async ({ databaseUrl, expect }) => {
const fixture = createDb(databaseUrl, { maxConnections: 1 });
const companyId = randomUUID(), agentId = randomUUID(), userId = randomUUID(), issueId = randomUUID(), requestId = randomUUID();
await fixture.insert(authUsers).values({ id: userId, name: "Goal control owner", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
await fixture.insert(companies).values({ id: companyId, name: "Startup failure isolation", issuePrefix: "MFI", defaultResponsibleUserId: userId });
@@ -210,7 +216,7 @@ describeDatabase("assembled maintenance server production wiring", () => {
await fixture.$client.unsafe(`create trigger maintenance_fail_goal_startup before insert on agent_wakeup_requests for each row execute function maintenance_fail_goal_startup()`);
let server: Awaited<ReturnType<typeof startMaintenanceServer>> | undefined;
try {
server = await startMaintenanceServer(databaseUrl!, { clock: true });
server = await startMaintenanceServer(databaseUrl, { clock: true });
await settleClock(server);
expect((await server.snapshot()).startup.phase).toBe("ready");
expect(server.output).toContain("maintenance_goal_startup_injected");
@@ -256,9 +262,8 @@ describeDatabase("assembled maintenance server production wiring", () => {
else await expect(access(server!.directory)).rejects.toMatchObject({ code: "ENOENT" });
}, 150_000);
it("exposes readiness while real deferred housekeeping is held and settles its work after release", async () => {
await clearDatabase();
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
maintenanceTest.concurrent("exposes readiness while real deferred housekeeping is held and settles its work after release", async ({ databaseUrl, expect }) => {
const fixture = createDb(databaseUrl, { maxConnections: 1 });
const companyId = randomUUID(), issueId = randomUUID(), interactionId = randomUUID();
await fixture.insert(companies).values({ id: companyId, name: "Readiness fixture", issuePrefix: "MRD" });
await fixture.insert(issues).values({ id: issueId, companyId, title: "Historical human decision", status: "todo" });
@@ -272,7 +277,7 @@ describeDatabase("assembled maintenance server production wiring", () => {
await reserved`select pg_advisory_lock(913703)`;
let server: Awaited<ReturnType<typeof startMaintenanceServer>> | undefined;
try {
server = await startMaintenanceServer(databaseUrl!, { suppressed: true });
server = await startMaintenanceServer(databaseUrl, { suppressed: true });
const current = server;
await expect.poll(() => current.proxy.queries.some(query => /insert into "decision_retention"/.test(query.sql)), { timeout: 10_000 }).toBe(true);
const snapshot = await server.snapshot();
@@ -296,9 +301,8 @@ describeDatabase("assembled maintenance server production wiring", () => {
}
}, 150_000);
it("does not advance the child clock over an actual driver socket whose startup write is held", async () => {
await clearDatabase();
const server = await startMaintenanceServer(databaseUrl!, { clock: true, suppressed: true });
maintenanceTest.concurrent("does not advance the child clock over an actual driver socket whose startup write is held", async ({ databaseUrl, expect }) => {
const server = await startMaintenanceServer(databaseUrl, { clock: true, suppressed: true });
const client = createDb(server.proxy.url, { maxConnections: 1, idleTimeoutSeconds: 0, connectTimeoutSeconds: 30 });
const address = new URL(server.proxy.url);
const socket = new Socket();
@@ -341,9 +345,8 @@ describeDatabase("assembled maintenance server production wiring", () => {
}
}, 150_000);
it("keeps the disabled-heartbeat boot quiet across two former hourly windows with a qualified child clock", async () => {
await clearDatabase();
const server = await startMaintenanceServer(databaseUrl!, { enabled: false, clock: true });
maintenanceTest.concurrent("keeps the disabled-heartbeat boot quiet across two former hourly windows with a qualified child clock", async ({ databaseUrl, expect }) => {
const server = await startMaintenanceServer(databaseUrl, { enabled: false, clock: true });
try {
await server.settled();
await server.freeze();
@@ -367,7 +370,7 @@ describeDatabase("assembled maintenance server production wiring", () => {
const history = await server.api<{ id: string }>("POST", `/companies/${company.id}/issues`, { title: "Retained completed history", status: "done" });
await server.advance(30_000);
await server.settled();
const persisted = createDb(databaseUrl!, { maxConnections: 1 });
const persisted = createDb(databaseUrl, { maxConnections: 1 });
try {
const [saved] = await persisted.$client`select company_id, status from issues where id=${history.id}`;
expect(saved).toMatchObject({ company_id: company.id, status: "done" });
@@ -387,8 +390,7 @@ describeDatabase("assembled maintenance server production wiring", () => {
} finally { await server.stop(); }
}, 150_000);
it("settles an in-flight provider claim on shutdown and reconstructs its unfinished object after restart", async () => {
await clearDatabase();
maintenanceTest.concurrent("settles an in-flight provider claim on shutdown and reconstructs its unfinished object after restart", async ({ databaseUrl, expect }) => {
let arrived!: () => void, cancelled = false;
const firstRequest = new Promise<void>(resolve => { arrived = resolve; });
const providerRequests: string[] = [];
@@ -401,10 +403,10 @@ describeDatabase("assembled maintenance server production wiring", () => {
});
await withMaintenanceProvider(provider, async origin => {
const options = { enabled: false, clock: true, githubOrigin: origin };
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
const fixture = createDb(databaseUrl, { maxConnections: 1 });
let server: Awaited<ReturnType<typeof startMaintenanceServer>> | undefined;
try {
server = await startMaintenanceServer(databaseUrl!, options);
server = await startMaintenanceServer(databaseUrl, options);
await server.settled();
const company = await server.api<{ id: string }>("POST", "/companies", { name: "Shutdown recovery" });
await server.api("PATCH", "/instance/settings/experimental", { enableExternalObjects: true });
@@ -420,7 +422,7 @@ describeDatabase("assembled maintenance server production wiring", () => {
expect(cancelled).toBe(true);
const [stopped] = await fixture.$client`select is_terminal, refresh_started_at, refresh_token from external_objects where company_id=${company.id}`;
expect(stopped).toMatchObject({ is_terminal: false, refresh_started_at: null, refresh_token: null });
server = await startMaintenanceServer(databaseUrl!, options);
server = await startMaintenanceServer(databaseUrl, options);
await settleClock(server);
const [recovered] = await fixture.$client`select is_terminal, status_key from external_objects where company_id=${company.id}`;
expect(recovered).toMatchObject({ is_terminal: true, status_key: "merged" });
@@ -436,8 +438,7 @@ describeDatabase("assembled maintenance server production wiring", () => {
});
}, 150_000);
it("processes the first ordinary external-object producer committed during idle drain and becomes quiet again", async () => {
await clearDatabase();
maintenanceTest.concurrent("processes the first ordinary external-object producer committed during idle drain and becomes quiet again", async ({ databaseUrl, expect }) => {
const providerRequests: string[] = [];
const provider = createServer((req, res) => {
providerRequests.push(req.url!);
@@ -451,12 +452,12 @@ describeDatabase("assembled maintenance server production wiring", () => {
title: "Controlled merged pull request", html_url: `https://github.com/maintenance/example/pull/${number}`, user: { login: "fixture" } }));
});
await withMaintenanceProvider(provider, async origin => {
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
const fixture = createDb(databaseUrl, { maxConnections: 1 });
let reserved: Awaited<ReturnType<typeof fixture.$client.reserve>> | undefined;
let pendingIssue: Promise<{ id: string }> | undefined;
let runtime: Awaited<ReturnType<typeof startMaintenanceServer>> | undefined;
try {
const server = runtime = await startMaintenanceServer(databaseUrl!, { enabled: false, clock: true,
const server = runtime = await startMaintenanceServer(databaseUrl, { enabled: false, clock: true,
githubOrigin: origin });
await server.settled();
const company = await server.api<{ id: string }>("POST", "/companies", { name: "Drain activation" });
@@ -538,8 +539,7 @@ describeDatabase("assembled maintenance server production wiring", () => {
});
}, 150_000);
it("discovers and acknowledges a sealed GitHub event after the first ordinary connection activation", async () => {
await clearDatabase();
maintenanceTest.concurrent("discovers and acknowledges a sealed GitHub event after the first ordinary connection activation", async ({ databaseUrl, expect }) => {
const signing = generateKeyPairSync("ed25519"), sealing = generateKeyPairSync("x25519");
const requests: string[] = [], acknowledgements: unknown[] = [];
let event: Record<string, unknown> | undefined, leased = false;
@@ -567,10 +567,10 @@ describeDatabase("assembled maintenance server production wiring", () => {
purpose: "events", provider: "github", profile: "github.code", epk: epk.toString("base64url"), iv: iv.toString("base64url"), ct: ct.toString("base64url") } }));
});
await withMaintenanceProvider(broker, async origin => {
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
const fixture = createDb(databaseUrl, { maxConnections: 1 });
let runtime: Awaited<ReturnType<typeof startMaintenanceServer>> | undefined;
try {
const server = runtime = await startMaintenanceServer(databaseUrl!, { enabled: false, clock: true,
const server = runtime = await startMaintenanceServer(databaseUrl, { enabled: false, clock: true,
githubOrigin: origin, connector: {
origin: origin, signPrivateKey: signing.privateKey.export({ format: "jwk" }).d!,
sealPrivateKey: sealing.privateKey.export({ format: "jwk" }).d!,
@@ -612,10 +612,9 @@ describeDatabase("assembled maintenance server production wiring", () => {
});
}, 150_000);
it("reacts to the first ordinary status-card source write and accepts authorized controlled summarizer output", async () => {
await clearDatabase();
const server = await startMaintenanceServer(databaseUrl!, { clock: true, suppressed: true });
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
maintenanceTest.concurrent("reacts to the first ordinary status-card source write and accepts authorized controlled summarizer output", async ({ databaseUrl, expect }) => {
const server = await startMaintenanceServer(databaseUrl, { clock: true, suppressed: true });
const fixture = createDb(databaseUrl, { maxConnections: 1 });
try {
await server.settled();
const company = await server.api<{ id: string }>("POST", "/companies", { name: "Status-card ingress" });
@@ -670,10 +669,9 @@ describeDatabase("assembled maintenance server production wiring", () => {
} finally { await fixture.$client.end(); await server.stop(); }
}, 150_000);
it("expires an ordinarily published backup after its ordinary task close without losing run history", async () => {
await clearDatabase();
const server = await startMaintenanceServer(databaseUrl!, { enabled: false, clock: true });
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
maintenanceTest.concurrent("expires an ordinarily published backup after its ordinary task close without losing run history", async ({ databaseUrl, expect }) => {
const server = await startMaintenanceServer(databaseUrl, { enabled: false, clock: true });
const fixture = createDb(databaseUrl, { maxConnections: 1 });
try {
await server.settled();
const company = await server.api<{ id: string }>("POST", "/companies", { name: "Backup ingress" });
@@ -715,10 +713,9 @@ describeDatabase("assembled maintenance server production wiring", () => {
}
}, 150_000);
it("archives the first ordinary overdue attention source through the assembled retention worker", async () => {
await clearDatabase();
const server = await startMaintenanceServer(databaseUrl!, { clock: true, suppressed: true });
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
maintenanceTest.concurrent("archives the first ordinary overdue attention source through the assembled retention worker", async ({ databaseUrl, expect }) => {
const server = await startMaintenanceServer(databaseUrl, { clock: true, suppressed: true });
const fixture = createDb(databaseUrl, { maxConnections: 1 });
try {
await server.settled();
const company = await server.api<{ id: string }>("POST", "/companies", { name: "Attention ingress" });
@@ -753,8 +750,7 @@ describeDatabase("assembled maintenance server production wiring", () => {
}
}, 150_000);
it("accepts the first ordinary merge confirmation through the assembled provider worker", async () => {
await clearDatabase();
maintenanceTest.concurrent("accepts the first ordinary merge confirmation through the assembled provider worker", async ({ databaseUrl, expect }) => {
const providerRequests: string[] = [];
const provider = createServer((req, res) => {
providerRequests.push(req.url!); res.setHeader("Content-Type", "application/json");
@@ -763,10 +759,10 @@ describeDatabase("assembled maintenance server production wiring", () => {
title: "Controlled merged confirmation", html_url: "https://github.com/maintenance/example/pull/39", user: { login: "fixture" } }));
});
await withMaintenanceProvider(provider, async origin => {
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
const fixture = createDb(databaseUrl, { maxConnections: 1 });
let runtime: Awaited<ReturnType<typeof startMaintenanceServer>> | undefined;
try {
const server = runtime = await startMaintenanceServer(databaseUrl!, { clock: true, suppressed: true,
const server = runtime = await startMaintenanceServer(databaseUrl, { clock: true, suppressed: true,
githubOrigin: origin });
await server.settled();
const company = await server.api<{ id: string }>("POST", "/companies", { name: "Merge ingress" });
@@ -794,22 +790,21 @@ describeDatabase("assembled maintenance server production wiring", () => {
});
}, 150_000);
it("projects the first ordinary routine completion in the actual suppressed Cloud runtime", async () => {
await clearDatabase();
maintenanceTest.concurrent("projects the first ordinary routine completion in the actual suppressed Cloud runtime", async ({ databaseUrl, expect }) => {
const companyId = randomUUID(), agentId = randomUUID();
const pair = generateKeyPairSync("ed25519");
const publicJwk = { ...pair.publicKey.export({ format: "jwk" }), kid: "maintenance-routine-key", use: "sig", alg: "EdDSA" };
const storageRequests: string[] = [];
const storage = createServer((req, res) => { storageRequests.push(req.url!); res.statusCode = 500; res.end("Unexpected S3 request in routine fixture"); });
await withMaintenanceProvider(storage, async origin => {
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
const fixture = createDb(databaseUrl, { maxConnections: 1 });
let runtime: Awaited<ReturnType<typeof startMaintenanceServer>> | undefined;
try {
await fixture.insert(companies).values({ id: companyId, name: "Routine ingress", issuePrefix: "MRT",
defaultResponsibleUserId: "maintenance-board", requireBoardApprovalForNewAgents: false });
await fixture.insert(agents).values({ id: agentId, companyId, name: "Suppressed routine worker", status: "active",
lifecycleState: "ready", adapterType: "process", adapterConfig: { command: "/usr/bin/false" } });
const server = runtime = await startMaintenanceServer(databaseUrl!, { clock: true, suppressed: true,
const server = runtime = await startMaintenanceServer(databaseUrl, { clock: true, suppressed: true,
cloud: { jwks: JSON.stringify({ keys: [publicJwk] }), s3Origin: origin, standby: false } });
await server.settled();
const routine = await server.api<{ id: string }>("POST", `/companies/${companyId}/routines`, {
@@ -871,8 +866,7 @@ describeDatabase("assembled maintenance server production wiring", () => {
});
}, 150_000);
it("keeps an actual Cloud warm standby quiet and activates after a normally signed durable claim", async () => {
await clearDatabase();
maintenanceTest.concurrent("keeps an actual Cloud warm standby quiet and activates after a normally signed durable claim", async ({ databaseUrl, expect }) => {
const pair = generateKeyPairSync("ed25519");
const publicJwk = { ...pair.publicKey.export({ format: "jwk" }), kid: "maintenance-test-key", use: "sig", alg: "EdDSA" };
const storageRequests: string[] = [];
@@ -880,7 +874,7 @@ describeDatabase("assembled maintenance server production wiring", () => {
await withMaintenanceProvider(storage, async origin => {
let runtime: Awaited<ReturnType<typeof startMaintenanceServer>> | undefined;
try {
const server = runtime = await startMaintenanceServer(databaseUrl!, { clock: true,
const server = runtime = await startMaintenanceServer(databaseUrl, { clock: true,
cloud: { jwks: JSON.stringify({ keys: [publicJwk] }), s3Origin: origin, standby: true } });
const standby = await server.settled(500, 1); await server.freeze();
expect(standby.idle.active).toBe(1);
@@ -904,7 +898,7 @@ describeDatabase("assembled maintenance server production wiring", () => {
await server.advance(30_000); const afterClaim = await server.settled();
expect(afterClaim.idle.active).toBe(0);
expect(server.proxy.queries.length).toBeGreaterThan(before);
const fixture = createDb(databaseUrl!, { maxConnections: 1 });
const fixture = createDb(databaseUrl, { maxConnections: 1 });
try {
const [transfers] = await fixture.$client`select count(*)::integer as count from company_transfer_runs`;
expect(transfers!.count).toBe(0);
@@ -0,0 +1,149 @@
import { startNativeSessionExecutionLeaseRenewal } from "./native-execution-lease.js";
import { writeFile } from "node:fs/promises";
import { resolve } from "node:path";
import { eq, sql } from "drizzle-orm";
import { checkpointDb, livenessDb, nativeRunFinalizations } from "@paperclipai/db";
import { beforeAll, expect } from "vitest";
import { createRunnerdCodexTransport, DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js";
import { registerRunnerPrpAuthority } from "../../realtime/runner-prp-ws.js";
import { nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js";
import { nativeRestartFixture, describeEmbeddedPostgres, realProcessIt, waitForCondition } from "../../__tests__/helpers/native-restart.js";
describeEmbeddedPostgres("native runner restart recovery: checkpoint-cost", () => {
const environment = nativeRestartFixture();
const { seedRun, transportOptions } = environment;
let runtimeRoot: typeof environment.current.runtimeRoot;
let companyId: typeof environment.current.companyId;
let successor: typeof environment.current.successor;
beforeAll(() => { ({ runtimeRoot, companyId, successor } = environment.current); });
realProcessIt("keeps checkpoint SQL independent of streamed output volume with a real Postgres writer", async () => {
const samples: Array<{ chunks: number; checkpointWrites: number; transactions: number;
changedRows: number; checkpointBytes: number; walBytes: number; elapsedMs: number; idleSql: number;
streamSql: number; streamTransactions: number; streamChangedRows: number; renewalSql: number }> = [];
for (const chunks of [8, 512, 8192]) {
const fixture = await seedRun(`CHECKPOINT-COST-${chunks}`);
const root = resolve(runtimeRoot, fixture.runId);
const releaseOutput = resolve(root, "release-output");
const binding = { companyId, issueId: fixture.issueId, runId: fixture.runId };
const owner = { ...binding, authorityRunId: fixture.runId, bootId: successor.bootId,
generation: 1, leaseOwner: "cost-controller" };
await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: owner.bootId,
controllerPid: successor.pid, controllerProcessStartedAt: successor.processStartedAt,
controllerGeneration: 1, leaseOwner: owner.leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000),
}).where(eq(nativeRunFinalizations.runId, fixture.runId));
let authority!: DurablePrpControlPlane;
let measuring = false;
let transactions = 0, changedRows = 0, checkpointBytes = 0;
const statements: string[] = [];
const renewalStatements: string[] = [];
let leaseLost = false;
const renewal = startNativeSessionExecutionLeaseRenewal({ db: fixture.db, ...binding, leaseOwner: owner.leaseOwner,
attempt: 0, controller: successor, remote: true, onLost: async () => { leaseLost = true; } });
const capturedClients = [fixture.db.$client, checkpointDb(fixture.db).$client, livenessDb(fixture.db).$client];
const priorDebug = capturedClients.map(client => client.options.debug);
// Capture statements only, never SQL parameters (encrypted state or secrets).
capturedClients.forEach((client, index) => {
client.options.debug = (_connection, query) => {
if (!measuring) return;
// Renewals are existing liveness traffic, reported separately. A
// checkpoint query on any pool still counts against the zero oracle.
if (index === 2 && !query.includes("native_runner_checkpoints")) renewalStatements.push(query);
else statements.push(query);
};
});
const write = nativeRunnerCheckpointWriter(fixture.db, owner);
let costFailure: unknown;
const transport = createRunnerdCodexTransport({ ...transportOptions(fixture, root),
codexArgs: ["--state-file", resolve(root, "provider.json"), "--linger-after-turn-start",
"--held-output", "--held-output-count", String(chunks), "--held-output-release-file", releaseOutput],
controlPlaneRegistration: async core => {
authority = core;
core.setCheckpointPersistence(async checkpoint => {
if (measuring) {
transactions++;
changedRows += checkpoint.changedRecords.size;
checkpointBytes += [...checkpoint.changedRecords.values()].reduce((sum, value) => sum + Buffer.byteLength(value), 0);
expect([...checkpoint.changedRecords.values()].join("\n")).not.toContain("still computing");
}
await write(checkpoint);
});
return registerRunnerPrpAuthority({ ...binding, authority: core });
},
});
try {
await transport.transport.request("thread/start", { cwd: root, dynamicTools: [] });
await authority.flushCheckpoint();
const [walStart] = await fixture.db.execute(sql`select pg_current_wal_insert_lsn()::text as lsn`);
const started = performance.now();
measuring = true;
await transport.transport.request("turn/start", { input: [{ type: "text", text: "Produce the scripted stream." }] });
await waitForCondition("settled control setup before releasing text", () =>
authority.store.state.commands.every(command => command.status === "completed")
&& authority.store.state.committedEvents.some(event => event.eventType === "turn.started"));
await authority.flushCheckpoint();
const beforeStream = { sql: statements.length, transactions, changedRows };
await writeFile(releaseOutput, "release\n", { flag: "wx" });
await waitForCondition(`the final scripted text chunk of ${chunks}`, () => authority.store.state.committedEvents
.some(event => event.eventType === "item.delta" && JSON.stringify(event.envelope).includes(`still computing ${chunks - 1} `)), chunks > 4096 ? 480_000 : 60_000)
.catch(error => { console.info("checkpoint-cost-stream-diagnostic", JSON.stringify({ chunks,
cursor: authority.store.state.ackedSourceSeq,
receivedTextEvents: authority.store.state.committedEvents.filter(event => event.eventType === "item.delta").length })); throw error; });
await authority.flushCheckpoint();
measuring = false;
const elapsedMs = performance.now() - started;
const [walEnd] = await fixture.db.execute(sql`select pg_wal_lsn_diff(pg_current_wal_insert_lsn(), ${String(walStart.lsn)}::pg_lsn)::text as bytes`);
const checkpointWrites = statements.filter(query => /^\s*(insert|update|delete)\b/i.test(query)
&& query.includes("native_runner_checkpoints")).length;
expect(checkpointWrites).toBeGreaterThan(0); // The collector must see real control writes.
const snapshot = { chunks, checkpointWrites, transactions, changedRows, checkpointBytes,
walBytes: Number(walEnd.bytes), elapsedMs, idleSql: 0, renewalSql: renewalStatements.length,
streamSql: statements.length - beforeStream.sql,
streamTransactions: transactions - beforeStream.transactions,
streamChangedRows: changedRows - beforeStream.changedRows };
// Setup control messages may coalesce differently. The released text-only
// interval must add exactly zero work, independent of that scheduling.
expect(snapshot).toMatchObject({ streamSql: 0, streamTransactions: 0, streamChangedRows: 0 });
expect(leaseLost).toBe(false);
statements.length = 0;
measuring = true;
await new Promise(done => setTimeout(done, 100));
await authority.flushCheckpoint();
snapshot.idleSql = statements.length;
expect(statements).toEqual([]);
// Positive calibration: an actual new control operation must trip the zero-write oracle.
authority.queueCommand("session.snapshot", {}, `cost-positive-control-${chunks}`);
await authority.flushCheckpoint();
expect(statements.some(query => /insert into "native_runner_checkpoints"/i.test(query))).toBe(true);
measuring = false;
samples.push(snapshot);
console.info("native-checkpoint-cost-sample", JSON.stringify(snapshot));
} catch (error) {
costFailure = error;
throw error;
} finally {
measuring = false;
capturedClients.forEach((client, index) => { client.options.debug = priorDebug[index]; });
try { await transport.transport.close(); }
catch (error) {
if (costFailure) throw new AggregateError([costFailure, error], "Checkpoint cost measurement and cleanup failed");
throw error;
}
finally {
await renewal.stop().catch(error => {
if (costFailure) throw new AggregateError([costFailure, error], "Checkpoint cost measurement and lease renewal failed");
throw error;
});
}
}
}
const report = { schema: "paperclip.checkpoint-cost/v1", samples,
scope: "isolated Postgres; control operations included; WAL is diagnostic; no production capacity claim" };
if (process.env.PAPERCLIP_TEST_CHECKPOINT_COST_REPORT) await writeFile(process.env.PAPERCLIP_TEST_CHECKPOINT_COST_REPORT,
`${JSON.stringify(report, null, 2)}\n`, { flag: "wx", mode: 0o600 });
// Each stream size was checked independently; total setup counts remain
// diagnostic because control-message batching is scheduling-dependent.
console.info("native-checkpoint-cost", JSON.stringify(report));
}, 540_000);
});
@@ -0,0 +1,173 @@
import { randomUUID } from "node:crypto";
import { createRunnerProviderLaunch, runnerLaunchRequirementsDigest, type RunnerProviderLaunchRequirements } from "@paperclipai/paperclip-runner";
import { spawn } from "node:child_process";
import { existsSync } from "node:fs";
import { mkdir, readFile, rm } from "node:fs/promises";
import { resolve } from "node:path";
import { eq } from "drizzle-orm";
import { environmentLeases, environments, heartbeatRuns, nativeRunFinalizations } from "@paperclipai/db";
import { beforeAll, expect, vi } from "vitest";
import {
createRunnerdCodexTransport,
defaultCapabilityRunnerdBinary,
DurablePrpControlPlane,
readRunnerdArtifactBinding,
type RunnerProcessHandle,
type RunnerProcessLaunchSpec,
} from "../../vendor/paperclip-runner/index.js";
import { registerRunnerPrpAuthority } from "../../realtime/runner-prp-ws.js";
import { readProcessStartedAt } from "../hot-restart.js";
import { loadNativeRunnerCheckpoint, registerNativeRunnerCheckpoint, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "./native-runner-recovery-contract.js";
import { claimNativeRestartRecoveries } from "./native-restart-recovery.js";
import {
nativeRestartFixture,
describeEmbeddedPostgres,
realProcessIt,
fakeCodexAppServer,
processAlive,
waitForCondition,
stopOwnedProcessGroup,
} from "../../__tests__/helpers/native-restart.js";
describeEmbeddedPostgres("native runner restart recovery: diskless", () => {
const environment = nativeRestartFixture();
const { seedRun, transportOptions } = environment;
let runtimeRoot: typeof environment.current.runtimeRoot;
let companyId: typeof environment.current.companyId;
let agentId: typeof environment.current.agentId;
let successor: typeof environment.current.successor;
beforeAll(() => { ({ runtimeRoot, companyId, agentId, successor } = environment.current); });
realProcessIt.each([false, true])("adopts the same runner from Postgres after losing the controller filesystem (prepared task: %s)", async preparedTask => {
const fixture = await seedRun(`EMPTY-CONTROLLER-DISK-${preparedTask}`);
const controllerRoot = resolve(runtimeRoot, `${fixture.runId}-controller`);
const sandboxRoot = resolve(runtimeRoot, `${fixture.runId}-sandbox`);
const binding = { companyId, issueId: fixture.issueId, runId: fixture.runId };
const outageMs = Number(process.env.PAPERCLIP_TEST_CONTROLLER_OUTAGE_MS ?? 1000);
if (!Number.isFinite(outageMs) || outageMs < 0 || outageMs > 300_000) throw new Error("Invalid controller outage duration");
await mkdir(sandboxRoot, { recursive: true });
await mkdir(resolve(sandboxRoot, "filesystem/codex-home"), { recursive: true });
let bootId = randomUUID();
const taskLeaseId = randomUUID();
if (preparedTask) {
const [environment] = await fixture.db.insert(environments).values({ name: "Surviving plugin task", driver: "plugin", status: "active", config: {} }).returning();
await fixture.db.insert(environmentLeases).values({ id: taskLeaseId, companyId, environmentId: environment.id,
heartbeatRunId: fixture.runId, issueId: fixture.issueId, status: "active", leasePolicy: "ephemeral",
provider: "plugin", providerLeaseId: "saved-task", metadata: { agentId, runnerTaskLease: true, runnerTaskDispatched: true } });
}
const originalLeaseExpiresAt = new Date(Date.now() + 60_000);
await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: bootId,
controllerGeneration: 1, controllerPid: process.pid, controllerProcessStartedAt: new Date(),
leaseOwner: "original-controller", leaseExpiresAt: originalLeaseExpiresAt,
}).where(eq(nativeRunFinalizations.runId, fixture.runId));
const options = {
...transportOptions(fixture, controllerRoot),
...(preparedTask ? { prpIdentity: { ...transportOptions(fixture, controllerRoot).prpIdentity, environmentLeaseId: taskLeaseId } } : {}),
runnerStateDirectory: resolve(sandboxRoot, "runner"),
runnerFilesystemRoot: resolve(sandboxRoot, "filesystem"),
runnerReconnectGraceMs: REMOTE_RUNNER_RECONNECT_GRACE_MS,
codexArgs: ["--state-file", resolve(sandboxRoot, "provider.json"),
"--call-log", resolve(sandboxRoot, "calls.log"), "--linger-after-turn-start", "--held-output"],
controlPlaneRegistration: async (authority: DurablePrpControlPlane) => {
await registerNativeRunnerCheckpoint({ db: fixture.db, binding, bootId, authority });
return registerRunnerPrpAuthority({ ...binding, authority });
},
};
const prepareLaunch = vi.fn(async (requirements: RunnerProviderLaunchRequirements) => ({
configurationId: "provider-owned-configuration", requirementsDigest: runnerLaunchRequirementsDigest(requirements),
provider: createRunnerProviderLaunch(requirements, { options: { codexArgs: options.codexArgs },
root: sandboxRoot, cwd: sandboxRoot, codexHome: resolve(sandboxRoot, "filesystem/codex-home"),
codexCommand: fakeCodexAppServer, providerNodeCommand: process.execPath, opencodeProxyPath: "", acpxSidecarPath: "" }),
}));
const first = createRunnerdCodexTransport({ ...options, ...(preparedTask ? {
prepareLaunch, runnerArtifact: readRunnerdArtifactBinding(defaultCapabilityRunnerdBinary()),
runnerProcessLauncher: (spec: RunnerProcessLaunchSpec): RunnerProcessHandle => {
const child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: { ...process.env, ...spec.environment },
detached: process.platform !== "win32", stdio: "ignore" });
return { child, processGroupId: child.pid ?? null, completion: new Promise((resolveExit, reject) => {
child.once("error", reject);
child.once("exit", (code, signal) => resolveExit({ code, signal, stdout: "", stderr: "" }));
}) };
},
} : {}) });
let adopted: ReturnType<typeof createRunnerdCodexTransport> | undefined;
let runnerPid: number | null = null;
try {
const thread = (await first.transport.request("thread/start", { cwd: sandboxRoot, dynamicTools: [] })).thread as Record<string, unknown>;
const turn = (await first.transport.request("turn/start", { input: [{ type: "text", text: "Keep computing while the controller is offline." }] })).turn as Record<string, unknown>;
runnerPid = first.evidence().runnerPid;
const providerPid = first.evidence().providerPid;
if (!runnerPid || !providerPid) throw new Error("Runner/provider process identity missing");
const startedAt = await readProcessStartedAt(runnerPid);
if (!startedAt) throw new Error("Runner start identity missing");
await waitForCondition("streaming ACK advances without a recovery checkpoint", async () => {
const runner = JSON.parse(await readFile(resolve(sandboxRoot, "runner/runner-state.json"), "utf8"));
const checkpoint = await loadNativeRunnerCheckpoint(fixture.db, binding);
const cursor = checkpoint && JSON.parse(checkpoint.records.get("cursor")!);
return cursor && runner.ackedSourceSeq >= cursor.ackedSourceSeq + 64;
});
const beforeRestart = (await loadNativeRunnerCheckpoint(fixture.db, binding))!;
expect([...beforeRestart.records.entries()].filter(([key]) => key.startsWith("event:"))
.some(([, value]) => JSON.parse(value).eventType === "item.delta")).toBe(false);
await fixture.db.update(heartbeatRuns).set({ processPid: runnerPid, processStartedAt: new Date(startedAt),
runnerProfileJson: { ...fixture.preparedProfile,
...(!preparedTask ? { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v1", state: "prepared",
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64),
workspaceId: "workspace", leaseId: "lease", providerLeaseId: "sandbox", remoteCwd: sandboxRoot } } : {}),
sessionCheckpoint: { identity: { ...binding, agentId, sessionId: fixture.native.normalizedSessionId },
providerSessionId: thread.id, process: { providerPid } },
},
}).where(eq(heartbeatRuns.id, fixture.runId));
await first.detachControllerForRestart();
await rm(controllerRoot, { recursive: true, force: true });
expect(existsSync(controllerRoot)).toBe(false);
await new Promise(resolveOutage => setTimeout(resolveOutage, outageMs));
expect(processAlive(runnerPid)).toBe(true);
expect(processAlive(providerPid)).toBe(true);
bootId = randomUUID();
if (originalLeaseExpiresAt.getTime() > Date.now()) {
const [waiting] = await claimNativeRestartRecoveries({ db: fixture.db, controller: { ...successor, bootId },
restartKind: "hard", runIds: [fixture.runId] });
expect(waiting).toMatchObject({ kind: "awaiting_evidence", reason: "remote_controller_lease_live", retryAt: originalLeaseExpiresAt });
await new Promise(resolveExpiry => setTimeout(resolveExpiry, Math.max(0, originalLeaseExpiresAt.getTime() - Date.now()) + 20));
}
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: { ...successor, bootId },
restartKind: "hard", runIds: [fixture.runId] });
expect(claim).toMatchObject({ kind: "reattach_remote_runner", providerAttempt: 0, controllerGeneration: 2 });
if (preparedTask) expect(claim).toMatchObject({ remote: { kind: "plugin_task", leaseId: taskLeaseId } });
const checkpoint = await restoreNativeRunnerCheckpoint(fixture.db, binding, controllerRoot);
if (!checkpoint) throw new Error("Missing durable authority");
const duplicateLauncher = vi.fn(() => { throw new Error("duplicate runner spawn attempted"); });
prepareLaunch.mockClear();
const authorizeRunnerAdoption = vi.fn(async () => {});
adopted = createRunnerdCodexTransport({ ...options,
...(preparedTask ? { prepareLaunch, authorizeRunnerAdoption } : {}),
runnerBinary: resolve(controllerRoot, "new-release-does-not-contain-old-runner"),
runnerArtifact: { version: checkpoint.runnerVersion, digest: checkpoint.runnerDigest },
resumeDynamicTools: [], runnerProcessLauncher: duplicateLauncher,
adoptExistingRunner: { pid: preparedTask ? null : runnerPid, processGroupId: preparedTask ? null : first.evidence().runnerProcessGroupId,
startedAt: preparedTask ? null : startedAt, isAlive: () => processAlive(runnerPid) },
});
await expect(adopted.transport.request("thread/read", {})).resolves.toMatchObject({
thread: { id: thread.id, turns: [{ id: turn.id, status: "inProgress" }] },
});
await adopted.transport.request("turn/steer", { input: [{ type: "text", text: "Continue the same turn." }],
expectedTurnId: turn.id, correlationId: "after-empty-disk-restart" });
expect(adopted.evidence().runnerPid).toBe(preparedTask ? null : runnerPid);
expect(adopted.evidence().providerPid).toBe(providerPid);
expect(await readProcessStartedAt(runnerPid)).toBe(startedAt);
expect(duplicateLauncher).not.toHaveBeenCalled();
expect(prepareLaunch).not.toHaveBeenCalled();
if (preparedTask) expect(authorizeRunnerAdoption).toHaveBeenCalledExactlyOnceWith("provider-owned-configuration", options.prpIdentity);
const calls = await readFile(resolve(sandboxRoot, "calls.log"), "utf8");
expect(calls.match(/^turn\/start$/gm)).toHaveLength(1);
expect(calls.match(/^turn\/steer$/gm)).toHaveLength(1);
} finally {
await adopted?.transport.close().catch(() => undefined);
await first.transport.close().catch(() => undefined);
await stopOwnedProcessGroup(runnerPid, sandboxRoot).catch(() => undefined);
}
}, 345_000);
});
@@ -0,0 +1,65 @@
import { startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js";
import { eq } from "drizzle-orm";
import { createDb, closeLivenessDb, livenessDb, nativeRunFinalizations } from "@paperclipai/db";
import { beforeAll, expect, it, vi } from "vitest";
import { nativeRestartFixture, describeEmbeddedPostgres } from "../../__tests__/helpers/native-restart.js";
describeEmbeddedPostgres("native runner restart recovery: lease-renewal", () => {
const environment = nativeRestartFixture();
const { seedRun } = environment;
let temporary: typeof environment.current.temporary;
let companyId: typeof environment.current.companyId;
let successor: typeof environment.current.successor;
beforeAll(() => { ({ temporary, companyId, successor } = environment.current); });
it("retains remote ownership through two delayed successful liveness-pool renewals", async () => {
const fixture = await seedRun("DELAYED-RENEWAL-SUCCESS");
const owner = createDb(temporary.connectionString);
const pool = livenessDb(owner);
const input = { db: owner, runId: fixture.runId, companyId, issueId: fixture.issueId,
controller: successor, leaseOwner: "delayed-renewal-owner", attempt: 1, remote: true,
onLost: vi.fn(async () => undefined) };
const initialExpiry = new Date(Date.now() + 60_000);
await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: successor.bootId,
controllerPid: successor.pid, controllerProcessStartedAt: successor.processStartedAt,
leaseOwner: input.leaseOwner, attempt: 1, leaseExpiresAt: initialExpiry })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
let release: (() => void) | undefined;
let occupied: Promise<unknown> | undefined;
let guard: NativeExecutionLeaseRenewal | undefined;
const occupyPool = async () => {
let entered = 0;
let ready!: () => void;
const acquired = new Promise<void>(resolve => { ready = resolve; });
const barrier = new Promise<void>(resolve => { release = resolve; });
occupied = Promise.all([0, 1].map(() => pool.transaction(async () => {
if (++entered === 2) ready();
await barrier;
})));
await acquired;
};
const savedExpiry = async () => {
const [saved] = await fixture.db.select({ expires: nativeRunFinalizations.leaseExpiresAt })
.from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
return saved.expires!.getTime();
};
try {
await occupyPool();
guard = startNativeSessionExecutionLeaseRenewal(input);
await new Promise(resolve => setTimeout(resolve, 50_000));
release!(); await occupied;
await vi.waitFor(async () => expect(await savedExpiry()).toBeGreaterThan(initialExpiry.getTime() + 40_000));
const firstExpiry = await savedExpiry();
await occupyPool();
await new Promise(resolve => setTimeout(resolve, 40_000));
release!(); await occupied;
await vi.waitFor(async () => expect(await savedExpiry()).toBeGreaterThan(firstExpiry + 30_000));
expect(input.onLost).not.toHaveBeenCalled();
await guard.stop();
} finally {
release?.(); await occupied; await guard?.stop().catch(() => undefined);
await closeLivenessDb(owner, 1); await owner.$client.end({ timeout: 1 });
}
}, 120_000);
});
@@ -0,0 +1,164 @@
import { startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js";
import { randomUUID } from "node:crypto";
import { resolve } from "node:path";
import { eq, sql } from "drizzle-orm";
import { environmentLeases, environments, heartbeatRuns, nativeRunFinalizations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
import { beforeAll, expect, it, vi } from "vitest";
import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js";
import { buildNativeExecutionInput } from "./native-execution-input.js";
import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
import { executePaperclipNativeSession } from "./native-session-executor.js";
import { registerNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
import { claimNativeRestartRecoveries } from "./native-restart-recovery.js";
import { remotePreparedWorkspace, nativeRestartFixture, describeEmbeddedPostgres } from "../../__tests__/helpers/native-restart.js";
describeEmbeddedPostgres("native runner restart recovery: preparation", () => {
const environment = nativeRestartFixture();
const { seedRun, transportOptions } = environment;
let runtimeRoot: typeof environment.current.runtimeRoot;
let companyId: typeof environment.current.companyId;
let agentId: typeof environment.current.agentId;
let successor: typeof environment.current.successor;
beforeAll(() => { ({ runtimeRoot, companyId, agentId, successor } = environment.current); });
it.each(["preparation", "claim lock"])("gives a fresh remote claim its full lease through blocked %s", async (blockedPhase) => {
const fixture = await seedRun(`FRESH-SLOW-${blockedPhase === "claim lock" ? "CLAIM-LOCK" : "PREPARATION"}`);
const environmentId = randomUUID(), leaseId = randomUUID();
await fixture.db.insert(environments).values({ id: environmentId, name: `Fresh preparation fixture ${environmentId}`, driver: "sandbox" });
await fixture.db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId,
heartbeatRunId: fixture.runId, issueId: fixture.issueId, status: "active",
provider: "fixture", providerLeaseId: `fresh-preparation-${leaseId}` });
const execution = buildNativeExecutionInput({ companyId, agentId, runId: fixture.runId,
issue: { id: fixture.issueId, identifier: "NRR-FRESH-SLOW", title: "Prepare slowly", description: null, workMode: "standard" },
taskPrompt: "Prepare a fresh execution.", provider: "codex",
workspace: { id: fixture.runId, cwd: runtimeRoot, repoUrl: null, repoRef: null, branchName: null },
normalizedSessionId: fixture.native.normalizedSessionId, runtimeContext: nativeRuntimeContextFixture(),
completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1",
contract: { revision: "1", objective: "Prepare once", criteria: [{ id: "objective", requirement: "Retain ownership" }] } },
});
// Stop at the provider boundary after checking ownership; no harness is launched.
const failure = new Error("end fresh preparation experiment");
const backend = { descriptor: async () => { throw failure; }, openSession: async () => { throw failure; } };
let reached!: () => void, release!: () => void;
const claimed = new Promise<void>(resolveReady => { reached = resolveReady; });
const preparation = new Promise<void>(resolveReady => { release = resolveReady; });
let releaseLock: (() => void) | undefined;
let holder: Promise<void> | undefined;
let lockPid: number | undefined;
let leaseStartsAfter: Date | undefined;
if (blockedPhase === "claim lock") {
let locked!: () => void;
const lockTaken = new Promise<void>(resolve => { locked = resolve; });
const lockReleased = new Promise<void>(resolve => { releaseLock = resolve; });
holder = fixture.db.transaction(async tx => {
const [connection] = await tx.execute(sql`select pg_backend_pid() as pid`);
lockPid = Number(connection.pid);
await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)).for("update");
locked(); await lockReleased;
});
await lockTaken;
}
const running = executePaperclipNativeSession({ db: fixture.db, execution,
runnerInstanceId: fixture.native.runnerInstanceId, backend,
runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: runtimeRoot },
onEvent: async event => {
if (event.payload?.span === "native.coordinator.claim") { reached(); await preparation; }
},
}).catch(error => error);
try {
if (blockedPhase === "claim lock") {
// Observe the real claim waiting on the run row after it has locked its
// coordinator. The lease must start after this wait, not before it.
await expect.poll(async () => {
const [waiting] = await fixture.db.execute(sql`select exists (
select 1 from pg_stat_activity where ${lockPid} = any(pg_blocking_pids(pid))
and query like 'select %from "heartbeat_runs"%for update%'
) as blocked`);
return waiting.blocked;
}, { timeout: 10_000 }).toBe(true);
await fixture.db.execute(sql`select pg_sleep(1)`);
const [observed] = await fixture.db.execute(sql`select clock_timestamp() as observed_at`);
leaseStartsAfter = new Date(String(observed.observed_at));
releaseLock!(); await holder;
}
await Promise.race([claimed, running.then(error => { throw error; })]);
const [initial] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
if (leaseStartsAfter) {
expect(initial.leaseExpiresAt!.getTime() - leaseStartsAfter.getTime()).toBeGreaterThan(59_900);
release();
expect(await running).toBe(failure);
return;
}
await new Promise(resolveWait => setTimeout(resolveWait, Math.max(0, initial.leaseExpiresAt!.getTime() - Date.now() + 2_000)));
const [current] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(current.leaseExpiresAt!.getTime()).toBeGreaterThan(Date.now());
expect(current.leaseOwner).toBe(initial.leaseOwner);
expect(current.controllerGeneration).toBe(initial.controllerGeneration);
expect(current.attempt).toBe(1);
release();
expect(await running).toBe(failure);
} finally { releaseLock?.(); await holder; release(); await running; }
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
it("renews a remote claim through slow preparation and shares renewal with execution", async () => {
const fixture = await seedRun("SLOW-RECOVERY-PREPARATION");
const root = resolve(runtimeRoot, fixture.runId);
const binding = { companyId, issueId: fixture.issueId, runId: fixture.runId };
await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: {
...fixture.preparedProfile, nativeWorkspaceSync: remotePreparedWorkspace(),
sessionCheckpoint: { identity: { ...binding, agentId, sessionId: fixture.native.normalizedSessionId },
providerSessionId: "surviving-provider" },
} }).where(eq(heartbeatRuns.id, fixture.runId));
await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: successor.bootId,
controllerGeneration: 1, leaseOwner: "previous-owner", leaseExpiresAt: new Date(Date.now() + 60_000) })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
const authority = new DurablePrpControlPlane({ stateDirectory: resolve(root, "control-plane"),
identity: transportOptions(fixture, root).prpIdentity,
expectedRunnerVersion: "lease-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
await registerNativeRunnerCheckpoint({ db: fixture.db, binding, bootId: successor.bootId, authority });
await authority.stop();
// Establish the abandoned predecessor before the fault; claimed leases are
// never edited below. The successor must retain ownership for a real minute.
await fixture.db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(0) })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
const controller = { ...successor, bootId: randomUUID() };
const onLost = vi.fn(async () => undefined);
let preparation: NativeExecutionLeaseRenewal | undefined;
let execution: NativeExecutionLeaseRenewal | undefined;
let acquired = false;
try {
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller,
restartKind: "hard", runIds: [fixture.runId], onClaim: (claim, scope) => {
expect(scope.remote).toBe(true);
preparation = startNativeSessionExecutionLeaseRenewal({ db: fixture.db, ...scope,
runId: claim.runId, leaseOwner: claim.leaseOwner, attempt: claim.providerAttempt, onLost });
acquired = true;
} });
expect(acquired).toBe(true);
expect(claim.kind).toBe("reattach_remote_runner");
if (!("leaseOwner" in claim)) throw new Error("Expected recovery claim");
const [initial] = await fixture.db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId));
await new Promise(resolveWait => setTimeout(resolveWait, 30_000));
execution = startNativeSessionExecutionLeaseRenewal({ db: fixture.db, ...binding,
leaseOwner: claim.leaseOwner, attempt: claim.providerAttempt, controller, remote: true, onLost });
await preparation!.stop();
await preparation!.stop(); // A caller cannot release another caller's reference.
const remaining = Math.max(0, initial.leaseExpiresAt!.getTime() - Date.now() + 2_000);
await new Promise(resolveWait => setTimeout(resolveWait, remaining));
const [current] = await fixture.db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(current.leaseExpiresAt!.getTime()).toBeGreaterThan(Date.now());
expect(current.controllerGeneration).toBe(claim.controllerGeneration);
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: { ...controller, bootId: randomUUID() },
restartKind: "hard", runIds: [fixture.runId] })).toEqual([
expect.objectContaining({ kind: "awaiting_evidence", runId: fixture.runId }),
]);
expect(onLost).not.toHaveBeenCalled();
} finally {
await preparation?.stop();
await execution?.stop();
}
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
});
@@ -1,17 +1,13 @@
import { recoveryPostgresSupport, startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js";
import { renewNativeSessionExecutionLease, startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js";
import { enableRunFailureReporting } from "../../__tests__/helpers/run-failure-reporting.js";
import { hasNativeLocalProcessStop } from "../native-local-process-stop.js";
import { randomUUID } from "node:crypto";
import { createRunnerProviderLaunch, runnerLaunchRequirementsDigest, type RunnerProviderLaunchRequirements } from "@paperclipai/paperclip-runner";
import { createRequire } from "node:module";
import { spawn } from "node:child_process";
import { existsSync } from "node:fs";
import { mkdir, mkdtemp, readFile, readdir, rename, rm, writeFile } from "node:fs/promises";
import { createServer, type Server } from "node:http";
import { mkdir, readFile, readdir, rename, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { resolve } from "node:path";
import { and, eq, inArray, sql } from "drizzle-orm";
import {
agents,
@@ -35,28 +31,17 @@ import {
nativeRunnerCheckpoints,
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
} from "@paperclipai/db";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { beforeAll, expect, it, vi } from "vitest";
import { heartbeatService } from "../heartbeat.js";
import {
canonicalNativeRuntimeContextDigest,
createRunnerdCodexTransport,
defaultCapabilityRunnerdBinary,
parseNativeExecutionInput,
DurablePrpControlPlane,
decodeDurablePrpCheckpoint,
readRunnerdArtifactBinding,
type RunnerProcessHandle,
type RunnerProcessLaunchSpec,
} from "../../vendor/paperclip-runner/index.js";
import {
startEmbeddedPostgresTestDatabase,
} from "../../__tests__/helpers/embedded-postgres.js";
import {
registerRunnerPrpAuthority,
runnerPrpWebSocketInternals,
setupRunnerPrpWebSocketServer,
} from "../../realtime/runner-prp-ws.js";
import { registerRunnerPrpAuthority } from "../../realtime/runner-prp-ws.js";
import { readProcessStartedAt } from "../hot-restart.js";
import { resolvePaperclipInstanceRoot } from "../../home-paths.js";
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
@@ -65,12 +50,22 @@ import { buildNativeExecutionWithCheckpoint, findNativeSessionResumeRun, nativeT
import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
import { materializeAsset } from "./runtime-context.js";
import { createRunnerTestTlsTunnel, dockerCommandRunner } from "../../__tests__/helpers/docker-command-runner.js";
import { createIsolatedTestHome } from "../../__tests__/helpers/isolated-test-home.js";
import { finalizeNativeRun } from "./native-run-finalizer.js";
import { waitForPendingRunFailureReports } from "../run-failure-report.js";
import { closeIdleWarmNativeSessionsForRestart, detachNativeSessionsForRestart, executePaperclipNativeSession, NativeControllerDetachedForRestartError } from "./native-session-executor.js";
import { listExpiredNativeRunnerAuthorities, loadNativeRunnerCheckpoint, nativeRunnerCheckpointRegistration, nativeRunnerCheckpointWriter, registerNativeRunnerCheckpoint, restoreNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "./native-runner-recovery-contract.js";
import {
closeIdleWarmNativeSessionsForRestart,
detachNativeSessionsForRestart,
executePaperclipNativeSession,
NativeControllerDetachedForRestartError,
} from "./native-session-executor.js";
import {
listExpiredNativeRunnerAuthorities,
loadNativeRunnerCheckpoint,
nativeRunnerCheckpointRegistration,
nativeRunnerCheckpointWriter,
registerNativeRunnerCheckpoint,
restoreNativeRunnerCheckpoint,
} from "./native-runner-checkpoint.js";
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
import { RUNNER_BRIDGE_READ_OPERATIONS, RUNNER_BRIDGE_SCHEMAS } from "./runner-bridge-tools.js";
import { HttpError } from "../../errors.js";
@@ -84,12 +79,7 @@ vi.mock("../../sentry.js", async () => {
captureRunFailure: mockCaptureRunFailure,
};
});
import {
claimNativeRestartRecoveries,
currentNativeControllerIdentity,
type NativeControllerIdentity,
} from "./native-restart-recovery.js";
import { claimNativeRestartRecoveries, currentNativeControllerIdentity, type NativeControllerIdentity } from "./native-restart-recovery.js";
// Exact v5 wire prompt from before the task-monitor upgrade (#15446).
const historicalPrompt = {
@@ -104,224 +94,25 @@ const unregisteredPrompt = {
text: "Complete the assigned task using the saved instructions.",
digest: "b0cab3306694028624bbccbf40984a7aca7c809f77b29bfd959325a2af3f7a66",
} as const;
// An explicitly supplied disposable database also permits qualification on hosts
// where native Postgres cannot allocate semaphores. Never fall back silently.
const describeEmbeddedPostgres = recoveryPostgresSupport.supported ? describe : describe.skip;
const fakeCodexAppServer = resolve(
import.meta.dirname,
"../../../../packages/paperclip-runner/runner/target/debug/fake-codex-app-server",
);
const binariesAvailable =
existsSync(defaultCapabilityRunnerdBinary()) && existsSync(fakeCodexAppServer);
const realProcessIt = binariesAvailable ? it : it.skip;
async function closeServer(server: Server | null): Promise<void> {
if (!server) return;
server.closeAllConnections();
if (!server.listening) return;
await new Promise<void>((resolveClose) => server.close(() => resolveClose()));
}
function processAlive(pid: number | null): boolean {
if (!pid) return false;
try {
process.kill(pid, 0);
return true;
} catch {
return false;
}
}
async function waitForCondition(
description: string,
condition: () => boolean | Promise<boolean>,
timeoutMs = 10_000,
): Promise<void> {
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
if (await condition()) return;
await new Promise<void>((resolveWait) => setTimeout(resolveWait, 25));
}
throw new Error(`Timed out waiting for ${description}`);
}
async function stopOwnedProcessGroup(
pid: number | null,
expectedStateRoot: string,
): Promise<void> {
if (!pid || !processAlive(pid)) return;
const command = await import("node:child_process").then(
({ execFileSync }) =>
execFileSync("ps", ["-p", String(pid), "-o", "command="], {
encoding: "utf8",
}).trim(),
);
if (!command.includes(expectedStateRoot)) {
throw new Error(`Refusing to signal process ${pid}; ownership changed`);
}
if (process.platform !== "win32") process.kill(-pid, "SIGKILL");
else process.kill(pid, "SIGKILL");
await waitForCondition(`owned process ${pid} to exit`, () => !processAlive(pid));
}
import {
remotePreparedWorkspace,
nativeRestartFixture,
describeEmbeddedPostgres,
realProcessIt,
processAlive,
waitForCondition,
stopOwnedProcessGroup,
} from "../../__tests__/helpers/native-restart.js";
describeEmbeddedPostgres("native runner restart recovery with real processes", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let runtimeRoot: string;
let paperclipHome: string;
let server: Server | null = null;
let apiUrl: string;
const originalPaperclipHome = process.env.PAPERCLIP_HOME;
let isolatedHome: ReturnType<typeof createIsolatedTestHome>;
const companyId = randomUUID();
const agentId = randomUUID();
let successor: NativeControllerIdentity;
beforeAll(async () => {
temporary = await startRecoveryPostgres("native-runner-restart-recovery-");
runtimeRoot = await mkdtemp(resolve(tmpdir(), "native-restart-runtime-"));
isolatedHome = createIsolatedTestHome();
paperclipHome = isolatedHome.home;
process.env.PAPERCLIP_HOME = paperclipHome;
const controllerStartedAt = await readProcessStartedAt(process.pid);
successor = {
bootId: randomUUID(),
pid: process.pid,
processStartedAt: controllerStartedAt
? new Date(controllerStartedAt)
: new Date(),
};
server = createServer();
await new Promise<void>((resolveListen) =>
server!.listen(0, "127.0.0.1", resolveListen),
);
const address = server.address();
if (!address || typeof address === "string") {
throw new Error("Expected restart recovery TCP listener");
}
apiUrl = `http://127.0.0.1:${address.port}`;
setupRunnerPrpWebSocketServer(server, { apiUrl });
const db = createDb(temporary.connectionString);
await db.insert(companies).values({
id: companyId,
name: "Native restart recovery",
issuePrefix: "NRR",
requireBoardApprovalForNewAgents: false,
});
await db.insert(agents).values({
id: agentId,
companyId,
name: "Native restart runner",
role: "engineer",
status: "active",
adapterType: "paperclip_runner",
adapterConfig: { provider: "codex" },
runtimeConfig: {},
permissions: {},
});
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => {
runnerPrpWebSocketInternals.resetForTests();
await closeServer(server);
await temporary?.cleanup();
if (runtimeRoot) await rm(runtimeRoot, { recursive: true, force: true });
isolatedHome?.cleanup();
if (originalPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME;
else process.env.PAPERCLIP_HOME = originalPaperclipHome;
});
async function seedRun(
label: string,
existingDb?: ReturnType<typeof createDb>,
) {
const db = existingDb ?? createDb(temporary.connectionString);
const issueId = randomUUID();
const runId = randomUUID();
await db.insert(issues).values({
id: issueId,
companyId,
identifier: `NRR-${label}`,
title: `Recover ${label}`,
status: "in_progress",
priority: "medium",
workMode: "standard",
assigneeAgentId: agentId,
});
const [run] = await db
.insert(heartbeatRuns)
.values({
id: runId,
companyId,
agentId,
status: "running",
runtimeMode: "native",
nativeIssueId: issueId,
invocationSource: "assignment",
triggerDetail: "system",
contextSnapshot: { issueId },
})
.returning();
if (!run) throw new Error("Failed to seed native restart run");
await db
.update(issues)
.set({ executionRunId: runId })
.where(eq(issues.id, issueId));
const native = await prepareNativeHeartbeatRun({
db,
run,
issue: {
id: issueId,
title: `Recover ${label}`,
description: null,
reviewPolicy: null,
},
environmentLeaseId: `lease-${label}`,
});
await db.insert(nativeRunFinalizations).values({
runId,
companyId,
issueId,
phase: "observed",
});
const [preparedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
return { db, issueId, runId, native, preparedProfile: preparedRun.runnerProfileJson };
}
function transportOptions(
fixture: Awaited<ReturnType<typeof seedRun>>,
stateDirectory: string,
) {
return {
runnerBinary: defaultCapabilityRunnerdBinary(),
codexCommand: fakeCodexAppServer,
codexArgs: [
"--state-file",
resolve(stateDirectory, "fake-codex-state.json"),
"--call-log",
resolve(stateDirectory, "fake-codex-calls.log"),
],
stateDirectory,
lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 },
prpIdentity: {
runnerInstanceId: fixture.native.runnerInstanceId,
environmentLeaseId: fixture.native.environmentLeaseId,
runId: fixture.runId,
normalizedSessionId: fixture.native.normalizedSessionId,
turnId: fixture.native.turnId,
itemId: fixture.native.itemId,
},
controlPlaneRegistration: (authority: Parameters<typeof registerRunnerPrpAuthority>[0]["authority"]) =>
registerRunnerPrpAuthority({
companyId,
runId: fixture.runId,
authority,
}),
};
}
const environment = nativeRestartFixture();
const { seedRun, transportOptions } = environment;
let temporary: typeof environment.current.temporary;
let runtimeRoot: typeof environment.current.runtimeRoot;
let companyId: typeof environment.current.companyId;
let agentId: typeof environment.current.agentId;
let successor: typeof environment.current.successor;
beforeAll(() => { ({ temporary, runtimeRoot, companyId, agentId, successor } = environment.current); });
it.each(["issue", "run", "coordinator"] as const)("defers a busy %s row without blocking replacement startup", async kind => {
const fixture = await seedRun(`OVERLAP-BUSY-${kind}`);
@@ -1066,56 +857,6 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
}
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
it("retains remote ownership through two delayed successful liveness-pool renewals", async () => {
const fixture = await seedRun("DELAYED-RENEWAL-SUCCESS");
const owner = createDb(temporary.connectionString);
const pool = livenessDb(owner);
const input = { db: owner, runId: fixture.runId, companyId, issueId: fixture.issueId,
controller: successor, leaseOwner: "delayed-renewal-owner", attempt: 1, remote: true,
onLost: vi.fn(async () => undefined) };
const initialExpiry = new Date(Date.now() + 60_000);
await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: successor.bootId,
controllerPid: successor.pid, controllerProcessStartedAt: successor.processStartedAt,
leaseOwner: input.leaseOwner, attempt: 1, leaseExpiresAt: initialExpiry })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
let release: (() => void) | undefined;
let occupied: Promise<unknown> | undefined;
let guard: NativeExecutionLeaseRenewal | undefined;
const occupyPool = async () => {
let entered = 0;
let ready!: () => void;
const acquired = new Promise<void>(resolve => { ready = resolve; });
const barrier = new Promise<void>(resolve => { release = resolve; });
occupied = Promise.all([0, 1].map(() => pool.transaction(async () => {
if (++entered === 2) ready();
await barrier;
})));
await acquired;
};
const savedExpiry = async () => {
const [saved] = await fixture.db.select({ expires: nativeRunFinalizations.leaseExpiresAt })
.from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
return saved.expires!.getTime();
};
try {
await occupyPool();
guard = startNativeSessionExecutionLeaseRenewal(input);
await new Promise(resolve => setTimeout(resolve, 50_000));
release!(); await occupied;
await vi.waitFor(async () => expect(await savedExpiry()).toBeGreaterThan(initialExpiry.getTime() + 40_000));
const firstExpiry = await savedExpiry();
await occupyPool();
await new Promise(resolve => setTimeout(resolve, 40_000));
release!(); await occupied;
await vi.waitFor(async () => expect(await savedExpiry()).toBeGreaterThan(firstExpiry + 30_000));
expect(input.onLost).not.toHaveBeenCalled();
await guard.stop();
} finally {
release?.(); await occupied; await guard?.stop().catch(() => undefined);
await closeLivenessDb(owner, 1); await owner.$client.end({ timeout: 1 });
}
}, 120_000);
it("renews remote ownership while the application pool is saturated and bounds row-lock waits", async () => {
const fixture = await seedRun("RENEWAL-LIVENESS-POOL");
const owner = createDb(temporary.connectionString, { maxConnections: 1 });
@@ -1174,146 +915,6 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
}
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
it.each(["preparation", "claim lock"])("gives a fresh remote claim its full lease through blocked %s", async (blockedPhase) => {
const fixture = await seedRun(`FRESH-SLOW-${blockedPhase === "claim lock" ? "CLAIM-LOCK" : "PREPARATION"}`);
const environmentId = randomUUID(), leaseId = randomUUID();
await fixture.db.insert(environments).values({ id: environmentId, name: `Fresh preparation fixture ${environmentId}`, driver: "sandbox" });
await fixture.db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId,
heartbeatRunId: fixture.runId, issueId: fixture.issueId, status: "active",
provider: "fixture", providerLeaseId: `fresh-preparation-${leaseId}` });
const execution = buildNativeExecutionInput({ companyId, agentId, runId: fixture.runId,
issue: { id: fixture.issueId, identifier: "NRR-FRESH-SLOW", title: "Prepare slowly", description: null, workMode: "standard" },
taskPrompt: "Prepare a fresh execution.", provider: "codex",
workspace: { id: fixture.runId, cwd: runtimeRoot, repoUrl: null, repoRef: null, branchName: null },
normalizedSessionId: fixture.native.normalizedSessionId, runtimeContext: nativeRuntimeContextFixture(),
completionContract: { id: randomUUID(), sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1",
contract: { revision: "1", objective: "Prepare once", criteria: [{ id: "objective", requirement: "Retain ownership" }] } },
});
// Stop at the provider boundary after checking ownership; no harness is launched.
const failure = new Error("end fresh preparation experiment");
const backend = { descriptor: async () => { throw failure; }, openSession: async () => { throw failure; } };
let reached!: () => void, release!: () => void;
const claimed = new Promise<void>(resolveReady => { reached = resolveReady; });
const preparation = new Promise<void>(resolveReady => { release = resolveReady; });
let releaseLock: (() => void) | undefined;
let holder: Promise<void> | undefined;
let lockPid: number | undefined;
let leaseStartsAfter: Date | undefined;
if (blockedPhase === "claim lock") {
let locked!: () => void;
const lockTaken = new Promise<void>(resolve => { locked = resolve; });
const lockReleased = new Promise<void>(resolve => { releaseLock = resolve; });
holder = fixture.db.transaction(async tx => {
const [connection] = await tx.execute(sql`select pg_backend_pid() as pid`);
lockPid = Number(connection.pid);
await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)).for("update");
locked(); await lockReleased;
});
await lockTaken;
}
const running = executePaperclipNativeSession({ db: fixture.db, execution,
runnerInstanceId: fixture.native.runnerInstanceId, backend,
runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: runtimeRoot },
onEvent: async event => {
if (event.payload?.span === "native.coordinator.claim") { reached(); await preparation; }
},
}).catch(error => error);
try {
if (blockedPhase === "claim lock") {
// Observe the real claim waiting on the run row after it has locked its
// coordinator. The lease must start after this wait, not before it.
await expect.poll(async () => {
const [waiting] = await fixture.db.execute(sql`select exists (
select 1 from pg_stat_activity where ${lockPid} = any(pg_blocking_pids(pid))
and query like 'select %from "heartbeat_runs"%for update%'
) as blocked`);
return waiting.blocked;
}, { timeout: 10_000 }).toBe(true);
await fixture.db.execute(sql`select pg_sleep(1)`);
const [observed] = await fixture.db.execute(sql`select clock_timestamp() as observed_at`);
leaseStartsAfter = new Date(String(observed.observed_at));
releaseLock!(); await holder;
}
await Promise.race([claimed, running.then(error => { throw error; })]);
const [initial] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
if (leaseStartsAfter) {
expect(initial.leaseExpiresAt!.getTime() - leaseStartsAfter.getTime()).toBeGreaterThan(59_900);
release();
expect(await running).toBe(failure);
return;
}
await new Promise(resolveWait => setTimeout(resolveWait, Math.max(0, initial.leaseExpiresAt!.getTime() - Date.now() + 2_000)));
const [current] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(current.leaseExpiresAt!.getTime()).toBeGreaterThan(Date.now());
expect(current.leaseOwner).toBe(initial.leaseOwner);
expect(current.controllerGeneration).toBe(initial.controllerGeneration);
expect(current.attempt).toBe(1);
release();
expect(await running).toBe(failure);
} finally { releaseLock?.(); await holder; release(); await running; }
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
it("renews a remote claim through slow preparation and shares renewal with execution", async () => {
const fixture = await seedRun("SLOW-RECOVERY-PREPARATION");
const root = resolve(runtimeRoot, fixture.runId);
const binding = { companyId, issueId: fixture.issueId, runId: fixture.runId };
await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: {
...fixture.preparedProfile, nativeWorkspaceSync: remotePreparedWorkspace(),
sessionCheckpoint: { identity: { ...binding, agentId, sessionId: fixture.native.normalizedSessionId },
providerSessionId: "surviving-provider" },
} }).where(eq(heartbeatRuns.id, fixture.runId));
await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: successor.bootId,
controllerGeneration: 1, leaseOwner: "previous-owner", leaseExpiresAt: new Date(Date.now() + 60_000) })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
const authority = new DurablePrpControlPlane({ stateDirectory: resolve(root, "control-plane"),
identity: transportOptions(fixture, root).prpIdentity,
expectedRunnerVersion: "lease-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` });
await registerNativeRunnerCheckpoint({ db: fixture.db, binding, bootId: successor.bootId, authority });
await authority.stop();
// Establish the abandoned predecessor before the fault; claimed leases are
// never edited below. The successor must retain ownership for a real minute.
await fixture.db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(0) })
.where(eq(nativeRunFinalizations.runId, fixture.runId));
const controller = { ...successor, bootId: randomUUID() };
const onLost = vi.fn(async () => undefined);
let preparation: NativeExecutionLeaseRenewal | undefined;
let execution: NativeExecutionLeaseRenewal | undefined;
let acquired = false;
try {
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller,
restartKind: "hard", runIds: [fixture.runId], onClaim: (claim, scope) => {
expect(scope.remote).toBe(true);
preparation = startNativeSessionExecutionLeaseRenewal({ db: fixture.db, ...scope,
runId: claim.runId, leaseOwner: claim.leaseOwner, attempt: claim.providerAttempt, onLost });
acquired = true;
} });
expect(acquired).toBe(true);
expect(claim.kind).toBe("reattach_remote_runner");
if (!("leaseOwner" in claim)) throw new Error("Expected recovery claim");
const [initial] = await fixture.db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId));
await new Promise(resolveWait => setTimeout(resolveWait, 30_000));
execution = startNativeSessionExecutionLeaseRenewal({ db: fixture.db, ...binding,
leaseOwner: claim.leaseOwner, attempt: claim.providerAttempt, controller, remote: true, onLost });
await preparation!.stop();
await preparation!.stop(); // A caller cannot release another caller's reference.
const remaining = Math.max(0, initial.leaseExpiresAt!.getTime() - Date.now() + 2_000);
await new Promise(resolveWait => setTimeout(resolveWait, remaining));
const [current] = await fixture.db.select().from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, fixture.runId));
expect(current.leaseExpiresAt!.getTime()).toBeGreaterThan(Date.now());
expect(current.controllerGeneration).toBe(claim.controllerGeneration);
expect(await claimNativeRestartRecoveries({ db: fixture.db, controller: { ...controller, bootId: randomUUID() },
restartKind: "hard", runIds: [fixture.runId] })).toEqual([
expect.objectContaining({ kind: "awaiting_evidence", runId: fixture.runId }),
]);
expect(onLost).not.toHaveBeenCalled();
} finally {
await preparation?.stop();
await execution?.stop();
}
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
it("fences plugin warm attachment checkpoints with the current run, including completed-session recovery", async () => {
const fixture = await seedRun("WARM-CHECKPOINT-CALLBACK");
const binding = { companyId, issueId: fixture.issueId, runId: fixture.runId };
@@ -1649,266 +1250,6 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
} finally { await authority.stop(); }
});
realProcessIt("keeps checkpoint SQL independent of streamed output volume with a real Postgres writer", async () => {
const samples: Array<{ chunks: number; checkpointWrites: number; transactions: number;
changedRows: number; checkpointBytes: number; walBytes: number; elapsedMs: number; idleSql: number;
streamSql: number; streamTransactions: number; streamChangedRows: number; renewalSql: number }> = [];
for (const chunks of [8, 512, 8192]) {
const fixture = await seedRun(`CHECKPOINT-COST-${chunks}`);
const root = resolve(runtimeRoot, fixture.runId);
const releaseOutput = resolve(root, "release-output");
const binding = { companyId, issueId: fixture.issueId, runId: fixture.runId };
const owner = { ...binding, authorityRunId: fixture.runId, bootId: successor.bootId,
generation: 1, leaseOwner: "cost-controller" };
await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: owner.bootId,
controllerPid: successor.pid, controllerProcessStartedAt: successor.processStartedAt,
controllerGeneration: 1, leaseOwner: owner.leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000),
}).where(eq(nativeRunFinalizations.runId, fixture.runId));
let authority!: DurablePrpControlPlane;
let measuring = false;
let transactions = 0, changedRows = 0, checkpointBytes = 0;
const statements: string[] = [];
const renewalStatements: string[] = [];
let leaseLost = false;
const renewal = startNativeSessionExecutionLeaseRenewal({ db: fixture.db, ...binding, leaseOwner: owner.leaseOwner,
attempt: 0, controller: successor, remote: true, onLost: async () => { leaseLost = true; } });
const capturedClients = [fixture.db.$client, checkpointDb(fixture.db).$client, livenessDb(fixture.db).$client];
const priorDebug = capturedClients.map(client => client.options.debug);
// Capture statements only, never SQL parameters (encrypted state or secrets).
capturedClients.forEach((client, index) => {
client.options.debug = (_connection, query) => {
if (!measuring) return;
// Renewals are existing liveness traffic, reported separately. A
// checkpoint query on any pool still counts against the zero oracle.
if (index === 2 && !query.includes("native_runner_checkpoints")) renewalStatements.push(query);
else statements.push(query);
};
});
const write = nativeRunnerCheckpointWriter(fixture.db, owner);
let costFailure: unknown;
const transport = createRunnerdCodexTransport({ ...transportOptions(fixture, root),
codexArgs: ["--state-file", resolve(root, "provider.json"), "--linger-after-turn-start",
"--held-output", "--held-output-count", String(chunks), "--held-output-release-file", releaseOutput],
controlPlaneRegistration: async core => {
authority = core;
core.setCheckpointPersistence(async checkpoint => {
if (measuring) {
transactions++;
changedRows += checkpoint.changedRecords.size;
checkpointBytes += [...checkpoint.changedRecords.values()].reduce((sum, value) => sum + Buffer.byteLength(value), 0);
expect([...checkpoint.changedRecords.values()].join("\n")).not.toContain("still computing");
}
await write(checkpoint);
});
return registerRunnerPrpAuthority({ ...binding, authority: core });
},
});
try {
await transport.transport.request("thread/start", { cwd: root, dynamicTools: [] });
await authority.flushCheckpoint();
const [walStart] = await fixture.db.execute(sql`select pg_current_wal_insert_lsn()::text as lsn`);
const started = performance.now();
measuring = true;
await transport.transport.request("turn/start", { input: [{ type: "text", text: "Produce the scripted stream." }] });
await waitForCondition("settled control setup before releasing text", () =>
authority.store.state.commands.every(command => command.status === "completed")
&& authority.store.state.committedEvents.some(event => event.eventType === "turn.started"));
await authority.flushCheckpoint();
const beforeStream = { sql: statements.length, transactions, changedRows };
await writeFile(releaseOutput, "release\n", { flag: "wx" });
await waitForCondition(`the final scripted text chunk of ${chunks}`, () => authority.store.state.committedEvents
.some(event => event.eventType === "item.delta" && JSON.stringify(event.envelope).includes(`still computing ${chunks - 1} `)), chunks > 4096 ? 480_000 : 60_000)
.catch(error => { console.info("checkpoint-cost-stream-diagnostic", JSON.stringify({ chunks,
cursor: authority.store.state.ackedSourceSeq,
receivedTextEvents: authority.store.state.committedEvents.filter(event => event.eventType === "item.delta").length })); throw error; });
await authority.flushCheckpoint();
measuring = false;
const elapsedMs = performance.now() - started;
const [walEnd] = await fixture.db.execute(sql`select pg_wal_lsn_diff(pg_current_wal_insert_lsn(), ${String(walStart.lsn)}::pg_lsn)::text as bytes`);
const checkpointWrites = statements.filter(query => /^\s*(insert|update|delete)\b/i.test(query)
&& query.includes("native_runner_checkpoints")).length;
expect(checkpointWrites).toBeGreaterThan(0); // The collector must see real control writes.
const snapshot = { chunks, checkpointWrites, transactions, changedRows, checkpointBytes,
walBytes: Number(walEnd.bytes), elapsedMs, idleSql: 0, renewalSql: renewalStatements.length,
streamSql: statements.length - beforeStream.sql,
streamTransactions: transactions - beforeStream.transactions,
streamChangedRows: changedRows - beforeStream.changedRows };
// Setup control messages may coalesce differently. The released text-only
// interval must add exactly zero work, independent of that scheduling.
expect(snapshot).toMatchObject({ streamSql: 0, streamTransactions: 0, streamChangedRows: 0 });
expect(leaseLost).toBe(false);
statements.length = 0;
measuring = true;
await new Promise(done => setTimeout(done, 100));
await authority.flushCheckpoint();
snapshot.idleSql = statements.length;
expect(statements).toEqual([]);
// Positive calibration: an actual new control operation must trip the zero-write oracle.
authority.queueCommand("session.snapshot", {}, `cost-positive-control-${chunks}`);
await authority.flushCheckpoint();
expect(statements.some(query => /insert into "native_runner_checkpoints"/i.test(query))).toBe(true);
measuring = false;
samples.push(snapshot);
console.info("native-checkpoint-cost-sample", JSON.stringify(snapshot));
} catch (error) {
costFailure = error;
throw error;
} finally {
measuring = false;
capturedClients.forEach((client, index) => { client.options.debug = priorDebug[index]; });
try { await transport.transport.close(); }
catch (error) {
if (costFailure) throw new AggregateError([costFailure, error], "Checkpoint cost measurement and cleanup failed");
throw error;
}
finally {
await renewal.stop().catch(error => {
if (costFailure) throw new AggregateError([costFailure, error], "Checkpoint cost measurement and lease renewal failed");
throw error;
});
}
}
}
const report = { schema: "paperclip.checkpoint-cost/v1", samples,
scope: "isolated Postgres; control operations included; WAL is diagnostic; no production capacity claim" };
if (process.env.PAPERCLIP_TEST_CHECKPOINT_COST_REPORT) await writeFile(process.env.PAPERCLIP_TEST_CHECKPOINT_COST_REPORT,
`${JSON.stringify(report, null, 2)}\n`, { flag: "wx", mode: 0o600 });
// Each stream size was checked independently; total setup counts remain
// diagnostic because control-message batching is scheduling-dependent.
console.info("native-checkpoint-cost", JSON.stringify(report));
}, 540_000);
realProcessIt.each([false, true])("adopts the same runner from Postgres after losing the controller filesystem (prepared task: %s)", async preparedTask => {
const fixture = await seedRun(`EMPTY-CONTROLLER-DISK-${preparedTask}`);
const controllerRoot = resolve(runtimeRoot, `${fixture.runId}-controller`);
const sandboxRoot = resolve(runtimeRoot, `${fixture.runId}-sandbox`);
const binding = { companyId, issueId: fixture.issueId, runId: fixture.runId };
const outageMs = Number(process.env.PAPERCLIP_TEST_CONTROLLER_OUTAGE_MS ?? 1000);
if (!Number.isFinite(outageMs) || outageMs < 0 || outageMs > 300_000) throw new Error("Invalid controller outage duration");
await mkdir(sandboxRoot, { recursive: true });
await mkdir(resolve(sandboxRoot, "filesystem/codex-home"), { recursive: true });
let bootId = randomUUID();
const taskLeaseId = randomUUID();
if (preparedTask) {
const [environment] = await fixture.db.insert(environments).values({ name: "Surviving plugin task", driver: "plugin", status: "active", config: {} }).returning();
await fixture.db.insert(environmentLeases).values({ id: taskLeaseId, companyId, environmentId: environment.id,
heartbeatRunId: fixture.runId, issueId: fixture.issueId, status: "active", leasePolicy: "ephemeral",
provider: "plugin", providerLeaseId: "saved-task", metadata: { agentId, runnerTaskLease: true, runnerTaskDispatched: true } });
}
const originalLeaseExpiresAt = new Date(Date.now() + 60_000);
await fixture.db.update(nativeRunFinalizations).set({ controllerBootId: bootId,
controllerGeneration: 1, controllerPid: process.pid, controllerProcessStartedAt: new Date(),
leaseOwner: "original-controller", leaseExpiresAt: originalLeaseExpiresAt,
}).where(eq(nativeRunFinalizations.runId, fixture.runId));
const options = {
...transportOptions(fixture, controllerRoot),
...(preparedTask ? { prpIdentity: { ...transportOptions(fixture, controllerRoot).prpIdentity, environmentLeaseId: taskLeaseId } } : {}),
runnerStateDirectory: resolve(sandboxRoot, "runner"),
runnerFilesystemRoot: resolve(sandboxRoot, "filesystem"),
runnerReconnectGraceMs: REMOTE_RUNNER_RECONNECT_GRACE_MS,
codexArgs: ["--state-file", resolve(sandboxRoot, "provider.json"),
"--call-log", resolve(sandboxRoot, "calls.log"), "--linger-after-turn-start", "--held-output"],
controlPlaneRegistration: async (authority: DurablePrpControlPlane) => {
await registerNativeRunnerCheckpoint({ db: fixture.db, binding, bootId, authority });
return registerRunnerPrpAuthority({ ...binding, authority });
},
};
const prepareLaunch = vi.fn(async (requirements: RunnerProviderLaunchRequirements) => ({
configurationId: "provider-owned-configuration", requirementsDigest: runnerLaunchRequirementsDigest(requirements),
provider: createRunnerProviderLaunch(requirements, { options: { codexArgs: options.codexArgs },
root: sandboxRoot, cwd: sandboxRoot, codexHome: resolve(sandboxRoot, "filesystem/codex-home"),
codexCommand: fakeCodexAppServer, providerNodeCommand: process.execPath, opencodeProxyPath: "", acpxSidecarPath: "" }),
}));
const first = createRunnerdCodexTransport({ ...options, ...(preparedTask ? {
prepareLaunch, runnerArtifact: readRunnerdArtifactBinding(defaultCapabilityRunnerdBinary()),
runnerProcessLauncher: (spec: RunnerProcessLaunchSpec): RunnerProcessHandle => {
const child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: { ...process.env, ...spec.environment },
detached: process.platform !== "win32", stdio: "ignore" });
return { child, processGroupId: child.pid ?? null, completion: new Promise((resolveExit, reject) => {
child.once("error", reject);
child.once("exit", (code, signal) => resolveExit({ code, signal, stdout: "", stderr: "" }));
}) };
},
} : {}) });
let adopted: ReturnType<typeof createRunnerdCodexTransport> | undefined;
let runnerPid: number | null = null;
try {
const thread = (await first.transport.request("thread/start", { cwd: sandboxRoot, dynamicTools: [] })).thread as Record<string, unknown>;
const turn = (await first.transport.request("turn/start", { input: [{ type: "text", text: "Keep computing while the controller is offline." }] })).turn as Record<string, unknown>;
runnerPid = first.evidence().runnerPid;
const providerPid = first.evidence().providerPid;
if (!runnerPid || !providerPid) throw new Error("Runner/provider process identity missing");
const startedAt = await readProcessStartedAt(runnerPid);
if (!startedAt) throw new Error("Runner start identity missing");
await waitForCondition("streaming ACK advances without a recovery checkpoint", async () => {
const runner = JSON.parse(await readFile(resolve(sandboxRoot, "runner/runner-state.json"), "utf8"));
const checkpoint = await loadNativeRunnerCheckpoint(fixture.db, binding);
const cursor = checkpoint && JSON.parse(checkpoint.records.get("cursor")!);
return cursor && runner.ackedSourceSeq >= cursor.ackedSourceSeq + 64;
});
const beforeRestart = (await loadNativeRunnerCheckpoint(fixture.db, binding))!;
expect([...beforeRestart.records.entries()].filter(([key]) => key.startsWith("event:"))
.some(([, value]) => JSON.parse(value).eventType === "item.delta")).toBe(false);
await fixture.db.update(heartbeatRuns).set({ processPid: runnerPid, processStartedAt: new Date(startedAt),
runnerProfileJson: { ...fixture.preparedProfile,
...(!preparedTask ? { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v1", state: "prepared",
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64),
workspaceId: "workspace", leaseId: "lease", providerLeaseId: "sandbox", remoteCwd: sandboxRoot } } : {}),
sessionCheckpoint: { identity: { ...binding, agentId, sessionId: fixture.native.normalizedSessionId },
providerSessionId: thread.id, process: { providerPid } },
},
}).where(eq(heartbeatRuns.id, fixture.runId));
await first.detachControllerForRestart();
await rm(controllerRoot, { recursive: true, force: true });
expect(existsSync(controllerRoot)).toBe(false);
await new Promise(resolveOutage => setTimeout(resolveOutage, outageMs));
expect(processAlive(runnerPid)).toBe(true);
expect(processAlive(providerPid)).toBe(true);
bootId = randomUUID();
if (originalLeaseExpiresAt.getTime() > Date.now()) {
const [waiting] = await claimNativeRestartRecoveries({ db: fixture.db, controller: { ...successor, bootId },
restartKind: "hard", runIds: [fixture.runId] });
expect(waiting).toMatchObject({ kind: "awaiting_evidence", reason: "remote_controller_lease_live", retryAt: originalLeaseExpiresAt });
await new Promise(resolveExpiry => setTimeout(resolveExpiry, Math.max(0, originalLeaseExpiresAt.getTime() - Date.now()) + 20));
}
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: { ...successor, bootId },
restartKind: "hard", runIds: [fixture.runId] });
expect(claim).toMatchObject({ kind: "reattach_remote_runner", providerAttempt: 0, controllerGeneration: 2 });
if (preparedTask) expect(claim).toMatchObject({ remote: { kind: "plugin_task", leaseId: taskLeaseId } });
const checkpoint = await restoreNativeRunnerCheckpoint(fixture.db, binding, controllerRoot);
if (!checkpoint) throw new Error("Missing durable authority");
const duplicateLauncher = vi.fn(() => { throw new Error("duplicate runner spawn attempted"); });
prepareLaunch.mockClear();
const authorizeRunnerAdoption = vi.fn(async () => {});
adopted = createRunnerdCodexTransport({ ...options,
...(preparedTask ? { prepareLaunch, authorizeRunnerAdoption } : {}),
runnerBinary: resolve(controllerRoot, "new-release-does-not-contain-old-runner"),
runnerArtifact: { version: checkpoint.runnerVersion, digest: checkpoint.runnerDigest },
resumeDynamicTools: [], runnerProcessLauncher: duplicateLauncher,
adoptExistingRunner: { pid: preparedTask ? null : runnerPid, processGroupId: preparedTask ? null : first.evidence().runnerProcessGroupId,
startedAt: preparedTask ? null : startedAt, isAlive: () => processAlive(runnerPid) },
});
await expect(adopted.transport.request("thread/read", {})).resolves.toMatchObject({
thread: { id: thread.id, turns: [{ id: turn.id, status: "inProgress" }] },
});
await adopted.transport.request("turn/steer", { input: [{ type: "text", text: "Continue the same turn." }],
expectedTurnId: turn.id, correlationId: "after-empty-disk-restart" });
expect(adopted.evidence().runnerPid).toBe(preparedTask ? null : runnerPid);
expect(adopted.evidence().providerPid).toBe(providerPid);
expect(await readProcessStartedAt(runnerPid)).toBe(startedAt);
expect(duplicateLauncher).not.toHaveBeenCalled();
expect(prepareLaunch).not.toHaveBeenCalled();
if (preparedTask) expect(authorizeRunnerAdoption).toHaveBeenCalledExactlyOnceWith("provider-owned-configuration", options.prpIdentity);
const calls = await readFile(resolve(sandboxRoot, "calls.log"), "utf8");
expect(calls.match(/^turn\/start$/gm)).toHaveLength(1);
expect(calls.match(/^turn\/steer$/gm)).toHaveLength(1);
} finally {
await adopted?.transport.close().catch(() => undefined);
await first.transport.close().catch(() => undefined);
await stopOwnedProcessGroup(runnerPid, sandboxRoot).catch(() => undefined);
}
}, 345_000);
realProcessIt("adopts one active runner across hot and hard controller restarts without duplicating steering", async () => {
const fixture = await seedRun("LIVE");
const stateDirectory = resolve(runtimeRoot, fixture.runId);
@@ -2635,13 +1976,6 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
}
});
function remotePreparedWorkspace() {
return { schema: "paperclip.native-workspace-sync/v3", state: "prepared",
descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null,
workspaceId: randomUUID(), leaseId: randomUUID(), providerLeaseId: randomUUID(),
remoteCwd: "/workspace", resourceDisposition: "destroy" };
}
it.each(["claimed", "generation", "process", "launch", "provider-event", "checkpoint", "invalid-workspace", "unfenced"])(
"keeps remote startup recovery blocked with contradictory %s evidence", async (kind) => {
const fixture = await seedRun(`REMOTE-${kind}`);
@@ -2829,5 +2163,4 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
}, 30_000);
});
enableRunFailureReporting();