mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 06:16:06 -10:00
test(server): split workspace-recovery tests into seven feature files (#19664)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - PR CI runs the general server tests in 12 shards, and each shard runs files in parallel on 4 workers > - `server/src/services/workspace-recovery.test.ts` takes about 699 seconds in CI by itself. A file runs whole on one worker, so shard 2/12 takes 11.5 minutes (median) when the other shards take about 7 minutes > - A different shard count cannot fix this, because the time of one file sets the minimum time of its shard > - This pull request splits the file into seven feature files with a shared fixture > - The benefit is that the shards can balance again. This removes about 2 minutes from the median PR CI time and about 5 minutes from the p90 ## Linked Issues or Issue Description **What existing behavior does this improve?** The time of the general server test shards in PR CI. **Subsystem affected** Server tests: workspace recovery (`server/src/services/workspace-recovery*.test.ts`) and the general-server shard weights. **Current behavior** One file holds 460 tests and takes about 699 seconds in CI. Its shard is the slowest in most PR runs. **Proposed behavior** Seven files of about 60 to 140 seconds each. The shard partitioner places them on different shards. **Reason and benefit** The general server shards finish close together. The PR CI critical path becomes shorter. **Breaking changes** None. No product code changes. ## What Changed - Split `server/src/services/workspace-recovery.test.ts` into seven files by feature area: - `retry-copyback` (76 tests); - `preparation` (80); - `preparation-proof` (53); - `preparation-dispatch` (25); - `preparation-gates` (43); - `warm-handoff` (36); - `admission` (147). - Add `server/src/services/workspace-recovery.test-fixture.ts` with the shared hooks, `seed`, helpers and an embedded Postgres per file. - Test bodies, hooks and helpers are moved without change. - Four lines that assigned the shared `db`, `root` or `connectionString` now call `replaceFixtureState(...)`. - `scripts/general-server-shard-durations.json`: replace the old key with the seven new files. The weights divide the current CI cost of the old file (636 seconds, master runs 42296943853 and 42294279748) in proportion to measured local time. A `splitEstimates` entry records the method, as for earlier splits. - Update the file references in `doc/task-recovery.md` and `maps/feature-map/recovery.md`. ## Verification - `vitest list`, before and after: 136 declarations each, with identical sorted names. At run time, 460 tests before and after. - Each new file passes when run alone: 76, 80, 53, 25, 43, 36 and 147 tests. - All seven files run together with `--maxWorkers=1`: 460/460 pass. - A timing check of the original file: - `afterEach` took 4.7 seconds in total and `beforeAll` 4.1 seconds, out of about 1,000 seconds, so the teardown is not the cost; - most of the time is in three `it.each` blocks. - Focused `tsc --noEmit` on the eight new files: clean. - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`: 39/39 pass. - `node scripts/check-maps.mjs`: aligned. - With the new weights, the 12-shard partition puts the seven files on seven different shards. ## Risks - The CI time of each new file is an estimate from local runs. If `workspace-recovery-preparation.test.ts` takes more than about 150 seconds in CI, the next split point is the "keeps ancestor Retry held" block. - Each file starts its own embedded Postgres, which adds a few seconds of setup per file. - The optional external-database environment variables keep their old behavior but were not tested. CI does not set them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent made the split, and the parent session reviewed it and corrected the shard weights. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Nicky LeachandClaude Opus 5.5 authored and GitHub committed
1 parent
d6f4155bad
commit
8ed712f00a
14 files changed
+6723
-6399
No files matched your search
@@ -253,7 +253,13 @@ Run the focused PostgreSQL/filesystem suites from `server/`:
|
||||
../node_modules/.bin/vitest run --config vitest.config.ts \
|
||||
src/services/system-recovery-state.test.ts \
|
||||
src/services/task-retry.test.ts \
|
||||
src/services/workspace-recovery.test.ts \
|
||||
src/services/workspace-recovery-retry-copyback.test.ts \
|
||||
src/services/workspace-recovery-preparation.test.ts \
|
||||
src/services/workspace-recovery-preparation-proof.test.ts \
|
||||
src/services/workspace-recovery-preparation-dispatch.test.ts \
|
||||
src/services/workspace-recovery-preparation-gates.test.ts \
|
||||
src/services/workspace-recovery-warm-handoff.test.ts \
|
||||
src/services/workspace-recovery-admission.test.ts \
|
||||
src/services/explicit-native-continuation.test.ts \
|
||||
src/services/heartbeat/workspace-preparation.test.ts \
|
||||
src/services/native-runtime/native-chat-workspace.test.ts
|
||||
|
||||
@@ -311,7 +311,7 @@ gap.
|
||||
|
||||
### `operator-retry`
|
||||
|
||||
The [workspace recovery regression](../../server/src/services/workspace-recovery.test.ts)
|
||||
The [workspace recovery regression](../../server/src/services/workspace-recovery-warm-handoff.test.ts)
|
||||
uses the real Retry producer and a quiet delivery coordinator. It checks that a
|
||||
committed request starts one physical source preservation, duplicate Retry keeps
|
||||
the active owner's receipt valid, and startup reconstructs a resolved files-only
|
||||
|
||||
@@ -1345,8 +1345,14 @@
|
||||
"server/src/services/vercel-connect.test.ts": 163,
|
||||
"server/src/services/workspace-git-operation-scheduler.test.ts": 1674,
|
||||
"server/src/services/workspace-git-snapshot-streaming.test.ts": 30880,
|
||||
"server/src/services/workspace-recovery-admission.test.ts": 66000,
|
||||
"server/src/services/workspace-recovery-preparation-dispatch.test.ts": 101000,
|
||||
"server/src/services/workspace-recovery-preparation-gates.test.ts": 60000,
|
||||
"server/src/services/workspace-recovery-preparation-proof.test.ts": 91000,
|
||||
"server/src/services/workspace-recovery-preparation.test.ts": 136000,
|
||||
"server/src/services/workspace-recovery-retry-copyback.test.ts": 66000,
|
||||
"server/src/services/workspace-recovery-source.test.ts": 7319,
|
||||
"server/src/services/workspace-recovery.test.ts": 309441,
|
||||
"server/src/services/workspace-recovery-warm-handoff.test.ts": 116000,
|
||||
"server/src/services/workspace-restore-reconciliation.test.ts": 5517,
|
||||
"server/src/services/workspace-runtime-exposure-backfill.test.ts": 4136,
|
||||
"server/src/services/workspace-runtime-exposure.test.ts": 14546,
|
||||
@@ -1382,6 +1388,20 @@
|
||||
"server/src/services/native-runtime/native-runner-restart-checkpoint-cost.integration.test.ts": 187000,
|
||||
"server/src/services/native-runtime/native-runner-restart-diskless.integration.test.ts": 172000
|
||||
}
|
||||
},
|
||||
"server/src/services/workspace-recovery.test.ts": {
|
||||
"sourceRun": 42296943853,
|
||||
"sourceDurationMs": 636000,
|
||||
"apportionedBy": "Local per-file test time for the same tests, averaged over a whole-file run and per-file runs; the predecessor's current CI cost (636s, master runs 42296943853 and 42294279748) is divided in those proportions.",
|
||||
"files": {
|
||||
"server/src/services/workspace-recovery-admission.test.ts": 66000,
|
||||
"server/src/services/workspace-recovery-preparation-dispatch.test.ts": 101000,
|
||||
"server/src/services/workspace-recovery-preparation-gates.test.ts": 60000,
|
||||
"server/src/services/workspace-recovery-preparation-proof.test.ts": 91000,
|
||||
"server/src/services/workspace-recovery-preparation.test.ts": 136000,
|
||||
"server/src/services/workspace-recovery-retry-copyback.test.ts": 66000,
|
||||
"server/src/services/workspace-recovery-warm-handoff.test.ts": 116000
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { sql } from "drizzle-orm";
|
||||
import { createDb } from "@paperclipai/db";
|
||||
|
||||
// Every workspace-recovery test file that uses
|
||||
// PAPERCLIP_WORKSPACE_RECOVERY_TEST_DATABASE_URL truncates that database.
|
||||
// A file holds this session advisory lock from setup to teardown, so the files
|
||||
// queue on the database, even across separate test processes. Ending the
|
||||
// returned client releases the lock.
|
||||
//
|
||||
// Poll with pg_try_advisory_lock rather than block in pg_advisory_lock: the
|
||||
// holder's migrations include CREATE INDEX CONCURRENTLY, which waits for every
|
||||
// open query in the database, and a blocked lock call would never finish.
|
||||
export async function lockWorkspaceRecoveryTestDatabase(url: string) {
|
||||
const lock = createDb(url, { maxConnections: 1, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
|
||||
for (;;) {
|
||||
const [row] = await lock.execute<{ locked: boolean }>(
|
||||
sql`SELECT pg_try_advisory_lock(hashtext('paperclip:workspace-recovery-tests')) AS locked`);
|
||||
if (row?.locked) return lock;
|
||||
await delay(500);
|
||||
}
|
||||
}
|
||||
|
||||
// Waiting for the lock can take as long as the other files run.
|
||||
export const WORKSPACE_RECOVERY_EXTERNAL_SETUP_TIMEOUT_MS = 30 * 60_000;
|
||||
@@ -0,0 +1,446 @@
|
||||
// Shared setup for the automatic retained workspace recovery suites
|
||||
// (workspace-recovery-retry-copyback, -preparation*, -warm-handoff and -admission),
|
||||
// which were split from one file by feature area so CI can run them in parallel.
|
||||
// Each suite calls useWorkspaceRecoveryFixture() inside its top-level describe, so
|
||||
// it gets its own database, temporary root and the same per-test cleanup. `db`,
|
||||
// `root` and `connectionString` are live bindings: the helpers and the importing
|
||||
// suite see the values that beforeAll (or replaceFixtureState) assigns.
|
||||
//
|
||||
// vi.mock() only applies to the file that declares it, so each suite keeps its
|
||||
// own environment-execution-target mock and passes its `targets` map in.
|
||||
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
|
||||
import { recordRequiredWorkspaceRestoreFailure, terminalizeLegacyExecution } from "../legacy-execution-recovery.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
||||
import { environmentService } from "../environments.js";
|
||||
import { classifyEnvironmentCapabilities, environmentRuntimeService } from "../environment-runtime.js";
|
||||
import { readSystemRecovery, createSystemRecovery, failSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "../system-recovery-state.js";
|
||||
import { adoptSystemRecovery } from "../system-recovery.js";
|
||||
import { requestTaskRetry } from "../task-retry.js";
|
||||
import { execFile } from "node:child_process";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { chmod, cp, lstat, mkdir, mkdtemp, readdir, realpath, rename, rm, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import express from "express";
|
||||
import { and, eq, inArray, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, expect, vi } from "vitest";
|
||||
import { agents, completionContracts, companies, createDb, applyPendingMigrations, environmentLeases, environments, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, agentWakeupRequests, executionWorkspaces } from "@paperclipai/db";
|
||||
import { CONTROL_PLANE_CONFORMANCE_RESULT, CONTROL_PLANE_CONFORMANCE_TERMINAL } from "../../vendor/paperclip-runner/testing.js";
|
||||
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
||||
import type { EnvironmentLease } from "@paperclipai/shared";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { lockWorkspaceRecoveryTestDatabase, WORKSPACE_RECOVERY_EXTERNAL_SETUP_TIMEOUT_MS } from "./workspace-recovery-database-lock.js";
|
||||
import { prepareNativeWorkspaceSync, resumeNativeWorkspaceSync } from "../native-runtime/native-workspace-sync.js";
|
||||
import { remoteTerminationReceipt } from "../remote-execution-termination.js";
|
||||
import { prepareSandboxStopAndRetain, settleStopOnlyCleanup, readStopOnlyCleanup } from "../sandbox-stop-and-retain.js";
|
||||
import * as heartbeatServices from "../heartbeat.js";
|
||||
import { heartbeatService } from "../heartbeat.js";
|
||||
import { prepareRemoteWorkspace, reserveRemoteWorkspace } from "../remote-workspaces.js";
|
||||
import { issueRoutes } from "../../routes/issues.js";
|
||||
import { errorHandler } from "../../middleware/index.js";
|
||||
import * as workerProof from "../native-runtime/native-workspace-platform-proof.js";
|
||||
|
||||
export let db: ReturnType<typeof createDb>;
|
||||
export let root: string;
|
||||
export let connectionString: string;
|
||||
export const fixtureCompanyIds = new Set<string>();
|
||||
let targets: Map<string, AdapterSandboxExecutionTarget>;
|
||||
|
||||
/** Swaps the connection the shared helpers use; a suite restores the previous one itself. */
|
||||
export function replaceFixtureState(next: { db: typeof db; root?: string; connectionString?: string }) {
|
||||
db = next.db;
|
||||
if (next.root !== undefined) root = next.root;
|
||||
if (next.connectionString !== undefined) connectionString = next.connectionString;
|
||||
}
|
||||
|
||||
export function useWorkspaceRecoveryFixture(executionTargets: Map<string, AdapterSandboxExecutionTarget>) {
|
||||
targets = executionTargets;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let externalLock: ReturnType<typeof createDb> | undefined;
|
||||
const priorHome = process.env.PAPERCLIP_HOME, priorInstance = process.env.PAPERCLIP_INSTANCE_ID;
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_RECOVERY_TEST_DATABASE_URL");
|
||||
beforeAll(async () => {
|
||||
root = await mkdtemp(path.join(os.tmpdir(), "workspace-recovery-"));
|
||||
process.env.PAPERCLIP_HOME = path.join(root, "home"); process.env.PAPERCLIP_INSTANCE_ID = "recovery";
|
||||
if (external) {
|
||||
externalLock = await lockWorkspaceRecoveryTestDatabase(external);
|
||||
await applyPendingMigrations(external);
|
||||
temporary = { connectionString: external, cleanup: async () => {} };
|
||||
} else temporary = await startEmbeddedPostgresTestDatabase("workspace-recovery-");
|
||||
connectionString = temporary.connectionString;
|
||||
db = createDb(connectionString);
|
||||
if (external) await db.execute(sql`TRUNCATE TABLE companies CASCADE`);
|
||||
}, external ? WORKSPACE_RECOVERY_EXTERNAL_SETUP_TIMEOUT_MS : 30_000);
|
||||
afterAll(async () => {
|
||||
await db?.$client.end(); await temporary?.cleanup();
|
||||
await externalLock?.$client.end(); externalLock = undefined;
|
||||
const makeWritable = async (directory: string): Promise<void> => {
|
||||
const entry = await lstat(directory);
|
||||
if (!entry.isDirectory() || entry.isSymbolicLink()) return;
|
||||
await chmod(directory, 0o700);
|
||||
for (const child of await readdir(directory, { withFileTypes: true })) {
|
||||
if (child.isDirectory()) await makeWritable(path.join(directory, child.name));
|
||||
}
|
||||
};
|
||||
await makeWritable(root); await rm(root, { recursive: true, force: true });
|
||||
if (priorHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = priorHome;
|
||||
if (priorInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; else process.env.PAPERCLIP_INSTANCE_ID = priorInstance;
|
||||
});
|
||||
afterEach(async () => {
|
||||
// Discovery is intentionally global and bounded. An unresolved source from
|
||||
// an earlier test must not consume the current fixture's recovery page.
|
||||
// Keep all history within each test until its assertions have completed.
|
||||
const companyIds = [...fixtureCompanyIds]; fixtureCompanyIds.clear();
|
||||
if (!companyIds.length) return;
|
||||
// Stop global queue sweeps from revisiting deferred input after this
|
||||
// fixture's recovery actions and lease bindings are removed below.
|
||||
await db.update(agents).set({ status: "paused" }).where(inArray(agents.companyId, companyIds));
|
||||
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() }).where(and(
|
||||
inArray(agentWakeupRequests.companyId, companyIds),
|
||||
inArray(agentWakeupRequests.status, ["queued", "claimed", "deferred_issue_execution"])));
|
||||
await heartbeatService(db).drainActiveRunExecutions();
|
||||
// Admission tests leave synthetic queue blockers. Retire them only after
|
||||
// assertions so they cannot trip the next test's instance-wide drain fence.
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(and(
|
||||
inArray(heartbeatRuns.companyId, companyIds), inArray(heartbeatRuns.status, ["queued", "running"])));
|
||||
const leases = await db.select({ id: environmentLeases.id }).from(environmentLeases)
|
||||
.where(inArray(environmentLeases.companyId, companyIds));
|
||||
for (const lease of leases) targets.delete(lease.id);
|
||||
await db.delete(environmentLeases).where(inArray(environmentLeases.companyId, companyIds));
|
||||
// Historical adoption can reconstruct an action from either the lease or
|
||||
// the run descriptor, so retire both fixture inputs before deleting it.
|
||||
await db.update(heartbeatRuns).set({ resultJson: sql`${heartbeatRuns.resultJson} - 'workspaceRestoreRecovery' - 'workspaceRestoreFailure'` })
|
||||
.where(inArray(heartbeatRuns.companyId, companyIds));
|
||||
await db.delete(issueRecoveryActions).where(inArray(issueRecoveryActions.companyId, companyIds));
|
||||
});
|
||||
}
|
||||
|
||||
export async function seed(mode = "native", nativeDescriptor = true, resourceDisposition: "destroy" | "keep_running" = "destroy", reusable = false, ephemeral = false) {
|
||||
const companyId = randomUUID(), issueId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), actionId: string = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Workspace recovery", defaultResponsibleUserId: "board", issuePrefix: `R${companyId.slice(0, 6)}` });
|
||||
fixtureCompanyIds.add(companyId);
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "paperclip_runner", status: "idle", runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } });
|
||||
await db.insert(issues).values({ id: issueId, companyId, assigneeAgentId: agentId, status: "blocked", title: "Recover existing work" });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, issueId, nativeIssueId: mode === "native" ? issueId : null,
|
||||
agentId, runtimeMode: mode, status: "failed", finishedAt: new Date(), errorCode: "execution_interrupted", contextSnapshot: { issueId } });
|
||||
if (mode === "legacy") await db.update(heartbeatRuns).set({ runnerProfileJson: { adapterDispatch: { adapterType: "codex_local" } } }).where(eq(heartbeatRuns.id, runId));
|
||||
if (mode === "native") await db.insert(nativeRunFinalizations).values({ companyId, issueId, runId, phase: "terminal_failure" });
|
||||
const [environment] = await db.insert(environments).values({ name: runId, driver: "sandbox" }).returning();
|
||||
const providerLeaseId: string = randomUUID(), remote = path.join(root, runId, "remote"), host = path.join(root, runId, "host");
|
||||
await mkdir(host, { recursive: true }); await writeFile(path.join(host, "existing.txt"), "base\n");
|
||||
const leaseValues = { companyId, issueId, heartbeatRunId: runId,
|
||||
environmentId: environment.id, provider: "daytona", providerLeaseId,
|
||||
metadata: { remoteCwd: remote, driver: "sandbox", pluginId: randomUUID(), sandboxProviderPlugin: true, ...(reusable ? { agentId } : {}) } };
|
||||
const acquired = reusable ? await environmentService(db).acquireLease({ ...leaseValues, leasePolicy: "reuse_by_environment" }) : null;
|
||||
const [lease] = acquired ? await db.select().from(environmentLeases).where(eq(environmentLeases.id, acquired.id))
|
||||
: await db.insert(environmentLeases).values({ ...leaseValues, leasePolicy: ephemeral ? "ephemeral" : "retain_on_failure", status: "released",
|
||||
cleanupStatus: "success", releasedAt: new Date() }).returning();
|
||||
const target: AdapterSandboxExecutionTarget = { kind: "remote", transport: "sandbox", providerKey: "test", leaseId: lease.id,
|
||||
remoteCwd: remote, timeoutMs: 30_000, sandboxLeaseAcquisition: { outcome: "created", providerLeaseId },
|
||||
runner: { execute: input => new Promise((resolve, reject) => {
|
||||
const startedAt = new Date().toISOString();
|
||||
const child = execFile(input.command, input.args ?? [], { cwd: input.cwd, env: { ...process.env, ...input.env }, timeout: input.timeoutMs, maxBuffer: 16 * 1024 * 1024 },
|
||||
(error, stdout, stderr) => resolve({ exitCode: error ? typeof error.code === "number" ? error.code : 1 : 0,
|
||||
signal: error?.signal ?? null, timedOut: error?.killed ?? false, stdout, stderr, pid: child.pid ?? null, startedAt }));
|
||||
child.stdin?.on("error", (error: NodeJS.ErrnoException) => { if (error.code !== "EPIPE") reject(error); });
|
||||
child.stdin?.end(input.stdin ?? "");
|
||||
}) } };
|
||||
targets.set(lease.id, target);
|
||||
if (nativeDescriptor) {
|
||||
const prepared = await prepareNativeWorkspaceSync({ db, companyId, workspaceId: issueId, workspaceLocalDir: host,
|
||||
runId, lease: lease as EnvironmentLease, target, resourceDisposition });
|
||||
expect(prepared).not.toBeNull();
|
||||
} else {
|
||||
// Provider boundary only: real temporary files, no native baseline. The
|
||||
// production provider's syncOut contract validates its archive itself.
|
||||
target.runner!.syncIn = async () => ({ filesTransferred: 0, bytesTransferred: 0, operations: [] });
|
||||
target.runner!.syncOut = async operations => {
|
||||
for (const operation of operations) for (const file of operation.files)
|
||||
await cp(file.sourcePath, file.targetPath, { recursive: true, dereference: false });
|
||||
return { filesTransferred: 2, bytesTransferred: 0, operations: [] };
|
||||
};
|
||||
await mkdir(remote, { recursive: true });
|
||||
await writeFile(path.join(remote, "existing.txt"), "base\n");
|
||||
}
|
||||
await writeFile(path.join(remote, "saved.txt"), "finished work\n");
|
||||
await writeFile(path.join(host, "newer.txt"), "newer local work\n");
|
||||
const recovery = { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [lease.id] };
|
||||
await db.update(heartbeatRuns).set({ resultJson: { workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: recovery,
|
||||
conversationContinuation: "continue_conversation_v1" } }).where(eq(heartbeatRuns.id, runId));
|
||||
await db.update(environmentLeases).set({ metadata: { ...lease.metadata,
|
||||
remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId, state: "stopped" }) } }).where(eq(environmentLeases.id, lease.id));
|
||||
await db.insert(issueRecoveryActions).values({ id: actionId, companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
|
||||
ownerType: "board", cause: mode === "native" ? "native_continuation_requires_reconciliation" : "legacy_execution_requires_reconciliation",
|
||||
fingerprint: runId, nextAction: "Manual repair required", evidence: { runId, workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: recovery } });
|
||||
const runtime = { resumeRunLease: vi.fn(async (_input: { lease: EnvironmentLease }) => ({ providerLeaseId, metadata: { remoteCwd: remote } })),
|
||||
retryPendingSandboxTeardown: vi.fn(async (_input: { lease: EnvironmentLease }) => ({ providerLeaseId, state: "stopped" })) };
|
||||
const environmentRuntime = environmentRuntimeService(db, { drivers: [{ driver: "sandbox",
|
||||
acquireRunLease: vi.fn(), releaseRunLease: vi.fn(), ...runtime,
|
||||
resolveCapabilities: async () => classifyEnvironmentCapabilities({ verifiedMethods: [], declared: null, narrowing: null }),
|
||||
}] });
|
||||
const input = { db, companyId, issueId, runId, actionId, environmentRuntime };
|
||||
return { ...input, input, runtime, host, remote, lease, providerLeaseId };
|
||||
}
|
||||
|
||||
export const action = async (id: string) => (await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!;
|
||||
|
||||
export const run = async (id: string) => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)))[0]!;
|
||||
|
||||
export async function failedWarmSource(exportOnly = false, hops = 0) {
|
||||
const f = hops ? await warmHandoffSource(hops) : await seed("native", true, "keep_running");
|
||||
if (hops) await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment" }).where(eq(environmentLeases.id, f.lease.id));
|
||||
const retry = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "board",
|
||||
actorSource: "local_implicit", idempotencyKey: randomUUID(), observedRunId: f.runId });
|
||||
const failed = failSystemRecovery(reserveSystemRecoveryAttempt(createSystemRecovery(f.runId, retry.requestId!))!,
|
||||
new SystemRecoveryFailure("source_binding_invalid", "service_required"), { now: Date.now() - 86_400_000 });
|
||||
await db.update(issueRecoveryActions).set({ evidence: { ...(await action(f.actionId)).evidence, systemRecovery: failed,
|
||||
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: f.runId, status: "blocked", stage: "verifying_stop",
|
||||
attempt: 1, attemptLimit: 6, requestIds: [retry.requestId!] } } }).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
if (exportOnly) {
|
||||
const contractId = randomUUID(), resultId = randomUUID();
|
||||
await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issueId,
|
||||
revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard",
|
||||
completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: {},
|
||||
canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" });
|
||||
await db.update(heartbeatRuns).set({ completionContractId: contractId, resultJson: {} }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId, runId: f.runId,
|
||||
completionContractId: contractId, schemaStatus: "accepted", serverFingerprint: resultId, canonicalSha256: resultId,
|
||||
resultJson: { terminal: { runTerminalState: "succeeded" } } });
|
||||
await db.update(nativeRunFinalizations).set({ resultId }).where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
const evidence = { ...(await action(f.actionId)).evidence };
|
||||
delete evidence.workspaceRestoreFailure; delete evidence.workspaceRestoreRecovery;
|
||||
await db.update(issueRecoveryActions).set({ cause: "native_workspace_sync_out_retry_exhausted", evidence }).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
}
|
||||
return { ...f, retry, failed };
|
||||
}
|
||||
|
||||
export function boardRecoveryApp(actor: { userId: string; source: "local_implicit" | "session"; companyIds?: string[] } = { userId: "local-board", source: "local_implicit" },
|
||||
heartbeat?: ReturnType<typeof heartbeatService>) {
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => { req.actor = { type: "board", ...actor }; next(); });
|
||||
const factory = heartbeat ? vi.spyOn(heartbeatServices, "heartbeatService").mockReturnValue(heartbeat) : undefined;
|
||||
try { app.use("/api", issueRoutes(db, {} as Parameters<typeof issueRoutes>[1])); }
|
||||
finally { factory?.mockRestore(); }
|
||||
app.use(errorHandler);
|
||||
return app;
|
||||
}
|
||||
|
||||
export function receiptOnlyRecoveryApp(actor?: Parameters<typeof boardRecoveryApp>[0]) {
|
||||
const routeHeartbeat = heartbeatService(db);
|
||||
vi.spyOn(routeHeartbeat, "resumeTaskRetry").mockImplementation(async () => undefined as never);
|
||||
return boardRecoveryApp(actor, routeHeartbeat);
|
||||
}
|
||||
|
||||
export type RecoverySourceLayout = "retained" | "published" | "raw";
|
||||
|
||||
export async function acceptedRemoteExport(sourceLayout: RecoverySourceLayout = "retained") {
|
||||
const f = await seed("native", false);
|
||||
const source = await run(f.runId);
|
||||
let workspaceId: string = randomUUID(), remoteCwd = await realpath(f.remote);
|
||||
let preparedReference: Awaited<ReturnType<typeof prepareRemoteWorkspace>>["reference"] | undefined;
|
||||
if (sourceLayout === "published") {
|
||||
// Fresh modern sources retain the actual publisher's path and authority.
|
||||
// The explicit raw fixture remains available for refusal coverage.
|
||||
targets.get(f.lease.id)!.remoteCwd = remoteCwd;
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
||||
await db.update(issues).set({ status: "in_progress", executionRunId: f.runId, executionLockedAt: new Date() })
|
||||
.where(eq(issues.id, f.issueId));
|
||||
await db.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
const leaseMetadata = { ...f.lease.metadata };
|
||||
delete leaseMetadata.remoteExecutionTermination;
|
||||
const [activeLease] = await db.update(environmentLeases).set({ status: "active", releasedAt: null,
|
||||
cleanupStatus: null, metadata: leaseMetadata }).where(eq(environmentLeases.id, f.lease.id)).returning();
|
||||
const reserved = await reserveRemoteWorkspace({ db, companyId: f.companyId, issueId: f.issueId,
|
||||
runId: f.runId, agentId: source.agentId, environmentId: activeLease.environmentId!,
|
||||
projectId: null, projectWorkspaceId: null, existingWorkspaceId: null, source: { kind: "empty" } });
|
||||
const prepared = await prepareRemoteWorkspace({ db, workspace: reserved, runId: f.runId,
|
||||
lease: activeLease as EnvironmentLease, target: targets.get(f.lease.id)! });
|
||||
workspaceId = prepared.workspace.id; remoteCwd = prepared.remoteRoot; preparedReference = prepared.reference;
|
||||
for (const name of ["existing.txt", "saved.txt"]) await cp(path.join(f.remote, name), path.join(remoteCwd, name));
|
||||
await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success", releasedAt: new Date(),
|
||||
metadata: { ...prepared.lease.metadata,
|
||||
remoteExecutionTermination: remoteTerminationReceipt(prepared.lease, { providerLeaseId: f.providerLeaseId, state: "stopped" }) },
|
||||
}).where(eq(environmentLeases.id, f.lease.id));
|
||||
await db.update(heartbeatRuns).set({ status: source.status, finishedAt: source.finishedAt,
|
||||
error: source.error, errorCode: source.errorCode }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.update(issues).set({ status: issue.status, executionRunId: issue.executionRunId,
|
||||
executionLockedAt: issue.executionLockedAt }).where(eq(issues.id, f.issueId));
|
||||
} else if (sourceLayout === "retained") {
|
||||
const publishedRoot = path.join(path.dirname(f.remote), ".paperclip-workspaces", workspaceId);
|
||||
await mkdir(path.dirname(publishedRoot), { recursive: true });
|
||||
await rename(f.remote, publishedRoot);
|
||||
f.remote = publishedRoot;
|
||||
remoteCwd = await realpath(f.remote);
|
||||
}
|
||||
const git = (args: string[], cwd = remoteCwd) => new Promise<string>((resolve, reject) =>
|
||||
execFile("git", args, { cwd }, (error, stdout) => error ? reject(error) : resolve(stdout.trim())));
|
||||
await git(["init"]);
|
||||
await git(["add", "."]);
|
||||
await git(["-c", "user.name=Recovery fixture", "-c", "[email protected]", "commit", "-m", "Completed work"]);
|
||||
const commit = await git(["rev-parse", "HEAD"]);
|
||||
targets.get(f.lease.id)!.remoteCwd = remoteCwd;
|
||||
f.runtime.resumeRunLease.mockImplementation(async () => ({ providerLeaseId: f.providerLeaseId, metadata: { remoteCwd } }));
|
||||
const authority = { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: null,
|
||||
source: { kind: "empty" }, writerRunId: f.runId, dirty: true,
|
||||
location: { environmentId: f.lease.environmentId!, leaseId: f.lease.id, providerLeaseId: f.providerLeaseId, remoteCwd } };
|
||||
if (sourceLayout !== "published") await db.insert(executionWorkspaces).values({ id: workspaceId, companyId: f.companyId, sourceIssueId: f.issueId,
|
||||
name: "Completed remote work", cwd: remoteCwd, providerType: "cloud_sandbox", mode: "isolated_workspace",
|
||||
strategyType: "project_primary", status: "active", metadata: { remoteWorkspace: authority } });
|
||||
await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, f.issueId));
|
||||
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
await db.update(environmentLeases).set({ executionWorkspaceId: workspaceId, leasePolicy: "reuse_by_environment",
|
||||
metadata: { ...lease.metadata, remoteCwd, agentId: source.agentId } }).where(eq(environmentLeases.id, lease.id));
|
||||
const contractId = randomUUID(), resultId = randomUUID();
|
||||
await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issueId,
|
||||
revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "low",
|
||||
completionAuthority: "agent_claim_policy", incompleteCriteriaPolicy: "preserve_non_terminal",
|
||||
contractJson: { revision: CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim.contractRevision,
|
||||
objective: "Complete the work", criteria: [{ id: "objective", requirement: "Complete the task" }] },
|
||||
canonicalSha256: "contract", createdByActorType: "system", createdByActorId: "test" });
|
||||
await db.update(heartbeatRuns).set({ completionContractId: contractId, resultJson: {}, runnerProfileJson: {
|
||||
adapterDispatch: { adapterType: "paperclip_runner" },
|
||||
remoteWorkspace: preparedReference ?? { schema: "paperclip.remote-workspace-run.v1", workspaceId, leaseId: lease.id,
|
||||
providerLeaseId: f.providerLeaseId, remoteCwd, epoch: 1, state: "prepared", snapshotId: null },
|
||||
} }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId, runId: f.runId,
|
||||
completionContractId: contractId, schemaStatus: "accepted", serverFingerprint: "result", canonicalSha256: "result",
|
||||
resultJson: { terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, result: CONTROL_PLANE_CONFORMANCE_RESULT } });
|
||||
await db.update(nativeRunFinalizations).set({ resultId, failureCode: "native_workspace_sync_out_retry_exhausted" })
|
||||
.where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
await db.update(issueRecoveryActions).set({ cause: "native_workspace_sync_out_retry_exhausted", evidence: { runId: f.runId } })
|
||||
.where(eq(issueRecoveryActions.id, f.actionId));
|
||||
await adoptSystemRecovery(db, f);
|
||||
return { ...f, ...(sourceLayout === "published" ? { remote: remoteCwd } : {}), source, workspaceId, resultId, git, commit };
|
||||
}
|
||||
|
||||
export async function interruptedRemotePreparation(interruption: "shutdown" | "lease" | "process_lost" = "shutdown", sourceLayout: RecoverySourceLayout = "retained") {
|
||||
const f = await acceptedRemoteExport(sourceLayout);
|
||||
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
await db.delete(nativeRunResults).where(eq(nativeRunResults.runId, f.runId));
|
||||
const recovery = { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [f.lease.id] };
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", nativeIssueId: null, completionContractId: null,
|
||||
status: interruption === "process_lost" ? "failed" : "interrupted", executionStage: "preparing",
|
||||
errorCode: interruption === "process_lost" ? "process_lost" : interruption === "lease" ? "lease_released_before_terminal" : "server_shutdown_interrupted",
|
||||
controllerBootId: randomUUID(),
|
||||
error: interruption === "process_lost" ? "Process lost -- server may have restarted; retrying once"
|
||||
: interruption === "lease" ? "Environment lease released before preparation settled" : "Interrupted by graceful server shutdown (SIGTERM)",
|
||||
resultJson: { workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: recovery,
|
||||
...(interruption === "process_lost" ? { executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
||||
processLossDiagnostic: { localCheck: "no_identifiers", pidRecorded: false, groupRecorded: false, retryEligible: true, runPredatesObserver: true } } : {}),
|
||||
...(interruption === "lease" ? { cancellation: { source: "unknown", expected: false,
|
||||
initiator: { type: "system" }, reason: "Environment lease released before preparation settled",
|
||||
recordedAt: new Date().toISOString() } } : {}) },
|
||||
}).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.update(issues).set({ statusVersion: 2 }).where(eq(issues.id, f.issueId));
|
||||
await db.update(issueRecoveryActions).set({ cause: "legacy_execution_requires_reconciliation",
|
||||
fingerprint: `legacy-execution:${f.runId}`, status: "resolved", outcome: "blocked",
|
||||
evidence: { runId: f.runId, workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: recovery,
|
||||
automaticRecovery: { policy: "preserve_without_replay_v1", runId: f.runId, replay: "blocked", actionOutcome: "unknown" },
|
||||
nativeBootstrapFailureBlock: { runId: f.runId, previousStatus: "in_progress", statusVersion: 2 } },
|
||||
}).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
await adoptSystemRecovery(db, f);
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.source.agentId, status: "running" });
|
||||
return f;
|
||||
}
|
||||
|
||||
export async function withheldPreparationRetry(f: { companyId: string; issueId: string; runId: string; actionId: string }) {
|
||||
const source = await run(f.runId);
|
||||
const [withheld] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId,
|
||||
responsibleUserId: source.responsibleUserId, scopeKind: "issue", issueId: f.issueId, runtimeMode: "legacy", status: "cancelled", finishedAt: new Date(),
|
||||
retryOfRunId: f.runId, scheduledRetryAttempt: 1, scheduledRetryReason: "transient_failure",
|
||||
errorCode: "execution_reconciliation_required", contextSnapshot: { issueId: f.issueId },
|
||||
resultJson: { stopReason: "execution_reconciliation_required", timeoutSource: "stale_queued_run_gate",
|
||||
executionWait: { issueId: f.issueId, recoveryActionId: f.actionId } },
|
||||
}).returning();
|
||||
return withheld;
|
||||
}
|
||||
|
||||
export async function failedRemotePreparation(sourceLayout: RecoverySourceLayout = "retained", startedMarker = false, warning = false) {
|
||||
const f = await interruptedRemotePreparation("shutdown", sourceLayout);
|
||||
const source = await run(f.runId);
|
||||
const error = "Repository access is no longer available; refresh the repository catalog";
|
||||
await terminalizeLegacyExecution({ db, run: source, status: "failed", reconcileIfNeeded: false,
|
||||
patch: { errorCode: "setup_failed", error, executionStage: "preparing",
|
||||
startedAt: new Date(Date.now() - 1000), contextSnapshot: { ...source.contextSnapshot, source: "native_status_decision" },
|
||||
resultJson: { ...source.resultJson, stopReason: "adapter_failed",
|
||||
executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } } });
|
||||
if (warning) await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId, agentId: source.agentId,
|
||||
eventType: "agent_observability.unavailable", stream: "system", level: "warn",
|
||||
payload: { reason: "unsupported_harness", revision: randomUUID() } });
|
||||
if (startedMarker) await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId, agentId: source.agentId,
|
||||
eventType: "lifecycle", stream: "system", level: "info", message: "run started" });
|
||||
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId, agentId: source.agentId,
|
||||
eventType: "error", stream: "system", level: "error", message: error });
|
||||
await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId, agentId: source.agentId,
|
||||
eventType: "lifecycle", stream: "system", level: "info", message: "task.run.measured",
|
||||
payload: { phase: "task.run.measured", outcome: "failed", durationMs: 1000, durableSettlement: true } });
|
||||
return f;
|
||||
}
|
||||
|
||||
export async function stoppedWorkspaceWorker(f: { runId: string }) {
|
||||
const source = await run(f.runId);
|
||||
const deploymentWorker = { provider: "railway" as const, projectId: randomUUID(), environmentId: randomUUID(), serviceId: randomUUID(), deploymentId: randomUUID() };
|
||||
const owner = { token: randomUUID(), hostname: "previous-controller", pid: 123,
|
||||
processStartedAt: new Date().toISOString(), worker: deploymentWorker };
|
||||
const stopProof = { v: 1 as const, iss: "paperclip-cloud" as const, aud: "paperclip-workspace-worker-stop/v1" as const,
|
||||
sub: "test-stack", stopped: true as const, iat: Math.floor(Date.now() / 1000),
|
||||
exp: Math.floor(Date.now() / 1000) + 30, runId: f.runId, ownerToken: owner.token, worker: deploymentWorker };
|
||||
const proof = vi.spyOn(workerProof, "requestWorkspaceWorkerStopProof").mockResolvedValue({ status: "verified", proof: stopProof });
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: {
|
||||
...source.runnerProfileJson, nativeWorkspaceFinalizationOwner: owner,
|
||||
} }).where(eq(heartbeatRuns.id, f.runId));
|
||||
return { proof, stopProof, owner, deploymentWorker };
|
||||
}
|
||||
|
||||
export async function warmHandoffSource(hops = 1, beforeStop?: (lease: typeof environmentLeases.$inferSelect) => Promise<void>) {
|
||||
const prior = await seed("native", true, "keep_running", true), source = await run(prior.runId);
|
||||
let previousTarget = targets.get(prior.lease.id)!, previousLeaseId = prior.lease.id;
|
||||
let previousRunId: string = prior.runId;
|
||||
let lease = prior.lease, current = source;
|
||||
const leaseIds = [previousLeaseId];
|
||||
for (let hop = 0; hop < hops; hop++) {
|
||||
expect(await resumeNativeWorkspaceSync({ db, runId: previousRunId, target: previousTarget })).toBe(true);
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", errorCode: null, resultJson: null }).where(eq(heartbeatRuns.id, previousRunId));
|
||||
await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, previousRunId));
|
||||
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.id, prior.actionId));
|
||||
await environmentService(db).releaseLease(previousLeaseId, "released", { cleanupStatus: "success" });
|
||||
const [released] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, previousLeaseId));
|
||||
[current] = await db.insert(heartbeatRuns).values({ companyId: prior.companyId, agentId: source.agentId,
|
||||
issueId: prior.issueId, nativeIssueId: prior.issueId, runtimeMode: "native", status: "cancelled", finishedAt: new Date(),
|
||||
contextSnapshot: { issueId: prior.issueId } }).returning();
|
||||
await db.insert(nativeRunFinalizations).values({ companyId: prior.companyId, issueId: prior.issueId, runId: current.id, phase: "terminal_failure" });
|
||||
const acquired = await environmentService(db).acquireLease({ companyId: prior.companyId, environmentId: prior.lease.environmentId!,
|
||||
issueId: prior.issueId, heartbeatRunId: current.id, provider: "daytona", providerLeaseId: prior.providerLeaseId,
|
||||
leasePolicy: "reuse_by_environment", replacesReusableLeaseId: previousLeaseId,
|
||||
metadata: { ...released.metadata, sandboxLeaseAcquisition: { outcome: "resumed" } } });
|
||||
[lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, acquired.id));
|
||||
const target = { ...previousTarget, leaseId: lease.id, sandboxLeaseAcquisition: { outcome: "resumed" as const, providerLeaseId: prior.providerLeaseId } };
|
||||
targets.set(lease.id, target);
|
||||
expect(await prepareNativeWorkspaceSync({ db, companyId: prior.companyId, workspaceId: prior.issueId,
|
||||
workspaceLocalDir: prior.host, runId: current.id, lease: lease as EnvironmentLease, target, resourceDisposition: "keep_running" })).not.toBeNull();
|
||||
leaseIds.push(lease.id); previousRunId = current.id; previousLeaseId = lease.id; previousTarget = target;
|
||||
}
|
||||
[lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id));
|
||||
await beforeStop?.(lease);
|
||||
await writeFile(path.join(prior.remote, "after-warm-stop.txt"), "stopped turn work\n");
|
||||
await recordRequiredWorkspaceRestoreFailure(db, await run(current.id), { workspaceRestoreFailure: "restore_failed" }, lease);
|
||||
const [action] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, prior.companyId),
|
||||
eq(issueRecoveryActions.fingerprint, `native-workspace:${current.id}`)));
|
||||
const [retained] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id));
|
||||
const stopping = readStopOnlyCleanup(retained) ? retained : await prepareSandboxStopAndRetain(db, retained);
|
||||
expect(stopping).not.toBeNull();
|
||||
expect(await settleStopOnlyCleanup(db, stopping!, { attemptId: String(stopping!.metadata!.pendingCleanupAttemptId),
|
||||
receipt: { providerLeaseId: prior.providerLeaseId, state: "stopped" } })).not.toBeNull();
|
||||
return { ...prior, priorRunId: prior.runId, priorLeaseId: prior.lease.id, leaseIds, runId: current.id, lease, actionId: action.id,
|
||||
input: { ...prior.input, runId: current.id, actionId: action.id } };
|
||||
}
|
||||
|
||||
export async function makeRetryDue(actionId: string) {
|
||||
const current = await action(actionId);
|
||||
const state = readSystemRecovery(current.evidence)!;
|
||||
await db.update(issueRecoveryActions).set({ evidence: { ...current.evidence,
|
||||
systemRecovery: { ...state, nextAttemptAt: 0 } } }).where(eq(issueRecoveryActions.id, actionId));
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,496 @@
|
||||
import { nextWorkspaceRecoveryAt } from "./system-recovery.js";
|
||||
import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js";
|
||||
import { isPreDispatchExecutionWaitVerified } from "./pre-dispatch-execution-wait.js";
|
||||
import { heartbeatRunEvents, companyObservability, executionWorkspaceRepositories, issueWorkProducts, authUsers, companyMemberships, agents, createDb, environments, heartbeatRuns, issueRecoveryActions, issues, nativeRunResults, executionWorkspaces } from "@paperclipai/db";
|
||||
import { settleInterruptedNativeBootstrap } from "./legacy-execution-recovery.js";
|
||||
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "./delivery-work-notifications.js";
|
||||
import { environmentService } from "./environments.js";
|
||||
import * as agentStarts from "./agent-start-lock.js";
|
||||
import { taskRepositoryRelativePath } from "./execution-workspace-repositories.js";
|
||||
import { classifyEnvironmentCapabilities, environmentRuntimeService } from "./environment-runtime.js";
|
||||
import { instanceSettingsService } from "./instance-settings.js";
|
||||
import * as storageBackend from "../storage/index.js";
|
||||
import { getStorageService } from "../storage/index.js";
|
||||
import { readFreshWorkspaceReadiness } from "./fresh-workspace-continuation.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { cp, mkdir, readFile, realpath, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import request from "supertest";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { CONTROL_PLANE_CONFORMANCE_RESULT, CONTROL_PLANE_CONFORMANCE_TERMINAL } from "../vendor/paperclip-runner/testing.js";
|
||||
import type { NativeExecutionInput, NativeSession, NativeSessionBackend, PrpEvent } from "../vendor/paperclip-runner/index.js";
|
||||
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
||||
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import { recoverRetainedWorkspace, reconcileRetainedWorkspaces, retainedWorkspaceCandidates } from "./workspace-recovery.js";
|
||||
import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
||||
import { registerWorkspaceRecoveryWork } from "./workspace-recovery-work.js";
|
||||
import { heartbeatService } from "./heartbeat.js";
|
||||
import { hasVerifiedWorkspaceRepair } from "./workspace-recovery-receipt.js";
|
||||
import { createSavedWorkspaceLoader } from "./remote-workspaces.js";
|
||||
import { getExecutionBlocker } from "./execution-blocker.js";
|
||||
import { useWorkspaceRecoveryFixture, action, boardRecoveryApp, connectionString, db, failedRemotePreparation, interruptedRemotePreparation, root, run, stoppedWorkspaceWorker, withheldPreparationRetry } from "./__tests__/workspace-recovery-fixture.js";
|
||||
const targets = vi.hoisted(() => new Map<string, AdapterSandboxExecutionTarget>());
|
||||
vi.mock("./environment-execution-target.js", async importOriginal => {
|
||||
const actual = await importOriginal<typeof import("./environment-execution-target.js")>();
|
||||
return { ...actual, resolveEnvironmentExecutionTarget: async (input: Parameters<typeof actual.resolveEnvironmentExecutionTarget>[0]) =>
|
||||
(input.leaseId ? targets.get(input.leaseId) : undefined) ?? actual.resolveEnvironmentExecutionTarget(input) };
|
||||
});
|
||||
|
||||
describe("automatic retained workspace recovery", () => {
|
||||
useWorkspaceRecoveryFixture(targets);
|
||||
|
||||
it.each(["complete", "stopped_worker", "automatic_lost", "catalog", "catalog_warning", "catalog_unknown_warning", "catalog_provider_warning", "invalid_published_root"] as const)(
|
||||
"dispatches the recovered failed-preparation Retry once through the native provider boundary (%s)", async mode => {
|
||||
const unavailableCatalog = mode !== "complete" && mode !== "stopped_worker" && mode !== "automatic_lost" && mode !== "invalid_published_root", withWarning = mode.includes("warning");
|
||||
const localStorage = getStorageService();
|
||||
const databaseUrl = process.env.DATABASE_URL, masterKey = process.env.PAPERCLIP_SECRETS_MASTER_KEY;
|
||||
const experimental = await instanceSettingsService(db).getExperimental();
|
||||
let stoppedWorker: Awaited<ReturnType<typeof stoppedWorkspaceWorker>> | undefined;
|
||||
let storage: { mockRestore(): void } | undefined;
|
||||
const starts = vi.spyOn(agentStarts, "withAgentStartLock");
|
||||
const retryResumes: Promise<void>[] = [];
|
||||
let resumeSpy: { mockRestore(): void } | undefined;
|
||||
let ownedHeartbeat: ReturnType<typeof heartbeatService> | undefined;
|
||||
let releaseStart: (() => void) | undefined;
|
||||
let heldStart: Promise<void> | undefined;
|
||||
let explicitResume: Promise<void> | undefined;
|
||||
let recoveryCoordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
let recoveryWorker: ReturnType<typeof registerWorkspaceRecoveryWork> | undefined;
|
||||
try {
|
||||
storage = vi.spyOn(storageBackend, "getStorageService").mockReturnValue({ ...localStorage, provider: "s3" });
|
||||
process.env.DATABASE_URL = connectionString; process.env.PAPERCLIP_SECRETS_MASTER_KEY = "63".repeat(32);
|
||||
await instanceSettingsService(db).updateExperimental({ enableNativeRunner: true, enableIsolatedWorkspaces: true });
|
||||
const f = mode === "automatic_lost" ? await interruptedRemotePreparation("process_lost", "published")
|
||||
: await failedRemotePreparation(mode === "invalid_published_root" ? "raw" : "published");
|
||||
if (mode === "automatic_lost") {
|
||||
await db.update(heartbeatRuns).set({ responsibleUserId: "local-board" }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await withheldPreparationRetry(f);
|
||||
}
|
||||
if (mode === "complete") await db.update(heartbeatRuns).set({ sessionIdBefore: randomUUID() }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (mode === "stopped_worker") stoppedWorker = await stoppedWorkspaceWorker(f);
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
if (stoppedWorker) {
|
||||
expect((await run(f.runId)).runnerProfileJson?.nativeWorkspaceOwnerStopProof).toEqual(stoppedWorker.stopProof);
|
||||
expect((await run(f.runId)).runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeUndefined();
|
||||
}
|
||||
const source = await run(f.runId), nextDirectory = path.join(root, randomUUID(), "next");
|
||||
await mkdir(nextDirectory, { recursive: true });
|
||||
const nextRemote = await realpath(nextDirectory);
|
||||
await db.insert(authUsers).values({ id: "local-board", name: "Local operator", email: "[email protected]",
|
||||
createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing();
|
||||
await db.insert(companyMemberships).values({ companyId: f.companyId, principalType: "user", principalId: "local-board", membershipRole: "owner" });
|
||||
await db.update(agents).set({ defaultEnvironmentId: f.lease.environmentId,
|
||||
adapterConfig: withWarning ? { provider: "acpx", acpxAgent: "pi", model: "openrouter/deepseek/deepseek-v4.1-flash-1017" }
|
||||
: { provider: "codex", model: "gpt-6.1-sol" } }).where(eq(agents.id, source.agentId));
|
||||
if (withWarning) await db.insert(companyObservability).values({ companyId: f.companyId, revision: randomUUID(), enabled: true,
|
||||
destination: { provider: "otlp", endpointBaseUrl: "https://collector.example.invalid/otel" },
|
||||
capture: { userPrompts: false, assistantResponses: false, toolDetails: false, toolContent: false } });
|
||||
await db.update(environments).set({ config: { provider: "fake", image: "fake:test" } }).where(eq(environments.id, f.lease.environmentId!));
|
||||
const environment = environmentService(db), oldTarget = targets.get(f.lease.id)!;
|
||||
oldTarget.runner!.syncIn = async operations => {
|
||||
for (const operation of operations) for (const file of operation.files) {
|
||||
await mkdir(path.dirname(file.targetPath), { recursive: true });
|
||||
await cp(file.sourcePath, file.targetPath, { recursive: true, dereference: false });
|
||||
}
|
||||
return { filesTransferred: operations.reduce((count, operation) => count + operation.files.length, 0), bytesTransferred: 0, operations: [] };
|
||||
};
|
||||
const filesystemExecute = oldTarget.runner!.execute;
|
||||
const fixtureCodex = "/opt/paperclip-fixture/bin/codex";
|
||||
const executeCommand: typeof filesystemExecute = async input => {
|
||||
if (input.command === "sh" && input.args?.some(arg => arg.includes("command -v codex"))) {
|
||||
return { exitCode: 0, signal: null, timedOut: false, pid: null, startedAt: new Date().toISOString(), stdout: `${fixtureCodex}\n`, stderr: "" };
|
||||
}
|
||||
if (input.command === fixtureCodex && input.args?.[0] === "--version") {
|
||||
return { exitCode: 0, signal: null, timedOut: false, pid: null, startedAt: new Date().toISOString(), stdout: "codex-cli 0.160.0\n", stderr: "" };
|
||||
}
|
||||
return filesystemExecute(input);
|
||||
};
|
||||
const runtime = environmentRuntimeService(db, { drivers: [{ driver: "sandbox",
|
||||
acquireRunLease: async input => {
|
||||
const lease = await environment.acquireLease({ companyId: input.companyId, environmentId: input.environment.id,
|
||||
issueId: input.issueId, heartbeatRunId: input.heartbeatRunId, executionWorkspaceId: input.executionWorkspaceId,
|
||||
provider: "fake", providerLeaseId: randomUUID(), leasePolicy: "retain_on_failure",
|
||||
metadata: { driver: "sandbox", remoteCwd: nextRemote, agentId: input.agentId } });
|
||||
targets.set(lease.id, { ...oldTarget, leaseId: lease.id, remoteCwd: nextRemote,
|
||||
runner: { ...oldTarget.runner!, execute: executeCommand },
|
||||
sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: lease.providerLeaseId! } });
|
||||
return lease;
|
||||
},
|
||||
releaseRunLease: async input => environment.releaseLease(input.lease.id, input.status, { cleanupStatus: "success",
|
||||
remoteExecutionTermination: remoteTerminationReceipt(input.lease, { providerLeaseId: input.lease.providerLeaseId, state: "stopped" }) }),
|
||||
resumeRunLease: async ({ lease }) => ({ providerLeaseId: lease.providerLeaseId!, metadata: { remoteCwd: lease.metadata?.remoteCwd } }),
|
||||
retryPendingSandboxTeardown: async ({ lease }) => ({ providerLeaseId: lease.providerLeaseId, state: "stopped" }),
|
||||
realizeWorkspace: async () => ({ cwd: nextRemote, metadata: { remoteCwd: nextRemote } }),
|
||||
execute: async input => executeCommand({ command: input.command, args: input.args, cwd: input.cwd,
|
||||
env: input.env, stdin: input.stdin, timeoutMs: input.timeoutMs }),
|
||||
resolveCapabilities: async () => classifyEnvironmentCapabilities({ verifiedMethods: [], declared: null, narrowing: null }),
|
||||
}] });
|
||||
const workProductId = randomUUID(), evidenceRef = `work_product:${workProductId}`;
|
||||
await db.insert(issueWorkProducts).values({ id: workProductId, companyId: f.companyId, issueId: f.issueId,
|
||||
type: "artifact", provider: "paperclip", title: "Continued saved work", status: "ready_for_review", reviewState: "approved" });
|
||||
const inputs: NativeExecutionInput[] = [], startTurn = vi.fn();
|
||||
const backendFactory = (execution: NativeExecutionInput): NativeSessionBackend => {
|
||||
inputs.push(execution);
|
||||
const identity = { companyId: f.companyId, agentId: source.agentId, issueId: f.issueId,
|
||||
runId: execution.binding.runId, sessionId: execution.session.normalizedSessionId! };
|
||||
const turnId = `turn-${identity.runId}`, result = structuredClone(CONTROL_PLANE_CONFORMANCE_RESULT);
|
||||
result.completionClaim.contractRevision = execution.completionContract.contract.revision;
|
||||
result.completionClaim.criteria = execution.completionContract.contract.criteria.map(criterion => ({
|
||||
...result.completionClaim.criteria[0]!, criterionId: criterion.id, evidenceRefs: [evidenceRef] }));
|
||||
result.evidence = [{ kind: "work_product", ref: evidenceRef }]; result.verification[0]!.artifactRef = evidenceRef;
|
||||
let started!: () => void;
|
||||
const turnStarted = new Promise<void>(resolve => { started = resolve; });
|
||||
const session: NativeSession = {
|
||||
identity: () => identity,
|
||||
async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; },
|
||||
async *events() {
|
||||
await turnStarted;
|
||||
const current = await run(identity.runId);
|
||||
yield { schema: "paperclip.prp.event.v1", sourceEventId: `${identity.runId}:terminal`, sourceSeq: 1,
|
||||
sourceInstanceId: current.runnerInstanceId!, sourceKind: "runner", runId: identity.runId,
|
||||
normalizedSessionId: identity.sessionId, turnId, eventType: "turn.completed", schemaVersion: 1,
|
||||
priority: 0, emittedAt: new Date().toISOString(), payload: {} } satisfies PrpEvent;
|
||||
},
|
||||
async startTurn(input) {
|
||||
expect(execution.provider).toMatchObject({ kind: "codex", model: "gpt-6.1-sol" });
|
||||
expect(await readFile(path.join(execution.workspace.cwd!, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
await writeFile(path.join(execution.workspace.cwd!, "continued.txt"), "continued from preserved work\n");
|
||||
startTurn(input); started(); return { turnId };
|
||||
},
|
||||
async result() { await turnStarted; return { result, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, turnId }; },
|
||||
async snapshot() { return { backendKind: "mock", sessionId: "recovered-preparation", identity,
|
||||
providerSessionId: "recovered-preparation", activeTurnId: null, pendingRuntimeRequests: [], lineage: [], cursor: "1" }; },
|
||||
async close() {},
|
||||
};
|
||||
return { async descriptor() { return { kind: "mock", name: "recovered-preparation", version: "1",
|
||||
capabilities: { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true },
|
||||
runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } }; },
|
||||
async openSession() { return session; } };
|
||||
};
|
||||
const heartbeat = heartbeatService(db, { environmentRuntime: runtime, nativeSessionBackendFactory: backendFactory });
|
||||
ownedHeartbeat = heartbeat;
|
||||
const resumeTaskRetry = heartbeat.resumeTaskRetry;
|
||||
resumeSpy = vi.spyOn(heartbeat, "resumeTaskRetry").mockImplementation((...args) => {
|
||||
const pending = resumeTaskRetry(...args);
|
||||
retryResumes.push(pending);
|
||||
return pending;
|
||||
});
|
||||
const app = boardRecoveryApp(undefined, heartbeat);
|
||||
const gate = new Promise<void>(resolve => { releaseStart = resolve; });
|
||||
heldStart = agentStarts.withAgentStartLock(source.agentId, () => gate);
|
||||
const receipt = mode === "automatic_lost" ? null : (await request(app).post(`/api/issues/${f.issueId}/retry`)
|
||||
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() }).expect(202)).body;
|
||||
if (mode === "automatic_lost") {
|
||||
await Promise.all([heartbeat.resumeRemoteStopComments(await run(f.runId)),
|
||||
heartbeat.resumeRemoteStopComments(await run(f.runId))]);
|
||||
explicitResume = heartbeat.promoteDueScheduledRetries(new Date(Date.now() + 300_000)).then(() => undefined);
|
||||
} else {
|
||||
await expect.poll(() => starts.mock.calls.filter(([agentId]) => agentId === source.agentId).length, { timeout: 10_000 }).toBe(2);
|
||||
explicitResume = heartbeat.resumeTaskRetry(f.companyId, receipt.requestId);
|
||||
}
|
||||
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
mode === "automatic_lost" ? eq(heartbeatRuns.scheduledRetryReason, "interrupted_preparation")
|
||||
: sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${receipt.requestId}`));
|
||||
expect(successors).toHaveLength(1);
|
||||
if (unavailableCatalog) {
|
||||
const repoUrl = "https://github.com/example/revoked.git";
|
||||
await db.insert(executionWorkspaceRepositories).values({ companyId: f.companyId, executionWorkspaceId: f.workspaceId,
|
||||
requestedByIssueId: f.issueId, repositoryIdentity: repoUrl, repoUrl, catalogRepositoryId: "revoked-before-dispatch",
|
||||
relativePath: taskRepositoryRelativePath(repoUrl), requestKey: randomUUID(), state: "ready", pinnedCommit: f.commit });
|
||||
}
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(and(
|
||||
eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "running")));
|
||||
releaseStart!(); await Promise.all([heldStart, explicitResume]);
|
||||
await heartbeat.resumeQueuedRuns(); await heartbeat.drainActiveRunExecutions();
|
||||
const completed = await run(successors[0].id);
|
||||
if (mode === "invalid_published_root") {
|
||||
expect(completed).toMatchObject({ status: "failed", runtimeMode: "legacy", errorCode: "setup_failed",
|
||||
error: "remote_workspace_published_path_invalid" });
|
||||
expect(inputs).toHaveLength(0); expect(startTurn).not.toHaveBeenCalled();
|
||||
expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, completed.id))).toHaveLength(0);
|
||||
expect(await readFile(path.join(f.remote, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect((await run(f.runId)).error).toBe(source.error);
|
||||
return;
|
||||
}
|
||||
if (unavailableCatalog) {
|
||||
expect(completed).toMatchObject({ status: "failed", errorCode: "setup_failed",
|
||||
error: "Repository access is no longer available; refresh the repository catalog" });
|
||||
expect(inputs).toHaveLength(0); expect(startTurn).not.toHaveBeenCalled();
|
||||
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId));
|
||||
expect(await readFile(path.join(workspace.cwd!, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect((await run(f.runId)).error).toBe(source.error);
|
||||
expect(completed).toMatchObject({ runtimeMode: "legacy", runtimeModeResolvedAt: null, executionStage: "preparing",
|
||||
nativeIssueId: null, nativeSessionId: null, usageJson: { ledgerScope: expect.any(Object) },
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } });
|
||||
expect(completed.usageJson).toEqual({ ledgerScope: { issueId: f.issueId, projectId: null, billingCode: null } });
|
||||
if (withWarning) {
|
||||
const [warning] = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, completed.id), eq(heartbeatRunEvents.seq, 1)));
|
||||
expect(warning).toMatchObject({ eventType: "agent_observability.unavailable", stream: "system", level: "warn",
|
||||
payload: { reason: "unsupported_harness", revision: expect.any(String) }, sourceEventId: null, sourceInstanceId: null });
|
||||
if (mode === "catalog_unknown_warning") await db.update(heartbeatRunEvents).set({ payload: { ...warning.payload, reason: "unknown" } }).where(eq(heartbeatRunEvents.id, warning.id));
|
||||
if (mode === "catalog_provider_warning") await db.update(heartbeatRunEvents).set({ sourceEventId: randomUUID() }).where(eq(heartbeatRunEvents.id, warning.id));
|
||||
}
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId, status: "running" });
|
||||
const errors = vi.fn();
|
||||
recoveryCoordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: errors });
|
||||
recoveryWorker = registerWorkspaceRecoveryWork(db, recoveryCoordinator, runtime, heartbeat.resumeRemoteStopComments);
|
||||
await recoveryWorker.ready;
|
||||
const retryIntent = { observedRunId: completed.id, idempotencyKey: randomUUID() };
|
||||
const retry = (await request(app).post(`/api/issues/${f.issueId}/retry`).send(retryIntent).expect(202)).body;
|
||||
const resumed = () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${retry.requestId}`));
|
||||
await expect.poll(async () => await hasVerifiedWorkspaceRepair(db, await run(completed.id)), { timeout: 20_000 }).toBe(true);
|
||||
if (mode === "catalog_unknown_warning" || mode === "catalog_provider_warning") {
|
||||
await heartbeat.resumeTaskRetry(f.companyId, retry.requestId);
|
||||
expect(await resumed()).toHaveLength(0); expect(startTurn).not.toHaveBeenCalled();
|
||||
return;
|
||||
}
|
||||
await expect.poll(async () => (await resumed()).length, { timeout: 20_000 }).toBe(1);
|
||||
const [continuation] = await resumed();
|
||||
expect(continuation).toMatchObject({ status: "queued", contextSnapshot: { previousRunId: completed.id,
|
||||
taskRetryRequestId: retry.requestId } });
|
||||
expect(await readFreshWorkspaceReadiness(db, continuation)).toMatchObject({ mode: "resume_preserved_workspace" });
|
||||
expect((await request(app).post(`/api/issues/${f.issueId}/retry`).send(retryIntent).expect(202)).body.requestId).toBe(retry.requestId);
|
||||
expect(await resumed()).toHaveLength(1); expect(startTurn).not.toHaveBeenCalled();
|
||||
expect(errors).not.toHaveBeenCalled();
|
||||
return;
|
||||
}
|
||||
expect({ status: completed.status, error: completed.error, runtimeMode: completed.runtimeMode })
|
||||
.toEqual({ status: "succeeded", error: null, runtimeMode: "native" });
|
||||
expect(inputs).toHaveLength(1); expect(startTurn).toHaveBeenCalledOnce();
|
||||
expect(path.basename(path.dirname(inputs[0].workspace.cwd!))).toBe(".paperclip-workspaces");
|
||||
expect(completed.runnerProfileJson?.nativeExecutionInput).toMatchObject({ provider: { kind: "codex", model: "gpt-6.1-sol" } });
|
||||
expect(await readFile(path.join(inputs[0].workspace.cwd!, "continued.txt"), "utf8")).toBe("continued from preserved work\n");
|
||||
expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, completed.id)))
|
||||
.toEqual([expect.objectContaining({ schemaStatus: "accepted" })]);
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("done");
|
||||
expect((await request(app).get(`/api/issues/${f.issueId}/queued-comments`).expect(200)).body.entries).toEqual([]);
|
||||
expect((await request(app).get(`/api/issues/${f.issueId}`).expect(200)).body.interruption)
|
||||
.toMatchObject({ state: "resolved", canRetry: false });
|
||||
if (mode === "automatic_lost") await heartbeat.resumeRemoteStopComments(await run(f.runId));
|
||||
else await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId);
|
||||
expect(startTurn).toHaveBeenCalledOnce();
|
||||
expect((await run(f.runId)).error).toBe(source.error);
|
||||
} finally {
|
||||
try {
|
||||
releaseStart?.(); await Promise.allSettled([heldStart, explicitResume, ...retryResumes]);
|
||||
await recoveryCoordinator?.stop(); await recoveryWorker?.stop();
|
||||
await ownedHeartbeat?.drainActiveRunExecutions();
|
||||
} finally {
|
||||
resumeSpy?.mockRestore(); starts.mockRestore();
|
||||
stoppedWorker?.proof.mockRestore();
|
||||
storage?.mockRestore();
|
||||
if (databaseUrl === undefined) delete process.env.DATABASE_URL; else process.env.DATABASE_URL = databaseUrl;
|
||||
if (masterKey === undefined) delete process.env.PAPERCLIP_SECRETS_MASTER_KEY; else process.env.PAPERCLIP_SECRETS_MASTER_KEY = masterKey;
|
||||
await instanceSettingsService(db).updateExperimental(experimental);
|
||||
}
|
||||
}
|
||||
}, 40_000);
|
||||
|
||||
it.each(["shutdown", "lease"] as const)("admits HTTP Retry once after an interrupted Runner preparation recovers its remote workspace (%s)", async interruption => {
|
||||
const f = await interruptedRemotePreparation(interruption);
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
|
||||
const intent = { observedRunId: f.runId, idempotencyKey: randomUUID() };
|
||||
const app = boardRecoveryApp();
|
||||
const receipt = (await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body;
|
||||
const heartbeat = heartbeatService(db);
|
||||
await Promise.all([heartbeat.resumeTaskRetry(f.companyId, receipt.requestId),
|
||||
heartbeat.resumeTaskRetry(f.companyId, receipt.requestId)]);
|
||||
expect((await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body.requestId).toBe(receipt.requestId);
|
||||
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
eq(heartbeatRuns.status, "queued")));
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0].contextSnapshot).toMatchObject({ previousRunId: f.runId });
|
||||
expect(await readFreshWorkspaceReadiness(db, successors[0])).toMatchObject({ mode: "resume_preserved_workspace" });
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
||||
expect((await run(f.runId)).status).toBe("interrupted");
|
||||
const remote = (await run(f.runId)).runnerProfileJson!.remoteWorkspace as { snapshotId: string };
|
||||
const saved = await createSavedWorkspaceLoader(db)({ companyId: f.companyId, workspaceId: f.workspaceId,
|
||||
snapshotId: remote.snapshotId });
|
||||
try { expect(await readFile(path.join(saved.root, "saved.txt"), "utf8")).toBe("finished work\n"); }
|
||||
finally { await saved.cleanup(); }
|
||||
}, 30_000);
|
||||
|
||||
it.each([false, true].flatMap(alreadyRecovered => (["shutdown", "lease"] as const).map(interruption => ({ alreadyRecovered, interruption }))))(
|
||||
"automatically retries interrupted preparation after preservation ($interruption, startup=$alreadyRecovered)", async ({ alreadyRecovered, interruption }) => {
|
||||
const f = await interruptedRemotePreparation(interruption);
|
||||
if (alreadyRecovered) await recoverRetainedWorkspace(f.input);
|
||||
const restarted = createDb(connectionString);
|
||||
try {
|
||||
const heartbeat = heartbeatService(restarted);
|
||||
await Promise.all(await reconcileRetainedWorkspaces(restarted, f.environmentRuntime,
|
||||
heartbeat.resumeRemoteStopComments, true) ?? []);
|
||||
await Promise.all([heartbeat.resumeRemoteStopComments(await run(f.runId)),
|
||||
heartbeat.resumeRemoteStopComments(await run(f.runId))]);
|
||||
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
eq(heartbeatRuns.retryOfRunId, f.runId)));
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryAttempt: 1 });
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("in_progress");
|
||||
expect((await run(f.runId)).status).toBe("interrupted");
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
|
||||
} finally { await restarted.$client.end(); }
|
||||
}, 30_000);
|
||||
|
||||
it.each([false, true])("continues recovered lost preparation once after its first retry was withheld (startup=%s)", async alreadyRecovered => {
|
||||
const f = await interruptedRemotePreparation("process_lost");
|
||||
const withheld = await withheldPreparationRetry(f);
|
||||
expect(await isPreDispatchExecutionWaitVerified(db, withheld)).toBe(true);
|
||||
if (alreadyRecovered) await recoverRetainedWorkspace(f.input);
|
||||
const restarted = createDb(connectionString);
|
||||
try {
|
||||
const heartbeat = heartbeatService(restarted);
|
||||
await Promise.all(await reconcileRetainedWorkspaces(restarted, f.environmentRuntime,
|
||||
heartbeat.resumeRemoteStopComments, true) ?? []);
|
||||
await Promise.all([heartbeat.resumeRemoteStopComments(await run(f.runId)),
|
||||
heartbeat.resumeRemoteStopComments(await run(f.runId))]);
|
||||
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
eq(heartbeatRuns.status, "scheduled_retry")));
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0]).toMatchObject({ retryOfRunId: withheld.id, scheduledRetryAttempt: 2,
|
||||
scheduledRetryReason: "interrupted_preparation", contextSnapshot: { executionRetryAccounting: { failureRetries: 2 } } });
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
||||
expect((await run(f.runId)).status).toBe("failed");
|
||||
expect(await run(withheld.id)).toMatchObject({ status: "cancelled", startedAt: null });
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
|
||||
} finally { await restarted.$client.end(); }
|
||||
}, 30_000);
|
||||
|
||||
it.each(["withheld_first", "successor_first"] as const)("reuses the newest retry child without replacing a withheld sibling (%s)", async order => {
|
||||
const f = await interruptedRemotePreparation("process_lost");
|
||||
const source = await run(f.runId);
|
||||
const now = Date.now();
|
||||
const addSuccessor = async () => (await db.insert(heartbeatRuns).values({
|
||||
companyId: f.companyId, agentId: source.agentId, issueId: f.issueId, retryOfRunId: source.id,
|
||||
status: "failed", runtimeMode: "native", nativeIssueId: f.issueId,
|
||||
startedAt: new Date(now - 1000), finishedAt: new Date(now), createdAt: new Date(now - 1000),
|
||||
errorCode: "native_provider_terminal_failed", contextSnapshot: { issueId: f.issueId },
|
||||
scheduledRetryAttempt: 2, scheduledRetryReason: "transient_failure",
|
||||
}).returning())[0];
|
||||
const addWithheld = async () => {
|
||||
const child = await withheldPreparationRetry(f);
|
||||
await db.update(heartbeatRuns).set({ createdAt: new Date(now - 2000) }).where(eq(heartbeatRuns.id, child.id));
|
||||
return child;
|
||||
};
|
||||
let successor: typeof heartbeatRuns.$inferSelect;
|
||||
let withheld: typeof heartbeatRuns.$inferSelect;
|
||||
if (order === "withheld_first") { withheld = await addWithheld(); successor = await addSuccessor(); }
|
||||
else { successor = await addSuccessor(); withheld = await addWithheld(); }
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const heartbeat = heartbeatService(db);
|
||||
await Promise.all([heartbeat.resumeRemoteStopComments(await run(f.runId)),
|
||||
heartbeat.resumeRemoteStopComments(await run(f.runId))]);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, withheld.id))).toHaveLength(0);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.id))).toHaveLength(2);
|
||||
expect(await run(successor!.id)).toEqual(successor!);
|
||||
expect((await action(f.actionId)).evidence).toMatchObject({ bootstrapPreparationContinuation: "delivered",
|
||||
bootstrapPreparationContinuationRunId: successor!.id });
|
||||
expect(await nextWorkspaceRecoveryAt(db)).toBeNull();
|
||||
}, 30_000);
|
||||
|
||||
it.each([false, true])("continues a recovered lost preparation without an earlier retry (startup=%s)", async alreadyRecovered => {
|
||||
const f = await interruptedRemotePreparation("process_lost");
|
||||
if (alreadyRecovered) await recoverRetainedWorkspace(f.input);
|
||||
const restarted = createDb(connectionString);
|
||||
try {
|
||||
const heartbeat = heartbeatService(restarted);
|
||||
await Promise.all(await reconcileRetainedWorkspaces(restarted, f.environmentRuntime,
|
||||
heartbeat.resumeRemoteStopComments, true) ?? []);
|
||||
await Promise.all([heartbeat.resumeRemoteStopComments(await run(f.runId)),
|
||||
heartbeat.resumeRemoteStopComments(await run(f.runId))]);
|
||||
const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.runId));
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryAttempt: 1 });
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
||||
expect(await nextWorkspaceRecoveryAt(restarted)).toBeNull();
|
||||
} finally { await restarted.$client.end(); }
|
||||
}, 30_000);
|
||||
|
||||
it.each(["restart", "pause", "claimed", "duplicate"] as const)("retains preparation continuation after settlement before enqueue (%s)", async mode => {
|
||||
const f = await interruptedRemotePreparation("process_lost"), withheld = await withheldPreparationRetry(f);
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const source = await run(f.runId);
|
||||
expect(await settleInterruptedNativeBootstrap(db, { run: source, providerDispatchStarted: false })).not.toBeNull();
|
||||
expect((await action(f.actionId)).evidence.automaticRecovery).toMatchObject({ replay: "allowed" });
|
||||
expect(await nextWorkspaceRecoveryAt(db)).not.toBeNull();
|
||||
const duplicateId = mode === "duplicate" ? randomUUID() : null;
|
||||
if (duplicateId) {
|
||||
const prior = await action(f.actionId);
|
||||
await db.insert(issueRecoveryActions).values({ ...prior, id: duplicateId });
|
||||
}
|
||||
if (mode === "claimed") {
|
||||
const first = await heartbeatService(db).scheduleBoundedRetry(f.runId);
|
||||
expect(first.outcome).toBe("scheduled");
|
||||
expect((await action(f.actionId)).evidence.bootstrapPreparationContinuation).toBe("pending");
|
||||
}
|
||||
if (mode === "pause") await db.update(agents).set({ status: "paused" }).where(eq(agents.id, source.agentId));
|
||||
const restarted = createDb(connectionString);
|
||||
try {
|
||||
const heartbeat = heartbeatService(restarted);
|
||||
await Promise.all(await reconcileRetainedWorkspaces(restarted, f.environmentRuntime,
|
||||
heartbeat.resumeRemoteStopComments, true) ?? []);
|
||||
if (mode === "pause") {
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, withheld.id))).toHaveLength(0);
|
||||
expect(await nextWorkspaceRecoveryAt(restarted)).toBeNull();
|
||||
expect((await action(f.actionId)).evidence.bootstrapPreparationContinuation).toBe("deferred");
|
||||
const wake = vi.fn(), unsubscribe = subscribeDeliveryWork(restarted, DELIVERY_QUEUES.workspaceRecovery, wake);
|
||||
try {
|
||||
const rollback = new Error("rollback eligibility change");
|
||||
await expect(restarted.transaction(async tx => {
|
||||
await notifyDeferredExecutionEligibility(tx, { companyId: f.companyId });
|
||||
await tx.update(agents).set({ status: "idle" }).where(eq(agents.id, source.agentId));
|
||||
throw rollback;
|
||||
})).rejects.toBe(rollback);
|
||||
expect((await db.select().from(agents).where(eq(agents.id, source.agentId)))[0].status).toBe("paused");
|
||||
await Promise.all(await reconcileRetainedWorkspaces(restarted, f.environmentRuntime,
|
||||
heartbeat.resumeRemoteStopComments, true) ?? []);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, withheld.id))).toHaveLength(0);
|
||||
expect(await nextWorkspaceRecoveryAt(restarted)).toBeNull();
|
||||
wake.mockClear();
|
||||
await restarted.transaction(async tx => {
|
||||
await notifyDeferredExecutionEligibility(tx, { companyId: f.companyId });
|
||||
await tx.update(agents).set({ status: "idle" }).where(eq(agents.id, source.agentId));
|
||||
});
|
||||
expect(wake).toHaveBeenCalled();
|
||||
} finally { unsubscribe(); }
|
||||
await Promise.all(await reconcileRetainedWorkspaces(restarted, f.environmentRuntime,
|
||||
heartbeat.resumeRemoteStopComments, true) ?? []);
|
||||
}
|
||||
const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, withheld.id));
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryAttempt: 2 });
|
||||
expect((await action(f.actionId)).evidence).toMatchObject({ bootstrapPreparationContinuation: "delivered",
|
||||
bootstrapPreparationContinuationRunId: successors[0].id });
|
||||
expect(await nextWorkspaceRecoveryAt(restarted)).toBeNull();
|
||||
if (duplicateId) expect((await action(duplicateId)).evidence.bootstrapPreparationContinuation).toBe("delivered");
|
||||
expect(await retainedWorkspaceCandidates(restarted, undefined, false, true)).not.toEqual(
|
||||
expect.arrayContaining([expect.objectContaining({ run: expect.objectContaining({ id: f.runId }) })]));
|
||||
} finally { await restarted.$client.end(); }
|
||||
}, 30_000);
|
||||
|
||||
it("revisits a pending continuation after enqueue fails before a new notification", async () => {
|
||||
const f = await interruptedRemotePreparation("process_lost"), withheld = await withheldPreparationRetry(f);
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
expect(await settleInterruptedNativeBootstrap(db, { run: await run(f.runId), providerDispatchStarted: false })).not.toBeNull();
|
||||
const heartbeat = heartbeatService(db), passiveSweep = {};
|
||||
let attempts = 0;
|
||||
const onRecovered = async (source: Awaited<ReturnType<typeof run>>) => {
|
||||
if (source.id !== f.runId) return;
|
||||
if (++attempts === 1) throw new Error("database read unavailable before enqueue notification");
|
||||
await heartbeat.resumeRemoteStopComments(source);
|
||||
};
|
||||
await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, onRecovered, true, undefined, passiveSweep) ?? []);
|
||||
expect(attempts).toBe(1);
|
||||
expect(await nextWorkspaceRecoveryAt(db)).not.toBeNull();
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, withheld.id))).toHaveLength(0);
|
||||
await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, onRecovered, true, undefined, passiveSweep) ?? []);
|
||||
expect(attempts).toBe(2);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, withheld.id)))
|
||||
.toEqual([expect.objectContaining({ status: "scheduled_retry", scheduledRetryAttempt: 2 })]);
|
||||
expect(await nextWorkspaceRecoveryAt(db)).toBeNull();
|
||||
}, 30_000);
|
||||
|
||||
});
|
||||
@@ -0,0 +1,584 @@
|
||||
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
||||
import { heartbeatRunEvents, agents, approvals, costEvents, issueApprovals, environmentLeases, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, agentWakeupRequests, workspaceOperations, issueThreadInteractions, executionWorkspaces, executionWorkspaceSnapshots } from "@paperclipai/db";
|
||||
import { terminalizeLegacyExecution } from "./legacy-execution-recovery.js";
|
||||
import * as recoverySources from "./workspace-recovery-source.js";
|
||||
import { requestedRunCancellation } from "./run-cancellation.js";
|
||||
import { readSystemRecovery } from "./system-recovery-state.js";
|
||||
import { getStorageService } from "../storage/index.js";
|
||||
import { attachPreservedWorkspaceSnapshots } from "./workspace-recovery-snapshot.js";
|
||||
import { requestTaskRetry } from "./task-retry.js";
|
||||
import { readFreshWorkspaceReadiness } from "./fresh-workspace-continuation.js";
|
||||
import { randomBytes, randomUUID } from "node:crypto";
|
||||
import { mkdir, readFile, realpath, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import request from "supertest";
|
||||
import { and, eq, inArray, sql } from "drizzle-orm";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { finalizeNativeRun } from "./native-runtime/native-run-finalizer.js";
|
||||
import { issueRecoveryActionService } from "./issue-recovery-actions.js";
|
||||
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
||||
import type { EnvironmentLease } from "@paperclipai/shared";
|
||||
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
import { recoverRetainedWorkspace, requestWorkspaceRecovery, scheduleRecoveredWorkspaceCleanup, reconcileRetainedWorkspaces } from "./workspace-recovery.js";
|
||||
import { heartbeatService } from "./heartbeat.js";
|
||||
import { hasVerifiedWorkspaceRepair } from "./workspace-recovery-receipt.js";
|
||||
import { createSavedWorkspaceLoader, readRemoteWorkspaceAuthority, saveRemoteWorkspace } from "./remote-workspaces.js";
|
||||
import { createReusableLeaseHandoff } from "./environment-lease-handoff.js";
|
||||
import { preserveRemoteWorkspacePreparationFailure } from "./native-runtime/native-workspace-finalization-state.js";
|
||||
import { getExecutionBlocker } from "./execution-blocker.js";
|
||||
import { useWorkspaceRecoveryFixture, acceptedRemoteExport, action, boardRecoveryApp, db, interruptedRemotePreparation, makeRetryDue, root, run, seed, withheldPreparationRetry } from "./__tests__/workspace-recovery-fixture.js";
|
||||
const targets = vi.hoisted(() => new Map<string, AdapterSandboxExecutionTarget>());
|
||||
vi.mock("./environment-execution-target.js", async importOriginal => {
|
||||
const actual = await importOriginal<typeof import("./environment-execution-target.js")>();
|
||||
return { ...actual, resolveEnvironmentExecutionTarget: async (input: Parameters<typeof actual.resolveEnvironmentExecutionTarget>[0]) =>
|
||||
(input.leaseId ? targets.get(input.leaseId) : undefined) ?? actual.resolveEnvironmentExecutionTarget(input) };
|
||||
});
|
||||
|
||||
describe("automatic retained workspace recovery", () => {
|
||||
useWorkspaceRecoveryFixture(targets);
|
||||
|
||||
it.each(["no_certificate", "provider_started", "runtime_selected", "live_controller", "operator_stop", "missing_stop_receipt", "wrong_snapshot", "retry_exhausted", "human_reblock", "paused", "reassigned", "approval"] as const)(
|
||||
"keeps lost preparation behind its %s gate after file preservation", async gate => {
|
||||
const f = await interruptedRemotePreparation("process_lost");
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const source = await run(f.runId);
|
||||
if (gate === "no_certificate") await db.update(heartbeatRuns).set({ resultJson: { ...source.resultJson,
|
||||
executionRecovery: null } }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "provider_started") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId,
|
||||
runId: f.runId, agentId: source.agentId, seq: 1, eventType: "native.process_start_requested", stream: "system" });
|
||||
if (gate === "runtime_selected") await db.update(heartbeatRuns).set({ runtimeModeResolvedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "live_controller") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "operator_stop") await db.update(heartbeatRuns).set({ resultJson: { ...source.resultJson,
|
||||
cancellation: requestedRunCancellation({ cancelledByActorType: "user", cancelledByUserId: "operator" }, "Stop requested"),
|
||||
} }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "missing_stop_receipt") {
|
||||
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
await db.update(environmentLeases).set({ metadata: { ...lease.metadata, remoteExecutionTermination: null } }).where(eq(environmentLeases.id, f.lease.id));
|
||||
}
|
||||
if (gate === "wrong_snapshot") await db.update(heartbeatRuns).set({ runnerProfileJson: { ...source.runnerProfileJson,
|
||||
remoteWorkspace: { ...(source.runnerProfileJson!.remoteWorkspace as Record<string, unknown>), snapshotId: randomUUID() },
|
||||
} }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "retry_exhausted") await db.update(heartbeatRuns).set({ scheduledRetryAttempt: 2 }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "human_reblock") await db.update(issues).set({ statusVersion: 3 }).where(eq(issues.id, f.issueId));
|
||||
if (gate === "paused") await db.update(agents).set({ status: "paused" }).where(eq(agents.id, source.agentId));
|
||||
if (gate === "reassigned") await db.update(issues).set({ assigneeAgentId: null, assigneeUserId: "board" }).where(eq(issues.id, f.issueId));
|
||||
if (gate === "approval") {
|
||||
const approvalId = randomUUID();
|
||||
await db.insert(approvals).values({ id: approvalId, companyId: f.companyId, type: "hire_agent", status: "pending", payload: {} });
|
||||
await db.insert(issueApprovals).values({ companyId: f.companyId, issueId: f.issueId, approvalId });
|
||||
}
|
||||
const heartbeat = heartbeatService(db);
|
||||
await heartbeat.resumeRemoteStopComments(await run(f.runId));
|
||||
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
eq(heartbeatRuns.retryOfRunId, f.runId)))).toHaveLength(0);
|
||||
expect((await run(f.runId)).error).toBe(source.error);
|
||||
}, 30_000);
|
||||
|
||||
it.each(["provider_evidence", "exhausted"] as const)("does not replace a withheld retry with %s", async gate => {
|
||||
const f = await interruptedRemotePreparation("process_lost"), withheld = await withheldPreparationRetry(f);
|
||||
if (gate === "provider_evidence") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId,
|
||||
runId: withheld.id, agentId: withheld.agentId, seq: 1, eventType: "tool.execution.started", stream: "system" });
|
||||
if (gate === "exhausted") await db.update(heartbeatRuns).set({ scheduledRetryAttempt: 2 }).where(eq(heartbeatRuns.id, withheld.id));
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const heartbeat = heartbeatService(db);
|
||||
await Promise.all([heartbeat.resumeRemoteStopComments(await run(f.runId)),
|
||||
heartbeat.resumeRemoteStopComments(await run(f.runId))]);
|
||||
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
eq(heartbeatRuns.retryOfRunId, withheld.id)))).toHaveLength(0);
|
||||
expect(await run(withheld.id)).toMatchObject({ status: "cancelled", startedAt: null });
|
||||
expect(await readFile(path.join(f.remote, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
}, 30_000);
|
||||
|
||||
it("retries unpublished preparation from its preserved ancestor without an old failure-block receipt", async () => {
|
||||
const f = await seed("legacy", false);
|
||||
await requestWorkspaceRecovery({ ...f, idempotencyKey: randomUUID(), actorId: "board" });
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const ancestor = await run(f.runId), leaseId = randomUUID(), providerLeaseId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: ancestor.agentId, status: "running" });
|
||||
const app = boardRecoveryApp(), heartbeat = heartbeatService(db);
|
||||
const first = (await request(app).post(`/api/issues/${f.issueId}/retry`).send({
|
||||
observedRunId: f.runId, idempotencyKey: randomUUID(),
|
||||
}).expect(202)).body;
|
||||
await heartbeat.resumeTaskRetry(f.companyId, first.requestId);
|
||||
const admitted = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
eq(heartbeatRuns.status, "queued")));
|
||||
expect(admitted).toHaveLength(1);
|
||||
const interruptedId = admitted[0].id;
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "legacy", status: "interrupted", executionStage: "preparing",
|
||||
startedAt: new Date(Date.now() - 60_000), finishedAt: new Date(), controllerBootId: randomUUID(),
|
||||
errorCode: "lease_released_before_terminal", error: "Lease released before preparation settled",
|
||||
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
||||
resultJson: { cancellation: { source: "unknown", expected: false, initiator: { type: "system" },
|
||||
reason: "Lease released before preparation settled", recordedAt: new Date().toISOString() } },
|
||||
}).where(eq(heartbeatRuns.id, interruptedId));
|
||||
await db.update(issues).set({ executionRunId: null, checkoutRunId: null }).where(eq(issues.id, f.issueId));
|
||||
const lease = { id: leaseId, companyId: f.companyId, heartbeatRunId: interruptedId,
|
||||
provider: "daytona", providerLeaseId };
|
||||
await db.insert(environmentLeases).values({ ...lease, issueId: f.issueId, environmentId: f.lease.environmentId,
|
||||
status: "released", cleanupStatus: "success", releasedAt: new Date(),
|
||||
metadata: { remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId, state: "stopped" }) },
|
||||
});
|
||||
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
|
||||
kind: "active_run_watchdog", ownerType: "board", returnOwnerAgentId: ancestor.agentId,
|
||||
cause: "legacy_execution_requires_reconciliation", fingerprint: `legacy-execution:${interruptedId}`,
|
||||
status: "resolved", outcome: "blocked", nextAction: "Recorded actions require reconciliation",
|
||||
evidence: { runId: interruptedId, originalFailureCode: "lease_released_before_terminal",
|
||||
automaticRecovery: { policy: "preserve_without_replay_v1", runId: interruptedId, replay: "blocked", actionOutcome: "unknown" } },
|
||||
});
|
||||
const intent = { observedRunId: interruptedId, idempotencyKey: randomUUID() };
|
||||
const receipt = (await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body;
|
||||
await Promise.all([heartbeat.resumeTaskRetry(f.companyId, receipt.requestId),
|
||||
heartbeat.resumeTaskRetry(f.companyId, receipt.requestId)]);
|
||||
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
eq(heartbeatRuns.status, "queued")));
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0].contextSnapshot).toMatchObject({ previousRunId: interruptedId });
|
||||
expect((await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body.requestId).toBe(receipt.requestId);
|
||||
const restored = path.join(root, randomUUID()); await mkdir(restored);
|
||||
const copies = await attachPreservedWorkspaceSnapshots({ db, companyId: f.companyId, issueId: f.issueId,
|
||||
sourceRunIds: [interruptedId], cwd: restored });
|
||||
expect(copies).toHaveLength(1);
|
||||
expect(await readFile(path.join(copies[0]!.path, "files", "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(await readFile(path.join(f.host, "newer.txt"), "utf8")).toBe("newer local work\n");
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
||||
}, 30_000);
|
||||
|
||||
it.each(["local", "none"])("continues verified interrupted preparation with %s leases", async provider => {
|
||||
const f = await interruptedRemotePreparation();
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
||||
resultJson: null }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (provider === "none") await db.delete(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
else await db.update(environmentLeases).set({ provider: "local", metadata: {} }).where(eq(environmentLeases.id, f.lease.id));
|
||||
const heartbeat = heartbeatService(db);
|
||||
await heartbeat.resumeRemoteStopComments(await run(f.runId));
|
||||
const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.runId));
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryAttempt: 1 });
|
||||
}, 30_000);
|
||||
|
||||
it.each(["provider_started", "provider_dispatch_recorded", "runtime_selected", "live_controller", "wrong_snapshot", "operator_stop",
|
||||
"operator_stop_metadata", "missing_stop_receipt", "unrecorded_interruption",
|
||||
"retry_exhausted", "human_reblock", "paused", "reassigned"] as const)(
|
||||
"keeps the interrupted preparation continuation behind %s", async gate => {
|
||||
const f = await interruptedRemotePreparation("lease");
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const source = await run(f.runId);
|
||||
if (gate === "provider_started") await db.insert(heartbeatRunEvents).values({ companyId: f.companyId,
|
||||
runId: f.runId, agentId: source.agentId, seq: 1, eventType: "native.process_start_requested", stream: "system" });
|
||||
if (gate === "provider_dispatch_recorded") await db.update(heartbeatRuns).set({ resultJson: { ...source.resultJson,
|
||||
executionRecovery: { kind: "bootstrap", providerWorkStarted: true }, output: randomBytes(128 * 1024).toString("base64"),
|
||||
} }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "runtime_selected") await db.update(heartbeatRuns).set({ runtimeModeResolvedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "live_controller") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "wrong_snapshot") await db.update(heartbeatRuns).set({ runnerProfileJson: { ...source.runnerProfileJson,
|
||||
remoteWorkspace: { ...(source.runnerProfileJson!.remoteWorkspace as Record<string, unknown>), snapshotId: randomUUID() },
|
||||
} }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "operator_stop") await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "operator_interrupted" }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "operator_stop_metadata") await db.update(heartbeatRuns).set({ resultJson: { ...source.resultJson,
|
||||
cancellation: requestedRunCancellation({ cancelledByActorType: "user", cancelledByUserId: "operator" }, "Stop requested"),
|
||||
} }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "unrecorded_interruption") await db.update(heartbeatRuns).set({ resultJson: { ...source.resultJson,
|
||||
cancellation: null,
|
||||
} }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "missing_stop_receipt") {
|
||||
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
await db.update(environmentLeases).set({ metadata: { ...lease.metadata, remoteExecutionTermination: null } })
|
||||
.where(eq(environmentLeases.id, f.lease.id));
|
||||
}
|
||||
if (gate === "retry_exhausted") await db.update(heartbeatRuns).set({ scheduledRetryAttempt: 2 }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "human_reblock") await db.update(issues).set({ statusVersion: 3 }).where(eq(issues.id, f.issueId));
|
||||
if (gate === "paused") await db.update(agents).set({ status: "paused" }).where(eq(agents.id, source.agentId));
|
||||
if (gate === "reassigned") await db.update(issues).set({ assigneeAgentId: null, assigneeUserId: "board" }).where(eq(issues.id, f.issueId));
|
||||
const heartbeat = heartbeatService(db);
|
||||
await heartbeat.resumeRemoteStopComments(await run(f.runId));
|
||||
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
eq(heartbeatRuns.retryOfRunId, f.runId)))).toHaveLength(0);
|
||||
if (["provider_started", "provider_dispatch_recorded", "runtime_selected", "live_controller", "wrong_snapshot", "operator_stop",
|
||||
"operator_stop_metadata", "missing_stop_receipt", "unrecorded_interruption"].includes(gate)) {
|
||||
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId,
|
||||
actorId: "local-board", actorSource: "local_implicit", observedRunId: f.runId, idempotencyKey: randomUUID() });
|
||||
await heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!);
|
||||
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
eq(heartbeatRuns.status, "queued")))).toHaveLength(0);
|
||||
}
|
||||
if (gate === "human_reblock" || gate === "reassigned") {
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("blocked");
|
||||
}
|
||||
expect((await run(f.runId)).error).toBe(source.error);
|
||||
}, 30_000);
|
||||
|
||||
it("uses a pending Retry as the sole successor after interrupted preparation recovers", async () => {
|
||||
const f = await interruptedRemotePreparation();
|
||||
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId,
|
||||
actorId: "local-board", actorSource: "local_implicit", observedRunId: f.runId, idempotencyKey: randomUUID() });
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const heartbeat = heartbeatService(db);
|
||||
await Promise.all([heartbeat.resumeRemoteStopComments(await run(f.runId)),
|
||||
heartbeat.resumeTaskRetry(f.companyId, receipt.requestId!)]);
|
||||
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
inArray(heartbeatRuns.status, ["queued", "scheduled_retry"])));
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0].status).toBe("queued");
|
||||
expect(await readFreshWorkspaceReadiness(db, successors[0])).toMatchObject({ mode: "resume_preserved_workspace" });
|
||||
}, 30_000);
|
||||
|
||||
it("cancels the interrupted preparation retry when a user blocks the task before it starts", async () => {
|
||||
const f = await interruptedRemotePreparation();
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const heartbeat = heartbeatService(db);
|
||||
await heartbeat.resumeRemoteStopComments(await run(f.runId));
|
||||
const [successor] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, f.runId));
|
||||
expect(successor.status).toBe("scheduled_retry");
|
||||
await db.update(issues).set({ status: "blocked", statusVersion: 4 }).where(eq(issues.id, f.issueId));
|
||||
await heartbeat.promoteDueScheduledRetries(new Date(Date.now() + 300_000));
|
||||
expect(await run(successor.id)).toMatchObject({ status: "cancelled", errorCode: "issue_blocked" });
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("blocked");
|
||||
}, 30_000);
|
||||
|
||||
it.each([false, true])("keeps Retry behind an accepted remote export and resumes only its saved result (review pending: %s)", async reviewPending => {
|
||||
const f = await acceptedRemoteExport();
|
||||
const { source, workspaceId, resultId, git, commit } = f;
|
||||
const interactionId = randomUUID();
|
||||
if (reviewPending) await db.insert(issueThreadInteractions).values({ id: interactionId, companyId: f.companyId,
|
||||
issueId: f.issueId, sourceRunId: f.runId, createdByAgentId: source.agentId, kind: "request_confirmation",
|
||||
title: "Review saved work", effectiveResolverPolicy: "human_only",
|
||||
payload: { version: 1, prompt: "Review before publishing", rejectRequiresReason: true,
|
||||
target: { type: "custom", key: "layout_review", label: "Review saved work" } } });
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId, status: "running" });
|
||||
const app = boardRecoveryApp();
|
||||
const intent = { observedRunId: f.runId, idempotencyKey: randomUUID() };
|
||||
const retry = (await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body;
|
||||
expect((await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body.requestId).toBe(retry.requestId);
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, retry.requestId!);
|
||||
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text,
|
||||
${heartbeatRuns.contextSnapshot}->>'issueId') = ${f.issueId}`))).toHaveLength(1);
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toMatchObject({ runId: f.runId, workspaceRepairRequired: true });
|
||||
await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => {
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, retry.requestId!);
|
||||
}) ?? []);
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
|
||||
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
|
||||
const saved = readRemoteWorkspaceAuthority(workspace.metadata)!;
|
||||
const restored = await createSavedWorkspaceLoader(db)({ companyId: f.companyId, workspaceId, snapshotId: saved.snapshotId! });
|
||||
try {
|
||||
expect(await readFile(path.join(restored.root, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(await git(["rev-parse", "HEAD"], restored.root)).toBe(commit);
|
||||
}
|
||||
finally { await restored.cleanup(); }
|
||||
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
expect(coordinator).toMatchObject({ phase: "workspace_finalizing", resultId });
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retry.requestId!)))[0])
|
||||
.toMatchObject({ status: "coalesced", runId: f.runId, payload: { taskRetry: { strategy: "finalize_result" } } });
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(2);
|
||||
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0])
|
||||
.toMatchObject({ status: reviewPending ? "in_review" : "done", executionRunId: null, checkoutRunId: null });
|
||||
expect((await run(f.runId)).status).toBe("succeeded");
|
||||
if (reviewPending) expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)))[0])
|
||||
.toMatchObject({ status: "pending", result: null, resolvedAt: null });
|
||||
}, 30_000);
|
||||
|
||||
it("preserves an explicitly requested superseded remote export without applying its result over newer work", async () => {
|
||||
const f = await acceptedRemoteExport();
|
||||
const [later] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.source.agentId,
|
||||
issueId: f.issueId, status: "cancelled", runtimeMode: "legacy", finishedAt: new Date(),
|
||||
contextSnapshot: { issueId: f.issueId }, createdAt: new Date(f.source.createdAt.getTime() + 1_000) }).returning();
|
||||
const [currentWorkspace] = await db.insert(executionWorkspaces).values({ companyId: f.companyId,
|
||||
sourceIssueId: f.issueId, name: "Newer workspace", cwd: path.join(root, later.id), providerType: "cloud_sandbox",
|
||||
mode: "isolated_workspace", strategyType: "project_primary", status: "active" }).returning();
|
||||
await db.update(issues).set({ executionWorkspaceId: currentWorkspace.id }).where(eq(issues.id, f.issueId));
|
||||
const laterAction = await issueRecoveryActionService(db).upsertSourceScoped({ companyId: f.companyId,
|
||||
sourceIssueId: f.issueId, kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation",
|
||||
fingerprint: later.id, evidence: { runId: later.id }, nextAction: "Inspect the stopped work",
|
||||
ownerType: "board", supersedeOnIdentityChange: true });
|
||||
expect((await action(f.actionId)).status).toBe("cancelled");
|
||||
await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => {}) ?? []);
|
||||
expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
|
||||
const app = boardRecoveryApp();
|
||||
const recoveryRequest = { actionId: f.actionId, runId: f.runId, idempotencyKey: randomUUID() };
|
||||
await request(app).post(`/api/issues/${f.issueId}/recovery-actions/recover-workspace`).send(recoveryRequest).expect(202);
|
||||
const admitted = await action(f.actionId);
|
||||
await request(app).post(`/api/issues/${f.issueId}/recovery-actions/recover-workspace`).send(recoveryRequest).expect(202);
|
||||
expect(await action(f.actionId)).toEqual(admitted);
|
||||
await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, later.id));
|
||||
await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => {}) ?? []);
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)?.failure?.code).toBe("workspace_writer_active");
|
||||
expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.id, later.id));
|
||||
await makeRetryDue(f.actionId);
|
||||
await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => {}) ?? []);
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
|
||||
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId));
|
||||
const saved = readRemoteWorkspaceAuthority(workspace.metadata)!;
|
||||
const restored = await createSavedWorkspaceLoader(db)({ companyId: f.companyId, workspaceId: f.workspaceId, snapshotId: saved.snapshotId! });
|
||||
try {
|
||||
expect(await f.git(["rev-parse", "HEAD"], restored.root)).toBe(f.commit);
|
||||
expect(await readFile(path.join(restored.root, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
} finally { await restored.cleanup(); }
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0])
|
||||
.toMatchObject({ status: "blocked", executionWorkspaceId: currentWorkspace.id });
|
||||
expect(await action(laterAction.id)).toEqual(expect.objectContaining({ status: "active", evidence: { runId: later.id } }));
|
||||
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0])
|
||||
.toMatchObject({ phase: "terminal_failure", resultId: f.resultId });
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toEqual([]);
|
||||
}, 30_000);
|
||||
|
||||
it.each([
|
||||
{ status: "failed", runtimeMode: "native", scenario: "none" },
|
||||
{ status: "failed", runtimeMode: "native", scenario: "registration" },
|
||||
{ status: "failed", runtimeMode: "native", scenario: "registration_after_dispatch" },
|
||||
{ status: "failed", runtimeMode: "native", scenario: "registration_existing_hold" },
|
||||
{ status: "failed", runtimeMode: "native", scenario: "registration_existing_hold_after_dispatch" },
|
||||
{ status: "cancelled", runtimeMode: "native", scenario: "none" },
|
||||
{ status: "failed", runtimeMode: "legacy", scenario: "final" },
|
||||
{ status: "failed", runtimeMode: "legacy", scenario: "initial" },
|
||||
] as const)("saves the authoritative remote workspace after $runtimeMode $status without replaying a turn or certifying external effects ($scenario)", async ({ status, runtimeMode, scenario }) => {
|
||||
const f = await seed(runtimeMode, false);
|
||||
let retryRequestId: string | undefined;
|
||||
const remoteCwd = await realpath(f.remote);
|
||||
targets.get(f.lease.id)!.remoteCwd = remoteCwd;
|
||||
f.runtime.resumeRunLease.mockImplementation(async () => ({ providerLeaseId: f.providerLeaseId, metadata: { remoteCwd } }));
|
||||
const workspaceId = randomUUID();
|
||||
let authority = { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: null as string | null,
|
||||
source: { kind: "empty" }, writerRunId: f.runId, dirty: true,
|
||||
location: { environmentId: f.lease.environmentId!, leaseId: f.lease.id,
|
||||
providerLeaseId: f.providerLeaseId, remoteCwd } };
|
||||
await db.insert(executionWorkspaces).values({ id: workspaceId, companyId: f.companyId, sourceIssueId: f.issueId,
|
||||
name: "Authoritative remote files", cwd: remoteCwd, providerRef: remoteCwd, providerType: "cloud_sandbox",
|
||||
mode: "isolated_workspace", strategyType: "project_primary", status: "active", metadata: { remoteWorkspace: authority } });
|
||||
await db.update(issues).set({ executionWorkspaceId: workspaceId }).where(eq(issues.id, f.issueId));
|
||||
const [boundLease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
await db.update(environmentLeases).set({ executionWorkspaceId: workspaceId, leasePolicy: "reuse_by_execution_workspace",
|
||||
metadata: { ...boundLease.metadata, remoteCwd, agentId: (await run(f.runId)).agentId } })
|
||||
.where(eq(environmentLeases.id, f.lease.id));
|
||||
const before = await run(f.runId);
|
||||
if (scenario === "final") {
|
||||
// A reused allocation can fail during preparation, before native input or
|
||||
// its finalization coordinator exists. Keep a real saved predecessor and
|
||||
// the ordinary exact lease handoff; neither grants a new provider turn.
|
||||
const [priorRun] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: before.agentId,
|
||||
issueId: f.issueId, nativeIssueId: f.issueId, runtimeMode: "native", status: "succeeded",
|
||||
createdAt: new Date(before.createdAt.getTime() - 2_000), finishedAt: new Date(before.createdAt.getTime() - 1_000),
|
||||
contextSnapshot: { issueId: f.issueId } }).returning();
|
||||
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
runId: priorRun.id, phase: "committed" });
|
||||
const [currentLease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
const [priorLease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
heartbeatRunId: priorRun.id, environmentId: currentLease.environmentId, executionWorkspaceId: workspaceId,
|
||||
provider: currentLease.provider, providerLeaseId: f.providerLeaseId, leasePolicy: "reuse_by_environment", status: "active",
|
||||
acquiredAt: new Date(currentLease.acquiredAt.getTime() - 1_000), metadata: currentLease.metadata }).returning();
|
||||
const priorAuthority = { ...authority, writerRunId: priorRun.id,
|
||||
location: { ...authority.location, leaseId: priorLease.id } };
|
||||
const [priorWorkspace] = await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: priorAuthority } })
|
||||
.where(eq(executionWorkspaces.id, workspaceId)).returning();
|
||||
const priorSaved = await saveRemoteWorkspace({ db, workspace: priorWorkspace, runId: priorRun.id,
|
||||
lease: priorLease as EnvironmentLease, target: { ...targets.get(f.lease.id)!, leaseId: priorLease.id },
|
||||
assertOwned: async () => undefined });
|
||||
const handoff = createReusableLeaseHandoff(priorLease, currentLease);
|
||||
await db.update(environmentLeases).set({ status: "expired", cleanupStatus: "success", releasedAt: currentLease.acquiredAt,
|
||||
metadata: { ...priorLease.metadata, reusableLeaseSuccessor: handoff } }).where(eq(environmentLeases.id, priorLease.id));
|
||||
await db.update(environmentLeases).set({ metadata: { ...currentLease.metadata, reusableLeasePredecessor: handoff } })
|
||||
.where(eq(environmentLeases.id, currentLease.id));
|
||||
authority = { ...authority, epoch: 2, snapshotId: priorSaved.snapshotId };
|
||||
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: authority } })
|
||||
.where(eq(executionWorkspaces.id, workspaceId));
|
||||
expect(before.runtimeMode).toBe("legacy");
|
||||
expect(before.runnerProfileJson?.nativeExecutionInput).toBeUndefined();
|
||||
expect(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId))).toEqual([]);
|
||||
}
|
||||
if (runtimeMode === "legacy") await writeFile(path.join(f.remote, "pre-native-change.txt"), "current preparation wrote these files\n");
|
||||
await db.update(heartbeatRuns).set({ status, runnerProfileJson: { ...before.runnerProfileJson,
|
||||
...(status === "failed" && runtimeMode === "native" ? { nativeResourceDisposition: "destroy" } : {}),
|
||||
remoteWorkspace: { schema: "paperclip.remote-workspace-run.v1", workspaceId, leaseId: f.lease.id,
|
||||
providerLeaseId: f.providerLeaseId, remoteCwd, epoch: authority.epoch, state: "prepared", snapshotId: authority.snapshotId } },
|
||||
resultJson: { ...before.resultJson, ...(status === "cancelled" ? {
|
||||
nativeCancellation: { dispatchState: "acknowledged", requestedAt: new Date().toISOString() },
|
||||
} : {}) },
|
||||
}).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: { ...authority, epoch: authority.epoch + 1 } } })
|
||||
.where(eq(executionWorkspaces.id, workspaceId));
|
||||
await expect(recoverySources.inspectWorkspaceRecoverySource(db, f.input)).rejects.toMatchObject({ code: "source_binding_invalid" });
|
||||
expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
|
||||
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: authority } })
|
||||
.where(eq(executionWorkspaces.id, workspaceId));
|
||||
if (runtimeMode === "legacy") {
|
||||
// Remote file ownership exists before native runtime selection.
|
||||
expect((await recoverySources.inspectWorkspaceRecoverySource(db, f.input)).sources).toHaveLength(1);
|
||||
const failedUpload = vi.spyOn(getStorageService(), "putFile").mockRejectedValueOnce(new Error("Synthetic saved-copy outage"));
|
||||
try {
|
||||
if (scenario === "initial") {
|
||||
// Exercise the original setup boundary, not a failed final save that
|
||||
// retries the same operation. The initial ledger reservation survives.
|
||||
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
await db.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null, resultJson: null })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
const [originalLease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
const activeMetadata = { ...originalLease.metadata };
|
||||
delete activeMetadata.remoteExecutionTermination;
|
||||
const [activeLease] = await db.update(environmentLeases).set({ status: "active", releasedAt: null, cleanupStatus: null,
|
||||
metadata: activeMetadata }).where(eq(environmentLeases.id, f.lease.id)).returning();
|
||||
const [preparing] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
|
||||
await expect(saveRemoteWorkspace({ db, workspace: preparing, runId: f.runId, lease: activeLease as EnvironmentLease,
|
||||
target: targets.get(f.lease.id)!, assertOwned: async () => undefined, boundary: "initial" }))
|
||||
.rejects.toThrow("Synthetic saved-copy outage");
|
||||
const preserved = await preserveRemoteWorkspacePreparationFailure(db, activeLease, { assertOwned: async () => undefined });
|
||||
expect(preserved?.resultJson).toMatchObject({ workspaceRestoreFailure: "restore_failed",
|
||||
workspaceRestoreRecovery: { leaseIds: [f.lease.id] } });
|
||||
await terminalizeLegacyExecution({ db, run: preserved!, status: "failed", patch: {
|
||||
errorCode: "setup_failed", error: "Synthetic saved-copy outage", finishedAt: new Date(),
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
||||
} });
|
||||
const repairs = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, f.companyId),
|
||||
sql`${issueRecoveryActions.evidence}->>'runId' = ${f.runId}`));
|
||||
expect(repairs).toHaveLength(1);
|
||||
f.actionId = repairs[0].id;
|
||||
f.input.actionId = repairs[0].id;
|
||||
expect(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId))).toEqual([]);
|
||||
expect((await run(f.runId)).runnerProfileJson).not.toHaveProperty("nativeExecutionInput");
|
||||
} else await recoverRetainedWorkspace(f.input);
|
||||
expect(failedUpload).toHaveBeenCalledTimes(1);
|
||||
} finally { failedUpload.mockRestore(); }
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).not.toBe("complete");
|
||||
const [unsaved] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
|
||||
expect(readRemoteWorkspaceAuthority(unsaved.metadata)).toMatchObject({ dirty: true, writerRunId: f.runId,
|
||||
epoch: authority.epoch, snapshotId: authority.snapshotId });
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(false);
|
||||
expect(await readFile(path.join(f.remote, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
if (scenario === "final") await makeRetryDue(f.actionId);
|
||||
}
|
||||
if (scenario.startsWith("registration")) {
|
||||
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
await db.update(heartbeatRuns).set({ resultJson: { workspaceRestoreFailure: "restore_failed", originalDiagnostic: true },
|
||||
runnerProfileJson: { ...(await run(f.runId)).runnerProfileJson, nativeResourceDisposition: "keep_running" },
|
||||
}).where(eq(heartbeatRuns.id, f.runId));
|
||||
const [stopped] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
const termination = stopped.metadata!.remoteExecutionTermination as Record<string, unknown>;
|
||||
await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment", releasedAt: new Date(String(termination.confirmedAt)), metadata: { ...stopped.metadata,
|
||||
sandboxStopAndRetainReceipt: { ...termination, schema: "paperclip.sandbox-stop-and-retain-receipt.v1",
|
||||
pluginId: stopped.metadata!.pluginId, method: "environmentStopLease", requestId: randomUUID() },
|
||||
} }).where(eq(environmentLeases.id, f.lease.id));
|
||||
const [later] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: before.agentId,
|
||||
issueId: f.issueId, runtimeMode: "legacy", status: "failed", errorCode: "setup_failed", finishedAt: new Date(),
|
||||
executionStage: "preparing", runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
||||
createdAt: new Date(before.createdAt.getTime() + 1), contextSnapshot: { issueId: f.issueId },
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
||||
}).returning();
|
||||
if (scenario.endsWith("after_dispatch")) await appendHeartbeatRunEvent(db, {
|
||||
companyId: f.companyId, runId: later.id, agentId: before.agentId, eventType: "adapter.invoke", stream: "system",
|
||||
level: "info", message: "Execution started before the contradictory bootstrap result.", payload: null,
|
||||
});
|
||||
const existingHold = scenario.includes("existing_hold");
|
||||
await issueRecoveryActionService(db).upsertSourceScoped({ companyId: f.companyId, sourceIssueId: f.issueId,
|
||||
kind: existingHold ? "active_run_watchdog" : "stranded_assigned_issue",
|
||||
cause: existingHold ? "legacy_execution_requires_reconciliation" : "stranded_assigned_issue", fingerprint: `latest-setup:${later.id}`,
|
||||
ownerType: "board", nextAction: "Inspect the failed setup before continuing.",
|
||||
evidence: existingHold ? { runId: later.id, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } }
|
||||
: { latestRunId: later.id, latestRunStatus: "failed", latestRunErrorCode: "setup_failed" } });
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: before.agentId, status: "running" });
|
||||
const app = boardRecoveryApp(), retryIntent = { observedRunId: later.id, idempotencyKey: randomUUID() };
|
||||
const retry = (await request(app).post(`/api/issues/${f.issueId}/retry`).send(retryIntent).expect(202)).body;
|
||||
retryRequestId = retry.requestId;
|
||||
expect((await request(app).post(`/api/issues/${f.issueId}/retry`).send(retryIntent).expect(202)).body.requestId).toBe(retryRequestId);
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, retryRequestId!);
|
||||
const [registered] = await db.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, f.companyId),
|
||||
sql`${issueRecoveryActions.evidence}->>'runId' = ${f.runId}`));
|
||||
expect(registered).toBeDefined();
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(3);
|
||||
f.actionId = f.input.actionId = registered.id;
|
||||
await Promise.all(await reconcileRetainedWorkspaces(db, f.environmentRuntime, async () => {}) ?? []);
|
||||
}
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const recoveredAction = await action(f.actionId);
|
||||
expect(readSystemRecovery(recoveredAction.evidence)?.state).toBe("complete");
|
||||
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspaceId));
|
||||
const saved = readRemoteWorkspaceAuthority(workspace.metadata);
|
||||
expect(saved).toMatchObject({ dirty: false, writerRunId: null });
|
||||
expect(saved?.snapshotId).toBeTruthy();
|
||||
const snapshots = await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.executionWorkspaceId, workspaceId));
|
||||
expect(snapshots).toHaveLength(runtimeMode === "legacy" ? 2 : 1);
|
||||
expect(snapshots.find(snapshot => snapshot.id === saved!.snapshotId))
|
||||
.toMatchObject({ status: "committed", sourceRunId: f.runId, generation: runtimeMode === "legacy" ? 2 : 1 });
|
||||
const extracted = await createSavedWorkspaceLoader(db)({ companyId: f.companyId, workspaceId, snapshotId: saved!.snapshotId! });
|
||||
try {
|
||||
expect(await readFile(path.join(extracted.root, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
if (runtimeMode === "legacy") expect(await readFile(path.join(extracted.root, "pre-native-change.txt"), "utf8"))
|
||||
.toBe("current preparation wrote these files\n");
|
||||
}
|
||||
finally { await extracted.cleanup(); }
|
||||
expect(await readFile(path.join(f.host, "newer.txt"), "utf8")).toBe("newer local work\n");
|
||||
expect(await run(f.runId)).toMatchObject({ status, errorCode: scenario === "initial" ? "setup_failed" : "execution_interrupted",
|
||||
runnerProfileJson: { remoteWorkspace: { state: "saved", snapshotId: saved!.snapshotId } } });
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId)))
|
||||
.toHaveLength(retryRequestId ? 1 : 0);
|
||||
expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId))).toHaveLength(0);
|
||||
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.companyId))).toHaveLength(0);
|
||||
expect(await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.heartbeatRunId, f.runId),
|
||||
eq(workspaceOperations.phase, "workspace_recovery_snapshot")))).toHaveLength(0);
|
||||
expect(f.runtime.resumeRunLease).toHaveBeenCalledTimes(scenario === "final" ? 2 : 1);
|
||||
expect(f.runtime.retryPendingSandboxTeardown).toHaveBeenCalledTimes(runtimeMode === "legacy" ? 2 : 1);
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
|
||||
await scheduleRecoveredWorkspaceCleanup(db, await run(f.runId));
|
||||
const [settledLease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
if (retryRequestId) {
|
||||
expect(settledLease).toMatchObject({ status: "released", leasePolicy: "reuse_by_environment" });
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, retryRequestId);
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, retryRequestId);
|
||||
const queued = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
||||
if (scenario.endsWith("after_dispatch")) {
|
||||
expect(queued).toHaveLength(0);
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, retryRequestId)))[0])
|
||||
.toMatchObject({ status: "deferred_issue_execution", runId: null });
|
||||
return;
|
||||
}
|
||||
expect(queued).toHaveLength(1);
|
||||
expect(await readFreshWorkspaceReadiness(db, queued[0]))
|
||||
.toMatchObject({ mode: "resume_preserved_workspace", preservationReceiptId: expect.any(String) });
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ executionWorkspaceId: workspaceId });
|
||||
expect((await run(f.runId)).resultJson).toMatchObject({ originalDiagnostic: true });
|
||||
expect(f.runtime.resumeRunLease).toHaveBeenCalledTimes(1);
|
||||
return;
|
||||
}
|
||||
if (runtimeMode === "legacy") {
|
||||
expect(settledLease).toMatchObject({ status: "released",
|
||||
leasePolicy: scenario === "initial" ? "retain_on_failure" : "reuse_by_execution_workspace" });
|
||||
expect((await run(f.runId)).runtimeMode).toBe("legacy");
|
||||
expect(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId))).toEqual([]);
|
||||
if (scenario === "final") expect(snapshots.find(snapshot => snapshot.id === authority.snapshotId)).toMatchObject({ status: "superseded" });
|
||||
else expect(snapshots.find(snapshot => snapshot.generation === 1)).toMatchObject({ status: "pending" });
|
||||
// Explicit Retry may admit the preserved files. Saturate agent capacity
|
||||
// so this tests admission without dispatching a new agent turn.
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: before.agentId, status: "running" });
|
||||
const retry = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "local-board",
|
||||
actorSource: "local_implicit", observedRunId: f.runId, idempotencyKey: randomUUID() });
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, retry.requestId!);
|
||||
const [successor] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
||||
expect(successor).toBeDefined();
|
||||
expect(await readFreshWorkspaceReadiness(db, successor))
|
||||
.toMatchObject({ mode: "resume_preserved_workspace", preservationReceiptId: expect.any(String) });
|
||||
expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, successor.id))).toEqual([]);
|
||||
expect(f.runtime.resumeRunLease).toHaveBeenCalledTimes(scenario === "final" ? 2 : 1);
|
||||
return;
|
||||
}
|
||||
if (status === "failed") {
|
||||
expect(settledLease).toMatchObject({ status: "pending_cleanup", leasePolicy: "ephemeral",
|
||||
metadata: { workspaceRecoveryCleanup: { runId: f.runId, snapshotId: saved!.snapshotId, epoch: authority.epoch } } });
|
||||
const source = await run(f.runId);
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId, status: "running" });
|
||||
const retry = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "local-board",
|
||||
actorSource: "local_implicit", observedRunId: source.id, idempotencyKey: randomUUID() });
|
||||
// The saved, re-stopped sandbox only awaits destroy. That cleanup does not
|
||||
// hold the successor: it continues now instead of on the cleanup retry.
|
||||
expect(settledLease.metadata?.workspaceRecoveryCleanup).toMatchObject({ stopReceipt: { runId: f.runId, leaseId: settledLease.id } });
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, retry.requestId!);
|
||||
const [successor] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
||||
expect(successor).toBeDefined();
|
||||
const readiness = await readFreshWorkspaceReadiness(db, successor);
|
||||
expect(readiness).toMatchObject({ mode: "resume_preserved_workspace", preservationReceiptId: expect.any(String) });
|
||||
} else {
|
||||
expect(settledLease).toMatchObject({ status: "released", leasePolicy: "reuse_by_execution_workspace" });
|
||||
expect(settledLease.metadata?.workspaceRecoveryCleanup).toBeUndefined();
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
});
|
||||
@@ -0,0 +1,199 @@
|
||||
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
||||
import { heartbeatRunEvents, authUsers, companyMemberships, agents, companies, environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, agentWakeupRequests, workspaceOperations, executionWorkspaces } from "@paperclipai/db";
|
||||
import { requestedRunCancellation } from "./run-cancellation.js";
|
||||
import { readFreshWorkspaceReadiness } from "./fresh-workspace-continuation.js";
|
||||
import { materializeIsolatedTaskDirectory } from "./isolated-task-directory.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import request from "supertest";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
||||
import { recoverRetainedWorkspace } from "./workspace-recovery.js";
|
||||
import { heartbeatService } from "./heartbeat.js";
|
||||
import { useWorkspaceRecoveryFixture, boardRecoveryApp, db, failedRemotePreparation, fixtureCompanyIds, receiptOnlyRecoveryApp, run } from "./__tests__/workspace-recovery-fixture.js";
|
||||
const targets = vi.hoisted(() => new Map<string, AdapterSandboxExecutionTarget>());
|
||||
vi.mock("./environment-execution-target.js", async importOriginal => {
|
||||
const actual = await importOriginal<typeof import("./environment-execution-target.js")>();
|
||||
return { ...actual, resolveEnvironmentExecutionTarget: async (input: Parameters<typeof actual.resolveEnvironmentExecutionTarget>[0]) =>
|
||||
(input.leaseId ? targets.get(input.leaseId) : undefined) ?? actual.resolveEnvironmentExecutionTarget(input) };
|
||||
});
|
||||
|
||||
describe("automatic retained workspace recovery", () => {
|
||||
useWorkspaceRecoveryFixture(targets);
|
||||
|
||||
it.each(["malformed_worker_stop", "foreign_run_worker_stop", "unstopped_worker", "active_workspace_owner", "delayed_provider_event", "native_input", "session_checkpoint", "session_after", "controller_recovery", "process", "coordinator",
|
||||
"runtime_selected", "live_controller", "missing_error", "missing_measurement", "foreign_error", "unknown_result",
|
||||
"wrong_snapshot", "wrong_generation", "active_writer", "missing_stop", "foreign_stop", "additional_unstopped_lease",
|
||||
"operator_stop", "newer_input", "reassigned", "wrong_actor", "actor_permission_revoked",
|
||||
"provider_usage", "unknown_usage", "invalid_ledger_scope", "foreign_ledger_scope",
|
||||
"array_usage", "false_usage", "zero_usage", "empty_usage_string", "empty_usage_object",
|
||||
"active_lease", "pending_cleanup", "cleanup_failed", "cleanup_unknown", "cleanup_inflight", "unreleased", "foreign_lease_issue", "local_provider_claim"] as const)(
|
||||
"holds a failed-preparation HTTP Retry when current proof changes: %s", async gate => {
|
||||
const f = await failedRemotePreparation();
|
||||
await db.update(heartbeatRuns).set({ sessionIdBefore: randomUUID() }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const source = await run(f.runId), heartbeat = heartbeatService(db);
|
||||
const stoppedWorkerAudit = { v: 1, iss: "paperclip-cloud", aud: "paperclip-workspace-worker-stop/v1",
|
||||
sub: "test-stack", stopped: true, iat: 100, exp: 160, runId: f.runId, ownerToken: randomUUID(),
|
||||
worker: { provider: "railway", projectId: randomUUID(), environmentId: randomUUID(), serviceId: randomUUID(), deploymentId: randomUUID() } };
|
||||
source.runnerProfileJson = { ...source.runnerProfileJson, nativeWorkspaceOwnerStopProof:
|
||||
gate === "malformed_worker_stop" ? { stopped: true } : {
|
||||
...stoppedWorkerAudit, ...(gate === "foreign_run_worker_stop" ? { runId: randomUUID() } : {}),
|
||||
...(gate === "unstopped_worker" ? { stopped: false } : {}),
|
||||
}, ...(gate === "active_workspace_owner" ? { nativeWorkspaceFinalizationOwner: { token: randomUUID() } } : {}) };
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: source.runnerProfileJson }).where(eq(heartbeatRuns.id, f.runId));
|
||||
const actorId = gate === "actor_permission_revoked" ? randomUUID() : "local-board";
|
||||
if (gate === "actor_permission_revoked") {
|
||||
await db.insert(authUsers).values({ id: actorId, name: "Retry owner", email: `${actorId}@example.invalid`, createdAt: new Date(), updatedAt: new Date() });
|
||||
await db.insert(companyMemberships).values({ companyId: f.companyId, principalType: "user", principalId: actorId, membershipRole: "owner" });
|
||||
}
|
||||
const app = receiptOnlyRecoveryApp({ userId: actorId, source: gate === "actor_permission_revoked" ? "session" : "local_implicit", companyIds: [f.companyId] });
|
||||
const receipt = (await request(app).post(`/api/issues/${f.issueId}/retry`)
|
||||
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() }).expect(202)).body;
|
||||
if (gate === "delayed_provider_event") await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: f.runId,
|
||||
agentId: source.agentId, eventType: "session.started", stream: "system", payload: { sessionId: randomUUID() },
|
||||
nativeSource: { sourceInstanceId: randomUUID(), sourceEventId: randomUUID(), sourceSeq: 1, protocolSchemaVersion: 1,
|
||||
canonicalPayload: { type: "session.started", sessionId: randomUUID() } } });
|
||||
if (["native_input", "session_checkpoint", "controller_recovery"].includes(gate)) {
|
||||
const key = gate === "native_input" ? "nativeExecutionInput" : gate === "session_checkpoint" ? "sessionCheckpoint" : "nativeControllerRecovery";
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...source.runnerProfileJson, [key]: { sessionId: randomUUID() } } })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
}
|
||||
if (gate === "process") await db.update(heartbeatRuns).set({ processPid: 1386 }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "session_after") await db.update(heartbeatRuns).set({ sessionIdAfter: randomUUID() }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "coordinator") {
|
||||
await db.update(heartbeatRuns).set({ nativeIssueId: f.issueId }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId,
|
||||
issueId: f.issueId, runId: f.runId, phase: "observed", attempt: 1 });
|
||||
}
|
||||
if (gate === "runtime_selected") await db.update(heartbeatRuns).set({ runtimeModeResolvedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "live_controller") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "missing_error" || gate === "missing_measurement") await db.delete(heartbeatRunEvents).where(and(
|
||||
eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.seq, gate === "missing_error" ? 1 : 2)));
|
||||
if (gate === "foreign_error") {
|
||||
const [foreign] = await db.insert(agents).values({ companyId: f.companyId, name: "Other owner", adapterType: "paperclip_runner" }).returning();
|
||||
await db.update(heartbeatRunEvents).set({ agentId: foreign.id })
|
||||
.where(and(eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.seq, 1)));
|
||||
}
|
||||
if (["array_usage", "false_usage", "zero_usage", "empty_usage_string", "empty_usage_object"].includes(gate)) {
|
||||
const invalid = gate === "array_usage" ? [] : gate === "false_usage" ? false : gate === "zero_usage" ? 0
|
||||
: gate === "empty_usage_string" ? "" : {};
|
||||
await db.update(heartbeatRuns).set({ usageJson: sql`${JSON.stringify(invalid)}::jsonb` }).where(eq(heartbeatRuns.id, f.runId));
|
||||
}
|
||||
if (["provider_usage", "unknown_usage", "invalid_ledger_scope", "foreign_ledger_scope"].includes(gate)) {
|
||||
const ledgerScope = { issueId: gate === "foreign_ledger_scope" ? randomUUID() : f.issueId, projectId: null, billingCode: null };
|
||||
await db.update(heartbeatRuns).set({ usageJson: gate === "provider_usage" ? { ledgerScope, inputTokens: 0 }
|
||||
: gate === "unknown_usage" ? { ledgerScope, unknown: true }
|
||||
: { ledgerScope: gate === "invalid_ledger_scope" ? { ...ledgerScope, providerWorkStarted: false } : ledgerScope } })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
}
|
||||
if (gate === "unknown_result") await db.update(heartbeatRuns).set({ resultJson: { ...source.resultJson,
|
||||
output: "Unknown provider work must not be replayed" } }).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "wrong_snapshot") await db.update(heartbeatRuns).set({ runnerProfileJson: { ...source.runnerProfileJson,
|
||||
remoteWorkspace: { ...(source.runnerProfileJson!.remoteWorkspace as Record<string, unknown>), snapshotId: randomUUID() } } })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "wrong_generation") await db.update(workspaceOperations).set({ metadata: sql`jsonb_set(${workspaceOperations.metadata}, '{generation}', '999')` })
|
||||
.where(and(eq(workspaceOperations.heartbeatRunId, f.runId), eq(workspaceOperations.phase, "workspace_recovery")));
|
||||
if (gate === "active_writer") await db.update(executionWorkspaces).set({ metadata: sql`jsonb_set(${executionWorkspaces.metadata}, '{remoteWorkspace,writerRunId}', ${JSON.stringify(randomUUID())}::jsonb)` })
|
||||
.where(eq(executionWorkspaces.id, f.workspaceId));
|
||||
if (gate === "missing_stop" || gate === "foreign_stop") {
|
||||
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
await db.update(environmentLeases).set({ metadata: { ...lease.metadata, remoteExecutionTermination: gate === "missing_stop"
|
||||
? null : { ...(lease.metadata!.remoteExecutionTermination as Record<string, unknown>), runId: randomUUID() } } })
|
||||
.where(eq(environmentLeases.id, lease.id));
|
||||
}
|
||||
if (["active_lease", "pending_cleanup", "cleanup_failed", "cleanup_unknown", "cleanup_inflight", "unreleased", "local_provider_claim"].includes(gate)) {
|
||||
await db.update(environmentLeases).set({
|
||||
...(gate === "active_lease" ? { status: "active" } : gate === "pending_cleanup" ? { status: "pending_cleanup" } : {}),
|
||||
...(gate === "cleanup_failed" ? { cleanupStatus: "failed" } : gate === "cleanup_unknown" ? { cleanupStatus: null } : {}),
|
||||
...(gate === "unreleased" ? { releasedAt: null } : {}),
|
||||
...(gate === "cleanup_inflight" ? { metadata: sql`jsonb_set(${environmentLeases.metadata}, '{pendingCleanupInFlight}', 'true')` } : {}),
|
||||
...(gate === "local_provider_claim" ? { provider: "local", providerLeaseId: null,
|
||||
metadata: sql`${environmentLeases.metadata} - 'remoteExecutionTermination'` } : {}),
|
||||
}).where(eq(environmentLeases.id, f.lease.id));
|
||||
}
|
||||
if (gate === "foreign_lease_issue") {
|
||||
const [foreign] = await db.insert(issues).values({ companyId: f.companyId, title: "Other task", status: "blocked", assigneeAgentId: source.agentId }).returning();
|
||||
await db.update(environmentLeases).set({ issueId: foreign.id }).where(eq(environmentLeases.id, f.lease.id));
|
||||
}
|
||||
if (gate === "additional_unstopped_lease") await db.insert(environmentLeases).values({ companyId: f.companyId,
|
||||
issueId: f.issueId, heartbeatRunId: f.runId, environmentId: f.lease.environmentId,
|
||||
provider: "daytona", providerLeaseId: randomUUID(), status: "active" });
|
||||
if (gate === "operator_stop") await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "operator_interrupted",
|
||||
resultJson: { ...source.resultJson, cancellation: requestedRunCancellation({ cancelledByActorType: "user", cancelledByUserId: "local-board" }, "Stop requested") } })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
if (gate === "newer_input") await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId,
|
||||
issueId: f.issueId, contextSnapshot: { issueId: f.issueId, source: "issue.comment.reopen" }, status: "running" });
|
||||
if (gate === "reassigned") await db.update(issues).set({ assigneeAgentId: null, assigneeUserId: "board" }).where(eq(issues.id, f.issueId));
|
||||
if (gate === "wrong_actor") await db.update(agentWakeupRequests).set({ requestedByActorId: "another-user" })
|
||||
.where(eq(agentWakeupRequests.id, receipt.requestId));
|
||||
if (gate === "actor_permission_revoked") await db.update(companyMemberships).set({ status: "suspended" })
|
||||
.where(and(eq(companyMemberships.companyId, f.companyId), eq(companyMemberships.principalId, actorId)));
|
||||
await Promise.all([heartbeat.resumeTaskRetry(f.companyId, receipt.requestId), heartbeat.resumeTaskRetry(f.companyId, receipt.requestId)]);
|
||||
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${receipt.requestId}`))).toHaveLength(0);
|
||||
expect((await run(f.runId)).error).toBe(source.error);
|
||||
}, 30_000);
|
||||
|
||||
it.each(["settled", "active", "provider_handle"] as const)("keeps unsupported mixed local and remote source leases held: %s", async state => {
|
||||
const f = await failedRemotePreparation();
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
await db.insert(environmentLeases).values({ companyId: f.companyId, heartbeatRunId: f.runId, issueId: f.issueId,
|
||||
provider: "local", providerLeaseId: state === "provider_handle" ? randomUUID() : null,
|
||||
status: state === "active" ? "active" : "released", releasedAt: new Date(), cleanupStatus: "success" });
|
||||
const receipt = (await request(boardRecoveryApp()).post(`/api/issues/${f.issueId}/retry`)
|
||||
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() }).expect(202)).body;
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, receipt.requestId);
|
||||
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${receipt.requestId}`))).toHaveLength(0);
|
||||
}, 30_000);
|
||||
|
||||
it.each(["plain", "unsupported_warning", "warning_presentation", "provider_warning", "measurement"] as const)(
|
||||
"keeps the existing-directory readiness settlement contract: %s", async mode => {
|
||||
const companyId = randomUUID(), issueId = randomUUID(), agentId = randomUUID(), sourceRunId = randomUUID(),
|
||||
successorRunId = randomUUID(), requestId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Preparation settlement", issuePrefix: `S${companyId.slice(0, 6)}` });
|
||||
fixtureCompanyIds.add(companyId);
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", adapterType: "paperclip_runner" });
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Continue existing directory", status: "blocked", assigneeAgentId: agentId });
|
||||
const cwd = await materializeIsolatedTaskDirectory({ companyId, issueId });
|
||||
await writeFile(path.join(cwd, "saved.txt"), "existing task work\n");
|
||||
const [workspace] = await db.insert(executionWorkspaces).values({ companyId, sourceIssueId: issueId,
|
||||
mode: "isolated_workspace", strategyType: "project_primary", providerType: "local_fs", name: "Task directory", cwd }).returning();
|
||||
await db.update(issues).set({ executionWorkspaceId: workspace.id, executionWorkspacePreference: "reuse_existing" }).where(eq(issues.id, issueId));
|
||||
const error = "The selected existing task directory has no Git checkout.";
|
||||
await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId, agentId, issueId, runtimeMode: "legacy",
|
||||
status: "failed", finishedAt: new Date(), errorCode: "workspace_validation_failed", error, executionStage: "preparing",
|
||||
contextSnapshot: { issueId }, runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
||||
usageJson: { ledgerScope: { issueId, projectId: null, billingCode: null } },
|
||||
resultJson: { stopReason: "adapter_failed", executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
|
||||
workspaceValidation: { reason: "git_worktree_base_not_git_checkout", issueId,
|
||||
resolvedWorkspaceSource: "task_session", requestedExecutionWorkspaceMode: "isolated_workspace",
|
||||
issueExecutionWorkspacePreference: "reuse_existing", issueExecutionWorkspaceId: workspace.id,
|
||||
issueProjectId: null, issueProjectWorkspaceId: null, resolvedWorkspaceCwd: cwd } } });
|
||||
if (mode.includes("warning")) await appendHeartbeatRunEvent(db, { companyId, agentId, runId: sourceRunId,
|
||||
eventType: "agent_observability.unavailable", level: "warn", stream: "system",
|
||||
message: mode === "warning_presentation" ? "Changed presentation" : "Company observability was not applied: this harness is not supported.",
|
||||
payload: { reason: "unsupported_harness", revision: randomUUID() },
|
||||
...(mode === "provider_warning" ? { nativeSource: { sourceInstanceId: randomUUID(), sourceEventId: randomUUID(), sourceSeq: 1,
|
||||
protocolSchemaVersion: 1, canonicalPayload: { type: "agent_observability.unavailable" } } } : {}) });
|
||||
await appendHeartbeatRunEvent(db, { companyId, agentId, runId: sourceRunId, eventType: "error", level: "error", stream: "system", message: error });
|
||||
await appendHeartbeatRunEvent(db, { companyId, agentId, runId: sourceRunId, eventType: "lifecycle", level: "info", stream: "system",
|
||||
message: "task.run.measured", payload: { phase: "task.run.measured", outcome: "failed", durationMs: 1, durableSettlement: mode !== "measurement" } });
|
||||
const [successor] = await db.insert(heartbeatRuns).values({ id: successorRunId, companyId, agentId, issueId, status: "queued",
|
||||
contextSnapshot: { issueId, previousRunId: sourceRunId, taskRetryRequestId: requestId } }).returning();
|
||||
await db.insert(agentWakeupRequests).values({ id: requestId, companyId, agentId, source: "on_demand", triggerDetail: "manual", reason: "task_retry",
|
||||
status: "coalesced", runId: successor.id, requestedByActorType: "user", requestedByActorId: "local-board",
|
||||
idempotencyKey: `task-retry:${randomUUID()}`, payload: { issueId, taskRetry: { schema: "paperclip.task-retry.v1",
|
||||
generationId: requestId, sourceRunId, actorId: "local-board", actorSource: "session", strategy: "evaluate" } } });
|
||||
const reuse = mode === "plain" || mode === "unsupported_warning";
|
||||
expect(await readFreshWorkspaceReadiness(db, successor)).toMatchObject(reuse
|
||||
? { mode: "reuse_current_workspace", retryIntentId: requestId, sourceRunId, workspaceId: workspace.id, cwd }
|
||||
: { mode: "fresh_isolated_workspace", retryIntentId: requestId, sourceRunId, retainedSourceIds: [] });
|
||||
expect(await readFile(path.join(cwd, "saved.txt"), "utf8")).toBe("existing task work\n");
|
||||
});
|
||||
|
||||
});
|
||||
@@ -0,0 +1,871 @@
|
||||
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
||||
import { heartbeatRunEvents, agents, completionContracts, companies, environmentLeases, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, agentWakeupRequests, workspaceOperations, executionWorkspaces, executionWorkspaceSnapshots } from "@paperclipai/db";
|
||||
import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js";
|
||||
import { selectHeartbeatEnvironment } from "./heartbeat/environment-selection.js";
|
||||
import { attachPreservedRemoteWorkspaceSnapshots } from "./remote-workspace-recovery-attachments.js";
|
||||
import { Readable } from "node:stream";
|
||||
import { renderPaperclipWakePrompt } from "@paperclipai/adapter-utils/server-utils";
|
||||
import { buildEnvironmentLeaseContext } from "./environment-runtime.js";
|
||||
import { createHeartbeatWorkspaceResolver, resolveExecutionWorkspaceReuseRequestForIssue } from "./heartbeat/workspaces.js";
|
||||
import { environmentService } from "./environments.js";
|
||||
import { environmentRunOrchestrator } from "./environment-run-orchestrator.js";
|
||||
import { executionWorkspaceService } from "./execution-workspaces.js";
|
||||
import { workspaceOperationService } from "./workspace-operations.js";
|
||||
import { instanceSettingsService } from "./instance-settings.js";
|
||||
import { readSystemRecovery, createSystemRecovery } from "./system-recovery-state.js";
|
||||
import * as storageBackend from "../storage/index.js";
|
||||
import { getStorageService } from "../storage/index.js";
|
||||
import { attachPreservedWorkspaceSnapshots, preserveWorkspaceSnapshot } from "./workspace-recovery-snapshot.js";
|
||||
import { requestTaskRetry } from "./task-retry.js";
|
||||
import { readFreshWorkspaceReadiness } from "./fresh-workspace-continuation.js";
|
||||
import { execFile } from "node:child_process";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { chmod, cp, lstat, mkdir, mkdtemp, readFile, readdir, realpath, rm, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import request from "supertest";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { CONTROL_PLANE_CONFORMANCE_RESULT, CONTROL_PLANE_CONFORMANCE_TERMINAL } from "../vendor/paperclip-runner/testing.js";
|
||||
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
||||
import type { EnvironmentLease } from "@paperclipai/shared";
|
||||
import { recoverRetainedWorkspace } from "./workspace-recovery.js";
|
||||
import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
||||
import { registerWorkspaceRecoveryWork } from "./workspace-recovery-work.js";
|
||||
import { heartbeatService } from "./heartbeat.js";
|
||||
import { hasVerifiedWorkspaceRepair } from "./workspace-recovery-receipt.js";
|
||||
import { createSavedWorkspaceLoader, prepareRemoteWorkspace, readRemoteWorkspaceAuthority, reserveRemoteWorkspace, saveRemoteWorkspace } from "./remote-workspaces.js";
|
||||
import { getExecutionBlocker } from "./execution-blocker.js";
|
||||
import { buildExecutionContinuation } from "./execution-continuation.js";
|
||||
import { createHeartbeatRunOutput } from "./heartbeat/run-output.js";
|
||||
import { objectLogFixture } from "./heartbeat/__tests__/object-log-fixture.js";
|
||||
import { useWorkspaceRecoveryFixture, action, boardRecoveryApp, connectionString, db, failedRemotePreparation, fixtureCompanyIds, receiptOnlyRecoveryApp, root, run, stoppedWorkspaceWorker } from "./__tests__/workspace-recovery-fixture.js";
|
||||
const targets = vi.hoisted(() => new Map<string, AdapterSandboxExecutionTarget>());
|
||||
vi.mock("./environment-execution-target.js", async importOriginal => {
|
||||
const actual = await importOriginal<typeof import("./environment-execution-target.js")>();
|
||||
return { ...actual, resolveEnvironmentExecutionTarget: async (input: Parameters<typeof actual.resolveEnvironmentExecutionTarget>[0]) =>
|
||||
(input.leaseId ? targets.get(input.leaseId) : undefined) ?? actual.resolveEnvironmentExecutionTarget(input) };
|
||||
});
|
||||
|
||||
describe("automatic retained workspace recovery", () => {
|
||||
useWorkspaceRecoveryFixture(targets);
|
||||
|
||||
it.each(["new message", "saved message", "Retry"] as const)(
|
||||
"continues recovered preparation with an inherited session through %s", async entry => {
|
||||
const f = await failedRemotePreparation();
|
||||
await db.update(heartbeatRuns).set({ sessionIdBefore: randomUUID() }).where(eq(heartbeatRuns.id, f.runId));
|
||||
const source = await run(f.runId), app = boardRecoveryApp(), heartbeat = heartbeatService(db);
|
||||
if (entry !== "saved message") await recoverRetainedWorkspace(f.input);
|
||||
let requestId: string | undefined, commentId: string | undefined;
|
||||
if (entry === "Retry") {
|
||||
const receipt = (await request(app).post(`/api/issues/${f.issueId}/retry`)
|
||||
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() }).expect(202)).body;
|
||||
requestId = receipt.requestId;
|
||||
await heartbeat.resumeTaskRetry(f.companyId, requestId!);
|
||||
} else {
|
||||
const body = { body: "Continue from the saved work", reopen: true, clientRequestId: randomUUID() };
|
||||
const comment = (await request(app).post(`/api/issues/${f.issueId}/comments`).send(body).expect(201)).body;
|
||||
commentId = comment.id;
|
||||
expect((await request(app).post(`/api/issues/${f.issueId}/comments`).send(body).expect(201)).body.id).toBe(commentId);
|
||||
await expect.poll(async () => (await db.select().from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, f.companyId),
|
||||
sql`${agentWakeupRequests.payload}->>'commentId' = ${commentId}`))).length).toBe(1);
|
||||
if (entry === "saved message") {
|
||||
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
eq(heartbeatRuns.status, "queued")))).toHaveLength(0);
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
}
|
||||
await heartbeat.resumeRemoteStopComments(await run(f.runId));
|
||||
}
|
||||
const successors = () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
eq(heartbeatRuns.status, "queued")));
|
||||
await expect.poll(async () => (await successors()).length).toBe(1);
|
||||
const [successor] = await successors();
|
||||
expect(successor.contextSnapshot).toMatchObject({ forceFreshSession: true, previousRunId: f.runId,
|
||||
...(commentId ? { wakeCommentIds: [commentId] } : { taskRetryRequestId: requestId }) });
|
||||
await Promise.all([heartbeat.resumeRemoteStopComments(await run(f.runId)),
|
||||
requestId ? heartbeat.resumeTaskRetry(f.companyId, requestId) : heartbeat.resumeExecutionWaitComments()]);
|
||||
expect(await successors()).toHaveLength(1);
|
||||
const queue = (await request(app).get(`/api/issues/${f.issueId}/queued-comments`).expect(200)).body;
|
||||
expect(queue.executionWait ?? null).toBeNull();
|
||||
if (commentId) expect(queue).toMatchObject({ state: "queued", entries: [{ comment: { id: commentId } }] });
|
||||
else expect(queue.entries).toEqual([]);
|
||||
expect((await request(app).get(`/api/issues/${f.issueId}`).expect(200)).body.interruption)
|
||||
.toMatchObject({ state: "working", canRetry: false });
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
||||
const original = await run(f.runId);
|
||||
expect(original).toMatchObject({ status: "failed", error: source.error, sessionIdBefore: source.sessionIdBefore });
|
||||
expect(await hasVerifiedWorkspaceRepair(db, original)).toBe(true);
|
||||
const remote = original.runnerProfileJson!.remoteWorkspace as { snapshotId: string };
|
||||
const saved = await createSavedWorkspaceLoader(db)({ companyId: f.companyId, workspaceId: f.workspaceId,
|
||||
snapshotId: remote.snapshotId });
|
||||
try { expect(await readFile(path.join(saved.root, "saved.txt"), "utf8")).toBe("finished work\n"); }
|
||||
finally { await saved.cleanup(); }
|
||||
}, 30_000);
|
||||
|
||||
it.each(["retained_history", "ordinary_followup"])("restores the newest recovered tick bytes after a later workspace selection: %s", async kind => {
|
||||
vi.stubEnv("DATABASE_URL", connectionString);
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "62".repeat(32));
|
||||
try {
|
||||
const f = await failedRemotePreparation();
|
||||
const tickLog = Array.from({ length: 12 }, (_, i) => `2026-11-06T07:${String(21 + i).padStart(2, "0")}:00Z\n`).join("");
|
||||
await writeFile(path.join(f.remote, "ticks.log"), tickLog);
|
||||
if (kind === "retained_history") {
|
||||
const readonlySource = path.join(f.remote, "source-assets");
|
||||
await mkdir(readonlySource); await writeFile(path.join(readonlySource, "asset.txt"), "Preserved source asset\n"); await chmod(readonlySource, 0o555);
|
||||
}
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const initial = await run(f.runId), oldRunId = randomUUID(), oldActionId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: oldRunId, companyId: f.companyId, issueId: f.issueId,
|
||||
agentId: initial.agentId, status: "failed", contextSnapshot: { issueId: f.issueId },
|
||||
createdAt: new Date(initial.createdAt.getTime() - 60_000), finishedAt: new Date(initial.createdAt.getTime() - 30_000) });
|
||||
await db.insert(issueRecoveryActions).values({ id: oldActionId, companyId: f.companyId, sourceIssueId: f.issueId,
|
||||
kind: "active_run_watchdog", status: "resolved", ownerType: "board", cause: "native_continuation_requires_reconciliation",
|
||||
fingerprint: oldRunId, nextAction: "Historical preserved files", evidence: { runId: oldRunId } });
|
||||
const oldRoot = path.join(root, oldRunId); await mkdir(oldRoot);
|
||||
await writeFile(path.join(oldRoot, "ticks.log"), "2026-11-06T07:21:00Z\n");
|
||||
const oldProviderId = randomUUID();
|
||||
const [oldLease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
heartbeatRunId: oldRunId, environmentId: f.lease.environmentId, provider: "daytona", providerLeaseId: oldProviderId,
|
||||
status: "released", cleanupStatus: "success", releasedAt: new Date() }).returning();
|
||||
await preserveWorkspaceSnapshot({ db, companyId: f.companyId, issueId: f.issueId, runId: oldRunId,
|
||||
actionId: oldActionId, leaseId: oldLease.id, providerLeaseId: oldProviderId,
|
||||
target: { ...targets.get(f.lease.id)!, leaseId: oldLease.id, remoteCwd: oldRoot }, assertHeld: async () => {} });
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: f.issueId,
|
||||
error: "native_remote_recovery_lease_mismatch", errorCode: "setup_failed", sessionIdBefore: randomUUID(),
|
||||
}).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
runId: f.runId, phase: "terminal_failure" });
|
||||
const [placeholder] = await db.insert(executionWorkspaces).values({ companyId: f.companyId,
|
||||
sourceIssueId: f.issueId, name: "Later empty placeholder", providerType: "cloud_sandbox",
|
||||
mode: "isolated_workspace", strategyType: "project_primary", cwd: `/workspaces/${randomUUID()}`,
|
||||
metadata: { remoteWorkspace: { schema: "paperclip.remote-workspace.v1", epoch: 1,
|
||||
snapshotId: null, source: { kind: "empty" }, writerRunId: f.runId, dirty: false, location: null },
|
||||
...(kind === "retained_history" ? { remoteWorkspaceRecoverySourceRunIds: [oldRunId] } : {}),
|
||||
remoteWorkspaceSnapshotRepositories: { snapshotId: randomUUID(), repositories: [] } },
|
||||
}).returning();
|
||||
await db.update(issues).set({ executionWorkspaceId: placeholder.id }).where(eq(issues.id, f.issueId));
|
||||
const source = await run(f.runId);
|
||||
expect(await hasVerifiedWorkspaceRepair(db, source)).toBe(true);
|
||||
const sourceReference = source.runnerProfileJson!.remoteWorkspace as { snapshotId: string };
|
||||
const savedSource = await createSavedWorkspaceLoader(db)({ companyId: f.companyId,
|
||||
workspaceId: f.workspaceId, snapshotId: sourceReference.snapshotId });
|
||||
try { expect(await readFile(path.join(savedSource.root, "ticks.log"), "utf8")).toBe(tickLog); }
|
||||
finally { await savedSource.cleanup(); }
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId, status: "running" });
|
||||
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId,
|
||||
actorId: "local-board", actorSource: "local_implicit", observedRunId: f.runId, idempotencyKey: randomUUID() });
|
||||
expect(receipt.accepted).toBe(true);
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, receipt.requestId!);
|
||||
const [successor] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${receipt.requestId}`));
|
||||
expect(successor).toBeDefined();
|
||||
const readiness = await readFreshWorkspaceReadiness(db, successor);
|
||||
expect(readiness).toMatchObject({ mode: "resume_preserved_workspace" });
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
||||
const [agent] = await db.select().from(agents).where(eq(agents.id, source.agentId));
|
||||
const selection = await selectHeartbeatEnvironment(db, {
|
||||
run: successor, agent,
|
||||
task: { issueRef: issue, issueContext: issue, issueId: f.issueId, context: successor.contextSnapshot!,
|
||||
executionProjectId: issue.projectId, workspaceProjectContext: null, boundSourceWorkspace: null,
|
||||
freshWorkspace: readiness, hasSelectedWorkspaceSource: false, taskWorkspaceIntent: null, responsibleUserId: "local-board" },
|
||||
policy: { trustPreset: { kind: "standard", preset: "standard", boundary: null, sourcePresets: {} },
|
||||
isolatedWorkspacesEnabled: true, requestedExecutionWorkspaceMode: "isolated_workspace",
|
||||
projectExecutionWorkspacePolicy: null, issueExecutionWorkspaceSettings: null },
|
||||
config: { config: {}, sourceWorkspaceConfig: {}, issueAssigneeOverrides: null },
|
||||
services: { environmentsSvc: environmentService(db), executionWorkspacesSvc: executionWorkspaceService(db),
|
||||
instanceSettings: instanceSettingsService(db), findSharedWorkspaceHolder: async () => null,
|
||||
findExecutionWorkspaceHolder: async () => null },
|
||||
});
|
||||
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, successor.id));
|
||||
const storage = { ...getStorageService(), provider: "s3" as const };
|
||||
const reserved = await reserveRemoteWorkspace({ db, storage, companyId: f.companyId, issueId: f.issueId,
|
||||
runId: successor.id, agentId: source.agentId, environmentId: f.lease.environmentId!,
|
||||
projectId: null, projectWorkspaceId: null, existingWorkspaceId: selection.existingExecutionWorkspace?.id ?? null,
|
||||
source: { kind: "empty" } });
|
||||
const allocationPath = path.join(root, successor.id); await mkdir(allocationPath);
|
||||
const allocation = await realpath(allocationPath);
|
||||
const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
heartbeatRunId: successor.id, environmentId: f.lease.environmentId, provider: "daytona",
|
||||
providerLeaseId: randomUUID(), status: "active" }).returning();
|
||||
const target = { ...targets.get(f.lease.id)!, leaseId: lease.id, remoteCwd: allocation,
|
||||
runner: { ...targets.get(f.lease.id)!.runner!, syncIn: async operations => {
|
||||
for (const operation of operations) for (const file of operation.files) {
|
||||
await mkdir(path.dirname(file.targetPath), { recursive: true });
|
||||
await cp(file.sourcePath, file.targetPath, { recursive: true, dereference: false });
|
||||
}
|
||||
return { filesTransferred: 1, bytesTransferred: 0, operations: [] };
|
||||
} } } satisfies AdapterSandboxExecutionTarget;
|
||||
const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: successor.id,
|
||||
lease: lease as EnvironmentLease, target });
|
||||
const recovered = prepared.workspace.metadata?.remoteWorkspaceRecoveredFiles as Array<{ path: string }> | undefined;
|
||||
const contents = await Promise.all(["", ...(recovered ?? []).map(item => `${item.path}/files`)]
|
||||
.map(relative => readFile(path.join(prepared.remoteRoot, relative, "ticks.log"), "utf8").catch(() => null)));
|
||||
if (kind === "retained_history") expect(contents).toContain("2026-11-06T07:21:00Z\n");
|
||||
expect(contents).toContain(tickLog);
|
||||
if (kind === "retained_history") {
|
||||
const currentSidecar = (recovered as Array<{ path: string; sourceRunId: string }>).find(item => item.sourceRunId === source.id)!;
|
||||
expect(await readFile(path.join(prepared.remoteRoot, currentSidecar.path, "files", "source-assets", "asset.txt"), "utf8"))
|
||||
.toBe("Preserved source asset\n");
|
||||
expect((await lstat(path.join(prepared.remoteRoot, currentSidecar.path, "files", "source-assets"))).mode & 0o777).toBe(0o555);
|
||||
}
|
||||
const continuation = await buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId,
|
||||
agentId: agent.id, runId: successor.id, context: successor.contextSnapshot!, summary: null, exposeLowTrustRaw: false });
|
||||
const providerContext: Record<string, unknown> = { ...successor.contextSnapshot,
|
||||
executionContinuation: continuation, paperclipWake: { issue: { id: f.issueId }, executionContinuation: continuation } };
|
||||
const workspaces = executionWorkspaceService(db);
|
||||
const current = (await workspaces.getStoredById(f.companyId, placeholder.id))!;
|
||||
const currentIssue = (await db.select().from(issues).where(eq(issues.id, f.issueId)))[0];
|
||||
const environment = (await environmentService(db).getById(f.lease.environmentId!))!;
|
||||
const orchestrator = environmentRunOrchestrator(db, { environmentRuntime: f.environmentRuntime });
|
||||
const acquire = vi.spyOn(orchestrator, "acquireForRun").mockResolvedValue({ environment,
|
||||
lease: prepared.lease, leaseContext: buildEnvironmentLeaseContext({ persistedExecutionWorkspace: current }), executionTransport: null });
|
||||
const storageRead = vi.spyOn(storageBackend, "getStorageService").mockReturnValue(storage);
|
||||
targets.set(lease.id, { ...target, remoteCwd: prepared.remoteRoot });
|
||||
const localEnvironment = await environmentService(db).ensureLocalEnvironment(f.companyId);
|
||||
try {
|
||||
const boundary = await prepareHeartbeatWorkspace(db, { run: await run(successor.id), agent,
|
||||
task: { issueRef: currentIssue, issueId: f.issueId, wasIssueAssignee: true,
|
||||
context: providerContext, responsibleUserId: "local-board", previousSessionParams: null, freshWorkspace: readiness },
|
||||
policy: { trustPreset: { kind: "standard", preset: "standard", boundary: null, sourcePresets: {} },
|
||||
isolatedWorkspacesEnabled: true, effectiveExecutionWorkspaceMode: "isolated_workspace",
|
||||
requestedExecutionWorkspaceMode: "isolated_workspace", useIsolatedTaskDirectory: true,
|
||||
nativeChatWorkspaceScope: null, projectExecutionWorkspacePolicy: null, issueExecutionWorkspaceSettings: null,
|
||||
environmentExecutionWorkspaceSettings: null, executionPolicy: { executionMode: "any" } },
|
||||
environment: { selectedEnvironmentId: environment.id, localEnvironment, selectedEnvironmentForConfig: environment,
|
||||
environmentResolution: { environmentId: environment.id, source: "agent" }, resolvedInstanceSettings: await instanceSettingsService(db).get() },
|
||||
config: { mergedConfig: {}, configSnapshot: null, secretManifest: [] },
|
||||
reuse: { ...selection, existingExecutionWorkspace: current, reusableExistingExecutionWorkspace: current,
|
||||
requestedShouldReuseExisting: true, workspaceReuseRequest: resolveExecutionWorkspaceReuseRequestForIssue({
|
||||
issueExecutionWorkspaceId: current.id, issueExecutionWorkspacePreference: "reuse_existing", existingExecutionWorkspaceStatus: current.status }),
|
||||
runOptions: {} },
|
||||
services: { envOrchestrator: orchestrator, executionWorkspacesSvc: workspaces, workspaceOperationsSvc: workspaceOperationService(db),
|
||||
...createHeartbeatWorkspaceResolver(db), appendRunEvent: vi.fn() },
|
||||
controllerLease: { assertOwned: vi.fn(async () => {}) }, nativeRunnerPreparationSpans: [],
|
||||
});
|
||||
const paths = providerContext.recoveredWorkspaceFiles as Array<{ sourceRunId: string; path: string }>;
|
||||
const newest = paths?.find(item => item.sourceRunId === source.id);
|
||||
expect(newest).toBeDefined();
|
||||
const prompt = renderPaperclipWakePrompt(providerContext.paperclipWake);
|
||||
expect(prompt).toContain(newest!.path);
|
||||
const readback = await new Promise<string>((resolve, reject) => execFile(process.execPath, ["-e",
|
||||
"const fs=require('node:fs');const p=JSON.parse(process.argv[1]);const text=fs.readFileSync(require('node:path').join(p.cwd,p.path,'files/ticks.log'),'utf8');const lines=text.trimEnd().split('\\n');process.stdout.write(JSON.stringify({count:lines.length,first:lines[0],last:lines.at(-1)}));",
|
||||
JSON.stringify({ cwd: boundary.executionWorkspace.cwd, path: newest!.path })],
|
||||
(error, stdout) => error ? reject(error) : resolve(stdout)));
|
||||
expect(JSON.parse(readback)).toEqual({ count: 12, first: "2026-11-06T07:21:00Z", last: "2026-11-06T07:32:00Z" });
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionWorkspaceId).toBe(placeholder.id);
|
||||
const editedLog = `${tickLog}User preserved annotation\n`;
|
||||
await writeFile(path.join(boundary.executionWorkspace.cwd, newest!.path, "files", "ticks.log"), editedLog);
|
||||
let [selectedWorkspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, placeholder.id));
|
||||
if (kind === "retained_history") {
|
||||
const olderPaths = selectedWorkspace.metadata!.remoteWorkspaceRecoveredFiles as Array<{ operationId: string; sourceRunId: string; path: string }>;
|
||||
const [legacy] = await db.update(executionWorkspaces).set({ metadata: { ...selectedWorkspace.metadata,
|
||||
remoteWorkspaceRecoveredFiles: olderPaths.map(({ operationId, sourceRunId, path }) => ({ operationId, sourceRunId, path })) } })
|
||||
.where(eq(executionWorkspaces.id, selectedWorkspace.id)).returning();
|
||||
const upgraded = await prepareRemoteWorkspace({ db, storage, workspace: legacy, runId: successor.id,
|
||||
lease: boundary.activeEnvironmentLease!.lease, target: boundary.executionTarget });
|
||||
expect(upgraded.workspace.metadata?.remoteWorkspaceRecoveredFiles).toEqual(expect.arrayContaining(olderPaths.map(item =>
|
||||
expect.objectContaining({ operationId: item.operationId, sourceRunId: item.sourceRunId, path: item.path }))));
|
||||
expect(await readFile(path.join(upgraded.remoteRoot, newest!.path, "files", "ticks.log"), "utf8")).toBe(editedLog);
|
||||
selectedWorkspace = upgraded.workspace;
|
||||
}
|
||||
await saveRemoteWorkspace({ db, storage, workspace: selectedWorkspace, runId: successor.id,
|
||||
lease: boundary.activeEnvironmentLease!.lease, target: boundary.executionTarget, assertOwned: async () => {} });
|
||||
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success" })
|
||||
.where(eq(environmentLeases.id, lease.id));
|
||||
const coldAllocationPath = path.join(root, randomUUID()); await mkdir(coldAllocationPath);
|
||||
const [coldLease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
heartbeatRunId: successor.id, environmentId: f.lease.environmentId, provider: "daytona",
|
||||
providerLeaseId: randomUUID(), status: "active" }).returning();
|
||||
const coldWorkspace = await reserveRemoteWorkspace({ db, storage, companyId: f.companyId, issueId: f.issueId,
|
||||
runId: successor.id, agentId: source.agentId, environmentId: f.lease.environmentId!, projectId: null,
|
||||
projectWorkspaceId: null, existingWorkspaceId: placeholder.id, source: { kind: "empty" } });
|
||||
const cold = await prepareRemoteWorkspace({ db, storage, workspace: coldWorkspace, runId: successor.id,
|
||||
lease: coldLease as EnvironmentLease, target: { ...target, leaseId: coldLease.id, remoteCwd: await realpath(coldAllocationPath) } });
|
||||
expect(await readFile(path.join(cold.remoteRoot, newest!.path, "files", "ticks.log"), "utf8")).toBe(editedLog);
|
||||
if (kind === "retained_history") {
|
||||
await saveRemoteWorkspace({ db, storage, workspace: cold.workspace, runId: successor.id,
|
||||
lease: cold.lease, target: { ...target, leaseId: coldLease.id, remoteCwd: cold.remoteRoot }, assertOwned: async () => {} });
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, successor.id));
|
||||
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success" }).where(eq(environmentLeases.id, coldLease.id));
|
||||
const [child] = await db.insert(issues).values({ companyId: f.companyId, parentId: f.issueId,
|
||||
assigneeAgentId: agent.id, title: "Shared workspace continuation", status: "in_progress", executionWorkspaceId: placeholder.id }).returning();
|
||||
const [childRun] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: agent.id, issueId: child.id,
|
||||
status: "running", contextSnapshot: { issueId: child.id } }).returning();
|
||||
const [saved] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, placeholder.id));
|
||||
const olderPaths = saved.metadata!.remoteWorkspaceRecoveredFiles as Array<{ operationId: string; sourceRunId: string; path: string }>;
|
||||
const [earlierChild] = await db.insert(issues).values({ companyId: f.companyId, parentId: f.issueId,
|
||||
title: "Earlier related recovery source", status: "done", executionWorkspaceId: placeholder.id }).returning();
|
||||
const [related] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, issueId: earlierChild.id,
|
||||
agentId: agent.id, status: "failed", contextSnapshot: { issueId: earlierChild.id } }).returning();
|
||||
const [relatedAction] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: earlierChild.id,
|
||||
kind: "active_run_watchdog", status: "resolved", ownerType: "board", cause: "native_continuation_requires_reconciliation",
|
||||
fingerprint: related.id, nextAction: "Saved related files", evidence: { runId: related.id } }).returning();
|
||||
const [relatedLease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: earlierChild.id,
|
||||
heartbeatRunId: related.id, environmentId: f.lease.environmentId, provider: "daytona", providerLeaseId: randomUUID(),
|
||||
status: "released", cleanupStatus: "success", releasedAt: new Date() }).returning();
|
||||
await preserveWorkspaceSnapshot({ db, companyId: f.companyId, issueId: earlierChild.id, runId: related.id,
|
||||
actionId: relatedAction.id, leaseId: relatedLease.id, providerLeaseId: relatedLease.providerLeaseId!,
|
||||
target: { ...targets.get(f.lease.id)!, leaseId: relatedLease.id, remoteCwd: oldRoot }, assertHeld: async () => {} });
|
||||
const [attachedRelated] = await attachPreservedWorkspaceSnapshots({ db, companyId: f.companyId, issueId: earlierChild.id,
|
||||
sourceRunIds: [related.id], cwd: cold.remoteRoot });
|
||||
olderPaths.push({ operationId: attachedRelated.operationId, sourceRunId: related.id,
|
||||
path: `.paperclip-recovered/${attachedRelated.operationId}` });
|
||||
const [migration] = await db.insert(workspaceOperations).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
executionWorkspaceId: placeholder.id, phase: "remote_workspace_migration", status: "succeeded",
|
||||
metadata: { legacyRecoverySources: [{ runId: related.id, issueId: earlierChild.id }] } }).returning();
|
||||
const marker = { schema: "paperclip.legacy-workspace-recovery-migration.v1", operationId: migration.id };
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { legacyWorkspaceRecoveryMigration:
|
||||
{ ...marker, workspaceId: placeholder.id } } }).where(eq(heartbeatRuns.id, related.id));
|
||||
await db.update(executionWorkspaces).set({ metadata: { ...saved.metadata, legacyWorkspaceRecoveryMigration: marker,
|
||||
remoteWorkspaceRecoveredFiles:
|
||||
olderPaths.map(({ operationId, sourceRunId, path }) => ({ operationId, sourceRunId, path })) } }).where(eq(executionWorkspaces.id, saved.id));
|
||||
const shared = await reserveRemoteWorkspace({ db, storage, companyId: f.companyId, issueId: child.id,
|
||||
runId: childRun.id, agentId: agent.id, environmentId: f.lease.environmentId!, projectId: null,
|
||||
projectWorkspaceId: null, existingWorkspaceId: placeholder.id, source: { kind: "empty" } });
|
||||
const [childLease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: child.id,
|
||||
heartbeatRunId: childRun.id, environmentId: f.lease.environmentId, provider: "daytona", providerLeaseId: coldLease.providerLeaseId, status: "active" }).returning();
|
||||
const inherited = await prepareRemoteWorkspace({ db, storage, workspace: shared, runId: childRun.id,
|
||||
lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: cold.remoteRoot } });
|
||||
expect(inherited.workspace.metadata?.remoteWorkspaceRecoveredFiles).toEqual(expect.arrayContaining(olderPaths.map(item =>
|
||||
expect.objectContaining({ operationId: item.operationId, sourceRunId: item.sourceRunId, path: item.path }))));
|
||||
expect(await readFile(path.join(inherited.remoteRoot, newest!.path, "files", "ticks.log"), "utf8")).toBe(editedLog);
|
||||
const [withoutProof] = await db.update(executionWorkspaces).set({ metadata: { ...inherited.workspace.metadata,
|
||||
remoteWorkspaceRecoveredFiles: olderPaths.map(({ operationId, sourceRunId, path }) => ({ operationId, sourceRunId, path })) } })
|
||||
.where(eq(executionWorkspaces.id, inherited.workspace.id)).returning();
|
||||
for (const fault of ["migration_status", "source_marker_workspace", "history_issue"]) {
|
||||
await db.update(workspaceOperations).set({ status: fault === "migration_status" ? "failed" : "succeeded",
|
||||
metadata: { legacyRecoverySources: [{ runId: related.id, issueId: fault === "history_issue" ? randomUUID() : earlierChild.id }] } })
|
||||
.where(eq(workspaceOperations.id, migration.id));
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { legacyWorkspaceRecoveryMigration:
|
||||
{ ...marker, workspaceId: fault === "source_marker_workspace" ? randomUUID() : placeholder.id } } })
|
||||
.where(eq(heartbeatRuns.id, related.id));
|
||||
await expect(prepareRemoteWorkspace({ db, storage, workspace: withoutProof, runId: childRun.id,
|
||||
lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: cold.remoteRoot } }))
|
||||
.rejects.toThrow("remote_workspace_recovery_receipt_invalid");
|
||||
}
|
||||
await db.update(workspaceOperations).set({ status: "succeeded", metadata: {
|
||||
legacyRecoverySources: [{ runId: related.id, issueId: earlierChild.id }] } }).where(eq(workspaceOperations.id, migration.id));
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { legacyWorkspaceRecoveryMigration:
|
||||
{ ...marker, workspaceId: placeholder.id } } }).where(eq(heartbeatRuns.id, related.id));
|
||||
await db.update(workspaceOperations).set({ status: "failed" }).where(eq(workspaceOperations.id,
|
||||
(olderPaths.find(item => item.sourceRunId === source.id))!.operationId));
|
||||
await expect(prepareRemoteWorkspace({ db, storage, workspace: withoutProof, runId: childRun.id,
|
||||
lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: cold.remoteRoot } }))
|
||||
.rejects.toThrow("remote_workspace_recovery_receipt_invalid");
|
||||
expect(await readFile(path.join(inherited.remoteRoot, newest!.path, "files", "ticks.log"), "utf8")).toBe(editedLog);
|
||||
await db.update(workspaceOperations).set({ status: "succeeded" }).where(eq(workspaceOperations.id,
|
||||
(olderPaths.find(item => item.sourceRunId === source.id))!.operationId));
|
||||
const immutableAssets = path.join(inherited.remoteRoot, "readonly-assets");
|
||||
await mkdir(immutableAssets); await writeFile(path.join(immutableAssets, "asset.txt"), "Read-only workspace asset\n");
|
||||
await chmod(immutableAssets, 0o555);
|
||||
await saveRemoteWorkspace({ db, storage, workspace: inherited.workspace, runId: childRun.id,
|
||||
lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: inherited.remoteRoot },
|
||||
assertOwned: async () => {} });
|
||||
const [selectedSaved] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, inherited.workspace.id));
|
||||
const [snapshotOnly] = await db.update(executionWorkspaces).set({ metadata: {
|
||||
...Object.fromEntries(Object.entries(selectedSaved.metadata!).filter(([key]) => key !== "legacyWorkspaceRecoveryMigration")),
|
||||
remoteWorkspaceRecoveredFiles: olderPaths.map(({ operationId, sourceRunId, path }) => ({ operationId, sourceRunId, path })) } })
|
||||
.where(eq(executionWorkspaces.id, inherited.workspace.id)).returning();
|
||||
const snapshotReserved = await reserveRemoteWorkspace({ db, storage, companyId: f.companyId, issueId: child.id,
|
||||
runId: childRun.id, agentId: agent.id, environmentId: f.lease.environmentId!, projectId: null,
|
||||
projectWorkspaceId: null, existingWorkspaceId: snapshotOnly.id, source: { kind: "empty" } });
|
||||
const snapshotRetained = await prepareRemoteWorkspace({ db, storage, workspace: snapshotReserved, runId: childRun.id,
|
||||
lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: inherited.remoteRoot } });
|
||||
expect(snapshotRetained.workspace.metadata?.remoteWorkspaceRecoveredFiles).toHaveLength(olderPaths.length);
|
||||
expect((await lstat(immutableAssets)).mode & 0o777).toBe(0o555);
|
||||
expect(await readFile(path.join(immutableAssets, "asset.txt"), "utf8")).toBe("Read-only workspace asset\n");
|
||||
expect(await readFile(path.join(snapshotRetained.remoteRoot, newest!.path, "files", "ticks.log"), "utf8")).toBe(editedLog);
|
||||
const legacySnapshot = { ...snapshotRetained.workspace, metadata: { ...snapshotRetained.workspace.metadata,
|
||||
remoteWorkspaceRecoveredFiles: olderPaths.map(({ operationId, sourceRunId, path }) => ({ operationId, sourceRunId, path })) } };
|
||||
const authority = readRemoteWorkspaceAuthority(legacySnapshot.metadata)!;
|
||||
await db.update(executionWorkspaceSnapshots).set({ status: "uploaded" }).where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!));
|
||||
await expect(prepareRemoteWorkspace({ db, storage, workspace: legacySnapshot, runId: childRun.id,
|
||||
lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: inherited.remoteRoot } }))
|
||||
.rejects.toThrow("remote_workspace_recovery_receipt_invalid");
|
||||
await db.update(executionWorkspaceSnapshots).set({ status: "committed" }).where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!));
|
||||
await expect(prepareRemoteWorkspace({ db, storage, workspace: { ...legacySnapshot, metadata: { ...legacySnapshot.metadata,
|
||||
remoteWorkspace: { ...authority, snapshotId: sourceReference.snapshotId } } }, runId: childRun.id,
|
||||
lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: inherited.remoteRoot } }))
|
||||
.rejects.toThrow("remote_workspace_recovery_receipt_invalid");
|
||||
const unreadable = { ...storage, getObject: vi.fn().mockRejectedValue(new Error("fixture: selected snapshot unavailable")) };
|
||||
await expect(prepareRemoteWorkspace({ db, storage: unreadable, workspace: legacySnapshot, runId: childRun.id,
|
||||
lease: childLease as EnvironmentLease, target: { ...target, leaseId: childLease.id, remoteCwd: inherited.remoteRoot } }))
|
||||
.rejects.toThrow("fixture: selected snapshot unavailable");
|
||||
expect(await readFile(path.join(snapshotRetained.remoteRoot, newest!.path, "files", "ticks.log"), "utf8")).toBe(editedLog);
|
||||
}
|
||||
if (kind === "ordinary_followup") {
|
||||
await rm(path.join(cold.remoteRoot, newest!.path), { recursive: true });
|
||||
await saveRemoteWorkspace({ db, storage, workspace: cold.workspace, runId: successor.id,
|
||||
lease: cold.lease, target: { ...target, leaseId: coldLease.id, remoteCwd: cold.remoteRoot }, assertOwned: async () => {} });
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, successor.id));
|
||||
await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success" })
|
||||
.where(eq(environmentLeases.id, coldLease.id));
|
||||
const [normal] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: agent.id, issueId: f.issueId,
|
||||
status: "running", runtimeMode: "legacy", contextSnapshot: { issueId: f.issueId } }).returning();
|
||||
const ordinaryWorkspace = await reserveRemoteWorkspace({ db, storage, companyId: f.companyId, issueId: f.issueId,
|
||||
runId: normal.id, agentId: agent.id, environmentId: f.lease.environmentId!, projectId: null,
|
||||
projectWorkspaceId: null, existingWorkspaceId: placeholder.id, source: { kind: "empty" } });
|
||||
const ordinaryAllocation = await realpath(await mkdtemp(path.join(root, "ordinary-")));
|
||||
const [ordinaryLease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
heartbeatRunId: normal.id, environmentId: f.lease.environmentId, provider: "daytona", providerLeaseId: randomUUID(), status: "active" }).returning();
|
||||
const ordinary = await prepareRemoteWorkspace({ db, storage, workspace: ordinaryWorkspace, runId: normal.id,
|
||||
lease: ordinaryLease as EnvironmentLease, target: { ...target, leaseId: ordinaryLease.id, remoteCwd: ordinaryAllocation } });
|
||||
expect(await lstat(path.join(ordinary.remoteRoot, newest!.path)).then(() => true, () => false)).toBe(false);
|
||||
expect(ordinary.workspace.metadata?.remoteWorkspaceRecoveredFiles ?? []).toEqual([]);
|
||||
}
|
||||
} finally { acquire.mockRestore(); storageRead.mockRestore(); }
|
||||
expect((await run(f.runId)).error).toBe("native_remote_recovery_lease_mismatch");
|
||||
} finally { vi.unstubAllEnvs(); }
|
||||
}, 30_000);
|
||||
|
||||
it("reads canonical attachments from the caller's storage backend", async () => {
|
||||
const f = await failedRemotePreparation(); await recoverRetainedWorkspace(f.input);
|
||||
const source = await run(f.runId), destination = await mkdtemp(path.join(root, "injected-storage-"));
|
||||
const [writer] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId,
|
||||
issueId: f.issueId, status: "running", responsibleUserId: "local-board", contextSnapshot: { issueId: f.issueId } }).returning();
|
||||
const global = getStorageService(), read = global.getObject.bind(global);
|
||||
const injected = { ...global, getObject: vi.fn(read) };
|
||||
const wrongBackend = vi.spyOn(global, "getObject").mockRejectedValue(new Error("wrong storage backend"));
|
||||
try {
|
||||
const scope = { db, companyId: f.companyId, issueId: f.issueId, runId: writer.id, workspaceId: randomUUID(),
|
||||
sourceRunIds: [source.id], cwd: destination, storage: injected };
|
||||
const attached = await attachPreservedRemoteWorkspaceSnapshots(scope);
|
||||
expect(await readFile(path.join(attached[0].path, "files", "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(injected.getObject).toHaveBeenCalled();
|
||||
expect(wrongBackend).not.toHaveBeenCalled();
|
||||
} finally { wrongBackend.mockRestore(); }
|
||||
}, 30_000);
|
||||
|
||||
it("restores ordinary same-workspace Retry after reservation advances current authority", async () => {
|
||||
vi.stubEnv("DATABASE_URL", connectionString);
|
||||
try {
|
||||
const f = await failedRemotePreparation(); await recoverRetainedWorkspace(f.input);
|
||||
const source = await run(f.runId);
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId, status: "running" });
|
||||
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "local-board",
|
||||
actorSource: "local_implicit", observedRunId: source.id, idempotencyKey: randomUUID() });
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, receipt.requestId!);
|
||||
const [successor] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${receipt.requestId}`));
|
||||
expect(await readFreshWorkspaceReadiness(db, successor)).toMatchObject({ mode: "resume_preserved_workspace" });
|
||||
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, successor.id));
|
||||
const storage = { ...getStorageService(), provider: "s3" as const };
|
||||
const reserved = await reserveRemoteWorkspace({ db, storage, companyId: f.companyId, issueId: f.issueId,
|
||||
runId: successor.id, agentId: source.agentId, environmentId: f.lease.environmentId!, projectId: null,
|
||||
projectWorkspaceId: null, existingWorkspaceId: f.workspaceId, source: { kind: "empty" } });
|
||||
expect(await hasVerifiedWorkspaceRepair(db, source)).toBe(false);
|
||||
const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
heartbeatRunId: successor.id, environmentId: f.lease.environmentId, provider: "daytona", providerLeaseId: randomUUID(), status: "active" }).returning();
|
||||
const target = { ...targets.get(f.lease.id)!, leaseId: lease.id,
|
||||
remoteCwd: await realpath(await mkdtemp(path.join(root, "same-workspace-"))),
|
||||
runner: { ...targets.get(f.lease.id)!.runner!, syncIn: async operations => {
|
||||
for (const operation of operations) for (const file of operation.files) {
|
||||
await mkdir(path.dirname(file.targetPath), { recursive: true }); await cp(file.sourcePath, file.targetPath, { recursive: true });
|
||||
}
|
||||
return { filesTransferred: 1, bytesTransferred: 0, operations: [] };
|
||||
} } } satisfies AdapterSandboxExecutionTarget;
|
||||
const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: successor.id,
|
||||
lease: lease as EnvironmentLease, target });
|
||||
expect(prepared.workspace.id).toBe(f.workspaceId);
|
||||
expect(await readFile(path.join(prepared.remoteRoot, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(prepared.workspace.metadata?.remoteWorkspaceRecoveredFiles ?? []).toEqual([]);
|
||||
} finally { vi.unstubAllEnvs(); }
|
||||
}, 30_000);
|
||||
|
||||
it.each(["missing_receipt", "missing_reference", "prepared_reference", "missing_snapshot_id", "uncommitted", "foreign_snapshot", "wrong_lease", "missing_object", "corrupt_object", "wrong_company", "wrong_issue"])(
|
||||
"refuses canonical recovery attachments with %s evidence", async fault => {
|
||||
const f = await failedRemotePreparation();
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const source = await run(f.runId), remote = source.runnerProfileJson!.remoteWorkspace as { snapshotId: string };
|
||||
const [writer] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId,
|
||||
issueId: f.issueId, status: "running", responsibleUserId: "local-board", contextSnapshot: { issueId: f.issueId } }).returning();
|
||||
const destination = path.join(root, writer.id); await mkdir(destination);
|
||||
const [receipt] = await db.select().from(workspaceOperations).where(and(eq(workspaceOperations.heartbeatRunId, f.runId),
|
||||
eq(workspaceOperations.phase, "workspace_recovery"), eq(workspaceOperations.status, "succeeded")));
|
||||
if (fault === "missing_receipt") await db.update(workspaceOperations).set({ status: "failed" }).where(eq(workspaceOperations.id, receipt.id));
|
||||
if (fault === "missing_reference") await db.update(heartbeatRuns).set({ runnerProfileJson:
|
||||
Object.fromEntries(Object.entries(source.runnerProfileJson!).filter(([key]) => key !== "remoteWorkspace")) }).where(eq(heartbeatRuns.id, source.id));
|
||||
if (fault === "prepared_reference" || fault === "missing_snapshot_id") await db.update(heartbeatRuns).set({ runnerProfileJson: {
|
||||
...source.runnerProfileJson, remoteWorkspace: { ...source.runnerProfileJson!.remoteWorkspace as object,
|
||||
...(fault === "prepared_reference" ? { state: "prepared" } : { snapshotId: null }) },
|
||||
} }).where(eq(heartbeatRuns.id, source.id));
|
||||
if (fault === "uncommitted") await db.update(executionWorkspaceSnapshots).set({ status: "uploaded" }).where(eq(executionWorkspaceSnapshots.id, remote.snapshotId));
|
||||
if (fault === "foreign_snapshot") await db.update(heartbeatRuns).set({ runnerProfileJson: { ...source.runnerProfileJson,
|
||||
remoteWorkspace: { ...source.runnerProfileJson!.remoteWorkspace as object, snapshotId: randomUUID() } } }).where(eq(heartbeatRuns.id, source.id));
|
||||
if (fault === "wrong_lease") await db.update(workspaceOperations).set({ metadata: { ...receipt.metadata, leaseId: randomUUID() } })
|
||||
.where(eq(workspaceOperations.id, receipt.id));
|
||||
const storage = getStorageService(), getObject = storage.getObject.bind(storage);
|
||||
const failingRead = ["missing_object", "corrupt_object"].includes(fault) ? vi.spyOn(storage, "getObject").mockImplementation(async (...args) => {
|
||||
if (fault === "missing_object") throw new Error("fixture: object missing");
|
||||
const object = await getObject(...args); object.stream.destroy();
|
||||
return { ...object, stream: Readable.from([Buffer.from("corrupt")]) };
|
||||
}) : undefined;
|
||||
try {
|
||||
await expect(attachPreservedRemoteWorkspaceSnapshots({ db,
|
||||
companyId: fault === "wrong_company" ? randomUUID() : f.companyId,
|
||||
issueId: fault === "wrong_issue" ? randomUUID() : f.issueId,
|
||||
runId: writer.id, workspaceId: randomUUID(), sourceRunIds: [source.id], cwd: destination })).rejects.toThrow();
|
||||
expect(await readdir(destination)).toEqual(fault === "missing_object" || fault === "corrupt_object" ? [".paperclip-recovered"] : []);
|
||||
expect(await readFile(path.join(f.remote, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
} finally { failingRead?.mockRestore(); }
|
||||
}, 30_000);
|
||||
|
||||
it.each(["unchanged", "advanced"])("deduplicates immutable canonical sidecars when the source workspace is %s", async state => {
|
||||
const f = await failedRemotePreparation(); await recoverRetainedWorkspace(f.input);
|
||||
const source = await run(f.runId), destination = path.join(root, randomUUID()); await mkdir(destination);
|
||||
await writeFile(path.join(destination, "later.txt"), "Later selected work\n");
|
||||
const [selected] = await db.insert(executionWorkspaces).values({ companyId: f.companyId, sourceIssueId: f.issueId,
|
||||
name: "Later selected workspace", cwd: destination, mode: "isolated_workspace", strategyType: "project_primary" }).returning();
|
||||
await db.update(issues).set({ executionWorkspaceId: selected.id }).where(eq(issues.id, f.issueId));
|
||||
const [writer] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId,
|
||||
issueId: f.issueId, status: "running", responsibleUserId: "local-board", contextSnapshot: { issueId: f.issueId } }).returning();
|
||||
const scope = { db, companyId: f.companyId, issueId: f.issueId, runId: writer.id,
|
||||
workspaceId: selected.id, sourceRunIds: [source.id, source.id], cwd: destination };
|
||||
const first = await attachPreservedRemoteWorkspaceSnapshots(scope);
|
||||
expect(first).toHaveLength(1);
|
||||
expect(await readFile(path.join(first[0].path, "files", "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
await writeFile(path.join(first[0].path, "files", "saved.txt"), "User edited sidecar\n");
|
||||
if (state === "advanced") {
|
||||
vi.stubEnv("DATABASE_URL", connectionString);
|
||||
try {
|
||||
const [child] = await db.insert(issues).values({ companyId: f.companyId, parentId: f.issueId,
|
||||
assigneeAgentId: source.agentId, title: "Later legitimate workspace writer", status: "in_progress", executionWorkspaceId: f.workspaceId }).returning();
|
||||
const [later] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId,
|
||||
issueId: child.id, status: "running", responsibleUserId: "local-board", contextSnapshot: { issueId: child.id } }).returning();
|
||||
const storage = { ...getStorageService(), provider: "s3" as const };
|
||||
const reserved = await reserveRemoteWorkspace({ db, storage, companyId: f.companyId, issueId: child.id,
|
||||
runId: later.id, agentId: source.agentId, environmentId: f.lease.environmentId!, projectId: null,
|
||||
projectWorkspaceId: null, existingWorkspaceId: f.workspaceId, source: { kind: "empty" } });
|
||||
const allocation = await realpath(await mkdtemp(path.join(root, "later-writer-")));
|
||||
const [lease] = await db.insert(environmentLeases).values({ companyId: f.companyId, issueId: child.id,
|
||||
heartbeatRunId: later.id, environmentId: f.lease.environmentId, provider: "daytona", providerLeaseId: randomUUID(), status: "active" }).returning();
|
||||
const target = { ...targets.get(f.lease.id)!, leaseId: lease.id, remoteCwd: allocation,
|
||||
runner: { ...targets.get(f.lease.id)!.runner!, syncIn: async operations => {
|
||||
for (const operation of operations) for (const file of operation.files) {
|
||||
await mkdir(path.dirname(file.targetPath), { recursive: true }); await cp(file.sourcePath, file.targetPath, { recursive: true });
|
||||
}
|
||||
return { filesTransferred: 1, bytesTransferred: 0, operations: [] };
|
||||
} } } satisfies AdapterSandboxExecutionTarget;
|
||||
const advanced = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: later.id, lease: lease as EnvironmentLease, target });
|
||||
await writeFile(path.join(advanced.remoteRoot, "later.txt"), "newer source workspace bytes\n");
|
||||
await saveRemoteWorkspace({ db, storage, workspace: advanced.workspace, runId: later.id, lease: advanced.lease,
|
||||
target: { ...target, remoteCwd: advanced.remoteRoot }, assertOwned: async () => {} });
|
||||
expect(await hasVerifiedWorkspaceRepair(db, source)).toBe(false);
|
||||
const old = source.runnerProfileJson!.remoteWorkspace as { snapshotId: string };
|
||||
expect((await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, old.snapshotId)))[0].status).toBe("superseded");
|
||||
} finally { vi.unstubAllEnvs(); }
|
||||
}
|
||||
expect(await attachPreservedRemoteWorkspaceSnapshots(scope)).toEqual(first);
|
||||
expect(await readFile(path.join(first[0].path, "files", "saved.txt"), "utf8")).toBe("User edited sidecar\n");
|
||||
expect(await readFile(path.join(destination, "later.txt"), "utf8")).toBe("Later selected work\n");
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionWorkspaceId).toBe(selected.id);
|
||||
expect(await readdir(path.join(destination, ".paperclip-recovered"))).toEqual([first[0].operationId]);
|
||||
}, 30_000);
|
||||
|
||||
it("admits one HTTP Retry after a failed native preparation has verified saved work", async () => {
|
||||
const f = await failedRemotePreparation();
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const source = await run(f.runId);
|
||||
expect(source).toMatchObject({ runtimeMode: "legacy", runtimeModeResolvedAt: null, executionStage: "preparing",
|
||||
status: "failed", errorCode: "setup_failed", nativeIssueId: null, nativeSessionId: null,
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } });
|
||||
expect(await hasVerifiedWorkspaceRepair(db, source)).toBe(true);
|
||||
const app = boardRecoveryApp(), heartbeat = heartbeatService(db);
|
||||
const intent = { observedRunId: f.runId, idempotencyKey: randomUUID() };
|
||||
const receipt = (await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body;
|
||||
await Promise.all([heartbeat.resumeTaskRetry(f.companyId, receipt.requestId),
|
||||
heartbeat.resumeTaskRetry(f.companyId, receipt.requestId)]);
|
||||
expect((await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body.requestId).toBe(receipt.requestId);
|
||||
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${receipt.requestId}`));
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0]).toMatchObject({ status: "queued", contextSnapshot: {
|
||||
previousRunId: f.runId, taskRetryRequestId: receipt.requestId } });
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId)))[0])
|
||||
.toMatchObject({ status: "coalesced", runId: successors[0].id });
|
||||
expect(await readFreshWorkspaceReadiness(db, successors[0])).toMatchObject({ mode: "resume_preserved_workspace" });
|
||||
const remote = source.runnerProfileJson!.remoteWorkspace as { snapshotId: string };
|
||||
const saved = await createSavedWorkspaceLoader(db)({ companyId: f.companyId, workspaceId: f.workspaceId,
|
||||
snapshotId: remote.snapshotId });
|
||||
try { expect(await readFile(path.join(saved.root, "saved.txt"), "utf8")).toBe("finished work\n"); }
|
||||
finally { await saved.cleanup(); }
|
||||
expect((await run(f.runId)).error).toBe(source.error);
|
||||
}, 30_000);
|
||||
|
||||
async function recoveredPreparationRetryChain(mode: "certified" | "inherited_session" | "historical_warning" | "empty_object_log" = "inherited_session") {
|
||||
const f = await failedRemotePreparation("retained", mode !== "certified", mode === "historical_warning");
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
if (mode !== "certified") await db.update(heartbeatRuns).set({ errorCode: "adapter_failed",
|
||||
sessionIdBefore: randomUUID() }).where(eq(heartbeatRuns.id, f.runId));
|
||||
const source = await run(f.runId);
|
||||
const [completed] = mode !== "certified" ? await db.insert(heartbeatRuns).values({
|
||||
companyId: f.companyId, issueId: f.issueId, agentId: source.agentId, status: "succeeded",
|
||||
createdAt: new Date(source.createdAt.getTime() - 1000), finishedAt: source.createdAt,
|
||||
sessionIdAfter: source.sessionIdBefore, resultJson: { summary: "Earlier turn completed" },
|
||||
}).returning() : [];
|
||||
if (completed) await db.update(heartbeatRuns).set({ retryOfRunId: completed.id }).where(eq(heartbeatRuns.id, source.id));
|
||||
if (mode === "empty_object_log") await recordPreparationObjectLog(source);
|
||||
const [failed] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
agentId: source.agentId, runtimeMode: "legacy", status: "failed", retryOfRunId: source.id,
|
||||
sessionIdBefore: source.sessionIdBefore,
|
||||
startedAt: new Date(), finishedAt: new Date(), executionStage: "preparing", errorCode: "setup_failed",
|
||||
error: "The workspace still has an unfinished writer or save operation.",
|
||||
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
||||
contextSnapshot: { issueId: f.issueId, source: "issue.continuation_recovery" },
|
||||
resultJson: { stopReason: "adapter_failed", executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
||||
}).returning();
|
||||
return { ...f, source, failed, completed };
|
||||
}
|
||||
|
||||
async function recordPreparationObjectLog(source: typeof heartbeatRuns.$inferSelect, outputText?: string) {
|
||||
const store = objectLogFixture();
|
||||
const output = createHeartbeatRunOutput(db, { run: source, runId: source.id, issueId: source.issueId,
|
||||
runLogStore: store, identityRedactor: { chunk: (_stream, value) => value, redact: value => value },
|
||||
secretKeys: new Set(), appendRunEvent: vi.fn(), getCurrentUserRedactionOptions: async () => ({ enabled: false }) });
|
||||
const handlers = await output.begin(source);
|
||||
if (outputText) await handlers.onLog("stdout", outputText);
|
||||
const summary = await store.finalize(output.handle!);
|
||||
await output.flushOutputProgress({ force: true });
|
||||
await db.update(heartbeatRuns).set({ logBytes: summary.bytes, logSha256: summary.sha256,
|
||||
logCompressed: summary.compressed, stdoutExcerpt: output.stdoutExcerpt, stderrExcerpt: output.stderrExcerpt,
|
||||
}).where(eq(heartbeatRuns.id, source.id));
|
||||
return { store, summary };
|
||||
}
|
||||
|
||||
it.each(["certified", "inherited_session", "historical_warning", "empty_object_log"] as const)("admits one HTTP Retry through a failed automatic successor after its preparation ancestor is recovered: %s", async mode => {
|
||||
const f = await recoveredPreparationRetryChain(mode);
|
||||
const app = boardRecoveryApp(), heartbeat = heartbeatService(db);
|
||||
const intent = { observedRunId: f.failed.id, idempotencyKey: randomUUID() };
|
||||
const receipt = (await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body;
|
||||
await Promise.all([heartbeat.resumeTaskRetry(f.companyId, receipt.requestId),
|
||||
heartbeat.resumeTaskRetry(f.companyId, receipt.requestId)]);
|
||||
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${receipt.requestId}`));
|
||||
expect(successors).toHaveLength(1);
|
||||
expect(successors[0]).toMatchObject({ status: "queued", contextSnapshot: {
|
||||
forceFreshSession: true, previousRunId: f.failed.id, taskRetryRequestId: receipt.requestId } });
|
||||
expect(await readFreshWorkspaceReadiness(db, successors[0])).toMatchObject({ mode: "resume_preserved_workspace" });
|
||||
expect(await buildExecutionContinuation({ db, companyId: f.companyId, issueId: f.issueId,
|
||||
agentId: f.source.agentId, runId: successors[0].id, context: successors[0].contextSnapshot!,
|
||||
summary: null, exposeLowTrustRaw: false })).toMatchObject({ interruptedRunId: f.failed.id });
|
||||
const remote = (await run(f.source.id)).runnerProfileJson!.remoteWorkspace as { snapshotId: string };
|
||||
const saved = await createSavedWorkspaceLoader(db)({ companyId: f.companyId, workspaceId: f.workspaceId,
|
||||
snapshotId: remote.snapshotId });
|
||||
try { expect(await readFile(path.join(saved.root, "saved.txt"), "utf8")).toBe("finished work\n"); }
|
||||
finally { await saved.cleanup(); }
|
||||
expect((await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body.requestId).toBe(receipt.requestId);
|
||||
expect((await run(f.runId)).error).toBe(f.source.error);
|
||||
if (f.completed) expect(await run(f.completed.id)).toEqual(f.completed);
|
||||
}, 30_000);
|
||||
|
||||
it.each(["unknown_size", "unknown_digest", "wrong_digest", "nonempty_size", "foreign_reference", "other_store",
|
||||
"stdout", "stderr", "output_sequence", "output_time", "output_bytes", "output_stream", "compressed", "recorded_output"] as const)(
|
||||
"keeps a logged preparation ancestor held without complete empty-output proof: %s", async gate => {
|
||||
const f = await recoveredPreparationRetryChain("empty_object_log");
|
||||
if (gate === "recorded_output") await recordPreparationObjectLog(await run(f.source.id), "Provider work was recorded.\n");
|
||||
const patch: Partial<typeof heartbeatRuns.$inferInsert> = gate === "unknown_size" ? { logBytes: null }
|
||||
: gate === "unknown_digest" ? { logSha256: null }
|
||||
: gate === "wrong_digest" ? { logSha256: "0".repeat(64) }
|
||||
: gate === "nonempty_size" ? { logBytes: 1 }
|
||||
: gate === "foreign_reference" ? { logRef: `${f.companyId}/${f.source.agentId}/${randomUUID()}` }
|
||||
: gate === "other_store" ? { logStore: "s3_archive" }
|
||||
: gate === "stdout" ? { stdoutExcerpt: "Recorded provider output" }
|
||||
: gate === "stderr" ? { stderrExcerpt: "Recorded diagnostic output" }
|
||||
: gate === "output_sequence" ? { lastOutputSeq: 1 }
|
||||
: gate === "output_time" ? { lastOutputAt: new Date() }
|
||||
: gate === "output_bytes" ? { lastOutputBytes: 1 }
|
||||
: gate === "output_stream" ? { lastOutputStream: "stdout" }
|
||||
: gate === "compressed" ? { logCompressed: true } : {};
|
||||
if (Object.keys(patch).length) await db.update(heartbeatRuns).set(patch).where(eq(heartbeatRuns.id, f.source.id));
|
||||
const receipt = (await request(boardRecoveryApp()).post(`/api/issues/${f.issueId}/retry`)
|
||||
.send({ observedRunId: f.failed.id, idempotencyKey: randomUUID() }).expect(202)).body;
|
||||
await heartbeatService(db).resumeTaskRetry(f.companyId, receipt.requestId);
|
||||
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${receipt.requestId}`))).toHaveLength(0);
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId)))[0])
|
||||
.toMatchObject({ status: "deferred_issue_execution", runId: null });
|
||||
expect((await action(f.actionId)).evidence.explicitUserContinuation).toBeUndefined();
|
||||
}, 30_000);
|
||||
|
||||
it.each(["foreign_company", "foreign_task", "foreign_agent", "conflicting_binding", "missing_edge", "cycle", "forged_context",
|
||||
"live_predecessor", "live_controller", "unknown_stop", "failed_cleanup", "pending_cleanup", "failed_save",
|
||||
"unsafe_workspace", "operator_stop", "wrong_actor", "wrong_generation", "wrong_recovery_source", "sibling_hold", "completed_ancestor",
|
||||
"provider_event", "native_input", "session_checkpoint", "controller_recovery", "native_result", "process", "runtime_selected",
|
||||
"unknown_result", "provider_usage", "invalid_accounting", "foreign_lease_issue", "foreign_started_event", "invalid_started_event",
|
||||
"warning_reason", "provider_warning", "extra_after_warning", "unknown_failure_code", "missing_failure_code", "operator_stop_mislabeled"] as const)(
|
||||
"keeps ancestor Retry held when current proof changes: %s", async gate => {
|
||||
const f = await recoveredPreparationRetryChain(["warning_reason", "provider_warning", "extra_after_warning"].includes(gate)
|
||||
? "historical_warning" : "inherited_session");
|
||||
const heartbeat = heartbeatService(db), app = receiptOnlyRecoveryApp();
|
||||
const original = readSystemRecovery((await action(f.actionId)).evidence)!;
|
||||
const receipt = (await request(app).post(`/api/issues/${f.issueId}/retry`)
|
||||
.send({ observedRunId: f.failed.id, idempotencyKey: randomUUID() }).expect(202)).body;
|
||||
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId));
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({
|
||||
generationId: original.generationId, attempts: original.attempts, intentId: receipt.requestId });
|
||||
if (["foreign_company", "foreign_task", "foreign_agent", "conflicting_binding"].includes(gate)) {
|
||||
const companyId = gate === "foreign_company" ? randomUUID() : f.companyId;
|
||||
if (companyId !== f.companyId) {
|
||||
await db.insert(companies).values({ id: companyId, name: "Other company", issuePrefix: `X${companyId.slice(0, 6)}` });
|
||||
fixtureCompanyIds.add(companyId);
|
||||
}
|
||||
const agentId = gate === "foreign_company" || gate === "foreign_agent" ? randomUUID() : f.source.agentId;
|
||||
if (agentId !== f.source.agentId) await db.insert(agents).values({ id: agentId, companyId,
|
||||
name: "Other owner", adapterType: "paperclip_runner" });
|
||||
const issueId = gate === "foreign_company" || gate === "foreign_task" || gate === "conflicting_binding" ? randomUUID() : f.issueId;
|
||||
if (issueId !== f.issueId) await db.insert(issues).values({ id: issueId, companyId, title: "Other work" });
|
||||
const [bridge] = await db.insert(heartbeatRuns).values({ companyId, agentId, issueId,
|
||||
nativeIssueId: gate === "conflicting_binding" ? f.issueId : issueId, status: "failed", finishedAt: new Date(), retryOfRunId: f.source.id,
|
||||
createdAt: f.source.createdAt, contextSnapshot: { issueId: f.issueId } }).returning();
|
||||
await db.update(heartbeatRuns).set({ retryOfRunId: bridge.id }).where(eq(heartbeatRuns.id, f.failed.id));
|
||||
}
|
||||
if (gate === "missing_edge" || gate === "forged_context") await db.update(heartbeatRuns).set({ retryOfRunId: null,
|
||||
...(gate === "forged_context" ? { contextSnapshot: { ...f.failed.contextSnapshot,
|
||||
previousRunId: f.source.id, retryOfRunId: f.source.id } } : {}) }).where(eq(heartbeatRuns.id, f.failed.id));
|
||||
if (gate === "cycle") await db.update(heartbeatRuns).set({ retryOfRunId: f.source.id }).where(eq(heartbeatRuns.id, f.source.id));
|
||||
if (gate === "live_predecessor") await db.update(heartbeatRuns).set({ status: "running", finishedAt: null })
|
||||
.where(eq(heartbeatRuns.id, f.source.id));
|
||||
if (gate === "live_controller") await db.update(heartbeatRuns).set({ controllerLeaseExpiresAt: new Date(Date.now() + 60_000) })
|
||||
.where(eq(heartbeatRuns.id, f.source.id));
|
||||
if (gate === "provider_event" || gate === "extra_after_warning") await appendHeartbeatRunEvent(db, {
|
||||
companyId: f.companyId, runId: f.source.id, agentId: f.source.agentId, stream: "system", level: "info",
|
||||
eventType: gate === "provider_event" ? "session.started" : "lifecycle", message: "Unexpected later event" });
|
||||
if (["native_input", "session_checkpoint", "controller_recovery"].includes(gate)) {
|
||||
const key = gate === "native_input" ? "nativeExecutionInput" : gate === "session_checkpoint" ? "sessionCheckpoint" : "nativeControllerRecovery";
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...f.source.runnerProfileJson, [key]: { sessionId: randomUUID() } } })
|
||||
.where(eq(heartbeatRuns.id, f.source.id));
|
||||
}
|
||||
if (gate === "native_result") {
|
||||
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.companyId, f.companyId));
|
||||
await db.update(heartbeatRuns).set({ nativeIssueId: f.issueId, completionContractId: contract.id })
|
||||
.where(eq(heartbeatRuns.id, f.source.id));
|
||||
await db.insert(nativeRunResults).values({ companyId: f.companyId, issueId: f.issueId, runId: f.source.id,
|
||||
completionContractId: contract.id, schemaStatus: "accepted", canonicalSha256: "unexpected", serverFingerprint: "unexpected",
|
||||
resultJson: { terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, result: CONTROL_PLANE_CONFORMANCE_RESULT } });
|
||||
}
|
||||
if (gate === "process") await db.update(heartbeatRuns).set({ processPid: 999999999 }).where(eq(heartbeatRuns.id, f.source.id));
|
||||
if (gate === "runtime_selected") await db.update(heartbeatRuns).set({ runtimeModeResolvedAt: new Date() }).where(eq(heartbeatRuns.id, f.source.id));
|
||||
if (gate === "unknown_result") await db.update(heartbeatRuns).set({ resultJson: { ...f.source.resultJson, unknownOutcome: true } })
|
||||
.where(eq(heartbeatRuns.id, f.source.id));
|
||||
if (gate === "provider_usage" || gate === "invalid_accounting") await db.update(heartbeatRuns).set({ usageJson: gate === "provider_usage"
|
||||
? { inputTokens: 1 } : { ledgerScope: { issueId: randomUUID(), projectId: null, billingCode: null } } }).where(eq(heartbeatRuns.id, f.source.id));
|
||||
if (["foreign_started_event", "invalid_started_event", "warning_reason", "provider_warning"].includes(gate))
|
||||
await db.update(heartbeatRunEvents).set(gate === "foreign_started_event" || gate === "provider_warning"
|
||||
? { sourceEventId: randomUUID() } : { payload: gate === "warning_reason" ? { reason: "unknown", revision: randomUUID() } : { unknown: true } })
|
||||
.where(and(eq(heartbeatRunEvents.runId, f.source.id), eq(heartbeatRunEvents.seq, 1)));
|
||||
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
if (gate === "foreign_lease_issue") {
|
||||
const [other] = await db.insert(issues).values({ companyId: f.companyId, title: "Other allocation owner" }).returning();
|
||||
await db.update(environmentLeases).set({ issueId: other.id }).where(eq(environmentLeases.id, lease.id));
|
||||
}
|
||||
if (gate === "unknown_stop") await db.update(environmentLeases).set({ metadata: { ...lease.metadata,
|
||||
remoteExecutionTermination: null } }).where(eq(environmentLeases.id, lease.id));
|
||||
if (gate === "failed_cleanup") await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, lease.id));
|
||||
if (gate === "pending_cleanup") await db.update(environmentLeases).set({ metadata: { ...lease.metadata,
|
||||
pendingCleanupInFlight: true } }).where(eq(environmentLeases.id, lease.id));
|
||||
if (gate === "failed_save") await db.update(workspaceOperations).set({ status: "failed" }).where(and(
|
||||
eq(workspaceOperations.companyId, f.companyId), eq(workspaceOperations.heartbeatRunId, f.source.id),
|
||||
eq(workspaceOperations.phase, "workspace_recovery")));
|
||||
if (gate === "unsafe_workspace") {
|
||||
const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, f.workspaceId));
|
||||
await db.update(executionWorkspaces).set({ metadata: { ...workspace.metadata,
|
||||
remoteWorkspace: { ...(workspace.metadata!.remoteWorkspace as object), dirty: true, writerRunId: f.source.id } } })
|
||||
.where(eq(executionWorkspaces.id, f.workspaceId));
|
||||
}
|
||||
if (gate === "operator_stop") await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "operator_interrupted" })
|
||||
.where(eq(heartbeatRuns.id, f.source.id));
|
||||
if (["unknown_failure_code", "missing_failure_code", "operator_stop_mislabeled"].includes(gate)) await db.update(heartbeatRuns)
|
||||
.set({ errorCode: gate === "missing_failure_code" ? null : gate === "operator_stop_mislabeled" ? "operator_interrupted" : "unknown_failure" })
|
||||
.where(eq(heartbeatRuns.id, f.source.id));
|
||||
if (gate === "wrong_actor") await db.update(agentWakeupRequests).set({ requestedByActorId: "other-user" })
|
||||
.where(eq(agentWakeupRequests.id, wake.id));
|
||||
if (gate === "wrong_generation") await db.update(agentWakeupRequests).set({ payload: { ...wake.payload,
|
||||
taskRetry: { ...(wake.payload!.taskRetry as object), generationId: randomUUID() } } }).where(eq(agentWakeupRequests.id, wake.id));
|
||||
if (gate === "wrong_recovery_source") {
|
||||
const record = await action(f.actionId);
|
||||
const systemRecovery = { ...readSystemRecovery(record.evidence)!, sourceRunId: randomUUID(),
|
||||
intentId: null, state: "exhausted" as const };
|
||||
await db.update(issueRecoveryActions).set({ evidence: { ...record.evidence, systemRecovery } })
|
||||
.where(eq(issueRecoveryActions.id, record.id));
|
||||
const next = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "local-board",
|
||||
actorSource: "local_implicit", observedRunId: f.failed.id, idempotencyKey: randomUUID() });
|
||||
expect(next.requestId).toBe(receipt.requestId);
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)).toEqual(systemRecovery);
|
||||
}
|
||||
if (gate === "sibling_hold" || gate === "completed_ancestor") {
|
||||
const [other] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.source.agentId,
|
||||
issueId: f.issueId, status: gate === "completed_ancestor" ? "succeeded" : "failed", finishedAt: new Date(),
|
||||
createdAt: new Date(f.source.createdAt.getTime() - 1000), runnerProfileJson: f.source.runnerProfileJson }).returning();
|
||||
const originalAction = await action(f.actionId);
|
||||
const systemRecovery = { ...createSystemRecovery(other.id), state: "exhausted" as const };
|
||||
const [hold] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
|
||||
kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation", fingerprint: other.id,
|
||||
status: "resolved", outcome: "blocked", nextAction: "Existing hold", evidence: { ...originalAction.evidence,
|
||||
runId: other.id, systemRecovery } }).returning();
|
||||
if (gate === "completed_ancestor") await db.update(heartbeatRuns).set({ retryOfRunId: other.id }).where(eq(heartbeatRuns.id, f.source.id));
|
||||
const next = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "local-board",
|
||||
actorSource: "local_implicit", observedRunId: f.failed.id, idempotencyKey: randomUUID() });
|
||||
expect(next.requestId).toBe(receipt.requestId);
|
||||
expect(readSystemRecovery((await action(hold.id)).evidence)).toEqual(systemRecovery);
|
||||
}
|
||||
await Promise.all([heartbeat.resumeTaskRetry(f.companyId, receipt.requestId), heartbeat.resumeTaskRetry(f.companyId, receipt.requestId)]);
|
||||
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${receipt.requestId}`))).toHaveLength(0);
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId)))[0])
|
||||
.toMatchObject({ status: "deferred_issue_execution", runId: null });
|
||||
expect((await action(f.actionId)).evidence.explicitUserContinuation).toBeUndefined();
|
||||
}, 30_000);
|
||||
|
||||
it.each(["current worker", "stopped worker"] as const)("resumes the same failed-preparation HTTP Retry from the real preservation worker: %s", async mode => {
|
||||
const f = await failedRemotePreparation();
|
||||
const stoppedWorker = mode === "stopped worker" ? await stoppedWorkspaceWorker(f) : undefined;
|
||||
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
await db.update(environmentLeases).set({ status: "active", releasedAt: null, cleanupStatus: null,
|
||||
metadata: { ...lease.metadata, remoteExecutionTermination: null } }).where(eq(environmentLeases.id, lease.id));
|
||||
let releaseStop!: () => void;
|
||||
const stopped = new Promise<void>(resolve => { releaseStop = resolve; });
|
||||
f.runtime.retryPendingSandboxTeardown.mockImplementation(async () => {
|
||||
await stopped;
|
||||
return { providerLeaseId: f.providerLeaseId, state: "stopped" };
|
||||
});
|
||||
const errors = vi.fn(), heartbeat = heartbeatService(db, { environmentRuntime: f.environmentRuntime });
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: errors });
|
||||
const worker = registerWorkspaceRecoveryWork(db, coordinator, f.environmentRuntime, heartbeat.resumeRemoteStopComments);
|
||||
const successors = () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${requestId}`));
|
||||
let requestId: string;
|
||||
try {
|
||||
await worker.ready;
|
||||
const app = boardRecoveryApp(), intent = { observedRunId: f.runId, idempotencyKey: randomUUID() };
|
||||
const receipt = (await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body;
|
||||
requestId = receipt.requestId;
|
||||
expect((await request(app).post(`/api/issues/${f.issueId}/retry`).send(intent).expect(202)).body.requestId).toBe(requestId);
|
||||
await expect.poll(() => f.runtime.retryPendingSandboxTeardown.mock.calls.length, { timeout: 20_000 }).toBe(1);
|
||||
expect(await successors()).toHaveLength(0);
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(false);
|
||||
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)))[0].metadata?.remoteExecutionTermination).toBeFalsy();
|
||||
releaseStop();
|
||||
await expect.poll(async () => (await successors()).length, { timeout: 20_000 }).toBe(1);
|
||||
const [successor] = await successors();
|
||||
expect(successor.contextSnapshot).toMatchObject({ previousRunId: f.runId, taskRetryRequestId: requestId });
|
||||
expect(await readFreshWorkspaceReadiness(db, successor)).toMatchObject({ mode: "resume_preserved_workspace" });
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
|
||||
if (mode === "stopped worker") {
|
||||
expect(stoppedWorker!.proof).toHaveBeenCalledWith({ runId: f.runId, ownerToken: stoppedWorker!.owner.token,
|
||||
worker: stoppedWorker!.deploymentWorker });
|
||||
expect((await run(f.runId)).runnerProfileJson?.nativeWorkspaceOwnerStopProof).toEqual(stoppedWorker!.stopProof);
|
||||
expect((await run(f.runId)).runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeUndefined();
|
||||
}
|
||||
const remote = (await run(f.runId)).runnerProfileJson!.remoteWorkspace as { snapshotId: string };
|
||||
const saved = await createSavedWorkspaceLoader(db)({ companyId: f.companyId, workspaceId: f.workspaceId, snapshotId: remote.snapshotId });
|
||||
try { expect(await readFile(path.join(saved.root, "saved.txt"), "utf8")).toBe("finished work\n"); }
|
||||
finally { await saved.cleanup(); }
|
||||
await Promise.all([heartbeat.resumeTaskRetry(f.companyId, requestId), heartbeat.resumeRemoteStopComments(await run(f.runId))]);
|
||||
expect(await successors()).toHaveLength(1);
|
||||
expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce();
|
||||
expect((await run(f.runId))).toMatchObject({ status: "failed", errorCode: "setup_failed" });
|
||||
expect(errors).not.toHaveBeenCalled();
|
||||
} finally { releaseStop(); await coordinator.stop(); await worker.stop(); stoppedWorker?.proof.mockRestore(); }
|
||||
}, 40_000);
|
||||
|
||||
});
|
||||
File diff suppressed because it is too large.
Load diff
@@ -9,6 +9,7 @@ import { eq, sql } from "drizzle-orm";
|
||||
import { activityLog, agents, agentWakeupRequests, applyPendingMigrations, companies, createDb, environmentLeases, environments,
|
||||
heartbeatRuns, issueRecoveryActions, issues, workspaceOperations, completionContracts, nativeRunResults, nativeRunFinalizations } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
||||
import { lockWorkspaceRecoveryTestDatabase, WORKSPACE_RECOVERY_EXTERNAL_SETUP_TIMEOUT_MS } from "./__tests__/workspace-recovery-database-lock.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
|
||||
import { inspectWorkspaceRecoverySource } from "./workspace-recovery-source.js";
|
||||
import { createSystemRecovery, failSystemRecovery, SystemRecoveryFailure } from "./system-recovery-state.js";
|
||||
@@ -19,16 +20,18 @@ import { errorHandler } from "../middleware/error-handler.js";
|
||||
describe("workspace source verification diagnostics", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let externalLock: ReturnType<typeof createDb> | undefined;
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_RECOVERY_TEST_DATABASE_URL");
|
||||
beforeAll(async () => {
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_RECOVERY_TEST_DATABASE_URL");
|
||||
if (external) {
|
||||
externalLock = await lockWorkspaceRecoveryTestDatabase(external);
|
||||
await applyPendingMigrations(external);
|
||||
temporary = { connectionString: external, cleanup: async () => {} };
|
||||
} else temporary = await startEmbeddedPostgresTestDatabase("source-verification-");
|
||||
db = createDb(temporary.connectionString);
|
||||
if (external) await db.execute(sql`TRUNCATE TABLE companies CASCADE`);
|
||||
}, 30_000);
|
||||
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); });
|
||||
}, external ? WORKSPACE_RECOVERY_EXTERNAL_SETUP_TIMEOUT_MS : 30_000);
|
||||
afterAll(async () => { await db?.$client.end(); await temporary?.cleanup(); await externalLock?.$client.end(); });
|
||||
|
||||
async function seed() {
|
||||
const companyId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), actionId = randomUUID(), agentId = randomUUID();
|
||||
|
||||
@@ -0,0 +1,668 @@
|
||||
import { inspectWorkspaceSource } from "./workspace-source-inspection.js";
|
||||
import { adapterExecutionControls, createAdapterExecutionControl } from "./adapter-execution-control.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
|
||||
import { environmentService } from "./environments.js";
|
||||
import { executionWorkspaceService } from "./execution-workspaces.js";
|
||||
import * as nativeWorkspaceSync from "./native-runtime/native-workspace-sync.js";
|
||||
import { nativeWorkspaceSyncStateRoot, prepareNativeWorkspaceSync, resumeNativeWorkspaceSync } from "./native-runtime/native-workspace-sync.js";
|
||||
import { readSystemRecovery, createSystemRecovery, failSystemRecovery, reserveSystemRecoveryAttempt, SystemRecoveryFailure } from "./system-recovery-state.js";
|
||||
import { adoptSystemRecovery } from "./system-recovery.js";
|
||||
import { verifiedWorkspaceSnapshotReceipt } from "./workspace-recovery-snapshot.js";
|
||||
import { requestTaskRetry } from "./task-retry.js";
|
||||
import { readFreshWorkspaceReadiness } from "./fresh-workspace-continuation.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { cp, mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, createDb, applyPendingMigrations, environmentLeases, environments, heartbeatRuns, issueComments, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, agentWakeupRequests, executionWorkspaces } from "@paperclipai/db";
|
||||
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
||||
import { nativeWorkspaceFinalizationLockKey } from "./native-runtime/native-workspace-finalization-ownership.js";
|
||||
import { recoverRetainedWorkspace, requestWorkspaceRecovery, reconcileRetainedWorkspaces, retainedWorkspaceCandidates } from "./workspace-recovery.js";
|
||||
import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
||||
import { registerWorkspaceRecoveryWork } from "./workspace-recovery-work.js";
|
||||
import { heartbeatService } from "./heartbeat.js";
|
||||
import { hasVerifiedWorkspaceRepair, readWorkspaceRecovery } from "./workspace-recovery-receipt.js";
|
||||
import { claimNativeSessionResumptions } from "./native-runtime/native-finalization-reconciler.js";
|
||||
import { getExecutionBlocker } from "./execution-blocker.js";
|
||||
import { issueRoutes } from "../routes/issues.js";
|
||||
import { errorHandler } from "../middleware/index.js";
|
||||
import { useWorkspaceRecoveryFixture, action, connectionString, db, failedWarmSource, replaceFixtureState, root, run, seed, warmHandoffSource } from "./__tests__/workspace-recovery-fixture.js";
|
||||
const targets = vi.hoisted(() => new Map<string, AdapterSandboxExecutionTarget>());
|
||||
vi.mock("./environment-execution-target.js", async importOriginal => {
|
||||
const actual = await importOriginal<typeof import("./environment-execution-target.js")>();
|
||||
return { ...actual, resolveEnvironmentExecutionTarget: async (input: Parameters<typeof actual.resolveEnvironmentExecutionTarget>[0]) =>
|
||||
(input.leaseId ? targets.get(input.leaseId) : undefined) ?? actual.resolveEnvironmentExecutionTarget(input) };
|
||||
});
|
||||
|
||||
describe("automatic retained workspace recovery", () => {
|
||||
useWorkspaceRecoveryFixture(targets);
|
||||
|
||||
describe("verified warm allocation handoffs", () => {
|
||||
let original: { db: typeof db; root: string; connectionString: string; home: string | undefined };
|
||||
let isolated: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let isolatedDb: typeof db | undefined;
|
||||
let isolatedRoot: string | undefined;
|
||||
beforeAll(async () => {
|
||||
original = { db, root, connectionString, home: process.env.PAPERCLIP_HOME };
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_RECOVERY_HANDOFF_TEST_DATABASE_URL");
|
||||
if (external) {
|
||||
await applyPendingMigrations(external);
|
||||
isolated = { connectionString: external, cleanup: async () => {} };
|
||||
} else isolated = await startEmbeddedPostgresTestDatabase("workspace-warm-handoff-");
|
||||
isolatedDb = createDb(isolated.connectionString);
|
||||
if (external) await isolatedDb.execute(sql`TRUNCATE TABLE companies CASCADE`);
|
||||
isolatedRoot = await mkdtemp(path.join(os.tmpdir(), "workspace-warm-handoff-files-"));
|
||||
replaceFixtureState({ db: isolatedDb, root: isolatedRoot, connectionString: isolated.connectionString });
|
||||
process.env.PAPERCLIP_HOME = path.join(root, "home");
|
||||
}, 30_000);
|
||||
afterAll(async () => {
|
||||
await isolatedDb?.$client.end(); await isolated?.cleanup();
|
||||
if (isolatedRoot) await rm(isolatedRoot, { recursive: true, force: true });
|
||||
if (original) {
|
||||
replaceFixtureState({ db: original.db, root: original.root, connectionString: original.connectionString });
|
||||
if (original.home === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = original.home;
|
||||
}
|
||||
});
|
||||
it.each([{ hops: 1, snapshot: false }, { hops: 2, snapshot: false }, { hops: 2, snapshot: true }])(
|
||||
"settles adopted warm source history: $hops handoffs, snapshot=$snapshot", async ({ hops, snapshot }) => {
|
||||
const f = await failedWarmSource(false, hops);
|
||||
if (snapshot) {
|
||||
const runner = targets.get(f.lease.id)!.runner!;
|
||||
runner.syncIn = async () => ({ filesTransferred: 0, bytesTransferred: 0, operations: [] });
|
||||
runner.syncOut = async operations => {
|
||||
for (const operation of operations) for (const file of operation.files)
|
||||
await cp(file.sourcePath, file.targetPath, { recursive: true, dereference: false });
|
||||
return { filesTransferred: 2, bytesTransferred: 0, operations: [] };
|
||||
};
|
||||
await rm(path.join(nativeWorkspaceSyncStateRoot(), f.runId), { recursive: true });
|
||||
}
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
|
||||
const [lease] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id));
|
||||
expect(lease).toMatchObject({ status: "released", cleanupStatus: "success",
|
||||
leasePolicy: snapshot ? "retain_on_failure" : "reuse_by_environment" });
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(!snapshot);
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.retry.requestId!)))[0])
|
||||
.toMatchObject({ status: "deferred_issue_execution", runId: null });
|
||||
expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce();
|
||||
if (snapshot) {
|
||||
const state = readSystemRecovery((await action(f.actionId)).evidence)!;
|
||||
expect(await verifiedWorkspaceSnapshotReceipt(db, { companyId: f.companyId, issueId: f.issueId,
|
||||
runId: f.runId, actionId: f.actionId, operationIds: state.preservationOperationIds })).not.toBeNull();
|
||||
expect(lease.metadata?.nativeSourceRetention).toMatchObject({ settlement: "snapshot_retained" });
|
||||
} else expect(await readFile(path.join(f.host, "after-warm-stop.txt"), "utf8")).toBe("stopped turn work\n");
|
||||
}, 30_000);
|
||||
|
||||
it("keeps adopted warm history held when a predecessor changes after export", async () => {
|
||||
const f = await failedWarmSource(false, 1);
|
||||
const history = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, f.companyId),
|
||||
eq(environmentLeases.providerLeaseId, f.providerLeaseId)));
|
||||
const predecessor = history.find(lease => lease.id !== f.lease.id)!;
|
||||
const stop = f.runtime.retryPendingSandboxTeardown.getMockImplementation()!;
|
||||
f.runtime.retryPendingSandboxTeardown.mockImplementationOnce(async input => {
|
||||
const receipt = await stop(input);
|
||||
await db.update(environmentLeases).set({ status: "active" }).where(eq(environmentLeases.id, predecessor.id));
|
||||
return receipt;
|
||||
});
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const state = readSystemRecovery((await action(f.actionId)).evidence)!;
|
||||
expect(state.state).not.toBe("complete");
|
||||
expect(state.failure?.code).toBe("allocation_ownership_ambiguous");
|
||||
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0].leasePolicy).toBe("retain_on_failure");
|
||||
expect(await readFile(path.join(f.host, "after-warm-stop.txt"), "utf8")).toBe("stopped turn work\n");
|
||||
}, 30_000);
|
||||
|
||||
it.each([1, 2])("recovers a stopped native source after %s verified warm predecessor handoffs", async hops => {
|
||||
const f = await warmHandoffSource(hops);
|
||||
expect((await run(f.priorRunId)).runnerProfileJson?.nativeWorkspaceSync).toMatchObject({ state: "finalized", leaseId: f.priorLeaseId });
|
||||
const [previous] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.priorLeaseId));
|
||||
expect(previous).toMatchObject({ status: "expired", cleanupStatus: "success", providerLeaseId: f.providerLeaseId });
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
|
||||
expect(await readFile(path.join(f.host, "after-warm-stop.txt"), "utf8")).toBe("stopped turn work\n");
|
||||
const source = await run(f.runId), freshProviderLeaseId = randomUUID();
|
||||
const [fresh] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId,
|
||||
issueId: f.issueId, nativeIssueId: f.issueId, runtimeMode: "native", status: "succeeded", finishedAt: new Date() }).returning();
|
||||
const remoteCwd = path.join(root, fresh.id, "remote");
|
||||
const freshLease = await environmentService(db).acquireLease({ companyId: f.companyId, environmentId: f.lease.environmentId!,
|
||||
issueId: f.issueId, heartbeatRunId: fresh.id, provider: "daytona", providerLeaseId: freshProviderLeaseId,
|
||||
metadata: { ...f.lease.metadata, remoteCwd } });
|
||||
const freshTarget = { ...targets.get(f.lease.id)!, leaseId: freshLease.id, remoteCwd,
|
||||
sandboxLeaseAcquisition: { outcome: "created" as const, providerLeaseId: freshProviderLeaseId } };
|
||||
expect(await prepareNativeWorkspaceSync({ db, companyId: f.companyId, workspaceId: f.issueId,
|
||||
workspaceLocalDir: f.host, runId: fresh.id, lease: freshLease, target: freshTarget, resourceDisposition: "keep_running" })).not.toBeNull();
|
||||
expect(await resumeNativeWorkspaceSync({ db, runId: fresh.id, target: freshTarget })).toBe(true);
|
||||
expect(await inspectWorkspaceSource(db, { companyId: f.companyId, issueId: f.issueId, runId: f.runId, actionId: f.actionId }))
|
||||
.toMatchObject({ status: "verified" });
|
||||
}, 30_000);
|
||||
|
||||
it.each(["active", "active_run", "active_control", "foreign_company", "environment", "workspace", "task", "agent", "unfinalized", "owner", "missing_proof", "conflicting_proof", "dangling_proof", "extra_history"])(
|
||||
"keeps ambiguous warm source history fenced: %s", async fault => {
|
||||
const f = await warmHandoffSource();
|
||||
const [previous] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.priorLeaseId));
|
||||
if (fault === "active") await db.update(environmentLeases).set({ status: "active" }).where(eq(environmentLeases.id, previous.id));
|
||||
if (fault === "active_run") await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, f.priorRunId));
|
||||
if (fault === "active_control") adapterExecutionControls.set(f.priorRunId, createAdapterExecutionControl());
|
||||
if (fault === "foreign_company") {
|
||||
const [foreign] = await db.insert(companies).values({ name: "Foreign owner", issuePrefix: `F${randomUUID().slice(0, 6)}` }).returning();
|
||||
await db.update(environmentLeases).set({ companyId: foreign.id }).where(eq(environmentLeases.id, previous.id));
|
||||
}
|
||||
if (fault === "environment") {
|
||||
const [other] = await db.insert(environments).values({ name: randomUUID(), driver: "sandbox" }).returning();
|
||||
await db.update(environmentLeases).set({ environmentId: other.id }).where(eq(environmentLeases.id, previous.id));
|
||||
}
|
||||
if (fault === "workspace") {
|
||||
const [other] = await db.insert(executionWorkspaces).values({ companyId: f.companyId, mode: "isolated_workspace", strategyType: "test", name: "Other workspace" }).returning();
|
||||
await db.update(environmentLeases).set({ executionWorkspaceId: other.id }).where(eq(environmentLeases.id, previous.id));
|
||||
}
|
||||
if (fault === "task") {
|
||||
const [other] = await db.insert(issues).values({ companyId: f.companyId, title: "Other task" }).returning();
|
||||
await db.update(environmentLeases).set({ issueId: other.id }).where(eq(environmentLeases.id, previous.id));
|
||||
}
|
||||
if (fault === "agent") {
|
||||
const [other] = await db.insert(agents).values({ companyId: f.companyId, name: "Other agent", adapterType: "paperclip_runner" }).returning();
|
||||
await db.update(heartbeatRuns).set({ agentId: other.id }).where(eq(heartbeatRuns.id, f.priorRunId));
|
||||
}
|
||||
if (fault === "unfinalized") {
|
||||
const prior = await run(f.priorRunId), reference = prior.runnerProfileJson!.nativeWorkspaceSync as Record<string, unknown>;
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { ...prior.runnerProfileJson, nativeWorkspaceSync: { ...reference, state: "prepared", finalHostSha256: null } } }).where(eq(heartbeatRuns.id, prior.id));
|
||||
}
|
||||
if (fault === "owner") await db.update(nativeRunFinalizations).set({ leaseOwner: randomUUID() }).where(eq(nativeRunFinalizations.runId, f.priorRunId));
|
||||
if (fault === "missing_proof") {
|
||||
const metadata = { ...previous.metadata }; delete metadata.reusableLeaseSuccessor;
|
||||
await db.update(environmentLeases).set({ metadata }).where(eq(environmentLeases.id, previous.id));
|
||||
}
|
||||
if (fault === "conflicting_proof") await db.update(environmentLeases).set({ metadata: { ...previous.metadata,
|
||||
reusableLeaseSuccessor: { ...(previous.metadata!.reusableLeaseSuccessor as Record<string, unknown>), successorRunId: randomUUID() } } }).where(eq(environmentLeases.id, previous.id));
|
||||
if (fault === "dangling_proof") await db.update(environmentLeases).set({ metadata: { ...previous.metadata, reusableLeasePredecessor: {} } }).where(eq(environmentLeases.id, previous.id));
|
||||
if (fault === "extra_history") await db.insert(environmentLeases).values({ companyId: f.companyId, environmentId: previous.environmentId,
|
||||
heartbeatRunId: f.priorRunId, issueId: f.issueId, provider: "daytona", providerLeaseId: f.providerLeaseId,
|
||||
status: "expired", leasePolicy: "reuse_by_environment", cleanupStatus: "success", metadata: previous.metadata });
|
||||
try {
|
||||
expect(await inspectWorkspaceSource(db, { companyId: f.companyId, issueId: f.issueId, runId: f.runId, actionId: f.actionId }))
|
||||
.toMatchObject({ status: "blocked", failureCode: "allocation_ownership_ambiguous" });
|
||||
await expect(readFile(path.join(f.host, "after-warm-stop.txt"))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
} finally { adapterExecutionControls.get(f.priorRunId)?.finish(); adapterExecutionControls.delete(f.priorRunId); }
|
||||
}, 30_000);
|
||||
|
||||
it("keeps server handoff receipts across metadata updates and same-run reacquisition", async () => {
|
||||
const f = await warmHandoffSource(1, async lease => {
|
||||
const proof = lease.metadata!.reusableLeasePredecessor;
|
||||
expect(proof).toMatchObject({ schema: "paperclip.reusable-lease-handoff.v1", successorLeaseId: lease.id });
|
||||
const forged = { schema: "forged-provider-metadata" };
|
||||
const updated = await environmentService(db).updateLeaseMetadata(lease.id, { ...lease.metadata,
|
||||
reusableLeasePredecessor: forged, reusableLeaseSuccessor: forged, marker: "updated" });
|
||||
expect(updated!.metadata?.reusableLeasePredecessor).toEqual(proof);
|
||||
expect(updated!.metadata?.reusableLeaseSuccessor).toBeUndefined();
|
||||
await environmentService(db).releaseLease(lease.id, "released", { cleanupStatus: "success" });
|
||||
const reacquired = await environmentService(db).acquireLease({ companyId: lease.companyId, environmentId: lease.environmentId!,
|
||||
issueId: lease.issueId, heartbeatRunId: lease.heartbeatRunId, provider: lease.provider, providerLeaseId: lease.providerLeaseId,
|
||||
leasePolicy: "reuse_by_environment", reusesReusableLeaseId: lease.id,
|
||||
metadata: { ...updated!.metadata, reusableLeasePredecessor: forged, reusableLeaseSuccessor: forged } });
|
||||
expect(reacquired.id).toBe(lease.id);
|
||||
expect(reacquired.metadata?.reusableLeasePredecessor).toEqual(proof);
|
||||
expect(reacquired.metadata?.reusableLeaseSuccessor).toBeUndefined();
|
||||
});
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
|
||||
}, 30_000);
|
||||
|
||||
it("does not manufacture handoff authority from injected fresh or orphan lease metadata", async () => {
|
||||
const f = await warmHandoffSource(), source = await run(f.runId);
|
||||
const forged = { schema: "paperclip.reusable-lease-handoff.v1", successorLeaseId: f.lease.id };
|
||||
const values = { companyId: f.companyId, environmentId: f.lease.environmentId!, issueId: f.issueId,
|
||||
heartbeatRunId: source.id, provider: "daytona", providerLeaseId: randomUUID(),
|
||||
metadata: { agentId: source.agentId, reusableLeasePredecessor: forged, reusableLeaseSuccessor: forged } };
|
||||
const fresh = await environmentService(db).acquireLease(values);
|
||||
const orphan = await environmentService(db).insertPendingCleanupLease({ ...values, providerLeaseId: randomUUID(), failureReason: "test" });
|
||||
for (const lease of [fresh, orphan]) {
|
||||
expect(lease.metadata?.reusableLeasePredecessor).toBeUndefined();
|
||||
expect(lease.metadata?.reusableLeaseSuccessor).toBeUndefined();
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
it("rolls back both handoff links and keeps provider mismatches or retained sources fenced", async () => {
|
||||
const f = await warmHandoffSource(), source = await run(f.runId), service = environmentService(db);
|
||||
const values = { companyId: f.companyId, environmentId: f.lease.environmentId!, issueId: f.issueId,
|
||||
heartbeatRunId: f.priorRunId, provider: "daytona", providerLeaseId: randomUUID(),
|
||||
leasePolicy: "reuse_by_environment" as const, metadata: { agentId: source.agentId } };
|
||||
const prior = await service.acquireLease(values);
|
||||
await service.releaseLease(prior.id, "released", { cleanupStatus: "success" });
|
||||
const [before] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, prior.id));
|
||||
const rollback = new Error("rollback_handoff");
|
||||
await expect(db.transaction(async tx => {
|
||||
const acquired = await environmentService(tx as unknown as typeof db).acquireLease({ ...values, heartbeatRunId: f.runId, replacesReusableLeaseId: prior.id });
|
||||
expect(acquired.metadata?.reusableLeasePredecessor).toMatchObject({ previousLeaseId: prior.id, successorLeaseId: acquired.id });
|
||||
throw rollback;
|
||||
})).rejects.toBe(rollback);
|
||||
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, prior.id)))[0]).toEqual(before);
|
||||
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.providerLeaseId, values.providerLeaseId))).toHaveLength(1);
|
||||
await expect(service.acquireLease({ ...values, heartbeatRunId: f.runId, provider: "foreign-provider", replacesReusableLeaseId: prior.id })).rejects.toThrow("ownership changed");
|
||||
await expect(service.acquireLease({ ...values, providerLeaseId: f.providerLeaseId, replacesReusableLeaseId: f.lease.id })).rejects.toThrow("ownership changed");
|
||||
const [later] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId, issueId: f.issueId,
|
||||
nativeIssueId: f.issueId, runtimeMode: "native", status: "succeeded", finishedAt: new Date() }).returning();
|
||||
await service.acquireLease({ ...values, heartbeatRunId: later.id, providerLeaseId: f.providerLeaseId });
|
||||
expect(await inspectWorkspaceSource(db, { companyId: f.companyId, issueId: f.issueId, runId: f.runId, actionId: f.actionId }))
|
||||
.toMatchObject({ status: "blocked", failureCode: "allocation_ownership_ambiguous" });
|
||||
}, 30_000);
|
||||
|
||||
});
|
||||
|
||||
it("unblocks a pending workspace selection only after a superseded retry's real files are preserved", async () => {
|
||||
const f = await seed();
|
||||
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
const source = await run(f.runId);
|
||||
await db.update(heartbeatRuns).set({ resultJson: null, nativePhase: "retryable_failure",
|
||||
startedAt: new Date(1), finishedAt: new Date(2) }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.update(nativeRunFinalizations).set({ phase: "retryable_failure", attempt: 1,
|
||||
recoveryState: "resuming_session", nextAttemptAt: new Date(0) }).where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
await db.update(environmentLeases).set({ status: "active", leasePolicy: "ephemeral",
|
||||
releasedAt: null, cleanupStatus: null, metadata: f.lease.metadata }).where(eq(environmentLeases.id, f.lease.id));
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId,
|
||||
issueId: f.issueId, nativeIssueId: f.issueId, runtimeMode: "native", status: "succeeded", nativePhase: "committed",
|
||||
startedAt: new Date(3), finishedAt: new Date(4), contextSnapshot: { previousRunId: f.runId, taskRetryRequestId: randomUUID() } });
|
||||
await db.insert(executionWorkspaces).values({ id: f.issueId, companyId: f.companyId, sourceIssueId: f.issueId,
|
||||
name: "Shared files", mode: "shared_workspace", strategyType: "project_primary", cwd: f.host });
|
||||
const actor = { type: "board" as const, source: "local_implicit" as const, isInstanceAdmin: true };
|
||||
const workspaces = executionWorkspaceService(db);
|
||||
await workspaces.selectTaskWorkspace({ companyId: f.companyId, issueId: f.issueId, actor,
|
||||
selection: { kind: "existing", workspaceId: f.issueId }, expectedBindingRevision: 0, requestKey: randomUUID() });
|
||||
const admission = { companyId: f.companyId, issueId: f.issueId, actor, runId: randomUUID() };
|
||||
await expect(workspaces.applyPendingTaskWorkspaceSelection(admission)).rejects.toThrow("Previous task run still owns workspace recovery");
|
||||
expect(await claimNativeSessionResumptions({ db, runnerInstanceId: "recovery", runIds: [f.runId] })).toEqual([]);
|
||||
await expect(workspaces.applyPendingTaskWorkspaceSelection(admission)).rejects.toThrow("Previous task run still owns workspace recovery");
|
||||
const [preservation] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
||||
await recoverRetainedWorkspace({ ...f.input, actionId: preservation.id });
|
||||
expect(readSystemRecovery((await action(preservation.id)).evidence)?.state).toBe("complete");
|
||||
expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(await readFile(path.join(f.host, "newer.txt"), "utf8")).toBe("newer local work\n");
|
||||
expect(await workspaces.applyPendingTaskWorkspaceSelection(admission)).toBe(true);
|
||||
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
||||
expect(task).toMatchObject({ status: "blocked", assigneeAgentId: source.agentId, executionWorkspaceId: f.issueId,
|
||||
workspacePendingSelection: null, executionWorkspaceSettings: { mode: "shared_workspace" } });
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(2);
|
||||
expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.companyId, f.companyId))).toHaveLength(0);
|
||||
}, 30_000);
|
||||
|
||||
it.each(["HTTP Retry", "manual admission Retry", "new comment", "automatic cleanup"] as const)("repairs a retained failure through %s without admitting work before verified stop", async entry => {
|
||||
const f = await seed("native", true, "keep_running");
|
||||
const source = await run(f.runId);
|
||||
const manual = entry === "manual admission Retry";
|
||||
await db.update(heartbeatRuns).set({ resultJson: null,
|
||||
errorCode: manual ? "native_restart_recovery_blocked" : "execution_finalization_deadline_exceeded",
|
||||
...(manual ? { processPid: 1386, nativePhase: "terminal_failure" } : {}) })
|
||||
.where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.update(environmentLeases).set({ status: "active", leasePolicy: "reuse_by_environment",
|
||||
releasedAt: null, cleanupStatus: null, metadata: f.lease.metadata }).where(eq(environmentLeases.id, f.lease.id));
|
||||
if (manual) {
|
||||
const systemRecovery = failSystemRecovery(reserveSystemRecoveryAttempt(createSystemRecovery(f.runId))!,
|
||||
new SystemRecoveryFailure("allocation_ownership_ambiguous", "user_action_required"));
|
||||
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", resultId: null,
|
||||
failureCode: "native_restart_recovery_blocked", failureDetail: { recoveryMode: "exact_checkpoint_resume", systemRecovery } })
|
||||
.where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
await db.update(issues).set({ executionRunId: f.runId, checkoutRunId: f.runId }).where(eq(issues.id, f.issueId));
|
||||
await db.update(issueRecoveryActions).set({ status: "active", outcome: "blocked", cause: "native_restart_recovery_blocked",
|
||||
fingerprint: `native-restart:${f.runId}`, returnOwnerAgentId: source.agentId, wakePolicy: null, maxAttempts: 3,
|
||||
evidence: { runId: f.runId, automaticAdmissionExhausted: true, executorStopEvidence: "unverified",
|
||||
providerPid: 1386, checkpointPreserved: true },
|
||||
nextAction: "Automatic admission checks exhausted; original executor stop and state remain unverified" })
|
||||
.where(eq(issueRecoveryActions.id, f.actionId));
|
||||
} else {
|
||||
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "blocked",
|
||||
cause: "execution_finalization_deadline_exceeded", evidence: { runId: f.runId, providerOwnership: "unverified",
|
||||
automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } } }).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
}
|
||||
// Leave admission observable without substituting a fake model turn.
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId, status: "running" });
|
||||
let releaseStop!: () => void;
|
||||
const stopped = new Promise<void>(resolve => { releaseStop = resolve; });
|
||||
f.runtime.retryPendingSandboxTeardown.mockImplementation(async () => {
|
||||
await stopped;
|
||||
return { providerLeaseId: f.providerLeaseId, state: "stopped" };
|
||||
});
|
||||
const errors = vi.fn(), heartbeat = heartbeatService(db, { environmentRuntime: f.environmentRuntime });
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: errors });
|
||||
const worker = registerWorkspaceRecoveryWork(db, coordinator, f.environmentRuntime, heartbeat.resumeRemoteStopComments);
|
||||
const successors = () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
||||
try {
|
||||
await worker.ready;
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => { req.actor = { type: "board", source: "local_implicit", userId: "board" }; next(); });
|
||||
app.use("/api", issueRoutes(db, {} as Parameters<typeof issueRoutes>[1]));
|
||||
app.use(errorHandler);
|
||||
let requestId: string | undefined;
|
||||
let commentId: string | undefined;
|
||||
if (manual) expect((await request(app).get(`/api/issues/${f.issueId}`).expect(200)).body.interruption)
|
||||
.toMatchObject({ state: "failed", canRetry: true, observedRunId: f.runId });
|
||||
if (entry === "HTTP Retry" || manual) {
|
||||
const body = { idempotencyKey: randomUUID(), observedRunId: f.runId };
|
||||
const retry = await request(app).post(`/api/issues/${f.issueId}/retry`).send(body).expect(202);
|
||||
requestId = retry.body.requestId;
|
||||
expect((await request(app).post(`/api/issues/${f.issueId}/retry`).send(body).expect(202)).body.requestId).toBe(requestId);
|
||||
} else if (entry === "new comment") {
|
||||
const response = await request(app).post(`/api/issues/${f.issueId}/comments`)
|
||||
.send({ body: "Is this fixed?" }).expect(201);
|
||||
commentId = response.body.id;
|
||||
expect(await successors()).toHaveLength(0);
|
||||
} else {
|
||||
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
authorType: "user", authorUserId: "board", body: "Is this fixed?" }).returning();
|
||||
commentId = comment.id;
|
||||
const wake = { source: "automation" as const, triggerDetail: "system" as const,
|
||||
reason: "issue_reopened_via_comment", requestedByActorType: "user" as const, requestedByActorId: "board",
|
||||
payload: { issueId: f.issueId, commentId: comment.id, mutation: "comment" },
|
||||
contextSnapshot: { source: "issue.comment.reopen", issueId: f.issueId, wakeCommentId: comment.id },
|
||||
};
|
||||
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: source.agentId,
|
||||
source: wake.source, triggerDetail: wake.triggerDetail, reason: wake.reason,
|
||||
requestedByActorType: "user", requestedByActorId: "board", status: "deferred_issue_execution",
|
||||
payload: { ...wake.payload, _paperclipWakeContext: { ...wake.contextSnapshot, wakeReason: wake.reason,
|
||||
wakeSource: wake.source, wakeTriggerDetail: wake.triggerDetail, wakeCommentIds: [comment.id] } },
|
||||
});
|
||||
expect(await successors()).toHaveLength(0);
|
||||
if (entry === "automatic cleanup") await heartbeat.sweepOrphanedActiveLeases({ backoffMs: 0 });
|
||||
}
|
||||
await expect.poll(() => f.runtime.retryPendingSandboxTeardown.mock.calls.length, { timeout: 20_000 }).toBe(1);
|
||||
if (!manual) {
|
||||
if (requestId) await heartbeat.resumeTaskRetry(f.companyId, requestId);
|
||||
else await heartbeat.resumeQueuedRuns();
|
||||
}
|
||||
expect(await successors()).toHaveLength(0);
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(false);
|
||||
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0].metadata)
|
||||
.not.toHaveProperty("remoteExecutionTermination");
|
||||
if (manual) expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0])
|
||||
.toMatchObject({ executionRunId: f.runId, checkoutRunId: f.runId });
|
||||
await expect(readFile(path.join(f.host, "saved.txt"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
|
||||
releaseStop();
|
||||
await expect.poll(async () => (await successors()).length, { timeout: 20_000 }).toBe(1);
|
||||
const [successor] = await successors();
|
||||
if (commentId) expect(successor.contextSnapshot?.wakeCommentIds).toContain(commentId);
|
||||
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(true);
|
||||
expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(await readFile(path.join(f.host, "newer.txt"), "utf8")).toBe("newer local work\n");
|
||||
if (requestId) expect(await readFreshWorkspaceReadiness(db, successor)).toMatchObject({ mode: "resume_preserved_workspace" });
|
||||
else expect(successor.contextSnapshot?.previousRunId).toBe(f.runId);
|
||||
await Promise.all([requestId ? heartbeat.resumeTaskRetry(f.companyId, requestId) : heartbeat.resumeQueuedRuns(),
|
||||
heartbeat.resumeRemoteStopComments(await run(f.runId))]);
|
||||
expect(await successors()).toHaveLength(1);
|
||||
expect(f.runtime.resumeRunLease).toHaveBeenCalledOnce();
|
||||
expect((await run(f.runId)).status).toBe("failed");
|
||||
const reloaded = await request(app).get(`/api/issues/${f.issueId}/recovery-actions`).expect(200);
|
||||
expect(reloaded.body.active).toBeNull();
|
||||
const records = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
||||
expect(records.some(record => readSystemRecovery(record.evidence)?.state === "complete")).toBe(true);
|
||||
expect(errors).not.toHaveBeenCalled();
|
||||
} finally { releaseStop(); await coordinator.stop(); await worker.stop(); }
|
||||
}, 40_000);
|
||||
|
||||
it("wakes physical preservation from HTTP Retry before passive history without replaying the provider", async () => {
|
||||
for (let index = 0; index < 5; index++) {
|
||||
const old = await seed();
|
||||
await recoverRetainedWorkspace(old.input);
|
||||
await db.update(issueRecoveryActions).set({
|
||||
id: `00000000-000${index}-4000-8000-${randomUUID().slice(-12)}`,
|
||||
}).where(eq(issueRecoveryActions.id, old.actionId));
|
||||
}
|
||||
const f = await seed();
|
||||
await adoptSystemRecovery(db, f);
|
||||
const before = await action(f.actionId);
|
||||
const exhausted = failSystemRecovery(readSystemRecovery(before.evidence)!,
|
||||
new SystemRecoveryFailure("source_binding_invalid", "service_required"));
|
||||
await db.update(issueRecoveryActions).set({ cause: "native_workspace_sync_out_retry_exhausted",
|
||||
evidence: { ...before.evidence, systemRecovery: exhausted } })
|
||||
.where(eq(issueRecoveryActions.id, f.actionId));
|
||||
const errors = vi.fn();
|
||||
const competingCleanup = await retainedWorkspaceCandidates(db, undefined, false, true);
|
||||
expect(competingCleanup).toHaveLength(5);
|
||||
expect(competingCleanup.every(row => row.action.id.startsWith("00000000"))).toBe(true);
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: errors });
|
||||
const recovered = vi.fn(async () => {});
|
||||
const worker = registerWorkspaceRecoveryWork(db, coordinator, f.environmentRuntime, recovered);
|
||||
try {
|
||||
await worker.ready;
|
||||
await db.update(agents).set({ status: "paused" }).where(eq(agents.companyId, f.companyId));
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => { req.actor = { type: "board", source: "local_implicit", userId: "board" }; next(); });
|
||||
app.use("/api", issueRoutes(db, {} as Parameters<typeof issueRoutes>[1]));
|
||||
app.use(errorHandler);
|
||||
const body = { idempotencyKey: randomUUID(), observedRunId: f.runId };
|
||||
const retry = await request(app).post(`/api/issues/${f.issueId}/retry`).send(body).expect(202);
|
||||
const repeated = await request(app).post(`/api/issues/${f.issueId}/retry`).send(body).expect(202);
|
||||
const coalesced = await request(app).post(`/api/issues/${f.issueId}/retry`)
|
||||
.send({ ...body, idempotencyKey: randomUUID() }).expect(202);
|
||||
expect(repeated.body.requestId).toBe(retry.body.requestId);
|
||||
expect(coalesced.body.requestId).toBe(retry.body.requestId);
|
||||
await expect.poll(async () => readSystemRecovery((await action(f.actionId)).evidence)?.state,
|
||||
{ timeout: 20_000 }).toBe("complete");
|
||||
expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
|
||||
expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.companyId, f.companyId))).toHaveLength(0);
|
||||
const reloaded = await request(app).get(`/api/issues/${f.issueId}/recovery-actions`).expect(200);
|
||||
expect(JSON.stringify(reloaded.body)).toContain('"state":"complete"');
|
||||
expect(errors).not.toHaveBeenCalled();
|
||||
} finally { await coordinator.stop(); await worker.stop(); }
|
||||
}, 30_000);
|
||||
|
||||
it("hands an exhausted system export repair to the board instead of leaving it system-owned", async () => {
|
||||
// Regression: the bounded export budget ran out, but the action stayed active and
|
||||
// system-owned for hours. The board saw no owner, and replies started nothing.
|
||||
const f = await seed();
|
||||
await adoptSystemRecovery(db, f);
|
||||
const before = await action(f.actionId);
|
||||
const adopted = readSystemRecovery(before.evidence)!;
|
||||
const now = Date.now();
|
||||
await db.update(issueRecoveryActions).set({ cause: "native_workspace_sync_out_retry_exhausted",
|
||||
nextAction: "Finalization retry budget exhausted. Retry workspace export and merge from the retained sandbox; do not submit another provider turn.",
|
||||
evidence: { ...before.evidence, systemRecovery: { ...adopted, revision: adopted.revision + 1, phase: "preserve_files",
|
||||
state: "scheduled", attempts: 6, retriesUsed: 5, firstFailureAt: now - 3_600_000, deadlineAt: now - 1_000, nextAttemptAt: null,
|
||||
failure: { code: "export_interrupted", retryClass: "retry_transient", evidenceFingerprint: "f".repeat(64) } } } })
|
||||
.where(eq(issueRecoveryActions.id, f.actionId));
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const exhausted = await action(f.actionId);
|
||||
expect(readSystemRecovery(exhausted.evidence)?.state).toBe("exhausted");
|
||||
expect(exhausted).toMatchObject({ status: "active", ownerType: "board" });
|
||||
expect(exhausted.nextAction).toMatch(/stopped after 6 attempts/);
|
||||
const blocker = await getExecutionBlocker(db, f.companyId, f.issueId);
|
||||
expect(blocker?.workspaceRecovery).toMatchObject({ status: "blocked", canRetry: true });
|
||||
expect(blocker?.nextAction).toMatch(/Retry recovery/);
|
||||
expect(blocker?.nextAction).not.toMatch(/send a (new )?message/i);
|
||||
// Nothing physical or provider-facing ran after the budget was spent.
|
||||
expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
|
||||
// The board's Retry starts one new bounded system generation.
|
||||
await requestWorkspaceRecovery({ ...f.input, idempotencyKey: randomUUID(), actorId: "board" });
|
||||
const retried = await action(f.actionId);
|
||||
expect(retried.ownerType).toBe("system");
|
||||
expect(readSystemRecovery(retried.evidence)).toMatchObject({ state: "scheduled", attempts: 0 });
|
||||
}, 30_000);
|
||||
|
||||
it("hands a repair that was already exhausted and system-owned to the board without another attempt", async () => {
|
||||
const f = await seed();
|
||||
await adoptSystemRecovery(db, f);
|
||||
const before = await action(f.actionId);
|
||||
const spent = failSystemRecovery({ ...readSystemRecovery(before.evidence)!, attempts: 6, retriesUsed: 5 },
|
||||
new SystemRecoveryFailure("export_interrupted", "retry_transient"));
|
||||
expect(spent.state).toBe("exhausted");
|
||||
await db.update(issueRecoveryActions).set({ cause: "native_workspace_sync_out_retry_exhausted", ownerType: "system",
|
||||
evidence: { ...before.evidence, systemRecovery: spent,
|
||||
workspaceRecovery: { ...readWorkspaceRecovery(before.evidence)!, status: "blocked", attempt: 6, retryable: true } } })
|
||||
.where(eq(issueRecoveryActions.id, f.actionId));
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const handed = await action(f.actionId);
|
||||
expect(handed).toMatchObject({ status: "active", ownerType: "board" });
|
||||
expect(readSystemRecovery(handed.evidence)).toEqual(spent);
|
||||
expect((await getExecutionBlocker(db, f.companyId, f.issueId))?.nextAction).toMatch(/Retry recovery/);
|
||||
expect(f.runtime.resumeRunLease).not.toHaveBeenCalled();
|
||||
// Repeat sweeps are idempotent and still grant no attempt.
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)).toEqual(spent);
|
||||
}, 30_000);
|
||||
|
||||
it("reconstructs a resolved files-only preservation obligation at startup without replay", async () => {
|
||||
const f = await seed("legacy");
|
||||
await adoptSystemRecovery(db, f);
|
||||
const before = await action(f.actionId);
|
||||
const automaticRecovery = { policy: "preserve_without_replay_v1", runId: f.runId, replay: "blocked" };
|
||||
await db.update(issueRecoveryActions).set({ status: "resolved", outcome: "blocked",
|
||||
evidence: { ...before.evidence, automaticRecovery } }).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
const errors = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: errors });
|
||||
const worker = registerWorkspaceRecoveryWork(db, coordinator, f.environmentRuntime, async () => {});
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(async () => readSystemRecovery((await action(f.actionId)).evidence)?.state,
|
||||
{ timeout: 20_000 }).toBe("complete");
|
||||
expect((await action(f.actionId)).evidence.automaticRecovery).toEqual(automaticRecovery);
|
||||
expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0);
|
||||
expect(errors).not.toHaveBeenCalled();
|
||||
} finally { await coordinator.stop(); await worker.stop(); }
|
||||
}, 30_000);
|
||||
|
||||
it("fills a free export slot across the candidate cursor while another export is held and joins both at shutdown", async () => {
|
||||
const orderedSeed = async (position: number) => {
|
||||
const fixture = await seed();
|
||||
const id = `00000002-000${position}-4000-8000-${randomUUID().slice(-12)}` as const;
|
||||
await db.update(issueRecoveryActions).set({ id }).where(eq(issueRecoveryActions.id, fixture.actionId));
|
||||
fixture.actionId = id; fixture.input.actionId = id;
|
||||
return fixture;
|
||||
};
|
||||
const held = await orderedSeed(2);
|
||||
const fixtures = [held];
|
||||
const runtimeFor = (runId: string) => fixtures.find(fixture => fixture.runId === runId)!.environmentRuntime;
|
||||
const runtime = { ...held.environmentRuntime,
|
||||
resumeWorkspaceRecoveryLease: (input: Parameters<typeof held.environmentRuntime.resumeWorkspaceRecoveryLease>[0]) =>
|
||||
runtimeFor(input.source.runId).resumeWorkspaceRecoveryLease(input),
|
||||
retryPendingSandboxTeardown: (input: Parameters<typeof held.environmentRuntime.retryPendingSandboxTeardown>[0]) =>
|
||||
runtimeFor(input.lease.heartbeatRunId!).retryPendingSandboxTeardown(input),
|
||||
};
|
||||
let active = 0, maximum = 0, release!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const countExport = (fixture: typeof held, pause = false) => {
|
||||
const resume = fixture.runtime.resumeRunLease.getMockImplementation()!;
|
||||
fixture.runtime.resumeRunLease.mockImplementation(async input => {
|
||||
active += 1; maximum = Math.max(maximum, active);
|
||||
try { if (pause) await gate; return await resume(input); } finally { active -= 1; }
|
||||
});
|
||||
};
|
||||
countExport(held, true);
|
||||
const errors = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: errors });
|
||||
const worker = registerWorkspaceRecoveryWork(db, coordinator, runtime, async () => {});
|
||||
try {
|
||||
await expect.poll(() => held.runtime.resumeRunLease.mock.calls.length, { timeout: 20_000 }).toBe(1);
|
||||
// Start the middle candidate only after the highest candidate occupies
|
||||
// one slot. The next producer then sorts before that saved cursor.
|
||||
const fast = await orderedSeed(1); fixtures.push(fast); countExport(fast);
|
||||
await requestTaskRetry({ db, companyId: fast.companyId, issueId: fast.issueId, actorId: "board",
|
||||
observedRunId: fast.runId, idempotencyKey: randomUUID() });
|
||||
await expect.poll(async () => readSystemRecovery((await action(fast.actionId)).evidence)?.state,
|
||||
{ timeout: 20_000 }).toBe("complete");
|
||||
const third = await orderedSeed(0); fixtures.push(third); countExport(third);
|
||||
await requestTaskRetry({ db, companyId: third.companyId, issueId: third.issueId, actorId: "board",
|
||||
observedRunId: third.runId, idempotencyKey: randomUUID() });
|
||||
await expect.poll(async () => readSystemRecovery((await action(third.actionId)).evidence)?.state,
|
||||
{ timeout: 20_000 }).toBe("complete");
|
||||
expect(readSystemRecovery((await action(held.actionId)).evidence)?.state).toBe("running");
|
||||
expect(maximum).toBeLessThanOrEqual(2);
|
||||
expect(await readFile(path.join(third.host, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, third.companyId))).toHaveLength(1);
|
||||
let stopped = false;
|
||||
const stopping = Promise.all([coordinator.stop(), worker.stop()]).then(() => { stopped = true; });
|
||||
await Promise.resolve(); expect(stopped).toBe(false);
|
||||
release(); await stopping;
|
||||
expect(readSystemRecovery((await action(held.actionId)).evidence)?.state).toBe("complete");
|
||||
expect(errors).not.toHaveBeenCalled();
|
||||
} finally { release(); await coordinator.stop(); await worker.stop(); }
|
||||
}, 60_000);
|
||||
|
||||
it("rotates past due sources whose physical locks belong to other workers", async () => {
|
||||
const blocked = [await seed(), await seed()], available = await seed();
|
||||
for (const [index, fixture] of [...blocked, available].entries()) {
|
||||
await adoptSystemRecovery(db, fixture);
|
||||
const id = `00000001-000${index}-4000-8000-${randomUUID().slice(-12)}` as const;
|
||||
await db.update(issueRecoveryActions).set({ id }).where(eq(issueRecoveryActions.id, fixture.actionId));
|
||||
fixture.actionId = id; fixture.input.actionId = id;
|
||||
}
|
||||
let unlock!: () => void;
|
||||
const gate = new Promise<void>(resolve => { unlock = resolve; });
|
||||
const held = blocked.map(fixture => {
|
||||
let ready!: () => void;
|
||||
const acquired = new Promise<void>(resolve => { ready = resolve; });
|
||||
const lock = db.transaction(async tx => {
|
||||
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${nativeWorkspaceFinalizationLockKey(fixture.companyId, fixture.runId)}, 0))`);
|
||||
ready(); await gate;
|
||||
});
|
||||
return { acquired, lock };
|
||||
});
|
||||
try {
|
||||
await Promise.all(held.map(item => item.acquired));
|
||||
await Promise.all(await reconcileRetainedWorkspaces(db, available.environmentRuntime, async () => {}, true) ?? []);
|
||||
expect(blocked.every(fixture => fixture.runtime.resumeRunLease.mock.calls.length === 0)).toBe(true);
|
||||
await Promise.all(await reconcileRetainedWorkspaces(db, available.environmentRuntime, async () => {}, true) ?? []);
|
||||
expect(readSystemRecovery((await action(available.actionId)).evidence)?.state).toBe("complete");
|
||||
expect(await readFile(path.join(available.host, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(blocked.every(fixture => fixture.runtime.resumeRunLease.mock.calls.length === 0)).toBe(true);
|
||||
} finally { unlock(); await Promise.all(held.map(item => item.lock)); }
|
||||
}, 30_000);
|
||||
|
||||
it("coalesces Retry while preservation owns its physical operation", async () => {
|
||||
const f = await seed();
|
||||
await adoptSystemRecovery(db, f);
|
||||
const resume = f.runtime.resumeRunLease.getMockImplementation()!;
|
||||
let release!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
f.runtime.resumeRunLease.mockImplementation(async input => { await gate; return resume(input); });
|
||||
const preserving = recoverRetainedWorkspace(f.input);
|
||||
let requestId: string | null | undefined;
|
||||
try {
|
||||
await expect.poll(() => f.runtime.resumeRunLease.mock.calls.length).toBe(1);
|
||||
const request = { db, companyId: f.companyId, issueId: f.issueId, actorId: "board",
|
||||
idempotencyKey: randomUUID(), observedRunId: f.runId };
|
||||
const first = await requestTaskRetry(request);
|
||||
requestId = first.requestId;
|
||||
const second = await requestTaskRetry({ ...request, idempotencyKey: randomUUID() });
|
||||
expect(second.requestId).toBe(first.requestId);
|
||||
} finally { release(); await preserving; }
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)?.intentId).toBe(requestId);
|
||||
expect(f.runtime.resumeRunLease.mock.calls.filter(([input]) => input.lease.id === f.lease.id)).toHaveLength(1);
|
||||
expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
|
||||
}, 30_000);
|
||||
|
||||
it("adopts only the exact historical infrastructure notice, preserving prose and user-action notices", async () => {
|
||||
const f = await seed("legacy", false);
|
||||
const metadata = { version: 1 as const, sourceRunId: f.runId,
|
||||
sections: [{ rows: [{ type: "key_value" as const, label: "Recovery action", value: f.actionId }] }] };
|
||||
const base = { companyId: f.companyId, issueId: f.issueId, body: "Original technical history",
|
||||
authorType: "system" as const, presentation: { kind: "system_notice" as const, tone: "danger" as const,
|
||||
title: "Recovery needs attention", detailsDefaultOpen: false }, metadata };
|
||||
const comments = await db.insert(issueComments).values([
|
||||
base,
|
||||
{ ...base, authorType: "user", authorUserId: "board" },
|
||||
{ ...base, metadata: { ...metadata, sourceRunId: randomUUID() } },
|
||||
{ ...base, metadata: { ...metadata, recoveryResponsibility: "user" as const } },
|
||||
]).returning();
|
||||
await adoptSystemRecovery(db, f);
|
||||
await adoptSystemRecovery(db, f);
|
||||
const rows = await db.select().from(issueComments).where(eq(issueComments.companyId, f.companyId));
|
||||
expect(rows.find(row => row.id === comments[0].id)?.metadata?.recoveryResponsibility).toBe("system");
|
||||
for (const original of comments.slice(1)) {
|
||||
expect(rows.find(row => row.id === original.id)?.metadata).toEqual(original.metadata);
|
||||
}
|
||||
expect(rows.every(row => row.body === base.body)).toBe(true);
|
||||
});
|
||||
|
||||
});
|
||||
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user