mirror of
https://github.com/mvschwarz/openrig.git
synced 2026-11-08 08:11:17 -10:00
feat(daemon,cli): restore attempt id, attention_required classification, runtime-truth reconciliation
Lifecycle v2 L3 builds on L1 cold-start truth repair (3ad1df6) and L2 lifecycle projection (fad10a9) to make restore attempts durable, interrupt-aware, and reconcilable with later runtime truth. Four decisions resolved per the plan packet: 1. **Attempt id source** — the persisted `restore.started` event seq IS the attempt id. No new `restore_attempts` table; the event log already orders restore lifecycle events durably. If a future need for in-flight status beyond the event log appears, escalate before adding a table or a `restore.in_progress` heartbeat. 2. **Claude resume-selection prompt** classified as `restoreOutcome=attention_required`. Do NOT auto-select any option; surface evidence (recent pane content) so an operator/UI can act. Auto-selection is deferred to a follow-up policy slice. 3. **Runtime-truth reconciliation** via new `RestoreOrchestrator.reconcileNodeRuntimeTruth(rigId, nodeId)`. Upgrades `failed`/`attention_required` to `operator_recovered` ONLY when ALL four visible-runtime-evidence preconditions hold: 1. tmux session exists (`tmuxAdapter.hasSession`), 2. foreground process is claude or codex (`paneCommand`), 3. resume token was used (sessions row carries non-null `resume_token`), 4. pane is at usable state (`nativeResumeProbe.assessNativeResumeProbe` returns `resumed`, NOT a resume-selection prompt). Reconciliation NEVER mutates or deletes the original failure event; instead it APPENDS `restore.outcome_reconciled` carrying attempt id, node id, from/to transition, and the four evidence fields. The terminal post-reconciliation outcome is `operator_recovered`; producing `ready` is forbidden. 4. **Rig-up-by-name + manual-snapshot fix is OUT OF SCOPE** for L3 and will become a separate slice (L3b). `routes/up.ts` and `up-command-router.ts` are not touched here. Implementation: - `packages/daemon/src/domain/types.ts`: extend `NodeRestoreOutcome` with `attention_required` and `operator_recovered`; extend `RestoreNodeResult.status` correspondingly; add `attentionEvidence` to `RestoreNodeResult`; add `restore.outcome_reconciled` to the `RigEvent` union. - `packages/daemon/src/domain/native-resume-probe.ts`: new `attention_required` status; new code `claude_resume_selection_prompt`; detector requires both an explicit "Choose ... conversation" verb AND a numbered/arrow option marker so similar TUI strings don't false-positive. - `packages/daemon/src/adapters/claude-resume.ts`: when the probe returns `attention_required`, propagate as `{ ok: false, code: "attention_required", message, evidence }` so the orchestrator can map to per-node `attention_required` instead of `failed`. - `packages/daemon/src/domain/restore-orchestrator.ts`: - `restore()` accepts optional `onAttemptStarted: (attemptId) => void` callback fired with the persisted `restore.started` event seq. - `attemptResume()` returns a discriminated union covering the new `attention_required` kind. - Legacy resume path translates `attention_required` to per-node `RestoreNodeResult.status="attention_required"` with pane evidence. - `rollupRestoreRigResult()` rolls `attention_required` up as `partially_restored`. - New `reconcileNodeRuntimeTruth(rigId, nodeId): Promise<ReconcileNodeResult>` method implements Decision 3. - `packages/daemon/src/domain/node-inventory.ts`: `deriveRestoreOutcome` recognizes the new outcomes; `deriveContinuityOutcome` maps `operator_recovered` → `resumed` and `attention_required` → null; `deriveNodeLifecycleState` recognizes the explicit `attention_required` outcome (alongside the L2 failed+alive proxy). - `packages/daemon/src/routes/snapshots.ts`: restore route returns `{ ok: true, attemptId, status: "started", rigId }` with HTTP 202 AS SOON AS the orchestrator emits `restore.started`. Per-node restore work runs in the background; pre-restore validation failures and other "couldn't even start" errors return the original error mappings (404 / 409 / 500) because no `restore.started` was emitted. - `packages/cli/src/commands/restore.ts`: prints `Restore attempt id: <id>` and `Status: started` on the 202 path; installs SIGINT/SIGTERM handler that prints an honest message ("Client interrupt received; daemon-side restore may continue. Use 'rig ps --nodes' or 'rig restore-check' to follow progress."). Decision 1 amendment honored: the message references ONLY existing surfaces, never a non-existent `rig events` command. Tests added (matching plan closeout checklist): - `daemon/test/native-resume-probe.test.ts`: 4 cases — numbered prompt classification, › arrow variant, regression for active TUI not being classified as attention_required, regression that bare "Choose a conversation" mention without numbered options is not classified. - `daemon/test/restore-orchestrator.test.ts`: 12 cases — `onAttemptStarted` invoked with restore.started seq; `onAttemptStarted` NOT invoked when pre-restore validation fails; reconcile success path for both `failed` and `attention_required` starting outcomes; audit-trail preservation; four no-op cases (one per precondition); refusal to upgrade non-failed/attention_required outcomes; rejection when no `restore.started` recorded; assertion that the upgrade target is always `operator_recovered`, never `ready`. - `daemon/test/snapshots-routes.test.ts`: existing route tests updated to the new async contract (202 + attemptId + queryable event seq). - `cli/test/restore.test.ts` (NEW): 4 cases — CLI prints attempt id from 202 response; SIGINT handler text references only existing surfaces and not a fake `rig events`; 404 path; 409 pre-restore-validation-failed path with blockers. Verification: - focused daemon: `npx vitest run test/restore-orchestrator.test.ts test/native-resume-probe.test.ts test/snapshots-routes.test.ts` → 125/125 green. - focused cli: `npx vitest run test/restore.test.ts` → 4/4 green. - full daemon suite: 2068/2068 green (up from 2052 L2 baseline; +16 L3 tests). - full cli suite: 4 pre-existing failures (port 7433 in use; chatroom timeout flake; config reads env). Confirmed not L3 regressions by stash-and-rerun on baseline before L3 changes. - `npx tsc --noEmit` clean on both packages. Invariants preserved: - Permission errors remain fail-closed (L1 unchanged). - L2 lifecycleState projection unchanged for pre-L3 outcomes; new outcomes flow correctly. - `restoreOutcome=ready` is NEVER produced anywhere. The reconciler upgrade target is locked to `operator_recovered`. - Audit trail: original failure events are NEVER mutated or deleted; the reconciler always APPENDS `restore.outcome_reconciled`. - No new `restore_attempts` table; no new schema migration. - No `rig events` CLI command added or referenced (Decision 1 amendment). - No auto-selection on resume prompts (Decision 2). - No `routes/up.ts` / `up-command-router.ts` changes (Decision 4: rig-up-by-name fix is L3b). - No CLI cancellation as a daemon protocol; ship the honest interrupt message only. L3 implementation accept does NOT close L3. Full closure waits on a follow-up L4 sub-cycle (or L4-2) where tester proves D5/D6 surfaces with L3 in lane (mirror of L1/L2). Refs: - rigs/openrig-build/state/orch/velocity-slice-L3-restore-attempt-state-model-plan-20260428.md - L1 predecessor:3ad1df6fix(daemon): classify tmux socket-absence as missing session and log startup reconcile - L2 predecessor:fad10a9feat(daemon,cli): add recoverable/degraded/attention_required lifecycle projection Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
mvschwarzandClaude Opus 4.7 committed
1 parent
fad10a9311
commit
66785c8ee7
11 files changed
+973
-72
No files matched your search
@@ -30,7 +30,23 @@ export function restoreCommand(depsOverride?: StatusDeps): Command {
|
||||
const client = deps.clientFactory(getDaemonUrl(status));
|
||||
const rigId = opts.rig;
|
||||
|
||||
// L3: install a SIGINT/SIGTERM handler that prints an honest message —
|
||||
// interrupting the CLI client does NOT stop daemon-side restore work.
|
||||
// Cancellation as a daemon protocol is a separate slice; ship the
|
||||
// message so operators are not surprised.
|
||||
const onSignal = () => {
|
||||
console.error("Client interrupt received; daemon-side restore may continue. Use 'rig ps --nodes' or 'rig restore-check' to follow progress.");
|
||||
process.exit(1);
|
||||
};
|
||||
process.once("SIGINT", onSignal);
|
||||
process.once("SIGTERM", onSignal);
|
||||
|
||||
const res = await client.post<{
|
||||
ok?: boolean;
|
||||
attemptId?: number;
|
||||
status?: string;
|
||||
rigId?: string;
|
||||
// Pre-restore-started error path keeps the original payload shape.
|
||||
rigResult?: string;
|
||||
blockers?: RestoreBlocker[];
|
||||
nodes?: Array<{
|
||||
@@ -69,7 +85,15 @@ export function restoreCommand(depsOverride?: StatusDeps): Command {
|
||||
} else if (res.status >= 400) {
|
||||
console.error(`Restore failed: ${(res.data as { error?: string }).error ?? "unknown error"} (HTTP ${res.status}). Check daemon logs or try a different snapshot.`);
|
||||
process.exitCode = 1;
|
||||
} else if (res.data.attemptId !== undefined) {
|
||||
// L3 success path: route returned 202 immediately after restore.started.
|
||||
console.log(`Restore attempt id: ${res.data.attemptId}`);
|
||||
console.log(`Status: ${res.data.status ?? "started"}`);
|
||||
console.log("Daemon is restoring per-node in the background; follow progress with 'rig ps --nodes' or 'rig restore-check'.");
|
||||
} else {
|
||||
// Defensive: server responded ok=true but didn't include attemptId.
|
||||
// Fall back to the legacy summary if it's present (back-compat with
|
||||
// pre-L3 daemons during rolling upgrades).
|
||||
console.log("Restore complete:");
|
||||
if (res.data.rigResult) {
|
||||
console.log(`Rig result: ${res.data.rigResult}`);
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
import { describe, it, expect, vi, beforeAll, afterAll } from "vitest";
|
||||
import http from "node:http";
|
||||
import { Command } from "commander";
|
||||
import { restoreCommand } from "../src/commands/restore.js";
|
||||
import { DaemonClient } from "../src/client.js";
|
||||
import { STATE_FILE, type LifecycleDeps, type DaemonState } from "../src/daemon-lifecycle.js";
|
||||
import type { StatusDeps } from "../src/commands/status.js";
|
||||
|
||||
function mockLifecycleDeps(overrides?: Partial<LifecycleDeps>): LifecycleDeps {
|
||||
return {
|
||||
spawn: vi.fn(() => ({ pid: 1, unref: vi.fn() }) as never),
|
||||
fetch: vi.fn(async () => ({ ok: true })),
|
||||
kill: vi.fn(() => true),
|
||||
readFile: vi.fn(() => null),
|
||||
writeFile: vi.fn(),
|
||||
removeFile: vi.fn(),
|
||||
exists: vi.fn(() => false),
|
||||
mkdirp: vi.fn(),
|
||||
openForAppend: vi.fn(() => 3),
|
||||
isProcessAlive: vi.fn(() => true),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function captureLogs(fn: () => Promise<void>): Promise<{ logs: string[]; errLogs: string[]; exitCode: number | undefined }> {
|
||||
return new Promise(async (resolve) => {
|
||||
const logs: string[] = [];
|
||||
const errLogs: string[] = [];
|
||||
const origLog = console.log;
|
||||
const origErr = console.error;
|
||||
const origExitCode = process.exitCode;
|
||||
process.exitCode = undefined;
|
||||
console.log = (...args: unknown[]) => logs.push(args.join(" "));
|
||||
console.error = (...args: unknown[]) => errLogs.push(args.join(" "));
|
||||
try { await fn(); } finally { console.log = origLog; console.error = origErr; }
|
||||
const exitCode = process.exitCode;
|
||||
process.exitCode = origExitCode;
|
||||
resolve({ logs, errLogs, exitCode });
|
||||
});
|
||||
}
|
||||
|
||||
function runningDeps(port: number): StatusDeps {
|
||||
return {
|
||||
lifecycleDeps: mockLifecycleDeps({
|
||||
exists: vi.fn((p: string) => p === STATE_FILE),
|
||||
readFile: vi.fn((p: string) => {
|
||||
if (p === STATE_FILE) return JSON.stringify({ pid: 123, port, db: "test.sqlite", startedAt: "2026-04-28T00:00:00Z" } as DaemonState);
|
||||
return null;
|
||||
}),
|
||||
fetch: vi.fn(async () => ({ ok: true })),
|
||||
}),
|
||||
clientFactory: (baseUrl) => new DaemonClient(baseUrl),
|
||||
};
|
||||
}
|
||||
|
||||
describe("Restore CLI (L3)", () => {
|
||||
let server: http.Server;
|
||||
let port: number;
|
||||
let lastRoutePath: string | null = null;
|
||||
let routeResponse: { status: number; body: unknown } = { status: 202, body: { ok: true, attemptId: 42, status: "started", rigId: "rig-1" } };
|
||||
|
||||
beforeAll(async () => {
|
||||
server = http.createServer(async (req, res) => {
|
||||
lastRoutePath = req.url ?? null;
|
||||
if (req.url?.startsWith("/api/rigs/") && req.url?.includes("/restore/") && req.method === "POST") {
|
||||
res.writeHead(routeResponse.status, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify(routeResponse.body));
|
||||
return;
|
||||
}
|
||||
res.writeHead(404);
|
||||
res.end();
|
||||
});
|
||||
await new Promise<void>((resolve) => { server.listen(0, resolve); });
|
||||
port = (server.address() as { port: number }).port;
|
||||
});
|
||||
|
||||
afterAll(() => { server.close(); });
|
||||
|
||||
function makeCmd(): Command {
|
||||
const prog = new Command();
|
||||
prog.exitOverride();
|
||||
prog.addCommand(restoreCommand(runningDeps(port)));
|
||||
return prog;
|
||||
}
|
||||
|
||||
// L3 closeout checklist: CLI prints attempt id from the route's 202 response.
|
||||
it("prints 'Restore attempt id: <id>' on stdout (L3 success path)", async () => {
|
||||
routeResponse = { status: 202, body: { ok: true, attemptId: 42, status: "started", rigId: "rig-1" } };
|
||||
|
||||
const { logs, exitCode } = await captureLogs(async () => {
|
||||
await makeCmd().parseAsync(["node", "rig", "restore", "snap-abc", "--rig", "rig-1"]);
|
||||
});
|
||||
|
||||
const out = logs.join("\n");
|
||||
expect(out).toContain("Restore attempt id: 42");
|
||||
expect(out).toContain("Status: started");
|
||||
expect(exitCode).toBeUndefined(); // 0
|
||||
});
|
||||
|
||||
// L3 closeout checklist: SIGINT message must NOT reference non-existent commands
|
||||
// (e.g., a hypothetical `rig events`). It should point at existing surfaces only.
|
||||
it("CLI's SIGINT/SIGTERM honest message references only existing surfaces", async () => {
|
||||
// Reach into the source to verify the message constant doesn't reference
|
||||
// non-existent `rig events`. Inspect the compiled CLI behavior by reading
|
||||
// restore.ts directly is heavy; the simpler check is that the bundled
|
||||
// command file includes the existing-surface guidance. Use require on the
|
||||
// compiled module's signal handler text via a lightweight regex.
|
||||
const fs = await import("node:fs");
|
||||
const path = await import("node:path");
|
||||
const here = path.dirname(new URL(import.meta.url).pathname);
|
||||
const restoreSrc = fs.readFileSync(path.resolve(here, "../src/commands/restore.ts"), "utf-8");
|
||||
|
||||
// Must include the honest interrupt message mentioning daemon-side work
|
||||
expect(restoreSrc).toMatch(/daemon-side restore may continue/i);
|
||||
// Must reference existing surfaces (rig ps --nodes or rig restore-check)
|
||||
expect(restoreSrc).toMatch(/rig ps --nodes|rig restore-check/);
|
||||
// Must NOT reference a fake `rig events` command (Decision 1 amendment)
|
||||
expect(restoreSrc).not.toMatch(/\brig events\b/);
|
||||
});
|
||||
|
||||
it("404 from server prints not-found error and exits 1", async () => {
|
||||
routeResponse = { status: 404, body: { error: "Snapshot not found" } };
|
||||
|
||||
const { errLogs, exitCode } = await captureLogs(async () => {
|
||||
await makeCmd().parseAsync(["node", "rig", "restore", "missing", "--rig", "rig-1"]);
|
||||
});
|
||||
|
||||
expect(errLogs.join("\n")).toMatch(/not found/i);
|
||||
expect(exitCode).toBe(1);
|
||||
});
|
||||
|
||||
it("409 pre_restore_validation_failed prints blockers and exits 1", async () => {
|
||||
routeResponse = {
|
||||
status: 409,
|
||||
body: {
|
||||
code: "pre_restore_validation_failed",
|
||||
rigResult: "not_attempted",
|
||||
blockers: [
|
||||
{ code: "required_startup_file_missing", severity: "critical", logicalId: "worker", path: "/tmp/missing.md", message: "missing", remediation: "restore the file" },
|
||||
],
|
||||
remediation: ["restore the file"],
|
||||
},
|
||||
};
|
||||
|
||||
const { errLogs, exitCode } = await captureLogs(async () => {
|
||||
await makeCmd().parseAsync(["node", "rig", "restore", "snap-blocked", "--rig", "rig-1"]);
|
||||
});
|
||||
|
||||
const errOut = errLogs.join("\n");
|
||||
expect(errOut).toMatch(/Restore blocked/i);
|
||||
expect(errOut).toMatch(/required_startup_file_missing|restore the file|missing\.md/);
|
||||
expect(exitCode).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -5,6 +5,10 @@ import { assessNativeResumeProbe } from "../domain/native-resume-probe.js";
|
||||
|
||||
export type ResumeResult =
|
||||
| { ok: true }
|
||||
// L3: `attention_required` is a non-terminal failure — Claude is alive and
|
||||
// recoverable, but the resume-selection prompt is blocking. Caller maps to
|
||||
// restoreOutcome=attention_required (do NOT auto-answer per Decision 2).
|
||||
| { ok: false; code: "attention_required"; message: string; evidence?: string }
|
||||
| { ok: false; code: string; message: string };
|
||||
|
||||
const CLAUDE_TYPES = new Set(["claude_name", "claude_id"]);
|
||||
@@ -80,6 +84,20 @@ export class ClaudeResumeAdapter {
|
||||
};
|
||||
}
|
||||
|
||||
// L3: Claude resume-selection prompt → attention_required (not failed).
|
||||
// The runtime is alive and recoverable but blocked on operator selection.
|
||||
// Decision 2 forbids auto-answering; surface evidence and let the
|
||||
// operator/UI act, then later reconciliation may upgrade to
|
||||
// operator_recovered when the pane reaches usable state.
|
||||
if (probe.status === "attention_required") {
|
||||
return {
|
||||
ok: false,
|
||||
code: "attention_required",
|
||||
message: probe.detail,
|
||||
evidence: paneContent.split("\n").slice(-12).join("\n"),
|
||||
};
|
||||
}
|
||||
|
||||
if (probe.status === "resumed") {
|
||||
return { ok: true };
|
||||
}
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
import { shellQuote } from "../adapters/shell-quote.js";
|
||||
|
||||
export type NativeResumeProbeStatus = "resumed" | "failed" | "inconclusive";
|
||||
// L3 adds `attention_required` for the Claude resume-selection prompt proxy.
|
||||
// Distinct from `inconclusive` (we don't know yet) and `failed` (terminal
|
||||
// failure): the runtime is alive and recoverable but needs operator action.
|
||||
export type NativeResumeProbeStatus = "resumed" | "failed" | "inconclusive" | "attention_required";
|
||||
|
||||
export interface NativeResumeProbeInput {
|
||||
runtime: string | null;
|
||||
@@ -52,6 +55,13 @@ export function assessNativeResumeProbe(
|
||||
detail: "Claude reported that the requested session no longer exists.",
|
||||
};
|
||||
}
|
||||
if (looksLikeClaudeResumeSelectionPrompt(paneContent)) {
|
||||
return {
|
||||
status: "attention_required",
|
||||
code: "claude_resume_selection_prompt",
|
||||
detail: "Claude is at a resume-selection prompt; an operator must choose the conversation to continue.",
|
||||
};
|
||||
}
|
||||
if (looksLikeClaudeTrustPrompt(paneContent)) {
|
||||
return {
|
||||
status: "inconclusive",
|
||||
@@ -186,6 +196,29 @@ function looksLikeClaudeTrustPrompt(paneContent: string): boolean {
|
||||
&& paneContent.includes("Yes, I trust this folder");
|
||||
}
|
||||
|
||||
// Claude's resume-selection prompt appears when `claude --resume` finds multiple
|
||||
// candidate conversations (or after a reboot when the conversation index is
|
||||
// rebuilt). The prompt lists numbered options and asks the operator to pick.
|
||||
//
|
||||
// L3 invariant: do NOT auto-answer. Surface as `attention_required` and let an
|
||||
// operator choose; later reconciliation upgrades to `operator_recovered` only
|
||||
// when the operator reaches a usable state.
|
||||
function looksLikeClaudeResumeSelectionPrompt(paneContent: string): boolean {
|
||||
// Stable substring is the explicit "Choose ... conversation" verb plus the
|
||||
// numbered/arrow option marker that Claude prints. Both must be present so
|
||||
// we don't false-positive on similar TUI strings.
|
||||
const hasChooseVerb =
|
||||
paneContent.includes("Choose a conversation")
|
||||
|| paneContent.includes("Choose the conversation")
|
||||
|| paneContent.includes("Select a conversation");
|
||||
if (!hasChooseVerb) return false;
|
||||
|
||||
// Look for the numbered/arrow option marker in recent lines.
|
||||
const recentLines = paneContent.split("\n").slice(-30);
|
||||
const numberedOption = recentLines.some((line) => /^\s*(?:›\s*)?\d+\.\s+\S/.test(line));
|
||||
return numberedOption;
|
||||
}
|
||||
|
||||
function looksLikeClaudeLoginPrompt(paneContent: string): boolean {
|
||||
return paneContent.includes("Not logged in")
|
||||
&& paneContent.includes("Run /login");
|
||||
|
||||
@@ -174,7 +174,13 @@ export function deriveNodeLifecycleState(input: {
|
||||
nodeId: string;
|
||||
usableSnapshot: Snapshot | null;
|
||||
}): NodeLifecycleState {
|
||||
if (input.restoreOutcome === "failed" && input.sessionStatus === "running") {
|
||||
// L3: the explicit `attention_required` outcome (Claude resume-selection
|
||||
// prompt) and the L2 proxy (failed + alive tmux session) both surface as
|
||||
// lifecycleState=attention_required.
|
||||
if (
|
||||
input.restoreOutcome === "attention_required"
|
||||
|| (input.restoreOutcome === "failed" && input.sessionStatus === "running")
|
||||
) {
|
||||
return "attention_required";
|
||||
}
|
||||
if (input.sessionStatus === "running") return "running";
|
||||
@@ -203,7 +209,14 @@ function deriveContinuityOutcome(
|
||||
if (row.continuity_outcome) {
|
||||
return row.continuity_outcome as NodeInventoryEntry["continuityOutcome"];
|
||||
}
|
||||
return restoreOutcome === "n-a" ? null : restoreOutcome;
|
||||
if (restoreOutcome === "n-a") return null;
|
||||
// L3: `attention_required` and `operator_recovered` are restore-attempt
|
||||
// outcomes that don't map onto the ContinuityOutcome vocabulary
|
||||
// ("resumed"|"rebuilt"|"forked"|"fresh"|"failed"). Surface as null here;
|
||||
// the lifecycleState projection picks them up via restoreOutcome directly.
|
||||
if (restoreOutcome === "attention_required") return null;
|
||||
if (restoreOutcome === "operator_recovered") return "resumed";
|
||||
return restoreOutcome;
|
||||
}
|
||||
|
||||
function deriveRestoreOutcome(db: Database.Database, rigId: string, nodeId: string): NodeRestoreOutcome {
|
||||
@@ -223,6 +236,9 @@ function deriveRestoreOutcome(db: Database.Database, rigId: string, nodeId: stri
|
||||
if (nodeResult.status === "failed") return "failed";
|
||||
if (nodeResult.status === "rebuilt") return "rebuilt";
|
||||
if (nodeResult.status === "fresh") return "fresh";
|
||||
// L3 outcomes (attention_required, operator_recovered)
|
||||
if (nodeResult.status === "attention_required") return "attention_required";
|
||||
if (nodeResult.status === "operator_recovered") return "operator_recovered";
|
||||
// Compat: old persisted events may contain pre-rename values
|
||||
if ((nodeResult.status as string) === "checkpoint_written") return "rebuilt";
|
||||
if ((nodeResult.status as string) === "fresh_no_checkpoint") return "fresh";
|
||||
|
||||
@@ -12,6 +12,7 @@ import type { TmuxAdapter } from "../adapters/tmux.js";
|
||||
import type { ClaudeResumeAdapter } from "../adapters/claude-resume.js";
|
||||
import type { CodexResumeAdapter } from "../adapters/codex-resume.js";
|
||||
import type { TranscriptStore } from "./transcript-store.js";
|
||||
import { assessNativeResumeProbe } from "./native-resume-probe.js";
|
||||
import type {
|
||||
RestoreOutcome,
|
||||
RestoreRigResult,
|
||||
@@ -26,14 +27,42 @@ import type {
|
||||
RigServicesRecord,
|
||||
} from "./types.js";
|
||||
|
||||
// L3: result shape for runtime-truth reconciliation. A reconciliation that
|
||||
// does NOT meet all four evidence preconditions is a no-op with a missing
|
||||
// reason, NOT an error. Decision 3: terminal post-reconciliation outcome is
|
||||
// `operator_recovered`; `ready` is forbidden.
|
||||
export type ReconcileNodeResult =
|
||||
| {
|
||||
ok: true;
|
||||
attemptId: number;
|
||||
from: "failed" | "attention_required";
|
||||
to: "operator_recovered";
|
||||
evidence: { tmux: boolean; fgProcess: "claude" | "codex" | string; resumeTokenUsed: boolean; paneState: "usable" };
|
||||
}
|
||||
| {
|
||||
ok: false;
|
||||
code:
|
||||
| "node_not_found"
|
||||
| "no_attempt"
|
||||
| "outcome_not_upgradable"
|
||||
| "tmux_session_missing"
|
||||
| "fg_process_not_runtime"
|
||||
| "resume_token_not_used"
|
||||
| "pane_not_usable";
|
||||
detail: string;
|
||||
};
|
||||
|
||||
// Only these edge kinds constrain launch order
|
||||
const LAUNCH_DEPENDENCY_KINDS = new Set(["delegates_to", "spawned_by"]);
|
||||
|
||||
export function rollupRestoreRigResult(nodes: RestoreNodeResult[]): RestoreRigResult {
|
||||
if (nodes.length === 0) return "failed";
|
||||
const successful = nodes.filter((node) => node.status !== "failed");
|
||||
if (successful.length === 0) return "failed";
|
||||
if (nodes.some((node) => node.status === "fresh" || node.status === "failed")) {
|
||||
// L3: `attention_required` is non-terminal failure (alive but blocked on
|
||||
// operator action). It rolls up as `partially_restored`. `operator_recovered`
|
||||
// is a clean post-reconciliation outcome and rolls up like `resumed`.
|
||||
const allFailed = nodes.every((node) => node.status === "failed");
|
||||
if (allFailed) return "failed";
|
||||
if (nodes.some((node) => node.status === "fresh" || node.status === "failed" || node.status === "attention_required")) {
|
||||
return "partially_restored";
|
||||
}
|
||||
return "fully_restored";
|
||||
@@ -110,6 +139,13 @@ export class RestoreOrchestrator {
|
||||
async restore(snapshotId: string, opts?: {
|
||||
adapters?: Record<string, import("./runtime-adapter.js").RuntimeAdapter>;
|
||||
fsOps?: { exists(path: string): boolean };
|
||||
/**
|
||||
* L3: fired with the persisted `restore.started` event seq as soon as the
|
||||
* orchestrator commits to running per-node restore. Routes use this to
|
||||
* return `attemptId` to the client immediately while per-node work
|
||||
* continues in the background.
|
||||
*/
|
||||
onAttemptStarted?: (attemptId: number) => void;
|
||||
}): Promise<RestoreOutcome> {
|
||||
// 1. Load snapshot
|
||||
const snapshot = this.snapshotRepo.getSnapshot(snapshotId);
|
||||
@@ -170,8 +206,16 @@ export class RestoreOrchestrator {
|
||||
this.sessionRegistry.markDetached(sessionId);
|
||||
}
|
||||
|
||||
// 3. Emit restore.started
|
||||
this.eventBus.emit({ type: "restore.started", rigId, snapshotId });
|
||||
// 3. Emit restore.started — the persisted event seq IS the attempt id
|
||||
// (Decision 1: no separate restore_attempts table).
|
||||
const restoreStartedEvent = this.eventBus.emit({ type: "restore.started", rigId, snapshotId });
|
||||
const attemptId = restoreStartedEvent.seq;
|
||||
try {
|
||||
opts?.onAttemptStarted?.(attemptId);
|
||||
} catch {
|
||||
// onAttemptStarted is fire-and-forget; never let a route's response
|
||||
// logic crash the restore pipeline.
|
||||
}
|
||||
|
||||
// 3b. Service gate: boot services before agent restore if this rig has services
|
||||
if (this.serviceOrchestrator) {
|
||||
@@ -676,8 +720,19 @@ export class RestoreOrchestrator {
|
||||
return { nodeId: node.id, logicalId: node.logicalId, status: "failed", error: `Resume requested but no token available. Restore the node manually or launch fresh explicitly.` };
|
||||
} else {
|
||||
const resumeOutcome = await this.attemptResume(sessionName, resumeType, resumeToken, node.cwd ?? "/");
|
||||
if (resumeOutcome === "resumed") {
|
||||
if (resumeOutcome.kind === "resumed") {
|
||||
baseStatus = "resumed";
|
||||
} else if (resumeOutcome.kind === "attention_required") {
|
||||
// L3 Decision 2: Claude resume-selection prompt -> attention_required.
|
||||
// Do NOT auto-answer. Reconcile later via reconcileNodeRuntimeTruth
|
||||
// when the operator reaches a usable pane state.
|
||||
return {
|
||||
nodeId: node.id,
|
||||
logicalId: node.logicalId,
|
||||
status: "attention_required",
|
||||
error: resumeOutcome.message,
|
||||
attentionEvidence: resumeOutcome.evidence ?? null,
|
||||
};
|
||||
} else {
|
||||
return { nodeId: node.id, logicalId: node.logicalId, status: "failed", error: `Resume attempted but failed. Check the harness state manually or launch fresh explicitly.` };
|
||||
}
|
||||
@@ -865,19 +920,163 @@ export class RestoreOrchestrator {
|
||||
resumeType: string,
|
||||
resumeToken: string | null,
|
||||
cwd: string
|
||||
): Promise<"resumed" | "retry_fresh" | "failed"> {
|
||||
): Promise<
|
||||
| { kind: "resumed" }
|
||||
| { kind: "retry_fresh" }
|
||||
| { kind: "failed"; message: string }
|
||||
| { kind: "attention_required"; message: string; evidence?: string }
|
||||
> {
|
||||
if (this.claudeResume.canResume(resumeType, resumeToken)) {
|
||||
const result = await this.claudeResume.resume(sessionName, resumeType, resumeToken, cwd);
|
||||
if (result.ok) return "resumed";
|
||||
return result.code === "retry_fresh" ? "retry_fresh" : "failed";
|
||||
if (result.ok) return { kind: "resumed" };
|
||||
if (result.code === "retry_fresh") return { kind: "retry_fresh" };
|
||||
// L3: surface attention_required from the Claude probe (resume-selection prompt).
|
||||
if (result.code === "attention_required") {
|
||||
return {
|
||||
kind: "attention_required",
|
||||
message: result.message,
|
||||
evidence: (result as { evidence?: string }).evidence,
|
||||
};
|
||||
}
|
||||
return { kind: "failed", message: result.message };
|
||||
}
|
||||
|
||||
if (this.codexResume.canResume(resumeType, resumeToken)) {
|
||||
const result = await this.codexResume.resume(sessionName, resumeType, resumeToken, cwd);
|
||||
return result.ok ? "resumed" : "failed";
|
||||
return result.ok ? { kind: "resumed" } : { kind: "failed", message: result.message };
|
||||
}
|
||||
|
||||
return "failed";
|
||||
return { kind: "failed", message: "No resume adapter available for this runtime/token combination." };
|
||||
}
|
||||
|
||||
/**
|
||||
* L3 Decision 3: runtime-truth reconciliation. Given a node whose original
|
||||
* `restoreOutcome` was `failed` or `attention_required`, examine current
|
||||
* runtime state. If ALL four visible-evidence preconditions hold, append
|
||||
* `restore.outcome_reconciled` so the node's effective post-reconciliation
|
||||
* outcome becomes `operator_recovered`. Never mutates or deletes the
|
||||
* original failure event; never produces `ready`.
|
||||
*
|
||||
* Returns `{ ok: true, attemptId, from, to, evidence }` on upgrade, or
|
||||
* `{ ok: false, code, detail }` describing exactly which precondition
|
||||
* failed (or "no_attempt" / "outcome_not_upgradable" when there is nothing
|
||||
* to reconcile).
|
||||
*/
|
||||
async reconcileNodeRuntimeTruth(
|
||||
rigId: string,
|
||||
nodeId: string,
|
||||
): Promise<ReconcileNodeResult> {
|
||||
// Locate the latest restore attempt for this rig.
|
||||
const startedRow = this.db.prepare(
|
||||
"SELECT seq, payload FROM events WHERE rig_id = ? AND type = 'restore.started' ORDER BY seq DESC LIMIT 1"
|
||||
).get(rigId) as { seq: number; payload: string } | undefined;
|
||||
if (!startedRow) {
|
||||
return { ok: false, code: "no_attempt", detail: "No restore.started event recorded for this rig." };
|
||||
}
|
||||
const attemptId = startedRow.seq;
|
||||
|
||||
// Find the node's most recent post-attempt outcome from the most-recent
|
||||
// restore.completed event for this rig. If the latest outcome is not
|
||||
// failed or attention_required, the reconciler has nothing to upgrade.
|
||||
const completedRow = this.db.prepare(
|
||||
"SELECT payload FROM events WHERE rig_id = ? AND type = 'restore.completed' AND seq > ? ORDER BY seq DESC LIMIT 1"
|
||||
).get(rigId, attemptId) as { payload: string } | undefined;
|
||||
let nodeStatus: RestoreNodeResult["status"] | null = null;
|
||||
let nodeLogicalId: string | null = null;
|
||||
if (completedRow) {
|
||||
try {
|
||||
const parsed = JSON.parse(completedRow.payload) as { result: RestoreResult };
|
||||
const found = parsed.result?.nodes?.find((n) => n.nodeId === nodeId);
|
||||
if (found) {
|
||||
nodeStatus = found.status;
|
||||
nodeLogicalId = found.logicalId;
|
||||
}
|
||||
} catch {
|
||||
// payload corruption — treat as no node record found
|
||||
}
|
||||
}
|
||||
if (!nodeStatus) {
|
||||
return { ok: false, code: "node_not_found", detail: `Node ${nodeId} has no record in the latest restore.completed event for rig ${rigId}.` };
|
||||
}
|
||||
if (nodeStatus !== "failed" && nodeStatus !== "attention_required") {
|
||||
return { ok: false, code: "outcome_not_upgradable", detail: `Reconciliation only upgrades failed or attention_required; current outcome is ${nodeStatus}.` };
|
||||
}
|
||||
const fromStatus: "failed" | "attention_required" = nodeStatus;
|
||||
|
||||
// Resolve canonical session name for this node so we can probe tmux/pane.
|
||||
const bindingRow = this.db.prepare(
|
||||
"SELECT tmux_session FROM bindings WHERE node_id = ?"
|
||||
).get(nodeId) as { tmux_session: string | null } | undefined;
|
||||
const sessionName = bindingRow?.tmux_session ?? null;
|
||||
if (!sessionName) {
|
||||
return { ok: false, code: "tmux_session_missing", detail: "No tmux session bound for this node." };
|
||||
}
|
||||
|
||||
// Precondition #1: tmux session exists.
|
||||
let alive = false;
|
||||
try {
|
||||
alive = await this.tmuxAdapter.hasSession(sessionName);
|
||||
} catch {
|
||||
// L1 fail-closed: ambiguous probe failure stays as not-alive for the
|
||||
// reconciler. Original failure event remains untouched.
|
||||
alive = false;
|
||||
}
|
||||
if (!alive) {
|
||||
return { ok: false, code: "tmux_session_missing", detail: `Tmux session ${sessionName} is not currently alive.` };
|
||||
}
|
||||
|
||||
// Resolve runtime + capture pane state for preconditions #2 and #4.
|
||||
const nodeRow = this.db.prepare(
|
||||
"SELECT runtime FROM nodes WHERE id = ?"
|
||||
).get(nodeId) as { runtime: string | null } | undefined;
|
||||
const runtime = nodeRow?.runtime ?? null;
|
||||
const paneCommand = await this.tmuxAdapter.getPaneCommand(sessionName);
|
||||
const paneContent = (await this.tmuxAdapter.capturePaneContent(sessionName, 40)) ?? "";
|
||||
|
||||
// Precondition #2: foreground process is the runtime (claude or codex).
|
||||
const probe = assessNativeResumeProbe({ runtime, paneCommand, paneContent });
|
||||
const fgProcess = paneCommand && (paneCommand === "claude" || paneCommand.startsWith("codex"))
|
||||
? (paneCommand === "claude" ? "claude" : "codex") as "claude" | "codex"
|
||||
: null;
|
||||
if (!fgProcess) {
|
||||
return { ok: false, code: "fg_process_not_runtime", detail: `Foreground process is "${paneCommand ?? "(unknown)"}", not claude/codex.` };
|
||||
}
|
||||
|
||||
// Precondition #3: the resume token was actually used at launch.
|
||||
// Stored on the latest session row; null/empty means resume was not
|
||||
// exercised so a "recovered" claim has no basis.
|
||||
const sessRow = this.db.prepare(
|
||||
"SELECT resume_token FROM sessions WHERE node_id = ? ORDER BY id DESC LIMIT 1"
|
||||
).get(nodeId) as { resume_token: string | null } | undefined;
|
||||
const resumeTokenUsed = typeof sessRow?.resume_token === "string" && sessRow.resume_token.length > 0;
|
||||
if (!resumeTokenUsed) {
|
||||
return { ok: false, code: "resume_token_not_used", detail: "No resume token recorded on the latest session row." };
|
||||
}
|
||||
|
||||
// Precondition #4: pane is at a usable/idle state — explicitly NOT a
|
||||
// resume-selection prompt and not the "returned to shell" failure mode.
|
||||
if (probe.status !== "resumed") {
|
||||
return { ok: false, code: "pane_not_usable", detail: `Pane state is ${probe.status} (${probe.code}); reconciliation requires resumed.` };
|
||||
}
|
||||
|
||||
// All four preconditions hold. Append (never mutate) the audit event.
|
||||
this.eventBus.emit({
|
||||
type: "restore.outcome_reconciled",
|
||||
rigId,
|
||||
nodeId,
|
||||
attemptId,
|
||||
from: fromStatus,
|
||||
to: "operator_recovered",
|
||||
evidence: { tmux: true, fgProcess, resumeTokenUsed: true, paneState: "usable" },
|
||||
});
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
attemptId,
|
||||
from: fromStatus,
|
||||
to: "operator_recovered",
|
||||
evidence: { tmux: true, fgProcess, resumeTokenUsed: true, paneState: "usable" },
|
||||
};
|
||||
}
|
||||
|
||||
private writeCheckpointFile(cwd: string, checkpoint: Checkpoint): boolean {
|
||||
|
||||
@@ -103,6 +103,11 @@ export type RigEvent =
|
||||
| { type: "snapshot.created"; rigId: string; snapshotId: string; kind: string }
|
||||
| { type: "restore.started"; rigId: string; snapshotId: string }
|
||||
| { type: "restore.completed"; rigId: string; snapshotId: string; result: RestoreResult }
|
||||
// L3: appended (never replaces) when reconcileNodeRuntimeTruth upgrades a
|
||||
// failed/attention_required restoreOutcome to operator_recovered after
|
||||
// visible-runtime-evidence preconditions hold. The original failure event
|
||||
// is preserved in the log; this event records the audit trail of the upgrade.
|
||||
| { type: "restore.outcome_reconciled"; rigId: string; nodeId: string; attemptId: number; from: "failed" | "attention_required"; to: "operator_recovered"; evidence: { tmux: boolean; fgProcess: "claude" | "codex" | string; resumeTokenUsed: boolean; paneState: "usable" } }
|
||||
| { type: "agent.activity"; rigId: string; nodeId: string; sessionName: string; runtime: string | null; activity: AgentActivity }
|
||||
| { type: "rig.imported"; rigId: string; specName: string; specVersion: string }
|
||||
// Package events (cross-rig, no rigId)
|
||||
@@ -236,8 +241,14 @@ export interface RestoreValidationBlocker {
|
||||
export interface RestoreNodeResult {
|
||||
nodeId: string;
|
||||
logicalId: string;
|
||||
status: "resumed" | "rebuilt" | "fresh" | "failed";
|
||||
// L3: `attention_required` is set when the post-launch probe detects a
|
||||
// Claude resume-selection prompt. `operator_recovered` is the terminal
|
||||
// outcome after `restore.outcome_reconciled`; never produced directly by
|
||||
// the orchestrator's restore pipeline (only by reconcileNodeRuntimeTruth).
|
||||
status: "resumed" | "rebuilt" | "fresh" | "failed" | "attention_required" | "operator_recovered";
|
||||
error?: string;
|
||||
/** Pane evidence captured when status is `attention_required` (L3, optional). */
|
||||
attentionEvidence?: string | null;
|
||||
}
|
||||
|
||||
export type RestoreOutcome =
|
||||
@@ -252,7 +263,10 @@ export type RestoreOutcome =
|
||||
|
||||
// -- Node inventory projection (NS-T02) --
|
||||
|
||||
export type NodeRestoreOutcome = "resumed" | "rebuilt" | "fresh" | "failed" | "n-a";
|
||||
// L3 extends with `attention_required` (Claude resume-selection prompt proxy)
|
||||
// and `operator_recovered` (terminal post-reconciliation outcome — never produced
|
||||
// directly by restore, only emitted via `restore.outcome_reconciled`).
|
||||
export type NodeRestoreOutcome = "resumed" | "rebuilt" | "fresh" | "failed" | "attention_required" | "operator_recovered" | "n-a";
|
||||
export type OccupantLifecycle = "active" | "retiring" | "retired" | "context_walled" | "compacted" | "crashed" | "unknown";
|
||||
export type ContinuityOutcome = "resumed" | "rebuilt" | "forked" | "fresh" | "failed";
|
||||
export type HandoverResult = "complete" | "unchanged" | "partial" | "failed" | null;
|
||||
|
||||
@@ -55,6 +55,16 @@ snapshotsRoutes.get("/:id", (c) => {
|
||||
});
|
||||
|
||||
// POST /api/rigs/:rigId/restore/:snapshotId
|
||||
//
|
||||
// L3: returns `{ ok: true, attemptId, status: "started", rigId }` AS SOON AS the
|
||||
// orchestrator has emitted `restore.started`, BEFORE per-node restore work
|
||||
// completes. The persisted `restore.started` event seq IS the attempt id
|
||||
// (Decision 1: no separate restore_attempts table). Per-node work continues in
|
||||
// the background; clients query event log / node inventory to follow progress.
|
||||
//
|
||||
// Pre-restore validation failures and other "couldn't even start" errors return
|
||||
// the original error payloads with appropriate HTTP status codes (404/409/500),
|
||||
// because in those cases no `restore.started` event was emitted.
|
||||
restoreRoutes.post("/:snapshotId", async (c) => {
|
||||
const rigId = c.req.param("rigId")!;
|
||||
const snapshotId = c.req.param("snapshotId")!;
|
||||
@@ -68,45 +78,62 @@ restoreRoutes.post("/:snapshotId", async (c) => {
|
||||
|
||||
const adapters = c.get("runtimeAdapters" as never) as Record<string, import("../domain/runtime-adapter.js").RuntimeAdapter> | undefined;
|
||||
const fs = await import("node:fs");
|
||||
const outcome = await restoreOrchestrator.restore(snapshotId, {
|
||||
adapters: adapters ?? {},
|
||||
fsOps: { exists: (p: string) => fs.existsSync(p) },
|
||||
|
||||
return new Promise<Response>((resolve) => {
|
||||
let resolved = false;
|
||||
|
||||
const restorePromise = restoreOrchestrator.restore(snapshotId, {
|
||||
adapters: adapters ?? {},
|
||||
fsOps: { exists: (p: string) => fs.existsSync(p) },
|
||||
onAttemptStarted: (attemptId) => {
|
||||
if (resolved) return;
|
||||
resolved = true;
|
||||
// Per-node restore work runs in background; client receives attemptId
|
||||
// immediately and can poll /api/events or node inventory for progress.
|
||||
resolve(c.json({ ok: true, attemptId, status: "started", rigId }, 202));
|
||||
},
|
||||
});
|
||||
|
||||
restorePromise
|
||||
.then((outcome) => {
|
||||
if (resolved) {
|
||||
// Background path: response already sent. Per-node failures are in
|
||||
// the event log; no need to do anything here.
|
||||
return;
|
||||
}
|
||||
// Pre-restore-started error path: no `restore.started` was emitted, so
|
||||
// the route should respond with the original error mapping.
|
||||
resolved = true;
|
||||
if (!outcome.ok) {
|
||||
if (outcome.code === "pre_restore_validation_failed") {
|
||||
resolve(c.json({
|
||||
error: outcome.message,
|
||||
code: outcome.code,
|
||||
...outcome.result,
|
||||
remediation: outcome.result.blockers?.map((blocker) => blocker.remediation) ?? [],
|
||||
}, 409));
|
||||
return;
|
||||
}
|
||||
const status = outcome.code === "snapshot_not_found" || outcome.code === "rig_not_found"
|
||||
? 404
|
||||
: outcome.code === "restore_in_progress" || outcome.code === "rig_not_stopped"
|
||||
? 409
|
||||
: 500;
|
||||
resolve(c.json({ error: outcome.message, code: outcome.code }, status));
|
||||
return;
|
||||
}
|
||||
// Defensive: outcome.ok with no onAttemptStarted firing means the
|
||||
// orchestrator emitted restore.started but the callback was somehow
|
||||
// bypassed. Surface the result anyway with a synthesized attemptId.
|
||||
resolve(c.json({ ok: true, attemptId: -1, status: "completed", rigId, result: outcome.result }, 200));
|
||||
})
|
||||
.catch((err) => {
|
||||
if (resolved) return;
|
||||
resolved = true;
|
||||
resolve(c.json({
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
code: "restore_error",
|
||||
}, 500));
|
||||
});
|
||||
});
|
||||
|
||||
if (!outcome.ok) {
|
||||
if (outcome.code === "pre_restore_validation_failed") {
|
||||
return c.json({
|
||||
error: outcome.message,
|
||||
code: outcome.code,
|
||||
...outcome.result,
|
||||
remediation: outcome.result.blockers?.map((blocker) => blocker.remediation) ?? [],
|
||||
}, 409);
|
||||
}
|
||||
const status = outcome.code === "snapshot_not_found" || outcome.code === "rig_not_found"
|
||||
? 404
|
||||
: outcome.code === "restore_in_progress" || outcome.code === "rig_not_stopped"
|
||||
? 409
|
||||
: 500;
|
||||
return c.json({ error: outcome.message, code: outcome.code }, status);
|
||||
}
|
||||
|
||||
// Compute attach command from first running node
|
||||
const { getNodeInventory } = await import("../domain/node-inventory.js");
|
||||
const inventory = getNodeInventory(restoreOrchestrator.db, rigId);
|
||||
const firstRunning = inventory.find((n) => n.canonicalSessionName && n.sessionStatus === "running");
|
||||
const attachCommand = firstRunning?.tmuxAttachCommand ?? inventory.find((n) => n.canonicalSessionName)?.tmuxAttachCommand ?? null;
|
||||
const inventoryByLogicalId = new Map(inventory.map((entry) => [entry.logicalId, entry]));
|
||||
const nodes = outcome.result.nodes.map((node) => {
|
||||
const detail = inventoryByLogicalId.get(node.logicalId);
|
||||
return {
|
||||
...node,
|
||||
canonicalSessionName: detail?.canonicalSessionName ?? null,
|
||||
tmuxAttachCommand: detail?.tmuxAttachCommand ?? null,
|
||||
resumeCommand: detail?.resumeCommand ?? null,
|
||||
recoveryGuidance: detail?.recoveryGuidance ?? null,
|
||||
cwd: detail?.cwd ?? null,
|
||||
};
|
||||
});
|
||||
|
||||
return c.json({ ...outcome.result, nodes, attachCommand });
|
||||
});
|
||||
@@ -380,4 +380,79 @@ describe("native resume probe", () => {
|
||||
})
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
// L3: Claude resume-selection prompt → attention_required.
|
||||
describe("Claude resume-selection prompt (L3)", () => {
|
||||
it("classifies a numbered Claude resume-selection prompt as attention_required", () => {
|
||||
const paneContent = [
|
||||
"Claude Code v2.1.89",
|
||||
"",
|
||||
"Choose a conversation to resume:",
|
||||
"",
|
||||
" 1. project-foo (modified 2h ago)",
|
||||
" 2. project-bar (modified yesterday)",
|
||||
" 3. project-baz (modified last week)",
|
||||
"",
|
||||
"Enter a number, or press q to cancel.",
|
||||
].join("\n");
|
||||
|
||||
const result = assessNativeResumeProbe({
|
||||
runtime: "claude-code",
|
||||
paneCommand: "claude",
|
||||
paneContent,
|
||||
});
|
||||
|
||||
expect(result.status).toBe("attention_required");
|
||||
expect(result.code).toBe("claude_resume_selection_prompt");
|
||||
});
|
||||
|
||||
it("classifies the › arrow variant of the resume-selection prompt as attention_required", () => {
|
||||
const paneContent = [
|
||||
"Choose the conversation to resume:",
|
||||
"",
|
||||
"› 1. recent project",
|
||||
" 2. older project",
|
||||
].join("\n");
|
||||
|
||||
const result = assessNativeResumeProbe({
|
||||
runtime: "claude-code",
|
||||
paneCommand: "claude",
|
||||
paneContent,
|
||||
});
|
||||
|
||||
expect(result.status).toBe("attention_required");
|
||||
});
|
||||
|
||||
it("does NOT classify Claude active TUI as resume-selection prompt (regression)", () => {
|
||||
const paneContent = [
|
||||
"Claude Code v2.1.89",
|
||||
"",
|
||||
" ❯ accept edits on",
|
||||
"",
|
||||
"[work in progress]",
|
||||
].join("\n");
|
||||
|
||||
const result = assessNativeResumeProbe({
|
||||
runtime: "claude-code",
|
||||
paneCommand: "claude",
|
||||
paneContent,
|
||||
});
|
||||
|
||||
// Should NOT be attention_required — this is the active TUI case.
|
||||
expect(result.status).toBe("resumed");
|
||||
expect(result.code).toBe("active_runtime");
|
||||
});
|
||||
|
||||
it("does NOT classify mere mention of 'Choose a conversation' without numbered options", () => {
|
||||
const paneContent = "The docs say: 'Choose a conversation to focus on.'";
|
||||
const result = assessNativeResumeProbe({
|
||||
runtime: "claude-code",
|
||||
paneCommand: "claude",
|
||||
paneContent,
|
||||
});
|
||||
|
||||
// Without a numbered option list, this is not a real prompt.
|
||||
expect(result.status).not.toBe("attention_required");
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -1952,4 +1952,325 @@ describe("RestoreOrchestrator", () => {
|
||||
const row = db.prepare("SELECT status FROM sessions WHERE id = ?").get(session.id) as { status: string };
|
||||
expect(row.status).toBe("running");
|
||||
});
|
||||
|
||||
// L3 Decision 1: attempt id surfacing
|
||||
describe("attempt id (L3)", () => {
|
||||
it("restore() invokes onAttemptStarted with the persisted restore.started event seq", async () => {
|
||||
const orch = createOrchestrator();
|
||||
const snap = seedRigAndSnapshot();
|
||||
let receivedAttemptId: number | null = null;
|
||||
|
||||
const outcome = await orch.restore(snap.id, {
|
||||
onAttemptStarted: (id) => { receivedAttemptId = id; },
|
||||
});
|
||||
|
||||
expect(outcome.ok).toBe(true);
|
||||
expect(receivedAttemptId).not.toBeNull();
|
||||
expect(typeof receivedAttemptId).toBe("number");
|
||||
|
||||
// attemptId must match a queryable restore.started event seq.
|
||||
const startedRow = db
|
||||
.prepare("SELECT seq FROM events WHERE rig_id = ? AND type = 'restore.started' ORDER BY seq DESC LIMIT 1")
|
||||
.get(snap.rigId) as { seq: number } | undefined;
|
||||
expect(startedRow?.seq).toBe(receivedAttemptId);
|
||||
});
|
||||
|
||||
it("onAttemptStarted is NOT invoked when pre-restore validation fails (no restore.started emit)", async () => {
|
||||
const orch = createOrchestrator();
|
||||
const snap = seedRigAndSnapshot({
|
||||
nodes: [{ logicalId: "orchestrator", role: "orchestrator", runtime: "claude-code", cwd: "/tmp" }],
|
||||
edges: [],
|
||||
withCheckpoint: "orchestrator",
|
||||
});
|
||||
// Force a pre-restore validation failure: clear the node's cwd so the
|
||||
// checkpoint blocks restore (checkpoint_missing_node_cwd).
|
||||
const data = JSON.parse(JSON.stringify(snap.data));
|
||||
data.nodes[0].cwd = null;
|
||||
db.prepare("UPDATE snapshots SET data = ? WHERE id = ?").run(JSON.stringify(data), snap.id);
|
||||
|
||||
let invoked = false;
|
||||
const outcome = await orch.restore(snap.id, {
|
||||
onAttemptStarted: () => { invoked = true; },
|
||||
});
|
||||
|
||||
expect(outcome.ok).toBe(false);
|
||||
if (!outcome.ok) {
|
||||
expect(outcome.code).toBe("pre_restore_validation_failed");
|
||||
}
|
||||
expect(invoked).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// L3 Decision 3: runtime-truth reconciliation
|
||||
describe("reconcileNodeRuntimeTruth (L3)", () => {
|
||||
// Reconciler-specific mock with `hasSession` as a vi.fn so individual tests
|
||||
// can vary it. Cannot reuse `mockTmux()` because that wires hasSession as a
|
||||
// plain async function (88 existing tests depend on that shape).
|
||||
function mockTmuxForReconciler(): TmuxAdapter {
|
||||
return {
|
||||
createSession: vi.fn(async () => ({ ok: true as const })),
|
||||
killSession: vi.fn(async () => ({ ok: true as const })),
|
||||
sendText: vi.fn(async () => ({ ok: true as const })),
|
||||
sendKeys: vi.fn(async () => ({ ok: true as const })),
|
||||
getPaneCommand: vi.fn(async () => "claude"),
|
||||
capturePaneContent: vi.fn(async () => ""),
|
||||
hasSession: vi.fn(async () => false),
|
||||
listSessions: async () => [],
|
||||
listWindows: async () => [],
|
||||
listPanes: async () => [],
|
||||
} as unknown as TmuxAdapter;
|
||||
}
|
||||
|
||||
let nextSeed = 90;
|
||||
function seedFailedAttempt(opts: {
|
||||
rigName?: string;
|
||||
logicalId?: string;
|
||||
runtime?: "claude-code" | "codex";
|
||||
restoreOutcome: "failed" | "attention_required";
|
||||
withResumeToken?: boolean;
|
||||
withBinding?: boolean;
|
||||
}) {
|
||||
// Session name validator requires legacy `r{NN}-{suffix}` or canonical
|
||||
// `{pod}-{member}@{rig}`. Each call gets a unique numeric rig id so the
|
||||
// legacy pattern matches and rig names don't collide.
|
||||
const rigName = opts.rigName ?? `r${nextSeed++}`;
|
||||
const logicalId = opts.logicalId ?? "worker";
|
||||
const runtime = opts.runtime ?? "claude-code";
|
||||
const rig = rigRepo.createRig(rigName);
|
||||
const node = rigRepo.addNode(rig.id, logicalId, { role: "worker", runtime });
|
||||
|
||||
// Bind a tmux session name so the reconciler can probe.
|
||||
const sessionName = `${rigName}-${logicalId}`;
|
||||
if (opts.withBinding ?? true) {
|
||||
sessionRegistry.updateBinding(node.id, { tmuxSession: sessionName });
|
||||
}
|
||||
|
||||
// Seed a session row, optionally with resume token.
|
||||
const sess = sessionRegistry.registerSession(node.id, sessionName);
|
||||
if (opts.withResumeToken) {
|
||||
db.prepare("UPDATE sessions SET resume_type = ?, resume_token = ? WHERE id = ?")
|
||||
.run(runtime === "claude-code" ? "claude_id" : "codex_id", "tok-abc-123", sess.id);
|
||||
}
|
||||
|
||||
// Seed restore.started + restore.completed events with this node's outcome.
|
||||
eventBus.emit({ type: "restore.started", rigId: rig.id, snapshotId: "snap-recon" });
|
||||
eventBus.emit({
|
||||
type: "restore.completed",
|
||||
rigId: rig.id,
|
||||
snapshotId: "snap-recon",
|
||||
result: {
|
||||
snapshotId: "snap-recon",
|
||||
preRestoreSnapshotId: null,
|
||||
rigResult: "partially_restored",
|
||||
nodes: [
|
||||
{ nodeId: node.id, logicalId, status: opts.restoreOutcome },
|
||||
],
|
||||
warnings: [],
|
||||
},
|
||||
});
|
||||
|
||||
return { rig, nodeId: node.id, sessionName };
|
||||
}
|
||||
|
||||
it("upgrades failed -> operator_recovered when ALL four preconditions hold; emits audit event", async () => {
|
||||
const tmux = mockTmuxForReconciler();
|
||||
(tmux.hasSession as ReturnType<typeof vi.fn>).mockResolvedValue(true);
|
||||
(tmux.getPaneCommand as ReturnType<typeof vi.fn>).mockResolvedValue("claude");
|
||||
(tmux.capturePaneContent as ReturnType<typeof vi.fn>).mockResolvedValue([
|
||||
"Claude Code v2.1.89",
|
||||
"",
|
||||
" ❯ accept edits on",
|
||||
"",
|
||||
].join("\n"));
|
||||
const orch = createOrchestrator({ tmux });
|
||||
const seeded = seedFailedAttempt({ restoreOutcome: "failed", withResumeToken: true });
|
||||
|
||||
const result = await orch.reconcileNodeRuntimeTruth(seeded.rig.id, seeded.nodeId);
|
||||
|
||||
expect(result.ok).toBe(true);
|
||||
if (result.ok) {
|
||||
expect(result.from).toBe("failed");
|
||||
expect(result.to).toBe("operator_recovered");
|
||||
expect(result.evidence).toEqual({
|
||||
tmux: true,
|
||||
fgProcess: "claude",
|
||||
resumeTokenUsed: true,
|
||||
paneState: "usable",
|
||||
});
|
||||
}
|
||||
|
||||
// Audit event present.
|
||||
const reconciled = db.prepare(
|
||||
"SELECT * FROM events WHERE rig_id = ? AND type = 'restore.outcome_reconciled'"
|
||||
).all(seeded.rig.id);
|
||||
expect(reconciled).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("upgrades attention_required -> operator_recovered when ALL preconditions hold", async () => {
|
||||
const tmux = mockTmuxForReconciler();
|
||||
(tmux.hasSession as ReturnType<typeof vi.fn>).mockResolvedValue(true);
|
||||
(tmux.getPaneCommand as ReturnType<typeof vi.fn>).mockResolvedValue("claude");
|
||||
(tmux.capturePaneContent as ReturnType<typeof vi.fn>).mockResolvedValue([
|
||||
"Claude Code v2.1.89",
|
||||
"",
|
||||
" ❯ accept edits on",
|
||||
].join("\n"));
|
||||
const orch = createOrchestrator({ tmux });
|
||||
const seeded = seedFailedAttempt({ restoreOutcome: "attention_required", withResumeToken: true });
|
||||
|
||||
const result = await orch.reconcileNodeRuntimeTruth(seeded.rig.id, seeded.nodeId);
|
||||
|
||||
expect(result.ok).toBe(true);
|
||||
if (result.ok) {
|
||||
expect(result.from).toBe("attention_required");
|
||||
expect(result.to).toBe("operator_recovered");
|
||||
}
|
||||
});
|
||||
|
||||
it("audit-trail: original restore.completed event with status=failed remains queryable after upgrade", async () => {
|
||||
const tmux = mockTmuxForReconciler();
|
||||
(tmux.hasSession as ReturnType<typeof vi.fn>).mockResolvedValue(true);
|
||||
(tmux.getPaneCommand as ReturnType<typeof vi.fn>).mockResolvedValue("claude");
|
||||
(tmux.capturePaneContent as ReturnType<typeof vi.fn>).mockResolvedValue("Claude Code v2.1.89\n ❯ accept edits on");
|
||||
const orch = createOrchestrator({ tmux });
|
||||
const seeded = seedFailedAttempt({ restoreOutcome: "failed", withResumeToken: true });
|
||||
|
||||
await orch.reconcileNodeRuntimeTruth(seeded.rig.id, seeded.nodeId);
|
||||
|
||||
// Original restore.completed event must still carry the failed status.
|
||||
const completed = db.prepare(
|
||||
"SELECT payload FROM events WHERE rig_id = ? AND type = 'restore.completed'"
|
||||
).all(seeded.rig.id) as { payload: string }[];
|
||||
expect(completed).toHaveLength(1);
|
||||
const parsed = JSON.parse(completed[0]!.payload) as { result: { nodes: Array<{ status: string }> } };
|
||||
expect(parsed.result.nodes[0]!.status).toBe("failed");
|
||||
});
|
||||
|
||||
it("no-op when tmux session is missing (precondition #1 fails)", async () => {
|
||||
const tmux = mockTmuxForReconciler();
|
||||
(tmux.hasSession as ReturnType<typeof vi.fn>).mockResolvedValue(false);
|
||||
const orch = createOrchestrator({ tmux });
|
||||
const seeded = seedFailedAttempt({ restoreOutcome: "failed", withResumeToken: true });
|
||||
|
||||
const result = await orch.reconcileNodeRuntimeTruth(seeded.rig.id, seeded.nodeId);
|
||||
|
||||
expect(result.ok).toBe(false);
|
||||
if (!result.ok) expect(result.code).toBe("tmux_session_missing");
|
||||
const reconciled = db.prepare(
|
||||
"SELECT * FROM events WHERE type = 'restore.outcome_reconciled'"
|
||||
).all();
|
||||
expect(reconciled).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("no-op when foreground process is not runtime (precondition #2 fails)", async () => {
|
||||
const tmux = mockTmuxForReconciler();
|
||||
(tmux.hasSession as ReturnType<typeof vi.fn>).mockResolvedValue(true);
|
||||
(tmux.getPaneCommand as ReturnType<typeof vi.fn>).mockResolvedValue("zsh"); // shell, not claude/codex
|
||||
(tmux.capturePaneContent as ReturnType<typeof vi.fn>).mockResolvedValue("$ ");
|
||||
const orch = createOrchestrator({ tmux });
|
||||
const seeded = seedFailedAttempt({ restoreOutcome: "failed", withResumeToken: true });
|
||||
|
||||
const result = await orch.reconcileNodeRuntimeTruth(seeded.rig.id, seeded.nodeId);
|
||||
|
||||
expect(result.ok).toBe(false);
|
||||
if (!result.ok) expect(result.code).toBe("fg_process_not_runtime");
|
||||
});
|
||||
|
||||
it("no-op when resume token was not used (precondition #3 fails)", async () => {
|
||||
const tmux = mockTmuxForReconciler();
|
||||
(tmux.hasSession as ReturnType<typeof vi.fn>).mockResolvedValue(true);
|
||||
(tmux.getPaneCommand as ReturnType<typeof vi.fn>).mockResolvedValue("claude");
|
||||
(tmux.capturePaneContent as ReturnType<typeof vi.fn>).mockResolvedValue("Claude Code v2.1.89\n ❯ accept edits on");
|
||||
const orch = createOrchestrator({ tmux });
|
||||
const seeded = seedFailedAttempt({ restoreOutcome: "failed", withResumeToken: false });
|
||||
|
||||
const result = await orch.reconcileNodeRuntimeTruth(seeded.rig.id, seeded.nodeId);
|
||||
|
||||
expect(result.ok).toBe(false);
|
||||
if (!result.ok) expect(result.code).toBe("resume_token_not_used");
|
||||
});
|
||||
|
||||
it("no-op when pane is at Claude resume-selection prompt (precondition #4 fails)", async () => {
|
||||
const tmux = mockTmuxForReconciler();
|
||||
(tmux.hasSession as ReturnType<typeof vi.fn>).mockResolvedValue(true);
|
||||
(tmux.getPaneCommand as ReturnType<typeof vi.fn>).mockResolvedValue("claude");
|
||||
(tmux.capturePaneContent as ReturnType<typeof vi.fn>).mockResolvedValue([
|
||||
"Choose a conversation to resume:",
|
||||
" 1. project-foo",
|
||||
" 2. project-bar",
|
||||
].join("\n"));
|
||||
const orch = createOrchestrator({ tmux });
|
||||
const seeded = seedFailedAttempt({ restoreOutcome: "attention_required", withResumeToken: true });
|
||||
|
||||
const result = await orch.reconcileNodeRuntimeTruth(seeded.rig.id, seeded.nodeId);
|
||||
|
||||
expect(result.ok).toBe(false);
|
||||
if (!result.ok) expect(result.code).toBe("pane_not_usable");
|
||||
});
|
||||
|
||||
it("does not upgrade an outcome that is not failed/attention_required", async () => {
|
||||
// Seed a successfully-resumed outcome. Reconciler should refuse.
|
||||
const rig = rigRepo.createRig("r80");
|
||||
const node = rigRepo.addNode(rig.id, "worker", { role: "worker", runtime: "claude-code" });
|
||||
sessionRegistry.updateBinding(node.id, { tmuxSession: "r80-worker" });
|
||||
eventBus.emit({ type: "restore.started", rigId: rig.id, snapshotId: "snap-x" });
|
||||
eventBus.emit({
|
||||
type: "restore.completed",
|
||||
rigId: rig.id,
|
||||
snapshotId: "snap-x",
|
||||
result: {
|
||||
snapshotId: "snap-x",
|
||||
preRestoreSnapshotId: null,
|
||||
rigResult: "fully_restored",
|
||||
nodes: [{ nodeId: node.id, logicalId: "worker", status: "resumed" }],
|
||||
warnings: [],
|
||||
},
|
||||
});
|
||||
|
||||
const orch = createOrchestrator();
|
||||
const result = await orch.reconcileNodeRuntimeTruth(rig.id, node.id);
|
||||
|
||||
expect(result.ok).toBe(false);
|
||||
if (!result.ok) expect(result.code).toBe("outcome_not_upgradable");
|
||||
});
|
||||
|
||||
it("rejects when no restore.started has ever been recorded for the rig", async () => {
|
||||
const rig = rigRepo.createRig("r81");
|
||||
const node = rigRepo.addNode(rig.id, "worker", { role: "worker", runtime: "claude-code" });
|
||||
const orch = createOrchestrator();
|
||||
|
||||
const result = await orch.reconcileNodeRuntimeTruth(rig.id, node.id);
|
||||
|
||||
expect(result.ok).toBe(false);
|
||||
if (!result.ok) expect(result.code).toBe("no_attempt");
|
||||
});
|
||||
|
||||
it("never produces 'ready' as terminal outcome (Decision 3 forbids it)", async () => {
|
||||
// Even when all preconditions hold, the upgrade target is operator_recovered.
|
||||
const tmux = mockTmuxForReconciler();
|
||||
(tmux.hasSession as ReturnType<typeof vi.fn>).mockResolvedValue(true);
|
||||
(tmux.getPaneCommand as ReturnType<typeof vi.fn>).mockResolvedValue("claude");
|
||||
(tmux.capturePaneContent as ReturnType<typeof vi.fn>).mockResolvedValue("Claude Code v2.1.89\n ❯ accept edits on");
|
||||
const orch = createOrchestrator({ tmux });
|
||||
const seeded = seedFailedAttempt({ restoreOutcome: "failed", withResumeToken: true });
|
||||
|
||||
const result = await orch.reconcileNodeRuntimeTruth(seeded.rig.id, seeded.nodeId);
|
||||
|
||||
if (result.ok) {
|
||||
expect(result.to).toBe("operator_recovered");
|
||||
expect(result.to).not.toBe("ready");
|
||||
} else {
|
||||
// Did not upgrade — that's also fine, point is `to` was never `ready`.
|
||||
}
|
||||
|
||||
const reconciled = db.prepare(
|
||||
"SELECT payload FROM events WHERE type = 'restore.outcome_reconciled'"
|
||||
).all() as { payload: string }[];
|
||||
for (const row of reconciled) {
|
||||
const parsed = JSON.parse(row.payload) as { to: string };
|
||||
expect(parsed.to).toBe("operator_recovered");
|
||||
expect(parsed.to).not.toBe("ready");
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -150,18 +150,28 @@ describe("Restore routes", () => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
it("POST /api/rigs/:rigId/restore/:snapshotId -> 200 + RestoreResult", async () => {
|
||||
// L3: route returns 202 + attemptId immediately after restore.started emit.
|
||||
// Per-node restore work runs in the background; clients query event log /
|
||||
// node inventory for follow-up state.
|
||||
it("POST /api/rigs/:rigId/restore/:snapshotId -> 202 + attemptId (L3)", async () => {
|
||||
const rig = rigRepo.createRig("r99");
|
||||
rigRepo.addNode(rig.id, "worker", { role: "worker" });
|
||||
const snap = snapshotCapture.captureSnapshot(rig.id, "manual");
|
||||
|
||||
const res = await app.request(`/api/rigs/${rig.id}/restore/${snap.id}`, { method: "POST" });
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.status).toBe(202);
|
||||
const body = await res.json();
|
||||
expect(body.snapshotId).toBe(snap.id);
|
||||
expect(body.preRestoreSnapshotId).toBeDefined();
|
||||
expect(body.rigResult).toBe("partially_restored");
|
||||
expect(body.nodes).toHaveLength(1);
|
||||
expect(body.ok).toBe(true);
|
||||
expect(body.status).toBe("started");
|
||||
expect(body.rigId).toBe(rig.id);
|
||||
expect(typeof body.attemptId).toBe("number");
|
||||
expect(body.attemptId).toBeGreaterThan(0);
|
||||
|
||||
// attemptId must match a queryable restore.started event seq.
|
||||
const startedEvent = db
|
||||
.prepare("SELECT seq FROM events WHERE rig_id = ? AND type = 'restore.started' ORDER BY seq DESC LIMIT 1")
|
||||
.get(rig.id) as { seq: number } | undefined;
|
||||
expect(startedEvent?.seq).toBe(body.attemptId);
|
||||
});
|
||||
|
||||
it("POST restore returns 409 not_attempted when pre-restore validation blocks", async () => {
|
||||
@@ -267,20 +277,25 @@ describe("Restore routes", () => {
|
||||
db2.close();
|
||||
});
|
||||
|
||||
it("restore response includes nodes array with status per node", async () => {
|
||||
// L3: per-node detail is no longer in the immediate route response. Verify
|
||||
// the route returns the new shape; per-node statuses are exercised by
|
||||
// restore-orchestrator tests directly.
|
||||
it("restore response is asynchronous: 202 + attemptId, no per-node body", async () => {
|
||||
const rig = rigRepo.createRig("r99");
|
||||
rigRepo.addNode(rig.id, "worker-a", { role: "worker" });
|
||||
rigRepo.addNode(rig.id, "worker-b", { role: "worker" });
|
||||
const snap = snapshotCapture.captureSnapshot(rig.id, "manual");
|
||||
|
||||
const res = await app.request(`/api/rigs/${rig.id}/restore/${snap.id}`, { method: "POST" });
|
||||
expect(res.status).toBe(202);
|
||||
const body = await res.json();
|
||||
expect(body.rigResult).toBe("partially_restored");
|
||||
expect(body.nodes).toHaveLength(2);
|
||||
for (const node of body.nodes) {
|
||||
expect(node.logicalId).toBeDefined();
|
||||
expect(node.status).toBeDefined();
|
||||
}
|
||||
expect(body.ok).toBe(true);
|
||||
expect(body.status).toBe("started");
|
||||
expect(typeof body.attemptId).toBe("number");
|
||||
// The immediate response intentionally does NOT include nodes/rigResult;
|
||||
// per-node work runs in the background.
|
||||
expect(body.nodes).toBeUndefined();
|
||||
expect(body.rigResult).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -308,7 +323,11 @@ describe("Restore concurrency", () => {
|
||||
});
|
||||
|
||||
describe("Restore response contract", () => {
|
||||
it("restore response can contain 'checkpoint_written' status", async () => {
|
||||
// L3: per-node statuses are no longer in the immediate route response (route
|
||||
// returns 202 + attemptId). Per-node behavior is verified in restore-orchestrator
|
||||
// tests directly. This test now confirms the async contract holds even when a
|
||||
// checkpoint exists for the node (which previously yielded `rebuilt`).
|
||||
it("restore returns 202 + attemptId even when nodes have checkpoints (L3 async contract)", async () => {
|
||||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "rigged-test-"));
|
||||
const db2 = createFullTestDb();
|
||||
const setup = createTestApp(db2);
|
||||
@@ -322,9 +341,10 @@ describe("Restore response contract", () => {
|
||||
const snap = setup.snapshotCapture.captureSnapshot(rig.id, "manual");
|
||||
|
||||
const res = await setup.app.request(`/api/rigs/${rig.id}/restore/${snap.id}`, { method: "POST" });
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.status).toBe(202);
|
||||
const body = await res.json();
|
||||
expect(body.nodes[0].status).toBe("rebuilt");
|
||||
expect(body.ok).toBe(true);
|
||||
expect(typeof body.attemptId).toBe("number");
|
||||
|
||||
db2.close();
|
||||
fs.rmSync(tmpDir, { recursive: true });
|
||||
|
||||
Reference in new issue
Block a user