mirror of
https://github.com/mvschwarz/openrig.git
synced 2026-11-08 08:11:17 -10:00
feat(daemon,cli): add recoverable/degraded/attention_required lifecycle projection
Lifecycle v2 L2 builds on the L1 cold-start truth repair (3ad1df6) by giving operators a first-class projection that distinguishes recoverable rigs/nodes from genuinely-stopped ones. Pre-L2, a detached session row and a stopped-from- scratch rig look identical to `rig ps` and `rig up`; the wording divergence between "Recover from snapshot" and "Turn on" is unreachable without this state. All new state is DERIVED — no schema migration, no new column, no auto-restore on daemon start. Decisions per plan packet: 1. v0 derives `recoverable` from session detachment + snapshot resume metadata; persistence deferred until performance/projection needs justify it. 2. CLI says "Recover" when the rig's latest usable snapshot can resume; says "Turn on" when no usable snapshot exists. Help-text honesty: "Recover" describes what `rig up` does, it does not promise success before tester L4. 3. Rig-level `degraded` = mixed live + detached on the same rig; per-node `attention_required` = the post-L3 Claude resume-prompt proxy (failed restoreOutcome with a still-alive tmux session). Per-node lifecycleState (additive on NodeInventoryEntry): attention_required > running > recoverable > detached, in priority order. Per-rig lifecycleState (additive on PsEntry): any attention_required → attention_required; all running → running; all non-running, any recoverable → recoverable; all non-running, none recoverable → stopped; mixed → degraded. Implementation: - packages/daemon/src/domain/types.ts: declare NodeLifecycleState + RigLifecycleState; add lifecycleState to NodeInventoryEntry. - packages/daemon/src/domain/rig-repository.ts: free function plus instance method `findLatestUsableSnapshot(rigId)` returns the latest snapshot whose data carries at least one non-null resume token; returns null otherwise. - packages/daemon/src/domain/node-inventory.ts: new helper `deriveNodeLifecycleState`; getNodeInventory loads the rig's usable snapshot once and folds per-node state. - packages/daemon/src/domain/ps-projection.ts: new helper `deriveRigLifecycleState`; getEntries derives per-rig lifecycleState by folding per-node states from getNodeInventory. - packages/daemon/src/routes/rigs.ts: GET /api/rigs/summary enriched with lifecycleState so CLI surfaces (rig up wording) work without a second round trip. - packages/cli/src/commands/ps.ts: add LIFECYCLE column to rig + node tables with abbreviated codes (run/rec/stp/deg/att, run/rec/det/att); JSON output passes through lifecycleState from API. - packages/cli/src/commands/up.ts: when isRigName resolves to an existing rig (no library spec match), fetch /api/rigs/summary and print "Recovering rig 'X' from latest snapshot..." for recoverable, "Turning on rig 'X'..." for stopped. JSON output is unaffected (no prefix). Tests added (matching plan closeout checklist): - packages/daemon/test/node-inventory.test.ts: 6 lifecycleState cases (running, recoverable with token, detached when token absent, detached with no snapshot, attention_required on failed+alive, exited+detached). - packages/daemon/test/ps-projection.test.ts: 5 rig-level cases plus deriveRigLifecycleState fold helper full-branch coverage. Also migrated the test fixture to createFullTestDb so the full migration set (including pods) is available — required because ps-projection now folds via getNodeInventory. - packages/daemon/test/rig-repository.test.ts: 5 findLatestUsableSnapshot cases (no snapshot, all-null tokens, partial tokens, latest-of-many ordering, returns even when only some nodes have tokens — per-node state then determines individual recoverability). - packages/cli/test/ps.test.ts: 4 cases (rig table LIFECYCLE column with all five abbreviated codes; rig JSON includes lifecycleState; node table column; node JSON includes lifecycleState). - packages/cli/test/up.test.ts: 4 wording-divergence cases (recoverable prints Recovering, stopped prints Turning on, running prints neither, --json output is clean). Verification: - npx vitest run focused (daemon node-inventory + ps-projection + rig-repository): 62/62 green. - npx vitest run focused (cli ps + up): 43/43 green. - Full daemon suite: 2052/2052 green. - Full CLI suite has 4 pre-existing failures (port 7433 in use, chatroom timeout flaky, config reads env) — confirmed by running them on the pre-L2 baseline. Not caused by L2. - npx tsc --noEmit clean on both packages. Invariants preserved: - No persisted column added (decision #1 derive-only). - Existing `sessionStatus`/`startupStatus`/`restoreOutcome` fields unchanged — `lifecycleState` is additive. - No auto-restore on daemon start; `createDaemon` reconcile loop is unchanged from L1. - Help text says "Recovering ... from latest snapshot" which describes what `rig up` does without promising success before tester L4 VM proof. - `recoverable` derivation handles null fields gracefully (snapshot exists but token is null → node stays detached, not falsely recoverable). - Permission errors (L1 fail-closed) propagate unchanged through the new projection; lifecycleState only consults sessionStatus + restoreOutcome + snapshot data. L2 implementation accept does NOT close L2 — full closure waits on tester L4 VM-reboot end-to-end proof, mirror of L1. Refs: - rigs/openrig-build/state/orch/velocity-slice-L2-recoverable-state-safe-defaults-plan-20260428.md - L1 predecessor commit:3ad1df6fix(daemon): classify tmux socket-absence as missing session and log startup reconcile Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
1 parent
3ad1df6339
commit
fad10a9311
12 files changed
+740
-33
No files matched your search
@@ -10,6 +10,7 @@ interface PsEntry {
|
||||
nodeCount: number;
|
||||
runningCount: number;
|
||||
status: "running" | "partial" | "stopped";
|
||||
lifecycleState?: "running" | "recoverable" | "stopped" | "degraded" | "attention_required";
|
||||
uptime: string | null;
|
||||
latestSnapshot: string | null;
|
||||
}
|
||||
@@ -26,6 +27,7 @@ interface NodeEntry {
|
||||
sessionStatus: string | null;
|
||||
startupStatus: "pending" | "ready" | "attention_required" | "failed" | null;
|
||||
restoreOutcome: string;
|
||||
lifecycleState?: "running" | "detached" | "recoverable" | "attention_required";
|
||||
tmuxAttachCommand: string | null;
|
||||
resumeCommand: string | null;
|
||||
latestError: string | null;
|
||||
@@ -39,6 +41,27 @@ interface NodeEntry {
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
// Compact rig-level lifecycle codes for the rig table header (3-char fixed width).
|
||||
function abbrevRigLifecycle(state: PsEntry["lifecycleState"] | undefined): string {
|
||||
if (!state) return "—";
|
||||
if (state === "running") return "run";
|
||||
if (state === "recoverable") return "rec";
|
||||
if (state === "stopped") return "stp";
|
||||
if (state === "degraded") return "deg";
|
||||
if (state === "attention_required") return "att";
|
||||
return "—";
|
||||
}
|
||||
|
||||
// Compact per-node lifecycle codes for the nodes table.
|
||||
function abbrevNodeLifecycle(state: NodeEntry["lifecycleState"] | undefined): string {
|
||||
if (!state) return "—";
|
||||
if (state === "running") return "run";
|
||||
if (state === "recoverable") return "rec";
|
||||
if (state === "detached") return "det";
|
||||
if (state === "attention_required") return "att";
|
||||
return "—";
|
||||
}
|
||||
|
||||
/**
|
||||
* `rig ps` — list running rigs and optionally their nodes.
|
||||
* @param depsOverride - injectable deps for testing
|
||||
@@ -101,7 +124,7 @@ Exit codes:
|
||||
}
|
||||
|
||||
// Formatted table
|
||||
const header = padRigRow("RIG", "NODES", "RUNNING", "STATUS", "UPTIME", "SNAPSHOT");
|
||||
const header = padRigRow("RIG", "NODES", "RUNNING", "STATUS", "LIFECYCLE", "UPTIME", "SNAPSHOT");
|
||||
console.log(header);
|
||||
for (const e of entries) {
|
||||
console.log(padRigRow(
|
||||
@@ -109,6 +132,7 @@ Exit codes:
|
||||
String(e.nodeCount),
|
||||
String(e.runningCount),
|
||||
e.status,
|
||||
abbrevRigLifecycle(e.lifecycleState),
|
||||
e.uptime ?? "—",
|
||||
e.latestSnapshot ?? "—",
|
||||
));
|
||||
@@ -147,7 +171,7 @@ async function handleNodes(client: DaemonClient, json: boolean): Promise<void> {
|
||||
return;
|
||||
}
|
||||
|
||||
const header = padNodeRow("RIG", "POD", "MEMBER", "SESSION", "RUNTIME", "STATUS", "STARTUP", "ACTIVITY", "RESTORE", "ERROR");
|
||||
const header = padNodeRow("RIG", "POD", "MEMBER", "SESSION", "RUNTIME", "STATUS", "STARTUP", "LIFECYCLE", "ACTIVITY", "RESTORE", "ERROR");
|
||||
console.log(header);
|
||||
for (const n of allNodes) {
|
||||
const parts = n.logicalId.split(".");
|
||||
@@ -162,6 +186,7 @@ async function handleNodes(client: DaemonClient, json: boolean): Promise<void> {
|
||||
n.runtime ?? "—",
|
||||
n.sessionStatus ?? "—",
|
||||
n.startupStatus ?? "—",
|
||||
abbrevNodeLifecycle(n.lifecycleState),
|
||||
formatActivity(n.agentActivity),
|
||||
n.restoreOutcome,
|
||||
n.latestError ? truncate(n.latestError, 30) : "—",
|
||||
@@ -185,18 +210,19 @@ function fitCell(value: string, width: number): string {
|
||||
return truncate(value, width).padEnd(width);
|
||||
}
|
||||
|
||||
function padRigRow(rig: string, nodes: string, running: string, status: string, uptime: string, snapshot: string): string {
|
||||
function padRigRow(rig: string, nodes: string, running: string, status: string, lifecycle: string, uptime: string, snapshot: string): string {
|
||||
return [
|
||||
fitCell(rig, 24),
|
||||
fitCell(nodes, 7),
|
||||
fitCell(running, 9),
|
||||
fitCell(status, 10),
|
||||
fitCell(lifecycle, 11),
|
||||
fitCell(uptime, 11),
|
||||
snapshot,
|
||||
].join("");
|
||||
}
|
||||
|
||||
function padNodeRow(rig: string, pod: string, member: string, session: string, runtime: string, status: string, startup: string, activity: string, restore: string, error: string): string {
|
||||
function padNodeRow(rig: string, pod: string, member: string, session: string, runtime: string, status: string, startup: string, lifecycle: string, activity: string, restore: string, error: string): string {
|
||||
return [
|
||||
fitCell(rig, 30),
|
||||
fitCell(pod, 10),
|
||||
@@ -205,6 +231,7 @@ function padNodeRow(rig: string, pod: string, member: string, session: string, r
|
||||
fitCell(runtime, 12),
|
||||
fitCell(status, 10),
|
||||
fitCell(startup, 10),
|
||||
fitCell(lifecycle, 11),
|
||||
fitCell(activity, 12),
|
||||
fitCell(restore, 10),
|
||||
error,
|
||||
|
||||
@@ -102,14 +102,27 @@ Examples:
|
||||
|
||||
// If it looks like a name, check for library spec match
|
||||
let defaultLibraryCwdOverride: string | undefined;
|
||||
// Cache rig summaries so we can both detect ambiguity AND derive lifecycleState
|
||||
// for "Recovering ..." vs "Turning on ..." wording (post-L2).
|
||||
let rigSummariesCache: Array<{ id: string; name: string; lifecycleState?: string }> | null = null;
|
||||
const fetchRigSummaries = async () => {
|
||||
if (rigSummariesCache !== null) return rigSummariesCache;
|
||||
try {
|
||||
const res = await client.get<Array<{ id: string; name: string; lifecycleState?: string }>>("/api/rigs/summary");
|
||||
rigSummariesCache = res.data ?? [];
|
||||
} catch {
|
||||
rigSummariesCache = [];
|
||||
}
|
||||
return rigSummariesCache;
|
||||
};
|
||||
if (isRigName) {
|
||||
try {
|
||||
const { resolveLibrarySpec } = await import("./specs.js");
|
||||
const entry = await resolveLibrarySpec(client, source);
|
||||
// Library match found — check for existing-rig collision
|
||||
// Use /api/rigs/summary which mirrors findRigsByName (includes stopped rigs)
|
||||
const rigRes = await client.get<Array<{ id: string; name: string }>>("/api/rigs/summary");
|
||||
const rigMatches = (rigRes.data ?? []).filter((r) => r.name === source);
|
||||
const rigSummaries = await fetchRigSummaries();
|
||||
const rigMatches = rigSummaries.filter((r) => r.name === source);
|
||||
if (rigMatches.length > 0) {
|
||||
console.error(`'${source}' is ambiguous — it matches both an existing rig restore target and a library spec.`);
|
||||
console.error(` To launch the library spec: rig up ${entry.sourcePath}`);
|
||||
@@ -131,6 +144,21 @@ Examples:
|
||||
}
|
||||
// Not found or other error — proceed with existing rig-name behavior
|
||||
}
|
||||
|
||||
// Wording divergence (post-L2): if sourceRef stayed as the rig name we are
|
||||
// routing through the existing-rig restore path. Print "Recovering ..." or
|
||||
// "Turning on ..." per the rig's derived lifecycleState. Help text honesty:
|
||||
// "Recover" describes what `rig up` does; it does not promise success before
|
||||
// tester L4 VM proof completes.
|
||||
if (sourceRef === source && !opts.json) {
|
||||
const summaries = await fetchRigSummaries();
|
||||
const match = summaries.find((r) => r.name === source);
|
||||
if (match?.lifecycleState === "recoverable") {
|
||||
console.log(`Recovering rig "${source}" from latest snapshot...`);
|
||||
} else if (match?.lifecycleState === "stopped") {
|
||||
console.log(`Turning on rig "${source}"...`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const isRigBundle = !isRigName && /\.rigbundle$/i.test(sourceRef);
|
||||
|
||||
@@ -412,4 +412,89 @@ describe("Ps CLI", () => {
|
||||
expect(logs.some((l) => l.includes("rig daemon start"))).toBe(true);
|
||||
expect(exitCode).toBe(1);
|
||||
});
|
||||
|
||||
// L2 lifecycle column / JSON shape (rig + node level)
|
||||
it("ps prints LIFECYCLE column with abbreviated rig-level state", async () => {
|
||||
psData = [
|
||||
{ rigId: "rig-1", name: "running-rig", nodeCount: 2, runningCount: 2, status: "running", lifecycleState: "running", uptime: "2h", latestSnapshot: null },
|
||||
{ rigId: "rig-2", name: "recover-rig", nodeCount: 2, runningCount: 0, status: "stopped", lifecycleState: "recoverable", uptime: null, latestSnapshot: "1h ago" },
|
||||
{ rigId: "rig-3", name: "stop-rig", nodeCount: 1, runningCount: 0, status: "stopped", lifecycleState: "stopped", uptime: null, latestSnapshot: null },
|
||||
{ rigId: "rig-4", name: "deg-rig", nodeCount: 2, runningCount: 1, status: "partial", lifecycleState: "degraded", uptime: "30m", latestSnapshot: null },
|
||||
{ rigId: "rig-5", name: "att-rig", nodeCount: 2, runningCount: 2, status: "running", lifecycleState: "attention_required", uptime: "10m", latestSnapshot: null },
|
||||
];
|
||||
const { logs } = await captureLogs(async () => {
|
||||
await makeCmd().parseAsync(["node", "rig", "ps"]);
|
||||
});
|
||||
const output = logs.join("\n");
|
||||
expect(output).toContain("LIFECYCLE");
|
||||
expect(output).toContain("run");
|
||||
expect(output).toContain("rec");
|
||||
expect(output).toContain("stp");
|
||||
expect(output).toContain("deg");
|
||||
expect(output).toContain("att");
|
||||
});
|
||||
|
||||
it("ps --json includes lifecycleState in rig entries", async () => {
|
||||
psData = [
|
||||
{ rigId: "rig-1", name: "test", nodeCount: 1, runningCount: 0, status: "stopped", lifecycleState: "recoverable", uptime: null, latestSnapshot: "5m ago" },
|
||||
];
|
||||
const { logs } = await captureLogs(async () => {
|
||||
await makeCmd().parseAsync(["node", "rig", "ps", "--json"]);
|
||||
});
|
||||
const parsed = JSON.parse(logs.join(""));
|
||||
expect(parsed[0].lifecycleState).toBe("recoverable");
|
||||
});
|
||||
|
||||
it("ps --nodes prints LIFECYCLE column with per-node abbreviated state", async () => {
|
||||
psData = [
|
||||
{ rigId: "rig-1", name: "mixed-rig", nodeCount: 3, runningCount: 1, status: "partial", lifecycleState: "degraded", uptime: "15m", latestSnapshot: null },
|
||||
];
|
||||
nodesData["rig-1"] = [
|
||||
{
|
||||
rigId: "rig-1", rigName: "mixed-rig", logicalId: "dev.impl", podId: "pod-1", podNamespace: "dev",
|
||||
canonicalSessionName: "dev-impl@mixed-rig", nodeKind: "agent", runtime: "claude-code",
|
||||
sessionStatus: "running", startupStatus: "ready", restoreOutcome: "n-a", lifecycleState: "running",
|
||||
tmuxAttachCommand: null, resumeCommand: null, latestError: null,
|
||||
},
|
||||
{
|
||||
rigId: "rig-1", rigName: "mixed-rig", logicalId: "dev.qa", podId: "pod-1", podNamespace: "dev",
|
||||
canonicalSessionName: "dev-qa@mixed-rig", nodeKind: "agent", runtime: "claude-code",
|
||||
sessionStatus: "detached", startupStatus: "ready", restoreOutcome: "n-a", lifecycleState: "recoverable",
|
||||
tmuxAttachCommand: null, resumeCommand: null, latestError: null,
|
||||
},
|
||||
{
|
||||
rigId: "rig-1", rigName: "mixed-rig", logicalId: "dev.lead", podId: "pod-1", podNamespace: "dev",
|
||||
canonicalSessionName: "dev-lead@mixed-rig", nodeKind: "agent", runtime: "claude-code",
|
||||
sessionStatus: "detached", startupStatus: "ready", restoreOutcome: "n-a", lifecycleState: "detached",
|
||||
tmuxAttachCommand: null, resumeCommand: null, latestError: null,
|
||||
},
|
||||
];
|
||||
const { logs } = await captureLogs(async () => {
|
||||
await makeCmd().parseAsync(["node", "rig", "ps", "--nodes"]);
|
||||
});
|
||||
const output = logs.join("\n");
|
||||
expect(output).toContain("LIFECYCLE");
|
||||
expect(output).toContain("run");
|
||||
expect(output).toContain("rec");
|
||||
expect(output).toContain("det");
|
||||
});
|
||||
|
||||
it("ps --nodes --json includes lifecycleState per node", async () => {
|
||||
psData = [
|
||||
{ rigId: "rig-1", name: "test", nodeCount: 1, runningCount: 0, status: "stopped", lifecycleState: "recoverable", uptime: null, latestSnapshot: null },
|
||||
];
|
||||
nodesData["rig-1"] = [
|
||||
{
|
||||
rigId: "rig-1", rigName: "test", logicalId: "dev.impl", podId: "pod-1", podNamespace: "dev",
|
||||
canonicalSessionName: "dev-impl@test", nodeKind: "agent", runtime: "claude-code",
|
||||
sessionStatus: "detached", startupStatus: "ready", restoreOutcome: "n-a", lifecycleState: "recoverable",
|
||||
tmuxAttachCommand: null, resumeCommand: null, latestError: null,
|
||||
},
|
||||
];
|
||||
const { logs } = await captureLogs(async () => {
|
||||
await makeCmd().parseAsync(["node", "rig", "ps", "--nodes", "--json"]);
|
||||
});
|
||||
const parsed = JSON.parse(logs.join(""));
|
||||
expect(parsed[0].lifecycleState).toBe("recoverable");
|
||||
});
|
||||
});
|
||||
@@ -689,4 +689,141 @@ describe("Up CLI", () => {
|
||||
expect(logs.join("\n")).toContain("/specs/alpha.yaml");
|
||||
expect(exitCode).toBe(1);
|
||||
});
|
||||
|
||||
// L2 wording divergence: rig name in recoverable state -> "Recovering ..."; stopped -> "Turning on ..."
|
||||
it("up <rig-name> prints 'Recovering ... from latest snapshot' when lifecycleState=recoverable", async () => {
|
||||
const origListeners = server.listeners("request");
|
||||
server.removeAllListeners("request");
|
||||
server.on("request", async (req: http.IncomingMessage, res: http.ServerResponse) => {
|
||||
let body = "";
|
||||
for await (const chunk of req) body += chunk;
|
||||
if (req.url === "/api/specs/library" && req.method === "GET") {
|
||||
// No library match — falls through to existing-rig path
|
||||
res.writeHead(200, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify([]));
|
||||
} else if (req.url === "/api/rigs/summary" && req.method === "GET") {
|
||||
res.writeHead(200, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify([{ id: "r-rec", name: "stale-velocity", nodeCount: 4, lifecycleState: "recoverable" }]));
|
||||
} else if (req.url === "/api/up" && req.method === "POST") {
|
||||
res.writeHead(201, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify({ status: "restored", rigId: "r-rec", rigName: "stale-velocity", rigResult: "restored", nodes: [], warnings: [] }));
|
||||
} else {
|
||||
res.writeHead(404);
|
||||
res.end();
|
||||
}
|
||||
});
|
||||
|
||||
const { logs, exitCode } = await captureLogs(async () => {
|
||||
await makeCmd().parseAsync(["node", "rig", "up", "stale-velocity"]);
|
||||
});
|
||||
|
||||
server.removeAllListeners("request");
|
||||
for (const l of origListeners) server.on("request", l as (...args: unknown[]) => void);
|
||||
|
||||
const output = logs.join("\n");
|
||||
expect(output).toContain('Recovering rig "stale-velocity" from latest snapshot');
|
||||
expect(output).not.toContain('Turning on rig "stale-velocity"');
|
||||
expect(exitCode).toBeUndefined(); // 0
|
||||
});
|
||||
|
||||
it("up <rig-name> prints 'Turning on ...' when lifecycleState=stopped (no usable snapshot)", async () => {
|
||||
const origListeners = server.listeners("request");
|
||||
server.removeAllListeners("request");
|
||||
server.on("request", async (req: http.IncomingMessage, res: http.ServerResponse) => {
|
||||
let body = "";
|
||||
for await (const chunk of req) body += chunk;
|
||||
if (req.url === "/api/specs/library" && req.method === "GET") {
|
||||
res.writeHead(200, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify([]));
|
||||
} else if (req.url === "/api/rigs/summary" && req.method === "GET") {
|
||||
res.writeHead(200, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify([{ id: "r-stop", name: "fresh-rig", nodeCount: 1, lifecycleState: "stopped" }]));
|
||||
} else if (req.url === "/api/up" && req.method === "POST") {
|
||||
res.writeHead(201, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify({ status: "completed", rigId: "r-stop", stages: [], errors: [], warnings: [] }));
|
||||
} else {
|
||||
res.writeHead(404);
|
||||
res.end();
|
||||
}
|
||||
});
|
||||
|
||||
const { logs, exitCode } = await captureLogs(async () => {
|
||||
await makeCmd().parseAsync(["node", "rig", "up", "fresh-rig"]);
|
||||
});
|
||||
|
||||
server.removeAllListeners("request");
|
||||
for (const l of origListeners) server.on("request", l as (...args: unknown[]) => void);
|
||||
|
||||
const output = logs.join("\n");
|
||||
expect(output).toContain('Turning on rig "fresh-rig"');
|
||||
expect(output).not.toContain('Recovering rig "fresh-rig"');
|
||||
expect(exitCode).toBeUndefined();
|
||||
});
|
||||
|
||||
it("up <rig-name> prints no recovery wording when lifecycleState=running (already up)", async () => {
|
||||
const origListeners = server.listeners("request");
|
||||
server.removeAllListeners("request");
|
||||
server.on("request", async (req: http.IncomingMessage, res: http.ServerResponse) => {
|
||||
let body = "";
|
||||
for await (const chunk of req) body += chunk;
|
||||
if (req.url === "/api/specs/library" && req.method === "GET") {
|
||||
res.writeHead(200, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify([]));
|
||||
} else if (req.url === "/api/rigs/summary" && req.method === "GET") {
|
||||
res.writeHead(200, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify([{ id: "r-run", name: "live-rig", nodeCount: 1, lifecycleState: "running" }]));
|
||||
} else if (req.url === "/api/up" && req.method === "POST") {
|
||||
res.writeHead(201, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify({ status: "completed", rigId: "r-run", stages: [], errors: [], warnings: [] }));
|
||||
} else {
|
||||
res.writeHead(404);
|
||||
res.end();
|
||||
}
|
||||
});
|
||||
|
||||
const { logs, exitCode } = await captureLogs(async () => {
|
||||
await makeCmd().parseAsync(["node", "rig", "up", "live-rig"]);
|
||||
});
|
||||
|
||||
server.removeAllListeners("request");
|
||||
for (const l of origListeners) server.on("request", l as (...args: unknown[]) => void);
|
||||
|
||||
const output = logs.join("\n");
|
||||
expect(output).not.toContain("Recovering rig");
|
||||
expect(output).not.toContain("Turning on rig");
|
||||
expect(exitCode).toBeUndefined();
|
||||
});
|
||||
|
||||
it("up --json suppresses the wording prefix even for recoverable rigs (machine output stays clean)", async () => {
|
||||
const origListeners = server.listeners("request");
|
||||
server.removeAllListeners("request");
|
||||
server.on("request", async (req: http.IncomingMessage, res: http.ServerResponse) => {
|
||||
let body = "";
|
||||
for await (const chunk of req) body += chunk;
|
||||
if (req.url === "/api/specs/library" && req.method === "GET") {
|
||||
res.writeHead(200, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify([]));
|
||||
} else if (req.url === "/api/rigs/summary" && req.method === "GET") {
|
||||
res.writeHead(200, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify([{ id: "r-rec", name: "json-rig", nodeCount: 1, lifecycleState: "recoverable" }]));
|
||||
} else if (req.url === "/api/up" && req.method === "POST") {
|
||||
res.writeHead(201, { "Content-Type": "application/json" });
|
||||
res.end(JSON.stringify({ status: "restored", rigId: "r-rec", rigName: "json-rig", rigResult: "restored", nodes: [], warnings: [] }));
|
||||
} else {
|
||||
res.writeHead(404);
|
||||
res.end();
|
||||
}
|
||||
});
|
||||
|
||||
const { logs } = await captureLogs(async () => {
|
||||
await makeCmd().parseAsync(["node", "rig", "up", "json-rig", "--json"]);
|
||||
});
|
||||
|
||||
server.removeAllListeners("request");
|
||||
for (const l of origListeners) server.on("request", l as (...args: unknown[]) => void);
|
||||
|
||||
// The body must be valid JSON (no preceding "Recovering ..." text).
|
||||
const joined = logs.join("");
|
||||
expect(() => JSON.parse(joined)).not.toThrow();
|
||||
});
|
||||
});
|
||||
@@ -1,10 +1,11 @@
|
||||
import type Database from "better-sqlite3";
|
||||
import type { NodeInventoryEntry, NodeDetailEntry, NodeDetailPeer, NodeDetailEdge, NodeDetailCompactSpec, NodeRestoreOutcome, Binding, RestoreResult, NodeRecoveryGuidance } from "./types.js";
|
||||
import type { NodeInventoryEntry, NodeDetailEntry, NodeDetailPeer, NodeDetailEdge, NodeDetailCompactSpec, NodeRestoreOutcome, NodeLifecycleState, Binding, RestoreResult, NodeRecoveryGuidance, Snapshot } from "./types.js";
|
||||
import type { RuntimeAdapter } from "./runtime-adapter.js";
|
||||
import type { ContextUsageStore } from "./context-usage-store.js";
|
||||
import type { AgentActivityStore } from "./agent-activity-store.js";
|
||||
import type { TmuxAdapter } from "../adapters/tmux.js";
|
||||
import { probeSessionActivity } from "./session-transport.js";
|
||||
import { findLatestUsableSnapshot } from "./rig-repository.js";
|
||||
|
||||
// -- Row types for SQL results --
|
||||
|
||||
@@ -152,6 +153,42 @@ function deriveNodeKind(runtime: string | null): "agent" | "infrastructure" {
|
||||
return runtime === "terminal" ? "infrastructure" : "agent";
|
||||
}
|
||||
|
||||
/**
|
||||
* Derives per-node lifecycle state from session/restore truth plus the rig's latest
|
||||
* usable snapshot.
|
||||
*
|
||||
* Priorities (post-L2):
|
||||
* attention_required — restoreOutcome=failed AND tmux session is alive
|
||||
* (v0 proxy for the Claude resume-prompt case; revisited in L3).
|
||||
* running — sessionStatus=running.
|
||||
* recoverable — non-running session AND the latest usable snapshot has a
|
||||
* non-null resume token for THIS node.
|
||||
* detached — anything else (no session, exited, detached without resume token).
|
||||
*
|
||||
* Permission/IO failures upstream (L1 fail-closed) leave sessionStatus unchanged, so
|
||||
* the projection stays honest without misclassifying ambiguous probe failures.
|
||||
*/
|
||||
export function deriveNodeLifecycleState(input: {
|
||||
sessionStatus: string | null;
|
||||
restoreOutcome: NodeRestoreOutcome;
|
||||
nodeId: string;
|
||||
usableSnapshot: Snapshot | null;
|
||||
}): NodeLifecycleState {
|
||||
if (input.restoreOutcome === "failed" && input.sessionStatus === "running") {
|
||||
return "attention_required";
|
||||
}
|
||||
if (input.sessionStatus === "running") return "running";
|
||||
if (input.usableSnapshot) {
|
||||
const nodeSession = (input.usableSnapshot.data.sessions ?? []).find(
|
||||
(s) => s.nodeId === input.nodeId,
|
||||
);
|
||||
if (typeof nodeSession?.resumeToken === "string" && nodeSession.resumeToken.length > 0) {
|
||||
return "recoverable";
|
||||
}
|
||||
}
|
||||
return "detached";
|
||||
}
|
||||
|
||||
function deriveOccupantLifecycle(row: InventoryRow): NodeInventoryEntry["occupantLifecycle"] {
|
||||
if (row.occupant_lifecycle) {
|
||||
return row.occupant_lifecycle as NodeInventoryEntry["occupantLifecycle"];
|
||||
@@ -233,6 +270,10 @@ function mapProjectionEntries(entries: unknown[]): Array<{ id: string; category:
|
||||
* Single source of truth consumed by CLI, UI, and MCP.
|
||||
*/
|
||||
export function getNodeInventory(db: Database.Database, rigId: string): NodeInventoryEntry[] {
|
||||
// Resolve the rig's latest usable snapshot once for the whole projection so per-node
|
||||
// recoverability checks share the same source of truth without N extra queries.
|
||||
const usableSnapshot = findLatestUsableSnapshot(db, rigId);
|
||||
|
||||
// Join nodes with newest session (max ULID = max session.id string comparison)
|
||||
// and the rig name
|
||||
const hasCodexConfigProfile = db.prepare("PRAGMA table_info(nodes)").all()
|
||||
@@ -282,6 +323,12 @@ export function getNodeInventory(db: Database.Database, rigId: string): NodeInve
|
||||
|
||||
return rows.map((row) => {
|
||||
const restoreOutcome = deriveRestoreOutcome(db, rigId, row.node_id);
|
||||
const lifecycleState = deriveNodeLifecycleState({
|
||||
sessionStatus: row.session_status,
|
||||
restoreOutcome,
|
||||
nodeId: row.node_id,
|
||||
usableSnapshot,
|
||||
});
|
||||
return {
|
||||
rigId: row.rig_id,
|
||||
rigName: row.rig_name,
|
||||
@@ -295,6 +342,7 @@ export function getNodeInventory(db: Database.Database, rigId: string): NodeInve
|
||||
sessionStatus: row.session_status,
|
||||
startupStatus: row.startup_status as NodeInventoryEntry["startupStatus"],
|
||||
restoreOutcome,
|
||||
lifecycleState,
|
||||
occupantLifecycle: deriveOccupantLifecycle(row),
|
||||
continuityOutcome: deriveContinuityOutcome(row, restoreOutcome),
|
||||
handoverResult: row.handover_result as NodeInventoryEntry["handoverResult"] ?? null,
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
import type Database from "better-sqlite3";
|
||||
import type { NodeLifecycleState, RigLifecycleState } from "./types.js";
|
||||
import { getNodeInventory } from "./node-inventory.js";
|
||||
|
||||
export interface PsEntry {
|
||||
rigId: string;
|
||||
@@ -6,10 +8,42 @@ export interface PsEntry {
|
||||
nodeCount: number;
|
||||
runningCount: number;
|
||||
status: "running" | "partial" | "stopped";
|
||||
lifecycleState: RigLifecycleState;
|
||||
uptime: string | null;
|
||||
latestSnapshot: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Folds per-node lifecycle states into a rig-level lifecycle state.
|
||||
*
|
||||
* attention_required > running > recoverable > stopped, with `degraded` for mixes.
|
||||
*
|
||||
* Rules (post-L2):
|
||||
* - any node attention_required → attention_required (priority over below).
|
||||
* - all nodes running → running.
|
||||
* - all nodes non-running, any recoverable → recoverable.
|
||||
* - all nodes non-running, none recoverable → stopped.
|
||||
* - mixed running + non-running → degraded.
|
||||
* - empty rig (no nodes) → stopped.
|
||||
*
|
||||
* `recoverable` at rig level depends on per-node recoverability, which already accounts
|
||||
* for whether the rig's latest usable snapshot has a resume token for that node.
|
||||
*/
|
||||
export function deriveRigLifecycleState(nodeStates: NodeLifecycleState[]): RigLifecycleState {
|
||||
if (nodeStates.length === 0) return "stopped";
|
||||
if (nodeStates.some((s) => s === "attention_required")) return "attention_required";
|
||||
|
||||
const runningCount = nodeStates.filter((s) => s === "running").length;
|
||||
const totalCount = nodeStates.length;
|
||||
|
||||
if (runningCount === totalCount) return "running";
|
||||
if (runningCount === 0) {
|
||||
const anyRecoverable = nodeStates.some((s) => s === "recoverable");
|
||||
return anyRecoverable ? "recoverable" : "stopped";
|
||||
}
|
||||
return "degraded";
|
||||
}
|
||||
|
||||
/**
|
||||
* Projects rig/run summary for `rig ps`.
|
||||
* Aggregates across all rigs: node counts, running counts, status, uptime, snapshot age.
|
||||
@@ -56,12 +90,18 @@ export class PsProjectionService {
|
||||
: r.running_count > 0 ? "partial"
|
||||
: "stopped";
|
||||
|
||||
// Derive rig-level lifecycleState by folding per-node states. The cost is one
|
||||
// node-inventory pass per rig; this matches the v0 "derive, no migration" decision.
|
||||
const inventory = getNodeInventory(this.db, r.rig_id);
|
||||
const lifecycleState = deriveRigLifecycleState(inventory.map((e) => e.lifecycleState));
|
||||
|
||||
return {
|
||||
rigId: r.rig_id,
|
||||
name: r.name,
|
||||
nodeCount: r.node_count,
|
||||
runningCount: r.running_count,
|
||||
status,
|
||||
lifecycleState,
|
||||
uptime: r.earliest_running_at ? formatDuration(now - new Date(r.earliest_running_at + "Z").getTime()) : null,
|
||||
latestSnapshot: r.latest_snapshot_at ? formatAge(now - new Date(r.latest_snapshot_at + "Z").getTime()) : null,
|
||||
};
|
||||
|
||||
@@ -11,8 +11,46 @@ import type {
|
||||
RigWithRelations,
|
||||
RigServicesRecord,
|
||||
RigServicesRecordInput,
|
||||
Snapshot,
|
||||
SnapshotData,
|
||||
} from "./types.js";
|
||||
|
||||
/**
|
||||
* Free helper: returns the latest snapshot for a rig whose data carries non-null
|
||||
* `resume_token` for at least one persisted session, or null otherwise.
|
||||
*
|
||||
* Used by node-inventory and ps-projection to derive `lifecycleState=recoverable`
|
||||
* post-L1 cold-start. Distinct from `findLatestAutoPreDown`, which filters by kind.
|
||||
*/
|
||||
export function findLatestUsableSnapshot(db: Database.Database, rigId: string): Snapshot | null {
|
||||
const row = db.prepare(
|
||||
"SELECT * FROM snapshots WHERE rig_id = ? ORDER BY created_at DESC, id DESC LIMIT 1"
|
||||
).get(rigId) as { id: string; rig_id: string; kind: string; status: string; data: string; created_at: string } | undefined;
|
||||
if (!row) return null;
|
||||
|
||||
let data: SnapshotData;
|
||||
try {
|
||||
data = JSON.parse(row.data) as SnapshotData;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
|
||||
const sessions = data.sessions ?? [];
|
||||
const hasAnyResumeToken = sessions.some(
|
||||
(s) => typeof s.resumeToken === "string" && s.resumeToken.length > 0,
|
||||
);
|
||||
if (!hasAnyResumeToken) return null;
|
||||
|
||||
return {
|
||||
id: row.id,
|
||||
rigId: row.rig_id,
|
||||
kind: row.kind,
|
||||
status: row.status,
|
||||
data,
|
||||
createdAt: row.created_at,
|
||||
};
|
||||
}
|
||||
|
||||
interface NodeOptions {
|
||||
role?: string;
|
||||
runtime?: string;
|
||||
@@ -173,6 +211,15 @@ export class RigRepository {
|
||||
return rows.map((r) => this.rowToRig(r));
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the latest snapshot for this rig with at least one non-null resume token,
|
||||
* or null if no usable snapshot exists. Used by the lifecycle projection to derive
|
||||
* `recoverable` state.
|
||||
*/
|
||||
findLatestUsableSnapshot(rigId: string): Snapshot | null {
|
||||
return findLatestUsableSnapshot(this.db, rigId);
|
||||
}
|
||||
|
||||
findRigsByName(name: string): Rig[] {
|
||||
const rows = this.db
|
||||
.prepare("SELECT * FROM rigs WHERE name = ? ORDER BY created_at")
|
||||
|
||||
@@ -284,6 +284,19 @@ export interface NodeRecoveryGuidance {
|
||||
notes: string[];
|
||||
}
|
||||
|
||||
// Per-node lifecycle projection derived from session/restore state plus snapshot resume metadata.
|
||||
// L2 cold-start truth model: distinguishes a recoverable detached node from a node that would
|
||||
// fresh-launch, and surfaces "attention required" for the post-L3 Claude resume-prompt proxy.
|
||||
export type NodeLifecycleState = "running" | "detached" | "recoverable" | "attention_required";
|
||||
|
||||
// Per-rig lifecycle aggregate folded from per-node states.
|
||||
// running — every node is running.
|
||||
// recoverable — every node is non-running and at least one node has a usable snapshot token.
|
||||
// stopped — every node is non-running and no node has a usable snapshot token.
|
||||
// degraded — mixed running + non-running on the same rig.
|
||||
// attention_required — any node is attention_required (priority over above).
|
||||
export type RigLifecycleState = "running" | "recoverable" | "stopped" | "degraded" | "attention_required";
|
||||
|
||||
export interface NodeInventoryEntry {
|
||||
rigId: string;
|
||||
rigName: string;
|
||||
@@ -297,6 +310,7 @@ export interface NodeInventoryEntry {
|
||||
sessionStatus: string | null;
|
||||
startupStatus: "pending" | "ready" | "attention_required" | "failed" | null;
|
||||
restoreOutcome: NodeRestoreOutcome;
|
||||
lifecycleState: NodeLifecycleState;
|
||||
occupantLifecycle: OccupantLifecycle;
|
||||
continuityOutcome: ContinuityOutcome | null;
|
||||
handoverResult: HandoverResult;
|
||||
|
||||
@@ -6,6 +6,7 @@ import type { SnapshotRepository } from "../domain/snapshot-repository.js";
|
||||
import type { RestoreOrchestrator } from "../domain/restore-orchestrator.js";
|
||||
import { projectRigToGraph, type InventoryOverlay } from "../domain/graph-projection.js";
|
||||
import { getNodeInventory, getNodeInventoryWithContext } from "../domain/node-inventory.js";
|
||||
import { deriveRigLifecycleState } from "../domain/ps-projection.js";
|
||||
import type { ContextUsageStore } from "../domain/context-usage-store.js";
|
||||
import type { Pod, ExpansionPodFragment } from "../domain/types.js";
|
||||
import type { RigExpansionService } from "../domain/rig-expansion-service.js";
|
||||
@@ -87,7 +88,14 @@ function getSelfAttachService(c: { get: (key: string) => unknown }): SelfAttachS
|
||||
rigsRoutes.get("/summary", (c) => {
|
||||
const repo = getRepo(c);
|
||||
const summaries = repo.getRigSummaries();
|
||||
return c.json(summaries);
|
||||
// Enrich with rig-level lifecycleState so CLI surfaces (rig up wording, recover vs
|
||||
// turn-on) can choose the right operator action without a second round trip.
|
||||
const enriched = summaries.map((s) => {
|
||||
const inventory = getNodeInventory(repo.db, s.id);
|
||||
const lifecycleState = deriveRigLifecycleState(inventory.map((e) => e.lifecycleState));
|
||||
return { ...s, lifecycleState };
|
||||
});
|
||||
return c.json(enriched);
|
||||
});
|
||||
|
||||
rigsRoutes.post("/", async (c) => {
|
||||
|
||||
@@ -422,4 +422,94 @@ describe("Node Inventory Projection", () => {
|
||||
expect(detail!.installedResources).toHaveLength(1);
|
||||
expect(detail!.installedResources[0]!.id).toBe("live-skill");
|
||||
});
|
||||
|
||||
// L2 lifecycleState projection
|
||||
describe("lifecycleState (L2)", () => {
|
||||
function seedSnapshotForRig(rigId: string, sessions: Array<{ nodeId: string; resumeToken: string | null }>): void {
|
||||
const data = {
|
||||
rig: { id: rigId, name: "rig-name", createdAt: "2026-04-28T00:00:00Z", updatedAt: "2026-04-28T00:00:00Z" },
|
||||
nodes: [],
|
||||
edges: [],
|
||||
sessions: sessions.map((s, i) => ({
|
||||
id: `sess-snap-${i}`,
|
||||
nodeId: s.nodeId,
|
||||
sessionName: `tmux-${s.nodeId}`,
|
||||
status: "detached",
|
||||
resumeType: s.resumeToken ? "claude" : null,
|
||||
resumeToken: s.resumeToken,
|
||||
restorePolicy: "resume_if_possible",
|
||||
lastSeenAt: null,
|
||||
createdAt: "2026-04-28T00:00:00Z",
|
||||
origin: "launched" as const,
|
||||
startupStatus: "ready" as const,
|
||||
startupCompletedAt: null,
|
||||
})),
|
||||
checkpoints: {},
|
||||
};
|
||||
db.prepare("INSERT INTO snapshots (id, rig_id, kind, status, data) VALUES (?, ?, ?, ?, ?)")
|
||||
.run(`snap-${rigId}`, rigId, "manual", "complete", JSON.stringify(data));
|
||||
}
|
||||
|
||||
it("running session -> lifecycleState=running", () => {
|
||||
seedPodAwareRig(db);
|
||||
seedSession(db, "node-1", "dev-impl@test-rig", { status: "running" });
|
||||
|
||||
const entries = getNodeInventory(db, "rig-1");
|
||||
const entry = entries.find((e) => e.logicalId === "dev.impl");
|
||||
expect(entry?.lifecycleState).toBe("running");
|
||||
});
|
||||
|
||||
it("detached session + usable snapshot with token for THIS node -> lifecycleState=recoverable", () => {
|
||||
seedPodAwareRig(db);
|
||||
seedSession(db, "node-1", "dev-impl@test-rig", { status: "detached" });
|
||||
seedSnapshotForRig("rig-1", [{ nodeId: "node-1", resumeToken: "abc-123" }]);
|
||||
|
||||
const entries = getNodeInventory(db, "rig-1");
|
||||
const entry = entries.find((e) => e.logicalId === "dev.impl");
|
||||
expect(entry?.lifecycleState).toBe("recoverable");
|
||||
});
|
||||
|
||||
it("detached session + snapshot exists but resume token is null for this node -> lifecycleState=detached", () => {
|
||||
seedPodAwareRig(db);
|
||||
seedSession(db, "node-1", "dev-impl@test-rig", { status: "detached" });
|
||||
// Snapshot has token for a DIFFERENT node, not this one
|
||||
seedSnapshotForRig("rig-1", [{ nodeId: "node-2", resumeToken: "xyz-789" }]);
|
||||
|
||||
const entries = getNodeInventory(db, "rig-1");
|
||||
const entry = entries.find((e) => e.logicalId === "dev.impl");
|
||||
expect(entry?.lifecycleState).toBe("detached");
|
||||
});
|
||||
|
||||
it("detached session + no snapshot -> lifecycleState=detached", () => {
|
||||
seedPodAwareRig(db);
|
||||
seedSession(db, "node-1", "dev-impl@test-rig", { status: "detached" });
|
||||
|
||||
const entries = getNodeInventory(db, "rig-1");
|
||||
const entry = entries.find((e) => e.logicalId === "dev.impl");
|
||||
expect(entry?.lifecycleState).toBe("detached");
|
||||
});
|
||||
|
||||
it("restoreOutcome=failed + tmux session alive -> lifecycleState=attention_required (Claude resume-prompt proxy)", () => {
|
||||
seedPodAwareRig(db);
|
||||
seedSession(db, "node-1", "dev-impl@test-rig", { status: "running" });
|
||||
// Persist a restore.completed event with this node failed
|
||||
seedEvent(db, "rig-1", "node-1", "restore.completed", {
|
||||
result: { rigResult: "partially_restored", nodes: [{ nodeId: "node-1", status: "failed" }] },
|
||||
});
|
||||
|
||||
const entries = getNodeInventory(db, "rig-1");
|
||||
const entry = entries.find((e) => e.logicalId === "dev.impl");
|
||||
expect(entry?.restoreOutcome).toBe("failed");
|
||||
expect(entry?.lifecycleState).toBe("attention_required");
|
||||
});
|
||||
|
||||
it("exited session + no snapshot -> lifecycleState=detached", () => {
|
||||
seedPodAwareRig(db);
|
||||
seedSession(db, "node-1", "dev-impl@test-rig", { status: "exited" });
|
||||
|
||||
const entries = getNodeInventory(db, "rig-1");
|
||||
const entry = entries.find((e) => e.logicalId === "dev.impl");
|
||||
expect(entry?.lifecycleState).toBe("detached");
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -1,36 +1,14 @@
|
||||
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
||||
import type Database from "better-sqlite3";
|
||||
import { createDb } from "../src/db/connection.js";
|
||||
import { migrate } from "../src/db/migrate.js";
|
||||
import { coreSchema } from "../src/db/migrations/001_core_schema.js";
|
||||
import { bindingsSessionsSchema } from "../src/db/migrations/002_bindings_sessions.js";
|
||||
import { eventsSchema } from "../src/db/migrations/003_events.js";
|
||||
import { snapshotsSchema } from "../src/db/migrations/004_snapshots.js";
|
||||
import { checkpointsSchema } from "../src/db/migrations/005_checkpoints.js";
|
||||
import { resumeMetadataSchema } from "../src/db/migrations/006_resume_metadata.js";
|
||||
import { nodeSpecFieldsSchema } from "../src/db/migrations/007_node_spec_fields.js";
|
||||
import { packagesSchema } from "../src/db/migrations/008_packages.js";
|
||||
import { installJournalSchema } from "../src/db/migrations/009_install_journal.js";
|
||||
import { journalSeqSchema } from "../src/db/migrations/010_journal_seq.js";
|
||||
import { bootstrapSchema } from "../src/db/migrations/011_bootstrap.js";
|
||||
import { discoverySchema } from "../src/db/migrations/012_discovery.js";
|
||||
import { discoveryFkFix } from "../src/db/migrations/013_discovery_fk_fix.js";
|
||||
import { PsProjectionService } from "../src/domain/ps-projection.js";
|
||||
|
||||
const ALL_MIGRATIONS = [
|
||||
coreSchema, bindingsSessionsSchema, eventsSchema, snapshotsSchema,
|
||||
checkpointsSchema, resumeMetadataSchema, nodeSpecFieldsSchema,
|
||||
packagesSchema, installJournalSchema, journalSeqSchema, bootstrapSchema,
|
||||
discoverySchema, discoveryFkFix,
|
||||
];
|
||||
import { createFullTestDb } from "./helpers/test-app.js";
|
||||
import { PsProjectionService, deriveRigLifecycleState } from "../src/domain/ps-projection.js";
|
||||
|
||||
describe("PsProjectionService", () => {
|
||||
let db: Database.Database;
|
||||
let ps: PsProjectionService;
|
||||
|
||||
beforeEach(() => {
|
||||
db = createDb();
|
||||
migrate(db, ALL_MIGRATIONS);
|
||||
db = createFullTestDb();
|
||||
ps = new PsProjectionService({ db });
|
||||
});
|
||||
|
||||
@@ -187,4 +165,109 @@ describe("PsProjectionService", () => {
|
||||
daemonDb.close();
|
||||
}
|
||||
});
|
||||
|
||||
// L2 rig-level lifecycleState
|
||||
describe("lifecycleState (L2)", () => {
|
||||
function seedSnapshotForRig(rigId: string, sessions: Array<{ nodeId: string; resumeToken: string | null }>): void {
|
||||
const data = {
|
||||
rig: { id: rigId, name: "rig-name", createdAt: "2026-04-28T00:00:00Z", updatedAt: "2026-04-28T00:00:00Z" },
|
||||
nodes: [],
|
||||
edges: [],
|
||||
sessions: sessions.map((s, i) => ({
|
||||
id: `sess-snap-${i}`,
|
||||
nodeId: s.nodeId,
|
||||
sessionName: `tmux-${s.nodeId}`,
|
||||
status: "detached",
|
||||
resumeType: s.resumeToken ? "claude" : null,
|
||||
resumeToken: s.resumeToken,
|
||||
restorePolicy: "resume_if_possible",
|
||||
lastSeenAt: null,
|
||||
createdAt: "2026-04-28T00:00:00Z",
|
||||
origin: "launched" as const,
|
||||
startupStatus: "ready" as const,
|
||||
startupCompletedAt: null,
|
||||
})),
|
||||
checkpoints: {},
|
||||
};
|
||||
db.prepare("INSERT INTO snapshots (id, rig_id, kind, status, data) VALUES (?, ?, ?, ?, ?)")
|
||||
.run(`snap-${rigId}`, rigId, "manual", "complete", JSON.stringify(data));
|
||||
}
|
||||
|
||||
it("all nodes running -> lifecycleState=running", () => {
|
||||
const rigId = seedRig("all-run");
|
||||
const n1 = seedNode(rigId, "dev");
|
||||
const n2 = seedNode(rigId, "qa");
|
||||
seedSession(n1, "running");
|
||||
seedSession(n2, "running");
|
||||
|
||||
const entries = ps.getEntries();
|
||||
expect(entries[0]!.lifecycleState).toBe("running");
|
||||
});
|
||||
|
||||
it("all nodes detached + usable snapshot -> lifecycleState=recoverable", () => {
|
||||
const rigId = seedRig("all-recoverable");
|
||||
const n1 = seedNode(rigId, "dev");
|
||||
const n2 = seedNode(rigId, "qa");
|
||||
seedSession(n1, "detached");
|
||||
seedSession(n2, "detached");
|
||||
seedSnapshotForRig(rigId, [
|
||||
{ nodeId: n1, resumeToken: "tok-1" },
|
||||
{ nodeId: n2, resumeToken: "tok-2" },
|
||||
]);
|
||||
|
||||
const entries = ps.getEntries();
|
||||
expect(entries[0]!.lifecycleState).toBe("recoverable");
|
||||
});
|
||||
|
||||
it("all nodes detached + no usable snapshot -> lifecycleState=stopped", () => {
|
||||
const rigId = seedRig("all-stopped");
|
||||
const n1 = seedNode(rigId, "dev");
|
||||
seedSession(n1, "detached");
|
||||
|
||||
const entries = ps.getEntries();
|
||||
expect(entries[0]!.lifecycleState).toBe("stopped");
|
||||
});
|
||||
|
||||
it("mixed running + detached -> lifecycleState=degraded", () => {
|
||||
const rigId = seedRig("mixed");
|
||||
const n1 = seedNode(rigId, "dev");
|
||||
const n2 = seedNode(rigId, "qa");
|
||||
seedSession(n1, "running");
|
||||
seedSession(n2, "detached");
|
||||
|
||||
const entries = ps.getEntries();
|
||||
expect(entries[0]!.lifecycleState).toBe("degraded");
|
||||
});
|
||||
|
||||
it("any node attention_required -> lifecycleState=attention_required (priority over running)", () => {
|
||||
const rigId = seedRig("att-priority");
|
||||
const n1 = seedNode(rigId, "dev");
|
||||
const n2 = seedNode(rigId, "qa");
|
||||
seedSession(n1, "running");
|
||||
seedSession(n2, "running");
|
||||
// Mark n2 as attention_required via failed restoreOutcome on a running session
|
||||
db.prepare(
|
||||
"INSERT INTO events (rig_id, node_id, type, payload) VALUES (?, ?, ?, ?)"
|
||||
).run(rigId, n2, "restore.completed", JSON.stringify({
|
||||
result: { rigResult: "partially_restored", nodes: [{ nodeId: n2, status: "failed" }] },
|
||||
type: "restore.completed",
|
||||
}));
|
||||
|
||||
const entries = ps.getEntries();
|
||||
expect(entries[0]!.lifecycleState).toBe("attention_required");
|
||||
});
|
||||
|
||||
// Pure unit-level coverage of the fold helper.
|
||||
it("deriveRigLifecycleState fold helper covers all branches", () => {
|
||||
expect(deriveRigLifecycleState([])).toBe("stopped");
|
||||
expect(deriveRigLifecycleState(["running", "running"])).toBe("running");
|
||||
expect(deriveRigLifecycleState(["detached", "detached"])).toBe("stopped");
|
||||
expect(deriveRigLifecycleState(["recoverable", "detached"])).toBe("recoverable");
|
||||
expect(deriveRigLifecycleState(["recoverable", "recoverable"])).toBe("recoverable");
|
||||
expect(deriveRigLifecycleState(["running", "detached"])).toBe("degraded");
|
||||
expect(deriveRigLifecycleState(["running", "recoverable"])).toBe("degraded");
|
||||
expect(deriveRigLifecycleState(["attention_required", "running"])).toBe("attention_required");
|
||||
expect(deriveRigLifecycleState(["attention_required", "detached"])).toBe("attention_required");
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -191,3 +191,103 @@ describe("Wiring regression", () => {
|
||||
db.close();
|
||||
});
|
||||
});
|
||||
|
||||
// L2 findLatestUsableSnapshot
|
||||
describe("RigRepository.findLatestUsableSnapshot (L2)", () => {
|
||||
let db: Database.Database;
|
||||
let repo: RigRepository;
|
||||
|
||||
beforeEach(() => {
|
||||
db = setupDb();
|
||||
repo = new RigRepository(db);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
function seedSnapshot(
|
||||
rigId: string,
|
||||
snapshotId: string,
|
||||
sessions: Array<{ nodeId: string; resumeToken: string | null }>,
|
||||
createdAt?: string,
|
||||
): void {
|
||||
const data = {
|
||||
rig: { id: rigId, name: "rig-name", createdAt: "2026-04-28T00:00:00Z", updatedAt: "2026-04-28T00:00:00Z" },
|
||||
nodes: [],
|
||||
edges: [],
|
||||
sessions: sessions.map((s, i) => ({
|
||||
id: `sess-snap-${i}`,
|
||||
nodeId: s.nodeId,
|
||||
sessionName: `tmux-${s.nodeId}`,
|
||||
status: "detached",
|
||||
resumeType: s.resumeToken ? "claude" : null,
|
||||
resumeToken: s.resumeToken,
|
||||
restorePolicy: "resume_if_possible",
|
||||
lastSeenAt: null,
|
||||
createdAt: "2026-04-28T00:00:00Z",
|
||||
origin: "launched" as const,
|
||||
startupStatus: "ready" as const,
|
||||
startupCompletedAt: null,
|
||||
})),
|
||||
checkpoints: {},
|
||||
};
|
||||
if (createdAt) {
|
||||
db.prepare("INSERT INTO snapshots (id, rig_id, kind, status, data, created_at) VALUES (?, ?, ?, ?, ?, ?)")
|
||||
.run(snapshotId, rigId, "manual", "complete", JSON.stringify(data), createdAt);
|
||||
} else {
|
||||
db.prepare("INSERT INTO snapshots (id, rig_id, kind, status, data) VALUES (?, ?, ?, ?, ?)")
|
||||
.run(snapshotId, rigId, "manual", "complete", JSON.stringify(data));
|
||||
}
|
||||
}
|
||||
|
||||
it("returns null when no snapshot exists for rig", () => {
|
||||
const rig = repo.createRig("no-snap");
|
||||
expect(repo.findLatestUsableSnapshot(rig.id)).toBeNull();
|
||||
});
|
||||
|
||||
it("returns null when only snapshots have null resume tokens for all nodes", () => {
|
||||
const rig = repo.createRig("null-tokens");
|
||||
seedSnapshot(rig.id, "snap-1", [
|
||||
{ nodeId: "node-a", resumeToken: null },
|
||||
{ nodeId: "node-b", resumeToken: null },
|
||||
]);
|
||||
expect(repo.findLatestUsableSnapshot(rig.id)).toBeNull();
|
||||
});
|
||||
|
||||
it("returns latest snapshot when at least one session has a non-null resume token", () => {
|
||||
const rig = repo.createRig("usable");
|
||||
seedSnapshot(rig.id, "snap-1", [
|
||||
{ nodeId: "node-a", resumeToken: "tok-a" },
|
||||
{ nodeId: "node-b", resumeToken: null },
|
||||
]);
|
||||
|
||||
const result = repo.findLatestUsableSnapshot(rig.id);
|
||||
expect(result).not.toBeNull();
|
||||
expect(result!.id).toBe("snap-1");
|
||||
expect(result!.data.sessions?.find((s) => s.nodeId === "node-a")?.resumeToken).toBe("tok-a");
|
||||
});
|
||||
|
||||
it("returns the snapshot even when only some nodes have valid tokens (per-node lifecycleState handles individual recoverability)", () => {
|
||||
const rig = repo.createRig("partial");
|
||||
seedSnapshot(rig.id, "snap-1", [
|
||||
{ nodeId: "node-a", resumeToken: "tok-a" }, // recoverable
|
||||
{ nodeId: "node-b", resumeToken: null }, // detached
|
||||
{ nodeId: "node-c", resumeToken: null }, // detached
|
||||
]);
|
||||
|
||||
const result = repo.findLatestUsableSnapshot(rig.id);
|
||||
expect(result).not.toBeNull();
|
||||
expect(result!.data.sessions).toHaveLength(3);
|
||||
});
|
||||
|
||||
it("returns the latest snapshot when multiple usable snapshots exist (ORDER BY created_at DESC, id DESC)", () => {
|
||||
const rig = repo.createRig("multi-snap");
|
||||
seedSnapshot(rig.id, "snap-old", [{ nodeId: "node-a", resumeToken: "tok-old" }], "2026-04-27 00:00:00");
|
||||
seedSnapshot(rig.id, "snap-new", [{ nodeId: "node-a", resumeToken: "tok-new" }], "2026-04-28 00:00:00");
|
||||
|
||||
const result = repo.findLatestUsableSnapshot(rig.id);
|
||||
expect(result).not.toBeNull();
|
||||
expect(result!.id).toBe("snap-new");
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user