feat(daemon,cli): PL-004 Phase C — daemon-native Watchdog supervision tree

Adds the Watchdog primitive from PL-004 Phase C:

Migrations (031, 032):
  - 031_watchdog_jobs: canonical scheduler state (policy enum + intervals
    + lifecycle). Three v1 policies: periodic-reminder, artifact-pool-ready,
    edge-artifact-required. workflow-keepalive is intentionally rejected
    at registration with policy_deferred_to_phase_d (ships in Phase D).
  - 032_watchdog_history: append-only audit. Records sent / skipped /
    terminal outcomes only; pure not_due polls are not recorded.

Domain (TypeScript ports of POC .mjs policies preserving observable semantics):
  - WatchdogJobsRepository (CRUD + register-time enum gate + lifecycle)
  - WatchdogHistoryLog (append-only writer; no update/delete API surface)
  - WatchdogPolicyEngine (resolve policy by name, parse spec_yaml context,
    dispatch evaluate, record meaningful outcomes, emit events, call
    injected delivery for action=send)
  - WatchdogScheduler (tick-driven daemon-native scheduler joining the
    supervision tree; isDue gate; sequential per-tick evaluation;
    graceful in-flight await on stop; SQLite-canonical restart recovery)
  - 3 policy modules + shared artifact-pool helpers

Routes (per Phase A R1 SSE route-order discipline; SSE before /:job_id):
  - POST /api/watchdog/register
  - GET /api/watchdog/list
  - GET /api/watchdog/sse + /watch
  - GET /api/watchdog/:job_id and :/:job_id/status
  - POST /api/watchdog/:job_id/stop

5 new RigEvents:
  - watchdog.evaluation_fired / _skipped / _terminal
  - watchdog.job_registered / _stopped

CLI:
  - rig watchdog register/list/show/status/stop

Wiring:
  - startup.ts constructs repo + log early, engine + scheduler after
    SessionTransport is available (delivery is wired via SessionTransport.send)
  - index.ts starts the scheduler post-listen and stops it on SIGINT/SIGTERM

Tests: 62 daemon tests + 7 CLI tests covering jobs-repo, history-log
append-only contract, three-policy semantics, engine dispatch+history+events,
scheduler dueness/restart-recovery/error-resilience, routes including SSE
route-order regression guard, CLI subcommand registration + happy paths +
workflow-keepalive Phase D rejection surfacing. Full daemon vitest baseline:
2471/2471 pass; CLI watchdog tests 7/7 pass; pre-existing CLI failures
(host-registry/config/specs/preflight/client/chatroom) confirmed present
on Phase B HEAD pre-Phase-C and unrelated to this change.

workflow-keepalive deferred to Phase D paired with workflow_instances.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
mvschwarzandClaude Opus 4.7 committed 2026-05-03 00:10:25 -07:00
1 parent 629d061325
commit 48c0267615
28 files changed
+3149 -4

No files matched your search

+2
View File
@@ -325,6 +325,8 @@ Migration boundary:
- `021_seat_handover_observability.ts` adds seat-handover observability fields.
- `022_node_codex_config_profile.ts` adds Codex config-profile field on nodes.
- `023_stream_items.ts` through `027_outbox_entries.ts` add the PL-004 Phase A coordination tables (see § Coordination Layer below).
- `028_project_classifications.ts` through `030_views_custom.ts` add the PL-004 Phase B classifier + view tables.
- `031_watchdog_jobs.ts` and `032_watchdog_history.ts` add the PL-004 Phase C watchdog supervision-tree tables. `watchdog_jobs` is the canonical scheduler state (policy, target_session, intervals, lifecycle); `watchdog_history` is the append-only audit of meaningful evaluations (`sent`/`skipped`/`terminal`). Pure `not_due` polls are not recorded. Three v1 policies: `periodic-reminder`, `artifact-pool-ready`, `edge-artifact-required`. The fourth POC policy `workflow-keepalive` is intentionally rejected at registration with `policy_deferred_to_phase_d` and ships in Phase D paired with workflow_instances.
### Coordination tables (PL-004 Phase A)
+14
View File
@@ -700,6 +700,20 @@ Closure-reason semantics:
- `no-follow-on` — terminal completion, nothing else needed.
- `escalation` — kicked up to a higher tier (target = escalation target).
## Coordination Watchdog (PL-004 Phase C)
`rig watchdog` registers, lists, inspects, and stops daemon-native scheduler jobs. The scheduler runs inside the daemon supervision tree and persists state in SQLite (`watchdog_jobs`/`watchdog_history`); jobs survive daemon restarts. Three policies are available at v1: `periodic-reminder`, `artifact-pool-ready`, and `edge-artifact-required`. The fourth POC policy `workflow-keepalive` is rejected with `policy_deferred_to_phase_d` and ships in Phase D.
### `rig watchdog`
- `register --spec <path> --policy <name> --target-session <s> --interval-seconds <n> --registered-by <s>` — register a job from a YAML spec; `--active-wake-interval-seconds` and `--scan-interval-seconds` are pool-ready-specific opt-ins.
- `list` — list all jobs (active + stopped + terminal).
- `show <job_id>` — show one job.
- `status <job_id>` — show job + recent evaluation history (last 20 entries).
- `stop <job_id> [--reason <text>]` — operator stop; scheduler skips the job thereafter.
History records only meaningful evaluations: `sent` (delivery executed), `skipped` (policy returned skip with a reason such as `no_actionable_artifacts`), or `terminal` (policy declared the job done). Pure `not_due` polls are not recorded.
## Commands Not Present
These are not current top-level `rig` commands:
+17
View File
@@ -89,6 +89,23 @@ Fast lookup:
- `packages/daemon/src/db/migrations/025_queue_transitions.ts`: PL-004 Phase A — append-only queue state transition log. Exports: queueTransitionsSchema. Related: `packages/daemon/src/db/migrate.js`.
- `packages/daemon/src/db/migrations/026_inbox_entries.ts`: PL-004 Phase A — mailbox-style inbox for asynchronous deposit. Exports: inboxEntriesSchema. Related: `packages/daemon/src/db/migrate.js`.
- `packages/daemon/src/db/migrations/027_outbox_entries.ts`: PL-004 Phase A — sender-side outbox audit. Exports: outboxEntriesSchema. Related: `packages/daemon/src/db/migrate.js`.
- `packages/daemon/src/db/migrations/028_project_classifications.ts`: PL-004 Phase B — L2 project classifications table (idempotent on stream_item_id; FK to stream_items). Exports: projectClassificationsSchema. Related: `packages/daemon/src/db/migrate.js`.
- `packages/daemon/src/db/migrations/029_classifier_leases.ts`: PL-004 Phase B — single-writer classifier lease table (partial UNIQUE INDEX on state='active'). Exports: classifierLeasesSchema. Related: `packages/daemon/src/db/migrate.js`.
- `packages/daemon/src/db/migrations/030_views_custom.ts`: PL-004 Phase B — custom view registry. Exports: viewsCustomSchema. Related: `packages/daemon/src/db/migrate.js`.
- `packages/daemon/src/db/migrations/031_watchdog_jobs.ts`: PL-004 Phase C — watchdog jobs (canonical scheduler state; policy enum + intervals + lifecycle). Exports: watchdogJobsSchema. Related: `packages/daemon/src/db/migrate.js`.
- `packages/daemon/src/db/migrations/032_watchdog_history.ts`: PL-004 Phase C — append-only watchdog evaluation history (sent/skipped/terminal; not_due polls not recorded). Exports: watchdogHistorySchema. Related: `packages/daemon/src/db/migrate.js`.
### Domain: Coordination Watchdog (PL-004 Phase C)
- `packages/daemon/src/domain/watchdog-jobs-repository.ts`: Persistence for watchdog_jobs. Enforces three-policy enum (periodic-reminder, artifact-pool-ready, edge-artifact-required) at register-time; rejects workflow-keepalive with policy_deferred_to_phase_d. Exports: WatchdogJobsRepository, WatchdogJobsError, WatchdogJob, PHASE_C_POLICIES, WORKFLOW_KEEPALIVE_DEFERRED_POLICY. Related: `packages/daemon/src/db/migrations/031_watchdog_jobs.js`.
- `packages/daemon/src/domain/watchdog-history-log.ts`: Append-only writer for watchdog_history. Records meaningful evaluations only (sent/skipped/terminal). API surface intentionally lacks update/delete. Exports: WatchdogHistoryLog, WatchdogHistoryEntry, WatchdogOutcome. Related: `packages/daemon/src/db/migrations/032_watchdog_history.js`.
- `packages/daemon/src/domain/watchdog-policy-engine.ts`: Resolves policy by name, parses spec_yaml context, dispatches policy.evaluate(), records meaningful outcomes via history-log, emits watchdog.* events, calls injected delivery callback for action=send. Exports: WatchdogPolicyEngine, DeliveryFn, DeliveryRequest, DeliveryOutcome. Related: `packages/daemon/src/domain/watchdog-jobs-repository.js`, `packages/daemon/src/domain/watchdog-history-log.js`, `packages/daemon/src/domain/event-bus.js`.
- `packages/daemon/src/domain/watchdog-scheduler.ts`: Tick-driven daemon-native scheduler that joins the supervision tree. Filters listActive() through isDue(), evaluates due jobs sequentially, supports start/stop with graceful in-flight await. Exports: WatchdogScheduler, isDue. Related: `packages/daemon/src/domain/watchdog-policy-engine.js`, `packages/daemon/src/domain/watchdog-jobs-repository.js`.
- `packages/daemon/src/domain/policies/types.ts`: Policy contract (Policy, PolicyJob, PolicyEvaluation). Exports: Policy, PolicyJob, PolicyEvaluation. Related: none.
- `packages/daemon/src/domain/policies/periodic-reminder.ts`: Policy that returns send {target, message} unconditionally; throws policy_spec_invalid if context.target.session or context.message missing. Translated from POC `lib/policies/periodic-reminder.mjs`. Exports: periodicReminderPolicy. Related: `packages/daemon/src/domain/policies/types.js`.
- `packages/daemon/src/domain/policies/artifact-pool-ready.ts`: Policy that scans context.pools and skips with no_actionable_artifacts when empty; otherwise sends a formatted message capped at max_items. Translated from POC `lib/policies/artifact-pool-ready.mjs`. Exports: artifactPoolReadyPolicy. Related: `packages/daemon/src/domain/policies/artifact-pool-helpers.js`, `packages/daemon/src/domain/policies/types.js`.
- `packages/daemon/src/domain/policies/edge-artifact-required.ts`: Policy that scans source_pools and target_pools, computes source-key per artifact, and sends an edge-repair message for sources lacking matching downstream artifacts. Skips with no_missing_edge_artifacts when all sources matched. Translated from POC `lib/policies/edge-artifact-required.mjs`. Exports: edgeArtifactRequiredPolicy. Related: `packages/daemon/src/domain/policies/artifact-pool-helpers.js`, `packages/daemon/src/domain/policies/types.js`.
- `packages/daemon/src/domain/policies/artifact-pool-helpers.ts`: Shared filesystem scanner + formatter for artifact-pool-ready and edge-artifact-required. Translated from POC `lib/policies/artifact-pool.js`. Exports: scanArtifactPools, formatArtifactList, sourceKeyFor, ArtifactPoolSpec, ScannedArtifact. Related: none.
- `packages/daemon/src/routes/watchdog.ts`: Hono route module for `/api/watchdog` (`POST /register`, `GET /list`, `GET /sse`, `GET /watch`, `GET /:job_id`, `GET /:job_id/status`, `POST /:job_id/stop`). SSE/literal paths mounted before /:job_id catchall per Phase A R1 route-order discipline. Exports: watchdogRoutes. Related: `packages/daemon/src/domain/watchdog-jobs-repository.js`, `packages/daemon/src/domain/watchdog-history-log.js`, `packages/daemon/src/domain/event-bus.js`.
### Domain: Core Runtime
- `packages/daemon/src/domain/errors.ts`: Domain-specific error types, currently including `RigNotFoundError`. Exports: RigNotFoundError. Related: none.
+153
View File
@@ -0,0 +1,153 @@
import { readFileSync } from "node:fs";
import { Command } from "commander";
import { DaemonClient } from "../client.js";
import { getDaemonStatus, getDaemonUrl } from "../daemon-lifecycle.js";
import { realDeps } from "./daemon.js";
import type { StatusDeps } from "./status.js";
/**
* `rig watchdog` — coordination primitive Watchdog (PL-004 Phase C).
*
* Backed by `/api/watchdog`. Operates only via the daemon HTTP API.
*
* Per PRD § Watchdog: scheduler is daemon-native and joins the
* supervision tree. Three policies in scope at v1; workflow-keepalive
* deferred to Phase D.
*/
export interface WatchdogDeps extends StatusDeps {}
async function withClient<T>(
deps: WatchdogDeps,
fn: (client: DaemonClient) => Promise<T>,
): Promise<T | undefined> {
const status = await getDaemonStatus(deps.lifecycleDeps);
if (status.state !== "running" || status.healthy === false) {
console.error("Daemon not running. Start it with: rig daemon start");
process.exitCode = 1;
return undefined;
}
const client = deps.clientFactory(getDaemonUrl(status));
return fn(client);
}
function printResult(json: boolean, body: unknown, status: number): void {
if (json) {
console.log(JSON.stringify(body));
} else {
console.log(JSON.stringify(body, null, 2));
}
if (status >= 400) process.exitCode = status >= 500 ? 2 : 1;
}
export function watchdogCommand(depsOverride?: WatchdogDeps): Command {
const cmd = new Command("watchdog").description(
"Coordination Watchdog — daemon-native scheduler for periodic-reminder, artifact-pool-ready, edge-artifact-required (PL-004 Phase C; workflow-keepalive ships in Phase D)",
);
const getDeps = (): WatchdogDeps =>
depsOverride ?? {
lifecycleDeps: realDeps(),
clientFactory: (url: string) => new DaemonClient(url),
};
cmd
.command("register")
.description("Register a new watchdog job from a YAML spec file")
.requiredOption("--spec <path>", "Path to YAML spec file (policy + target + interval + context)")
.requiredOption("--policy <policy>", "Policy name (one of: periodic-reminder, artifact-pool-ready, edge-artifact-required)")
.requiredOption("--target-session <session>", "Canonical <member>@<rig> target")
.requiredOption("--interval-seconds <n>", "Evaluation interval (positive integer)")
.requiredOption("--registered-by <session>", "Registering session (for audit)")
.option("--active-wake-interval-seconds <n>", "(pool-ready-specific) wake-up cadence when actionable artifacts exist")
.option("--scan-interval-seconds <n>", "(pool-ready-specific) artifact pool scan cadence")
.option("--json", "JSON output for agents")
.action(async (opts: {
spec: string;
policy: string;
targetSession: string;
intervalSeconds: string;
registeredBy: string;
activeWakeIntervalSeconds?: string;
scanIntervalSeconds?: string;
json?: boolean;
}) => {
const deps = getDeps();
let specYaml: string;
try {
specYaml = readFileSync(opts.spec, "utf-8");
} catch (err) {
console.error(`Failed to read spec file ${opts.spec}: ${err instanceof Error ? err.message : err}`);
process.exitCode = 1;
return;
}
await withClient(deps, async (client) => {
const res = await client.post<unknown>("/api/watchdog/register", {
policy: opts.policy,
specYaml,
targetSession: opts.targetSession,
intervalSeconds: Number.parseInt(opts.intervalSeconds, 10),
activeWakeIntervalSeconds: opts.activeWakeIntervalSeconds
? Number.parseInt(opts.activeWakeIntervalSeconds, 10)
: undefined,
scanIntervalSeconds: opts.scanIntervalSeconds
? Number.parseInt(opts.scanIntervalSeconds, 10)
: undefined,
registeredBySession: opts.registeredBy,
});
printResult(opts.json ?? false, res.data, res.status);
});
});
cmd
.command("list")
.description("List all watchdog jobs")
.option("--json", "JSON output for agents")
.action(async (opts: { json?: boolean }) => {
const deps = getDeps();
await withClient(deps, async (client) => {
const res = await client.get<unknown>("/api/watchdog/list");
printResult(opts.json ?? false, res.data, res.status);
});
});
cmd
.command("show <jobId>")
.description("Show one watchdog job")
.option("--json", "JSON output for agents")
.action(async (jobId: string, opts: { json?: boolean }) => {
const deps = getDeps();
await withClient(deps, async (client) => {
const res = await client.get<unknown>(`/api/watchdog/${encodeURIComponent(jobId)}`);
printResult(opts.json ?? false, res.data, res.status);
});
});
cmd
.command("status <jobId>")
.description("Show one watchdog job + recent evaluation history")
.option("--json", "JSON output for agents")
.action(async (jobId: string, opts: { json?: boolean }) => {
const deps = getDeps();
await withClient(deps, async (client) => {
const res = await client.get<unknown>(`/api/watchdog/${encodeURIComponent(jobId)}/status`);
printResult(opts.json ?? false, res.data, res.status);
});
});
cmd
.command("stop <jobId>")
.description("Stop a watchdog job (operator-stopped; scheduler skips it)")
.option("--reason <text>", "Stop reason (free-form; recorded in terminal_reason)")
.option("--json", "JSON output for agents")
.action(async (jobId: string, opts: { reason?: string; json?: boolean }) => {
const deps = getDeps();
await withClient(deps, async (client) => {
const res = await client.post<unknown>(`/api/watchdog/${encodeURIComponent(jobId)}/stop`, {
reason: opts.reason,
});
printResult(opts.json ?? false, res.data, res.status);
});
});
return cmd;
}
+3
View File
@@ -29,6 +29,7 @@ import { streamCommand, type StreamDeps } from "./commands/stream.js";
import { queueCommand, type QueueDeps } from "./commands/queue.js";
import { projectCommand, type ProjectDeps } from "./commands/project.js";
import { viewCommand, type ViewDeps } from "./commands/view.js";
import { watchdogCommand, type WatchdogDeps } from "./commands/watchdog.js";
import { captureCommand } from "./commands/capture.js";
import { broadcastCommand } from "./commands/broadcast.js";
import { configCommand } from "./commands/config.js";
@@ -84,6 +85,7 @@ export interface ProgramDeps {
queueDeps?: QueueDeps;
projectDeps?: ProjectDeps;
viewDeps?: ViewDeps;
watchdogDeps?: WatchdogDeps;
captureDeps?: StatusDeps;
broadcastDeps?: StatusDeps;
askDeps?: StatusDeps;
@@ -140,6 +142,7 @@ export function createProgram(depsOverride?: ProgramDeps): Command {
program.addCommand(queueCommand(depsOverride?.queueDeps));
program.addCommand(projectCommand(depsOverride?.projectDeps));
program.addCommand(viewCommand(depsOverride?.viewDeps));
program.addCommand(watchdogCommand(depsOverride?.watchdogDeps));
program.addCommand(captureCommand(depsOverride?.captureDeps));
program.addCommand(broadcastCommand(depsOverride?.broadcastDeps));
program.addCommand(askCommand(depsOverride?.askDeps));
+233
View File
@@ -0,0 +1,233 @@
import { mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, it, expect, vi, beforeEach } from "vitest";
import type { WatchdogDeps } from "../src/commands/watchdog.js";
import { createProgram } from "../src/index.js";
vi.mock("../src/daemon-lifecycle.js", async () => {
const actual = await vi.importActual<Record<string, unknown>>("../src/daemon-lifecycle.js");
return {
...actual,
getDaemonStatus: vi.fn(async () => ({
state: "running",
healthy: true,
pid: 1234,
port: 7433,
})),
getDaemonUrl: vi.fn(() => "http://localhost:7433"),
};
});
interface StubResponse {
status: number;
data: unknown;
}
function makeDeps(opts?: {
routes?: Record<string, StubResponse>;
}): {
deps: WatchdogDeps;
calls: Array<{ method: string; path: string; body?: unknown }>;
} {
const calls: Array<{ method: string; path: string; body?: unknown }> = [];
const routes = opts?.routes ?? {};
return {
calls,
deps: {
lifecycleDeps: {} as WatchdogDeps["lifecycleDeps"],
clientFactory: () =>
({
get: vi.fn(async (path: string) => {
calls.push({ method: "GET", path });
return routes[`GET ${path}`] ?? { status: 200, data: {} };
}),
getText: vi.fn(async (path: string) => {
calls.push({ method: "GET", path });
return { status: 200, data: "" };
}),
post: vi.fn(async (path: string, body: unknown) => {
calls.push({ method: "POST", path, body });
return routes[`POST ${path}`] ?? { status: 201, data: {} };
}),
delete: vi.fn(async (path: string) => {
calls.push({ method: "DELETE", path });
return { status: 204, data: null };
}),
postText: vi.fn(async (path: string) => {
calls.push({ method: "POST", path });
return { status: 200, data: "" };
}),
postExpectText: vi.fn(async (path: string) => {
calls.push({ method: "POST", path });
return { status: 200, data: "" };
}),
}) as unknown as ReturnType<WatchdogDeps["clientFactory"]>,
},
};
}
describe("rig watchdog CLI (PL-004 Phase C)", () => {
let logs: string[];
let errors: string[];
beforeEach(() => {
logs = [];
errors = [];
vi.spyOn(console, "log").mockImplementation((...a) => logs.push(a.join(" ")));
vi.spyOn(console, "error").mockImplementation((...a) => errors.push(a.join(" ")));
process.exitCode = undefined;
});
it("watchdog is registered with all v1 subcommands", () => {
const { deps } = makeDeps();
const program = createProgram({ watchdogDeps: deps });
const cmd = program.commands.find((c) => c.name() === "watchdog");
expect(cmd).toBeDefined();
const subs = cmd!.commands.map((c) => c.name()).sort();
expect(subs).toContain("register");
expect(subs).toContain("list");
expect(subs).toContain("show");
expect(subs).toContain("status");
expect(subs).toContain("stop");
});
it("register POSTs /api/watchdog/register with spec file contents + structured fields", async () => {
const { deps, calls } = makeDeps({
routes: {
"POST /api/watchdog/register": {
status: 201,
data: { jobId: "01J..." },
},
},
});
const tmp = mkdtempSync(join(tmpdir(), "wd-cli-"));
const spec = join(tmp, "spec.yaml");
const yaml =
"policy: periodic-reminder\ntarget: alice@rig\ninterval_seconds: 60\ncontext:\n target:\n session: alice@rig\n message: ping\n";
writeFileSync(spec, yaml);
const program = createProgram({ watchdogDeps: deps });
program.exitOverride();
await program.parseAsync([
"node",
"rig",
"watchdog",
"register",
"--spec",
spec,
"--policy",
"periodic-reminder",
"--target-session",
"alice@rig",
"--interval-seconds",
"60",
"--registered-by",
"ops@kernel",
"--json",
]);
const call = calls.find((c) => c.path === "/api/watchdog/register");
expect(call).toBeDefined();
const body = call!.body as Record<string, unknown>;
expect(body.policy).toBe("periodic-reminder");
expect(body.specYaml).toBe(yaml);
expect(body.targetSession).toBe("alice@rig");
expect(body.intervalSeconds).toBe(60);
expect(body.registeredBySession).toBe("ops@kernel");
});
it("register propagates daemon-side workflow-keepalive rejection (Phase D deferral surfaced via 400)", async () => {
const { deps } = makeDeps({
routes: {
"POST /api/watchdog/register": {
status: 400,
data: {
error: "policy_deferred_to_phase_d",
message: "policy 'workflow-keepalive' ships in PL-004 Phase D",
},
},
},
});
const tmp = mkdtempSync(join(tmpdir(), "wd-cli-"));
const spec = join(tmp, "spec.yaml");
writeFileSync(spec, "policy: workflow-keepalive\n");
const program = createProgram({ watchdogDeps: deps });
program.exitOverride();
await program.parseAsync([
"node",
"rig",
"watchdog",
"register",
"--spec",
spec,
"--policy",
"workflow-keepalive",
"--target-session",
"alice@rig",
"--interval-seconds",
"60",
"--registered-by",
"ops@kernel",
"--json",
]);
expect(process.exitCode).toBe(1);
expect(logs.some((l) => l.includes("policy_deferred_to_phase_d"))).toBe(true);
});
it("list GETs /api/watchdog/list", async () => {
const { deps, calls } = makeDeps({
routes: { "GET /api/watchdog/list": { status: 200, data: [] } },
});
const program = createProgram({ watchdogDeps: deps });
program.exitOverride();
await program.parseAsync(["node", "rig", "watchdog", "list", "--json"]);
expect(calls.find((c) => c.path === "/api/watchdog/list")).toBeDefined();
});
it("show GETs /api/watchdog/:jobId", async () => {
const { deps, calls } = makeDeps({
routes: { "GET /api/watchdog/job-1": { status: 200, data: { jobId: "job-1" } } },
});
const program = createProgram({ watchdogDeps: deps });
program.exitOverride();
await program.parseAsync(["node", "rig", "watchdog", "show", "job-1", "--json"]);
expect(calls.find((c) => c.path === "/api/watchdog/job-1")).toBeDefined();
});
it("status GETs /api/watchdog/:jobId/status", async () => {
const { deps, calls } = makeDeps({
routes: {
"GET /api/watchdog/job-1/status": {
status: 200,
data: { job: { jobId: "job-1" }, recentHistory: [] },
},
},
});
const program = createProgram({ watchdogDeps: deps });
program.exitOverride();
await program.parseAsync(["node", "rig", "watchdog", "status", "job-1", "--json"]);
expect(calls.find((c) => c.path === "/api/watchdog/job-1/status")).toBeDefined();
});
it("stop POSTs /api/watchdog/:jobId/stop with reason", async () => {
const { deps, calls } = makeDeps({
routes: {
"POST /api/watchdog/job-1/stop": { status: 200, data: { state: "stopped" } },
},
});
const program = createProgram({ watchdogDeps: deps });
program.exitOverride();
await program.parseAsync([
"node",
"rig",
"watchdog",
"stop",
"job-1",
"--reason",
"tester stop",
"--json",
]);
const call = calls.find((c) => c.path === "/api/watchdog/job-1/stop");
expect(call).toBeDefined();
expect((call!.body as { reason: string }).reason).toBe("tester stop");
});
});
@@ -0,0 +1,46 @@
import type { Migration } from "../migrate.js";
/**
* Watchdog jobs (PL-004 Phase C; daemon-native Watchdog supervision tree).
*
* Per PRD § Watchdog + slice IMPL § Write Set: SQLite-canonical watchdog
* jobs that join the daemon supervision tree. Three policy enum values
* SHIPPED at Phase C v1: periodic-reminder, artifact-pool-ready,
* edge-artifact-required. The fourth POC policy (workflow-keepalive)
* ships in Phase D paired with workflow_instances; daemon registration
* MUST reject it with a structured deferred-to-phase-d error.
*
* State enum:
* active - scheduler is evaluating this job on its interval
* stopped - operator stopped the job; scheduler skips it
* terminal - policy declared the job complete (e.g., workflow finished)
*
* Schema details mirror the POC declarative spec (target_session,
* interval_seconds, optional pool-ready-specific scan/active intervals).
* spec_yaml preserves the original operator-supplied YAML for audit.
*/
export const watchdogJobsSchema: Migration = {
name: "031_watchdog_jobs.sql",
sql: `
CREATE TABLE IF NOT EXISTS watchdog_jobs (
job_id TEXT PRIMARY KEY,
policy TEXT NOT NULL,
spec_yaml TEXT NOT NULL,
target_session TEXT NOT NULL,
interval_seconds INTEGER NOT NULL,
active_wake_interval_seconds INTEGER,
scan_interval_seconds INTEGER,
last_evaluation_at TEXT,
last_fire_at TEXT,
state TEXT NOT NULL DEFAULT 'active',
registered_by_session TEXT NOT NULL,
registered_at TEXT NOT NULL,
terminal_reason TEXT
);
CREATE INDEX IF NOT EXISTS idx_watchdog_jobs_state ON watchdog_jobs(state);
CREATE INDEX IF NOT EXISTS idx_watchdog_jobs_policy ON watchdog_jobs(policy);
CREATE INDEX IF NOT EXISTS idx_watchdog_jobs_target_session ON watchdog_jobs(target_session);
CREATE INDEX IF NOT EXISTS idx_watchdog_jobs_active_next_eval
ON watchdog_jobs(last_evaluation_at) WHERE state = 'active';
`,
};
@@ -0,0 +1,47 @@
import type { Migration } from "../migrate.js";
/**
* Watchdog history (PL-004 Phase C; append-only audit log).
*
* Per PRD § Watchdog + slice IMPL § Guard Checkpoint Focus item 2:
* append-only history of meaningful watchdog events. Pure `not_due`
* polls are NOT recorded (matches POC; minimizes table size). Only
* `sent` (delivery executed), `skipped` (policy ran but skipped per
* its own logic — e.g., suppress_if_recent_success or
* no_actionable_artifacts), and `terminal` (policy declared job done)
* are recorded.
*
* Append-only contract: writers only INSERT. UPDATE/DELETE are not
* exposed by WatchdogHistoryLog API; direct SQL UPDATE/DELETE would
* succeed at the DB level (SQLite has no view/role layer) but is a
* contract violation enforced at the domain-layer API boundary.
*
* Outcome enum:
* sent - policy.evaluate() returned action=send and delivery executed
* skipped - policy.evaluate() returned action=skip (with reason)
* terminal - policy.evaluate() returned action=terminal (job declared done)
*
* FK to watchdog_jobs(job_id) for referential integrity. Indexes on
* (job_id, evaluated_at DESC) for "recent history for one job" queries
* and (outcome, evaluated_at DESC) for cross-job outcome queries.
*/
export const watchdogHistorySchema: Migration = {
name: "032_watchdog_history.sql",
sql: `
CREATE TABLE IF NOT EXISTS watchdog_history (
history_id TEXT PRIMARY KEY,
job_id TEXT NOT NULL REFERENCES watchdog_jobs(job_id),
evaluated_at TEXT NOT NULL,
outcome TEXT NOT NULL,
skip_reason TEXT,
delivery_target_session TEXT,
delivery_status TEXT,
delivery_message TEXT,
evaluation_notes TEXT
);
CREATE INDEX IF NOT EXISTS idx_watchdog_history_job_recent
ON watchdog_history(job_id, evaluated_at DESC);
CREATE INDEX IF NOT EXISTS idx_watchdog_history_outcome_recent
ON watchdog_history(outcome, evaluated_at DESC);
`,
};
@@ -0,0 +1,136 @@
// PL-004 Phase C: shared artifact-pool helpers (TypeScript port of POC
// `lib/policies/artifact-pool.js`).
//
// Used by artifact-pool-ready and edge-artifact-required policies. Pure
// filesystem scanner; no event-bus, no DB, no Hono. Reverse-engineers
// the POC contract for include_statuses + frontmatter extraction.
import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
import { join, basename } from "node:path";
export interface ArtifactPoolSpec {
/** Absolute path to the pool directory (or `paths` array variant). */
path?: string;
paths?: string[];
/** File extensions to include (default: ['.md']). */
extensions?: string[];
/** Filter on frontmatter `status:` field (default: include all). */
include_statuses?: string[];
/** Frontmatter field used for keying artifacts (default: 'entry'). */
key_field?: string;
}
export interface ScannedArtifact {
/** Absolute path to the artifact file. */
path: string;
/** Parsed YAML frontmatter (top-level keys only). */
frontmatter: Record<string, string>;
}
const FRONTMATTER_RE = /^---\n([\s\S]*?)\n---/;
/**
* Parse top-level YAML frontmatter from a markdown-style document.
* Only extracts `key: value` pairs (no nested structures, no arrays).
* Mirrors POC's lightweight extraction in lib/policies/artifact-pool.js.
*/
function parseFrontmatter(content: string): Record<string, string> {
const match = FRONTMATTER_RE.exec(content);
if (!match) return {};
const block = match[1] ?? "";
const fm: Record<string, string> = {};
for (const line of block.split("\n")) {
const idx = line.indexOf(":");
if (idx === -1) continue;
const key = line.slice(0, idx).trim();
let value = line.slice(idx + 1).trim();
if (value.startsWith('"') && value.endsWith('"')) value = value.slice(1, -1);
if (value.startsWith("'") && value.endsWith("'")) value = value.slice(1, -1);
if (key) fm[key] = value;
}
return fm;
}
/**
* Scan a single pool directory for artifacts matching extensions +
* include_statuses. Returns ScannedArtifact[] with parsed frontmatter.
*/
function scanPool(spec: ArtifactPoolSpec & { path: string }): ScannedArtifact[] {
if (!existsSync(spec.path)) return [];
let entries: string[];
try {
entries = readdirSync(spec.path);
} catch {
return [];
}
const exts = spec.extensions ?? [".md"];
const includeStatuses = spec.include_statuses;
const out: ScannedArtifact[] = [];
for (const entry of entries) {
if (!exts.some((e) => entry.endsWith(e))) continue;
const fullPath = join(spec.path, entry);
let st;
try {
st = statSync(fullPath);
} catch {
continue;
}
if (!st.isFile()) continue;
let content: string;
try {
content = readFileSync(fullPath, "utf-8");
} catch {
continue;
}
const fm = parseFrontmatter(content);
if (includeStatuses && includeStatuses.length > 0) {
if (!includeStatuses.includes(fm.status ?? "")) continue;
}
out.push({ path: fullPath, frontmatter: fm });
}
return out;
}
/**
* Scan one or more artifact pools per spec(s). Mirrors POC
* `scanArtifactPools(pools)`. Returns the flat union of matches across
* all pools. Missing/unreadable pools yield empty contributions.
*/
export async function scanArtifactPools(
pools: ArtifactPoolSpec | ArtifactPoolSpec[] | undefined,
): Promise<ScannedArtifact[]> {
if (!pools) return [];
const list = Array.isArray(pools) ? pools : [pools];
const out: ScannedArtifact[] = [];
for (const spec of list) {
if (spec.path) {
out.push(...scanPool({ ...spec, path: spec.path }));
} else if (Array.isArray(spec.paths)) {
for (const p of spec.paths) out.push(...scanPool({ ...spec, path: p }));
}
}
return out;
}
/**
* Format an artifact list as bullet lines. Mirrors POC formatArtifactList.
*/
export function formatArtifactList(artifacts: ScannedArtifact[], maxItems: number): string {
return artifacts
.slice(0, maxItems)
.map((a) => `- ${basename(a.path)} (${a.path})`)
.join("\n");
}
/**
* Compute the source-key for an artifact (used by edge-artifact-required).
* Mirrors POC sourceKeyFor: prefer frontmatter[keyField], else basename
* sans .md extension.
*/
export function sourceKeyFor(artifact: ScannedArtifact, keyField: string): string {
const value = artifact.frontmatter[keyField];
if (value === undefined || value === null || value === "") {
return basename(artifact.path).replace(/\.md$/, "");
}
return value;
}
@@ -0,0 +1,53 @@
// PL-004 Phase C: artifact-pool-ready policy (TypeScript port of
// POC `lib/policies/artifact-pool-ready.mjs`, 29 lines).
//
// Scans context.pools, returns:
// - skip { reason: "no_actionable_artifacts" } if pool empty
// - send { message: "Artifact pool ready: <label> has N actionable
// artifact(s)..." + bullet list capped at max_items (default 5) }
import type { Policy, PolicyEvaluation, PolicyJob } from "./types.js";
import {
type ArtifactPoolSpec,
formatArtifactList,
scanArtifactPools,
} from "./artifact-pool-helpers.js";
interface ArtifactPoolReadyContext {
target?: { session?: string };
pools?: ArtifactPoolSpec | ArtifactPoolSpec[];
label?: string;
max_items?: number;
message_template?: string;
}
export const artifactPoolReadyPolicy: Policy = {
name: "artifact-pool-ready",
async evaluate(job: PolicyJob): Promise<PolicyEvaluation> {
const context = job.context as ArtifactPoolReadyContext;
const targetSession = context.target?.session;
if (!targetSession) {
throw Object.assign(new Error("artifact-pool-ready requires context.target.session"), {
code: "policy_spec_invalid",
policy: "artifact-pool-ready",
field: "context.target.session",
});
}
const artifacts = await scanArtifactPools(context.pools);
if (artifacts.length === 0) {
return { action: "skip", reason: "no_actionable_artifacts" };
}
const label = context.label ?? "artifact pool";
const maxItems = context.max_items ?? 5;
const list = formatArtifactList(artifacts, maxItems);
const message =
context.message_template ??
`Artifact pool ready: ${label} has ${artifacts.length} actionable artifact(s).\n\n${list}`;
return {
action: "send",
target: targetSession,
message,
notes: { artifact_count: artifacts.length, label },
};
},
};
@@ -0,0 +1,89 @@
// PL-004 Phase C: edge-artifact-required policy (TypeScript port of
// POC `lib/policies/edge-artifact-required.mjs`, 70 lines).
//
// Scans both source pools and target pools. For each source artifact,
// computes its source-key (frontmatter[key_field] OR basename-sans-md).
// If targets array contains no artifact with matching source-key,
// the source is "missing a downstream edge artifact." Emits a send
// only when one or more sources are missing edges; otherwise skip with
// reason "no_missing_edge_artifacts".
import type { Policy, PolicyEvaluation, PolicyJob } from "./types.js";
import {
type ArtifactPoolSpec,
type ScannedArtifact,
formatArtifactList,
scanArtifactPools,
sourceKeyFor,
} from "./artifact-pool-helpers.js";
interface EdgeArtifactRequiredContext {
target?: { session?: string };
source_pools?: ArtifactPoolSpec | ArtifactPoolSpec[];
target_pools?: ArtifactPoolSpec | ArtifactPoolSpec[];
source_key_field?: string;
target_key_field?: string;
label?: string;
max_items?: number;
message_template?: string;
}
function targetContainsKey(
targets: ScannedArtifact[],
key: string,
targetKeyField: string,
): boolean {
for (const t of targets) {
if (sourceKeyFor(t, targetKeyField) === key) return true;
}
return false;
}
export const edgeArtifactRequiredPolicy: Policy = {
name: "edge-artifact-required",
async evaluate(job: PolicyJob): Promise<PolicyEvaluation> {
const context = job.context as EdgeArtifactRequiredContext;
const targetSession = context.target?.session;
if (!targetSession) {
throw Object.assign(new Error("edge-artifact-required requires context.target.session"), {
code: "policy_spec_invalid",
policy: "edge-artifact-required",
field: "context.target.session",
});
}
const sourceKeyField = context.source_key_field ?? "entry";
const targetKeyField = context.target_key_field ?? "entry";
const sources = await scanArtifactPools(context.source_pools);
// POC override: target scan ignores include_statuses (so we can detect
// any matching downstream artifact regardless of its lifecycle state).
const targetPoolsRaw = context.target_pools;
const targetPoolsList = targetPoolsRaw
? Array.isArray(targetPoolsRaw)
? targetPoolsRaw
: [targetPoolsRaw]
: [];
const targets = await scanArtifactPools(
targetPoolsList.map((p) => ({ ...p, include_statuses: [] })),
);
const missing: ScannedArtifact[] = [];
for (const s of sources) {
const key = sourceKeyFor(s, sourceKeyField);
if (!targetContainsKey(targets, key, targetKeyField)) missing.push(s);
}
if (missing.length === 0) {
return { action: "skip", reason: "no_missing_edge_artifacts" };
}
const label = context.label ?? "edge";
const maxItems = context.max_items ?? 5;
const list = formatArtifactList(missing, maxItems);
const message =
context.message_template ??
`Artifact edge repair needed: ${label} has ${missing.length} upstream artifact(s) with no matching downstream artifact.\n\n${list}`;
return {
action: "send",
target: targetSession,
message,
notes: { missing_count: missing.length, label },
};
},
};
@@ -0,0 +1,38 @@
// PL-004 Phase C: periodic-reminder policy (TypeScript port of
// POC `lib/policies/periodic-reminder.mjs`, 15 lines).
//
// Contract: explicit context.target.session and context.message both
// required. Throws for missing either (hard contract violation).
// Returns action=send unconditionally when invoked (the scheduler
// gates on interval; this policy itself is stateless).
import type { Policy, PolicyEvaluation, PolicyJob } from "./types.js";
interface PeriodicReminderContext {
target?: { session?: string };
message?: string;
}
export const periodicReminderPolicy: Policy = {
name: "periodic-reminder",
async evaluate(job: PolicyJob): Promise<PolicyEvaluation> {
const context = job.context as PeriodicReminderContext;
const targetSession = context.target?.session;
const message = context.message;
if (!targetSession) {
throw Object.assign(new Error("periodic-reminder requires context.target.session"), {
code: "policy_spec_invalid",
policy: "periodic-reminder",
field: "context.target.session",
});
}
if (!message) {
throw Object.assign(new Error("periodic-reminder requires context.message"), {
code: "policy_spec_invalid",
policy: "periodic-reminder",
field: "context.message",
});
}
return { action: "send", target: targetSession, message };
},
};
@@ -0,0 +1,39 @@
// PL-004 Phase C: shared policy contract types.
//
// Each watchdog policy implements `evaluate(job)` returning a
// PolicyEvaluation. Pure: no side-effects, no event-bus, no DB.
// The watchdog-policy-engine maps `action: send` to a delivery call,
// records meaningful outcomes via watchdog-history-log, and emits
// the corresponding RigEvent.
export interface PolicyJob {
jobId: string;
policy: string;
targetSession: string;
intervalSeconds: number;
activeWakeIntervalSeconds: number | null;
scanIntervalSeconds: number | null;
/** Parsed `context:` block from the operator-supplied spec_yaml. */
context: Record<string, unknown>;
lastEvaluationAt: string | null;
lastFireAt: string | null;
registeredBySession: string;
registeredAt: string;
}
export type PolicyEvaluation =
| { action: "send"; target: string; message: string; notes?: Record<string, unknown> }
| { action: "skip"; reason: string; notes?: Record<string, unknown> }
| { action: "terminal"; reason: string; notes?: Record<string, unknown> };
export interface Policy {
/** Stable identifier matching watchdog_jobs.policy enum. */
readonly name: string;
/**
* Pure evaluation. No I/O beyond filesystem reads (artifact-pool
* scans). Throws only for hard contract violations (missing required
* spec fields); recoverable conditions (no actionable artifacts,
* recent successful run) MUST return action=skip.
*/
evaluate(job: PolicyJob): Promise<PolicyEvaluation>;
}
+11 -1
View File
@@ -172,7 +172,17 @@ export type RigEvent =
| { type: "classifier.lease_expired"; leaseId: string; classifierSession: string; expiredAt: string }
| { type: "classifier.dead"; leaseId: string; classifierSession: string; lastHeartbeat: string; detectedAt: string }
| { type: "classifier.reclaimed"; leaseId: string; previousClassifierSession: string; reclaimedBySession: string; reason: string; reclaimedAt: string }
| { type: "view.changed"; viewName: string; cause: string };
| { type: "view.changed"; viewName: string; cause: string }
// PL-004 Phase C: daemon-native Watchdog supervision tree events.
// Three policies in scope (periodic-reminder, artifact-pool-ready,
// edge-artifact-required); workflow-keepalive deferred to Phase D.
// Pure `not_due` polls are NOT recorded in history and NOT emitted
// as events; only meaningful evaluations + lifecycle transitions are.
| { type: "watchdog.evaluation_fired"; jobId: string; policy: string; targetSession: string; deliveryStatus: string }
| { type: "watchdog.evaluation_skipped"; jobId: string; policy: string; skipReason: string }
| { type: "watchdog.evaluation_terminal"; jobId: string; policy: string; terminalReason: string }
| { type: "watchdog.job_registered"; jobId: string; policy: string; targetSession: string; registeredBy: string }
| { type: "watchdog.job_stopped"; jobId: string; reason: string };
export type PersistedEvent = RigEvent & {
seq: number;
@@ -0,0 +1,136 @@
import type Database from "better-sqlite3";
import { ulid } from "ulid";
/**
* Watchdog history log (PL-004 Phase C; append-only audit writer).
*
* Per slice IMPL § Guard Checkpoint Focus item 2:
* - Pure `not_due` polls are NOT recorded (matches POC; minimizes
* table size). Only `sent` / `skipped` / `terminal` outcomes pass
* through this log.
* - Append-only API surface: only `record()` exposed. UPDATE/DELETE
* are not exposed. SQLite has no view/role layer so direct SQL
* could mutate, but the contract is enforced at this domain
* boundary.
*/
export type WatchdogOutcome = "sent" | "skipped" | "terminal";
export interface WatchdogHistoryRecordInput {
jobId: string;
evaluatedAt: string;
outcome: WatchdogOutcome;
skipReason?: string | null;
deliveryTargetSession?: string | null;
deliveryStatus?: string | null;
deliveryMessage?: string | null;
evaluationNotes?: Record<string, unknown> | null;
}
export interface WatchdogHistoryEntry {
historyId: string;
jobId: string;
evaluatedAt: string;
outcome: WatchdogOutcome;
skipReason: string | null;
deliveryTargetSession: string | null;
deliveryStatus: string | null;
deliveryMessage: string | null;
evaluationNotes: Record<string, unknown> | null;
}
interface HistoryRow {
history_id: string;
job_id: string;
evaluated_at: string;
outcome: string;
skip_reason: string | null;
delivery_target_session: string | null;
delivery_status: string | null;
delivery_message: string | null;
evaluation_notes: string | null;
}
export class WatchdogHistoryLog {
constructor(private readonly db: Database.Database) {}
/**
* Append a meaningful evaluation outcome. Returns the persisted entry.
* Caller MUST NOT invoke this for `not_due` pure-skip outcomes — those
* are not recorded (per POC + IMPL guidance).
*/
record(input: WatchdogHistoryRecordInput): WatchdogHistoryEntry {
const historyId = ulid();
this.db
.prepare(
`INSERT INTO watchdog_history (
history_id, job_id, evaluated_at, outcome,
skip_reason, delivery_target_session, delivery_status,
delivery_message, evaluation_notes
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
)
.run(
historyId,
input.jobId,
input.evaluatedAt,
input.outcome,
input.skipReason ?? null,
input.deliveryTargetSession ?? null,
input.deliveryStatus ?? null,
input.deliveryMessage ?? null,
input.evaluationNotes ? JSON.stringify(input.evaluationNotes) : null,
);
return {
historyId,
jobId: input.jobId,
evaluatedAt: input.evaluatedAt,
outcome: input.outcome,
skipReason: input.skipReason ?? null,
deliveryTargetSession: input.deliveryTargetSession ?? null,
deliveryStatus: input.deliveryStatus ?? null,
deliveryMessage: input.deliveryMessage ?? null,
evaluationNotes: input.evaluationNotes ?? null,
};
}
listForJob(jobId: string, limit = 50): WatchdogHistoryEntry[] {
const rows = this.db
.prepare(
`SELECT * FROM watchdog_history WHERE job_id = ?
ORDER BY evaluated_at DESC, history_id DESC LIMIT ?`,
)
.all(jobId, limit) as HistoryRow[];
return rows.map(rowToEntry);
}
listAll(limit = 100): WatchdogHistoryEntry[] {
const rows = this.db
.prepare(
`SELECT * FROM watchdog_history
ORDER BY evaluated_at DESC, history_id DESC LIMIT ?`,
)
.all(limit) as HistoryRow[];
return rows.map(rowToEntry);
}
countForJob(jobId: string): number {
const row = this.db
.prepare(`SELECT COUNT(*) AS n FROM watchdog_history WHERE job_id = ?`)
.get(jobId) as { n: number };
return row.n;
}
}
function rowToEntry(row: HistoryRow): WatchdogHistoryEntry {
return {
historyId: row.history_id,
jobId: row.job_id,
evaluatedAt: row.evaluated_at,
outcome: row.outcome as WatchdogOutcome,
skipReason: row.skip_reason,
deliveryTargetSession: row.delivery_target_session,
deliveryStatus: row.delivery_status,
deliveryMessage: row.delivery_message,
evaluationNotes: row.evaluation_notes ? JSON.parse(row.evaluation_notes) : null,
};
}
@@ -0,0 +1,230 @@
import type Database from "better-sqlite3";
import { ulid } from "ulid";
/**
* Watchdog jobs repository (PL-004 Phase C).
*
* Owns all reads/writes against `watchdog_jobs`. Pure persistence; no
* event-bus, no scheduler, no policy dispatch. Composed by the
* scheduler and policy engine.
*
* Phase C v1 enforces the policy enum against three values:
* periodic-reminder, artifact-pool-ready, edge-artifact-required.
* `workflow-keepalive` is intentionally rejected with a structured
* `policy_deferred_to_phase_d` error.
*/
export const PHASE_C_POLICIES = [
"periodic-reminder",
"artifact-pool-ready",
"edge-artifact-required",
] as const;
export type WatchdogPolicyName = (typeof PHASE_C_POLICIES)[number];
export const WORKFLOW_KEEPALIVE_DEFERRED_POLICY = "workflow-keepalive";
export type WatchdogJobState = "active" | "stopped" | "terminal";
export interface WatchdogJob {
jobId: string;
policy: WatchdogPolicyName;
specYaml: string;
targetSession: string;
intervalSeconds: number;
activeWakeIntervalSeconds: number | null;
scanIntervalSeconds: number | null;
lastEvaluationAt: string | null;
lastFireAt: string | null;
state: WatchdogJobState;
registeredBySession: string;
registeredAt: string;
terminalReason: string | null;
}
export interface RegisterWatchdogJobInput {
policy: string;
specYaml: string;
targetSession: string;
intervalSeconds: number;
activeWakeIntervalSeconds?: number | null;
scanIntervalSeconds?: number | null;
registeredBySession: string;
}
interface JobRow {
job_id: string;
policy: string;
spec_yaml: string;
target_session: string;
interval_seconds: number;
active_wake_interval_seconds: number | null;
scan_interval_seconds: number | null;
last_evaluation_at: string | null;
last_fire_at: string | null;
state: string;
registered_by_session: string;
registered_at: string;
terminal_reason: string | null;
}
export class WatchdogJobsError extends Error {
constructor(
public readonly code: string,
message: string,
public readonly details?: Record<string, unknown>,
) {
super(message);
this.name = "WatchdogJobsError";
}
}
export class WatchdogJobsRepository {
constructor(
private readonly db: Database.Database,
private readonly now: () => Date = () => new Date(),
) {}
register(input: RegisterWatchdogJobInput): WatchdogJob {
if (input.policy === WORKFLOW_KEEPALIVE_DEFERRED_POLICY) {
throw new WatchdogJobsError(
"policy_deferred_to_phase_d",
`policy 'workflow-keepalive' ships in PL-004 Phase D paired with workflow_instances; not available in Phase C v1`,
{ policy: input.policy, phase: "C", deferredTo: "D" },
);
}
if (!PHASE_C_POLICIES.includes(input.policy as WatchdogPolicyName)) {
throw new WatchdogJobsError(
"policy_unknown",
`unknown watchdog policy '${input.policy}'; Phase C v1 supports: ${PHASE_C_POLICIES.join(", ")}`,
{ policy: input.policy, supported: [...PHASE_C_POLICIES] },
);
}
if (!Number.isInteger(input.intervalSeconds) || input.intervalSeconds <= 0) {
throw new WatchdogJobsError(
"interval_invalid",
`interval_seconds must be a positive integer (got ${input.intervalSeconds})`,
{ intervalSeconds: input.intervalSeconds },
);
}
if (!input.targetSession || !input.targetSession.includes("@")) {
throw new WatchdogJobsError(
"target_session_invalid",
`target_session must be canonical '<member>@<rig>' (got '${input.targetSession}')`,
{ targetSession: input.targetSession },
);
}
const jobId = ulid();
const registeredAt = this.now().toISOString();
this.db
.prepare(
`INSERT INTO watchdog_jobs (
job_id, policy, spec_yaml, target_session,
interval_seconds, active_wake_interval_seconds, scan_interval_seconds,
last_evaluation_at, last_fire_at, state,
registered_by_session, registered_at, terminal_reason
) VALUES (?, ?, ?, ?, ?, ?, ?, NULL, NULL, 'active', ?, ?, NULL)`,
)
.run(
jobId,
input.policy,
input.specYaml,
input.targetSession,
input.intervalSeconds,
input.activeWakeIntervalSeconds ?? null,
input.scanIntervalSeconds ?? null,
input.registeredBySession,
registeredAt,
);
return this.getByIdOrThrow(jobId);
}
getById(jobId: string): WatchdogJob | null {
const row = this.db
.prepare(`SELECT * FROM watchdog_jobs WHERE job_id = ?`)
.get(jobId) as JobRow | undefined;
return row ? rowToJob(row) : null;
}
getByIdOrThrow(jobId: string): WatchdogJob {
const job = this.getById(jobId);
if (!job) {
throw new WatchdogJobsError(
"job_not_found",
`watchdog job ${jobId} not found`,
{ jobId },
);
}
return job;
}
listAll(): WatchdogJob[] {
const rows = this.db
.prepare(`SELECT * FROM watchdog_jobs ORDER BY registered_at ASC`)
.all() as JobRow[];
return rows.map(rowToJob);
}
listActive(): WatchdogJob[] {
const rows = this.db
.prepare(`SELECT * FROM watchdog_jobs WHERE state = 'active' ORDER BY registered_at ASC`)
.all() as JobRow[];
return rows.map(rowToJob);
}
recordEvaluation(jobId: string, evaluatedAt: string, fired: boolean): void {
if (fired) {
this.db
.prepare(
`UPDATE watchdog_jobs SET last_evaluation_at = ?, last_fire_at = ? WHERE job_id = ?`,
)
.run(evaluatedAt, evaluatedAt, jobId);
} else {
this.db
.prepare(`UPDATE watchdog_jobs SET last_evaluation_at = ? WHERE job_id = ?`)
.run(evaluatedAt, jobId);
}
}
markTerminal(jobId: string, reason: string): void {
this.db
.prepare(
`UPDATE watchdog_jobs SET state = 'terminal', terminal_reason = ? WHERE job_id = ?`,
)
.run(reason, jobId);
}
stop(jobId: string, reason = "operator_stopped"): WatchdogJob {
const existing = this.getByIdOrThrow(jobId);
if (existing.state === "terminal") {
throw new WatchdogJobsError(
"job_terminal",
`cannot stop watchdog job ${jobId}: state is terminal`,
{ jobId, state: existing.state },
);
}
if (existing.state === "stopped") return existing;
this.db
.prepare(`UPDATE watchdog_jobs SET state = 'stopped', terminal_reason = ? WHERE job_id = ?`)
.run(reason, jobId);
return this.getByIdOrThrow(jobId);
}
}
function rowToJob(row: JobRow): WatchdogJob {
return {
jobId: row.job_id,
policy: row.policy as WatchdogPolicyName,
specYaml: row.spec_yaml,
targetSession: row.target_session,
intervalSeconds: row.interval_seconds,
activeWakeIntervalSeconds: row.active_wake_interval_seconds,
scanIntervalSeconds: row.scan_interval_seconds,
lastEvaluationAt: row.last_evaluation_at,
lastFireAt: row.last_fire_at,
state: row.state as WatchdogJobState,
registeredBySession: row.registered_by_session,
registeredAt: row.registered_at,
terminalReason: row.terminal_reason,
};
}
@@ -0,0 +1,411 @@
import type { EventBus } from "./event-bus.js";
import type { Policy, PolicyEvaluation, PolicyJob } from "./policies/types.js";
import { artifactPoolReadyPolicy } from "./policies/artifact-pool-ready.js";
import { edgeArtifactRequiredPolicy } from "./policies/edge-artifact-required.js";
import { periodicReminderPolicy } from "./policies/periodic-reminder.js";
import {
type WatchdogJob,
type WatchdogJobsRepository,
} from "./watchdog-jobs-repository.js";
import type { WatchdogHistoryEntry, WatchdogHistoryLog } from "./watchdog-history-log.js";
/**
* Watchdog policy engine (PL-004 Phase C).
*
* Owns:
* 1. Policy resolution by name (registry).
* 2. spec_yaml → context parse for the policy.
* 3. Dispatch evaluate(job) → PolicyEvaluation.
* 4. For action=send: invoke delivery callback, record history,
* emit watchdog.evaluation_fired.
* 5. For action=skip: record history, emit watchdog.evaluation_skipped.
* 6. For action=terminal: mark job terminal, record history, emit
* watchdog.evaluation_terminal.
*
* The actual delivery is injected as a callback so tests can stub it
* and so the daemon supervision tree can wire it to SessionTransport
* without this module needing a concrete dependency.
*/
export interface DeliveryRequest {
targetSession: string;
message: string;
}
export interface DeliveryOutcome {
status: "ok" | "failed";
error?: string;
}
export type DeliveryFn = (req: DeliveryRequest) => Promise<DeliveryOutcome>;
export interface PolicyContextParser {
/** Parse the operator-supplied spec_yaml into the `context:` block. */
(specYaml: string): Record<string, unknown>;
}
interface WatchdogPolicyEngineDeps {
jobsRepo: WatchdogJobsRepository;
historyLog: WatchdogHistoryLog;
eventBus: EventBus;
deliver: DeliveryFn;
/** Optional override: defaults to a YAML-aware parser. */
parseContext?: PolicyContextParser;
now?: () => Date;
}
const POLICY_REGISTRY: Map<string, Policy> = new Map([
[periodicReminderPolicy.name, periodicReminderPolicy],
[artifactPoolReadyPolicy.name, artifactPoolReadyPolicy],
[edgeArtifactRequiredPolicy.name, edgeArtifactRequiredPolicy],
]);
export interface EvaluationResult {
job: WatchdogJob;
outcome: PolicyEvaluation;
history: WatchdogHistoryEntry | null;
delivery: DeliveryOutcome | null;
}
export class WatchdogPolicyEngine {
private readonly jobsRepo: WatchdogJobsRepository;
private readonly historyLog: WatchdogHistoryLog;
private readonly eventBus: EventBus;
private readonly deliver: DeliveryFn;
private readonly parseContext: PolicyContextParser;
private readonly now: () => Date;
constructor(deps: WatchdogPolicyEngineDeps) {
this.jobsRepo = deps.jobsRepo;
this.historyLog = deps.historyLog;
this.eventBus = deps.eventBus;
this.deliver = deps.deliver;
this.parseContext = deps.parseContext ?? defaultParseContext;
this.now = deps.now ?? (() => new Date());
}
resolvePolicy(name: string): Policy | undefined {
return POLICY_REGISTRY.get(name);
}
/**
* Evaluate one watchdog job. Side-effects: history record (for
* meaningful outcomes), event emission, and (on send) delivery.
*
* Pure `not_due` decisions are NOT this engine's concern; the
* scheduler decides dueness and only calls evaluate for due jobs.
* If the policy itself returns skip with an internal reason
* (no_actionable_artifacts, etc.) we DO record that.
*/
async evaluate(job: WatchdogJob): Promise<EvaluationResult> {
const policy = this.resolvePolicy(job.policy);
if (!policy) {
const reason = `unknown_policy:${job.policy}`;
this.jobsRepo.markTerminal(job.jobId, reason);
const evaluatedAt = this.now().toISOString();
const history = this.historyLog.record({
jobId: job.jobId,
evaluatedAt,
outcome: "terminal",
skipReason: reason,
});
this.eventBus.emit({
type: "watchdog.evaluation_terminal",
jobId: job.jobId,
policy: job.policy,
terminalReason: reason,
});
return {
job: this.jobsRepo.getByIdOrThrow(job.jobId),
outcome: { action: "terminal", reason },
history,
delivery: null,
};
}
const context = this.parseContext(job.specYaml);
const policyJob: PolicyJob = {
jobId: job.jobId,
policy: job.policy,
targetSession: job.targetSession,
intervalSeconds: job.intervalSeconds,
activeWakeIntervalSeconds: job.activeWakeIntervalSeconds,
scanIntervalSeconds: job.scanIntervalSeconds,
context,
lastEvaluationAt: job.lastEvaluationAt,
lastFireAt: job.lastFireAt,
registeredBySession: job.registeredBySession,
registeredAt: job.registeredAt,
};
const outcome = await policy.evaluate(policyJob);
const evaluatedAt = this.now().toISOString();
if (outcome.action === "send") {
const delivery = await this.deliver({
targetSession: outcome.target,
message: outcome.message,
});
const history = this.historyLog.record({
jobId: job.jobId,
evaluatedAt,
outcome: "sent",
deliveryTargetSession: outcome.target,
deliveryStatus: delivery.status,
deliveryMessage: outcome.message,
evaluationNotes: outcome.notes ?? null,
});
this.jobsRepo.recordEvaluation(job.jobId, evaluatedAt, true);
this.eventBus.emit({
type: "watchdog.evaluation_fired",
jobId: job.jobId,
policy: job.policy,
targetSession: outcome.target,
deliveryStatus: delivery.status,
});
return {
job: this.jobsRepo.getByIdOrThrow(job.jobId),
outcome,
history,
delivery,
};
}
if (outcome.action === "skip") {
const history = this.historyLog.record({
jobId: job.jobId,
evaluatedAt,
outcome: "skipped",
skipReason: outcome.reason,
evaluationNotes: outcome.notes ?? null,
});
this.jobsRepo.recordEvaluation(job.jobId, evaluatedAt, false);
this.eventBus.emit({
type: "watchdog.evaluation_skipped",
jobId: job.jobId,
policy: job.policy,
skipReason: outcome.reason,
});
return {
job: this.jobsRepo.getByIdOrThrow(job.jobId),
outcome,
history,
delivery: null,
};
}
// terminal
this.jobsRepo.markTerminal(job.jobId, outcome.reason);
const history = this.historyLog.record({
jobId: job.jobId,
evaluatedAt,
outcome: "terminal",
skipReason: outcome.reason,
evaluationNotes: outcome.notes ?? null,
});
this.eventBus.emit({
type: "watchdog.evaluation_terminal",
jobId: job.jobId,
policy: job.policy,
terminalReason: outcome.reason,
});
return {
job: this.jobsRepo.getByIdOrThrow(job.jobId),
outcome,
history,
delivery: null,
};
}
}
/**
* Default spec_yaml parser. Extracts the `context:` block as a
* generic Record<string, unknown>. Uses a minimal recursive
* indentation-based parser sufficient for the POC spec shape:
* policy: ...
* target: <member>@<rig>
* interval_seconds: 1800
* context:
* target:
* session: ...
* message: "..."
* pools:
* - path: /abs/...
* include_statuses: [ready]
*
* For robust YAML, a future maintenance pass can swap in `yaml` or
* `js-yaml`; the POC schemas are simple enough that this parser
* handles them in-tree without an extra dependency.
*/
function defaultParseContext(specYaml: string): Record<string, unknown> {
const lines = specYaml.split("\n");
let i = 0;
while (i < lines.length) {
const line = lines[i] ?? "";
if (/^context\s*:\s*$/.test(line)) {
i++;
const block: string[] = [];
while (i < lines.length) {
const sub = lines[i] ?? "";
if (sub.length === 0) {
block.push(sub);
i++;
continue;
}
if (sub.startsWith(" ") || sub.startsWith("\t")) {
block.push(sub);
i++;
continue;
}
break;
}
// Strip leading two-space indent to make it a top-level YAML doc.
const dedented = block
.map((l) => (l.startsWith(" ") ? l.slice(2) : l))
.join("\n");
return parseSimpleYaml(dedented) as Record<string, unknown>;
}
i++;
}
return {};
}
/**
* Minimal indentation-aware YAML parser. Supports nested mappings,
* inline `[a, b]` arrays, dash-prefixed sequences of mappings, and
* scalar string values (quoted or unquoted). Sufficient for the POC
* spec set; not a general YAML implementation.
*/
function parseSimpleYaml(src: string): unknown {
const lines = src.split("\n").filter((l) => l.trim().length > 0 && !l.trim().startsWith("#"));
const idx = { i: 0 };
return parseYamlBlock(lines, idx, 0);
}
function indentOf(line: string): number {
let n = 0;
while (n < line.length && line[n] === " ") n++;
return n;
}
function parseScalar(raw: string): unknown {
const trimmed = raw.trim();
if (trimmed === "null" || trimmed === "~") return null;
if (trimmed === "true") return true;
if (trimmed === "false") return false;
if (/^-?\d+$/.test(trimmed)) return Number.parseInt(trimmed, 10);
if (/^-?\d+\.\d+$/.test(trimmed)) return Number.parseFloat(trimmed);
if (trimmed.startsWith("[") && trimmed.endsWith("]")) {
const inner = trimmed.slice(1, -1).trim();
if (inner === "") return [];
return inner.split(",").map((p) => parseScalar(p));
}
if (trimmed.startsWith('"') && trimmed.endsWith('"')) return trimmed.slice(1, -1);
if (trimmed.startsWith("'") && trimmed.endsWith("'")) return trimmed.slice(1, -1);
return trimmed;
}
function parseYamlBlock(lines: string[], idx: { i: number }, expectedIndent: number): unknown {
if (idx.i >= lines.length) return null;
const first = lines[idx.i] ?? "";
const firstIndent = indentOf(first);
if (firstIndent < expectedIndent) return null;
const trimmed = first.trim();
if (trimmed.startsWith("- ")) {
return parseYamlSeq(lines, idx, firstIndent);
}
return parseYamlMap(lines, idx, firstIndent);
}
function parseYamlMap(
lines: string[],
idx: { i: number },
blockIndent: number,
): Record<string, unknown> {
const result: Record<string, unknown> = {};
while (idx.i < lines.length) {
const line = lines[idx.i] ?? "";
const ind = indentOf(line);
if (ind < blockIndent) break;
if (ind > blockIndent) break;
const trimmed = line.trim();
const colonIdx = trimmed.indexOf(":");
if (colonIdx === -1) break;
const key = trimmed.slice(0, colonIdx).trim();
const valuePart = trimmed.slice(colonIdx + 1).trim();
idx.i++;
if (valuePart.length > 0) {
result[key] = parseScalar(valuePart);
} else {
// Nested block: probe next line indent.
if (idx.i < lines.length) {
const next = lines[idx.i] ?? "";
const nextIndent = indentOf(next);
if (nextIndent > blockIndent) {
result[key] = parseYamlBlock(lines, idx, nextIndent);
continue;
}
}
result[key] = null;
}
}
return result;
}
function parseYamlSeq(
lines: string[],
idx: { i: number },
blockIndent: number,
): unknown[] {
const result: unknown[] = [];
while (idx.i < lines.length) {
const line = lines[idx.i] ?? "";
const ind = indentOf(line);
if (ind < blockIndent) break;
if (ind > blockIndent) break;
const trimmed = line.trim();
if (!trimmed.startsWith("- ")) break;
const rest = trimmed.slice(2).trim();
idx.i++;
if (rest.includes(":")) {
// First key of an inline mapping element. Synthesize a virtual
// sub-block by re-injecting the rest as a key + scanning further
// lines at deeper indent.
const colonIdx = rest.indexOf(":");
const k = rest.slice(0, colonIdx).trim();
const v = rest.slice(colonIdx + 1).trim();
const elem: Record<string, unknown> = {};
if (v.length > 0) {
elem[k] = parseScalar(v);
} else if (idx.i < lines.length && indentOf(lines[idx.i] ?? "") > blockIndent + 2) {
elem[k] = parseYamlBlock(lines, idx, indentOf(lines[idx.i] ?? ""));
} else {
elem[k] = null;
}
// Continue gathering same-element keys at indent blockIndent + 2.
while (idx.i < lines.length) {
const sub = lines[idx.i] ?? "";
const subIndent = indentOf(sub);
if (subIndent !== blockIndent + 2) break;
const subTrim = sub.trim();
if (subTrim.startsWith("- ")) break;
const sCol = subTrim.indexOf(":");
if (sCol === -1) break;
const sk = subTrim.slice(0, sCol).trim();
const sv = subTrim.slice(sCol + 1).trim();
idx.i++;
if (sv.length > 0) {
elem[sk] = parseScalar(sv);
} else if (idx.i < lines.length && indentOf(lines[idx.i] ?? "") > blockIndent + 2) {
elem[sk] = parseYamlBlock(lines, idx, indentOf(lines[idx.i] ?? ""));
} else {
elem[sk] = null;
}
}
result.push(elem);
} else {
result.push(parseScalar(rest));
}
}
return result;
}
@@ -0,0 +1,146 @@
import type { WatchdogJob, WatchdogJobsRepository } from "./watchdog-jobs-repository.js";
import type { WatchdogPolicyEngine } from "./watchdog-policy-engine.js";
/**
* Watchdog scheduler (PL-004 Phase C; daemon-native supervision tree
* member). Joins createDaemon's lifecycle: started by startup.ts after
* the policy engine is ready, stopped during graceful shutdown.
*
* Loop semantics:
* - Wakes every `tickIntervalMs` (default 1000) and queries the
* repository for active jobs whose `interval_seconds` has elapsed
* since `last_evaluation_at` (or who have never been evaluated).
* - For each due job, calls policyEngine.evaluate(job). The engine
* records meaningful outcomes; pure not-due polls are filtered
* here and never reach history.
* - SQLite is the canonical schedule state — restart recovery is
* automatic: on startup, `listActive()` returns the same set
* and `last_evaluation_at` determines next-due.
*
* Concurrency:
* - One in-flight tick at a time. If a tick takes longer than
* tickIntervalMs (long policy.evaluate or slow delivery), the
* next tick is delayed; we don't queue overlapping ticks.
* - Within a tick, jobs are evaluated sequentially. This bounds
* resource use and matches the POC's single-loop behavior.
*
* Shutdown:
* - stop() sets shuttingDown=true, clears the timer, and awaits the
* in-flight tick (if any). Idempotent.
*/
export interface WatchdogSchedulerDeps {
jobsRepo: WatchdogJobsRepository;
policyEngine: WatchdogPolicyEngine;
/** Tick wake-up cadence. Default 1000 ms. */
tickIntervalMs?: number;
/** Override clock for tests. */
now?: () => Date;
/** Override timer scheduler for tests. */
setTimer?: (cb: () => void, ms: number) => NodeJS.Timeout;
clearTimer?: (handle: NodeJS.Timeout) => void;
/** Notification on tick errors (for telemetry; defaults to console.error). */
onTickError?: (err: unknown) => void;
}
export class WatchdogScheduler {
private readonly jobsRepo: WatchdogJobsRepository;
private readonly policyEngine: WatchdogPolicyEngine;
private readonly tickIntervalMs: number;
private readonly now: () => Date;
private readonly setTimer: (cb: () => void, ms: number) => NodeJS.Timeout;
private readonly clearTimer: (handle: NodeJS.Timeout) => void;
private readonly onTickError: (err: unknown) => void;
private timer: NodeJS.Timeout | null = null;
private inflight: Promise<void> | null = null;
private shuttingDown = false;
private started = false;
constructor(deps: WatchdogSchedulerDeps) {
this.jobsRepo = deps.jobsRepo;
this.policyEngine = deps.policyEngine;
this.tickIntervalMs = deps.tickIntervalMs ?? 1000;
this.now = deps.now ?? (() => new Date());
this.setTimer = deps.setTimer ?? ((cb, ms) => setTimeout(cb, ms));
this.clearTimer = deps.clearTimer ?? ((handle) => clearTimeout(handle));
this.onTickError = deps.onTickError ?? ((err) => console.error("[watchdog] tick error", err));
}
start(): void {
if (this.started) return;
this.started = true;
this.shuttingDown = false;
this.scheduleNextTick();
}
async stop(): Promise<void> {
if (!this.started) return;
this.shuttingDown = true;
if (this.timer) {
this.clearTimer(this.timer);
this.timer = null;
}
if (this.inflight) {
try {
await this.inflight;
} catch {
// Errors already logged via onTickError; suppress during shutdown.
}
}
this.started = false;
}
isRunning(): boolean {
return this.started && !this.shuttingDown;
}
/**
* Run one tick synchronously (await result). Exposed for tests so
* they can drive ticks without timers. Production path calls this
* via the timer loop.
*/
async runTickNow(): Promise<void> {
if (this.inflight) {
await this.inflight;
return;
}
this.inflight = this.tick();
try {
await this.inflight;
} finally {
this.inflight = null;
}
}
private scheduleNextTick(): void {
if (this.shuttingDown) return;
this.timer = this.setTimer(() => {
void this.runTickNow()
.catch((err) => this.onTickError(err))
.finally(() => this.scheduleNextTick());
}, this.tickIntervalMs);
}
private async tick(): Promise<void> {
if (this.shuttingDown) return;
const nowMs = this.now().getTime();
const active = this.jobsRepo.listActive();
for (const job of active) {
if (this.shuttingDown) return;
if (!isDue(job, nowMs)) continue;
try {
await this.policyEngine.evaluate(job);
} catch (err) {
this.onTickError(err);
}
}
}
}
export function isDue(job: WatchdogJob, nowMs: number): boolean {
if (!job.lastEvaluationAt) return true;
const last = Date.parse(job.lastEvaluationAt);
if (Number.isNaN(last)) return true;
return nowMs - last >= job.intervalSeconds * 1000;
}
+22 -2
View File
@@ -6,15 +6,35 @@ export async function startServer(port?: number) {
const p = port ?? parseInt(readOpenRigEnv("OPENRIG_PORT", "RIGGED_PORT") ?? "7433", 10);
const dbPath = readOpenRigEnv("OPENRIG_DB", "RIGGED_DB") ?? "openrig.sqlite";
const { app, contextMonitor } = await createDaemon({ dbPath });
const { app, contextMonitor, deps } = await createDaemon({ dbPath });
const h = readOpenRigEnv("OPENRIG_HOST", "RIGGED_HOST") ?? "127.0.0.1";
return serve({ fetch: app.fetch, port: p, hostname: h }, (info) => {
const server = serve({ fetch: app.fetch, port: p, hostname: h }, (info) => {
console.log(`OpenRig daemon listening on http://localhost:${info.port}`);
// Start context monitor polling only after successful server bind
contextMonitor.start();
// PL-004 Phase C: start watchdog scheduler. Joins the supervision
// tree post-bind so the HTTP surface is ready before the first
// tick (matches contextMonitor pattern).
deps.watchdogScheduler?.start();
});
// PL-004 Phase C: graceful shutdown — stop scheduler before process
// exit so any in-flight policy evaluation completes (or is awaited).
const shutdown = async (sig: string) => {
console.log(`OpenRig daemon received ${sig}; shutting down`);
try {
await deps.watchdogScheduler?.stop();
} catch (err) {
console.error("[watchdog] shutdown error", err);
}
process.exit(0);
};
process.once("SIGINT", () => void shutdown("SIGINT"));
process.once("SIGTERM", () => void shutdown("SIGTERM"));
return server;
}
// Only start the server when this file is executed directly (not imported).
+170
View File
@@ -0,0 +1,170 @@
import { Hono } from "hono";
import { streamSSE } from "hono/streaming";
import type { EventBus } from "../domain/event-bus.js";
import type { WatchdogHistoryLog } from "../domain/watchdog-history-log.js";
import {
type WatchdogJobsRepository,
WatchdogJobsError,
} from "../domain/watchdog-jobs-repository.js";
/**
* Watchdog HTTP routes (PL-004 Phase C). Backs `rig watchdog` CLI verb.
*
* Per Phase A R1 SSE route-order lesson: SSE/static routes mounted
* BEFORE the bare-param /:job_id catchall so the literal `/sse` and
* literal action paths win over the param route.
*
* Endpoints:
* POST /register register a new watchdog job
* GET /list list all watchdog jobs (active + stopped + terminal)
* GET /sse SSE stream of watchdog.* events
* GET /:job_id show one job
* GET /:job_id/status job + recent history (compact summary)
* POST /:job_id/stop operator stop
*/
export function watchdogRoutes(): Hono {
const app = new Hono();
function getJobsRepo(c: { get: (key: string) => unknown }): WatchdogJobsRepository {
return c.get("watchdogJobsRepo" as never) as WatchdogJobsRepository;
}
function getHistoryLog(c: { get: (key: string) => unknown }): WatchdogHistoryLog {
return c.get("watchdogHistoryLog" as never) as WatchdogHistoryLog;
}
function getEventBus(c: { get: (key: string) => unknown }): EventBus {
return c.get("eventBus" as never) as EventBus;
}
function errorResponse(
c: { json: (body: unknown, status?: number) => Response },
err: unknown,
): Response {
if (err instanceof WatchdogJobsError) {
const status =
err.code === "job_not_found" ? 404
: err.code === "policy_unknown" ? 400
: err.code === "policy_deferred_to_phase_d" ? 400
: err.code === "interval_invalid" ? 400
: err.code === "target_session_invalid" ? 400
: err.code === "job_terminal" ? 409
: 500;
return c.json(
{ error: err.code, message: err.message, ...(err.details ?? {}) },
status as 200,
);
}
const message = err instanceof Error ? err.message : "internal error";
return c.json({ error: "internal_error", message }, 500);
}
app.post("/register", async (c) => {
const body = await c.req
.json<{
policy?: string;
specYaml?: string;
targetSession?: string;
intervalSeconds?: number;
activeWakeIntervalSeconds?: number;
scanIntervalSeconds?: number;
registeredBySession?: string;
}>()
.catch(() => ({} as never));
if (!body.policy) return c.json({ error: "policy is required" }, 400);
if (!body.specYaml) return c.json({ error: "specYaml is required" }, 400);
if (!body.targetSession) return c.json({ error: "targetSession is required" }, 400);
if (typeof body.intervalSeconds !== "number") {
return c.json({ error: "intervalSeconds is required" }, 400);
}
if (!body.registeredBySession) return c.json({ error: "registeredBySession is required" }, 400);
try {
const job = getJobsRepo(c).register({
policy: body.policy,
specYaml: body.specYaml,
targetSession: body.targetSession,
intervalSeconds: body.intervalSeconds,
activeWakeIntervalSeconds: body.activeWakeIntervalSeconds,
scanIntervalSeconds: body.scanIntervalSeconds,
registeredBySession: body.registeredBySession,
});
getEventBus(c).emit({
type: "watchdog.job_registered",
jobId: job.jobId,
policy: job.policy,
targetSession: job.targetSession,
registeredBy: job.registeredBySession,
});
return c.json(job, 201);
} catch (err) {
return errorResponse(c, err);
}
});
// GET /list — list all watchdog jobs.
// Literal path before /:job_id (per Phase A R1 SSE route-order lesson).
app.get("/list", (c) => {
const jobs = getJobsRepo(c).listAll();
return c.json(jobs);
});
// SSE for watchdog.* events. MUST precede /:job_id so the literal
// path wins. Per Phase A R1 SSE route-order lesson.
const sseHandler = (c: Parameters<typeof streamSSE>[0]) => {
const eventBus = getEventBus(c);
return streamSSE(c, async (stream) => {
const unsubscribe = eventBus.subscribe((event) => {
if (
event.type !== "watchdog.evaluation_fired" &&
event.type !== "watchdog.evaluation_skipped" &&
event.type !== "watchdog.evaluation_terminal" &&
event.type !== "watchdog.job_registered" &&
event.type !== "watchdog.job_stopped"
) return;
const sse = { id: String(event.seq), data: JSON.stringify(event) };
stream.writeSSE(sse).catch(() => {});
});
try {
await new Promise<void>((resolve) => stream.onAbort(() => resolve()));
} finally {
unsubscribe();
}
});
};
app.get("/sse", sseHandler);
app.get("/watch", sseHandler);
app.get("/:job_id/status", (c) => {
const jobId = c.req.param("job_id");
const job = getJobsRepo(c).getById(jobId);
if (!job) return c.json({ error: "job_not_found", jobId }, 404);
const recentHistory = getHistoryLog(c).listForJob(jobId, 20);
return c.json({ job, recentHistory });
});
app.post("/:job_id/stop", async (c) => {
const jobId = c.req.param("job_id");
const body = await c.req
.json<{ reason?: string }>()
.catch(() => ({} as { reason?: string }));
try {
const job = getJobsRepo(c).stop(jobId, body.reason ?? "operator_stopped");
getEventBus(c).emit({
type: "watchdog.job_stopped",
jobId: job.jobId,
reason: body.reason ?? "operator_stopped",
});
return c.json(job);
} catch (err) {
return errorResponse(c, err);
}
});
app.get("/:job_id", (c) => {
const jobId = c.req.param("job_id");
const job = getJobsRepo(c).getById(jobId);
if (!job) return c.json({ error: "job_not_found", jobId }, 404);
return c.json(job);
});
return app;
}
+14
View File
@@ -63,6 +63,7 @@ import { streamRoutes } from "./routes/stream.js";
import { queueRoutes } from "./routes/queue.js";
import { projectsRoutes } from "./routes/projects.js";
import { viewsRoutes } from "./routes/views.js";
import { watchdogRoutes } from "./routes/watchdog.js";
import type { StreamStore } from "./domain/stream-store.js";
import type { QueueRepository } from "./domain/queue-repository.js";
import type { InboxHandler } from "./domain/inbox-handler.js";
@@ -70,6 +71,10 @@ import type { OutboxHandler } from "./domain/outbox-handler.js";
import type { ProjectClassifier } from "./domain/project-classifier.js";
import type { ClassifierLeaseManager } from "./domain/classifier-lease-manager.js";
import type { ViewProjector } from "./domain/view-projector.js";
import type { WatchdogJobsRepository } from "./domain/watchdog-jobs-repository.js";
import type { WatchdogHistoryLog } from "./domain/watchdog-history-log.js";
import type { WatchdogPolicyEngine } from "./domain/watchdog-policy-engine.js";
import type { WatchdogScheduler } from "./domain/watchdog-scheduler.js";
import { envRoutes } from "./routes/env.js";
import type { RigLifecycleService } from "./domain/rig-lifecycle-service.js";
import { seatRoutes } from "./routes/seat.js";
@@ -116,6 +121,10 @@ export interface AppDeps {
projectClassifier?: ProjectClassifier;
classifierLeaseManager?: ClassifierLeaseManager;
viewProjector?: ViewProjector;
watchdogJobsRepo?: WatchdogJobsRepository;
watchdogHistoryLog?: WatchdogHistoryLog;
watchdogPolicyEngine?: WatchdogPolicyEngine;
watchdogScheduler?: WatchdogScheduler;
specReviewService?: SpecReviewService;
specLibraryService?: SpecLibraryService;
whoamiService?: WhoamiService;
@@ -264,6 +273,10 @@ export function createApp(deps: AppDeps): Hono {
c.set("projectClassifier" as never, deps.projectClassifier);
c.set("classifierLeaseManager" as never, deps.classifierLeaseManager);
c.set("viewProjector" as never, deps.viewProjector);
c.set("watchdogJobsRepo" as never, deps.watchdogJobsRepo);
c.set("watchdogHistoryLog" as never, deps.watchdogHistoryLog);
c.set("watchdogPolicyEngine" as never, deps.watchdogPolicyEngine);
c.set("watchdogScheduler" as never, deps.watchdogScheduler);
c.set("specReviewService" as never, deps.specReviewService);
c.set("specLibraryService" as never, deps.specLibraryService);
c.set("whoamiService" as never, deps.whoamiService);
@@ -314,6 +327,7 @@ export function createApp(deps: AppDeps): Hono {
app.route("/api/queue", queueRoutes());
app.route("/api/projects", projectsRoutes());
app.route("/api/views", viewsRoutes());
app.route("/api/watchdog", watchdogRoutes());
app.route("/api/rigs/:rigId/env", envRoutes());
app.route("/api/restore-check", restoreCheckRoutes);
+43 -1
View File
@@ -67,6 +67,10 @@ import { ProjectClassifier } from "./domain/project-classifier.js";
import { ClassifierLeaseManager } from "./domain/classifier-lease-manager.js";
import { ViewProjector } from "./domain/view-projector.js";
import { wireViewEventBridge } from "./domain/view-event-bridge.js";
import { WatchdogJobsRepository } from "./domain/watchdog-jobs-repository.js";
import { WatchdogHistoryLog } from "./domain/watchdog-history-log.js";
import { WatchdogPolicyEngine } from "./domain/watchdog-policy-engine.js";
import { WatchdogScheduler } from "./domain/watchdog-scheduler.js";
import { SpecReviewService } from "./domain/spec-review-service.js";
import { SpecLibraryService } from "./domain/spec-library-service.js";
import { WhoamiService } from "./domain/whoami-service.js";
@@ -103,6 +107,8 @@ import { outboxEntriesSchema } from "./db/migrations/027_outbox_entries.js";
import { projectClassificationsSchema } from "./db/migrations/028_project_classifications.js";
import { classifierLeasesSchema } from "./db/migrations/029_classifier_leases.js";
import { viewsCustomSchema } from "./db/migrations/030_views_custom.js";
import { watchdogJobsSchema } from "./db/migrations/031_watchdog_jobs.js";
import { watchdogHistorySchema } from "./db/migrations/032_watchdog_history.js";
import { OPENRIG_HOME } from "./openrig-compat.js";
import {
getCompatibleOpenRigPath,
@@ -128,7 +134,7 @@ interface DaemonResult {
export async function createDaemon(opts?: DaemonOptions): Promise<DaemonResult> {
const dbPath = opts?.dbPath ?? ":memory:";
const db = createDb(dbPath);
migrate(db, [coreSchema, bindingsSessionsSchema, eventsSchema, snapshotsSchema, checkpointsSchema, resumeMetadataSchema, nodeSpecFieldsSchema, packagesSchema, installJournalSchema, journalSeqSchema, bootstrapSchema, discoverySchema, discoveryFkFix, agentspecRebootSchema, startupContextSchema, chatMessagesSchema, podNamespaceSchema, contextUsageSchema, externalCliAttachmentSchema, rigServicesSchema, seatHandoverObservabilitySchema, nodeCodexConfigProfileSchema, streamItemsSchema, queueItemsSchema, queueTransitionsSchema, inboxEntriesSchema, outboxEntriesSchema, projectClassificationsSchema, classifierLeasesSchema, viewsCustomSchema]);
migrate(db, [coreSchema, bindingsSessionsSchema, eventsSchema, snapshotsSchema, checkpointsSchema, resumeMetadataSchema, nodeSpecFieldsSchema, packagesSchema, installJournalSchema, journalSeqSchema, bootstrapSchema, discoverySchema, discoveryFkFix, agentspecRebootSchema, startupContextSchema, chatMessagesSchema, podNamespaceSchema, contextUsageSchema, externalCliAttachmentSchema, rigServicesSchema, seatHandoverObservabilitySchema, nodeCodexConfigProfileSchema, streamItemsSchema, queueItemsSchema, queueTransitionsSchema, inboxEntriesSchema, outboxEntriesSchema, projectClassificationsSchema, classifierLeasesSchema, viewsCustomSchema, watchdogJobsSchema, watchdogHistorySchema]);
const rigRepo = new RigRepository(db);
const sessionRegistry = new SessionRegistry(db);
@@ -163,6 +169,12 @@ export async function createDaemon(opts?: DaemonOptions): Promise<DaemonResult>
// when underlying state mutates.
wireViewEventBridge(eventBus, viewProjectorInstance);
// PL-004 Phase C — watchdog supervision tree. Repository + history-log
// are constructed early; the policy engine + scheduler are constructed
// after SessionTransport is available so the engine can wire delivery.
const watchdogJobsRepoInstance = new WatchdogJobsRepository(db);
const watchdogHistoryLogInstance = new WatchdogHistoryLog(db);
const tmuxAdapter = new TmuxAdapter(opts?.tmuxExec ?? execCommand);
// cmuxFactory takes precedence (for tests), then cmuxExec-based CLI transport, then default
const cmuxFactory = opts?.cmuxFactory
@@ -436,6 +448,8 @@ export async function createDaemon(opts?: DaemonOptions): Promise<DaemonResult>
classifierLeaseManager: classifierLeaseManagerInstance,
projectClassifier: new ProjectClassifier(db, eventBus, classifierLeaseManagerInstance),
viewProjector: viewProjectorInstance,
watchdogJobsRepo: watchdogJobsRepoInstance,
watchdogHistoryLog: watchdogHistoryLogInstance,
askService: (() => {
const psProjectionService = new PsProjectionService({ db });
const execDep = (cmd: string, args: string[]): Promise<{ stdout: string; exitCode: number }> =>
@@ -514,6 +528,34 @@ export async function createDaemon(opts?: DaemonOptions): Promise<DaemonResult>
}
} catch { /* best-effort — reference docs are not critical to daemon operation */ }
// PL-004 Phase C — watchdog policy engine + scheduler. Wired here
// (after deps construction) so the engine can dispatch deliveries
// through the live SessionTransport. Scheduler is started by
// index.ts after listen() so the daemon's HTTP surface is ready
// before the scheduler's first tick.
const sessionTransport = deps.sessionTransport;
if (sessionTransport) {
const watchdogPolicyEngine = new WatchdogPolicyEngine({
jobsRepo: watchdogJobsRepoInstance,
historyLog: watchdogHistoryLogInstance,
eventBus,
deliver: async ({ targetSession, message }) => {
try {
const result = await sessionTransport.send(targetSession, message);
return result.ok ? { status: "ok" } : { status: "failed", error: result.error };
} catch (err) {
return { status: "failed", error: err instanceof Error ? err.message : String(err) };
}
},
});
const watchdogScheduler = new WatchdogScheduler({
jobsRepo: watchdogJobsRepoInstance,
policyEngine: watchdogPolicyEngine,
});
deps.watchdogPolicyEngine = watchdogPolicyEngine;
deps.watchdogScheduler = watchdogScheduler;
}
// Context monitor — constructed before createApp so routes can access pollOnce for refresh.
// Caller (index.ts) starts polling after listen.
const { ContextMonitor } = await import("./domain/context-monitor.js");
@@ -0,0 +1,121 @@
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import type Database from "better-sqlite3";
import { createDb } from "../src/db/connection.js";
import { migrate } from "../src/db/migrate.js";
import { coreSchema } from "../src/db/migrations/001_core_schema.js";
import { eventsSchema } from "../src/db/migrations/003_events.js";
import { watchdogJobsSchema } from "../src/db/migrations/031_watchdog_jobs.js";
import { watchdogHistorySchema } from "../src/db/migrations/032_watchdog_history.js";
import { WatchdogJobsRepository } from "../src/domain/watchdog-jobs-repository.js";
import { WatchdogHistoryLog } from "../src/domain/watchdog-history-log.js";
describe("WatchdogHistoryLog (PL-004 Phase C; append-only audit)", () => {
let db: Database.Database;
let jobsRepo: WatchdogJobsRepository;
let log: WatchdogHistoryLog;
let jobId: string;
beforeEach(() => {
db = createDb();
migrate(db, [coreSchema, eventsSchema, watchdogJobsSchema, watchdogHistorySchema]);
jobsRepo = new WatchdogJobsRepository(db);
log = new WatchdogHistoryLog(db);
jobId = jobsRepo.register({
policy: "periodic-reminder",
specYaml: "policy: periodic-reminder",
targetSession: "a@rig",
intervalSeconds: 60,
registeredBySession: "ops@kernel",
}).jobId;
});
afterEach(() => db.close());
it("record(sent) persists delivery target/status/message + ULID id", () => {
const e = log.record({
jobId,
evaluatedAt: "2026-05-03T07:00:00.000Z",
outcome: "sent",
deliveryTargetSession: "a@rig",
deliveryStatus: "ok",
deliveryMessage: "hello",
});
expect(e.historyId).toMatch(/^[0-9A-Z]{26}$/);
expect(e.outcome).toBe("sent");
expect(e.deliveryTargetSession).toBe("a@rig");
expect(e.deliveryStatus).toBe("ok");
expect(e.deliveryMessage).toBe("hello");
});
it("record(skipped) persists reason + null delivery fields", () => {
const e = log.record({
jobId,
evaluatedAt: "2026-05-03T07:00:00.000Z",
outcome: "skipped",
skipReason: "no_actionable_artifacts",
});
expect(e.outcome).toBe("skipped");
expect(e.skipReason).toBe("no_actionable_artifacts");
expect(e.deliveryTargetSession).toBeNull();
});
it("record(terminal) persists reason", () => {
const e = log.record({
jobId,
evaluatedAt: "2026-05-03T07:00:00.000Z",
outcome: "terminal",
skipReason: "policy_done",
});
expect(e.outcome).toBe("terminal");
expect(e.skipReason).toBe("policy_done");
});
it("record JSON-encodes evaluation_notes and decodes on read", () => {
log.record({
jobId,
evaluatedAt: "2026-05-03T07:00:00.000Z",
outcome: "sent",
deliveryTargetSession: "a@rig",
deliveryStatus: "ok",
deliveryMessage: "x",
evaluationNotes: { artifact_count: 7, label: "foo" },
});
const list = log.listForJob(jobId);
expect(list[0]?.evaluationNotes).toEqual({ artifact_count: 7, label: "foo" });
});
it("listForJob returns DESC by evaluated_at", () => {
log.record({ jobId, evaluatedAt: "2026-05-03T07:00:00.000Z", outcome: "sent", deliveryTargetSession: "a@rig", deliveryStatus: "ok", deliveryMessage: "first" });
log.record({ jobId, evaluatedAt: "2026-05-03T07:01:00.000Z", outcome: "sent", deliveryTargetSession: "a@rig", deliveryStatus: "ok", deliveryMessage: "second" });
const list = log.listForJob(jobId);
expect(list[0]?.deliveryMessage).toBe("second");
expect(list[1]?.deliveryMessage).toBe("first");
});
it("countForJob returns the number of recorded entries", () => {
expect(log.countForJob(jobId)).toBe(0);
log.record({ jobId, evaluatedAt: "2026-05-03T07:00:00.000Z", outcome: "skipped", skipReason: "x" });
log.record({ jobId, evaluatedAt: "2026-05-03T07:00:00.000Z", outcome: "skipped", skipReason: "x" });
expect(log.countForJob(jobId)).toBe(2);
});
it("FK violation: record() against unknown job_id throws SQLite FK error", () => {
expect(() =>
log.record({
jobId: "unknown-job-id",
evaluatedAt: "2026-05-03T07:00:00.000Z",
outcome: "skipped",
skipReason: "x",
}),
).toThrow();
});
it("API surface does NOT expose update or delete (append-only contract)", () => {
const proto = Object.getPrototypeOf(log) as Record<string, unknown>;
const names = Object.getOwnPropertyNames(proto);
expect(names).not.toContain("update");
expect(names).not.toContain("delete");
expect(names).not.toContain("remove");
expect(names).not.toContain("modify");
});
});
@@ -0,0 +1,150 @@
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import type Database from "better-sqlite3";
import { createDb } from "../src/db/connection.js";
import { migrate } from "../src/db/migrate.js";
import { coreSchema } from "../src/db/migrations/001_core_schema.js";
import { eventsSchema } from "../src/db/migrations/003_events.js";
import { watchdogJobsSchema } from "../src/db/migrations/031_watchdog_jobs.js";
import { watchdogHistorySchema } from "../src/db/migrations/032_watchdog_history.js";
import {
PHASE_C_POLICIES,
WatchdogJobsError,
WatchdogJobsRepository,
} from "../src/domain/watchdog-jobs-repository.js";
describe("WatchdogJobsRepository (PL-004 Phase C)", () => {
let db: Database.Database;
let repo: WatchdogJobsRepository;
beforeEach(() => {
db = createDb();
migrate(db, [coreSchema, eventsSchema, watchdogJobsSchema, watchdogHistorySchema]);
repo = new WatchdogJobsRepository(db);
});
afterEach(() => db.close());
function validInput(overrides: Record<string, unknown> = {}) {
return {
policy: "periodic-reminder",
specYaml: "policy: periodic-reminder\ntarget: a@rig\ninterval_seconds: 60\ncontext:\n target:\n session: a@rig\n message: hello\n",
targetSession: "a@rig",
intervalSeconds: 60,
registeredBySession: "ops@kernel",
...overrides,
};
}
it("register stores all three v1 policies", () => {
for (const p of PHASE_C_POLICIES) {
const job = repo.register(validInput({ policy: p }));
expect(job.policy).toBe(p);
expect(job.state).toBe("active");
expect(job.jobId).toMatch(/^[0-9A-Z]{26}$/);
}
});
it("register rejects workflow-keepalive with policy_deferred_to_phase_d", () => {
try {
repo.register(validInput({ policy: "workflow-keepalive" }));
throw new Error("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(WatchdogJobsError);
expect((err as WatchdogJobsError).code).toBe("policy_deferred_to_phase_d");
}
});
it("register rejects unknown policy with policy_unknown", () => {
try {
repo.register(validInput({ policy: "totally-bogus" }));
throw new Error("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(WatchdogJobsError);
expect((err as WatchdogJobsError).code).toBe("policy_unknown");
}
});
it("register rejects non-positive interval_seconds with interval_invalid", () => {
for (const bad of [0, -1, 1.5]) {
try {
repo.register(validInput({ intervalSeconds: bad }));
throw new Error(`should have thrown for ${bad}`);
} catch (err) {
expect(err).toBeInstanceOf(WatchdogJobsError);
expect((err as WatchdogJobsError).code).toBe("interval_invalid");
}
}
});
it("register rejects target_session without @ as target_session_invalid", () => {
try {
repo.register(validInput({ targetSession: "no-at-here" }));
throw new Error("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(WatchdogJobsError);
expect((err as WatchdogJobsError).code).toBe("target_session_invalid");
}
});
it("listActive returns only state=active jobs in registration order", () => {
const a = repo.register(validInput({ targetSession: "a@rig" }));
const b = repo.register(validInput({ targetSession: "b@rig" }));
const c = repo.register(validInput({ targetSession: "c@rig" }));
repo.stop(b.jobId);
const active = repo.listActive();
expect(active.map((j) => j.jobId)).toEqual([a.jobId, c.jobId]);
});
it("recordEvaluation(fired=true) updates last_evaluation_at + last_fire_at", () => {
const job = repo.register(validInput());
repo.recordEvaluation(job.jobId, "2026-05-03T07:00:00.000Z", true);
const after = repo.getByIdOrThrow(job.jobId);
expect(after.lastEvaluationAt).toBe("2026-05-03T07:00:00.000Z");
expect(after.lastFireAt).toBe("2026-05-03T07:00:00.000Z");
});
it("recordEvaluation(fired=false) updates only last_evaluation_at", () => {
const job = repo.register(validInput());
repo.recordEvaluation(job.jobId, "2026-05-03T07:00:00.000Z", false);
const after = repo.getByIdOrThrow(job.jobId);
expect(after.lastEvaluationAt).toBe("2026-05-03T07:00:00.000Z");
expect(after.lastFireAt).toBeNull();
});
it("markTerminal sets state=terminal + terminal_reason", () => {
const job = repo.register(validInput());
repo.markTerminal(job.jobId, "policy_returned_terminal");
const after = repo.getByIdOrThrow(job.jobId);
expect(after.state).toBe("terminal");
expect(after.terminalReason).toBe("policy_returned_terminal");
});
it("stop sets state=stopped + records reason", () => {
const job = repo.register(validInput());
const stopped = repo.stop(job.jobId, "operator stop reason");
expect(stopped.state).toBe("stopped");
expect(stopped.terminalReason).toBe("operator stop reason");
});
it("stop on already-terminal job throws job_terminal", () => {
const job = repo.register(validInput());
repo.markTerminal(job.jobId, "done");
try {
repo.stop(job.jobId);
throw new Error("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(WatchdogJobsError);
expect((err as WatchdogJobsError).code).toBe("job_terminal");
}
});
it("getByIdOrThrow throws job_not_found for unknown id", () => {
try {
repo.getByIdOrThrow("does-not-exist");
throw new Error("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(WatchdogJobsError);
expect((err as WatchdogJobsError).code).toBe("job_not_found");
}
});
});
@@ -0,0 +1,240 @@
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { mkdirSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { artifactPoolReadyPolicy } from "../src/domain/policies/artifact-pool-ready.js";
import { edgeArtifactRequiredPolicy } from "../src/domain/policies/edge-artifact-required.js";
import { periodicReminderPolicy } from "../src/domain/policies/periodic-reminder.js";
import type { PolicyJob } from "../src/domain/policies/types.js";
function makeJob(overrides: Partial<PolicyJob> & { context: Record<string, unknown> }): PolicyJob {
return {
jobId: "job-1",
policy: "periodic-reminder",
targetSession: "a@rig",
intervalSeconds: 60,
activeWakeIntervalSeconds: null,
scanIntervalSeconds: null,
lastEvaluationAt: null,
lastFireAt: null,
registeredBySession: "ops@kernel",
registeredAt: "2026-05-03T07:00:00.000Z",
...overrides,
};
}
describe("periodicReminderPolicy", () => {
it("returns send with explicit target+message from context", async () => {
const out = await periodicReminderPolicy.evaluate(
makeJob({
context: { target: { session: "alice@rig" }, message: "ping" },
}),
);
expect(out).toEqual({ action: "send", target: "alice@rig", message: "ping" });
});
it("throws policy_spec_invalid when context.target.session is missing", async () => {
try {
await periodicReminderPolicy.evaluate(makeJob({ context: { message: "x" } }));
throw new Error("should have thrown");
} catch (err) {
expect((err as Error & { code: string }).code).toBe("policy_spec_invalid");
}
});
it("throws policy_spec_invalid when context.message is missing", async () => {
try {
await periodicReminderPolicy.evaluate(makeJob({ context: { target: { session: "a@rig" } } }));
throw new Error("should have thrown");
} catch (err) {
expect((err as Error & { code: string }).code).toBe("policy_spec_invalid");
}
});
});
describe("artifactPoolReadyPolicy", () => {
let tmp: string;
beforeEach(() => {
tmp = join(tmpdir(), `watchdog-pool-${Date.now()}-${Math.random()}`);
mkdirSync(tmp, { recursive: true });
});
afterEach(() => rmSync(tmp, { recursive: true, force: true }));
it("skip with reason no_actionable_artifacts when pool empty", async () => {
const out = await artifactPoolReadyPolicy.evaluate(
makeJob({
context: { target: { session: "a@rig" }, pools: { path: tmp } },
}),
);
expect(out).toEqual({ action: "skip", reason: "no_actionable_artifacts" });
});
it("send with formatted message when pool has actionable artifacts", async () => {
writeFileSync(join(tmp, "a.md"), "---\nstatus: ready\n---\nbody-a\n");
writeFileSync(join(tmp, "b.md"), "---\nstatus: ready\n---\nbody-b\n");
const out = await artifactPoolReadyPolicy.evaluate(
makeJob({
context: {
target: { session: "a@rig" },
pools: { path: tmp, include_statuses: ["ready"] },
label: "things",
},
}),
);
expect(out.action).toBe("send");
if (out.action !== "send") return;
expect(out.target).toBe("a@rig");
expect(out.message).toMatch(/things has 2 actionable artifact/);
expect(out.message).toContain("a.md");
expect(out.message).toContain("b.md");
});
it("respects include_statuses filter (artifact with non-matching status is excluded)", async () => {
writeFileSync(join(tmp, "a.md"), "---\nstatus: ready\n---\nbody-a\n");
writeFileSync(join(tmp, "b.md"), "---\nstatus: draft\n---\nbody-b\n");
const out = await artifactPoolReadyPolicy.evaluate(
makeJob({
context: {
target: { session: "a@rig" },
pools: { path: tmp, include_statuses: ["ready"] },
},
}),
);
expect(out.action).toBe("send");
if (out.action !== "send") return;
expect(out.message).toMatch(/has 1 actionable/);
expect(out.message).toContain("a.md");
expect(out.message).not.toContain("b.md");
});
it("respects max_items cap in formatted bullet list", async () => {
for (const c of ["a", "b", "c", "d", "e", "f", "g"]) {
writeFileSync(join(tmp, `${c}.md`), "---\nstatus: ready\n---\n");
}
const out = await artifactPoolReadyPolicy.evaluate(
makeJob({
context: {
target: { session: "a@rig" },
pools: { path: tmp, include_statuses: ["ready"] },
max_items: 3,
},
}),
);
expect(out.action).toBe("send");
if (out.action !== "send") return;
expect(out.message).toMatch(/has 7 actionable/);
const bulletCount = out.message.split("\n").filter((l) => l.startsWith("- ")).length;
expect(bulletCount).toBe(3);
});
it("missing pool directory yields skip (no actionable)", async () => {
rmSync(tmp, { recursive: true, force: true });
const out = await artifactPoolReadyPolicy.evaluate(
makeJob({
context: { target: { session: "a@rig" }, pools: { path: tmp } },
}),
);
expect(out).toEqual({ action: "skip", reason: "no_actionable_artifacts" });
});
});
describe("edgeArtifactRequiredPolicy", () => {
let src: string;
let tgt: string;
beforeEach(() => {
src = join(tmpdir(), `wd-edge-src-${Date.now()}-${Math.random()}`);
tgt = join(tmpdir(), `wd-edge-tgt-${Date.now()}-${Math.random()}`);
mkdirSync(src, { recursive: true });
mkdirSync(tgt, { recursive: true });
});
afterEach(() => {
rmSync(src, { recursive: true, force: true });
rmSync(tgt, { recursive: true, force: true });
});
it("skip when every source key has matching target", async () => {
writeFileSync(join(src, "x.md"), "---\nentry: x\nstatus: ready\n---\n");
writeFileSync(join(tgt, "x-downstream.md"), "---\nentry: x\nstatus: anything\n---\n");
const out = await edgeArtifactRequiredPolicy.evaluate(
makeJob({
context: {
target: { session: "a@rig" },
source_pools: { path: src, include_statuses: ["ready"] },
target_pools: { path: tgt },
},
}),
);
expect(out).toEqual({ action: "skip", reason: "no_missing_edge_artifacts" });
});
it("send with missing-edge list when sources lack matching targets", async () => {
writeFileSync(join(src, "x.md"), "---\nentry: x\nstatus: ready\n---\n");
writeFileSync(join(src, "y.md"), "---\nentry: y\nstatus: ready\n---\n");
writeFileSync(join(tgt, "x-downstream.md"), "---\nentry: x\n---\n");
const out = await edgeArtifactRequiredPolicy.evaluate(
makeJob({
context: {
target: { session: "a@rig" },
source_pools: { path: src, include_statuses: ["ready"] },
target_pools: { path: tgt },
label: "myedge",
},
}),
);
expect(out.action).toBe("send");
if (out.action !== "send") return;
expect(out.target).toBe("a@rig");
expect(out.message).toMatch(/myedge has 1 upstream artifact/);
expect(out.message).toContain("y.md");
expect(out.message).not.toContain("x.md");
});
it("source key falls back to basename-sans-md when frontmatter[key_field] absent", async () => {
writeFileSync(join(src, "no-frontmatter-key.md"), "---\nstatus: ready\n---\n");
const out = await edgeArtifactRequiredPolicy.evaluate(
makeJob({
context: {
target: { session: "a@rig" },
source_pools: { path: src, include_statuses: ["ready"] },
target_pools: { path: tgt },
},
}),
);
expect(out.action).toBe("send");
if (out.action !== "send") return;
expect(out.message).toContain("no-frontmatter-key.md");
});
it("target match honors target_key_field override", async () => {
writeFileSync(join(src, "x.md"), "---\nentry: foo\nstatus: ready\n---\n");
writeFileSync(join(tgt, "x.md"), "---\nlinks_to: foo\n---\n");
const out = await edgeArtifactRequiredPolicy.evaluate(
makeJob({
context: {
target: { session: "a@rig" },
source_pools: { path: src, include_statuses: ["ready"] },
target_pools: { path: tgt },
source_key_field: "entry",
target_key_field: "links_to",
},
}),
);
expect(out).toEqual({ action: "skip", reason: "no_missing_edge_artifacts" });
});
it("target scan ignores include_statuses (matches POC override)", async () => {
writeFileSync(join(src, "x.md"), "---\nentry: x\nstatus: ready\n---\n");
// Target has matching key but a non-listed status; match should still count.
writeFileSync(join(tgt, "x-down.md"), "---\nentry: x\nstatus: rejected\n---\n");
const out = await edgeArtifactRequiredPolicy.evaluate(
makeJob({
context: {
target: { session: "a@rig" },
source_pools: { path: src, include_statuses: ["ready"] },
target_pools: { path: tgt, include_statuses: ["accepted"] },
},
}),
);
expect(out).toEqual({ action: "skip", reason: "no_missing_edge_artifacts" });
});
});
@@ -0,0 +1,160 @@
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import type Database from "better-sqlite3";
import { createDb } from "../src/db/connection.js";
import { migrate } from "../src/db/migrate.js";
import { coreSchema } from "../src/db/migrations/001_core_schema.js";
import { eventsSchema } from "../src/db/migrations/003_events.js";
import { watchdogJobsSchema } from "../src/db/migrations/031_watchdog_jobs.js";
import { watchdogHistorySchema } from "../src/db/migrations/032_watchdog_history.js";
import { EventBus } from "../src/domain/event-bus.js";
import { WatchdogJobsRepository } from "../src/domain/watchdog-jobs-repository.js";
import { WatchdogHistoryLog } from "../src/domain/watchdog-history-log.js";
import {
type DeliveryFn,
WatchdogPolicyEngine,
} from "../src/domain/watchdog-policy-engine.js";
import type { PersistedEvent } from "../src/domain/types.js";
describe("WatchdogPolicyEngine (PL-004 Phase C)", () => {
let db: Database.Database;
let bus: EventBus;
let jobsRepo: WatchdogJobsRepository;
let log: WatchdogHistoryLog;
let captured: PersistedEvent[];
let deliveryCalls: Array<{ targetSession: string; message: string }>;
let deliver: DeliveryFn;
function makeEngine(deliverOverride?: DeliveryFn): WatchdogPolicyEngine {
return new WatchdogPolicyEngine({
jobsRepo,
historyLog: log,
eventBus: bus,
deliver: deliverOverride ?? deliver,
});
}
beforeEach(() => {
db = createDb();
migrate(db, [coreSchema, eventsSchema, watchdogJobsSchema, watchdogHistorySchema]);
bus = new EventBus(db);
jobsRepo = new WatchdogJobsRepository(db);
log = new WatchdogHistoryLog(db);
captured = [];
bus.subscribe((e) => captured.push(e));
deliveryCalls = [];
deliver = async (req) => {
deliveryCalls.push(req);
return { status: "ok" };
};
});
afterEach(() => db.close());
it("evaluate(periodic-reminder) routes through delivery + records sent + emits evaluation_fired", async () => {
const engine = makeEngine();
const job = jobsRepo.register({
policy: "periodic-reminder",
specYaml: "policy: periodic-reminder\ntarget: alice@rig\ninterval_seconds: 60\ncontext:\n target:\n session: alice@rig\n message: ping\n",
targetSession: "alice@rig",
intervalSeconds: 60,
registeredBySession: "ops@kernel",
});
const result = await engine.evaluate(job);
expect(result.outcome.action).toBe("send");
expect(deliveryCalls).toEqual([{ targetSession: "alice@rig", message: "ping" }]);
expect(log.countForJob(job.jobId)).toBe(1);
const list = log.listForJob(job.jobId);
expect(list[0]?.outcome).toBe("sent");
expect(list[0]?.deliveryStatus).toBe("ok");
expect(captured.some((e) => e.type === "watchdog.evaluation_fired")).toBe(true);
const after = jobsRepo.getByIdOrThrow(job.jobId);
expect(after.lastEvaluationAt).not.toBeNull();
expect(after.lastFireAt).not.toBeNull();
});
it("evaluate records skipped + emits evaluation_skipped + does NOT touch last_fire_at", async () => {
const engine = makeEngine(async () => {
throw new Error("delivery should NOT be called for skip");
});
// artifact-pool-ready against an empty pool returns skip(no_actionable_artifacts).
const job = jobsRepo.register({
policy: "artifact-pool-ready",
specYaml:
"policy: artifact-pool-ready\ntarget: alice@rig\ninterval_seconds: 60\ncontext:\n target:\n session: alice@rig\n pools:\n path: /nonexistent-pool-dir-for-test\n",
targetSession: "alice@rig",
intervalSeconds: 60,
registeredBySession: "ops@kernel",
});
const result = await engine.evaluate(job);
expect(result.outcome.action).toBe("skip");
if (result.outcome.action !== "skip") return;
expect(result.outcome.reason).toBe("no_actionable_artifacts");
expect(deliveryCalls).toEqual([]);
expect(log.countForJob(job.jobId)).toBe(1);
expect(log.listForJob(job.jobId)[0]?.outcome).toBe("skipped");
expect(captured.some((e) => e.type === "watchdog.evaluation_skipped")).toBe(true);
const after = jobsRepo.getByIdOrThrow(job.jobId);
expect(after.lastEvaluationAt).not.toBeNull();
expect(after.lastFireAt).toBeNull();
});
it("delivery failure is recorded as sent outcome with delivery_status=failed", async () => {
const failingDeliver: DeliveryFn = async () => ({ status: "failed", error: "transport denied" });
const engine = makeEngine(failingDeliver);
const job = jobsRepo.register({
policy: "periodic-reminder",
specYaml:
"policy: periodic-reminder\ntarget: alice@rig\ninterval_seconds: 60\ncontext:\n target:\n session: alice@rig\n message: ping\n",
targetSession: "alice@rig",
intervalSeconds: 60,
registeredBySession: "ops@kernel",
});
await engine.evaluate(job);
const list = log.listForJob(job.jobId);
expect(list[0]?.outcome).toBe("sent");
expect(list[0]?.deliveryStatus).toBe("failed");
});
it("unknown policy at evaluate-time marks job terminal + emits evaluation_terminal", async () => {
const engine = makeEngine();
const job = jobsRepo.register({
policy: "periodic-reminder",
specYaml: "policy: periodic-reminder\ncontext:\n target:\n session: a@rig\n message: x\n",
targetSession: "a@rig",
intervalSeconds: 60,
registeredBySession: "ops@kernel",
});
// Hand-craft a job carrying an unknown policy name. The engine should
// treat this defensively: terminal + history record + event.
const synthetic = { ...job, policy: "totally-bogus" as never };
const result = await engine.evaluate(synthetic);
expect(result.outcome.action).toBe("terminal");
const after = jobsRepo.getByIdOrThrow(job.jobId);
expect(after.state).toBe("terminal");
expect(captured.some((e) => e.type === "watchdog.evaluation_terminal")).toBe(true);
});
it("policy registry resolves the three v1 policies", () => {
const engine = makeEngine();
expect(engine.resolvePolicy("periodic-reminder")?.name).toBe("periodic-reminder");
expect(engine.resolvePolicy("artifact-pool-ready")?.name).toBe("artifact-pool-ready");
expect(engine.resolvePolicy("edge-artifact-required")?.name).toBe("edge-artifact-required");
expect(engine.resolvePolicy("workflow-keepalive")).toBeUndefined();
});
it("default YAML parser extracts nested context block", async () => {
const engine = makeEngine();
const job = jobsRepo.register({
policy: "periodic-reminder",
specYaml:
"policy: periodic-reminder\ntarget: a@rig\ninterval_seconds: 60\ncontext:\n target:\n session: parsed-session@rig\n message: parsed-message\n",
targetSession: "a@rig",
intervalSeconds: 60,
registeredBySession: "ops@kernel",
});
await engine.evaluate(job);
expect(deliveryCalls).toEqual([
{ targetSession: "parsed-session@rig", message: "parsed-message" },
]);
});
});
@@ -0,0 +1,204 @@
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { Hono } from "hono";
import type Database from "better-sqlite3";
import { createDb } from "../src/db/connection.js";
import { migrate } from "../src/db/migrate.js";
import { coreSchema } from "../src/db/migrations/001_core_schema.js";
import { eventsSchema } from "../src/db/migrations/003_events.js";
import { watchdogJobsSchema } from "../src/db/migrations/031_watchdog_jobs.js";
import { watchdogHistorySchema } from "../src/db/migrations/032_watchdog_history.js";
import { EventBus } from "../src/domain/event-bus.js";
import { WatchdogJobsRepository } from "../src/domain/watchdog-jobs-repository.js";
import { WatchdogHistoryLog } from "../src/domain/watchdog-history-log.js";
import { watchdogRoutes } from "../src/routes/watchdog.js";
function buildApp(opts: {
eventBus: EventBus;
jobsRepo: WatchdogJobsRepository;
historyLog: WatchdogHistoryLog;
}): Hono {
const app = new Hono();
app.use("*", async (c, next) => {
c.set("eventBus" as never, opts.eventBus);
c.set("watchdogJobsRepo" as never, opts.jobsRepo);
c.set("watchdogHistoryLog" as never, opts.historyLog);
await next();
});
app.route("/api/watchdog", watchdogRoutes());
return app;
}
describe("watchdog routes (PL-004 Phase C)", () => {
let db: Database.Database;
let bus: EventBus;
let jobsRepo: WatchdogJobsRepository;
let log: WatchdogHistoryLog;
let app: Hono;
const validRegisterBody = {
policy: "periodic-reminder",
specYaml:
"policy: periodic-reminder\ntarget: alice@rig\ninterval_seconds: 60\ncontext:\n target:\n session: alice@rig\n message: ping\n",
targetSession: "alice@rig",
intervalSeconds: 60,
registeredBySession: "ops@kernel",
};
beforeEach(() => {
db = createDb();
migrate(db, [coreSchema, eventsSchema, watchdogJobsSchema, watchdogHistorySchema]);
bus = new EventBus(db);
jobsRepo = new WatchdogJobsRepository(db);
log = new WatchdogHistoryLog(db);
app = buildApp({ eventBus: bus, jobsRepo, historyLog: log });
});
afterEach(() => db.close());
it("POST /register returns 201 + persists job + emits watchdog.job_registered", async () => {
const captured: Array<{ type: string }> = [];
bus.subscribe((e) => captured.push(e));
const res = await app.request("/api/watchdog/register", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(validRegisterBody),
});
expect(res.status).toBe(201);
const body = (await res.json()) as { jobId: string; policy: string; state: string };
expect(body.jobId).toMatch(/^[0-9A-Z]{26}$/);
expect(body.policy).toBe("periodic-reminder");
expect(body.state).toBe("active");
expect(captured.some((e) => e.type === "watchdog.job_registered")).toBe(true);
});
it("POST /register rejects workflow-keepalive with 400 + policy_deferred_to_phase_d", async () => {
const res = await app.request("/api/watchdog/register", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ ...validRegisterBody, policy: "workflow-keepalive" }),
});
expect(res.status).toBe(400);
const body = (await res.json()) as { error: string };
expect(body.error).toBe("policy_deferred_to_phase_d");
});
it("POST /register rejects unknown policy with 400 + policy_unknown", async () => {
const res = await app.request("/api/watchdog/register", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ ...validRegisterBody, policy: "totally-bogus" }),
});
expect(res.status).toBe(400);
const body = (await res.json()) as { error: string };
expect(body.error).toBe("policy_unknown");
});
it("POST /register rejects missing required field with 400", async () => {
const res = await app.request("/api/watchdog/register", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ policy: "periodic-reminder" }),
});
expect(res.status).toBe(400);
});
it("GET /list lists all jobs", async () => {
jobsRepo.register({ ...validRegisterBody, targetSession: "a@rig" });
jobsRepo.register({ ...validRegisterBody, targetSession: "b@rig" });
const res = await app.request("/api/watchdog/list");
expect(res.status).toBe(200);
const list = (await res.json()) as Array<{ targetSession: string }>;
expect(list).toHaveLength(2);
});
it("GET /:job_id returns job by id", async () => {
const job = jobsRepo.register(validRegisterBody);
const res = await app.request(`/api/watchdog/${job.jobId}`);
expect(res.status).toBe(200);
const body = (await res.json()) as { jobId: string };
expect(body.jobId).toBe(job.jobId);
});
it("GET /:job_id returns 404 for unknown id", async () => {
const res = await app.request("/api/watchdog/unknown-id");
expect(res.status).toBe(404);
const body = (await res.json()) as { error: string };
expect(body.error).toBe("job_not_found");
});
it("GET /:job_id/status returns job + recent history", async () => {
const job = jobsRepo.register(validRegisterBody);
log.record({
jobId: job.jobId,
evaluatedAt: "2026-05-03T07:00:00.000Z",
outcome: "sent",
deliveryTargetSession: "alice@rig",
deliveryStatus: "ok",
deliveryMessage: "ping",
});
const res = await app.request(`/api/watchdog/${job.jobId}/status`);
expect(res.status).toBe(200);
const body = (await res.json()) as { job: { jobId: string }; recentHistory: Array<{ outcome: string }> };
expect(body.job.jobId).toBe(job.jobId);
expect(body.recentHistory).toHaveLength(1);
expect(body.recentHistory[0]?.outcome).toBe("sent");
});
it("POST /:job_id/stop stops job + emits watchdog.job_stopped", async () => {
const captured: Array<{ type: string }> = [];
bus.subscribe((e) => captured.push(e));
const job = jobsRepo.register(validRegisterBody);
const res = await app.request(`/api/watchdog/${job.jobId}/stop`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ reason: "tester stopped" }),
});
expect(res.status).toBe(200);
const body = (await res.json()) as { state: string };
expect(body.state).toBe("stopped");
expect(captured.some((e) => e.type === "watchdog.job_stopped")).toBe(true);
});
it("POST /:job_id/stop on terminal job returns 409 + job_terminal", async () => {
const job = jobsRepo.register(validRegisterBody);
jobsRepo.markTerminal(job.jobId, "done");
const res = await app.request(`/api/watchdog/${job.jobId}/stop`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({}),
});
expect(res.status).toBe(409);
const body = (await res.json()) as { error: string };
expect(body.error).toBe("job_terminal");
});
it("R1 SSE pattern: GET /api/watchdog/sse returns 200 + content-type text/event-stream", async () => {
const res = await app.request("/api/watchdog/sse");
try {
expect(res.status).toBe(200);
expect(res.headers.get("content-type") ?? "").toContain("text/event-stream");
} finally {
await res.body?.cancel();
}
});
it("R1 SSE pattern: GET /api/watchdog/watch returns 200 + content-type text/event-stream", async () => {
const res = await app.request("/api/watchdog/watch");
try {
expect(res.status).toBe(200);
expect(res.headers.get("content-type") ?? "").toContain("text/event-stream");
} finally {
await res.body?.cancel();
}
});
it("R1 SSE pattern: GET /api/watchdog/sse does NOT return job_not_found (route-order regression guard)", async () => {
const res = await app.request("/api/watchdog/sse");
try {
expect(res.status).not.toBe(404);
expect(res.headers.get("content-type") ?? "").not.toContain("application/json");
} finally {
await res.body?.cancel();
}
});
});
@@ -0,0 +1,221 @@
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import type Database from "better-sqlite3";
import { createDb } from "../src/db/connection.js";
import { migrate } from "../src/db/migrate.js";
import { coreSchema } from "../src/db/migrations/001_core_schema.js";
import { eventsSchema } from "../src/db/migrations/003_events.js";
import { watchdogJobsSchema } from "../src/db/migrations/031_watchdog_jobs.js";
import { watchdogHistorySchema } from "../src/db/migrations/032_watchdog_history.js";
import { EventBus } from "../src/domain/event-bus.js";
import { WatchdogJobsRepository } from "../src/domain/watchdog-jobs-repository.js";
import { WatchdogHistoryLog } from "../src/domain/watchdog-history-log.js";
import { WatchdogPolicyEngine } from "../src/domain/watchdog-policy-engine.js";
import { WatchdogScheduler, isDue } from "../src/domain/watchdog-scheduler.js";
describe("WatchdogScheduler (PL-004 Phase C)", () => {
let db: Database.Database;
let bus: EventBus;
let jobsRepo: WatchdogJobsRepository;
let log: WatchdogHistoryLog;
let deliveries: Array<{ targetSession: string; message: string }>;
beforeEach(() => {
db = createDb();
migrate(db, [coreSchema, eventsSchema, watchdogJobsSchema, watchdogHistorySchema]);
bus = new EventBus(db);
jobsRepo = new WatchdogJobsRepository(db);
log = new WatchdogHistoryLog(db);
deliveries = [];
});
afterEach(() => db.close());
function makeEngine() {
return new WatchdogPolicyEngine({
jobsRepo,
historyLog: log,
eventBus: bus,
deliver: async (req) => {
deliveries.push(req);
return { status: "ok" };
},
});
}
it("isDue returns true for never-evaluated jobs", () => {
const job = jobsRepo.register({
policy: "periodic-reminder",
specYaml: "context:\n target:\n session: a@rig\n message: x\n",
targetSession: "a@rig",
intervalSeconds: 30,
registeredBySession: "ops@kernel",
});
expect(isDue(job, Date.now())).toBe(true);
});
it("isDue returns false when interval not yet elapsed", () => {
const job = jobsRepo.register({
policy: "periodic-reminder",
specYaml: "context:\n target:\n session: a@rig\n message: x\n",
targetSession: "a@rig",
intervalSeconds: 30,
registeredBySession: "ops@kernel",
});
const tenSecondsAgo = new Date(Date.now() - 10_000).toISOString();
jobsRepo.recordEvaluation(job.jobId, tenSecondsAgo, true);
const updated = jobsRepo.getByIdOrThrow(job.jobId);
expect(isDue(updated, Date.now())).toBe(false);
});
it("isDue returns true when interval elapsed", () => {
const job = jobsRepo.register({
policy: "periodic-reminder",
specYaml: "context:\n target:\n session: a@rig\n message: x\n",
targetSession: "a@rig",
intervalSeconds: 30,
registeredBySession: "ops@kernel",
});
const longAgo = new Date(Date.now() - 60_000).toISOString();
jobsRepo.recordEvaluation(job.jobId, longAgo, true);
const updated = jobsRepo.getByIdOrThrow(job.jobId);
expect(isDue(updated, Date.now())).toBe(true);
});
it("runTickNow only evaluates due active jobs", async () => {
const j1 = jobsRepo.register({
policy: "periodic-reminder",
specYaml: "context:\n target:\n session: a@rig\n message: m1\n",
targetSession: "a@rig",
intervalSeconds: 30,
registeredBySession: "ops@kernel",
});
const j2 = jobsRepo.register({
policy: "periodic-reminder",
specYaml: "context:\n target:\n session: b@rig\n message: m2\n",
targetSession: "b@rig",
intervalSeconds: 30,
registeredBySession: "ops@kernel",
});
jobsRepo.recordEvaluation(j1.jobId, new Date(Date.now() - 5_000).toISOString(), true);
const engine = makeEngine();
const sched = new WatchdogScheduler({ jobsRepo, policyEngine: engine });
await sched.runTickNow();
expect(deliveries).toEqual([{ targetSession: "b@rig", message: "m2" }]);
});
it("runTickNow evaluates a due job at every subsequent due tick", async () => {
const job = jobsRepo.register({
policy: "periodic-reminder",
specYaml: "context:\n target:\n session: a@rig\n message: m\n",
targetSession: "a@rig",
intervalSeconds: 1,
registeredBySession: "ops@kernel",
});
const engine = makeEngine();
const sched = new WatchdogScheduler({ jobsRepo, policyEngine: engine });
await sched.runTickNow();
expect(deliveries.length).toBe(1);
// Re-mark as if evaluation was 5s ago to make the next tick due again.
jobsRepo.recordEvaluation(job.jobId, new Date(Date.now() - 5_000).toISOString(), true);
await sched.runTickNow();
expect(deliveries.length).toBe(2);
});
it("stopped jobs are excluded from evaluation", async () => {
const job = jobsRepo.register({
policy: "periodic-reminder",
specYaml: "context:\n target:\n session: a@rig\n message: m\n",
targetSession: "a@rig",
intervalSeconds: 30,
registeredBySession: "ops@kernel",
});
jobsRepo.stop(job.jobId);
const engine = makeEngine();
const sched = new WatchdogScheduler({ jobsRepo, policyEngine: engine });
await sched.runTickNow();
expect(deliveries).toEqual([]);
});
it("terminal jobs are excluded from evaluation", async () => {
const job = jobsRepo.register({
policy: "periodic-reminder",
specYaml: "context:\n target:\n session: a@rig\n message: m\n",
targetSession: "a@rig",
intervalSeconds: 30,
registeredBySession: "ops@kernel",
});
jobsRepo.markTerminal(job.jobId, "done");
const engine = makeEngine();
const sched = new WatchdogScheduler({ jobsRepo, policyEngine: engine });
await sched.runTickNow();
expect(deliveries).toEqual([]);
});
it("policy evaluation errors are caught and tick continues for siblings", async () => {
jobsRepo.register({
policy: "periodic-reminder",
specYaml: "context:\n message: only-message-no-target\n",
targetSession: "a@rig",
intervalSeconds: 30,
registeredBySession: "ops@kernel",
});
const ok = jobsRepo.register({
policy: "periodic-reminder",
specYaml: "context:\n target:\n session: ok@rig\n message: ok\n",
targetSession: "ok@rig",
intervalSeconds: 30,
registeredBySession: "ops@kernel",
});
const engine = makeEngine();
const errors: unknown[] = [];
const sched = new WatchdogScheduler({
jobsRepo,
policyEngine: engine,
onTickError: (err) => errors.push(err),
});
await sched.runTickNow();
expect(errors.length).toBe(1);
expect(deliveries).toEqual([{ targetSession: "ok@rig", message: "ok" }]);
// After successful eval the second job's last_evaluation_at should be set.
const after = jobsRepo.getByIdOrThrow(ok.jobId);
expect(after.lastEvaluationAt).not.toBeNull();
});
it("start + stop are idempotent and do not throw", async () => {
const engine = makeEngine();
const sched = new WatchdogScheduler({ jobsRepo, policyEngine: engine, tickIntervalMs: 60_000 });
sched.start();
sched.start();
expect(sched.isRunning()).toBe(true);
await sched.stop();
await sched.stop();
expect(sched.isRunning()).toBe(false);
});
it("recovers schedule across restart via SQLite (new repo + new scheduler picks up active jobs)", async () => {
jobsRepo.register({
policy: "periodic-reminder",
specYaml: "context:\n target:\n session: persist@rig\n message: persisted\n",
targetSession: "persist@rig",
intervalSeconds: 1,
registeredBySession: "ops@kernel",
});
// Simulate restart: new repo + new history-log + new engine + new scheduler
// sharing the same DB handle. Phase A/B pattern; SQLite is canonical.
const repo2 = new WatchdogJobsRepository(db);
const log2 = new WatchdogHistoryLog(db);
const engine2 = new WatchdogPolicyEngine({
jobsRepo: repo2,
historyLog: log2,
eventBus: bus,
deliver: async (req) => {
deliveries.push(req);
return { status: "ok" };
},
});
const sched2 = new WatchdogScheduler({ jobsRepo: repo2, policyEngine: engine2 });
await sched2.runTickNow();
expect(deliveries).toEqual([{ targetSession: "persist@rig", message: "persisted" }]);
});
});