fix(watchdog): the idle gate fires once per material state, not once per window

An unchanged human-blocked row re-woke its orchestrator on a timer. Reproduced on
an isolated daemon before any of this was written, two distinct routes:

  R-1  window expiry: wakes at +0s and +120.3s, exactly the configured
       active-wake window, with the row's only transitions being `created` and
       its park.
  R-2  flicker: anchor fire, seat briefly active (observed driving actionable=0),
       seat idle again, re-fire 60.1s into a 120s window.

R-2 is the route that explains the field report. The active-wake window was never
a cooldown: watchdog-policy-engine clears `actionable` on EVERY skip, and the
throttle's precondition includes `job.actionable`, so a moment of seat busyness
skips the window entirely. At the production 900s window a flickering seat can be
re-woken as fast as the 60s scan — which window expiry alone cannot produce, and
which matches the reported ~3 minutes.

The gate now fires once per MATERIAL STATE of the gated set. A fingerprint over
the qualifying row ids and each row's material state — row state, blocker and
blocker state, claim, and the latest transition that is NOT wake machinery —
is compared against the one the job last fired for.

LAYER: the policy, not the engine. The engine keeps today's throttle semantics
for every other policy by construction rather than by assurance — the only engine
edit is adding one skip-reason string to the quiet list, which changes no
behaviour and which no other policy emits. The window remains beneath the
fingerprint gate as a floor.

The memory keys to the CONDITION, never to seat activity: skips for seat_active,
seat_needs_input and activity_stale_unknown return before it is read or written,
so a flicker can no longer launder a fresh wake out of an unchanged row.

Wake-machinery markers (wake-attempt:, escalation-rung:, ladder-exhausted:,
ladder-suspend:/resume:, park wake fired:, idle-gate:) are excluded from
materiality — if they counted, every wake would justify the next one.

State: one nullable column, overwritten, never appended to. A per-wake ledger is
the bookkeeping this repair is not allowed to add. Suppression stays derivable
without one: the row is untouched and still appears in held/escalations, and the
job carries the exact condition it fired for.

Two existing assertions are AMENDED, not loosened: both asserted the skip reason
`active_wake_not_due`, which is now `gate_condition_unchanged` because a stricter
gate decides first. Their subject — fires once, delivers once — is unchanged and
still asserted by the delivery counts. The new reason is asserted exactly rather
than relaxed to "some skip", because the two are not interchangeable: one expires
with the window and is bypassed by a flicker, the other holds until the gated set
materially changes.
This commit is contained in:
v-openrig-build committed 2026-09-01 11:50:44 +00:00
1 parent fdd26628f3
commit f610eec7d2
6 files changed
+261 -7

No files matched your search

+3 -1
View File
@@ -80,9 +80,10 @@ import { contextUsageWatchdogSchema } from "./migrations/074_context_usage_watch
import { contextUsageWatchdogGenerationSchema } from "./migrations/075_context_usage_watchdog_generation.js";
import { ownerNotificationLevelsSchema } from "./migrations/076_owner_notification_levels.js";
import { nodeSessionSourceSchema } from "./migrations/077_node_session_source.js";
import { idleGateFiredConditionSchema } from "./migrations/078_idle_gate_fired_condition.js";
import type { Migration } from "./migrate.js";
/** Ordered 001→077 (068 enforcer decisions, dropped forward by 071; 069/070 W3 launch truth). */
/** Ordered 001→078 (068 enforcer decisions, dropped forward by 071; 069/070 W3 launch truth). */
export const ALL_MIGRATIONS: Migration[] = [
coreSchema,
bindingsSessionsSchema,
@@ -161,4 +162,5 @@ export const ALL_MIGRATIONS: Migration[] = [
contextUsageWatchdogGenerationSchema,
ownerNotificationLevelsSchema,
nodeSessionSourceSchema,
idleGateFiredConditionSchema,
];
@@ -0,0 +1,22 @@
import type { Migration } from "../migrate.js";
/**
* OPR.0.5.8.1 S2 — one overwritten value per job: the gated CONDITION the
* idle-gate last fired for.
*
* The idle-gate wake repeated on a timer rather than on a change, because its
* only cooldown was the engine's active-wake window — and every skip clears
* `actionable`, so a moment of seat busyness bypassed that window entirely.
* Remembering what was last fired for lets the policy fire once per material
* state of the gated set instead.
*
* Nullable and single-valued on purpose. It is overwritten, never appended to:
* a per-wake ledger would be the bookkeeping this repair is explicitly not
* allowed to add. A null means "nothing fired for yet", which fires once.
*/
export const idleGateFiredConditionSchema: Migration = {
name: "078_idle_gate_fired_condition.sql",
sql: `
ALTER TABLE watchdog_jobs ADD COLUMN last_fired_condition TEXT;
`,
};
@@ -17,6 +17,7 @@
// active_wake_interval_seconds on the registered job). It WAKES ONLY — it
// never claims or acts on the gate.
import { createHash } from "node:crypto";
import type Database from "better-sqlite3";
import type { AgentActivityStore } from "../agent-activity-store.js";
import { effectiveGateRoles } from "../gate-predicate.js";
@@ -102,8 +103,37 @@ export function makeIdleGateQitemPolicy(deps: IdleGateQitemDeps): Policy {
return { action: "skip", reason: "seat_active", notes: { seat, activityState: activity.state } };
}
// Signal C — OPR.0.5.8.1 S2. Fire once per MATERIAL STATE of the gated
// set, not once per window and not once per idle episode.
//
// The engine's active-wake window was the only cooldown, and it is not
// one: `watchdog-policy-engine.ts` clears `actionable` on EVERY skip, and
// the throttle's precondition includes `job.actionable`. So a seat that
// went briefly active between scans re-opened immediate firing —
// reproduced at 60.1s into a 120s window. Window expiry alone also
// re-woke an unchanged row every interval.
//
// The memory keys to the CONDITION, never to seat activity: a skip for
// seat_active / needs_input / activity_stale returns above WITHOUT
// touching it, so a flicker can no longer manufacture a fresh wake.
const fingerprint = gatedConditionFingerprint(db, gated.map((g) => g.qitemId));
const firedFor = (
db.prepare("SELECT last_fired_condition AS c FROM watchdog_jobs WHERE job_id = ?")
.get(job.jobId) as { c: string | null } | undefined
)?.c ?? null;
if (firedFor !== null && firedFor === fingerprint) {
return {
action: "skip",
reason: "gate_condition_unchanged",
notes: { seat, pendingGateCount: gated.length },
};
}
// Both signals joined → ONE bounded wake (single-target, keepalive
// shape). Engine active-wake throttle provides the cooldown.
// shape). Recorded at decision time, mirroring how the engine stamps
// last_fire_at on the send path regardless of delivery outcome.
db.prepare("UPDATE watchdog_jobs SET last_fired_condition = ? WHERE job_id = ?")
.run(fingerprint, job.jobId);
const primary = gated[0]!;
const message =
job.message ??
@@ -129,6 +159,56 @@ export function makeIdleGateQitemPolicy(deps: IdleGateQitemDeps): Policy {
};
}
/**
* Wake-machinery markers are NEVER material. If they were, every wake would
* justify the next one and the gate would be decorative.
*/
const WAKE_MACHINERY_NOTE_PREFIXES = [
"wake-attempt:",
"escalation-rung:",
"ladder-exhausted:",
"ladder-suspend:",
"ladder-resume:",
"park wake fired:",
"idle-gate:",
];
/**
* A stable digest of the gated condition: which rows qualify, and the material
* state of each.
*
* Material state per the ruled definition — a substantive transition on the row
* (state / claim / content-bearing note / resolution), a transition on its
* blocker or the blocker reaching terminal, and pickup evidence. Row identity
* alone is not enough: the same two rows can be in different states.
*
* Derived entirely from existing queue facts. Nothing new is recorded per wake;
* the only stored value is the single overwritten fingerprint on the job.
*/
function gatedConditionFingerprint(db: Database.Database, qitemIds: readonly string[]): string {
const notMachinery = WAKE_MACHINERY_NOTE_PREFIXES.map(() => "COALESCE(t.transition_note,'') NOT LIKE ?").join(" AND ");
const latestSubstantive = db.prepare(
`SELECT MAX(t.transition_id) AS marker
FROM queue_transitions t
WHERE t.qitem_id = ? AND ${notMachinery}`,
);
const rowFacts = db.prepare(
`SELECT q.state AS state, COALESCE(q.blocked_on,'') AS blockedOn,
COALESCE(q.claimed_at,'') AS claimedAt,
COALESCE((SELECT b.state FROM queue_items b WHERE b.qitem_id = q.blocked_on),'') AS blockerState
FROM queue_items q WHERE q.qitem_id = ?`,
);
const likeArgs = WAKE_MACHINERY_NOTE_PREFIXES.map((p) => `${p}%`);
const parts = [...qitemIds].sort().map((id) => {
const f = rowFacts.get(id) as
| { state: string; blockedOn: string; claimedAt: string; blockerState: string }
| undefined;
const marker = (latestSubstantive.get(id, ...likeArgs) as { marker: number | null } | undefined)?.marker ?? 0;
return [id, f?.state ?? "", f?.blockedOn ?? "", f?.blockerState ?? "", f?.claimedAt ?? "", marker].join("|");
});
return createHash("sha256").update(parts.join("\n")).digest("hex").slice(0, 32);
}
function buildIdleGateMessage(input: {
seat: string;
qitemId: string;
@@ -124,6 +124,15 @@ const QUIET_SKIP_REASONS = new Set<string>([
// audited signal is the WAKE (fired) path.
"no_pending_gate",
"seat_active",
// OPR.0.5.8.1 S2 — the gated condition has not materially changed since the
// wake that was already delivered for it. Quiet for the same reason as its
// siblings: a 60s scan must not write a history row every minute. The
// suppression stays derivable without one — the row itself remains visible in
// the held/escalations views (this suppresses the WAKE, never the record),
// and the job carries `last_fired_condition`, the exact condition it fired
// for. Adding this string changes no throttle semantics and no other policy
// emits it.
"gate_condition_unchanged",
// OPR.0.5.6.24 — the parked-owner consumer's clean-scan no-op: nothing
// parked, nothing closed, nothing deferred. Suppressed so a routine rig
// scan writes no history (the loud, audited signals are the SENT wake,
@@ -4,6 +4,7 @@ import { createFullTestDb } from "./helpers/test-app.js";
import { migrate } from "../src/db/migrate.js";
import { watchdogJobsSchema } from "../src/db/migrations/031_watchdog_jobs.js";
import { watchdogHistorySchema } from "../src/db/migrations/032_watchdog_history.js";
import { idleGateFiredConditionSchema } from "../src/db/migrations/078_idle_gate_fired_condition.js";
import { RigRepository } from "../src/domain/rig-repository.js";
import { SessionRegistry } from "../src/domain/session-registry.js";
import { EventBus } from "../src/domain/event-bus.js";
@@ -71,7 +72,7 @@ describe("idle-gate-qitem policy (OPR.0.4.3.16)", () => {
beforeEach(() => {
db = createFullTestDb();
migrate(db, [watchdogJobsSchema, watchdogHistorySchema]); // idempotent; adds watchdog tables
migrate(db, [watchdogJobsSchema, watchdogHistorySchema, idleGateFiredConditionSchema]); // idempotent; adds watchdog tables
eventBus = new EventBus(db);
store = new AgentActivityStore({ db, eventBus, now: () => NOW });
seedSeat();
@@ -94,6 +95,122 @@ describe("idle-gate-qitem policy (OPR.0.4.3.16)", () => {
expect(out.notes?.activityEvidenceSource).toBe("runtime_hook");
});
// --- OPR.0.5.8.1 S2: fire once per MATERIAL STATE of the gated set ---
//
// Reproduced on a real daemon before any of this was written: an unchanged
// gate-shaped blocked row re-woke the seat at +0s and +120.3s (window expiry),
// and a brief seat-active flicker cleared `actionable` and produced a re-fire
// 60.1s into a 120s window. The window was never a cooldown.
function seedJobRow(jobId = "job-1"): void {
db.prepare(
`INSERT INTO watchdog_jobs (job_id, policy, spec_yaml, target_session, interval_seconds,
active_wake_interval_seconds, state, registered_by_session, registered_at)
VALUES (?, 'idle-gate-qitem', 'policy: idle-gate-qitem', ?, 30, 300, 'active', 'ops@kernel', '2026-07-03T07:00:00Z')`,
).run(jobId, SEAT);
}
const firedCondition = (jobId = "job-1") =>
(db.prepare("SELECT last_fired_condition AS c FROM watchdog_jobs WHERE job_id = ?").get(jobId) as
| { c: string | null }
| undefined)?.c ?? null;
function appendTransition(qitemId: string, note: string, id: number): void {
db.prepare(
`INSERT INTO queue_transitions (transition_id, qitem_id, ts, state, actor_session, transition_note)
VALUES (?, ?, '2026-07-03T08:00:00Z', 'blocked', 'someone@rig', ?)`,
).run(id, qitemId, note);
}
it("S2 R-1 — an UNCHANGED gated set does not re-wake when the window expires", async () => {
seedJobRow();
seedGateQitem("q-gate-1");
seedActivity("Stop", FRESH);
const policy = makeIdleGateQitemPolicy({ db, agentActivityStore: store });
expect((await policy.evaluate(makeJob())).action).toBe("send");
// Second evaluation with the window fully elapsed and nothing changed.
const out = await policy.evaluate(makeJob({ lastFireAt: "2026-07-03T07:00:00.000Z" }));
expect(out.action).toBe("skip");
expect((out as { reason: string }).reason).toBe("gate_condition_unchanged");
});
it("S2 R-2 — a seat_active FLICKER does not reset the fired-for memory", async () => {
// The engine clears `actionable` on every skip, which is what made a brief
// busy moment re-open immediate firing. The condition memory lives in the
// policy and keys to the CONDITION, so an activity skip cannot launder it.
seedJobRow();
seedGateQitem("q-gate-1");
seedActivity("Stop", FRESH);
const policy = makeIdleGateQitemPolicy({ db, agentActivityStore: store });
expect((await policy.evaluate(makeJob())).action).toBe("send");
const afterFire = firedCondition();
expect(afterFire).not.toBeNull();
seedActivity("UserPromptSubmit", FRESH); // seat goes active
const busy = await policy.evaluate(makeJob());
expect((busy as { reason: string }).reason).toBe("seat_active");
expect(firedCondition()).toBe(afterFire); // memory untouched by the flicker
seedActivity("Stop", FRESH); // idle again
const out = await policy.evaluate(makeJob({ lastFireAt: null, actionable: false } as never));
expect(out.action).toBe("skip");
expect((out as { reason: string }).reason).toBe("gate_condition_unchanged");
});
it("S2 preserve — a NEW gate qitem arriving at the seat wakes", async () => {
seedJobRow();
seedGateQitem("q-gate-1");
seedActivity("Stop", FRESH);
const policy = makeIdleGateQitemPolicy({ db, agentActivityStore: store });
expect((await policy.evaluate(makeJob())).action).toBe("send");
seedGateQitem("q-gate-2");
expect((await policy.evaluate(makeJob())).action).toBe("send");
});
it("S2 preserve — a MATERIAL transition on the row re-wakes promptly", async () => {
seedJobRow();
seedGateQitem("q-gate-1");
seedActivity("Stop", FRESH);
const policy = makeIdleGateQitemPolicy({ db, agentActivityStore: store });
expect((await policy.evaluate(makeJob())).action).toBe("send");
expect((await policy.evaluate(makeJob())).action).toBe("skip");
appendTransition("q-gate-1", "reviewer asked a question", 9001);
expect((await policy.evaluate(makeJob())).action).toBe("send");
});
it("S2 — wake-machinery markers are NEVER material (else every wake justifies the next)", async () => {
seedJobRow();
seedGateQitem("q-gate-1");
seedActivity("Stop", FRESH);
const policy = makeIdleGateQitemPolicy({ db, agentActivityStore: store });
expect((await policy.evaluate(makeJob())).action).toBe("send");
appendTransition("q-gate-1", "wake-attempt: 1/3 outcome=delivered", 9002);
appendTransition("q-gate-1", "escalation-rung: orchestrator -> orch@rig", 9003);
const out = await policy.evaluate(makeJob());
expect(out.action).toBe("skip");
expect((out as { reason: string }).reason).toBe("gate_condition_unchanged");
});
it("S2 — a suppressed wake stays derivable: the job records the condition it fired for", async () => {
// Suppressing the WAKE, never the record. The row itself is untouched and
// still appears in held/escalations; the job carries the exact condition.
seedJobRow();
seedGateQitem("q-gate-1");
seedActivity("Stop", FRESH);
const policy = makeIdleGateQitemPolicy({ db, agentActivityStore: store });
expect(firedCondition()).toBeNull();
await policy.evaluate(makeJob());
const recorded = firedCondition();
expect(recorded).toMatch(/^[0-9a-f]{32}$/);
await policy.evaluate(makeJob());
expect(firedCondition()).toBe(recorded); // one overwritten value, not a ledger
expect(
(db.prepare("SELECT COUNT(*) AS n FROM queue_items WHERE qitem_id = 'q-gate-1'").get() as { n: number }).n,
).toBe(1);
});
it("human-gate tier (no gate:* tag) + FRESH idle → send (secondary predicate, gate:human)", async () => {
seedGateQitem("q-human-1", { tags: null, tier: "human-gate" });
seedActivity("Stop", FRESH);
@@ -215,11 +332,23 @@ describe("idle-gate-qitem policy (OPR.0.4.3.16)", () => {
expect(deliveries[0]?.targetSession).toBe(SEAT);
expect(historyLog.listForJob(registered.jobId)[0]?.outcome).toBe("sent");
// Second immediate evaluation → engine active-wake throttle quiet-skips
// (cooldown is FREE; no duplicate wake).
// Second immediate evaluation → quiet skip, no duplicate wake.
//
// AMENDED by OPR.0.5.8.1 S2. This test's subject — fires once, then no
// duplicate wake — is unchanged and still asserted by the delivery count,
// which is what the user actually experiences. Only the REASON moved:
// the condition gate now decides before the engine's active-wake window
// is consulted, so the skip is `gate_condition_unchanged` rather than
// `active_wake_not_due`.
//
// Asserting the new reason rather than loosening to "any skip" on
// purpose: the two reasons are not interchangeable. `active_wake_not_due`
// expires with the window and is bypassed entirely once a skip has
// cleared `actionable`; `gate_condition_unchanged` holds until the gated
// set materially changes. Which one fired is the whole repair.
const r2 = await engine.evaluate(jobsRepo.getByIdOrThrow(registered.jobId));
expect(r2.outcome.action).toBe("skip");
expect((r2.outcome as { reason: string }).reason).toBe("active_wake_not_due");
expect((r2.outcome as { reason: string }).reason).toBe("gate_condition_unchanged");
expect(deliveries).toHaveLength(1);
});
});
@@ -412,8 +412,20 @@ describe("W2c watchdog auto-registration — production composition", () => {
expect(deliveries[0]?.message).toContain("q-w2c-delivery");
expect(history.listForJob(rows[0]!.jobId).map((entry) => entry.outcome)).toEqual(["sent"]);
// AMENDED by OPR.0.5.8.1 S2. The subject — "delivers once" — is unchanged
// and still asserted by the delivery count below. Only the skip REASON
// moved: the gated-condition gate now decides before the engine's
// active-wake window is consulted. The two are not interchangeable, so
// the new reason is asserted exactly rather than relaxed to "some skip".
const second = await engine.evaluate(deps.watchdogJobsRepo!.getByIdOrThrow(rows[0]!.jobId));
expect(second.outcome).toEqual({ action: "skip", reason: "active_wake_not_due" });
expect(second.outcome).toEqual({
action: "skip",
reason: "gate_condition_unchanged",
// Notes match the convention of this policy's sibling quiet skips
// (seat_active carries them too): available to a caller inspecting the
// outcome, while the quiet classification keeps them out of history.
notes: { seat: sessionName, pendingGateCount: 1 },
});
expect(deliveries).toHaveLength(1);
expect(autoRows(db, sessionName)).toHaveLength(1);