mirror of
https://github.com/mvschwarz/openrig.git
synced 2026-11-08 08:11:17 -10:00
fix(watchdog): the idle gate fires once per material state, not once per window
An unchanged human-blocked row re-woke its orchestrator on a timer. Reproduced on
an isolated daemon before any of this was written, two distinct routes:
R-1 window expiry: wakes at +0s and +120.3s, exactly the configured
active-wake window, with the row's only transitions being `created` and
its park.
R-2 flicker: anchor fire, seat briefly active (observed driving actionable=0),
seat idle again, re-fire 60.1s into a 120s window.
R-2 is the route that explains the field report. The active-wake window was never
a cooldown: watchdog-policy-engine clears `actionable` on EVERY skip, and the
throttle's precondition includes `job.actionable`, so a moment of seat busyness
skips the window entirely. At the production 900s window a flickering seat can be
re-woken as fast as the 60s scan — which window expiry alone cannot produce, and
which matches the reported ~3 minutes.
The gate now fires once per MATERIAL STATE of the gated set. A fingerprint over
the qualifying row ids and each row's material state — row state, blocker and
blocker state, claim, and the latest transition that is NOT wake machinery —
is compared against the one the job last fired for.
LAYER: the policy, not the engine. The engine keeps today's throttle semantics
for every other policy by construction rather than by assurance — the only engine
edit is adding one skip-reason string to the quiet list, which changes no
behaviour and which no other policy emits. The window remains beneath the
fingerprint gate as a floor.
The memory keys to the CONDITION, never to seat activity: skips for seat_active,
seat_needs_input and activity_stale_unknown return before it is read or written,
so a flicker can no longer launder a fresh wake out of an unchanged row.
Wake-machinery markers (wake-attempt:, escalation-rung:, ladder-exhausted:,
ladder-suspend:/resume:, park wake fired:, idle-gate:) are excluded from
materiality — if they counted, every wake would justify the next one.
State: one nullable column, overwritten, never appended to. A per-wake ledger is
the bookkeeping this repair is not allowed to add. Suppression stays derivable
without one: the row is untouched and still appears in held/escalations, and the
job carries the exact condition it fired for.
Two existing assertions are AMENDED, not loosened: both asserted the skip reason
`active_wake_not_due`, which is now `gate_condition_unchanged` because a stricter
gate decides first. Their subject — fires once, delivers once — is unchanged and
still asserted by the delivery counts. The new reason is asserted exactly rather
than relaxed to "some skip", because the two are not interchangeable: one expires
with the window and is bypassed by a flicker, the other holds until the gated set
materially changes.
This commit is contained in:
1 parent
fdd26628f3
commit
f610eec7d2
6 files changed
+261
-7
No files matched your search
@@ -80,9 +80,10 @@ import { contextUsageWatchdogSchema } from "./migrations/074_context_usage_watch
|
||||
import { contextUsageWatchdogGenerationSchema } from "./migrations/075_context_usage_watchdog_generation.js";
|
||||
import { ownerNotificationLevelsSchema } from "./migrations/076_owner_notification_levels.js";
|
||||
import { nodeSessionSourceSchema } from "./migrations/077_node_session_source.js";
|
||||
import { idleGateFiredConditionSchema } from "./migrations/078_idle_gate_fired_condition.js";
|
||||
import type { Migration } from "./migrate.js";
|
||||
|
||||
/** Ordered 001→077 (068 enforcer decisions, dropped forward by 071; 069/070 W3 launch truth). */
|
||||
/** Ordered 001→078 (068 enforcer decisions, dropped forward by 071; 069/070 W3 launch truth). */
|
||||
export const ALL_MIGRATIONS: Migration[] = [
|
||||
coreSchema,
|
||||
bindingsSessionsSchema,
|
||||
@@ -161,4 +162,5 @@ export const ALL_MIGRATIONS: Migration[] = [
|
||||
contextUsageWatchdogGenerationSchema,
|
||||
ownerNotificationLevelsSchema,
|
||||
nodeSessionSourceSchema,
|
||||
idleGateFiredConditionSchema,
|
||||
];
|
||||
@@ -0,0 +1,22 @@
|
||||
import type { Migration } from "../migrate.js";
|
||||
|
||||
/**
|
||||
* OPR.0.5.8.1 S2 — one overwritten value per job: the gated CONDITION the
|
||||
* idle-gate last fired for.
|
||||
*
|
||||
* The idle-gate wake repeated on a timer rather than on a change, because its
|
||||
* only cooldown was the engine's active-wake window — and every skip clears
|
||||
* `actionable`, so a moment of seat busyness bypassed that window entirely.
|
||||
* Remembering what was last fired for lets the policy fire once per material
|
||||
* state of the gated set instead.
|
||||
*
|
||||
* Nullable and single-valued on purpose. It is overwritten, never appended to:
|
||||
* a per-wake ledger would be the bookkeeping this repair is explicitly not
|
||||
* allowed to add. A null means "nothing fired for yet", which fires once.
|
||||
*/
|
||||
export const idleGateFiredConditionSchema: Migration = {
|
||||
name: "078_idle_gate_fired_condition.sql",
|
||||
sql: `
|
||||
ALTER TABLE watchdog_jobs ADD COLUMN last_fired_condition TEXT;
|
||||
`,
|
||||
};
|
||||
@@ -17,6 +17,7 @@
|
||||
// active_wake_interval_seconds on the registered job). It WAKES ONLY — it
|
||||
// never claims or acts on the gate.
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
import type Database from "better-sqlite3";
|
||||
import type { AgentActivityStore } from "../agent-activity-store.js";
|
||||
import { effectiveGateRoles } from "../gate-predicate.js";
|
||||
@@ -102,8 +103,37 @@ export function makeIdleGateQitemPolicy(deps: IdleGateQitemDeps): Policy {
|
||||
return { action: "skip", reason: "seat_active", notes: { seat, activityState: activity.state } };
|
||||
}
|
||||
|
||||
// Signal C — OPR.0.5.8.1 S2. Fire once per MATERIAL STATE of the gated
|
||||
// set, not once per window and not once per idle episode.
|
||||
//
|
||||
// The engine's active-wake window was the only cooldown, and it is not
|
||||
// one: `watchdog-policy-engine.ts` clears `actionable` on EVERY skip, and
|
||||
// the throttle's precondition includes `job.actionable`. So a seat that
|
||||
// went briefly active between scans re-opened immediate firing —
|
||||
// reproduced at 60.1s into a 120s window. Window expiry alone also
|
||||
// re-woke an unchanged row every interval.
|
||||
//
|
||||
// The memory keys to the CONDITION, never to seat activity: a skip for
|
||||
// seat_active / needs_input / activity_stale returns above WITHOUT
|
||||
// touching it, so a flicker can no longer manufacture a fresh wake.
|
||||
const fingerprint = gatedConditionFingerprint(db, gated.map((g) => g.qitemId));
|
||||
const firedFor = (
|
||||
db.prepare("SELECT last_fired_condition AS c FROM watchdog_jobs WHERE job_id = ?")
|
||||
.get(job.jobId) as { c: string | null } | undefined
|
||||
)?.c ?? null;
|
||||
if (firedFor !== null && firedFor === fingerprint) {
|
||||
return {
|
||||
action: "skip",
|
||||
reason: "gate_condition_unchanged",
|
||||
notes: { seat, pendingGateCount: gated.length },
|
||||
};
|
||||
}
|
||||
|
||||
// Both signals joined → ONE bounded wake (single-target, keepalive
|
||||
// shape). Engine active-wake throttle provides the cooldown.
|
||||
// shape). Recorded at decision time, mirroring how the engine stamps
|
||||
// last_fire_at on the send path regardless of delivery outcome.
|
||||
db.prepare("UPDATE watchdog_jobs SET last_fired_condition = ? WHERE job_id = ?")
|
||||
.run(fingerprint, job.jobId);
|
||||
const primary = gated[0]!;
|
||||
const message =
|
||||
job.message ??
|
||||
@@ -129,6 +159,56 @@ export function makeIdleGateQitemPolicy(deps: IdleGateQitemDeps): Policy {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Wake-machinery markers are NEVER material. If they were, every wake would
|
||||
* justify the next one and the gate would be decorative.
|
||||
*/
|
||||
const WAKE_MACHINERY_NOTE_PREFIXES = [
|
||||
"wake-attempt:",
|
||||
"escalation-rung:",
|
||||
"ladder-exhausted:",
|
||||
"ladder-suspend:",
|
||||
"ladder-resume:",
|
||||
"park wake fired:",
|
||||
"idle-gate:",
|
||||
];
|
||||
|
||||
/**
|
||||
* A stable digest of the gated condition: which rows qualify, and the material
|
||||
* state of each.
|
||||
*
|
||||
* Material state per the ruled definition — a substantive transition on the row
|
||||
* (state / claim / content-bearing note / resolution), a transition on its
|
||||
* blocker or the blocker reaching terminal, and pickup evidence. Row identity
|
||||
* alone is not enough: the same two rows can be in different states.
|
||||
*
|
||||
* Derived entirely from existing queue facts. Nothing new is recorded per wake;
|
||||
* the only stored value is the single overwritten fingerprint on the job.
|
||||
*/
|
||||
function gatedConditionFingerprint(db: Database.Database, qitemIds: readonly string[]): string {
|
||||
const notMachinery = WAKE_MACHINERY_NOTE_PREFIXES.map(() => "COALESCE(t.transition_note,'') NOT LIKE ?").join(" AND ");
|
||||
const latestSubstantive = db.prepare(
|
||||
`SELECT MAX(t.transition_id) AS marker
|
||||
FROM queue_transitions t
|
||||
WHERE t.qitem_id = ? AND ${notMachinery}`,
|
||||
);
|
||||
const rowFacts = db.prepare(
|
||||
`SELECT q.state AS state, COALESCE(q.blocked_on,'') AS blockedOn,
|
||||
COALESCE(q.claimed_at,'') AS claimedAt,
|
||||
COALESCE((SELECT b.state FROM queue_items b WHERE b.qitem_id = q.blocked_on),'') AS blockerState
|
||||
FROM queue_items q WHERE q.qitem_id = ?`,
|
||||
);
|
||||
const likeArgs = WAKE_MACHINERY_NOTE_PREFIXES.map((p) => `${p}%`);
|
||||
const parts = [...qitemIds].sort().map((id) => {
|
||||
const f = rowFacts.get(id) as
|
||||
| { state: string; blockedOn: string; claimedAt: string; blockerState: string }
|
||||
| undefined;
|
||||
const marker = (latestSubstantive.get(id, ...likeArgs) as { marker: number | null } | undefined)?.marker ?? 0;
|
||||
return [id, f?.state ?? "", f?.blockedOn ?? "", f?.blockerState ?? "", f?.claimedAt ?? "", marker].join("|");
|
||||
});
|
||||
return createHash("sha256").update(parts.join("\n")).digest("hex").slice(0, 32);
|
||||
}
|
||||
|
||||
function buildIdleGateMessage(input: {
|
||||
seat: string;
|
||||
qitemId: string;
|
||||
|
||||
@@ -124,6 +124,15 @@ const QUIET_SKIP_REASONS = new Set<string>([
|
||||
// audited signal is the WAKE (fired) path.
|
||||
"no_pending_gate",
|
||||
"seat_active",
|
||||
// OPR.0.5.8.1 S2 — the gated condition has not materially changed since the
|
||||
// wake that was already delivered for it. Quiet for the same reason as its
|
||||
// siblings: a 60s scan must not write a history row every minute. The
|
||||
// suppression stays derivable without one — the row itself remains visible in
|
||||
// the held/escalations views (this suppresses the WAKE, never the record),
|
||||
// and the job carries `last_fired_condition`, the exact condition it fired
|
||||
// for. Adding this string changes no throttle semantics and no other policy
|
||||
// emits it.
|
||||
"gate_condition_unchanged",
|
||||
// OPR.0.5.6.24 — the parked-owner consumer's clean-scan no-op: nothing
|
||||
// parked, nothing closed, nothing deferred. Suppressed so a routine rig
|
||||
// scan writes no history (the loud, audited signals are the SENT wake,
|
||||
|
||||
@@ -4,6 +4,7 @@ import { createFullTestDb } from "./helpers/test-app.js";
|
||||
import { migrate } from "../src/db/migrate.js";
|
||||
import { watchdogJobsSchema } from "../src/db/migrations/031_watchdog_jobs.js";
|
||||
import { watchdogHistorySchema } from "../src/db/migrations/032_watchdog_history.js";
|
||||
import { idleGateFiredConditionSchema } from "../src/db/migrations/078_idle_gate_fired_condition.js";
|
||||
import { RigRepository } from "../src/domain/rig-repository.js";
|
||||
import { SessionRegistry } from "../src/domain/session-registry.js";
|
||||
import { EventBus } from "../src/domain/event-bus.js";
|
||||
@@ -71,7 +72,7 @@ describe("idle-gate-qitem policy (OPR.0.4.3.16)", () => {
|
||||
|
||||
beforeEach(() => {
|
||||
db = createFullTestDb();
|
||||
migrate(db, [watchdogJobsSchema, watchdogHistorySchema]); // idempotent; adds watchdog tables
|
||||
migrate(db, [watchdogJobsSchema, watchdogHistorySchema, idleGateFiredConditionSchema]); // idempotent; adds watchdog tables
|
||||
eventBus = new EventBus(db);
|
||||
store = new AgentActivityStore({ db, eventBus, now: () => NOW });
|
||||
seedSeat();
|
||||
@@ -94,6 +95,122 @@ describe("idle-gate-qitem policy (OPR.0.4.3.16)", () => {
|
||||
expect(out.notes?.activityEvidenceSource).toBe("runtime_hook");
|
||||
});
|
||||
|
||||
// --- OPR.0.5.8.1 S2: fire once per MATERIAL STATE of the gated set ---
|
||||
//
|
||||
// Reproduced on a real daemon before any of this was written: an unchanged
|
||||
// gate-shaped blocked row re-woke the seat at +0s and +120.3s (window expiry),
|
||||
// and a brief seat-active flicker cleared `actionable` and produced a re-fire
|
||||
// 60.1s into a 120s window. The window was never a cooldown.
|
||||
|
||||
function seedJobRow(jobId = "job-1"): void {
|
||||
db.prepare(
|
||||
`INSERT INTO watchdog_jobs (job_id, policy, spec_yaml, target_session, interval_seconds,
|
||||
active_wake_interval_seconds, state, registered_by_session, registered_at)
|
||||
VALUES (?, 'idle-gate-qitem', 'policy: idle-gate-qitem', ?, 30, 300, 'active', 'ops@kernel', '2026-07-03T07:00:00Z')`,
|
||||
).run(jobId, SEAT);
|
||||
}
|
||||
const firedCondition = (jobId = "job-1") =>
|
||||
(db.prepare("SELECT last_fired_condition AS c FROM watchdog_jobs WHERE job_id = ?").get(jobId) as
|
||||
| { c: string | null }
|
||||
| undefined)?.c ?? null;
|
||||
function appendTransition(qitemId: string, note: string, id: number): void {
|
||||
db.prepare(
|
||||
`INSERT INTO queue_transitions (transition_id, qitem_id, ts, state, actor_session, transition_note)
|
||||
VALUES (?, ?, '2026-07-03T08:00:00Z', 'blocked', 'someone@rig', ?)`,
|
||||
).run(id, qitemId, note);
|
||||
}
|
||||
|
||||
it("S2 R-1 — an UNCHANGED gated set does not re-wake when the window expires", async () => {
|
||||
seedJobRow();
|
||||
seedGateQitem("q-gate-1");
|
||||
seedActivity("Stop", FRESH);
|
||||
const policy = makeIdleGateQitemPolicy({ db, agentActivityStore: store });
|
||||
|
||||
expect((await policy.evaluate(makeJob())).action).toBe("send");
|
||||
// Second evaluation with the window fully elapsed and nothing changed.
|
||||
const out = await policy.evaluate(makeJob({ lastFireAt: "2026-07-03T07:00:00.000Z" }));
|
||||
expect(out.action).toBe("skip");
|
||||
expect((out as { reason: string }).reason).toBe("gate_condition_unchanged");
|
||||
});
|
||||
|
||||
it("S2 R-2 — a seat_active FLICKER does not reset the fired-for memory", async () => {
|
||||
// The engine clears `actionable` on every skip, which is what made a brief
|
||||
// busy moment re-open immediate firing. The condition memory lives in the
|
||||
// policy and keys to the CONDITION, so an activity skip cannot launder it.
|
||||
seedJobRow();
|
||||
seedGateQitem("q-gate-1");
|
||||
seedActivity("Stop", FRESH);
|
||||
const policy = makeIdleGateQitemPolicy({ db, agentActivityStore: store });
|
||||
expect((await policy.evaluate(makeJob())).action).toBe("send");
|
||||
const afterFire = firedCondition();
|
||||
expect(afterFire).not.toBeNull();
|
||||
|
||||
seedActivity("UserPromptSubmit", FRESH); // seat goes active
|
||||
const busy = await policy.evaluate(makeJob());
|
||||
expect((busy as { reason: string }).reason).toBe("seat_active");
|
||||
expect(firedCondition()).toBe(afterFire); // memory untouched by the flicker
|
||||
|
||||
seedActivity("Stop", FRESH); // idle again
|
||||
const out = await policy.evaluate(makeJob({ lastFireAt: null, actionable: false } as never));
|
||||
expect(out.action).toBe("skip");
|
||||
expect((out as { reason: string }).reason).toBe("gate_condition_unchanged");
|
||||
});
|
||||
|
||||
it("S2 preserve — a NEW gate qitem arriving at the seat wakes", async () => {
|
||||
seedJobRow();
|
||||
seedGateQitem("q-gate-1");
|
||||
seedActivity("Stop", FRESH);
|
||||
const policy = makeIdleGateQitemPolicy({ db, agentActivityStore: store });
|
||||
expect((await policy.evaluate(makeJob())).action).toBe("send");
|
||||
|
||||
seedGateQitem("q-gate-2");
|
||||
expect((await policy.evaluate(makeJob())).action).toBe("send");
|
||||
});
|
||||
|
||||
it("S2 preserve — a MATERIAL transition on the row re-wakes promptly", async () => {
|
||||
seedJobRow();
|
||||
seedGateQitem("q-gate-1");
|
||||
seedActivity("Stop", FRESH);
|
||||
const policy = makeIdleGateQitemPolicy({ db, agentActivityStore: store });
|
||||
expect((await policy.evaluate(makeJob())).action).toBe("send");
|
||||
expect((await policy.evaluate(makeJob())).action).toBe("skip");
|
||||
|
||||
appendTransition("q-gate-1", "reviewer asked a question", 9001);
|
||||
expect((await policy.evaluate(makeJob())).action).toBe("send");
|
||||
});
|
||||
|
||||
it("S2 — wake-machinery markers are NEVER material (else every wake justifies the next)", async () => {
|
||||
seedJobRow();
|
||||
seedGateQitem("q-gate-1");
|
||||
seedActivity("Stop", FRESH);
|
||||
const policy = makeIdleGateQitemPolicy({ db, agentActivityStore: store });
|
||||
expect((await policy.evaluate(makeJob())).action).toBe("send");
|
||||
|
||||
appendTransition("q-gate-1", "wake-attempt: 1/3 outcome=delivered", 9002);
|
||||
appendTransition("q-gate-1", "escalation-rung: orchestrator -> orch@rig", 9003);
|
||||
const out = await policy.evaluate(makeJob());
|
||||
expect(out.action).toBe("skip");
|
||||
expect((out as { reason: string }).reason).toBe("gate_condition_unchanged");
|
||||
});
|
||||
|
||||
it("S2 — a suppressed wake stays derivable: the job records the condition it fired for", async () => {
|
||||
// Suppressing the WAKE, never the record. The row itself is untouched and
|
||||
// still appears in held/escalations; the job carries the exact condition.
|
||||
seedJobRow();
|
||||
seedGateQitem("q-gate-1");
|
||||
seedActivity("Stop", FRESH);
|
||||
const policy = makeIdleGateQitemPolicy({ db, agentActivityStore: store });
|
||||
expect(firedCondition()).toBeNull();
|
||||
await policy.evaluate(makeJob());
|
||||
const recorded = firedCondition();
|
||||
expect(recorded).toMatch(/^[0-9a-f]{32}$/);
|
||||
await policy.evaluate(makeJob());
|
||||
expect(firedCondition()).toBe(recorded); // one overwritten value, not a ledger
|
||||
expect(
|
||||
(db.prepare("SELECT COUNT(*) AS n FROM queue_items WHERE qitem_id = 'q-gate-1'").get() as { n: number }).n,
|
||||
).toBe(1);
|
||||
});
|
||||
|
||||
it("human-gate tier (no gate:* tag) + FRESH idle → send (secondary predicate, gate:human)", async () => {
|
||||
seedGateQitem("q-human-1", { tags: null, tier: "human-gate" });
|
||||
seedActivity("Stop", FRESH);
|
||||
@@ -215,11 +332,23 @@ describe("idle-gate-qitem policy (OPR.0.4.3.16)", () => {
|
||||
expect(deliveries[0]?.targetSession).toBe(SEAT);
|
||||
expect(historyLog.listForJob(registered.jobId)[0]?.outcome).toBe("sent");
|
||||
|
||||
// Second immediate evaluation → engine active-wake throttle quiet-skips
|
||||
// (cooldown is FREE; no duplicate wake).
|
||||
// Second immediate evaluation → quiet skip, no duplicate wake.
|
||||
//
|
||||
// AMENDED by OPR.0.5.8.1 S2. This test's subject — fires once, then no
|
||||
// duplicate wake — is unchanged and still asserted by the delivery count,
|
||||
// which is what the user actually experiences. Only the REASON moved:
|
||||
// the condition gate now decides before the engine's active-wake window
|
||||
// is consulted, so the skip is `gate_condition_unchanged` rather than
|
||||
// `active_wake_not_due`.
|
||||
//
|
||||
// Asserting the new reason rather than loosening to "any skip" on
|
||||
// purpose: the two reasons are not interchangeable. `active_wake_not_due`
|
||||
// expires with the window and is bypassed entirely once a skip has
|
||||
// cleared `actionable`; `gate_condition_unchanged` holds until the gated
|
||||
// set materially changes. Which one fired is the whole repair.
|
||||
const r2 = await engine.evaluate(jobsRepo.getByIdOrThrow(registered.jobId));
|
||||
expect(r2.outcome.action).toBe("skip");
|
||||
expect((r2.outcome as { reason: string }).reason).toBe("active_wake_not_due");
|
||||
expect((r2.outcome as { reason: string }).reason).toBe("gate_condition_unchanged");
|
||||
expect(deliveries).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -412,8 +412,20 @@ describe("W2c watchdog auto-registration — production composition", () => {
|
||||
expect(deliveries[0]?.message).toContain("q-w2c-delivery");
|
||||
expect(history.listForJob(rows[0]!.jobId).map((entry) => entry.outcome)).toEqual(["sent"]);
|
||||
|
||||
// AMENDED by OPR.0.5.8.1 S2. The subject — "delivers once" — is unchanged
|
||||
// and still asserted by the delivery count below. Only the skip REASON
|
||||
// moved: the gated-condition gate now decides before the engine's
|
||||
// active-wake window is consulted. The two are not interchangeable, so
|
||||
// the new reason is asserted exactly rather than relaxed to "some skip".
|
||||
const second = await engine.evaluate(deps.watchdogJobsRepo!.getByIdOrThrow(rows[0]!.jobId));
|
||||
expect(second.outcome).toEqual({ action: "skip", reason: "active_wake_not_due" });
|
||||
expect(second.outcome).toEqual({
|
||||
action: "skip",
|
||||
reason: "gate_condition_unchanged",
|
||||
// Notes match the convention of this policy's sibling quiet skips
|
||||
// (seat_active carries them too): available to a caller inspecting the
|
||||
// outcome, while the quiet classification keeps them out of history.
|
||||
notes: { seat: sessionName, pendingGateCount: 1 },
|
||||
});
|
||||
expect(deliveries).toHaveLength(1);
|
||||
expect(autoRows(db, sessionName)).toHaveLength(1);
|
||||
|
||||
|
||||
Reference in new issue
Block a user