feat(telemetry): ACTIVITY D1+D2 — fold window_activity motion into the ACTIVITY ladder

`rig ps` reported ACTIVITY=unknown for every seat on this fleet while the web
UI showed the same seats working. Two causes, one code path in
attachAgentActivity:

  D1 (Codex -> unknown) = COVERAGE. A Codex seat has no runtime hook, so the
    positive-hook early return never fires and the verdict falls to a
    structural matcher that reasons over a fixed 8-line tail window. A tall
    footer pushes the real work line above that window, so the matcher misses
    and the seat reads unknown.

  D2 (Claude -> idle) = PRECEDENCE. A seat that Stopped and then RESUMED still
    carries the stale positive `idle` hook as its latest, and that hook was
    returned as authoritative before anything else was consulted. Live motion
    placed further down the ladder would still have lost to it.

The fix is one precedence rule applied at every exit of the ladder:

    needs_input > running > MOTION > idle / unknown

Motion is tmux `#{window_activity}` — a fact about bytes on the pane, already
computed per seat at 1Hz by SeatActivityService and already surfaced in the
separate TERMINAL column. ACTIVITY was the only consumer that never read it.
Because it is a byte-level fact it is runtime-AGNOSTIC, so it covers Codex and
Claude identically instead of needing a matcher per provider reskin.

Motion only ever UPGRADES `idle`/`unknown` to `running`. It never overrides a
`needs_input` verdict (a seat waiting at a prompt keeps redrawing, and
relabelling it `running` would step over the one seat that is actually
blocked), and it never MANUFACTURES `idle` from silence (that would satisfy
"not unknown" by relabelling the default and rot the closed union). A silent
seat's prior verdict is returned exactly as it was.

The UI has folded this same signal under the name `terminal_activity` since
slice 15 (activity-visuals.ts). The daemon now reports under that existing
name rather than minting a second word for one fact — a second vocabulary for
one signal is how the two surfaces came to disagree in the first place.

Gate: packages/daemon/test/activity-motion-fold.test.ts, RED-first. A1 (idle
hook + motion), A2 (Codex coverage) and A8 (the demoted-hook shape this fleet
actually has) were observed failing by name before the fix; the six invariant
assertions passed before and after.

Live canary on the upgraded daemon, sampled DURING the turn rather than after:
a generating Codex seat read `running` for six consecutive 2s samples and
returned to `unknown` when it stopped, while a second Codex seat sitting still
read `unknown` in those same polls as a concurrent in-band negative control.
The structural matcher caught the middle of that turn and missed both EDGES —
motion covered exactly the boundaries the diagnosis predicted.

D2's precedence path is proven at the unit gate with an observed RED but is
UNTESTABLE-ON-THIS-FLEET live: every hook here demotes to unknown for
unresolvable generation, so no seat can produce the generation-RESOLVED
positive idle hook the precedence branch requires. Recorded as a third verdict,
not as a pass.

Full daemon suite 7277 passed / 0 failed, tsc clean.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
v-openrig-buildandClaude Opus 5 committed 2026-08-11 05:08:28 +00:00
1 parent 60ee1e57be
commit 3d0fda724d
5 files changed
+321 -18

No files matched your search

+78 -14
View File
@@ -1,5 +1,5 @@
import type Database from "better-sqlite3";
import type { NodeInventoryEntry, NodeDetailEntry, NodeDetailPeer, NodeDetailEdge, NodeDetailCompactSpec, NodeRestoreOutcome, NodeOriented, NodeLifecycleState, Binding, RestoreResult, NodeRecoveryGuidance, Snapshot, WorkspaceSpec, SeatIdentityVerdict, SeatIdentityVerdictKind } from "./types.js";
import type { NodeInventoryEntry, NodeDetailEntry, NodeDetailPeer, NodeDetailEdge, NodeDetailCompactSpec, NodeRestoreOutcome, NodeOriented, NodeLifecycleState, Binding, RestoreResult, NodeRecoveryGuidance, Snapshot, WorkspaceSpec, SeatIdentityVerdict, SeatIdentityVerdictKind, AgentActivity, SeatActivity } from "./types.js";
import { identityVerdictDownranksRunning } from "./types.js";
import { SeatIdentityStore } from "./seat-identity-store.js";
import { buildOrientedMap } from "./startup-proof.js";
@@ -1066,6 +1066,15 @@ export async function attachAgentActivity(
* and capture-FREE: lifts structural motion into the ACTIVITY signal on the DEFAULT path so a
* live hook-less/stale-hook seat stops rendering as `unknown`. Absent = the pre-5b behavior. */
structuralActivity?: { getStructuralActivity(sessionName: string): StructuralObservation | null };
/** ACTIVITY D1+D2 — the cached MOTION observation (SeatActivityService, tmux
* `#{window_activity}` at 1Hz). READ-only and capture-FREE, same as `structuralActivity`.
*
* This signal was already computed per seat and already rendered — in the SEPARATE `TERMINAL`
* column, and in the UI's own activity fold (activity-visuals.ts has returned
* `{state: "running", source: "terminal_activity"}` on `terminalActive === true` since slice
* 15). ACTIVITY was the one consumer that never read it, which is why the web UI could show a
* seat working while `rig ps` called the same seat `unknown`. Absent = the pre-fix behavior. */
seatActivity?: { getSeatActivity(sessionName: string): SeatActivity | null };
now?: Date;
// OPR.0.4.3 healthz-wedge amplification fix: cheap by default. The per-node
// tmux `capturePaneContent` fallback (probeSessionActivity) is the storm that
@@ -1085,15 +1094,66 @@ export async function attachAgentActivity(
const sampledAt = deps.now ?? new Date();
const captureFallback = deps.captureFallback ?? false;
return Promise.all(entries.map(async (entry) => {
// ACTIVITY D1+D2 — the MOTION reading, resolved once and applied to every exit below.
const motion = entry.canonicalSessionName
? deps.seatActivity?.getSeatActivity(entry.canonicalSessionName) ?? null
: null;
const motionRunning = motion?.isActiveWithinWindow === true;
/**
* ACTIVITY D1+D2 — ONE precedence rule, applied at EVERY exit of the ladder:
*
* needs_input > running > MOTION > idle / unknown
*
* Live motion UPGRADES a verdict of `idle` or `unknown`; it never touches `needs_input` or an
* already-`running` verdict, and it never MANUFACTURES `idle`. That asymmetry is the whole
* safety argument, and it is what each half of the defect needs:
*
* D1 (Codex → unknown) is COVERAGE. A Codex seat has no hook, and the structural matcher is
* Claude-shaped over a fixed 8-line tail window, so a tall Codex footer pushes the real work
* line out of view and the seat falls to `unknown`. Motion is a fact about bytes on the pane,
* so it covers every runtime without a matcher per provider — the treadmill this avoids.
* D2 (Claude → idle) is PRECEDENCE. A seat that Stopped and then RESUMED still carries the
* stale positive `idle` hook as its latest, and the old early-return handed that back as
* authoritative before anything else was consulted. A perfect motion source placed further
* down the ladder would still have lost to it. Motion has to outrank a POSITIVE idle hook,
* not merely an absent one.
*
* Why `needs_input` outranks motion: motion cannot tell "working" from "waiting at a prompt" —
* any per-second redraw keeps the window fresh. If motion outranked a needs_input TEXT verdict,
* a seat asking for permission would be relabelled `running`, and the parking watch would step
* over the one seat that is actually blocked.
*
* Why motion never yields `idle`: a silent window is not evidence of a quiet agent (a seat can
* think for a long time without printing). Deriving `idle` from silence would satisfy
* "not unknown" by relabelling the default, which is the closed union rotting rather than the
* signal improving. Silence therefore leaves the prior verdict exactly as it was.
*/
const withMotion = (activity: AgentActivity): AgentActivity => {
if (!motionRunning) return activity;
if (activity.state !== "idle" && activity.state !== "unknown") return activity;
return {
state: "running",
reason: "window_activity_motion",
evidenceSource: "terminal_activity",
sampledAt: motion!.lastObservedAt,
// The RAW `#{window_activity}` timestamp that earned the verdict — a reader can age it
// against their own clock and see for themselves why this seat reads running.
evidence: motion!.lastActivityAt ?? null,
runtime: entry.runtime ?? null,
};
};
const hookActivity = deps.activityStore?.getLatestForNode({
sessionName: entry.canonicalSessionName,
now: sampledAt,
});
// A fresh POSITIVE hook (running/needs_input/idle) is authoritative — use it directly.
// A fresh POSITIVE hook (running/needs_input/idle) is authoritative — EXCEPT that a positive
// `idle` hook now yields to live motion (D2). running/needs_input hooks are untouched.
if (hookActivity && hookActivity.state !== "unknown") {
return {
...entry,
agentActivity: hookActivity,
agentActivity: withMotion(hookActivity),
};
}
@@ -1110,51 +1170,55 @@ export async function attachAgentActivity(
if (structural && structural.state !== "unknown") {
return {
...entry,
agentActivity: {
agentActivity: withMotion({
state: mapPaneState(structural.state),
reason: structural.reason,
evidenceSource: "pane_heuristic",
sampledAt: structural.observedAt,
evidence: structural.evidence,
},
}),
};
}
// No positive hook and no structural motion. A stale/unknown hook, if one exists, is delivered
// No positive hook and no structural verdict. A stale/unknown hook, if one exists, is delivered
// HONESTLY as-is (unknown/stale) — never upgraded to a quiet-seat verdict on arrival age alone.
// Live motion DOES upgrade it, and on this fleet that is the common case rather than the exotic
// one: every seat's hook currently arrives and is then demoted to unknown because the occupant
// generation cannot be resolved, so a demoted hook — not a positive idle one — is what stands
// between a working seat and a truthful label.
if (hookActivity) {
return {
...entry,
agentActivity: hookActivity,
agentActivity: withMotion(hookActivity),
};
}
if (!captureFallback) {
// CHEAP DEFAULT — no per-node tmux capture. running/idle is supplied by the
// SeatActivityService snapshot (terminalActive) at higher precedence; a
// hook-less seat's pane-heuristic needs_input requires ?full/?refresh.
// CHEAP DEFAULT — no per-node tmux capture. A hook-less seat's pane-heuristic needs_input
// still requires ?full/?refresh; running now comes from the capture-FREE motion read above,
// so the honest `unknown` placeholder is reached only by a seat that is genuinely silent.
return {
...entry,
agentActivity: {
agentActivity: withMotion({
state: "unknown",
reason: "no_runtime_hook",
evidenceSource: "session_registry",
sampledAt: sampledAt.toISOString(),
evidence: null,
fallback: true,
},
}),
};
}
return {
...entry,
agentActivity: await probeSessionActivity({
agentActivity: withMotion(await probeSessionActivity({
sessionName: entry.canonicalSessionName,
runtime: entry.runtime,
attachmentType: entry.attachmentType,
tmuxAdapter: deps.tmuxAdapter,
now: sampledAt,
}),
})),
};
}));
}
+6
View File
@@ -428,6 +428,12 @@ export type AgentActivityState = "running" | "needs_input" | "idle" | "unknown";
export type AgentActivityEvidenceSource =
| "runtime_hook"
| "pane_heuristic"
/** ACTIVITY D1+D2 — tmux `#{window_activity}` motion (SeatActivityService). A fact about BYTES on
* the pane, so it is runtime-AGNOSTIC: it sees a generating Codex seat and a generating Claude
* seat identically, where a TUI-string matcher has to be re-taught on every provider reskin. The
* UI has called this source `terminal_activity` since slice 15 (activity-visuals.ts); the daemon
* reports under the SAME name rather than minting a second word for one signal. */
| "terminal_activity"
| "tmux_session"
| "external_cli"
| "session_registry";
+6 -3
View File
@@ -323,10 +323,13 @@ rigsRoutes.get("/:id/graph", async (c) => {
// hook activity; ?full=true opts into the per-node needs_input capture.
const graphFull = c.req.query("full") === "true";
const seatStructuralActivityService = c.get("seatStructuralActivityService" as never) as SeatStructuralActivityService | undefined;
const inventoryWithActivityOnly = tmuxAdapter
? await attachAgentActivity(inventory, { tmuxAdapter, activityStore: agentActivityStore, structuralActivity: seatStructuralActivityService, captureFallback: graphFull })
: inventory;
// ACTIVITY D1+D2 — resolved BEFORE attachAgentActivity now, because the ACTIVITY ladder reads the
// same motion observation the TERMINAL column reads. Order matters only for this declaration; the
// two enrichments still read their own sources independently.
const seatActivityService = c.get("seatActivityService" as never) as SeatActivityService | undefined;
const inventoryWithActivityOnly = tmuxAdapter
? await attachAgentActivity(inventory, { tmuxAdapter, activityStore: agentActivityStore, structuralActivity: seatStructuralActivityService, seatActivity: seatActivityService, captureFallback: graphFull })
: inventory;
const inventoryWithActivity = attachTerminalActivityAndWork(inventoryWithActivityOnly, {
db: getRepo(c).db,
seatActivity: seatActivityService,
+5 -1
View File
@@ -122,6 +122,10 @@ nodesRoutes.get("/", async (c) => {
tmuxAdapter: deps.tmuxAdapter,
activityStore: deps.agentActivityStore,
structuralActivity: deps.seatStructuralActivityService,
// ACTIVITY D1+D2 — the SAME SeatActivityService instance `attachTerminalActivityAndWork` reads
// just below. One observation, now serving both the TERMINAL column and the ACTIVITY verdict,
// so the two columns can no longer disagree about whether a seat is producing output.
seatActivity: deps.seatActivityService,
captureFallback: full,
});
const withTerminalAndWork = attachTerminalActivityAndWork(withActivity, {
@@ -145,7 +149,7 @@ nodesRoutes.get("/:logicalId", async (c) => {
if (!detail) return c.json({ error: `Node "${logicalId}" not found in rig "${rigId}". Check node IDs with: rig ps --nodes` }, 404);
// Node DETAIL is a single node — the per-node tmux capture is cheap here, so
// detail always runs the full fallback (freshest needs_input for the one seat).
const [detailWithActivity] = await attachAgentActivity([detail], { tmuxAdapter: deps.tmuxAdapter, activityStore: deps.agentActivityStore, captureFallback: true });
const [detailWithActivity] = await attachAgentActivity([detail], { tmuxAdapter: deps.tmuxAdapter, activityStore: deps.agentActivityStore, seatActivity: deps.seatActivityService, captureFallback: true });
const [detailWithTerminalAndWork] = attachTerminalActivityAndWork(detailWithActivity ? [detailWithActivity] : [detail], {
db: deps.rigRepo.db,
seatActivity: deps.seatActivityService,
@@ -0,0 +1,226 @@
// ACTIVITY D1+D2 — fold the window_activity MOTION signal into the ACTIVITY ladder. RED-first.
//
// TWO CAUSES, ONE CODE PATH (attachAgentActivity, node-inventory.ts):
// D1 (Codex → unknown) = COVERAGE. A Codex seat has no hook, so the positive-hook early return
// never fires, and the cached STRUCTURAL verdict comes from a Claude-shaped matcher over a fixed
// 8-line tail window. Codex's tall footer pushes `◦ Working (… esc to interrupt)` above that
// window → no match → unknown. A generating seat reports nothing.
// D2 (Claude → idle) = PRECEDENCE. A positive `idle` hook early-returns as authoritative, so live
// motion is NEVER consulted. A Claude seat that Stopped then RESUMED still carries the stale idle
// hook as latest → reports idle WHILE GENERATING. A perfect motion source alone does not fix this.
//
// THE FIX = a runtime-agnostic MOTION source (tmux `#{window_activity}`, already computed per seat by
// SeatActivityService and already surfaced as `terminalActive` in a SEPARATE column) folded into the
// ACTIVITY ladder, ordered `needs_input(TEXT) > motion > idle`. Motion can only ever UPGRADE to
// running; it never manufactures `idle`, so a hook-less MOTIONLESS seat still reads `unknown` and the
// closed union does not rot.
//
// WHY MOTION AND NOT A BETTER MATCHER: a per-runtime TUI matcher is a treadmill — every provider
// reskin re-breaks it. `window_activity` is a tmux fact about bytes on the pane, identical for Claude,
// Codex, and anything else we ever seat.
//
// LIVE MEASUREMENT BEHIND THIS DESIGN (daemon route, reverted runtime cb662b9d, 2026-08-11 04:52Z):
// terminalActive was TRUE for exactly the two seats generating at that instant (dev-driver,
// dev50-driver) and FALSE for the other 12 — including every Codex seat, whose tall footers do NOT
// keep the window fresh. The motion source discriminates live and is not sprayed by clock-tick
// redraws, which was the design's main risk.
//
// NOTE ON THE NEGATIVE CONTROL: these unit assertions are NOT the gate on their own. The reverted
// runtime is uniformly inert (all 14 seats unknown, reason=generation_unverifiable), so D2 cannot be
// STAGED there — assertions that look satisfied against it can be vacuous. The live control is staged
// separately by deliberately reintroducing the precedence and watching A1 fail BY NAME.
import { describe, it, expect } from "vitest";
import { attachAgentActivity } from "../src/domain/node-inventory.js";
import type { AgentActivity } from "../src/domain/types.js";
function mkTmux(counter: { captures: number }) {
return {
hasSession: async () => true,
getPaneCommand: async () => "claude",
capturePaneContent: async () => {
counter.captures++;
return "some pane output\n> awaiting input";
},
} as never;
}
function entries(sessionName: string, runtime = "claude-code") {
return [{ canonicalSessionName: sessionName, runtime, attachmentType: "tmux", logicalId: "dev.impl" }] as never;
}
/** The cached MOTION observation, as SeatActivityService exposes it. A cache read, never a capture. */
function mkMotion(isActiveWithinWindow: boolean | null) {
return {
getSeatActivity: () =>
isActiveWithinWindow === null
? null
: {
paneId: "s@rig",
isActiveWithinWindow,
silenceWindowSeconds: 3,
lastObservedAt: "2026-08-11T04:52:00.000Z",
lastActivityAt: "2026-08-11T04:51:59.000Z",
},
} as never;
}
function mkStructural(state: "agent_active" | "agent_idle" | "attention" | "unknown" | null) {
return {
getStructuralActivity: () =>
state
? { state, reason: `structural_${state}`, evidence: "pane", observedAt: "2026-08-11T04:52:00.000Z" }
: null,
} as never;
}
function hook(state: AgentActivity["state"], reason: string, extra: Partial<AgentActivity> = {}): AgentActivity {
return {
state,
reason,
evidenceSource: "runtime_hook",
sampledAt: "2026-08-11T04:30:00.000Z",
evidence: null,
...extra,
};
}
const store = (activity: AgentActivity | null) => ({ getLatestForNode: () => activity }) as never;
async function activityOf(deps: Record<string, unknown>, runtime = "claude-code"): Promise<AgentActivity> {
const out = (await attachAgentActivity(entries("s@rig", runtime), deps as never)) as Array<{
agentActivity: AgentActivity;
}>;
return out[0]!.agentActivity;
}
describe("ACTIVITY D1+D2 — motion folded into the ladder", () => {
it("A1 [D2 precedence] a positive IDLE hook + LIVE motion must NOT report idle — it reports running", async () => {
const counter = { captures: 0 };
const activity = await activityOf({
tmuxAdapter: mkTmux(counter),
activityStore: store(hook("idle", "stop_hook")),
structuralActivity: mkStructural(null),
seatActivity: mkMotion(true),
});
// The assertion the defect fails BY NAME: a generating Claude seat reporting idle is a false
// ASSERTION of quiet, strictly worse than an honest unknown — the parking watch reports all-clear
// over a working seat.
expect(activity.state).not.toBe("idle");
expect(activity.state).toBe("running");
expect(counter.captures).toBe(0); // cache read, never a per-request capture (healthz-wedge invariant)
});
it("A2 [D1 coverage] a hook-less seat whose structural matcher MISSES + LIVE motion reports running", async () => {
// The Codex shape: no hook at all, and the Claude-shaped structural matcher returns unknown
// because the tall footer pushed the work line out of the 8-line tail window.
const activity = await activityOf(
{
tmuxAdapter: mkTmux({ captures: 0 }),
activityStore: store(null),
structuralActivity: mkStructural("unknown"),
seatActivity: mkMotion(true),
},
"codex",
);
// RUNNING exactly — "not unknown" is the indicator, and it passes for a generating Codex seat that
// reads idle (D2 on Codex). Assert the correctness standard, not its shadow.
expect(activity.state).toBe("running");
});
it("A3 [control] a hook-less MOTIONLESS seat still reports unknown — the fix must not relabel the default", async () => {
const activity = await activityOf({
tmuxAdapter: mkTmux({ captures: 0 }),
activityStore: store(null),
structuralActivity: mkStructural("unknown"),
seatActivity: mkMotion(false),
});
expect(activity.state).toBe("unknown");
});
it("A3b [honest absence] no motion observation at all → unknown, never a quiet-seat verdict", async () => {
const noObs = await activityOf({
tmuxAdapter: mkTmux({ captures: 0 }),
activityStore: store(null),
structuralActivity: mkStructural("unknown"),
seatActivity: mkMotion(null),
});
expect(noObs.state).toBe("unknown");
// And with the dep entirely absent (older wiring), behavior is unchanged.
const noDep = await activityOf({
tmuxAdapter: mkTmux({ captures: 0 }),
activityStore: store(null),
structuralActivity: mkStructural("unknown"),
});
expect(noDep.state).toBe("unknown");
});
it("A4 [order] needs_input TEXT outranks motion — a seat waiting at a prompt keeps redrawing", async () => {
// Motion cannot tell "waiting at a prompt" from "working": any per-second redraw keeps the window
// fresh. So a positive needs_input verdict — from the hook OR from the structural text read — must
// survive live motion, or the fix converts "answer me" into "busy, leave it alone".
const fromHook = await activityOf({
tmuxAdapter: mkTmux({ captures: 0 }),
activityStore: store(hook("needs_input", "permission_prompt")),
structuralActivity: mkStructural(null),
seatActivity: mkMotion(true),
});
expect(fromHook.state).toBe("needs_input");
const fromStructural = await activityOf({
tmuxAdapter: mkTmux({ captures: 0 }),
activityStore: store(null),
structuralActivity: mkStructural("attention"),
seatActivity: mkMotion(true),
});
expect(fromStructural.state).toBe("needs_input");
});
it("A5 [unchanged] a positive RUNNING hook keeps its authority with no motion observation", async () => {
const activity = await activityOf({
tmuxAdapter: mkTmux({ captures: 0 }),
activityStore: store(hook("running", "prompt_submit")),
structuralActivity: mkStructural(null),
seatActivity: mkMotion(null),
});
expect(activity.state).toBe("running");
expect(activity.evidenceSource).toBe("runtime_hook");
});
it("A6 [no fabrication] a MOTIONLESS seat with a positive idle hook still reads idle", async () => {
// The mirror of A1: motion only ever upgrades. Absent motion, the idle hook stands, so the fix
// cannot be accused of laundering every idle seat into running.
const activity = await activityOf({
tmuxAdapter: mkTmux({ captures: 0 }),
activityStore: store(hook("idle", "stop_hook")),
structuralActivity: mkStructural(null),
seatActivity: mkMotion(false),
});
expect(activity.state).toBe("idle");
});
it("A7 [no fabrication] a stale/unknown hook + no motion is still delivered honestly, never idle", async () => {
const activity = await activityOf({
tmuxAdapter: mkTmux({ captures: 0 }),
activityStore: store(hook("unknown", "generation_unverifiable", { stale: true })),
structuralActivity: mkStructural("unknown"),
seatActivity: mkMotion(false),
});
expect(activity.state).toBe("unknown");
expect(activity.reason).toBe("generation_unverifiable");
});
it("A8 [the live fleet shape] an unverifiable-generation hook + LIVE motion reports running", async () => {
// Measured live: all 14 seats carry state=unknown reason=generation_unverifiable, so on this fleet
// the demoted hook — not a positive idle one — is what stands between a generating seat and a
// truthful label. Motion must beat it, or the fix changes nothing on the machine it ships to.
const activity = await activityOf({
tmuxAdapter: mkTmux({ captures: 0 }),
activityStore: store(hook("unknown", "generation_unverifiable", { stale: true })),
structuralActivity: mkStructural("unknown"),
seatActivity: mkMotion(true),
});
expect(activity.state).toBe("running");
});
});