Files
ompweb/instrumentation.ts
mikannseandkahme247 a29b77d404 fix(runtime): recover long-idle sessions, survive terminal loss, journal crashes (#61)
* fix(runtime): recover long-idle sessions, survive terminal loss, journal crashes

A server left running while the user is away could leave the open page
unable to load anything (empty sidebar, endless Loading) until the
process was restarted, with no trace of why. Harden both sides:

- SessionSidebar: refresh the session list when the tab becomes visible
  again or the network returns — the SSE reconnect after laptop sleep /
  network change carries no list invalidation.
- useAgentSession: a CLOSED event stream now retries with capped backoff
  even while the agent is idle; previously only running sessions were
  retried, leaving idle streams dead silently until a page reload.
- process-lifecycle: translate SIGHUP into a graceful SIGTERM tree kill so
  a CLI launcher losing its terminal disposes the detached omp children
  instead of orphaning them.
- instrumentation: journal uncaughtException/unhandledRejection and
  event-loop stalls to ~/.omp/omp-web/diagnostics.log, so a crash-vs-stall
  wedge is diagnosable from disk when no terminal is watching.
- session-reader: an unsettled in-flight list load stops being coalesced
  after 60s instead of pinning the wedge until process restart.
- mcp-config: bound the synchronous git rev-parse with a 10s timeout so a
  hung spawn cannot block the entire event loop.

* fix(review): clear stale reconnect timer, gate late cache writes, fix MCP fallback caching, label sleep gaps

- useAgentSession: connectEvents drops any pending backoff timer first so a send-during-backoff cannot orphan a timer that later tears down the healthy stream; contract test pins it.
- session-reader: a watchdog-retired load can no longer overwrite fresher cache data when it resolves late (retired flag).
- mcp-config: transient git failures return the cwd fallback without poisoning the 30s project-root cache.
- instrumentation: stall watchdog compares CPU time so laptop sleep is journaled as sleep, not a misleading stall.

---------

Co-authored-by: kahme247 <[email protected]>
2026-09-13 18:01:21 +03:00

102 lines
4.9 KiB
TypeScript

import { appendFileSync, mkdirSync, renameSync, statSync } from "fs";
import { join } from "path";
import { getConfigRoot } from "@/lib/omp/paths";
export async function register(): Promise<void> {
if (process.env.NEXT_RUNTIME !== "nodejs") return;
// Honor HTTP(S)_PROXY/NO_PROXY for server-side fetch (update checks, skill
// search, model connection tests). Node's built-in fetch ignores proxy env
// vars (NODE_USE_ENV_PROXY is Node 24+ only; engines floor is 22).
const { configureHttpDispatcher } = await import("@/lib/http-dispatcher");
configureHttpDispatcher();
// Startup diagnostics: agent dir. Kept to one line so it greps cleanly;
// failures here must never block boot.
try {
const { getAgentDir } = await import("@/lib/session-reader");
console.log(
`[omp-web] starting (agent-dir ${getAgentDir()})`,
);
} catch {
// Diagnostics are best-effort.
}
// Warm the shared utility omp process so the first models/auth request does
// not pay the multi-second cold spawn (measured 1.2-4s on a real install).
// Fire-and-forget: register() must not block boot, and a missing omp binary
// is reported per-request by the routes — log once here and move on.
// The shared process registers its own SIGINT/SIGTERM/exit disposal hook on
// first use (lib/omp/rpc-utility.ts), as the session registry does.
void (async () => {
try {
const { runUtilityCommand } = await import("@/lib/omp/rpc-utility");
await runUtilityCommand({ type: "get_state" });
const { getOmpVersion } = await import("@/lib/omp/omp-cli");
const version = await getOmpVersion();
console.log(`[omp-web] omp utility ready (${version ?? "version unknown"})`);
} catch (error) {
const { resolveOmpBin } = await import("@/lib/omp/omp-cli");
const bin = resolveOmpBin();
const detail = error instanceof Error ? error.message : String(error);
const hint = bin
? `resolved ${bin}; repair with: omp update (or: bun install -g @oh-my-pi/pi-coding-agent@latest)`
: "omp binary not found; install oh-my-pi or set OMP_WEB_OMP_BIN";
console.warn(`[omp-web] omp utility warm-up failed (routes will retry on demand): ${detail} — ${hint}`);
}
})();
// Crash/stall journal: a long-running server that dies or wedges while the
// user is away leaves no trace in a terminal that no longer exists (CLI runs
// are killed with their terminal; pages then show endless loading until the
// process is restarted). Append fatal errors and event-loop stalls to a file
// so the next incident explains itself. Node's default crash semantics are
// preserved — this only adds the record before exiting.
const logDir = join(getConfigRoot(), "omp-web");
const logPath = join(logDir, "diagnostics.log");
const appendDiag = (kind: string, detail: string) => {
try {
mkdirSync(logDir, { recursive: true });
try {
if (statSync(logPath).size > 1_000_000) renameSync(logPath, `${logPath}.old`);
} catch { /* first write or unreadable — append anyway */ }
appendFileSync(logPath, `${new Date().toISOString()} [${kind}] ${detail}\n`, { encoding: "utf8" });
} catch { /* diagnostics must never crash the server */ }
};
const describe = (value: unknown) => (value instanceof Error ? `${value.name}: ${value.message}\n${value.stack ?? ""}` : String(value));
process.on("uncaughtException", (error) => {
appendDiag("crash", `uncaughtException ${describe(error)}`);
// An uncaughtException listener suppresses Node's default exit; keep the
// crash-visible semantics by exiting explicitly.
process.exit(2);
});
process.on("unhandledRejection", (reason) => {
appendDiag("crash", `unhandledRejection ${describe(reason)}`);
// Same as above: preserve Node's crash-on-unhandled-rejection default.
process.exit(2);
});
let lastTick = Date.now();
let lastCpu = process.cpuUsage();
const watchdog = setInterval(() => {
const now = Date.now();
const cpu = process.cpuUsage();
const drift = now - lastTick;
const cpuMs = (cpu.user - lastCpu.user + cpu.system - lastCpu.system) / 1000;
lastTick = now;
lastCpu = cpu;
if (drift > 45_000) {
// Wall-clock drift alone cannot tell a blocked loop from a sleeping
// machine: an hour with the lid closed looks like an hour-long stall
// but burns no CPU. Label accordingly so the journal does not mislead
// the next long-idle investigation.
const seconds = Math.round(drift / 1000);
if (cpuMs < Math.min(5_000, drift / 2)) {
appendDiag("sleep", `event loop gap of ~${seconds}s with negligible CPU time — machine was asleep/suspended or CPU-starved, not a synchronous block`);
} else {
appendDiag("stall", `event loop unresponsive for ~${seconds}s — a synchronous operation is blocking every request`);
}
}
}, 15_000);
watchdog.unref?.();
}