mirror of
https://github.com/mvschwarz/openrig.git
synced 2026-11-11 11:41:38 -10:00
Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c85623c299 | ||
|
|
002e4451e2 | ||
|
|
076c851227 | ||
|
|
66a0cc7e69 | ||
|
|
40ccca1fb1 | ||
|
|
95a9a85a7f | ||
|
|
d547b32dad |
No files matched your search
@@ -0,0 +1,114 @@
|
||||
# Network connector: a local seat as a hub participant
|
||||
|
||||
Source of truth: `packages/daemon/src/domain/network/` (`network-bindings.ts`, `network-connector.ts`, `hub-client.ts`),
|
||||
`packages/daemon/src/routes/network.ts`, `packages/cli/src/commands/network.ts`.
|
||||
|
||||
A hub is another OpenRig queue that its participants reach over MCP, such as a daemon served by `rig mcp serve --http`
|
||||
with one grant per participant. The network connector puts one selected local seat on a hub as one of its participants.
|
||||
The connector opens no port: your daemon makes every connection outward, and the hub never connects to your machine. Work the hub
|
||||
sends that participant runs on your seat, under your accounts, and the result goes back to the hub item.
|
||||
|
||||
## Connect
|
||||
|
||||
```bash
|
||||
rig network connect <seat> --hub <url> --as <participant> --token-file <path>
|
||||
```
|
||||
|
||||
- `<participant>` is the participant's address in the hub (`member@rig`), and the token file holds its bearer.
|
||||
- The binding follows the seat's node, so a relaunched or renamed seat keeps it.
|
||||
- The token is stored at `<OpenRig home>/network-secrets/<binding>/credential.json` (a 0700 folder and a 0600 file). It
|
||||
is never returned by the API or shown in status. A token in the hub URL is refused.
|
||||
- A plain-http hub URL needs `--allow-http` (see Limits).
|
||||
- One active binding per local seat, and one per hub participant.
|
||||
- On a daemon with an operator bearer configured, `connect` and `disconnect` send `OPENRIG_AUTH_BEARER_TOKEN`, as the
|
||||
`rig host` writes do.
|
||||
|
||||
## What the connector does
|
||||
|
||||
The daemon runs the connector for every active binding. It normally polls every 30 seconds
|
||||
(`OPENRIG_NETWORK_POLL_SECONDS`, at least 5); a failing hub, backoff or a backlog can delay it. Each binding's pass has
|
||||
a deadline of 60 seconds, so one slow hub doesn't hold the others. With no active binding it makes no network call.
|
||||
|
||||
1. It lists the participant's own items on the hub, claims each pending one, and creates a local queue item for the
|
||||
seat's current address. The local item comes from OpenRig itself, and it says where the work came from first: its
|
||||
summary starts "From <requester> via network hub", and its body opens with the requester, the hub and the hub item,
|
||||
before the requester's own words.
|
||||
2. It reports each stage on the hub item, kept apart:
|
||||
- **Queued:** the hub claim means the connector holds the item. The note says "Not picked up yet."
|
||||
- **Picked up:** once the seat claims the local item.
|
||||
- **The result:** once the local item closes, following any local handoffs to the item that finished. The hub item
|
||||
closes the way its queue closes one: done with the resolution and the closure reason (`no-follow-on`, `canceled`
|
||||
or `denied`), or failed when the local item failed. A close with no resolution sends the closer's own note on that
|
||||
close, or "Closed without a stated result." No other note on the item is sent.
|
||||
|
||||
To answer, close the local item with your result:
|
||||
|
||||
```bash
|
||||
rig queue update <local item> --state done --closure-reason no-follow-on --resolution "<your answer>"
|
||||
```
|
||||
|
||||
## Retries, restarts and unknown outcomes
|
||||
|
||||
- Each hub item maps to one local item, under an id derived from the seat's node, the hub, the participant and the hub
|
||||
item. A lost answer, a retry, a daemon restart, or disconnecting and connecting the same seat to the same hub and
|
||||
participant again lands on that same item, so it isn't queued twice. A new binding reports what the earlier one
|
||||
took.
|
||||
- If the local record of an item is gone, or it was handed on without one successor here, the hub item says its outcome
|
||||
is unknown, and it is never run again.
|
||||
- If someone else closed the hub item first (its requester canceled it, say), the local result stays here, the connector
|
||||
doesn't count it as delivered, and `rig network status` (and its `--json`) shows it under the binding as closed
|
||||
elsewhere. The connector checks the hub item just before closing it. It does not write over a close it has already
|
||||
seen, and reads the item back after a close attempt. A hub item someone there handed on counts as closed elsewhere
|
||||
too.
|
||||
- The derived id makes the queue item unique, not the work's outside effects. Anything your agent did outside OpenRig
|
||||
before a crash is neither undone nor deduplicated.
|
||||
|
||||
## Status
|
||||
|
||||
```bash
|
||||
rig network status [--json]
|
||||
```
|
||||
|
||||
Each binding shows its seat, participant, hub, state, and its last contact or last error. `--json` adds the
|
||||
connector's view: each binding's next pass, backoff and last pass.
|
||||
|
||||
## When things go wrong
|
||||
|
||||
- **The seat has no session:** its hub items stay pending and unclaimed, and the last error says why. They're taken
|
||||
once the seat has a session again. A stopped seat still has its session, so it still takes items; they wait for it.
|
||||
- **The hub can't be reached, or a pass runs past its deadline:** the binding stays active, the error is recorded, and
|
||||
retries back off, doubling up to 5 minutes. A failure on this machine is recorded as local, not as the hub's.
|
||||
- **The hub refuses the credential** (HTTP 401): the binding is marked revoked and its credential removed. No new work
|
||||
comes in; local items and results stay.
|
||||
|
||||
## Disconnect
|
||||
|
||||
```bash
|
||||
rig network disconnect <binding-or-seat> [--reason "<why>"]
|
||||
```
|
||||
|
||||
The binding ends here, so no new work comes in through it, and its credential is removed. Local items and results
|
||||
stay. The hub still lists the participant until it's removed there too.
|
||||
|
||||
If the credential can't be removed, the binding still ends, and the error names the
|
||||
`rig network disconnect <binding id>` to run again. The daemon also removes credentials whose binding isn't active
|
||||
each time it starts.
|
||||
|
||||
## Limits
|
||||
|
||||
- The connector polls, so new work normally waits up to one interval before it reaches the seat, and longer when the
|
||||
hub fails or a backlog builds.
|
||||
- If someone else closes a hub item between the connector's check and its close, the hub may keep either outcome, or
|
||||
combine the later state with the earlier resolution. The connector counts its result as delivered only when the
|
||||
readback matches its state, closure reason and resolution; otherwise a closed item is reported as closed elsewhere.
|
||||
An atomic close-if-open needs the hub to offer a conditional close.
|
||||
- After a disconnect or a revocation, work already taken keeps running here, but its pickup and result no longer reach
|
||||
the hub item, which stays at its last reported stage.
|
||||
- **Plain http needs your choice.** `rig network connect` refuses an `http:` hub URL until you add `--allow-http`, because
|
||||
the participant's token and its work then cross the network unencrypted, where they can be read. Use the hub's
|
||||
`https:` URL where it has one. The choice is recorded on that binding, so the connector doesn't ask again, and
|
||||
`rig network status` (and its `--json`) marks the binding unencrypted. OpenRig doesn't judge the address: a private or
|
||||
tailnet address doesn't prove the connection is protected.
|
||||
- **Redirects:** the connector follows a hub's redirect only within its own origin, or to its `https:` form on the same
|
||||
host. Any other redirect stops the connection with its target named, and the token isn't sent there. Connect the
|
||||
binding to the hub's own URL.
|
||||
Generated
+1
@@ -7341,6 +7341,7 @@
|
||||
"dependencies": {
|
||||
"@hono/node-server": "^1.13.0",
|
||||
"@hono/node-ws": "^1.3.1",
|
||||
"@modelcontextprotocol/sdk": "^1.32.2",
|
||||
"better-sqlite3": "^13.0.3",
|
||||
"hono": "^4.7.0",
|
||||
"smol-toml": "^1.8.0",
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
import { Command } from "commander";
|
||||
import { lstatSync, readFileSync } from "node:fs";
|
||||
import { DaemonClient, operatorAuthHeaders } from "../client.js";
|
||||
import { getDaemonStatus, getDaemonUrl, daemonStatusGuard, type LifecycleDeps } from "../daemon-lifecycle.js";
|
||||
import { realDeps } from "./daemon.js";
|
||||
|
||||
export interface NetworkDeps {
|
||||
lifecycleDeps: LifecycleDeps;
|
||||
clientFactory: (url: string) => Pick<DaemonClient, "get" | "post">;
|
||||
readTokenFile?: (path: string) => string;
|
||||
}
|
||||
|
||||
interface Binding {
|
||||
id: string; localSession: string; hubUrl: string; hubSeat: string; state: string; stateReason: string | null;
|
||||
lastContactAt: string | null; lastError: string | null; unencrypted?: boolean;
|
||||
/** Hub items someone else closed before this binding's result reached them. */
|
||||
closedElsewhere?: Array<{ hubItem: string; localItem: string }>;
|
||||
}
|
||||
|
||||
/** The participant's token, read from the file the account gave the person. A file others can read is used, with a note. */
|
||||
function defaultReadTokenFile(path: string): string {
|
||||
const stat = lstatSync(path);
|
||||
if (!stat.isFile()) throw new Error(`${path} isn't a regular file.`);
|
||||
if ((stat.mode & 0o077) !== 0) console.error(`Note: ${path} can be read by other users of this machine; OpenRig keeps its own private copy.`);
|
||||
return readFileSync(path, "utf8").trim();
|
||||
}
|
||||
|
||||
function line(b: Binding): string {
|
||||
const contact = b.lastContactAt ? `last contact ${b.lastContactAt}` : "no contact yet";
|
||||
const state = b.state === "active" ? contact : `${b.state}${b.stateReason ? ` (${b.stateReason})` : ""}`;
|
||||
const plain = b.unencrypted ? " UNENCRYPTED (http, allowed for this binding)" : "";
|
||||
const conflicts = (b.closedElsewhere ?? []).map((c) => `\n hub item ${c.hubItem} was closed elsewhere before its result arrived; the result stays on ${c.localItem}`).join("");
|
||||
return `${b.id} ${b.localSession} -> ${b.hubSeat} at ${b.hubUrl}${plain} ${state}${b.lastError && b.state === "active" ? ` last error: ${b.lastError}` : ""}${conflicts}`;
|
||||
}
|
||||
|
||||
/** `rig network`: connect a selected local seat to a participant of a remote hub (an account's network rig, or
|
||||
* another OpenRig hub), see where each connection stands, and disconnect one. */
|
||||
export function networkCommand(depsOverride?: NetworkDeps): Command {
|
||||
const cmd = new Command("network")
|
||||
.description("Connect selected local seats to a remote hub, see their status, and disconnect them");
|
||||
const getDeps = (): NetworkDeps => depsOverride ?? { lifecycleDeps: realDeps(), clientFactory: (url: string) => new DaemonClient(url) };
|
||||
|
||||
const client = async () => {
|
||||
const deps = getDeps();
|
||||
const daemon = await getDaemonStatus(deps.lifecycleDeps);
|
||||
if (!daemonStatusGuard(daemon)) return null;
|
||||
return deps.clientFactory(getDaemonUrl(daemon));
|
||||
};
|
||||
const finish = (status: number, data: unknown, json: boolean | undefined, ok: (data: Record<string, unknown>) => string) => {
|
||||
if (status >= 400) {
|
||||
const error = data as { error?: string; message?: string };
|
||||
console.error(json ? JSON.stringify(data) : `${error.message ?? "The daemon refused this"} (${error.error ?? status})`);
|
||||
process.exitCode = status >= 500 ? 2 : 1;
|
||||
return;
|
||||
}
|
||||
console.log(json ? JSON.stringify(data, null, 2) : ok(data as Record<string, unknown>));
|
||||
};
|
||||
|
||||
cmd.command("connect")
|
||||
.description("Connect a local seat to a hub participant, using that participant's token")
|
||||
.argument("<seat>", "The local seat to connect (its address, member@rig)")
|
||||
.requiredOption("--hub <url>", "The hub's MCP URL")
|
||||
.requiredOption("--as <participant>", "The participant's address in the hub (member@rig)")
|
||||
.requiredOption("--token-file <path>", "A file holding the participant's token")
|
||||
.option("--allow-http", "Accept a plain-http hub for this binding: its token and work cross the network unencrypted")
|
||||
.option("--json", "Print JSON")
|
||||
.action(async (seat: string, opts: { hub: string; as: string; tokenFile: string; allowHttp?: boolean; json?: boolean }) => {
|
||||
let token: string;
|
||||
try { token = (getDeps().readTokenFile ?? defaultReadTokenFile)(opts.tokenFile); } catch (error) {
|
||||
console.error(`Can't read the token file: ${(error as Error).message}`);
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
const daemon = await client();
|
||||
if (!daemon) return;
|
||||
const res = await daemon.post<Record<string, unknown>>("/api/network/bindings", { session: seat, hubUrl: opts.hub, hubSeat: opts.as, token, allowHttp: opts.allowHttp === true }, { headers: operatorAuthHeaders() });
|
||||
finish(res.status, res.data, opts.json, (b) => `Connected ${line(b as unknown as Binding)}\nWork for ${opts.as} reaches ${seat} once the connector runs; rig network status shows its last contact.`);
|
||||
});
|
||||
|
||||
cmd.command("status")
|
||||
.description("List the connections and where each stands")
|
||||
.option("--json", "Print JSON")
|
||||
.action(async (opts: { json?: boolean }) => {
|
||||
const daemon = await client();
|
||||
if (!daemon) return;
|
||||
const res = await daemon.get<{ bindings: Binding[] }>("/api/network/bindings");
|
||||
finish(res.status, res.data, opts.json, (d) => {
|
||||
const bindings = (d.bindings ?? []) as Binding[];
|
||||
return bindings.length ? bindings.map(line).join("\n") : "No seats are connected to a hub.";
|
||||
});
|
||||
});
|
||||
|
||||
cmd.command("disconnect")
|
||||
.description("Disconnect a seat: no new work comes in through it; its local work and results stay")
|
||||
.argument("<binding-or-seat>", "The binding id, or the connected seat's address")
|
||||
.option("--reason <text>", "Why, for the record")
|
||||
.option("--json", "Print JSON")
|
||||
.action(async (ref: string, opts: { reason?: string; json?: boolean }) => {
|
||||
const daemon = await client();
|
||||
if (!daemon) return;
|
||||
const res = await daemon.post<Record<string, unknown>>(`/api/network/bindings/${encodeURIComponent(ref)}/disconnect`, { reason: opts.reason }, { headers: operatorAuthHeaders() });
|
||||
finish(res.status, res.data, opts.json, (b) => `Disconnected ${line(b as unknown as Binding)}\nRemove the participant on the hub too, so the account stops listing it.`);
|
||||
});
|
||||
|
||||
return cmd;
|
||||
}
|
||||
@@ -91,6 +91,7 @@ import { compactPlanCommand, type CompactPlanDeps } from "./commands/compact-pla
|
||||
import { compactCommand, type CompactDeps } from "./commands/compact.js";
|
||||
import { heartbeatCommand, type HeartbeatDeps } from "./commands/heartbeat.js";
|
||||
import { seatCommand, handoverCommand, type SeatDeps } from "./commands/seat.js";
|
||||
import { networkCommand, type NetworkDeps } from "./commands/network.js";
|
||||
import { rigModeCommand, type RigModeDeps } from "./commands/rig-mode.js";
|
||||
import { policyCommand } from "./commands/policy.js";
|
||||
import { startupProofCommand, type StartupProofDeps } from "./commands/startup-proof.js";
|
||||
@@ -168,6 +169,7 @@ export interface ProgramDeps {
|
||||
compactDeps?: CompactDeps;
|
||||
heartbeatDeps?: HeartbeatDeps;
|
||||
seatDeps?: SeatDeps;
|
||||
networkDeps?: NetworkDeps;
|
||||
rigModeDeps?: RigModeDeps;
|
||||
startupProofDeps?: StartupProofDeps;
|
||||
healthDeps?: HealthDeps;
|
||||
@@ -269,6 +271,7 @@ export function createProgram(depsOverride?: ProgramDeps): Command {
|
||||
program.addCommand(heartbeatCommand(depsOverride?.heartbeatDeps));
|
||||
program.addCommand(seatCommand(depsOverride?.seatDeps));
|
||||
program.addCommand(handoverCommand(depsOverride?.seatDeps));
|
||||
program.addCommand(networkCommand(depsOverride?.networkDeps));
|
||||
program.addCommand(startupProofCommand(depsOverride?.startupProofDeps));
|
||||
// release-0.3.2 slice 12 — rig scope CLI primitive.
|
||||
program.addCommand(scopeCommand());
|
||||
|
||||
@@ -102,6 +102,7 @@ export const COMMAND_LOADERS: ReadonlyMap<string, Load> = new Map<string, Load>(
|
||||
["heartbeat", async () => (await import("./commands/heartbeat.js")).heartbeatCommand()],
|
||||
["seat", async () => (await import("./commands/seat.js")).seatCommand()],
|
||||
["handover", async () => (await import("./commands/seat.js")).handoverCommand()],
|
||||
["network", async () => (await import("./commands/network.js")).networkCommand()],
|
||||
["startup-proof", async () => (await import("./commands/startup-proof.js")).startupProofCommand()],
|
||||
["scope", async () => (await import("./commands/scope.js")).scopeCommand()],
|
||||
["proof", async () => (await import("./commands/proof.js")).proofCommand()],
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
// `rig network connect | status | disconnect` against the daemon's real network routes.
|
||||
import { afterEach, beforeEach, expect, it, vi } from "vitest";
|
||||
import { Command } from "commander";
|
||||
import { Hono } from "hono";
|
||||
import Database from "better-sqlite3";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { networkCommand, type NetworkDeps } from "../src/commands/network.js";
|
||||
import { STATE_FILE, type LifecycleDeps } from "../src/daemon-lifecycle.js";
|
||||
import { ALL_MIGRATIONS } from "../../daemon/src/db/all-migrations.js";
|
||||
import { migrate } from "../../daemon/src/db/migrate.js";
|
||||
import { RigRepository } from "../../daemon/src/domain/rig-repository.js";
|
||||
import { networkRoutes } from "../../daemon/src/routes/network.js";
|
||||
|
||||
let db: Database.Database;
|
||||
let home: string;
|
||||
let app: Hono;
|
||||
let rigs: RigRepository;
|
||||
const routesWith = (bearerToken: string | null) => {
|
||||
const next = new Hono();
|
||||
next.use("*", async (c, n) => { c.set("rigRepo" as never, rigs as never); await n(); });
|
||||
next.route("/api/network", networkRoutes({ home, bearerToken }));
|
||||
return next;
|
||||
};
|
||||
beforeEach(() => {
|
||||
db = new Database(":memory:");
|
||||
migrate(db, ALL_MIGRATIONS);
|
||||
rigs = new RigRepository(db);
|
||||
const node = rigs.addNode(rigs.createRig("lab").id, "impl", { runtime: "claude-code" });
|
||||
db.prepare("INSERT INTO sessions (id, node_id, session_name, status) VALUES ('s1', ?, 'dev-impl@lab', 'running')").run(node.id);
|
||||
home = mkdtempSync(join(tmpdir(), "openrig-network-cli-"));
|
||||
app = routesWith(null);
|
||||
});
|
||||
afterEach(() => { db.close(); rmSync(home, { recursive: true, force: true }); vi.restoreAllMocks(); vi.unstubAllEnvs(); process.exitCode = undefined; });
|
||||
|
||||
function cli(tokenFile: Record<string, string> = {}) {
|
||||
const lifecycleDeps: LifecycleDeps = {
|
||||
spawn: vi.fn(), fetch: vi.fn(async () => ({ ok: true })), kill: vi.fn(),
|
||||
readFile: (p) => p === STATE_FILE ? JSON.stringify({ pid: 123, port: 19731, db: "/fixture/test.sqlite", startedAt: new Date().toISOString() }) : null,
|
||||
writeFile: vi.fn(), removeFile: vi.fn(), exists: (p) => p === STATE_FILE, mkdirp: vi.fn(), openForAppend: () => 3, isProcessAlive: () => true,
|
||||
} as LifecycleDeps;
|
||||
const call = async (method: "GET" | "POST", url: string, body?: unknown, headers: Record<string, string> = {}) => {
|
||||
const res = await app.request(url, { method, headers: { "Content-Type": "application/json", ...headers }, ...(body === undefined ? {} : { body: JSON.stringify(body) }) });
|
||||
return { status: res.status, data: await res.json() };
|
||||
};
|
||||
const deps: NetworkDeps = {
|
||||
lifecycleDeps,
|
||||
clientFactory: () => ({ get: (url: string) => call("GET", url), post: (url: string, body: unknown, options?: { headers?: Record<string, string> }) => call("POST", url, body, options?.headers) }) as never,
|
||||
readTokenFile: (path) => { if (!(path in tokenFile)) throw new Error("no such file"); return tokenFile[path]!; },
|
||||
};
|
||||
return new Command().exitOverride().addCommand(networkCommand(deps));
|
||||
}
|
||||
|
||||
it("connects a seat with its token file, shows it in status without the token, and disconnects it", async () => {
|
||||
const out = vi.spyOn(console, "log").mockImplementation(() => {});
|
||||
await cli({ "/tok": "tok-abc" }).parseAsync(["node", "rig", "network", "connect", "dev-impl@lab", "--hub", "https://hub.example/mcp", "--as", "alex@team", "--token-file", "/tok"]);
|
||||
expect(String(out.mock.calls.at(-1)?.[0])).toMatch(/^Connected netb-[0-9a-f]{20} dev-impl@lab -> alex@team at https:\/\/hub\.example\/mcp no contact yet/);
|
||||
|
||||
await cli().parseAsync(["node", "rig", "network", "status", "--json"]);
|
||||
const status = String(out.mock.calls.at(-1)?.[0]);
|
||||
expect(JSON.parse(status).bindings).toEqual([expect.objectContaining({ localSession: "dev-impl@lab", hubSeat: "alex@team", state: "active" })]);
|
||||
expect(status).not.toContain("tok-abc");
|
||||
|
||||
await cli().parseAsync(["node", "rig", "network", "disconnect", "dev-impl@lab", "--reason", "done"]);
|
||||
expect(String(out.mock.calls.at(-1)?.[0])).toMatch(/^Disconnected netb-[0-9a-f]{20} dev-impl@lab -> alex@team .*disconnected \(done\)/);
|
||||
expect(process.exitCode).toBeUndefined();
|
||||
});
|
||||
|
||||
it("names the daemon's refusal and exits 1, and an unreadable token file sends nothing", async () => {
|
||||
const err = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||
await cli({ "/tok": "tok-abc" }).parseAsync(["node", "rig", "network", "connect", "nobody@lab", "--hub", "https://hub.example/mcp", "--as", "alex@team", "--token-file", "/tok"]);
|
||||
expect(String(err.mock.calls.at(-1)?.[0])).toBe("No local seat is named 'nobody@lab'. (seat_not_found)");
|
||||
expect(process.exitCode).toBe(1);
|
||||
process.exitCode = undefined;
|
||||
await cli().parseAsync(["node", "rig", "network", "connect", "dev-impl@lab", "--hub", "https://hub.example/mcp", "--as", "alex@team", "--token-file", "/missing"]);
|
||||
expect(String(err.mock.calls.at(-1)?.[0])).toBe("Can't read the token file: no such file");
|
||||
expect(process.exitCode).toBe(1);
|
||||
expect((await app.request("/api/network/bindings").then((r) => r.json()) as { bindings: unknown[] }).bindings).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("a plain-http hub needs --allow-http: refused with the remedy, then connected and shown unencrypted in status and JSON", async () => {
|
||||
const out = vi.spyOn(console, "log").mockImplementation(() => {});
|
||||
const err = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||
const args = ["node", "rig", "network", "connect", "dev-impl@lab", "--hub", "http://192.0.2.7:7433/mcp", "--as", "alex@team", "--token-file", "/tok"];
|
||||
await cli({ "/tok": "tok-abc" }).parseAsync(args);
|
||||
expect(process.exitCode).toBe(1);
|
||||
expect(String(err.mock.calls.at(-1)?.[0])).toMatch(/plain http, .*unencrypted.*--allow-http to accept that for this binding\. \(http_not_allowed\)$/);
|
||||
process.exitCode = undefined;
|
||||
|
||||
await cli({ "/tok": "tok-abc" }).parseAsync([...args, "--allow-http"]);
|
||||
expect(process.exitCode).toBeUndefined();
|
||||
expect(String(out.mock.calls.at(-1)?.[0])).toMatch(/^Connected netb-[0-9a-f]{20} dev-impl@lab -> alex@team at http:\/\/192\.0\.2\.7:7433\/mcp UNENCRYPTED \(http, allowed for this binding\)/);
|
||||
await cli().parseAsync(["node", "rig", "network", "status"]);
|
||||
expect(String(out.mock.calls.at(-1)?.[0])).toContain("UNENCRYPTED (http, allowed for this binding)");
|
||||
await cli().parseAsync(["node", "rig", "network", "status", "--json"]);
|
||||
const json = String(out.mock.calls.at(-1)?.[0]);
|
||||
expect(JSON.parse(json).bindings).toEqual([expect.objectContaining({ hubUrl: "http://192.0.2.7:7433/mcp", allowHttp: true, unencrypted: true })]);
|
||||
expect(json).not.toContain("tok-abc");
|
||||
});
|
||||
|
||||
it("status shows a hub item closed elsewhere under its binding, in text and JSON", async () => {
|
||||
const out = vi.spyOn(console, "log").mockImplementation(() => {});
|
||||
await cli({ "/tok": "tok-abc" }).parseAsync(["node", "rig", "network", "connect", "dev-impl@lab", "--hub", "https://hub.example/mcp", "--as", "alex@team", "--token-file", "/tok"]);
|
||||
const bindingId = String(out.mock.calls.at(-1)?.[0]).match(/netb-[0-9a-f]{20}/)![0];
|
||||
db.prepare(`INSERT INTO network_receipts (local_qitem_id, binding_id, hub_qitem_id, created_at, settled_at, outcome, last_error)
|
||||
VALUES ('qitem-net-0001', ?, 'hub-9', '2026-11-11T19:00:00Z', '2026-11-11T19:01:00Z', 'closed_on_hub', 'closed elsewhere (done, canceled)')`).run(bindingId);
|
||||
await cli().parseAsync(["node", "rig", "network", "status"]);
|
||||
expect(String(out.mock.calls.at(-1)?.[0])).toContain("\n hub item hub-9 was closed elsewhere before its result arrived; the result stays on qitem-net-0001");
|
||||
await cli().parseAsync(["node", "rig", "network", "status", "--json"]);
|
||||
expect(JSON.parse(String(out.mock.calls.at(-1)?.[0])).bindings[0].closedElsewhere).toEqual([{ hubItem: "hub-9", localItem: "qitem-net-0001", note: "closed elsewhere (done, canceled)" }]);
|
||||
});
|
||||
|
||||
it("sends the operator bearer on its writes, so a daemon with one configured takes them", async () => {
|
||||
app = routesWith("op-secret");
|
||||
const out = vi.spyOn(console, "log").mockImplementation(() => {});
|
||||
const err = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||
const connectArgs = ["node", "rig", "network", "connect", "dev-impl@lab", "--hub", "https://hub.example/mcp", "--as", "alex@team", "--token-file", "/tok"];
|
||||
vi.stubEnv("OPENRIG_AUTH_BEARER_TOKEN", "wrong");
|
||||
await cli({ "/tok": "tok-abc" }).parseAsync(connectArgs);
|
||||
expect(process.exitCode).toBe(1);
|
||||
expect(String(err.mock.calls.at(-1)?.[0])).toContain("bearer token does not match");
|
||||
process.exitCode = undefined;
|
||||
vi.stubEnv("OPENRIG_AUTH_BEARER_TOKEN", "op-secret");
|
||||
await cli({ "/tok": "tok-abc" }).parseAsync(connectArgs);
|
||||
expect(process.exitCode).toBeUndefined();
|
||||
expect(String(out.mock.calls.at(-1)?.[0])).toMatch(/^Connected netb-/);
|
||||
await cli().parseAsync(["node", "rig", "network", "disconnect", "dev-impl@lab"]);
|
||||
expect(process.exitCode).toBeUndefined();
|
||||
expect(String(out.mock.calls.at(-1)?.[0])).toMatch(/^Disconnected netb-/);
|
||||
});
|
||||
@@ -0,0 +1,159 @@
|
||||
// The outbound connector against a real hub: the daemon's routes behind `rig mcp serve --http` with a grant per
|
||||
// participant, reached by the connector's own MCP client. A second, local queue takes the work, and what happens there
|
||||
// comes back to the hub item: queued, picked up, the result, and a revoked grant ending the binding.
|
||||
import { afterEach, beforeEach, expect, it } from "vitest";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import type { AddressInfo } from "node:net";
|
||||
import { serve, type ServerType } from "@hono/node-server";
|
||||
import type Database from "better-sqlite3";
|
||||
import { createTestApp } from "../../daemon/test/helpers/test-app.js";
|
||||
import { createDb } from "../../daemon/src/db/connection.js";
|
||||
import { migrate } from "../../daemon/src/db/migrate.js";
|
||||
import { ALL_MIGRATIONS } from "../../daemon/src/db/all-migrations.js";
|
||||
import { EventBus } from "../../daemon/src/domain/event-bus.js";
|
||||
import { QueueRepository } from "../../daemon/src/domain/queue-repository.js";
|
||||
import { OutboxHandler } from "../../daemon/src/domain/outbox-handler.js";
|
||||
import { RigRepository } from "../../daemon/src/domain/rig-repository.js";
|
||||
import { NetworkBindingStore } from "../../daemon/src/domain/network/network-bindings.js";
|
||||
import { networkClosedElsewhere, runNetworkBindingPass, type NetworkConnectorDeps } from "../../daemon/src/domain/network/network-connector.js";
|
||||
import { addGrant, revokeGrant, startMcpHttpServer } from "../src/mcp-http.js";
|
||||
|
||||
let hubDb: Database.Database;
|
||||
let localDb: Database.Database;
|
||||
let hubHome: string;
|
||||
let localHome: string;
|
||||
let hubRepo: QueueRepository;
|
||||
let daemon: ServerType;
|
||||
let hubUrl: string;
|
||||
let tokenFile: string;
|
||||
const closers: Array<() => Promise<void>> = [];
|
||||
const saved = { home: process.env.OPENRIG_HOME, session: process.env.OPENRIG_SESSION_NAME };
|
||||
|
||||
const queue = (db: Database.Database) => {
|
||||
const repo = new QueueRepository(db, new EventBus(db), { validateRig: () => true, loadHumanRegistry: () => ({ ok: true as const, entities: [] }) });
|
||||
repo.attachOutbox(new OutboxHandler(db));
|
||||
return repo;
|
||||
};
|
||||
|
||||
beforeEach(async () => {
|
||||
hubDb = createDb(":memory:");
|
||||
migrate(hubDb, ALL_MIGRATIONS);
|
||||
hubHome = fs.mkdtempSync(path.join(os.tmpdir(), "network-hub-"));
|
||||
localHome = fs.mkdtempSync(path.join(os.tmpdir(), "network-local-"));
|
||||
process.env.OPENRIG_HOME = hubHome;
|
||||
process.env.OPENRIG_SESSION_NAME = "env-seat@elsewhere";
|
||||
tokenFile = path.join(hubHome, "mcp-http-tokens.json");
|
||||
hubRepo = queue(hubDb);
|
||||
const inert = async () => ({ ok: false, error: "inert fixture" });
|
||||
const { app } = createTestApp(hubDb, { appDeps: {
|
||||
sessionTransport: { resolveSessions: async () => ({ ok: false, code: "not_found", error: "inert fixture" }), send: inert, capture: inert, broadcast: inert } as never,
|
||||
queueRepo: hubRepo,
|
||||
} });
|
||||
daemon = await new Promise<ServerType>((resolve) => { const s = serve({ fetch: app.fetch, port: 0, hostname: "127.0.0.1" }, () => resolve(s)); });
|
||||
const served = await startMcpHttpServer({ daemonUrl: `http://127.0.0.1:${(daemon.address() as AddressInfo).port}`, host: "127.0.0.1", port: 0, tokenFile });
|
||||
closers.push(() => served.close());
|
||||
hubUrl = served.url;
|
||||
|
||||
localDb = createDb(":memory:");
|
||||
migrate(localDb, ALL_MIGRATIONS);
|
||||
const rigs = new RigRepository(localDb);
|
||||
const node = rigs.addNode(rigs.createRig("lab").id, "impl", { runtime: "claude-code" });
|
||||
localDb.prepare("INSERT INTO sessions (id, node_id, session_name, status) VALUES ('s1', ?, 'dev-impl@lab', 'running')").run(node.id);
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
for (const close of closers.splice(0).reverse()) await close();
|
||||
await new Promise<void>((resolve) => daemon.close(() => resolve()));
|
||||
hubDb.close();
|
||||
localDb.close();
|
||||
fs.rmSync(hubHome, { recursive: true, force: true });
|
||||
fs.rmSync(localHome, { recursive: true, force: true });
|
||||
if (saved.home === undefined) delete process.env.OPENRIG_HOME; else process.env.OPENRIG_HOME = saved.home;
|
||||
if (saved.session === undefined) delete process.env.OPENRIG_SESSION_NAME; else process.env.OPENRIG_SESSION_NAME = saved.session;
|
||||
});
|
||||
|
||||
it("takes the participant's item over the wire, carries pickup and the result back, and a revoked grant ends the binding", async () => {
|
||||
const token = await addGrant(tokenFile, { name: "alex", seat: "alex@team", rigs: ["team"], roots: [], workspace: false, owner: false });
|
||||
// Longer than a list preview, with its end marked, so a shortened body would show.
|
||||
const body = `Summarize the release notes. ${"Context. ".repeat(400)}END-OF-BODY`;
|
||||
const task = await hubRepo.create({ sourceSession: "lead@team", destinationSession: "alex@team", summary: "Summarize the notes", body, nudge: false } as never);
|
||||
|
||||
const localRepo = queue(localDb);
|
||||
const bindings = new NetworkBindingStore(localDb, localHome);
|
||||
// The test hub listens on loopback http, so the binding records the person's --allow-http choice.
|
||||
const binding = bindings.create({ session: "dev-impl@lab", hubUrl, hubSeat: "alex@team", token, allowHttp: true });
|
||||
const deps: NetworkConnectorDeps = { db: localDb, queueRepo: localRepo, bindings, home: localHome };
|
||||
const pass = () => runNetworkBindingPass(deps, bindings.get(binding.id)!);
|
||||
const notes = () => hubRepo.listTransitions(task.qitemId).map((t) => t.transitionNote).filter(Boolean);
|
||||
|
||||
const first = await pass();
|
||||
expect(first.error).toBeUndefined();
|
||||
expect(first).toMatchObject({ taken: 1 });
|
||||
const [local] = localRepo.list({ limit: 50 });
|
||||
expect(local).toMatchObject({ sourceSession: "daemon@kernel", destinationSession: "dev-impl@lab", summary: "From lead@team via network hub: Summarize the notes", state: "pending" });
|
||||
expect(local!.body!.split("\n")[0]).toBe(`From lead@team through the network hub ${hubUrl}, as alex@team (hub item ${task.qitemId}).`);
|
||||
expect(local!.body).toContain(body);
|
||||
expect(hubRepo.getById(task.qitemId)!.state).toBe("in-progress");
|
||||
expect(notes().at(-1)).toBe(`Queued for the connected local seat as ${local!.qitemId}. Not picked up yet.`);
|
||||
expect(hubRepo.listTransitions(task.qitemId).find((t) => t.state === "in-progress")?.actorSession).toMatch(/^alex@team/);
|
||||
|
||||
expect(await pass()).toMatchObject({ taken: 0, pickupsSent: 0 });
|
||||
expect(localRepo.list({ limit: 50 })).toHaveLength(1);
|
||||
|
||||
localRepo.claim({ qitemId: local!.qitemId, destinationSession: "dev-impl@lab" });
|
||||
expect(await pass()).toMatchObject({ pickupsSent: 1 });
|
||||
expect(notes().at(-1)).toMatch(/^Picked up by the local seat at /);
|
||||
expect(hubRepo.getById(task.qitemId)!.state).toBe("in-progress");
|
||||
|
||||
await localRepo.update({ qitemId: local!.qitemId, actorSession: "dev-impl@lab", state: "done", closureReason: "no-follow-on", resolution: "Three changes; none breaking." });
|
||||
expect(await pass()).toMatchObject({ resultsSent: 1 });
|
||||
expect(hubRepo.getById(task.qitemId)).toMatchObject({ state: "done", resolution: "Three changes; none breaking." });
|
||||
expect(bindings.get(binding.id)!.lastError).toBeNull();
|
||||
|
||||
// A failure and a cancel close the hub item the way its queue takes them: failed, and done with closure reason canceled.
|
||||
const failing = await hubRepo.create({ sourceSession: "lead@team", destinationSession: "alex@team", summary: "Build it", body: "Build the release.", nudge: false } as never);
|
||||
const dropped = await hubRepo.create({ sourceSession: "lead@team", destinationSession: "alex@team", summary: "Skip it", body: "Maybe not.", nudge: false } as never);
|
||||
expect(await pass()).toMatchObject({ taken: 2 });
|
||||
const byHub = (id: string) => localRepo.list({ limit: 50 }).find((i) => (i.tags ?? []).includes(`network-hub-item:${id}`))!;
|
||||
await localRepo.update({ qitemId: byHub(failing.qitemId).qitemId, actorSession: "dev-impl@lab", state: "failed", transitionNote: "The build broke." });
|
||||
await localRepo.update({ qitemId: byHub(dropped.qitemId).qitemId, actorSession: "dev-impl@lab", state: "done", closureReason: "canceled", resolution: "Not needed." });
|
||||
expect(await pass()).toMatchObject({ resultsSent: 2 });
|
||||
expect(hubRepo.getById(failing.qitemId)).toMatchObject({ state: "failed", resolution: "The build broke." });
|
||||
expect(hubRepo.getById(dropped.qitemId)).toMatchObject({ state: "done", closureReason: "canceled", resolution: "Not needed." });
|
||||
|
||||
// Two items the requester fails on the hub while they're queued here. Our failed and done results then aren't ours to
|
||||
// deliver: the hub items keep their failure, and the receipts say closed elsewhere.
|
||||
const theirs = [
|
||||
await hubRepo.create({ sourceSession: "lead@team", destinationSession: "alex@team", summary: "One", body: "First.", nudge: false } as never),
|
||||
await hubRepo.create({ sourceSession: "lead@team", destinationSession: "alex@team", summary: "Two", body: "Second.", nudge: false } as never),
|
||||
];
|
||||
expect(await pass()).toMatchObject({ taken: 2 });
|
||||
for (const item of theirs) await hubRepo.update({ qitemId: item.qitemId, actorSession: "lead@team", state: "failed", resolution: "Dropped by the requester." } as never);
|
||||
await localRepo.update({ qitemId: byHub(theirs[0]!.qitemId).qitemId, actorSession: "dev-impl@lab", state: "failed", transitionNote: "Could not finish." });
|
||||
await localRepo.update({ qitemId: byHub(theirs[1]!.qitemId).qitemId, actorSession: "dev-impl@lab", state: "done", closureReason: "no-follow-on", resolution: "Finished it." });
|
||||
expect(await pass()).toMatchObject({ resultsSent: 0 });
|
||||
for (const item of theirs) expect(hubRepo.getById(item.qitemId)).toMatchObject({ state: "failed", resolution: "Dropped by the requester." });
|
||||
expect(localDb.prepare("SELECT DISTINCT outcome FROM network_receipts WHERE hub_qitem_id IN (?, ?)").all(theirs[0]!.qitemId, theirs[1]!.qitemId)).toEqual([{ outcome: "closed_on_hub" }]);
|
||||
|
||||
// An operator on the hub hands the original on to another seat while it's queued here. It's closed to the connector:
|
||||
// nothing is written to it, the original and its successor are unchanged, status shows it, and it isn't retried.
|
||||
const handed = await hubRepo.create({ sourceSession: "lead@team", destinationSession: "alex@team", summary: "Three", body: "Third.", nudge: false } as never);
|
||||
expect(await pass()).toMatchObject({ taken: 1 });
|
||||
const { created: successor } = await hubRepo.handoff({ qitemId: handed.qitemId, fromSession: "lead@team", toSession: "other@team", nudge: false });
|
||||
const before = hubRepo.listTransitions(handed.qitemId).length;
|
||||
await localRepo.update({ qitemId: byHub(handed.qitemId).qitemId, actorSession: "dev-impl@lab", state: "done", closureReason: "no-follow-on", resolution: "Done here." });
|
||||
expect(await pass()).toMatchObject({ resultsSent: 0 });
|
||||
expect(hubRepo.getById(handed.qitemId)!.state).toBe("handed-off");
|
||||
expect(hubRepo.listTransitions(handed.qitemId)).toHaveLength(before);
|
||||
expect(hubRepo.getById(successor.qitemId)).toMatchObject({ state: "pending", destinationSession: "other@team" });
|
||||
expect(networkClosedElsewhere(localDb, binding.id)).toContainEqual(expect.objectContaining({ hubItem: handed.qitemId, localItem: byHub(handed.qitemId).qitemId,
|
||||
note: expect.stringContaining("closed elsewhere (handed-off") }));
|
||||
await pass();
|
||||
expect(hubRepo.listTransitions(handed.qitemId)).toHaveLength(before);
|
||||
|
||||
await revokeGrant(tokenFile, "alex");
|
||||
expect(await pass()).toMatchObject({ error: "revoked" });
|
||||
expect(bindings.get(binding.id)!.state).toBe("revoked");
|
||||
});
|
||||
@@ -107,6 +107,7 @@
|
||||
"dependencies": {
|
||||
"@hono/node-server": "^1.13.0",
|
||||
"@hono/node-ws": "^1.3.1",
|
||||
"@modelcontextprotocol/sdk": "^1.32.2",
|
||||
"better-sqlite3": "^13.0.3",
|
||||
"hono": "^4.7.0",
|
||||
"smol-toml": "^1.8.0",
|
||||
|
||||
@@ -7,6 +7,8 @@ import { queueErasureLinkIndexesSchema } from "./migrations/156_queue_erasure_li
|
||||
import { queueErasureBatchesSchema } from "./migrations/157_queue_erasure_batches.js";
|
||||
import { queueInboxCheckinsSchema } from "./migrations/158_queue_inbox_checkins.js";
|
||||
import { queueItemsHandedOffFromIndexSchema } from "./migrations/159_queue_items_handed_off_from_index.js";
|
||||
import { networkBindingsSchema } from "./migrations/160_network_bindings.js";
|
||||
import { networkReceiptsSchema } from "./migrations/161_network_receipts.js";
|
||||
import { activityGrantsSchema } from "./migrations/143_activity_grants.js";
|
||||
import { environmentBindingsSchema } from "./migrations/144_environment_bindings.js";
|
||||
import { nodePermissionsToSchema } from "./migrations/145_node_permissions_to.js";
|
||||
@@ -321,4 +323,6 @@ export const ALL_MIGRATIONS: Migration[] = [
|
||||
queueErasureBatchesSchema,
|
||||
queueInboxCheckinsSchema,
|
||||
queueItemsHandedOffFromIndexSchema,
|
||||
networkBindingsSchema,
|
||||
networkReceiptsSchema,
|
||||
];
|
||||
@@ -0,0 +1,27 @@
|
||||
import type { Migration } from "../migrate.js";
|
||||
|
||||
/** A selected local seat joined to one participant of a remote hub (an account's network rig, reached over MCP). The
|
||||
* participant's credential lives in a private file under the OpenRig home, never in this table. One active binding per
|
||||
* local seat and per hub participant; a disconnected or revoked binding stays as history. `allow_http` records the
|
||||
* person's choice to send the participant's token to a plain-http hub. */
|
||||
export const networkBindingsSchema: Migration = {
|
||||
name: "160_network_bindings.sql",
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS network_bindings (
|
||||
id TEXT PRIMARY KEY,
|
||||
node_id TEXT NOT NULL,
|
||||
local_session TEXT NOT NULL,
|
||||
hub_url TEXT NOT NULL,
|
||||
hub_seat TEXT NOT NULL,
|
||||
allow_http INTEGER NOT NULL DEFAULT 0 CHECK (allow_http IN (0, 1)),
|
||||
state TEXT NOT NULL CHECK (state IN ('active', 'revoked', 'disconnected')),
|
||||
state_reason TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
last_contact_at TEXT,
|
||||
last_error TEXT
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_network_bindings_active_node ON network_bindings(node_id) WHERE state = 'active';
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_network_bindings_active_participant ON network_bindings(hub_url, hub_seat) WHERE state = 'active';
|
||||
`,
|
||||
};
|
||||
@@ -0,0 +1,26 @@
|
||||
import type { Migration } from "../migrate.js";
|
||||
|
||||
/** One hub item taken in, and the local item that carries it: what was reported back, and when. It's a receipt, not a
|
||||
* task: the work itself is the local queue item, found by its id, which is derived from the seat's node, the hub, the
|
||||
* participant and the hub item. `binding_id` is the binding that reports it now, and a new binding for the same seat,
|
||||
* hub and participant takes it over. `settled_at` is when nothing more is reported (the result was sent, the outcome is
|
||||
* unknown, or the hub item was closed elsewhere: `outcome` says which); `checked_at` gives every open receipt a turn. */
|
||||
export const networkReceiptsSchema: Migration = {
|
||||
name: "161_network_receipts.sql",
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS network_receipts (
|
||||
local_qitem_id TEXT PRIMARY KEY,
|
||||
binding_id TEXT NOT NULL,
|
||||
hub_qitem_id TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
queued_sent_at TEXT,
|
||||
pickup_sent_at TEXT,
|
||||
settled_at TEXT,
|
||||
checked_at TEXT,
|
||||
outcome TEXT,
|
||||
last_error TEXT
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_network_receipts_hub ON network_receipts(binding_id, hub_qitem_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_network_receipts_open ON network_receipts(binding_id) WHERE settled_at IS NULL;
|
||||
`,
|
||||
};
|
||||
@@ -0,0 +1,120 @@
|
||||
// The connector's view of a remote hub: the queue tools an external grant offers its participant (list its own items,
|
||||
// claim, show, update), reached over MCP's Streamable HTTP with that participant's bearer. Nothing here decides what to
|
||||
// do with an item; it only says what the hub answered, and sorts a failure into what the caller must do about it.
|
||||
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
|
||||
import { StreamableHTTPClientTransport, StreamableHTTPError } from "@modelcontextprotocol/sdk/client/streamableHttp.js";
|
||||
|
||||
export interface HubItem {
|
||||
qitemId: string;
|
||||
sourceSession: string;
|
||||
destinationSession: string;
|
||||
state: string;
|
||||
body?: string | null;
|
||||
summary?: string | null;
|
||||
resolution?: string | null;
|
||||
closureReason?: string | null;
|
||||
}
|
||||
|
||||
/** `revoked`: the hub refused the credential (401), so the binding ends. `unavailable`: the hub or its grant authority
|
||||
* couldn't answer, or the connection failed, so try later. `refused`: the hub answered this one call with an error. */
|
||||
export type HubErrorKind = "revoked" | "unavailable" | "refused";
|
||||
|
||||
export class HubError extends Error {
|
||||
constructor(readonly kind: HubErrorKind, message: string) {
|
||||
super(message);
|
||||
}
|
||||
}
|
||||
|
||||
export interface HubSession {
|
||||
listOwned(state: "pending" | "in-progress", limit: number): Promise<HubItem[]>;
|
||||
claim(qitemId: string): Promise<void>;
|
||||
show(qitemId: string): Promise<HubItem>;
|
||||
note(qitemId: string, note: string): Promise<void>;
|
||||
/** Close the way the queue closes an item: done with a closure reason (no-follow-on, canceled or denied), or failed. */
|
||||
close(qitemId: string, input: { state: "done" | "failed"; closureReason: string | null; resolution: string }): Promise<void>;
|
||||
end(): Promise<void>;
|
||||
}
|
||||
|
||||
/** `signal` closes the connection when it aborts, which ends any call still waiting on the hub. */
|
||||
export type HubConnect = (hubUrl: string, token: string, signal?: AbortSignal) => Promise<HubSession>;
|
||||
|
||||
/** A failure, sorted. The hub's own MCP endpoint answers 401 for an unknown or revoked credential and 503 when its grant
|
||||
* authority can't answer (`packages/cli/src/mcp-http.ts`). */
|
||||
export function hubFailure(error: unknown): HubError {
|
||||
if (error instanceof HubError) return error;
|
||||
if (error instanceof StreamableHTTPError && error.code === 401) {
|
||||
return new HubError("revoked", "The hub refused this participant's credential: it's unknown there or was revoked.");
|
||||
}
|
||||
if (error instanceof StreamableHTTPError && typeof error.code === "number" && error.code >= 300 && error.code < 400) {
|
||||
return new HubError("unavailable", `The hub redirected the connection somewhere else, and the participant's token isn't sent there (${error.message.slice(0, 300)}). Connect the binding to the hub's own URL.`);
|
||||
}
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
return new HubError("unavailable", `The hub couldn't be reached: ${message.slice(0, 300)}`);
|
||||
}
|
||||
|
||||
function parsed(result: { isError?: boolean; content?: unknown }): unknown {
|
||||
const content = Array.isArray(result.content) ? result.content as Array<{ type?: string; text?: string }> : [];
|
||||
const text = content.filter((part) => part.type === "text").map((part) => part.text ?? "").join("\n");
|
||||
if (result.isError) throw new HubError("refused", text.slice(0, 500) || "The hub refused this call.");
|
||||
try { return JSON.parse(text); } catch { throw new HubError("refused", "The hub answered with something that isn't JSON."); }
|
||||
}
|
||||
|
||||
const item = (value: unknown): HubItem => {
|
||||
const row = (value ?? {}) as Record<string, unknown>;
|
||||
if (typeof row.qitemId !== "string") throw new HubError("refused", "The hub answered without an item id.");
|
||||
return {
|
||||
qitemId: row.qitemId,
|
||||
sourceSession: String(row.sourceSession ?? ""),
|
||||
destinationSession: String(row.destinationSession ?? ""),
|
||||
state: String(row.state ?? ""),
|
||||
body: typeof row.body === "string" ? row.body : null,
|
||||
summary: typeof row.summary === "string" ? row.summary : null,
|
||||
resolution: typeof row.resolution === "string" ? row.resolution : null,
|
||||
closureReason: typeof row.closureReason === "string" ? row.closureReason : null,
|
||||
};
|
||||
};
|
||||
|
||||
/** Connect to a hub as one participant. The bearer goes only in this transport's Authorization header. The SDK's
|
||||
* same-origin policy, set explicitly, follows a redirect only within the hub's origin, or to its https form on the same
|
||||
* host with default ports, so the bearer follows only those; any other redirect fails with its target named. */
|
||||
export const connectMcpHub: HubConnect = async (hubUrl, token, signal) => {
|
||||
const transport = new StreamableHTTPClientTransport(new URL(hubUrl), {
|
||||
requestInit: { headers: { Authorization: `Bearer ${token}` } },
|
||||
redirectPolicy: "same-origin",
|
||||
});
|
||||
const client = new Client({ name: "openrig-network-connector", version: "1" });
|
||||
const close = () => { void client.close().catch(() => {}); };
|
||||
if (signal?.aborted) throw new HubError("unavailable", "The pass ended before it reached the hub.");
|
||||
signal?.addEventListener("abort", close, { once: true });
|
||||
try {
|
||||
await client.connect(transport);
|
||||
} catch (error) {
|
||||
signal?.removeEventListener("abort", close);
|
||||
await client.close().catch(() => {});
|
||||
throw hubFailure(error);
|
||||
}
|
||||
const call = async (name: string, args: Record<string, unknown>): Promise<unknown> => {
|
||||
try {
|
||||
return parsed(await client.callTool({ name, arguments: args }) as { isError?: boolean; content?: unknown });
|
||||
} catch (error) {
|
||||
throw hubFailure(error);
|
||||
}
|
||||
};
|
||||
return {
|
||||
listOwned: async (state, limit) => {
|
||||
const rows = await call("rig_queue_list", { owned: true, state, limit });
|
||||
if (!Array.isArray(rows)) throw new HubError("refused", "The hub's list wasn't a list.");
|
||||
return rows.map(item);
|
||||
},
|
||||
claim: async (qitemId) => { await call("rig_queue_claim", { qitemId }); },
|
||||
show: async (qitemId) => {
|
||||
const shown = await call("rig_queue_show", { qitemId }) as Record<string, unknown>;
|
||||
return item(shown?.item ?? shown);
|
||||
},
|
||||
note: async (qitemId, note) => { await call("rig_queue_update", { qitemId, note }); },
|
||||
close: async (qitemId, input) => {
|
||||
await call("rig_queue_update", { qitemId, state: input.state, ...(input.closureReason ? { closureReason: input.closureReason } : {}), resolution: input.resolution });
|
||||
},
|
||||
end: async () => { signal?.removeEventListener("abort", close); await client.close().catch(() => {}); },
|
||||
};
|
||||
};
|
||||
@@ -0,0 +1,238 @@
|
||||
// A selected local seat joined to one participant of a remote hub: the account's network rig, which this daemon reaches
|
||||
// as an MCP client with that participant's own credential (an external grant). The binding names the local seat by its
|
||||
// durable node id, so a renamed or relaunched seat keeps it; the hub address represents the seat and never renames it.
|
||||
// The credential is a private file under the OpenRig home; this table only says where the binding stands.
|
||||
import type Database from "better-sqlite3";
|
||||
import { randomBytes } from "node:crypto";
|
||||
import { closeSync, existsSync, lstatSync, mkdirSync, openSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { resolveSessionNodeId } from "../queue-owner.js";
|
||||
|
||||
export type NetworkBindingState = "active" | "revoked" | "disconnected";
|
||||
|
||||
export interface NetworkBinding {
|
||||
id: string;
|
||||
nodeId: string;
|
||||
/** The seat's address when it was bound, for display; the node id is what the binding follows. */
|
||||
localSession: string;
|
||||
hubUrl: string;
|
||||
/** The participant's address in the hub (`member@rig`). */
|
||||
hubSeat: string;
|
||||
/** The person chose, for this binding, to send the participant's token to a plain-http hub. */
|
||||
allowHttp: boolean;
|
||||
/** The hub URL is plain http, so the token and the work cross the network unencrypted. */
|
||||
unencrypted: boolean;
|
||||
state: NetworkBindingState;
|
||||
stateReason: string | null;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
lastContactAt: string | null;
|
||||
lastError: string | null;
|
||||
}
|
||||
|
||||
export class NetworkBindingError extends Error {
|
||||
constructor(readonly code: string, message: string, readonly status: 400 | 404 | 409 | 500 = 400) {
|
||||
super(message);
|
||||
}
|
||||
}
|
||||
|
||||
interface Row {
|
||||
id: string; node_id: string; local_session: string; hub_url: string; hub_seat: string; allow_http: number; state: NetworkBindingState;
|
||||
state_reason: string | null; created_at: string; updated_at: string; last_contact_at: string | null; last_error: string | null;
|
||||
}
|
||||
|
||||
const project = (row: Row): NetworkBinding => ({
|
||||
id: row.id, nodeId: row.node_id, localSession: row.local_session, hubUrl: row.hub_url, hubSeat: row.hub_seat,
|
||||
allowHttp: row.allow_http === 1, unencrypted: row.hub_url.startsWith("http:"), state: row.state, stateReason: row.state_reason, createdAt: row.created_at, updatedAt: row.updated_at,
|
||||
lastContactAt: row.last_contact_at, lastError: row.last_error,
|
||||
});
|
||||
|
||||
const MAX_CREDENTIAL_BYTES = 16384;
|
||||
|
||||
/** Where a binding's credential lives: `<home>/network-secrets/<binding id>/credential.json`, private to this user. */
|
||||
export function networkCredentialFile(home: string, bindingId: string): string {
|
||||
if (!/^netb-[0-9a-f]{20}$/.test(bindingId)) throw new NetworkBindingError("invalid_binding_id", `'${bindingId}' isn't a network binding id.`);
|
||||
return join(home, "network-secrets", bindingId, "credential.json");
|
||||
}
|
||||
|
||||
/** The participant's bearer, read from its private file. A file anyone else can read, a link or an oversized file is
|
||||
* refused, with the path named and the value never shown. */
|
||||
export function readNetworkCredential(home: string, bindingId: string): string {
|
||||
const file = networkCredentialFile(home, bindingId);
|
||||
let stat;
|
||||
try { stat = lstatSync(file); } catch {
|
||||
throw new NetworkBindingError("credential_missing", `The credential for ${bindingId} isn't at ${file}. Connect the seat again with its token file.`, 404);
|
||||
}
|
||||
if (!stat.isFile() || stat.isSymbolicLink() || stat.size > MAX_CREDENTIAL_BYTES || (stat.mode & 0o077) !== 0) {
|
||||
throw new NetworkBindingError("credential_unsafe", `${file} must be a regular file readable only by this user.`);
|
||||
}
|
||||
// A parse error can quote the file's text, which holds the token: say only which file.
|
||||
let value: { version?: unknown; token?: unknown };
|
||||
try { value = JSON.parse(readFileSync(file, "utf8")) as typeof value; } catch {
|
||||
throw new NetworkBindingError("credential_invalid", `${file} isn't a network credential.`);
|
||||
}
|
||||
if (value === null || typeof value !== "object" || value.version !== 1 || typeof value.token !== "string" || !value.token) {
|
||||
throw new NetworkBindingError("credential_invalid", `${file} isn't a network credential.`);
|
||||
}
|
||||
return value.token;
|
||||
}
|
||||
|
||||
function validHubUrl(raw: unknown, allowHttp: boolean): string {
|
||||
if (typeof raw !== "string" || !raw.trim()) throw new NetworkBindingError("invalid_hub_url", "A hub URL is required.");
|
||||
let url: URL;
|
||||
try { url = new URL(raw.trim()); } catch { throw new NetworkBindingError("invalid_hub_url", `'${raw}' isn't a URL.`); }
|
||||
if (url.protocol !== "https:" && url.protocol !== "http:") throw new NetworkBindingError("invalid_hub_url", "The hub URL must use https (or http).");
|
||||
// A credential in the URL would be stored in plain text in this table and shown in status.
|
||||
if (url.username || url.password) throw new NetworkBindingError("invalid_hub_url", "Put the participant's token in its token file, not in the hub URL.");
|
||||
// Plain http is the person's choice per binding. The scheme is kept as given, and the address isn't judged: a private
|
||||
// or tailnet address doesn't prove the connection is protected.
|
||||
if (url.protocol === "http:" && !allowHttp) {
|
||||
throw new NetworkBindingError("http_not_allowed", `${url.toString()} is plain http, so the participant's token and its work would cross the network unencrypted, where they can be read. Use the hub's https URL, or connect again with --allow-http to accept that for this binding.`);
|
||||
}
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
function validHubSeat(raw: unknown): string {
|
||||
if (typeof raw !== "string" || !/^[^@\s]+@[^@\s]+$/.test(raw.trim())) {
|
||||
throw new NetworkBindingError("invalid_hub_seat", "The hub participant is its address in the hub, member@rig.");
|
||||
}
|
||||
return raw.trim();
|
||||
}
|
||||
|
||||
function validToken(raw: unknown): string {
|
||||
const token = typeof raw === "string" ? raw.trim() : "";
|
||||
if (!token || /\s/.test(token) || Buffer.byteLength(token) > 8192) {
|
||||
throw new NetworkBindingError("invalid_token", "The participant's token must be one non-empty line with no spaces.");
|
||||
}
|
||||
return token;
|
||||
}
|
||||
|
||||
export class NetworkBindingStore {
|
||||
constructor(
|
||||
private readonly db: Database.Database,
|
||||
private readonly home: string,
|
||||
private readonly now: () => Date = () => new Date(),
|
||||
private readonly newId: () => string = () => `netb-${randomBytes(10).toString("hex")}`,
|
||||
) {}
|
||||
|
||||
list(): NetworkBinding[] {
|
||||
return (this.db.prepare("SELECT * FROM network_bindings ORDER BY created_at, id").all() as Row[]).map(project);
|
||||
}
|
||||
|
||||
active(): NetworkBinding[] {
|
||||
return (this.db.prepare("SELECT * FROM network_bindings WHERE state = 'active' ORDER BY created_at, id").all() as Row[]).map(project);
|
||||
}
|
||||
|
||||
get(id: string): NetworkBinding | null {
|
||||
const row = this.db.prepare("SELECT * FROM network_bindings WHERE id = ?").get(id) as Row | undefined;
|
||||
return row ? project(row) : null;
|
||||
}
|
||||
|
||||
/** A binding by its id, or the active binding of a local seat named by its address. */
|
||||
find(ref: string): NetworkBinding | null {
|
||||
const byId = this.get(ref);
|
||||
if (byId) return byId;
|
||||
const nodeId = resolveSessionNodeId(this.db, ref);
|
||||
const row = nodeId ? this.db.prepare("SELECT * FROM network_bindings WHERE node_id = ? AND state = 'active'").get(nodeId) as Row | undefined : undefined;
|
||||
return row ? project(row) : null;
|
||||
}
|
||||
|
||||
/** Bind a local seat to a hub participant with that participant's token. The credential is written first, exclusively
|
||||
* and private, then the row. If either step fails, the credential this call created is removed, so a refused bind
|
||||
* leaves none behind; a file that was already at that path is never touched. */
|
||||
create(input: { session: unknown; hubUrl: unknown; hubSeat: unknown; token: unknown; allowHttp?: unknown }): NetworkBinding {
|
||||
const session = typeof input.session === "string" ? input.session.trim() : "";
|
||||
if (!session) throw new NetworkBindingError("invalid_seat", "Name the local seat to connect.");
|
||||
const nodeId = resolveSessionNodeId(this.db, session);
|
||||
if (!nodeId) throw new NetworkBindingError("seat_not_found", `No local seat is named '${session}'.`, 404);
|
||||
const allowHttp = input.allowHttp === true;
|
||||
const hubUrl = validHubUrl(input.hubUrl, allowHttp);
|
||||
const hubSeat = validHubSeat(input.hubSeat);
|
||||
const token = validToken(input.token);
|
||||
const bound = this.db.prepare("SELECT id FROM network_bindings WHERE node_id = ? AND state = 'active'").get(nodeId) as { id: string } | undefined;
|
||||
if (bound) throw new NetworkBindingError("seat_already_connected", `${session} is already connected (${bound.id}). Disconnect it first.`, 409);
|
||||
const taken = this.db.prepare("SELECT id FROM network_bindings WHERE hub_url = ? AND hub_seat = ? AND state = 'active'").get(hubUrl, hubSeat) as { id: string } | undefined;
|
||||
if (taken) throw new NetworkBindingError("participant_already_connected", `${hubSeat} at ${hubUrl} is already connected (${taken.id}).`, 409);
|
||||
|
||||
const id = this.newId();
|
||||
const file = networkCredentialFile(this.home, id);
|
||||
let created = false;
|
||||
let madeFolder: string | undefined;
|
||||
try {
|
||||
madeFolder = mkdirSync(dirname(file), { recursive: true, mode: 0o700 });
|
||||
const fd = openSync(file, "wx", 0o600);
|
||||
created = true;
|
||||
try { writeFileSync(fd, `${JSON.stringify({ version: 1, token })}\n`); } finally { closeSync(fd); }
|
||||
const at = this.now().toISOString();
|
||||
this.db.prepare(`INSERT INTO network_bindings (id, node_id, local_session, hub_url, hub_seat, allow_http, state, created_at, updated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, 'active', ?, ?)`).run(id, nodeId, session, hubUrl, hubSeat, allowHttp ? 1 : 0, at, at);
|
||||
} catch (error) {
|
||||
// Only what this call made: the file it opened, and this binding's folder if it made it (`mkdirSync` returns the
|
||||
// first folder it made, so any value means this one is new). A folder that was already there keeps its other files.
|
||||
if (created) rmSync(file, { force: true });
|
||||
if (madeFolder !== undefined) rmSync(dirname(file), { recursive: true, force: true });
|
||||
throw error;
|
||||
}
|
||||
return this.get(id)!;
|
||||
}
|
||||
|
||||
/** The person ends this binding here. No new work comes in through it; local rows, results and other bindings stay. The
|
||||
* credential is removed, and the hub still lists the participant until it's removed there too. */
|
||||
disconnect(ref: string, reason?: string): NetworkBinding {
|
||||
return this.end(ref, "disconnected", reason?.trim() || "disconnected here");
|
||||
}
|
||||
|
||||
/** The hub refused the credential: it was revoked there. Same effect as a local disconnect. */
|
||||
markRevoked(id: string, reason: string): NetworkBinding {
|
||||
return this.end(id, "revoked", reason);
|
||||
}
|
||||
|
||||
/** The connector reached the hub as this binding. */
|
||||
recordContact(id: string, at: Date = this.now()): void {
|
||||
this.db.prepare("UPDATE network_bindings SET last_contact_at = ?, last_error = NULL, updated_at = ? WHERE id = ?")
|
||||
.run(at.toISOString(), at.toISOString(), id);
|
||||
}
|
||||
|
||||
/** The connector couldn't reach the hub as this binding, or the hub answered with an error; the binding stays active.
|
||||
* Control characters are replaced, since the text can come from the hub and `rig network status` prints it. */
|
||||
recordError(id: string, message: string): void {
|
||||
this.db.prepare("UPDATE network_bindings SET last_error = ?, updated_at = ? WHERE id = ?")
|
||||
.run(message.replace(/[\u0000-\u001f\u007f-\u009f]/g, " ").slice(0, 500), this.now().toISOString(), id);
|
||||
}
|
||||
|
||||
/** The binding ends first, so no new work comes in even if the credential can't be removed; removal is tried on every
|
||||
* call, including again on an already ended binding, until its credential is gone. */
|
||||
private end(ref: string, state: "disconnected" | "revoked", reason: string): NetworkBinding {
|
||||
const binding = this.find(ref);
|
||||
if (!binding) throw new NetworkBindingError("binding_not_found", `No network binding or connected seat is named '${ref}'.`, 404);
|
||||
if (binding.state === "active") {
|
||||
this.db.prepare("UPDATE network_bindings SET state = ?, state_reason = ?, updated_at = ? WHERE id = ? AND state = 'active'")
|
||||
.run(state, reason, this.now().toISOString(), binding.id);
|
||||
}
|
||||
try {
|
||||
this.removeCredential(binding.id);
|
||||
} catch (error) {
|
||||
throw new NetworkBindingError("credential_cleanup_failed",
|
||||
`${binding.id} is ${this.get(binding.id)!.state}, but its credential at ${networkCredentialFile(this.home, binding.id)} couldn't be removed (${(error as Error).message}). Run \`rig network disconnect ${binding.id}\` again to retry.`, 500);
|
||||
}
|
||||
return this.get(binding.id)!;
|
||||
}
|
||||
|
||||
/** Removes the credential folders whose binding isn't active: one a stop left between the credential write and its
|
||||
* row, or one whose removal failed when its binding ended. Other names under the folder are left alone. */
|
||||
sweepCredentials(): { removed: string[]; failed: string[] } {
|
||||
const swept = { removed: [] as string[], failed: [] as string[] };
|
||||
let names: string[];
|
||||
try { names = readdirSync(join(this.home, "network-secrets")); } catch { return swept; }
|
||||
for (const name of names.sort()) {
|
||||
if (!/^netb-[0-9a-f]{20}$/.test(name) || this.get(name)?.state === "active") continue;
|
||||
try { this.removeCredential(name); swept.removed.push(name); } catch { swept.failed.push(name); }
|
||||
}
|
||||
return swept;
|
||||
}
|
||||
|
||||
private removeCredential(id: string): void {
|
||||
const file = networkCredentialFile(this.home, id);
|
||||
if (existsSync(dirname(file))) rmSync(dirname(file), { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,405 @@
|
||||
// The outbound connector: for each active network binding, reach the hub as that participant, take the items addressed
|
||||
// to it into the bound local seat's queue, and carry what happens locally back to the hub item.
|
||||
//
|
||||
// Honest stages, kept apart on the hub item:
|
||||
// - a hub claim is the connector taking custody of the item, not the agent picking it up;
|
||||
// - "picked up" is reported only once the local item is claimed;
|
||||
// - the result is the local item's own closing outcome.
|
||||
//
|
||||
// Each hub item maps to one local item, under an id derived from what it is: the seat's node, the hub, the participant
|
||||
// and the hub item. The queue's same-id absorption makes every retry, restart or new binding for the same seat, hub and
|
||||
// participant land on that one item, so a lost response never starts a second one. An item whose local record is gone
|
||||
// reads unknown and is never run again.
|
||||
import type Database from "better-sqlite3";
|
||||
import { createHash } from "node:crypto";
|
||||
import { QueueRepositoryError, type QueueItem, type QueueRepository } from "../queue-repository.js";
|
||||
import { NetworkBindingError, readNetworkCredential, type NetworkBinding, type NetworkBindingStore } from "./network-bindings.js";
|
||||
import { connectMcpHub, HubError, type HubConnect, type HubItem, type HubSession } from "./hub-client.js";
|
||||
|
||||
/** OpenRig's own sender: the connector creates the local item for the hub's requester, who is named first in it. */
|
||||
const LOCAL_SOURCE = "daemon@kernel";
|
||||
const PAGE = 25;
|
||||
const MAX_PAGES = 8;
|
||||
/** Receipts looked at per pass, the least recently looked at first, so a long backlog still gives every item a turn. */
|
||||
const REPORT_LIMIT = 50;
|
||||
/** How long one binding's pass may take before its connection is closed, so one slow hub can't hold the others. */
|
||||
const PASS_TIMEOUT_MS = 60_000;
|
||||
|
||||
/** The local item for a hub item, by what it is. A retry, a restart or a new binding for the same seat, hub and
|
||||
* participant lands on the same item. */
|
||||
export function networkLocalItemId(binding: Pick<NetworkBinding, "nodeId" | "hubUrl" | "hubSeat">, hubQitemId: string): string {
|
||||
return `qitem-net-${createHash("sha256").update([binding.nodeId, binding.hubUrl, binding.hubSeat, hubQitemId].join("\n")).digest("hex").slice(0, 20)}`;
|
||||
}
|
||||
|
||||
export interface NetworkConnectorDeps {
|
||||
db: Database.Database;
|
||||
queueRepo: QueueRepository;
|
||||
bindings: NetworkBindingStore;
|
||||
home: string;
|
||||
connect?: HubConnect;
|
||||
now?: () => Date;
|
||||
log?: (line: string) => void;
|
||||
passTimeoutMs?: number;
|
||||
reportLimit?: number;
|
||||
}
|
||||
|
||||
export interface NetworkPassResult {
|
||||
bindingId: string;
|
||||
taken: number;
|
||||
pickupsSent: number;
|
||||
resultsSent: number;
|
||||
unknown: number;
|
||||
/** Why the pass stopped early, if it did. `revoked` ended the binding; `no_session` left the hub's items pending;
|
||||
* `local` is a failure here, not the hub's. */
|
||||
error?: HubError["kind"] | "credential" | "no_session" | "http_not_allowed" | "local";
|
||||
}
|
||||
|
||||
/** The seat's current address: the node's latest session, as the queue's other node lookups read it. */
|
||||
function currentSession(db: Database.Database, nodeId: string): string | null {
|
||||
const row = db.prepare("SELECT session_name FROM sessions WHERE node_id = ? ORDER BY created_at DESC, id DESC LIMIT 1")
|
||||
.get(nodeId) as { session_name: string } | undefined;
|
||||
return row?.session_name ?? null;
|
||||
}
|
||||
|
||||
interface Receipt {
|
||||
local_qitem_id: string; binding_id: string; hub_qitem_id: string; queued_sent_at: string | null; pickup_sent_at: string | null;
|
||||
settled_at: string | null; outcome: string | null;
|
||||
}
|
||||
|
||||
/** The states a local item ends in. A handed-off local item ends at its successor, which is followed. */
|
||||
const FINAL = new Set(["done", "canceled", "denied", "failed"]);
|
||||
/** The states in which the hub item is closed to this connector. A hub item handed on by someone there is closed here:
|
||||
* its successor isn't followed, and it's never written to again. */
|
||||
const HUB_CLOSED = new Set([...FINAL, "handed-off"]);
|
||||
|
||||
/** One pass for one binding, bounded by its deadline and the connector's stop. Never throws: a failure is recorded on
|
||||
* the binding and returned. */
|
||||
export async function runNetworkBindingPass(deps: NetworkConnectorDeps, binding: NetworkBinding,
|
||||
opts: { signal?: AbortSignal } = {}): Promise<NetworkPassResult> {
|
||||
const now = deps.now ?? (() => new Date());
|
||||
const result: NetworkPassResult = { bindingId: binding.id, taken: 0, pickupsSent: 0, resultsSent: 0, unknown: 0 };
|
||||
// The token goes to a plain-http hub only on the person's recorded choice for this binding.
|
||||
if (binding.unencrypted && !binding.allowHttp) {
|
||||
deps.bindings.recordError(binding.id, `${binding.hubUrl} is plain http and this binding has no --allow-http choice, so its token isn't sent. Connect again with the hub's https URL, or with --allow-http.`);
|
||||
return { ...result, error: "http_not_allowed" };
|
||||
}
|
||||
let token: string;
|
||||
try {
|
||||
token = readNetworkCredential(deps.home, binding.id);
|
||||
} catch (error) {
|
||||
deps.bindings.recordError(binding.id, error instanceof NetworkBindingError ? error.message : String(error));
|
||||
return { ...result, error: "credential" };
|
||||
}
|
||||
|
||||
const timeoutMs = deps.passTimeoutMs ?? PASS_TIMEOUT_MS;
|
||||
const deadline = new AbortController();
|
||||
const timer = setTimeout(() => deadline.abort(), timeoutMs);
|
||||
timer.unref?.();
|
||||
const onStop = () => deadline.abort();
|
||||
if (opts.signal?.aborted) deadline.abort();
|
||||
opts.signal?.addEventListener("abort", onStop, { once: true });
|
||||
// Checked before every take, and again just before a local item is made: an ended binding, the pass's deadline or
|
||||
// the connector's stop starts no local work, even when a hub answer arrives late.
|
||||
const live = () => !deadline.signal.aborted && deps.bindings.get(binding.id)?.state === "active";
|
||||
|
||||
let hub: HubSession | null = null;
|
||||
try {
|
||||
hub = await (deps.connect ?? connectMcpHub)(binding.hubUrl, token, deadline.signal);
|
||||
// Work goes to the seat at its current address. With no session there's nobody to give it to, so nothing is taken.
|
||||
const destination = currentSession(deps.db, binding.nodeId);
|
||||
if (destination) {
|
||||
// What this participant already holds (claimed before a restart, or through an earlier binding) comes first.
|
||||
for (const held of await hub.listOwned("in-progress", PAGE * MAX_PAGES)) {
|
||||
if (!live()) break;
|
||||
if (held.destinationSession === binding.hubSeat && await ensureLocal(deps, binding, hub, held, destination, now, live)) result.taken++;
|
||||
}
|
||||
// New work, a page at a time: a claim moves an item out of pending, so each page is new items.
|
||||
for (let page = 0; page < MAX_PAGES && live(); page++) {
|
||||
const pending = await hub.listOwned("pending", PAGE);
|
||||
for (const item of pending) {
|
||||
if (!live()) break;
|
||||
if (item.destinationSession !== binding.hubSeat) continue;
|
||||
try {
|
||||
await hub.claim(item.qitemId);
|
||||
} catch (error) {
|
||||
// Another claimant, or the item closed meanwhile: it isn't this binding's to take.
|
||||
if (error instanceof HubError && error.kind === "refused") continue;
|
||||
throw error;
|
||||
}
|
||||
if (await ensureLocal(deps, binding, hub, item, destination, now, live)) result.taken++;
|
||||
}
|
||||
if (pending.length < PAGE) break;
|
||||
}
|
||||
}
|
||||
// Report back what happened locally, for the items this binding holds. A revoked binding never gets here.
|
||||
if (live()) await reportBack(deps, binding, hub, now, result);
|
||||
deps.bindings.recordContact(binding.id, now());
|
||||
if (!destination) {
|
||||
deps.bindings.recordError(binding.id, `${binding.localSession} has no session here, so its hub items stay pending until it has one.`);
|
||||
return { ...result, error: "no_session" };
|
||||
}
|
||||
return result;
|
||||
} catch (error) {
|
||||
if (deadline.signal.aborted) {
|
||||
deps.bindings.recordError(binding.id, opts.signal?.aborted
|
||||
? "The connector stopped during this pass and closed its connection to the hub."
|
||||
: `This pass took longer than ${timeoutMs >= 1000 ? `${Math.round(timeoutMs / 1000)} s` : `${timeoutMs} ms`}, so its connection to the hub was closed. The next pass tries again.`);
|
||||
return { ...result, error: "unavailable" };
|
||||
}
|
||||
// Only the hub session's own failures are the hub's (it reports them as HubError); anything else happened here.
|
||||
if (!(error instanceof HubError)) {
|
||||
deps.bindings.recordError(binding.id, `A local error stopped this pass: ${error instanceof Error ? error.message : String(error)}`);
|
||||
return { ...result, error: "local" };
|
||||
}
|
||||
if (error.kind === "revoked") {
|
||||
try {
|
||||
deps.bindings.markRevoked(binding.id, error.message);
|
||||
} catch (cleanup) {
|
||||
// The binding is revoked either way; only its credential's removal failed, and the error says how to retry.
|
||||
if (!(cleanup instanceof NetworkBindingError) || cleanup.code !== "credential_cleanup_failed") throw cleanup;
|
||||
deps.bindings.recordError(binding.id, cleanup.message);
|
||||
}
|
||||
deps.log?.(`network ${binding.id}: revoked by the hub; no new work comes in through it`);
|
||||
} else {
|
||||
deps.bindings.recordError(binding.id, error.message);
|
||||
}
|
||||
return { ...result, error: error.kind };
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
opts.signal?.removeEventListener("abort", onStop);
|
||||
await hub?.end();
|
||||
}
|
||||
}
|
||||
|
||||
/** The local item for a hub item this participant holds: created once, or found again. Returns true when it was new. */
|
||||
async function ensureLocal(deps: NetworkConnectorDeps, binding: NetworkBinding, hub: HubSession, item: HubItem, destination: string,
|
||||
now: () => Date, live: () => boolean): Promise<boolean> {
|
||||
const localId = networkLocalItemId(binding, item.qitemId);
|
||||
const taken = deps.db.prepare("SELECT binding_id FROM network_receipts WHERE local_qitem_id = ?").get(localId) as { binding_id: string } | undefined;
|
||||
if (taken) {
|
||||
// Taken already, perhaps through an earlier binding for this seat, hub and participant: this binding reports it now,
|
||||
// with what was already sent and its outcome, unknown included, kept.
|
||||
if (taken.binding_id !== binding.id) deps.db.prepare("UPDATE network_receipts SET binding_id = ? WHERE local_qitem_id = ?").run(binding.id, localId);
|
||||
return false;
|
||||
}
|
||||
const full = item.body == null ? await hub.show(item.qitemId) : item;
|
||||
if (!live()) return false;
|
||||
let outcome: string | null = null;
|
||||
try {
|
||||
await deps.queueRepo.create({
|
||||
qitemId: localId,
|
||||
sourceSession: LOCAL_SOURCE,
|
||||
destinationSession: destination,
|
||||
// Where it came from is read first: the requester and the hub, before the requester's own words.
|
||||
summary: full.summary ? `From ${item.sourceSession} via network hub: ${full.summary}` : `From ${item.sourceSession} via network hub`,
|
||||
body: [
|
||||
`From ${item.sourceSession} through the network hub ${binding.hubUrl}, as ${binding.hubSeat} (hub item ${item.qitemId}).`,
|
||||
"",
|
||||
full.body ?? "",
|
||||
"",
|
||||
"---",
|
||||
`Answer by closing this item with your result: rig queue update ${localId} --state done --closure-reason no-follow-on --resolution "<your answer>".`,
|
||||
"OpenRig returns the resolution to the hub item.",
|
||||
].join("\n"),
|
||||
tags: [`network-binding:${binding.id}`, `network-hub-item:${item.qitemId}`],
|
||||
});
|
||||
} catch (error) {
|
||||
const code = error instanceof QueueRepositoryError ? error.code : null;
|
||||
// Its id was used before and its record erased: whether it ran is unknown, and it's never run again. The hub hears
|
||||
// that from reportBack, which retries the note until it lands.
|
||||
if (code === "qitem_erased") outcome = "unknown";
|
||||
// The same id with another destination: the seat's address changed between attempts; the item already exists.
|
||||
else if (code !== "qitem_id_reuse" || !deps.queueRepo.getById(localId)) throw error;
|
||||
}
|
||||
const inserted = deps.db.prepare(`INSERT INTO network_receipts (local_qitem_id, binding_id, hub_qitem_id, created_at, outcome)
|
||||
VALUES (?, ?, ?, ?, ?) ON CONFLICT DO NOTHING`).run(localId, binding.id, item.qitemId, now().toISOString(), outcome).changes > 0;
|
||||
// Every stage note, queued included, goes out from reportBack, which marks it sent only once it lands.
|
||||
return inserted && outcome === null;
|
||||
}
|
||||
|
||||
/** The item a local item ends at, following local handoffs: a successor names the item it came from. Null when a
|
||||
* handed-off item has no one successor here (handed to another host, or the record is gone): its outcome is unknown. */
|
||||
function finalItem(repo: QueueRepository, start: QueueItem): QueueItem | null {
|
||||
let current = start;
|
||||
for (let hop = 0; hop < 20 && current.state === "handed-off"; hop++) {
|
||||
const next = repo.list({ handedOffFrom: current.qitemId, limit: 2 });
|
||||
if (next.length !== 1) return null;
|
||||
current = next[0]!;
|
||||
}
|
||||
return current.state === "handed-off" ? null : current;
|
||||
}
|
||||
|
||||
/** What the hub hears as the result: the resolution, else the closer's own note on the close itself. The close is read
|
||||
* with `closingTransition`, since later notes on a closed item repeat its state. Other notes on the item can be
|
||||
* OpenRig's or another seat's, and name local seats, so they're never sent. */
|
||||
function closingWords(repo: QueueRepository, item: QueueItem): string {
|
||||
if (item.resolution) return item.resolution;
|
||||
const close = repo.closingTransition(item.qitemId, item.state);
|
||||
return close?.transitionNote && close.actorSession === item.destinationSession ? close.transitionNote : "Closed without a stated result.";
|
||||
}
|
||||
|
||||
/** The hub items a binding's result never reached because someone else closed them first, for status. */
|
||||
export function networkClosedElsewhere(db: Database.Database, bindingId: string): Array<{ hubItem: string; localItem: string; note: string | null }> {
|
||||
return (db.prepare("SELECT hub_qitem_id, local_qitem_id, last_error FROM network_receipts WHERE binding_id = ? AND outcome = 'closed_on_hub' ORDER BY settled_at, local_qitem_id")
|
||||
.all(bindingId) as Array<{ hub_qitem_id: string; local_qitem_id: string; last_error: string | null }>)
|
||||
.map((row) => ({ hubItem: row.hub_qitem_id, localItem: row.local_qitem_id, note: row.last_error }));
|
||||
}
|
||||
|
||||
async function reportBack(deps: NetworkConnectorDeps, binding: NetworkBinding, hub: HubSession, now: () => Date, result: NetworkPassResult): Promise<void> {
|
||||
const open = deps.db.prepare(`SELECT * FROM network_receipts WHERE binding_id = ? AND settled_at IS NULL
|
||||
ORDER BY COALESCE(checked_at, ''), created_at LIMIT ?`).all(binding.id, deps.reportLimit ?? REPORT_LIMIT) as Receipt[];
|
||||
const mark = (column: "queued_sent_at" | "pickup_sent_at" | "settled_at", receipt: Receipt, outcome?: string, lastError: string | null = null) =>
|
||||
deps.db.prepare(`UPDATE network_receipts SET ${column} = ?, outcome = COALESCE(?, outcome), last_error = ? WHERE local_qitem_id = ?`)
|
||||
.run(now().toISOString(), outcome ?? null, lastError, receipt.local_qitem_id);
|
||||
for (const receipt of open) {
|
||||
deps.db.prepare("UPDATE network_receipts SET checked_at = ? WHERE local_qitem_id = ?").run(now().toISOString(), receipt.local_qitem_id);
|
||||
try {
|
||||
const local = deps.queueRepo.getById(receipt.local_qitem_id);
|
||||
const unknown = async (why: string) => {
|
||||
await hub.note(receipt.hub_qitem_id, why);
|
||||
mark("settled_at", receipt, "unknown");
|
||||
result.unknown++;
|
||||
};
|
||||
if (!local || receipt.outcome === "unknown") {
|
||||
await unknown(`This item's local record (${receipt.local_qitem_id}) is gone, so whether it ran is unknown. It won't be run again.`);
|
||||
continue;
|
||||
}
|
||||
if (!receipt.queued_sent_at) {
|
||||
await hub.note(receipt.hub_qitem_id, `Queued for the connected local seat as ${local.qitemId}. Not picked up yet.`);
|
||||
mark("queued_sent_at", receipt);
|
||||
}
|
||||
if (!receipt.pickup_sent_at && local.claimedAt) {
|
||||
await hub.note(receipt.hub_qitem_id, `Picked up by the local seat at ${local.claimedAt}.`);
|
||||
mark("pickup_sent_at", receipt);
|
||||
result.pickupsSent++;
|
||||
}
|
||||
const end = local.state === "handed-off" ? finalItem(deps.queueRepo, local) : local;
|
||||
if (!end) {
|
||||
await unknown(`This item was handed on from ${local.qitemId} without one successor here, so its outcome is unknown. It won't be run again.`);
|
||||
continue;
|
||||
}
|
||||
if (!FINAL.has(end.state)) continue;
|
||||
// The hub item closes the way the queue closes one: done with the local closure reason, or failed. A refusal, a
|
||||
// cancel or a failure never reads as success.
|
||||
const reason = end.state === "canceled" || end.closureReason === "canceled" ? "canceled"
|
||||
: end.state === "denied" || end.closureReason === "denied" ? "denied" : "no-follow-on";
|
||||
const state = end.state === "failed" ? "failed" as const : "done" as const;
|
||||
const closureReason = state === "failed" ? null : reason;
|
||||
const words = closingWords(deps.queueRepo, end);
|
||||
// A final check before the close: a hub item already closed, by our own close whose answer was lost or by someone
|
||||
// else, gets no write, so another actor's failure or cancel never becomes our done.
|
||||
let shown = await hub.show(receipt.hub_qitem_id);
|
||||
if (!HUB_CLOSED.has(shown.state)) {
|
||||
try {
|
||||
await hub.close(receipt.hub_qitem_id, { state, closureReason, resolution: words });
|
||||
} catch (error) {
|
||||
// Refused: closed by someone else between the check and the close. The readback says how.
|
||||
if (!(error instanceof HubError) || error.kind !== "refused") throw error;
|
||||
}
|
||||
shown = await hub.show(receipt.hub_qitem_id);
|
||||
if (!HUB_CLOSED.has(shown.state)) {
|
||||
deps.db.prepare("UPDATE network_receipts SET last_error = ? WHERE local_qitem_id = ?").run("The close didn't take: the hub item still reads open. Trying again.", receipt.local_qitem_id);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
// It counts as our result only when the hub item shows our state, closure reason and resolution. Otherwise the
|
||||
// local result stays here, and status shows the hub item as closed elsewhere.
|
||||
if (shown.state !== state || shown.resolution !== words || (shown.closureReason ?? null) !== closureReason) {
|
||||
mark("settled_at", receipt, "closed_on_hub", `The hub item was closed elsewhere (${shown.state}${shown.closureReason ? `, ${shown.closureReason}` : ""}) before this result reached it. The result stays on ${local.qitemId}.`);
|
||||
deps.log?.(`network ${binding.id}: hub item ${receipt.hub_qitem_id} was closed elsewhere; its local result stays on ${local.qitemId}`);
|
||||
continue;
|
||||
}
|
||||
mark("settled_at", receipt, end.state);
|
||||
result.resultsSent++;
|
||||
} catch (error) {
|
||||
if (!(error instanceof HubError) || error.kind !== "refused") throw error;
|
||||
deps.db.prepare("UPDATE network_receipts SET last_error = ? WHERE local_qitem_id = ?").run(error.message.slice(0, 500), receipt.local_qitem_id);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export interface NetworkConnectorSnapshot {
|
||||
running: boolean;
|
||||
intervalMs: number;
|
||||
bindings: Array<{ id: string; nextPassAt: string | null; backoffMs: number; lastPass: NetworkPassResult | null }>;
|
||||
}
|
||||
|
||||
/** The connector the daemon runs: one pass per active binding when it's due, one round at a time, off the event loop's
|
||||
* hot path (the hub calls are async network I/O). Each pass is bounded by its deadline, and stop closes a pass in
|
||||
* flight. A binding that fails backs off, doubling to a cap, with jitter. With no active binding it makes no network
|
||||
* call at all. */
|
||||
export class NetworkConnectorService {
|
||||
private timer: ReturnType<typeof setTimeout> | null = null;
|
||||
private inFlight: Promise<void> | null = null;
|
||||
private stopped = false;
|
||||
private stopping: AbortController | null = null;
|
||||
private readonly due = new Map<string, { at: number; backoffMs: number; last: NetworkPassResult | null }>();
|
||||
readonly intervalMs: number;
|
||||
private readonly maxBackoffMs: number;
|
||||
|
||||
constructor(private readonly deps: NetworkConnectorDeps, opts: { intervalMs?: number; maxBackoffMs?: number; random?: () => number } = {}) {
|
||||
this.intervalMs = opts.intervalMs ?? 30_000;
|
||||
this.maxBackoffMs = Math.max(this.intervalMs, opts.maxBackoffMs ?? 300_000);
|
||||
this.random = opts.random ?? Math.random;
|
||||
}
|
||||
private readonly random: () => number;
|
||||
|
||||
start(): void {
|
||||
this.stopped = false;
|
||||
this.stopping = new AbortController();
|
||||
// A credential whose binding isn't active has nothing left to do here: one a stop left before its row, or one
|
||||
// whose removal failed when its binding ended.
|
||||
const swept = this.deps.bindings.sweepCredentials();
|
||||
if (swept.removed.length) this.deps.log?.(`network connector: removed the credentials of ended bindings ${swept.removed.join(", ")}`);
|
||||
if (swept.failed.length) this.deps.log?.(`network connector: couldn't remove the credentials of ended bindings ${swept.failed.join(", ")}`);
|
||||
this.schedule(0);
|
||||
}
|
||||
|
||||
async stop(): Promise<void> {
|
||||
this.stopped = true;
|
||||
this.stopping?.abort();
|
||||
if (this.timer) clearTimeout(this.timer);
|
||||
this.timer = null;
|
||||
await this.inFlight;
|
||||
}
|
||||
|
||||
/** One round: a pass for each active binding that's due. Exposed for tests and status. */
|
||||
async round(): Promise<void> {
|
||||
if (this.inFlight) return this.inFlight;
|
||||
this.inFlight = (async () => {
|
||||
const at = (this.deps.now ?? (() => new Date()))().getTime();
|
||||
const active = this.deps.bindings.active();
|
||||
for (const id of [...this.due.keys()]) if (!active.some((b) => b.id === id)) this.due.delete(id);
|
||||
for (const binding of active) {
|
||||
if (this.stopped) break;
|
||||
const due = this.due.get(binding.id);
|
||||
if (due && due.at > at) continue;
|
||||
const pass = await runNetworkBindingPass(this.deps, binding, { signal: this.stopping?.signal });
|
||||
const failing = pass.error === "unavailable" || pass.error === "refused" || pass.error === "credential" || pass.error === "local";
|
||||
const backoffMs = failing ? Math.min(this.maxBackoffMs, (due?.backoffMs ?? this.intervalMs) * 2) : this.intervalMs;
|
||||
const jitter = 1 + (this.random() * 0.4 - 0.2);
|
||||
this.due.set(binding.id, { at: at + Math.round(backoffMs * jitter), backoffMs, last: pass });
|
||||
}
|
||||
})();
|
||||
try { await this.inFlight; } finally { this.inFlight = null; }
|
||||
}
|
||||
|
||||
snapshot(): NetworkConnectorSnapshot {
|
||||
return {
|
||||
running: !this.stopped,
|
||||
intervalMs: this.intervalMs,
|
||||
bindings: [...this.due.entries()].map(([id, d]) => ({ id, nextPassAt: new Date(d.at).toISOString(), backoffMs: d.backoffMs, lastPass: d.last })),
|
||||
};
|
||||
}
|
||||
|
||||
private schedule(ms: number): void {
|
||||
if (this.stopped) return;
|
||||
this.timer = setTimeout(() => {
|
||||
void this.round().catch((error) => this.deps.log?.(`network connector round failed: ${(error as Error).message}`))
|
||||
.finally(() => this.schedule(Math.min(this.intervalMs, 5_000)));
|
||||
}, ms);
|
||||
this.timer.unref?.();
|
||||
}
|
||||
}
|
||||
|
||||
export { HubError };
|
||||
@@ -384,6 +384,7 @@ export async function startServer(port?: number) {
|
||||
// tree post-bind so the HTTP surface is ready before the first
|
||||
// tick (matches contextMonitor pattern).
|
||||
deps.watchdogScheduler?.start();
|
||||
deps.networkConnector?.start();
|
||||
deps.sessionTransport?.guardedDelivery?.recover();
|
||||
// S10 — start the gateway subsystem's NETWORK services (replay, outbound poll,
|
||||
// Socket Mode inbound) post-bind, contextMonitor pattern: the composed wire is
|
||||
@@ -452,6 +453,7 @@ export async function startServer(port?: number) {
|
||||
["proof-source-watch", () => deps.proofSourceWatch?.close()],
|
||||
["health-diagnosis", () => deps.healthDiagnosis?.stop()],
|
||||
["watchdog", () => deps.watchdogScheduler?.stop()],
|
||||
["network-connector", () => deps.networkConnector?.stop()],
|
||||
["stream-intake", () => deps.streamIntakeCoordinator?.stop()],
|
||||
["draft-delivery", () => deps.sessionTransport?.guardedDelivery?.stop()],
|
||||
["seat-activity", () => deps.seatActivityService?.stop()],
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
import { Hono, type Context } from "hono";
|
||||
import type { RigRepository } from "../domain/rig-repository.js";
|
||||
import { NetworkBindingError, NetworkBindingStore } from "../domain/network/network-bindings.js";
|
||||
import { networkClosedElsewhere } from "../domain/network/network-connector.js";
|
||||
import { authBearerTokenMiddleware } from "../middleware/auth-bearer-token.js";
|
||||
import { OPENRIG_HOME } from "../openrig-compat.js";
|
||||
|
||||
/** A local seat's connections to a remote hub (`rig network`). Reads list every binding with its state and last contact;
|
||||
* the token is accepted on connect and never returned. Writes are gated like the hosts routes: enforced when an operator
|
||||
* bearer is configured, passed through on a loopback or tailnet-trusted daemon. */
|
||||
export function networkRoutes(opts?: { bearerToken?: string | null; home?: string }): Hono {
|
||||
const router = new Hono();
|
||||
const requireAuth = authBearerTokenMiddleware({ expectedToken: opts?.bearerToken ?? null });
|
||||
router.use("/bindings", async (c, next) => (c.req.method === "GET" ? next() : requireAuth(c, next)));
|
||||
router.use("/bindings/:id/disconnect", requireAuth);
|
||||
|
||||
const store = (c: Context) => new NetworkBindingStore((c.get("rigRepo" as never) as RigRepository).db, opts?.home ?? OPENRIG_HOME);
|
||||
const refused = (c: Context, error: unknown) => {
|
||||
if (error instanceof NetworkBindingError) return c.json({ error: error.code, message: error.message }, error.status);
|
||||
throw error;
|
||||
};
|
||||
|
||||
// The connector's own view (next pass, backoff, last pass) when it runs in this daemon; null otherwise.
|
||||
router.get("/bindings", (c) => {
|
||||
const connector = c.get("networkConnector" as never) as { snapshot(): unknown } | undefined;
|
||||
const db = (c.get("rigRepo" as never) as RigRepository).db;
|
||||
// Hub items someone else closed before a binding's result reached them: a conflict the person sees.
|
||||
const bindings = store(c).list().map((b) => ({ ...b, closedElsewhere: networkClosedElsewhere(db, b.id) }));
|
||||
return c.json({ bindings, connector: connector?.snapshot() ?? null });
|
||||
});
|
||||
|
||||
router.post("/bindings", async (c) => {
|
||||
let body: Record<string, unknown>;
|
||||
try { body = await c.req.json(); } catch { return c.json({ error: "invalid_body", message: "The body must be JSON." }, 400); }
|
||||
try {
|
||||
return c.json(store(c).create({ session: body.session, hubUrl: body.hubUrl, hubSeat: body.hubSeat, token: body.token, allowHttp: body.allowHttp }), 201);
|
||||
} catch (error) { return refused(c, error); }
|
||||
});
|
||||
|
||||
router.post("/bindings/:id/disconnect", async (c) => {
|
||||
let body: Record<string, unknown> = {};
|
||||
try { body = await c.req.json(); } catch { /* no reason given */ }
|
||||
try {
|
||||
return c.json(store(c).disconnect(c.req.param("id"), typeof body.reason === "string" ? body.reason : undefined));
|
||||
} catch (error) { return refused(c, error); }
|
||||
});
|
||||
|
||||
return router;
|
||||
}
|
||||
@@ -84,6 +84,7 @@ import { skillsRoutes } from "./routes/skills.js";
|
||||
import type { SkillLibraryDiscoveryService } from "./domain/skill-library-discovery.js";
|
||||
import { configRoutes } from "./routes/config.js";
|
||||
import { hostsRoutes } from "./routes/hosts.js";
|
||||
import { networkRoutes } from "./routes/network.js";
|
||||
import { hostReadThrough } from "./domain/hosts/read-through.js";
|
||||
import { jsonBodyErrorHandler, trackJsonBodyParseErrors } from "./middleware/json-body-error.js";
|
||||
import { getSelfHostId, getSelfHostIdSource } from "./domain/hosts/fanout-contract.js";
|
||||
@@ -242,6 +243,8 @@ export interface AppDeps {
|
||||
watchdogHistoryLog?: WatchdogHistoryLog;
|
||||
watchdogPolicyEngine?: WatchdogPolicyEngine;
|
||||
watchdogScheduler?: WatchdogScheduler;
|
||||
/** The outbound network connector (`rig network`); started after the listener binds. */
|
||||
networkConnector?: import("./domain/network/network-connector.js").NetworkConnectorService;
|
||||
/** B8 / slice-07 A3 — the model-divergence monitor (effective-vs-pinned, four-channel proclaim). */
|
||||
modelDivergenceMonitor?: import("./domain/model-divergence/model-divergence-monitor.js").ModelDivergenceMonitor;
|
||||
/** S10 — the in-daemon gateway subsystem (amended M1 §3: in-process, no second deployable).
|
||||
@@ -634,6 +637,7 @@ export function createApp(deps: AppDeps): Hono {
|
||||
c.set("watchdogHistoryLog" as never, deps.watchdogHistoryLog);
|
||||
c.set("watchdogPolicyEngine" as never, deps.watchdogPolicyEngine);
|
||||
c.set("watchdogScheduler" as never, deps.watchdogScheduler);
|
||||
c.set("networkConnector" as never, deps.networkConnector);
|
||||
c.set("workflowRuntime" as never, deps.workflowRuntime);
|
||||
c.set("workflowBuiltinSpecsDir" as never, deps.workflowBuiltinSpecsDir);
|
||||
c.set("workflowsFolderDir" as never, deps.workflowsFolderDir);
|
||||
@@ -872,6 +876,9 @@ export function createApp(deps: AppDeps): Hono {
|
||||
"/api/hosts",
|
||||
hostsRoutes({ bearerToken: deps.missionControlBearerToken ?? null }),
|
||||
);
|
||||
// A selected local seat's connections to a remote hub (`rig network`): bindings, status and disconnect. Its writes carry
|
||||
// a participant's token, so they take the same operator-bearer posture as the hosts routes.
|
||||
app.route("/api/network", networkRoutes({ bearerToken: deps.missionControlBearerToken ?? null }));
|
||||
// Slice Story View v0 — slice indexer + per-tab payload routes.
|
||||
app.route("/api/slices", slicesRoutes());
|
||||
// Living Notes Packet 2 (OPR.0.4.4.20) — composed-review read contract.
|
||||
|
||||
@@ -2544,6 +2544,19 @@ export async function createDaemon(opts?: DaemonOptions): Promise<DaemonResult>
|
||||
});
|
||||
deps.watchdogPolicyEngine = watchdogPolicyEngine;
|
||||
deps.watchdogScheduler = watchdogScheduler;
|
||||
// The outbound network connector: a pass per active `rig network` binding. With none, it makes no network call.
|
||||
{
|
||||
const { NetworkConnectorService } = await import("./domain/network/network-connector.js");
|
||||
const { NetworkBindingStore } = await import("./domain/network/network-bindings.js");
|
||||
const pollSeconds = Number(process.env.OPENRIG_NETWORK_POLL_SECONDS);
|
||||
deps.networkConnector = new NetworkConnectorService({
|
||||
db: queueRepoInstance.db,
|
||||
queueRepo: queueRepoInstance,
|
||||
bindings: new NetworkBindingStore(queueRepoInstance.db, OPENRIG_HOME),
|
||||
home: OPENRIG_HOME,
|
||||
log: (line) => console.log(`[network] ${line}`),
|
||||
}, Number.isFinite(pollSeconds) && pollSeconds >= 5 ? { intervalMs: pollSeconds * 1000 } : {});
|
||||
}
|
||||
|
||||
// OPR.0.5.6.24 F-14 — ONE parked-owner supervisor job per rig (mini-req 1):
|
||||
// ensured idempotently on the (policy, target_session) tuple; the anchor is
|
||||
|
||||
@@ -4,6 +4,8 @@ import { queueErasureSchema } from "../../src/db/migrations/155_queue_erasure.js
|
||||
import { queueErasureBatchesSchema } from "../../src/db/migrations/157_queue_erasure_batches.js";
|
||||
import { queueInboxCheckinsSchema } from "../../src/db/migrations/158_queue_inbox_checkins.js";
|
||||
import { queueItemsHandedOffFromIndexSchema } from "../../src/db/migrations/159_queue_items_handed_off_from_index.js";
|
||||
import { networkBindingsSchema } from "../../src/db/migrations/160_network_bindings.js";
|
||||
import { networkReceiptsSchema } from "../../src/db/migrations/161_network_receipts.js";
|
||||
import { activityGrantsSchema } from "../../src/db/migrations/143_activity_grants.js";
|
||||
import { environmentBindingsSchema } from "../../src/db/migrations/144_environment_bindings.js";
|
||||
import { nodePermissionsToSchema } from "../../src/db/migrations/145_node_permissions_to.js";
|
||||
@@ -166,7 +168,7 @@ import fs from "node:fs";
|
||||
|
||||
/** Seam B R6: the canonical full-fixture migration list, exported so file-backed
|
||||
* DB-reopen tests migrate IDENTICALLY to createFullTestDb. */
|
||||
export const migrationsForFullTestDb = [coreSchema, bindingsSessionsSchema, eventsSchema, snapshotsSchema, checkpointsSchema, resumeMetadataSchema, nodeSpecFieldsSchema, packagesSchema, installJournalSchema, journalSeqSchema, bootstrapSchema, discoverySchema, discoveryFkFix, agentspecRebootSchema, startupContextSchema, chatMessagesSchema, podNamespaceSchema, contextUsageSchema, externalCliAttachmentSchema, rigServicesSchema, seatHandoverObservabilitySchema, nodeCodexConfigProfileSchema, nodePermissionPolicySchema, rigPermissionPolicySchema, nodePolicyProvenanceSchema, rigPolicyProvenanceSchema, streamItemsSchema, queueItemsSchema, queueTransitionsSchema, rigPolicySchema, rigArchiveSchema, resumeProvenanceSchema, resumeVerificationSchema, seatIdentityVerdictsSchema, selfHostIdentitySchema, occupantTenuresSchema, daemonLifecycleSchema, watchdogJobsSchema, occupantGenerationStampsSchema, projectionManifestSchema, watchdogTargetGenerationSchema, appliedLaunchObservationsSchema, appliedLaunchObservationInvalidationsSchema, threadSeatMapSchema, queueTransitionWakesSchema, nodeSessionSourceSchema, scopedOperatingPostureSchema, humanNotificationIntentSchema, reviewReadIndexesSchema, inventoryEventIndexesSchema, rigClaudeManagedBlockFileSchema, nodePermissionSelectionsSchema, humanReplyToSchema, humanQuestionsSchema, nodeEffortSchema, rigNonInterruptiveSchema, rigInstallRootSchema, threadPartMapSchema, queueTransitionsQitemIdOrderSchema, queueItemsRecoveryTagsSchema, resumeRotationSchema, sessionStopIntentSchema, nodeEffortChoiceSchema, modePostureCoexistSchema, seatEnvSchema, appliedLaunchDaemonOptOutSchema, sessionCommunityHarnessSchema, threadSeatMapChannelKeySchema, queueItemWakeInputBoxSchema, providerAccountReadingsSchema, humanQuietWindowsSchema, seatIdentityPanePidSchema, seatIdentityProcessEvidenceSchema, rigStartupLayersSchema, humanOperatingModesSchema, queueDelegatedDecisionSchema, nodeAdvisorSchema, operationCorrelationSchema, terminalPolicySchema, seatRuntimeSwitchesSchema, claudeTokenReadingsSchema, nodeLaunchRecipeSchema, nodeClaudeTokenRefSchema, nodeClaudePermissionsSchema, nodeCodexPermissionsSchema, escalationTargetRefsSchema, sessionAgentRunnerSchema, nodeAccountHomeSchema, nodeClaudeMcpSchema, activityGrantsSchema, environmentBindingsSchema, nodePermissionsToSchema, nodeStartupGuidanceFileSchema, managedBlockWritesSchema, streamIntakeCursorSchema, nodeLaunchWrapperSchema, piAccountSwitchesSchema, eventsRetentionMarksSchema, queueErasureSchema, queueErasureBatchesSchema, queueInboxCheckinsSchema, queueItemsHandedOffFromIndexSchema];
|
||||
export const migrationsForFullTestDb = [coreSchema, bindingsSessionsSchema, eventsSchema, snapshotsSchema, checkpointsSchema, resumeMetadataSchema, nodeSpecFieldsSchema, packagesSchema, installJournalSchema, journalSeqSchema, bootstrapSchema, discoverySchema, discoveryFkFix, agentspecRebootSchema, startupContextSchema, chatMessagesSchema, podNamespaceSchema, contextUsageSchema, externalCliAttachmentSchema, rigServicesSchema, seatHandoverObservabilitySchema, nodeCodexConfigProfileSchema, nodePermissionPolicySchema, rigPermissionPolicySchema, nodePolicyProvenanceSchema, rigPolicyProvenanceSchema, streamItemsSchema, queueItemsSchema, queueTransitionsSchema, rigPolicySchema, rigArchiveSchema, resumeProvenanceSchema, resumeVerificationSchema, seatIdentityVerdictsSchema, selfHostIdentitySchema, occupantTenuresSchema, daemonLifecycleSchema, watchdogJobsSchema, occupantGenerationStampsSchema, projectionManifestSchema, watchdogTargetGenerationSchema, appliedLaunchObservationsSchema, appliedLaunchObservationInvalidationsSchema, threadSeatMapSchema, queueTransitionWakesSchema, nodeSessionSourceSchema, scopedOperatingPostureSchema, humanNotificationIntentSchema, reviewReadIndexesSchema, inventoryEventIndexesSchema, rigClaudeManagedBlockFileSchema, nodePermissionSelectionsSchema, humanReplyToSchema, humanQuestionsSchema, nodeEffortSchema, rigNonInterruptiveSchema, rigInstallRootSchema, threadPartMapSchema, queueTransitionsQitemIdOrderSchema, queueItemsRecoveryTagsSchema, resumeRotationSchema, sessionStopIntentSchema, nodeEffortChoiceSchema, modePostureCoexistSchema, seatEnvSchema, appliedLaunchDaemonOptOutSchema, sessionCommunityHarnessSchema, threadSeatMapChannelKeySchema, queueItemWakeInputBoxSchema, providerAccountReadingsSchema, humanQuietWindowsSchema, seatIdentityPanePidSchema, seatIdentityProcessEvidenceSchema, rigStartupLayersSchema, humanOperatingModesSchema, queueDelegatedDecisionSchema, nodeAdvisorSchema, operationCorrelationSchema, terminalPolicySchema, seatRuntimeSwitchesSchema, claudeTokenReadingsSchema, nodeLaunchRecipeSchema, nodeClaudeTokenRefSchema, nodeClaudePermissionsSchema, nodeCodexPermissionsSchema, escalationTargetRefsSchema, sessionAgentRunnerSchema, nodeAccountHomeSchema, nodeClaudeMcpSchema, activityGrantsSchema, environmentBindingsSchema, nodePermissionsToSchema, nodeStartupGuidanceFileSchema, managedBlockWritesSchema, streamIntakeCursorSchema, nodeLaunchWrapperSchema, piAccountSwitchesSchema, eventsRetentionMarksSchema, queueErasureSchema, queueErasureBatchesSchema, queueInboxCheckinsSchema, queueItemsHandedOffFromIndexSchema, networkBindingsSchema, networkReceiptsSchema];
|
||||
|
||||
/**
|
||||
* P24 — the DECLARED exclusions for {@link migrationsForFullTestDb}. That list is deliberately a
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
// A bind whose credential write fails partway leaves neither a credential nor a binding behind.
|
||||
import { afterEach, beforeEach, expect, it, vi } from "vitest";
|
||||
|
||||
const failWrite = { on: false };
|
||||
vi.mock("node:fs", async (importOriginal) => {
|
||||
const real = await importOriginal<typeof import("node:fs")>();
|
||||
return {
|
||||
...real,
|
||||
writeFileSync: (...args: Parameters<typeof real.writeFileSync>) => {
|
||||
// A partial write: some bytes reach the file, then the disk is full.
|
||||
if (failWrite.on && typeof args[0] === "number") {
|
||||
real.writeSync(args[0], "{\"version\":1,");
|
||||
throw Object.assign(new Error("ENOSPC: no space left on device, write"), { code: "ENOSPC" });
|
||||
}
|
||||
return real.writeFileSync(...args);
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
const { default: Database } = await import("better-sqlite3");
|
||||
const { existsSync, mkdtempSync, rmSync } = await import("node:fs");
|
||||
const { tmpdir } = await import("node:os");
|
||||
const { join } = await import("node:path");
|
||||
const { ALL_MIGRATIONS } = await import("../src/db/all-migrations.js");
|
||||
const { migrate } = await import("../src/db/migrate.js");
|
||||
const { RigRepository } = await import("../src/domain/rig-repository.js");
|
||||
const { NetworkBindingStore, networkCredentialFile } = await import("../src/domain/network/network-bindings.js");
|
||||
|
||||
let db: InstanceType<typeof Database>;
|
||||
let home: string;
|
||||
beforeEach(() => {
|
||||
db = new Database(":memory:");
|
||||
migrate(db, ALL_MIGRATIONS);
|
||||
const rigs = new RigRepository(db);
|
||||
const node = rigs.addNode(rigs.createRig("lab").id, "impl", { runtime: "claude-code" });
|
||||
db.prepare("INSERT INTO sessions (id, node_id, session_name, status) VALUES ('s1', ?, 'a@lab', 'running')").run(node.id);
|
||||
home = mkdtempSync(join(tmpdir(), "openrig-network-write-"));
|
||||
});
|
||||
afterEach(() => { failWrite.on = false; db.close(); rmSync(home, { recursive: true, force: true }); });
|
||||
|
||||
it("removes the partly written credential and its folder, and records no binding", () => {
|
||||
const id = "netb-00000000000000000003";
|
||||
const store = new NetworkBindingStore(db, home, () => new Date(), () => id);
|
||||
failWrite.on = true;
|
||||
expect(() => store.create({ session: "a@lab", hubUrl: "https://hub.example/mcp", hubSeat: "alex@team", token: "tok" })).toThrow(/ENOSPC/);
|
||||
expect(existsSync(networkCredentialFile(home, id))).toBe(false);
|
||||
expect(existsSync(join(networkCredentialFile(home, id), ".."))).toBe(false);
|
||||
expect(store.list()).toHaveLength(0);
|
||||
failWrite.on = false;
|
||||
expect(store.create({ session: "a@lab", hubUrl: "https://hub.example/mcp", hubSeat: "alex@team", token: "tok" }).state).toBe("active");
|
||||
});
|
||||
@@ -0,0 +1,260 @@
|
||||
// `rig network` bindings: a selected local seat joined to one hub participant. The credential lives in a private file
|
||||
// and never comes back over the API; disconnect stops the binding and removes the credential, and local work stays.
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import Database from "better-sqlite3";
|
||||
import { Hono } from "hono";
|
||||
import { existsSync, mkdtempSync, readFileSync, rmSync, statSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { ALL_MIGRATIONS } from "../src/db/all-migrations.js";
|
||||
import { migrate } from "../src/db/migrate.js";
|
||||
import { RigRepository } from "../src/domain/rig-repository.js";
|
||||
import { NetworkBindingError, NetworkBindingStore, networkCredentialFile, readNetworkCredential } from "../src/domain/network/network-bindings.js";
|
||||
import { networkRoutes } from "../src/routes/network.js";
|
||||
|
||||
describe("network bindings", () => {
|
||||
let db: Database.Database;
|
||||
let rigs: RigRepository;
|
||||
let home: string;
|
||||
let store: NetworkBindingStore;
|
||||
let seq = 0;
|
||||
beforeEach(() => {
|
||||
db = new Database(":memory:");
|
||||
migrate(db, ALL_MIGRATIONS);
|
||||
rigs = new RigRepository(db);
|
||||
home = mkdtempSync(join(tmpdir(), "openrig-network-"));
|
||||
store = new NetworkBindingStore(db, home);
|
||||
});
|
||||
afterEach(() => { db.close(); rmSync(home, { recursive: true, force: true }); });
|
||||
|
||||
function seat(session: string) {
|
||||
const rig = rigs.listRigs().find((r) => r.name === "lab") ?? rigs.createRig("lab");
|
||||
const node = rigs.addNode(rig.id, `n${++seq}`, { runtime: "claude-code" });
|
||||
db.prepare("INSERT INTO sessions (id, node_id, session_name, status) VALUES (?, ?, ?, 'running')").run(`s${seq}`, node.id, session);
|
||||
return node;
|
||||
}
|
||||
const connect = (session: string, over: Record<string, unknown> = {}) =>
|
||||
store.create({ session, hubUrl: "https://hub.example/mcp", hubSeat: "alex@team", token: "tok-123", ...over });
|
||||
|
||||
it("binds a local seat by its node, keeps the token in a private file, and never lists it", () => {
|
||||
const node = seat("dev-impl@lab");
|
||||
const binding = connect("dev-impl@lab");
|
||||
expect(binding).toMatchObject({ nodeId: node.id, localSession: "dev-impl@lab", hubUrl: "https://hub.example/mcp", hubSeat: "alex@team",
|
||||
state: "active", lastContactAt: null });
|
||||
expect(binding.id).toMatch(/^netb-[0-9a-f]{20}$/);
|
||||
const file = networkCredentialFile(home, binding.id);
|
||||
expect(statSync(file).mode & 0o777).toBe(0o600);
|
||||
expect(statSync(join(file, "..")).mode & 0o777).toBe(0o700);
|
||||
expect(readNetworkCredential(home, binding.id)).toBe("tok-123");
|
||||
expect(JSON.stringify(store.list())).not.toContain("tok-123");
|
||||
});
|
||||
|
||||
it("refuses an unknown seat, a second binding for a seat or a participant, and a credential in the URL", () => {
|
||||
expect(() => connect("nobody@lab")).toThrow(expect.objectContaining({ code: "seat_not_found", status: 404 }));
|
||||
seat("a@lab"); seat("b@lab");
|
||||
connect("a@lab");
|
||||
expect(() => connect("a@lab", { hubSeat: "other@team" })).toThrow(expect.objectContaining({ code: "seat_already_connected", status: 409 }));
|
||||
expect(() => connect("b@lab")).toThrow(expect.objectContaining({ code: "participant_already_connected", status: 409 }));
|
||||
expect(() => connect("b@lab", { hubSeat: "c@team", hubUrl: "https://user:[email protected]/mcp" })).toThrow(expect.objectContaining({ code: "invalid_hub_url" }));
|
||||
expect(() => connect("b@lab", { hubSeat: "c@team", token: "two words" })).toThrow(expect.objectContaining({ code: "invalid_token" }));
|
||||
expect(() => connect("b@lab", { hubSeat: "not-an-address" })).toThrow(expect.objectContaining({ code: "invalid_hub_seat" }));
|
||||
// Every refusal above left no credential behind.
|
||||
expect(store.list()).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("plain http needs the person's choice: refused with the remedy, then recorded and marked unencrypted with --allow-http", () => {
|
||||
seat("a@lab");
|
||||
const http = { hubUrl: "http://hub.internal:7433/mcp" };
|
||||
let message = "";
|
||||
try { connect("a@lab", http); } catch (error) {
|
||||
expect(error).toMatchObject({ code: "http_not_allowed", status: 400 });
|
||||
message = (error as Error).message;
|
||||
}
|
||||
expect(message).toContain("unencrypted");
|
||||
expect(message).toContain("https URL");
|
||||
expect(message).toContain("--allow-http");
|
||||
expect(store.list()).toHaveLength(0);
|
||||
expect(existsSync(join(home, "network-secrets"))).toBe(false);
|
||||
|
||||
const binding = connect("a@lab", { ...http, allowHttp: true });
|
||||
// The scheme is kept as given, never switched.
|
||||
expect(binding).toMatchObject({ hubUrl: "http://hub.internal:7433/mcp", allowHttp: true, unencrypted: true });
|
||||
expect(store.list()).toEqual([expect.objectContaining({ id: binding.id, allowHttp: true, unencrypted: true })]);
|
||||
expect(JSON.stringify(store.list())).not.toContain("tok-123");
|
||||
// https needs no choice and isn't marked.
|
||||
seat("b@lab");
|
||||
expect(connect("b@lab", { hubSeat: "b@team" })).toMatchObject({ hubUrl: "https://hub.example/mcp", allowHttp: false, unencrypted: false });
|
||||
});
|
||||
|
||||
it("disconnect ends the binding by id or by seat, removes the credential, and frees the seat to connect again", () => {
|
||||
seat("a@lab");
|
||||
const first = connect("a@lab");
|
||||
const ended = store.disconnect("a@lab", "moving to another account");
|
||||
expect(ended).toMatchObject({ id: first.id, state: "disconnected", stateReason: "moving to another account" });
|
||||
expect(existsSync(networkCredentialFile(home, first.id))).toBe(false);
|
||||
expect(() => readNetworkCredential(home, first.id)).toThrow(expect.objectContaining({ code: "credential_missing" }));
|
||||
// A second disconnect is the same answer, not an error.
|
||||
expect(store.disconnect(first.id).state).toBe("disconnected");
|
||||
const again = connect("a@lab");
|
||||
expect(again.id).not.toBe(first.id);
|
||||
expect(store.active().map((b) => b.id)).toEqual([again.id]);
|
||||
// Two binds in the same millisecond list in id order, so compare by id.
|
||||
expect(Object.fromEntries(store.list().map((b) => [b.id, b.state]))).toEqual({ [first.id]: "disconnected", [again.id]: "active" });
|
||||
});
|
||||
|
||||
it("a revoked binding is ended the same way, and contact and errors are recorded on the active one", () => {
|
||||
seat("a@lab");
|
||||
const binding = connect("a@lab");
|
||||
store.recordError(binding.id, "hub answered 503");
|
||||
expect(store.get(binding.id)!.lastError).toBe("hub answered 503");
|
||||
// Hub text can carry terminal escapes; status prints it, so control characters are replaced.
|
||||
store.recordError(binding.id, "hub said \u001b[31mred\u0007");
|
||||
expect(store.get(binding.id)!.lastError).toBe("hub said [31mred ");
|
||||
store.recordContact(binding.id, new Date("2026-11-11T18:30:00.000Z"));
|
||||
expect(store.get(binding.id)).toMatchObject({ lastContactAt: "2026-11-11T18:30:00.000Z", lastError: null });
|
||||
expect(store.markRevoked(binding.id, "the hub refused the credential")).toMatchObject({ state: "revoked", stateReason: "the hub refused the credential" });
|
||||
expect(existsSync(networkCredentialFile(home, binding.id))).toBe(false);
|
||||
});
|
||||
|
||||
it("refuses a credential file that others can read", async () => {
|
||||
seat("a@lab");
|
||||
const binding = connect("a@lab");
|
||||
const { chmodSync } = await import("node:fs");
|
||||
chmodSync(networkCredentialFile(home, binding.id), 0o644);
|
||||
expect(() => readNetworkCredential(home, binding.id)).toThrow(NetworkBindingError);
|
||||
expect(readFileSync(networkCredentialFile(home, binding.id), "utf8")).toContain("tok-123");
|
||||
});
|
||||
|
||||
it("a refused bind removes only what it created: a file already at the path stays, and a failed insert takes its credential with it", async () => {
|
||||
const { mkdirSync, writeFileSync } = await import("node:fs");
|
||||
seat("a@lab");
|
||||
const fixed = "netb-00000000000000000001";
|
||||
const pinned = new NetworkBindingStore(db, home, () => new Date(), () => fixed);
|
||||
const path = networkCredentialFile(home, fixed);
|
||||
mkdirSync(join(path, ".."), { recursive: true, mode: 0o700 });
|
||||
writeFileSync(path, "someone else's file", { mode: 0o600 });
|
||||
expect(() => pinned.create({ session: "a@lab", hubUrl: "https://hub.example/mcp", hubSeat: "alex@team", token: "tok" })).toThrow(/EEXIST/);
|
||||
expect(readFileSync(path, "utf8")).toBe("someone else's file");
|
||||
expect(store.list()).toHaveLength(0);
|
||||
|
||||
db.exec("CREATE TRIGGER refuse_bind BEFORE INSERT ON network_bindings BEGIN SELECT RAISE(ABORT, 'refused for the test'); END;");
|
||||
const other = new NetworkBindingStore(db, home, () => new Date(), () => "netb-00000000000000000002");
|
||||
expect(() => other.create({ session: "a@lab", hubUrl: "https://hub.example/mcp", hubSeat: "alex@team", token: "tok" })).toThrow(/refused for the test/);
|
||||
expect(existsSync(networkCredentialFile(home, "netb-00000000000000000002"))).toBe(false);
|
||||
expect(store.list()).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("a disconnect whose credential can't be removed still ends the binding, says so, and the retry it names removes it", async () => {
|
||||
const { chmodSync } = await import("node:fs");
|
||||
seat("a@lab");
|
||||
const binding = connect("a@lab");
|
||||
const secrets = join(home, "network-secrets");
|
||||
chmodSync(secrets, 0o500);
|
||||
let message = "";
|
||||
try {
|
||||
try { store.disconnect("a@lab"); } catch (error) {
|
||||
expect(error).toMatchObject({ code: "credential_cleanup_failed", status: 500 });
|
||||
message = (error as Error).message;
|
||||
}
|
||||
expect(store.get(binding.id)!.state).toBe("disconnected");
|
||||
expect(store.active()).toHaveLength(0);
|
||||
} finally {
|
||||
chmodSync(secrets, 0o700);
|
||||
}
|
||||
// The seat's name no longer finds an ended binding, so the message names the id to retry with.
|
||||
expect(() => store.disconnect("a@lab")).toThrow(expect.objectContaining({ code: "binding_not_found" }));
|
||||
const retry = /`rig network disconnect (\S+)`/.exec(message)?.[1];
|
||||
expect(retry).toBe(binding.id);
|
||||
expect(store.disconnect(retry!).state).toBe("disconnected");
|
||||
expect(existsSync(networkCredentialFile(home, binding.id))).toBe(false);
|
||||
});
|
||||
|
||||
it("a failed bind into a folder that was already there removes only its own credential file", async () => {
|
||||
const { mkdirSync, readFileSync: read, writeFileSync } = await import("node:fs");
|
||||
seat("a@lab");
|
||||
const id = "netb-00000000000000000004";
|
||||
const path = networkCredentialFile(home, id);
|
||||
mkdirSync(join(path, ".."), { recursive: true, mode: 0o700 });
|
||||
writeFileSync(join(path, "..", "other.txt"), "keep me");
|
||||
db.exec("CREATE TRIGGER refuse_bind BEFORE INSERT ON network_bindings BEGIN SELECT RAISE(ABORT, 'refused for the test'); END;");
|
||||
const pinned = new NetworkBindingStore(db, home, () => new Date(), () => id);
|
||||
expect(() => pinned.create({ session: "a@lab", hubUrl: "https://hub.example/mcp", hubSeat: "alex@team", token: "tok" })).toThrow(/refused for the test/);
|
||||
expect(existsSync(path)).toBe(false);
|
||||
expect(read(join(path, "..", "other.txt"), "utf8")).toBe("keep me");
|
||||
});
|
||||
|
||||
it("the credential sweep removes folders whose binding isn't active and leaves the active one and other names", async () => {
|
||||
const { mkdirSync, writeFileSync } = await import("node:fs");
|
||||
seat("a@lab"); seat("b@lab");
|
||||
const kept = connect("a@lab");
|
||||
const ended = connect("b@lab", { hubSeat: "b@team" });
|
||||
store.disconnect(ended.id);
|
||||
// An ended binding whose removal failed, a folder a stop left before its row, and a name that isn't a binding.
|
||||
const leftovers = [ended.id, "netb-00000000000000000009"];
|
||||
for (const id of leftovers) {
|
||||
mkdirSync(join(networkCredentialFile(home, id), ".."), { recursive: true, mode: 0o700 });
|
||||
writeFileSync(networkCredentialFile(home, id), "{}", { mode: 0o600 });
|
||||
}
|
||||
mkdirSync(join(home, "network-secrets", "notes"));
|
||||
expect(store.sweepCredentials()).toEqual({ removed: [...leftovers].sort(), failed: [] });
|
||||
for (const id of leftovers) expect(existsSync(join(networkCredentialFile(home, id), ".."))).toBe(false);
|
||||
expect(readNetworkCredential(home, kept.id)).toBe("tok-123");
|
||||
expect(existsSync(join(home, "network-secrets", "notes"))).toBe(true);
|
||||
});
|
||||
|
||||
it("a corrupt credential file is refused by its path alone, without quoting what it holds", async () => {
|
||||
const { writeFileSync } = await import("node:fs");
|
||||
seat("a@lab");
|
||||
const binding = connect("a@lab");
|
||||
writeFileSync(networkCredentialFile(home, binding.id), "tok-SECRET-not-json", { mode: 0o600 });
|
||||
let message = "";
|
||||
try { readNetworkCredential(home, binding.id); } catch (error) {
|
||||
expect(error).toMatchObject({ code: "credential_invalid" });
|
||||
message = (error as Error).message;
|
||||
}
|
||||
expect(message).toContain(networkCredentialFile(home, binding.id));
|
||||
expect(message).not.toContain("SECRET");
|
||||
});
|
||||
|
||||
describe("routes", () => {
|
||||
const app = (bearerToken: string | null = null) => {
|
||||
const router = new Hono();
|
||||
router.use("*", async (c, next) => { c.set("rigRepo" as never, rigs as never); await next(); });
|
||||
router.route("/api/network", networkRoutes({ bearerToken, home }));
|
||||
return router;
|
||||
};
|
||||
const post = (a: Hono, path: string, body: unknown, auth?: string) => a.request(path, {
|
||||
method: "POST", headers: { "Content-Type": "application/json", ...(auth ? { Authorization: `Bearer ${auth}` } : {}) }, body: JSON.stringify(body),
|
||||
});
|
||||
|
||||
it("connects, lists without the token, refuses with the store's code, and disconnects", async () => {
|
||||
seat("a@lab");
|
||||
const a = app();
|
||||
const created = await post(a, "/api/network/bindings", { session: "a@lab", hubUrl: "https://hub.example/mcp", hubSeat: "alex@team", token: "tok-9" });
|
||||
expect(created.status).toBe(201);
|
||||
const binding = await created.json() as { id: string };
|
||||
const listed = await (await a.request("/api/network/bindings")).text();
|
||||
expect(listed).toContain(binding.id);
|
||||
expect(listed).not.toContain("tok-9");
|
||||
const dup = await post(a, "/api/network/bindings", { session: "a@lab", hubUrl: "https://hub.example/mcp", hubSeat: "x@team", token: "tok-9" });
|
||||
expect(dup.status).toBe(409);
|
||||
expect(await dup.json()).toMatchObject({ error: "seat_already_connected" });
|
||||
const ended = await post(a, `/api/network/bindings/${binding.id}/disconnect`, { reason: "done" });
|
||||
expect(ended.status).toBe(200);
|
||||
expect(await ended.json()).toMatchObject({ state: "disconnected", stateReason: "done" });
|
||||
});
|
||||
|
||||
it("with an operator bearer configured, writes need it and reads don't", async () => {
|
||||
seat("a@lab");
|
||||
const a = app("op-secret");
|
||||
const body = { session: "a@lab", hubUrl: "https://hub.example/mcp", hubSeat: "alex@team", token: "tok-9" };
|
||||
expect((await post(a, "/api/network/bindings", body)).status).toBe(401);
|
||||
expect((await a.request("/api/network/bindings")).status).toBe(200);
|
||||
const created = await post(a, "/api/network/bindings", body, "op-secret");
|
||||
expect(created.status).toBe(201);
|
||||
const { id } = await created.json() as { id: string };
|
||||
expect((await post(a, `/api/network/bindings/${id}/disconnect`, {})).status).toBe(401);
|
||||
expect((await post(a, `/api/network/bindings/${id}/disconnect`, {}, "op-secret")).status).toBe(200);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,566 @@
|
||||
// The outbound connector against a hub that behaves like an external grant's queue: it lists the participant's own items,
|
||||
// a claim moves an item out of pending, and a close stores the resolution. Each test counts the local queue items that
|
||||
// carry a hub item: a lost answer, a restart, a rebinding or a revoke must never make a second one. No agent consumes
|
||||
// these items here, so the cells prove one local item per hub item, not one agent run.
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import Database from "better-sqlite3";
|
||||
import { existsSync, mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { ALL_MIGRATIONS } from "../src/db/all-migrations.js";
|
||||
import { migrate } from "../src/db/migrate.js";
|
||||
import { RigRepository } from "../src/domain/rig-repository.js";
|
||||
import { EventBus } from "../src/domain/event-bus.js";
|
||||
import { QueueRepository, QueueRepositoryError } from "../src/domain/queue-repository.js";
|
||||
import { NetworkBindingStore, networkCredentialFile } from "../src/domain/network/network-bindings.js";
|
||||
import { Hono } from "hono";
|
||||
import { connectMcpHub, HubError, type HubConnect, type HubItem, type HubSession } from "../src/domain/network/hub-client.js";
|
||||
import { networkRoutes } from "../src/routes/network.js";
|
||||
import { NetworkConnectorService, networkLocalItemId, runNetworkBindingPass, type NetworkConnectorDeps } from "../src/domain/network/network-connector.js";
|
||||
|
||||
const PARTICIPANT = "alex@team";
|
||||
|
||||
class FakeHub {
|
||||
readonly items = new Map<string, HubItem & { notes: string[] }>();
|
||||
readonly tokens: string[] = [];
|
||||
revoked = false;
|
||||
down = false;
|
||||
/** The next claim or close of these ids takes effect, then its answer is lost. */
|
||||
readonly loseClaimAnswer = new Set<string>();
|
||||
readonly loseCloseAnswer = new Set<string>();
|
||||
/** The next note on these ids fails; later ones land. */
|
||||
readonly failNote = new Set<string>();
|
||||
/** Shows of these ids wait until the pass closes its connection. */
|
||||
readonly stallShow = new Set<string>();
|
||||
/** Runs while a show is waiting on the hub, before it answers. */
|
||||
duringShow: ((id: string) => void) | null = null;
|
||||
/** Someone else closes these items just as ours arrives: the hub keeps theirs and answers ours as done. */
|
||||
readonly raceClose = new Map<string, Partial<HubItem>>();
|
||||
closes = 0;
|
||||
|
||||
add(qitemId: string, body = "Summarize the release notes", destination = PARTICIPANT): void {
|
||||
this.items.set(qitemId, { qitemId, sourceSession: "assistant@team", destinationSession: destination, state: "pending", body, summary: body, resolution: null, closureReason: null, notes: [] });
|
||||
}
|
||||
|
||||
readonly connect: HubConnect = async (_url, token, signal) => {
|
||||
this.tokens.push(token);
|
||||
if (this.revoked) throw new HubError("revoked", "The hub refused this participant's credential.");
|
||||
if (this.down) throw new HubError("unavailable", "The hub couldn't be reached: connect ECONNREFUSED");
|
||||
const get = (id: string) => {
|
||||
const item = this.items.get(id);
|
||||
if (!item) throw new HubError("refused", `qitem_not_found ${id}`);
|
||||
return item;
|
||||
};
|
||||
const session: HubSession = {
|
||||
listOwned: async (state, limit) => [...this.items.values()].filter((i) => i.destinationSession === PARTICIPANT && i.state === state).slice(0, limit)
|
||||
.map(({ notes: _notes, ...item }) => ({ ...item, body: null })),
|
||||
claim: async (id) => {
|
||||
const item = get(id);
|
||||
if (item.state !== "pending") throw new HubError("refused", `qitem ${id} is ${item.state}`);
|
||||
item.state = "in-progress";
|
||||
if (this.loseClaimAnswer.delete(id)) throw new HubError("unavailable", "The hub couldn't be reached: socket hang up");
|
||||
},
|
||||
show: async (id) => {
|
||||
this.duringShow?.(id);
|
||||
if (this.stallShow.has(id)) {
|
||||
await new Promise((_resolve, reject) => signal?.addEventListener("abort", () => reject(new HubError("unavailable", "The hub couldn't be reached: closed")), { once: true }));
|
||||
}
|
||||
const { notes: _notes, ...item } = get(id);
|
||||
return { ...item };
|
||||
},
|
||||
note: async (id, note) => {
|
||||
if (this.failNote.delete(id)) throw new HubError("unavailable", "The hub couldn't be reached: socket hang up");
|
||||
get(id).notes.push(note);
|
||||
},
|
||||
close: async (id, input) => {
|
||||
this.closes++;
|
||||
const item = get(id);
|
||||
if (["done", "canceled", "denied", "failed"].includes(item.state)) throw new HubError("refused", `qitem ${id} is already closed`);
|
||||
const theirs = this.raceClose.get(id);
|
||||
if (theirs) { Object.assign(item, theirs); return; }
|
||||
Object.assign(item, { state: input.state, closureReason: input.closureReason, resolution: input.resolution });
|
||||
if (this.loseCloseAnswer.delete(id)) throw new HubError("unavailable", "The hub couldn't be reached: socket hang up");
|
||||
},
|
||||
end: async () => {},
|
||||
};
|
||||
return session;
|
||||
};
|
||||
}
|
||||
|
||||
describe("network connector", () => {
|
||||
let db: Database.Database;
|
||||
let home: string;
|
||||
let repo: QueueRepository;
|
||||
let bindings: NetworkBindingStore;
|
||||
let hub: FakeHub;
|
||||
let deps: NetworkConnectorDeps;
|
||||
beforeEach(() => {
|
||||
db = new Database(":memory:");
|
||||
migrate(db, ALL_MIGRATIONS);
|
||||
const rigs = new RigRepository(db);
|
||||
const node = rigs.addNode(rigs.createRig("lab").id, "impl", { runtime: "claude-code" });
|
||||
db.prepare("INSERT INTO sessions (id, node_id, session_name, status) VALUES ('s1', ?, 'dev-impl@lab', 'running')").run(node.id);
|
||||
home = mkdtempSync(join(tmpdir(), "openrig-connector-"));
|
||||
repo = new QueueRepository(db, new EventBus(db), { validateRig: () => true, loadHumanRegistry: () => ({ ok: true, entities: [] }) });
|
||||
bindings = new NetworkBindingStore(db, home);
|
||||
hub = new FakeHub();
|
||||
deps = { db, queueRepo: repo, bindings, home, connect: hub.connect };
|
||||
});
|
||||
afterEach(() => { vi.restoreAllMocks(); db.close(); rmSync(home, { recursive: true, force: true }); });
|
||||
|
||||
const bind = () => bindings.create({ session: "dev-impl@lab", hubUrl: "https://hub.example/mcp", hubSeat: PARTICIPANT, token: "tok-1" });
|
||||
const pass = (bindingId: string) => runNetworkBindingPass(deps, bindings.get(bindingId)!);
|
||||
/** The local items carrying this hub item. */
|
||||
const runs = (hubId: string) => repo.list({ limit: 500 }).filter((i) => (i.tags ?? []).includes(`network-hub-item:${hubId}`));
|
||||
|
||||
it("takes a hub item into the bound seat once, says where it came from first, then carries pickup and the result back in order", async () => {
|
||||
const b = bind();
|
||||
hub.add("hub-1");
|
||||
const first = await pass(b.id);
|
||||
expect(first).toMatchObject({ taken: 1 });
|
||||
expect(first.error).toBeUndefined();
|
||||
expect(hub.tokens).toEqual(["tok-1"]);
|
||||
const [local] = runs("hub-1");
|
||||
expect(local).toMatchObject({ qitemId: networkLocalItemId(b, "hub-1"), sourceSession: "daemon@kernel", destinationSession: "dev-impl@lab", state: "pending",
|
||||
summary: "From assistant@team via network hub: Summarize the release notes" });
|
||||
// Where it came from is read before the requester's own words.
|
||||
expect(local!.body!.split("\n")[0]).toBe("From assistant@team through the network hub https://hub.example/mcp, as alex@team (hub item hub-1).");
|
||||
expect(local!.body).toContain("Summarize the release notes");
|
||||
// The hub claim is custody, not pickup: the item says it's queued and not picked up.
|
||||
expect(hub.items.get("hub-1")).toMatchObject({ state: "in-progress" });
|
||||
expect(hub.items.get("hub-1")!.notes).toEqual([`Queued for the connected local seat as ${local!.qitemId}. Not picked up yet.`]);
|
||||
expect(bindings.get(b.id)!.lastContactAt).not.toBeNull();
|
||||
|
||||
await pass(b.id);
|
||||
expect(runs("hub-1")).toHaveLength(1);
|
||||
expect(hub.items.get("hub-1")!.notes).toHaveLength(1);
|
||||
|
||||
repo.claim({ qitemId: local!.qitemId, destinationSession: "dev-impl@lab" });
|
||||
expect(await pass(b.id)).toMatchObject({ pickupsSent: 1, resultsSent: 0 });
|
||||
expect(hub.items.get("hub-1")!.notes.at(-1)).toMatch(/^Picked up by the local seat at /);
|
||||
|
||||
await repo.update({ qitemId: local!.qitemId, actorSession: "dev-impl@lab", state: "done", closureReason: "no-follow-on", resolution: "Three changes; none breaking." });
|
||||
expect(await pass(b.id)).toMatchObject({ resultsSent: 1 });
|
||||
expect(hub.items.get("hub-1")).toMatchObject({ state: "done", resolution: "Three changes; none breaking." });
|
||||
await pass(b.id);
|
||||
expect(hub.closes).toBe(1);
|
||||
expect(runs("hub-1")).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("a claim whose answer was lost: the next pass finds the item it holds and makes its one local item", async () => {
|
||||
const b = bind();
|
||||
hub.add("hub-2");
|
||||
hub.loseClaimAnswer.add("hub-2");
|
||||
expect(await pass(b.id)).toMatchObject({ taken: 0, error: "unavailable" });
|
||||
expect(runs("hub-2")).toHaveLength(0);
|
||||
expect(bindings.get(b.id)!.state).toBe("active");
|
||||
expect(await pass(b.id)).toMatchObject({ taken: 1 });
|
||||
expect(runs("hub-2")).toHaveLength(1);
|
||||
await pass(b.id);
|
||||
expect(runs("hub-2")).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("a restart between the local create and its receipt finds the same local item, not a second one", async () => {
|
||||
const b = bind();
|
||||
hub.add("hub-3");
|
||||
hub.items.get("hub-3")!.state = "in-progress"; // claimed before the restart
|
||||
await repo.create({ qitemId: networkLocalItemId(b, "hub-3"), sourceSession: "daemon@kernel", destinationSession: "dev-impl@lab", body: "already made",
|
||||
tags: [`network-binding:${b.id}`, "network-hub-item:hub-3"] });
|
||||
await pass(b.id);
|
||||
expect(runs("hub-3")).toHaveLength(1);
|
||||
expect(runs("hub-3")[0]!.body).toBe("already made");
|
||||
expect(db.prepare("SELECT count(*) AS n FROM network_receipts WHERE hub_qitem_id = 'hub-3'").get()).toEqual({ n: 1 });
|
||||
});
|
||||
|
||||
it("a close whose answer was lost is seen closed on the next pass and recorded as sent, without closing twice", async () => {
|
||||
const b = bind();
|
||||
hub.add("hub-4");
|
||||
await pass(b.id);
|
||||
const [local] = runs("hub-4");
|
||||
await repo.update({ qitemId: local!.qitemId, actorSession: "dev-impl@lab", state: "done", closureReason: "no-follow-on", resolution: "done it" });
|
||||
hub.loseCloseAnswer.add("hub-4");
|
||||
expect(await pass(b.id)).toMatchObject({ error: "unavailable" });
|
||||
expect(hub.items.get("hub-4")).toMatchObject({ state: "done", resolution: "done it" });
|
||||
const settled = await pass(b.id);
|
||||
expect(settled).toMatchObject({ resultsSent: 1 });
|
||||
expect(settled.error).toBeUndefined();
|
||||
expect(hub.items.get("hub-4")!.resolution).toBe("done it");
|
||||
expect(db.prepare("SELECT outcome, settled_at IS NOT NULL AS sent FROM network_receipts WHERE hub_qitem_id = 'hub-4'").get()).toEqual({ outcome: "done", sent: 1 });
|
||||
});
|
||||
|
||||
it("a revoked credential ends the binding: no new work comes in, and the work already taken stays", async () => {
|
||||
const b = bind();
|
||||
hub.add("hub-5");
|
||||
await pass(b.id);
|
||||
hub.revoked = true;
|
||||
expect(await pass(b.id)).toMatchObject({ error: "revoked" });
|
||||
expect(bindings.get(b.id)).toMatchObject({ state: "revoked" });
|
||||
expect(existsSync(networkCredentialFile(home, b.id))).toBe(false);
|
||||
hub.revoked = false;
|
||||
hub.add("hub-6");
|
||||
const service = new NetworkConnectorService(deps, { intervalMs: 1000 });
|
||||
await service.round();
|
||||
expect(runs("hub-6")).toHaveLength(0);
|
||||
expect(runs("hub-5")).toHaveLength(1);
|
||||
expect(hub.items.get("hub-6")!.state).toBe("pending");
|
||||
});
|
||||
|
||||
it("an unreachable hub is recorded, the binding stays active, and its next pass waits out a growing backoff", async () => {
|
||||
const b = bind();
|
||||
hub.down = true;
|
||||
let now = new Date("2026-11-11T19:00:00.000Z");
|
||||
const service = new NetworkConnectorService({ ...deps, now: () => now }, { intervalMs: 30_000, random: () => 0.5 });
|
||||
await service.round();
|
||||
expect(bindings.get(b.id)).toMatchObject({ state: "active", lastError: "The hub couldn't be reached: connect ECONNREFUSED" });
|
||||
expect(service.snapshot().bindings[0]).toMatchObject({ id: b.id, backoffMs: 60_000, lastPass: expect.objectContaining({ error: "unavailable" }) });
|
||||
const attempts = hub.tokens.length;
|
||||
now = new Date(now.getTime() + 30_000);
|
||||
await service.round();
|
||||
expect(hub.tokens.length).toBe(attempts);
|
||||
now = new Date(now.getTime() + 30_001);
|
||||
await service.round();
|
||||
expect(hub.tokens.length).toBe(attempts + 1);
|
||||
expect(service.snapshot().bindings[0]!.backoffMs).toBe(120_000);
|
||||
hub.down = false;
|
||||
now = new Date(now.getTime() + 120_001);
|
||||
await service.round();
|
||||
expect(service.snapshot().bindings[0]!.backoffMs).toBe(30_000);
|
||||
expect(bindings.get(b.id)!.lastError).toBeNull();
|
||||
});
|
||||
|
||||
it("a local item whose record is gone reads unknown on the hub and is never run again", async () => {
|
||||
const b = bind();
|
||||
hub.add("hub-7");
|
||||
await pass(b.id);
|
||||
const [local] = runs("hub-7");
|
||||
db.prepare("DELETE FROM queue_items WHERE qitem_id = ?").run(local!.qitemId);
|
||||
expect(await pass(b.id)).toMatchObject({ unknown: 1 });
|
||||
expect(hub.items.get("hub-7")!.notes.at(-1)).toBe(`This item's local record (${local!.qitemId}) is gone, so whether it ran is unknown. It won't be run again.`);
|
||||
expect(hub.items.get("hub-7")!.state).toBe("in-progress");
|
||||
await pass(b.id);
|
||||
expect(runs("hub-7")).toHaveLength(0);
|
||||
expect(db.prepare("SELECT outcome FROM network_receipts WHERE hub_qitem_id = 'hub-7'").get()).toEqual({ outcome: "unknown" });
|
||||
});
|
||||
|
||||
it("a seat with no session leaves its hub items pending and says why; once it has one, work goes to its current address", async () => {
|
||||
const b = bind();
|
||||
hub.add("hub-8");
|
||||
db.prepare("DELETE FROM sessions WHERE node_id = ?").run(b.nodeId);
|
||||
expect(await pass(b.id)).toMatchObject({ taken: 0, error: "no_session" });
|
||||
expect(hub.items.get("hub-8")).toMatchObject({ state: "pending", notes: [] });
|
||||
expect(runs("hub-8")).toHaveLength(0);
|
||||
expect(bindings.get(b.id)).toMatchObject({ state: "active", lastError: "dev-impl@lab has no session here, so its hub items stay pending until it has one." });
|
||||
// The seat comes back under another address; the binding follows its node.
|
||||
db.prepare("INSERT INTO sessions (id, node_id, session_name, status) VALUES ('s2', ?, 'impl-v2@lab', 'running')").run(b.nodeId);
|
||||
const taken = await pass(b.id);
|
||||
expect(taken).toMatchObject({ taken: 1 });
|
||||
expect(taken.error).toBeUndefined();
|
||||
expect(runs("hub-8")).toEqual([expect.objectContaining({ destinationSession: "impl-v2@lab" })]);
|
||||
expect(bindings.get(b.id)!.lastError).toBeNull();
|
||||
});
|
||||
|
||||
it("a revoke whose credential can't be removed still ends the binding, records why, and the next start sweeps it", async () => {
|
||||
const { chmodSync } = await import("node:fs");
|
||||
const b = bind();
|
||||
hub.revoked = true;
|
||||
const secrets = join(home, "network-secrets");
|
||||
chmodSync(secrets, 0o500);
|
||||
try {
|
||||
expect(await pass(b.id)).toMatchObject({ error: "revoked" });
|
||||
expect(bindings.get(b.id)!.state).toBe("revoked");
|
||||
expect(bindings.get(b.id)!.lastError).toContain(`Run \`rig network disconnect ${b.id}\` again to retry.`);
|
||||
expect(existsSync(networkCredentialFile(home, b.id))).toBe(true);
|
||||
} finally {
|
||||
chmodSync(secrets, 0o700);
|
||||
}
|
||||
const lines: string[] = [];
|
||||
const service = new NetworkConnectorService({ ...deps, log: (line) => lines.push(line) }, { intervalMs: 1000 });
|
||||
service.start();
|
||||
await service.stop();
|
||||
expect(existsSync(networkCredentialFile(home, b.id))).toBe(false);
|
||||
expect(lines).toEqual([`network connector: removed the credentials of ended bindings ${b.id}`]);
|
||||
});
|
||||
|
||||
it("a plain-http binding with its recorded choice is used pass after pass, and one without a choice sends no token", async () => {
|
||||
const b = bindings.create({ session: "dev-impl@lab", hubUrl: "http://192.0.2.7:7433/mcp", hubSeat: PARTICIPANT, token: "tok-1", allowHttp: true });
|
||||
hub.add("hub-9");
|
||||
expect((await pass(b.id)).error).toBeUndefined();
|
||||
expect((await pass(b.id)).error).toBeUndefined();
|
||||
expect(hub.tokens).toEqual(["tok-1", "tok-1"]);
|
||||
expect(runs("hub-9")).toHaveLength(1);
|
||||
|
||||
db.prepare("UPDATE network_bindings SET allow_http = 0 WHERE id = ?").run(b.id);
|
||||
expect(await pass(b.id)).toMatchObject({ error: "http_not_allowed" });
|
||||
expect(hub.tokens).toHaveLength(2);
|
||||
expect(bindings.get(b.id)!.lastError).toContain("--allow-http");
|
||||
});
|
||||
|
||||
it("a hub that redirects to another origin gets no token there, and the error names where it pointed", async () => {
|
||||
const { createServer } = await import("node:http");
|
||||
const listen = (handler: Parameters<typeof createServer>[1]) => new Promise<{ url: string; close: () => Promise<void> }>((resolve) => {
|
||||
const server = createServer(handler).listen(0, "127.0.0.1", () => {
|
||||
const port = (server.address() as { port: number }).port;
|
||||
resolve({ url: `http://127.0.0.1:${port}/mcp`, close: () => new Promise((done) => server.close(() => done())) });
|
||||
});
|
||||
});
|
||||
const elsewhere: Array<string | undefined> = [];
|
||||
const other = await listen((req, res) => { elsewhere.push(req.headers.authorization); res.writeHead(500).end(); });
|
||||
const asked: Array<string | undefined> = [];
|
||||
const hubServer = await listen((req, res) => { asked.push(req.headers.authorization); res.writeHead(307, { Location: other.url }).end(); });
|
||||
try {
|
||||
const b = bindings.create({ session: "dev-impl@lab", hubUrl: hubServer.url, hubSeat: PARTICIPANT, token: "tok-redirect", allowHttp: true });
|
||||
const result = await runNetworkBindingPass({ db, queueRepo: repo, bindings, home }, bindings.get(b.id)!);
|
||||
expect(result.error).toBe("unavailable");
|
||||
expect(asked[0]).toBe("Bearer tok-redirect");
|
||||
expect(elsewhere).toEqual([]);
|
||||
expect(bindings.get(b.id)!.lastError).toContain("redirected the connection somewhere else");
|
||||
expect(bindings.get(b.id)!.lastError).toContain(other.url);
|
||||
expect(bindings.get(b.id)!.state).toBe("active");
|
||||
} finally {
|
||||
await hubServer.close();
|
||||
await other.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("a local handoff's result comes back from its successor; failed and denied closes come back as themselves; no successor reads unknown", async () => {
|
||||
const b = bind();
|
||||
for (const id of ["hub-h1", "hub-h2", "hub-h3", "hub-h4"]) hub.add(id);
|
||||
await pass(b.id);
|
||||
const local = (id: string) => runs(id)[0]!;
|
||||
repo.claim({ qitemId: local("hub-h1").qitemId, destinationSession: "dev-impl@lab" });
|
||||
const { created } = await repo.handoff({ qitemId: local("hub-h1").qitemId, fromSession: "dev-impl@lab", toSession: "review@lab", nudge: false });
|
||||
repo.claim({ qitemId: created.qitemId, destinationSession: "review@lab" });
|
||||
await repo.update({ qitemId: created.qitemId, actorSession: "review@lab", state: "done", closureReason: "no-follow-on", resolution: "Reviewed: fine." });
|
||||
await repo.update({ qitemId: local("hub-h2").qitemId, actorSession: "dev-impl@lab", state: "failed", transitionNote: "The build broke." });
|
||||
// The queue's own refusal form: done with closure reason denied.
|
||||
await repo.update({ qitemId: local("hub-h3").qitemId, actorSession: "dev-impl@lab", state: "done", closureReason: "denied", resolution: "Not mine to do." });
|
||||
// Handed on with no successor here, as a handoff to another host leaves it.
|
||||
db.prepare("UPDATE queue_items SET state = 'handed-off' WHERE qitem_id = ?").run(local("hub-h4").qitemId);
|
||||
|
||||
expect(await pass(b.id)).toMatchObject({ resultsSent: 3, unknown: 1 });
|
||||
expect(hub.items.get("hub-h1")).toMatchObject({ state: "done", closureReason: "no-follow-on", resolution: "Reviewed: fine." });
|
||||
expect(hub.items.get("hub-h2")).toMatchObject({ state: "failed", closureReason: null, resolution: "The build broke." });
|
||||
expect(hub.items.get("hub-h3")).toMatchObject({ state: "done", closureReason: "denied", resolution: "Not mine to do." });
|
||||
expect(hub.items.get("hub-h4")).toMatchObject({ state: "in-progress" });
|
||||
expect(hub.items.get("hub-h4")!.notes.at(-1)).toBe(`This item was handed on from ${local("hub-h4").qitemId} without one successor here, so its outcome is unknown. It won't be run again.`);
|
||||
});
|
||||
|
||||
it("a disconnect while a hub answer is pending starts no local work, and work already queued stays", async () => {
|
||||
const b = bind();
|
||||
hub.add("hub-d1");
|
||||
await pass(b.id);
|
||||
hub.add("hub-d2");
|
||||
hub.duringShow = (id) => { if (id === "hub-d2") bindings.disconnect(b.id); };
|
||||
await pass(b.id);
|
||||
expect(runs("hub-d2")).toHaveLength(0);
|
||||
expect(runs("hub-d1")).toHaveLength(1);
|
||||
expect(hub.items.get("hub-d2")).toMatchObject({ state: "in-progress", notes: [] });
|
||||
});
|
||||
|
||||
it("disconnecting and connecting the same seat, hub and participant again keeps one local item per hub item, unknown included", async () => {
|
||||
const first = bind();
|
||||
for (const id of ["r-queued", "r-picked", "r-done", "r-gone"]) hub.add(id);
|
||||
await pass(first.id);
|
||||
const local = (id: string) => runs(id)[0]!;
|
||||
repo.claim({ qitemId: local("r-picked").qitemId, destinationSession: "dev-impl@lab" });
|
||||
repo.claim({ qitemId: local("r-done").qitemId, destinationSession: "dev-impl@lab" });
|
||||
await repo.update({ qitemId: local("r-done").qitemId, actorSession: "dev-impl@lab", state: "done", closureReason: "no-follow-on", resolution: "Finished before the reconnect." });
|
||||
const gone = local("r-gone").qitemId;
|
||||
db.prepare("DELETE FROM queue_items WHERE qitem_id = ?").run(gone);
|
||||
bindings.disconnect(first.id);
|
||||
|
||||
const second = bind();
|
||||
expect(second.id).not.toBe(first.id);
|
||||
expect(await pass(second.id)).toMatchObject({ taken: 0, pickupsSent: 2, resultsSent: 1, unknown: 1 });
|
||||
for (const id of ["r-queued", "r-picked", "r-done"]) expect(runs(id)).toHaveLength(1);
|
||||
expect(runs("r-gone")).toHaveLength(0);
|
||||
expect(hub.items.get("r-picked")!.notes.at(-1)).toMatch(/^Picked up by the local seat at /);
|
||||
expect(hub.items.get("r-done")).toMatchObject({ state: "done", resolution: "Finished before the reconnect." });
|
||||
expect(hub.items.get("r-gone")!.notes.at(-1)).toBe(`This item's local record (${gone}) is gone, so whether it ran is unknown. It won't be run again.`);
|
||||
expect(db.prepare("SELECT DISTINCT binding_id FROM network_receipts").all()).toEqual([{ binding_id: second.id }]);
|
||||
await pass(second.id);
|
||||
expect(runs("r-gone")).toHaveLength(0);
|
||||
expect(hub.items.get("r-gone")!.notes.filter((n) => n.includes("is unknown"))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("a stage note that fails is sent on the next pass, and an unknown outcome is marked sent only once its note lands", async () => {
|
||||
const b = bind();
|
||||
hub.add("hub-n1");
|
||||
hub.failNote.add("hub-n1");
|
||||
expect(await pass(b.id)).toMatchObject({ taken: 1, error: "unavailable" });
|
||||
expect(hub.items.get("hub-n1")!.notes).toEqual([]);
|
||||
await pass(b.id);
|
||||
expect(hub.items.get("hub-n1")!.notes).toEqual([`Queued for the connected local seat as ${runs("hub-n1")[0]!.qitemId}. Not picked up yet.`]);
|
||||
|
||||
// An id used before and erased: the queue refuses it, so whether it ran is unknown.
|
||||
hub.add("hub-n2");
|
||||
vi.spyOn(repo, "create").mockRejectedValueOnce(new QueueRepositoryError("qitem_erased", "erased"));
|
||||
hub.failNote.add("hub-n2");
|
||||
expect(await pass(b.id)).toMatchObject({ error: "unavailable" });
|
||||
const receipt = () => db.prepare("SELECT outcome, settled_at FROM network_receipts WHERE hub_qitem_id = 'hub-n2'").get() as { outcome: string; settled_at: string | null };
|
||||
expect(receipt()).toMatchObject({ outcome: "unknown", settled_at: null });
|
||||
await pass(b.id);
|
||||
expect(hub.items.get("hub-n2")!.notes).toEqual([expect.stringMatching(/is gone, so whether it ran is unknown\. It won't be run again\.$/)]);
|
||||
expect(receipt().settled_at).not.toBeNull();
|
||||
expect(runs("hub-n2")).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("a hub item someone else closed first isn't taken as our result: the receipt says so and the local result stays", async () => {
|
||||
const b = bind();
|
||||
hub.add("hub-c1");
|
||||
await pass(b.id);
|
||||
const [local] = runs("hub-c1");
|
||||
Object.assign(hub.items.get("hub-c1")!, { state: "done", closureReason: "canceled", resolution: "Not needed any more." });
|
||||
await repo.update({ qitemId: local!.qitemId, actorSession: "dev-impl@lab", state: "done", closureReason: "no-follow-on", resolution: "Here it is." });
|
||||
const closes = hub.closes;
|
||||
expect(await pass(b.id)).toMatchObject({ resultsSent: 0 });
|
||||
// Checked before the close, so nothing is written over theirs.
|
||||
expect(hub.closes).toBe(closes);
|
||||
expect(hub.items.get("hub-c1")).toMatchObject({ state: "done", closureReason: "canceled", resolution: "Not needed any more." });
|
||||
expect(db.prepare("SELECT outcome, settled_at IS NOT NULL AS settled, last_error FROM network_receipts WHERE hub_qitem_id = 'hub-c1'").get()).toEqual({
|
||||
outcome: "closed_on_hub", settled: 1,
|
||||
last_error: `The hub item was closed elsewhere (done, canceled) before this result reached it. The result stays on ${local!.qitemId}.`,
|
||||
});
|
||||
expect(repo.getById(local!.qitemId)!.resolution).toBe("Here it is.");
|
||||
});
|
||||
|
||||
it("a hub item already failed elsewhere gets no write and isn't our result, and a close that loses a race is read back as closed elsewhere", async () => {
|
||||
const b = bind();
|
||||
hub.add("hub-f1");
|
||||
hub.add("hub-f2");
|
||||
await pass(b.id);
|
||||
const [f1] = runs("hub-f1");
|
||||
const [f2] = runs("hub-f2");
|
||||
Object.assign(hub.items.get("hub-f1")!, { state: "failed", closureReason: null, resolution: "Their failure." });
|
||||
hub.raceClose.set("hub-f2", { state: "done", closureReason: "no-follow-on", resolution: "Someone else's answer." });
|
||||
for (const item of [f1!, f2!]) {
|
||||
await repo.update({ qitemId: item.qitemId, actorSession: "dev-impl@lab", state: "done", closureReason: "no-follow-on", resolution: "Ours." });
|
||||
}
|
||||
const closes = hub.closes;
|
||||
expect(await pass(b.id)).toMatchObject({ resultsSent: 0 });
|
||||
expect(hub.closes).toBe(closes + 1);
|
||||
expect(hub.items.get("hub-f1")).toMatchObject({ state: "failed", resolution: "Their failure." });
|
||||
expect(hub.items.get("hub-f2")).toMatchObject({ state: "done", resolution: "Someone else's answer." });
|
||||
const outcomes = db.prepare("SELECT hub_qitem_id, outcome FROM network_receipts WHERE hub_qitem_id IN ('hub-f1', 'hub-f2') ORDER BY hub_qitem_id").all();
|
||||
expect(outcomes).toEqual([{ hub_qitem_id: "hub-f1", outcome: "closed_on_hub" }, { hub_qitem_id: "hub-f2", outcome: "closed_on_hub" }]);
|
||||
|
||||
// The person sees both conflicts on the binding.
|
||||
const app = new Hono();
|
||||
app.use("*", async (c, next) => { c.set("rigRepo" as never, new RigRepository(db) as never); await next(); });
|
||||
app.route("/api/network", networkRoutes({ home }));
|
||||
// A later pass's ordinary contact doesn't clear it, and it's read from the receipts, so a restart keeps it.
|
||||
expect((await pass(b.id)).error).toBeUndefined();
|
||||
const listed = await (await app.request("/api/network/bindings")).json() as { bindings: Array<{ id: string; closedElsewhere: unknown[] }> };
|
||||
expect(listed.bindings.find((x) => x.id === b.id)!.closedElsewhere).toEqual([
|
||||
{ hubItem: "hub-f1", localItem: f1!.qitemId, note: `The hub item was closed elsewhere (failed) before this result reached it. The result stays on ${f1!.qitemId}.` },
|
||||
{ hubItem: "hub-f2", localItem: f2!.qitemId, note: `The hub item was closed elsewhere (done, no-follow-on) before this result reached it. The result stays on ${f2!.qitemId}.` },
|
||||
].sort((x, y) => x.localItem.localeCompare(y.localItem)));
|
||||
});
|
||||
|
||||
it("a close without a resolution sends only the closer's own closing note, never another note on the item", async () => {
|
||||
const b = bind();
|
||||
hub.add("hub-w1");
|
||||
hub.add("hub-w2");
|
||||
await pass(b.id);
|
||||
const [w1] = runs("hub-w1");
|
||||
const [w2] = runs("hub-w2");
|
||||
db.prepare("INSERT INTO queue_transitions (qitem_id, ts, state, transition_note, actor_session) VALUES (?, ?, 'pending', ?, 'daemon@kernel')")
|
||||
.run(w1!.qitemId, new Date().toISOString(), "Stuck sweep: dev-impl@lab idle 17 min, recovery qitem-recovery-x.");
|
||||
await repo.update({ qitemId: w1!.qitemId, actorSession: "dev-impl@lab", state: "done", closureReason: "no-follow-on" });
|
||||
await repo.update({ qitemId: w2!.qitemId, actorSession: "dev-impl@lab", state: "done", closureReason: "no-follow-on", transitionNote: "Done; the notes are in the item." });
|
||||
// Notes after a close repeat its state: the closer's own later note isn't the result, and a later OpenRig note
|
||||
// doesn't hide the closer's note on the close.
|
||||
await repo.update({ qitemId: w1!.qitemId, actorSession: "dev-impl@lab", transitionNote: "Also: I left a scratch file in /tmp." });
|
||||
db.prepare("INSERT INTO queue_transitions (qitem_id, ts, state, transition_note, actor_session) VALUES (?, ?, 'done', ?, 'daemon@kernel')")
|
||||
.run(w2!.qitemId, new Date().toISOString(), "Delivery receipt: wake confirmed for dev-impl@lab.");
|
||||
await pass(b.id);
|
||||
expect(hub.items.get("hub-w1")!.resolution).toBe("Closed without a stated result.");
|
||||
expect(hub.items.get("hub-w2")!.resolution).toBe("Done; the notes are in the item.");
|
||||
});
|
||||
|
||||
it("a failure here is recorded as local, not as the hub's", async () => {
|
||||
const b = bind();
|
||||
hub.add("hub-l1");
|
||||
vi.spyOn(repo, "create").mockRejectedValueOnce(new Error("disk I/O error"));
|
||||
expect(await pass(b.id)).toMatchObject({ error: "local" });
|
||||
expect(bindings.get(b.id)!.lastError).toBe("A local error stopped this pass: disk I/O error");
|
||||
});
|
||||
|
||||
it("reporting looks at a bounded batch per pass and rotates it, so a later item isn't hidden behind older open ones", async () => {
|
||||
const b = bind();
|
||||
for (const id of ["hub-q1", "hub-q2", "hub-q3"]) hub.add(id);
|
||||
let tick = Date.parse("2026-11-11T19:00:00.000Z");
|
||||
const limited = { ...deps, reportLimit: 1, now: () => new Date(tick += 1000) };
|
||||
await runNetworkBindingPass(limited, bindings.get(b.id)!);
|
||||
// q1 and q2 stay unclaimed and open; q3 is picked up.
|
||||
repo.claim({ qitemId: runs("hub-q3")[0]!.qitemId, destinationSession: "dev-impl@lab" });
|
||||
for (let i = 0; i < 3; i++) await runNetworkBindingPass(limited, bindings.get(b.id)!);
|
||||
expect(hub.items.get("hub-q3")!.notes.at(-1)).toMatch(/^Picked up by the local seat at /);
|
||||
});
|
||||
|
||||
it("a pass whose hub stalls is closed at its deadline before any late local work, and the next binding still gets its turn", async () => {
|
||||
const node = new RigRepository(db).addNode(new RigRepository(db).listRigs()[0]!.id, "review", { runtime: "claude-code" });
|
||||
db.prepare("INSERT INTO sessions (id, node_id, session_name, status) VALUES ('s9', ?, 'dev-review@lab', 'running')").run(node.id);
|
||||
const slow = bind();
|
||||
hub.add("hub-s1");
|
||||
hub.stallShow.add("hub-s1");
|
||||
const other = new FakeHub();
|
||||
other.add("hub-o1");
|
||||
const fast = bindings.create({ session: "dev-review@lab", hubUrl: "https://other-hub.example/mcp", hubSeat: PARTICIPANT, token: "tok-2" });
|
||||
const connect: HubConnect = (url, token, signal) => (url === "https://other-hub.example/mcp" ? other : hub).connect(url, token, signal);
|
||||
const service = new NetworkConnectorService({ ...deps, connect, passTimeoutMs: 200 }, { intervalMs: 1000 });
|
||||
await service.round();
|
||||
expect(bindings.get(slow.id)!.lastError).toBe("This pass took longer than 200 ms, so its connection to the hub was closed. The next pass tries again.");
|
||||
expect(runs("hub-s1")).toHaveLength(0);
|
||||
// The other binding's pass ran in the same round and took its item.
|
||||
expect(runs("hub-o1")).toEqual([expect.objectContaining({ destinationSession: "dev-review@lab" })]);
|
||||
expect(bindings.get(fast.id)!.lastError).toBeNull();
|
||||
});
|
||||
|
||||
it("an initialize the hub answers but a stalled initialized notification is closed at the deadline, and stop doesn't wait for a pass", async () => {
|
||||
const { createServer } = await import("node:http");
|
||||
const waiting: Array<() => void> = [];
|
||||
const held: Array<import("node:http").ServerResponse> = [];
|
||||
const server = createServer((req, res) => {
|
||||
let raw = "";
|
||||
req.on("data", (chunk) => { raw += chunk; });
|
||||
req.on("end", () => {
|
||||
const message = raw ? JSON.parse(raw) as { id?: number; method?: string; params?: { protocolVersion?: string } } : {};
|
||||
if (message.method === "initialize") {
|
||||
res.writeHead(200, { "Content-Type": "application/json" }).end(JSON.stringify({ jsonrpc: "2.0", id: message.id,
|
||||
result: { protocolVersion: message.params?.protocolVersion, capabilities: {}, serverInfo: { name: "stalling hub", version: "1" } } }));
|
||||
} else {
|
||||
held.push(res); // notifications/initialized: never answered
|
||||
waiting.shift()?.();
|
||||
}
|
||||
});
|
||||
});
|
||||
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", () => resolve()));
|
||||
const url = `http://127.0.0.1:${(server.address() as { port: number }).port}/mcp`;
|
||||
try {
|
||||
const b = bindings.create({ session: "dev-impl@lab", hubUrl: url, hubSeat: PARTICIPANT, token: "tok-stall", allowHttp: true });
|
||||
const real = { ...deps, connect: connectMcpHub };
|
||||
expect(await runNetworkBindingPass({ ...real, passTimeoutMs: 300 }, bindings.get(b.id)!)).toMatchObject({ error: "unavailable" });
|
||||
expect(bindings.get(b.id)!.lastError).toBe("This pass took longer than 300 ms, so its connection to the hub was closed. The next pass tries again.");
|
||||
|
||||
const service = new NetworkConnectorService({ ...real, passTimeoutMs: 60_000 }, { intervalMs: 1000 });
|
||||
const stalled = new Promise<void>((resolve) => waiting.push(resolve));
|
||||
service.start();
|
||||
await stalled;
|
||||
const asked = Date.now();
|
||||
await service.stop();
|
||||
expect(Date.now() - asked).toBeLessThan(5_000);
|
||||
expect(bindings.get(b.id)!.lastError).toBe("The connector stopped during this pass and closed its connection to the hub.");
|
||||
} finally {
|
||||
for (const res of held) res.destroy();
|
||||
server.closeAllConnections?.();
|
||||
await new Promise<void>((resolve) => server.close(() => resolve()));
|
||||
}
|
||||
});
|
||||
|
||||
it("with no active binding, a round reaches no hub", async () => {
|
||||
const service = new NetworkConnectorService(deps, { intervalMs: 1000 });
|
||||
await service.round();
|
||||
expect(hub.tokens).toEqual([]);
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user