openrig/docs/as-built/test-layers.md

kind, title, status, applies-when, siblings, last-verified-against-source, last-updated

kind	title	status	applies-when	siblings	last-verified-against-source	last-updated
as-built	Test layers — what to run before you push, and what each layer proves	active	You changed OpenRig and want to know which checks to run before opening a pull request, what each check actually proves, and what CI will run for you. Also read it before claiming a stub-agent scenario covers a behaviour.	arteries.md README.md codemap.md	e8f0ab340d	2026-10-08

Test layers: what to run before you push, and what each layer proves

This page is a map, not the territory. It was checked against source at e8f0ab34 and it will drift. It is also incomplete on purpose. Before you rely on a command or a claim here, read the script it names or run it. If a behaviour is missing from this page (or from arteries.md), that tells you nothing about whether a change to it is safe.

Short version

For every change, run what the pull-request template asks for:

npm run build
npm run lint      # typecheck only; there is no formatter or style linter
npm test          # = npm run test:repo && npm run test:workspaces

Then add the layers your change calls for, using the ladder below. CI also runs a network-denied macOS test pass and a containerised installed-package scenario on every PR to main. In the PR description, say which layers you ran and which you could not run.

The ladder

Layer	Command	What it proves	What it cannot prove	When to run it
Typecheck	npm run lint	Builds the daemon, then tsc --noEmit passes for daemon, ui, cli and tui	Any runtime behaviour. Formatting.	Every change
Build	npm run build	All four workspaces compile (the UI also runs vite build)	That the published package is complete (see Packaging)	Every change
Repo checks	npm run test:repo	Builds the daemon, then runs node --test --test-concurrency=1 scripts/*.test.mjs, scripts/check-first-run-references.mjs, scripts/check-docs-guard.mjs, mirror-skills --check (skill homes, generated copies, layout and digests) and generate-context-packs --check	Package behaviour	Every change. It matters most when you touch scripts/, docs/ or skills.
Package suites	npm run test:workspaces. One package: npm run test -w packages/cli. One file, from the repo root: npx vitest run packages/cli/test/<file>.test.ts	Vitest unit and integration tests for daemon, cli and tui. Some of them start real scenario daemons, tmux sessions and stub runners.	The installed package. Real Claude Code or Codex. CI's sandbox conditions (see CI).	Every change (npm test covers it)
Web UI suite	npm run test:ui	jsdom unit tests for packages/ui	Real browser rendering	When you touch packages/ui, or a daemon API the UI reads. The web UI is in maintenance mode, so this suite is advisory locally and is not part of npm test.
Generated files	npm run mirror-skills:check, npm run generate-context-packs:check (both part of npm test)	Shipped skill homes are publishable, generated copies match their homes, all edges match their recorded hashes, and every static context pack passes the daemon's own manifest parser and the leak scan	Whether the skill content is any good	Edit a shipped skill's one in-repo home, run npm run mirror-skills to regenerate its other copies and refresh digests, then run the checks and commit the result. The apply reads nothing outside the repository. See "A shipped skill" in ARCHITECTURE.md.
Local gate	npm run gate	Lint plus npm test on a clean tree, tied to HEAD, one run per machine at a time, with a JSON receipt	Anything that lint and npm test don't already prove	Optional. Useful when several agents share one machine, or when you want a receipt tied to a commit.
Packaging	npm run build:package, npm run test:tui-package, bash scripts/smoke-fresh-install.sh	The publishable @openrig/cli assembles, the packed TUI launches, and a fresh install starts a daemon	Behaviour beyond startup	When you touch dependencies, bin entries, build scripts, or files a package ships
Stub scenarios, host mode	node --import tsx packages/daemon/scripts/run-scenarios.mjs [scenario.yaml …]	A scenario's steps and assertions pass against a scratch daemon, real tmux and runtime: stub seats	The installed package. Seeded-regression pairing. Everything under What a stub can and cannot prove.	While you write or debug a scenario
Stub scenarios, installed package in a container	CI job installed-scenario. Before pushing: DOCKER_HOST=ssh://<your-host> bash scripts/run-pr-scenarios.sh --remote …	A clean Linux install starts. A claimed queue item survives a daemon restart. A planted fault is caught at the intended assertion. Two stub seats' scripted output reads back through rig transcript and rig capture (passing form only).	Native runtimes. Most command families (see Help wanted).	Changes that can affect daemon startup, packaging, queue durability or the scenario harness
Live-model evals	npm run eval -w packages/daemon -- --provider rig …	Whether a real Claude Code seat runs the expected rig context get <entry> for natural prompts	Determinism. Codex. Anything without a case.	Changes to shipped skills or context entries meant to steer agents. Uses provider credits.
Testbed runbooks	docker/testbed/runbooks/ (manual)	The testbed image itself works as a test environment	Product behaviour	Only when you change the testbed image or the container plumbing

Some package tests are gated. OPENRIG_E2E_REAL_CODEX=1 runs two daemon e2e tests against a real Codex (they also need tmux, codex and ~/.codex/auth.json, and they use provider credits). OPENRIG_REAL_CLAUDE_INTEGRATION=1 enables only a placeholder: it requires OPENRIG_PARENT_NATIVE_ID and checks its format, and no real-Claude fork test exists yet. Without tmux, some tmux tests skip and others (such as transcript-seat-env-native.test.ts) fail. The CLI's release-tag tests skip without local tags unless CI=true, so a green local run without tags has skipped them. CI installs tmux and fetches full history.

The TUI has its own vitest suite (npm run test -w packages/tui, which builds the TUI first) and runs in the tui leg of package-tests; npm run test:tui-package launches the packed TUI in demo mode. The tui_socket scenario surface exists, but the only scenarios that use it are not yet runnable checks.

Gotchas that cost people time

• Running inside a seat. The root vitest.config.ts makes a root-level file run use its package's config. The cli, daemon and TUI configs all run the shared root test/hermetic-env.setup.ts before any test file. It clears the inherited OPENRIG_* and RIGGED_* instance selectors (connection, database, workspace and topology roots, seat identity, bearer tokens), forces a fixture OPENRIG_HOME and fails if the home is not fixture-scoped. It also wraps fetch, so a real request to anything other than a loopback ephemeral-port fixture or a registered target fails. It points HOME and the XDG directories at a fixture user home and clears CODEX_HOME and CLAUDE_CONFIG_DIR. OPENRIG_E2E_REAL_CODEX=1 keeps native homes and binaries only for seat-handover-model-fidelity-e2e.test.ts and seat-lifecycle-set-model-resume-e2e.test.ts; all other files remain isolated. The scenario scaffold always uses fixture homes and runtime stubs. The two native files rely on Vitest's default per-file isolation; runs without it are unsupported. Scenario scaffolds scrub inherited provider roots but preserve those passed in an explicit baseEnv. A test can still set its own overrides after setup. The UI suite (jsdom) uses its own packages/ui/test/setup.ts.

• Stale vendored daemon. npm run build:package leaves an assembled copy of the daemon at packages/cli/daemon/ (gitignored). If you change daemon source afterwards, scripts/check-cli-daemon-freshness.test.mjs inside test:repo can fail on the stale copy. Re-run npm run build:package or delete packages/cli/daemon/. npm run gate deletes it for you.

• Local passes, CI fails. CI runs the package suites with a scratch HOME, no credentials, and outbound network blocked except to localhost (see below). A test that reads your real home directory, a running daemon, the network or a credential can pass on your machine and fail in CI.

• Mixed fixture clocks. compaction-command-submission.test.ts uses one clock for the enforcer, the transport's injected now and sleeps, and its Date.now idle deadline. Previously, real-time transport waits could exhaust the fixture's 20 ms budget during a scheduler pause even though the modeled seat was idle. Restore the Date.now spy after each test; keep real deadline behavior covered by the transport tests. This changes the fixture, not production timeouts.

• Docs location. check-docs-guard fails if a tracked file under docs/ sits outside docs/as-built/, docs/reference/ or docs/releases/. docs/DESIGN.md is the one allowed exception.

What CI runs on your pull request

.github/workflows/tests.yml runs on pull requests to main and on manual dispatch, all on Node 22. Each suite is its own job, so one failure doesn't hide the others. No provider credentials are supplied.

Job	Runner	What it does
build-and-package	ubuntu	npm ci and npm run build. Then npm pack --dry-run for cli, daemon and tui, with each file list uploaded as the pack-lists artifact. The job only records the lists; reviewers read them.
typecheck	ubuntu	npm run lint
repo-checks	ubuntu	npm run test:repo
package-tests	macOS, matrix daemon, cli, tui, ui (fail-fast off)	Installs tmux, runs npm ci and npm run build. Then runs npm run test -w packages/<pkg> under env -i with a scratch HOME/TMPDIR/TMUX_TMPDIR and a sandbox-exec profile that blocks outbound network except localhost and the job's own socket directory. scripts/ci-macos-process-probe.mjs runs first.
linux-seat-environment	ubuntu	Builds all packages, then runs seat-env-parent-session.integration.test.ts with real CLI, daemon, tmux and stub seats. A network namespace has loopback only; tests run as the unprivileged runner user under env -i with scratch HOME/TMPDIR/TMUX_TMPDIR. Runs on pull requests and manual dispatch, like package-tests.
installed-scenario	ubuntu	bash scripts/run-pr-scenarios.sh (fifteen container runs, described below). Uploads dist/pr-scenarios/ as installed-scenario-evidence, including on failure.

The Linux seat-environment job covers a gap in the macOS matrix: the integration test reads /proc/<pid>/environ, so macOS skips it. It checks both a new tmux server and an existing server carrying a parent's session identity. It verifies the scoped environment, the runtime directory and disabled session-bus default, and tmux's terminal identity in the launched panes; the existing owner session stays unchanged. This adds one focused Linux job to the existing suite. Stub seats prove the environment passed to the pane; native Claude image paste and Shift+Enter handling still need separate verification (see #2041).

A second workflow, .github/workflows/portability-report.yml (job portability-report), lists lines your PR adds that contain machine-specific values: credentials, home or temp paths, network addresses, email addresses. It never fails because of a finding. To see the matched text locally (credentials show only their first characters), run node scripts/portability-report.mjs. By default it compares the merge-base with origin/main against HEAD; pass --staged to check staged changes instead.

The UI suite is a CI matrix leg even though it is advisory locally. Check your PR's status to see whether a red UI leg is treated as blocking. Branch-protection settings are not in the repository.

The local gate: npm run gate

scripts/gate-lane.mjs runs two legs: npm run lint and npm run test. It does not run test:ui, the scenarios or the packaging checks. On top of those legs it:

• holds a machine-wide lock on localhost port 40404 (override with OPENRIG_GATE_LANE_PORT). A second gate exits 2 straight away instead of waiting.
• refuses a dirty worktree. Refuses a node_modules that is a symlink, or @openrig/* workspace links that resolve outside this worktree. Both are common with shared git worktrees.
• deletes the stale vendored daemon bundle at packages/cli/daemon/ before the legs run.
• writes gate-lane-verdict.json at the repo root, or at OPENRIG_GATE_VERDICT, which must point inside the current worktree (a path outside it exits 3 before the legs run). The file holds the HEAD SHA, per-leg results and durations, and a note of other node/vitest/tsc processes running at the time. It fails if HEAD changes during the run. The file is not gitignored, so don't commit it.

Exit codes: 0 pass, 1 a leg failed, 2 lane busy, 3 the runner itself errored. OPENRIG_GATE_LANE_SMOKE=1 skips the legs to test the wiring only, and the verdict records smoke: true.

Stub-agent scenarios

What's in the box

Piece	Where
runtime: stub adapter	packages/daemon/src/adapters/stub-runtime-adapter.ts, with stub-runner.ts (the process that runs in the pane), stub-script.ts (the launch-script format) and stub-compaction.ts / stub-restore.ts
Scenario format and runner	packages/daemon/test/helpers/scenario-*.ts. Step verbs: up, down, send, restart, daemon, seed_regression are bound. restore, mutate, policy are rejected at load time (UNBOUND_STEP_VERB) because the runner has no shipped binding for them; emit parses but only refocus is bound. Assertion surfaces: ps, queue, stream, scope, pane, transcript, tui_socket, policy_provenance.
Scenario library	packages/test-system/scenarios/: 20 scenarios, 7 stub topologies, agents/, culture.md, per-seat stub scripts in scripts/
Runner fixtures	packages/daemon/test/fixtures/scenarios/ (scenario-01-per-seat-scripts, scenario-02-baton, scenario-10-one-view-state)
Host runner	packages/daemon/scripts/run-scenarios.mjs
In-container entry and result check	packages/test-system/ci/run.mjs, packages/test-system/ci/result.mjs
PR orchestrator	scripts/run-pr-scenarios.sh
Image	docker/testbed/Dockerfile, built by scripts/build-testbed-image.sh; docker/testbed/Dockerfile.scenarios is layered on top by scripts/run-pr-scenarios.sh

Host mode: while you're writing a scenario

The tmux server selection regression also uses these scaffold and lifecycle helpers in packages/daemon/test/tmux-server-isolation-daemon.test.ts. It runs a real stub seat through config changes, daemon restarts, capture, self-attachment and cleanup, while native socket observations verify that a personal namesake survives. The test also removes the effective-server record and verifies that destroy --all leaves the daemon, database and both servers' sessions intact before restoring the record and checking normal cleanup. The YAML grammar has no config-write or socket-inventory surface, so these assertions live in the Vitest integration test. It does not exercise a real provider model, SSH connection or graphical terminal window.

packages/daemon/test/tmux-server-two-instances.test.ts adds two real daemons sharing a scratch OS home and inherited personal tmux server, with separate OpenRig homes, databases and ports. Both enable isolation and control mode. It checks same-named seats through inventory, capture and verified writes, observes the marked control session on each selected socket, then stops one instance and checks that the other instance's pane identity, control session and delivery still work. The personal namesake remains untouched. This test requires a tmux version that supports control mode and the built CLI and daemon. Like CI's package suites, it should run with scratch homes and outbound network restricted to localhost; scripted seats prove terminal routing and rendered delivery, not model receipt.

npm run build    # the runner drives packages/cli/dist/bin-wrapper.js
node --import tsx packages/daemon/scripts/run-scenarios.mjs     # the fixture scenario-*.yaml files
node --import tsx packages/daemon/scripts/run-scenarios.mjs packages/daemon/test/fixtures/scenarios/scenario-02-baton.yaml

• The runner creates a scratch HOME/OPENRIG_HOME, its own tmux server and its own daemon. It builds the scenario environment from HOME, PATH and TERM only, so your shell's daemon-target variables (OPENRIG_URL, OPENRIG_PORT and similar), TMUX and OPENRIG_TEST_CLOCK_NOW are dropped rather than inherited. It does not touch the daemon or tmux server your own session uses.
• It accepts YAML paths only. Any flag, including --container, is refused before anything runs.
• It supplies no fault controller, so any scenario with a seed_regression step fails loudly at that step. That includes the library's queue-baton-survives-restart. Use the container path to run the seeded pair.
• scenario-10-one-view-state declares its own normaliser for its cross-surface equals check. Whether it passes in host mode has not been verified.

Container mode: the scenario pack CI runs

In CI the installed-scenario job runs scripts/run-pr-scenarios.sh with no arguments: both baton cases (fixture = scenario-02-baton.yaml, library = queue-baton-survives-restart.yaml), each run three times as healthy, then lost-baton, then healthy (six containers); the refocus-parked effect case runs healthy, seeded, then healthy (three containers); and the six passing-only cases (transcript = transcript-reads-addressed-seat.yaml, capture = capture-returns-addressed-seat.yaml, daemon-restart, refocus-active, refocus-parked-off, prompt-state) run once each, healthy only (six containers). That is fifteen fresh containers.

To run a case before you push, point DOCKER_HOST at a Docker engine you control over SSH:

npm ci
DOCKER_HOST=ssh://<your-host> bash scripts/run-pr-scenarios.sh \
  --remote --case library --mode healthy --out dist/scenario-check

• Options: --case fixture|library|transcript|capture|daemon-restart|refocus-parked| refocus-active|refocus-parked-off|prompt-state (default: all nine), --mode healthy|lost-baton|seeded (default: healthy, lost-baton, healthy) and --out <dir> (default dist/pr-scenarios). --mode applies to the baton cases (fixture and library) and to the refocus-parked effect case; the six passing-only cases (transcript, capture, daemon-restart, refocus-active, refocus-parked-off, prompt-state) always run once each, healthy. --mode lost-baton leaves them out of a full run, and --case transcript|capture --mode lost-baton is refused with exit 2. Use a fresh --out directory for each run you want to keep.
• --remote accepts only ssh:// values for DOCKER_HOST, and it refuses if DOCKER_CONTEXT is set. The engine must be Linux on amd64 or arm64. The script reads the server's platform, so a macOS/arm64 laptop driving a Linux/amd64 engine works.
• Without --remote, the script refuses to run anywhere except GitHub Actions on Linux.
• The source build and pack run on your machine. The image build and the containers run on the engine. Building the image needs network access for the pinned base image, Node and dependencies. The scenario containers have no network.
• The image is built from your working tree but tagged with git rev-parse HEAD. Commit first, so the tag names what you actually tested.
• It never pushes images. It removes only the containers it named. If the SSH connection drops, it prints the name of any container it could not remove.

What happens, in order:

1. scripts/build-testbed-image.sh packs @openrig/cli from your tree. It installs the package into a digest-pinned Debian image with Node, checks rig --version, and checks that the installed daemon loads. The result is tagged openrig-testbed:<sha>.
2. docker/testbed/Dockerfile.scenarios adds a bundled copy of packages/test-system/ci/run.mjs plus the fixture and library scenarios.
3. Each run uses a fresh container: --network none, read-only root plus a 512 MB /tmp tmpfs, --cap-drop ALL, no-new-privileges, non-root, 2 CPUs, 2 GiB, 256 PIDs and a 300-second deadline.
4. In lost-baton mode, the runner stops the daemon, changes exactly one queue row in the scratch database from in-progress to pending, and restarts the daemon. packages/test-system/ci/result.mjs then accepts only one outcome: a failure at the post-restart queue assertion, with baton-1 observed as pending at the expected destination. A startup error, a timeout or any other failure does not count as a caught regression. Healthy runs must pass with no fault armed.
5. The transcript and capture runs are passing form only: result.mjs accepts only a clean PASS of the named scenario, with no seed or fault. Their planted failures are product-code mutations, so they stay per-PR evidence and are not CI runs. Stub scripts work here because the runner executes the pipeline in host mode inside the image.

A single healthy run proves that one leg only. The paired fault run is what shows the assertion can catch the failure.

Adding a new case to this pack today means changing three files: scripts/run-pr-scenarios.sh (the --case list), CASES in packages/test-system/ci/result.mjs, and the fault controller in packages/test-system/ci/run.mjs, which is currently specific to the baton row. A passing-only case needs no fault controller: add it to PASSING_CASES and the --case list in scripts/run-pr-scenarios.sh, and to PASSING_CASES in result.mjs.

What a green stub scenario proves

• The packaged CLI installs into a clean Linux container and starts a daemon.
• rig up and rig down of a stub topology go through the real daemon, tmux, SQLite and CLI.
• Queue state, read through the shipped rig queue list path, survives a daemon restart.
• With the paired seeded run: the assertion fails when that state is broken.

What a stub can and cannot prove

The stub does not fabricate product outputs. It runs the real daemon, tmux and CLI. But it is not Claude Code or Codex, and it differs from them in ways that decide what a stub green means:

Area	What the stub does at e8f0ab34	What that means for your test
Consuming a message	stub-runner.ts runs its launch script once and then idles. It has no stdin reader, socket or other input channel. The default script prints [stub] scripted reply: acknowledged at boot, before anything has been sent.	A pane showing a reply, or your echoed text, does not prove the message was consumed. No stub scenario can currently prove "delivered and answered". Separately, rig send --verify means "appeared in the pane" (with transport.verify_input_box on, a Claude Code or Codex send means "the seat was idle before the paste and its input box took the exact text"), not acknowledgement (see rig send --help), and the scenario send step doesn't pass --verify at all.
Launch path	StubRuntimeAdapter types node <stub-runner> … into the pane (tmux.sendText, then Enter). Claude Code with an explicit permission mode launches through a managed launch (ClaudeManagedLaunch.prepare, tmux.sendShellCommand). Otherwise Claude Code, Codex and Pi launch through SeatLaunchEnvironment.command and tmux.sendShellCommand; the stub adapter does not get that environment. The shell-foreground check in session-transport.ts (unverifiedShellForeground) runs for every runtime except terminal and claude-code, and only codex has a native-process proof; Claude Code ordinary delivery applies its own uncertainty policy at the input boundary.	A stub seat doesn't exercise the seat launch environment, managed launch, wrappers or native-process identity (the class behind #197). Wrapping the stub in a shell wouldn't change that.
Permissions	validateNativePermissionSelection accepts only codex and claude-code. The stub carries the resolved floor / full_bypass posture, including one set by a declarative permission_policy, into its runner's --posture argument and READY line. It ignores non-interruptive launches, and it doesn't exercise Claude or Codex native permission behaviour.	Testing Claude permission modes or per-seat permission selection needs a real runtime.
Queue pickup and wake	No stub worker reacts to a nudge by claiming an item, working on it and handing it back.	The baton scenarios prove the claim survives a restart. They don't prove delivery, pickup or wake.
Reboot / tmux reset	daemon: {op: restart} restarts only the scenario daemon, and the tmux server keeps running. No step resets tmux.	Recycled pane IDs and stale bindings after a real reboot (the class behind #141) are not covered by a daemon restart.
Multi-rig and same-name identities	The up step always uses the scenario's top-level topology and ignores a per-step override. No step archives or removes a rig.	Cross-rig scope, same-name generations and archive/remove routing (the class behind #174) need runner work before they can be tested.
Step-time restore, compaction, mutation, policy	restore, mutate and policy steps are refused at load time (UNBOUND_STEP_VERB); emit parses but a non-refocus behavior throws UnboundActionError when the runner reaches it. Behaviours run only from a per-seat launch script (env.stub_scripts). A seat restart step maps to rig launch, which is not native resume or fork.	Step-time compaction and restore, and native resume and fork, are not covered.
Skill projection	The stub projects skills into <cwd>/.openrig/stub/skills/. Claude Code projects a team seat's skills into the seat's generated plugin in OpenRig's state, and into <cwd>/.claude/skills/ only for a seat without one.	A stub green says nothing about provider-specific projection (the class behind #159).
Model and provider	No model, no tokens. Stub scripts refuse usage_limit because it only exists on real runtimes.	Model behaviour, provider usage limits and real context usage need live evals or a real runtime.
Platform	CI's scenario job runs in a Linux container.	macOS process behaviour is covered by the macOS package-tests jobs, not by scenarios.

The most useful next pieces of harness work, roughly in dependency order:

1. A stub that consumes input and answers a fresh nonce.
2. A model-free test entry point shaped like a provider, that goes through the real native adapter and managed launch. Package tests such as claude-interactive-launch.test.ts and the *-native.test.ts files already drive the real adapters through tmux against an offline recorder; they prove command resolution and arguments. claude-executable-foreground.test.ts holds that recorder open in private Bash, zsh, dash and ksh panes to check parent PID, process-group ownership, tmux's pane command and parent-shell restoration for unmanaged fresh, fork and resume launches. These fixtures do not prove native readiness, history or answers.
3. Lifecycle and identity steps (tmux reset, archive/remove).
4. Then pickup, wake and restore scenarios.

Library status

Seven library scenarios run in CI: queue-baton-survives-restart (since #243) and refocus-parked-baton with their seeded pairs, and transcript-reads-addressed-seat, capture-returns-addressed-seat, daemon-restart-preserves-team-identity, refocus-active-baton and refocus-parked-baton-packet-off in passing form only. The fixture case runs the scenario-02-baton test fixture through the same seeded pair. Three more, down-stops-every-seat, send-renders-in-addressed-pane and up-relaunch-restores-a-two-seat-team, run with the local runner but not in CI, and compaction-restore-resumes-role runs the compaction round-trip's success path the same way. Those ten later scenarios are not in this table; "Status today" under Help wanted covers seven of them. The other nine scenarios are authored but not yet runnable as meaningful checks:

Scenario	Why it doesn't run as a check yet	Smallest useful next step
queue-baton-survives-restart	Runs in CI: healthy / seeded baton-drop / healthy	Extend it to handoff, block, resolve and unclaim
clean-lifecycle-no-residue	The post-down "nothing left over" assertion is commented out, because the format has no way to say "empty"	Add a real absence assertion
ps-scope-honesty	The second up reuses the first topology, and the expected match tolerates the other rig	Make up honour the per-step topology, then assert the exclusion
send-verify-means-rendered	The send step drops verify, and the stub has no input loop	Build the input-consuming stub and assert a response derived from the nonce
home-divergence-preseed-visible	The pipeline ignores env.pre_existing_tmux, and the default script never prints the expected trust reached	Recreate the real HOME split and whatever consumes it
stream-emit-durable-replay-live	Step-time emit is unbound	Assert the actual stream contract
policy-posture-survives-structural-edit	The validator refuses its policy and mutate steps at load time (UNBOUND_STEP_VERB — no shipped runner binding)	Rewrite around current per-seat permission selection and an observed launch
locked-restore-never-permissive	The validator refuses its policy and restore steps at load time (UNBOUND_STEP_VERB — no shipped runner binding)	Keep the invariant (the selection survives a restore) and use current modes
one-view-state-after-mutation-storm	The validator rejects its bare-list equals (EQUALS_NOT_DECLARATIVE) and refuses its mutate steps at load time (UNBOUND_STEP_VERB)	Declare the comparable fields and real mutations
kill-daemon-mid-handoff	A pane send doesn't create the queue item the next assertion expects, and later assertions use a shape the queue reader doesn't return	Rebuild the setup before treating a red result as meaningful

In most of these files, seed_regression steps that come after the assertions are historical markers, not executable fault injections.

Ordinary CLI journeys with stub seats

packages/daemon/test/stub-cli-journeys.integration.test.ts reuses the scenario helpers with one private daemon and two stub rigs. It checks three contracts through the real CLI: verified send appears in the addressed pane but not its sibling; current-rig and explicit rig listings contain exactly the expected seats; and a filtered stream item survives daemon restart and an idempotent emit retry.

After building, run the focused file with:

npm test -w @openrig/daemon -- test/stub-cli-journeys.integration.test.ts

The file also runs in the daemon package-test job. It covers these command contracts without extending the YAML grammar. The corresponding library scenarios above remain pending: pane rendering does not prove input consumption, durable stream replay does not prove live subscription, and this test does not exercise native providers, restore or policy changes.

packages/daemon/test/rig-edge-cli.integration.test.ts uses the same private scenario daemon and tmux helpers for edge repair. It removes a stub seat, re-adds its logical ID with the terminal runtime, then restores its cross-pod edges with one rig edge add per edge. It checks repeat IDs and edge counts, refusal without a write, unchanged sibling node/binding identities and pane PIDs, export, snapshot capture and exact rollback. It models the cross-runtime replacement workaround without invoking Claude Code or Codex. The YAML runner has no bound edge/remove/add verbs, so this journey drives the real CLI directly:

npm test -w @openrig/daemon -- test/rig-edge-cli.integration.test.ts

If your change touches an artery, add or extend a scenario (or say why not)

arteries.md lists the behaviours that everything else depends on. If your diff can affect one, your PR description should do one of these:

• link the scenario that covers it, and say which layer you ran it in;
• add or extend a scenario. For a regression scenario, show that the fixed code passes and that the planted old defect fails at the intended assertion. A setup crash or a timeout doesn't count;
• or say precisely why the stub can't exercise it. Use the table above, which covers message consumption, native launch and permissions, tmux reset and so on. Then say what you ran instead, for example a manual run with the real runtime.

This is part of normal review, not an extra approval step. It also isn't a claim that changes outside the arteries map are safe.

Live-model evals

packages/test-system/evals/ is a separate check from scenarios. A real seat receives a natural prompt and decides what to do. A case passes when the seat's transcript shows the expected rig context get <entry> command, in the expected order. Cases live in cases/selection.yaml (10) and cases/loading.yaml (4).

# Harness check: a fake provider fed a JSON map of prompt -> transcript.
# Any prompt without a transcript counts as an error, never as a pass.
npm run eval -w packages/daemon -- --transcripts <transcripts.json> --out <result.json>

# Live run: drives a real Claude Code seat
npm run eval -w packages/daemon -- --provider rig --seat <session> --out <result.json>
npm run eval -w packages/daemon -- --provider rig --seat-spec "$PWD/<one-seat-rig.yaml>" --out <result.json>

• Live runs use provider credits, and the results vary from run to run. CI never runs them.
• --provider rig needs OPENRIG_HOME set, and it supports Claude Code seats only. It refuses a Codex seat.
• --seat-spec runs rig up of a scratch rig against your current daemon, then rig down at the end. --seat attaches to an existing seat and leaves it running.
• evals/rig-seat/example-scratch-rig.yaml sets permission_policy: builtin:yolo and cwd: ".". It also references local:agents/blank, and no such agent directory ships beside it. Read the file and expect to adapt it before you use it.
• Before any run, the runner builds the production context package in a temp directory and refuses if any case's entry doesn't resolve in it.
• npm test covers the harness without a model: the case schema, the grader, the fixtures and production entry resolution (packages/daemon/test/eval-*.test.ts).

Docker testbed runbooks

docker/testbed/runbooks/ holds manual procedures, L0 to L6, that you run on a machine with Docker. They check the testbed image itself:

• L0: resolve the pinned base image and the stub asset list
• L1: PTY allocation
• L2: tmux server lifecycle
• L3: the daemon inside a container, settling a stub rig
• L4: the scenario environment still refuses a foreign daemon target inside a container
• L5: several containers acting as named hosts on one Docker network
• L6: the scenario runner driving a scenario through a container

Use them when you change docker/testbed/, scripts/build-testbed-image.sh or the container staging helpers. Ordinary PRs don't need them. They were written before the CI scenario job and contain internal planning references, so treat them as procedures to adapt, not as current status.

Help wanted: command families without a behavioural scenario

This list may be stale. It was taken from the rig --help tree of 0.6.3: 85 visible top-level families. At e8f0ab34, packages/cli/src/index.ts registers 87 top-level commands; roster and telemetry were added since and have no row yet. Before picking one up, check packages/test-system/scenarios/, the cases in scripts/run-pr-scenarios.sh, and the current rig --help. One scenario doesn't cover a family's every option, sequence or platform. A --help check only proves the command is discoverable. Existing unit tests are not counted here. They exist, but this list is about behavioural scenarios.

Status today: queue has partial coverage, namely create, claim and list, plus the claim surviving a restart. down has partial coverage in packages/test-system/scenarios/down-stops-every-seat.yaml: after rig down, rig ps reports every seat stopped and the rig still listed. send has partial coverage in send-renders-in-addressed-pane.yaml beside it: the envelope and the message body render in the addressed seat's pane. transcript has partial coverage in transcript-reads-addressed-seat.yaml: each of two seats' scripted output is readable through that seat's rig transcript --tail 200 --json. capture has partial coverage in capture-returns-addressed-seat.yaml: in a rig whose member names overlap by prefix (impl, impl2; dev-impl, dev-impl2 with the pod), rig capture <seat> returns each seat's own pane. The scenario catches a capture that resolves the seat by a prefix match on its member name (so dev-impl also matches dev-impl2) and takes the first hit, or any resolver that returns the same pane for both seats; it does not exercise a prefix match on the full seat name. All six run with the local runner (run-scenarios.mjs). The transcript, capture and daemon scenarios also run in the CI job, in passing form only; the down, send and up scenarios don't. They do not prove that panes are gone or that a seat consumed a message. The transcript case does not establish ordering, uniqueness, sibling-token absence, tail limits or restart persistence. up has partial coverage in up-relaunch-restores-a-two-seat-team.yaml: a two-seat stub team comes up, goes down, and the same spec starts a new team in place of the stopped one with the same name, exactly two seats, both running, not archived, so a relaunch that re-records the old seats into the team's row fails on the final ps counts. It does not prove that conversations survive the round trip (that is the --existing restore path), that a seat consumed a message, or that the panes are gone after down. daemon has partial coverage in daemon-restart-preserves-team-identity.yaml: a two-seat stub team is up, the runner restarts the scratch daemon, and rig ps --json must report the same name still running with exactly two seats, both running, not archived, so a restart that loses the team's row, or changes its seat counts or status, fails on the post-restart ps; rig down then records it stopped, so the restart left a governable team. It does not prove that queue items survive the restart (that is queue-baton-survives-restart.yaml), that a kill inside a handoff window is reconciled (that is kill-daemon-mid-handoff.yaml), the port-and-process half of the row, or that the panes are gone after down. compaction-restore-resumes-role covers the compaction round-trip's success path: the seat's launch script (delivered through env.stub_scripts) fires the real precompact and restore seams in order, and impl's own pane carries the injected directive. It runs with the local runner, not in CI (container mode refuses env.stub_scripts loud and named), and its seeded wrong-seat-restore negative case is not armed yet.

Before you start, two honest constraints:

• Today's step verbs can't run arbitrary rig commands. Many rows below need a runner binding first. One proposal (not implemented) is a single structured-argv step that runs the installed CLI through the existing rigBin, instead of one new verb per command.
• Rows that need a seat to answer (anything with "nonce") depend on the input-consuming stub described above.
• expect has no negative, absence or count form. match is a structural subset and contains is a substring, so "the sibling received nothing", "the panes are gone" or "no duplicates" can't be asserted yet. An exact field value, such as runningCount: 0 on the ps surface, is the closest available form.
• The restart step runs rig launch <rig> <node> --json and passes no other flag; the node is the logical ID (for example dev.qa). A stub seat has no resume token. rig launch has no --fresh option: a deliberate fresh start of a stub seat after down goes through rig up --existing <rig> --fresh <seat> or rig seat launch <seat> --fresh --reason "<why>", so the launch row needs a runner binding to one of those first.

Group	Family	Proposed first check (observable result)
lifecycle	start	Restore two stopped stub seats. Both answer distinct fresh nonces, and no second daemon starts.
lifecycle	daemon (partial)	Restart-preserving-the-persisted-team-row is covered by daemon-restart-preserves-team-identity.yaml. Next: port and process agreement through public commands, and stop, which needs a runner binding.
lifecycle	bootstrap	Bootstrap a minimal stub spec. Seats come up ready and the selected spec is the one used.
lifecycle	up (partial)	Relaunch-not-duplicating-identities is covered by up-relaunch-restores-a-two-seat-team.yaml. Next: each seat consumes its own send, which needs the input-consuming stub.
lifecycle	down	(partial) Next: its panes are gone, a sibling rig still answers, retained data follows the docs.
lifecycle	create	Create a one-seat rig through the public path. It appears and answers without a hand-written topology.
lifecycle	launch	Fresh, relaunch, resume and fork on provider-shaped fixtures. Correct restored token, one live process.
reads	status	Two rigs in different states. Status matches their exact identities and transitions.
reads	ps	Two rigs with overlapping seat names. Rig and node filters return exactly the intended members.
reads	usage	Deterministic stub telemetry. top and series respect seat and time bounds. Missing data isn't reported as zero.
reads	health	Known fresh, stale and cleared findings. Scope, explain and diagnosis values match the fixture.
continuity	snapshot	Snapshot two seats. Rig, membership and snapshot bytes survive a daemon restart.
continuity	restore	Restore after a daemon and tmux reset. Seat identity and nonce answers survive.
continuity	agent-image	create/show/pin/unpin/delete for a provider-shaped fixture. References and identity match.
continuity	fork	Fork at a known history token. The child keeps the prefix with its own identity, and the parent still answers.
continuity	restore-check	One restorable and one missing transcript. Readiness tells them apart without relaunching.
continuity	restore-packet	write/read/validate round trip. A malformed packet fails clearly.
continuity	compact-plan	Scripted history and usage fixtures. Candidates and exclusions match thresholds, and nothing is compacted.
continuity	compact	Input-driven compact and restore keep the role and token. Messages before and after reach the same seat.
continuity	handover	Scripted predecessor and successor. Context is delivered before the rebind, and the successor answers.
recovery	crash-cart	Stop only the scratch daemon. The output names paths and state, and the command changes nothing.
recovery	preflight	Ready and missing-dependency fixtures produce named checks without launching seats.
recovery	doctor	Healthy and damaged scratch installs. Exact diagnosis, no automatic repair.
recovery	destroy	In a disposable scenario HOME only. Removes the intended state and keeps explicitly retained markers.
registry	gateway	Add, show, update and remove a synthetic human. Reads match exact fields, and removal sticks.
registry	host	add/list/select/rename isolated entries that resolve to fixture endpoints. Pairing needs two container daemons.
coordination	parked	An owed item with a stopped worker is reported. A held worker with a live wake is distinguished.
coordination	stream	emit/list/show/archive an exact item. A restart and a bounded watch keep the order with no duplicates.
coordination	queue	(partial) handoff → successor owned by --to and source asserted closed (handed-off), block → parked blocked, resolve → unpark to in-progress (owner nudge suppressed), unclaim → pending with destination kept — each with exact items and destinations asserted via expect queue (rig queue list --json --full --all) in queue-transitions-reach-exact-states.yaml; baton restart coverage stays in queue-baton-survives-restart.yaml.
coordination	project	Acquire a lease, classify one item twice, read one projection. The wrong holder cannot act.
coordination	view	Register a view over known items. list/show return the exact filtered result after a mutation.
coordination	watchdog	Register one short wake, see one response, stop it, prove no further deliveries.
coordination	workflow	validate/compile/instantiate a tiny flow. It transitions and resumes after a restart with no duplicate queue work.
coordination	chatroom	send/history/topic/clear in a scratch room. A bounded watcher sees exactly the ordered messages.
coordination	heartbeat	Known execution-proof and missing-closure fixtures report the right status. Empty output isn't completion.
authoring	export	Export a stub rig, then validate and re-import it. Membership and configuration are equivalent.
authoring	import	Import a minimal spec, and the export matches it. Malformed input leaves state unchanged.
authoring	package	validate/plan/install/rollback in scratch. Installed files and prior state match.
authoring	requirements	Complete and missing capability fixtures produce the right named requirements.
authoring	bundle	create/inspect/install a local bundle and round-trip it. A missing asset gives a named failure.
authoring	agent	Validate a good and a malformed stub AgentSpec. The diagnostic names the bad field.
authoring	spec	validate/audit/preflight/show a minimal spec, with predictable diagnostics for a malformed field.
authoring	specs	add/show/preview/rename/remove a library fixture. A missing reference fails explicitly.
authoring	workspace	validate/doctor known roots plus one missing-frontmatter case. Reports the exact affected path.
authoring	config	set/get/reset a scratch key and init a workspace. Readback and files match the requested root.
presentation	ui	Open the URL through a fake browser executable and read a local UI route. Rendering is out of scope.
presentation	tui	Drive the TUI command bar and socket over two stub rigs. select, mutate and back show exact identities.
presentation	terminal	open/status/views against a fake terminal-app endpoint. Correct target and layout arguments.
adoption	discover	One unmanaged and one owned scratch tmux session. Only the intended candidate is discovered.
adoption	attach	Attach to the intended seat. whoami reports it without duplicating the process.
adoption	bind	Bind one discovered pane. A different, already-occupied identity is refused.
adoption	adopt	Adopt two scratch sessions. PIDs and history are preserved, and the identities become addressable.
adoption	reconcile-session	Rebind a hand-resumed provider-shaped fixture without starting or typing into another process.
adoption	unclaim	Release an adopted seat. The tmux process and its history stay alive.
adoption	release	Release one rig's claimed sessions. Processes survive, and a sibling rig is unchanged.
identity	archive	Archive an old same-name generation. The current one stays routable, and the old data is kept.
identity	unarchive	Unarchive a retained generation without taking over another live seat.
identity	remove	Remove an archived generation or one member. Only that identity goes, and a reused name still routes.
identity	whoami	Inside each stub seat, it returns that seat, not a sibling or the caller.
identity	seat	Status, held-message guard, stop/launch and resume token keep the right process and permissions.
integration	mcp	Serve against a scratch daemon. One listed tool's value matches CLI state.
integration	slack	Manifest and config round trip, plus deterministic inbound and outbound fixture requests. No real Slack account.
integration	ask	Search fixture transcripts via a fake model endpoint. Citations resolve to the stored lines.
integration	file	Copy a fixture payload through isolated transport endpoints. Verify bytes and paths.
integration	env	A fixture service-backed rig. status/logs/down reflect that service and stop only it.
delivery	transcript (partial)	Own scripted output from two addressed seats is covered by transcript-reads-addressed-seat.yaml. Remaining: ordering, uniqueness, no other seat's token, tail limits and restart persistence; absence/count assertions need a runner binding.
delivery	send	(partial) Next: send a fresh nonce and require an answer derived from it. Boot text or echo can't pass.
delivery	capture	(partial) Each seat's own token, under prefix-overlapping member names. Next: the not-found, tail (--lines), sibling-absence and --rig/--pod cases, which need an error form, a tail option, an absence form and a multi-target surface in the runner.
delivery	broadcast	Broadcast a nonce to a rig or pod. Each intended seat answers once, and an excluded seat gets nothing.
delivery	walk	Walk two pieces through an input-consuming stub. Order is kept, and a bad piece isn't reported as delivered.
context	context	add/get/compose/trace/profile over a fixture project. Output bytes, order and roots match.
context	plugin	list/show/used-by/validate a fixture plugin. A missing resource is not the same as a valid empty selection.
context	skill	Resolve and audit a fixture skill, then inspect the projection the consumer actually gets.
context	mode	set/effective/cite/unset at nested scopes. Precedence holds, and the inherited value returns after unset.
context	startup-proof	Submit a proof from a real stub identity. Check the stored attribution and the duplicate behaviour.
permissions	policy	Apply and read an explicit permission selection using a provider-shaped fixture. It survives a restore, and the default is unchanged.
accounts	auth	Synthetic auth files in scratch: save, list, validate, switch. No provider contact, no real credentials.
accounts	provider	Synthetic account, binding and signal state. The precheck refuses where it should, and a switch keeps seat history.
topology	expand	Add a pod to a live stub rig. Old seats survive, and new members answer nonces.
topology	add	Add one seat. Exact binding, siblings preserved, and the duplicate-name behaviour is explicit.
topology	grow	Grow two seats. Identities are unique, config is inherited, and existing history is kept.
topology	shrink	Remove a pod. Unrelated seats still answer, and the removed membership is gone.
onboarding	setup	Scripted newcomer answers against synthetic providers. No install or login for an unused provider.
work	scope	Create a mission and a slice. move/progress/notes/graph readback keep identity and paths.
work	proof	add/judge/show evidence in a fixture scope. Readiness updates without overwriting earlier proof.

How to make a case useful:

• Assert actual state or content, not just exit code 0. Where ordinary use includes bad input, add an invalid or missing input case too.
• Reuse the existing runner and the installed CLI. Don't add a second scenario engine.
• If a scenario exposes an existing product bug, keep the failing case and open an issue. Include the exact source revision, the shortest failing sequence, and the intended versus observed result. Don't weaken the assertion to make it pass. Keep "the harness setup failed" separate from "the product misbehaved".
• A check that only covers the success path is still useful. Just don't call it a regression detector until you have seen its negative case fail.

Existing docs in this area that are out of date

Read these with care. This page notes them and proposes no changes.

• The headers of transcript-reads-addressed-seat.yaml and capture-returns-addressed-seat.yaml say "host mode only". Both run in the CI container job, which executes the host-mode pipeline inside the image.
• packages/test-system/scripts/README.md and packages/test-system/ROUTING.md are dependency-routing notes from when the scenarios were first written. Several of the items they list have since landed. Treat them as history.
• docker/testbed/runbooks/: see above. They are manual procedures that predate the CI job.