M10: results, report, CLI run/validate/init/report with required-vs-allowed

.gitignore

@@ -3,3 +3,4 @@ review/
 .opencode-ci/out/
 coverage/
 *.log
+**/.opencode-ci/out/

biome.json

@@ -15,7 +15,8 @@
 			"!review",
 			"!schema",
 			"!test/fixtures/**/node_modules",
-			"!.opencode-ci/out"
+			"!.opencode-ci/out",
+			"!**/.opencode-ci/out"
 		]
 	}
 }

docs/reports/m10-implementation.md

@@ -0,0 +1,128 @@
+# M10 Implementation Report — Results, report, CLI run/validate/init/report
+
+- **Stage ID:** M10 (report: `src/report/` + `src/run.ts`, utils, CLI, `src/index.ts`, examples, unit/e2e tests)
+- **Date:** 2026-10-17
+- **Branch:** `impl/v1` (not committed, per task instructions)
+
+## Summary
+
+Implemented the M10 runner surface: a central `CaseResult`/`ResultsFile`
+shape, a pure Markdown renderer with required/allowed-failure/skip cells
+and per-failure `<details>` blocks, the `run` orchestrator (load → tier 1 →
+tier 2 → skipped-live → bun-test → results.json + report.md → exit code),
+a hand-parsed CLI (`init`, `validate`, `run`, `report`, `--version`,
+`--help`), the public programmatic API, and the `init` example templates.
+Acceptance verified manually and by automation: `run` in `tool-basic`
+passes and writes `report.md`, `run` in `throws` exits 1, `validate` in
+`no-entry` exits 2, and a unit test snapshot-checks the Markdown.
+
+## Files created
+
+- `src/report/results.ts` — `CaseResult` (exact M10 shape: `id`,
+  `tier: "load"|"scripted"|"live"`, `opencode`, `pass`, `skipped?`,
+  `description?`, `failures?`, `tools?`, `steps?`, `costUSD?`,
+  `tokens?`, `durationMs`, `textExcerpt?`, `attempts?`, `judge?`,
+  `kept?`) and `ResultsFile { version: 1, at, tool, plugin, results }`.
+- `src/report/markdown.ts` — `renderMarkdown(results, opts?
+  { allowedVersions?, sha? })`, `gitSha()` (`GITHUB_SHA` else
+  `git rev-parse --short HEAD` else `"unknown"`), `appendStepSummary()`
+  (appends to `$GITHUB_STEP_SUMMARY` when set, never throws). Header with
+  plugin/tool/opencode versions/SHA/totals; one `Tier|Case|Result|Tools|
+  Steps|Cost|Time` table per opencode version; cells `✅ pass`, `❌ fail`,
+  `⚠️ allowed failure`, `⏭️ skipped (reason)`; `<details>` per failure
+  with the failure list and `kept` evidence path; live cost/token totals.
+- `src/util/log.ts` — `createLogger({quiet, verbose})`; info suppressed
+  when quiet, debug only when verbose, warn/error always (error even when
+  quiet).
+- `src/util/rows.ts` — `rows<T>(json)`: normalizes `[]` or `{data: []}`.
+- `src/util/fs.ts` — `ensureDir`, `isFile` (never throws), `writeJsonl`,
+  `readJsonl` (missing file → `[]`, bad lines skipped), `appendFile`
+  (never throws).
+- `src/run.ts` — `runCi(opts?)` orchestrator plus `isTuiOnly()` and the
+  `TierName`/`RunOptions`/`RunOutcome` types. Order per spec: load+validate
+  → resolve entries (exit 2 upstream) → install/build once → per version:
+  resolve binary, tier 1 (`runLoad` or `runUiStatic`), tier-1-failure marks
+  tier 2/3 `skipped: "tier 1 failed"`, TUI-only marks tier 2/3
+  `skipped: "TUI-only plugin"`, tier 2 via `pool(concurrencyLimit())`,
+  live scenarios `skipped: "live tier not implemented (M11)"` → bun-test
+  escape hatch (once, `OPENCODE_BIN` = required binary, last-40-lines
+  failures) → `results.json` + `report.md` + step summary. `--gate` warns
+  (M12) and continues. `--keep` sets `OPENCODE_PLUGIN_CI_KEEP=1` (chose env
+  over churning M8/M9 tier signatures). Exit: 0 unless a required-version
+  case failed (`!pass && !skipped`).
+- `src/index.ts` — programmatic API: `Sandbox`, `startStub`,
+  `readProbe`, `evaluate`, `check`, `resolveEntries`, `writeShim`,
+  `resolveOpencode`, `parseEvents`, `textOf`, `toolUses`,
+  `probePluginDirs()`, plus related types, `renderMarkdown`,
+  `appendStepSummary`, `runCi`.
+- `examples/.opencode-ci/config.yaml` + `scenarios/example.yaml` —
+  minimal valid init templates (M13 will expand).
+- `test/unit/report.test.ts` — fixed-`ResultsFile` assertions (header,
+  tables, all four result cells, `<details>` + evidence, live totals)
+  plus `toMatchSnapshot()`.
+- `test/e2e/cli.test.ts` — spawns the real CLI with `Bun.spawn` (async)
+  and `OPENCODE_BIN`: tool-basic `run --scenario tool-basic-pass` → 0
+  with `results.json`+`report.md`; throws `run` → 1; no-entry `validate`
+  → 2 matching `/no server entry/i`; tool-basic `validate` → 0.
+- `test/unit/__snapshots__/report.test.ts.snap` — generated by bun test.
+
+## Files modified
+
+- `src/tiers/load.ts` — `CaseResult` moved to `src/report/results.ts`;
+  imports it and re-exports for backward compatibility (`./load.ts`
+  importers keep working).
+- `src/tiers/scripted.ts`, `src/tiers/ui-static.ts` — import
+  `CaseResult` from `../report/results.ts`.
+- `src/cli.ts` — rewritten: hand-parsed `init [--force]` (copies
+  `examples/.opencode-ci`, refuses overwrite without `--force`),
+  `validate` (load+validate, entries, summary; exit 2 on `ConfigError`),
+  `run` (repeatable/comma `--tier/--scenario/--opencode`, `--out`,
+  `--keep/--gate/--verbose/--quiet`; `ConfigError` → 2, unexpected → 1),
+  `report [<results.json>]` (re-renders `report.md` beside the results
+  file + step summary), `--version`/`--help`, unknown → 2.
+- `package.json` — added `examples` to `files` so `init` works installed.
+- `biome.json` — added `!**/.opencode-ci/out` ignore: generated fixture
+  evidence (e.g. `test/fixtures/.../.opencode-ci/out/`) was failing
+  `bun run lint` on format.
+- `.gitignore` — added `**/.opencode-ci/out/` so per-fixture evidence
+  from e2e runs is never committed.
+- `src/opencode/logs.ts`, `src/sandbox/sandbox.ts`,
+  `test/e2e/sandbox.test.ts`, `test/unit/logs.test.ts` — mechanical
+  `obj["key"]` → `obj.key` fixes for biome `useLiteralKeys` (newer biome
+  than the milestone was written against; behavior-identical; `check`
+  was red on the pristine tree).
+
+## Test results
+
+- `bun run check` — green: `tsc --noEmit` clean, `biome check .` clean,
+  `bun test ./test/unit`: **154 pass, 0 fail** (13 files, incl. new
+  report snapshot test).
+- `bun test ./test/e2e/cli.test.ts --timeout 600000` — **4 pass, 0 fail**
+  (~5 s; stub runs are local and fast).
+- Manual: `run --scenario tool-basic-pass` in `tool-basic` → exit 0,
+  real tier-1 + scripted evidence, `report.md` written; `run` in
+  `throws` → exit 1 with `fixture boom` detail block and tier-1-failed
+  skip; `validate` in `no-entry` → exit 2 "No server entry"; `init`
+  writes 2 files, refuses overwrite, `--force` overwrites; `report`
+  re-renders `report.md` and appends `$GITHUB_STEP_SUMMARY`; `--version`
+  prints `0.1.0`, unknown args exit 2.
+
+## Self-audit notes
+
+- Fail-closed: schema/entry errors → `ConfigError` → exit 2; unexpected
+  runtime errors in `run` → exit 1; bad results files in `report` →
+  exit 2 with usage. Skips never pass a failing gate: exit counts only
+  `!pass && !skipped` on the required version.
+- No stubs: `live` is an explicit skip with reason (M11 owns it); gate is
+  an accepted-but-warn flag (M12 owns it). Both are reported, not hidden.
+- Async spawn only for opencode and `bun test` (`Bun.spawn`); sync spawn
+  used solely for `git rev-parse` in `gitSha()` (no event loop at stake).
+- Two off-by-one `import.meta.url` bugs caught by manual testing:
+  `src/run.ts` and `src/cli.ts` used `../../package.json` /
+  `../../examples/...` (one level too many); fixed to `../…`.
+- A `**/` sequence inside a `/** … */` comment in `src/run.ts` closed
+  the comment early and broke typecheck; reworded. Grep confirms no
+  other `/**/`-style `*/` sequences in block comments (one remaining
+  `**/` is in a `//` comment, safe).
+- Not committed, per task instructions (`git status` shows M10 files
+  staged as modified/untracked, ready for review).

examples/.opencode-ci/config.yaml

@@ -0,0 +1,7 @@
+# Example config written by `opencode-plugin-ci init`. Edit to match your plugin.
+plugin:
+  path: .
+expect:
+  tools: []
+opencode:
+  versions: ["2.0.26"]

examples/.opencode-ci/scenarios/example.yaml

@@ -0,0 +1,8 @@
+id: example
+description: Example scripted scenario (edit me).
+mode: scripted
+prompt: Say hello.
+script:
+  - text: done
+expect:
+  text: { contains: done }

package.json

@@ -19,6 +19,7 @@
 		"src",
 		"probe",
 		"schema",
+		"examples",
 		"README.md",
 		"LICENSE",
 		"NOTICE"

src/cli.ts

@@ -1,10 +1,54 @@
+// Argument parsing and command dispatch. Parsed by hand; no CLI framework.
+//
+// opencode-plugin-ci init        [--force]
+// opencode-plugin-ci validate
+// opencode-plugin-ci run         [--tier load|scripted|live]... [--scenario id]...
+//                                [--opencode v]... [--out dir] [--keep] [--gate]
+//                                [--verbose] [--quiet]
+// opencode-plugin-ci report      [<results.json>]
+// opencode-plugin-ci --version | --help
+//
+// Exit codes: 0 pass, 1 failure, 2 configuration error / unknown arguments.
+import {
+	copyFileSync,
+	existsSync,
+	readdirSync,
+	readFileSync,
+	statSync,
+} from "node:fs";
+import { join, resolve } from "node:path";
+import { fileURLToPath } from "node:url";
+import { ConfigError, loadAll } from "./config/load.ts";
+import { ensureEntriesOrThrow } from "./plugin/entry.ts";
+import {
+	appendStepSummary,
+	deriveAllowedVersions,
+	renderMarkdown,
+} from "./report/markdown.ts";
+import type { ResultsFile } from "./report/results.ts";
+import { isTuiOnly, runCi, type TierName } from "./run.ts";
+import { ensureDir } from "./util/fs.ts";
+import { createLogger } from "./util/log.ts";
+
 const USAGE = `opencode-plugin-ci
 
 Usage:
+  opencode-plugin-ci init [--force]
+  opencode-plugin-ci validate
+  opencode-plugin-ci run [--tier load|scripted|live] [--scenario <id>] [--opencode <v>] [--out <dir>] [--keep] [--gate] [--verbose] [--quiet]
+  opencode-plugin-ci report [<results.json>]
   opencode-plugin-ci --version
   opencode-plugin-ci --help
 `;
 
+/** Usage/validation error inside the CLI: always exits 2. */
+class UsageError extends Error {
+	constructor(message: string) {
+		super(message);
+		this.name = "UsageError";
+	}
+}
+
 async function readVersion(): Promise<string> {
 	const pkgUrl = new URL("../package.json", import.meta.url);
 	const file = Bun.file(pkgUrl);
@@ -15,15 +59,339 @@ async function readVersion(): Promise<string> {
 	return pkg.version;
 }
 
+function takeValue(args: string[], i: number, flag: string): string {
+	const value = args[i + 1];
+	if (value === undefined || value.startsWith("-")) {
+		throw new UsageError(`${flag} needs a value.`);
+	}
+	return value;
+}
+
+/** Split `--flag a,b --flag c` style repeatable values on commas. */
+function collectValues(raw: string[]): string[] {
+	const out: string[] = [];
+	for (const item of raw) {
+		for (const part of item.split(",")) {
+			const trimmed = part.trim();
+			if (trimmed.length > 0) {
+				out.push(trimmed);
+			}
+		}
+	}
+	return out;
+}
+
+interface RunFlags {
+	tiers: TierName[];
+	scenarios: string[];
+	opencodes: string[];
+	out?: string;
+	keep: boolean;
+	gate: boolean;
+	verbose: boolean;
+	quiet: boolean;
+}
+
+function parseRunArgs(args: string[]): RunFlags {
+	const tiers: TierName[] = [];
+	const scenarios: string[] = [];
+	const opencodes: string[] = [];
+	let out: string | undefined;
+	let keep = false;
+	let gate = false;
+	let verbose = false;
+	let quiet = false;
+	for (let i = 0; i < args.length; i++) {
+		const arg = args[i] as string;
+		if (arg === "--tier" || arg.startsWith("--tier=")) {
+			const raw =
+				arg === "--tier"
+					? takeValue(args, i, "--tier")
+					: arg.slice("--tier=".length);
+			if (arg === "--tier") {
+				i++;
+			}
+			for (const tier of collectValues([raw])) {
+				if (tier !== "load" && tier !== "scripted" && tier !== "live") {
+					throw new UsageError(
+						`Unknown --tier ${JSON.stringify(tier)}: expected load, scripted or live.`,
+					);
+				}
+				tiers.push(tier);
+			}
+		} else if (arg === "--scenario" || arg.startsWith("--scenario=")) {
+			const raw =
+				arg === "--scenario"
+					? takeValue(args, i, "--scenario")
+					: arg.slice("--scenario=".length);
+			if (arg === "--scenario") {
+				i++;
+			}
+			scenarios.push(...collectValues([raw]));
+		} else if (arg === "--opencode" || arg.startsWith("--opencode=")) {
+			const raw =
+				arg === "--opencode"
+					? takeValue(args, i, "--opencode")
+					: arg.slice("--opencode=".length);
+			if (arg === "--opencode") {
+				i++;
+			}
+			opencodes.push(...collectValues([raw]));
+		} else if (arg === "--out" || arg.startsWith("--out=")) {
+			out =
+				arg === "--out"
+					? takeValue(args, i, "--out")
+					: arg.slice("--out=".length);
+			if (arg === "--out") {
+				i++;
+			}
+		} else if (arg === "--keep") {
+			keep = true;
+		} else if (arg === "--gate") {
+			gate = true;
+		} else if (arg === "--verbose") {
+			verbose = true;
+		} else if (arg === "--quiet") {
+			quiet = true;
+		} else if (arg === "--help" || arg === "-h") {
+			console.log(USAGE);
+			process.exit(0);
+		} else {
+			throw new UsageError(`Unknown argument for run: ${arg}`);
+		}
+	}
+	return { tiers, scenarios, opencodes, out, keep, gate, verbose, quiet };
+}
+
+function exampleSourceDir(): string {
+	return fileURLToPath(new URL("../examples/.opencode-ci", import.meta.url));
+}
+
+function listFilesRecursive(dir: string): string[] {
+	const out: string[] = [];
+	for (const name of readdirSync(dir).sort()) {
+		const full = join(dir, name);
+		let isDir = false;
+		try {
+			isDir = statSync(full).isDirectory();
+		} catch {
+			continue;
+		}
+		if (isDir) {
+			out.push(...listFilesRecursive(full));
+		} else {
+			out.push(full);
+		}
+	}
+	return out;
+}
+
+async function cmdInit(args: string[]): Promise<number> {
+	let force = false;
+	for (const arg of args) {
+		if (arg === "--force") {
+			force = true;
+		} else if (arg === "--help" || arg === "-h") {
+			console.log(USAGE);
+			return 0;
+		} else {
+			console.error(`Unknown argument for init: ${arg}\n`);
+			console.error(USAGE);
+			return 2;
+		}
+	}
+	const cwd = process.cwd();
+	const source = exampleSourceDir();
+	if (!existsSync(source)) {
+		console.error(`Init templates missing: ${source} does not exist.`);
+		return 2;
+	}
+	const files = listFilesRecursive(source);
+	const dest = join(cwd, ".opencode-ci");
+	const planned = files.map((file) =>
+		join(dest, file.slice(source.length + 1)),
+	);
+	const existing = planned.filter((file) => existsSync(file));
+	if (existing.length > 0 && !force) {
+		console.error(
+			`Refusing to overwrite ${existing.length} existing file(s) (e.g. ${existing[0]}). Re-run with --force to overwrite.`,
+		);
+		return 2;
+	}
+	for (let i = 0; i < files.length; i++) {
+		ensureDir(join(planned[i] as string, ".."));
+		copyFileSync(files[i] as string, planned[i] as string);
+	}
+	console.log(
+		`Wrote ${planned.length} file(s) to ${dest}${force ? " (overwrote existing files)." : "."}`,
+	);
+	return 0;
+}
+
+async function cmdValidate(): Promise<number> {
+	const cwd = process.cwd();
+	const logger = createLogger();
+	try {
+		const { config, scenarios, warnings } = loadAll(cwd);
+		for (const warning of warnings) {
+			logger.warn(warning);
+		}
+		const pluginPath = resolve(cwd, config.plugin.path);
+		const entries = ensureEntriesOrThrow(pluginPath, config.plugin.entry);
+		const tuiOnly = isTuiOnly(config, entries);
+		const byMode = new Map<string, number>();
+		for (const scenario of scenarios) {
+			byMode.set(scenario.mode, (byMode.get(scenario.mode) ?? 0) + 1);
+		}
+		console.log(`plugin: ${pluginPath}${tuiOnly ? " (TUI-only)" : ""}`);
+		console.log(
+			`entries: server=${entries.server ?? "-"} tui=${entries.tui ?? "-"}`,
+		);
+		console.log(`opencode versions: ${config.opencode.versions.join(", ")}`);
+		console.log(
+			`scenarios: ${scenarios.length}` +
+				(byMode.size > 0
+					? ` (${[...byMode.entries()].map(([mode, n]) => `${n} ${mode}`).join(", ")})`
+					: ""),
+		);
+		return 0;
+	} catch (error) {
+		if (error instanceof ConfigError) {
+			console.error(`Configuration error: ${error.message}`);
+			return 2;
+		}
+		throw error;
+	}
+}
+
+async function cmdRun(args: string[]): Promise<number> {
+	let flags: RunFlags;
+	try {
+		flags = parseRunArgs(args);
+	} catch (error) {
+		if (error instanceof UsageError) {
+			console.error(`${error.message}\n`);
+			console.error(USAGE);
+			return 2;
+		}
+		throw error;
+	}
+	try {
+		const outcome = await runCi({
+			tiers: flags.tiers.length > 0 ? flags.tiers : undefined,
+			scenarios: flags.scenarios.length > 0 ? flags.scenarios : undefined,
+			opencodes: flags.opencodes.length > 0 ? flags.opencodes : undefined,
+			out: flags.out,
+			keep: flags.keep,
+			gate: flags.gate,
+			verbose: flags.verbose,
+			quiet: flags.quiet,
+		});
+		return outcome.exitCode;
+	} catch (error) {
+		if (error instanceof ConfigError) {
+			console.error(`Configuration error: ${error.message}`);
+			return 2;
+		}
+		console.error(error instanceof Error ? error.message : String(error));
+		return 1;
+	}
+}
+
+function readResultsFile(path: string): ResultsFile {
+	let text: string;
+	try {
+		text = readFileSync(path, "utf8");
+	} catch {
+		throw new UsageError(`Results file not found: ${path}`);
+	}
+	let parsed: unknown;
+	try {
+		parsed = JSON.parse(text) as unknown;
+	} catch {
+		throw new UsageError(`Results file is not valid JSON: ${path}`);
+	}
+	if (
+		parsed === null ||
+		typeof parsed !== "object" ||
+		!Array.isArray((parsed as { results?: unknown }).results)
+	) {
+		throw new UsageError(
+			`Results file has no results array: ${path} (expected results.json from \`run\`).`,
+		);
+	}
+	return parsed as ResultsFile;
+}
+
+async function cmdReport(args: string[]): Promise<number> {
+	let target: string | undefined;
+	for (const arg of args) {
+		if (arg === "--help" || arg === "-h") {
+			console.log(USAGE);
+			return 0;
+		}
+		if (arg.startsWith("-")) {
+			console.error(`Unknown argument for report: ${arg}\n`);
+			console.error(USAGE);
+			return 2;
+		}
+		if (target !== undefined) {
+			console.error("report takes at most one results.json path.\n");
+			console.error(USAGE);
+			return 2;
+		}
+		target = arg;
+	}
+	const resultsPath = resolve(
+		target ?? join(process.cwd(), ".opencode-ci/out/results.json"),
+	);
+	try {
+		const results = readResultsFile(resultsPath);
+		const markdown = renderMarkdown(results, {
+			allowedVersions: deriveAllowedVersions(results),
+		});
+		const reportPath = join(resultsPath, "..", "report.md");
+		await Bun.write(reportPath, markdown);
+		appendStepSummary(markdown);
+		console.log(`Wrote ${reportPath}.`);
+		return 0;
+	} catch (error) {
+		if (error instanceof UsageError) {
+			console.error(`${error.message}\n`);
+			console.error(USAGE);
+			return 2;
+		}
+		throw error;
+	}
+}
+
 export async function main(argv: string[]): Promise<number> {
 	if (argv.includes("--version")) {
 		console.log(await readVersion());
 		return 0;
 	}
-	if (argv.includes("--help") || argv.includes("-h") || argv.length === 0) {
+	if (argv.length === 0 || argv.includes("--help") || argv.includes("-h")) {
 		console.log(USAGE);
 		return 0;
 	}
+	const [command, ...rest] = argv as [string, ...string[]];
+	if (command === "init") {
+		return cmdInit(rest);
+	}
+	if (command === "validate") {
+		if (rest.length > 0) {
+			console.error(`validate takes no arguments.\n`);
+			console.error(USAGE);
+			return 2;
+		}
+		return cmdValidate();
+	}
+	if (command === "run") {
+		return cmdRun(rest);
+	}
+	if (command === "report") {
+		return cmdReport(rest);
+	}
 	console.error(`Unknown arguments: ${argv.join(" ")}\n`);
 	console.error(USAGE);
 	return 2;

src/index.ts

@@ -0,0 +1,54 @@
+// Public programmatic API for `bun:test` users (SPEC 5.3): the primitives
+// the runner itself uses, so authors who outgrow YAML can assert directly.
+
+export type { Evidence } from "./assert/expect.ts";
+export { evaluate } from "./assert/expect.ts";
+export { check } from "./assert/matchers.ts";
+export type { ResolvedOpencode } from "./opencode/binary.ts";
+export { resolveOpencode } from "./opencode/binary.ts";
+export type {
+	InnerToolCall,
+	RunEvent,
+	RunEventError,
+	RunEventPart,
+	ToolCallState,
+	ToolUse,
+	Usage,
+} from "./opencode/events.ts";
+export {
+	errors,
+	parseEvents,
+	textOf,
+	toolUses,
+	usage,
+} from "./opencode/events.ts";
+export type { Entries } from "./plugin/entry.ts";
+export { ensureEntriesOrThrow, resolveEntries } from "./plugin/entry.ts";
+export { writeShim } from "./plugin/shim.ts";
+export type {
+	InventoryPlugin,
+	ProbeInventory,
+	ProbeRecord,
+	ToolCallPair,
+} from "./probe/read.ts";
+export { ProbeLog, probePluginDirs, readProbe } from "./probe/read.ts";
+export { appendStepSummary, renderMarkdown } from "./report/markdown.ts";
+export type { CaseResult, ResultsFile } from "./report/results.ts";
+export type { RunOptions, RunOutcome, TierName } from "./run.ts";
+export { runCi } from "./run.ts";
+export type {
+	RunOutput,
+	SandboxOptions,
+	SandboxPlugin,
+	SandboxRunOptions,
+} from "./sandbox/sandbox.ts";
+export { Sandbox } from "./sandbox/sandbox.ts";
+export type {
+	Recorded,
+	RequestKind,
+	Step,
+	Stub,
+	StubOptions,
+	ToolCall,
+} from "./stub/stub.ts";
+export { EXHAUSTED, JUDGE_SENTINEL, startStub } from "./stub/stub.ts";

src/opencode/logs.ts

@@ -90,7 +90,7 @@ export function pluginLogProblems(
 ): LogLine[] {
 	const problems: LogLine[] = [];
 	for (const line of lines) {
-		const level = line["level"];
+		const level = line.level;
 		if (level === "ERROR" || level === "FATAL") {
 			const namesId = filter.id !== undefined && line._raw.includes(filter.id);
 			const namesPath =
@@ -100,7 +100,7 @@ export function pluginLogProblems(
 				continue;
 			}
 		}
-		const message = line["message"] ?? line["msg"];
+		const message = line.message ?? line.msg;
 		if (
 			message === "failed to load plugin" &&
 			filter.id !== undefined &&

src/report/markdown.ts

@@ -0,0 +1,202 @@
+// Markdown report rendering (SPEC 7.1, layout ported from e2e-report.ts):
+// a header, one table per opencode version, a <details> block per failure,
+// and live cost/token totals. Pure function + a step-summary helper; file
+// writing lives in run.ts / cli.ts.
+import { appendFile } from "../util/fs.ts";
+import type { CaseResult, ResultsFile } from "./results.ts";
+
+export interface RenderOptions {
+	/** Versions whose failures are allowed (non-required matrix legs). */
+	allowedVersions?: string[];
+	/** Override the git SHA (defaults to GITHUB_SHA or `git rev-parse`). */
+	sha?: string;
+}
+
+/** GITHUB_SHA in CI, else `git rev-parse --short HEAD`, else "unknown". */
+export function gitSha(): string {
+	const fromEnv = process.env.GITHUB_SHA;
+	if (fromEnv !== undefined && fromEnv.length > 0) {
+		return fromEnv;
+	}
+	try {
+		const proc = Bun.spawnSync(["git", "rev-parse", "--short", "HEAD"]);
+		const out = new TextDecoder().decode(proc.stdout).trim();
+		return out.length > 0 ? out : "unknown";
+	} catch {
+		return "unknown";
+	}
+}
+
+/**
+ * Append the report to $GITHUB_STEP_SUMMARY when set. Best effort: never
+ * throws, so a missing summary file cannot fail the run.
+ */
+export function appendStepSummary(markdown: string): void {
+	const file = process.env.GITHUB_STEP_SUMMARY;
+	if (file === undefined || file.length === 0) {
+		return;
+	}
+	appendFile(file, `${markdown}\n`);
+}
+
+function isSkipped(result: CaseResult): boolean {
+	return result.skipped !== undefined;
+}
+
+function isFailed(result: CaseResult): boolean {
+	return !result.pass && !isSkipped(result);
+}
+
+function resultCell(result: CaseResult, allowed: boolean): string {
+	if (isSkipped(result)) {
+		return result.skipped !== undefined && result.skipped.length > 0
+			? `⏭️ skipped (${result.skipped})`
+			: "⏭️ skipped";
+	}
+	if (result.pass) {
+		return "✅ pass";
+	}
+	return allowed ? "⚠️ allowed failure" : "❌ fail";
+}
+
+function formatCost(costUSD: number | undefined): string {
+	if (costUSD === undefined) {
+		return "-";
+	}
+	return `$${costUSD.toFixed(4)}`;
+}
+
+function formatTime(durationMs: number): string {
+	if (durationMs >= 1000) {
+		return `${(durationMs / 1000).toFixed(1)}s`;
+	}
+	return `${durationMs}ms`;
+}
+
+function formatCount(value: number | undefined): string {
+	return value === undefined ? "-" : String(value);
+}
+
+/**
+ * Derive allowed-failure versions from a results file: unique opencode
+ * values in order of first appearance, minus the required (first) version.
+ * Mirrors run.ts (`versions.slice(1)`) so `report` re-renders identically.
+ */
+export function deriveAllowedVersions(results: ResultsFile): string[] {
+	const versions: string[] = [];
+	for (const result of results.results) {
+		if (!versions.includes(result.opencode)) {
+			versions.push(result.opencode);
+		}
+	}
+	return versions.slice(1);
+}
+
+/** Render one per-version table plus failure details for that version. */
+function renderVersionSection(
+	version: string,
+	cases: CaseResult[],
+	required: boolean,
+	allowedVersions: string[],
+): string {
+	const lines: string[] = [];
+	lines.push(
+		`## opencode ${version}${required ? " (required)" : " (allowed to fail)"}`,
+	);
+	lines.push("");
+	lines.push("| Tier | Case | Result | Tools | Steps | Cost | Time |");
+	lines.push("|---|---|---|---|---|---|---|");
+	for (const result of cases) {
+		const allowed = allowedVersions.includes(result.opencode);
+		const tools =
+			result.tools !== undefined && result.tools.length > 0
+				? result.tools.join(", ")
+				: "-";
+		lines.push(
+			`| ${result.tier} | ${result.id} | ${resultCell(result, allowed)} | ${tools} | ${formatCount(result.steps)} | ${formatCost(result.costUSD)} | ${formatTime(result.durationMs)} |`,
+		);
+	}
+	lines.push("");
+	for (const result of cases) {
+		if (!isFailed(result)) {
+			continue;
+		}
+		lines.push(
+			`<details><summary><code>${result.opencode} / ${result.id}</code> — ${result.tier} failed</summary>`,
+		);
+		lines.push("");
+		for (const failure of result.failures ?? ["(no failure details)"]) {
+			lines.push(`- ${failure}`);
+		}
+		lines.push("");
+		if (result.kept !== undefined) {
+			lines.push(`Evidence: \`${result.kept}\``);
+			lines.push("");
+		}
+		lines.push("</details>");
+		lines.push("");
+	}
+	return lines.join("\n");
+}
+
+/** Render the full report.md for a results file. */
+export function renderMarkdown(
+	results: ResultsFile,
+	opts?: RenderOptions,
+): string {
+	const allowedVersions = opts?.allowedVersions ?? [];
+	const sha = opts?.sha ?? gitSha();
+	const versions: string[] = [];
+	for (const result of results.results) {
+		if (!versions.includes(result.opencode)) {
+			versions.push(result.opencode);
+		}
+	}
+	const required = versions.length > 0 ? versions[0] : undefined;
+
+	const passed = results.results.filter((r) => r.pass && !isSkipped(r)).length;
+	const failed = results.results.filter((r) => isFailed(r)).length;
+	const skipped = results.results.filter((r) => isSkipped(r)).length;
+
+	const lines: string[] = [];
+	lines.push("# opencode-plugin-ci report");
+	lines.push("");
+	lines.push(`- plugin: ${results.plugin}`);
+	lines.push(`- tool: ${results.tool}`);
+	lines.push(`- opencode: ${versions.length > 0 ? versions.join(", ") : "-"}`);
+	lines.push(`- sha: ${sha}`);
+	lines.push(`- date: ${results.at}`);
+	lines.push("");
+	lines.push(
+		`**Totals:** ${passed} passed, ${failed} failed, ${skipped} skipped ` +
+			`(${results.results.length} cases across ${versions.length} opencode version${versions.length === 1 ? "" : "s"})`,
+	);
+	lines.push("");
+
+	for (const version of versions) {
+		const cases = results.results.filter((r) => r.opencode === version);
+		lines.push(
+			renderVersionSection(
+				version,
+				cases,
+				version === required,
+				allowedVersions,
+			),
+		);
+	}
+
+	const live = results.results.filter(
+		(r) => r.tier === "live" && !isSkipped(r),
+	);
+	if (live.length > 0) {
+		const cost = live.reduce((sum, r) => sum + (r.costUSD ?? 0), 0);
+		const input = live.reduce((sum, r) => sum + (r.tokens?.input ?? 0), 0);
+		const output = live.reduce((sum, r) => sum + (r.tokens?.output ?? 0), 0);
+		lines.push(
+			`Live totals: $${cost.toFixed(4)} across ${live.length} case${live.length === 1 ? "" : "s"}; ${input} input / ${output} output tokens.`,
+		);
+		lines.push("");
+	}
+
+	return `${lines.join("\n").trimEnd()}\n`;
+}

src/report/results.ts

@@ -0,0 +1,30 @@
+// Results file shape: the IUMBTEMS EvalRow shape (`id`, `pass`, `tools`,
+// `failures`, `steps`, `costUSD`, `textExcerpt`, `kept`, ...) extended with
+// `tier` and `opencode` (SPEC 7.1). Written to `<out>/results.json` by run.ts
+// and re-rendered by the `report` command.
+export interface CaseResult {
+	id: string; // scenario id, or "load" / "bun-test"
+	tier: "load" | "scripted" | "live";
+	opencode: string; // resolved version label, e.g. "2.0.26"
+	pass: boolean;
+	skipped?: string; // reason, when the case did not run
+	description?: string;
+	failures?: string[];
+	tools?: string[]; // tool names called (probe, post)
+	steps?: number; // count of step_finish
+	costUSD?: number;
+	tokens?: { input: number; output: number };
+	durationMs: number;
+	textExcerpt?: string; // first 300 chars of text
+	attempts?: { pass: boolean; failures: string[] }[]; // live only
+	judge?: { pass: boolean; reason: string };
+	kept?: string; // evidence dir
+}
+
+export interface ResultsFile {
+	version: 1;
+	at: string; // ISO timestamp of the run
+	tool: string; // opencode-plugin-ci version
+	plugin: string; // plugin package name (or path fallback)
+	results: CaseResult[];
+}

src/run.ts

@@ -0,0 +1,400 @@
+// Orchestrator for `opencode-plugin-ci run` (M10 order):
+// 1. Load + validate config/scenarios.
+// 2. Resolve entries (exit 2 when neither server nor TUI exists).
+// 3. Run plugin.install / plugin.build once.
+// 4. For each opencode version: resolve binary, tier 1 (or static tier 1
+//    for TUI-only plugins), then tier 2; tier 3 live scenarios are marked
+//    skipped until M11 implements the live tier.
+// 5. Run `.opencode-ci/**/*.test.ts` once via `bun test` (programmatic API
+//    escape hatch) and fold it in as a `bun-test` scripted case.
+// 6. Write results.json + report.md (+ $GITHUB_STEP_SUMMARY).
+// 7. Gate (M12; the --gate flag is accepted but warns for now).
+// 8. Exit code: 0 when all required cases pass or skip, 1 on a required
+//    failure. Results on non-first versions are allowed failures.
+import { readdirSync, readFileSync, statSync, writeFileSync } from "node:fs";
+import { join, relative, resolve } from "node:path";
+import { loadAll } from "./config/load.ts";
+import type { Config, Scenario } from "./config/schema.ts";
+import { resolveOpencode } from "./opencode/binary.ts";
+import { type Entries, ensureEntriesOrThrow } from "./plugin/entry.ts";
+import { runStep } from "./plugin/install.ts";
+import { appendStepSummary, renderMarkdown } from "./report/markdown.ts";
+import type { CaseResult, ResultsFile } from "./report/results.ts";
+import { runLoad } from "./tiers/load.ts";
+import { concurrencyLimit, pool, runScenario } from "./tiers/scripted.ts";
+import { runUiStatic } from "./tiers/ui-static.ts";
+import { createLogger, type Logger } from "./util/log.ts";
+
+export type TierName = "load" | "scripted" | "live";
+
+export interface RunOptions {
+	cwd?: string;
+	tiers?: TierName[];
+	scenarios?: string[];
+	opencodes?: string[];
+	out?: string;
+	keep?: boolean;
+	gate?: boolean;
+	verbose?: boolean;
+	quiet?: boolean;
+}
+
+export interface RunOutcome {
+	exitCode: number;
+	resultsFile: ResultsFile;
+	resultsPath: string;
+	reportPath: string;
+}
+
+const VALID_TIERS: TierName[] = ["load", "scripted", "live"];
+
+/** True for TUI-only plugins: forced via config, or no server entry. */
+export function isTuiOnly(config: Config, entries: Entries): boolean {
+	if (config.plugin.ui === true) {
+		return true;
+	}
+	if (config.plugin.ui === false) {
+		return false;
+	}
+	return entries.server === undefined;
+}
+
+function toolVersion(): string {
+	try {
+		const text = readFileSync(
+			new URL("../package.json", import.meta.url),
+			"utf8",
+		);
+		const pkg = JSON.parse(text) as { version?: unknown };
+		return typeof pkg.version === "string" ? pkg.version : "0.0.0";
+	} catch {
+		return "0.0.0";
+	}
+}
+
+function pluginDisplayName(pluginPath: string): string {
+	try {
+		const pkg = JSON.parse(
+			readFileSync(join(pluginPath, "package.json"), "utf8"),
+		) as { name?: unknown };
+		if (typeof pkg.name === "string" && pkg.name.length > 0) {
+			return pkg.name;
+		}
+	} catch {
+		// Fall through to the path fallback.
+	}
+	return pluginPath;
+}
+
+// Recursively collect ".opencode-ci" test files ("*.test.ts") under cwd.
+function findProgrammaticTests(cwd: string): string[] {
+	const root = join(cwd, ".opencode-ci");
+	const out: string[] = [];
+	const walk = (dir: string): void => {
+		let names: string[];
+		try {
+			names = readdirSync(dir);
+		} catch {
+			return;
+		}
+		for (const name of names.sort()) {
+			const full = join(dir, name);
+			let isDir = false;
+			try {
+				isDir = statSync(full).isDirectory();
+			} catch {
+				continue;
+			}
+			if (isDir) {
+				walk(full);
+			} else if (name.endsWith(".test.ts")) {
+				out.push(full);
+			}
+		}
+	};
+	walk(root);
+	return out;
+}
+
+/** M10 placeholder for the live tier: every live scenario is skipped. */
+function skippedLive(
+	scenario: Scenario,
+	version: string,
+	reason: string,
+): CaseResult {
+	return {
+		id: scenario.id,
+		tier: "live",
+		opencode: version,
+		pass: true,
+		skipped: reason,
+		description: scenario.description,
+		durationMs: 0,
+	};
+}
+
+function skippedForTier1(scenario: Scenario, version: string): CaseResult {
+	return {
+		id: scenario.id,
+		tier: scenario.mode === "live" ? "live" : "scripted",
+		opencode: version,
+		pass: true,
+		skipped: "tier 1 failed",
+		description: scenario.description,
+		durationMs: 0,
+	};
+}
+
+function skippedTuiOnly(scenario: Scenario, version: string): CaseResult {
+	return {
+		id: scenario.id,
+		tier: scenario.mode === "live" ? "live" : "scripted",
+		opencode: version,
+		pass: true,
+		skipped: "TUI-only plugin",
+		description: scenario.description,
+		durationMs: 0,
+	};
+}
+
+export async function runCi(opts?: RunOptions): Promise<RunOutcome> {
+	const cwd = resolve(opts?.cwd ?? process.cwd());
+	const logger: Logger = createLogger({
+		quiet: opts?.quiet,
+		verbose: opts?.verbose,
+	});
+	const wantedTiers = opts?.tiers ?? VALID_TIERS;
+	const wantedScenarios = opts?.scenarios;
+
+	// 1. Load + validate (fail closed: ConfigError -> exit 2 upstream).
+	const { config, scenarios, warnings } = loadAll(cwd);
+	for (const warning of warnings) {
+		logger.warn(warning);
+	}
+
+	const pluginPath = resolve(cwd, config.plugin.path);
+	// 2. Resolve entries (throws ConfigError when nothing resolves).
+	const entries = ensureEntriesOrThrow(pluginPath, config.plugin.entry);
+	const tuiOnly = isTuiOnly(config, entries);
+	if (tuiOnly) {
+		logger.info("TUI-only plugin: tier 1 is static; tiers 2/3 are skipped.");
+	}
+
+	// 3. Install + build once, before any tier.
+	if (config.plugin.install !== undefined) {
+		logger.info(`Running install step: ${config.plugin.install}`);
+		await runStep("install", config.plugin.install, pluginPath);
+	}
+	if (config.plugin.build !== undefined) {
+		logger.info(`Running build step: ${config.plugin.build}`);
+		await runStep("build", config.plugin.build, pluginPath);
+	}
+
+	const versions =
+		opts?.opencodes !== undefined && opts.opencodes.length > 0
+			? opts.opencodes
+			: config.opencode.versions;
+	const required = versions[0] as string;
+	const allowedVersions = versions.slice(1);
+	const outDir = resolve(opts?.out ?? join(cwd, ".opencode-ci/out"));
+
+	if (opts?.keep === true) {
+		// Sandboxes honor this env var in dispose(); threading a flag
+		// through every tier API would churn the M8/M9 signatures.
+		process.env.OPENCODE_PLUGIN_CI_KEEP = "1";
+	}
+
+	const logLevel = opts?.verbose === true ? "all" : config.logLevel;
+	const matchScenario = (scenario: Scenario): boolean =>
+		wantedScenarios === undefined ||
+		wantedScenarios.length === 0 ||
+		wantedScenarios.includes(scenario.id);
+
+	const results: CaseResult[] = [];
+
+	// 4. One pass per opencode version.
+	for (const version of versions) {
+		logger.info(`Resolving opencode ${version}...`);
+		const { bin } = await resolveOpencode(version);
+
+		const runTier1 = wantedTiers.includes("load");
+		const runScripted = wantedTiers.includes("scripted");
+		const runLive = wantedTiers.includes("live");
+
+		let tier1Pass = true;
+		if (runTier1) {
+			logger.info(`[${version}] tier 1 (load)...`);
+			const tier1 = tuiOnly
+				? await runUiStatic({ pluginPath, outDir, version })
+				: await runLoad({
+						pluginPath,
+						pluginOptions: config.plugin.options,
+						opencodeBin: bin,
+						outDir,
+						version,
+						expected: {
+							tools: config.expect.tools,
+							commands: config.expect.commands,
+							agents: config.expect.agents,
+							skills: config.expect.skills,
+							mcp: config.expect.mcp,
+						},
+						logLevel,
+					});
+			results.push(tier1);
+			tier1Pass = tier1.pass;
+			logger.info(`[${version}] tier 1 ${tier1.pass ? "passed" : "failed"}.`);
+		}
+
+		const scripted = scenarios.filter(
+			(s) => s.mode === "scripted" && matchScenario(s),
+		);
+		const live = scenarios.filter((s) => s.mode === "live" && matchScenario(s));
+
+		if (!tier1Pass) {
+			// A failing tier 1 would fail every later tier for the same
+			// reason, so report them as skipped instead of noisy red.
+			for (const scenario of [...scripted, ...live]) {
+				if (
+					(scenario.mode === "scripted" && !runScripted) ||
+					(scenario.mode === "live" && !runLive)
+				) {
+					continue;
+				}
+				results.push(skippedForTier1(scenario, version));
+			}
+			continue;
+		}
+
+		if (tuiOnly) {
+			for (const scenario of [...scripted, ...live]) {
+				if (
+					(scenario.mode === "scripted" && !runScripted) ||
+					(scenario.mode === "live" && !runLive)
+				) {
+					continue;
+				}
+				results.push(skippedTuiOnly(scenario, version));
+			}
+			continue;
+		}
+
+		if (runScripted && entries.server !== undefined) {
+			logger.info(
+				`[${version}] tier 2: ${scripted.length} scripted scenario(s)...`,
+			);
+			const ran = await pool(scripted, concurrencyLimit(), (scenario) =>
+				runScenario({
+					scenario,
+					config,
+					pluginPath,
+					serverEntry: entries.server as string,
+					opencodeBin: bin,
+					outDir,
+					version,
+					logLevel,
+				}),
+			);
+			results.push(...ran);
+		}
+
+		if (runLive) {
+			// M11 implements the live tier; until then every live scenario
+			// is reported skipped (never failed) so forks stay green.
+			for (const scenario of live) {
+				results.push(
+					skippedLive(scenario, version, "live tier not implemented (M11)"),
+				);
+			}
+		}
+	}
+
+	// 5. Programmatic escape hatch: run "bun test .opencode-ci" once, with
+	// OPENCODE_BIN pointing at the required version's binary.
+	if (wantedTiers.includes("scripted")) {
+		const testFiles = findProgrammaticTests(cwd);
+		if (testFiles.length > 0) {
+			logger.info(`Running programmatic tests: ${testFiles.length} file(s)...`);
+			const { bin } = await resolveOpencode(required);
+			const started = Date.now();
+			const proc = Bun.spawn(["bun", "test", ".opencode-ci"], {
+				cwd,
+				env: { ...process.env, OPENCODE_BIN: bin },
+				stdout: "pipe",
+				stderr: "pipe",
+				stdin: "ignore",
+			});
+			const [stdout, stderr] = await Promise.all([
+				new Response(proc.stdout as ReadableStream).text(),
+				new Response(proc.stderr as ReadableStream).text(),
+			]);
+			const code = await proc.exited;
+			const output = `${stdout}\n${stderr}`.trim();
+			const tail = output.length > 0 ? output.split("\n").slice(-40) : [];
+			results.push({
+				id: "bun-test",
+				tier: "scripted",
+				opencode: required,
+				pass: code === 0,
+				failures:
+					code === 0
+						? undefined
+						: tail.length > 0
+							? tail
+							: [`bun test .opencode-ci exited with code ${code}.`],
+				steps: testFiles.length,
+				durationMs: Date.now() - started,
+				textExcerpt: output.slice(0, 300),
+				kept: outDir,
+			});
+			logger.info(
+				`Programmatic tests ${code === 0 ? "passed" : `failed (exit ${code})`}.`,
+			);
+		}
+	}
+
+	// 6. Write results.json + report.md (+ step summary).
+	const resultsFile: ResultsFile = {
+		version: 1,
+		at: new Date().toISOString(),
+		tool: toolVersion(),
+		plugin: pluginDisplayName(pluginPath),
+		results,
+	};
+	const resultsPath = join(outDir, "results.json");
+	const reportPath = join(outDir, "report.md");
+	{
+		const { ensureDir } = await import("./util/fs.ts");
+		ensureDir(outDir);
+	}
+	writeFileSync(
+		resultsPath,
+		`${JSON.stringify(resultsFile, null, 2)}\n`,
+		"utf8",
+	);
+	const markdown = renderMarkdown(resultsFile, { allowedVersions });
+	writeFileSync(reportPath, markdown, "utf8");
+	appendStepSummary(markdown);
+	logger.info(
+		`Wrote ${relative(cwd, resultsPath)} and ${relative(cwd, reportPath)}.`,
+	);
+
+	// 7. Gate (M12): accepted but not implemented yet.
+	if (opts?.gate === true) {
+		logger.warn(
+			"The --gate flag is accepted but the flake gate lands in M12; continuing without gating.",
+		);
+	}
+
+	// 8. Exit code: required-version failures fail; allowed versions and
+	// skips never do.
+	const requiredFailures = results.filter(
+		(r) => r.opencode === required && !r.pass && r.skipped === undefined,
+	);
+	return {
+		exitCode: requiredFailures.length > 0 ? 1 : 0,
+		resultsFile,
+		resultsPath,
+		reportPath,
+	};
+}

src/sandbox/sandbox.ts

@@ -121,10 +121,10 @@ export class Sandbox {
 			plugins: opts.plugins,
 		};
 		if (opts.stubUrl !== undefined) {
-			config["providers"] = stubProvider(opts.stubUrl);
+			config.providers = stubProvider(opts.stubUrl);
 		}
 		if (opts.agents !== undefined) {
-			config["agents"] = opts.agents;
+			config.agents = opts.agents;
 		}
 		writeFileSync(
 			join(configHome, "opencode", "opencode.jsonc"),

src/tiers/load.ts

@@ -2,8 +2,6 @@
 // assert it registers (inventory), stays silent in the logs, and provides
 // every expected tool/command/agent/skill/mcp name.
 //
-// TODO(M10): move CaseResult to src/report/results.ts and import it here.
-// Until then this local definition is the canonical tier-1 result shape.
 import { copyFileSync, mkdirSync, writeFileSync } from "node:fs";
 import { join, resolve } from "node:path";
 import { ConfigError } from "../config/load.ts";
@@ -12,27 +10,12 @@ import { parseLogs, pluginLogProblems } from "../opencode/logs.ts";
 import { resolveEntries } from "../plugin/entry.ts";
 import { writeShim } from "../plugin/shim.ts";
 import { probePluginDirs, readProbe } from "../probe/read.ts";
+import type { CaseResult } from "../report/results.ts";
 import { type RunOutput, Sandbox } from "../sandbox/sandbox.ts";
 import { startStub } from "../stub/stub.ts";
 
-export interface CaseResult {
-	id: string;
-	tier: "load" | "scripted" | "live";
-	opencode: string;
-	pass: boolean;
-	skipped?: string;
-	description?: string;
-	failures?: string[];
-	tools?: string[];
-	steps?: number;
-	costUSD?: number;
-	tokens?: { input: number; output: number };
-	durationMs: number;
-	textExcerpt?: string;
-	attempts?: { pass: boolean; failures: string[] }[];
-	judge?: { pass: boolean; reason: string };
-	kept?: string;
-}
+// Re-exported so existing importers (`./load.ts`) keep working.
+export type { CaseResult } from "../report/results.ts";
 
 export interface LoadTierExpected {
 	tools?: string[];

src/tiers/scripted.ts

@@ -11,9 +11,9 @@ import type { Config, Scenario } from "../config/schema.ts";
 import { textOf, usage } from "../opencode/events.ts";
 import { writeShim } from "../plugin/shim.ts";
 import { probePluginDirs, readProbe } from "../probe/read.ts";
+import type { CaseResult } from "../report/results.ts";
 import { type RunOutput, Sandbox } from "../sandbox/sandbox.ts";
 import { startStub } from "../stub/stub.ts";
-import type { CaseResult } from "./load.ts";
 
 export interface RunScenarioOptions {
 	scenario: Scenario;

src/tiers/ui-static.ts

@@ -7,7 +7,7 @@ import { join, resolve } from "node:path";
 import { pathToFileURL } from "node:url";
 import { ConfigError } from "../config/load.ts";
 import { resolveEntries } from "../plugin/entry.ts";
-import type { CaseResult } from "./load.ts";
+import type { CaseResult } from "../report/results.ts";
 
 export interface RunUiStaticOptions {
 	pluginPath: string;

src/util/fs.ts

@@ -0,0 +1,62 @@
+// Small filesystem helpers shared by the runner and the report renderer.
+import {
+	appendFileSync,
+	mkdirSync,
+	readFileSync,
+	statSync,
+	writeFileSync,
+} from "node:fs";
+
+/** mkdir -p. */
+export function ensureDir(dir: string): void {
+	mkdirSync(dir, { recursive: true });
+}
+
+/** True when path exists and is a regular file. Never throws. */
+export function isFile(path: string): boolean {
+	try {
+		return statSync(path).isFile();
+	} catch {
+		return false;
+	}
+}
+
+/** Write records as JSON lines (one object per line, trailing newline). */
+export function writeJsonl(file: string, records: unknown[]): void {
+	const text = records.map((r) => JSON.stringify(r)).join("\n");
+	writeFileSync(file, text.length > 0 ? `${text}\n` : "", "utf8");
+}
+
+/**
+ * Read JSON lines defensively: missing files yield [], blank lines are
+ * skipped, and bad lines are skipped (a torn final line must not be fatal).
+ */
+export function readJsonl(file: string): unknown[] {
+	let text: string;
+	try {
+		text = readFileSync(file, "utf8");
+	} catch {
+		return [];
+	}
+	const out: unknown[] = [];
+	for (const line of text.split("\n")) {
+		if (line.trim() === "") {
+			continue;
+		}
+		try {
+			out.push(JSON.parse(line) as unknown);
+		} catch {
+			// Skip torn lines.
+		}
+	}
+	return out;
+}
+
+/** Append text to a file (used for $GITHUB_STEP_SUMMARY). Never throws. */
+export function appendFile(file: string, text: string): void {
+	try {
+		appendFileSync(file, text, "utf8");
+	} catch {
+		// Best effort: a failed step summary must not fail the run.
+	}
+}

src/util/log.ts

@@ -0,0 +1,39 @@
+// Tiny logger that respects --quiet/--verbose. info is suppressed when
+// quiet; debug only shows when verbose (and not quiet); warn and error
+// always show.
+export interface LoggerOptions {
+	quiet?: boolean;
+	verbose?: boolean;
+}
+
+export interface Logger {
+	info(message: string): void;
+	warn(message: string): void;
+	error(message: string): void;
+	debug(message: string): void;
+}
+
+export function createLogger(opts?: LoggerOptions): Logger {
+	const quiet = opts?.quiet === true;
+	const verbose = opts?.verbose === true && !quiet;
+	return {
+		info(message: string): void {
+			if (!quiet) {
+				console.log(message);
+			}
+		},
+		warn(message: string): void {
+			if (!quiet) {
+				console.warn(message);
+			}
+		},
+		error(message: string): void {
+			console.error(message);
+		},
+		debug(message: string): void {
+			if (verbose) {
+				console.log(message);
+			}
+		},
+	};
+}

src/util/rows.ts

@@ -0,0 +1,16 @@
+// rows(): normalize `[]` or `{ data: [] }` to an array. Probe inventory
+// lists and opencode client lists come back in either shape depending on
+// the domain, so every consumer normalizes through here instead of
+// assuming one form.
+export function rows<T>(json: unknown): T[] {
+	if (Array.isArray(json)) {
+		return json as T[];
+	}
+	if (json !== null && typeof json === "object") {
+		const data = (json as { data?: unknown }).data;
+		if (Array.isArray(data)) {
+			return data as T[];
+		}
+	}
+	return [];
+}

test/e2e/cli.test.ts

@@ -0,0 +1,74 @@
+// CLI acceptance (e2e, needs OPENCODE_BIN or the dev install at
+// review/oc): `run` in tool-basic passes with --scenario filtering and
+// writes report.md, `run` in throws exits 1, `validate` in no-entry exits 2.
+import { describe, expect, test } from "bun:test";
+import { existsSync } from "node:fs";
+import { join, resolve } from "node:path";
+
+const CLI = resolve(import.meta.dir, "../../bin/opencode-plugin-ci.ts");
+const FIXTURES = resolve(import.meta.dir, "../fixtures/plugins");
+
+function opencodeBin(): string {
+	return (
+		process.env.OPENCODE_BIN ??
+		resolve(import.meta.dir, "../../review/oc/node_modules/.bin/opencode")
+	);
+}
+
+interface CliResult {
+	code: number;
+	stdout: string;
+	stderr: string;
+}
+
+async function runCli(args: string[], cwd: string): Promise<CliResult> {
+	const proc = Bun.spawn(["bun", CLI, ...args], {
+		cwd,
+		env: { ...process.env, OPENCODE_BIN: opencodeBin() },
+		stdout: "pipe",
+		stderr: "pipe",
+		stdin: "ignore",
+	});
+	const [stdout, stderr] = await Promise.all([
+		new Response(proc.stdout as ReadableStream).text(),
+		new Response(proc.stderr as ReadableStream).text(),
+	]);
+	return { code: await proc.exited, stdout, stderr };
+}
+
+describe("cli e2e", () => {
+	test("run in tool-basic passes and writes report.md", async () => {
+		const dir = join(FIXTURES, "tool-basic");
+		const { code, stdout, stderr } = await runCli(
+			["run", "--scenario", "tool-basic-pass"],
+			dir,
+		);
+		expect(`${stdout}\n${stderr}`).toContain("report.md");
+		expect(code).toBe(0);
+		expect(existsSync(join(dir, ".opencode-ci/out/results.json"))).toBe(true);
+		expect(existsSync(join(dir, ".opencode-ci/out/report.md"))).toBe(true);
+	}, 600_000);
+
+	test("run in throws exits 1", async () => {
+		const { code } = await runCli(["run"], join(FIXTURES, "throws"));
+		expect(code).toBe(1);
+	}, 600_000);
+
+	test("validate in no-entry exits 2", async () => {
+		const { code, stdout, stderr } = await runCli(
+			["validate"],
+			join(FIXTURES, "no-entry"),
+		);
+		expect(code).toBe(2);
+		expect(`${stdout}\n${stderr}`).toMatch(/no server entry/i);
+	}, 120_000);
+
+	test("validate in tool-basic exits 0", async () => {
+		const { code, stdout } = await runCli(
+			["validate"],
+			join(FIXTURES, "tool-basic"),
+		);
+		expect(code).toBe(0);
+		expect(stdout).toContain("scenarios: 2");
+	}, 120_000);
+});

test/e2e/sandbox.test.ts

@@ -35,7 +35,7 @@ function respondSSE(
 		id,
 		object: "chat.completion.chunk",
 		created: Math.floor(Date.now() / 1000),
-		model: (body["model"] as string) ?? "scripted",
+		model: (body.model as string) ?? "scripted",
 	};
 	res.writeHead(200, {
 		"content-type": "text/event-stream",
@@ -74,7 +74,7 @@ function respondJSON(
 			id: `chatcmpl-stub-${Date.now()}`,
 			object: "chat.completion",
 			created: Math.floor(Date.now() / 1000),
-			model: body["model"] ?? "scripted",
+			model: body.model ?? "scripted",
 			choices: [
 				{
 					index: 0,
@@ -115,9 +115,9 @@ async function startInlineStub(): Promise<InlineStub> {
 		}
 		const body = JSON.parse(await readBody(req)) as Record<string, unknown>;
 		const hasTools =
-			Array.isArray(body["tools"]) && (body["tools"] as unknown[]).length > 0;
+			Array.isArray(body.tools) && (body.tools as unknown[]).length > 0;
 		const text = hasTools ? "ok" : "Stub title";
-		if (body["stream"] === true) {
+		if (body.stream === true) {
 			respondSSE(res, body, text);
 		} else {
 			respondJSON(res, body, text);

test/unit/__snapshots__/report.test.ts.snap

@@ -0,0 +1,48 @@
+// Bun Snapshot v1, https://bun.sh/docs/test/snapshots
+
+exports[`renderMarkdown matches snapshot 1`] = `
+"# opencode-plugin-ci report
+
+- plugin: fixture-tool-basic
+- tool: 0.1.0
+- opencode: 2.0.26, latest
+- sha: abc1234
+- date: 2026-10-17T00:00:00.000Z
+
+**Totals:** 3 passed, 2 failed, 1 skipped (6 cases across 2 opencode versions)
+
+## opencode 2.0.26 (required)
+
+| Tier | Case | Result | Tools | Steps | Cost | Time |
+|---|---|---|---|---|---|---|
+| load | load | ✅ pass | echo_tool | 1 | $0.0000 | 1.5s |
+| scripted | tool-basic-pass | ✅ pass | echo_tool, shell | 2 | - | 23.4s |
+| scripted | tool-basic-fail | ❌ fail | echo_tool | 2 | - | 22.1s |
+| live | future-live | ⏭️ skipped (live tier not implemented (M11)) | - | - | - | 0ms |
+| live | live-real | ✅ pass | - | 3 | $0.0123 | 60.0s |
+
+<details><summary><code>2.0.26 / tool-basic-fail</code> — scripted failed</summary>
+
+- tools[0].output: expected to contain "echo:bye", got "echo:hi"
+
+Evidence: \`.opencode-ci/out/2.0.26/tool-basic-fail\`
+
+</details>
+
+## opencode latest (allowed to fail)
+
+| Tier | Case | Result | Tools | Steps | Cost | Time |
+|---|---|---|---|---|---|---|
+| scripted | tool-basic-fail | ⚠️ allowed failure | - | - | - | 5.0s |
+
+<details><summary><code>latest / tool-basic-fail</code> — scripted failed</summary>
+
+- opencode exited with code 1.
+
+Evidence: \`.opencode-ci/out/latest/tool-basic-fail\`
+
+</details>
+
+Live totals: $0.0123 across 1 case; 1000 input / 200 output tokens.
+"
+`;

test/unit/logs.test.ts

@@ -30,25 +30,23 @@ describe("parseLogLine", () => {
 
 	test("handles msg= instead of message=", () => {
 		const parsed = parseLogLine(LOADING);
-		expect(parsed?.["msg"]).toBe("loading plugin");
-		expect(parsed?.["id"]).toBe("/tmp/claude-0/spike/plugins/a");
-		expect(parsed?.["entrypoint"]).toBe(
+		expect(parsed?.msg).toBe("loading plugin");
+		expect(parsed?.id).toBe("/tmp/claude-0/spike/plugins/a");
+		expect(parsed?.entrypoint).toBe(
 			"file:///tmp/claude-0/spike/plugins/a/server.ts",
 		);
-		expect(parsed?.["level"]).toBe("INFO");
+		expect(parsed?.level).toBe("INFO");
 	});
 
 	test("parses a path target with slashes and dots", () => {
 		const parsed = parseLogLine(NOT_A_DIR);
-		expect(parsed?.["message"]).toBe(
-			"configured plugin path must be a directory",
-		);
-		expect(parsed?.["target"]).toBe("/tmp/claude-0/spike/plugins/a.ts");
+		expect(parsed?.message).toBe("configured plugin path must be a directory");
+		expect(parsed?.target).toBe("/tmp/claude-0/spike/plugins/a.ts");
 	});
 
 	test("unescapes quoted quotes", () => {
 		const parsed = parseLogLine('level=INFO message="say \\"hi\\""');
-		expect(parsed?.["message"]).toBe('say "hi"');
+		expect(parsed?.message).toBe('say "hi"');
 	});
 
 	test("returns undefined for empty lines", () => {

test/unit/report.test.ts

@@ -0,0 +1,158 @@
+// Unit snapshot-check of the Markdown report for a fixed ResultsFile:
+// one required pass, one required failure, one allowed failure, one skip,
+// and a live row so cost/token totals render.
+import { describe, expect, test } from "bun:test";
+import {
+	deriveAllowedVersions,
+	renderMarkdown,
+} from "../../src/report/markdown.ts";
+import type { ResultsFile } from "../../src/report/results.ts";
+
+const FIXED: ResultsFile = {
+	version: 1,
+	at: "2026-10-17T00:00:00.000Z",
+	tool: "0.1.0",
+	plugin: "fixture-tool-basic",
+	results: [
+		{
+			id: "load",
+			tier: "load",
+			opencode: "2.0.26",
+			pass: true,
+			tools: ["echo_tool"],
+			steps: 1,
+			costUSD: 0,
+			tokens: { input: 10, output: 5 },
+			durationMs: 1500,
+			textExcerpt: "ok",
+			kept: ".opencode-ci/out/2.0.26/load",
+		},
+		{
+			id: "tool-basic-pass",
+			tier: "scripted",
+			opencode: "2.0.26",
+			pass: true,
+			description: "echo_tool is called and returns echo:hi",
+			tools: ["echo_tool", "shell"],
+			steps: 2,
+			durationMs: 23400,
+			textExcerpt: "done",
+			kept: ".opencode-ci/out/2.0.26/tool-basic-pass",
+		},
+		{
+			id: "tool-basic-fail",
+			tier: "scripted",
+			opencode: "2.0.26",
+			pass: false,
+			failures: [
+				'tools[0].output: expected to contain "echo:bye", got "echo:hi"',
+			],
+			tools: ["echo_tool"],
+			steps: 2,
+			durationMs: 22100,
+			textExcerpt: "done",
+			kept: ".opencode-ci/out/2.0.26/tool-basic-fail",
+		},
+		{
+			id: "tool-basic-fail",
+			tier: "scripted",
+			opencode: "latest",
+			pass: false,
+			failures: ["opencode exited with code 1."],
+			durationMs: 5000,
+			textExcerpt: "",
+			kept: ".opencode-ci/out/latest/tool-basic-fail",
+		},
+		{
+			id: "future-live",
+			tier: "live",
+			opencode: "2.0.26",
+			pass: true,
+			skipped: "live tier not implemented (M11)",
+			description: "placeholder live scenario",
+			durationMs: 0,
+		},
+		{
+			id: "live-real",
+			tier: "live",
+			opencode: "2.0.26",
+			pass: true,
+			costUSD: 0.0123,
+			tokens: { input: 1000, output: 200 },
+			steps: 3,
+			durationMs: 60000,
+			textExcerpt: "the tool output shows echo:hi",
+			kept: ".opencode-ci/out/2.0.26/live-real",
+		},
+	],
+};
+
+describe("renderMarkdown", () => {
+	test("renders header, tables, failures and live totals", () => {
+		const md = renderMarkdown(FIXED, {
+			allowedVersions: ["latest"],
+			sha: "abc1234",
+		});
+		// Header.
+		expect(md).toContain("# opencode-plugin-ci report");
+		expect(md).toContain("plugin: fixture-tool-basic");
+		expect(md).toContain("tool: 0.1.0");
+		expect(md).toContain("opencode: 2.0.26, latest");
+		expect(md).toContain("sha: abc1234");
+		expect(md).toContain("**Totals:** 3 passed, 2 failed, 1 skipped");
+		// One table per opencode version.
+		expect(md).toContain("## opencode 2.0.26 (required)");
+		expect(md).toContain("## opencode latest (allowed to fail)");
+		expect(md).toContain(
+			"| Tier | Case | Result | Tools | Steps | Cost | Time |",
+		);
+		// Result cells.
+		expect(md).toContain("✅ pass");
+		expect(md).toContain("❌ fail");
+		expect(md).toContain("⚠️ allowed failure");
+		expect(md).toContain("⏭️ skipped (live tier not implemented (M11))");
+		// Failure details with evidence path.
+		expect(md).toContain("<details>");
+		expect(md).toContain('expected to contain "echo:bye"');
+		expect(md).toContain("Evidence: `.opencode-ci/out/2.0.26/tool-basic-fail`");
+		// Live totals line.
+		expect(md).toContain("Live totals: $0.0123");
+		expect(md).toContain("1000 input / 200 output tokens");
+	});
+
+	test("matches snapshot", () => {
+		const md = renderMarkdown(FIXED, {
+			allowedVersions: ["latest"],
+			sha: "abc1234",
+		});
+		expect(md).toMatchSnapshot();
+	});
+
+	test("deriveAllowedVersions mirrors run.ts versions.slice(1)", () => {
+		expect(deriveAllowedVersions(FIXED)).toEqual(["latest"]);
+		expect(
+			deriveAllowedVersions({
+				version: 1,
+				at: "2026-10-17T00:00:00.000Z",
+				tool: "0.1.0",
+				plugin: "p",
+				results: [],
+			}),
+		).toEqual([]);
+	});
+
+	test("report path derivation renders allowed failure for non-first version", () => {
+		// Same derivation cmdReport uses: unique opencode values slice(1).
+		const allowedVersions = deriveAllowedVersions(FIXED);
+		const md = renderMarkdown(FIXED, {
+			allowedVersions,
+			sha: "abc1234",
+		});
+		// Non-first version (latest) failure is allowed …
+		expect(md).toContain("## opencode latest (allowed to fail)");
+		expect(md).toContain("⚠️ allowed failure");
+		// … while the required (first) version failure stays hard.
+		expect(md).toContain("## opencode 2.0.26 (required)");
+		expect(md).toContain("❌ fail");
+	});
+});