Claude
795839066d
- Probe every OpenCode V2 capability SwarmOS uses at startup and run each setup module in isolation; /doctor and tools.swarmos.doctor report registered, skipped (with the missing capability) and failed modules plus caught hook errors. - scripts/check-host-contract.ts scans an OpenCode V2 checkout for the hook names and APIs SwarmOS relies on (also runnable via OPENCODE_V2_SRC in tests). - tools.verify.run discovers the repo's own checks, scopes tests to files that import the changed code, and returns structured counts and file:line failures. QA PASS receipts and the release gate now require a passing run on the current working tree (opt out with requireVerifiedReceipts: false). - Per-session telemetry and an observe-only mode used as the benchmark baseline. - bench/: 12 tasks with hidden tests and reference solutions, reference/noop/ agent drivers, and a SwarmOS-vs-baseline report. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0171AqkGJuWMw9gY25tb1Q7T
1.7 KiB
SwarmOS Benchmark
Measures whether SwarmOS actually helps by running the same tasks with SwarmOS fully on
(swarmos) and with SwarmOS loaded in observe-only mode (baseline: telemetry and /doctor, nothing else).
Tasks
Each task in tasks/<id>/ has:
| Path | Purpose |
|---|---|
task.json |
id, category (bug, refactor, constraints, ui), the user-style prompt, the verify command and a timeout |
repo/ |
Starting repository the agent works in |
hidden/ |
Tests copied to __hidden__/ only after the agent finishes |
solution/ |
Reference fix, used to prove the task is solvable |
Running
bun run bench:check # reference driver must pass 12/12, noop must pass 0/12
bun bench/run.ts --driver agent --repeat 3 # real comparison, both variants
bun bench/run.ts --driver agent --tasks bug-paginate,ui-spacing-tokens --variants swarmos
The agent driver runs BENCH_AGENT_CMD (default opencode run "$BENCH_PROMPT") inside a fresh git
workspace with an opencode.json that loads swarmos-plugin.ts (override with --plugin <path>).
Run bun run build first so the bundle is current.
Reports land in bench/results/ as JSON plus a Markdown table: success rate (overall and per category),
mean turns, mean tool calls, tool failure rate, circuit-breaker fires, tokens and cache-hit ratio,
and wall time. Turn, tool and breaker numbers come from SwarmOS telemetry; token numbers appear only
when the host reports usage on session.generate events.
Adding a task
Create the four parts above, then run bun run bench:check. test/bench.test.ts runs the same
check in CI, so a task whose hidden tests pass untouched or fail on the reference solution breaks the build.