Verify your opencode v2 plugin actually works — locally and in GitHub Actions — by driving the real opencode binary with your plugin loaded and asserting on what happened.

What it does: for each opencode version you configure, it creates an isolated sandbox (fresh HOME/XDG_*, git-initialized project dir), loads your plugin between two probe plugins, runs opencode run, and checks tier-1 activation plus your scripted (stub model, deterministic) and live (real provider) scenarios. Results become results.json, a Markdown report (appended to $GITHUB_STEP_SUMMARY in Actions), and an exit code: 0 pass, 1 failure, 2 configuration error.

The design is in docs/SPEC.md. Machine-readable schemas live in schema/config.json and schema/scenario.json (generated by bun scripts/gen-schema.ts, checked in).

Prerequisites

Bun ≥ 1.2, Node ≥ 22.
Network access the first time each opencode version is used (the tool downloads @opencode/cli@<version> into a cache), or set OPENCODE_BIN to an existing binary.

5-minute quickstart

From your plugin repo root:

# 1. Scaffold starter files (.opencode-ci/config.yaml + an example scenario).
#    Plain `init` refuses to overwrite existing files; --force re-scaffolds
#    the starters (used here so this quickstart is repeatable — back up any
#    hand-written scenarios first).
bunx @heretek-ai/opencode-plugin-ci init --force

# 2. Edit .opencode-ci/config.yaml: set plugin.path if your package lives in
#    a subdirectory, list the tools/commands/agents/skills/mcp your plugin
#    contributes under `expect`, and pick your opencode versions + live model.
#    See "Config reference" below.

# 3. Check that everything validates and the plugin entry resolves.
bunx @heretek-ai/opencode-plugin-ci validate

# 4. Run the starter scenario. The live tier auto-skips when no API key
#    is configured, so this is free and offline.
bunx @heretek-ai/opencode-plugin-ci run --scenario example

Add your own scenarios under .opencode-ci/scenarios/ (see "Scenario reference" below) and run everything with plain bunx @heretek-ai/opencode-plugin-ci run.

Open .opencode-ci/out/report.md for the report. Each scenario keeps its evidence under .opencode-ci/out/<opencode-version>/<scenario-id>/ (events.jsonl, probe.jsonl, stub-requests.jsonl, stderr.log).

Config reference

.opencode-ci/config.yaml (or config.json; exactly one may exist). Every object is strict: unknown keys fail validation so typos never pass silently. Full schema: schema/config.json.

# yaml-language-server: $schema=https://unpkg.com/@heretek-ai/opencode-plugin-ci/schema/config.json
plugin:
  path: .                    # plugin package dir in the checkout
  entry: dist/server.js       # optional override; else auto-detected
  install: bun install        # optional, runs once before any tier
  build: bun run build        # optional, runs once before any tier
  options: {}                 # default plugin options for every scenario
  ui: false                   # true = TUI-only plugin (tier 1 load check only)
opencode:
  versions: ["2.0.26", "latest"]  # first is required; the rest may fail
expect:                       # tier 1 inventory expectations
  tools: [my_tool]
  commands: []
  agents: []
  skills: []
  mcp: []
  providers: []               # for provider plugins (tier 1)
  models: []                  # for provider plugins (tier 1)
live:
  model: opencode/some-model  # provider/model, or a bare id resolved via `opencode models`
  keys: [OPENCODE_API_KEY]    # env vars passed into the sandbox
  repeat: 3
  pass: 2                     # must be <= repeat
  budgetUSD: 1.00             # hard stop for the whole live tier
gate:
  enabled: true               # historical flake gate, live tier only
  window: 5
  threshold: 0.9
  bootstrap: fail             # fail|allow — allow tolerates "no history yet"
  workflow: plugin-ci.yml     # optional; else inferred from GITHUB_WORKFLOW_REF
  artifact: opencode-plugin-ci-results  # must match the Action's artifact-name input
logLevel: info                # info|all

Scenarios are .opencode-ci/scenarios/*.{yaml,yml,json}, sorted by filename; duplicate ids are an error. Schema: schema/scenario.json.

id: redacts-secrets           # ^[a-z0-9][a-z0-9-]*$
description: execute.before strips API keys from bash commands
mode: scripted                # scripted | live
plugin:
  options: { mode: strict }   # merged over config plugin.options
workspace:                    # files written into the project dir before the run
  .env: "API_KEY=sk-test\n"
prompt: "print the env file"
script:                       # scripted only: one step per agent-loop request
  - toolCalls:
      - name: shell
        arguments: { command: "cat .env", description: "show env" }
  - text: done
timeoutMs: 120000
expect:
  exitCode: 0
  tools:
    - name: shell
      status: completed
judge:                        # live only, optional
  rubric: "The reply explains that the key was redacted and does not reveal it."

Cross-field rules (all validated): script is required in scripted mode and forbidden in live mode; expect.request and judgeReply are scripted-only; judge is live-only; live.pass <= live.repeat.

Scenario reference: one example per assertion type

All assertions live under expect: and are AND-ed. Matchers available on most fields: equals, contains, notContains, matches / notMatches (JavaScript regex source; a leading (?i) enables case-insensitivity), includes (arrays), exists, count ({ eq?, gte?, lte? }). Non-string values are compared as JSON for the string matchers; equals uses deep equality.

exitCode — the opencode run exit code (default 0): exitCode: 0
noErrors — no error events in --format json stdout (default true): noErrors: true

tools — tool calls from the probe's tool.hook records (which see Code Mode inner calls under their real names). At least one call must match status, input and output together; count defaults to gte: 1:

tools:
  - name: shell
    status: completed
    input: { command: { notContains: "sk-test" } }
    output: { contains: "redacted" }
    count: { gte: 1 }

noTools — the named tools must never be called (neither in the probe log nor as tool_use events):
```
noTools: [webfetch]
```

hooks — raw probe records. hook is one of tool.execute.before, tool.execute.after, shell.create.before, session.context, session.prompt; phase is pre (what your plugin received) or post (after its mutation, default). env values are redacted for secret-like keys before recording:

hooks:
  - hook: tool.execute.before
    phase: post
    tool: shell
    input: { command: { notContains: "sk-test" } }
  - hook: shell.create.before
    env: { MY_PLUGIN_ACTIVE: { equals: "1" } }
  - hook: session.context
    system: { contains: "Never print secrets" }

request — scripted only. Picks the nth (1-based) model request the stub recorded: system is the system-role text, tools covers both directly-offered tools and Code Mode catalog entries (matched by full path, last segment, or _-joined name):
```
request:
  nth: 1
  system: { contains: "Never print secrets" }
  tools: { includes: [my_tool] }
```

files — project dir after the run. Paths are relative; any .. escaping the project dir is rejected:

files:
  - path: .my-plugin/log.jsonl
    exists: true
    contains: "redacted"

stderr — the raw server logs (--print-logs):
```
stderr:
  notMatches: "(?i)unhandled"
```
text — concatenated text events (the model's visible reply):
```
text:
  notContains: "sk-test"
```

Your tools are in Code Mode

Tools your plugin adds with ctx.tool.transform are not offered to the model as normal tools by default. They are listed in a catalog inside the system prompt, and the model calls them through the built-in execute tool: execute({ code: "return await tools.my_tool({ text: \"hi\" })" }). This means:

In scripted scenarios, just name your tool normally — the stub auto-wraps the call into execute using the catalog path it finds in the recorded system text. (A step can opt out with raw: true.)
Live models do the same thing on their own: they read the catalog and call execute.
Set options: { codemode: false } on the tool to make it a direct tool instead (it then appears in the request's tools array).
Because --format json only shows the outer execute call, write tool assertions against the probe (tools: / hooks:), which sees every inner call under its real name — never against tool_use events.

Tiers

Tier	Name	Model	Runs
1	`load`	stub	always — plugin activates, inventory matches `expect`
2	`scripted`	stub	always — deterministic scenarios, no tokens
3	`live`	real provider	only with a key; `pass` of `repeat` attempts must pass

Filter with run --tier load --tier scripted, --scenario <id> (repeatable, comma-separated), --opencode <v>, --out <dir>, --keep (keep sandboxes for debugging), --gate (force the gate locally), --verbose / --quiet.

TUI-only plugins (no server entry, or plugin.ui: true) get a static tier-1 check (default export has a string id and function setup) and tiers 2–3 report skipped. A failing tier 1 marks that version's later tiers skipped: "tier 1 failed". Results for opencode.versions[0] are required; other versions are allowed failures (reported, exit code unaffected). .opencode-ci/**/*.test.ts files run once via bun test with OPENCODE_BIN set, folded in as a bun-test case.

Gate (live tier only)

Tiers 1 and 2 fail on any failure. Live failures go through a historical flake gate instead: the live tier fails only when a stable case (passed ≥ 90% of ≥ 5 sampled runs) fails now, plus sticky-red and human-written .opencode-ci/acknowledgments.json ({ version: 1, acknowledgments: [{ case, reason, issue, date }] }; malformed fails closed).

History comes from earlier opencode-plugin-ci-results artifacts on the same branch via gh, so the workflow needs actions: read. The gate runs only when the live tier ran at least one non-skipped case, gate.enabled is true, and --gate was passed or GITHUB_ACTIONS == "true". Without gh/token it falls back to "fail on any live failure". No history exits 2 unless gate.bootstrap: allow.

Matrix builds: artifact names must be unique within a workflow run, so give each matrix leg a distinct artifact-name Action input and set the same value in that leg's gate.artifact config (the gate downloads history by that name). Example:

strategy:
  matrix:
    opencode: ["2.0.26", "latest"]
steps:
  - uses: Heretek-AI/opencode-v2-plugin-teser@v1
    with:
      artifact-name: opencode-plugin-ci-results-${{ matrix.opencode }}

with gate: { artifact: opencode-plugin-ci-results-2.0.26 } in the required leg's config (each leg reads its own checkout's config, so parameterize it per leg or keep one artifact per workflow).

GitHub Action

Heretek-AI/opencode-v2-plugin-teser@v1 (see action.yml). Inputs tiers, opencode-versions (overrides config), working-directory, upload-artifact (default true), artifact-name (default opencode-plugin-ci-results). Outputs result (pass/fail) and report-path. Inputs travel to the script through env:, never interpolated. Minimal workflow (examples/workflow.yml):

name: plugin-ci
on: [push, pull_request]
permissions:
  contents: read
  actions: read # flake gate history
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: Heretek-AI/opencode-v2-plugin-teser@v1
        with:
          tiers: load,scripted,live # live auto-skips without a key
        env:
          OPENCODE_API_KEY: ${{ secrets.OPENCODE_API_KEY }}

Security

Never use pull_request_target with the PR's code checked out and secrets exposed. The live tier runs the plugin under test's code with your real API key: a malicious PR could exfiltrate the key. Use pull_request (fork PRs get no secrets, so live auto-skips and stays green) and run live checks on push to trusted branches.
Treat scenario workspace files and prompts as untrusted input to the plugin, and keep secrets out of assertions: the probe redacts secret-like env/header values, and you should too — never print or record secret values.

README.md Unescape Escape

opencode-plugin-ci