opencode-plugin-ci

Verify your opencode v2 plugin actually works — locally and in GitHub Actions — by driving the real opencode binary with your plugin loaded and asserting on what happened.

What it does: for each opencode version you configure, it creates an isolated sandbox (fresh HOME/XDG_*, git-initialized project dir), loads your plugin between two probe plugins, runs opencode run, and checks tier-1 activation plus your scripted (stub model, deterministic) and live (real provider) scenarios. Results become results.json, a Markdown report (appended to $GITHUB_STEP_SUMMARY in Actions), and an exit code: 0 pass, 1 failure, 2 configuration error.

The design is in docs/SPEC.md. Machine-readable schemas live in schema/config.json and schema/scenario.json (generated by bun scripts/gen-schema.ts, checked in).

Prerequisites

• Bun ≥ 1.2, Node ≥ 22.
• Network access the first time each opencode version is used (the tool downloads @opencode/cli@<version> into a cache), or set OPENCODE_BIN to an existing binary.

5-minute quickstart

From your plugin repo root:

NOTE: the package is published as @heretek-ai/opencode-plugin-ci, so bunx @heretek-ai/opencode-plugin-ci ... works directly. To run from a source checkout instead, <checkout> below is your clone of this repo, and bun <checkout>/bin/opencode-plugin-ci.ts ... takes the place of bunx ....

# 1. Scaffold starter files (.opencode-ci/config.yaml + an example scenario).
#    Plain `init` refuses to overwrite existing files; --force re-scaffolds
#    the starters (used here so this quickstart is repeatable — back up any
#    hand-written scenarios first).
bun <checkout>/bin/opencode-plugin-ci.ts init --force

# 2. Edit .opencode-ci/config.yaml: set plugin.path if your package lives in
#    a subdirectory, list the tools/commands/agents/skills/mcp your plugin
#    contributes under `expect`, and pick your opencode versions + live model.
#    See "Config reference" below.

# 3. Check that everything validates and the plugin entry resolves.
bun <checkout>/bin/opencode-plugin-ci.ts validate

# 4. Run the starter scenario. The live tier auto-skips when no API key
#    is configured, so this is free and offline.
bun <checkout>/bin/opencode-plugin-ci.ts run --scenario example

Add your own scenarios under .opencode-ci/scenarios/ (see "Scenario reference" below) and run everything with plain bun <checkout>/bin/opencode-plugin-ci.ts run.

Open .opencode-ci/out/report.md for the report. Each scenario keeps its evidence under .opencode-ci/out/<opencode-version>/<scenario-id>/ (events.jsonl, probe.jsonl, stub-requests.jsonl, stderr.log).

Config reference

.opencode-ci/config.yaml (or config.json; exactly one may exist). Every object is strict: unknown keys fail validation — errors name the file and field, not a line number — so typos never pass silently. Full schema: schema/config.json.

# yaml-language-server: $schema=https://unpkg.com/@heretek-ai/opencode-plugin-ci/schema/config.json
plugin:
  path: .                    # plugin package dir in the checkout
  entry: dist/server.js       # optional; must exist after build, else auto-detected
  install: bun install        # optional, runs once before any tier
  build: bun run build        # optional, runs once before entries are resolved
  options: {}                 # default plugin options for every scenario
  ui: false                   # true = TUI-only plugin (tier 1 load check only)
opencode:
  versions: ["2.0.26", "latest"]  # first is required; the rest may fail
expect:                       # tier 1 inventory expectations
  tools: [my_tool]
  commands: []
  agents: []
  skills: []
  mcp: []
live:
  model: opencode/some-model  # provider/model, or a bare id resolved via `opencode models`
  keys: [OPENCODE_API_KEY]    # env vars passed into the sandbox
  repeat: 3
  pass: 2                     # must be <= repeat
  budgetUSD: 1.00             # hard stop for the whole live tier, across all versions
  timeoutMs: 240000           # per live attempt, unless the scenario sets timeoutMs
gate:
  enabled: true               # historical flake gate, live tier only
  bootstrap: fail             # fail|allow — allow tolerates "no history yet"
  workflow: plugin-ci.yml     # optional; else inferred from GITHUB_WORKFLOW_REF
  artifact: opencode-plugin-ci-results  # history artifact; --gate-artifact (the Action's artifact-name) overrides it
logLevel: info                # info|all

Scenarios are .opencode-ci/scenarios/*.{yaml,yml,json}, sorted by filename; duplicate ids are an error. Schema: schema/scenario.json.

id: redacts-secrets           # ^[a-z0-9][a-z0-9-]*$; "load" and "bun-test" are reserved
description: execute.before strips API keys from bash commands
mode: scripted                # scripted | live
plugin:
  options: { mode: strict }   # merged over config plugin.options
workspace:                    # files written into the project dir before the run (relative paths, no "..")
  .env: "API_KEY=sk-test\n"
prompt: "print the env file"
script:                       # scripted only: one step per agent-loop request
  - toolCalls:
      - name: shell
        arguments: { command: "cat .env", description: "show env" }
  - text: done
timeoutMs: 120000             # optional; default 120000 (scripted) or live.timeoutMs (live)
expect:
  exitCode: 0
  tools:
    - name: shell
      status: completed
judge:                        # live only, optional
  rubric: "The reply explains that the key was redacted and does not reveal it."

Cross-field rules (all validated): script is required in scripted mode and forbidden in live mode; expect.request is scripted-only; judge is live-only; live.pass <= live.repeat; id must not be load or bun-test; workspace keys must be relative paths that stay inside the project dir.

Scenario reference: one example per assertion type

All assertions live under expect: and are AND-ed. Matchers available on most fields: equals, contains, notContains, matches / notMatches (JavaScript regex source; a leading (?i) enables case-insensitivity), includes (arrays), exists, count ({ eq?, gte?, lte? }). Non-string values are compared as JSON for the string matchers; equals uses deep equality. A field with no recorded value fails every matcher except exists: false, so a misspelled field name (input: { comand: … }) fails with no value was recorded (check the field name) instead of passing a notContains.

• exitCode — the opencode run exit code (default 0): exitCode: 0

• noErrors — no error events in --format json stdout (default true): noErrors: true

• tools — tool calls from the probe's tool.hook records (which see Code Mode inner calls under their real names). At least one call must match status, input and output together; count defaults to gte: 1:

  tools:
    - name: shell
      status: completed
      input: { command: { notContains: "sk-test" } }
      output: { contains: "redacted" }
      count: { gte: 1 }
  

• noTools — the named tools must never be called (neither in the probe log nor as tool_use events):

  noTools: [webfetch]
  

• hooks — raw probe records. hook is one of tool.execute.before, tool.execute.after, shell.create.before, session.context, session.prompt; phase is pre (what your plugin received) or post (after its mutation, default). env values are redacted for secret-like keys before recording:

  hooks:
    - hook: tool.execute.before
      phase: post
      tool: shell
      input: { command: { notContains: "sk-test" } }
    - hook: shell.create.before
      env: { MY_PLUGIN_ACTIVE: { equals: "1" } }
    - hook: session.context
      system: { contains: "Never print secrets" }
  

• request — scripted only. Picks the nth (1-based) model request the stub recorded: system is the system-role text, tools covers both directly-offered tools and Code Mode catalog entries (matched by full path, last segment, or _-joined name):

  request:
    nth: 1
    system: { contains: "Never print secrets" }
    tools: { includes: [my_tool] }
  

• files — project dir after the run. Paths are relative; any .. escaping the project dir is rejected:

  files:
    - path: .my-plugin/log.jsonl
      exists: true
      contains: "redacted"
  

• stderr — the raw server logs (--print-logs):

  stderr:
    notMatches: "(?i)unhandled"
  

• text — concatenated text events (the model's visible reply):

  text:
    notContains: "sk-test"
  

Your tools are in Code Mode

Tools your plugin adds with ctx.tool.transform are not offered to the model as normal tools by default. They are listed in a catalog inside the system prompt, and the model calls them through the built-in execute tool: execute({ code: "return await tools.my_tool({ text: \"hi\" })" }). This means:

• In scripted scenarios, just name your tool normally — the stub auto-wraps the call into execute using the catalog path it finds in the recorded system text. (A step can opt out with raw: true.)
• Live models do the same thing on their own: they read the catalog and call execute.
• Set options: { codemode: false } on the tool to make it a direct tool instead (it then appears in the request's tools array).
• Because --format json only shows the outer execute call, write tool assertions against the probe (tools: / hooks:), which sees every inner call under its real name — never against tool_use events.
• Tools registered with a namespace are recorded as <namespace>_<name> (dots become _). In tools:, noTools: and hooks[].tool you may write that name, ns.name or the catalog path tools.ns.name; all three match. A bare name does not match, and the failure says which namespaced name was recorded.

Tiers

Tier	Name	Model	Runs
1	load	stub	always — plugin activates, inventory matches expect
2	scripted	stub	always — deterministic scenarios, no tokens
3	live	real provider	only with a key; pass of repeat attempts must pass

Filter with run --tier load --tier scripted, --scenario <id> (repeatable, comma-separated; an unknown id is a config error, exit 2), --opencode <v> (x.y.z or latest), --out <dir>, --keep (keep sandboxes for debugging), --gate (force the gate locally), --gate-artifact <name> (overrides gate.artifact), --verbose / --quiet. Each case's evidence dir under --out is wiped when the case starts, so re-running into the same out dir never mixes runs.

TUI-only plugins (no server entry, or plugin.ui: true) get a static tier-1 check (default export has a string id and function setup) and tiers 2–3 report skipped. A failing tier 1 marks that version's later tiers skipped: "tier 1 failed". Results for opencode.versions[0] are required; other versions are allowed failures (reported, exit code unaffected). .opencode-ci/**/*.test.ts files run once via bun test with OPENCODE_BIN set, folded in as a bun-test case (the tool passes each file as an explicit ./ path, because bun test skips dot-dirs when given a directory filter). The live budget is shared by every version in one run. Once it is spent, a case that has not started is skipped, and a case that already ran attempts but has fewer than pass passes fails with (budget exhausted …) in its failure line.

Gate (live tier only)

Tiers 1 and 2 fail on any failure. Live failures go through a historical flake gate instead: the live tier fails only when a stable case (passed ≥ 90% of ≥ 5 sampled runs) fails now, plus sticky-red and human-written .opencode-ci/acknowledgments.json ({ version: 1, acknowledgments: [{ case, reason, issue, date }] }; malformed fails closed).

History comes from earlier opencode-plugin-ci-results artifacts via gh, so the workflow needs actions: read. Push runs read their own branch's history; pull request runs read the base branch's history (GITHUB_BASE_REF), since a PR's own ref has no earlier runs. The gate runs only when the live tier ran at least one non-skipped case, gate.enabled is true, and --gate was passed or GITHUB_ACTIONS == "true". Without gh/token it falls back to "fail on any live failure". No history exits 2 unless gate.bootstrap: allow.

Matrix builds: artifact names must be unique within a workflow run, so give each matrix leg a distinct artifact-name Action input. The Action passes it to the CLI as --gate-artifact, so each leg reads its own history; no config change is needed, and validation does NOT enforce this. Example:

strategy:
  matrix:
    opencode: ["2.0.26", "latest"]
steps:
  - uses: Heretek-AI/opencode-v2-plugin-teser@v1
    with:
      opencode-versions: ${{ matrix.opencode }}
      artifact-name: opencode-plugin-ci-results-${{ matrix.opencode }}

GitHub Action

Heretek-AI/opencode-v2-plugin-teser@v1 (see action.yml). Inputs tiers, opencode-versions (overrides config), working-directory (must stay inside GITHUB_WORKSPACE; absolute paths inside the workspace are fine, paths outside exit 2), upload-artifact (default true), artifact-name (default opencode-plugin-ci-results; also the gate's history name). There is no config input: the tool reads .opencode-ci/config.* in working-directory. Outputs result (pass/fail; a fully skipped run is pass) and report-path. Inputs travel to the script through env:, never interpolated. Minimal workflow (examples/workflow.yml):

name: plugin-ci
on: [push, pull_request]
permissions:
  contents: read
  actions: read # flake gate history
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: Heretek-AI/opencode-v2-plugin-teser@v1
        with:
          tiers: load,scripted,live # live auto-skips without a key
        env:
          OPENCODE_API_KEY: ${{ secrets.OPENCODE_API_KEY }}

Security

• Never use pull_request_target with the PR's code checked out and secrets exposed. The live tier runs the plugin under test's code with your real API key: a malicious PR could exfiltrate the key. Use pull_request (fork PRs get no secrets, so live auto-skips and stays green) and run live checks on push to trusted branches.
• Treat scenario workspace files and prompts as untrusted input to the plugin, and keep secrets out of assertions: the probe redacts secret-like env/header values at the source, but every other evidence surface (prompt, tool I/O, session context, stub requests, events, report) MUST NOT contain secrets — that is the scenario author's responsibility. Never print or record secret values (a scenario that runs env or cat .env persists the key into the uploaded evidence).
• As defense in depth, explicitly configured live-key values (live.keys) are scrubbed to [redacted] in report.md, results.json failures, stub-requests.jsonl, events.jsonl and stderr.log before they are written. probe.jsonl stays raw on purpose, so notContains-style assertions still see — and fail on — leaked values.