Validate submitted bundles in CI (#2)

* Validate submitted bundle source in registry CI

* Remove the validated CI proof submission

* docs: state bundle check limits and validator upkeep

---------

Co-authored-by: OpenRig contributors <[email protected]>

.github/workflows/registry-check.yml

@@ -1,5 +1,5 @@
 # Checks registry entries, behaviour views, journeys and the generated status file. It parses files in this
-# repository only: nothing a submission contains is fetched, installed or run, and the job holds a read-only token.
+# repository only. The separate bundle job fetches submitted source for validation, never installs or launches it.
 name: Registry check
 
 on:
@@ -27,3 +27,30 @@ jobs:
       - run: npm ci --ignore-scripts --no-audit --no-fund
       - run: node --test test/*.test.mjs
       - run: node registry-check.mjs
+
+  bundles:
+    name: Submitted bundle check
+    needs: check
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    defaults:
+      run:
+        working-directory: tools
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
+        with:
+          persist-credentials: false
+      - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
+        with:
+          node-version: 22
+      - run: npm ci --ignore-scripts --no-audit --no-fund
+      - name: Install the published validator (no lifecycle scripts)
+        run: npm install --prefix "$RUNNER_TEMP/bundle-validator" --ignore-scripts --no-audit --no-fund @openrig/[email protected]
+      - name: Exercise source validation without installing or launching a bundle
+        run: node --test ci/bundle-check.test.mjs
+        env:
+          RIG_BIN: ${{ runner.temp }}/bundle-validator/node_modules/.bin/rig
+      - name: Check submitted bundles
+        run: node ci/bundle-check.mjs
+        env:
+          RIG_BIN: ${{ runner.temp }}/bundle-validator/node_modules/.bin/rig

README.md

@@ -43,8 +43,10 @@ npm test
 npm run check
 ```
 
-CI runs these checks on pull requests and pushes to main. It reads registry files; it does not execute submitted
-bundles. Status is generated from run records; it is not edited by hand.
+CI runs these checks on pull requests and pushes to main. A separate **Submitted bundle check** job fetches each
+submission's source and runs the pinned OpenRig validator. Its PR annotation and job summary show the exact commit
+and results. Submitted bundles are not installed or launched. Status is generated from run records; it is not edited
+by hand.
 
 ## License
 

registry/README.md

@@ -21,6 +21,7 @@ missing or describe another package, and a status file that was edited by hand.
 ## Submitting a rig
 
 Add `registry/submissions/<your-team>.yaml` with three fields (see `submissions/README.md`): your repository, the
-folder holding `rig.yaml`, and a branch, tag or commit. The check accepts it and says "Submission received". A
-maintainer then writes the full entry here, at a pinned commit, and removes the submission. A submission is never
-listed: the site and the status file read only the entries in this folder.
+folder holding `rig.yaml`, and a branch, tag or commit. The registry check accepts it and says "Submission received".
+The separate bundle check fetches the submitted revision and reports its `rig bundle check` results without
+installing or launching it. A maintainer then writes the full entry here, at a pinned commit, and removes the
+submission. A submission is never listed: the site and the status file read only the entries in this folder.

registry/submissions/README.md

@@ -9,9 +9,14 @@ folder: rigs/my-team   # the folder holding rig.yaml; "." for the repository roo
 ref: main              # a branch, tag or commit
 ```
 
-The registry check accepts it and says "Submission received". A maintainer then pins your link to an exact commit,
-reviews the bundle, writes the full entry in `registry/`, and removes this file. If it isn't listed, you get a short
-reply saying why.
+The registry check accepts it and says "Submission received". The **Submitted bundle check** job then fetches that
+ref and runs `rig bundle check` using OpenRig 0.6.6. Its PR annotation and job summary show the resolved commit,
+findings and checks left unverified. Findings or an unreadable source fail the check. The job does not install
+dependencies from the bundle, run its scripts, or launch its team. To check before submitting, run
+`rig bundle check <folder>` locally.
+
+A maintainer then pins your link to an exact commit, reviews the bundle, writes the full entry in `registry/`, and
+removes this file. If it isn't listed, you get a short reply saying why.
 
 Rig names on openrig.dev/rigs are unique. If `registry/<your-team>.yaml` already exists, the check still passes but
 says the name is taken; rename your file to a distinct one, for example `<taken-name>-<your-name>.yaml`.

tools/README.md

@@ -2,11 +2,13 @@
 
 Small Node programs that check the registry and generate its public status. They read files in this repository
 and, for the generator, run records and receipts on the machine running it. They do not fetch, install or run
-anything a submitted bundle contains.
+anything a submitted bundle contains. The separate CI bundle validator downloads source as data for the published
+CLI's read-only check; it does not execute bundle scripts.
 
 | Program | What it does | Where it runs |
 |---|---|---|
 | `registry-check.mjs` | Checks `registry/*.yaml`, each entry's behaviour views, `registry/submissions/`, `status/journeys/` and `status/status.json` | CI on every pull request and push to main |
+| `ci/bundle-check.mjs` | Fetches each submission's ref, records its exact commit, then runs `rig bundle check <folder> --json` | A separate CI job on every pull request and push to main |
 | `status.mjs` | Generates `status/status.json` from run records, or checks that the committed file equals regenerated output | Wherever the run records and their receipts are kept |
 
 ```sh
@@ -17,6 +19,22 @@ node status.mjs generate --records <folder> [--records <folder> ...]
 node status.mjs check --records <folder> [--records <folder> ...]
 ```
 
+The bundle job installs exactly `@openrig/[email protected]` with lifecycle scripts disabled, then runs the checks against
+temporary source checkouts. It never installs submitted dependencies, invokes bundle startup actions, or launches a
+daemon or team. The job has a read-only token and checkout does not persist credentials. Source fetches and validator
+processes each have a two-minute timeout; the job is capped at ten minutes. Every submission is checked on each run,
+including unchanged submissions; branch and tag results are bound to the resolved commit shown in that run.
+The check has time limits but no repository-size cap: a repository too large for the runner fails at its limits,
+rather than with a named size error.
+
+At each OpenRig release, maintainers must revisit the pinned validator version and the labels synchronized with it.
+Update the pin and its labels through a normal reviewed pull request.
+
+Findings and fetch/validator errors fail the job. `not_checked` results remain visible in its JSON summary; a pass
+does not establish README quality, absence of arbitrary secrets, or runtime behaviour. Maintainer review still
+decides listing. The integration controls in `ci/bundle-check.test.mjs` use local Git transport and the same installed
+CLI. To run them, set `RIG_BIN` to the validator executable and run `node --test ci/bundle-check.test.mjs`.
+
 ## Formats
 
 The formats are defined in the OpenRig repository's `docs/reference/bundle-formats.md`. This folder keeps a copy of

tools/ci/bundle-check.mjs

@@ -0,0 +1,91 @@
+#!/usr/bin/env node
+// Fetch source as data, then invoke the published validator. Never install or launch a submitted bundle.
+import fs from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { execFileSync } from "node:child_process";
+import { fileURLToPath, pathToFileURL } from "node:url";
+import { loadSubmissions, loadValidators } from "../lib/load.mjs";
+
+const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../..");
+const options = { encoding: "utf8", timeout: 120_000, maxBuffer: 2 * 1024 * 1024, stdio: ["ignore", "pipe", "pipe"] };
+const git = (cwd, args) => execFileSync("git", ["-c", "core.hooksPath=/dev/null", ...args], {
+  ...options, cwd, env: { ...process.env, GIT_TERMINAL_PROMPT: "0" },
+}).trim();
+
+/** The caller supplies a schema-validated submission. Each fetch and validator gets its own temporary folder. */
+export function checkSubmission(submission, rig = process.env.RIG_BIN || "rig") {
+  const temp = fs.mkdtempSync(path.join(os.tmpdir(), "openrig-bundle-check-"));
+  const checkout = path.join(temp, "source");
+  let resolvedCommit;
+  try {
+    fs.mkdirSync(checkout);
+    // A ref names one branch, tag or commit, not a fetch refspec or an option.
+    if (submission.ref.startsWith("-")) throw new Error("ref must be a branch, tag or commit");
+    git(checkout, ["check-ref-format", "--allow-onelevel", submission.ref]);
+    git(checkout, ["init", "--quiet"]);
+    git(checkout, ["fetch", "--quiet", "--depth=1", "--no-tags", "--", submission.repository, submission.ref]);
+    resolvedCommit = git(checkout, ["rev-parse", "--verify", "FETCH_HEAD^{commit}"]);
+    git(checkout, ["checkout", "--quiet", "--detach", resolvedCommit]);
+    const folder = fs.realpathSync(path.resolve(checkout, submission.folder));
+    const relative = path.relative(fs.realpathSync(checkout), folder);
+    if (relative === ".." || relative.startsWith(`..${path.sep}`) || path.isAbsolute(relative)) {
+      throw new Error("bundle folder resolves outside the fetched repository");
+    }
+    let stdout;
+    let status = 0;
+    try {
+      stdout = execFileSync(rig, ["bundle", "check", folder, "--json"], {
+        ...options, cwd: temp, env: { ...process.env, OPENRIG_HOME: path.join(temp, "home") },
+      });
+    } catch (error) {
+      // Exit 1 is the CLI's normal findings result. Launch failures and timeouts are not validation results.
+      if (error.status !== 1 || error.signal) throw error;
+      stdout = String(error.stdout);
+      status = 1;
+    }
+    const result = JSON.parse(stdout);
+    if (result.standardVersion !== "openrig.bundle-standard/v1" || !Array.isArray(result.checks) || !result.checks.length ||
+        result.checks.some(check => !["pass", "finding", "not_checked"].includes(check.status))) {
+      throw new Error("validator did not return a bundle-standard result");
+    }
+    const findings = result.checks.filter(check => check.status === "finding").length;
+    return { source: submission, resolvedCommit, status: status || (findings ? 1 : 0), result };
+  } catch (error) {
+    return { source: submission, ...(resolvedCommit ? { resolvedCommit } : {}), status: 1, error: error.message };
+  } finally {
+    fs.rmSync(temp, { recursive: true, force: true });
+  }
+}
+
+export function checkSubmissions(root = ROOT) {
+  const validators = loadValidators();
+  if (validators.problems.length) throw new Error(validators.problems.join("; "));
+  return loadSubmissions(root, validators).map(({ file, submission, problem }) => ({
+    file, ...(problem ? { status: 1, error: problem } : checkSubmission(submission)),
+  }));
+}
+
+// Neither annotation data nor a JSON code block may become workflow commands or Markdown supplied by a bundle.
+const escapeAnnotation = text => String(text).replaceAll("%", "%25").replaceAll("\r", "%0D").replaceAll("\n", "%0A").replaceAll(":", "%3A").replaceAll(",", "%2C");
+export function report(results, { log = console.log, summary = process.env.GITHUB_STEP_SUMMARY, actions = process.env.GITHUB_ACTIONS === "true" } = {}) {
+  for (const result of results) {
+    log(JSON.stringify(result));
+    if (actions) {
+      const checks = result.result?.checks || [];
+      const counts = `${checks.filter(c => c.status === "finding").length} finding(s), ${checks.filter(c => c.status === "not_checked").length} not checked`;
+      const message = result.error || `${counts}; resolved commit ${result.resolvedCommit}. See the job summary for every rule.`;
+      log(`::${result.status ? "error" : "notice"} file=${escapeAnnotation(result.file)},title=Bundle check::${escapeAnnotation(message)}`);
+    }
+  }
+  if (!results.length) log("No submitted bundles to check.");
+  if (summary) {
+    const json = JSON.stringify(results, null, 2).replaceAll("`", "\\u0060").replaceAll("<", "\\u003c");
+    fs.appendFileSync(summary, `## Submitted bundle checks\n\nValidator: @openrig/cli 0.6.6. Validation only; no bundle is installed or launched.\n\n${results.length ? "```json\n" + json + "\n```" : "No submitted bundles to check."}\n`);
+  }
+  return results.some(result => result.status !== 0) ? 1 : 0;
+}
+
+if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) {
+  process.exitCode = report(checkSubmissions());
+}

tools/ci/bundle-check.test.mjs

@@ -0,0 +1,117 @@
+import assert from "node:assert/strict";
+import fs from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { execFileSync } from "node:child_process";
+import { test } from "node:test";
+import { checkSubmission, checkSubmissions, report } from "./bundle-check.mjs";
+
+const SPEC = `version: "0.2"
+name: validator-fixture
+docs: [{path: README.md}]
+pods:
+  - id: infra
+    label: Infrastructure
+    members:
+      - { id: observer, runtime: terminal, agent_ref: "builtin:terminal", profile: none, cwd: "." }
+    edges: []
+edges: []
+`;
+
+function fixture(t) {
+  const root = fs.mkdtempSync(path.join(os.tmpdir(), "submission-test-"));
+  t.after(() => fs.rmSync(root, { recursive: true, force: true }));
+  const source = path.join(root, "source");
+  fs.mkdirSync(source);
+  const git = (...args) => execFileSync("git", args, { cwd: source, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim();
+  git("init", "-q", "-b", "main");
+  git("config", "user.name", "Test");
+  git("config", "user.email", "[email protected]");
+  fs.writeFileSync(path.join(source, "rig.yaml"), SPEC);
+  fs.writeFileSync(path.join(source, "README.md"), "# Fixture team\n");
+  // These would leave a receipt if anything installs dependencies or launches the source.
+  const marker = path.join(root, "executed");
+  const command = `node -e 'require("fs").writeFileSync(${JSON.stringify(marker)},"ran")'`;
+  fs.writeFileSync(path.join(source, "package.json"), JSON.stringify({ scripts: { preinstall: command, postinstall: command, start: command } }));
+  git("add", "."); git("commit", "-qm", "Valid bundle");
+  const commit = git("rev-parse", "HEAD");
+  git("tag", "valid");
+  const saved = { ...process.env };
+  // Only transport is substituted; production still parses a real, schema-valid GitHub submission.
+  process.env.GIT_CONFIG_COUNT = "2";
+  process.env.GIT_CONFIG_KEY_0 = `url.file://${source}.insteadOf`;
+  process.env.GIT_CONFIG_VALUE_0 = "https://github.com/example/team";
+  process.env.GIT_CONFIG_KEY_1 = "protocol.file.allow";
+  process.env.GIT_CONFIG_VALUE_1 = "always";
+  t.after(() => { for (const key of Object.keys(process.env)) if (!(key in saved)) delete process.env[key]; Object.assign(process.env, saved); });
+  const registry = path.join(root, "registry");
+  fs.mkdirSync(path.join(registry, "registry", "submissions"), { recursive: true });
+  const submission = { repository: "https://github.com/example/team", folder: ".", ref: "main" };
+  const writeSubmission = (value = submission) => fs.writeFileSync(path.join(registry, "registry/submissions/team.yaml"), JSON.stringify(value));
+  return { root, source, git, commit, marker, registry, submission, writeSubmission };
+}
+
+test("a submitted branch reaches the real CLI, preserves not_checked, and executes no source scripts", t => {
+  const f = fixture(t); f.writeSubmission();
+  const results = checkSubmissions(f.registry);
+  assert.equal(results.length, 1);
+  assert.equal(results[0].status, 0, JSON.stringify(results));
+  assert.equal(results[0].resolvedCommit, f.commit);
+  assert.ok(results[0].result.checks.some(c => c.ruleId === "pod_aware_rig" && c.status === "pass"));
+  assert.ok(results[0].result.checks.some(c => c.ruleId === "embedded_secrets" && c.status === "not_checked"));
+  assert.equal(fs.existsSync(f.marker), false);
+  assert.equal(f.git("status", "--porcelain"), "");
+});
+
+test("a pinned commit and tag check those bytes even when the branch has become invalid", t => {
+  const f = fixture(t);
+  fs.writeFileSync(path.join(f.source, "rig.yaml"), "not: a rig\n");
+  f.git("add", "."); f.git("commit", "-qm", "Broken bundle");
+  for (const ref of [f.commit, "valid"]) {
+    const result = checkSubmission({ ...f.submission, ref });
+    assert.equal(result.status, 0, JSON.stringify(result));
+    assert.equal(result.resolvedCommit, f.commit);
+  }
+  const result = checkSubmission(f.submission);
+  assert.equal(result.status, 1);
+  assert.equal(result.resolvedCommit, f.git("rev-parse", "HEAD"));
+  assert.ok(result.result.checks.some(c => c.ruleId === "pod_aware_rig" && c.status === "finding"));
+  assert.equal(report([result], { log() {}, summary: null, actions: false }), 1);
+});
+
+test("invalid input, missing refs and escaping folder links fail instead of silently skipping validation", t => {
+  const f = fixture(t);
+  f.writeSubmission({ ...f.submission, repository: "file:///not-a-submission" });
+  assert.equal(checkSubmissions(f.registry)[0].status, 1);
+  assert.equal(checkSubmission({ ...f.submission, ref: "missing" }).status, 1);
+  assert.equal(checkSubmission({ ...f.submission, ref: "--upload-pack=unexpected" }).status, 1);
+  fs.symlinkSync(os.tmpdir(), path.join(f.source, "outside"));
+  f.git("add", "."); f.git("commit", "-qm", "Outside folder");
+  const result = checkSubmission({ ...f.submission, folder: "outside" });
+  assert.equal(result.status, 1);
+  assert.match(result.error, /outside the fetched repository/);
+});
+
+test("missing README is a real CLI finding; a missing validator is a distinct error", t => {
+  const f = fixture(t);
+  fs.unlinkSync(path.join(f.source, "README.md"));
+  f.git("add", "."); f.git("commit", "-qm", "Missing README");
+  const result = checkSubmission(f.submission);
+  assert.equal(result.status, 1);
+  assert.ok(result.result.checks.some(c => c.ruleId === "readme_in_docs" && c.status === "finding"));
+  const failed = checkSubmission(f.submission, path.join(f.root, "no-validator"));
+  assert.equal(failed.status, 1); assert.ok(failed.error); assert.equal(failed.result, undefined);
+});
+
+test("no submissions pass without fetching, and CI output quotes authored text", t => {
+  const f = fixture(t);
+  assert.deepEqual(checkSubmissions(f.registry), []);
+  const output = []; const summary = path.join(f.root, "summary.md");
+  const result = { file: "registry/submissions/team.yaml", status: 1, error: "bad\n::notice::forged\n```\n<script>" };
+  assert.equal(report([result], { log: s => output.push(s), summary, actions: true }), 1);
+  assert.equal(output.length, 2);
+  assert.ok(output.every(line => !line.includes("\n")));
+  assert.match(output[1], /%0A%3A%3Anotice/);
+  assert.equal(fs.readFileSync(summary, "utf8").match(/```/g).length, 2);
+  assert.doesNotMatch(fs.readFileSync(summary, "utf8"), /<script>/);
+});