ci: run the Runner support suite as a required PR check (#19537)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Runner support suite (`pnpm test:e2e:runner:unit`) checks the code behind paid Runner runs: workflow security policy, evidence oracles, cost receipts and controller recovery. > - Only a manual Docker Runner check ran this suite. PR CI did not run it. > - So changes on master broke 16 of its tests, and nobody saw it. #19527 and #19529 fix those tests. > - This pull request runs the suite in PR CI and makes the `verify` check require it. > - The benefit is that a change that breaks the suite fails on its own pull request, before merge. ## Linked Issues or Issue Description Refs #19527 and #19529 (both merged; they repair the suite). Also accounts for #19533, which added the sandbox provider job. **Merge order:** #19527 and #19529 have merged, so the suite now passes on master. This PR can merge. **What existing behavior does this improve?** PR CI for `pr-trusted.yml`. **Current behavior** `pnpm test:e2e:runner:unit` runs only in `docker-runner-check.yml`, by manual dispatch. Master changes broke 12–13 tests by Nov 3 and 16 by Nov 6, with no signal on any pull request. **Proposed behavior** A required "Runner support suite" job runs the suite on pull requests. **Reason and benefit** Drift fails on the pull request that causes it. Authors update deliberate pins (profile versions, suite hashes, workflow policy) in the same pull request. ## What Changed - `pr-trusted.yml`: new job `verify_runner_e2e_support` ("Runner support suite"). It uses the same steps as other trusted jobs: checkout without credentials, setup-node directly before pnpm, the npm overrides, the shared `setup-ci-dependencies` restore, and CI profiling. Then it runs `pnpm test:e2e:runner:unit`. It has no secrets. Timeout: 20 minutes. - `pr-trusted.yml`, job `verify`: the job now needs the new job and checks its result. A failed, cancelled or wrongly skipped run fails `verify`. - `turbo.json`: new routing task `ci:runner-e2e-support`. Its inputs are `server/**`, `cli/**`, `packages/**` and `ui/**`. Changes to `tests/**`, `scripts/**` and `.github/**` already force full CI, so they also run the job. - `.github/ci-task-matrix.json`: one lane for the new job. - Tests: `ci-plan.test.mjs`, `ci-profile.test.mjs`, `e2e-shard.test.mjs` and `workflow-security.test.ts` now include the new job. A UI component change now selects this one Runner task. The suite imports UI modules and checks that catalog-listed UI files stay tracked in git, so UI changes can break it. - CI profiling: when `PAPERCLIP_CI_PROFILE` is on, the suite's Vitest config adds `scripts/ci-profile/vitest-reporter.mjs`, and `run-unit.mjs` runs the Node tests through `scripts/ci-profile/node-test.mjs`. So the job uploads per-test timings, like the root and Runner package suites. The launcher loads the profiling helper only after its "Vitest is not installed" check. - `tests/runner-e2e/README.md`: says that PR CI runs the suite, and that deliberate pin changes go in the same pull request. ## Verification - `node --test .github/scripts/tests/*.test.mjs scripts/ci/*.test.mjs scripts/__tests__/e2e-shard.test.mjs scripts/__tests__/release-verify-workflow.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs`: 830 passed, 3 failed. The same 3 fail on the base branch without this change. They need local tools that are not on this Mac. - A local run with `PAPERCLIP_CI_PROFILE=true` wrote per-test and per-file timing records for Vitest and for Node. Without profiling, output and exit codes do not change. Both `unit-launcher.test.mjs` tests pass. - The aggregate test now covers the new job: `verify` rejects it when it fails, is cancelled, or is skipped while selected. - The pnpm bootstrap policy tests in `workflow-security.test.ts` pass with 10 setups in `pr-trusted.yml`. - On this PR's merged tree (`80b526d697`, master plus this PR): `pnpm test:e2e:runner:unit` passed, Vitest 320 files (6,830 tests) and node:test 373, in 58s. It also passes on master `4c1f629c31`. - Suite runtime on a 10-core Mac: 43–63s. With `--maxWorkers=4`: Vitest 118s, about 490s of CPU in total. On a 4-vCPU GitHub runner I expect about 3–4 minutes for the tests and 1–2 minutes for setup. The job runs in parallel with the longer test shards, so it should not lengthen CI. - `pr.yml` calls `pr-trusted.yml@master`. This PR's own CI does not run the new job. The first real CI timing comes from the first pull request after merge. ## Risks - More runner minutes: one extra job on most pull requests (all except documentation-only ones). - The suite has not run on Linux CI recently: old `docker-runner-check` logs have expired. A Linux-only failure would show up on the first pull requests after merge. The job can be removed from the `verify` list quickly if needed. - #19229 removes `version: 9.15.4` from the pnpm steps in this file. Whichever PR merges second needs a small rebase. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), by Anthropic, in Claude Code with tool use (shell, file edits, git, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-07 06:16:06 -10:00 · 2026-11-06 11:50:42 -08:00
1 parent e8f07b2a39
commit 79c724b292
@@ -284,5 +284,10 @@
      "shard_label": "8/8",
      "task": "ci:e2e"
    }
+  ],
+  "verify_runner_e2e_support": [
+    {
+      "task": "ci:runner-e2e-support"
+    }
  ]
 }
@@ -33,7 +33,9 @@ test("a UI component retains browser and build coverage without unrelated server
  assert.equal(selected(result, "build"), true);
  assert.equal(selected(result, "e2e_shards"), true);
  assert.equal(selected(result, "verify_serialized_server"), false);
-  assert.deepEqual([...selectedTasks(result)].filter((task) => /server|chat|runner/.test(task)), []);
+  // The Runner support suite imports UI modules and checks that catalog-listed
+  // UI files stay tracked, so UI source changes are its inputs too.
+  assert.deepEqual([...selectedTasks(result)].filter((task) => /server|chat|runner/.test(task)), ["ci:runner-e2e-support"]);
  assert.equal(result.jobs.general_tests.selected, 2);
 });

@@ -330,10 +330,10 @@ test("workflow exporter runs trusted code and keeps keys out of the PR jobs", ()
  assertNoPrWorkflowCredentials(pr);
  assert.match(pr, /if: always\(\) && env.PAPERCLIP_CI_PROFILE == 'true'/);
  assert.doesNotMatch(pr, /PAPERCLIP_CI_PROFILE:.*labels/);
-  assert.equal(pr.match(/PAPERCLIP_CI_PROFILE: \$\{\{ needs.gate.outputs.profile_ci \}\}/g).length, 10);
+  assert.equal(pr.match(/PAPERCLIP_CI_PROFILE: \$\{\{ needs.gate.outputs.profile_ci \}\}/g).length, 11);
  // upload-artifact excludes files inside dot-directories by default. Keep
  // this opt-in scoped to timing NDJSON, never the workspace or other secrets.
-  assert.equal(pr.match(/path: \.ci-profile\/\*\.ndjson\n {10}include-hidden-files: true/g).length, 10);
+  assert.equal(pr.match(/path: \.ci-profile\/\*\.ndjson\n {10}include-hidden-files: true/g).length, 11);
  const routing = readFileSync(new URL("../../workflows/ci-routing.yml", import.meta.url), "utf8");
  assert.match(routing, /PROFILE_CI: \$\{\{ steps.profile.outputs.profile_ci \}\}/);
  assert.match(routing, /\/git\/ref\/heads\/master/);
@@ -727,10 +727,71 @@ jobs:
          if-no-files-found: warn
          retention-days: 14

+  verify_runner_e2e_support:
+    name: Runner support suite
+    needs: [gate]
+    if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.verify_runner_e2e_support.run) }}
+    runs-on: ${{ needs.gate.outputs.runner }}
+    timeout-minutes: 20
+
+    env:
+      PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
+      PAPERCLIP_CI_PROFILE_JOB: Runner support suite
+
+    # The paid Runner workflows depend on this suite: workflow security
+    # policy, evidence oracles, cost receipts and controller recovery. It
+    # used to run only in a manual Docker Runner check, so master changes
+    # broke it unseen (#19527, #19529). It needs no provider credentials.
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+        with:
+          persist-credentials: false
+
+      - name: Initialize CI profile
+        continue-on-error: true
+        if: env.PAPERCLIP_CI_PROFILE == 'true'
+        run: node scripts/ci-profile/initialize.mjs
+
+      - name: Setup Node.js for pnpm bootstrap
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
+        with:
+          node-version: 24
+          package-manager-cache: false
+
+      - name: Setup pnpm
+        uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
+        env:
+          NPM_CONFIG_AUDIT: "false"
+          NPM_CONFIG_FUND: "false"
+          NPM_CONFIG_UPDATE_NOTIFIER: "false"
+        with:
+          version: 9.15.4
+
+      # Share the checked-in lockfile key with master. PR merge refs must not
+      # save full copies of the store or evict the post-merge build caches.
+      - name: Restore materialized dependencies and shared builds
+        uses: ./.github/actions/setup-ci-dependencies
+
+      # Builds the plugin SDK first, then runs Vitest and the node:test files.
+      - name: Run the Runner support suite
+        run: pnpm test:e2e:runner:unit
+
+      - name: Upload CI profile timings
+        continue-on-error: true
+        if: always() && env.PAPERCLIP_CI_PROFILE == 'true'
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+        with:
+          name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
+          path: .ci-profile/*.ndjson
+          include-hidden-files: true
+          if-no-files-found: warn
+          retention-days: 14
+
  verify:
    name: verify
    if: ${{ always() }}
-    needs: [gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, sandbox_provider_tests, verify_serialized_server, canary_dry_run]
+    needs: [gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, sandbox_provider_tests, verify_serialized_server, canary_dry_run, verify_runner_e2e_support]
    runs-on: ${{ needs.gate.outputs.runner }}
    timeout-minutes: 5
    steps:
@@ -749,7 +810,7 @@ jobs:
          const plan = process.env.CI_PLAN ? JSON.parse(process.env.CI_PLAN) : null;
          if (plan) assert.equal(plan.version, 1, 'Supported task plan required');
          if (plan?.testSelection) assert.equal(results.test_selection.result, 'success', 'Published selection must succeed');
-          for (const job of ["typecheck_release_registry","general_tests","verify_paperclip_runner","build","docker_context_integrity","sandbox_provider_tests","verify_serialized_server","canary_dry_run"]) {
+          for (const job of ["typecheck_release_registry","general_tests","verify_paperclip_runner","build","docker_context_integrity","sandbox_provider_tests","verify_serialized_server","canary_dry_run","verify_runner_e2e_support"]) {
            if (plan) assert.equal(typeof plan.jobs[job]?.run, 'boolean', 'Missing suite decision: ' + job);
            const selected = process.env.FULL_CI === 'true' && (!plan || plan.jobs[job].run);
            assert.equal(results[job].result, selected ? 'success' : 'skipped', job);
@@ -246,6 +246,7 @@ test("the trusted PR workflow limits full CI to merge-relevant stack layers", ()
    "verify_serialized_server",
    "canary_dry_run",
    "e2e_shards",
+    "verify_runner_e2e_support",
  ]) {
    assert.match(
      jobs.get(jobId),
@@ -263,7 +264,7 @@ test("the trusted PR workflow limits full CI to merge-relevant stack layers", ()
  const verify = jobs.get("verify");
  assert.match(
    verify,
-    /^ {4}needs: \[gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, sandbox_provider_tests, verify_serialized_server, canary_dry_run\]$/m,
+    /^ {4}needs: \[gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, sandbox_provider_tests, verify_serialized_server, canary_dry_run, verify_runner_e2e_support\]$/m,
  );
  assert.equal(runAggregate(verify, true).status, 0);
  assert.equal(runAggregate(verify, false).status, 0);
@@ -272,7 +273,7 @@ test("the trusted PR workflow limits full CI to merge-relevant stack layers", ()
      "an optional selector cannot block the successful original matrix");
  }
  for (const job of ["gate", "policy", "typecheck_release_registry", "general_tests", "verify_paperclip_runner",
-    "build", "docker_context_integrity", "sandbox_provider_tests", "verify_serialized_server", "canary_dry_run"]) {
+    "build", "docker_context_integrity", "sandbox_provider_tests", "verify_serialized_server", "canary_dry_run", "verify_runner_e2e_support"]) {
    for (const result of ["failure", "cancelled", "skipped"]) {
      assert.notEqual(runAggregate(verify, true, { [job]: result }).status, 0, `${job}: ${result}`);
    }
@@ -1184,6 +1184,13 @@ unavailable/terminated test process produce a `launcher/environment failure`
 diagnostic and exit 2. Assertion failures retain the test runner's exit status
 and output; assess installed-product failures from those assertions separately.

+PR CI runs `pnpm test:e2e:runner:unit` as the required "Runner support suite"
+job in `pr-trusted.yml`. It runs when server, CLI, package or UI source
+changes, and on every full CI run. The suite pins workflow security policy,
+evidence oracles, profile versions, suite hashes and dated fixtures. When a
+change to one of those is deliberate, update the pin in the same pull request
+and name the reason. Do not weaken a security check to make it pass.
+
 The OpenRouter ranking snapshot is tracked in `openrouter-models.json`; nightly
 runs never mutate it. Refresh it deliberately, review the source/capture/hash
 diff, and rerun credential-free checks:
@@ -13,6 +13,8 @@ if (!existsSync(vitest)) {
  console.error(`${prefix} launcher/environment failure: Vitest is not installed. Run pnpm install before validation.`);
  process.exit(2);
 }
+// Load after the check: a missing install must still report the launcher failure.
+const { invocationEnv } = await import(new URL("../../scripts/ci-profile/record.mjs", import.meta.url).href);

 const env = { ...process.env };
 delete env.NODE_PATH;
@@ -22,13 +24,16 @@ for (const [name, command] of [
  // Server modules under test import the built plugin SDK (and shared) package.
  ["Plugin build dependencies", ["scripts/ensure-plugin-build-deps.mjs"]],
  ["Vitest", [vitest, "run", "--config", "tests/runner-e2e/vitest.config.ts", ...args]],
-  ["Node tests", ["--test", "tests/runner-e2e/product-regression-testops.test.mjs", "tests/runner-e2e/artifact-budget.test.mjs", "tests/runner-e2e/native-completion-checks.test.mjs",
+  // The wrapper adds the CI profiling reporter when PAPERCLIP_CI_PROFILE is set.
+  ["Node tests", ["scripts/ci-profile/node-test.mjs", "tests/runner-e2e/product-regression-testops.test.mjs", "tests/runner-e2e/artifact-budget.test.mjs", "tests/runner-e2e/native-completion-checks.test.mjs",
    "tests/runner-e2e/native-completion-git-source.test.mjs", "tests/runner-e2e/native-completion-source-contract.test.mjs",
    "tests/runner-e2e/chat-feature-testops-api.test.mjs", "tests/runner-e2e/chat-feature-testops-cli.test.mjs", "tests/runner-e2e/chat-feature-testops-results.test.mjs", "tests/runner-e2e/testops-chat-reporter.test.mjs", "tests/runner-e2e/chat-feature-testops-retained-verification.node-test.mjs",
    "tests/runner-e2e/chat-completion-testops-api.test.mjs", "tests/runner-e2e/chat-completion-testops-contract.test.mjs",
    "tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs", "tests/runner-e2e/chat-full-catalog-testops-reconcile.test.mjs", "tests/runner-e2e/chat-full-catalog-testops-adapter.test.mjs", "tests/runner-e2e/chat-catalog-testops-streaming.test.mjs"]],
 ]) {
-  const result = spawnSync(process.execPath, command, { cwd: root, env, stdio: "inherit" });
+  const result = spawnSync(process.execPath, command, {
+    cwd: root, env: name === "Vitest" ? invocationEnv("vitest", env) : env, stdio: "inherit",
+  });
  if (result.error || result.signal) {
    console.error(`${prefix} launcher/environment failure: ${name} ${result.error?.message ?? `terminated by ${result.signal}`}`);
    process.exit(2);
@@ -1,9 +1,14 @@
+import { fileURLToPath } from "node:url";
 import { configDefaults, defineConfig } from "vitest/config";

 export default defineConfig({
  root: import.meta.dirname,
  test: {
    name: "runner-e2e-support",
+    // CI profiling records per-test timings, like the root and Runner package configs.
+    ...(process.env.PAPERCLIP_CI_PROFILE === "true"
+      ? { reporters: [...configDefaults.reporters, fileURLToPath(new URL("../../scripts/ci-profile/vitest-reporter.mjs", import.meta.url))] }
+      : {}),
    globalSetup: ["./verified-provider-bundles.setup.mjs"],
    include: ["**/*.test.ts", "**/*.test.mjs"],
    // These use node:test and run separately in test:e2e:runner:unit.
@@ -83,10 +83,10 @@ describe("public repository paid workflow security", () => {
  });

  // PR #18348 added the ninth pr-trusted bootstrap, in the Vitest selection job.
-  // #19533 added the tenth, for the sandbox provider tests.
+  // #19533 added the tenth (sandbox provider tests); the eleventh runs this suite.
  const trustedPnpmBootstraps = {
    "runner-full-stack-e2e.yml": 8,
-    "pr-trusted.yml": 10,
+    "pr-trusted.yml": 11,
  } as const;

  it("keeps pnpm bootstrap registry telemetry out of trusted workflow setup", async () => {
@@ -234,6 +234,16 @@
      ],
      "outputs": [],
      "cache": false
+    },
+    "//#ci:runner-e2e-support": {
+      "inputs": [
+        "server/**",
+        "cli/**",
+        "packages/**",
+        "ui/**"
+      ],
+      "outputs": [],
+      "cache": false
    }
  }
 }