diff --git a/.claude/plugins/n8n/skills/mutate-changed/SKILL.md b/.claude/plugins/n8n/skills/mutate-changed/SKILL.md new file mode 100644 index 00000000000..dde0c1a1363 --- /dev/null +++ b/.claude/plugins/n8n/skills/mutate-changed/SKILL.md @@ -0,0 +1,124 @@ +--- +description: Run Stryker mutation testing on the source files changed in the current branch (vs origin/master). One command for "did my work hold up under mutation?" before pushing. Triages on the side which files dropped below threshold and offers to invoke n8n:strengthen-tests on them. Use when the user says /mutate-changed, "mutate what I changed", "check my changes", or has just finished writing a feature and wants pre-merge feedback. Scope: only packages/workflow/src/** changes are mutated today. +--- + +# Mutate what I changed + +Closes the local dev loop. Single command to run Stryker against every source file the current branch touched (vs `origin/master`), then point at any reds that need strengthening. + +## When to use + +- User says `/mutate-changed`, "mutate the files I changed", "check my changes", "did my tests stick" +- Mid-feature: dev wants pre-merge feedback before pushing +- Pre-PR: cheaper than waiting for the nightly cron + +**Don't** use: +- For a single specific file (`/n8n:mutation-test ` is faster) +- For non-`packages/workflow` changes — Stryker is only wired up there today +- After the user already ran `/n8n:strengthen-tests` (which calls mutation-test internally for verification — running both again is wasted compute) + +## Inputs + +- **Default base**: `origin/master`. Override with `--base ` if comparing against another branch (e.g. `--base HEAD~5`). +- **Default scope**: `packages/workflow/src/**/*.ts`. The only package with Stryker wired up today. + +## Steps + +### 1. Identify changed source files + +```bash +git diff --name-only origin/master...HEAD -- 'packages/workflow/src/**/*.ts' +``` + +(`...` is correct — three-dot means "since the branch diverged from base," which is what we want.) + +If `git fetch` hasn't been run recently, suggest the user `git fetch origin master` first; otherwise the base ref is stale. + +Filter out: +- `**/*.d.ts` (declarations, no behaviour) +- `**/*.stories.ts` (Storybook scaffolding, not present in workflow but defensive) +- `index.ts` files (barrels) +- `interfaces.ts`, `types.ts`, `constants.ts` (same low-value filter as `seed-ledger.mjs`) + +### 2. Surface the plan to the user + +Print the filtered list before running anything. Each Stryker run is 1–5 minutes; the user should confirm if there are many. + +``` +Found N changed source files to mutate: + - packages/workflow/src/foo.ts + - packages/workflow/src/bar.ts + ... + +Estimated runtime: ~M-K minutes (M minutes minimum if every Stryker run is fast). +Proceed? (skill default: yes if N ≤ 3, ask if N > 3) +``` + +If the filtered list is empty: report "No source files under packages/workflow/src/** changed vs $base — nothing to mutate." and stop. Exit cleanly. + +If N > 8: refuse and ask the user to narrow scope (a different base ref, or invoke per-file). Running 8+ mutations sequentially is a 30+ minute session that should be a deliberate choice. + +### 3. Run mutation testing per file + +For each file in the plan, invoke `pnpm --filter=n8n-workflow mutate `. The `summary.json` and other artefacts get overwritten on each run, so capture the score per file as you go. + +After each run completes, print one line: + +``` +✓ src/foo.ts 95.12% (39/41 killed) GREEN +✗ src/bar.ts 54.83% (17/31 killed) RED — 13 survivors, top: ConditionalExpression, EqualityOperator +``` + +If a Stryker run hard-fails (exit 3, no `summary.json`), print `! src/foo.ts Stryker failed — see stderr` and continue to the next file. Don't abort the whole batch. + +### 4. Summary table + +After all files have been mutated, print one compact table: + +``` +=== Mutation results: N files, M green, K red, J failed === +| File | Score | Verdict | Survivors | +|-----------------------------|---------|---------|-----------| +| src/foo.ts | 95.12% | GREEN | 2 | +| src/bar.ts | 54.83% | RED | 13 | +| src/baz.ts | n/a | FAILED | - | +``` + +### 5. Offer the strengthen step on the worst red file + +If any file came back red: + +> The lowest-score red file is `src/bar.ts` (54.83%, 13 survivors). Run `/n8n:strengthen-tests` to triage them and write assertion changes? (suggesting; don't auto-invoke) + +Only suggest one file at a time — `n8n:strengthen-tests` caps at 5 survivors per invocation, and re-running this skill after edits is cheap. + +If everything is green: report it and stop. No follow-up needed. + +## Output shape + +Three deliverable sections per invocation: + +1. **Plan** (before running) — list of files, estimated runtime +2. **Per-file progress** (during) — one line per file as it completes +3. **Summary table + recommendation** (after) — compact view + +Don't dump full `summary.json` payloads — the per-file mutate runs already write them to disk under `packages/workflow/reports/mutation/` (overwriting each time, since the orchestrator uses fixed filenames). The user can read the latest one if they want detail. + +## Constraints + +- **Hardcoded to `packages/workflow`.** Generalise when Stryker is wired up to other packages. +- **Max 8 files per invocation.** Above that, ask user to narrow. +- **Don't auto-invoke `/n8n:strengthen-tests`.** Suggest, don't act. Same reasoning as the other skills: each pass should be a deliberate human-approved step. +- **No commits.** Edits land in working tree; user reviews. +- **No fabricated scores.** If a Stryker run fails, mark FAILED in the table — never guess a value. + +## Common follow-ups + +- "strengthen them all" → loop the user through `/n8n:strengthen-tests`, one file at a time +- "what changed?" → `git diff origin/master...HEAD -- ` for the file in question +- "ignore " → re-run with the user's exclude added to the filter + +## Related + +- `n8n:mutation-test` — single-file version of this skill +- `n8n:strengthen-tests` — the natural next step when reds show up diff --git a/.claude/plugins/n8n/skills/mutation-test/SKILL.md b/.claude/plugins/n8n/skills/mutation-test/SKILL.md new file mode 100644 index 00000000000..aed1e385884 --- /dev/null +++ b/.claude/plugins/n8n/skills/mutation-test/SKILL.md @@ -0,0 +1,115 @@ +--- +description: Run Stryker mutation testing on a single source file and return a structured, token-frugal report that's pipeable to a follow-up "strengthen tests" loop. Use when the user says /mutation-test, "mutation test this file", or has just edited tests and wants to verify they actually assert behaviour. Per-file only — full-package mutation runs are out of scope. +--- + +# Mutation testing — single file + +Wraps `pnpm --filter= mutate ` and parses `summary.json` into a compact, structured shape suitable for downstream "strengthen the surviving mutants" iteration. + +## When to use + +- User explicitly invokes: `/mutation-test `, "mutation test this file", "check my test effectiveness on X" +- User has just edited a test file and wants to know if their assertions are load-bearing +- Follow-up loop after a `red` verdict — feed the structured output back to a "fix" iteration + +**Don't** use this skill for: +- Whole-package or whole-repo mutation runs — single file only +- Coverage % questions (use the existing coverage workflow) +- Files outside `packages/workflow/` — Stryker is only wired up there today + +## Inputs + +One required argument: the source file to mutate, as either a repo-relative path or a package-relative path. Examples that all mean the same thing: + +- `packages/workflow/src/cron.ts` +- `src/cron.ts` (assumes packages/workflow) +- `workflow/src/cron.ts` (assumes packages/) + +If ambiguous, ask the user once which package; do not guess. + +## Steps + +1. **Resolve package + relative source path.** Today only `n8n-workflow` (`packages/workflow`) has Stryker wired. If the user passes a file outside that, say so and stop — don't fabricate output. + +2. **Run Stryker with trimmed output:** + ```bash + pnpm --filter=n8n-workflow mutate 2>&1 | tail -40 + ``` + `tail -40` discards the Stryker progress bar spam; the relevant numbers + survivor list always land in the last ~30 lines. Exit codes: `0` = pass, `1` = below threshold (still valid, summary.json exists), `2` = usage error, `3` = Stryker failure (no summary.json). + +3. **If exit code 3**, surface the trimmed tail to the user, suggest checking that workspace deps are built (`pnpm build`), and stop. Don't fabricate a report. + +4. **Read `packages/workflow/reports/mutation/summary.json`** — never `raw.json`. raw.json is 600KB+ and not needed for the strengthen loop. summary.json already contains every surviving mutant with its location, replacement, mutator name, and the names of tests that covered the line. + +5. **Cap covering_tests at 3 per survivor.** If a mutant was covered by more than 3 tests, keep the first 3 and append `+N more` as a count. Names beyond 3 add tokens without adding actionable signal — the strengthen loop only needs to know *which test* to extend, not all of them. + +6. **Compute `minimum_kills_needed`** to reach the threshold: + ``` + killed_now = summary.overall.counts.killed + summary.overall.counts.timeout + valid_total = killed_now + summary.overall.counts.survived + summary.overall.counts.noCoverage + needed = ceil((threshold/100) * valid_total) - killed_now + ``` + This tells the next loop the minimum number of survivors it has to kill to flip `red` → `green`. Cap at the number of survivors. + +7. **Output the structured shape** described below. Keep prose to one headline line; the rest is the JSON block. + +## Output shape + +One headline line, then a fenced JSON block. Nothing else — no preamble, no per-survivor commentary, no risk triage (that's the next loop's job). + +```` +[red|green] % (threshold %) — survivors; need to kill ≥ to flip green. + +```json +{ + "verdict": "red", + "target": "packages/workflow/src/augment-object.ts", + "package": "n8n-workflow", + "score": 76.74, + "threshold": 80, + "delta_to_threshold": 3.26, + "minimum_kills_needed": 5, + "counts": { + "killed": 99, + "survived": 28, + "no_coverage": 2, + "timeout": 0 + }, + "survivors": [ + { + "id": "77", + "mutator": "ConditionalExpression", + "location": "src/augment-object.ts:95:6", + "original": "value === null", + "replacement": "false", + "covering_tests": [ + "augmentObject should handle null values", + "augmentObject should handle nested nulls" + ], + "covering_tests_overflow": 0 + } + ] +} +``` +```` + +Order the survivors array by `location` (ascending line number, then column) so the strengthen loop processes them top-to-bottom of the file. + +## Constraints + +- **No raw.json** — never read or surface it. summary.json is the only input. +- **No HTML report** — don't `open` raw.html or paste links to it. If the user wants visual exploration they'll ask. +- **No automatic triage** — don't categorise survivors by "real bug" vs "refactor insurance." That's a separate analysis step that should happen on demand, not by default. Keeps token cost predictable. +- **No "I'll regenerate tests for you now"** — this skill reports the gap. Use `n8n:strengthen-tests` if you want assertion edits. + +## Common follow-ups (don't do unless asked) + +- User says "fix these" → start a strengthen loop using the JSON output as input. Read covering_tests source, propose changes per mutant, run the skill again to verify. +- User says "explain survivor #N" → fetch that mutant from summary.json, show its surrounding ~5 lines from the source file, no analysis beyond what summary.json contains. +- User says "what's the threshold?" → 80% provisional; see `scripts/mutation-health/README.md` for the rationale. +- User says "run it on the changed files" → not wired yet. Suggest `git diff` to find candidates, then invoke this skill per file. + +## Related + +- `scripts/mutation-health/README.md` — the broader BQ-backed observability story +- `packages/workflow/stryker.config.mjs` — the Stryker config this skill drives diff --git a/.claude/plugins/n8n/skills/strengthen-tests/SKILL.md b/.claude/plugins/n8n/skills/strengthen-tests/SKILL.md new file mode 100644 index 00000000000..9772d16b130 --- /dev/null +++ b/.claude/plugins/n8n/skills/strengthen-tests/SKILL.md @@ -0,0 +1,147 @@ +--- +description: Take a Stryker summary.json (from n8n:mutation-test), triage the surviving mutants by user-reachable-behaviour risk, write minimal assertion changes to kill the top 3-5 highest-leverage survivors, then verify by re-running n8n:mutation-test. Use when the user has just run mutation testing and wants to strengthen the test suite, or says "kill the survivors / strengthen tests / fix the red." Pairs with n8n:mutation-test as the inner write side of a single iteration. +--- + +# Strengthen tests — kill the highest-leverage survivors + +The other half of the local mutation-testing loop. `n8n:mutation-test` reports which mutations escaped the tests; this skill picks the ones that matter and writes minimal assertion changes to kill them. + +## When to use + +- User has just run `/n8n:mutation-test ` and the verdict was `red` +- User says: "strengthen tests", "kill the survivors", "fix the red", "iterate on the tests for X" +- Mid-loop: this skill's verify step calls `n8n:mutation-test` again, so the loop closes here + +**Don't** use this skill: +- Before any mutation testing has been run for the target file (no `summary.json` to triage) +- For a `green` verdict — there's nothing to strengthen; if user insists, push back and ask which file actually needs work +- To bulk-kill every survivor — explicitly capped at 5 per invocation. Re-invoke for more. + +## Inputs + +- **Default**: read `packages/workflow/reports/mutation/summary.json` (the last `n8n:mutation-test` run's output). +- **Override**: `--summary ` to point at a different summary file. + +## Steps + +### 1. Read the summary + +`packages/workflow/reports/mutation/summary.json`. Already compact (~50 KB). Pull: +- `files[0].file` — the source file under test +- `files[0].score` — current mutation score +- `files[0].survivors[]` — every surviving (and no-coverage) mutant with location, replacement, covering test names + +If `summary.json` is missing, stop. Tell the user to run `n8n:mutation-test` first. + +### 2. Read the source under test, sparingly + +Read the source file referenced in `summary.json`. Read **once**, the whole file (typical n8n-workflow source files are 50-500 lines; the cost is bounded). This is the only file read; don't load test files yet. + +### 3. Triage the survivors + +Categorise each survivor into one of three buckets. Use the rubric below — don't apply it mechanically, but lean on it. + +**HIGH leverage — "real regression vector":** mutant guards behaviour that real users actually hit through the public API surface. + +- Type checks against shapes user data routinely takes: `null`, `undefined`, `Date`, `Buffer`, `Uint8Array`, `Array`, plain objects with arbitrary keys +- Conditional branches that gate critical fall-through (e.g. "if this returns early, the wrong code path runs") +- Code paths that handle user-controlled flow: expressions, Code-node behaviour, binary item handling, deep-clone semantics +- Mutants on guard clauses that prevent crashes (`if (value == null) return ...`) + +**MODERATE leverage — "user-observable invariant":** real but lower-frequency user impact. + +- Proxy traps that change `Object.keys` / `in` / `hasOwnProperty` semantics after mutation +- Set-then-delete / re-set-after-delete sequences (Code-node assignment patterns) +- Treating `undefined` assignment as deletion (`obj.foo = undefined` → key removed) +- Edge cases that only fire under specific iteration patterns + +**LOW leverage — "refactor insurance" or noise:** skip these. Document the skip in the output but don't write tests for them. + +- Constructor property checks (`arr.constructor === Array`) unless production code is known to rely on them +- Idempotency invariants only triggered by other library code (Lodash, native spread) +- Equivalent mutants — mutations that produce semantically identical code (e.g. swapping an unused conditional) +- Mutants on internal helpers users can't reach through any public API + +If the summary lists `noCoverage` survivors, treat them as their own bucket: "the test suite doesn't even execute these lines." Triage them by the same rubric, but flag separately since they need a new test case rather than an assertion extension. + +### 4. Pick the work set: up to 5 mutants + +Order: all HIGH first, then MODERATE if budget remains, never LOW. Hard cap at 5 total. If HIGH alone exceeds 5, pick the 5 most distinct (don't pick 5 mutants on the same line — they probably share a fix; pick representatives across the file). + +If fewer than 3 HIGH+MODERATE candidates exist, just do what's there. Don't pad with LOW just to hit a number. + +Write up the work set to the user **before editing**: + +``` +Picked N survivors to address (M skipped as refactor-insurance / low-leverage): + 1. [HIGH] location — original → replacement + plan: assert in + 2. [HIGH] ... + 3. [MODERATE] ... + ... +``` + +This is the user's chance to redirect. Don't write code yet. + +### 5. Read covering tests for the picked survivors + +For each picked survivor, the summary lists the test names that covered the line. Find those tests in the test file (usually `packages//test/.test.ts`, but check) and read **just the relevant `test('...')` blocks** — not the whole file. Use `Grep` + `Read` with line offsets to keep token cost down. + +Goal: understand what the existing test asserts so the new assertion is additive, not contradictory. + +### 6. Write the changes + +Constraints: +- **Prefer extending an existing covering test** over adding a new one. Lower file churn, easier review. +- **Match the existing style** — same assertion library, same matcher idioms (`.toBe` vs `.toEqual` vs `.toStrictEqual`). +- **Minimal additions** — one or two assertions per mutant, not a new it-block per mutant. +- **No fabrication** — only assert what the source code actually does. If you can't tell from the source, stop and ask the user. +- **For `noCoverage` survivors**, add a new test case named after the behaviour being pinned. Place it next to related tests. + +Use `Edit` with exact-string matches. Never rewrite entire test files. + +### 7. Verify + +Re-invoke `n8n:mutation-test` on the same source file. Report: + +``` +Before: red 76.74% (28 survivors) +After: green 82.34% (22 survivors) +Killed: 6 of 5 targeted (1 bonus — fix for #77 also killed #78) +Still surviving: 22 — re-invoke /n8n:strengthen-tests for another batch. +``` + +If the score went UP but threshold still not met: the iteration is working, recommend another pass. +If the score went DOWN or stayed the same: at least one new test isn't asserting what we think it asserts. Surface the diff to the user and stop — do not auto-revert. +If a test now fails (not survives — actually fails): we asserted something the code doesn't do. Revert that specific assertion, leave the rest, report which one was wrong. + +## Output shape + +Each invocation produces: + +1. **Work plan** (before edits) — the picked survivors and the plan for each +2. **Diffs** (during edits) — Edit tool calls, visible in transcript +3. **Verify** (after) — re-run + before/after comparison + +Keep prose minimal between sections. The plan and verify steps are the structured outputs; everything else is mechanical. + +## Constraints + +- **5 mutants max per invocation.** Re-invoke for more. Prevents runaway sessions on 30-survivor files. +- **Never fabricate assertions.** If the source doesn't clearly do X, don't claim it does. +- **No new test files unless absolutely necessary.** Extend the existing covering test file. +- **No reverting other people's tests.** Only edit tests in the package being mutated. +- **No re-running mutation-test more than once per invocation.** That's the verify step. Don't loop within a single invocation; let the user re-invoke. +- **No commits.** Edits land in the working tree; user reviews and commits. + +## Common follow-ups + +- User says "go again" → re-invoke this skill. The summary.json now reflects the post-edit state. +- User says "why was #N classified as LOW?" → explain the rubric application for that specific mutant, no re-triage of others. +- User says "kill #N specifically" → override the triage for that mutant, treat it as picked. +- User says "skip the verify step" → don't; the verify step is the contract that the edits actually moved the score. + +## Related + +- `n8n:mutation-test` — the read side of this loop +- `scripts/mutation-health/README.md` — the BQ-backed observability story this slots into diff --git a/.github/workflows/mutation-health-nightly.yml b/.github/workflows/mutation-health-nightly.yml new file mode 100644 index 00000000000..7d06c5dffbb --- /dev/null +++ b/.github/workflows/mutation-health-nightly.yml @@ -0,0 +1,118 @@ +name: 'Mutation Health (nightly)' + +on: + schedule: + # 03:30 UTC daily — outside CI rush, before EU morning. + - cron: '30 3 * * *' + workflow_dispatch: + inputs: + package: + description: 'Workspace package to mutate' + type: choice + options: + - n8n-workflow + default: n8n-workflow + +permissions: + contents: read + +# Prevent overlapping scheduled + manual runs from racing on the same +# ledger row. The writer MERGE is idempotent, but two concurrent runs +# would emit duplicate event rows for the same picked file. +concurrency: + group: mutation-health-${{ github.event.inputs.package || 'n8n-workflow' }} + cancel-in-progress: false + +env: + PACKAGE_NAME: ${{ github.event.inputs.package || 'n8n-workflow' }} + PKG_DIR: packages/workflow # 1-to-1 mapping today; generalise when more packages are wired up + REPORTS_DIR: packages/workflow/reports/mutation + READER_URL: https://internal.users.n8n.cloud/webhook/mutation-health-ledger + +jobs: + run: + name: Mutate one file, ledger writeback + runs-on: blacksmith-4vcpu-ubuntu-2204 + timeout-minutes: 60 + steps: + - name: Checkout + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + # Shallow clone is fine — pick-next no longer reads git history. + # Staleness is purely age-based via last_checked_at. + + - name: Setup Environment + uses: ./.github/actions/setup-nodejs + + - name: Fetch live ledger from BigQuery + run: | + mkdir -p "$REPORTS_DIR" + curl --fail -sS "$READER_URL?package=$PACKAGE_NAME" -o "$REPORTS_DIR/live-ledger.json" + + - name: Pick next source file + id: pick + run: | + picked_json=$(node scripts/mutation-health/pick-next.mjs \ + --package-dir "$PKG_DIR" \ + --ledger-file "$REPORTS_DIR/live-ledger.json") + echo "$picked_json" + src_repo=$(echo "$picked_json" | jq -r '.picked.source_file_path // ""') + if [ -z "$src_repo" ]; then + echo "::notice::Picker returned no work (all-green / empty ledger). Exiting cleanly." + echo "skip=true" >> "$GITHUB_OUTPUT" + exit 0 + fi + src_rel="${src_repo#"$PKG_DIR"/}" + echo "skip=false" >> "$GITHUB_OUTPUT" + echo "source-rel=$src_rel" >> "$GITHUB_OUTPUT" + + - name: Mutate the picked source file + if: steps.pick.outputs.skip != 'true' + # Exit code semantics from mutate.mjs: + # 0 — score >= threshold (green) + # 1 — score < threshold (red) — emit step still runs, this is normal + # 2 — usage error + # 3 — Stryker hard failure (no summary.json) — must fail the job + # We capture rc explicitly so we can distinguish "below threshold" from + # "Stryker crashed." continue-on-error: true would collapse them. + run: | + set +e + pnpm --filter "$PACKAGE_NAME" mutate "${{ steps.pick.outputs.source-rel }}" + rc=$? + set -e + if [ "$rc" -gt 1 ]; then + echo "::error::Stryker hard-failed with exit code $rc." + exit "$rc" + fi + echo "Mutate exit $rc ($([ "$rc" = "0" ] && echo green || echo red))." + + - name: Emit BQ payload + if: steps.pick.outputs.skip != 'true' + run: node scripts/mutation-health/emit-payload.mjs --summary "$REPORTS_DIR/summary.json" --package "$PACKAGE_NAME" + + - name: POST result payload + if: steps.pick.outputs.skip != 'true' + env: + MUTATION_HEALTH_WEBHOOK: ${{ secrets.MUTATION_HEALTH_WEBHOOK }} + run: | + if [ -z "$MUTATION_HEALTH_WEBHOOK" ]; then + echo "::notice::MUTATION_HEALTH_WEBHOOK not set — dry-run, POST skipped (payload uploaded as artefact)." + exit 0 + fi + curl --fail -sS -X POST \ + -H 'Content-Type: application/json' \ + --data @"$REPORTS_DIR/bq-payload.json" \ + "$MUTATION_HEALTH_WEBHOOK" + + - name: Upload artefacts + if: always() + uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0 + with: + name: mutation-health-${{ env.PACKAGE_NAME }}-${{ github.run_id }} + path: | + ${{ env.REPORTS_DIR }}/raw.json + ${{ env.REPORTS_DIR }}/raw.html + ${{ env.REPORTS_DIR }}/summary.json + ${{ env.REPORTS_DIR }}/bq-payload.json + ${{ env.REPORTS_DIR }}/live-ledger.json + retention-days: 14 + if-no-files-found: warn diff --git a/.gitignore b/.gitignore index 6960719dd6d..fff5145a663 100644 --- a/.gitignore +++ b/.gitignore @@ -20,6 +20,8 @@ nodelinter.config.json !.github/scripts/package-lock.json packages/**/.turbo .turbo +.stryker-tmp/ +**/reports/mutation/ *.tsbuildinfo .stylelintcache *.swp @@ -72,3 +74,4 @@ packages/cli/src/commands/export/outputs .n8n lefthook-local.yml .playwright-mcp +stryker.log diff --git a/packages/workflow/package.json b/packages/workflow/package.json index 19f153d6d56..aa6356670d7 100644 --- a/packages/workflow/package.json +++ b/packages/workflow/package.json @@ -31,7 +31,8 @@ "watch": "tsc --build tsconfig.build.esm.json tsconfig.build.cjs.json --watch", "test": "vitest run", "test:unit": "vitest run", - "test:dev": "vitest --watch" + "test:dev": "vitest --watch", + "mutate": "node scripts/mutate.mjs" }, "files": [ "dist/**/*" @@ -41,6 +42,8 @@ "@n8n/config": "workspace:*", "@n8n/typescript-config": "workspace:*", "@n8n/vitest-config": "workspace:*", + "@stryker-mutator/core": "catalog:", + "@stryker-mutator/vitest-runner": "catalog:", "@types/express": "catalog:", "@types/jmespath": "^0.15.0", "@types/lodash": "catalog:", diff --git a/packages/workflow/scripts/mutate.mjs b/packages/workflow/scripts/mutate.mjs new file mode 100644 index 00000000000..cd35a6dcc09 --- /dev/null +++ b/packages/workflow/scripts/mutate.mjs @@ -0,0 +1,213 @@ +#!/usr/bin/env node +/** + * Run Stryker on a single source file and emit an actionable summary. + * + * Usage: pnpm --filter=n8n-workflow mutate + * Example: pnpm --filter=n8n-workflow mutate src/cron.ts + * + * Outputs (under packages/workflow/reports/mutation/): + * raw.json — full Stryker Mutation Testing Elements report + * raw.html — Stryker's HTML report (browse for human review) + * summary.json — compact actionable summary (this script) + * + * Exit codes: + * 0 — mutation score ≥ threshold + * 1 — score < threshold (AI loop should iterate) + * 2 — usage / config error + * 3 — Stryker run failed + */ + +import { spawn } from 'node:child_process'; +import { readFile, writeFile } from 'node:fs/promises'; +import { existsSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import path from 'node:path'; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const pkgRoot = path.resolve(__dirname, '..'); + +const THRESHOLD = Number(process.env.STRYKER_THRESHOLD ?? 80); + +function die(code, msg) { + process.stderr.write(`${msg}\n`); + process.exit(code); +} + +const targetArg = process.argv[2]; +if (!targetArg) { + die( + 2, + 'Usage: pnpm --filter=n8n-workflow mutate \n' + + 'Example: pnpm --filter=n8n-workflow mutate src/cron.ts', + ); +} + +const target = path.isAbsolute(targetArg) ? path.relative(pkgRoot, targetArg) : targetArg; +if (!target.startsWith('src/') || target.includes('..')) { + die(2, `Target must be under src/ within this package. Got: ${target}`); +} +if (!existsSync(path.join(pkgRoot, target))) { + die(2, `Target not found: ${path.join(pkgRoot, target)}`); +} + +const reportDir = path.join(pkgRoot, 'reports/mutation'); +const rawJsonPath = path.join(reportDir, 'raw.json'); +const summaryJsonPath = path.join(reportDir, 'summary.json'); + +process.stderr.write(`Running Stryker on ${target} (threshold: ${THRESHOLD}%)\n`); + +await new Promise((resolve) => { + const child = spawn('node_modules/.bin/stryker', ['run', '--mutate', target], { + cwd: pkgRoot, + stdio: 'inherit', + }); + child.on('exit', (code) => { + if (code !== 0) die(3, `Stryker exited with code ${code}`); + resolve(); + }); + child.on('error', (err) => die(3, `Stryker failed to start: ${err.message}`)); +}); + +if (!existsSync(rawJsonPath)) { + die(3, `Stryker did not produce ${rawJsonPath}`); +} + +const raw = JSON.parse(await readFile(rawJsonPath, 'utf8')); + +// Build a test-id → test-name lookup so survivors can name the tests that +// covered the mutated line without killing the mutant. +const testIdToName = {}; +for (const info of Object.values(raw.testFiles ?? {})) { + for (const t of info.tests ?? []) { + testIdToName[t.id] = t.name; + } +} + +function sliceFromLocation(source, loc) { + const lines = source.split('\n'); + const { start, end } = loc; + if (start.line === end.line) { + return lines[start.line - 1].slice(start.column, end.column); + } + return [ + lines[start.line - 1].slice(start.column), + ...lines.slice(start.line, end.line - 1), + lines[end.line - 1].slice(0, end.column), + ].join('\n'); +} + +function scoreFromCounts(c) { + const detected = c.killed + c.timeout; + const valid = c.killed + c.timeout + c.survived + c.noCoverage; + return valid === 0 ? 0 : +((detected / valid) * 100).toFixed(2); +} + +const filesSummary = []; +for (const [file, info] of Object.entries(raw.files)) { + const counts = { + killed: 0, + survived: 0, + noCoverage: 0, + timeout: 0, + compileError: 0, + runtimeError: 0, + ignored: 0, + }; + const survivors = []; + for (const m of info.mutants) { + switch (m.status) { + case 'Killed': + counts.killed++; + break; + case 'Survived': + counts.survived++; + break; + case 'NoCoverage': + counts.noCoverage++; + break; + case 'Timeout': + counts.timeout++; + break; + case 'CompileError': + counts.compileError++; + break; + case 'RuntimeError': + counts.runtimeError++; + break; + case 'Ignored': + counts.ignored++; + break; + } + if (m.status === 'Survived' || m.status === 'NoCoverage') { + survivors.push({ + id: m.id, + mutator: m.mutatorName, + status: m.status, + location: `${file}:${m.location.start.line}:${m.location.start.column}`, + line: m.location.start.line, + original: sliceFromLocation(info.source, m.location), + replacement: m.replacement, + coveringTests: (m.coveredBy ?? []).map((id) => testIdToName[id] ?? id), + }); + } + } + survivors.sort((a, b) => a.line - b.line); + const score = scoreFromCounts(counts); + filesSummary.push({ + file, + score, + thresholdMet: score >= THRESHOLD, + counts, + survivors, + }); +} + +const overallCounts = filesSummary.reduce( + (acc, f) => { + for (const k of Object.keys(acc)) acc[k] += f.counts[k]; + return acc; + }, + { + killed: 0, + survived: 0, + noCoverage: 0, + timeout: 0, + compileError: 0, + runtimeError: 0, + ignored: 0, + }, +); + +const summary = { + generatedAt: new Date().toISOString(), + threshold: THRESHOLD, + target, + overall: { + score: scoreFromCounts(overallCounts), + counts: overallCounts, + thresholdMet: scoreFromCounts(overallCounts) >= THRESHOLD, + }, + files: filesSummary, +}; + +await writeFile(summaryJsonPath, JSON.stringify(summary, null, 2)); + +process.stderr.write('\n=== Mutation summary ===\n'); +for (const f of filesSummary) { + const mark = f.thresholdMet ? '✓' : '✗'; + process.stderr.write( + `${mark} ${f.file} ${f.score.toFixed(2)}% ` + + `(killed ${f.counts.killed} / survived ${f.counts.survived} / no-cov ${f.counts.noCoverage} / timeout ${f.counts.timeout})\n`, + ); + for (const s of f.survivors) { + process.stderr.write( + ` - ${s.status.toLowerCase().padEnd(10)} ${s.mutator.padEnd(22)} ${s.location}\n`, + ); + } +} +process.stderr.write( + `\nThreshold: ${THRESHOLD}% • overall: ${summary.overall.score.toFixed(2)}%\n`, +); +process.stderr.write(`Summary written: ${summaryJsonPath}\n`); + +process.exit(summary.overall.thresholdMet ? 0 : 1); diff --git a/packages/workflow/stryker.config.mjs b/packages/workflow/stryker.config.mjs new file mode 100644 index 00000000000..dc1f2326d61 --- /dev/null +++ b/packages/workflow/stryker.config.mjs @@ -0,0 +1,26 @@ +/** @type {import('@stryker-mutator/api/core').PartialStrykerOptions} */ +export default { + packageManager: 'pnpm', + testRunner: 'vitest', + plugins: ['@stryker-mutator/vitest-runner'], + // Use the Stryker-specific vitest config (vm-engine only) — see + // vitest.stryker.config.ts for the rationale. + vitest: { + configFile: 'vitest.stryker.config.ts', + }, + reporters: ['progress', 'clear-text', 'html', 'json'], + coverageAnalysis: 'perTest', + // Default empty — the `mutate` npm script always passes --mutate . + // Direct invocation with no --mutate will fail fast (allowEmpty: false). + mutate: [], + htmlReporter: { fileName: 'reports/mutation/raw.html' }, + jsonReporter: { fileName: 'reports/mutation/raw.json' }, + timeoutMS: 60_000, + // Each Stryker worker spawns vitest with one project (vm-engine, via + // vitest.stryker.config.ts). Default 4 is fine on CI runners; lower + // locally if you hit OOM during the initial dry run via + // `STRYKER_CONCURRENCY=1`. + concurrency: Number(process.env.STRYKER_CONCURRENCY ?? 4), + tempDirName: '.stryker-tmp', + cleanTempDir: true, +}; diff --git a/packages/workflow/test/setup-vm-evaluator.ts b/packages/workflow/test/setup-vm-evaluator.ts index 3ce6f67d4ac..18c61cf997b 100644 --- a/packages/workflow/test/setup-vm-evaluator.ts +++ b/packages/workflow/test/setup-vm-evaluator.ts @@ -14,7 +14,13 @@ if (process.env.N8N_EXPRESSION_ENGINE === 'vm') { }); }); - afterAll(async () => { - await Expression.disposeExpressionEngine(); - }); + // Under Stryker, the worker process exits the moment vitest finishes — the + // OS reclaims isolated-vm native handles either way. Calling dispose here + // aborts the worker on Node 24 with a native finaliser assertion, which + // Stryker reports as a dry-run failure. + if (!process.env.STRYKER_RUN) { + afterAll(async () => { + await Expression.disposeExpressionEngine(); + }); + } } diff --git a/packages/workflow/vitest.stryker.config.ts b/packages/workflow/vitest.stryker.config.ts new file mode 100644 index 00000000000..9fe1d23ab37 --- /dev/null +++ b/packages/workflow/vitest.stryker.config.ts @@ -0,0 +1,40 @@ +// Vitest config used by Stryker only — NOT by `pnpm test`, NOT by CI's +// unit-test runs. Runs the single forward-looking `vm-engine` project +// (N8N_EXPRESSION_ENGINE=vm) rather than both engines. +// +// Two reasons: +// 1. Halves Stryker's dry-run cost — only one vitest project loads per +// Stryker worker, removing concurrent isolated-vm initialisation +// pressure that occasionally crashes the dry-run on local machines. +// 2. Mutation score reflects test effectiveness against the engine n8n +// is moving to. Legacy-engine is being phased out; tests that pass +// only under it shouldn't pad the mutation score. +// +// The default `vitest.config.ts` still runs both projects for `pnpm test` +// and CI — engine-equivalence is asserted there. + +import { defineConfig } from 'vitest/config'; +import { createBaseInlineConfig } from '@n8n/vitest-config/node'; + +const { reporters, outputFile, ...sharedTestConfig } = createBaseInlineConfig({ + include: ['test/**/*.test.ts'], + setupFiles: ['./test/setup-vm-evaluator.ts'], +}); + +export default defineConfig({ + test: { + reporters, + outputFile, + projects: [ + { + test: { + ...sharedTestConfig, + name: 'vm-engine', + // STRYKER_RUN tells setup-vm-evaluator.ts to skip the + // isolated-vm disposer on teardown — see that file for why. + env: { N8N_EXPRESSION_ENGINE: 'vm', STRYKER_RUN: 'true' }, + }, + }, + ], + }, +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 4780e5916be..23a18e9c8dc 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -99,6 +99,12 @@ catalogs: '@rudderstack/rudder-sdk-node': specifier: 3.0.5 version: 3.0.5 + '@stryker-mutator/core': + specifier: 9.6.1 + version: 9.6.1 + '@stryker-mutator/vitest-runner': + specifier: 9.6.1 + version: 9.6.1 '@supabase/supabase-js': specifier: 2.50.0 version: 2.50.0 @@ -4941,6 +4947,12 @@ importers: '@n8n/vitest-config': specifier: workspace:* version: link:../@n8n/vitest-config + '@stryker-mutator/core': + specifier: 'catalog:' + version: 9.6.1(@types/node@20.19.21) + '@stryker-mutator/vitest-runner': + specifier: 'catalog:' + version: 9.6.1(@stryker-mutator/core@9.6.1(@types/node@20.19.21))(vitest@4.1.1(@opentelemetry/api@1.9.0)(@types/node@20.19.21)(jsdom@23.0.1(bufferutil@4.0.9)(utf-8-validate@5.0.10))(vite@8.0.2(@types/node@20.19.21)(esbuild@0.25.10)(jiti@2.6.1)(sass-embedded@1.98.0)(sass@1.98.0)(terser@5.16.1)(tsx@4.19.3)(yaml@2.8.3))) '@types/express': specifier: 'catalog:' version: 5.0.1 @@ -5815,6 +5827,12 @@ packages: peerDependencies: '@babel/core': ^7.0.0 + '@babel/plugin-proposal-decorators@7.29.0': + resolution: {integrity: sha512-CVBVv3VY/XRMxRYq5dwr2DS7/MvqPm23cOCjbwNnVrfOqcWlnefua1uUs0sjdKOGjvPUG633o07uWzJq4oI6dA==} + engines: {node: '>=6.9.0'} + peerDependencies: + '@babel/core': ^7.0.0-0 + '@babel/plugin-proposal-private-property-in-object@7.21.0-placeholder-for-preset-env.2': resolution: {integrity: sha512-SOSkfJDddaM7mak6cPEpswyTRnuRltl429hMraQEglW+OkovnCzsiszTmsrlY//qLFjCpQDFRvjdm2wA5pPm9w==} engines: {node: '>=6.9.0'} @@ -5842,6 +5860,12 @@ packages: peerDependencies: '@babel/core': ^7.0.0-0 + '@babel/plugin-syntax-decorators@7.28.6': + resolution: {integrity: sha512-71EYI0ONURHJBL4rSFXnITXqXrrY8q4P0q006DPfN+Rk+ASM+++IBXem/ruokgBZR8YNEWZ8R6B+rCb8VcUTqA==} + engines: {node: '>=6.9.0'} + peerDependencies: + '@babel/core': ^7.0.0-0 + '@babel/plugin-syntax-import-assertions@7.28.6': resolution: {integrity: sha512-pSJUpFHdx9z5nqTSirOCMtYVP2wFgoWhP0p3g8ONK/4IHhLIBd0B9NYqAvIUAhq+OkhO4VM1tENCt0cjlsNShw==} engines: {node: '>=6.9.0'} @@ -5918,6 +5942,12 @@ packages: peerDependencies: '@babel/core': ^7.0.0-0 + '@babel/plugin-syntax-typescript@7.28.6': + resolution: {integrity: sha512-+nDNmQye7nlnuuHDboPbGm00Vqg3oO8niRRL27/4LYHUsHYh0zJ1xWOz0uRwNFmM1Avzk8wZbc6rdiYhomzv/A==} + engines: {node: '>=6.9.0'} + peerDependencies: + '@babel/core': ^7.0.0-0 + '@babel/plugin-syntax-unicode-sets-regex@7.18.6': resolution: {integrity: sha512-727YkEAPwSIQTv5im8QHz3upqp92JTWhidIC81Tdx4VJYIte/VndKf1qKrfnnhPLiPghStWfvC/iFaMCQu7Nqg==} engines: {node: '>=6.9.0'} @@ -6206,6 +6236,12 @@ packages: peerDependencies: '@babel/core': ^7.0.0-0 + '@babel/plugin-transform-typescript@7.28.6': + resolution: {integrity: sha512-0YWL2RFxOqEm9Efk5PvreamxPME8OyY0wM5wh5lHjF+VtVhdneCWGzZeSqzOfiobVqQaNCd2z0tQvnI9DaPWPw==} + engines: {node: '>=6.9.0'} + peerDependencies: + '@babel/core': ^7.0.0-0 + '@babel/plugin-transform-unicode-escapes@7.27.1': resolution: {integrity: sha512-Ysg4v6AmF26k9vpfFuTZg8HRfVWzsh1kVfowA23y9j/Gu6dOuahdUVhkLqpObp3JIv27MLSii6noRnuKN8H0Mg==} engines: {node: '>=6.9.0'} @@ -6241,6 +6277,12 @@ packages: peerDependencies: '@babel/core': ^7.0.0-0 || ^8.0.0-0 <8.0.0 + '@babel/preset-typescript@7.28.5': + resolution: {integrity: sha512-+bQy5WOI2V6LJZpPVxY+yp66XdZ2yifu0Mc1aP5CQKgjn4QM5IN2i5fAZ4xKop47pr8rpVhiAeu+nDQa12C8+g==} + engines: {node: '>=6.9.0'} + peerDependencies: + '@babel/core': ^7.0.0-0 + '@babel/runtime@7.28.4': resolution: {integrity: sha512-Q/N6JNWvIvPnLDvjlE1OUBLPQHH6l3CltCEsHIujp45zQUSSh8K+gHnaEX45yAT1nyngnINhvWtzN+Nb9D8RAQ==} engines: {node: '>=6.9.0'} @@ -10139,6 +10181,29 @@ packages: storybook: ^10.1.11 vue: ^3.0.0 + '@stryker-mutator/api@9.6.1': + resolution: {integrity: sha512-g8VNoFWQWbx0pdal3Vt8jVCZW+v3sc3gi94iI0GVtVgUGTqphAjJF6EAruPTx0lqvtonsaAxn5TD36hcG1d6Wg==} + engines: {node: '>=20.0.0'} + + '@stryker-mutator/core@9.6.1': + resolution: {integrity: sha512-WMgnvf+Wyh/yiruhNZwc8w8DlzmmjXhPjSn5MR8RhAXzlnWji8TQrUYgBUkHk9bEgSaIlB3KZHm37iiU5Q2cLQ==} + engines: {node: '>=20.0.0'} + hasBin: true + + '@stryker-mutator/instrumenter@9.6.1': + resolution: {integrity: sha512-5K8wH4Pthly25c2uKKik4Dfcoeou7sbJdFS6u3QIYHlulgFVDJwtEMWTZGkZfs7IiUEXIDNa0keRACq5jn5AvA==} + engines: {node: '>=20.0.0'} + + '@stryker-mutator/util@9.6.1': + resolution: {integrity: sha512-Lk/ALVctJjFv1vvwR+CFoKzDCWvsBlq7flDUnmnpuwTrGbm156EdZD1Jjq4o8KdOap0ezUZqQNE9OAI1m2+pUQ==} + + '@stryker-mutator/vitest-runner@9.6.1': + resolution: {integrity: sha512-eyUHTCf3Ui+SUn/tpFJwzw6MV391kyBLZk/cDHFUfKFELqKMLbvd7e81axArlApKqO6cOnLfrxlwED+2SRN0ow==} + engines: {node: '>=14.18.0'} + peerDependencies: + '@stryker-mutator/core': 9.6.1 + vitest: '>=2.0.0' + '@stylistic/eslint-plugin@5.0.0': resolution: {integrity: sha512-nVV2FSzeTJ3oFKw+3t9gQYQcrgbopgCASSY27QOtkhEGgSfdQQjDmzZd41NeT1myQ8Wc6l+pZllST9qIu4NKzg==} engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} @@ -11541,6 +11606,10 @@ packages: resolution: {integrity: sha512-TGZJ/Q6PO0ns/a72zw/d3FI0ywqY7oMqTbRzji2/AsoA/1frIhIOuVoqZMapDt6XFppbbdT0NEzd9dYwmKI0rQ==} engines: {node: '>=10'} + angular-html-parser@10.4.0: + resolution: {integrity: sha512-++nLNyZwRfHqFh7akH5Gw/JYizoFlMRz0KRigfwfsLqV8ZqlcVRb1LkPEWdYvEKDnbktknM2J4BXaYUGrQZPww==} + engines: {node: '>= 14'} + ansi-colors@4.1.3: resolution: {integrity: sha512-/6w/C21Pm1A7aZitlI5Ni/2J6FFQN8i1Cvz3kHABAAbw93v/NlvKdVOqz7CCWz/3iv/JplRSEEZ83XION15ovw==} engines: {node: '>=6'} @@ -12238,6 +12307,10 @@ packages: resolution: {integrity: sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==} engines: {node: '>=10'} + chalk@5.6.2: + resolution: {integrity: sha512-7NzBL0rN6fMUW+f7A6Io4h40qQlG+xGmtMxfbnH/K7TAtt8JQWVQK+6g0UXKMeVJoyV5EkkNsErQ8pVD3bLHbA==} + engines: {node: ^12.17.0 || ^14.13 || >=16.0.0} + change-case@4.1.2: resolution: {integrity: sha512-bSxY2ws9OtviILG1EiY5K7NNxkqg/JnRnFxLtKQ96JaviiIxi7djMrSd0ECT9AC+lttClmYwKw53BWpOMblo7A==} @@ -13445,6 +13518,9 @@ packages: resolution: {integrity: sha512-DeWwawk6r5yR9jFgnDKYt4sLS0LmHJJi3ZOnb5/JdbYwj3nW+FxQnHIjhBKz8YLC7oRNPVM9NQ47I3CVx34eqQ==} engines: {node: '>=12'} + emoji-regex@10.6.0: + resolution: {integrity: sha512-toUI84YS5YmxW219erniWD0CIVOo46xGKColeNQRgOzDorgBi1v4D71/OFzgD9GO2UGKIv1C3Sp8DAn0+j5w7A==} + emoji-regex@8.0.0: resolution: {integrity: sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==} @@ -15536,6 +15612,9 @@ packages: json-pointer@0.6.2: resolution: {integrity: sha512-vLWcKbOaXlO+jvRy4qNd+TI1QUPZzfJj1tpJ3vAXDych5XJf93ftpUKe5pKCrzyIIwgBJcOcCVRUfqQP25afBw==} + json-rpc-2.0@1.7.1: + resolution: {integrity: sha512-JqZjhjAanbpkXIzFE7u8mE/iFblawwlXtONaCvRqI+pyABVz7B4M1EUNpyVW+dZjqgQ2L5HFmZCmOCgUKm00hg==} + json-schema-to-ts@3.1.1: resolution: {integrity: sha512-+DWg8jCJG2TEnpy7kOm/7/AxaYoaRbjVB4LFZLySZlWn8exGs3A4OLJR966cVvU26N7X9TWxl+Jsw7dzAqKT6g==} engines: {node: '>=16'} @@ -15951,6 +16030,9 @@ packages: resolution: {integrity: sha512-z+Uw/vLuy6gQe8cfaFWD7p0wVv8fJl3mbzXh33RS+0oW2wvUqiRXiQ69gLWSLpgB5/6sU+r6BlQR0MBILadqTQ==} deprecated: This package is deprecated. Use the optional chaining (?.) operator instead. + lodash.groupby@4.6.0: + resolution: {integrity: sha512-5dcWxm23+VAoz+awKmBaiBvzox8+RqMgFhi7UvX9DHZr2HdxHXM/Wrf8cfKpsW37RNrvtPn6hSwNqurSILbmJw==} + lodash.includes@4.3.0: resolution: {integrity: sha512-W3Bx6mdkRTGtlJISOvVD/lbqjTlPPUDTMnlXZFnVwi9NKJ6tiAk6LVdlhZMm17VZisqhKcgzpO5Wz91PCt5b0w==} @@ -16793,6 +16875,19 @@ packages: resolution: {integrity: sha512-71ippSywq5Yb7/tVYyGbkBggbU8H3u5Rz56fH60jGFgr8uHwxs+aSKeqmluIVzM0m0kB7xQjKS6qPfd0b2ZoqQ==} hasBin: true + mutation-server-protocol@0.4.1: + resolution: {integrity: sha512-SBGK0j8hLDne7bktgThKI8kGvGTx3rY3LAeQTmOKZ5bVnL/7TorLMvcVF7dIPJCu5RNUWhkkuF53kurygYVt3g==} + engines: {node: '>=18'} + + mutation-testing-elements@3.7.3: + resolution: {integrity: sha512-SMeIPxngJpfjfNYctFpYQQtlBlZaVO0aoB3FKdwrI8Ee/2bkyUuCZzAOCLv1U9fnmfA37dPFq0Owduoxs2XgGQ==} + + mutation-testing-metrics@3.7.3: + resolution: {integrity: sha512-B8QrP0ZomErzTPNlhrzKWPNBln+3afwBZPHv0Q7N8wZZTYxMptzb/Gdm3ExXVmioVYrtZAtsDs7W/T/b2AixOQ==} + + mutation-testing-report-schema@3.7.3: + resolution: {integrity: sha512-BHm3MYq+ckO+t5CtlG8zpqxc75rdJCkxVlE+fGuGJM3F7tNCQ/OW2N+TQVHN3BHsYa84+BFc6g3AwDYkUsw2MA==} + mute-stream@0.0.8: resolution: {integrity: sha512-nnbWWOkoWyUsTjKrhgD0dcz22mdkSnpYqbEjIm2nhwhuxlSkpywJmBo8h0ZqJdkp73mb90SssHkN4rsRaBAfAA==} @@ -19938,9 +20033,17 @@ packages: resolution: {integrity: sha512-3KS2b+kL7fsuk/eJZ7EQdnEmQoaho/r6KUef7hxvltNA5DR8NAUM+8wJMbJyZ4G9/7i3v5zPBIMN5aybAh2/Jg==} engines: {node: '>= 0.4'} + typed-inject@5.0.0: + resolution: {integrity: sha512-0Ql2ORqBORLMdAW89TQKZsb1PQkFGImFfVmncXWe7a+AA3+7dh7Se9exxZowH4kbnlvKEFkMxUYdHUpjYWFJaA==} + engines: {node: '>=18'} + typed-query-selector@2.12.1: resolution: {integrity: sha512-uzR+FzI8qrUEIu96oaeBJmd9E7CFEiQ3goA5qCVgc4s5llSubcfGHq9yUstZx/k4s9dXHVKsE35YWoFyvEqEHA==} + typed-rest-client@2.3.1: + resolution: {integrity: sha512-k4kX5Up6qA68D0Cby2AK+6+vM5k3qTxe+/3FqhnHRExjY5cfbOnzjQZbP/LXleF8hVoDvDqxlgk9KK83HoBZlQ==} + engines: {node: '>= 16.0.0'} + typedarray-dts@1.0.0: resolution: {integrity: sha512-Ka0DBegjuV9IPYFT1h0Qqk5U4pccebNIJCGl8C5uU7xtOs+jpJvKGAY4fHGK25hTmXZOEUl9Cnsg5cS6K/b5DA==} @@ -20548,6 +20651,9 @@ packages: weak-map@1.0.8: resolution: {integrity: sha512-lNR9aAefbGPpHO7AEnY0hCFjz1eTkWCXYvkTRrTHs9qv8zJp+SkVYpzfLIFXQQiG3tVvbNFQgVg2bQS8YGgxyw==} + weapon-regex@1.3.6: + resolution: {integrity: sha512-wsf1m1jmMrso5nhwVFJJHSubEBf3+pereGd7+nBKtYJ18KoB/PWJOHS3WRkwS04VrOU0iJr2bZU+l1QaTJ+9nA==} + weaviate-client@3.9.0: resolution: {integrity: sha512-7qwg7YONAaT4zWnohLrFdzky+rZegVe76J+Tky/+7tuyvjFpdKgSrdqI/wPDh8aji0ZGZrL4DdGwGfFnZ+uV4w==} engines: {node: '>=18.0.0'} @@ -23126,6 +23232,15 @@ snapshots: transitivePeerDependencies: - supports-color + '@babel/plugin-proposal-decorators@7.29.0(@babel/core@7.29.0)': + dependencies: + '@babel/core': 7.29.0 + '@babel/helper-create-class-features-plugin': 7.28.6(@babel/core@7.29.0) + '@babel/helper-plugin-utils': 7.28.6 + '@babel/plugin-syntax-decorators': 7.28.6(@babel/core@7.29.0) + transitivePeerDependencies: + - supports-color + '@babel/plugin-proposal-private-property-in-object@7.21.0-placeholder-for-preset-env.2(@babel/core@7.29.0)': dependencies: '@babel/core': 7.29.0 @@ -23150,6 +23265,11 @@ snapshots: '@babel/core': 7.29.0 '@babel/helper-plugin-utils': 7.28.6 + '@babel/plugin-syntax-decorators@7.28.6(@babel/core@7.29.0)': + dependencies: + '@babel/core': 7.29.0 + '@babel/helper-plugin-utils': 7.28.6 + '@babel/plugin-syntax-import-assertions@7.28.6(@babel/core@7.29.0)': dependencies: '@babel/core': 7.29.0 @@ -23220,6 +23340,11 @@ snapshots: '@babel/core': 7.29.0 '@babel/helper-plugin-utils': 7.28.6 + '@babel/plugin-syntax-typescript@7.28.6(@babel/core@7.29.0)': + dependencies: + '@babel/core': 7.29.0 + '@babel/helper-plugin-utils': 7.28.6 + '@babel/plugin-syntax-unicode-sets-regex@7.18.6(@babel/core@7.29.0)': dependencies: '@babel/core': 7.29.0 @@ -23536,6 +23661,17 @@ snapshots: '@babel/core': 7.29.0 '@babel/helper-plugin-utils': 7.28.6 + '@babel/plugin-transform-typescript@7.28.6(@babel/core@7.29.0)': + dependencies: + '@babel/core': 7.29.0 + '@babel/helper-annotate-as-pure': 7.27.3 + '@babel/helper-create-class-features-plugin': 7.28.6(@babel/core@7.29.0) + '@babel/helper-plugin-utils': 7.28.6 + '@babel/helper-skip-transparent-expression-wrappers': 7.27.1 + '@babel/plugin-syntax-typescript': 7.28.6(@babel/core@7.29.0) + transitivePeerDependencies: + - supports-color + '@babel/plugin-transform-unicode-escapes@7.27.1(@babel/core@7.29.0)': dependencies: '@babel/core': 7.29.0 @@ -23642,6 +23778,17 @@ snapshots: '@babel/types': 7.29.0 esutils: 2.0.3 + '@babel/preset-typescript@7.28.5(@babel/core@7.29.0)': + dependencies: + '@babel/core': 7.29.0 + '@babel/helper-plugin-utils': 7.28.6 + '@babel/helper-validator-option': 7.27.1 + '@babel/plugin-syntax-jsx': 7.27.1(@babel/core@7.29.0) + '@babel/plugin-transform-modules-commonjs': 7.28.6(@babel/core@7.29.0) + '@babel/plugin-transform-typescript': 7.28.6(@babel/core@7.29.0) + transitivePeerDependencies: + - supports-color + '@babel/runtime@7.28.4': {} '@babel/template@7.28.6': @@ -28008,6 +28155,73 @@ snapshots: vue: 3.5.26(typescript@6.0.2) vue-component-type-helpers: 3.3.1 + '@stryker-mutator/api@9.6.1': + dependencies: + mutation-testing-metrics: 3.7.3 + mutation-testing-report-schema: 3.7.3 + tslib: 2.8.1 + typed-inject: 5.0.0 + + '@stryker-mutator/core@9.6.1(@types/node@20.19.21)': + dependencies: + '@inquirer/prompts': 8.3.2(@types/node@20.19.21) + '@stryker-mutator/api': 9.6.1 + '@stryker-mutator/instrumenter': 9.6.1 + '@stryker-mutator/util': 9.6.1 + ajv: 8.18.0 + chalk: 5.6.2 + commander: 14.0.1 + diff-match-patch: 1.0.5 + emoji-regex: 10.6.0 + execa: 9.6.1 + json-rpc-2.0: 1.7.1 + lodash.groupby: 4.6.0 + minimatch: 10.2.3 + mutation-server-protocol: 0.4.1 + mutation-testing-elements: 3.7.3 + mutation-testing-metrics: 3.7.3 + mutation-testing-report-schema: 3.7.3 + npm-run-path: 6.0.0 + progress: 2.0.3 + rxjs: 7.8.1 + semver: 7.7.3 + source-map: 0.7.6 + tree-kill: 1.2.2 + tslib: 2.8.1 + typed-inject: 5.0.0 + typed-rest-client: 2.3.1 + transitivePeerDependencies: + - '@types/node' + - supports-color + + '@stryker-mutator/instrumenter@9.6.1': + dependencies: + '@babel/core': 7.29.0 + '@babel/generator': 7.29.1 + '@babel/parser': 7.29.2 + '@babel/plugin-proposal-decorators': 7.29.0(@babel/core@7.29.0) + '@babel/plugin-transform-explicit-resource-management': 7.28.6(@babel/core@7.29.0) + '@babel/preset-typescript': 7.28.5(@babel/core@7.29.0) + '@stryker-mutator/api': 9.6.1 + '@stryker-mutator/util': 9.6.1 + angular-html-parser: 10.4.0 + semver: 7.7.3 + tslib: 2.8.1 + weapon-regex: 1.3.6 + transitivePeerDependencies: + - supports-color + + '@stryker-mutator/util@9.6.1': {} + + '@stryker-mutator/vitest-runner@9.6.1(@stryker-mutator/core@9.6.1(@types/node@20.19.21))(vitest@4.1.1(@opentelemetry/api@1.9.0)(@types/node@20.19.21)(jsdom@23.0.1(bufferutil@4.0.9)(utf-8-validate@5.0.10))(vite@8.0.2(@types/node@20.19.21)(esbuild@0.25.10)(jiti@2.6.1)(sass-embedded@1.98.0)(sass@1.98.0)(terser@5.16.1)(tsx@4.19.3)(yaml@2.8.3)))': + dependencies: + '@stryker-mutator/api': 9.6.1 + '@stryker-mutator/core': 9.6.1(@types/node@20.19.21) + '@stryker-mutator/util': 9.6.1 + semver: 7.7.3 + tslib: 2.8.1 + vitest: 4.1.1(@opentelemetry/api@1.9.0)(@types/node@20.19.21)(@vitest/browser-playwright@4.0.16)(jsdom@23.0.1(bufferutil@4.0.9)(utf-8-validate@5.0.10))(vite@8.0.2(@types/node@20.19.21)(esbuild@0.25.10)(jiti@2.6.1)(sass-embedded@1.98.0)(sass@1.98.0)(terser@5.16.1)(tsx@4.19.3)(yaml@2.8.3)) + '@stylistic/eslint-plugin@5.0.0(eslint@9.29.0(jiti@2.6.1))': dependencies: '@eslint-community/eslint-utils': 4.7.0(eslint@9.29.0(jiti@2.6.1)) @@ -29646,6 +29860,8 @@ snapshots: transitivePeerDependencies: - supports-color + angular-html-parser@10.4.0: {} + ansi-colors@4.1.3: {} ansi-escapes@4.3.2: @@ -30536,6 +30752,8 @@ snapshots: ansi-styles: 4.3.0 supports-color: 7.2.0 + chalk@5.6.2: {} + change-case@4.1.2: dependencies: camel-case: 4.1.2 @@ -31886,6 +32104,8 @@ snapshots: emittery@0.13.1: {} + emoji-regex@10.6.0: {} + emoji-regex@8.0.0: {} emoji-regex@9.2.2: {} @@ -34619,6 +34839,8 @@ snapshots: dependencies: foreach: 2.0.6 + json-rpc-2.0@1.7.1: {} + json-schema-to-ts@3.1.1: dependencies: '@babel/runtime': 7.28.4 @@ -35049,6 +35271,8 @@ snapshots: lodash.get@4.4.2: {} + lodash.groupby@4.6.0: {} + lodash.includes@4.3.0: {} lodash.isarguments@3.1.0: {} @@ -36252,6 +36476,18 @@ snapshots: mustache@4.2.0: {} + mutation-server-protocol@0.4.1: + dependencies: + zod: 3.25.67 + + mutation-testing-elements@3.7.3: {} + + mutation-testing-metrics@3.7.3: + dependencies: + mutation-testing-report-schema: 3.7.3 + + mutation-testing-report-schema@3.7.3: {} + mute-stream@0.0.8: {} mute-stream@3.0.0: {} @@ -39990,8 +40226,18 @@ snapshots: possible-typed-array-names: 1.1.0 reflect.getprototypeof: 1.0.10 + typed-inject@5.0.0: {} + typed-query-selector@2.12.1: {} + typed-rest-client@2.3.1: + dependencies: + des.js: 1.1.0 + js-md4: 0.3.2 + qs: 6.14.2 + tunnel: 0.0.6 + underscore: 1.13.8 + typedarray-dts@1.0.0: {} typedarray-to-buffer@3.1.5: @@ -40604,6 +40850,8 @@ snapshots: weak-map@1.0.8: {} + weapon-regex@1.3.6: {} + weaviate-client@3.9.0(encoding@0.1.13): dependencies: abort-controller-x: 0.4.3 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 808108bf363..aa5daffc63e 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -57,6 +57,8 @@ catalog: '@types/node': 24.10.1 '@types/uuid': ^10.0.0 '@types/xml2js': ^0.4.14 + '@stryker-mutator/core': 9.6.1 + '@stryker-mutator/vitest-runner': 9.6.1 '@vitest/coverage-v8': 4.1.1 agent-browser: 0.26.0 axios: 1.16.1 diff --git a/scripts/mutation-health/DEMO-HANDOVER.md b/scripts/mutation-health/DEMO-HANDOVER.md new file mode 100644 index 00000000000..c7966f2941c --- /dev/null +++ b/scripts/mutation-health/DEMO-HANDOVER.md @@ -0,0 +1,92 @@ +# Demo handover: stacked PR on #30956 + +Use this prompt to drive the strengthen-tests loop end-to-end and open a stacked PR that demonstrates the trial. + +--- + +## Prompt + +> I want to demo the mutation-health strengthen-tests loop from PR #30956. Drive the whole flow from a fresh branch and open a stacked PR. +> +> **Base branch**: `devp-stryker-mvp-spike` (the PR's branch — not master yet). +> +> **Steps**: +> +> 1. `git fetch origin && git checkout devp-stryker-mvp-spike && git pull && git checkout -b demo/strengthen-` +> +> 2. Query the live ledger to find the lowest-score red file: +> ```bash +> curl -sS 'https://internal.users.n8n.cloud/webhook/mutation-health-ledger?package=n8n-workflow' \ +> | jq '.ledger | map(select(.status == "red")) | sort_by(.last_score | tonumber) | .[0]' +> ``` +> Use whatever it returns. As of 2026-05-22, that's `src/workflow-checksum.ts` at 38.64% — but check live state first. +> +> 3. Run the local mutation-testing skill on that file: +> `/n8n:mutation-test packages/workflow/src/` +> +> Confirm the output JSON shows the score and a list of survivors with mutator + location + covering tests. +> +> 4. Run the strengthen skill: +> `/n8n:strengthen-tests` +> +> It'll triage survivors (HIGH/MODERATE/LOW), edit the covering test file with targeted assertions, and re-run `n8n:mutation-test` to verify the score climbed. Max 5 survivors per pass. +> +> 5. Review the diff yourself: `git diff packages/workflow/test/` +> +> Sanity-check each new assertion. Reject anything that's mocking-the-mock, asserting trivia, or pinning behaviour the source doesn't actually have. The skill is supposed to refuse to fabricate but humans verify. +> +> 6. If you want to push further, re-invoke `/n8n:strengthen-tests` for the next 5 survivors. Or move on. +> +> 7. Final verification: +> `/n8n:mutation-test packages/workflow/src/` +> +> Capture the before/after score for the PR body. +> +> 8. Push and open a stacked PR **against `devp-stryker-mvp-spike`** (not master): +> ```bash +> git push -u origin demo/strengthen- +> gh pr create --draft --base devp-stryker-mvp-spike \ +> --title "test(core): strengthen assertions (demo) (no-changelog)" \ +> --body "" +> ``` +> +> **PR body template:** +> +> ```markdown +> ## Summary +> +> Demo PR for #30956. Drives the `n8n:strengthen-tests` loop against `packages/workflow/src/` to show the trial loop end-to-end. +> +> **Before**: % mutation score, survivors +> **After**: % mutation score, survivors +> +> Survivors addressed (with rationale): +> 1. — added to +> 2. ... +> +> ## Test plan +> - [ ] `pnpm --filter=n8n-workflow mutate src/` reproduces the post-score locally +> - [ ] `pnpm --filter=n8n-workflow test test/.test.ts` passes +> - [ ] Each new assertion has a clear "this would have caught X bug" justification +> ``` +> +> **Goals of the demo:** +> +> - Reviewer sees a real diff with surgical assertion edits, not big-bang test rewrites +> - The before/after numbers are reproducible (`pnpm mutate` gives the same answer to anyone) +> - The skill refused to fabricate or split low-leverage survivors out — what landed is what mattered +> +> **What NOT to do:** +> +> - Don't rewrite whole test files. The skill should only add/extend covering tests. +> - Don't bypass the verify step. Every change must be backed by a re-run that shows the score moved. +> - Don't auto-merge. This is a draft demo; the reviewer takes it forward. + +--- + +## Why this is a good first PR for a reviewer + +- Small (a handful of assertion lines) +- Numerically verifiable (run `pnpm mutate` yourself, see the same number) +- Demonstrates the full loop without committing to the AI auto-PR pipeline yet +- Stacked on `devp-stryker-mvp-spike` so the loop's machinery is already on the branch diff --git a/scripts/mutation-health/README.md b/scripts/mutation-health/README.md new file mode 100644 index 00000000000..5889cc34dd0 --- /dev/null +++ b/scripts/mutation-health/README.md @@ -0,0 +1,231 @@ +# `scripts/mutation-health/` + +Phase 1 substrate for the Mutation Health Observability initiative. + +## What is mutation testing? + +Line coverage tells you which lines your tests **execute**. Mutation testing tells you which behavioural changes your tests **catch**. A file can have 100% line coverage and a 0% mutation score: every line runs during the test suite, but no test would fail if the code were silently broken. + +### How it works + +A mutation testing tool (n8n uses [Stryker](https://stryker-mutator.io/)) does this for each source file: + +1. **Parse the source into an AST.** +2. **Generate small variants ("mutants")** by changing nodes in the AST. Examples: + + | Mutator | Original | Mutated | + | --- | --- | --- | + | Conditional | `if (item.mode === 'everyX')` | `if (true)`, `if (false)` | + | Equality | `a === b` | `a !== b` | + | Boundary | `value > 0` | `value >= 0` | + | Arithmetic | `return a + b` | `return a - b` | + | String literal | `'hello'` | `''`, `"Stryker was here!"` | + | Block statement | `{ x(); return; }` | `{}` | + | Conditional (ternary) | `cond ? a : b` | `a`, `b`, `cond ? a : a`, `cond ? b : b` | + + There are ~40 mutator categories. One source line typically produces several mutants. + +3. **For each mutant, run the test suite against the mutated code.** One of these outcomes: + + | Outcome | Meaning | + | --- | --- | + | **Killed** | At least one test failed → tests caught the change. ✓ | + | **Survived** | All tests passed → tests didn't catch the change. ✗ | + | **NoCoverage** | No test even ran the mutated line. | + | **Timeout** | Tests hung (counted as detected). | + +4. **Mutation score** = `(killed + timeout) / (killed + timeout + survived + no_coverage)`. Higher = more load-bearing assertions. + +### Line coverage vs mutation score — a real example + +`packages/workflow/src/workflow-checksum.ts`: + +- Line coverage: **87.09%** +- Mutation score: **38.64%** + +Mutating `let hexString = ''` to `let hexString = "Stryker was here!"` survived the test suite. The tests assert that two similar workflows produce different checksums — but never pin the actual output format. Line coverage calls this fine; mutation testing flags it as assertion-light test theatre. + +That divergence is exactly why this project exists. + +--- + +## What's in this directory + +| File | Purpose | +| --- | --- | +| `pick-next.mjs` | Walk `/src/`, merge with the live ledger, return the next source file to mutate | +| `emit-payload.mjs` | Turn a Stryker `summary.json` into a BQ-ready writer payload | + +The Stryker run itself lives in `packages/workflow/scripts/mutate.mjs` and is invoked via `pnpm --filter=n8n-workflow mutate `. + +The reader and writer webhooks are plain HTTP — the GHA hits them with `curl`. There is no fetch/post wrapper script; if you want to call them locally, see [Local usage](#local-usage). + +The BQ table schema lives with the writer workflow (in n8n's internal Quality project), not in this repo — the writer owns the MERGE statement and is the single source of truth. + +## End-to-end pipeline + +``` +[GHA nightly cron, .github/workflows/mutation-health-nightly.yml] + │ + ├─► curl GET reader webhook → live-ledger.json (current BQ state) + │ │ + │ └─► [n8n: QA Mutation Health Reader] ──► SELECT from BQ ledger + │ + ├─► pick-next.mjs → one source file + │ walks /src/, merges with live ledger + │ files missing from ledger are synthesised as `new` + │ priority: new → red → stale → skip green + │ within new: alphabetical + │ within red/stale: lowest score first + │ + ├─► pnpm --filter=n8n-workflow mutate → summary.json + │ + ├─► emit-payload.mjs → bq-payload.json + │ + └─► curl POST writer webhook → INSERTs event + MERGEs ledger row + ↓ + [n8n writer workflow: QA: Mutation Health Writer] + ↓ + ┌───────────────────────────────────┐ + │ qa_mutation_health_ledger (MERGE) │ + │ qa_performance_metrics (INSERT) │ + └───────────────────────────────────┘ +``` + +The writer workflow lives in n8n's internal Quality project. It's created and maintained outside this repo. This README documents the contract it implements. + +## State transitions + +| Trigger | Stored `status` | +| --- | --- | +| Source file in `src/` but no row yet | synthesised as `new` at pick time; not stored | +| Last run scored ≥ `threshold_at_run` | `green` | +| Last run scored < `threshold_at_run` | `red` | + +Stored statuses are just two: `red` and `green`. `new` is computed in-memory by the picker for any file in the source tree that has no ledger row yet — the row is only persisted after that file's first scored run. The picker also computes a transient `stale` state — any `green` row whose `last_checked_at` is older than 4 weeks is treated as `stale` for that pick. No `last_checked_sha` is needed; no git history is consulted. + +Picker priority: `new` → `red` → `stale` → skip fresh `green`. + +- Within `new`: alphabetical (rows exit the bucket as they're scored) +- Within `red`: lowest score first (weakest tests revisited first) +- Within `stale`: oldest `last_checked_at` first (natural cycling of long-stable files) + +If every row is green and fresh, the picker exits 0 with `{"picked": null, "reason": "all-green"}` — a healthy "nothing to do" state, not a failure. + +## Webhook contracts + +Two n8n workflows back the pipeline. Both live in the internal Quality project (`L8csxtEbFpFOWlf8`) and are created/maintained outside this repo. Both run unauthenticated (URL-as-secret pattern, matching existing `qa_*` workers). + +| Endpoint | Method | Workflow | Purpose | +| --- | --- | --- | --- | +| `https://internal.users.n8n.cloud/webhook/mutation-health-writer` | POST | `QA: Mutation Health Writer` (`iYEBmBat8OscRTVq`) | INSERT events + MERGE ledger | +| `https://internal.users.n8n.cloud/webhook/mutation-health-ledger?package=` | GET | `QA: Mutation Health Reader` (`ZmRsNUwvgfCSq0JI`) | Read current ledger state | + +### Writer webhook + +`POST https://internal.users.n8n.cloud/webhook/mutation-health-writer` with `Content-Type: application/json`: + +```json +{ + "ledger": [ + { + "source_file_path": "packages/workflow/src/cron.ts", + "package": "n8n-workflow", + "last_score": 95.12, + "threshold_at_run": 80, + "last_checked_at": "2026-05-22T10:03:55.660Z", + "status": "green", + "mutants_killed": 39, + "mutants_survived": 2, + "mutants_no_coverage": 0, + "mutants_timeout": 0 + } + ], + "events": [ + { + "benchmark_name": "mutation_health", + "value": 95.12, + "timestamp": "2026-05-22T10:03:55.660Z", + "dimensions": { + "package": "n8n-workflow", + "source_file": "packages/workflow/src/cron.ts", + "sha": "095239e175", + "status_after": "green", + "threshold": 80, + "mutants_killed": 39, + "mutants_survived": 2, + "mutants_no_coverage": 0, + "mutants_timeout": 0 + } + } + ] +} +``` + +Either array may be empty (manual smoke tests sometimes send only `events`). + +The writer: + +1. For each `events[]` row → `INSERT` into `qa_performance_metrics`. +2. For each `ledger[]` row → `MERGE` into `qa_mutation_health_ledger` on `source_file_path`. Status is always `red` or `green` — the picker synthesises `new` in-memory and never posts it. + +The webhook URL is delivered to GHA via the `MUTATION_HEALTH_WEBHOOK` repo secret. The secret URL itself is the only auth (matches existing `qa_*` writer pattern); rotate the secret if leaked. + +### Reader webhook + +`GET https://internal.users.n8n.cloud/webhook/mutation-health-ledger?package=`: + +```json +{ + "ledger": [ + { + "source_file_path": "packages/workflow/src/cron.ts", + "package": "n8n-workflow", + "last_score": 95.12, + "threshold_at_run": 80, + "last_checked_at": "2026-05-22T10:03:55.660Z", + "status": "green", + "mutants_killed": 39, + "mutants_survived": 2, + "mutants_no_coverage": 0, + "mutants_timeout": 0 + } + ] +} +``` + +The `package` query param is validated server-side against the same pnpm-workspace allowlist regex used elsewhere in the pipeline; invalid input returns 500. No SQL is constructed or accepted on the client side — the SELECT is hardcoded in the workflow. + +Unauthenticated — the URL is not a secret. The data isn't sensitive (file paths + integer scores), but treat the URL as low-trust: anyone with it can read all current ledger state for the queried package. + +## Threshold (provisional) + +Runs use `STRYKER_THRESHOLD=80` as a placeholder. The threshold moves to evidence-based after ~4 weeks of accumulated data. Until then, treat `red`/`green` verdicts as preliminary. + +## Local usage + +```bash +# Run Stryker on one file (the inner loop — also invokable via /n8n:mutation-test skill) +pnpm --filter=n8n-workflow mutate src/cron.ts + +# Pull current ledger from BQ +curl --fail -sS \ + 'https://internal.users.n8n.cloud/webhook/mutation-health-ledger?package=n8n-workflow' \ + -o /tmp/ledger.json + +# Pick the next file to score +node scripts/mutation-health/pick-next.mjs \ + --package-dir packages/workflow \ + --ledger-file /tmp/ledger.json + +# Build a BQ payload from a Stryker run +node scripts/mutation-health/emit-payload.mjs \ + --summary packages/workflow/reports/mutation/summary.json \ + --package n8n-workflow + +# POST the result (requires MUTATION_HEALTH_WEBHOOK to be set) +curl --fail -sS -X POST \ + -H 'Content-Type: application/json' \ + --data @packages/workflow/reports/mutation/bq-payload.json \ + "$MUTATION_HEALTH_WEBHOOK" +``` diff --git a/scripts/mutation-health/emit-payload.mjs b/scripts/mutation-health/emit-payload.mjs new file mode 100644 index 00000000000..434dfd61bc8 --- /dev/null +++ b/scripts/mutation-health/emit-payload.mjs @@ -0,0 +1,118 @@ +#!/usr/bin/env node +/** + * Emit a BQ payload from a mutate.mjs summary.json. + * + * Input: packages//reports/mutation/summary.json (from `pnpm mutate `) + * Output: JSON document with two keys, `ledger` (rows for qa_mutation_health_ledger) + * and `events` (rows for qa_performance_metrics, benchmark_name="mutation_health"). + * + * The output is what the n8n writer workflow consumes via webhook. This script + * intentionally does NOT call BigQuery directly — the writer workflow owns + * BQ credentials and the MERGE statement for the ledger upsert. + * + * Usage: + * node scripts/mutation-health/emit-payload.mjs \ + * --summary packages/workflow/reports/mutation/summary.json \ + * --package n8n-workflow \ + * [--out ] # default: /reports/mutation/bq-payload.json + */ + +import { execFileSync } from 'node:child_process'; +import { readFile, writeFile, mkdir } from 'node:fs/promises'; +import { existsSync } from 'node:fs'; +import path from 'node:path'; + +function die(code, msg) { + process.stderr.write(`${msg}\n`); + process.exit(code); +} + +function parseArgs(argv) { + const out = {}; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (!a.startsWith('--')) continue; + const key = a.slice(2); + const next = argv[i + 1]; + if (next === undefined || next.startsWith('--')) { + out[key] = true; + } else { + out[key] = next; + i++; + } + } + return out; +} + +const args = parseArgs(process.argv.slice(2)); + +const summaryPath = args.summary; +const pkg = args.package; + +if (!summaryPath) die(2, 'Missing required --summary '); +if (!pkg) die(2, 'Missing required --package '); +if (!existsSync(summaryPath)) die(2, `Summary not found: ${summaryPath}`); + +const sha = execFileSync('git', ['rev-parse', 'HEAD'], { encoding: 'utf8' }).trim(); + +const summary = JSON.parse(await readFile(summaryPath, 'utf8')); + +if (!Array.isArray(summary.files)) { + die(2, 'Summary missing `files` array.'); +} + +// pkg-root = two dirs up from the summary (reports/mutation/summary.json) +const pkgRoot = path.resolve(path.dirname(summaryPath), '../..'); +const repoRoot = path.resolve( + execFileSync('git', ['rev-parse', '--show-toplevel'], { encoding: 'utf8' }).trim(), +); +const pkgRelToRepo = path.relative(repoRoot, pkgRoot); + +const threshold = Number(summary.threshold); +const timestamp = summary.generatedAt; + +const ledger = []; +const events = []; + +for (const f of summary.files) { + const sourceRel = path.posix.join(pkgRelToRepo, f.file); + const status = f.thresholdMet ? 'green' : 'red'; + + ledger.push({ + source_file_path: sourceRel, + package: pkg, + last_score: f.score, + threshold_at_run: threshold, + last_checked_at: timestamp, + status, + mutants_killed: f.counts.killed, + mutants_survived: f.counts.survived, + mutants_no_coverage: f.counts.noCoverage, + mutants_timeout: f.counts.timeout, + }); + + events.push({ + benchmark_name: 'mutation_health', + value: f.score, + timestamp, + dimensions: { + package: pkg, + source_file: sourceRel, + sha, + status_after: status, + threshold, + mutants_killed: f.counts.killed, + mutants_survived: f.counts.survived, + mutants_no_coverage: f.counts.noCoverage, + mutants_timeout: f.counts.timeout, + }, + }); +} + +const outPath = args.out ?? path.join(pkgRoot, 'reports/mutation/bq-payload.json'); +await mkdir(path.dirname(outPath), { recursive: true }); +await writeFile(outPath, JSON.stringify({ ledger, events }, null, 2)); + +process.stderr.write( + `Emitted ${ledger.length} ledger row(s) + ${events.length} event row(s) → ${outPath}\n`, +); diff --git a/scripts/mutation-health/pick-next.mjs b/scripts/mutation-health/pick-next.mjs new file mode 100644 index 00000000000..1ca1cc0a7b2 --- /dev/null +++ b/scripts/mutation-health/pick-next.mjs @@ -0,0 +1,228 @@ +#!/usr/bin/env node +/** + * Walk a package's source tree, merge with the live BQ ledger snapshot, + * return the next pair to mutate. + * + * Files present in src/ but absent from the live ledger are synthesised as + * status='new'. No separate seed step needed — the ledger fills in + * organically as files get scored. + * + * Stored statuses (from BQ): new | red | green + * Effective statuses (computed at pick time): new | red | stale | green + * + * Picker priority: new → red → stale → skip green + * Tiebreaks within each bucket: + * - new: alphabetical by source_file_path + * - red: lowest score first (focus on weakest tests) + * - stale: oldest last_checked_at first (natural cycling) + * + * "Stale" is an in-memory promotion of green rows older than + * STALE_AFTER_WEEKS (default 4). Not stored. + * + * Inputs: + * --package-dir Required. Repo-relative path to the package, e.g. packages/workflow + * --ledger-file Required. Live ledger JSON: { "ledger": [ ... ] } + * --stale-after-weeks Optional. Default 4. + * + * Output (stdout): { picked: { source_file_path, package, prior_status, effective_status } } + * OR { picked: null, reason: "all-green" | "empty-source-tree" }. + * + * Exit codes: + * 0 — picked a row OR nothing to do (with picked: null sentinel) + * 2 — usage / config error + */ + +import { readdir, readFile } from 'node:fs/promises'; +import { existsSync } from 'node:fs'; +import { execFileSync } from 'node:child_process'; +import path from 'node:path'; + +function die(code, msg) { + process.stderr.write(`${msg}\n`); + process.exit(code); +} + +function parseArgs(argv) { + const out = {}; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (!a.startsWith('--')) continue; + const key = a.slice(2); + const next = argv[i + 1]; + if (next === undefined || next.startsWith('--')) { + out[key] = true; + } else { + out[key] = next; + i++; + } + } + return out; +} + +// Files with no useful mutation surface: barrels, declarations, type-only modules. +const LOW_VALUE_BASENAMES = new Set(['interfaces', 'index', 'constants', 'types']); + +function isMutationWorthy(absPath) { + if (absPath.endsWith('.d.ts')) return false; + const base = path.basename(absPath, '.ts'); + if (LOW_VALUE_BASENAMES.has(base)) return false; + return true; +} + +async function walkSources(dir) { + const entries = await readdir(dir, { withFileTypes: true }); + const out = []; + for (const e of entries) { + const full = path.join(dir, e.name); + if (e.isDirectory()) { + out.push(...(await walkSources(full))); + } else if (e.isFile() && e.name.endsWith('.ts')) { + out.push(full); + } + } + return out; +} + +const args = parseArgs(process.argv.slice(2)); + +const STALE_AFTER_WEEKS_DEFAULT = 4; +const staleArg = args['stale-after-weeks']; +const parsedStale = Number(staleArg); +let STALE_AFTER_WEEKS; +if (staleArg === undefined) { + STALE_AFTER_WEEKS = STALE_AFTER_WEEKS_DEFAULT; +} else if (Number.isFinite(parsedStale) && parsedStale > 0) { + STALE_AFTER_WEEKS = parsedStale; +} else { + process.stderr.write( + `Invalid --stale-after-weeks=${staleArg}, falling back to ${STALE_AFTER_WEEKS_DEFAULT}.\n`, + ); + STALE_AFTER_WEEKS = STALE_AFTER_WEEKS_DEFAULT; +} + +const pkgDirArg = args['package-dir']; +const ledgerFile = args['ledger-file']; +if (!pkgDirArg) die(2, 'Missing required --package-dir '); +if (!ledgerFile) die(2, 'Missing required --ledger-file '); + +const repoRoot = path.resolve( + execFileSync('git', ['rev-parse', '--show-toplevel'], { encoding: 'utf8' }).trim(), +); +const pkgDir = path.isAbsolute(pkgDirArg) ? pkgDirArg : path.join(repoRoot, pkgDirArg); +if (!existsSync(pkgDir)) die(2, `Package dir not found: ${pkgDir}`); + +const pkgJsonPath = path.join(pkgDir, 'package.json'); +if (!existsSync(pkgJsonPath)) die(2, `No package.json at ${pkgJsonPath}`); +const pkgName = JSON.parse(await readFile(pkgJsonPath, 'utf8')).name; + +const srcDir = path.join(pkgDir, 'src'); +if (!existsSync(srcDir)) die(2, `No src/ in ${pkgDir}`); + +const ledgerPath = path.isAbsolute(ledgerFile) ? ledgerFile : path.join(process.cwd(), ledgerFile); +if (!existsSync(ledgerPath)) die(2, `Ledger file not found: ${ledgerPath}`); + +const ledgerPayload = JSON.parse(await readFile(ledgerPath, 'utf8')); +const liveLedger = ledgerPayload.ledger; +if (!Array.isArray(liveLedger)) die(2, 'Ledger payload missing `ledger` array.'); + +const allSources = (await walkSources(srcDir)).sort(); +const worthy = allSources.filter(isMutationWorthy).map((abs) => path.relative(repoRoot, abs)); + +if (worthy.length === 0) { + process.stderr.write('No mutation-worthy source files found under src/.\n'); + process.stdout.write(JSON.stringify({ picked: null, reason: 'empty-source-tree' }) + '\n'); + process.exit(0); +} + +// Merge: live ledger row wins over synthesised "new" row. +const byPath = new Map(); +for (const row of liveLedger) { + byPath.set(row.source_file_path, row); +} +const merged = worthy.map( + (p) => + byPath.get(p) ?? { + source_file_path: p, + package: pkgName, + last_score: null, + threshold_at_run: null, + last_checked_at: null, + status: 'new', + }, +); + +const NOW = Date.now(); +const STALE_AFTER_MS = STALE_AFTER_WEEKS * 7 * 24 * 60 * 60 * 1000; + +function computeEffectiveStatus(row) { + if (row.status === 'new') return 'new'; + if (row.status === 'red') return 'red'; + // status === 'green' — promote to 'stale' if old enough + if (row.last_checked_at) { + const age = NOW - Date.parse(row.last_checked_at); + if (age > STALE_AFTER_MS) return 'stale'; + } + return 'green'; +} + +const PRIORITY = { new: 0, red: 1, stale: 2, green: 3 }; + +const annotated = merged.map((row) => ({ ...row, effective_status: computeEffectiveStatus(row) })); + +annotated.sort((a, b) => { + const pa = PRIORITY[a.effective_status] ?? 99; + const pb = PRIORITY[b.effective_status] ?? 99; + if (pa !== pb) return pa - pb; + + if (a.effective_status === 'new') { + return a.source_file_path.localeCompare(b.source_file_path); + } + + if (a.effective_status === 'red') { + const sa = a.last_score == null ? Infinity : Number(a.last_score); + const sb = b.last_score == null ? Infinity : Number(b.last_score); + if (sa !== sb) return sa - sb; + return a.source_file_path.localeCompare(b.source_file_path); + } + + // stale: oldest last_checked_at first + const ta = a.last_checked_at ? Date.parse(a.last_checked_at) : 0; + const tb = b.last_checked_at ? Date.parse(b.last_checked_at) : 0; + if (ta !== tb) return ta - tb; + return a.source_file_path.localeCompare(b.source_file_path); +}); + +const counts = annotated.reduce((acc, r) => { + acc[r.effective_status] = (acc[r.effective_status] ?? 0) + 1; + return acc; +}, {}); + +process.stderr.write( + `Source files: ${worthy.length} • ` + + `new=${counts.new ?? 0} red=${counts.red ?? 0} stale=${counts.stale ?? 0} green=${counts.green ?? 0}\n`, +); + +const top = annotated[0]; + +if (top.effective_status === 'green') { + process.stderr.write(`All actionable rows green (stale threshold ${STALE_AFTER_WEEKS} weeks) — nothing to do.\n`); + process.stdout.write(JSON.stringify({ picked: null, reason: 'all-green' }) + '\n'); + process.exit(0); +} + +process.stderr.write( + `Picked: ${top.source_file_path}\n` + + ` priority=${top.effective_status} ` + + `(was ${top.status}, last_checked_at=${top.last_checked_at ?? 'never'})\n`, +); + +process.stdout.write( + JSON.stringify({ + picked: { + source_file_path: top.source_file_path, + package: top.package, + prior_status: top.status, + effective_status: top.effective_status, + }, + }) + '\n', +);