mirror of
https://github.com/n8n-io/n8n.git
synced 2026-10-11 22:50:06 +00:00
chore: Stryker mutation testing for packages/workflow (no-changelog) (#30956)
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
Declan Carroll
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
76c432c53f
commit
3bff0f52b9
@@ -0,0 +1,124 @@
|
||||
---
|
||||
description: Run Stryker mutation testing on the source files changed in the current branch (vs origin/master). One command for "did my work hold up under mutation?" before pushing. Triages on the side which files dropped below threshold and offers to invoke n8n:strengthen-tests on them. Use when the user says /mutate-changed, "mutate what I changed", "check my changes", or has just finished writing a feature and wants pre-merge feedback. Scope: only packages/workflow/src/** changes are mutated today.
|
||||
---
|
||||
|
||||
# Mutate what I changed
|
||||
|
||||
Closes the local dev loop. Single command to run Stryker against every source file the current branch touched (vs `origin/master`), then point at any reds that need strengthening.
|
||||
|
||||
## When to use
|
||||
|
||||
- User says `/mutate-changed`, "mutate the files I changed", "check my changes", "did my tests stick"
|
||||
- Mid-feature: dev wants pre-merge feedback before pushing
|
||||
- Pre-PR: cheaper than waiting for the nightly cron
|
||||
|
||||
**Don't** use:
|
||||
- For a single specific file (`/n8n:mutation-test <path>` is faster)
|
||||
- For non-`packages/workflow` changes — Stryker is only wired up there today
|
||||
- After the user already ran `/n8n:strengthen-tests` (which calls mutation-test internally for verification — running both again is wasted compute)
|
||||
|
||||
## Inputs
|
||||
|
||||
- **Default base**: `origin/master`. Override with `--base <ref>` if comparing against another branch (e.g. `--base HEAD~5`).
|
||||
- **Default scope**: `packages/workflow/src/**/*.ts`. The only package with Stryker wired up today.
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Identify changed source files
|
||||
|
||||
```bash
|
||||
git diff --name-only origin/master...HEAD -- 'packages/workflow/src/**/*.ts'
|
||||
```
|
||||
|
||||
(`...` is correct — three-dot means "since the branch diverged from base," which is what we want.)
|
||||
|
||||
If `git fetch` hasn't been run recently, suggest the user `git fetch origin master` first; otherwise the base ref is stale.
|
||||
|
||||
Filter out:
|
||||
- `**/*.d.ts` (declarations, no behaviour)
|
||||
- `**/*.stories.ts` (Storybook scaffolding, not present in workflow but defensive)
|
||||
- `index.ts` files (barrels)
|
||||
- `interfaces.ts`, `types.ts`, `constants.ts` (same low-value filter as `seed-ledger.mjs`)
|
||||
|
||||
### 2. Surface the plan to the user
|
||||
|
||||
Print the filtered list before running anything. Each Stryker run is 1–5 minutes; the user should confirm if there are many.
|
||||
|
||||
```
|
||||
Found N changed source files to mutate:
|
||||
- packages/workflow/src/foo.ts
|
||||
- packages/workflow/src/bar.ts
|
||||
...
|
||||
|
||||
Estimated runtime: ~M-K minutes (M minutes minimum if every Stryker run is fast).
|
||||
Proceed? (skill default: yes if N ≤ 3, ask if N > 3)
|
||||
```
|
||||
|
||||
If the filtered list is empty: report "No source files under packages/workflow/src/** changed vs $base — nothing to mutate." and stop. Exit cleanly.
|
||||
|
||||
If N > 8: refuse and ask the user to narrow scope (a different base ref, or invoke per-file). Running 8+ mutations sequentially is a 30+ minute session that should be a deliberate choice.
|
||||
|
||||
### 3. Run mutation testing per file
|
||||
|
||||
For each file in the plan, invoke `pnpm --filter=n8n-workflow mutate <package-relative-path>`. The `summary.json` and other artefacts get overwritten on each run, so capture the score per file as you go.
|
||||
|
||||
After each run completes, print one line:
|
||||
|
||||
```
|
||||
✓ src/foo.ts 95.12% (39/41 killed) GREEN
|
||||
✗ src/bar.ts 54.83% (17/31 killed) RED — 13 survivors, top: ConditionalExpression, EqualityOperator
|
||||
```
|
||||
|
||||
If a Stryker run hard-fails (exit 3, no `summary.json`), print `! src/foo.ts Stryker failed — see stderr` and continue to the next file. Don't abort the whole batch.
|
||||
|
||||
### 4. Summary table
|
||||
|
||||
After all files have been mutated, print one compact table:
|
||||
|
||||
```
|
||||
=== Mutation results: N files, M green, K red, J failed ===
|
||||
| File | Score | Verdict | Survivors |
|
||||
|-----------------------------|---------|---------|-----------|
|
||||
| src/foo.ts | 95.12% | GREEN | 2 |
|
||||
| src/bar.ts | 54.83% | RED | 13 |
|
||||
| src/baz.ts | n/a | FAILED | - |
|
||||
```
|
||||
|
||||
### 5. Offer the strengthen step on the worst red file
|
||||
|
||||
If any file came back red:
|
||||
|
||||
> The lowest-score red file is `src/bar.ts` (54.83%, 13 survivors). Run `/n8n:strengthen-tests` to triage them and write assertion changes? (suggesting; don't auto-invoke)
|
||||
|
||||
Only suggest one file at a time — `n8n:strengthen-tests` caps at 5 survivors per invocation, and re-running this skill after edits is cheap.
|
||||
|
||||
If everything is green: report it and stop. No follow-up needed.
|
||||
|
||||
## Output shape
|
||||
|
||||
Three deliverable sections per invocation:
|
||||
|
||||
1. **Plan** (before running) — list of files, estimated runtime
|
||||
2. **Per-file progress** (during) — one line per file as it completes
|
||||
3. **Summary table + recommendation** (after) — compact view
|
||||
|
||||
Don't dump full `summary.json` payloads — the per-file mutate runs already write them to disk under `packages/workflow/reports/mutation/` (overwriting each time, since the orchestrator uses fixed filenames). The user can read the latest one if they want detail.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Hardcoded to `packages/workflow`.** Generalise when Stryker is wired up to other packages.
|
||||
- **Max 8 files per invocation.** Above that, ask user to narrow.
|
||||
- **Don't auto-invoke `/n8n:strengthen-tests`.** Suggest, don't act. Same reasoning as the other skills: each pass should be a deliberate human-approved step.
|
||||
- **No commits.** Edits land in working tree; user reviews.
|
||||
- **No fabricated scores.** If a Stryker run fails, mark FAILED in the table — never guess a value.
|
||||
|
||||
## Common follow-ups
|
||||
|
||||
- "strengthen them all" → loop the user through `/n8n:strengthen-tests`, one file at a time
|
||||
- "what changed?" → `git diff origin/master...HEAD -- <file>` for the file in question
|
||||
- "ignore <pattern>" → re-run with the user's exclude added to the filter
|
||||
|
||||
## Related
|
||||
|
||||
- `n8n:mutation-test` — single-file version of this skill
|
||||
- `n8n:strengthen-tests` — the natural next step when reds show up
|
||||
@@ -0,0 +1,115 @@
|
||||
---
|
||||
description: Run Stryker mutation testing on a single source file and return a structured, token-frugal report that's pipeable to a follow-up "strengthen tests" loop. Use when the user says /mutation-test, "mutation test this file", or has just edited tests and wants to verify they actually assert behaviour. Per-file only — full-package mutation runs are out of scope.
|
||||
---
|
||||
|
||||
# Mutation testing — single file
|
||||
|
||||
Wraps `pnpm --filter=<pkg> mutate <file>` and parses `summary.json` into a compact, structured shape suitable for downstream "strengthen the surviving mutants" iteration.
|
||||
|
||||
## When to use
|
||||
|
||||
- User explicitly invokes: `/mutation-test <path>`, "mutation test this file", "check my test effectiveness on X"
|
||||
- User has just edited a test file and wants to know if their assertions are load-bearing
|
||||
- Follow-up loop after a `red` verdict — feed the structured output back to a "fix" iteration
|
||||
|
||||
**Don't** use this skill for:
|
||||
- Whole-package or whole-repo mutation runs — single file only
|
||||
- Coverage % questions (use the existing coverage workflow)
|
||||
- Files outside `packages/workflow/` — Stryker is only wired up there today
|
||||
|
||||
## Inputs
|
||||
|
||||
One required argument: the source file to mutate, as either a repo-relative path or a package-relative path. Examples that all mean the same thing:
|
||||
|
||||
- `packages/workflow/src/cron.ts`
|
||||
- `src/cron.ts` (assumes packages/workflow)
|
||||
- `workflow/src/cron.ts` (assumes packages/)
|
||||
|
||||
If ambiguous, ask the user once which package; do not guess.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Resolve package + relative source path.** Today only `n8n-workflow` (`packages/workflow`) has Stryker wired. If the user passes a file outside that, say so and stop — don't fabricate output.
|
||||
|
||||
2. **Run Stryker with trimmed output:**
|
||||
```bash
|
||||
pnpm --filter=n8n-workflow mutate <package-relative-src> 2>&1 | tail -40
|
||||
```
|
||||
`tail -40` discards the Stryker progress bar spam; the relevant numbers + survivor list always land in the last ~30 lines. Exit codes: `0` = pass, `1` = below threshold (still valid, summary.json exists), `2` = usage error, `3` = Stryker failure (no summary.json).
|
||||
|
||||
3. **If exit code 3**, surface the trimmed tail to the user, suggest checking that workspace deps are built (`pnpm build`), and stop. Don't fabricate a report.
|
||||
|
||||
4. **Read `packages/workflow/reports/mutation/summary.json`** — never `raw.json`. raw.json is 600KB+ and not needed for the strengthen loop. summary.json already contains every surviving mutant with its location, replacement, mutator name, and the names of tests that covered the line.
|
||||
|
||||
5. **Cap covering_tests at 3 per survivor.** If a mutant was covered by more than 3 tests, keep the first 3 and append `+N more` as a count. Names beyond 3 add tokens without adding actionable signal — the strengthen loop only needs to know *which test* to extend, not all of them.
|
||||
|
||||
6. **Compute `minimum_kills_needed`** to reach the threshold:
|
||||
```
|
||||
killed_now = summary.overall.counts.killed + summary.overall.counts.timeout
|
||||
valid_total = killed_now + summary.overall.counts.survived + summary.overall.counts.noCoverage
|
||||
needed = ceil((threshold/100) * valid_total) - killed_now
|
||||
```
|
||||
This tells the next loop the minimum number of survivors it has to kill to flip `red` → `green`. Cap at the number of survivors.
|
||||
|
||||
7. **Output the structured shape** described below. Keep prose to one headline line; the rest is the JSON block.
|
||||
|
||||
## Output shape
|
||||
|
||||
One headline line, then a fenced JSON block. Nothing else — no preamble, no per-survivor commentary, no risk triage (that's the next loop's job).
|
||||
|
||||
````
|
||||
[red|green] <score>% (threshold <T>%) — <N> survivors; need to kill ≥<K> to flip green.
|
||||
|
||||
```json
|
||||
{
|
||||
"verdict": "red",
|
||||
"target": "packages/workflow/src/augment-object.ts",
|
||||
"package": "n8n-workflow",
|
||||
"score": 76.74,
|
||||
"threshold": 80,
|
||||
"delta_to_threshold": 3.26,
|
||||
"minimum_kills_needed": 5,
|
||||
"counts": {
|
||||
"killed": 99,
|
||||
"survived": 28,
|
||||
"no_coverage": 2,
|
||||
"timeout": 0
|
||||
},
|
||||
"survivors": [
|
||||
{
|
||||
"id": "77",
|
||||
"mutator": "ConditionalExpression",
|
||||
"location": "src/augment-object.ts:95:6",
|
||||
"original": "value === null",
|
||||
"replacement": "false",
|
||||
"covering_tests": [
|
||||
"augmentObject should handle null values",
|
||||
"augmentObject should handle nested nulls"
|
||||
],
|
||||
"covering_tests_overflow": 0
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
````
|
||||
|
||||
Order the survivors array by `location` (ascending line number, then column) so the strengthen loop processes them top-to-bottom of the file.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **No raw.json** — never read or surface it. summary.json is the only input.
|
||||
- **No HTML report** — don't `open` raw.html or paste links to it. If the user wants visual exploration they'll ask.
|
||||
- **No automatic triage** — don't categorise survivors by "real bug" vs "refactor insurance." That's a separate analysis step that should happen on demand, not by default. Keeps token cost predictable.
|
||||
- **No "I'll regenerate tests for you now"** — this skill reports the gap. Use `n8n:strengthen-tests` if you want assertion edits.
|
||||
|
||||
## Common follow-ups (don't do unless asked)
|
||||
|
||||
- User says "fix these" → start a strengthen loop using the JSON output as input. Read covering_tests source, propose changes per mutant, run the skill again to verify.
|
||||
- User says "explain survivor #N" → fetch that mutant from summary.json, show its surrounding ~5 lines from the source file, no analysis beyond what summary.json contains.
|
||||
- User says "what's the threshold?" → 80% provisional; see `scripts/mutation-health/README.md` for the rationale.
|
||||
- User says "run it on the changed files" → not wired yet. Suggest `git diff` to find candidates, then invoke this skill per file.
|
||||
|
||||
## Related
|
||||
|
||||
- `scripts/mutation-health/README.md` — the broader BQ-backed observability story
|
||||
- `packages/workflow/stryker.config.mjs` — the Stryker config this skill drives
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
description: Take a Stryker summary.json (from n8n:mutation-test), triage the surviving mutants by user-reachable-behaviour risk, write minimal assertion changes to kill the top 3-5 highest-leverage survivors, then verify by re-running n8n:mutation-test. Use when the user has just run mutation testing and wants to strengthen the test suite, or says "kill the survivors / strengthen tests / fix the red." Pairs with n8n:mutation-test as the inner write side of a single iteration.
|
||||
---
|
||||
|
||||
# Strengthen tests — kill the highest-leverage survivors
|
||||
|
||||
The other half of the local mutation-testing loop. `n8n:mutation-test` reports which mutations escaped the tests; this skill picks the ones that matter and writes minimal assertion changes to kill them.
|
||||
|
||||
## When to use
|
||||
|
||||
- User has just run `/n8n:mutation-test <file>` and the verdict was `red`
|
||||
- User says: "strengthen tests", "kill the survivors", "fix the red", "iterate on the tests for X"
|
||||
- Mid-loop: this skill's verify step calls `n8n:mutation-test` again, so the loop closes here
|
||||
|
||||
**Don't** use this skill:
|
||||
- Before any mutation testing has been run for the target file (no `summary.json` to triage)
|
||||
- For a `green` verdict — there's nothing to strengthen; if user insists, push back and ask which file actually needs work
|
||||
- To bulk-kill every survivor — explicitly capped at 5 per invocation. Re-invoke for more.
|
||||
|
||||
## Inputs
|
||||
|
||||
- **Default**: read `packages/workflow/reports/mutation/summary.json` (the last `n8n:mutation-test` run's output).
|
||||
- **Override**: `--summary <path>` to point at a different summary file.
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Read the summary
|
||||
|
||||
`packages/workflow/reports/mutation/summary.json`. Already compact (~50 KB). Pull:
|
||||
- `files[0].file` — the source file under test
|
||||
- `files[0].score` — current mutation score
|
||||
- `files[0].survivors[]` — every surviving (and no-coverage) mutant with location, replacement, covering test names
|
||||
|
||||
If `summary.json` is missing, stop. Tell the user to run `n8n:mutation-test` first.
|
||||
|
||||
### 2. Read the source under test, sparingly
|
||||
|
||||
Read the source file referenced in `summary.json`. Read **once**, the whole file (typical n8n-workflow source files are 50-500 lines; the cost is bounded). This is the only file read; don't load test files yet.
|
||||
|
||||
### 3. Triage the survivors
|
||||
|
||||
Categorise each survivor into one of three buckets. Use the rubric below — don't apply it mechanically, but lean on it.
|
||||
|
||||
**HIGH leverage — "real regression vector":** mutant guards behaviour that real users actually hit through the public API surface.
|
||||
|
||||
- Type checks against shapes user data routinely takes: `null`, `undefined`, `Date`, `Buffer`, `Uint8Array`, `Array`, plain objects with arbitrary keys
|
||||
- Conditional branches that gate critical fall-through (e.g. "if this returns early, the wrong code path runs")
|
||||
- Code paths that handle user-controlled flow: expressions, Code-node behaviour, binary item handling, deep-clone semantics
|
||||
- Mutants on guard clauses that prevent crashes (`if (value == null) return ...`)
|
||||
|
||||
**MODERATE leverage — "user-observable invariant":** real but lower-frequency user impact.
|
||||
|
||||
- Proxy traps that change `Object.keys` / `in` / `hasOwnProperty` semantics after mutation
|
||||
- Set-then-delete / re-set-after-delete sequences (Code-node assignment patterns)
|
||||
- Treating `undefined` assignment as deletion (`obj.foo = undefined` → key removed)
|
||||
- Edge cases that only fire under specific iteration patterns
|
||||
|
||||
**LOW leverage — "refactor insurance" or noise:** skip these. Document the skip in the output but don't write tests for them.
|
||||
|
||||
- Constructor property checks (`arr.constructor === Array`) unless production code is known to rely on them
|
||||
- Idempotency invariants only triggered by other library code (Lodash, native spread)
|
||||
- Equivalent mutants — mutations that produce semantically identical code (e.g. swapping an unused conditional)
|
||||
- Mutants on internal helpers users can't reach through any public API
|
||||
|
||||
If the summary lists `noCoverage` survivors, treat them as their own bucket: "the test suite doesn't even execute these lines." Triage them by the same rubric, but flag separately since they need a new test case rather than an assertion extension.
|
||||
|
||||
### 4. Pick the work set: up to 5 mutants
|
||||
|
||||
Order: all HIGH first, then MODERATE if budget remains, never LOW. Hard cap at 5 total. If HIGH alone exceeds 5, pick the 5 most distinct (don't pick 5 mutants on the same line — they probably share a fix; pick representatives across the file).
|
||||
|
||||
If fewer than 3 HIGH+MODERATE candidates exist, just do what's there. Don't pad with LOW just to hit a number.
|
||||
|
||||
Write up the work set to the user **before editing**:
|
||||
|
||||
```
|
||||
Picked N survivors to address (M skipped as refactor-insurance / low-leverage):
|
||||
1. [HIGH] location — original → replacement
|
||||
plan: assert <X> in <covering test name>
|
||||
2. [HIGH] ...
|
||||
3. [MODERATE] ...
|
||||
...
|
||||
```
|
||||
|
||||
This is the user's chance to redirect. Don't write code yet.
|
||||
|
||||
### 5. Read covering tests for the picked survivors
|
||||
|
||||
For each picked survivor, the summary lists the test names that covered the line. Find those tests in the test file (usually `packages/<pkg>/test/<source-basename>.test.ts`, but check) and read **just the relevant `test('...')` blocks** — not the whole file. Use `Grep` + `Read` with line offsets to keep token cost down.
|
||||
|
||||
Goal: understand what the existing test asserts so the new assertion is additive, not contradictory.
|
||||
|
||||
### 6. Write the changes
|
||||
|
||||
Constraints:
|
||||
- **Prefer extending an existing covering test** over adding a new one. Lower file churn, easier review.
|
||||
- **Match the existing style** — same assertion library, same matcher idioms (`.toBe` vs `.toEqual` vs `.toStrictEqual`).
|
||||
- **Minimal additions** — one or two assertions per mutant, not a new it-block per mutant.
|
||||
- **No fabrication** — only assert what the source code actually does. If you can't tell from the source, stop and ask the user.
|
||||
- **For `noCoverage` survivors**, add a new test case named after the behaviour being pinned. Place it next to related tests.
|
||||
|
||||
Use `Edit` with exact-string matches. Never rewrite entire test files.
|
||||
|
||||
### 7. Verify
|
||||
|
||||
Re-invoke `n8n:mutation-test` on the same source file. Report:
|
||||
|
||||
```
|
||||
Before: red 76.74% (28 survivors)
|
||||
After: green 82.34% (22 survivors)
|
||||
Killed: 6 of 5 targeted (1 bonus — fix for #77 also killed #78)
|
||||
Still surviving: 22 — re-invoke /n8n:strengthen-tests for another batch.
|
||||
```
|
||||
|
||||
If the score went UP but threshold still not met: the iteration is working, recommend another pass.
|
||||
If the score went DOWN or stayed the same: at least one new test isn't asserting what we think it asserts. Surface the diff to the user and stop — do not auto-revert.
|
||||
If a test now fails (not survives — actually fails): we asserted something the code doesn't do. Revert that specific assertion, leave the rest, report which one was wrong.
|
||||
|
||||
## Output shape
|
||||
|
||||
Each invocation produces:
|
||||
|
||||
1. **Work plan** (before edits) — the picked survivors and the plan for each
|
||||
2. **Diffs** (during edits) — Edit tool calls, visible in transcript
|
||||
3. **Verify** (after) — re-run + before/after comparison
|
||||
|
||||
Keep prose minimal between sections. The plan and verify steps are the structured outputs; everything else is mechanical.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **5 mutants max per invocation.** Re-invoke for more. Prevents runaway sessions on 30-survivor files.
|
||||
- **Never fabricate assertions.** If the source doesn't clearly do X, don't claim it does.
|
||||
- **No new test files unless absolutely necessary.** Extend the existing covering test file.
|
||||
- **No reverting other people's tests.** Only edit tests in the package being mutated.
|
||||
- **No re-running mutation-test more than once per invocation.** That's the verify step. Don't loop within a single invocation; let the user re-invoke.
|
||||
- **No commits.** Edits land in the working tree; user reviews and commits.
|
||||
|
||||
## Common follow-ups
|
||||
|
||||
- User says "go again" → re-invoke this skill. The summary.json now reflects the post-edit state.
|
||||
- User says "why was #N classified as LOW?" → explain the rubric application for that specific mutant, no re-triage of others.
|
||||
- User says "kill #N specifically" → override the triage for that mutant, treat it as picked.
|
||||
- User says "skip the verify step" → don't; the verify step is the contract that the edits actually moved the score.
|
||||
|
||||
## Related
|
||||
|
||||
- `n8n:mutation-test` — the read side of this loop
|
||||
- `scripts/mutation-health/README.md` — the BQ-backed observability story this slots into
|
||||
Reference in New Issue
Block a user