refactor: Group quality engineering workspaces by capability (#40007)

This commit is contained in:
Declan Carroll
2026-10-01 11:20:00 +00:00
committed by GitHub
parent 06b27d1461
commit a4b87fccbe
1775 changed files with 791 additions and 1213 deletions
+1 -1
View File
@@ -17,7 +17,7 @@ clear, when to stay silent, what never to say — lives once in
| `backend/` | Backend | `cli`, `@n8n/db`, `core`, `workflow`, node packages |
| `db-migrations/` | DB migrations | `@n8n/db` migrations + their tests in `cli` |
| `frontend/` | Frontend | `packages/frontend` |
| `qa-dx/` | QA & DX | `.github`, `docker`, `scripts`, `patches`, `packages/testing`, the lint/test/TS config packages, baselines |
| `qa-dx/` | QA & DX | `.github`, `docker`, `scripts`, `patches`, `packages/quality`, the lint/test/TS config packages, baselines |
| `testing/` | Backend + Frontend | any package with a test suite |
A directory maps to one agent unless, like `testing/`, the policy is identical
@@ -13,7 +13,7 @@ Flag a diff that **adds** entries to any of these, and ask for the fix instead:
|------|---------|
| `.code-health-baseline.json` | `@n8n/code-health` violations |
| `.boundaries-baseline.json` | `turbo boundaries` issue count |
| `packages/testing/playwright/.janitor-baseline.json` | Playwright janitor findings |
| `packages/quality/testing/playwright/.janitor-baseline.json` | Playwright janitor findings |
| `packages/cli/eslint.config.mjs` | the `misplaced-n8n-typeorm-import` and public-API allowlists, each captioned "NEVER add to this list" |
| `.code-health-baseline.json` (`lint-config-layering`) | package-wide rule downgrades left in package ESLint configs |
@@ -35,7 +35,7 @@ fed from a `needs:` list. Flag:
Jobs gated on a `ci-filter` filter only run when a matching file changed, so a
filter that is too narrow means the test quietly stops covering new code. When
a PR adds a directory that an existing filter was meant to cover — a new package
under `packages/testing/`, a new script under `.github/scripts/` — check the
under `packages/quality/`, a new script under `.github/scripts/` — check the
filter still matches it.
## Least privilege
+1 -1
View File
@@ -32,7 +32,7 @@ Based on the affected area, pick the test layer and pattern:
| CLI / API | Vitest integration | setupTestServer + supertest | `packages/cli/test/integration/` |
| Config | Vitest unit | GlobalConfig + Container | `packages/@n8n/config/src/configs/__tests__/` |
| Editor UI | Vitest | Vue Test Utils + Pinia | `packages/frontend/editor-ui/src/**/__tests__/` |
| E2E / Canvas | Playwright | Test containers + composables | `packages/testing/playwright/` |
| E2E / Canvas | Playwright | Test containers + composables | `packages/quality/testing/playwright/` |
## Step 3: Locate Source Files
+1 -1
View File
@@ -8,7 +8,7 @@ exclude:
# Machine-generated instance-AI expectation traces. Contain ephemeral
# credential reference IDs (nanoids) that re-trip the secrets scanner
# on every regeneration. No secret material.
- packages/testing/playwright/expectations
- packages/quality/testing/playwright/expectations
# Test code repo-wide. Fixtures routinely contain synthetic PEM blocks,
# basic-auth strings, and deliberately fake keys (including the secret
+17 -17
View File
@@ -435,12 +435,12 @@
"hash": "4a6882cb8d62"
}
],
"packages/@n8n/benchmark/package.json": [
"packages/quality/efficiency/scale/benchmark/package.json": [
{
"rule": "catalog-violations",
"line": 38,
"message": "@oclif/core appears in 4 packages with 2 different versions — add to pnpm-workspace.yaml catalog",
"hash": "67f9d81d9528"
"hash": "e07bb1da47d7"
}
],
"packages/@n8n/cli/package.json": [
@@ -2043,48 +2043,48 @@
"hash": "0f94a71d5ae8"
}
],
"packages/testing/playwright/eslint.config.mjs": [
"packages/quality/testing/playwright/eslint.config.mjs": [
{
"rule": "lint-config-layering",
"line": 21,
"message": "packages/testing/playwright turns '@typescript-eslint/no-unused-expressions' down to 'off' for the whole package.",
"hash": "e2f93b5b591a"
"message": "packages/quality/testing/playwright turns '@typescript-eslint/no-unused-expressions' down to 'off' for the whole package.",
"hash": "c789a142996b"
},
{
"rule": "lint-config-layering",
"line": 22,
"message": "packages/testing/playwright turns '@typescript-eslint/no-use-before-define' down to 'off' for the whole package.",
"hash": "75d3ec1677ba"
"message": "packages/quality/testing/playwright turns '@typescript-eslint/no-use-before-define' down to 'off' for the whole package.",
"hash": "fcc3a69d8a2e"
},
{
"rule": "lint-config-layering",
"line": 23,
"message": "packages/testing/playwright turns '@typescript-eslint/promise-function-async' down to 'off' for the whole package.",
"hash": "35dbe9e44ada"
"message": "packages/quality/testing/playwright turns '@typescript-eslint/promise-function-async' down to 'off' for the whole package.",
"hash": "5ab4c080de15"
},
{
"rule": "lint-config-layering",
"line": 24,
"message": "packages/testing/playwright turns 'playwright/expect-expect' down to 'warn' for the whole package.",
"hash": "e8b5f4b0f0dd"
"message": "packages/quality/testing/playwright turns 'playwright/expect-expect' down to 'warn' for the whole package.",
"hash": "3a584940f6c7"
},
{
"rule": "lint-config-layering",
"line": 25,
"message": "packages/testing/playwright turns 'playwright/max-nested-describe' down to 'warn' for the whole package.",
"hash": "77d82ee8a802"
"message": "packages/quality/testing/playwright turns 'playwright/max-nested-describe' down to 'warn' for the whole package.",
"hash": "8ea7fd515ed8"
},
{
"rule": "lint-config-layering",
"line": 27,
"message": "packages/testing/playwright turns 'playwright/no-skipped-test' down to 'warn' for the whole package.",
"hash": "14eb9eecd1b4"
"message": "packages/quality/testing/playwright turns 'playwright/no-skipped-test' down to 'warn' for the whole package.",
"hash": "6ce1d38312f9"
},
{
"rule": "lint-config-layering",
"line": 78,
"message": "packages/testing/playwright turns 'unicorn/filename-case' down to 'off' for the whole package.",
"hash": "de86b9645930"
"message": "packages/quality/testing/playwright turns 'unicorn/filename-case' down to 'off' for the whole package.",
"hash": "949214800425"
}
],
"packages/frontend/@n8n/design-system/eslint.config.mjs": [
+6 -4
View File
@@ -26,7 +26,7 @@
!docker/images/runners
!docker/images/runners/n8n-task-runners.json
# === benchmark image (packages/@n8n/benchmark/Dockerfile) ===
# === benchmark image (packages/quality/efficiency/scale/benchmark/Dockerfile) ===
!package.json
!pnpm-lock.yaml
!pnpm-workspace.yaml
@@ -34,8 +34,10 @@
!patches/**
!scripts
!scripts/**
!packages/@n8n/benchmark
!packages/@n8n/benchmark/**
!packages/quality
!packages/quality/efficiency
!packages/quality/efficiency/scale
!packages/quality/efficiency/scale/benchmark
!packages/quality/efficiency/scale/benchmark/**
!packages/@n8n/typescript-config
!packages/@n8n/typescript-config/**
+3 -3
View File
@@ -81,11 +81,11 @@ return 'blacksmith';
| Telemetry | Source | Metrics |
|-----------|--------|---------|
| Playwright perf/benchmark | `packages/testing/playwright/reporters/metrics-reporter.ts` | Any metric attached via `attachMetric()` |
| Accessibility buckets | `packages/testing/playwright/reporters/a11y-reporter.ts` | Per-bucket axe score, violated rules, violating elements |
| Playwright perf/benchmark | `packages/quality/testing/playwright/reporters/metrics-reporter.ts` | Any metric attached via `attachMetric()` |
| Accessibility buckets | `packages/quality/testing/playwright/reporters/a11y-reporter.ts` | Per-bucket axe score, violated rules, violating elements |
| Build stats | `.github/scripts/send-build-stats.mjs` | Per-package build duration, cache hit/miss, run total |
| Docker stats | `.github/scripts/send-docker-stats.mjs` | Image size per platform, docker build duration |
| Container stack | `packages/testing/containers/telemetry.ts` | E2E stack startup times per service |
| Container stack | `packages/quality/environments/containers/telemetry.ts` | E2E stack startup times per service |
## Secrets
+1 -1
View File
@@ -178,7 +178,7 @@ These only run if specific files changed:
| `docker/images/n8n-base/Dockerfile` | `build-base-image.yml` | any |
| `**/package.json`, `**/turbo.json` | `build-windows.yml` | master |
| `packages/@n8n/ai-workflow-builder.ee/evaluations/programmatic/python/**` | `test-evals-python.yml` | any |
| `packages/@n8n/benchmark/**` | `build-benchmark-image.yml` | master |
| `packages/quality/efficiency/scale/benchmark/**` | `build-benchmark-image.yml` | master |
| `packages/cli/src/public-api/**/*.yml`, `packages/cli/src/public-api/**/*.yaml`, `packages/cli/src/public-api/**/*.css`, `packages/cli/src/public-api/v1/openapi-gen/**/*.ts`, `packages/cli/scripts/build.mjs`, `packages/cli/package.json` | `util-publish-api-schema.yml` | master |
| `packages/@n8n/instance-ai/src/**`, `packages/@n8n/instance-ai/skills/**`, `packages/@n8n/instance-ai/knowledge-base/**`, `packages/@n8n/instance-ai/evaluations/**`, `packages/cli/src/modules/instance-ai/**`, `packages/core/src/execution-engine/eval-mock-helpers.ts`, `packages/@n8n/agents/src/**` | `ci-instance-ai-evals.yml` | on PR `opened` / `reopened` / `ready_for_review` |
| `docker/get-n8n.sh`, `docker/get-n8n-compose.yml`, `docker/test-get-n8n.sh` | `test-get-n8n.yml` | any |
@@ -242,7 +242,7 @@ describe('runtime filter', () => {
'!**/CHANGELOG.md',
'!**/*.test.ts',
'!**/*.spec.ts',
'!packages/testing/playwright/**',
'!packages/quality/testing/playwright/**',
'!packages/frontend/@n8n/storybook/**',
'!scripts/agent-setup.mjs',
'!scripts/backend-module/**',
@@ -265,7 +265,7 @@ describe('runtime filter', () => {
it('does not trigger on playwright tests', () => {
assert.equal(
evaluateFilter(['packages/testing/playwright/tests/x.spec.ts'], runtimePatterns),
evaluateFilter(['packages/quality/testing/playwright/tests/x.spec.ts'], runtimePatterns),
false,
);
});
+1 -1
View File
@@ -1,6 +1,6 @@
services:
postgres:
image: ${TEST_IMAGE_POSTGRES:?set it from packages/testing/containers/postgres-versions.json}
image: ${TEST_IMAGE_POSTGRES:?set it from packages/quality/environments/containers/postgres-versions.json}
restart: always
environment:
- POSTGRES_DB=n8n
+1 -1
View File
@@ -7,7 +7,7 @@ import { fileURLToPath } from 'node:url';
const REPO_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../..');
export const POSTGRES_VERSIONS_PATH = 'packages/testing/containers/postgres-versions.json';
export const POSTGRES_VERSIONS_PATH = 'packages/quality/environments/containers/postgres-versions.json';
/** The matrix wall is bounded by its slowest leg, so a larger runner buys nothing. */
const RUNNER = 'blacksmith-4vcpu-ubuntu-2204';
+5 -5
View File
@@ -30,9 +30,9 @@ describe('matchesGlob', () => {
});
it('matches a prefix pattern', () => {
assert.equal(matchesGlob('packages/testing/playwright/a.ts', 'packages/testing/**'), true);
assert.equal(matchesGlob('packages/testing', 'packages/testing/**'), false);
assert.equal(matchesGlob('packages/cli/a.ts', 'packages/testing/**'), false);
assert.equal(matchesGlob('packages/quality/testing/playwright/a.ts', 'packages/quality/**'), true);
assert.equal(matchesGlob('packages/quality', 'packages/quality/**'), false);
assert.equal(matchesGlob('packages/cli/a.ts', 'packages/quality/**'), false);
});
it('matches a literal path only exactly', () => {
@@ -55,8 +55,8 @@ describe('matchesGlob', () => {
'packages/cli/__snapshots__/a.ts.snap',
'packages/nodes-base/nodes/Foo/test/fixtures/a.json',
'packages/frontend/editor-ui/src/__mocks__/a.ts',
'packages/testing/playwright/a.ts',
'packages/testing',
'packages/quality/testing/playwright/a.ts',
'packages/quality',
'.github/scripts/glob.test.mjs',
'.github/workflows/ci.yml',
'a/.hidden/b.spec.js',
@@ -217,7 +217,7 @@ describe('computeLineStats', () => {
['*.snap extension', 'src/foo.snap'],
['fixtures/ directory', 'src/fixtures/data.json'],
['__mocks__/ directory', 'src/__mocks__/axios.ts'],
['packages/testing/**', 'packages/testing/playwright/spec.ts'],
['packages/quality/**', 'packages/quality/testing/playwright/spec.ts'],
['dot-directory *.test.mjs', '.github/scripts/owners/owners.test.mjs'],
]) {
it(`classifies ${label} as testFiles`, () => {
+5 -2
View File
@@ -36,8 +36,11 @@ export const TEST_PATTERNS = [
// Fixtures and mocks
'**/fixtures/**',
'**/__mocks__/**',
// Dedicated testing package
'packages/testing/**',
// Former packages/testing workspaces
'packages/quality/policy/**',
'packages/quality/testing/**',
'packages/quality/environments/**',
'packages/quality/efficiency/microbenchmarks/**',
];
export const MISC_PATTERNS = [
+11 -4
View File
@@ -203,14 +203,21 @@ describe('countFilteredAdditions', () => {
assert.equal(countFilteredAdditions(files, EXCLUDE_PATTERNS), 50);
});
it('applies EXCLUDE_PATTERNS to packages/testing and pnpm-lock.yaml', () => {
it('excludes former testing workspaces but counts benchmark CLI source', () => {
const files = [
{ filename: 'packages/cli/src/service.ts', additions: 50 },
{ filename: 'packages/testing/playwright/tests/workflow.spec.ts', additions: 100 },
{ filename: 'packages/testing/playwright/pages/CanvasPage.ts', additions: 100 },
{ filename: 'packages/quality/testing/playwright/tests/workflow.spec.ts', additions: 100 },
{ filename: 'packages/quality/testing/playwright/pages/CanvasPage.ts', additions: 100 },
{ filename: 'packages/quality/policy/code-health/src/cli.ts', additions: 100 },
{ filename: 'packages/quality/environments/containers/service-stack.ts', additions: 100 },
{
filename: 'packages/quality/efficiency/microbenchmarks/benchmarks/bench-options.ts',
additions: 100,
},
{ filename: 'packages/quality/efficiency/scale/benchmark/src/commands/list.ts', additions: 100 },
{ filename: 'pnpm-lock.yaml', additions: 500 },
];
assert.equal(countFilteredAdditions(files, EXCLUDE_PATTERNS), 50);
assert.equal(countFilteredAdditions(files, EXCLUDE_PATTERNS), 150);
});
it('applies EXCLUDE_PATTERNS to markdown files', () => {
+2 -2
View File
@@ -6,7 +6,7 @@ on:
branches:
- master
paths:
- 'packages/@n8n/benchmark/**'
- 'packages/quality/efficiency/scale/benchmark/**'
- 'pnpm-lock.yaml'
- 'pnpm-workspace.yaml'
- '.github/workflows/build-benchmark-image.yml'
@@ -33,7 +33,7 @@ jobs:
DOCKER_BUILD_SUMMARY: false
with:
context: .
file: ./packages/@n8n/benchmark/Dockerfile
file: ./packages/quality/efficiency/scale/benchmark/Dockerfile
platforms: linux/amd64
provenance: false
push: true
+2 -2
View File
@@ -99,8 +99,8 @@ jobs:
mode: filter
filters: |
janitor:
packages/testing/playwright/**
packages/testing/janitor/**
packages/quality/testing/playwright/**
packages/quality/testing/janitor/**
check-static-analysis:
name: Static Analysis
+23 -23
View File
@@ -79,7 +79,7 @@ jobs:
!**/CHANGELOG.md
!**/*.test.ts
!**/*.spec.ts
!packages/testing/playwright/**
!packages/quality/testing/playwright/**
!packages/frontend/@n8n/storybook/**
!.devcontainer/**
!scripts/agent-setup.mjs
@@ -92,7 +92,7 @@ jobs:
unit:
**
!packages/@n8n/task-runner-python/**
!packages/testing/playwright/**
!packages/quality/testing/playwright/**
!.github/**
.github/workflows/test-unit-reusable.yml
!**/*.md
@@ -111,8 +111,8 @@ jobs:
.github/workflows/prepare-docker-reusable.yml
.github/actions/build-n8n-docker/**
.github/actions/load-n8n-docker/**
packages/testing/playwright/**
packages/testing/containers/**
packages/quality/testing/playwright/**
packages/quality/environments/containers/**
dev-server-smoke:
packages/frontend/editor-ui/vite.config.mts
# If this job becomes too slow, use a smaller glob. Do not delete the assertion.
@@ -120,10 +120,10 @@ jobs:
pnpm-workspace.yaml
packages/@n8n/*/package.json
packages/frontend/@n8n/frontend-vite-config/**
packages/testing/playwright/tests/dev-server-smoke/**
packages/testing/playwright/playwright.config.ts
packages/testing/playwright/playwright-projects.ts
packages/testing/playwright/package.json
packages/quality/testing/playwright/tests/dev-server-smoke/**
packages/quality/testing/playwright/playwright.config.ts
packages/quality/testing/playwright/playwright-projects.ts
packages/quality/testing/playwright/package.json
.github/workflows/test-dev-server-smoke-reusable.yml
frontend-declarations:
packages/frontend/@n8n/design-system/**
@@ -146,9 +146,9 @@ jobs:
cubic.yaml
.agents/review-rules/**
e2e-performance:
packages/testing/playwright/tests/performance/**
packages/testing/playwright/utils/performance-helper.ts
packages/testing/containers/**
packages/quality/testing/playwright/tests/performance/**
packages/quality/testing/playwright/utils/performance-helper.ts
packages/quality/environments/containers/**
.github/workflows/test-e2e-performance-reusable.yml
# Whole directories, not the `engine-*` files alone. Engine v2
# routing lives in ordinary control plane code: `workflow-runner.ts`
@@ -169,8 +169,8 @@ jobs:
packages/cli/src/webhooks/**
packages/cli/src/workflows/triggers/**
packages/cli/src/workflow-runner.ts
packages/testing/containers/**
packages/testing/playwright/**
packages/quality/environments/containers/**
packages/quality/testing/playwright/**
instance-ai-workflow-eval:
packages/@n8n/instance-ai/src/**
packages/@n8n/instance-ai/evaluations/**
@@ -193,10 +193,10 @@ jobs:
packages/cli/src/**/*.integration.test.ts
packages/cli/src/modules/instance-registry/**
packages/cli/vitest.*
packages/testing/containers/services/postgres.ts
packages/testing/containers/services/redis.ts
packages/testing/containers/postgres-versions.json
packages/testing/containers/test-containers.ts
packages/quality/environments/containers/services/postgres.ts
packages/quality/environments/containers/services/redis.ts
packages/quality/environments/containers/postgres-versions.json
packages/quality/environments/containers/test-containers.ts
.github/workflows/test-db-reusable.yml
.github/scripts/db-test-matrix.mjs
docs/generated/**
@@ -214,10 +214,10 @@ jobs:
packages/cli/vitest.config.base.ts
packages/cli/vitest.config.migration.ts
packages/cli/vitest.config.migration.testcontainers.ts
packages/testing/containers/services/postgres.ts
packages/testing/containers/services/redis.ts
packages/testing/containers/postgres-versions.json
packages/testing/containers/test-containers.ts
packages/quality/environments/containers/services/postgres.ts
packages/quality/environments/containers/services/redis.ts
packages/quality/environments/containers/postgres-versions.json
packages/quality/environments/containers/test-containers.ts
.github/workflows/test-db-reusable.yml
.github/scripts/db-test-matrix.mjs
@@ -247,7 +247,7 @@ jobs:
env:
CHANGED_FILES: ${{ steps.ci-filter.outputs.changed-files }}
run: |
PACKAGES=$(node packages/testing/janitor/dist/cli.js affected-packages | tr '\n' ' ' | sed 's/ *$//')
PACKAGES=$(node packages/quality/testing/janitor/dist/cli.js affected-packages | tr '\n' ' ' | sed 's/ *$//')
echo "Affected packages: $PACKAGES"
echo "list=$PACKAGES" >> "$GITHUB_OUTPUT"
@@ -393,7 +393,7 @@ jobs:
PLAYWRIGHT_PROJECT: ${{ github.event.pull_request.head.repo.fork == true && 'sqlite:e2e' || 'multi-main:e2e' }}
run: |
FILES_CSV=$(echo "$CHANGED_FILES" | tr '\n' ',' | sed 's/,$//')
MATRIX=$(node packages/testing/playwright/scripts/distribute-tests.mjs --matrix 20 --orchestrate --impact "--project=$PLAYWRIGHT_PROJECT" "--files=$FILES_CSV" "--base=$MERGE_BASE")
MATRIX=$(node packages/quality/testing/playwright/scripts/distribute-tests.mjs --matrix 20 --orchestrate --impact "--project=$PLAYWRIGHT_PROJECT" "--files=$FILES_CSV" "--base=$MERGE_BASE")
echo "matrix=$MATRIX" >> "$GITHUB_OUTPUT"
echo "skip-tests=$(node -e "process.stdout.write(JSON.parse(process.argv[1])[0]?.skip === true ? 'true' : 'false')" "$MATRIX")" >> "$GITHUB_OUTPUT"
+2 -2
View File
@@ -3,7 +3,7 @@ name: 'CI: Terraform'
on:
pull_request:
paths:
- packages/@n8n/benchmark/infra/**
- packages/quality/efficiency/scale/benchmark/infra/**
- .github/workflows/ci-terraform.yml
# The job only reads the repo and runs terraform. It writes nothing.
@@ -16,7 +16,7 @@ jobs:
runs-on: ubuntu-latest
defaults:
run:
working-directory: packages/@n8n/benchmark/infra
working-directory: packages/quality/efficiency/scale/benchmark/infra
steps:
- name: Check out project
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -29,10 +29,10 @@ on:
# verify themselves (both modes) before the merge.
pull_request:
paths:
- 'packages/testing/playwright/tests/infrastructure/encryption/**'
- 'packages/testing/containers/services/n8n.ts'
- 'packages/testing/containers/services/types.ts'
- 'packages/testing/containers/stack.ts'
- 'packages/quality/testing/playwright/tests/infrastructure/encryption/**'
- 'packages/quality/environments/containers/services/n8n.ts'
- 'packages/quality/environments/containers/services/types.ts'
- 'packages/quality/environments/containers/stack.ts'
- '.github/workflows/encryption-upgrade-test.yml'
permissions:
@@ -36,4 +36,4 @@ jobs:
- name: Destroy cloud env
run: pnpm destroy-cloud-env
working-directory: packages/@n8n/benchmark
working-directory: packages/quality/efficiency/scale/benchmark
+4 -4
View File
@@ -62,11 +62,11 @@ jobs:
- name: Destroy any existing environment
run: pnpm destroy-cloud-env
working-directory: packages/@n8n/benchmark
working-directory: packages/quality/efficiency/scale/benchmark
- name: Provision the environment
run: pnpm provision-cloud-env ${{ env.DEBUG }}
working-directory: packages/@n8n/benchmark
working-directory: packages/quality/efficiency/scale/benchmark
- name: Run the benchmark
id: benchmark
@@ -81,7 +81,7 @@ jobs:
--n8nTag ${{ env.N8N_TAG }} \
--benchmarkTag ${{ env.N8N_BENCHMARK_TAG }} \
${{ env.DEBUG }}
working-directory: packages/@n8n/benchmark
working-directory: packages/quality/efficiency/scale/benchmark
# We need to login again because the access token expires
- name: Azure login
@@ -95,7 +95,7 @@ jobs:
- name: Destroy the environment
if: always()
run: pnpm destroy-cloud-env ${{ env.DEBUG }}
working-directory: packages/@n8n/benchmark
working-directory: packages/quality/efficiency/scale/benchmark
- name: Fail `build` job if `benchmark` step failed
if: steps.benchmark.outcome == 'failure'
@@ -10,7 +10,7 @@ on:
env:
NODE_OPTIONS: '--max-old-space-size=6144'
PLAYWRIGHT_BROWSERS_PATH: packages/testing/playwright/.playwright-browsers
PLAYWRIGHT_BROWSERS_PATH: packages/quality/testing/playwright/.playwright-browsers
jobs:
smoke:
@@ -53,6 +53,6 @@ jobs:
with:
name: dev-server-smoke-report
path: |
packages/testing/playwright/test-results/
packages/testing/playwright/playwright-report/
packages/quality/testing/playwright/test-results/
packages/quality/testing/playwright/playwright-report/
retention-days: 7
@@ -53,7 +53,7 @@ jobs:
- name: Generate matrix (up to 20 shards, duration-weighted)
id: gen
run: |
MATRIX=$(node packages/testing/playwright/scripts/distribute-tests.mjs --matrix 20 --orchestrate --project=coverage)
MATRIX=$(node packages/quality/testing/playwright/scripts/distribute-tests.mjs --matrix 20 --orchestrate --project=coverage)
echo "matrix=$MATRIX" >> "$GITHUB_OUTPUT"
e2e:
@@ -215,7 +215,7 @@ jobs:
# Merges all lcovs into one lcov.info (SF: paths normalised to repo-root-
# relative) and builds the per-spec E2E impact map for PR selection.
- name: Aggregate coverage + build impact map
working-directory: packages/testing/playwright
working-directory: packages/quality/testing/playwright
run: |
node scripts/aggregate-coverage.mjs \
--shards=/tmp/shards \
@@ -227,7 +227,7 @@ jobs:
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
with:
name: coverage-report
path: packages/testing/playwright/coverage/
path: packages/quality/testing/playwright/coverage/
retention-days: 14
- name: Upload Combined Coverage to Codecov
@@ -235,7 +235,7 @@ jobs:
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: packages/testing/playwright/coverage/lcov.info
files: packages/quality/testing/playwright/coverage/lcov.info
flags: nightly-full
name: nightly-full-coverage
fail_ci_if_error: false
@@ -256,7 +256,7 @@ jobs:
env:
CODECOV_API_TOKEN: ${{ secrets.CODECOV_API_TOKEN }}
run: |
node packages/testing/playwright/scripts/coverage-analysis.mjs \
node packages/quality/testing/playwright/scripts/coverage-analysis.mjs \
--md --top=15 --out-json=coverage-gaps.json >> "$GITHUB_STEP_SUMMARY"
- name: Upload Coverage Gap Report
+6 -6
View File
@@ -5,8 +5,8 @@ on:
pull_request:
paths:
- '.github/workflows/test-e2e-helm.yml'
- 'packages/testing/containers/helm-stack.ts'
- 'packages/testing/containers/helm-start-stack.ts'
- 'packages/quality/environments/containers/helm-stack.ts'
- 'packages/quality/environments/containers/helm-start-stack.ts'
workflow_dispatch:
inputs:
branch:
@@ -83,7 +83,7 @@ jobs:
build-command: 'pnpm build'
- name: Install Browsers
working-directory: packages/testing/playwright
working-directory: packages/quality/testing/playwright
run: pnpm exec playwright install chromium
- name: Install Helm
@@ -115,7 +115,7 @@ jobs:
echo "KUBECONFIG=${KUBECONFIG_PATH}" >> "$GITHUB_ENV"
- name: Run Building Blocks E2E Tests
working-directory: packages/testing/playwright
working-directory: packages/quality/testing/playwright
run: |
N8N_BASE_URL="${{ env.N8N_BASE_URL }}" \
RESET_E2E_DB=true \
@@ -150,8 +150,8 @@ jobs:
with:
name: helm-e2e-report-${{ inputs.mode || 'queue' }}
path: |
packages/testing/playwright/test-results/
packages/testing/playwright/playwright-report/
packages/quality/testing/playwright/test-results/
packages/quality/testing/playwright/playwright-report/
retention-days: 7
cleanup-ghcr:
@@ -74,7 +74,7 @@ jobs:
# The aggregator auto-detects n8n version and commit SHA from the repo
# (anchored to its own file location), so no path/version glue is needed.
- name: Aggregate sizing matrix
working-directory: packages/testing/playwright
working-directory: packages/quality/testing/playwright
run: |
pnpm exec tsx scripts/sizing-matrix-aggregate.ts \
--input /tmp/benchmark-reports \
@@ -86,8 +86,8 @@ jobs:
with:
name: sizing-matrix
path: |
packages/testing/playwright/sizing-matrix.json
packages/testing/playwright/sizing-matrix.md
packages/quality/testing/playwright/sizing-matrix.json
packages/quality/testing/playwright/sizing-matrix.md
retention-days: 14
if-no-files-found: error
+1 -1
View File
@@ -77,7 +77,7 @@ jobs:
- name: Generate matrix (up to 20 shards, duration-weighted)
id: gen
run: |
MATRIX=$(node packages/testing/playwright/scripts/distribute-tests.mjs --matrix 20 --orchestrate --project=sqlite:e2e)
MATRIX=$(node packages/quality/testing/playwright/scripts/distribute-tests.mjs --matrix 20 --orchestrate --project=sqlite:e2e)
echo "matrix=$MATRIX" >> "$GITHUB_OUTPUT"
e2e:
@@ -37,7 +37,7 @@ jobs:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
sparse-checkout: packages/testing/playwright/scripts/canvas-perf-sentinels.mjs
sparse-checkout: packages/quality/testing/playwright/scripts/canvas-perf-sentinels.mjs
sparse-checkout-cone-mode: false
- name: Download performance shard artifacts
@@ -54,4 +54,4 @@ jobs:
echo "::warning::No test-results.json found in performance artifacts — sentinel skipped."
exit 0
fi
node packages/testing/playwright/scripts/canvas-perf-sentinels.mjs "${RESULTS_PATHS[@]}"
node packages/quality/testing/playwright/scripts/canvas-perf-sentinels.mjs "${RESULTS_PATHS[@]}"
+10 -10
View File
@@ -69,7 +69,7 @@ env:
# Browser cache location - must match install-browsers script.
# Absolute so nested pnpm invocations (e.g. coverage shard runner) don't
# re-resolve a relative path against the package's INIT_CWD and double it.
PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/packages/testing/playwright/.playwright-browsers
PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/packages/quality/testing/playwright/.playwright-browsers
# docker-artifact loads the image locally as n8nio/n8n:local; unused in local mode.
TEST_IMAGE_N8N: 'n8nio/n8n:local'
N8N_SKIP_LICENSES: 'true'
@@ -123,7 +123,7 @@ jobs:
GITHUB_RUN_ID: ${{ github.run_id }}
GITHUB_RUN_ATTEMPT: ${{ github.run_attempt }}
run: |
FILTERED=$(echo "$CANDIDATES" | node packages/testing/janitor/dist/cli.js filter-shard | tr '\n' ' ' | sed 's/ *$//')
FILTERED=$(echo "$CANDIDATES" | node packages/quality/testing/janitor/dist/cli.js filter-shard | tr '\n' ' ' | sed 's/ *$//')
echo "specs=$FILTERED" >> "$GITHUB_OUTPUT"
- name: Resolve shard args
@@ -195,13 +195,13 @@ jobs:
with:
name: ${{ inputs.artifact-prefix }}-shard-${{ matrix.shard }}
path: |
packages/testing/playwright/test-results/
packages/testing/playwright/test-results.json
packages/testing/playwright/playwright-report/
packages/testing/playwright/coverage/lcov.info
packages/testing/playwright/coverage/by-spec/
packages/testing/playwright/coverage/index.html
packages/testing/playwright/coverage/assets/
packages/quality/testing/playwright/test-results/
packages/quality/testing/playwright/test-results.json
packages/quality/testing/playwright/playwright-report/
packages/quality/testing/playwright/coverage/lcov.info
packages/quality/testing/playwright/coverage/by-spec/
packages/quality/testing/playwright/coverage/index.html
packages/quality/testing/playwright/coverage/assets/
retention-days: 1
if-no-files-found: ignore
@@ -214,7 +214,7 @@ jobs:
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
with:
name: ${{ inputs.artifact-prefix }}-a11y-report-shard-${{ matrix.shard }}
path: packages/testing/playwright/a11y-report/
path: packages/quality/testing/playwright/a11y-report/
retention-days: 7
if-no-files-found: ignore
@@ -90,13 +90,13 @@ jobs:
env:
BASE_BRANCH: ${{ inputs.base-branch }}
run: |
if [ -f packages/testing/code-health/package.json ]; then
if [ -f packages/quality/policy/code-health/package.json ]; then
echo 'present=true' >> "$GITHUB_OUTPUT"
elif [ -n "$BASE_BRANCH" ]; then
echo 'present=false' >> "$GITHUB_OUTPUT"
echo "::notice::packages/testing/code-health is absent on \"$BASE_BRANCH\"; nothing to verify."
echo "::notice::packages/quality/policy/code-health is absent on \"$BASE_BRANCH\"; nothing to verify."
else
echo '::error::packages/testing/code-health is missing (moved or renamed?).'
echo '::error::packages/quality/policy/code-health is missing (moved or renamed?).'
exit 1
fi
@@ -119,10 +119,10 @@ jobs:
# REPORT_ONLY is a single flag or empty. Quoting it would pass an empty argument, which
# the CLI reads as an (unknown) package name and rejects.
# shellcheck disable=SC2086
pnpm --dir packages/testing/code-health exec tsx src/cli.ts verify-npm-install --changed="$BASE_REF" $REPORT_ONLY
pnpm --dir packages/quality/policy/code-health exec tsx src/cli.ts verify-npm-install --changed="$BASE_REF" $REPORT_ONLY
- name: Verify npm-install closure (all packages)
if: steps.verifier.outputs.present == 'true' && inputs.scope == 'all'
run: |
# shellcheck disable=SC2086 # see the scoped step above
pnpm --dir packages/testing/code-health exec tsx src/cli.ts verify-npm-install --all $REPORT_ONLY
pnpm --dir packages/quality/policy/code-health exec tsx src/cli.ts verify-npm-install --all $REPORT_ONLY
+1 -1
View File
@@ -71,7 +71,7 @@ jobs:
- name: Resolve Postgres image
if: fromJSON(steps.paths-filter.outputs.results).db
run: |
IMAGE=$(node -p "require('./packages/testing/containers/postgres-versions.json').primary")
IMAGE=$(node -p "require('./packages/quality/environments/containers/postgres-versions.json').primary")
echo "TEST_IMAGE_POSTGRES=$IMAGE" >> "$GITHUB_ENV"
- name: Start Postgres
+9 -9
View File
@@ -46,17 +46,17 @@ sbom-source.cdx.json
junit.xml
test-results.json
*.0x
packages/testing/playwright/playwright-report
packages/testing/playwright/eval-report
packages/testing/playwright/a11y-report
packages/testing/playwright/test-results
packages/testing/playwright/results.xml
packages/testing/playwright/eval-results.json
packages/quality/testing/playwright/playwright-report
packages/quality/testing/playwright/eval-report
packages/quality/testing/playwright/a11y-report
packages/quality/testing/playwright/test-results
packages/quality/testing/playwright/results.xml
packages/quality/testing/playwright/eval-results.json
packages/@n8n/instance-ai/eval-results.json
packages/@n8n/instance-ai/.eval-output/
packages/@n8n/instance-ai/eval-pr-comment.md
packages/testing/playwright/.playwright-browsers
packages/testing/playwright/.playwright-cli
packages/quality/testing/playwright/.playwright-browsers
packages/quality/testing/playwright/.playwright-cli
test-results/
compiled_app_output
trivy_report*
@@ -65,7 +65,7 @@ docker-build-manifest.json
packages/cli/src/modules/my-feature
.secrets
act-event.json
packages/testing/**/.cursor/rules/
packages/quality/**/.cursor/rules/
.venv
.ruff_cache
__pycache__
+3 -3
View File
@@ -310,14 +310,14 @@ extends one of those layers):
- Run that package's `pnpm lint` and `pnpm typecheck` before committing code.
Build first when shared types or cross-package dependencies change.
- Use Vitest for unit tests. Use
[Playwright](packages/testing/playwright/AGENTS.md) when a test needs its
[Playwright](packages/quality/testing/playwright/AGENTS.md) when a test needs its
browser, fixtures, or managed containers.
- For Vitest packages with `@n8n/di` decorators, use
`createVitestConfigWithDecorators` from `@n8n/vitest-config/node-decorators`.
- Check import and mock side effects before running tests. Keep tests out of
user-owned directories. Set `N8N_USER_FOLDER` to a test-owned directory before
importing n8n settings. Clean up only paths that the test created.
- CI runs [`@n8n/code-health`](packages/testing/code-health/README.md) static
- CI runs [`@n8n/code-health`](packages/quality/policy/code-health/README.md) static
analysis on PRs. It checks monorepo rules, including dependency hygiene and
encryption-boundary coverage.
@@ -331,7 +331,7 @@ extends one of those layers):
| Start a Codespace backend and share its port | `pnpm dev:up` |
The root `pnpm dev` command does not start a server. See the
[Playwright guide](packages/testing/playwright/README.md) and the
[Playwright guide](packages/quality/testing/playwright/README.md) and the
[Codespaces guide](.devcontainer/codespaces/README.md) for details.
### Common Development Tasks
+2 -2
View File
@@ -432,7 +432,7 @@ Each instance gets its own port, and the webhook URL matches the main URL. Multi
>- This does not work with Podman out of the box. You need to [configure Testcontainers for Podman](https://podman-desktop.io/tutorial/testcontainers-with-podman) first.
>- Alternatively, you can use colima and set DOCKER_HOST and TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE environment variables as described [here](https://node.testcontainers.org/supported-container-runtimes/#colima)
Refer to [packages/testing/containers/README.md](packages/testing/containers/README.md) for more information.
Refer to [packages/quality/environments/containers/README.md](packages/quality/environments/containers/README.md) for more information.
### Work locally with syslog
@@ -661,7 +661,7 @@ E2E tests can be started via one of the following commands:
- `pnpm --filter=n8n-playwright test:local --ui` - Run tests in interactive UI mode (useful for debugging)
- `pnpm --filter=n8n-playwright test:local --grep="test-name"` - Run specific tests matching pattern
See `packages/testing/playwright/README.md` for more test commands and `packages/testing/playwright/CONTRIBUTING.md` for writing guidelines.
See `packages/quality/testing/playwright/README.md` for more test commands and `packages/quality/testing/playwright/CONTRIBUTING.md` for writing guidelines.
## Create custom nodes
+1 -2
View File
@@ -254,8 +254,7 @@ docker/get-n8n.sh @n8n-io/qa-dx required
# QA
packages/testing/ @n8n-io/qa-dx
packages/@n8n/benchmark/ @n8n-io/qa-dx
packages/quality/ @n8n-io/qa-dx
packages/@n8n/vitest-config/ @n8n-io/qa-dx
# Encryption boundary (IAM) — the encryption area and its guardrail rules.
+1
View File
@@ -18,6 +18,7 @@
"**/cl100k_base.json",
"**/o200k_base.json",
"**/*.generated.ts",
"**/benchmark/scripts/mock-api/**",
"**/expectations/**"
]
},
+1 -1
View File
@@ -150,5 +150,5 @@ ignore:
- (?s:.*/migrations/(postgresdb|sqlite|mysqldb)/.*\.ts.*)\Z
- (?s:.*/scripts/.*)\Z
# Test/build infrastructure packages
- (?s:.*/packages/testing/.*)\Z
- (?s:.*/packages/quality/.*)\Z
- (?s:.*/packages/@n8n/(benchmark|eslint-config|eslint-plugin-community-nodes|stylelint-config|typescript-config|vitest-config)/.*)\Z
+1 -1
View File
@@ -202,7 +202,7 @@ reviews:
- docker/**
- scripts/**
- patches/**
- packages/testing/**
- packages/quality/**
- packages/@n8n/eslint-config/**
- packages/@n8n/stylelint-config/**
- packages/@n8n/vitest-config/**
+2 -2
View File
@@ -147,7 +147,7 @@ const config: KnipConfig = {
// Only named in JSDoc type imports; @stryker-mutator/core provides it.
ignoreDependencies: ['@stryker-mutator/api'],
}),
'packages/@n8n/benchmark': pkg({
'packages/quality/efficiency/scale/benchmark': pkg({
// The k6 runtime provides its own modules; only @types/k6 is installed.
ignoreDependencies: ['k6'],
}),
@@ -204,7 +204,7 @@ const config: KnipConfig = {
// Spawned as a binary through execFile.
ignoreDependencies: ['agent-browser'],
}),
'packages/testing/playwright': pkg({
'packages/quality/testing/playwright': pkg({
ignoreDependencies: [
// The e2e suite runs against the built app; the edge orders the turbo build.
'n8n',
+1 -1
View File
@@ -87,7 +87,7 @@ pre-commit:
- merge
- rebase
playwright_janitor:
glob: 'packages/testing/playwright/**/*.ts'
glob: 'packages/quality/testing/playwright/**/*.ts'
run: |
files=$(echo {staged_files} | tr ' ' ',')
pnpm --filter=n8n-playwright janitor --files="$files"
-7
View File
@@ -1,7 +0,0 @@
{
"$schema": "../node_modules/@biomejs/biome/configuration_schema.json",
"extends": ["../../../biome.jsonc"],
"files": {
"ignore": ["scripts/mock-api/**"]
}
}
+1 -1
View File
@@ -33,7 +33,7 @@ const TBLS_IMAGE =
const POSTGRES_VERSIONS_PATH = resolve(
REPO_ROOT,
'packages/testing/containers/postgres-versions.json',
'packages/quality/environments/containers/postgres-versions.json',
);
function primaryPostgresImage() {
@@ -26,7 +26,7 @@ export const backendNetworkBoundaryConfig = tseslint.config({
allow: [
// The factory itself: this is where the guarded client is built.
'packages/@n8n/backend-network/',
'packages/@n8n/benchmark/',
'packages/quality/efficiency/scale/benchmark/',
],
},
],
@@ -25,7 +25,7 @@ const TEST_FILE_PATTERN = /(\.(test|spec)\.ts$)|([\\/]__tests__[\\/])|([\\/]test
* so bare or self-naming directives of those forms swallow this rule's report,
* and a package-level `'off'` override is invisible to lint altogether. The
* authoritative check is the code-health rule `encryption-boundary`
* (packages/testing/code-health): it scans the source out of band for every
* (packages/quality/policy/code-health): it scans the source out of band for every
* directive form and verifies that each package that can reach the primitives
* composes the boundary config at error severity.
*/
@@ -7,7 +7,7 @@ import { ExpressionEvaluator } from '../evaluator/expression-evaluator';
// can a real bridge be built from inside the synchronous evaluate() path?
// isolated-vm uses its sync APIs directly; QuickJS relies on the WASM module
// cached by pool warmup (an earlier async initialize). The cold start's cost
// is tracked in packages/testing/performance (cold-start.bench.ts).
// is tracked in packages/quality/efficiency/microbenchmarks (cold-start.bench.ts).
describe(`Lazy acquisition: synchronous cold start (${engineName})`, () => {
it('builds a working bridge synchronously when the pool is exhausted', async () => {
+3 -3
View File
@@ -29,12 +29,12 @@ Read these docs before starting any implementation:
## E2E Testing
Tests live in `packages/testing/playwright/tests/e2e/instance-ai/`.
Tests live in `packages/quality/testing/playwright/tests/e2e/instance-ai/`.
### Local-build mode (no docker, no recording — hits real Anthropic API)
```bash
cd packages/testing/playwright
cd packages/quality/testing/playwright
export ANTHROPIC_API_KEY=sk-ant-...
pnpm test:local:instance-ai # full suite
pnpm test:local:instance-ai --grep "preview" # single test
@@ -52,7 +52,7 @@ mode with a real key. This captures LLM traffic + tool traces into
```bash
pnpm build:docker # from repo root — build the local n8n image first
cd packages/testing/playwright
cd packages/quality/testing/playwright
ANTHROPIC_API_KEY=sk-ant-... pnpm test:container:sqlite tests/e2e/instance-ai --workers 1
```
@@ -263,8 +263,8 @@ N8N_INSTANCE_AI_SEARXNG_URL=http://searxng:8080
INSTANCE_AI_BRAVE_SEARCH_API_KEY=BSA-xxx
# With sandbox (n8n sandbox service)
# CI can start it with:
# pnpm tsx packages/testing/containers/start-sandbox.ts --network n8n-eval-net
# For local development, start it with:
# pnpm --filter n8n-containers services --services sandbox --network n8n-eval-net
N8N_INSTANCE_AI_SANDBOX_ENABLED=true
N8N_INSTANCE_AI_SANDBOX_PROVIDER=n8n-sandbox
N8N_SANDBOX_SERVICE_URL=https://sandbox.example.com
+4 -4
View File
@@ -255,13 +255,13 @@ volatile request body.
| File | Purpose |
|------|---------|
| `packages/testing/playwright/tests/e2e/instance-ai/fixtures.ts` | Test fixtures — proxy setup, recording/replay orchestration |
| `packages/quality/testing/playwright/tests/e2e/instance-ai/fixtures.ts` | Test fixtures — proxy setup, recording/replay orchestration |
| `packages/@n8n/instance-ai/src/tracing/trace-replay.ts` | `TraceIndex`, `IdRemapper`, `TraceWriter`, JSONL parsing |
| `packages/@n8n/instance-ai/src/tracing/langsmith-tracing.ts` | Tool wrapping — `replayWrapTools`, `recordWrapTools` |
| `packages/@n8n/instance-ai/src/types.ts` | `InstanceAiTraceContext`, `TraceReplayMode` |
| `packages/cli/src/modules/instance-ai/instance-ai.service.ts` | Trace mode initialization, shared state management |
| `packages/cli/src/modules/instance-ai/instance-ai.controller.ts` | Test-only REST endpoints for trace delivery |
| `packages/testing/containers/services/proxy.ts` | `ProxyServer` class (MockServer client) |
| `packages/quality/environments/containers/services/proxy.ts` | `ProxyServer` class (MockServer client) |
### Test-Only Endpoints
@@ -323,7 +323,7 @@ stack entirely. Tests hit the real Anthropic API directly. This mode does
**not** record proxy expectations.
```bash
cd packages/testing/playwright
cd packages/quality/testing/playwright
export ANTHROPIC_API_KEY=sk-ant-...
pnpm test:local:instance-ai # full suite
pnpm test:local:instance-ai --grep "preview" # single test
@@ -351,7 +351,7 @@ pnpm test:local:instance-ai --grep "preview" --headed
vars over the generic `test:local:isolated` runner, which provides random free
ports, a throwaway `N8N_USER_FOLDER` (so `~/.n8n` is never touched), and
process-group cleanup. See the
[Playwright README](../../../testing/playwright/README.md) for full details on
[Playwright README](../../../quality/testing/playwright/README.md) for full details on
`test:local:isolated`.
> **Cost note:** Each run makes real Anthropic calls. Scope with `--grep` or
@@ -5,7 +5,7 @@ export const backendNetworkBoundaryConfig = defineConfig({
'n8n-local-rules/no-uncentralized-http': [
'error',
{
allow: ['packages/@n8n/backend-network/', 'packages/@n8n/benchmark/'],
allow: ['packages/@n8n/backend-network/', 'packages/quality/efficiency/scale/benchmark/'],
},
],
},
@@ -3,7 +3,7 @@ import type { App, ComponentPublicInstance } from 'vue';
/**
* localStorage key that switches on app-wide component re-render counting.
*
* Set by the canvas performance benchmark (packages/testing/playwright) before
* Set by the canvas performance benchmark (packages/quality/testing/playwright) before
* the SPA boots. Real users never set it, so the global mixin below is never
* installed in normal use and the feature carries no runtime cost for them.
*/
+19
View File
@@ -0,0 +1,19 @@
# Quality engineering instructions
Read the repository [AGENTS.md](../../AGENTS.md) first. Use the [quality engineering index](README.md) to find the tool for your task.
| Area | Responsibility | Current location |
| --- | --- | --- |
| Policy | Repository rules and static analysis | [`rules-engine`](policy/rules-engine/), [`code-health`](policy/code-health/) |
| Testing | Test selection, architecture checks, and test orchestration | [`test-impact`](testing/test-impact/), [`janitor`](testing/janitor/), [`playwright`](testing/playwright/) |
| Environments | Test services and deployment stacks | [`containers`](environments/containers/) |
| Efficiency | Speed, memory, load, latency, throughput, capacity, and resource use | [Microbenchmarks](efficiency/microbenchmarks/), [Playwright performance tests](testing/playwright/tests/performance/), [infrastructure benchmarks](testing/playwright/tests/infrastructure/benchmarks/), [benchmark CLI](efficiency/scale/benchmark/) |
## Working rules
- Read the instructions in the owning package before you change its files.
- Keep benchmark methods and results comparable when you move files. Do not change behavior during a path migration.
- Keep existing package names and `pnpm --filter` commands when you move workspaces.
- Update workspace discovery, CI paths, test-impact rules, ownership, documentation, and the lockfile when you move a package. Verify that each moved workspace still runs its intended checks.
- Prefer local efficiency measurements that do not require an n8n Docker image rebuild. State clearly when a benchmark does require an image.
- Put task instructions in focused README files. Keep inherited `AGENTS.md` files short and limited to rules for their area.
+12
View File
@@ -0,0 +1,12 @@
# Quality engineering
Each area owns workspaces. A test stays with its runner, even when it measures another quality concern.
| Area | Owns | Start here |
| --- | --- | --- |
| Policy | Repository rules and static checks | [Policy](policy/README.md) |
| Testing | Test selection, architecture checks, and the Playwright runner | [Testing](testing/README.md) |
| Environments | Services and deployment stacks for tests | [Environments](environments/README.md) |
| Efficiency | Standalone microbenchmark and load-test tools | [Efficiency](efficiency/README.md) |
Use the [Playwright suite map](testing/playwright/README.md#choose-a-suite) to find browser, accessibility, memory, and infrastructure tests. Their specs remain in Playwright because they use its fixtures and projects.
+8
View File
@@ -0,0 +1,8 @@
# Efficiency instructions
Read the parent [quality instructions](../AGENTS.md) before you change a benchmark.
- Compare results with the same machine, workload, and process profile.
- Do not change benchmark methods during a path migration.
- Keep local measurements possible without an n8n Docker image rebuild. State when a benchmark needs an image.
- Put task steps and result interpretation in the owning benchmark's README.
+10
View File
@@ -0,0 +1,10 @@
# Efficiency tooling
This area owns tools that measure speed and scale outside the Playwright runner.
| Tool | Use it to | Guide |
| --- | --- | --- |
| `@n8n/performance` | Compare code paths without building an n8n Docker image | [Microbenchmarks](microbenchmarks/README.md) |
| `@n8n/n8n-benchmark` | Run load scenarios against an n8n instance | [Benchmark CLI](scale/benchmark/README.md) |
For idle memory, canvas performance, throughput, and local soak tests, use the [Playwright suite map](../testing/playwright/README.md#choose-a-suite). These specs need Playwright fixtures, projects, metrics, or managed containers.
@@ -1,5 +1,5 @@
{
"extends": "../../../tsconfig.json",
"extends": "../../../../tsconfig.json",
"compilerOptions": {
"module": "esnext",
"moduleResolution": "bundler",
@@ -38,7 +38,7 @@ WORKDIR /app
COPY --chown=node:node ./pnpm-lock.yaml /app/pnpm-lock.yaml
COPY --chown=node:node ./pnpm-workspace.yaml /app/pnpm-workspace.yaml
COPY --chown=node:node ./package.json /app/package.json
COPY --chown=node:node ./packages/@n8n/benchmark/package.json /app/packages/@n8n/benchmark/package.json
COPY --chown=node:node ./packages/quality/efficiency/scale/benchmark/package.json /app/packages/quality/efficiency/scale/benchmark/package.json
COPY --chown=node:node ./patches /app/patches
COPY --chown=node:node ./scripts /app/scripts
@@ -47,15 +47,15 @@ RUN pnpm install --frozen-lockfile
# TS config files
COPY --chown=node:node ./packages/@n8n/typescript-config /app/packages/@n8n/typescript-config
COPY --chown=node:node ./packages/@n8n/benchmark/tsconfig.json /app/packages/@n8n/benchmark/tsconfig.json
COPY --chown=node:node ./packages/@n8n/benchmark/tsconfig.build.json /app/packages/@n8n/benchmark/tsconfig.build.json
COPY --chown=node:node ./packages/quality/efficiency/scale/benchmark/tsconfig.json /app/packages/quality/efficiency/scale/benchmark/tsconfig.json
COPY --chown=node:node ./packages/quality/efficiency/scale/benchmark/tsconfig.build.json /app/packages/quality/efficiency/scale/benchmark/tsconfig.build.json
# Source files
COPY --chown=node:node ./packages/@n8n/benchmark/src /app/packages/@n8n/benchmark/src
COPY --chown=node:node ./packages/@n8n/benchmark/bin /app/packages/@n8n/benchmark/bin
COPY --chown=node:node ./packages/@n8n/benchmark/scenarios /app/packages/@n8n/benchmark/scenarios
COPY --chown=node:node ./packages/quality/efficiency/scale/benchmark/src /app/packages/quality/efficiency/scale/benchmark/src
COPY --chown=node:node ./packages/quality/efficiency/scale/benchmark/bin /app/packages/quality/efficiency/scale/benchmark/bin
COPY --chown=node:node ./packages/quality/efficiency/scale/benchmark/scenarios /app/packages/quality/efficiency/scale/benchmark/scenarios
WORKDIR /app/packages/@n8n/benchmark
WORKDIR /app/packages/quality/efficiency/scale/benchmark
RUN pnpm build
#
@@ -64,7 +64,7 @@ FROM base AS runner
COPY --from=builder /app /app
WORKDIR /app/packages/@n8n/benchmark
WORKDIR /app/packages/quality/efficiency/scale/benchmark
USER node
ENTRYPOINT [ "/app/packages/@n8n/benchmark/bin/n8n-benchmark" ]
ENTRYPOINT [ "/app/packages/quality/efficiency/scale/benchmark/bin/n8n-benchmark" ]
@@ -2,10 +2,12 @@
Tool for executing benchmarks against an n8n instance.
Run the `pnpm --filter @n8n/n8n-benchmark` commands below from the repository root. pnpm runs each script in the benchmark package directory.
## Directory structure
```text
packages/@n8n/benchmark
packages/quality/efficiency/scale/benchmark
├── scenarios Benchmark scenarios
├── src Source code for the n8n-benchmark cli
├── Dockerfile Dockerfile for the n8n-benchmark cli
@@ -50,17 +52,17 @@ docker run -v ./scenarios:/scenarios ghcr.io/n8n-io/n8n-benchmark:latest run \
The benchmark suite consists of [benchmark scenarios](#benchmark-scenarios) and different [n8n setups](#n8n-setups).
### locally
### Locally
```sh
pnpm benchmark-locally
pnpm --filter @n8n/n8n-benchmark benchmark-locally
```
You can filter to a specific scenario and setup:
```sh
# Run only the http-node scenario with the sqlite setup
pnpm benchmark-locally --runDir /tmp/n8n-data --scenarioFilter http-node sqlite
pnpm --filter @n8n/n8n-benchmark benchmark-locally --runDir /tmp/n8n-data --scenarioFilter http-node sqlite
```
### In the cloud
@@ -71,9 +73,9 @@ Terraform code that creates it.
Create the environment, run the benchmarks, then delete the environment:
```sh
pnpm provision-cloud-env
pnpm benchmark-in-cloud
pnpm destroy-cloud-env
pnpm --filter @n8n/n8n-benchmark provision-cloud-env
pnpm --filter @n8n/n8n-benchmark benchmark-in-cloud
pnpm --filter @n8n/n8n-benchmark destroy-cloud-env
```
## Running the `n8n-benchmark` cli
@@ -87,7 +89,7 @@ Build the Docker image:
```sh
# Must be run in the repository root
# k6 doesn't have an arm64 build available for linux, we need to build against amd64
docker build --platform linux/amd64 -t n8n-benchmark -f packages/@n8n/benchmark/Dockerfile .
docker build --platform linux/amd64 -t n8n-benchmark -f packages/quality/efficiency/scale/benchmark/Dockerfile .
```
Run the image
@@ -108,11 +110,13 @@ Requirements:
- [k6](https://grafana.com/docs/k6/latest/set-up/install-k6/)
- Node.js v24 or higher
Set `N8N_USER_EMAIL` and `N8N_USER_PASSWORD` in your environment before you run the CLI.
```sh
pnpm build
pnpm --filter @n8n/n8n-benchmark build > benchmark-build.log 2>&1
# Run tests against http://localhost:5678 with specified email and password
N8N_USER_EMAIL=user@n8n.io N8N_USER_PASSWORD=password ./bin/n8n-benchmark run
pnpm --filter @n8n/n8n-benchmark exec ./bin/n8n-benchmark run
```
## Benchmark scenarios
@@ -127,4 +131,4 @@ Available scenarios are located in [`./scenarios`](./scenarios/).
## n8n setups
A n8n setup defines a single n8n runtime configuration using Docker compose. Different n8n setups are located in [`./scripts/n8nSetups`](./scripts/n8nSetups).
An n8n setup defines one n8n runtime configuration with Docker Compose. The setups are in [`scripts/n8n-setups`](scripts/n8n-setups/).
@@ -0,0 +1,7 @@
{
"$schema": "../../../../../node_modules/@biomejs/biome/configuration_schema.json",
"extends": ["../../../../../biome.jsonc"],
"files": {
"ignore": ["scripts/mock-api/**"]
}
}

Some files were not shown because too many files have changed in this diff Show More