fix(verify-pr): keep the mutation matrix off the tree live cells measure on (#15999)

* fix(verify-pr): keep the mutation matrix off the tree live cells measure on

A local round interleaved its mutation matrix with 80 live daemon cells
on one tree. A build inside a mutation row propagated the mutant into
dist/, and the hand-patched repair still carried the mutant expression,
so every live cell and the whole matrix were withdrawn and the round
shipped as inconclusive.

Require a dedicated tree for the mutation matrix, gate every daemon or
dist-level harness launch on a scripted dist-versus-source anchor check
with a mismatch-reporting control, and forbid hand-patching dist to
repair a contaminated build. Pin the new rule in the workflow meta-test.

* test(verify-pr): pin the launch gate's coverage and control clauses

Review on 0d06209b2d showed the four rig-isolation pins left two
clauses of the launch gate unasserted: deleting "covers every PR
anchor" or the mismatch-reporting control still passed 4/4. Pin both
so a future edit cannot silently weaken the gate.

Also name the clean-dist launch gate in the mutation-runner pointer,
so the section that proves a mutant reached dist and the rule that
proves the measurement tree's dist is clean cross-reference each other.

* fix(verify-pr): define the launch gate's reference side and isolate per-tree artifacts

Review on 21014c30d6 (R1-1/R1-2/R1-3, RC1-RC5): the gate compared dist
against an unnamed source, so a self-consistent mutant build passed it;
"PR anchor" was undefined and collided with the mutation-anchor sense;
the control proved liveness, not refusal; the gate had no carve-out for
the producing row's own run; and tree isolation stopped at the checkout
while the workspace links, the shared sandbox tag, and the single Maven
repository stayed host-global.

Name the reference side as a sha256 manifest of each measurement arm's
clean dist recorded before the matrix starts, retire the undefined
anchor term, require the gate to refuse a known-mutant build, exempt the
producing row's own launches, and extend isolation to node_modules, the
sandbox image tag, and the Maven repository. Replace the fragment pins
with the whole bullet pinned inside its section plus the whole pointer
line; a 17-row mutation acceptance goes red on every row, and a
real-file M8 run turned the suite red before a byte-identical restore.

* fix(verify-pr): qualify the matrix-tree copies and name both sandbox levers

Review on ff867ec035: the cp -al recipe dropped the per-package
qualifier the install strategy already requires (root marked 17.0.1
vs packages/core marked 15.0.12 shadow for real), hardlinks share
inodes so a writing row truncates the round's copy, the sandbox clause
named only the launch-side variable while the build reads
QWEN_SANDBOX_IMAGE_TAG and --sandbox-image outranks the export, and the
pin's section slice failed open when the closing heading was renamed.

Carry the qualifier and the never-symlink rule, require real copies
from the store for rows that write under node_modules, name the build
lever and the flag precedence, and assert the slice bounds. The
acceptance matrix grows to 26 rows, all red on mutation including both
heading renames; a real-file rename run turned the suite red before a
byte-identical restore.

* fix(verify-pr): scope the sandbox image per launch and bound the pin slice by the next heading

Review on b99f52a5e3: <tree> sat where a Docker tag must go and the
export's value was never spelled out; a shell-wide export leaks the
matrix image into every later cell; the two cp -al copies disagreed on
when hardlinks are safe (the base build's postinstall step writes under
node_modules); the slice bound was a named heading, so an inserted
section could shelter a moved bullet; and the prose named a build
variable no test pinned at its read site.

Spell a sanitized label and the full image reference, inject the
variable per launch instead of exporting it, carry the write caveat in
both copies with an explicit copy import method (pnpm's auto falls back
to hardlinks without clones), bound the slice on the next heading, and
pin process.env.QWEN_SANDBOX_IMAGE_TAG in build_sandbox.js. The
acceptance matrix grows to 30 rows, all red on mutation; real-file
heading-insertion and variable-rename runs each turned the suite red
before byte-identical restores. Merged main (#15988, #16002); the suite
now runs 197 tests, all green.
This commit is contained in:
Shaojin Wen authored and GitHub committed 2026-10-31 21:08:39 +08:00
1 parent 812a8c7ccb
commit 94448d368d
2 files changed
+106 -2

No files matched your search

+42 -2
View File
@@ -583,7 +583,11 @@ differs only by the change under test; the verdict is the pair of counts.
hardlinks are free, and the relative `@qwen-code/*` symlinks then resolve
into the base tree, which the realpath assertion above can confirm. Do not
symlink the directory itself — the internal links would resolve through it
back into the head tree and both arms would run head code.
back into the head tree and both arms would run head code. Hardlinks
share inodes, and the base build can write under `node_modules` (its
postinstall patch step), so a base arm that rebuilds needs real copies
from the store instead
(`pnpm install --offline --frozen-lockfile --config.package-import-method=copy`).
- Report the cell table: environment per cell, observable oracle per cell
(exit code, stderr line, wire request, rendered frame), and `X/Y` at head
vs control. "5/9 flip from broken to fixed" is the shape to aim for.
@@ -975,7 +979,8 @@ least one test that imports it.
(`../escape` has two dots and a slash). Likewise a mode-000 fixture
violates read and execute at once, so dropping either check survived 990
unit tests.
- Runner hygiene (time caps, PID-only kills, restoring the tree, syntax
- Runner hygiene (time caps, PID-only kills, restoring the tree, a tree of
its own apart from every live cell, the clean-`dist` launch gate, syntax
checks that do not execute) is in **Rig hygiene (local rounds)** below.
**A test that can skip is coverage only where it executes.** A test gated on
@@ -1267,6 +1272,41 @@ results in both directions.
- Run scenarios that kill processes, revoke grants or wedge a shared
resource alone on their arm; three runs that overlapped a concurrent
revocation probe had to be set aside and repeated.
- Give the mutation matrix a tree of its own — never the tree a live cell
is measured on. Restoring the source after each row is not enough: a
build inside a row propagates the mutant into `dist/`, and a daemon
started mid-row loads it. Isolate every artifact a launch resolves, not
just the checkout: `cp -al` the round's `node_modules` — root plus any
per-package ones — into the matrix tree (never symlink the directory
itself) and assert each `@qwen-code/*` link resolves inside it (a nested
tree without its own `node_modules` walks up to the round's); a row that
writes under `node_modules` needs real copies from the store
(`pnpm install --offline --frozen-lockfile --config.package-import-method=copy`),
because hardlinks share the round's inodes and pnpm's default import
falls back to hardlinks where clones are unavailable. Build the sandbox
image under a per-tree tag — a sanitized tree label, never a path
(`QWEN_SANDBOX_IMAGE_TAG=<label>`; the build reads that variable, not
`QWEN_SANDBOX_IMAGE`, and interpolates it into the tag verbatim) — and
inject `QWEN_SANDBOX_IMAGE=ghcr.io/qwenlm/qwen-code:<label>` per launch
of that tree's cells, never as a shell-wide export, which every later
cell inherits; pass no `--sandbox-image` in those cells — the flag
outranks the variable (the tag in `packages/cli/package.json` is one
shared name any tree's build retags). Give Java rows their own
`-Dmaven.repo.local` copied from the warmed repository. Before the
matrix starts, record a sha256 manifest of each measurement arm's
clean `dist/`, and gate every daemon or dist-level harness launch
outside the row that produced it on a scripted comparison against that
manifest — never against the matrix tree's live source, which mid-row
is the mutant and agrees with its own build. Prove the gate live by
requiring it to refuse a known-mutant build, not merely to report a
fabricated mismatch. Never hand-patch `dist/` to repair a contaminated
build; rebuild from source you have verified clean and re-record the
manifest from that build — a hand-patch that restores only the visible
sites leaves the rest of the build mutant. Measured example: a round
interleaved its mutation matrix with 80 live daemon cells on one tree,
then hand-patched the mutant it found in `dist/`; the patched build
still carried the mutant expression, so every live cell and the whole
matrix were withdrawn and the round shipped as `inconclusive`.
- Stop processes only by the PIDs you recorded. After every batch, list
orphaned (ppid 1) processes under the rig path and expect zero.
- Restore the tree and tear the rig down before exiting. `process.exit()`
@@ -5113,6 +5113,70 @@ describe('qwen-triage verify hardening round 2', () => {
'fresh database, storage and workspace per scenario and per arm',
);
expect(flat).toContain('orphaned (ppid 1) processes');
// PR #13748 local round (2026-10-31): the mutation matrix and 80 live
// daemon cells shared one tree; a hand-patched dist still carried the
// mutant expression, and the whole round was withdrawn as inconclusive.
// Review on 21014c30d6: fragment pins left the operative clauses and the
// measured example unasserted, so the bullet is pinned whole inside its
// section: deleting any clause, flipping the exclusion, or moving the
// bullet out of Rig hygiene turns this red. Review on ff867ec035: the
// slice bounds are asserted, because a renamed closing heading would
// otherwise fail open (slice(start, -1) spans the rest of the document).
// Review on b99f52a5e3: the end bound is the NEXT heading, not a named
// one, so a section inserted before the old bound cannot shelter a moved
// bullet; and the sandbox variable the prose names is pinned at its read
// site, so renaming it cannot leave the recipe pointing at nothing.
const rigHygieneStart = flat.indexOf('### Rig hygiene (local rounds)');
const rigHygieneEnd = flat.indexOf('### ', rigHygieneStart + 4);
expect(rigHygieneStart).toBeGreaterThanOrEqual(0);
expect(rigHygieneEnd).toBeGreaterThan(rigHygieneStart);
const rigHygiene = flat.slice(rigHygieneStart, rigHygieneEnd);
expect(rigHygiene).toContain(
'Give the mutation matrix a tree of its own — never the tree a live ' +
'cell is measured on. Restoring the source after each row is not ' +
'enough: a build inside a row propagates the mutant into `dist/`, and a ' +
'daemon started mid-row loads it. Isolate every artifact a launch ' +
"resolves, not just the checkout: `cp -al` the round's `node_modules` — " +
'root plus any per-package ones — into the matrix tree (never symlink ' +
'the directory itself) and assert each `@qwen-code/*` link resolves ' +
'inside it (a nested tree without its own `node_modules` walks up to ' +
"the round's); a row that writes under `node_modules` needs real copies " +
'from the store (`pnpm install --offline --frozen-lockfile ' +
'--config.package-import-method=copy`), because hardlinks share the ' +
"round's inodes and pnpm's default import falls back to hardlinks where " +
'clones are unavailable. Build the sandbox image under a per-tree tag — ' +
'a sanitized tree label, never a path ' +
'(`QWEN_SANDBOX_IMAGE_TAG=<label>`; the build reads that variable, not ' +
'`QWEN_SANDBOX_IMAGE`, and interpolates it into the tag verbatim) — and ' +
'inject `QWEN_SANDBOX_IMAGE=ghcr.io/qwenlm/qwen-code:<label>` per ' +
"launch of that tree's cells, never as a shell-wide export, which every " +
'later cell inherits; pass no `--sandbox-image` in those cells — the ' +
'flag outranks the variable (the tag in `packages/cli/package.json` is ' +
"one shared name any tree's build retags). Give Java rows their own " +
'`-Dmaven.repo.local` copied from the warmed repository. Before the ' +
"matrix starts, record a sha256 manifest of each measurement arm's " +
'clean `dist/`, and gate every daemon or dist-level harness launch ' +
'outside the row that produced it on a scripted comparison against that ' +
"manifest — never against the matrix tree's live source, which mid-row " +
'is the mutant and agrees with its own build. Prove the gate live by ' +
'requiring it to refuse a known-mutant build, not merely to report a ' +
'fabricated mismatch. Never hand-patch `dist/` to repair a contaminated ' +
'build; rebuild from source you have verified clean and re-record the ' +
'manifest from that build — a hand-patch that restores only the visible ' +
'sites leaves the rest of the build mutant. Measured example: a round ' +
'interleaved its mutation matrix with 80 live daemon cells on one tree, ' +
'then hand-patched the mutant it found in `dist/`; the patched build ' +
'still carried the mutant expression, so every live cell and the whole ' +
'matrix were withdrawn and the round shipped as `inconclusive`.',
);
expect(flat).toContain(
'Runner hygiene (time caps, PID-only kills, restoring the tree, a tree ' +
'of its own apart from every live cell, the clean-`dist` launch gate, ' +
'syntax checks that do not execute) is in **Rig hygiene (local ' +
'rounds)** below.',
);
const sandboxBuild = readFileSync('scripts/build_sandbox.js', 'utf8');
expect(sandboxBuild).toContain('process.env.QWEN_SANDBOX_IMAGE_TAG');
expect(flat).toContain('`process.exit()` skips `finally`');
expect(flat).toContain('with `node --check`, never `import()`');
expect(flat).toContain('Do not edit a shell runner while it executes');