mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
fix(verify-pr): keep the mutation matrix off the tree live cells measure on (#15999)
* fix(verify-pr): keep the mutation matrix off the tree live cells measure on A local round interleaved its mutation matrix with 80 live daemon cells on one tree. A build inside a mutation row propagated the mutant into dist/, and the hand-patched repair still carried the mutant expression, so every live cell and the whole matrix were withdrawn and the round shipped as inconclusive. Require a dedicated tree for the mutation matrix, gate every daemon or dist-level harness launch on a scripted dist-versus-source anchor check with a mismatch-reporting control, and forbid hand-patching dist to repair a contaminated build. Pin the new rule in the workflow meta-test. * test(verify-pr): pin the launch gate's coverage and control clauses Review on0d06209b2dshowed the four rig-isolation pins left two clauses of the launch gate unasserted: deleting "covers every PR anchor" or the mismatch-reporting control still passed 4/4. Pin both so a future edit cannot silently weaken the gate. Also name the clean-dist launch gate in the mutation-runner pointer, so the section that proves a mutant reached dist and the rule that proves the measurement tree's dist is clean cross-reference each other. * fix(verify-pr): define the launch gate's reference side and isolate per-tree artifacts Review on21014c30d6(R1-1/R1-2/R1-3, RC1-RC5): the gate compared dist against an unnamed source, so a self-consistent mutant build passed it; "PR anchor" was undefined and collided with the mutation-anchor sense; the control proved liveness, not refusal; the gate had no carve-out for the producing row's own run; and tree isolation stopped at the checkout while the workspace links, the shared sandbox tag, and the single Maven repository stayed host-global. Name the reference side as a sha256 manifest of each measurement arm's clean dist recorded before the matrix starts, retire the undefined anchor term, require the gate to refuse a known-mutant build, exempt the producing row's own launches, and extend isolation to node_modules, the sandbox image tag, and the Maven repository. Replace the fragment pins with the whole bullet pinned inside its section plus the whole pointer line; a 17-row mutation acceptance goes red on every row, and a real-file M8 run turned the suite red before a byte-identical restore. * fix(verify-pr): qualify the matrix-tree copies and name both sandbox levers Review onff867ec035: the cp -al recipe dropped the per-package qualifier the install strategy already requires (root marked 17.0.1 vs packages/core marked 15.0.12 shadow for real), hardlinks share inodes so a writing row truncates the round's copy, the sandbox clause named only the launch-side variable while the build reads QWEN_SANDBOX_IMAGE_TAG and --sandbox-image outranks the export, and the pin's section slice failed open when the closing heading was renamed. Carry the qualifier and the never-symlink rule, require real copies from the store for rows that write under node_modules, name the build lever and the flag precedence, and assert the slice bounds. The acceptance matrix grows to 26 rows, all red on mutation including both heading renames; a real-file rename run turned the suite red before a byte-identical restore. * fix(verify-pr): scope the sandbox image per launch and bound the pin slice by the next heading Review onb99f52a5e3: <tree> sat where a Docker tag must go and the export's value was never spelled out; a shell-wide export leaks the matrix image into every later cell; the two cp -al copies disagreed on when hardlinks are safe (the base build's postinstall step writes under node_modules); the slice bound was a named heading, so an inserted section could shelter a moved bullet; and the prose named a build variable no test pinned at its read site. Spell a sanitized label and the full image reference, inject the variable per launch instead of exporting it, carry the write caveat in both copies with an explicit copy import method (pnpm's auto falls back to hardlinks without clones), bound the slice on the next heading, and pin process.env.QWEN_SANDBOX_IMAGE_TAG in build_sandbox.js. The acceptance matrix grows to 30 rows, all red on mutation; real-file heading-insertion and variable-rename runs each turned the suite red before byte-identical restores. Merged main (#15988, #16002); the suite now runs 197 tests, all green.
This commit is contained in:
1 parent
812a8c7ccb
commit
94448d368d
2 files changed
+106
-2
No files matched your search
@@ -583,7 +583,11 @@ differs only by the change under test; the verdict is the pair of counts.
|
||||
hardlinks are free, and the relative `@qwen-code/*` symlinks then resolve
|
||||
into the base tree, which the realpath assertion above can confirm. Do not
|
||||
symlink the directory itself — the internal links would resolve through it
|
||||
back into the head tree and both arms would run head code.
|
||||
back into the head tree and both arms would run head code. Hardlinks
|
||||
share inodes, and the base build can write under `node_modules` (its
|
||||
postinstall patch step), so a base arm that rebuilds needs real copies
|
||||
from the store instead
|
||||
(`pnpm install --offline --frozen-lockfile --config.package-import-method=copy`).
|
||||
- Report the cell table: environment per cell, observable oracle per cell
|
||||
(exit code, stderr line, wire request, rendered frame), and `X/Y` at head
|
||||
vs control. "5/9 flip from broken to fixed" is the shape to aim for.
|
||||
@@ -975,7 +979,8 @@ least one test that imports it.
|
||||
(`../escape` has two dots and a slash). Likewise a mode-000 fixture
|
||||
violates read and execute at once, so dropping either check survived 990
|
||||
unit tests.
|
||||
- Runner hygiene (time caps, PID-only kills, restoring the tree, syntax
|
||||
- Runner hygiene (time caps, PID-only kills, restoring the tree, a tree of
|
||||
its own apart from every live cell, the clean-`dist` launch gate, syntax
|
||||
checks that do not execute) is in **Rig hygiene (local rounds)** below.
|
||||
|
||||
**A test that can skip is coverage only where it executes.** A test gated on
|
||||
@@ -1267,6 +1272,41 @@ results in both directions.
|
||||
- Run scenarios that kill processes, revoke grants or wedge a shared
|
||||
resource alone on their arm; three runs that overlapped a concurrent
|
||||
revocation probe had to be set aside and repeated.
|
||||
- Give the mutation matrix a tree of its own — never the tree a live cell
|
||||
is measured on. Restoring the source after each row is not enough: a
|
||||
build inside a row propagates the mutant into `dist/`, and a daemon
|
||||
started mid-row loads it. Isolate every artifact a launch resolves, not
|
||||
just the checkout: `cp -al` the round's `node_modules` — root plus any
|
||||
per-package ones — into the matrix tree (never symlink the directory
|
||||
itself) and assert each `@qwen-code/*` link resolves inside it (a nested
|
||||
tree without its own `node_modules` walks up to the round's); a row that
|
||||
writes under `node_modules` needs real copies from the store
|
||||
(`pnpm install --offline --frozen-lockfile --config.package-import-method=copy`),
|
||||
because hardlinks share the round's inodes and pnpm's default import
|
||||
falls back to hardlinks where clones are unavailable. Build the sandbox
|
||||
image under a per-tree tag — a sanitized tree label, never a path
|
||||
(`QWEN_SANDBOX_IMAGE_TAG=<label>`; the build reads that variable, not
|
||||
`QWEN_SANDBOX_IMAGE`, and interpolates it into the tag verbatim) — and
|
||||
inject `QWEN_SANDBOX_IMAGE=ghcr.io/qwenlm/qwen-code:<label>` per launch
|
||||
of that tree's cells, never as a shell-wide export, which every later
|
||||
cell inherits; pass no `--sandbox-image` in those cells — the flag
|
||||
outranks the variable (the tag in `packages/cli/package.json` is one
|
||||
shared name any tree's build retags). Give Java rows their own
|
||||
`-Dmaven.repo.local` copied from the warmed repository. Before the
|
||||
matrix starts, record a sha256 manifest of each measurement arm's
|
||||
clean `dist/`, and gate every daemon or dist-level harness launch
|
||||
outside the row that produced it on a scripted comparison against that
|
||||
manifest — never against the matrix tree's live source, which mid-row
|
||||
is the mutant and agrees with its own build. Prove the gate live by
|
||||
requiring it to refuse a known-mutant build, not merely to report a
|
||||
fabricated mismatch. Never hand-patch `dist/` to repair a contaminated
|
||||
build; rebuild from source you have verified clean and re-record the
|
||||
manifest from that build — a hand-patch that restores only the visible
|
||||
sites leaves the rest of the build mutant. Measured example: a round
|
||||
interleaved its mutation matrix with 80 live daemon cells on one tree,
|
||||
then hand-patched the mutant it found in `dist/`; the patched build
|
||||
still carried the mutant expression, so every live cell and the whole
|
||||
matrix were withdrawn and the round shipped as `inconclusive`.
|
||||
- Stop processes only by the PIDs you recorded. After every batch, list
|
||||
orphaned (ppid 1) processes under the rig path and expect zero.
|
||||
- Restore the tree and tear the rig down before exiting. `process.exit()`
|
||||
|
||||
@@ -5113,6 +5113,70 @@ describe('qwen-triage verify hardening round 2', () => {
|
||||
'fresh database, storage and workspace per scenario and per arm',
|
||||
);
|
||||
expect(flat).toContain('orphaned (ppid 1) processes');
|
||||
// PR #13748 local round (2026-10-31): the mutation matrix and 80 live
|
||||
// daemon cells shared one tree; a hand-patched dist still carried the
|
||||
// mutant expression, and the whole round was withdrawn as inconclusive.
|
||||
// Review on 21014c30d6: fragment pins left the operative clauses and the
|
||||
// measured example unasserted, so the bullet is pinned whole inside its
|
||||
// section: deleting any clause, flipping the exclusion, or moving the
|
||||
// bullet out of Rig hygiene turns this red. Review on ff867ec035: the
|
||||
// slice bounds are asserted, because a renamed closing heading would
|
||||
// otherwise fail open (slice(start, -1) spans the rest of the document).
|
||||
// Review on b99f52a5e3: the end bound is the NEXT heading, not a named
|
||||
// one, so a section inserted before the old bound cannot shelter a moved
|
||||
// bullet; and the sandbox variable the prose names is pinned at its read
|
||||
// site, so renaming it cannot leave the recipe pointing at nothing.
|
||||
const rigHygieneStart = flat.indexOf('### Rig hygiene (local rounds)');
|
||||
const rigHygieneEnd = flat.indexOf('### ', rigHygieneStart + 4);
|
||||
expect(rigHygieneStart).toBeGreaterThanOrEqual(0);
|
||||
expect(rigHygieneEnd).toBeGreaterThan(rigHygieneStart);
|
||||
const rigHygiene = flat.slice(rigHygieneStart, rigHygieneEnd);
|
||||
expect(rigHygiene).toContain(
|
||||
'Give the mutation matrix a tree of its own — never the tree a live ' +
|
||||
'cell is measured on. Restoring the source after each row is not ' +
|
||||
'enough: a build inside a row propagates the mutant into `dist/`, and a ' +
|
||||
'daemon started mid-row loads it. Isolate every artifact a launch ' +
|
||||
"resolves, not just the checkout: `cp -al` the round's `node_modules` — " +
|
||||
'root plus any per-package ones — into the matrix tree (never symlink ' +
|
||||
'the directory itself) and assert each `@qwen-code/*` link resolves ' +
|
||||
'inside it (a nested tree without its own `node_modules` walks up to ' +
|
||||
"the round's); a row that writes under `node_modules` needs real copies " +
|
||||
'from the store (`pnpm install --offline --frozen-lockfile ' +
|
||||
'--config.package-import-method=copy`), because hardlinks share the ' +
|
||||
"round's inodes and pnpm's default import falls back to hardlinks where " +
|
||||
'clones are unavailable. Build the sandbox image under a per-tree tag — ' +
|
||||
'a sanitized tree label, never a path ' +
|
||||
'(`QWEN_SANDBOX_IMAGE_TAG=<label>`; the build reads that variable, not ' +
|
||||
'`QWEN_SANDBOX_IMAGE`, and interpolates it into the tag verbatim) — and ' +
|
||||
'inject `QWEN_SANDBOX_IMAGE=ghcr.io/qwenlm/qwen-code:<label>` per ' +
|
||||
"launch of that tree's cells, never as a shell-wide export, which every " +
|
||||
'later cell inherits; pass no `--sandbox-image` in those cells — the ' +
|
||||
'flag outranks the variable (the tag in `packages/cli/package.json` is ' +
|
||||
"one shared name any tree's build retags). Give Java rows their own " +
|
||||
'`-Dmaven.repo.local` copied from the warmed repository. Before the ' +
|
||||
"matrix starts, record a sha256 manifest of each measurement arm's " +
|
||||
'clean `dist/`, and gate every daemon or dist-level harness launch ' +
|
||||
'outside the row that produced it on a scripted comparison against that ' +
|
||||
"manifest — never against the matrix tree's live source, which mid-row " +
|
||||
'is the mutant and agrees with its own build. Prove the gate live by ' +
|
||||
'requiring it to refuse a known-mutant build, not merely to report a ' +
|
||||
'fabricated mismatch. Never hand-patch `dist/` to repair a contaminated ' +
|
||||
'build; rebuild from source you have verified clean and re-record the ' +
|
||||
'manifest from that build — a hand-patch that restores only the visible ' +
|
||||
'sites leaves the rest of the build mutant. Measured example: a round ' +
|
||||
'interleaved its mutation matrix with 80 live daemon cells on one tree, ' +
|
||||
'then hand-patched the mutant it found in `dist/`; the patched build ' +
|
||||
'still carried the mutant expression, so every live cell and the whole ' +
|
||||
'matrix were withdrawn and the round shipped as `inconclusive`.',
|
||||
);
|
||||
expect(flat).toContain(
|
||||
'Runner hygiene (time caps, PID-only kills, restoring the tree, a tree ' +
|
||||
'of its own apart from every live cell, the clean-`dist` launch gate, ' +
|
||||
'syntax checks that do not execute) is in **Rig hygiene (local ' +
|
||||
'rounds)** below.',
|
||||
);
|
||||
const sandboxBuild = readFileSync('scripts/build_sandbox.js', 'utf8');
|
||||
expect(sandboxBuild).toContain('process.env.QWEN_SANDBOX_IMAGE_TAG');
|
||||
expect(flat).toContain('`process.exit()` skips `finally`');
|
||||
expect(flat).toContain('with `node --check`, never `import()`');
|
||||
expect(flat).toContain('Do not edit a shell runner while it executes');
|
||||
|
||||
Reference in new issue
Block a user