mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
fix(ci): clear the pinned corepack cache before every install (#16351)
* fix(ci): clear the pinned corepack cache before every install The Linux test shards have been dying seconds into Install dependencies with MODULE_NOT_FOUND on the cached pnpm's bin/pnpm.mjs since #16328 pinned COREPACK_HOME to runner.temp (main runs 41876940141, 41889761157, 41927887452; six different self-hosted hosts). The pin assumed runner.temp is per-run, but self-hosted runners keep _work/_temp across jobs — the lint job's isolated-HOME step already documents this. corepack trusts an existing cache entry without re-verifying its files, so a partial pnpm tree left by an earlier job fails every later job on that host. Clear the pinned cache at the start of the shared install body in all three jobs that copy it, enforcing the per-run contract the pin documents. The guard keeps the reset a no-op where COREPACK_HOME is unset (test_build) or already run-unique (lint). The test step still reuses the cache install warms in the same job, and poisoned hosts self-heal on their next job without manual cleanup. Pin the reset and the install/test COREPACK_HOME match in the workflow meta-test, and bump the size baseline for the added lines. * fix(ci): scope the corepack cache reset to the shard-owned path Review found two over-deletion shapes in the nonempty guard: a runner that exports COREPACK_HOME would lose its entire shared cache in the build job, and hosted lint would lose the cache its pnpm-store step had just warmed, re-exposing the install to a registry outage. Match only the exact shard-owned path instead; the build and lint copies become unconditional no-ops and the three install bodies stay byte-identical. Add an execution test that runs the shipped guard fragment under a scrubbed environment across five cache shapes: it kills the dead-guard mutant the string pins survived, and a widened-guard mutant dies on the shared-cache survival cell. State the incident as observations with the corruption source undetermined, and correct the stale per-run rationale beside the pin.
This commit is contained in:
1 parent
f3488b47fa
commit
ef9c891b8f
3 files changed
+193
-11
No files matched your search
@@ -19,7 +19,7 @@
|
||||
9551 build-and-publish-image.yml
|
||||
55276 cd-cua-driver.yml
|
||||
2222 cd-mobile-mcp.yml
|
||||
192263 ci.yml
|
||||
195906 ci.yml
|
||||
4193 codeql.yml
|
||||
9389 comment-attachment-guard.yml
|
||||
2090 desktop-packaging-check.yml
|
||||
|
||||
@@ -679,6 +679,17 @@ jobs:
|
||||
( while sleep 10; do sample_disk; done ) &
|
||||
SAMPLER_PID=$!
|
||||
trap 'pkill -TERM -P "$SAMPLER_PID" 2>/dev/null || true; kill "$SAMPLER_PID" 2>/dev/null || true' EXIT
|
||||
# Observed: the pinned cache entry existed with its pnpm binary
|
||||
# missing, corepack trusts an existing entry without re-verifying
|
||||
# its files, and the install died instantly with MODULE_NOT_FOUND
|
||||
# (main run 41927887452). What left the entry incomplete is
|
||||
# undetermined; _work/_temp survives across jobs on self-hosted
|
||||
# runners, so such state persists. Start the shard-owned cache
|
||||
# empty — and only that exact path, so a runner-exported shared
|
||||
# cache and lint's already-warmed run-unique home stay untouched.
|
||||
if [ "${COREPACK_HOME:-}" = "${RUNNER_TEMP:-}/qwen-ci-corepack" ]; then
|
||||
rm -rf -- "${COREPACK_HOME}"
|
||||
fi
|
||||
corepack pnpm install --frozen-lockfile --prefer-offline --reporter=append-only
|
||||
|
||||
- name: 'Report npm cache usage (self-hosted)'
|
||||
@@ -1078,11 +1089,13 @@ jobs:
|
||||
env:
|
||||
QWEN_SKIP_PREPARE: '1'
|
||||
QWEN_SKIP_NOTICE_GENERATION: '1'
|
||||
# Pin COREPACK_HOME to a per-run path so the corepack cache written
|
||||
# here is visible to the test step below, which overrides HOME for
|
||||
# isolation. corepack does not re-verify extracted entries on cache
|
||||
# hit, so this must NOT be a shared/persistent path (poisoning
|
||||
# channel) — per-run is safe and sufficient.
|
||||
# Pin COREPACK_HOME to a fixed path under runner.temp so the
|
||||
# corepack cache written here is visible to the test step below,
|
||||
# which overrides HOME for isolation. runner.temp survives across
|
||||
# jobs on self-hosted runners, and corepack does not re-verify
|
||||
# extracted entries on cache hit, so the install body below clears
|
||||
# this exact path before use; the test step then reuses the cache
|
||||
# this step warms within the same job.
|
||||
COREPACK_HOME: '${{ runner.temp }}/qwen-ci-corepack'
|
||||
run: |-
|
||||
DISK_SAMPLES="${RUNNER_TEMP}/disk-pressure-samples.log"
|
||||
@@ -1097,6 +1110,17 @@ jobs:
|
||||
( while sleep 10; do sample_disk; done ) &
|
||||
SAMPLER_PID=$!
|
||||
trap 'pkill -TERM -P "$SAMPLER_PID" 2>/dev/null || true; kill "$SAMPLER_PID" 2>/dev/null || true' EXIT
|
||||
# Observed: the pinned cache entry existed with its pnpm binary
|
||||
# missing, corepack trusts an existing entry without re-verifying
|
||||
# its files, and the install died instantly with MODULE_NOT_FOUND
|
||||
# (main run 41927887452). What left the entry incomplete is
|
||||
# undetermined; _work/_temp survives across jobs on self-hosted
|
||||
# runners, so such state persists. Start the shard-owned cache
|
||||
# empty — and only that exact path, so a runner-exported shared
|
||||
# cache and lint's already-warmed run-unique home stay untouched.
|
||||
if [ "${COREPACK_HOME:-}" = "${RUNNER_TEMP:-}/qwen-ci-corepack" ]; then
|
||||
rm -rf -- "${COREPACK_HOME}"
|
||||
fi
|
||||
corepack pnpm install --frozen-lockfile --prefer-offline --reporter=append-only
|
||||
|
||||
# Eight shards pull this same artifact at once, so the cap has to clear
|
||||
@@ -1873,6 +1897,17 @@ jobs:
|
||||
( while sleep 10; do sample_disk; done ) &
|
||||
SAMPLER_PID=$!
|
||||
trap 'pkill -TERM -P "$SAMPLER_PID" 2>/dev/null || true; kill "$SAMPLER_PID" 2>/dev/null || true' EXIT
|
||||
# Observed: the pinned cache entry existed with its pnpm binary
|
||||
# missing, corepack trusts an existing entry without re-verifying
|
||||
# its files, and the install died instantly with MODULE_NOT_FOUND
|
||||
# (main run 41927887452). What left the entry incomplete is
|
||||
# undetermined; _work/_temp survives across jobs on self-hosted
|
||||
# runners, so such state persists. Start the shard-owned cache
|
||||
# empty — and only that exact path, so a runner-exported shared
|
||||
# cache and lint's already-warmed run-unique home stay untouched.
|
||||
if [ "${COREPACK_HOME:-}" = "${RUNNER_TEMP:-}/qwen-ci-corepack" ]; then
|
||||
rm -rf -- "${COREPACK_HOME}"
|
||||
fi
|
||||
corepack pnpm install --frozen-lockfile --prefer-offline --reporter=append-only
|
||||
|
||||
- name: 'Report npm cache usage (self-hosted)'
|
||||
|
||||
@@ -27,7 +27,14 @@
|
||||
// place so that stays a revert.
|
||||
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import {
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
readFileSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
@@ -839,9 +846,10 @@ describe('post-merge push lane', () => {
|
||||
QWEN_SKIP_NOTICE_GENERATION: '1',
|
||||
// The shards' test step overrides HOME, which would hide the
|
||||
// corepack cache install warms here; pin COREPACK_HOME to a
|
||||
// per-run path so the warmup hits it. lint_and_static routes
|
||||
// HOME and COREPACK_HOME together via GITHUB_ENV, so its
|
||||
// install needs no step-level override.
|
||||
// fixed path under runner.temp so the warmup hits it (the
|
||||
// install body clears that exact path first). lint_and_static
|
||||
// routes HOME and COREPACK_HOME together via GITHUB_ENV, so
|
||||
// its install needs no step-level override.
|
||||
COREPACK_HOME: '${{ runner.temp }}/qwen-ci-corepack',
|
||||
});
|
||||
} else {
|
||||
@@ -897,9 +905,48 @@ describe('post-merge push lane', () => {
|
||||
QWEN_SKIP_PREPARE: '1',
|
||||
QWEN_SKIP_NOTICE_GENERATION: '1',
|
||||
// The test step's HOME override would hide the corepack cache install
|
||||
// warms here; pin COREPACK_HOME to a per-run path so the warmup hits it.
|
||||
// warms here; pin COREPACK_HOME to a fixed path under runner.temp so
|
||||
// the warmup hits it (the install body clears that exact path first).
|
||||
COREPACK_HOME: '${{ runner.temp }}/qwen-ci-corepack',
|
||||
});
|
||||
// Observed on the fleet: the pinned cache entry existed with its pnpm
|
||||
// binary missing, corepack trusts an existing entry without re-verifying
|
||||
// its files, and Install dependencies died instantly with
|
||||
// MODULE_NOT_FOUND (main runs 41876940141, 41889761157, 41927887452).
|
||||
// What left the entry incomplete is undetermined; _work/_temp survives
|
||||
// across jobs on self-hosted runners (the lint job's isolated-HOME step
|
||||
// documents the same fact), so such state persists. Every copy of the
|
||||
// shared install body must clear the cache before corepack runs, and the
|
||||
// guard must match ONLY the shard-owned path: a runner-exported shared
|
||||
// COREPACK_HOME (test_build) and lint's run-unique home — already warmed
|
||||
// by the hosted pnpm-store step — must never be deletion targets. The
|
||||
// execution test below holds the semantics; these pins hold the spelling.
|
||||
for (const job of ['test_build', 'test', 'lint_and_static']) {
|
||||
const body = String(install(job).run ?? '');
|
||||
expect(
|
||||
body,
|
||||
`${job} must scope the reset to the shard-owned path`,
|
||||
).toContain(
|
||||
'if [ "${COREPACK_HOME:-}" = "${RUNNER_TEMP:-}/qwen-ci-corepack" ]; then',
|
||||
);
|
||||
const reset = body.indexOf('rm -rf -- "${COREPACK_HOME}"');
|
||||
expect(
|
||||
reset,
|
||||
`${job} must reset the pinned corepack cache`,
|
||||
).toBeGreaterThanOrEqual(0);
|
||||
expect(
|
||||
body.indexOf('corepack pnpm install'),
|
||||
`${job} must reset the cache before corepack runs`,
|
||||
).toBeGreaterThan(reset);
|
||||
}
|
||||
// The test step must keep sharing the cache install warms: a drifted
|
||||
// COREPACK_HOME sends the vitest globalSetup warmup back to the cold
|
||||
// npmjs download whose ETIMEDOUT flake #16328 removed.
|
||||
expect(
|
||||
ci.jobs.test.steps.find(
|
||||
(s) => s.name === 'Run tests and generate reports',
|
||||
).env.COREPACK_HOME,
|
||||
).toBe(install('test').env.COREPACK_HOME);
|
||||
const namesIn = (job) => (ci.jobs[job].steps ?? []).map((x) => x.name);
|
||||
// Values alone are order-blind, and order is load-bearing here: `steps`
|
||||
// exposes only steps that already ran, and a missing property reads as
|
||||
@@ -1028,6 +1075,106 @@ describe('post-merge push lane', () => {
|
||||
expect(fetchCode).toContain('Incremental lint history:');
|
||||
});
|
||||
|
||||
it.skipIf(process.platform === 'win32')(
|
||||
'executes the shared corepack reset only against the shard-owned path',
|
||||
() => {
|
||||
// Semantic witness for the reset guard. The string pins above hold its
|
||||
// spelling, but a guard that never executes satisfies every one of
|
||||
// them (the verify-pr round on 01d2b116 survived exactly that mutant),
|
||||
// so run the shipped fragment — sliced from the real install body, not
|
||||
// a copy — and watch what it deletes. The environment is built from
|
||||
// scratch: this suite also runs inside CI shards that export their own
|
||||
// COREPACK_HOME and RUNNER_TEMP, and inheriting those would point the
|
||||
// cleanup at the running job's own directories.
|
||||
const step = ci.jobs.test.steps.find(
|
||||
(s) => s.name === 'Install dependencies',
|
||||
);
|
||||
const body = String(step.run ?? '');
|
||||
const start = body.indexOf('if [ "${COREPACK_HOME:-}"');
|
||||
const end = body.indexOf('corepack pnpm install');
|
||||
expect(
|
||||
start,
|
||||
'reset guard is missing from the install body',
|
||||
).toBeGreaterThanOrEqual(0);
|
||||
expect(end, 'reset must sit before the corepack install').toBeGreaterThan(
|
||||
start,
|
||||
);
|
||||
const fragment = body.slice(start, end);
|
||||
const cells = [
|
||||
{
|
||||
name: 'shard-owned cache is cleared',
|
||||
home: (temp) => join(temp, 'qwen-ci-corepack'),
|
||||
cleared: true,
|
||||
},
|
||||
{
|
||||
name: 'runner-exported shared cache survives',
|
||||
home: (_temp, root) => join(root, 'shared-corepack'),
|
||||
cleared: false,
|
||||
},
|
||||
{
|
||||
name: 'lint run-unique warmed home survives',
|
||||
home: (temp) =>
|
||||
join(temp, 'qwen-lint-home-42-1', '.cache', 'node', 'corepack'),
|
||||
cleared: false,
|
||||
},
|
||||
{
|
||||
name: 'unset variable leaves the default cache alone',
|
||||
home: null,
|
||||
cleared: false,
|
||||
},
|
||||
{
|
||||
name: 'empty variable is not a deletion target',
|
||||
home: () => '',
|
||||
cleared: false,
|
||||
},
|
||||
];
|
||||
for (const cell of cells) {
|
||||
const root = mkdtempSync(join(tmpdir(), 'corepack-reset-'));
|
||||
try {
|
||||
const temp = join(root, 'temp');
|
||||
const home = join(root, 'home');
|
||||
mkdirSync(temp, { recursive: true });
|
||||
mkdirSync(home, { recursive: true });
|
||||
// With no COREPACK_HOME the default cache is the deletion target
|
||||
// a naive guard would reach; give every cell a marker to watch.
|
||||
const target =
|
||||
cell.home === null
|
||||
? join(home, '.cache', 'node', 'corepack')
|
||||
: cell.home(temp, root);
|
||||
const marker = join(target, 'v1', 'pnpm', '11.24.0', '.corepack');
|
||||
if (target) {
|
||||
mkdirSync(join(target, 'v1', 'pnpm', '11.24.0'), {
|
||||
recursive: true,
|
||||
});
|
||||
writeFileSync(marker, '{}');
|
||||
}
|
||||
const bystander = join(root, 'bystander');
|
||||
mkdirSync(bystander, { recursive: true });
|
||||
writeFileSync(join(bystander, 'sentinel'), 'keep');
|
||||
const env = {
|
||||
PATH: process.env.PATH,
|
||||
HOME: home,
|
||||
RUNNER_TEMP: temp,
|
||||
};
|
||||
if (cell.home !== null) env.COREPACK_HOME = target;
|
||||
execFileSync(
|
||||
'bash',
|
||||
['--noprofile', '--norc', '-e', '-o', 'pipefail', '-c', fragment],
|
||||
{ env, cwd: root },
|
||||
);
|
||||
if (cell.cleared) {
|
||||
expect(existsSync(target), cell.name).toBe(false);
|
||||
} else if (target) {
|
||||
expect(existsSync(marker), cell.name).toBe(true);
|
||||
}
|
||||
expect(existsSync(join(bystander, 'sentinel')), cell.name).toBe(true);
|
||||
} finally {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
['first fetch aborts', 1, 99, 99, 1, 0, 'deepen fetch aborted'],
|
||||
['third fetch aborts', 3, 99, 99, 3, 2, 'deepen fetch aborted'],
|
||||
|
||||
Reference in new issue
Block a user