fix(ci): clear the pinned corepack cache before every install (#16351)

* fix(ci): clear the pinned corepack cache before every install

The Linux test shards have been dying seconds into Install dependencies
with MODULE_NOT_FOUND on the cached pnpm's bin/pnpm.mjs since #16328
pinned COREPACK_HOME to runner.temp (main runs 41876940141, 41889761157,
41927887452; six different self-hosted hosts). The pin assumed runner.temp
is per-run, but self-hosted runners keep _work/_temp across jobs — the
lint job's isolated-HOME step already documents this. corepack trusts an
existing cache entry without re-verifying its files, so a partial pnpm
tree left by an earlier job fails every later job on that host.

Clear the pinned cache at the start of the shared install body in all
three jobs that copy it, enforcing the per-run contract the pin
documents. The guard keeps the reset a no-op where COREPACK_HOME is
unset (test_build) or already run-unique (lint). The test step still
reuses the cache install warms in the same job, and poisoned hosts
self-heal on their next job without manual cleanup.

Pin the reset and the install/test COREPACK_HOME match in the workflow
meta-test, and bump the size baseline for the added lines.

* fix(ci): scope the corepack cache reset to the shard-owned path

Review found two over-deletion shapes in the nonempty guard: a runner
that exports COREPACK_HOME would lose its entire shared cache in the
build job, and hosted lint would lose the cache its pnpm-store step had
just warmed, re-exposing the install to a registry outage. Match only
the exact shard-owned path instead; the build and lint copies become
unconditional no-ops and the three install bodies stay byte-identical.

Add an execution test that runs the shipped guard fragment under a
scrubbed environment across five cache shapes: it kills the dead-guard
mutant the string pins survived, and a widened-guard mutant dies on the
shared-cache survival cell. State the incident as observations with the
corruption source undetermined, and correct the stale per-run rationale
beside the pin.
This commit is contained in:
Shaojin Wen authored and GitHub committed 2026-11-05 09:27:53 +08:00
1 parent f3488b47fa
commit ef9c891b8f
3 files changed
+193 -11

No files matched your search

+1 -1
View File
@@ -19,7 +19,7 @@
9551 build-and-publish-image.yml
55276 cd-cua-driver.yml
2222 cd-mobile-mcp.yml
192263 ci.yml
195906 ci.yml
4193 codeql.yml
9389 comment-attachment-guard.yml
2090 desktop-packaging-check.yml
+40 -5
View File
@@ -679,6 +679,17 @@ jobs:
( while sleep 10; do sample_disk; done ) &
SAMPLER_PID=$!
trap 'pkill -TERM -P "$SAMPLER_PID" 2>/dev/null || true; kill "$SAMPLER_PID" 2>/dev/null || true' EXIT
# Observed: the pinned cache entry existed with its pnpm binary
# missing, corepack trusts an existing entry without re-verifying
# its files, and the install died instantly with MODULE_NOT_FOUND
# (main run 41927887452). What left the entry incomplete is
# undetermined; _work/_temp survives across jobs on self-hosted
# runners, so such state persists. Start the shard-owned cache
# empty — and only that exact path, so a runner-exported shared
# cache and lint's already-warmed run-unique home stay untouched.
if [ "${COREPACK_HOME:-}" = "${RUNNER_TEMP:-}/qwen-ci-corepack" ]; then
rm -rf -- "${COREPACK_HOME}"
fi
corepack pnpm install --frozen-lockfile --prefer-offline --reporter=append-only
- name: 'Report npm cache usage (self-hosted)'
@@ -1078,11 +1089,13 @@ jobs:
env:
QWEN_SKIP_PREPARE: '1'
QWEN_SKIP_NOTICE_GENERATION: '1'
# Pin COREPACK_HOME to a per-run path so the corepack cache written
# here is visible to the test step below, which overrides HOME for
# isolation. corepack does not re-verify extracted entries on cache
# hit, so this must NOT be a shared/persistent path (poisoning
# channel) — per-run is safe and sufficient.
# Pin COREPACK_HOME to a fixed path under runner.temp so the
# corepack cache written here is visible to the test step below,
# which overrides HOME for isolation. runner.temp survives across
# jobs on self-hosted runners, and corepack does not re-verify
# extracted entries on cache hit, so the install body below clears
# this exact path before use; the test step then reuses the cache
# this step warms within the same job.
COREPACK_HOME: '${{ runner.temp }}/qwen-ci-corepack'
run: |-
DISK_SAMPLES="${RUNNER_TEMP}/disk-pressure-samples.log"
@@ -1097,6 +1110,17 @@ jobs:
( while sleep 10; do sample_disk; done ) &
SAMPLER_PID=$!
trap 'pkill -TERM -P "$SAMPLER_PID" 2>/dev/null || true; kill "$SAMPLER_PID" 2>/dev/null || true' EXIT
# Observed: the pinned cache entry existed with its pnpm binary
# missing, corepack trusts an existing entry without re-verifying
# its files, and the install died instantly with MODULE_NOT_FOUND
# (main run 41927887452). What left the entry incomplete is
# undetermined; _work/_temp survives across jobs on self-hosted
# runners, so such state persists. Start the shard-owned cache
# empty — and only that exact path, so a runner-exported shared
# cache and lint's already-warmed run-unique home stay untouched.
if [ "${COREPACK_HOME:-}" = "${RUNNER_TEMP:-}/qwen-ci-corepack" ]; then
rm -rf -- "${COREPACK_HOME}"
fi
corepack pnpm install --frozen-lockfile --prefer-offline --reporter=append-only
# Eight shards pull this same artifact at once, so the cap has to clear
@@ -1873,6 +1897,17 @@ jobs:
( while sleep 10; do sample_disk; done ) &
SAMPLER_PID=$!
trap 'pkill -TERM -P "$SAMPLER_PID" 2>/dev/null || true; kill "$SAMPLER_PID" 2>/dev/null || true' EXIT
# Observed: the pinned cache entry existed with its pnpm binary
# missing, corepack trusts an existing entry without re-verifying
# its files, and the install died instantly with MODULE_NOT_FOUND
# (main run 41927887452). What left the entry incomplete is
# undetermined; _work/_temp survives across jobs on self-hosted
# runners, so such state persists. Start the shard-owned cache
# empty — and only that exact path, so a runner-exported shared
# cache and lint's already-warmed run-unique home stay untouched.
if [ "${COREPACK_HOME:-}" = "${RUNNER_TEMP:-}/qwen-ci-corepack" ]; then
rm -rf -- "${COREPACK_HOME}"
fi
corepack pnpm install --frozen-lockfile --prefer-offline --reporter=append-only
- name: 'Report npm cache usage (self-hosted)'
+152 -5
View File
@@ -27,7 +27,14 @@
// place so that stays a revert.
import { execFileSync } from 'node:child_process';
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
import {
existsSync,
mkdirSync,
mkdtempSync,
readFileSync,
rmSync,
writeFileSync,
} from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { describe, expect, it } from 'vitest';
@@ -839,9 +846,10 @@ describe('post-merge push lane', () => {
QWEN_SKIP_NOTICE_GENERATION: '1',
// The shards' test step overrides HOME, which would hide the
// corepack cache install warms here; pin COREPACK_HOME to a
// per-run path so the warmup hits it. lint_and_static routes
// HOME and COREPACK_HOME together via GITHUB_ENV, so its
// install needs no step-level override.
// fixed path under runner.temp so the warmup hits it (the
// install body clears that exact path first). lint_and_static
// routes HOME and COREPACK_HOME together via GITHUB_ENV, so
// its install needs no step-level override.
COREPACK_HOME: '${{ runner.temp }}/qwen-ci-corepack',
});
} else {
@@ -897,9 +905,48 @@ describe('post-merge push lane', () => {
QWEN_SKIP_PREPARE: '1',
QWEN_SKIP_NOTICE_GENERATION: '1',
// The test step's HOME override would hide the corepack cache install
// warms here; pin COREPACK_HOME to a per-run path so the warmup hits it.
// warms here; pin COREPACK_HOME to a fixed path under runner.temp so
// the warmup hits it (the install body clears that exact path first).
COREPACK_HOME: '${{ runner.temp }}/qwen-ci-corepack',
});
// Observed on the fleet: the pinned cache entry existed with its pnpm
// binary missing, corepack trusts an existing entry without re-verifying
// its files, and Install dependencies died instantly with
// MODULE_NOT_FOUND (main runs 41876940141, 41889761157, 41927887452).
// What left the entry incomplete is undetermined; _work/_temp survives
// across jobs on self-hosted runners (the lint job's isolated-HOME step
// documents the same fact), so such state persists. Every copy of the
// shared install body must clear the cache before corepack runs, and the
// guard must match ONLY the shard-owned path: a runner-exported shared
// COREPACK_HOME (test_build) and lint's run-unique home — already warmed
// by the hosted pnpm-store step — must never be deletion targets. The
// execution test below holds the semantics; these pins hold the spelling.
for (const job of ['test_build', 'test', 'lint_and_static']) {
const body = String(install(job).run ?? '');
expect(
body,
`${job} must scope the reset to the shard-owned path`,
).toContain(
'if [ "${COREPACK_HOME:-}" = "${RUNNER_TEMP:-}/qwen-ci-corepack" ]; then',
);
const reset = body.indexOf('rm -rf -- "${COREPACK_HOME}"');
expect(
reset,
`${job} must reset the pinned corepack cache`,
).toBeGreaterThanOrEqual(0);
expect(
body.indexOf('corepack pnpm install'),
`${job} must reset the cache before corepack runs`,
).toBeGreaterThan(reset);
}
// The test step must keep sharing the cache install warms: a drifted
// COREPACK_HOME sends the vitest globalSetup warmup back to the cold
// npmjs download whose ETIMEDOUT flake #16328 removed.
expect(
ci.jobs.test.steps.find(
(s) => s.name === 'Run tests and generate reports',
).env.COREPACK_HOME,
).toBe(install('test').env.COREPACK_HOME);
const namesIn = (job) => (ci.jobs[job].steps ?? []).map((x) => x.name);
// Values alone are order-blind, and order is load-bearing here: `steps`
// exposes only steps that already ran, and a missing property reads as
@@ -1028,6 +1075,106 @@ describe('post-merge push lane', () => {
expect(fetchCode).toContain('Incremental lint history:');
});
it.skipIf(process.platform === 'win32')(
'executes the shared corepack reset only against the shard-owned path',
() => {
// Semantic witness for the reset guard. The string pins above hold its
// spelling, but a guard that never executes satisfies every one of
// them (the verify-pr round on 01d2b116 survived exactly that mutant),
// so run the shipped fragment — sliced from the real install body, not
// a copy — and watch what it deletes. The environment is built from
// scratch: this suite also runs inside CI shards that export their own
// COREPACK_HOME and RUNNER_TEMP, and inheriting those would point the
// cleanup at the running job's own directories.
const step = ci.jobs.test.steps.find(
(s) => s.name === 'Install dependencies',
);
const body = String(step.run ?? '');
const start = body.indexOf('if [ "${COREPACK_HOME:-}"');
const end = body.indexOf('corepack pnpm install');
expect(
start,
'reset guard is missing from the install body',
).toBeGreaterThanOrEqual(0);
expect(end, 'reset must sit before the corepack install').toBeGreaterThan(
start,
);
const fragment = body.slice(start, end);
const cells = [
{
name: 'shard-owned cache is cleared',
home: (temp) => join(temp, 'qwen-ci-corepack'),
cleared: true,
},
{
name: 'runner-exported shared cache survives',
home: (_temp, root) => join(root, 'shared-corepack'),
cleared: false,
},
{
name: 'lint run-unique warmed home survives',
home: (temp) =>
join(temp, 'qwen-lint-home-42-1', '.cache', 'node', 'corepack'),
cleared: false,
},
{
name: 'unset variable leaves the default cache alone',
home: null,
cleared: false,
},
{
name: 'empty variable is not a deletion target',
home: () => '',
cleared: false,
},
];
for (const cell of cells) {
const root = mkdtempSync(join(tmpdir(), 'corepack-reset-'));
try {
const temp = join(root, 'temp');
const home = join(root, 'home');
mkdirSync(temp, { recursive: true });
mkdirSync(home, { recursive: true });
// With no COREPACK_HOME the default cache is the deletion target
// a naive guard would reach; give every cell a marker to watch.
const target =
cell.home === null
? join(home, '.cache', 'node', 'corepack')
: cell.home(temp, root);
const marker = join(target, 'v1', 'pnpm', '11.24.0', '.corepack');
if (target) {
mkdirSync(join(target, 'v1', 'pnpm', '11.24.0'), {
recursive: true,
});
writeFileSync(marker, '{}');
}
const bystander = join(root, 'bystander');
mkdirSync(bystander, { recursive: true });
writeFileSync(join(bystander, 'sentinel'), 'keep');
const env = {
PATH: process.env.PATH,
HOME: home,
RUNNER_TEMP: temp,
};
if (cell.home !== null) env.COREPACK_HOME = target;
execFileSync(
'bash',
['--noprofile', '--norc', '-e', '-o', 'pipefail', '-c', fragment],
{ env, cwd: root },
);
if (cell.cleared) {
expect(existsSync(target), cell.name).toBe(false);
} else if (target) {
expect(existsSync(marker), cell.name).toBe(true);
}
expect(existsSync(join(bystander, 'sentinel')), cell.name).toBe(true);
} finally {
rmSync(root, { recursive: true, force: true });
}
}
},
);
it.each([
['first fetch aborts', 1, 99, 99, 1, 0, 'deepen fetch aborted'],
['third fetch aborts', 3, 99, 99, 3, 2, 'deepen fetch aborted'],