fix(ci): apply the apt-mirror probe to the release docker lane too (#16278)

* fix(ci): apply the apt-mirror probe to the release docker lane too

The release docker integration lane builds the same sandbox image as the E2E
leg but never selected an apt mirror, and it prunes the builder cache on every
run so its apt layers are always cold. On mainland pool hosts that rebuilt
against the throttled deb.debian.org path (11-45 kB/s) — the same defect #16214
fixed for E2E, left unfixed here because it was out of scope (#16271).

Extract the probe (apt_mirror_probe_rate / select_apt_mirror) and a new
apply_apt_mirror helper into a shared .github/scripts/apt-mirror-probe.sh,
sourced by both run-e2e-tests.sh and run-release-docker-integration.sh. Each
calls apply_apt_mirror before build:sandbox, so a mainland host builds against
the VPC intranet mirror (6-9 MB/s, verified for E2E) while a healthy default is
left untouched. The Dockerfile APT_MIRROR ARG already exists from #16218.

Contract tests read the shared probe and pin the candidate loop and the source
statement (not substrings the comments also satisfy); both callers must apply
the probe before the build. Mutation-verified: removing the release call, a
candidate, or a source line each reddens the suite.

* fix(ci): carry the apt-mirror probe into the shard-retry test rig

run-e2e-tests.sh now sources its sibling apt-mirror-probe.sh at startup.
The shard-retry meta-test copies only run-e2e-tests.sh into a temp dir, so
the source resolved to a missing file and 'set -e' aborted before any shard
ran — all eight cases saw zero npm calls. Copy the probe file alongside the
script in the rig so the source resolves; sourcing only defines functions
(no network at source time) and the sandbox:none path never calls
apply_apt_mirror, so the rig stays hermetic.

Witness: CI at 315733fe showed all 8 e2e-shard-retry cases failing
(npmCalls expected 2, got 0); with the probe carried into the rig all 8
pass locally. eslint exit 0, prettier clean.

* fix(ci): address #16278 review — hermetic probe test, shebang, scope header

R1-1: the release-lane exec harness now stubs curl (per-URL rates) so the suite
no longer probes the live network; the npm stub logs BUILD_SANDBOX_FLAGS and two
execution tests assert the fast mirror is passed as --build-arg APT_MIRROR with
the CI label preserved, and that an all-slow probe adds no mirror arg. Flipping
the mirror-selection guard reddens them.

R1-2: add the inert shebang so file(1) reports text/x-shellscript and the
shellcheck lane enrols the sourced probe; a lint test pins the mime type.

R1-3: narrow the header to the two covered build sites and record the third
cold caller (release-sdk.yml build-sdk) as out of scope, tracked in #16317.

* test(ci): drive the shellcheck lane for the probe; copy scripts dir in shard harness

R2-1: the shebang test now runs the lane's own candidate selection with only
shellcheck stubbed, so a filter change dropping the probe is caught (removing the
shebang reddens it). R2-2: the shard-retry harness copies the whole .github/scripts
dir so any sourced sibling comes along instead of a hand-maintained file list.

* merge(ci): sync origin/main to pick up the pinned COREPACK_HOME gate

The lint-gate-freshness lane went red on 9b3d131cdb because main moved
.github/workflows/ci.yml at 2b9c4a98f4, and the shard-4 Linux lane died on
the very corepack 'pnpm.mjs' miss that commit fixes.

Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmw2t0at7dy

* docs(ci): stop promising the release lane a benefit its routed host cannot get

`integration_docker` is pinned to `ecs-qwen-hk4-host`, or `ubuntu-latest` when
the ECS pool is unavailable. On the Hong Kong host `deb.debian.org` already
answers at MB/s and the VPC mirror is unreachable, so `select_apt_mirror`
correctly keeps the built-in source; on a hosted runner the same is true. The
call-site comment asserted a mainland benefit this call site cannot obtain, and
the always-cold builder cache it sits next to is unchanged.

The comment now says what the call does: apply the same probe the E2E leg uses,
so a host whose default is throttled builds against a faster endpoint, while a
host whose default is already fast keeps it. The `Fixes #16271` claim is
narrowed to `Refs #16271` in the PR body — the always-cold half stays open.

Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmw420o4bfo

* fix(ci): keep the release docker lane script executable

The previous delivery of 6218478b62 went through the Git Data API, which
recorded .github/scripts/run-release-docker-integration.sh as 100644 while
the base has it 100755. release.yml runs these by bare path, so the mode is
load-bearing, and scripts/tests/release-workflow.test.js
> keeps the extracted release scripts executable already fails on it
(Linux unit tests shard 3/8 on 6218478b62).

Restore the recorded mode to 100755.

Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmw48g5gafx

* test(ci): stage the probe fixture at its real path (#16278 R2-1)

The rewritten case kept its fixture at the temp-repo root, where the
lane's `^([^.]+)` candidate alternative already matches, so deleting the
`.*\.(sh|zsh|bash)` arm left it green. Put the file at
`.github/scripts/apt-mirror-probe.sh` — a leading dot makes the `.sh` arm
the only way it can be selected — teach `setup()` to create parent
directories for nested names, and assert the exact shellcheck argv.

Two-arm check on the lane's own run string: as authored, exit 0 with
`... --color=never .github/scripts/apt-mirror-probe.sh`; with the `.sh`
arm deleted, exit 1 ("matched no shell-script candidates").

Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmw4cqh0dg3

---------

Co-authored-by: Qwen-Coder <[email protected]>
Co-authored-by: yiliang114 <[email protected]>

scripts/tests/e2e-shard-retry.test.js

@@ -7,6 +7,7 @@
 import { execFileSync } from 'node:child_process';
 import {
   chmodSync,
+  cpSync,
   mkdtempSync,
   readFileSync,
   rmSync,
@@ -26,6 +27,10 @@ describe('e2e workflow sandbox:none shard retry execution', () => {
   function runStepScript({ failCalls, elapsedSeconds }) {
     const dir = mkdtempSync(join(tmpdir(), 'qwen-e2e-retry-'));
     try {
+      // Copy the whole scripts dir so any sibling the runner sources comes
+      // along, instead of hand-maintaining a per-file list (#16278 R2-2). The
+      // probe lands at dir/apt-mirror-probe.sh, where dirname BASH_SOURCE looks.
+      cpSync('.github/scripts', dir, { recursive: true });
       const callCountFile = join(dir, 'npm-call-count');
       const npmStub = join(dir, 'npm');
       writeFileSync(callCountFile, '0');

scripts/tests/e2e-workflow.test.js

@@ -1096,18 +1096,25 @@ describe('e2e workflow', () => {
 
     it('probes the apt mirror instead of pinning a region', () => {
       // #16214: deb.debian.org serves this pool at MB/s from Hong Kong and
-      // 11-20 kB/s from the mainland hosts, where the two apt layers then
-      // outlast the job's 60-minute cap. So neither a single mirror nor a
+      // 11-45 kB/s from the mainland hosts, where the two apt layers then
+      // outlast the job's execution cap. So neither a single mirror nor a
       // host-name map is correct — the build probes and overrides only on a
-      // clear win, which leaves healthy hosts on the built-in source.
-      // The VPC intranet mirror is the durable fix: it bypasses the capped
-      // public egress that throttles the public mirrors (the maven script's
-      // rationale). It must stay a probe candidate — unreachable off-VPC, it
-      // measures 0 and loses, so no host map is needed.
-      expect(e2eRunScript).toContain('mirrors.cloud.aliyuncs.com');
-      expect(e2eRunScript).toContain('mirrors.aliyun.com');
-      expect(e2eRunScript).toContain('mirrors.tuna.tsinghua.edu.cn');
-      expect(e2eRunScript).toContain('--build-arg APT_MIRROR=');
+      // clear win, which leaves healthy hosts on the built-in source. The
+      // probe lives in a shared script so the release docker lane (#16271)
+      // selects the same source; the VPC intranet mirror bypasses the capped
+      // public egress (the maven script's rationale) and, unreachable off-VPC,
+      // measures 0 and loses — no host map needed.
+      const aptProbeScript = readFileSync(
+        '.github/scripts/apt-mirror-probe.sh',
+        'utf8',
+      );
+      // Pin the candidate loop itself, not a substring: the mirrors are also
+      // named in the comments, so a toContain would survive removing a
+      // candidate from the actual `for` list.
+      expect(aptProbeScript).toMatch(
+        /for mirror in mirrors\.cloud\.aliyuncs\.com mirrors\.aliyun\.com mirrors\.tuna\.tsinghua\.edu\.cn; do/,
+      );
+      expect(aptProbeScript).toContain('--build-arg APT_MIRROR=');
       // Both stages need the ARG: a stage cannot inherit an ARG declared
       // before its own FROM, so one declaration per stage is the contract.
       // Counting (not toMatch) catches deleting either one — R1-1 witnessed
@@ -1116,16 +1123,15 @@ describe('e2e workflow', () => {
         readFileSync('Dockerfile', 'utf8').match(/^ARG APT_MIRROR=$/gm),
       ).toHaveLength(2);
 
-      const selectIndex = e2eRunScript.indexOf('select_apt_mirror()');
-      expect(selectIndex).toBeGreaterThan(-1);
-      const callIndex = e2eRunScript.indexOf(
-        'apt_mirror="$(select_apt_mirror)"',
-      );
-      expect(callIndex).toBeGreaterThan(selectIndex);
-      // The probe has to run before the build consumes the flag, otherwise the
-      // mirror is decided after the layer it was meant to speed up.
+      // The E2E leg sources the shared probe and applies it before the build
+      // consumes the flag, otherwise the mirror is decided after the layer it
+      // was meant to speed up. Match the source statement, not a substring —
+      // the shellcheck directive above it also names the file.
+      expect(e2eRunScript).toMatch(/^source .*apt-mirror-probe\.sh"$/m);
+      const applyIndex = e2eRunScript.indexOf('apply_apt_mirror');
+      expect(applyIndex).toBeGreaterThan(-1);
       expect(e2eRunScript.indexOf('if ! build_image;')).toBeGreaterThan(
-        callIndex,
+        applyIndex,
       );
       // Two gates hold the override back, both on the mirror side: an absolute
       // floor (no "slow vs slower" override) and a 4x factor (a genuinely fast
@@ -1133,16 +1139,16 @@ describe('e2e workflow', () => {
       // pinned, not just the names: a 10x mutation of the floor flips the
       // both-slow shape onto a still-slow mirror. \b, not toContain, so the
       // 10x-up mutation 'fast_mirror=10000000' cannot satisfy the pin.
-      expect(e2eRunScript).toMatch(/fast_mirror=1000000\b/);
-      expect(e2eRunScript).toMatch(/-gt "\$\{fast_mirror\}" \]/);
-      expect(e2eRunScript).toMatch(/-gt \$\(\(default_rate \* 4\)\)/);
+      expect(aptProbeScript).toMatch(/fast_mirror=1000000\b/);
+      expect(aptProbeScript).toMatch(/-gt "\$\{fast_mirror\}" \]/);
+      expect(aptProbeScript).toMatch(/-gt \$\(\(default_rate \* 4\)\)/);
       // There must be NO "default looks healthy" gate: the mainland paths are
       // bimodal, so a short probe of the default cannot veto the override
       // (#16214 recurrence on hz4-2, run 41743539611). Pin its absence.
-      expect(e2eRunScript).not.toContain('slow_default');
+      expect(aptProbeScript).not.toContain('slow_default');
       // The measurements have to reach the log, or the next slow build is
       // another manual investigation.
-      expect(e2eRunScript).toContain('apt probe:');
+      expect(aptProbeScript).toContain('apt probe:');
     });
 
     it('keeps one bounded retry without pruning the shared daemon', () => {

scripts/tests/lint.test.js

@@ -479,7 +479,9 @@ describe('git-sourced lint lanes', () => {
       repo = path.join(root, 'repo');
       mkdirSync(repo);
       for (const [name, content] of Object.entries(files)) {
-        writeFileSync(path.join(repo, name), content);
+        const file = path.join(repo, name);
+        mkdirSync(path.dirname(file), { recursive: true });
+        writeFileSync(file, content);
       }
       execSync('git init -q && git add -A', { cwd: repo });
     }
@@ -702,6 +704,41 @@ describe('git-sourced lint lanes', () => {
     }
   });
 
+  it.skipIf(process.platform === 'win32')(
+    'enrols the sourced apt-mirror probe in the shellcheck lane via its shebang',
+    async (ctx) => {
+      // Drive the lane's own candidate selection (name regex + real file(1)
+      // mime filter) with only shellcheck stubbed, so a change to the lane's
+      // filter that drops the probe is caught here (#16278 R2-1). The fixture
+      // sits at the probe's real dot-directory path: `^([^.]+)` cannot match a
+      // leading dot, so only the `.*\.(sh|zsh|bash)` arm can select it —
+      // at the repo root that arm could be deleted without reddening this
+      // case. Removing the probe's shebang still makes file(1) report
+      // text/plain and reddens it too.
+      const linters = await getLintersOrSkip(ctx);
+      const probeSrc = readFileSync(
+        '.github/scripts/apt-mirror-probe.sh',
+        'utf8',
+      );
+      const probePath = '.github/scripts/apt-mirror-probe.sh';
+      const { root, repo, bin } = setup({ [probePath]: probeSrc });
+      try {
+        const shellcheckLog = path.join(root, 'shellcheck.log');
+        stub(bin, 'shellcheck', `echo "$@" >> '${shellcheckLog}'`);
+        const result = runLane(linters.shellcheck.run, { repo, bin });
+        expect(result.status).toBe(0);
+        // Exact argv: pins the lane options and the awk colon-strip, and
+        // proves this file (not a root-level namesake) reached shellcheck.
+        expect(readFileSync(shellcheckLog, 'utf8').trim()).toBe(
+          '--check-sourced --enable=all --exclude=SC2002,SC2129,SC2310' +
+            ` --severity=style --format=gcc --color=never ${probePath}`,
+        );
+      } finally {
+        rmSync(root, { recursive: true, force: true });
+      }
+    },
+  );
+
   it('appends the pip --user bin dir after the inherited PATH', async () => {
     const { getLinterPath } = await import('../lint.js');
     const env = { HOME: '/home/runner', PATH: '/usr/bin:/bin' };

scripts/tests/release-workflow.test.js

@@ -2516,6 +2516,23 @@ describe('release workflow', () => {
     );
   });
 
+  it('selects the apt mirror before the always-cold sandbox build', () => {
+    // #16271: this lane prunes the builder cache, so its apt layers are always
+    // cold — the same defect #16214 fixed for the E2E leg. It must source the
+    // shared probe and apply it before `build:sandbox`, so a mainland host
+    // builds against the VPC intranet mirror rather than the throttled default.
+    // Match the source statement, not a substring — the shellcheck directive
+    // above it also names the file.
+    expect(dockerIntegrationScript).toMatch(
+      /^source .*apt-mirror-probe\.sh"$/m,
+    );
+    const applyIndex = dockerIntegrationScript.indexOf('apply_apt_mirror');
+    expect(applyIndex).toBeGreaterThan(-1);
+    expect(
+      dockerIntegrationScript.indexOf('npm run build:sandbox'),
+    ).toBeGreaterThan(applyIndex);
+  });
+
   it('prunes BuildKit cache and gates the docker data root before the build and the vitest phase', () => {
     // #13479: run 37374675168 passed the job-start disk floor gate and the
     // runner still died on ENOSPC 24 minutes into the docker test step. The
@@ -2591,6 +2608,9 @@ describe('release workflow', () => {
     dockerPostFloorKb,
     postGateExit,
     inspectTimesOut = false,
+    aptDefaultRate,
+    aptAliyunRate,
+    aptTunaRate,
   } = {}) => {
     const directory = mkdtempSync(
       join(tmpdir(), 'release-docker-integration-'),
@@ -2647,8 +2667,25 @@ if [ "$1" = 'info' ]; then
 fi
 exit 0`,
     );
-    stub('npm', `printf 'npm %s\\n' "$*" >> "$CALLS_LOG"`);
+    stub(
+      'npm',
+      `printf 'npm %s flags=%s\\n' "$*" "\${BUILD_SANDBOX_FLAGS:-}" >> "$CALLS_LOG"`,
+    );
     stub('npx', `printf 'npx %s\\n' "$*" >> "$CALLS_LOG"`);
+    // Deterministic stand-in for the probe's range reads so the suite stays
+    // hermetic (#16278 R1-1): answer the -w '%{http_code} %{speed_download}'
+    // projection per URL, default slow + one fast mirror.
+    stub(
+      'curl',
+      `for url; do :; done
+case "$url" in
+  *deb.debian.org*) printf '206 %s' "\${CURL_DEFAULT_RATE:-10000}" ;;
+  *mirrors.aliyun.com*) printf '206 %s' "\${CURL_ALIYUN_RATE:-5000000}" ;;
+  *mirrors.tuna.tsinghua.edu.cn*) printf '206 %s' "\${CURL_TUNA_RATE:-0}" ;;
+  *mirrors.cloud.aliyuncs.com*) printf '000 0' ;;
+  *) printf '000 0' ;;
+esac`,
+    );
     if (!helperAbsent) {
       // Both checkpoints share this stub: count invocations so a case can
       // fail the pre-vitest gate while the pre-build gate passes.
@@ -2683,6 +2720,10 @@ exit 0`,
         PRUNE_FAILS: pruneFails ? '1' : '0',
         IMAGE_PRUNE_FAILS: imagePruneFails ? '1' : '0',
         DOCKER_INFO_FAILS: dockerInfoFails ? '1' : '0',
+        BUILD_SANDBOX_FLAGS: '--label org.qwen-code.ci.sandbox=true',
+        CURL_DEFAULT_RATE: String(aptDefaultRate ?? 10000),
+        CURL_ALIYUN_RATE: String(aptAliyunRate ?? 5000000),
+        CURL_TUNA_RATE: String(aptTunaRate ?? 0),
       };
       // The floor knobs come from the test, never the ambient environment:
       // the suite's verdict must not depend on the env of whoever runs it.
@@ -2721,6 +2762,30 @@ exit 0`,
     }
   };
 
+  it.skipIf(process.platform === 'win32')(
+    'selects the fast apt mirror and preserves the CI label in the build flags',
+    () => {
+      const { result, calls } = runDockerIntegrationScript();
+      expect(result.status).toBe(0);
+      expect(calls).toContain('--build-arg APT_MIRROR=mirrors.aliyun.com');
+      expect(calls).toContain('--label org.qwen-code.ci.sandbox=true');
+    },
+  );
+
+  it.skipIf(process.platform === 'win32')(
+    'adds no apt mirror build arg when every endpoint is slow',
+    () => {
+      const { result, calls } = runDockerIntegrationScript({
+        aptDefaultRate: 10000,
+        aptAliyunRate: 20000,
+        aptTunaRate: 15000,
+      });
+      expect(result.status).toBe(0);
+      expect(calls).not.toContain('--build-arg APT_MIRROR=');
+      expect(calls).toContain('--label org.qwen-code.ci.sandbox=true');
+    },
+  );
+
   it.skipIf(process.platform === 'win32')(
     'reclaims without the disk gate or the lock protocol on hosted runners',
     () => {