From 45f71b371465850592d24217c8258d2113f14d84 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=98=93=E8=89=AF?= <1204183885@qq.com> Date: Fri, 6 Nov 2026 15:42:59 +0800 Subject: [PATCH] fix(ci): apply the apt-mirror probe to the release docker lane too (#16278) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(ci): apply the apt-mirror probe to the release docker lane too The release docker integration lane builds the same sandbox image as the E2E leg but never selected an apt mirror, and it prunes the builder cache on every run so its apt layers are always cold. On mainland pool hosts that rebuilt against the throttled deb.debian.org path (11-45 kB/s) — the same defect #16214 fixed for E2E, left unfixed here because it was out of scope (#16271). Extract the probe (apt_mirror_probe_rate / select_apt_mirror) and a new apply_apt_mirror helper into a shared .github/scripts/apt-mirror-probe.sh, sourced by both run-e2e-tests.sh and run-release-docker-integration.sh. Each calls apply_apt_mirror before build:sandbox, so a mainland host builds against the VPC intranet mirror (6-9 MB/s, verified for E2E) while a healthy default is left untouched. The Dockerfile APT_MIRROR ARG already exists from #16218. Contract tests read the shared probe and pin the candidate loop and the source statement (not substrings the comments also satisfy); both callers must apply the probe before the build. Mutation-verified: removing the release call, a candidate, or a source line each reddens the suite. * fix(ci): carry the apt-mirror probe into the shard-retry test rig run-e2e-tests.sh now sources its sibling apt-mirror-probe.sh at startup. The shard-retry meta-test copies only run-e2e-tests.sh into a temp dir, so the source resolved to a missing file and 'set -e' aborted before any shard ran — all eight cases saw zero npm calls. Copy the probe file alongside the script in the rig so the source resolves; sourcing only defines functions (no network at source time) and the sandbox:none path never calls apply_apt_mirror, so the rig stays hermetic. Witness: CI at 315733fe showed all 8 e2e-shard-retry cases failing (npmCalls expected 2, got 0); with the probe carried into the rig all 8 pass locally. eslint exit 0, prettier clean. * fix(ci): address #16278 review — hermetic probe test, shebang, scope header R1-1: the release-lane exec harness now stubs curl (per-URL rates) so the suite no longer probes the live network; the npm stub logs BUILD_SANDBOX_FLAGS and two execution tests assert the fast mirror is passed as --build-arg APT_MIRROR with the CI label preserved, and that an all-slow probe adds no mirror arg. Flipping the mirror-selection guard reddens them. R1-2: add the inert shebang so file(1) reports text/x-shellscript and the shellcheck lane enrols the sourced probe; a lint test pins the mime type. R1-3: narrow the header to the two covered build sites and record the third cold caller (release-sdk.yml build-sdk) as out of scope, tracked in #16317. * test(ci): drive the shellcheck lane for the probe; copy scripts dir in shard harness R2-1: the shebang test now runs the lane's own candidate selection with only shellcheck stubbed, so a filter change dropping the probe is caught (removing the shebang reddens it). R2-2: the shard-retry harness copies the whole .github/scripts dir so any sourced sibling comes along instead of a hand-maintained file list. * merge(ci): sync origin/main to pick up the pinned COREPACK_HOME gate The lint-gate-freshness lane went red on 9b3d131cdb because main moved .github/workflows/ci.yml at 2b9c4a98f4, and the shard-4 Linux lane died on the very corepack 'pnpm.mjs' miss that commit fixes. Co-authored-by: Qwen-Coder Patrol-Run: qwen-pr-closeout/jmw2t0at7dy * docs(ci): stop promising the release lane a benefit its routed host cannot get `integration_docker` is pinned to `ecs-qwen-hk4-host`, or `ubuntu-latest` when the ECS pool is unavailable. On the Hong Kong host `deb.debian.org` already answers at MB/s and the VPC mirror is unreachable, so `select_apt_mirror` correctly keeps the built-in source; on a hosted runner the same is true. The call-site comment asserted a mainland benefit this call site cannot obtain, and the always-cold builder cache it sits next to is unchanged. The comment now says what the call does: apply the same probe the E2E leg uses, so a host whose default is throttled builds against a faster endpoint, while a host whose default is already fast keeps it. The `Fixes #16271` claim is narrowed to `Refs #16271` in the PR body — the always-cold half stays open. Co-authored-by: Qwen-Coder Patrol-Run: qwen-pr-closeout/jmw420o4bfo * fix(ci): keep the release docker lane script executable The previous delivery of 6218478b62 went through the Git Data API, which recorded .github/scripts/run-release-docker-integration.sh as 100644 while the base has it 100755. release.yml runs these by bare path, so the mode is load-bearing, and scripts/tests/release-workflow.test.js > keeps the extracted release scripts executable already fails on it (Linux unit tests shard 3/8 on 6218478b62). Restore the recorded mode to 100755. Co-authored-by: Qwen-Coder Patrol-Run: qwen-pr-closeout/jmw48g5gafx * test(ci): stage the probe fixture at its real path (#16278 R2-1) The rewritten case kept its fixture at the temp-repo root, where the lane's `^([^.]+)` candidate alternative already matches, so deleting the `.*\.(sh|zsh|bash)` arm left it green. Put the file at `.github/scripts/apt-mirror-probe.sh` — a leading dot makes the `.sh` arm the only way it can be selected — teach `setup()` to create parent directories for nested names, and assert the exact shellcheck argv. Two-arm check on the lane's own run string: as authored, exit 0 with `... --color=never .github/scripts/apt-mirror-probe.sh`; with the `.sh` arm deleted, exit 1 ("matched no shell-script candidates"). Co-authored-by: Qwen-Coder Patrol-Run: qwen-pr-closeout/jmw4cqh0dg3 --------- Co-authored-by: Qwen-Coder Co-authored-by: yiliang114 --- .github/scripts/apt-mirror-probe.sh | 90 +++++++++++++++++++ .github/scripts/run-e2e-tests.sh | 76 +--------------- .../scripts/run-release-docker-integration.sh | 10 +++ scripts/tests/e2e-shard-retry.test.js | 5 ++ scripts/tests/e2e-workflow.test.js | 56 ++++++------ scripts/tests/lint.test.js | 39 +++++++- scripts/tests/release-workflow.test.js | 67 +++++++++++++- 7 files changed, 244 insertions(+), 99 deletions(-) create mode 100644 .github/scripts/apt-mirror-probe.sh diff --git a/.github/scripts/apt-mirror-probe.sh b/.github/scripts/apt-mirror-probe.sh new file mode 100644 index 00000000000..2d5687b2dcc --- /dev/null +++ b/.github/scripts/apt-mirror-probe.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Sourced by run-e2e-tests.sh and run-release-docker-integration.sh — the two +# sandbox-image build sites this helper covers. A third cold `build:sandbox` +# caller (release-sdk.yml build-sdk) is deliberately not wired here yet; it is +# tracked in #16317. Defines apt_mirror_probe_rate / select_apt_mirror / +# apply_apt_mirror; executes nothing on its own. The shebang is inert for a +# sourced file but makes file(1) report text/x-shellscript so the shellcheck +# lane enrols it (#16278 R1-2). +# +# The Dockerfile's built-in apt source is deb.debian.org. On the pool's +# mainland-China hosts that CDN path has been measured at 11-45 kB/s, which +# turns its two apt layers into a 2-3 hour build against the job's execution +# cap, so the leg is cancelled before any test runs (#16214). The same path +# runs at MB/s from the Hong Kong hosts, and it recovers on the mainland hosts +# at off-peak hours — so pinning either a mirror or a host list is wrong. +# Probe every build instead: the fastest endpoint wins, a healthy default is +# left alone, and no fleet map needs upkeep. + +apt_mirror_probe_rate() { + # A 512 KiB range read of a file every Debian mirror carries. Any large + # immutable index would do. The window matters more than the target: a + # shorter read cannot tell a warm CDN edge from a throttled one, while this + # one still finishes in well under a second wherever the path is healthy, + # so a slow candidate costs at most the 4-second cap. + local reply code rate + reply="$(curl -s -o /dev/null --max-time 4 --range 0-524287 \ + -w '%{http_code} %{speed_download}' \ + "http://$1/debian/dists/bookworm/main/binary-arm64/Packages.gz" 2>/dev/null)" || true + code="${reply%% *}" + rate="${reply##* }" + case "${code}" in + 200 | 206) printf '%s' "${rate%%.*}" ;; + *) printf '0' ;; + esac +} + +select_apt_mirror() { + command -v curl > /dev/null 2>&1 || return 0 + # Two gates, both on the mirror side. An absolute floor stops a "slow vs + # slower" override (two crawling endpoints must not move a host onto a + # mirror it never needed); the 4x factor keeps a genuinely fast default — + # the Hong Kong hosts — on the built-in source, since a mirror there is + # rarely 4x quicker. There is deliberately no "default looks healthy" gate: + # the mainland paths are bimodal, a 512 KiB probe can catch a fast burst + # (372 kB/s on hz4-2) while the sustained 200 MB transfer then throttles to + # 45 kB/s and times out (#16214 recurrence, run 41743539611). Trusting the + # mirror's own measured speed, not the default's, is what survives that. + local fast_mirror=1000000 + local default_rate best_rate=0 best='' mirror rate + default_rate="$(apt_mirror_probe_rate deb.debian.org)" + default_rate="${default_rate:-0}" + # mirrors.cloud.aliyuncs.com is the Alibaba Cloud VPC intranet mirror: it + # answers from inside the VPC and does not consume the instance's capped + # 100 Mbps public egress that throttles the public mirrors under concurrent + # load — the same reason configure-maven-mirror.sh prefers it for Central. + # Hosted runners and the Hong Kong hosts cannot reach it, so the probe just + # measures 0 there and it loses to a reachable public mirror; no host map is + # needed. Listed first because where it answers it is the fastest path. + for mirror in mirrors.cloud.aliyuncs.com mirrors.aliyun.com mirrors.tuna.tsinghua.edu.cn; do + rate="$(apt_mirror_probe_rate "${mirror}")" + rate="${rate:-0}" + if [ "${rate}" -gt "${best_rate}" ]; then + best="${mirror}" + best_rate="${rate}" + fi + done + # Print the measurements: the next person debugging a slow image build + # should not have to re-run this probe by hand (#16214). + echo "apt probe: deb.debian.org=${default_rate} B/s best=${best_rate} B/s (${best:-none})" >&2 + if [ -n "${best}" ] && [ "${best_rate}" -gt "${fast_mirror}" ] && + [ "${best_rate}" -gt $((default_rate * 4)) ]; then + printf '%s' "${best}" + fi +} + +# Applies the probe to the sandbox build: exports BUILD_SANDBOX_FLAGS with the +# selected APT_MIRROR build arg (preserving any flags already set, e.g. the CI +# sandbox label) and logs the decision. A no-op selection leaves the flags +# untouched so the Dockerfile keeps its built-in deb.debian.org source. +apply_apt_mirror() { + local apt_mirror + apt_mirror="$(select_apt_mirror)" + if [ -n "${apt_mirror}" ]; then + export BUILD_SANDBOX_FLAGS="${BUILD_SANDBOX_FLAGS:-} --build-arg APT_MIRROR=${apt_mirror}" + echo "apt mirror: ${apt_mirror} (probe measured deb.debian.org as slower)" + else + echo "apt mirror: deb.debian.org (probe found no faster mirror)" + fi +} diff --git a/.github/scripts/run-e2e-tests.sh b/.github/scripts/run-e2e-tests.sh index 4107f2e814f..40e6d3cb3a9 100644 --- a/.github/scripts/run-e2e-tests.sh +++ b/.github/scripts/run-e2e-tests.sh @@ -2,6 +2,9 @@ set -euo pipefail +# shellcheck source=.github/scripts/apt-mirror-probe.sh +source "$(dirname "${BASH_SOURCE[0]}")/apt-mirror-probe.sh" + sandbox="${1:?usage: run-e2e-tests.sh }" shard="${2:?usage: run-e2e-tests.sh }" @@ -31,71 +34,6 @@ if [ "$sandbox" = 'sandbox:docker' ]; then npm run build:sandbox -- -s --no-prune -i "$sandbox_image" 7>&- 8>&- 9>&- } - # The Dockerfile's built-in apt source is deb.debian.org. On the pool's - # mainland-China hosts that CDN path has been measured at 11-20 kB/s, which - # turns its two apt layers into a 2-3 hour build against this job's 60-minute - # cap, so the leg is cancelled before any test runs (#16214). The same path - # runs at MB/s from the Hong Kong hosts, and it recovers on the mainland hosts - # at off-peak hours — so pinning either a mirror or a host list is wrong. - # Probe every build instead: the fastest endpoint wins, a healthy default is - # left alone, and no fleet map needs upkeep. - apt_mirror_probe_rate() { - # A 512 KiB range read of a file every Debian mirror carries. Any large - # immutable index would do. The window matters more than the target: a - # shorter read cannot tell a warm CDN edge from a throttled one, while this - # one still finishes in well under a second wherever the path is healthy, - # so a slow candidate costs at most the 4-second cap. - local reply code rate - reply="$(curl -s -o /dev/null --max-time 4 --range 0-524287 \ - -w '%{http_code} %{speed_download}' \ - "http://$1/debian/dists/bookworm/main/binary-arm64/Packages.gz" 2>/dev/null)" || true - code="${reply%% *}" - rate="${reply##* }" - case "${code}" in - 200 | 206) printf '%s' "${rate%%.*}" ;; - *) printf '0' ;; - esac - } - - select_apt_mirror() { - command -v curl > /dev/null 2>&1 || return 0 - # Two gates, both on the mirror side. An absolute floor stops a "slow vs - # slower" override (two crawling endpoints must not move a host onto a - # mirror it never needed); the 4x factor keeps a genuinely fast default — - # the Hong Kong hosts — on the built-in source, since a mirror there is - # rarely 4x quicker. There is deliberately no "default looks healthy" gate: - # the mainland paths are bimodal, a 512 KiB probe can catch a fast burst - # (372 kB/s on hz4-2) while the sustained 200 MB transfer then throttles to - # 45 kB/s and times out (#16214 recurrence, run 41743539611). Trusting the - # mirror's own measured speed, not the default's, is what survives that. - local fast_mirror=1000000 - local default_rate best_rate=0 best='' mirror rate - default_rate="$(apt_mirror_probe_rate deb.debian.org)" - default_rate="${default_rate:-0}" - # mirrors.cloud.aliyuncs.com is the Alibaba Cloud VPC intranet mirror: it - # answers from inside the VPC and does not consume the instance's capped - # 100 Mbps public egress that throttles the public mirrors under concurrent - # load — the same reason configure-maven-mirror.sh prefers it for Central. - # Hosted runners and the Hong Kong hosts cannot reach it, so the probe just - # measures 0 there and it loses to a reachable public mirror; no host map is - # needed. Listed first because where it answers it is the fastest path. - for mirror in mirrors.cloud.aliyuncs.com mirrors.aliyun.com mirrors.tuna.tsinghua.edu.cn; do - rate="$(apt_mirror_probe_rate "${mirror}")" - rate="${rate:-0}" - if [ "${rate}" -gt "${best_rate}" ]; then - best="${mirror}" - best_rate="${rate}" - fi - done - # Print the measurements: the next person debugging a slow image build - # should not have to re-run this probe by hand (#16214). - echo "apt probe: deb.debian.org=${default_rate} B/s best=${best_rate} B/s (${best:-none})" >&2 - if [ -n "${best}" ] && [ "${best_rate}" -gt "${fast_mirror}" ] && - [ "${best_rate}" -gt $((default_rate * 4)) ]; then - printf '%s' "${best}" - fi - } - if [ "$RUNNER_ENVIRONMENT" = 'self-hosted' ]; then mkdir -p "${HOME}/.cache/qwen-code-ci" # Host daemon lock, shared for the whole step and never @@ -136,13 +74,7 @@ if [ "$sandbox" = 'sandbox:docker' ]; then # Label- and age-filtered, so it cannot touch the image a # concurrent shard is testing. timeout 20m docker image prune --all --force --filter 'label=org.qwen-code.ci.sandbox=true' --filter 'until=24h' || echo "::warning::old CI sandbox image cleanup failed on ${RUNNER_NAME:-this runner}" - apt_mirror="$(select_apt_mirror)" - if [ -n "${apt_mirror}" ]; then - export BUILD_SANDBOX_FLAGS="${BUILD_SANDBOX_FLAGS:-} --build-arg APT_MIRROR=${apt_mirror}" - echo "apt mirror: ${apt_mirror} (probe measured deb.debian.org as slower)" - else - echo "apt mirror: deb.debian.org (probe found no faster mirror)" - fi + apply_apt_mirror if ! build_image; then echo "::warning::sandbox image build failed; retrying once" build_image diff --git a/.github/scripts/run-release-docker-integration.sh b/.github/scripts/run-release-docker-integration.sh index 024c046076b..bca96195d06 100755 --- a/.github/scripts/run-release-docker-integration.sh +++ b/.github/scripts/run-release-docker-integration.sh @@ -1,6 +1,9 @@ #!/usr/bin/env bash set -euo pipefail +# shellcheck source=.github/scripts/apt-mirror-probe.sh +source "$(dirname "${BASH_SOURCE[0]}")/apt-mirror-probe.sh" + cleanup_release_containers() { container_ids="$(timeout 30 docker ps -aq --filter "label=org.qwen-code.ci.owner=${RELEASE_CONTAINER_OWNER}" 2>/dev/null)" || container_ids='' if [ -n "$container_ids" ]; then @@ -139,6 +142,13 @@ if [ "$image_inspect_status" -ne 0 ]; then # bounding that side too is a cross-lane change tracked separately. timeout 20m docker image prune --all --force --filter 'label=org.qwen-code.ci.sandbox=true' --filter 'until=24h' || echo "::warning::old CI sandbox image cleanup failed on ${RUNNER_NAME:-this runner}" check_docker_data_root_floor "${DISK_FLOOR_DOCKER_MIN_FREE_KB:-8388608}" + # This lane prunes the builder cache above, so its apt layers stay cold, and + # its routed host (`ecs-qwen-hk4-host`, or `ubuntu-latest` when the ECS pool + # is unavailable) already reaches `deb.debian.org` at MB/s — there the probe + # keeps the built-in source, as designed. It applies the probe the E2E leg + # uses (#16214) so a host whose default is throttled builds against a faster + # endpoint; the always-cold builder cache is unchanged here (#16271). + apply_apt_mirror # See e2e.yml: closing the lock descriptors in the child keeps a descendant # that outlives this job from holding the lock. npm run build:sandbox -- -s --no-prune -i "$sandbox_image" 7>&- 8>&- 9>&- diff --git a/scripts/tests/e2e-shard-retry.test.js b/scripts/tests/e2e-shard-retry.test.js index 342e9d882b8..f90188c8224 100644 --- a/scripts/tests/e2e-shard-retry.test.js +++ b/scripts/tests/e2e-shard-retry.test.js @@ -7,6 +7,7 @@ import { execFileSync } from 'node:child_process'; import { chmodSync, + cpSync, mkdtempSync, readFileSync, rmSync, @@ -26,6 +27,10 @@ describe('e2e workflow sandbox:none shard retry execution', () => { function runStepScript({ failCalls, elapsedSeconds }) { const dir = mkdtempSync(join(tmpdir(), 'qwen-e2e-retry-')); try { + // Copy the whole scripts dir so any sibling the runner sources comes + // along, instead of hand-maintaining a per-file list (#16278 R2-2). The + // probe lands at dir/apt-mirror-probe.sh, where dirname BASH_SOURCE looks. + cpSync('.github/scripts', dir, { recursive: true }); const callCountFile = join(dir, 'npm-call-count'); const npmStub = join(dir, 'npm'); writeFileSync(callCountFile, '0'); diff --git a/scripts/tests/e2e-workflow.test.js b/scripts/tests/e2e-workflow.test.js index e830a5d9a81..7059157b9bf 100644 --- a/scripts/tests/e2e-workflow.test.js +++ b/scripts/tests/e2e-workflow.test.js @@ -1096,18 +1096,25 @@ describe('e2e workflow', () => { it('probes the apt mirror instead of pinning a region', () => { // #16214: deb.debian.org serves this pool at MB/s from Hong Kong and - // 11-20 kB/s from the mainland hosts, where the two apt layers then - // outlast the job's 60-minute cap. So neither a single mirror nor a + // 11-45 kB/s from the mainland hosts, where the two apt layers then + // outlast the job's execution cap. So neither a single mirror nor a // host-name map is correct — the build probes and overrides only on a - // clear win, which leaves healthy hosts on the built-in source. - // The VPC intranet mirror is the durable fix: it bypasses the capped - // public egress that throttles the public mirrors (the maven script's - // rationale). It must stay a probe candidate — unreachable off-VPC, it - // measures 0 and loses, so no host map is needed. - expect(e2eRunScript).toContain('mirrors.cloud.aliyuncs.com'); - expect(e2eRunScript).toContain('mirrors.aliyun.com'); - expect(e2eRunScript).toContain('mirrors.tuna.tsinghua.edu.cn'); - expect(e2eRunScript).toContain('--build-arg APT_MIRROR='); + // clear win, which leaves healthy hosts on the built-in source. The + // probe lives in a shared script so the release docker lane (#16271) + // selects the same source; the VPC intranet mirror bypasses the capped + // public egress (the maven script's rationale) and, unreachable off-VPC, + // measures 0 and loses — no host map needed. + const aptProbeScript = readFileSync( + '.github/scripts/apt-mirror-probe.sh', + 'utf8', + ); + // Pin the candidate loop itself, not a substring: the mirrors are also + // named in the comments, so a toContain would survive removing a + // candidate from the actual `for` list. + expect(aptProbeScript).toMatch( + /for mirror in mirrors\.cloud\.aliyuncs\.com mirrors\.aliyun\.com mirrors\.tuna\.tsinghua\.edu\.cn; do/, + ); + expect(aptProbeScript).toContain('--build-arg APT_MIRROR='); // Both stages need the ARG: a stage cannot inherit an ARG declared // before its own FROM, so one declaration per stage is the contract. // Counting (not toMatch) catches deleting either one — R1-1 witnessed @@ -1116,16 +1123,15 @@ describe('e2e workflow', () => { readFileSync('Dockerfile', 'utf8').match(/^ARG APT_MIRROR=$/gm), ).toHaveLength(2); - const selectIndex = e2eRunScript.indexOf('select_apt_mirror()'); - expect(selectIndex).toBeGreaterThan(-1); - const callIndex = e2eRunScript.indexOf( - 'apt_mirror="$(select_apt_mirror)"', - ); - expect(callIndex).toBeGreaterThan(selectIndex); - // The probe has to run before the build consumes the flag, otherwise the - // mirror is decided after the layer it was meant to speed up. + // The E2E leg sources the shared probe and applies it before the build + // consumes the flag, otherwise the mirror is decided after the layer it + // was meant to speed up. Match the source statement, not a substring — + // the shellcheck directive above it also names the file. + expect(e2eRunScript).toMatch(/^source .*apt-mirror-probe\.sh"$/m); + const applyIndex = e2eRunScript.indexOf('apply_apt_mirror'); + expect(applyIndex).toBeGreaterThan(-1); expect(e2eRunScript.indexOf('if ! build_image;')).toBeGreaterThan( - callIndex, + applyIndex, ); // Two gates hold the override back, both on the mirror side: an absolute // floor (no "slow vs slower" override) and a 4x factor (a genuinely fast @@ -1133,16 +1139,16 @@ describe('e2e workflow', () => { // pinned, not just the names: a 10x mutation of the floor flips the // both-slow shape onto a still-slow mirror. \b, not toContain, so the // 10x-up mutation 'fast_mirror=10000000' cannot satisfy the pin. - expect(e2eRunScript).toMatch(/fast_mirror=1000000\b/); - expect(e2eRunScript).toMatch(/-gt "\$\{fast_mirror\}" \]/); - expect(e2eRunScript).toMatch(/-gt \$\(\(default_rate \* 4\)\)/); + expect(aptProbeScript).toMatch(/fast_mirror=1000000\b/); + expect(aptProbeScript).toMatch(/-gt "\$\{fast_mirror\}" \]/); + expect(aptProbeScript).toMatch(/-gt \$\(\(default_rate \* 4\)\)/); // There must be NO "default looks healthy" gate: the mainland paths are // bimodal, so a short probe of the default cannot veto the override // (#16214 recurrence on hz4-2, run 41743539611). Pin its absence. - expect(e2eRunScript).not.toContain('slow_default'); + expect(aptProbeScript).not.toContain('slow_default'); // The measurements have to reach the log, or the next slow build is // another manual investigation. - expect(e2eRunScript).toContain('apt probe:'); + expect(aptProbeScript).toContain('apt probe:'); }); it('keeps one bounded retry without pruning the shared daemon', () => { diff --git a/scripts/tests/lint.test.js b/scripts/tests/lint.test.js index f458173d1dc..b18ad2f8f3d 100644 --- a/scripts/tests/lint.test.js +++ b/scripts/tests/lint.test.js @@ -479,7 +479,9 @@ describe('git-sourced lint lanes', () => { repo = path.join(root, 'repo'); mkdirSync(repo); for (const [name, content] of Object.entries(files)) { - writeFileSync(path.join(repo, name), content); + const file = path.join(repo, name); + mkdirSync(path.dirname(file), { recursive: true }); + writeFileSync(file, content); } execSync('git init -q && git add -A', { cwd: repo }); } @@ -702,6 +704,41 @@ describe('git-sourced lint lanes', () => { } }); + it.skipIf(process.platform === 'win32')( + 'enrols the sourced apt-mirror probe in the shellcheck lane via its shebang', + async (ctx) => { + // Drive the lane's own candidate selection (name regex + real file(1) + // mime filter) with only shellcheck stubbed, so a change to the lane's + // filter that drops the probe is caught here (#16278 R2-1). The fixture + // sits at the probe's real dot-directory path: `^([^.]+)` cannot match a + // leading dot, so only the `.*\.(sh|zsh|bash)` arm can select it — + // at the repo root that arm could be deleted without reddening this + // case. Removing the probe's shebang still makes file(1) report + // text/plain and reddens it too. + const linters = await getLintersOrSkip(ctx); + const probeSrc = readFileSync( + '.github/scripts/apt-mirror-probe.sh', + 'utf8', + ); + const probePath = '.github/scripts/apt-mirror-probe.sh'; + const { root, repo, bin } = setup({ [probePath]: probeSrc }); + try { + const shellcheckLog = path.join(root, 'shellcheck.log'); + stub(bin, 'shellcheck', `echo "$@" >> '${shellcheckLog}'`); + const result = runLane(linters.shellcheck.run, { repo, bin }); + expect(result.status).toBe(0); + // Exact argv: pins the lane options and the awk colon-strip, and + // proves this file (not a root-level namesake) reached shellcheck. + expect(readFileSync(shellcheckLog, 'utf8').trim()).toBe( + '--check-sourced --enable=all --exclude=SC2002,SC2129,SC2310' + + ` --severity=style --format=gcc --color=never ${probePath}`, + ); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }, + ); + it('appends the pip --user bin dir after the inherited PATH', async () => { const { getLinterPath } = await import('../lint.js'); const env = { HOME: '/home/runner', PATH: '/usr/bin:/bin' }; diff --git a/scripts/tests/release-workflow.test.js b/scripts/tests/release-workflow.test.js index eb78c4259b1..ed646abcb3e 100644 --- a/scripts/tests/release-workflow.test.js +++ b/scripts/tests/release-workflow.test.js @@ -2516,6 +2516,23 @@ describe('release workflow', () => { ); }); + it('selects the apt mirror before the always-cold sandbox build', () => { + // #16271: this lane prunes the builder cache, so its apt layers are always + // cold — the same defect #16214 fixed for the E2E leg. It must source the + // shared probe and apply it before `build:sandbox`, so a mainland host + // builds against the VPC intranet mirror rather than the throttled default. + // Match the source statement, not a substring — the shellcheck directive + // above it also names the file. + expect(dockerIntegrationScript).toMatch( + /^source .*apt-mirror-probe\.sh"$/m, + ); + const applyIndex = dockerIntegrationScript.indexOf('apply_apt_mirror'); + expect(applyIndex).toBeGreaterThan(-1); + expect( + dockerIntegrationScript.indexOf('npm run build:sandbox'), + ).toBeGreaterThan(applyIndex); + }); + it('prunes BuildKit cache and gates the docker data root before the build and the vitest phase', () => { // #13479: run 37374675168 passed the job-start disk floor gate and the // runner still died on ENOSPC 24 minutes into the docker test step. The @@ -2591,6 +2608,9 @@ describe('release workflow', () => { dockerPostFloorKb, postGateExit, inspectTimesOut = false, + aptDefaultRate, + aptAliyunRate, + aptTunaRate, } = {}) => { const directory = mkdtempSync( join(tmpdir(), 'release-docker-integration-'), @@ -2647,8 +2667,25 @@ if [ "$1" = 'info' ]; then fi exit 0`, ); - stub('npm', `printf 'npm %s\\n' "$*" >> "$CALLS_LOG"`); + stub( + 'npm', + `printf 'npm %s flags=%s\\n' "$*" "\${BUILD_SANDBOX_FLAGS:-}" >> "$CALLS_LOG"`, + ); stub('npx', `printf 'npx %s\\n' "$*" >> "$CALLS_LOG"`); + // Deterministic stand-in for the probe's range reads so the suite stays + // hermetic (#16278 R1-1): answer the -w '%{http_code} %{speed_download}' + // projection per URL, default slow + one fast mirror. + stub( + 'curl', + `for url; do :; done +case "$url" in + *deb.debian.org*) printf '206 %s' "\${CURL_DEFAULT_RATE:-10000}" ;; + *mirrors.aliyun.com*) printf '206 %s' "\${CURL_ALIYUN_RATE:-5000000}" ;; + *mirrors.tuna.tsinghua.edu.cn*) printf '206 %s' "\${CURL_TUNA_RATE:-0}" ;; + *mirrors.cloud.aliyuncs.com*) printf '000 0' ;; + *) printf '000 0' ;; +esac`, + ); if (!helperAbsent) { // Both checkpoints share this stub: count invocations so a case can // fail the pre-vitest gate while the pre-build gate passes. @@ -2683,6 +2720,10 @@ exit 0`, PRUNE_FAILS: pruneFails ? '1' : '0', IMAGE_PRUNE_FAILS: imagePruneFails ? '1' : '0', DOCKER_INFO_FAILS: dockerInfoFails ? '1' : '0', + BUILD_SANDBOX_FLAGS: '--label org.qwen-code.ci.sandbox=true', + CURL_DEFAULT_RATE: String(aptDefaultRate ?? 10000), + CURL_ALIYUN_RATE: String(aptAliyunRate ?? 5000000), + CURL_TUNA_RATE: String(aptTunaRate ?? 0), }; // The floor knobs come from the test, never the ambient environment: // the suite's verdict must not depend on the env of whoever runs it. @@ -2721,6 +2762,30 @@ exit 0`, } }; + it.skipIf(process.platform === 'win32')( + 'selects the fast apt mirror and preserves the CI label in the build flags', + () => { + const { result, calls } = runDockerIntegrationScript(); + expect(result.status).toBe(0); + expect(calls).toContain('--build-arg APT_MIRROR=mirrors.aliyun.com'); + expect(calls).toContain('--label org.qwen-code.ci.sandbox=true'); + }, + ); + + it.skipIf(process.platform === 'win32')( + 'adds no apt mirror build arg when every endpoint is slow', + () => { + const { result, calls } = runDockerIntegrationScript({ + aptDefaultRate: 10000, + aptAliyunRate: 20000, + aptTunaRate: 15000, + }); + expect(result.status).toBe(0); + expect(calls).not.toContain('--build-arg APT_MIRROR='); + expect(calls).toContain('--label org.qwen-code.ci.sandbox=true'); + }, + ); + it.skipIf(process.platform === 'win32')( 'reclaims without the disk gate or the lock protocol on hosted runners', () => {