fix(ci): probe the apt mirror so mainland E2E hosts can build the sandbox image (#16218)

* fix(ci): probe the apt mirror so mainland E2E hosts can build the sandbox image

The docker E2E leg builds the sandbox image per push under a SHA-scoped tag, so
on any host whose image store no longer holds a recent build, the two apt-get
layers are downloaded again. deb.debian.org serves the Hong Kong hosts at MB/s
and the mainland-China hosts at 11-20 kB/s, which turns that build into 2-3
hours against the job's 60-minute cap. Over the past week 97 of 127 docker legs
on the ecs-qwen-hz* hosts were cancelled this way, and the run reports
`cancelled` instead of a verdict.

Probe rather than pin a mirror or a host-name map: the build measures
deb.debian.org and two domestic mirrors, and passes --build-arg APT_MIRROR only
when the default is slower than 250 kB/s and a mirror beats it by 4x. The image
keeps deb.debian.org as its default source, so release builds, local sandbox
builds and the healthy Hong Kong hosts are unchanged, and no fleet inventory
needs to be maintained.

The probe's measurements are logged, so the next slow build can be diagnosed
from its own output.

Refs #16214

* fix(ci): require an absolutely fast mirror before overriding the apt source

The relative guard alone only asked a mirror to be 4x faster than the default,
so on a host where both endpoints were slow the mirror still "won" and moved
the host onto a source it never needed. Require 1 MB/s in absolute terms too:
below that the build is slow either way, and the extra cache key is not worth
it.

* test(ci): pin the apt-probe threshold values, not just the gate structure

The new contract test asserted the gate variables and comparisons but not
their values: mutating slow_default 10x up or fast_mirror 10x down left the
suite green while flipping the both-slow host shape onto a still-slow mirror
(review finding on #16218). Pin both literals with a word boundary — a plain
substring pin would still match the 10x-up mutation — and witness both
mutations reddening the suite before restoring the values.

* test(ci): require the mirror ARG in both Dockerfile stages

A single-match assertion survived deleting the runtime-stage ARG APT_MIRROR
(R1-1, mutation-witnessed by review): the builder declaration satisfied the
regex while the runtime apt layer silently kept the slow default. Count both
declarations instead; deleting either stage's ARG now reddens the suite.
This commit is contained in:
易良 authored and GitHub committed 2026-11-04 05:52:30 +08:00
1 parent 4203138bc7
commit aaf523cfc4
3 files changed
+134 -2

No files matched your search

+46
View File
@@ -1094,6 +1094,52 @@ describe('e2e workflow', () => {
);
});
it('probes the apt mirror instead of pinning a region', () => {
// #16214: deb.debian.org serves this pool at MB/s from Hong Kong and
// 11-20 kB/s from the mainland hosts, where the two apt layers then
// outlast the job's 60-minute cap. So neither a single mirror nor a
// host-name map is correct — the build probes and overrides only on a
// clear win, which leaves healthy hosts on the built-in source.
expect(e2eRunScript).toContain('mirrors.aliyun.com');
expect(e2eRunScript).toContain('mirrors.tuna.tsinghua.edu.cn');
expect(e2eRunScript).toContain('--build-arg APT_MIRROR=');
// Both stages need the ARG: a stage cannot inherit an ARG declared
// before its own FROM, so one declaration per stage is the contract.
// Counting (not toMatch) catches deleting either one — R1-1 witnessed
// the runtime-stage removal surviving a single-match assertion.
expect(
readFileSync('Dockerfile', 'utf8').match(/^ARG APT_MIRROR=$/gm),
).toHaveLength(2);
const selectIndex = e2eRunScript.indexOf('select_apt_mirror()');
expect(selectIndex).toBeGreaterThan(-1);
const callIndex = e2eRunScript.indexOf(
'apt_mirror="$(select_apt_mirror)"',
);
expect(callIndex).toBeGreaterThan(selectIndex);
// The probe has to run before the build consumes the flag, otherwise the
// mirror is decided after the layer it was meant to speed up.
expect(e2eRunScript.indexOf('if ! build_image;')).toBeGreaterThan(
callIndex,
);
// Three conditions hold the override back, and dropping any one is a real
// regression: a healthy default host keeps its source, a mirror has to be
// fast in absolute terms rather than merely faster than a broken default,
// and it has to beat the default decisively. The literals are pinned, not
// just the variable names: a 10x mutation of either threshold leaves the
// structure intact but flips the both-slow shape onto a still-slow mirror.
// \b, not toContain: 'slow_default=250000' is a substring of the 10x
// mutation 'slow_default=2500000', which would keep a naive pin green.
expect(e2eRunScript).toMatch(/slow_default=250000\b/);
expect(e2eRunScript).toMatch(/-lt "\$\{slow_default\}" \]/);
expect(e2eRunScript).toMatch(/fast_mirror=1000000\b/);
expect(e2eRunScript).toMatch(/-gt "\$\{fast_mirror\}" \]/);
expect(e2eRunScript).toMatch(/-gt \$\(\(default_rate \* 4\)\)/);
// The measurements have to reach the log, or the next slow build is
// another manual investigation.
expect(e2eRunScript).toContain('apt probe:');
});
it('keeps one bounded retry without pruning the shared daemon', () => {
expect(e2eRunScript.match(/build_image/g)).toHaveLength(3);
expect(e2eRunScript).toContain(