mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
易良
c2921217ef
* ci: gate on the OS temp directory and shorten /tmp retention to 1d
The 2026-10-31 hz incident: /tmp tmpfs hit 100% inodes while the
workspace and RUNNER_TEMP sat on a data disk with millions free, so
every disk-floor gate was green and jobs died on mkdtemp ENOSPC
mid-suite. The gate now always checks the OS temp directory
(deduplicated against caller arguments); the tmpfiles policy drops
7d->1d because the measured 120-260k files/day steady state exceeds
the tmpfs inode ceiling at 7d. Mutation-verified: dropping the
os.tmpdir() check reds the suite.
* test(ci): cover the TMPDIR-unset default and pin the 1d retention
Review findings: the new OS-temp test never executed the production
incident path (TMPDIR unset -> /tmp default) — a stripped default
survived the suite green; and the 7d->1d retention change had no test
coverage — a revert shipped the incident again. Add the unset-TMPDIR
sub-run, a set-equality dedup assertion, and a qwen-docker-cleanup
suite pin on the exact tmpfiles line. Mutations verified red.
* test(ci): pin RUNNER_TEMP apart so the /tmp fallback test bites
The first version of the TMPDIR-unset test let the RUNNER_TEMP default
(/tmp) satisfy the assertion, so a stripped ${TMPDIR:-/tmp} default
survived green — the exact regression the test exists to catch. Pin
RUNNER_TEMP to a distinct directory; the /tmp sample can then only come
from the OS-temp append. Mutation now reds.
* test(ci): unset TMPDIR as empty string; destructure was a no-op
The previous fix still failed on clean trees: spreading a destructured
env cannot DELETE a key — process.env.TMPDIR re-entered via runGate's
own spread, so the /tmp fallback never fired and the test failed
against the correct script. An empty-string TMPDIR is bash-unset for
${TMPDIR:-/tmp} (and bare for the regression), which is the exact
distinction under test. Clean 8/8; stripped-default mutation reds.
* ci(ecs-runner): enforce the /tmp retention from the owned daily service
Review finding: the 1d policy's enforcement agent was the distro's
systemd-tmpfiles-clean.timer, which can be masked on hardened images and
reports 'static' (unverifiable via is-enabled) on stock Ubuntu — with
nothing in the qwen stack running systemd-tmpfiles --clean, the measured
120-260k files/day steady state regrows past the tmpfs inode ceiling and
the 2026-10-31 incident class recurs with every test green. Add
ExecStartPost=systemd-tmpfiles --clean --prefix=/tmp to the daily
qwen-docker-cleanup.service, pin it in its test suite, and document the
enforcement owner in the README verification block. Mutation-verified.
100 lines
4.2 KiB
Bash
Executable File
100 lines
4.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Pre-job disk floor gate for self-hosted Linux runners (#10035).
|
|
#
|
|
# ENOSPC failed `npm ci` mid-install on a saturated host while the workspace
|
|
# cleanup timer skipped busy runners, so the job died mid-suite and the host
|
|
# looked healthy by the time anyone ran `df`. Failing BEFORE the heavy step
|
|
# turns that into a clear, retryable gate failure instead of a corrupted
|
|
# half-run; a re-run lands on a host with headroom.
|
|
# Checkout populates this script first, so the gate cannot protect checkout.
|
|
#
|
|
# Usage: check-disk-floor.sh [DIR ...]
|
|
# With no directories, checks GITHUB_WORKSPACE and RUNNER_TEMP.
|
|
#
|
|
# The OS temp directory (TMPDIR, defaulting to /tmp) is ALWAYS checked in
|
|
# addition to the caller's directories (deduplicated): on 2026-10-31 the hz
|
|
# fleet's /tmp tmpfs hit 100% inodes while the workspace and RUNNER_TEMP sat
|
|
# on a data disk with millions free, so every gate was green and the jobs
|
|
# died on mkdtemp ENOSPC mid-suite. os.tmpdir() writes (node, surefire,
|
|
# tsx, git subprocess temp files) do not follow RUNNER_TEMP.
|
|
#
|
|
# Floors, checked per directory's filesystem (env-overridable):
|
|
# DISK_FLOOR_MIN_FREE_KB default 2097152 (2 GiB)
|
|
# DISK_FLOOR_MIN_FREE_INODES default 100000
|
|
#
|
|
# Every check emits a DISKFLOOR sample tagged with runner and run ids so a
|
|
# breached gate can be correlated with the GitHub job that hit it.
|
|
set -u
|
|
|
|
MIN_FREE_KB="${DISK_FLOOR_MIN_FREE_KB:-2097152}"
|
|
MIN_FREE_INODES="${DISK_FLOOR_MIN_FREE_INODES:-100000}"
|
|
|
|
validate_floor_override() {
|
|
local name="$1"
|
|
local value="$2"
|
|
|
|
if ! [[ "$value" =~ ^[0-9]{1,18}$ ]]; then
|
|
echo "::error::${name} must be a non-negative integer of at most 18 digits"
|
|
exit 1
|
|
fi
|
|
}
|
|
|
|
validate_floor_override DISK_FLOOR_MIN_FREE_KB "$MIN_FREE_KB"
|
|
validate_floor_override DISK_FLOOR_MIN_FREE_INODES "$MIN_FREE_INODES"
|
|
|
|
TAG="runner[${RUNNER_NAME:-unknown}] run[${GITHUB_RUN_ID:-local}/${GITHUB_RUN_ATTEMPT:-1}] job[${GITHUB_JOB:-check-disk-floor}]"
|
|
|
|
if [ "$#" -eq 0 ]; then
|
|
set -- "${GITHUB_WORKSPACE:-.}" "${RUNNER_TEMP:-/tmp}"
|
|
fi
|
|
|
|
# Always include the OS temp directory: it is a distinct filesystem from the
|
|
# caller's directories on this fleet, and its exhaustion is exactly the
|
|
# failure class this gate exists to catch before the heavy step.
|
|
os_tmp="${TMPDIR:-/tmp}"
|
|
for existing in "$@"; do
|
|
if [ "$existing" = "$os_tmp" ]; then
|
|
os_tmp=''
|
|
break
|
|
fi
|
|
done
|
|
if [ -n "$os_tmp" ]; then
|
|
set -- "$@" "$os_tmp"
|
|
fi
|
|
|
|
status=0
|
|
for dir in "$@"; do
|
|
if [ ! -d "$dir" ]; then
|
|
echo "::warning::disk floor check skipped for missing directory: ${dir}"
|
|
continue
|
|
fi
|
|
# df -kP is POSIX-portable: one line per filesystem, sizes in KiB, mount
|
|
# point last. With -i the same layout reports inodes instead of blocks.
|
|
space_line="$(df -kP "$dir" 2>/dev/null | tail -n 1 || true)"
|
|
inode_line="$(df -kPi "$dir" 2>/dev/null | tail -n 1 || true)"
|
|
avail_kb="$(awk '{print $4}' <<< "$space_line")"
|
|
mount="$(awk '{for (i = 6; i <= NF; i++) printf "%s%s", $i, (i < NF ? " " : ""); print ""}' <<< "$space_line")"
|
|
free_inodes="$(awk '{print $4}' <<< "$inode_line")"
|
|
|
|
case "$avail_kb" in '' | *[!0-9]*) avail_kb='' ;; esac
|
|
case "$free_inodes" in '' | *[!0-9]*) free_inodes='' ;; esac
|
|
|
|
echo "DISKFLOOR $(date -u +%Y-%m-%dT%H:%M:%SZ) ${TAG} dir[${dir}] fs[${mount}] avail_kb[${avail_kb:-unknown}] floor_kb[${MIN_FREE_KB}] free_inodes[${free_inodes:-unknown}] floor_inodes[${MIN_FREE_INODES}]"
|
|
|
|
if [ -z "$avail_kb" ] && [ -z "$free_inodes" ]; then
|
|
echo "::warning::could not read disk usage for ${dir}; letting the job proceed"
|
|
continue
|
|
fi
|
|
|
|
if [ -n "$avail_kb" ] && [ "$avail_kb" -lt "$MIN_FREE_KB" ]; then
|
|
echo "::error::Disk floor breached on ${RUNNER_NAME:-unknown}: ${dir} has ${avail_kb} KiB free, below the ${MIN_FREE_KB} KiB floor. Failing before the heavy step instead of dying on ENOSPC mid-run — re-run the job to land on a host with headroom."
|
|
status=1
|
|
fi
|
|
if [ -n "$free_inodes" ] && [ "$free_inodes" -lt "$MIN_FREE_INODES" ]; then
|
|
echo "::error::Disk floor breached on ${RUNNER_NAME:-unknown}: ${dir} has ${free_inodes} free inodes, below the ${MIN_FREE_INODES} floor. Failing before the heavy step instead of dying on ENOSPC mid-run — re-run the job to land on a host with headroom."
|
|
status=1
|
|
fi
|
|
done
|
|
|
|
exit "$status"
|