ci: gate on the OS temp directory and shorten /tmp retention to 1d (#16126)

* ci: gate on the OS temp directory and shorten /tmp retention to 1d

The 2026-10-31 hz incident: /tmp tmpfs hit 100% inodes while the
workspace and RUNNER_TEMP sat on a data disk with millions free, so
every disk-floor gate was green and jobs died on mkdtemp ENOSPC
mid-suite. The gate now always checks the OS temp directory
(deduplicated against caller arguments); the tmpfiles policy drops
7d->1d because the measured 120-260k files/day steady state exceeds
the tmpfs inode ceiling at 7d. Mutation-verified: dropping the
os.tmpdir() check reds the suite.

* test(ci): cover the TMPDIR-unset default and pin the 1d retention

Review findings: the new OS-temp test never executed the production
incident path (TMPDIR unset -> /tmp default) — a stripped default
survived the suite green; and the 7d->1d retention change had no test
coverage — a revert shipped the incident again. Add the unset-TMPDIR
sub-run, a set-equality dedup assertion, and a qwen-docker-cleanup
suite pin on the exact tmpfiles line. Mutations verified red.

* test(ci): pin RUNNER_TEMP apart so the /tmp fallback test bites

The first version of the TMPDIR-unset test let the RUNNER_TEMP default
(/tmp) satisfy the assertion, so a stripped ${TMPDIR:-/tmp} default
survived green — the exact regression the test exists to catch. Pin
RUNNER_TEMP to a distinct directory; the /tmp sample can then only come
from the OS-temp append. Mutation now reds.

* test(ci): unset TMPDIR as empty string; destructure was a no-op

The previous fix still failed on clean trees: spreading a destructured
env cannot DELETE a key — process.env.TMPDIR re-entered via runGate's
own spread, so the /tmp fallback never fired and the test failed
against the correct script. An empty-string TMPDIR is bash-unset for
${TMPDIR:-/tmp} (and bare for the regression), which is the exact
distinction under test. Clean 8/8; stripped-default mutation reds.

* ci(ecs-runner): enforce the /tmp retention from the owned daily service

Review finding: the 1d policy's enforcement agent was the distro's
systemd-tmpfiles-clean.timer, which can be masked on hardened images and
reports 'static' (unverifiable via is-enabled) on stock Ubuntu — with
nothing in the qwen stack running systemd-tmpfiles --clean, the measured
120-260k files/day steady state regrows past the tmpfs inode ceiling and
the 2026-10-31 incident class recurs with every test green. Add
ExecStartPost=systemd-tmpfiles --clean --prefix=/tmp to the daily
qwen-docker-cleanup.service, pin it in its test suite, and document the
enforcement owner in the README verification block. Mutation-verified.

.github/scripts/check-disk-floor.sh

@@ -11,6 +11,13 @@
 # Usage: check-disk-floor.sh [DIR ...]
 # With no directories, checks GITHUB_WORKSPACE and RUNNER_TEMP.
 #
+# The OS temp directory (TMPDIR, defaulting to /tmp) is ALWAYS checked in
+# addition to the caller's directories (deduplicated): on 2026-10-31 the hz
+# fleet's /tmp tmpfs hit 100% inodes while the workspace and RUNNER_TEMP sat
+# on a data disk with millions free, so every gate was green and the jobs
+# died on mkdtemp ENOSPC mid-suite. os.tmpdir() writes (node, surefire,
+# tsx, git subprocess temp files) do not follow RUNNER_TEMP.
+#
 # Floors, checked per directory's filesystem (env-overridable):
 #   DISK_FLOOR_MIN_FREE_KB      default 2097152 (2 GiB)
 #   DISK_FLOOR_MIN_FREE_INODES  default 100000
@@ -41,6 +48,20 @@ if [ "$#" -eq 0 ]; then
   set -- "${GITHUB_WORKSPACE:-.}" "${RUNNER_TEMP:-/tmp}"
 fi
 
+# Always include the OS temp directory: it is a distinct filesystem from the
+# caller's directories on this fleet, and its exhaustion is exactly the
+# failure class this gate exists to catch before the heavy step.
+os_tmp="${TMPDIR:-/tmp}"
+for existing in "$@"; do
+  if [ "$existing" = "$os_tmp" ]; then
+    os_tmp=''
+    break
+  fi
+done
+if [ -n "$os_tmp" ]; then
+  set -- "$@" "$os_tmp"
+fi
+
 status=0
 for dir in "$@"; do
   if [ ! -d "$dir" ]; then

.github/scripts/check-disk-floor.test.mjs

@@ -124,4 +124,92 @@ describe('check-disk-floor', () => {
       rmSync(temp, { recursive: true, force: true });
     }
   });
+
+  it('always checks the OS temp directory, deduplicated', () => {
+    // The 2026-10-31 hz incident: /tmp hit 100% inodes while the workspace
+    // and RUNNER_TEMP sat on a data disk, so caller directories alone were
+    // green while every mkdtemp died mid-suite. The gate must add the OS
+    // temp directory on its own — and not duplicate it when a caller
+    // already passed it.
+    const dir = mkdtempSync(join(tmpdir(), 'disk-floor-os-'));
+    const osTmp = mkdtempSync(join(tmpdir(), 'disk-floor-ostmp-'));
+    try {
+      // Explicit dirs: the OS temp dir is still checked (huge inode floor
+      // forces the breach on it deterministically).
+      const breached = runGate({
+        dirs: [dir],
+        env: {
+          TMPDIR: osTmp,
+          DISK_FLOOR_MIN_FREE_KB: '1',
+          DISK_FLOOR_MIN_FREE_INODES: '999999999999',
+        },
+      });
+      assert.equal(breached.status, 1);
+      assert.ok(breached.stdout.includes(`dir[${osTmp}]`));
+      assert.ok(breached.stdout.includes('free inodes'));
+
+      // Deduplication: passing the OS temp dir explicitly checks it once,
+      // and the sample set is exactly the caller's dirs (an appended-then-
+      // deduped path and a silently-dropped path differ only in set shape).
+      const counted = runGate({
+        dirs: [dir, osTmp],
+        env: {
+          TMPDIR: osTmp,
+          DISK_FLOOR_MIN_FREE_KB: '1',
+          DISK_FLOOR_MIN_FREE_INODES: '1',
+        },
+      });
+      assert.equal(counted.status, 0, counted.stderr);
+      const samples = counted.stdout
+        .split('\n')
+        .filter((line) => line.startsWith('DISKFLOOR '));
+      assert.equal(samples.length, 2);
+      const sampledDirs = samples
+        .map((line) => /dir\[([^\]]+)\]/.exec(line)?.[1])
+        .sort();
+      assert.deepEqual(sampledDirs, [dir, osTmp].sort());
+    } finally {
+      rmSync(dir, { recursive: true, force: true });
+      rmSync(osTmp, { recursive: true, force: true });
+    }
+  });
+
+  it('falls back to /tmp when TMPDIR is unset', () => {
+    // The incident fleet's os.tmpdir() is /tmp with TMPDIR unset; the
+    // script's `${TMPDIR:-/tmp}` default must keep gating that path. A
+    // regression to plain `${TMPDIR}` appends nothing and the incident
+    // class returns with the suite green. An empty-string TMPDIR is the
+    // bash-equivalent of unset for `:-` (but not for a bare expansion),
+    // which is exactly the distinction under test. RUNNER_TEMP is pinned
+    // to a distinct directory so a dir[/tmp] sample can only come from the
+    // OS-temp append, not the RUNNER_TEMP default (mutation-verified).
+    const ws = mkdtempSync(join(tmpdir(), 'disk-floor-ws-'));
+    const temp = mkdtempSync(join(tmpdir(), 'disk-floor-tmp-'));
+    try {
+      const result = runGate({
+        env: {
+          GITHUB_WORKSPACE: ws,
+          RUNNER_TEMP: temp,
+          TMPDIR: '',
+          DISK_FLOOR_MIN_FREE_KB: '1',
+          DISK_FLOOR_MIN_FREE_INODES: '1',
+        },
+      });
+      assert.equal(result.status, 0, result.stderr);
+      assert.ok(result.stdout.includes(`dir[${ws}]`));
+      assert.ok(result.stdout.includes(`dir[${temp}]`));
+      assert.ok(
+        result.stdout
+          .split('\n')
+          .some(
+            (line) =>
+              line.startsWith('DISKFLOOR ') && line.includes('dir[/tmp]'),
+          ),
+        `expected the /tmp fallback sample, got: ${result.stdout}`,
+      );
+    } finally {
+      rmSync(ws, { recursive: true, force: true });
+      rmSync(temp, { recursive: true, force: true });
+    }
+  });
 });

.github/scripts/ecs-runner/README.md

@@ -18,8 +18,9 @@ replace it.
 - Uses the existing Qwen sandbox daemon lock for labelled image pruning.
   Build-cache pruning relies on Docker's in-use protection and runs even
   when that lock is busy or missing.
-- Changes the host `/tmp` retention policy from 30 days to 7 days through
-  `systemd-tmpfiles`.
+- Changes the host `/tmp` retention policy from 30 days to 1 day through
+  `systemd-tmpfiles` (measured 2026-10-31: CI hosts generate ~120-260k
+  temp files/day and a 7d steady state exceeds the tmpfs inode ceiling).
 
 Use this only on dedicated CI hosts. A sandbox-labelled or `qwen-code-*`-named
 container older than 24 hours is considered leaked and may be removed even if
@@ -42,7 +43,7 @@ sudo .github/scripts/ecs-runner/install-qwen-docker-cleanup.sh
 ```
 
 The installer copies the cleanup command and systemd units into the host,
-installs the seven-day `/tmp` policy, reloads systemd, and enables the timer.
+installs the one-day `/tmp` policy, reloads systemd, and enables the timer.
 Run the same command again after pulling a newer version to update the host.
 Merging changes does not update installed copies. Run the installer on every
 host, including hosts that do not yet have the timer, and check each host's
@@ -56,14 +57,21 @@ systemctl is-enabled qwen-docker-cleanup.timer
 systemctl is-active qwen-docker-cleanup.timer
 systemctl list-timers qwen-docker-cleanup.timer
 systemd-tmpfiles --cat-config | grep '^D /tmp '
+journalctl -u qwen-docker-cleanup.service -n 5 --no-pager
 ```
 
 The expected `/tmp` policy is:
 
 ```text
-D /tmp 1777 root root 7d
+D /tmp 1777 root root 1d
 ```
 
+The retention policy is enforced by the daily `qwen-docker-cleanup.service`
+itself (`ExecStartPost=systemd-tmpfiles --clean --prefix=/tmp`), not by the
+distro's `systemd-tmpfiles-clean.timer` — on hardened images that timer can
+be masked, and it reports `static` (unverifiable via `is-enabled`) on stock
+Ubuntu. The journal line above confirms the ExecStartPost ran.
+
 ## Run manually
 
 Drain the host first if it may contain a legitimate job or container running
@@ -110,7 +118,7 @@ the job log.
 
 This is a temporary-directory convention, not filesystem isolation: commands
 that explicitly write elsewhere bypass it. Existing arbitrary copies in `/tmp`
-remain covered only by the seven-day policy. This service does not remove them
+remain covered only by the one-day policy. This service does not remove them
 by name because they may belong to an active job. A later review job on the
 same runner registration retries removing stale `qwen-review-scratch.*`
 directories from `RUNNER_TEMP`; residue on an idle registration still requires

.github/scripts/ecs-runner/qwen-ci-tmp.conf

@@ -1,2 +1,5 @@
-# Dedicated CI hosts generate large numbers of disposable test directories.
-D /tmp 1777 root root 7d
+# Dedicated CI hosts generate large numbers of disposable test directories
+# (node-compile-cache, tsx, surefire, playwright — ~120-260k files/day).
+# Measured 2026-10-31: a 7d steady state exceeds the 1,048,576-inode tmpfs
+# ceiling; 1d keeps the steady state at 11-25% with the cleanup timer.
+D /tmp 1777 root root 1d

.github/scripts/ecs-runner/qwen-docker-cleanup.service

@@ -12,3 +12,10 @@ Type=oneshot
 # case instead of cutting the sweep off mid-prune.
 TimeoutStartSec=1h
 ExecStart=/usr/local/sbin/qwen-docker-cleanup
+# Enforce the /etc/tmpfiles.d/tmp.conf (1d) retention here rather than
+# relying on the distro's systemd-tmpfiles-clean.timer: on hardened images
+# that timer can be masked, and it reports "static" on stock Ubuntu —
+# unverifiable via is-enabled — while the fleet's measured 120-260k
+# temp files/day regrows past the tmpfs inode ceiling without an
+# enforcement agent (the 2026-10-31 incident class).
+ExecStartPost=/usr/bin/systemd-tmpfiles --clean --prefix=/tmp

.github/scripts/ecs-runner/qwen-docker-cleanup.test.mjs

@@ -89,3 +89,32 @@ test('reports cache-prune failure instead of a successful service exit', () => {
   assert.match(result.stderr, /Docker build cache cleanup failed/);
   assert.match(result.calls, /^image prune --all --force/m);
 });
+
+test('installs a one-day /tmp retention policy', () => {
+  // Measured 2026-10-31: CI hosts generate ~120-260k temp files/day, and a
+  // 7d steady state exceeds the ~1M-inode tmpfs ceiling — that is the hz
+  // fleet incident. The installer's conf must pin 1d; a revert ships the
+  // incident again with every test green.
+  const conf = readFileSync(
+    new URL('./qwen-ci-tmp.conf', import.meta.url),
+    'utf8',
+  );
+  assert.match(conf, /^D \/tmp 1777 root root 1d$/m);
+  assert.doesNotMatch(conf, /^D \/tmp 1777 root root 7d$/m);
+});
+
+test('the daily service enforces the /tmp retention itself', () => {
+  // The distro's systemd-tmpfiles-clean.timer can be masked (hardened
+  // images) and reports "static" on stock Ubuntu — unverifiable via
+  // is-enabled — so the retention policy needs an enforcement agent the
+  // qwen stack owns. Dropping the ExecStartPost regrows the 2026-10-31
+  // incident class with every doc and test green.
+  const service = readFileSync(
+    new URL('./qwen-docker-cleanup.service', import.meta.url),
+    'utf8',
+  );
+  assert.match(
+    service,
+    /^ExecStartPost=\/usr\/bin\/systemd-tmpfiles --clean --prefix=\/tmp$/m,
+  );
+});