fix(ci): close the round-5 patrol findings, including a macOS lane regression

R5-1 was a regression this PR introduced: three fixtures hardcoded
RM_BIN '/usr/bin/rm', which macOS does not have, and the vitest exclude is
win32-only so the macOS scripts lane collects them. Resolve rm and mktemp with
command -v instead. Witnessed against the real step body with an absent binary:
bash -e exits 127 with .git still present, bash -c exits 0 with .git present —
matching the reported shape exactly.

R5-2: git layers env-provided config above both file scopes, so neutralising
the config files left core.hooksPath reachable through GIT_CONFIG_COUNT and
GIT_CONFIG_PARAMETERS. Pin both in the checkout step's own env in both jobs, as
qwen-autofix already does.

R5-3: the actionable-round gate accepted any non-empty array while the consumer
silently dropped entries matching no candidate, so a non-empty but unmatchable
list still produced a green round that acted on nothing — the verdict step is
skipped once has_decisions is true. Close the loop in actOnDecisions: count
matches and throw when a non-empty list matched nothing. Per-decision failures
stay swallowed, and a matched-but-failed decision still counts.

R5-4: pin both guard copies, not just classify's. Dropping gh from act's loop
did not even abort, because no act step interpolates GH_BIN and the driver falls
back to a bare-name spawn. Also assert every consumed guard output is published
by that job's own loop.

R5-5: cap the capture and identity steps at 1 minute and check the output path
is a regular file before appending, so a planted FIFO cannot block them until
the job cap and conclude the round cancelled. Job cap 35 -> 36 keeps the
documented headroom now that two more steps are capped.

R5-6: gate the accept-path case on the uid alone and only the symlink arms on a
readlink -f capability probe, so macOS still pins the accept verdict and
publish()'s writes. Probed: the base accept arm invokes readlink zero times.

9 targeted mutations caught, byte-exact restore; the two not witnessable on
Linux carry the macOS-model probes above. 500 passed | 1 skipped across 7
suites, 134/134 shared node gates, size ratchet and lint clean.
This commit is contained in:
wenshao committed 2026-11-07 23:44:02 +08:00
1 parent df8f175bfe
commit fa9e3ca963
6 files changed
+209 -47

No files matched your search

+13
View File
@@ -547,6 +547,7 @@ export async function actOnDecisions(client, targets, decisions) {
if (!Array.isArray(targets) || !Array.isArray(decisions)) return;
const byKey = new Map(targets.map((target) => [decisionKey(target), target]));
const handled = new Set();
let matched = 0;
for (const decision of decisions) {
const key = decisionKey(decision);
if (handled.has(key)) continue;
@@ -559,6 +560,7 @@ export async function actOnDecisions(client, targets, decisions) {
);
continue;
}
matched += 1;
await actOnDecision(client, target, decision);
} catch (error) {
process.stderr.write(
@@ -566,6 +568,17 @@ export async function actOnDecisions(client, targets, decisions) {
);
}
}
// A non-empty list that matched nothing is the hollow-green shape: the round
// uploads, runs act, prints "no candidate" per entry and exits 0, while the
// workflow's verdict step is skipped because has_decisions is true. Fail
// loudly instead. Per-decision failures stay swallowed above — a candidate
// superseded between scan and act must not fail the round, and it still
// counts as matched because the decision did address a real scanned target.
if (decisions.length > 0 && matched === 0) {
throw new Error(
`act: ${decisions.length} decision(s) matched no scanned candidate; this round acted on nothing.`,
);
}
}
export async function resetSuccessfulFailures(client, prs) {
+1 -1
View File
@@ -40,7 +40,7 @@
9646 qwen-autofix-fork-bridge.yml
5942 qwen-autofix-fork-signal.yml
451750 qwen-autofix.yml
44551 qwen-ci-flaky-rerun.yml
46266 qwen-ci-flaky-rerun.yml
278504 qwen-code-pr-review.yml
107075 qwen-fleet-shepherd.yml
22680 qwen-issue-followup-bot.yml
+38 -9
View File
@@ -27,15 +27,15 @@ jobs:
# hosted pool: one round waited 2h09m for a runner, and 17 of 20 recent
# ticks were cancelled with zero jobs — they never received one at all.
runs-on: '${{ (github.repository == ''QwenLM/qwen-code'' && vars.MAINTAINER_ECS_RUNNER_DISABLED != ''true'') && fromJSON(''["self-hosted", "linux", "x64", "ecs-qwen"]'') || fromJSON(''["ubuntu-latest"]'') }}'
# Budget: capped steps sum to guard 2 + docker recheck 2 + resolve 12 +
# classifier 10 = 26, and scan (~1m measured), checkout (~22s), setup-node
# and the capture still have to fit. Two earlier caps were wrong: 20 omitted
# the guard's own 2 minutes, 25 omitted the classifier raise below. The
# resolver can attempt TWO sequential 10-minute pulls, so its step stays
# capped below the job: a cold host costs one round, not the job. Cron still
# fires every 10 minutes, so a worst-case cold round costs two ticks — the
# deliberate trade for not killing a round mid-pull.
timeout-minutes: 35
# Budget: capped steps sum to guard 2 + capture 1 + docker recheck 2 +
# identity 1 + ensure 5 + resolve 12 + classifier 10 = 33, and scan (~1m
# measured), checkout (~22s), setup-node and the upload still have to fit.
# Earlier caps were wrong twice: 20 omitted the guard's own 2 minutes, 25
# omitted the classifier raise. The resolver can attempt TWO sequential
# 10-minute pulls, so its step stays capped below the job: a cold host costs
# one round, not the job. Cron fires every 10 minutes, so a worst-case cold
# round costs ticks — the deliberate trade for not killing a round mid-pull.
timeout-minutes: 36
permissions:
actions: 'read'
contents: 'read'
@@ -263,6 +263,12 @@ jobs:
PATH: '${{ steps.guard.outputs.system_path }}'
GIT_CONFIG_GLOBAL: '${{ steps.gitmeta.outputs.gitconfig }}'
GIT_CONFIG_SYSTEM: '${{ steps.gitmeta.outputs.gitconfig }}'
# git layers env-provided config ABOVE both file scopes, so
# neutralising the files alone leaves core.hooksPath reachable through
# $GITHUB_ENV. Same channel qwen-autofix closes; step env, because the
# runner takes the last value for a duplicated $GITHUB_ENV key.
GIT_CONFIG_COUNT: '0'
GIT_CONFIG_PARAMETERS: ''
with:
ref: '${{ github.sha }}'
persist-credentials: false
@@ -291,10 +297,20 @@ jobs:
# therefore keeps the post-setup-node PATH as-is.
- name: 'Capture trusted PATH'
id: 'stage'
# A co-tenant can replace $GITHUB_OUTPUT with a FIFO; the append below
# would then block until this cap instead of the whole job cap.
timeout-minutes: 1
env:
SYSTEM_PATH: '${{ steps.guard.outputs.system_path }}'
RUNNER_ENVIRONMENT: '${{ runner.environment }}'
run: |-
# A co-tenant can replace this path with a FIFO; the append would then
# block. Check the shell-visible half of fstat's isFile() first, for
# BOTH branches, and let the step cap bound the residual race.
if [ ! -f "${GITHUB_OUTPUT}" ]; then
echo "::error::step output file is not a regular file; refusing to append to it."
exit 1
fi
if [ "${RUNNER_ENVIRONMENT}" = 'self-hosted' ]; then
echo "trusted_path=${SYSTEM_PATH}:${PATH}" >> "${GITHUB_OUTPUT}"
else
@@ -322,6 +338,8 @@ jobs:
# CI_BOT_PAT. It needs no checkout, so moving it here costs nothing.
- name: 'Resolve patrol identity'
id: 'identity'
# Same FIFO exposure as the capture step, and this one holds the PAT.
timeout-minutes: 1
env:
# The PAT rides in GITHUB_TOKEN and is handed to gh inline, so the
# preamble below can unset GH_TOKEN without losing the credential.
@@ -348,6 +366,11 @@ jobs:
export GH_CONFIG_DIR
bot_login="$(GH_TOKEN="${GITHUB_TOKEN}" "${GH_BIN:?}" api user --jq .login)"
test -n "${bot_login}"
# Same FIFO exposure as the capture step, on the step that holds the PAT.
if [ ! -f "${GITHUB_OUTPUT}" ]; then
echo "::error::step output file is not a regular file; refusing to append to it."
exit 1
fi
echo "bot_login=${bot_login}" >> "${GITHUB_OUTPUT}"
- name: 'Scan stale PR failures'
@@ -764,6 +787,12 @@ jobs:
PATH: '${{ steps.guard.outputs.system_path }}'
GIT_CONFIG_GLOBAL: '${{ steps.gitmeta.outputs.gitconfig }}'
GIT_CONFIG_SYSTEM: '${{ steps.gitmeta.outputs.gitconfig }}'
# git layers env-provided config ABOVE both file scopes, so
# neutralising the files alone leaves core.hooksPath reachable through
# $GITHUB_ENV. Same channel qwen-autofix closes; step env, because the
# runner takes the last value for a duplicated $GITHUB_ENV key.
GIT_CONFIG_COUNT: '0'
GIT_CONFIG_PARAMETERS: ''
with:
ref: '${{ github.sha }}'
persist-credentials: false
+108 -35
View File
@@ -15,6 +15,27 @@ import { join } from 'node:path';
import { describe, expect, it } from 'vitest';
import { parse } from 'yaml';
// macOS ships rm at /bin/rm, so a hardcoded /usr/bin/rm exits 127 on the macOS
// scripts lane (which collects these files — the vitest exclude is win32-only).
// Resolve external tools the way the workflows themselves do.
// macOS readlink has no -f, so only the symlink arms need this capability;
// the base accept, writable-root and missing-docker arms never resolve a link.
const hasReadlinkF = (() => {
try {
execFileSync('bash', ['-c', 'readlink -f / >/dev/null 2>&1'], {
stdio: 'ignore',
});
return true;
} catch {
return false;
}
})();
const resolvedBin = (name) =>
execFileSync('bash', ['-c', `command -v ${name}`], {
encoding: 'utf8',
}).trim();
const workflow = readFileSync(
'.github/workflows/qwen-ci-flaky-rerun.yml',
'utf8',
@@ -492,8 +513,10 @@ describe('ci failure patrol workflow', () => {
});
// Root bypasses every -w test, so the accept path is only meaningful for a
// non-root uid; macOS readlink has no -f, so the symlink cases need Linux.
it.skipIf(process.getuid?.() === 0 || process.platform !== 'linux')(
// non-root uid. The platform is NOT the gate: the base accept, writable-root
// and missing-docker arms never call readlink -f, so they must keep running on
// macOS — they are the only pins on the accept verdict and publish()'s writes.
it.skipIf(process.getuid?.() === 0)(
'behaviorally proves the guard accepts a non-writable base and publishes it',
() => {
const guard = yml.jobs.classify.steps[0];
@@ -547,36 +570,39 @@ describe('ci failure patrol workflow', () => {
expect(writableDir.outputs.system_path).toBeUndefined();
// A symlink is accepted only when the link's directory, the resolved
// target and the target's directory are all non-writable.
const linkOk = runGuard(guard, {
...approved,
...modes,
nodeLink: { targetMode: 0o555, targetDirMode: 0o555 },
});
expect(linkOk.status).toBe(0);
expect(linkOk.outputs.node).toMatch(/\/bin\/node$/);
const linkBad = runGuard(guard, {
...approved,
...modes,
nodeLink: { targetMode: 0o555, targetDirMode: 0o755 },
});
expect(linkBad.status).not.toBe(0);
expect(linkBad.output).toContain('no verified node');
// target and the target's directory are all non-writable. These three
// arms are the only ones that need readlink -f.
if (hasReadlinkF) {
const linkOk = runGuard(guard, {
...approved,
...modes,
nodeLink: { targetMode: 0o555, targetDirMode: 0o555 },
});
expect(linkOk.status).toBe(0);
expect(linkOk.outputs.node).toMatch(/\/bin\/node$/);
const linkBad = runGuard(guard, {
...approved,
...modes,
nodeLink: { targetMode: 0o555, targetDirMode: 0o755 },
});
expect(linkBad.status).not.toBe(0);
expect(linkBad.output).toContain('no verified node');
// The trust anchor itself: a lying readlink first on the ambient PATH
// must never run. If it did, it would report a harmless target for the
// writable-directory symlink above AND append its own gh= output line,
// which the runner would take as the last value for that key.
const planted = runGuard(guard, {
...approved,
...modes,
nodeLink: { targetMode: 0o555, targetDirMode: 0o755 },
plantReadlink: true,
});
expect(planted.status).not.toBe(0);
expect(planted.output).toContain('no verified node');
expect(planted.outputs.gh).not.toBe('/attacker/bin/gh');
expect(planted.outputs.node).toBeUndefined();
// The trust anchor itself: a lying readlink first on the ambient PATH
// must never run. If it did, it would report a harmless target for the
// writable-directory symlink above AND append its own gh= output line,
// which the runner would take as the last value for that key.
const planted = runGuard(guard, {
...approved,
...modes,
nodeLink: { targetMode: 0o555, targetDirMode: 0o755 },
plantReadlink: true,
});
expect(planted.status).not.toBe(0);
expect(planted.output).toContain('no verified node');
expect(planted.outputs.gh).not.toBe('/attacker/bin/gh');
expect(planted.outputs.node).toBeUndefined();
}
// The docker branch has its own refusal: the pinned binary missing must
// name docker, not whichever tool the fixture happened to fail on first.
@@ -617,6 +643,36 @@ describe('ci failure patrol workflow', () => {
// by builtins and invoked absolutely — a bare `readlink -f` would resolve
// through the plantable PATH this step exists to distrust, and a planted
// readlink runs with $GITHUB_OUTPUT and $GITHUB_ENV in its environment.
// Both copies: act's guard is byte-identical and holds the write PAT, but
// nothing pinned its tool loop or roots. Dropping gh there does not even
// abort — no act step interpolates GH_BIN, so PATROL_GH would silently fall
// back to a bare-name spawn on the plantable PATH.
for (const job of [yml.jobs.classify, yml.jobs.act]) {
const copy = job.steps[0];
expect(copy.env.PATROL_BIN_ROOTS, job.name).toBe(
'/usr/bin /bin /usr/local/bin',
);
expect(copy.run, job.name).toContain(
'for tool in rm git timeout mktemp gh node mkdir grep; do',
);
// Every guard output a later step consumes must be published by that
// job's own loop (docker is verified separately, from SANDBOX_COMMAND).
const consumed = new Set(
[
...JSON.stringify(job.steps).matchAll(
/steps\.guard\.outputs\.([\w-]+)/g,
),
].map((m) => m[1]),
);
const published = new Set(
(copy.run.match(/for tool in ([^;]+); do/)?.[1] ?? '').split(/\s+/),
);
published.add('docker');
published.add('system_path');
for (const name of consumed) {
expect(published.has(name), `${job.name} consumes ${name}`).toBe(true);
}
}
expect(guard.env.PATROL_BIN_ROOTS).toBe('/usr/bin /bin /usr/local/bin');
expect(guard.run).toContain('usable()');
expect(guard.run).toContain('[ ! -L "${c}" ] && [ -x "${c}" ]');
@@ -1096,8 +1152,8 @@ describe('ci failure patrol workflow', () => {
RUNNER_TEMP: runnerTemp,
GITHUB_ENV: githubEnv,
GITHUB_OUTPUT: join(dir, 'gh_output'),
RM_BIN: '/usr/bin/rm',
MKTEMP_BIN: '/usr/bin/mktemp',
RM_BIN: resolvedBin('rm'),
MKTEMP_BIN: resolvedBin('mktemp'),
GIT_CONFIG_GLOBAL: '/dev/null',
GIT_CONFIG_SYSTEM: '/dev/null',
},
@@ -1215,12 +1271,24 @@ describe('ci failure patrol workflow', () => {
.map((l) => l.trim())
.filter((l) => l !== '' && !l.startsWith('#'));
expect(stageLines).toEqual([
'if [ ! -f "${GITHUB_OUTPUT}" ]; then',
'echo "::error::step output file is not a regular file; refusing to append to it."',
'exit 1',
'fi',
'if [ "${RUNNER_ENVIRONMENT}" = \'self-hosted\' ]; then',
'echo "trusted_path=${SYSTEM_PATH}:${PATH}" >> "${GITHUB_OUTPUT}"',
'else',
'echo "trusted_path=${PATH}" >> "${GITHUB_OUTPUT}"',
'fi',
]);
// Both uncapped-before steps now carry a cap: a planted FIFO at
// $GITHUB_OUTPUT would otherwise block them until the JOB cap and the round
// would conclude cancelled, skipping the verdict step entirely.
const identity = classify.steps.find((step) => step.id === 'identity');
for (const step of [stage, identity]) {
expect(step['timeout-minutes'], step.id).toBeGreaterThan(0);
expect(step['timeout-minutes'], step.id).toBeLessThanOrEqual(2);
}
expect(stage.env.SYSTEM_PATH).toBe(
'${{ steps.guard.outputs.system_path }}',
);
@@ -1320,7 +1388,7 @@ describe('ci failure patrol workflow', () => {
// The planted directory wins an ambient PATH lookup.
PATH: `${planted}:${trusted}:${process.env.PATH}`,
GH_BIN: join(trusted, 'gh'),
MKTEMP_BIN: '/usr/bin/mktemp',
MKTEMP_BIN: resolvedBin('mktemp'),
GITHUB_TOKEN: 'SENTINEL-PAT-VALUE',
GITHUB_OUTPUT: outFile,
RUNNER_TEMP: root,
@@ -1372,6 +1440,11 @@ describe('ci failure patrol workflow', () => {
expect(checkout.env.GIT_CONFIG_SYSTEM).toBe(
'${{ steps.gitmeta.outputs.gitconfig }}',
);
// git layers env-provided config ABOVE both file scopes, so neutralising
// the files alone leaves core.hooksPath reachable through $GITHUB_ENV.
// Step env, not $GITHUB_ENV: the runner takes the last duplicated key.
expect(checkout.env.GIT_CONFIG_COUNT).toBe('0');
expect(checkout.env.GIT_CONFIG_PARAMETERS).toBe('');
// checkout spawns git by name, so it gets only verified directories.
expect(checkout.env.PATH).toBe('${{ steps.guard.outputs.system_path }}');
}
@@ -1430,7 +1503,7 @@ describe('ci failure patrol workflow', () => {
RUNNER_TEMP: runnerTemp,
GITHUB_ENV: envFile,
GITHUB_OUTPUT: join(root, 'gh_output'),
RM_BIN: '/usr/bin/rm',
RM_BIN: resolvedBin('rm'),
};
execFileSync('bash', ['-c', rebuild.run], {
env: stepEnv,
+39
View File
@@ -904,6 +904,45 @@ describe('ci flaky rerun patrol', () => {
expect(c.calls.filter((call) => call[0] === 'comment')).toHaveLength(1);
});
it('fails the round when a non-empty decision list matches no candidate', async () => {
// The hollow-green shape the workflow's verdict step cannot catch: validate
// sees a non-empty array and sets has_decisions=true, so the verdict step is
// skipped, act prints "no candidate" per entry and exits 0. The consumer has
// to close the loop. `{"pr":123,"action":"rerun"}` is exactly the shape a
// model emits when it ignores the five-field key.
const c = client();
await expect(
actOnDecisions(c, [target()], [{ pr: 123, action: 'rerun' }]),
).rejects.toThrow(/matched no scanned candidate/);
expect(c.calls).toEqual([]);
// A mistyped headSha is the same failure, not a per-decision skip.
const c2 = client();
await expect(
actOnDecisions(c2, [target()], [decision({ headSha: 'f'.repeat(40) })]),
).rejects.toThrow(/matched no scanned candidate/);
expect(c2.calls).toEqual([]);
// An empty list is still a legitimate no-op: the workflow's verdict step
// owns that case, and the consumer must not throw on it.
const c3 = client();
await expect(actOnDecisions(c3, [target()], [])).resolves.toBeUndefined();
expect(c3.calls).toEqual([]);
});
it('keeps a matched decision that failed per-target from failing the round', async () => {
// A candidate superseded between scan and act must not fail the round. It
// still counts as matched, because the decision addressed a real target.
const c = client({
async rerunFailedJobs() {
throw new Error('temporary API failure');
},
});
await expect(
actOnDecisions(c, [target()], [decision()]),
).resolves.toBeUndefined();
});
it('persists a zero-count reset marker after success', async () => {
const c = client({
async comments() {
@@ -11,6 +11,14 @@ import {
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { parse } from 'yaml';
// macOS ships rm at /bin/rm, so a hardcoded /usr/bin/rm exits 127 on the macOS
// scripts lane (which collects these files — the vitest exclude is win32-only).
// Resolve external tools the way the workflows themselves do.
const resolvedBin = (name) =>
execFileSync('bash', ['-c', `command -v ${name}`], {
encoding: 'utf8',
}).trim();
import { expect, it } from 'vitest';
// The qwen-triage entries are DERIVED, not hand-typed: every pool-routed job
@@ -113,8 +121,8 @@ it.each(jobs)(
// Steps that use the guard's verified base need those binaries and
// an output file; a real runner always supplies GITHUB_OUTPUT.
GITHUB_OUTPUT: join(root, 'gh-output'),
RM_BIN: '/usr/bin/rm',
MKTEMP_BIN: '/usr/bin/mktemp',
RM_BIN: resolvedBin('rm'),
MKTEMP_BIN: resolvedBin('mktemp'),
},
});