mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
fix(verify-pr): post flaky demotions on closed PRs and pin the blank class (#16135)
* fix(verify-pr): post flaky demotions on closed PRs and pin the blank class The closed-PR skip from #16050 also caught the flaky variants of the cancelled, failed-before-report and results-unavailable notices, and the report body of a round that wrote no report.md. Those render ❌ and, on a merged PR, name non-deterministic tests now on main, so they post unless this run's own report already carries the demotion. - publisher: the closed-PR skip leaves FLAKE_VERDICT=flaky alone, except beside this run's own report - publisher: the gates line says its tests are "not counted individually above", since each suite run now counts as one assertion (zh: 不逐条计入上面的断言) - checker tests: pin the blank class on their own (whitespace refused, U+0085 and U+FEFF accepted); before, only the shared parity table in the workflow suite caught a revert to trim() - SKILL: rewrap the over-long local-path line * fix(verify-pr): scope re-run advice to open PRs and pin the gate condition The results-unavailable notice can now land on a merged PR, where the verify lane refuses `@qwen-code /verify`: both variants now offer the re-run only while the PR is open. The flaky exemption keys on the gate verdict, which holds only because the gate step runs on a prepared round; a test now pins that condition. * fix(verify-pr): skip a flaky notice only beside a report that carries it On a closed PR, a flaky notice with no live status of its own to finalize is skipped beside this run's own report, as a duplicate of the demotion that report carries. That report can come from an earlier attempt whose gate did not diverge. A re-run attempt whose own live status is missing (that post is best-effort) then lost its ❌ to the run log, beside a report that never showed it. A flaky verdict now adds a verify-flaky marker to the header, and the closed-PR skip drops a flaky notice only when this run's own report carries that marker in its header. * test(verify-pr): pin the own-comment lookup's initializers The publish step runs under `set -u`, and a failed comment listing or bot identity lookup leaves OWN_ID, OWN_SUBSTANTIVE and OWN_FLAKY at their initial values. Dropping any of those three lines kept the suite green: no case made either lookup fail. On such a run the step would instead abort with "unbound variable" and post nothing. The upsert stub can now fail the listing or the identity lookup. Under either failure, a report posts fresh rather than patching a status it cannot see. A flaky notice on a closed PR also posts, because the lookup cannot tell whether this run's report already carries the demotion. Deleting any one initializer now fails the test on its own unbound variable, which the harness now prints with the failed exit status.
This commit is contained in:
1 parent
2800ea18da
commit
b9e20830f7
4 files changed
+199
-29
No files matched your search
@@ -5775,11 +5775,20 @@ jobs:
|
||||
PARTIAL_ZH='本次验证运行未正常结束,下列内容可能不完整甚至缺失。仅作为评审证据,**不构成评审、批准或 CI 检查**。'
|
||||
|
||||
# Every body opens with a header: the verify marker, this run's
|
||||
# marker and, on a body that carries findings, the substantive
|
||||
# marker, ended by a blank line. Every lookup reads markers only
|
||||
# marker, the flaky marker when the gate demoted the run and, on a
|
||||
# body that carries findings, the substantive marker, ended by a
|
||||
# blank line. Every lookup reads markers only
|
||||
# from a header, so it is started once, here, and each branch below
|
||||
# appends to it.
|
||||
printf '%s\n%s\n' '<!-- qwen-triage:verify -->' "$RUN_MARKER" > "$BODY_FILE"
|
||||
# Every branch a flaky verdict can reach demotes on it (the gate step
|
||||
# runs only on a prepared round it was asked to verify), so the
|
||||
# header records it once, here: a later attempt of this run can
|
||||
# then tell whether the report it would post beside carries it.
|
||||
FLAKY_MARKER='<!-- qwen-triage:verify-flaky -->'
|
||||
if [ "${FLAKE_VERDICT:-}" = 'flaky' ]; then
|
||||
printf '%s\n' "$FLAKY_MARKER" >> "$BODY_FILE"
|
||||
fi
|
||||
|
||||
# The gate verdict travels via job outputs: it survives a later
|
||||
# agent-step failure, a job timeout, and a cancellation. A
|
||||
@@ -5863,16 +5872,16 @@ jobs:
|
||||
if [ "${FLAKE_VERDICT:-}" = 'flaky' ]; then
|
||||
printf '**Sandboxed verification: ❌ not passed — non-deterministic tests (flakiness gate); result artifact unavailable** - [workflow run](%s)\n\n' "$RUN_URL"
|
||||
printf 'Flakiness gate: ❌ %s\n\n' "${FLAKE_SUMMARY:-changed test files returned different results across identical re-runs}"
|
||||
printf 'The verification ran and its deterministic flakiness gate observed run-to-run divergence, but the result artifact (report and per-round matrix) could not be retrieved for publishing. The run log still has the full output; re-run `@qwen-code /verify` for a fresh report.\n\n'
|
||||
printf 'The verification ran and its deterministic flakiness gate observed run-to-run divergence, but the result artifact (report and per-round matrix) could not be retrieved for publishing. The run log still has the full output; while the PR is open, re-run `@qwen-code /verify` for a fresh report.\n\n'
|
||||
printf '<details>\n<summary>中文 — 判定:❌ 不通过 · 测试结果不确定(抖动门);结果产物不可用</summary>\n\n'
|
||||
printf '抖动门:❌ %s\n\n' "${FLAKE_SUMMARY:-changed test files returned different results across identical re-runs}"
|
||||
printf '验证已执行,其确定性抖动门观测到轮间分歧,但结果产物(报告与各轮矩阵)未能取回用于发布。运行日志中仍有完整输出;如需完整报告请重新运行 `@qwen-code /verify`。\n\n'
|
||||
printf '验证已执行,其确定性抖动门观测到轮间分歧,但结果产物(报告与各轮矩阵)未能取回用于发布。运行日志中仍有完整输出;PR 未关闭时,如需完整报告请重新运行 `@qwen-code /verify`。\n\n'
|
||||
printf '</details>\n\n'
|
||||
else
|
||||
printf '**Sandboxed verification: ⚠️ incomplete — results unavailable** - [workflow run](%s)\n\n' "$RUN_URL"
|
||||
printf 'The verification ran, but its result artifact could not be retrieved for publishing, so there is nothing to report here. The run log still has the agent output; re-run `@qwen-code /verify` for a fresh report.\n\n'
|
||||
printf 'The verification ran, but its result artifact could not be retrieved for publishing, so there is nothing to report here. The run log still has the agent output; while the PR is open, re-run `@qwen-code /verify` for a fresh report.\n\n'
|
||||
printf '<details>\n<summary>中文 — 判定:⚠️ 未完成 · 结果不可用</summary>\n\n'
|
||||
printf '验证已执行,但结果产物未能取回用于发布,因此此处没有可报告的内容。运行日志中仍有 agent 输出;如需完整报告请重新运行 `@qwen-code /verify`。\n\n'
|
||||
printf '验证已执行,但结果产物未能取回用于发布,因此此处没有可报告的内容。运行日志中仍有 agent 输出;PR 未关闭时,如需完整报告请重新运行 `@qwen-code /verify`。\n\n'
|
||||
printf '</details>\n\n'
|
||||
fi
|
||||
printf '%s\n' '— _Qwen Code · sandboxed verification_'
|
||||
@@ -6033,8 +6042,8 @@ jobs:
|
||||
G_SKIP_CLAUSE=" · ${G_SKIP} skipped"
|
||||
G_SKIP_CLAUSE_ZH=" · ${G_SKIP} 跳过"
|
||||
fi
|
||||
GATES_LINE="Targeted gates (existing tests, not counted above): ${G_PASS} passed · ${G_FAIL} failed${G_SKIP_CLAUSE} across ${G_SUITES} suite(s)"
|
||||
GATES_LINE_ZH="定向门禁(已有测试,不计入上面的断言):${G_PASS} 通过 · ${G_FAIL} 失败${G_SKIP_CLAUSE_ZH},共 ${G_SUITES} 个套件"
|
||||
GATES_LINE="Targeted gates (existing tests, not counted individually above): ${G_PASS} passed · ${G_FAIL} failed${G_SKIP_CLAUSE} across ${G_SUITES} suite(s)"
|
||||
GATES_LINE_ZH="定向门禁(已有测试,不逐条计入上面的断言):${G_PASS} 通过 · ${G_FAIL} 失败${G_SKIP_CLAUSE_ZH},共 ${G_SUITES} 个套件"
|
||||
fi
|
||||
else
|
||||
echo "::warning::assertions.json missing or inconsistent; not treating it as evidence."
|
||||
@@ -6310,6 +6319,7 @@ jobs:
|
||||
# run produced, can quote one further down.
|
||||
OWN_ID=''
|
||||
OWN_SUBSTANTIVE=''
|
||||
OWN_FLAKY=''
|
||||
if ! BOT_LOGIN="$(gh api user --jq '.login')" || [ -z "$BOT_LOGIN" ]; then
|
||||
echo "::warning::Could not resolve the bot identity; posting a fresh comment instead of reusing one."
|
||||
BOT_LOGIN=''
|
||||
@@ -6319,15 +6329,17 @@ jobs:
|
||||
--method GET \
|
||||
--paginate \
|
||||
-F per_page=100 \
|
||||
| jq -sr --arg bot "$BOT_LOGIN" --arg run "$RUN_MARKER" --arg sub "$SUBSTANTIVE_MARKER" \
|
||||
| jq -sr --arg bot "$BOT_LOGIN" --arg run "$RUN_MARKER" --arg sub "$SUBSTANTIVE_MARKER" --arg flaky "$FLAKY_MARKER" \
|
||||
'def header: split("\n\n")[0] | split("\n");
|
||||
[.[][] | select((.body | startswith("<!-- qwen-triage:verify -->")) and .user.login == $bot)
|
||||
| (.body | header) as $h | select(any($h[]; . == $run))
|
||||
| "\(.id)\t\(any($h[]; . == $sub))"]
|
||||
| "\(.id)\t\(any($h[]; . == $sub))\t\(any($h[]; . == $flaky))"]
|
||||
| last // empty'
|
||||
)"; then
|
||||
OWN_ID="${OWN_META%%$'\t'*}"
|
||||
OWN_SUBSTANTIVE="${OWN_META#*$'\t'}"
|
||||
OWN_META="${OWN_META#*$'\t'}"
|
||||
OWN_SUBSTANTIVE="${OWN_META%%$'\t'*}"
|
||||
OWN_FLAKY="${OWN_META#*$'\t'}"
|
||||
elif [ -n "$BOT_LOGIN" ]; then
|
||||
echo "::warning::Failed to look up this run's comment; will create a new one."
|
||||
fi
|
||||
@@ -6342,9 +6354,14 @@ jobs:
|
||||
OWN_ID=''
|
||||
fi
|
||||
# A round can queue for hours: a notice with no status of its own
|
||||
# to finalize is noise on a PR that closed meanwhile. A failed
|
||||
# lookup posts.
|
||||
# to finalize is noise on a PR that closed meanwhile. A flaky
|
||||
# demotion still posts, since on a merged PR it names tests now on
|
||||
# main, unless this run's own report already carries it. An
|
||||
# attempt whose own live status is missing (that post is
|
||||
# best-effort) finds an earlier attempt's report instead, which
|
||||
# can predate the verdict. A failed lookup posts.
|
||||
if [ -z "$OWN_ID" ] && [ "$BODY_SUBSTANTIVE" != true ] &&
|
||||
{ [ "${FLAKE_VERDICT:-}" != flaky ] || [ "$OWN_FLAKY" = true ]; } &&
|
||||
[ "$(gh api "repos/$GITHUB_REPOSITORY/pulls/$PR_NUMBER" --jq '.state' 2>/dev/null || true)" = closed ]; then
|
||||
echo "PR #${PR_NUMBER} closed before this round published; notice not posted (verdict=${VERDICT})." >> "$GITHUB_STEP_SUMMARY"
|
||||
exit 0
|
||||
|
||||
@@ -705,9 +705,10 @@ and clear its
|
||||
errors and its verdict-shape and footer warnings before publishing (a
|
||||
missing footer is legitimate only when the model could not be resolved),
|
||||
act on a gates-size warning by recording the suites it ran under `gates`
|
||||
(with none run, say in the report what the count holds), then state the verdict word from `verdict.txt` verbatim. A published
|
||||
filename is spent: raw.githubusercontent serves a URL from cache, so
|
||||
overwriting `02-matrix.png` leaves every earlier link showing the old image.
|
||||
(with none run, say in the report what the count holds), then state the
|
||||
verdict word from `verdict.txt` verbatim. A published filename is spent:
|
||||
raw.githubusercontent serves a URL from cache, so overwriting
|
||||
`02-matrix.png` leaves every earlier link showing the old image.
|
||||
Republish as `02-matrix-v2.png`, update the body and every comment that quotes
|
||||
it, then read the file back at that ref (`gh api
|
||||
repos/<assets-repo>/contents/<path>?ref=<branch>`) to confirm its size and
|
||||
|
||||
@@ -4473,7 +4473,8 @@ describe('qwen-triage verify hardening round 2', () => {
|
||||
try {
|
||||
// The stub records which HTTP verb the publisher used and against
|
||||
// which comment id, keeps the body it sent, and serves a comments
|
||||
// listing from a fixture.
|
||||
// listing from a fixture. STUB_FAIL_LIST / STUB_FAIL_USER fail the
|
||||
// listing or the bot-identity lookup.
|
||||
writeFileSync(
|
||||
join(dir, 'gh'),
|
||||
[
|
||||
@@ -4481,10 +4482,10 @@ describe('qwen-triage verify hardening round 2', () => {
|
||||
'all="$*"',
|
||||
'for a in "$@"; do case "$a" in body=@*) cp "${a#body=@}" "$BODY_OUT";; esac; done',
|
||||
'case "$all" in',
|
||||
' *"api user"*|*" user "*) echo qwen-code-ci-bot ;;',
|
||||
' *"api user"*|*" user "*) [ -n "${STUB_FAIL_USER:-}" ] && exit 1; echo qwen-code-ci-bot ;;',
|
||||
' */pulls/7999*) [ "$STUB_PR_STATE" = fail ] && exit 1; echo "${STUB_PR_STATE:-open}" ;;',
|
||||
' *"-X PATCH"*) echo "PATCH $all" >> "$CALLS" ;;',
|
||||
' *comments*--method*GET*) cat "$LISTING" ;;',
|
||||
' *comments*--method*GET*) [ -n "${STUB_FAIL_LIST:-}" ] && exit 1; cat "$LISTING" ;;',
|
||||
' *issues/*/comments*) echo "POST $all" >> "$CALLS" ;;',
|
||||
'esac',
|
||||
'exit 0',
|
||||
@@ -4534,7 +4535,7 @@ describe('qwen-triage verify hardening round 2', () => {
|
||||
...env,
|
||||
},
|
||||
});
|
||||
expect(res.status).toBe(0);
|
||||
expect(res.status, res.stderr).toBe(0);
|
||||
return readFileSync(calls, 'utf8');
|
||||
};
|
||||
|
||||
@@ -4581,6 +4582,17 @@ describe('qwen-triage verify hardening round 2', () => {
|
||||
expect(foreign).toContain('POST');
|
||||
expect(foreign).not.toContain('/issues/comments/777');
|
||||
|
||||
// A lookup that cannot read the listing or the bot identity finds no
|
||||
// comment of this run's, so the report is posted fresh. The step runs
|
||||
// under its own `set -u`: each value the lookup would set needs its
|
||||
// initializer, or this path aborts with nothing posted.
|
||||
const blindLookups = [{ STUB_FAIL_LIST: '1' }, { STUB_FAIL_USER: '1' }];
|
||||
for (const env of blindLookups) {
|
||||
const blind = run([status(555, 1)], env);
|
||||
expect(blind).toContain('POST');
|
||||
expect(blind).not.toContain('/issues/comments/555');
|
||||
}
|
||||
|
||||
// Publishers queue, so the newest marker comment can be another
|
||||
// round's: its live status is that round's to replace, and an earlier
|
||||
// round's report is kept. This real report is posted fresh beside
|
||||
@@ -4675,6 +4687,7 @@ describe('qwen-triage verify hardening round 2', () => {
|
||||
for (const env of [
|
||||
{ DOWNLOAD_OUTCOME: 'failure' },
|
||||
{ VERIFY_RESULT: 'cancelled' },
|
||||
{ VERIFY_RESULT: 'failure' },
|
||||
{ VERDICT: 'infra-error', PREPARE_FAILURE_PHASE: 'install' },
|
||||
]) {
|
||||
writeFileSync(summary, '');
|
||||
@@ -4682,6 +4695,9 @@ describe('qwen-triage verify hardening round 2', () => {
|
||||
expect(readFileSync(summary, 'utf8')).toContain(
|
||||
'PR #7999 closed before this round published; notice not posted',
|
||||
);
|
||||
// Only a flaky verdict is exempt; the gate's other verdicts are
|
||||
// still notices.
|
||||
expect(run([], { ...closed, ...env, FLAKE_VERDICT: 'pass' })).toBe('');
|
||||
expect(run([ownReport], { ...closed, ...env })).toBe('');
|
||||
const finalized = run([status(555, 1)], { ...closed, ...env });
|
||||
expect(finalized).toContain('/issues/comments/555');
|
||||
@@ -4693,6 +4709,90 @@ describe('qwen-triage verify hardening round 2', () => {
|
||||
expect(
|
||||
run([], { ...closed, VERDICT: 'fail', PREPARE_FAILURE_PHASE: 'build' }),
|
||||
).toContain('POST');
|
||||
// A flaky demotion is a finding, not noise: on a merged PR it names
|
||||
// tests now on main. Each body that carries one without the
|
||||
// substantive marker posts, unless this run's own report already
|
||||
// carries the demotion. Without a live status of its own, a later
|
||||
// attempt finds an earlier attempt's report, which can predate the
|
||||
// verdict; the marker counts only in that report's header.
|
||||
const flaky = { ...closed, FLAKE_VERDICT: 'flaky' };
|
||||
const ownFlakyReport = {
|
||||
...ownReport,
|
||||
body: `${M}\n<!-- qwen-triage:verify-run=1 -->\n<!-- qwen-triage:verify-flaky -->\n<!-- qwen-triage:verify-substantive -->\n\nreport`,
|
||||
};
|
||||
for (const [env, headline] of [
|
||||
[{ DOWNLOAD_OUTCOME: 'failure' }, 'result artifact unavailable'],
|
||||
[{ VERIFY_RESULT: 'cancelled' }, 'job cancelled before the report'],
|
||||
[{ VERIFY_RESULT: 'failure' }, 'the job then failed before the report'],
|
||||
]) {
|
||||
writeFileSync(bodyOut, '');
|
||||
expect(run([], { ...flaky, ...env })).toContain('POST');
|
||||
expect(readFileSync(bodyOut, 'utf8')).toContain(
|
||||
`❌ not passed — non-deterministic tests (flakiness gate); ${headline}**`,
|
||||
);
|
||||
expect(run([ownFlakyReport], { ...flaky, ...env })).toBe('');
|
||||
// Unable to read that report, the lookup cannot tell whether it
|
||||
// carries the demotion, so the notice posts.
|
||||
for (const blind of blindLookups) {
|
||||
expect(
|
||||
run([ownFlakyReport], { ...flaky, ...env, ...blind }),
|
||||
).toContain('POST');
|
||||
}
|
||||
for (const earlier of [
|
||||
ownReport,
|
||||
{
|
||||
...ownReport,
|
||||
body: `${ownReport.body}\n<!-- qwen-triage:verify-flaky -->`,
|
||||
},
|
||||
]) {
|
||||
const beside = run([earlier], { ...flaky, ...env });
|
||||
expect(beside).toContain('POST');
|
||||
expect(beside).not.toContain('/issues/comments/555');
|
||||
}
|
||||
// A full re-run posts its own live status first, and that status
|
||||
// is what this attempt finalizes.
|
||||
const rerun = run([ownReport, status(600, 1)], { ...flaky, ...env });
|
||||
expect(rerun).toContain('/issues/comments/600');
|
||||
expect(rerun).not.toContain('POST');
|
||||
}
|
||||
// The exemption keys on the gate verdict, not on the rendered body.
|
||||
// The skipped, n/a and prepare-failure bodies stay ⚠️ under `flaky`;
|
||||
// only the gate step's condition keeps them from meeting one.
|
||||
expect(
|
||||
stepIn('verify', 'Flakiness gate: re-run changed test files'),
|
||||
).toContain(
|
||||
`if: "steps.pr.outputs.decision == 'run' && steps.prepare.outputs.verdict == ''"`,
|
||||
);
|
||||
// The verify lane runs only on an open PR, and a closed PR is where
|
||||
// a results-unavailable notice can now land: either variant scopes
|
||||
// its re-run advice to an open PR.
|
||||
for (const [comments, env] of [
|
||||
[[], { ...flaky, DOWNLOAD_OUTCOME: 'failure' }],
|
||||
[[status(555, 1)], { ...closed, DOWNLOAD_OUTCOME: 'failure' }],
|
||||
]) {
|
||||
writeFileSync(bodyOut, '');
|
||||
run(comments, env);
|
||||
const body = readFileSync(bodyOut, 'utf8');
|
||||
expect(body).toContain(
|
||||
'while the PR is open, re-run `@qwen-code /verify` for a fresh report',
|
||||
);
|
||||
expect(body).toContain(
|
||||
'PR 未关闭时,如需完整报告请重新运行 `@qwen-code /verify`',
|
||||
);
|
||||
}
|
||||
// A round whose agent wrote no report.md gets the report body
|
||||
// without the marker: a notice, posted only when the gate demoted it.
|
||||
rmSync(join(art, 'report.md'));
|
||||
try {
|
||||
expect(run([], closed)).toBe('');
|
||||
writeFileSync(bodyOut, '');
|
||||
expect(run([], flaky)).toContain('POST');
|
||||
expect(readFileSync(bodyOut, 'utf8')).toContain(
|
||||
'❌ not passed — non-deterministic tests (flakiness gate)**',
|
||||
);
|
||||
} finally {
|
||||
writeFileSync(join(art, 'report.md'), '## real report\n');
|
||||
}
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
@@ -6350,8 +6450,9 @@ describe('qwen-triage verify publish fidelity', () => {
|
||||
// A re-run of the publisher finds what it already published by the run
|
||||
// marker, and a notice must not replace findings, so every terminal body
|
||||
// opens with a header: the verify marker both lookups select on, this
|
||||
// run's marker, and the substantive marker when it carries findings. The
|
||||
// lookups read markers only from the lines before the first blank line.
|
||||
// run's marker, the flaky marker when the gate demoted the run, and the
|
||||
// substantive marker when it carries findings. The lookups read markers
|
||||
// only from the lines before the first blank line.
|
||||
it('opens every terminal body with its header', () => {
|
||||
const dir = fixture();
|
||||
try {
|
||||
@@ -6360,6 +6461,14 @@ describe('qwen-triage verify publish fidelity', () => {
|
||||
'<!-- qwen-triage:verify-run=41 -->',
|
||||
];
|
||||
const strong = [...weak, '<!-- qwen-triage:verify-substantive -->'];
|
||||
// Every body a flaky verdict can reach is demoted, and its header says
|
||||
// so ahead of the substantive marker.
|
||||
const flakyMark = '<!-- qwen-triage:verify-flaky -->';
|
||||
const flakyWeak = [...weak, flakyMark];
|
||||
const flakyStrong = [
|
||||
...flakyWeak,
|
||||
'<!-- qwen-triage:verify-substantive -->',
|
||||
];
|
||||
const variants = [
|
||||
['real report', strong, { AGENT_VERDICT: 'merge-ready' }],
|
||||
['incomplete — cancelled', weak, { VERIFY_RESULT: 'cancelled' }],
|
||||
@@ -6385,6 +6494,31 @@ describe('qwen-triage verify publish fidelity', () => {
|
||||
weak,
|
||||
{ VERDICT: 'infra-error', PREPARE_FAILURE_PHASE: 'install' },
|
||||
],
|
||||
[
|
||||
'non-deterministic tests (flakiness gate)**',
|
||||
flakyStrong,
|
||||
{ AGENT_VERDICT: 'merge-ready', FLAKE_VERDICT: 'flaky' },
|
||||
],
|
||||
[
|
||||
'job cancelled before the report',
|
||||
flakyWeak,
|
||||
{ VERIFY_RESULT: 'cancelled', FLAKE_VERDICT: 'flaky' },
|
||||
],
|
||||
[
|
||||
'the job then failed before the report',
|
||||
flakyWeak,
|
||||
{ VERIFY_RESULT: 'failure', VERDICT: '', FLAKE_VERDICT: 'flaky' },
|
||||
],
|
||||
[
|
||||
'result artifact unavailable',
|
||||
flakyWeak,
|
||||
{ DOWNLOAD_OUTCOME: 'failure', FLAKE_VERDICT: 'flaky' },
|
||||
],
|
||||
[
|
||||
'real report',
|
||||
strong,
|
||||
{ AGENT_VERDICT: 'merge-ready', FLAKE_VERDICT: 'pass' },
|
||||
],
|
||||
];
|
||||
variants.forEach(([shown, header, env], i) => {
|
||||
const body = render(dir, {
|
||||
@@ -6403,6 +6537,12 @@ describe('qwen-triage verify publish fidelity', () => {
|
||||
});
|
||||
expect(reportless).toContain('No report.md was found');
|
||||
expect(reportless.split('\n\n')[0].split('\n')).toEqual(weak);
|
||||
const flakyReportless = render(dir, {
|
||||
NAME: 'hdr-noreport-flaky',
|
||||
GITHUB_RUN_ID: '41',
|
||||
FLAKE_VERDICT: 'flaky',
|
||||
});
|
||||
expect(flakyReportless.split('\n\n')[0].split('\n')).toEqual(flakyWeak);
|
||||
|
||||
// Written once, before the branches, which only append: a new branch
|
||||
// cannot open its body without it.
|
||||
@@ -6650,11 +6790,11 @@ describe('qwen-triage verify publish fidelity', () => {
|
||||
'Scripted assertions: 10 passed · 0 failed · 10 total',
|
||||
);
|
||||
expect(body).toContain(
|
||||
'Targeted gates (existing tests, not counted above): 1909 passed · 2 failed · 1 skipped across 2 suite(s)',
|
||||
'Targeted gates (existing tests, not counted individually above): 1909 passed · 2 failed · 1 skipped across 2 suite(s)',
|
||||
);
|
||||
expect(
|
||||
body.indexOf(
|
||||
'定向门禁(已有测试,不计入上面的断言):1909 通过 · 2 失败 · 1 跳过,共 2 个套件',
|
||||
'定向门禁(已有测试,不逐条计入上面的断言):1909 通过 · 2 失败 · 1 跳过,共 2 个套件',
|
||||
),
|
||||
).toBeGreaterThan(body.indexOf('<details>'));
|
||||
expect(body).not.toContain('@everyone');
|
||||
@@ -6677,10 +6817,10 @@ describe('qwen-triage verify publish fidelity', () => {
|
||||
gates: [{ suite: 'cli', pass: 1292, fail: 0 }],
|
||||
});
|
||||
expect(noSkip).toContain(
|
||||
'Targeted gates (existing tests, not counted above): 1292 passed · 0 failed across 1 suite(s)',
|
||||
'Targeted gates (existing tests, not counted individually above): 1292 passed · 0 failed across 1 suite(s)',
|
||||
);
|
||||
expect(noSkip).toContain(
|
||||
'定向门禁(已有测试,不计入上面的断言):1292 通过 · 0 失败,共 1 个套件',
|
||||
'定向门禁(已有测试,不逐条计入上面的断言):1292 通过 · 0 失败,共 1 个套件',
|
||||
);
|
||||
expect(noSkip).not.toContain('0 skipped');
|
||||
expect(noSkip).not.toContain('Gate totals do not decide the verdict');
|
||||
@@ -6695,10 +6835,10 @@ describe('qwen-triage verify publish fidelity', () => {
|
||||
],
|
||||
});
|
||||
expect(mixed).toContain(
|
||||
'Targeted gates (existing tests, not counted above): 1909 passed · 0 failed across 2 suite(s)',
|
||||
'Targeted gates (existing tests, not counted individually above): 1909 passed · 0 failed across 2 suite(s)',
|
||||
);
|
||||
expect(mixed).toContain(
|
||||
'定向门禁(已有测试,不计入上面的断言):1909 通过 · 0 失败,共 2 个套件',
|
||||
'定向门禁(已有测试,不逐条计入上面的断言):1909 通过 · 0 失败,共 2 个套件',
|
||||
);
|
||||
expect(mixed).not.toContain('12 skipped');
|
||||
|
||||
@@ -6709,7 +6849,7 @@ describe('qwen-triage verify publish fidelity', () => {
|
||||
gates: [{ suite: 'cli', pass: 1292, fail: 0, skipped: 0 }],
|
||||
});
|
||||
expect(zeroSkip).toContain(
|
||||
'Targeted gates (existing tests, not counted above): 1292 passed · 0 failed · 0 skipped across 1 suite(s)',
|
||||
'Targeted gates (existing tests, not counted individually above): 1292 passed · 0 failed · 0 skipped across 1 suite(s)',
|
||||
);
|
||||
|
||||
// The advisory is gated on a trusted verdict: an untrusted run with
|
||||
|
||||
@@ -354,6 +354,18 @@ describe('gates', () => {
|
||||
expect(raw('9007199254740991').errors).toEqual([]);
|
||||
});
|
||||
|
||||
it('calls a suite blank only when it is JSON whitespace', () => {
|
||||
// The publisher's jq tests the same class. trim() also calls U+FEFF
|
||||
// blank and jq's \S calls U+0085 blank, so with either one here the
|
||||
// checker would refuse a suite the publisher renders.
|
||||
expect(
|
||||
withGates([{ suite: ' \t\n\r', pass: 1, fail: 0 }]).errors.join('\n'),
|
||||
).toMatch(GATES_ERROR);
|
||||
for (const suite of ['\u0085', '\ufeff']) {
|
||||
expect(withGates([{ suite, pass: 1, fail: 0 }]).errors).toEqual([]);
|
||||
}
|
||||
});
|
||||
|
||||
it('warns when a suite-sized total comes with no gate results', () => {
|
||||
const big = check({
|
||||
assertions: JSON.stringify({ pass: 1001, fail: 0, total: 1001 }),
|
||||
|
||||
Reference in new issue
Block a user