QwenLM/qwen-code
1
0 Fork 0
mirror of https://github.com/QwenLM/qwen-code.git synced 2026-11-07 06:19:43 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity

fix(verify-pr): keep each verify round in its own comment (#15988)

Browse Source 
* fix(verify-pr): keep each verify round in its own comment

Verify jobs are serialized per PR, but their publishers queue on hosted
runners, so a round's live status is often still "running" when the
next round's verify job starts. The status step took that comment over
and both publishers then wrote into it, so one round's report replaced
the other's. A cancelled pending run's notice could likewise replace
another round's live status.

The status comment now carries a run marker. The status step reuses a
live status only when it is this run's own (a re-run attempt), and the
publisher writes only into that status, posting fresh otherwise.
WEAK_BODY no longer decides anything and is removed.

Also, from the #15966 follow-ups: evidence images are collected from
the round the comment publishes instead of from every round, oldest
first, and the pre-agent sweep comment no longer describes the
head -1 ordering.

* fix(verify-pr): let a re-run publisher find its own comment

The publisher looked only for this run's live status, but the body it
publishes carries neither the running marker nor the run marker. A
publish job re-run after it had posted therefore found nothing and
posted the report a second time. Every terminal body now keeps the run
marker right after the leading verify marker, and the publisher matches
its own run's comment whether it is still the live status or already
the report. The status step still reuses only a live status.

Tests now pin the status body's leading markers in order, use a re-run
attempt number distinct from the run id, check the run marker in every
terminal body, and check that the evidence skip count comes from the
published round.

* fix(verify-pr): never let a notice replace a published report

A publish job re-run on its own downloads its own attempt's artifact,
which the verify job never uploaded, so it always renders "results
unavailable". Since the previous commit matched this run's comment
whatever it held, that notice replaced the round's report. The lookup
now also reports whether this run's comment carries findings, and a
body without findings is posted beside it instead. A notice may still
replace this run's live status or its earlier notice.

Every body now opens with a header written once, before the branches:
the verify marker, the run marker and, when the body carries findings,
the substantive marker, ended by a blank line. The build-failure body's
substantive marker moves up into it. Both lookups that choose a comment
to write into read markers only from the header, so a marker quoted
further down a body is text.

Tests pin the notice/report gate in both directions, the header of every
terminal body and that it is written once, header-only matching in both
lookups, and that the status step does not reuse this run's report.

* fix(verify-pr): read the substantive marker from the header for the snapshot

The previous-round snapshot was the last lookup still matching its
marker anywhere in a body, so a newer notice quoting the substantive
marker below its header would have been snapshotted in place of the
newest real report. It now reads the header like the other lookups.
Build-failure reports posted before the header change carry the marker
below their header and are skipped by the snapshot during the
transition.

Tests add that notice shape to the snapshot case, and render the
reportless body to check it carries no substantive marker.
This commit is contained in:
Shaojin Wen authored and GitHub committed 2026-10-31 18:56:36 +08:00
1 parent 39fd1b45ba
commit d811674c55
2 files changed
+527 -193
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/qwen-triage.yml
+128 -100
View File
@@ -2552,34 +2552,42 @@ jobs:
exit 0
fi
fi
# Status comment with the live run link, upserted by marker so
# re-runs reuse one comment; publish-verify rewrites the same
# comment with the final report. Best-effort.
# Status comment with the live run link; publish-verify rewrites
# this same comment with the final report. Best-effort.
MARKER='<!-- qwen-triage:verify -->'
# A dedicated machine marker for the live status. Inferring the
# lifecycle state from prose would misread a REPORT that merely
# quotes the running sentence as a live status — and the next run
# would then overwrite that report.
STATUS_MARKER='<!-- qwen-triage:verify-state=running -->'
# Which run the live status belongs to. Verify jobs are serialized
# per PR, but their publishers queue on hosted runners, so the
# previous round's status is often still "running" here. Taking it
# over would put two rounds into one comment, and the later report
# would overwrite the earlier one. A run reuses only its own status
# (a re-run attempt keeps the run id).
RUN_MARKER="<!-- qwen-triage:verify-run=${GITHUB_RUN_ID} -->"
# Bodies that carry findings mark themselves. Selecting "newest
# comment without the running marker" would pick a cancelled or
# infra notice, silently dropping the last real round's findings.
SUBSTANTIVE_MARKER='<!-- qwen-triage:verify-substantive -->'
printf -v BODY '%s\n%s\n\n%s\n\n%s' \
printf -v BODY '%s\n%s\n%s\n\n%s\n\n%s' \
"$MARKER" \
"$STATUS_MARKER" \
"$RUN_MARKER" \
"🔬 **Sandboxed verification is running** — [watch live progress]($RUN_URL). The report will be posted here when the run completes." \
"🔬 **沙箱验证正在运行** —— [查看实时进度]($RUN_URL)。运行结束后验证报告会发布在这里。"
# Marker lookup, shared with publish-verify: only BOT-OWNED
# comments whose body STARTS with the marker count. Any user can
# paste the marker into a comment of their own; matching on
# `contains` anywhere would make the bot try to PATCH a stranger's
# comment (403) and strand the run. `stale` marks a marker comment
# that is a previous round's terminal report rather than a live
# status.
# comment (403) and strand the run.
# Fail CLOSED on an identity lookup failure: an empty bot login
# used to widen the filter to every user's comments, so one
# transient API error could point the bot at an attacker's comment.
# Markers count only in the header, the lines before the first
# blank line, where they are written: a report or a notice can
# quote them further down.
if ! BOT_LOGIN="$(gh api user --jq '.login')" || [ -z "$BOT_LOGIN" ]; then
echo "::warning::Could not resolve the bot identity; posting a fresh status comment instead of reusing one."
BOT_LOGIN=''
@@ -2589,41 +2597,38 @@ jobs:
EXISTING_META="$(
gh api "repos/$GITHUB_REPOSITORY/issues/$PR_NUMBER/comments" \
--method GET --paginate -F per_page=100 |
jq -rs --arg m "$MARKER" --arg s "$STATUS_MARKER" --arg sub "$SUBSTANTIVE_MARKER" --arg bot "$BOT_LOGIN" \
jq -rs --arg m "$MARKER" --arg s "$STATUS_MARKER" --arg run "$RUN_MARKER" --arg sub "$SUBSTANTIVE_MARKER" --arg bot "$BOT_LOGIN" \
'[.[][] | select((.body | startswith($m)) and .user.login == $bot)] as $mine
| ($mine | last) as $latest
| ($mine | map(select(.body | contains($sub))) | last) as $report
| if $latest == null then ""
else "\($latest.id)\t\($latest.body | contains($s))\t\($report.id // "")" end'
| ($mine | map(select(.body | split("\n\n")[0] | split("\n") | any(.[]; . == $s) and any(.[]; . == $run))) | last) as $own
| ($mine | map(select(.body | split("\n\n")[0] | split("\n") | any(.[]; . == $sub))) | last) as $report
| "\($own.id // "")\t\($report.id // "")"'
)" || EXISTING_META=''
fi
EXISTING_ID="${EXISTING_META%%$'\t'*}"
OWN_ID="${EXISTING_META%%$'\t'*}"
# Snapshot the newest SUBSTANTIVE report, not simply the newest
# marker comment: after a real report followed by a weak notice
# (cancelled / infra), snapshotting the notice would drop the prior
# round's unresolved findings from the follow-up table.
REPORT_ID="${EXISTING_META##*$'\t'}"
if [ -n "$REPORT_ID" ] && [ "$REPORT_ID" != "$EXISTING_META" ]; then
REPORT_ID="${EXISTING_META#*$'\t'}"
if [ -n "$REPORT_ID" ]; then
gh api "repos/$GITHUB_REPOSITORY/issues/comments/$REPORT_ID" \
--jq '.body' > "$RUNNER_TEMP/verify-context/previous-report.md" 2>/dev/null ||
rm -f "$RUNNER_TEMP/verify-context/previous-report.md"
fi
# Only reuse the marker comment when it is a live status: replacing
# a previous round's REPORT with "running" would destroy evidence
# that a cancelled or infra-failed re-run then never restores.
case "$EXISTING_META" in
*$'\t'true$'\t'*)
gh api --method PATCH \
"repos/$GITHUB_REPOSITORY/issues/comments/$EXISTING_ID" \
-f body="$BODY" >/dev/null ||
echo "::warning::Failed to update verify status comment; continuing." >&2
;;
*)
gh api "repos/$GITHUB_REPOSITORY/issues/$PR_NUMBER/comments" \
-f body="$BODY" >/dev/null ||
echo "::warning::Failed to post verify status comment; continuing." >&2
;;
esac
# Anything that is not this run's own live status gets a fresh
# comment: replacing a previous round's REPORT with "running" would
# destroy evidence that a cancelled or infra-failed re-run then
# never restores.
if [ -n "$OWN_ID" ]; then
gh api --method PATCH \
"repos/$GITHUB_REPOSITORY/issues/comments/$OWN_ID" \
-f body="$BODY" >/dev/null ||
echo "::warning::Failed to update verify status comment; continuing." >&2
else
gh api "repos/$GITHUB_REPOSITORY/issues/$PR_NUMBER/comments" \
-f body="$BODY" >/dev/null ||
echo "::warning::Failed to post verify status comment; continuing." >&2
fi
echo "decision=run" >> "$GITHUB_OUTPUT"
# Install before checkout so PR-controlled .npmrc cannot affect npm.
@@ -4247,10 +4252,11 @@ jobs:
CLI_VERSION="$(timeout 30s qwen --version 2>/dev/null | head -1 | tr -d '[:space:]' | cut -c1-64 || true)"
SKILL_HASH="$(timeout 30s git hash-object --no-filters .qwen/skills/verify-pr/SKILL.md 2>/dev/null | cut -c1-12 || true)"
# 3. Fake artifact dirs planted during the build: a zeroed
# timestamp sorts AHEAD of the agent's real dir at collection
# time. (A steered agent can still forge its own artifacts —
# that residual is why the report is advisory, never a review.)
# 3. Fake artifact dirs planted during the build: the publisher
# takes the round that sorts last, so a plant dated after the
# agent's real dir would headline the run. (A steered agent can
# still forge its own artifacts — that residual is why the
# report is advisory, never a review.)
find tmp -maxdepth 2 -type d -name '*-verify-*' -exec rm -rf {} + 2>/dev/null || true
# 4. A fresh agent home, created after the lifecycle processes are
# gone: qwen loads user-scope file commands from $HOME/.qwen, and
@@ -5187,8 +5193,9 @@ jobs:
# per-PR group here would let a second /verify cancel a completed run's
# pending publisher and drop its report entirely. Overlap between
# publishers is instead made safe by the marker discipline below
# (bot-owned + prefix match, live-status-only reuse, post-fresh
# fallback), which is a correctness property rather than a scheduling one.
# (bot-owned + prefix match, each run writes only into its own
# comment, post-fresh fallback), which is a correctness property rather
# than a scheduling one.
concurrency:
group: "${{ format('{0}-publish-verify-{1}', github.workflow, github.run_id) }}"
cancel-in-progress: false
@@ -5333,6 +5340,11 @@ jobs:
exit 0
fi
BODY_FILE="${RUNNER_TEMP:-/tmp}/verify-comment.md"
# Every body keeps the run marker the status step wrote, so this
# job, re-run after it published, finds its own comment again
# instead of posting the report twice.
RUN_MARKER="<!-- qwen-triage:verify-run=${GITHUB_RUN_ID} -->"
SUBSTANTIVE_MARKER='<!-- qwen-triage:verify-substantive -->'
# Set only from the published report's own verdict.txt below.
AGENT_VERDICT=''
@@ -5638,23 +5650,26 @@ jobs:
# find predicates enforce both), and any failure degrades to a
# text-only comment rather than blocking the report. The
# VERIFY_ASSETS_UPLOADER override exists as a test seam only.
# $1 is the round directory the comment publishes: its images
# illustrate its report, never another round's.
EVIDENCE_SECTION=''
collect_and_host_evidence() {
local imgs=() f base safe seen=' ' hosted=0 total=0 skipped=0
local round="${1:-}" imgs=() f base safe seen=' ' hosted=0 total=0 skipped=0
local prefix="pr-assets/verify/pr${PR_NUMBER}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT:-1}"
local stage_dir="${RUNNER_TEMP:-/tmp}/verify-assets-stage"
local uploader="${VERIFY_ASSETS_UPLOADER:-scripts/upload-aliyun-oss-assets.js}"
[ -n "$round" ] || return 0
# `|| true` on every find: the artifact download is
# continue-on-error, so verify-results may not exist at all, and
# a bare non-zero find under `pipefail` would abort the step
# before the missing-report notice and the final upsert.
total="$(find verify-results -type f -path '*/evidence/*.png' 2>/dev/null | wc -l | tr -d ' ' || true)"
total="$(find "$round" -type f -path '*/evidence/*.png' 2>/dev/null | wc -l | tr -d ' ' || true)"
[ -n "$total" ] && [ "$total" -gt 0 ] || return 0
# Byte-exact size cap: find's -2M unit rounds file sizes UP to
# whole MiB first, silently turning a documented 2 MB cap into a
# 1 MiB one; the c (bytes) unit does not round.
while IFS= read -r f; do imgs+=("$f"); done < <(
find verify-results -type f -path '*/evidence/*.png' -size -2097153c 2>/dev/null | sort | head -8
find "$round" -type f -path '*/evidence/*.png' -size -2097153c 2>/dev/null | sort | head -8
)
if [ "${#imgs[@]}" -eq 0 ]; then
EVIDENCE_SECTION="_${total} evidence image(s) were produced but none passed the hosting caps (8 images, ≤2 MB each); see the run artifacts._"$'\n\n'
@@ -5671,8 +5686,10 @@ jobs:
*) continue ;;
esac
[ -n "${safe%.png}" ] || continue
# Two artifact dirs can sanitize to the same name; the second
# copy would silently overwrite the first and render twice.
# Two images can sanitize to the same name (nested evidence/
# dirs, or names that differ only in stripped characters); the
# second copy would silently overwrite the first and render
# twice.
case "$seen" in *" $safe "*) continue ;; esac
# Extension is attacker-choosable, and it is the enforcement
# point for what OSS serves: ossutil derives the object's
@@ -5757,20 +5774,21 @@ jobs:
PARTIAL_EN='The verification run did not complete, so the phases below may be partial or missing entirely. **Advisory evidence for human reviewers — not a review, an approval, or a CI check.**'
PARTIAL_ZH='本次验证运行未正常结束，下列内容可能不完整甚至缺失。仅作为评审证据，**不构成评审、批准或 CI 检查**。'
# Weak terminal notices (cancelled / infra / skipped / n-a) carry no
# new evidence, so they must never overwrite a previous round's real
# report — they only replace this run's own "running" status (see
# the upsert below). Real outcomes (report, prepare-fail) upsert.
WEAK_BODY=false
# Every body opens with a header: the verify marker, this run's
# marker and, on a body that carries findings, the substantive
# marker, ended by a blank line. Every lookup reads markers only
# from a header, so it is started once, here, and each branch below
# appends to it.
printf '%s\n%s\n' '<!-- qwen-triage:verify -->' "$RUN_MARKER" > "$BODY_FILE"
# The gate verdict travels via job outputs: it survives a later
# agent-step failure, a job timeout, and a cancellation. A
# recorded `flaky` must demote the headline in EVERY terminal
# branch — these two used to post the neutral notice and silently
# drop the demotion the download-failure branch already honors.
if [ "${VERIFY_RESULT:-}" = "cancelled" ]; then
WEAK_BODY=true
{
printf '%s\n\n' '<!-- qwen-triage:verify -->'
printf '\n'
if [ "${FLAKE_VERDICT:-}" = 'flaky' ]; then
printf '**Sandboxed verification: ❌ not passed — non-deterministic tests (flakiness gate); job cancelled before the report** - [workflow run](%s)\n\n' "$RUN_URL"
printf 'Flakiness gate: ❌ %s\n\n' "${FLAKE_SUMMARY:-changed test files returned different results across identical re-runs}"
@@ -5787,11 +5805,10 @@ jobs:
printf '</details>\n\n'
fi
printf '%s\n' '— _Qwen Code · sandboxed verification_'
} > "$BODY_FILE"
} >> "$BODY_FILE"
elif [ "${VERIFY_RESULT:-}" != "success" ] || [ -z "${VERDICT:-}" ]; then
WEAK_BODY=true
{
printf '%s\n\n' '<!-- qwen-triage:verify -->'
printf '\n'
if [ "${FLAKE_VERDICT:-}" = 'flaky' ]; then
printf '**Sandboxed verification: ❌ not passed — non-deterministic tests (flakiness gate); the job then failed before the report** - [workflow run](%s)\n\n' "$RUN_URL"
printf 'Flakiness gate: ❌ %s\n\n' "${FLAKE_SUMMARY:-changed test files returned different results across identical re-runs}"
@@ -5808,14 +5825,13 @@ jobs:
printf '</details>\n\n'
fi
printf '%s\n' '— _Qwen Code · sandboxed verification_'
} > "$BODY_FILE"
} >> "$BODY_FILE"
elif [ "${VERDICT:-}" = "skipped" ] || [ "${VERDICT:-}" = "n/a" ]; then
# These outcomes deliberately upload nothing, so their download
# always "fails"; they must be answered before the
# download-failure branch or their real reason is unreachable.
WEAK_BODY=true
{
printf '%s\n\n' '<!-- qwen-triage:verify -->'
printf '\n'
if [ "${VERDICT:-}" = "skipped" ]; then
printf '**Sandboxed verification: ⚠️ not run — skipped** - [workflow run](%s)\n\n' "$RUN_URL"
printf 'Skipped because %s.\n\n' "${SKIP_REASON:-the PR was not in a verifiable state}"
@@ -5830,21 +5846,20 @@ jobs:
printf '</details>\n\n'
fi
printf '%s\n' '— _Qwen Code · sandboxed verification_'
} > "$BODY_FILE"
} >> "$BODY_FILE"
elif [ "${DOWNLOAD_OUTCOME:-success}" != "success" ]; then
# The verify job may have succeeded, but its artifact never
# arrived (the download step is continue-on-error). Without this
# branch the full-report path still runs and its scope paragraph
# claims the A/B, the harnesses and the gates were delivered —
# when nothing was.
WEAK_BODY=true
# The flake verdict travels via job outputs, independent of the
# artifact — the gate's demotion contract ("reported as not
# passed regardless of the agent verdict") must fire even when
# the download did not, or a flaky PR gets a neutral
# "results unavailable" notice instead of its ❌.
{
printf '%s\n\n' '<!-- qwen-triage:verify -->'
printf '\n'
if [ "${FLAKE_VERDICT:-}" = 'flaky' ]; then
printf '**Sandboxed verification: ❌ not passed — non-deterministic tests (flakiness gate); result artifact unavailable** - [workflow run](%s)\n\n' "$RUN_URL"
printf 'Flakiness gate: ❌ %s\n\n' "${FLAKE_SUMMARY:-changed test files returned different results across identical re-runs}"
@@ -5861,11 +5876,8 @@ jobs:
printf '</details>\n\n'
fi
printf '%s\n' '— _Qwen Code · sandboxed verification_'
} > "$BODY_FILE"
} >> "$BODY_FILE"
elif [ -n "${PREPARE_FAILURE_PHASE:-}" ]; then
# An infra-classified prepare failure says nothing about the PR,
# so it must not overwrite a previous round's real report.
[ "${VERDICT:-}" = 'infra-error' ] && WEAK_BODY=true
PREPARE_LOG="$(find verify-results -name 'prepare.log' 2>/dev/null | head -1 || true)"
# Initialised, not merely defaulted at use: an inherited value
# would otherwise tell a reader that a single-shot `npm run
@@ -5900,10 +5912,10 @@ jobs:
# mis-attribution this commit set out to remove, pointed the
# other way.
{
printf '%s\n\n' '<!-- qwen-triage:verify -->'
# Reachable: the prepare step reports infra-error when the
# registry was unreachable from the runner at failure time.
if [ "${VERDICT:-}" = 'infra-error' ]; then
printf '\n'
printf '**Sandboxed verification: ⚠️ incomplete — infrastructure failure** - [workflow run](%s)\n\n' "$RUN_URL"
printf '`%s` failed before any verification started, and `registry.npmjs.org` was unreachable from the runner at that moment — so this looks like an infrastructure incident rather than a problem with this PR. Re-running `@qwen-code /verify` may well succeed; the install log below is the place to confirm.\n\n' "$PREPARE_COMMAND"
printf '<details>\n<summary>中文 — 判定：⚠️ 未完成 · 基础设施故障</summary>\n\n'
@@ -5919,9 +5931,8 @@ jobs:
fi
emit_block 'Install/build log' "$PREPARE_LOG" 20000
printf '%s\n' '— _Qwen Code · sandboxed verification_'
} > "$BODY_FILE"
} >> "$BODY_FILE"
else
collect_and_host_evidence
# Pin the artifact-dir shape and sort: a bare -name search takes
# whatever directory find visits first, which is unordered. Rounds
# are named pr<n>-verify-<YYYYMMDD>-<HHMMSS>, so the last in sort
@@ -5930,13 +5941,15 @@ jobs:
# round (scratch/) sorts past the round's own and would stand in
# for it.
REPORT="$(find verify-results -mindepth 2 -maxdepth 2 -type f -path '*-verify-*/report.md' 2>/dev/null | sort | tail -1 || true)"
# The report's own directory is the round: its counts and its
# verdict word headline it, never another directory's. This is
# the only reader of verdict.txt, which untrusted PR code wrote.
# The report's own directory is the round: its counts, its
# verdict word and its images go with it, never another
# directory's. This is the only reader of verdict.txt, which
# untrusted PR code wrote.
if [ -n "$REPORT" ]; then
ASSERTIONS_FILE="$(dirname "$REPORT")/assertions.json"
ROUND_DIR="$(dirname "$REPORT")"
ASSERTIONS_FILE="$ROUND_DIR/assertions.json"
[ -f "$ASSERTIONS_FILE" ] || ASSERTIONS_FILE=''
AGENT_VERDICT="$(head -c 4096 "$(dirname "$REPORT")/verdict.txt" 2>/dev/null | tr -d '[:space:]' | cut -c1-32 || true)"
AGENT_VERDICT="$(head -c 4096 "$ROUND_DIR/verdict.txt" 2>/dev/null | tr -d '[:space:]' | cut -c1-32 || true)"
case "$AGENT_VERDICT" in
merge-ready|findings|blocked|inconclusive) ;;
*)
@@ -5946,9 +5959,16 @@ jobs:
esac
else
# No report, so no verdict to trust: the newest round's counts
# are shown only as partial evidence under the process outcome.
# and images are shown only as partial evidence under the
# process outcome.
ASSERTIONS_FILE="$(find verify-results -mindepth 2 -maxdepth 2 -type f -path '*-verify-*/assertions.json' 2>/dev/null | sort | tail -1 || true)"
if [ -n "$ASSERTIONS_FILE" ]; then
ROUND_DIR="$(dirname "$ASSERTIONS_FILE")"
else
ROUND_DIR="$(find verify-results -mindepth 1 -maxdepth 1 -type d -name '*-verify-*' 2>/dev/null | sort | tail -1 || true)"
fi
fi
collect_and_host_evidence "$ROUND_DIR"
# Exact root-level path, never a find: the staging step writes
# this name last as root, and a same-named file nested inside a
# collected agent artifact dir must not shadow it.
@@ -6133,15 +6153,7 @@ jobs:
MISSING_REPORT_NOTE='No report.md was found in the run artifacts, so the report section is omitted — see the workflow run output.'
echo "::warning::${MISSING_REPORT_NOTE}"
fi
# A run that timed out or crashed before writing report.md has
# no findings to preserve: marking it substantive would let a
# headline plus "no report found" overwrite the previous round's
# real evidence, which is the opposite of the rule.
if [ -z "$REPORT" ]; then
WEAK_BODY=true
fi
{
printf '%s\n' '<!-- qwen-triage:verify -->'
# Substantive marker: this body carries findings, so a later
# weak notice must not displace it as the follow-up round's
# previous-report snapshot.
@@ -6222,52 +6234,68 @@ jobs:
# Attribution footer, the same shape /review posts: model, CLI
# version, and the content hash of the skill that ran. Only on
# a body that actually carries a report and only when all three
# workflow-owned values arrived — a missing report makes this
# a weak notice (WEAK_BODY above), and a half-known identity
# keeps the generic signature too.
# workflow-owned values arrived — without a report the body
# is only a notice, and a half-known identity keeps the
# generic signature too.
if [ -n "$REPORT" ] && [ -n "${VERIFY_MODEL:-}" ] && [ -n "${VERIFY_CLI_VERSION:-}" ] && [ -n "${VERIFY_SKILL_HASH:-}" ]; then
printf '_— %s via Qwen Code /verify-pr (v%s; skill %s)_\n' "$VERIFY_MODEL" "$VERIFY_CLI_VERSION" "$VERIFY_SKILL_HASH"
else
printf '%s\n' '— _Qwen Code · sandboxed verification_'
fi
} > "$BODY_FILE"
} >> "$BODY_FILE"
fi
# Upsert by marker: a real outcome (report / prepare-fail) replaces
# whatever carries the marker — the "running" status or an older
# report. A WEAK_BODY notice only replaces this run's own "running"
# status; if the marker comment is a previous round's real report,
# post fresh so that report survives.
# Write only into this run's own comment, found by its run marker:
# the live status its verify job posted, or the body this job
# already published if it is re-run. Anything else gets a fresh
# comment. The newest marker comment may belong to another round —
# a later round's live status, or an earlier round's report — and
# replacing either would make one round overwrite the other.
# Same ownership discipline as the resolve step: only BOT-OWNED
# comments STARTING with the marker are candidates, so a marker
# pasted by any user cannot divert the bot into PATCHing (and
# failing on) someone else's comment.
# Fail CLOSED on identity failure (an empty login previously
# widened the filter to every user's comments), and match the
# live-status MARKER rather than prose a report could quote.
EXISTING='' EXISTING_RUNNING='false'
# widened the filter to every user's comments), and read the
# machine markers only from a header: a report, or a value a PR
# run produced, can quote one further down.
OWN_ID=''
OWN_SUBSTANTIVE=''
if ! BOT_LOGIN="$(gh api user --jq '.login')" || [ -z "$BOT_LOGIN" ]; then
echo "::warning::Could not resolve the bot identity; posting a fresh comment instead of reusing one."
BOT_LOGIN=''
fi
if [ -n "$BOT_LOGIN" ] && EXISTING_META="$(
if [ -n "$BOT_LOGIN" ] && OWN_META="$(
gh api "repos/$GITHUB_REPOSITORY/issues/$PR_NUMBER/comments" \
--method GET \
--paginate \
-F per_page=100 \
| jq -sr --arg bot "$BOT_LOGIN" \
'[.[][] | select((.body | startswith("<!-- qwen-triage:verify -->")) and .user.login == $bot)] | last
| if . == null then "" else "\(.id)\t\(.body | contains("<!-- qwen-triage:verify-state=running -->"))" end'
| jq -sr --arg bot "$BOT_LOGIN" --arg run "$RUN_MARKER" --arg sub "$SUBSTANTIVE_MARKER" \
'def header: split("\n\n")[0] | split("\n");
[.[][] | select((.body | startswith("<!-- qwen-triage:verify -->")) and .user.login == $bot)
| (.body | header) as $h | select(any($h[]; . == $run))
| "\(.id)\t\(any($h[]; . == $sub))"]
| last // empty'
)"; then
EXISTING="${EXISTING_META%%$'\t'*}"
case "$EXISTING_META" in *$'\t'true) EXISTING_RUNNING='true' ;; esac
OWN_ID="${OWN_META%%$'\t'*}"
OWN_SUBSTANTIVE="${OWN_META#*$'\t'}"
elif [ -n "$BOT_LOGIN" ]; then
echo "::warning::Failed to look up existing verify comments; will create a new one."
echo "::warning::Failed to look up this run's comment; will create a new one."
fi
# A notice never replaces a body that carries findings. Re-run on
# its own, this job cannot download results (the artifact name
# carries the attempt), so its "results unavailable" notice would
# otherwise replace the round's report; it is posted beside it.
BODY_SUBSTANTIVE="$(
jq -Rrs --arg sub "$SUBSTANTIVE_MARKER" 'split("\n\n")[0] | split("\n") | any(.[]; . == $sub)' "$BODY_FILE" || true
)"
if [ "$OWN_SUBSTANTIVE" = true ] && [ "$BODY_SUBSTANTIVE" != true ]; then
OWN_ID=''
fi
# A failed PATCH (comment deleted, or ownership changed under us)
# must still leave a terminal report on the PR, never silence.
if [ -n "$EXISTING" ] && { [ "$WEAK_BODY" = false ] || [ "$EXISTING_RUNNING" = 'true' ]; }; then
gh api -X PATCH "repos/$GITHUB_REPOSITORY/issues/comments/$EXISTING" -F body=@"$BODY_FILE" >/dev/null ||
if [ -n "$OWN_ID" ]; then
gh api -X PATCH "repos/$GITHUB_REPOSITORY/issues/comments/$OWN_ID" -F body=@"$BODY_FILE" >/dev/null ||
gh api "repos/$GITHUB_REPOSITORY/issues/$PR_NUMBER/comments" -F body=@"$BODY_FILE" >/dev/null
else
gh api "repos/$GITHUB_REPOSITORY/issues/$PR_NUMBER/comments" -F body=@"$BODY_FILE" >/dev/null
scripts/tests/qwen-triage-workflow.test.js
+399 -93
View File
@@ -2789,6 +2789,7 @@ describe('qwen-triage verify hardening', () => {
GH_STUB_DIFF: diffFile,
GH_STUB_OID: prOid ?? 'oid-1',
GITHUB_REPOSITORY: 'QwenLM/qwen-code',
GITHUB_RUN_ID: '1',
GITHUB_STEP_SUMMARY: '/dev/null',
GITHUB_OUTPUT: out,
RUNNER_TEMP: temp,
@@ -3942,18 +3943,21 @@ describe('qwen-triage verify hardening round 2', () => {
);
});
// Status/report lifecycle is keyed on a machine marker, and identity
// Status/report lifecycle is keyed on machine markers, and identity
// failures must not widen the ownership filter to every user.
it('keys the status comment on a marker and fails closed on identity', () => {
it('keys the status comment on markers and fails closed on identity', () => {
for (const name of [
'Resolve PR and snapshot metadata',
'Post verification report comment',
]) {
const s = step(name);
expect(s).toContain('qwen-triage:verify-state=running');
expect(s).toContain('qwen-triage:verify-run=${GITHUB_RUN_ID}');
expect(s).not.toContain('$bot == ""');
expect(s).toContain('.user.login == $bot');
}
expect(step('Resolve PR and snapshot metadata')).toContain(
'qwen-triage:verify-state=running',
);
});
// The evidence-hosting path carries the untrusted-image checks; exercise
@@ -4018,9 +4022,7 @@ describe('qwen-triage verify hardening round 2', () => {
const work = join(dir, 'work');
const art = join(work, 'verify-results', 'prA-verify-1');
mkdirSync(join(art, 'evidence'), { recursive: true });
mkdirSync(join(work, 'verify-results', 'prB-verify-2', 'evidence'), {
recursive: true,
});
mkdirSync(join(art, 'scratch', 'evidence'), { recursive: true });
const png = (p, bytes) =>
writeFileSync(
p,
@@ -4030,11 +4032,8 @@ describe('qwen-triage verify hardening round 2', () => {
]),
);
png(join(art, 'evidence', '01-ab.png'), 500);
// Same sanitized name in a second artifact dir -> must not duplicate.
png(
join(work, 'verify-results', 'prB-verify-2', 'evidence', '01-ab.png'),
500,
);
// Same sanitized name in a nested evidence dir -> must not duplicate.
png(join(art, 'scratch', 'evidence', '01-ab.png'), 500);
writeFileSync(join(art, 'evidence', '02-fake.png'), 'not a png');
png(join(art, 'evidence', '03-big.png'), 2 * 1024 * 1024);
png(join(art, 'evidence', '04-edge.png'), 2 * 1024 * 1024 - 9);
@@ -4084,7 +4083,11 @@ describe('qwen-triage verify hardening round 2', () => {
'![01-ab](https://assets.example.test/pr-assets/verify/pr7999-77-1/01-ab.png)',
);
expect(comment).not.toContain('02-fake');
expect(comment).toContain('did not pass the hosting checks');
// The text file, the oversize file and the duplicate: a duplicate
// copied over the first counts as hosted and drops this to 2.
expect(comment).toContain(
'_3 additional image(s) did not pass the hosting checks',
);
// Upload failure -> text-only, never an aborted report.
const out2 = join(dir, 'comment2.md');
@@ -4451,8 +4454,10 @@ describe('qwen-triage verify hardening round 2', () => {
// Every publish fixture returned [] for the comments listing, so the
// PATCH arm — the one that must reuse a live status comment instead of
// stranding it — was never executed by any test.
it('patches a live status comment instead of posting a duplicate', () => {
// stranding it — was never executed by any test. The publisher used to
// take the newest marker comment, which under queued publishers was
// often another round's, and one round's report overwrote the other's.
it("patches this run's own status comment and no other", () => {
const publishStep = stepIn(
'publish-verify',
'Post verification report comment',
@@ -4463,12 +4468,14 @@ describe('qwen-triage verify hardening round 2', () => {
const dir = mkdtempSync(join(tmpdir(), 'verify-upsert-'));
try {
// The stub records which HTTP verb the publisher used and against
// which comment id, and serves a comments listing from a fixture.
// which comment id, keeps the body it sent, and serves a comments
// listing from a fixture.
writeFileSync(
join(dir, 'gh'),
[
'#!/usr/bin/env bash',
'all="$*"',
'for a in "$@"; do case "$a" in body=@*) cp "${a#body=@}" "$BODY_OUT";; esac; done',
'case "$all" in',
' *"api user"*|*" user "*) echo qwen-code-ci-bot ;;',
' *"-X PATCH"*) echo "PATCH $all" >> "$CALLS" ;;',
@@ -4488,6 +4495,7 @@ describe('qwen-triage verify hardening round 2', () => {
);
const listing = join(dir, 'listing.json');
const calls = join(dir, 'calls');
const bodyOut = join(dir, 'body.md');
const M = '<!-- qwen-triage:verify -->';
const RUNNING = '<!-- qwen-triage:verify-state=running -->';
const run = (comments, env = {}) => {
@@ -4501,13 +4509,15 @@ describe('qwen-triage verify hardening round 2', () => {
PATH: `${dir}:${process.env.PATH}`,
LISTING: listing,
CALLS: calls,
GH_STUB_OUT: join(dir, 'body.md'),
BODY_OUT: bodyOut,
GH_TOKEN: 'x',
GITHUB_REPOSITORY: 'QwenLM/qwen-code',
RUNNER_TEMP: dir,
GITHUB_STEP_SUMMARY: '/dev/null',
// A second attempt: the comment belongs to the run, not to one
// attempt of it.
GITHUB_RUN_ID: '1',
GITHUB_RUN_ATTEMPT: '1',
GITHUB_RUN_ATTEMPT: '2',
PR_NUMBER: '7999',
RUN_URL: 'u',
VERIFY_RESULT: 'success',
@@ -4523,34 +4533,133 @@ describe('qwen-triage verify hardening round 2', () => {
return readFileSync(calls, 'utf8');
};
// A live status comment owned by the bot must be PATCHed, not
// duplicated — otherwise the "running" line is stranded forever.
// One page of comments, matching `gh --paginate` output shape.
const patched = run([
{
id: 555,
user: { login: 'qwen-code-ci-bot' },
body: `${M}\n${RUNNING}\n\nrunning`,
},
]);
// A live status as the status step writes it, for the given run
// (the fixture publishes as run 1).
const status = (id, runId, login = 'qwen-code-ci-bot') => ({
id,
user: { login },
body: `${M}\n${RUNNING}\n<!-- qwen-triage:verify-run=${runId} -->\n\nrunning`,
});
// This run's own live status must be PATCHed, not duplicated —
// otherwise the "running" line is stranded forever. One page of
// comments, matching `gh --paginate` output shape.
const patched = run([status(555, 1)]);
expect(patched).toContain('PATCH');
expect(patched).toContain('/issues/comments/555');
expect(patched).not.toContain('POST');
// Both lookups select on the leading marker, and the run marker is
// what a re-run of this job finds.
expect(readFileSync(bodyOut, 'utf8').split('\n').slice(0, 2)).toEqual([
M,
'<!-- qwen-triage:verify-run=1 -->',
]);
// Re-run after it published: the report it already wrote is replaced,
// not posted a second time.
const rerun = run([
{
id: 555,
user: { login: 'qwen-code-ci-bot' },
body: `${M}\n<!-- qwen-triage:verify-run=1 -->\n<!-- qwen-triage:verify-substantive -->\n\nreport`,
},
]);
expect(rerun).toContain('/issues/comments/555');
expect(rerun).not.toContain('POST');
// No prior comment: post fresh.
expect(run([])).toContain('POST');
// A comment carrying the marker but owned by someone else must not be
// touched; the report is posted fresh instead.
const foreign = run([
{
id: 777,
user: { login: 'someone-else' },
body: `${M}\n${RUNNING}\n\nrunning`,
},
]);
const foreign = run([status(777, 1, 'someone-else')]);
expect(foreign).toContain('POST');
expect(foreign).not.toContain('/issues/comments/777');
// Publishers queue, so the newest marker comment can be another
// round's: its live status is that round's to replace, and an earlier
// round's report is kept. This real report is posted fresh beside
// either.
const report = (id) => ({
id,
user: { login: 'qwen-code-ci-bot' },
body: `${M}\n<!-- qwen-triage:verify-run=2 -->\n<!-- qwen-triage:verify-substantive -->\n\nearlier report`,
});
for (const comments of [
[report(600), status(601, 2)],
[status(601, 2), report(602)],
]) {
const calls = run(comments);
expect(calls).toContain('POST');
expect(calls).not.toContain('PATCH');
}
// The run marker is matched whole: run 1 does not own run 12's.
expect(run([status(612, 12)])).not.toContain('PATCH');
// The own status is found even when a later round's sits after it,
// and a weak notice replaces it just the same.
const own = run([status(555, 1), status(556, 2)]);
expect(own).toContain('/issues/comments/555');
expect(own).not.toContain('/issues/comments/556');
const cancelled = run([status(555, 1), status(556, 2)], {
VERIFY_RESULT: 'cancelled',
});
expect(cancelled).toContain('PATCH');
expect(cancelled).toContain('/issues/comments/555');
expect(cancelled).not.toContain('POST');
// A notice never replaces this run's published report. Re-run on its
// own, the job cannot download results (the artifact name carries the
// attempt), so a body that would say "results unavailable" is posted
// beside the report. A build failure carries findings and replaces it.
const ownReport = {
id: 555,
user: { login: 'qwen-code-ci-bot' },
body: `${M}\n<!-- qwen-triage:verify-run=1 -->\n<!-- qwen-triage:verify-substantive -->\n\nreport`,
};
for (const env of [
{ DOWNLOAD_OUTCOME: 'failure' },
{ VERIFY_RESULT: 'cancelled' },
{ VERDICT: 'infra-error', PREPARE_FAILURE_PHASE: 'install' },
]) {
const calls = run([ownReport], env);
expect(calls).toContain('POST');
expect(calls).not.toContain('/issues/comments/555');
}
const buildFail = run([ownReport], {
VERDICT: 'fail',
PREPARE_FAILURE_PHASE: 'build',
});
expect(buildFail).toContain('/issues/comments/555');
expect(buildFail).not.toContain('POST');
// A notice may replace this run's earlier notice.
const notice = run(
[
{
id: 555,
user: { login: 'qwen-code-ci-bot' },
body: `${M}\n<!-- qwen-triage:verify-run=1 -->\n\nresults unavailable`,
},
],
{ DOWNLOAD_OUTCOME: 'failure' },
);
expect(notice).toContain('/issues/comments/555');
expect(notice).not.toContain('POST');
// Markers count only in the header: another round's report quoting
// this run's marker further down is not this run's comment.
const quoting = {
id: 556,
user: { login: 'qwen-code-ci-bot' },
body: `${M}\n<!-- qwen-triage:verify-run=2 -->\n<!-- qwen-triage:verify-substantive -->\n\n\`\`\`\n${M}\n<!-- qwen-triage:verify-run=1 -->\n\`\`\``,
};
const quoted = run([status(555, 1), quoting]);
expect(quoted).toContain('/issues/comments/555');
expect(quoted).not.toContain('/issues/comments/556');
const quotedOnly = run([quoting]);
expect(quotedOnly).toContain('POST');
expect(quotedOnly).not.toContain('PATCH');
} finally {
rmSync(dir, { recursive: true, force: true });
}
@@ -6068,6 +6177,136 @@ describe('qwen-triage verify publish fidelity', () => {
}
});
// A re-run of the publisher finds what it already published by the run
// marker, and a notice must not replace findings, so every terminal body
// opens with a header: the verify marker both lookups select on, this
// run's marker, and the substantive marker when it carries findings. The
// lookups read markers only from the lines before the first blank line.
it('opens every terminal body with its header', () => {
const dir = fixture();
try {
const weak = [
'<!-- qwen-triage:verify -->',
'<!-- qwen-triage:verify-run=41 -->',
];
const strong = [...weak, '<!-- qwen-triage:verify-substantive -->'];
const variants = [
['real report', strong, { AGENT_VERDICT: 'merge-ready' }],
['incomplete — cancelled', weak, { VERIFY_RESULT: 'cancelled' }],
[
'incomplete — infrastructure failure',
weak,
{ VERIFY_RESULT: 'failure', VERDICT: '' },
],
[
'not run — skipped',
weak,
{ VERDICT: 'skipped', SKIP_REASON: 'closed' },
],
['not run — n/a', weak, { VERDICT: 'n/a' }],
['results unavailable', weak, { DOWNLOAD_OUTCOME: 'failure' }],
[
'the PR could not be built',
strong,
{ VERDICT: 'fail', PREPARE_FAILURE_PHASE: 'build' },
],
[
'was unreachable from the runner',
weak,
{ VERDICT: 'infra-error', PREPARE_FAILURE_PHASE: 'install' },
],
];
variants.forEach(([shown, header, env], i) => {
const body = render(dir, {
NAME: `hdr-${i}`,
GITHUB_RUN_ID: '41',
...env,
});
expect(body).toContain(shown);
expect(body.split('\n\n')[0].split('\n')).toEqual(header);
});
// A run whose artifact holds no report has no findings to keep.
rmSync(join(dir, 'work', 'verify-results', 'prA-verify-1', 'report.md'));
const reportless = render(dir, {
NAME: 'hdr-noreport',
GITHUB_RUN_ID: '41',
});
expect(reportless).toContain('No report.md was found');
expect(reportless.split('\n\n')[0].split('\n')).toEqual(weak);
// Written once, before the branches, which only append: a new branch
// cannot open its body without it.
const code = publishScript()
.split('\n')
.filter((line) => !/^\s*#/.test(line))
.join('\n');
expect(code.match(/'<!-- qwen-triage:verify -->'/g)).toHaveLength(1);
expect(code.match(/[^>]> "\$BODY_FILE"/g)).toHaveLength(1);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
// Images were collected across every round, oldest first, while the
// report came from the newest: an earlier round holding eight images
// filled the cap, and the published report showed none of its own.
it("illustrates the published round with that round's own images", () => {
const dir = fixture();
const results = join(dir, 'work', 'verify-results');
const png = (round, name) => {
mkdirSync(join(results, round, 'evidence'), { recursive: true });
writeFileSync(
join(results, round, 'evidence', `${name}.png`),
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0]),
);
};
const images = (env) => {
const body = render(dir, {
AGENT_VERDICT: 'merge-ready',
VERIFY_ASSETS_UPLOADER: 'true',
ALIYUN_OSS_BUCKET: 'b',
ALIYUN_OSS_PUBLIC_BASE_URL: 'https://oss.example.test',
...env,
});
// Each published round holds one image and hosts it; another round's
// images must not be counted as failing the checks either.
expect(body).not.toContain('did not pass the hosting checks');
return [
...body.matchAll(/!\[([^\]]+)\]\(https:\/\/oss\.example\.test\//g),
].map((m) => m[1]);
};
try {
// An earlier complete round with a full set, the newest complete
// round with one image, and a newer directory holding images only.
mkdirSync(join(results, 'prA-verify-0'));
writeFileSync(join(results, 'prA-verify-0', 'report.md'), '## old\n');
writeFileSync(
join(results, 'prA-verify-0', 'assertions.json'),
'{"pass":10,"fail":0,"total":10}',
);
for (let i = 1; i <= 8; i++) png('prA-verify-0', `0${i}-old`);
png('prA-verify-1', '09-new');
png('prA-verify-2', '10-stray');
expect(images({ NAME: 'ev-report' })).toEqual(['09-new']);
// Reportless: the round whose counts are shown.
rmSync(join(results, 'prA-verify-0', 'report.md'));
rmSync(join(results, 'prA-verify-1', 'report.md'));
expect(images({ NAME: 'ev-counts', VERDICT: 'timeout' })).toEqual([
'09-new',
]);
// Neither report nor counts anywhere: the newest round directory.
rmSync(join(results, 'prA-verify-0', 'assertions.json'));
rmSync(join(results, 'prA-verify-1', 'assertions.json'));
expect(images({ NAME: 'ev-none', VERDICT: 'timeout' })).toEqual([
'10-stray',
]);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
// verdict.txt has one reader. The verify job used to read its own copy
// from whichever round directory its search found first and hand the
// publisher that word.
@@ -6305,70 +6544,137 @@ describe('qwen-triage verify publish fidelity', () => {
}
});
// The follow-up snapshot must select the newest SUBSTANTIVE report, not
// the newest non-running comment: a cancelled notice passes that test.
it('snapshots the newest substantive report, not a weak notice', () => {
// The status step's comment block, run verbatim as run 1 with a stubbed
// gh that serves one page of `gh api --paginate` output and records every
// write and the snapshot read.
const statusComment = (comments) => {
const resolve = step('Resolve PR and snapshot metadata');
// Run the workflow's own selector verbatim against a fixture shaped
// exactly like one page of `gh api --paginate` output, so no rewriting
// of the expression is needed.
const jqProgram = resolve.match(/jq -rs --arg m[^']*'([\s\S]*?end)'/)?.[1];
expect(jqProgram).toBeTruthy();
expect(jqProgram).toContain('contains($sub)');
const dir = mkdtempSync(join(tmpdir(), 'verify-snap-'));
const start = resolve.indexOf("MARKER='<!-- qwen-triage:verify -->'");
const end = resolve.indexOf('echo "decision=run"', start);
expect(start).toBeGreaterThan(-1);
expect(end).toBeGreaterThan(start);
const dir = mkdtempSync(join(tmpdir(), 'verify-status-'));
try {
const comments = join(dir, 'c.json');
writeFileSync(
comments,
JSON.stringify([
{
id: 101,
user: { login: 'qwen-code-ci-bot' },
body: '<!-- qwen-triage:verify -->\n<!-- qwen-triage:verify-substantive -->\n\nREAL REPORT',
},
{
id: 102,
user: { login: 'qwen-code-ci-bot' },
body: '<!-- qwen-triage:verify -->\n\ncancelled notice',
},
]),
join(dir, 'gh'),
[
'#!/usr/bin/env bash',
'case "$*" in',
' "api user --jq .login") echo qwen-code-ci-bot ;;',
' *"--method GET"*) cat "$LISTING" ;;',
' *"--method PATCH"*) echo "PATCH $*" >> "$CALLS" ;;',
' *"--jq .body") echo "SNAP $*" >> "$CALLS"; echo previous ;;',
' *"/issues/7999/comments -f body="*) echo "POST $*" >> "$CALLS" ;;',
' *) echo "UNEXPECTED $*" >> "$CALLS"; exit 1 ;;',
'esac',
].join('\n'),
{ mode: 0o755 },
);
const runJq = (program) =>
spawnSync(
'jq',
[
'-rs',
'--arg',
'm',
'<!-- qwen-triage:verify -->',
'--arg',
's',
'<!-- qwen-triage:verify-state=running -->',
'--arg',
'sub',
'<!-- qwen-triage:verify-substantive -->',
'--arg',
'bot',
'qwen-code-ci-bot',
program,
comments,
],
{ encoding: 'utf8' },
).stdout.trim();
// The newest comment is the weak notice, but the snapshot must be the
// substantive report behind it.
expect(runJq(jqProgram)).toBe('102\tfalse\t101');
// Control: selecting "newest non-running comment" picks the notice.
expect(
runJq(
'[.[][] | select((.body | startswith($m)) and .user.login == $bot)] | map(select(.body | contains($s) | not)) | last | "\\(.id)"',
),
).toBe('102');
writeFileSync(join(dir, 'listing.json'), JSON.stringify(comments));
writeFileSync(join(dir, 'calls'), '');
mkdirSync(join(dir, 'verify-context'));
const script = resolve.slice(start, end).replace(/^ {10}/gm, '');
const res = spawnSync('bash', ['-c', `set -euo pipefail\n${script}`], {
cwd: dir,
encoding: 'utf8',
env: {
...process.env,
PATH: `${dir}:${process.env.PATH}`,
LISTING: join(dir, 'listing.json'),
CALLS: join(dir, 'calls'),
GH_TOKEN: 'x',
GITHUB_REPOSITORY: 'QwenLM/qwen-code',
GITHUB_RUN_ID: '1',
GITHUB_RUN_ATTEMPT: '2',
PR_NUMBER: '7999',
RUN_URL: 'u',
RUNNER_TEMP: dir,
},
});
expect(res.status).toBe(0);
const calls = readFileSync(join(dir, 'calls'), 'utf8');
expect(calls).not.toContain('UNEXPECTED');
return calls;
} finally {
rmSync(dir, { recursive: true, force: true });
}
};
const BOT = { login: 'qwen-code-ci-bot' };
const liveStatus = (id, runId, user = BOT) => ({
id,
user,
body: `<!-- qwen-triage:verify -->\n<!-- qwen-triage:verify-state=running -->\n<!-- qwen-triage:verify-run=${runId} -->\n\nrunning`,
});
// The follow-up snapshot must select the newest SUBSTANTIVE report, not
// the newest non-running comment: a cancelled notice passes that test,
// and so does a notice quoting the marker below its header.
it('snapshots the newest substantive report, not a weak notice', () => {
const calls = statusComment([
{
id: 101,
user: BOT,
body: '<!-- qwen-triage:verify -->\n<!-- qwen-triage:verify-substantive -->\n\nREAL REPORT',
},
{
id: 102,
user: BOT,
body: '<!-- qwen-triage:verify -->\n\ncancelled notice',
},
{
id: 103,
user: BOT,
body: '<!-- qwen-triage:verify -->\n\nnotice quoting\n<!-- qwen-triage:verify-substantive -->',
},
]);
expect(calls).toContain(
'SNAP api repos/QwenLM/qwen-code/issues/comments/101 ',
);
expect(calls).not.toContain('comments/102');
expect(calls).not.toContain('comments/103');
});
// Verify jobs run one at a time per PR, but the previous round's
// publisher is often still queued, so its live status is still there.
// Taking it over put both rounds into one comment, and the later report
// overwrote the earlier one.
it("reuses only its own run's status comment", () => {
// The body exactly as both lookups read it: the verify marker first,
// then the state and this run's marker.
const written =
'-f body=<!-- qwen-triage:verify -->\n<!-- qwen-triage:verify-state=running -->\n<!-- qwen-triage:verify-run=1 -->\n\n';
const own = statusComment([liveStatus(201, 1), liveStatus(202, 2)]);
expect(own).toContain(
'PATCH api --method PATCH repos/QwenLM/qwen-code/issues/comments/201 ',
);
expect(own).not.toContain('comments/202');
expect(own).toContain(written);
// Another round's live status, a run id that only starts with ours,
// and our run marker in a comment the bot does not own: none is ours.
// ...nor is this run's published report, which a re-run must not turn
// back into "running", nor a report quoting both markers below its
// header.
for (const comment of [
liveStatus(202, 2),
liveStatus(212, 12),
liveStatus(203, 1, { login: 'someone-else' }),
{
id: 204,
user: BOT,
body: '<!-- qwen-triage:verify -->\n<!-- qwen-triage:verify-run=1 -->\n<!-- qwen-triage:verify-substantive -->\n\nreport',
},
{
id: 205,
user: BOT,
body: '<!-- qwen-triage:verify -->\n<!-- qwen-triage:verify-run=2 -->\n\n```\n<!-- qwen-triage:verify-state=running -->\n<!-- qwen-triage:verify-run=1 -->\n```',
},
]) {
const calls = statusComment([comment]);
expect(calls).not.toContain('PATCH');
expect(calls).toContain('POST');
expect(calls).toContain(written);
}
});
// The report comment carries the same attribution /review posts: model,
@@ -7431,13 +7737,13 @@ describe('qwen-triage verify round-3 hardening', () => {
});
// A crashed run with no report has nothing to preserve, so it must not
// claim the substantive marker and overwrite the previous round's report.
// claim the substantive marker and displace the previous round's report
// as the next round's snapshot.
it('marks a body substantive only when a report exists', () => {
const publish = stepIn(
'publish-verify',
'Post verification report comment',
);
expect(publish).toMatch(/if \[ -z "\$REPORT" \]; then\s+WEAK_BODY=true/);
expect(publish).toMatch(
/if \[ -n "\$REPORT" \]; then\s+printf '%s\\n' '<!-- qwen-triage:verify-substantive -->'/,
);