feat(verify-pr): give the sandboxed round the PR's review threads (#16300)

* feat(verify-pr): give the sandboxed round the PR's review threads

The sandboxed /verify round had no token and no review context. Its only
inputs were the metadata snapshot and, on a follow-up round, the previous
report. On #13652, Critical R1-26 was confirmed and then marked fixed
three days before a round that published merge-ready without mentioning
it. Eight hours later the review bot, at the head that round had
certified, still could not rule out its residual.

The verify job's trusted resolve step now writes review-threads.json
beside pr.json. It holds every review thread, resolved ones included,
since resolving is the author's claim, together with each reviewer's
standing CHANGES_REQUESTED review. It comes from one paginated GraphQL
query reshaped with jq. The snapshot is best-effort: on failure no file
is written and the step warns.

verify-pr-prepare.mjs sends the same query and writes the same shape for
the local path. It removes a previous copy first, so a failed fetch
cannot leave a stale file. A test runs the workflow's block verbatim and
prepare's transform over the same pages and requires equal output.

The skill adds report item 4, Existing review findings. Each term the
verdict turns on is computed from snapshot fields:

- a thread is blocking when its first comment is tagged Critical,
  Blocker, P0 or P1 by an owner, member or collaborator;
- a blocking thread is open when it is unresolved, or when that comment
  belongs to a standing change request, resolved or not. That is the rule
  GitHub's review decision applies.

An open blocking thread left unmeasured rules out merge-ready, and with
nothing failed the verdict is inconclusive. Findings at fail 0 would
publish as "passed with non-blocking findings". Other blocking threads are
re-measured as budget allows, and the verdict line counts those left
unmeasured. Change request bodies are listed but do not hold the verdict,
because their severities are prose. The duty is also a line in the scope
budget. A missing file reads as unknown, never as "nothing open".

The file adds 2396 bytes, so the size baseline records the exact 378242.

* test(verify-pr): pin the review snapshot's jq guard, copy order and odd answers

- The CI/local parity test runs the step's real jq program. Like the
  file's four other jq cases, it now skips when jq is missing, instead of
  failing one assertion later with a message that names neither jq nor
  the missing tool.
- The local recipe must clear the cached review-threads.json before it
  copies prepare's file. The test pinned the three lines but not their
  order. With rm after cp, every local round would read the review state
  as unknown.
- prepare's two other failure arms are now driven through the CLI: an
  answer that is not JSON, and pages without a pull request. Each must
  leave no file and say why. An empty state there would read as "nothing
  open".

* test(verify-pr): drive the review snapshot through the shared gh harness

The review-snapshot case had its own copy of the executed-step harness:
a temp dir, a stub gh on PATH and GitHub's bash flags. makeGhHarness
exists so that fidelity contract lives in one place. The factory now
records every call's argv, returns its temp dir, and has an
empty-stdout arm for paginated reads, GH_STUB_EMPTY_LIST. The snapshot
case runs through it. Its failure arms are the factory's
GH_STUB_FAIL_LIST and GH_STUB_EMPTY_LIST.

Deleting the shared stub's fail arm, its empty-stdout arm or its argv
capture turns the case red, and so does recording only the first
argument.
This commit is contained in:
Shaojin Wen authored and GitHub committed 2026-11-05 23:50:44 +08:00
1 parent 0c620f62d8
commit 5d45b42940
9 files changed
+887 -30

No files matched your search

+1 -1
View File
@@ -47,7 +47,7 @@
5760 qwen-pr-safety-precheck.yml
3243 qwen-review-runner-schedule.yml
27648 qwen-triage-finalize.yml
375110 qwen-triage.yml
378242 qwen-triage.yml
15726 release-chrome-extension.yml
26801 release-managed-agent-reference.yml
13464 release-managed-runtime.yml
+26
View File
@@ -2552,6 +2552,32 @@ jobs:
exit 0
fi
fi
# Review threads, resolved ones included, and standing change
# requests for the same token-free agent: a round that never sees a
# reviewer's Critical, or the author's claim to have fixed one, can
# certify a head that still carries it. Same query and shape as
# scripts/verify-pr-prepare.mjs; a test runs both over one set of
# pages. Best-effort, and absent on failure: the skill reads a
# missing file as unknown, never as nothing open.
# shellcheck disable=SC2016 # $owner and friends are GraphQL's
if ! gh api graphql --paginate --slurp -f owner="${GITHUB_REPOSITORY%/*}" \
-f name="${GITHUB_REPOSITORY#*/}" -F number="$PR_NUMBER" \
-f query='query($owner: String!, $name: String!, $number: Int!, $endCursor: String) { repository(owner: $owner, name: $name) { pullRequest(number: $number) { latestOpinionatedReviews(first: 100) { nodes { state submittedAt url body author { login } authorAssociation commit { oid } } } reviewThreads(first: 50, after: $endCursor) { pageInfo { hasNextPage endCursor } nodes { isResolved isOutdated path line originalLine comments(first: 50) { totalCount nodes { createdAt url body author { login } authorAssociation originalCommit { oid } pullRequestReview { url } } } } } } } }' |
jq 'def who: {author: .author.login, association: .authorAssociation};
[.[].data.repository.pullRequest] as $p
| {threads: [$p[].reviewThreads.nodes[]
| {resolved: .isResolved, outdated: .isOutdated, path, line, originalLine,
commentCount: .comments.totalCount,
comments: [.comments.nodes[] | who + {createdAt, url,
review: .pullRequestReview.url, commit: .originalCommit.oid, body}]}],
changeRequests: [$p[0].latestOpinionatedReviews.nodes[]
| select(.state == "CHANGES_REQUESTED")
| who + {submittedAt, commit: .commit.oid, url, body}]}' \
> "$RUNNER_TEMP/verify-context/review-threads.json" ||
[ ! -s "$RUNNER_TEMP/verify-context/review-threads.json" ]; then
rm -f "$RUNNER_TEMP/verify-context/review-threads.json"
echo "::warning::Could not snapshot the PR's review threads; the round reports them as not covered."
fi
# Status comment with the live run link; publish-verify rewrites
# this same comment with the final report. Best-effort.
MARKER='<!-- qwen-triage:verify -->'