fix(verify-pr): count test suites apart from the round's assertions (#16013)

* fix(verify-pr): count test suites apart from the round's assertions Rounds folded whole workspace suites into assertions.json, so the published "Scripted assertions" line measured the repo, not the round: 37 of the 252 counted reports in the 30-day census exceeded 1000, and one round's 7737 held a few dozen checks of its own, the rest being five suites counted again for every mutant run. assertions.json gains gates: [{ suite, pass, fail, skipped? }], one entry per existing test suite run as a targeted gate. The skill says suite results and mutant runs stay out of pass/fail/total, and that a gate red the PR causes still reaches the verdict as an assertion. The checker refuses a malformed list and warns when over 1000 assertions come with no gates. The publisher prints the gate totals on a line of their own, numbers only, and does not let them decide the verdict. Also, deferred from #15938: the paragraph for a merge-ready round with failures now states the checker's rule instead of inviting the reader to excuse the failures, and the artifact-check guard's empty-value legs are pinned. * fix(verify-pr): warn on empty gates and mirror the checker's shape rules Address the three stage-2 triage suggestions: an empty gates list no longer silences the suite-sized warning; the publisher's jq refuses blank suite names and non-count skipped values (false/null) instead of defaulting them, so a shape the checker refuses never renders totals; and the design docs say 'all three' / '三者' where the rule count grew from two to three. Co-authored-by: Qwen-Coder <[email protected]> * fix(verify-pr): give the gates shape rule one witness and honest totals Address the five R1 review suggestions: - one shared fixture table drives both the publisher's jq and checkArtifacts, and both bound counts at 2^53-1, closing the 1e309 divergence where jq clamped an overflow the checker refuses - the skipped clause prints only when some entry records the field, so an unrecorded count is no longer published as a measured zero - a trusted verdict over gate reds carries one advisory sentence in both languages pointing at the assertion ledger; still verdict-inert - pin the skill's compensation sentence and name the gates-size warning in the local clear-list and the CI-path sentence - bump the workflow size baseline to 375110: the parity comment, the conditional clause and the advisory exceed the remaining allowance Co-authored-by: Qwen-Coder <[email protected]> * fix(verify-pr): print the skip clause only for fully recorded gates Address the two R2 review suggestions: - the clause now requires every entry to record skipped; a mixed list published a partial sum as the run's skip total - pin the advisory's trusted-verdict guard from the false side: an untrusted run with gate reds prints the totals but no advisory - fixtures: a mixed list drops the clause, a recorded skipped:0 keeps it, and a timeout headline over gate reds stays advisory-free Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]>
2026-11-07 06:19:43 -10:00 · 2026-11-01 02:26:31 +08:00
1 parent e739b9fd3f
commit 06c1f98971
@@ -47,7 +47,7 @@
 5760 qwen-pr-safety-precheck.yml
 3243 qwen-review-runner-schedule.yml
 27648 qwen-triage-finalize.yml
-369831 qwen-triage.yml
+375110 qwen-triage.yml
 15726 release-chrome-extension.yml
 26801 release-managed-agent-reference.yml
 13464 release-managed-runtime.yml
@@ -5975,6 +5975,8 @@ jobs:
            FLAKE_LOG='verify-results/flake-gate.log'
            ASSERT_LINE=''
            ASSERT_LINE_ZH=''
+            GATES_LINE=''
+            GATES_LINE_ZH=''
            if [ -n "$ASSERTIONS_FILE" ]; then
              # Numbers only — coerce anything non-numeric to 0 so untrusted
              # JSON cannot smuggle text into the comment body.
@@ -6001,6 +6003,37 @@ jobs:
                read -r A_PASS A_FAIL A_TOTAL <<< "$ASSERT_META"
                ASSERT_LINE="Scripted assertions: ${A_PASS} passed · ${A_FAIL} failed · ${A_TOTAL} total"
                ASSERT_LINE_ZH="脚本断言：${A_PASS} 通过 · ${A_FAIL} 失败 · ${A_TOTAL} 总计"
+                # Existing test suites run as targeted gates are counted
+                # apart from the round's own assertions: folded in, a
+                # workspace's thousands of tests made "N scripted
+                # assertions" a measure of the repo, not of the round. Totals
+                # only, so no agent-chosen suite name reaches the body; a
+                # malformed list gets no line, and the checker refuses it.
+                # Shape rules mirror the checker's, and one shared fixture
+                # table pins the parity: a blank suite or a non-count skipped
+                # (false/null) is refused, not defaulted, and counts stop at
+                # 2^53-1, the last integer JSON literals parse to exactly.
+                GATES_META="$(
+                  jq -r 'def count: type == "number" and . >= 0 and . == floor and . <= 9007199254740991;
+                         if (.gates | type) == "array" and (.gates | length) > 0
+                            and (.gates | all(type == "object" and (.suite | type) == "string" and (.suite | test("\\S")) and (.pass | count) and (.fail | count) and ((has("skipped") | not) or (.skipped | count))))
+                         then "\(.gates | length) \(.gates | map(.pass) | add | floor) \(.gates | map(.fail) | add | floor) \(.gates | map(.skipped // 0) | add | floor) \(.gates | map(select(.skipped != null)) | length)"
+                         else empty end' "$ASSERTIONS_FILE" 2>/dev/null || true
+                )"
+                if [ -n "$GATES_META" ]; then
+                  read -r G_SUITES G_PASS G_FAIL G_SKIP G_RECORDED <<< "$GATES_META"
+                  # Absent skipped is "not recorded", not a measured zero,
+                  # and a partial sum is not the run's total: the clause
+                  # prints only when every entry carries the field.
+                  G_SKIP_CLAUSE=''
+                  G_SKIP_CLAUSE_ZH=''
+                  if [ "$G_RECORDED" = "$G_SUITES" ]; then
+                    G_SKIP_CLAUSE=" · ${G_SKIP} skipped"
+                    G_SKIP_CLAUSE_ZH=" · ${G_SKIP} 跳过"
+                  fi
+                  GATES_LINE="Targeted gates (existing tests, not counted above): ${G_PASS} passed · ${G_FAIL} failed${G_SKIP_CLAUSE} across ${G_SUITES} suite(s)"
+                  GATES_LINE_ZH="定向门禁（已有测试，不计入上面的断言）：${G_PASS} 通过 · ${G_FAIL} 失败${G_SKIP_CLAUSE_ZH}，共 ${G_SUITES} 个套件"
+                fi
              else
                echo "::warning::assertions.json missing or inconsistent; not treating it as evidence."
              fi
@@ -6175,11 +6208,19 @@ jobs:
              if [ -n "$ASSERT_LINE" ]; then
                printf '%s\n\n' "$ASSERT_LINE"
              fi
+              if [ -n "$GATES_LINE" ]; then
+                printf '%s\n\n' "$GATES_LINE"
+                # Advisory only: a gate can be red for a pre-existing main
+                # failure, so this never refuses or demotes the verdict.
+                if [ "$TRUST_AGENT_VERDICT" = true ] && [ "${G_FAIL:-0}" != '0' ]; then
+                  printf 'Gate totals do not decide the verdict: a gate red the PR causes belongs in the assertion ledger above — read that count before trusting the verdict.\n\n'
+                fi
+              fi
              if [ -n "$FLAKE_LINE" ]; then
                printf '%s\n\n' "$FLAKE_LINE"
              fi
              if [ "${AGENT_VERDICT:-}" = 'merge-ready' ] && [ "$TRUST_AGENT_VERDICT" != true ] && [ "${A_FAIL:-0}" != '0' ]; then
-                printf 'The agent reported `merge-ready`, but `assertions.json` recorded %s failures, so that claim was not trusted — see the report for whether these are expected A/B control cells.\n\n' "$A_FAIL"
+                printf 'The agent reported `merge-ready`, but `assertions.json` recorded %s failures, so that claim was not trusted: an expected A/B control red counts as a pass, so a recorded failure contradicts `merge-ready`.\n\n' "$A_FAIL"
              fi
              if [ -n "$CHECK_NOTE" ]; then
                printf 'The agent reported `%s`, but %s.\n\n' "$AGENT_VERDICT" "$CHECK_NOTE"
@@ -6203,11 +6244,17 @@ jobs:
              if [ -n "$ASSERT_LINE_ZH" ]; then
                printf '%s\n\n' "$ASSERT_LINE_ZH"
              fi
+              if [ -n "$GATES_LINE_ZH" ]; then
+                printf '%s\n\n' "$GATES_LINE_ZH"
+                if [ "$TRUST_AGENT_VERDICT" = true ] && [ "${G_FAIL:-0}" != '0' ]; then
+                  printf '门禁合计不影响判定：PR 导致的门禁红灯应记入上面的断言账本——采信判定前请先看那个数。\n\n'
+                fi
+              fi
              if [ -n "$FLAKE_LINE_ZH" ]; then
                printf '%s\n\n' "$FLAKE_LINE_ZH"
              fi
              if [ "${AGENT_VERDICT:-}" = 'merge-ready' ] && [ "$TRUST_AGENT_VERDICT" != true ] && [ "${A_FAIL:-0}" != '0' ]; then
-                printf 'agent 报告了 `merge-ready`，但 `assertions.json` 记录了 %s 个失败，因此该判定未被采信——请参阅报告确认这些是否为预期的 A/B 对照单元。\n\n' "$A_FAIL"
+                printf 'agent 报告了 `merge-ready`，但 `assertions.json` 记录了 %s 个失败，因此该判定未被采信：预期中的 A/B 对照红灯应计为通过，所以记录在案的失败与 `merge-ready` 相矛盾。\n\n' "$A_FAIL"
              fi
              if [ -n "$CHECK_NOTE_ZH" ]; then
                printf 'agent 报告了 `%s`，但%s。\n\n' "$AGENT_VERDICT" "$CHECK_NOTE_ZH"