ci(sdk-java): run the Hosted admission gates even when the Hosted verify fails (#16010)

* ci(sdk-java): run the Hosted admission gates even when the Hosted verify fails

The admission step had no condition, so it ran only when every earlier
step passed. The Hosted verify fails most nights, and the step was skipped
on all 13 nights from 10-14 to 10-30 that the verify failed, including the
three since the Runtime Broker fault gates stopped being skipped the same
way. It ran on the other three nights and passed each time, in about two
minutes.

The step rebuilds the module with `clean verify` and needs only the CLI
bundle, the installed SDK and the MySQL service. Each gate class works
under its own random tenant, and on green nights it already runs against
the database the Hosted verify used. So it takes the condition the fault
gates and the O4 step use: run unless the run was cancelled, as long as
the SDK install succeeded.

The class check after it reads only the admission step's reports. It now
follows that step's outcome instead of the job status; otherwise it would
still skip after a red Hosted verify while the step it audits ran.

The pin for the fault-gate condition now covers both steps, and a new case
pins the class check's condition and position. The setup-step assertion
now names the offending steps in its message: vitest abbreviates the
array once a step name is long enough, and the install step's own name
already is.

* ci(sdk-java): keep the O4 gates reachable on red-verify nights

The admission step now runs on red-verify nights too, so the job's worst
case sums all four capped steps (291 min) plus the uncapped setup. Raise
the hosted-harness-mysql ceiling from 304 to 324 so a slow setup (the CLI
install retries three times) cannot cancel the job before the O4 gates,
whose !cancelled() condition would record them skipped.

Close the two pin gaps the review found: the red-verify condition test
gains the O4 row (all three conditioned steps are now pinned), and the
admission class-check test pins that the check step carries no
continue-on-error, so a dropped gate class still fails the job.

All three pins are mutation-witnessed at this head (ceiling back to 304,
O4 if: deleted, continue-on-error on the check step each redden exactly
their case); the nine sdk-java.yml parsing suites pass 709/709.

* ci(sdk-java): boot the admission gates on their own database and bump the size baseline

Now that the admission step runs after a red Hosted verify, it must not
share that verify's database. Managed schema validation refuses any table
outside the managed set, and four Hosted ITs create fg*_fire tables they
drop only during cleanup (the relay classes only while their Spring context
is active), while the isolation extension truncates only ma_ tables. A
class killed before its cleanup would leave one behind and fail both
admission gates at boot. On a MySQL 8.4.6 rig, an fg9b_fire table in
hosted_harness_test made the base command fail both classes with "Managed
schema must contain exactly the 38 target tables", while this command
passed both on a fresh qwen_admission_ci. The step now creates that
database the way O4 creates qwen_o4_ci.

The pin test adds the O4 row, pins that the class check has no
continue-on-error, custom shell or altered command, that no gate command
swallows its failure, and that admission boots on its own database. Both
steps' names are checked before use.

sdk-java.yml outgrew the ratchet: main's copy was already 3971 bytes over
its recorded 144802 without a bump, and this change adds the rest. Even
without its comments the file would exceed the ceiling, so the baseline
moves to the file's size.

* test(ci): pin the admission step whole and keep the install ahead of the verify

The admission step's command, env and shell are now pinned with literal
values. The hosted-gates class check counts a failed class as one that
ran, so an `||` fallback, a pipe, or -fn in the command or in MAVEN_ARGS
would have turned a red admission gate green, and a second -Dmysql.url
would have pointed it back at the shared database. A list of forbidden
substrings caught some of these and rejected harmless commands, so the
fault-gate and O4 rows go back to pinning only their condition and
continue-on-error; their commands are unchanged by this PR.

The three gate rows also pin that the SDK install comes before the Hosted
verify. After it, a red verify would skip the install and every gate with
it, on exactly the nights the condition is for.

Comments: the job comment opened with stale ceilings (162 + 20 + 50 + 20
= 252); it now gives the measured setup time (under 10 minutes on every
nightly from 10-14 to 10-30), states that what changed is the timed-out
Verify path, which used to skip admission, and says the CLI install makes
up to three attempts rather than retrying three times. The O4 row comment
says O4 has carried its condition since it was added.

* test(ci): derive the red-verify contract from the job and pin the setup allowance

The red-verify condition was asserted over three hand-listed step names,
so a gate added to the job later sat outside every assertion and a red
verify would skip it silently, as it skipped the admission gates. A new
case derives the contract from the steps after the Hosted verify:

- Only the two report checks that need the verify's output may run on a
  green job alone. Every other step's condition must be `always()`, or
  `${{ }}` around `always()`, `!cancelled()` or `failure()`, optionally
  narrowed by one step's outcome. Any other condition either has no
  status function, so GitHub prefixes `success() &&`, or still reads the
  job status. A narrowing step must itself run regardless of the verify.
- No step after the verify carries continue-on-error.
- Every step that invokes Maven, matched as a word so folded commands,
  wrappers and quoted paths count, carries the exact red-verify condition
  and a ceiling, so it enters the summed step budget.

The job-budget rows now carry the setup allowance (33 for the MySQL job,
15 for failover) and assert ceiling - steps against it, so raising a step
ceiling re-derives the allowance instead of quietly spending it.
This commit is contained in:
Shaojin Wen authored and GitHub committed 2026-11-01 02:25:44 +08:00
1 parent 7838a9c7ee
commit 143a8b0653
3 files changed
+229 -31

No files matched your search

+1 -1
View File
@@ -60,7 +60,7 @@
1079 scorecard-monthly.yml
9751 sdk-java-contract-open-prs.yml
9555 sdk-java-flyway-open-prs.yml
144802 sdk-java.yml
150332 sdk-java.yml
3886 sdk-python.yml
5595 security-checks.yml
6777 serve-ab-publish.yml
+33 -12
View File
@@ -1097,7 +1097,7 @@ jobs:
name: 'Hosted process fault gates / MySQL 8.4 / Java 21'
if: "${{ inputs.tier != 'fast' }}"
runs-on: 'ubuntu-latest'
# The step ceilings today (162 + 20 + 50 + 20 = 252 — the FG8 letters
# The step ceilings today (201 + 20 + 50 + 20 = 291 — the FG8 letters
# grew the shared failsafe fork to 3600 s, FG9 slice 3 to 3990 s, the
# FG10b–f workflow relay letters to 4484 s, the FG12a MCP reply-loss
# letter to 4499 s, the FG12c restart matrix to 4548 s, the FG13c Hook
@@ -1171,10 +1171,16 @@ jobs:
# pending CI measurement: fork 11444 s, Verify 201 minutes, and 12044 s
# of fork plus wrap with the same 16 s left;
# the Broker fault gates use 20,
# and admission plus O4 keep per-fork ceilings) plus ~13 minutes for the
# uncapped setup steps (Checkout.. cgroup delegation, the report
# checks): 291 + 13. The failover arms run in hosted-failover-e2e.
timeout-minutes: 304
# and admission plus O4 keep per-fork ceilings) plus a 33-minute
# allowance for the uncapped setup steps (Checkout.. cgroup delegation,
# the report checks), which took under 10 minutes on every nightly from
# 10-14 to 10-30: 291 + 33. A Verify that timed out used to skip
# admission, leaving that path 50 minutes spare; admission now runs
# after it, so that path too can fill all four capped ceilings. The
# allowance sits above the ~13-minute floor so that a slow setup (the
# CLI install makes up to three attempts) still leaves the O4 gates
# time to run. The failover arms run in hosted-failover-e2e.
timeout-minutes: 324
services:
mysql:
image: 'mysql:8.4.6'
@@ -1329,24 +1335,39 @@ jobs:
retention-days: 7
- name: 'Verify Hosted admission gates'
id: 'hosted_admission'
# Its clean would otherwise discard the hosted lane's failsafe
# reports and the fg7-cases evidence the hosts upload above, so it
# runs after the fixture. The hosted-harness-gates profile carries a
# 2400 s failsafe fork budget for both admission witness classes
# (@Timeout(1800) each); the same killed-after-timeout pattern as the
# hosted-harness-mysql verify applies.
# Like the fault gates it rebuilds the module and needs only what the
# bundle step installs and builds, the installed SDK and the MySQL
# service, so a red Hosted verify must not skip it: without this it
# was skipped on all 13 nights from 10-14 to 10-30 that the verify
# failed, three of them after the fault gates got this condition.
# It boots on a fresh database, as O4 does: schema validation refuses
# any table outside the managed set, and a Hosted IT killed before
# its cleanup can leave its fg*_fire table behind.
if: "${{ !cancelled() && steps.managed_dependencies.outcome == 'success' }}"
timeout-minutes: 50
env:
MAVEN_ARGS: '--settings ${{ runner.temp }}/setup-java-m2/settings.xml --toolchains ${{ runner.temp }}/setup-java-m2/toolchains.xml'
MYSQL_PORT: "${{ job.services.mysql.ports['3306'] }}"
run: >-
mvn --batch-mode --no-transfer-progress -f packages/managed-agent-server/pom.xml
-Phosted-harness-gates
-Dnode.executable="$(command -v node)"
-Dqwen.cli.entry="${GITHUB_WORKSPACE}/dist/cli.js"
-Dmysql.url="jdbc:mysql://127.0.0.1:${MYSQL_PORT}/hosted_harness_test?allowPublicKeyRetrieval=true&useSSL=false"
-Dmysql.user=root -Dmysql.password=hosted-fixture clean verify checkstyle:check
MYSQL_PWD: 'hosted-fixture'
run: |-
mysql --no-defaults --protocol=TCP --host=127.0.0.1 --port="${MYSQL_PORT}" --user=root -e 'CREATE DATABASE qwen_admission_ci'
mvn --batch-mode --no-transfer-progress -f packages/managed-agent-server/pom.xml \
-Phosted-harness-gates \
-Dnode.executable="$(command -v node)" \
-Dqwen.cli.entry="${GITHUB_WORKSPACE}/dist/cli.js" \
-Dmysql.url="jdbc:mysql://127.0.0.1:${MYSQL_PORT}/qwen_admission_ci?allowPublicKeyRetrieval=true&useSSL=false" \
-Dmysql.user=root -Dmysql.password=hosted-fixture clean verify checkstyle:check
- name: 'Check that every Hosted admission gate class ran'
# It reads only the admission step's reports, so it follows that
# step rather than the job status a red Hosted verify has already set.
if: "${{ !cancelled() && steps.hosted_admission.outcome == 'success' }}"
run: 'node scripts/check-failsafe-reports.js hosted-gates packages/managed-agent-server'
- name: 'Verify O4 filesystem process and capacity gates'