fix(ci): scope the sdk-java host lock to the cache write-back (#15128)

* fix(ci): scope the sdk-java host lock to the cache write-back (#15117) Run 39627609266's fast-tier job arrived at 17:38, stayed head of the sdk-java host-lock queue, and failed at 19:15 without starting Maven. The MariaDB integration lane held the same lock across its install and full MySQL clean verify; under pool contention that hold ran from 15:54 to 18:57 (run 39628985638), past the fast tier's 90-minute budget. Both MariaDB Maven runs already resolve from a job-local repository, so they touch no host-shared state. Seed that repository before the lock, run both Maven commands unlocked, and take the host lock only for the cache write-back, the sole remaining host-shared mutation. Drop fd 9 on the write-back children so a cancelled job's survivor cannot retain the lock. Raise the lane ceiling to 300 minutes for the ~3-hour verify plus one lock wait and setup, and pin the new order and fd closes. Fixes #15117 * test(ci): exempt the OpenTUI parity suites from the no-AK gate (#15117) The main merge brought three interactive suites from #14964. The no-AK gate requires every runnable integration test to be selected or explicitly exempted, so the Test job fails until they are listed. Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen Autofix <[email protected]> Co-authored-by: Qwen-Coder <[email protected]> Co-authored-by: jinjing.zzj <[email protected]> Co-authored-by: 易良 <[email protected]>
2026-11-07 06:19:43 -10:00 · 2026-10-22 21:44:45 +08:00
1 parent cf5a473cab
commit cfd95f6abb
@@ -59,7 +59,7 @@
 1079 scorecard-monthly.yml
 9751 sdk-java-contract-open-prs.yml
 9555 sdk-java-flyway-open-prs.yml
-114180 sdk-java.yml
+114901 sdk-java.yml
 3886 sdk-python.yml
 5595 security-checks.yml
 6777 serve-ab-publish.yml
@@ -450,7 +450,8 @@ jobs:
    # Pool routing per the flyway job (#13245): hosted queue starvation fails
    # a zero-step job outright (#13623), the pool does not. The job is
    # pool-safe: the database is a per-leg container on a random host port,
-    # and every Maven run holds the test job's per-host sdk-java lock.
+    # both Maven runs use a job-local repository, and only the shared cache
+    # write-back holds the test job's per-host sdk-java lock.
    # Both legs run the same non-Hosted *IT family, as before #14936 removed
    # the MariaDB service. The MariaDB leg is what keeps the published
    # MariaDB-only V12 repair (RTRIM, error 1901) and the schema catalog's
@@ -471,17 +472,16 @@ jobs:
            image: 'mariadb:10.11.18'
            client: 'mariadb'
            engine_args: '--innodb-use-native-aio=0'
-    # Each leg takes the per-host sdk-java lock once, around the single step
-    # that runs both Maven commands. Run 39353663670's verify
-    # acquisition waited the full 90 minutes after the install step released
-    # the lock, so a per-step lock let this job lose its place in the host
-    # queue between installing and verifying. The ceiling must cover that
-    # 90-minute wait, up to ~100 minutes from acquisition to the end of
-    # verify (job 118848599935: a 71-minute seed, 26 of Maven), the 5-minute
-    # write-back backstop and ~30 minutes of setup; 280 leaves margin. Run
-    # 39563394993 hit the old 200. Hosted lanes never take the lock
-    # and inherit this ceiling as a bound, not as an expected duration.
-    timeout-minutes: 280
+    # On the pool this job takes the per-host sdk-java lock only for its
+    # shared Maven cache write-back: both Maven runs resolve from a job-local
+    # repository, so holding the lock across the whole integration suite
+    # serialized the fast tier behind a multi-hour hold (run 39628985638 held
+    # it 15:54-18:57 while run 39627609266's fast-tier job burned its full
+    # 90-minute wait). The ceiling absorbs that wait, the ~3-hour Maven run,
+    # the 5-minute write-back backstop and 30 minutes of uncapped setup: 310.
+    # Hosted lanes never take the lock and inherit this ceiling as a bound,
+    # not as an expected duration.
+    timeout-minutes: 310
    env:
      MYSQL_CONTAINER: 'qwen-code-sdk-java-${{ matrix.database }}-${{ github.run_id }}-${{ github.run_attempt }}'
      DATABASE_IMAGE: '${{ matrix.image }}'
@@ -732,33 +732,17 @@ jobs:
          MAVEN_ARGS: '--settings ${{ runner.temp }}/setup-java-m2/settings.xml --toolchains ${{ runner.temp }}/setup-java-m2/toolchains.xml -Dmaven.repo.local=${{ runner.temp }}/m2-repo'
        run: |-
          set -o pipefail
-          # One acquisition spans both Maven runs: run 39353663670's verify
-          # step waited the full 90 minutes after the install step released
-          # the lock, so a per-step lock let this job lose its place in the
-          # host queue between installing and verifying. The ticketed queue
-          # in sdk-java-lock.sh orders that one wait (fast PR legs overtake
-          # it until it parks or is promoted) instead of a wake-up race
-          # against sibling holds (qwen-code#14743).
-          if [ -f "${GITHUB_WORKSPACE}/.github/scripts/sdk-java-lock.sh" ]; then
-            source "${GITHUB_WORKSPACE}/.github/scripts/sdk-java-lock.sh"
-            sdk_java_lock_acquire 5400
-          else
-            # release-sdk-java.yml resumes an existing tag by checking out
-            # that immutable commit, which may predate this helper. Fall
-            # back to the bare lock this queue replaced rather than dying
-            # before Maven and wedging the release.
-            mkdir -p "${HOME}/.cache/qwen-code-ci"
-            exec 9>"${HOME}/.cache/qwen-code-ci/sdk-java-tests.lock"
-            if ! flock --wait 5400 9; then
-              echo "::error::sdk-java host lock not acquired within 90 minutes"
-              exit 1
-            fi
-          fi
          # Seed from the host-shared repo (cache-restored on hosted runners)
          # so it and the egress proxy keep paying off for the job-local copy.
+          # The seed only reads the host cache and is best-effort, so it takes
+          # no lock and cannot delay the lane.
          cp -aln "${HOME}/.m2/repository" "${RUNNER_TEMP}/m2-repo" 2>/dev/null || true
-          # The shell owns the lock; Maven children must not extend its lifetime.
-          mvn --batch-mode --no-transfer-progress -f packages/sdk-java/qwencode/pom.xml -DskipTests -Dgpg.skip=true -Dmaven.javadoc.skip=true install 9>&-
+          # Both Maven runs resolve from the job-local copy above, so they do
+          # not touch host-shared state and must not hold the per-host lock:
+          # a MySQL verify can run for hours under pool contention, and the
+          # fast tier's 90-minute wait expires behind it (run 39627609266
+          # waited 17:38-19:15 behind run 39628985638's integration hold).
+          mvn --batch-mode --no-transfer-progress -f packages/sdk-java/qwencode/pom.xml -DskipTests -Dgpg.skip=true -Dmaven.javadoc.skip=true install
          cd packages/managed-agent-server
          # Keep mysql.url for release runs that check out an older tag.
          mvn --batch-mode --no-transfer-progress -Pmysql-integration \
@@ -766,15 +750,38 @@ jobs:
            -Dmysql.url="jdbc:mysql://127.0.0.1:${MYSQL_PORT_1}/managed_agent_test?createDatabaseIfNotExist=true&allowPublicKeyRetrieval=true&useSSL=false" \
            -Dmysql.url.1="jdbc:mysql://127.0.0.1:${MYSQL_PORT_1}/managed_agent_test?createDatabaseIfNotExist=true&allowPublicKeyRetrieval=true&useSSL=false" \
            -Dmysql.url.2="jdbc:mysql://127.0.0.1:${MYSQL_PORT_2}/managed_agent_test?createDatabaseIfNotExist=true&allowPublicKeyRetrieval=true&useSSL=false" \
-            -Dmysql.user=root -Dmysql.password=runtime-broker clean verify checkstyle:check 9>&-
-          # Close the cache loop inside the same lock: the one-way seed
-          # above alone does not recapture the Server dependency tree after
-          # a pom key rotation. The excludes name the two in-house
-          # fixed release coordinates rather than the whole com/alibaba
-          # group, so third-party artifacts under it (druid) still
-          # recapture into the host-shared ~/.m2. Best-effort like the
-          # seed: the warning arms keep a copy failure from reddening the
-          # gate after mvn has already passed.
+            -Dmysql.user=root -Dmysql.password=runtime-broker clean verify checkstyle:check
+          # Only the write-back mutates host-shared state, so only it takes
+          # the per-host lock. One acquisition spans the write-back: run
+          # 39353663670's verify step waited the full 90 minutes after the
+          # install step released the lock, so a per-step lock let this job
+          # lose its place in the host queue between installing and verifying.
+          # The ticketed queue in sdk-java-lock.sh orders that one wait (fast
+          # PR legs overtake it until it parks or is promoted) instead of a
+          # wake-up race against sibling holds (qwen-code#14743).
+          if [ -f "${GITHUB_WORKSPACE}/.github/scripts/sdk-java-lock.sh" ]; then
+            source "${GITHUB_WORKSPACE}/.github/scripts/sdk-java-lock.sh"
+            sdk_java_lock_acquire 5400
+          else
+            # release-sdk-java.yml resumes an existing tag by checking out
+            # that immutable commit, which may predate this helper. Fall
+            # back to the bare lock this queue replaced rather than dying
+            # before the write-back and wedging the release.
+            mkdir -p "${HOME}/.cache/qwen-code-ci"
+            exec 9>"${HOME}/.cache/qwen-code-ci/sdk-java-tests.lock"
+            if ! flock --wait 5400 9; then
+              echo "::error::sdk-java host lock not acquired within 90 minutes"
+              exit 1
+            fi
+          fi
+          # Close the cache loop inside that lock: the one-way seed above
+          # alone does not recapture the Server dependency tree after a pom
+          # key rotation. The excludes name the two in-house fixed release
+          # coordinates rather than the whole com/alibaba group, so
+          # third-party artifacts under it (druid) still recapture into the
+          # host-shared ~/.m2. Best-effort like the seed: the warning arms
+          # keep a copy failure from reddening the gate after mvn has already
+          # passed.
          # On a fresh ECS host nothing creates ~/.m2/repository anymore
          # (the hosted-only cache no longer restores it), so create it
          # before the write-back or every job re-resolves through the
@@ -784,10 +791,12 @@ jobs:
          # took up to 61 minutes (run 39563394993). Each link is atomic and
          # -n never replaces a host file, so the backstop can leave an
          # artifact missing for the next job to re-resolve, never truncated.
-          mkdir -p "${HOME}/.m2/repository"
+          # No write-back child inherits fd 9, or a cancelled job's survivor
+          # would hold the host lock until it exits.
+          mkdir -p "${HOME}/.m2/repository" 9>&-
          cd "${RUNNER_TEMP}/m2-repo"
          write_back=0
-          timeout 300 find . \( -path ./com/alibaba/qwencode-sdk -o -path ./com/alibaba/qwen-managed-agent-server \) -prune -o -type f -links 1 -exec cp -ln --parents -t "${HOME}/.m2/repository" {} + || write_back=$?
+          timeout 300 find . \( -path ./com/alibaba/qwencode-sdk -o -path ./com/alibaba/qwen-managed-agent-server \) -prune -o -type f -links 1 -exec cp -ln --parents -t "${HOME}/.m2/repository" {} + 9>&- || write_back=$?
          if [ "${write_back}" -eq 124 ]; then
            echo "::warning::Maven cache write-back stopped after 300 s; the gate result above is unaffected"
          elif [ "${write_back}" -ne 0 ]; then