Files
70be140e14 fix(release): reduce hosted runner queue delays (#15832)
* perf(release): reuse verified native audio prebuilds across releases

Every release rebuilds the five native audio prebuilds even when nothing
that affects those binaries changed. The build is short; the wait is not.
For v0.26.3 the Linux x64 prebuild queued 74 minutes to execute for 50
seconds, and the collection step queued again after it — and `publish`
depends on that collection, so the whole release sits behind it.

Fingerprint the audio build inputs and reuse an artifact from an earlier
successful official main release when the fingerprint matches. The key
covers the audio package, the dependency lock, the package manager pin and
the contents of the producing workflow and this helper, so the Node pin and
the prebuildify target arguments are part of it. The package version is
excluded deliberately: no part of the native build reads it, and including
it would make reuse impossible between releases.

Reuse means executing bytes an earlier run produced, so provenance is
checked rather than assumed: exact fingerprint name, unexpired, same
repository for both run and head run, and a completed successful
release.yml run on main from workflow_dispatch or schedule. The pinned
download-artifact already verifies the artifact digest.

Retention is checked differently at selection and consumption. Selection
requires 48 hours of headroom so the artifact outlives the release;
consumers require only that it has not expired. Re-applying the margin
against a later clock would reject an artifact the release had already
committed to, after the native matrix was skipped and with no fallback.

Lookup, trial download and payload validation are all non-fatal, and the
reuse ID is published only once a trial download has validated; an empty ID
runs the full matrix, which is today's behaviour. The publisher does not
trust that decision: it recomputes the fingerprint from its own checkout,
re-verifies the source run, and validates the payload again, fatally — by
then the matrix was skipped, so a bad artifact must stop the release.

This does not depend on ECS_RELEASE_HOST_RUNNER_ENABLED and takes effect
with the switch unset.

The collector needed three fixes to be a trustworthy producer. Its
`rm -rf -- "$GITHUB_WORKSPACE/packages/audio-capture/prebuilds"` resolved
through whatever `packages` was on a persistent host: with a leftover
symlink it deleted an external tree and exited 0, and the following
download wrote through the same symlink. It now wipes through a shared
preflight that canonicalises the workspace and refuses paths resolving
outside it. It also ran `node` with no Node contract, inheriting Node from
the hosted image by accident. And its release-only label pinned `x64` while
the runners holding that label are ARM64, which never matches and queues
until timeout.

Contract tests pin the wiring, each confirmed to fail against an injected
regression. Design doc in both languages.

* fix(release): move the release build and SDK build off the hosted queue

Behind the existing ECS_RELEASE_HOST_RUNNER_ENABLED switch, prepare the
version commit and produce the CLI build outputs on an isolated release
runner, and split the TypeScript SDK into an isolated producer plus the
existing hosted publisher. Eligible Linux downstream jobs join the same
pool. With the switch unset every path is exactly today's behaviour.

The final npm publication stays on GitHub-hosted runners. npm trusted
publishing and provenance require a cloud-hosted runner, so moving the
publisher itself would break the release rather than speed it up — only
the build work ahead of it can leave the queue.

Each migrated job runs one shared preflight before checkout, referenced
as a '$/' self-repository action so it resolves at the running commit
without a prior checkout. Release-only labels carry no architecture
component: the runners holding that label are ARM64, and an "x64"
component never matches instead of falling back to hosted. Jobs that
genuinely need amd64 — the OSS mirrors and the managed runtime, which
execute a hardcoded linux-amd64 ossutil — keep their hosted path, tracked
in #15847. The VSCode build routes by matrix.universal so only the
dependency-free leg migrates.

The SDK publisher consumes the exact tested tarball through a same-run
artifact and refuses to overwrite different already-published bytes on
retry. Audio prebuild reuse is the base of this branch and is specified
separately; the isolated build job is a second consumer of that artifact
and is held to the same re-verification contract as the publisher.

Rebased onto the audio prebuild reuse branch, so that mechanism and its
contract tests are no longer duplicated here.

* fix(release): reconcile audio reuse workflow contracts

Share the pre-checkout wipe, preserve the workflow size cap, and let fingerprint failures rebuild native audio.

Co-authored-by: Qwen-Coder <[email protected]>

* fix(ci): use block-style release permissions

Co-authored-by: Qwen-Coder <[email protected]>

* fix(release): drop x64 from publish's release-host expression

The review's unresolved item: collect got the ARM64 label fix but publish kept
["self-hosted", "linux", "x64", "ecs-qwen-release-host"], which can never
match the ARM64 machines holding that label, so the OIDC-carrying job would
queue forever once ECS_RELEASE_HOST_RUNNER_ENABLED is true. Same expression as
the collect fix, same one-word drop.

* docs(release): clarify audio reuse rollout and validation

Co-authored-by: Qwen-Coder <[email protected]>

* docs(release): reserve runner slots on existing ECS hosts

* fix(release): quote audio workflow scalar values

---------

Co-authored-by: Qwen-Coder <[email protected]>
Co-authored-by: zzj <[email protected]>
2026-10-30 01:08:54 +08:00

308 lines
13 KiB
YAML

name: 'Finalize Stable Release'
on:
release:
types: ['published']
workflow_dispatch:
inputs:
tag:
description: 'The stable release tag to finalize (e.g., v0.19.10).'
required: true
type: 'string'
concurrency:
group: 'finalize-stable-release-${{ github.event.release.tag_name || inputs.tag }}'
cancel-in-progress: false
jobs:
finalize:
name: 'Finalize Stable Release'
runs-on: '${{ (github.repository == ''QwenLM/qwen-code'' && vars.ECS_RELEASE_HOST_RUNNER_ENABLED == ''true'') && fromJSON(''["self-hosted", "linux", "ecs-qwen-release-host"]'') || fromJSON(''["ubuntu-latest"]'') }}'
if: |-
${{ github.repository == 'QwenLM/qwen-code' && (github.event_name == 'workflow_dispatch' || github.event.release.prerelease == false) }}
environment:
name: 'production-release'
permissions:
contents: 'read'
pull-requests: 'write'
env:
RELEASE_TAG: '${{ github.event.release.tag_name || inputs.tag }}'
steps:
- uses: '$/.github/actions/prepare-release-runner'
- name: 'Validate stable release tag'
id: 'meta'
env:
TAG: '${{ env.RELEASE_TAG }}'
run: |-
is_stable="false"
if [[ "${TAG}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
is_stable="true"
fi
echo "is_stable=${is_stable}" >> "${GITHUB_OUTPUT}"
echo "release_branch=release/${TAG}" >> "${GITHUB_OUTPUT}"
if [[ "${is_stable}" != "true" ]]; then
echo "::error::${TAG} is not a stable release tag; finalize handles stable tags only."
exit 1
fi
- name: 'Checkout release branch'
if: |-
${{ steps.meta.outputs.is_stable == 'true' }}
uses: 'actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10' # v6.0.3
with:
token: '${{ secrets.CI_BOT_PAT }}'
ref: '${{ steps.meta.outputs.release_branch }}'
fetch-depth: 0
persist-credentials: false
- name: 'Setup Node.js'
if: |-
${{ runner.environment != 'self-hosted' && steps.meta.outputs.is_stable == 'true' }}
uses: 'actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e' # v6.4.0
with:
node-version-file: '.nvmrc'
package-manager-cache: false
- name: 'Install Dependencies'
if: |-
${{ steps.meta.outputs.is_stable == 'true' }}
env:
NPM_CONFIG_PREFER_OFFLINE: 'true'
run: |-
if [ -f pnpm-lock.yaml ]; then
corepack pnpm install --frozen-lockfile --ignore-scripts --prefer-offline --reporter=append-only
else
npm ci --ignore-scripts --no-audit --progress=false
fi
# Replay the root postinstall (patch-package) and commit-info
# generation; dependency and workspace lifecycle scripts remain
# disabled.
npm run postinstall
npm run generate
- name: 'Configure Git User'
if: |-
${{ steps.meta.outputs.is_stable == 'true' }}
run: |-
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git config core.hooksPath .husky
- name: 'Resolve previous stable release tag'
if: |-
${{ steps.meta.outputs.is_stable == 'true' }}
id: 'previous'
env:
TAG: '${{ env.RELEASE_TAG }}'
run: |-
previous_tag=""
found_current="false"
while IFS= read -r stable_tag; do
if [[ "${found_current}" == "true" ]]; then
previous_tag="${stable_tag}"
break
fi
if [[ "${stable_tag}" == "${TAG}" ]]; then
found_current="true"
fi
done < <(git tag --list 'v*' --sort=-version:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$')
if [[ -z "${previous_tag}" ]]; then
echo "::error::Could not resolve the stable release before ${TAG}."
exit 1
fi
echo "tag=${previous_tag}" >> "${GITHUB_OUTPUT}"
- name: 'Generate AI-assisted release notes'
if: |-
${{ steps.meta.outputs.is_stable == 'true' }}
continue-on-error: true
timeout-minutes: 35
env:
GITHUB_TOKEN: '${{ secrets.CI_BOT_PAT }}'
OPENAI_API_KEY: '${{ secrets.OPENAI_API_KEY }}'
OPENAI_BASE_URL: '${{ secrets.OPENAI_BASE_URL }}'
OPENAI_MODEL: '${{ secrets.OPENAI_MODEL }}'
PREVIOUS_RELEASE_TAG: '${{ steps.previous.outputs.tag }}'
RELEASE_BRANCH: '${{ steps.meta.outputs.release_branch }}'
RELEASE_NOTES_FILE: '${{ runner.temp }}/release-notes.md'
RELEASE_TAG: '${{ env.RELEASE_TAG }}'
run: |-
# The first publish keeps GitHub-generated notes; this improves them after the tag exists.
node scripts/generate-release-notes.js \
--repo="${GITHUB_REPOSITORY}" \
--tag="${RELEASE_TAG}" \
--previous-tag="${PREVIOUS_RELEASE_TAG}" \
--target="${RELEASE_BRANCH}" \
--output="${RELEASE_NOTES_FILE}"
- name: 'Update GitHub Release notes'
if: |-
${{ steps.meta.outputs.is_stable == 'true' }}
continue-on-error: true
env:
GITHUB_TOKEN: '${{ secrets.CI_BOT_PAT }}'
RELEASE_NOTES_FILE: '${{ runner.temp }}/release-notes.md'
RELEASE_TAG: '${{ env.RELEASE_TAG }}'
run: |-
if [[ -s "${RELEASE_NOTES_FILE}" ]]; then
gh release edit "${RELEASE_TAG}" --notes-file "${RELEASE_NOTES_FILE}"
else
echo "::warning::AI release notes were not generated; keeping GitHub-generated notes."
fi
- name: 'Comment released-in version on merged PRs'
if: |-
${{ steps.meta.outputs.is_stable == 'true' }}
continue-on-error: true
env:
GITHUB_TOKEN: '${{ secrets.CI_BOT_PAT }}'
PREVIOUS_TAG: '${{ steps.previous.outputs.tag }}'
RELEASE_TAG: '${{ env.RELEASE_TAG }}'
run: |-
set -euo pipefail
# Squash-merge commit subjects carry the PR as "(#NNN)"; collect those between the two tags.
# The trailing "|| true" keeps "no matches" (empty range) from failing under pipefail.
pr_numbers="$(git log --pretty=format:'%s' "${PREVIOUS_TAG}..${RELEASE_TAG}" | grep -oE '\(#[0-9]+\)$' | tr -d '()#' | sort -un || true)"
if [[ -z "${pr_numbers}" ]]; then
echo "No PR references found between ${PREVIOUS_TAG} and ${RELEASE_TAG}."
exit 0
fi
echo "Found PRs between ${PREVIOUS_TAG} and ${RELEASE_TAG}:"
echo "${pr_numbers}"
marker='<!-- qwen-release-comment:v1 -->'
release_url="https://github.com/${GITHUB_REPOSITORY}/releases/tag/${RELEASE_TAG}"
while IFS= read -r num; do
if [[ -z "${num}" ]]; then
continue
fi
# Re-run safety: skip PRs that already carry this release marker.
existing="$(gh pr view "${num}" --json comments --jq '.comments[].body' 2>/dev/null || true)"
if grep -qF "${marker}" <<<"${existing}"; then
echo "PR #${num} already has a release comment; skipping."
continue
fi
body="${marker}"$'\n'"Released in [${RELEASE_TAG}](${release_url})."
gh pr comment "${num}" --body "${body}" && echo "Commented on PR #${num}." || echo "::warning::Failed to comment on PR #${num}."
done <<< "${pr_numbers}"
- name: 'Regenerate CHANGELOG.md'
if: |-
${{ steps.meta.outputs.is_stable == 'true' }}
env:
GH_TOKEN: '${{ secrets.CI_BOT_PAT }}'
BRANCH_NAME: '${{ steps.meta.outputs.release_branch }}'
RELEASE_TAG: '${{ env.RELEASE_TAG }}'
run: |-
set -euo pipefail
gh auth setup-git
node scripts/generate-changelog.js
git add CHANGELOG.md
if git diff --cached --quiet -- CHANGELOG.md; then
echo "CHANGELOG.md already up to date."
else
git commit -m "docs(changelog): sync for ${RELEASE_TAG}"
git push origin "${BRANCH_NAME}"
fi
- name: 'Create PR to merge release branch into main'
if: |-
${{ steps.meta.outputs.is_stable == 'true' }}
id: 'pr'
env:
# Author the PR as the review bot (a third identity) so that NEITHER
# approve PAT (ci-bot / dev-bot) is the author: GitHub forbids
# self-approval, and the two approve steps below need both PAT
# identities free to supply the branch protection's two required
# approvals without a human. GITHUB_TOKEN cannot be used here
# because the org disables GitHub Actions from creating PRs.
GITHUB_TOKEN: '${{ secrets.CI_REVIEW_BOT_PAT }}'
RELEASE_BRANCH: '${{ steps.meta.outputs.release_branch }}'
RELEASE_TAG: '${{ env.RELEASE_TAG }}'
run: |-
set -euo pipefail
# Merge the release branch back to main so package metadata and CHANGELOG converge.
pr_data="$(gh pr list --head "${RELEASE_BRANCH}" --base main --state all --json url,state --jq '(map(select(.state == "MERGED"))[0] // map(select(.state == "OPEN"))[0]) // empty | [.url, .state] | @tsv')"
pr_url=""
pr_state=""
if [[ -n "${pr_data}" ]]; then
IFS=$'\t' read -r pr_url pr_state <<< "${pr_data}"
fi
if [[ "${pr_state}" == "MERGED" ]]; then
echo "Release PR already merged; skipping approval and auto-merge."
echo "PR_URL=${pr_url}" >> "${GITHUB_OUTPUT}"
echo "SHOULD_MERGE=false" >> "${GITHUB_OUTPUT}"
exit 0
fi
if [[ "${pr_state}" != "OPEN" ]]; then
# `gh pr create` runs right after the branch push and release
# creation; wait so GitHub doesn't reject the burst with its
# "was submitted too quickly" secondary rate limit.
sleep 30
pr_url="$(gh pr create \
--base main \
--head "${RELEASE_BRANCH}" \
--label 'skip-changelog' \
--title "chore(release): ${RELEASE_TAG}" \
--body "Automated release PR for ${RELEASE_TAG}. Syncs package.json versions and CHANGELOG.md on main.")"
fi
gh pr update-branch "${pr_url}"
echo "PR_URL=${pr_url}" >> "${GITHUB_OUTPUT}"
echo "SHOULD_MERGE=true" >> "${GITHUB_OUTPUT}"
- name: 'Approve release PR'
if: |-
${{ steps.meta.outputs.is_stable == 'true' && steps.pr.outputs.SHOULD_MERGE == 'true' }}
env:
GITHUB_TOKEN: '${{ secrets.CI_DEV_BOT_PAT }}'
PR_URL: '${{ steps.pr.outputs.PR_URL }}'
run: |-
set -euo pipefail
gh pr review "${PR_URL}" --approve --body "Automated approval for the release version bump."
# Second approval uses CI_BOT_PAT which MUST be a different GitHub user
# from both CI_REVIEW_BOT_PAT (PR author) and CI_DEV_BOT_PAT (first
# approver). GitHub rejects self-approval at the API level. If this step
# fails with "Can not approve your own pull request", rotate CI_BOT_PAT
# to a distinct account. The step is non-blocking so that auto-merge can
# still proceed when a human has already approved or when branch
# protection only requires one approval. See #14341.
- name: 'Approve release PR (second bot)'
id: 'second_approve'
if: |-
${{ steps.meta.outputs.is_stable == 'true' && steps.pr.outputs.SHOULD_MERGE == 'true' }}
continue-on-error: true
env:
GITHUB_TOKEN: '${{ secrets.CI_BOT_PAT }}'
PR_URL: '${{ steps.pr.outputs.PR_URL }}'
run: |-
set -euo pipefail
gh pr review "${PR_URL}" --approve --body "Automated second approval for the release version bump."
- name: 'Warn on second-bot approve failure'
if: |-
${{ steps.meta.outputs.is_stable == 'true' && steps.pr.outputs.SHOULD_MERGE == 'true' && steps.second_approve.outcome == 'failure' }}
run: |-
echo "::warning::Second-bot approve failed — CI_BOT_PAT may be the same GitHub user as the PR author (CI_REVIEW_BOT_PAT). Rotate CI_BOT_PAT to a distinct account. See https://github.com/QwenLM/qwen-code/issues/14341"
- name: 'Enable auto-merge for release PR'
if: |-
${{ steps.meta.outputs.is_stable == 'true' && steps.pr.outputs.SHOULD_MERGE == 'true' }}
env:
GITHUB_TOKEN: '${{ secrets.CI_BOT_PAT }}'
PR_URL: '${{ steps.pr.outputs.PR_URL }}'
RELEASE_TAG: '${{ env.RELEASE_TAG }}'
run: |-
set -euo pipefail
gh pr merge "${PR_URL}" \
--squash \
--auto \
--subject "chore(release): ${RELEASE_TAG} [skip ci]"