Files
70be140e14 fix(release): reduce hosted runner queue delays (#15832)
* perf(release): reuse verified native audio prebuilds across releases

Every release rebuilds the five native audio prebuilds even when nothing
that affects those binaries changed. The build is short; the wait is not.
For v0.26.3 the Linux x64 prebuild queued 74 minutes to execute for 50
seconds, and the collection step queued again after it — and `publish`
depends on that collection, so the whole release sits behind it.

Fingerprint the audio build inputs and reuse an artifact from an earlier
successful official main release when the fingerprint matches. The key
covers the audio package, the dependency lock, the package manager pin and
the contents of the producing workflow and this helper, so the Node pin and
the prebuildify target arguments are part of it. The package version is
excluded deliberately: no part of the native build reads it, and including
it would make reuse impossible between releases.

Reuse means executing bytes an earlier run produced, so provenance is
checked rather than assumed: exact fingerprint name, unexpired, same
repository for both run and head run, and a completed successful
release.yml run on main from workflow_dispatch or schedule. The pinned
download-artifact already verifies the artifact digest.

Retention is checked differently at selection and consumption. Selection
requires 48 hours of headroom so the artifact outlives the release;
consumers require only that it has not expired. Re-applying the margin
against a later clock would reject an artifact the release had already
committed to, after the native matrix was skipped and with no fallback.

Lookup, trial download and payload validation are all non-fatal, and the
reuse ID is published only once a trial download has validated; an empty ID
runs the full matrix, which is today's behaviour. The publisher does not
trust that decision: it recomputes the fingerprint from its own checkout,
re-verifies the source run, and validates the payload again, fatally — by
then the matrix was skipped, so a bad artifact must stop the release.

This does not depend on ECS_RELEASE_HOST_RUNNER_ENABLED and takes effect
with the switch unset.

The collector needed three fixes to be a trustworthy producer. Its
`rm -rf -- "$GITHUB_WORKSPACE/packages/audio-capture/prebuilds"` resolved
through whatever `packages` was on a persistent host: with a leftover
symlink it deleted an external tree and exited 0, and the following
download wrote through the same symlink. It now wipes through a shared
preflight that canonicalises the workspace and refuses paths resolving
outside it. It also ran `node` with no Node contract, inheriting Node from
the hosted image by accident. And its release-only label pinned `x64` while
the runners holding that label are ARM64, which never matches and queues
until timeout.

Contract tests pin the wiring, each confirmed to fail against an injected
regression. Design doc in both languages.

* fix(release): move the release build and SDK build off the hosted queue

Behind the existing ECS_RELEASE_HOST_RUNNER_ENABLED switch, prepare the
version commit and produce the CLI build outputs on an isolated release
runner, and split the TypeScript SDK into an isolated producer plus the
existing hosted publisher. Eligible Linux downstream jobs join the same
pool. With the switch unset every path is exactly today's behaviour.

The final npm publication stays on GitHub-hosted runners. npm trusted
publishing and provenance require a cloud-hosted runner, so moving the
publisher itself would break the release rather than speed it up — only
the build work ahead of it can leave the queue.

Each migrated job runs one shared preflight before checkout, referenced
as a '$/' self-repository action so it resolves at the running commit
without a prior checkout. Release-only labels carry no architecture
component: the runners holding that label are ARM64, and an "x64"
component never matches instead of falling back to hosted. Jobs that
genuinely need amd64 — the OSS mirrors and the managed runtime, which
execute a hardcoded linux-amd64 ossutil — keep their hosted path, tracked
in #15847. The VSCode build routes by matrix.universal so only the
dependency-free leg migrates.

The SDK publisher consumes the exact tested tarball through a same-run
artifact and refuses to overwrite different already-published bytes on
retry. Audio prebuild reuse is the base of this branch and is specified
separately; the isolated build job is a second consumer of that artifact
and is held to the same re-verification contract as the publisher.

Rebased onto the audio prebuild reuse branch, so that mechanism and its
contract tests are no longer duplicated here.

* fix(release): reconcile audio reuse workflow contracts

Share the pre-checkout wipe, preserve the workflow size cap, and let fingerprint failures rebuild native audio.

Co-authored-by: Qwen-Coder <[email protected]>

* fix(ci): use block-style release permissions

Co-authored-by: Qwen-Coder <[email protected]>

* fix(release): drop x64 from publish's release-host expression

The review's unresolved item: collect got the ARM64 label fix but publish kept
["self-hosted", "linux", "x64", "ecs-qwen-release-host"], which can never
match the ARM64 machines holding that label, so the OIDC-carrying job would
queue forever once ECS_RELEASE_HOST_RUNNER_ENABLED is true. Same expression as
the collect fix, same one-word drop.

* docs(release): clarify audio reuse rollout and validation

Co-authored-by: Qwen-Coder <[email protected]>

* docs(release): reserve runner slots on existing ECS hosts

* fix(release): quote audio workflow scalar values

---------

Co-authored-by: Qwen-Coder <[email protected]>
Co-authored-by: zzj <[email protected]>
2026-10-30 01:08:54 +08:00
..