mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-11 04:02:45 -10:00
feat/btw-qodercli-alignment
224
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Shaojin Wen
|
37e29fd656 |
fix(ci): read prettier's whole ignore set at the base tip (#16729)
A Critical landed with the previous change and was reproduced on the merged tree. The prettier narrowing fetched .gitignore through the same helper as .prettierignore, so it read the newest commit touching .prettierignore rather than the base tip. .gitignore owns no gate verdict of its own, so a later base-side change to it was invisible: the equality held, the subset held, and a branch whose prettier scope was narrower than the base gate's was certified fresh. Three-ref probe on the merged module returns no stale file; the same probe on this one returns the verdict. Rather than patch the ref for one file, stop guessing which files matter. Prettier discovers .gitignore and .prettierignore at every depth — the lane groups by the nearest ancestor carrying one, and the repository tracks 29 of them — so the narrowing now reads both recursive trees and compares the whole ignore-file set by path and blob SHA. Everything except the root .prettierignore must be identical, which blob SHAs answer without fetching content. A truncated tree never narrows. This closes the nested-file gap the same review raised, at the same call count. Also from that review's deferred list: Extraction failure returned false, so it never reached the catch that emits the warning — the design promised an annotation on that path and did not deliver one. It throws now. The Chinese design still said two content fetches while its own constraints bullet said four; both languages now describe the tree calls. The eslint real-file drift guard checked anchors but not negations, so a bang in the global array would have disabled that narrowing silently too. The relocated-block shape did not reproduce: every placement leaves the files property in the byte-compared remainder. Four regression cases pin that rather than adding a brace matcher for a shape already covered. |
||
|
Shaojin Wen
|
56ea1bc3fb |
fix(ci): stop staling branches for eslint ignores they never excluded (#16706)
* fix(ci): stop staling branches for eslint ignores they never excluded
The lint gate freshness check treats every gate file as indivisible, so a
base-side addition to the global eslint ignores reddens every in-flight PR.
All nine freshness reds in a measured eight-hour window named one such
addition, and because the lane checks out the branch head alone, the same
commit can never turn green on a rerun — the only remedy was merging main
and restarting the whole matrix.
Adding an ignore only shrinks what the base checks. Lint scope is
tree \ ignores, so a branch's green still covers the base gate exactly when
the branch's ignores are a subset of the base's. Narrow the stale verdict on
that condition, comparing everything outside the array byte for byte so a
change that also tightens a rule still fails. A base-side removal fails the
subset test and keeps reddening the branch.
The subset argument needs entries to be purely subtractive, and a negated
glob re-includes files. Measured through lintFiles on eslint 9.29.0, a base
carrying '!**/keep.gen.ts' lints a file the branch never checks while still
being a superset of it, so refuse to narrow when either side has a negated
entry. isPathIgnored is not a usable oracle for this: it reports both shapes
as ignored.
Fail closed throughout. A fetch or extraction failure keeps the verdict
instead of reaching main()'s fail-open, which would discard every other gate
file's verdict too, and the two content fetches happen only on a verdict that
was already going to fail.
* fix(ci): refuse escaped negations and narrow prettierignore too
Review round 1 found three gaps, all reproduced before fixing.
The negation guard compared source text while the consumer evaluates the
decoded pattern, so an entry written '\x21foo' passed startsWith('!') and
still re-included files — the exact unsoundness the guard existed to close.
Refuse any backslash rather than decoding, since decoding would evaluate
branch-controlled text and the dep-free lane has no parser.
The raw media type and main()'s fetchText wiring were pinned by nothing:
every case injected its own fetcher, so flipping the media type to JSON or
deleting the wiring each left the suite fully green while the refinement
silently stopped firing behind the fail-closed catch. Export the media type
and pin both, one of them by reading the source like the ci.yml wiring case.
The subset argument is a property of any purely subtractive scope list, not
of one filename. .prettierignore decides what prettier --check covers and its
last three commits were pure additions, so the same treadmill ran on it.
Dispatch the narrowing per file instead of hardcoding eslint.config.js.
Self-audit of that extension caught a fourth issue: the gitignore-style
parser trimmed both ends, but prettier treats leading whitespace as part of
the pattern — 'foo.js' and ' foo.js' ignore different files, and ' # foo' is
a pattern rather than a comment. Strip trailing whitespace only.
* docs(ci): correct the isPathIgnored rationale and the scoped-ignores count
Two published claims were wrong; both came from review of the previous
commit and neither affects behaviour.
The guard's rationale said isPathIgnored reports the re-including and
non-re-including shapes alike. It does not. The probe behind that claim used
a config holding only an ignores object, so nothing matched the target
extension and ESLint ignored the file for lack of a matching configuration.
With a config object that matches, isPathIgnored distinguishes the pair
correctly; it is blind only to directory-pruned shapes. The guard is
unchanged and still rests on the lintFiles measurement.
The designs also said "the two config-scoped ignores arrays". There are
five, three of them not array literals, which is why an earlier grep for
`ignores: [` counted only two beside the global one. All five stay in the
byte-compared remainder, so the behaviour claim was right and only the
count was wrong.
* fix(ci): split on engine terminators and pin prettier's second ignore file
Round 2 review. The Critical is real and reproduced: the extractor split on
\n alone, so a branch line written `// x<U+2028> 'hidden/**',` was skipped
whole as a comment while the engine ended the comment at U+2028 and read a
real entry. The extractor under-reported the branch's own ignores, the
remainder matched, the subset held, and a branch ignoring strictly more than
the base was certified fresh. Same for U+2029 and a lone CR. Split on the
engine's terminator set instead.
The report's second shape — base moving the array into a files-scoped object
— did not reproduce: the files property lands in the byte-compared
remainder, which already rejects it in all three placements tried. Added a
regression case pinning that, rather than a brace matcher for a shape the
remainder compare already covers.
Prettier 3 reads .gitignore as well as .prettierignore: measured on 3.6.1,
adding apps/host/ there drops the checked-file count from 2 to 1. The
narrowing now fetches both and requires .gitignore byte-identical. It is not
folded into the subset test — the real .gitignore carries a dozen bang
re-inclusions, so a semantic comparison would reject every input and leave
the narrowing dead from day one. The new real-file guard caught that during
implementation.
Also from this round: a drift guard so a negated line in the real
.prettierignore reddens instead of silently disabling the narrowing, plus
the two deferred items — a warning annotation on the fail-closed path so a
stopped refinement leaves a trace, and a pin on the JSON half of the fetch
seam.
|
||
|
|
2aac39a17b |
ci: route the web-shell visuals capture for fork PRs to the disposable pool (#16617)
* ci: route the Test aggregate gate and web-shell visuals capture to the disposable pool Both jobs match the ecs-qwen-ephemeral pool's security model: the Test aggregate runs one shell line and checks out nothing, and the visuals capture builds untrusted PR code with a read-only token and no long-lived secrets. Routing them off the hosted queue removes the 9-minute waits the one-line gate saw after its shards finished, and takes a 45-minute render job off hosted capacity. The MAINTAINER_ECS_RUNNER_DISABLED kill-switch reverts both to hosted. The E2E OpenTUI leg deliberately stays hosted: it carries long-lived model secrets, which the disposable pool's boundary forbids until a separate trusted pool exists. Co-authored-by: Qwen-Coder <[email protected]> * ci: complete the pool split for the visuals capture and sync the design docs Review follow-up on #16617. The capture kept the unconditional setup-node download, which can fail outright on the pool where nodejs.org may be unreachable: gate it to hosted, add the self-hosted Node preflight and the disk-floor gate (the tui-parity shape), and pin all of it in the routing battery. The slim image also lacks the DejaVu font baseline and ffmpeg; one best-effort self-hosted apt step installs both so a blocked Debian archive degrades to fallback fonts and .webm-only recordings instead of sinking the preview. Record the migration in both design-doc languages (superseding the browser-dependency exclusion for this job), refresh the deployed slot count, and document that the aggregate gate intentionally no longer follows the dispatch fleet inputs. Extend the secret scan to the bracket-indexed access form and drop the no-op sameRepo loop. Co-authored-by: Qwen-Coder <[email protected]> * test(ci): pin the ffmpeg install line and the degradation branch Both assertions matched text that survives the change they exist to catch: /ffmpeg/ also matches the degradation warning's own wording, so dropping the package from the apt list stayed green, and /::warning:: only proved the string appears somewhere in the step. - Match the install statement itself: `apt-get install … ffmpeg` on one line, so only the package list satisfies it. - Require the warning to sit in the guard's failure branch (`else` today, `||` accepted). That branch is what makes the step degrade instead of sinking the job — the step carries no continue-on-error. Verified by mutation on web-shell-visuals.yml: dropping ffmpeg from the apt list, deleting the else branch, and removing the guard entirely each turn this suite red (110 -> 109 passed) at the new assertion lines; every revert restores the workflow byte-for-byte. * fix(ci): drop fonts-dejavu-mono from the pool apt step and qualify the degradation claim The pool image is node:22-bookworm-slim (Debian 12, now oldstable), where fonts-dejavu-mono has no build — the mono face already ships inside fonts-dejavu-core. One unknown package fails the whole apt transaction before anything installs, so the step would have taken its warning branch on every pooled capture and installed neither the fonts nor ffmpeg. The workflow comment and both design docs also overclaimed the degradation contract: the earlier `playwright install --with-deps` step installs apt packages without continue-on-error, so a blocked Debian archive sinks the preview rather than degrading it. The best-effort wording now covers only the extra font/ffmpeg install this step performs. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): route the required gate with the shards and harden the pooled capture The required Test aggregate queued 110 minutes on the manually sized disposable pool with an empty runner (job 128729488158) while sibling jobs were served in ~3; it now follows the classifier's ubuntu_runner like the sibling aggregates, so the kill-switch also works on fork runs. The first pooled capture runs measured every remaining review finding: pnpm ENOSPC after the shared 2 GiB floor passed an 89%-used slot (run 42478651795), a runner lost to CPU/memory starvation from unpinned Playwright workers (run 42494910412), and a 45-minute cap cancelled before any render with installs alone at 22m18s. The capture now carries a lane-calibrated floor (8 GiB / 85%), --workers=2 on both render passes, and a routed 90/45 timeout. The apt step gets bounded calls and a post-install font check; its comment no longer claims the archive itself degrades. Pins: the secrets guard walks parsed expressions (toJSON(secrets) and bracket forms now fail) and bounds job-level permissions; the pool label derives from the pool config; the whole apt package list, the disk-floor overrides, the routed timeout and both worker pins are asserted. Six single-point mutations (mono re-add, workers drop, bare 45, floor drop, toJSON(secrets), label rename) all red the suite. Size baseline updated for the reviewed capture growth. Suites: routing 110/110, platform-lanes + workflow-size + main-ci-failure-issue 498 passed / 1 pre-existing skip, Prettier clean. Co-authored-by: Qwen-Coder <[email protected]> * ci: record the visuals capture's routed timeout in the CI budget The budget gate counts expression timeouts as unbounded by design; the routed 90/45 ceiling moves web-shell-visuals.yml from a fixed 45 to one expression-timeout job, exactly the entry the gate's error names. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): keep the pooled visuals capture fork-only The capture's runs-on was rewritten in |
||
|
易良
|
2a191acfce |
ci: drop the hz-5 leg that no runner can pick up (#16666)
* ci: drop the hz-5 leg that no runner can pick up The update matrix routes every leg by a dedicated per-host label, and no runner in the fleet carries ecs-update-hz-5: all 29 hz5 machines are ephemeral container runners labelled only self-hosted and ecs-qwen-ephemeral. That is the same shape hz4 has, and hz4 is already excluded from the matrix for exactly that reason. The leg therefore queues until GitHub cancels it, and the failure reporter deliberately ignores cancelled runs, so the pool would have gone stale silently. Drop the leg, its actionlint label declaration, and the host from the workflow test's expectations. Also name every hk leg in that test: the shape check accepts any digit string, so renaming one hk leg together with its declaration survived the whole suite while that leg queued forever. * test(ci): pin the actionlint declaration to a matrix leg in both directions A declaration left behind after a partial revert re-authorises a label no runner carries, and the suite stayed green. Also record the workflow's current size in the ratchet so the next PR that edits it is not charged for drift it did not cause. * test(ci): read the actionlint declarations with comments stripped Both halves of the pairing grepped the raw file, so a commented-out declaration counted twice over: it satisfied the forward check for a leg that had lost its declaration, and it tripped the reverse check as a label declared with no leg. One stripped read, shared by both directions, so a comment is a comment. |
||
|
Shaojin WenandQwen-Coder
|
0dd106d42f |
feat(ci): give the autofix address agent failed-check evidence (#16659)
* feat(ci): give the autofix address agent failed-check evidence The review loop selects a PR when its checks are red, but the agent that fixes it receives check names only and holds no GitHub credentials, so a shard name or a Test check that failed on a lint step costs a round of guessing. The CI Failure Patrol already distils job logs for its classifier; this wires the same redaction and line selection into the address feedback. A collector prints the failed step names plus a bounded, redacted log excerpt per failing job (first three jobs, 60 lines each, 8 MiB buffer). It runs in prepare before the PR branch is checked out, so the collector and the patrol helper it imports execute from the trusted base. Selection mirrors the workflow's failed-check gates; every failure degrades to less evidence, never a failed round. Adds behavioral tests, workflow wiring pins with mutation witnesses, and a bilingual design doc. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): render the autofix check evidence against a current gh Review found the log fetch dead on a current gh: it refuses the jobs/<id>/logs endpoint without --allow-escape-sequences (exit 1, empty stdout), and the helper swallowed that into null, so a real round got Failed steps and no excerpt. Apply the three-part shape from autofix-pr/ci-log.ts: pass the flag, fall back to the bare call only on an unknown-flag stderr, and stripVTControlCharacters so raw ANSI never reaches feedback.md or step stdout. The fake gh now reproduces the refusal, so a missing flag turns the suite red. Also address the non-blocking notes: keep skillLog's whole ~120-line selection so the failure-summary tail survives a noisy log; size the excerpt fence past any backtick run; make the selector comment precise about Still-red vs Failed checks (CANCELLED); and apply the same fetch fix to the patrol's GhClient.jobLog, which was equally log-blind. Tighten the tests the verify report flagged as unpinned (usage guard per argument, 80-char name cap, realistic StatusContext). Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
易良andQwen-Coder
|
7d99e7d26e |
fix(ci): cover hz hosts in the fleet qwen update and pin both prefixes (#16619)
* fix(ci): cover hz hosts in the fleet qwen update and pin both prefixes The update matrix only carried the five ecs-update-hk-N legs, so every hz host kept serving 0.26.0 after 0.27.0 shipped. Add one leg per hz host (routed by runner name, no re-registration needed; hz4 is ephemeral-only), install into both /usr/local and the legacy /usr prefix the hz images carry, and verify both absolute binary paths so a stale /usr/bin copy can never hide behind the PATH winner again. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): make the legacy-prefix install shape-independent and pin behavior npm exits EEXIST when /usr/bin/qwen is a regular file or a symlink into another prefix, so remove that bin path (only after the /usr/local install succeeded) before installing into /usr, and skip the second install entirely where no legacy copy exists — hk hosts no longer grow a second 156 MB tree nothing reads. -L catches the dangling-symlink shape -e alone would skip. Sharpen the pins per review: read step bodies with comments stripped so commenting a command out turns red, assert the verify loop actually tests the version, pin each leg's routing mode (hz name / hk label), and span both routing families in the stale-pool reporter fixture. All three green mutations from the review now turn red. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): route hz legs by dedicated labels and gate the retry on a pre-loop snapshot GitHub schedules self-hosted jobs by runner label, never by runner name, so all five name-routed hz legs could only queue until the 24 h auto-cancel — which the failure reporter deliberately ignores. Each hz host's first runner now carries a dedicated ecs-update-hz-N label (added via the labels API; pool labels untouched), the matrix is a static array so actionlint can check every label, and the labels are declared in .github/actionlint.yaml. The retry loop's early exit read the CURRENT existence of /usr/bin/qwen, which the loop's own rm -f had just removed: after a transient /usr install failure the next attempt exited 0 with the binary deleted, and every later run repeated the silent skip. Snapshot legacy presence once before the loop, export it through GITHUB_ENV, and fail Verify version when a copy that existed at the start is missing at the end. The pinned verify loop now uses -e || -L, the same definition of exists as the update step, so a dangling symlink fails instead of skipping the pin. The suite moves with the workflow: the routing contract now pins label-only legs against the actionlint inventory, and new sudo/npm-stub replays exercise the transient-failure retry, the exhausted-retry error, the no-legacy single install, and the dangling-symlink path end to end (28/28). Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): let npm --force replace the legacy bin and pin verify's failure direction The pre-emptive rm of /usr/bin/qwen deleted the binary even when all three /usr install attempts failed; the next run then snapshotted 'no legacy copy', skipped /usr and verified green forever. npm --force replaces a foreign bin only on a successful install and leaves it untouched on failure, so a failed leg now keeps the old CLI and a rerun recovers both prefixes (R1-2 cross-run half, pr-verify F6). Also pin the verify step's failure direction behaviorally: a stale second copy and a vanished legacy copy must both exit non-zero (R2-1, pr-verify F7). The replay stub now mirrors real npm's EEXIST/--force semantics, and a two-run replay on one host tree covers the cross-run recovery. Suite: 31/31. Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
易良andQwen-Coder
|
64f93e55b4 |
fix(ci): check out all of .github for owner assignment (#16599)
* fix(ci): check out all of .github for owner assignment A reused ECS workspace can be left in non-cone sparse-checkout mode by another job, and git sparse-checkout set reuses the stored mode. The narrow .github/scripts pattern then drops issue-owners.json and the assignment script fails with ENOENT. Checking out .github covers the scripts and the owner map under either mode. Co-authored-by: Qwen-Coder <[email protected]> * test(ci): pin the widened .github sparse checkout in the meta-tests Update the three workflow pins to the new .github pattern and widen the auto-minimize-spam checkout the routing pin also covers: its blocklist sits directly under .github/ and the same non-cone residue would drop it, silently disabling spam minimization. Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
易良andQwen-Coder
|
cbc6164c26 |
feat(ci): switch review runner day pool to a numeric count over hz runners (#16544)
* feat(ci): switch review runner day pool to a numeric count over hz runners Migrate the review runner schedule off hk1/hk2 and onto the hz1-hz6 fleet, replacing the QWEN_REVIEW_DAY_HOSTS runner-name list with a single QWEN_REVIEW_DAY_RUNNER_COUNT number. The old scheme made the operator maintain a space-separated list of runner names, defended by a runtime typo check; the new scheme picks the first N online runners by name sort, fails loudly if N exceeds the online fleet, and is deterministic across runs. Default day pool size is 40. Manual workflow_dispatch with pool=day reconverges against the current var value, so an operator can change the count and re-apply it without waiting for the next 05:00 cron. Co-authored-by: Qwen-Coder <[email protected]> * style(ci): apply prettier to review runner schedule script Co-authored-by: Qwen-Coder <[email protected]> * feat(ci): drive fleet by pattern var, filter by capability, soften day cap Address review feedback on PR #16544: - Read QWEN_REVIEW_FLEET_NAME_PATTERN from the environment so a region change is a var edit, not a code change. Default stays on the hz fleet. - Filter the managed set by capability: a runner must carry the X64 label (the architecture review jobs target) and must not carry a reserved label such as ecs-qwen-release-host. Without this, the day pool nominally sized at 40 actually has 24 usable slots because hz1-17..hz1-32 are ARM64-only, and the planner would also try to relabel the dedicated release hosts. - Strict integer parsing for the day count: a /^\d+$/ gate before Number() so trailing garbage (4O, 40.9, 1e2) cannot silently coerce and shrink the daytime pool. - Soft cap instead of hard fail when dayCount exceeds the online fleet: truncate to whatever is online and warn in the run summary. The previous behaviour left every runner in its previous pool for up to 12 hours when a stale var overran reality. - Restore the per-name "keeping on review" listing in the step summary so operators can see which hosts are in the day pool without re- running the planner by hand. - Drop hk1/hk2 references from the schedule workflow and the review workflow's runs-on comment; the script no longer cares about regions. Add an end-to-end day-mode CLI test that drives the planner through a fake gh and asserts the label calls and the kept-names summary. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): address PR #16544 review feedback on the runner scheduler - day mode now reserves one online runner for CI: the keep budget is onlineManaged - 1, so an over-large count can no longer consume the whole fleet and leave the ecs-qwen pool empty for the daytime half (review R1-1). - The truncation warning is emitted as a ::warning:: workflow command on its own stdout line instead of an inert bullet inside the step summary, and now says "eligible and online" (R1-1, R1-7). - The empty-fleet guard splits: name-mismatch keeps the original message, while a pattern-matching but capability-ineligible fleet gets an error that names the X64/reserved-label filter (R1-7). - Runners that still carry a managed label but are no longer eligible are listed in the step summary as orphaned (labels untouched) (R1-3). - The workflow no longer duplicates the fleet-pattern default; an unset repository variable falls through to the script's single fallback (R2-1). The 05:00 cron comment now names QWEN_REVIEW_DAY_RUNNER_COUNT instead of the removed REVIEW_DAY_HOSTS (R1-4). - Tests: drop the vacuous X64+ARM64 assertion (R1-5), re-add the day-mode demotion case (R1-2), route the day-mode end-to-end test through spawnPlanner (R2-2), and pin the truncation summary plus the stdout ::warning:: line (R2-3). Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
|
3016bf8453 |
fix(ci): keep sdk-java resolver bookkeeping and in-house jars out of the host cache (#15281) (#15286)
* fix(ci): scope the sdk-java test-job lock to the cache write-back (#15281) * fix(ci): keep sdk-java SDK resolution job-local (#15281) * test(cli): keep malformed-Unicode case titles JUnit-safe (#15281) The parametrized malformed-Unicode cases interpolate lone surrogates into the test title. Vitest's JUnit reporter drops a lone low surrogate together with the preceding character — the semicolon of an ' entity — which makes packages/cli/junit.xml invalid and fails the CI report publisher. Render the case as JSON in the title and keep the raw value in the body. * fix(ci): unlink seeded Maven metadata before the unlocked suite (#15281) Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): harden the SDK Java host-cache isolation (#15281) Widen the seeded resolver-bookkeeping unlink to every seed site and the release lane, prune pre-PR in-house coordinates inside the locked write-back, and keep the suite's exit status so a failed leg still publishes its downloads. Extend the workflow meta-tests to pin each invariant and update the bilingual design record. Co-authored-by: Qwen-Coder <[email protected]> * test(ci): pin the first mvn failure guard in the SDK Java suite-status test (#15281) * fix(ci): harden SDK Java write-back isolation and lock verdicts (#15281) Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): keep Maven resolver bookkeeping out of the shared cache (#15281) * chore(ci): restore the size baseline final newline (#15281) Co-authored-by: Qwen-Coder <[email protected]> * test(managed-agent): align workspace upload admission probes (#15281) The two Workspace upload routes joined the WORKSPACE_FILES rule class without extending the acceptance probe's request builder, so the generic JSON/empty-body request was rejected by argument binding before admission. Send the octet-stream upload its required query parameters and the status query its uploadId, and pin the status route's own not-found code for a stranger. * fix(ci): close remaining Maven cache isolation gaps Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): update Maven cache cleanup contract Co-authored-by: Qwen-Coder <[email protected]> * test(ci): create runner temp for Maven seed replay Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: qwen-autofix <[email protected]> Co-authored-by: Qwen-Coder <[email protected]> Co-authored-by: Shaojin Wen <[email protected]> Co-authored-by: yiliang114 <[email protected]> |
||
|
Shaojin Wen
|
dfc37dffa4 |
fix(ci): run the failure patrol on ECS and time its sandbox image pull separately (#16462)
* fix(ci): run the failure patrol on ECS and pull the sandbox image outside the classifier budget
The patrol had effectively stopped. Of its last 20 scheduled rounds, 17 were
cancelled with zero jobs — they never received a runner at all — and one round
waited 2h09m in the queue. The single round that did run spent 304s in the
classifier, hit the 5-minute step timeout and produced no decisions, yet still
reported success: the timeout is continue-on-error and Validate only warns, so
a completely ineffective patrol looked green.
Both jobs move to the ecs-qwen pool behind the existing
MAINTAINER_ECS_RUNNER_DISABLED kill switch with a hosted fallback, using the
same routing expression as the E2E workflow. That pool had 199 idle hosts while
the patrol queued. Schedule and dispatch both check out github.sha, so there is
no fork-PR trust gate to add.
The sandbox image is resolved and pulled in its own timed step ahead of the
classifier, so the 5-minute budget belongs to the model alone. Whether the
observed 304s went to a cold image pull or to inference is NOT established; the
timing notice, reported on failure too, is what settles it. The hard-coded
0.19.9 CLI pin becomes the version the job actually runs — on the pool, the
preinstalled one — so the agent no longer forces an image the fleet does not
keep warm. Resolving the CLI is gated on the scan finding a candidate, so empty
ticks on the hosted fallback do not pay for a global npm install.
Sharing a persistent workspace required two guards. The runner allowlist is now
the first step of both jobs, ahead of the step that hands CI_BOT_PAT to the
runner, and classify preflights the docker daemon so a host without one fails
in seconds instead of burning the classifier budget. Scan clears the previous
round's decisions file and agent home: otherwise a stale file would pass
Validate and be acted on even when this round's classifier timed out without
writing anything.
The classify job cap rises from 10 to 20 minutes to cover the resolver's own
10-minute pull budget plus scan and the classifier; a lower cap would kill a
cold pull mid-flight. Cron and the classifier's 5-minute cap are unchanged, so
a worst-case cold round can still cost one tick to the concurrency group.
The workflow grows past its size allowance because the two guards are inlined,
matching how the autofix and repo-hygiene workflows already carry theirs;
.size-baseline is bumped in this commit rather than extracting a shared script
that would refactor three unrelated workflows.
* fix(ci): bound the patrol's image resolve step and pin the directory it deletes
Two findings from an adversarial pass over the previous commit.
The image resolver can attempt two sequential ten-minute pulls: the requested
image, then a fallback to the latest GHCR semver tag. The twenty-minute job cap
added in the previous commit therefore did not cover the resolve step alone,
let alone scan and the classifier, and a cold host would have been killed
mid-round with nothing produced. The resolve step now carries its own
twelve-minute cap below the job, mirroring the bound the classifier already
had, so a cold host costs one round instead of the job. The budget assertion
in the meta-test encoded a single pull and was wrong; it now requires both
bounded steps to fit inside the job cap with room left for scan.
The scan step deletes the previous round's agent home with rm -rf, and took
that directory's name from a job-level variable. On the shared pool
$GITHUB_ENV is writable by anything running at the runner's uid, which is the
threat model the image resolver's own step-file guards already encode. The
name is now pinned in the env of both steps that use it, so the destructive
command reads a step-level value, and a test binds the two spellings together
and rejects a job-level definition.
Also pins the gate shared by the CLI, image and classifier steps: if those
three conditions ever desync, the classifier runs with an empty image
reference or an empty CLI version instead of being skipped.
Five further mutations targeting only these new assertions — reordering the
CLI install ahead of the scan, dropping the resolve cap, raising it to the
whole job, drifting the two agent-home spellings apart, and desynchronizing
the classifier gate — were all caught, with the workflow restored byte-exactly
after each.
* fix(ci): close the patrol's persistent-runner credential paths and the sandbox consumer gate
Four review findings, all reproduced locally before fixing.
The always() reset step defeated the new runner allowlist. Reset hands
CI_BOT_PAT to a driver script read out of the workspace, and always() fires
even when the guard rejected the host or checkout failed — so on a persistent
runner the allowlist did not actually close the write path. The guard and
checkout now carry ids and reset requires both to have succeeded, keeping
always() so a failed act still resets state.
A reused workspace also carried executable Git metadata. The classifier's
sandbox mounts the workspace read-write and its two file tools do not exclude
.git, so a previous round needed nothing but write_file to plant a filter that
this round's checkout executes during its own clean/reset — with HEAD still
resolving to the trusted SHA, which is why a pinned ref and
persist-credentials: false do not help. Both jobs now discard .git before
checkout and let it rebuild, as triage already does, and the classifier denies
writes to .git config and hooks so the same-round post-checkout git commands
are covered too. The regression test plants a real filter and asserts the
payload fires without the rebuild, so it cannot pass vacuously.
The docker preflight had no bound of its own. A daemon that accepts the
connection and never answers hangs before the scan, where neither the resolve
nor the classifier step cap applies, so even an empty-candidate round would
hold the serialized patrol for the whole job budget. The probe is now wrapped
in a ten-second timeout under a two-minute step cap, with a test that stubs a
wedged daemon and asserts the guard returns.
Adding a resolver step also enrolled this workflow in a gate whose protected
set is derived from the tree, and that gate's consumer detector only recognized
a run: body or env.SETTINGS_JSON — not a uses: step binding
env.QWEN_SANDBOX_IMAGE. The workflow therefore contributed zero consumers and
failed the closing vacuity assert, which CI confirmed on the required Lint &
Static lane. The detector now recognizes that binding shape; no other workflow
gains a newly detected consumer, and the workflow is deliberately not exempted,
which would have disabled every binding check for it permanently.
Six mutations targeting only these fixes were all caught, including narrowing
the detector back, which reproduces the exact CI failure.
* fix(ci): make the patrol guard testable off Linux and probe the daemon it vouches for
Three review findings on the previous commit, each witnessed against the
acceptance criteria the reviewer specified.
The guard behavioural test executes the real step body, which now ends in
`timeout 10 …`, but only the wedged-daemon test carried a capability probe. On
a host without coreutils timeout the approved-runner case therefore failed
deterministically: bash reports 127, `if !` inverts it, and the guard exits 1
where the test expects 0. That suite is collected by the macOS scripts shard
and by Windows, and neither lane runs on pull_request, so the first red would
have been a merge-queue ejection. The allowlist fixture now ships a forwarding
timeout shim, which keeps those assertions live on every platform, and the
suite joins the win32 exclusion list beside its bash-driven siblings — its name
does not match the existing qwen-* glob. Witnessed as an A/B under a PATH whose
timeout exits 127: with the shim the suite is green, without it exactly one
case fails and the other eighteen still pass.
The preflight also probed a different daemon and a different binary than the
steps it vouches for. It ran a PATH-resolved `docker` with no endpoint pin,
while the classifier pins DOCKER_HOST/DOCKER_CONTEXT and the resolver spawns an
absolute path. Because $DOCKER_CONFIG/config.json is shared across pool
registrations, the guard could reject a host whose default daemon is healthy,
or pass on one whose default daemon is dead and let the classifier burn its
budget. The guard now carries the same two pins and probes through
SANDBOX_COMMAND, and a test binds both to the consumer's values so the three
cannot drift. The behavioural stub answers only for the pinned endpoint, so
dropping either pin from the workflow reds the approved-host case; witnessed,
along with the structural test.
The two new Git-metadata rebuilds were also outside the cross-workflow
symlink-safety registry, with a subset of its assertions re-implemented inline.
That registry runs the real body against a workspace whose .git is a symlink
and asserts the external target survives, which the inline copy could not
distinguish: a trailing-slash rewrite follows the link and empties the target
while `existsSync('.git') === false` still holds. Both jobs are now registered
and the duplicated structural loop is gone, keeping the planted-filter fixture
that the registry does not cover. Witnessed: the trailing-slash mutation reds
exactly the two new rows.
The wedged-daemon test gained a lower time bound as well. Pointing the probe at
a stub through SANDBOX_COMMAND means an unset variable would otherwise fail
instantly on an empty command name and satisfy the upper bound without any
timeout ever firing.
* fix(ci): pin the patrol's command lookup and neutralise global Git config
Four review findings on the previous head. Each was verified against the code
before fixing, and each fix is witnessed against the acceptance criterion its
reviewer specified.
The allowlist decided which host runs the job but said nothing about which
binary a credential-bearing step then executed. Resolve patrol identity ran a
PATH-resolved gh with CI_BOT_PAT already in its environment, and act and reset
did the same through a PATH-resolved node. This pool runs untrusted fork CI at
the same uid and the workflow itself writes a PATH directory via npm install
-g, so a co-located job can plant either name. Both jobs now capture PATH once
through expression context and every later run step exports it before its
first command word; the classifier takes it as step env, because the CLI
resolves its sandbox command as a bare name too. The capture sits after
setup-node on purpose — capturing earlier would pin the ambient node and
silently drop the version the job asked for — and identity moved below it,
which costs nothing since it needs no checkout. Validate is covered as well:
it holds no token, but a planted node there could rewrite the driver that act
then runs with the PAT.
Deleting the repository's .git also left the wider Git configuration untouched.
A core.hooksPath planted in ~/.gitconfig survives that deletion, and
actions/checkout copies $HOME/.gitconfig into its temp home before running git
checkout --force, so a planted post-checkout hook fires during the very
checkout the rebuild exists to make trustworthy and can rewrite the driver
before act runs it. Both rebuild steps now write an empty config under
RUNNER_TEMP and export GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM, and both
checkout steps pin the same two in step env, which outranks the appendable
$GITHUB_ENV copy. Neither touches the developer's or runner's own ~/.gitconfig.
The preflight also vouched for a binary the classifier does not use.
SANDBOX_COMMAND reaches only the image resolver; the CLI returns the bare name
docker from sandboxConfig and spawns it through PATH, so a planted docker
passed the absolute-path probe and would have run the classifier on the host
instead of in a container. The guard now refuses the round unless command -v
docker is the pinned binary. And it no longer discards the probe's own
diagnostic: a missing binary (127), a non-executable one (126) and a socket
permission failure are not hangs, and reporting all of them as a ten-second
daemon timeout pointed every round at a healthy daemon.
Two items the review recorded as deferred are fixed here rather than carried:
the job cap omitted the guard's own two minutes, so the capped steps summed to
19 of a 20-minute budget before scan and setup — raised to 25 with a test that
sums every capped step; and nothing pinned the resolver's image argument, so
dropping it would have left a green suite over a total patrol outage. The
Git-filter fixture is also isolated from the developer's global config, where
commit.gpgsign would otherwise red a test about workflow contamination.
Six mutations targeting these fixes were all caught, each hitting both the
structural and the behavioral assertion where both exist. The shared checkout
registry fixture now provides RUNNER_TEMP and GITHUB_ENV, which a real runner
always sets and the widened rebuild body needs.
* fix(ci): establish a verified execution base for the patrol on the shared pool
Root-cause fix for the fourth review round, whose findings kept regenerating
siblings because each earlier patch closed one instance rather than the shared
cause: on a persistent pool where untrusted fork CI runs at the same uid, any
name resolved through PATH and any config read from a predictable path is
attacker-controlled.
The first step now establishes an execution base using shell builtins only. A
candidate absolute path is accepted only when the job uid can write neither it
nor its containing directory; a symlink is accepted when the link's directory,
the resolved target and the target's directory are all non-writable. A blanket
symlink refusal was over-strict and a pool probe showed it rejecting node on a
correctly provisioned host. The guard publishes those paths and a system_path of
verified directories, and every later step uses them: the Git rebuild runs the
verified rm and mktemp, checkout gets only system_path, and credential-bearing
steps invoke absolute gh and node. The driver spawned a bare gh itself, so it
takes an explicit path too — pinning only the workflow's PATH would have left
that child resolving through the plantable one.
gh's own config channel is closed on every PAT-bearing step, following the
autofix preamble: pin GH_HOST, unset the enterprise and ambient tokens, and mint
a fresh GH_CONFIG_DIR per step with a failing mktemp aborting rather than
leaving gh to fall back to the shared ~/.config/gh. The gitconfig neutralisation
likewise mints an unpredictable name instead of truncating a guessable one.
The docker probe is now unconditional and goes through the verified base, and a
separate step re-checks the bare-name resolution on the exact PATH the
classifier receives, since setup-node prepends to it after the guard runs.
Two defects the pool probe exposed are fixed here as well. The classifier hit a
five-minute cap after a two-second image pull, so the model alone needs more
than five minutes: its cap is raised to ten and the job cap to thirty-five,
which covers every capped step plus the measured uncapped ones. And a round that
found candidates but produced no decisions still reported success — the exact
shape that hid twenty ineffective rounds and that the probe reproduced. A final
step now fails such a round loudly, after the evidence upload, while a round
with no candidates stays a green no-op.
Ten mutations target these controls and all ten are caught, each reverted
byte-exactly: removing or softening the verdict step, restoring either known-bad
budget, dropping the symlink branch, dropping the gh preamble, returning the
driver to a bare gh, restoring the guessable gitconfig name, removing checkout's
verified PATH, and removing the post-capture equality check. Three of those were
initially MISSED and the assertions were added before claiming coverage; the
driver's spawn had no test at all because every existing GhClient test stubs it.
* fix(ci): keep the patrol pull timing when the resolver outlives its budget
The resolve step is capped at 12 minutes, but the resolver can attempt two
sequential 10-minute pulls plus the tag fetch. In that worst case the runner
killed the step, so the elapsed-time notice never ran — losing the measurement
in precisely the slow case the separate step exists to surface. Bound the
resolver inside the step with the guard's verified timeout instead: the shell
regains control at 660s, reports the duration and exit 124, and still fails the
round. Verified on this host that coreutils timeout signals the child's process
group, so the docker pull is reaped rather than orphaned.
Also pin the ECS routing by evaluation rather than substring. `== 'true'`
contains every term the old includes() assertions looked for, so an inverted
kill switch stayed green. The routing gate now evaluates both patrol jobs for
the switch on, the switch off and a foreign repository, and the workflow suite
pins the whole expression against the canonical pool routing that gate
evaluates.
Refresh the two comments that still described the classifier's old 5-minute
budget, and bump the size baseline for the added prose (39678 -> 40452).
* test(ci): strip comments before pinning the patrol's gh reroute channel
A mutation witness on the final head commented out `export GH_HOST=github.com`
in all four token-bearing steps and the gate stayed green: the assertion used
toContain against the raw run body, which a commented-out line still satisfies.
The stage assertion in the same test already filters comment lines, so use that
idiom here too. With it, the commented-out pin reds the gate.
Found while re-running the consolidated mutation matrix; the production steps
were already correct, this only closes the hole in their regression gate.
* fix(ci): close the round-4 patrol findings at their shared root
Three Criticals, all the same class — a trust anchor that still resolved
something through a channel the step exists to distrust:
- readlink is an external command, so the builtin-only predicate was resolving
its own trust anchor through the plantable PATH. Bootstrap it with builtins
first and invoke it absolutely in both jobs; a planted readlink runs with
GITHUB_OUTPUT and GITHUB_ENV in its environment and could both vouch for a
writable symlink target and append its own output key.
- The scan resets resolved rm, mkdir and grep as bare names on a step that
declares no PATH. A planted rm exiting 0 without deleting leaves the previous
round's decisions file to be replayed by act, which the verdict step cannot
catch because has_decisions is true. Bind the guard's verified absolutes and
add mkdir and grep to its tool list; grep writes the provenance anchor.
- An unguessable git config name does not protect a LISTABLE file: a same-uid
co-tenant can enumerate RUNNER_TEMP, match the minted name and refill it
before git reads it. Use /dev/null for both scopes, as repo-hygiene does.
Also address the eight Suggestions: delete act's consumer-less capture step and
its misleading comment; state the real node precedence and keep the hosted
toolchain winning there, since leading with the root-owned system prefix would
fail Ensure Qwen Code's global install; pin the verdict operator and evaluate
the workflow's own expression instead of a hand-copied replica; pin PATROL_GH
and NODE_BIN on the driver-spawning steps; cap Ensure Qwen Code at 5 minutes;
make has_decisions a content verdict so an empty decision list fails the round;
and gate the POSIX-only driver spawn test off the Windows lane.
The accept path is now tested: mode bits alone reach it for a non-root uid, so
the earlier claim that it needed root-owned files was wrong. The new case reads
GITHUB_OUTPUT back, covers the symlink accept and refusal, isolates usable()'s
directory check from the readlink bootstrap's own, and plants a lying readlink.
14 targeted mutations all caught on this tree, byte-exact restore, baseline
green. 496 passed | 1 skipped across 7 suites; 134/134 shared node gates.
* fix(ci): close the round-5 patrol findings, including a macOS lane regression
R5-1 was a regression this PR introduced: three fixtures hardcoded
RM_BIN '/usr/bin/rm', which macOS does not have, and the vitest exclude is
win32-only so the macOS scripts lane collects them. Resolve rm and mktemp with
command -v instead. Witnessed against the real step body with an absent binary:
bash -e exits 127 with .git still present, bash -c exits 0 with .git present —
matching the reported shape exactly.
R5-2: git layers env-provided config above both file scopes, so neutralising
the config files left core.hooksPath reachable through GIT_CONFIG_COUNT and
GIT_CONFIG_PARAMETERS. Pin both in the checkout step's own env in both jobs, as
qwen-autofix already does.
R5-3: the actionable-round gate accepted any non-empty array while the consumer
silently dropped entries matching no candidate, so a non-empty but unmatchable
list still produced a green round that acted on nothing — the verdict step is
skipped once has_decisions is true. Close the loop in actOnDecisions: count
matches and throw when a non-empty list matched nothing. Per-decision failures
stay swallowed, and a matched-but-failed decision still counts.
R5-4: pin both guard copies, not just classify's. Dropping gh from act's loop
did not even abort, because no act step interpolates GH_BIN and the driver falls
back to a bare-name spawn. Also assert every consumed guard output is published
by that job's own loop.
R5-5: cap the capture and identity steps at 1 minute and check the output path
is a regular file before appending, so a planted FIFO cannot block them until
the job cap and conclude the round cancelled. Job cap 35 -> 36 keeps the
documented headroom now that two more steps are capped.
R5-6: gate the accept-path case on the uid alone and only the symlink arms on a
readlink -f capability probe, so macOS still pins the accept verdict and
publish()'s writes. Probed: the base accept arm invokes readlink zero times.
9 targeted mutations caught, byte-exact restore; the two not witnessable on
Linux carry the macOS-model probes above. 500 passed | 1 skipped across 7
suites, 134/134 shared node gates, size ratchet and lint clean.
* fix(ci): count actions not key matches, and stop minting a plantable gh config
Both round-6 Criticals are defects in this PR's own recent fixes.
R6-1: matched incremented as soon as the key resolved, but actOnDecision opens
with validDecision and returns silently. A list carrying all five identity
fields with a 240-char reason, rerun at low confidence, or string-typed ids
matched every key, acted on nothing, and still reported success — the exact
hollow green the throw was added to end. Validate the schema before counting
and name the rejected decision on stderr. Superseded, already-handled and
at-cap targets still count as deliberate declines, an attempted action that
threw still counts, and no_action counts because it posts a marker.
R3-2: the GH_CONFIG_DIR minted last round sits under the uid-writable, listable
$RUNNER_TEMP at 0700 owned by the job uid, and is reused by every gh call in the
step, so a config.yml written between calls captures the next Authorization
header. This is the construction the same diff rejects for the Git config sixty
lines above. Point all four steps at a path the job uid cannot create and drop
the now-unused mktemp from them and from the guard's tool list. Probed with gh:
/nonexistent/... falls back to defaults with http_unix_socket empty and creates
nothing; /dev/null/... was tried too and makes gh fail outright, so it is not used.
The planted-gh behavioural test now asserts the step leaves nothing in
RUNNER_TEMP, and deliberately still supplies MKTEMP_BIN so a regression to
minting would SUCCEED there — without that the assertion passed for the wrong
reason and only the structural pin caught it.
5 mutations caught, byte-exact restore. 503 passed | 1 skipped across 7 suites,
138/138 shared node gates, size ratchet and lint clean.
* fix(ci): declare the residue threat model and pin its channels, not the next one
Round 7's finding is not an eighth defect: against a live same-uid process no
userspace check holds, because the guard publishes its verdict through
$GITHUB_OUTPUT, which that process can append to (the runner takes the last
value for a key). Six rounds each closed one channel and found another.
So declare the model instead of implying an unbounded guarantee. The workflow
now states it defends against RESIDUE from earlier jobs — planted binaries,
config files, $GITHUB_ENV entries, uncleaned trees — and explicitly does not
claim to hold against a process running as the same uid during the job.
Closing that needs a real boundary (per-job container, dedicated uid, or an
unshared runner) and is out of scope here.
Stopgaps for the residue channels, each verified rather than assumed:
- Re-verify every published binary AT THE POINT OF USE, so a forged output
value fails before the exec instead of running with the PAT.
- usable() and both sibling arms reject a candidate the job uid OWNS:
ownership implies chmod, and under the residue model a uid-owned file's
content was authored by the attacker regardless of its mode.
- The checkout overrides the REPOSITORY scope via GIT_CONFIG_KEY_*, which the
reset cannot reach because it deletes .git and checkout re-creates it.
Probed: with only the file scopes pinned the planted post-checkout fires;
with these entries it does not. Also pin GIT_EXEC_PATH, GIT_TEMPLATE_DIR,
GIT_DIR and GIT_WORK_TREE.
- Pin BASH_ENV/SHELLOPTS/LD_*/NODE_OPTIONS on every PAT step and the
classifier, since they take effect before the body's first line.
- Pin SANDBOX and QWEN_SANDBOX on the classifier: sandboxConfig.ts returns ''
when SANDBOX is set and lets QWEN_SANDBOX override settings, so residue
there would switch the sandbox off and run the agent's tools on the host.
- Pin npm_config_userconfig/globalconfig, and reset QWEN_HOME again
immediately before the classifier instead of leaving it unattended through
the install and image pull. Job cap 36 -> 37 keeps the headroom invariant.
Two model-level tests, both with positive controls: a forged node= in the
outputs file must fail before the binary runs, and a re-planted repository
core.hooksPath must not execute during checkout.
Adding the ownership rule made the accept arm unconstructible for a same-uid
fixture, so it now runs against the real system roots and skips where the
host lacks that toolset — it skips here, and that is reported, not hidden.
8 of 9 mutations caught; the ninth (empty hooksPath override) was probed and
is a benign mutation, not a gap. 504 passed | 2 skipped, 138/138 node gates.
* fix(ci): keep the patrol's hosted fallback alive and pin its interpreter
The point-of-use recheck applied the pool's ownership/writability predicate
unconditionally, so it refused exactly the uid-owned tool-cache node the
guard's hosted branch exists to publish. Every round on ubuntu-latest aborted
at Scan, turning the emergency routing lever into a hard outage. Gate those
arms on runner.environment the way the guard gates them, and require every
declared name to be present: a blank PATROL_GH used to be skipped, which sends
the driver back to a bare `gh` lookup on the plantable PATH.
The interpreter was the one entrance the guard could not cover, because it is
chosen before the guard's first line. Pin it absolutely, and extend the same
predicate to the values nothing re-checked (rm, timeout, system_path) and to
npm/qwen, whose bare resolution decided whether to install and whose stdout
became the version pin the classifier runs. Pin the loader, proxy and trust
store channels on the node/gh consumers that lacked them.
The pre-pull needs no docker endpoint pins: every spawn inside the resolver
already goes through sandboxSpawnEnv(), verified against a rogue currentContext
and a hostile DOCKER_HOST.
* fix(ci): unset the lowercase proxy variants instead of duplicating env keys
GitHub env keys are case-insensitive, so pinning HTTPS_PROXY beside
https_proxy is a duplicate key: actionlint rejects the file and the whole
Lint & Static lane reds on it. A plain YAML parse does not see it because
the keys differ as strings, which is how it reached CI. Keep the uppercase
pins and unset the lowercase variants in the body, next to the existing
token unset — gh reads them at exec time, not at shell startup, so the
channel closes at the same point.
Add a local AST walk for case-insensitive duplicates so the class is caught
without depending on actionlint being installed, with a non-vacuity bound on
the number of env blocks it actually reaches.
* fix(ci): give npm two distinct config scopes and pin the test runner family
Pinning both npm config scopes to /dev/null makes npm abort with
`double-loading config "/dev/null" as "global", previously loaded as
"user"` before it resolves anything — even `npm --version` exits 1. That
killed the install branch, which is the entire hosted fallback, and every
step behind it. Use two distinct non-creatable paths instead, the same
construction the gh config dir already uses: away from HOME, and not
plantable because / is root-owned. A new arm runs the real npm under the
step's own values with a residue $HOME/.npmrc present.
The resolve-step and validate harnesses spread the ambient environment, so
on the ECS fleet RUNNER_ENVIRONMENT=self-hosted reached the step body and
the new point-of-use recheck refused their uid-owned stubs. That was the
required Test lane's red. Pin the hosted policy in both fixtures; the
ownership predicate keeps its own tests and is not loosened.
* fix(ci): pin the sandbox proxy command the classifier actually executes
The residue fence pinned SANDBOX and QWEN_SANDBOX but not
QWEN_SANDBOX_PROXY_COMMAND, which serve/sandbox.ts interpolates unquoted
into the proxy container's host-side command line and spawns with bash -c.
A residue value is shell syntax, not data: arbitrary commands at runner uid
inside the step holding OPENAI_API_KEY, plus a proxy network the sandbox's
HTTPS_PROXY is repointed at. The runtime refusal only fires under
tools.executionSandbox; this step sets tools.sandbox.
Also close three defects in this branch's own new tests, found while
auditing the deferred list: READLINK_BIN was executed with no point-of-use
recheck; the recheck-loop count was a floor one below the real count, so a
whole loop could vanish unnoticed; and the planted-qwen arm refused at npm
and never reached the qwen predicate, so it now drives the real predicate
on qwen alone with a hosted positive control. The npm arm reports a skip
instead of a green when npm is absent.
|
||
|
|
274c8a5283 |
fix(ci): report timed-out main CI runs instead of letting cancelled hide them (#16280)
* fix(ci): report timed-out main CI runs instead of letting cancelled hide them
main-ci-failure-issue.yml only admitted `failure` conclusions, but a job that
hits its execution cap concludes `cancelled` — so timed-out main runs never
filed an issue. That is how #16214 (the E2E docker leg rebuilding the sandbox
image past its 60-minute cap on cold mainland hosts) hid for weeks (#16272).
Admit `cancelled` at the trigger, then discriminate in the live step: a
cancelled run is reportable only when a job carries the execution-time-timeout
annotation; a plain concurrency supersede is skipped, and an unreadable
annotation fails safe toward not reporting. The download projections include
cancelled jobs so the timed-out leg is named in the report. The infra
classifier is unchanged, so a pure-timeout run (no failure jobs) files rather
than reruns, and normal failure handling is untouched.
Meta-tests: the trigger pin and routing matrix now reflect the admitted
cancelled conclusions; three new live-step tests witness timeout→report,
concurrency→skip, and unreadable→skip; the download-selector and jobs-fetch
pins are updated, as is the ci-platform-lanes trigger pin.
* fix(ci): classify cancelled runs by positive evidence, fail open when unknown
Review on #16280: real concurrency cancels carry the runner shutdown / higher-
priority / operation-canceled annotations, not just "operation was canceled",
and a timed-out job can carry a shutdown annotation alongside the timeout. The
previous discriminator only looked for the timeout text and skipped everything
else, so it rested on "absence of timeout = concurrency" and dropped unreadable
cancellations — re-creating the hiding problem #16272 fixes.
Classify by positive evidence on both sides instead: a timeout annotation
reports (even with a coexisting shutdown annotation); a concurrency annotation
with no timeout skips; neither, or unreadable jobs/annotations, fails open and
reports like the rest of the live step. Fixtures cover the shutdown and
higher-priority concurrency forms, timeout+shutdown coexistence, and the
fail-open paths; breaking the concurrency signature reddens the skip tests.
* fix(ci): address #16280 review — trust gate, renderer, successor dedup, docs
R1-4 (security): only trusted triggers (push/schedule/dispatch on main) run the
cancellation classifier; PR-triggered cancellations stay on the supersede path
so contributor-authored annotation text cannot drive the PAT-backed reporter.
R1-1: renderFailureComment now keeps cancelled jobs, so a timed-out leg is named
in the issue/PR comment even when no job concluded failure (the #16214 shape).
R1-2: factor the classifier into cancelled_class with a total_count receipt and
explicit unreadable handling; unknown fails open with a warning instead of
asserting a supersede it never established.
R1-3: a timeout-backed cancelled successor owns the older run's result (same
shared classifier), preventing a duplicate report; log wording corrected.
R1-6: the classifier runs after the stale-attempt check so a stale event is
attributed to staleness, not to a cancellation cause.
R1-7: the test stub resolves the check-run id (job id != check-run id) and
applies the caller's --jq with real jq; RUN_CONCLUSION env is pinned; fixtures
cover shutdown / higher-priority / coexisting timeout / unreadable / PR arms.
R1-5: sync the two design docs (both languages) with the new behaviour.
* fix(ci): move the cancelled-run classifier out of the workflow body
main-ci-failure-issue.yml grew to 63933 bytes, 6793 over its recorded
57140 (allowance 4096), so the workflow-size ratchet failed. Move the new
cancelled_class helper into .github/scripts/ci/classify-cancelled-run.sh,
which is what the ratchet asks for ("long steps into .github/scripts/"),
and trim review-narrative prose to the load-bearing why. Behaviour is
unchanged: both call sites invoke the script.
The workflow-shape pin that read the classifier's attempt-jobs fetch out
of the workflow now reads the script.
Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmw1wuw3xco
* fix(ci): do not report a cancelled run that ran no job (#16280 R1-2, R2-1, R2-4)
Critical R1-2: the classifier returned unknown (fail-open report) for a
cancelled run whose attempt has no cancelled job at all — a run displaced while
queued certifies 0==0 and reads no annotation, so it filed a "Main CI failed"
issue for a run that tested nothing. Track whether any cancelled job was seen;
none means supersede (concurrency), while unknown now only covers cancelled jobs
whose annotations are unreadable/unmatched or uncertifiable pages.
R2-1: tests for the certification false branch (partial pages -> unknown) and
the no-jobs case (-> concurrency). R2-4: a negative successor case with a real
id whose cancelled job is a supersede, exercising the == 'timeout' false branch.
* fix(ci): report a cancelled run that also carries a failed job (#16280)
R3-1 (Critical): the classifier read only `cancelled` jobs, so a run whose
job already concluded `failure` before the rest was cancelled classified as a
deliberate supersede and dropped the regression. Read `failure`/`timed_out`
jobs first and fail open on them.
R1-2 (Critical): a cancelled run whose jobs all stopped before their first
step still fell through to `unknown`, which reports. Treat "no cancelled job
ran a step" as nothing-executed, preserving the zero-job behaviour.
R3-2: give that outcome its own token and case arm, so the caller's notice no
longer claims an annotation this path never read.
R3-4: the annotation-error test now uses two cancelled jobs, because with one
the fall-through prints the same `unknown` as the guard and cannot see it
removed.
Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmw2mktggdp
* test(ci): correct successor-gate comment and pin the classifier's full fetch
R3-3: the comment no longer claims the gate admits only failures. R2-4: pin the
classifier fetch including ?per_page=100 and --paginate so flag-stripping reddens.
* fix(ci): keep a failed cancel-classification read from deferring the report
The classifier printed `unknown` both for a run it read and could not match
and for a read that failed. Crediting `unknown` as the successor's own
verdict would then let a transient API failure drop the only report of a
regression, because the successor's gate may have read the same run as
concurrency and skipped. Print `unreadable` for those read failures and
keep the ownership arm on `timeout`/`unknown`.
Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmw4alb71g0
* fix(ci): keep a confirmed timeout over a sibling's failed annotation read (#16280 R5-1)
The per-job annotation read exited unreadable before the verdict block, so
a sibling job's failed read buried timeout evidence another job already
carried — the same hiding this PR removes, one layer over. Remember the
failed read, keep scanning, and fall back to unreadable only when no
timeout was confirmed; a concurrency supersede is likewise not certifiable
over an unread sibling. Adds real-script regression tests (stubbed gh)
that go red on the pre-fix script.
* test(ci): exercise the successor jq projection end to end (#16280 R5-5)
The cancelled-successor deferral binds successor_id/successor_attempt from the
workflows/runs --jq projection, but every stub handed back a pre-formatted
string, so dropping the id field stayed green while production bound the
attempt number as the id. The new case returns raw JSON and applies the
caller's filter with real jq (verified red with the id field dropped), plus a
shape pin on both projected fields.
* docs(ci): align the design docs with the cancelled-run verdicts (#16280 R4-1)
The R4-1 fix widened the successor-ownership rule from `timeout` to
`timeout | unknown`, and the classifier now skips on `nothing-executed` as
well as `concurrency`, but both design docs this PR touches still described
the pre-fix rule: only a timeout-backed cancelled successor owns the result,
and a cancelled run reports only when a job carries a timeout annotation.
Restate the shipped rule in all four files (both languages): the two verdicts
that stay silent, the fail-open `unknown`/`unreadable` reports, a cancelled
run that also carries a `failure`/`timed_out` job, and the `unreadable`
successor that deliberately does not take ownership because a failed read
certifies nothing about what that run concluded.
Docs only — no workflow or script bytes change, so the workflow-size ratchet
is unaffected.
* ci: record main's 501-byte growth of main-ci-failure-issue.yml (#16280)
Merging main into this branch brought in
|
||
|
易良andQwen-Coder
|
65e1f1aeaa |
fix(ci): route external PR CI directly to disposable runners (#16551)
* fix(ci): relay runner settings to fork PR jobs * fix(ci): route external PR jobs directly to disposable runners * fix(ci): give ephemeral runners readable slot names * fix(ci): drop deleted smoke workflow from size baseline The ephemeral-runner-smoke.yml removal left a stale .size-baseline entry that workflow-size.test.js rejects, and ci.yml shrank below the ratchet slack, so record its current size in the same pass. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): propagate disposable-pool routing into comments and docs Address review follow-ups: refresh the two ci.yml comments that still described the removed variable switch, rename the test_build display name off the hardcoded ubuntu-latest, mark the 2026-10-31 routing design as superseded in both languages, and pin the example runner registration to the fork selector in the routing test. Co-authored-by: Qwen-Coder <[email protected]> * style(docs): prettier-format the updated routing tables Co-authored-by: Qwen-Coder <[email protected]> * test(ci): pin the disposable-pool labels to the CI selectors `pr_test_runner`/`lint_runner` publish `["self-hosted", "ecs-qwen-ephemeral"]` and `scripts/ci/ephemeral-runner/config.example.json` registers the same set, but nothing tied the two: a rename on either side leaves every fork PR queued forever with no failed check to point at. Assert the equality, and move the manager fixture off the retired `["self-hosted", "linux", "ARM64", "trial-only"]` set. --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
易良andyiliang114
|
708ba347d3 |
feat(ci): add disposable external PR runners (#16513)
* feat(ci): add disposable external PR runners * test(ci): align platform lanes with disposable runner routing --------- Co-authored-by: yiliang114 <[email protected]> |
||
|
Shaojin WenandQwen-Coder
|
cf29f90062 |
fix(ci): route the review/resolve permission gate to the ECS pool (#16482)
* fix(ci): route the review/resolve permission gate to the ECS pool The authorize job gates every @qwen-code /review and /resolve command, but its ECS routing required github.event.pull_request, which issue_comment and workflow_dispatch payloads do not carry. Every command therefore fell through to ubuntu-latest, and the gate sat in the hosted queue for hours (PR #15286: 3h10m for a 4-second job) while the ECS pool was idle. Route issue_comment and workflow_dispatch to the persistent ECS pool, matching qwen-autofix.yml's route/review-scan pattern for PAT-bearing API-only steps. authorize runs no agent, checks out nothing, and processes no PR content, so this does not widen the fork attack surface. PR-payload events keep the same-repo fork guard; the hosted fallback and ECS kill switch are unchanged. resolve-pr and publish-resolution stay on hosted by design (ephemerality); this removes the gate wait only. * fix(ci): harden the authorize ECS-routing pins and risk docs Address review feedback on the gate routing change: - Pin the checkout-free invariant the ECS routing depends on: a uses: step on authorize would execute contributor-authored files on the persistent pool with CI_BOT_PAT in the step env while the expression pin stayed green (R1-1). Scoped to authorize; mutation-witnessed (adding actions/checkout reds it). - Record the trigger-set widening in the design-doc Risks (both languages): issue_comment makes the gate fireable by any GitHub user on a fork PR, and the should_review authorization fact is now produced on the shared host and consumed downstream without re-deriving (R1-2). - Correct the precedent wording: route loads only the read-only github.token; review-scan is the job that loads CI_DEV_BOT_PAT on the pool. * fix(ci): harden the shared-pool authorize gate against gh config reroute Address the round-2 CHANGES_REQUESTED review (R2-1 Critical, R2-2): R2-1: routing the gate to the persistent ecs-qwen pool moved its CI_BOT_PAT gh api call onto a host with a shared, writable ~/.config/gh. A co-resident survivor could plant http_unix_socket there, receive the PAT, and forge the permission response downstream consumes as should_review. Prepend the same pins as qwen-autofix.yml's route/review-scan — export GH_HOST=github.com plus a private, guarded GH_CONFIG_DIR (a failed mktemp exits 1; the gate fails closed). No `unset GH_TOKEN`: it is this step's only credential. Witnessed with real gh 2.103.0: the as-written step leaks the PAT to a planted socket and approves a stranger; the hardened step denies with the socket untouched. R2-2: extend the checkout-free guard to the run: half — pin the step inventory to the single permission step and forbid `${{` spliced into any run body (event values must arrive via env:). Mutation-witnessed: removing the preamble and adding a run-only second step each red the test. Also correct the design-doc parity claim (both languages): the exposure matches review-scan only WITH these gh pins; document the hardening. * fix(ci): extend the ECS gate routing to triage and harden the gate pins Two batches, verified together before push: 1. Extend the command-gate routing to qwen-triage.yml's authorize (the /triage, /tmux, /verify sibling blind spot): the same issue_comment / workflow_dispatch disjuncts, and — because that job has three steps that spend CI_BOT_PAT on gh — the same self-contained GH_HOST + guarded GH_CONFIG_DIR preamble on each, per the repo's per-step convention. Raise the recorded workflow size baseline to match the added hardening. 2. Address round-3 review (R2-2 fix-induced, R3-1): the gate pins tested substring presence only. Now evaluate the comment-stripped live lines and pin the guarded mktemp form + fail-closed shape, require GH_HOST / GH_CONFIG_DIR exported before the first gh api, and forbid executing repository code via relative paths or interpreters (the pool cwd is the reused $GITHUB_WORKSPACE). Applied to both the review and triage gate tests. Mutation-witnessed: an unguarded one-liner, a relocated preamble, and an appended `node ./…` line each red the tests. Tests: qwen-pr-review-workflow.test.js 271, qwen-triage-workflow.test.mjs 131, ci-runner-routing.test.mjs 103 — all pass; prettier + ESLint clean. * fix(ci): supply RUNNER_TEMP to the triage gate replay tests; harden the gate pins R4-1 (Critical): the new ${RUNNER_TEMP} gh-config preamble is the first RUNNER_TEMP reference in the triage permission step, and three replay harnesses in scripts/tests/qwen-triage-workflow.test.js execute that body verbatim without setting RUNNER_TEMP — off-CI (npm run test:scripts) the fail-closed guard exits 1 before writing should_run. Set RUNNER_TEMP to each harness's existing mkdtempSync dir (the :1773 precedent); the workflow's fail-closed guard is unchanged. Verified: the file is red under `env -u RUNNER_TEMP` before and 203/203 green after; dropping the line reds it again. (The review named two harnesses; running the full file surfaced a third.) Also clear the round-4 deferred notes: - R4-2: the /verify ack comment no longer promises a receipt "even when the sandbox lane is backed up" — the gate now also queues with the pool. - R4-3/R4-4: the triage routing pin is now an exact expression equality, the step inventory is pinned, and PAT-bearing steps are identified by env GH_TOKEN rather than gh subcommand spelling. - R3-1: tighten the no-repository-code deny-list (quote boundaries + interpreter as a word) so `bash -c './x.sh'` no longer slips past; mutation-witnessed in both the review and triage gate tests. Tests: triage vitest 203 (off-CI), triage .mjs 131, review vitest 271, ci-runner-routing 103 — all pass; prettier + ESLint clean; size gate green. * fix(ci): correct the gate-routing narrative; close the round-5 verify gaps R5-1 / F1 (narrative): the motivating incident is a /resolve, whose end-to-end latency this PR does not change — resolve-pr stays on ubuntu-latest, so only the gate wait is removed. State this in both design-doc languages and the pinning test comment; the saving lands on /review, /triage, /tmux, /verify (pool-routed downstream). Correct the "cancelled without ever running" sentence to the run's timeline (resolve-pr started when the gate finished and was cancelled ~29m36s later). S1: the no-repository-code deny-list missed absolute-path interpreters (/usr/bin/node x.js) and versioned Python (python3.11 x.py); widen the regex (optional path prefix + python[0-9.]*) in both gate tests. Mutation-witnessed: both previously-surviving mutants now red, clean tree green. S2: each hardened gh step minted a $RUNNER_TEMP/*gh-config.* dir that nothing removed; add trap 'rm -rf "$GH_CONFIG_DIR"' EXIT to all four steps (RUNNER_TEMP survives between jobs on the pool). S3: record the exact final workflow sizes in .size-baseline. Tests: triage vitest 203 (off-CI, RUNNER_TEMP unset), review vitest 271, triage .mjs 131, ci-runner-routing 103 — all pass; prettier + ESLint clean; size gate green. * fix(ci): pin the no-repository-code invariant on every triage gate step The triage gate meta-test applied the relative-path/interpreter denylist only to PAT-bearing steps, so a repository-code execution line added to the credential-free budget step survived the suite (witnessed by mutation). The no-repository-code invariant is a job-wide safety premise of the ECS routing: even a credential-free step that runs a planted file leaves a survivor on the persistent host. Pin it on every step's comment-stripped live lines, and align both design-doc versions with the widened coverage and the witnessed mutation set. Co-authored-by: Qwen-Coder <[email protected]> * docs(ci): correct the routing-matrix coverage claim for the triage gate The validation plan said the triage gate is pinned by the triage meta-test AND the ci-runner-routing matrix, but the matrix never reads either authorize job (it pins the sibling pool jobs). State the actual coverage in both language versions. Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
|
4d61487419 |
ci(release): only file issues on failure and reconcile stale ones (#15852)
* ci(release): file failure issues only for failures and close stale ones The VSCode companion and Chrome extension notifiers filed an issue when a job was merely cancelled, and nothing closed that issue once a later attempt shipped, leaving a live failure report against a released version. Drop the cancelled arm from both notifiers and add a success-gated reconcile-failure job to each, reusing the shared release step. That step now takes the title prefix and release version so all three release workflows can close the bot-authored issue an earlier attempt filed. Fixes #15840 * test(release): pin the version boundary when closing a stale issue A nightly sibling of the same release sorts first in the gh issue list result, so dropping the " on " suffix from the title guard silently closes another release's issue. The fixture now covers that ordering and the boundary mutant is killed. * test(release): tie each reconcile prefix to the title its notifier files The fan-out suites asserted `FAILURE_TITLE_PREFIX` against a hand-written constant, so renaming the notifier's `--title` left both green while the real reconcile would match no issue and the stale release-failure issue would stay open against a shipped release (QwenLM/qwen-code#15852). Read the prefix out of the notifier's own `--title` argument and assert it starts with the prefix the reconcile job passes, instead of restating the literal a third time. * test(ci): pin the version token and separator in the release title contract reconcile-failure reads a notifier title through startswith(prefix + version + " on "), but the suite only pinned the prefix. A notifier that dropped the version or separated it with something other than " on " closed nothing while both suites stayed green, which is a silent failure of the stale-issue cleanup (#15840). Derive the rest of the title from the notifier's own --title, accept the version variable with or without its :-unknown default, and assert the reconcile job reads the same variable the notifier writes. --------- Co-authored-by: Shaojin Wen <[email protected]> Co-authored-by: 2SinceJuly <[email protected]> |
||
|
易良andQwen-Coder
|
5890329390 |
fix(ci): probe the host egress proxy so mainland release hosts can build the sandbox image (#16281)
The docker-container driver's buildkitd inherits neither the runner's proxy env nor the host network namespace. On ecs-qwen-hz1-17/18 the DNS answer for registry-1.docker.io is poisoned, so the pinned node:22-slim resolution dies with a direct-dial timeout before any application step — five consecutive image-build failures (#16039, #16162) that forced ECS_RELEASE_HOST_RUNNER_ENABLED=false. Probe the fleet-managed proxy marker and inject network=host plus env.*_PROXY driver opts only when the proxy answers a live registry probe. Hosted and HK runners have no marker file, so the probe emits nothing and their builds keep the default direct route. Verified live on hz1: unopted builder reproduces the incident, opted builder resolves the pinned digest for both platforms. Co-authored-by: Qwen-Coder <[email protected]> |
||
|
Shaojin WenandQwen-Coder
|
f387c3a75a |
fix(ci): recreate git metadata before the publish-triage checkout (#16494)
* docs(ci): state the real publish-triage safety argument The publish-triage header comment claimed a 'clean runner', but the job routes to the persistent ECS pool like publish-tmux and runs without a container. The guarantee that actually holds is trusted-code-only: a sparse checkout pinned to github.sha, never agent-provided code, and a publisher that imports only node builtins plus one relative helper, so a reused workspace offers no bare-specifier node_modules to hijack. Reword the comment to say that, per the non-blocking note in the #16468 re-review. Comment-only change; no job, timeout, or behavior delta. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): recreate git metadata before the publish-triage checkout The patrol closeout on the comment rewrite checked the claims against the file and found two gaps. First, publish-triage was the only job reaching actions/checkout on the shared pool without recreating the workspace .git: an earlier agent run owned that workspace and could have planted hooks/config that checkout's git commands would execute in the one job holding the write PAT. Add the same Recreate runner Git metadata step the triage/tmux/verify jobs run before their checkouts, and add publish-triage to the qwen-runner-checkout-workflow matrix that behaviorally executes it (removal witness: deleting the step turns the new entry red). Second, narrow the header: drop the 'it is safe there' certification in favor of naming the controls, and replace 'sole writer' — the triage job's reaction and lifecycle host steps also write with the PAT — with 'the only path from the agent's drafts to GitHub'. Design pair updated in both languages. Co-authored-by: Qwen-Coder <[email protected]> * docs(ci): keep the new step comment off the PAT-holder claim The publish-triage step comment and the design pair said the checkout runs in "the job that holds the write PAT", which is the same overstatement this PR removes: the triage job's reaction, status comment, and finalize step also write with that PAT. Name the publisher as what it is instead, so the header's new parenthetical and the design prose agree. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): pin the publish-triage safety claims and derive the checkout guard Addresses the remaining review findings in this PR. R1-2: publish-tmux's banner still claimed a clean GitHub-hosted runner while its runs-on selects the ECS pool; reword it to the controls that actually hold (checks out nothing, runs no repository code), routing untouched. R1-4: the publisher's bare-specifier-free import closure is load-bearing for the pool argument but nothing pinned it; add a meta-test that scans the closure and matches it against the sparse-checkout list (witness: adding import 'yaml' reddens exactly that test). R2-2: the runner-checkout guard's hand-typed job list is what missed publish-triage; derive the qwen-triage entries from pool-routed jobs that check out, so the next such job reddens with no test edit (witnesses: a new pool checkout job without the reset, and deleting publish-triage's reset, both redden). R2-3: pin publish-triage's ownership-restore fallback chain with assertUnconditional (witness: deleting the sudo fallback reddens). R2-1: the design pair's Files-affected list still said the runner-checkout test needs no change; correct both languages and narrow the pinned-by parenthetical to what each suite actually pins. Co-authored-by: Qwen-Coder <[email protected]> * test(ci): make the new pins layout-independent, floored, and complete Round 3 found three holes in the pins added by the previous commit, all in the guards themselves. R3-1: specifiersOf used a single-line pattern, so Prettier-wrapped imports were invisible in both directions — a wrapped bare specifier kept the closure test green while the pool job would die ERR_MODULE_NOT_FOUND, and a wrapped relative one dropped a needed entry from the expected sparse list. Span newlines, hoist the scanner to module scope, and fixture-pin it (wrapped relative, wrapped bare, and the publisher's own multi-line node:fs import); those three fixtures were verified red against the old pattern and green after, and the sparse comparison stays exact equality. R3-2: the derived job set had no floor, so rewriting one pool job's runs-on into a vars lookup silently dropped its row; add an arrayContaining floor that names the lost job while still admitting new pool jobs with no test edit. R3-3: the ownership test's name promised the full fallback chain but assertUnconditional matches chmod unanchored, so deleting the sudo chmod arm stayed green; pin that arm in this test rather than the shared helper, which has five other call sites. Witnesses: wrapped bare import reddens the closure test, a vars-lookup runs-on reddens the floor naming publish-triage, and deleting the sudo chmod arm reddens the ownership pin — each with byte-level restore. Co-authored-by: Qwen-Coder <[email protected]> * test(ci): prove the sparse checkout by loading it, not by scanning syntax The APPROVED review deferred three findings against the closure guard, all instances of one root cause: the guard modelled JS module syntax with a regex. Proven on the shipped code — a 1164-character wrapped bare import left it green (the 400-char bound), and an out-of-tree helper turned the byte-offset slice into the garbage entry /.github/scripts/elper.mjs. Replace the scanner with the condition the pool actually creates: materialize exactly the sparse-checkout list into a temp tree, assert no node_modules exists above it, and import the publisher there. Sufficiency is the load succeeding; necessity is each .mjs entry omitted making it fail; the owner policy is tied by resolving the publisher's own new URL literal and requiring that exact path in the list, instead of an unanchored substring plus a hard-coded entry. This removes the scanner and its layout fixtures rather than patching them again — wrapped imports of any length, prose that looks like an import, and out-of-tree helpers are all covered by the load itself. Witnesses: the long wrapped bare import that was previously green now fails, as do a short bare import, an out-of-tree helper, a helper dropped from the list, a stale extra entry, and a moved owner-policy path; each with byte-level restore. Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
Shaojin WenandQwen-Coder
|
dd4515ca27 |
feat(ci): isolate triage agent credentials and run all triage on ECS (#16468)
* feat(ci): isolate triage agent credentials and run all triage on ECS The triage agent carried a write PAT, so comment reruns and external fork-PR triage had to stay on ephemeral hosted runners while hosted capacity is tight and the ECS pool has headroom. Wire the token-isolation follow-up the workflow already documented: the agent reads GitHub with the job's read-only GITHUB_TOKEN and expresses every write as a draft; a separate publish job validates and executes drafts with the bot PAT, using one publisher script shared with local /triage runs. Contain the agent job, launch the CLI directly behind the hardened loopback model proxy, and move the deterministic maintainer resolver into the publisher. With no write credential in the agent environment, route every triage trigger to ECS behind the existing kill switch, and retire author_can_write. Markers, bot identity, and the finalize workflow's approve-on-green protocol are unchanged. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): bump triage CI budget and address triage review findings The new publish-triage job crossed the .ci-budget ratchet (7->8 jobs, 280->290 declared timeout-minutes), aborting Lint & Static before any lint or helper test ran. Bump the budget entry, mirror the artifact-name fallback expression on the upload side so an early-death run's drafts stay addressable, and enforce SKILL.md's process-label rule at the publisher chokepoint (welcome-pr, maintainer, help wanted, good first issue) with fail-closed tests, aligning the issue-workflow instruction that previously contradicted the rule. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): harden the triage draft pipeline after an adversarial audit Six findings from a multi-round pre-push audit, each with a mutation witness: kill lingering agent-user processes before the drafts strip/upload so a detached child cannot re-plant symlinks in the race window (upload-artifact follows them and RUNNER_TEMP is host-shared; unkillable survivors drop the draft set deny-by-default); lstat the draft containers themselves in the publisher preflight, since a symlinked bodies/ redirects the realpath containment base; reject cross-kind actions (review/close-pr on issues, close-issue on PRs); install util-linux for runuser as the sibling lanes do; make the mktemp'd proxy-wrapper directory traversable by the agent user so the gh/git wrappers do not silently fall out of its PATH; pin the artifact-name symmetry, the kill-before-strip ordering, and the toolchain install line. Design pair updated in both languages. Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
Shaojin Wen
|
478c2f87ab |
feat(ci): make flaky-patrol rerun and no_action decisions visible (#16459)
* feat(ci): make flaky-patrol rerun and no_action decisions visible The CI Failure Patrol posted only a hidden HTML marker when it re-ran a flaky failure or could not classify one, so maintainers saw a red PR with no explanation and clicked Update Branch — advancing the head, dismissing approvals, and re-running the whole CI for a failure that was already being handled on the same commit. Post a short bilingual notice alongside the marker for both decisions: the rerun notice states the transient cause and that no Update Branch is needed; the no_action notice asks for manual review before rerunning or updating the branch. The deterministic-failure comment is byte-identical, and the post-success reset marker stays hidden. Pin the visible notices, the marker round-trip through dedup and the per-head action count, and reason escaping for all three actions. * fix(ci): strip bidi overrides from flaky-patrol notice reasons An adversarial audit of the visible patrol notices found that bidi control characters (U+200E/U+200F, U+202A-U+202E, U+2066-U+2069) passed through the reason escaping untouched. The reason is classifier output derived from untrusted CI logs, and a right-to-left override can visually rewrite the very sentence maintainers trust when deciding whether to update a branch. Drop those characters in the shared escaping helper so all three visible decisions (and the deflake issue signature) are covered, and pin the stripping for each action. Removing the strip line fails the new test. * fix(ci): cover the Arabic letter mark in the patrol bidi strip An escalated audit round compared the strip class against the full Unicode Bidi_Control set and found U+061C (Arabic letter mark) missing; it can reorder adjacent Arabic-script text in a visible notice. Add it to the class and extend the per-action pin payload, so the class now matches the complete Bidi_Control set. Removing just U+061C fails the pin test. * fix(ci): collapse newlines in patrol reasons and pin the audit gaps The verification round found that a newline plus a line-start fence in a classifier reason opens a block-level code fence in the visible notice: the folded Chinese block and the tracking marker render as visible code. Collapse newlines and their surrounding whitespace to a single space in the shared escaping helper, keeping every reason single-line. Close the coverage gaps the same round measured: feed all twelve Bidi_Control code points individually, pin the English-before-fold and Chinese-inside-fold placement of the deterministic comment, and assert the fold's closing tag in all three notices. Witnesses: removing the collapse, narrowing the bidi class, dropping U+200F alone, swapping EN/ZH, or dropping the closing tag each fails the suite; a marked rendering probe shows the fence swallowing the fold before the fix and a clean render after; benign comment bodies remain byte-identical to base. |
||
|
|
7cb46d9fed |
feat(hooks): add FileChanged notifications for local sessions (#13969)
* feat(hooks): add FileChanged notifications for local sessions * fix(ci): preserve shell gate coverage and trim transcript data * fix(ci): flush debug prompts and isolate git fixtures * fix(hooks): respect ACP watcher ownership and skill registration * test: stabilize hosted recovery and failed spawn fixtures * fix(ci): budget the complete hosted integration fork * fix(runtime-broker): accept MariaDB CSI guard metadata * test: align schema baseline and hook dispatch waits * fix(ci): allow hosted shared fork to finish all classes * fix(hooks): preserve file watching across errors and cwd changes * fix(ci): isolate reference archive extraction per run * test(ci): keep reference archive checks dependency-free * fix(hooks): constrain recursive watches and repair CI * test(web-shell): synchronize verified connection return * fix(ci): correct runtime broker callback indentation * test(web-shell): match the managed workspace registration label Keep the remote workspace registration smoke checks in sync with the service label added by unified workspace navigation. Preserve exact text matching and all registration and replay assertions. * fix(ci): reduce standalone runtime fixture overhead * fix(hooks): ignore implicit atomic-write temporary files * test(managed-agent): recover initial node port conflicts * fix(hooks): observe parents before discovery anchors disappear * test(managed-agent): await stream cleanup before recovery teardown * test(managed-agent): align hosted shell fixture capabilities * test(managed-agent): let OS choose initial harness port * fix(ci): restore runner-local Java cache permissions * test(runtime-broker): use native execution profile paths * test(managed-agent): bind valid workspace upload probes * test(managed-agent): reuse the upload probe digest fixture * test(managed-agent): bind hook fault listeners to available ports * test(managed-agent): align the native profile path fixture * test(managed-agent): Wait for projection refusal retries * ci: Prepare daemon Java cache before Node preflight * test(managed-agent): check channel model calls after delivery * fix(hooks): observe explicitly registered symlink targets * test(managed-agent): separate consumption from relay snapshots * test(managed-agent): target tamper injection by resource key * test(managed-agent): exhaust attestation EOF retry in fault gates * fix(ci): bound RocketMQ release downloads * test(tui): await the restored assistant before capturing history * fix(ci): use a checksum-pinned RocketMQ mirror * fix(ci): provision missing browser test fonts * fix(ci): quote the font probe environment * fix(ci): prepare browser fonts without elevation * test(core): compare complete prune reports without ordering * fix(ci): repair only runner-local Java tool caches * test(managed-agent): qualify fixture JDBC templates * fix: isolate reconnect hooks and cancellation test fixtures Skip chat initialization so MCP reconnect cannot re-enable file hooks, and keep cancellation fixtures isolated from the background worktree scanner. * fix(ci): tolerate a missing Java cache helper on resumed releases Resuming an existing release checks out a tag that may predate the helper, which failed all five Java jobs before setup-java ran. * fix: finish the main merge for harness deadlines and workflow size Use the renamed settle-timeout accessor in the merged hosted-harness tests, and record sdk-java.yml's new size after the resumed-release cache guard. * style(managed-agent): restore import grouping in merged verification tests * test(managed-agent): require consumed child result mirrors * test(web-shell): center repository settings capture --------- Co-authored-by: yiliang114 <[email protected]> Co-authored-by: yungsen.hys <[email protected]> Co-authored-by: Shaojin Wen <[email protected]> |
||
|
chencheng (云谦)
|
1b1f4455d9 |
ci: build the PR tree once and shorten the required path (#16450)
* ci: build the PR tree once and shorten the required path The no-AK integration gate, the four web-shell smoke shards and Lint & Static each rebuilt the tree test_build already uploads, and the build cache rarely helped: most runs logged "29 of 29 workspaces changed" because the arch-keyed cache was split across the mixed x64/ARM64 pool and a prefix restore returned the newest entry instead of the base commit's. - test_build restores the build its base commit saved (ci-build-cache.mjs key --at), drops runner.arch from the key, and runs the serve fast-path closure and core subpath checks against the built tree. - The no-AK integration gate unpacks that tree and runs as three shards; integration_no_ak_result keeps the required check name. - The web-shell smoke unpacks the same tree instead of building; consumers recopy the opentui native assets for their host (host-assets). - The standalone runtime regression moves from unit shard 1 to shard 4. - A java_only profile skips the TypeScript lanes for changes confined to the Java packages; Lint & Static runs the scripts/tests suites that name them. - Drafts skip the integration gate and the browser smoke; ready_for_review starts the full run. * ci: check the settings schema in test_build, which has the core dist Lint & Static only had core's dist because the serve fast-path check built it; with that check moved, Generate settings schema failed to import it. * ci: keep the standalone regression on unit shard 1 Moving it did not shorten shard 1: the regression runs in the side lane, in parallel with the CLI and core suites, and shard 1's time is its CLI slice (576 s against 147 s on shard 4 for the same 181 files). That imbalance is left to duration-based sharding. |
||
|
易良andQwen-Coder
|
1a20792cfa |
ci: route the no-AK integration gate to ECS for every pull request (#16132)
* ci: route the no-AK integration gate to ECS for every pull request
Same trust class as lint_and_static (2026-11-02 extension of the
2026-10-31 decision): the integration step executes PR code, but the
lane carries zero secrets by construction — every model key is emptied
at the step and the job's only token is the read-only checkout grant.
The gate's step-level runner.temp HOME becomes run-unique and is
cleared before use (the R2-1 lesson: RUNNER_TEMP survives across jobs
on this pool). test and integration_cli keep the association-gated
ubuntu_runner. Mutation-verified: runs-on revert, fixed HOME, and a
real key in the PR-code step each red the routing battery.
* fix(ci): isolate no-AK HOME from the first step; value-pinned zero-secrets
Round-2 review findings: (1) Critical — the gate-scoped HOME missed the
lane's actual fork-code start: checkout's local action and the install
(lifecycle scripts plus scripts/prepare.js, un-skipped) ran under the
runner's real shared home with a shared COREPACK_HOME — the exact
cross-job poisoning channel the lint acceptance closed. Mirror lint:
first-step GITHUB_ENV run-unique HOME + COREPACK_HOME inside it +
shared verified XDG cache + always() reclamation. (2) Critical — the
zero-secrets guard was name-based and missed odd-named secrets; enforce
by VALUE (every non-path env key empty, no secrets.* anywhere in the
gate). (3) runs-on literal pin tightened (timeout line also mentions
lint_runner). Docs: EN single-lane phrasing, stale merge-queue claim,
2026-11-02 paragraph rewritten for the first-step shape. Six mutations
red (first-step removal, shared corepack, odd-name secret, no cleanup,
runs-on revert, fixed HOME).
* docs(ci): correct the enforcement-order claim and stale trust-basis note
Round-2 review: the doc claimed the routing battery runs before
dependency install — false (HELPER_TESTS imports yaml, runs after
install, and not at all on github_ci_only PRs' own runs); and the
ci.yml trust-basis comment still described the round-1 gate-scoped
HOME shape. State the actual timing and detection latency, and the
shipped first-step shape.
* fix(ci): scrub ANTHROPIC_AUTH_TOKEN at the no-AK gate; honest cache naming
Round-3 review: the Anthropic SDK's auth resolver back-fills
ANTHROPIC_AUTH_TOKEN from env, so a runner-account residue would survive
into fork PR code on the pool — add it to the gate's scrub list (pinned
by both suites). The 'Configure persistent npm cache' step name promised
persistence the per-run HOME removes; rename it and record the accepted
cold-install cost (shared pnpm store stays a follow-up).
* fix(ci): scrub ANTHROPIC_AUTH_TOKEN at the no-AK gate; honest cache naming
Round-3 review: the Anthropic SDK's auth resolver back-fills
ANTHROPIC_AUTH_TOKEN from env, so a runner-account residue would survive
into fork PR code on the pool — add it to the gate's scrub list. The
value whitelist only iterates present keys, so the deletion mutation
escaped it; pin the key's presence-and-empty in the battery and the
28-key literal list. The 'Configure persistent npm cache' step name
promised persistence the per-run HOME removes; rename and record the
accepted cold-install cost (shared pnpm store stays a follow-up).
* test(ci): pin the no-AK grant, HOME template and reclamation path
Three assertions in the no-AK routing battery proved less than they read:
- the job's `permissions` was never inspected, so the read-only-checkout
grant that justifies routing untrusted fork code onto the pool could be
widened without a red test;
- `create.includes('HOME=')` is satisfied by `COREPACK_HOME=` and
`XDG_CACHE_HOME=`, so deleting the real HOME/USERPROFILE publication kept
the test green;
- the cleanup only had to contain the `qwen-no-ak-home-` prefix, so a path
that dropped `-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}` would still pass
while removing a directory nothing used.
Pin the grant by value, the publication by its printf template, and the
reclamation by the exact run-unique path the create step builds.
Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmw48g5gafx
* test(ci): pin the realized HOME values in the shared ungated lane loop
R2-1: assert the create `home=` line, the gate QWEN_HOME and the cleanup
command by value, and require the exact set of `$GITHUB_ENV` writers. An
`echo` stand-in for the cleanup, a `-v2` create path and an appended second
HOME writer all kept the battery green under containment checks.
R1-2 / R1-1: hoist those pins into the shared `ungated` loop so
`lint_and_static` gets them too. Its create step was pinned by
`includes('HOME=')` (satisfied by `COREPACK_HOME=`) and its cleanup by
`includes('qwen-')`, so a dropped `HOME=%s` or a retargeted `rm -rf` survived.
Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmw4h0sl6g9
* test(ci): red on later republishes of the isolated HOME keys
R3-1: the GITHUB_ENV equality guard read step 0 only, so a later step that
republished HOME reverted the run-unique home by last-write-wins while every
assertion stayed green — the isolation would silently stop applying to the
steps that run PR code. Scan the remaining steps for isolated-key
$GITHUB_ENV writers and for step-level HOME/USERPROFILE/COREPACK_HOME/
XDG_CACHE_HOME overrides, leaving the npm cache step's own key a legitimate
later write.
Mutation-verified on this head: appending printf 'HOME=%s\n' ... to
integration_no_ak's 'Configure per-run npm cache (self-hosted)' step and
adding a step-level `env: HOME` each turn the suite red (103 pass -> 102
pass / 1 fail); reverting restores 103/103.
Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmw4ng9wvgi
---------
Co-authored-by: Qwen-Coder <[email protected]>
|
||
|
|
45f71b3714 |
fix(ci): apply the apt-mirror probe to the release docker lane too (#16278)
* fix(ci): apply the apt-mirror probe to the release docker lane too The release docker integration lane builds the same sandbox image as the E2E leg but never selected an apt mirror, and it prunes the builder cache on every run so its apt layers are always cold. On mainland pool hosts that rebuilt against the throttled deb.debian.org path (11-45 kB/s) — the same defect #16214 fixed for E2E, left unfixed here because it was out of scope (#16271). Extract the probe (apt_mirror_probe_rate / select_apt_mirror) and a new apply_apt_mirror helper into a shared .github/scripts/apt-mirror-probe.sh, sourced by both run-e2e-tests.sh and run-release-docker-integration.sh. Each calls apply_apt_mirror before build:sandbox, so a mainland host builds against the VPC intranet mirror (6-9 MB/s, verified for E2E) while a healthy default is left untouched. The Dockerfile APT_MIRROR ARG already exists from #16218. Contract tests read the shared probe and pin the candidate loop and the source statement (not substrings the comments also satisfy); both callers must apply the probe before the build. Mutation-verified: removing the release call, a candidate, or a source line each reddens the suite. * fix(ci): carry the apt-mirror probe into the shard-retry test rig run-e2e-tests.sh now sources its sibling apt-mirror-probe.sh at startup. The shard-retry meta-test copies only run-e2e-tests.sh into a temp dir, so the source resolved to a missing file and 'set -e' aborted before any shard ran — all eight cases saw zero npm calls. Copy the probe file alongside the script in the rig so the source resolves; sourcing only defines functions (no network at source time) and the sandbox:none path never calls apply_apt_mirror, so the rig stays hermetic. Witness: CI at |
||
|
|
c0782a8cb6 |
test(ci): pin the zero-skip timeout shape in awaitTwoSkipLines (#16384)
Refs #10737 Patrol-Run: qwen-issue-patrol/jmw3fvcf8y3 Co-authored-by: yiliang114 <[email protected]> Co-authored-by: Qwen-Coder <[email protected]> |
||
|
|
a5fec03684 |
fix(ci): fail Java lanes before disk exhaustion (#16068)
* fix(ci): fail Java lanes before disk exhaustion Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): gate the MySQL pool lane instead of a hosted job The disk floor gate landed in hosted-harness-mysql, which runs on ubuntu-latest, so `runner.environment == 'self-hosted'` is never true there and the step was dead code. The job that actually shares the ECS pool disk - mysql-integration, two Maven suites plus a per-leg MySQL container - stayed ungated. Move the step to mysql-integration, immediately after its checkout so the sdk-java-workflow pin still holds. This also clears the three Hosted real-process gates pins, which reject a conditioned step among the setup steps that precede managed_dependencies in hosted-harness-mysql, and reflow the new test to the repo's prettier style. Co-authored-by: Qwen-Coder <[email protected]> Patrol-Run: qwen-pr-closeout/jmvya43o17i * style(ci): drop the blank line left behind in hosted-harness-mysql Moving the disk floor gate out of hosted-harness-mysql left a stray blank line between its checkout and 'Set up Java'. Restoring it makes the whole workflow diff additive on mysql-integration only, so 'hosted-only jobs are unchanged' holds byte for byte and an A/B census of that job reads clean. Co-authored-by: Qwen-Coder <[email protected]> * ci(sdk-java): record the real workflow size after the disk-floor gates The six #16003 disk-floor gate steps grow sdk-java.yml from 154074 to 155650 bytes. main had already drifted 3742 bytes past the recorded 150332 — inside the 4096 allowance, but it left only 354 bytes of ratchet headroom, so this real growth tripped Lint & Static. Record the file's actual size rather than shrinking the gates: each one is a three-line step calling .github/scripts/check-disk-floor.sh, and the workflow test pins all six gated jobs. Co-authored-by: Qwen-Coder <[email protected]> Patrol-Run: qwen-pr-closeout/jmw02ewzoa2 * fix(ci): derive the disk-floor corpus from runs-on and narrow the broker claim Address the two blocking review findings on the Java disk-floor gate: R1-1: the meta-test skipped any job without the gate and compared a closed six-name list, so a newly pool-routed lane that forgot the gate stayed green, and a neutered gate (continue-on-error) kept every assertion true. Derive the corpus from each job's runs-on (same hazard as the setup-java corpus), require gateIndex > -1 per pool-capable lane, and assert the gate carries no continue-on-error. The derived set is the same six jobs today. Verified by mutation: adding continue-on-error to a gate reddens it, and renaming a gate (ungated pool job) reddens it with 'expected -1 to be greater than -1'. R1-2: the step comment claimed RUNNER_TEMP covers the broker store, but the absolute free-KiB floor cannot express the broker's used-ratio refusal (a 952 GiB volume at 94% used still has ample free bytes). Narrow the comment to the gate's real scope (near-full filesystem / inode exhaustion, #10035) and state the broker disk-threshold investigation stays out of scope, matching the PR description. Bump .size-baseline for the comment change (155650 -> 155888). sdk-java workflow meta-test 117 passed / 5 skipped; eslint exit 0, prettier clean; check-workflow-size.sh passes. * fix(ci): calibrate the Java disk floors and fail closed in the meta-test Review round on this PR found the gate could not fire in the two shapes it exists for, and that the meta-test could go green through the same drift it was added to prevent. R1-2: every lane ran the shared 2 GiB / 100k-inode default. The #16003 incident was a used-ratio refusal — the broker reported 0.94 while the 952 GiB volume still had tens of GiB free — so no absolute floor can express that threshold, and 2 GiB is below what these lanes write: a host with 4 GiB free passed the gate and then died mid-suite. Give the five heavy lanes the repo's build-sized 8 GiB floor, leave flyway-migrations and the shared value alone, and add an opt-in DISK_FLOOR_MAX_USED_PCT arm to check-disk-floor.sh for the two broker lanes (90%). R2-1/R2-2: the corpus came from `runs-on` text and skipped jobs outside it, so a lane routed through an expression or a job output shipped ungated while the suite stayed green, and nothing asserted the hosted-only jobs carry no gate. Fail closed: an explicit hosted-only allowlist, a gate required for every other job, a total-count assertion, and a negative assertion on those two jobs. Pin the per-lane floors so dropping one is a test failure. Verified: sdk-java-workflow suite 117 passed, 5 skipped; it goes red when a gate is deleted, when one is added to a hosted job, and when a floor is removed. check-disk-floor suite 12 passed, including four new ratio cases. --------- Co-authored-by: Qwen-Coder <[email protected]> Co-authored-by: yiliang114 <[email protected]> Co-authored-by: qwen-code patrol <[email protected]> |
||
|
|
56fec7f3f5 |
ci: shard the Linux unit tests and web-shell smoke, and reuse cached builds (#16133)
* ci: shard the Linux unit-test lane eight ways behind one build The Linux Test job ran 45-59 minutes: a ten-minute prepare build, then the CLI and core suites back to back. A new test_build job runs the prepare build once and uploads its outputs; eight shards install without the build, unpack them and run --shard=N/8 of every Vitest suite. Test (ubuntu-latest, Node 22.x) becomes an aggregate that passes only when the build and every shard pass. * ci: record ci.yml's growth from the test_build prelude test_build carries the same pre-checkout cleanup, checkout and install prelude as the test shards; ci-platform-lanes.test.js pins the two copies identical. * ci: record the Linux lane's two new jobs in the PR-stage budget test_build and the test_ubuntu aggregate add two jobs; the declared timeouts drop from 305 to 295 minutes. * ci: shard the web-shell E2E smoke four ways The smoke lane ran 44-46 minutes on every recent PR, about 33 of them in the browser tests at two workers, and is the longest lane once the Linux unit tests are sharded. Each shard runs --shard=N/4 of the smoke; the transcript gate runs on the first. web-shell E2E Smoke (ubuntu-latest, Node 22.x) becomes an aggregate over the shards. * ci: bound the unit shards' tree download and retry it once A pool host stalled 17 minutes in download-artifact while the other shards took under a minute. * ci: rebuild only the workspaces a change reaches scripts/ci-build-cache.mjs hashes each workspace's tracked build inputs (tests excluded) with its workspace dependencies' hashes and the shared root inputs. test_build and the web-shell smoke shards restore the newest cached build and rebuild only workspaces whose hash changed. A push to main always builds from scratch and saves, so a pull request starts from a clean build of main. * docs(ci): describe the incremental build cache * ci: keep the build cache path the same on every pool host actions/cache versions an entry by its path; under runner.temp the path differs per host, so a run could never restore another host's save. * ci: save the ARM64 build cache from the smoke lane, and run the aggregates on the shards' runners The smoke shards run on the ARM64 pool, which test_build may not, so the first shard saves the clean build it made when its restore found nothing. The two aggregate checks queued 9 and 13 minutes for a hosted runner after their shards finished; they now take the shards' runners. * ci: bound the sharded test step and skip shards when the build fails Address the first automated review round on the Linux sharding lane: - Gate every build-consuming shard step on `test_build` success. `!cancelled()` stays true when a needed job fails, so a failed build still started all eight shards, each paying a full install before dying at the missing artifact and reporting artifact plumbing as its failing step. - Drop the test step cap to 45 minutes. The job cap is 60, so the old 110 could never fire; a step timeout keeps the diagnostic steps below it alive while a job timeout does not. - Keep the built-tree artifact for a week so a late "re-run failed jobs" still finds it, and correct the trigger comment that still named three post-merge jobs when five now accept push. Co-authored-by: Qwen-Coder <[email protected]> * ci: keep disk telemetry for a failed build and correct the design doc `test_build` installs with the disk sampler and then builds for up to its whole timeout, but the sampler dies with the install step's trap and no collector ran, so an ENOSPC build (or the OOM that took a shard host) kept no evidence. Give it the same failure-gated dump and last-step upload as `lint_and_static`, under its own artifact name. The incremental-lint design still said the test job keeps normal prepare behaviour; this lane now installs with `QWEN_SKIP_PREPARE=1` and unpacks the tree built once by `test_build`. Correct both language versions. Co-authored-by: Qwen-Coder <[email protected]> * ci: share the built-tree definition and the build preflight, harden the guards The archive step re-implemented `builtPaths()` from `scripts/ci-build-cache.mjs`, so the tree the shards unpack had a second definition that could drift from the one `pack` stores. A `paths` mode now prints that list for `tar -T -`, and a new test pins the four artifact-name literals that tie the producer to the eight consumers. The cached branch of `build` also hand-reproduced a subset of `scripts/build.js` and so skipped its ink-patch preflight: the same missing `frame-controller.d.ts` self-healed on a cache miss and surfaced as ~50 tsc errors inside ink's declarations on a hit. The preflight is now a function reachable as `scripts/build.js --preflight` and runs before both branches. The remaining review suggestions were all meta-test gaps: the standalone-runtime expression and the `test_build`/`test` prelude were guarded by nothing that compares them in order, `ci-build-cache.mjs`'s three CI modes were never executed by a test, and the infra-classification filter's step name was not joined to `ci.yml`. Each now has an assertion that goes red when the guard is removed. Co-authored-by: Qwen-Coder <[email protected]> * ci: bound the build steps so the failure collectors can run Round-2 review found four issues in the previous round's fixes and one gap in the new build tests. A job timeout cancels the job and skips the failure-gated collectors, so the `test_build` steps that can block now carry caps below the job's 45 minutes: the install, the build, the pack and the archive. The install caps are identical across `test_build`, `test` and `lint_and_static` so the prelude parity guard still compares the three copies field for field. A test pins every blocking cap under the job ceiling. The `--preflight` early exit was only asserted as a string, so it could be renamed with nothing going red while the cache-hit path silently ran the full build. A spawned case now copies `scripts/build.js` into the fixture and asserts the flag exits without generate, a workspace build or the schema step. The CLI mode tests write extension-less `#!/bin/sh` stubs and join PATH with ':', which cmd.exe cannot resolve; the Windows lane collects that file and downgrades its failure to a warning, so it would have carried a permanently red block. The suite is excluded on win32 alongside the other bash-fixture suites. The `test_build`/`test` intersection grew to 16 with the disk collector, but the floor stayed at 15, leaving exactly one shared step uncompared; the floor now matches the membership. The cache-hit-with-changes branch — the code that decides what is rebuilt and what is deleted — gains a case: a changed workspace is purged and rebuilt by directory, and an untouched workspace keeps what the cache restored. ci.yml's recorded size baseline is updated to the current file so the next PR gets the full growth allowance back. Co-authored-by: Qwen-Coder <[email protected]> * test(ci): keep cache fixture call logs outside the archived tree Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: 易良 <[email protected]> Co-authored-by: Qwen-Coder <[email protected]> |
||
|
易良
|
b5975157d1 |
fix(ci): drop the healthy-default apt gate and prefer the VPC intranet mirror (#16260)
* fix(ci): drop the healthy-default gate so a fast apt burst cannot veto the mirror The merged probe (#16218) kept a third gate: override only when the default measured below 250 kB/s. On the mainland hosts the deb.debian.org path is bimodal — run 41743539611 on hz4-2 probed 372 kB/s (a fast 512 KiB burst), so the gate kept the default, and the sustained 200 MB transfer then throttled to 45 kB/s and hit the 60-minute cap again. Remove that gate. The two mirror-side gates already cover the cases it was meant to protect: the 1 MB/s absolute floor rejects a "slow vs slower" override, and the 4x factor keeps a genuinely fast default (the Hong Kong hosts) on the built-in source. The hz4-2 sample (default 372 kB/s, mirror 1.97 MB/s) now selects the mirror; the Hong Kong and both-slow shapes are unchanged. The contract test pins the floor, the factor, and the absence of any healthy-default gate. * fix(ci): prefer the VPC intranet apt mirror to bypass the capped public egress The public mirrors still traverse the pool's capped 100 Mbps public egress that throttles under concurrent load — the exact problem configure-maven-mirror.sh solves for Central by preferring mirrors.cloud.aliyuncs.com, which answers from inside the VPC and consumes no public bandwidth. Add it as the first apt probe candidate. Off the VPC (hosted runners, Hong Kong hosts) it is unreachable, the probe measures 0, and a reachable public mirror wins — so no host map is needed and the probe self-verifies on each build. |
||
|
易良
|
aaf523cfc4 |
fix(ci): probe the apt mirror so mainland E2E hosts can build the sandbox image (#16218)
* fix(ci): probe the apt mirror so mainland E2E hosts can build the sandbox image The docker E2E leg builds the sandbox image per push under a SHA-scoped tag, so on any host whose image store no longer holds a recent build, the two apt-get layers are downloaded again. deb.debian.org serves the Hong Kong hosts at MB/s and the mainland-China hosts at 11-20 kB/s, which turns that build into 2-3 hours against the job's 60-minute cap. Over the past week 97 of 127 docker legs on the ecs-qwen-hz* hosts were cancelled this way, and the run reports `cancelled` instead of a verdict. Probe rather than pin a mirror or a host-name map: the build measures deb.debian.org and two domestic mirrors, and passes --build-arg APT_MIRROR only when the default is slower than 250 kB/s and a mirror beats it by 4x. The image keeps deb.debian.org as its default source, so release builds, local sandbox builds and the healthy Hong Kong hosts are unchanged, and no fleet inventory needs to be maintained. The probe's measurements are logged, so the next slow build can be diagnosed from its own output. Refs #16214 * fix(ci): require an absolutely fast mirror before overriding the apt source The relative guard alone only asked a mirror to be 4x faster than the default, so on a host where both endpoints were slow the mirror still "won" and moved the host onto a source it never needed. Require 1 MB/s in absolute terms too: below that the build is slow either way, and the extra cache key is not worth it. * test(ci): pin the apt-probe threshold values, not just the gate structure The new contract test asserted the gate variables and comparisons but not their values: mutating slow_default 10x up or fast_mirror 10x down left the suite green while flipping the both-slow host shape onto a still-slow mirror (review finding on #16218). Pin both literals with a word boundary — a plain substring pin would still match the 10x-up mutation — and witness both mutations reddening the suite before restoring the values. * test(ci): require the mirror ARG in both Dockerfile stages A single-match assertion survived deleting the runtime-stage ARG APT_MIRROR (R1-1, mutation-witnessed by review): the builder declaration satisfied the regex while the runtime apt layer silently kept the slow default. Count both declarations instead; deleting either stage's ARG now reddens the suite. |
||
|
易良
|
c2921217ef |
ci: gate on the OS temp directory and shorten /tmp retention to 1d (#16126)
* ci: gate on the OS temp directory and shorten /tmp retention to 1d
The 2026-10-31 hz incident: /tmp tmpfs hit 100% inodes while the
workspace and RUNNER_TEMP sat on a data disk with millions free, so
every disk-floor gate was green and jobs died on mkdtemp ENOSPC
mid-suite. The gate now always checks the OS temp directory
(deduplicated against caller arguments); the tmpfiles policy drops
7d->1d because the measured 120-260k files/day steady state exceeds
the tmpfs inode ceiling at 7d. Mutation-verified: dropping the
os.tmpdir() check reds the suite.
* test(ci): cover the TMPDIR-unset default and pin the 1d retention
Review findings: the new OS-temp test never executed the production
incident path (TMPDIR unset -> /tmp default) — a stripped default
survived the suite green; and the 7d->1d retention change had no test
coverage — a revert shipped the incident again. Add the unset-TMPDIR
sub-run, a set-equality dedup assertion, and a qwen-docker-cleanup
suite pin on the exact tmpfiles line. Mutations verified red.
* test(ci): pin RUNNER_TEMP apart so the /tmp fallback test bites
The first version of the TMPDIR-unset test let the RUNNER_TEMP default
(/tmp) satisfy the assertion, so a stripped ${TMPDIR:-/tmp} default
survived green — the exact regression the test exists to catch. Pin
RUNNER_TEMP to a distinct directory; the /tmp sample can then only come
from the OS-temp append. Mutation now reds.
* test(ci): unset TMPDIR as empty string; destructure was a no-op
The previous fix still failed on clean trees: spreading a destructured
env cannot DELETE a key — process.env.TMPDIR re-entered via runGate's
own spread, so the /tmp fallback never fired and the test failed
against the correct script. An empty-string TMPDIR is bash-unset for
${TMPDIR:-/tmp} (and bare for the regression), which is the exact
distinction under test. Clean 8/8; stripped-default mutation reds.
* ci(ecs-runner): enforce the /tmp retention from the owned daily service
Review finding: the 1d policy's enforcement agent was the distro's
systemd-tmpfiles-clean.timer, which can be masked on hardened images and
reports 'static' (unverifiable via is-enabled) on stock Ubuntu — with
nothing in the qwen stack running systemd-tmpfiles --clean, the measured
120-260k files/day steady state regrows past the tmpfs inode ceiling and
the 2026-10-31 incident class recurs with every test green. Add
ExecStartPost=systemd-tmpfiles --clean --prefix=/tmp to the daily
qwen-docker-cleanup.service, pin it in its test suite, and document the
enforcement owner in the README verification block. Mutation-verified.
|
||
|
易良
|
eaafba5d14 |
ci: route the classify_pr gate to ECS for every pull request (#16019)
The gate is the needs-prerequisite of every Linux lane, but its own runner stayed association-gated: untrusted fork PRs queued the whole chain on the scarce hosted pool (observed 30+ min on PR #15851). classify executes only trusted base-SHA code with a read-only token — a strict subset of the lint exposure accepted on 2026-10-31 — so route it unconditionally to ECS, kill-switch excepted. The ubuntu_runner association gate for the test/integration lanes is unchanged. Routing battery: gate matrix across all associations, structural pin against clause creep; association-clause mutation reds the suite. |
||
|
jinyeandQwen-Coder
|
4cd3372bb6 |
fix(managed-agent): close post-split verification gaps (#15927)
* test(managed-agent): close post-split verification gaps Establish an active response before probing caller cancellation, retain close deadlines, run both independent embedding hosts from the full MySQL lane, and record post-merge acceptance boundaries. Co-authored-by: Qwen-Coder <[email protected]> * ci(managed-agent): keep embedding checks within workflow size gate Move the existing embedding-host verification commands unchanged into a shell script so sdk-java.yml stays within the existing growth allowance. Extend the workflow guards to inspect the script and retain the test and cache-lock requirements. Co-authored-by: Qwen-Coder <[email protected]> * fix(cli): finish hosted settle budget call migration Replace the 25 remaining references to the removed settle timeout constant with the existing per-test budget function. This repairs the TypeScript compilation failure exposed by the main merge while preserving the new shared deadline policy. Co-authored-by: Qwen-Coder <[email protected]> * fix(managed-agent): preserve Hosted fault checks after module split Preserve the typed stream-idle failure and its duration across the SDK boundary. Update fault-injection fixtures for wrapped connectors and the verification module directory, and wait for the documented pending Items projection within the existing bounded poll. Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
|
ca7d487ce9 |
ci: route Lint & Static to ECS for every pull request (#15974)
* ci: route Lint & Static to ECS for every pull request * ci: isolate HOME and complete routing pins for lint_and_static - lint_and_static admits untrusted forks to the shared pool, so pin a job-level HOME/USERPROFILE under runner.temp and create it before the first write (git safe.directory, npm config). - Derive the classifier output from each job's own runs-on in the ceiling helper, so a timeout pointed at the wrong output throws instead of being substituted; parameterise runnerOn the same way. - Pin test's own runs-on to ubuntu_runner and add the HOME-isolation guard for every lane routed past the association gate. - Correct the qwen-autofix premise and the integration_no_ak comment, and record the routing decision in a bilingual design note. Co-authored-by: Qwen-Coder <[email protected]> Patrol-Run: qwen-pr-closeout/jmvvwdojb45 * fix(ci): isolate lint lane HOME and restore routing guard strength Resolve R1-1 (Critical): per-job HOME for lint_and_static so untrusted fork code cannot plant git/npm config a later trusted job inherits; shared warm caches re-pointed via getent + XDG_DATA_HOME/COREPACK_HOME. Resolve R1-2: timeout ceilings derive the output name from each job's own runs-on. Resolve R1-3: explicit ubuntu_runner pin for test. * docs(ci): record lint ECS fork-routing design and correct stale premises Resolve R1-5: bilingual design record (container tradeoff, acceptance, residual risks), qwen-autofix.md section 38 premise, no_ak scoping comment. * fix(ci): publish run-unique isolated HOME from step env, not job env Resolve P1/R1-1: jobs.<job_id>.env cannot read the runner context, so the workflow failed GitHub validation entirely; publish HOME/USERPROFILE via GITHUB_ENV from the first step instead. Resolve R2-1: RUNNER_TEMP survives across jobs on this pool, so the path is run-unique, cleared before create, and reclaimed by an always() step before the failure collector. Resolve R2-3: share content-verified linter/corepack caches via RUNNER_TOOL_CACHE. Resolve R2-5: move the HOME guard into the PR-gated routing battery with a single-lane set pin; resolve wenshao R2-4 with one fleet ladder for both outputs. Witness: actionlint 1.7.12 exits clean on ci.yml; six mutations each red the matching guard. * docs(ci): correct lint routing record per round-2 review Supersede link points at qwen-autofix.md section 38 (R2-4); read-only token and HOME-relative write class stated precisely (R2-2); residual risks are examples not a census, naming web_shell_e2e_smoke and other PAT pool jobs (wenshao R2-3); association gate recorded as throughput policy, not an enforceable boundary (R2-6); dispatch-over-fleet priority and per-lane hosted ceilings (P3); cache cost list and PR-gated test placement updated. * chore(ci): bump ci.yml size ratchet for merged tmpdir routing and HOME isolation Growth is real: #15979's helper-test tmpdir routing arrived via the main merge the freshness gate required, plus this PR's run-unique HOME mechanism. 156377 -> 161281 bytes. * fix(ci): keep COREPACK_HOME per-run; shared cache is a poisoning channel Corepack does not re-verify extracted entries on cache hit (reproduced locally on 0.36.0 with the repo's pnpm pin: tampered entry executes with the recorded hash unchanged, network disabled). Move COREPACK_HOME inside the run-unique HOME; XDG_CACHE_HOME stays shared because lint.js SHA-256-verifies linter archives on every hit. Guard and bilingual record updated; shared-corepack mutation reds the guard. --------- Co-authored-by: Qwen-Coder <[email protected]> Co-authored-by: Shaojin Wen <[email protected]> |
||
|
易良
|
46d825d115 |
Revert "fix(ci): isolate web-shell smoke from shared runner networks (#15886)" (#15961)
This reverts commit
|
||
|
Dragon
|
318babe4d7 | fix(ci): weight PR owner assignment by open issue load (#15194) | ||
|
jinyeandQwen-Coder
|
f3af3dd857 |
refactor(managed-agent): Split server into Maven modules (#15871)
* refactor(managed-agent): split server into Maven modules Separate contracts, orchestration, persistence, adapters, assembly, and verification while retaining the existing service delivery model. Move tests and build consumers together and document local evidence and pending CI acceptance. Co-authored-by: Qwen-Coder <[email protected]> * refactor(managed-agent): reduce configuration and SDK boilerplate Co-authored-by: Qwen-Coder <[email protected]> * refactor(managed-agent): narrow contracts and simplify test scaffolding Co-authored-by: Qwen-Coder <[email protected]> * fix(managed-agent): repair verification after module migration Resolve shared fixtures and worker paths from the Maven module root, update CI packaging assertions, and align existing test fixtures with the extracted contracts. Restore restricted Workspace constructors and remove redundant test visibility changes without altering inherited MySQL coverage. Co-authored-by: Qwen-Coder <[email protected]> * test(managed-agent): align remaining Harness and OSS fixtures Preserve SDK refusal messages and register the managed OSS adapter at fixture injection points. Keep the existing retry and client shutdown assertions intact. Co-authored-by: Qwen-Coder <[email protected]> * style(managed-agent): order Hosted cleanup Harness imports Co-authored-by: Qwen-Coder <[email protected]> * test(managed-agent): fix recovery smoke fixture and import order Supply the required transcript coverage in the recovery browser fixture and wait for history before testing a complete handoff. Preserve its existing safety and submission assertions. Order the cleanup coordinator test imports for Checkstyle. Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
易良andQwen-Coder
|
cc6de38cb5 |
fix(ci): isolate web-shell smoke from shared runner networks (#15886)
* fix(ci): isolate web-shell smoke from shared runner networks Co-authored-by: Qwen-Coder <[email protected]> * docs(ci): clarify hosted smoke font guard behavior Co-authored-by: Qwen-Coder <[email protected]> * docs(ci): explain dormant smoke pool steps Co-authored-by: Qwen-Coder <[email protected]> * docs(ci): stop overstating which dormant smoke guards are pinned The retained comment claimed every self-hosted step's guard is pinned by checkout/preflight tests. Only two of six are: `Recreate runner Git metadata` (scripts/tests/qwen-runner-checkout-workflow.test.js) and `Use pre-installed Node.js (self-hosted)` (scripts/tests/no-ak-integration-ci.test.js). `Check host font baseline (self-hosted)` is named by no test file, and its `runner.environment == 'self-hosted'` guard can never be true under `runs-on: 'ubuntu-latest'` — so it is parked, not pinned. Say which two are pinned and that the font baseline guard is parked, and re-measure the ci.yml size ratchet entry (155625 -> 155708). Co-authored-by: Qwen-Coder <[email protected]> Patrol-Run: qwen-pr-closeout/jmvugxtx22a --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
|
7c9e79e70b |
perf(release): reuse verified native audio prebuilds across releases (#15854)
* perf(release): reuse verified native audio prebuilds across releases Every release rebuilds the five native audio prebuilds even when nothing that affects those binaries changed. The build is short; the wait is not. For v0.26.3 the Linux x64 prebuild queued 74 minutes to execute for 50 seconds, and the collection step queued again after it — and `publish` depends on that collection, so the whole release sits behind it. Fingerprint the audio build inputs and reuse an artifact from an earlier successful official main release when the fingerprint matches. The key covers the audio package, the dependency lock, the package manager pin and the contents of the producing workflow and this helper, so the Node pin and the prebuildify target arguments are part of it. The package version is excluded deliberately: no part of the native build reads it, and including it would make reuse impossible between releases. Reuse means executing bytes an earlier run produced, so provenance is checked rather than assumed: exact fingerprint name, unexpired, same repository for both run and head run, and a completed successful release.yml run on main from workflow_dispatch or schedule. The pinned download-artifact already verifies the artifact digest. Retention is checked differently at selection and consumption. Selection requires 48 hours of headroom so the artifact outlives the release; consumers require only that it has not expired. Re-applying the margin against a later clock would reject an artifact the release had already committed to, after the native matrix was skipped and with no fallback. Lookup, trial download and payload validation are all non-fatal, and the reuse ID is published only once a trial download has validated; an empty ID runs the full matrix, which is today's behaviour. The publisher does not trust that decision: it recomputes the fingerprint from its own checkout, re-verifies the source run, and validates the payload again, fatally — by then the matrix was skipped, so a bad artifact must stop the release. This does not depend on ECS_RELEASE_HOST_RUNNER_ENABLED and takes effect with the switch unset. The collector needed three fixes to be a trustworthy producer. Its `rm -rf -- "$GITHUB_WORKSPACE/packages/audio-capture/prebuilds"` resolved through whatever `packages` was on a persistent host: with a leftover symlink it deleted an external tree and exited 0, and the following download wrote through the same symlink. It now wipes through a shared preflight that canonicalises the workspace and refuses paths resolving outside it. It also ran `node` with no Node contract, inheriting Node from the hosted image by accident. And its release-only label pinned `x64` while the runners holding that label are ARM64, which never matches and queues until timeout. Contract tests pin the wiring, each confirmed to fail against an injected regression. Design doc in both languages. * fix(release): reconcile audio reuse workflow contracts Share the pre-checkout wipe, preserve the workflow size cap, and let fingerprint failures rebuild native audio. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): use block-style release permissions Co-authored-by: Qwen-Coder <[email protected]> * fix(release): drop x64 from publish's release-host expression The review's unresolved item: collect got the ARM64 label fix but publish kept ["self-hosted", "linux", "x64", "ecs-qwen-release-host"], which can never match the ARM64 machines holding that label, so the OIDC-carrying job would queue forever once ECS_RELEASE_HOST_RUNNER_ENABLED is true. Same expression as the collect fix, same one-word drop. * docs(release): clarify audio reuse rollout and validation Co-authored-by: Qwen-Coder <[email protected]> * test(release): cover artifact overwrite and contained symlink guards * test(web-shell): include transcript coverage in recovery fixture Bring the fixture correction from #15883 into this branch to address the observed smoke failure. Co-authored-by: Qwen-Coder <[email protected]> * fix(release): make audio cache seeding best effort Keep the validated same-run artifact usable when retaining a keyed copy fails. Pin the reuse wiring and correct the retention and producer fixtures without changing release trust boundaries. Co-authored-by: Qwen-Coder <[email protected]> * chore(test): apply prettier to the audio prebuild fixtures * docs(release): clarify audio reuse run provenance Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]> Co-authored-by: zzj <[email protected]> |
||
|
易良andQwen-Coder
|
84aee5b9b7 |
test(web-shell): include transcript coverage in recovery fixture (#15883)
* test(web-shell): include transcript coverage in recovery fixture Co-authored-by: Qwen-Coder <[email protected]> * test(web-shell): complete recovery event fixtures Remove the unevaluated transcript type guard and supply required event terminal flags without changing assertions. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): use runner temp for Web Shell browser smoke (#15884) * fix(ci): use runner temp for Web Shell browser smoke Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): resolve runner temp in the browser step context Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): partition the complete macOS test suite into bounded jobs (#15888) * fix(ci): use runner temp for Web Shell browser smoke Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): resolve runner temp in the browser step context Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): partition the complete macOS test suite into bounded jobs Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): account for the macOS aggregate job budget Record the additional five-minute aggregate without changing any test execution timeout. Co-authored-by: Qwen-Coder <[email protected]> * test(install): isolate Windows fixtures from persistent PATH (#15889) Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): write the macOS shard matrix in block style and pin its gates The seven-row matrix used YAML flow mappings, which `.yamllint.yml` forbids (`braces: forbid: 'non-empty'`), so the `Run yamllint` step cannot go green on this branch. Rewrite the rows in the block style every other workflow matrix in this repo already uses. Add the test pins the shard plumbing was missing: - the `cli`/`core` dispatch argv, including the bare `--` that keeps `--shard` forwarded to vitest instead of consumed by npm, and the per-group `NODE_OPTIONS` choice - `id: 'install'` on `test_macos_shards`, which the suite's `steps.install.outcome` gate silently reads as '' if it is renamed - the aggregate `test_macos` gate against the shards' gate, so the preserved check name cannot report `skipped` while the legs fail Co-authored-by: Qwen-Coder <[email protected]> Patrol-Run: qwen-pr-closeout/jmvu6810z1y * fix(ci): preserve shell fixture gates on macOS system Bash Handle empty issue families under nounset without mapfile and explicitly reject either missing OSS credential. Keep failure assertions and include shell diagnostics. Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): keep macOS aggregate failures eligible for infra retry Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
|
70be140e14 |
fix(release): reduce hosted runner queue delays (#15832)
* perf(release): reuse verified native audio prebuilds across releases Every release rebuilds the five native audio prebuilds even when nothing that affects those binaries changed. The build is short; the wait is not. For v0.26.3 the Linux x64 prebuild queued 74 minutes to execute for 50 seconds, and the collection step queued again after it — and `publish` depends on that collection, so the whole release sits behind it. Fingerprint the audio build inputs and reuse an artifact from an earlier successful official main release when the fingerprint matches. The key covers the audio package, the dependency lock, the package manager pin and the contents of the producing workflow and this helper, so the Node pin and the prebuildify target arguments are part of it. The package version is excluded deliberately: no part of the native build reads it, and including it would make reuse impossible between releases. Reuse means executing bytes an earlier run produced, so provenance is checked rather than assumed: exact fingerprint name, unexpired, same repository for both run and head run, and a completed successful release.yml run on main from workflow_dispatch or schedule. The pinned download-artifact already verifies the artifact digest. Retention is checked differently at selection and consumption. Selection requires 48 hours of headroom so the artifact outlives the release; consumers require only that it has not expired. Re-applying the margin against a later clock would reject an artifact the release had already committed to, after the native matrix was skipped and with no fallback. Lookup, trial download and payload validation are all non-fatal, and the reuse ID is published only once a trial download has validated; an empty ID runs the full matrix, which is today's behaviour. The publisher does not trust that decision: it recomputes the fingerprint from its own checkout, re-verifies the source run, and validates the payload again, fatally — by then the matrix was skipped, so a bad artifact must stop the release. This does not depend on ECS_RELEASE_HOST_RUNNER_ENABLED and takes effect with the switch unset. The collector needed three fixes to be a trustworthy producer. Its `rm -rf -- "$GITHUB_WORKSPACE/packages/audio-capture/prebuilds"` resolved through whatever `packages` was on a persistent host: with a leftover symlink it deleted an external tree and exited 0, and the following download wrote through the same symlink. It now wipes through a shared preflight that canonicalises the workspace and refuses paths resolving outside it. It also ran `node` with no Node contract, inheriting Node from the hosted image by accident. And its release-only label pinned `x64` while the runners holding that label are ARM64, which never matches and queues until timeout. Contract tests pin the wiring, each confirmed to fail against an injected regression. Design doc in both languages. * fix(release): move the release build and SDK build off the hosted queue Behind the existing ECS_RELEASE_HOST_RUNNER_ENABLED switch, prepare the version commit and produce the CLI build outputs on an isolated release runner, and split the TypeScript SDK into an isolated producer plus the existing hosted publisher. Eligible Linux downstream jobs join the same pool. With the switch unset every path is exactly today's behaviour. The final npm publication stays on GitHub-hosted runners. npm trusted publishing and provenance require a cloud-hosted runner, so moving the publisher itself would break the release rather than speed it up — only the build work ahead of it can leave the queue. Each migrated job runs one shared preflight before checkout, referenced as a '$/' self-repository action so it resolves at the running commit without a prior checkout. Release-only labels carry no architecture component: the runners holding that label are ARM64, and an "x64" component never matches instead of falling back to hosted. Jobs that genuinely need amd64 — the OSS mirrors and the managed runtime, which execute a hardcoded linux-amd64 ossutil — keep their hosted path, tracked in #15847. The VSCode build routes by matrix.universal so only the dependency-free leg migrates. The SDK publisher consumes the exact tested tarball through a same-run artifact and refuses to overwrite different already-published bytes on retry. Audio prebuild reuse is the base of this branch and is specified separately; the isolated build job is a second consumer of that artifact and is held to the same re-verification contract as the publisher. Rebased onto the audio prebuild reuse branch, so that mechanism and its contract tests are no longer duplicated here. * fix(release): reconcile audio reuse workflow contracts Share the pre-checkout wipe, preserve the workflow size cap, and let fingerprint failures rebuild native audio. Co-authored-by: Qwen-Coder <[email protected]> * fix(ci): use block-style release permissions Co-authored-by: Qwen-Coder <[email protected]> * fix(release): drop x64 from publish's release-host expression The review's unresolved item: collect got the ARM64 label fix but publish kept ["self-hosted", "linux", "x64", "ecs-qwen-release-host"], which can never match the ARM64 machines holding that label, so the OIDC-carrying job would queue forever once ECS_RELEASE_HOST_RUNNER_ENABLED is true. Same expression as the collect fix, same one-word drop. * docs(release): clarify audio reuse rollout and validation Co-authored-by: Qwen-Coder <[email protected]> * docs(release): reserve runner slots on existing ECS hosts * fix(release): quote audio workflow scalar values --------- Co-authored-by: Qwen-Coder <[email protected]> Co-authored-by: zzj <[email protected]> |
||
|
Shaojin Wen
|
35fd1f44ff |
feat(ci): bound autofix takeover with a round budget (#15388)
* feat(ci): bound autofix takeover with a round budget
`@qwen-code /takeover` engages a persistent loop capped at 100 rounds,
so a maintainer who wants one bounded pass has to spend the whole cap
in a single Critical-only round and accept a "reached its round cap"
pause, or watch the run and stop it by hand.
Add `<cmd> for N` (1-99), which engages takeover and records a
window-scoped round budget on the acknowledge comment. The scan lowers
the effective cap to N for that window; once the budget is spent, a
human-authored PR is released — one bilingual note, the label removed,
no needs-human escalation — while a bot-authored PR keeps the paused
path, since releasing it would hand the PR back to standard management.
Failed, timed-out and crashed rounds consume budget, so N means N
attempts.
The workflow sits against its 470 KB start-runs gate, so the duplicated
env commentary moves into the rationale record (af-160), leaving a
one-line summary plus a pointer per constant.
* fix(ci): count attempts and keep breaker stops in the takeover budget
Three review findings on the round budget:
- A no-op round writes its eval marker with the round UNCHANGED, so a
budget riding on ROUND never spent for a PR whose rounds kept finding
nothing — with `for 1` it stayed managed indefinitely. Spend is now
the number of eval markers in the window: one per attempt, no-op
included.
- The same reading separates a breaker stop from real exhaustion: a
breaker stamps its terminal marker at round=MAX_ROUNDS after far
fewer attempts, and that sentinel jump adds no marker, so a
twenty-round window that hit five failures stays paused and escalated
instead of being released with a "nothing is waiting" note.
- A queued conflict round carries the OLD cap, and conflicts were
exempt from the superseded-window check, so after a smaller re-arm it
could run past the new limit. A budgeted live window makes it stale
too; the next scan re-selects the conflict under the new window.
The workflow sits on its 470 KB gate, so the env commentary these reads
pushed past the limit moves into the rationale record the same way as
af-160.
* fix(ci): keep the budget bounded across audit re-arms and queued rounds
Two review findings on the round budget:
- The growth audit appends `<!-- autofix-rearm -->` to open a fresh
window for its own round accounting. A window opened that way carries
no budget marker, so an automatic re-arm turned a `for 2` engagement
back into a 100-round one with no maintainer asking. The report script
now skips that automatic re-arm while the current window is budgeted,
so the audit round stays inside the budget it was given. Both copies
of the helper (script and the failure-path inline one) change together,
which the existing equality test pins.
- A queued round can outlive the sibling that spent the window's last
attempt: a no-op leaves ROUND at 0 and the window key unchanged, so
neither the superseded-window check nor the round cap discarded it and
with fresh feedback it ran one more attempt. `review-address` now
recounts the window's eval markers and discards on a spent budget.
The new tests also caught a real defect in the first draft of the
re-arm guard: a `[[ ... ]] && echo` as the function's last command
returned 1 under `set -e` and would have aborted the report.
* fix(ci): scope the budget to the engagement, not the round window
Third review round. Three correctness findings, all real:
- Suppressing the growth-audit re-arm (the previous round's fix) also
froze the growth baseline: it is the only automatic mover of that key,
so after one over-budget round every later round recomputed growth
against the pinned baseline and went Critical-only. The re-arm is
unconditional again. The budget survives it because budget and spend
are now read per ENGAGEMENT: a maintainer command opens one, while an
automatic re-arm (whose comment also carries that round's eval marker)
does not.
- The attempt census was a raw marker count with no identity, so a
GitHub "Re-run failed jobs" re-posted a round's report and charged
that round twice — releasing the PR early. Attempts are counted as
distinct (ts, round) markers in both the scan and prepare.
- The release fired on a spent budget alone, so a circuit breaker that
tripped exactly on the last budgeted attempt was reported as a
finished engagement with "nothing here is waiting on a human". A
trailing un-evaluated-failure streak of at least TIMEOUT_WINDOW_CAP
now skips the release and lands on the paused path with
autofix/needs-human; a provenance-verified forced dispatch likewise
falls through to its own refusal instead of being preempted.
Design decision 7, its Chinese twin and the operator guide now describe
the carry-forward instead of the suppression.
* fix(ci): end the budget with the engagement that opened it
A spent budget marker outlived the label it belonged to. `stop` posts a
`takeover-ack released` ack, which is not one of the window-key anchors,
so the marker kept reading back: on a bot-authored PR the scan re-entered
the cap branch on every tick and re-applied `autofix/needs-human` at N/N,
and a hand-removed label had the same effect. The budget is now honoured
only while `autofix/takeover` is present, which ends it in both cases.
The regression harnesses now set HAS_TAKEOVER for the managed case.
* fix(ci): repair the two checks the last commit broke
Both failures were introduced by the label-gate commit and are now fixed:
- The pointer suite stopped recognising `autofix.yml`'s env pointers
because they had been shortened to `→ af-NNN`. Restored the full
`→ qwen-autofix.md#af-NNN` form the suite matches, so every section is
pointed at again.
- `HAS_TAKEOVER` was read unguarded while a pre-existing replay harness
runs its shell with `set -u`, which aborted that harness with
"unbound variable". The read is `${HAS_TAKEOVER:-false}` now: callers
that never set it keep the pre-budget behaviour.
Verified locally: 72 focused workflow/size cases pass; the file is
469,982 bytes, under the 470,000 gate.
* fix(ci): attribute budget attempts to the window that dispatched them
The engagement census filtered markers by publication time (`.at >=
floor`), so a round dispatched under the previous engagement and posted
after a fresh `for N` ack was charged to the new allowance: `for 1` could
be spent before its own work ran. Markers carry the window they ran
under, so the census now selects `.win >= floor` (and still skips the
legacy `win=none` markers), which associates an attempt with its
originating engagement while keeping the automatic child windows of that
engagement inside it.
Verified: 73 focused workflow/size cases pass; 469,943 bytes.
* test(ci): pin the budget's lifetime to the takeover label
Audit finding: the label gate that ends a budget when `autofix/takeover`
goes away had no regression. The budget-read harness now takes the label
state as a parameter, and a fixture asserts that the same anchor yields
the takeover cap (100) once the label is absent — the shape that used to
re-park a bot-authored PR at N/N with autofix/needs-human on every scan.
* fix(ci): read the carried budget per engagement in prepare too
`review-address` still resolved `LIVE_ROUND_BUDGET` from the latest growth
window, so after an audit re-arm a queued task read `live_budget=0` and
the attempt cap could not see the engagement's spend. The leg now uses
the scan's engagement scope: a maintainer command opens one, an automatic
re-arm (whose comment also carries an eval marker) does not. That also
makes the superseded-conflict clause see the real budget instead of the
audit window's empty one.
* docs(ci): align the budget record with what the loop now does
Three claims had drifted from the implementation: the effective-cap
bullet said nothing else reads a budget (prepare and the report script
both do), the growth-audit paragraph still described the suppressed
re-arm (it is unconditional; budget and spend are read per engagement),
and the queued-round paragraph said a no-op leaves the round "at 0"
(it leaves it UNCHANGED).
* fix(ci): let the shepherd see the parameterized takeover commands
The auto-release lever builds its resume candidates by exact body match
against the bare takeover and retry commands, so a paused PR re-engaged
with `@qwen-code /takeover for 2` (or `from N`) matched neither and the
release could fire after the pause window despite a fresh engagement.
The predicate now also accepts the two parameterized forms as prefixes of
the configured command, interpolating TAKEOVER_COMMAND rather than adding
a literal.
* docs(ci): name every site that carries budget logic in af-159
The site index listed the three original steps while the prepare leg and the report script's guard read and enforce the same budget.
* docs(ci): name each af-160 entry by the constant it records
The section's own preamble promises that every entry names the constant it
documents; none did, so a maintainer following an env pointer had to read
all eight to find the one they came for. Each header now leads with its
constant. The command entry also drops its "ONE parameterized form" claim,
which this PR's second form falsified, and names both.
* docs(ci): nest the seed sections under their own heading
The guide was retitled for both forms, but the seed's sections stayed at H2, so a document-wide reading of 'does not shorten the round cap' sat directly above the form that does. The seed's sections are now nested and its Picking N no longer collides with the budget one's anchor.
* docs(ci): complete the budget record's inventory and risk list
The files-affected inventory omitted the report script and the prepare
leg's reads, and the risk list only covered the DELETE-failure direction.
Both now name the release's actual shape: the label gate ends the budget,
and the note-then-DELETE order means a failed note still releases.
* docs(ci): bring the Chinese budget record in line with the English one
The previous commit's script aborted after writing only the English file, so the pair disagreed. This lands the same inventory and risk additions in zh-CN.
* fix(ci): keep the budget release note ahead of the label, and read dispatch identity correctly
The release note is the only terminal comment the budget path ever posts;
once the label is gone the PR is in neither selection pool, so a failed post
could never be retried. Skip the DELETE when the note fails, so the next scan
retries both.
The release gate keyed on FORCED_PR being empty, but route publishes pr_number
for every targeted event, not just workflow_dispatch. A pull_request or
pull_request_review run that reached the spend-to-release window therefore fell
through to the round-cap copy and applied autofix/needs-human to a PR that hit
no cap. Test the dispatch identity the way the sibling refusal gate does.
* docs(ci): correct the budget record against the shipped mechanism
The record and the guide disagreed with the code in four ways, all of them in
the direction that misleads whoever debugs it next:
- the budget is read per engagement (the eval-marker exclusion is what tells a
command from the automatic re-arm), not by window-key equality;
- the re-arm is not the only thing that re-anchors the growth baseline — a
base-branch update and the scan's heal ack do too, so the exclusivity claim
is gone while the per-engagement conclusion stays;
- a cancelled run that reaches the finalizer DOES spend an attempt, so the
error direction is an early release, not a late one;
- a crash, timeout or breaker stamps the cap, ending a budgeted window at once
rather than spending the rest of the budget.
Also: the label — not the command — is what engages management (a hand-applied
label cannot be bounded), the 470 KB number is this repo's gate rather than
GitHub's limit, /retry does nothing on a released human PR, and the queued
conflict discard is this change's new clause rather than the window-key check.
af-160 gains the TRUSTED_ASSOC entry the prose move deleted, af-007 names both
parameterized readers, and the guide index advertises both forms.
* docs(ci): label the takeover input by its scope, not by a window
* test(ci): stop calling the seed the one parameterized form
* fix(ci): read prepare's budget only while the takeover label is present
The scan zeroes a spent budget once the label is gone, but prepare still read
it from the comment history alone. A bot-authored PR whose label was removed
after its budget ran out was therefore dispatched by the scan and discarded by
prepare on every tick, with no marker written - a livelock rather than a
release. Gate the read the same way the scan does, on the same live label
re-read prepare already performs.
* fix(ci): count attempts per run, park breaker stops, keep a spent budget
The budget census keyed on `ts|round`, so two executions reporting the same
watermark and round — the conflict no-op, which re-dispatches every tick —
collapsed into one attempt and a for-N engagement could never spend. The eval
marker now carries `run=GITHUB_RUN_ID`, the dedup identity the growth marker
already uses, and both censuses key on it with `ts|round` as the legacy
fallback, so a re-run of one run still charges once.
A breaker stop also spent the budget on its way out. The consecutive-failure
breaker jumps the round to MAX_ROUNDS with a real timestamp, the cumulative
timeout breaker counts timeouts across the window with pushes between them, and
the convergence breaker only pushes — none leaves the trailing sentinel run the
release gate tested. The gate now reads BUDGET_HALTED: a sentinel anywhere in
the engagement, a round past ROUND_START + attempts, or a convergence-break
notice newer than the window key all park the PR instead of releasing it.
Finally, the shepherd's three-day auto-release strips the label from a paused
bot PR. The budget used to die with it, dropping the effective cap back to 10
and re-selecting the PR for rounds N+1..10. A spent budget now survives for a
bot-authored engagement; an unspent one still dies with the label, and a human
release keeps no cap at all.
* test(ci): pin the budget's lower-only clamp and both census filters
The cap clamp's -lt half was exercised by no fixture: the replay hardcoded
EFF_MAX_ROUNDS=100 and every budget is <=99, so deleting the comparison left
the suite green while a stale marker could raise a strict cap. Parameterize the
incoming cap and replay a 20-round marker against a 10-round cap.
The census filters were equally unpinned: BUDGET_FLOOR was frozen at '' and no
fixture carried win=none, so dropping either clause stayed green. jq compares
window keys as strings, so 'none' sorts above every ISO timestamp and only the
win != none clause keeps pre-takeover rounds out. Thread the floor through and
add the two cases plus an in-engagement one.
* chore(ci): re-baseline after merging main
* fix(ci): describe a budget stop as a budget, and record the release only once it happens
Two operator-facing copies lied at the boundary. A bot-authored PR whose
BUDGET_SPENT brought it there fell through to the round-cap body and printed
'hit the round cap (0/3)' - a round count of zero against a cap of three, on a
PR that never reached a cap. It now gets its own line and fleet state naming the
attempts used.
The fleet row claiming a release was written before the block could skip it five
ways (dry run, unreadable labels, changed consent, PAT identity, failed DELETE),
so a dry-run scan reported every budgeted PR as released. The row now follows the
DELETE that actually happened.
Reclaimed the bytes from three pointer comments whose rationale lives in their
sections (af-045, af-118, af-120).
* test(ci): replay the budget boundary end to end, bot path included
The release harness stubbed fleet_row to a no-op and extracted from the inner
if, so neither the fleet row nor the bot-authored half of the boundary was
executed: a dry-run scan could report a release it never performed, and a
restructure that stopped the bot check reaching the branch would release a bot
PR's label with the suite green.
The stub now records rows, every skipped arm asserts no release row, and a
second replay runs the whole outer block with a bot author: no DELETE, and the
notice names the budget instead of a round cap the PR never reached. The outer
entry condition is replayed on its own for the no-op-only window, whose ROUND
never advances past 0 - the only reason a for-2 PR that found nothing twice can
still end.
* fix(ci): state the ack's one exception instead of promising an unconditional release
The budget ack said 'the loop removes the label' for every reader, but a
bot-authored PR keeps the label and pauses for a human - the exact case a
maintainer reading their own ack would mispredict. Both languages now carry the
exception in the same sentence, so the ack is accurate whichever PR it lands on
without a second authorship read at the ack site.
* test(ci): feed the parameterized commands to the shepherd's resume detector
The widened matcher was exercised by nothing: deleting either startswith arm
kept both suites green, and the anti-drift cross-check pins only the two exact
forms. A future grammar change would leave the shepherd blind between a
maintainer's command and the ack landing, auto-releasing a just-re-engaged PR -
the regression the cross-check exists to prevent. Add the two forms plus a
near-miss that must not count.
* fix(ci): match the shepherds resume commands as strictly as the router does
The startswith arms accepted bodies the router rejects - for 0, for 100, for 3
please, from 3 for 2 - so a command that did nothing (no label, no ack, no log
line) vetoed the auto-release for the whole grace window, renewable by any
further malformed variant, and three such bodies from distinct authors in one
window tripped the two-read budget into a permission-fault deferral.
The matcher now strips the prefix and requires a 1-2 digit tail, with the
for-arm also requiring a value >= 1 - the same grammar the parser enforces,
including leading zeros read as decimal.
* docs(ci): correct the prose move's warranty and the inventory's map
af-160 claimed the block was moved verbatim, but the TAKEOVER_COMMAND entry is
the one rewrite - and the rewrite had dropped the base's assurance that the
captured value reaches nothing but an integer comparison. The preamble now names
that exception and the entry restores the clause.
The inventory named the report script, which this change does not touch, and
omitted the fleet shepherd, which it does; the scope boundary repeated the
omission. af-159's site index pointed at the script instead of the shepherd's
resume detector, and still credited the script with carrying the budget.
* test(ci): pin each language of the budget ack separately
Both ack bodies are single printfs with positional arguments, and the only
assertions on the budget copy searched the whole posted body - so the exact
mutation this file documents elsewhere (swapping the EN/ZH arguments, shipping
the Chinese sentence inside the English paragraph) stayed green. Split at the
details summary and assert each language names its own sentence, for both the
engage and the re-arm ack.
* test(ci): replay the prepare budget reads instead of injecting them
Both discard tests handed LIVE_ROUND_BUDGET and LIVE_ATTEMPTS in as literals, so
the two jq programs decision 8 rests on were executed by nothing: a wrong window
key or a dropped marker scan stayed green while a queued round either ran past
the budget or was discarded on an unbudgeted PR.
Replay both reads against a real census: a budgeted ack plus an eval-carrying
re-arm must still yield the budget (the re-arm does not open an engagement), a
human release must zero it, and a spent budget on a bot PR must survive that
removal.
* test(ci): read the report script through the module constant
The re-arm test resolved the script path a second way (resolve/dirname/
fileURLToPath) where ~35 sibling sites use the module-scope read, so renaming or
moving the script meant finding a second spelling whose stale literal would only
surface as a lone ENOENT in this test.
* fix(ci): keep legacy eval markers distinct and migrate the last reader
Two defects in the run= marker format, both from readers that assumed the old
shape:
- The marker builder stored an absent run= as "", and jq's `//` does NOT fall
back for an empty string, so two DIFFERENT pre-change reports collapsed onto
one dedup key: a for-2 window under-counted its attempts and could run one
more round. Both builders now keep a missing run null, so `//` falls back to
the ts|round key as intended, and a replay of two distinct legacy reports
asserts 2 (a re-posted one still asserts 1).
- OVER_BUDGET_AUTHORS still required ` -->` immediately after the optional
win=, so it rejected every new report and left the per-maintainer regular-
feedback batch limit disabled during Critical-only rounds. Its scan now
accepts the optional run= field like the other readers.
* test(ci): pin the eval marker's run field where the assertions read it
Appending `run=${GITHUB_RUN_ID}` to the eval marker moved five expectations out
from under the tests: the report step's terminal-handoff marker and the report
script's two pushed/no-op markers. The assertions still pinned the pre-`run=`
text, so `Test (ubuntu-latest)` failed on `posts a human-handoff marker…` and
`stamps the push marker…` while the writers were correct.
The markers themselves are unchanged — only the expected literals now carry the
field the writers emit.
* fix(ci): let an explicit release close the budget, and keep idle wedges out of the halt
Two defects at the engagement floor, both reported against the shipped code:
- The spent-budget survival clause read label absence plus bot authorship, so it
could not tell the shepherd's silent removal from a maintainer's `stop` or a
hand-removal. A bot PR that spent `for 2` stayed parked at 2/2 after `stop`:
the release leg was skipped by the bot check and the paused leg re-applied
needs-human, contradicting the ack's "standard management continues". The
floor now treats `takeover-ack released` as a boundary, so an explicit release
ends the engagement while a silent removal still keeps a spent budget.
- The halt census counted idle (silent-sandbox) sentinels, which the cumulative
breaker deliberately excludes, so three infra wedges parked a bounded
engagement instead of releasing it. Their runs are mapped from the idle
headlines and subtracted, mirroring the breaker's own `TIMEOUT_N - IDLE_N`.
Reclaimed ~2 KB from three pointer comments (af-155, af-149, af-152) whose
rationale stays in their sections, taking the file from 113 bytes of headroom
to 1,451.
* docs(ci): state the budget's real lifetime, halt thresholds and prose warranty
Four records told the operator the opposite of what shipped:
- the design record (both languages) and af-159 said the shepherd's auto-release
ends the budget and a paused bot PR returns to ten-round management. A SPENT
budget now survives any removal that leaves no `takeover-ack released` marker,
and only an explicit `stop` posts one — so `stop` is the release that ends it.
- af-159 and the guide said any sentinel halts the release, and that the first
crashed or timed-out round parks the PR. The halt needs TIMEOUT_WINDOW_CAP (3)
sentinels (idle ones subtracted), or a round past ROUND_START + attempts, or a
convergence notice; one crash only spends its attempt.
- af-160's warranty promised ONE exception; TRUSTED_ASSOC is a second, now
marked at its entry and counted in the preamble.
- af-157's quoted block had lost its first line, leaving an orphaned fragment
under a heading that still quotes it.
The pointer convention now says a summary line may be rewritten once its block
was compressed (the section is authoritative), and af-159 points at the
growth-audit record for the re-arm invariants it depends on.
* test(ci): round-trip the run field and witness both release suppressors
Two gaps the last round opened:
- The `run=` format was pinned only on the writer side. Both builders now get a
round-trip fixture: a body carrying `run=1111` parses round, window and run
through the real jq, a legacy body still parses round and window with run
absent (null, not ""), and the census keeps two distinct runs apart. Dropping
the optional group from either reader now reds it instead of silently
emptying the census.
- The release gate's two suppressors had no failing witness: `runBoundary`
hardcoded `BUDGET_HALTED: 'false'` and `runRelease` omitted the variable. Both
are parameterised now, with a halted engagement and a real forced dispatch
asserting no label DELETE, a passive event carrying the same pr_number still
releasing, and `runRelease` passing the literal production always has.
The halt census also gains its false side: one sentinel is not a halt (the
threshold is TIMEOUT_WINDOW_CAP), and a convergence notice older than the window
key is not one either.
* fix(ci): retry the handoff post once
The handoff comment is the only carrier for a round that errors, times out or
crashes: its gate excludes fixed/noop rounds, and the always() finalizer only
patches the status comment. It posted single-shot, so one transient 5xx during a
comment-API blip left the round uncounted — the census did not move and the next
scan re-dispatched the same round, a free attempt.
It now retries once, five seconds apart, like the re-arm ack. A sustained
outage still costs the attempt, which the guide already states: a lost marker
delays the release, it never ends the engagement early.
* test(ci): watch the seed beside the budget, and both notice-dedup filters
Three assertions that could not fail before:
- the route replay printed only TAKEOVER_CMD and TAKEOVER_FOR, so "the two
forms never co-emit" was pinned in one direction only. It now prints the seed
too: `for N` must leave the seed empty and `from N` the budget.
- the budget-spent dedup fixture was newer AND bot-authored, so dropping either
filter stayed green. A human-quoted marker and an older bot notice must both
fail to suppress the note now.
- the growth marker's window prefers GROWTH_BASE_WIN over WINDOW; every existing
case passed the same value for both, so swapping them was invisible.
* docs(ci): record what happens when a breaker meets the spending round
The boundary table covered a breaker that stops the loop BEFORE the budget is
spent and said nothing about its condition being met exactly on the attempt that
reaches N. That attempt is already counted, so the PR takes the release path and
no breaker diagnosis is posted: the consecutive-failure gate cannot fire on the
cap round. Stated in both languages rather than left to be inferred from code.
* test(ci): witness a failed label DELETE without aborting the scan
The release harness's gh stub exited 0 on every write, so the DELETE's failure
arm was dead in the suite: unwrapping the 'if !' would have shipped green, and in
production one transient 502 would abort the scan job — costing every other PR in
that tick its round.
The stub can now fail a label write; the case asserts the warning, no fleet row
claiming a release, and that the block still reaches its continue.
* test(ci): inline the round-trip inspection command
A single-use constant left over from iterating on the fixture; the comment above
it already says why the command is what it is.
* docs(ci): finish removing the claim that the budget ends with the label
Two sites still carried the old rule. The guide told the operator that the
shepherd's auto-release returns a paused bot PR to ten-round management, and
af-160's TAKEOVER_MAX_ROUNDS entry promised that removing the label restores the
strict cap — which a still-live for-N budget keeps clamping, exactly in the
spent-bot case af-159 now documents. Both now say what the code does.
Found on an audit pass over this round's diff; no behaviour change.
* docs(ci): list the released ack among the floor's boundaries
Decision 2 enumerated two of the three markers that move the engagement floor;
the explicit-release marker, which is what tells `stop` apart from a silent
removal, was missing in both languages.
* fix(ci): clear the escalation a deferred budget release applied
A forced dispatch that meets a spent budget answers the dispatch on the
paused path, so the release lands one scan later — and left the
needs-human label that pause applied on a PR whose release note says
nothing is waiting on a human. The release now deletes it too, 404
tolerant and only after the takeover DELETE landed. The notices on that
deferred pass quote the attempts the budget counts: a no-op-only
engagement reaches the gate at round 0, and 0/N named a cap the PR never
reached.
* docs(ci): record the forced-dispatch deferral of a budget release
The guide, af-159 and both design languages promised an unconditional
release with no escalation label. A forced dispatch is the one pass that
cannot release, so the deferral, the cleanup it now owes, and the
attempts numerator are written down where a maintainer reads them.
* docs(ci): stop claiming the breaker is the only thing that withholds a release
The guide said the release is withheld ONLY on the three breaker
thresholds, and read any pause as 'a human should look'. A forced
dispatch withholds it too, for one scan, and that pause means the budget
finished. Both design languages lose the same exclusivity claim.
* docs(ci): spell out the label pointer like every other one
The compressed TAKEOVER/SKIP comment kept a bare 'af-160' while the
other 184 pointers in the file name the record they live in.
|
||
|
易良
|
8ef796af37 |
ci(review): keep four runners on review through the daytime (#15769)
The review pool is label-switched: the whole hk1/hk2 fleet carries `ecs-review` overnight and none of it does between 05:00 and 17:00, so a review pushed during the day waits until the evening switch. Today that backlog is 65 queued review runs at 13:45. Give the daytime a small dedicated pool instead of moving the whole fleet: the morning switch now runs a new `day` mode, which keeps a named handful of runners on `ecs-review` and returns everyone else to `ecs-qwen`. Four hk2 runners are the default; a repository variable overrides the list. A name that matches no registered runner fails the tick instead of silently shrinking the pool, and an offline kept runner stays untouched like every other offline runner. The 17:00 switch still moves the whole fleet to review, so the four merge into the full pool at night. |
||
|
易良andQwen-Coder
|
9acb3a5a1d |
fix(ci): pin PR retries to the triggering head SHA (#15734)
* feat(managed-agent): materialize inert external workspaces * feat(managed-agent): enable external files with verified close (#15685) * feat(managed-agent): add the external-workspace provisioning prerequisite (#15512) Step 3a of the external Hosted Workspace plan: a new `external-workspace` Runtime provisioner kind that places a session-isolated worker on a registered EXTERNAL storage tree and mints the boot document the shipped worker already admits (`managed-context/1` plus the existing `storage` member). The storage identity comes only from the durable registration: a read-only primary-key lookup that admits `REGISTERED` rows, bounds the revision to 2^53-1, joins the owner row on its own primary key, and fails closed when a persisted row disagrees with itself. A registration that cannot be read to completion stays retryable (`runtime_external_unavailable`) so the Broker keeps the placement adoptable instead of burying it. `external-workspace` accepts only the closed evidence set {storageRef, mountRoot, continuity}; `mountNonce` from the design is deferred and explicitly refused, because the registration record cannot persist it yet and an unverifiable claim must not be ignored. Hosted tools, EXECUTION admission, publication consumption, `stopDrained` and close gating are not enabled: `supportsDrainedStop()` stays false, every managed Workspace feature remains refused, and preflight now states the prerequisite in an `external-prerequisite` warning. No Session can be placed yet, and every placement fails closed. `external-scratch` behavior and wording, the shipped worker boot/storage/registration.json contract, the schema and the CLI are unchanged. * test(managed-agent): pin external storage refusal in lock canary * fix(managed-agent): require exact external workspace digest * fix(managed-agent): reject storage evidence for non-running workers * feat(managed-agent): add inert exact-worker external drain * feat(managed-agent): fence external workspace ownership and drain * feat(managed-agent): enable external files with verified close * docs: record verified external files and remaining phase two work * fix(web-shell): avoid recovery warning for closed sessions * test(ci): pin HostedExternalPublicFilesIT in the hosted-process class list #15685 adds HostedExternalPublicFilesIT, so hosted-process-ci.test.js's expected class list must carry it; the meta-test pins which IT classes the hosted-process lane runs and had 35 entries against 36 discovered. * fix(managed-agent): restore admission precedence and honour a LOST drain claim The external-files increment resolved the Workspace above the Agent pin, so a request whose stored definition cannot take effect, or whose revision cannot be selected, was refused on Workspace grounds first: it answered 404 workspace_required and a bare IllegalStateException instead of 409 agent_definition_unsupported / 400 unsupported_feature. Resolve the pin first again and keep the external pin rewrite once the Workspace is known. The close-claim validation admitted only DRAINING, but drainClaimedBinding deliberately keeps a LOST binding in LOST until its stop proof commits, so the arm refused its own claim with runtime_close_claim_pending instead of surfacing the stop's runtime_broker_recovery_blocked. BindingRenewal now records that its claim was taken on a LOST binding and admits that state. The settlement probe reads the storage kind to refuse an EXTERNAL tree the bridge holds no mount for, so its fixture needs the managed catalog rather than an empty database. The lock-order canary gains the new ExternalWork.apply/3 ambiguity, whose callee locks the Session row first and never reads ma_turn. * fix(ci): pin PR retries to the triggering head SHA * docs(ci): correct two comments the head-SHA pin made stale Pinning the four pull_request lanes to github.event.pull_request.head.sha left two comments describing the old checkout: the one above test_macos still cross-referenced the Ubuntu gate's superseded wording, and check-lint-gate-freshness.mjs still said the lint lane checks out refs/pull/N/head. Comment-only. ci.yml shrinks by 5 bytes (156090 -> 156085), so neither the .size-baseline ratchet nor the .ci-budget ledger moves. Co-authored-by: Qwen-Coder <[email protected]> Patrol-Run: qwen-pr-closeout/jmvr2rokbyb * test(ci): pin all four PR checkout lanes to their whole ref expression The whole-expression `toBe` pin covered only `test` and `lint_and_static`. `web_shell_e2e_smoke` and `integration_no_ak` were guarded by `toContain(head.sha)` plus `not.toContain('refs/pull/')`, neither of which can see arm order: moving `|| github.ref` ahead of the pull-request arm keeps both green while the lane checks out the asynchronously rebuilt merge ref this PR exists to get away from, and the ancestor guard still passes because a newer merge tree contains the expected head. Pin both lanes against their own full ref literal, so arm order is part of the contract the same way it already is for the push-arm lanes. Verified one mutant at a time on each of the two lines: the new assertion turns red (1 failed | 79 passed) while the two substring assertions stay green, and the tree is byte-identical after reverting. Co-authored-by: Qwen-Coder <[email protected]> Patrol-Run: qwen-pr-closeout/jmvr7204nyh --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
易良
|
b227af2b58 |
fix(release): close the failure issue once a re-run passes (#15711)
notify_failure files an issue whenever the latest attempt of a release run fails, but the job is skipped as soon as a later attempt succeeds, so nothing ever closes the issue it filed. v0.26.2-preview.0 shipped on a successful re-run while #15672 stayed open, was picked up by the scheduled autofix agent, and produced a PR against a release that had already landed. Add a success-gated sibling job that reconciles that issue: it comments the passing attempt, clears the autofix labels, and closes as not planned. Closing requires bot authorship, so a human's same-titled issue is never touched, and the job also stays away when any validation job failed so it cannot race the notifier under force_skip_tests. |
||
|
易良andQwen-Coder
|
74efa46640 |
fix(ci): preserve setup-java JDKs during pool fallback (#15623)
* fix(ci): isolate pool JDK fallback cache publication
* fix(ci): use the aarch64 JDK mirror directory
* fix(ci): normalize setup-java cache versions
* fix(ci): match setup-java's real tool-cache directory name
The pool fallback probed setup-java's cache at
Java_Temurin-Hotspot_jdk/${dir%%-*}/${arch}, stripping the build suffix.
setup-java does not strip it: getToolcacheVersionName() rewrites the first
'+' of Adoptium's semver to '-' BEFORE tc.cacheDir() applies semver.clean(),
and clean() keeps a prerelease -- so the real directory keeps the build
suffix (11.0.32-101, 21.0.13-9.0.LTS). The probe path never existed, the
adoption branch was dead code, and every fallback re-downloaded the pinned
archive even on a host that already held that exact JDK.
Key the probe by the full ${dir}. Our own namespace is unchanged: the script
still publishes only into qwen-pool-jdk/, never into setup-java's.
The test fixture derived the same directory with semver.clean(setupVersion),
encoding the same wrong premise, so both sides agreed by construction and the
suite stayed green over a dead reuse path. Derive it the way setup-java does
instead; reverting only the script line now turns 8 of 14 tests red.
Also assert GITHUB_PATH and toolchains.xml's jdkHome on the reuse path, and
make success() report stdout -- the script writes its ::error:: diagnostics
there, so the old stderr-only message was empty exactly when it was needed.
Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmvp8bpg2w9
* fix(ci): defer private JDK staging until cold install
* fix(ci): reinstall JDKs with orphaned cache markers
* test(ci): pin the cold-install failure path of the pool JDK fallback
Deferring JDK staging into the cold-install branch left `::error::cannot
create` reachable with no witness. Add the cold-path arm: no shared entry,
private namespace shadowed by a regular file, assert the script fails closed
on its own marker and publishes nothing to GITHUB_ENV.
The assertion has to name `::error::cannot create` rather than the bare
phrase: `mkdir` prints "cannot create directory" itself, so `toContain
('cannot create')` still passes when the guard is dropped and the script
falls through to `::error::cannot stage the download`.
Also drop `expect(f.downloads()).toBe(0)` from the sibling-marker case.
Under `MODE=network` the curl shim exits 22 before it appends to the
counter, so that assertion could not fail; `success()` already pins zero
downloads there, because any curl attempt fails the script closed.
Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmvprm5gzwv
---------
Co-authored-by: Qwen-Coder <[email protected]>
|
||
|
Shaojin Wen
|
eea44208b8 |
ci: run every suite but CLI and core in the side test lane (#15570)
* ci: run every suite but CLI and core in the side test lane
The Linux test job's main lane ran the CLI suite and then, one after
another, core, about twenty smaller workspace suites and scripts/tests,
while the side lane (web-shell, then the standalone regression) went idle
long before the CLI suite finished. The main lane now runs only the CLI
and core suites; the side lane runs every other workspace's test:ci, the
standalone regression and scripts/tests, each at half the Vitest worker
cap. Every side-lane suite runs even after another fails.
* ci: print the side lane of a test step killed before its replay
The lane script replays the side lane's log only after the main lane
ends, so a step timeout or cancel lost every suite but the CLI and core
ones. The script now writes that log to SIDE_LANE_LOG when the caller
names one; ci.yml names it under runner.temp, outside the TMPDIR the step
removes, and a failure()/cancelled() step prints whatever is left of it.
The lane test also checks that the main lane's output is present before
it checks its order against the replay, and a comment that still placed
scripts/tests in the main lane now names the side lane.
* ci: keep the lane script's log out of a nested run
scripts/tests runs `npm run test:scripts:unit`, which runs the lane script
again, inside the step that names SIDE_LANE_LOG. The nested run inherited
that name and truncated, replayed and removed the outer side lane's log, so
the outer replay had nothing to print. The script now unsets the variable
once it has read it, and the lane test drives a nested run with a named log
to pin that: the nested run must not see the caller's log, and the outer
side lane's own output must still reach its replay.
The print-step test also pins the `if [ -f "${side_log}" ]` guard around the
cat, which is what keeps that step quiet on an ordinary red test job.
|
||
|
易良
|
2c9fe85a58 |
fix(ci): resolve the pool's Maven dependencies from the intranet mirror (#15641)
The pool's public bandwidth is 100 Mbps for 32 runners, so a burst of dependency fetches saturates it, and the resulting loss turns into the setup-java stalls and 429s seen on the Hangzhou hosts. Alibaba Cloud serves Maven from mirrors.cloud.aliyuncs.com inside the VPC, where the traffic never touches that cap: byte-buddy 1.17.8 arrives as the same 9,018,553 bytes in 0.54 s, faster than the public cap allows. Probe a real artifact (the directory root answers 404) and keep the public mirror for the hosts that cannot reach the intranet one. |
||
|
易良
|
f60b865615 |
fix(ci): bound setup-java on the pool and fall back to the tool cache (#15605)
* fix(ci): bound setup-java on the pool and fall back to the tool cache actions/setup-java resolves the requested version through api.adoptium.net on every run, even when the tool cache already holds that JDK, and the lookup carries no timeout. The Hangzhou hosts cannot always reach that endpoint: one job sat in 'Set up Java' for close to two hours in run 40262799523, and this host currently shows six such processes. Preloading the tool cache therefore never helped — the step never gets as far as consulting it. Bound the attempt at three minutes and, when it does not finish, install the JDK from the host's cache or the Tsinghua mirror with the pinned SHA-256, and write the settings/toolchains files the Maven steps already point at. Applied to the daemon E2E lane first; the other five call sites follow once this one reports green from the pool. * fix(ci): serialise the pool JDK install and bound the fallback Every runner on a host shares RUNNER_TOOL_CACHE, and the stall that sends jobs here is common-mode: one host's egress fails, so concurrent jobs reach the fallback together and the three-minute bound synchronises them further. Take a host lock and only replace a cache entry that carries no .complete marker, so a JDK another job is already building with is never removed. Bound the fallback step itself, and lower the download ceiling to match. |
||
|
易良
|
0975e493fe |
fix(ci): resolve the pool's Maven dependencies through a domestic mirror (#15610)
The pool fetches from one egress with dozens of jobs at a time. Central answers 429 once that egress passes its rate limit, and a job whose host lacks a warm ~/.m2 pulls several hundred megabytes, so the bursts are routine: eleven attempts fifteen seconds apart all answered 429, and a separate job failed on a truncated byte-buddy transfer (4.3 MB of 9 MB). actions/setup-java writes the settings file the Maven steps already pass with --settings, and it holds servers but no mirrors. Rewrite it with the same github server entry plus a central mirror on the pool, where jobs run behind that shared egress; the hosted lanes keep the action's own file. |
||
|
易良andQwen-Coder
|
3c5532278e |
fix(ci): route the tui-parity changes gate to the ECS pool for trusted PRs (#15561)
* fix(ci): route the tui-parity changes gate to the ECS pool for trusted PRs The Check TUI impact job feeds every tui-parity lane, but sat 40+ minutes in the hosted queue while the pool it gates stood idle (#13245 pattern). Route it like the parity/noflicker lanes it guards: trusted PRs, in-repo heads and dispatches reach ecs-qwen, untrusted forks stay hosted, the kill-switch still wins. The job runs no checkout and no PR code, so the pool workspace-hygiene steps do not apply; a new routing meta-test pins both the routing and the checkout-free invariant. * test(ci): pin the tui-parity changes gate to its lanes and to a shell-free step list Two cells the shared #13245 lane harness pins for every other lane were missing from the hand-rolled changes-gate test, and its checkout-only pin under-claimed the pool-hygiene invariant it names: - the MAINTAINER_ECS_RUNNER_DISABLED kill-switch was asserted for pull_request only, so relocating that conjunct inside the event arm left workflow_dispatch runs selecting the pool with the switch flipped on during a pool incident while this suite stayed green; - nothing pinned the gate's runs-on to the parity/noflicker expressions, which is this change's own acceptance criterion (character-identical); - the pin matched only uses: steps prefixed actions/checkout, so a run: step added to jobs.changes passed it while executing and writing inside the pool's reused, uncleaned $GITHUB_WORKSPACE. Assert the kill-switch on every event, assert byte-identity with both lanes, and pin the whole step list rather than just checkouts. The job stays checkout-free: check-disk-floor.sh rides the checkout, so adding one would re-add the pool-hygiene requirement instead of satisfying it. Verified with single mutants, one at a time, each reverted and proven byte-identical: relocating the kill-switch conjunct inside the event arm on changes only reddens the workflow_dispatch iteration (92 tests / 1 fail), and appending "- run: 'true'" to jobs.changes reddens the step-list pin (92 tests / 1 fail). Head is 92/92 green. Co-authored-by: Qwen-Coder <[email protected]> Patrol-Run: qwen-pr-closeout/jmvoc6arqva --------- Co-authored-by: Qwen-Coder <[email protected]> |
||
|
易良
|
9c9088ee50 | fix(ci): run credential-free release fanout on ecs runners (#15517) |