mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
* ci: shard the Linux unit-test lane eight ways behind one build The Linux Test job ran 45-59 minutes: a ten-minute prepare build, then the CLI and core suites back to back. A new test_build job runs the prepare build once and uploads its outputs; eight shards install without the build, unpack them and run --shard=N/8 of every Vitest suite. Test (ubuntu-latest, Node 22.x) becomes an aggregate that passes only when the build and every shard pass. * ci: record ci.yml's growth from the test_build prelude test_build carries the same pre-checkout cleanup, checkout and install prelude as the test shards; ci-platform-lanes.test.js pins the two copies identical. * ci: record the Linux lane's two new jobs in the PR-stage budget test_build and the test_ubuntu aggregate add two jobs; the declared timeouts drop from 305 to 295 minutes. * ci: shard the web-shell E2E smoke four ways The smoke lane ran 44-46 minutes on every recent PR, about 33 of them in the browser tests at two workers, and is the longest lane once the Linux unit tests are sharded. Each shard runs --shard=N/4 of the smoke; the transcript gate runs on the first. web-shell E2E Smoke (ubuntu-latest, Node 22.x) becomes an aggregate over the shards. * ci: bound the unit shards' tree download and retry it once A pool host stalled 17 minutes in download-artifact while the other shards took under a minute. * ci: rebuild only the workspaces a change reaches scripts/ci-build-cache.mjs hashes each workspace's tracked build inputs (tests excluded) with its workspace dependencies' hashes and the shared root inputs. test_build and the web-shell smoke shards restore the newest cached build and rebuild only workspaces whose hash changed. A push to main always builds from scratch and saves, so a pull request starts from a clean build of main. * docs(ci): describe the incremental build cache * ci: keep the build cache path the same on every pool host actions/cache versions an entry by its path; under runner.temp the path differs per host, so a run could never restore another host's save. * ci: save the ARM64 build cache from the smoke lane, and run the aggregates on the shards' runners The smoke shards run on the ARM64 pool, which test_build may not, so the first shard saves the clean build it made when its restore found nothing. The two aggregate checks queued 9 and 13 minutes for a hosted runner after their shards finished; they now take the shards' runners. * ci: bound the sharded test step and skip shards when the build fails Address the first automated review round on the Linux sharding lane: - Gate every build-consuming shard step on `test_build` success. `!cancelled()` stays true when a needed job fails, so a failed build still started all eight shards, each paying a full install before dying at the missing artifact and reporting artifact plumbing as its failing step. - Drop the test step cap to 45 minutes. The job cap is 60, so the old 110 could never fire; a step timeout keeps the diagnostic steps below it alive while a job timeout does not. - Keep the built-tree artifact for a week so a late "re-run failed jobs" still finds it, and correct the trigger comment that still named three post-merge jobs when five now accept push. Co-authored-by: Qwen-Coder <[email protected]> * ci: keep disk telemetry for a failed build and correct the design doc `test_build` installs with the disk sampler and then builds for up to its whole timeout, but the sampler dies with the install step's trap and no collector ran, so an ENOSPC build (or the OOM that took a shard host) kept no evidence. Give it the same failure-gated dump and last-step upload as `lint_and_static`, under its own artifact name. The incremental-lint design still said the test job keeps normal prepare behaviour; this lane now installs with `QWEN_SKIP_PREPARE=1` and unpacks the tree built once by `test_build`. Correct both language versions. Co-authored-by: Qwen-Coder <[email protected]> * ci: share the built-tree definition and the build preflight, harden the guards The archive step re-implemented `builtPaths()` from `scripts/ci-build-cache.mjs`, so the tree the shards unpack had a second definition that could drift from the one `pack` stores. A `paths` mode now prints that list for `tar -T -`, and a new test pins the four artifact-name literals that tie the producer to the eight consumers. The cached branch of `build` also hand-reproduced a subset of `scripts/build.js` and so skipped its ink-patch preflight: the same missing `frame-controller.d.ts` self-healed on a cache miss and surfaced as ~50 tsc errors inside ink's declarations on a hit. The preflight is now a function reachable as `scripts/build.js --preflight` and runs before both branches. The remaining review suggestions were all meta-test gaps: the standalone-runtime expression and the `test_build`/`test` prelude were guarded by nothing that compares them in order, `ci-build-cache.mjs`'s three CI modes were never executed by a test, and the infra-classification filter's step name was not joined to `ci.yml`. Each now has an assertion that goes red when the guard is removed. Co-authored-by: Qwen-Coder <[email protected]> * ci: bound the build steps so the failure collectors can run Round-2 review found four issues in the previous round's fixes and one gap in the new build tests. A job timeout cancels the job and skips the failure-gated collectors, so the `test_build` steps that can block now carry caps below the job's 45 minutes: the install, the build, the pack and the archive. The install caps are identical across `test_build`, `test` and `lint_and_static` so the prelude parity guard still compares the three copies field for field. A test pins every blocking cap under the job ceiling. The `--preflight` early exit was only asserted as a string, so it could be renamed with nothing going red while the cache-hit path silently ran the full build. A spawned case now copies `scripts/build.js` into the fixture and asserts the flag exits without generate, a workspace build or the schema step. The CLI mode tests write extension-less `#!/bin/sh` stubs and join PATH with ':', which cmd.exe cannot resolve; the Windows lane collects that file and downgrades its failure to a warning, so it would have carried a permanently red block. The suite is excluded on win32 alongside the other bash-fixture suites. The `test_build`/`test` intersection grew to 16 with the disk collector, but the floor stayed at 15, leaving exactly one shared step uncompared; the floor now matches the membership. The cache-hit-with-changes branch — the code that decides what is rebuilt and what is deleted — gains a case: a changed workspace is purged and rebuilt by directory, and an untouched workspace keeps what the cache restored. ci.yml's recorded size baseline is updated to the current file so the next PR gets the full growth allowance back. Co-authored-by: Qwen-Coder <[email protected]> * test(ci): keep cache fixture call logs outside the archived tree Co-authored-by: Qwen-Coder <[email protected]> --------- Co-authored-by: 易良 <[email protected]> Co-authored-by: Qwen-Coder <[email protected]>
103 lines
3.4 KiB
JavaScript
103 lines
3.4 KiB
JavaScript
/**
|
|
* @license
|
|
* Copyright 2025 Google LLC
|
|
* SPDX-License-Identifier: Apache-2.0
|
|
*/
|
|
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
import { execSync } from 'node:child_process';
|
|
import { existsSync } from 'node:fs';
|
|
import { dirname, join } from 'node:path';
|
|
import { fileURLToPath } from 'node:url';
|
|
|
|
const __dirname = dirname(fileURLToPath(import.meta.url));
|
|
const root = join(__dirname, '..');
|
|
|
|
// Shared with the incremental CI build (scripts/ci-build-cache.mjs), which
|
|
// rebuilds only changed workspaces and so never reaches the rest of this file.
|
|
const preflight = () => {
|
|
// Install if node_modules was removed (e.g. via npm run clean or scripts/clean.js)
|
|
if (!existsSync(join(root, 'node_modules'))) {
|
|
execSync('corepack pnpm install --frozen-lockfile', {
|
|
stdio: 'inherit',
|
|
cwd: root,
|
|
});
|
|
}
|
|
|
|
// The root postinstall applies patches/ink+7.0.3.patch, and packages/cli
|
|
// compiles against ink types only that patch adds: an install that skipped
|
|
// lifecycle scripts fails the build with ~50 errors that all blame ink. The
|
|
// patch creates frame-controller.d.ts, which pristine [email protected] lacks.
|
|
const inkFrameController = join(
|
|
root,
|
|
'node_modules/ink/build/frame-controller.d.ts',
|
|
);
|
|
if (!existsSync(inkFrameController)) {
|
|
execSync('npm run postinstall', { stdio: 'inherit', cwd: root });
|
|
if (!existsSync(inkFrameController)) {
|
|
console.error(
|
|
'build: patches/ink+7.0.3.patch is not applied to node_modules/ink, so\n' +
|
|
'packages/cli cannot typecheck against it. Run "npm run postinstall"\n' +
|
|
'and resolve what patch-package reports.',
|
|
);
|
|
process.exit(1);
|
|
}
|
|
}
|
|
};
|
|
|
|
preflight();
|
|
if (process.argv.includes('--preflight')) {
|
|
process.exit(0);
|
|
}
|
|
|
|
// build all workspaces/packages in dependency order
|
|
execSync('npm run generate', { stdio: 'inherit', cwd: root });
|
|
|
|
// Select the CLI by path: the repository root has the same package name.
|
|
const cliOnly = process.argv.includes('--cli-only');
|
|
// Mobile MCP has an independent build and was never part of the root build.
|
|
const filter = cliOnly
|
|
? '--filter "{./packages/cli}..." --filter @qwen-code/node-repl-mcp --filter @qwen-code/channel-plugin-example --filter @qwen-code/generated-ui'
|
|
: '--filter "!@qwen-code/mobile-mcp"';
|
|
execSync(`corepack pnpm -r ${filter} run build`, {
|
|
stdio: 'inherit',
|
|
cwd: root,
|
|
});
|
|
|
|
execSync('node --import tsx/esm scripts/generate-settings-schema.ts', {
|
|
stdio: 'inherit',
|
|
cwd: root,
|
|
});
|
|
|
|
// also build container image if sandboxing is enabled
|
|
// skip (-s) npm install + build since we did that above
|
|
try {
|
|
execSync('node scripts/sandbox_command.js -q', {
|
|
stdio: 'inherit',
|
|
cwd: root,
|
|
});
|
|
if (
|
|
process.env.BUILD_SANDBOX === '1' ||
|
|
process.env.BUILD_SANDBOX === 'true'
|
|
) {
|
|
execSync('node scripts/build_sandbox.js -s', {
|
|
stdio: 'inherit',
|
|
cwd: root,
|
|
});
|
|
}
|
|
} catch {
|
|
// ignore
|
|
}
|