docs(design): shard CI tests by measured run time

docs/design/2026-11-07-ci-duration-sharding.md

@@ -0,0 +1,133 @@
+# Shard CI tests by measured run time
+
+[English](2026-11-07-ci-duration-sharding.md) | [简体中文](2026-11-07-ci-duration-sharding.zh-CN.md)
+
+Status: implemented in this change. Follows
+[Shorten the pull-request CI critical path](2026-11-06-ci-pr-critical-path.md).
+
+## Problem
+
+After #16450, the required-path median for internal pull requests fell from
+28.5 to 21.4 minutes (95 runs, 2026-11-06 to 11-07), short of the 15-minute
+target. The Linux unit-test lane finished last in 66 of those 95 runs, and the
+no-AK integration lane in the rest.
+
+- **Unit shards are uneven.** Vitest's `--shard` gives every shard the same
+  number of files, ordered by a hash of the path, whatever they cost. Shard 1
+  (CLI 589 s) and shard 6 (CLI 743 s) took 12–14 minutes while shard 4
+  finished its CLI files in 146 s.
+- **A single file sets a shard's floor.** A file runs on one worker, so no
+  split can finish sooner than its slowest file. `Session.test.ts` (66k
+  lines, ~556 s), `hosted-harness-session.test.ts` (~292 s) and
+  `hosted-workspace-tool-turn.test.ts` (~206 s in pull-request runs, ~650 s
+  with coverage) set that floor for whichever shard drew them.
+- **Integration shards are uneven too** (median test step 425 s, 163 s,
+  498 s). `mods-tool-middleware.test.ts` alone takes ~430 s, and shard 1 also
+  type-checks the integration sources (~100 s) before its tests.
+- **The build lands on slow hosts half the time.** The x64 pool includes ARM64
+  hosts (hz). `test_build` took 435 s there against 206 s on the x64 (hk)
+  hosts, and every unit and integration shard waits for it.
+
+## Design
+
+### Duration-based sharding
+
+- **The data.** `.github/scripts/ci/test-durations.json` holds, per suite
+  (`packages/cli`, `packages/core`, `integration-tests`), the worker count a
+  shard runs with on ECS, a per-file overhead (import and collect) and the
+  median seconds of every file that takes at least 2 s. Faster files cost the
+  overhead alone.
+- **The plan.** `test-shard-plan.mjs` places the files longest first. A
+  shard's finish is estimated as the larger of its longest file and its total
+  over the workers. Each file goes to the shard that keeps the slowest finish
+  lowest; when several shards keep it equal, the least loaded one wins.
+  Ordering and tie-breaks depend only on paths and durations, so every shard
+  computes the same plan on any host. The worker count is part of the data,
+  not read from the host, for the same reason.
+- **The sequencer.** `vitest-duration-sequencer.mjs` extends Vitest's
+  `BaseSequencer`. With `--shard`, it returns this shard's part of the plan and
+  runs its longest files first. Suites missing from the data keep Vitest's
+  default. The cli, core and integration configs use it; the other workspaces
+  are small and run in the side lane.
+- **The refresh.** `update-test-durations.mjs` reads the per-file times
+  Vitest prints in the logs of the latest successful pull-request runs. It
+  skips push runs, whose coverage slows them, and uses only shards that ran
+  on the ECS pool. Someone runs it by hand and commits the diff. A new or
+  renamed file falls back to the overhead until the next refresh.
+
+### Split the slowest files
+
+The tests move unchanged into sibling files that copy the shared setup, as
+`hosted-workspace-tool-turn.*.test.ts` already did. Every split keeps the
+same set of test names (checked with `vitest list`).
+
+| Original                             | Parts                                                                     |
+| ------------------------------------ | ------------------------------------------------------------------------- |
+| `Session.test.ts`                    | `Session`, `Session.prompt`, `Session.prompt-queue`, `Session.tool-calls` |
+| `hosted-harness-session.test.ts`     | the original, `.no-tool`, `.tools`                                        |
+| `hosted-workspace-tool-turn.test.ts` | the original, `.approvals`, `.hooks`                                      |
+| `mods-tool-middleware.test.ts`       | the original and `.rest`, over a shared `.suite.ts`                       |
+
+`Session.test.ts` also runs faster split: alone it took 294 s locally,
+and its four parts took 128 s in total. Its later tests slow down as state
+builds up across the 66k-line file.
+
+### Integration type-check beside the tests
+
+Shard 1 starts `typecheck:integration` in the background, runs its tests,
+then waits for the type-check and prints its log. The step fails if either
+fails.
+
+### Pin the build to the x64 hosts
+
+`classify_pr` publishes `build_runner`. It equals `ubuntu_runner` unless the
+repository variable `BUILD_RUNNER_LABELS` is set and `ubuntu_runner` chose the
+x64 fleet; then it takes the variable's labels. Only `test_build` reads it.
+The hk hosts need a label of their own, proposed as `ecs-qwen-hk`, because
+`runs-on` matches every listed label and cannot exclude one, and both host
+groups carry the same four. Until operators add the label and set the
+variable, nothing changes; clearing the variable reverts it.
+
+## Decisions
+
+| Decision                                     | Rationale                                                                                                |
+| -------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
+| Durations checked in, not in `actions/cache` | Reviewable and deterministic; works for forks and locally; integration tests do not run on push          |
+| Per-file times from job logs, not JUnit      | Pull-request runs carry no coverage; the integration lane writes no JUnit                                |
+| Worker count fixed in the data               | Hosts differ (`maxWorkers: '25%'`); a host-dependent plan would drop or repeat files across shards       |
+| Split files in the same change               | Without it the slowest file sets the floor: the plan alone saves about 1–1.5 minutes per lane            |
+| `build_runner` gated by a variable           | The hk label does not exist yet; the change ships inert and operators can turn it on or off without a PR |
+
+## Risks
+
+- **Stale durations.** New and renamed files cost the overhead until a
+  refresh. A plan built on stale data is uneven, never wrong: every file
+  still runs in exactly one shard.
+- **The first runs use estimates** for the split files. Refresh the data from
+  this change's own runs.
+- **Pinning the build to hk** queues it when those hosts are busy. Clear the
+  variable to return to the whole pool.
+
+## Validation
+
+- `.github/scripts/ci/test-shard-plan.test.mjs`: every file lands in exactly
+  one shard for 1–60 shards; the plan ignores input order; the slow files
+  spread; the durations file is well formed.
+- Against the real file lists (1463 cli, 1015 core), the eight plans cover
+  every file once; their estimated finishes are 183 s for cli and 106 s for
+  core. A real `vitest run --shard=1/8` in core runs exactly the planned file.
+- `.github/scripts/ci-runner-routing.test.mjs`: `build_runner` follows
+  `ubuntu_runner` while the variable is empty. With the variable set, it
+  narrows only the x64-fleet routing, leaving forks, the kill-switch and the
+  hosted and ARM64 selectors alone. Only `test_build` reads it.
+- `no-ak-integration-ci`: the new integration file is listed, and the gate
+  waits for the type-check and fails with it.
+
+## Acceptance criteria
+
+- Every Linux unit-test file and every no-AK integration file still runs on
+  every pull request, once.
+- The slowest Linux unit shard's test step drops below 8 minutes (median of
+  20 runs).
+- The required-path median drops below 17 minutes over 20 runs, and below 15
+  once `BUILD_RUNNER_LABELS` is set.

docs/design/2026-11-07-ci-duration-sharding.zh-CN.md

@@ -0,0 +1,73 @@
+# CI 测试按实测耗时分片
+
+[English](2026-11-07-ci-duration-sharding.md) | [简体中文](2026-11-07-ci-duration-sharding.zh-CN.md)
+
+状态：已在本次改动中实现。前置文档：[缩短 Pull Request CI 的关键路径](2026-11-06-ci-pr-critical-path.zh-CN.md)。
+
+## 问题
+
+#16450 合入后，内部 PR 的 required 路径中位数从 28.5 分钟降到 21.4 分钟（2026-11-06 至 11-07，95 次 run），离 15 分钟的目标还差一截。这 95 次里，Linux 单测 lane 有 66 次最后完成，其余是 no-AK 集成测试 lane。
+
+- **单测分片不均。** Vitest 的 `--shard` 按路径哈希排序，给每片分相同数量的文件，不看耗时。shard 1（CLI 589 s）和 shard 6（CLI 743 s）跑了 12–14 分钟，shard 4 的 CLI 文件 146 s 就跑完了。
+- **单个文件决定分片下限。** 一个文件只能在一个 worker 上跑，所以怎么切都快不过最慢的那个文件。`Session.test.ts`（6.6 万行，约 556 s）、`hosted-harness-session.test.ts`（约 292 s）和 `hosted-workspace-tool-turn.test.ts`（PR run 里约 206 s，带 coverage 时约 650 s）落在哪一片，哪一片就慢。
+- **集成测试分片也不均**（测试步骤中位数分别是 425 s、163 s、498 s）。`mods-tool-middleware.test.ts` 一个文件就约 430 s；shard 1 还要先做集成测试源码的类型检查（约 100 s）再跑测试。
+- **构建有一半时间落在慢主机上。** x64 池里混着 ARM64 主机（hz）。`test_build` 在 hz 上要 435 s，在 x64 的 hk 主机上 206 s，而所有单测和集成测试分片都要等它。
+
+## 设计
+
+### 按耗时分片
+
+- **数据。** `.github/scripts/ci/test-durations.json` 按 suite（`packages/cli`、`packages/core`、`integration-tests`）记录三项：ECS 上每片用的 worker 数、每个文件的固定开销（import 和 collect），以及耗时 ≥2 s 的文件的中位耗时（秒）。更快的文件只计固定开销。
+- **分配。** `test-shard-plan.mjs` 从最慢的文件开始放。每片的完成时间估为"最大单文件"和"总耗时 ÷ worker 数"中较大的那个。每个文件放进能让最慢一片的完成时间保持最低的那一片；有几片持平时，选当前负载最小的。排序和平局只看路径和耗时，所以每一片在任何主机上算出的分配都一样。worker 数也写在数据里，而不是从主机读取，原因相同。
+- **sequencer。** `vitest-duration-sequencer.mjs` 继承 Vitest 的 `BaseSequencer`。传了 `--shard` 时，它返回本片在分配中的那部分文件，并让最慢的文件先跑。数据里没有的 suite 沿用 Vitest 默认行为。cli、core 和集成测试的配置接入了它；其余 workspace 都很小，在 side lane 里跑。
+- **刷新。** `update-test-durations.mjs` 读取最近几次成功的 PR run 的日志，取 Vitest 打印的每个文件耗时。push run 带 coverage，会更慢，所以不用；也只用跑在 ECS 池上的分片。由人手动运行并提交 diff。新增或改名的文件在下次刷新前只计固定开销。
+
+### 拆分最慢的文件
+
+测试原样搬到同目录的兄弟文件里，各自复制共用的 setup，沿用 `hosted-workspace-tool-turn.*.test.ts` 已有的做法。每组拆分前后的测试名称集合完全一致（用 `vitest list` 核对过）。
+
+| 原文件                               | 拆分后                                                                    |
+| ------------------------------------ | ------------------------------------------------------------------------- |
+| `Session.test.ts`                    | `Session`、`Session.prompt`、`Session.prompt-queue`、`Session.tool-calls` |
+| `hosted-harness-session.test.ts`     | 原文件、`.no-tool`、`.tools`                                              |
+| `hosted-workspace-tool-turn.test.ts` | 原文件、`.approvals`、`.hooks`                                            |
+| `mods-tool-middleware.test.ts`       | 原文件和 `.rest`，共用 `.suite.ts`                                        |
+
+`Session.test.ts` 拆开后本身也更快：本地单独跑要 294 s，拆成 4 个文件后合计 128 s。在这个 6.6 万行的文件里，越靠后的测试越慢，因为前面积累的状态越来越多。
+
+### 集成测试类型检查与测试并行
+
+shard 1 在后台启动 `typecheck:integration`，然后跑测试，跑完再等类型检查结束并打印它的日志。两者任何一个失败，这一步都失败。
+
+### 把构建固定到 x64 主机
+
+`classify_pr` 新增输出 `build_runner`。默认等于 `ubuntu_runner`；只有设置了仓库变量 `BUILD_RUNNER_LABELS`、并且 `ubuntu_runner` 选的是 x64 池时，才改用这个变量里的标签。只有 `test_build` 读取它。hk 主机需要一个自己独有的标签，建议叫 `ecs-qwen-hk`：`runs-on` 要求所列标签全部匹配，没法排除某个标签，而两组主机现在挂的是同样的 4 个标签。在运维加上标签并设置变量之前，什么都不会变；清空变量即可回退。
+
+## 决策
+
+| 决策                                     | 理由                                                                          |
+| ---------------------------------------- | ----------------------------------------------------------------------------- |
+| 耗时数据提交进仓库，不放 `actions/cache` | 可 review、结果确定；fork PR 和本地都能用；集成测试不在 push 上跑             |
+| 从 job 日志取每个文件的耗时，不用 JUnit  | PR run 不带 coverage；集成测试 lane 不写 JUnit                                |
+| worker 数固定写在数据里                  | 主机配置不同（`maxWorkers: '25%'`）；分配如果依赖主机，文件可能被漏跑或重复跑 |
+| 拆文件放在同一个改动里                   | 不拆的话最慢文件就是下限：只改分配，每条 lane 只省约 1–1.5 分钟               |
+| `build_runner` 由变量开关                | hk 标签还不存在；改动合入后先不生效，运维不发 PR 就能开关                     |
+
+## 风险
+
+- **耗时数据过期。** 新增或改名的文件在刷新前只计固定开销。基于过期数据的分配只会不均，不会出错：每个文件仍然只在一片里跑一次。
+- **头几次 run 用的是估算值。** 拆出来的文件目前是估算耗时，用本改动自己的 run 刷新数据即可。
+- **构建固定到 hk 后**，hk 忙时构建会排队。清空变量即可回到整个池。
+
+## 验证
+
+- `.github/scripts/ci/test-shard-plan.test.mjs`：1–60 片时每个文件都恰好落在一片里；输入顺序不影响结果；慢文件会被分散；耗时数据文件格式正确。
+- 用真实文件列表（cli 1463 个、core 1015 个）验证：8 片的分配覆盖每个文件恰好一次；估算的最慢一片 cli 为 183 s，core 为 106 s。在 core 里实际跑 `vitest run --shard=1/8`，跑的正是分配给它的那个文件。
+- `.github/scripts/ci-runner-routing.test.mjs`：变量为空时 `build_runner` 等于 `ubuntu_runner`；设置变量后只收窄 x64 池的路由，fork PR、紧急开关、托管和 ARM64 选项都不受影响；只有 `test_build` 读取它。
+- `no-ak-integration-ci`：新增的集成测试文件已列入；gate 会等类型检查结束，并在它失败时失败。
+
+## 验收标准
+
+- 每个 PR 上，每个 Linux 单测文件和每个 no-AK 集成测试文件都仍然跑且只跑一次。
+- 取 20 次 run，最慢的 Linux 单测分片测试步骤中位数低于 8 分钟。
+- 取 20 次 run，required 路径中位数低于 17 分钟；设置 `BUILD_RUNNER_LABELS` 后低于 15 分钟。