paperclip/.github/ci-duration-pr.md at master

Nicky LeachandClaude Opus 5.5 97953e9b59 fix(ci): repair the weekly test-weight refresh and extend it to e2e and Runner lanes (#19650 )
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors wait for PR CI before merge, so CI wall-clock time sets
the pace of work
> - CI test shards are balanced with per-file durations that a weekly
workflow refreshes
> - The weekly refresh has failed since 2026-11-02, and the e2e weights
have not been measured since 2026-09-15. Today e2e shards 3 and 6 take
7.0–7.7 minutes of tests, and shard 1 takes 4.3 minutes
> - A single file can also grow until it sets the time of its shard, and
nothing tells us about it
> - This pull request repairs the refresh, extends it to the e2e and
Runner package lanes, and warns about test files over 180 seconds
> - The benefit is balanced shards that stay balanced, and early notice
of oversized test files

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The weekly refresh of the test-duration weights that balance CI shards.

**Subsystem affected**

CI: `scripts/ci/refresh-test-durations.mjs`,
`.github/workflows/ci-test-durations.yml`, and the Vitest and Playwright
CI reporters.

**Current behavior**

- The 2026-11-02 refresh (run 41408033686) failed. `gh api …/logs` now
refuses output that contains terminal escape sequences.
- Most successful `cloud-readiness.yml` master runs are admission-only
and have no server jobs. The script took the newest 2 runs, so it would
fail with "No server timing jobs found" on most weeks.
- The e2e weights (`scripts/e2e-shard-durations.json`) have no refresh
and are from 2026-09-15.
- No signal tells us when one test file becomes too slow.

**Proposed behavior**

The refresh runs again, selects master runs that ran the server jobs,
refreshes the e2e and Runner package-lane weights from merged PRs, and
lists oversized files. CI writes a warning for each test file over 180
seconds.

**Reason and benefit**

Unbalanced shards add minutes to every PR. The current e2e weights put
472 seconds on the slowest shard. The new weights put about 300 seconds
on each shard.

**Breaking changes**

None. Weights change only shard placement, never which tests run.

## What Changed

- Add `scripts/ci/gh-api.mjs`. It always passes
`--allow-escape-sequences`, retries without the flag when an older `gh`
does not know it, and strips ANSI codes from logs.
- `scripts/ci/refresh-test-durations.mjs`:
- scans up to 50 master runs and uses the 2 newest that ran the server
jobs;
- adds the `--dry-run`, `--pr-body=<path>` and
`--lanes=server,e2e,runner` options;
- validates every lane before it writes a file. If a merged-PR lane has
fewer than 2 valid runs, it keeps the current weights and writes a
warning.
- Add `scripts/ci/pr-run-durations.mjs`. It uses the final passing
`pr.yml` run of each merged PR, and it validates:
  - the PR is merged into `master`;
  - the merge commit is in the checkout;
  - the run's head SHA is the PR's final head SHA;
  - every shard of the run succeeded.

  It parses Playwright per-spec times and Runner per-file times.
- Add `scripts/ci/oversized-test-files.mjs` (Vitest) and
`scripts/ci/oversized-test-files-playwright.mjs` (Playwright). In GitHub
Actions they write a `::warning` for each test file over
`PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS` (default 180). They never
fail a job. The root and Runner Vitest configs and the e2e Playwright
config load them.
- `scripts/ci/test-timing-reporter.mjs`: move the per-file duration
calculation into an exported `moduleDurationMs()` helper. The behavior
does not change.
- `.github/workflows/ci-test-durations.yml` and
`.github/ci-duration-pr.md`:
  - the bot PR can also update the e2e and Runner manifests;
  - the PR body lists oversized files;
  - the verify step also runs the new tests and the e2e shard tests.
- Refresh `scripts/e2e-shard-durations.json`: 56 of 56 specs measured
(before: 30), from merged PRs #19613, #19191 and #19176. The file now
uses the shared schema with `sources` and `coverage`.

## Verification

- `node --test scripts/ci/test-selection-pr-timings.test.mjs
scripts/ci/test-selection-timings.test.mjs
scripts/__tests__/e2e-shard.test.mjs`: 32/32 pass. These include the e2e
shard balance check with the new weights.
- `node scripts/ci/refresh-test-durations.mjs --dry-run` against real
GitHub data, all lanes:

| Manifest | Measured | Sources |
| --- | --- | --- |
| `general-server-shard-durations.json` | 1488/1534 files | master runs
42296943853, 42294279748 |
| `serialized-shard-durations.json` | 167/167 files | the same two runs
|
| `e2e-shard-durations.json` | 56/56 specs | PR runs 42296374365,
42297700476, 42296994179 |
| Runner `vitest-lane-durations.json` | 310/311 files | the same three
PR runs |

- e2e placement over 8 shards, with the new medians as the real costs:
  - with the old weights, the slowest shard has 472 seconds;
  - with the new weights, every shard has about 300 seconds.
- The dry run lists 12 files over 180 seconds. The largest is
`server/src/services/workspace-recovery.test.ts`, at 636 seconds.
- In this PR, only the e2e weight file changes. The weekly run refreshes
the server weights. The Runner manifest is added by #19646.

## Risks

- The `--allow-escape-sequences` path ran only in unit tests. The local
`gh` did not support the flag, so the real dry run used the fallback.
The next weekly run is the first real test of the flag.
- The changes to the bot-PR shell steps did not run in Actions yet.
- The e2e weights come from 3 PR runs. Real shard times vary more than
the estimate.
- This PR and #19637 both edit the `reporters` lists in
`vitest.config.ts` and `packages/paperclip-runner/vitest.config.ts`. The
second PR to merge needs a small merge resolution.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. A Claude sub-agent wrote the change, and the parent
session reviewed it.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
3.7 KiB Raw Permalink Blame History

Thinking Path

Linked Issues or Issue Description

What Changed

Verification

Risks

Model Used

Checklist

3.7 KiB

Raw Permalink Blame History
