mirror of
https://github.com/codewhale-hq/Codewhale.git
synced 2026-11-11 01:51:34 -10:00
fix(ci): bound telemetry_kill_switch_dispatch so it stops flaking on Ubuntu (#6270)
`codewhale-cli::telemetry_kill_switch_dispatch::missing_preference_defaults_on_without_inventing_acceptance`
has failed twice on Ubuntu CI at `telemetry_kill_switch_dispatch.rs:180`
("default-on writes dry run") on changes that touch neither telemetry nor
the CLI: PR #6104 (`c810bc0458`, 2026-09-12) and PR #6267 (`75e032b61d`,
2026-09-16). Both times macOS and Windows passed.
It is not an env race. CI runs cargo-nextest, which gives every test its own
process, and the test never mutates process-global state — each case spawns
the real binary with `env_clear()` and a fresh `TempDir` HOME/CODEWHALE_HOME
(`telemetry_kill_switch_dispatch.rs:231-276`). The panic is a *missing file*:
`$CODEWHALE_HOME/telemetry/dryrun.jsonl` was never written.
The mechanism is a wall-clock deadline. `features list` resolves to
`Surface::Cli`, whose exit path waits `CLI_PERSIST_TIMEOUT` — 250 ms
(`crates/telemetry/src/lib.rs:75`) — for a detached writer thread that must
re-run `decision::re_decide` against disk and then fsync an append before the
process exits. The code deliberately fails open, so a missed deadline silently
produces no receipt.
All three existing test-group overrides filter `binary(integration)`, and this
is a *different* binary, so none of them ever matched it: the five cases ran at
full parallelism beside 15,774 tests. The CI log shows the runner was saturated
at that moment — neighbouring subprocess-spawning tests took 3.5-3.6 s for work
that normally finishes well under a second, while this one failed in 0.747 s.
This adds the missing override, putting the binary in the existing
`telemetry-contract` group (max-threads = 1) for the same reason that group
exists: these tests spawn the real binary and cannot absorb scheduler latency.
Verified on this machine (macOS aarch64):
cargo nextest show-config test-groups -p codewhale-cli --all-features \
--locked -E 'binary(telemetry_kill_switch_dispatch)'
group: telemetry-contract (max threads = 1)
* override for default profile with filter
'binary(telemetry_kill_switch_dispatch)':
codewhale-cli::telemetry_kill_switch_dispatch: (all 5 tests)
group: spawns-binaries (max threads = 3) (no matches)
sh scripts/with-hermetic-test-home.sh cargo nextest run -p codewhale-cli \
--all-features --locked -E 'binary(telemetry_kill_switch_dispatch)'
Summary [0.621s] 5 tests run: 5 passed, 0 skipped
Not fixed here: the 250 ms budget itself. Bounding the group removes the load
that makes the deadline reachable, but a slow enough host can still miss it.
The product-side options — raise CLI_PERSIST_TIMEOUT, hoist the `re_decide`
disk read out of the deadline window, or have the CLI path join the writer for
the local non-network case — are a behavioural decision, not a test fix, and
are left for the issue.
Signed-off-by: CodeWhale Bot <[email protected]>
Co-authored-by: CodeWhale Bot <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 file changed
+14
@@ -42,6 +42,20 @@ exec-persistent-service = { max-threads = 1 }
|
||||
filter = 'binary(integration) & test(/^telemetry_contract::/)'
|
||||
test-group = 'telemetry-contract'
|
||||
|
||||
# Same hazard, different binary. `telemetry_kill_switch_dispatch` is a
|
||||
# codewhale-cli integration binary, not `integration`, so none of the filters
|
||||
# below ever matched it and it ran at full parallelism beside ~15.7k tests.
|
||||
# Each of its cases spawns the real binary, which gives a detached writer
|
||||
# thread only CLI_PERSIST_TIMEOUT (250 ms, crates/telemetry/src/lib.rs) to
|
||||
# re-read config from disk and fsync its batch before the process exits. Under
|
||||
# Ubuntu CI load that deadline is missed and the dry-run receipt never lands —
|
||||
# observed at telemetry_kill_switch_dispatch.rs:180 on PR #6104 (2026-09-12)
|
||||
# and again on PR #6267 (2026-09-16), both Ubuntu-only, both on changes that
|
||||
# touch neither telemetry nor the CLI.
|
||||
[[profile.default.overrides]]
|
||||
filter = 'binary(telemetry_kill_switch_dispatch)'
|
||||
test-group = 'telemetry-contract'
|
||||
|
||||
[[profile.default.overrides]]
|
||||
filter = 'binary(integration) & test(/^exec_persistent_service::/)'
|
||||
test-group = 'exec-persistent-service'
|
||||
|
||||
Reference in new issue
Block a user