Compare commits

...
2 Commits
Author SHA1 Message Date
CodeWhale BotandClaude Sonnet 5.5 aaf478da4c web: regenerate agent docs for the telemetry turns rows
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
2026-11-12 18:18:31 -08:00
CodeWhale BotandClaude Sonnet 5.5 facb35e16d telemetry: count session turns in the engine on every surface
Counter::Turns and the session turn_wall histogram were bumped only by
the terminal UI event loop, so exec and serve sessions always reported
zero turns and an empty histogram. The bump now sits in
record_model_turn_telemetry, before the zero-usage early return, so every
surface reports the turns its engine ran. The terminal UI bump is
removed in the same commit so a TUI turn is counted once.

Child engines do not add session turns: the bump is skipped when the
engine has a child host (sub-agent) or an RLM host, though their model
usage rows are still recorded as before. The wire shape is unchanged.

Terminal-UI counts differ only for turns that never reach the engine's
model-turn settlement: shell-command turns, rejected submissions,
history purges and admitted operations that failed early no longer
count. docs/TELEMETRY.md describes the new anchor and these differences.

record_model_turn_telemetry now takes the counters as an argument so the
new engine unit test uses a local instance: two turns (one with tokens,
one with none) give turns = 2 and two turn_wall observations, and a
child-engine turn adds neither.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
2026-11-12 18:06:29 -08:00
5 changed files with 73 additions and 19 deletions

No files matched your search

+15 -2
View File
@@ -7853,7 +7853,14 @@ impl Engine {
// Update session usage
self.session.total_usage.add(&turn.usage);
record_model_turn_telemetry(self.api_provider, &self.session.model, &turn, status);
record_model_turn_telemetry(
codewhale_telemetry::session_counters(),
self.api_provider,
&self.session.model,
&turn,
status,
self.child_host.is_none() && self.rlm_host.is_none(),
);
if !self.is_acp_turn() {
self.record_goal_usage_for_turn(&turn.usage, turn.started_at);
}
@@ -11783,11 +11790,17 @@ use codewhale_tools::names::{REQUEST_USER_INPUT_NAME, is_tool_search_tool};
pub mod tests;
fn record_model_turn_telemetry(
counters: &codewhale_telemetry::SessionCounters,
provider: ProviderKind,
model: &str,
turn: &TurnContext,
status: TurnOutcomeStatus,
counts_as_session_turn: bool,
) {
if counts_as_session_turn {
counters.bump(codewhale_telemetry::Counter::Turns);
counters.observe_turn_secs(turn.started_at.elapsed().as_secs());
}
let usage = &turn.parent_route_usage;
if usage.input_tokens == 0 && usage.output_tokens == 0 && status == TurnOutcomeStatus::Completed
{
@@ -11798,7 +11811,7 @@ fn record_model_turn_telemetry(
TurnOutcomeStatus::Failed => codewhale_telemetry::TurnOutcome::Failed,
TurnOutcomeStatus::Interrupted => codewhale_telemetry::TurnOutcome::Interrupted,
};
codewhale_telemetry::session_counters().record_model_turn(
counters.record_model_turn(
provider,
model,
outcome,
@@ -1614,3 +1614,57 @@ async fn computer_use_autonomous_postures_block_before_the_plugin() {
.await;
}
}
#[test]
fn session_turns_count_completed_and_zero_usage_turns_but_not_child_engine_turns() {
let counters = codewhale_telemetry::SessionCounters::default();
let mut used = TurnContext::new(4);
used.parent_route_usage = Usage {
input_tokens: 120,
output_tokens: 30,
..Usage::default()
};
record_model_turn_telemetry(
&counters,
ProviderKind::Deepseek,
"deepseek-v4-flash",
&used,
TurnOutcomeStatus::Completed,
true,
);
let idle = TurnContext::new(4);
record_model_turn_telemetry(
&counters,
ProviderKind::Deepseek,
"deepseek-v4-flash",
&idle,
TurnOutcomeStatus::Completed,
true,
);
assert_eq!(counters.counters().turns, 2);
let wall = counters.turn_wall();
assert_eq!(
wall.lt_5s + wall.five_to_thirty + wall.thirty_to_onetwenty + wall.gte_120s,
2
);
let usage_rows = counters.usage_events();
assert_eq!(usage_rows.len(), 1, "only the turn that used tokens has a row");
record_model_turn_telemetry(
&counters,
ProviderKind::Deepseek,
"deepseek-v4-flash",
&used,
TurnOutcomeStatus::Completed,
false,
);
assert_eq!(counters.counters().turns, 2, "a child engine turn adds no session turn");
let wall = counters.turn_wall();
assert_eq!(
wall.lt_5s + wall.five_to_thirty + wall.thirty_to_onetwenty + wall.gte_120s,
2
);
}
-13
View File
@@ -3967,19 +3967,6 @@ pub(crate) async fn run_event_loop(
}
}
// Counted here, at the caller, never inside
// `execute_turn_end_observer_hook`: that function's
// first statement returns early for anyone with no
// TurnEnd hooks, and the natural future optimization
// hoists that check up to this call site — which would
// silently zero the counter for every user who does
// not use hooks.
{
let telemetry = codewhale_telemetry::session_counters();
telemetry.bump(codewhale_telemetry::Counter::Turns);
telemetry.observe_turn_secs(turn_elapsed.as_secs());
}
if let Err(error) = execute_turn_end_observer_hook(
app,
completed_turn.as_ref(),
+2 -2
View File
@@ -327,7 +327,7 @@ The workhorse. Everything a session accumulated ships here, once.
| field | source anchor |
|---|---|
| `turns` | `run_event_loop` in `crates/tui/src/tui/ui/event_loop.rs`, immediately before it calls `execute_turn_end_observer_hook`. Never inside that hook: its first statement is `if !app.hooks.has_hooks_for_event(HookEvent::TurnEnd) { return Ok(()); }` (`crates/tui/src/tui/ui/observer_hooks.rs`), and the natural future optimization hoists that check to the call site, silently zeroing the counter for every user without hooks. |
| `turns` | `record_model_turn_telemetry` in `crates/engine/src/core/engine.rs`, called once per turn the engine runs, before the zero-usage early return, so a turn that reported no tokens still counts. The engine counts on every surface (terminal UI, `exec`, `serve`, desktop), so an `exec` or `serve` session reports the turns it ran. It counts turns of the session's own engine only: a sub-agent's or an RLM call's child engine adds no session turn, though its model usage rows still count. The terminal UI no longer counts a turn itself. Autonomous continuation turns and review turns are turns the engine runs and count. Compared with the earlier terminal-UI count, shell-command turns (`!` commands), rejected submissions, history purges and turns that failed before the model was called no longer count. The call site never lives inside a conditionally entered handler such as the turn-end hook. |
| `tool_calls` | `execute_tool_with_lock` in `crates/engine/src/core/engine/tool_execution.rs` — surface-agnostic, fires for exec and CLI too |
| `fleet_dispatch` | `create_queued_run_with_descriptor` in `crates/engine/src/fleet/manager.rs` — the single funnel that `create_run` and `create_queued_run` both land in; counting at either caller would double-count a plain `fleet run`. |
| `workflow_run` | bumped in `WorkflowTool::execute` (`crates/engine/src/tools/workflow/mod.rs`) only after `parse_workflow_action` returns an `Ok(WorkflowAction)`, never from `input["action"]`. The JSON Schema `enum` in `WorkflowTool::input_schema` is what is published *to the model* — a declaration, not a guard; the real parse also accepts `spawn\|wait\|list\|inspect\|stop\|abort`, and its reject arm (`Invalid workflow action '…'`) embeds the model string verbatim, so a rejected action is never counted. |
@@ -351,7 +351,7 @@ The workhorse. Everything a session accumulated ships here, once.
Why discriminants and nothing else: `ToolError::PathEscape`'s `Display` *is* an absolute path (`crates/tools/src/lib.rs:61`); `fim.rs:48-50`'s `Display` *is* a literal source fragment the model emitted; `secrets/src/lib.rs:50`'s `Display` carries the secret store's absolute path; every `LlmError` variant carries the raw provider HTTP body verbatim (`crates/engine/src/llm_client/mod.rs:327`), and a 400 from a content filter routinely echoes the prompt.
**`turn_wall`** — a per-session histogram of counts, never per-turn events. `lt_5s`, `5_30s`, `30_120s`, `gte_120s`. Recorded by `observe_turn_secs` next to the `turns` bump in `run_event_loop`, which already has the turn duration in hand.
**`turn_wall`** — a per-session histogram of counts, never per-turn events. `lt_5s`, `5_30s`, `30_120s`, `gte_120s`. Recorded by `observe_turn_secs` next to the `turns` bump in `record_model_turn_telemetry`; the duration is the time from the engine starting the turn to the turn settling, and the same sub-agent and RLM exclusion as `turns` applies. The sum of its four buckets equals `turns`.
### Event: `panic`
+2 -2
View File
@@ -259,8 +259,8 @@
},
{
"path": "docs/TELEMETRY.md",
"sha256": "0ac65dbd0e1d9d196d1c4421116852c3ab6a855062c375c25b09db993084c563",
"content": "# Codewhale product telemetry\n\n> 阅读简体中文版:[zh_hans/TELEMETRY.md](zh_hans/TELEMETRY.md)\n\n**Usage analytics are on by default in the current source.** The first\ninteractive launch gives a localized, nonblocking disclosure naming **Codewhale\nand PostHog**, with a direct route to turn counting off. Notice version `6`\ndescribes this policy; displaying it is not recorded as human acceptance.\nMissing an older acceptance record does not turn a fresh installation off.\nExisting explicit opt-outs remain off across upgrades, and unreadable privacy\nstate fails closed. Environment and CLI kill switches still take precedence.\n\nRead the disclosure with `codewhale config telemetry`. Turn usage off in\n`/settings` or run `codewhale config set telemetry false`. Turning it back on\nin Settings or with `codewhale config set telemetry true` updates the existing\ndurable preference and privacy records for new sessions. The compatibility\ncommand `codewhale config telemetry --accept-notice 5` remains available, but\nis not required for a fresh installation. Collection never includes work content or credentials.\n\nThe PostHog sink is optional and inert until the ingest operator explicitly\nconfigures a project token and approved regional host, and records staging\nproof that the actual egress path does not forward the original client IP.\nSource readiness does not establish a deployment, an activated processor,\nan egress receipt, or a retention setting.\n\nCodewhale does not collect conversations, code, prompts, files, file/repo/branch\nnames, model content, or credentials. It sends no per-turn or per-tool timeline.\nIt does send the closed, aggregate schema below: version and platform classes,\nsession duration/outcome, feature/error counters, and a random install id that\nrotates every 90 days.\n\n**There is now a real endpoint.** An enabled session sends its batches to the\nfirst-party ingest service at `https://telemetry.codewhale.net/v1/telemetry`,\nwhich is the shipped default for `telemetry_endpoint`. What that service is,\nwhat it stores, and what it structurally cannot store is spelled out in \"What\nthe endpoint does\" below.\n\n**To send nothing anywhere, turn telemetry off** (see \"Turning it off\"). To stay\nenabled but contact nobody, set `telemetry_endpoint = \"\"`: batches are then\nappended to `$CODEWHALE_HOME/telemetry/dryrun.jsonl` on your own machine, byte\nfor byte what the server would have received, and no HTTP client is ever\nconstructed. That file is how you audit this document against reality.\n\nThis document is the schema. It is not a summary of the schema: a test in\n`crates/telemetry` parses the field names out of this file and asserts set\nequality against the structs the serializer actually uses, so a field that is\nhere and not in the code — or in the code and not here — fails the build.\n\nLocal saved sessions and `codewhale metrics` remain available with telemetry\ndisabled and require no account or hosted reporting service. Disabling telemetry\ndoes not disable local usage receipts.\n\n## Turning it off\n\nThere are two off switches and they do different things. Both stop collection\ncompletely; only one of them erases anything.\n\n```sh\ncodewhale config set telemetry false # opt out: stops collection and erases state\nCODEWHALE_TELEMETRY=0 codewhale # kill switch: stops collection, erases nothing\nDO_NOT_TRACK=1 codewhale # the same kill switch, by the common convention\ncodewhale --telemetry false # the same kill switch, for one command\n```\n\n**`telemetry = false` in the config file is the opt-out.** It is a floor:\n`--telemetry true` and `CODEWHALE_TELEMETRY=1` both lose to it, because a\nsetting you can undo by accident from a wrapper script is not a setting. It\ndeletes the random install id, truncates every buffered event and every dry-run\nrecord, and writes a tombstone. Appends, identity/state writes, and delivery all\nshare the wipe's ordering lock, so once opt-out returns no pre-opt-out write or\nPOST remains in flight. If any part of that wipe fails, the tombstone is still\nthere and the buffer is undrainable — a failed wipe fails closed. Every later\nrun re-asserts the same tombstone for as long as the setting stands, so it\nsurvives. Turning telemetry back on in `/settings` records that explicit preference\nin both existing privacy registers; a fresh launch can then clear the tombstone. Nothing buffered before that point is\never sent.\n\n**The environment variable and the flag are kill switches, not opt-outs.**\nTelemetry is off for the run, nothing is written, nothing is sent — and\nnothing on disk is touched or deleted. That is deliberate: a harness or agent\nthat sets `CODEWHALE_TELEMETRY=0` for one command must not silently discard the\ninstall id and the dry-run records of the person who owns the machine. If you\nwant the erasing kind, use the config file.\n\n`CODEWHALE_TELEMELine truncated
"sha256": "6d50b9b54afd857a11057cfb42dbe1a2e648c9385e81352328114a2cc97bec73",
"content": "# Codewhale product telemetry\n\n> 阅读简体中文版:[zh_hans/TELEMETRY.md](zh_hans/TELEMETRY.md)\n\n**Usage analytics are on by default in the current source.** The first\ninteractive launch gives a localized, nonblocking disclosure naming **Codewhale\nand PostHog**, with a direct route to turn counting off. Notice version `6`\ndescribes this policy; displaying it is not recorded as human acceptance.\nMissing an older acceptance record does not turn a fresh installation off.\nExisting explicit opt-outs remain off across upgrades, and unreadable privacy\nstate fails closed. Environment and CLI kill switches still take precedence.\n\nRead the disclosure with `codewhale config telemetry`. Turn usage off in\n`/settings` or run `codewhale config set telemetry false`. Turning it back on\nin Settings or with `codewhale config set telemetry true` updates the existing\ndurable preference and privacy records for new sessions. The compatibility\ncommand `codewhale config telemetry --accept-notice 5` remains available, but\nis not required for a fresh installation. Collection never includes work content or credentials.\n\nThe PostHog sink is optional and inert until the ingest operator explicitly\nconfigures a project token and approved regional host, and records staging\nproof that the actual egress path does not forward the original client IP.\nSource readiness does not establish a deployment, an activated processor,\nan egress receipt, or a retention setting.\n\nCodewhale does not collect conversations, code, prompts, files, file/repo/branch\nnames, model content, or credentials. It sends no per-turn or per-tool timeline.\nIt does send the closed, aggregate schema below: version and platform classes,\nsession duration/outcome, feature/error counters, and a random install id that\nrotates every 90 days.\n\n**There is now a real endpoint.** An enabled session sends its batches to the\nfirst-party ingest service at `https://telemetry.codewhale.net/v1/telemetry`,\nwhich is the shipped default for `telemetry_endpoint`. What that service is,\nwhat it stores, and what it structurally cannot store is spelled out in \"What\nthe endpoint does\" below.\n\n**To send nothing anywhere, turn telemetry off** (see \"Turning it off\"). To stay\nenabled but contact nobody, set `telemetry_endpoint = \"\"`: batches are then\nappended to `$CODEWHALE_HOME/telemetry/dryrun.jsonl` on your own machine, byte\nfor byte what the server would have received, and no HTTP client is ever\nconstructed. That file is how you audit this document against reality.\n\nThis document is the schema. It is not a summary of the schema: a test in\n`crates/telemetry` parses the field names out of this file and asserts set\nequality against the structs the serializer actually uses, so a field that is\nhere and not in the code — or in the code and not here — fails the build.\n\nLocal saved sessions and `codewhale metrics` remain available with telemetry\ndisabled and require no account or hosted reporting service. Disabling telemetry\ndoes not disable local usage receipts.\n\n## Turning it off\n\nThere are two off switches and they do different things. Both stop collection\ncompletely; only one of them erases anything.\n\n```sh\ncodewhale config set telemetry false # opt out: stops collection and erases state\nCODEWHALE_TELEMETRY=0 codewhale # kill switch: stops collection, erases nothing\nDO_NOT_TRACK=1 codewhale # the same kill switch, by the common convention\ncodewhale --telemetry false # the same kill switch, for one command\n```\n\n**`telemetry = false` in the config file is the opt-out.** It is a floor:\n`--telemetry true` and `CODEWHALE_TELEMETRY=1` both lose to it, because a\nsetting you can undo by accident from a wrapper script is not a setting. It\ndeletes the random install id, truncates every buffered event and every dry-run\nrecord, and writes a tombstone. Appends, identity/state writes, and delivery all\nshare the wipe's ordering lock, so once opt-out returns no pre-opt-out write or\nPOST remains in flight. If any part of that wipe fails, the tombstone is still\nthere and the buffer is undrainable — a failed wipe fails closed. Every later\nrun re-asserts the same tombstone for as long as the setting stands, so it\nsurvives. Turning telemetry back on in `/settings` records that explicit preference\nin both existing privacy registers; a fresh launch can then clear the tombstone. Nothing buffered before that point is\never sent.\n\n**The environment variable and the flag are kill switches, not opt-outs.**\nTelemetry is off for the run, nothing is written, nothing is sent — and\nnothing on disk is touched or deleted. That is deliberate: a harness or agent\nthat sets `CODEWHALE_TELEMETRY=0` for one command must not silently discard the\ninstall id and the dry-run records of the person who owns the machine. If you\nwant the erasing kind, use the config file.\n\n`CODEWHALE_TELEMELine truncated
},
{
"path": "docs/public-surface-facts.json",