mirror of
https://github.com/codewhale-hq/Codewhale.git
synced 2026-11-07 17:09:13 -10:00
fix(engine): retain captured usage and settle interrupted workers
Bind native usage revisions, frozen route prices and original spend owners across Engine, child/review/Auto and native client projections. Preserve unknown usage instead of deriving money from unverified legacy totals. Keep Source ownership until real publication acknowledgment and retain failed settlement for later admitted flush/recovery. Copy frozen pricing audit labels into session snapshots without changing numeric corrections. Store continuable interrupted workers as terminal Interrupted while keeping the public waiting-for-follow-up view. This fixes a real workflow-child HTTP500 completion hang without weakening publication/rollback guards, replaying a request, discarding its checkpoint or changing live approvals. Verification: 102 unique existing cases covered by source-scoped reuse: 80 v7, 17 v8, 4 actual v10 Engine passes and 1 actual v11 TUI pass. V11 TUI-lib compile/list passed in450.30s; exact end-to-end case passed 1/1 against2338unchanged inputs and unchanged executable hash. Existing Apple __eh_frame compact-unwind linker warning remains. Independent bounded v9/v11 reviews justify reuse; this is not102fresh executions. Four obsolete existing tests are renamed/updated; no additional cases or code comments. Earlier ordinary production-library/CLI/Runtime HTTP observations retain their exact receipts. V9 ordinaryCLI build passed and actual HTTP500 child control exited naturally1 with task/run terminal failure,1POST/1fsynced synthetic effect, retained Missingrev2/checkpoint, real pipe EOF, joined handlers/closed sockets and no forced cleanup. Known/missingHTTP200 and 13CLI/3RuntimeHTTP controls retain earlier source identities. No external provider calls or fresh v11 ordinaryCLI claim. All failed generations kept. Receipt: CW/artifacts/engine-0103-execution-20261102/accounting-native-v11/RECEIPT.json SHA256:832ecfe1c1548edc9712bc31bb78288b79a375dbc0d2d4ba0cae454e9179a2b9 FullP04 remains open: raw presence, physical-attempt durability, original workspace, prompt layers, cache economics and finite retirement. Other release packets, native/installed/final gates and CW7704 remain open. Exclude the two temporary originalPR7187 web files; preserve their history.
This commit is contained in:
1 parent
39660a1d04
commit
ccc087723c
72 files changed
+8423
-2413
No files matched your search
@@ -160,6 +160,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Fixed
|
||||
|
||||
- Child, review and Auto usage keeps its captured session and route through late completion and persistence retries. Unreported usage stays unknown, cache writes and reasoning remain distinct, and cost calculations use the quote captured for the request.
|
||||
- Interrupted children persist their continuation checkpoint and complete their workflow task instead of leaving it waiting on a worker whose execution already ended. Live approval waits retain their existing behavior.
|
||||
- Session publication compares the document and crash checkpoint captured before work was queued. A delayed save refuses changed records without replacing newer history or clearing its checkpoint. Ordered completion retains real checkpoint acknowledgments, and native turn/control identity follows restores, undo and terminal saves. Legacy checkpoints require explicit recovery; canonical history recovery retains its original source and exact prepared generation.
|
||||
- Damaged usage sidecars keep valid recovery data readable with incomplete coverage and no publication receipt. Known retirement, changed raw records and unsafe writes still refuse.
|
||||
- Local control and ACP output serialize within the existing 8 MiB frame limit and stop a damaged or stalled output stream. Compatibility replies, events and relay stdout now share the 30-second write deadline; owner forwarding keeps its tighter 5 seconds. Standalone and attached ACP recover after complete invalid UTF-8 or oversized input frames, discard oversized tails through the real newline, and preserve cancellation, permissions and actual Core receipts.
|
||||
|
||||
+142
-55
@@ -205,6 +205,7 @@ pub struct TranslationProviderResponse {
|
||||
pub translated: Result<String>,
|
||||
pub route: crate::cost_status::EffectiveRouteEnvelope,
|
||||
pub usage: Option<Usage>,
|
||||
pub accounting: Option<crate::cost_status::RuntimeUsageRecord>,
|
||||
}
|
||||
|
||||
/// Universal client for the resolved provider route.
|
||||
@@ -3620,10 +3621,79 @@ impl CodewhaleClient {
|
||||
text: &str,
|
||||
model: &str,
|
||||
target_language: &str,
|
||||
source: &crate::cost_status::NativeUsageSource,
|
||||
) -> Result<TranslationProviderResponse> {
|
||||
let route = self.effective_route_envelope(model, chrono::Utc::now());
|
||||
let observation = source.observe_route(&route)?;
|
||||
source.admit_dispatch(&route).await?;
|
||||
let wire = observation.wire_observations.clone();
|
||||
let response = crate::llm_client::observe_request_retries(
|
||||
Some(observation),
|
||||
self.translate_prepared_with_usage(text, model, target_language, route.clone()),
|
||||
)
|
||||
.await;
|
||||
let wire = wire.drain();
|
||||
let usage = response
|
||||
.as_ref()
|
||||
.ok()
|
||||
.and_then(|response| response.usage.clone())
|
||||
.unwrap_or_default();
|
||||
let coverage = if wire.last_transmission.is_some() {
|
||||
wire.usage_coverage(usage != Usage::default(), response.is_ok())
|
||||
} else {
|
||||
match &response {
|
||||
Ok(_) if usage != Usage::default() => {
|
||||
crate::llm_client::ProviderUsageCoverage::Reported
|
||||
}
|
||||
Ok(_) => crate::llm_client::ProviderUsageCoverage::Missing,
|
||||
Err(error) => match error
|
||||
.downcast_ref::<crate::llm_client::LlmError>()
|
||||
.map(crate::llm_client::LlmError::transmission)
|
||||
{
|
||||
Some(crate::llm_client::RequestTransmission::NotDispatched) => {
|
||||
crate::llm_client::ProviderUsageCoverage::NotDispatched
|
||||
}
|
||||
Some(crate::llm_client::RequestTransmission::Rejected) => {
|
||||
crate::llm_client::ProviderUsageCoverage::Rejected
|
||||
}
|
||||
_ => crate::llm_client::ProviderUsageCoverage::Missing,
|
||||
},
|
||||
}
|
||||
};
|
||||
let record = source.prepare_observed(
|
||||
&route,
|
||||
&usage,
|
||||
coverage,
|
||||
if response.is_ok() {
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage
|
||||
} else {
|
||||
crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown
|
||||
},
|
||||
&wire,
|
||||
)?;
|
||||
if record.has_captured_origin() {
|
||||
crate::cost_status::settle_runtime_usage_record(
|
||||
crate::cost_status::scope_token(),
|
||||
None,
|
||||
&record,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
response.map(|mut response| {
|
||||
response.accounting = Some(record);
|
||||
response
|
||||
})
|
||||
}
|
||||
|
||||
async fn translate_prepared_with_usage(
|
||||
&self,
|
||||
text: &str,
|
||||
model: &str,
|
||||
target_language: &str,
|
||||
route: crate::cost_status::EffectiveRouteEnvelope,
|
||||
) -> Result<TranslationProviderResponse> {
|
||||
// Freeze pricing before either the remote-control gate or the provider
|
||||
// permit. A later live-catalog refresh must not reprice this request.
|
||||
let route = self.effective_route_envelope(model, chrono::Utc::now());
|
||||
let _inference = self.acquire_remote_control_inference_permit().await;
|
||||
let _permit = self.acquire_provider_request_permit().await;
|
||||
let model = self.wire_model_for_route(model);
|
||||
@@ -3666,6 +3736,8 @@ impl CodewhaleClient {
|
||||
translation_text_from_response(&response)
|
||||
};
|
||||
return Ok(TranslationProviderResponse {
|
||||
accounting: None,
|
||||
|
||||
translated,
|
||||
route,
|
||||
usage,
|
||||
@@ -3759,6 +3831,8 @@ impl CodewhaleClient {
|
||||
};
|
||||
|
||||
Ok(TranslationProviderResponse {
|
||||
accounting: None,
|
||||
|
||||
translated,
|
||||
route,
|
||||
usage,
|
||||
@@ -3768,9 +3842,17 @@ impl CodewhaleClient {
|
||||
/// Test adapter for asserting the translated text; production retains receipts.
|
||||
#[cfg(test)]
|
||||
async fn translate(&self, text: &str, model: &str, target_language: &str) -> Result<String> {
|
||||
self.translate_with_usage(text, model, target_language)
|
||||
.await?
|
||||
.translated
|
||||
self.translate_with_usage(
|
||||
text,
|
||||
model,
|
||||
target_language,
|
||||
&crate::cost_status::NativeUsageSource::capture(
|
||||
format!("translation-test:{}", uuid::Uuid::new_v4()),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
),
|
||||
)
|
||||
.await?
|
||||
.translated
|
||||
}
|
||||
|
||||
/// List every available model under the endpoint's verified pagination contract.
|
||||
@@ -4716,44 +4798,40 @@ impl CodewhaleClient {
|
||||
{
|
||||
let mut retry_cfg: LlmRetryConfig = self.retry.clone().into();
|
||||
retry_cfg.operation = operation;
|
||||
let request_result = crate::llm_client::observe_request_retries(
|
||||
None,
|
||||
with_retry(
|
||||
&retry_cfg,
|
||||
|| {
|
||||
let response = attempt();
|
||||
async move {
|
||||
self.wait_for_rate_limit().await;
|
||||
let response = response.await?;
|
||||
let status = response.status();
|
||||
if status.is_success() {
|
||||
return Ok(response);
|
||||
}
|
||||
let header_retry_after = extract_retry_after(response.headers());
|
||||
let raw = bounded_error_text(response, ERROR_BODY_MAX_BYTES).await;
|
||||
let body_retry_after = retry_delay_from_error_body(&raw);
|
||||
let body =
|
||||
self.disclosed_http_error_body(disclosure, status.as_u16(), &raw);
|
||||
let error = self.http_error_with_route_context(
|
||||
status.as_u16(),
|
||||
&body,
|
||||
header_retry_after,
|
||||
body_retry_after,
|
||||
);
|
||||
crate::llm_client::settle_outbound_http_error(&error);
|
||||
Err(error)
|
||||
let request_result = with_retry(
|
||||
&retry_cfg,
|
||||
|| {
|
||||
let response = attempt();
|
||||
async move {
|
||||
self.wait_for_rate_limit().await;
|
||||
let response = response.await?;
|
||||
let status = response.status();
|
||||
if status.is_success() {
|
||||
return Ok(response);
|
||||
}
|
||||
},
|
||||
Some(Box::new(|err, attempt, delay| {
|
||||
let (reason_label, _) = retry_reason_label_and_human(err);
|
||||
logging::warn(format!(
|
||||
"Isolated HTTP retry reason={} attempt={} delay={:.2}s",
|
||||
reason_label,
|
||||
attempt + 1,
|
||||
delay.as_secs_f64(),
|
||||
));
|
||||
})),
|
||||
),
|
||||
let header_retry_after = extract_retry_after(response.headers());
|
||||
let raw = bounded_error_text(response, ERROR_BODY_MAX_BYTES).await;
|
||||
let body_retry_after = retry_delay_from_error_body(&raw);
|
||||
let body = self.disclosed_http_error_body(disclosure, status.as_u16(), &raw);
|
||||
let error = self.http_error_with_route_context(
|
||||
status.as_u16(),
|
||||
&body,
|
||||
header_retry_after,
|
||||
body_retry_after,
|
||||
);
|
||||
crate::llm_client::settle_outbound_http_error(&error);
|
||||
Err(error)
|
||||
}
|
||||
},
|
||||
Some(Box::new(|err, attempt, delay| {
|
||||
let (reason_label, _) = retry_reason_label_and_human(err);
|
||||
logging::warn(format!(
|
||||
"Isolated HTTP retry reason={} attempt={} delay={:.2}s",
|
||||
reason_label,
|
||||
attempt + 1,
|
||||
delay.as_secs_f64(),
|
||||
));
|
||||
})),
|
||||
)
|
||||
.await;
|
||||
|
||||
@@ -4842,6 +4920,7 @@ impl CodewhaleClient {
|
||||
pub async fn create_message_without_response_cache(
|
||||
&self,
|
||||
request: MessageRequest,
|
||||
observation: crate::llm_client::RequestRetryObservation,
|
||||
) -> Result<MessageResponse> {
|
||||
let mut isolated = self.clone();
|
||||
isolated.isolated_request_state = true;
|
||||
@@ -4850,9 +4929,11 @@ impl CodewhaleClient {
|
||||
// auxiliary classifier call, however: it must neither consume nor
|
||||
// inherit that mutable foreground state.
|
||||
isolated.rate_limiter = Arc::new(AsyncMutex::new(TokenBucket::from_env()));
|
||||
isolated
|
||||
.create_message_with_cache_policy(request, false)
|
||||
.await
|
||||
crate::llm_client::observe_request_retries(
|
||||
Some(observation),
|
||||
isolated.create_message_with_cache_policy(request, false),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn create_message_with_cache_policy(
|
||||
@@ -6601,22 +6682,28 @@ pub fn main_turn_pricing_quote_at(
|
||||
endpoint_fingerprint: &str,
|
||||
dispatched_at: u64,
|
||||
) -> Option<crate::provider_catalog_live::ProviderLivePricingQuote> {
|
||||
let declared = client
|
||||
.filter(|client| {
|
||||
crate::cost_status::endpoint_fingerprint(client.base_url()).as_deref()
|
||||
== Some(endpoint_fingerprint)
|
||||
})
|
||||
.and_then(|client| {
|
||||
client.configured_pricing_quote_at(provider, provider_identity, model, dispatched_at)
|
||||
});
|
||||
crate::provider_catalog_live::declared_or_catalog_quote(declared, || {
|
||||
crate::provider_catalog_live::fresh_provider_live_pricing_quote_at(
|
||||
let client = client.filter(|client| {
|
||||
crate::cost_status::endpoint_fingerprint(client.base_url()).as_deref()
|
||||
== Some(endpoint_fingerprint)
|
||||
});
|
||||
let declared = client.and_then(|client| {
|
||||
client.configured_pricing_quote_at(provider, provider_identity, model, dispatched_at)
|
||||
});
|
||||
crate::provider_catalog_live::declared_or_catalog_quote(declared, || match client {
|
||||
Some(client) => crate::provider_catalog_live::fresh_dispatch_pricing_quote_at(
|
||||
provider,
|
||||
provider_identity,
|
||||
model,
|
||||
client.base_url(),
|
||||
dispatched_at,
|
||||
),
|
||||
None => crate::provider_catalog_live::fresh_provider_live_pricing_quote_at(
|
||||
provider,
|
||||
provider_identity,
|
||||
model,
|
||||
endpoint_fingerprint,
|
||||
dispatched_at,
|
||||
)
|
||||
),
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -7984,7 +7984,10 @@ mod google_thought_signature_tests {
|
||||
client.create_message_stream(request).await.map(|_| ())
|
||||
} else {
|
||||
client
|
||||
.create_message_without_response_cache(request)
|
||||
.create_message_without_response_cache(
|
||||
request,
|
||||
crate::llm_client::RequestRetryObservation::isolated(),
|
||||
)
|
||||
.await
|
||||
.map(|_| ())
|
||||
};
|
||||
|
||||
@@ -340,56 +340,58 @@ impl CodewhaleClient {
|
||||
/// Only a failure class leaves this function — provider error bodies can
|
||||
/// echo the prompt and must never reach receipts.
|
||||
///
|
||||
/// `dispatched` is set once both permits are held and the request is
|
||||
/// handed to the transport, so a caller whose deadline cancels this
|
||||
/// future can tell a possibly-billed request from one never sent.
|
||||
/// The caller retains its isolated physical observations across cancellation,
|
||||
/// including admission waits, HTTP refusal and response-body parsing.
|
||||
pub async fn system_one_decide(
|
||||
&self,
|
||||
body: &Value,
|
||||
dispatched: &std::sync::atomic::AtomicBool,
|
||||
observation: crate::llm_client::RequestRetryObservation,
|
||||
) -> std::result::Result<SystemOneResponse, AutoRouterFailure> {
|
||||
if serde_json::to_vec(body).map_or(true, |bytes| bytes.len() > DECISION_REQUEST_MAX_BYTES)
|
||||
|| !valid_decision_request(body)
|
||||
{
|
||||
return Err(AutoRouterFailure::NotRunnable);
|
||||
}
|
||||
let mut isolated = self.clone();
|
||||
isolated.isolated_request_state = true;
|
||||
isolated.retry.max_retries = 0;
|
||||
let _inference = isolated.acquire_remote_control_inference_permit().await;
|
||||
let _permit = isolated.acquire_provider_request_permit().await;
|
||||
let url = if isolated.api_provider == ProviderKind::Openrouter {
|
||||
// The OpenRouter Decisions API is a sibling of /api/v1, so keep
|
||||
// the configured origin/proxy prefix and replace only /v1.
|
||||
let mut url = reqwest::Url::parse(&isolated.base_url)
|
||||
.map_err(|_| AutoRouterFailure::NotRunnable)?;
|
||||
let prefix = url.path().trim_end_matches('/').trim_end_matches("/v1");
|
||||
url.set_path(&format!("{prefix}/alpha/decisions"));
|
||||
url.to_string()
|
||||
} else {
|
||||
api_url(&isolated.base_url, "systemone")
|
||||
};
|
||||
dispatched.store(true, std::sync::atomic::Ordering::Release);
|
||||
let response = isolated
|
||||
.send_json_with_retry(&url, body)
|
||||
.await
|
||||
.map_err(|error| router_failure_from_error(&error))?;
|
||||
let text = bounded_provider_catalog_text(response, DECISION_RESPONSE_MAX_BYTES)
|
||||
.await
|
||||
.map_err(|error| match error {
|
||||
CatalogRefreshError::Network => AutoRouterFailure::Transport,
|
||||
_ => AutoRouterFailure::InvalidAnswer,
|
||||
})?;
|
||||
let mut response: SystemOneResponse =
|
||||
serde_json::from_str(&text).map_err(|_| AutoRouterFailure::InvalidAnswer)?;
|
||||
response.answers_validated = Some(valid_decision_response(body, &response));
|
||||
response.model = response.model.map(|model| {
|
||||
self.redact_model_bound_text(&model)
|
||||
.chars()
|
||||
.take(128)
|
||||
.collect()
|
||||
});
|
||||
Ok(response)
|
||||
crate::llm_client::observe_request_retries(Some(observation), async {
|
||||
if serde_json::to_vec(body)
|
||||
.map_or(true, |bytes| bytes.len() > DECISION_REQUEST_MAX_BYTES)
|
||||
|| !valid_decision_request(body)
|
||||
{
|
||||
return Err(AutoRouterFailure::NotRunnable);
|
||||
}
|
||||
let mut isolated = self.clone();
|
||||
isolated.isolated_request_state = true;
|
||||
isolated.retry.max_retries = 0;
|
||||
let _inference = isolated.acquire_remote_control_inference_permit().await;
|
||||
let _permit = isolated.acquire_provider_request_permit().await;
|
||||
let url = if isolated.api_provider == ProviderKind::Openrouter {
|
||||
// The OpenRouter Decisions API is a sibling of /api/v1, so keep
|
||||
// the configured origin/proxy prefix and replace only /v1.
|
||||
let mut url = reqwest::Url::parse(&isolated.base_url)
|
||||
.map_err(|_| AutoRouterFailure::NotRunnable)?;
|
||||
let prefix = url.path().trim_end_matches('/').trim_end_matches("/v1");
|
||||
url.set_path(&format!("{prefix}/alpha/decisions"));
|
||||
url.to_string()
|
||||
} else {
|
||||
api_url(&isolated.base_url, "systemone")
|
||||
};
|
||||
let response = isolated
|
||||
.send_json_with_retry(&url, body)
|
||||
.await
|
||||
.map_err(|error| router_failure_from_error(&error))?;
|
||||
let text = bounded_provider_catalog_text(response, DECISION_RESPONSE_MAX_BYTES)
|
||||
.await
|
||||
.map_err(|error| match error {
|
||||
CatalogRefreshError::Network => AutoRouterFailure::Transport,
|
||||
_ => AutoRouterFailure::InvalidAnswer,
|
||||
})?;
|
||||
let mut response: SystemOneResponse =
|
||||
serde_json::from_str(&text).map_err(|_| AutoRouterFailure::InvalidAnswer)?;
|
||||
response.answers_validated = Some(valid_decision_response(body, &response));
|
||||
response.model = response.model.map(|model| {
|
||||
self.redact_model_bound_text(&model)
|
||||
.chars()
|
||||
.take(128)
|
||||
.collect()
|
||||
});
|
||||
Ok(response)
|
||||
})
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
||||
@@ -670,17 +672,21 @@ mod decisions_compatibility_tests {
|
||||
.acquire_provider_request_permit()
|
||||
.await
|
||||
.expect("shared permit");
|
||||
let dispatched = std::sync::atomic::AtomicBool::new(false);
|
||||
let observation = crate::llm_client::RequestRetryObservation::isolated();
|
||||
assert!(
|
||||
tokio::time::timeout(
|
||||
Duration::from_millis(30),
|
||||
clone.system_one_decide(&request(), &dispatched)
|
||||
clone.system_one_decide(&request(), observation.clone())
|
||||
)
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
!dispatched.load(std::sync::atomic::Ordering::Acquire),
|
||||
observation
|
||||
.wire_observations
|
||||
.drain()
|
||||
.transmissions
|
||||
.is_empty(),
|
||||
"a queued request has no dispatched-spend marker"
|
||||
);
|
||||
assert!(
|
||||
@@ -692,10 +698,10 @@ mod decisions_compatibility_tests {
|
||||
);
|
||||
drop(held);
|
||||
let answer = client
|
||||
.system_one_decide(&request(), &dispatched)
|
||||
.system_one_decide(&request(), observation.clone())
|
||||
.await
|
||||
.expect("one admitted request");
|
||||
assert!(dispatched.load(std::sync::atomic::Ordering::Acquire));
|
||||
assert_eq!(observation.wire_observations.drain().transmissions.len(), 1);
|
||||
assert_eq!(answer.answers_validated, Some(true));
|
||||
assert_eq!(
|
||||
client
|
||||
@@ -718,13 +724,23 @@ mod decisions_compatibility_tests {
|
||||
let mut body = response();
|
||||
body["usage"] = json!({"inputTokens": 287, "cost": 0.000012054});
|
||||
let response: SystemOneResponse = serde_json::from_value(body).expect("partial response");
|
||||
let batch = crate::model_routing::decision_usage_batch(&route, &response);
|
||||
assert!(
|
||||
batch.records.is_empty(),
|
||||
"a partial token count cannot produce a complete priced subtotal"
|
||||
let batch = crate::model_routing::decision_usage_batch(
|
||||
&route,
|
||||
&response,
|
||||
&crate::cost_status::NativeUsageSource::capture(
|
||||
"partial",
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
),
|
||||
&crate::llm_client::OutboundWireObservationBatch::default(),
|
||||
);
|
||||
assert_eq!(batch.dropped_records, 1);
|
||||
assert_eq!(batch.drop_records.len(), 1);
|
||||
assert_eq!(batch.records.len(), 1);
|
||||
assert_eq!(batch.records[0].usage.usage.input_tokens, 287);
|
||||
assert_eq!(
|
||||
batch.records[0].settlement.as_ref().unwrap().coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
);
|
||||
assert_eq!(batch.dropped_records, 0);
|
||||
assert!(batch.drop_records.is_empty());
|
||||
let mut invalid = request();
|
||||
invalid["questions"]["urgency"]["criteria"] = json!([null]);
|
||||
assert!(!valid_decision_request(&invalid));
|
||||
@@ -777,13 +793,31 @@ mod decisions_compatibility_tests {
|
||||
assert!(!usage.complete);
|
||||
assert_eq!(usage.reported_cost().as_deref(), Some("0.000012054"));
|
||||
let route = crate::cost_status::decision_receipt_fixture("malformed-count").route;
|
||||
let batch = crate::model_routing::decision_usage_batch(&route, &decoded);
|
||||
assert!(
|
||||
batch.records.is_empty(),
|
||||
"invalid counters cannot fabricate priced usage"
|
||||
let batch = crate::model_routing::decision_usage_batch(
|
||||
&route,
|
||||
&decoded,
|
||||
&crate::cost_status::NativeUsageSource::capture(
|
||||
"malformed-count",
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
),
|
||||
&crate::llm_client::OutboundWireObservationBatch::default(),
|
||||
);
|
||||
assert_eq!(batch.dropped_records, 1);
|
||||
assert_eq!(batch.drop_records.len(), 1);
|
||||
assert_eq!(batch.records.len(), 1);
|
||||
assert_eq!(
|
||||
batch.records[0].usage.usage.input_tokens,
|
||||
usage.input_tokens
|
||||
);
|
||||
assert_eq!(
|
||||
batch.records[0].usage.usage.output_tokens,
|
||||
usage.output_tokens
|
||||
);
|
||||
assert!(matches!(
|
||||
batch.records[0].settlement.as_ref().unwrap().coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
| crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
));
|
||||
assert_eq!(batch.dropped_records, 0);
|
||||
assert!(batch.drop_records.is_empty());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,3 @@
|
||||
|
||||
use super::*;
|
||||
use crate::client::chat::{
|
||||
build_chat_messages, build_chat_messages_for_request,
|
||||
@@ -1301,7 +1300,10 @@ async fn cache_free_message_call_neither_reads_nor_writes_global_cache() {
|
||||
);
|
||||
|
||||
let response = client
|
||||
.create_message_without_response_cache(request)
|
||||
.create_message_without_response_cache(
|
||||
request,
|
||||
crate::llm_client::RequestRetryObservation::isolated(),
|
||||
)
|
||||
.await
|
||||
.expect("cache-free provider call succeeds");
|
||||
assert_eq!(response.id, "chatcmpl-cache-free-provider");
|
||||
@@ -1375,7 +1377,10 @@ async fn cache_free_classifier_429_does_not_publish_global_retry_or_rate_limit_s
|
||||
top_p: None,
|
||||
};
|
||||
let error = client
|
||||
.create_message_without_response_cache(request)
|
||||
.create_message_without_response_cache(
|
||||
request,
|
||||
crate::llm_client::RequestRetryObservation::isolated(),
|
||||
)
|
||||
.await
|
||||
.expect_err("429 must fail when isolated retries are disabled");
|
||||
assert!(
|
||||
|
||||
@@ -52,8 +52,9 @@
|
||||
ProviderKind::Custom
|
||||
);
|
||||
|
||||
let route = crate::route_runtime::resolve_runtime_route(&config, ProviderKind::Custom, None)
|
||||
.expect("custom route should resolve");
|
||||
let route =
|
||||
crate::route_runtime::resolve_runtime_route(&config, ProviderKind::Custom, None)
|
||||
.expect("custom route should resolve");
|
||||
|
||||
// Provide the key the route's auth path will read.
|
||||
let client = {
|
||||
@@ -99,7 +100,15 @@
|
||||
|
||||
let client = deepseek_anthropic_client(&server);
|
||||
let response = client
|
||||
.translate_with_usage("Hello", "deepseek-chat", "Spanish")
|
||||
.translate_with_usage(
|
||||
"Hello",
|
||||
"deepseek-chat",
|
||||
"Spanish",
|
||||
&crate::cost_status::NativeUsageSource::capture(
|
||||
format!("translation-test:{}", uuid::Uuid::new_v4()),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
),
|
||||
)
|
||||
.await
|
||||
.expect("decoded provider response retains its receipt");
|
||||
|
||||
@@ -135,7 +144,15 @@
|
||||
|
||||
let client = deepseek_request_boundary_client("https://api.deepseek.com/v1", server.uri());
|
||||
let response = client
|
||||
.translate_with_usage("Hello", "deepseek-chat", "Spanish")
|
||||
.translate_with_usage(
|
||||
"Hello",
|
||||
"deepseek-chat",
|
||||
"Spanish",
|
||||
&crate::cost_status::NativeUsageSource::capture(
|
||||
format!("translation-test:{}", uuid::Uuid::new_v4()),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
),
|
||||
)
|
||||
.await
|
||||
.expect("provider success must retain its frozen route");
|
||||
assert_eq!(
|
||||
@@ -168,7 +185,15 @@
|
||||
|
||||
let client = deepseek_request_boundary_client("https://api.deepseek.com/v1", server.uri());
|
||||
let error = match client
|
||||
.translate_with_usage("Hello", "deepseek-chat", "Spanish")
|
||||
.translate_with_usage(
|
||||
"Hello",
|
||||
"deepseek-chat",
|
||||
"Spanish",
|
||||
&crate::cost_status::NativeUsageSource::capture(
|
||||
format!("translation-test:{}", uuid::Uuid::new_v4()),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(_) => panic!("HTTP failure must not become a provider-success receipt"),
|
||||
@@ -263,5 +288,7 @@
|
||||
"a trailing slash still recognizes the host"
|
||||
);
|
||||
assert!(!client_for("baseten", "https://127.0.0.1:9/v1").catalog_endpoint_is_baseten());
|
||||
assert!(!client_for("groq", "https://api.groq.com/openai/v1").catalog_endpoint_is_baseten());
|
||||
assert!(
|
||||
!client_for("groq", "https://api.groq.com/openai/v1").catalog_endpoint_is_baseten()
|
||||
);
|
||||
}
|
||||
@@ -257,7 +257,13 @@ async fn orcarouter_live_catalog_and_chat_through_the_provider_path() {
|
||||
"English",
|
||||
16,
|
||||
);
|
||||
match client.create_message_without_response_cache(request).await {
|
||||
match client
|
||||
.create_message_without_response_cache(
|
||||
request,
|
||||
crate::llm_client::RequestRetryObservation::isolated(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(response) => {
|
||||
assert!(
|
||||
!response.content.is_empty(),
|
||||
|
||||
@@ -82,17 +82,15 @@ pub trait CompactionNoticeSink: Send + Sync + std::fmt::Debug {
|
||||
/// Deliver one already-rendered, user-visible sentence.
|
||||
fn notice(&self, message: String);
|
||||
/// Captured dispatch origin for a child; ordinary engines retain the existing billing path.
|
||||
fn accounting_origin(&self) -> Option<(crate::cost_status::CostScopeToken, String, String)> {
|
||||
None
|
||||
fn accounting_origin(&self) -> std::io::Result<Option<crate::cost_status::NativeUsageOrigin>> {
|
||||
Ok(None)
|
||||
}
|
||||
/// Projection after the existing billing block has settled this exact response once.
|
||||
fn settled_usage<'a>(
|
||||
&'a self,
|
||||
_source: &'a str,
|
||||
_route: &'a crate::cost_status::EffectiveRouteEnvelope,
|
||||
_usage: &'a Usage,
|
||||
) -> std::pin::Pin<Box<dyn std::future::Future<Output = ()> + Send + 'a>> {
|
||||
Box::pin(async {})
|
||||
_record: &'a crate::cost_status::RuntimeUsageRecord,
|
||||
) -> std::pin::Pin<Box<dyn std::future::Future<Output = std::io::Result<()>> + Send + 'a>> {
|
||||
Box::pin(async { Ok(()) })
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1990,11 +1988,66 @@ async fn create_summary(
|
||||
|
||||
// Capture the session scope before awaiting so a late response cannot
|
||||
// accrue into a subsequently loaded/new session.
|
||||
let accounting_origin = notice_sink.and_then(CompactionNoticeSink::accounting_origin);
|
||||
let cost_scope = accounting_origin
|
||||
.as_ref()
|
||||
.map_or_else(crate::cost_status::scope_token, |origin| origin.0);
|
||||
let response = match client.create_message(request).await {
|
||||
let accounting_origin = notice_sink
|
||||
.map(CompactionNoticeSink::accounting_origin)
|
||||
.transpose()?
|
||||
.flatten();
|
||||
let cost_scope = accounting_origin.as_ref().map_or_else(
|
||||
crate::cost_status::scope_token,
|
||||
crate::cost_status::NativeUsageOrigin::scope,
|
||||
);
|
||||
let native_origin = if accounting_origin.is_some() {
|
||||
accounting_origin
|
||||
} else if let Some(owner) = config.runtime_cost_owner.as_deref() {
|
||||
Some(crate::cost_status::NativeUsageOrigin::runtime(
|
||||
cost_scope, owner,
|
||||
)?)
|
||||
} else if let Some(session_id) = prepared.session_id.as_ref() {
|
||||
let session_id = session_id.clone();
|
||||
Some(
|
||||
codewhale_runtime::runtime_control::owner_work(move || {
|
||||
let manager = crate::session_manager::SessionManager::default_location()?;
|
||||
Ok(crate::cost_status::NativeUsageOrigin::session(
|
||||
cost_scope,
|
||||
manager.sessions_dir().to_path_buf(),
|
||||
session_id,
|
||||
))
|
||||
})
|
||||
.await?,
|
||||
)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let source = crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
format!("compaction:{}", uuid::Uuid::new_v4()),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
native_origin,
|
||||
);
|
||||
let observation = source.observe_route(&cost_route)?;
|
||||
if let Some(record) = source.admit_dispatch(&cost_route).await? {
|
||||
if let Some(sink) = notice_sink {
|
||||
sink.settled_usage(&record).await?;
|
||||
}
|
||||
}
|
||||
let wire = observation.wire_observations.clone();
|
||||
let response = crate::llm_client::observe_request_retries(
|
||||
Some(observation),
|
||||
client.create_message(request),
|
||||
)
|
||||
.await;
|
||||
let record = source.response_record(&cost_route, &response, &wire.drain())?;
|
||||
if record.has_captured_origin() {
|
||||
crate::cost_status::settle_runtime_usage_record(
|
||||
cost_scope,
|
||||
config.runtime_cost_owner.as_deref(),
|
||||
&record,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
if let Some(sink) = notice_sink {
|
||||
sink.settled_usage(&record).await?;
|
||||
}
|
||||
let response = match response {
|
||||
Ok(response) => response,
|
||||
// A byte-side size rejection can also read like a length problem
|
||||
// (gateway HTML pages carry their own wording); the size ladder
|
||||
@@ -2079,64 +2132,6 @@ async fn create_summary(
|
||||
// A rejected summary or canceled retry still consumed these tokens.
|
||||
crate::core::turn::add_usage_to(invocation_usage, &response.usage);
|
||||
|
||||
let source_id = format!(
|
||||
"compaction:dispatch:{}:response:{}",
|
||||
cost_route
|
||||
.dispatched_at
|
||||
.timestamp_nanos_opt()
|
||||
.unwrap_or_default(),
|
||||
response.id
|
||||
);
|
||||
if let Some((scope, session, agent)) = accounting_origin.as_ref() {
|
||||
if crate::core::engine::turn_loop::usage_has_reported_data(&response.usage) {
|
||||
if let Some(owner) = config.runtime_cost_owner.as_deref() {
|
||||
crate::cost_status::report_effective_route_for_runtime(
|
||||
*scope,
|
||||
Some(owner),
|
||||
&source_id,
|
||||
&cost_route,
|
||||
&response.usage,
|
||||
);
|
||||
} else {
|
||||
crate::usage_persistence::report_effective_route_for_interactive_origin(
|
||||
*scope,
|
||||
session,
|
||||
agent,
|
||||
&source_id,
|
||||
&cost_route,
|
||||
&response.usage,
|
||||
);
|
||||
}
|
||||
} else if let Some(owner) = config.runtime_cost_owner.as_deref() {
|
||||
crate::cost_status::report_unreceipted_provider_success(
|
||||
*scope,
|
||||
Some(owner),
|
||||
&source_id,
|
||||
&cost_route,
|
||||
);
|
||||
} else {
|
||||
crate::usage_persistence::report_unreceipted_for_interactive_origin(
|
||||
*scope,
|
||||
session,
|
||||
agent,
|
||||
&source_id,
|
||||
&cost_route,
|
||||
);
|
||||
}
|
||||
} else {
|
||||
crate::cost_status::report_effective_route_for_runtime(
|
||||
cost_scope,
|
||||
config.runtime_cost_owner.as_deref(),
|
||||
&source_id,
|
||||
&cost_route,
|
||||
&response.usage,
|
||||
);
|
||||
}
|
||||
if let Some(sink) = notice_sink {
|
||||
sink.settled_usage(&source_id, &cost_route, &response.usage)
|
||||
.await;
|
||||
}
|
||||
|
||||
// Usage above is already billed; a provider-declared incomplete
|
||||
// summary must still fail rather than replace the session history
|
||||
// with a fragment.
|
||||
|
||||
@@ -2880,6 +2880,8 @@ impl Engine {
|
||||
let pending_post_turn = self.reserve_post_turn_snapshot();
|
||||
let status = terminal_turn_status_at_settlement(status, self.cancel_token.is_cancelled());
|
||||
terminal_permit.send(Event::TurnComplete {
|
||||
source_accounting: true,
|
||||
|
||||
// The composer shell turn started above; it never carries a host
|
||||
// submission envelope or compaction id.
|
||||
turn_id: Some(turn_id.clone()),
|
||||
@@ -4150,7 +4152,7 @@ impl Engine {
|
||||
}
|
||||
}
|
||||
Op::StartFleetMember { start, tx } => {
|
||||
let result = match self.off_turn_subagent_runtime() {
|
||||
let result = match self.off_turn_subagent_runtime().await {
|
||||
Some(runtime) => {
|
||||
crate::tools::subagent::start_fleet_member(
|
||||
Arc::clone(&self.subagent_manager),
|
||||
@@ -4183,7 +4185,7 @@ impl Engine {
|
||||
}
|
||||
Op::FollowUpSubAgent { agent_id, text } => {
|
||||
let active_session_id = self.session.id.clone();
|
||||
let runtime = self.off_turn_subagent_runtime();
|
||||
let runtime = self.off_turn_subagent_runtime().await;
|
||||
let manager_handle = Arc::clone(&self.subagent_manager);
|
||||
let manager = Arc::clone(&manager_handle);
|
||||
let session = active_session_id.clone();
|
||||
@@ -5574,6 +5576,8 @@ impl Engine {
|
||||
let _ = self.send_event(Event::error(envelope)).await;
|
||||
let _ = self
|
||||
.send_event(Event::TurnComplete {
|
||||
source_accounting: true,
|
||||
|
||||
turn_id: None,
|
||||
submission_id,
|
||||
compaction_id: None,
|
||||
@@ -6035,6 +6039,13 @@ impl Engine {
|
||||
|
||||
let mut tool_context = self.build_tool_context_for_turn(input_policy, &route);
|
||||
if wiring.is_live() {
|
||||
if let Some(owner) = self.config.compaction.runtime_cost_owner.as_deref() {
|
||||
tool_context.accounting_origin = crate::cost_status::NativeUsageOrigin::runtime(
|
||||
crate::cost_status::scope_token(),
|
||||
owner,
|
||||
)
|
||||
.ok();
|
||||
}
|
||||
tool_context.rlm_caller = rlm_host::CapturedRlmCaller::capture(
|
||||
self,
|
||||
input_policy,
|
||||
@@ -6602,6 +6613,39 @@ impl Engine {
|
||||
let compaction = *compaction;
|
||||
let initial_routed_usage = *initial_routed_usage;
|
||||
let initial_usage_owner = compaction.runtime_cost_owner.clone();
|
||||
for record in initial_routed_usage
|
||||
.records
|
||||
.iter()
|
||||
.filter(|record| record.settlement.is_some())
|
||||
{
|
||||
let settled = if record.has_captured_origin() {
|
||||
crate::cost_status::settle_runtime_usage_record(
|
||||
crate::cost_status::scope_token(),
|
||||
initial_usage_owner.as_deref(),
|
||||
record,
|
||||
)
|
||||
.await
|
||||
} else {
|
||||
Err(std::io::Error::new(
|
||||
std::io::ErrorKind::PermissionDenied,
|
||||
"initial usage has no captured native origin",
|
||||
))
|
||||
};
|
||||
if let Err(error) = settled {
|
||||
let message = format!(
|
||||
"Cannot start the turn while classifier usage remains unsettled: {error}"
|
||||
);
|
||||
let _ = self
|
||||
.send_event(Event::error(ErrorEnvelope::classify(
|
||||
message.clone(),
|
||||
false,
|
||||
)))
|
||||
.await;
|
||||
return SendMessageOutcome::NotStarted {
|
||||
error: Some(message),
|
||||
};
|
||||
}
|
||||
}
|
||||
if autonomous && self.cancel_token.is_cancelled() {
|
||||
crate::cost_status::report_runtime_usage_batch(
|
||||
crate::cost_status::scope_token(),
|
||||
@@ -6789,6 +6833,8 @@ impl Engine {
|
||||
.send_event(Event::error(ErrorEnvelope::fatal_auth(message.clone())))
|
||||
.await;
|
||||
terminal_permit.send(Event::TurnComplete {
|
||||
source_accounting: true,
|
||||
|
||||
turn_id: None,
|
||||
submission_id,
|
||||
compaction_id: None,
|
||||
@@ -7038,17 +7084,11 @@ impl Engine {
|
||||
initial_usage_owner.as_deref(),
|
||||
&crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: initial_routed_usage.decisions.clone(),
|
||||
records: initial_routed_usage.records.clone(),
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
for record in &initial_routed_usage.records {
|
||||
crate::cost_status::report_effective_route_for_runtime(
|
||||
initial_cost_scope,
|
||||
initial_usage_owner.as_deref(),
|
||||
&record.source_id,
|
||||
&record.usage.route,
|
||||
&record.usage.usage,
|
||||
);
|
||||
let _ = self
|
||||
.send_event(Event::RoutedTurnUsage {
|
||||
usage: record.usage.usage.clone(),
|
||||
@@ -7115,6 +7155,8 @@ impl Engine {
|
||||
);
|
||||
let error = (status == TurnOutcomeStatus::Failed).then_some(message.clone());
|
||||
terminal_permit.send(Event::TurnComplete {
|
||||
source_accounting: true,
|
||||
|
||||
// `TurnStarted` already named this turn; the failed terminal
|
||||
// echoes it and the submission that started it.
|
||||
turn_id: Some(turn.id.clone()),
|
||||
@@ -7144,13 +7186,12 @@ impl Engine {
|
||||
return outcome;
|
||||
}
|
||||
|
||||
// Headless/runtime hosts supply their durable turn owner. Interactive
|
||||
// turns historically supplied none, leaving a detached child with
|
||||
// only the soon-to-be-sealed mailbox. Install this turn-local sink
|
||||
// only after every pre-dispatch failure return, and retire/clear it at
|
||||
// settlement so the next turn always receives a fresh owner.
|
||||
let interactive_runtime_cost_owner = if self.config.terminal_chrome_enabled
|
||||
&& self.config.compaction.runtime_cost_owner.is_none()
|
||||
// Headless/runtime hosts supply their durable turn owner; an embedded
|
||||
// host may supply its captured Session origin. Only an otherwise
|
||||
// unowned interactive turn installs this turn-local fallback, after
|
||||
// pre-dispatch failures, and retires it at settlement.
|
||||
let interactive_runtime_cost_owner = if self.config.compaction.runtime_cost_owner.is_none()
|
||||
&& self.config.runtime_services.accounting_origin.is_none()
|
||||
{
|
||||
let owner = format!("interactive:{}:{}", self.session.id, turn.id);
|
||||
crate::usage_persistence::register_persistent_interactive_runtime_usage_sink(
|
||||
@@ -7165,6 +7206,22 @@ impl Engine {
|
||||
None
|
||||
};
|
||||
|
||||
let turn_accounting_origin = self
|
||||
.config
|
||||
.compaction
|
||||
.runtime_cost_owner
|
||||
.as_deref()
|
||||
.map(|owner| {
|
||||
crate::cost_status::NativeUsageOrigin::runtime(
|
||||
crate::cost_status::scope_token(),
|
||||
owner,
|
||||
)
|
||||
})
|
||||
.transpose()
|
||||
.map(|origin| {
|
||||
origin.or_else(|| self.config.runtime_services.accounting_origin.clone())
|
||||
});
|
||||
|
||||
let previous_goal_objective = self.config.goal_objective.clone();
|
||||
let previous_goal_token_budget = self.config.goal_token_budget;
|
||||
let previous_goal_status = self.config.goal_status;
|
||||
@@ -7441,7 +7498,7 @@ impl Engine {
|
||||
// so the stall watchdog never reports a turn that already ended.
|
||||
self.turn_heartbeat.idle();
|
||||
let _post_timing = codewhale_runtime::logging::PhaseTimer::start("turn_postprocessing");
|
||||
let (mut status, error) = match turn_result {
|
||||
let (mut status, mut error) = match turn_result {
|
||||
Ok(outcome) => outcome,
|
||||
Err(panic) => {
|
||||
let detail = codewhale_util::utils::panic_message(&*panic);
|
||||
@@ -7457,6 +7514,20 @@ impl Engine {
|
||||
}
|
||||
};
|
||||
|
||||
let accounting_settlement = match turn_accounting_origin {
|
||||
Ok(Some(origin)) => origin.flush_retained_usage().await,
|
||||
Ok(None) => Ok(()),
|
||||
Err(error) => Err(error),
|
||||
};
|
||||
if let Err(accounting_error) = accounting_settlement {
|
||||
if status == TurnOutcomeStatus::Completed {
|
||||
status = TurnOutcomeStatus::Failed;
|
||||
}
|
||||
error.get_or_insert_with(|| {
|
||||
format!("Turn usage remains unacknowledged: {accounting_error}")
|
||||
});
|
||||
}
|
||||
|
||||
// Update session usage
|
||||
self.session.total_usage.add(&turn.usage);
|
||||
record_model_turn_telemetry(self.api_provider, &self.session.model, &turn, status);
|
||||
@@ -7518,6 +7589,7 @@ impl Engine {
|
||||
.then(|| {
|
||||
crate::tools::subagent::advisor::AdvisorUsageContext::capture(
|
||||
self.config.compaction.runtime_cost_owner.as_deref(),
|
||||
self.config.runtime_services.accounting_origin.clone(),
|
||||
)
|
||||
});
|
||||
if let Some(owner) = interactive_runtime_cost_owner.as_deref() {
|
||||
@@ -7555,6 +7627,8 @@ impl Engine {
|
||||
// name was read as user-visible delivery on Interrupted turns that
|
||||
// rendered nothing.)
|
||||
let completion_sender = terminal_permit.send(Event::TurnComplete {
|
||||
source_accounting: true,
|
||||
|
||||
// The authoritative terminal names the turn that actually ran and
|
||||
// the host submission that started it (`None` for self-starts).
|
||||
turn_id: Some(turn.id.clone()),
|
||||
@@ -7704,6 +7778,8 @@ impl Engine {
|
||||
.await;
|
||||
let _ = self
|
||||
.send_event(Event::TurnComplete {
|
||||
source_accounting: true,
|
||||
|
||||
// Purge carries none of the turn, submission or compaction
|
||||
// identities: it never starts a model turn.
|
||||
turn_id: None,
|
||||
@@ -7728,17 +7804,38 @@ impl Engine {
|
||||
.await;
|
||||
let messages_before = self.session.messages.len();
|
||||
|
||||
let (status, error) = match run_purge(
|
||||
&client,
|
||||
self.api_provider,
|
||||
&self.session.id,
|
||||
&self.session.messages,
|
||||
&self.session.model,
|
||||
self.session.reasoning_effort.clone(),
|
||||
client.effective_max_output_tokens(&self.session.model),
|
||||
)
|
||||
.await
|
||||
{
|
||||
let purge = async {
|
||||
let origin =
|
||||
if let Some(origin) = self.config.runtime_services.accounting_origin.clone() {
|
||||
origin
|
||||
} else {
|
||||
let scope = crate::cost_status::scope_token();
|
||||
let session_id = self.session.id.clone();
|
||||
codewhale_runtime::runtime_control::owner_work(move || {
|
||||
let manager = crate::session_manager::SessionManager::default_location()?;
|
||||
Ok(crate::cost_status::NativeUsageOrigin::session(
|
||||
scope,
|
||||
manager.sessions_dir().to_path_buf(),
|
||||
session_id,
|
||||
))
|
||||
})
|
||||
.await
|
||||
.map_err(|error| format!("Purge accounting admission failed: {error}"))?
|
||||
};
|
||||
run_purge(
|
||||
&client,
|
||||
self.api_provider,
|
||||
&self.session.id,
|
||||
&self.session.messages,
|
||||
&self.session.model,
|
||||
self.session.reasoning_effort.clone(),
|
||||
client.effective_max_output_tokens(&self.session.model),
|
||||
origin,
|
||||
)
|
||||
.await
|
||||
}
|
||||
.await;
|
||||
let (status, error) = match purge {
|
||||
Ok(result) => {
|
||||
let messages_after = result.messages.len();
|
||||
self.session.replace_messages(result.messages);
|
||||
@@ -7768,6 +7865,8 @@ impl Engine {
|
||||
|
||||
let _ = self
|
||||
.send_event(Event::TurnComplete {
|
||||
source_accounting: true,
|
||||
|
||||
// Purge carries none of the turn, submission or compaction
|
||||
// identities: it never starts a model turn.
|
||||
turn_id: None,
|
||||
@@ -7876,7 +7975,11 @@ impl Engine {
|
||||
reasoning_effort: self.session.reasoning_effort.clone(),
|
||||
reasoning_effort_auto: self.session.reasoning_effort_auto,
|
||||
};
|
||||
self.build_tool_context_for_turn(&authority, &route)
|
||||
let mut context = self.build_tool_context_for_turn(&authority, &route);
|
||||
if context.accounting_origin.is_none() {
|
||||
context.accounting_origin = self.config.runtime_services.accounting_origin.clone();
|
||||
}
|
||||
context
|
||||
}
|
||||
|
||||
/// Build a child runtime from the installed session route, outside any
|
||||
@@ -7884,12 +7987,32 @@ impl Engine {
|
||||
/// (`Op::FollowUpSubAgent`). Mirrors the per-turn runtime the `agent` tool
|
||||
/// receives, minus the turn-scoped fork context and mailbox barrier: a
|
||||
/// continued fork is a background child of the session, not of a turn.
|
||||
fn off_turn_subagent_runtime(&self) -> Option<SubAgentRuntime> {
|
||||
async fn off_turn_subagent_runtime(&self) -> Option<SubAgentRuntime> {
|
||||
let client = self.codewhale_client.clone()?;
|
||||
let mode = self.current_mode;
|
||||
let allow_shell = self.session.allow_shell && !matches!(mode, AppMode::Plan);
|
||||
let shell_policy = shell_policy_for_mode(mode, allow_shell);
|
||||
let tool_context = self.build_tool_context(mode, self.session.auto_approve);
|
||||
let mut tool_context = self.build_tool_context(mode, self.session.auto_approve);
|
||||
if tool_context.accounting_origin.is_none() {
|
||||
let scope = crate::cost_status::scope_token();
|
||||
let session_id = self.session.id.clone();
|
||||
match codewhale_runtime::runtime_control::owner_work(move || {
|
||||
let manager = crate::session_manager::SessionManager::default_location()?;
|
||||
Ok(crate::cost_status::NativeUsageOrigin::session(
|
||||
scope,
|
||||
manager.sessions_dir().to_path_buf(),
|
||||
session_id,
|
||||
))
|
||||
})
|
||||
.await
|
||||
{
|
||||
Ok(origin) => tool_context.accounting_origin = Some(origin),
|
||||
Err(error) => {
|
||||
tracing::warn!(%error, "off-turn child accounting admission failed");
|
||||
return None;
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut rt = SubAgentRuntime::new(
|
||||
client,
|
||||
self.session.model.clone(),
|
||||
|
||||
@@ -35,25 +35,39 @@ pub(super) enum AutoCompactionStep {
|
||||
struct EngineCompactionNoticeSink {
|
||||
tx: mpsc::Sender<Event>,
|
||||
child: Option<Arc<crate::tools::subagent::engine::ChildAuthority>>,
|
||||
origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
}
|
||||
|
||||
impl crate::compaction::CompactionNoticeSink for EngineCompactionNoticeSink {
|
||||
fn notice(&self, message: String) {
|
||||
let _ = self.tx.try_send(Event::Status { message });
|
||||
}
|
||||
fn accounting_origin(&self) -> Option<(crate::cost_status::CostScopeToken, String, String)> {
|
||||
self.child.as_ref().map(|child| child.accounting_origin())
|
||||
fn accounting_origin(&self) -> std::io::Result<Option<crate::cost_status::NativeUsageOrigin>> {
|
||||
if let Some(child) = self.child.as_ref() {
|
||||
return child
|
||||
.runtime
|
||||
.context
|
||||
.accounting_origin
|
||||
.clone()
|
||||
.map(Some)
|
||||
.ok_or_else(|| {
|
||||
std::io::Error::new(
|
||||
std::io::ErrorKind::PermissionDenied,
|
||||
"child compaction has no admitted accounting origin",
|
||||
)
|
||||
});
|
||||
}
|
||||
Ok(self.origin.clone())
|
||||
}
|
||||
fn settled_usage<'a>(
|
||||
&'a self,
|
||||
source: &'a str,
|
||||
route: &'a crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &'a Usage,
|
||||
) -> std::pin::Pin<Box<dyn std::future::Future<Output = ()> + Send + 'a>> {
|
||||
record: &'a crate::cost_status::RuntimeUsageRecord,
|
||||
) -> std::pin::Pin<Box<dyn std::future::Future<Output = std::io::Result<()>> + Send + 'a>> {
|
||||
Box::pin(async move {
|
||||
if let Some(child) = self.child.as_ref() {
|
||||
child.project_settled_response(source, route, usage).await;
|
||||
child.project_settled_response(record).await?;
|
||||
}
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -295,6 +309,13 @@ impl Engine {
|
||||
prepared.session_id = Some(self.session.id.clone());
|
||||
prepared.notice_sink = Some(std::sync::Arc::new(EngineCompactionNoticeSink {
|
||||
tx: self.tx_event.clone(),
|
||||
origin: self
|
||||
.config
|
||||
.compaction
|
||||
.runtime_cost_owner
|
||||
.is_none()
|
||||
.then(|| self.config.runtime_services.accounting_origin.clone())
|
||||
.flatten(),
|
||||
child: self
|
||||
.child_host
|
||||
.as_ref()
|
||||
@@ -326,6 +347,8 @@ impl Engine {
|
||||
self.emit_compaction_cancelled(id, false, message).await;
|
||||
let _ = self
|
||||
.send_event(Event::TurnComplete {
|
||||
source_accounting: true,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id,
|
||||
@@ -364,6 +387,8 @@ impl Engine {
|
||||
// operation: hosts waiting on its lifecycle must be released.
|
||||
let _ = self
|
||||
.send_event(Event::TurnComplete {
|
||||
source_accounting: true,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id,
|
||||
@@ -419,6 +444,8 @@ impl Engine {
|
||||
.await;
|
||||
let _ = self
|
||||
.send_event(Event::TurnComplete {
|
||||
source_accounting: true,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id,
|
||||
@@ -472,6 +499,8 @@ impl Engine {
|
||||
.await;
|
||||
let _ = self
|
||||
.send_event(Event::TurnComplete {
|
||||
source_accounting: true,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id,
|
||||
@@ -502,6 +531,8 @@ impl Engine {
|
||||
.await;
|
||||
let _ = self
|
||||
.send_event(Event::TurnComplete {
|
||||
source_accounting: true,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id,
|
||||
@@ -589,6 +620,8 @@ impl Engine {
|
||||
|
||||
let _ = self
|
||||
.send_event(Event::TurnComplete {
|
||||
source_accounting: true,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id,
|
||||
@@ -1238,7 +1271,11 @@ mod tests {
|
||||
#[tokio::test]
|
||||
async fn compaction_notice_sink_delivers_a_status_event() {
|
||||
let (tx, mut rx) = mpsc::channel(4);
|
||||
let sink = EngineCompactionNoticeSink { tx, child: None };
|
||||
let sink = EngineCompactionNoticeSink {
|
||||
tx,
|
||||
child: None,
|
||||
origin: None,
|
||||
};
|
||||
sink.notice("Making room re-encoded 2 inline image(s)".to_string());
|
||||
match rx.recv().await {
|
||||
Some(Event::Status { message }) => {
|
||||
|
||||
@@ -804,6 +804,7 @@ async fn plan_with_reasoning(
|
||||
prompt: &str,
|
||||
) -> crate::turn_route_plan::PlannedTurnRoute {
|
||||
crate::turn_route_plan::plan_turn_route(crate::turn_route_plan::TurnRoutePlanRequest {
|
||||
accounting_origin: None,
|
||||
route_config: config,
|
||||
app_route_identity: identity,
|
||||
api_provider: provider,
|
||||
|
||||
@@ -382,8 +382,15 @@ impl Engine {
|
||||
let transport_retries = observation.retries.clone();
|
||||
let mut pending = Some(super::turn_loop::PendingProviderRequestReceipt {
|
||||
request_id: observation.request_id.clone(),
|
||||
accounting_source: crate::cost_status::NativeUsageSource::capture(
|
||||
observation.request_id.clone(),
|
||||
crate::cost_status::UsageSpendOwner::Primary,
|
||||
),
|
||||
accounting_scope: crate::cost_status::scope_token(),
|
||||
accounting_owner: self.config.compaction.runtime_cost_owner.clone(),
|
||||
route: client.effective_route_envelope(&self.session.model, chrono::Utc::now()),
|
||||
primary: true,
|
||||
accounting_owned_elsewhere: true,
|
||||
invoked: true,
|
||||
turn_id: turn.id.clone(),
|
||||
request_index: turn.stop_diagnostics.model_requests_started,
|
||||
@@ -398,6 +405,30 @@ impl Engine {
|
||||
transport_retries: transport_retries.clone(),
|
||||
started_at,
|
||||
});
|
||||
core.accounting_owner = crate::cost_status::UsageSpendOwner::Primary;
|
||||
core.accounting_origin =
|
||||
if let Some(owner) = self.config.compaction.runtime_cost_owner.as_deref() {
|
||||
Some(crate::cost_status::NativeUsageOrigin::runtime(
|
||||
crate::cost_status::scope_token(),
|
||||
owner,
|
||||
)?)
|
||||
} else if let Some(origin) = self.config.runtime_services.accounting_origin.clone() {
|
||||
Some(origin)
|
||||
} else {
|
||||
let scope = crate::cost_status::scope_token();
|
||||
let session_id = self.session.id.clone();
|
||||
Some(
|
||||
codewhale_runtime::runtime_control::owner_work(move || {
|
||||
let manager = crate::session_manager::SessionManager::default_location()?;
|
||||
Ok(crate::cost_status::NativeUsageOrigin::session(
|
||||
scope,
|
||||
manager.sessions_dir().to_path_buf(),
|
||||
session_id,
|
||||
))
|
||||
})
|
||||
.await?,
|
||||
)
|
||||
};
|
||||
let result = tokio::select! {
|
||||
biased;
|
||||
result = crate::llm_client::observe_request_retries(Some(observation),
|
||||
@@ -408,23 +439,27 @@ impl Engine {
|
||||
.stop_diagnostics
|
||||
.transport_retries
|
||||
.saturating_add(transport_retries.load(std::sync::atomic::Ordering::Relaxed));
|
||||
self.settle_provider_request_receipt(
|
||||
&mut pending,
|
||||
result.as_ref().ok().map(|value| &value.usage),
|
||||
if result.is_ok() {
|
||||
"review_response"
|
||||
} else if self.cancel_token.is_cancelled() {
|
||||
"review_cancelled"
|
||||
} else {
|
||||
"review_failed"
|
||||
},
|
||||
)
|
||||
.await;
|
||||
let accounting = self
|
||||
.settle_provider_request_receipt(
|
||||
&mut pending,
|
||||
result.as_ref().ok().map(|value| &value.usage),
|
||||
if result.is_ok() {
|
||||
"review_response"
|
||||
} else if self.cancel_token.is_cancelled() {
|
||||
"review_cancelled"
|
||||
} else {
|
||||
"review_failed"
|
||||
},
|
||||
)
|
||||
.await
|
||||
.map_err(anyhow::Error::msg)?;
|
||||
let mut result = result?;
|
||||
let accounting = result.source_records.last().cloned().or(accounting);
|
||||
turn.add_parent_usage(&result.usage);
|
||||
let elapsed_ms = u64::try_from(started_at.elapsed().as_millis()).unwrap_or(u64::MAX);
|
||||
let _ = self
|
||||
.send_event(Event::TurnUsage {
|
||||
accounting,
|
||||
max_output_tokens: Some(cap),
|
||||
usage: result.usage.clone(),
|
||||
duration_ms: elapsed_ms,
|
||||
|
||||
@@ -46,10 +46,7 @@ pub struct CapturedRlmCaller {
|
||||
review_policy: Arc<AutoReviewPolicy>,
|
||||
deadline: Option<tokio::time::Instant>,
|
||||
origin_scope: crate::cost_status::CostScopeToken,
|
||||
origin_session: String,
|
||||
origin_turn: String,
|
||||
child_accounting: Option<crate::tools::subagent::engine::ChildAccountingProjection>,
|
||||
scheduler: tokio::runtime::Handle,
|
||||
runtime_owner: Option<String>,
|
||||
_runtime_lease: Option<crate::cost_status::RuntimeUsageLease>,
|
||||
}
|
||||
@@ -106,8 +103,6 @@ impl CapturedRlmCaller {
|
||||
.child_host
|
||||
.as_ref()
|
||||
.map(|child| child.authority.accounting_projection());
|
||||
let scheduler = tokio::runtime::Handle::try_current()
|
||||
.map_err(|_| ToolError::not_available("RLM caller has no held Engine scheduler"))?;
|
||||
let runtime_owner = engine.config.compaction.runtime_cost_owner.clone();
|
||||
let runtime_lease = runtime_owner
|
||||
.as_deref()
|
||||
@@ -146,10 +141,7 @@ impl CapturedRlmCaller {
|
||||
review_policy: Arc::clone(&engine.shared_auto_review_policy),
|
||||
deadline,
|
||||
origin_scope: crate::cost_status::scope_token(),
|
||||
origin_session: context.state_namespace.clone(),
|
||||
origin_turn: origin_turn.to_string(),
|
||||
child_accounting,
|
||||
scheduler,
|
||||
runtime_owner,
|
||||
_runtime_lease: runtime_lease,
|
||||
};
|
||||
@@ -685,40 +677,121 @@ impl Engine {
|
||||
}
|
||||
}
|
||||
|
||||
/// One actual producer dispatch, reserved before entering the client. Its Drop
|
||||
/// records unknown execution synchronously; no future is needed to retain it.
|
||||
/// One actual producer dispatch with a durable source reservation before the client.
|
||||
pub(super) struct RlmDispatchedRequest {
|
||||
caller: Arc<CapturedRlmCaller>,
|
||||
usage: RlmUsageAccumulator,
|
||||
reservation: RlmUsageReservation,
|
||||
source: String,
|
||||
accounting_source: crate::cost_status::NativeUsageSource,
|
||||
route: crate::cost_status::EffectiveRouteEnvelope,
|
||||
settled: Option<(
|
||||
Usage,
|
||||
Option<u64>,
|
||||
crate::cost_status::RuntimeUsageMissingReason,
|
||||
)>,
|
||||
settled: Option<crate::cost_status::RuntimeUsageRecord>,
|
||||
projected: bool,
|
||||
refused: bool,
|
||||
}
|
||||
impl RlmDispatchedRequest {
|
||||
pub(super) async fn settle_open_error(&mut self, error: &anyhow::Error) {
|
||||
if crate::tools::subagent::engine::provider_request_refusal_proven(error) {
|
||||
self.cancel_unexecuted();
|
||||
} else {
|
||||
self.settle(&Usage::default(), false).await;
|
||||
}
|
||||
pub(crate) fn settlement_acknowledged(&self) -> bool {
|
||||
self.refused || self.projected
|
||||
}
|
||||
pub(super) fn cancel_unexecuted(&mut self) {
|
||||
self.usage.cancel_sync(
|
||||
self.reservation,
|
||||
false,
|
||||
pub(crate) async fn bind_dispatch(
|
||||
&mut self,
|
||||
observation: &crate::llm_client::RequestRetryObservation,
|
||||
) -> std::io::Result<()> {
|
||||
#[cfg(not(any(test, feature = "test-support")))]
|
||||
if !self.accounting_source.has_captured_origin() {
|
||||
return Err(std::io::Error::new(
|
||||
std::io::ErrorKind::PermissionDenied,
|
||||
"RLM provider call has no captured accounting origin",
|
||||
));
|
||||
}
|
||||
self.accounting_source
|
||||
.bind_dispatch(&self.route, observation)?;
|
||||
if let Some(record) = self.accounting_source.admit_dispatch(&self.route).await? {
|
||||
self.usage.complete_record(self.reservation, &record).await;
|
||||
if let Some(child) = &self.caller.child_accounting {
|
||||
child.project_settled(&record).await?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
pub(super) async fn settle_open_error(&mut self, error: &anyhow::Error) {
|
||||
let wire = self.accounting_source.observed_wire();
|
||||
let coverage = if wire.last_transmission.is_some() {
|
||||
wire.usage_coverage(false, false)
|
||||
} else {
|
||||
match error
|
||||
.downcast_ref::<crate::llm_client::LlmError>()
|
||||
.map(crate::llm_client::LlmError::transmission)
|
||||
{
|
||||
Some(crate::llm_client::RequestTransmission::NotDispatched) => {
|
||||
crate::llm_client::ProviderUsageCoverage::NotDispatched
|
||||
}
|
||||
Some(crate::llm_client::RequestTransmission::Rejected) => {
|
||||
crate::llm_client::ProviderUsageCoverage::Rejected
|
||||
}
|
||||
_ => crate::llm_client::ProviderUsageCoverage::Missing,
|
||||
}
|
||||
};
|
||||
let record = match self.accounting_source.prepare_observed(
|
||||
&self.route,
|
||||
&Usage::default(),
|
||||
coverage,
|
||||
crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown,
|
||||
);
|
||||
self.refused = true;
|
||||
&wire,
|
||||
) {
|
||||
Ok(record) => record,
|
||||
Err(error) => {
|
||||
tracing::warn!(%error, "RLM refusal receipt was not admitted");
|
||||
return;
|
||||
}
|
||||
};
|
||||
let published = self.caller.publish_response_async(&record).await;
|
||||
self.usage.complete_record(self.reservation, &record).await;
|
||||
let projected = if let Some(child) = &self.caller.child_accounting {
|
||||
child.project_settled(&record).await
|
||||
} else {
|
||||
Ok(())
|
||||
};
|
||||
self.projected = published.is_ok() && projected.is_ok();
|
||||
self.settled = Some(record);
|
||||
self.refused = self.projected
|
||||
&& matches!(
|
||||
coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Rejected
|
||||
| crate::llm_client::ProviderUsageCoverage::NotDispatched
|
||||
);
|
||||
}
|
||||
pub(super) async fn cancel_unexecuted(&mut self) {
|
||||
let wire = self.accounting_source.observed_wire();
|
||||
if wire.last_transmission.is_some() {
|
||||
self.settle(&Usage::default(), false).await;
|
||||
return;
|
||||
}
|
||||
let record = match self.accounting_source.prepare_observed(
|
||||
&self.route,
|
||||
&Usage::default(),
|
||||
crate::llm_client::ProviderUsageCoverage::NotDispatched,
|
||||
crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown,
|
||||
&wire,
|
||||
) {
|
||||
Ok(record) => record,
|
||||
Err(error) => {
|
||||
tracing::warn!(%error, "uninvoked RLM reservation remains unsettled");
|
||||
return;
|
||||
}
|
||||
};
|
||||
let published = self.caller.publish_response_async(&record).await;
|
||||
self.usage.complete_record(self.reservation, &record).await;
|
||||
let projected = if let Some(child) = &self.caller.child_accounting {
|
||||
child.project_settled(&record).await
|
||||
} else {
|
||||
Ok(())
|
||||
};
|
||||
self.projected = published.is_ok() && projected.is_ok();
|
||||
self.refused = self.projected;
|
||||
self.settled = Some(record);
|
||||
}
|
||||
pub(super) async fn settle(&mut self, usage: &Usage, complete: bool) {
|
||||
if self.settled.is_some() || self.refused {
|
||||
if self.refused {
|
||||
return;
|
||||
}
|
||||
let reason = if complete {
|
||||
@@ -726,102 +799,82 @@ impl RlmDispatchedRequest {
|
||||
} else {
|
||||
crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown
|
||||
};
|
||||
// Publication precedes any cancellable worker projection. Drop retains
|
||||
// this exact settlement and never synthesizes a second charge.
|
||||
let priced = self
|
||||
.caller
|
||||
.publish_response(&self.source, &self.route, usage, reason);
|
||||
self.settled = Some((usage.clone(), priced, reason));
|
||||
if usage_has_reported_data(usage) {
|
||||
self.usage.complete(self.reservation, usage).await;
|
||||
let coverage = if complete && usage_has_reported_data(usage) {
|
||||
crate::llm_client::ProviderUsageCoverage::Reported
|
||||
} else if usage_has_reported_data(usage) {
|
||||
crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
} else {
|
||||
self.usage.cancel_sync(self.reservation, true, reason);
|
||||
}
|
||||
if let Some(child) = &self.caller.child_accounting {
|
||||
child
|
||||
.project_settled(&self.source, &self.route, usage, priced, reason)
|
||||
.await;
|
||||
}
|
||||
self.projected = true;
|
||||
}
|
||||
}
|
||||
impl Drop for RlmDispatchedRequest {
|
||||
fn drop(&mut self) {
|
||||
if self.refused || self.projected {
|
||||
return;
|
||||
}
|
||||
let (usage, priced, reason) = self.settled.clone().unwrap_or_else(|| {
|
||||
let reason = crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown;
|
||||
let usage = Usage::default();
|
||||
let priced = self
|
||||
.caller
|
||||
.publish_response(&self.source, &self.route, &usage, reason);
|
||||
self.usage.cancel_sync(self.reservation, true, reason);
|
||||
(usage, priced, reason)
|
||||
});
|
||||
if let Some(child) = &self.caller.child_accounting {
|
||||
child.recover_settled(
|
||||
&self.caller.scheduler,
|
||||
self.source.clone(),
|
||||
self.route.clone(),
|
||||
usage,
|
||||
priced,
|
||||
reason,
|
||||
);
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
};
|
||||
let record = match self.accounting_source.prepare_observed(
|
||||
&self.route,
|
||||
usage,
|
||||
coverage,
|
||||
reason,
|
||||
&self.accounting_source.observed_wire(),
|
||||
) {
|
||||
Ok(record) => record,
|
||||
Err(error) => {
|
||||
tracing::error!(%error, "RLM usage predecessor remains unsettled");
|
||||
return;
|
||||
}
|
||||
};
|
||||
if let Err(error) = self.caller.publish_response_async(&record).await {
|
||||
tracing::warn!(%error, "RLM usage has no durable acknowledgement");
|
||||
}
|
||||
self.settled = Some(record.clone());
|
||||
self.usage.complete_record(self.reservation, &record).await;
|
||||
let projected = if let Some(child) = &self.caller.child_accounting {
|
||||
child.project_settled(&record).await
|
||||
} else {
|
||||
Ok(())
|
||||
};
|
||||
self.projected = record.is_acknowledged() && projected.is_ok();
|
||||
}
|
||||
}
|
||||
|
||||
impl CapturedRlmCaller {
|
||||
async fn publish_response_async(
|
||||
self: &Arc<Self>,
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
) -> std::io::Result<Option<u64>> {
|
||||
let caller = Arc::clone(self);
|
||||
let record = record.clone();
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
let origin = crate::cost_status::test_cost_scope_id();
|
||||
codewhale_runtime::runtime_control::owner_work(move || {
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
let _origin = crate::cost_status::bind_test_cost_scope(origin);
|
||||
caller
|
||||
.publish_response(&record)
|
||||
.map_err(anyhow::Error::from)
|
||||
})
|
||||
.await
|
||||
.map_err(std::io::Error::other)
|
||||
}
|
||||
|
||||
fn publish_response(
|
||||
&self,
|
||||
source: &str,
|
||||
route: &crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &Usage,
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason,
|
||||
) -> Option<u64> {
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
) -> std::io::Result<Option<u64>> {
|
||||
if let Some(child) = &self.child_accounting {
|
||||
return child.publish(source, route, usage, reason);
|
||||
return child.publish(record);
|
||||
}
|
||||
if usage_has_reported_data(usage) {
|
||||
if let Some(owner) = self.runtime_owner.as_deref() {
|
||||
crate::cost_status::report_effective_route_for_runtime(
|
||||
self.origin_scope,
|
||||
Some(owner),
|
||||
source,
|
||||
route,
|
||||
usage,
|
||||
);
|
||||
} else {
|
||||
crate::usage_persistence::report_effective_route_for_interactive_origin(
|
||||
self.origin_scope,
|
||||
&self.origin_session,
|
||||
&self.origin_turn,
|
||||
source,
|
||||
route,
|
||||
usage,
|
||||
);
|
||||
}
|
||||
} else if let Some(owner) = self.runtime_owner.as_deref() {
|
||||
crate::cost_status::report_missing_runtime_usage(
|
||||
self.origin_scope,
|
||||
Some(owner),
|
||||
source,
|
||||
route,
|
||||
reason,
|
||||
);
|
||||
} else {
|
||||
crate::usage_persistence::report_missing_usage_for_interactive_origin(
|
||||
self.origin_scope,
|
||||
&self.origin_session,
|
||||
&self.origin_turn,
|
||||
source,
|
||||
route,
|
||||
reason,
|
||||
);
|
||||
}
|
||||
None
|
||||
crate::cost_status::report_runtime_usage_record(
|
||||
self.origin_scope,
|
||||
self.runtime_owner.as_deref(),
|
||||
record,
|
||||
)?;
|
||||
Ok(usage_has_reported_data(&record.usage.usage)
|
||||
.then(|| {
|
||||
crate::tools::subagent::priced_usd_microusd(
|
||||
&record.usage.route.audit_frozen(&record.usage.usage),
|
||||
)
|
||||
})
|
||||
.flatten())
|
||||
}
|
||||
}
|
||||
|
||||
impl Engine {
|
||||
pub(super) async fn rlm_provider_request(
|
||||
&mut self,
|
||||
@@ -847,7 +900,11 @@ impl Engine {
|
||||
caller: Arc::clone(&state.caller),
|
||||
usage: state.usage.clone(),
|
||||
reservation,
|
||||
source,
|
||||
accounting_source: crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
source,
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
state.caller.context.accounting_origin.clone(),
|
||||
),
|
||||
route,
|
||||
settled: None,
|
||||
projected: false,
|
||||
|
||||
@@ -26,9 +26,9 @@ async fn event_capacity_cancel_before_admission_has_no_started_turn_and_keeps_cl
|
||||
};
|
||||
spec.initial_routed_usage
|
||||
.records
|
||||
.push(crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: "event-capacity:classifier".into(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
.push(crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
"event-capacity:classifier",
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
None,
|
||||
ProviderKind::Openai,
|
||||
@@ -43,7 +43,7 @@ async fn event_capacity_cancel_before_admission_has_no_started_turn_and_keeps_cl
|
||||
..Usage::default()
|
||||
},
|
||||
},
|
||||
});
|
||||
));
|
||||
let controls = Arc::clone(&engine.turn_controls);
|
||||
handle.send(op).await.unwrap();
|
||||
let task = tokio::spawn(engine.run());
|
||||
@@ -214,9 +214,9 @@ async fn event_capacity_invalid_images_release_queued_control_and_keep_classifie
|
||||
});
|
||||
spec.initial_routed_usage
|
||||
.records
|
||||
.push(crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: "event-capacity:invalid-image-classifier".into(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
.push(crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
"event-capacity:invalid-image-classifier",
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
None,
|
||||
ProviderKind::Openai,
|
||||
@@ -231,7 +231,7 @@ async fn event_capacity_invalid_images_release_queued_control_and_keep_classifie
|
||||
..Usage::default()
|
||||
},
|
||||
},
|
||||
});
|
||||
));
|
||||
handle.send(op).await.unwrap();
|
||||
let task = tokio::spawn(engine.run());
|
||||
if full {
|
||||
|
||||
@@ -66,7 +66,9 @@ async fn queued_goal_clear_refreshes_prompt_and_cancels_stale_continuation() {
|
||||
}
|
||||
Event::SessionUpdated {
|
||||
source: _,
|
||||
system_prompt, .. } => {
|
||||
system_prompt,
|
||||
..
|
||||
} => {
|
||||
let prompt = match system_prompt.expect("clear SessionUpdated prompt") {
|
||||
SystemPrompt::Text(text) => text,
|
||||
SystemPrompt::Blocks(blocks) => blocks
|
||||
@@ -1173,6 +1175,8 @@ async fn headless_host_drains_existing_engine_completion_inbox_before_exit() {
|
||||
engine
|
||||
.tx_event
|
||||
.send(Event::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
|
||||
@@ -426,8 +426,7 @@ async fn idle_subagent_delivery_releases_claim_when_route_fails_before_recording
|
||||
use crate::tools::subagent::SubAgentCompletion;
|
||||
|
||||
let workspace = tempdir().expect("tempdir");
|
||||
let api_config = Config::default()
|
||||
.with_legacy_root(
|
||||
let api_config = Config::default().with_legacy_root(
|
||||
Some("test-key".to_string()),
|
||||
Some("http://127.0.0.1:1/v1".to_string()),
|
||||
);
|
||||
@@ -508,6 +507,7 @@ fn subagent_mailbox_keeps_lifecycle_events_reliable() {
|
||||
));
|
||||
assert!(!subagent_mailbox_message_is_best_effort(
|
||||
&MailboxMessage::TokenUsage {
|
||||
record: None,
|
||||
agent_id: "agent_a".to_string(),
|
||||
source_id: "response-a".to_string(),
|
||||
route: Box::new(crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
@@ -603,6 +603,7 @@ fn subagent_mailbox_never_samples_lifecycle_or_usage_events() {
|
||||
assert!(subagent_mailbox_best_effort_send_permitted(
|
||||
&mut last_sent_at,
|
||||
&MailboxMessage::TokenUsage {
|
||||
record: None,
|
||||
agent_id: "agent_a".to_string(),
|
||||
source_id: "response-a".to_string(),
|
||||
route: Box::new(crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
|
||||
@@ -19,8 +19,8 @@ async fn repl_fence_obeys_auto_review_block_under_full_access() {
|
||||
let client: crate::core::model_client::SharedModelClient = mock.clone();
|
||||
let mut config = deterministic_engine_config(workspace.path());
|
||||
// Built through the config entry point, as production does.
|
||||
config.auto_review_policy =
|
||||
crate::core::authority::auto_review::AutoReviewPolicy::from_config(&{
|
||||
config.auto_review_policy = crate::core::authority::auto_review::AutoReviewPolicy::from_config(
|
||||
&{
|
||||
let __cw_config_init_0_0 = Some(codewhale_runtime::config::AutoReviewConfig {
|
||||
block: vec![{
|
||||
let __cw_config_init_1_0 = Some("no-code".to_string());
|
||||
@@ -38,7 +38,8 @@ async fn repl_fence_obeys_auto_review_block_under_full_access() {
|
||||
let mut __cw_config_init_0_value: Config = Config::default();
|
||||
__cw_config_init_0_value.auto_review = __cw_config_init_0_0;
|
||||
__cw_config_init_0_value
|
||||
});
|
||||
},
|
||||
);
|
||||
let (mut engine, handle) = Engine::new_with_model_client(config, &Config::default(), client);
|
||||
engine.session.auto_approve = true;
|
||||
engine.session.add_message(Message {
|
||||
@@ -990,9 +991,9 @@ async fn initial_routed_usage_is_total_only_emitted_once_and_keeps_parent_route_
|
||||
missing_usage_route.billing_mode = crate::cost_status::RouteBillingMode::Metered;
|
||||
**initial_routed_usage = crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: vec![crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: "auto-router:engine-fixture".to_string(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
records: vec![crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
"auto-router:engine-fixture".to_string(),
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
None,
|
||||
ProviderKind::Openai,
|
||||
@@ -1003,7 +1004,7 @@ async fn initial_routed_usage_is_total_only_emitted_once_and_keeps_parent_route_
|
||||
),
|
||||
usage: classifier_usage.clone(),
|
||||
},
|
||||
}],
|
||||
)],
|
||||
drop_records: vec![crate::cost_status::RuntimeUsageDropRecord {
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason::default(),
|
||||
source_id: "auto-router:engine-fixture:missing-usage".to_string(),
|
||||
|
||||
@@ -72,8 +72,12 @@ struct PreparedModelStep {
|
||||
|
||||
pub(super) struct PendingProviderRequestReceipt {
|
||||
pub(super) request_id: String,
|
||||
pub(super) accounting_source: crate::cost_status::NativeUsageSource,
|
||||
pub(super) accounting_scope: crate::cost_status::CostScopeToken,
|
||||
pub(super) accounting_owner: Option<String>,
|
||||
pub(super) route: crate::cost_status::EffectiveRouteEnvelope,
|
||||
pub(super) primary: bool,
|
||||
pub(super) accounting_owned_elsewhere: bool,
|
||||
pub(super) invoked: bool,
|
||||
pub(super) turn_id: String,
|
||||
pub(super) request_index: u32,
|
||||
@@ -159,6 +163,7 @@ struct StreamOutcome {
|
||||
tool_uses: Vec<ToolUseState>,
|
||||
usage: Usage,
|
||||
usage_reported: bool,
|
||||
accounting: Option<crate::cost_status::RuntimeUsageRecord>,
|
||||
stop_reason: Option<String>,
|
||||
pending_message_complete: bool,
|
||||
last_text_index: Option<usize>,
|
||||
@@ -1378,25 +1383,85 @@ impl Engine {
|
||||
child.accounting_origin().0
|
||||
});
|
||||
let review_route = client.effective_route_envelope(client.model(), chrono::Utc::now());
|
||||
let started = Instant::now();
|
||||
let review =
|
||||
super::reviewer::consult_reviewer(client, &context_text, &self.cancel_token).await;
|
||||
if let Some(usage) = &review.usage {
|
||||
turn.add_usage(usage);
|
||||
let source = format!("auto-review:{}:{tool_id}", turn.id);
|
||||
let origin = if let Some(child) = child_accounting.as_ref() {
|
||||
child.runtime.context.accounting_origin.clone()
|
||||
} else {
|
||||
self.config
|
||||
.compaction
|
||||
.runtime_cost_owner
|
||||
.as_deref()
|
||||
.map(|owner| crate::cost_status::NativeUsageOrigin::runtime(cost_scope, owner))
|
||||
.transpose()
|
||||
.map_err(|error| {
|
||||
ToolError::execution_failed(format!(
|
||||
"review accounting admission failed: {error}"
|
||||
))
|
||||
})?
|
||||
.or_else(|| self.config.runtime_services.accounting_origin.clone())
|
||||
};
|
||||
let source = crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
format!("auto-review:{}:{tool_id}", turn.id),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
origin,
|
||||
);
|
||||
let observation = source
|
||||
.observe_route(&review_route)
|
||||
.map_err(|error| ToolError::execution_failed(error.to_string()))?;
|
||||
if let Some(record) = source
|
||||
.admit_dispatch(&review_route)
|
||||
.await
|
||||
.map_err(|error| ToolError::execution_failed(error.to_string()))?
|
||||
{
|
||||
if let Some(child) = child_accounting.as_ref() {
|
||||
child
|
||||
.settle_response(&source, review_route.clone(), usage)
|
||||
.await;
|
||||
} else {
|
||||
crate::cost_status::report_effective_route_for_runtime(
|
||||
cost_scope,
|
||||
self.config.compaction.runtime_cost_owner.as_deref(),
|
||||
&format!("auto-review:{}:{tool_id}", turn.id),
|
||||
&review_route,
|
||||
usage,
|
||||
);
|
||||
.project_settled_response(&record)
|
||||
.await
|
||||
.map_err(|error| ToolError::execution_failed(error.to_string()))?;
|
||||
}
|
||||
}
|
||||
let wire = observation.wire_observations.clone();
|
||||
let started = Instant::now();
|
||||
let review = crate::llm_client::observe_request_retries(
|
||||
Some(observation),
|
||||
super::reviewer::consult_reviewer(client, &context_text, &self.cancel_token),
|
||||
)
|
||||
.await;
|
||||
let wire = wire.drain();
|
||||
let empty = Usage::default();
|
||||
let usage = review.usage.as_ref().unwrap_or(&empty);
|
||||
let has_response = review.usage.is_some();
|
||||
let coverage = wire.usage_coverage(usage_has_reported_data(usage), has_response);
|
||||
let record = source
|
||||
.prepare_observed(
|
||||
&review_route,
|
||||
usage,
|
||||
coverage,
|
||||
if has_response {
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage
|
||||
} else {
|
||||
crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown
|
||||
},
|
||||
&wire,
|
||||
)
|
||||
.map_err(|error| ToolError::execution_failed(error.to_string()))?;
|
||||
if let Some(child) = child_accounting.as_ref() {
|
||||
child
|
||||
.settle_response(&record)
|
||||
.await
|
||||
.map_err(|error| ToolError::execution_failed(error.to_string()))?;
|
||||
} else {
|
||||
crate::cost_status::settle_runtime_usage_record(
|
||||
cost_scope,
|
||||
self.config.compaction.runtime_cost_owner.as_deref(),
|
||||
&record,
|
||||
)
|
||||
.await
|
||||
.map_err(|error| {
|
||||
ToolError::execution_failed(format!("review usage remains unacknowledged: {error}"))
|
||||
})?;
|
||||
}
|
||||
if let Some(usage) = &review.usage {
|
||||
turn.add_usage(usage);
|
||||
if usage_has_reported_data(usage) {
|
||||
let request_ms = u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX);
|
||||
let _ = self
|
||||
@@ -1408,12 +1473,6 @@ impl Engine {
|
||||
})
|
||||
.await;
|
||||
}
|
||||
} else if matches!(
|
||||
&review.outcome,
|
||||
super::reviewer::ReviewerOutcome::Unavailable { reason }
|
||||
if reason == "the reviewer timed out" || reason == "the reviewer request failed"
|
||||
) {
|
||||
turn.add_routed_usage_dropped_records(1);
|
||||
}
|
||||
match &review.outcome {
|
||||
super::reviewer::ReviewerOutcome::Allow { .. } => {
|
||||
@@ -4669,12 +4728,12 @@ impl Engine {
|
||||
pending: &mut Option<PendingProviderRequestReceipt>,
|
||||
usage: Option<&Usage>,
|
||||
outcome: &str,
|
||||
) {
|
||||
) -> Result<Option<crate::cost_status::RuntimeUsageRecord>, String> {
|
||||
let Some(pending) = pending.take() else {
|
||||
return;
|
||||
return Ok(None);
|
||||
};
|
||||
let usage = usage.filter(|usage| usage_has_reported_data(usage));
|
||||
let wire = pending.wire.drain();
|
||||
let wire = pending.wire.snapshot();
|
||||
let source_id = wire
|
||||
.last_transmission
|
||||
.as_ref()
|
||||
@@ -4689,25 +4748,76 @@ impl Engine {
|
||||
},
|
||||
|attempt| attempt.transmission,
|
||||
);
|
||||
let coverage = match transmission {
|
||||
crate::llm_client::RequestTransmission::NotDispatched => {
|
||||
crate::llm_client::ProviderUsageCoverage::NotDispatched
|
||||
}
|
||||
crate::llm_client::RequestTransmission::Rejected => {
|
||||
crate::llm_client::ProviderUsageCoverage::Rejected
|
||||
}
|
||||
crate::llm_client::RequestTransmission::OutcomeUnknown => {
|
||||
if usage.is_some() && matches!(outcome, "completed" | "review_response") {
|
||||
crate::llm_client::ProviderUsageCoverage::Reported
|
||||
} else if usage.is_some() {
|
||||
crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
} else {
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
let coverage = if wire.last_transmission.is_some() {
|
||||
wire.usage_coverage(
|
||||
usage.is_some(),
|
||||
matches!(outcome, "completed" | "review_response"),
|
||||
)
|
||||
} else {
|
||||
match transmission {
|
||||
crate::llm_client::RequestTransmission::NotDispatched => {
|
||||
crate::llm_client::ProviderUsageCoverage::NotDispatched
|
||||
}
|
||||
crate::llm_client::RequestTransmission::Rejected => {
|
||||
crate::llm_client::ProviderUsageCoverage::Rejected
|
||||
}
|
||||
crate::llm_client::RequestTransmission::OutcomeUnknown
|
||||
if usage.is_some() && matches!(outcome, "completed" | "review_response") =>
|
||||
{
|
||||
crate::llm_client::ProviderUsageCoverage::Reported
|
||||
}
|
||||
crate::llm_client::RequestTransmission::OutcomeUnknown if usage.is_some() => {
|
||||
crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
}
|
||||
_ => crate::llm_client::ProviderUsageCoverage::Missing,
|
||||
}
|
||||
};
|
||||
let mut accounting_record = None;
|
||||
let accounting_error = if pending.primary && !pending.accounting_owned_elsewhere {
|
||||
let reason = if matches!(outcome, "completed" | "review_response") {
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage
|
||||
} else {
|
||||
crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown
|
||||
};
|
||||
let prepared = pending.accounting_source.prepare_observed(
|
||||
&pending.route,
|
||||
usage.unwrap_or(&Usage::default()),
|
||||
coverage,
|
||||
reason,
|
||||
&wire,
|
||||
);
|
||||
match prepared {
|
||||
Ok(record) => {
|
||||
accounting_record = Some(record.clone());
|
||||
match crate::cost_status::settle_runtime_usage_record(
|
||||
pending.accounting_scope,
|
||||
pending.accounting_owner.as_deref(),
|
||||
&record,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(()) => None,
|
||||
Err(error) => Some(error.to_string()),
|
||||
}
|
||||
}
|
||||
Err(error) => Some(error.to_string()),
|
||||
}
|
||||
} else {
|
||||
None
|
||||
};
|
||||
if let Some(error) = accounting_error.as_ref() {
|
||||
tracing::error!(%error, "provider usage has no durable acknowledgement");
|
||||
let _ = self
|
||||
.send_event(Event::status(format!(
|
||||
"Provider usage settlement is uncertain: {error}"
|
||||
)))
|
||||
.await;
|
||||
}
|
||||
let failed = accounting_error.clone();
|
||||
let _ = self.send_event(Event::ProviderRequestReceipt {
|
||||
receipt: json!({
|
||||
"source_accounting": pending.primary,
|
||||
"accounting_error": accounting_error,
|
||||
"session_id": self.session.id,
|
||||
"turn_id": pending.turn_id,
|
||||
"request_index": pending.request_index,
|
||||
@@ -4731,6 +4841,10 @@ impl Engine {
|
||||
"cache_attribution": "Local prefix changes and wire schema measurements are context for reported usage, not proof that eviction caused a miss or that unreported retries were free.",
|
||||
}),
|
||||
}).await;
|
||||
match failed {
|
||||
Some(error) => Err(error),
|
||||
None => Ok(accounting_record),
|
||||
}
|
||||
}
|
||||
|
||||
async fn process_stream(
|
||||
@@ -5315,12 +5429,20 @@ impl Engine {
|
||||
} else {
|
||||
"completion_unknown"
|
||||
};
|
||||
self.settle_provider_request_receipt(
|
||||
&mut request_receipt,
|
||||
attempt_usage_reported.then_some(&usage),
|
||||
receipt_outcome,
|
||||
)
|
||||
.await;
|
||||
let accounting = match self
|
||||
.settle_provider_request_receipt(
|
||||
&mut request_receipt,
|
||||
attempt_usage_reported.then_some(&usage),
|
||||
receipt_outcome,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(record) => record,
|
||||
Err(error) => {
|
||||
stream_error.get_or_insert(format!("Provider usage settlement failed: {error}"));
|
||||
None
|
||||
}
|
||||
};
|
||||
StreamOutcome {
|
||||
current_text_raw,
|
||||
current_text_visible,
|
||||
@@ -5331,6 +5453,7 @@ impl Engine {
|
||||
tool_uses,
|
||||
usage,
|
||||
usage_reported,
|
||||
accounting,
|
||||
stop_reason,
|
||||
pending_message_complete,
|
||||
last_text_index,
|
||||
|
||||
@@ -106,10 +106,54 @@ impl Engine {
|
||||
.map(|_| turn.steps_used().saturating_add(1));
|
||||
let retry_observation = self.request_retry_observation();
|
||||
let transport_retries = retry_observation.retries.clone();
|
||||
let request_route =
|
||||
client.effective_route_envelope(&stream_request.model, chrono::Utc::now());
|
||||
let accounting_origin = match self
|
||||
.config
|
||||
.compaction
|
||||
.runtime_cost_owner
|
||||
.as_deref()
|
||||
.map(|owner| {
|
||||
crate::cost_status::NativeUsageOrigin::runtime(
|
||||
crate::cost_status::scope_token(),
|
||||
owner,
|
||||
)
|
||||
})
|
||||
.transpose()
|
||||
{
|
||||
Ok(origin) => origin.or_else(|| self.config.runtime_services.accounting_origin.clone()),
|
||||
Err(error) => {
|
||||
return PhaseResult::Return((
|
||||
TurnOutcomeStatus::Failed,
|
||||
Some(format!("provider accounting admission failed: {error}")),
|
||||
));
|
||||
}
|
||||
};
|
||||
let accounting_source = crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
retry_observation.request_id.clone(),
|
||||
crate::cost_status::UsageSpendOwner::Primary,
|
||||
accounting_origin.filter(|_| self.child_host.is_none() && self.rlm_host.is_none()),
|
||||
);
|
||||
if let Err(error) = accounting_source.bind_dispatch(&request_route, &retry_observation) {
|
||||
return PhaseResult::Return((TurnOutcomeStatus::Failed, Some(error.to_string())));
|
||||
}
|
||||
if self.child_host.is_none() && self.rlm_host.is_none() {
|
||||
if let Err(error) = accounting_source.admit_dispatch(&request_route).await {
|
||||
return PhaseResult::Return((
|
||||
TurnOutcomeStatus::Failed,
|
||||
Some(format!("provider source admission failed: {error}")),
|
||||
));
|
||||
}
|
||||
}
|
||||
let dispatch_observation = retry_observation.clone();
|
||||
let mut request_receipt = Some(PendingProviderRequestReceipt {
|
||||
request_id: retry_observation.request_id.clone(),
|
||||
route: client.effective_route_envelope(&stream_request.model, chrono::Utc::now()),
|
||||
accounting_source,
|
||||
accounting_scope: crate::cost_status::scope_token(),
|
||||
accounting_owner: self.config.compaction.runtime_cost_owner.clone(),
|
||||
route: request_route,
|
||||
primary: self.child_host.is_none() && self.rlm_host.is_none(),
|
||||
accounting_owned_elsewhere: false,
|
||||
invoked: false,
|
||||
turn_id: turn.id.clone(),
|
||||
request_index: turn
|
||||
@@ -125,7 +169,7 @@ impl Engine {
|
||||
let rlm_request = tokio::select! {
|
||||
biased;
|
||||
() = request_cancel.cancelled() => {
|
||||
self.settle_provider_request_receipt(&mut request_receipt, None, "cancelled_before_stream").await;
|
||||
let _ = self.settle_provider_request_receipt(&mut request_receipt, None, "cancelled_before_stream").await;
|
||||
let _ = self.send_event(Event::status("Request cancelled")).await;
|
||||
return PhaseResult::Return((TurnOutcomeStatus::Interrupted, None));
|
||||
}
|
||||
@@ -137,6 +181,14 @@ impl Engine {
|
||||
Err(error) => Some(Err(error)),
|
||||
Ok(request) => {
|
||||
rlm_dispatch = request;
|
||||
if let Some(dispatch) = rlm_dispatch.as_mut() {
|
||||
if let Err(error) = dispatch.bind_dispatch(&dispatch_observation).await {
|
||||
return PhaseResult::Return((
|
||||
TurnOutcomeStatus::Failed,
|
||||
Some(error.to_string()),
|
||||
));
|
||||
}
|
||||
}
|
||||
let opening =
|
||||
crate::llm_client::observe_request_retries(Some(retry_observation), async {
|
||||
request_started = true;
|
||||
@@ -168,6 +220,9 @@ impl Engine {
|
||||
source,
|
||||
route,
|
||||
));
|
||||
if let Some(dispatch) = child_dispatch.as_mut() {
|
||||
dispatch.bind_dispatch(&dispatch_observation).await?;
|
||||
}
|
||||
}
|
||||
if let Some(job) = child_job.as_ref() {
|
||||
tokio::time::timeout(
|
||||
@@ -201,21 +256,22 @@ impl Engine {
|
||||
.transport_retries
|
||||
.saturating_add(transport_retries.load(std::sync::atomic::Ordering::Relaxed));
|
||||
let Some(stream_result) = stream_result else {
|
||||
self.settle_provider_request_receipt(
|
||||
&mut request_receipt,
|
||||
None,
|
||||
if opening_steer.is_some() {
|
||||
"steered_before_stream"
|
||||
} else {
|
||||
"cancelled_before_stream"
|
||||
},
|
||||
)
|
||||
.await;
|
||||
let _ = self
|
||||
.settle_provider_request_receipt(
|
||||
&mut request_receipt,
|
||||
None,
|
||||
if opening_steer.is_some() {
|
||||
"steered_before_stream"
|
||||
} else {
|
||||
"cancelled_before_stream"
|
||||
},
|
||||
)
|
||||
.await;
|
||||
if let Some(dispatch) = rlm_dispatch.as_mut() {
|
||||
if request_started {
|
||||
dispatch.settle(&Usage::default(), false).await;
|
||||
} else {
|
||||
dispatch.cancel_unexecuted();
|
||||
dispatch.cancel_unexecuted().await;
|
||||
if let Some(state) = self.rlm_host.as_mut() {
|
||||
state.model_rounds = state.model_rounds.saturating_sub(1);
|
||||
}
|
||||
@@ -249,7 +305,8 @@ impl Engine {
|
||||
s
|
||||
}
|
||||
Err(e) => {
|
||||
self.settle_provider_request_receipt(&mut request_receipt, None, "open_failed")
|
||||
let _ = self
|
||||
.settle_provider_request_receipt(&mut request_receipt, None, "open_failed")
|
||||
.await;
|
||||
if let Some(dispatch) = rlm_dispatch.as_mut() {
|
||||
dispatch.settle_open_error(&e).await;
|
||||
@@ -469,6 +526,7 @@ impl Engine {
|
||||
mut tool_uses,
|
||||
usage,
|
||||
usage_reported,
|
||||
accounting,
|
||||
stop_reason,
|
||||
pending_message_complete,
|
||||
last_text_index,
|
||||
@@ -478,7 +536,7 @@ impl Engine {
|
||||
stream_start,
|
||||
first_token_at,
|
||||
request_dispatched_at,
|
||||
stream_error,
|
||||
mut stream_error,
|
||||
} = self
|
||||
.process_stream(
|
||||
stream,
|
||||
@@ -503,7 +561,13 @@ impl Engine {
|
||||
// C02-05: a response whose stream failed or was steered — a provider error
|
||||
// frame, a transport error, a stall, or a cap — is not a complete
|
||||
// response. Nothing it collected may execute or continue the turn.
|
||||
let response_stream_failed = steer_preempted || stream_error.is_some();
|
||||
if rlm_dispatch
|
||||
.as_ref()
|
||||
.is_some_and(|dispatch| !dispatch.settlement_acknowledged())
|
||||
{
|
||||
stream_error.get_or_insert_with(|| "RLM usage settlement failed".to_string());
|
||||
}
|
||||
let mut response_stream_failed = steer_preempted || stream_error.is_some();
|
||||
progress.turn_error = progress.turn_error.take().or(stream_error.clone());
|
||||
turn.stop_diagnostics
|
||||
.observe_provider_response(stop_reason.as_deref(), tool_uses.len());
|
||||
@@ -565,6 +629,14 @@ impl Engine {
|
||||
)
|
||||
.await;
|
||||
}
|
||||
if child_dispatch
|
||||
.as_ref()
|
||||
.is_some_and(|dispatch| !dispatch.settlement_acknowledged())
|
||||
{
|
||||
stream_error.get_or_insert_with(|| "Child usage settlement failed".to_string());
|
||||
response_stream_failed = true;
|
||||
progress.turn_error = progress.turn_error.take().or(stream_error.clone());
|
||||
}
|
||||
drop(child_dispatch.take());
|
||||
if let Some(job) = child_job.as_ref() {
|
||||
job.response_settled(!response_stream_failed);
|
||||
@@ -582,6 +654,7 @@ impl Engine {
|
||||
if usage_reported {
|
||||
let _ = self
|
||||
.send_event(Event::TurnUsage {
|
||||
accounting,
|
||||
max_output_tokens: turn.max_output_tokens.map(|_| stream_request.max_tokens),
|
||||
usage: usage.clone(),
|
||||
duration_ms: u64::try_from(stream_start.elapsed().as_millis())
|
||||
|
||||
@@ -668,6 +668,7 @@ impl Engine {
|
||||
&self.api_config,
|
||||
&stream_request.messages,
|
||||
self.config.compaction.runtime_cost_owner.as_deref(),
|
||||
self.config.runtime_services.accounting_origin.clone(),
|
||||
&self.cancel_token,
|
||||
));
|
||||
}
|
||||
|
||||
@@ -558,6 +558,7 @@ pub enum Event {
|
||||
/// price this under the parent quote and price routed children from
|
||||
/// their own receipts, avoiding double billing without subtraction.
|
||||
parent_route_usage: Usage,
|
||||
source_accounting: bool,
|
||||
/// Provider calls whose execution/usage could not be receipted.
|
||||
/// Non-zero makes cost coverage explicitly incomplete.
|
||||
routed_usage_dropped_records: u64,
|
||||
@@ -581,6 +582,7 @@ pub enum Event {
|
||||
/// provider never reported usage for the call — absence is honest, and
|
||||
/// fields inside `usage` stay `None` when the provider omits them.
|
||||
TurnUsage {
|
||||
accounting: Option<crate::cost_status::RuntimeUsageRecord>,
|
||||
/// Primary request allowance; not a claim of provider-reported usage.
|
||||
max_output_tokens: Option<u32>,
|
||||
usage: Usage,
|
||||
|
||||
@@ -630,6 +630,7 @@ pub fn event_to_protocol(event: &Event, ids: &ProtocolIds) -> wire::EventMsg {
|
||||
error,
|
||||
tool_catalog,
|
||||
base_url,
|
||||
..
|
||||
} => wire::EventMsg::TurnComplete {
|
||||
thread_id,
|
||||
session_id,
|
||||
@@ -658,6 +659,7 @@ pub fn event_to_protocol(event: &Event, ids: &ProtocolIds) -> wire::EventMsg {
|
||||
first_token_ms,
|
||||
request_ms,
|
||||
finish_reason: _,
|
||||
..
|
||||
} => wire::EventMsg::TurnUsage {
|
||||
max_output_tokens: *max_output_tokens,
|
||||
thread_id,
|
||||
@@ -1445,6 +1447,8 @@ mod tests {
|
||||
};
|
||||
let complete = event_to_protocol(
|
||||
&Event::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -1489,6 +1493,7 @@ mod tests {
|
||||
for (event, tag) in [
|
||||
(
|
||||
Event::TurnUsage {
|
||||
accounting: None,
|
||||
max_output_tokens: None,
|
||||
usage: parent,
|
||||
duration_ms: 12,
|
||||
@@ -1521,6 +1526,8 @@ mod tests {
|
||||
#[test]
|
||||
fn manual_compaction_terminal_keeps_its_operation_id_in_projection() {
|
||||
let event = Event::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: Some("compact-1".into()),
|
||||
@@ -1588,6 +1595,8 @@ mod tests {
|
||||
submission_id: Some("sub-host-1".into()),
|
||||
},
|
||||
Event::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: Some("turn-1".into()),
|
||||
submission_id: Some("sub-host-1".into()),
|
||||
compaction_id: None,
|
||||
@@ -1617,6 +1626,7 @@ mod tests {
|
||||
request_ms: None,
|
||||
},
|
||||
Event::TurnUsage {
|
||||
accounting: None,
|
||||
max_output_tokens: None,
|
||||
usage,
|
||||
duration_ms: 12,
|
||||
|
||||
+1832
-147
File diff suppressed because it is too large.
Load diff
@@ -624,6 +624,7 @@ pub trait FleetRouterCaller: Send + Sync + std::fmt::Debug {
|
||||
/// A Reasoning Router bound to its own exact preflighted route.
|
||||
#[derive(Clone)]
|
||||
pub struct LiveFleetRouter {
|
||||
accounting_origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
client: crate::client::CodewhaleClient,
|
||||
captured: CapturedReasoningRouter,
|
||||
route: PreflightedRoute,
|
||||
@@ -660,6 +661,7 @@ impl LiveFleetRouter {
|
||||
pub fn bind(
|
||||
captured: &CapturedReasoningRouter,
|
||||
config: &Config,
|
||||
accounting_origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
) -> Result<Self, RouterBindError> {
|
||||
let route = preflight_route(
|
||||
&captured.id,
|
||||
@@ -698,6 +700,7 @@ impl LiveFleetRouter {
|
||||
let call = router_call_plan(captured.requested_call_reasoning, &route.capability);
|
||||
|
||||
Ok(Self {
|
||||
accounting_origin,
|
||||
client,
|
||||
captured: captured.clone(),
|
||||
route,
|
||||
@@ -768,11 +771,44 @@ impl FleetRouterCaller for LiveFleetRouter {
|
||||
top_p: None,
|
||||
};
|
||||
|
||||
let response = self
|
||||
#[cfg(not(any(test, feature = "test-support")))]
|
||||
if self.accounting_origin.is_none() {
|
||||
return Err("reasoning router has no admitted accounting origin".to_string());
|
||||
}
|
||||
let route = self
|
||||
.client
|
||||
.create_message(request)
|
||||
.effective_route_envelope(&self.route.wire_model, chrono::Utc::now());
|
||||
let source = crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
format!("fleet-router:{}", uuid::Uuid::new_v4()),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
self.accounting_origin.clone(),
|
||||
);
|
||||
let observation = source
|
||||
.observe_route(&route)
|
||||
.map_err(|error| error.to_string())?;
|
||||
source
|
||||
.admit_dispatch(&route)
|
||||
.await
|
||||
.map_err(|error| error.to_string())?;
|
||||
let wire = observation.wire_observations.clone();
|
||||
let response = crate::llm_client::observe_request_retries(
|
||||
Some(observation),
|
||||
self.client.create_message(request),
|
||||
)
|
||||
.await;
|
||||
let record = source
|
||||
.response_record(&route, &response, &wire.drain())
|
||||
.map_err(|error| error.to_string())?;
|
||||
if record.has_captured_origin() {
|
||||
crate::cost_status::settle_runtime_usage_record(
|
||||
crate::cost_status::scope_token(),
|
||||
None,
|
||||
&record,
|
||||
)
|
||||
.await
|
||||
.map_err(|error| error.to_string())?;
|
||||
}
|
||||
let response = response.map_err(|error| error.to_string())?;
|
||||
if codewhale_models::is_incomplete_stop_reason(response.stop_reason.as_deref()) {
|
||||
return Err(format!(
|
||||
"reasoning router response incomplete: provider stop reason `{}`",
|
||||
@@ -1070,6 +1106,7 @@ impl ExactFleetWorkflow {
|
||||
captured_at: impl Into<String>,
|
||||
config: Option<&Config>,
|
||||
search_roots: &[FleetSearchRoot],
|
||||
accounting_origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
) -> Result<Self, String> {
|
||||
let exact = document
|
||||
.exact()
|
||||
@@ -1106,7 +1143,8 @@ impl ExactFleetWorkflow {
|
||||
.map_err(|error| error.to_string())?;
|
||||
|
||||
// Preflight every worker route before anything else can happen.
|
||||
let (preflight, router) = Self::preflight_and_bind(&snapshot, captured_router, config)?;
|
||||
let (preflight, router) =
|
||||
Self::preflight_and_bind(&snapshot, captured_router, config, accounting_origin)?;
|
||||
|
||||
let router_unavailable = match (snapshot.router(), &router) {
|
||||
(Some(_), None) => {
|
||||
@@ -1130,6 +1168,7 @@ impl ExactFleetWorkflow {
|
||||
snapshot: &FleetSnapshot,
|
||||
captured_router: Option<CapturedReasoningRouter>,
|
||||
config: Option<&Config>,
|
||||
accounting_origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
) -> Result<(RoutePreflight, Option<Arc<dyn FleetRouterCaller>>), String> {
|
||||
let Some(config) = config else {
|
||||
return Err(format!(
|
||||
@@ -1180,7 +1219,7 @@ impl ExactFleetWorkflow {
|
||||
let mut router: Option<Arc<dyn FleetRouterCaller>> = None;
|
||||
let mut router_route = None;
|
||||
if let Some(captured) = &captured_router {
|
||||
match LiveFleetRouter::bind(captured, config) {
|
||||
match LiveFleetRouter::bind(captured, config, accounting_origin.clone()) {
|
||||
Ok(live) => {
|
||||
router_route = Some(live.route().clone());
|
||||
router = Some(Arc::new(live));
|
||||
@@ -3036,6 +3075,7 @@ permissions = "read_only"
|
||||
"2026-08-14T00:00:00Z",
|
||||
Some(&config),
|
||||
&[],
|
||||
None,
|
||||
)
|
||||
.expect("legacy Cloud Fleet starts");
|
||||
let route = workflow
|
||||
@@ -3068,7 +3108,7 @@ call_reasoning = "low"
|
||||
.expect("legacy Cloud router profile parses");
|
||||
let captured =
|
||||
CapturedReasoningRouter::from_profile(&router_profile, "workspace".to_string());
|
||||
let live = LiveFleetRouter::bind(&captured, &config)
|
||||
let live = LiveFleetRouter::bind(&captured, &config, None)
|
||||
.expect("legacy Cloud Router binds its source table and secret");
|
||||
assert_eq!(live.route.provider_id, "ollama-cloud");
|
||||
assert_eq!(live.route.provider_config_id.as_deref(), Some("ollama"));
|
||||
@@ -3649,6 +3689,7 @@ mod saved_fleet_tests {
|
||||
"2026-09-22T00:00:00Z",
|
||||
Some(&config),
|
||||
&roots,
|
||||
None,
|
||||
)
|
||||
.expect("capture");
|
||||
|
||||
|
||||
@@ -363,7 +363,7 @@ impl QuotaExhaustionError {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize)]
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum RequestTransmission {
|
||||
NotDispatched,
|
||||
@@ -1471,7 +1471,7 @@ pub struct OutboundWireObservation {
|
||||
pub image_projection: Option<ImageProjectionReceipt>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize)]
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct OutboundTransmissionObservation {
|
||||
pub source_id: String,
|
||||
pub method: String,
|
||||
@@ -1479,7 +1479,7 @@ pub struct OutboundTransmissionObservation {
|
||||
pub http_status: Option<u16>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Default, serde::Serialize)]
|
||||
#[derive(Debug, Clone, Default, serde::Serialize)]
|
||||
pub struct OutboundWireObservationBatch {
|
||||
pub observations: Vec<OutboundWireObservation>,
|
||||
pub dropped: u64,
|
||||
@@ -1501,6 +1501,14 @@ pub struct OutboundWireObservations {
|
||||
}
|
||||
|
||||
impl OutboundWireObservations {
|
||||
pub fn snapshot(&self) -> OutboundWireObservationBatch {
|
||||
self.buffer
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.batch
|
||||
.clone()
|
||||
}
|
||||
|
||||
pub fn drain(&self) -> OutboundWireObservationBatch {
|
||||
let mut buffer = self
|
||||
.buffer
|
||||
@@ -1618,6 +1626,50 @@ pub struct RequestRetryObservation {
|
||||
pub wire_observations: OutboundWireObservations,
|
||||
}
|
||||
|
||||
impl RequestRetryObservation {
|
||||
pub fn isolated() -> Self {
|
||||
Self::for_auxiliary_request(uuid::Uuid::new_v4().to_string())
|
||||
}
|
||||
|
||||
pub fn for_auxiliary_request(request_id: impl Into<String>) -> Self {
|
||||
Self {
|
||||
request_id: request_id.into(),
|
||||
retries: std::sync::Arc::new(std::sync::atomic::AtomicU32::new(0)),
|
||||
emit: std::sync::Arc::new(|_| Box::pin(async {})),
|
||||
wire_observations: OutboundWireObservations::default(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl OutboundWireObservationBatch {
|
||||
pub fn usage_coverage(&self, reported: bool, complete: bool) -> ProviderUsageCoverage {
|
||||
let last = self.last_transmission.as_ref();
|
||||
let unknown_prior = self.transmissions_dropped != 0
|
||||
|| self.transmissions.iter().any(|attempt| {
|
||||
attempt.transmission == RequestTransmission::OutcomeUnknown
|
||||
&& last.is_none_or(|last| last.source_id != attempt.source_id)
|
||||
});
|
||||
if unknown_prior {
|
||||
return if reported {
|
||||
ProviderUsageCoverage::Incomplete
|
||||
} else {
|
||||
ProviderUsageCoverage::Missing
|
||||
};
|
||||
}
|
||||
match last.map(|last| last.transmission) {
|
||||
None | Some(RequestTransmission::NotDispatched) => ProviderUsageCoverage::NotDispatched,
|
||||
Some(RequestTransmission::Rejected) => ProviderUsageCoverage::Rejected,
|
||||
Some(RequestTransmission::OutcomeUnknown) if reported && complete => {
|
||||
ProviderUsageCoverage::Reported
|
||||
}
|
||||
Some(RequestTransmission::OutcomeUnknown) if reported => {
|
||||
ProviderUsageCoverage::Incomplete
|
||||
}
|
||||
Some(RequestTransmission::OutcomeUnknown) => ProviderUsageCoverage::Missing,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
tokio::task_local! {
|
||||
static REQUEST_RETRY_OBSERVATION: Option<RequestRetryObservation>;
|
||||
}
|
||||
|
||||
+451
-219
@@ -11,16 +11,14 @@ use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::client::CodewhaleClient;
|
||||
use crate::client::system_one::{DecisionRouterRoute, SystemOneAnswer, SystemOneResponse};
|
||||
use crate::cost_status::{
|
||||
EffectiveRouteEnvelope, EffectiveRouteUsage, RuntimeUsageDropRecord, RuntimeUsageRecord,
|
||||
};
|
||||
use crate::cost_status::{EffectiveRouteEnvelope, RuntimeUsageDropRecord, RuntimeUsageRecord};
|
||||
use crate::llm_client::LlmClient;
|
||||
use crate::model_inventory::{ModelInventory, probability_bp};
|
||||
use crate::reasoning_preference::ReasoningEffort;
|
||||
use codewhale_config::ProviderKind;
|
||||
use codewhale_models::Role;
|
||||
use codewhale_models::auto_route_evidence::AutoRouteDecisionEvidence;
|
||||
use codewhale_models::{ContentBlock, Message, MessageRequest, MessageResponse, SystemPrompt};
|
||||
use codewhale_models::{Role, Usage};
|
||||
use codewhale_runtime::config::{
|
||||
AutoRouterKind, Config, ProviderIdentity, normalize_model_name_for_provider,
|
||||
};
|
||||
@@ -782,6 +780,7 @@ pub async fn resolve_auto_route_with_inventory_for_session(
|
||||
selected_model_mode,
|
||||
selected_thinking_mode,
|
||||
true,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
}
|
||||
@@ -794,6 +793,7 @@ pub async fn resolve_auto_route_with_inventory_for_session_and_cache_policy(
|
||||
selected_model_mode: &str,
|
||||
selected_thinking_mode: &str,
|
||||
allow_response_cache: bool,
|
||||
accounting_source: Option<crate::cost_status::NativeUsageSource>,
|
||||
) -> Result<AutoRouteSelection> {
|
||||
let mut inventory = ModelInventory::from_config(config).map_err(anyhow::Error::msg)?;
|
||||
if let Some(active) = inventory.active_default()
|
||||
@@ -836,8 +836,9 @@ pub async fn resolve_auto_route_with_inventory_for_session_and_cache_policy(
|
||||
selected_model_mode,
|
||||
selected_thinking_mode,
|
||||
allow_response_cache,
|
||||
accounting_source,
|
||||
)
|
||||
.await;
|
||||
.await?;
|
||||
Ok(normalize_auto_route_selection_for_config(config, selection))
|
||||
}
|
||||
|
||||
@@ -856,6 +857,13 @@ fn auto_route_without_router(config: &Config, inventory: &ModelInventory) -> Aut
|
||||
/// Ask the configured router (either kind) for this turn's route. Callers
|
||||
/// have already checked `inventory.router_available`. The `cfg!(any(test, feature = "test-support"))`
|
||||
/// short-circuit lives in the caller, so tests exercise this directly.
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
#[error("classifier accounting admission or settlement failed")]
|
||||
struct ClassifierAccountingFailure {
|
||||
#[source]
|
||||
source: anyhow::Error,
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
async fn auto_route_via_router(
|
||||
config: &Config,
|
||||
@@ -866,7 +874,8 @@ async fn auto_route_via_router(
|
||||
selected_model_mode: &str,
|
||||
selected_thinking_mode: &str,
|
||||
allow_response_cache: bool,
|
||||
) -> AutoRouteSelection {
|
||||
accounting_source: Option<crate::cost_status::NativeUsageSource>,
|
||||
) -> Result<AutoRouteSelection> {
|
||||
let fallback = auto_route_declared_fallback(config, inventory);
|
||||
let attempt = match inventory.router_kind {
|
||||
AutoRouterKind::Decision => {
|
||||
@@ -878,7 +887,7 @@ async fn auto_route_via_router(
|
||||
receipt.reason =
|
||||
AutoRouteReason::LocalFallback(AutoRouteHeuristicReason::NoFastSibling);
|
||||
}
|
||||
return selection;
|
||||
return Ok(selection);
|
||||
};
|
||||
auto_route_decision_recommendation(
|
||||
config,
|
||||
@@ -888,6 +897,7 @@ async fn auto_route_via_router(
|
||||
recent_context,
|
||||
session_mode,
|
||||
selected_thinking_mode,
|
||||
accounting_source,
|
||||
)
|
||||
.await
|
||||
}
|
||||
@@ -901,21 +911,23 @@ async fn auto_route_via_router(
|
||||
selected_model_mode,
|
||||
selected_thinking_mode,
|
||||
allow_response_cache,
|
||||
accounting_source,
|
||||
)
|
||||
.await
|
||||
}
|
||||
};
|
||||
match attempt {
|
||||
Ok(attempt) => auto_route_from_classifier_attempt(fallback, inventory, attempt),
|
||||
// Client construction/preparation failed before a provider request was
|
||||
// admitted. There is no provider usage to invent and no dropped
|
||||
// response receipt to claim.
|
||||
Ok(attempt) => Ok(auto_route_from_classifier_attempt(
|
||||
fallback, inventory, attempt,
|
||||
)),
|
||||
Err(error) if error.is::<ClassifierAccountingFailure>() => Err(error),
|
||||
// Only client construction/preflight failures use the local fallback.
|
||||
Err(_) => {
|
||||
let mut selection = auto_route_classifier_fallback(fallback, inventory);
|
||||
if let Some(receipt) = selection.receipt.as_mut() {
|
||||
receipt.router_failure = Some(AutoRouterFailure::NotRunnable);
|
||||
}
|
||||
selection
|
||||
Ok(selection)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -929,6 +941,7 @@ pub const ROUTER_TEST_REQUEST: &str = "Rename the variable foo to bar in src/lib
|
||||
/// the setup view never reports a test that did not happen.
|
||||
pub async fn test_auto_router(
|
||||
config: &Config,
|
||||
accounting_origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
) -> std::result::Result<(AutoRouteSelection, u64), String> {
|
||||
let inventory = ModelInventory::from_config(config)?;
|
||||
if !inventory.router_available {
|
||||
@@ -944,6 +957,13 @@ pub async fn test_auto_router(
|
||||
{
|
||||
return Err(AutoRouteHeuristicReason::NoFastSibling.label().to_string());
|
||||
}
|
||||
let accounting_source = accounting_origin.map(|origin| {
|
||||
crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
format!("router-test:{}", uuid::Uuid::new_v4()),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
Some(origin),
|
||||
)
|
||||
});
|
||||
let started = Instant::now();
|
||||
let selection = auto_route_via_router(
|
||||
config,
|
||||
@@ -954,8 +974,10 @@ pub async fn test_auto_router(
|
||||
"auto",
|
||||
"auto",
|
||||
false,
|
||||
accounting_source,
|
||||
)
|
||||
.await;
|
||||
.await
|
||||
.map_err(|error| error.to_string())?;
|
||||
let latency_ms = u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX);
|
||||
// `NotRunnable` is set only when the client could not be built or the
|
||||
// request failed preflight: nothing reached the network.
|
||||
@@ -1372,23 +1394,12 @@ pub fn auto_route_usage_source_id(route: &EffectiveRouteEnvelope, response_id: &
|
||||
}
|
||||
|
||||
fn auto_route_attempt_from_response(
|
||||
request_route: EffectiveRouteEnvelope,
|
||||
_request_route: EffectiveRouteEnvelope,
|
||||
response: &MessageResponse,
|
||||
inventory: &ModelInventory,
|
||||
record: RuntimeUsageRecord,
|
||||
) -> InventoryAutoRouteAttempt {
|
||||
// All-zero usage cannot price a routed segment. The dispatch caller owns
|
||||
// the stronger cache/provenance context and must explicitly classify this
|
||||
// as either a proven cache replay or missing provider billing evidence.
|
||||
let routed_usage = auto_route_usage_has_reported_data(&response.usage)
|
||||
.then(|| RuntimeUsageRecord {
|
||||
source_id: auto_route_usage_source_id(&request_route, &response.id),
|
||||
usage: EffectiveRouteUsage {
|
||||
route: request_route.sanitized_for_persistence(),
|
||||
usage: response.usage.clone(),
|
||||
},
|
||||
})
|
||||
.into_iter()
|
||||
.collect();
|
||||
let routed_usage = vec![record];
|
||||
let recommendation =
|
||||
(!codewhale_models::is_incomplete_stop_reason(response.stop_reason.as_deref()))
|
||||
.then(|| {
|
||||
@@ -1415,40 +1426,17 @@ fn auto_route_attempt_from_provider_response(
|
||||
request_route: EffectiveRouteEnvelope,
|
||||
response: &MessageResponse,
|
||||
inventory: &ModelInventory,
|
||||
record: RuntimeUsageRecord,
|
||||
) -> InventoryAutoRouteAttempt {
|
||||
let drop_route = request_route.sanitized_for_persistence();
|
||||
let mut attempt = auto_route_attempt_from_response(request_route, response, inventory);
|
||||
// This classifier request is currently not cacheable (temperature is
|
||||
// provider-default, not the deterministic Some(0.0) cache contract), so a
|
||||
// decoded all-zero response is missing provider billing evidence even when
|
||||
// the caller permits response-cache use. Do not silently reinterpret the
|
||||
// policy boolean as cache-hit provenance.
|
||||
if attempt.routed_usage.is_empty() {
|
||||
attempt.routed_usage_drop_records = vec![RuntimeUsageDropRecord {
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason::default(),
|
||||
source_id: auto_route_usage_source_id(
|
||||
&drop_route,
|
||||
&format!("missing-usage:{}", response.id),
|
||||
),
|
||||
route: drop_route,
|
||||
}];
|
||||
attempt.routed_usage_dropped_records = 1;
|
||||
}
|
||||
attempt
|
||||
auto_route_attempt_from_response(request_route, response, inventory, record)
|
||||
}
|
||||
|
||||
fn auto_route_attempt_with_dropped_response(
|
||||
request_route: EffectiveRouteEnvelope,
|
||||
record: RuntimeUsageRecord,
|
||||
failure: AutoRouterFailure,
|
||||
) -> InventoryAutoRouteAttempt {
|
||||
let request_route = request_route.sanitized_for_persistence();
|
||||
InventoryAutoRouteAttempt {
|
||||
routed_usage_drop_records: vec![RuntimeUsageDropRecord {
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason::default(),
|
||||
source_id: auto_route_usage_source_id(&request_route, "transport-error"),
|
||||
route: request_route,
|
||||
}],
|
||||
routed_usage_dropped_records: 1,
|
||||
routed_usage: vec![record],
|
||||
..InventoryAutoRouteAttempt::failed(failure)
|
||||
}
|
||||
}
|
||||
@@ -1470,6 +1458,7 @@ async fn auto_route_inventory_recommendation(
|
||||
selected_model_mode: &str,
|
||||
selected_thinking_mode: &str,
|
||||
allow_response_cache: bool,
|
||||
accounting_source: Option<crate::cost_status::NativeUsageSource>,
|
||||
) -> Result<InventoryAutoRouteAttempt> {
|
||||
let mut router_config = config.clone();
|
||||
// The classifier runs on the inventory's router route: the explicit
|
||||
@@ -1527,46 +1516,106 @@ async fn auto_route_inventory_recommendation(
|
||||
preflight_auto_route_request(&client, &request)?;
|
||||
let request_route =
|
||||
client.effective_route_envelope(&inventory.router_model, chrono::Utc::now());
|
||||
let response = if allow_response_cache {
|
||||
tokio::time::timeout(
|
||||
Duration::from_secs(inventory.router_timeout_secs),
|
||||
client.create_message(request),
|
||||
)
|
||||
.await
|
||||
} else {
|
||||
tokio::time::timeout(
|
||||
Duration::from_secs(inventory.router_timeout_secs),
|
||||
client.create_message_without_response_cache(request),
|
||||
)
|
||||
.await
|
||||
};
|
||||
let response = match response {
|
||||
Ok(Ok(response)) => response,
|
||||
// The request crossed Codewhale's dispatch boundary, but no exact
|
||||
// provider usage came back. Preserve the fallback while explicitly
|
||||
// failing cost coverage closed.
|
||||
Ok(Err(error)) => {
|
||||
return Ok(auto_route_attempt_with_dropped_response(
|
||||
request_route,
|
||||
crate::client::system_one::router_failure_from_error(&error),
|
||||
));
|
||||
let accounting_source = accounting_source
|
||||
.or_else(|| {
|
||||
cfg!(any(test, feature = "test-support")).then(|| {
|
||||
crate::cost_status::NativeUsageSource::capture(
|
||||
format!("auto-router:{}", uuid::Uuid::new_v4()),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
)
|
||||
})
|
||||
})
|
||||
.ok_or_else(|| ClassifierAccountingFailure {
|
||||
source: anyhow::anyhow!("classifier requires a captured native accounting source"),
|
||||
})?;
|
||||
if !cfg!(any(test, feature = "test-support")) && !accounting_source.has_captured_origin() {
|
||||
return Err(ClassifierAccountingFailure {
|
||||
source: anyhow::anyhow!("classifier has no admitted usage origin"),
|
||||
}
|
||||
// The local deadline cancels the future and can fire while the request
|
||||
// is still waiting on an application/provider permit. With no response
|
||||
// evidence we must not invent a provider call or a missing-usage
|
||||
// receipt. Transport errors returned by the client remain the
|
||||
// conservative explicit-dropped path above.
|
||||
Err(_) => {
|
||||
.into());
|
||||
}
|
||||
async {
|
||||
let observation = accounting_source.observe_route(&request_route)?;
|
||||
accounting_source.admit_dispatch(&request_route).await?;
|
||||
let response = if allow_response_cache {
|
||||
tokio::time::timeout(
|
||||
Duration::from_secs(inventory.router_timeout_secs),
|
||||
crate::llm_client::observe_request_retries(
|
||||
Some(observation.clone()),
|
||||
client.create_message(request),
|
||||
),
|
||||
)
|
||||
.await
|
||||
} else {
|
||||
tokio::time::timeout(
|
||||
Duration::from_secs(inventory.router_timeout_secs),
|
||||
client.create_message_without_response_cache(request, observation.clone()),
|
||||
)
|
||||
.await
|
||||
};
|
||||
let wire = observation.wire_observations.drain();
|
||||
let (response, failure) = match response {
|
||||
Ok(Ok(response)) => (Some(response), None),
|
||||
Ok(Err(error)) => (
|
||||
None,
|
||||
Some(crate::client::system_one::router_failure_from_error(&error)),
|
||||
),
|
||||
Err(_) => (None, Some(AutoRouterFailure::Timeout)),
|
||||
};
|
||||
if response.is_none() && wire.transmissions.is_empty() {
|
||||
return Ok(InventoryAutoRouteAttempt::failed(
|
||||
AutoRouterFailure::Timeout,
|
||||
failure.unwrap_or(AutoRouterFailure::Transport),
|
||||
));
|
||||
}
|
||||
};
|
||||
Ok(auto_route_attempt_from_provider_response(
|
||||
request_route,
|
||||
&response,
|
||||
inventory,
|
||||
))
|
||||
let usage = response
|
||||
.as_ref()
|
||||
.map(|response| &response.usage)
|
||||
.cloned()
|
||||
.unwrap_or_default();
|
||||
let complete = response.is_some();
|
||||
let coverage = if wire.last_transmission.is_none() {
|
||||
if auto_route_usage_has_reported_data(&usage) {
|
||||
if complete {
|
||||
crate::llm_client::ProviderUsageCoverage::Reported
|
||||
} else {
|
||||
crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
}
|
||||
} else {
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
}
|
||||
} else {
|
||||
wire.usage_coverage(auto_route_usage_has_reported_data(&usage), complete)
|
||||
};
|
||||
let reason = if response.is_some() {
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage
|
||||
} else {
|
||||
crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown
|
||||
};
|
||||
let record =
|
||||
accounting_source.prepare_observed(&request_route, &usage, coverage, reason, &wire)?;
|
||||
if record.has_captured_origin() {
|
||||
crate::cost_status::settle_runtime_usage_record(
|
||||
crate::cost_status::scope_token(),
|
||||
None,
|
||||
&record,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
match response {
|
||||
Some(response) => Ok(auto_route_attempt_from_provider_response(
|
||||
request_route,
|
||||
&response,
|
||||
inventory,
|
||||
record,
|
||||
)),
|
||||
None => Ok(auto_route_attempt_with_dropped_response(
|
||||
record,
|
||||
failure.unwrap_or(AutoRouterFailure::Transport),
|
||||
)),
|
||||
}
|
||||
}
|
||||
.await
|
||||
.map_err(|source| ClassifierAccountingFailure { source }.into())
|
||||
}
|
||||
|
||||
/// The active provider's runnable strong/fast pair, when both tiers can run.
|
||||
@@ -1669,7 +1718,26 @@ async fn auto_route_decision_recommendation(
|
||||
recent_context: &str,
|
||||
session_mode: &str,
|
||||
selected_thinking_mode: &str,
|
||||
accounting_source: Option<crate::cost_status::NativeUsageSource>,
|
||||
) -> Result<InventoryAutoRouteAttempt> {
|
||||
let accounting_source = accounting_source
|
||||
.or_else(|| {
|
||||
cfg!(any(test, feature = "test-support")).then(|| {
|
||||
crate::cost_status::NativeUsageSource::capture(
|
||||
format!("auto-router:{}", uuid::Uuid::new_v4()),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
)
|
||||
})
|
||||
})
|
||||
.ok_or_else(|| ClassifierAccountingFailure {
|
||||
source: anyhow::anyhow!("classifier requires a captured native accounting source"),
|
||||
})?;
|
||||
if !cfg!(any(test, feature = "test-support")) && !accounting_source.has_captured_origin() {
|
||||
return Err(ClassifierAccountingFailure {
|
||||
source: anyhow::anyhow!("classifier has no admitted usage origin"),
|
||||
}
|
||||
.into());
|
||||
}
|
||||
let route = inventory
|
||||
.router_decision_route
|
||||
.ok_or_else(|| anyhow::anyhow!("decision router has no route"))?;
|
||||
@@ -1695,40 +1763,88 @@ async fn auto_route_decision_recommendation(
|
||||
// Both transports enter the existing usage ledger. TypeSafe carries its
|
||||
// own Custom/Unknown identity; no chat price or zero-cost claim is inherited.
|
||||
let request_route =
|
||||
Some(client.effective_route_envelope(&inventory.router_model, chrono::Utc::now()));
|
||||
let started = Instant::now();
|
||||
let dispatched = std::sync::atomic::AtomicBool::new(false);
|
||||
let outcome = tokio::time::timeout(
|
||||
Duration::from_secs(inventory.router_timeout_secs),
|
||||
client.system_one_decide(&body, &dispatched),
|
||||
)
|
||||
.await;
|
||||
let latency_ms = u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX);
|
||||
Ok(match outcome {
|
||||
// A deadline that fired after the request was handed to the transport
|
||||
// may still be billed: record the coverage gap. One that fired while
|
||||
// waiting on a permit sent nothing and invents no receipt.
|
||||
Err(_) => match request_route {
|
||||
Some(route) if dispatched.load(std::sync::atomic::Ordering::Acquire) => {
|
||||
auto_route_attempt_with_dropped_response(route, AutoRouterFailure::Timeout)
|
||||
client.effective_route_envelope(&inventory.router_model, chrono::Utc::now());
|
||||
async {
|
||||
let observation = accounting_source.observe_route(&request_route)?;
|
||||
accounting_source.admit_dispatch(&request_route).await?;
|
||||
let started = Instant::now();
|
||||
let outcome = tokio::time::timeout(
|
||||
Duration::from_secs(inventory.router_timeout_secs),
|
||||
client.system_one_decide(&body, observation.clone()),
|
||||
)
|
||||
.await;
|
||||
let latency_ms = u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX);
|
||||
let wire = observation.wire_observations.drain();
|
||||
let (response, failure) = match outcome {
|
||||
Ok(Ok(response)) => (Some(response), None),
|
||||
Ok(Err(failure)) => (None, Some(failure)),
|
||||
Err(_) => (None, Some(AutoRouterFailure::Timeout)),
|
||||
};
|
||||
if response.is_none() && wire.transmissions.is_empty() {
|
||||
return Ok(InventoryAutoRouteAttempt::failed(
|
||||
failure.unwrap_or(AutoRouterFailure::Transport),
|
||||
));
|
||||
}
|
||||
let usage = Usage {
|
||||
input_tokens: response
|
||||
.as_ref()
|
||||
.and_then(|response| response.usage.as_ref())
|
||||
.map_or(0, |usage| usage.input_tokens),
|
||||
output_tokens: response
|
||||
.as_ref()
|
||||
.and_then(|response| response.usage.as_ref())
|
||||
.map_or(0, |usage| usage.output_tokens),
|
||||
..Default::default()
|
||||
};
|
||||
let complete = response
|
||||
.as_ref()
|
||||
.and_then(|response| response.usage.as_ref())
|
||||
.is_some_and(|usage| usage.complete);
|
||||
let coverage = if wire.last_transmission.is_none() {
|
||||
if auto_route_usage_has_reported_data(&usage) {
|
||||
if complete {
|
||||
crate::llm_client::ProviderUsageCoverage::Reported
|
||||
} else {
|
||||
crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
}
|
||||
} else {
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
}
|
||||
_ => InventoryAutoRouteAttempt::failed(AutoRouterFailure::Timeout),
|
||||
},
|
||||
Ok(Err(failure)) => match request_route {
|
||||
Some(route) if dispatched.load(std::sync::atomic::Ordering::Acquire) => {
|
||||
auto_route_attempt_with_dropped_response(route, failure)
|
||||
}
|
||||
_ => InventoryAutoRouteAttempt::failed(failure),
|
||||
},
|
||||
Ok(Ok(response)) => decision_attempt_from_response(
|
||||
config.auto_cost_saving(),
|
||||
inventory,
|
||||
pair,
|
||||
request_route,
|
||||
&response,
|
||||
latency_ms,
|
||||
),
|
||||
})
|
||||
} else {
|
||||
wire.usage_coverage(auto_route_usage_has_reported_data(&usage), complete)
|
||||
};
|
||||
let reason = if response.is_some() {
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage
|
||||
} else {
|
||||
crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown
|
||||
};
|
||||
let record =
|
||||
accounting_source.prepare_observed(&request_route, &usage, coverage, reason, &wire)?;
|
||||
if record.has_captured_origin() {
|
||||
crate::cost_status::settle_runtime_usage_record(
|
||||
crate::cost_status::scope_token(),
|
||||
None,
|
||||
&record,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
Ok(match response {
|
||||
Some(response) => decision_attempt_from_response(
|
||||
config.auto_cost_saving(),
|
||||
inventory,
|
||||
pair,
|
||||
Some(record),
|
||||
&response,
|
||||
latency_ms,
|
||||
),
|
||||
None => auto_route_attempt_with_dropped_response(
|
||||
record,
|
||||
failure.unwrap_or(AutoRouterFailure::Transport),
|
||||
),
|
||||
})
|
||||
}
|
||||
.await
|
||||
.map_err(|source| ClassifierAccountingFailure { source }.into())
|
||||
}
|
||||
|
||||
/// A validated `choice` answer.
|
||||
@@ -1787,36 +1903,51 @@ pub fn validated_choice(answer: Option<&SystemOneAnswer>, options: &[&str]) -> O
|
||||
pub fn decision_usage_batch(
|
||||
request_route: &EffectiveRouteEnvelope,
|
||||
response: &SystemOneResponse,
|
||||
source: &crate::cost_status::NativeUsageSource,
|
||||
wire: &crate::llm_client::OutboundWireObservationBatch,
|
||||
) -> crate::cost_status::RuntimeUsageBatch {
|
||||
let usage = codewhale_models::Usage {
|
||||
input_tokens: response.usage.as_ref().map_or(0, |u| u.input_tokens),
|
||||
output_tokens: response.usage.as_ref().map_or(0, |u| u.output_tokens),
|
||||
let usage = Usage {
|
||||
input_tokens: response
|
||||
.usage
|
||||
.as_ref()
|
||||
.map_or(0, |usage| usage.input_tokens),
|
||||
output_tokens: response
|
||||
.usage
|
||||
.as_ref()
|
||||
.map_or(0, |usage| usage.output_tokens),
|
||||
..Default::default()
|
||||
};
|
||||
let source_id =
|
||||
auto_route_usage_source_id(request_route, response.id.as_deref().unwrap_or("systemone"));
|
||||
let route = request_route.sanitized_for_persistence();
|
||||
if response.usage.as_ref().is_some_and(|u| u.complete)
|
||||
&& auto_route_usage_has_reported_data(&usage)
|
||||
{
|
||||
crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: vec![RuntimeUsageRecord {
|
||||
source_id,
|
||||
usage: EffectiveRouteUsage { route, usage },
|
||||
}],
|
||||
..Default::default()
|
||||
let complete = response.usage.as_ref().is_some_and(|usage| usage.complete);
|
||||
let coverage = if wire.last_transmission.is_none() {
|
||||
if auto_route_usage_has_reported_data(&usage) {
|
||||
if complete {
|
||||
crate::llm_client::ProviderUsageCoverage::Reported
|
||||
} else {
|
||||
crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
}
|
||||
} else {
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
}
|
||||
} else {
|
||||
crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
drop_records: vec![RuntimeUsageDropRecord {
|
||||
source_id,
|
||||
route,
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason::default(),
|
||||
}],
|
||||
dropped_records: 1,
|
||||
wire.usage_coverage(auto_route_usage_has_reported_data(&usage), complete)
|
||||
};
|
||||
match source.prepare_observed(
|
||||
request_route,
|
||||
&usage,
|
||||
coverage,
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage,
|
||||
wire,
|
||||
) {
|
||||
Ok(record) => crate::cost_status::RuntimeUsageBatch {
|
||||
records: vec![record],
|
||||
..Default::default()
|
||||
},
|
||||
Err(error) => {
|
||||
tracing::error!(%error, "decision source predecessor remains unresolved");
|
||||
crate::cost_status::RuntimeUsageBatch {
|
||||
dropped_records: 1,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1827,7 +1958,7 @@ fn decision_attempt_from_response(
|
||||
cost_saving: bool,
|
||||
inventory: &ModelInventory,
|
||||
pair: &ActiveTierPair,
|
||||
request_route: Option<EffectiveRouteEnvelope>,
|
||||
usage_record: Option<RuntimeUsageRecord>,
|
||||
response: &SystemOneResponse,
|
||||
latency_ms: u64,
|
||||
) -> InventoryAutoRouteAttempt {
|
||||
@@ -1840,11 +1971,8 @@ fn decision_attempt_from_response(
|
||||
failure: None,
|
||||
fallback_reason: None,
|
||||
};
|
||||
if let Some(request_route) = request_route {
|
||||
let batch = decision_usage_batch(&request_route, response);
|
||||
attempt.routed_usage = batch.records;
|
||||
attempt.routed_usage_dropped_records = batch.dropped_records;
|
||||
attempt.routed_usage_drop_records = batch.drop_records;
|
||||
if let Some(record) = usage_record {
|
||||
attempt.routed_usage.push(record);
|
||||
}
|
||||
|
||||
// Rejected answers still incurred spend. Keep bounded diagnostic evidence
|
||||
@@ -2215,9 +2343,16 @@ mod tests {
|
||||
4.0,
|
||||
));
|
||||
let first_route = client.effective_route_envelope(model, first_at);
|
||||
let valid = auto_route_attempt_from_response(
|
||||
first_route,
|
||||
&classifier_response(
|
||||
let valid = {
|
||||
let native = crate::cost_status::NativeUsageSource::capture(
|
||||
format!(
|
||||
"auto-router:{}",
|
||||
crate::cost_status::usage_source_fingerprint(&uuid::Uuid::new_v4().to_string())
|
||||
),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
);
|
||||
let route = first_route;
|
||||
let response = classifier_response(
|
||||
"same-provider-response-id",
|
||||
&format!(r#"{{"provider":"openrouter","model":"{model}","thinking":"off"}}"#),
|
||||
"stop",
|
||||
@@ -2227,9 +2362,12 @@ mod tests {
|
||||
prompt_cache_hit_tokens: Some(3),
|
||||
..Default::default()
|
||||
},
|
||||
),
|
||||
&inventory,
|
||||
);
|
||||
);
|
||||
let record = native
|
||||
.reported(&route, &response.usage)
|
||||
.expect("native fixture response");
|
||||
auto_route_attempt_from_response(route, &response, &inventory, record)
|
||||
};
|
||||
|
||||
// Replace the live row in the same Unix second. The first routed
|
||||
// record must keep its old immutable revision and the new attempt must
|
||||
@@ -2242,9 +2380,16 @@ mod tests {
|
||||
19.0,
|
||||
));
|
||||
let second_at = first_at + chrono::Duration::nanoseconds(1);
|
||||
let invalid = auto_route_attempt_from_response(
|
||||
client.effective_route_envelope(model, second_at),
|
||||
&classifier_response(
|
||||
let invalid = {
|
||||
let native = crate::cost_status::NativeUsageSource::capture(
|
||||
format!(
|
||||
"auto-router:{}",
|
||||
crate::cost_status::usage_source_fingerprint(&uuid::Uuid::new_v4().to_string())
|
||||
),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
);
|
||||
let route = client.effective_route_envelope(model, second_at);
|
||||
let response = classifier_response(
|
||||
"same-provider-response-id",
|
||||
"not valid route json",
|
||||
"stop",
|
||||
@@ -2253,12 +2398,23 @@ mod tests {
|
||||
output_tokens: 3,
|
||||
..Default::default()
|
||||
},
|
||||
),
|
||||
&inventory,
|
||||
);
|
||||
let incomplete = auto_route_attempt_from_response(
|
||||
client.effective_route_envelope(model, second_at + chrono::Duration::nanoseconds(1)),
|
||||
&classifier_response(
|
||||
);
|
||||
let record = native
|
||||
.reported(&route, &response.usage)
|
||||
.expect("native fixture response");
|
||||
auto_route_attempt_from_response(route, &response, &inventory, record)
|
||||
};
|
||||
let incomplete = {
|
||||
let native = crate::cost_status::NativeUsageSource::capture(
|
||||
format!(
|
||||
"auto-router:{}",
|
||||
crate::cost_status::usage_source_fingerprint(&uuid::Uuid::new_v4().to_string())
|
||||
),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
);
|
||||
let route = client
|
||||
.effective_route_envelope(model, second_at + chrono::Duration::nanoseconds(1));
|
||||
let response = classifier_response(
|
||||
"same-provider-response-id",
|
||||
&format!(r#"{{"provider":"openrouter","model":"{model}"}}"#),
|
||||
"length",
|
||||
@@ -2267,46 +2423,88 @@ mod tests {
|
||||
output_tokens: 4,
|
||||
..Default::default()
|
||||
},
|
||||
),
|
||||
&inventory,
|
||||
);
|
||||
);
|
||||
let record = native
|
||||
.reported(&route, &response.usage)
|
||||
.expect("native fixture response");
|
||||
auto_route_attempt_from_response(route, &response, &inventory, record)
|
||||
};
|
||||
let missing_usage_route =
|
||||
client.effective_route_envelope(model, second_at + chrono::Duration::nanoseconds(2));
|
||||
let missing_usage = auto_route_attempt_from_provider_response(
|
||||
missing_usage_route.clone(),
|
||||
&classifier_response(
|
||||
let missing_usage = {
|
||||
let native = crate::cost_status::NativeUsageSource::capture(
|
||||
format!(
|
||||
"auto-router:{}",
|
||||
crate::cost_status::usage_source_fingerprint(&uuid::Uuid::new_v4().to_string())
|
||||
),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
);
|
||||
let route = missing_usage_route.clone();
|
||||
let response = classifier_response(
|
||||
"missing-usage-response-id",
|
||||
"not valid route json",
|
||||
"stop",
|
||||
codewhale_models::Usage::default(),
|
||||
),
|
||||
&inventory,
|
||||
);
|
||||
assert!(missing_usage.routed_usage.is_empty());
|
||||
assert_eq!(missing_usage.routed_usage_dropped_records, 1);
|
||||
assert_eq!(missing_usage.routed_usage_drop_records.len(), 1);
|
||||
);
|
||||
let record = native
|
||||
.reported(&route, &response.usage)
|
||||
.expect("native fixture response");
|
||||
auto_route_attempt_from_provider_response(route, &response, &inventory, record)
|
||||
};
|
||||
assert_eq!(missing_usage.routed_usage.len(), 1);
|
||||
assert_eq!(missing_usage.routed_usage_dropped_records, 0);
|
||||
assert!(missing_usage.routed_usage_drop_records.is_empty());
|
||||
assert_eq!(
|
||||
missing_usage.routed_usage_drop_records[0].route,
|
||||
missing_usage.routed_usage[0].usage.route,
|
||||
missing_usage_route.sanitized_for_persistence()
|
||||
);
|
||||
assert_eq!(
|
||||
missing_usage.routed_usage[0]
|
||||
.settlement
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
);
|
||||
assert!(
|
||||
missing_usage.routed_usage_drop_records[0]
|
||||
missing_usage.routed_usage[0]
|
||||
.source_id
|
||||
.starts_with("auto-router:")
|
||||
);
|
||||
assert!(
|
||||
!missing_usage.routed_usage_drop_records[0]
|
||||
!missing_usage.routed_usage[0]
|
||||
.source_id
|
||||
.contains("missing-usage-response-id")
|
||||
);
|
||||
|
||||
let transport = auto_route_attempt_with_dropped_response(
|
||||
client.effective_route_envelope(model, second_at + chrono::Duration::nanoseconds(3)),
|
||||
AutoRouterFailure::Transport,
|
||||
let transport = {
|
||||
let native = crate::cost_status::NativeUsageSource::capture(
|
||||
format!("auto-router:{}", uuid::Uuid::new_v4()),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
);
|
||||
let route = client
|
||||
.effective_route_envelope(model, second_at + chrono::Duration::nanoseconds(3));
|
||||
let record = native
|
||||
.prepare(
|
||||
&route,
|
||||
&codewhale_models::Usage::default(),
|
||||
crate::llm_client::ProviderUsageCoverage::Missing,
|
||||
crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown,
|
||||
)
|
||||
.unwrap();
|
||||
auto_route_attempt_with_dropped_response(record, AutoRouterFailure::Transport)
|
||||
};
|
||||
assert_eq!(transport.routed_usage_dropped_records, 0);
|
||||
assert!(transport.routed_usage_drop_records.is_empty());
|
||||
assert_eq!(transport.routed_usage.len(), 1);
|
||||
assert_eq!(
|
||||
transport.routed_usage[0]
|
||||
.settlement
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
);
|
||||
assert_eq!(transport.routed_usage_dropped_records, 1);
|
||||
assert_eq!(transport.routed_usage_drop_records.len(), 1);
|
||||
assert!(transport.routed_usage.is_empty());
|
||||
|
||||
let fallback = auto_route_declared_fallback(&config, &inventory);
|
||||
let valid = auto_route_from_classifier_attempt(fallback.clone(), &inventory, valid);
|
||||
@@ -2363,30 +2561,30 @@ mod tests {
|
||||
// replaying any record cannot add parent-route spend or a second
|
||||
// routed segment, while distinct dispatches remain distinct.
|
||||
let _cost_scope = crate::cost_status::test_scope();
|
||||
let owner = "auto-router-selection-test-owner";
|
||||
crate::cost_status::register_interactive_runtime_usage_sink(
|
||||
owner,
|
||||
crate::cost_status::scope_token(),
|
||||
);
|
||||
let lease = crate::cost_status::acquire_runtime_usage_lease(owner)
|
||||
.expect("runtime usage owner lease");
|
||||
let directory = tempfile::tempdir().unwrap();
|
||||
let manager =
|
||||
crate::session_manager::SessionManager::new(directory.path().to_path_buf()).unwrap();
|
||||
for record in &records {
|
||||
for _ in 0..2 {
|
||||
crate::cost_status::report_effective_route_for_runtime(
|
||||
crate::usage_persistence::report_runtime_usage_record_for_origin_with_manager(
|
||||
crate::cost_status::scope_token(),
|
||||
Some(lease.owner()),
|
||||
&record.source_id,
|
||||
&record.usage.route,
|
||||
&record.usage.usage,
|
||||
);
|
||||
"router-cost-fixture",
|
||||
"router-turn",
|
||||
record,
|
||||
&manager,
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
}
|
||||
crate::cost_status::finish_runtime_usage_owner(owner);
|
||||
drop(lease);
|
||||
let pending = crate::cost_status::drain();
|
||||
assert_eq!(pending.usage_source_fingerprints.len(), records.len());
|
||||
assert_eq!(pending.priced_turns, records.len() as u32);
|
||||
assert_eq!(pending.unpriced_turns, 0);
|
||||
assert_eq!(pending.source_records.len(), records.len());
|
||||
let mut cost = crate::session_manager::SessionCostSnapshot::default();
|
||||
for record in &pending.source_records {
|
||||
cost.project_usage_record(record).unwrap();
|
||||
}
|
||||
assert_eq!(cost.usage_source_fingerprints.len(), records.len());
|
||||
assert_eq!(cost.priced_turns, records.len() as u32);
|
||||
assert_eq!(cost.unpriced_turns, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -3561,9 +3759,10 @@ mod decision_router_tests {
|
||||
"decision router must be available"
|
||||
);
|
||||
auto_route_via_router(
|
||||
config, &inventory, latest, context, "agent", "auto", "auto", false,
|
||||
config, &inventory, latest, context, "agent", "auto", "auto", false, None,
|
||||
)
|
||||
.await
|
||||
.expect("fixture classifier accounting must settle")
|
||||
}
|
||||
|
||||
fn receipt(selection: &AutoRouteSelection) -> &AutoRouteReceipt {
|
||||
@@ -3769,9 +3968,17 @@ mod decision_router_tests {
|
||||
let json = serde_json::to_string(receipt).expect("receipt json");
|
||||
assert!(!json.contains(MARKER), "status {status}: body leaked");
|
||||
assert!(!receipt.router_failure.unwrap().label().contains(MARKER));
|
||||
assert!(selection.routed_usage.is_empty());
|
||||
assert_eq!(selection.routed_usage_dropped_records, 1, "status {status}");
|
||||
assert_eq!(selection.routed_usage_drop_records.len(), 1);
|
||||
assert_eq!(selection.routed_usage.len(), 1);
|
||||
assert_eq!(selection.routed_usage_dropped_records, 0, "status {status}");
|
||||
assert!(selection.routed_usage_drop_records.is_empty());
|
||||
assert_eq!(
|
||||
selection.routed_usage[0]
|
||||
.settlement
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Rejected
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3797,10 +4004,26 @@ mod decision_router_tests {
|
||||
receipt.reason,
|
||||
AutoRouteReason::ClassifierFallback(_)
|
||||
));
|
||||
assert!(selection.routed_usage.is_empty());
|
||||
// The POST was sent before the deadline: coverage must fail closed.
|
||||
assert_eq!(selection.routed_usage_drop_records.len(), 1);
|
||||
assert_eq!(selection.routed_usage_dropped_records, 1);
|
||||
assert_eq!(selection.routed_usage.len(), 1);
|
||||
assert!(selection.routed_usage_drop_records.is_empty());
|
||||
assert_eq!(selection.routed_usage_dropped_records, 0);
|
||||
assert_eq!(
|
||||
selection.routed_usage[0]
|
||||
.settlement
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
);
|
||||
assert_eq!(
|
||||
selection.routed_usage[0]
|
||||
.settlement
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.physical_requests
|
||||
.len(),
|
||||
1
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -3827,7 +4050,7 @@ mod decision_router_tests {
|
||||
assert!(selection.routed_usage.is_empty());
|
||||
|
||||
// The setup test must say no call was made, not report a result.
|
||||
let reason = test_auto_router(&config)
|
||||
let reason = test_auto_router(&config, None)
|
||||
.await
|
||||
.expect_err("no test call without a strong/fast pair");
|
||||
assert_eq!(reason, AutoRouteHeuristicReason::NoFastSibling.label());
|
||||
@@ -4046,9 +4269,18 @@ mod decision_router_tests {
|
||||
Some("0.000019992")
|
||||
);
|
||||
if incomplete_usage {
|
||||
assert!(selection.routed_usage.is_empty());
|
||||
assert_eq!(selection.routed_usage_dropped_records, 1);
|
||||
assert_eq!(selection.routed_usage_drop_records.len(), 1);
|
||||
assert_eq!(selection.routed_usage.len(), 1);
|
||||
assert_eq!(selection.routed_usage_dropped_records, 0);
|
||||
assert!(selection.routed_usage_drop_records.is_empty());
|
||||
assert!(matches!(
|
||||
selection.routed_usage[0]
|
||||
.settlement
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
| crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
));
|
||||
} else {
|
||||
assert_eq!(selection.routed_usage.len(), 1);
|
||||
assert!(selection.routed_usage_drop_records.is_empty());
|
||||
|
||||
@@ -1725,7 +1725,7 @@ pub fn declared_or_catalog_quote(
|
||||
) -> Option<ProviderLivePricingQuote> {
|
||||
match declared {
|
||||
Some(quote) if quote.carries_pricing_decision() => Some(quote),
|
||||
declared => catalog()
|
||||
Some(declared) => catalog()
|
||||
.filter(|quote| {
|
||||
matches!(
|
||||
quote.provenance,
|
||||
@@ -1734,7 +1734,8 @@ pub fn declared_or_catalog_quote(
|
||||
| PricingProvenance::CloudFacts
|
||||
)
|
||||
})
|
||||
.or(declared),
|
||||
.or(Some(declared)),
|
||||
None => catalog(),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+55
-26
@@ -671,6 +671,7 @@ pub async fn run_purge(
|
||||
model: &str,
|
||||
reasoning_effort: Option<String>,
|
||||
max_tokens: u32,
|
||||
accounting_origin: crate::cost_status::NativeUsageOrigin,
|
||||
) -> Result<PurgeResult, String> {
|
||||
// 1. Build the purge prompt from the current conversation.
|
||||
let prompt = build_purge_prompt(messages);
|
||||
@@ -718,35 +719,33 @@ pub async fn run_purge(
|
||||
top_p: None,
|
||||
};
|
||||
|
||||
// 4. Send to the model. Capture the session scope before awaiting so a
|
||||
// late response cannot accrue into a subsequently loaded/new session.
|
||||
let cost_scope = crate::cost_status::scope_token();
|
||||
let cost_route = client.effective_route_envelope(model, chrono::Utc::now());
|
||||
let response = client
|
||||
.create_message(request)
|
||||
let source = crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
format!("purge:{}", uuid::Uuid::new_v4()),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
Some(accounting_origin),
|
||||
);
|
||||
let observation = source
|
||||
.observe_route(&cost_route)
|
||||
.map_err(|error| error.to_string())?;
|
||||
source
|
||||
.admit_dispatch(&cost_route)
|
||||
.await
|
||||
.map_err(|e| format!("Purge API error: {e}"))?;
|
||||
|
||||
// Report the route, not just the provider name: the endpoint decides
|
||||
// whether this is a metered public API, a plan quota, or a local runtime.
|
||||
// Purge currently has TUI admission only. Freeze that session origin rather
|
||||
// than borrowing a previous Runtime turn's owner from ambient config.
|
||||
let source_id = format!(
|
||||
"purge:{}:{}",
|
||||
cost_route
|
||||
.dispatched_at
|
||||
.timestamp_nanos_opt()
|
||||
.unwrap_or_default(),
|
||||
response.id
|
||||
);
|
||||
crate::usage_persistence::report_effective_route_for_interactive_origin(
|
||||
cost_scope,
|
||||
session_id,
|
||||
&source_id,
|
||||
&source_id,
|
||||
&cost_route,
|
||||
&response.usage,
|
||||
);
|
||||
.map_err(|error| error.to_string())?;
|
||||
let wire = observation.wire_observations.clone();
|
||||
let response = crate::llm_client::observe_request_retries(
|
||||
Some(observation),
|
||||
client.create_message(request),
|
||||
)
|
||||
.await;
|
||||
let record = source
|
||||
.response_record(&cost_route, &response, &wire.drain())
|
||||
.map_err(|error| error.to_string())?;
|
||||
crate::cost_status::settle_runtime_usage_record(cost_scope, None, &record)
|
||||
.await
|
||||
.map_err(|error| format!("Purge usage settlement failed: {error}"))?;
|
||||
let response = response.map_err(|error| format!("Purge API error: {error}"))?;
|
||||
|
||||
// A truncated response can still carry a complete-looking `purge_context`
|
||||
// call; executing it would mutate the session from incomplete output.
|
||||
@@ -886,6 +885,12 @@ mod tests {
|
||||
"mock",
|
||||
None,
|
||||
4096,
|
||||
crate::session_manager::SessionManager::default_location()
|
||||
.expect("fixture manager")
|
||||
.expect_absent_session(session_id)
|
||||
.expect("native fixture source")
|
||||
.usage_origin(crate::cost_status::scope_token())
|
||||
.expect("native fixture origin"),
|
||||
))
|
||||
.unwrap();
|
||||
assert_eq!(messages, original);
|
||||
@@ -1228,6 +1233,12 @@ mod tests {
|
||||
"mock",
|
||||
None,
|
||||
4096,
|
||||
crate::session_manager::SessionManager::default_location()
|
||||
.expect("fixture manager")
|
||||
.expect_absent_session("purge-test")
|
||||
.expect("native fixture source")
|
||||
.usage_origin(crate::cost_status::scope_token())
|
||||
.expect("native fixture origin"),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
@@ -1274,6 +1285,12 @@ mod tests {
|
||||
"mock",
|
||||
None,
|
||||
4096,
|
||||
crate::session_manager::SessionManager::default_location()
|
||||
.expect("fixture manager")
|
||||
.expect_absent_session("purge-test")
|
||||
.expect("native fixture source")
|
||||
.usage_origin(crate::cost_status::scope_token())
|
||||
.expect("native fixture origin"),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
@@ -1308,6 +1325,12 @@ mod tests {
|
||||
"mock",
|
||||
None,
|
||||
4096,
|
||||
crate::session_manager::SessionManager::default_location()
|
||||
.expect("fixture manager")
|
||||
.expect_absent_session("purge-test")
|
||||
.expect("native fixture source")
|
||||
.usage_origin(crate::cost_status::scope_token())
|
||||
.expect("native fixture origin"),
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
@@ -1331,6 +1354,12 @@ mod tests {
|
||||
"mock",
|
||||
None,
|
||||
4096,
|
||||
crate::session_manager::SessionManager::default_location()
|
||||
.expect("fixture manager")
|
||||
.expect_absent_session("purge-test")
|
||||
.expect("native fixture source")
|
||||
.usage_origin(crate::cost_status::scope_token())
|
||||
.expect("native fixture origin"),
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
|
||||
@@ -105,13 +105,13 @@ impl RlmUsageAccumulator {
|
||||
let index = state.records.len();
|
||||
let source_id = format!("rlm:{}:request:{index}", state.ledger_id);
|
||||
state.records.push(Some(RlmUsageSlot {
|
||||
record: crate::cost_status::RuntimeUsageRecord {
|
||||
record: crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
source_id,
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: route.sanitized_for_persistence(),
|
||||
usage: Usage::default(),
|
||||
},
|
||||
},
|
||||
),
|
||||
completed: false,
|
||||
}));
|
||||
Ok(RlmUsageReservation { index })
|
||||
@@ -128,6 +128,51 @@ impl RlmUsageAccumulator {
|
||||
}
|
||||
|
||||
/// Attach a provider's reported usage to its already-reserved exact route.
|
||||
pub async fn complete_record(
|
||||
&self,
|
||||
reservation: RlmUsageReservation,
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
) {
|
||||
self.complete_record_sync(reservation, record);
|
||||
}
|
||||
|
||||
pub fn complete_record_sync(
|
||||
&self,
|
||||
reservation: RlmUsageReservation,
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
) {
|
||||
let mut state = self
|
||||
.state
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
let Some(Some(slot)) = state.records.get_mut(reservation.index) else {
|
||||
return;
|
||||
};
|
||||
if crate::cost_status::usage_source_fingerprint(&slot.record.source_id)
|
||||
!= crate::cost_status::usage_source_fingerprint(&record.source_id)
|
||||
|| slot.record.usage.route != record.usage.route
|
||||
{
|
||||
return;
|
||||
}
|
||||
match record.permits_replacement(slot.completed.then_some(&slot.record)) {
|
||||
Ok(true) => {
|
||||
slot.record = record.clone();
|
||||
slot.completed = true;
|
||||
}
|
||||
Ok(false) => return,
|
||||
Err(error) => {
|
||||
tracing::warn!(%error, "RLM source projection refused its predecessor");
|
||||
return;
|
||||
}
|
||||
}
|
||||
let mut usage = Usage::default();
|
||||
for slot in state.records.iter().flatten().filter(|slot| slot.completed) {
|
||||
crate::core::turn::add_usage_to(&mut usage, &slot.record.usage.usage);
|
||||
}
|
||||
state.usage = usage;
|
||||
}
|
||||
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
pub async fn complete(&self, reservation: RlmUsageReservation, usage: &Usage) {
|
||||
let mut state = self
|
||||
.state
|
||||
|
||||
@@ -93,7 +93,7 @@ const RUNTIME_STORE_WORK_DIRS: [&str; 7] = [
|
||||
"agent-mail",
|
||||
"turn-operations",
|
||||
];
|
||||
const TURN_OPERATION_BINDING_SCHEMA_VERSION: u32 = 1;
|
||||
const TURN_OPERATION_BINDING_SCHEMA_VERSION: u32 = 2;
|
||||
pub const MAX_ROUTED_USAGE_RECORDS_PER_TURN: usize = 64;
|
||||
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
@@ -203,7 +203,8 @@ const IMAGE_RUNTIME_SCHEMA_VERSION: u32 = 3;
|
||||
// Explicit allowances need a newer reader so old binaries cannot retry uncapped.
|
||||
const OUTPUT_LIMIT_RUNTIME_SCHEMA_VERSION: u32 = 4;
|
||||
const PREFIX_CHECKPOINT_RUNTIME_SCHEMA_VERSION: u32 = 5;
|
||||
const MAX_SUPPORTED_RUNTIME_SCHEMA_VERSION: u32 = OUTPUT_LIMIT_RUNTIME_SCHEMA_VERSION;
|
||||
const SOURCE_ACCOUNTING_RUNTIME_SCHEMA_VERSION: u32 = 6;
|
||||
const MAX_SUPPORTED_RUNTIME_SCHEMA_VERSION: u32 = SOURCE_ACCOUNTING_RUNTIME_SCHEMA_VERSION;
|
||||
|
||||
fn is_zero_u64(value: &u64) -> bool {
|
||||
*value == 0
|
||||
@@ -963,6 +964,13 @@ pub struct TurnRecord {
|
||||
/// inline RLM/guardian calls.
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub routed_usage: Vec<EffectiveRouteUsage>,
|
||||
#[serde(default)]
|
||||
pub source_accounting: bool,
|
||||
#[serde(default, skip_serializing_if = "std::collections::BTreeMap::is_empty")]
|
||||
pub source_usage_records:
|
||||
std::collections::BTreeMap<String, crate::cost_status::RuntimeUsageRecord>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub source_usage_origin_turn_id: Option<String>,
|
||||
/// Decision evidence shares this durable origin-turn ledger, including
|
||||
/// responses arriving after the parent terminal event.
|
||||
#[serde(
|
||||
@@ -1067,11 +1075,39 @@ impl TurnRecord {
|
||||
|| self.schema_version >= PREFIX_CHECKPOINT_RUNTIME_SCHEMA_VERSION,
|
||||
"Runtime prefix checkpoint requires its recovery schema"
|
||||
);
|
||||
if self.schema_version > PREFIX_CHECKPOINT_RUNTIME_SCHEMA_VERSION {
|
||||
anyhow::ensure!(
|
||||
(!self.source_accounting
|
||||
&& self.source_usage_records.is_empty()
|
||||
&& self.source_usage_origin_turn_id.is_none())
|
||||
|| self.schema_version >= SOURCE_ACCOUNTING_RUNTIME_SCHEMA_VERSION,
|
||||
"native source accounting requires its recovery schema"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
self.source_usage_records.is_empty() || self.source_accounting,
|
||||
"source records require native accounting reconciliation"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
self.source_usage_records.is_empty() || self.source_usage_origin_turn_id.is_some(),
|
||||
"native source receipts have no original owning turn"
|
||||
);
|
||||
if let Some(origin) = self.source_usage_origin_turn_id.as_deref() {
|
||||
validated_record_id(origin, "native usage origin turn id")?;
|
||||
}
|
||||
anyhow::ensure!(
|
||||
self.source_usage_records.len() <= MAX_ROUTED_USAGE_RECORDS_PER_TURN
|
||||
&& self.source_usage_records.iter().all(|(key, record)| record
|
||||
.settlement
|
||||
.is_some()
|
||||
&& record.validate().is_ok()
|
||||
&& *key == crate::cost_status::usage_source_fingerprint(&record.source_id)
|
||||
&& self.routed_usage_source_ids.contains(key)),
|
||||
"native source receipts exceed their bound or identity contract"
|
||||
);
|
||||
if self.schema_version > MAX_SUPPORTED_RUNTIME_SCHEMA_VERSION {
|
||||
bail!(
|
||||
"Turn schema v{} is newer than supported v{}",
|
||||
self.schema_version,
|
||||
PREFIX_CHECKPOINT_RUNTIME_SCHEMA_VERSION
|
||||
MAX_SUPPORTED_RUNTIME_SCHEMA_VERSION
|
||||
);
|
||||
}
|
||||
if (self.max_output_tokens.is_some()
|
||||
@@ -1845,11 +1881,6 @@ impl RuntimeThreadStore {
|
||||
}
|
||||
}
|
||||
|
||||
/// Remove a binding left before its turn record by a process crash.
|
||||
///
|
||||
/// Bindings are committed before turns, while engine submission happens
|
||||
/// only after both are durable. A binding with no turn therefore never
|
||||
/// reached the engine and is safe to discard during startup recovery.
|
||||
fn recover_incomplete_turn_operations(&self) -> Result<()> {
|
||||
let operations_dir = checked_existing_runtime_store_dir(&self.turn_operations_dir)?;
|
||||
for entry in fs::read_dir(&operations_dir)
|
||||
@@ -1896,24 +1927,13 @@ impl RuntimeThreadStore {
|
||||
self.with_turn_operation_claim(Some(&observed.operation_key_fingerprint), || {
|
||||
// A live writer may have replaced the file between the
|
||||
// directory scan and this claim. Re-read under the same
|
||||
// cross-process lock used by `start_turn` before deciding
|
||||
// that the binding is torn.
|
||||
let Some(binding) =
|
||||
// cross-process lock used by `start_turn`. Ambiguous
|
||||
// reservations remain available for exact receipt lookup.
|
||||
let Some(_binding) =
|
||||
self.load_turn_operation_binding(&observed.operation_key_fingerprint)?
|
||||
else {
|
||||
return Ok(());
|
||||
};
|
||||
if !self.turn_path(&binding.turn_id)?.exists() {
|
||||
// Persistence is binding -> item -> turn. A process can
|
||||
// stop after the item write but before the turn commit;
|
||||
// that item was never submitted to an engine and has no
|
||||
// authoritative parent. Remove it under the same operation
|
||||
// claim before making the key retryable.
|
||||
for item in self.list_items_for_turn(&binding.turn_id)? {
|
||||
self.remove_item(&item.id)?;
|
||||
}
|
||||
remove_file_if_exists(&path)?;
|
||||
}
|
||||
Ok(())
|
||||
})?;
|
||||
}
|
||||
@@ -1956,10 +1976,6 @@ impl RuntimeThreadStore {
|
||||
Ok(Some(binding))
|
||||
}
|
||||
|
||||
fn remove_turn_operation_binding(&self, operation_key_fingerprint: &str) -> Result<()> {
|
||||
remove_file_if_exists(&self.turn_operation_path(operation_key_fingerprint)?)
|
||||
}
|
||||
|
||||
fn recover_claimed_agent_mail(&self) -> Result<()> {
|
||||
let _mail_mutation = self.mail_mutation.lock();
|
||||
for mut mail in self.list_agent_mail()? {
|
||||
@@ -4432,6 +4448,15 @@ struct RuntimeTurnOperationBinding {
|
||||
operation_key_fingerprint: String,
|
||||
request_fingerprint: String,
|
||||
created_at: DateTime<Utc>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
preparation: Option<RuntimeTurnPreparation>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
struct RuntimeTurnPreparation {
|
||||
id: String,
|
||||
source_fingerprint: String,
|
||||
admitted: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
@@ -4476,6 +4501,17 @@ impl RuntimeTurnOperationBinding {
|
||||
validated_record_id(&self.turn_id, "operation turn id")?;
|
||||
validate_sha256_fingerprint(&self.operation_key_fingerprint, "operation key fingerprint")?;
|
||||
validate_sha256_fingerprint(&self.request_fingerprint, "operation request fingerprint")?;
|
||||
if let Some(preparation) = &self.preparation {
|
||||
anyhow::ensure!(
|
||||
self.schema_version >= 2,
|
||||
"turn preparation requires its admission schema"
|
||||
);
|
||||
validated_record_id(&preparation.id, "turn preparation id")?;
|
||||
validate_sha256_fingerprint(
|
||||
&preparation.source_fingerprint,
|
||||
"classifier source fingerprint",
|
||||
)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -5632,8 +5668,21 @@ fn write_json_atomic<T: Serialize>(path: &Path, value: &T) -> Result<()> {
|
||||
}
|
||||
reject_symlinked_store_file(path)?;
|
||||
let payload = serde_json::to_string_pretty(value)?;
|
||||
codewhale_util::utils::write_atomic(path, payload.as_bytes())
|
||||
.with_context(|| format!("Failed to write {}", path.display()))
|
||||
let parent = path
|
||||
.parent()
|
||||
.ok_or_else(|| anyhow!("Runtime record has no parent"))?;
|
||||
let name = path
|
||||
.file_name()
|
||||
.ok_or_else(|| anyhow!("Runtime record has no basename"))?;
|
||||
codewhale_util::files::WorkspaceFile::open_resolved_root(parent, Path::new(name), false)?
|
||||
.replace(payload.as_bytes())
|
||||
.with_context(|| format!("Failed to write {}", path.display()))?;
|
||||
#[cfg(unix)]
|
||||
if let (Some(ancestor), Some(name)) = (parent.parent(), parent.file_name()) {
|
||||
codewhale_util::files::WorkspaceFile::open_resolved_root(ancestor, Path::new(name), false)?
|
||||
.sync_parent()?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn remove_file_if_exists(path: &Path) -> Result<()> {
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -70,7 +70,7 @@ const SESSION_GOALS_DIR: &str = ".goals";
|
||||
const CURRENT_SESSION_GOAL_SCHEMA_VERSION: u32 = 2;
|
||||
const MAX_SESSION_GOAL_OBJECTIVE_CHARS: usize = 8_192;
|
||||
const MAX_SESSION_GOAL_FILE_BYTES: u64 = 64 * 1_024;
|
||||
pub const CURRENT_SESSION_SCHEMA_VERSION: u32 = 1;
|
||||
pub const CURRENT_SESSION_SCHEMA_VERSION: u32 = 2;
|
||||
pub(crate) const SESSION_METADATA_READ_LIMIT: usize = 16 * 1024 * 1024;
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
@@ -200,6 +200,84 @@ enum SessionWriteState {
|
||||
}
|
||||
|
||||
impl SessionWriteReceipt {
|
||||
fn session_scope(&self, require_open: bool) -> io::Result<(PathBuf, String)> {
|
||||
let predecessor = match &*self
|
||||
.0
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
{
|
||||
SessionWriteState::Observed(base) => {
|
||||
return Ok((
|
||||
base.observation.sessions_dir.clone(),
|
||||
base.observation.session_id.clone(),
|
||||
));
|
||||
}
|
||||
SessionWriteState::Pending { closed: true, .. } if require_open => {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::ResourceBusy,
|
||||
"session producer was refused",
|
||||
));
|
||||
}
|
||||
SessionWriteState::Pending {
|
||||
result: Some(Err(_)),
|
||||
acknowledgement_required: true,
|
||||
..
|
||||
} if require_open => {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::ResourceBusy,
|
||||
"session checkpoint acknowledgement failed",
|
||||
));
|
||||
}
|
||||
SessionWriteState::Pending {
|
||||
result: Some(Ok(publication)),
|
||||
..
|
||||
} => {
|
||||
return Ok((
|
||||
publication.observation.sessions_dir.clone(),
|
||||
publication.observation.session_id.clone(),
|
||||
));
|
||||
}
|
||||
SessionWriteState::Pending { predecessor, .. } => predecessor.clone(),
|
||||
};
|
||||
predecessor.session_scope(require_open)
|
||||
}
|
||||
|
||||
pub fn usage_origin(
|
||||
&self,
|
||||
scope: crate::cost_status::CostScopeToken,
|
||||
) -> io::Result<crate::cost_status::NativeUsageOrigin> {
|
||||
let (store, session) = self.session_scope(true)?;
|
||||
Ok(crate::cost_status::NativeUsageOrigin::session(
|
||||
scope, store, session,
|
||||
))
|
||||
}
|
||||
|
||||
pub fn project_usage_record(
|
||||
&self,
|
||||
scope: crate::cost_status::CostScopeToken,
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
cost: &mut SessionCostSnapshot,
|
||||
) -> io::Result<Option<(u64, u64)>> {
|
||||
if !record.has_captured_origin() {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::PermissionDenied,
|
||||
"usage projection has no captured native origin",
|
||||
));
|
||||
}
|
||||
if record.is_acknowledged() && !record.has_live_projection() {
|
||||
return Ok(None);
|
||||
}
|
||||
let (store, session) = self.session_scope(false)?;
|
||||
if !record.matches_session_projection(&store, &session)? {
|
||||
return Ok(None);
|
||||
}
|
||||
let mut candidate = cost.clone();
|
||||
let delta = candidate.project_usage_record(record)?;
|
||||
crate::cost_status::acknowledge_usage_projection(scope, record)?;
|
||||
*cost = candidate;
|
||||
Ok(Some(delta))
|
||||
}
|
||||
|
||||
fn from_base(base: SessionPublicationBase) -> Self {
|
||||
Self(Arc::new(Mutex::new(SessionWriteState::Observed(base))))
|
||||
}
|
||||
@@ -599,7 +677,7 @@ pub(crate) fn read_session_reference_bytes(path: &Path) -> io::Result<Vec<u8>> {
|
||||
|
||||
const CURRENT_QUEUE_SCHEMA_VERSION: u32 = 1;
|
||||
const LATE_USAGE_DIR: &str = ".late-usage";
|
||||
const CURRENT_LATE_USAGE_SCHEMA_VERSION: u32 = 1;
|
||||
const CURRENT_LATE_USAGE_SCHEMA_VERSION: u32 = 2;
|
||||
const MAX_LATE_USAGE_UNRESOLVED_RECORDS_PER_SESSION: usize = 64;
|
||||
const MAX_LATE_USAGE_LEDGER_BYTES: u64 = 1024 * 1024;
|
||||
const LATE_USAGE_DELETED: &[u8] = b"codewhale-session-deleted-v1\n";
|
||||
@@ -629,9 +707,45 @@ struct LateUsageRecord {
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
decision: Option<crate::cost_status::RuntimeDecisionReceipt>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
settlement: Option<crate::cost_status::UsageSourceRevision>,
|
||||
#[serde(default, skip_serializing_if = "is_false")]
|
||||
diagnostic_only: bool,
|
||||
}
|
||||
|
||||
impl LateUsageRecord {
|
||||
fn runtime_record(&self) -> crate::cost_status::RuntimeUsageRecord {
|
||||
let mut record = crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
self.source_fingerprint.clone(),
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: self.route.clone(),
|
||||
usage: self.usage.clone().unwrap_or_default(),
|
||||
},
|
||||
);
|
||||
record.settlement = self.settlement.clone();
|
||||
record
|
||||
}
|
||||
|
||||
fn unresolved(&self) -> bool {
|
||||
if self.diagnostic_only {
|
||||
return false;
|
||||
}
|
||||
self.settlement.as_ref().map_or_else(
|
||||
|| {
|
||||
self.usage
|
||||
.as_ref()
|
||||
.is_none_or(|usage| *usage == codewhale_models::Usage::default())
|
||||
},
|
||||
|revision| {
|
||||
matches!(
|
||||
revision.coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
| crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
)
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
fn captured(
|
||||
turn_id: &str,
|
||||
source_id: &str,
|
||||
@@ -649,6 +763,8 @@ impl LateUsageRecord {
|
||||
.cloned(),
|
||||
reason,
|
||||
decision: decision.map(crate::cost_status::RuntimeDecisionReceipt::sanitized),
|
||||
settlement: None,
|
||||
diagnostic_only: false,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -680,7 +796,7 @@ fn is_sha256_fingerprint(value: &str) -> bool {
|
||||
}
|
||||
|
||||
const fn default_session_schema_version() -> u32 {
|
||||
CURRENT_SESSION_SCHEMA_VERSION
|
||||
1
|
||||
}
|
||||
|
||||
const fn default_queue_schema_version() -> u32 {
|
||||
@@ -1244,6 +1360,8 @@ pub struct SessionCostSnapshot {
|
||||
/// changing these receipts or reading the full transcript.
|
||||
#[serde(default, skip_serializing_if = "BTreeSet::is_empty")]
|
||||
pub usage_source_fingerprints: BTreeSet<String>,
|
||||
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
|
||||
pub accounted_usage_sources: BTreeMap<String, crate::cost_status::UsageSourceContribution>,
|
||||
#[serde(
|
||||
default,
|
||||
skip_serializing_if = "BTreeMap::is_empty",
|
||||
@@ -1260,6 +1378,184 @@ pub struct SessionCostSnapshot {
|
||||
}
|
||||
|
||||
impl SessionCostSnapshot {
|
||||
pub fn project_usage_record(
|
||||
&mut self,
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
) -> io::Result<(u64, u64)> {
|
||||
self.project_usage_record_inner(record, false)
|
||||
}
|
||||
|
||||
pub(crate) fn project_usage_record_inner(
|
||||
&mut self,
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
protected_ledger: bool,
|
||||
) -> io::Result<(u64, u64)> {
|
||||
let key = crate::cost_status::usage_source_fingerprint(&record.source_id);
|
||||
let previous = self.accounted_usage_sources.get(&key);
|
||||
if let Some(previous) = previous {
|
||||
previous.validate()?;
|
||||
}
|
||||
if previous.is_some_and(|previous| previous.record == *record) {
|
||||
return Ok((0, 0));
|
||||
}
|
||||
if self.usage_source_fingerprints.contains(&key) && previous.is_none() {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::InvalidData,
|
||||
"legacy accounted usage has no provable prior contribution; correction retained",
|
||||
));
|
||||
}
|
||||
if !protected_ledger {
|
||||
if !record.has_live_projection() {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::WouldBlock,
|
||||
"usage source has no admitted live publication",
|
||||
));
|
||||
}
|
||||
record.permits_replacement(previous.map(|previous| &previous.record))?;
|
||||
} else {
|
||||
record.validate()?;
|
||||
if let Some(previous) = previous {
|
||||
if !record.follows_protected(&previous.record)? {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::InvalidData,
|
||||
"protected usage projection changed route or spend owner",
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
let next = crate::cost_status::UsageSourceContribution::for_record(record);
|
||||
let old_tokens =
|
||||
previous.map_or(0, crate::cost_status::UsageSourceContribution::total_tokens);
|
||||
let new_tokens = next.total_tokens();
|
||||
let old_usd = previous.map_or(0.0, |previous| previous.usd);
|
||||
let old_cny = previous.map_or(0.0, |previous| previous.cny);
|
||||
let (usd, cny) = if next.owner() == crate::cost_status::UsageSpendOwner::Primary {
|
||||
(&mut self.session_cost_usd, &mut self.session_cost_cny)
|
||||
} else {
|
||||
(&mut self.subagent_cost_usd, &mut self.subagent_cost_cny)
|
||||
};
|
||||
if *usd + f64::EPSILON < old_usd || *cny + f64::EPSILON < old_cny {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::InvalidData,
|
||||
"usage projection is missing its prior monetary contribution",
|
||||
));
|
||||
}
|
||||
if previous.is_some_and(|old| {
|
||||
self.priced_turns < old.priced
|
||||
|| self.unpriced_turns < old.unpriced
|
||||
|| self.cny_priced_turns < old.cny_priced
|
||||
|| self.cny_unpriced_turns < old.cny_unpriced
|
||||
}) {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::InvalidData,
|
||||
"usage projection is missing its prior coverage contribution",
|
||||
));
|
||||
}
|
||||
let updated = crate::pricing::CostEstimate {
|
||||
usd: (*usd - old_usd).max(0.0),
|
||||
cny: (*cny - old_cny).max(0.0),
|
||||
}
|
||||
.saturating_add(crate::pricing::CostEstimate {
|
||||
usd: next.usd,
|
||||
cny: next.cny,
|
||||
});
|
||||
*usd = updated.usd;
|
||||
*cny = updated.cny;
|
||||
let replace = |total: &mut u32, old: u32, new: u32| {
|
||||
*total = total.saturating_sub(old).saturating_add(new);
|
||||
};
|
||||
replace(
|
||||
&mut self.priced_turns,
|
||||
previous.map_or(0, |old| old.priced),
|
||||
next.priced,
|
||||
);
|
||||
replace(
|
||||
&mut self.unpriced_turns,
|
||||
previous.map_or(0, |old| old.unpriced),
|
||||
next.unpriced,
|
||||
);
|
||||
replace(
|
||||
&mut self.cny_priced_turns,
|
||||
previous.map_or(0, |old| old.cny_priced),
|
||||
next.cny_priced,
|
||||
);
|
||||
replace(
|
||||
&mut self.cny_unpriced_turns,
|
||||
previous.map_or(0, |old| old.cny_unpriced),
|
||||
next.cny_unpriced,
|
||||
);
|
||||
let audit = record.usage.route.audit_frozen(&record.usage.usage);
|
||||
if let Some(provenance) = audit.provenance.as_ref() {
|
||||
self.pricing_provenances
|
||||
.insert(provenance.label().to_string());
|
||||
}
|
||||
if let Some(defect) = audit.live_pricing_defect.as_ref() {
|
||||
let defects = if audit.estimate.is_some() {
|
||||
&mut self.live_pricing_defects
|
||||
} else {
|
||||
&mut self.live_pricing_unusable_defects
|
||||
};
|
||||
defects.insert(defect.label().to_string());
|
||||
}
|
||||
if audit.counts_toward_money_coverage() {
|
||||
self.unpriced_classes.extend(
|
||||
audit
|
||||
.unpriced_classes
|
||||
.iter()
|
||||
.map(|class| class.label().to_string()),
|
||||
);
|
||||
}
|
||||
self.route_receipts
|
||||
.insert(record.usage.route.receipt(&audit));
|
||||
if let Some(reason) = audit.unpriced_reason {
|
||||
self.unpriced_reasons.insert(reason.label().to_string());
|
||||
self.cny_unpriced_reasons.insert(reason.label().to_string());
|
||||
}
|
||||
if let Some(revision) = record.settlement.as_ref() {
|
||||
if matches!(
|
||||
revision.coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
| crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
) {
|
||||
self.unpriced_reasons
|
||||
.insert(revision.reason.label().to_string());
|
||||
self.cny_unpriced_reasons
|
||||
.insert(revision.reason.label().to_string());
|
||||
if self.missing_usage_sources.len() < crate::cost_status::MAX_MISSING_USAGE_SOURCES
|
||||
|| self.missing_usage_sources.contains_key(&key)
|
||||
{
|
||||
self.missing_usage_sources.insert(
|
||||
key.clone(),
|
||||
crate::cost_status::MissingUsageCoverage::for_route(
|
||||
&record.usage.route,
|
||||
revision.reason,
|
||||
),
|
||||
);
|
||||
} else {
|
||||
self.missing_usage_overflowed = true;
|
||||
}
|
||||
} else {
|
||||
self.missing_usage_sources.remove(&key);
|
||||
}
|
||||
}
|
||||
if self.unpriced_turns == 0 {
|
||||
self.unpriced_reasons.remove("request_outcome_unknown");
|
||||
self.unpriced_reasons
|
||||
.remove("provider_success_missing_usage");
|
||||
}
|
||||
if self.cny_unpriced_turns == 0 {
|
||||
self.cny_unpriced_reasons.remove("request_outcome_unknown");
|
||||
self.cny_unpriced_reasons
|
||||
.remove("provider_success_missing_usage");
|
||||
}
|
||||
self.usage_source_fingerprints.insert(key.clone());
|
||||
self.accounted_usage_sources.insert(key, next);
|
||||
let total = self.total_estimate();
|
||||
self.displayed_cost_high_water_usd = self.displayed_cost_high_water_usd.max(total.usd);
|
||||
self.displayed_cost_high_water_cny = self.displayed_cost_high_water_cny.max(total.cny);
|
||||
Ok((old_tokens, new_tokens))
|
||||
}
|
||||
|
||||
fn absorb_late_background_cost(&mut self, pool: &crate::cost_status::PendingBackgroundCost) {
|
||||
let pool = crate::cost_status::project_missing_usage_ledger(
|
||||
&mut self.missing_usage_sources,
|
||||
@@ -1300,7 +1596,6 @@ impl SessionCostSnapshot {
|
||||
.extend(pool.route_receipts.iter().cloned());
|
||||
self.usage_source_fingerprints
|
||||
.extend(pool.usage_source_fingerprints.iter().cloned());
|
||||
self.coverage_recorded = true;
|
||||
let total = self.total_estimate();
|
||||
self.displayed_cost_high_water_usd = self.displayed_cost_high_water_usd.max(total.usd);
|
||||
self.displayed_cost_high_water_cny = self.displayed_cost_high_water_cny.max(total.cny);
|
||||
@@ -1764,6 +2059,9 @@ impl SavedSession {
|
||||
)
|
||||
})?
|
||||
.capture(owner)?;
|
||||
if !self.metadata.cost.accounted_usage_sources.is_empty() {
|
||||
self.schema_version = self.schema_version.max(CURRENT_SESSION_SCHEMA_VERSION);
|
||||
}
|
||||
self.metadata.publication_id = receipt.publication_id();
|
||||
self.write_receipt = Some(receipt.clone());
|
||||
Ok(receipt)
|
||||
@@ -2657,7 +2955,43 @@ impl SessionManager {
|
||||
"late usage ledger exceeds its size bound",
|
||||
));
|
||||
}
|
||||
write_atomic(path, &bytes)
|
||||
let parent = path.parent().ok_or_else(|| {
|
||||
io::Error::new(io::ErrorKind::InvalidInput, "usage ledger has no parent")
|
||||
})?;
|
||||
let name = path.file_name().ok_or_else(|| {
|
||||
io::Error::new(io::ErrorKind::InvalidInput, "usage ledger has no basename")
|
||||
})?;
|
||||
codewhale_util::files::WorkspaceFile::open_resolved_root(parent, Path::new(name), false)?
|
||||
.replace(&bytes)?;
|
||||
Self::sync_late_usage_ledger(path)
|
||||
}
|
||||
|
||||
fn sync_late_usage_ledger(path: &Path) -> io::Result<()> {
|
||||
let parent = path.parent().ok_or_else(|| {
|
||||
io::Error::new(io::ErrorKind::InvalidInput, "usage ledger has no parent")
|
||||
})?;
|
||||
let name = path.file_name().ok_or_else(|| {
|
||||
io::Error::new(io::ErrorKind::InvalidInput, "usage ledger has no basename")
|
||||
})?;
|
||||
let ledger = codewhale_util::files::WorkspaceFile::open_resolved_root(
|
||||
parent,
|
||||
Path::new(name),
|
||||
false,
|
||||
)?;
|
||||
ledger.open_update(false, false)?.sync_all()?;
|
||||
ledger.sync_parent()?;
|
||||
#[cfg(unix)]
|
||||
for directory in [parent, parent.parent().unwrap_or(parent)] {
|
||||
if let (Some(ancestor), Some(name)) = (directory.parent(), directory.file_name()) {
|
||||
codewhale_util::files::WorkspaceFile::open_resolved_root(
|
||||
ancestor,
|
||||
Path::new(name),
|
||||
false,
|
||||
)?
|
||||
.sync_parent()?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn load_late_usage_unlocked(path: &Path) -> io::Result<LateUsageLedger> {
|
||||
@@ -2695,20 +3029,21 @@ impl SessionManager {
|
||||
}
|
||||
let ledger: LateUsageLedger = serde_json::from_slice(&raw)
|
||||
.map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))?;
|
||||
if ledger.schema_version != CURRENT_LATE_USAGE_SCHEMA_VERSION
|
||||
if !(1..=CURRENT_LATE_USAGE_SCHEMA_VERSION).contains(&ledger.schema_version)
|
||||
|| (ledger.schema_version < 2
|
||||
&& ledger
|
||||
.records
|
||||
.iter()
|
||||
.any(|record| record.settlement.is_some()))
|
||||
|| ledger
|
||||
.records
|
||||
.iter()
|
||||
.filter(|record| {
|
||||
record
|
||||
.usage
|
||||
.as_ref()
|
||||
.is_none_or(|usage| *usage == codewhale_models::Usage::default())
|
||||
})
|
||||
.filter(|record| record.unresolved())
|
||||
.count()
|
||||
> MAX_LATE_USAGE_UNRESOLVED_RECORDS_PER_SESSION
|
||||
|| ledger.records.iter().any(|record| {
|
||||
!is_sha256_fingerprint(&record.source_fingerprint)
|
||||
record.runtime_record().validate().is_err()
|
||||
|| !is_sha256_fingerprint(&record.source_fingerprint)
|
||||
|| !is_sha256_fingerprint(&record.turn_fingerprint)
|
||||
|| record.decision.as_ref().is_some_and(|r| !r.is_bounded())
|
||||
})
|
||||
@@ -2731,6 +3066,8 @@ impl SessionManager {
|
||||
let mut lock = fd_lock::RwLock::new(lock_file);
|
||||
let _guard = lock.write()?;
|
||||
if Self::late_usage_is_deleted(&path)? {
|
||||
Self::sync_late_usage_ledger(&path.with_extension("deleted"))?;
|
||||
crate::cost_status::retire_session_usage_records(&self.sessions_dir, session_id)?;
|
||||
return Ok(None);
|
||||
}
|
||||
write().map(Some)
|
||||
@@ -2786,9 +3123,18 @@ impl SessionManager {
|
||||
let _guard = lock.write()?;
|
||||
if Self::late_usage_is_deleted(&path)? {
|
||||
// Handled, rather than a failed sink that should queue a retry.
|
||||
Self::sync_late_usage_ledger(&path.with_extension("deleted"))?;
|
||||
return Ok(true);
|
||||
}
|
||||
let mut ledger = Self::load_late_usage_unlocked(&path)?;
|
||||
self.persist_late_usage_record_admitted(&path, incoming)
|
||||
}
|
||||
|
||||
fn persist_late_usage_record_admitted(
|
||||
&self,
|
||||
path: &Path,
|
||||
incoming: LateUsageRecord,
|
||||
) -> io::Result<bool> {
|
||||
let mut ledger = Self::load_late_usage_unlocked(path)?;
|
||||
let original = ledger.clone();
|
||||
let source_fingerprint = incoming.source_fingerprint.clone();
|
||||
if let Some(record) = ledger
|
||||
@@ -2804,14 +3150,27 @@ impl SessionManager {
|
||||
"late usage source changed its captured origin or route",
|
||||
));
|
||||
}
|
||||
if record.settlement.is_some()
|
||||
&& record.usage != incoming.usage
|
||||
&& incoming.decision.is_none()
|
||||
{
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::PermissionDenied,
|
||||
"serialized usage cannot replace a native source revision",
|
||||
));
|
||||
}
|
||||
let mut changed = false;
|
||||
if record
|
||||
.usage
|
||||
.as_ref()
|
||||
.is_none_or(|usage| *usage == codewhale_models::Usage::default())
|
||||
if record.settlement.is_none()
|
||||
&& record
|
||||
.usage
|
||||
.as_ref()
|
||||
.is_none_or(|usage| *usage == codewhale_models::Usage::default())
|
||||
&& let Some(usage) = incoming.usage.as_ref()
|
||||
{
|
||||
record.usage = Some(usage.clone());
|
||||
if !incoming.diagnostic_only {
|
||||
record.diagnostic_only = false;
|
||||
}
|
||||
changed = true;
|
||||
}
|
||||
if let Some(decision) = incoming.decision.as_ref() {
|
||||
@@ -2834,12 +3193,7 @@ impl SessionManager {
|
||||
&& ledger
|
||||
.records
|
||||
.iter()
|
||||
.filter(|record| {
|
||||
record
|
||||
.usage
|
||||
.as_ref()
|
||||
.is_none_or(|usage| *usage == codewhale_models::Usage::default())
|
||||
})
|
||||
.filter(|record| record.unresolved())
|
||||
.count()
|
||||
== MAX_LATE_USAGE_UNRESOLVED_RECORDS_PER_SESSION
|
||||
{
|
||||
@@ -2889,17 +3243,16 @@ impl SessionManager {
|
||||
"decision receipt exceeds its bound",
|
||||
));
|
||||
}
|
||||
self.persist_late_usage_record(
|
||||
session_id,
|
||||
LateUsageRecord::captured(
|
||||
turn_id,
|
||||
&receipt.source_id,
|
||||
&receipt.route,
|
||||
receipt.usage.as_ref().filter(|_| receipt.usage_complete),
|
||||
Some(receipt),
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage,
|
||||
),
|
||||
)
|
||||
let mut incoming = LateUsageRecord::captured(
|
||||
turn_id,
|
||||
&receipt.source_id,
|
||||
&receipt.route,
|
||||
None,
|
||||
Some(receipt),
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage,
|
||||
);
|
||||
incoming.diagnostic_only = true;
|
||||
self.persist_late_usage_record(session_id, incoming)
|
||||
}
|
||||
|
||||
/// Read the bounded provider decision evidence from the existing origin ledger.
|
||||
@@ -2916,6 +3269,202 @@ impl SessionManager {
|
||||
.collect())
|
||||
}
|
||||
|
||||
pub fn flush_pending_usage(&self, session_id: &str) -> io::Result<()> {
|
||||
self.with_session_write_admission(session_id, || {
|
||||
self.flush_pending_usage_admitted(session_id)
|
||||
})
|
||||
.map(|_| ())
|
||||
}
|
||||
|
||||
fn flush_pending_usage_admitted(&self, session_id: &str) -> io::Result<()> {
|
||||
for record in
|
||||
crate::cost_status::retained_session_usage_records(&self.sessions_dir, session_id)
|
||||
{
|
||||
let Some((store, session, operation, scope)) = record.captured_session_origin()? else {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::PermissionDenied,
|
||||
"retained usage lacks its native Session origin",
|
||||
));
|
||||
};
|
||||
if store != self.sessions_dir || session != session_id {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::PermissionDenied,
|
||||
"retained usage belongs to another Session origin",
|
||||
));
|
||||
}
|
||||
if !record.is_acknowledged() {
|
||||
self.persist_runtime_usage_record_admitted(session_id, &operation, &record)?;
|
||||
record.acknowledge_persisted()?;
|
||||
record.acknowledge_live_projection()?;
|
||||
crate::cost_status::record_interactive_runtime_usage(scope, record.clone());
|
||||
}
|
||||
crate::cost_status::retire_session_usage_record(
|
||||
&self.sessions_dir,
|
||||
session_id,
|
||||
&record,
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn persist_runtime_usage_record(
|
||||
&self,
|
||||
session_id: &str,
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
) -> io::Result<bool> {
|
||||
if !record.has_native_producer() {
|
||||
return self
|
||||
.with_session_write_admission(session_id, || {
|
||||
let (path, _) = self.late_usage_paths(session_id)?;
|
||||
let ledger = Self::load_late_usage_unlocked(&path)?;
|
||||
if ledger
|
||||
.records
|
||||
.iter()
|
||||
.any(|old| !old.diagnostic_only && old.runtime_record() == *record)
|
||||
{
|
||||
Self::sync_late_usage_ledger(&path)?;
|
||||
return Ok(true);
|
||||
}
|
||||
Err(io::Error::new(
|
||||
io::ErrorKind::PermissionDenied,
|
||||
"serialized usage cannot admit a new source or correction",
|
||||
))
|
||||
})
|
||||
.map(|handled| handled.unwrap_or(true));
|
||||
}
|
||||
let (store, origin_session, operation) = record.session_destination()?;
|
||||
if store != self.sessions_dir || origin_session != session_id {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::PermissionDenied,
|
||||
"usage source belongs to another captured session origin",
|
||||
));
|
||||
}
|
||||
let handled = self.with_session_write_admission(session_id, || {
|
||||
let handled =
|
||||
self.persist_runtime_usage_record_admitted(session_id, &operation, record)?;
|
||||
record.acknowledge_persisted()?;
|
||||
record.acknowledge_live_projection()?;
|
||||
Ok(handled)
|
||||
})?;
|
||||
if handled.is_none() {
|
||||
record.acknowledge_persisted()?;
|
||||
}
|
||||
Ok(handled.unwrap_or(true))
|
||||
}
|
||||
|
||||
pub(crate) fn persist_runtime_usage_record_admitted(
|
||||
&self,
|
||||
session_id: &str,
|
||||
turn_id: &str,
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
) -> io::Result<bool> {
|
||||
let (path, _) = self.late_usage_paths(session_id)?;
|
||||
let mut ledger = Self::load_late_usage_unlocked(&path)?;
|
||||
let original = ledger.clone();
|
||||
let key = crate::cost_status::usage_source_fingerprint(&record.source_id);
|
||||
let index = ledger
|
||||
.records
|
||||
.iter()
|
||||
.position(|old| old.source_fingerprint == key);
|
||||
let previous = index.map(|index| &ledger.records[index]);
|
||||
if previous.is_some_and(|old| {
|
||||
old.turn_fingerprint != crate::cost_status::usage_source_fingerprint(turn_id)
|
||||
|| old.route != record.usage.route.sanitized_for_persistence()
|
||||
}) {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::InvalidData,
|
||||
"usage source changed its captured origin or route",
|
||||
));
|
||||
}
|
||||
let previous_record = previous.filter(|old| !old.diagnostic_only).map(|old| {
|
||||
let mut record = crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
old.source_fingerprint.clone(),
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: old.route.clone(),
|
||||
usage: old.usage.clone().unwrap_or_default(),
|
||||
},
|
||||
);
|
||||
record.settlement = old.settlement.clone();
|
||||
record
|
||||
});
|
||||
if !record.permits_replacement(previous_record.as_ref())? {
|
||||
Self::sync_late_usage_ledger(&path)?;
|
||||
return Ok(true);
|
||||
}
|
||||
let unresolved = record.settlement.as_ref().is_some_and(|revision| {
|
||||
matches!(
|
||||
revision.coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
| crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
)
|
||||
});
|
||||
if index.is_none()
|
||||
&& unresolved
|
||||
&& ledger.records.iter().filter(|old| old.unresolved()).count()
|
||||
>= MAX_LATE_USAGE_UNRESOLVED_RECORDS_PER_SESSION
|
||||
{
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::StorageFull,
|
||||
"late usage unresolved-source capacity reached; exact receipt retained",
|
||||
));
|
||||
}
|
||||
let mut incoming = LateUsageRecord::captured(
|
||||
turn_id,
|
||||
&record.source_id,
|
||||
&record.usage.route,
|
||||
Some(&record.usage.usage),
|
||||
None,
|
||||
record.settlement.as_ref().map_or(
|
||||
crate::cost_status::RuntimeUsageMissingReason::default(),
|
||||
|revision| revision.reason,
|
||||
),
|
||||
);
|
||||
incoming.usage = Some(record.usage.usage.clone());
|
||||
incoming.settlement = record.settlement.clone();
|
||||
ledger.schema_version = CURRENT_LATE_USAGE_SCHEMA_VERSION;
|
||||
if let Some(index) = index {
|
||||
incoming.decision = ledger.records[index].decision.clone();
|
||||
ledger.records[index] = incoming;
|
||||
} else {
|
||||
ledger.records.push(incoming);
|
||||
}
|
||||
Self::write_late_usage_candidate(&path, &original, &ledger)?;
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
pub fn has_native_usage_records(&self, session_id: &str) -> io::Result<bool> {
|
||||
self.with_session_write_admission(session_id, || {
|
||||
let (path, _) = self.late_usage_paths(session_id)?;
|
||||
Ok(Self::load_late_usage_unlocked(&path)?
|
||||
.records
|
||||
.iter()
|
||||
.any(|record| record.settlement.is_some() && !record.diagnostic_only))
|
||||
})?
|
||||
.ok_or_else(Self::retired_session_write_error)
|
||||
}
|
||||
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
pub(crate) fn persist_legacy_usage_admitted(
|
||||
&self,
|
||||
session_id: &str,
|
||||
turn_id: &str,
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
) -> io::Result<bool> {
|
||||
let (path, _) = self.late_usage_paths(session_id)?;
|
||||
self.persist_late_usage_record_admitted(
|
||||
&path,
|
||||
LateUsageRecord::captured(
|
||||
turn_id,
|
||||
&record.source_id,
|
||||
&record.usage.route,
|
||||
Some(&record.usage.usage),
|
||||
None,
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage,
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
pub fn persist_late_runtime_usage(
|
||||
&self,
|
||||
session_id: &str,
|
||||
@@ -3019,11 +3568,54 @@ impl SessionManager {
|
||||
.cost
|
||||
.cny_unpriced_reasons
|
||||
.insert(LATE_USAGE_UNAVAILABLE_REASON.to_string());
|
||||
metadata.cost.coverage_recorded = true;
|
||||
return;
|
||||
}
|
||||
};
|
||||
for mut record in ledger.records {
|
||||
if record.diagnostic_only {
|
||||
if let Some(receipt) = record
|
||||
.decision
|
||||
.as_ref()
|
||||
.filter(|receipt| receipt.is_bounded())
|
||||
{
|
||||
metadata
|
||||
.cost
|
||||
.route_receipts
|
||||
.insert(receipt.diagnostic_receipt());
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if record.settlement.is_some() {
|
||||
let mut source = crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
record.source_fingerprint.clone(),
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: record.route.clone(),
|
||||
usage: record.usage.clone().unwrap_or_default(),
|
||||
},
|
||||
);
|
||||
source.settlement = record.settlement.clone();
|
||||
match metadata.cost.project_usage_record_inner(&source, true) {
|
||||
Ok((old, new)) => {
|
||||
metadata.total_tokens = metadata
|
||||
.total_tokens
|
||||
.saturating_sub(old)
|
||||
.saturating_add(new)
|
||||
}
|
||||
Err(error) => {
|
||||
tracing::warn!(%error, "saved usage projection refused an unprovable correction");
|
||||
metadata
|
||||
.cost
|
||||
.unpriced_reasons
|
||||
.insert("usage_correction_unverified".into());
|
||||
metadata
|
||||
.cost
|
||||
.cny_unpriced_reasons
|
||||
.insert("usage_correction_unverified".into());
|
||||
metadata.cost.missing_usage_overflowed = true;
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if record.usage.as_ref() == Some(&codewhale_models::Usage::default()) {
|
||||
record.usage = None;
|
||||
}
|
||||
@@ -3041,13 +3633,13 @@ impl SessionManager {
|
||||
let source_id = format!("late:{}", record.source_fingerprint);
|
||||
let mut pending = if let Some(usage) = record.usage.as_ref() {
|
||||
crate::cost_status::background_cost_for_runtime_usage(
|
||||
&crate::cost_status::RuntimeUsageRecord {
|
||||
&crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
source_id,
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: record.route,
|
||||
usage: usage.clone(),
|
||||
},
|
||||
},
|
||||
),
|
||||
)
|
||||
} else {
|
||||
crate::cost_status::background_cost_for_runtime_drop(
|
||||
@@ -3114,7 +3706,6 @@ impl SessionManager {
|
||||
.cost
|
||||
.cny_unpriced_reasons
|
||||
.insert("late_usage_ledger_overflow".to_string());
|
||||
metadata.cost.coverage_recorded = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3223,6 +3814,9 @@ impl SessionManager {
|
||||
persisted: &SessionWriteMetadata,
|
||||
preserve_lifecycle: bool,
|
||||
) -> io::Result<()> {
|
||||
if !session.metadata.cost.accounted_usage_sources.is_empty() {
|
||||
session.schema_version = session.schema_version.max(CURRENT_SESSION_SCHEMA_VERSION);
|
||||
}
|
||||
for metadata in persisted.saved.iter().chain(persisted.checkpoint.iter()) {
|
||||
Self::resolve_runtime_binding(session, Some(metadata))?;
|
||||
}
|
||||
@@ -3423,6 +4017,7 @@ impl SessionManager {
|
||||
let result = self
|
||||
.with_session_write_admission(&id, || {
|
||||
let base = self.validate_publication_admitted(&session, &receipt)?;
|
||||
self.flush_pending_usage_admitted(&id)?;
|
||||
let persisted = self.load_session_write_metadata(&id)?;
|
||||
session.make_storage_compatible();
|
||||
let checkpoint_receipt = Self::publication_receipt(&session)?;
|
||||
@@ -3465,6 +4060,7 @@ impl SessionManager {
|
||||
.with_session_write_admission(&id, || {
|
||||
let base = self.validate_publication_admitted(&session, &receipt)?;
|
||||
Self::validate_checkpoint_authority(&base)?;
|
||||
self.flush_pending_usage_admitted(&id)?;
|
||||
let persisted = self.load_session_write_metadata(&id)?;
|
||||
Self::prepare_session_write(&mut session, &persisted, true)?;
|
||||
let session_path = self.validated_session_path(&id)?;
|
||||
@@ -3546,6 +4142,7 @@ impl SessionManager {
|
||||
));
|
||||
}
|
||||
}
|
||||
self.flush_pending_usage_admitted(&id)?;
|
||||
let persisted = self.load_session_write_metadata(&id)?;
|
||||
session.make_storage_compatible();
|
||||
let checkpoint_receipt = Self::publication_receipt(&session)?;
|
||||
|
||||
@@ -84,9 +84,9 @@
|
||||
}
|
||||
|
||||
fn late_usage_test_record(source_id: &str) -> crate::cost_status::RuntimeUsageRecord {
|
||||
crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: source_id.to_string(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
source_id.to_string(),
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
None,
|
||||
ProviderKind::Deepseek,
|
||||
@@ -100,7 +100,7 @@
|
||||
..Default::default()
|
||||
},
|
||||
},
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -116,13 +116,13 @@
|
||||
.persist_late_runtime_usage(
|
||||
"decision-origin",
|
||||
"origin-turn",
|
||||
&crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: receipt.source_id.clone(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
&crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
receipt.source_id.clone(),
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: receipt.route.clone(),
|
||||
usage: receipt.usage.clone().expect("usage"),
|
||||
},
|
||||
},
|
||||
),
|
||||
)
|
||||
.expect("token append dedupes");
|
||||
drop(manager);
|
||||
@@ -173,6 +173,37 @@
|
||||
partial.source_id = "partial-provider-response-id".into();
|
||||
partial.usage_complete = false;
|
||||
partial.usage.as_mut().expect("partial usage").output_tokens = 0;
|
||||
let source = crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
partial.source_id.clone(),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
Some(
|
||||
manager
|
||||
.load_session_snapshot("decision-partial-origin")
|
||||
.unwrap()
|
||||
.write_receipt
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.usage_origin(crate::cost_status::scope_token())
|
||||
.unwrap(),
|
||||
),
|
||||
);
|
||||
let record = source
|
||||
.prepare(
|
||||
&partial.route,
|
||||
partial.usage.as_ref().unwrap(),
|
||||
crate::llm_client::ProviderUsageCoverage::Incomplete,
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage,
|
||||
)
|
||||
.unwrap();
|
||||
manager
|
||||
.with_session_write_admission("decision-partial-origin", || {
|
||||
manager.persist_runtime_usage_record_admitted(
|
||||
"decision-partial-origin",
|
||||
"partial-turn",
|
||||
&record,
|
||||
)
|
||||
})
|
||||
.unwrap();
|
||||
manager
|
||||
.persist_late_decision_receipt("decision-partial-origin", "partial-turn", &partial)
|
||||
.expect("partial diagnostic append");
|
||||
@@ -180,8 +211,8 @@
|
||||
.load_session_snapshot("decision-partial-origin")
|
||||
.expect("partial replay");
|
||||
assert_eq!(
|
||||
partial_session.metadata.total_tokens, 0,
|
||||
"partial provider counters remain diagnostic rather than an authoritative subtotal"
|
||||
partial_session.metadata.total_tokens, 9,
|
||||
"known partial counters survive independently of incomplete coverage"
|
||||
);
|
||||
assert_eq!(partial_session.metadata.cost.unpriced_turns, 1);
|
||||
assert!(
|
||||
@@ -315,7 +346,11 @@
|
||||
assert_eq!(restored.metadata.cost.usage_source_fingerprints.len(), 1);
|
||||
if malformed == "tombstone" {
|
||||
assert!(restored.write_receipt.is_none());
|
||||
assert!(restored.capture_publication(SessionWriteOwner::control()).is_err());
|
||||
assert!(
|
||||
restored
|
||||
.capture_publication(SessionWriteOwner::control())
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
manager.save_session(&restored).is_err(),
|
||||
"an invalid deletion marker must fail closed for writes"
|
||||
@@ -896,7 +931,12 @@
|
||||
.expect("finish deletion");
|
||||
assert!(manager.list_sessions().expect("list").is_empty());
|
||||
assert_eq!(fs::read(&legacy).expect("legacy preserved"), legacy_before);
|
||||
assert!(manager.load_legacy_checkpoint().expect("legacy read").is_none());
|
||||
assert!(
|
||||
manager
|
||||
.load_legacy_checkpoint()
|
||||
.expect("legacy read")
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1031,13 +1071,13 @@
|
||||
output_tokens: 5,
|
||||
..codewhale_models::Usage::default()
|
||||
};
|
||||
let usage_record = crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: "translation:old-turn:assistant:1".to_string(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
let usage_record = crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
"translation:old-turn:assistant:1".to_string(),
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: priced_route.clone(),
|
||||
usage: usage.clone(),
|
||||
},
|
||||
};
|
||||
);
|
||||
let missing_record = crate::cost_status::RuntimeUsageDropRecord {
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason::default(),
|
||||
source_id: "advisor:old-turn:provider-response:0".to_string(),
|
||||
@@ -1103,7 +1143,9 @@
|
||||
)
|
||||
.expect("late ledger");
|
||||
assert!(!ledger.contains("translation:old-turn"));
|
||||
assert!(!ledger.contains(codewhale_config::descriptors::defaults::DEFAULT_DEEPSEEK_BASE_URL));
|
||||
assert!(
|
||||
!ledger.contains(codewhale_config::descriptors::defaults::DEFAULT_DEEPSEEK_BASE_URL)
|
||||
);
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
@@ -1162,7 +1204,8 @@
|
||||
session_id,
|
||||
"bounded-turn",
|
||||
&crate::cost_status::RuntimeUsageDropRecord {
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown,
|
||||
reason:
|
||||
crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown,
|
||||
source_id: format!("late-bounded:{index}"),
|
||||
route: route.clone(),
|
||||
},
|
||||
@@ -1173,10 +1216,7 @@
|
||||
let loaded = manager
|
||||
.load_session_snapshot(session_id)
|
||||
.expect("load bounded");
|
||||
assert_eq!(
|
||||
loaded.metadata.total_tokens,
|
||||
0
|
||||
);
|
||||
assert_eq!(loaded.metadata.total_tokens, 0);
|
||||
assert_eq!(loaded.metadata.cost.unpriced_turns, 1);
|
||||
assert!(
|
||||
loaded
|
||||
@@ -1186,7 +1226,10 @@
|
||||
.contains("late_usage_ledger_overflow")
|
||||
);
|
||||
let ledger = manager.load_late_usage(session_id).expect("bounded ledger");
|
||||
assert_eq!(ledger.records.len(), MAX_LATE_USAGE_UNRESOLVED_RECORDS_PER_SESSION);
|
||||
assert_eq!(
|
||||
ledger.records.len(),
|
||||
MAX_LATE_USAGE_UNRESOLVED_RECORDS_PER_SESSION
|
||||
);
|
||||
assert!(ledger.overflowed);
|
||||
}
|
||||
|
||||
@@ -1209,16 +1252,16 @@
|
||||
Some(codewhale_config::descriptors::defaults::DEFAULT_DEEPSEEK_BASE_URL),
|
||||
Utc::now(),
|
||||
);
|
||||
let record = crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: "linked-sidecar-response".to_string(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
let record = crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
"linked-sidecar-response".to_string(),
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route,
|
||||
usage: codewhale_models::Usage {
|
||||
input_tokens: 1,
|
||||
..codewhale_models::Usage::default()
|
||||
},
|
||||
},
|
||||
};
|
||||
);
|
||||
|
||||
let outside_lock = tmp.path().join("outside.lock");
|
||||
fs::write(&outside_lock, b"outside-lock").expect("outside lock");
|
||||
@@ -1398,6 +1441,7 @@
|
||||
|
||||
// Full round-trip of every coverage field.
|
||||
let full = SessionCostSnapshot {
|
||||
accounted_usage_sources: BTreeMap::new(),
|
||||
session_cost_usd: 2.5,
|
||||
session_cost_cny: 3.0,
|
||||
subagent_cost_usd: 0.5,
|
||||
@@ -1580,14 +1624,19 @@
|
||||
Some("agent"),
|
||||
);
|
||||
manager.save_session(&session).unwrap();
|
||||
let config = toml::from_str(include_str!(
|
||||
"../../../config/tests/fixtures/custom_models.toml"
|
||||
))
|
||||
.unwrap();
|
||||
let route = crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
None,
|
||||
Some(&config),
|
||||
ProviderKind::Deepseek,
|
||||
"deepseek",
|
||||
"deepseek-v4-flash",
|
||||
Some("https://api.deepseek.com/v1"),
|
||||
"deepseek-v4.1-flash-expires-on-0910",
|
||||
Some("https://models.example.test/v1"),
|
||||
Utc::now(),
|
||||
);
|
||||
assert!(route.provider_live_pricing.is_some());
|
||||
for index in 0..65 {
|
||||
manager
|
||||
.persist_late_runtime_drop(
|
||||
@@ -1610,9 +1659,9 @@
|
||||
let manager = SessionManager::new(sessions).unwrap();
|
||||
let mut changed = route.clone();
|
||||
changed.model = "different-model".into();
|
||||
let invalid = crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: "attempt-0".into(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
let invalid = crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
"attempt-0",
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: changed,
|
||||
usage: codewhale_models::Usage {
|
||||
input_tokens: 1,
|
||||
@@ -1620,7 +1669,7 @@
|
||||
..Default::default()
|
||||
},
|
||||
},
|
||||
};
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.persist_late_runtime_usage("overflow-origin", "turn", &invalid)
|
||||
@@ -1629,9 +1678,9 @@
|
||||
io::ErrorKind::InvalidData
|
||||
);
|
||||
for index in 0..65 {
|
||||
let known = crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: format!("attempt-{index}"),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
let known = crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
format!("attempt-{index}"),
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: route.clone(),
|
||||
usage: codewhale_models::Usage {
|
||||
input_tokens: 1,
|
||||
@@ -1639,7 +1688,7 @@
|
||||
..Default::default()
|
||||
},
|
||||
},
|
||||
};
|
||||
);
|
||||
for _ in 0..2 {
|
||||
assert!(
|
||||
manager
|
||||
@@ -1693,6 +1742,8 @@
|
||||
}),
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason::default(),
|
||||
decision: None,
|
||||
settlement: None,
|
||||
diagnostic_only: false,
|
||||
};
|
||||
let empty_bytes = serde_json::to_vec(&LateUsageLedger::default())
|
||||
.unwrap()
|
||||
@@ -1713,16 +1764,16 @@
|
||||
};
|
||||
let (path, _) = manager.ensure_late_usage_paths("byte-limit").unwrap();
|
||||
SessionManager::write_late_usage_ledger(&path, &original).unwrap();
|
||||
let extra = crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: "must-not-be-marked-handled".into(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
let extra = crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
"must-not-be-marked-handled",
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route,
|
||||
usage: codewhale_models::Usage {
|
||||
input_tokens: 1,
|
||||
..Default::default()
|
||||
},
|
||||
},
|
||||
};
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.persist_late_runtime_usage("byte-limit", "turn", &extra)
|
||||
|
||||
+150
-50
@@ -57,7 +57,6 @@
|
||||
//! (Von, OpenJev, Laya) are third-party open models; only the integration
|
||||
//! architecture and the routing method here are covered by that attribution.
|
||||
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use serde_json::{Value, json};
|
||||
@@ -67,15 +66,14 @@ use codewhale_core::role::Role;
|
||||
|
||||
use crate::client::CodewhaleClient;
|
||||
use crate::client::system_one::{DecisionRouterRoute, SystemOneResponse};
|
||||
use crate::model_routing::{
|
||||
AutoRouterFailure, auto_route_usage_source_id, decision_usage_batch, truncate_for_auto_router,
|
||||
};
|
||||
use crate::model_routing::{AutoRouterFailure, decision_usage_batch, truncate_for_auto_router};
|
||||
use codewhale_runtime::config::Config;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
#[derive(Clone)]
|
||||
struct ShadowUsageContext {
|
||||
scope: crate::cost_status::CostScopeToken,
|
||||
source: crate::cost_status::NativeUsageSource,
|
||||
runtime_owner: Option<String>,
|
||||
// Retains the origin's existing durable ledger until settlement finishes.
|
||||
_lease: Option<crate::cost_status::RuntimeUsageLease>,
|
||||
@@ -84,10 +82,26 @@ struct ShadowUsageContext {
|
||||
}
|
||||
|
||||
impl ShadowUsageContext {
|
||||
fn capture(owner: Option<&str>) -> Self {
|
||||
fn capture(
|
||||
owner: Option<&str>,
|
||||
fallback_origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
) -> Self {
|
||||
let owner = owner.map(str::trim).filter(|owner| !owner.is_empty());
|
||||
let scope = crate::cost_status::scope_token();
|
||||
let origin = match owner {
|
||||
Some(owner) => crate::cost_status::NativeUsageOrigin::runtime(scope, owner).ok(),
|
||||
None => fallback_origin,
|
||||
};
|
||||
Self {
|
||||
scope: crate::cost_status::scope_token(),
|
||||
scope,
|
||||
source: crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
format!(
|
||||
"shadow:{}",
|
||||
crate::cost_status::usage_source_fingerprint(&uuid::Uuid::new_v4().to_string())
|
||||
),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
origin,
|
||||
),
|
||||
runtime_owner: owner.map(str::to_owned),
|
||||
_lease: owner.and_then(crate::cost_status::acquire_runtime_usage_lease),
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
@@ -99,7 +113,7 @@ impl ShadowUsageContext {
|
||||
let context = self.clone();
|
||||
// Existing sinks may write the origin-session ledger. Keep their
|
||||
// filesystem/SQLite work off Tokio workers as well.
|
||||
let settled = tokio::task::spawn_blocking(move || {
|
||||
let settled = codewhale_runtime::runtime_control::owner_work(move || {
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
let _origin = crate::cost_status::bind_test_cost_scope(context.test_origin);
|
||||
crate::cost_status::report_runtime_usage_batch(
|
||||
@@ -109,6 +123,7 @@ impl ShadowUsageContext {
|
||||
);
|
||||
// Release the lease while its captured test binding is still live.
|
||||
drop(context);
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
if settled.is_err() {
|
||||
@@ -263,6 +278,7 @@ pub fn spawn_shadow_gate(
|
||||
config: &Config,
|
||||
messages: &[Message],
|
||||
runtime_owner: Option<&str>,
|
||||
fallback_origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
cancel_token: &CancellationToken,
|
||||
) -> Option<tokio::task::JoinHandle<ShadowOutcome>> {
|
||||
let settings = match ShadowSettings::from_env()? {
|
||||
@@ -276,7 +292,11 @@ pub fn spawn_shadow_gate(
|
||||
let runtime = tokio::runtime::Handle::try_current().ok()?;
|
||||
let config = config.clone();
|
||||
// Capture before detaching; the next session/turn must never acquire it.
|
||||
let usage_context = ShadowUsageContext::capture(runtime_owner);
|
||||
let usage_context = ShadowUsageContext::capture(runtime_owner, fallback_origin);
|
||||
if !usage_context.source.has_captured_origin() {
|
||||
tracing::warn!(target: "superfast", "decision gate has no admitted accounting origin");
|
||||
return None;
|
||||
}
|
||||
let cancel_token = cancel_token.clone();
|
||||
Some(runtime.spawn(async move {
|
||||
let outcome = evaluate(
|
||||
@@ -356,32 +376,39 @@ async fn evaluate(
|
||||
};
|
||||
let body = decision_body(&client, &settings.model, latest_request);
|
||||
let request_route = client.effective_route_envelope(&settings.model, chrono::Utc::now());
|
||||
let dispatched = AtomicBool::new(false);
|
||||
let observation = match usage_context.source.observe_route(&request_route) {
|
||||
Ok(observation) => observation,
|
||||
Err(_) => {
|
||||
return ShadowOutcome::Failed {
|
||||
failure: AutoRouterFailure::NotRunnable,
|
||||
latency_ms: 0,
|
||||
};
|
||||
}
|
||||
};
|
||||
if let Err(error) = usage_context.source.admit_dispatch(&request_route).await {
|
||||
tracing::warn!(%error, "provider source admission failed");
|
||||
return ShadowOutcome::Failed {
|
||||
failure: AutoRouterFailure::NotRunnable,
|
||||
latency_ms: 0,
|
||||
};
|
||||
}
|
||||
let started = Instant::now();
|
||||
let answer = tokio::select! {
|
||||
biased;
|
||||
() = cancel_token.cancelled() => Err(AutoRouterFailure::Cancelled),
|
||||
result = tokio::time::timeout(settings.timeout, client.system_one_decide(&body, &dispatched)) => {
|
||||
result = tokio::time::timeout(settings.timeout, client.system_one_decide(&body, observation.clone())) => {
|
||||
result.unwrap_or(Err(AutoRouterFailure::Timeout))
|
||||
}
|
||||
};
|
||||
let latency_ms = u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX);
|
||||
let wire = observation.wire_observations.drain();
|
||||
match answer {
|
||||
Err(failure) => {
|
||||
if dispatched.load(Ordering::Acquire) {
|
||||
if !wire.transmissions.is_empty() {
|
||||
usage_context
|
||||
.report(crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
drop_records: vec![crate::cost_status::RuntimeUsageDropRecord {
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason::default(),
|
||||
source_id: auto_route_usage_source_id(
|
||||
&request_route,
|
||||
"shadow:dispatched-unreceipted",
|
||||
),
|
||||
route: request_route.sanitized_for_persistence(),
|
||||
}],
|
||||
dropped_records: 1,
|
||||
..Default::default()
|
||||
.report(match usage_context.source.prepare_observed(&request_route, &codewhale_models::Usage::default(), wire.usage_coverage(false, false), crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown, &wire) {
|
||||
Ok(record) => crate::cost_status::RuntimeUsageBatch { records: vec![record], ..Default::default() },
|
||||
Err(error) => { tracing::error!(%error, "shadow usage predecessor remains unsettled"); crate::cost_status::RuntimeUsageBatch { dropped_records: 1, ..Default::default() } }
|
||||
})
|
||||
.await;
|
||||
}
|
||||
@@ -393,7 +420,8 @@ async fn evaluate(
|
||||
Ok(response) => {
|
||||
// Billing evidence settles even when strict answer validation
|
||||
// prevents the policy from producing a recommendation.
|
||||
let mut batch = decision_usage_batch(&request_route, &response);
|
||||
let mut batch =
|
||||
decision_usage_batch(&request_route, &response, &usage_context.source, &wire);
|
||||
let intent = crate::model_routing::validated_choice(
|
||||
response.answers.get("intent"),
|
||||
&["code_change", "code_question", "command", "chat", "other"],
|
||||
@@ -401,10 +429,7 @@ async fn evaluate(
|
||||
batch
|
||||
.decisions
|
||||
.push(crate::cost_status::RuntimeDecisionReceipt {
|
||||
source_id: auto_route_usage_source_id(
|
||||
&request_route,
|
||||
response.id.as_deref().unwrap_or("systemone"),
|
||||
),
|
||||
source_id: usage_context.source.source_id(),
|
||||
route: request_route.sanitized_for_persistence(),
|
||||
usage: response.usage.as_ref().map(|u| codewhale_models::Usage {
|
||||
input_tokens: u.input_tokens,
|
||||
@@ -573,6 +598,10 @@ mod tests {
|
||||
EnvVarGuard::remove(TIMEOUT_VAR),
|
||||
]),
|
||||
}
|
||||
crate::cost_status::register_runtime_usage_sink(
|
||||
"shadow-loopback-fixture",
|
||||
std::sync::Arc::new(|_| true),
|
||||
);
|
||||
Env {
|
||||
_guards: guards,
|
||||
_home: home,
|
||||
@@ -644,7 +673,13 @@ mod tests {
|
||||
}
|
||||
|
||||
async fn run(messages: &[Message]) -> Option<ShadowOutcome> {
|
||||
let handle = spawn_shadow_gate(&config(), messages, None, &CancellationToken::new())?;
|
||||
let handle = spawn_shadow_gate(
|
||||
&config(),
|
||||
messages,
|
||||
Some("shadow-loopback-fixture"),
|
||||
None,
|
||||
&CancellationToken::new(),
|
||||
)?;
|
||||
Some(handle.await.expect("shadow task"))
|
||||
}
|
||||
|
||||
@@ -778,6 +813,7 @@ mod tests {
|
||||
let handle = spawn_shadow_gate(
|
||||
&config(),
|
||||
&turn("Run the tests"),
|
||||
Some("shadow-loopback-fixture"),
|
||||
None,
|
||||
&CancellationToken::new(),
|
||||
)
|
||||
@@ -943,10 +979,16 @@ mod tests {
|
||||
let _env = hermetic_env(Some((&server, 2_000)));
|
||||
let cancel = CancellationToken::new();
|
||||
cancel.cancel();
|
||||
let outcome = spawn_shadow_gate(&config(), &turn("hi"), None, &cancel)
|
||||
.expect("task")
|
||||
.await
|
||||
.expect("joined");
|
||||
let outcome = spawn_shadow_gate(
|
||||
&config(),
|
||||
&turn("hi"),
|
||||
Some("shadow-loopback-fixture"),
|
||||
None,
|
||||
&cancel,
|
||||
)
|
||||
.expect("task")
|
||||
.await
|
||||
.expect("joined");
|
||||
assert!(matches!(
|
||||
outcome,
|
||||
ShadowOutcome::Failed {
|
||||
@@ -976,15 +1018,16 @@ mod tests {
|
||||
let sink = dropped.clone();
|
||||
crate::cost_status::register_runtime_usage_sink_with_drop(
|
||||
owner,
|
||||
std::sync::Arc::new(|_| true),
|
||||
Some(std::sync::Arc::new(move |record| {
|
||||
std::sync::Arc::new(move |record| {
|
||||
sink.lock().expect("drop sink").push(record);
|
||||
true
|
||||
})),
|
||||
}),
|
||||
None,
|
||||
);
|
||||
|
||||
let cancel = CancellationToken::new();
|
||||
let handle = spawn_shadow_gate(&config(), &turn("hi"), Some(owner), &cancel).expect("task");
|
||||
let handle =
|
||||
spawn_shadow_gate(&config(), &turn("hi"), Some(owner), None, &cancel).expect("task");
|
||||
tokio::time::timeout(Duration::from_secs(1), async {
|
||||
while requests(&server).await.is_empty() {
|
||||
tokio::task::yield_now().await;
|
||||
@@ -1007,8 +1050,34 @@ mod tests {
|
||||
));
|
||||
assert!(start.elapsed() < Duration::from_millis(300));
|
||||
let records = dropped.lock().expect("drop records");
|
||||
assert_eq!(records.len(), 1);
|
||||
assert_eq!(records[0].route.provider_identity, "typesafe");
|
||||
assert_eq!(records.len(), 2);
|
||||
assert_eq!(records[0].usage.usage, codewhale_models::Usage::default());
|
||||
assert!(
|
||||
records[0]
|
||||
.settlement
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.physical_requests
|
||||
.is_empty()
|
||||
);
|
||||
assert_eq!(
|
||||
records[0].settlement.as_ref().unwrap().coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
);
|
||||
assert_eq!(records[1].usage.route.provider_identity, "typesafe");
|
||||
assert_eq!(
|
||||
records[1].settlement.as_ref().unwrap().coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
);
|
||||
assert_eq!(
|
||||
records[1]
|
||||
.settlement
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.physical_requests
|
||||
.len(),
|
||||
1
|
||||
);
|
||||
let batch = crate::cost_status::take_runtime_usage(owner);
|
||||
assert!(batch.records.is_empty());
|
||||
assert!(batch.drop_records.is_empty());
|
||||
@@ -1055,6 +1124,7 @@ mod tests {
|
||||
&config(),
|
||||
&turn("hi"),
|
||||
Some(owner),
|
||||
None,
|
||||
&CancellationToken::new(),
|
||||
)
|
||||
.expect("task");
|
||||
@@ -1070,14 +1140,23 @@ mod tests {
|
||||
let records = observed.lock().expect("records");
|
||||
assert_eq!(
|
||||
records.len(),
|
||||
1,
|
||||
"retired origin still owns its late response"
|
||||
2,
|
||||
"retired origin still owns its reservation and late response"
|
||||
);
|
||||
assert_eq!(records[0].usage.usage.input_tokens, 121);
|
||||
assert_eq!(records[0].usage.usage.output_tokens, 8);
|
||||
assert_eq!(records[0].usage.route.provider_identity, "typesafe");
|
||||
assert_eq!(records[0].usage.usage, codewhale_models::Usage::default());
|
||||
assert!(
|
||||
records[0]
|
||||
.settlement
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.physical_requests
|
||||
.is_empty()
|
||||
);
|
||||
assert_eq!(records[1].usage.usage.input_tokens, 121);
|
||||
assert_eq!(records[1].usage.usage.output_tokens, 8);
|
||||
assert_eq!(records[1].usage.route.provider_identity, "typesafe");
|
||||
assert_eq!(
|
||||
records[0].usage.route.billing_mode,
|
||||
records[1].usage.route.billing_mode,
|
||||
crate::cost_status::RouteBillingMode::Unknown
|
||||
);
|
||||
let receipts = decisions.lock().expect("decision receipts");
|
||||
@@ -1111,16 +1190,17 @@ mod tests {
|
||||
let sink = drops.clone();
|
||||
crate::cost_status::register_runtime_usage_sink_with_drop(
|
||||
owner,
|
||||
std::sync::Arc::new(|_| true),
|
||||
Some(std::sync::Arc::new(move |record| {
|
||||
std::sync::Arc::new(move |record| {
|
||||
sink.lock().expect("drops").push(record);
|
||||
true
|
||||
})),
|
||||
}),
|
||||
None,
|
||||
);
|
||||
let handle = spawn_shadow_gate(
|
||||
&config(),
|
||||
&turn("hi"),
|
||||
Some(owner),
|
||||
None,
|
||||
&CancellationToken::new(),
|
||||
)
|
||||
.expect("task");
|
||||
@@ -1131,13 +1211,33 @@ mod tests {
|
||||
..
|
||||
}
|
||||
));
|
||||
assert_eq!(drops.lock().expect("drop receipts").len(), 1);
|
||||
let receipts = drops.lock().expect("drop receipts");
|
||||
assert_eq!(receipts.len(), 2);
|
||||
assert_eq!(receipts[0].usage.usage, codewhale_models::Usage::default());
|
||||
assert!(
|
||||
receipts[0]
|
||||
.settlement
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.physical_requests
|
||||
.is_empty()
|
||||
);
|
||||
assert_eq!(
|
||||
receipts[1]
|
||||
.settlement
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.physical_requests
|
||||
.len(),
|
||||
1
|
||||
);
|
||||
drop(receipts);
|
||||
crate::cost_status::finish_runtime_usage_owner(owner);
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn empty_origin_and_receipt_only_batch_use_the_existing_interactive_pool() {
|
||||
let _scope = crate::cost_status::test_scope();
|
||||
let context = ShadowUsageContext::capture(Some(" "));
|
||||
let context = ShadowUsageContext::capture(Some(" "), None);
|
||||
assert!(context.runtime_owner.is_none());
|
||||
context
|
||||
.report(crate::cost_status::RuntimeUsageBatch {
|
||||
|
||||
@@ -1566,6 +1566,12 @@ impl ToolSpec for ReviewTool {
|
||||
}]
|
||||
};
|
||||
|
||||
#[cfg(not(any(test, feature = "test-support")))]
|
||||
if context.accounting_origin.is_none() {
|
||||
return Err(ToolError::not_available(
|
||||
"review has no native accounting origin",
|
||||
));
|
||||
}
|
||||
let route = client.effective_route_envelope(&self.model, chrono::Utc::now());
|
||||
let review_diff = match &source {
|
||||
ReviewSource::PullRequest { diff, .. } => diff.clone(),
|
||||
@@ -1606,6 +1612,7 @@ impl ToolSpec for ReviewTool {
|
||||
.to_string()
|
||||
});
|
||||
let mut usage = Usage::default();
|
||||
let mut source_records = Vec::new();
|
||||
let mut accumulator = plan.as_ref().map(PrReviewAccumulator::new);
|
||||
let mut single_output = None;
|
||||
let mut anchors = super::review_core::ReviewAnchorReport::default();
|
||||
@@ -1618,6 +1625,7 @@ impl ToolSpec for ReviewTool {
|
||||
let mut spend_usd = None;
|
||||
for (index, prompt) in prompts.into_iter().enumerate() {
|
||||
let mut request = super::review_core::ReviewCoreRequest::new(&review_diff, prompt);
|
||||
request.accounting_origin = context.accounting_origin.clone();
|
||||
request.workspace = tool_workspace.clone();
|
||||
request.budget = budget.tool_budget();
|
||||
request.max_output_tokens = Some(allowance);
|
||||
@@ -1629,6 +1637,7 @@ impl ToolSpec for ReviewTool {
|
||||
return Ok(review_error_with_usage(
|
||||
&route,
|
||||
&usage,
|
||||
&source_records,
|
||||
format!(
|
||||
"{}; no partial review was accepted.",
|
||||
request_failure_message(
|
||||
@@ -1640,6 +1649,7 @@ impl ToolSpec for ReviewTool {
|
||||
));
|
||||
}
|
||||
};
|
||||
source_records.extend(result.source_records.iter().cloned());
|
||||
add_review_usage(&mut usage, &result.usage);
|
||||
for tool in &result.tools_used {
|
||||
if !tools_used.iter().any(|seen| seen == tool) {
|
||||
@@ -1659,6 +1669,7 @@ impl ToolSpec for ReviewTool {
|
||||
return Ok(review_error_with_usage(
|
||||
&route,
|
||||
&usage,
|
||||
&source_records,
|
||||
format!(
|
||||
"Review pass {}/{} response incomplete: provider stop reason `{}`; the pass {}. The partial review was not accepted.",
|
||||
index + 1,
|
||||
@@ -1672,6 +1683,7 @@ impl ToolSpec for ReviewTool {
|
||||
return Ok(review_error_with_usage(
|
||||
&route,
|
||||
&usage,
|
||||
&source_records,
|
||||
leaked_tool_call_message(
|
||||
index + 1,
|
||||
plan.as_ref().map_or(1, |plan| plan.passes.len()),
|
||||
@@ -1680,19 +1692,34 @@ impl ToolSpec for ReviewTool {
|
||||
}
|
||||
if let (Some(plan), Some(accumulator)) = (&plan, accumulator.as_mut()) {
|
||||
if let Err(error) = accumulator.accept(&plan.passes[index], result.raw) {
|
||||
return Ok(review_error_with_usage(&route, &usage, error.to_string()));
|
||||
return Ok(review_error_with_usage(
|
||||
&route,
|
||||
&usage,
|
||||
&source_records,
|
||||
error.to_string(),
|
||||
));
|
||||
}
|
||||
} else {
|
||||
match accept_single_review(&result.raw) {
|
||||
Ok(_) => single_output = Some(result.output),
|
||||
Err(error) => {
|
||||
return Ok(review_error_with_usage(&route, &usage, error));
|
||||
return Ok(review_error_with_usage(
|
||||
&route,
|
||||
&usage,
|
||||
&source_records,
|
||||
error,
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Err(error) = ensure_pr_source_current(&source, &context.workspace).await {
|
||||
return Ok(review_error_with_usage(&route, &usage, error.to_string()));
|
||||
return Ok(review_error_with_usage(
|
||||
&route,
|
||||
&usage,
|
||||
&source_records,
|
||||
error.to_string(),
|
||||
));
|
||||
}
|
||||
let mut coverage = None;
|
||||
let output = if let (Some(accumulator), ReviewSource::PullRequest { diff, .. }) =
|
||||
@@ -1701,7 +1728,12 @@ impl ToolSpec for ReviewTool {
|
||||
let (output, _, completed) = match accumulator.finish(diff) {
|
||||
Ok(completed) => completed,
|
||||
Err(error) => {
|
||||
return Ok(review_error_with_usage(&route, &usage, error.to_string()));
|
||||
return Ok(review_error_with_usage(
|
||||
&route,
|
||||
&usage,
|
||||
&source_records,
|
||||
error.to_string(),
|
||||
));
|
||||
}
|
||||
};
|
||||
coverage = Some(completed);
|
||||
@@ -1719,7 +1751,7 @@ impl ToolSpec for ReviewTool {
|
||||
}
|
||||
output.summary.push_str(¬e);
|
||||
}
|
||||
let mut metadata = review_usage_metadata(&route, &usage);
|
||||
let mut metadata = review_usage_metadata(&route, &usage, &source_records);
|
||||
metadata["review_anchors"] = json!(anchors);
|
||||
metadata["review_tools_used"] = json!(tools_used);
|
||||
metadata["review_repository_reads"] = json!(tool_workspace.is_some());
|
||||
@@ -1743,14 +1775,24 @@ impl ToolSpec for ReviewTool {
|
||||
metadata["review_coverage"] = match serde_json::to_value(coverage) {
|
||||
Ok(coverage) => coverage,
|
||||
Err(error) => {
|
||||
return Ok(review_error_with_usage(&route, &usage, error.to_string()));
|
||||
return Ok(review_error_with_usage(
|
||||
&route,
|
||||
&usage,
|
||||
&source_records,
|
||||
error.to_string(),
|
||||
));
|
||||
}
|
||||
};
|
||||
}
|
||||
let result = match ToolResult::json(&output) {
|
||||
Ok(result) => result,
|
||||
Err(error) => {
|
||||
return Ok(review_error_with_usage(&route, &usage, error.to_string()));
|
||||
return Ok(review_error_with_usage(
|
||||
&route,
|
||||
&usage,
|
||||
&source_records,
|
||||
error.to_string(),
|
||||
));
|
||||
}
|
||||
};
|
||||
Ok(result.with_metadata(metadata))
|
||||
@@ -1801,14 +1843,16 @@ fn accept_single_review(response_text: &str) -> Result<ReviewOutput, String> {
|
||||
fn review_error_with_usage(
|
||||
route: &crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &Usage,
|
||||
records: &[crate::cost_status::RuntimeUsageRecord],
|
||||
message: impl Into<String>,
|
||||
) -> ToolResult {
|
||||
ToolResult::error(message.into()).with_metadata(review_usage_metadata(route, usage))
|
||||
ToolResult::error(message.into()).with_metadata(review_usage_metadata(route, usage, records))
|
||||
}
|
||||
|
||||
fn review_usage_metadata(
|
||||
route: &crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &Usage,
|
||||
records: &[crate::cost_status::RuntimeUsageRecord],
|
||||
) -> Value {
|
||||
let mut metadata = json!({
|
||||
"tool": "review",
|
||||
@@ -1818,6 +1862,13 @@ fn review_usage_metadata(
|
||||
// Every billable class, from the one shared producer, so a child turn can be
|
||||
// priced with the same completeness as a parent turn (#4318).
|
||||
crate::cost_status::attach_child_usage_metadata(&mut metadata, route, usage);
|
||||
crate::cost_status::attach_child_usage_batch_metadata(
|
||||
&mut metadata,
|
||||
&crate::cost_status::RuntimeUsageBatch {
|
||||
records: records.to_vec(),
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
metadata
|
||||
}
|
||||
|
||||
@@ -2929,7 +2980,7 @@ mod tests {
|
||||
.accept(&plan.passes[0], clean_pass("first"))
|
||||
.unwrap();
|
||||
let error = malformed.accept(&plan.passes[1], "{}".into()).unwrap_err();
|
||||
let result = review_error_with_usage(&route, &usage, error.to_string());
|
||||
let result = review_error_with_usage(&route, &usage, &[], error.to_string());
|
||||
assert!(!result.success);
|
||||
let metadata = result.metadata.unwrap();
|
||||
assert_eq!(metadata["input_tokens"], 24);
|
||||
@@ -2945,7 +2996,7 @@ mod tests {
|
||||
drift.accept(&plan.passes[0], clean_pass("first")).unwrap();
|
||||
drift.accept(&plan.passes[1], clean_pass("second")).unwrap();
|
||||
let error = drift.finish(&(diff + "drift")).unwrap_err();
|
||||
let result = review_error_with_usage(&route, &usage, error.to_string());
|
||||
let result = review_error_with_usage(&route, &usage, &[], error.to_string());
|
||||
assert!(!result.success);
|
||||
assert_eq!(result.metadata.unwrap()["input_tokens"], 24);
|
||||
}
|
||||
@@ -3163,6 +3214,7 @@ mod tests {
|
||||
reasoning_tokens: Some(7),
|
||||
..Default::default()
|
||||
},
|
||||
&[],
|
||||
);
|
||||
|
||||
assert_eq!(metadata["tool"], "review");
|
||||
|
||||
@@ -9,7 +9,7 @@ use codewhale_models::{ContentBlock, Message, MessageRequest, SystemPrompt, Tool
|
||||
|
||||
use super::review::{ReviewEvidence, ReviewIssue, ReviewOutput};
|
||||
use super::review_hunks::DiffHunks;
|
||||
use super::review_triage::{ReviewStop, SpendRates};
|
||||
use super::review_triage::ReviewStop;
|
||||
use super::spec::{ToolContext, ToolSpec};
|
||||
use crate::core::model_client::ModelClient;
|
||||
|
||||
@@ -457,6 +457,7 @@ fn leaks_tool_call_markup(text: &str) -> bool {
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ReviewCoreResult {
|
||||
pub source_records: Vec<crate::cost_status::RuntimeUsageRecord>,
|
||||
pub output: ReviewOutput,
|
||||
pub raw: String,
|
||||
pub stop_reason: Option<String>,
|
||||
@@ -521,6 +522,8 @@ fn clip(value: &str, max_chars: usize) -> String {
|
||||
pub const MAX_INTENT_CHARS: usize = 8_000;
|
||||
|
||||
pub struct ReviewCoreRequest<'a> {
|
||||
pub accounting_origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
pub accounting_owner: crate::cost_status::UsageSpendOwner,
|
||||
pub diff: &'a str,
|
||||
pub prompt: String,
|
||||
pub intent: Option<ReviewIntent>,
|
||||
@@ -530,7 +533,6 @@ pub struct ReviewCoreRequest<'a> {
|
||||
pub reasoning_effort: Option<String>,
|
||||
pub max_claims: usize,
|
||||
pub max_nits: usize,
|
||||
pub spend_rates: Option<SpendRates>,
|
||||
pub money_cap_usd: Option<f64>,
|
||||
/// Run an independent refutation pass before returning claims. Only worth
|
||||
/// its cost when the budget already bought repository tools, so callers
|
||||
@@ -542,6 +544,8 @@ impl<'a> ReviewCoreRequest<'a> {
|
||||
#[must_use]
|
||||
pub fn new(diff: &'a str, prompt: impl Into<String>) -> Self {
|
||||
Self {
|
||||
accounting_origin: None,
|
||||
accounting_owner: crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
diff,
|
||||
prompt: prompt.into(),
|
||||
intent: None,
|
||||
@@ -551,7 +555,6 @@ impl<'a> ReviewCoreRequest<'a> {
|
||||
reasoning_effort: None,
|
||||
max_claims: 40,
|
||||
max_nits: 3,
|
||||
spend_rates: None,
|
||||
money_cap_usd: None,
|
||||
refute: false,
|
||||
}
|
||||
@@ -709,6 +712,69 @@ fn apply_refutations(output: &mut ReviewOutput, refutations: &[Refutation]) -> u
|
||||
refuted
|
||||
}
|
||||
|
||||
fn source_spend_usd(records: &[crate::cost_status::RuntimeUsageRecord]) -> Option<f64> {
|
||||
records
|
||||
.iter()
|
||||
.try_fold(crate::pricing::CostEstimate::default(), |total, record| {
|
||||
let contribution = crate::cost_status::UsageSourceContribution::for_record(record);
|
||||
(contribution.unpriced == 0).then(|| {
|
||||
total.saturating_add(crate::pricing::CostEstimate {
|
||||
usd: contribution.usd,
|
||||
cny: contribution.cny,
|
||||
})
|
||||
})
|
||||
})
|
||||
.map(|total| total.usd)
|
||||
}
|
||||
|
||||
pub(crate) async fn accounted_model_call(
|
||||
client: &dyn ModelClient,
|
||||
body: MessageRequest,
|
||||
origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
owner: crate::cost_status::UsageSpendOwner,
|
||||
require_usd_price: bool,
|
||||
) -> Result<(
|
||||
codewhale_models::MessageResponse,
|
||||
crate::cost_status::RuntimeUsageRecord,
|
||||
)> {
|
||||
let route = client.effective_route_envelope(&body.model, chrono::Utc::now());
|
||||
if require_usd_price {
|
||||
let audit = route.audit_frozen(&Usage {
|
||||
input_tokens: 1,
|
||||
output_tokens: 1,
|
||||
..Default::default()
|
||||
});
|
||||
anyhow::ensure!(
|
||||
audit.usd_priced
|
||||
|| audit
|
||||
.unpriced_reason
|
||||
.is_some_and(|reason| !reason.counts_toward_money_coverage()),
|
||||
"review money cap requires a verified frozen USD quote before dispatch"
|
||||
);
|
||||
}
|
||||
let source = crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
format!("auxiliary-request:{}", uuid::Uuid::new_v4()),
|
||||
owner,
|
||||
origin,
|
||||
);
|
||||
let observation = source.observe_route(&route)?;
|
||||
source.admit_dispatch(&route).await?;
|
||||
let wire = observation.wire_observations.clone();
|
||||
let response =
|
||||
crate::llm_client::observe_request_retries(Some(observation), client.create_message(body))
|
||||
.await;
|
||||
let record = source.response_record(&route, &response, &wire.drain())?;
|
||||
if record.has_captured_origin() {
|
||||
crate::cost_status::settle_runtime_usage_record(
|
||||
crate::cost_status::scope_token(),
|
||||
None,
|
||||
&record,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
Ok((response?, record))
|
||||
}
|
||||
|
||||
pub async fn run_review_core(
|
||||
client: &dyn ModelClient,
|
||||
model: &str,
|
||||
@@ -729,6 +795,7 @@ pub async fn run_review_core(
|
||||
}],
|
||||
}];
|
||||
let mut usage = Usage::default();
|
||||
let mut source_records = Vec::new();
|
||||
let mut steps = 0;
|
||||
let mut tool_calls = 0;
|
||||
let mut used: Vec<String> = Vec::new();
|
||||
@@ -742,17 +809,21 @@ pub async fn run_review_core(
|
||||
.unwrap_or_else(|| client.effective_max_output_tokens(model));
|
||||
|
||||
while steps < request.budget.max_steps {
|
||||
if let (Some(cap), Some(rates)) = (request.money_cap_usd, request.spend_rates) {
|
||||
let spent = rates.cost_usd(&usage);
|
||||
spend_usd = Some(spent);
|
||||
if steps > 0 && spent >= cap {
|
||||
if let Some(cap) = request.money_cap_usd {
|
||||
spend_usd = source_spend_usd(&source_records);
|
||||
anyhow::ensure!(
|
||||
spend_usd.is_some(),
|
||||
"review money cap cannot be verified from its dispatched source receipts"
|
||||
);
|
||||
if steps > 0 && spend_usd.is_some_and(|spent| spent >= cap) {
|
||||
stop = Some(ReviewStop::MoneyCap);
|
||||
break;
|
||||
}
|
||||
}
|
||||
steps += 1;
|
||||
let response = client
|
||||
.create_message(MessageRequest {
|
||||
let (response, record) = accounted_model_call(
|
||||
client,
|
||||
MessageRequest {
|
||||
model: model.to_string(),
|
||||
messages: messages.clone(),
|
||||
max_tokens,
|
||||
@@ -765,8 +836,13 @@ pub async fn run_review_core(
|
||||
stream: Some(false),
|
||||
temperature: None,
|
||||
top_p: None,
|
||||
})
|
||||
.await?;
|
||||
},
|
||||
request.accounting_origin.clone(),
|
||||
request.accounting_owner,
|
||||
request.money_cap_usd.is_some(),
|
||||
)
|
||||
.await?;
|
||||
source_records.push(record);
|
||||
super::review::add_review_usage(&mut usage, &response.usage);
|
||||
stop_reason = response.stop_reason.clone();
|
||||
let calls: Vec<(String, String, Value)> = response
|
||||
@@ -856,8 +932,12 @@ pub async fn run_review_core(
|
||||
});
|
||||
}
|
||||
|
||||
if let (Some(cap), Some(rates)) = (request.money_cap_usd, request.spend_rates) {
|
||||
spend_usd = Some(rates.cost_usd(&usage));
|
||||
if let Some(cap) = request.money_cap_usd {
|
||||
spend_usd = source_spend_usd(&source_records);
|
||||
anyhow::ensure!(
|
||||
spend_usd.is_some(),
|
||||
"review money cap cannot be verified from its dispatched source receipts"
|
||||
);
|
||||
if spend_usd.is_some_and(|spent| spent >= cap) && stop.is_none() {
|
||||
stop = Some(ReviewStop::MoneyCap);
|
||||
}
|
||||
@@ -896,8 +976,17 @@ pub async fn run_review_core(
|
||||
&& !output.issues.is_empty()
|
||||
&& context.is_some()
|
||||
&& stop.is_none()
|
||||
&& let Some(refutations) =
|
||||
refute_claims(client, model, &request, &output, &tools, &steps).await
|
||||
&& let Some(refutations) = refute_claims(
|
||||
client,
|
||||
model,
|
||||
&request,
|
||||
&output,
|
||||
&tools,
|
||||
&steps,
|
||||
&mut source_records,
|
||||
&mut usage,
|
||||
)
|
||||
.await?
|
||||
{
|
||||
anchors.refuted = apply_refutations(&mut output, &refutations);
|
||||
}
|
||||
@@ -915,6 +1004,13 @@ pub async fn run_review_core(
|
||||
output.summary.push_str(&entry);
|
||||
}
|
||||
}
|
||||
if request.money_cap_usd.is_some() {
|
||||
spend_usd = source_spend_usd(&source_records);
|
||||
anyhow::ensure!(
|
||||
spend_usd.is_some(),
|
||||
"review money cap cannot be verified after refutation"
|
||||
);
|
||||
}
|
||||
// A negative-result list that grows without bound is its own kind of noise.
|
||||
output.ruled_out.truncate(MAX_RULED_OUT);
|
||||
let changed_lines = request
|
||||
@@ -923,6 +1019,7 @@ pub async fn run_review_core(
|
||||
.filter(|line| line.starts_with('+') && !line.starts_with("+++"))
|
||||
.count();
|
||||
Ok(ReviewCoreResult {
|
||||
source_records,
|
||||
output,
|
||||
raw: final_text,
|
||||
stop_reason,
|
||||
@@ -953,8 +1050,12 @@ async fn refute_claims(
|
||||
output: &ReviewOutput,
|
||||
tools: &[Arc<dyn ToolSpec>],
|
||||
steps_used: &u32,
|
||||
) -> Option<Vec<Refutation>> {
|
||||
let context = request.workspace.as_ref().map(ToolContext::new)?;
|
||||
source_records: &mut Vec<crate::cost_status::RuntimeUsageRecord>,
|
||||
usage: &mut Usage,
|
||||
) -> Result<Option<Vec<Refutation>>> {
|
||||
let Some(context) = request.workspace.as_ref().map(ToolContext::new) else {
|
||||
return Ok(None);
|
||||
};
|
||||
let claims: Vec<serde_json::Value> = output
|
||||
.issues
|
||||
.iter()
|
||||
@@ -993,9 +1094,15 @@ async fn refute_claims(
|
||||
let mut steps = *steps_used;
|
||||
let mut final_text = String::new();
|
||||
while steps < request.budget.max_steps.saturating_add(2) {
|
||||
if let Some(cap) = request.money_cap_usd {
|
||||
if source_spend_usd(source_records).is_none_or(|spent| spent >= cap) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
steps += 1;
|
||||
let response = client
|
||||
.create_message(MessageRequest {
|
||||
let (response, record) = accounted_model_call(
|
||||
client,
|
||||
MessageRequest {
|
||||
model: model.to_string(),
|
||||
messages: messages.clone(),
|
||||
max_tokens: request
|
||||
@@ -1010,9 +1117,14 @@ async fn refute_claims(
|
||||
stream: Some(false),
|
||||
temperature: None,
|
||||
top_p: None,
|
||||
})
|
||||
.await
|
||||
.ok()?;
|
||||
},
|
||||
request.accounting_origin.clone(),
|
||||
request.accounting_owner,
|
||||
request.money_cap_usd.is_some(),
|
||||
)
|
||||
.await?;
|
||||
source_records.push(record);
|
||||
super::review::add_review_usage(usage, &response.usage);
|
||||
final_text.clear();
|
||||
let calls: Vec<(String, String, Value)> = response
|
||||
.content
|
||||
@@ -1064,7 +1176,7 @@ async fn refute_claims(
|
||||
});
|
||||
}
|
||||
let refutations = parse_refutations(&final_text, output.issues.len());
|
||||
(!refutations.is_empty()).then_some(refutations)
|
||||
Ok((!refutations.is_empty()).then_some(refutations))
|
||||
}
|
||||
|
||||
/// The lines this change actually added, as `path:line` on the post-image.
|
||||
|
||||
@@ -1074,13 +1074,13 @@ mod tests {
|
||||
reasoning_replay_tokens: Some(7),
|
||||
..Default::default()
|
||||
};
|
||||
let record = crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: "rlm:test:request:0".to_string(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
let record = crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
"rlm:test:request:0".to_string(),
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: route.clone(),
|
||||
usage: usage.clone(),
|
||||
},
|
||||
};
|
||||
);
|
||||
let drop_record = crate::cost_status::RuntimeUsageDropRecord {
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason::default(),
|
||||
source_id: "rlm:test:request:1".to_string(),
|
||||
|
||||
@@ -97,6 +97,7 @@ pub trait DynamicToolExecutor: Send + Sync {
|
||||
/// attached.
|
||||
#[derive(Clone)]
|
||||
pub struct RuntimeToolServices {
|
||||
pub accounting_origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
pub shell_manager: Option<SharedShellManager>,
|
||||
/// True only for the real headless exec host after it has established the
|
||||
/// explicit authority required to transfer `persist:true` services.
|
||||
@@ -128,6 +129,7 @@ pub struct RuntimeToolServices {
|
||||
impl Default for RuntimeToolServices {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
accounting_origin: None,
|
||||
shell_manager: None,
|
||||
persist_services_enabled: false,
|
||||
task_manager: None,
|
||||
@@ -148,6 +150,7 @@ impl Default for RuntimeToolServices {
|
||||
impl std::fmt::Debug for RuntimeToolServices {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("RuntimeToolServices")
|
||||
.field("accounting_origin", &self.accounting_origin.is_some())
|
||||
.field("shell_manager", &self.shell_manager.is_some())
|
||||
.field("persist_services_enabled", &self.persist_services_enabled)
|
||||
.field("task_manager", &self.task_manager.is_some())
|
||||
@@ -579,6 +582,7 @@ pub struct ToolExecutionState {
|
||||
/// Actual Core caller admission for per-round Python RPCs. Kernels do not
|
||||
/// retain it; unattached tool contexts cannot make model calls.
|
||||
pub rlm_caller: Option<Arc<crate::core::engine::rlm_host::CapturedRlmCaller>>,
|
||||
pub accounting_origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
/// Effective session/ancestor tool ceiling, carried to MCP dispatch and runtime registration.
|
||||
pub disallowed_tools: Vec<String>,
|
||||
/// Trusted local host narrowing; inherited by every alias/nested context.
|
||||
@@ -780,6 +784,7 @@ impl ToolContext {
|
||||
workspace,
|
||||
execution: Box::new(ToolExecutionState {
|
||||
rlm_caller: None,
|
||||
accounting_origin: None,
|
||||
disallowed_tools: Vec::new(),
|
||||
acp_host: None,
|
||||
child_host: None,
|
||||
|
||||
@@ -112,56 +112,40 @@ pub struct EmissionGuard {
|
||||
|
||||
/// Accounting ownership captured while the originating turn is still live.
|
||||
///
|
||||
/// Runtime turns retain their synchronous durable sink through the lease;
|
||||
/// ordinary interactive turns fall back to the exact session cost generation
|
||||
/// captured here. Neither path can spill into a later session.
|
||||
/// Runtime turns retain their durable sink through the lease; otherwise the
|
||||
/// already captured native Session origin remains authoritative. Neither path
|
||||
/// can spill into a later session.
|
||||
#[derive(Debug)]
|
||||
pub struct AdvisorUsageContext {
|
||||
cost_scope: crate::cost_status::CostScopeToken,
|
||||
runtime_usage_lease: Option<crate::cost_status::RuntimeUsageLease>,
|
||||
source: crate::cost_status::NativeUsageSource,
|
||||
}
|
||||
|
||||
impl AdvisorUsageContext {
|
||||
#[must_use]
|
||||
pub fn capture(runtime_owner: Option<&str>) -> Self {
|
||||
pub fn capture(
|
||||
runtime_owner: Option<&str>,
|
||||
fallback_origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
) -> Self {
|
||||
Self {
|
||||
cost_scope: crate::cost_status::scope_token(),
|
||||
source: crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
format!("advisor:{}", uuid::Uuid::new_v4()),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
match runtime_owner {
|
||||
Some(owner) => crate::cost_status::NativeUsageOrigin::runtime(
|
||||
crate::cost_status::scope_token(),
|
||||
owner,
|
||||
)
|
||||
.ok(),
|
||||
None => fallback_origin,
|
||||
},
|
||||
),
|
||||
runtime_usage_lease: runtime_owner
|
||||
.and_then(crate::cost_status::acquire_runtime_usage_lease),
|
||||
}
|
||||
}
|
||||
|
||||
fn report(
|
||||
&self,
|
||||
source_id: &str,
|
||||
route: &crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &codewhale_models::Usage,
|
||||
) {
|
||||
crate::cost_status::report_effective_route_for_runtime(
|
||||
self.cost_scope,
|
||||
self.runtime_usage_lease
|
||||
.as_ref()
|
||||
.map(crate::cost_status::RuntimeUsageLease::owner),
|
||||
source_id,
|
||||
route,
|
||||
usage,
|
||||
);
|
||||
}
|
||||
|
||||
fn report_unreceipted(
|
||||
&self,
|
||||
source_id: &str,
|
||||
route: &crate::cost_status::EffectiveRouteEnvelope,
|
||||
) {
|
||||
crate::cost_status::report_unreceipted_provider_success(
|
||||
self.cost_scope,
|
||||
self.runtime_usage_lease
|
||||
.as_ref()
|
||||
.map(crate::cost_status::RuntimeUsageLease::owner),
|
||||
source_id,
|
||||
route,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
impl EmissionGuard {
|
||||
@@ -398,26 +382,53 @@ pub async fn run_advisor_for_turn(
|
||||
top_p: None,
|
||||
};
|
||||
|
||||
let response = match client.create_message(request).await {
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
tracing::warn!(target: "advisor", "advisor LLM call failed for turn {turn_id}: {e}");
|
||||
let observation = match usage_context.source.observe_route(&route) {
|
||||
Ok(observation) => observation,
|
||||
Err(error) => {
|
||||
tracing::warn!(%error, "advisor accounting admission failed");
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
// A decoded provider response is billable even when its partial/empty
|
||||
// content is rejected below or the emission guard suppresses a duplicate.
|
||||
let usage_source_id = format!("advisor:{turn_id}:provider-response:0");
|
||||
if response.usage == codewhale_models::Usage::default() {
|
||||
usage_context.report_unreceipted(&usage_source_id, &route);
|
||||
tracing::warn!(
|
||||
target: "advisor",
|
||||
"advisor provider response omitted usage for turn {turn_id}; cost coverage is unknown"
|
||||
);
|
||||
} else {
|
||||
usage_context.report(&usage_source_id, &route, &response.usage);
|
||||
if let Err(error) = usage_context.source.admit_dispatch(&route).await {
|
||||
tracing::warn!(%error, "advisor source admission failed");
|
||||
return;
|
||||
}
|
||||
let wire = observation.wire_observations.clone();
|
||||
let response = crate::llm_client::observe_request_retries(
|
||||
Some(observation),
|
||||
client.create_message(request),
|
||||
)
|
||||
.await;
|
||||
let record = match usage_context
|
||||
.source
|
||||
.response_record(&route, &response, &wire.drain())
|
||||
{
|
||||
Ok(record) => record,
|
||||
Err(error) => {
|
||||
tracing::warn!(%error, "advisor usage source was refused");
|
||||
return;
|
||||
}
|
||||
};
|
||||
if let Err(error) = crate::cost_status::settle_runtime_usage_record(
|
||||
usage_context.cost_scope,
|
||||
usage_context
|
||||
.runtime_usage_lease
|
||||
.as_ref()
|
||||
.map(crate::cost_status::RuntimeUsageLease::owner),
|
||||
&record,
|
||||
)
|
||||
.await
|
||||
{
|
||||
tracing::warn!(%error, "advisor usage remains unacknowledged");
|
||||
return;
|
||||
}
|
||||
let response = match response {
|
||||
Ok(response) => response,
|
||||
Err(error) => {
|
||||
tracing::warn!(%error, "advisor request failed");
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
if codewhale_models::is_incomplete_stop_reason(response.stop_reason.as_deref()) {
|
||||
tracing::warn!(
|
||||
@@ -1057,6 +1068,16 @@ mod tests {
|
||||
}
|
||||
let guard = std::sync::Arc::new(tokio::sync::Mutex::new(emission_guard));
|
||||
let (tx, mut rx) = mpsc::channel(1);
|
||||
let directory = tempfile::tempdir().unwrap();
|
||||
let manager =
|
||||
crate::session_manager::SessionManager::new(directory.path().to_path_buf()).unwrap();
|
||||
crate::usage_persistence::register_persistent_interactive_runtime_usage_sink_for_test(
|
||||
"advisor-loopback-fixture",
|
||||
crate::cost_status::scope_token(),
|
||||
"advisor-fixture",
|
||||
"advisor-turn",
|
||||
&manager,
|
||||
);
|
||||
run_advisor_for_turn(
|
||||
"advisor-turn".to_string(),
|
||||
make_messages_with_n_tool_calls(1),
|
||||
@@ -1070,12 +1091,31 @@ mod tests {
|
||||
client,
|
||||
route_config,
|
||||
"deepseek-chat".to_string(),
|
||||
AdvisorUsageContext::capture(None),
|
||||
AdvisorUsageContext::capture(Some("advisor-loopback-fixture"), None),
|
||||
guard,
|
||||
tx,
|
||||
)
|
||||
.await;
|
||||
(crate::cost_status::drain(), rx.try_recv().ok())
|
||||
crate::cost_status::finish_runtime_usage_owner("advisor-loopback-fixture");
|
||||
let mut pending = crate::cost_status::drain();
|
||||
let mut cost = crate::session_manager::SessionCostSnapshot::default();
|
||||
for record in &pending.source_records {
|
||||
cost.project_usage_record(record).unwrap();
|
||||
}
|
||||
pending.estimate = cost.total_estimate();
|
||||
pending.priced_turns = cost.priced_turns;
|
||||
pending.unpriced_turns = cost.unpriced_turns;
|
||||
pending.cny_priced_turns = cost.cny_priced_turns;
|
||||
pending.cny_unpriced_turns = cost.cny_unpriced_turns;
|
||||
if cost
|
||||
.unpriced_reasons
|
||||
.contains("provider_success_missing_usage")
|
||||
{
|
||||
pending
|
||||
.unpriced_reasons
|
||||
.insert("provider_success_missing_usage");
|
||||
}
|
||||
(pending, rx.try_recv().ok())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -24,116 +24,57 @@ impl ChildAccountingProjection {
|
||||
}
|
||||
pub fn publish(
|
||||
&self,
|
||||
source: &str,
|
||||
route: &crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &Usage,
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason,
|
||||
) -> Option<u64> {
|
||||
let known = usage_has_reported_data(usage);
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
) -> std::io::Result<Option<u64>> {
|
||||
let owner = self
|
||||
.runtime_usage_lease
|
||||
.as_ref()
|
||||
.map(crate::cost_status::RuntimeUsageLease::owner);
|
||||
if known {
|
||||
match owner {
|
||||
Some(owner) => crate::cost_status::report_effective_route_for_runtime(
|
||||
self.origin.cost_scope,
|
||||
Some(owner),
|
||||
source,
|
||||
route,
|
||||
usage,
|
||||
),
|
||||
None => self
|
||||
.origin
|
||||
.report_ownerless_usage(&self.owner, source, route, usage),
|
||||
}
|
||||
if let Some(mailbox) = &self.mailbox {
|
||||
let _ = mailbox.send(MailboxMessage::token_usage(
|
||||
&self.owner,
|
||||
source,
|
||||
route.clone(),
|
||||
usage.clone(),
|
||||
));
|
||||
}
|
||||
} else {
|
||||
match owner {
|
||||
Some(owner) => crate::cost_status::report_missing_runtime_usage(
|
||||
self.origin.cost_scope,
|
||||
Some(owner),
|
||||
source,
|
||||
route,
|
||||
reason,
|
||||
),
|
||||
None => crate::usage_persistence::report_missing_usage_for_interactive_origin(
|
||||
self.origin.cost_scope,
|
||||
&self.origin.session_id,
|
||||
&self.owner,
|
||||
source,
|
||||
route,
|
||||
reason,
|
||||
),
|
||||
}
|
||||
}
|
||||
known
|
||||
.then(|| priced_usd_microusd(&route.audit(usage)))
|
||||
.flatten()
|
||||
}
|
||||
/// Finish only the worker projection of an already-published receipt.
|
||||
/// The caller supplies its held Engine scheduler; no runtime is created.
|
||||
pub fn recover_settled(
|
||||
&self,
|
||||
scheduler: &tokio::runtime::Handle,
|
||||
source: String,
|
||||
route: crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: Usage,
|
||||
priced: Option<u64>,
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason,
|
||||
) {
|
||||
let manager = self.manager.clone();
|
||||
let owner = self.owner.clone();
|
||||
scheduler.spawn(async move {
|
||||
Self::project_worker_receipt(&manager, &owner, &source, &route, &usage, priced, reason)
|
||||
.await;
|
||||
crate::cost_status::report_runtime_usage_record(self.origin.cost_scope, owner, record)?;
|
||||
let reported = record.settlement.as_ref().is_some_and(|revision| {
|
||||
revision.coverage == crate::llm_client::ProviderUsageCoverage::Reported
|
||||
|| (revision.coverage == crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
&& record.usage.usage != Usage::default())
|
||||
});
|
||||
if reported && let Some(mailbox) = &self.mailbox {
|
||||
let _ = mailbox.send(MailboxMessage::usage_record(&self.owner, record.clone()));
|
||||
}
|
||||
Ok(reported
|
||||
.then(|| priced_usd_microusd(&record.usage.route.audit_frozen(&record.usage.usage)))
|
||||
.flatten())
|
||||
}
|
||||
|
||||
async fn publish_async(
|
||||
&self,
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
) -> std::io::Result<Option<u64>> {
|
||||
let accounting = self.clone();
|
||||
let record = record.clone();
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
let origin = crate::cost_status::test_cost_scope_id();
|
||||
codewhale_runtime::runtime_control::owner_work(move || {
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
let _origin = crate::cost_status::bind_test_cost_scope(origin);
|
||||
accounting.publish(&record).map_err(anyhow::Error::from)
|
||||
})
|
||||
.await
|
||||
.map_err(std::io::Error::other)
|
||||
}
|
||||
|
||||
pub async fn project_settled(
|
||||
&self,
|
||||
source: &str,
|
||||
route: &crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &Usage,
|
||||
priced: Option<u64>,
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason,
|
||||
) {
|
||||
Self::project_worker_receipt(
|
||||
&self.manager,
|
||||
&self.owner,
|
||||
source,
|
||||
route,
|
||||
usage,
|
||||
priced,
|
||||
reason,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
async fn project_worker_receipt(
|
||||
manager: &SharedSubAgentManager,
|
||||
owner: &str,
|
||||
source: &str,
|
||||
route: &crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &Usage,
|
||||
priced: Option<u64>,
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason,
|
||||
) {
|
||||
let mut manager = manager.write().await;
|
||||
if usage_has_reported_data(usage) {
|
||||
manager.record_worker_routed_usage(owner, source, route, usage, priced);
|
||||
} else {
|
||||
manager.record_worker_missing_usage(
|
||||
owner,
|
||||
source,
|
||||
crate::cost_status::MissingUsageCoverage::for_route(route, reason),
|
||||
);
|
||||
}
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
) -> std::io::Result<()> {
|
||||
let manager = self.manager.clone();
|
||||
let owner = self.owner.clone();
|
||||
let record = record.clone();
|
||||
codewhale_runtime::runtime_control::owner_work(move || {
|
||||
let mut manager = manager.blocking_write();
|
||||
manager.record_worker_source(&owner, &record)?;
|
||||
Ok(())
|
||||
})
|
||||
.await
|
||||
.map_err(std::io::Error::other)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -229,33 +170,17 @@ impl ChildAuthority {
|
||||
}
|
||||
pub async fn settle_response(
|
||||
&self,
|
||||
source_id: &str,
|
||||
route: crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &Usage,
|
||||
) {
|
||||
record_provider_response_usage(
|
||||
&self.runtime,
|
||||
&self.owner_agent_id,
|
||||
source_id,
|
||||
route,
|
||||
usage,
|
||||
)
|
||||
.await;
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
) -> std::io::Result<()> {
|
||||
let accounting = self.accounting_projection();
|
||||
accounting.publish_async(record).await?;
|
||||
accounting.project_settled(record).await
|
||||
}
|
||||
pub async fn project_settled_response(
|
||||
&self,
|
||||
source_id: &str,
|
||||
route: &crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &Usage,
|
||||
) {
|
||||
let priced = usage_has_reported_data(usage)
|
||||
.then(|| priced_usd_microusd(&route.audit(usage)))
|
||||
.flatten();
|
||||
self.runtime
|
||||
.manager
|
||||
.write()
|
||||
.await
|
||||
.record_worker_routed_usage(&self.owner_agent_id, source_id, route, usage, priced);
|
||||
record: &crate::cost_status::RuntimeUsageRecord,
|
||||
) -> std::io::Result<()> {
|
||||
self.accounting_projection().project_settled(record).await
|
||||
}
|
||||
pub fn new_execution_id(&self) -> String {
|
||||
new_child_execution_id(&self.owner_agent_id)
|
||||
@@ -967,7 +892,6 @@ pub struct ChildJob {
|
||||
pub fork_context: bool,
|
||||
parking: Option<Arc<std::sync::atomic::AtomicBool>>,
|
||||
artifact: Arc<tokio::sync::Mutex<Option<SubAgentTranscriptArtifactWriter>>>,
|
||||
scheduler: tokio::runtime::Handle,
|
||||
pub requests: std::sync::atomic::AtomicU32,
|
||||
logical_steps: std::sync::atomic::AtomicU32,
|
||||
initial_submission_assigned: std::sync::atomic::AtomicBool,
|
||||
@@ -995,6 +919,8 @@ impl ChildJob {
|
||||
)
|
||||
.await?;
|
||||
let selected_model = authority.runtime.model.clone();
|
||||
let _ = tokio::runtime::Handle::try_current()
|
||||
.map_err(|_| anyhow!("child admission requires the existing Engine scheduler"))?;
|
||||
Ok(Arc::new(Self {
|
||||
authority,
|
||||
assignment,
|
||||
@@ -1004,8 +930,6 @@ impl ChildJob {
|
||||
fork_context,
|
||||
parking,
|
||||
artifact: Arc::new(tokio::sync::Mutex::new(Some(artifact))),
|
||||
scheduler: tokio::runtime::Handle::try_current()
|
||||
.map_err(|_| anyhow!("child admission requires the existing Engine scheduler"))?,
|
||||
requests: std::sync::atomic::AtomicU32::new(0),
|
||||
logical_steps: std::sync::atomic::AtomicU32::new(0),
|
||||
initial_submission_assigned: std::sync::atomic::AtomicBool::new(false),
|
||||
@@ -1242,8 +1166,12 @@ impl ChildJob {
|
||||
self.requests
|
||||
.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
|
||||
ChildDispatchedRequest {
|
||||
accounting_source: crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
source,
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
self.authority.runtime.context.accounting_origin.clone(),
|
||||
),
|
||||
job: self.clone(),
|
||||
source,
|
||||
route,
|
||||
settled: None,
|
||||
refused: false,
|
||||
@@ -1516,18 +1444,14 @@ impl ChildJob {
|
||||
}
|
||||
}
|
||||
|
||||
/// Exact primary dispatch ownership. Dropping an actually invoked request
|
||||
/// records ambiguity under its captured origin; an unopened/queued turn never
|
||||
/// constructs this guard. Billing is synchronous before worker projection.
|
||||
/// Exact primary dispatch ownership with durable reservation before invocation.
|
||||
/// Dropping retains the captured origin's Unknown reservation and any unacknowledged
|
||||
/// final in the existing journal. Billing precedes worker projection.
|
||||
pub struct ChildDispatchedRequest {
|
||||
accounting_source: crate::cost_status::NativeUsageSource,
|
||||
job: Arc<ChildJob>,
|
||||
source: String,
|
||||
route: crate::cost_status::EffectiveRouteEnvelope,
|
||||
settled: Option<(
|
||||
Usage,
|
||||
Option<u64>,
|
||||
crate::cost_status::RuntimeUsageMissingReason,
|
||||
)>,
|
||||
settled: Option<crate::cost_status::RuntimeUsageRecord>,
|
||||
refused: bool,
|
||||
projected: bool,
|
||||
}
|
||||
@@ -1550,11 +1474,74 @@ pub fn provider_request_refusal_proven(error: &anyhow::Error) -> bool {
|
||||
)
|
||||
}
|
||||
impl ChildDispatchedRequest {
|
||||
pub async fn settle_open_error(&mut self, error: &anyhow::Error) {
|
||||
self.refused = provider_request_refusal_proven(error);
|
||||
if !self.refused {
|
||||
self.settle(&Usage::default(), false).await;
|
||||
pub(crate) fn settlement_acknowledged(&self) -> bool {
|
||||
self.refused || self.projected
|
||||
}
|
||||
pub(crate) async fn bind_dispatch(
|
||||
&mut self,
|
||||
observation: &crate::llm_client::RequestRetryObservation,
|
||||
) -> std::io::Result<()> {
|
||||
#[cfg(not(any(test, feature = "test-support")))]
|
||||
if !self.accounting_source.has_captured_origin() {
|
||||
return Err(std::io::Error::new(
|
||||
std::io::ErrorKind::PermissionDenied,
|
||||
"child provider call has no captured accounting origin",
|
||||
));
|
||||
}
|
||||
self.accounting_source
|
||||
.bind_dispatch(&self.route, observation)?;
|
||||
if let Some(record) = self.accounting_source.admit_dispatch(&self.route).await? {
|
||||
self.job
|
||||
.authority
|
||||
.accounting_projection()
|
||||
.project_settled(&record)
|
||||
.await?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
pub async fn settle_open_error(&mut self, error: &anyhow::Error) {
|
||||
let wire = self.accounting_source.observed_wire();
|
||||
let coverage = if wire.last_transmission.is_some() {
|
||||
wire.usage_coverage(false, false)
|
||||
} else {
|
||||
match error.downcast_ref::<LlmError>().map(LlmError::transmission) {
|
||||
Some(crate::llm_client::RequestTransmission::NotDispatched) => {
|
||||
crate::llm_client::ProviderUsageCoverage::NotDispatched
|
||||
}
|
||||
Some(crate::llm_client::RequestTransmission::Rejected) => {
|
||||
crate::llm_client::ProviderUsageCoverage::Rejected
|
||||
}
|
||||
_ => crate::llm_client::ProviderUsageCoverage::Missing,
|
||||
}
|
||||
};
|
||||
let record = match self.accounting_source.prepare_observed(
|
||||
&self.route,
|
||||
&Usage::default(),
|
||||
coverage,
|
||||
crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown,
|
||||
&wire,
|
||||
) {
|
||||
Ok(record) => record,
|
||||
Err(error) => {
|
||||
tracing::warn!(%error, "child refusal receipt was not admitted");
|
||||
return;
|
||||
}
|
||||
};
|
||||
let accounting = self.job.authority.accounting_projection();
|
||||
let published = accounting.publish_async(&record).await;
|
||||
let projected = if published.is_ok() {
|
||||
accounting.project_settled(&record).await
|
||||
} else {
|
||||
Err(std::io::Error::other("child refusal persistence failed"))
|
||||
};
|
||||
self.settled = Some(record);
|
||||
self.projected = projected.is_ok();
|
||||
self.refused = self.projected
|
||||
&& matches!(
|
||||
coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Rejected
|
||||
| crate::llm_client::ProviderUsageCoverage::NotDispatched
|
||||
);
|
||||
}
|
||||
pub async fn settle(&mut self, usage: &Usage, complete: bool) {
|
||||
let reason = if complete {
|
||||
@@ -1562,59 +1549,38 @@ impl ChildDispatchedRequest {
|
||||
} else {
|
||||
crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown
|
||||
};
|
||||
let priced = report_provider_response_usage_origin(
|
||||
&self.job.authority.runtime,
|
||||
&self.job.authority.owner_agent_id,
|
||||
&self.source,
|
||||
let coverage = if complete && usage_has_reported_data(usage) {
|
||||
crate::llm_client::ProviderUsageCoverage::Reported
|
||||
} else if usage_has_reported_data(usage) {
|
||||
crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
} else {
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
};
|
||||
let record = match self.accounting_source.prepare_observed(
|
||||
&self.route,
|
||||
usage,
|
||||
coverage,
|
||||
reason,
|
||||
);
|
||||
self.settled = Some((usage.clone(), priced, reason));
|
||||
self.project(usage, priced, reason).await;
|
||||
self.projected = true;
|
||||
}
|
||||
async fn project(
|
||||
&self,
|
||||
usage: &Usage,
|
||||
priced: Option<u64>,
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason,
|
||||
) {
|
||||
self.job
|
||||
.authority
|
||||
.accounting_projection()
|
||||
.project_settled(&self.source, &self.route, usage, priced, reason)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
impl Drop for ChildDispatchedRequest {
|
||||
fn drop(&mut self) {
|
||||
if self.refused || self.projected {
|
||||
return;
|
||||
&self.accounting_source.observed_wire(),
|
||||
) {
|
||||
Ok(record) => record,
|
||||
Err(error) => {
|
||||
tracing::warn!(%error, "child usage predecessor remains unsettled");
|
||||
return;
|
||||
}
|
||||
};
|
||||
let accounting = self.job.authority.accounting_projection();
|
||||
if let Err(error) = accounting.publish_async(&record).await {
|
||||
tracing::warn!(%error, "child provider usage remains unacknowledged");
|
||||
}
|
||||
let (usage, priced, reason) = self.settled.clone().unwrap_or_else(|| {
|
||||
let reason = crate::cost_status::RuntimeUsageMissingReason::RequestOutcomeUnknown;
|
||||
report_provider_response_usage_origin(
|
||||
&self.job.authority.runtime,
|
||||
&self.job.authority.owner_agent_id,
|
||||
&self.source,
|
||||
&self.route,
|
||||
&Usage::default(),
|
||||
reason,
|
||||
);
|
||||
(Usage::default(), None, reason)
|
||||
});
|
||||
// If the actor was aborted while waiting for this projection, the
|
||||
// exact already-settled cost is not billed again. The existing held
|
||||
// scheduler retires only worker metadata under the same source id.
|
||||
self.job.authority.accounting_projection().recover_settled(
|
||||
&self.job.scheduler,
|
||||
self.source.clone(),
|
||||
self.route.clone(),
|
||||
usage,
|
||||
priced,
|
||||
reason,
|
||||
);
|
||||
self.settled = Some(record.clone());
|
||||
self.projected = match accounting.project_settled(&record).await {
|
||||
Ok(()) => record.is_acknowledged(),
|
||||
Err(error) => {
|
||||
tracing::warn!(%error, "worker usage publication failed");
|
||||
false
|
||||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -79,6 +79,8 @@ pub enum MailboxMessage {
|
||||
/// Incremental token usage from a sub-agent's API call.
|
||||
/// Published after each turn so the parent's cost counter updates live.
|
||||
TokenUsage {
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
record: Option<crate::cost_status::RuntimeUsageRecord>,
|
||||
agent_id: String,
|
||||
/// Stable identity of the provider response. Runtime accounting uses
|
||||
/// this across direct durability, mailbox replay, and restart dedupe.
|
||||
@@ -134,6 +136,19 @@ impl MailboxMessage {
|
||||
}
|
||||
}
|
||||
|
||||
pub fn usage_record(
|
||||
agent_id: impl Into<String>,
|
||||
record: crate::cost_status::RuntimeUsageRecord,
|
||||
) -> Self {
|
||||
Self::TokenUsage {
|
||||
agent_id: agent_id.into(),
|
||||
source_id: record.source_id.clone(),
|
||||
route: Box::new(record.usage.route.clone()),
|
||||
usage: record.usage.usage.clone(),
|
||||
record: Some(record),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn token_usage(
|
||||
agent_id: impl Into<String>,
|
||||
source_id: impl Into<String>,
|
||||
@@ -141,6 +156,7 @@ impl MailboxMessage {
|
||||
usage: Usage,
|
||||
) -> Self {
|
||||
Self::TokenUsage {
|
||||
record: None,
|
||||
agent_id: agent_id.into(),
|
||||
source_id: source_id.into(),
|
||||
route: Box::new(route),
|
||||
@@ -618,6 +634,7 @@ mod tests {
|
||||
),
|
||||
(
|
||||
MailboxMessage::TokenUsage {
|
||||
record: None,
|
||||
agent_id: "a9".into(),
|
||||
source_id: "response-a9".into(),
|
||||
route: Box::new(test_route(ProviderKind::Deepseek, "deepseek-v4-flash")),
|
||||
|
||||
@@ -986,6 +986,9 @@ pub struct AgentWorkerRecord {
|
||||
/// retried delivery idempotent in both projections after reload.
|
||||
#[serde(default, skip_serializing_if = "BTreeSet::is_empty")]
|
||||
pub usage_source_fingerprints: BTreeSet<String>,
|
||||
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
|
||||
pub source_usage_records:
|
||||
BTreeMap<String, (crate::cost_status::RuntimeUsageRecord, Option<u64>)>,
|
||||
#[serde(
|
||||
default,
|
||||
skip_serializing_if = "BTreeMap::is_empty",
|
||||
@@ -1109,6 +1112,7 @@ impl AgentWorkerRecord {
|
||||
artifacts,
|
||||
usage: default_agent_run_usage(),
|
||||
usage_source_fingerprints: BTreeSet::new(),
|
||||
source_usage_records: BTreeMap::new(),
|
||||
missing_usage_sources: BTreeMap::new(),
|
||||
missing_usage_overflowed: false,
|
||||
legacy_unreported_usage: false,
|
||||
@@ -1248,7 +1252,7 @@ fn usage_has_reported_data(usage: &Usage) -> bool {
|
||||
/// Convert an authoritative USD audit into the workflow IR's integer
|
||||
/// microdollar receipt. Route coverage stays on the cost-status path; this
|
||||
/// narrow projection deliberately preserves only a priced subtotal.
|
||||
fn priced_usd_microusd(audit: &crate::pricing::TurnCostAudit) -> Option<u64> {
|
||||
pub(crate) fn priced_usd_microusd(audit: &crate::pricing::TurnCostAudit) -> Option<u64> {
|
||||
if !audit.usd_priced {
|
||||
return None;
|
||||
}
|
||||
@@ -1293,28 +1297,6 @@ impl SubAgentAccountingOrigin {
|
||||
session_id: context.state_namespace.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Settle one reported provider response that has no runtime owner. A
|
||||
/// live origin scope charges the interactive pool exactly once per source
|
||||
/// id; a retired one appends the exact frozen receipt to the origin
|
||||
/// session's sidecar, labelled by the child run that dispatched it. Either
|
||||
/// way the replacement session never sees it.
|
||||
fn report_ownerless_usage(
|
||||
&self,
|
||||
agent_id: &str,
|
||||
source_id: &str,
|
||||
route: &crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &Usage,
|
||||
) {
|
||||
crate::usage_persistence::report_effective_route_for_interactive_origin(
|
||||
self.cost_scope,
|
||||
&self.session_id,
|
||||
agent_id,
|
||||
source_id,
|
||||
route,
|
||||
usage,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Publish one child provider response into every projection that owns it.
|
||||
@@ -1322,6 +1304,7 @@ impl SubAgentAccountingOrigin {
|
||||
/// and the durable worker record hash it with the same canonical function.
|
||||
/// Accounting ownership is the runtime's dispatch-time
|
||||
/// [`SubAgentAccountingOrigin`]; nothing here consults the live cost scope.
|
||||
#[cfg(test)]
|
||||
async fn record_provider_response_usage(
|
||||
runtime: &SubAgentRuntime,
|
||||
agent_id: &str,
|
||||
@@ -1329,36 +1312,45 @@ async fn record_provider_response_usage(
|
||||
route: crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &Usage,
|
||||
) {
|
||||
let priced = report_provider_response_usage_origin(
|
||||
runtime,
|
||||
agent_id,
|
||||
source_id,
|
||||
&route,
|
||||
usage,
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage,
|
||||
);
|
||||
runtime
|
||||
let origin = runtime
|
||||
.context
|
||||
.accounting_origin
|
||||
.clone()
|
||||
.expect("fixture origin captured before dispatch");
|
||||
let prior = runtime
|
||||
.manager
|
||||
.write()
|
||||
.read()
|
||||
.await
|
||||
.record_worker_routed_usage(agent_id, source_id, &route, usage, priced);
|
||||
.worker_records
|
||||
.get(agent_id)
|
||||
.and_then(|worker| {
|
||||
worker
|
||||
.source_usage_records
|
||||
.get(&crate::cost_status::usage_source_fingerprint(source_id))
|
||||
})
|
||||
.map(|record| record.0.clone());
|
||||
let source = if let Some(prior) = prior {
|
||||
crate::cost_status::NativeUsageSource::resume_admitted(&prior, Some(origin))
|
||||
.expect("fixture native predecessor")
|
||||
} else {
|
||||
crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
source_id,
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
Some(origin),
|
||||
)
|
||||
};
|
||||
let record = source
|
||||
.reported(&route, usage)
|
||||
.expect("fixture usage revision");
|
||||
let accounting = engine::ChildAccountingProjection::capture(runtime, agent_id);
|
||||
accounting
|
||||
.publish(&record)
|
||||
.expect("fixture durable origin publication");
|
||||
accounting
|
||||
.project_settled(&record)
|
||||
.await
|
||||
.expect("fixture worker publication");
|
||||
}
|
||||
fn report_provider_response_usage_origin(
|
||||
runtime: &SubAgentRuntime,
|
||||
agent_id: &str,
|
||||
source_id: &str,
|
||||
route: &crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &Usage,
|
||||
missing_reason: crate::cost_status::RuntimeUsageMissingReason,
|
||||
) -> Option<u64> {
|
||||
engine::ChildAccountingProjection::capture(runtime, agent_id).publish(
|
||||
source_id,
|
||||
route,
|
||||
usage,
|
||||
missing_reason,
|
||||
)
|
||||
}
|
||||
|
||||
fn new_child_execution_id(agent_id: &str) -> String {
|
||||
format!("agent:{agent_id}:approval:{}", Uuid::new_v4())
|
||||
}
|
||||
@@ -2254,15 +2246,6 @@ pub struct SubAgentCheckpoint {
|
||||
/// This checkpoint was written because the parent's turn ended while the
|
||||
/// child was still working — not because the child asked anyone anything
|
||||
/// (#5906).
|
||||
///
|
||||
/// Both cases land on `AgentWorkerStatus::WaitingForUser` (the projection
|
||||
/// keys off `continuable`), and coordination has to tell them apart: a
|
||||
/// child that asked a question keeps its write claim, because a user can
|
||||
/// still answer it and the child will write again. A parked child is
|
||||
/// waiting for nothing — it is resumed as a *new* agent through
|
||||
/// `followup`, which registers a claim of its own — so its claim must
|
||||
/// stop gating peers the moment it parks. Carried on the checkpoint rather
|
||||
/// than sniffed out of the reason string, which would be a guess.
|
||||
#[serde(default, skip_serializing_if = "is_false")]
|
||||
pub parked_at_turn_end: bool,
|
||||
}
|
||||
@@ -3249,6 +3232,22 @@ impl SubAgentRuntime {
|
||||
// Accounting ownership is fixed here, at dispatch, before any child
|
||||
// provider request exists; see `SubAgentAccountingOrigin`.
|
||||
let accounting_origin = SubAgentAccountingOrigin::capture(&context);
|
||||
#[cfg(test)]
|
||||
let context = {
|
||||
let mut context = context;
|
||||
if context.accounting_origin.is_none() {
|
||||
context.accounting_origin = crate::session_manager::default_sessions_dir()
|
||||
.ok()
|
||||
.map(|dir| {
|
||||
crate::cost_status::NativeUsageOrigin::session(
|
||||
accounting_origin.cost_scope,
|
||||
dir,
|
||||
accounting_origin.session_id.clone(),
|
||||
)
|
||||
});
|
||||
}
|
||||
context
|
||||
};
|
||||
Self {
|
||||
client,
|
||||
api_config: None,
|
||||
@@ -7710,6 +7709,161 @@ impl SubAgentManager {
|
||||
})
|
||||
}
|
||||
|
||||
fn record_worker_source(
|
||||
&mut self,
|
||||
worker_id: &str,
|
||||
source: &crate::cost_status::RuntimeUsageRecord,
|
||||
) -> Result<bool> {
|
||||
self.with_store_transaction(|manager| {
|
||||
anyhow::ensure!(
|
||||
manager.owns_store_record(worker_id),
|
||||
"usage worker belongs to a live foreign writer"
|
||||
);
|
||||
let changed = manager.record_worker_source_admitted(worker_id, source)?;
|
||||
manager.persist_state_synchronously()?;
|
||||
Ok(changed)
|
||||
})
|
||||
}
|
||||
|
||||
fn record_worker_source_admitted(
|
||||
&mut self,
|
||||
worker_id: &str,
|
||||
source: &crate::cost_status::RuntimeUsageRecord,
|
||||
) -> Result<bool> {
|
||||
let key = crate::cost_status::usage_source_fingerprint(&source.source_id);
|
||||
let record = self
|
||||
.worker_records
|
||||
.get_mut(worker_id)
|
||||
.ok_or_else(|| anyhow!("usage worker is unavailable"))?;
|
||||
if record
|
||||
.missing_usage_sources
|
||||
.get(&key)
|
||||
.is_some_and(|coverage| !coverage.matches_route(&source.usage.route))
|
||||
{
|
||||
return Err(anyhow!("worker usage source changed its captured route"));
|
||||
}
|
||||
let previous = record.source_usage_records.get(&key);
|
||||
if record.usage_source_fingerprints.contains(&key) && previous.is_none() {
|
||||
return Err(anyhow!(
|
||||
"legacy worker source has no provable prior contribution"
|
||||
));
|
||||
}
|
||||
if !source.permits_replacement(previous.map(|old| &old.0))? {
|
||||
return Ok(false);
|
||||
}
|
||||
anyhow::ensure!(
|
||||
source.is_acknowledged(),
|
||||
"worker source has no durable acknowledgement"
|
||||
);
|
||||
let revision = source
|
||||
.settlement
|
||||
.as_ref()
|
||||
.ok_or_else(|| anyhow!("worker source has no revision"))?;
|
||||
let reported = revision.coverage == crate::llm_client::ProviderUsageCoverage::Reported;
|
||||
let incomplete = revision.coverage == crate::llm_client::ProviderUsageCoverage::Incomplete;
|
||||
let old_usage = previous.map(|old| &old.0.usage.usage);
|
||||
let old_cost = previous.and_then(|old| old.1);
|
||||
if let Some((prior, cost)) = previous {
|
||||
prior.validate()?;
|
||||
let expected_cost = crate::cost_status::UsageSourceContribution::for_record(prior);
|
||||
let expected_cost = (expected_cost.priced > 0)
|
||||
.then(|| priced_usd_microusd(&prior.usage.route.audit_frozen(&prior.usage.usage)))
|
||||
.flatten();
|
||||
anyhow::ensure!(
|
||||
*cost == expected_cost,
|
||||
"worker prior cost does not match its frozen receipt"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
record.usage.input_tokens.unwrap_or(0) >= u64::from(prior.usage.usage.input_tokens)
|
||||
&& record.usage.output_tokens.unwrap_or(0)
|
||||
>= u64::from(prior.usage.usage.output_tokens)
|
||||
&& record.usage.total_tokens.unwrap_or(0)
|
||||
>= usage_total_tokens(&prior.usage.usage)
|
||||
&& record.usage.cost_microusd.unwrap_or(0) >= cost.unwrap_or(0),
|
||||
"worker projection is missing its prior source contribution"
|
||||
);
|
||||
}
|
||||
let contribution = crate::cost_status::UsageSourceContribution::for_record(source);
|
||||
let priced = (contribution.priced > 0)
|
||||
.then(|| priced_usd_microusd(&source.usage.route.audit_frozen(&source.usage.usage)))
|
||||
.flatten();
|
||||
let update = |total: &mut Option<u64>, old: u64, new: u64| {
|
||||
*total = Some(total.unwrap_or(0).saturating_sub(old).saturating_add(new));
|
||||
};
|
||||
if reported
|
||||
|| (incomplete
|
||||
&& (source.usage.usage.input_tokens > 0
|
||||
|| old_usage.is_some_and(|usage| usage.input_tokens > 0)))
|
||||
{
|
||||
update(
|
||||
&mut record.usage.input_tokens,
|
||||
old_usage.map_or(0, |usage| u64::from(usage.input_tokens)),
|
||||
u64::from(source.usage.usage.input_tokens),
|
||||
);
|
||||
}
|
||||
if reported
|
||||
|| (incomplete
|
||||
&& (source.usage.usage.output_tokens > 0
|
||||
|| old_usage.is_some_and(|usage| usage.output_tokens > 0)))
|
||||
{
|
||||
update(
|
||||
&mut record.usage.output_tokens,
|
||||
old_usage.map_or(0, |usage| u64::from(usage.output_tokens)),
|
||||
u64::from(source.usage.usage.output_tokens),
|
||||
);
|
||||
}
|
||||
if reported
|
||||
|| (incomplete
|
||||
&& (contribution.total_tokens() > 0
|
||||
|| old_usage.is_some_and(|usage| usage_total_tokens(usage) > 0)))
|
||||
{
|
||||
update(
|
||||
&mut record.usage.total_tokens,
|
||||
old_usage.map_or(0, usage_total_tokens),
|
||||
contribution.total_tokens(),
|
||||
);
|
||||
}
|
||||
if old_cost.is_some() || priced.is_some() {
|
||||
update(
|
||||
&mut record.usage.cost_microusd,
|
||||
old_cost.unwrap_or(0),
|
||||
priced.unwrap_or(0),
|
||||
);
|
||||
}
|
||||
if matches!(
|
||||
revision.coverage,
|
||||
crate::llm_client::ProviderUsageCoverage::Missing
|
||||
| crate::llm_client::ProviderUsageCoverage::Incomplete
|
||||
) {
|
||||
if record.missing_usage_sources.len() < crate::cost_status::MAX_MISSING_USAGE_SOURCES
|
||||
|| record.missing_usage_sources.contains_key(&key)
|
||||
{
|
||||
record.missing_usage_sources.insert(
|
||||
key.clone(),
|
||||
crate::cost_status::MissingUsageCoverage::for_route(
|
||||
&source.usage.route,
|
||||
revision.reason,
|
||||
),
|
||||
);
|
||||
} else {
|
||||
record.missing_usage_overflowed = true;
|
||||
}
|
||||
} else {
|
||||
record.missing_usage_sources.remove(&key);
|
||||
}
|
||||
record.usage_source_fingerprints.insert(key.clone());
|
||||
record
|
||||
.source_usage_records
|
||||
.insert(key, (source.persisted(), priced));
|
||||
record.has_unreported_usage = record.legacy_unreported_usage
|
||||
|| record.missing_usage_overflowed
|
||||
|| !record.missing_usage_sources.is_empty();
|
||||
record.updated_at_ms = epoch_millis_now();
|
||||
refresh_usage_note(&mut record.usage);
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn record_worker_routed_usage(
|
||||
&mut self,
|
||||
worker_id: &str,
|
||||
@@ -7731,6 +7885,7 @@ impl SubAgentManager {
|
||||
}
|
||||
self.record_worker_usage(worker_id, source_id, usage, priced_cost_microusd);
|
||||
}
|
||||
#[cfg(test)]
|
||||
fn record_worker_usage(
|
||||
&mut self,
|
||||
worker_id: &str,
|
||||
@@ -7801,6 +7956,7 @@ impl SubAgentManager {
|
||||
self.persist_state_debounced();
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn record_worker_missing_usage(
|
||||
&mut self,
|
||||
worker_id: &str,
|
||||
@@ -8114,7 +8270,7 @@ impl SubAgentManager {
|
||||
// Delivery verification is deferred: `ensure_worker_delivery_verified`
|
||||
// computes it off the lock before the terminal commit (natural and
|
||||
// panic paths) or heals it on the next detail read (#6210).
|
||||
let status = worker_status_from_subagent_result(result);
|
||||
let status = worker_status_from_subagent_status(&result.status);
|
||||
let message = match &result.status {
|
||||
SubAgentStatus::Completed => Some("completed".to_string()),
|
||||
SubAgentStatus::Failed(err) => Some(err.clone()),
|
||||
@@ -12327,12 +12483,10 @@ async fn subagent_session_projection(
|
||||
// Interrupted snapshot that carries a continuable checkpoint
|
||||
// (`continuable`/`needs_continuation` true, `terminal` true) means the
|
||||
// worker is parked waiting for the parent to act, so it must project as
|
||||
// `waiting_for_user` rather than a bare `interrupted`. When a worker
|
||||
// record exists its status was already derived via
|
||||
// `worker_status_from_subagent_result`; mirror that derivation when there
|
||||
// is no record so both paths agree on the "needs parent action" signal.
|
||||
// `waiting_for_user` rather than a bare `interrupted`.
|
||||
let status = worker_record
|
||||
.as_ref()
|
||||
.filter(|_| !continuable)
|
||||
.map(|record| agent_worker_status_name(record.status))
|
||||
.unwrap_or_else(|| agent_worker_status_name(worker_status_from_subagent_result(&snapshot)))
|
||||
.to_string();
|
||||
@@ -13153,7 +13307,15 @@ struct DurableSubAgentStore {
|
||||
impl DurableSubAgentStore {
|
||||
fn from_legacy(state: PersistedSubAgentState) -> Self {
|
||||
Self {
|
||||
store_schema: 3,
|
||||
store_schema: if state
|
||||
.workers
|
||||
.iter()
|
||||
.any(|worker| !worker.source_usage_records.is_empty())
|
||||
{
|
||||
4
|
||||
} else {
|
||||
3
|
||||
},
|
||||
revision: 0,
|
||||
state,
|
||||
record_writers: HashMap::new(),
|
||||
@@ -13162,7 +13324,19 @@ impl DurableSubAgentStore {
|
||||
}
|
||||
|
||||
fn validate(&mut self) -> Result<()> {
|
||||
anyhow::ensure!(self.store_schema == 3, "unsupported delegated store format");
|
||||
anyhow::ensure!(
|
||||
(3..=4).contains(&self.store_schema),
|
||||
"unsupported delegated store format"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
self.store_schema >= 4
|
||||
|| self
|
||||
.state
|
||||
.workers
|
||||
.iter()
|
||||
.all(|worker| worker.source_usage_records.is_empty()),
|
||||
"native source accounting requires delegated format4"
|
||||
);
|
||||
self.state
|
||||
.validate_worker_origins()
|
||||
.map_err(anyhow::Error::msg)?;
|
||||
@@ -13657,6 +13831,7 @@ impl SubAgentStoreAuthority {
|
||||
for key in [
|
||||
"usage",
|
||||
"usage_source_fingerprints",
|
||||
"source_usage_records",
|
||||
"missing_usage_sources",
|
||||
"missing_usage_overflowed",
|
||||
"legacy_unreported_usage",
|
||||
@@ -13946,6 +14121,14 @@ impl SubAgentStoreAuthority {
|
||||
.revision
|
||||
.checked_add(1)
|
||||
.ok_or_else(|| anyhow!("delegated store revision exhausted"))?;
|
||||
if next
|
||||
.state
|
||||
.workers
|
||||
.iter()
|
||||
.any(|worker| !worker.source_usage_records.is_empty())
|
||||
{
|
||||
next.store_schema = 4;
|
||||
}
|
||||
next.validate()?;
|
||||
write_json_atomic(&self.state_root, &self.path, &next)?;
|
||||
self.revision
|
||||
|
||||
@@ -2402,10 +2402,48 @@ pub(super) async fn delayed_chat_client(
|
||||
(client, calls, bodies, config)
|
||||
}
|
||||
|
||||
fn freeze_usage_fixture_quote(route: &mut crate::cost_status::EffectiveRouteEnvelope) {
|
||||
let base_url = route.provider.provider().default_base_url();
|
||||
let models = serde_json::from_value::<
|
||||
Vec<codewhale_config::catalog::configured::ConfiguredModel>,
|
||||
>(json!([{
|
||||
"provider": &route.provider_identity,
|
||||
"base_url": base_url,
|
||||
"id": &route.model,
|
||||
"cost": {"input": 1.0, "output": 2.0}
|
||||
}]))
|
||||
.expect("fixture declared rates");
|
||||
route.provider_live_pricing = Some(
|
||||
crate::provider_catalog_live::configured_dispatch_pricing_quote_at(
|
||||
&models,
|
||||
route.provider,
|
||||
&route.provider_identity,
|
||||
&route.model,
|
||||
base_url,
|
||||
u64::try_from(route.dispatched_at.timestamp()).expect("fixture dispatch time"),
|
||||
)
|
||||
.expect("fixture frozen quote"),
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn detached_interactive_usage_after_mailbox_seal_reaches_session_accounting() {
|
||||
let _cost_scope = crate::cost_status::test_scope();
|
||||
let tmp = tempdir().expect("tempdir");
|
||||
let sessions = crate::session_manager::SessionManager::new(tmp.path().join("sessions"))
|
||||
.expect("fixture session store");
|
||||
let session = crate::session_manager::create_saved_session_with_id_and_mode(
|
||||
"session-detached-usage".to_string(),
|
||||
&[],
|
||||
"deepseek-v4-flash",
|
||||
tmp.path(),
|
||||
0,
|
||||
None,
|
||||
Some("agent"),
|
||||
);
|
||||
sessions
|
||||
.save_session(&session)
|
||||
.expect("fixture saved origin");
|
||||
let manager = Arc::new(RwLock::new(SubAgentManager::new(
|
||||
tmp.path().to_path_buf(),
|
||||
2,
|
||||
@@ -2414,23 +2452,34 @@ async fn detached_interactive_usage_after_mailbox_seal_reaches_session_accountin
|
||||
let mut root = stub_runtime();
|
||||
root.manager = Arc::clone(&manager);
|
||||
root.context = ToolContext::new(tmp.path()).with_state_namespace("session-detached-usage");
|
||||
root.accounting_origin = SubAgentAccountingOrigin::capture(&root.context);
|
||||
root.mailbox = Some(mailbox.clone());
|
||||
let runtime_owner = "interactive:session-detached-usage:turn-parent";
|
||||
crate::cost_status::register_interactive_runtime_usage_sink(
|
||||
crate::usage_persistence::register_persistent_interactive_runtime_usage_sink_for_test(
|
||||
runtime_owner,
|
||||
crate::cost_status::scope_token(),
|
||||
"session-detached-usage",
|
||||
"turn-parent",
|
||||
&sessions,
|
||||
);
|
||||
root.runtime_usage_lease = crate::cost_status::acquire_runtime_usage_lease(runtime_owner);
|
||||
root.context.accounting_origin = Some(
|
||||
crate::cost_status::NativeUsageOrigin::runtime(
|
||||
crate::cost_status::scope_token(),
|
||||
runtime_owner,
|
||||
)
|
||||
.expect("fixture captured runtime origin"),
|
||||
);
|
||||
let detached = root.background_runtime();
|
||||
|
||||
let cases = [
|
||||
let mut cases = [
|
||||
(
|
||||
"agent_deepseek",
|
||||
"subagent:agent_deepseek:step:1:response:late",
|
||||
crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
None,
|
||||
ProviderKind::Deepseek,
|
||||
"deepseek-direct",
|
||||
ProviderKind::Deepseek.as_str(),
|
||||
"deepseek-v4-flash",
|
||||
Some(ProviderKind::Deepseek.provider().default_base_url()),
|
||||
chrono::Utc::now(),
|
||||
@@ -2447,7 +2496,7 @@ async fn detached_interactive_usage_after_mailbox_seal_reaches_session_accountin
|
||||
crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
None,
|
||||
ProviderKind::Anthropic,
|
||||
"anthropic-direct",
|
||||
ProviderKind::Anthropic.as_str(),
|
||||
"claude-sonnet-4-5",
|
||||
Some(ProviderKind::Anthropic.provider().default_base_url()),
|
||||
chrono::Utc::now(),
|
||||
@@ -2476,6 +2525,11 @@ async fn detached_interactive_usage_after_mailbox_seal_reaches_session_accountin
|
||||
},
|
||||
),
|
||||
];
|
||||
for (_, _, route, _) in &mut cases {
|
||||
if route.provider != ProviderKind::Custom {
|
||||
freeze_usage_fixture_quote(route);
|
||||
}
|
||||
}
|
||||
for (worker_id, ..) in &cases {
|
||||
manager.write().await.register_worker_for_session(
|
||||
make_worker_spec(worker_id, tmp.path().to_path_buf()),
|
||||
@@ -2514,18 +2568,18 @@ async fn detached_interactive_usage_after_mailbox_seal_reaches_session_accountin
|
||||
"post-seal usage must not weaken the mailbox ordering boundary"
|
||||
);
|
||||
|
||||
let session_usage = crate::cost_status::drain();
|
||||
let pending = crate::cost_status::drain();
|
||||
assert_eq!(pending.source_records.len(), cases.len());
|
||||
let session_usage = sessions
|
||||
.load_session_snapshot("session-detached-usage")
|
||||
.expect("durable detached usage")
|
||||
.metadata
|
||||
.cost;
|
||||
assert_eq!(session_usage.usage_source_fingerprints.len(), cases.len());
|
||||
assert!(
|
||||
session_usage.estimate.is_positive(),
|
||||
"direct provider routes must reach the live money total"
|
||||
);
|
||||
assert!(session_usage.subagent_cost_usd > 0.0);
|
||||
assert_eq!(
|
||||
session_usage
|
||||
.priced_turns
|
||||
.saturating_add(session_usage.unpriced_turns),
|
||||
cases.len() as u32,
|
||||
"every metered or unknown-basis route contributes exactly one coverage receipt"
|
||||
session_usage.priced_turns + session_usage.unpriced_turns,
|
||||
cases.len() as u32
|
||||
);
|
||||
for (worker_id, source_id, _route, usage) in cases {
|
||||
let worker = manager
|
||||
@@ -2559,6 +2613,20 @@ async fn detached_interactive_usage_after_mailbox_seal_reaches_session_accountin
|
||||
async fn child_guardian_usage_source_is_sanitized_and_replay_idempotent() {
|
||||
let _cost_scope = crate::cost_status::test_scope();
|
||||
let tmp = tempdir().expect("tempdir");
|
||||
let sessions = crate::session_manager::SessionManager::new(tmp.path().join("sessions"))
|
||||
.expect("fixture session store");
|
||||
let session = crate::session_manager::create_saved_session_with_id_and_mode(
|
||||
"guardian-usage-session".to_string(),
|
||||
&[],
|
||||
"deepseek-v4-flash",
|
||||
tmp.path(),
|
||||
0,
|
||||
None,
|
||||
Some("agent"),
|
||||
);
|
||||
sessions
|
||||
.save_session(&session)
|
||||
.expect("fixture saved origin");
|
||||
let manager = Arc::new(RwLock::new(SubAgentManager::new(
|
||||
tmp.path().to_path_buf(),
|
||||
1,
|
||||
@@ -2570,13 +2638,25 @@ async fn child_guardian_usage_source_is_sanitized_and_replay_idempotent() {
|
||||
);
|
||||
|
||||
let runtime_owner = "interactive:guardian-usage-session:turn-parent";
|
||||
crate::cost_status::register_interactive_runtime_usage_sink(
|
||||
crate::usage_persistence::register_persistent_interactive_runtime_usage_sink_for_test(
|
||||
runtime_owner,
|
||||
crate::cost_status::scope_token(),
|
||||
"guardian-usage-session",
|
||||
"turn-parent",
|
||||
&sessions,
|
||||
);
|
||||
let mut runtime = stub_runtime();
|
||||
runtime.manager = Arc::clone(&manager);
|
||||
runtime.context = ToolContext::new(tmp.path()).with_state_namespace("guardian-usage-session");
|
||||
runtime.accounting_origin = SubAgentAccountingOrigin::capture(&runtime.context);
|
||||
runtime.runtime_usage_lease = crate::cost_status::acquire_runtime_usage_lease(runtime_owner);
|
||||
runtime.context.accounting_origin = Some(
|
||||
crate::cost_status::NativeUsageOrigin::runtime(
|
||||
crate::cost_status::scope_token(),
|
||||
runtime_owner,
|
||||
)
|
||||
.expect("fixture captured runtime origin"),
|
||||
);
|
||||
|
||||
let source_id = child_guardian_usage_source_id("agent_guardian", "tool-fixed");
|
||||
assert_eq!(
|
||||
@@ -2592,14 +2672,15 @@ async fn child_guardian_usage_source_is_sanitized_and_replay_idempotent() {
|
||||
assert!(!source_id.contains("agent_guardian"));
|
||||
assert!(!source_id.contains("tool-fixed"));
|
||||
|
||||
let route = crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
let mut route = crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
None,
|
||||
ProviderKind::Deepseek,
|
||||
"deepseek-direct",
|
||||
ProviderKind::Deepseek.as_str(),
|
||||
"deepseek-v4-flash",
|
||||
Some(ProviderKind::Deepseek.provider().default_base_url()),
|
||||
chrono::Utc::now(),
|
||||
);
|
||||
freeze_usage_fixture_quote(&mut route);
|
||||
let usage = Usage {
|
||||
input_tokens: 7,
|
||||
output_tokens: 5,
|
||||
@@ -2618,7 +2699,13 @@ async fn child_guardian_usage_source_is_sanitized_and_replay_idempotent() {
|
||||
record_provider_response_usage(&runtime, "agent_guardian", &source_id, route, &usage).await;
|
||||
|
||||
crate::cost_status::finish_runtime_usage_owner(runtime_owner);
|
||||
let session_usage = crate::cost_status::drain();
|
||||
let pending = crate::cost_status::drain();
|
||||
assert_eq!(pending.source_records.len(), 1);
|
||||
let session_usage = sessions
|
||||
.load_session_snapshot("guardian-usage-session")
|
||||
.expect("durable fixture accounting")
|
||||
.metadata
|
||||
.cost;
|
||||
let fingerprint = crate::cost_status::usage_source_fingerprint(&source_id);
|
||||
assert_eq!(
|
||||
session_usage.usage_source_fingerprints,
|
||||
@@ -2646,6 +2733,20 @@ async fn child_guardian_usage_source_is_sanitized_and_replay_idempotent() {
|
||||
async fn ownerless_no_mailbox_provider_usage_reaches_accounting_once() {
|
||||
let _cost_scope = crate::cost_status::test_scope();
|
||||
let tmp = tempdir().expect("tempdir");
|
||||
let sessions = crate::session_manager::SessionManager::new(tmp.path().join("sessions"))
|
||||
.expect("fixture session store");
|
||||
let session = crate::session_manager::create_saved_session_with_id_and_mode(
|
||||
"direct-usage-session".to_string(),
|
||||
&[],
|
||||
"deepseek-v4-flash",
|
||||
tmp.path(),
|
||||
0,
|
||||
None,
|
||||
Some("agent"),
|
||||
);
|
||||
sessions
|
||||
.save_session(&session)
|
||||
.expect("fixture saved origin");
|
||||
let manager = Arc::new(RwLock::new(SubAgentManager::new(
|
||||
tmp.path().to_path_buf(),
|
||||
1,
|
||||
@@ -2657,18 +2758,26 @@ async fn ownerless_no_mailbox_provider_usage_reaches_accounting_once() {
|
||||
);
|
||||
let mut runtime = stub_runtime();
|
||||
runtime.manager = Arc::clone(&manager);
|
||||
runtime.context = ToolContext::new(tmp.path()).with_state_namespace("direct-usage-session");
|
||||
runtime.accounting_origin = SubAgentAccountingOrigin::capture(&runtime.context);
|
||||
runtime.context.accounting_origin = Some(crate::cost_status::NativeUsageOrigin::session(
|
||||
runtime.accounting_origin.cost_scope,
|
||||
sessions.sessions_dir().to_path_buf(),
|
||||
"direct-usage-session".to_string(),
|
||||
));
|
||||
assert!(runtime.runtime_usage_lease.is_none());
|
||||
assert!(runtime.mailbox.is_none());
|
||||
|
||||
let source_id = "subagent:agent_direct:step:1:response:direct";
|
||||
let route = crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
let mut route = crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
None,
|
||||
ProviderKind::Deepseek,
|
||||
"deepseek-direct",
|
||||
ProviderKind::Deepseek.as_str(),
|
||||
"deepseek-v4-flash",
|
||||
Some(ProviderKind::Deepseek.provider().default_base_url()),
|
||||
chrono::Utc::now(),
|
||||
);
|
||||
freeze_usage_fixture_quote(&mut route);
|
||||
let usage = Usage {
|
||||
input_tokens: 19,
|
||||
output_tokens: 7,
|
||||
@@ -2679,7 +2788,13 @@ async fn ownerless_no_mailbox_provider_usage_reaches_accounting_once() {
|
||||
.await;
|
||||
}
|
||||
|
||||
let session_usage = crate::cost_status::drain();
|
||||
let pending = crate::cost_status::drain();
|
||||
assert_eq!(pending.source_records.len(), 1);
|
||||
let session_usage = sessions
|
||||
.load_session_snapshot("direct-usage-session")
|
||||
.expect("durable fixture accounting")
|
||||
.metadata
|
||||
.cost;
|
||||
let fingerprint = crate::cost_status::usage_source_fingerprint(source_id);
|
||||
assert_eq!(
|
||||
session_usage.usage_source_fingerprints,
|
||||
@@ -2692,7 +2807,7 @@ async fn ownerless_no_mailbox_provider_usage_reaches_accounting_once() {
|
||||
1,
|
||||
"a replayed ownerless provider response must have one cost receipt"
|
||||
);
|
||||
assert!(session_usage.estimate.is_positive());
|
||||
assert!(session_usage.subagent_cost_usd > 0.0);
|
||||
|
||||
let worker = manager
|
||||
.read()
|
||||
@@ -2757,11 +2872,16 @@ async fn ownerless_child_usage_crossing_new_settles_to_its_dispatch_origin_once(
|
||||
root.manager = Arc::clone(&manager);
|
||||
root.context = ToolContext::new(tmp.path()).with_state_namespace(origin_session_id);
|
||||
root.accounting_origin = SubAgentAccountingOrigin::capture(&root.context);
|
||||
root.context.accounting_origin = Some(crate::cost_status::NativeUsageOrigin::session(
|
||||
root.accounting_origin.cost_scope,
|
||||
sessions.sessions_dir().to_path_buf(),
|
||||
origin_session_id.to_string(),
|
||||
));
|
||||
let child = root.background_runtime();
|
||||
assert!(child.runtime_usage_lease.is_none());
|
||||
assert!(child.mailbox.is_none());
|
||||
|
||||
let route = crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
let mut route = crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
None,
|
||||
ProviderKind::Deepseek,
|
||||
ProviderKind::Deepseek.as_str(),
|
||||
@@ -2769,6 +2889,7 @@ async fn ownerless_child_usage_crossing_new_settles_to_its_dispatch_origin_once(
|
||||
Some(ProviderKind::Deepseek.provider().default_base_url()),
|
||||
chrono::Utc::now(),
|
||||
);
|
||||
freeze_usage_fixture_quote(&mut route);
|
||||
let usage = Usage {
|
||||
input_tokens: 19,
|
||||
output_tokens: 7,
|
||||
@@ -2786,12 +2907,17 @@ async fn ownerless_child_usage_crossing_new_settles_to_its_dispatch_origin_once(
|
||||
// `/new` retires the origin scope exactly as the TUI does, while the
|
||||
// child keeps running.
|
||||
let settled_origin = crate::cost_status::close_current_scope();
|
||||
assert_eq!(settled_origin.source_records.len(), 1);
|
||||
assert_eq!(
|
||||
settled_origin.usage_source_fingerprints,
|
||||
[first_fingerprint.clone()].into()
|
||||
crate::cost_status::usage_source_fingerprint(&settled_origin.source_records[0].source_id),
|
||||
first_fingerprint
|
||||
);
|
||||
assert_eq!(settled_origin.priced_turns, 1);
|
||||
assert!(settled_origin.estimate.is_positive());
|
||||
let first_origin = sessions
|
||||
.load_session_snapshot(origin_session_id)
|
||||
.expect("first durable origin");
|
||||
assert_eq!(first_origin.metadata.total_tokens, 26);
|
||||
assert_eq!(first_origin.metadata.cost.priced_turns, 1);
|
||||
assert!(first_origin.metadata.cost.subagent_cost_usd > 0.0);
|
||||
|
||||
// The next response, its monitor replay, and a guardian reply without a
|
||||
// usage payload all arrive after the scope generation moved on.
|
||||
@@ -2808,16 +2934,43 @@ async fn ownerless_child_usage_crossing_new_settles_to_its_dispatch_origin_once(
|
||||
}
|
||||
|
||||
let replacement_live = crate::cost_status::drain();
|
||||
assert!(
|
||||
replacement_live.is_empty(),
|
||||
"late origin receipts charged the replacement scope: {replacement_live:?}"
|
||||
assert_eq!(
|
||||
replacement_live.estimate,
|
||||
crate::pricing::CostEstimate::default()
|
||||
);
|
||||
assert_eq!(
|
||||
replacement_live.priced_turns + replacement_live.unpriced_turns,
|
||||
0
|
||||
);
|
||||
let mut replacement_projection = sessions
|
||||
.load_session_by_prefix(replacement_session_id)
|
||||
.expect("replacement projection");
|
||||
let receipt = replacement_projection
|
||||
.capture_publication(crate::session_manager::SessionWriteOwner::control())
|
||||
.expect("replacement publication");
|
||||
let before_projection = serde_json::to_value(&replacement_projection.metadata.cost).unwrap();
|
||||
for record in &replacement_live.source_records {
|
||||
assert!(
|
||||
receipt
|
||||
.project_usage_record(
|
||||
crate::cost_status::scope_token(),
|
||||
record,
|
||||
&mut replacement_projection.metadata.cost,
|
||||
)
|
||||
.expect("foreign source projection")
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
serde_json::to_value(&replacement_projection.metadata.cost).unwrap(),
|
||||
before_projection
|
||||
);
|
||||
|
||||
let origin = sessions
|
||||
.load_session_snapshot(origin_session_id)
|
||||
.expect("origin session");
|
||||
assert_eq!(origin.metadata.total_tokens, 26);
|
||||
assert_eq!(origin.metadata.cost.priced_turns, 1);
|
||||
assert_eq!(origin.metadata.total_tokens, 52);
|
||||
assert_eq!(origin.metadata.cost.priced_turns, 2);
|
||||
assert_eq!(origin.metadata.cost.unpriced_turns, 1);
|
||||
assert!(
|
||||
origin
|
||||
@@ -2828,7 +2981,12 @@ async fn ownerless_child_usage_crossing_new_settles_to_its_dispatch_origin_once(
|
||||
);
|
||||
assert_eq!(
|
||||
origin.metadata.cost.usage_source_fingerprints,
|
||||
[late_fingerprint.clone(), missing_fingerprint.clone()].into()
|
||||
[
|
||||
first_fingerprint.clone(),
|
||||
late_fingerprint.clone(),
|
||||
missing_fingerprint.clone()
|
||||
]
|
||||
.into()
|
||||
);
|
||||
|
||||
let replacement = sessions
|
||||
@@ -2863,6 +3021,20 @@ async fn ownerless_child_usage_crossing_new_settles_to_its_dispatch_origin_once(
|
||||
async fn provider_success_without_usage_records_one_route_aware_gap_and_no_zero_mail() {
|
||||
let _cost_scope = crate::cost_status::test_scope();
|
||||
let tmp = tempdir().expect("tempdir");
|
||||
let sessions = crate::session_manager::SessionManager::new(tmp.path().join("sessions"))
|
||||
.expect("fixture session store");
|
||||
let session = crate::session_manager::create_saved_session_with_id_and_mode(
|
||||
"missing-usage-session".to_string(),
|
||||
&[],
|
||||
"deepseek-v4-flash",
|
||||
tmp.path(),
|
||||
0,
|
||||
None,
|
||||
Some("agent"),
|
||||
);
|
||||
sessions
|
||||
.save_session(&session)
|
||||
.expect("fixture saved origin");
|
||||
let manager = Arc::new(RwLock::new(SubAgentManager::new(
|
||||
tmp.path().to_path_buf(),
|
||||
1,
|
||||
@@ -2874,21 +3046,33 @@ async fn provider_success_without_usage_records_one_route_aware_gap_and_no_zero_
|
||||
);
|
||||
|
||||
let runtime_owner = "interactive:missing-usage-session:turn-parent";
|
||||
crate::cost_status::register_interactive_runtime_usage_sink(
|
||||
crate::usage_persistence::register_persistent_interactive_runtime_usage_sink_for_test(
|
||||
runtime_owner,
|
||||
crate::cost_status::scope_token(),
|
||||
"missing-usage-session",
|
||||
"turn-parent",
|
||||
&sessions,
|
||||
);
|
||||
let (mailbox, mut mailbox_rx) = Mailbox::new(CancellationToken::new());
|
||||
let mut runtime = stub_runtime();
|
||||
runtime.manager = Arc::clone(&manager);
|
||||
runtime.context = ToolContext::new(tmp.path()).with_state_namespace("missing-usage-session");
|
||||
runtime.accounting_origin = SubAgentAccountingOrigin::capture(&runtime.context);
|
||||
runtime.mailbox = Some(mailbox);
|
||||
runtime.runtime_usage_lease = crate::cost_status::acquire_runtime_usage_lease(runtime_owner);
|
||||
runtime.context.accounting_origin = Some(
|
||||
crate::cost_status::NativeUsageOrigin::runtime(
|
||||
crate::cost_status::scope_token(),
|
||||
runtime_owner,
|
||||
)
|
||||
.expect("fixture captured runtime origin"),
|
||||
);
|
||||
|
||||
let source_id = child_guardian_usage_source_id("agent_missing_usage", "tool-fixed");
|
||||
let route = crate::cost_status::EffectiveRouteEnvelope::capture(
|
||||
None,
|
||||
ProviderKind::Deepseek,
|
||||
"deepseek-direct",
|
||||
ProviderKind::Deepseek.as_str(),
|
||||
"deepseek-v4-flash",
|
||||
Some(ProviderKind::Deepseek.provider().default_base_url()),
|
||||
chrono::Utc::now(),
|
||||
@@ -2909,7 +3093,13 @@ async fn provider_success_without_usage_records_one_route_aware_gap_and_no_zero_
|
||||
"missing usage must not also publish a priced-zero TokenUsage message"
|
||||
);
|
||||
crate::cost_status::finish_runtime_usage_owner(runtime_owner);
|
||||
let session_usage = crate::cost_status::drain();
|
||||
let pending = crate::cost_status::drain();
|
||||
assert_eq!(pending.source_records.len(), 1);
|
||||
let session_usage = sessions
|
||||
.load_session_snapshot("missing-usage-session")
|
||||
.expect("durable fixture accounting")
|
||||
.metadata
|
||||
.cost;
|
||||
let fingerprint = crate::cost_status::usage_source_fingerprint(&source_id);
|
||||
assert_eq!(
|
||||
session_usage.usage_source_fingerprints,
|
||||
@@ -7396,7 +7586,7 @@ async fn coordination_interrupt_fans_in_once_and_preserves_checkpoint() {
|
||||
.count();
|
||||
assert_eq!(complete_events, 1);
|
||||
let worker = manager.get_worker_record(&agent_id).expect("worker record");
|
||||
assert_eq!(worker.status, AgentWorkerStatus::WaitingForUser);
|
||||
assert_eq!(worker.status, AgentWorkerStatus::Interrupted);
|
||||
assert_eq!(
|
||||
worker
|
||||
.events
|
||||
@@ -10245,7 +10435,7 @@ async fn api_timeout_preserves_checkpoint_and_returns_needs_input_without_parkin
|
||||
.as_ref()
|
||||
.expect("worker record")
|
||||
.status,
|
||||
AgentWorkerStatus::WaitingForUser
|
||||
AgentWorkerStatus::Interrupted
|
||||
);
|
||||
assert_eq!(
|
||||
projection
|
||||
@@ -19266,6 +19456,23 @@ pub(super) async fn run_incomplete_response_worker(
|
||||
) {
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
let accounting_home = tempdir().expect("private incomplete-response accounting home");
|
||||
let sessions =
|
||||
crate::session_manager::SessionManager::new(accounting_home.path().join("sessions"))
|
||||
.expect("incomplete-response session store");
|
||||
let session_id = "incomplete-response-session";
|
||||
let session = crate::session_manager::create_saved_session_with_id_and_mode(
|
||||
session_id.to_string(),
|
||||
&[],
|
||||
"deepseek-v4-flash",
|
||||
workspace,
|
||||
0,
|
||||
None,
|
||||
Some("agent"),
|
||||
);
|
||||
sessions
|
||||
.save_session(&session)
|
||||
.expect("saved incomplete-response origin");
|
||||
let manager = Arc::new(RwLock::new(SubAgentManager::new(
|
||||
workspace.to_path_buf(),
|
||||
2,
|
||||
@@ -19287,7 +19494,11 @@ pub(super) async fn run_incomplete_response_worker(
|
||||
{
|
||||
let mut manager = manager.write().await;
|
||||
manager.agents.insert(agent_id.clone(), agent);
|
||||
manager.register_worker(make_worker_spec(&agent_id, workspace.to_path_buf()));
|
||||
manager.register_worker_for_session(
|
||||
make_worker_spec(&agent_id, workspace.to_path_buf()),
|
||||
session_id,
|
||||
None,
|
||||
);
|
||||
}
|
||||
|
||||
let (client, calls, fixture_config) = incomplete_then_complete_chat_client(stop_reason).await;
|
||||
@@ -19296,7 +19507,13 @@ pub(super) async fn run_incomplete_response_worker(
|
||||
runtime.client = client;
|
||||
runtime.api_config = Some(Arc::new(fixture_config));
|
||||
runtime.manager = Arc::clone(&manager);
|
||||
runtime.context = ToolContext::new(workspace.to_path_buf());
|
||||
runtime.context = ToolContext::new(workspace.to_path_buf()).with_state_namespace(session_id);
|
||||
runtime.accounting_origin = SubAgentAccountingOrigin::capture(&runtime.context);
|
||||
runtime.context.accounting_origin = Some(crate::cost_status::NativeUsageOrigin::session(
|
||||
runtime.accounting_origin.cost_scope,
|
||||
sessions.sessions_dir().to_path_buf(),
|
||||
session_id.to_string(),
|
||||
));
|
||||
runtime.mailbox = Some(mailbox);
|
||||
|
||||
run_subagent_task(SubAgentTask {
|
||||
@@ -27904,10 +28121,13 @@ async fn settled_projection_waiter_releases_origin_lease_and_mailbox_before_meta
|
||||
let owner = "interactive:projection-session:origin-turn";
|
||||
let sink_lifetime = Arc::new(());
|
||||
let sink_weak = Arc::downgrade(&sink_lifetime);
|
||||
let sink_calls = Arc::new(AtomicUsize::new(0));
|
||||
let captured_calls = sink_calls.clone();
|
||||
crate::cost_status::register_runtime_usage_sink(
|
||||
owner,
|
||||
Arc::new(move |_| {
|
||||
let _held = &sink_lifetime;
|
||||
captured_calls.fetch_add(1, Ordering::SeqCst);
|
||||
true
|
||||
}),
|
||||
);
|
||||
@@ -27932,20 +28152,43 @@ async fn settled_projection_waiter_releases_origin_lease_and_mailbox_before_meta
|
||||
output_tokens: 7,
|
||||
..Usage::default()
|
||||
};
|
||||
let held_metadata = manager.write().await;
|
||||
projection.recover_settled(
|
||||
&tokio::runtime::Handle::current(),
|
||||
"settled-response".into(),
|
||||
route,
|
||||
usage,
|
||||
Some(7),
|
||||
crate::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage,
|
||||
let source = crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
"settled-response",
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
Some(
|
||||
crate::cost_status::NativeUsageOrigin::runtime(
|
||||
crate::cost_status::scope_token(),
|
||||
owner,
|
||||
)
|
||||
.unwrap(),
|
||||
),
|
||||
);
|
||||
tokio::task::yield_now().await;
|
||||
let reservation = source.admit_dispatch(&route).await.unwrap().unwrap();
|
||||
projection.project_settled(&reservation).await.unwrap();
|
||||
drop(reservation);
|
||||
let settled = source.reported(&route, &usage).unwrap();
|
||||
crate::cost_status::report_runtime_usage_record(
|
||||
crate::cost_status::scope_token(),
|
||||
Some(owner),
|
||||
&settled,
|
||||
)
|
||||
.unwrap();
|
||||
assert!(settled.is_acknowledged());
|
||||
let contribution = crate::cost_status::UsageSourceContribution::for_record(&settled);
|
||||
let expected_cost = (contribution.priced > 0)
|
||||
.then(|| priced_usd_microusd(&route.audit_frozen(&usage)))
|
||||
.flatten();
|
||||
drop(source);
|
||||
assert_eq!(sink_calls.load(Ordering::SeqCst), 2);
|
||||
let held_metadata = manager.write().await;
|
||||
let mut waiting = Box::pin(projection.project_settled(&settled));
|
||||
assert!(futures_util::poll!(&mut waiting).is_pending());
|
||||
drop(waiting);
|
||||
drop(projection);
|
||||
drop(settled);
|
||||
assert!(
|
||||
sink_weak.upgrade().is_none(),
|
||||
"the metadata waiter must not extend the billing lease"
|
||||
crate::cost_status::acquire_runtime_usage_lease(owner).is_none(),
|
||||
"the metadata waiter must not extend the registered billing lease"
|
||||
);
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_secs(1), receiver.recv())
|
||||
@@ -27954,20 +28197,39 @@ async fn settled_projection_waiter_releases_origin_lease_and_mailbox_before_meta
|
||||
.is_none(),
|
||||
"the metadata waiter must not retain the originating mailbox"
|
||||
);
|
||||
assert_eq!(
|
||||
sink_calls.load(Ordering::SeqCst),
|
||||
2,
|
||||
"metadata projection never republishes billing"
|
||||
);
|
||||
assert!(
|
||||
crate::cost_status::take_runtime_usage(owner)
|
||||
.records
|
||||
.is_empty(),
|
||||
"recovery never bills"
|
||||
.is_empty()
|
||||
);
|
||||
drop(held_metadata);
|
||||
let manager = tokio::time::timeout(Duration::from_secs(1), manager.read())
|
||||
.await
|
||||
.unwrap();
|
||||
let record = manager
|
||||
.get_result_by_ref_for_session("projection-session", &worker)
|
||||
.unwrap();
|
||||
let usage = record.usage.as_ref().unwrap();
|
||||
assert_eq!(usage.total_tokens, Some(20));
|
||||
assert_eq!(usage.cost_microusd, Some(7));
|
||||
tokio::time::timeout(Duration::from_secs(1), async {
|
||||
loop {
|
||||
let manager = manager.read().await;
|
||||
let record = manager
|
||||
.get_result_by_ref_for_session("projection-session", &worker)
|
||||
.unwrap();
|
||||
if record.usage.as_ref().and_then(|usage| usage.total_tokens) == Some(20) {
|
||||
assert_eq!(record.usage.as_ref().unwrap().cost_microusd, expected_cost);
|
||||
break;
|
||||
}
|
||||
drop(manager);
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
tokio::time::timeout(Duration::from_secs(1), async {
|
||||
while sink_weak.upgrade().is_some() {
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(sink_calls.load(Ordering::SeqCst), 2);
|
||||
}
|
||||
@@ -268,6 +268,7 @@ struct EvidenceBlock {
|
||||
struct CritiqueRun {
|
||||
report: CritiqueReport,
|
||||
usage: Usage,
|
||||
source_record: crate::cost_status::RuntimeUsageRecord,
|
||||
/// Provider stop reason when the critic response was incomplete; billed
|
||||
/// usage above still counts, but the partial report must not.
|
||||
incomplete_stop_reason: Option<String>,
|
||||
@@ -430,6 +431,13 @@ self-check of whether what you just did is actually correct and complete."
|
||||
));
|
||||
};
|
||||
|
||||
#[cfg(not(any(test, feature = "test-support")))]
|
||||
if context.accounting_origin.is_none() {
|
||||
return Err(ToolError::execution_failed(
|
||||
"verify has no admitted usage origin".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
// --- Deterministic evidence gathering ---
|
||||
let mut evidence: Vec<EvidenceBlock> = Vec::new();
|
||||
if gather_diff_scope {
|
||||
@@ -465,6 +473,7 @@ self-check of whether what you just did is actually correct and complete."
|
||||
self.critic_effort,
|
||||
max_tokens,
|
||||
&critique_input,
|
||||
context.accounting_origin.clone(),
|
||||
),
|
||||
)
|
||||
.await?;
|
||||
@@ -472,6 +481,13 @@ self-check of whether what you just did is actually correct and complete."
|
||||
if let Some(reason) = &run.incomplete_stop_reason {
|
||||
let mut metadata = json!({ "tool": "verify" });
|
||||
crate::cost_status::attach_child_usage_metadata(&mut metadata, &route, &run.usage);
|
||||
crate::cost_status::attach_child_usage_batch_metadata(
|
||||
&mut metadata,
|
||||
&crate::cost_status::RuntimeUsageBatch {
|
||||
records: vec![run.source_record.clone()],
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
return Ok(ToolResult::error(format!(
|
||||
"Verify critic response incomplete: provider stop reason `{reason}`; the partial critique was not accepted."
|
||||
))
|
||||
@@ -489,6 +505,13 @@ self-check of whether what you just did is actually correct and complete."
|
||||
// Previously this reported only input/output, so a verify child's cache
|
||||
// reads *and* cache writes were invisible to the cost audit (#4318).
|
||||
crate::cost_status::attach_child_usage_metadata(&mut metadata, &route, &run.usage);
|
||||
crate::cost_status::attach_child_usage_batch_metadata(
|
||||
&mut metadata,
|
||||
&crate::cost_status::RuntimeUsageBatch {
|
||||
records: vec![run.source_record.clone()],
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
|
||||
let result = ToolResult::json(&run.report)
|
||||
.map_err(|e| ToolError::execution_failed(e.to_string()))?;
|
||||
@@ -504,13 +527,21 @@ async fn run_critique<C: LlmClient>(
|
||||
effort: ReasoningEffort,
|
||||
max_tokens: u32,
|
||||
input: &CritiqueInput,
|
||||
origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
) -> Result<CritiqueRun, ToolError> {
|
||||
let prompt = build_critic_prompt(input, DEFAULT_MAX_EVIDENCE_CHARS);
|
||||
let request = build_critic_request(model, effort, max_tokens, prompt);
|
||||
let response = client
|
||||
.create_message(request)
|
||||
.await
|
||||
.map_err(|e| ToolError::execution_failed(format!("verify critic request failed: {e}")))?;
|
||||
let (response, source_record) = super::review_core::accounted_model_call(
|
||||
client,
|
||||
request,
|
||||
origin,
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
false,
|
||||
)
|
||||
.await
|
||||
.map_err(|error| {
|
||||
ToolError::execution_failed(format!("verify critic request failed: {error}"))
|
||||
})?;
|
||||
let incomplete_stop_reason =
|
||||
codewhale_models::is_incomplete_stop_reason(response.stop_reason.as_deref()).then(|| {
|
||||
codewhale_models::stop_reason_detail(response.stop_reason.as_deref()).to_string()
|
||||
@@ -519,6 +550,7 @@ async fn run_critique<C: LlmClient>(
|
||||
Ok(CritiqueRun {
|
||||
report: CritiqueReport::from_model_text(&text),
|
||||
usage: response.usage,
|
||||
source_record,
|
||||
incomplete_stop_reason,
|
||||
})
|
||||
}
|
||||
@@ -961,6 +993,7 @@ mod tests {
|
||||
ReasoningEffort::Max,
|
||||
65_536,
|
||||
&planted_bug_input(),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect("critique runs");
|
||||
@@ -1002,6 +1035,7 @@ mod tests {
|
||||
ReasoningEffort::High,
|
||||
65_536,
|
||||
&planted_bug_input(),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect("runs");
|
||||
|
||||
@@ -1411,8 +1411,9 @@ async fn start_workflow(
|
||||
let fleet = if let Some(name) = workflow_fleet_name(&input)? {
|
||||
let workspace = context.workspace.clone();
|
||||
let api_config = runtime.api_config.clone();
|
||||
let accounting_origin = context.accounting_origin.clone();
|
||||
tokio::task::spawn_blocking(move || {
|
||||
workflow_fleet_binding(&name, &workspace, api_config.as_deref())
|
||||
workflow_fleet_binding(&name, &workspace, api_config.as_deref(), accounting_origin)
|
||||
})
|
||||
.await
|
||||
.map_err(|error| ToolError::execution_failed(format!("Fleet loading failed: {error}")))??
|
||||
@@ -1800,6 +1801,7 @@ fn workflow_fleet_binding(
|
||||
name: &str,
|
||||
workspace: &std::path::Path,
|
||||
api_config: Option<&codewhale_runtime::config::Config>,
|
||||
accounting_origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
) -> Result<WorkflowFleetBinding, ToolError> {
|
||||
let roots = crate::fleet::exact::fleet_search_roots(workspace);
|
||||
let (document, id) = crate::fleet::exact::load_fleet_document(name, workspace, api_config)
|
||||
@@ -1839,6 +1841,7 @@ fn workflow_fleet_binding(
|
||||
chrono::Utc::now().to_rfc3339(),
|
||||
api_config,
|
||||
&roots,
|
||||
accounting_origin,
|
||||
)
|
||||
.map_err(ToolError::invalid_input)?;
|
||||
Ok(WorkflowFleetBinding::Exact(Arc::new(operation)))
|
||||
|
||||
@@ -304,6 +304,7 @@ async fn exact_fleet_frozen_route_survives_a_conflicting_selected_roster() {
|
||||
"2026-09-13T00:00:00Z",
|
||||
Some(&fixture.config),
|
||||
&[],
|
||||
None,
|
||||
)
|
||||
.expect("preflight exact route against loopback provider");
|
||||
select_role(root.path(), "deepseek-v4-pro", "deepseek");
|
||||
@@ -368,6 +369,7 @@ async fn exact_fleet_changed_provider_endpoint_is_rejected_before_dispatch() {
|
||||
"2026-09-13T00:00:00Z",
|
||||
Some(&fixture.config),
|
||||
&[],
|
||||
None,
|
||||
)
|
||||
.expect("capture exact route");
|
||||
let mut changed = fixture.config.clone();
|
||||
@@ -412,6 +414,7 @@ async fn native_exact_fleet_builder_keeps_the_plan_read_only_ceiling() {
|
||||
"2026-09-13T00:00:00Z",
|
||||
Some(&fixture.config),
|
||||
&[],
|
||||
None,
|
||||
)
|
||||
.expect("capture exact builder");
|
||||
let source =
|
||||
|
||||
@@ -30,6 +30,7 @@ use codewhale_runtime::config::{Config, ProviderIdentity};
|
||||
/// captured `UserDispatchPrepare` fields and `/preview-request` can pass a
|
||||
/// hypothetical prompt, without either one duplicating the other's logic.
|
||||
pub struct TurnRoutePlanRequest<'a> {
|
||||
pub accounting_origin: Option<crate::cost_status::NativeUsageOrigin>,
|
||||
pub route_config: &'a Config,
|
||||
pub app_route_identity: &'a ProviderIdentity,
|
||||
pub api_provider: ProviderKind,
|
||||
@@ -158,6 +159,13 @@ pub async fn plan_turn_route(
|
||||
request.reasoning_effort,
|
||||
),
|
||||
request.allow_auto_router_response_cache,
|
||||
request.accounting_origin.clone().map(|origin| {
|
||||
crate::cost_status::NativeUsageSource::capture_with_origin(
|
||||
format!("auto-router:{}", uuid::Uuid::new_v4()),
|
||||
crate::cost_status::UsageSpendOwner::Auxiliary,
|
||||
Some(origin),
|
||||
)
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.map_err(|err| err.to_string())?,
|
||||
@@ -341,9 +349,9 @@ mod tests {
|
||||
);
|
||||
let batch = crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: vec![crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: "auto-router:plan-usage".to_string(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
records: vec![crate::cost_status::RuntimeUsageRecord::legacy(
|
||||
"auto-router:plan-usage".to_string(),
|
||||
crate::cost_status::EffectiveRouteUsage {
|
||||
route: route.clone(),
|
||||
usage: codewhale_models::Usage {
|
||||
input_tokens: 4,
|
||||
@@ -351,7 +359,7 @@ mod tests {
|
||||
..Default::default()
|
||||
},
|
||||
},
|
||||
}],
|
||||
)],
|
||||
drop_records: vec![crate::cost_status::RuntimeUsageDropRecord {
|
||||
reason: crate::cost_status::RuntimeUsageMissingReason::default(),
|
||||
source_id: "auto-router:plan-drop".to_string(),
|
||||
@@ -426,6 +434,7 @@ mod tests {
|
||||
let identity = deepseek_identity();
|
||||
|
||||
let planned = plan_turn_route(TurnRoutePlanRequest {
|
||||
accounting_origin: None,
|
||||
route_config: &config,
|
||||
app_route_identity: &identity,
|
||||
api_provider: ProviderKind::Deepseek,
|
||||
|
||||
@@ -1,12 +1,14 @@
|
||||
use std::sync::Arc;
|
||||
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
use crate::cost_status::EffectiveRouteUsage;
|
||||
use crate::cost_status::{
|
||||
CostScopeToken, EffectiveRouteEnvelope, EffectiveRouteUsage, RuntimeUsageDropRecord,
|
||||
RuntimeUsageMissingReason, RuntimeUsageRecord, acknowledge_retired_usage_source,
|
||||
record_interactive_runtime_usage, record_interactive_runtime_usage_drop,
|
||||
register_runtime_decision_sink, register_runtime_usage_sink_with_drop,
|
||||
report_interactive_decision,
|
||||
CostScopeToken, EffectiveRouteEnvelope, RuntimeUsageDropRecord, RuntimeUsageMissingReason,
|
||||
RuntimeUsageRecord, acknowledge_retired_usage_source, record_interactive_runtime_usage,
|
||||
record_interactive_runtime_usage_drop, register_interactive_runtime_usage_sink_with_projection,
|
||||
register_runtime_decision_sink, report_interactive_decision,
|
||||
};
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
use codewhale_models::Usage;
|
||||
|
||||
/// Install an interactive sink whose stale-scope fallback is an origin-session
|
||||
@@ -49,20 +51,38 @@ pub fn register_persistent_interactive_runtime_usage_sink_at(
|
||||
let decision_sessions_dir = sessions_dir.clone();
|
||||
let decision_session_id = session_id.to_string();
|
||||
let decision_turn_id = turn_id.to_string();
|
||||
register_runtime_usage_sink_with_drop(
|
||||
let projection_sessions_dir = usage_sessions_dir.clone();
|
||||
register_interactive_runtime_usage_sink_with_projection(
|
||||
owner,
|
||||
Arc::new(move |record| {
|
||||
crate::session_manager::SessionManager::new(usage_sessions_dir.clone())
|
||||
.map(|manager| {
|
||||
report_effective_route_for_interactive_origin_with_manager(
|
||||
scope,
|
||||
&usage_session_id,
|
||||
&usage_turn_id,
|
||||
&record.source_id,
|
||||
&record.usage.route,
|
||||
&record.usage.usage,
|
||||
&manager,
|
||||
)
|
||||
if record.settlement.is_some() {
|
||||
return report_runtime_usage_record_for_origin_with_manager(
|
||||
scope,
|
||||
&usage_session_id,
|
||||
&usage_turn_id,
|
||||
&record,
|
||||
&manager,
|
||||
)
|
||||
.is_ok();
|
||||
}
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
{
|
||||
report_effective_route_for_interactive_origin_with_manager(
|
||||
scope,
|
||||
&usage_session_id,
|
||||
&usage_turn_id,
|
||||
&record.source_id,
|
||||
&record.usage.route,
|
||||
&record.usage.usage,
|
||||
&manager,
|
||||
)
|
||||
}
|
||||
#[cfg(not(any(test, feature = "test-support")))]
|
||||
{
|
||||
false
|
||||
}
|
||||
})
|
||||
.unwrap_or(false)
|
||||
}),
|
||||
@@ -81,6 +101,8 @@ pub fn register_persistent_interactive_runtime_usage_sink_at(
|
||||
})
|
||||
.unwrap_or(false)
|
||||
})),
|
||||
projection_sessions_dir,
|
||||
session_id.to_string(),
|
||||
);
|
||||
register_runtime_decision_sink(
|
||||
owner,
|
||||
@@ -123,29 +145,43 @@ pub fn register_persistent_interactive_runtime_usage_sink_for_test(
|
||||
);
|
||||
}
|
||||
|
||||
/// Report an interactive auxiliary response against its immutable origin.
|
||||
/// A stale foreground scope is not an error: it means `/new` or session load
|
||||
/// already moved on, so the exact receipt is appended to the old session's
|
||||
/// durable sidecar instead of being redirected to the active session.
|
||||
pub fn report_effective_route_for_interactive_origin(
|
||||
pub(crate) fn report_runtime_usage_record_for_origin_with_manager(
|
||||
scope: CostScopeToken,
|
||||
session_id: &str,
|
||||
turn_id: &str,
|
||||
source_id: &str,
|
||||
route: &EffectiveRouteEnvelope,
|
||||
usage: &Usage,
|
||||
) {
|
||||
let persisted =
|
||||
crate::session_manager::SessionManager::default_location().is_ok_and(|manager| {
|
||||
report_effective_route_for_interactive_origin_with_manager(
|
||||
scope, session_id, turn_id, source_id, route, usage, &manager,
|
||||
)
|
||||
});
|
||||
if !persisted {
|
||||
tracing::warn!("late interactive usage could not be persisted for its origin session");
|
||||
record: &RuntimeUsageRecord,
|
||||
manager: &crate::session_manager::SessionManager,
|
||||
) -> std::io::Result<()> {
|
||||
let mut publication = Ok(());
|
||||
let admitted = manager.with_live_session_origin(session_id, || {
|
||||
publication = manager
|
||||
.persist_runtime_usage_record_admitted(session_id, turn_id, record)
|
||||
.and_then(|_| {
|
||||
if record.has_native_producer() {
|
||||
record
|
||||
.acknowledge_persisted()
|
||||
.and_then(|_| record.acknowledge_live_projection())
|
||||
} else {
|
||||
Ok(())
|
||||
}
|
||||
});
|
||||
if publication.is_err() {
|
||||
return false;
|
||||
}
|
||||
let _ = record_interactive_runtime_usage(scope, record.clone());
|
||||
true
|
||||
})?;
|
||||
publication?;
|
||||
if admitted.is_none() {
|
||||
acknowledge_retired_usage_source(scope, &record.source_id);
|
||||
if record.has_native_producer() {
|
||||
record.acknowledge_persisted()?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
pub fn report_effective_route_for_interactive_origin_with_manager(
|
||||
scope: CostScopeToken,
|
||||
session_id: &str,
|
||||
@@ -155,22 +191,29 @@ pub fn report_effective_route_for_interactive_origin_with_manager(
|
||||
usage: &Usage,
|
||||
manager: &crate::session_manager::SessionManager,
|
||||
) -> bool {
|
||||
let record = RuntimeUsageRecord {
|
||||
source_id: source_id.to_string(),
|
||||
usage: EffectiveRouteUsage {
|
||||
let record = RuntimeUsageRecord::legacy(
|
||||
source_id.to_string(),
|
||||
EffectiveRouteUsage {
|
||||
route: route.sanitized_for_persistence(),
|
||||
usage: usage.clone(),
|
||||
},
|
||||
};
|
||||
);
|
||||
let mut durability = Ok(true);
|
||||
match manager.with_live_session_origin(session_id, || {
|
||||
record_interactive_runtime_usage(scope, record.clone())
|
||||
durability = manager.persist_legacy_usage_admitted(session_id, turn_id, &record);
|
||||
durability.as_ref().is_ok_and(|handled| *handled)
|
||||
&& record_interactive_runtime_usage(scope, record.clone())
|
||||
}) {
|
||||
Ok(None) => {
|
||||
acknowledge_retired_usage_source(scope, source_id);
|
||||
return true;
|
||||
}
|
||||
Ok(Some(true)) => return true,
|
||||
Ok(Some(false)) => {}
|
||||
Ok(Some(false)) => {
|
||||
if durability.is_err() {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
Err(_) => return false,
|
||||
}
|
||||
manager
|
||||
|
||||
@@ -714,6 +714,8 @@ mod tests {
|
||||
engine
|
||||
.tx_event
|
||||
.send(Event::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
|
||||
@@ -296,6 +296,25 @@ pub(crate) async fn run_exec_agent(
|
||||
|| fleet_capture.is_some()
|
||||
|| output_format == ExecOutputFormat::StreamJson;
|
||||
let mut exec_session_manager = fleet_capture.map(|(_, manager)| Arc::new(manager));
|
||||
let _exec_operation_lease = if exec_is_fresh {
|
||||
let (manager, source, lease) = admit_cli_saved_operation(
|
||||
exec_session_manager.clone(),
|
||||
exec_session_id.clone(),
|
||||
exec_session_workspace.clone(),
|
||||
model.to_string(),
|
||||
config
|
||||
.active_provider_identity()
|
||||
.map_err(anyhow::Error::msg)?,
|
||||
prompt.to_string(),
|
||||
)
|
||||
.await
|
||||
.context("could not admit exec's saved accounting operation")?;
|
||||
exec_session_manager = Some(manager);
|
||||
exec_write_receipt = Some(source);
|
||||
Some(lease)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let mut open_exec_session_manager = || -> Result<Arc<SessionManager>> {
|
||||
if exec_session_manager.is_none() {
|
||||
exec_session_manager = Some(Arc::new(
|
||||
@@ -309,13 +328,21 @@ pub(crate) async fn run_exec_agent(
|
||||
))
|
||||
};
|
||||
if should_persist_session {
|
||||
let manager = open_exec_session_manager()?;
|
||||
if exec_is_fresh {
|
||||
exec_write_receipt = Some(manager.expect_absent_session(&exec_session_id)?);
|
||||
}
|
||||
open_exec_session_manager()?;
|
||||
}
|
||||
|
||||
let route = resolve_cli_exec_route(config, model, prompt, force_configured_route).await?;
|
||||
let accounting_origin = exec_write_receipt
|
||||
.as_ref()
|
||||
.ok_or_else(|| anyhow::anyhow!("exec has no admitted original session source"))?
|
||||
.usage_origin(codewhale_engine::cost_status::scope_token())?;
|
||||
let route = resolve_cli_exec_route(
|
||||
config,
|
||||
model,
|
||||
prompt,
|
||||
force_configured_route,
|
||||
Some(accounting_origin.clone()),
|
||||
)
|
||||
.await?;
|
||||
let execution_config = config_for_cli_route(config, &route)?;
|
||||
codewhale_engine::provider_lake::refresh_account_catalog_before_first_request(
|
||||
&execution_config,
|
||||
@@ -497,6 +524,7 @@ pub(crate) async fn run_exec_agent(
|
||||
codewhale_engine::tools::shell::new_shared_shell_manager(workspace.clone());
|
||||
let exec_automations = exec_automation_services(fleet_authority_active || one_shot)?;
|
||||
let runtime_services = codewhale_engine::tools::spec::RuntimeToolServices {
|
||||
accounting_origin: Some(accounting_origin),
|
||||
shell_manager: Some(exec_shell_manager.clone()),
|
||||
persist_services_enabled,
|
||||
automations: exec_automations,
|
||||
@@ -1774,6 +1802,8 @@ mod tests {
|
||||
..Usage::default()
|
||||
};
|
||||
Event::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
|
||||
+261
-35
@@ -10563,6 +10563,53 @@ fn pick_session_id() -> Result<String> {
|
||||
Ok(session.id.clone())
|
||||
}
|
||||
|
||||
async fn admit_cli_saved_operation(
|
||||
manager: Option<std::sync::Arc<session_manager::SessionManager>>,
|
||||
id: String,
|
||||
workspace: PathBuf,
|
||||
model: String,
|
||||
provider: codewhale_runtime::config::ProviderIdentity,
|
||||
prompt: String,
|
||||
) -> Result<(
|
||||
std::sync::Arc<session_manager::SessionManager>,
|
||||
session_manager::SessionWriteReceipt,
|
||||
session_manager::SessionLease,
|
||||
)> {
|
||||
codewhale_runtime::runtime_control::owner_work(move || {
|
||||
let manager = match manager {
|
||||
Some(manager) => manager,
|
||||
None => std::sync::Arc::new(session_manager::SessionManager::default_location()?),
|
||||
};
|
||||
let lease = manager.reserve_session_for_external_write(&id)?;
|
||||
let source = manager.expect_absent_session(&id)?;
|
||||
let messages = vec![codewhale_models::Message {
|
||||
role: codewhale_models::Role::User,
|
||||
content: vec![codewhale_models::ContentBlock::Text {
|
||||
text: prompt,
|
||||
cache_control: None,
|
||||
}],
|
||||
}];
|
||||
let mut saved = session_manager::create_saved_session_with_id_and_mode(
|
||||
id,
|
||||
&messages,
|
||||
&model,
|
||||
&workspace,
|
||||
0,
|
||||
None,
|
||||
Some("exec"),
|
||||
);
|
||||
let (kind, provider_id) = exec_stream_provider_route(&provider);
|
||||
saved.metadata.model_provider = kind;
|
||||
saved.metadata.model_provider_id = provider_id;
|
||||
saved.metadata.cost.coverage_recorded = true;
|
||||
saved.write_receipt = Some(source);
|
||||
saved.capture_publication(session_manager::SessionWriteOwner::control())?;
|
||||
let publication = manager.publish_session(&saved)?;
|
||||
Ok((manager, publication.write_receipt(), lease))
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
async fn run_review(config: &Config, args: ReviewArgs) -> Result<()> {
|
||||
initialize_cloud_facts(config);
|
||||
use codewhale_engine::client::CodewhaleClient;
|
||||
@@ -10622,9 +10669,35 @@ async fn run_review(config: &Config, args: ReviewArgs) -> Result<()> {
|
||||
})
|
||||
.transpose()?;
|
||||
let review_workspace = std::env::current_dir()?;
|
||||
// A stateless CLI review has no Engine session or Native caller to invent.
|
||||
let review_context = codewhale_engine::tools::spec::ToolContext::new(&review_workspace)
|
||||
let model = resolve_review_model(config, args.model.as_deref());
|
||||
anyhow::ensure!(
|
||||
args.money_cap_usd.is_none() || !model.trim().eq_ignore_ascii_case("auto"),
|
||||
"--money-cap-usd requires a fixed review model; Auto may dispatch a separately priced classifier"
|
||||
);
|
||||
let review_rates = match (args.input_usd_per_mtok, args.output_usd_per_mtok) {
|
||||
(Some(input), Some(output)) => Some(codewhale_engine::tools::review_triage::SpendRates {
|
||||
input_usd_per_million: input,
|
||||
output_usd_per_million: output,
|
||||
}),
|
||||
_ => None,
|
||||
};
|
||||
let (_, review_source, _review_lease) = admit_cli_saved_operation(
|
||||
None,
|
||||
uuid::Uuid::new_v4().to_string(),
|
||||
review_workspace.clone(),
|
||||
model.clone(),
|
||||
config
|
||||
.active_provider_identity()
|
||||
.map_err(anyhow::Error::msg)?,
|
||||
format!("Review the following diff:\n\n{diff}"),
|
||||
)
|
||||
.await
|
||||
.context("could not admit the review's saved accounting operation")?;
|
||||
let review_accounting_origin =
|
||||
review_source.usage_origin(codewhale_engine::cost_status::scope_token())?;
|
||||
let mut review_context = codewhale_engine::tools::spec::ToolContext::new(&review_workspace)
|
||||
.with_features(config.features());
|
||||
review_context.accounting_origin = Some(review_accounting_origin.clone());
|
||||
// #6510: one review prompt authority. Plain diffs and PRs both ask for
|
||||
// the structured review contract; a plain diff renders it locally.
|
||||
//
|
||||
@@ -10657,13 +10730,19 @@ async fn run_review(config: &Config, args: ReviewArgs) -> Result<()> {
|
||||
format!("Review the following diff and provide feedback:\n\n{diff}\n\nEnd of diff.")
|
||||
}]
|
||||
};
|
||||
let model = resolve_review_model(config, args.model.as_deref());
|
||||
let route_input = prompts
|
||||
.iter()
|
||||
.max_by_key(|prompt| prompt.chars().count())
|
||||
.expect("review has at least one prompt");
|
||||
let route = resolve_cli_exec_route(config, &model, route_input, force_configured_route).await?;
|
||||
let execution_config = config_for_cli_route(config, &route)?;
|
||||
let route = resolve_cli_exec_route(
|
||||
config,
|
||||
&model,
|
||||
route_input,
|
||||
force_configured_route,
|
||||
Some(review_accounting_origin.clone()),
|
||||
)
|
||||
.await?;
|
||||
let execution_config = review_config_with_spend_rates(config, &route, review_rates)?;
|
||||
codewhale_engine::provider_lake::refresh_account_catalog_before_first_request(
|
||||
&execution_config,
|
||||
)
|
||||
@@ -10700,13 +10779,6 @@ async fn run_review(config: &Config, args: ReviewArgs) -> Result<()> {
|
||||
.as_ref()
|
||||
.map(codewhale_engine::tools::review::PrReviewAccumulator::new);
|
||||
let review_triage = codewhale_engine::tools::review_triage::triage_review(&diff);
|
||||
let review_rates = match (args.input_usd_per_mtok, args.output_usd_per_mtok) {
|
||||
(Some(input), Some(output)) => Some(codewhale_engine::tools::review_triage::SpendRates {
|
||||
input_usd_per_million: input,
|
||||
output_usd_per_million: output,
|
||||
}),
|
||||
_ => None,
|
||||
};
|
||||
// Visible-text reserve for this exact route/model (#6285). A reasoning route
|
||||
// shares one `max_tokens` allowance between hidden reasoning and visible
|
||||
// text, so the review request has to keep room for the review itself.
|
||||
@@ -10739,7 +10811,7 @@ async fn run_review(config: &Config, args: ReviewArgs) -> Result<()> {
|
||||
let mut review_anchors = codewhale_engine::tools::review_core::ReviewAnchorReport::default();
|
||||
let mut review_tools_used: Vec<String> = Vec::new();
|
||||
let mut review_stop: Option<codewhale_engine::tools::review_triage::ReviewStop> = None;
|
||||
let mut review_spend_usd = None;
|
||||
let mut review_spend_usd = review_budget.money_limit_supported.then_some(0.0);
|
||||
for (index, user_prompt) in prompts.into_iter().enumerate() {
|
||||
let reasoning_effort = route.reasoning_effort.and_then(|effort| {
|
||||
review_reasoning_effort_value_for_prompt(
|
||||
@@ -10756,10 +10828,22 @@ async fn run_review(config: &Config, args: ReviewArgs) -> Result<()> {
|
||||
request.budget = review_budget.tool_budget();
|
||||
request.max_output_tokens = Some(review_allowance);
|
||||
request.reasoning_effort = reasoning_effort;
|
||||
request.spend_rates = review_rates;
|
||||
request.money_cap_usd = review_budget
|
||||
request.accounting_origin = Some(review_accounting_origin.clone());
|
||||
if let Some(cap) = review_budget
|
||||
.money_cap_usd
|
||||
.filter(|_| review_budget.money_limit_supported);
|
||||
.filter(|_| review_budget.money_limit_supported)
|
||||
{
|
||||
let remaining = cap - review_spend_usd.unwrap_or(0.0);
|
||||
if remaining <= 0.0 {
|
||||
return report_failure(
|
||||
&usage,
|
||||
index,
|
||||
publication,
|
||||
"the per-review money cap was reached; no additional pass was dispatched and no partial review was accepted or posted".to_string(),
|
||||
);
|
||||
}
|
||||
request.money_cap_usd = Some(remaining);
|
||||
}
|
||||
request.intent = Some(intent.clone());
|
||||
request.refute = matches!(
|
||||
review_triage.risk,
|
||||
@@ -10797,7 +10881,26 @@ async fn run_review(config: &Config, args: ReviewArgs) -> Result<()> {
|
||||
review_anchors.anchored += result.anchors.anchored;
|
||||
review_anchors.unanchored += result.anchors.unanchored;
|
||||
review_stop = result.stop.clone().or(review_stop);
|
||||
review_spend_usd = result.spend_usd.or(review_spend_usd);
|
||||
if review_budget.money_limit_supported {
|
||||
let Some(spend) = result.spend_usd else {
|
||||
return report_failure(
|
||||
&usage,
|
||||
index,
|
||||
publication,
|
||||
"the review's frozen USD source receipts could not verify its money cap; no partial review was accepted or posted".to_string(),
|
||||
);
|
||||
};
|
||||
let total = review_spend_usd.unwrap_or(0.0) + spend;
|
||||
if !total.is_finite() {
|
||||
return report_failure(
|
||||
&usage,
|
||||
index,
|
||||
publication,
|
||||
"the review's accumulated source spend is not finite; no partial review was accepted or posted".to_string(),
|
||||
);
|
||||
}
|
||||
review_spend_usd = Some(total);
|
||||
}
|
||||
if codewhale_models::is_incomplete_stop_reason(review_stop_reason.as_deref()) {
|
||||
// #6285: budget exhaustion is an infrastructure outcome, not a
|
||||
// review verdict. The PR was never judged, so the message must not
|
||||
@@ -11142,6 +11245,8 @@ fn review_failure_payload(
|
||||
"publication": publication.as_str(),
|
||||
"error": message,
|
||||
"usage": usage,
|
||||
"usage_scope": "completed_review_passes",
|
||||
"usage_coverage": "incomplete",
|
||||
"completed_review_passes": completed_passes,
|
||||
"planned_review_passes": planned_passes,
|
||||
})
|
||||
@@ -11175,7 +11280,7 @@ fn report_review_failure(
|
||||
let usage = serde_json::to_string(usage)?;
|
||||
let publication = publication.as_str();
|
||||
bail!(
|
||||
"{message}; publication: {publication}; completed review passes: {completed_passes}/{planned_passes}; accumulated usage: {usage}"
|
||||
"{message}; publication: {publication}; completed review passes: {completed_passes}/{planned_passes}; known usage from completed passes (incomplete total coverage): {usage}"
|
||||
)
|
||||
}
|
||||
|
||||
@@ -11198,6 +11303,62 @@ fn review_execution_route(config: &Config, args: &ReviewArgs) -> Result<(Config,
|
||||
Ok((resolved, force_configured_route))
|
||||
}
|
||||
|
||||
fn review_config_with_spend_rates(
|
||||
config: &Config,
|
||||
route: &CliAutoRoute,
|
||||
rates: Option<codewhale_engine::tools::review_triage::SpendRates>,
|
||||
) -> Result<Config> {
|
||||
let mut scoped = config_for_cli_route(config, route)?;
|
||||
let Some(rates) = rates else {
|
||||
return Ok(scoped);
|
||||
};
|
||||
let client = codewhale_engine::client::CodewhaleClient::new(&scoped)?;
|
||||
let envelope = client.effective_route_envelope(&route.model, chrono::Utc::now());
|
||||
let provider = client.admitted_provider_identity().key.to_string();
|
||||
let endpoint = client.base_url().to_string();
|
||||
let models = scoped.custom_models.get_or_insert_with(Vec::new);
|
||||
let index = models
|
||||
.iter()
|
||||
.position(|row| row.id == envelope.model && row.matches_route(&provider, &endpoint));
|
||||
let row = if let Some(index) = index {
|
||||
&mut models[index]
|
||||
} else {
|
||||
models.push(codewhale_config::catalog::configured::ConfiguredModel {
|
||||
provider,
|
||||
base_url: endpoint,
|
||||
id: envelope.model,
|
||||
display_name: None,
|
||||
limit: None,
|
||||
cost: None,
|
||||
modalities: None,
|
||||
attachment: None,
|
||||
reasoning: None,
|
||||
tool_call: None,
|
||||
structured_output: None,
|
||||
extras: Default::default(),
|
||||
});
|
||||
models
|
||||
.last_mut()
|
||||
.expect("inserted review route declaration")
|
||||
};
|
||||
row.cost = Some(codewhale_config::models_dev::ModelsDevCost {
|
||||
input: Some(rates.input_usd_per_million),
|
||||
output: Some(rates.output_usd_per_million),
|
||||
..Default::default()
|
||||
});
|
||||
codewhale_config::catalog::configured::validate_configured_models(models)?;
|
||||
let priced = codewhale_engine::client::CodewhaleClient::new(&scoped)?
|
||||
.effective_route_envelope(&route.model, chrono::Utc::now());
|
||||
anyhow::ensure!(
|
||||
priced.provider_live_pricing.as_ref().is_some_and(|quote| {
|
||||
quote.provenance == codewhale_config::pricing::PricingProvenance::UserOverride
|
||||
&& quote.currency == codewhale_config::pricing::Currency::Usd
|
||||
}),
|
||||
"the explicit review rates could not be frozen for this exact route"
|
||||
);
|
||||
Ok(scoped)
|
||||
}
|
||||
|
||||
fn non_empty_flag(value: Option<&str>) -> Option<&str> {
|
||||
value.map(str::trim).filter(|value| !value.is_empty())
|
||||
}
|
||||
@@ -11211,6 +11372,26 @@ fn resolve_review_model(config: &Config, explicit_model: Option<&str>) -> String
|
||||
}
|
||||
|
||||
fn validate_review_receipt_args(args: &ReviewArgs) -> Result<()> {
|
||||
anyhow::ensure!(
|
||||
args.money_cap_usd
|
||||
.is_none_or(|cap| cap.is_finite() && cap > 0.0),
|
||||
"--money-cap-usd must be finite and greater than zero"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
args.input_usd_per_mtok.is_some() == args.output_usd_per_mtok.is_some(),
|
||||
"--input-usd-per-mtok and --output-usd-per-mtok must be supplied together"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
[args.input_usd_per_mtok, args.output_usd_per_mtok]
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.all(|rate| rate.is_finite() && rate >= 0.0),
|
||||
"review token rates must be finite and nonnegative"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
args.input_usd_per_mtok.is_none() || args.money_cap_usd.is_some(),
|
||||
"review token rates require --money-cap-usd"
|
||||
);
|
||||
if args.receipt_path.is_some() && !args.write_receipt && !args.check_receipt {
|
||||
bail!("--receipt-path requires --write-receipt or --check-receipt");
|
||||
}
|
||||
@@ -14105,14 +14286,30 @@ async fn resolve_cli_auto_route(
|
||||
config: &Config,
|
||||
model: &str,
|
||||
prompt: &str,
|
||||
accounting_origin: Option<codewhale_engine::cost_status::NativeUsageOrigin>,
|
||||
) -> Result<CliAutoRoute> {
|
||||
let active = config
|
||||
.active_provider_identity()
|
||||
.map_err(anyhow::Error::msg)?;
|
||||
if model.trim().eq_ignore_ascii_case("auto") {
|
||||
let selection =
|
||||
model_routing::resolve_auto_route_with_inventory(config, prompt, "", "auto", "auto")
|
||||
.await?;
|
||||
model_routing::resolve_auto_route_with_inventory_for_session_and_cache_policy(
|
||||
config,
|
||||
prompt,
|
||||
"",
|
||||
"agent",
|
||||
"auto",
|
||||
"auto",
|
||||
true,
|
||||
accounting_origin.map(|origin| {
|
||||
codewhale_engine::cost_status::NativeUsageSource::capture_with_origin(
|
||||
format!("cli-auto-router:{}", uuid::Uuid::new_v4()),
|
||||
codewhale_engine::cost_status::UsageSpendOwner::Auxiliary,
|
||||
Some(origin),
|
||||
)
|
||||
}),
|
||||
)
|
||||
.await?;
|
||||
let preference = config
|
||||
.reasoning_effort()
|
||||
.filter(|_| config.reasoning_effort_is_explicit())
|
||||
@@ -14183,6 +14380,7 @@ async fn resolve_cli_exec_route(
|
||||
model: &str,
|
||||
prompt: &str,
|
||||
force_configured_route: bool,
|
||||
accounting_origin: Option<codewhale_engine::cost_status::NativeUsageOrigin>,
|
||||
) -> Result<CliAutoRoute> {
|
||||
let mut local_default = None;
|
||||
if let Ok(identity) = config.active_provider_identity()
|
||||
@@ -14219,7 +14417,7 @@ async fn resolve_cli_exec_route(
|
||||
auto_model: false,
|
||||
});
|
||||
}
|
||||
resolve_cli_auto_route(config, model, prompt).await
|
||||
resolve_cli_auto_route(config, model, prompt, accounting_origin).await
|
||||
}
|
||||
|
||||
fn should_force_configured_exec_route(
|
||||
@@ -14631,11 +14829,27 @@ async fn run_workflow_tool_command_inner(
|
||||
|
||||
initialize_cloud_facts(&config);
|
||||
let model = resolve_exec_model(&config, None);
|
||||
let workflow_session_id = uuid::Uuid::new_v4().to_string();
|
||||
let (_, workflow_source, _workflow_lease) = admit_cli_saved_operation(
|
||||
None,
|
||||
workflow_session_id.clone(),
|
||||
workspace.clone(),
|
||||
model.clone(),
|
||||
config
|
||||
.active_provider_identity()
|
||||
.map_err(anyhow::Error::msg)?,
|
||||
format!("Run this Workflow:\n{}", serde_json::to_string(&input)?),
|
||||
)
|
||||
.await
|
||||
.context("could not admit the Workflow's saved accounting operation")?;
|
||||
let workflow_accounting_origin =
|
||||
workflow_source.usage_origin(codewhale_engine::cost_status::scope_token())?;
|
||||
let route = resolve_cli_exec_route(
|
||||
&config,
|
||||
&model,
|
||||
"Run a checked-in Workflow through the host runtime",
|
||||
true,
|
||||
Some(workflow_accounting_origin.clone()),
|
||||
)
|
||||
.await?;
|
||||
let execution_config = config_for_cli_route(&config, &route)?;
|
||||
@@ -14672,6 +14886,7 @@ async fn run_workflow_tool_command_inner(
|
||||
&workspace,
|
||||
event_tx,
|
||||
plugin_registry,
|
||||
workflow_accounting_origin,
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -14754,10 +14969,10 @@ async fn run_workflow_tool_command_inner(
|
||||
input_analysis: ExecStreamInputAnalysis::default(),
|
||||
visible_final_answer_chars: result.content.chars().count(),
|
||||
visible_final_answer_excerpt: exec_stream_final_answer_excerpt(&result.content),
|
||||
session_id: String::new(),
|
||||
resume_command: String::new(),
|
||||
session_id: workflow_session_id.clone(),
|
||||
resume_command: format!("codewhale --resume {workflow_session_id}"),
|
||||
workspace: workspace.display().to_string(),
|
||||
message_count: 0,
|
||||
message_count: 1,
|
||||
status: Some(workflow_status.clone()),
|
||||
termination_reason: Some(if completed { "resolved" } else { "tool_error" }.to_string()),
|
||||
error_category: (!completed).then(|| "tool".to_string()),
|
||||
@@ -14841,6 +15056,7 @@ async fn build_direct_workflow_tool(
|
||||
workspace: &Path,
|
||||
event_tx: tokio::sync::mpsc::Sender<codewhale_engine::core::events::Event>,
|
||||
plugin_registry: std::sync::Arc<codewhale_plugins::PluginRegistry>,
|
||||
accounting_origin: codewhale_engine::cost_status::NativeUsageOrigin,
|
||||
) -> Result<(
|
||||
codewhale_engine::tools::workflow::WorkflowTool,
|
||||
codewhale_engine::tools::ToolContext,
|
||||
@@ -14903,6 +15119,7 @@ async fn build_direct_workflow_tool(
|
||||
config.sandbox_network_access,
|
||||
),
|
||||
));
|
||||
context.accounting_origin = Some(accounting_origin);
|
||||
let network_policy = config.network.clone().map(|network| {
|
||||
codewhale_net::network_policy::NetworkPolicyDecider::with_default_audit(
|
||||
codewhale_net::network_policy::NetworkPolicy::from(network),
|
||||
@@ -15265,21 +15482,30 @@ fn persist_exec_session(
|
||||
if checkpoint.is_some() {
|
||||
captured.require_checkpoint_acknowledgement();
|
||||
}
|
||||
let native_accounting = session_id
|
||||
.map(|id| manager.has_native_usage_records(id))
|
||||
.transpose()?
|
||||
.unwrap_or(false)
|
||||
|| provider_request_receipts
|
||||
.iter()
|
||||
.any(|receipt| receipt["source_accounting"].as_bool() == Some(true));
|
||||
let mut saved = if let Some(id) = session_id.filter(|id| !id.trim().is_empty()) {
|
||||
match manager.load_session_snapshot(id) {
|
||||
Ok(existing) => session_manager::update_session(
|
||||
existing,
|
||||
messages,
|
||||
total_tokens,
|
||||
system_prompt.as_ref(),
|
||||
),
|
||||
Ok(existing) => {
|
||||
let tokens = if native_accounting {
|
||||
existing.metadata.total_tokens
|
||||
} else {
|
||||
total_tokens
|
||||
};
|
||||
session_manager::update_session(existing, messages, tokens, system_prompt.as_ref())
|
||||
}
|
||||
Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
|
||||
session_manager::create_saved_session_with_id_and_mode(
|
||||
id.to_string(),
|
||||
messages,
|
||||
model,
|
||||
workspace,
|
||||
total_tokens,
|
||||
if native_accounting { 0 } else { total_tokens },
|
||||
system_prompt.as_ref(),
|
||||
Some("exec"),
|
||||
)
|
||||
@@ -15334,10 +15560,10 @@ fn persist_exec_session(
|
||||
}
|
||||
}
|
||||
}
|
||||
for receipt in provider_request_receipts
|
||||
.iter()
|
||||
.filter(|receipt| receipt["primary"].as_bool() == Some(true))
|
||||
{
|
||||
for receipt in provider_request_receipts.iter().filter(|receipt| {
|
||||
receipt["primary"].as_bool() == Some(true)
|
||||
&& receipt["source_accounting"].as_bool() != Some(true)
|
||||
}) {
|
||||
let Some(source) = receipt["source_id"].as_str() else {
|
||||
continue;
|
||||
};
|
||||
|
||||
@@ -4,8 +4,8 @@ fn parse_cli(args: &[&str]) -> Cli {
|
||||
|
||||
#[test]
|
||||
fn worker_command_policy_prompt_that_looks_like_a_flag_parses() {
|
||||
use codewhale_engine::fleet::executor::build_worker_exec_command;
|
||||
use codewhale_config::FleetExecConfig;
|
||||
use codewhale_engine::fleet::executor::build_worker_exec_command;
|
||||
use codewhale_protocol::fleet::FleetTaskSpec;
|
||||
|
||||
let task: FleetTaskSpec = serde_json::from_value(serde_json::json!({
|
||||
@@ -509,8 +509,25 @@ async fn direct_workflow_tool_runs_without_an_operator_model_turn() {
|
||||
};
|
||||
let (event_tx, mut event_rx) = tokio::sync::mpsc::channel(64);
|
||||
let plugins = Arc::new(codewhale_plugins::PluginRegistry::empty(workspace.path()));
|
||||
let manager = Arc::new(
|
||||
session_manager::SessionManager::new(workspace.path().join("sessions"))
|
||||
.expect("isolated Workflow session store"),
|
||||
);
|
||||
let (_, source, _lease) = admit_cli_saved_operation(
|
||||
Some(manager),
|
||||
uuid::Uuid::new_v4().to_string(),
|
||||
workspace.path().to_path_buf(),
|
||||
route.model.clone(),
|
||||
route.provider.clone(),
|
||||
"Run a model-free Workflow".to_string(),
|
||||
)
|
||||
.await
|
||||
.expect("admitted Workflow invocation");
|
||||
let origin = source
|
||||
.usage_origin(codewhale_engine::cost_status::scope_token())
|
||||
.expect("captured Workflow usage origin");
|
||||
let (tool, context) =
|
||||
build_direct_workflow_tool(&config, &route, workspace.path(), event_tx, plugins)
|
||||
build_direct_workflow_tool(&config, &route, workspace.path(), event_tx, plugins, origin)
|
||||
.await
|
||||
.expect("build direct workflow runtime");
|
||||
|
||||
@@ -803,6 +820,7 @@ async fn explicit_exec_model_routes_to_unique_authenticated_provider_candidate()
|
||||
&config,
|
||||
codewhale_config::catalog::reviewed::constants::ZAI_GLM_5_2_MODEL,
|
||||
"pong",
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect("explicit GLM should route to the configured Z.ai provider");
|
||||
@@ -835,6 +853,7 @@ async fn explicit_exec_model_reports_ambiguous_authenticated_provider_candidates
|
||||
&config,
|
||||
codewhale_config::catalog::reviewed::constants::ZAI_GLM_5_2_MODEL,
|
||||
"pong",
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect_err("ambiguous GLM route should ask for an explicit provider");
|
||||
@@ -869,7 +888,7 @@ async fn cli_auto_model_honors_a_fixed_reasoning_preference() {
|
||||
__cw_config_init_13_value
|
||||
};
|
||||
|
||||
let route = resolve_cli_auto_route(&config, "auto", "debug a failing test")
|
||||
let route = resolve_cli_auto_route(&config, "auto", "debug a failing test", None)
|
||||
.await
|
||||
.expect("Auto model route");
|
||||
|
||||
@@ -963,7 +982,8 @@ fn cli_route_execution_config_preserves_the_legacy_literal_custom_route() {
|
||||
.expect("legacy identity remains repeatedly resolvable");
|
||||
assert_eq!(identity.key.as_str(), "custom");
|
||||
}
|
||||
let client = codewhale_engine::client::CodewhaleClient::new(&execution).expect("legacy execution client");
|
||||
let client = codewhale_engine::client::CodewhaleClient::new(&execution)
|
||||
.expect("legacy execution client");
|
||||
assert_eq!(client.base_url(), "http://127.0.0.1:18183/v1");
|
||||
}
|
||||
|
||||
@@ -1529,6 +1549,7 @@ async fn forced_exec_route_keeps_custom_provider_when_model_matches_builtin_cata
|
||||
codewhale_config::catalog::reviewed::constants::ZAI_GLM_5_2_MODEL,
|
||||
"audit",
|
||||
true,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect("forced route");
|
||||
@@ -1577,7 +1598,7 @@ async fn no_flag_exec_keeps_configured_named_custom_route_for_matching_builtin_m
|
||||
Some("override-model")
|
||||
));
|
||||
|
||||
let route = resolve_cli_exec_route(&config, &model, "audit", force)
|
||||
let route = resolve_cli_exec_route(&config, &model, "audit", force, None)
|
||||
.await
|
||||
.expect("no-flag configured route");
|
||||
let execution = config_for_cli_route(&config, &route).expect("admitted execution route");
|
||||
@@ -1611,7 +1632,7 @@ async fn configured_review_default_keeps_named_custom_route_and_exact_receipt()
|
||||
"explicit-review-model"
|
||||
);
|
||||
|
||||
let route = resolve_cli_exec_route(&config, &model, "review diff", true)
|
||||
let route = resolve_cli_exec_route(&config, &model, "review diff", true, None)
|
||||
.await
|
||||
.expect("configured review route");
|
||||
let execution = config_for_cli_route(&config, &route).expect("admitted execution route");
|
||||
@@ -1709,6 +1730,8 @@ fn review_failure_payload_preserves_usage_without_complete_claim() {
|
||||
assert_eq!(payload["completed_review_passes"], 1);
|
||||
assert_eq!(payload["planned_review_passes"], 2);
|
||||
assert_eq!(payload["publication"], "not_attempted");
|
||||
assert_eq!(payload["usage_scope"], "completed_review_passes");
|
||||
assert_eq!(payload["usage_coverage"], "incomplete");
|
||||
assert_eq!(payload["usage"]["input_tokens"], 24);
|
||||
assert_eq!(payload["usage"]["reasoning_tokens"], 3);
|
||||
assert!(payload.get("review").is_none());
|
||||
|
||||
@@ -40,6 +40,8 @@ fn review_failure_after_publication_preserves_all_usage_and_post_state() {
|
||||
assert_eq!(payload["success"], false);
|
||||
assert_eq!(payload["complete"], false);
|
||||
assert_eq!(payload["publication"], publication.as_str());
|
||||
assert_eq!(payload["usage_scope"], "completed_review_passes");
|
||||
assert_eq!(payload["usage_coverage"], "incomplete");
|
||||
assert_eq!(payload["usage"]["input_tokens"], 55);
|
||||
assert_eq!(payload["usage"]["output_tokens"], 13);
|
||||
assert_eq!(payload["usage"]["reasoning_tokens"], 5);
|
||||
@@ -88,6 +90,7 @@ async fn review_provider_flag_pins_route_for_multi_route_model() {
|
||||
&resolve_review_model(&inferred_config, inferred.model.as_deref()),
|
||||
"review diff",
|
||||
inferred_force,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect_err("a model offered by two configured routes must hard-error");
|
||||
@@ -120,6 +123,7 @@ async fn review_provider_flag_pins_route_for_multi_route_model() {
|
||||
&resolve_review_model(&pinned_config, pinned.model.as_deref()),
|
||||
"review diff",
|
||||
pinned_force,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect("pinned review route");
|
||||
@@ -155,6 +159,7 @@ async fn review_provider_flag_pins_route_for_multi_route_model() {
|
||||
&resolve_review_model(&custom_config, pinned_custom.model.as_deref()),
|
||||
"review diff",
|
||||
custom_force,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect("pinned custom review route");
|
||||
@@ -820,7 +825,8 @@ fn pr_review_unreviewed_note_names_budget_stops_and_plan_skips() {
|
||||
assert!(note.contains("a/b.txt b/b.txt"), "{note}");
|
||||
assert!(note.contains("a/c.txt b/c.txt"), "{note}");
|
||||
// A complete plan with every pass done names nothing.
|
||||
let complete = codewhale_engine::tools::review::plan_pr_review(&diff, &view, max_chars, 3).unwrap();
|
||||
let complete =
|
||||
codewhale_engine::tools::review::plan_pr_review(&diff, &view, max_chars, 3).unwrap();
|
||||
assert_eq!(
|
||||
pr_review_unreviewed_note(Some(&complete), 3),
|
||||
"no planned file was left unread"
|
||||
@@ -831,7 +837,8 @@ fn pr_review_unreviewed_note_names_budget_stops_and_plan_skips() {
|
||||
async fn actual_cli_review_report_host_matches_legacy_and_core_keeps_inline_authority() {
|
||||
let _home = crate::test_support::SealedHome::new();
|
||||
let _policy = codewhale_plugins::activation::TestPolicyGuard::extension_host(false);
|
||||
let Some(node) = codewhale_engine::extension_host::test_support::node_for_tests("cli_review_report_parity")
|
||||
let Some(node) =
|
||||
codewhale_engine::extension_host::test_support::node_for_tests("cli_review_report_parity")
|
||||
else {
|
||||
return;
|
||||
};
|
||||
@@ -844,8 +851,9 @@ async fn actual_cli_review_report_host_matches_legacy_and_core_keeps_inline_auth
|
||||
..Default::default()
|
||||
},
|
||||
));
|
||||
let _manager =
|
||||
codewhale_engine::extension_host::TestManagerGuard::install(std::sync::Arc::clone(&manager));
|
||||
let _manager = codewhale_engine::extension_host::TestManagerGuard::install(
|
||||
std::sync::Arc::clone(&manager),
|
||||
);
|
||||
let mut flags = codewhale_runtime::features::Features::with_defaults();
|
||||
flags.enable(codewhale_runtime::features::Feature::ReviewHost);
|
||||
let context = codewhale_engine::tools::spec::ToolContext::new(root.path()).with_features(flags);
|
||||
@@ -897,9 +905,10 @@ async fn actual_cli_review_report_host_matches_legacy_and_core_keeps_inline_auth
|
||||
.unwrap(),
|
||||
"plain prose"
|
||||
);
|
||||
let prompt = codewhale_engine::tools::review_host::interactive(7, &view, &review_test_diff(), &context)
|
||||
.await
|
||||
.unwrap();
|
||||
let prompt =
|
||||
codewhale_engine::tools::review_host::interactive(7, &view, &review_test_diff(), &context)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(prompt, format_pr_prompt(7, &view, &review_test_diff()));
|
||||
manager.shutdown().await;
|
||||
}
|
||||
@@ -1035,6 +1044,7 @@ async fn configured_workflow_default_keeps_named_custom_route() {
|
||||
&model,
|
||||
"Run a checked-in Workflow through the host runtime",
|
||||
true,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect("configured workflow route");
|
||||
@@ -1052,7 +1062,8 @@ async fn configured_workflow_default_keeps_named_custom_route() {
|
||||
execution.active_route_base_url(),
|
||||
"http://127.0.0.1:18181/v1"
|
||||
);
|
||||
let client = codewhale_engine::client::CodewhaleClient::new(&execution).expect("workflow client");
|
||||
let client =
|
||||
codewhale_engine::client::CodewhaleClient::new(&execution).expect("workflow client");
|
||||
assert_eq!(client.base_url(), "http://127.0.0.1:18181/v1");
|
||||
}
|
||||
|
||||
|
||||
@@ -765,7 +765,8 @@
|
||||
exec_stream_final_answer_excerpt(" short reply \n"),
|
||||
"short reply"
|
||||
);
|
||||
let long = "x".repeat(codewhale_engine::fleet::executor::EXEC_STREAM_FINAL_ANSWER_EXCERPT_CHARS + 5);
|
||||
let long = "x"
|
||||
.repeat(codewhale_engine::fleet::executor::EXEC_STREAM_FINAL_ANSWER_EXCERPT_CHARS + 5);
|
||||
let excerpt = exec_stream_final_answer_excerpt(&long);
|
||||
assert_eq!(
|
||||
excerpt.chars().count(),
|
||||
@@ -1366,8 +1367,10 @@
|
||||
"--probe-search",
|
||||
] {
|
||||
assert!(
|
||||
crate::Cli::try_parse_from(["codewhale-tui", "doctor", output_flag, live_flag,])
|
||||
.is_err(),
|
||||
crate::Cli::try_parse_from(
|
||||
["codewhale-tui", "doctor", output_flag, live_flag,]
|
||||
)
|
||||
.is_err(),
|
||||
"{output_flag} unexpectedly accepted live flag {live_flag}"
|
||||
);
|
||||
}
|
||||
@@ -1376,9 +1379,9 @@
|
||||
|
||||
#[test]
|
||||
fn default_max_turns_does_not_add_a_hidden_worker_cap() {
|
||||
use codewhale_config::FleetExecConfig;
|
||||
use codewhale_engine::fleet::executor::build_worker_exec_command_with_launch_spec;
|
||||
use codewhale_engine::fleet::worker_runtime::fleet_task_to_worker_spec_with_profiles;
|
||||
use codewhale_config::FleetExecConfig;
|
||||
use codewhale_protocol::fleet::{FleetHostSpec, FleetTaskSpec, FleetWorkerSpec};
|
||||
|
||||
let exec = FleetExecConfig::default();
|
||||
@@ -1438,7 +1441,8 @@
|
||||
assert_eq!(args.max_tool_calls, None);
|
||||
// Follow omission beyond argv through the real CLI resolver. This
|
||||
// previously installed 200 despite correct unbounded launch args.
|
||||
let turn = codewhale_engine::core::turn::TurnContext::new(exec_max_steps(args.max_turns));
|
||||
let turn =
|
||||
codewhale_engine::core::turn::TurnContext::new(exec_max_steps(args.max_turns));
|
||||
assert_eq!(turn.step_limit(), expected);
|
||||
assert_eq!(turn.stop_diagnostics.effective_max_steps, expected);
|
||||
}
|
||||
|
||||
@@ -7794,6 +7794,8 @@ mod tests {
|
||||
|
||||
fn turn_complete_event() -> EngineEvent {
|
||||
EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -8364,6 +8366,8 @@ mod tests {
|
||||
..codewhale_models::Usage::default()
|
||||
};
|
||||
controller.observe_engine_event(&EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
|
||||
@@ -13882,6 +13882,13 @@ fn map_automation_err(err: anyhow::Error) -> ApiError {
|
||||
}
|
||||
|
||||
fn map_thread_err(err: anyhow::Error) -> ApiError {
|
||||
if let Some(error) = err.downcast_ref::<codewhale_engine::runtime_threads::manager::RuntimeTurnOperationLookupError>() {
|
||||
return match error {
|
||||
codewhale_engine::runtime_threads::manager::RuntimeTurnOperationLookupError::InvalidRequest => ApiError::bad_request(error.to_string()),
|
||||
codewhale_engine::runtime_threads::manager::RuntimeTurnOperationLookupError::Incomplete => ApiError::conflict(error.to_string()),
|
||||
codewhale_engine::runtime_threads::manager::RuntimeTurnOperationLookupError::Unavailable => ApiError::internal(error.to_string()),
|
||||
};
|
||||
}
|
||||
let message = err.to_string();
|
||||
let lower = message.to_ascii_lowercase();
|
||||
if (lower.starts_with("thread '") && lower.ends_with("' not found"))
|
||||
|
||||
@@ -728,72 +728,19 @@ pub(super) fn messages_from_thread_detail(
|
||||
runtime.history_from_records(&detail.turns, &items_by_turn)
|
||||
}
|
||||
|
||||
/// Merge the thread's authoritative cost into a session about to be saved.
|
||||
///
|
||||
/// The engine snapshot carries messages/tokens but no cost — cost lives in
|
||||
/// the turn records' route-audited usage — so derive it from the same
|
||||
/// accumulation that powers `/v1/usage` (recorded-time pricing, both
|
||||
/// published currencies). The parent/child split mirrors the TUI writer's
|
||||
/// field semantics (`sync_cost_to_metadata`): `session_cost_*` carries
|
||||
/// parent-turn spend and `subagent_cost_*` routed-child spend, so a session
|
||||
/// previously saved by the TUI never gets child spend counted twice in
|
||||
/// `total_estimate()`. Merging each side with max keeps a session resumed
|
||||
/// across threads from losing previously persisted spend, and extends the
|
||||
/// monotonic display guarantee (#244) to the persisted shape. Coverage
|
||||
/// travels with the money it qualifies (#4318): the counters are
|
||||
/// parent-turn coverage (the TUI's own session-level accounting), CNY
|
||||
/// included, and `coverage_recorded` marks that this writer computed them
|
||||
/// from audited turn records rather than deserializing a legacy default.
|
||||
async fn persist_thread_cost(
|
||||
runtime: &std::sync::Arc<RuntimeThreadManager>,
|
||||
thread_id: &str,
|
||||
session: &mut codewhale_engine::session_manager::SavedSession,
|
||||
) -> Result<(), ApiError> {
|
||||
let usage = runtime
|
||||
.aggregate_usage_for_thread(thread_id)
|
||||
session.metadata.cost = runtime
|
||||
.project_session_cost_for_thread(thread_id, &session.metadata.cost)
|
||||
.await
|
||||
.map_err(|e| ApiError::internal(format!("Failed to aggregate thread usage: {e}")))?;
|
||||
let combined = usage.combined();
|
||||
let cost = &mut session.metadata.cost;
|
||||
cost.session_cost_usd = cost.session_cost_usd.max(usage.parent.cost_usd);
|
||||
cost.session_cost_cny = cost.session_cost_cny.max(usage.parent.cost_cny);
|
||||
cost.subagent_cost_usd = cost.subagent_cost_usd.max(usage.routed_children.cost_usd);
|
||||
cost.subagent_cost_cny = cost.subagent_cost_cny.max(usage.routed_children.cost_cny);
|
||||
// The display total is session + subagent, so the high-water mark rides
|
||||
// the combined figure in both currencies.
|
||||
cost.displayed_cost_high_water_usd = cost.displayed_cost_high_water_usd.max(combined.cost_usd);
|
||||
cost.displayed_cost_high_water_cny = cost.displayed_cost_high_water_cny.max(combined.cost_cny);
|
||||
cost.priced_turns = cost
|
||||
.priced_turns
|
||||
.max(u32::try_from(usage.parent.priced_turns).unwrap_or(u32::MAX));
|
||||
cost.unpriced_turns = cost
|
||||
.unpriced_turns
|
||||
.max(u32::try_from(usage.parent.unpriced_turns).unwrap_or(u32::MAX));
|
||||
cost.cny_priced_turns = cost
|
||||
.cny_priced_turns
|
||||
.max(u32::try_from(usage.parent.cny_priced_turns).unwrap_or(u32::MAX));
|
||||
cost.cny_unpriced_turns = cost
|
||||
.cny_unpriced_turns
|
||||
.max(u32::try_from(usage.parent.cny_unpriced_turns).unwrap_or(u32::MAX));
|
||||
// Coverage travels with the money (#4318): reasons and classes are the
|
||||
// qualifiers a reload needs to treat these totals as known, not a
|
||||
// legacy-unknown complete zero. Parent-turn coverage only — the same
|
||||
// field the TUI writer uses; routed-child spend lives in subagent_cost_*.
|
||||
cost.unpriced_reasons
|
||||
.extend(usage.parent.unpriced_reasons.iter().cloned());
|
||||
cost.cny_unpriced_reasons
|
||||
.extend(usage.parent.cny_unpriced_reasons.iter().cloned());
|
||||
cost.unpriced_classes
|
||||
.extend(usage.parent.unpriced_classes.iter().cloned());
|
||||
cost.pricing_provenances
|
||||
.extend(usage.parent.pricing_provenances.iter().cloned());
|
||||
cost.live_pricing_defects
|
||||
.extend(usage.parent.live_pricing_defects.iter().cloned());
|
||||
cost.live_pricing_unusable_defects
|
||||
.extend(usage.parent.live_pricing_unusable_defects.iter().cloned());
|
||||
cost.route_receipts
|
||||
.extend(usage.parent.route_receipts.iter().cloned());
|
||||
cost.coverage_recorded = true;
|
||||
.map_err(|error| {
|
||||
ApiError::internal(format!(
|
||||
"Failed to reconcile thread source accounting: {error}"
|
||||
))
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
@@ -486,6 +486,11 @@ fn messages_from_thread_detail_batches_tool_results() {
|
||||
last_read: None,
|
||||
};
|
||||
let turn = TurnRecord {
|
||||
source_usage_origin_turn_id: None,
|
||||
|
||||
source_accounting: false,
|
||||
source_usage_records: Default::default(),
|
||||
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
@@ -3102,6 +3107,7 @@ async fn agent_runs_runtime_api_exposes_persisted_worker_receipts() -> Result<()
|
||||
note: "not reported".to_string(),
|
||||
},
|
||||
usage_source_fingerprints: Default::default(),
|
||||
source_usage_records: Default::default(),
|
||||
missing_usage_sources: Default::default(),
|
||||
missing_usage_overflowed: false,
|
||||
legacy_unreported_usage: false,
|
||||
@@ -3542,6 +3548,8 @@ async fn compatibility_stream_closes_losslessly_across_replay_live_handoff() ->
|
||||
.await;
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -3819,6 +3827,8 @@ async fn compatibility_stream_exposes_and_resolves_user_input_without_answer_ech
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -4225,6 +4235,8 @@ async fn thread_endpoints_expose_lifecycle_contract() -> Result<()> {
|
||||
.await;
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -4249,6 +4261,8 @@ async fn thread_endpoints_expose_lifecycle_contract() -> Result<()> {
|
||||
Op::CompactContext { .. } => {
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -4392,6 +4406,8 @@ async fn turn_endpoint_operation_key_returns_original_and_conflicts_on_mismatch(
|
||||
sleep(Duration::from_millis(20)).await;
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -4619,6 +4635,8 @@ async fn turn_operation_lookup_is_authenticated_read_only_and_survives_restart()
|
||||
engine
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -4752,6 +4770,8 @@ async fn events_endpoint_respects_since_seq_cursor() -> Result<()> {
|
||||
.await;
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -5695,6 +5715,8 @@ async fn steer_and_interrupt_endpoints_work_on_active_turn() -> Result<()> {
|
||||
sleep(Duration::from_millis(60)).await;
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -6250,6 +6272,8 @@ async fn stream_endpoint_remains_backward_compatible() -> Result<()> {
|
||||
.await;
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -8828,6 +8852,8 @@ async fn session_create_from_thread_rejects_active_turn() -> Result<()> {
|
||||
.await;
|
||||
let _ = tx_event
|
||||
.send(EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -10049,6 +10075,11 @@ async fn session_save_merges_thread_cost_split_and_records_coverage() -> Result<
|
||||
let store = runtime_threads.test_store();
|
||||
let now = Utc::now();
|
||||
let mut turn = TurnRecord {
|
||||
source_usage_origin_turn_id: None,
|
||||
|
||||
source_accounting: false,
|
||||
source_usage_records: Default::default(),
|
||||
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
@@ -10253,6 +10284,11 @@ async fn session_save_persists_parent_cny_unpriced_reasons_without_double_count(
|
||||
let store = runtime_threads.test_store();
|
||||
let now = Utc::now();
|
||||
let mut turn = TurnRecord {
|
||||
source_usage_origin_turn_id: None,
|
||||
|
||||
source_accounting: false,
|
||||
source_usage_records: Default::default(),
|
||||
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
@@ -11614,6 +11650,11 @@ fn seed_summary_search_transcript(
|
||||
item_ids.push(item_id);
|
||||
}
|
||||
store.save_turn(&TurnRecord {
|
||||
source_usage_origin_turn_id: None,
|
||||
|
||||
source_accounting: false,
|
||||
source_usage_records: Default::default(),
|
||||
|
||||
prefix_checkpoint: None,
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
@@ -20099,6 +20140,8 @@ async fn runtime_image_http_rejects_before_dispatch_and_accepts_large_canonical_
|
||||
harness
|
||||
.tx_event
|
||||
.send(EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
|
||||
@@ -460,6 +460,9 @@ async fn import_into_owner(
|
||||
serde_json::to_writer(&mut bound, &observed)?;
|
||||
session = observed;
|
||||
}
|
||||
session.schema_version = session
|
||||
.schema_version
|
||||
.max(codewhale_engine::session_manager::CURRENT_SESSION_SCHEMA_VERSION);
|
||||
let document_digest = saved_document_digest(&session)?;
|
||||
let prepare_runtime = Arc::clone(&runtime);
|
||||
let prepared_operation = operation.clone();
|
||||
@@ -1702,6 +1705,9 @@ async fn mutate_in_owner(
|
||||
.ok_or_else(|| anyhow!("canonical thread has no provider identity"))?,
|
||||
thread.model_provider_id.as_deref(),
|
||||
);
|
||||
session.schema_version = session
|
||||
.schema_version
|
||||
.max(codewhale_engine::session_manager::CURRENT_SESSION_SCHEMA_VERSION);
|
||||
let document_digest = saved_document_digest(&session)?;
|
||||
let prepare_runtime = Arc::clone(&runtime);
|
||||
let operation = history_owner_work(move || {
|
||||
|
||||
+217
-24
@@ -597,6 +597,7 @@ pub struct AgentProgressMeta {
|
||||
/// spend is unknown renders no token figure at all rather than a
|
||||
/// fabricated `0`.
|
||||
pub received_tokens: Option<u64>,
|
||||
pub received_usage_sources: HashMap<String, u64>,
|
||||
/// Unsettled items on this child's own To-do list, from the latest
|
||||
/// `WorkState` envelope. `None` until a real list is published — the
|
||||
/// strip never invents a `0 left` chip for agents with no checklist.
|
||||
@@ -1179,6 +1180,8 @@ pub struct SessionState {
|
||||
/// Redacted provider-response identities settled by parent or routed owners.
|
||||
/// Membership never adds a child amount; parent usage keeps its own pricing.
|
||||
pub subagent_usage_sources: HashSet<String>,
|
||||
pub accounted_usage_sources:
|
||||
std::collections::BTreeMap<String, codewhale_engine::cost_status::UsageSourceContribution>,
|
||||
pub missing_usage_sources:
|
||||
std::collections::BTreeMap<String, codewhale_engine::cost_status::MissingUsageCoverage>,
|
||||
pub missing_usage_overflowed: bool,
|
||||
@@ -1359,6 +1362,7 @@ impl Default for SessionState {
|
||||
subagent_cost: 0.0,
|
||||
subagent_cost_cny: 0.0,
|
||||
subagent_usage_sources: HashSet::new(),
|
||||
accounted_usage_sources: Default::default(),
|
||||
missing_usage_sources: std::collections::BTreeMap::new(),
|
||||
missing_usage_overflowed: false,
|
||||
displayed_cost_high_water: 0.0,
|
||||
@@ -4662,7 +4666,23 @@ impl App {
|
||||
pool.cache_write_tokens,
|
||||
);
|
||||
self.absorb_background_cost_coverage(&pool);
|
||||
runtime_usage_arrived
|
||||
let mut source_arrived = false;
|
||||
for record in &pool.source_records {
|
||||
match self.absorb_usage_source(record) {
|
||||
Ok(changed) => source_arrived |= changed,
|
||||
Err(error) => {
|
||||
tracing::warn!(%error, "live usage source projection refused");
|
||||
self.session
|
||||
.cost_unpriced_reasons
|
||||
.insert("usage_correction_unverified".into());
|
||||
self.session
|
||||
.cost_cny_unpriced_reasons
|
||||
.insert("usage_correction_unverified".into());
|
||||
self.session.missing_usage_overflowed = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
runtime_usage_arrived || source_arrived
|
||||
}
|
||||
|
||||
/// Clear every live cost-coverage counter.
|
||||
@@ -4671,6 +4691,7 @@ impl App {
|
||||
/// leave the previous session's priced/unpriced turns attached to a total
|
||||
/// that no longer contains them (#4318).
|
||||
pub fn reset_cost_coverage(&mut self) {
|
||||
self.session.accounted_usage_sources.clear();
|
||||
self.session.missing_usage_sources.clear();
|
||||
self.session.missing_usage_overflowed = false;
|
||||
self.session.cost_priced_turns = 0;
|
||||
@@ -4745,33 +4766,33 @@ impl App {
|
||||
|
||||
/// Copy current session/subagent cost accumulators into session metadata
|
||||
/// for persistence.
|
||||
pub fn sync_cost_to_metadata(
|
||||
fn copy_cost_to_snapshot(
|
||||
&self,
|
||||
metadata: &mut codewhale_engine::session_manager::SessionMetadata,
|
||||
cost: &mut codewhale_engine::session_manager::SessionCostSnapshot,
|
||||
) {
|
||||
metadata.cost.session_cost_usd = self.session.session_cost;
|
||||
metadata.cost.session_cost_cny = self.session.session_cost_cny;
|
||||
metadata.cost.subagent_cost_usd = self.session.subagent_cost;
|
||||
metadata.cost.subagent_cost_cny = self.session.subagent_cost_cny;
|
||||
metadata.cost.displayed_cost_high_water_usd = self.session.displayed_cost_high_water;
|
||||
metadata.cost.displayed_cost_high_water_cny = self.session.displayed_cost_high_water_cny;
|
||||
cost.session_cost_usd = self.session.session_cost;
|
||||
cost.session_cost_cny = self.session.session_cost_cny;
|
||||
cost.subagent_cost_usd = self.session.subagent_cost;
|
||||
cost.subagent_cost_cny = self.session.subagent_cost_cny;
|
||||
cost.displayed_cost_high_water_usd = self.session.displayed_cost_high_water;
|
||||
cost.displayed_cost_high_water_cny = self.session.displayed_cost_high_water_cny;
|
||||
// Coverage travels with the money it qualifies. A restored total without
|
||||
// these fields cannot say what it covers, and its serde defaults read as
|
||||
// a *complete* total covering zero turns — so they are persisted together
|
||||
// and `coverage_recorded` marks that this writer actually knew (#4318).
|
||||
metadata.cost.priced_turns = self.session.cost_priced_turns;
|
||||
metadata.cost.unpriced_turns = self.session.cost_unpriced_turns;
|
||||
metadata.cost.cny_priced_turns = self.session.cost_cny_priced_turns;
|
||||
metadata.cost.cny_unpriced_turns = self.session.cost_cny_unpriced_turns;
|
||||
metadata.cost.unpriced_reasons = self.session.cost_unpriced_reasons.clone();
|
||||
metadata.cost.cny_unpriced_reasons = self.session.cost_cny_unpriced_reasons.clone();
|
||||
metadata.cost.unpriced_classes = self.session.cost_unpriced_classes.clone();
|
||||
metadata.cost.pricing_provenances = self.session.cost_pricing_provenances.clone();
|
||||
metadata.cost.live_pricing_defects = self.session.cost_live_pricing_defects.clone();
|
||||
metadata.cost.live_pricing_unusable_defects =
|
||||
cost.priced_turns = self.session.cost_priced_turns;
|
||||
cost.unpriced_turns = self.session.cost_unpriced_turns;
|
||||
cost.cny_priced_turns = self.session.cost_cny_priced_turns;
|
||||
cost.cny_unpriced_turns = self.session.cost_cny_unpriced_turns;
|
||||
cost.unpriced_reasons = self.session.cost_unpriced_reasons.clone();
|
||||
cost.cny_unpriced_reasons = self.session.cost_cny_unpriced_reasons.clone();
|
||||
cost.unpriced_classes = self.session.cost_unpriced_classes.clone();
|
||||
cost.pricing_provenances = self.session.cost_pricing_provenances.clone();
|
||||
cost.live_pricing_defects = self.session.cost_live_pricing_defects.clone();
|
||||
cost.live_pricing_unusable_defects =
|
||||
self.session.cost_live_pricing_unusable_defects.clone();
|
||||
metadata.cost.route_receipts = self.session.cost_route_receipts.clone();
|
||||
metadata.cost.usage_source_fingerprints = self
|
||||
cost.route_receipts = self.session.cost_route_receipts.clone();
|
||||
cost.usage_source_fingerprints = self
|
||||
.session
|
||||
.subagent_usage_sources
|
||||
.iter()
|
||||
@@ -4780,9 +4801,181 @@ impl App {
|
||||
// A session restored as legacy-unknown stays unknown when re-saved:
|
||||
// re-writing it as "recorded" would launder the missing evidence into an
|
||||
// apparently complete zero.
|
||||
metadata.cost.missing_usage_sources = self.session.missing_usage_sources.clone();
|
||||
metadata.cost.missing_usage_overflowed = self.session.missing_usage_overflowed;
|
||||
metadata.cost.coverage_recorded = !self.session.cost_coverage_unknown_legacy;
|
||||
cost.missing_usage_sources = self.session.missing_usage_sources.clone();
|
||||
cost.missing_usage_overflowed = self.session.missing_usage_overflowed;
|
||||
cost.coverage_recorded = !self.session.cost_coverage_unknown_legacy;
|
||||
cost.accounted_usage_sources = self.session.accounted_usage_sources.clone();
|
||||
}
|
||||
|
||||
pub(crate) fn restore_cost_snapshot(
|
||||
&mut self,
|
||||
cost: &codewhale_engine::session_manager::SessionCostSnapshot,
|
||||
) {
|
||||
let parent = CostEstimate {
|
||||
usd: cost.session_cost_usd,
|
||||
cny: cost.session_cost_cny,
|
||||
}
|
||||
.sanitized();
|
||||
let auxiliary = CostEstimate {
|
||||
usd: cost.subagent_cost_usd,
|
||||
cny: cost.subagent_cost_cny,
|
||||
}
|
||||
.sanitized();
|
||||
self.session.session_cost = parent.usd;
|
||||
self.session.session_cost_cny = parent.cny;
|
||||
self.session.subagent_cost = auxiliary.usd;
|
||||
self.session.subagent_cost_cny = auxiliary.cny;
|
||||
self.session.subagent_usage_sources =
|
||||
cost.usage_source_fingerprints.iter().cloned().collect();
|
||||
self.session.accounted_usage_sources = cost.accounted_usage_sources.clone();
|
||||
self.session.missing_usage_sources = cost.missing_usage_sources.clone();
|
||||
self.session.missing_usage_overflowed = cost.missing_usage_overflowed;
|
||||
self.session.cost_priced_turns = cost.priced_turns;
|
||||
self.session.cost_unpriced_turns = cost.unpriced_turns;
|
||||
self.session.cost_cny_priced_turns = cost.cny_priced_turns;
|
||||
self.session.cost_cny_unpriced_turns = cost.cny_unpriced_turns;
|
||||
self.session.cost_unpriced_reasons = cost.unpriced_reasons.clone();
|
||||
self.session.cost_cny_unpriced_reasons = cost.cny_unpriced_reasons.clone();
|
||||
self.session.cost_unpriced_classes = cost.unpriced_classes.clone();
|
||||
self.session.cost_pricing_provenances = cost.pricing_provenances.clone();
|
||||
self.session.cost_live_pricing_defects = cost.live_pricing_defects.clone();
|
||||
self.session.cost_live_pricing_unusable_defects =
|
||||
cost.live_pricing_unusable_defects.clone();
|
||||
self.session.cost_route_receipts = cost.route_receipts.clone();
|
||||
self.session.cost_coverage_unknown_legacy = cost.coverage_is_legacy_unknown();
|
||||
self.session.displayed_cost_high_water = self
|
||||
.session
|
||||
.displayed_cost_high_water
|
||||
.max(cost.displayed_cost_high_water_usd)
|
||||
.max(parent.usd + auxiliary.usd);
|
||||
self.session.displayed_cost_high_water_cny = self
|
||||
.session
|
||||
.displayed_cost_high_water_cny
|
||||
.max(cost.displayed_cost_high_water_cny)
|
||||
.max(parent.cny + auxiliary.cny);
|
||||
}
|
||||
|
||||
pub(crate) fn mark_usage_uncertain(&mut self, reason: &'static str) {
|
||||
if self
|
||||
.session
|
||||
.cost_unpriced_reasons
|
||||
.insert(reason.to_string())
|
||||
{
|
||||
self.session.cost_unpriced_turns = self.session.cost_unpriced_turns.saturating_add(1);
|
||||
}
|
||||
if self
|
||||
.session
|
||||
.cost_cny_unpriced_reasons
|
||||
.insert(reason.to_string())
|
||||
{
|
||||
self.session.cost_cny_unpriced_turns =
|
||||
self.session.cost_cny_unpriced_turns.saturating_add(1);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn absorb_usage_source(
|
||||
&mut self,
|
||||
record: &codewhale_engine::cost_status::RuntimeUsageRecord,
|
||||
) -> Result<bool, String> {
|
||||
let key = codewhale_engine::cost_status::usage_source_fingerprint(&record.source_id);
|
||||
let previous = self.session.accounted_usage_sources.get(&key).cloned();
|
||||
let mut cost = codewhale_engine::session_manager::SessionCostSnapshot::default();
|
||||
self.copy_cost_to_snapshot(&mut cost);
|
||||
if previous
|
||||
.as_ref()
|
||||
.is_some_and(|previous| previous.record == *record)
|
||||
{
|
||||
return Ok(false);
|
||||
}
|
||||
let source = self
|
||||
.session_write_receipt
|
||||
.as_ref()
|
||||
.ok_or_else(|| "cost projection has no admitted session scope".to_string())?;
|
||||
let Some((old, new)) = source
|
||||
.project_usage_record(
|
||||
codewhale_engine::cost_status::scope_token(),
|
||||
record,
|
||||
&mut cost,
|
||||
)
|
||||
.map_err(|error| error.to_string())?
|
||||
else {
|
||||
return Ok(false);
|
||||
};
|
||||
if old == 0
|
||||
&& new == 0
|
||||
&& previous
|
||||
.as_ref()
|
||||
.is_some_and(|previous| previous.record == *record)
|
||||
{
|
||||
return Ok(false);
|
||||
}
|
||||
self.restore_cost_snapshot(&cost);
|
||||
let next_usage = &record.usage.usage;
|
||||
let old_usage = previous
|
||||
.as_ref()
|
||||
.map(|previous| &previous.record.usage.usage);
|
||||
let replace = |total: &mut u32, old: u32, new: u32| {
|
||||
*total = total.saturating_sub(old).saturating_add(new);
|
||||
};
|
||||
replace(
|
||||
&mut self.session.total_tokens,
|
||||
u32::try_from(old).unwrap_or(u32::MAX),
|
||||
u32::try_from(new).unwrap_or(u32::MAX),
|
||||
);
|
||||
replace(
|
||||
&mut self.session.total_conversation_tokens,
|
||||
u32::try_from(old).unwrap_or(u32::MAX),
|
||||
u32::try_from(new).unwrap_or(u32::MAX),
|
||||
);
|
||||
let replace_breakdown = |total: &mut u32, old: u32, new: u32| {
|
||||
*total = if new >= old {
|
||||
total.saturating_add(new - old)
|
||||
} else {
|
||||
total.saturating_sub(old - new)
|
||||
};
|
||||
};
|
||||
replace_breakdown(
|
||||
&mut self.session.total_input_tokens,
|
||||
old_usage.map_or(0, |usage| usage.input_tokens),
|
||||
next_usage.input_tokens,
|
||||
);
|
||||
replace_breakdown(
|
||||
&mut self.session.total_output_tokens,
|
||||
old_usage.map_or(0, |usage| usage.output_tokens),
|
||||
next_usage.output_tokens,
|
||||
);
|
||||
let before = old_usage.map(codewhale_engine::pricing::token_usage_for_pricing);
|
||||
let after = codewhale_engine::pricing::token_usage_for_pricing(next_usage);
|
||||
replace_breakdown(
|
||||
&mut self.session.total_cache_hit_tokens,
|
||||
before.as_ref().map_or(0, |usage| {
|
||||
u32::try_from(usage.cache_read).unwrap_or(u32::MAX)
|
||||
}),
|
||||
u32::try_from(after.cache_read).unwrap_or(u32::MAX),
|
||||
);
|
||||
replace_breakdown(
|
||||
&mut self.session.total_cache_miss_tokens,
|
||||
before
|
||||
.as_ref()
|
||||
.map_or(0, |usage| u32::try_from(usage.input).unwrap_or(u32::MAX)),
|
||||
u32::try_from(after.input).unwrap_or(u32::MAX),
|
||||
);
|
||||
replace_breakdown(
|
||||
&mut self.session.total_cache_write_tokens,
|
||||
before.as_ref().map_or(0, |usage| {
|
||||
u32::try_from(usage.cache_write).unwrap_or(u32::MAX)
|
||||
}),
|
||||
u32::try_from(after.cache_write).unwrap_or(u32::MAX),
|
||||
);
|
||||
self.needs_redraw = true;
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
pub fn sync_cost_to_metadata(
|
||||
&self,
|
||||
metadata: &mut codewhale_engine::session_manager::SessionMetadata,
|
||||
) {
|
||||
self.copy_cost_to_snapshot(&mut metadata.cost);
|
||||
// Persist cumulative turn duration so the footer "worked" chip
|
||||
// survives session save/restore (#2038).
|
||||
metadata.cumulative_turn_secs = self.cumulative_turn_duration.as_secs();
|
||||
|
||||
@@ -889,6 +889,8 @@ mod tests {
|
||||
observe(
|
||||
&mut app,
|
||||
&Event::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -1092,6 +1094,8 @@ mod tests {
|
||||
|
||||
let done = metadata(
|
||||
&Event::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -1190,6 +1194,8 @@ mod tests {
|
||||
|
||||
let turn = metadata(
|
||||
&Event::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
|
||||
@@ -460,75 +460,40 @@ pub(super) fn subagent_message_refreshes_workspace_context(message: &MailboxMess
|
||||
/// allocating a `DelegateCard` or `FanoutCard` on first sight (issue #128).
|
||||
pub(super) fn handle_subagent_mailbox_for_turn(
|
||||
app: &mut App,
|
||||
turn_id: &str,
|
||||
seq: u64,
|
||||
_turn_id: &str,
|
||||
_seq: u64,
|
||||
message: &MailboxMessage,
|
||||
) -> bool {
|
||||
// Accumulate sub-agent token costs for the real-time footer counter (#166).
|
||||
if let MailboxMessage::TokenUsage {
|
||||
record,
|
||||
agent_id: _,
|
||||
source_id,
|
||||
route,
|
||||
usage,
|
||||
..
|
||||
} = message
|
||||
{
|
||||
// The child's own route truth always wins and is never guessed from
|
||||
// provider identity: `route` is the immutable envelope its client was
|
||||
// frozen with at construction, so its billing mode, billing surface
|
||||
// and endpoint fingerprint are the child's dispatch receipt. A child
|
||||
// whose route froze as Unknown stays Unknown.
|
||||
//
|
||||
// Sub-agent spend joins the parent total, so it also joins the
|
||||
// completeness counters `/cost` reports against that total.
|
||||
let stable_source = if source_id.trim().is_empty() {
|
||||
format!("mailbox:{turn_id}:{seq}")
|
||||
} else {
|
||||
source_id.clone()
|
||||
};
|
||||
let source_fingerprint =
|
||||
codewhale_engine::cost_status::usage_source_fingerprint(&stable_source);
|
||||
if codewhale_engine::cost_status::usage_source_seen(&stable_source) {
|
||||
// The runtime sink runs synchronously before mailbox publication.
|
||||
// Fold its identity and money into App together, before a following
|
||||
// TurnComplete can snapshot either half. A post-seal response has
|
||||
// no mailbox event and is swept by the event loop instead.
|
||||
let first_live_delivery = !app
|
||||
.session
|
||||
.subagent_usage_sources
|
||||
.contains(&source_fingerprint);
|
||||
let pending_runtime_cost = codewhale_engine::cost_status::drain();
|
||||
if !pending_runtime_cost.is_empty() {
|
||||
app.absorb_pending_background_cost(&pending_runtime_cost);
|
||||
if let Some(record) = record.as_ref().filter(|record| record.settlement.is_some()) {
|
||||
let pending = codewhale_engine::cost_status::drain();
|
||||
app.absorb_pending_background_cost(&pending);
|
||||
if let Err(error) = app.absorb_usage_source(record) {
|
||||
tracing::warn!(%error, "mailbox usage revision was not admitted");
|
||||
app.mark_usage_uncertain("usage_correction_unverified");
|
||||
}
|
||||
if first_live_delivery
|
||||
&& app
|
||||
.session
|
||||
.subagent_usage_sources
|
||||
.contains(&source_fingerprint)
|
||||
let fingerprint =
|
||||
codewhale_engine::cost_status::usage_source_fingerprint(&record.source_id);
|
||||
if app
|
||||
.session
|
||||
.accounted_usage_sources
|
||||
.get(&fingerprint)
|
||||
.is_some_and(|contribution| contribution.record == *record)
|
||||
{
|
||||
record_agent_current_activity(app, message);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
if app
|
||||
.session
|
||||
.subagent_usage_sources
|
||||
.insert(source_fingerprint)
|
||||
{
|
||||
// Update the Work-row receipt under the same replay guard as
|
||||
// cost. A mailbox replay must not make either total grow twice.
|
||||
// Usage confirms or replaces the child's frozen spawn model; the
|
||||
// parent's configured/default route is never child evidence.
|
||||
record_agent_current_activity(app, message);
|
||||
let audit = route.audit(usage);
|
||||
app.record_turn_cost_audit(&audit);
|
||||
app.record_turn_cost_route_receipt(route.receipt(&audit));
|
||||
if let Some(cost) = audit.estimate {
|
||||
app.accrue_subagent_cost_estimate(cost);
|
||||
}
|
||||
}
|
||||
// No transcript card changed. The event-loop redraw still repaints
|
||||
// the Work row and footer from the updated receipt state.
|
||||
let pending = codewhale_engine::cost_status::drain();
|
||||
app.absorb_pending_background_cost(&pending);
|
||||
app.mark_usage_uncertain("legacy_usage_unverified");
|
||||
record_agent_current_activity(app, message);
|
||||
return false;
|
||||
}
|
||||
|
||||
@@ -705,15 +670,32 @@ fn record_agent_current_activity(app: &mut App, message: &MailboxMessage) {
|
||||
let agent_id = message.agent_id().to_string();
|
||||
let meta = app.agent_progress_meta.entry(agent_id).or_default();
|
||||
meta.last_progress_at = Some(Instant::now());
|
||||
if let MailboxMessage::TokenUsage { route, usage, .. } = message {
|
||||
// The child's own used-token tally (input + output), matching the
|
||||
// worker budget's `usage_total_tokens`. Counting only completions made
|
||||
// the strip look "stuck" on tiny numbers while the child was burning
|
||||
// context. Absent until a real envelope lands, so an agent with no
|
||||
// reported usage shows no number instead of a zero.
|
||||
let turn_total =
|
||||
u64::from(usage.input_tokens).saturating_add(u64::from(usage.output_tokens));
|
||||
meta.received_tokens = Some(meta.received_tokens.unwrap_or(0).saturating_add(turn_total));
|
||||
if let MailboxMessage::TokenUsage {
|
||||
record,
|
||||
route,
|
||||
usage,
|
||||
..
|
||||
} = message
|
||||
{
|
||||
let (route, usage) = record
|
||||
.as_ref()
|
||||
.filter(|record| record.settlement.is_some())
|
||||
.map_or((route.as_ref(), usage), |record| {
|
||||
(&record.usage.route, &record.usage.usage)
|
||||
});
|
||||
let total = u64::from(usage.input_tokens).saturating_add(u64::from(usage.output_tokens));
|
||||
if let Some(record) = record.as_ref().filter(|record| record.settlement.is_some()) {
|
||||
let key = codewhale_engine::cost_status::usage_source_fingerprint(&record.source_id);
|
||||
let previous = meta.received_usage_sources.insert(key, total).unwrap_or(0);
|
||||
meta.received_tokens = Some(
|
||||
meta.received_tokens
|
||||
.unwrap_or(0)
|
||||
.saturating_sub(previous)
|
||||
.saturating_add(total),
|
||||
);
|
||||
} else {
|
||||
meta.received_tokens = Some(meta.received_tokens.unwrap_or(0).saturating_add(total));
|
||||
}
|
||||
meta.resolved_provider = Some(route.provider.as_str().to_string());
|
||||
meta.resolved_model = Some(bound_agent_activity_text(
|
||||
&codewhale_runtime::safe_label::sanitize_persisted_route_label(&route.model),
|
||||
@@ -1352,6 +1334,7 @@ mod tests {
|
||||
&mut app,
|
||||
91,
|
||||
&MailboxMessage::TokenUsage {
|
||||
record: None,
|
||||
agent_id: "agent_route".to_string(),
|
||||
source_id: "response-route".to_string(),
|
||||
route: Box::new(test_route(
|
||||
@@ -1380,6 +1363,7 @@ mod tests {
|
||||
&mut app,
|
||||
1,
|
||||
&MailboxMessage::TokenUsage {
|
||||
record: None,
|
||||
agent_id: "agent_spend".to_string(),
|
||||
source_id: "response-1".to_string(),
|
||||
route: Box::new(route.clone()),
|
||||
@@ -1394,6 +1378,7 @@ mod tests {
|
||||
&mut app,
|
||||
2,
|
||||
&MailboxMessage::TokenUsage {
|
||||
record: None,
|
||||
agent_id: "agent_spend".to_string(),
|
||||
source_id: "response-2".to_string(),
|
||||
route: Box::new(route),
|
||||
|
||||
@@ -502,8 +502,16 @@ fn accrue_child_token_cost_if_any(app: &mut App, result: &Result<ToolResult, Too
|
||||
};
|
||||
if let Some(batch) = codewhale_engine::cost_status::child_usage_records_from_metadata(metadata)
|
||||
{
|
||||
let pending = codewhale_engine::cost_status::drain();
|
||||
app.absorb_pending_background_cost(&pending);
|
||||
for record in &batch.records {
|
||||
accrue_child_route_usage(app, &record.usage);
|
||||
if record.settlement.is_some() {
|
||||
if let Err(error) = app.absorb_usage_source(record) {
|
||||
tracing::warn!(%error, "tool usage revision was not admitted");
|
||||
}
|
||||
} else {
|
||||
app.mark_usage_uncertain("legacy_usage_unverified");
|
||||
}
|
||||
}
|
||||
for record in &batch.drop_records {
|
||||
let pending = codewhale_engine::cost_status::background_cost_for_runtime_drop(record);
|
||||
@@ -527,42 +535,10 @@ fn accrue_child_token_cost_if_any(app: &mut App, result: &Result<ToolResult, Too
|
||||
}
|
||||
return;
|
||||
}
|
||||
let Some(route) = codewhale_engine::cost_status::child_route_envelope_from_metadata(metadata)
|
||||
else {
|
||||
return;
|
||||
};
|
||||
// Use the same parser as the runtime host. It deliberately returns a
|
||||
// zero-valued usage record when the producer emitted the canonical child
|
||||
// fields: a model-backed call is still an auditable/priced-zero call, and
|
||||
// replay/server-tool telemetry must not disappear in the TUI projection.
|
||||
let Some(usage) = codewhale_engine::cost_status::child_usage_from_metadata(metadata) else {
|
||||
return;
|
||||
};
|
||||
accrue_child_route_usage(
|
||||
app,
|
||||
&codewhale_engine::cost_status::EffectiveRouteUsage { route, usage },
|
||||
);
|
||||
}
|
||||
|
||||
fn accrue_child_route_usage(
|
||||
app: &mut App,
|
||||
routed: &codewhale_engine::cost_status::EffectiveRouteUsage,
|
||||
) {
|
||||
// `route` is the child's own dispatch receipt, rehydrated from the
|
||||
// complete `child_*` metadata `attach_child_usage_metadata` emits at the
|
||||
// child's wire boundary (review/verify/rlm are the three producers). An
|
||||
// incomplete or legacy payload rehydrates as `RouteBillingMode::Unknown`,
|
||||
// so a child never inherits the live `app.billing_presentation` chip and a
|
||||
// `/provider` switch between dispatch and arrival cannot retro-bill it.
|
||||
//
|
||||
// Sub-agent spend lands in the same displayed total as parent turns, so it
|
||||
// has to feed the same completeness counters — otherwise `/cost` would call
|
||||
// a total complete while an unpriced child turn is missing from it.
|
||||
let audit = routed.route.audit(&routed.usage);
|
||||
app.record_turn_cost_audit(&audit);
|
||||
app.record_turn_cost_route_receipt(routed.route.receipt(&audit));
|
||||
if let Some(cost) = audit.estimate {
|
||||
app.accrue_subagent_cost_estimate(cost);
|
||||
if codewhale_engine::cost_status::child_route_envelope_from_metadata(metadata).is_some()
|
||||
&& codewhale_engine::cost_status::child_usage_from_metadata(metadata).is_some()
|
||||
{
|
||||
app.mark_usage_uncertain("legacy_usage_unverified");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -4903,30 +4903,10 @@ pub(crate) fn apply_loaded_session_with_goal(
|
||||
}
|
||||
app.session.total_tokens = u32::try_from(session.metadata.total_tokens).unwrap_or(u32::MAX);
|
||||
app.session.total_conversation_tokens = app.session.total_tokens;
|
||||
let restored_parent = codewhale_engine::pricing::CostEstimate {
|
||||
usd: session.metadata.cost.session_cost_usd,
|
||||
cny: session.metadata.cost.session_cost_cny,
|
||||
}
|
||||
.sanitized();
|
||||
let restored_background = codewhale_engine::pricing::CostEstimate {
|
||||
usd: session.metadata.cost.subagent_cost_usd,
|
||||
cny: session.metadata.cost.subagent_cost_cny,
|
||||
}
|
||||
.sanitized();
|
||||
// A restored session has no live billed receipt; the estimate rules the
|
||||
// meter until the next model call reports one.
|
||||
app.session.displayed_cost_high_water = 0.0;
|
||||
app.session.displayed_cost_high_water_cny = 0.0;
|
||||
app.last_billed_input_tokens = None;
|
||||
app.session.session_cost = restored_parent.usd;
|
||||
app.session.session_cost_cny = restored_parent.cny;
|
||||
app.session.subagent_cost = restored_background.usd;
|
||||
app.session.subagent_cost_cny = restored_background.cny;
|
||||
app.session.subagent_usage_sources = session
|
||||
.metadata
|
||||
.cost
|
||||
.usage_source_fingerprints
|
||||
.iter()
|
||||
.cloned()
|
||||
.collect();
|
||||
app.restore_cost_snapshot(&session.metadata.cost);
|
||||
codewhale_engine::cost_status::restore_usage_source_ledger(
|
||||
session
|
||||
.metadata
|
||||
@@ -4937,44 +4917,9 @@ pub(crate) fn apply_loaded_session_with_goal(
|
||||
&session.metadata.cost.missing_usage_sources,
|
||||
session.metadata.cost.missing_usage_overflowed,
|
||||
);
|
||||
// Coverage is restored *with* the money, and the live counters are cleared
|
||||
// first: whatever the previous session in this process priced is not inside
|
||||
// the total being loaded, so carrying those counters over would describe the
|
||||
// wrong total (#4318).
|
||||
app.reset_cost_coverage();
|
||||
app.session.missing_usage_sources = session.metadata.cost.missing_usage_sources.clone();
|
||||
app.session.missing_usage_overflowed = session.metadata.cost.missing_usage_overflowed;
|
||||
app.session.cost_priced_turns = session.metadata.cost.priced_turns;
|
||||
app.session.cost_unpriced_turns = session.metadata.cost.unpriced_turns;
|
||||
app.session.cost_cny_priced_turns = session.metadata.cost.cny_priced_turns;
|
||||
app.session.cost_cny_unpriced_turns = session.metadata.cost.cny_unpriced_turns;
|
||||
app.session.cost_unpriced_reasons = session.metadata.cost.unpriced_reasons.clone();
|
||||
app.session.cost_cny_unpriced_reasons = session.metadata.cost.cny_unpriced_reasons.clone();
|
||||
app.session.cost_unpriced_classes = session.metadata.cost.unpriced_classes.clone();
|
||||
app.session.cost_pricing_provenances = session.metadata.cost.pricing_provenances.clone();
|
||||
app.session.cost_live_pricing_defects = session.metadata.cost.live_pricing_defects.clone();
|
||||
app.session.cost_live_pricing_unusable_defects =
|
||||
session.metadata.cost.live_pricing_unusable_defects.clone();
|
||||
app.session.cost_route_receipts = session.metadata.cost.route_receipts.clone();
|
||||
// A pre-coverage session deserializes its new fields from serde defaults,
|
||||
// which are indistinguishable from "complete total, zero turns". Flag it so
|
||||
// `/cost` says the coverage is unknown rather than claiming completeness,
|
||||
// including for an all-zero record.
|
||||
app.session.cost_coverage_unknown_legacy = session.metadata.cost.coverage_is_legacy_unknown();
|
||||
// Restore the high-water marks from persisted metadata so the
|
||||
// monotonic cost guarantee (#244) survives session restarts.
|
||||
// Take the max with the current totals — old sessions without
|
||||
// persisted high-water fields deserialise to 0.0 and fall back to
|
||||
// the restored total with no regression.
|
||||
let total_restored_usd = session.metadata.cost.total_usd();
|
||||
let total_restored_cny = session.metadata.cost.total_cny();
|
||||
let restored_high_water = codewhale_engine::pricing::CostEstimate {
|
||||
usd: session.metadata.cost.displayed_cost_high_water_usd,
|
||||
cny: session.metadata.cost.displayed_cost_high_water_cny,
|
||||
}
|
||||
.sanitized();
|
||||
app.session.displayed_cost_high_water = restored_high_water.usd.max(total_restored_usd);
|
||||
app.session.displayed_cost_high_water_cny = restored_high_water.cny.max(total_restored_cny);
|
||||
codewhale_engine::cost_status::restore_accounted_usage_sources(
|
||||
&session.metadata.cost.accounted_usage_sources,
|
||||
);
|
||||
app.session.last_prompt_tokens = None;
|
||||
app.session.last_completion_tokens = None;
|
||||
app.session.last_prompt_cache_hit_tokens = None;
|
||||
|
||||
@@ -722,6 +722,15 @@ pub(crate) fn prepare_user_dispatch(
|
||||
};
|
||||
let allowed_tools = allowed_tools_for_message(app.active_allowed_tools.clone(), &message);
|
||||
|
||||
let accounting_origin = app
|
||||
.session_write_receipt
|
||||
.as_ref()
|
||||
.map(|source| source.usage_origin(codewhale_engine::cost_status::scope_token()))
|
||||
.transpose()?;
|
||||
anyhow::ensure!(
|
||||
!should_auto_resolve || accounting_origin.is_some(),
|
||||
"Auto routing has no admitted session accounting origin"
|
||||
);
|
||||
Ok(UserDispatchPrepare {
|
||||
submission_id: uuid::Uuid::new_v4().to_string(),
|
||||
message,
|
||||
@@ -758,6 +767,7 @@ pub(crate) fn prepare_user_dispatch(
|
||||
auto_compact_threshold_percent: app.auto_compact_threshold_percent,
|
||||
snapshot,
|
||||
cost_scope: codewhale_engine::cost_status::scope_token(),
|
||||
accounting_origin,
|
||||
message_index,
|
||||
history_cell,
|
||||
})
|
||||
@@ -944,6 +954,7 @@ pub(crate) async fn spawned_dispatch_inner(
|
||||
// Bound in its own statement: the planner borrows `prepare`, and the error
|
||||
// arm moves it into the failure closure.
|
||||
let plan_result = plan_turn_route(TurnRoutePlanRequest {
|
||||
accounting_origin: prepare.accounting_origin.clone(),
|
||||
route_config: &prepare.route_config,
|
||||
app_route_identity: &prepare.app_route_identity,
|
||||
api_provider: prepare.api_provider,
|
||||
|
||||
@@ -271,6 +271,7 @@ pub(crate) struct UserDispatchPrepare {
|
||||
pub(super) auto_compact_threshold_percent: f64,
|
||||
pub(super) snapshot: UserDispatchSnapshot,
|
||||
pub(super) cost_scope: codewhale_engine::cost_status::CostScopeToken,
|
||||
pub(super) accounting_origin: Option<codewhale_engine::cost_status::NativeUsageOrigin>,
|
||||
pub(super) message_index: usize,
|
||||
pub(super) history_cell: usize,
|
||||
}
|
||||
|
||||
+207
-168
@@ -353,10 +353,7 @@ pub(super) fn persist_settled_session_on_shutdown(
|
||||
|
||||
#[derive(Debug)]
|
||||
struct TranslationAccountingContext {
|
||||
cost_scope: codewhale_engine::cost_status::CostScopeToken,
|
||||
origin_session_id: Option<String>,
|
||||
origin_turn_id: Option<String>,
|
||||
source_id: String,
|
||||
source: codewhale_engine::cost_status::NativeUsageSource,
|
||||
}
|
||||
|
||||
struct SettledTranslation {
|
||||
@@ -366,18 +363,16 @@ struct SettledTranslation {
|
||||
|
||||
impl TranslationAccountingContext {
|
||||
fn capture(app: &App, kind: &str, sequence: u64) -> Self {
|
||||
let raw_source = format!(
|
||||
"translation:{}:{}:{kind}:{sequence}",
|
||||
app.current_session_id.as_deref().unwrap_or("no-session"),
|
||||
app.runtime_turn_id.as_deref().unwrap_or("no-turn")
|
||||
);
|
||||
let origin = app.session_write_receipt.as_ref().and_then(|receipt| {
|
||||
receipt
|
||||
.usage_origin(codewhale_engine::cost_status::scope_token())
|
||||
.ok()
|
||||
});
|
||||
Self {
|
||||
cost_scope: codewhale_engine::cost_status::scope_token(),
|
||||
origin_session_id: app.current_session_id.clone(),
|
||||
origin_turn_id: app.runtime_turn_id.clone(),
|
||||
source_id: format!(
|
||||
"translation:{}",
|
||||
codewhale_engine::cost_status::usage_source_fingerprint(&raw_source)
|
||||
source: codewhale_engine::cost_status::NativeUsageSource::capture_with_origin(
|
||||
format!("translation:{kind}:{sequence}:{}", uuid::Uuid::new_v4()),
|
||||
codewhale_engine::cost_status::UsageSpendOwner::Auxiliary,
|
||||
origin,
|
||||
),
|
||||
}
|
||||
}
|
||||
@@ -386,61 +381,19 @@ impl TranslationAccountingContext {
|
||||
self,
|
||||
response: anyhow::Result<codewhale_engine::client::TranslationProviderResponse>,
|
||||
) -> SettledTranslation {
|
||||
let response = match response {
|
||||
Ok(response) => response,
|
||||
Err(error) => {
|
||||
return SettledTranslation {
|
||||
translated: Err(error),
|
||||
usage: None,
|
||||
};
|
||||
}
|
||||
};
|
||||
if let Some(usage) = response.usage.as_ref() {
|
||||
if let (Some(session_id), Some(turn_id)) = (
|
||||
self.origin_session_id.as_deref(),
|
||||
self.origin_turn_id.as_deref(),
|
||||
) {
|
||||
codewhale_engine::usage_persistence::report_effective_route_for_interactive_origin(
|
||||
self.cost_scope,
|
||||
session_id,
|
||||
turn_id,
|
||||
&self.source_id,
|
||||
&response.route,
|
||||
usage,
|
||||
);
|
||||
} else {
|
||||
codewhale_engine::cost_status::report_effective_route_for_runtime(
|
||||
self.cost_scope,
|
||||
None,
|
||||
&self.source_id,
|
||||
&response.route,
|
||||
usage,
|
||||
);
|
||||
}
|
||||
} else {
|
||||
if let (Some(session_id), Some(turn_id)) = (
|
||||
self.origin_session_id.as_deref(),
|
||||
self.origin_turn_id.as_deref(),
|
||||
) {
|
||||
codewhale_engine::usage_persistence::report_unreceipted_for_interactive_origin(
|
||||
self.cost_scope,
|
||||
session_id,
|
||||
turn_id,
|
||||
&self.source_id,
|
||||
&response.route,
|
||||
);
|
||||
} else {
|
||||
codewhale_engine::cost_status::report_unreceipted_provider_success(
|
||||
self.cost_scope,
|
||||
None,
|
||||
&self.source_id,
|
||||
&response.route,
|
||||
);
|
||||
}
|
||||
}
|
||||
SettledTranslation {
|
||||
translated: response.translated,
|
||||
usage: response.usage,
|
||||
match response {
|
||||
Ok(response) => SettledTranslation {
|
||||
translated: response.translated,
|
||||
usage: if response.accounting.is_some() {
|
||||
None
|
||||
} else {
|
||||
response.usage
|
||||
},
|
||||
},
|
||||
Err(error) => SettledTranslation {
|
||||
translated: Err(error),
|
||||
usage: None,
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1348,6 +1301,8 @@ pub async fn run_tui(
|
||||
app.runtime_services.hook_executor = Some(std::sync::Arc::new(app.hooks.clone()));
|
||||
}
|
||||
app.runtime_services = RuntimeToolServices {
|
||||
accounting_origin: None,
|
||||
|
||||
shell_manager: Some(shell_manager),
|
||||
persist_services_enabled: false,
|
||||
task_manager: Some(task_manager.clone()),
|
||||
@@ -2960,15 +2915,21 @@ pub(crate) async fn run_event_loop(
|
||||
let target_language =
|
||||
app.ui_locale.translation_target_name().to_string();
|
||||
tokio::spawn(async move {
|
||||
let settled = accounting.settle(
|
||||
let response = if !accounting.source.has_captured_origin() {
|
||||
Err(anyhow::anyhow!(
|
||||
"Translation has no admitted session accounting origin"
|
||||
))
|
||||
} else {
|
||||
client
|
||||
.translate_with_usage(
|
||||
&original_text,
|
||||
&translation_model,
|
||||
&target_language,
|
||||
&accounting.source,
|
||||
)
|
||||
.await,
|
||||
);
|
||||
.await
|
||||
};
|
||||
let settled = accounting.settle(response);
|
||||
let _ = tx.send(TranslationEvent::AssistantMessage {
|
||||
origin_session_fingerprint,
|
||||
origin_turn_fingerprint,
|
||||
@@ -3076,15 +3037,21 @@ pub(crate) async fn run_event_loop(
|
||||
let target_language =
|
||||
app.ui_locale.translation_target_name().to_string();
|
||||
tokio::spawn(async move {
|
||||
let settled = accounting.settle(
|
||||
let response = if !accounting.source.has_captured_origin() {
|
||||
Err(anyhow::anyhow!(
|
||||
"Translation has no admitted session accounting origin"
|
||||
))
|
||||
} else {
|
||||
client
|
||||
.translate_with_usage(
|
||||
&original_thinking,
|
||||
&translation_model,
|
||||
&target_language,
|
||||
&accounting.source,
|
||||
)
|
||||
.await,
|
||||
);
|
||||
.await
|
||||
};
|
||||
let settled = accounting.settle(response);
|
||||
let _ = tx.send(TranslationEvent::Thinking {
|
||||
origin_session_fingerprint,
|
||||
origin_turn_fingerprint,
|
||||
@@ -3389,6 +3356,7 @@ pub(crate) async fn run_event_loop(
|
||||
compaction_id,
|
||||
usage,
|
||||
parent_route_usage,
|
||||
source_accounting,
|
||||
routed_usage_dropped_records,
|
||||
status,
|
||||
error,
|
||||
@@ -3540,47 +3508,54 @@ pub(crate) async fn run_event_loop(
|
||||
codewhale_runtime::retry_status::clear();
|
||||
crate::tui::notifications::stop_title_animation_quietly();
|
||||
}
|
||||
let turn_tokens = usage.input_tokens.saturating_add(usage.output_tokens);
|
||||
app.session.total_tokens =
|
||||
app.session.total_tokens.saturating_add(turn_tokens);
|
||||
app.session.total_conversation_tokens = app
|
||||
.session
|
||||
.total_conversation_tokens
|
||||
.saturating_add(turn_tokens);
|
||||
app.session.total_input_tokens = app
|
||||
.session
|
||||
.total_input_tokens
|
||||
.saturating_add(usage.input_tokens);
|
||||
app.session.total_output_tokens = app
|
||||
.session
|
||||
.total_output_tokens
|
||||
.saturating_add(usage.output_tokens);
|
||||
// Only accumulate cache telemetry when the provider
|
||||
// reported at least one cache class. Use pricing's
|
||||
// canonical mutually-exclusive hit/miss/write split so
|
||||
// cache writes are never counted again as misses.
|
||||
if usage.prompt_cache_hit_tokens.is_some()
|
||||
|| usage.prompt_cache_miss_tokens.is_some()
|
||||
|| usage.prompt_cache_write_tokens.is_some()
|
||||
{
|
||||
let classes =
|
||||
codewhale_engine::pricing::token_usage_for_pricing(&usage);
|
||||
let hit_tokens = u32::try_from(classes.cache_read).unwrap_or(u32::MAX);
|
||||
let miss_tokens = u32::try_from(classes.input).unwrap_or(u32::MAX);
|
||||
let write_tokens =
|
||||
u32::try_from(classes.cache_write).unwrap_or(u32::MAX);
|
||||
app.session.total_cache_hit_tokens = app
|
||||
if source_accounting {
|
||||
let pending = codewhale_engine::cost_status::drain();
|
||||
app.absorb_pending_background_cost(&pending);
|
||||
} else {
|
||||
let turn_tokens =
|
||||
usage.input_tokens.saturating_add(usage.output_tokens);
|
||||
app.session.total_tokens =
|
||||
app.session.total_tokens.saturating_add(turn_tokens);
|
||||
app.session.total_conversation_tokens = app
|
||||
.session
|
||||
.total_cache_hit_tokens
|
||||
.saturating_add(hit_tokens);
|
||||
app.session.total_cache_miss_tokens = app
|
||||
.total_conversation_tokens
|
||||
.saturating_add(turn_tokens);
|
||||
app.session.total_input_tokens = app
|
||||
.session
|
||||
.total_cache_miss_tokens
|
||||
.saturating_add(miss_tokens);
|
||||
app.session.total_cache_write_tokens = app
|
||||
.total_input_tokens
|
||||
.saturating_add(usage.input_tokens);
|
||||
app.session.total_output_tokens = app
|
||||
.session
|
||||
.total_cache_write_tokens
|
||||
.saturating_add(write_tokens);
|
||||
.total_output_tokens
|
||||
.saturating_add(usage.output_tokens);
|
||||
// Only accumulate cache telemetry when the provider
|
||||
// reported at least one cache class. Use pricing's
|
||||
// canonical mutually-exclusive hit/miss/write split so
|
||||
// cache writes are never counted again as misses.
|
||||
if usage.prompt_cache_hit_tokens.is_some()
|
||||
|| usage.prompt_cache_miss_tokens.is_some()
|
||||
|| usage.prompt_cache_write_tokens.is_some()
|
||||
{
|
||||
let classes =
|
||||
codewhale_engine::pricing::token_usage_for_pricing(&usage);
|
||||
let hit_tokens =
|
||||
u32::try_from(classes.cache_read).unwrap_or(u32::MAX);
|
||||
let miss_tokens = u32::try_from(classes.input).unwrap_or(u32::MAX);
|
||||
let write_tokens =
|
||||
u32::try_from(classes.cache_write).unwrap_or(u32::MAX);
|
||||
app.session.total_cache_hit_tokens = app
|
||||
.session
|
||||
.total_cache_hit_tokens
|
||||
.saturating_add(hit_tokens);
|
||||
app.session.total_cache_miss_tokens = app
|
||||
.session
|
||||
.total_cache_miss_tokens
|
||||
.saturating_add(miss_tokens);
|
||||
app.session.total_cache_write_tokens = app
|
||||
.session
|
||||
.total_cache_write_tokens
|
||||
.saturating_add(write_tokens);
|
||||
}
|
||||
}
|
||||
app.session.last_prompt_tokens = Some(usage.input_tokens);
|
||||
app.session.last_completion_tokens = Some(usage.output_tokens);
|
||||
@@ -3653,7 +3628,10 @@ pub(crate) async fn run_event_loop(
|
||||
.as_ref()
|
||||
.and_then(|turn| turn.route.as_ref())
|
||||
.and_then(codewhale_engine::core::events::TurnRoute::cost_envelope)
|
||||
.filter(|_| parent_route_usage != codewhale_models::Usage::default())
|
||||
.filter(|_| {
|
||||
!source_accounting
|
||||
&& parent_route_usage != codewhale_models::Usage::default()
|
||||
})
|
||||
.map(|route| route.audit(&parent_route_usage));
|
||||
app.push_turn_cache_record(crate::tui::app::TurnCacheRecord {
|
||||
provider,
|
||||
@@ -4873,7 +4851,9 @@ pub(crate) async fn run_event_loop(
|
||||
>(value)
|
||||
.ok()
|
||||
});
|
||||
if let Some(source) = receipt["source_id"].as_str() {
|
||||
if receipt["source_accounting"].as_bool() != Some(true)
|
||||
&& let Some(source) = receipt["source_id"].as_str()
|
||||
{
|
||||
if receipt["usage_reported"].as_bool() == Some(true) {
|
||||
app.session.subagent_usage_sources.insert(
|
||||
codewhale_engine::cost_status::usage_source_fingerprint(
|
||||
@@ -4899,6 +4879,18 @@ pub(crate) async fn run_event_loop(
|
||||
app.absorb_pending_background_cost(&pool);
|
||||
}
|
||||
}
|
||||
if let Some(error) = receipt["accounting_error"].as_str() {
|
||||
app.session
|
||||
.cost_unpriced_reasons
|
||||
.insert("usage_settlement_unacknowledged".into());
|
||||
app.session
|
||||
.cost_cny_unpriced_reasons
|
||||
.insert("usage_settlement_unacknowledged".into());
|
||||
app.session.missing_usage_overflowed = true;
|
||||
app.status_message = Some(format!(
|
||||
"Provider usage settlement is uncertain: {error}"
|
||||
));
|
||||
}
|
||||
let leaf = app.session_journal.leaf_id.clone();
|
||||
app.session_journal.append(
|
||||
codewhale_runtime::session_tree::SessionEntryKind::System {
|
||||
@@ -4961,12 +4953,14 @@ pub(crate) async fn run_event_loop(
|
||||
}
|
||||
}
|
||||
EngineEvent::TurnUsage {
|
||||
accounting,
|
||||
max_output_tokens: _,
|
||||
usage,
|
||||
duration_ms,
|
||||
first_token_ms,
|
||||
request_ms,
|
||||
finish_reason: _,
|
||||
..
|
||||
} => {
|
||||
// Per-step usage receipt. The session metrics strip
|
||||
// folds each model call's timing (stream time, TTFT,
|
||||
@@ -4999,16 +4993,21 @@ pub(crate) async fn run_event_loop(
|
||||
// this and lands the authoritative cumulative price
|
||||
// through the same audit path, so nothing counts
|
||||
// twice and the two can never disagree on route.
|
||||
let step_cost = app
|
||||
.active_turn
|
||||
.as_ref()
|
||||
.and_then(|turn| turn.route.as_ref())
|
||||
.and_then(codewhale_engine::core::events::TurnRoute::cost_envelope)
|
||||
.and_then(|route| route.audit(&usage).estimate);
|
||||
if let Some(cost) = step_cost {
|
||||
app.accrue_pending_turn_cost_estimate(cost);
|
||||
if accounting.is_none() {
|
||||
let step_cost = app
|
||||
.active_turn
|
||||
.as_ref()
|
||||
.and_then(|turn| turn.route.as_ref())
|
||||
.and_then(codewhale_engine::core::events::TurnRoute::cost_envelope)
|
||||
.and_then(|route| route.audit(&usage).estimate);
|
||||
if let Some(cost) = step_cost {
|
||||
app.accrue_pending_turn_cost_estimate(cost);
|
||||
}
|
||||
app.session.accrue_pending_turn_usage(&usage);
|
||||
} else {
|
||||
let pending = codewhale_engine::cost_status::drain();
|
||||
app.absorb_pending_background_cost(&pending);
|
||||
}
|
||||
app.session.accrue_pending_turn_usage(&usage);
|
||||
}
|
||||
EngineEvent::RoutedTurnUsage {
|
||||
usage,
|
||||
@@ -8669,91 +8668,131 @@ mod session_boot_event_tests {
|
||||
#[test]
|
||||
fn assistant_and_thinking_translation_usage_each_accrue_once() {
|
||||
let _scope = codewhale_engine::cost_status::test_scope();
|
||||
let directory = tempfile::tempdir().unwrap();
|
||||
let manager = SessionManager::new(directory.path().to_path_buf()).unwrap();
|
||||
let mut app = test_app();
|
||||
app.current_session_id = Some("session-translation".to_string());
|
||||
app.runtime_turn_id = Some("turn-translation".to_string());
|
||||
app.session_write_receipt = Some(
|
||||
manager
|
||||
.expect_absent_session("session-translation")
|
||||
.unwrap(),
|
||||
);
|
||||
let usage_a = codewhale_models::Usage {
|
||||
input_tokens: 5,
|
||||
output_tokens: 2,
|
||||
..codewhale_models::Usage::default()
|
||||
..Default::default()
|
||||
};
|
||||
let usage_b = codewhale_models::Usage {
|
||||
input_tokens: 3,
|
||||
output_tokens: 1,
|
||||
..codewhale_models::Usage::default()
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let assistant = TranslationAccountingContext::capture(&app, "assistant", 1).settle(Ok(
|
||||
codewhale_engine::client::TranslationProviderResponse {
|
||||
let assistant = TranslationAccountingContext::capture(&app, "assistant", 1);
|
||||
let record_a = assistant
|
||||
.source
|
||||
.reported(&translation_test_route(), &usage_a)
|
||||
.unwrap();
|
||||
manager
|
||||
.persist_runtime_usage_record("session-translation", &record_a)
|
||||
.unwrap();
|
||||
let assistant =
|
||||
assistant.settle(Ok(codewhale_engine::client::TranslationProviderResponse {
|
||||
accounting: Some(record_a.clone()),
|
||||
translated: Ok("助理".to_string()),
|
||||
route: translation_test_route(),
|
||||
route: record_a.usage.route.clone(),
|
||||
usage: Some(usage_a.clone()),
|
||||
},
|
||||
));
|
||||
let thinking = TranslationAccountingContext::capture(&app, "thinking", 2).settle(Ok(
|
||||
codewhale_engine::client::TranslationProviderResponse {
|
||||
translated: Err(anyhow::anyhow!("incomplete: max_tokens")),
|
||||
route: translation_test_route(),
|
||||
usage: Some(usage_b.clone()),
|
||||
},
|
||||
));
|
||||
|
||||
assert_eq!(assistant.usage.as_ref(), Some(&usage_a));
|
||||
assert_eq!(thinking.usage.as_ref(), Some(&usage_b));
|
||||
}));
|
||||
let thinking = TranslationAccountingContext::capture(&app, "thinking", 2);
|
||||
let record_b = thinking
|
||||
.source
|
||||
.reported(&translation_test_route(), &usage_b)
|
||||
.unwrap();
|
||||
manager
|
||||
.persist_runtime_usage_record("session-translation", &record_b)
|
||||
.unwrap();
|
||||
let thinking = thinking.settle(Ok(codewhale_engine::client::TranslationProviderResponse {
|
||||
accounting: Some(record_b.clone()),
|
||||
translated: Err(anyhow::anyhow!("incomplete: max_tokens")),
|
||||
route: record_b.usage.route.clone(),
|
||||
usage: Some(usage_b.clone()),
|
||||
}));
|
||||
assert_eq!(assistant.usage, None);
|
||||
assert_eq!(thinking.usage, None);
|
||||
assert!(
|
||||
thinking.translated.is_err(),
|
||||
"semantic rejection is preserved"
|
||||
);
|
||||
accrue_translation_usage(&mut app, assistant.usage.as_ref().expect("assistant usage"));
|
||||
accrue_translation_usage(&mut app, thinking.usage.as_ref().expect("thinking usage"));
|
||||
for record in [&record_a, &record_b, &record_a, &record_b] {
|
||||
let _ = app.absorb_usage_source(record).unwrap();
|
||||
}
|
||||
assert_eq!(app.session.total_input_tokens, 8);
|
||||
assert_eq!(app.session.total_output_tokens, 3);
|
||||
assert_eq!(app.session.total_tokens, 11);
|
||||
|
||||
let pending = codewhale_engine::cost_status::drain();
|
||||
assert_eq!(
|
||||
pending.priced_turns.saturating_add(pending.unpriced_turns),
|
||||
2,
|
||||
"each decoded provider response is audited exactly once"
|
||||
app.session
|
||||
.cost_priced_turns
|
||||
.saturating_add(app.session.cost_unpriced_turns),
|
||||
2
|
||||
);
|
||||
assert!(
|
||||
codewhale_engine::cost_status::drain()
|
||||
.source_records
|
||||
.is_empty()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn translation_unreceipted_success_is_marked_once_but_transport_failure_is_not() {
|
||||
let _scope = codewhale_engine::cost_status::test_scope();
|
||||
let directory = tempfile::tempdir().unwrap();
|
||||
let manager = SessionManager::new(directory.path().to_path_buf()).unwrap();
|
||||
let mut app = test_app();
|
||||
app.current_session_id = Some("session-translation-missing-usage".to_string());
|
||||
app.runtime_turn_id = Some("turn-translation-missing-usage".to_string());
|
||||
|
||||
app.session_write_receipt = Some(
|
||||
manager
|
||||
.expect_absent_session("session-translation-missing-usage")
|
||||
.unwrap(),
|
||||
);
|
||||
let context = TranslationAccountingContext::capture(&app, "assistant", 7);
|
||||
let record = context
|
||||
.source
|
||||
.reported(
|
||||
&translation_test_route(),
|
||||
&codewhale_models::Usage::default(),
|
||||
)
|
||||
.unwrap();
|
||||
manager
|
||||
.persist_runtime_usage_record("session-translation-missing-usage", &record)
|
||||
.unwrap();
|
||||
for _ in 0..2 {
|
||||
let settled = TranslationAccountingContext::capture(&app, "assistant", 7).settle(Ok(
|
||||
codewhale_engine::client::TranslationProviderResponse {
|
||||
translated: Ok("translation remains usable".to_string()),
|
||||
route: translation_test_route(),
|
||||
usage: None,
|
||||
},
|
||||
));
|
||||
let settled = TranslationAccountingContext {
|
||||
source: context.source.clone(),
|
||||
}
|
||||
.settle(Ok(codewhale_engine::client::TranslationProviderResponse {
|
||||
accounting: Some(record.clone()),
|
||||
translated: Ok("translation remains usable".to_string()),
|
||||
route: record.usage.route.clone(),
|
||||
usage: None,
|
||||
}));
|
||||
assert_eq!(
|
||||
settled.translated.expect("semantic output remains usable"),
|
||||
"translation remains usable"
|
||||
);
|
||||
assert_eq!(settled.usage, None);
|
||||
let _ = app.absorb_usage_source(&record).unwrap();
|
||||
}
|
||||
let transport = TranslationAccountingContext::capture(&app, "assistant", 8)
|
||||
.settle(Err(anyhow::anyhow!("HTTP 429")));
|
||||
assert!(transport.translated.is_err());
|
||||
assert_eq!(transport.usage, None);
|
||||
|
||||
let pending = codewhale_engine::cost_status::drain();
|
||||
assert_eq!(pending.priced_turns, 0);
|
||||
assert_eq!(
|
||||
pending.unpriced_turns, 1,
|
||||
"stable response id dedupes replay"
|
||||
);
|
||||
assert_eq!(pending.cny_unpriced_turns, 1);
|
||||
assert_eq!(app.session.cost_priced_turns, 0);
|
||||
assert_eq!(app.session.cost_unpriced_turns, 1);
|
||||
assert_eq!(app.session.cost_cny_unpriced_turns, 1);
|
||||
assert!(
|
||||
pending
|
||||
.unpriced_reasons
|
||||
app.session
|
||||
.cost_unpriced_reasons
|
||||
.contains("provider_success_missing_usage")
|
||||
);
|
||||
}
|
||||
|
||||
@@ -920,6 +920,7 @@ pub(crate) async fn build_preview_request_inputs(
|
||||
Err(error) => return posture(None, PreviewUnresolved::PlanFailed(error)),
|
||||
};
|
||||
let planned = plan_turn_route(TurnRoutePlanRequest {
|
||||
accounting_origin: None,
|
||||
route_config: &route_config,
|
||||
app_route_identity: &app_route_identity,
|
||||
api_provider: app.api_provider,
|
||||
|
||||
+277
-111
@@ -76,9 +76,9 @@ fn failed_engine_channel_settles_classifier_batch_once() {
|
||||
);
|
||||
let batch = codewhale_engine::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: vec![codewhale_engine::cost_status::RuntimeUsageRecord {
|
||||
source_id: "auto-router:dispatch-usage".to_string(),
|
||||
usage: codewhale_engine::cost_status::EffectiveRouteUsage {
|
||||
records: vec![codewhale_engine::cost_status::RuntimeUsageRecord::legacy(
|
||||
"auto-router:dispatch-usage".to_string(),
|
||||
codewhale_engine::cost_status::EffectiveRouteUsage {
|
||||
route: route.clone(),
|
||||
usage: codewhale_models::Usage {
|
||||
input_tokens: 6,
|
||||
@@ -86,7 +86,7 @@ fn failed_engine_channel_settles_classifier_batch_once() {
|
||||
..Default::default()
|
||||
},
|
||||
},
|
||||
}],
|
||||
)],
|
||||
drop_records: vec![codewhale_engine::cost_status::RuntimeUsageDropRecord {
|
||||
reason: codewhale_engine::cost_status::RuntimeUsageMissingReason::default(),
|
||||
source_id: "auto-router:dispatch-drop".to_string(),
|
||||
@@ -818,6 +818,44 @@ fn test_mailbox_route(
|
||||
)
|
||||
}
|
||||
|
||||
fn settled_child_fixture_record(
|
||||
app: &mut App,
|
||||
manager: &codewhale_engine::session_manager::SessionManager,
|
||||
source_id: &str,
|
||||
route: &codewhale_engine::cost_status::EffectiveRouteEnvelope,
|
||||
usage: &codewhale_models::Usage,
|
||||
) -> codewhale_engine::cost_status::RuntimeUsageRecord {
|
||||
if app.session_write_receipt.is_none() {
|
||||
let snapshot = build_session_snapshot(app, manager).expect("native fixture snapshot");
|
||||
app.current_session_id = Some(snapshot.metadata.id.clone());
|
||||
app.session_write_receipt = snapshot.write_receipt;
|
||||
app.current_session_metadata = Some(snapshot.metadata);
|
||||
}
|
||||
let scope = codewhale_engine::cost_status::scope_token();
|
||||
let origin = app
|
||||
.session_write_receipt
|
||||
.as_ref()
|
||||
.expect("native fixture source")
|
||||
.usage_origin(scope)
|
||||
.expect("native fixture origin");
|
||||
let source = codewhale_engine::cost_status::NativeUsageSource::capture_with_origin(
|
||||
source_id,
|
||||
codewhale_engine::cost_status::UsageSpendOwner::Auxiliary,
|
||||
Some(origin),
|
||||
);
|
||||
let record = source
|
||||
.prepare(
|
||||
route,
|
||||
usage,
|
||||
codewhale_engine::llm_client::ProviderUsageCoverage::Reported,
|
||||
codewhale_engine::cost_status::RuntimeUsageMissingReason::SuccessWithoutUsage,
|
||||
)
|
||||
.expect("native fixture record");
|
||||
codewhale_engine::cost_status::report_runtime_usage_record(scope, None, &record)
|
||||
.expect("durable fixture publication");
|
||||
record
|
||||
}
|
||||
|
||||
fn served_endpoint_fingerprint() -> Option<String> {
|
||||
codewhale_engine::cost_status::endpoint_fingerprint("https://api.deepseek.com/v1")
|
||||
}
|
||||
@@ -9934,6 +9972,10 @@ fn apply_loaded_session_resets_unpersisted_telemetry() {
|
||||
fn apply_loaded_session_replaces_cost_coverage_with_the_loaded_total() {
|
||||
let _cost_scope = codewhale_engine::cost_status::test_scope();
|
||||
let mut app = create_test_app();
|
||||
app.session.displayed_cost_high_water = 900.0;
|
||||
app.session.displayed_cost_high_water_cny = 6000.0;
|
||||
app.last_billed_input_tokens = Some(700);
|
||||
|
||||
// Live state from the session being replaced.
|
||||
app.session.cost_priced_turns = 9;
|
||||
app.session.cost_unpriced_turns = 4;
|
||||
@@ -9974,6 +10016,9 @@ fn apply_loaded_session_replaces_cost_coverage_with_the_loaded_total() {
|
||||
|
||||
apply_loaded_session(&mut app, &mut Config::default(), &session).expect("restore session");
|
||||
|
||||
assert_eq!(app.session.displayed_cost_high_water, 3.75);
|
||||
assert_eq!(app.session.displayed_cost_high_water_cny, 0.0);
|
||||
assert_eq!(app.last_billed_input_tokens, None);
|
||||
assert_eq!(app.session.cost_priced_turns, 2);
|
||||
assert_eq!(app.session.subagent_cost, 1.25);
|
||||
assert_eq!(app.session.cost_unpriced_turns, 1);
|
||||
@@ -16751,6 +16796,8 @@ fn user_input_completion_preserves_newer_question_and_dispatch() {
|
||||
observe_human_request_settlement(
|
||||
&mut app,
|
||||
&EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -16798,6 +16845,8 @@ fn user_input_turn_end_cancel_and_disconnect_retire_question_views() {
|
||||
_ => observe_human_request_settlement(
|
||||
&mut app,
|
||||
&EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -18002,12 +18051,13 @@ fn reconcile_subagent_activity_state_caps_terminal_card_bursts() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn subagent_token_usage_updates_live_cost_counter_without_card_change() {
|
||||
fn legacy_subagent_token_usage_marks_uncertainty_without_card_change() {
|
||||
let mut app = create_test_app();
|
||||
handle_subagent_mailbox(
|
||||
&mut app,
|
||||
1,
|
||||
&codewhale_engine::tools::subagent::MailboxMessage::TokenUsage {
|
||||
record: None,
|
||||
agent_id: "agent-a".to_string(),
|
||||
source_id: "response-a".to_string(),
|
||||
route: Box::new(test_mailbox_route(
|
||||
@@ -18022,7 +18072,12 @@ fn subagent_token_usage_updates_live_cost_counter_without_card_change() {
|
||||
},
|
||||
);
|
||||
|
||||
assert!(app.session.subagent_cost > 0.0);
|
||||
assert_eq!(app.session.subagent_cost, 0.0);
|
||||
assert!(
|
||||
app.session
|
||||
.cost_unpriced_reasons
|
||||
.contains("legacy_usage_unverified")
|
||||
);
|
||||
assert!(
|
||||
app.history.is_empty(),
|
||||
"usage-only mailbox messages should not allocate a sub-agent card"
|
||||
@@ -18038,6 +18093,7 @@ fn subagent_token_usage_prices_the_child_route_not_the_parent_route() {
|
||||
&mut app,
|
||||
2,
|
||||
&codewhale_engine::tools::subagent::MailboxMessage::TokenUsage {
|
||||
record: None,
|
||||
agent_id: "agent-codex".to_string(),
|
||||
source_id: "response-codex".to_string(),
|
||||
route: Box::new(test_mailbox_route(ProviderKind::OpenaiCodex, "gpt-5.5")),
|
||||
@@ -18058,31 +18114,37 @@ fn subagent_token_usage_prices_the_child_route_not_the_parent_route() {
|
||||
#[test]
|
||||
fn subagent_token_usage_is_deduped_by_response_source() {
|
||||
let mut app = create_test_app();
|
||||
let usage = codewhale_engine::tools::subagent::MailboxMessage::TokenUsage {
|
||||
agent_id: "agent-a".to_string(),
|
||||
source_id: "response-a".to_string(),
|
||||
route: Box::new(test_mailbox_route(
|
||||
ProviderKind::Deepseek,
|
||||
"deepseek-v4-flash",
|
||||
)),
|
||||
usage: codewhale_models::Usage {
|
||||
input_tokens: 10_000,
|
||||
output_tokens: 1_000,
|
||||
..Default::default()
|
||||
},
|
||||
let fixture_dir = tempfile::tempdir().expect("fixture dir");
|
||||
let fixture_manager =
|
||||
codewhale_engine::session_manager::SessionManager::new(fixture_dir.path().join("sessions"))
|
||||
.expect("fixture manager");
|
||||
let route = test_mailbox_route(ProviderKind::Deepseek, "deepseek-v4-flash");
|
||||
let source_usage = codewhale_models::Usage {
|
||||
input_tokens: 10_000,
|
||||
output_tokens: 1_000,
|
||||
..Default::default()
|
||||
};
|
||||
let record = settled_child_fixture_record(
|
||||
&mut app,
|
||||
&fixture_manager,
|
||||
"response-a",
|
||||
&route,
|
||||
&source_usage,
|
||||
);
|
||||
let usage = codewhale_engine::tools::subagent::MailboxMessage::usage_record("agent-a", record);
|
||||
|
||||
handle_subagent_mailbox(&mut app, 7, &usage);
|
||||
let first = app.session.subagent_cost;
|
||||
handle_subagent_mailbox(&mut app, 8, &usage);
|
||||
assert_eq!(app.session.subagent_cost, first);
|
||||
let mut distinct = usage.clone();
|
||||
let codewhale_engine::tools::subagent::MailboxMessage::TokenUsage { source_id, .. } =
|
||||
&mut distinct
|
||||
else {
|
||||
unreachable!()
|
||||
};
|
||||
*source_id = "response-b".to_string();
|
||||
let next = settled_child_fixture_record(
|
||||
&mut app,
|
||||
&fixture_manager,
|
||||
"independent-response",
|
||||
&route,
|
||||
&source_usage,
|
||||
);
|
||||
let distinct = codewhale_engine::tools::subagent::MailboxMessage::usage_record("agent-a", next);
|
||||
handle_subagent_mailbox(&mut app, 9, &distinct);
|
||||
assert!(app.session.subagent_cost > first);
|
||||
}
|
||||
@@ -18107,6 +18169,10 @@ fn active_deleted_origin_mailbox_usage_does_not_accrue_or_restore_the_session()
|
||||
manager.save_session(&session).expect("save origin");
|
||||
app.current_session_id = Some(session.metadata.id.clone());
|
||||
app.current_session_metadata = Some(session.metadata.clone());
|
||||
app.session_write_receipt = manager
|
||||
.load_session(&session.metadata.id)
|
||||
.expect("native origin read")
|
||||
.write_receipt;
|
||||
let origin_scope = codewhale_engine::cost_status::scope_token();
|
||||
let owner = "interactive:active-deleted-origin:origin-turn";
|
||||
codewhale_engine::usage_persistence::register_persistent_interactive_runtime_usage_sink_for_test(
|
||||
@@ -18129,11 +18195,21 @@ fn active_deleted_origin_mailbox_usage_does_not_accrue_or_restore_the_session()
|
||||
.is_some_and(|cost| cost.usd > 0.0)
|
||||
);
|
||||
let source_id = "subagent:deleted-origin:response";
|
||||
let mailbox = codewhale_engine::tools::subagent::MailboxMessage::token_usage(
|
||||
"agent-deleted-origin",
|
||||
let source = codewhale_engine::cost_status::NativeUsageSource::capture_with_origin(
|
||||
source_id,
|
||||
route.clone(),
|
||||
usage.clone(),
|
||||
codewhale_engine::cost_status::UsageSpendOwner::Auxiliary,
|
||||
Some(
|
||||
app.session_write_receipt
|
||||
.as_ref()
|
||||
.expect("native source")
|
||||
.usage_origin(origin_scope)
|
||||
.expect("native origin"),
|
||||
),
|
||||
);
|
||||
let record = source.reported(&route, &usage).expect("native receipt");
|
||||
let mailbox = codewhale_engine::tools::subagent::MailboxMessage::usage_record(
|
||||
"agent-deleted-origin",
|
||||
record.clone(),
|
||||
);
|
||||
manager
|
||||
.delete_session(&session.metadata.id)
|
||||
@@ -18142,13 +18218,8 @@ fn active_deleted_origin_mailbox_usage_does_not_accrue_or_restore_the_session()
|
||||
// an active-origin callback, not the already-covered stale-scope path.
|
||||
assert_eq!(codewhale_engine::cost_status::scope_token(), origin_scope);
|
||||
for seq in 1..=2 {
|
||||
codewhale_engine::cost_status::report_effective_route_for_runtime(
|
||||
origin_scope,
|
||||
Some(owner),
|
||||
source_id,
|
||||
&route,
|
||||
&usage,
|
||||
);
|
||||
codewhale_engine::cost_status::report_runtime_usage_record(origin_scope, None, &record)
|
||||
.expect("handled retired receipt");
|
||||
handle_subagent_mailbox_for_turn(&mut app, "origin-turn", seq, &mailbox);
|
||||
}
|
||||
assert!(codewhale_engine::cost_status::drain().is_empty());
|
||||
@@ -18183,31 +18254,37 @@ fn active_deleted_origin_mailbox_usage_does_not_accrue_or_restore_the_session()
|
||||
#[test]
|
||||
fn subagent_token_usage_source_is_stable_across_engine_turns() {
|
||||
let mut app = create_test_app();
|
||||
let usage = codewhale_engine::tools::subagent::MailboxMessage::TokenUsage {
|
||||
agent_id: "agent-a".to_string(),
|
||||
source_id: "response-a".to_string(),
|
||||
route: Box::new(test_mailbox_route(
|
||||
ProviderKind::Deepseek,
|
||||
"deepseek-v4-flash",
|
||||
)),
|
||||
usage: codewhale_models::Usage {
|
||||
input_tokens: 10_000,
|
||||
output_tokens: 1_000,
|
||||
..Default::default()
|
||||
},
|
||||
let fixture_dir = tempfile::tempdir().expect("fixture dir");
|
||||
let fixture_manager =
|
||||
codewhale_engine::session_manager::SessionManager::new(fixture_dir.path().join("sessions"))
|
||||
.expect("fixture manager");
|
||||
let route = test_mailbox_route(ProviderKind::Deepseek, "deepseek-v4-flash");
|
||||
let source_usage = codewhale_models::Usage {
|
||||
input_tokens: 10_000,
|
||||
output_tokens: 1_000,
|
||||
..Default::default()
|
||||
};
|
||||
let record = settled_child_fixture_record(
|
||||
&mut app,
|
||||
&fixture_manager,
|
||||
"response-a",
|
||||
&route,
|
||||
&source_usage,
|
||||
);
|
||||
let usage = codewhale_engine::tools::subagent::MailboxMessage::usage_record("agent-a", record);
|
||||
|
||||
handle_subagent_mailbox_for_turn(&mut app, "engine-turn-a", 1, &usage);
|
||||
let first = app.session.subagent_cost;
|
||||
handle_subagent_mailbox_for_turn(&mut app, "engine-turn-b", 1, &usage);
|
||||
assert_eq!(app.session.subagent_cost, first, "same response dedupes");
|
||||
let mut distinct = usage.clone();
|
||||
let codewhale_engine::tools::subagent::MailboxMessage::TokenUsage { source_id, .. } =
|
||||
&mut distinct
|
||||
else {
|
||||
unreachable!()
|
||||
};
|
||||
*source_id = "response-b".to_string();
|
||||
let next = settled_child_fixture_record(
|
||||
&mut app,
|
||||
&fixture_manager,
|
||||
"independent-response",
|
||||
&route,
|
||||
&source_usage,
|
||||
);
|
||||
let distinct = codewhale_engine::tools::subagent::MailboxMessage::usage_record("agent-a", next);
|
||||
handle_subagent_mailbox_for_turn(&mut app, "engine-turn-b", 2, &distinct);
|
||||
assert!(
|
||||
app.session.subagent_cost > first,
|
||||
@@ -19441,6 +19518,8 @@ fn local_cancel_marks_late_stream_events_for_suppression() {
|
||||
));
|
||||
assert!(!suppress_engine_event_after_local_cancel(
|
||||
&EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -19472,6 +19551,8 @@ fn turn_end_outbox_uses_the_terminal_events_owner() {
|
||||
app.runtime_turn_id = Some("turn-model".to_string());
|
||||
|
||||
let model_terminal = EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: Some("turn-model".to_string()),
|
||||
submission_id: None,
|
||||
compaction_id: None,
|
||||
@@ -19493,6 +19574,8 @@ fn turn_end_outbox_uses_the_terminal_events_owner() {
|
||||
assert_eq!(model.turn_id.as_deref(), Some("turn-model"));
|
||||
|
||||
let rejected_terminal = EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: Some("submission-rejected".to_string()),
|
||||
compaction_id: None,
|
||||
@@ -19514,6 +19597,8 @@ fn turn_end_outbox_uses_the_terminal_events_owner() {
|
||||
assert!(rejected.turn_id.is_none());
|
||||
|
||||
let compaction_terminal = EngineEvent::TurnComplete {
|
||||
source_accounting: false,
|
||||
|
||||
turn_id: None,
|
||||
submission_id: None,
|
||||
compaction_id: Some("compact-manual".to_string()),
|
||||
@@ -23690,10 +23775,6 @@ fn shell_wait_without_command_uses_task_id_until_command_metadata_arrives() {
|
||||
#[test]
|
||||
fn legacy_child_usage_metadata_fails_closed_without_parent_route_fallback() {
|
||||
let mut app = create_test_app();
|
||||
// An in-process review child publishes no route of its own, so it runs on
|
||||
// the parent turn's client and is billed from the parent's frozen receipt.
|
||||
// Without a receipt there is nothing sound to bill from, so the turn has to
|
||||
// be active for the child to accrue anything.
|
||||
app.active_turn = Some(crate::tui::app::ActiveTurnMetadata {
|
||||
turn_id: "turn-child-usage".to_string(),
|
||||
created_at: chrono::Utc::now(),
|
||||
@@ -23743,7 +23824,7 @@ fn legacy_child_usage_metadata_fails_closed_without_parent_route_fallback() {
|
||||
assert!(
|
||||
app.session
|
||||
.cost_unpriced_reasons
|
||||
.contains("unknown_billing_basis")
|
||||
.contains("legacy_usage_unverified")
|
||||
);
|
||||
}
|
||||
|
||||
@@ -23774,24 +23855,20 @@ fn child_usage_metadata_carries_cache_write_and_reasoning_end_to_end() {
|
||||
};
|
||||
|
||||
// The shared producer emits every class.
|
||||
let priced_route = codewhale_engine::cost_status::EffectiveRouteEnvelope {
|
||||
openrouter_vendor: None,
|
||||
provider: codewhale_config::ProviderKind::Anthropic,
|
||||
provider_identity: "anthropic-api".to_string(),
|
||||
model: "claude-haiku-4-5".to_string(),
|
||||
billing_surface: Some(
|
||||
codewhale_engine::pricing::FIRST_PARTY_PAYG_BILLING_SURFACE.to_string(),
|
||||
),
|
||||
// Must be a real fingerprint: persistence and receipt boundaries drop
|
||||
// anything that is not one, and a dropped field would have made the
|
||||
// round-trip assertion below vacuously pass on both sides.
|
||||
endpoint_fingerprint: codewhale_engine::cost_status::endpoint_fingerprint(
|
||||
"https://api.anthropic.com/v1",
|
||||
),
|
||||
provider_live_pricing: None,
|
||||
billing_mode: codewhale_engine::cost_status::RouteBillingMode::Metered,
|
||||
dispatched_at: chrono::DateTime::<chrono::Utc>::from_timestamp(0, 0).expect("epoch"),
|
||||
};
|
||||
let api_config = codewhale_runtime::config::Config::default();
|
||||
let priced_route = codewhale_engine::cost_status::EffectiveRouteEnvelope::capture(
|
||||
Some(&api_config),
|
||||
ProviderKind::Anthropic,
|
||||
ProviderKind::Anthropic.as_str(),
|
||||
"claude-haiku-4-5",
|
||||
Some(ProviderKind::Anthropic.provider().default_base_url()),
|
||||
chrono::Utc::now(),
|
||||
);
|
||||
assert_eq!(
|
||||
priced_route.billing_mode,
|
||||
codewhale_engine::cost_status::RouteBillingMode::Metered,
|
||||
);
|
||||
assert!(priced_route.provider_live_pricing.is_some());
|
||||
let fields =
|
||||
codewhale_engine::cost_status::child_usage_metadata_fields(&priced_route, &child_usage);
|
||||
assert_eq!(fields["child_prompt_cache_write_tokens"], 100_000);
|
||||
@@ -23809,6 +23886,10 @@ fn child_usage_metadata_carries_cache_write_and_reasoning_end_to_end() {
|
||||
// The parent deliberately stays DeepSeek: only the frozen child envelope
|
||||
// may control this price.
|
||||
let mut priced_app = create_test_app();
|
||||
let fixture_dir = tempfile::tempdir().expect("fixture dir");
|
||||
let fixture_manager =
|
||||
codewhale_engine::session_manager::SessionManager::new(fixture_dir.path().join("sessions"))
|
||||
.expect("fixture manager");
|
||||
priced_app.api_provider = codewhale_config::ProviderKind::Deepseek;
|
||||
priced_app.billing_presentation = codewhale_engine::route_billing::BillingPresentation::Metered;
|
||||
let mut metadata = serde_json::json!({});
|
||||
@@ -23817,6 +23898,20 @@ fn child_usage_metadata_carries_cache_write_and_reasoning_end_to_end() {
|
||||
&priced_route,
|
||||
&child_usage,
|
||||
);
|
||||
let record = settled_child_fixture_record(
|
||||
&mut priced_app,
|
||||
&fixture_manager,
|
||||
"cache-class-response",
|
||||
&priced_route,
|
||||
&child_usage,
|
||||
);
|
||||
codewhale_engine::cost_status::attach_child_usage_batch_metadata(
|
||||
&mut metadata,
|
||||
&codewhale_engine::cost_status::RuntimeUsageBatch {
|
||||
records: vec![record],
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
assert_eq!(
|
||||
codewhale_engine::cost_status::child_route_envelope_from_metadata(&metadata),
|
||||
Some(priced_route.clone()),
|
||||
@@ -23843,25 +23938,45 @@ fn child_usage_metadata_carries_cache_write_and_reasoning_end_to_end() {
|
||||
unpriced_app.api_provider = codewhale_config::ProviderKind::Deepseek;
|
||||
unpriced_app.billing_presentation =
|
||||
codewhale_engine::route_billing::BillingPresentation::Metered;
|
||||
let unpriced_route = codewhale_engine::cost_status::EffectiveRouteEnvelope {
|
||||
openrouter_vendor: None,
|
||||
provider: codewhale_config::ProviderKind::Moonshot,
|
||||
provider_identity: "moonshot-api".to_string(),
|
||||
model: "kimi-k2.7-code".to_string(),
|
||||
billing_surface: Some(
|
||||
codewhale_engine::pricing::FIRST_PARTY_PAYG_BILLING_SURFACE.to_string(),
|
||||
),
|
||||
endpoint_fingerprint: Some("test-moonshot-endpoint".to_string()),
|
||||
provider_live_pricing: None,
|
||||
billing_mode: codewhale_engine::cost_status::RouteBillingMode::Metered,
|
||||
dispatched_at: chrono::DateTime::<chrono::Utc>::from_timestamp(0, 0).expect("epoch"),
|
||||
};
|
||||
let mut unpriced_config = codewhale_runtime::config::Config::default();
|
||||
unpriced_config.custom_models = Some(
|
||||
serde_json::from_value(serde_json::json!([{
|
||||
"provider": "moonshot",
|
||||
"base_url": ProviderKind::Moonshot.provider().default_base_url(),
|
||||
"id": "kimi-k2.7-code",
|
||||
"cost": { "input": 1.0, "output": 2.0, "cache_read": 0.1 }
|
||||
}]))
|
||||
.expect("fixture rates without a cache-write price"),
|
||||
);
|
||||
let unpriced_route = codewhale_engine::cost_status::EffectiveRouteEnvelope::capture(
|
||||
Some(&unpriced_config),
|
||||
ProviderKind::Moonshot,
|
||||
ProviderKind::Moonshot.as_str(),
|
||||
"kimi-k2.7-code",
|
||||
Some(ProviderKind::Moonshot.provider().default_base_url()),
|
||||
chrono::Utc::now(),
|
||||
);
|
||||
assert!(unpriced_route.provider_live_pricing.is_some());
|
||||
let mut metadata = serde_json::json!({});
|
||||
codewhale_engine::cost_status::attach_child_usage_metadata(
|
||||
&mut metadata,
|
||||
&unpriced_route,
|
||||
&child_usage,
|
||||
);
|
||||
let record = settled_child_fixture_record(
|
||||
&mut unpriced_app,
|
||||
&fixture_manager,
|
||||
"unpriced-cache-class-response",
|
||||
&unpriced_route,
|
||||
&child_usage,
|
||||
);
|
||||
codewhale_engine::cost_status::attach_child_usage_batch_metadata(
|
||||
&mut metadata,
|
||||
&codewhale_engine::cost_status::RuntimeUsageBatch {
|
||||
records: vec![record],
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
let unpriced =
|
||||
Ok(codewhale_engine::tools::spec::ToolResult::success("ok").with_metadata(metadata));
|
||||
handle_tool_call_complete(&mut unpriced_app, "rlm-1", "rlm", &unpriced);
|
||||
@@ -23884,10 +23999,19 @@ fn child_usage_metadata_carries_cache_write_and_reasoning_end_to_end() {
|
||||
pathological.api_provider = codewhale_config::ProviderKind::Deepseek;
|
||||
pathological.billing_presentation =
|
||||
codewhale_engine::route_billing::BillingPresentation::Metered;
|
||||
let pathological_route = codewhale_engine::cost_status::EffectiveRouteEnvelope::capture(
|
||||
Some(&api_config),
|
||||
ProviderKind::Deepseek,
|
||||
ProviderKind::Deepseek.as_str(),
|
||||
"deepseek-v4-flash",
|
||||
Some(ProviderKind::Deepseek.provider().default_base_url()),
|
||||
chrono::Utc::now(),
|
||||
);
|
||||
assert!(pathological_route.provider_live_pricing.is_some());
|
||||
let mut metadata = serde_json::json!({});
|
||||
codewhale_engine::cost_status::attach_child_usage_metadata(
|
||||
&mut metadata,
|
||||
&test_mailbox_route(ProviderKind::Deepseek, "deepseek-v4-flash"),
|
||||
&pathological_route,
|
||||
&codewhale_models::Usage {
|
||||
input_tokens: 1_000,
|
||||
output_tokens: 100,
|
||||
@@ -23895,6 +24019,24 @@ fn child_usage_metadata_carries_cache_write_and_reasoning_end_to_end() {
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
let usage = codewhale_engine::cost_status::child_usage_from_metadata(&metadata)
|
||||
.expect("typed pathological usage");
|
||||
let route = codewhale_engine::cost_status::child_route_envelope_from_metadata(&metadata)
|
||||
.expect("typed pathological route");
|
||||
let record = settled_child_fixture_record(
|
||||
&mut pathological,
|
||||
&fixture_manager,
|
||||
"reasoning-subset-response",
|
||||
&route,
|
||||
&usage,
|
||||
);
|
||||
codewhale_engine::cost_status::attach_child_usage_batch_metadata(
|
||||
&mut metadata,
|
||||
&codewhale_engine::cost_status::RuntimeUsageBatch {
|
||||
records: vec![record],
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
let result =
|
||||
Ok(codewhale_engine::tools::spec::ToolResult::success("ok").with_metadata(metadata));
|
||||
handle_tool_call_complete(&mut pathological, "rlm-2", "rlm", &result);
|
||||
@@ -23919,6 +24061,10 @@ fn child_usage_metadata_carries_cache_write_and_reasoning_end_to_end() {
|
||||
#[test]
|
||||
fn zero_usage_model_child_still_records_priced_receipt() {
|
||||
let mut app = create_test_app();
|
||||
let fixture_dir = tempfile::tempdir().expect("fixture dir");
|
||||
let fixture_manager =
|
||||
codewhale_engine::session_manager::SessionManager::new(fixture_dir.path().join("sessions"))
|
||||
.expect("fixture manager");
|
||||
let route = test_mailbox_route(ProviderKind::Deepseek, "deepseek-v4-flash");
|
||||
let mut metadata = serde_json::json!({});
|
||||
codewhale_engine::cost_status::attach_child_usage_metadata(
|
||||
@@ -23926,6 +24072,20 @@ fn zero_usage_model_child_still_records_priced_receipt() {
|
||||
&route,
|
||||
&codewhale_models::Usage::default(),
|
||||
);
|
||||
let record = settled_child_fixture_record(
|
||||
&mut app,
|
||||
&fixture_manager,
|
||||
"explicit-zero-response",
|
||||
&route,
|
||||
&codewhale_models::Usage::default(),
|
||||
);
|
||||
codewhale_engine::cost_status::attach_child_usage_batch_metadata(
|
||||
&mut metadata,
|
||||
&codewhale_engine::cost_status::RuntimeUsageBatch {
|
||||
records: vec![record],
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
let result = Ok(
|
||||
codewhale_engine::tools::spec::ToolResult::error("kernel closed").with_metadata(metadata),
|
||||
);
|
||||
@@ -29985,19 +30145,24 @@ fn displayed_session_cost_is_monotonic_under_negative_reconciliation() {
|
||||
#[test]
|
||||
fn duplicate_mailbox_token_usage_does_not_regress_displayed_cost() {
|
||||
let mut app = create_test_app();
|
||||
let usage = codewhale_engine::tools::subagent::MailboxMessage::TokenUsage {
|
||||
agent_id: "agent-x".to_string(),
|
||||
source_id: "response-x".to_string(),
|
||||
route: Box::new(test_mailbox_route(
|
||||
ProviderKind::Deepseek,
|
||||
"deepseek-v4-flash",
|
||||
)),
|
||||
usage: codewhale_models::Usage {
|
||||
input_tokens: 10_000,
|
||||
output_tokens: 1_000,
|
||||
..Default::default()
|
||||
},
|
||||
let fixture_dir = tempfile::tempdir().expect("fixture dir");
|
||||
let fixture_manager =
|
||||
codewhale_engine::session_manager::SessionManager::new(fixture_dir.path().join("sessions"))
|
||||
.expect("fixture manager");
|
||||
let route = test_mailbox_route(ProviderKind::Deepseek, "deepseek-v4-flash");
|
||||
let source_usage = codewhale_models::Usage {
|
||||
input_tokens: 10_000,
|
||||
output_tokens: 1_000,
|
||||
..Default::default()
|
||||
};
|
||||
let record = settled_child_fixture_record(
|
||||
&mut app,
|
||||
&fixture_manager,
|
||||
"response-x",
|
||||
&route,
|
||||
&source_usage,
|
||||
);
|
||||
let usage = codewhale_engine::tools::subagent::MailboxMessage::usage_record("agent-x", record);
|
||||
handle_subagent_mailbox(&mut app, 11, &usage);
|
||||
let baseline = app.displayed_session_cost();
|
||||
assert!(baseline > 0.0);
|
||||
@@ -30011,13 +30176,14 @@ fn duplicate_mailbox_token_usage_does_not_regress_displayed_cost() {
|
||||
);
|
||||
|
||||
// A distinct provider response must extend the displayed cost upward.
|
||||
let mut distinct = usage.clone();
|
||||
let codewhale_engine::tools::subagent::MailboxMessage::TokenUsage { source_id, .. } =
|
||||
&mut distinct
|
||||
else {
|
||||
unreachable!()
|
||||
};
|
||||
*source_id = "response-y".to_string();
|
||||
let next = settled_child_fixture_record(
|
||||
&mut app,
|
||||
&fixture_manager,
|
||||
"independent-response",
|
||||
&route,
|
||||
&source_usage,
|
||||
);
|
||||
let distinct = codewhale_engine::tools::subagent::MailboxMessage::usage_record("agent-x", next);
|
||||
handle_subagent_mailbox(&mut app, 13, &distinct);
|
||||
assert!(app.displayed_session_cost() > baseline);
|
||||
}
|
||||
|
||||
@@ -305,7 +305,16 @@ async fn test_preset(app: &mut App, config: &Config, preset: RouterPreset) {
|
||||
let mut lines = vec![router_summary(router, locale)];
|
||||
// Captured before the call, like any background provider request.
|
||||
let cost_scope = codewhale_engine::cost_status::scope_token();
|
||||
match codewhale_engine::model_routing::test_auto_router(&candidate).await {
|
||||
let origin = app
|
||||
.session_write_receipt
|
||||
.as_ref()
|
||||
.and_then(|source| source.usage_origin(cost_scope).ok());
|
||||
if origin.is_none() {
|
||||
app.status_message =
|
||||
Some("Router test has no admitted session accounting origin".to_string());
|
||||
return;
|
||||
}
|
||||
match codewhale_engine::model_routing::test_auto_router(&candidate, origin).await {
|
||||
Ok((selection, latency_ms)) => {
|
||||
// The test call is real spend: settle it into session
|
||||
// cost exactly as a turn's routing call is settled.
|
||||
|
||||
@@ -1408,6 +1408,7 @@ mod tests {
|
||||
chrono::Utc::now(),
|
||||
);
|
||||
let usage = |source_id: &str, input_tokens, output_tokens| MailboxMessage::TokenUsage {
|
||||
record: None,
|
||||
agent_id: "agent_stream".to_string(),
|
||||
source_id: source_id.to_string(),
|
||||
route: Box::new(route.clone()),
|
||||
|
||||
Reference in new issue
Block a user