mirror of
https://github.com/codewhale-hq/Codewhale.git
synced 2026-11-09 01:49:49 -10:00
wip: complete Decisions compatibility and owned shadow settlement (#6604)
OpenRouter uses /api/alpha/decisions and TypeSafe /v1/systemone on the existing authenticated transport. Validate Choice, Noul and fractional Score against each request, retain provider usage on rejected policy answers, and bound request/response bodies. Preserve original contributor branch/authorship and off-by-default shadow behavior. Detached shadow calls capture the existing turn/session usage owner and lease before dispatch, obey cancellation, and settle usage or missing-coverage evidence through the same runtime ledger. Bounded raw-cost decision receipts persist in that ledger, including late responses, with deletion admission and turn_mutation intact. Existing child metadata and TurnRecord constructors/decoders are migrated without another event authority. Source checkpoint only: rustfmt and git diff --check passed. Focused Rust tests, npm test && npm run check:web, clippy, negative controls and fresh hosted exact-head Linux/macOS/Windows proof are pending. No Jev spend, native UI acceptance or release claim. Signed-off-by: CodeWhale Bot <[email protected]>
This commit is contained in:
1 parent
346708429c
commit
3185d9ceaf
16 files changed
+1382
-121
No files matched your search
@@ -2,9 +2,9 @@
|
||||
//!
|
||||
//! `[auto.router] kind = "decision"` asks a non-generative decision model
|
||||
//! (TypeSafe's Jev) one typed Choice per turn. The wire is a plain JSON
|
||||
//! `POST {base}/systemone` — not chat completions — served on two routes:
|
||||
//! `POST to the route’s decision endpoint` — not chat completions — served on two routes:
|
||||
//!
|
||||
//! * **OpenRouter** — `{openrouter base}/systemone` with the user's OpenRouter
|
||||
//! * **OpenRouter** — `https://openrouter.ai/api/alpha/decisions` with the user's OpenRouter
|
||||
//! key, built exactly like every other OpenRouter client.
|
||||
//! * **TypeSafe direct** — `https://api.typesafe.ai/v1/systemone` with a
|
||||
//! TypeSafe key.
|
||||
@@ -16,13 +16,15 @@
|
||||
//! * TypeSafe is **not** an [`ApiProvider`]: it serves no chat route, so it is
|
||||
//! the decision router's own endpoint + key (`TYPESAFE_API_KEY`, the
|
||||
//! `typesafe` secret-store slot, or `[providers.typesafe] api_key` /
|
||||
//! `api_key_env`). Its spend is shown on the Auto receipt as the
|
||||
//! provider-reported cost; it is not entered in session cost totals.
|
||||
//! `api_key_env`). Its tokens enter the shared usage ledger under a frozen
|
||||
//! `custom` / `typesafe` route with unknown billing; reported cost is retained
|
||||
//! on decision receipts. Unknown pricing is never interpreted as free.
|
||||
//! * The routing call makes one attempt (no retry): it is bounded by the
|
||||
//! router timeout, and a retried decision would arrive after the turn has
|
||||
//! already fallen back.
|
||||
//! * The router parses the `choice` answer shape; the shadow Decision Gate
|
||||
//! (`crate::superfast`) also reads `noul`. `score` is not parsed.
|
||||
//! (`crate::superfast`) also reads `noul`. The transport strictly validates
|
||||
//! Choice, Noul and fractional Score responses before either policy uses them.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
@@ -38,6 +40,9 @@ pub(crate) const TYPESAFE_DEFAULT_BASE_URL: &str = "https://api.typesafe.ai/v1";
|
||||
pub(crate) const TYPESAFE_API_KEY_ENV: &str = "TYPESAFE_API_KEY";
|
||||
/// Secret-store slot and `[providers.<name>]` table name for the TypeSafe key.
|
||||
pub(crate) const TYPESAFE_KEY_NAME: &str = "typesafe";
|
||||
/// Bound a non-generative decision response before allocating/decoding it.
|
||||
const DECISION_RESPONSE_MAX_BYTES: usize = 256 * 1024;
|
||||
const DECISION_REQUEST_MAX_BYTES: usize = 1024 * 1024;
|
||||
|
||||
/// Which endpoint serves a decision router.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
@@ -128,6 +133,10 @@ pub(crate) struct SystemOneResponse {
|
||||
pub(crate) answers: BTreeMap<String, SystemOneAnswer>,
|
||||
#[serde(default)]
|
||||
pub(crate) usage: Option<SystemOneUsage>,
|
||||
/// Transport validation is separate from decoding so a rejected policy
|
||||
/// answer still preserves the provider's usage/cost evidence.
|
||||
#[serde(skip)]
|
||||
pub(crate) answers_validated: Option<bool>,
|
||||
}
|
||||
|
||||
/// One answer. The `choice` and `noul` subsets are interpreted.
|
||||
@@ -141,6 +150,10 @@ pub(crate) struct SystemOneAnswer {
|
||||
#[serde(default)]
|
||||
pub(crate) noul: Option<f64>,
|
||||
#[serde(default)]
|
||||
pub(crate) score: Option<f64>,
|
||||
#[serde(default)]
|
||||
pub(crate) legend: BTreeMap<String, Value>,
|
||||
#[serde(default)]
|
||||
pub(crate) probabilities: BTreeMap<String, Option<f64>>,
|
||||
#[serde(default)]
|
||||
pub(crate) confidence: Option<f64>,
|
||||
@@ -166,7 +179,7 @@ impl SystemOneUsage {
|
||||
pub(crate) fn reported_cost(&self) -> Option<String> {
|
||||
let raw = self.cost.as_ref()?.get().trim();
|
||||
let value: f64 = raw.parse().ok()?;
|
||||
(value.is_finite() && value >= 0.0).then(|| raw.to_string())
|
||||
(raw.len() <= 128 && value.is_finite() && value >= 0.0).then(|| raw.to_string())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -176,8 +189,8 @@ impl CodewhaleClient {
|
||||
/// OpenRouter is the ordinary OpenRouter client (its key, base URL,
|
||||
/// attribution headers). TypeSafe re-points a clone of the active route's
|
||||
/// client — keeping its retry, TLS and redaction policy — at the TypeSafe
|
||||
/// endpoint with the TypeSafe key only, and drops the active provider's
|
||||
/// concurrency permit so a routing call never holds a chat slot.
|
||||
/// endpoint with the TypeSafe key only. Its captured request budget and
|
||||
/// remote-control ownership continue to apply to the one decision attempt.
|
||||
pub(crate) fn for_decision_route(
|
||||
config: &Config,
|
||||
route: DecisionRouterRoute,
|
||||
@@ -216,6 +229,15 @@ impl CodewhaleClient {
|
||||
)?
|
||||
.build()?;
|
||||
client.base_url = base_url;
|
||||
client.http1_client = client.http_client.clone();
|
||||
// Repointing auth/URL must also replace the inherited chat
|
||||
// route identity. No TypeSafe price/product is guessed.
|
||||
client.api_provider = ApiProvider::Custom;
|
||||
client.provider_identity = TYPESAFE_KEY_NAME.to_string();
|
||||
client.openrouter_vendor = None;
|
||||
client.billing_surface = None;
|
||||
client.billing_mode = crate::cost_status::RouteBillingMode::Unknown;
|
||||
client.route_limits = None;
|
||||
// `Self::new` froze the redaction set from the chat provider's
|
||||
// secrets; the TypeSafe key is none of them, so add it before
|
||||
// any decision body is built from untrusted context.
|
||||
@@ -223,15 +245,13 @@ impl CodewhaleClient {
|
||||
push_model_bound_secret(&mut secrets, Some(&key));
|
||||
client.model_bound_secret_values = Arc::new(secrets);
|
||||
client.api_key = key;
|
||||
client.request_concurrency = None;
|
||||
client.remote_control_inference_participant = false;
|
||||
Ok(client)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `POST {base}/systemone` once, isolated like the Auto chat classifier:
|
||||
/// no global retry banners, no shared token bucket, no response cache.
|
||||
/// `POST to the route’s decision endpoint` once, isolated like the Auto chat classifier:
|
||||
/// no global retry banners or response cache; shared admission still applies.
|
||||
///
|
||||
/// Only a failure class leaves this function — provider error bodies can
|
||||
/// echo the prompt and must never reach receipts.
|
||||
@@ -244,26 +264,147 @@ impl CodewhaleClient {
|
||||
body: &Value,
|
||||
dispatched: &std::sync::atomic::AtomicBool,
|
||||
) -> std::result::Result<SystemOneResponse, AutoRouterFailure> {
|
||||
if serde_json::to_vec(body).map_or(true, |bytes| bytes.len() > DECISION_REQUEST_MAX_BYTES)
|
||||
|| !valid_decision_request(body)
|
||||
{
|
||||
return Err(AutoRouterFailure::NotRunnable);
|
||||
}
|
||||
let mut isolated = self.clone();
|
||||
isolated.isolated_request_state = true;
|
||||
isolated.rate_limiter = Arc::new(AsyncMutex::new(TokenBucket::from_env()));
|
||||
isolated.retry.max_retries = 0;
|
||||
let _inference = isolated.acquire_remote_control_inference_permit().await;
|
||||
let _permit = isolated.acquire_provider_request_permit().await;
|
||||
let url = api_url(&isolated.base_url, "systemone");
|
||||
let url = if isolated.api_provider == ApiProvider::Openrouter {
|
||||
// The OpenRouter Decisions API is a sibling of /api/v1, so keep
|
||||
// the configured origin/proxy prefix and replace only /v1.
|
||||
let mut url = reqwest::Url::parse(&isolated.base_url)
|
||||
.map_err(|_| AutoRouterFailure::NotRunnable)?;
|
||||
let prefix = url.path().trim_end_matches('/').trim_end_matches("/v1");
|
||||
url.set_path(&format!("{prefix}/alpha/decisions"));
|
||||
url.to_string()
|
||||
} else {
|
||||
api_url(&isolated.base_url, "systemone")
|
||||
};
|
||||
isolated.wait_for_rate_limit().await;
|
||||
dispatched.store(true, std::sync::atomic::Ordering::Release);
|
||||
let response = isolated
|
||||
.send_json_with_retry(&url, body)
|
||||
.await
|
||||
.map_err(|error| router_failure_from_error(&error))?;
|
||||
let bytes = response
|
||||
.bytes()
|
||||
let text = bounded_provider_catalog_text(response, DECISION_RESPONSE_MAX_BYTES)
|
||||
.await
|
||||
.map_err(|_| AutoRouterFailure::Transport)?;
|
||||
serde_json::from_slice(&bytes).map_err(|_| AutoRouterFailure::InvalidAnswer)
|
||||
.map_err(|error| match error {
|
||||
CatalogRefreshError::Network => AutoRouterFailure::Transport,
|
||||
_ => AutoRouterFailure::InvalidAnswer,
|
||||
})?;
|
||||
let mut response: SystemOneResponse =
|
||||
serde_json::from_str(&text).map_err(|_| AutoRouterFailure::InvalidAnswer)?;
|
||||
response.answers_validated = Some(valid_decision_response(body, &response));
|
||||
response.model = response.model.map(|model| {
|
||||
self.redact_model_bound_text(&model)
|
||||
.chars()
|
||||
.take(128)
|
||||
.collect()
|
||||
});
|
||||
Ok(response)
|
||||
}
|
||||
}
|
||||
|
||||
fn valid_decision_request(body: &Value) -> bool {
|
||||
let Some(questions) = body.get("questions").and_then(Value::as_object) else {
|
||||
return false;
|
||||
};
|
||||
let model = body.get("model").and_then(Value::as_str);
|
||||
model.is_some_and(|m| !m.trim().is_empty() && m.len() <= 256)
|
||||
&& body.get("state").is_some_and(|state| {
|
||||
matches!(state, Value::String(_) | Value::Object(_) | Value::Array(_))
|
||||
})
|
||||
&& !questions.is_empty()
|
||||
&& questions.len() <= 64
|
||||
&& questions
|
||||
.values()
|
||||
.all(|q| match q.get("type").and_then(Value::as_str) {
|
||||
Some("noul") => true,
|
||||
Some("choice") => q
|
||||
.get("criteria")
|
||||
.and_then(Value::as_object)
|
||||
.is_some_and(|c| !c.is_empty() && c.len() <= 64),
|
||||
Some("score") => q
|
||||
.get("criteria")
|
||||
.and_then(Value::as_array)
|
||||
.is_some_and(|c| !c.is_empty() && c.len() <= 10),
|
||||
_ => false,
|
||||
})
|
||||
}
|
||||
|
||||
fn valid_decision_response(body: &Value, response: &SystemOneResponse) -> bool {
|
||||
let Some(questions) = body.get("questions").and_then(Value::as_object) else {
|
||||
return false;
|
||||
};
|
||||
response
|
||||
.model
|
||||
.as_deref()
|
||||
.is_some_and(|m| !m.trim().is_empty() && m.len() <= 256)
|
||||
&& response.usage.is_some()
|
||||
&& response.answers.len() == questions.len()
|
||||
&& questions.iter().all(|(name, question)| {
|
||||
let Some(answer) = response.answers.get(name) else {
|
||||
return false;
|
||||
};
|
||||
if Some(answer.kind.as_str()) != question.get("type").and_then(Value::as_str) {
|
||||
return false;
|
||||
}
|
||||
match answer.kind.as_str() {
|
||||
"noul" => answer
|
||||
.noul
|
||||
.is_some_and(|v| v.is_finite() && (0.0..=1.0).contains(&v)),
|
||||
"choice" => {
|
||||
let Some(criteria) = question.get("criteria").and_then(Value::as_object) else {
|
||||
return false;
|
||||
};
|
||||
let options = criteria.keys().map(String::as_str).collect::<Vec<_>>();
|
||||
crate::model_routing::validated_choice(Some(answer), &options).is_some()
|
||||
}
|
||||
"score" => {
|
||||
let Some(criteria) = question.get("criteria").and_then(Value::as_array) else {
|
||||
return false;
|
||||
};
|
||||
let Some(score) = answer.score else {
|
||||
return false;
|
||||
};
|
||||
if !score.is_finite()
|
||||
|| !(0.0..=(criteria.len() - 1) as f64).contains(&score)
|
||||
|| !answer
|
||||
.confidence
|
||||
.is_some_and(|v| v.is_finite() && (0.0..=1.0).contains(&v))
|
||||
|| answer.legend.len() != criteria.len()
|
||||
|| answer.probabilities.len() != criteria.len()
|
||||
{
|
||||
return false;
|
||||
}
|
||||
let mut sum = 0.0;
|
||||
let mut expected_score = 0.0;
|
||||
for (level, criterion) in criteria.iter().enumerate() {
|
||||
let key = level.to_string();
|
||||
let Some(Some(probability)) = answer.probabilities.get(&key) else {
|
||||
return false;
|
||||
};
|
||||
if answer.legend.get(&key) != Some(criterion)
|
||||
|| !probability.is_finite()
|
||||
|| !(0.0..=1.0).contains(probability)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
sum += probability;
|
||||
expected_score += level as f64 * probability;
|
||||
}
|
||||
(sum - 1.0).abs() <= 0.02 && (score - expected_score).abs() <= 0.02
|
||||
}
|
||||
_ => false,
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
/// Collapse a client error into a non-secret failure class. The HTTP status
|
||||
/// is kept where the error carries it; the body never is.
|
||||
pub(crate) fn router_failure_from_error(error: &anyhow::Error) -> AutoRouterFailure {
|
||||
@@ -293,3 +434,88 @@ pub(crate) fn router_failure_from_error(error: &anyhow::Error) -> AutoRouterFail
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod decisions_compatibility_tests {
|
||||
use super::*;
|
||||
|
||||
fn request() -> Value {
|
||||
json!({"model":"typesafe/jev-1.13", "state":{"ticket":"charged twice"}, "questions": {
|
||||
"intent": {"type":"choice", "criteria":{"billing":"money", "other":"anything else"}},
|
||||
"refund": {"type":"noul", "instructions":"Does it ask for a refund?"},
|
||||
"urgency": {"type":"score", "criteria":["Can wait", "Needs attention this week", "Needs attention today"]}
|
||||
}})
|
||||
}
|
||||
|
||||
fn response() -> Value {
|
||||
json!({"id":"fixture-decision", "model":"typesafe/jev-1.13-20260917", "usage":{"input_tokens":287,"output_tokens":20,"cost":0.000012054}, "answers": {
|
||||
"intent":{"type":"choice","choice":"billing","confidence":0.9,"probabilities":{"billing":0.9,"other":0.1}},
|
||||
"refund":{"type":"noul","noul":0.99},
|
||||
"urgency":{"type":"score","score":1.7,"confidence":0.9,"legend":{"0":"Can wait","1":"Needs attention this week","2":"Needs attention today"},"probabilities":{"0":0.1,"1":0.1,"2":0.8}}
|
||||
}})
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn all_documented_primitives_validate_without_rescaling_score() {
|
||||
let decoded: SystemOneResponse =
|
||||
serde_json::from_value(response()).expect("documented fixture");
|
||||
assert!(valid_decision_request(&request()));
|
||||
assert!(valid_decision_response(&request(), &decoded));
|
||||
assert_eq!(decoded.answers["urgency"].score, Some(1.7));
|
||||
assert_eq!(
|
||||
decoded.usage.expect("usage").reported_cost().as_deref(),
|
||||
Some("0.000012054")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wrong_types_unoffered_choices_scores_and_partial_shapes_are_rejected() {
|
||||
let changes = [
|
||||
("/answers/refund/noul", json!(1.01)),
|
||||
("/answers/refund/type", json!("score")),
|
||||
("/answers/intent/choice", json!("not-offered")),
|
||||
("/answers/intent/choice", json!("other")),
|
||||
("/answers/intent/confidence", json!(-0.1)),
|
||||
("/answers/intent/probabilities", json!({"billing":0.9})),
|
||||
(
|
||||
"/answers/intent/probabilities",
|
||||
json!({"billing":0.6,"other":0.6}),
|
||||
),
|
||||
("/answers/urgency/score", json!(2.01)),
|
||||
("/answers/urgency/score", json!(0.5)),
|
||||
("/answers/urgency/legend/1", json!("different rubric")),
|
||||
("/answers/urgency/probabilities/1", Value::Null),
|
||||
("/answers/urgency/confidence", json!(1.1)),
|
||||
("/model", Value::Null),
|
||||
("/usage", Value::Null),
|
||||
("/answers", json!({})),
|
||||
];
|
||||
for (pointer, value) in changes {
|
||||
let mut body = response();
|
||||
*body.pointer_mut(pointer).expect("fixture field") = value;
|
||||
let decoded: SystemOneResponse =
|
||||
serde_json::from_value(body).expect("structural decode");
|
||||
assert!(
|
||||
!valid_decision_response(&request(), &decoded),
|
||||
"accepted {pointer}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn non_finite_in_memory_answers_and_unbounded_request_shapes_are_rejected() {
|
||||
let mut decoded: SystemOneResponse = serde_json::from_value(response()).expect("fixture");
|
||||
decoded.answers.get_mut("refund").expect("noul").noul = Some(f64::NAN);
|
||||
assert!(!valid_decision_response(&request(), &decoded));
|
||||
let mut decoded: SystemOneResponse = serde_json::from_value(response()).expect("fixture");
|
||||
decoded.answers.get_mut("urgency").expect("score").score = Some(f64::INFINITY);
|
||||
assert!(!valid_decision_response(&request(), &decoded));
|
||||
for shape in [
|
||||
json!({}),
|
||||
json!({"model":"jev", "state":"hello", "questions":{}}),
|
||||
json!({"model":"jev", "state":"hello", "questions":{"q":{"type":"score","criteria":vec!["level";11]}}}),
|
||||
] {
|
||||
assert!(!valid_decision_request(&shape));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -5562,6 +5562,14 @@ impl Engine {
|
||||
);
|
||||
turn.add_routed_usage_dropped_records(residual_dropped_records);
|
||||
let initial_cost_scope = crate::cost_status::scope_token();
|
||||
crate::cost_status::report_runtime_usage_batch(
|
||||
initial_cost_scope,
|
||||
initial_usage_owner.as_deref(),
|
||||
&crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: initial_routed_usage.decisions.clone(),
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
for record in &initial_routed_usage.records {
|
||||
crate::cost_status::report_effective_route_for_runtime(
|
||||
initial_cost_scope,
|
||||
|
||||
@@ -8520,6 +8520,7 @@ async fn initial_routed_usage_is_total_only_emitted_once_and_keeps_parent_route_
|
||||
);
|
||||
missing_usage_route.billing_mode = crate::cost_status::RouteBillingMode::Metered;
|
||||
**initial_routed_usage = crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: vec![crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: "auto-router:engine-fixture".to_string(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
|
||||
@@ -1825,6 +1825,8 @@ impl Engine {
|
||||
drop(crate::superfast::spawn_shadow_gate(
|
||||
&self.api_config,
|
||||
&stream_request.messages,
|
||||
self.config.compaction.runtime_cost_owner.as_deref(),
|
||||
&self.cancel_token,
|
||||
));
|
||||
}
|
||||
let _ = self
|
||||
|
||||
@@ -488,6 +488,7 @@ pub fn attach_child_usage_metadata(
|
||||
pub const MAX_CHILD_USAGE_RECORDS: usize = 64;
|
||||
|
||||
const CHILD_USAGE_RECORDS_KEY: &str = "child_usage_records";
|
||||
const CHILD_DECISION_RECEIPTS_KEY: &str = "child_decision_receipts";
|
||||
const CHILD_USAGE_DROP_RECORDS_KEY: &str = "child_usage_drop_records";
|
||||
const CHILD_USAGE_DROPPED_RECORDS_KEY: &str = "child_usage_dropped_records";
|
||||
|
||||
@@ -506,6 +507,18 @@ pub fn attach_child_usage_batch_metadata(
|
||||
let Some(object) = metadata.as_object_mut() else {
|
||||
return;
|
||||
};
|
||||
object.insert(
|
||||
CHILD_DECISION_RECEIPTS_KEY.into(),
|
||||
serde_json::json!(
|
||||
batch
|
||||
.decisions
|
||||
.iter()
|
||||
.filter(|receipt| receipt.is_bounded())
|
||||
.take(MAX_CHILD_USAGE_RECORDS)
|
||||
.map(RuntimeDecisionReceipt::sanitized)
|
||||
.collect::<Vec<_>>()
|
||||
),
|
||||
);
|
||||
let retained_records = batch
|
||||
.records
|
||||
.iter()
|
||||
@@ -545,10 +558,23 @@ pub fn attach_child_usage_batch_metadata(
|
||||
serde_json::json!(retained_drops),
|
||||
);
|
||||
let usage_overflow = batch.records.len().saturating_sub(MAX_CHILD_USAGE_RECORDS);
|
||||
let decision_overflow = batch
|
||||
.decisions
|
||||
.len()
|
||||
.saturating_sub(MAX_CHILD_USAGE_RECORDS)
|
||||
.saturating_add(
|
||||
batch
|
||||
.decisions
|
||||
.iter()
|
||||
.take(MAX_CHILD_USAGE_RECORDS)
|
||||
.filter(|r| !r.is_bounded())
|
||||
.count(),
|
||||
);
|
||||
let dropped_records = batch
|
||||
.dropped_records
|
||||
.max(u64::try_from(batch.drop_records.len()).unwrap_or(u64::MAX))
|
||||
.saturating_add(u64::try_from(usage_overflow).unwrap_or(u64::MAX));
|
||||
.saturating_add(u64::try_from(usage_overflow).unwrap_or(u64::MAX))
|
||||
.saturating_add(u64::try_from(decision_overflow).unwrap_or(u64::MAX));
|
||||
if dropped_records > 0 {
|
||||
object.insert(
|
||||
CHILD_USAGE_DROPPED_RECORDS_KEY.into(),
|
||||
@@ -581,6 +607,7 @@ pub fn child_usage_records_from_metadata(
|
||||
.unwrap_or(0);
|
||||
let Some(values) = value.as_array() else {
|
||||
return Some(RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: Vec::new(),
|
||||
drop_records: Vec::new(),
|
||||
dropped_records: declared_dropped.saturating_add(1),
|
||||
@@ -589,6 +616,7 @@ pub fn child_usage_records_from_metadata(
|
||||
|
||||
let overflow = values.len().saturating_sub(MAX_CHILD_USAGE_RECORDS);
|
||||
let mut batch = RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: Vec::with_capacity(values.len().min(MAX_CHILD_USAGE_RECORDS)),
|
||||
drop_records: Vec::with_capacity(drop_values.len().min(MAX_CHILD_USAGE_RECORDS)),
|
||||
dropped_records: declared_dropped
|
||||
@@ -636,6 +664,27 @@ pub fn child_usage_records_from_metadata(
|
||||
// Every declared drop slot already contributes to dropped_records,
|
||||
// including malformed entries; do not count the same gap twice.
|
||||
}
|
||||
if let Some(decisions) = metadata.get(CHILD_DECISION_RECEIPTS_KEY) {
|
||||
if let Some(values) = decisions.as_array() {
|
||||
for value in values.iter().take(MAX_CHILD_USAGE_RECORDS) {
|
||||
let bounded = serde_json::to_vec(value).is_ok_and(|bytes| bytes.len() <= 16 * 1024);
|
||||
let parsed = bounded
|
||||
.then(|| serde_json::from_value::<RuntimeDecisionReceipt>(value.clone()).ok())
|
||||
.flatten();
|
||||
if let Some(receipt) = parsed.filter(RuntimeDecisionReceipt::is_bounded) {
|
||||
batch.decisions.push(receipt.sanitized());
|
||||
} else {
|
||||
batch.dropped_records = batch.dropped_records.saturating_add(1);
|
||||
}
|
||||
}
|
||||
batch.dropped_records = batch.dropped_records.saturating_add(
|
||||
u64::try_from(values.len().saturating_sub(MAX_CHILD_USAGE_RECORDS))
|
||||
.unwrap_or(u64::MAX),
|
||||
);
|
||||
} else {
|
||||
batch.dropped_records = batch.dropped_records.saturating_add(1);
|
||||
}
|
||||
}
|
||||
Some(batch)
|
||||
}
|
||||
|
||||
@@ -794,7 +843,7 @@ fn with_pending_state_mut<R>(f: impl FnOnce(&mut ScopedPendingBackgroundCost) ->
|
||||
.get_or_init(|| Mutex::new(std::collections::HashMap::new()))
|
||||
.lock()
|
||||
.unwrap_or_else(|e| e.into_inner());
|
||||
f(by_thread.entry(std::thread::current().id()).or_default())
|
||||
f(by_thread.entry(test_cost_scope_id()).or_default())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -807,6 +856,7 @@ const MAX_RUNTIME_USAGE_RECORDS_PER_OWNER: usize = 64;
|
||||
#[derive(Default)]
|
||||
struct OwnerRuntimeUsageJournal {
|
||||
records: VecDeque<RuntimeUsageRecord>,
|
||||
decisions: VecDeque<RuntimeDecisionReceipt>,
|
||||
drop_records: VecDeque<RuntimeUsageDropRecord>,
|
||||
dropped_records: u64,
|
||||
dropped_source_fingerprints: HashSet<String>,
|
||||
@@ -820,6 +870,8 @@ type RuntimeUsageJournal = HashMap<String, OwnerRuntimeUsageJournal>;
|
||||
/// closed instead of silently presenting a partial cost as complete.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Eq)]
|
||||
pub struct RuntimeUsageBatch {
|
||||
/// Bounded decision evidence, owned and retired with the same usage ledger.
|
||||
pub decisions: Vec<RuntimeDecisionReceipt>,
|
||||
pub records: Vec<RuntimeUsageRecord>,
|
||||
/// Exact provider-success calls whose usage payload was absent. The
|
||||
/// bounded records retain route billing truth; `dropped_records` remains
|
||||
@@ -846,16 +898,118 @@ pub struct RuntimeUsageDropRecord {
|
||||
pub route: EffectiveRouteEnvelope,
|
||||
}
|
||||
|
||||
/// Provider decision evidence attached to the existing origin-turn ledger.
|
||||
/// It is diagnostic evidence, never an instruction to change the model route.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct RuntimeDecisionReceipt {
|
||||
pub source_id: String,
|
||||
pub route: EffectiveRouteEnvelope,
|
||||
pub usage: Option<Usage>,
|
||||
pub shadow: bool,
|
||||
pub valid_answers: bool,
|
||||
pub evidence: crate::model_routing::AutoRouteDecisionEvidence,
|
||||
}
|
||||
|
||||
impl RuntimeDecisionReceipt {
|
||||
pub(crate) fn is_bounded(&self) -> bool {
|
||||
self.source_id.len() <= 128
|
||||
&& self.evidence.choice.len() <= 128
|
||||
&& self.evidence.probabilities_bp.len() <= 64
|
||||
&& self
|
||||
.evidence
|
||||
.probabilities_bp
|
||||
.iter()
|
||||
.all(|(key, value)| key.len() <= 128 && *value <= 10_000)
|
||||
&& self.evidence.confidence_bp <= 10_000
|
||||
&& self.evidence.min_confidence_bp <= 10_000
|
||||
&& self
|
||||
.evidence
|
||||
.thinking
|
||||
.as_ref()
|
||||
.is_none_or(|v| v.len() <= 128)
|
||||
&& self
|
||||
.evidence
|
||||
.response_model
|
||||
.as_ref()
|
||||
.is_none_or(|v| v.len() <= 128)
|
||||
&& self
|
||||
.evidence
|
||||
.provider_reported_cost_usd
|
||||
.as_ref()
|
||||
.is_none_or(|v| {
|
||||
v.len() <= 128
|
||||
&& v.parse::<f64>()
|
||||
.is_ok_and(|cost| cost.is_finite() && cost >= 0.0)
|
||||
})
|
||||
&& serde_json::to_vec(self).is_ok_and(|bytes| bytes.len() <= 16 * 1024)
|
||||
}
|
||||
|
||||
pub(crate) fn sanitized(&self) -> Self {
|
||||
Self {
|
||||
source_id: usage_source_fingerprint(&self.source_id),
|
||||
route: self.route.sanitized_for_persistence(),
|
||||
..self.clone()
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn diagnostic_receipt(&self) -> String {
|
||||
// Reuse cost_status's persisted diagnostic receipt surface. No prompt,
|
||||
// response id, URL, auth header or unoffered choice enters this string.
|
||||
format!(
|
||||
"decision:{}",
|
||||
serde_json::to_string(&self.sanitized()).unwrap_or_default()
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) type RuntimeDecisionSink = Arc<dyn Fn(RuntimeDecisionReceipt) -> bool + Send + Sync>;
|
||||
|
||||
pub(crate) type RuntimeUsageSink = Arc<dyn Fn(RuntimeUsageRecord) -> bool + Send + Sync>;
|
||||
pub(crate) type RuntimeUsageDropSink = Arc<dyn Fn(RuntimeUsageDropRecord) -> bool + Send + Sync>;
|
||||
|
||||
struct RuntimeUsageSinkEntry {
|
||||
sink: RuntimeUsageSink,
|
||||
dropped_sink: Option<RuntimeUsageDropSink>,
|
||||
decision_sink: Option<RuntimeDecisionSink>,
|
||||
leases: usize,
|
||||
terminal: bool,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn decision_receipt_fixture(source_id: &str) -> RuntimeDecisionReceipt {
|
||||
let mut route = EffectiveRouteEnvelope::capture(
|
||||
None,
|
||||
crate::config::ApiProvider::Custom,
|
||||
"typesafe",
|
||||
"jev-latest",
|
||||
Some("https://api.typesafe.ai/v1"),
|
||||
Utc::now(),
|
||||
);
|
||||
route.billing_mode = RouteBillingMode::Unknown;
|
||||
RuntimeDecisionReceipt {
|
||||
source_id: source_id.to_string(),
|
||||
route,
|
||||
usage: Some(Usage {
|
||||
input_tokens: 9,
|
||||
output_tokens: 4,
|
||||
..Default::default()
|
||||
}),
|
||||
shadow: true,
|
||||
valid_answers: false,
|
||||
evidence: crate::model_routing::AutoRouteDecisionEvidence {
|
||||
choice: "invalid".to_string(),
|
||||
probabilities_bp: Default::default(),
|
||||
confidence_bp: 0,
|
||||
min_confidence_bp: 5_000,
|
||||
cost_saving_kept_fast: false,
|
||||
thinking: None,
|
||||
provider_reported_cost_usd: Some("0.000012054".to_string()),
|
||||
latency_ms: 12,
|
||||
response_model: Some("jev-latest".to_string()),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/// Keeps an owner sink alive while a detached child can still report usage.
|
||||
/// The runtime turn may already be terminal; the last child release retires
|
||||
/// the sink only after its final provider response has been durably appended.
|
||||
@@ -908,7 +1062,7 @@ fn with_runtime_usage_sinks<R>(
|
||||
.get_or_init(|| Mutex::new(HashMap::new()))
|
||||
.lock()
|
||||
.unwrap_or_else(|error| error.into_inner());
|
||||
f(by_thread.entry(std::thread::current().id()).or_default())
|
||||
f(by_thread.entry(test_cost_scope_id()).or_default())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -928,7 +1082,7 @@ fn with_existing_runtime_usage_sinks<R>(
|
||||
{
|
||||
let by_thread = TEST_RUNTIME_USAGE_SINKS.get()?;
|
||||
let mut by_thread = by_thread.lock().unwrap_or_else(|error| error.into_inner());
|
||||
let sinks = by_thread.get_mut(&std::thread::current().id())?;
|
||||
let sinks = by_thread.get_mut(&test_cost_scope_id())?;
|
||||
Some(f(sinks))
|
||||
}
|
||||
}
|
||||
@@ -948,7 +1102,7 @@ fn with_runtime_usage_journal_mut<R>(f: impl FnOnce(&mut RuntimeUsageJournal) ->
|
||||
.get_or_init(|| Mutex::new(std::collections::HashMap::new()))
|
||||
.lock()
|
||||
.unwrap_or_else(|error| error.into_inner());
|
||||
f(by_thread.entry(std::thread::current().id()).or_default())
|
||||
f(by_thread.entry(test_cost_scope_id()).or_default())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1079,6 +1233,7 @@ pub(crate) fn register_runtime_usage_sink_with_drop(
|
||||
RuntimeUsageSinkEntry {
|
||||
sink,
|
||||
dropped_sink,
|
||||
decision_sink: None,
|
||||
leases: 0,
|
||||
terminal: false,
|
||||
},
|
||||
@@ -1086,6 +1241,74 @@ pub(crate) fn register_runtime_usage_sink_with_drop(
|
||||
});
|
||||
}
|
||||
|
||||
/// Extend the existing owner entry; this is not another sink registry.
|
||||
pub(crate) fn register_runtime_decision_sink(owner: &str, sink: RuntimeDecisionSink) {
|
||||
with_runtime_usage_sinks(|sinks| {
|
||||
if let Some(entry) = sinks.get_mut(owner.trim()) {
|
||||
entry.decision_sink = Some(sink);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
fn record_runtime_decision(owner: &str, record: &RuntimeDecisionReceipt) {
|
||||
if !record.is_bounded() {
|
||||
record_runtime_usage_drop(owner, &record.source_id, &record.route);
|
||||
return;
|
||||
}
|
||||
let record = record.sanitized();
|
||||
let sink = with_runtime_usage_sinks(|sinks| {
|
||||
sinks
|
||||
.get(owner)
|
||||
.and_then(|entry| entry.decision_sink.clone())
|
||||
});
|
||||
if sink.is_some_and(|sink| sink(record.clone())) {
|
||||
return;
|
||||
}
|
||||
with_runtime_usage_journal_mut(|journal| {
|
||||
let entry = journal.entry(owner.to_string()).or_default();
|
||||
if entry
|
||||
.decisions
|
||||
.iter()
|
||||
.any(|old| old.source_id == record.source_id)
|
||||
{
|
||||
return;
|
||||
}
|
||||
if entry.decisions.len() == MAX_RUNTIME_USAGE_RECORDS_PER_OWNER {
|
||||
entry.decisions.pop_front();
|
||||
entry.dropped_records = entry.dropped_records.saturating_add(1);
|
||||
}
|
||||
entry.decisions.push_back(record);
|
||||
});
|
||||
}
|
||||
|
||||
fn report_interactive_decision(scope: CostScopeToken, receipt: &RuntimeDecisionReceipt) {
|
||||
if !receipt.is_bounded() {
|
||||
return;
|
||||
}
|
||||
with_pending_state_mut(|state| {
|
||||
if state.generation == scope.0 {
|
||||
if state
|
||||
.pending
|
||||
.route_receipts
|
||||
.iter()
|
||||
.filter(|v| v.starts_with("decision:"))
|
||||
.count()
|
||||
< MAX_RUNTIME_USAGE_RECORDS_PER_OWNER
|
||||
{
|
||||
state
|
||||
.pending
|
||||
.route_receipts
|
||||
.insert(receipt.diagnostic_receipt());
|
||||
} else {
|
||||
state
|
||||
.pending
|
||||
.route_receipts
|
||||
.insert("decision:diagnostic_receipt_bound_reached".to_string());
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Redacted durable identity shared by runtime-turn, worker, and interactive
|
||||
/// session accounting. Raw response ids never need to be persisted merely to
|
||||
/// make replay idempotent.
|
||||
@@ -1151,6 +1374,9 @@ fn register_persistent_interactive_runtime_usage_sink_at(
|
||||
let drop_session_id = usage_session_id.clone();
|
||||
let drop_turn_id = usage_turn_id.clone();
|
||||
let usage_sessions_dir = sessions_dir.clone();
|
||||
let decision_sessions_dir = sessions_dir.clone();
|
||||
let decision_session_id = session_id.to_string();
|
||||
let decision_turn_id = turn_id.to_string();
|
||||
register_runtime_usage_sink_with_drop(
|
||||
owner,
|
||||
Arc::new(move |record| {
|
||||
@@ -1183,6 +1409,28 @@ fn register_persistent_interactive_runtime_usage_sink_at(
|
||||
.unwrap_or(false)
|
||||
})),
|
||||
);
|
||||
register_runtime_decision_sink(
|
||||
owner,
|
||||
Arc::new(move |receipt| {
|
||||
let Ok(manager) =
|
||||
crate::session_manager::SessionManager::new(decision_sessions_dir.clone())
|
||||
else {
|
||||
return false;
|
||||
};
|
||||
let accepted = manager
|
||||
.persist_late_decision_receipt(&decision_session_id, &decision_turn_id, &receipt)
|
||||
.unwrap_or(false);
|
||||
if accepted {
|
||||
// The append may have been handled by a deletion tombstone. Only
|
||||
// a live origin may enter the foreground diagnostic projection.
|
||||
let _ = manager.with_live_session_origin(&decision_session_id, || {
|
||||
report_interactive_decision(scope, &receipt);
|
||||
true
|
||||
});
|
||||
}
|
||||
accepted
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -1288,6 +1536,7 @@ pub fn take_runtime_usage(owner: &str) -> RuntimeUsageBatch {
|
||||
journal
|
||||
.remove(owner)
|
||||
.map_or_else(RuntimeUsageBatch::default, |entry| RuntimeUsageBatch {
|
||||
decisions: entry.decisions.into_iter().collect(),
|
||||
records: entry.records.into_iter().collect(),
|
||||
drop_records: entry.drop_records.into_iter().collect(),
|
||||
dropped_records: entry.dropped_records,
|
||||
@@ -1295,6 +1544,33 @@ pub fn take_runtime_usage(owner: &str) -> RuntimeUsageBatch {
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
thread_local! {
|
||||
static TEST_COST_ORIGIN: std::cell::Cell<Option<std::thread::ThreadId>> = const { std::cell::Cell::new(None) };
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn test_cost_scope_id() -> std::thread::ThreadId {
|
||||
TEST_COST_ORIGIN
|
||||
.with(|scope| scope.get())
|
||||
.unwrap_or_else(|| std::thread::current().id())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) struct TestCostScopeBinding(Option<std::thread::ThreadId>);
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn bind_test_cost_scope(origin: std::thread::ThreadId) -> TestCostScopeBinding {
|
||||
TestCostScopeBinding(TEST_COST_ORIGIN.with(|scope| scope.replace(Some(origin))))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
impl Drop for TestCostScopeBinding {
|
||||
fn drop(&mut self) {
|
||||
TEST_COST_ORIGIN.with(|scope| scope.set(self.0));
|
||||
}
|
||||
}
|
||||
|
||||
/// Capture the current session/run generation before starting a background
|
||||
/// provider request. The same token must be supplied when its usage returns.
|
||||
#[must_use]
|
||||
@@ -1845,6 +2121,13 @@ pub(crate) fn report_runtime_usage_batch(
|
||||
runtime_owner: Option<&str>,
|
||||
batch: &RuntimeUsageBatch,
|
||||
) {
|
||||
for receipt in &batch.decisions {
|
||||
if let Some(owner) = runtime_owner {
|
||||
record_runtime_decision(owner, receipt);
|
||||
} else {
|
||||
report_interactive_decision(scope, receipt);
|
||||
}
|
||||
}
|
||||
for record in &batch.records {
|
||||
report_effective_route_for_runtime(
|
||||
scope,
|
||||
@@ -3189,6 +3472,7 @@ mod tests {
|
||||
attach_child_usage_batch_metadata(
|
||||
&mut metadata,
|
||||
&RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records,
|
||||
drop_records: Vec::new(),
|
||||
dropped_records: 0,
|
||||
|
||||
+122
-58
@@ -443,7 +443,7 @@ pub(crate) struct AutoRouteReceipt {
|
||||
/// What a decision model answered and what it cost. Probabilities are basis
|
||||
/// points (0..=10000) so the receipt stays `Eq` and never re-renders floats.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub(crate) struct AutoRouteDecisionEvidence {
|
||||
pub struct AutoRouteDecisionEvidence {
|
||||
/// The tier the decision model chose (`fast` | `strong`).
|
||||
pub(crate) choice: String,
|
||||
pub(crate) probabilities_bp: BTreeMap<String, u16>,
|
||||
@@ -473,6 +473,7 @@ pub(crate) enum AutoRouterFailure {
|
||||
/// Declared but unusable: missing key, unknown kind, or client setup.
|
||||
NotRunnable,
|
||||
Timeout,
|
||||
Cancelled,
|
||||
Http {
|
||||
status: u16,
|
||||
},
|
||||
@@ -492,6 +493,7 @@ impl AutoRouterFailure {
|
||||
"not runnable (check the router key and [auto.router])".to_string()
|
||||
}
|
||||
Self::Timeout => "timed out".to_string(),
|
||||
Self::Cancelled => "cancelled".to_string(),
|
||||
Self::Http { status } => format!("HTTP {status}"),
|
||||
Self::QuotaExhausted => "provider quota or credits exhausted".to_string(),
|
||||
Self::Rejected => "request rejected by the provider".to_string(),
|
||||
@@ -1223,7 +1225,10 @@ fn auto_route_usage_has_reported_data(usage: &codewhale_models::Usage) -> bool {
|
||||
/// Stable, persistence-safe identity for one classifier response. The raw
|
||||
/// provider response id is hashed with the frozen dispatch route and instant;
|
||||
/// neither it nor any custom route label crosses into telemetry/persistence.
|
||||
fn auto_route_usage_source_id(route: &EffectiveRouteEnvelope, response_id: &str) -> String {
|
||||
pub(crate) fn auto_route_usage_source_id(
|
||||
route: &EffectiveRouteEnvelope,
|
||||
response_id: &str,
|
||||
) -> String {
|
||||
use sha2::{Digest as _, Sha256};
|
||||
|
||||
let mut digest = Sha256::new();
|
||||
@@ -1556,11 +1561,10 @@ async fn auto_route_decision_recommendation(
|
||||
session_mode,
|
||||
selected_thinking_mode,
|
||||
);
|
||||
// OpenRouter spend enters the session through the ordinary routed-usage
|
||||
// path. TypeSafe is not a chat provider, so its spend is shown on the
|
||||
// receipt only (see `client::system_one`).
|
||||
let request_route = (route == DecisionRouterRoute::Openrouter)
|
||||
.then(|| client.effective_route_envelope(&inventory.router_model, chrono::Utc::now()));
|
||||
// Both transports enter the existing usage ledger. TypeSafe carries its
|
||||
// own Custom/Unknown identity; no chat price or zero-cost claim is inherited.
|
||||
let request_route =
|
||||
Some(client.effective_route_envelope(&inventory.router_model, chrono::Utc::now()));
|
||||
let started = Instant::now();
|
||||
let dispatched = std::sync::atomic::AtomicBool::new(false);
|
||||
let outcome = tokio::time::timeout(
|
||||
@@ -1595,15 +1599,18 @@ async fn auto_route_decision_recommendation(
|
||||
}
|
||||
|
||||
/// A validated `choice` answer.
|
||||
struct ValidChoice {
|
||||
choice: String,
|
||||
probabilities_bp: BTreeMap<String, u16>,
|
||||
confidence_bp: u16,
|
||||
pub(crate) struct ValidChoice {
|
||||
pub(crate) choice: String,
|
||||
pub(crate) probabilities_bp: BTreeMap<String, u16>,
|
||||
pub(crate) confidence_bp: u16,
|
||||
}
|
||||
|
||||
/// Validate one `choice` answer against the offered options. Any violation
|
||||
/// rejects the whole answer; nothing is repaired.
|
||||
fn validated_choice(answer: Option<&SystemOneAnswer>, options: &[&str]) -> Option<ValidChoice> {
|
||||
pub(crate) fn validated_choice(
|
||||
answer: Option<&SystemOneAnswer>,
|
||||
options: &[&str],
|
||||
) -> Option<ValidChoice> {
|
||||
let answer = answer?;
|
||||
if answer.kind != "choice" {
|
||||
return None;
|
||||
@@ -1625,6 +1632,14 @@ fn validated_choice(answer: Option<&SystemOneAnswer>, options: &[&str]) -> Optio
|
||||
if (sum - 1.0).abs() > 0.02 {
|
||||
return None;
|
||||
}
|
||||
let selected = (*answer.probabilities.get(choice)?)?;
|
||||
if answer
|
||||
.probabilities
|
||||
.values()
|
||||
.any(|probability| probability.is_some_and(|v| v > selected))
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let confidence = answer.confidence?;
|
||||
if !confidence.is_finite() || !(0.0..=1.0).contains(&confidence) {
|
||||
return None;
|
||||
@@ -1636,6 +1651,40 @@ fn validated_choice(answer: Option<&SystemOneAnswer>, options: &[&str]) -> Optio
|
||||
})
|
||||
}
|
||||
|
||||
/// One decision response settled through the existing routed-usage ledger.
|
||||
/// Malformed policy answers still carry provider usage; missing/all-zero usage
|
||||
/// remains an explicit coverage gap. The source identity is route-bound.
|
||||
pub(crate) fn decision_usage_batch(
|
||||
request_route: &EffectiveRouteEnvelope,
|
||||
response: &SystemOneResponse,
|
||||
) -> crate::cost_status::RuntimeUsageBatch {
|
||||
let usage = codewhale_models::Usage {
|
||||
input_tokens: response.usage.as_ref().map_or(0, |u| u.input_tokens),
|
||||
output_tokens: response.usage.as_ref().map_or(0, |u| u.output_tokens),
|
||||
..Default::default()
|
||||
};
|
||||
let source_id =
|
||||
auto_route_usage_source_id(request_route, response.id.as_deref().unwrap_or("systemone"));
|
||||
let route = request_route.sanitized_for_persistence();
|
||||
if auto_route_usage_has_reported_data(&usage) {
|
||||
crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: vec![RuntimeUsageRecord {
|
||||
source_id,
|
||||
usage: EffectiveRouteUsage { route, usage },
|
||||
}],
|
||||
..Default::default()
|
||||
}
|
||||
} else {
|
||||
crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
drop_records: vec![RuntimeUsageDropRecord { source_id, route }],
|
||||
dropped_records: 1,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Turn a decoded System One response into a routing attempt. All policy is
|
||||
/// here, in code: the decision model does no arithmetic.
|
||||
fn decision_attempt_from_response(
|
||||
@@ -1656,41 +1705,35 @@ fn decision_attempt_from_response(
|
||||
fallback_reason: None,
|
||||
};
|
||||
if let Some(request_route) = request_route {
|
||||
let usage = codewhale_models::Usage {
|
||||
input_tokens: response
|
||||
.usage
|
||||
.as_ref()
|
||||
.map_or(0, |usage| usage.input_tokens),
|
||||
output_tokens: response
|
||||
.usage
|
||||
.as_ref()
|
||||
.map_or(0, |usage| usage.output_tokens),
|
||||
..Default::default()
|
||||
};
|
||||
let response_id = response.id.as_deref().unwrap_or("systemone");
|
||||
if auto_route_usage_has_reported_data(&usage) {
|
||||
attempt.routed_usage.push(RuntimeUsageRecord {
|
||||
source_id: auto_route_usage_source_id(&request_route, response_id),
|
||||
usage: EffectiveRouteUsage {
|
||||
route: request_route.sanitized_for_persistence(),
|
||||
usage,
|
||||
},
|
||||
});
|
||||
} else {
|
||||
let drop_route = request_route.sanitized_for_persistence();
|
||||
attempt
|
||||
.routed_usage_drop_records
|
||||
.push(RuntimeUsageDropRecord {
|
||||
source_id: auto_route_usage_source_id(
|
||||
&drop_route,
|
||||
&format!("missing-usage:{response_id}"),
|
||||
),
|
||||
route: drop_route,
|
||||
});
|
||||
attempt.routed_usage_dropped_records = 1;
|
||||
}
|
||||
let batch = decision_usage_batch(&request_route, response);
|
||||
attempt.routed_usage = batch.records;
|
||||
attempt.routed_usage_dropped_records = batch.dropped_records;
|
||||
attempt.routed_usage_drop_records = batch.drop_records;
|
||||
}
|
||||
|
||||
// Rejected answers still incurred spend. Keep bounded diagnostic evidence
|
||||
// before any policy return; valid tier evidence below replaces this shell.
|
||||
attempt.decision = Some(AutoRouteDecisionEvidence {
|
||||
choice: "invalid".to_string(),
|
||||
probabilities_bp: Default::default(),
|
||||
confidence_bp: 0,
|
||||
min_confidence_bp: inventory.router_min_confidence_bp,
|
||||
cost_saving_kept_fast: false,
|
||||
thinking: None,
|
||||
provider_reported_cost_usd: response
|
||||
.usage
|
||||
.as_ref()
|
||||
.and_then(|usage| usage.reported_cost()),
|
||||
latency_ms,
|
||||
response_model: response
|
||||
.model
|
||||
.as_deref()
|
||||
.map(|model| model.chars().take(128).collect()),
|
||||
});
|
||||
if response.answers_validated == Some(false) {
|
||||
attempt.failure = Some(AutoRouterFailure::InvalidAnswer);
|
||||
return attempt;
|
||||
}
|
||||
let Some(tier) = validated_choice(response.answers.get("tier"), &DECISION_TIER_OPTIONS) else {
|
||||
attempt.failure = Some(AutoRouterFailure::InvalidAnswer);
|
||||
return attempt;
|
||||
@@ -3280,13 +3323,13 @@ mod decision_router_tests {
|
||||
let _env = hermetic_env();
|
||||
let server = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/systemone"))
|
||||
.and(path("/api/alpha/decisions"))
|
||||
.and(header("authorization", "Bearer or-test-key"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(answer_body(0.82, 0.64)))
|
||||
.expect(1)
|
||||
.mount(&server)
|
||||
.await;
|
||||
let mut config = decision_config(&server.uri(), false, 2);
|
||||
let mut config = decision_config(&format!("{}/api/v1", server.uri()), false, 2);
|
||||
// A configured credential that a tool result echoed back.
|
||||
let secret = "cw-router-secret-should-never-leave-process";
|
||||
config
|
||||
@@ -3363,11 +3406,11 @@ mod decision_router_tests {
|
||||
let _env = hermetic_env();
|
||||
let server = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/systemone"))
|
||||
.and(path("/api/alpha/decisions"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(answer_body(0.82, 0.64)))
|
||||
.mount(&server)
|
||||
.await;
|
||||
let config = decision_config(&server.uri(), false, 2);
|
||||
let config = decision_config(&format!("{}/api/v1", server.uri()), false, 2);
|
||||
let selection = route(&config, "Debug the release pipeline", "").await;
|
||||
|
||||
assert_eq!(selection.provider, ApiProvider::Deepseek);
|
||||
@@ -3418,11 +3461,11 @@ mod decision_router_tests {
|
||||
let _env = hermetic_env();
|
||||
let server = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/systemone"))
|
||||
.and(path("/api/alpha/decisions"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_json(answer_body(0.35, 0.3)))
|
||||
.mount(&server)
|
||||
.await;
|
||||
let config = decision_config(&server.uri(), false, 2);
|
||||
let config = decision_config(&format!("{}/api/v1", server.uri()), false, 2);
|
||||
let selection = route(&config, "What does this function do?", "").await;
|
||||
|
||||
assert_eq!(selection.model, "deepseek-v4-pro", "declared default");
|
||||
@@ -3449,7 +3492,7 @@ mod decision_router_tests {
|
||||
let _env = hermetic_env();
|
||||
let server = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/systemone"))
|
||||
.and(path("/api/alpha/decisions"))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(status)
|
||||
.set_body_string(format!(r#"{{"error":{{"message":"{MARKER}"}}}}"#)),
|
||||
@@ -3457,7 +3500,7 @@ mod decision_router_tests {
|
||||
.expect(1)
|
||||
.mount(&server)
|
||||
.await;
|
||||
let config = decision_config(&server.uri(), false, 2);
|
||||
let config = decision_config(&format!("{}/api/v1", server.uri()), false, 2);
|
||||
let selection = route(&config, "Explain the diff", "").await;
|
||||
|
||||
assert_eq!(selection.model, "deepseek-v4-pro", "status {status}");
|
||||
@@ -3485,7 +3528,7 @@ mod decision_router_tests {
|
||||
let _env = hermetic_env();
|
||||
let server = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/systemone"))
|
||||
.and(path("/api/alpha/decisions"))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200)
|
||||
.set_body_json(answer_body(0.9, 0.8))
|
||||
@@ -3493,7 +3536,7 @@ mod decision_router_tests {
|
||||
)
|
||||
.mount(&server)
|
||||
.await;
|
||||
let config = decision_config(&server.uri(), false, 1);
|
||||
let config = decision_config(&format!("{}/api/v1", server.uri()), false, 1);
|
||||
let selection = route(&config, "Explain the diff", "").await;
|
||||
|
||||
let receipt = receipt(&selection);
|
||||
@@ -3517,7 +3560,7 @@ mod decision_router_tests {
|
||||
.expect(0)
|
||||
.mount(&server)
|
||||
.await;
|
||||
let mut config = decision_config(&server.uri(), false, 2);
|
||||
let mut config = decision_config(&format!("{}/api/v1", server.uri()), false, 2);
|
||||
// A single-tier active provider: nothing for the decision to choose.
|
||||
config.provider = Some("openrouter".to_string());
|
||||
config.default_text_model = Some("synthetic/single-tier-model".to_string());
|
||||
@@ -3590,8 +3633,17 @@ mod decision_router_tests {
|
||||
}
|
||||
);
|
||||
assert!(receipt.decision.is_some());
|
||||
// TypeSafe is not a chat provider: spend stays on the receipt only.
|
||||
assert!(selection.routed_usage.is_empty());
|
||||
// A decision endpoint never inherits the active DeepSeek billing identity.
|
||||
assert_eq!(selection.routed_usage.len(), 1);
|
||||
let usage = &selection.routed_usage[0];
|
||||
assert_eq!(usage.usage.route.provider, ApiProvider::Custom);
|
||||
assert_eq!(usage.usage.route.provider_identity, "typesafe");
|
||||
assert_eq!(
|
||||
usage.usage.route.billing_mode,
|
||||
crate::cost_status::RouteBillingMode::Unknown
|
||||
);
|
||||
assert_eq!(usage.usage.usage.input_tokens, 476);
|
||||
assert_eq!(usage.usage.usage.output_tokens, 70);
|
||||
assert!(selection.routed_usage_drop_records.is_empty());
|
||||
}
|
||||
|
||||
@@ -3672,6 +3724,18 @@ mod decision_router_tests {
|
||||
for body in [outside, bad_sum, missing, null_probability, bad_confidence] {
|
||||
let attempt = policy(false, &parsed(body.clone()));
|
||||
assert_eq!(attempt.recommendation, None, "{body}");
|
||||
assert_eq!(
|
||||
attempt
|
||||
.decision
|
||||
.as_ref()
|
||||
.and_then(|r| r.provider_reported_cost_usd.as_deref()),
|
||||
parsed(body.clone())
|
||||
.usage
|
||||
.as_ref()
|
||||
.and_then(|u| u.reported_cost())
|
||||
.as_deref(),
|
||||
"rejected policy retains cost"
|
||||
);
|
||||
assert_eq!(
|
||||
attempt.failure,
|
||||
Some(AutoRouterFailure::InvalidAnswer),
|
||||
|
||||
@@ -449,6 +449,7 @@ fn messages_from_thread_detail_batches_tool_results() {
|
||||
saved_session_checkpoint: None,
|
||||
};
|
||||
let turn = TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: 2,
|
||||
id: turn_id.clone(),
|
||||
@@ -9381,6 +9382,7 @@ async fn session_save_merges_thread_cost_split_and_records_coverage() -> Result<
|
||||
let store = runtime_threads.test_store();
|
||||
let now = Utc::now();
|
||||
let mut turn = TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: 2,
|
||||
id: "turn_cost_merge".to_string(),
|
||||
@@ -9579,6 +9581,7 @@ async fn session_save_persists_parent_cny_unpriced_reasons_without_double_count(
|
||||
let store = runtime_threads.test_store();
|
||||
let now = Utc::now();
|
||||
let mut turn = TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: 2,
|
||||
id: "turn_cny_reasons".to_string(),
|
||||
@@ -10829,6 +10832,7 @@ fn seed_summary_search_transcript(
|
||||
item_ids.push(item_id);
|
||||
}
|
||||
store.save_turn(&TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: 2,
|
||||
id: turn_id.clone(),
|
||||
|
||||
@@ -603,6 +603,20 @@ where
|
||||
.collect::<Vec<_>>()
|
||||
.serialize(serializer)
|
||||
}
|
||||
fn serialize_decision_receipts<S>(
|
||||
values: &[crate::cost_status::RuntimeDecisionReceipt],
|
||||
serializer: S,
|
||||
) -> std::result::Result<S::Ok, S::Error>
|
||||
where
|
||||
S: serde::Serializer,
|
||||
{
|
||||
values
|
||||
.iter()
|
||||
.map(crate::cost_status::RuntimeDecisionReceipt::sanitized)
|
||||
.collect::<Vec<_>>()
|
||||
.serialize(serializer)
|
||||
}
|
||||
|
||||
const RUNTIME_RESTART_REASON: &str = "Interrupted by process restart";
|
||||
const EMPTY_TURN_REASON: &str = "Turn completed without engine output";
|
||||
const DYNAMIC_TOOL_RESULT_TIMEOUT: Duration = Duration::from_secs(300);
|
||||
@@ -1457,6 +1471,14 @@ pub struct TurnRecord {
|
||||
/// inline RLM/guardian calls.
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub routed_usage: Vec<EffectiveRouteUsage>,
|
||||
/// Decision evidence shares this durable origin-turn ledger, including
|
||||
/// responses arriving after the parent terminal event.
|
||||
#[serde(
|
||||
default,
|
||||
skip_serializing_if = "Vec::is_empty",
|
||||
serialize_with = "serialize_decision_receipts"
|
||||
)]
|
||||
pub decision_receipts: Vec<crate::cost_status::RuntimeDecisionReceipt>,
|
||||
/// Exact missing-usage provider responses paired with their frozen route.
|
||||
/// Sources are persisted only as fingerprints, routes are sanitized, and
|
||||
/// the shared source ledger bounds usage plus missing-usage records.
|
||||
@@ -1523,6 +1545,11 @@ pub struct TurnRecord {
|
||||
|
||||
impl TurnRecord {
|
||||
fn validate_output_token_limit(&self) -> Result<()> {
|
||||
if self.decision_receipts.len() > MAX_ROUTED_USAGE_RECORDS_PER_TURN
|
||||
|| self.decision_receipts.iter().any(|r| !r.is_bounded())
|
||||
{
|
||||
bail!("Turn decision evidence exceeds its bound");
|
||||
}
|
||||
if self.schema_version > MAX_SUPPORTED_RUNTIME_SCHEMA_VERSION {
|
||||
bail!(
|
||||
"Turn schema v{} is newer than supported v{}",
|
||||
@@ -1669,6 +1696,32 @@ fn append_routed_usage_drop_record(turn: &mut TurnRecord, record: RuntimeUsageDr
|
||||
true
|
||||
}
|
||||
|
||||
fn append_decision_receipt(
|
||||
turn: &mut TurnRecord,
|
||||
receipt: &crate::cost_status::RuntimeDecisionReceipt,
|
||||
) -> bool {
|
||||
if !receipt.is_bounded() {
|
||||
return false;
|
||||
}
|
||||
let receipt = receipt.sanitized();
|
||||
if turn
|
||||
.decision_receipts
|
||||
.iter()
|
||||
.any(|old| old.source_id == receipt.source_id)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
if turn.decision_receipts.len() >= MAX_ROUTED_USAGE_RECORDS_PER_TURN {
|
||||
if turn.routed_usage_dropped_records == 0 {
|
||||
turn.routed_usage_dropped_records = 1;
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
turn.decision_receipts.push(receipt);
|
||||
true
|
||||
}
|
||||
|
||||
/// Bind pre-parent auxiliary calls to a reserved Runtime turn before the
|
||||
/// engine accepts the parent operation. Exact records are persisted now so a
|
||||
/// crash cannot erase them; the engine's terminal event remains the sole
|
||||
@@ -1678,6 +1731,9 @@ fn append_initial_routed_usage_to_turn(
|
||||
turn: &mut TurnRecord,
|
||||
batch: &crate::cost_status::RuntimeUsageBatch,
|
||||
) {
|
||||
for receipt in &batch.decisions {
|
||||
append_decision_receipt(turn, receipt);
|
||||
}
|
||||
for record in &batch.records {
|
||||
append_routed_usage_record(turn, &record.source_id, record.usage.clone());
|
||||
}
|
||||
@@ -1694,7 +1750,10 @@ const UNACCEPTED_TURN_REASON: &str =
|
||||
"Turn was not accepted; this record keeps the completed pre-turn provider call";
|
||||
|
||||
fn routed_usage_batch_is_empty(batch: &crate::cost_status::RuntimeUsageBatch) -> bool {
|
||||
batch.records.is_empty() && batch.drop_records.is_empty() && batch.dropped_records == 0
|
||||
batch.decisions.is_empty()
|
||||
&& batch.records.is_empty()
|
||||
&& batch.drop_records.is_empty()
|
||||
&& batch.dropped_records == 0
|
||||
}
|
||||
|
||||
/// Durable identity of the settlement record for one completed pre-turn
|
||||
@@ -1721,6 +1780,12 @@ fn unaccepted_routed_usage_turn_id(
|
||||
.map(|record| routed_usage_source_fingerprint(&record.source_id)),
|
||||
)
|
||||
.collect::<Vec<_>>();
|
||||
identities.extend(
|
||||
batch
|
||||
.decisions
|
||||
.iter()
|
||||
.map(|record| routed_usage_source_fingerprint(&record.source_id)),
|
||||
);
|
||||
if identities.is_empty() {
|
||||
return runtime_record_id("turn_unaccepted");
|
||||
}
|
||||
@@ -1758,6 +1823,7 @@ fn settle_unaccepted_routed_usage(
|
||||
} else {
|
||||
let now = Utc::now();
|
||||
TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
id: turn_id.clone(),
|
||||
@@ -11432,6 +11498,7 @@ impl RuntimeThreadManager {
|
||||
// Only create a turn if there's content.
|
||||
if !item_ids.is_empty() {
|
||||
self.store.save_turn(&TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: if turn_seed.image_content.is_empty() {
|
||||
CURRENT_RUNTIME_SCHEMA_VERSION
|
||||
@@ -11800,6 +11867,9 @@ impl RuntimeThreadManager {
|
||||
let _turn_mutation = self.store.turn_mutation.lock();
|
||||
match self.store.load_turn(turn_id) {
|
||||
Ok(mut turn) => {
|
||||
for receipt in &background_usage.decisions {
|
||||
append_decision_receipt(&mut turn, receipt);
|
||||
}
|
||||
for record in background_usage.records.iter().cloned() {
|
||||
append_routed_usage_record(&mut turn, &record.source_id, record.usage);
|
||||
}
|
||||
@@ -12478,6 +12548,7 @@ impl RuntimeThreadManager {
|
||||
// malformed/stale parent catalog entry cannot erase real
|
||||
// auxiliary spend on the error path.
|
||||
let initial_routed_usage = crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: std::mem::take(&mut selection.routed_usage),
|
||||
drop_records: std::mem::take(&mut selection.routed_usage_drop_records),
|
||||
dropped_records: std::mem::take(
|
||||
@@ -13087,6 +13158,7 @@ impl RuntimeThreadManager {
|
||||
thread.auto_approve,
|
||||
);
|
||||
let turn = TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
id: turn_id.clone(),
|
||||
@@ -14309,6 +14381,21 @@ impl RuntimeThreadManager {
|
||||
drop_store.save_turn(&turn).is_ok()
|
||||
})),
|
||||
);
|
||||
let decision_store = self.store.clone();
|
||||
let decision_turn_id = turn_id.to_string();
|
||||
crate::cost_status::register_runtime_decision_sink(
|
||||
turn_id,
|
||||
Arc::new(move |receipt| {
|
||||
let _turn_mutation = decision_store.turn_mutation.lock();
|
||||
let Ok(mut turn) = decision_store.load_turn(&decision_turn_id) else {
|
||||
return false;
|
||||
};
|
||||
if !append_decision_receipt(&mut turn, &receipt) {
|
||||
return true;
|
||||
}
|
||||
decision_store.save_turn(&turn).is_ok()
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
/// Persist an engine status line as a completed `status` item.
|
||||
@@ -14813,6 +14900,14 @@ impl RuntimeThreadManager {
|
||||
if let Some(batch) =
|
||||
crate::cost_status::child_usage_records_from_metadata(metadata)
|
||||
{
|
||||
if !batch.decisions.is_empty() {
|
||||
let _turn_mutation = self.store.turn_mutation.lock();
|
||||
let mut turn = self.store.load_turn(&turn_id)?;
|
||||
for receipt in &batch.decisions {
|
||||
append_decision_receipt(&mut turn, receipt);
|
||||
}
|
||||
self.store.save_turn(&turn)?;
|
||||
}
|
||||
for record in batch.records {
|
||||
self.append_routed_usage_to_turn(
|
||||
&turn_id,
|
||||
@@ -16191,6 +16286,9 @@ impl RuntimeThreadManager {
|
||||
turn.duration_ms = turn.started_at.map(|start| duration_ms(start, ended_at));
|
||||
turn.usage = turn_usage;
|
||||
turn.effective_route_usage = turn_effective_route_usage;
|
||||
for receipt in &background_usage.decisions {
|
||||
append_decision_receipt(&mut turn, receipt);
|
||||
}
|
||||
for record in background_usage.records {
|
||||
append_routed_usage_record(&mut turn, &record.source_id, record.usage);
|
||||
}
|
||||
|
||||
@@ -2447,6 +2447,7 @@ fn sample_thread(thread_id: &str) -> ThreadRecord {
|
||||
fn sample_turn(thread_id: &str, turn_id: &str, status: RuntimeTurnStatus) -> TurnRecord {
|
||||
let now = Utc::now();
|
||||
TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
id: turn_id.to_string(),
|
||||
@@ -8464,6 +8465,7 @@ async fn initial_classifier_usage_is_persisted_before_terminal_and_merged_exactl
|
||||
Utc::now(),
|
||||
);
|
||||
let classifier_batch = crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: vec![crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: "auto-router:runtime-fixture".to_string(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
@@ -8600,6 +8602,7 @@ fn classifier_settlement_batch(
|
||||
);
|
||||
route.billing_mode = crate::cost_status::RouteBillingMode::Subscription;
|
||||
crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: vec![crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: format!("auto-router:{source_prefix}-usage"),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
@@ -16747,6 +16750,7 @@ fn opening_manager_recovers_stale_queued_and_in_progress_work() -> Result<()> {
|
||||
manager.store.save_item(&queued_item)?;
|
||||
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
id: "turn_in_progress".to_string(),
|
||||
@@ -16785,6 +16789,7 @@ fn opening_manager_recovers_stale_queued_and_in_progress_work() -> Result<()> {
|
||||
workspace_snapshots: Vec::new(),
|
||||
})?;
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
id: "turn_queued".to_string(),
|
||||
@@ -16981,6 +16986,7 @@ fn seed_turns_with_user_messages(
|
||||
ended_at: Some(created_at),
|
||||
})?;
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
id: turn_id.clone(),
|
||||
@@ -17800,6 +17806,7 @@ fn restart_rebuild_restores_tool_call_identity_from_persisted_items() -> Result<
|
||||
manager.store.save_item(&user_item)?;
|
||||
manager.store.save_item(&call_item)?;
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
id: "turn_5823".to_string(),
|
||||
@@ -17905,6 +17912,7 @@ fn restart_rebuild_keeps_in_flight_tool_call_identity() -> Result<()> {
|
||||
};
|
||||
manager.store.save_item(&call_item)?;
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
id: "turn_5823_inflight".to_string(),
|
||||
@@ -18005,6 +18013,7 @@ fn restart_rebuild_skips_steers_the_engine_never_delivered() -> Result<()> {
|
||||
manager.store.save_item(&dropped)?;
|
||||
manager.store.save_item(&pending)?;
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
id: turn_id.clone(),
|
||||
@@ -18106,6 +18115,7 @@ fn restart_rebuild_skips_legacy_tool_items_without_identity() -> Result<()> {
|
||||
manager.store.save_item(&user_item)?;
|
||||
manager.store.save_item(&legacy_tool_item)?;
|
||||
manager.store.save_turn(&TurnRecord {
|
||||
decision_receipts: Vec::new(),
|
||||
max_output_tokens: None,
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
id: "turn_5823_legacy".to_string(),
|
||||
@@ -21962,3 +21972,55 @@ mod execution_identity {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn decision_receipt_lease_persists_terminal_turn_and_replays_exactly_once() -> Result<()> {
|
||||
let _cost_scope = crate::cost_status::test_scope();
|
||||
let directory = test_runtime_dir();
|
||||
let manager = test_manager(directory.clone())?;
|
||||
let thread = manager
|
||||
.create_thread(CreateThreadRequest::default())
|
||||
.await?;
|
||||
let turn = sample_turn(
|
||||
&thread.id,
|
||||
"decision-terminal-origin",
|
||||
RuntimeTurnStatus::Completed,
|
||||
);
|
||||
manager.store.save_turn(&turn)?;
|
||||
manager.register_runtime_usage_sink(&turn.id);
|
||||
let lease = crate::cost_status::acquire_runtime_usage_lease(&turn.id).expect("origin lease");
|
||||
crate::cost_status::finish_runtime_usage_owner(&turn.id);
|
||||
let receipt = crate::cost_status::decision_receipt_fixture("raw-runtime-decision-id");
|
||||
let batch = crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: vec![receipt.clone()],
|
||||
..Default::default()
|
||||
};
|
||||
for _ in 0..2 {
|
||||
crate::cost_status::report_runtime_usage_batch(
|
||||
crate::cost_status::scope_token(),
|
||||
Some(&turn.id),
|
||||
&batch,
|
||||
);
|
||||
}
|
||||
drop(lease);
|
||||
let reloaded = test_manager(directory)?.store.load_turn(&turn.id)?;
|
||||
assert_eq!(reloaded.status, RuntimeTurnStatus::Completed);
|
||||
assert_eq!(reloaded.decision_receipts, vec![receipt.sanitized()]);
|
||||
assert_eq!(
|
||||
reloaded.decision_receipts[0]
|
||||
.evidence
|
||||
.provider_reported_cost_usd
|
||||
.as_deref(),
|
||||
Some("0.000012054")
|
||||
);
|
||||
assert!(
|
||||
reloaded.routed_usage.is_empty(),
|
||||
"diagnostic receipt alone must not mint a second token charge"
|
||||
);
|
||||
assert_eq!(
|
||||
unaccepted_routed_usage_turn_id(&thread.id, &batch),
|
||||
unaccepted_routed_usage_turn_id(&thread.id, &batch)
|
||||
);
|
||||
assert!(!serde_json::to_string(&reloaded)?.contains("raw-runtime-decision-id"));
|
||||
Ok(())
|
||||
}
|
||||
@@ -71,6 +71,8 @@ struct LateUsageRecord {
|
||||
route: crate::cost_status::EffectiveRouteEnvelope,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
usage: Option<codewhale_models::Usage>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
decision: Option<crate::cost_status::RuntimeDecisionReceipt>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
@@ -1681,6 +1683,7 @@ impl SessionManager {
|
||||
|| ledger.records.iter().any(|record| {
|
||||
!is_sha256_fingerprint(&record.source_fingerprint)
|
||||
|| !is_sha256_fingerprint(&record.turn_fingerprint)
|
||||
|| record.decision.as_ref().is_some_and(|r| !r.is_bounded())
|
||||
})
|
||||
{
|
||||
return Err(io::Error::new(
|
||||
@@ -1742,6 +1745,7 @@ impl SessionManager {
|
||||
source_id: &str,
|
||||
route: &crate::cost_status::EffectiveRouteEnvelope,
|
||||
usage: Option<&codewhale_models::Usage>,
|
||||
decision: Option<&crate::cost_status::RuntimeDecisionReceipt>,
|
||||
) -> io::Result<bool> {
|
||||
let (path, lock_path) = self.ensure_late_usage_paths(session_id)?;
|
||||
let lock_file = open_private_lock_file(&lock_path)?;
|
||||
@@ -1753,11 +1757,21 @@ impl SessionManager {
|
||||
}
|
||||
let mut ledger = Self::load_late_usage_unlocked(&path)?;
|
||||
let source_fingerprint = crate::cost_status::usage_source_fingerprint(source_id);
|
||||
if ledger
|
||||
if let Some(record) = ledger
|
||||
.records
|
||||
.iter()
|
||||
.any(|record| record.source_fingerprint == source_fingerprint)
|
||||
.iter_mut()
|
||||
.find(|record| record.source_fingerprint == source_fingerprint)
|
||||
{
|
||||
if let Some(decision) = decision {
|
||||
if !decision.is_bounded() {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::InvalidData,
|
||||
"decision receipt exceeds its bound",
|
||||
));
|
||||
}
|
||||
record.decision = Some(decision.sanitized());
|
||||
Self::write_late_usage_ledger(&path, &ledger)?;
|
||||
}
|
||||
return Ok(true);
|
||||
}
|
||||
if ledger.records.len() == MAX_LATE_USAGE_RECORDS_PER_SESSION {
|
||||
@@ -1772,11 +1786,48 @@ impl SessionManager {
|
||||
turn_fingerprint: crate::cost_status::usage_source_fingerprint(turn_id),
|
||||
route: route.sanitized_for_persistence(),
|
||||
usage: usage.cloned(),
|
||||
decision: decision.map(crate::cost_status::RuntimeDecisionReceipt::sanitized),
|
||||
});
|
||||
Self::write_late_usage_ledger(&path, &ledger)?;
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
pub(crate) fn persist_late_decision_receipt(
|
||||
&self,
|
||||
session_id: &str,
|
||||
turn_id: &str,
|
||||
receipt: &crate::cost_status::RuntimeDecisionReceipt,
|
||||
) -> io::Result<bool> {
|
||||
if !receipt.is_bounded() {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::InvalidData,
|
||||
"decision receipt exceeds its bound",
|
||||
));
|
||||
}
|
||||
self.persist_late_usage_record(
|
||||
session_id,
|
||||
turn_id,
|
||||
&receipt.source_id,
|
||||
&receipt.route,
|
||||
receipt.usage.as_ref(),
|
||||
Some(receipt),
|
||||
)
|
||||
}
|
||||
|
||||
/// Read the bounded provider decision evidence from the existing origin ledger.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn decision_receipts_for_session(
|
||||
&self,
|
||||
session_id: &str,
|
||||
) -> io::Result<Vec<crate::cost_status::RuntimeDecisionReceipt>> {
|
||||
Ok(self
|
||||
.load_late_usage(session_id)?
|
||||
.records
|
||||
.into_iter()
|
||||
.filter_map(|record| record.decision)
|
||||
.collect())
|
||||
}
|
||||
|
||||
pub(crate) fn persist_late_runtime_usage(
|
||||
&self,
|
||||
session_id: &str,
|
||||
@@ -1789,6 +1840,7 @@ impl SessionManager {
|
||||
&record.source_id,
|
||||
&record.usage.route,
|
||||
Some(&record.usage.usage),
|
||||
None,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -1798,7 +1850,14 @@ impl SessionManager {
|
||||
turn_id: &str,
|
||||
record: &crate::cost_status::RuntimeUsageDropRecord,
|
||||
) -> io::Result<bool> {
|
||||
self.persist_late_usage_record(session_id, turn_id, &record.source_id, &record.route, None)
|
||||
self.persist_late_usage_record(
|
||||
session_id,
|
||||
turn_id,
|
||||
&record.source_id,
|
||||
&record.route,
|
||||
None,
|
||||
None,
|
||||
)
|
||||
}
|
||||
|
||||
fn with_session_read_lock<T>(
|
||||
@@ -1863,6 +1922,14 @@ impl SessionManager {
|
||||
}
|
||||
};
|
||||
for record in ledger.records {
|
||||
if let Some(receipt) = &record.decision {
|
||||
if receipt.is_bounded() {
|
||||
metadata
|
||||
.cost
|
||||
.route_receipts
|
||||
.insert(receipt.diagnostic_receipt());
|
||||
}
|
||||
}
|
||||
let source_fingerprint = record.source_fingerprint.clone();
|
||||
let source_id = format!("late:{}", record.source_fingerprint);
|
||||
let mut pending = if let Some(usage) = record.usage.as_ref() {
|
||||
@@ -4148,6 +4215,90 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decision_receipt_survives_restart_replay_and_session_deletion() {
|
||||
let tmp = tempdir().expect("tempdir");
|
||||
let manager = SessionManager::new(tmp.path().join("sessions")).expect("manager");
|
||||
save_late_usage_test_session(&manager, "decision-origin");
|
||||
let receipt = crate::cost_status::decision_receipt_fixture("raw-decision-response-id");
|
||||
manager
|
||||
.persist_late_decision_receipt("decision-origin", "origin-turn", &receipt)
|
||||
.expect("decision append");
|
||||
manager
|
||||
.persist_late_runtime_usage(
|
||||
"decision-origin",
|
||||
"origin-turn",
|
||||
&crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: receipt.source_id.clone(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
route: receipt.route.clone(),
|
||||
usage: receipt.usage.clone().expect("usage"),
|
||||
},
|
||||
},
|
||||
)
|
||||
.expect("token append dedupes");
|
||||
drop(manager);
|
||||
let manager = SessionManager::new(tmp.path().join("sessions")).expect("restart");
|
||||
let retained = manager
|
||||
.decision_receipts_for_session("decision-origin")
|
||||
.expect("receipts");
|
||||
assert_eq!(retained, vec![receipt.sanitized()]);
|
||||
let mut restored = manager
|
||||
.load_session_snapshot("decision-origin")
|
||||
.expect("restore");
|
||||
assert_eq!(restored.metadata.total_tokens, 13);
|
||||
assert_eq!(
|
||||
restored.metadata.cost.unpriced_turns, 1,
|
||||
"unknown TypeSafe billing remains explicit"
|
||||
);
|
||||
assert!(
|
||||
restored
|
||||
.metadata
|
||||
.cost
|
||||
.route_receipts
|
||||
.iter()
|
||||
.any(|r| r.contains("0.000012054"))
|
||||
);
|
||||
manager.apply_late_usage_to_metadata(&mut restored.metadata);
|
||||
assert_eq!(
|
||||
restored.metadata.total_tokens, 13,
|
||||
"replay must not count tokens twice"
|
||||
);
|
||||
assert_eq!(restored.metadata.cost.unpriced_turns, 1);
|
||||
manager.save_session(&restored).expect("save overlay");
|
||||
assert_eq!(
|
||||
manager
|
||||
.load_session_snapshot("decision-origin")
|
||||
.expect("resume again")
|
||||
.metadata
|
||||
.total_tokens,
|
||||
13
|
||||
);
|
||||
let (ledger, _) = manager.late_usage_paths("decision-origin").expect("paths");
|
||||
assert!(
|
||||
!fs::read_to_string(&ledger)
|
||||
.expect("ledger")
|
||||
.contains("raw-decision-response-id")
|
||||
);
|
||||
manager.delete_session("decision-origin").expect("delete");
|
||||
assert!(
|
||||
manager
|
||||
.persist_late_decision_receipt("decision-origin", "origin-turn", &receipt)
|
||||
.expect("retired replay")
|
||||
);
|
||||
assert!(
|
||||
!ledger.exists(),
|
||||
"late evidence must never recreate a deleted origin"
|
||||
);
|
||||
let mut oversized = receipt;
|
||||
oversized.evidence.response_model = Some("x".repeat(129));
|
||||
assert!(
|
||||
manager
|
||||
.persist_late_decision_receipt("decision-origin", "origin-turn", &oversized)
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn late_usage_reads_do_not_create_accounting_storage() {
|
||||
let tmp = tempdir().expect("tempdir");
|
||||
|
||||
+371
-31
@@ -44,8 +44,9 @@
|
||||
//! - The TypeSafe route needs a TypeSafe key even for a self-hosted server
|
||||
//! (the transport always authenticates); a server that ignores auth can
|
||||
//! be given any placeholder key.
|
||||
//! - The gate's own spend is the configured endpoint's and is not entered
|
||||
//! in session cost totals or Auto receipts.
|
||||
//! - Usage settles through the originating turn's shared ledger. Missing
|
||||
//! usage or cancellation after dispatch records a coverage gap. TypeSafe
|
||||
//! is an unpriced Custom route until a billing basis is reviewed.
|
||||
//! - Misconfiguration while enabled (missing or unknown provider, bad
|
||||
//! timeout, missing key) is logged at `warn` for each turn and nothing is
|
||||
//! sent.
|
||||
@@ -56,7 +57,7 @@
|
||||
//! (Von, OpenJev, Laya) are third-party open models; only the integration
|
||||
//! architecture and the routing method here are covered by that attribution.
|
||||
|
||||
use std::sync::atomic::AtomicBool;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use serde_json::{Value, json};
|
||||
@@ -67,7 +68,53 @@ use codewhale_core::role::Role;
|
||||
use crate::client::CodewhaleClient;
|
||||
use crate::client::system_one::{DecisionRouterRoute, SystemOneResponse};
|
||||
use crate::config::Config;
|
||||
use crate::model_routing::{AutoRouterFailure, truncate_for_auto_router};
|
||||
use crate::model_routing::{
|
||||
AutoRouterFailure, auto_route_usage_source_id, decision_usage_batch, truncate_for_auto_router,
|
||||
};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
#[derive(Clone)]
|
||||
struct ShadowUsageContext {
|
||||
scope: crate::cost_status::CostScopeToken,
|
||||
runtime_owner: Option<String>,
|
||||
// Retains the origin's existing durable ledger until settlement finishes.
|
||||
_lease: Option<crate::cost_status::RuntimeUsageLease>,
|
||||
#[cfg(test)]
|
||||
test_origin: std::thread::ThreadId,
|
||||
}
|
||||
|
||||
impl ShadowUsageContext {
|
||||
fn capture(owner: Option<&str>) -> Self {
|
||||
Self {
|
||||
scope: crate::cost_status::scope_token(),
|
||||
runtime_owner: owner.map(str::to_owned),
|
||||
_lease: owner.and_then(crate::cost_status::acquire_runtime_usage_lease),
|
||||
#[cfg(test)]
|
||||
test_origin: crate::cost_status::test_cost_scope_id(),
|
||||
}
|
||||
}
|
||||
|
||||
async fn report(&self, batch: crate::cost_status::RuntimeUsageBatch) {
|
||||
let context = self.clone();
|
||||
// Existing sinks may write the origin-session ledger. Keep their
|
||||
// filesystem/SQLite work off Tokio workers as well.
|
||||
let settled = tokio::task::spawn_blocking(move || {
|
||||
#[cfg(test)]
|
||||
let _origin = crate::cost_status::bind_test_cost_scope(context.test_origin);
|
||||
crate::cost_status::report_runtime_usage_batch(
|
||||
context.scope,
|
||||
context.runtime_owner.as_deref(),
|
||||
&batch,
|
||||
);
|
||||
// Release the lease while its captured test binding is still live.
|
||||
drop(context);
|
||||
})
|
||||
.await;
|
||||
if settled.is_err() {
|
||||
tracing::warn!(target: "superfast", "decision usage settlement worker failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Master switch. The gate never runs unless this env var is truthy.
|
||||
const ENABLED_VAR: &str = "SUPERFAST_ENABLED";
|
||||
@@ -207,13 +254,15 @@ fn last_user_text(messages: &[Message]) -> Option<String> {
|
||||
}
|
||||
|
||||
/// Fire the shadow gate for a turn's first model request. Returns at once:
|
||||
/// the evaluation runs on a detached task and only logs. No task is started
|
||||
/// the detached evaluation uses the turn's cancellation and accounting owner. No task is started
|
||||
/// (and `None` is returned) when the gate is off or misconfigured, when there
|
||||
/// is no user text, or when no Tokio runtime is present. The handle exists
|
||||
/// for tests; the turn loop drops it.
|
||||
pub(crate) fn spawn_shadow_gate(
|
||||
config: &Config,
|
||||
messages: &[Message],
|
||||
runtime_owner: Option<&str>,
|
||||
cancel_token: &CancellationToken,
|
||||
) -> Option<tokio::task::JoinHandle<ShadowOutcome>> {
|
||||
let settings = match ShadowSettings::from_env()? {
|
||||
Ok(settings) => settings,
|
||||
@@ -225,8 +274,18 @@ pub(crate) fn spawn_shadow_gate(
|
||||
let latest_request = last_user_text(messages)?;
|
||||
let runtime = tokio::runtime::Handle::try_current().ok()?;
|
||||
let config = config.clone();
|
||||
// Capture before detaching; the next session/turn must never acquire it.
|
||||
let usage_context = ShadowUsageContext::capture(runtime_owner);
|
||||
let cancel_token = cancel_token.clone();
|
||||
Some(runtime.spawn(async move {
|
||||
let outcome = evaluate(&config, &settings, &latest_request).await;
|
||||
let outcome = evaluate(
|
||||
&config,
|
||||
&settings,
|
||||
&latest_request,
|
||||
&usage_context,
|
||||
&cancel_token,
|
||||
)
|
||||
.await;
|
||||
log_outcome(outcome);
|
||||
outcome
|
||||
}))
|
||||
@@ -264,7 +323,15 @@ async fn evaluate(
|
||||
config: &Config,
|
||||
settings: &ShadowSettings,
|
||||
latest_request: &str,
|
||||
usage_context: &ShadowUsageContext,
|
||||
cancel_token: &CancellationToken,
|
||||
) -> ShadowOutcome {
|
||||
if cancel_token.is_cancelled() {
|
||||
return ShadowOutcome::Failed {
|
||||
failure: AutoRouterFailure::Cancelled,
|
||||
latency_ms: 0,
|
||||
};
|
||||
}
|
||||
// Client construction resolves keys (environment, secret store), so it
|
||||
// runs off the async worker (#6149).
|
||||
let built = {
|
||||
@@ -287,27 +354,95 @@ async fn evaluate(
|
||||
};
|
||||
};
|
||||
let body = decision_body(&client, &settings.model, latest_request);
|
||||
let request_route = client.effective_route_envelope(&settings.model, chrono::Utc::now());
|
||||
let dispatched = AtomicBool::new(false);
|
||||
let started = Instant::now();
|
||||
let answer = tokio::time::timeout(
|
||||
settings.timeout,
|
||||
client.system_one_decide(&body, &dispatched),
|
||||
)
|
||||
.await;
|
||||
let answer = tokio::select! {
|
||||
biased;
|
||||
() = cancel_token.cancelled() => Err(AutoRouterFailure::Cancelled),
|
||||
result = tokio::time::timeout(settings.timeout, client.system_one_decide(&body, &dispatched)) => {
|
||||
result.unwrap_or(Err(AutoRouterFailure::Timeout))
|
||||
}
|
||||
};
|
||||
let latency_ms = u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX);
|
||||
match answer {
|
||||
Err(_) => ShadowOutcome::Failed {
|
||||
failure: AutoRouterFailure::Timeout,
|
||||
latency_ms,
|
||||
},
|
||||
Ok(Err(failure)) => ShadowOutcome::Failed {
|
||||
failure,
|
||||
latency_ms,
|
||||
},
|
||||
Ok(Ok(response)) => ShadowOutcome::Recommendation {
|
||||
route: derive_route(&response),
|
||||
latency_ms,
|
||||
},
|
||||
Err(failure) => {
|
||||
if dispatched.load(Ordering::Acquire) {
|
||||
usage_context
|
||||
.report(crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
drop_records: vec![crate::cost_status::RuntimeUsageDropRecord {
|
||||
source_id: auto_route_usage_source_id(
|
||||
&request_route,
|
||||
"shadow:dispatched-unreceipted",
|
||||
),
|
||||
route: request_route.sanitized_for_persistence(),
|
||||
}],
|
||||
dropped_records: 1,
|
||||
..Default::default()
|
||||
})
|
||||
.await;
|
||||
}
|
||||
ShadowOutcome::Failed {
|
||||
failure,
|
||||
latency_ms,
|
||||
}
|
||||
}
|
||||
Ok(response) => {
|
||||
// Billing evidence settles even when strict answer validation
|
||||
// prevents the policy from producing a recommendation.
|
||||
let mut batch = decision_usage_batch(&request_route, &response);
|
||||
let intent = crate::model_routing::validated_choice(
|
||||
response.answers.get("intent"),
|
||||
&["code_change", "code_question", "command", "chat", "other"],
|
||||
);
|
||||
batch
|
||||
.decisions
|
||||
.push(crate::cost_status::RuntimeDecisionReceipt {
|
||||
source_id: auto_route_usage_source_id(
|
||||
&request_route,
|
||||
response.id.as_deref().unwrap_or("systemone"),
|
||||
),
|
||||
route: request_route.sanitized_for_persistence(),
|
||||
usage: response.usage.as_ref().map(|u| codewhale_models::Usage {
|
||||
input_tokens: u.input_tokens,
|
||||
output_tokens: u.output_tokens,
|
||||
..Default::default()
|
||||
}),
|
||||
shadow: true,
|
||||
valid_answers: response.answers_validated != Some(false),
|
||||
evidence: crate::model_routing::AutoRouteDecisionEvidence {
|
||||
choice: intent
|
||||
.as_ref()
|
||||
.map_or_else(|| "invalid".to_string(), |v| v.choice.clone()),
|
||||
probabilities_bp: intent
|
||||
.as_ref()
|
||||
.map_or_else(Default::default, |v| v.probabilities_bp.clone()),
|
||||
confidence_bp: intent.as_ref().map_or(0, |v| v.confidence_bp),
|
||||
min_confidence_bp: 5_000,
|
||||
cost_saving_kept_fast: false,
|
||||
thinking: None,
|
||||
provider_reported_cost_usd: response
|
||||
.usage
|
||||
.as_ref()
|
||||
.and_then(|u| u.reported_cost()),
|
||||
latency_ms,
|
||||
response_model: response.model.clone(),
|
||||
},
|
||||
});
|
||||
usage_context.report(batch).await;
|
||||
if response.answers_validated == Some(false) {
|
||||
ShadowOutcome::Failed {
|
||||
failure: AutoRouterFailure::InvalidAnswer,
|
||||
latency_ms,
|
||||
}
|
||||
} else {
|
||||
ShadowOutcome::Recommendation {
|
||||
route: derive_route(&response),
|
||||
latency_ms,
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -378,11 +513,11 @@ fn derive_route(response: &SystemOneResponse) -> Route {
|
||||
}
|
||||
|
||||
// Clearly chat, with a calibrated intent and a present, low tool-need signal.
|
||||
let intent_is_calibrated_chat = response.answers.get("intent").is_some_and(|intent| {
|
||||
intent.kind == "choice"
|
||||
&& intent.choice.as_deref() == Some("chat")
|
||||
&& unit_interval(intent.confidence).is_some_and(|c| c >= 0.5)
|
||||
});
|
||||
let intent_is_calibrated_chat = crate::model_routing::validated_choice(
|
||||
response.answers.get("intent"),
|
||||
&["code_change", "code_question", "command", "chat", "other"],
|
||||
)
|
||||
.is_some_and(|intent| intent.choice == "chat" && intent.confidence_bp >= 5_000);
|
||||
if intent_is_calibrated_chat && needs_tool.is_some_and(|nt| nt <= 0.2) {
|
||||
return Route::PlainChat;
|
||||
}
|
||||
@@ -477,6 +612,7 @@ mod tests {
|
||||
json!({
|
||||
"id": "sf-1",
|
||||
"model": "jev-latest",
|
||||
"usage": { "input_tokens": 121, "output_tokens": 8 },
|
||||
"answers": {
|
||||
"needs_tool": { "type": "noul", "noul": needs_tool },
|
||||
"answerable_from_context": { "type": "noul", "noul": from_context },
|
||||
@@ -498,7 +634,7 @@ mod tests {
|
||||
}
|
||||
|
||||
async fn run(messages: &[Message]) -> Option<ShadowOutcome> {
|
||||
let handle = spawn_shadow_gate(&config(), messages)?;
|
||||
let handle = spawn_shadow_gate(&config(), messages, None, &CancellationToken::new())?;
|
||||
Some(handle.await.expect("shadow task"))
|
||||
}
|
||||
|
||||
@@ -629,7 +765,13 @@ mod tests {
|
||||
let _env = hermetic_env(Some((&server, 100)));
|
||||
|
||||
let started = Instant::now();
|
||||
let handle = spawn_shadow_gate(&config(), &turn("Run the tests")).expect("task");
|
||||
let handle = spawn_shadow_gate(
|
||||
&config(),
|
||||
&turn("Run the tests"),
|
||||
None,
|
||||
&CancellationToken::new(),
|
||||
)
|
||||
.expect("task");
|
||||
assert!(
|
||||
started.elapsed() < Duration::from_millis(100),
|
||||
"spawning must not wait on the decision call"
|
||||
@@ -744,7 +886,8 @@ mod tests {
|
||||
(
|
||||
json!({
|
||||
"needs_tool": { "type": "noul", "noul": 0.05 },
|
||||
"intent": { "type": "choice", "choice": "chat", "confidence": 0.8 }
|
||||
"intent": { "type": "choice", "choice": "chat", "confidence": 0.8,
|
||||
"probabilities": { "chat": 0.9, "code_change": 0.04, "code_question": 0.03, "command": 0.02, "other": 0.01 } }
|
||||
}),
|
||||
Route::PlainChat,
|
||||
),
|
||||
@@ -784,4 +927,201 @@ mod tests {
|
||||
None
|
||||
);
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn cancelled_before_dispatch_sends_nothing() {
|
||||
let server = MockServer::start().await;
|
||||
let _env = hermetic_env(Some((&server, 2_000)));
|
||||
let cancel = CancellationToken::new();
|
||||
cancel.cancel();
|
||||
let outcome = spawn_shadow_gate(&config(), &turn("hi"), None, &cancel)
|
||||
.expect("task")
|
||||
.await
|
||||
.expect("joined");
|
||||
assert!(matches!(
|
||||
outcome,
|
||||
ShadowOutcome::Failed {
|
||||
failure: AutoRouterFailure::Cancelled,
|
||||
..
|
||||
}
|
||||
));
|
||||
assert!(requests(&server).await.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cancelled_dispatched_call_records_gap_for_its_owner() {
|
||||
let server = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/systemone"))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200)
|
||||
.set_delay(Duration::from_secs(2))
|
||||
.set_body_json(noul_body(0.9, 0.1)),
|
||||
)
|
||||
.expect(1)
|
||||
.mount(&server)
|
||||
.await;
|
||||
let _env = hermetic_env(Some((&server, 3_000)));
|
||||
let owner = "superfast-cancel-owner";
|
||||
let dropped = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
|
||||
let sink = dropped.clone();
|
||||
crate::cost_status::register_runtime_usage_sink_with_drop(
|
||||
owner,
|
||||
std::sync::Arc::new(|_| true),
|
||||
Some(std::sync::Arc::new(move |record| {
|
||||
sink.lock().expect("drop sink").push(record);
|
||||
true
|
||||
})),
|
||||
);
|
||||
|
||||
let cancel = CancellationToken::new();
|
||||
let handle = spawn_shadow_gate(&config(), &turn("hi"), Some(owner), &cancel).expect("task");
|
||||
tokio::time::timeout(Duration::from_secs(1), async {
|
||||
while requests(&server).await.is_empty() {
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("request admitted");
|
||||
let start = Instant::now();
|
||||
cancel.cancel();
|
||||
let outcome = tokio::time::timeout(Duration::from_millis(300), handle)
|
||||
.await
|
||||
.expect("cancellation stops the pending request")
|
||||
.expect("joined");
|
||||
assert!(matches!(
|
||||
outcome,
|
||||
ShadowOutcome::Failed {
|
||||
failure: AutoRouterFailure::Cancelled,
|
||||
..
|
||||
}
|
||||
));
|
||||
assert!(start.elapsed() < Duration::from_millis(300));
|
||||
let records = dropped.lock().expect("drop records");
|
||||
assert_eq!(records.len(), 1);
|
||||
assert_eq!(records[0].route.provider_identity, "typesafe");
|
||||
let batch = crate::cost_status::take_runtime_usage(owner);
|
||||
assert!(batch.records.is_empty());
|
||||
assert!(batch.drop_records.is_empty());
|
||||
crate::cost_status::finish_runtime_usage_owner(owner);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn owner_lease_retains_late_usage_and_invalid_answer_cost() {
|
||||
let server = MockServer::start().await;
|
||||
let mut body = noul_body(0.9, 0.1);
|
||||
body["answers"]["needs_tool"]["noul"] = 2.0.into();
|
||||
body["usage"]["cost"] = 0.000012054.into();
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/systemone"))
|
||||
.respond_with(
|
||||
ResponseTemplate::new(200)
|
||||
.set_delay(Duration::from_millis(100))
|
||||
.set_body_json(body),
|
||||
)
|
||||
.expect(1)
|
||||
.mount(&server)
|
||||
.await;
|
||||
let _env = hermetic_env(Some((&server, 2_000)));
|
||||
let owner = "superfast-late-owner";
|
||||
let observed = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
|
||||
let sink = observed.clone();
|
||||
crate::cost_status::register_runtime_usage_sink(
|
||||
owner,
|
||||
std::sync::Arc::new(move |record| {
|
||||
sink.lock().expect("sink").push(record);
|
||||
true
|
||||
}),
|
||||
);
|
||||
let decisions = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
|
||||
let decision_sink = decisions.clone();
|
||||
crate::cost_status::register_runtime_decision_sink(
|
||||
owner,
|
||||
std::sync::Arc::new(move |receipt| {
|
||||
decision_sink.lock().expect("decision sink").push(receipt);
|
||||
true
|
||||
}),
|
||||
);
|
||||
let handle = spawn_shadow_gate(
|
||||
&config(),
|
||||
&turn("hi"),
|
||||
Some(owner),
|
||||
&CancellationToken::new(),
|
||||
)
|
||||
.expect("task");
|
||||
crate::cost_status::finish_runtime_usage_owner(owner);
|
||||
let outcome = handle.await.expect("joined");
|
||||
assert!(matches!(
|
||||
outcome,
|
||||
ShadowOutcome::Failed {
|
||||
failure: AutoRouterFailure::InvalidAnswer,
|
||||
..
|
||||
}
|
||||
));
|
||||
let records = observed.lock().expect("records");
|
||||
assert_eq!(
|
||||
records.len(),
|
||||
1,
|
||||
"retired origin still owns its late response"
|
||||
);
|
||||
assert_eq!(records[0].usage.usage.input_tokens, 121);
|
||||
assert_eq!(records[0].usage.usage.output_tokens, 8);
|
||||
assert_eq!(records[0].usage.route.provider_identity, "typesafe");
|
||||
assert_eq!(
|
||||
records[0].usage.route.billing_mode,
|
||||
crate::cost_status::RouteBillingMode::Unknown
|
||||
);
|
||||
let receipts = decisions.lock().expect("decision receipts");
|
||||
assert_eq!(receipts.len(), 1);
|
||||
assert!(!receipts[0].valid_answers);
|
||||
assert!(receipts[0].shadow);
|
||||
assert_eq!(
|
||||
receipts[0].evidence.provider_reported_cost_usd.as_deref(),
|
||||
Some("0.000012054")
|
||||
);
|
||||
assert!(
|
||||
crate::cost_status::take_runtime_usage(owner)
|
||||
.records
|
||||
.is_empty(),
|
||||
"settled response must not enter fallback journal"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn oversized_decision_body_fails_closed_with_dispatched_coverage_gap() {
|
||||
let server = MockServer::start().await;
|
||||
Mock::given(method("POST"))
|
||||
.and(path("/v1/systemone"))
|
||||
.respond_with(ResponseTemplate::new(200).set_body_string("x".repeat(256 * 1024 + 1)))
|
||||
.expect(1)
|
||||
.mount(&server)
|
||||
.await;
|
||||
let _env = hermetic_env(Some((&server, 2_000)));
|
||||
let owner = "superfast-oversized-owner";
|
||||
let drops = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
|
||||
let sink = drops.clone();
|
||||
crate::cost_status::register_runtime_usage_sink_with_drop(
|
||||
owner,
|
||||
std::sync::Arc::new(|_| true),
|
||||
Some(std::sync::Arc::new(move |record| {
|
||||
sink.lock().expect("drops").push(record);
|
||||
true
|
||||
})),
|
||||
);
|
||||
let handle = spawn_shadow_gate(
|
||||
&config(),
|
||||
&turn("hi"),
|
||||
Some(owner),
|
||||
&CancellationToken::new(),
|
||||
)
|
||||
.expect("task");
|
||||
assert!(matches!(
|
||||
handle.await.expect("joined"),
|
||||
ShadowOutcome::Failed {
|
||||
failure: AutoRouterFailure::InvalidAnswer,
|
||||
..
|
||||
}
|
||||
));
|
||||
assert_eq!(drops.lock().expect("drop receipts").len(), 1);
|
||||
crate::cost_status::finish_runtime_usage_owner(owner);
|
||||
}
|
||||
}
|
||||
@@ -494,6 +494,7 @@ impl RlmTool {
|
||||
&error.to_string(),
|
||||
started.elapsed(),
|
||||
&crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: usage.records,
|
||||
drop_records: usage.drop_records,
|
||||
dropped_records: usage.dropped_records,
|
||||
@@ -504,6 +505,7 @@ impl RlmTool {
|
||||
(
|
||||
round,
|
||||
crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: usage.records,
|
||||
drop_records: usage.drop_records,
|
||||
dropped_records: usage.dropped_records,
|
||||
@@ -996,6 +998,7 @@ mod tests {
|
||||
"kernel stdout closed",
|
||||
Duration::from_millis(11),
|
||||
&crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: vec![record],
|
||||
drop_records: vec![drop_record],
|
||||
dropped_records: 1,
|
||||
|
||||
@@ -72,6 +72,7 @@ fn failed_engine_channel_settles_classifier_batch_once() {
|
||||
chrono::Utc::now(),
|
||||
);
|
||||
let batch = crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: vec![crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: "auto-router:dispatch-usage".to_string(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
@@ -21865,6 +21866,7 @@ fn routed_missing_usage_batch_prices_exact_routes_and_only_residual_as_generic()
|
||||
local_route.provider_identity = "local-computer".to_string();
|
||||
local_route.billing_mode = crate::cost_status::RouteBillingMode::Local;
|
||||
let batch = crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: Vec::new(),
|
||||
drop_records: vec![
|
||||
crate::cost_status::RuntimeUsageDropRecord {
|
||||
|
||||
@@ -312,6 +312,7 @@ async fn test_preset(app: &mut App, config: &Config, preset: RouterPreset) {
|
||||
cost_scope,
|
||||
None,
|
||||
&crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: selection.routed_usage.clone(),
|
||||
drop_records: selection.routed_usage_drop_records.clone(),
|
||||
dropped_records: selection.routed_usage_dropped_records,
|
||||
|
||||
@@ -186,6 +186,7 @@ pub(crate) async fn plan_turn_route(
|
||||
let initial_routed_usage = auto_selection
|
||||
.as_mut()
|
||||
.map(|selection| crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: std::mem::take(&mut selection.routed_usage),
|
||||
drop_records: std::mem::take(&mut selection.routed_usage_drop_records),
|
||||
dropped_records: std::mem::take(&mut selection.routed_usage_dropped_records),
|
||||
@@ -339,6 +340,7 @@ mod tests {
|
||||
chrono::Utc::now(),
|
||||
);
|
||||
let batch = crate::cost_status::RuntimeUsageBatch {
|
||||
decisions: Vec::new(),
|
||||
records: vec![crate::cost_status::RuntimeUsageRecord {
|
||||
source_id: "auto-router:plan-usage".to_string(),
|
||||
usage: crate::cost_status::EffectiveRouteUsage {
|
||||
|
||||
+17
-4
@@ -771,8 +771,13 @@ min_confidence = 0.5 # default 0.5, clamped to 0..1
|
||||
0.75, or the turn stays on the fast tier.
|
||||
- An unknown `kind`, or a decision `provider` other than `openrouter` /
|
||||
`typesafe`, leaves the router unconfigured and shown as failing.
|
||||
- `thinking` is ignored for decision routers. OpenRouter spend is recorded like
|
||||
any routed usage; TypeSafe-direct spend appears on the receipt only.
|
||||
- `thinking` is ignored for decision routers. Both routes settle tokens through
|
||||
the originating session's routed-usage ledger. TypeSafe is a named Custom route
|
||||
with unknown billing; its price is never borrowed from the active chat provider.
|
||||
Provider-reported cost remains verbatim on the decision receipt.
|
||||
- OpenRouter uses `POST /api/alpha/decisions`; TypeSafe uses `POST /v1/systemone`.
|
||||
The shared client validates Choice, Noul and Score against the offered questions
|
||||
and bounds responses to 256 KiB. Malformed answers fail open with usage retained.
|
||||
|
||||
#### Shadow Decision Gate (experimental, off by default)
|
||||
|
||||
@@ -800,8 +805,10 @@ SUPERFAST_TIMEOUT_MS=150 # 1..=10000, default 150
|
||||
- Only the latest user message is sent, truncated to 4,000 characters and
|
||||
redacted of configured secrets. The log (target `superfast`) carries the
|
||||
route, failure class and latency, never prompt text.
|
||||
- The gate's own spend is the configured endpoint's; it is not added to session
|
||||
cost totals or Auto receipts.
|
||||
- The gate retains the originating turn's accounting owner and cancellation.
|
||||
Its tokens settle through the shared ledger; missing usage or a cancelled/timed
|
||||
out request after dispatch creates an explicit coverage gap. Unknown TypeSafe
|
||||
pricing is recorded as unpriced rather than free.
|
||||
|
||||
The Decision Gate concept and reference implementation are by Andrea Bruno,
|
||||
released under CC BY 4.0:
|
||||
@@ -3480,3 +3487,9 @@ codewhale sessions scrub-secrets --apply # rewrite the affected files
|
||||
so it never loses a concurrent save. A session that is still open can write
|
||||
its in-memory copy back on its next save, so close open sessions first, and
|
||||
rotate any credential that was exposed — masking a stored copy cannot un-leak it.
|
||||
|
||||
Decision API compatibility uses OpenRouter `/api/alpha/decisions` and TypeSafe
|
||||
`/v1/systemone` on the existing authenticated client. Provider-reported cost is
|
||||
preserved verbatim in bounded decision receipts on the original turn or session,
|
||||
including rejected answers and late shadow responses. Receipts are diagnostic;
|
||||
they do not turn unknown pricing into an authoritative dollar total.
|
||||
Reference in new issue
Block a user