wip: complete Decisions compatibility and owned shadow settlement (#6604)

OpenRouter uses /api/alpha/decisions and TypeSafe /v1/systemone on the existing authenticated transport. Validate Choice, Noul and fractional Score against each request, retain provider usage on rejected policy answers, and bound request/response bodies. Preserve original contributor branch/authorship and off-by-default shadow behavior.

Detached shadow calls capture the existing turn/session usage owner and lease before dispatch, obey cancellation, and settle usage or missing-coverage evidence through the same runtime ledger. Bounded raw-cost decision receipts persist in that ledger, including late responses, with deletion admission and turn_mutation intact. Existing child metadata and TurnRecord constructors/decoders are migrated without another event authority.

Source checkpoint only: rustfmt and git diff --check passed. Focused Rust tests, npm test && npm run check:web, clippy, negative controls and fresh hosted exact-head Linux/macOS/Windows proof are pending. No Jev spend, native UI acceptance or release claim.

Signed-off-by: CodeWhale Bot <[email protected]>
This commit is contained in:
CodeWhale Bot committed 2026-09-30 03:03:33 -07:00
1 parent 346708429c
commit 3185d9ceaf
16 files changed
+1382 -121

No files matched your search

+244 -18
View File
@@ -2,9 +2,9 @@
//!
//! `[auto.router] kind = "decision"` asks a non-generative decision model
//! (TypeSafe's Jev) one typed Choice per turn. The wire is a plain JSON
//! `POST {base}/systemone` — not chat completions — served on two routes:
//! `POST to the route’s decision endpoint` — not chat completions — served on two routes:
//!
//! * **OpenRouter** — `{openrouter base}/systemone` with the user's OpenRouter
//! * **OpenRouter** — `https://openrouter.ai/api/alpha/decisions` with the user's OpenRouter
//! key, built exactly like every other OpenRouter client.
//! * **TypeSafe direct** — `https://api.typesafe.ai/v1/systemone` with a
//! TypeSafe key.
@@ -16,13 +16,15 @@
//! * TypeSafe is **not** an [`ApiProvider`]: it serves no chat route, so it is
//! the decision router's own endpoint + key (`TYPESAFE_API_KEY`, the
//! `typesafe` secret-store slot, or `[providers.typesafe] api_key` /
//! `api_key_env`). Its spend is shown on the Auto receipt as the
//! provider-reported cost; it is not entered in session cost totals.
//! `api_key_env`). Its tokens enter the shared usage ledger under a frozen
//! `custom` / `typesafe` route with unknown billing; reported cost is retained
//! on decision receipts. Unknown pricing is never interpreted as free.
//! * The routing call makes one attempt (no retry): it is bounded by the
//! router timeout, and a retried decision would arrive after the turn has
//! already fallen back.
//! * The router parses the `choice` answer shape; the shadow Decision Gate
//! (`crate::superfast`) also reads `noul`. `score` is not parsed.
//! (`crate::superfast`) also reads `noul`. The transport strictly validates
//! Choice, Noul and fractional Score responses before either policy uses them.
use std::collections::BTreeMap;
@@ -38,6 +40,9 @@ pub(crate) const TYPESAFE_DEFAULT_BASE_URL: &str = "https://api.typesafe.ai/v1";
pub(crate) const TYPESAFE_API_KEY_ENV: &str = "TYPESAFE_API_KEY";
/// Secret-store slot and `[providers.<name>]` table name for the TypeSafe key.
pub(crate) const TYPESAFE_KEY_NAME: &str = "typesafe";
/// Bound a non-generative decision response before allocating/decoding it.
const DECISION_RESPONSE_MAX_BYTES: usize = 256 * 1024;
const DECISION_REQUEST_MAX_BYTES: usize = 1024 * 1024;
/// Which endpoint serves a decision router.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
@@ -128,6 +133,10 @@ pub(crate) struct SystemOneResponse {
pub(crate) answers: BTreeMap<String, SystemOneAnswer>,
#[serde(default)]
pub(crate) usage: Option<SystemOneUsage>,
/// Transport validation is separate from decoding so a rejected policy
/// answer still preserves the provider's usage/cost evidence.
#[serde(skip)]
pub(crate) answers_validated: Option<bool>,
}
/// One answer. The `choice` and `noul` subsets are interpreted.
@@ -141,6 +150,10 @@ pub(crate) struct SystemOneAnswer {
#[serde(default)]
pub(crate) noul: Option<f64>,
#[serde(default)]
pub(crate) score: Option<f64>,
#[serde(default)]
pub(crate) legend: BTreeMap<String, Value>,
#[serde(default)]
pub(crate) probabilities: BTreeMap<String, Option<f64>>,
#[serde(default)]
pub(crate) confidence: Option<f64>,
@@ -166,7 +179,7 @@ impl SystemOneUsage {
pub(crate) fn reported_cost(&self) -> Option<String> {
let raw = self.cost.as_ref()?.get().trim();
let value: f64 = raw.parse().ok()?;
(value.is_finite() && value >= 0.0).then(|| raw.to_string())
(raw.len() <= 128 && value.is_finite() && value >= 0.0).then(|| raw.to_string())
}
}
@@ -176,8 +189,8 @@ impl CodewhaleClient {
/// OpenRouter is the ordinary OpenRouter client (its key, base URL,
/// attribution headers). TypeSafe re-points a clone of the active route's
/// client — keeping its retry, TLS and redaction policy — at the TypeSafe
/// endpoint with the TypeSafe key only, and drops the active provider's
/// concurrency permit so a routing call never holds a chat slot.
/// endpoint with the TypeSafe key only. Its captured request budget and
/// remote-control ownership continue to apply to the one decision attempt.
pub(crate) fn for_decision_route(
config: &Config,
route: DecisionRouterRoute,
@@ -216,6 +229,15 @@ impl CodewhaleClient {
)?
.build()?;
client.base_url = base_url;
client.http1_client = client.http_client.clone();
// Repointing auth/URL must also replace the inherited chat
// route identity. No TypeSafe price/product is guessed.
client.api_provider = ApiProvider::Custom;
client.provider_identity = TYPESAFE_KEY_NAME.to_string();
client.openrouter_vendor = None;
client.billing_surface = None;
client.billing_mode = crate::cost_status::RouteBillingMode::Unknown;
client.route_limits = None;
// `Self::new` froze the redaction set from the chat provider's
// secrets; the TypeSafe key is none of them, so add it before
// any decision body is built from untrusted context.
@@ -223,15 +245,13 @@ impl CodewhaleClient {
push_model_bound_secret(&mut secrets, Some(&key));
client.model_bound_secret_values = Arc::new(secrets);
client.api_key = key;
client.request_concurrency = None;
client.remote_control_inference_participant = false;
Ok(client)
}
}
}
/// `POST {base}/systemone` once, isolated like the Auto chat classifier:
/// no global retry banners, no shared token bucket, no response cache.
/// `POST to the route’s decision endpoint` once, isolated like the Auto chat classifier:
/// no global retry banners or response cache; shared admission still applies.
///
/// Only a failure class leaves this function — provider error bodies can
/// echo the prompt and must never reach receipts.
@@ -244,26 +264,147 @@ impl CodewhaleClient {
body: &Value,
dispatched: &std::sync::atomic::AtomicBool,
) -> std::result::Result<SystemOneResponse, AutoRouterFailure> {
if serde_json::to_vec(body).map_or(true, |bytes| bytes.len() > DECISION_REQUEST_MAX_BYTES)
|| !valid_decision_request(body)
{
return Err(AutoRouterFailure::NotRunnable);
}
let mut isolated = self.clone();
isolated.isolated_request_state = true;
isolated.rate_limiter = Arc::new(AsyncMutex::new(TokenBucket::from_env()));
isolated.retry.max_retries = 0;
let _inference = isolated.acquire_remote_control_inference_permit().await;
let _permit = isolated.acquire_provider_request_permit().await;
let url = api_url(&isolated.base_url, "systemone");
let url = if isolated.api_provider == ApiProvider::Openrouter {
// The OpenRouter Decisions API is a sibling of /api/v1, so keep
// the configured origin/proxy prefix and replace only /v1.
let mut url = reqwest::Url::parse(&isolated.base_url)
.map_err(|_| AutoRouterFailure::NotRunnable)?;
let prefix = url.path().trim_end_matches('/').trim_end_matches("/v1");
url.set_path(&format!("{prefix}/alpha/decisions"));
url.to_string()
} else {
api_url(&isolated.base_url, "systemone")
};
isolated.wait_for_rate_limit().await;
dispatched.store(true, std::sync::atomic::Ordering::Release);
let response = isolated
.send_json_with_retry(&url, body)
.await
.map_err(|error| router_failure_from_error(&error))?;
let bytes = response
.bytes()
let text = bounded_provider_catalog_text(response, DECISION_RESPONSE_MAX_BYTES)
.await
.map_err(|_| AutoRouterFailure::Transport)?;
serde_json::from_slice(&bytes).map_err(|_| AutoRouterFailure::InvalidAnswer)
.map_err(|error| match error {
CatalogRefreshError::Network => AutoRouterFailure::Transport,
_ => AutoRouterFailure::InvalidAnswer,
})?;
let mut response: SystemOneResponse =
serde_json::from_str(&text).map_err(|_| AutoRouterFailure::InvalidAnswer)?;
response.answers_validated = Some(valid_decision_response(body, &response));
response.model = response.model.map(|model| {
self.redact_model_bound_text(&model)
.chars()
.take(128)
.collect()
});
Ok(response)
}
}
fn valid_decision_request(body: &Value) -> bool {
let Some(questions) = body.get("questions").and_then(Value::as_object) else {
return false;
};
let model = body.get("model").and_then(Value::as_str);
model.is_some_and(|m| !m.trim().is_empty() && m.len() <= 256)
&& body.get("state").is_some_and(|state| {
matches!(state, Value::String(_) | Value::Object(_) | Value::Array(_))
})
&& !questions.is_empty()
&& questions.len() <= 64
&& questions
.values()
.all(|q| match q.get("type").and_then(Value::as_str) {
Some("noul") => true,
Some("choice") => q
.get("criteria")
.and_then(Value::as_object)
.is_some_and(|c| !c.is_empty() && c.len() <= 64),
Some("score") => q
.get("criteria")
.and_then(Value::as_array)
.is_some_and(|c| !c.is_empty() && c.len() <= 10),
_ => false,
})
}
fn valid_decision_response(body: &Value, response: &SystemOneResponse) -> bool {
let Some(questions) = body.get("questions").and_then(Value::as_object) else {
return false;
};
response
.model
.as_deref()
.is_some_and(|m| !m.trim().is_empty() && m.len() <= 256)
&& response.usage.is_some()
&& response.answers.len() == questions.len()
&& questions.iter().all(|(name, question)| {
let Some(answer) = response.answers.get(name) else {
return false;
};
if Some(answer.kind.as_str()) != question.get("type").and_then(Value::as_str) {
return false;
}
match answer.kind.as_str() {
"noul" => answer
.noul
.is_some_and(|v| v.is_finite() && (0.0..=1.0).contains(&v)),
"choice" => {
let Some(criteria) = question.get("criteria").and_then(Value::as_object) else {
return false;
};
let options = criteria.keys().map(String::as_str).collect::<Vec<_>>();
crate::model_routing::validated_choice(Some(answer), &options).is_some()
}
"score" => {
let Some(criteria) = question.get("criteria").and_then(Value::as_array) else {
return false;
};
let Some(score) = answer.score else {
return false;
};
if !score.is_finite()
|| !(0.0..=(criteria.len() - 1) as f64).contains(&score)
|| !answer
.confidence
.is_some_and(|v| v.is_finite() && (0.0..=1.0).contains(&v))
|| answer.legend.len() != criteria.len()
|| answer.probabilities.len() != criteria.len()
{
return false;
}
let mut sum = 0.0;
let mut expected_score = 0.0;
for (level, criterion) in criteria.iter().enumerate() {
let key = level.to_string();
let Some(Some(probability)) = answer.probabilities.get(&key) else {
return false;
};
if answer.legend.get(&key) != Some(criterion)
|| !probability.is_finite()
|| !(0.0..=1.0).contains(probability)
{
return false;
}
sum += probability;
expected_score += level as f64 * probability;
}
(sum - 1.0).abs() <= 0.02 && (score - expected_score).abs() <= 0.02
}
_ => false,
}
})
}
/// Collapse a client error into a non-secret failure class. The HTTP status
/// is kept where the error carries it; the body never is.
pub(crate) fn router_failure_from_error(error: &anyhow::Error) -> AutoRouterFailure {
@@ -293,3 +434,88 @@ pub(crate) fn router_failure_from_error(error: &anyhow::Error) -> AutoRouterFail
}
}
}
#[cfg(test)]
mod decisions_compatibility_tests {
use super::*;
fn request() -> Value {
json!({"model":"typesafe/jev-1.13", "state":{"ticket":"charged twice"}, "questions": {
"intent": {"type":"choice", "criteria":{"billing":"money", "other":"anything else"}},
"refund": {"type":"noul", "instructions":"Does it ask for a refund?"},
"urgency": {"type":"score", "criteria":["Can wait", "Needs attention this week", "Needs attention today"]}
}})
}
fn response() -> Value {
json!({"id":"fixture-decision", "model":"typesafe/jev-1.13-20260917", "usage":{"input_tokens":287,"output_tokens":20,"cost":0.000012054}, "answers": {
"intent":{"type":"choice","choice":"billing","confidence":0.9,"probabilities":{"billing":0.9,"other":0.1}},
"refund":{"type":"noul","noul":0.99},
"urgency":{"type":"score","score":1.7,"confidence":0.9,"legend":{"0":"Can wait","1":"Needs attention this week","2":"Needs attention today"},"probabilities":{"0":0.1,"1":0.1,"2":0.8}}
}})
}
#[test]
fn all_documented_primitives_validate_without_rescaling_score() {
let decoded: SystemOneResponse =
serde_json::from_value(response()).expect("documented fixture");
assert!(valid_decision_request(&request()));
assert!(valid_decision_response(&request(), &decoded));
assert_eq!(decoded.answers["urgency"].score, Some(1.7));
assert_eq!(
decoded.usage.expect("usage").reported_cost().as_deref(),
Some("0.000012054")
);
}
#[test]
fn wrong_types_unoffered_choices_scores_and_partial_shapes_are_rejected() {
let changes = [
("/answers/refund/noul", json!(1.01)),
("/answers/refund/type", json!("score")),
("/answers/intent/choice", json!("not-offered")),
("/answers/intent/choice", json!("other")),
("/answers/intent/confidence", json!(-0.1)),
("/answers/intent/probabilities", json!({"billing":0.9})),
(
"/answers/intent/probabilities",
json!({"billing":0.6,"other":0.6}),
),
("/answers/urgency/score", json!(2.01)),
("/answers/urgency/score", json!(0.5)),
("/answers/urgency/legend/1", json!("different rubric")),
("/answers/urgency/probabilities/1", Value::Null),
("/answers/urgency/confidence", json!(1.1)),
("/model", Value::Null),
("/usage", Value::Null),
("/answers", json!({})),
];
for (pointer, value) in changes {
let mut body = response();
*body.pointer_mut(pointer).expect("fixture field") = value;
let decoded: SystemOneResponse =
serde_json::from_value(body).expect("structural decode");
assert!(
!valid_decision_response(&request(), &decoded),
"accepted {pointer}"
);
}
}
#[test]
fn non_finite_in_memory_answers_and_unbounded_request_shapes_are_rejected() {
let mut decoded: SystemOneResponse = serde_json::from_value(response()).expect("fixture");
decoded.answers.get_mut("refund").expect("noul").noul = Some(f64::NAN);
assert!(!valid_decision_response(&request(), &decoded));
let mut decoded: SystemOneResponse = serde_json::from_value(response()).expect("fixture");
decoded.answers.get_mut("urgency").expect("score").score = Some(f64::INFINITY);
assert!(!valid_decision_response(&request(), &decoded));
for shape in [
json!({}),
json!({"model":"jev", "state":"hello", "questions":{}}),
json!({"model":"jev", "state":"hello", "questions":{"q":{"type":"score","criteria":vec!["level";11]}}}),
] {
assert!(!valid_decision_request(&shape));
}
}
}
+8
View File
@@ -5562,6 +5562,14 @@ impl Engine {
);
turn.add_routed_usage_dropped_records(residual_dropped_records);
let initial_cost_scope = crate::cost_status::scope_token();
crate::cost_status::report_runtime_usage_batch(
initial_cost_scope,
initial_usage_owner.as_deref(),
&crate::cost_status::RuntimeUsageBatch {
decisions: initial_routed_usage.decisions.clone(),
..Default::default()
},
);
for record in &initial_routed_usage.records {
crate::cost_status::report_effective_route_for_runtime(
initial_cost_scope,
+1
View File
@@ -8520,6 +8520,7 @@ async fn initial_routed_usage_is_total_only_emitted_once_and_keeps_parent_route_
);
missing_usage_route.billing_mode = crate::cost_status::RouteBillingMode::Metered;
**initial_routed_usage = crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
records: vec![crate::cost_status::RuntimeUsageRecord {
source_id: "auto-router:engine-fixture".to_string(),
usage: crate::cost_status::EffectiveRouteUsage {
+2
View File
@@ -1825,6 +1825,8 @@ impl Engine {
drop(crate::superfast::spawn_shadow_gate(
&self.api_config,
&stream_request.messages,
self.config.compaction.runtime_cost_owner.as_deref(),
&self.cancel_token,
));
}
let _ = self
+289 -5
View File
@@ -488,6 +488,7 @@ pub fn attach_child_usage_metadata(
pub const MAX_CHILD_USAGE_RECORDS: usize = 64;
const CHILD_USAGE_RECORDS_KEY: &str = "child_usage_records";
const CHILD_DECISION_RECEIPTS_KEY: &str = "child_decision_receipts";
const CHILD_USAGE_DROP_RECORDS_KEY: &str = "child_usage_drop_records";
const CHILD_USAGE_DROPPED_RECORDS_KEY: &str = "child_usage_dropped_records";
@@ -506,6 +507,18 @@ pub fn attach_child_usage_batch_metadata(
let Some(object) = metadata.as_object_mut() else {
return;
};
object.insert(
CHILD_DECISION_RECEIPTS_KEY.into(),
serde_json::json!(
batch
.decisions
.iter()
.filter(|receipt| receipt.is_bounded())
.take(MAX_CHILD_USAGE_RECORDS)
.map(RuntimeDecisionReceipt::sanitized)
.collect::<Vec<_>>()
),
);
let retained_records = batch
.records
.iter()
@@ -545,10 +558,23 @@ pub fn attach_child_usage_batch_metadata(
serde_json::json!(retained_drops),
);
let usage_overflow = batch.records.len().saturating_sub(MAX_CHILD_USAGE_RECORDS);
let decision_overflow = batch
.decisions
.len()
.saturating_sub(MAX_CHILD_USAGE_RECORDS)
.saturating_add(
batch
.decisions
.iter()
.take(MAX_CHILD_USAGE_RECORDS)
.filter(|r| !r.is_bounded())
.count(),
);
let dropped_records = batch
.dropped_records
.max(u64::try_from(batch.drop_records.len()).unwrap_or(u64::MAX))
.saturating_add(u64::try_from(usage_overflow).unwrap_or(u64::MAX));
.saturating_add(u64::try_from(usage_overflow).unwrap_or(u64::MAX))
.saturating_add(u64::try_from(decision_overflow).unwrap_or(u64::MAX));
if dropped_records > 0 {
object.insert(
CHILD_USAGE_DROPPED_RECORDS_KEY.into(),
@@ -581,6 +607,7 @@ pub fn child_usage_records_from_metadata(
.unwrap_or(0);
let Some(values) = value.as_array() else {
return Some(RuntimeUsageBatch {
decisions: Vec::new(),
records: Vec::new(),
drop_records: Vec::new(),
dropped_records: declared_dropped.saturating_add(1),
@@ -589,6 +616,7 @@ pub fn child_usage_records_from_metadata(
let overflow = values.len().saturating_sub(MAX_CHILD_USAGE_RECORDS);
let mut batch = RuntimeUsageBatch {
decisions: Vec::new(),
records: Vec::with_capacity(values.len().min(MAX_CHILD_USAGE_RECORDS)),
drop_records: Vec::with_capacity(drop_values.len().min(MAX_CHILD_USAGE_RECORDS)),
dropped_records: declared_dropped
@@ -636,6 +664,27 @@ pub fn child_usage_records_from_metadata(
// Every declared drop slot already contributes to dropped_records,
// including malformed entries; do not count the same gap twice.
}
if let Some(decisions) = metadata.get(CHILD_DECISION_RECEIPTS_KEY) {
if let Some(values) = decisions.as_array() {
for value in values.iter().take(MAX_CHILD_USAGE_RECORDS) {
let bounded = serde_json::to_vec(value).is_ok_and(|bytes| bytes.len() <= 16 * 1024);
let parsed = bounded
.then(|| serde_json::from_value::<RuntimeDecisionReceipt>(value.clone()).ok())
.flatten();
if let Some(receipt) = parsed.filter(RuntimeDecisionReceipt::is_bounded) {
batch.decisions.push(receipt.sanitized());
} else {
batch.dropped_records = batch.dropped_records.saturating_add(1);
}
}
batch.dropped_records = batch.dropped_records.saturating_add(
u64::try_from(values.len().saturating_sub(MAX_CHILD_USAGE_RECORDS))
.unwrap_or(u64::MAX),
);
} else {
batch.dropped_records = batch.dropped_records.saturating_add(1);
}
}
Some(batch)
}
@@ -794,7 +843,7 @@ fn with_pending_state_mut<R>(f: impl FnOnce(&mut ScopedPendingBackgroundCost) ->
.get_or_init(|| Mutex::new(std::collections::HashMap::new()))
.lock()
.unwrap_or_else(|e| e.into_inner());
f(by_thread.entry(std::thread::current().id()).or_default())
f(by_thread.entry(test_cost_scope_id()).or_default())
}
}
@@ -807,6 +856,7 @@ const MAX_RUNTIME_USAGE_RECORDS_PER_OWNER: usize = 64;
#[derive(Default)]
struct OwnerRuntimeUsageJournal {
records: VecDeque<RuntimeUsageRecord>,
decisions: VecDeque<RuntimeDecisionReceipt>,
drop_records: VecDeque<RuntimeUsageDropRecord>,
dropped_records: u64,
dropped_source_fingerprints: HashSet<String>,
@@ -820,6 +870,8 @@ type RuntimeUsageJournal = HashMap<String, OwnerRuntimeUsageJournal>;
/// closed instead of silently presenting a partial cost as complete.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct RuntimeUsageBatch {
/// Bounded decision evidence, owned and retired with the same usage ledger.
pub decisions: Vec<RuntimeDecisionReceipt>,
pub records: Vec<RuntimeUsageRecord>,
/// Exact provider-success calls whose usage payload was absent. The
/// bounded records retain route billing truth; `dropped_records` remains
@@ -846,16 +898,118 @@ pub struct RuntimeUsageDropRecord {
pub route: EffectiveRouteEnvelope,
}
/// Provider decision evidence attached to the existing origin-turn ledger.
/// It is diagnostic evidence, never an instruction to change the model route.
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub struct RuntimeDecisionReceipt {
pub source_id: String,
pub route: EffectiveRouteEnvelope,
pub usage: Option<Usage>,
pub shadow: bool,
pub valid_answers: bool,
pub evidence: crate::model_routing::AutoRouteDecisionEvidence,
}
impl RuntimeDecisionReceipt {
pub(crate) fn is_bounded(&self) -> bool {
self.source_id.len() <= 128
&& self.evidence.choice.len() <= 128
&& self.evidence.probabilities_bp.len() <= 64
&& self
.evidence
.probabilities_bp
.iter()
.all(|(key, value)| key.len() <= 128 && *value <= 10_000)
&& self.evidence.confidence_bp <= 10_000
&& self.evidence.min_confidence_bp <= 10_000
&& self
.evidence
.thinking
.as_ref()
.is_none_or(|v| v.len() <= 128)
&& self
.evidence
.response_model
.as_ref()
.is_none_or(|v| v.len() <= 128)
&& self
.evidence
.provider_reported_cost_usd
.as_ref()
.is_none_or(|v| {
v.len() <= 128
&& v.parse::<f64>()
.is_ok_and(|cost| cost.is_finite() && cost >= 0.0)
})
&& serde_json::to_vec(self).is_ok_and(|bytes| bytes.len() <= 16 * 1024)
}
pub(crate) fn sanitized(&self) -> Self {
Self {
source_id: usage_source_fingerprint(&self.source_id),
route: self.route.sanitized_for_persistence(),
..self.clone()
}
}
pub(crate) fn diagnostic_receipt(&self) -> String {
// Reuse cost_status's persisted diagnostic receipt surface. No prompt,
// response id, URL, auth header or unoffered choice enters this string.
format!(
"decision:{}",
serde_json::to_string(&self.sanitized()).unwrap_or_default()
)
}
}
pub(crate) type RuntimeDecisionSink = Arc<dyn Fn(RuntimeDecisionReceipt) -> bool + Send + Sync>;
pub(crate) type RuntimeUsageSink = Arc<dyn Fn(RuntimeUsageRecord) -> bool + Send + Sync>;
pub(crate) type RuntimeUsageDropSink = Arc<dyn Fn(RuntimeUsageDropRecord) -> bool + Send + Sync>;
struct RuntimeUsageSinkEntry {
sink: RuntimeUsageSink,
dropped_sink: Option<RuntimeUsageDropSink>,
decision_sink: Option<RuntimeDecisionSink>,
leases: usize,
terminal: bool,
}
#[cfg(test)]
pub(crate) fn decision_receipt_fixture(source_id: &str) -> RuntimeDecisionReceipt {
let mut route = EffectiveRouteEnvelope::capture(
None,
crate::config::ApiProvider::Custom,
"typesafe",
"jev-latest",
Some("https://api.typesafe.ai/v1"),
Utc::now(),
);
route.billing_mode = RouteBillingMode::Unknown;
RuntimeDecisionReceipt {
source_id: source_id.to_string(),
route,
usage: Some(Usage {
input_tokens: 9,
output_tokens: 4,
..Default::default()
}),
shadow: true,
valid_answers: false,
evidence: crate::model_routing::AutoRouteDecisionEvidence {
choice: "invalid".to_string(),
probabilities_bp: Default::default(),
confidence_bp: 0,
min_confidence_bp: 5_000,
cost_saving_kept_fast: false,
thinking: None,
provider_reported_cost_usd: Some("0.000012054".to_string()),
latency_ms: 12,
response_model: Some("jev-latest".to_string()),
},
}
}
/// Keeps an owner sink alive while a detached child can still report usage.
/// The runtime turn may already be terminal; the last child release retires
/// the sink only after its final provider response has been durably appended.
@@ -908,7 +1062,7 @@ fn with_runtime_usage_sinks<R>(
.get_or_init(|| Mutex::new(HashMap::new()))
.lock()
.unwrap_or_else(|error| error.into_inner());
f(by_thread.entry(std::thread::current().id()).or_default())
f(by_thread.entry(test_cost_scope_id()).or_default())
}
}
@@ -928,7 +1082,7 @@ fn with_existing_runtime_usage_sinks<R>(
{
let by_thread = TEST_RUNTIME_USAGE_SINKS.get()?;
let mut by_thread = by_thread.lock().unwrap_or_else(|error| error.into_inner());
let sinks = by_thread.get_mut(&std::thread::current().id())?;
let sinks = by_thread.get_mut(&test_cost_scope_id())?;
Some(f(sinks))
}
}
@@ -948,7 +1102,7 @@ fn with_runtime_usage_journal_mut<R>(f: impl FnOnce(&mut RuntimeUsageJournal) ->
.get_or_init(|| Mutex::new(std::collections::HashMap::new()))
.lock()
.unwrap_or_else(|error| error.into_inner());
f(by_thread.entry(std::thread::current().id()).or_default())
f(by_thread.entry(test_cost_scope_id()).or_default())
}
}
@@ -1079,6 +1233,7 @@ pub(crate) fn register_runtime_usage_sink_with_drop(
RuntimeUsageSinkEntry {
sink,
dropped_sink,
decision_sink: None,
leases: 0,
terminal: false,
},
@@ -1086,6 +1241,74 @@ pub(crate) fn register_runtime_usage_sink_with_drop(
});
}
/// Extend the existing owner entry; this is not another sink registry.
pub(crate) fn register_runtime_decision_sink(owner: &str, sink: RuntimeDecisionSink) {
with_runtime_usage_sinks(|sinks| {
if let Some(entry) = sinks.get_mut(owner.trim()) {
entry.decision_sink = Some(sink);
}
});
}
fn record_runtime_decision(owner: &str, record: &RuntimeDecisionReceipt) {
if !record.is_bounded() {
record_runtime_usage_drop(owner, &record.source_id, &record.route);
return;
}
let record = record.sanitized();
let sink = with_runtime_usage_sinks(|sinks| {
sinks
.get(owner)
.and_then(|entry| entry.decision_sink.clone())
});
if sink.is_some_and(|sink| sink(record.clone())) {
return;
}
with_runtime_usage_journal_mut(|journal| {
let entry = journal.entry(owner.to_string()).or_default();
if entry
.decisions
.iter()
.any(|old| old.source_id == record.source_id)
{
return;
}
if entry.decisions.len() == MAX_RUNTIME_USAGE_RECORDS_PER_OWNER {
entry.decisions.pop_front();
entry.dropped_records = entry.dropped_records.saturating_add(1);
}
entry.decisions.push_back(record);
});
}
fn report_interactive_decision(scope: CostScopeToken, receipt: &RuntimeDecisionReceipt) {
if !receipt.is_bounded() {
return;
}
with_pending_state_mut(|state| {
if state.generation == scope.0 {
if state
.pending
.route_receipts
.iter()
.filter(|v| v.starts_with("decision:"))
.count()
< MAX_RUNTIME_USAGE_RECORDS_PER_OWNER
{
state
.pending
.route_receipts
.insert(receipt.diagnostic_receipt());
} else {
state
.pending
.route_receipts
.insert("decision:diagnostic_receipt_bound_reached".to_string());
}
}
});
}
/// Redacted durable identity shared by runtime-turn, worker, and interactive
/// session accounting. Raw response ids never need to be persisted merely to
/// make replay idempotent.
@@ -1151,6 +1374,9 @@ fn register_persistent_interactive_runtime_usage_sink_at(
let drop_session_id = usage_session_id.clone();
let drop_turn_id = usage_turn_id.clone();
let usage_sessions_dir = sessions_dir.clone();
let decision_sessions_dir = sessions_dir.clone();
let decision_session_id = session_id.to_string();
let decision_turn_id = turn_id.to_string();
register_runtime_usage_sink_with_drop(
owner,
Arc::new(move |record| {
@@ -1183,6 +1409,28 @@ fn register_persistent_interactive_runtime_usage_sink_at(
.unwrap_or(false)
})),
);
register_runtime_decision_sink(
owner,
Arc::new(move |receipt| {
let Ok(manager) =
crate::session_manager::SessionManager::new(decision_sessions_dir.clone())
else {
return false;
};
let accepted = manager
.persist_late_decision_receipt(&decision_session_id, &decision_turn_id, &receipt)
.unwrap_or(false);
if accepted {
// The append may have been handled by a deletion tombstone. Only
// a live origin may enter the foreground diagnostic projection.
let _ = manager.with_live_session_origin(&decision_session_id, || {
report_interactive_decision(scope, &receipt);
true
});
}
accepted
}),
);
}
#[cfg(test)]
@@ -1288,6 +1536,7 @@ pub fn take_runtime_usage(owner: &str) -> RuntimeUsageBatch {
journal
.remove(owner)
.map_or_else(RuntimeUsageBatch::default, |entry| RuntimeUsageBatch {
decisions: entry.decisions.into_iter().collect(),
records: entry.records.into_iter().collect(),
drop_records: entry.drop_records.into_iter().collect(),
dropped_records: entry.dropped_records,
@@ -1295,6 +1544,33 @@ pub fn take_runtime_usage(owner: &str) -> RuntimeUsageBatch {
})
}
#[cfg(test)]
thread_local! {
static TEST_COST_ORIGIN: std::cell::Cell<Option<std::thread::ThreadId>> = const { std::cell::Cell::new(None) };
}
#[cfg(test)]
pub(crate) fn test_cost_scope_id() -> std::thread::ThreadId {
TEST_COST_ORIGIN
.with(|scope| scope.get())
.unwrap_or_else(|| std::thread::current().id())
}
#[cfg(test)]
pub(crate) struct TestCostScopeBinding(Option<std::thread::ThreadId>);
#[cfg(test)]
pub(crate) fn bind_test_cost_scope(origin: std::thread::ThreadId) -> TestCostScopeBinding {
TestCostScopeBinding(TEST_COST_ORIGIN.with(|scope| scope.replace(Some(origin))))
}
#[cfg(test)]
impl Drop for TestCostScopeBinding {
fn drop(&mut self) {
TEST_COST_ORIGIN.with(|scope| scope.set(self.0));
}
}
/// Capture the current session/run generation before starting a background
/// provider request. The same token must be supplied when its usage returns.
#[must_use]
@@ -1845,6 +2121,13 @@ pub(crate) fn report_runtime_usage_batch(
runtime_owner: Option<&str>,
batch: &RuntimeUsageBatch,
) {
for receipt in &batch.decisions {
if let Some(owner) = runtime_owner {
record_runtime_decision(owner, receipt);
} else {
report_interactive_decision(scope, receipt);
}
}
for record in &batch.records {
report_effective_route_for_runtime(
scope,
@@ -3189,6 +3472,7 @@ mod tests {
attach_child_usage_batch_metadata(
&mut metadata,
&RuntimeUsageBatch {
decisions: Vec::new(),
records,
drop_records: Vec::new(),
dropped_records: 0,
+122 -58
View File
@@ -443,7 +443,7 @@ pub(crate) struct AutoRouteReceipt {
/// What a decision model answered and what it cost. Probabilities are basis
/// points (0..=10000) so the receipt stays `Eq` and never re-renders floats.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub(crate) struct AutoRouteDecisionEvidence {
pub struct AutoRouteDecisionEvidence {
/// The tier the decision model chose (`fast` | `strong`).
pub(crate) choice: String,
pub(crate) probabilities_bp: BTreeMap<String, u16>,
@@ -473,6 +473,7 @@ pub(crate) enum AutoRouterFailure {
/// Declared but unusable: missing key, unknown kind, or client setup.
NotRunnable,
Timeout,
Cancelled,
Http {
status: u16,
},
@@ -492,6 +493,7 @@ impl AutoRouterFailure {
"not runnable (check the router key and [auto.router])".to_string()
}
Self::Timeout => "timed out".to_string(),
Self::Cancelled => "cancelled".to_string(),
Self::Http { status } => format!("HTTP {status}"),
Self::QuotaExhausted => "provider quota or credits exhausted".to_string(),
Self::Rejected => "request rejected by the provider".to_string(),
@@ -1223,7 +1225,10 @@ fn auto_route_usage_has_reported_data(usage: &codewhale_models::Usage) -> bool {
/// Stable, persistence-safe identity for one classifier response. The raw
/// provider response id is hashed with the frozen dispatch route and instant;
/// neither it nor any custom route label crosses into telemetry/persistence.
fn auto_route_usage_source_id(route: &EffectiveRouteEnvelope, response_id: &str) -> String {
pub(crate) fn auto_route_usage_source_id(
route: &EffectiveRouteEnvelope,
response_id: &str,
) -> String {
use sha2::{Digest as _, Sha256};
let mut digest = Sha256::new();
@@ -1556,11 +1561,10 @@ async fn auto_route_decision_recommendation(
session_mode,
selected_thinking_mode,
);
// OpenRouter spend enters the session through the ordinary routed-usage
// path. TypeSafe is not a chat provider, so its spend is shown on the
// receipt only (see `client::system_one`).
let request_route = (route == DecisionRouterRoute::Openrouter)
.then(|| client.effective_route_envelope(&inventory.router_model, chrono::Utc::now()));
// Both transports enter the existing usage ledger. TypeSafe carries its
// own Custom/Unknown identity; no chat price or zero-cost claim is inherited.
let request_route =
Some(client.effective_route_envelope(&inventory.router_model, chrono::Utc::now()));
let started = Instant::now();
let dispatched = std::sync::atomic::AtomicBool::new(false);
let outcome = tokio::time::timeout(
@@ -1595,15 +1599,18 @@ async fn auto_route_decision_recommendation(
}
/// A validated `choice` answer.
struct ValidChoice {
choice: String,
probabilities_bp: BTreeMap<String, u16>,
confidence_bp: u16,
pub(crate) struct ValidChoice {
pub(crate) choice: String,
pub(crate) probabilities_bp: BTreeMap<String, u16>,
pub(crate) confidence_bp: u16,
}
/// Validate one `choice` answer against the offered options. Any violation
/// rejects the whole answer; nothing is repaired.
fn validated_choice(answer: Option<&SystemOneAnswer>, options: &[&str]) -> Option<ValidChoice> {
pub(crate) fn validated_choice(
answer: Option<&SystemOneAnswer>,
options: &[&str],
) -> Option<ValidChoice> {
let answer = answer?;
if answer.kind != "choice" {
return None;
@@ -1625,6 +1632,14 @@ fn validated_choice(answer: Option<&SystemOneAnswer>, options: &[&str]) -> Optio
if (sum - 1.0).abs() > 0.02 {
return None;
}
let selected = (*answer.probabilities.get(choice)?)?;
if answer
.probabilities
.values()
.any(|probability| probability.is_some_and(|v| v > selected))
{
return None;
}
let confidence = answer.confidence?;
if !confidence.is_finite() || !(0.0..=1.0).contains(&confidence) {
return None;
@@ -1636,6 +1651,40 @@ fn validated_choice(answer: Option<&SystemOneAnswer>, options: &[&str]) -> Optio
})
}
/// One decision response settled through the existing routed-usage ledger.
/// Malformed policy answers still carry provider usage; missing/all-zero usage
/// remains an explicit coverage gap. The source identity is route-bound.
pub(crate) fn decision_usage_batch(
request_route: &EffectiveRouteEnvelope,
response: &SystemOneResponse,
) -> crate::cost_status::RuntimeUsageBatch {
let usage = codewhale_models::Usage {
input_tokens: response.usage.as_ref().map_or(0, |u| u.input_tokens),
output_tokens: response.usage.as_ref().map_or(0, |u| u.output_tokens),
..Default::default()
};
let source_id =
auto_route_usage_source_id(request_route, response.id.as_deref().unwrap_or("systemone"));
let route = request_route.sanitized_for_persistence();
if auto_route_usage_has_reported_data(&usage) {
crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
records: vec![RuntimeUsageRecord {
source_id,
usage: EffectiveRouteUsage { route, usage },
}],
..Default::default()
}
} else {
crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
drop_records: vec![RuntimeUsageDropRecord { source_id, route }],
dropped_records: 1,
..Default::default()
}
}
}
/// Turn a decoded System One response into a routing attempt. All policy is
/// here, in code: the decision model does no arithmetic.
fn decision_attempt_from_response(
@@ -1656,41 +1705,35 @@ fn decision_attempt_from_response(
fallback_reason: None,
};
if let Some(request_route) = request_route {
let usage = codewhale_models::Usage {
input_tokens: response
.usage
.as_ref()
.map_or(0, |usage| usage.input_tokens),
output_tokens: response
.usage
.as_ref()
.map_or(0, |usage| usage.output_tokens),
..Default::default()
};
let response_id = response.id.as_deref().unwrap_or("systemone");
if auto_route_usage_has_reported_data(&usage) {
attempt.routed_usage.push(RuntimeUsageRecord {
source_id: auto_route_usage_source_id(&request_route, response_id),
usage: EffectiveRouteUsage {
route: request_route.sanitized_for_persistence(),
usage,
},
});
} else {
let drop_route = request_route.sanitized_for_persistence();
attempt
.routed_usage_drop_records
.push(RuntimeUsageDropRecord {
source_id: auto_route_usage_source_id(
&drop_route,
&format!("missing-usage:{response_id}"),
),
route: drop_route,
});
attempt.routed_usage_dropped_records = 1;
}
let batch = decision_usage_batch(&request_route, response);
attempt.routed_usage = batch.records;
attempt.routed_usage_dropped_records = batch.dropped_records;
attempt.routed_usage_drop_records = batch.drop_records;
}
// Rejected answers still incurred spend. Keep bounded diagnostic evidence
// before any policy return; valid tier evidence below replaces this shell.
attempt.decision = Some(AutoRouteDecisionEvidence {
choice: "invalid".to_string(),
probabilities_bp: Default::default(),
confidence_bp: 0,
min_confidence_bp: inventory.router_min_confidence_bp,
cost_saving_kept_fast: false,
thinking: None,
provider_reported_cost_usd: response
.usage
.as_ref()
.and_then(|usage| usage.reported_cost()),
latency_ms,
response_model: response
.model
.as_deref()
.map(|model| model.chars().take(128).collect()),
});
if response.answers_validated == Some(false) {
attempt.failure = Some(AutoRouterFailure::InvalidAnswer);
return attempt;
}
let Some(tier) = validated_choice(response.answers.get("tier"), &DECISION_TIER_OPTIONS) else {
attempt.failure = Some(AutoRouterFailure::InvalidAnswer);
return attempt;
@@ -3280,13 +3323,13 @@ mod decision_router_tests {
let _env = hermetic_env();
let server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/systemone"))
.and(path("/api/alpha/decisions"))
.and(header("authorization", "Bearer or-test-key"))
.respond_with(ResponseTemplate::new(200).set_body_json(answer_body(0.82, 0.64)))
.expect(1)
.mount(&server)
.await;
let mut config = decision_config(&server.uri(), false, 2);
let mut config = decision_config(&format!("{}/api/v1", server.uri()), false, 2);
// A configured credential that a tool result echoed back.
let secret = "cw-router-secret-should-never-leave-process";
config
@@ -3363,11 +3406,11 @@ mod decision_router_tests {
let _env = hermetic_env();
let server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/systemone"))
.and(path("/api/alpha/decisions"))
.respond_with(ResponseTemplate::new(200).set_body_json(answer_body(0.82, 0.64)))
.mount(&server)
.await;
let config = decision_config(&server.uri(), false, 2);
let config = decision_config(&format!("{}/api/v1", server.uri()), false, 2);
let selection = route(&config, "Debug the release pipeline", "").await;
assert_eq!(selection.provider, ApiProvider::Deepseek);
@@ -3418,11 +3461,11 @@ mod decision_router_tests {
let _env = hermetic_env();
let server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/systemone"))
.and(path("/api/alpha/decisions"))
.respond_with(ResponseTemplate::new(200).set_body_json(answer_body(0.35, 0.3)))
.mount(&server)
.await;
let config = decision_config(&server.uri(), false, 2);
let config = decision_config(&format!("{}/api/v1", server.uri()), false, 2);
let selection = route(&config, "What does this function do?", "").await;
assert_eq!(selection.model, "deepseek-v4-pro", "declared default");
@@ -3449,7 +3492,7 @@ mod decision_router_tests {
let _env = hermetic_env();
let server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/systemone"))
.and(path("/api/alpha/decisions"))
.respond_with(
ResponseTemplate::new(status)
.set_body_string(format!(r#"{{"error":{{"message":"{MARKER}"}}}}"#)),
@@ -3457,7 +3500,7 @@ mod decision_router_tests {
.expect(1)
.mount(&server)
.await;
let config = decision_config(&server.uri(), false, 2);
let config = decision_config(&format!("{}/api/v1", server.uri()), false, 2);
let selection = route(&config, "Explain the diff", "").await;
assert_eq!(selection.model, "deepseek-v4-pro", "status {status}");
@@ -3485,7 +3528,7 @@ mod decision_router_tests {
let _env = hermetic_env();
let server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/systemone"))
.and(path("/api/alpha/decisions"))
.respond_with(
ResponseTemplate::new(200)
.set_body_json(answer_body(0.9, 0.8))
@@ -3493,7 +3536,7 @@ mod decision_router_tests {
)
.mount(&server)
.await;
let config = decision_config(&server.uri(), false, 1);
let config = decision_config(&format!("{}/api/v1", server.uri()), false, 1);
let selection = route(&config, "Explain the diff", "").await;
let receipt = receipt(&selection);
@@ -3517,7 +3560,7 @@ mod decision_router_tests {
.expect(0)
.mount(&server)
.await;
let mut config = decision_config(&server.uri(), false, 2);
let mut config = decision_config(&format!("{}/api/v1", server.uri()), false, 2);
// A single-tier active provider: nothing for the decision to choose.
config.provider = Some("openrouter".to_string());
config.default_text_model = Some("synthetic/single-tier-model".to_string());
@@ -3590,8 +3633,17 @@ mod decision_router_tests {
}
);
assert!(receipt.decision.is_some());
// TypeSafe is not a chat provider: spend stays on the receipt only.
assert!(selection.routed_usage.is_empty());
// A decision endpoint never inherits the active DeepSeek billing identity.
assert_eq!(selection.routed_usage.len(), 1);
let usage = &selection.routed_usage[0];
assert_eq!(usage.usage.route.provider, ApiProvider::Custom);
assert_eq!(usage.usage.route.provider_identity, "typesafe");
assert_eq!(
usage.usage.route.billing_mode,
crate::cost_status::RouteBillingMode::Unknown
);
assert_eq!(usage.usage.usage.input_tokens, 476);
assert_eq!(usage.usage.usage.output_tokens, 70);
assert!(selection.routed_usage_drop_records.is_empty());
}
@@ -3672,6 +3724,18 @@ mod decision_router_tests {
for body in [outside, bad_sum, missing, null_probability, bad_confidence] {
let attempt = policy(false, &parsed(body.clone()));
assert_eq!(attempt.recommendation, None, "{body}");
assert_eq!(
attempt
.decision
.as_ref()
.and_then(|r| r.provider_reported_cost_usd.as_deref()),
parsed(body.clone())
.usage
.as_ref()
.and_then(|u| u.reported_cost())
.as_deref(),
"rejected policy retains cost"
);
assert_eq!(
attempt.failure,
Some(AutoRouterFailure::InvalidAnswer),
+4
View File
@@ -449,6 +449,7 @@ fn messages_from_thread_detail_batches_tool_results() {
saved_session_checkpoint: None,
};
let turn = TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: 2,
id: turn_id.clone(),
@@ -9381,6 +9382,7 @@ async fn session_save_merges_thread_cost_split_and_records_coverage() -> Result<
let store = runtime_threads.test_store();
let now = Utc::now();
let mut turn = TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: 2,
id: "turn_cost_merge".to_string(),
@@ -9579,6 +9581,7 @@ async fn session_save_persists_parent_cny_unpriced_reasons_without_double_count(
let store = runtime_threads.test_store();
let now = Utc::now();
let mut turn = TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: 2,
id: "turn_cny_reasons".to_string(),
@@ -10829,6 +10832,7 @@ fn seed_summary_search_transcript(
item_ids.push(item_id);
}
store.save_turn(&TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: 2,
id: turn_id.clone(),
+99 -1
View File
@@ -603,6 +603,20 @@ where
.collect::<Vec<_>>()
.serialize(serializer)
}
fn serialize_decision_receipts<S>(
values: &[crate::cost_status::RuntimeDecisionReceipt],
serializer: S,
) -> std::result::Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
values
.iter()
.map(crate::cost_status::RuntimeDecisionReceipt::sanitized)
.collect::<Vec<_>>()
.serialize(serializer)
}
const RUNTIME_RESTART_REASON: &str = "Interrupted by process restart";
const EMPTY_TURN_REASON: &str = "Turn completed without engine output";
const DYNAMIC_TOOL_RESULT_TIMEOUT: Duration = Duration::from_secs(300);
@@ -1457,6 +1471,14 @@ pub struct TurnRecord {
/// inline RLM/guardian calls.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub routed_usage: Vec<EffectiveRouteUsage>,
/// Decision evidence shares this durable origin-turn ledger, including
/// responses arriving after the parent terminal event.
#[serde(
default,
skip_serializing_if = "Vec::is_empty",
serialize_with = "serialize_decision_receipts"
)]
pub decision_receipts: Vec<crate::cost_status::RuntimeDecisionReceipt>,
/// Exact missing-usage provider responses paired with their frozen route.
/// Sources are persisted only as fingerprints, routes are sanitized, and
/// the shared source ledger bounds usage plus missing-usage records.
@@ -1523,6 +1545,11 @@ pub struct TurnRecord {
impl TurnRecord {
fn validate_output_token_limit(&self) -> Result<()> {
if self.decision_receipts.len() > MAX_ROUTED_USAGE_RECORDS_PER_TURN
|| self.decision_receipts.iter().any(|r| !r.is_bounded())
{
bail!("Turn decision evidence exceeds its bound");
}
if self.schema_version > MAX_SUPPORTED_RUNTIME_SCHEMA_VERSION {
bail!(
"Turn schema v{} is newer than supported v{}",
@@ -1669,6 +1696,32 @@ fn append_routed_usage_drop_record(turn: &mut TurnRecord, record: RuntimeUsageDr
true
}
fn append_decision_receipt(
turn: &mut TurnRecord,
receipt: &crate::cost_status::RuntimeDecisionReceipt,
) -> bool {
if !receipt.is_bounded() {
return false;
}
let receipt = receipt.sanitized();
if turn
.decision_receipts
.iter()
.any(|old| old.source_id == receipt.source_id)
{
return false;
}
if turn.decision_receipts.len() >= MAX_ROUTED_USAGE_RECORDS_PER_TURN {
if turn.routed_usage_dropped_records == 0 {
turn.routed_usage_dropped_records = 1;
return true;
}
return false;
}
turn.decision_receipts.push(receipt);
true
}
/// Bind pre-parent auxiliary calls to a reserved Runtime turn before the
/// engine accepts the parent operation. Exact records are persisted now so a
/// crash cannot erase them; the engine's terminal event remains the sole
@@ -1678,6 +1731,9 @@ fn append_initial_routed_usage_to_turn(
turn: &mut TurnRecord,
batch: &crate::cost_status::RuntimeUsageBatch,
) {
for receipt in &batch.decisions {
append_decision_receipt(turn, receipt);
}
for record in &batch.records {
append_routed_usage_record(turn, &record.source_id, record.usage.clone());
}
@@ -1694,7 +1750,10 @@ const UNACCEPTED_TURN_REASON: &str =
"Turn was not accepted; this record keeps the completed pre-turn provider call";
fn routed_usage_batch_is_empty(batch: &crate::cost_status::RuntimeUsageBatch) -> bool {
batch.records.is_empty() && batch.drop_records.is_empty() && batch.dropped_records == 0
batch.decisions.is_empty()
&& batch.records.is_empty()
&& batch.drop_records.is_empty()
&& batch.dropped_records == 0
}
/// Durable identity of the settlement record for one completed pre-turn
@@ -1721,6 +1780,12 @@ fn unaccepted_routed_usage_turn_id(
.map(|record| routed_usage_source_fingerprint(&record.source_id)),
)
.collect::<Vec<_>>();
identities.extend(
batch
.decisions
.iter()
.map(|record| routed_usage_source_fingerprint(&record.source_id)),
);
if identities.is_empty() {
return runtime_record_id("turn_unaccepted");
}
@@ -1758,6 +1823,7 @@ fn settle_unaccepted_routed_usage(
} else {
let now = Utc::now();
TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: turn_id.clone(),
@@ -11432,6 +11498,7 @@ impl RuntimeThreadManager {
// Only create a turn if there's content.
if !item_ids.is_empty() {
self.store.save_turn(&TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: if turn_seed.image_content.is_empty() {
CURRENT_RUNTIME_SCHEMA_VERSION
@@ -11800,6 +11867,9 @@ impl RuntimeThreadManager {
let _turn_mutation = self.store.turn_mutation.lock();
match self.store.load_turn(turn_id) {
Ok(mut turn) => {
for receipt in &background_usage.decisions {
append_decision_receipt(&mut turn, receipt);
}
for record in background_usage.records.iter().cloned() {
append_routed_usage_record(&mut turn, &record.source_id, record.usage);
}
@@ -12478,6 +12548,7 @@ impl RuntimeThreadManager {
// malformed/stale parent catalog entry cannot erase real
// auxiliary spend on the error path.
let initial_routed_usage = crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
records: std::mem::take(&mut selection.routed_usage),
drop_records: std::mem::take(&mut selection.routed_usage_drop_records),
dropped_records: std::mem::take(
@@ -13087,6 +13158,7 @@ impl RuntimeThreadManager {
thread.auto_approve,
);
let turn = TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: turn_id.clone(),
@@ -14309,6 +14381,21 @@ impl RuntimeThreadManager {
drop_store.save_turn(&turn).is_ok()
})),
);
let decision_store = self.store.clone();
let decision_turn_id = turn_id.to_string();
crate::cost_status::register_runtime_decision_sink(
turn_id,
Arc::new(move |receipt| {
let _turn_mutation = decision_store.turn_mutation.lock();
let Ok(mut turn) = decision_store.load_turn(&decision_turn_id) else {
return false;
};
if !append_decision_receipt(&mut turn, &receipt) {
return true;
}
decision_store.save_turn(&turn).is_ok()
}),
);
}
/// Persist an engine status line as a completed `status` item.
@@ -14813,6 +14900,14 @@ impl RuntimeThreadManager {
if let Some(batch) =
crate::cost_status::child_usage_records_from_metadata(metadata)
{
if !batch.decisions.is_empty() {
let _turn_mutation = self.store.turn_mutation.lock();
let mut turn = self.store.load_turn(&turn_id)?;
for receipt in &batch.decisions {
append_decision_receipt(&mut turn, receipt);
}
self.store.save_turn(&turn)?;
}
for record in batch.records {
self.append_routed_usage_to_turn(
&turn_id,
@@ -16191,6 +16286,9 @@ impl RuntimeThreadManager {
turn.duration_ms = turn.started_at.map(|start| duration_ms(start, ended_at));
turn.usage = turn_usage;
turn.effective_route_usage = turn_effective_route_usage;
for receipt in &background_usage.decisions {
append_decision_receipt(&mut turn, receipt);
}
for record in background_usage.records {
append_routed_usage_record(&mut turn, &record.source_id, record.usage);
}
+62
View File
@@ -2447,6 +2447,7 @@ fn sample_thread(thread_id: &str) -> ThreadRecord {
fn sample_turn(thread_id: &str, turn_id: &str, status: RuntimeTurnStatus) -> TurnRecord {
let now = Utc::now();
TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: turn_id.to_string(),
@@ -8464,6 +8465,7 @@ async fn initial_classifier_usage_is_persisted_before_terminal_and_merged_exactl
Utc::now(),
);
let classifier_batch = crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
records: vec![crate::cost_status::RuntimeUsageRecord {
source_id: "auto-router:runtime-fixture".to_string(),
usage: crate::cost_status::EffectiveRouteUsage {
@@ -8600,6 +8602,7 @@ fn classifier_settlement_batch(
);
route.billing_mode = crate::cost_status::RouteBillingMode::Subscription;
crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
records: vec![crate::cost_status::RuntimeUsageRecord {
source_id: format!("auto-router:{source_prefix}-usage"),
usage: crate::cost_status::EffectiveRouteUsage {
@@ -16747,6 +16750,7 @@ fn opening_manager_recovers_stale_queued_and_in_progress_work() -> Result<()> {
manager.store.save_item(&queued_item)?;
manager.store.save_turn(&TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: "turn_in_progress".to_string(),
@@ -16785,6 +16789,7 @@ fn opening_manager_recovers_stale_queued_and_in_progress_work() -> Result<()> {
workspace_snapshots: Vec::new(),
})?;
manager.store.save_turn(&TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: "turn_queued".to_string(),
@@ -16981,6 +16986,7 @@ fn seed_turns_with_user_messages(
ended_at: Some(created_at),
})?;
manager.store.save_turn(&TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: turn_id.clone(),
@@ -17800,6 +17806,7 @@ fn restart_rebuild_restores_tool_call_identity_from_persisted_items() -> Result<
manager.store.save_item(&user_item)?;
manager.store.save_item(&call_item)?;
manager.store.save_turn(&TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: "turn_5823".to_string(),
@@ -17905,6 +17912,7 @@ fn restart_rebuild_keeps_in_flight_tool_call_identity() -> Result<()> {
};
manager.store.save_item(&call_item)?;
manager.store.save_turn(&TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: "turn_5823_inflight".to_string(),
@@ -18005,6 +18013,7 @@ fn restart_rebuild_skips_steers_the_engine_never_delivered() -> Result<()> {
manager.store.save_item(&dropped)?;
manager.store.save_item(&pending)?;
manager.store.save_turn(&TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: turn_id.clone(),
@@ -18106,6 +18115,7 @@ fn restart_rebuild_skips_legacy_tool_items_without_identity() -> Result<()> {
manager.store.save_item(&user_item)?;
manager.store.save_item(&legacy_tool_item)?;
manager.store.save_turn(&TurnRecord {
decision_receipts: Vec::new(),
max_output_tokens: None,
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
id: "turn_5823_legacy".to_string(),
@@ -21962,3 +21972,55 @@ mod execution_identity {
Ok(())
}
}
#[tokio::test]
async fn decision_receipt_lease_persists_terminal_turn_and_replays_exactly_once() -> Result<()> {
let _cost_scope = crate::cost_status::test_scope();
let directory = test_runtime_dir();
let manager = test_manager(directory.clone())?;
let thread = manager
.create_thread(CreateThreadRequest::default())
.await?;
let turn = sample_turn(
&thread.id,
"decision-terminal-origin",
RuntimeTurnStatus::Completed,
);
manager.store.save_turn(&turn)?;
manager.register_runtime_usage_sink(&turn.id);
let lease = crate::cost_status::acquire_runtime_usage_lease(&turn.id).expect("origin lease");
crate::cost_status::finish_runtime_usage_owner(&turn.id);
let receipt = crate::cost_status::decision_receipt_fixture("raw-runtime-decision-id");
let batch = crate::cost_status::RuntimeUsageBatch {
decisions: vec![receipt.clone()],
..Default::default()
};
for _ in 0..2 {
crate::cost_status::report_runtime_usage_batch(
crate::cost_status::scope_token(),
Some(&turn.id),
&batch,
);
}
drop(lease);
let reloaded = test_manager(directory)?.store.load_turn(&turn.id)?;
assert_eq!(reloaded.status, RuntimeTurnStatus::Completed);
assert_eq!(reloaded.decision_receipts, vec![receipt.sanitized()]);
assert_eq!(
reloaded.decision_receipts[0]
.evidence
.provider_reported_cost_usd
.as_deref(),
Some("0.000012054")
);
assert!(
reloaded.routed_usage.is_empty(),
"diagnostic receipt alone must not mint a second token charge"
);
assert_eq!(
unaccepted_routed_usage_turn_id(&thread.id, &batch),
unaccepted_routed_usage_turn_id(&thread.id, &batch)
);
assert!(!serde_json::to_string(&reloaded)?.contains("raw-runtime-decision-id"));
Ok(())
}
+155 -4
View File
@@ -71,6 +71,8 @@ struct LateUsageRecord {
route: crate::cost_status::EffectiveRouteEnvelope,
#[serde(default, skip_serializing_if = "Option::is_none")]
usage: Option<codewhale_models::Usage>,
#[serde(default, skip_serializing_if = "Option::is_none")]
decision: Option<crate::cost_status::RuntimeDecisionReceipt>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -1681,6 +1683,7 @@ impl SessionManager {
|| ledger.records.iter().any(|record| {
!is_sha256_fingerprint(&record.source_fingerprint)
|| !is_sha256_fingerprint(&record.turn_fingerprint)
|| record.decision.as_ref().is_some_and(|r| !r.is_bounded())
})
{
return Err(io::Error::new(
@@ -1742,6 +1745,7 @@ impl SessionManager {
source_id: &str,
route: &crate::cost_status::EffectiveRouteEnvelope,
usage: Option<&codewhale_models::Usage>,
decision: Option<&crate::cost_status::RuntimeDecisionReceipt>,
) -> io::Result<bool> {
let (path, lock_path) = self.ensure_late_usage_paths(session_id)?;
let lock_file = open_private_lock_file(&lock_path)?;
@@ -1753,11 +1757,21 @@ impl SessionManager {
}
let mut ledger = Self::load_late_usage_unlocked(&path)?;
let source_fingerprint = crate::cost_status::usage_source_fingerprint(source_id);
if ledger
if let Some(record) = ledger
.records
.iter()
.any(|record| record.source_fingerprint == source_fingerprint)
.iter_mut()
.find(|record| record.source_fingerprint == source_fingerprint)
{
if let Some(decision) = decision {
if !decision.is_bounded() {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
"decision receipt exceeds its bound",
));
}
record.decision = Some(decision.sanitized());
Self::write_late_usage_ledger(&path, &ledger)?;
}
return Ok(true);
}
if ledger.records.len() == MAX_LATE_USAGE_RECORDS_PER_SESSION {
@@ -1772,11 +1786,48 @@ impl SessionManager {
turn_fingerprint: crate::cost_status::usage_source_fingerprint(turn_id),
route: route.sanitized_for_persistence(),
usage: usage.cloned(),
decision: decision.map(crate::cost_status::RuntimeDecisionReceipt::sanitized),
});
Self::write_late_usage_ledger(&path, &ledger)?;
Ok(true)
}
pub(crate) fn persist_late_decision_receipt(
&self,
session_id: &str,
turn_id: &str,
receipt: &crate::cost_status::RuntimeDecisionReceipt,
) -> io::Result<bool> {
if !receipt.is_bounded() {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
"decision receipt exceeds its bound",
));
}
self.persist_late_usage_record(
session_id,
turn_id,
&receipt.source_id,
&receipt.route,
receipt.usage.as_ref(),
Some(receipt),
)
}
/// Read the bounded provider decision evidence from the existing origin ledger.
#[cfg(test)]
pub(crate) fn decision_receipts_for_session(
&self,
session_id: &str,
) -> io::Result<Vec<crate::cost_status::RuntimeDecisionReceipt>> {
Ok(self
.load_late_usage(session_id)?
.records
.into_iter()
.filter_map(|record| record.decision)
.collect())
}
pub(crate) fn persist_late_runtime_usage(
&self,
session_id: &str,
@@ -1789,6 +1840,7 @@ impl SessionManager {
&record.source_id,
&record.usage.route,
Some(&record.usage.usage),
None,
)
}
@@ -1798,7 +1850,14 @@ impl SessionManager {
turn_id: &str,
record: &crate::cost_status::RuntimeUsageDropRecord,
) -> io::Result<bool> {
self.persist_late_usage_record(session_id, turn_id, &record.source_id, &record.route, None)
self.persist_late_usage_record(
session_id,
turn_id,
&record.source_id,
&record.route,
None,
None,
)
}
fn with_session_read_lock<T>(
@@ -1863,6 +1922,14 @@ impl SessionManager {
}
};
for record in ledger.records {
if let Some(receipt) = &record.decision {
if receipt.is_bounded() {
metadata
.cost
.route_receipts
.insert(receipt.diagnostic_receipt());
}
}
let source_fingerprint = record.source_fingerprint.clone();
let source_id = format!("late:{}", record.source_fingerprint);
let mut pending = if let Some(usage) = record.usage.as_ref() {
@@ -4148,6 +4215,90 @@ mod tests {
}
}
#[test]
fn decision_receipt_survives_restart_replay_and_session_deletion() {
let tmp = tempdir().expect("tempdir");
let manager = SessionManager::new(tmp.path().join("sessions")).expect("manager");
save_late_usage_test_session(&manager, "decision-origin");
let receipt = crate::cost_status::decision_receipt_fixture("raw-decision-response-id");
manager
.persist_late_decision_receipt("decision-origin", "origin-turn", &receipt)
.expect("decision append");
manager
.persist_late_runtime_usage(
"decision-origin",
"origin-turn",
&crate::cost_status::RuntimeUsageRecord {
source_id: receipt.source_id.clone(),
usage: crate::cost_status::EffectiveRouteUsage {
route: receipt.route.clone(),
usage: receipt.usage.clone().expect("usage"),
},
},
)
.expect("token append dedupes");
drop(manager);
let manager = SessionManager::new(tmp.path().join("sessions")).expect("restart");
let retained = manager
.decision_receipts_for_session("decision-origin")
.expect("receipts");
assert_eq!(retained, vec![receipt.sanitized()]);
let mut restored = manager
.load_session_snapshot("decision-origin")
.expect("restore");
assert_eq!(restored.metadata.total_tokens, 13);
assert_eq!(
restored.metadata.cost.unpriced_turns, 1,
"unknown TypeSafe billing remains explicit"
);
assert!(
restored
.metadata
.cost
.route_receipts
.iter()
.any(|r| r.contains("0.000012054"))
);
manager.apply_late_usage_to_metadata(&mut restored.metadata);
assert_eq!(
restored.metadata.total_tokens, 13,
"replay must not count tokens twice"
);
assert_eq!(restored.metadata.cost.unpriced_turns, 1);
manager.save_session(&restored).expect("save overlay");
assert_eq!(
manager
.load_session_snapshot("decision-origin")
.expect("resume again")
.metadata
.total_tokens,
13
);
let (ledger, _) = manager.late_usage_paths("decision-origin").expect("paths");
assert!(
!fs::read_to_string(&ledger)
.expect("ledger")
.contains("raw-decision-response-id")
);
manager.delete_session("decision-origin").expect("delete");
assert!(
manager
.persist_late_decision_receipt("decision-origin", "origin-turn", &receipt)
.expect("retired replay")
);
assert!(
!ledger.exists(),
"late evidence must never recreate a deleted origin"
);
let mut oversized = receipt;
oversized.evidence.response_model = Some("x".repeat(129));
assert!(
manager
.persist_late_decision_receipt("decision-origin", "origin-turn", &oversized)
.is_err()
);
}
#[test]
fn late_usage_reads_do_not_create_accounting_storage() {
let tmp = tempdir().expect("tempdir");
+371 -31
View File
@@ -44,8 +44,9 @@
//! - The TypeSafe route needs a TypeSafe key even for a self-hosted server
//! (the transport always authenticates); a server that ignores auth can
//! be given any placeholder key.
//! - The gate's own spend is the configured endpoint's and is not entered
//! in session cost totals or Auto receipts.
//! - Usage settles through the originating turn's shared ledger. Missing
//! usage or cancellation after dispatch records a coverage gap. TypeSafe
//! is an unpriced Custom route until a billing basis is reviewed.
//! - Misconfiguration while enabled (missing or unknown provider, bad
//! timeout, missing key) is logged at `warn` for each turn and nothing is
//! sent.
@@ -56,7 +57,7 @@
//! (Von, OpenJev, Laya) are third-party open models; only the integration
//! architecture and the routing method here are covered by that attribution.
use std::sync::atomic::AtomicBool;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, Instant};
use serde_json::{Value, json};
@@ -67,7 +68,53 @@ use codewhale_core::role::Role;
use crate::client::CodewhaleClient;
use crate::client::system_one::{DecisionRouterRoute, SystemOneResponse};
use crate::config::Config;
use crate::model_routing::{AutoRouterFailure, truncate_for_auto_router};
use crate::model_routing::{
AutoRouterFailure, auto_route_usage_source_id, decision_usage_batch, truncate_for_auto_router,
};
use tokio_util::sync::CancellationToken;
#[derive(Clone)]
struct ShadowUsageContext {
scope: crate::cost_status::CostScopeToken,
runtime_owner: Option<String>,
// Retains the origin's existing durable ledger until settlement finishes.
_lease: Option<crate::cost_status::RuntimeUsageLease>,
#[cfg(test)]
test_origin: std::thread::ThreadId,
}
impl ShadowUsageContext {
fn capture(owner: Option<&str>) -> Self {
Self {
scope: crate::cost_status::scope_token(),
runtime_owner: owner.map(str::to_owned),
_lease: owner.and_then(crate::cost_status::acquire_runtime_usage_lease),
#[cfg(test)]
test_origin: crate::cost_status::test_cost_scope_id(),
}
}
async fn report(&self, batch: crate::cost_status::RuntimeUsageBatch) {
let context = self.clone();
// Existing sinks may write the origin-session ledger. Keep their
// filesystem/SQLite work off Tokio workers as well.
let settled = tokio::task::spawn_blocking(move || {
#[cfg(test)]
let _origin = crate::cost_status::bind_test_cost_scope(context.test_origin);
crate::cost_status::report_runtime_usage_batch(
context.scope,
context.runtime_owner.as_deref(),
&batch,
);
// Release the lease while its captured test binding is still live.
drop(context);
})
.await;
if settled.is_err() {
tracing::warn!(target: "superfast", "decision usage settlement worker failed");
}
}
}
/// Master switch. The gate never runs unless this env var is truthy.
const ENABLED_VAR: &str = "SUPERFAST_ENABLED";
@@ -207,13 +254,15 @@ fn last_user_text(messages: &[Message]) -> Option<String> {
}
/// Fire the shadow gate for a turn's first model request. Returns at once:
/// the evaluation runs on a detached task and only logs. No task is started
/// the detached evaluation uses the turn's cancellation and accounting owner. No task is started
/// (and `None` is returned) when the gate is off or misconfigured, when there
/// is no user text, or when no Tokio runtime is present. The handle exists
/// for tests; the turn loop drops it.
pub(crate) fn spawn_shadow_gate(
config: &Config,
messages: &[Message],
runtime_owner: Option<&str>,
cancel_token: &CancellationToken,
) -> Option<tokio::task::JoinHandle<ShadowOutcome>> {
let settings = match ShadowSettings::from_env()? {
Ok(settings) => settings,
@@ -225,8 +274,18 @@ pub(crate) fn spawn_shadow_gate(
let latest_request = last_user_text(messages)?;
let runtime = tokio::runtime::Handle::try_current().ok()?;
let config = config.clone();
// Capture before detaching; the next session/turn must never acquire it.
let usage_context = ShadowUsageContext::capture(runtime_owner);
let cancel_token = cancel_token.clone();
Some(runtime.spawn(async move {
let outcome = evaluate(&config, &settings, &latest_request).await;
let outcome = evaluate(
&config,
&settings,
&latest_request,
&usage_context,
&cancel_token,
)
.await;
log_outcome(outcome);
outcome
}))
@@ -264,7 +323,15 @@ async fn evaluate(
config: &Config,
settings: &ShadowSettings,
latest_request: &str,
usage_context: &ShadowUsageContext,
cancel_token: &CancellationToken,
) -> ShadowOutcome {
if cancel_token.is_cancelled() {
return ShadowOutcome::Failed {
failure: AutoRouterFailure::Cancelled,
latency_ms: 0,
};
}
// Client construction resolves keys (environment, secret store), so it
// runs off the async worker (#6149).
let built = {
@@ -287,27 +354,95 @@ async fn evaluate(
};
};
let body = decision_body(&client, &settings.model, latest_request);
let request_route = client.effective_route_envelope(&settings.model, chrono::Utc::now());
let dispatched = AtomicBool::new(false);
let started = Instant::now();
let answer = tokio::time::timeout(
settings.timeout,
client.system_one_decide(&body, &dispatched),
)
.await;
let answer = tokio::select! {
biased;
() = cancel_token.cancelled() => Err(AutoRouterFailure::Cancelled),
result = tokio::time::timeout(settings.timeout, client.system_one_decide(&body, &dispatched)) => {
result.unwrap_or(Err(AutoRouterFailure::Timeout))
}
};
let latency_ms = u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX);
match answer {
Err(_) => ShadowOutcome::Failed {
failure: AutoRouterFailure::Timeout,
latency_ms,
},
Ok(Err(failure)) => ShadowOutcome::Failed {
failure,
latency_ms,
},
Ok(Ok(response)) => ShadowOutcome::Recommendation {
route: derive_route(&response),
latency_ms,
},
Err(failure) => {
if dispatched.load(Ordering::Acquire) {
usage_context
.report(crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
drop_records: vec![crate::cost_status::RuntimeUsageDropRecord {
source_id: auto_route_usage_source_id(
&request_route,
"shadow:dispatched-unreceipted",
),
route: request_route.sanitized_for_persistence(),
}],
dropped_records: 1,
..Default::default()
})
.await;
}
ShadowOutcome::Failed {
failure,
latency_ms,
}
}
Ok(response) => {
// Billing evidence settles even when strict answer validation
// prevents the policy from producing a recommendation.
let mut batch = decision_usage_batch(&request_route, &response);
let intent = crate::model_routing::validated_choice(
response.answers.get("intent"),
&["code_change", "code_question", "command", "chat", "other"],
);
batch
.decisions
.push(crate::cost_status::RuntimeDecisionReceipt {
source_id: auto_route_usage_source_id(
&request_route,
response.id.as_deref().unwrap_or("systemone"),
),
route: request_route.sanitized_for_persistence(),
usage: response.usage.as_ref().map(|u| codewhale_models::Usage {
input_tokens: u.input_tokens,
output_tokens: u.output_tokens,
..Default::default()
}),
shadow: true,
valid_answers: response.answers_validated != Some(false),
evidence: crate::model_routing::AutoRouteDecisionEvidence {
choice: intent
.as_ref()
.map_or_else(|| "invalid".to_string(), |v| v.choice.clone()),
probabilities_bp: intent
.as_ref()
.map_or_else(Default::default, |v| v.probabilities_bp.clone()),
confidence_bp: intent.as_ref().map_or(0, |v| v.confidence_bp),
min_confidence_bp: 5_000,
cost_saving_kept_fast: false,
thinking: None,
provider_reported_cost_usd: response
.usage
.as_ref()
.and_then(|u| u.reported_cost()),
latency_ms,
response_model: response.model.clone(),
},
});
usage_context.report(batch).await;
if response.answers_validated == Some(false) {
ShadowOutcome::Failed {
failure: AutoRouterFailure::InvalidAnswer,
latency_ms,
}
} else {
ShadowOutcome::Recommendation {
route: derive_route(&response),
latency_ms,
}
}
}
}
}
@@ -378,11 +513,11 @@ fn derive_route(response: &SystemOneResponse) -> Route {
}
// Clearly chat, with a calibrated intent and a present, low tool-need signal.
let intent_is_calibrated_chat = response.answers.get("intent").is_some_and(|intent| {
intent.kind == "choice"
&& intent.choice.as_deref() == Some("chat")
&& unit_interval(intent.confidence).is_some_and(|c| c >= 0.5)
});
let intent_is_calibrated_chat = crate::model_routing::validated_choice(
response.answers.get("intent"),
&["code_change", "code_question", "command", "chat", "other"],
)
.is_some_and(|intent| intent.choice == "chat" && intent.confidence_bp >= 5_000);
if intent_is_calibrated_chat && needs_tool.is_some_and(|nt| nt <= 0.2) {
return Route::PlainChat;
}
@@ -477,6 +612,7 @@ mod tests {
json!({
"id": "sf-1",
"model": "jev-latest",
"usage": { "input_tokens": 121, "output_tokens": 8 },
"answers": {
"needs_tool": { "type": "noul", "noul": needs_tool },
"answerable_from_context": { "type": "noul", "noul": from_context },
@@ -498,7 +634,7 @@ mod tests {
}
async fn run(messages: &[Message]) -> Option<ShadowOutcome> {
let handle = spawn_shadow_gate(&config(), messages)?;
let handle = spawn_shadow_gate(&config(), messages, None, &CancellationToken::new())?;
Some(handle.await.expect("shadow task"))
}
@@ -629,7 +765,13 @@ mod tests {
let _env = hermetic_env(Some((&server, 100)));
let started = Instant::now();
let handle = spawn_shadow_gate(&config(), &turn("Run the tests")).expect("task");
let handle = spawn_shadow_gate(
&config(),
&turn("Run the tests"),
None,
&CancellationToken::new(),
)
.expect("task");
assert!(
started.elapsed() < Duration::from_millis(100),
"spawning must not wait on the decision call"
@@ -744,7 +886,8 @@ mod tests {
(
json!({
"needs_tool": { "type": "noul", "noul": 0.05 },
"intent": { "type": "choice", "choice": "chat", "confidence": 0.8 }
"intent": { "type": "choice", "choice": "chat", "confidence": 0.8,
"probabilities": { "chat": 0.9, "code_change": 0.04, "code_question": 0.03, "command": 0.02, "other": 0.01 } }
}),
Route::PlainChat,
),
@@ -784,4 +927,201 @@ mod tests {
None
);
}
#[tokio::test]
async fn cancelled_before_dispatch_sends_nothing() {
let server = MockServer::start().await;
let _env = hermetic_env(Some((&server, 2_000)));
let cancel = CancellationToken::new();
cancel.cancel();
let outcome = spawn_shadow_gate(&config(), &turn("hi"), None, &cancel)
.expect("task")
.await
.expect("joined");
assert!(matches!(
outcome,
ShadowOutcome::Failed {
failure: AutoRouterFailure::Cancelled,
..
}
));
assert!(requests(&server).await.is_empty());
}
#[tokio::test]
async fn cancelled_dispatched_call_records_gap_for_its_owner() {
let server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/systemone"))
.respond_with(
ResponseTemplate::new(200)
.set_delay(Duration::from_secs(2))
.set_body_json(noul_body(0.9, 0.1)),
)
.expect(1)
.mount(&server)
.await;
let _env = hermetic_env(Some((&server, 3_000)));
let owner = "superfast-cancel-owner";
let dropped = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
let sink = dropped.clone();
crate::cost_status::register_runtime_usage_sink_with_drop(
owner,
std::sync::Arc::new(|_| true),
Some(std::sync::Arc::new(move |record| {
sink.lock().expect("drop sink").push(record);
true
})),
);
let cancel = CancellationToken::new();
let handle = spawn_shadow_gate(&config(), &turn("hi"), Some(owner), &cancel).expect("task");
tokio::time::timeout(Duration::from_secs(1), async {
while requests(&server).await.is_empty() {
tokio::task::yield_now().await;
}
})
.await
.expect("request admitted");
let start = Instant::now();
cancel.cancel();
let outcome = tokio::time::timeout(Duration::from_millis(300), handle)
.await
.expect("cancellation stops the pending request")
.expect("joined");
assert!(matches!(
outcome,
ShadowOutcome::Failed {
failure: AutoRouterFailure::Cancelled,
..
}
));
assert!(start.elapsed() < Duration::from_millis(300));
let records = dropped.lock().expect("drop records");
assert_eq!(records.len(), 1);
assert_eq!(records[0].route.provider_identity, "typesafe");
let batch = crate::cost_status::take_runtime_usage(owner);
assert!(batch.records.is_empty());
assert!(batch.drop_records.is_empty());
crate::cost_status::finish_runtime_usage_owner(owner);
}
#[tokio::test]
async fn owner_lease_retains_late_usage_and_invalid_answer_cost() {
let server = MockServer::start().await;
let mut body = noul_body(0.9, 0.1);
body["answers"]["needs_tool"]["noul"] = 2.0.into();
body["usage"]["cost"] = 0.000012054.into();
Mock::given(method("POST"))
.and(path("/v1/systemone"))
.respond_with(
ResponseTemplate::new(200)
.set_delay(Duration::from_millis(100))
.set_body_json(body),
)
.expect(1)
.mount(&server)
.await;
let _env = hermetic_env(Some((&server, 2_000)));
let owner = "superfast-late-owner";
let observed = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
let sink = observed.clone();
crate::cost_status::register_runtime_usage_sink(
owner,
std::sync::Arc::new(move |record| {
sink.lock().expect("sink").push(record);
true
}),
);
let decisions = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
let decision_sink = decisions.clone();
crate::cost_status::register_runtime_decision_sink(
owner,
std::sync::Arc::new(move |receipt| {
decision_sink.lock().expect("decision sink").push(receipt);
true
}),
);
let handle = spawn_shadow_gate(
&config(),
&turn("hi"),
Some(owner),
&CancellationToken::new(),
)
.expect("task");
crate::cost_status::finish_runtime_usage_owner(owner);
let outcome = handle.await.expect("joined");
assert!(matches!(
outcome,
ShadowOutcome::Failed {
failure: AutoRouterFailure::InvalidAnswer,
..
}
));
let records = observed.lock().expect("records");
assert_eq!(
records.len(),
1,
"retired origin still owns its late response"
);
assert_eq!(records[0].usage.usage.input_tokens, 121);
assert_eq!(records[0].usage.usage.output_tokens, 8);
assert_eq!(records[0].usage.route.provider_identity, "typesafe");
assert_eq!(
records[0].usage.route.billing_mode,
crate::cost_status::RouteBillingMode::Unknown
);
let receipts = decisions.lock().expect("decision receipts");
assert_eq!(receipts.len(), 1);
assert!(!receipts[0].valid_answers);
assert!(receipts[0].shadow);
assert_eq!(
receipts[0].evidence.provider_reported_cost_usd.as_deref(),
Some("0.000012054")
);
assert!(
crate::cost_status::take_runtime_usage(owner)
.records
.is_empty(),
"settled response must not enter fallback journal"
);
}
#[tokio::test]
async fn oversized_decision_body_fails_closed_with_dispatched_coverage_gap() {
let server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/v1/systemone"))
.respond_with(ResponseTemplate::new(200).set_body_string("x".repeat(256 * 1024 + 1)))
.expect(1)
.mount(&server)
.await;
let _env = hermetic_env(Some((&server, 2_000)));
let owner = "superfast-oversized-owner";
let drops = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
let sink = drops.clone();
crate::cost_status::register_runtime_usage_sink_with_drop(
owner,
std::sync::Arc::new(|_| true),
Some(std::sync::Arc::new(move |record| {
sink.lock().expect("drops").push(record);
true
})),
);
let handle = spawn_shadow_gate(
&config(),
&turn("hi"),
Some(owner),
&CancellationToken::new(),
)
.expect("task");
assert!(matches!(
handle.await.expect("joined"),
ShadowOutcome::Failed {
failure: AutoRouterFailure::InvalidAnswer,
..
}
));
assert_eq!(drops.lock().expect("drop receipts").len(), 1);
crate::cost_status::finish_runtime_usage_owner(owner);
}
}
+3
View File
@@ -494,6 +494,7 @@ impl RlmTool {
&error.to_string(),
started.elapsed(),
&crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
records: usage.records,
drop_records: usage.drop_records,
dropped_records: usage.dropped_records,
@@ -504,6 +505,7 @@ impl RlmTool {
(
round,
crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
records: usage.records,
drop_records: usage.drop_records,
dropped_records: usage.dropped_records,
@@ -996,6 +998,7 @@ mod tests {
"kernel stdout closed",
Duration::from_millis(11),
&crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
records: vec![record],
drop_records: vec![drop_record],
dropped_records: 1,
+2
View File
@@ -72,6 +72,7 @@ fn failed_engine_channel_settles_classifier_batch_once() {
chrono::Utc::now(),
);
let batch = crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
records: vec![crate::cost_status::RuntimeUsageRecord {
source_id: "auto-router:dispatch-usage".to_string(),
usage: crate::cost_status::EffectiveRouteUsage {
@@ -21865,6 +21866,7 @@ fn routed_missing_usage_batch_prices_exact_routes_and_only_residual_as_generic()
local_route.provider_identity = "local-computer".to_string();
local_route.billing_mode = crate::cost_status::RouteBillingMode::Local;
let batch = crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
records: Vec::new(),
drop_records: vec![
crate::cost_status::RuntimeUsageDropRecord {
+1
View File
@@ -312,6 +312,7 @@ async fn test_preset(app: &mut App, config: &Config, preset: RouterPreset) {
cost_scope,
None,
&crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
records: selection.routed_usage.clone(),
drop_records: selection.routed_usage_drop_records.clone(),
dropped_records: selection.routed_usage_dropped_records,
+2
View File
@@ -186,6 +186,7 @@ pub(crate) async fn plan_turn_route(
let initial_routed_usage = auto_selection
.as_mut()
.map(|selection| crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
records: std::mem::take(&mut selection.routed_usage),
drop_records: std::mem::take(&mut selection.routed_usage_drop_records),
dropped_records: std::mem::take(&mut selection.routed_usage_dropped_records),
@@ -339,6 +340,7 @@ mod tests {
chrono::Utc::now(),
);
let batch = crate::cost_status::RuntimeUsageBatch {
decisions: Vec::new(),
records: vec![crate::cost_status::RuntimeUsageRecord {
source_id: "auto-router:plan-usage".to_string(),
usage: crate::cost_status::EffectiveRouteUsage {
+17 -4
View File
@@ -771,8 +771,13 @@ min_confidence = 0.5 # default 0.5, clamped to 0..1
0.75, or the turn stays on the fast tier.
- An unknown `kind`, or a decision `provider` other than `openrouter` /
`typesafe`, leaves the router unconfigured and shown as failing.
- `thinking` is ignored for decision routers. OpenRouter spend is recorded like
any routed usage; TypeSafe-direct spend appears on the receipt only.
- `thinking` is ignored for decision routers. Both routes settle tokens through
the originating session's routed-usage ledger. TypeSafe is a named Custom route
with unknown billing; its price is never borrowed from the active chat provider.
Provider-reported cost remains verbatim on the decision receipt.
- OpenRouter uses `POST /api/alpha/decisions`; TypeSafe uses `POST /v1/systemone`.
The shared client validates Choice, Noul and Score against the offered questions
and bounds responses to 256 KiB. Malformed answers fail open with usage retained.
#### Shadow Decision Gate (experimental, off by default)
@@ -800,8 +805,10 @@ SUPERFAST_TIMEOUT_MS=150 # 1..=10000, default 150
- Only the latest user message is sent, truncated to 4,000 characters and
redacted of configured secrets. The log (target `superfast`) carries the
route, failure class and latency, never prompt text.
- The gate's own spend is the configured endpoint's; it is not added to session
cost totals or Auto receipts.
- The gate retains the originating turn's accounting owner and cancellation.
Its tokens settle through the shared ledger; missing usage or a cancelled/timed
out request after dispatch creates an explicit coverage gap. Unknown TypeSafe
pricing is recorded as unpriced rather than free.
The Decision Gate concept and reference implementation are by Andrea Bruno,
released under CC BY 4.0:
@@ -3480,3 +3487,9 @@ codewhale sessions scrub-secrets --apply # rewrite the affected files
so it never loses a concurrent save. A session that is still open can write
its in-memory copy back on its next save, so close open sessions first, and
rotate any credential that was exposed — masking a stored copy cannot un-leak it.
Decision API compatibility uses OpenRouter `/api/alpha/decisions` and TypeSafe
`/v1/systemone` on the existing authenticated client. Provider-reported cost is
preserved verbatim in bounded decision receipts on the original turn or session,
including rejected answers and late shadow responses. Receipts are diagnostic;
they do not turn unknown pricing into an authoritative dollar total.