mirror of
https://github.com/codewhale-hq/Codewhale.git
synced 2026-11-07 17:09:13 -10:00
fix(pricing): keep the OpenRouter roster when one row is bad
Review follow-up for #6691. The live OpenRouter /v1/models lists six routers (openrouter/auto, auto-beta, fusion, pareto-code, bodybuilder, typesafe/jev-router) at "prompt":"-1","completion":"-1". parse_price rejected negatives and the conversion collected with `?`, so the whole refresh still failed and every turn stayed "rate unavailable". - A negative OpenRouter price now means "no fixed rate": the row is kept with cost None. Rows that do not decode, carry an id the catalog cannot hold, or an unreadable/implausible price are skipped and counted with a warning. A response that is not a `{"data": [...]}` list, or whose rows all fail, is still rejected. - declared_or_catalog_quote: a [[custom_models]] row with no rates yields to the exact endpoint's catalog price at every dispatch boundary (main turn, background envelope, EffectiveRouteEnvelope::capture); it stays frozen rate-less only when the catalog has none. The turn-loop logic moved into client::main_turn_pricing_quote_at so it is testable. - A pinned OpenRouter vendor no longer blocks an explicit declared rate for the exact route; the aggregate catalog price stays blocked. Against the saved live list (458 rows): refresh succeeds with 440 rows, 434 priced; deepseek/deepseek-v4-pro prices at 0.95526/1.91052 per M. Focused (cargo test -p codewhale-tui --lib): fixed: test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 13793 filtered out reverted (neg price, rate-less fallthrough, vendor pin): test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 13800 filtered out reverted (per-row decode skip): test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 13802 filtered out related (fetch_catalog openrouter configured_model effective_route provider_live_pricing exact_turn_snapshot main_turn cost_status provider_catalog_live): test result: ok. 120 passed; 0 failed; 0 ignored; 0 measured; 13683 filtered out Fixes #6690 Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_014ZwqatxgVFxHvovngywnks
This commit is contained in:
1 parent
2412a6ed11
commit
79d9815c76
7 files changed
+399
-95
No files matched your search
+7
-3
@@ -128,9 +128,13 @@ quieter, and Fleet runs can be checked before they spend anything.
|
||||
### Fixed
|
||||
|
||||
- OpenRouter turns are priced again instead of always showing "rate
|
||||
unavailable": the model-list refresh skips OpenRouter's `~` "latest" alias
|
||||
ids, which used to fail the whole refresh, and main interactive turns now
|
||||
honor a `[[custom_models]]` rate the same way background work does
|
||||
unavailable". The model-list refresh no longer fails as a whole on one bad
|
||||
row: `~` "latest" aliases are dropped, routers that publish a `-1`
|
||||
variable price are listed without a rate, and other malformed rows are
|
||||
skipped and counted. Main interactive turns now honor a `[[custom_models]]`
|
||||
rate the same way background work does, a declaration without rates no
|
||||
longer hides the catalog price, and an explicit declared rate prices a
|
||||
vendor-pinned OpenRouter route
|
||||
([#6690](https://github.com/Hmbown/Codewhale/issues/6690)).
|
||||
- Keep reasoning effort labels, including `xhigh` and `ultra`, visible beside
|
||||
short model names in the 80-column terminal footer.
|
||||
|
||||
@@ -128,9 +128,13 @@ quieter, and Fleet runs can be checked before they spend anything.
|
||||
### Fixed
|
||||
|
||||
- OpenRouter turns are priced again instead of always showing "rate
|
||||
unavailable": the model-list refresh skips OpenRouter's `~` "latest" alias
|
||||
ids, which used to fail the whole refresh, and main interactive turns now
|
||||
honor a `[[custom_models]]` rate the same way background work does
|
||||
unavailable". The model-list refresh no longer fails as a whole on one bad
|
||||
row: `~` "latest" aliases are dropped, routers that publish a `-1`
|
||||
variable price are listed without a rate, and other malformed rows are
|
||||
skipped and counted. Main interactive turns now honor a `[[custom_models]]`
|
||||
rate the same way background work does, a declaration without rates no
|
||||
longer hides the catalog price, and an explicit declared rate prices a
|
||||
vendor-pinned OpenRouter route
|
||||
([#6690](https://github.com/Hmbown/Codewhale/issues/6690)).
|
||||
- Keep reasoning effort labels, including `xhigh` and `ultra`, visible beside
|
||||
short model names in the 80-column terminal footer.
|
||||
|
||||
+186
-53
@@ -2834,21 +2834,23 @@ impl CodewhaleClient {
|
||||
let provider_live_pricing = u64::try_from(dispatched_at.timestamp())
|
||||
.ok()
|
||||
.and_then(|at| {
|
||||
self.configured_pricing_quote_at(
|
||||
self.api_provider,
|
||||
&self.provider_identity,
|
||||
&model,
|
||||
at,
|
||||
)
|
||||
.or_else(|| {
|
||||
crate::provider_catalog_live::fresh_dispatch_pricing_quote_at(
|
||||
crate::provider_catalog_live::declared_or_catalog_quote(
|
||||
self.configured_pricing_quote_at(
|
||||
self.api_provider,
|
||||
&self.provider_identity,
|
||||
&model,
|
||||
&self.base_url,
|
||||
at,
|
||||
)
|
||||
})
|
||||
),
|
||||
|| {
|
||||
crate::provider_catalog_live::fresh_dispatch_pricing_quote_at(
|
||||
self.api_provider,
|
||||
&self.provider_identity,
|
||||
&model,
|
||||
&self.base_url,
|
||||
at,
|
||||
)
|
||||
},
|
||||
)
|
||||
});
|
||||
crate::cost_status::EffectiveRouteEnvelope {
|
||||
openrouter_vendor: self.openrouter_vendor.clone(),
|
||||
@@ -3213,12 +3215,26 @@ impl CodewhaleClient {
|
||||
if or_models.is_empty() {
|
||||
return Err(CatalogRefreshError::EmptyList);
|
||||
}
|
||||
or_models
|
||||
// A row with an unreadable or implausible price is skipped and
|
||||
// counted, not allowed to fail every other row (#6690).
|
||||
let offerings: Vec<_> = or_models
|
||||
.iter()
|
||||
.map(|item| {
|
||||
openrouter_to_catalog_offering(item, &provider, &fingerprint, fetched_at)
|
||||
.filter_map(|item| {
|
||||
openrouter_to_catalog_offering(item, &provider, &fingerprint, fetched_at).ok()
|
||||
})
|
||||
.collect::<Result<Vec<_>, _>>()?
|
||||
.collect();
|
||||
let skipped = or_models.len() - offerings.len();
|
||||
if skipped > 0 {
|
||||
tracing::warn!(
|
||||
skipped,
|
||||
listed = or_models.len(),
|
||||
"skipped OpenRouter model rows with invalid pricing"
|
||||
);
|
||||
}
|
||||
if offerings.is_empty() {
|
||||
return Err(CatalogRefreshError::InvalidResponse);
|
||||
}
|
||||
offerings
|
||||
} else if self.catalog_endpoint_is_baseten() {
|
||||
let baseten_models = parse_baseten_models_response(&body)?;
|
||||
if baseten_models.is_empty() {
|
||||
@@ -4042,9 +4058,11 @@ struct ModelsListResponse {
|
||||
data: Vec<ModelListItem>,
|
||||
}
|
||||
|
||||
/// The list envelope is validated as a whole; each row is decoded on its own
|
||||
/// so one malformed row cannot fail the entire roster (#6690).
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct OpenRouterModelsResponse {
|
||||
data: Vec<OpenRouterModelItem>,
|
||||
data: Vec<serde_json::Value>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
@@ -4446,16 +4464,43 @@ fn parse_openrouter_models_response(
|
||||
) -> Result<Vec<OpenRouterModelItem>, CatalogRefreshError> {
|
||||
let parsed: OpenRouterModelsResponse =
|
||||
serde_json::from_str(payload).map_err(|_| CatalogRefreshError::InvalidResponse)?;
|
||||
let listed = parsed.data.len();
|
||||
let mut seen = std::collections::HashSet::new();
|
||||
// `~`-prefixed ids (`~deepseek/deepseek-pro-latest`) are OpenRouter's
|
||||
// moving "latest" aliases, not billing identities. They fail the catalog
|
||||
// id validator, and one such row used to fail the whole roster closed, so
|
||||
// no OpenRouter route could ever be priced from the lake (#6690).
|
||||
let models: Vec<_> = parsed
|
||||
.data
|
||||
.into_iter()
|
||||
.filter(|item| !item.id.starts_with('~') && seen.insert(item.id.clone()))
|
||||
.collect();
|
||||
let mut malformed = 0usize;
|
||||
let mut models = Vec::with_capacity(listed);
|
||||
for row in parsed.data {
|
||||
// `~`-prefixed ids (`~deepseek/deepseek-pro-latest`) are OpenRouter's
|
||||
// moving "latest" aliases, not billing identities, so they are dropped
|
||||
// silently. Any other row that does not decode or carries an id the
|
||||
// catalog cannot hold is skipped and counted: one such row used to
|
||||
// fail the whole roster closed, so no OpenRouter route could ever be
|
||||
// priced from the lake (#6690).
|
||||
let Ok(item) = serde_json::from_value::<OpenRouterModelItem>(row) else {
|
||||
malformed += 1;
|
||||
continue;
|
||||
};
|
||||
if item.id.starts_with('~') {
|
||||
continue;
|
||||
}
|
||||
if !crate::provider_lake::valid_catalog_model_id(&item.id) {
|
||||
malformed += 1;
|
||||
continue;
|
||||
}
|
||||
if seen.insert(item.id.clone()) {
|
||||
models.push(item);
|
||||
}
|
||||
}
|
||||
if malformed > 0 {
|
||||
tracing::warn!(
|
||||
malformed,
|
||||
listed,
|
||||
"skipped malformed OpenRouter model rows in the catalog refresh"
|
||||
);
|
||||
}
|
||||
// Rows were listed but not one survived: the response is not a roster.
|
||||
if models.is_empty() && malformed > 0 {
|
||||
return Err(CatalogRefreshError::InvalidResponse);
|
||||
}
|
||||
Ok(models)
|
||||
}
|
||||
|
||||
@@ -4741,30 +4786,45 @@ fn openrouter_to_catalog_offering(
|
||||
|
||||
let cost = if let Some(p) = item.pricing.as_ref() {
|
||||
// OpenRouter quotes per-token USD strings; ModelsDevCost is per million.
|
||||
// Its routers (`openrouter/auto`, `openrouter/fusion`, ...) publish
|
||||
// `"-1"`: the price depends on the model the router picks. A negative
|
||||
// price therefore means "no fixed rate" for the whole row, never a
|
||||
// partial one.
|
||||
let parse_price = |value: &Option<String>| -> Result<Option<f64>, CatalogRefreshError> {
|
||||
value
|
||||
.as_ref()
|
||||
.map(|value| {
|
||||
let parsed = value
|
||||
value
|
||||
.trim()
|
||||
.parse::<f64>()
|
||||
.ok()
|
||||
.filter(|value| value.is_finite() && *value >= 0.0)
|
||||
.ok_or(CatalogRefreshError::InvalidResponse)?;
|
||||
checked_per_token_to_per_million(parsed)
|
||||
.filter(|value| value.is_finite())
|
||||
.ok_or(CatalogRefreshError::InvalidResponse)
|
||||
})
|
||||
.transpose()
|
||||
};
|
||||
let cost = ModelsDevCost {
|
||||
input: parse_price(&p.prompt)?,
|
||||
output: parse_price(&p.completion)?,
|
||||
cache_read: parse_price(&p.input_cache_read)?,
|
||||
cache_write: parse_price(&p.input_cache_write)?,
|
||||
};
|
||||
if !codewhale_config::pricing::catalog_cost_is_valid(&cost) {
|
||||
return Err(CatalogRefreshError::InvalidResponse);
|
||||
let raw = [
|
||||
parse_price(&p.prompt)?,
|
||||
parse_price(&p.completion)?,
|
||||
parse_price(&p.input_cache_read)?,
|
||||
parse_price(&p.input_cache_write)?,
|
||||
];
|
||||
if raw.iter().flatten().any(|price| *price < 0.0) {
|
||||
None
|
||||
} else {
|
||||
let per_million =
|
||||
|price: Option<f64>| price.map(checked_per_token_to_per_million).transpose();
|
||||
let cost = ModelsDevCost {
|
||||
input: per_million(raw[0])?,
|
||||
output: per_million(raw[1])?,
|
||||
cache_read: per_million(raw[2])?,
|
||||
cache_write: per_million(raw[3])?,
|
||||
};
|
||||
if !codewhale_config::pricing::catalog_cost_is_valid(&cost) {
|
||||
return Err(CatalogRefreshError::InvalidResponse);
|
||||
}
|
||||
Some(cost)
|
||||
}
|
||||
Some(cost)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
@@ -4832,6 +4892,38 @@ fn openrouter_to_catalog_offering(
|
||||
})
|
||||
}
|
||||
|
||||
/// The rate a main interactive turn freezes at its dispatch boundary (#6690).
|
||||
/// An operator-declared `[[custom_models]]` rate wins, exactly as on the
|
||||
/// background envelope path; `client` must be installed on the endpoint this
|
||||
/// turn dispatches to, so a declaration never leaks onto another endpoint. A
|
||||
/// declaration with no rates yields to the catalog price for that endpoint.
|
||||
pub(crate) fn main_turn_pricing_quote_at(
|
||||
client: Option<&CodewhaleClient>,
|
||||
provider: ApiProvider,
|
||||
provider_identity: &str,
|
||||
model: &str,
|
||||
endpoint_fingerprint: &str,
|
||||
dispatched_at: u64,
|
||||
) -> Option<crate::provider_catalog_live::ProviderLivePricingQuote> {
|
||||
let declared = client
|
||||
.filter(|client| {
|
||||
crate::cost_status::endpoint_fingerprint(client.base_url()).as_deref()
|
||||
== Some(endpoint_fingerprint)
|
||||
})
|
||||
.and_then(|client| {
|
||||
client.configured_pricing_quote_at(provider, provider_identity, model, dispatched_at)
|
||||
});
|
||||
crate::provider_catalog_live::declared_or_catalog_quote(declared, || {
|
||||
crate::provider_catalog_live::fresh_provider_live_pricing_quote_at(
|
||||
provider,
|
||||
provider_identity,
|
||||
model,
|
||||
endpoint_fingerprint,
|
||||
dispatched_at,
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
pub(super) fn system_to_instructions(system: Option<SystemPrompt>) -> Option<String> {
|
||||
match system {
|
||||
Some(SystemPrompt::Text(text)) => Some(text),
|
||||
@@ -5646,11 +5738,17 @@ mod tests {
|
||||
})
|
||||
.to_string();
|
||||
let items = parse_openrouter_models_response(&openrouter).expect("OpenRouter shape");
|
||||
assert_eq!(
|
||||
openrouter_to_catalog_offering(&items[0], "openrouter", "fp", 1).unwrap_err(),
|
||||
CatalogRefreshError::InvalidResponse,
|
||||
"OpenRouter must reject {invalid:?}"
|
||||
);
|
||||
let offering = openrouter_to_catalog_offering(&items[0], "openrouter", "fp", 1);
|
||||
if invalid.starts_with('-') {
|
||||
// OpenRouter's negative sentinel means "no fixed rate" (#6690).
|
||||
assert_eq!(offering.expect("variable-price row").cost, None);
|
||||
} else {
|
||||
assert_eq!(
|
||||
offering.unwrap_err(),
|
||||
CatalogRefreshError::InvalidResponse,
|
||||
"OpenRouter must reject {invalid:?}"
|
||||
);
|
||||
}
|
||||
|
||||
let baseten = json!({
|
||||
"data": [{
|
||||
@@ -12776,9 +12874,11 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// #6690: OpenRouter's roster lists `~`-prefixed "latest" aliases. One
|
||||
/// such row used to fail the whole refresh closed (`invalid_response`), so
|
||||
/// the lake never went fresh and no OpenRouter turn could be priced.
|
||||
/// #6690: rows shaped like the live OpenRouter roster. A `~` "latest"
|
||||
/// alias, a router's `"-1"` variable price, an undecodable row and an id
|
||||
/// the catalog cannot hold each used to fail the whole refresh closed
|
||||
/// (`invalid_response`), so the lake never went fresh and no OpenRouter
|
||||
/// turn could be priced. Each now costs only its own row.
|
||||
#[tokio::test]
|
||||
async fn fetch_catalog_delta_skips_openrouter_latest_aliases_and_keeps_prices() {
|
||||
let server = MockServer::start().await;
|
||||
@@ -12787,9 +12887,20 @@ mod tests {
|
||||
200,
|
||||
json!({"data": [
|
||||
{"id": "~deepseek/deepseek-pro-latest",
|
||||
"pricing": {"prompt": "0.0000002523", "completion": "0.0000035",
|
||||
"input_cache_read": "0.0000002518"}},
|
||||
{"id": "openrouter/auto", "context_length": 2000000,
|
||||
"pricing": {"prompt": "-1", "completion": "-1"},
|
||||
"top_provider": {"context_length": null, "max_completion_tokens": null}},
|
||||
{"id": "synthetic/malformed-row", "context_length": "very long",
|
||||
"pricing": {"prompt": "0.000001", "completion": "0.000002"}},
|
||||
{"id": "deepseek/deepseek-v4-pro",
|
||||
"pricing": {"prompt": "0.000001", "completion": "0.000002"}}
|
||||
{"id": "synthetic/unreadable-price",
|
||||
"pricing": {"prompt": "not-a-number", "completion": "0.000002"}},
|
||||
{"id": "synthetic/bad id"},
|
||||
{"id": "deepseek/deepseek-v4-pro", "context_length": 1048576,
|
||||
"pricing": {"prompt": "0.00000095526", "completion": "0.00000191052",
|
||||
"input_cache_read": "0.000000079605"},
|
||||
"top_provider": {"context_length": 1024000, "max_completion_tokens": 384000}}
|
||||
]}),
|
||||
)
|
||||
.await;
|
||||
@@ -12798,16 +12909,38 @@ mod tests {
|
||||
let delta = client
|
||||
.fetch_catalog_delta()
|
||||
.await
|
||||
.expect("an alias row must not fail the whole roster");
|
||||
.expect("a bad row must not fail the whole roster");
|
||||
let ids: Vec<&str> = delta
|
||||
.offerings
|
||||
.iter()
|
||||
.map(|offering| offering.wire_model_id.as_str())
|
||||
.collect();
|
||||
assert_eq!(ids, ["deepseek/deepseek-v4-pro"]);
|
||||
let cost = delta.offerings[0].cost.as_ref().expect("served price");
|
||||
assert_eq!(cost.input, Some(1.0));
|
||||
assert_eq!(cost.output, Some(2.0));
|
||||
assert_eq!(ids, ["openrouter/auto", "deepseek/deepseek-v4-pro"]);
|
||||
assert_eq!(delta.offerings[0].cost, None, "router price is variable");
|
||||
let cost = delta.offerings[1].cost.as_ref().expect("served price");
|
||||
let close = |got: Option<f64>, want: f64| got.is_some_and(|got| (got - want).abs() < 1e-9);
|
||||
assert!(close(cost.input, 0.95526), "{cost:?}");
|
||||
assert!(close(cost.output, 1.91052), "{cost:?}");
|
||||
assert!(close(cost.cache_read, 0.079605), "{cost:?}");
|
||||
assert_eq!(cost.cache_write, None);
|
||||
|
||||
// Only structurally invalid responses still fail the refresh.
|
||||
for body in [
|
||||
json!({"data": {"id": "deepseek/deepseek-v4-pro"}}),
|
||||
json!({"models": []}),
|
||||
json!({"data": [{"name": "no id"}, {"id": "synthetic/bad id"}]}),
|
||||
] {
|
||||
let server = MockServer::start().await;
|
||||
mount_models_json(&server, 200, body.clone()).await;
|
||||
assert_eq!(
|
||||
openrouter_client_for(&server)
|
||||
.fetch_catalog_delta()
|
||||
.await
|
||||
.expect_err("structurally invalid roster"),
|
||||
CatalogRefreshError::InvalidResponse,
|
||||
"{body}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -1278,6 +1278,9 @@ async fn main_turn_dispatch_freezes_declared_custom_model_rate() {
|
||||
quote.provenance,
|
||||
codewhale_config::pricing::PricingProvenance::UserOverride
|
||||
);
|
||||
assert_eq!(quote.input_per_million.as_deref(), Some("1"));
|
||||
assert_eq!(quote.output_per_million.as_deref(), Some("2"));
|
||||
assert_eq!(quote.cache_read_per_million, None);
|
||||
handle.send(Op::Shutdown).await.expect("shutdown engine");
|
||||
run_task.await.expect("engine task");
|
||||
}
|
||||
|
||||
@@ -1829,35 +1829,14 @@ impl Engine {
|
||||
billing.provider_live_pricing = u64::try_from(dispatched_at.timestamp())
|
||||
.ok()
|
||||
.and_then(|dispatched_at_unix| {
|
||||
// An operator-declared `[[custom_models]]` rate
|
||||
// wins, exactly as on the background envelope
|
||||
// path; the fingerprint check keeps it scoped to
|
||||
// the endpoint this turn was installed on (#6690).
|
||||
let fingerprint = billing.endpoint_fingerprint.as_deref()?;
|
||||
self.codewhale_client
|
||||
.as_ref()
|
||||
.filter(|client| {
|
||||
crate::cost_status::endpoint_fingerprint(client.base_url())
|
||||
.as_deref()
|
||||
== Some(fingerprint)
|
||||
})
|
||||
.and_then(|client| {
|
||||
client.configured_pricing_quote_at(
|
||||
route.provider,
|
||||
&route.provider_identity,
|
||||
&route.model,
|
||||
dispatched_at_unix,
|
||||
)
|
||||
})
|
||||
.or_else(|| {
|
||||
crate::provider_catalog_live::fresh_provider_live_pricing_quote_at(
|
||||
route.provider,
|
||||
&route.provider_identity,
|
||||
&route.model,
|
||||
fingerprint,
|
||||
dispatched_at_unix,
|
||||
)
|
||||
})
|
||||
crate::client::main_turn_pricing_quote_at(
|
||||
self.codewhale_client.as_ref(),
|
||||
route.provider,
|
||||
&route.provider_identity,
|
||||
&route.model,
|
||||
billing.endpoint_fingerprint.as_deref()?,
|
||||
dispatched_at_unix,
|
||||
)
|
||||
});
|
||||
}
|
||||
let _ = self
|
||||
|
||||
@@ -201,8 +201,8 @@ impl EffectiveRouteEnvelope {
|
||||
u64::try_from(dispatched_at.timestamp())
|
||||
.ok()
|
||||
.and_then(|at| {
|
||||
config
|
||||
.and_then(|config| {
|
||||
crate::provider_catalog_live::declared_or_catalog_quote(
|
||||
config.and_then(|config| {
|
||||
crate::provider_catalog_live::configured_dispatch_pricing_quote_at(
|
||||
config.custom_models.as_deref().unwrap_or_default(),
|
||||
provider,
|
||||
@@ -211,8 +211,8 @@ impl EffectiveRouteEnvelope {
|
||||
base_url,
|
||||
at,
|
||||
)
|
||||
})
|
||||
.or_else(|| {
|
||||
}),
|
||||
|| {
|
||||
crate::provider_catalog_live::fresh_dispatch_pricing_quote_at(
|
||||
provider,
|
||||
&provider_identity,
|
||||
@@ -220,7 +220,8 @@ impl EffectiveRouteEnvelope {
|
||||
base_url,
|
||||
at,
|
||||
)
|
||||
})
|
||||
},
|
||||
)
|
||||
})
|
||||
});
|
||||
Self {
|
||||
@@ -251,7 +252,7 @@ impl EffectiveRouteEnvelope {
|
||||
self.provider,
|
||||
self.endpoint_fingerprint.as_deref(),
|
||||
);
|
||||
let declared_estimate = self.provider_live_pricing.as_ref().is_some_and(|quote| {
|
||||
let declared_quote = self.provider_live_pricing.as_ref().filter(|quote| {
|
||||
quote.provenance == codewhale_config::pricing::PricingProvenance::UserOverride
|
||||
&& self
|
||||
.endpoint_fingerprint
|
||||
@@ -269,6 +270,7 @@ impl EffectiveRouteEnvelope {
|
||||
.is_some()
|
||||
})
|
||||
});
|
||||
let declared_estimate = declared_quote.is_some();
|
||||
match self.billing_mode {
|
||||
RouteBillingMode::Subscription | RouteBillingMode::Local => {
|
||||
return TurnCostAudit::unpriced(crate::pricing::UnpricedReason::NotMoneyMetered);
|
||||
@@ -282,7 +284,12 @@ impl EffectiveRouteEnvelope {
|
||||
}
|
||||
// The OpenRouter model catalog does not identify a pinned upstream's
|
||||
// price. An endpoint match alone must not promote that aggregate rate.
|
||||
if self.provider == ApiProvider::Openrouter && self.openrouter_vendor.is_some() {
|
||||
// An operator's own declared rate for this exact route is not the
|
||||
// aggregate catalog, so it still prices the pinned turn.
|
||||
if self.provider == ApiProvider::Openrouter
|
||||
&& self.openrouter_vendor.is_some()
|
||||
&& !declared_quote.is_some_and(|quote| quote.carries_rates())
|
||||
{
|
||||
return TurnCostAudit::unpriced(crate::pricing::UnpricedReason::RoutingDependentPrice);
|
||||
}
|
||||
crate::pricing::audit_turn_cost_for_route_on_endpoint_for_identity_at(
|
||||
@@ -2262,6 +2269,151 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
const DECLARED_OPENROUTER_MODEL: &str = "synthetic/declared-model";
|
||||
|
||||
/// An OpenRouter config on the official endpoint with one
|
||||
/// `[[custom_models]]` row; `extra` adds fields such as `cost`.
|
||||
fn openrouter_declared_config(extra: &str, vendor: Option<&str>) -> crate::config::Config {
|
||||
let vendor = vendor.map_or_else(String::new, |vendor| format!("vendor = \"{vendor}\"\n"));
|
||||
toml::from_str(&format!(
|
||||
"provider = \"openrouter\"\ntelemetry = false\n\n\
|
||||
[[custom_models]]\nprovider = \"openrouter\"\n\
|
||||
base_url = \"{base}\"\nid = \"{DECLARED_OPENROUTER_MODEL}\"\n{extra}\n\n\
|
||||
[providers.openrouter]\nbase_url = \"{base}\"\n\
|
||||
api_key = \"fixture-not-a-provider-credential\"\n{vendor}",
|
||||
base = crate::config::DEFAULT_OPENROUTER_BASE_URL,
|
||||
))
|
||||
.expect("declared OpenRouter config")
|
||||
}
|
||||
|
||||
fn declared_openrouter_client(
|
||||
config: &crate::config::Config,
|
||||
) -> crate::client::CodewhaleClient {
|
||||
let route = crate::route_runtime::resolve_runtime_route(
|
||||
config,
|
||||
ApiProvider::Openrouter,
|
||||
Some(DECLARED_OPENROUTER_MODEL),
|
||||
)
|
||||
.expect("declared OpenRouter route");
|
||||
crate::client::CodewhaleClient::from_candidate(config, &route.candidate)
|
||||
.expect("declared OpenRouter client")
|
||||
}
|
||||
|
||||
/// #6690: a `[[custom_models]]` row declared only to add a model (no
|
||||
/// rates) used to freeze a rate-less quote on the main turn and hide the
|
||||
/// endpoint's own catalog price. A declared rate still wins, and with no
|
||||
/// catalog row the rate-less declaration stays frozen so no same-named
|
||||
/// bundled price can fill it.
|
||||
#[test]
|
||||
fn main_turn_rateless_declaration_yields_to_the_endpoint_catalog_price() {
|
||||
let _env = crate::test_support::lock_test_env();
|
||||
let home = tempfile::tempdir().expect("isolated catalog home");
|
||||
let _home = crate::test_support::EnvVarGuard::set("CODEWHALE_HOME", home.path());
|
||||
let _reset = ProviderCatalogTestReset;
|
||||
crate::provider_catalog_live::reset_cache_for_test();
|
||||
let _live = crate::provider_lake::lock_live_snapshot();
|
||||
crate::provider_lake::clear_live_snapshot();
|
||||
|
||||
let fingerprint = endpoint_fingerprint(crate::config::DEFAULT_OPENROUTER_BASE_URL)
|
||||
.expect("official endpoint");
|
||||
let now = u64::try_from(Utc::now().timestamp()).expect("timestamp");
|
||||
let rateless = declared_openrouter_client(&openrouter_declared_config("", None));
|
||||
let declared = declared_openrouter_client(&openrouter_declared_config(
|
||||
"cost = { input = 0.4, output = 1.6 }",
|
||||
None,
|
||||
));
|
||||
let quote_for = |client: &crate::client::CodewhaleClient| {
|
||||
crate::client::main_turn_pricing_quote_at(
|
||||
Some(client),
|
||||
ApiProvider::Openrouter,
|
||||
"openrouter",
|
||||
DECLARED_OPENROUTER_MODEL,
|
||||
&fingerprint,
|
||||
now,
|
||||
)
|
||||
.expect("a frozen main-turn quote")
|
||||
};
|
||||
|
||||
// No catalog row yet: the rate-less declaration is frozen as-is.
|
||||
let frozen = quote_for(&rateless);
|
||||
assert_eq!(
|
||||
frozen.provenance,
|
||||
codewhale_config::pricing::PricingProvenance::UserOverride
|
||||
);
|
||||
assert!(!frozen.carries_rates());
|
||||
|
||||
crate::provider_catalog_live::record_success(priced_provider_delta(
|
||||
"openrouter",
|
||||
DECLARED_OPENROUTER_MODEL,
|
||||
&fingerprint,
|
||||
now,
|
||||
));
|
||||
let catalog = quote_for(&rateless);
|
||||
assert_eq!(
|
||||
catalog.provenance,
|
||||
codewhale_config::pricing::PricingProvenance::ProviderLive
|
||||
);
|
||||
assert_eq!(catalog.input_per_million.as_deref(), Some("1.25"));
|
||||
assert_eq!(catalog.output_per_million.as_deref(), Some("5"));
|
||||
assert_eq!(catalog.cache_read_per_million.as_deref(), Some("0.25"));
|
||||
|
||||
let explicit = quote_for(&declared);
|
||||
assert_eq!(
|
||||
explicit.provenance,
|
||||
codewhale_config::pricing::PricingProvenance::UserOverride
|
||||
);
|
||||
assert_eq!(explicit.input_per_million.as_deref(), Some("0.4"));
|
||||
assert_eq!(explicit.output_per_million.as_deref(), Some("1.6"));
|
||||
}
|
||||
|
||||
/// #6690 review: a pinned OpenRouter vendor blocks the aggregate catalog
|
||||
/// price, but an operator's own declared rate for the exact route is not
|
||||
/// that aggregate and must still price the turn.
|
||||
#[test]
|
||||
fn openrouter_vendor_pin_is_priced_by_an_explicit_declared_rate() {
|
||||
let _env = crate::test_support::lock_test_env();
|
||||
let home = tempfile::tempdir().expect("isolated catalog home");
|
||||
let _home = crate::test_support::EnvVarGuard::set("CODEWHALE_HOME", home.path());
|
||||
let _reset = ProviderCatalogTestReset;
|
||||
crate::provider_catalog_live::reset_cache_for_test();
|
||||
let _live = crate::provider_lake::lock_live_snapshot();
|
||||
crate::provider_lake::clear_live_snapshot();
|
||||
|
||||
let usage = Usage {
|
||||
input_tokens: 1_000_000,
|
||||
output_tokens: 1_000_000,
|
||||
..Usage::default()
|
||||
};
|
||||
let capture = |extra: &str| {
|
||||
EffectiveRouteEnvelope::capture(
|
||||
Some(&openrouter_declared_config(extra, Some("cerebras"))),
|
||||
ApiProvider::Openrouter,
|
||||
"openrouter",
|
||||
DECLARED_OPENROUTER_MODEL,
|
||||
Some(crate::config::DEFAULT_OPENROUTER_BASE_URL),
|
||||
Utc::now(),
|
||||
)
|
||||
};
|
||||
|
||||
let declared = capture("cost = { input = 0.4, output = 1.6 }");
|
||||
assert_eq!(declared.openrouter_vendor.as_deref(), Some("cerebras"));
|
||||
let audit = declared.audit(&usage);
|
||||
assert_eq!(audit.unpriced_reason, None, "{audit:?}");
|
||||
assert_eq!(
|
||||
audit.provenance,
|
||||
Some(codewhale_config::pricing::PricingProvenance::UserOverride)
|
||||
);
|
||||
let usd = audit.estimate.expect("declared estimate").usd;
|
||||
assert!((usd - 2.0).abs() < 1e-12, "{usd}");
|
||||
|
||||
// A rate-less declaration is no explicit rate: the pin still wins.
|
||||
let rateless = capture("");
|
||||
assert_eq!(
|
||||
rateless.audit(&usage).unpriced_reason,
|
||||
Some(crate::pricing::UnpricedReason::RoutingDependentPrice)
|
||||
);
|
||||
}
|
||||
|
||||
struct ProviderCatalogTestReset;
|
||||
|
||||
impl Drop for ProviderCatalogTestReset {
|
||||
|
||||
@@ -203,6 +203,20 @@ where
|
||||
}
|
||||
|
||||
impl ProviderLivePricingQuote {
|
||||
/// Whether the quote names at least one rate. A `[[custom_models]]` row
|
||||
/// declared only to add a model to a roster freezes a rate-less quote.
|
||||
#[must_use]
|
||||
pub(crate) fn carries_rates(&self) -> bool {
|
||||
[
|
||||
&self.input_per_million,
|
||||
&self.output_per_million,
|
||||
&self.cache_read_per_million,
|
||||
&self.cache_write_per_million,
|
||||
]
|
||||
.into_iter()
|
||||
.any(Option::is_some)
|
||||
}
|
||||
|
||||
fn is_structurally_valid(&self) -> bool {
|
||||
self.pricing_for_route(
|
||||
self.provider,
|
||||
@@ -1385,6 +1399,21 @@ pub(crate) fn configured_dispatch_pricing_quote_at(
|
||||
)
|
||||
}
|
||||
|
||||
/// Pick the dispatch quote from an operator declaration and the endpoint's
|
||||
/// catalog. A declared rate wins. A declaration with no rates yields to the
|
||||
/// catalog's price for the same exact endpoint (#6690), and is kept (frozen,
|
||||
/// rate-less) only when the catalog has none, so a same-named bundled price
|
||||
/// still cannot fill the gap.
|
||||
pub(crate) fn declared_or_catalog_quote(
|
||||
declared: Option<ProviderLivePricingQuote>,
|
||||
catalog: impl FnOnce() -> Option<ProviderLivePricingQuote>,
|
||||
) -> Option<ProviderLivePricingQuote> {
|
||||
match declared {
|
||||
Some(quote) if quote.carries_rates() => Some(quote),
|
||||
declared => catalog().or(declared),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn fresh_dispatch_pricing_quote_at(
|
||||
provider: ApiProvider,
|
||||
provider_identity: &str,
|
||||
|
||||
Reference in new issue
Block a user