Compare commits

...
Author SHA1 Message Date
Vasanthdev2004 a454931e66 fix(reasoning): address round-2 review — accessor deep-copy, max tier, provenance
- EffortControl/BudgetControl now return a cloned Control, so a caller holding a
  Capability that shares state with the catalog cannot mutate Min/Max pointers or
  the Values slice through an accessor result (CodeRabbit). Lookup already deep
  -copies; this closes the accessor path too.
- Add modelregistry.ReasoningEffortMax ("max") + accept it in ValidReasoningEffort,
  so the snapshot's "max" tier (e.g. gpt-5.1-codex-max, newer Anthropic) has a
  constant to map to. No curated model lists it yet.
- Snapshot provenance: add a _fetched date and a regeneration recipe (documented
  in catalog.go), so refreshes are reproducible/auditable.
- Document that Lookup takes the API model id (wire name), not the friendly
  registry id, and that the match is exact (no case-fold/prefix-strip).
- Record the source-of-truth intent in the package doc: this catalog is the
  intended authoritative source; modelregistry's name-pattern inference is demoted
  to a fallback when it is wired in (a later change). Rename the `cap` builtin
  shadow to `entry`.

Tests pin the accessor deep-copy isolation.
2026-06-28 14:09:09 +05:30
Vasanthdev2004 35145a519b fix(reasoning): address review — presence-aware budget bounds, deep-copy lookup, first-party-only snapshot
- Control.Min/Max are now *int so an explicit bound is distinct from an omitted
  one: Gemini's `min: 0` ("thinking can be disabled") and Anthropic's missing
  max (unbounded) are preserved instead of both collapsing to 0. Budget() becomes
  BudgetControl() returning the control so callers read the pointers.
- Catalog.Lookup returns a deep copy (clone) of the matched Capability, so a
  caller can no longer mutate the shared embedded catalog through the returned
  Controls / Values slices or budget pointers.
- Drop the google-vertex section from the snapshot: it re-hosts third-party MaaS
  models (Anthropic/Moonshot/OpenAI/DeepSeek/Qwen/Meta), violating first-party
  -only. google (AI Studio) already covers first-party Gemini, so the gemini
  provider kind resolves there. Snapshot 158 -> 123 models, 16.8 KB.

Tests pin the explicit min: 0 preservation, the nil max on budget-only Anthropic
models, and that a mutation of a Lookup result does not leak into the catalog.
2026-06-28 13:34:05 +05:30
Vasanthdev2004 1282a23ab1 feat(reasoning): typed per-provider reasoning capability from a models.dev catalog
Providers describe reasoning control with incompatible concepts — OpenAI a
discrete effort enum (split by minor version), Anthropic and Gemini 2.5 a
thinking-token budget, some models only a thinking on/off toggle — which a flat
effort string cannot represent and a model-name heuristic cannot detect.

Add internal/reasoning: a Capability type (a model's reasoning flag plus its
typed Controls — effort/budget_tokens/toggle) and a Catalog that looks capability
up by provider + api model id. The data is an embedded, trimmed snapshot of the
community capability database (first-party providers only; routers excluded since
they carry generic/stale option lists), so detection is offline and exact instead
of guessed.

The package depends only on the standard library, so the model registry and
provider adapters can consume it without an import cycle. This change is purely
additive — nothing is wired into the live resolution path yet.

Tests pin the per-version ground truth (gpt-5 minimal-not-none, gpt-5.1
none-not-minimal, gpt-5.1-codex-max xhigh, gpt-5-pro locked to high; Anthropic
budget-only legacy vs effort-only newer with the budget removed; Gemini 2.5
budget/toggle vs Gemini 3 effort) and that every reasoning model Zero ships today
is covered by the snapshot.
2026-06-28 13:22:55 +05:30
5 changed files with 1582 additions and 1 deletions

No files matched your search

+5 -1
View File
@@ -25,6 +25,10 @@ const (
ReasoningEffortMedium ReasoningEffort = "medium"
ReasoningEffortHigh ReasoningEffort = "high"
ReasoningEffortXHigh ReasoningEffort = "xhigh"
// ReasoningEffortMax is the top tier some providers expose above xhigh (e.g.
// newer Anthropic models). Defined so a capability catalog that lists "max"
// has a constant to map to; no curated model lists it yet.
ReasoningEffortMax ReasoningEffort = "max"
)
type ModelCapability string
@@ -406,7 +410,7 @@ func ValidRuntimeProviderKind(provider ProviderKind) bool {
func ValidReasoningEffort(effort ReasoningEffort) bool {
switch effort {
case ReasoningEffortNone, ReasoningEffortMinimal, ReasoningEffortLow, ReasoningEffortMedium, ReasoningEffortHigh, ReasoningEffortXHigh:
case ReasoningEffortNone, ReasoningEffortMinimal, ReasoningEffortLow, ReasoningEffortMedium, ReasoningEffortHigh, ReasoningEffortXHigh, ReasoningEffortMax:
return true
default:
return false
+147
View File
@@ -0,0 +1,147 @@
// Package reasoning models how each provider's models expose reasoning-effort
// control. Providers disagree on the concept — OpenAI uses a discrete effort
// enum, Anthropic and Gemini 2.5 use a thinking-token budget, some models only
// toggle thinking on/off — so a single flat effort string cannot describe them.
//
// Capability is the typed, per-model description of that control, sourced from a
// community capability catalog (models.dev). It is the data the rest of Zero
// consults to decide which reasoning tiers a model actually supports, replacing
// model-name guessing. This package depends only on the standard library, so the
// model registry and provider adapters can import it without a cycle.
//
// Source of truth: this catalog is intended to become the authoritative reasoning
// -capability source. When it is wired into the live path, modelregistry's
// name-pattern inference (reasoningEffortsForModelName) is demoted to a fallback
// for models the catalog does not cover; the catalog wins where it has an entry.
// This commit is additive groundwork — nothing consumes it yet — so there is no
// behavioral divergence until that wiring lands.
package reasoning
import "strings"
// ControlKind enumerates how a model exposes reasoning control. The values
// mirror the models.dev reasoning_options[].type field.
type ControlKind string
const (
// ControlEffort is a discrete effort enum (OpenAI reasoning_effort, Gemini 3
// thinkingLevel, newer Claude output_config.effort).
ControlEffort ControlKind = "effort"
// ControlBudget is a thinking-token budget (Gemini 2.5 thinkingBudget, legacy
// Claude thinking.budget_tokens).
ControlBudget ControlKind = "budget_tokens"
// ControlToggle is an on/off thinking switch with no levels.
ControlToggle ControlKind = "toggle"
)
// Control is one reasoning control a model accepts. For an effort control,
// Values lists the accepted tiers ordered weakest to strongest. For a budget
// control, Min/Max bound the thinking-token budget; each is nil when the
// provider does not bound that side, kept distinct from an explicit 0 (Gemini
// uses min: 0 to mean "thinking can be disabled", which a plain int would lose).
type Control struct {
Kind ControlKind `json:"type"`
Values []string `json:"values,omitempty"`
Min *int `json:"min,omitempty"`
Max *int `json:"max,omitempty"`
}
// clone returns a deep copy of the control so the embedded catalog cannot be
// mutated through a returned Control's slice or budget pointers.
func (ctrl Control) clone() Control {
out := Control{Kind: ctrl.Kind}
if ctrl.Values != nil {
out.Values = append([]string(nil), ctrl.Values...)
}
if ctrl.Min != nil {
min := *ctrl.Min
out.Min = &min
}
if ctrl.Max != nil {
max := *ctrl.Max
out.Max = &max
}
return out
}
// Capability is the reasoning capability of a single model: whether it reasons
// at all, and through which controls. A model may carry more than one control
// (e.g. a newer Claude model exposes both an effort enum and a token budget); a
// reasoning model with no controls reasons but exposes no knob (always-on).
type Capability struct {
Reasoning bool `json:"reasoning"`
Controls []Control `json:"reasoning_options,omitempty"`
}
// clone returns a deep copy of the capability, cloning each control, so a caller
// cannot mutate the shared catalog through the returned value's slices.
func (c Capability) clone() Capability {
out := Capability{Reasoning: c.Reasoning}
if c.Controls != nil {
out.Controls = make([]Control, len(c.Controls))
for i, ctrl := range c.Controls {
out.Controls[i] = ctrl.clone()
}
}
return out
}
// Supported reports whether the model performs any reasoning.
func (c Capability) Supported() bool { return c.Reasoning }
// EffortControl returns the model's effort control and whether it has one. The
// returned Control is a deep copy, so a caller cannot mutate the shared catalog
// through its Values slice.
func (c Capability) EffortControl() (Control, bool) {
for _, ctrl := range c.Controls {
if ctrl.Kind == ControlEffort {
return ctrl.clone(), true
}
}
return Control{}, false
}
// EffortValues returns the ordered effort tiers the model accepts, or nil when
// it has no effort control (a budget- or toggle-only model, or a non-reasoning
// model).
func (c Capability) EffortValues() []string {
if ctrl, ok := c.EffortControl(); ok {
return ctrl.Values
}
return nil
}
// SupportsEffort reports whether tier is one of the model's accepted effort
// values (case-insensitive).
func (c Capability) SupportsEffort(tier string) bool {
tier = strings.ToLower(strings.TrimSpace(tier))
for _, v := range c.EffortValues() {
if strings.ToLower(v) == tier {
return true
}
}
return false
}
// BudgetControl returns the model's token-budget control and whether it has one.
// The returned Control is a deep copy: its Min/Max are independent pointers, so a
// caller cannot mutate the shared catalog through them. Min/Max are nil when that
// bound is unspecified (a real 0, e.g. Gemini's min: 0, is a non-nil pointer to 0).
func (c Capability) BudgetControl() (Control, bool) {
for _, ctrl := range c.Controls {
if ctrl.Kind == ControlBudget {
return ctrl.clone(), true
}
}
return Control{}, false
}
// HasControl reports whether the model exposes a control of the given kind.
func (c Capability) HasControl(kind ControlKind) bool {
for _, ctrl := range c.Controls {
if ctrl.Kind == kind {
return true
}
}
return false
}
+99
View File
@@ -0,0 +1,99 @@
package reasoning
import (
_ "embed"
"encoding/json"
"fmt"
"strings"
)
// modelsdev_snapshot.json is a trimmed snapshot of https://models.dev/api.json:
// for a set of first-party providers, each model's api id mapped to its
// reasoning capability (the `reasoning` flag and `reasoning_options`). Routers
// (OpenRouter/Azure/etc.) are intentionally excluded — they carry generic or
// stale option lists, so a lookup resolves to the first-party provider instead.
//
// Provenance is recorded in the snapshot's `_source`/`_fetched` header fields.
// To regenerate: fetch api.json, then for each provider in
// {anthropic, openai, google, xai, groq, deepseek} keep `{reasoning,
// reasoning_options}` per api id, and bump `_fetched`. Runtime auto-refresh is a
// separate change.
//
//go:embed modelsdev_snapshot.json
var snapshotBytes []byte
// snapshotDoc is the on-disk shape of the embedded snapshot.
type snapshotDoc struct {
Providers map[string]map[string]Capability `json:"providers"`
}
// Catalog is a reasoning-capability lookup keyed by provider slug and api model
// id. It is read-only after construction.
type Catalog struct {
byProvider map[string]map[string]Capability
}
// ParseCatalog builds a Catalog from a models.dev-style snapshot document.
func ParseCatalog(data []byte) (Catalog, error) {
var doc snapshotDoc
if err := json.Unmarshal(data, &doc); err != nil {
return Catalog{}, fmt.Errorf("reasoning: parse catalog: %w", err)
}
if len(doc.Providers) == 0 {
return Catalog{}, fmt.Errorf("reasoning: catalog has no providers")
}
return Catalog{byProvider: doc.Providers}, nil
}
// embedded is the catalog parsed from the bundled snapshot. A malformed embed is
// a build/release error, so it panics at init rather than failing silently.
var embedded = mustParseEmbedded()
func mustParseEmbedded() Catalog {
c, err := ParseCatalog(snapshotBytes)
if err != nil {
panic("reasoning: invalid embedded snapshot: " + err.Error())
}
return c
}
// Embedded returns the catalog parsed from the bundled models.dev snapshot.
func Embedded() Catalog { return embedded }
// Lookup returns the reasoning capability for a model identified by its Zero
// provider kind and the API model id — the provider's wire name (e.g.
// "claude-opus-4-1-20250805"), NOT the friendly registry id ("claude-opus-4.1").
// The match is exact: the api model id is only trimmed, not case-folded, and no
// router prefixes ("openai/…") or suffixes (":cloud") are stripped. ok is false
// when the provider or model is not in the snapshot, in which case the caller
// falls back to its next capability source.
func (c Catalog) Lookup(provider, apiModel string) (Capability, bool) {
apiModel = strings.TrimSpace(apiModel)
if apiModel == "" {
return Capability{}, false
}
for _, slug := range providerSlugs(provider) {
models, ok := c.byProvider[slug]
if !ok {
continue
}
if entry, ok := models[apiModel]; ok {
// Deep copy so callers cannot mutate the shared catalog through the
// returned Controls / Values slices or budget pointers.
return entry.clone(), true
}
}
return Capability{}, false
}
// providerSlugs maps a Zero provider kind to the models.dev provider slug to
// look up. Gemini and Google both resolve to "google" (AI Studio), the
// first-party source for Gemini models.
func providerSlugs(provider string) []string {
switch strings.ToLower(strings.TrimSpace(provider)) {
case "gemini":
return []string{"google"}
default:
return []string{strings.ToLower(strings.TrimSpace(provider))}
}
}
+249
View File
@@ -0,0 +1,249 @@
package reasoning
import "testing"
func TestEmbeddedCatalogLoads(t *testing.T) {
c := Embedded()
if len(c.byProvider) == 0 {
t.Fatal("embedded catalog is empty")
}
if _, ok := c.Lookup("openai", "gpt-5"); !ok {
t.Fatal("embedded catalog missing a well-known model (gpt-5)")
}
}
func TestParseCatalogRejectsEmpty(t *testing.T) {
if _, err := ParseCatalog([]byte(`{}`)); err == nil {
t.Fatal("expected an error for a catalog with no providers")
}
if _, err := ParseCatalog([]byte(`not json`)); err == nil {
t.Fatal("expected an error for malformed json")
}
}
func TestGroundTruthOpenAI(t *testing.T) {
c := Embedded()
cases := []struct {
api string
reason bool
kind ControlKind
values []string
}{
{"gpt-5", true, ControlEffort, []string{"minimal", "low", "medium", "high"}},
{"gpt-5-codex", true, ControlEffort, []string{"low", "medium", "high"}},
{"gpt-5-pro", true, ControlEffort, []string{"high"}},
{"gpt-5.1", true, ControlEffort, []string{"none", "low", "medium", "high"}},
{"gpt-5.1-codex-max", true, ControlEffort, []string{"low", "medium", "high", "xhigh"}},
{"o3", true, ControlEffort, []string{"low", "medium", "high"}},
}
for _, tc := range cases {
entry, ok := c.Lookup("openai", tc.api)
if !ok {
t.Errorf("%s: not found", tc.api)
continue
}
if entry.Supported() != tc.reason {
t.Errorf("%s: Supported=%v want %v", tc.api, entry.Supported(), tc.reason)
}
if !equalStrings(entry.EffortValues(), tc.values) {
t.Errorf("%s: efforts=%v want %v", tc.api, entry.EffortValues(), tc.values)
}
}
// Non-reasoning models report reasoning:false and expose no effort.
for _, api := range []string{"gpt-4o", "gpt-4o-mini", "gpt-4.1", "gpt-4-turbo"} {
entry, ok := c.Lookup("openai", api)
if !ok {
t.Errorf("%s: not found", api)
continue
}
if entry.Supported() {
t.Errorf("%s: Supported=true, want false (non-reasoning)", api)
}
if entry.EffortValues() != nil {
t.Errorf("%s: efforts=%v, want nil", api, entry.EffortValues())
}
}
// Version splits: minimal vs none are not interchangeable.
gpt5, _ := c.Lookup("openai", "gpt-5")
if !gpt5.SupportsEffort("minimal") || gpt5.SupportsEffort("none") {
t.Error("gpt-5 should support minimal, not none")
}
gpt51, _ := c.Lookup("openai", "gpt-5.1")
if gpt51.SupportsEffort("minimal") || !gpt51.SupportsEffort("none") {
t.Error("gpt-5.1 should support none, not minimal")
}
}
func TestGroundTruthAnthropic(t *testing.T) {
c := Embedded()
// Legacy / budget-only models: a thinking-token budget, no effort enum. The
// budget carries an explicit min (1024) and no max — Max stays nil.
for _, api := range []string{
"claude-opus-4-1-20250805", "claude-sonnet-4-5-20250929", "claude-haiku-4-5-20251001",
} {
entry, ok := c.Lookup("anthropic", api)
if !ok {
t.Errorf("%s: not found", api)
continue
}
ctrl, ok := entry.BudgetControl()
if !ok {
t.Errorf("%s: expected a budget control", api)
continue
}
if ctrl.Min == nil || *ctrl.Min != 1024 {
t.Errorf("%s: budget Min=%v, want non-nil 1024", api, ctrl.Min)
}
if ctrl.Max != nil {
t.Errorf("%s: budget Max=%d, want nil (unbounded)", api, *ctrl.Max)
}
if _, ok := entry.EffortControl(); ok {
t.Errorf("%s: did not expect an effort control", api)
}
}
// Newer models: native effort enum with xhigh and the budget control removed.
opus48, ok := c.Lookup("anthropic", "claude-opus-4-8")
if !ok {
t.Fatal("claude-opus-4-8 not found")
}
if !equalStrings(opus48.EffortValues(), []string{"low", "medium", "high", "xhigh", "max"}) {
t.Errorf("opus-4-8 efforts=%v", opus48.EffortValues())
}
if _, ok := opus48.BudgetControl(); ok {
t.Error("opus-4-8 should not carry a budget control (removed)")
}
}
func TestGroundTruthGemini(t *testing.T) {
c := Embedded()
pro, ok := c.Lookup("gemini", "gemini-2.5-pro") // gemini kind -> google slug
if !ok {
t.Fatal("gemini-2.5-pro not found via the gemini provider kind")
}
ctrl, ok := pro.BudgetControl()
if !ok || ctrl.Min == nil || *ctrl.Min != 128 || ctrl.Max == nil || *ctrl.Max != 32768 {
t.Errorf("gemini-2.5-pro budget=%+v ok=%v, want min=128 max=32768", ctrl, ok)
}
if _, ok := pro.EffortControl(); ok {
t.Error("gemini-2.5-pro should be budget-only, no effort control")
}
// gemini-2.5-flash can disable thinking: a toggle plus a budget whose explicit
// min: 0 must survive (a plain int + omitempty would drop it).
flash, _ := c.Lookup("google", "gemini-2.5-flash")
if !flash.HasControl(ControlToggle) {
t.Error("gemini-2.5-flash should expose a toggle (thinking can be disabled)")
}
if fctrl, ok := flash.BudgetControl(); !ok || fctrl.Min == nil || *fctrl.Min != 0 {
t.Errorf("gemini-2.5-flash budget Min=%v, want a non-nil pointer to 0", fctrl.Min)
}
// Gemini 3.x switched to an effort enum.
if g3, ok := c.Lookup("google", "gemini-3-pro-preview"); ok {
if _, ok := g3.EffortControl(); !ok {
t.Error("gemini-3-pro-preview should expose an effort control")
}
}
}
// TestCoversZeroShippedReasoningModels pins that every reasoning model Zero
// currently ships resolves in the catalog (by its api model id), so the
// models.dev fallback actually covers Zero's catalog rather than just well-known
// ids. The api ids mirror internal/modelregistry's curated entries.
func TestCoversZeroShippedReasoningModels(t *testing.T) {
c := Embedded()
shipped := []struct {
provider, api string
wantKind ControlKind
}{
{"anthropic", "claude-opus-4-1-20250805", ControlBudget},
{"anthropic", "claude-sonnet-4-5-20250929", ControlBudget},
{"anthropic", "claude-haiku-4-5-20251001", ControlBudget},
{"google", "gemini-2.5-pro", ControlBudget},
{"google", "gemini-2.5-flash", ControlToggle},
{"google", "gemini-2.5-flash-lite", ControlToggle},
}
for _, m := range shipped {
entry, ok := c.Lookup(m.provider, m.api)
if !ok {
t.Errorf("%s/%s: not covered by the snapshot", m.provider, m.api)
continue
}
if !entry.Supported() {
t.Errorf("%s/%s: Supported=false, want a reasoning model", m.provider, m.api)
}
if !entry.HasControl(m.wantKind) {
t.Errorf("%s/%s: missing expected control %q (controls=%+v)", m.provider, m.api, m.wantKind, entry.Controls)
}
}
}
// TestLookupReturnsDeepCopy pins that a caller cannot corrupt the shared
// embedded catalog by mutating a returned Capability's slices.
func TestLookupReturnsDeepCopy(t *testing.T) {
c := Embedded()
first, ok := c.Lookup("openai", "gpt-5")
if !ok || len(first.Controls) == 0 || len(first.Controls[0].Values) == 0 {
t.Fatal("setup: gpt-5 should have an effort control with values")
}
first.Controls[0].Values[0] = "MUTATED"
first.Controls[0].Kind = "MUTATED"
second, _ := c.Lookup("openai", "gpt-5")
if second.Controls[0].Values[0] == "MUTATED" || second.Controls[0].Kind == "MUTATED" {
t.Error("Lookup leaked a shared reference; a caller's mutation reached the catalog")
}
}
// TestAccessorsReturnDeepCopies pins that EffortControl/BudgetControl hand back
// independent copies, so a caller holding a Capability that shares state with the
// catalog cannot corrupt it by mutating an accessor result's slice or pointers.
func TestAccessorsReturnDeepCopies(t *testing.T) {
min := 100
c := Capability{Reasoning: true, Controls: []Control{
{Kind: ControlEffort, Values: []string{"low", "high"}},
{Kind: ControlBudget, Min: &min},
}}
eff, _ := c.EffortControl()
eff.Values[0] = "MUTATED"
if c.Controls[0].Values[0] != "low" {
t.Error("EffortControl leaked the Values slice")
}
bud, _ := c.BudgetControl()
*bud.Min = 999
if *c.Controls[1].Min != 100 {
t.Error("BudgetControl leaked the Min pointer")
}
}
func TestLookupMissFallsThrough(t *testing.T) {
c := Embedded()
if _, ok := c.Lookup("openai", "totally-made-up-model"); ok {
t.Error("unknown model should miss")
}
if _, ok := c.Lookup("no-such-provider", "gpt-5"); ok {
t.Error("unknown provider should miss")
}
if _, ok := c.Lookup("openai", " "); ok {
t.Error("blank api model should miss")
}
}
func equalStrings(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
File diff suppressed because it is too large. Load diff