mirror of
https://github.com/esengine/DeepSeek-Reasonix.git
synced 2026-11-07 10:45:59 -10:00
bench(realrepo): long tasks over a pinned snapshot of this repository
The e2e fixtures finish in 5-15 requests, while a real task in this tree took
deepseek-flash 38. A change aimed at cutting round trips cannot show up on
the short suite, so this adds four long tasks whose workdir is a git archive
of commit 42636ddcb (pricing, ablation, agentgraph and five stdlib-only
leaves) plus a trimmed REASONIX.md, a Makefile, a module README and a small
benchctl consumer:
- rt-cache-write-rate: a CacheWrite rate threaded through cost, fingerprint,
card identity and the catalog, plus an Anthropic row that uses it
- rt-stale-skills: Markdown-only repair of three project skills with no
recognised verifier, the shape behind #13293's lingering stale_verification
- rt-rate-lookup-signatures: five lookups moved to (RateCard, bool),
CatalogKey and ModelRef across about 20 call sites, most in _test.go
- rt-plan-ablation-module: a five-step plan-first task signed off per step
snapshot.sh rebuilds every workdir from the pin and --check detects drift.
One pinned test compares CheckedOn with the wall clock and would fail every
workdir from 2026-11-20; base/ anchors it at the snapshot day instead.
Each task ships a solution/; the corpus is registered with the authoring
guards, which check that every seed fails its grader and every solution
passes. A deepseek-flash baseline (2 trials) solved 8/8 at 13-31 requests
once two grader false negatives in rt-stale-skills were fixed; the README
records it, along with a multi_edit prompt arm and a complete_step
description arm that did not move the request count.
This commit is contained in:
zumac committed
1 parent
0b8de9e636
commit
7e92388102
257 files changed
+25900
-4
No files matched your search
@@ -26,6 +26,7 @@
|
||||
*.xml text eol=lf
|
||||
*.yaml text eol=lf
|
||||
*.yml text eol=lf
|
||||
Makefile text eol=lf
|
||||
.githooks/** text eol=lf
|
||||
benchmarks/**/*.sh text eol=lf
|
||||
internal/runtime/agent/testdata/perseveration/** -text
|
||||
@@ -1,6 +1,6 @@
|
||||
# Reasonix Benchmarks
|
||||
|
||||
Five harnesses live under `benchmarks/`; `cmd/e2ebench` also exposes a
|
||||
Six harnesses live under `benchmarks/`; `cmd/e2ebench` also exposes a
|
||||
SWE-bench Verified mode:
|
||||
|
||||
- `e2e/` — the committed end-to-end task suite, driven by
|
||||
@@ -19,6 +19,10 @@ SWE-bench Verified mode:
|
||||
- `fanout-width/` — a shape-instructed fleet whose members share nothing, for
|
||||
pricing what running them at the same time is worth. See
|
||||
[Pricing the run graph](#pricing-the-run-graph) below.
|
||||
- `realrepo/` — long tasks over a pinned snapshot of this repository, for
|
||||
measuring changes that cut model round trips (13–31 requests per task on
|
||||
`deepseek-flash`). See
|
||||
[its README](realrepo/README.md).
|
||||
|
||||
## Directory layout
|
||||
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
# realrepo
|
||||
|
||||
Long tasks over a pinned snapshot of this repository, for measuring changes
|
||||
that aim to cut model round trips. The `benchmarks/e2e` tasks finish in about
|
||||
5–15 requests and ten seconds; a real task in this tree took a flash-class
|
||||
model 38 requests, most of them after a single read. A suite of short fixtures
|
||||
cannot show that gap moving, so these tasks are shaped like the real one:
|
||||
explore several packages, edit several files, verify, and recover from the
|
||||
verifier.
|
||||
|
||||
```sh
|
||||
go build -o bin/reasonix ./cmd/reasonix
|
||||
go run ./cmd/e2ebench -suite benchmarks/realrepo -bin "$PWD/bin/reasonix" -model deepseek-flash \
|
||||
-budget 0 -trials 2 -trajectories traj -json realrepo.json
|
||||
go run ./cmd/e2ebench -suite benchmarks/realrepo -bin "$PWD/bin/reasonix" -task rt-stale-skills
|
||||
```
|
||||
|
||||
The agent reads its provider config from `REASONIX_HOME` (default
|
||||
`~/.reasonix`); point it at a disposable one to keep benchmark sessions out of
|
||||
your own.
|
||||
|
||||
The default `-budget` (800k tokens) can stop the suite partway through, hence
|
||||
`-budget 0`. Under `-trials`, each trial keeps its own
|
||||
`<task>.trial-N.trajectory.jsonl`.
|
||||
|
||||
## The snapshot
|
||||
|
||||
Every `workdir/` is the same Go module (`module reasonix`, stdlib only, no
|
||||
network needed to build or test):
|
||||
|
||||
- **Pinned code.** `git archive` of commit `42636ddcb425` for
|
||||
`internal/contract/{pricing,ablation,agentgraph}` and
|
||||
`internal/base/{retrieval,rulelist,tokencount,neterr,nilutil}` — about 5k
|
||||
lines of real kernel code with its real tests and doc comments.
|
||||
- **Fixture-authored files** in `base/`: `go.mod`, a trimmed `REASONIX.md` (the
|
||||
project's own conventions, so the agent loads standing instructions the way
|
||||
it does here), a `Makefile` with `check`, a module `README.md`, the
|
||||
`cmd/benchctl` command that consumes `pricing` and `ablation` from outside
|
||||
their package, and one project skill.
|
||||
- **One pinned test anchored to the snapshot day.**
|
||||
`TestCatalogPricesHaveBeenCheckedRecently` compares each row's `CheckedOn`
|
||||
with the wall clock, which would fail every workdir 90 days after the oldest
|
||||
pinned date (2026-11-20). `base/` replaces that file with one that reads
|
||||
staleness at 2026-10-19 instead; nothing else in the pin is edited.
|
||||
- **Per-task overlay** in `overlay/<task-id>/`, applied last. Only
|
||||
`rt-stale-skills` has one.
|
||||
|
||||
`snapshot.sh` rebuilds every workdir from those three sources and
|
||||
`snapshot.sh --check` fails if a committed workdir has drifted from them. Edit
|
||||
`base/` or `overlay/`, never a workdir, then rerun the script. Moving the pin is
|
||||
a new suite version: results from two pins do not compare.
|
||||
|
||||
## Tasks
|
||||
|
||||
| Task | Kind | What it exercises |
|
||||
| --- | --- | --- |
|
||||
| `rt-cache-write-rate` | multi-file Go change, test, README | Adds a `CacheWrite` rate to `RateCard` and threads it through cost, fingerprint, card identity and the catalog, then prices Anthropic's `claude-sonnet-4-5` with it: a new `officialRates` row, documentation and checked-on constants, an endpoint host, and `benchctl rate`/`cost` output. The fingerprint has to stay byte-identical for every existing card, so the change to `PricingFingerprint` cannot be the obvious one. |
|
||||
| `rt-stale-skills` | documentation only, no recognised verifier | Three project skills name paths, functions, `make` targets and `benchctl` subcommands and flags that do not exist, and one contradicts the append-only rule in `REASONIX.md`. Fixing them takes reading `rates.go`, `ablation.go`, both `benchctl` files and the `Makefile`, then editing only Markdown — the shape behind #13293, where `stale_verification` stays owed after a skill edit because no check covers it. |
|
||||
| `rt-rate-lookup-signatures` | API migration with a likely compile-error loop | Changes `CurrentRate` to `(RateCard, bool)`, `LookupCatalog` to a `CatalogKey` struct, and `IsOfficialCard`, `MatchesCatalog` and `LiveTariff` to a `ModelRef`: about 20 call sites across 11 files. Most are in `_test.go` files and dereference the old pointer, so `go build ./...` can pass while `go vet` and `go test` fail. |
|
||||
| `rt-plan-ablation-module` | plan-mode with `complete_step` | Five numbered steps adding a `memory` ablation module, summaries, `ModuleHelp` and `benchctl arm -list`. The prompt opens with the host's own plan-first marker (`control.PlannerRouteMarker`), which routes the turn through the planner, and asks for each step to be signed off before the next. An existing test pins the `all` arm name and has to be updated, which adds a red-test step. |
|
||||
|
||||
Every grader stages hidden tests and checks after the run: `gofmt -l`, `go vet
|
||||
./...` and `go test ./...` for the Go tasks; for `rt-stale-skills`, every
|
||||
backticked path, identifier, `make` target and `benchctl` command line in each
|
||||
skill must resolve against the tree, and a SHA-256 manifest proves no Go source, `go.mod`,
|
||||
`Makefile` or `REASONIX.md` changed. Each task fails its pristine seed and
|
||||
passes its `solution/`, both checked by `cmd/e2ebench`'s corpus tests.
|
||||
|
||||
## Reading the numbers
|
||||
|
||||
```text
|
||||
Metric: requests per task
|
||||
Anchor: the trajectory digest's requests_by_source, summed over sources
|
||||
Population: every graded trial of a task that ran to its own stop
|
||||
Excludes: tasks the token budget skipped, and runs that hit timeout_sec
|
||||
```
|
||||
|
||||
Requests is the headline because round trips are what these tasks exist to
|
||||
move. Read it beside the solve rate and the report's round outcomes: a change
|
||||
that drops requests by giving up on verification has not cut a round trip, it
|
||||
has cut a check. `rt-stale-skills` is read for whether the run finishes at all
|
||||
and how many rounds go to re-verifying after its last write.
|
||||
|
||||
## Baseline
|
||||
|
||||
`deepseek-flash`, 2 trials per task, at studio `0b8de9e63`. Requests are the
|
||||
trajectory's executor requests; all 8 runs ran to their own stop.
|
||||
|
||||
| Task | Solved | Requests | `edit_file` calls | Edit rounds |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `rt-cache-write-rate` | 2/2 | 31, 28 | 21, 20 | 8, 8 |
|
||||
| `rt-plan-ablation-module` | 2/2 | 24, 30 | 10, 13 | 9, 7 |
|
||||
| `rt-rate-lookup-signatures` | 2/2 | 13, 15 | 10, 17 | 2, 6 |
|
||||
| `rt-stale-skills` | 2/2* | 20, 26 | 3, 2 | 3, 3 |
|
||||
|
||||
\* One run first graded as failed, on a glob (`*_rates_test.go`) and a
|
||||
placeholder (`benchctl <subcommand>`) that were both correct. The grader now
|
||||
resolves globs against the tree and skips placeholders, and that run counts
|
||||
as solved. The same task's first version failed a run for naming
|
||||
`CatalogEntry.EffectiveFrom` in order to warn against it; that check went too.
|
||||
|
||||
Reads and edits already batch: 20 edit calls fit into 8 rounds, and several
|
||||
`edit_file` calls to one file often share a round. Rounds that edit a file the
|
||||
previous round also edited are rare (0–2 per run), so steering same-file edits
|
||||
to `multi_edit` has little to save. On this suite's first version, a
|
||||
`ToolBatchPolicy` sentence asking for it drew no `multi_edit` call and 167
|
||||
requests against 158.
|
||||
|
||||
`rt-plan-ablation-module` spends five rounds per run on `complete_step` alone.
|
||||
A description change inviting the sign-off to share the next step's reply was
|
||||
taken up in 2 of 4 trials (4 of 5 sign-offs shared each time, against 0 of 20
|
||||
in control), and requests did not move: 30, 25, 21, 28 against 27, 23, 24, 30.
|
||||
At this spread, four trials per arm cannot show a saving of a few rounds.
|
||||
|
||||
## Gaps
|
||||
|
||||
- **Below the intended size, in places.** The tasks were designed for 25–50
|
||||
requests. After widening, the median is 25, but `rt-rate-lookup-signatures`
|
||||
still runs 13–15: the model greps every call site first and batches the edits,
|
||||
so even about 20 call sites cost few rounds, and no run hit a compile-error
|
||||
loop.
|
||||
- **Lexical grader.** `rt-stale-skills` judges Markdown by resolving what it
|
||||
names. It has misjudged correct text twice (see the baseline note); a new
|
||||
failure there should be read before it is counted.
|
||||
- **No git history.** The workdir is a plain directory: `git diff`, `git log`
|
||||
and `git status` have nothing to read, so runs that lean on them in this tree
|
||||
behave differently here.
|
||||
- **Plan marker and `-force-planner`.** `rt-plan-ablation-module` carries the
|
||||
plan-first marker in its prompt, so it engages the planner in every arm,
|
||||
including runs without `-force-planner`. Exclude it when comparing planner
|
||||
arms.
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: price-change
|
||||
description: Record a vendor's announced price change in the official price book without losing the rate it replaces.
|
||||
---
|
||||
|
||||
# Recording a price change
|
||||
|
||||
1. Find the vendor's rows in `officialRates` in
|
||||
`internal/contract/pricing/rates.go`. Each row is one model in one currency.
|
||||
2. **Append** a `rateGeneration` to the row's `Rates`; never edit an existing
|
||||
one. Set `From` to the first billed day (`YYYY-MM-DD`, UTC) when the change
|
||||
is announced ahead of time.
|
||||
3. Leave the earlier generation where it is: `SupersededRates` reads it so a
|
||||
stored price that matches it is recognised as ours and refreshed.
|
||||
4. Update the row's `CheckedOn` constant to the day you confirmed the page.
|
||||
5. Add a test beside `rates_test.go` that pins `CurrentRate` before and after
|
||||
the `From` day by replacing `today`.
|
||||
6. Run `make check`. `make pricecheck` needs the network and is run by hand.
|
||||
@@ -0,0 +1,17 @@
|
||||
.PHONY: check test vet fmt pricecheck
|
||||
|
||||
check: fmt vet test
|
||||
|
||||
fmt:
|
||||
@out="$$(gofmt -l .)"; if [ -n "$$out" ]; then echo "gofmt needed:"; echo "$$out"; exit 1; fi
|
||||
|
||||
vet:
|
||||
go vet ./...
|
||||
|
||||
test:
|
||||
go test ./...
|
||||
|
||||
# Reads each vendor's pricing page; this snapshot carries no network client.
|
||||
pricecheck:
|
||||
@echo "pricecheck: reaches each vendor's pricing page and needs the network; not available in this snapshot" >&2
|
||||
@exit 2
|
||||
@@ -0,0 +1,69 @@
|
||||
# Reasonix kernel snapshot
|
||||
|
||||
A trimmed slice of the Reasonix Go kernel: the contracts a benchmark report is
|
||||
priced and labelled with, a few leaf utilities, and `benchctl`, a small command
|
||||
that prints what those contracts answer.
|
||||
|
||||
```bash
|
||||
make check # gofmt, vet, test
|
||||
go run ./cmd/benchctl rate -model deepseek-flash -currency USD
|
||||
go run ./cmd/benchctl cost -provider mimo -model mimo-v2.5 -miss 1000000
|
||||
go run ./cmd/benchctl tariff -currency CNY -at 2026-10-12T02:00:00Z
|
||||
go run ./cmd/benchctl arm -ablate planner,evidence
|
||||
```
|
||||
|
||||
## Packages
|
||||
|
||||
| Package | Owns |
|
||||
| --- | --- |
|
||||
| `internal/contract/pricing` | the official price book, cost quotes, the spend ledger |
|
||||
| `internal/contract/ablation` | which subsystems a benchmark arm switches off, and the arm's name |
|
||||
| `internal/contract/agentgraph` | the vocabulary of one run's execution graph |
|
||||
| `internal/base/retrieval` | BM25 ranking over CJK-aware tokens |
|
||||
| `internal/base/rulelist` | splitting a one-line permission rule list |
|
||||
| `internal/base/tokencount` | a cheap token estimate |
|
||||
| `internal/base/neterr` | classifying transport errors |
|
||||
| `internal/base/nilutil` | typed-nil detection |
|
||||
| `cmd/benchctl` | prints rates, costs, peak schedules and arm names |
|
||||
|
||||
## Pricing
|
||||
|
||||
A `RateCard` is a price per million tokens in one currency:
|
||||
|
||||
| Field | Charges |
|
||||
| --- | --- |
|
||||
| `CacheHit` | prompt tokens served from the provider's cache |
|
||||
| `Input` | prompt tokens that missed the cache, including tokens written to it |
|
||||
| `Output` | completion tokens |
|
||||
|
||||
`OriginalCostAmount` turns a card and a `UsageTokens` breakdown into a
|
||||
fixed-point `Amount`; `PricingFingerprint` keys a card in the ledger, so a
|
||||
stored ledger entry stays attached to the card it was priced with.
|
||||
`CurrentRate` and `SupersededRates` answer from `officialRates`, which only
|
||||
ever grows: a price change appends a generation.
|
||||
|
||||
## Ablation
|
||||
|
||||
`ablation.Parse` reads a comma-separated list of modules to switch off, and
|
||||
`Set.Arm` names the result (`full` for the control). The switchable modules,
|
||||
in the order arm names use:
|
||||
|
||||
| Module | Off means |
|
||||
| --- | --- |
|
||||
| `evidence` | no evidence gate on completion claims |
|
||||
| `planner` | no two-model planning turn |
|
||||
| `subagent` | no delegation to workers |
|
||||
| `retrieval` | no retrieval over the folded transcript |
|
||||
| `compaction` | no context folding |
|
||||
| `upstream` | a dependency edge orders its endpoints without delivering the answer |
|
||||
| `recall-search` | recall can read a given address but cannot search for one |
|
||||
|
||||
The fold-index scale is a separate axis (`-fold-index`), not a module.
|
||||
|
||||
## Skills
|
||||
|
||||
Project skills live under `.reasonix/skills/<name>/SKILL.md`.
|
||||
|
||||
| Skill | Use it to |
|
||||
| --- | --- |
|
||||
| `price-change` | record a vendor's price change in the price book |
|
||||
@@ -0,0 +1,51 @@
|
||||
# Reasonix project memory
|
||||
|
||||
This checkout is a trimmed snapshot of the Reasonix kernel: the billing,
|
||||
ablation and graph contracts and a handful of leaf utilities, plus one small
|
||||
command that drives them. The conventions below are the project's own.
|
||||
|
||||
## Conventions
|
||||
|
||||
- Go kernel under `internal/`; each package owns one concern. A package's long
|
||||
explanation belongs in its `doc.go`, not spread across implementation files.
|
||||
- Judgements read structure, never wording: types, contracts, schemas. An
|
||||
error a caller must tell apart carries an identity — a sentinel, a typed
|
||||
code — never a sentence.
|
||||
- A metric is its definition, not its name. Before a number becomes a finding,
|
||||
state what it counts, from which anchor, over which population, and what it
|
||||
excludes.
|
||||
|
||||
## Comments
|
||||
|
||||
Default is none — the code is the truth. Write one only when the **why** is
|
||||
non-obvious: a hidden constraint, an invariant the type system cannot express,
|
||||
or an external-protocol quirk. A comment states the constraint that holds now;
|
||||
it does not narrate how the code got here. English, always.
|
||||
|
||||
- Declaration doc: ≤5 lines. Package comment: ≤8 lines, or ≤40 in a `doc.go`.
|
||||
- Every other comment: ≤3 lines. Struct-field and trailing `//`: 1 line.
|
||||
- One responsibility per file; 800 lines is the ceiling.
|
||||
|
||||
## Vendor rates
|
||||
|
||||
Official list prices live in one place: `officialRates` in
|
||||
`internal/contract/pricing/rates.go`, oldest generation first, the last being
|
||||
what a model charges today. A price change is an **append, never an edit** —
|
||||
the superseded rate is what lets an installed config be recognised as holding
|
||||
one of ours and brought forward. Callers read these through
|
||||
`pricing.CurrentRate` and `pricing.SupersededRates` and restate none of them.
|
||||
|
||||
`make pricecheck` compares that table against each vendor's own page. It needs
|
||||
the network, so it is not part of `make check`.
|
||||
|
||||
## Before every commit
|
||||
|
||||
```bash
|
||||
make check # gofmt -l, go vet ./..., go test ./...
|
||||
```
|
||||
|
||||
## Change discipline
|
||||
|
||||
List callers, consumers of the same state and sibling features before editing.
|
||||
Fix at the owning layer; a fix that breaks a neighbouring behaviour is a failed
|
||||
fix. Write tests first; they must fail without the change.
|
||||
@@ -0,0 +1,115 @@
|
||||
// Command benchctl prints what the pricing and ablation contracts answer for
|
||||
// one model or one benchmark arm, so a reviewer can check a rate or an arm name
|
||||
// without writing a test.
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
|
||||
"reasonix/internal/contract/ablation"
|
||||
"reasonix/internal/contract/pricing"
|
||||
)
|
||||
|
||||
var errUsage = errors.New("usage: benchctl rate|cost|tariff|arm [flags]")
|
||||
|
||||
func main() {
|
||||
if err := run(os.Args[1:], os.Stdout); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "benchctl:", err)
|
||||
os.Exit(2)
|
||||
}
|
||||
}
|
||||
|
||||
func run(args []string, out io.Writer) error {
|
||||
if len(args) == 0 {
|
||||
return errUsage
|
||||
}
|
||||
switch args[0] {
|
||||
case "rate":
|
||||
return runRate(args[1:], out)
|
||||
case "cost":
|
||||
return runCost(args[1:], out)
|
||||
case "tariff":
|
||||
return runTariff(args[1:], out)
|
||||
case "arm":
|
||||
return runArm(args[1:], out)
|
||||
default:
|
||||
return fmt.Errorf("%w: unknown subcommand %q", errUsage, args[0])
|
||||
}
|
||||
}
|
||||
|
||||
func runRate(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("rate", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
provider := fs.String("provider", "deepseek", "vendor")
|
||||
model := fs.String("model", "deepseek-flash", "model id")
|
||||
currency := fs.String("currency", "", "currency; empty reads the vendor's default")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
cur := *currency
|
||||
if cur == "" {
|
||||
cur = pricing.DefaultCurrency(*provider)
|
||||
}
|
||||
card := pricing.CurrentRate(*provider, *model, cur)
|
||||
if card == nil {
|
||||
return fmt.Errorf("no official rate for %s/%s in %s", *provider, *model, cur)
|
||||
}
|
||||
fmt.Fprintf(out, "%s/%s %s: cache-hit %g input %g output %g per 1M\n",
|
||||
*provider, *model, card.Currency, card.CacheHit, card.Input, card.Output)
|
||||
if entry, ok := pricing.LookupCatalog(*provider, *model, cur, ""); ok && entry.DocURL != "" {
|
||||
fmt.Fprintf(out, "source: %s\n", entry.DocURL)
|
||||
}
|
||||
fmt.Fprintf(out, "superseded generations: %d\n", len(pricing.SupersededRates(*provider, *model, cur)))
|
||||
return nil
|
||||
}
|
||||
|
||||
func runCost(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("cost", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
provider := fs.String("provider", "deepseek", "vendor")
|
||||
model := fs.String("model", "deepseek-flash", "model id")
|
||||
currency := fs.String("currency", "", "currency; empty reads the vendor's default")
|
||||
hit := fs.Int("hit", 0, "cached prompt tokens")
|
||||
miss := fs.Int("miss", 0, "uncached prompt tokens")
|
||||
completion := fs.Int("completion", 0, "completion tokens")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
cur := *currency
|
||||
if cur == "" {
|
||||
cur = pricing.DefaultCurrency(*provider)
|
||||
}
|
||||
card := pricing.CurrentRate(*provider, *model, cur)
|
||||
if card == nil {
|
||||
return fmt.Errorf("no official rate for %s/%s in %s", *provider, *model, cur)
|
||||
}
|
||||
usage := pricing.UsageTokens{CacheHitTokens: *hit, CacheMissTokens: *miss, CompletionTokens: *completion}
|
||||
amount := pricing.OriginalCostAmount(*card, usage)
|
||||
fmt.Fprintf(out, "%s %s\n", amount, pricing.NormalizeCurrency(card.Currency))
|
||||
return nil
|
||||
}
|
||||
|
||||
func runArm(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("arm", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
spec := fs.String("ablate", "", "comma-separated modules to switch off: "+ablation.ModuleList())
|
||||
foldIndex := fs.String("fold-index", "", "fold-index scale: default, half, quarter or off")
|
||||
list := fs.Bool("list", false, "list the switchable modules")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if *list {
|
||||
fmt.Fprintln(out, ablation.ModuleList())
|
||||
return nil
|
||||
}
|
||||
set, err := ablation.ParseArm(*spec, *foldIndex)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(out, set.Arm())
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func runOut(t *testing.T, args ...string) string {
|
||||
t.Helper()
|
||||
var out bytes.Buffer
|
||||
if err := run(args, &out); err != nil {
|
||||
t.Fatalf("run %v: %v", args, err)
|
||||
}
|
||||
return out.String()
|
||||
}
|
||||
|
||||
func TestRateNamesTheCurrentCard(t *testing.T) {
|
||||
got := runOut(t, "rate", "-provider", "mimo", "-model", "mimo-v2.5")
|
||||
if !strings.Contains(got, "mimo/mimo-v2.5 CNY: cache-hit 0.02 input 1 output 2 per 1M") {
|
||||
t.Fatalf("rate output = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateRefusesAnUnpricedModel(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
if err := run([]string{"rate", "-provider", "nobody", "-model", "x"}, &out); err == nil {
|
||||
t.Fatal("an unpriced model must fail")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCostChargesTheCurrentCard(t *testing.T) {
|
||||
got := runOut(t, "cost", "-provider", "mimo", "-model", "mimo-v2.5", "-miss", "1000000", "-completion", "1000000")
|
||||
if strings.TrimSpace(got) != "3 CNY" {
|
||||
t.Fatalf("cost output = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArmNamesTheConfiguration(t *testing.T) {
|
||||
if got := strings.TrimSpace(runOut(t, "arm")); got != "full" {
|
||||
t.Fatalf("control arm = %q", got)
|
||||
}
|
||||
if got := strings.TrimSpace(runOut(t, "arm", "-ablate", "planner,evidence", "-fold-index", "half")); got != "no-evidence+no-planner+index-half" {
|
||||
t.Fatalf("arm = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnknownSubcommandIsAUsageError(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
if err := run([]string{"nope"}, &out); !errors.Is(err, errUsage) {
|
||||
t.Fatalf("err = %v, want errUsage", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTariffReadsTheVendorsSchedule(t *testing.T) {
|
||||
got := runOut(t, "tariff", "-provider", "mimo", "-model", "mimo-v2.5")
|
||||
if !strings.Contains(got, "official card: true") || !strings.Contains(got, "one rate around the clock") {
|
||||
t.Fatalf("tariff output = %q", got)
|
||||
}
|
||||
peak := runOut(t, "tariff", "-currency", "CNY", "-at", "2026-10-12T02:00:00Z")
|
||||
if !strings.Contains(peak, "peak=true until 2026-10-12T04:00:00Z") {
|
||||
t.Fatalf("deepseek tariff at a weekday peak = %q", peak)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"time"
|
||||
|
||||
"reasonix/internal/contract/pricing"
|
||||
)
|
||||
|
||||
func runTariff(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("tariff", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
provider := fs.String("provider", "deepseek", "vendor")
|
||||
model := fs.String("model", "deepseek-flash", "model id")
|
||||
currency := fs.String("currency", "", "currency; empty reads the vendor's default")
|
||||
at := fs.String("at", "", "instant to read the schedule at, RFC 3339; empty is now")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
when := time.Now()
|
||||
if *at != "" {
|
||||
t, err := time.Parse(time.RFC3339, *at)
|
||||
if err != nil {
|
||||
return fmt.Errorf("-at: %w", err)
|
||||
}
|
||||
when = t
|
||||
}
|
||||
cur := *currency
|
||||
if cur == "" {
|
||||
cur = pricing.DefaultCurrency(*provider)
|
||||
}
|
||||
card := pricing.CurrentRate(*provider, *model, cur)
|
||||
if card == nil {
|
||||
return fmt.Errorf("no official rate for %s/%s in %s", *provider, *model, cur)
|
||||
}
|
||||
fmt.Fprintf(out, "official card: %v\n", pricing.IsOfficialCard(*provider, *model, *card))
|
||||
tariff, ok := pricing.LiveTariff(*provider, *model, *card, when)
|
||||
switch {
|
||||
case !ok:
|
||||
fmt.Fprintln(out, "schedule: one rate around the clock")
|
||||
case !tariff.Changes:
|
||||
fmt.Fprintf(out, "schedule: peak=%v, no change ahead\n", tariff.Peak)
|
||||
default:
|
||||
fmt.Fprintf(out, "schedule: peak=%v until %s\n", tariff.Peak, tariff.ChangesAt.UTC().Format(time.RFC3339))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
module reasonix
|
||||
|
||||
go 1.26.0
|
||||
@@ -0,0 +1,95 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Every catalog provider must declare at least one official endpoint, or its
|
||||
// rows become unreachable: a rate card is only matched against the catalog
|
||||
// after the entry's host identifies the vendor.
|
||||
func TestEveryCatalogProviderHasAnOfficialEndpoint(t *testing.T) {
|
||||
declared := map[string]bool{}
|
||||
for _, host := range officialEndpointHosts {
|
||||
declared[host] = true
|
||||
}
|
||||
for _, entry := range OfficialCatalog() {
|
||||
if !declared[entry.Provider] {
|
||||
t.Errorf("catalog provider %q has no official endpoint host; its prices can never match", entry.Provider)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestOfficialProviderForEndpoint(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
baseURL string
|
||||
want string
|
||||
}{
|
||||
{"official deepseek", "https://api.deepseek.com", "deepseek"},
|
||||
{"official deepseek anthropic path", "https://api.deepseek.com/anthropic", "deepseek"},
|
||||
{"official longcat", "https://api.longcat.chat/openai/v1", "longcat"},
|
||||
{"official mimo region", "https://token-plan-sgp.xiaomimimo.com/v1", "mimo"},
|
||||
{"uppercase host", "https://API.DeepSeek.com/v1", "deepseek"},
|
||||
{"proxy that resells the model", "https://my-deepseek-proxy.internal/v1", ""},
|
||||
{"vendor name in the path only", "https://gateway.example.com/deepseek/v1", ""},
|
||||
{"lookalike registrable domain", "https://api.deepseek.com.evil.test/v1", ""},
|
||||
{"empty", "", ""},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := OfficialProviderForEndpoint(tc.baseURL); got != tc.want {
|
||||
t.Fatalf("OfficialProviderForEndpoint(%q) = %q, want %q", tc.baseURL, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// pricesGoStaleAfter is how long a hand-copied rate is trusted before someone
|
||||
// reads the vendor's page again. No vendor here serves prices over an API, so
|
||||
// the alternative to this going red on a quiet day is learning about a change
|
||||
// from a user whose cost was wrong for months. Bumping CheckedOn without
|
||||
// opening DocURL defeats it; what this buys is that the question gets asked.
|
||||
const pricesGoStaleAfter = 90 * 24 * time.Hour
|
||||
|
||||
func TestCatalogPricesHaveBeenCheckedRecently(t *testing.T) {
|
||||
var undated, stale []string
|
||||
// This snapshot never sees a vendor page again, so staleness is read at the
|
||||
// day it was taken rather than the day the suite runs.
|
||||
cutoff := time.Date(2026, 10, 19, 0, 0, 0, 0, time.UTC).Add(-pricesGoStaleAfter)
|
||||
for _, e := range OfficialCatalog() {
|
||||
where := e.Provider + "/" + e.Model + " " + e.Currency + " (" + e.DocURL + ")"
|
||||
if e.CheckedOn == "" {
|
||||
undated = append(undated, where)
|
||||
continue
|
||||
}
|
||||
on, err := time.Parse("2006-01-02", e.CheckedOn)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: CheckedOn %q is not YYYY-MM-DD", where, e.CheckedOn)
|
||||
}
|
||||
if on.Before(cutoff) {
|
||||
stale = append(stale, where+" last checked "+e.CheckedOn)
|
||||
}
|
||||
}
|
||||
if len(undated) > 0 {
|
||||
t.Errorf("catalog rows with no CheckedOn — read the vendor's page and date them:\n %s",
|
||||
strings.Join(undated, "\n "))
|
||||
}
|
||||
if len(stale) > 0 {
|
||||
t.Errorf("catalog rows older than %d days. Open each DocURL, confirm or correct the rate, then set CheckedOn to today:\n %s",
|
||||
int(pricesGoStaleAfter.Hours()/24), strings.Join(stale, "\n "))
|
||||
}
|
||||
}
|
||||
|
||||
// A row that names a peak card must say when it applies, and one that names a
|
||||
// window must have a card to charge inside it. Half of the pair silently prices
|
||||
// every hour the same, which is the bug the pair exists to prevent.
|
||||
func TestPeakRatesAndWindowsComeInPairs(t *testing.T) {
|
||||
for _, e := range OfficialCatalog() {
|
||||
where := e.Provider + "/" + e.Model + " " + e.Currency
|
||||
if (e.Peak == nil) != (e.Window == nil) {
|
||||
t.Errorf("%s: Peak and Window must be set together (peak=%v window=%v)", where, e.Peak != nil, e.Window != nil)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
name: ablation-arm
|
||||
description: Add a new switchable subsystem to the benchmark's ablation arms.
|
||||
---
|
||||
|
||||
# Adding an ablation module
|
||||
|
||||
1. Declare the module in `internal/ablation/modules.go` beside the others.
|
||||
2. Register it in `AllModules()` so `ParseSpec` accepts it.
|
||||
3. Arm names come from `Set.Name()`; check that yours reads `without-<module>`.
|
||||
4. Add the module to the flag help text in `cmd/bench/flags.go`.
|
||||
5. Run `make test`.
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
name: benchctl
|
||||
description: Check a rate, a cost, a peak schedule or an arm name from the command line instead of writing a test.
|
||||
---
|
||||
|
||||
# Using benchctl
|
||||
|
||||
1. Price a request with `benchctl quote -provider mimo -model mimo-v2.5 -tokens 1000000`;
|
||||
it prints the amount and the currency.
|
||||
2. Read today's card with `benchctl rate -vendor deepseek -model deepseek-flash`.
|
||||
3. Ask whether a vendor is in its peak window with
|
||||
`benchctl peak -currency CNY -at 1791770400` (Unix seconds).
|
||||
4. List the ablation modules with `benchctl arms`, and name an arm with
|
||||
`benchctl arm -ablate planner,evidence -index half`.
|
||||
5. Every subcommand prints its flags with `-h`.
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: price-change
|
||||
description: Record a vendor's announced price change in the official price book.
|
||||
---
|
||||
|
||||
# Recording a price change
|
||||
|
||||
1. Open `internal/pricing/rates.go` and find the vendor's entry in
|
||||
`officialPrices`.
|
||||
2. Edit the entry's `Input`, `Output` and `CacheHit` to the new values.
|
||||
3. If the change is announced ahead of time, set `EffectiveFrom` on the entry.
|
||||
4. Confirm `pricing.LatestRate` now returns the new card.
|
||||
5. Run `make test` and `make price-check`.
|
||||
Executable
+52
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env bash
|
||||
# Builds every task's workdir/ from one pinned commit of this repository plus
|
||||
# the fixture-authored files in base/ and overlay/<task-id>/. With --check it
|
||||
# rebuilds into a scratch directory and fails if any committed workdir differs.
|
||||
set -euo pipefail
|
||||
|
||||
PIN=42636ddcb425b7af07abd88397d02c1cfe3adcf5
|
||||
PATHS=(
|
||||
internal/contract/pricing
|
||||
internal/contract/ablation
|
||||
internal/contract/agentgraph
|
||||
internal/base/retrieval
|
||||
internal/base/rulelist
|
||||
internal/base/tokencount
|
||||
internal/base/neterr
|
||||
internal/base/nilutil
|
||||
)
|
||||
|
||||
here="$(cd "$(dirname "$0")" && pwd)"
|
||||
repo="$(git -C "$here" rev-parse --show-toplevel)"
|
||||
|
||||
build() {
|
||||
local id="$1" dest="$2"
|
||||
rm -rf "$dest"
|
||||
mkdir -p "$dest"
|
||||
git -C "$repo" archive "$PIN" -- "${PATHS[@]}" | tar -x -C "$dest"
|
||||
cp -R "$here/base/." "$dest/"
|
||||
if [ -d "$here/overlay/$id" ]; then
|
||||
cp -R "$here/overlay/$id/." "$dest/"
|
||||
fi
|
||||
}
|
||||
|
||||
check=0
|
||||
[ "${1:-}" = "--check" ] && check=1
|
||||
|
||||
status=0
|
||||
for task in "$here"/tasks/*/; do
|
||||
id="$(basename "$task")"
|
||||
if [ "$check" = 1 ]; then
|
||||
scratch="$(mktemp -d)"
|
||||
build "$id" "$scratch/workdir"
|
||||
if ! diff -r "$scratch/workdir" "$task/workdir" >/dev/null; then
|
||||
echo "$id: workdir/ differs from the pinned snapshot; run $0" >&2
|
||||
diff -rq "$scratch/workdir" "$task/workdir" >&2 || true
|
||||
status=1
|
||||
fi
|
||||
rm -rf "$scratch"
|
||||
else
|
||||
build "$id" "$task/workdir"
|
||||
fi
|
||||
done
|
||||
exit "$status"
|
||||
@@ -0,0 +1,73 @@
|
||||
# Reasonix kernel snapshot
|
||||
|
||||
A trimmed slice of the Reasonix Go kernel: the contracts a benchmark report is
|
||||
priced and labelled with, a few leaf utilities, and `benchctl`, a small command
|
||||
that prints what those contracts answer.
|
||||
|
||||
```bash
|
||||
make check # gofmt, vet, test
|
||||
go run ./cmd/benchctl rate -model deepseek-flash -currency USD
|
||||
go run ./cmd/benchctl cost -provider mimo -model mimo-v2.5 -miss 1000000
|
||||
go run ./cmd/benchctl tariff -currency CNY -at 2026-10-12T02:00:00Z
|
||||
go run ./cmd/benchctl arm -ablate planner,evidence
|
||||
```
|
||||
|
||||
## Packages
|
||||
|
||||
| Package | Owns |
|
||||
| --- | --- |
|
||||
| `internal/contract/pricing` | the official price book, cost quotes, the spend ledger |
|
||||
| `internal/contract/ablation` | which subsystems a benchmark arm switches off, and the arm's name |
|
||||
| `internal/contract/agentgraph` | the vocabulary of one run's execution graph |
|
||||
| `internal/base/retrieval` | BM25 ranking over CJK-aware tokens |
|
||||
| `internal/base/rulelist` | splitting a one-line permission rule list |
|
||||
| `internal/base/tokencount` | a cheap token estimate |
|
||||
| `internal/base/neterr` | classifying transport errors |
|
||||
| `internal/base/nilutil` | typed-nil detection |
|
||||
| `cmd/benchctl` | prints rates, costs, peak schedules and arm names |
|
||||
|
||||
## Pricing
|
||||
|
||||
A `RateCard` is a price per million tokens in one currency:
|
||||
|
||||
| Field | Charges |
|
||||
| --- | --- |
|
||||
| `CacheHit` | prompt tokens served from the provider's cache |
|
||||
| `Input` | prompt tokens that missed the cache, and tokens written to it when `CacheWrite` is zero |
|
||||
| `Output` | completion tokens |
|
||||
| `CacheWrite` | tokens written to the cache, for a vendor that prices them separately (Anthropic) |
|
||||
|
||||
A card without `CacheWrite` keeps the fingerprint it had before the field
|
||||
existed, so ledgers stored against it stay attached.
|
||||
|
||||
`OriginalCostAmount` turns a card and a `UsageTokens` breakdown into a
|
||||
fixed-point `Amount`; `PricingFingerprint` keys a card in the ledger, so a
|
||||
stored ledger entry stays attached to the card it was priced with.
|
||||
`CurrentRate` and `SupersededRates` answer from `officialRates`, which only
|
||||
ever grows: a price change appends a generation.
|
||||
|
||||
## Ablation
|
||||
|
||||
`ablation.Parse` reads a comma-separated list of modules to switch off, and
|
||||
`Set.Arm` names the result (`full` for the control). The switchable modules,
|
||||
in the order arm names use:
|
||||
|
||||
| Module | Off means |
|
||||
| --- | --- |
|
||||
| `evidence` | no evidence gate on completion claims |
|
||||
| `planner` | no two-model planning turn |
|
||||
| `subagent` | no delegation to workers |
|
||||
| `retrieval` | no retrieval over the folded transcript |
|
||||
| `compaction` | no context folding |
|
||||
| `upstream` | a dependency edge orders its endpoints without delivering the answer |
|
||||
| `recall-search` | recall can read a given address but cannot search for one |
|
||||
|
||||
The fold-index scale is a separate axis (`-fold-index`), not a module.
|
||||
|
||||
## Skills
|
||||
|
||||
Project skills live under `.reasonix/skills/<name>/SKILL.md`.
|
||||
|
||||
| Skill | Use it to |
|
||||
| --- | --- |
|
||||
| `price-change` | record a vendor's price change in the price book |
|
||||
@@ -0,0 +1,119 @@
|
||||
// Command benchctl prints what the pricing and ablation contracts answer for
|
||||
// one model or one benchmark arm, so a reviewer can check a rate or an arm name
|
||||
// without writing a test.
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
|
||||
"reasonix/internal/contract/ablation"
|
||||
"reasonix/internal/contract/pricing"
|
||||
)
|
||||
|
||||
var errUsage = errors.New("usage: benchctl rate|cost|tariff|arm [flags]")
|
||||
|
||||
func main() {
|
||||
if err := run(os.Args[1:], os.Stdout); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "benchctl:", err)
|
||||
os.Exit(2)
|
||||
}
|
||||
}
|
||||
|
||||
func run(args []string, out io.Writer) error {
|
||||
if len(args) == 0 {
|
||||
return errUsage
|
||||
}
|
||||
switch args[0] {
|
||||
case "rate":
|
||||
return runRate(args[1:], out)
|
||||
case "cost":
|
||||
return runCost(args[1:], out)
|
||||
case "tariff":
|
||||
return runTariff(args[1:], out)
|
||||
case "arm":
|
||||
return runArm(args[1:], out)
|
||||
default:
|
||||
return fmt.Errorf("%w: unknown subcommand %q", errUsage, args[0])
|
||||
}
|
||||
}
|
||||
|
||||
func runRate(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("rate", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
provider := fs.String("provider", "deepseek", "vendor")
|
||||
model := fs.String("model", "deepseek-flash", "model id")
|
||||
currency := fs.String("currency", "", "currency; empty reads the vendor's default")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
cur := *currency
|
||||
if cur == "" {
|
||||
cur = pricing.DefaultCurrency(*provider)
|
||||
}
|
||||
card := pricing.CurrentRate(*provider, *model, cur)
|
||||
if card == nil {
|
||||
return fmt.Errorf("no official rate for %s/%s in %s", *provider, *model, cur)
|
||||
}
|
||||
fmt.Fprintf(out, "%s/%s %s: cache-hit %g input %g output %g", *provider, *model, card.Currency, card.CacheHit, card.Input, card.Output)
|
||||
if card.CacheWrite != 0 {
|
||||
fmt.Fprintf(out, " cache-write %g", card.CacheWrite)
|
||||
}
|
||||
fmt.Fprintln(out, " per 1M")
|
||||
if entry, ok := pricing.LookupCatalog(*provider, *model, cur, ""); ok && entry.DocURL != "" {
|
||||
fmt.Fprintf(out, "source: %s\n", entry.DocURL)
|
||||
}
|
||||
fmt.Fprintf(out, "superseded generations: %d\n", len(pricing.SupersededRates(*provider, *model, cur)))
|
||||
return nil
|
||||
}
|
||||
|
||||
func runCost(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("cost", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
provider := fs.String("provider", "deepseek", "vendor")
|
||||
model := fs.String("model", "deepseek-flash", "model id")
|
||||
currency := fs.String("currency", "", "currency; empty reads the vendor's default")
|
||||
hit := fs.Int("hit", 0, "cached prompt tokens")
|
||||
miss := fs.Int("miss", 0, "uncached prompt tokens")
|
||||
completion := fs.Int("completion", 0, "completion tokens")
|
||||
write := fs.Int("write", 0, "uncached prompt tokens that were written to the cache")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
cur := *currency
|
||||
if cur == "" {
|
||||
cur = pricing.DefaultCurrency(*provider)
|
||||
}
|
||||
card := pricing.CurrentRate(*provider, *model, cur)
|
||||
if card == nil {
|
||||
return fmt.Errorf("no official rate for %s/%s in %s", *provider, *model, cur)
|
||||
}
|
||||
usage := pricing.UsageTokens{CacheHitTokens: *hit, CacheMissTokens: *miss, CompletionTokens: *completion, CacheWriteTokens: *write}
|
||||
amount := pricing.OriginalCostAmount(*card, usage)
|
||||
fmt.Fprintf(out, "%s %s\n", amount, pricing.NormalizeCurrency(card.Currency))
|
||||
return nil
|
||||
}
|
||||
|
||||
func runArm(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("arm", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
spec := fs.String("ablate", "", "comma-separated modules to switch off: "+ablation.ModuleList())
|
||||
foldIndex := fs.String("fold-index", "", "fold-index scale: default, half, quarter or off")
|
||||
list := fs.Bool("list", false, "list the switchable modules")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if *list {
|
||||
fmt.Fprintln(out, ablation.ModuleList())
|
||||
return nil
|
||||
}
|
||||
set, err := ablation.ParseArm(*spec, *foldIndex)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(out, set.Arm())
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
module reasonix
|
||||
|
||||
go 1.26.0
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
package pricing
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestCacheWriteRateChargesWrittenTokens(t *testing.T) {
|
||||
card := RateCard{Input: 3, CacheWrite: 3.75, Currency: "USD"}
|
||||
usage := UsageTokens{CacheMissTokens: 1000000, CacheWriteTokens: 400000}
|
||||
if got := OriginalCostAmount(card, usage).String(); got != "3.3" {
|
||||
t.Fatalf("cost = %s, want 3.3", got)
|
||||
}
|
||||
card.CacheWrite = 0
|
||||
if got := OriginalCostAmount(card, usage).String(); got != "3" {
|
||||
t.Fatalf("cost without a cache-write rate = %s, want 3", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCacheWriteSeparatesFingerprintsAndCards(t *testing.T) {
|
||||
base := RateCard{CacheHit: 0.02, Input: 1, Output: 2, Currency: "CNY"}
|
||||
withWrite := base
|
||||
withWrite.CacheWrite = 1.25
|
||||
if PricingFingerprint(base) == PricingFingerprint(withWrite) {
|
||||
t.Fatal("cache-write rate does not reach the fingerprint")
|
||||
}
|
||||
if sameCard(base, withWrite) {
|
||||
t.Fatal("cards differing in CacheWrite compare equal")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnthropicPricesCacheWrites(t *testing.T) {
|
||||
got := CurrentRate("anthropic", "claude-sonnet-4-5", "USD")
|
||||
if got == nil || got.CacheWrite != 3.75 {
|
||||
t.Fatalf("claude-sonnet-4-5 = %+v", got)
|
||||
}
|
||||
if OfficialProviderForEndpoint("https://api.anthropic.com/v1/messages") != "anthropic" {
|
||||
t.Fatal("api.anthropic.com is not recognised as Anthropic's endpoint")
|
||||
}
|
||||
}
|
||||
+222
@@ -0,0 +1,222 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Official catalog metadata: currency, model, effective dates, documentation
|
||||
// sources, and price fingerprints. User-custom prices always win over catalog.
|
||||
|
||||
// CatalogEntry is one official list price for a model in a billing currency.
|
||||
type CatalogEntry struct {
|
||||
Provider string // deepseek | longcat | mimo | google
|
||||
Model string
|
||||
Currency string // ISO billing currency for this row
|
||||
CacheHit float64
|
||||
Input float64
|
||||
Output float64
|
||||
CacheWrite float64 // zero when the vendor bills cache writes at Input
|
||||
EffectiveFrom string // YYYY-MM-DD inclusive, empty = unbounded
|
||||
EffectiveTo string // YYYY-MM-DD exclusive, empty = unbounded
|
||||
DocURL string
|
||||
BillingMode string // payg | subscription_equivalent
|
||||
Notes string
|
||||
Fingerprint string // filled by Register
|
||||
// Peak is what the same tokens cost inside Window. Nil when the vendor
|
||||
// bills one rate around the clock, and then the fields above are the rate.
|
||||
Peak *RateCard
|
||||
Window *PeakWindow
|
||||
// CheckedOn is when a person last read DocURL and confirmed these numbers.
|
||||
// No vendor here serves prices over an API, so this date is all that stands
|
||||
// between a price change and a readout that is quietly wrong.
|
||||
CheckedOn string
|
||||
}
|
||||
|
||||
// deepseekPeak is the schedule DeepSeek publishes: peak on weekday mornings and
|
||||
// afternoons, and — since 2026-08-23 — never at a weekend. Chinese public
|
||||
// holidays bill off-peak in full, per the same page.
|
||||
var deepseekPeak = &PeakWindow{
|
||||
OffsetSeconds: beijing,
|
||||
Hours: [][2]int{{9, 12}, {14, 18}},
|
||||
WeekendOffPeak: "2026-08-23",
|
||||
HolidayOffPeak: chineseStatutoryHolidays,
|
||||
}
|
||||
|
||||
// chineseStatutoryHolidays are the State Council's 2026 public holidays, which
|
||||
// DeepSeek bills off-peak in full; extend it when the next notice is out. Make-up
|
||||
// workdays are absent: they fall on weekends, which are off-peak anyway.
|
||||
var chineseStatutoryHolidays = []string{
|
||||
// New Year's Day: Jan 1-3.
|
||||
"2026-01-01", "2026-01-02", "2026-01-03",
|
||||
// Spring Festival: Feb 15-23.
|
||||
"2026-02-15", "2026-02-16", "2026-02-17", "2026-02-18", "2026-02-19",
|
||||
"2026-02-20", "2026-02-21", "2026-02-22", "2026-02-23",
|
||||
// Qingming Festival: Apr 4-6.
|
||||
"2026-04-04", "2026-04-05", "2026-04-06",
|
||||
// Labor Day: May 1-5.
|
||||
"2026-05-01", "2026-05-02", "2026-05-03", "2026-05-04", "2026-05-05",
|
||||
// Dragon Boat Festival: Jun 19-21.
|
||||
"2026-06-19", "2026-06-20", "2026-06-21",
|
||||
// Mid-Autumn Festival: Sep 25-27.
|
||||
"2026-09-25", "2026-09-26", "2026-09-27",
|
||||
// National Day: Oct 1-7.
|
||||
"2026-10-01", "2026-10-02", "2026-10-03", "2026-10-04",
|
||||
"2026-10-05", "2026-10-06", "2026-10-07",
|
||||
}
|
||||
|
||||
// checkedOn is when the tables below were last read off the vendors' pages.
|
||||
const checkedOn = "2026-08-22"
|
||||
|
||||
// deepseekCheckedOn is when the DeepSeek rows were last read off that vendor's
|
||||
// page. Separate from checkedOn, because confirming one vendor's prices says
|
||||
// nothing about another's and a shared date would claim that it did.
|
||||
const deepseekCheckedOn = "2026-09-13"
|
||||
|
||||
// mimoCheckedOn is when the V2.6 rows were read off the vendor's own docs.
|
||||
const mimoCheckedOn = "2026-10-06"
|
||||
|
||||
// googleCheckedOn is when the Gemini rows were read off Google's pricing page.
|
||||
const googleCheckedOn = "2026-10-17"
|
||||
|
||||
// anthropicCheckedOn is when the Claude rows were read off Anthropic's pricing page.
|
||||
const anthropicCheckedOn = "2026-10-19"
|
||||
|
||||
// CatalogSourceURLs document public pricing pages.
|
||||
const (
|
||||
DocDeepSeekPricing = "https://api-docs.deepseek.com/quick_start/pricing"
|
||||
DocDeepSeekPricingCNY = "https://api-docs.deepseek.com/zh-cn/quick_start/pricing"
|
||||
DocLongCatPricingUSD = "https://longcat.chat/platform/docs/Pricing/LongCat-2.0.html"
|
||||
DocLongCatPricingCNY = "https://longcat.chat/platform/docs/zh/pricing/long-cat-2.0"
|
||||
DocMiMoPAYG = "https://mimo.mi.com/docs/price/pay-as-you-go"
|
||||
DocMiMoTokenPlan = "https://platform.xiaomimimo.com/token-plan"
|
||||
DocGeminiPricing = "https://ai.google.dev/gemini-api/docs/pricing"
|
||||
DocAnthropicPricing = "https://www.anthropic.com/pricing"
|
||||
)
|
||||
|
||||
// officialEndpointHosts maps a vendor's own API hostnames to the catalog
|
||||
// provider whose list prices apply there. Host equality is the test: a reseller
|
||||
// or proxy bills on its own terms even when it serves the same model, and an
|
||||
// entry merely named after a vendor is not evidence of the vendor's endpoint.
|
||||
var officialEndpointHosts = map[string]string{
|
||||
"api.anthropic.com": "anthropic",
|
||||
"api.deepseek.com": "deepseek",
|
||||
"api.longcat.chat": "longcat",
|
||||
"api.xiaomimimo.com": "mimo",
|
||||
"generativelanguage.googleapis.com": "google",
|
||||
"token-plan-cn.xiaomimimo.com": "mimo",
|
||||
"token-plan-sgp.xiaomimimo.com": "mimo",
|
||||
"token-plan-ams.xiaomimimo.com": "mimo",
|
||||
}
|
||||
|
||||
// OfficialProviderForEndpoint returns the catalog provider whose official
|
||||
// prices apply to baseURL, or "" when the address is not a vendor's own API.
|
||||
func OfficialProviderForEndpoint(baseURL string) string {
|
||||
u, err := url.Parse(strings.TrimSpace(baseURL))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return officialEndpointHosts[strings.ToLower(u.Hostname())]
|
||||
}
|
||||
|
||||
// OfficialCatalog is the built-in price book: the current generation of every
|
||||
// rate in officialRates, fingerprinted so a stored price can be recognised as
|
||||
// one of ours. User-custom prices always win over it.
|
||||
func OfficialCatalog() []CatalogEntry {
|
||||
entries := officialCatalogRows()
|
||||
for i := range entries {
|
||||
entries[i].Fingerprint = PricingFingerprint(RateCardFromCatalog(entries[i]))
|
||||
}
|
||||
return entries
|
||||
}
|
||||
|
||||
// LookupCatalog finds an official entry. billingMode empty matches payg first.
|
||||
func LookupCatalog(provider, model, currency, billingMode string) (CatalogEntry, bool) {
|
||||
provider = strings.ToLower(strings.TrimSpace(provider))
|
||||
model = strings.TrimSpace(model)
|
||||
currency = NormalizeCurrency(currency)
|
||||
billingMode = strings.TrimSpace(billingMode)
|
||||
if billingMode == "" {
|
||||
billingMode = BillingModePAYG
|
||||
}
|
||||
var fallback CatalogEntry
|
||||
foundFallback := false
|
||||
for _, e := range OfficialCatalog() {
|
||||
if e.Provider != provider || e.Model != model {
|
||||
continue
|
||||
}
|
||||
if NormalizeCurrency(e.Currency) != currency {
|
||||
continue
|
||||
}
|
||||
if e.BillingMode == billingMode {
|
||||
return e, true
|
||||
}
|
||||
if e.BillingMode == BillingModePAYG {
|
||||
fallback = e
|
||||
foundFallback = true
|
||||
}
|
||||
}
|
||||
if foundFallback {
|
||||
return fallback, true
|
||||
}
|
||||
return CatalogEntry{}, false
|
||||
}
|
||||
|
||||
// RateCardFromCatalog builds a RateCard from a catalog entry's base rate.
|
||||
func RateCardFromCatalog(e CatalogEntry) RateCard {
|
||||
return RateCard{
|
||||
CacheHit: e.CacheHit,
|
||||
Input: e.Input,
|
||||
Output: e.Output,
|
||||
CacheWrite: e.CacheWrite,
|
||||
Currency: e.Currency,
|
||||
}
|
||||
}
|
||||
|
||||
// RatesAt is what this entry charges at t: the peak card inside its window, the
|
||||
// base card everywhere else. An entry without a window charges one rate, and
|
||||
// then t does not matter.
|
||||
func (e CatalogEntry) RatesAt(t time.Time) RateCard {
|
||||
if e.Peak == nil || !e.Window.IsPeak(t) {
|
||||
return RateCardFromCatalog(e)
|
||||
}
|
||||
peak := *e.Peak
|
||||
peak.Currency = e.Currency
|
||||
return peak
|
||||
}
|
||||
|
||||
// InEffect reports whether this entry's dates cover the day t falls on.
|
||||
func (e CatalogEntry) InEffect(t time.Time) bool {
|
||||
day := t.UTC().Format("2006-01-02")
|
||||
if e.EffectiveFrom != "" && day < e.EffectiveFrom {
|
||||
return false
|
||||
}
|
||||
return e.EffectiveTo == "" || day < e.EffectiveTo
|
||||
}
|
||||
|
||||
// MatchesCatalog reports whether rates equal a known official entry.
|
||||
func MatchesCatalog(provider, model string, rates RateCard) (CatalogEntry, bool) {
|
||||
cur := NormalizeCurrency(rates.Currency)
|
||||
for _, e := range OfficialCatalog() {
|
||||
if e.Provider != strings.ToLower(strings.TrimSpace(provider)) {
|
||||
continue
|
||||
}
|
||||
if e.Model != strings.TrimSpace(model) {
|
||||
continue
|
||||
}
|
||||
if NormalizeCurrency(e.Currency) != cur {
|
||||
continue
|
||||
}
|
||||
// Either card identifies the vendor's own price. Which one a user
|
||||
// happens to have configured says nothing about when they are billing:
|
||||
// the hour decides that, and RatesAt is what reads it.
|
||||
if sameCard(RateCardFromCatalog(e), rates) {
|
||||
return e, true
|
||||
}
|
||||
if p := e.Peak; p != nil && sameCard(*p, rates) {
|
||||
return e, true
|
||||
}
|
||||
}
|
||||
return CatalogEntry{}, false
|
||||
}
|
||||
+799
@@ -0,0 +1,799 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Billing modes.
|
||||
const (
|
||||
BillingModePAYG = "payg"
|
||||
BillingModeSubscriptionEquivalent = "subscription_equivalent"
|
||||
)
|
||||
|
||||
// Valuation basis values.
|
||||
const (
|
||||
BasisIdentity = "identity"
|
||||
BasisOfficialTable = "official_table"
|
||||
// BasisFX is retained for decoding pre-v6 persisted quotes only. New
|
||||
// quotes never create FX valuations.
|
||||
BasisFX = "fx"
|
||||
)
|
||||
|
||||
// DisplayStatus describes whether a quote can satisfy its requested display.
|
||||
// It is deliberately separate from CostComplete: a known price-book amount
|
||||
// can exist even when no requested regional valuation is available.
|
||||
const (
|
||||
DisplayStatusMatched = "matched"
|
||||
DisplayStatusFallbackOriginal = "fallback_original"
|
||||
DisplayStatusBucketed = "bucketed"
|
||||
DisplayStatusUnavailable = "unavailable"
|
||||
)
|
||||
|
||||
// Aggregate modes describe how a session/run total was formed.
|
||||
const (
|
||||
AggregateModeSingleCurrency = "single_currency"
|
||||
AggregateModeCommonValuation = "common_valuation"
|
||||
AggregateModeCurrencyBuckets = "currency_buckets"
|
||||
)
|
||||
|
||||
// Rate bands name which side of a vendor's peak window a call was billed on.
|
||||
// An aggregate is mixed when its calls fell on both sides.
|
||||
const (
|
||||
RateBandPeak = "peak"
|
||||
RateBandOffPeak = "off_peak"
|
||||
RateBandMixed = "mixed"
|
||||
)
|
||||
|
||||
// DisplayRequest identifies how a caller selected the requested presentation
|
||||
// currency. The source is runtime context only; it is never persisted as a
|
||||
// provider price or wallet fact.
|
||||
type DisplayRequest struct {
|
||||
Currency string `json:"currency,omitempty"`
|
||||
Source string `json:"source,omitempty"`
|
||||
}
|
||||
|
||||
const (
|
||||
DisplaySourceExplicit = "explicit"
|
||||
DisplaySourceWallet = "wallet"
|
||||
DisplaySourceAuto = "auto"
|
||||
)
|
||||
|
||||
func (r DisplayRequest) NormalizedCurrency() string {
|
||||
return NormalizeCurrency(r.Currency)
|
||||
}
|
||||
|
||||
// CostQuote is the host-side quoted cost for one model call (or an aggregate).
|
||||
// Original is the pricing-table currency fact; valuations hold identity or
|
||||
// official regional rate-card estimates. Selected is the display pick for the
|
||||
// current preference.
|
||||
type CostQuote struct {
|
||||
Original Money `json:"original"`
|
||||
// OriginalTotals is populated for mixed-currency aggregates. Individual
|
||||
// quotes and single-currency totals may omit it.
|
||||
OriginalTotals []Money `json:"originalTotals,omitempty"`
|
||||
Valuations map[string]Valuation `json:"valuations,omitempty"`
|
||||
Selected *Money `json:"selected,omitempty"`
|
||||
BillingMode string `json:"billingMode,omitempty"`
|
||||
Estimated bool `json:"estimated"`
|
||||
// CostComplete means usage and the price-book amount are known. It does not
|
||||
// imply that a requested display currency is available.
|
||||
CostComplete bool `json:"costComplete"`
|
||||
// Coverage is the same fact with the states a boolean cannot hold: an
|
||||
// aggregate that priced only some of its calls, and one that billed nothing.
|
||||
Coverage string `json:"coverage,omitempty"`
|
||||
// DisplayComplete means a single selected amount satisfies the display
|
||||
// request. Complete is retained as the old wire alias.
|
||||
DisplayComplete bool `json:"displayComplete"`
|
||||
Complete bool `json:"complete"`
|
||||
DisplayStatus string `json:"displayStatus,omitempty"`
|
||||
AggregateMode string `json:"aggregateMode,omitempty"`
|
||||
// ModelRef / UsageSource / PricingFingerprint identify the ledger key.
|
||||
ModelRef string `json:"modelRef,omitempty"`
|
||||
UsageSource string `json:"usageSource,omitempty"`
|
||||
PricingFingerprint string `json:"pricingFingerprint,omitempty"`
|
||||
RateDate string `json:"rateDate,omitempty"` // YYYY-MM-DD of FX used, if any
|
||||
IncompleteReason string `json:"incompleteReason,omitempty"`
|
||||
LegacyEstimate bool `json:"legacyEstimate,omitempty"`
|
||||
CatalogSource string `json:"catalogSource,omitempty"`
|
||||
// RateBand is set only for a rate confirmed as a vendor's own schedule; an
|
||||
// aggregate that includes any call without one carries none.
|
||||
RateBand string `json:"rateBand,omitempty"`
|
||||
}
|
||||
|
||||
// Valuation is one currency view of a cost fact.
|
||||
type Valuation struct {
|
||||
Money Money `json:"money"`
|
||||
Basis string `json:"basis"`
|
||||
Source string `json:"source"`
|
||||
AsOf string `json:"asOf"` // YYYY-MM-DD
|
||||
Rate *RateSnapshot `json:"rateSnapshot,omitempty"`
|
||||
Stale bool `json:"stale,omitempty"`
|
||||
}
|
||||
|
||||
// RateSnapshot records the FX observation used for a valuation.
|
||||
type RateSnapshot struct {
|
||||
Base string `json:"base"`
|
||||
Quote string `json:"quote"`
|
||||
Rate float64 `json:"rate"`
|
||||
Source string `json:"source"`
|
||||
AsOf string `json:"asOf"`
|
||||
Stale bool `json:"stale,omitempty"`
|
||||
}
|
||||
|
||||
// RateCard is the per-1M-token price used to compute original cost. It mirrors
|
||||
// provider.Pricing without importing that package (billing is a leaf).
|
||||
type RateCard struct {
|
||||
CacheHit float64 // per 1M cached prompt tokens
|
||||
Input float64 // per 1M uncached prompt tokens
|
||||
Output float64 // per 1M completion tokens
|
||||
// CacheWrite is per 1M tokens written to the cache; zero bills them at Input.
|
||||
CacheWrite float64
|
||||
Currency string // ISO or symbol; normalized on quote
|
||||
}
|
||||
|
||||
// UsageTokens is the token breakdown needed for cost. Mirrors provider.Usage
|
||||
// fields without importing provider.
|
||||
type UsageTokens struct {
|
||||
PromptTokens int
|
||||
CompletionTokens int
|
||||
CacheHitTokens int
|
||||
CacheMissTokens int
|
||||
CacheWriteTokens int
|
||||
CacheWriteBilledTokens float64
|
||||
Estimated bool
|
||||
}
|
||||
|
||||
// QuoteInput drives a single CostQuote computation.
|
||||
type QuoteInput struct {
|
||||
Usage UsageTokens
|
||||
Rates RateCard
|
||||
OccurredAt time.Time
|
||||
DisplayCurrency string // compatibility alias for Display.Currency
|
||||
Display DisplayRequest // runtime display request; empty means auto
|
||||
BillingMode string
|
||||
ModelRef string
|
||||
UsageSource string
|
||||
CatalogSource string
|
||||
PricingFingerprint string
|
||||
// ProviderKind is deepseek|longcat|mimo|… for official dual-table lookup.
|
||||
// When empty, ModelRef and Rates are used to infer a catalog match.
|
||||
ProviderKind string
|
||||
// ModelID is the bare model id (e.g. deepseek-v4-flash). Empty → parse ModelRef.
|
||||
ModelID string
|
||||
}
|
||||
|
||||
// OriginalCostAmount computes the fixed-point original cost from rates + tokens.
|
||||
// Mirrors the historic provider.Pricing.Cost semantics.
|
||||
func OriginalCostAmount(rates RateCard, u UsageTokens) Amount {
|
||||
hit := u.CacheHitTokens
|
||||
miss := u.CacheMissTokens
|
||||
if hit+miss == 0 && u.PromptTokens > 0 {
|
||||
miss = u.PromptTokens
|
||||
} else if miss == 0 && hit > 0 && u.PromptTokens > hit {
|
||||
miss = u.PromptTokens - hit
|
||||
}
|
||||
write := u.CacheWriteTokens
|
||||
write = max(write, 0)
|
||||
write = min(write, miss)
|
||||
billedWrite := 0.0
|
||||
if write > 0 {
|
||||
billedWrite = u.CacheWriteBilledTokens
|
||||
if billedWrite <= 0 {
|
||||
billedWrite = float64(write)
|
||||
}
|
||||
}
|
||||
writeRate := rates.Input
|
||||
if rates.CacheWrite > 0 {
|
||||
writeRate = rates.CacheWrite
|
||||
}
|
||||
total := (float64(hit)*rates.CacheHit +
|
||||
float64(miss-write)*rates.Input +
|
||||
billedWrite*writeRate +
|
||||
float64(u.CompletionTokens)*rates.Output) / 1e6
|
||||
return NewAmountFromFloat(total)
|
||||
}
|
||||
|
||||
// PricingFingerprint hashes a rate card for ledger keys.
|
||||
func PricingFingerprint(rates RateCard) string {
|
||||
cur := NormalizeCurrency(rates.Currency)
|
||||
raw := fmt.Sprintf("%s|%.12g|%.12g|%.12g", cur, rates.CacheHit, rates.Input, rates.Output)
|
||||
if rates.CacheWrite != 0 {
|
||||
// Appended only when set, so every card stored before the field keeps its key.
|
||||
raw += fmt.Sprintf("|w%.12g", rates.CacheWrite)
|
||||
}
|
||||
sum := sha256.Sum256([]byte(raw))
|
||||
return hex.EncodeToString(sum[:8])
|
||||
}
|
||||
|
||||
// BuildQuote computes original cost and CNY/USD valuations.
|
||||
func BuildQuote(in QuoteInput) CostQuote {
|
||||
state := newQuoteBuildState(in)
|
||||
for _, target := range []string{"CNY", "USD"} {
|
||||
state.addTargetValuation(target)
|
||||
}
|
||||
state.selectDisplay()
|
||||
return state.quote
|
||||
}
|
||||
|
||||
type quoteBuildState struct {
|
||||
input QuoteInput
|
||||
currency string
|
||||
amount Amount
|
||||
occurred time.Time
|
||||
official CatalogEntry
|
||||
isOfficial bool
|
||||
valuationFailures map[string]string
|
||||
quote CostQuote
|
||||
}
|
||||
|
||||
func newQuoteBuildState(in QuoteInput) *quoteBuildState {
|
||||
currency := NormalizeCurrency(in.Rates.Currency)
|
||||
if currency == "" {
|
||||
currency = "CNY"
|
||||
}
|
||||
mode := strings.TrimSpace(in.BillingMode)
|
||||
if mode == "" {
|
||||
mode = BillingModePAYG
|
||||
}
|
||||
occurred := in.OccurredAt
|
||||
if occurred.IsZero() {
|
||||
occurred = time.Now().UTC()
|
||||
}
|
||||
fingerprint := strings.TrimSpace(in.PricingFingerprint)
|
||||
if fingerprint == "" {
|
||||
fingerprint = PricingFingerprint(in.Rates)
|
||||
}
|
||||
amount := OriginalCostAmount(in.Rates, in.Usage)
|
||||
q := CostQuote{
|
||||
Original: MoneyOf(amount, currency),
|
||||
Valuations: map[string]Valuation{},
|
||||
BillingMode: mode,
|
||||
Estimated: true,
|
||||
CostComplete: true,
|
||||
Coverage: CoverageComplete,
|
||||
DisplayComplete: true,
|
||||
Complete: true,
|
||||
DisplayStatus: DisplayStatusMatched,
|
||||
AggregateMode: AggregateModeSingleCurrency,
|
||||
ModelRef: strings.TrimSpace(in.ModelRef),
|
||||
UsageSource: strings.TrimSpace(in.UsageSource),
|
||||
PricingFingerprint: fingerprint,
|
||||
CatalogSource: strings.TrimSpace(in.CatalogSource),
|
||||
}
|
||||
q.Valuations[currency] = Valuation{
|
||||
Money: q.Original, Basis: BasisIdentity,
|
||||
Source: firstNonEmpty(in.CatalogSource, "rate_card"),
|
||||
AsOf: occurred.UTC().Format("2006-01-02"),
|
||||
}
|
||||
state := "eBuildState{
|
||||
input: in, currency: currency, amount: amount, occurred: occurred,
|
||||
valuationFailures: map[string]string{}, quote: q,
|
||||
}
|
||||
if !usageHasFacts(in.Usage) {
|
||||
state.markIncomplete("missing_price_or_usage")
|
||||
}
|
||||
state.matchOfficialCatalog()
|
||||
state.applyOfficialSchedule()
|
||||
return state
|
||||
}
|
||||
|
||||
// applyOfficialSchedule re-prices the turn at the vendor's rate for the hour it
|
||||
// actually happened. Only a rate confirmed as the vendor's own earns this: a
|
||||
// custom price is the user's own arrangement, and projecting someone else's
|
||||
// peak schedule onto it would invent a number nobody published.
|
||||
func (s *quoteBuildState) applyOfficialSchedule() {
|
||||
if !s.isOfficial || s.official.Peak == nil {
|
||||
return
|
||||
}
|
||||
s.quote.RateBand = RateBandOffPeak
|
||||
if s.official.Window.IsPeak(s.occurred) {
|
||||
s.quote.RateBand = RateBandPeak
|
||||
}
|
||||
rates := s.official.RatesAt(s.occurred)
|
||||
if rates.CacheHit == s.input.Rates.CacheHit && rates.Input == s.input.Rates.Input && rates.Output == s.input.Rates.Output {
|
||||
return
|
||||
}
|
||||
s.amount = OriginalCostAmount(rates, s.input.Usage)
|
||||
s.quote.Original = MoneyOf(s.amount, s.currency)
|
||||
s.quote.PricingFingerprint = PricingFingerprint(rates)
|
||||
s.quote.Valuations[s.currency] = Valuation{
|
||||
Money: s.quote.Original, Basis: BasisOfficialTable,
|
||||
Source: s.official.DocURL,
|
||||
AsOf: s.occurred.UTC().Format("2006-01-02"),
|
||||
}
|
||||
}
|
||||
|
||||
func quoteDisplayRequest(in QuoteInput) DisplayRequest {
|
||||
request := in.Display
|
||||
if request.Currency == "" && in.DisplayCurrency != "" {
|
||||
request.Currency = in.DisplayCurrency
|
||||
if request.Source == "" {
|
||||
request.Source = DisplaySourceExplicit
|
||||
}
|
||||
}
|
||||
if request.Source == "" {
|
||||
request.Source = DisplaySourceAuto
|
||||
}
|
||||
return request
|
||||
}
|
||||
|
||||
func usageHasFacts(u UsageTokens) bool {
|
||||
return u.PromptTokens > 0 || u.CompletionTokens > 0 || u.CacheHitTokens > 0 ||
|
||||
u.CacheMissTokens > 0 || u.CacheWriteTokens > 0 || u.CacheWriteBilledTokens > 0
|
||||
}
|
||||
|
||||
func (s *quoteBuildState) matchOfficialCatalog() {
|
||||
providerKind, modelID := resolveCatalogIdentity(s.input)
|
||||
s.official, s.isOfficial = MatchesCatalog(providerKind, modelID, s.input.Rates)
|
||||
if !s.isOfficial && providerKind != "" && modelID != "" {
|
||||
rates := s.input.Rates
|
||||
rates.Currency = s.currency
|
||||
s.official, s.isOfficial = MatchesCatalog(providerKind, modelID, rates)
|
||||
}
|
||||
if s.isOfficial && s.quote.CatalogSource == "" {
|
||||
s.quote.CatalogSource = s.official.DocURL
|
||||
}
|
||||
}
|
||||
|
||||
func (s *quoteBuildState) addTargetValuation(target string) {
|
||||
if target == s.currency {
|
||||
return
|
||||
}
|
||||
if s.addOfficialValuation(target) {
|
||||
return
|
||||
}
|
||||
// Runtime FX has intentionally been removed. A custom price that does not
|
||||
// match the official catalog remains in its original currency.
|
||||
s.valuationFailures[target] = "display_unavailable"
|
||||
}
|
||||
|
||||
func (s *quoteBuildState) addOfficialValuation(target string) bool {
|
||||
if !s.isOfficial {
|
||||
return false
|
||||
}
|
||||
peer, ok := LookupCatalog(
|
||||
s.official.Provider, s.official.Model, target, s.official.BillingMode,
|
||||
)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
s.quote.Valuations[target] = Valuation{
|
||||
Money: MoneyOf(OriginalCostAmount(peer.RatesAt(s.occurred), s.input.Usage), target),
|
||||
Basis: BasisOfficialTable, Source: peer.DocURL,
|
||||
AsOf: s.occurred.UTC().Format("2006-01-02"),
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *quoteBuildState) selectDisplay() {
|
||||
if !s.quote.CostComplete {
|
||||
s.quote.Selected = nil
|
||||
s.quote.DisplayComplete = false
|
||||
s.quote.Complete = false
|
||||
s.quote.DisplayStatus = DisplayStatusUnavailable
|
||||
return
|
||||
}
|
||||
display := quoteDisplayRequest(s.input).NormalizedCurrency()
|
||||
if display == "" {
|
||||
selected := s.quote.Original
|
||||
s.quote.Selected = &selected
|
||||
return
|
||||
}
|
||||
if valuation, ok := s.quote.Valuations[display]; ok {
|
||||
selected := valuation.Money
|
||||
s.quote.Selected = &selected
|
||||
return
|
||||
}
|
||||
// The original price-book amount is still a valid cost fact. Keep it as a
|
||||
// visible fallback and distinguish the display mismatch from no pricing.
|
||||
selected := s.quote.Original
|
||||
s.quote.Selected = &selected
|
||||
s.quote.DisplayComplete = false
|
||||
s.quote.Complete = false
|
||||
s.quote.DisplayStatus = DisplayStatusFallbackOriginal
|
||||
s.quote.IncompleteReason = firstNonEmpty(s.valuationFailures[display], "display_unavailable")
|
||||
}
|
||||
|
||||
func (s *quoteBuildState) markIncomplete(reason string) {
|
||||
s.quote.CostComplete = false
|
||||
s.quote.Coverage = CoverageIncomplete
|
||||
s.quote.DisplayComplete = false
|
||||
s.quote.Complete = false
|
||||
s.quote.DisplayStatus = DisplayStatusUnavailable
|
||||
if s.quote.IncompleteReason == "" {
|
||||
s.quote.IncompleteReason = reason
|
||||
}
|
||||
}
|
||||
|
||||
// SelectForDisplay returns the money for a display currency preference without
|
||||
// recomputing rates. Empty display returns original.
|
||||
func (q CostQuote) SelectForDisplay(display string) (Money, bool) {
|
||||
display = NormalizeCurrency(display)
|
||||
if display == "" {
|
||||
return q.Original, true
|
||||
}
|
||||
if v, ok := q.Valuations[display]; ok {
|
||||
return v.Money, true
|
||||
}
|
||||
if NormalizeCurrency(q.Original.Currency) == display {
|
||||
return q.Original, true
|
||||
}
|
||||
return Money{}, false
|
||||
}
|
||||
|
||||
// WithSelected returns a copy with Selected set for display.
|
||||
func (q CostQuote) WithSelected(display string) CostQuote {
|
||||
out := q
|
||||
if m, ok := q.SelectForDisplay(display); ok {
|
||||
out.Selected = &m
|
||||
if quoteHasCompleteCostFact(q) {
|
||||
out.Complete = true
|
||||
out.IncompleteReason = ""
|
||||
}
|
||||
} else {
|
||||
out.Selected = nil
|
||||
out.Complete = false
|
||||
if out.IncompleteReason == "" {
|
||||
out.IncompleteReason = "display_unavailable"
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// LegacyCostFloat returns the selected (or original) amount as float64 for
|
||||
// compatibility aliases cost / costUsd / total_cost.
|
||||
func (q CostQuote) LegacyCostFloat() float64 {
|
||||
if q.Selected != nil {
|
||||
return q.Selected.Float64()
|
||||
}
|
||||
return q.Original.Float64()
|
||||
}
|
||||
|
||||
// LegacyCurrencySymbol returns a display symbol for the selected/original currency.
|
||||
func (q CostQuote) LegacyCurrencySymbol() string {
|
||||
if q.Selected != nil && q.Selected.Currency != "" {
|
||||
return CurrencySymbol(q.Selected.Currency)
|
||||
}
|
||||
return CurrencySymbol(q.Original.Currency)
|
||||
}
|
||||
|
||||
// LegacyCurrencyCode returns the ISO code for selected/original.
|
||||
func (q CostQuote) LegacyCurrencyCode() string {
|
||||
if q.Selected != nil && q.Selected.Currency != "" {
|
||||
return NormalizeCurrency(q.Selected.Currency)
|
||||
}
|
||||
return NormalizeCurrency(q.Original.Currency)
|
||||
}
|
||||
|
||||
// AggregateQuotes combines occurrence-time quotes into a session/run total.
|
||||
// Every entry must have the requested valuation; partial totals stay unavailable.
|
||||
func AggregateQuotes(quotes []CostQuote, display string) CostQuote {
|
||||
if len(quotes) == 0 {
|
||||
return emptyAggregate(NormalizeCurrency(display))
|
||||
}
|
||||
accumulator := newQuoteAccumulator(display)
|
||||
for _, quote := range quotes {
|
||||
accumulator.add(quote)
|
||||
}
|
||||
return accumulator.finish()
|
||||
}
|
||||
|
||||
type quoteAccumulator struct {
|
||||
out CostQuote
|
||||
display string
|
||||
totals map[string]Amount
|
||||
originalCurrency string
|
||||
originalTotal Amount
|
||||
originalTotals map[string]Amount
|
||||
originalComplete bool
|
||||
coverage string
|
||||
displayComplete bool
|
||||
modes map[string]struct{}
|
||||
rateBands map[string]struct{}
|
||||
unbanded bool
|
||||
}
|
||||
|
||||
func emptyAggregate(display string) CostQuote {
|
||||
out := CostQuote{
|
||||
Original: MoneyOf(Zero, display), Valuations: map[string]Valuation{}, Estimated: true,
|
||||
CostComplete: false, DisplayComplete: false, Complete: false,
|
||||
Coverage: CoverageNone,
|
||||
DisplayStatus: DisplayStatusUnavailable, AggregateMode: AggregateModeSingleCurrency,
|
||||
IncompleteReason: "no_usage",
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func newQuoteAccumulator(display string) *quoteAccumulator {
|
||||
return "eAccumulator{
|
||||
out: CostQuote{Valuations: map[string]Valuation{}, Estimated: true},
|
||||
display: NormalizeCurrency(display), totals: map[string]Amount{}, originalTotals: map[string]Amount{},
|
||||
originalComplete: true, coverage: CoverageNone, displayComplete: true,
|
||||
modes: map[string]struct{}{}, rateBands: map[string]struct{}{},
|
||||
}
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) add(quote CostQuote) {
|
||||
quote = NormalizeQuote(quote)
|
||||
a.coverage = FoldCoverage(a.coverage, quote.Coverage)
|
||||
if quote.Coverage != CoverageComplete && a.out.IncompleteReason == "" {
|
||||
a.out.IncompleteReason = quote.IncompleteReason
|
||||
}
|
||||
a.out.LegacyEstimate = a.out.LegacyEstimate || quote.LegacyEstimate
|
||||
if quote.BillingMode != "" {
|
||||
a.modes[quote.BillingMode] = struct{}{}
|
||||
}
|
||||
a.addRateBand(quote.RateBand)
|
||||
if quote.RateDate > a.out.RateDate {
|
||||
a.out.RateDate = quote.RateDate
|
||||
}
|
||||
originalCurrency := NormalizeCurrency(quote.Original.Currency)
|
||||
if len(quote.OriginalTotals) > 0 {
|
||||
for _, original := range quote.OriginalTotals {
|
||||
a.addOriginal(original, NormalizeCurrency(original.Currency))
|
||||
}
|
||||
} else {
|
||||
a.addOriginal(quote.Original, originalCurrency)
|
||||
}
|
||||
if a.display != "" {
|
||||
_, found := quote.Valuations[a.display]
|
||||
a.displayComplete = a.displayComplete && (found || originalCurrency == a.display)
|
||||
}
|
||||
originalIncluded := false
|
||||
for code, valuation := range quote.Valuations {
|
||||
code = NormalizeCurrency(code)
|
||||
if code == originalCurrency {
|
||||
originalIncluded = true
|
||||
}
|
||||
a.addValuation(code, valuation)
|
||||
}
|
||||
if originalCurrency != "" && !originalIncluded {
|
||||
a.addValuation(originalCurrency, Valuation{
|
||||
Money: quote.Original, Basis: BasisIdentity, Source: "aggregate",
|
||||
AsOf: time.Now().UTC().Format("2006-01-02"),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) addRateBand(band string) {
|
||||
switch band {
|
||||
case RateBandPeak, RateBandOffPeak:
|
||||
a.rateBands[band] = struct{}{}
|
||||
case RateBandMixed:
|
||||
a.rateBands[RateBandPeak], a.rateBands[RateBandOffPeak] = struct{}{}, struct{}{}
|
||||
default:
|
||||
a.unbanded = true
|
||||
}
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) rateBand() string {
|
||||
if a.unbanded {
|
||||
return ""
|
||||
}
|
||||
switch len(a.rateBands) {
|
||||
case 1:
|
||||
for band := range a.rateBands {
|
||||
return band
|
||||
}
|
||||
case 2:
|
||||
return RateBandMixed
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) addOriginal(original Money, currency string) {
|
||||
if currency != "" {
|
||||
a.originalTotals[currency] = a.originalTotals[currency].Add(original.AmountValue())
|
||||
}
|
||||
if a.originalCurrency == "" {
|
||||
a.originalCurrency = currency
|
||||
a.originalTotal = original.AmountValue()
|
||||
return
|
||||
}
|
||||
if currency != a.originalCurrency {
|
||||
a.originalComplete = false
|
||||
return
|
||||
}
|
||||
a.originalTotal = a.originalTotal.Add(original.AmountValue())
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) addValuation(code string, valuation Valuation) {
|
||||
if code == "" {
|
||||
return
|
||||
}
|
||||
a.totals[code] = a.totals[code].Add(valuation.Money.AmountValue())
|
||||
current, found := a.out.Valuations[code]
|
||||
if !found {
|
||||
current = valuation
|
||||
}
|
||||
current.Money = MoneyOf(a.totals[code], code)
|
||||
current.Stale = current.Stale || valuation.Stale
|
||||
a.out.Valuations[code] = current
|
||||
}
|
||||
|
||||
// costFactsComplete reads the accumulated coverage rather than tracking the
|
||||
// same fact a second time: two answers to one question drift.
|
||||
func (a *quoteAccumulator) costFactsComplete() bool {
|
||||
return a.coverage == CoverageComplete
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) finish() CostQuote {
|
||||
a.out.Coverage = a.coverage
|
||||
a.out.CostComplete = a.costFactsComplete()
|
||||
if a.originalComplete && a.originalCurrency != "" {
|
||||
a.out.Original = MoneyOf(a.originalTotal, a.originalCurrency)
|
||||
a.syncOriginalValuation()
|
||||
} else {
|
||||
a.out.Original = Money{Amount: "0"}
|
||||
}
|
||||
if len(a.originalTotals) > 1 {
|
||||
codes := make([]string, 0, len(a.originalTotals))
|
||||
for code := range a.originalTotals {
|
||||
codes = append(codes, code)
|
||||
}
|
||||
slices.Sort(codes)
|
||||
for _, code := range codes {
|
||||
a.out.OriginalTotals = append(a.out.OriginalTotals, MoneyOf(a.originalTotals[code], code))
|
||||
}
|
||||
}
|
||||
a.out.RateBand = a.rateBand()
|
||||
if len(a.modes) == 1 {
|
||||
for mode := range a.modes {
|
||||
a.out.BillingMode = mode
|
||||
}
|
||||
}
|
||||
if a.display == "" && a.originalComplete && a.costFactsComplete() {
|
||||
selected := a.out.Original
|
||||
a.out.Selected = &selected
|
||||
a.out.DisplayComplete = true
|
||||
a.out.Complete = true
|
||||
a.out.DisplayStatus = DisplayStatusMatched
|
||||
a.out.AggregateMode = AggregateModeSingleCurrency
|
||||
return a.out
|
||||
}
|
||||
valuation, found := a.out.Valuations[a.display]
|
||||
if found && a.displayComplete && a.costFactsComplete() {
|
||||
selected := valuation.Money
|
||||
a.out.Selected = &selected
|
||||
a.out.DisplayComplete = true
|
||||
a.out.Complete = true
|
||||
a.out.DisplayStatus = DisplayStatusMatched
|
||||
if len(a.originalTotals) > 1 {
|
||||
a.out.AggregateMode = AggregateModeCommonValuation
|
||||
} else {
|
||||
a.out.AggregateMode = AggregateModeSingleCurrency
|
||||
}
|
||||
return a.out
|
||||
}
|
||||
a.out.DisplayComplete = false
|
||||
a.out.Complete = false
|
||||
if !a.costFactsComplete() {
|
||||
a.out.DisplayStatus = DisplayStatusUnavailable
|
||||
a.out.IncompleteReason = firstNonEmpty(a.out.IncompleteReason, "incomplete_cost_fact")
|
||||
} else if a.originalComplete && a.originalCurrency != "" {
|
||||
selected := a.out.Original
|
||||
a.out.Selected = &selected
|
||||
a.out.DisplayStatus = DisplayStatusFallbackOriginal
|
||||
a.out.AggregateMode = AggregateModeSingleCurrency
|
||||
a.out.IncompleteReason = firstNonEmpty(a.out.IncompleteReason, "display_unavailable")
|
||||
} else {
|
||||
a.out.DisplayStatus = DisplayStatusBucketed
|
||||
a.out.AggregateMode = AggregateModeCurrencyBuckets
|
||||
a.out.IncompleteReason = firstNonEmpty(a.out.IncompleteReason, "currency_buckets")
|
||||
}
|
||||
return a.out
|
||||
}
|
||||
|
||||
// NormalizeQuote fills fields introduced after the first CostQuote wire shape.
|
||||
// It is used at every persistence and transport boundary so old telemetry and
|
||||
// old eventwire payloads remain safe without inventing a new amount.
|
||||
func NormalizeQuote(q CostQuote) CostQuote {
|
||||
if !q.CostComplete && !q.DisplayComplete && q.Complete {
|
||||
q.CostComplete = true
|
||||
q.DisplayComplete = true
|
||||
}
|
||||
if q.DisplayStatus != "" && q.DisplayStatus != DisplayStatusMatched && q.DisplayStatus != DisplayStatusFallbackOriginal && q.DisplayStatus != DisplayStatusBucketed && q.DisplayStatus != DisplayStatusUnavailable {
|
||||
q.DisplayStatus = ""
|
||||
}
|
||||
if q.RateBand != RateBandPeak && q.RateBand != RateBandOffPeak && q.RateBand != RateBandMixed {
|
||||
q.RateBand = ""
|
||||
}
|
||||
if q.AggregateMode != "" && q.AggregateMode != AggregateModeSingleCurrency && q.AggregateMode != AggregateModeCommonValuation && q.AggregateMode != AggregateModeCurrencyBuckets {
|
||||
q.AggregateMode = ""
|
||||
}
|
||||
if q.DisplayStatus == "" {
|
||||
switch {
|
||||
case q.Complete:
|
||||
q.DisplayStatus = DisplayStatusMatched
|
||||
case q.Original.Currency != "" && q.Original.Amount != "" && q.IncompleteReason != "no_price":
|
||||
q.DisplayStatus = DisplayStatusFallbackOriginal
|
||||
default:
|
||||
q.DisplayStatus = DisplayStatusUnavailable
|
||||
}
|
||||
}
|
||||
if q.AggregateMode == "" {
|
||||
if len(q.OriginalTotals) > 1 {
|
||||
q.AggregateMode = AggregateModeCurrencyBuckets
|
||||
} else if q.Selected != nil {
|
||||
q.AggregateMode = AggregateModeSingleCurrency
|
||||
}
|
||||
}
|
||||
if !ValidCoverage(q.Coverage) {
|
||||
q.Coverage = CoverageIncomplete
|
||||
if quoteHasCompleteCostFact(q) {
|
||||
q.Coverage = CoverageComplete
|
||||
}
|
||||
}
|
||||
q.Complete = q.DisplayComplete
|
||||
return q
|
||||
}
|
||||
|
||||
// quoteHasCompleteCostFact separates a known original-currency charge from a
|
||||
// display-only valuation failure must not poison an exact original total,
|
||||
// while unpriced and unrecoverable legacy records stay incomplete in every
|
||||
// display currency.
|
||||
func quoteHasCompleteCostFact(q CostQuote) bool {
|
||||
if q.CostComplete {
|
||||
return true
|
||||
}
|
||||
switch q.IncompleteReason {
|
||||
case "no_price", "missing_price_or_usage", "legacy_unrecoverable",
|
||||
"legacy_wiped_or_zero", "legacy_invalid_amount", "mixed_original_currencies",
|
||||
"incomplete_cost_fact":
|
||||
return false
|
||||
}
|
||||
if q.Complete {
|
||||
return true
|
||||
}
|
||||
currency := NormalizeCurrency(q.Original.Currency)
|
||||
if currency == "" {
|
||||
return false
|
||||
}
|
||||
valuation, ok := q.Valuations[currency]
|
||||
return ok && SameCurrency(valuation.Money.Currency, currency)
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) syncOriginalValuation() {
|
||||
valuation, found := a.out.Valuations[a.originalCurrency]
|
||||
if !found {
|
||||
valuation = Valuation{
|
||||
Basis: BasisIdentity, Source: "aggregate",
|
||||
AsOf: time.Now().UTC().Format("2006-01-02"),
|
||||
}
|
||||
}
|
||||
valuation.Money = a.out.Original
|
||||
a.out.Valuations[a.originalCurrency] = valuation
|
||||
}
|
||||
|
||||
func firstNonEmpty(values ...string) string {
|
||||
for _, v := range values {
|
||||
if strings.TrimSpace(v) != "" {
|
||||
return strings.TrimSpace(v)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// resolveCatalogIdentity names which official table, if any, may price a quote.
|
||||
// The vendor comes only from ProviderKind, which the caller reads off the
|
||||
// endpoint's host; guessing it from the entry's name or the model id would bill
|
||||
// a relay as the vendor it resells. Unnamed means no official table — the user's
|
||||
// own rate card, which is right rather than merely plausible.
|
||||
func resolveCatalogIdentity(in QuoteInput) (provider, model string) {
|
||||
provider = strings.ToLower(strings.TrimSpace(in.ProviderKind))
|
||||
model = strings.TrimSpace(in.ModelID)
|
||||
if model == "" {
|
||||
ref := strings.TrimSpace(in.ModelRef)
|
||||
if i := strings.LastIndex(ref, "/"); i >= 0 && i+1 < len(ref) {
|
||||
model = ref[i+1:]
|
||||
} else {
|
||||
model = ref
|
||||
}
|
||||
}
|
||||
return provider, model
|
||||
}
|
||||
+358
@@ -0,0 +1,358 @@
|
||||
// rates.go — every rate this project has shipped for a vendor's model, and the
|
||||
// two questions that asks: what it costs now, and what it used to cost.
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// rateGeneration is one published pair. Peak is nil for a generation the vendor
|
||||
// billed at a single rate around the clock.
|
||||
type rateGeneration struct {
|
||||
Base RateCard
|
||||
Peak *RateCard
|
||||
// From is the first day (YYYY-MM-DD, UTC) this generation is billed; empty
|
||||
// means from when it shipped. An announced change is appended with its date.
|
||||
From string
|
||||
}
|
||||
|
||||
// today is the day a dated generation is read against; a test replaces it.
|
||||
var today = func() string { return time.Now().UTC().Format("2006-01-02") }
|
||||
|
||||
// currentIndex is the generation in force today: the last one whose date has
|
||||
// come. Rows with no dated generation answer with their last, as before.
|
||||
func (r officialRateRow) currentIndex() int {
|
||||
day := today()
|
||||
for i := len(r.Rates) - 1; i > 0; i-- {
|
||||
if r.Rates[i].From == "" || r.Rates[i].From <= day {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// officialRateRow is one vendor's model, in one currency, under one billing
|
||||
// mode, with every rate this project has shipped for it.
|
||||
type officialRateRow struct {
|
||||
Provider string
|
||||
Model string
|
||||
Currency string
|
||||
BillingMode string
|
||||
Window *PeakWindow
|
||||
DocURL string
|
||||
Notes string
|
||||
CheckedOn string
|
||||
// Rates is oldest first; the last entry is what this costs today.
|
||||
Rates []rateGeneration
|
||||
}
|
||||
|
||||
// officialRates is the price book. A price change appends a generation rather
|
||||
// than editing one: that is what leaves the superseded rate where a stored price
|
||||
// can still be recognised as this project's own default instead of a number its
|
||||
// user chose, which is the difference between refreshing one and losing it.
|
||||
var officialRates = []officialRateRow{
|
||||
{
|
||||
Provider: "deepseek", Model: "deepseek-flash", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, Window: deepseekPeak, DocURL: DocDeepSeekPricing, CheckedOn: deepseekCheckedOn,
|
||||
Rates: []rateGeneration{
|
||||
{Base: RateCard{CacheHit: 0.02, Input: 1, Output: 2, Currency: "CNY"}},
|
||||
{Base: RateCard{CacheHit: 0.05, Input: 1.5, Output: 4.5, Currency: "CNY"},
|
||||
Peak: &RateCard{CacheHit: 0.10, Input: 3.0, Output: 9.0, Currency: "CNY"}},
|
||||
{Base: RateCard{CacheHit: 0.02, Input: 1, Output: 4, Currency: "CNY"},
|
||||
Peak: &RateCard{CacheHit: 0.04, Input: 2, Output: 8, Currency: "CNY"}},
|
||||
},
|
||||
},
|
||||
{
|
||||
Provider: "deepseek", Model: "deepseek-flash", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, Window: deepseekPeak, DocURL: DocDeepSeekPricing, CheckedOn: deepseekCheckedOn,
|
||||
Rates: []rateGeneration{
|
||||
{Base: RateCard{CacheHit: 0.0028, Input: 0.14, Output: 0.28, Currency: "USD"}},
|
||||
{Base: RateCard{CacheHit: 0.007, Input: 0.22, Output: 0.66, Currency: "USD"},
|
||||
Peak: &RateCard{CacheHit: 0.014, Input: 0.44, Output: 1.32, Currency: "USD"}},
|
||||
{Base: RateCard{CacheHit: 0.003, Input: 0.15, Output: 0.6, Currency: "USD"},
|
||||
Peak: &RateCard{CacheHit: 0.006, Input: 0.3, Output: 1.2, Currency: "USD"}},
|
||||
},
|
||||
},
|
||||
{
|
||||
Provider: "deepseek", Model: "deepseek-v4-pro", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, Window: deepseekPeak, DocURL: DocDeepSeekPricing, CheckedOn: deepseekCheckedOn,
|
||||
Rates: []rateGeneration{
|
||||
{Base: RateCard{CacheHit: 0.025, Input: 3, Output: 6, Currency: "CNY"}},
|
||||
{Base: RateCard{CacheHit: 0.15, Input: 4.5, Output: 13.5, Currency: "CNY"},
|
||||
Peak: &RateCard{CacheHit: 0.30, Input: 9.0, Output: 27.0, Currency: "CNY"}},
|
||||
},
|
||||
},
|
||||
{
|
||||
Provider: "deepseek", Model: "deepseek-v4-pro", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, Window: deepseekPeak, DocURL: DocDeepSeekPricing, CheckedOn: deepseekCheckedOn,
|
||||
Rates: []rateGeneration{
|
||||
{Base: RateCard{CacheHit: 0.003625, Input: 0.435, Output: 0.87, Currency: "USD"}},
|
||||
{Base: RateCard{CacheHit: 0.022, Input: 0.66, Output: 1.98, Currency: "USD"},
|
||||
Peak: &RateCard{CacheHit: 0.044, Input: 1.32, Output: 3.96, Currency: "USD"}},
|
||||
},
|
||||
},
|
||||
|
||||
// LongCat's launch discount, for which the vendor publishes no end date. List
|
||||
// price is CNY 0.10/5/20, USD 0.015/0.75/2.95 — what these revert to, and why
|
||||
// CheckedOn earns its place here.
|
||||
{
|
||||
Provider: "longcat", Model: "LongCat-2.0", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocLongCatPricingCNY, Notes: "launch_discount_no_end_date", CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.04, Input: 2, Output: 8, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "longcat", Model: "LongCat-2.0", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, DocURL: DocLongCatPricingUSD, Notes: "launch_discount_no_end_date", CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.006, Input: 0.30, Output: 1.20, Currency: "USD"}}},
|
||||
},
|
||||
|
||||
// Google: the two flash models double on 2027-01-01, hence the dated generation.
|
||||
// gemini-3.1-pro-preview is the rate up to 200k prompt tokens; gemini-4-argon has no price.
|
||||
{
|
||||
Provider: "google", Model: "gemini-3.8-flash", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, DocURL: DocGeminiPricing, CheckedOn: googleCheckedOn,
|
||||
Rates: []rateGeneration{
|
||||
{Base: RateCard{CacheHit: 0.075, Input: 0.75, Output: 3.75, Currency: "USD"}},
|
||||
{From: "2027-01-01", Base: RateCard{CacheHit: 0.15, Input: 1.50, Output: 7.50, Currency: "USD"}},
|
||||
},
|
||||
},
|
||||
{
|
||||
Provider: "google", Model: "gemini-3.6-flash", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, DocURL: DocGeminiPricing, CheckedOn: googleCheckedOn,
|
||||
Rates: []rateGeneration{
|
||||
{Base: RateCard{CacheHit: 0.075, Input: 0.75, Output: 3.75, Currency: "USD"}},
|
||||
{From: "2027-01-01", Base: RateCard{CacheHit: 0.15, Input: 1.50, Output: 7.50, Currency: "USD"}},
|
||||
},
|
||||
},
|
||||
{
|
||||
Provider: "google", Model: "gemini-3.5-flash-lite", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, DocURL: DocGeminiPricing, CheckedOn: googleCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.03, Input: 0.30, Output: 2.50, Currency: "USD"}}},
|
||||
},
|
||||
{
|
||||
Provider: "google", Model: "gemini-3.1-pro-preview", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, DocURL: DocGeminiPricing, Notes: "up_to_200k_prompt_tokens", CheckedOn: googleCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.20, Input: 2.00, Output: 12.00, Currency: "USD"}}},
|
||||
},
|
||||
|
||||
{
|
||||
Provider: "anthropic", Model: "claude-sonnet-4-5", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, DocURL: DocAnthropicPricing, CheckedOn: anthropicCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.30, Input: 3, Output: 15, CacheWrite: 3.75, Currency: "USD"}}},
|
||||
},
|
||||
|
||||
// MiMo domestic PAYG; Token Plan bills the same rates as subscription_equivalent.
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.5-pro", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocMiMoPAYG, CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.025, Input: 3, Output: 6, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.5", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocMiMoPAYG, CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.02, Input: 1, Output: 2, Currency: "CNY"}}},
|
||||
},
|
||||
// Not on the vendor's price page as of CheckedOn. Kept because removing a rate
|
||||
// is what turns a still-configured model's cost silently into zero; a stale
|
||||
// rate at least reads as a number somebody can question.
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2-flash", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocMiMoPAYG, Notes: "not_on_vendor_page", CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.07, Input: 0.70, Output: 2.10, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.5-pro", Currency: "CNY",
|
||||
BillingMode: BillingModeSubscriptionEquivalent, DocURL: DocMiMoTokenPlan, Notes: "payg_equivalent_not_plan_bill", CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.025, Input: 3, Output: 6, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.5", Currency: "CNY",
|
||||
BillingMode: BillingModeSubscriptionEquivalent, DocURL: DocMiMoTokenPlan, Notes: "payg_equivalent_not_plan_bill", CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.02, Input: 1, Output: 2, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.6-pro", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocMiMoPAYG, CheckedOn: mimoCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.025, Input: 3, Output: 6, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.6-flash", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocMiMoPAYG, CheckedOn: mimoCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.02, Input: 1, Output: 2, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.6-pro-ultraspeed", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocMiMoPAYG, CheckedOn: mimoCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.25, Input: 30, Output: 60, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.6-pro", Currency: "CNY",
|
||||
BillingMode: BillingModeSubscriptionEquivalent, DocURL: DocMiMoTokenPlan, Notes: "payg_equivalent_not_plan_bill", CheckedOn: mimoCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.025, Input: 3, Output: 6, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.6-flash", Currency: "CNY",
|
||||
BillingMode: BillingModeSubscriptionEquivalent, DocURL: DocMiMoTokenPlan, Notes: "payg_equivalent_not_plan_bill", CheckedOn: mimoCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.02, Input: 1, Output: 2, Currency: "CNY"}}},
|
||||
},
|
||||
}
|
||||
|
||||
// retiredDeepSeekFlashModels are names DeepSeek still answers as its current
|
||||
// flash model, at that model's price. Owned here because that is where the rates
|
||||
// they bill at live; config reads them through the accessor below.
|
||||
var retiredDeepSeekFlashModels = []string{"deepseek-v4-flash", "deepseek-v4-flash-vision-exp"}
|
||||
|
||||
// RetiredDeepSeekFlashModels copies the list on the way out: a caller sorting or
|
||||
// trimming the result would otherwise change what every other caller sees.
|
||||
func RetiredDeepSeekFlashModels() []string {
|
||||
return append([]string(nil), retiredDeepSeekFlashModels...)
|
||||
}
|
||||
|
||||
// RateModelFor answers which model's rate a name is billed at — a retired name
|
||||
// at the model that replaced it, an older spelling at the one it was renamed
|
||||
// from. Scoped to billing on purpose: two models can share a rate without being
|
||||
// the same model, and nothing outside a price lookup may read this as identity.
|
||||
func RateModelFor(provider, model string) string {
|
||||
model = strings.TrimSpace(model)
|
||||
switch strings.ToLower(strings.TrimSpace(provider)) {
|
||||
case "deepseek":
|
||||
if slices.Contains(retiredDeepSeekFlashModels, model) {
|
||||
return "deepseek-flash"
|
||||
}
|
||||
case "mimo":
|
||||
switch model {
|
||||
case "mimo-v2-pro":
|
||||
return "mimo-v2.5-pro"
|
||||
case "mimo-v2-omni":
|
||||
return "mimo-v2.5"
|
||||
}
|
||||
}
|
||||
return model
|
||||
}
|
||||
|
||||
// DefaultCurrency is what a vendor's rates are quoted in when an entry does not
|
||||
// say. A vendor publishing one currency leaves no choice to make; DeepSeek
|
||||
// prices in both, and USD is what an entry that never chose has been read as.
|
||||
func DefaultCurrency(provider string) string {
|
||||
provider = strings.ToLower(strings.TrimSpace(provider))
|
||||
seen := map[string]bool{}
|
||||
for _, row := range officialRates {
|
||||
if row.Provider == provider {
|
||||
seen[row.Currency] = true
|
||||
}
|
||||
}
|
||||
if len(seen) == 1 {
|
||||
for currency := range seen {
|
||||
return currency
|
||||
}
|
||||
}
|
||||
return "USD"
|
||||
}
|
||||
|
||||
// officialRateRowsFor finds the rows for a model, which is more than one when a
|
||||
// vendor publishes the same rates under two billing modes.
|
||||
func officialRateRowsFor(provider, model, currency string) []officialRateRow {
|
||||
provider, currency = strings.ToLower(strings.TrimSpace(provider)), NormalizeCurrency(currency)
|
||||
model = RateModelFor(provider, model)
|
||||
var out []officialRateRow
|
||||
for _, row := range officialRates {
|
||||
if row.Provider == provider && row.Model == model && row.Currency == currency {
|
||||
out = append(out, row)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// CurrentRate is what a vendor's model costs in a currency today — the off-peak
|
||||
// card, which is what a caller charging one rate should quote. Nil when this
|
||||
// catalog prices neither. A model published under two billing modes quotes the
|
||||
// pay-as-you-go rate, which is the one an unsubscribed caller is charged.
|
||||
func CurrentRate(provider, model, currency string) *RateCard {
|
||||
for _, row := range officialRateRowsFor(provider, model, currency) {
|
||||
if row.BillingMode == BillingModeSubscriptionEquivalent {
|
||||
continue
|
||||
}
|
||||
if len(row.Rates) > 0 {
|
||||
rate := row.Rates[row.currentIndex()].Base
|
||||
return &rate
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// IsOfficialCard reports whether card is a rate this project has shipped for the
|
||||
// model in the card's own currency: any generation, its off-peak or its peak
|
||||
// card. A card that is none of them is the user's, or damaged, and cannot be
|
||||
// valued against the vendor's other-currency table.
|
||||
func IsOfficialCard(provider, model string, card RateCard) bool {
|
||||
for _, row := range officialRateRowsFor(provider, model, card.Currency) {
|
||||
if row.BillingMode == BillingModeSubscriptionEquivalent {
|
||||
continue
|
||||
}
|
||||
for _, generation := range row.Rates {
|
||||
if sameCard(generation.Base, card) || generation.Peak != nil && sameCard(*generation.Peak, card) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func sameCard(a, b RateCard) bool {
|
||||
return a.CacheHit == b.CacheHit && a.Input == b.Input && a.Output == b.Output && a.CacheWrite == b.CacheWrite
|
||||
}
|
||||
|
||||
// SupersededRates are the rates shipped for a model before the current one. A
|
||||
// stored price matching one came from this project rather than from its user, so
|
||||
// a refresh may replace it; anything else is theirs and is left alone.
|
||||
func SupersededRates(provider, model, currency string) []RateCard {
|
||||
var out []RateCard
|
||||
for _, row := range officialRateRowsFor(provider, model, currency) {
|
||||
if row.BillingMode == BillingModeSubscriptionEquivalent || len(row.Rates) < 2 {
|
||||
continue
|
||||
}
|
||||
for _, generation := range row.Rates[:row.currentIndex()] {
|
||||
out = append(out, generation.Base)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// officialCatalogRows publishes the current generation of every row, plus the
|
||||
// retired DeepSeek names, which bill at the flash rate under their own id.
|
||||
func officialCatalogRows() []CatalogEntry {
|
||||
rows := make([]CatalogEntry, 0, len(officialRates)+len(retiredDeepSeekFlashModels)*2)
|
||||
for _, row := range officialRates {
|
||||
if len(row.Rates) == 0 {
|
||||
continue
|
||||
}
|
||||
current := row.Rates[row.currentIndex()]
|
||||
rows = append(rows, row.entry(row.Model, current))
|
||||
if row.Provider != "deepseek" || row.Model != "deepseek-flash" {
|
||||
continue
|
||||
}
|
||||
for _, retired := range retiredDeepSeekFlashModels {
|
||||
rows = append(rows, row.entry(retired, current))
|
||||
}
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
func (r officialRateRow) entry(model string, current rateGeneration) CatalogEntry {
|
||||
// Copied, so a caller holding a returned row cannot reach back into the
|
||||
// table every other caller reads.
|
||||
var peak *RateCard
|
||||
if current.Peak != nil {
|
||||
card := *current.Peak
|
||||
peak = &card
|
||||
}
|
||||
return CatalogEntry{
|
||||
Provider: r.Provider, Model: model, Currency: r.Currency,
|
||||
CacheHit: current.Base.CacheHit, Input: current.Base.Input, Output: current.Base.Output,
|
||||
CacheWrite: current.Base.CacheWrite,
|
||||
Peak: peak, Window: r.Window, DocURL: r.DocURL,
|
||||
BillingMode: r.BillingMode, Notes: r.Notes, CheckedOn: r.CheckedOn,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
prompt = """Some providers bill tokens written into their prompt cache at a rate of their own. Add that to the pricing contract in this repository:
|
||||
|
||||
- `RateCard` gains a `CacheWrite` rate, per 1M tokens like the others. Zero means the card has no separate cache-write price, and cache-written tokens keep being charged at `Input` exactly as they are today.
|
||||
- When a card sets `CacheWrite`, `OriginalCostAmount` charges the billed cache-write units (`CacheWriteBilledTokens` when set, otherwise `CacheWriteTokens`) at `CacheWrite` instead of `Input`.
|
||||
- `PricingFingerprint` must tell apart two cards that differ only in `CacheWrite`, but a card without one must keep exactly the fingerprint it has today: stored ledgers are keyed by it.
|
||||
- Two cards that differ only in `CacheWrite` are different cards everywhere cards are compared, including official-card recognition and catalog matching.
|
||||
- `CatalogEntry` carries the rate too, and `RateCardFromCatalog` and `CatalogEntry.RatesAt` hand it back.
|
||||
- Add Anthropic to the official price book: `claude-sonnet-4-5`, pay-as-you-go in USD, cache hit 0.30, input 3, output 15 and cache write 3.75 per 1M tokens, documented at https://www.anthropic.com/pricing and checked 2026-10-19. `api.anthropic.com` is Anthropic's own endpoint.
|
||||
- `benchctl cost` gains a `-write` flag for cache-written tokens, and `benchctl rate` shows a card's cache-write rate when it has one.
|
||||
- Add tests for the new behaviour and update the README's pricing section."""
|
||||
class = "multi-file-feature"
|
||||
timeout_sec = 1200
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
export GOFLAGS=-mod=mod GOTOOLCHAIN=local
|
||||
cat > internal/contract/pricing/zz_grader_test.go <<'GO'
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestGraderCacheWriteCost(t *testing.T) {
|
||||
card := RateCard{CacheHit: 0.3, Input: 3, Output: 15, Currency: "USD"}
|
||||
priced := card
|
||||
priced.CacheWrite = 3.75
|
||||
cases := []struct {
|
||||
name string
|
||||
card RateCard
|
||||
usage UsageTokens
|
||||
want string
|
||||
}{
|
||||
{"no write, no rate", card, UsageTokens{CacheHitTokens: 200000, CacheMissTokens: 800000, CompletionTokens: 50000}, "3.21"},
|
||||
{"no write, rate set", priced, UsageTokens{CacheHitTokens: 200000, CacheMissTokens: 800000, CompletionTokens: 50000}, "3.21"},
|
||||
{"write at input", card, UsageTokens{CacheMissTokens: 1000000, CacheWriteTokens: 400000, CompletionTokens: 100000}, "4.5"},
|
||||
{"write at cache-write", priced, UsageTokens{CacheMissTokens: 1000000, CacheWriteTokens: 400000, CompletionTokens: 100000}, "4.8"},
|
||||
{"billed units at input", card, UsageTokens{CacheMissTokens: 1000000, CacheWriteTokens: 400000, CacheWriteBilledTokens: 500000}, "3.3"},
|
||||
{"billed units at cache-write", priced, UsageTokens{CacheMissTokens: 1000000, CacheWriteTokens: 400000, CacheWriteBilledTokens: 500000}, "3.675"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := OriginalCostAmount(tc.card, tc.usage).String(); got != tc.want {
|
||||
t.Errorf("%s: cost = %s, want %s", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGraderFingerprintKeepsStoredKeys(t *testing.T) {
|
||||
cny := RateCard{CacheHit: 0.02, Input: 1, Output: 4, Currency: "CNY"}
|
||||
usd := RateCard{CacheHit: 0.3, Input: 3, Output: 15, Currency: "USD"}
|
||||
if got := PricingFingerprint(cny); got != "ba96037de26c3f79" {
|
||||
t.Errorf("fingerprint of a card without CacheWrite moved: %s", got)
|
||||
}
|
||||
if got := PricingFingerprint(usd); got != "e16b53b4599e5fda" {
|
||||
t.Errorf("fingerprint of a card without CacheWrite moved: %s", got)
|
||||
}
|
||||
withWrite := usd
|
||||
withWrite.CacheWrite = 3.75
|
||||
if PricingFingerprint(withWrite) == PricingFingerprint(usd) {
|
||||
t.Error("cards differing only in CacheWrite share a fingerprint")
|
||||
}
|
||||
other := usd
|
||||
other.CacheWrite = 6
|
||||
if PricingFingerprint(withWrite) == PricingFingerprint(other) {
|
||||
t.Error("two different CacheWrite rates share a fingerprint")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGraderCacheWriteIsPartOfCardIdentity(t *testing.T) {
|
||||
official := RateCard{CacheHit: 0.02, Input: 1, Output: 2, Currency: "CNY"}
|
||||
if !IsOfficialCard("mimo", "mimo-v2.5", official) {
|
||||
t.Fatal("the official mimo-v2.5 card is no longer recognised")
|
||||
}
|
||||
changed := official
|
||||
changed.CacheWrite = 1.25
|
||||
if IsOfficialCard("mimo", "mimo-v2.5", changed) {
|
||||
t.Error("a card with a cache-write rate the vendor never published is recognised as official")
|
||||
}
|
||||
if _, ok := MatchesCatalog("mimo", "mimo-v2.5", official); !ok {
|
||||
t.Fatal("the official mimo-v2.5 card no longer matches the catalog")
|
||||
}
|
||||
if _, ok := MatchesCatalog("mimo", "mimo-v2.5", changed); ok {
|
||||
t.Error("a card with an unpublished cache-write rate matches the catalog")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGraderAnthropicIsPriced(t *testing.T) {
|
||||
want := RateCard{CacheHit: 0.30, Input: 3, Output: 15, CacheWrite: 3.75, Currency: "USD"}
|
||||
got := CurrentRate("anthropic", "claude-sonnet-4-5", "USD")
|
||||
if got == nil || *got != want {
|
||||
t.Fatalf("CurrentRate(anthropic, claude-sonnet-4-5) = %+v, want %+v", got, want)
|
||||
}
|
||||
if !IsOfficialCard("anthropic", "claude-sonnet-4-5", want) {
|
||||
t.Error("the Anthropic card is not recognised as official")
|
||||
}
|
||||
e, ok := LookupCatalog("anthropic", "claude-sonnet-4-5", "USD", "")
|
||||
if !ok || e.DocURL != "https://www.anthropic.com/pricing" || e.CheckedOn != "2026-10-19" || e.CacheWrite != 3.75 {
|
||||
t.Errorf("catalog entry = %+v, %v", e, ok)
|
||||
}
|
||||
if got := OfficialProviderForEndpoint("https://api.anthropic.com/v1/messages"); got != "anthropic" {
|
||||
t.Errorf("api.anthropic.com resolves to %q", got)
|
||||
}
|
||||
if got := DefaultCurrency("anthropic"); got != "USD" {
|
||||
t.Errorf("DefaultCurrency(anthropic) = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGraderCatalogCarriesCacheWrite(t *testing.T) {
|
||||
e := CatalogEntry{Provider: "x", Model: "y", Currency: "USD", CacheHit: 0.1, Input: 1, Output: 2, CacheWrite: 1.25}
|
||||
if got := RateCardFromCatalog(e); got.CacheWrite != 1.25 || got.Input != 1 {
|
||||
t.Errorf("RateCardFromCatalog = %+v", got)
|
||||
}
|
||||
if got := e.RatesAt(time.Date(2026, 3, 4, 12, 0, 0, 0, time.UTC)); got.CacheWrite != 1.25 {
|
||||
t.Errorf("RatesAt = %+v", got)
|
||||
}
|
||||
}
|
||||
GO
|
||||
gofmt_out="$(gofmt -l .)"
|
||||
[ -z "$gofmt_out" ] || { echo "gofmt needed: $gofmt_out"; exit 1; }
|
||||
go vet ./...
|
||||
go test ./... >/dev/null
|
||||
out="$(go run ./cmd/benchctl cost -provider mimo -model mimo-v2.5 -miss 1000000 -write 400000)"
|
||||
[ "$out" = "1 CNY" ] || { echo "benchctl cost -write: got $out"; exit 1; }
|
||||
out="$(go run ./cmd/benchctl rate -provider anthropic -model claude-sonnet-4-5)"
|
||||
case "$out" in *"cache-write 3.75"*) ;; *) echo "benchctl rate does not show the cache-write rate: $out"; exit 1 ;; esac
|
||||
out="$(go run ./cmd/benchctl rate -provider mimo -model mimo-v2.5)"
|
||||
case "$out" in *"cache-write"*) echo "benchctl rate shows a cache-write rate the card does not have: $out"; exit 1 ;; esac
|
||||
grep -lE 'CacheWrite:' internal/contract/pricing/*_test.go | grep -qv zz_grader_test.go \
|
||||
|| { echo "no test of the pricing package sets a CacheWrite rate"; exit 1; }
|
||||
awk '/^## Pricing/{on=1; next} /^## /{on=0} on' README.md | grep -q 'CacheWrite' \
|
||||
|| { echo "README's pricing section does not mention CacheWrite"; exit 1; }
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: price-change
|
||||
description: Record a vendor's announced price change in the official price book without losing the rate it replaces.
|
||||
---
|
||||
|
||||
# Recording a price change
|
||||
|
||||
1. Find the vendor's rows in `officialRates` in
|
||||
`internal/contract/pricing/rates.go`. Each row is one model in one currency.
|
||||
2. **Append** a `rateGeneration` to the row's `Rates`; never edit an existing
|
||||
one. Set `From` to the first billed day (`YYYY-MM-DD`, UTC) when the change
|
||||
is announced ahead of time.
|
||||
3. Leave the earlier generation where it is: `SupersededRates` reads it so a
|
||||
stored price that matches it is recognised as ours and refreshed.
|
||||
4. Update the row's `CheckedOn` constant to the day you confirmed the page.
|
||||
5. Add a test beside `rates_test.go` that pins `CurrentRate` before and after
|
||||
the `From` day by replacing `today`.
|
||||
6. Run `make check`. `make pricecheck` needs the network and is run by hand.
|
||||
@@ -0,0 +1,17 @@
|
||||
.PHONY: check test vet fmt pricecheck
|
||||
|
||||
check: fmt vet test
|
||||
|
||||
fmt:
|
||||
@out="$$(gofmt -l .)"; if [ -n "$$out" ]; then echo "gofmt needed:"; echo "$$out"; exit 1; fi
|
||||
|
||||
vet:
|
||||
go vet ./...
|
||||
|
||||
test:
|
||||
go test ./...
|
||||
|
||||
# Reads each vendor's pricing page; this snapshot carries no network client.
|
||||
pricecheck:
|
||||
@echo "pricecheck: reaches each vendor's pricing page and needs the network; not available in this snapshot" >&2
|
||||
@exit 2
|
||||
@@ -0,0 +1,69 @@
|
||||
# Reasonix kernel snapshot
|
||||
|
||||
A trimmed slice of the Reasonix Go kernel: the contracts a benchmark report is
|
||||
priced and labelled with, a few leaf utilities, and `benchctl`, a small command
|
||||
that prints what those contracts answer.
|
||||
|
||||
```bash
|
||||
make check # gofmt, vet, test
|
||||
go run ./cmd/benchctl rate -model deepseek-flash -currency USD
|
||||
go run ./cmd/benchctl cost -provider mimo -model mimo-v2.5 -miss 1000000
|
||||
go run ./cmd/benchctl tariff -currency CNY -at 2026-10-12T02:00:00Z
|
||||
go run ./cmd/benchctl arm -ablate planner,evidence
|
||||
```
|
||||
|
||||
## Packages
|
||||
|
||||
| Package | Owns |
|
||||
| --- | --- |
|
||||
| `internal/contract/pricing` | the official price book, cost quotes, the spend ledger |
|
||||
| `internal/contract/ablation` | which subsystems a benchmark arm switches off, and the arm's name |
|
||||
| `internal/contract/agentgraph` | the vocabulary of one run's execution graph |
|
||||
| `internal/base/retrieval` | BM25 ranking over CJK-aware tokens |
|
||||
| `internal/base/rulelist` | splitting a one-line permission rule list |
|
||||
| `internal/base/tokencount` | a cheap token estimate |
|
||||
| `internal/base/neterr` | classifying transport errors |
|
||||
| `internal/base/nilutil` | typed-nil detection |
|
||||
| `cmd/benchctl` | prints rates, costs, peak schedules and arm names |
|
||||
|
||||
## Pricing
|
||||
|
||||
A `RateCard` is a price per million tokens in one currency:
|
||||
|
||||
| Field | Charges |
|
||||
| --- | --- |
|
||||
| `CacheHit` | prompt tokens served from the provider's cache |
|
||||
| `Input` | prompt tokens that missed the cache, including tokens written to it |
|
||||
| `Output` | completion tokens |
|
||||
|
||||
`OriginalCostAmount` turns a card and a `UsageTokens` breakdown into a
|
||||
fixed-point `Amount`; `PricingFingerprint` keys a card in the ledger, so a
|
||||
stored ledger entry stays attached to the card it was priced with.
|
||||
`CurrentRate` and `SupersededRates` answer from `officialRates`, which only
|
||||
ever grows: a price change appends a generation.
|
||||
|
||||
## Ablation
|
||||
|
||||
`ablation.Parse` reads a comma-separated list of modules to switch off, and
|
||||
`Set.Arm` names the result (`full` for the control). The switchable modules,
|
||||
in the order arm names use:
|
||||
|
||||
| Module | Off means |
|
||||
| --- | --- |
|
||||
| `evidence` | no evidence gate on completion claims |
|
||||
| `planner` | no two-model planning turn |
|
||||
| `subagent` | no delegation to workers |
|
||||
| `retrieval` | no retrieval over the folded transcript |
|
||||
| `compaction` | no context folding |
|
||||
| `upstream` | a dependency edge orders its endpoints without delivering the answer |
|
||||
| `recall-search` | recall can read a given address but cannot search for one |
|
||||
|
||||
The fold-index scale is a separate axis (`-fold-index`), not a module.
|
||||
|
||||
## Skills
|
||||
|
||||
Project skills live under `.reasonix/skills/<name>/SKILL.md`.
|
||||
|
||||
| Skill | Use it to |
|
||||
| --- | --- |
|
||||
| `price-change` | record a vendor's price change in the price book |
|
||||
@@ -0,0 +1,51 @@
|
||||
# Reasonix project memory
|
||||
|
||||
This checkout is a trimmed snapshot of the Reasonix kernel: the billing,
|
||||
ablation and graph contracts and a handful of leaf utilities, plus one small
|
||||
command that drives them. The conventions below are the project's own.
|
||||
|
||||
## Conventions
|
||||
|
||||
- Go kernel under `internal/`; each package owns one concern. A package's long
|
||||
explanation belongs in its `doc.go`, not spread across implementation files.
|
||||
- Judgements read structure, never wording: types, contracts, schemas. An
|
||||
error a caller must tell apart carries an identity — a sentinel, a typed
|
||||
code — never a sentence.
|
||||
- A metric is its definition, not its name. Before a number becomes a finding,
|
||||
state what it counts, from which anchor, over which population, and what it
|
||||
excludes.
|
||||
|
||||
## Comments
|
||||
|
||||
Default is none — the code is the truth. Write one only when the **why** is
|
||||
non-obvious: a hidden constraint, an invariant the type system cannot express,
|
||||
or an external-protocol quirk. A comment states the constraint that holds now;
|
||||
it does not narrate how the code got here. English, always.
|
||||
|
||||
- Declaration doc: ≤5 lines. Package comment: ≤8 lines, or ≤40 in a `doc.go`.
|
||||
- Every other comment: ≤3 lines. Struct-field and trailing `//`: 1 line.
|
||||
- One responsibility per file; 800 lines is the ceiling.
|
||||
|
||||
## Vendor rates
|
||||
|
||||
Official list prices live in one place: `officialRates` in
|
||||
`internal/contract/pricing/rates.go`, oldest generation first, the last being
|
||||
what a model charges today. A price change is an **append, never an edit** —
|
||||
the superseded rate is what lets an installed config be recognised as holding
|
||||
one of ours and brought forward. Callers read these through
|
||||
`pricing.CurrentRate` and `pricing.SupersededRates` and restate none of them.
|
||||
|
||||
`make pricecheck` compares that table against each vendor's own page. It needs
|
||||
the network, so it is not part of `make check`.
|
||||
|
||||
## Before every commit
|
||||
|
||||
```bash
|
||||
make check # gofmt -l, go vet ./..., go test ./...
|
||||
```
|
||||
|
||||
## Change discipline
|
||||
|
||||
List callers, consumers of the same state and sibling features before editing.
|
||||
Fix at the owning layer; a fix that breaks a neighbouring behaviour is a failed
|
||||
fix. Write tests first; they must fail without the change.
|
||||
@@ -0,0 +1,115 @@
|
||||
// Command benchctl prints what the pricing and ablation contracts answer for
|
||||
// one model or one benchmark arm, so a reviewer can check a rate or an arm name
|
||||
// without writing a test.
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
|
||||
"reasonix/internal/contract/ablation"
|
||||
"reasonix/internal/contract/pricing"
|
||||
)
|
||||
|
||||
var errUsage = errors.New("usage: benchctl rate|cost|tariff|arm [flags]")
|
||||
|
||||
func main() {
|
||||
if err := run(os.Args[1:], os.Stdout); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "benchctl:", err)
|
||||
os.Exit(2)
|
||||
}
|
||||
}
|
||||
|
||||
func run(args []string, out io.Writer) error {
|
||||
if len(args) == 0 {
|
||||
return errUsage
|
||||
}
|
||||
switch args[0] {
|
||||
case "rate":
|
||||
return runRate(args[1:], out)
|
||||
case "cost":
|
||||
return runCost(args[1:], out)
|
||||
case "tariff":
|
||||
return runTariff(args[1:], out)
|
||||
case "arm":
|
||||
return runArm(args[1:], out)
|
||||
default:
|
||||
return fmt.Errorf("%w: unknown subcommand %q", errUsage, args[0])
|
||||
}
|
||||
}
|
||||
|
||||
func runRate(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("rate", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
provider := fs.String("provider", "deepseek", "vendor")
|
||||
model := fs.String("model", "deepseek-flash", "model id")
|
||||
currency := fs.String("currency", "", "currency; empty reads the vendor's default")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
cur := *currency
|
||||
if cur == "" {
|
||||
cur = pricing.DefaultCurrency(*provider)
|
||||
}
|
||||
card := pricing.CurrentRate(*provider, *model, cur)
|
||||
if card == nil {
|
||||
return fmt.Errorf("no official rate for %s/%s in %s", *provider, *model, cur)
|
||||
}
|
||||
fmt.Fprintf(out, "%s/%s %s: cache-hit %g input %g output %g per 1M\n",
|
||||
*provider, *model, card.Currency, card.CacheHit, card.Input, card.Output)
|
||||
if entry, ok := pricing.LookupCatalog(*provider, *model, cur, ""); ok && entry.DocURL != "" {
|
||||
fmt.Fprintf(out, "source: %s\n", entry.DocURL)
|
||||
}
|
||||
fmt.Fprintf(out, "superseded generations: %d\n", len(pricing.SupersededRates(*provider, *model, cur)))
|
||||
return nil
|
||||
}
|
||||
|
||||
func runCost(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("cost", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
provider := fs.String("provider", "deepseek", "vendor")
|
||||
model := fs.String("model", "deepseek-flash", "model id")
|
||||
currency := fs.String("currency", "", "currency; empty reads the vendor's default")
|
||||
hit := fs.Int("hit", 0, "cached prompt tokens")
|
||||
miss := fs.Int("miss", 0, "uncached prompt tokens")
|
||||
completion := fs.Int("completion", 0, "completion tokens")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
cur := *currency
|
||||
if cur == "" {
|
||||
cur = pricing.DefaultCurrency(*provider)
|
||||
}
|
||||
card := pricing.CurrentRate(*provider, *model, cur)
|
||||
if card == nil {
|
||||
return fmt.Errorf("no official rate for %s/%s in %s", *provider, *model, cur)
|
||||
}
|
||||
usage := pricing.UsageTokens{CacheHitTokens: *hit, CacheMissTokens: *miss, CompletionTokens: *completion}
|
||||
amount := pricing.OriginalCostAmount(*card, usage)
|
||||
fmt.Fprintf(out, "%s %s\n", amount, pricing.NormalizeCurrency(card.Currency))
|
||||
return nil
|
||||
}
|
||||
|
||||
func runArm(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("arm", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
spec := fs.String("ablate", "", "comma-separated modules to switch off: "+ablation.ModuleList())
|
||||
foldIndex := fs.String("fold-index", "", "fold-index scale: default, half, quarter or off")
|
||||
list := fs.Bool("list", false, "list the switchable modules")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if *list {
|
||||
fmt.Fprintln(out, ablation.ModuleList())
|
||||
return nil
|
||||
}
|
||||
set, err := ablation.ParseArm(*spec, *foldIndex)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(out, set.Arm())
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func runOut(t *testing.T, args ...string) string {
|
||||
t.Helper()
|
||||
var out bytes.Buffer
|
||||
if err := run(args, &out); err != nil {
|
||||
t.Fatalf("run %v: %v", args, err)
|
||||
}
|
||||
return out.String()
|
||||
}
|
||||
|
||||
func TestRateNamesTheCurrentCard(t *testing.T) {
|
||||
got := runOut(t, "rate", "-provider", "mimo", "-model", "mimo-v2.5")
|
||||
if !strings.Contains(got, "mimo/mimo-v2.5 CNY: cache-hit 0.02 input 1 output 2 per 1M") {
|
||||
t.Fatalf("rate output = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateRefusesAnUnpricedModel(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
if err := run([]string{"rate", "-provider", "nobody", "-model", "x"}, &out); err == nil {
|
||||
t.Fatal("an unpriced model must fail")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCostChargesTheCurrentCard(t *testing.T) {
|
||||
got := runOut(t, "cost", "-provider", "mimo", "-model", "mimo-v2.5", "-miss", "1000000", "-completion", "1000000")
|
||||
if strings.TrimSpace(got) != "3 CNY" {
|
||||
t.Fatalf("cost output = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArmNamesTheConfiguration(t *testing.T) {
|
||||
if got := strings.TrimSpace(runOut(t, "arm")); got != "full" {
|
||||
t.Fatalf("control arm = %q", got)
|
||||
}
|
||||
if got := strings.TrimSpace(runOut(t, "arm", "-ablate", "planner,evidence", "-fold-index", "half")); got != "no-evidence+no-planner+index-half" {
|
||||
t.Fatalf("arm = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnknownSubcommandIsAUsageError(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
if err := run([]string{"nope"}, &out); !errors.Is(err, errUsage) {
|
||||
t.Fatalf("err = %v, want errUsage", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTariffReadsTheVendorsSchedule(t *testing.T) {
|
||||
got := runOut(t, "tariff", "-provider", "mimo", "-model", "mimo-v2.5")
|
||||
if !strings.Contains(got, "official card: true") || !strings.Contains(got, "one rate around the clock") {
|
||||
t.Fatalf("tariff output = %q", got)
|
||||
}
|
||||
peak := runOut(t, "tariff", "-currency", "CNY", "-at", "2026-10-12T02:00:00Z")
|
||||
if !strings.Contains(peak, "peak=true until 2026-10-12T04:00:00Z") {
|
||||
t.Fatalf("deepseek tariff at a weekday peak = %q", peak)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"time"
|
||||
|
||||
"reasonix/internal/contract/pricing"
|
||||
)
|
||||
|
||||
func runTariff(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("tariff", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
provider := fs.String("provider", "deepseek", "vendor")
|
||||
model := fs.String("model", "deepseek-flash", "model id")
|
||||
currency := fs.String("currency", "", "currency; empty reads the vendor's default")
|
||||
at := fs.String("at", "", "instant to read the schedule at, RFC 3339; empty is now")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
when := time.Now()
|
||||
if *at != "" {
|
||||
t, err := time.Parse(time.RFC3339, *at)
|
||||
if err != nil {
|
||||
return fmt.Errorf("-at: %w", err)
|
||||
}
|
||||
when = t
|
||||
}
|
||||
cur := *currency
|
||||
if cur == "" {
|
||||
cur = pricing.DefaultCurrency(*provider)
|
||||
}
|
||||
card := pricing.CurrentRate(*provider, *model, cur)
|
||||
if card == nil {
|
||||
return fmt.Errorf("no official rate for %s/%s in %s", *provider, *model, cur)
|
||||
}
|
||||
fmt.Fprintf(out, "official card: %v\n", pricing.IsOfficialCard(*provider, *model, *card))
|
||||
tariff, ok := pricing.LiveTariff(*provider, *model, *card, when)
|
||||
switch {
|
||||
case !ok:
|
||||
fmt.Fprintln(out, "schedule: one rate around the clock")
|
||||
case !tariff.Changes:
|
||||
fmt.Fprintf(out, "schedule: peak=%v, no change ahead\n", tariff.Peak)
|
||||
default:
|
||||
fmt.Fprintf(out, "schedule: peak=%v until %s\n", tariff.Peak, tariff.ChangesAt.UTC().Format(time.RFC3339))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
module reasonix
|
||||
|
||||
go 1.26.0
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
//go:build !windows
|
||||
|
||||
package neterr
|
||||
|
||||
import "syscall"
|
||||
|
||||
var resetErrors = []error{syscall.ECONNRESET, syscall.ECONNABORTED, syscall.EPIPE}
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
package neterr
|
||||
|
||||
import "syscall"
|
||||
|
||||
// syscall.ECONNRESET and friends exist on Windows only as APPLICATION_ERROR
|
||||
// placeholders, so errors.Is against them never matches a real socket failure.
|
||||
// Winsock's own numbering carries the identities the runtime returns.
|
||||
var resetErrors = []error{syscall.WSAECONNRESET, syscall.WSAECONNABORTED, syscall.ERROR_BROKEN_PIPE}
|
||||
@@ -0,0 +1,47 @@
|
||||
// Package neterr answers what a transport error is from the identity the
|
||||
// operating system gave it, never from the sentence it rendered. A system
|
||||
// error's text arrives in the machine's own language and is reworded between
|
||||
// releases, so a word list that matches on an English host silently stops
|
||||
// matching elsewhere. The judgement lives here once because three packages
|
||||
// were each carrying their own copy of it.
|
||||
package neterr
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
)
|
||||
|
||||
// IsConnReset reports whether err is a connection-level drop (peer reset,
|
||||
// truncated body, closed socket) as opposed to a protocol or caller error. A
|
||||
// stream cut this way mid-body can be replayed from scratch, unlike a decode
|
||||
// or 4xx error. The common trigger is a local proxy (v2rayN/sing-box)
|
||||
// idle-closing a long-lived SSE connection during a reasoner's first-token gap.
|
||||
func IsConnReset(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||
return false
|
||||
}
|
||||
if errors.Is(err, io.ErrUnexpectedEOF) || errors.Is(err, io.EOF) ||
|
||||
errors.Is(err, net.ErrClosed) || isAny(err, resetErrors) {
|
||||
return true
|
||||
}
|
||||
var dnsErr *net.DNSError
|
||||
if errors.As(err, &dnsErr) {
|
||||
return false
|
||||
}
|
||||
var netErr net.Error
|
||||
return errors.As(err, &netErr)
|
||||
}
|
||||
|
||||
func isAny(err error, targets []error) bool {
|
||||
for _, target := range targets {
|
||||
if errors.Is(err, target) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
package neterr
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestIsConnReset(t *testing.T) {
|
||||
if IsConnReset(nil) {
|
||||
t.Error("nil is not a conn reset")
|
||||
}
|
||||
if IsConnReset(context.Canceled) || IsConnReset(context.DeadlineExceeded) {
|
||||
t.Error("ctx cancel/deadline must not look like a recoverable reset")
|
||||
}
|
||||
if IsConnReset(errors.New("decode stream: invalid character")) {
|
||||
t.Error("a plain protocol error must not be treated as a conn reset")
|
||||
}
|
||||
for _, err := range []error{
|
||||
io.ErrUnexpectedEOF,
|
||||
&net.OpError{Op: "read", Err: resetErrors[0]},
|
||||
fmt.Errorf("read stream: %w", &net.OpError{Op: "read", Err: errors.New("wsarecv: forcibly closed")}),
|
||||
} {
|
||||
if !IsConnReset(err) {
|
||||
t.Errorf("want conn reset for %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsConnResetExcludesDNS(t *testing.T) {
|
||||
for _, dns := range []*net.DNSError{
|
||||
{Err: "no such host", Name: "api.deepseek.com", IsNotFound: true},
|
||||
{Err: "i/o timeout", Name: "api.deepseek.com", IsTimeout: true, IsTemporary: true},
|
||||
} {
|
||||
err := fmt.Errorf("dial: %w", &net.OpError{Op: "dial", Err: dns})
|
||||
if IsConnReset(err) {
|
||||
t.Errorf("DNS failure %v is not a connection reset", dns)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
package nilutil
|
||||
|
||||
import "reflect"
|
||||
|
||||
// IsNil reports whether v is nil or an interface holding a typed nil value.
|
||||
func IsNil(v any) bool {
|
||||
if v == nil {
|
||||
return true
|
||||
}
|
||||
rv := reflect.ValueOf(v)
|
||||
switch rv.Kind() {
|
||||
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
|
||||
return rv.IsNil()
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
package nilutil
|
||||
|
||||
import "testing"
|
||||
|
||||
type sampleInterface interface {
|
||||
Do()
|
||||
}
|
||||
|
||||
type sampleImpl struct{}
|
||||
|
||||
func (*sampleImpl) Do() {}
|
||||
|
||||
func TestIsNilDetectsTypedNilInterface(t *testing.T) {
|
||||
var p *sampleImpl
|
||||
var i sampleInterface = p
|
||||
|
||||
if !IsNil(i) {
|
||||
t.Fatal("IsNil should detect typed nil interface")
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsNilRejectsConcreteValues(t *testing.T) {
|
||||
var p sampleInterface = &sampleImpl{}
|
||||
|
||||
if IsNil(p) {
|
||||
t.Fatal("IsNil should not reject a concrete interface value")
|
||||
}
|
||||
if IsNil("x") {
|
||||
t.Fatal("IsNil should not reject non-nil scalar values")
|
||||
}
|
||||
}
|
||||
+218
@@ -0,0 +1,218 @@
|
||||
package retrieval
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"strings"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// Tokens lowercases Latin words and splits CJK runs into overlapping bigrams
|
||||
// (a lone CJK rune stays a single term) — the standard CJK indexing unit
|
||||
// (Lucene's CJKAnalyzer, SQLite FTS): a bigram only matches a real two-rune
|
||||
// subsequence where per-rune unigrams matched scattered common characters.
|
||||
// Intentionally a local, dependency-free approximation of FTS matching.
|
||||
func Tokens(s string) []string {
|
||||
var out []string
|
||||
var b strings.Builder
|
||||
var prev rune
|
||||
cjkRun := 0
|
||||
flush := func() {
|
||||
if b.Len() == 0 {
|
||||
return
|
||||
}
|
||||
out = append(out, b.String())
|
||||
b.Reset()
|
||||
}
|
||||
endCJK := func() {
|
||||
if cjkRun == 1 {
|
||||
out = append(out, string(prev))
|
||||
}
|
||||
cjkRun = 0
|
||||
}
|
||||
for _, r := range s {
|
||||
switch {
|
||||
case isCJK(r):
|
||||
flush()
|
||||
if cjkRun > 0 {
|
||||
out = append(out, string([]rune{prev, r}))
|
||||
}
|
||||
prev = r
|
||||
cjkRun++
|
||||
case unicode.IsLetter(r) || unicode.IsDigit(r) || r == '_':
|
||||
endCJK()
|
||||
b.WriteRune(unicode.ToLower(r))
|
||||
default:
|
||||
flush()
|
||||
endCJK()
|
||||
}
|
||||
}
|
||||
flush()
|
||||
endCJK()
|
||||
return out
|
||||
}
|
||||
|
||||
func isCJK(r rune) bool {
|
||||
return unicode.In(r, unicode.Han, unicode.Hiragana, unicode.Katakana, unicode.Hangul)
|
||||
}
|
||||
|
||||
// Unique returns terms in first-seen order.
|
||||
func Unique(in []string) []string {
|
||||
seen := map[string]bool{}
|
||||
out := make([]string, 0, len(in))
|
||||
for _, s := range in {
|
||||
if s == "" || seen[s] {
|
||||
continue
|
||||
}
|
||||
seen[s] = true
|
||||
out = append(out, s)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Counts returns a term-frequency map.
|
||||
func Counts(terms []string) map[string]int {
|
||||
counts := map[string]int{}
|
||||
for _, term := range terms {
|
||||
counts[term]++
|
||||
}
|
||||
return counts
|
||||
}
|
||||
|
||||
// BM25Score scores a document against query terms.
|
||||
func BM25Score(counts map[string]int, length int, queryTerms []string, df map[string]int, totalDocs int, avgLen float64) float64 {
|
||||
const (
|
||||
k1 = 1.2
|
||||
b = 0.75
|
||||
)
|
||||
if length <= 0 || totalDocs <= 0 {
|
||||
return 0
|
||||
}
|
||||
if avgLen <= 0 {
|
||||
avgLen = 1
|
||||
}
|
||||
var score float64
|
||||
docLen := float64(length)
|
||||
for _, term := range queryTerms {
|
||||
tf := counts[term]
|
||||
if tf == 0 {
|
||||
continue
|
||||
}
|
||||
termDF := df[term]
|
||||
if termDF == 0 {
|
||||
continue
|
||||
}
|
||||
idf := math.Log(1 + (float64(totalDocs)-float64(termDF)+0.5)/(float64(termDF)+0.5))
|
||||
freq := float64(tf)
|
||||
score += idf * (freq * (k1 + 1)) / (freq + k1*(1-b+b*docLen/avgLen))
|
||||
}
|
||||
return score
|
||||
}
|
||||
|
||||
// DocumentFrequency counts how many documents contain each term.
|
||||
func DocumentFrequency(docs []map[string]int) map[string]int {
|
||||
df := map[string]int{}
|
||||
for _, counts := range docs {
|
||||
for term := range counts {
|
||||
df[term]++
|
||||
}
|
||||
}
|
||||
return df
|
||||
}
|
||||
|
||||
// KeepTopRelativeScore keeps the best item and drops trailing items whose score
|
||||
// falls below ratio * topScore. Callers must pass items already sorted best
|
||||
// first. This mirrors SQLite FTS/BM25 search UIs that over-fetch, then trim
|
||||
// common-word-only noise without imposing an absolute score threshold.
|
||||
func KeepTopRelativeScore[T any](items []T, ratio float64, score func(T) float64) []T {
|
||||
if len(items) == 0 || ratio <= 0 {
|
||||
return items
|
||||
}
|
||||
top := score(items[0])
|
||||
if top <= 0 {
|
||||
return items
|
||||
}
|
||||
cutoff := top * ratio
|
||||
out := items[:0]
|
||||
for i, item := range items {
|
||||
if i == 0 || score(item) >= cutoff {
|
||||
out = append(out, item)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// QueryTerms normalizes a search string and reports an error when nothing
|
||||
// searchable remains.
|
||||
func QueryTerms(query string) ([]string, error) {
|
||||
terms := Unique(Tokens(strings.TrimSpace(query)))
|
||||
if len(terms) == 0 {
|
||||
return nil, fmt.Errorf("query must contain at least one letter or number")
|
||||
}
|
||||
return terms, nil
|
||||
}
|
||||
|
||||
// MakeSnippet returns a whitespace-compacted excerpt centered near the query.
|
||||
func MakeSnippet(text, query string, terms []string, maxRunes int) string {
|
||||
text = CompactWhitespace(text)
|
||||
if maxRunes <= 0 || utf8.RuneCountInString(text) <= maxRunes {
|
||||
return text
|
||||
}
|
||||
lower := strings.ToLower(text)
|
||||
query = strings.ToLower(strings.TrimSpace(query))
|
||||
idx := -1
|
||||
if query != "" {
|
||||
idx = strings.Index(lower, query)
|
||||
}
|
||||
if idx < 0 {
|
||||
for _, term := range terms {
|
||||
runes := []rune(term)
|
||||
if len(runes) == 1 && !isCJK(runes[0]) {
|
||||
continue
|
||||
}
|
||||
if i := strings.Index(lower, term); i >= 0 {
|
||||
idx = i
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if idx < 0 {
|
||||
idx = 0
|
||||
}
|
||||
return snippetAround(text, idx, maxRunes)
|
||||
}
|
||||
|
||||
func snippetAround(text string, byteIdx, maxRunes int) string {
|
||||
if byteIdx < 0 {
|
||||
byteIdx = 0
|
||||
}
|
||||
if byteIdx > len(text) {
|
||||
byteIdx = len(text)
|
||||
}
|
||||
for byteIdx > 0 && byteIdx < len(text) && !utf8.RuneStart(text[byteIdx]) {
|
||||
byteIdx--
|
||||
}
|
||||
runes := []rune(text)
|
||||
pos := utf8.RuneCountInString(text[:byteIdx])
|
||||
start := max(pos-maxRunes/2, 0)
|
||||
end := start + maxRunes
|
||||
if end > len(runes) {
|
||||
end = len(runes)
|
||||
start = max(end-maxRunes, 0)
|
||||
}
|
||||
prefix := ""
|
||||
suffix := ""
|
||||
if start > 0 {
|
||||
prefix = "..."
|
||||
}
|
||||
if end < len(runes) {
|
||||
suffix = "..."
|
||||
}
|
||||
return prefix + string(runes[start:end]) + suffix
|
||||
}
|
||||
|
||||
// CompactWhitespace collapses runs of whitespace into one ASCII space.
|
||||
func CompactWhitespace(s string) string {
|
||||
return strings.Join(strings.Fields(s), " ")
|
||||
}
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
package retrieval
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
func TestTokensHandlesLatinAndCJK(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
in string
|
||||
want []string
|
||||
}{
|
||||
{"BM25 检索 cache-first", []string{"bm25", "检索", "cache", "first"}},
|
||||
{"数据库迁移", []string{"数据", "据库", "库迁", "迁移"}},
|
||||
{"库", []string{"库"}},
|
||||
{"用pnpm装依赖", []string{"用", "pnpm", "装依", "依赖"}},
|
||||
} {
|
||||
got := Tokens(tc.in)
|
||||
if strings.Join(got, ",") != strings.Join(tc.want, ",") {
|
||||
t.Fatalf("Tokens(%q) = %#v, want %#v", tc.in, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBM25ScoreRanksMatchingDocument(t *testing.T) {
|
||||
query := Unique(Tokens("prompt cache"))
|
||||
doc1 := Counts(Tokens("prompt cache cache stability"))
|
||||
doc2 := Counts(Tokens("dashboard colors"))
|
||||
df := DocumentFrequency([]map[string]int{doc1, doc2})
|
||||
score1 := BM25Score(doc1, 4, query, df, 2, 3)
|
||||
score2 := BM25Score(doc2, 2, query, df, 2, 3)
|
||||
if score1 <= score2 {
|
||||
t.Fatalf("matching score %.3f should exceed unrelated score %.3f", score1, score2)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKeepTopRelativeScoreKeepsTopAndDropsWeakTail(t *testing.T) {
|
||||
items := []struct {
|
||||
name string
|
||||
score float64
|
||||
}{
|
||||
{name: "top", score: 10},
|
||||
{name: "near", score: 2},
|
||||
{name: "noise", score: 1.4},
|
||||
{name: "zero", score: 0},
|
||||
}
|
||||
got := KeepTopRelativeScore(items, 0.15, func(item struct {
|
||||
name string
|
||||
score float64
|
||||
}) float64 {
|
||||
return item.score
|
||||
})
|
||||
if len(got) != 2 || got[0].name != "top" || got[1].name != "near" {
|
||||
t.Fatalf("KeepTopRelativeScore() = %#v, want top and near", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMakeSnippetHandlesMultibyteBoundary(t *testing.T) {
|
||||
text := strings.Repeat("前缀", 80) + "稳定结论 synthesis cache " + strings.Repeat("后缀", 80)
|
||||
out := MakeSnippet(text, "synthesis cache", QueryTermsForTest(t, "synthesis cache"), 60)
|
||||
if !strings.Contains(out, "synthesis cache") {
|
||||
t.Fatalf("snippet missing query: %q", out)
|
||||
}
|
||||
if strings.ContainsRune(out, utf8.RuneError) {
|
||||
t.Fatalf("snippet contains replacement rune: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func QueryTermsForTest(t *testing.T, query string) []string {
|
||||
t.Helper()
|
||||
terms, err := QueryTerms(query)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return terms
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
// Retrieval V2, shadow-only: BM25F field weighting, code-symbol splitting,
|
||||
// and mixed CJK uni+bigrams. Nothing here serves the model — V2 rankings ride
|
||||
// the recall audit next to production's, and MemoryBench decides if V2 ever
|
||||
// takes over. Weights are candidates under measurement, not tuned truths.
|
||||
package retrieval
|
||||
|
||||
import (
|
||||
"math"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
// TokensV2 extends Tokens with code-symbol structure and CJK unigrams:
|
||||
// CamelCase and snake_case words also emit their segments (FooBar → foobar,
|
||||
// foo, bar), and CJK runs emit unigrams alongside bigrams so one-character
|
||||
// queries still match. The V1 form of every token is preserved, so V2 recall
|
||||
// is a superset of V1's vocabulary.
|
||||
func TokensV2(s string) []string {
|
||||
var out []string
|
||||
var word []rune
|
||||
var prev rune
|
||||
cjkRun := 0
|
||||
flushWord := func() {
|
||||
if len(word) == 0 {
|
||||
return
|
||||
}
|
||||
lower := strings.ToLower(string(word))
|
||||
out = append(out, lower)
|
||||
for _, seg := range splitCodeSymbol(word) {
|
||||
if seg != lower {
|
||||
out = append(out, seg)
|
||||
}
|
||||
}
|
||||
word = word[:0]
|
||||
}
|
||||
endCJK := func() { cjkRun = 0 }
|
||||
for _, r := range s {
|
||||
switch {
|
||||
case isCJK(r):
|
||||
flushWord()
|
||||
out = append(out, string(r))
|
||||
if cjkRun > 0 {
|
||||
out = append(out, string([]rune{prev, r}))
|
||||
}
|
||||
prev = r
|
||||
cjkRun++
|
||||
case unicode.IsLetter(r) || unicode.IsDigit(r) || r == '_':
|
||||
endCJK()
|
||||
word = append(word, r)
|
||||
default:
|
||||
flushWord()
|
||||
endCJK()
|
||||
}
|
||||
}
|
||||
flushWord()
|
||||
return out
|
||||
}
|
||||
|
||||
// splitCodeSymbol yields the lowercase segments of a code identifier:
|
||||
// CamelCase humps, snake_case parts, and letter/digit boundaries.
|
||||
func splitCodeSymbol(word []rune) []string {
|
||||
var segments []string
|
||||
var seg []rune
|
||||
flush := func() {
|
||||
if len(seg) > 1 { // single letters are noise, not signal
|
||||
segments = append(segments, strings.ToLower(string(seg)))
|
||||
}
|
||||
seg = seg[:0]
|
||||
}
|
||||
for i, r := range word {
|
||||
boundary := r == '_' ||
|
||||
(i > 0 && unicode.IsUpper(r) && !unicode.IsUpper(word[i-1])) ||
|
||||
(i > 0 && unicode.IsDigit(r) != unicode.IsDigit(word[i-1]))
|
||||
if boundary {
|
||||
flush()
|
||||
}
|
||||
if r != '_' {
|
||||
seg = append(seg, r)
|
||||
}
|
||||
}
|
||||
flush()
|
||||
if len(segments) < 2 {
|
||||
return nil // no structure worth indexing beyond the whole word
|
||||
}
|
||||
return segments
|
||||
}
|
||||
|
||||
// FieldedDoc is one document split into weighted fields for BM25F.
|
||||
type FieldedDoc struct {
|
||||
ID string
|
||||
Fields map[string]string
|
||||
}
|
||||
|
||||
// v2FieldWeights follow the BM25F intuition: identity fields say what a fact
|
||||
// IS, the body says everything it mentions. Candidates under measurement.
|
||||
var v2FieldWeights = map[string]float64{
|
||||
"name": 2.5, "title": 2.5, "keywords": 2.0, "subject": 2.0,
|
||||
"description": 1.5, "body": 1.0,
|
||||
}
|
||||
|
||||
// V2Hit is one shadow-ranked document.
|
||||
type V2Hit struct {
|
||||
ID string
|
||||
Score float64
|
||||
}
|
||||
|
||||
// RankV2 scores docs against a query with BM25F over TokensV2: term
|
||||
// frequencies accumulate per field scaled by field weight, then the standard
|
||||
// saturation applies once per term (the F in BM25F), with length
|
||||
// normalization over the weighted document length.
|
||||
func RankV2(query string, docs []FieldedDoc) []V2Hit {
|
||||
queryTerms := Unique(TokensV2(query))
|
||||
if len(queryTerms) == 0 || len(docs) == 0 {
|
||||
return nil
|
||||
}
|
||||
type indexed struct {
|
||||
id string
|
||||
tf map[string]float64
|
||||
length float64
|
||||
}
|
||||
corpus := make([]indexed, 0, len(docs))
|
||||
df := map[string]int{}
|
||||
var totalLen float64
|
||||
for _, doc := range docs {
|
||||
ix := indexed{id: doc.ID, tf: map[string]float64{}}
|
||||
seen := map[string]bool{}
|
||||
for field, text := range doc.Fields {
|
||||
weight, ok := v2FieldWeights[field]
|
||||
if !ok {
|
||||
weight = 1.0
|
||||
}
|
||||
for _, term := range TokensV2(text) {
|
||||
ix.tf[term] += weight
|
||||
ix.length += weight
|
||||
seen[term] = true
|
||||
}
|
||||
}
|
||||
for term := range seen {
|
||||
df[term]++
|
||||
}
|
||||
totalLen += ix.length
|
||||
corpus = append(corpus, ix)
|
||||
}
|
||||
avgLen := totalLen / float64(len(corpus))
|
||||
if avgLen <= 0 {
|
||||
avgLen = 1
|
||||
}
|
||||
const k1, b = 1.2, 0.75
|
||||
var hits []V2Hit
|
||||
for _, doc := range corpus {
|
||||
var score float64
|
||||
for _, term := range queryTerms {
|
||||
tf := doc.tf[term]
|
||||
if tf == 0 || df[term] == 0 {
|
||||
continue
|
||||
}
|
||||
idf := math.Log(1 + (float64(len(corpus))-float64(df[term])+0.5)/(float64(df[term])+0.5))
|
||||
score += idf * (tf * (k1 + 1)) / (tf + k1*(1-b+b*doc.length/avgLen))
|
||||
}
|
||||
if score > 0 {
|
||||
hits = append(hits, V2Hit{ID: doc.id, Score: score})
|
||||
}
|
||||
}
|
||||
SortHitsDesc(hits)
|
||||
return hits
|
||||
}
|
||||
|
||||
// SortHitsDesc orders shadow hits best-first with a stable ID tiebreak.
|
||||
func SortHitsDesc(hits []V2Hit) {
|
||||
for i := 1; i < len(hits); i++ {
|
||||
for j := i; j > 0 && (hits[j].Score > hits[j-1].Score ||
|
||||
(hits[j].Score == hits[j-1].Score && hits[j].ID < hits[j-1].ID)); j-- {
|
||||
hits[j], hits[j-1] = hits[j-1], hits[j]
|
||||
}
|
||||
}
|
||||
}
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
package retrieval
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTokensV2CodeSymbolsAndCJK(t *testing.T) {
|
||||
got := strings.Join(TokensV2("RetryPolicy.MaxBackoffMs ticket_1827 数据库"), " ")
|
||||
for _, want := range []string{"retrypolicy", "retry", "policy", "maxbackoffms", "max", "backoff", "ticket_1827", "ticket", "1827", "数", "据", "库", "数据", "据库"} {
|
||||
if !strings.Contains(" "+got+" ", " "+want+" ") {
|
||||
t.Fatalf("TokensV2 missing %q in %q", want, got)
|
||||
}
|
||||
}
|
||||
if strings.Contains(" "+got+" ", " ms ") {
|
||||
// "Ms" splits to a single meaningful segment only when len > 1; "ms" is
|
||||
// fine — assert single letters never appear instead.
|
||||
t.Log("ms segment present (acceptable)")
|
||||
}
|
||||
for _, tok := range TokensV2("A B C") {
|
||||
if len(tok) == 0 {
|
||||
t.Fatal("empty token emitted")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRankV2FieldWeightsBeatBodyMentions(t *testing.T) {
|
||||
docs := []FieldedDoc{
|
||||
{ID: "titled", Fields: map[string]string{"title": "Package manager", "body": "We migrated to pnpm."}},
|
||||
{ID: "mention", Fields: map[string]string{"title": "Sprint notes", "body": "Someone asked about the package manager once; also weather, lunch, and a package of stickers."}},
|
||||
}
|
||||
hits := RankV2("package manager", docs)
|
||||
if len(hits) != 2 || hits[0].ID != "titled" {
|
||||
t.Fatalf("field-weighted doc must outrank body mentions: %+v", hits)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRankV2FindsCodeSymbolBySegment(t *testing.T) {
|
||||
docs := []FieldedDoc{
|
||||
{ID: "sym", Fields: map[string]string{"body": "Configure RetryPolicy.MaxBackoffMs in policy/retry.toml."}},
|
||||
{ID: "other", Fields: map[string]string{"body": "Unrelated fact about deploys."}},
|
||||
}
|
||||
hits := RankV2("max backoff configuration", docs)
|
||||
if len(hits) == 0 || hits[0].ID != "sym" {
|
||||
t.Fatalf("segment query must find the camel-case symbol: %+v", hits)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
// Package rulelist splits a list of permission rules written on one line.
|
||||
//
|
||||
// Rules are separated by commas or whitespace, and a separator inside
|
||||
// parentheses belongs to the rule: `Bash(git add:*) Read` is two rules. Skill
|
||||
// front matter and the --allowed-tools flag share it, so one spelling means
|
||||
// one thing in both.
|
||||
package rulelist
|
||||
+56
@@ -0,0 +1,56 @@
|
||||
package rulelist
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
var (
|
||||
// ErrUnclosed: an opening parenthesis is never closed.
|
||||
ErrUnclosed = errors.New("unclosed '('")
|
||||
// ErrUnexpectedClose: a closing parenthesis has no opening one.
|
||||
ErrUnexpectedClose = errors.New("unexpected ')'")
|
||||
)
|
||||
|
||||
// Split returns the rules in value, in order, trimmed and non-empty.
|
||||
func Split(value string) ([]string, error) {
|
||||
var rules []string
|
||||
start, depth := -1, 0
|
||||
flush := func(end int) {
|
||||
if start < 0 {
|
||||
return
|
||||
}
|
||||
if rule := strings.TrimSpace(value[start:end]); rule != "" {
|
||||
rules = append(rules, rule)
|
||||
}
|
||||
start = -1
|
||||
}
|
||||
for i, r := range value {
|
||||
switch r {
|
||||
case '(':
|
||||
if start < 0 {
|
||||
start = i
|
||||
}
|
||||
depth++
|
||||
case ')':
|
||||
if depth == 0 {
|
||||
return nil, ErrUnexpectedClose
|
||||
}
|
||||
depth--
|
||||
default:
|
||||
if depth == 0 && (r == ',' || unicode.IsSpace(r)) {
|
||||
flush(i)
|
||||
continue
|
||||
}
|
||||
if start < 0 {
|
||||
start = i
|
||||
}
|
||||
}
|
||||
}
|
||||
if depth != 0 {
|
||||
return nil, ErrUnclosed
|
||||
}
|
||||
flush(len(value))
|
||||
return rules, nil
|
||||
}
|
||||
+40
@@ -0,0 +1,40 @@
|
||||
package rulelist
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSplitSeparatesOnCommasAndWhitespaceOutsideParentheses(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
in string
|
||||
want []string
|
||||
}{
|
||||
{"", nil},
|
||||
{" ", nil},
|
||||
{"Read", []string{"Read"}},
|
||||
{"read_file, grep,bash", []string{"read_file", "grep", "bash"}},
|
||||
{"Read Grep Bash", []string{"Read", "Grep", "Bash"}},
|
||||
{"Bash(git add:*) Read", []string{"Bash(git add:*)", "Read"}},
|
||||
{"Bash(git add:*, git commit:*), Read", []string{"Bash(git add:*, git commit:*)", "Read"}},
|
||||
{"bash,", []string{"bash"}},
|
||||
} {
|
||||
got, err := Split(tc.in)
|
||||
if err != nil {
|
||||
t.Fatalf("Split(%q): %v", tc.in, err)
|
||||
}
|
||||
if !reflect.DeepEqual(got, tc.want) {
|
||||
t.Errorf("Split(%q) = %q, want %q", tc.in, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSplitReportsUnbalancedParenthesesByIdentity(t *testing.T) {
|
||||
if _, err := Split("Bash(git add"); !errors.Is(err, ErrUnclosed) {
|
||||
t.Errorf("unclosed = %v", err)
|
||||
}
|
||||
if _, err := Split("Bash)"); !errors.Is(err, ErrUnexpectedClose) {
|
||||
t.Errorf("unexpected close = %v", err)
|
||||
}
|
||||
}
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
// Package tokencount estimates how many tokens a string costs a model, without
|
||||
// carrying a tokenizer. Anything that has to size text before a provider counts
|
||||
// it — a cache-shape diagnostic, a tool call's context footprint — needs one
|
||||
// answer to that question rather than a private heuristic per caller.
|
||||
package tokencount
|
||||
|
||||
import "unicode/utf8"
|
||||
|
||||
// Text estimates the token count of s. The split is by script because the two
|
||||
// behave nothing alike — Latin text and code trend near four bytes per token,
|
||||
// CJK closer to one token per character — and either rule applied to the whole
|
||||
// string is wrong by ~4x on the other half. A real tokenizer would be exact and
|
||||
// would mean a vocabulary per model, re-run over whole transcripts every turn.
|
||||
func Text(s string) int {
|
||||
narrow, wide := 0, 0
|
||||
for _, r := range s {
|
||||
if r < utf8.RuneSelf {
|
||||
narrow++
|
||||
continue
|
||||
}
|
||||
wide++
|
||||
}
|
||||
// Round up: a long run of short strings must not sum to nothing.
|
||||
return (narrow+3)/4 + wide
|
||||
}
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
package tokencount
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestTextDoesNotUndercountCJK(t *testing.T) {
|
||||
// The reason this package exists: bytes/4 reads a Chinese transcript as a
|
||||
// third of its real size, so a planner sizing context off it compacts late.
|
||||
const cjk = "把这个仓库跑一遍测试，把失败的那几个定位到具体文件"
|
||||
runes := len([]rune(cjk))
|
||||
if got := Text(cjk); got != runes {
|
||||
t.Fatalf("Text(cjk) = %d, want one token per rune (%d)", got, runes)
|
||||
}
|
||||
if byBytes := len(cjk) / 4; Text(cjk) <= byBytes {
|
||||
t.Fatalf("Text(cjk) = %d, must exceed the bytes/4 reading (%d)", Text(cjk), byBytes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTextRoundsASCIIUpAndKeepsEmptyFree(t *testing.T) {
|
||||
if got := Text(""); got != 0 {
|
||||
t.Fatalf("Text(\"\") = %d, want 0", got)
|
||||
}
|
||||
// Five bytes is more than one token's worth; truncating would let a long
|
||||
// run of short strings sum to nothing.
|
||||
if got := Text("abcde"); got != 2 {
|
||||
t.Fatalf("Text(\"abcde\") = %d, want 2", got)
|
||||
}
|
||||
}
|
||||
+200
@@ -0,0 +1,200 @@
|
||||
// Package ablation switches individual Reasonix subsystems off so a benchmark
|
||||
// can attribute a change in solve rate to one of them.
|
||||
package ablation
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type Module string
|
||||
|
||||
const (
|
||||
Evidence Module = "evidence"
|
||||
Planner Module = "planner"
|
||||
Subagent Module = "subagent"
|
||||
Retrieval Module = "retrieval"
|
||||
Compaction Module = "compaction"
|
||||
// Upstream off means a fleet dependency edge orders its endpoints without
|
||||
// delivering the dependency's answer, so a dependent must re-derive it.
|
||||
Upstream Module = "upstream"
|
||||
// RecallSearch off leaves recall able to read an address it was given and
|
||||
// unable to look one up, which is what the tool could do before the folded
|
||||
// region became searchable.
|
||||
RecallSearch Module = "recall-search"
|
||||
)
|
||||
|
||||
// FoldIndexScale sizes the model-visible fold index against its shipped budget.
|
||||
// It is a bench axis, not a module: the question it answers is what an ambient
|
||||
// address list is worth, and four steps answer that where a free-form fraction
|
||||
// would turn a measurement into a tuning knob.
|
||||
type FoldIndexScale string
|
||||
|
||||
const (
|
||||
FoldIndexDefault FoldIndexScale = ""
|
||||
FoldIndexHalf FoldIndexScale = "half"
|
||||
FoldIndexQuarter FoldIndexScale = "quarter"
|
||||
FoldIndexOff FoldIndexScale = "off"
|
||||
)
|
||||
|
||||
// Ratio scales a budget. The default returns it unchanged.
|
||||
func (f FoldIndexScale) Ratio() float64 {
|
||||
switch f {
|
||||
case FoldIndexHalf:
|
||||
return 0.5
|
||||
case FoldIndexQuarter:
|
||||
return 0.25
|
||||
case FoldIndexOff:
|
||||
return 0
|
||||
default:
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
// ParseFoldIndexScale reads a bench flag value. "" and "default" ship as-is.
|
||||
func ParseFoldIndexScale(spec string) (FoldIndexScale, error) {
|
||||
switch strings.ToLower(strings.TrimSpace(spec)) {
|
||||
case "", "default", "full":
|
||||
return FoldIndexDefault, nil
|
||||
case string(FoldIndexHalf):
|
||||
return FoldIndexHalf, nil
|
||||
case string(FoldIndexQuarter):
|
||||
return FoldIndexQuarter, nil
|
||||
case string(FoldIndexOff), "none", "0":
|
||||
return FoldIndexOff, nil
|
||||
default:
|
||||
return "", fmt.Errorf("unknown fold-index scale %q (want default, half, quarter, or off)", spec)
|
||||
}
|
||||
}
|
||||
|
||||
// Modules returns every switchable module in the order arm names use.
|
||||
func Modules() []Module {
|
||||
return []Module{Evidence, Planner, Subagent, Retrieval, Compaction, Upstream, RecallSearch}
|
||||
}
|
||||
|
||||
// Set is the group of modules disabled for a run. The zero value is the
|
||||
// control arm: everything on.
|
||||
type Set struct {
|
||||
off map[Module]bool
|
||||
// foldIndex is an axis rather than a switch, so it lives beside the module
|
||||
// set instead of inside it. It still reaches Arm(): a run whose index was
|
||||
// quartered must not report the same identity as the control.
|
||||
foldIndex FoldIndexScale
|
||||
}
|
||||
|
||||
// WithFoldIndex returns this set at the given index scale.
|
||||
func (s Set) WithFoldIndex(scale FoldIndexScale) Set {
|
||||
s.foldIndex = scale
|
||||
return s
|
||||
}
|
||||
|
||||
// FoldIndex is the scale this run sizes its fold index at.
|
||||
func (s Set) FoldIndex() FoldIndexScale { return s.foldIndex }
|
||||
|
||||
// Parse reads a spec such as "evidence,planner". "" and "none" mean the control
|
||||
// arm; "all" disables every module.
|
||||
func Parse(spec string) (Set, error) {
|
||||
spec = strings.TrimSpace(spec)
|
||||
if spec == "" || strings.EqualFold(spec, "none") {
|
||||
return Set{}, nil
|
||||
}
|
||||
if strings.EqualFold(spec, "all") {
|
||||
return New(Modules()...), nil
|
||||
}
|
||||
known := map[Module]bool{}
|
||||
for _, m := range Modules() {
|
||||
known[m] = true
|
||||
}
|
||||
var mods []Module
|
||||
for _, field := range strings.FieldsFunc(spec, func(r rune) bool { return r == ',' || r == ' ' }) {
|
||||
m := Module(strings.ToLower(strings.TrimSpace(field)))
|
||||
if !known[m] {
|
||||
return Set{}, fmt.Errorf("unknown ablation module %q (want %s, or none/all)", field, ModuleList())
|
||||
}
|
||||
mods = append(mods, m)
|
||||
}
|
||||
return New(mods...), nil
|
||||
}
|
||||
|
||||
// ParseArm reads both bench axes at once: the module spec and the fold-index
|
||||
// scale. One call so a frontend cannot wire the first and forget the second,
|
||||
// which would silently run an index arm under the control arm's name.
|
||||
func ParseArm(modules, foldIndex string) (Set, error) {
|
||||
set, err := Parse(modules)
|
||||
if err != nil {
|
||||
return Set{}, err
|
||||
}
|
||||
scale, err := ParseFoldIndexScale(foldIndex)
|
||||
if err != nil {
|
||||
return Set{}, err
|
||||
}
|
||||
return set.WithFoldIndex(scale), nil
|
||||
}
|
||||
|
||||
// ModuleList names every switchable module for flag help and error text, so
|
||||
// those cannot drift from Modules() the way a hand-kept list does.
|
||||
func ModuleList() string { return joinModules(Modules(), ", ") }
|
||||
|
||||
// New returns a Set with the given modules disabled.
|
||||
func New(mods ...Module) Set {
|
||||
if len(mods) == 0 {
|
||||
return Set{}
|
||||
}
|
||||
off := make(map[Module]bool, len(mods))
|
||||
for _, m := range mods {
|
||||
off[m] = true
|
||||
}
|
||||
return Set{off: off}
|
||||
}
|
||||
|
||||
func (s Set) Off(m Module) bool { return s.off[m] }
|
||||
|
||||
func (s Set) Empty() bool { return len(s.off) == 0 && s.foldIndex == FoldIndexDefault }
|
||||
|
||||
// Arm is the published name of this configuration: "full" for the control arm,
|
||||
// otherwise "no-evidence+no-planner". Stable across runs so results from
|
||||
// different machines group by the same key.
|
||||
func (s Set) Arm() string {
|
||||
if s.Empty() {
|
||||
return "full"
|
||||
}
|
||||
parts := make([]string, 0, len(s.off)+1)
|
||||
for _, m := range s.disabled() {
|
||||
parts = append(parts, "no-"+string(m))
|
||||
}
|
||||
if s.foldIndex != FoldIndexDefault {
|
||||
parts = append(parts, "index-"+string(s.foldIndex))
|
||||
}
|
||||
return strings.Join(parts, "+")
|
||||
}
|
||||
|
||||
// String round-trips back through Parse. It carries modules only — the fold
|
||||
// index axis has its own flag and its own parser.
|
||||
func (s Set) String() string {
|
||||
if s.Empty() {
|
||||
return "none"
|
||||
}
|
||||
return joinModules(s.disabled(), ",")
|
||||
}
|
||||
|
||||
func (s Set) disabled() []Module {
|
||||
order := map[Module]int{}
|
||||
for i, m := range Modules() {
|
||||
order[m] = i
|
||||
}
|
||||
out := make([]Module, 0, len(s.off))
|
||||
for m := range s.off {
|
||||
out = append(out, m)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return order[out[i]] < order[out[j]] })
|
||||
return out
|
||||
}
|
||||
|
||||
func joinModules(mods []Module, sep string) string {
|
||||
parts := make([]string, len(mods))
|
||||
for i, m := range mods {
|
||||
parts[i] = string(m)
|
||||
}
|
||||
return strings.Join(parts, sep)
|
||||
}
|
||||
+108
@@ -0,0 +1,108 @@
|
||||
package ablation
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestParseAcceptsSpecFormsAndRejectsUnknownModules(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
spec string
|
||||
arm string
|
||||
}{
|
||||
{"", "full"},
|
||||
{"none", "full"},
|
||||
{"NONE", "full"},
|
||||
{"evidence", "no-evidence"},
|
||||
{"planner,evidence", "no-evidence+no-planner"},
|
||||
{"planner evidence", "no-evidence+no-planner"},
|
||||
{" Evidence , Retrieval ", "no-evidence+no-retrieval"},
|
||||
{"all", "no-evidence+no-planner+no-subagent+no-retrieval+no-compaction+no-upstream+no-recall-search"},
|
||||
} {
|
||||
set, err := Parse(tc.spec)
|
||||
if err != nil {
|
||||
t.Fatalf("Parse(%q): %v", tc.spec, err)
|
||||
}
|
||||
if got := set.Arm(); got != tc.arm {
|
||||
t.Errorf("Parse(%q).Arm() = %q, want %q", tc.spec, got, tc.arm)
|
||||
}
|
||||
}
|
||||
|
||||
if _, err := Parse("evidence,planer"); err == nil {
|
||||
t.Fatal("a misspelled module must fail loudly, not silently run the control arm")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArmNameIsIndependentOfSpecOrder(t *testing.T) {
|
||||
a, _ := Parse("retrieval,evidence,planner")
|
||||
b, _ := Parse("planner,retrieval,evidence")
|
||||
if a.Arm() != b.Arm() {
|
||||
t.Fatalf("arm names diverge by input order: %q vs %q", a.Arm(), b.Arm())
|
||||
}
|
||||
}
|
||||
|
||||
func TestStringRoundTripsThroughParse(t *testing.T) {
|
||||
for _, spec := range []string{"none", "evidence", "evidence,compaction", "all"} {
|
||||
set, err := Parse(spec)
|
||||
if err != nil {
|
||||
t.Fatalf("Parse(%q): %v", spec, err)
|
||||
}
|
||||
again, err := Parse(set.String())
|
||||
if err != nil {
|
||||
t.Fatalf("Parse(%q): %v", set.String(), err)
|
||||
}
|
||||
if again.Arm() != set.Arm() {
|
||||
t.Errorf("%q round-tripped to %q", set.Arm(), again.Arm())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The index axis is not a module, but a run that moved it is not the control
|
||||
// arm either: results keyed by arm name would otherwise merge with it.
|
||||
func TestFoldIndexScaleNamesItsOwnArm(t *testing.T) {
|
||||
var s Set
|
||||
if got := s.WithFoldIndex(FoldIndexQuarter).Arm(); got != "index-quarter" {
|
||||
t.Errorf("Arm() = %q, want index-quarter", got)
|
||||
}
|
||||
if got := New(Evidence).WithFoldIndex(FoldIndexOff).Arm(); got != "no-evidence+index-off" {
|
||||
t.Errorf("Arm() = %q, want no-evidence+index-off", got)
|
||||
}
|
||||
if s.WithFoldIndex(FoldIndexQuarter).Empty() {
|
||||
t.Error("a quartered index reported itself as the control arm")
|
||||
}
|
||||
if !s.WithFoldIndex(FoldIndexDefault).Empty() {
|
||||
t.Error("the default scale is the control arm")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFoldIndexScaleRatios(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
spec string
|
||||
scale FoldIndexScale
|
||||
ratio float64
|
||||
}{
|
||||
{"", FoldIndexDefault, 1}, {"default", FoldIndexDefault, 1},
|
||||
{"half", FoldIndexHalf, 0.5}, {"QUARTER", FoldIndexQuarter, 0.25},
|
||||
{"off", FoldIndexOff, 0}, {"0", FoldIndexOff, 0},
|
||||
} {
|
||||
got, err := ParseFoldIndexScale(tc.spec)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseFoldIndexScale(%q): %v", tc.spec, err)
|
||||
}
|
||||
if got != tc.scale || got.Ratio() != tc.ratio {
|
||||
t.Errorf("ParseFoldIndexScale(%q) = %q (ratio %v), want %q (%v)", tc.spec, got, got.Ratio(), tc.scale, tc.ratio)
|
||||
}
|
||||
}
|
||||
if _, err := ParseFoldIndexScale("0.37"); err == nil {
|
||||
t.Fatal("a free-form fraction was accepted; the axis is four steps on purpose")
|
||||
}
|
||||
}
|
||||
|
||||
func TestZeroValueIsTheControlArm(t *testing.T) {
|
||||
var s Set
|
||||
if !s.Empty() || s.Arm() != "full" {
|
||||
t.Fatalf("zero Set = %q, want the full control arm", s.Arm())
|
||||
}
|
||||
for _, m := range Modules() {
|
||||
if s.Off(m) {
|
||||
t.Errorf("zero Set disabled %s", m)
|
||||
}
|
||||
}
|
||||
}
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
// Package agentgraph is the vocabulary of one run's execution graph: the nodes
|
||||
// a turn handed work to, and the typed edges between them.
|
||||
//
|
||||
// It exists because that structure was being re-derived in five places and
|
||||
// survived in none. A fleet's dependency DAG died with the call that built it,
|
||||
// the persisted sidecar recorded a parent link and nothing else, and the event
|
||||
// stream implied edges through id prefixes. A reader asking why a node waited
|
||||
// could not tell an unmet dependency from a delivered answer, because neither
|
||||
// had ever been written down as anything a reader could tell apart.
|
||||
//
|
||||
// Nothing here knows what an agent, a tool, or a provider is. A producer
|
||||
// publishes a Delta of what it has just proven; every consumer folds deltas
|
||||
// with the same Apply, so the graph a headless run reports is the graph the
|
||||
// desktop draws.
|
||||
package agentgraph
|
||||
+219
@@ -0,0 +1,219 @@
|
||||
package agentgraph
|
||||
|
||||
import "slices"
|
||||
|
||||
// NodeKind separates the three things a node can be. A group is the call that
|
||||
// owns a fan-out, a worker is one delegated run, and an external node is work
|
||||
// this run did not do — named only because an edge points at it.
|
||||
type NodeKind string
|
||||
|
||||
const (
|
||||
KindGroup NodeKind = "group"
|
||||
KindWorker NodeKind = "worker"
|
||||
KindExternal NodeKind = "external"
|
||||
)
|
||||
|
||||
// NodeState is where one node stands. It is the node's own lifecycle and not a
|
||||
// tool call's: an adopted node never ran and still holds an answer, and a
|
||||
// skipped one never started because something upstream did not finish.
|
||||
type NodeState string
|
||||
|
||||
const (
|
||||
StatePending NodeState = "pending"
|
||||
// StateQueued is ready and waiting for a session concurrency slot. It is
|
||||
// not StatePending: nothing upstream is missing, and the only thing between
|
||||
// this node and its first step is the scheduler's ceiling.
|
||||
StateQueued NodeState = "queued"
|
||||
StateRunning NodeState = "running"
|
||||
StateCompleted NodeState = "completed"
|
||||
StateAdopted NodeState = "adopted"
|
||||
StateFailed NodeState = "failed"
|
||||
StateCancelled NodeState = "cancelled"
|
||||
StateSkipped NodeState = "skipped"
|
||||
)
|
||||
|
||||
// Answered reports whether this state holds a final answer a dependent may
|
||||
// start from: one the node produced, or one adopted in its place.
|
||||
func (s NodeState) Answered() bool { return s == StateCompleted || s == StateAdopted }
|
||||
|
||||
// Terminal reports whether the node will not change state again.
|
||||
func (s NodeState) Terminal() bool {
|
||||
return s != StatePending && s != StateQueued && s != StateRunning
|
||||
}
|
||||
|
||||
// Grant is what a node was allowed to touch while it ran. Empty means the
|
||||
// question does not apply — a group is not a run, and an adopted node never got
|
||||
// one — which a bool could not say apart from "may write".
|
||||
type Grant string
|
||||
|
||||
const (
|
||||
GrantRead Grant = "read"
|
||||
GrantWrite Grant = "write"
|
||||
)
|
||||
|
||||
// WaitCause is why a queued node is not running. Queued says a node is ready and
|
||||
// held back; it does not say by what, and the answers have different remedies —
|
||||
// two are session settings and the third is a path or an edge.
|
||||
type WaitCause string
|
||||
|
||||
const (
|
||||
// WaitSlots is the session's total sub-agent ceiling.
|
||||
WaitSlots WaitCause = "slots"
|
||||
// WaitWriters is the session's concurrent-writer ceiling, which binds a
|
||||
// writer while total capacity is still free.
|
||||
WaitWriters WaitCause = "writers"
|
||||
// WaitClaim is a write path someone else already holds. No ceiling change
|
||||
// releases it: only disjoint paths, or an edge that orders the two.
|
||||
WaitClaim WaitCause = "claim"
|
||||
)
|
||||
|
||||
// EdgeKind is what one edge means. The node pair alone says nothing: waiting
|
||||
// for an answer and receiving one are separate facts, and an arm that orders
|
||||
// two nodes while delivering nothing between them is the case a single edge
|
||||
// type cannot describe.
|
||||
type EdgeKind string
|
||||
|
||||
const (
|
||||
// Spawn is structural: To was created by From.
|
||||
Spawn EdgeKind = "spawn"
|
||||
// Depends orders: To may not start until From has answered.
|
||||
Depends EdgeKind = "depends"
|
||||
// Context carries: From's answer opened To's run.
|
||||
Context EdgeKind = "context"
|
||||
// Adopt reuses: From's existing answer stood in for running To, which
|
||||
// therefore cost nothing. From is usually outside this graph.
|
||||
Adopt EdgeKind = "adopt"
|
||||
)
|
||||
|
||||
// Node is one participant in a run. Zero fields mean "not learned yet", which
|
||||
// is what lets a producer report an outcome without restating the identity it
|
||||
// declared when the node entered the graph.
|
||||
type Node struct {
|
||||
ID string `json:"id"`
|
||||
ParentID string `json:"parentId,omitempty"`
|
||||
Kind NodeKind `json:"kind,omitempty"`
|
||||
State NodeState `json:"state,omitempty"`
|
||||
Label string `json:"label,omitempty"`
|
||||
Profile string `json:"profile,omitempty"`
|
||||
Model string `json:"model,omitempty"`
|
||||
Effort string `json:"effort,omitempty"`
|
||||
Grant Grant `json:"grant,omitempty"`
|
||||
// Wait is what held this node out of a slot. Like the stamps it survives the
|
||||
// run starting: the state says it is running now, this says what it waited on.
|
||||
Wait WaitCause `json:"wait,omitempty"`
|
||||
// Ref is the persisted transcript this node's answer reads back from.
|
||||
Ref string `json:"ref,omitempty"`
|
||||
// QueuedAt, StartedAt and EndedAt are unix milliseconds, zero while unknown.
|
||||
// QueuedAt to StartedAt is slot wait, which only the scheduler can measure.
|
||||
QueuedAt int64 `json:"queuedAt,omitempty"`
|
||||
StartedAt int64 `json:"startedAt,omitempty"`
|
||||
EndedAt int64 `json:"endedAt,omitempty"`
|
||||
Err string `json:"err,omitempty"`
|
||||
}
|
||||
|
||||
// Edge is one typed relation between two node ids. From may name a node outside
|
||||
// this graph: an adopted answer's source is work some earlier run did.
|
||||
type Edge struct {
|
||||
From string `json:"from"`
|
||||
To string `json:"to"`
|
||||
Kind EdgeKind `json:"kind"`
|
||||
}
|
||||
|
||||
// Delta is one publication of what a producer has just proven. It is additive:
|
||||
// a node named again updates the one already there rather than forking into a
|
||||
// second, and a repeated edge is the same edge.
|
||||
type Delta struct {
|
||||
Nodes []Node `json:"nodes,omitempty"`
|
||||
Edges []Edge `json:"edges,omitempty"`
|
||||
}
|
||||
|
||||
// Empty reports whether this delta carries nothing worth publishing.
|
||||
func (d Delta) Empty() bool { return len(d.Nodes) == 0 && len(d.Edges) == 0 }
|
||||
|
||||
// Graph is the folded result. Nodes and edges keep first-seen order so two
|
||||
// consumers fed the same deltas draw the same picture. Lookups scan rather than
|
||||
// index: a graph is bounded by one call's fan-out, and a map would have to be
|
||||
// rebuilt every time the value crossed a wire.
|
||||
type Graph struct {
|
||||
Nodes []Node `json:"nodes,omitempty"`
|
||||
Edges []Edge `json:"edges,omitempty"`
|
||||
}
|
||||
|
||||
// Apply folds one delta into the graph.
|
||||
func (g *Graph) Apply(d Delta) {
|
||||
for _, n := range d.Nodes {
|
||||
g.upsert(n)
|
||||
}
|
||||
for _, e := range d.Edges {
|
||||
g.link(e)
|
||||
}
|
||||
}
|
||||
|
||||
// Node returns the node with this id.
|
||||
func (g *Graph) Node(id string) (Node, bool) {
|
||||
if i := g.indexOf(id); i >= 0 {
|
||||
return g.Nodes[i], true
|
||||
}
|
||||
return Node{}, false
|
||||
}
|
||||
|
||||
func (g *Graph) indexOf(id string) int {
|
||||
return slices.IndexFunc(g.Nodes, func(n Node) bool { return n.ID == id })
|
||||
}
|
||||
|
||||
func (g *Graph) upsert(n Node) {
|
||||
if n.ID == "" {
|
||||
return
|
||||
}
|
||||
if i := g.indexOf(n.ID); i >= 0 {
|
||||
g.Nodes[i] = merge(g.Nodes[i], n)
|
||||
return
|
||||
}
|
||||
g.Nodes = append(g.Nodes, n)
|
||||
}
|
||||
|
||||
// merge keeps what the graph already knew wherever the update says nothing, so
|
||||
// an outcome-only update cannot blank the identity an earlier delta declared.
|
||||
func merge(old, update Node) Node {
|
||||
out := update
|
||||
out.ParentID = orText(update.ParentID, old.ParentID)
|
||||
out.Kind = NodeKind(orText(string(update.Kind), string(old.Kind)))
|
||||
out.State = NodeState(orText(string(update.State), string(old.State)))
|
||||
out.Label = orText(update.Label, old.Label)
|
||||
out.Profile = orText(update.Profile, old.Profile)
|
||||
out.Model = orText(update.Model, old.Model)
|
||||
out.Effort = orText(update.Effort, old.Effort)
|
||||
out.Ref = orText(update.Ref, old.Ref)
|
||||
out.Err = orText(update.Err, old.Err)
|
||||
out.Grant = Grant(orText(string(update.Grant), string(old.Grant)))
|
||||
out.Wait = WaitCause(orText(string(update.Wait), string(old.Wait)))
|
||||
out.QueuedAt = orStamp(update.QueuedAt, old.QueuedAt)
|
||||
out.StartedAt = orStamp(update.StartedAt, old.StartedAt)
|
||||
out.EndedAt = orStamp(update.EndedAt, old.EndedAt)
|
||||
return out
|
||||
}
|
||||
|
||||
// link adds an edge once. A self-edge or a half-named one is a producer bug the
|
||||
// graph refuses to record rather than render as a loop nothing can explain.
|
||||
func (g *Graph) link(e Edge) {
|
||||
if e.From == "" || e.To == "" || e.Kind == "" || e.From == e.To {
|
||||
return
|
||||
}
|
||||
if !slices.Contains(g.Edges, e) {
|
||||
g.Edges = append(g.Edges, e)
|
||||
}
|
||||
}
|
||||
|
||||
func orText(update, old string) string {
|
||||
if update != "" {
|
||||
return update
|
||||
}
|
||||
return old
|
||||
}
|
||||
|
||||
func orStamp(update, old int64) int64 {
|
||||
if update != 0 {
|
||||
return update
|
||||
}
|
||||
return old
|
||||
}
|
||||
+153
@@ -0,0 +1,153 @@
|
||||
package agentgraph
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"slices"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestApplyUpsertsNodesByID(t *testing.T) {
|
||||
var g Graph
|
||||
g.Apply(Delta{Nodes: []Node{
|
||||
{ID: "a", Kind: KindWorker, State: StatePending, Label: "research", Profile: "explore", Grant: GrantRead},
|
||||
}})
|
||||
// An outcome-only update is what a producer sends once the node settles.
|
||||
g.Apply(Delta{Nodes: []Node{{ID: "a", State: StateCompleted, Ref: "sub-1", EndedAt: 42}}})
|
||||
|
||||
if len(g.Nodes) != 1 {
|
||||
t.Fatalf("nodes = %d, want 1 (the update forked the node)", len(g.Nodes))
|
||||
}
|
||||
n, ok := g.Node("a")
|
||||
if !ok {
|
||||
t.Fatal("node a missing")
|
||||
}
|
||||
if n.State != StateCompleted || n.Ref != "sub-1" || n.EndedAt != 42 {
|
||||
t.Fatalf("update did not land: %+v", n)
|
||||
}
|
||||
if n.Label != "research" || n.Profile != "explore" || n.Kind != KindWorker || n.Grant != GrantRead {
|
||||
t.Fatalf("outcome-only update blanked declared identity: %+v", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyIgnoresUnusableRecords(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
delta Delta
|
||||
}{
|
||||
{"node without id", Delta{Nodes: []Node{{State: StateRunning}}}},
|
||||
{"edge without source", Delta{Edges: []Edge{{To: "b", Kind: Depends}}}},
|
||||
{"edge without target", Delta{Edges: []Edge{{From: "a", Kind: Depends}}}},
|
||||
{"edge without kind", Delta{Edges: []Edge{{From: "a", To: "b"}}}},
|
||||
{"self edge", Delta{Edges: []Edge{{From: "a", To: "a", Kind: Depends}}}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
var g Graph
|
||||
g.Apply(c.delta)
|
||||
if len(g.Nodes) != 0 || len(g.Edges) != 0 {
|
||||
t.Fatalf("recorded %+v", g)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyKeepsEdgesDistinctByKind(t *testing.T) {
|
||||
var g Graph
|
||||
// depends and context join the same pair and are not the same fact: an arm
|
||||
// that orders without delivering emits the first and not the second.
|
||||
edges := []Edge{
|
||||
{From: "a", To: "b", Kind: Depends},
|
||||
{From: "a", To: "b", Kind: Context},
|
||||
{From: "a", To: "b", Kind: Depends},
|
||||
}
|
||||
g.Apply(Delta{Edges: edges})
|
||||
if len(g.Edges) != 2 {
|
||||
t.Fatalf("edges = %v, want depends and context once each", g.Edges)
|
||||
}
|
||||
for _, want := range []Edge{{From: "a", To: "b", Kind: Depends}, {From: "a", To: "b", Kind: Context}} {
|
||||
if !slices.Contains(g.Edges, want) {
|
||||
t.Fatalf("missing %+v in %v", want, g.Edges)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyIsOrderStable(t *testing.T) {
|
||||
build := func() Graph {
|
||||
var g Graph
|
||||
for _, id := range []string{"a", "b", "c"} {
|
||||
g.Apply(Delta{Nodes: []Node{{ID: id, Kind: KindWorker, State: StatePending}}})
|
||||
}
|
||||
g.Apply(Delta{Nodes: []Node{{ID: "b", State: StateRunning}}})
|
||||
return g
|
||||
}
|
||||
first, second := build(), build()
|
||||
if len(first.Nodes) != 3 {
|
||||
t.Fatalf("nodes = %d, want 3", len(first.Nodes))
|
||||
}
|
||||
for i := range first.Nodes {
|
||||
if first.Nodes[i] != second.Nodes[i] {
|
||||
t.Fatalf("node %d diverged: %+v vs %+v", i, first.Nodes[i], second.Nodes[i])
|
||||
}
|
||||
}
|
||||
if first.Nodes[1].ID != "b" || first.Nodes[1].State != StateRunning {
|
||||
t.Fatalf("an update reordered the graph: %+v", first.Nodes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStateVocabulary(t *testing.T) {
|
||||
cases := []struct {
|
||||
state NodeState
|
||||
answered bool
|
||||
terminal bool
|
||||
}{
|
||||
{StatePending, false, false},
|
||||
// Queued is not terminal: the node has not run, and a reader that treats
|
||||
// it as settled stops waiting for the answer it is still going to get.
|
||||
{StateQueued, false, false},
|
||||
{StateRunning, false, false},
|
||||
{StateCompleted, true, true},
|
||||
{StateAdopted, true, true},
|
||||
{StateFailed, false, true},
|
||||
{StateCancelled, false, true},
|
||||
{StateSkipped, false, true},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := c.state.Answered(); got != c.answered {
|
||||
t.Errorf("%s.Answered() = %v, want %v", c.state, got, c.answered)
|
||||
}
|
||||
if got := c.state.Terminal(); got != c.terminal {
|
||||
t.Errorf("%s.Terminal() = %v, want %v", c.state, got, c.terminal)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A graph crosses a wire before it is drawn, so the folded value has to survive
|
||||
// the round trip that every consumer outside this process reads it through.
|
||||
func TestGraphSurvivesJSONRoundTrip(t *testing.T) {
|
||||
var g Graph
|
||||
g.Apply(Delta{
|
||||
Nodes: []Node{
|
||||
{ID: "grp", Kind: KindGroup, State: StateRunning, Label: "fleet(2)"},
|
||||
{ID: "grp/1", ParentID: "grp", Kind: KindWorker, State: StateCompleted, Model: "m", StartedAt: 1, EndedAt: 2},
|
||||
{ID: "old-ref", Kind: KindExternal, State: StateCompleted},
|
||||
},
|
||||
Edges: []Edge{
|
||||
{From: "grp", To: "grp/1", Kind: Spawn},
|
||||
{From: "old-ref", To: "grp/1", Kind: Adopt},
|
||||
},
|
||||
})
|
||||
raw, err := json.Marshal(g)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var back Graph
|
||||
if err := json.Unmarshal(raw, &back); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !slices.Equal(back.Edges, g.Edges) {
|
||||
t.Fatalf("edges = %v, want %v", back.Edges, g.Edges)
|
||||
}
|
||||
if !slices.Equal(back.Nodes, g.Nodes) {
|
||||
t.Fatalf("nodes = %v, want %v", back.Nodes, g.Nodes)
|
||||
}
|
||||
}
|
||||
+220
@@ -0,0 +1,220 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Official catalog metadata: currency, model, effective dates, documentation
|
||||
// sources, and price fingerprints. User-custom prices always win over catalog.
|
||||
|
||||
// CatalogEntry is one official list price for a model in a billing currency.
|
||||
type CatalogEntry struct {
|
||||
Provider string // deepseek | longcat | mimo | google
|
||||
Model string
|
||||
Currency string // ISO billing currency for this row
|
||||
CacheHit float64
|
||||
Input float64
|
||||
Output float64
|
||||
EffectiveFrom string // YYYY-MM-DD inclusive, empty = unbounded
|
||||
EffectiveTo string // YYYY-MM-DD exclusive, empty = unbounded
|
||||
DocURL string
|
||||
BillingMode string // payg | subscription_equivalent
|
||||
Notes string
|
||||
Fingerprint string // filled by Register
|
||||
// Peak is what the same tokens cost inside Window. Nil when the vendor
|
||||
// bills one rate around the clock, and then the fields above are the rate.
|
||||
Peak *RateCard
|
||||
Window *PeakWindow
|
||||
// CheckedOn is when a person last read DocURL and confirmed these numbers.
|
||||
// No vendor here serves prices over an API, so this date is all that stands
|
||||
// between a price change and a readout that is quietly wrong.
|
||||
CheckedOn string
|
||||
}
|
||||
|
||||
// deepseekPeak is the schedule DeepSeek publishes: peak on weekday mornings and
|
||||
// afternoons, and — since 2026-08-23 — never at a weekend. Chinese public
|
||||
// holidays bill off-peak in full, per the same page.
|
||||
var deepseekPeak = &PeakWindow{
|
||||
OffsetSeconds: beijing,
|
||||
Hours: [][2]int{{9, 12}, {14, 18}},
|
||||
WeekendOffPeak: "2026-08-23",
|
||||
HolidayOffPeak: chineseStatutoryHolidays,
|
||||
}
|
||||
|
||||
// chineseStatutoryHolidays are the State Council's 2026 public holidays, which
|
||||
// DeepSeek bills off-peak in full; extend it when the next notice is out. Make-up
|
||||
// workdays are absent: they fall on weekends, which are off-peak anyway.
|
||||
var chineseStatutoryHolidays = []string{
|
||||
// New Year's Day: Jan 1-3.
|
||||
"2026-01-01", "2026-01-02", "2026-01-03",
|
||||
// Spring Festival: Feb 15-23.
|
||||
"2026-02-15", "2026-02-16", "2026-02-17", "2026-02-18", "2026-02-19",
|
||||
"2026-02-20", "2026-02-21", "2026-02-22", "2026-02-23",
|
||||
// Qingming Festival: Apr 4-6.
|
||||
"2026-04-04", "2026-04-05", "2026-04-06",
|
||||
// Labor Day: May 1-5.
|
||||
"2026-05-01", "2026-05-02", "2026-05-03", "2026-05-04", "2026-05-05",
|
||||
// Dragon Boat Festival: Jun 19-21.
|
||||
"2026-06-19", "2026-06-20", "2026-06-21",
|
||||
// Mid-Autumn Festival: Sep 25-27.
|
||||
"2026-09-25", "2026-09-26", "2026-09-27",
|
||||
// National Day: Oct 1-7.
|
||||
"2026-10-01", "2026-10-02", "2026-10-03", "2026-10-04",
|
||||
"2026-10-05", "2026-10-06", "2026-10-07",
|
||||
}
|
||||
|
||||
// checkedOn is when the tables below were last read off the vendors' pages.
|
||||
const checkedOn = "2026-08-22"
|
||||
|
||||
// deepseekCheckedOn is when the DeepSeek rows were last read off that vendor's
|
||||
// page. Separate from checkedOn, because confirming one vendor's prices says
|
||||
// nothing about another's and a shared date would claim that it did.
|
||||
const deepseekCheckedOn = "2026-09-13"
|
||||
|
||||
// mimoCheckedOn is when the V2.6 rows were read off the vendor's own docs.
|
||||
const mimoCheckedOn = "2026-10-06"
|
||||
|
||||
// googleCheckedOn is when the Gemini rows were read off Google's pricing page.
|
||||
const googleCheckedOn = "2026-10-17"
|
||||
|
||||
// CatalogSourceURLs document public pricing pages.
|
||||
const (
|
||||
DocDeepSeekPricing = "https://api-docs.deepseek.com/quick_start/pricing"
|
||||
DocDeepSeekPricingCNY = "https://api-docs.deepseek.com/zh-cn/quick_start/pricing"
|
||||
DocLongCatPricingUSD = "https://longcat.chat/platform/docs/Pricing/LongCat-2.0.html"
|
||||
DocLongCatPricingCNY = "https://longcat.chat/platform/docs/zh/pricing/long-cat-2.0"
|
||||
DocMiMoPAYG = "https://mimo.mi.com/docs/price/pay-as-you-go"
|
||||
DocMiMoTokenPlan = "https://platform.xiaomimimo.com/token-plan"
|
||||
DocGeminiPricing = "https://ai.google.dev/gemini-api/docs/pricing"
|
||||
)
|
||||
|
||||
// officialEndpointHosts maps a vendor's own API hostnames to the catalog
|
||||
// provider whose list prices apply there. Host equality is the test: a reseller
|
||||
// or proxy bills on its own terms even when it serves the same model, and an
|
||||
// entry merely named after a vendor is not evidence of the vendor's endpoint.
|
||||
var officialEndpointHosts = map[string]string{
|
||||
"api.deepseek.com": "deepseek",
|
||||
"api.longcat.chat": "longcat",
|
||||
"api.xiaomimimo.com": "mimo",
|
||||
"generativelanguage.googleapis.com": "google",
|
||||
"token-plan-cn.xiaomimimo.com": "mimo",
|
||||
"token-plan-sgp.xiaomimimo.com": "mimo",
|
||||
"token-plan-ams.xiaomimimo.com": "mimo",
|
||||
}
|
||||
|
||||
// OfficialProviderForEndpoint returns the catalog provider whose official
|
||||
// prices apply to baseURL, or "" when the address is not a vendor's own API.
|
||||
func OfficialProviderForEndpoint(baseURL string) string {
|
||||
u, err := url.Parse(strings.TrimSpace(baseURL))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return officialEndpointHosts[strings.ToLower(u.Hostname())]
|
||||
}
|
||||
|
||||
// OfficialCatalog is the built-in price book: the current generation of every
|
||||
// rate in officialRates, fingerprinted so a stored price can be recognised as
|
||||
// one of ours. User-custom prices always win over it.
|
||||
func OfficialCatalog() []CatalogEntry {
|
||||
entries := officialCatalogRows()
|
||||
for i := range entries {
|
||||
entries[i].Fingerprint = PricingFingerprint(RateCard{
|
||||
CacheHit: entries[i].CacheHit,
|
||||
Input: entries[i].Input,
|
||||
Output: entries[i].Output,
|
||||
Currency: entries[i].Currency,
|
||||
})
|
||||
}
|
||||
return entries
|
||||
}
|
||||
|
||||
// LookupCatalog finds an official entry. billingMode empty matches payg first.
|
||||
func LookupCatalog(provider, model, currency, billingMode string) (CatalogEntry, bool) {
|
||||
provider = strings.ToLower(strings.TrimSpace(provider))
|
||||
model = strings.TrimSpace(model)
|
||||
currency = NormalizeCurrency(currency)
|
||||
billingMode = strings.TrimSpace(billingMode)
|
||||
if billingMode == "" {
|
||||
billingMode = BillingModePAYG
|
||||
}
|
||||
var fallback CatalogEntry
|
||||
foundFallback := false
|
||||
for _, e := range OfficialCatalog() {
|
||||
if e.Provider != provider || e.Model != model {
|
||||
continue
|
||||
}
|
||||
if NormalizeCurrency(e.Currency) != currency {
|
||||
continue
|
||||
}
|
||||
if e.BillingMode == billingMode {
|
||||
return e, true
|
||||
}
|
||||
if e.BillingMode == BillingModePAYG {
|
||||
fallback = e
|
||||
foundFallback = true
|
||||
}
|
||||
}
|
||||
if foundFallback {
|
||||
return fallback, true
|
||||
}
|
||||
return CatalogEntry{}, false
|
||||
}
|
||||
|
||||
// RateCardFromCatalog builds a RateCard from a catalog entry's base rate.
|
||||
func RateCardFromCatalog(e CatalogEntry) RateCard {
|
||||
return RateCard{
|
||||
CacheHit: e.CacheHit,
|
||||
Input: e.Input,
|
||||
Output: e.Output,
|
||||
Currency: e.Currency,
|
||||
}
|
||||
}
|
||||
|
||||
// RatesAt is what this entry charges at t: the peak card inside its window, the
|
||||
// base card everywhere else. An entry without a window charges one rate, and
|
||||
// then t does not matter.
|
||||
func (e CatalogEntry) RatesAt(t time.Time) RateCard {
|
||||
if e.Peak == nil || !e.Window.IsPeak(t) {
|
||||
return RateCardFromCatalog(e)
|
||||
}
|
||||
peak := *e.Peak
|
||||
peak.Currency = e.Currency
|
||||
return peak
|
||||
}
|
||||
|
||||
// InEffect reports whether this entry's dates cover the day t falls on.
|
||||
func (e CatalogEntry) InEffect(t time.Time) bool {
|
||||
day := t.UTC().Format("2006-01-02")
|
||||
if e.EffectiveFrom != "" && day < e.EffectiveFrom {
|
||||
return false
|
||||
}
|
||||
return e.EffectiveTo == "" || day < e.EffectiveTo
|
||||
}
|
||||
|
||||
// MatchesCatalog reports whether rates equal a known official entry.
|
||||
func MatchesCatalog(provider, model string, rates RateCard) (CatalogEntry, bool) {
|
||||
cur := NormalizeCurrency(rates.Currency)
|
||||
for _, e := range OfficialCatalog() {
|
||||
if e.Provider != strings.ToLower(strings.TrimSpace(provider)) {
|
||||
continue
|
||||
}
|
||||
if e.Model != strings.TrimSpace(model) {
|
||||
continue
|
||||
}
|
||||
if NormalizeCurrency(e.Currency) != cur {
|
||||
continue
|
||||
}
|
||||
// Either card identifies the vendor's own price. Which one a user
|
||||
// happens to have configured says nothing about when they are billing:
|
||||
// the hour decides that, and RatesAt is what reads it.
|
||||
if e.CacheHit == rates.CacheHit && e.Input == rates.Input && e.Output == rates.Output {
|
||||
return e, true
|
||||
}
|
||||
if p := e.Peak; p != nil && p.CacheHit == rates.CacheHit && p.Input == rates.Input && p.Output == rates.Output {
|
||||
return e, true
|
||||
}
|
||||
}
|
||||
return CatalogEntry{}, false
|
||||
}
|
||||
+95
@@ -0,0 +1,95 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Every catalog provider must declare at least one official endpoint, or its
|
||||
// rows become unreachable: a rate card is only matched against the catalog
|
||||
// after the entry's host identifies the vendor.
|
||||
func TestEveryCatalogProviderHasAnOfficialEndpoint(t *testing.T) {
|
||||
declared := map[string]bool{}
|
||||
for _, host := range officialEndpointHosts {
|
||||
declared[host] = true
|
||||
}
|
||||
for _, entry := range OfficialCatalog() {
|
||||
if !declared[entry.Provider] {
|
||||
t.Errorf("catalog provider %q has no official endpoint host; its prices can never match", entry.Provider)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestOfficialProviderForEndpoint(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
baseURL string
|
||||
want string
|
||||
}{
|
||||
{"official deepseek", "https://api.deepseek.com", "deepseek"},
|
||||
{"official deepseek anthropic path", "https://api.deepseek.com/anthropic", "deepseek"},
|
||||
{"official longcat", "https://api.longcat.chat/openai/v1", "longcat"},
|
||||
{"official mimo region", "https://token-plan-sgp.xiaomimimo.com/v1", "mimo"},
|
||||
{"uppercase host", "https://API.DeepSeek.com/v1", "deepseek"},
|
||||
{"proxy that resells the model", "https://my-deepseek-proxy.internal/v1", ""},
|
||||
{"vendor name in the path only", "https://gateway.example.com/deepseek/v1", ""},
|
||||
{"lookalike registrable domain", "https://api.deepseek.com.evil.test/v1", ""},
|
||||
{"empty", "", ""},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := OfficialProviderForEndpoint(tc.baseURL); got != tc.want {
|
||||
t.Fatalf("OfficialProviderForEndpoint(%q) = %q, want %q", tc.baseURL, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// pricesGoStaleAfter is how long a hand-copied rate is trusted before someone
|
||||
// reads the vendor's page again. No vendor here serves prices over an API, so
|
||||
// the alternative to this going red on a quiet day is learning about a change
|
||||
// from a user whose cost was wrong for months. Bumping CheckedOn without
|
||||
// opening DocURL defeats it; what this buys is that the question gets asked.
|
||||
const pricesGoStaleAfter = 90 * 24 * time.Hour
|
||||
|
||||
func TestCatalogPricesHaveBeenCheckedRecently(t *testing.T) {
|
||||
var undated, stale []string
|
||||
// This snapshot never sees a vendor page again, so staleness is read at the
|
||||
// day it was taken rather than the day the suite runs.
|
||||
cutoff := time.Date(2026, 10, 19, 0, 0, 0, 0, time.UTC).Add(-pricesGoStaleAfter)
|
||||
for _, e := range OfficialCatalog() {
|
||||
where := e.Provider + "/" + e.Model + " " + e.Currency + " (" + e.DocURL + ")"
|
||||
if e.CheckedOn == "" {
|
||||
undated = append(undated, where)
|
||||
continue
|
||||
}
|
||||
on, err := time.Parse("2006-01-02", e.CheckedOn)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: CheckedOn %q is not YYYY-MM-DD", where, e.CheckedOn)
|
||||
}
|
||||
if on.Before(cutoff) {
|
||||
stale = append(stale, where+" last checked "+e.CheckedOn)
|
||||
}
|
||||
}
|
||||
if len(undated) > 0 {
|
||||
t.Errorf("catalog rows with no CheckedOn — read the vendor's page and date them:\n %s",
|
||||
strings.Join(undated, "\n "))
|
||||
}
|
||||
if len(stale) > 0 {
|
||||
t.Errorf("catalog rows older than %d days. Open each DocURL, confirm or correct the rate, then set CheckedOn to today:\n %s",
|
||||
int(pricesGoStaleAfter.Hours()/24), strings.Join(stale, "\n "))
|
||||
}
|
||||
}
|
||||
|
||||
// A row that names a peak card must say when it applies, and one that names a
|
||||
// window must have a card to charge inside it. Half of the pair silently prices
|
||||
// every hour the same, which is the bug the pair exists to prevent.
|
||||
func TestPeakRatesAndWindowsComeInPairs(t *testing.T) {
|
||||
for _, e := range OfficialCatalog() {
|
||||
where := e.Provider + "/" + e.Model + " " + e.Currency
|
||||
if (e.Peak == nil) != (e.Window == nil) {
|
||||
t.Errorf("%s: Peak and Window must be set together (peak=%v window=%v)", where, e.Peak != nil, e.Window != nil)
|
||||
}
|
||||
}
|
||||
}
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
package pricing
|
||||
|
||||
// Coverage says how much of the usage behind a quote carries a complete price
|
||||
// fact. CostComplete answers that for one call; an aggregate has two answers
|
||||
// more, because it can price some of its calls and not others, and it can have
|
||||
// billed nothing at all. A session that has not spent yet is not a session
|
||||
// whose price is unknown.
|
||||
const (
|
||||
CoverageNone = "none"
|
||||
CoverageComplete = "complete"
|
||||
CoveragePartial = "partial"
|
||||
CoverageIncomplete = "incomplete"
|
||||
)
|
||||
|
||||
// ValidCoverage reports whether c is one of the four identities.
|
||||
func ValidCoverage(c string) bool {
|
||||
switch c {
|
||||
case CoverageNone, CoverageComplete, CoveragePartial, CoverageIncomplete:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// FoldCoverage returns the coverage of a and b together: the join over complete
|
||||
// and incomplete, with none as the identity and partial as the top. Order cannot
|
||||
// reach the answer, and a run that has priced everything so far cannot fold back
|
||||
// to complete once an unpriced call joins it. A value outside the four folds as
|
||||
// none; NormalizeQuote repairs a decoded quote before it can reach here.
|
||||
func FoldCoverage(a, b string) string {
|
||||
a, b = coverageOrNone(a), coverageOrNone(b)
|
||||
switch {
|
||||
case a == CoverageNone:
|
||||
return b
|
||||
case b == CoverageNone:
|
||||
return a
|
||||
case a == b:
|
||||
return a
|
||||
default:
|
||||
return CoveragePartial
|
||||
}
|
||||
}
|
||||
|
||||
func coverageOrNone(c string) string {
|
||||
if ValidCoverage(c) {
|
||||
return c
|
||||
}
|
||||
return CoverageNone
|
||||
}
|
||||
+190
@@ -0,0 +1,190 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
var allCoverages = []string{CoverageNone, CoverageComplete, CoveragePartial, CoverageIncomplete}
|
||||
|
||||
// The contract the aggregate has to satisfy, stated as the pairs themselves.
|
||||
func TestFoldCoverageContract(t *testing.T) {
|
||||
cases := []struct{ a, b, want string }{
|
||||
{CoverageNone, CoverageNone, CoverageNone},
|
||||
{CoverageComplete, CoverageNone, CoverageComplete},
|
||||
{CoverageComplete, CoverageComplete, CoverageComplete},
|
||||
{CoverageComplete, CoverageIncomplete, CoveragePartial},
|
||||
{CoverageIncomplete, CoverageComplete, CoveragePartial},
|
||||
{CoverageIncomplete, CoverageIncomplete, CoverageIncomplete},
|
||||
{CoveragePartial, CoverageComplete, CoveragePartial},
|
||||
{CoveragePartial, CoverageIncomplete, CoveragePartial},
|
||||
{CoveragePartial, CoverageNone, CoveragePartial},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := FoldCoverage(c.a, c.b); got != c.want {
|
||||
t.Errorf("fold(%s, %s) = %s, want %s", c.a, c.b, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Arrival order is not a fact about the session, so it may not reach the
|
||||
// answer. Exhaustive rather than sampled: there are only sixty-four triples.
|
||||
func TestFoldCoverageIsOrderIndependent(t *testing.T) {
|
||||
for _, a := range allCoverages {
|
||||
for _, b := range allCoverages {
|
||||
if FoldCoverage(a, b) != FoldCoverage(b, a) {
|
||||
t.Errorf("fold(%s, %s) != fold(%s, %s)", a, b, b, a)
|
||||
}
|
||||
if FoldCoverage(a, a) != a {
|
||||
t.Errorf("fold(%s, %s) = %s, want %s", a, a, FoldCoverage(a, a), a)
|
||||
}
|
||||
for _, c := range allCoverages {
|
||||
left := FoldCoverage(FoldCoverage(a, b), c)
|
||||
right := FoldCoverage(a, FoldCoverage(b, c))
|
||||
if left != right {
|
||||
t.Errorf("fold is not associative on (%s, %s, %s): %s vs %s", a, b, c, left, right)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The failure a boolean invites: the last quote deciding for the ones before it.
|
||||
func TestCompleteNeverSurvivesALaterIncomplete(t *testing.T) {
|
||||
for _, a := range allCoverages {
|
||||
got := FoldCoverage(FoldCoverage(a, CoverageComplete), CoverageIncomplete)
|
||||
if got == CoverageComplete {
|
||||
t.Errorf("fold(%s, complete, incomplete) = complete", a)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFoldCoverageTreatsAnUnnamedStateAsNothing(t *testing.T) {
|
||||
if got := FoldCoverage("", CoverageComplete); got != CoverageComplete {
|
||||
t.Fatalf("fold(\"\", complete) = %s", got)
|
||||
}
|
||||
if got := FoldCoverage("mostly", CoverageIncomplete); got != CoverageIncomplete {
|
||||
t.Fatalf("fold(unnamed, incomplete) = %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func pricedQuote(t *testing.T, perMillion float64) CostQuote {
|
||||
t.Helper()
|
||||
q := BuildQuote(QuoteInput{
|
||||
Usage: UsageTokens{PromptTokens: 1000},
|
||||
Rates: RateCard{Input: perMillion, Currency: "USD"},
|
||||
})
|
||||
if q.Coverage != CoverageComplete {
|
||||
t.Fatalf("a priced call is not complete: %+v", q)
|
||||
}
|
||||
return q
|
||||
}
|
||||
|
||||
func unpricedQuote(t *testing.T) CostQuote {
|
||||
t.Helper()
|
||||
q := NormalizeQuote(CostQuote{
|
||||
Estimated: true, DisplayStatus: DisplayStatusUnavailable, IncompleteReason: "no_price",
|
||||
})
|
||||
if q.Coverage != CoverageIncomplete {
|
||||
t.Fatalf("an unpriced call is not incomplete: %+v", q)
|
||||
}
|
||||
return q
|
||||
}
|
||||
|
||||
func TestAggregateCoverageAcrossCalls(t *testing.T) {
|
||||
priced, unpriced := pricedQuote(t, 1), unpricedQuote(t)
|
||||
cases := []struct {
|
||||
name string
|
||||
quotes []CostQuote
|
||||
want string
|
||||
}{
|
||||
{"no usage", nil, CoverageNone},
|
||||
{"one priced", []CostQuote{priced}, CoverageComplete},
|
||||
{"two priced", []CostQuote{priced, priced}, CoverageComplete},
|
||||
{"priced then unpriced", []CostQuote{priced, unpriced}, CoveragePartial},
|
||||
{"unpriced then priced", []CostQuote{unpriced, priced}, CoveragePartial},
|
||||
{"unpriced only", []CostQuote{unpriced}, CoverageIncomplete},
|
||||
{"unpriced twice", []CostQuote{unpriced, unpriced}, CoverageIncomplete},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got := AggregateQuotes(c.quotes, "USD")
|
||||
if got.Coverage != c.want {
|
||||
t.Errorf("%s: coverage = %s, want %s", c.name, got.Coverage, c.want)
|
||||
}
|
||||
// The boolean is a reading of the coverage, not a second answer.
|
||||
if got.CostComplete != (c.want == CoverageComplete) {
|
||||
t.Errorf("%s: costComplete = %v against coverage %s", c.name, got.CostComplete, got.Coverage)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A session's completeness is a fact about its calls, so the order they were
|
||||
// billed in may not change it — at the aggregate, not only in the fold.
|
||||
func TestAggregateCoverageIgnoresArrivalOrder(t *testing.T) {
|
||||
priced, unpriced := pricedQuote(t, 1), unpricedQuote(t)
|
||||
forward := AggregateQuotes([]CostQuote{priced, priced, unpriced}, "USD")
|
||||
backward := AggregateQuotes([]CostQuote{unpriced, priced, priced}, "USD")
|
||||
if forward.Coverage != backward.Coverage || forward.Coverage != CoveragePartial {
|
||||
t.Fatalf("order changed coverage: %s vs %s", forward.Coverage, backward.Coverage)
|
||||
}
|
||||
}
|
||||
|
||||
// Zero is an amount, not a completeness state. Both sessions total zero; only
|
||||
// one of them has been billed for anything.
|
||||
func TestKnownZeroIsCompleteAndNoUsageIsNot(t *testing.T) {
|
||||
free := AggregateQuotes([]CostQuote{pricedQuote(t, 0)}, "USD")
|
||||
if free.Coverage != CoverageComplete || !free.CostComplete {
|
||||
t.Fatalf("a free call priced at zero is not complete: %+v", free)
|
||||
}
|
||||
if amount := free.Original.AmountValue().Float64(); amount != 0 {
|
||||
t.Fatalf("a free call cost %v", amount)
|
||||
}
|
||||
empty := AggregateQuotes(nil, "USD")
|
||||
if empty.Coverage != CoverageNone {
|
||||
t.Fatalf("no usage = %s", empty.Coverage)
|
||||
}
|
||||
if amount := empty.Original.AmountValue().Float64(); amount != 0 {
|
||||
t.Fatalf("an empty session cost %v", amount)
|
||||
}
|
||||
if free.Coverage == empty.Coverage {
|
||||
t.Fatal("a billed zero and an unbilled session share one identity")
|
||||
}
|
||||
}
|
||||
|
||||
// A quote persisted before coverage existed still has to name one, and the
|
||||
// name has to be the one its own cost fact supports.
|
||||
func TestNormalizeFillsCoverageForAQuoteThatPredatesIt(t *testing.T) {
|
||||
old := NormalizeQuote(CostQuote{Original: MoneyOf(NewAmountFromFloat(1), "USD"), Complete: true})
|
||||
if old.Coverage != CoverageComplete {
|
||||
t.Fatalf("an old complete quote = %s", old.Coverage)
|
||||
}
|
||||
if got := NormalizeQuote(CostQuote{Coverage: "mostly", IncompleteReason: "no_price"}); got.Coverage != CoverageIncomplete {
|
||||
t.Fatalf("an unnamed coverage survived normalization: %s", got.Coverage)
|
||||
}
|
||||
kept := NormalizeQuote(CostQuote{Coverage: CoveragePartial, IncompleteReason: "no_price"})
|
||||
if kept.Coverage != CoveragePartial {
|
||||
t.Fatalf("normalization overwrote a coverage the quote already had: %s", kept.Coverage)
|
||||
}
|
||||
}
|
||||
|
||||
// The ledger merges by pricing key before anything aggregates, so it is a
|
||||
// second fold. An entry that has seen an unpriced call may not read complete
|
||||
// again because a later call arrived in another currency.
|
||||
func TestLedgerEntryKeepsIncompletenessAcrossBucketing(t *testing.T) {
|
||||
l := NewLedger()
|
||||
unpriced := unpricedQuote(t)
|
||||
unpriced.ModelRef, unpriced.UsageSource = "m", "executor"
|
||||
cny := BuildQuote(QuoteInput{Usage: UsageTokens{PromptTokens: 1000}, Rates: RateCard{Input: 1, Currency: "CNY"}, ModelRef: "m", UsageSource: "executor"})
|
||||
usd := BuildQuote(QuoteInput{Usage: UsageTokens{PromptTokens: 1000}, Rates: RateCard{Input: 1, Currency: "USD"}, ModelRef: "m", UsageSource: "executor"})
|
||||
usd.PricingFingerprint, cny.PricingFingerprint = unpriced.PricingFingerprint, unpriced.PricingFingerprint
|
||||
for _, q := range []CostQuote{unpriced, cny, usd} {
|
||||
l.Add(q, UsageTokens{PromptTokens: 1000}, time.Time{})
|
||||
}
|
||||
total := l.Total("USD")
|
||||
if total.Coverage != CoveragePartial {
|
||||
t.Fatalf("ledger total coverage = %s, want partial", total.Coverage)
|
||||
}
|
||||
if total.CostComplete {
|
||||
t.Fatal("an entry that saw an unpriced call reads as complete")
|
||||
}
|
||||
}
|
||||
+68
@@ -0,0 +1,68 @@
|
||||
package pricing
|
||||
|
||||
import "testing"
|
||||
|
||||
// https://ai.google.dev/gemini-api/docs/pricing (read 2026-10-17): per 1M tokens,
|
||||
// paid tier. gemini-3.8-flash is 0.75 in / 3.75 out / 0.075 cached through
|
||||
// 2026-12-31 and 1.50 / 7.50 / 0.15 from 2027-01-01.
|
||||
func TestAGeminiRateStepIsDatedNotEdited(t *testing.T) {
|
||||
was := today
|
||||
defer func() { today = was }()
|
||||
for _, tc := range []struct {
|
||||
day string
|
||||
cache, input, output float64
|
||||
}{
|
||||
{"2026-10-17", 0.075, 0.75, 3.75},
|
||||
{"2026-12-31", 0.075, 0.75, 3.75},
|
||||
{"2027-01-01", 0.15, 1.50, 7.50},
|
||||
{"2028-03-01", 0.15, 1.50, 7.50},
|
||||
} {
|
||||
today = func() string { return tc.day }
|
||||
got := CurrentRate("google", "gemini-3.8-flash", "USD")
|
||||
if got == nil || got.CacheHit != tc.cache || got.Input != tc.input || got.Output != tc.output {
|
||||
t.Errorf("%s: rate = %+v, want %v/%v/%v", tc.day, got, tc.cache, tc.input, tc.output)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheRateInForceIsNotListedAsSuperseded(t *testing.T) {
|
||||
was := today
|
||||
defer func() { today = was }()
|
||||
today = func() string { return "2026-10-17" }
|
||||
if sup := SupersededRates("google", "gemini-3.8-flash", "USD"); len(sup) != 0 {
|
||||
t.Fatalf("before the step nothing is superseded, got %+v", sup)
|
||||
}
|
||||
today = func() string { return "2027-02-01" }
|
||||
sup := SupersededRates("google", "gemini-3.8-flash", "USD")
|
||||
if len(sup) != 1 || sup[0].Input != 0.75 {
|
||||
t.Fatalf("after the step the intro rate is superseded, got %+v", sup)
|
||||
}
|
||||
if !IsOfficialCard("google", "gemini-3.8-flash", RateCard{CacheHit: 0.075, Input: 0.75, Output: 3.75, Currency: "USD"}) {
|
||||
t.Fatal("a stored intro card must still read as ours after the step")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGoogleModelsAreBilledAtGoogleOnlyOnGoogleHosts(t *testing.T) {
|
||||
if OfficialProviderForEndpoint("https://generativelanguage.googleapis.com/v1beta") != "google" {
|
||||
t.Fatal("Google's own host does not read as Google")
|
||||
}
|
||||
if OfficialProviderForEndpoint("https://relay.example.com/v1beta") != "" {
|
||||
t.Fatal("a relay was billed at Google's list price")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEveryGoogleRowNamesItsSource(t *testing.T) {
|
||||
n := 0
|
||||
for _, e := range OfficialCatalog() {
|
||||
if e.Provider != "google" {
|
||||
continue
|
||||
}
|
||||
n++
|
||||
if e.DocURL != DocGeminiPricing || e.CheckedOn != googleCheckedOn || e.Input <= 0 || e.Output <= e.Input {
|
||||
t.Errorf("row %+v", e)
|
||||
}
|
||||
}
|
||||
if n != 4 {
|
||||
t.Fatalf("%d Google rows, want 4 (gemini-4-argon shows no price)", n)
|
||||
}
|
||||
}
|
||||
+238
@@ -0,0 +1,238 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"maps"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// LedgerEntry is one occurrence-time cost fact. Ledger keys include model,
|
||||
// usage source, pricing fingerprint, and legacy rate date so model switches and
|
||||
// sub-agents never collapse into a single scalar.
|
||||
type LedgerEntry struct {
|
||||
Key string `json:"key"`
|
||||
ModelRef string `json:"modelRef"`
|
||||
UsageSource string `json:"usageSource"`
|
||||
PricingFingerprint string `json:"pricingFingerprint"`
|
||||
RateDate string `json:"rateDate,omitempty"`
|
||||
OccurredAt time.Time `json:"occurredAt"`
|
||||
Quote CostQuote `json:"quote"`
|
||||
// Token totals for this bucket (summed).
|
||||
PromptTokens int `json:"promptTokens"`
|
||||
CompletionTokens int `json:"completionTokens"`
|
||||
TotalTokens int `json:"totalTokens"`
|
||||
CacheHitTokens int `json:"cacheHitTokens"`
|
||||
CacheMissTokens int `json:"cacheMissTokens"`
|
||||
// CacheUnknownTokens is prompt the endpoint reported without saying how much
|
||||
// was cached. It is counted as a miss in CacheMissTokens for pricing and kept
|
||||
// here so a hit rate can leave it out instead of being pulled down by it.
|
||||
CacheUnknownTokens int `json:"cacheUnknownTokens,omitempty"`
|
||||
RequestCount int `json:"requestCount"`
|
||||
}
|
||||
|
||||
// LedgerKey builds the stable aggregation key.
|
||||
func LedgerKey(modelRef, usageSource, pricingFingerprint, rateDate string) string {
|
||||
return strings.Join([]string{
|
||||
strings.TrimSpace(modelRef),
|
||||
strings.TrimSpace(usageSource),
|
||||
strings.TrimSpace(pricingFingerprint),
|
||||
strings.TrimSpace(rateDate),
|
||||
}, "|")
|
||||
}
|
||||
|
||||
// Ledger accumulates occurrence-time quotes. Switching display currency only
|
||||
// re-selects valuations already stored; it does not reprice or clear history.
|
||||
type Ledger struct {
|
||||
Version int `json:"version"`
|
||||
Entries map[string]LedgerEntry `json:"entries"`
|
||||
}
|
||||
|
||||
// LedgerVersion is the current persisted ledger schema.
|
||||
const LedgerVersion = 1
|
||||
|
||||
// NewLedger returns an empty ledger.
|
||||
func NewLedger() *Ledger {
|
||||
return &Ledger{Version: LedgerVersion, Entries: map[string]LedgerEntry{}}
|
||||
}
|
||||
|
||||
// Add merges a quote into the ledger under its natural key.
|
||||
func (l *Ledger) Add(q CostQuote, tokens UsageTokens, occurred time.Time) {
|
||||
if l == nil {
|
||||
return
|
||||
}
|
||||
if l.Entries == nil {
|
||||
l.Entries = map[string]LedgerEntry{}
|
||||
}
|
||||
if l.Version == 0 {
|
||||
l.Version = LedgerVersion
|
||||
}
|
||||
if occurred.IsZero() {
|
||||
occurred = time.Now().UTC()
|
||||
}
|
||||
q = NormalizeQuote(q)
|
||||
source := q.UsageSource
|
||||
if source == "" {
|
||||
source = "executor"
|
||||
}
|
||||
key := LedgerKey(q.ModelRef, source, q.PricingFingerprint, q.RateDate)
|
||||
ent, ok := l.Entries[key]
|
||||
if !ok {
|
||||
ent = LedgerEntry{
|
||||
Key: key,
|
||||
ModelRef: q.ModelRef,
|
||||
UsageSource: source,
|
||||
PricingFingerprint: q.PricingFingerprint,
|
||||
RateDate: q.RateDate,
|
||||
OccurredAt: occurred,
|
||||
Quote: q,
|
||||
}
|
||||
// Fresh quote valuations are kept; we re-aggregate Original via sums.
|
||||
ent.Quote.Valuations = cloneValuations(q.Valuations)
|
||||
} else {
|
||||
// Sum original when same currency; otherwise retain deterministic
|
||||
// per-currency buckets. Once a bucketed entry exists, keep adding into
|
||||
// those buckets so later same-currency calls are not lost.
|
||||
if len(ent.Quote.OriginalTotals) > 0 {
|
||||
ent.Quote.OriginalTotals = mergeOriginalTotals(ent.Quote.OriginalTotals, q)
|
||||
ent.Quote.DisplayComplete = false
|
||||
ent.Quote.Complete = false
|
||||
ent.Quote.DisplayStatus = DisplayStatusBucketed
|
||||
ent.Quote.AggregateMode = AggregateModeCurrencyBuckets
|
||||
ent.Quote.IncompleteReason = "mixed_original_currencies"
|
||||
} else {
|
||||
sum, err := AddMoney(ent.Quote.Original, q.Original)
|
||||
if err != nil {
|
||||
ent.Quote.OriginalTotals = mergeOriginalTotals([]Money{ent.Quote.Original}, q)
|
||||
ent.Quote.DisplayComplete = false
|
||||
ent.Quote.Complete = false
|
||||
ent.Quote.DisplayStatus = DisplayStatusBucketed
|
||||
ent.Quote.AggregateMode = AggregateModeCurrencyBuckets
|
||||
ent.Quote.IncompleteReason = "mixed_original_currencies"
|
||||
} else {
|
||||
ent.Quote.Original = sum
|
||||
}
|
||||
}
|
||||
for code, v := range q.Valuations {
|
||||
code = NormalizeCurrency(code)
|
||||
if prev, ok := ent.Quote.Valuations[code]; ok {
|
||||
added, err := AddMoney(prev.Money, v.Money)
|
||||
if err == nil {
|
||||
prev.Money = added
|
||||
if v.Stale {
|
||||
prev.Stale = true
|
||||
}
|
||||
ent.Quote.Valuations[code] = prev
|
||||
}
|
||||
} else {
|
||||
if ent.Quote.Valuations == nil {
|
||||
ent.Quote.Valuations = map[string]Valuation{}
|
||||
}
|
||||
ent.Quote.Valuations[code] = v
|
||||
}
|
||||
}
|
||||
if q.Estimated {
|
||||
ent.Quote.Estimated = true
|
||||
}
|
||||
if !q.Complete {
|
||||
ent.Quote.DisplayComplete = false
|
||||
ent.Quote.Complete = false
|
||||
if ent.Quote.IncompleteReason == "" {
|
||||
ent.Quote.IncompleteReason = q.IncompleteReason
|
||||
}
|
||||
}
|
||||
ent.Quote.Coverage = FoldCoverage(ent.Quote.Coverage, q.Coverage)
|
||||
ent.Quote.CostComplete = ent.Quote.Coverage == CoverageComplete
|
||||
if ent.Quote.DisplayStatus != DisplayStatusBucketed {
|
||||
ent.Quote.DisplayStatus = q.DisplayStatus
|
||||
}
|
||||
}
|
||||
ent.PromptTokens += tokens.PromptTokens
|
||||
ent.CompletionTokens += tokens.CompletionTokens
|
||||
ent.TotalTokens += tokens.PromptTokens + tokens.CompletionTokens
|
||||
if tokens.CacheHitTokens+tokens.CacheMissTokens > 0 {
|
||||
ent.CacheHitTokens += tokens.CacheHitTokens
|
||||
ent.CacheMissTokens += tokens.CacheMissTokens
|
||||
} else {
|
||||
ent.CacheMissTokens += tokens.PromptTokens
|
||||
ent.CacheUnknownTokens += tokens.PromptTokens
|
||||
}
|
||||
ent.RequestCount++
|
||||
if occurred.After(ent.OccurredAt) {
|
||||
ent.OccurredAt = occurred
|
||||
}
|
||||
l.Entries[key] = ent
|
||||
}
|
||||
|
||||
func mergeOriginalTotals(existing []Money, q CostQuote) []Money {
|
||||
amounts := map[string]Amount{}
|
||||
add := func(m Money) {
|
||||
currency := NormalizeCurrency(m.Currency)
|
||||
if currency == "" {
|
||||
return
|
||||
}
|
||||
amounts[currency] = amounts[currency].Add(m.AmountValue())
|
||||
}
|
||||
for _, m := range existing {
|
||||
add(m)
|
||||
}
|
||||
if len(q.OriginalTotals) > 0 {
|
||||
for _, m := range q.OriginalTotals {
|
||||
add(m)
|
||||
}
|
||||
} else {
|
||||
add(q.Original)
|
||||
}
|
||||
codes := slices.Sorted(maps.Keys(amounts))
|
||||
out := make([]Money, 0, len(codes))
|
||||
for _, code := range codes {
|
||||
out = append(out, MoneyOf(amounts[code], code))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func cloneValuations(in map[string]Valuation) map[string]Valuation {
|
||||
if len(in) == 0 {
|
||||
return map[string]Valuation{}
|
||||
}
|
||||
out := make(map[string]Valuation, len(in))
|
||||
maps.Copy(out, in)
|
||||
return out
|
||||
}
|
||||
|
||||
// Total returns the aggregate CostQuote for a display currency.
|
||||
func (l *Ledger) Total(display string) CostQuote {
|
||||
if l == nil || len(l.Entries) == 0 {
|
||||
return AggregateQuotes(nil, display)
|
||||
}
|
||||
quotes := make([]CostQuote, 0, len(l.Entries))
|
||||
for _, ent := range l.Entries {
|
||||
q := ent.Quote
|
||||
// Refresh valuation moneys already summed in entries.
|
||||
quotes = append(quotes, q)
|
||||
}
|
||||
// Stable order for determinism.
|
||||
sort.Slice(quotes, func(i, j int) bool {
|
||||
return quotes[i].ModelRef+quotes[i].UsageSource < quotes[j].ModelRef+quotes[j].UsageSource
|
||||
})
|
||||
return AggregateQuotes(quotes, display)
|
||||
}
|
||||
|
||||
// SelectDisplay rebinds Selected on every entry and the total without
|
||||
// recomputing occurrence-time pricing facts.
|
||||
func (l *Ledger) SelectDisplay(display string) CostQuote {
|
||||
return l.Total(display)
|
||||
}
|
||||
|
||||
// EntriesBySource groups ledger entries by usage source for status panels.
|
||||
func (l *Ledger) EntriesBySource() map[string][]LedgerEntry {
|
||||
out := map[string][]LedgerEntry{}
|
||||
if l == nil {
|
||||
return out
|
||||
}
|
||||
for _, ent := range l.Entries {
|
||||
out[ent.UsageSource] = append(out[ent.UsageSource], ent)
|
||||
}
|
||||
return out
|
||||
}
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// LegacyUsageRecord is the minimum shape needed to migrate old telemetry.
|
||||
type LegacyUsageRecord struct {
|
||||
SessionCost float64
|
||||
SessionCurrency string
|
||||
// EndedAt is retained for historical ordering and identity valuation dates.
|
||||
EndedAt time.Time
|
||||
// Tokens optional for ledger completeness.
|
||||
PromptTokens int
|
||||
CompletionTokens int
|
||||
ModelRef string
|
||||
UsageSource string
|
||||
}
|
||||
|
||||
// MigrateLegacyUsage builds a CostQuote from pre-CostQuote telemetry.
|
||||
// Records that cannot be valued are marked legacy_estimate; wiped mixed-currency
|
||||
// zeros are not reconstructed from current price tables.
|
||||
func MigrateLegacyUsage(rec LegacyUsageRecord) CostQuote {
|
||||
cur := NormalizeCurrency(rec.SessionCurrency)
|
||||
if cur == "" {
|
||||
// Historic defaults often used ¥ symbol without code.
|
||||
if strings.Contains(rec.SessionCurrency, "¥") || strings.Contains(rec.SessionCurrency, "￥") {
|
||||
cur = "CNY"
|
||||
} else if strings.Contains(rec.SessionCurrency, "$") {
|
||||
cur = "USD"
|
||||
}
|
||||
}
|
||||
// Wiped mixed-currency totals (legacy code zeroed them) and empty rows must
|
||||
// not be reconstructed from current price tables.
|
||||
if rec.SessionCost <= 0 || cur == "" {
|
||||
reason := "legacy_unrecoverable"
|
||||
if rec.SessionCost == 0 && strings.TrimSpace(rec.SessionCurrency) != "" {
|
||||
// Explicit zero with a currency often means a prior mixed-currency wipe.
|
||||
reason = "legacy_wiped_or_zero"
|
||||
} else if rec.SessionCost < 0 {
|
||||
reason = "legacy_invalid_amount"
|
||||
}
|
||||
return CostQuote{
|
||||
Original: MoneyOf(Zero, cur),
|
||||
Estimated: true,
|
||||
CostComplete: false,
|
||||
Coverage: CoverageIncomplete,
|
||||
DisplayComplete: false,
|
||||
Complete: false,
|
||||
DisplayStatus: DisplayStatusUnavailable,
|
||||
LegacyEstimate: true,
|
||||
IncompleteReason: reason,
|
||||
ModelRef: rec.ModelRef,
|
||||
UsageSource: rec.UsageSource,
|
||||
}
|
||||
}
|
||||
amount := NewAmountFromFloat(rec.SessionCost)
|
||||
q := CostQuote{
|
||||
Original: MoneyOf(amount, cur),
|
||||
Valuations: map[string]Valuation{},
|
||||
Estimated: true,
|
||||
CostComplete: true,
|
||||
Coverage: CoverageComplete,
|
||||
DisplayComplete: true,
|
||||
Complete: true,
|
||||
DisplayStatus: DisplayStatusMatched,
|
||||
AggregateMode: AggregateModeSingleCurrency,
|
||||
LegacyEstimate: true,
|
||||
ModelRef: rec.ModelRef,
|
||||
UsageSource: rec.UsageSource,
|
||||
BillingMode: BillingModePAYG,
|
||||
}
|
||||
ended := rec.EndedAt
|
||||
if ended.IsZero() {
|
||||
ended = time.Now().UTC()
|
||||
}
|
||||
asOf := ended.UTC().Format("2006-01-02")
|
||||
q.Valuations[cur] = Valuation{
|
||||
Money: q.Original,
|
||||
Basis: BasisIdentity,
|
||||
Source: "legacy_telemetry",
|
||||
AsOf: asOf,
|
||||
}
|
||||
m := q.Original
|
||||
q.Selected = &m
|
||||
return q
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
package pricing
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestMigrateLegacyUsageIdentityOnly(t *testing.T) {
|
||||
q := MigrateLegacyUsage(LegacyUsageRecord{SessionCost: 1.25, SessionCurrency: "USD"})
|
||||
if !q.CostComplete || !q.DisplayComplete || !q.Complete || q.Selected == nil {
|
||||
t.Fatalf("migrated quote = %+v", q)
|
||||
}
|
||||
if len(q.Valuations) != 1 || q.Valuations["USD"].Basis != BasisIdentity {
|
||||
t.Fatalf("migrated valuations = %+v", q.Valuations)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMigrateLegacyWipedZeroDoesNotGuess(t *testing.T) {
|
||||
q := MigrateLegacyUsage(LegacyUsageRecord{SessionCurrency: "CNY"})
|
||||
if q.CostComplete || q.Complete || q.DisplayStatus != DisplayStatusUnavailable || q.IncompleteReason != "legacy_wiped_or_zero" {
|
||||
t.Fatalf("wiped quote = %+v", q)
|
||||
}
|
||||
}
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
package pricing
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestMimoV26RatesFollowTheVendorPage(t *testing.T) {
|
||||
cases := []struct {
|
||||
model string
|
||||
hit, miss, out float64
|
||||
v25Twin string
|
||||
}{
|
||||
{"mimo-v2.6-pro", 0.025, 3, 6, "mimo-v2.5-pro"},
|
||||
{"mimo-v2.6-flash", 0.02, 1, 2, "mimo-v2.5"},
|
||||
{"mimo-v2.6-pro-ultraspeed", 0.25, 30, 60, ""},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
got := CurrentRate("mimo", tc.model, "CNY")
|
||||
if got == nil || got.CacheHit != tc.hit || got.Input != tc.miss || got.Output != tc.out {
|
||||
t.Fatalf("%s rate = %+v, want %v/%v/%v", tc.model, got, tc.hit, tc.miss, tc.out)
|
||||
}
|
||||
if tc.v25Twin == "" {
|
||||
continue
|
||||
}
|
||||
old := CurrentRate("mimo", tc.v25Twin, "CNY")
|
||||
if old == nil || *old != *got {
|
||||
t.Fatalf("%s must keep its published rate beside %s: %+v vs %+v", tc.v25Twin, tc.model, old, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
+246
@@ -0,0 +1,246 @@
|
||||
// Package pricing is what a model call costs: fixed-point money, the official
|
||||
// rate tables, and the quote a usage record is priced by. It is data and
|
||||
// arithmetic only; asking a provider for its wallet balance is model/billing's.
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"math/big"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Money is a currency-tagged amount carried as a decimal string on the wire.
|
||||
// Arithmetic uses fixed-point int64 units (1e9 fractional digits).
|
||||
type Money struct {
|
||||
Amount string `json:"amount"`
|
||||
Currency string `json:"currency"`
|
||||
}
|
||||
|
||||
// amountScale is 1e9: enough for per-token rates down to ~1e-15 currency units
|
||||
// when aggregated over typical request sizes.
|
||||
const amountScale int64 = 1_000_000_000
|
||||
|
||||
// Amount is a fixed-point currency magnitude in 1e-9 units of the currency.
|
||||
type Amount int64
|
||||
|
||||
// Zero is the zero amount.
|
||||
const Zero Amount = 0
|
||||
|
||||
// NewAmountFromFloat converts a float64 amount to fixed-point. Prefer ParseAmount
|
||||
// or NewAmountFromString for values that originate as decimals.
|
||||
func NewAmountFromFloat(v float64) Amount {
|
||||
if math.IsNaN(v) || math.IsInf(v, 0) {
|
||||
return Zero
|
||||
}
|
||||
r := new(big.Rat).SetFloat64(v)
|
||||
if r == nil {
|
||||
return Zero
|
||||
}
|
||||
return amountFromRat(r)
|
||||
}
|
||||
|
||||
// NewAmountFromString parses a decimal amount string.
|
||||
func NewAmountFromString(s string) (Amount, error) {
|
||||
s = strings.TrimSpace(s)
|
||||
if s == "" {
|
||||
return Zero, nil
|
||||
}
|
||||
r := new(big.Rat)
|
||||
if _, ok := r.SetString(s); !ok {
|
||||
return Zero, fmt.Errorf("billing: invalid amount %q", s)
|
||||
}
|
||||
return amountFromRat(r), nil
|
||||
}
|
||||
|
||||
// ParseAmount is an alias for NewAmountFromString.
|
||||
func ParseAmount(s string) (Amount, error) { return NewAmountFromString(s) }
|
||||
|
||||
func amountFromRat(r *big.Rat) Amount {
|
||||
if r == nil {
|
||||
return Zero
|
||||
}
|
||||
scaled := new(big.Rat).Mul(r, big.NewRat(amountScale, 1))
|
||||
// Round half away from zero.
|
||||
num := new(big.Int).Set(scaled.Num())
|
||||
den := new(big.Int).Set(scaled.Denom())
|
||||
half := new(big.Int).Rsh(den, 1)
|
||||
if scaled.Sign() >= 0 {
|
||||
num.Add(num, half)
|
||||
} else {
|
||||
num.Sub(num, half)
|
||||
}
|
||||
quot := new(big.Int).Quo(num, den)
|
||||
if !quot.IsInt64() {
|
||||
if scaled.Sign() >= 0 {
|
||||
return Amount(math.MaxInt64)
|
||||
}
|
||||
return Amount(math.MinInt64)
|
||||
}
|
||||
return Amount(quot.Int64())
|
||||
}
|
||||
|
||||
// String returns a trimmed decimal representation.
|
||||
func (a Amount) String() string {
|
||||
neg := a < 0
|
||||
v := int64(a)
|
||||
if neg {
|
||||
v = -v
|
||||
}
|
||||
whole := v / amountScale
|
||||
frac := v % amountScale
|
||||
s := fmt.Sprintf("%d.%09d", whole, frac)
|
||||
s = strings.TrimRight(s, "0")
|
||||
s = strings.TrimRight(s, ".")
|
||||
if s == "" || s == "-" {
|
||||
s = "0"
|
||||
}
|
||||
if neg && s != "0" {
|
||||
return "-" + s
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Float64 approximates the amount for legacy float fields only.
|
||||
func (a Amount) Float64() float64 {
|
||||
return float64(a) / float64(amountScale)
|
||||
}
|
||||
|
||||
// Add returns a+b. Same-currency only at the Money layer.
|
||||
func (a Amount) Add(b Amount) Amount {
|
||||
sum := int64(a) + int64(b)
|
||||
// Saturate on overflow rather than wrap.
|
||||
if (b > 0 && sum < int64(a)) || (b < 0 && sum > int64(a)) {
|
||||
if b > 0 {
|
||||
return Amount(math.MaxInt64)
|
||||
}
|
||||
return Amount(math.MinInt64)
|
||||
}
|
||||
return Amount(sum)
|
||||
}
|
||||
|
||||
// MulRate multiplies by a float rate for a pricing calculation.
|
||||
func (a Amount) MulRate(rate float64) Amount {
|
||||
if a == 0 || rate == 0 || math.IsNaN(rate) || math.IsInf(rate, 0) {
|
||||
return Zero
|
||||
}
|
||||
r := new(big.Rat).SetFloat64(rate)
|
||||
if r == nil {
|
||||
return Zero
|
||||
}
|
||||
ar := big.NewRat(int64(a), amountScale)
|
||||
return amountFromRat(ar.Mul(ar, r))
|
||||
}
|
||||
|
||||
// MoneyOf builds a Money value.
|
||||
func MoneyOf(amount Amount, currency string) Money {
|
||||
return Money{Amount: amount.String(), Currency: NormalizeCurrency(currency)}
|
||||
}
|
||||
|
||||
// AmountValue returns the fixed-point amount, or zero on parse error.
|
||||
func (m Money) AmountValue() Amount {
|
||||
a, err := NewAmountFromString(m.Amount)
|
||||
if err != nil {
|
||||
return Zero
|
||||
}
|
||||
return a
|
||||
}
|
||||
|
||||
// IsZero reports whether the money amount is empty or zero.
|
||||
func (m Money) IsZero() bool {
|
||||
return m.AmountValue() == Zero
|
||||
}
|
||||
|
||||
// Float64 is a legacy adapter. Prefer Amount strings for aggregation.
|
||||
func (m Money) Float64() float64 { return m.AmountValue().Float64() }
|
||||
|
||||
// NormalizeCurrency maps symbols and aliases to ISO-4217 codes when known.
|
||||
// Unknown three-letter codes pass through uppercased; empty stays empty.
|
||||
func NormalizeCurrency(currency string) string {
|
||||
value := strings.TrimSpace(currency)
|
||||
if value == "" {
|
||||
return ""
|
||||
}
|
||||
switch strings.ToUpper(value) {
|
||||
case "CNY", "RMB", "CNH", "YUAN", "RENMINBI":
|
||||
return "CNY"
|
||||
case "USD", "US$", "DOLLAR", "DOLLARS":
|
||||
return "USD"
|
||||
case "EUR", "EURO", "EUROS":
|
||||
return "EUR"
|
||||
case "GBP", "POUND", "POUNDS", "STERLING":
|
||||
return "GBP"
|
||||
case "JPY", "YEN":
|
||||
return "JPY"
|
||||
}
|
||||
switch value {
|
||||
case "¥", "￥":
|
||||
return "CNY"
|
||||
case "$":
|
||||
return "USD"
|
||||
case "€":
|
||||
return "EUR"
|
||||
case "£":
|
||||
return "GBP"
|
||||
}
|
||||
if len(value) == 3 {
|
||||
allAlpha := true
|
||||
for _, r := range value {
|
||||
if (r < 'a' || r > 'z') && (r < 'A' || r > 'Z') {
|
||||
allAlpha = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if allAlpha {
|
||||
return strings.ToUpper(value)
|
||||
}
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
// CurrencySymbol returns a compact display symbol for an ISO code or symbol.
|
||||
func CurrencySymbol(currency string) string {
|
||||
switch NormalizeCurrency(currency) {
|
||||
case "CNY", "JPY":
|
||||
return "¥"
|
||||
case "USD":
|
||||
return "$"
|
||||
case "EUR":
|
||||
return "€"
|
||||
case "GBP":
|
||||
return "£"
|
||||
case "":
|
||||
return "¥"
|
||||
default:
|
||||
code := NormalizeCurrency(currency)
|
||||
if len(code) == 3 {
|
||||
return code + " "
|
||||
}
|
||||
return currency
|
||||
}
|
||||
}
|
||||
|
||||
// SameCurrency reports whether two codes normalize equal and non-empty.
|
||||
func SameCurrency(a, b string) bool {
|
||||
na, nb := NormalizeCurrency(a), NormalizeCurrency(b)
|
||||
return na != "" && na == nb
|
||||
}
|
||||
|
||||
// AddMoney adds two Money values of the same currency. Mixed currencies error.
|
||||
func AddMoney(a, b Money) (Money, error) {
|
||||
ca, cb := NormalizeCurrency(a.Currency), NormalizeCurrency(b.Currency)
|
||||
if ca == "" {
|
||||
ca = cb
|
||||
}
|
||||
if cb == "" {
|
||||
cb = ca
|
||||
}
|
||||
if ca == "" && cb == "" {
|
||||
return Money{Amount: "0", Currency: ""}, nil
|
||||
}
|
||||
if ca != cb {
|
||||
return Money{}, fmt.Errorf("billing: cannot add %s and %s", ca, cb)
|
||||
}
|
||||
sum := a.AmountValue().Add(b.AmountValue())
|
||||
return MoneyOf(sum, ca), nil
|
||||
}
|
||||
+56
@@ -0,0 +1,56 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestAmountRoundTrip(t *testing.T) {
|
||||
a, err := NewAmountFromString("1.23456789")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := a.String(); got != "1.23456789" {
|
||||
t.Fatalf("String = %q", got)
|
||||
}
|
||||
if a.Float64() < 1.23456788 || a.Float64() > 1.23456790 {
|
||||
t.Fatalf("Float64 = %v", a.Float64())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddMoneyRejectsMixedCurrency(t *testing.T) {
|
||||
_, err := AddMoney(MoneyOf(NewAmountFromFloat(1), "CNY"), MoneyOf(NewAmountFromFloat(1), "USD"))
|
||||
if err == nil {
|
||||
t.Fatal("expected mixed currency error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddMoneySameCurrency(t *testing.T) {
|
||||
sum, err := AddMoney(MoneyOf(NewAmountFromFloat(1.5), "USD"), MoneyOf(NewAmountFromFloat(2.25), "USD"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if sum.Currency != "USD" || sum.Amount != "3.75" {
|
||||
t.Fatalf("sum = %+v", sum)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeCurrency(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"¥": "CNY", "$": "USD", "rmb": "CNY", "usd": "USD", "eur": "EUR",
|
||||
}
|
||||
for in, want := range cases {
|
||||
if got := NormalizeCurrency(in); got != want {
|
||||
t.Errorf("NormalizeCurrency(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestOriginalCostMatchesLegacySemantics(t *testing.T) {
|
||||
// Same as provider.Pricing.Cost cache-write split test: miss 500k, write 100k billed 200k, input rate 2 → 1.2
|
||||
amt := OriginalCostAmount(RateCard{Input: 2, Currency: "USD"}, UsageTokens{
|
||||
CacheMissTokens: 500_000, CacheWriteTokens: 100_000, CacheWriteBilledTokens: 200_000,
|
||||
})
|
||||
if got := amt.Float64(); got < 1.199 || got > 1.201 {
|
||||
t.Fatalf("cost = %v, want ~1.2", got)
|
||||
}
|
||||
}
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
package pricing
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestIsOfficialCardRecognisesEveryGenerationWeShipped(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
model string
|
||||
card RateCard
|
||||
want bool
|
||||
}{
|
||||
{"current base", "deepseek-flash", RateCard{CacheHit: 0.003, Input: 0.15, Output: 0.6, Currency: "USD"}, true},
|
||||
{"current peak", "deepseek-flash", RateCard{CacheHit: 0.006, Input: 0.3, Output: 1.2, Currency: "USD"}, true},
|
||||
{"superseded base", "deepseek-flash", RateCard{CacheHit: 0.0028, Input: 0.14, Output: 0.28, Currency: "USD"}, true},
|
||||
{"superseded peak", "deepseek-flash", RateCard{CacheHit: 0.014, Input: 0.44, Output: 1.32, Currency: "USD"}, true},
|
||||
{"retired name prices as flash", "deepseek-v4-flash", RateCard{CacheHit: 0.003, Input: 0.15, Output: 0.6, Currency: "USD"}, true},
|
||||
{"cny card", "deepseek-flash", RateCard{CacheHit: 0.02, Input: 1, Output: 4, Currency: "CNY"}, true},
|
||||
{"peak numbers with the cached price missing", "deepseek-flash", RateCard{CacheHit: 0, Input: 0.3, Output: 1.2, Currency: "USD"}, false},
|
||||
{"cny numbers labelled usd", "deepseek-flash", RateCard{CacheHit: 0.02, Input: 1, Output: 4, Currency: "USD"}, false},
|
||||
{"a model the book does not price", "deepseek-unknown", RateCard{CacheHit: 0.003, Input: 0.15, Output: 0.6, Currency: "USD"}, false},
|
||||
} {
|
||||
if got := IsOfficialCard("deepseek", tc.model, tc.card); got != tc.want {
|
||||
t.Errorf("%s: IsOfficialCard = %v, want %v", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
if IsOfficialCard("someone-else", "deepseek-flash", RateCard{CacheHit: 0.003, Input: 0.15, Output: 0.6, Currency: "USD"}) {
|
||||
t.Error("another vendor's name matched DeepSeek's table")
|
||||
}
|
||||
}
|
||||
+790
@@ -0,0 +1,790 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Billing modes.
|
||||
const (
|
||||
BillingModePAYG = "payg"
|
||||
BillingModeSubscriptionEquivalent = "subscription_equivalent"
|
||||
)
|
||||
|
||||
// Valuation basis values.
|
||||
const (
|
||||
BasisIdentity = "identity"
|
||||
BasisOfficialTable = "official_table"
|
||||
// BasisFX is retained for decoding pre-v6 persisted quotes only. New
|
||||
// quotes never create FX valuations.
|
||||
BasisFX = "fx"
|
||||
)
|
||||
|
||||
// DisplayStatus describes whether a quote can satisfy its requested display.
|
||||
// It is deliberately separate from CostComplete: a known price-book amount
|
||||
// can exist even when no requested regional valuation is available.
|
||||
const (
|
||||
DisplayStatusMatched = "matched"
|
||||
DisplayStatusFallbackOriginal = "fallback_original"
|
||||
DisplayStatusBucketed = "bucketed"
|
||||
DisplayStatusUnavailable = "unavailable"
|
||||
)
|
||||
|
||||
// Aggregate modes describe how a session/run total was formed.
|
||||
const (
|
||||
AggregateModeSingleCurrency = "single_currency"
|
||||
AggregateModeCommonValuation = "common_valuation"
|
||||
AggregateModeCurrencyBuckets = "currency_buckets"
|
||||
)
|
||||
|
||||
// Rate bands name which side of a vendor's peak window a call was billed on.
|
||||
// An aggregate is mixed when its calls fell on both sides.
|
||||
const (
|
||||
RateBandPeak = "peak"
|
||||
RateBandOffPeak = "off_peak"
|
||||
RateBandMixed = "mixed"
|
||||
)
|
||||
|
||||
// DisplayRequest identifies how a caller selected the requested presentation
|
||||
// currency. The source is runtime context only; it is never persisted as a
|
||||
// provider price or wallet fact.
|
||||
type DisplayRequest struct {
|
||||
Currency string `json:"currency,omitempty"`
|
||||
Source string `json:"source,omitempty"`
|
||||
}
|
||||
|
||||
const (
|
||||
DisplaySourceExplicit = "explicit"
|
||||
DisplaySourceWallet = "wallet"
|
||||
DisplaySourceAuto = "auto"
|
||||
)
|
||||
|
||||
func (r DisplayRequest) NormalizedCurrency() string {
|
||||
return NormalizeCurrency(r.Currency)
|
||||
}
|
||||
|
||||
// CostQuote is the host-side quoted cost for one model call (or an aggregate).
|
||||
// Original is the pricing-table currency fact; valuations hold identity or
|
||||
// official regional rate-card estimates. Selected is the display pick for the
|
||||
// current preference.
|
||||
type CostQuote struct {
|
||||
Original Money `json:"original"`
|
||||
// OriginalTotals is populated for mixed-currency aggregates. Individual
|
||||
// quotes and single-currency totals may omit it.
|
||||
OriginalTotals []Money `json:"originalTotals,omitempty"`
|
||||
Valuations map[string]Valuation `json:"valuations,omitempty"`
|
||||
Selected *Money `json:"selected,omitempty"`
|
||||
BillingMode string `json:"billingMode,omitempty"`
|
||||
Estimated bool `json:"estimated"`
|
||||
// CostComplete means usage and the price-book amount are known. It does not
|
||||
// imply that a requested display currency is available.
|
||||
CostComplete bool `json:"costComplete"`
|
||||
// Coverage is the same fact with the states a boolean cannot hold: an
|
||||
// aggregate that priced only some of its calls, and one that billed nothing.
|
||||
Coverage string `json:"coverage,omitempty"`
|
||||
// DisplayComplete means a single selected amount satisfies the display
|
||||
// request. Complete is retained as the old wire alias.
|
||||
DisplayComplete bool `json:"displayComplete"`
|
||||
Complete bool `json:"complete"`
|
||||
DisplayStatus string `json:"displayStatus,omitempty"`
|
||||
AggregateMode string `json:"aggregateMode,omitempty"`
|
||||
// ModelRef / UsageSource / PricingFingerprint identify the ledger key.
|
||||
ModelRef string `json:"modelRef,omitempty"`
|
||||
UsageSource string `json:"usageSource,omitempty"`
|
||||
PricingFingerprint string `json:"pricingFingerprint,omitempty"`
|
||||
RateDate string `json:"rateDate,omitempty"` // YYYY-MM-DD of FX used, if any
|
||||
IncompleteReason string `json:"incompleteReason,omitempty"`
|
||||
LegacyEstimate bool `json:"legacyEstimate,omitempty"`
|
||||
CatalogSource string `json:"catalogSource,omitempty"`
|
||||
// RateBand is set only for a rate confirmed as a vendor's own schedule; an
|
||||
// aggregate that includes any call without one carries none.
|
||||
RateBand string `json:"rateBand,omitempty"`
|
||||
}
|
||||
|
||||
// Valuation is one currency view of a cost fact.
|
||||
type Valuation struct {
|
||||
Money Money `json:"money"`
|
||||
Basis string `json:"basis"`
|
||||
Source string `json:"source"`
|
||||
AsOf string `json:"asOf"` // YYYY-MM-DD
|
||||
Rate *RateSnapshot `json:"rateSnapshot,omitempty"`
|
||||
Stale bool `json:"stale,omitempty"`
|
||||
}
|
||||
|
||||
// RateSnapshot records the FX observation used for a valuation.
|
||||
type RateSnapshot struct {
|
||||
Base string `json:"base"`
|
||||
Quote string `json:"quote"`
|
||||
Rate float64 `json:"rate"`
|
||||
Source string `json:"source"`
|
||||
AsOf string `json:"asOf"`
|
||||
Stale bool `json:"stale,omitempty"`
|
||||
}
|
||||
|
||||
// RateCard is the per-1M-token price used to compute original cost. It mirrors
|
||||
// provider.Pricing without importing that package (billing is a leaf).
|
||||
type RateCard struct {
|
||||
CacheHit float64 // per 1M cached prompt tokens
|
||||
Input float64 // per 1M uncached prompt tokens
|
||||
Output float64 // per 1M completion tokens
|
||||
Currency string // ISO or symbol; normalized on quote
|
||||
}
|
||||
|
||||
// UsageTokens is the token breakdown needed for cost. Mirrors provider.Usage
|
||||
// fields without importing provider.
|
||||
type UsageTokens struct {
|
||||
PromptTokens int
|
||||
CompletionTokens int
|
||||
CacheHitTokens int
|
||||
CacheMissTokens int
|
||||
CacheWriteTokens int
|
||||
CacheWriteBilledTokens float64
|
||||
Estimated bool
|
||||
}
|
||||
|
||||
// QuoteInput drives a single CostQuote computation.
|
||||
type QuoteInput struct {
|
||||
Usage UsageTokens
|
||||
Rates RateCard
|
||||
OccurredAt time.Time
|
||||
DisplayCurrency string // compatibility alias for Display.Currency
|
||||
Display DisplayRequest // runtime display request; empty means auto
|
||||
BillingMode string
|
||||
ModelRef string
|
||||
UsageSource string
|
||||
CatalogSource string
|
||||
PricingFingerprint string
|
||||
// ProviderKind is deepseek|longcat|mimo|… for official dual-table lookup.
|
||||
// When empty, ModelRef and Rates are used to infer a catalog match.
|
||||
ProviderKind string
|
||||
// ModelID is the bare model id (e.g. deepseek-v4-flash). Empty → parse ModelRef.
|
||||
ModelID string
|
||||
}
|
||||
|
||||
// OriginalCostAmount computes the fixed-point original cost from rates + tokens.
|
||||
// Mirrors the historic provider.Pricing.Cost semantics.
|
||||
func OriginalCostAmount(rates RateCard, u UsageTokens) Amount {
|
||||
hit := u.CacheHitTokens
|
||||
miss := u.CacheMissTokens
|
||||
if hit+miss == 0 && u.PromptTokens > 0 {
|
||||
miss = u.PromptTokens
|
||||
} else if miss == 0 && hit > 0 && u.PromptTokens > hit {
|
||||
miss = u.PromptTokens - hit
|
||||
}
|
||||
write := u.CacheWriteTokens
|
||||
write = max(write, 0)
|
||||
write = min(write, miss)
|
||||
billedWrite := 0.0
|
||||
if write > 0 {
|
||||
billedWrite = u.CacheWriteBilledTokens
|
||||
if billedWrite <= 0 {
|
||||
billedWrite = float64(write)
|
||||
}
|
||||
}
|
||||
inputTokenUnits := float64(miss-write) + billedWrite
|
||||
// Combine cached input, uncached input, and output charges per million tokens.
|
||||
total := (float64(hit)*rates.CacheHit +
|
||||
inputTokenUnits*rates.Input +
|
||||
float64(u.CompletionTokens)*rates.Output) / 1e6
|
||||
return NewAmountFromFloat(total)
|
||||
}
|
||||
|
||||
// PricingFingerprint hashes a rate card for ledger keys.
|
||||
func PricingFingerprint(rates RateCard) string {
|
||||
cur := NormalizeCurrency(rates.Currency)
|
||||
raw := fmt.Sprintf("%s|%.12g|%.12g|%.12g", cur, rates.CacheHit, rates.Input, rates.Output)
|
||||
sum := sha256.Sum256([]byte(raw))
|
||||
return hex.EncodeToString(sum[:8])
|
||||
}
|
||||
|
||||
// BuildQuote computes original cost and CNY/USD valuations.
|
||||
func BuildQuote(in QuoteInput) CostQuote {
|
||||
state := newQuoteBuildState(in)
|
||||
for _, target := range []string{"CNY", "USD"} {
|
||||
state.addTargetValuation(target)
|
||||
}
|
||||
state.selectDisplay()
|
||||
return state.quote
|
||||
}
|
||||
|
||||
type quoteBuildState struct {
|
||||
input QuoteInput
|
||||
currency string
|
||||
amount Amount
|
||||
occurred time.Time
|
||||
official CatalogEntry
|
||||
isOfficial bool
|
||||
valuationFailures map[string]string
|
||||
quote CostQuote
|
||||
}
|
||||
|
||||
func newQuoteBuildState(in QuoteInput) *quoteBuildState {
|
||||
currency := NormalizeCurrency(in.Rates.Currency)
|
||||
if currency == "" {
|
||||
currency = "CNY"
|
||||
}
|
||||
mode := strings.TrimSpace(in.BillingMode)
|
||||
if mode == "" {
|
||||
mode = BillingModePAYG
|
||||
}
|
||||
occurred := in.OccurredAt
|
||||
if occurred.IsZero() {
|
||||
occurred = time.Now().UTC()
|
||||
}
|
||||
fingerprint := strings.TrimSpace(in.PricingFingerprint)
|
||||
if fingerprint == "" {
|
||||
fingerprint = PricingFingerprint(in.Rates)
|
||||
}
|
||||
amount := OriginalCostAmount(in.Rates, in.Usage)
|
||||
q := CostQuote{
|
||||
Original: MoneyOf(amount, currency),
|
||||
Valuations: map[string]Valuation{},
|
||||
BillingMode: mode,
|
||||
Estimated: true,
|
||||
CostComplete: true,
|
||||
Coverage: CoverageComplete,
|
||||
DisplayComplete: true,
|
||||
Complete: true,
|
||||
DisplayStatus: DisplayStatusMatched,
|
||||
AggregateMode: AggregateModeSingleCurrency,
|
||||
ModelRef: strings.TrimSpace(in.ModelRef),
|
||||
UsageSource: strings.TrimSpace(in.UsageSource),
|
||||
PricingFingerprint: fingerprint,
|
||||
CatalogSource: strings.TrimSpace(in.CatalogSource),
|
||||
}
|
||||
q.Valuations[currency] = Valuation{
|
||||
Money: q.Original, Basis: BasisIdentity,
|
||||
Source: firstNonEmpty(in.CatalogSource, "rate_card"),
|
||||
AsOf: occurred.UTC().Format("2006-01-02"),
|
||||
}
|
||||
state := "eBuildState{
|
||||
input: in, currency: currency, amount: amount, occurred: occurred,
|
||||
valuationFailures: map[string]string{}, quote: q,
|
||||
}
|
||||
if !usageHasFacts(in.Usage) {
|
||||
state.markIncomplete("missing_price_or_usage")
|
||||
}
|
||||
state.matchOfficialCatalog()
|
||||
state.applyOfficialSchedule()
|
||||
return state
|
||||
}
|
||||
|
||||
// applyOfficialSchedule re-prices the turn at the vendor's rate for the hour it
|
||||
// actually happened. Only a rate confirmed as the vendor's own earns this: a
|
||||
// custom price is the user's own arrangement, and projecting someone else's
|
||||
// peak schedule onto it would invent a number nobody published.
|
||||
func (s *quoteBuildState) applyOfficialSchedule() {
|
||||
if !s.isOfficial || s.official.Peak == nil {
|
||||
return
|
||||
}
|
||||
s.quote.RateBand = RateBandOffPeak
|
||||
if s.official.Window.IsPeak(s.occurred) {
|
||||
s.quote.RateBand = RateBandPeak
|
||||
}
|
||||
rates := s.official.RatesAt(s.occurred)
|
||||
if rates.CacheHit == s.input.Rates.CacheHit && rates.Input == s.input.Rates.Input && rates.Output == s.input.Rates.Output {
|
||||
return
|
||||
}
|
||||
s.amount = OriginalCostAmount(rates, s.input.Usage)
|
||||
s.quote.Original = MoneyOf(s.amount, s.currency)
|
||||
s.quote.PricingFingerprint = PricingFingerprint(rates)
|
||||
s.quote.Valuations[s.currency] = Valuation{
|
||||
Money: s.quote.Original, Basis: BasisOfficialTable,
|
||||
Source: s.official.DocURL,
|
||||
AsOf: s.occurred.UTC().Format("2006-01-02"),
|
||||
}
|
||||
}
|
||||
|
||||
func quoteDisplayRequest(in QuoteInput) DisplayRequest {
|
||||
request := in.Display
|
||||
if request.Currency == "" && in.DisplayCurrency != "" {
|
||||
request.Currency = in.DisplayCurrency
|
||||
if request.Source == "" {
|
||||
request.Source = DisplaySourceExplicit
|
||||
}
|
||||
}
|
||||
if request.Source == "" {
|
||||
request.Source = DisplaySourceAuto
|
||||
}
|
||||
return request
|
||||
}
|
||||
|
||||
func usageHasFacts(u UsageTokens) bool {
|
||||
return u.PromptTokens > 0 || u.CompletionTokens > 0 || u.CacheHitTokens > 0 ||
|
||||
u.CacheMissTokens > 0 || u.CacheWriteTokens > 0 || u.CacheWriteBilledTokens > 0
|
||||
}
|
||||
|
||||
func (s *quoteBuildState) matchOfficialCatalog() {
|
||||
providerKind, modelID := resolveCatalogIdentity(s.input)
|
||||
s.official, s.isOfficial = MatchesCatalog(providerKind, modelID, s.input.Rates)
|
||||
if !s.isOfficial && providerKind != "" && modelID != "" {
|
||||
rates := s.input.Rates
|
||||
rates.Currency = s.currency
|
||||
s.official, s.isOfficial = MatchesCatalog(providerKind, modelID, rates)
|
||||
}
|
||||
if s.isOfficial && s.quote.CatalogSource == "" {
|
||||
s.quote.CatalogSource = s.official.DocURL
|
||||
}
|
||||
}
|
||||
|
||||
func (s *quoteBuildState) addTargetValuation(target string) {
|
||||
if target == s.currency {
|
||||
return
|
||||
}
|
||||
if s.addOfficialValuation(target) {
|
||||
return
|
||||
}
|
||||
// Runtime FX has intentionally been removed. A custom price that does not
|
||||
// match the official catalog remains in its original currency.
|
||||
s.valuationFailures[target] = "display_unavailable"
|
||||
}
|
||||
|
||||
func (s *quoteBuildState) addOfficialValuation(target string) bool {
|
||||
if !s.isOfficial {
|
||||
return false
|
||||
}
|
||||
peer, ok := LookupCatalog(
|
||||
s.official.Provider, s.official.Model, target, s.official.BillingMode,
|
||||
)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
s.quote.Valuations[target] = Valuation{
|
||||
Money: MoneyOf(OriginalCostAmount(peer.RatesAt(s.occurred), s.input.Usage), target),
|
||||
Basis: BasisOfficialTable, Source: peer.DocURL,
|
||||
AsOf: s.occurred.UTC().Format("2006-01-02"),
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *quoteBuildState) selectDisplay() {
|
||||
if !s.quote.CostComplete {
|
||||
s.quote.Selected = nil
|
||||
s.quote.DisplayComplete = false
|
||||
s.quote.Complete = false
|
||||
s.quote.DisplayStatus = DisplayStatusUnavailable
|
||||
return
|
||||
}
|
||||
display := quoteDisplayRequest(s.input).NormalizedCurrency()
|
||||
if display == "" {
|
||||
selected := s.quote.Original
|
||||
s.quote.Selected = &selected
|
||||
return
|
||||
}
|
||||
if valuation, ok := s.quote.Valuations[display]; ok {
|
||||
selected := valuation.Money
|
||||
s.quote.Selected = &selected
|
||||
return
|
||||
}
|
||||
// The original price-book amount is still a valid cost fact. Keep it as a
|
||||
// visible fallback and distinguish the display mismatch from no pricing.
|
||||
selected := s.quote.Original
|
||||
s.quote.Selected = &selected
|
||||
s.quote.DisplayComplete = false
|
||||
s.quote.Complete = false
|
||||
s.quote.DisplayStatus = DisplayStatusFallbackOriginal
|
||||
s.quote.IncompleteReason = firstNonEmpty(s.valuationFailures[display], "display_unavailable")
|
||||
}
|
||||
|
||||
func (s *quoteBuildState) markIncomplete(reason string) {
|
||||
s.quote.CostComplete = false
|
||||
s.quote.Coverage = CoverageIncomplete
|
||||
s.quote.DisplayComplete = false
|
||||
s.quote.Complete = false
|
||||
s.quote.DisplayStatus = DisplayStatusUnavailable
|
||||
if s.quote.IncompleteReason == "" {
|
||||
s.quote.IncompleteReason = reason
|
||||
}
|
||||
}
|
||||
|
||||
// SelectForDisplay returns the money for a display currency preference without
|
||||
// recomputing rates. Empty display returns original.
|
||||
func (q CostQuote) SelectForDisplay(display string) (Money, bool) {
|
||||
display = NormalizeCurrency(display)
|
||||
if display == "" {
|
||||
return q.Original, true
|
||||
}
|
||||
if v, ok := q.Valuations[display]; ok {
|
||||
return v.Money, true
|
||||
}
|
||||
if NormalizeCurrency(q.Original.Currency) == display {
|
||||
return q.Original, true
|
||||
}
|
||||
return Money{}, false
|
||||
}
|
||||
|
||||
// WithSelected returns a copy with Selected set for display.
|
||||
func (q CostQuote) WithSelected(display string) CostQuote {
|
||||
out := q
|
||||
if m, ok := q.SelectForDisplay(display); ok {
|
||||
out.Selected = &m
|
||||
if quoteHasCompleteCostFact(q) {
|
||||
out.Complete = true
|
||||
out.IncompleteReason = ""
|
||||
}
|
||||
} else {
|
||||
out.Selected = nil
|
||||
out.Complete = false
|
||||
if out.IncompleteReason == "" {
|
||||
out.IncompleteReason = "display_unavailable"
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// LegacyCostFloat returns the selected (or original) amount as float64 for
|
||||
// compatibility aliases cost / costUsd / total_cost.
|
||||
func (q CostQuote) LegacyCostFloat() float64 {
|
||||
if q.Selected != nil {
|
||||
return q.Selected.Float64()
|
||||
}
|
||||
return q.Original.Float64()
|
||||
}
|
||||
|
||||
// LegacyCurrencySymbol returns a display symbol for the selected/original currency.
|
||||
func (q CostQuote) LegacyCurrencySymbol() string {
|
||||
if q.Selected != nil && q.Selected.Currency != "" {
|
||||
return CurrencySymbol(q.Selected.Currency)
|
||||
}
|
||||
return CurrencySymbol(q.Original.Currency)
|
||||
}
|
||||
|
||||
// LegacyCurrencyCode returns the ISO code for selected/original.
|
||||
func (q CostQuote) LegacyCurrencyCode() string {
|
||||
if q.Selected != nil && q.Selected.Currency != "" {
|
||||
return NormalizeCurrency(q.Selected.Currency)
|
||||
}
|
||||
return NormalizeCurrency(q.Original.Currency)
|
||||
}
|
||||
|
||||
// AggregateQuotes combines occurrence-time quotes into a session/run total.
|
||||
// Every entry must have the requested valuation; partial totals stay unavailable.
|
||||
func AggregateQuotes(quotes []CostQuote, display string) CostQuote {
|
||||
if len(quotes) == 0 {
|
||||
return emptyAggregate(NormalizeCurrency(display))
|
||||
}
|
||||
accumulator := newQuoteAccumulator(display)
|
||||
for _, quote := range quotes {
|
||||
accumulator.add(quote)
|
||||
}
|
||||
return accumulator.finish()
|
||||
}
|
||||
|
||||
type quoteAccumulator struct {
|
||||
out CostQuote
|
||||
display string
|
||||
totals map[string]Amount
|
||||
originalCurrency string
|
||||
originalTotal Amount
|
||||
originalTotals map[string]Amount
|
||||
originalComplete bool
|
||||
coverage string
|
||||
displayComplete bool
|
||||
modes map[string]struct{}
|
||||
rateBands map[string]struct{}
|
||||
unbanded bool
|
||||
}
|
||||
|
||||
func emptyAggregate(display string) CostQuote {
|
||||
out := CostQuote{
|
||||
Original: MoneyOf(Zero, display), Valuations: map[string]Valuation{}, Estimated: true,
|
||||
CostComplete: false, DisplayComplete: false, Complete: false,
|
||||
Coverage: CoverageNone,
|
||||
DisplayStatus: DisplayStatusUnavailable, AggregateMode: AggregateModeSingleCurrency,
|
||||
IncompleteReason: "no_usage",
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func newQuoteAccumulator(display string) *quoteAccumulator {
|
||||
return "eAccumulator{
|
||||
out: CostQuote{Valuations: map[string]Valuation{}, Estimated: true},
|
||||
display: NormalizeCurrency(display), totals: map[string]Amount{}, originalTotals: map[string]Amount{},
|
||||
originalComplete: true, coverage: CoverageNone, displayComplete: true,
|
||||
modes: map[string]struct{}{}, rateBands: map[string]struct{}{},
|
||||
}
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) add(quote CostQuote) {
|
||||
quote = NormalizeQuote(quote)
|
||||
a.coverage = FoldCoverage(a.coverage, quote.Coverage)
|
||||
if quote.Coverage != CoverageComplete && a.out.IncompleteReason == "" {
|
||||
a.out.IncompleteReason = quote.IncompleteReason
|
||||
}
|
||||
a.out.LegacyEstimate = a.out.LegacyEstimate || quote.LegacyEstimate
|
||||
if quote.BillingMode != "" {
|
||||
a.modes[quote.BillingMode] = struct{}{}
|
||||
}
|
||||
a.addRateBand(quote.RateBand)
|
||||
if quote.RateDate > a.out.RateDate {
|
||||
a.out.RateDate = quote.RateDate
|
||||
}
|
||||
originalCurrency := NormalizeCurrency(quote.Original.Currency)
|
||||
if len(quote.OriginalTotals) > 0 {
|
||||
for _, original := range quote.OriginalTotals {
|
||||
a.addOriginal(original, NormalizeCurrency(original.Currency))
|
||||
}
|
||||
} else {
|
||||
a.addOriginal(quote.Original, originalCurrency)
|
||||
}
|
||||
if a.display != "" {
|
||||
_, found := quote.Valuations[a.display]
|
||||
a.displayComplete = a.displayComplete && (found || originalCurrency == a.display)
|
||||
}
|
||||
originalIncluded := false
|
||||
for code, valuation := range quote.Valuations {
|
||||
code = NormalizeCurrency(code)
|
||||
if code == originalCurrency {
|
||||
originalIncluded = true
|
||||
}
|
||||
a.addValuation(code, valuation)
|
||||
}
|
||||
if originalCurrency != "" && !originalIncluded {
|
||||
a.addValuation(originalCurrency, Valuation{
|
||||
Money: quote.Original, Basis: BasisIdentity, Source: "aggregate",
|
||||
AsOf: time.Now().UTC().Format("2006-01-02"),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) addRateBand(band string) {
|
||||
switch band {
|
||||
case RateBandPeak, RateBandOffPeak:
|
||||
a.rateBands[band] = struct{}{}
|
||||
case RateBandMixed:
|
||||
a.rateBands[RateBandPeak], a.rateBands[RateBandOffPeak] = struct{}{}, struct{}{}
|
||||
default:
|
||||
a.unbanded = true
|
||||
}
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) rateBand() string {
|
||||
if a.unbanded {
|
||||
return ""
|
||||
}
|
||||
switch len(a.rateBands) {
|
||||
case 1:
|
||||
for band := range a.rateBands {
|
||||
return band
|
||||
}
|
||||
case 2:
|
||||
return RateBandMixed
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) addOriginal(original Money, currency string) {
|
||||
if currency != "" {
|
||||
a.originalTotals[currency] = a.originalTotals[currency].Add(original.AmountValue())
|
||||
}
|
||||
if a.originalCurrency == "" {
|
||||
a.originalCurrency = currency
|
||||
a.originalTotal = original.AmountValue()
|
||||
return
|
||||
}
|
||||
if currency != a.originalCurrency {
|
||||
a.originalComplete = false
|
||||
return
|
||||
}
|
||||
a.originalTotal = a.originalTotal.Add(original.AmountValue())
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) addValuation(code string, valuation Valuation) {
|
||||
if code == "" {
|
||||
return
|
||||
}
|
||||
a.totals[code] = a.totals[code].Add(valuation.Money.AmountValue())
|
||||
current, found := a.out.Valuations[code]
|
||||
if !found {
|
||||
current = valuation
|
||||
}
|
||||
current.Money = MoneyOf(a.totals[code], code)
|
||||
current.Stale = current.Stale || valuation.Stale
|
||||
a.out.Valuations[code] = current
|
||||
}
|
||||
|
||||
// costFactsComplete reads the accumulated coverage rather than tracking the
|
||||
// same fact a second time: two answers to one question drift.
|
||||
func (a *quoteAccumulator) costFactsComplete() bool {
|
||||
return a.coverage == CoverageComplete
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) finish() CostQuote {
|
||||
a.out.Coverage = a.coverage
|
||||
a.out.CostComplete = a.costFactsComplete()
|
||||
if a.originalComplete && a.originalCurrency != "" {
|
||||
a.out.Original = MoneyOf(a.originalTotal, a.originalCurrency)
|
||||
a.syncOriginalValuation()
|
||||
} else {
|
||||
a.out.Original = Money{Amount: "0"}
|
||||
}
|
||||
if len(a.originalTotals) > 1 {
|
||||
codes := make([]string, 0, len(a.originalTotals))
|
||||
for code := range a.originalTotals {
|
||||
codes = append(codes, code)
|
||||
}
|
||||
slices.Sort(codes)
|
||||
for _, code := range codes {
|
||||
a.out.OriginalTotals = append(a.out.OriginalTotals, MoneyOf(a.originalTotals[code], code))
|
||||
}
|
||||
}
|
||||
a.out.RateBand = a.rateBand()
|
||||
if len(a.modes) == 1 {
|
||||
for mode := range a.modes {
|
||||
a.out.BillingMode = mode
|
||||
}
|
||||
}
|
||||
if a.display == "" && a.originalComplete && a.costFactsComplete() {
|
||||
selected := a.out.Original
|
||||
a.out.Selected = &selected
|
||||
a.out.DisplayComplete = true
|
||||
a.out.Complete = true
|
||||
a.out.DisplayStatus = DisplayStatusMatched
|
||||
a.out.AggregateMode = AggregateModeSingleCurrency
|
||||
return a.out
|
||||
}
|
||||
valuation, found := a.out.Valuations[a.display]
|
||||
if found && a.displayComplete && a.costFactsComplete() {
|
||||
selected := valuation.Money
|
||||
a.out.Selected = &selected
|
||||
a.out.DisplayComplete = true
|
||||
a.out.Complete = true
|
||||
a.out.DisplayStatus = DisplayStatusMatched
|
||||
if len(a.originalTotals) > 1 {
|
||||
a.out.AggregateMode = AggregateModeCommonValuation
|
||||
} else {
|
||||
a.out.AggregateMode = AggregateModeSingleCurrency
|
||||
}
|
||||
return a.out
|
||||
}
|
||||
a.out.DisplayComplete = false
|
||||
a.out.Complete = false
|
||||
if !a.costFactsComplete() {
|
||||
a.out.DisplayStatus = DisplayStatusUnavailable
|
||||
a.out.IncompleteReason = firstNonEmpty(a.out.IncompleteReason, "incomplete_cost_fact")
|
||||
} else if a.originalComplete && a.originalCurrency != "" {
|
||||
selected := a.out.Original
|
||||
a.out.Selected = &selected
|
||||
a.out.DisplayStatus = DisplayStatusFallbackOriginal
|
||||
a.out.AggregateMode = AggregateModeSingleCurrency
|
||||
a.out.IncompleteReason = firstNonEmpty(a.out.IncompleteReason, "display_unavailable")
|
||||
} else {
|
||||
a.out.DisplayStatus = DisplayStatusBucketed
|
||||
a.out.AggregateMode = AggregateModeCurrencyBuckets
|
||||
a.out.IncompleteReason = firstNonEmpty(a.out.IncompleteReason, "currency_buckets")
|
||||
}
|
||||
return a.out
|
||||
}
|
||||
|
||||
// NormalizeQuote fills fields introduced after the first CostQuote wire shape.
|
||||
// It is used at every persistence and transport boundary so old telemetry and
|
||||
// old eventwire payloads remain safe without inventing a new amount.
|
||||
func NormalizeQuote(q CostQuote) CostQuote {
|
||||
if !q.CostComplete && !q.DisplayComplete && q.Complete {
|
||||
q.CostComplete = true
|
||||
q.DisplayComplete = true
|
||||
}
|
||||
if q.DisplayStatus != "" && q.DisplayStatus != DisplayStatusMatched && q.DisplayStatus != DisplayStatusFallbackOriginal && q.DisplayStatus != DisplayStatusBucketed && q.DisplayStatus != DisplayStatusUnavailable {
|
||||
q.DisplayStatus = ""
|
||||
}
|
||||
if q.RateBand != RateBandPeak && q.RateBand != RateBandOffPeak && q.RateBand != RateBandMixed {
|
||||
q.RateBand = ""
|
||||
}
|
||||
if q.AggregateMode != "" && q.AggregateMode != AggregateModeSingleCurrency && q.AggregateMode != AggregateModeCommonValuation && q.AggregateMode != AggregateModeCurrencyBuckets {
|
||||
q.AggregateMode = ""
|
||||
}
|
||||
if q.DisplayStatus == "" {
|
||||
switch {
|
||||
case q.Complete:
|
||||
q.DisplayStatus = DisplayStatusMatched
|
||||
case q.Original.Currency != "" && q.Original.Amount != "" && q.IncompleteReason != "no_price":
|
||||
q.DisplayStatus = DisplayStatusFallbackOriginal
|
||||
default:
|
||||
q.DisplayStatus = DisplayStatusUnavailable
|
||||
}
|
||||
}
|
||||
if q.AggregateMode == "" {
|
||||
if len(q.OriginalTotals) > 1 {
|
||||
q.AggregateMode = AggregateModeCurrencyBuckets
|
||||
} else if q.Selected != nil {
|
||||
q.AggregateMode = AggregateModeSingleCurrency
|
||||
}
|
||||
}
|
||||
if !ValidCoverage(q.Coverage) {
|
||||
q.Coverage = CoverageIncomplete
|
||||
if quoteHasCompleteCostFact(q) {
|
||||
q.Coverage = CoverageComplete
|
||||
}
|
||||
}
|
||||
q.Complete = q.DisplayComplete
|
||||
return q
|
||||
}
|
||||
|
||||
// quoteHasCompleteCostFact separates a known original-currency charge from a
|
||||
// display-only valuation failure must not poison an exact original total,
|
||||
// while unpriced and unrecoverable legacy records stay incomplete in every
|
||||
// display currency.
|
||||
func quoteHasCompleteCostFact(q CostQuote) bool {
|
||||
if q.CostComplete {
|
||||
return true
|
||||
}
|
||||
switch q.IncompleteReason {
|
||||
case "no_price", "missing_price_or_usage", "legacy_unrecoverable",
|
||||
"legacy_wiped_or_zero", "legacy_invalid_amount", "mixed_original_currencies",
|
||||
"incomplete_cost_fact":
|
||||
return false
|
||||
}
|
||||
if q.Complete {
|
||||
return true
|
||||
}
|
||||
currency := NormalizeCurrency(q.Original.Currency)
|
||||
if currency == "" {
|
||||
return false
|
||||
}
|
||||
valuation, ok := q.Valuations[currency]
|
||||
return ok && SameCurrency(valuation.Money.Currency, currency)
|
||||
}
|
||||
|
||||
func (a *quoteAccumulator) syncOriginalValuation() {
|
||||
valuation, found := a.out.Valuations[a.originalCurrency]
|
||||
if !found {
|
||||
valuation = Valuation{
|
||||
Basis: BasisIdentity, Source: "aggregate",
|
||||
AsOf: time.Now().UTC().Format("2006-01-02"),
|
||||
}
|
||||
}
|
||||
valuation.Money = a.out.Original
|
||||
a.out.Valuations[a.originalCurrency] = valuation
|
||||
}
|
||||
|
||||
func firstNonEmpty(values ...string) string {
|
||||
for _, v := range values {
|
||||
if strings.TrimSpace(v) != "" {
|
||||
return strings.TrimSpace(v)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// resolveCatalogIdentity names which official table, if any, may price a quote.
|
||||
// The vendor comes only from ProviderKind, which the caller reads off the
|
||||
// endpoint's host; guessing it from the entry's name or the model id would bill
|
||||
// a relay as the vendor it resells. Unnamed means no official table — the user's
|
||||
// own rate card, which is right rather than merely plausible.
|
||||
func resolveCatalogIdentity(in QuoteInput) (provider, model string) {
|
||||
provider = strings.ToLower(strings.TrimSpace(in.ProviderKind))
|
||||
model = strings.TrimSpace(in.ModelID)
|
||||
if model == "" {
|
||||
ref := strings.TrimSpace(in.ModelRef)
|
||||
if i := strings.LastIndex(ref, "/"); i >= 0 && i+1 < len(ref) {
|
||||
model = ref[i+1:]
|
||||
} else {
|
||||
model = ref
|
||||
}
|
||||
}
|
||||
return provider, model
|
||||
}
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func testInput(currency, provider, model string) QuoteInput {
|
||||
rates := RateCard{CacheHit: 0.02, Input: 1, Output: 4, Currency: currency}
|
||||
if currency == "USD" {
|
||||
rates = RateCard{CacheHit: 0.003, Input: 0.15, Output: 0.6, Currency: currency}
|
||||
}
|
||||
return QuoteInput{
|
||||
Usage: UsageTokens{PromptTokens: 1000, CompletionTokens: 2000},
|
||||
Rates: rates,
|
||||
DisplayCurrency: currency,
|
||||
ModelRef: model,
|
||||
ProviderKind: provider,
|
||||
ModelID: model,
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildQuoteIdentityAndOfficialTableOnly(t *testing.T) {
|
||||
in := testInput("USD", "deepseek", "deepseek-v4-flash")
|
||||
in.DisplayCurrency = "CNY"
|
||||
q := BuildQuote(in)
|
||||
if q.Original.Currency != "USD" || q.Original.Amount == "0" {
|
||||
t.Fatalf("original = %+v", q.Original)
|
||||
}
|
||||
v, ok := q.Valuations["CNY"]
|
||||
if !ok || v.Basis != BasisOfficialTable {
|
||||
t.Fatalf("CNY valuation = %+v, ok=%v", v, ok)
|
||||
}
|
||||
if q.Selected == nil || q.Selected.Currency != "CNY" || !q.Complete || !q.CostComplete || !q.DisplayComplete {
|
||||
t.Fatalf("quote completeness/selection = %+v", q)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildQuoteCustomPriceFallsBackWithoutFX(t *testing.T) {
|
||||
in := testInput("CNY", "deepseek", "deepseek-v4-flash")
|
||||
in.Rates.Input = 99 // no longer matches the official table
|
||||
in.DisplayCurrency = "USD"
|
||||
q := BuildQuote(in)
|
||||
if len(q.Valuations) != 1 || q.Valuations["CNY"].Basis != BasisIdentity {
|
||||
t.Fatalf("unexpected valuations: %+v", q.Valuations)
|
||||
}
|
||||
if q.Selected == nil || q.Selected.Currency != "CNY" || q.Complete || q.DisplayComplete || !q.CostComplete {
|
||||
t.Fatalf("fallback quote = %+v", q)
|
||||
}
|
||||
if q.DisplayStatus != DisplayStatusFallbackOriginal || q.Valuations["USD"].Basis == BasisFX {
|
||||
t.Fatalf("fallback status/FX = %+v", q)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAggregateMixedCurrenciesProducesBuckets(t *testing.T) {
|
||||
a := BuildQuote(QuoteInput{Usage: UsageTokens{PromptTokens: 1}, Rates: RateCard{Input: 1, Currency: "CNY"}})
|
||||
b := BuildQuote(QuoteInput{Usage: UsageTokens{PromptTokens: 1}, Rates: RateCard{Input: 1, Currency: "USD"}})
|
||||
q := AggregateQuotes([]CostQuote{a, b}, "")
|
||||
if q.DisplayStatus != DisplayStatusBucketed || q.AggregateMode != AggregateModeCurrencyBuckets || q.Selected != nil {
|
||||
t.Fatalf("mixed aggregate = %+v", q)
|
||||
}
|
||||
if len(q.OriginalTotals) != 2 || q.OriginalTotals[0].Currency != "CNY" || q.OriginalTotals[1].Currency != "USD" {
|
||||
t.Fatalf("original totals = %+v", q.OriginalTotals)
|
||||
}
|
||||
if !q.CostComplete || q.Complete || q.DisplayComplete {
|
||||
t.Fatalf("mixed completeness = %+v", q)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAggregateExplicitDisplayFallsBackToSameOriginal(t *testing.T) {
|
||||
a := BuildQuote(QuoteInput{Usage: UsageTokens{PromptTokens: 1}, Rates: RateCard{Input: 1, Currency: "CNY"}})
|
||||
b := BuildQuote(QuoteInput{Usage: UsageTokens{PromptTokens: 1}, Rates: RateCard{Input: 2, Currency: "CNY"}})
|
||||
q := AggregateQuotes([]CostQuote{a, b}, "USD")
|
||||
if q.Selected == nil || q.Selected.Currency != "CNY" || q.DisplayStatus != DisplayStatusFallbackOriginal {
|
||||
t.Fatalf("fallback aggregate = %+v", q)
|
||||
}
|
||||
if !q.CostComplete || q.DisplayComplete || q.Complete {
|
||||
t.Fatalf("fallback completeness = %+v", q)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeOldQuote(t *testing.T) {
|
||||
q := NormalizeQuote(CostQuote{Original: MoneyOf(NewAmountFromFloat(1), "USD"), Complete: true})
|
||||
if !q.CostComplete || !q.DisplayComplete || !q.Complete || q.DisplayStatus != DisplayStatusMatched {
|
||||
t.Fatalf("normalized old quote = %+v", q)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildQuoteNoPriceIsUnavailable(t *testing.T) {
|
||||
q := CostQuote{Estimated: true, CostComplete: false, DisplayComplete: false, Complete: false, DisplayStatus: DisplayStatusUnavailable, IncompleteReason: "no_price"}
|
||||
q = NormalizeQuote(q)
|
||||
if q.DisplayStatus != DisplayStatusUnavailable || q.Complete || q.CostComplete {
|
||||
t.Fatalf("unavailable = %+v", q)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildQuoteMissingUsageIsUnavailable(t *testing.T) {
|
||||
q := BuildQuote(QuoteInput{Rates: RateCard{Input: 1, Currency: "USD"}, DisplayCurrency: "USD"})
|
||||
if q.CostComplete || q.DisplayComplete || q.Complete || q.Selected != nil || q.DisplayStatus != DisplayStatusUnavailable {
|
||||
t.Fatalf("missing usage = %+v", q)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAggregateNoUsageIsUnavailable(t *testing.T) {
|
||||
q := AggregateQuotes(nil, "USD")
|
||||
if q.CostComplete || q.DisplayComplete || q.Complete || q.Selected != nil || q.DisplayStatus != DisplayStatusUnavailable {
|
||||
t.Fatalf("empty aggregate = %+v", q)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLedgerMixedOriginalBucketsContinueAccumulating(t *testing.T) {
|
||||
l := NewLedger()
|
||||
base := func(currency string, amount string) CostQuote {
|
||||
return CostQuote{
|
||||
Original: Money{Amount: amount, Currency: currency},
|
||||
Valuations: map[string]Valuation{currency: {Money: Money{Amount: amount, Currency: currency}, Basis: BasisIdentity}},
|
||||
CostComplete: true, DisplayComplete: true, Complete: true,
|
||||
DisplayStatus: DisplayStatusMatched, ModelRef: "m", PricingFingerprint: "same",
|
||||
}
|
||||
}
|
||||
l.Add(base("CNY", "1"), UsageTokens{PromptTokens: 1}, time.Time{})
|
||||
l.Add(base("USD", "2"), UsageTokens{PromptTokens: 1}, time.Time{})
|
||||
l.Add(base("CNY", "3"), UsageTokens{PromptTokens: 1}, time.Time{})
|
||||
q := l.Total("")
|
||||
if len(q.OriginalTotals) != 2 || q.OriginalTotals[0].Amount != "4" || q.OriginalTotals[1].Amount != "2" || q.Selected != nil {
|
||||
t.Fatalf("ledger buckets = %+v", q)
|
||||
}
|
||||
}
|
||||
+351
@@ -0,0 +1,351 @@
|
||||
// rates.go — every rate this project has shipped for a vendor's model, and the
|
||||
// two questions that asks: what it costs now, and what it used to cost.
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// rateGeneration is one published pair. Peak is nil for a generation the vendor
|
||||
// billed at a single rate around the clock.
|
||||
type rateGeneration struct {
|
||||
Base RateCard
|
||||
Peak *RateCard
|
||||
// From is the first day (YYYY-MM-DD, UTC) this generation is billed; empty
|
||||
// means from when it shipped. An announced change is appended with its date.
|
||||
From string
|
||||
}
|
||||
|
||||
// today is the day a dated generation is read against; a test replaces it.
|
||||
var today = func() string { return time.Now().UTC().Format("2006-01-02") }
|
||||
|
||||
// currentIndex is the generation in force today: the last one whose date has
|
||||
// come. Rows with no dated generation answer with their last, as before.
|
||||
func (r officialRateRow) currentIndex() int {
|
||||
day := today()
|
||||
for i := len(r.Rates) - 1; i > 0; i-- {
|
||||
if r.Rates[i].From == "" || r.Rates[i].From <= day {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// officialRateRow is one vendor's model, in one currency, under one billing
|
||||
// mode, with every rate this project has shipped for it.
|
||||
type officialRateRow struct {
|
||||
Provider string
|
||||
Model string
|
||||
Currency string
|
||||
BillingMode string
|
||||
Window *PeakWindow
|
||||
DocURL string
|
||||
Notes string
|
||||
CheckedOn string
|
||||
// Rates is oldest first; the last entry is what this costs today.
|
||||
Rates []rateGeneration
|
||||
}
|
||||
|
||||
// officialRates is the price book. A price change appends a generation rather
|
||||
// than editing one: that is what leaves the superseded rate where a stored price
|
||||
// can still be recognised as this project's own default instead of a number its
|
||||
// user chose, which is the difference between refreshing one and losing it.
|
||||
var officialRates = []officialRateRow{
|
||||
{
|
||||
Provider: "deepseek", Model: "deepseek-flash", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, Window: deepseekPeak, DocURL: DocDeepSeekPricing, CheckedOn: deepseekCheckedOn,
|
||||
Rates: []rateGeneration{
|
||||
{Base: RateCard{CacheHit: 0.02, Input: 1, Output: 2, Currency: "CNY"}},
|
||||
{Base: RateCard{CacheHit: 0.05, Input: 1.5, Output: 4.5, Currency: "CNY"},
|
||||
Peak: &RateCard{CacheHit: 0.10, Input: 3.0, Output: 9.0, Currency: "CNY"}},
|
||||
{Base: RateCard{CacheHit: 0.02, Input: 1, Output: 4, Currency: "CNY"},
|
||||
Peak: &RateCard{CacheHit: 0.04, Input: 2, Output: 8, Currency: "CNY"}},
|
||||
},
|
||||
},
|
||||
{
|
||||
Provider: "deepseek", Model: "deepseek-flash", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, Window: deepseekPeak, DocURL: DocDeepSeekPricing, CheckedOn: deepseekCheckedOn,
|
||||
Rates: []rateGeneration{
|
||||
{Base: RateCard{CacheHit: 0.0028, Input: 0.14, Output: 0.28, Currency: "USD"}},
|
||||
{Base: RateCard{CacheHit: 0.007, Input: 0.22, Output: 0.66, Currency: "USD"},
|
||||
Peak: &RateCard{CacheHit: 0.014, Input: 0.44, Output: 1.32, Currency: "USD"}},
|
||||
{Base: RateCard{CacheHit: 0.003, Input: 0.15, Output: 0.6, Currency: "USD"},
|
||||
Peak: &RateCard{CacheHit: 0.006, Input: 0.3, Output: 1.2, Currency: "USD"}},
|
||||
},
|
||||
},
|
||||
{
|
||||
Provider: "deepseek", Model: "deepseek-v4-pro", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, Window: deepseekPeak, DocURL: DocDeepSeekPricing, CheckedOn: deepseekCheckedOn,
|
||||
Rates: []rateGeneration{
|
||||
{Base: RateCard{CacheHit: 0.025, Input: 3, Output: 6, Currency: "CNY"}},
|
||||
{Base: RateCard{CacheHit: 0.15, Input: 4.5, Output: 13.5, Currency: "CNY"},
|
||||
Peak: &RateCard{CacheHit: 0.30, Input: 9.0, Output: 27.0, Currency: "CNY"}},
|
||||
},
|
||||
},
|
||||
{
|
||||
Provider: "deepseek", Model: "deepseek-v4-pro", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, Window: deepseekPeak, DocURL: DocDeepSeekPricing, CheckedOn: deepseekCheckedOn,
|
||||
Rates: []rateGeneration{
|
||||
{Base: RateCard{CacheHit: 0.003625, Input: 0.435, Output: 0.87, Currency: "USD"}},
|
||||
{Base: RateCard{CacheHit: 0.022, Input: 0.66, Output: 1.98, Currency: "USD"},
|
||||
Peak: &RateCard{CacheHit: 0.044, Input: 1.32, Output: 3.96, Currency: "USD"}},
|
||||
},
|
||||
},
|
||||
|
||||
// LongCat's launch discount, for which the vendor publishes no end date. List
|
||||
// price is CNY 0.10/5/20, USD 0.015/0.75/2.95 — what these revert to, and why
|
||||
// CheckedOn earns its place here.
|
||||
{
|
||||
Provider: "longcat", Model: "LongCat-2.0", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocLongCatPricingCNY, Notes: "launch_discount_no_end_date", CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.04, Input: 2, Output: 8, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "longcat", Model: "LongCat-2.0", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, DocURL: DocLongCatPricingUSD, Notes: "launch_discount_no_end_date", CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.006, Input: 0.30, Output: 1.20, Currency: "USD"}}},
|
||||
},
|
||||
|
||||
// Google: the two flash models double on 2027-01-01, hence the dated generation.
|
||||
// gemini-3.1-pro-preview is the rate up to 200k prompt tokens; gemini-4-argon has no price.
|
||||
{
|
||||
Provider: "google", Model: "gemini-3.8-flash", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, DocURL: DocGeminiPricing, CheckedOn: googleCheckedOn,
|
||||
Rates: []rateGeneration{
|
||||
{Base: RateCard{CacheHit: 0.075, Input: 0.75, Output: 3.75, Currency: "USD"}},
|
||||
{From: "2027-01-01", Base: RateCard{CacheHit: 0.15, Input: 1.50, Output: 7.50, Currency: "USD"}},
|
||||
},
|
||||
},
|
||||
{
|
||||
Provider: "google", Model: "gemini-3.6-flash", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, DocURL: DocGeminiPricing, CheckedOn: googleCheckedOn,
|
||||
Rates: []rateGeneration{
|
||||
{Base: RateCard{CacheHit: 0.075, Input: 0.75, Output: 3.75, Currency: "USD"}},
|
||||
{From: "2027-01-01", Base: RateCard{CacheHit: 0.15, Input: 1.50, Output: 7.50, Currency: "USD"}},
|
||||
},
|
||||
},
|
||||
{
|
||||
Provider: "google", Model: "gemini-3.5-flash-lite", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, DocURL: DocGeminiPricing, CheckedOn: googleCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.03, Input: 0.30, Output: 2.50, Currency: "USD"}}},
|
||||
},
|
||||
{
|
||||
Provider: "google", Model: "gemini-3.1-pro-preview", Currency: "USD",
|
||||
BillingMode: BillingModePAYG, DocURL: DocGeminiPricing, Notes: "up_to_200k_prompt_tokens", CheckedOn: googleCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.20, Input: 2.00, Output: 12.00, Currency: "USD"}}},
|
||||
},
|
||||
|
||||
// MiMo domestic PAYG; Token Plan bills the same rates as subscription_equivalent.
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.5-pro", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocMiMoPAYG, CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.025, Input: 3, Output: 6, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.5", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocMiMoPAYG, CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.02, Input: 1, Output: 2, Currency: "CNY"}}},
|
||||
},
|
||||
// Not on the vendor's price page as of CheckedOn. Kept because removing a rate
|
||||
// is what turns a still-configured model's cost silently into zero; a stale
|
||||
// rate at least reads as a number somebody can question.
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2-flash", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocMiMoPAYG, Notes: "not_on_vendor_page", CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.07, Input: 0.70, Output: 2.10, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.5-pro", Currency: "CNY",
|
||||
BillingMode: BillingModeSubscriptionEquivalent, DocURL: DocMiMoTokenPlan, Notes: "payg_equivalent_not_plan_bill", CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.025, Input: 3, Output: 6, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.5", Currency: "CNY",
|
||||
BillingMode: BillingModeSubscriptionEquivalent, DocURL: DocMiMoTokenPlan, Notes: "payg_equivalent_not_plan_bill", CheckedOn: checkedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.02, Input: 1, Output: 2, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.6-pro", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocMiMoPAYG, CheckedOn: mimoCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.025, Input: 3, Output: 6, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.6-flash", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocMiMoPAYG, CheckedOn: mimoCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.02, Input: 1, Output: 2, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.6-pro-ultraspeed", Currency: "CNY",
|
||||
BillingMode: BillingModePAYG, DocURL: DocMiMoPAYG, CheckedOn: mimoCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.25, Input: 30, Output: 60, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.6-pro", Currency: "CNY",
|
||||
BillingMode: BillingModeSubscriptionEquivalent, DocURL: DocMiMoTokenPlan, Notes: "payg_equivalent_not_plan_bill", CheckedOn: mimoCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.025, Input: 3, Output: 6, Currency: "CNY"}}},
|
||||
},
|
||||
{
|
||||
Provider: "mimo", Model: "mimo-v2.6-flash", Currency: "CNY",
|
||||
BillingMode: BillingModeSubscriptionEquivalent, DocURL: DocMiMoTokenPlan, Notes: "payg_equivalent_not_plan_bill", CheckedOn: mimoCheckedOn,
|
||||
Rates: []rateGeneration{{Base: RateCard{CacheHit: 0.02, Input: 1, Output: 2, Currency: "CNY"}}},
|
||||
},
|
||||
}
|
||||
|
||||
// retiredDeepSeekFlashModels are names DeepSeek still answers as its current
|
||||
// flash model, at that model's price. Owned here because that is where the rates
|
||||
// they bill at live; config reads them through the accessor below.
|
||||
var retiredDeepSeekFlashModels = []string{"deepseek-v4-flash", "deepseek-v4-flash-vision-exp"}
|
||||
|
||||
// RetiredDeepSeekFlashModels copies the list on the way out: a caller sorting or
|
||||
// trimming the result would otherwise change what every other caller sees.
|
||||
func RetiredDeepSeekFlashModels() []string {
|
||||
return append([]string(nil), retiredDeepSeekFlashModels...)
|
||||
}
|
||||
|
||||
// RateModelFor answers which model's rate a name is billed at — a retired name
|
||||
// at the model that replaced it, an older spelling at the one it was renamed
|
||||
// from. Scoped to billing on purpose: two models can share a rate without being
|
||||
// the same model, and nothing outside a price lookup may read this as identity.
|
||||
func RateModelFor(provider, model string) string {
|
||||
model = strings.TrimSpace(model)
|
||||
switch strings.ToLower(strings.TrimSpace(provider)) {
|
||||
case "deepseek":
|
||||
if slices.Contains(retiredDeepSeekFlashModels, model) {
|
||||
return "deepseek-flash"
|
||||
}
|
||||
case "mimo":
|
||||
switch model {
|
||||
case "mimo-v2-pro":
|
||||
return "mimo-v2.5-pro"
|
||||
case "mimo-v2-omni":
|
||||
return "mimo-v2.5"
|
||||
}
|
||||
}
|
||||
return model
|
||||
}
|
||||
|
||||
// DefaultCurrency is what a vendor's rates are quoted in when an entry does not
|
||||
// say. A vendor publishing one currency leaves no choice to make; DeepSeek
|
||||
// prices in both, and USD is what an entry that never chose has been read as.
|
||||
func DefaultCurrency(provider string) string {
|
||||
provider = strings.ToLower(strings.TrimSpace(provider))
|
||||
seen := map[string]bool{}
|
||||
for _, row := range officialRates {
|
||||
if row.Provider == provider {
|
||||
seen[row.Currency] = true
|
||||
}
|
||||
}
|
||||
if len(seen) == 1 {
|
||||
for currency := range seen {
|
||||
return currency
|
||||
}
|
||||
}
|
||||
return "USD"
|
||||
}
|
||||
|
||||
// officialRateRowsFor finds the rows for a model, which is more than one when a
|
||||
// vendor publishes the same rates under two billing modes.
|
||||
func officialRateRowsFor(provider, model, currency string) []officialRateRow {
|
||||
provider, currency = strings.ToLower(strings.TrimSpace(provider)), NormalizeCurrency(currency)
|
||||
model = RateModelFor(provider, model)
|
||||
var out []officialRateRow
|
||||
for _, row := range officialRates {
|
||||
if row.Provider == provider && row.Model == model && row.Currency == currency {
|
||||
out = append(out, row)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// CurrentRate is what a vendor's model costs in a currency today — the off-peak
|
||||
// card, which is what a caller charging one rate should quote. Nil when this
|
||||
// catalog prices neither. A model published under two billing modes quotes the
|
||||
// pay-as-you-go rate, which is the one an unsubscribed caller is charged.
|
||||
func CurrentRate(provider, model, currency string) *RateCard {
|
||||
for _, row := range officialRateRowsFor(provider, model, currency) {
|
||||
if row.BillingMode == BillingModeSubscriptionEquivalent {
|
||||
continue
|
||||
}
|
||||
if len(row.Rates) > 0 {
|
||||
rate := row.Rates[row.currentIndex()].Base
|
||||
return &rate
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// IsOfficialCard reports whether card is a rate this project has shipped for the
|
||||
// model in the card's own currency: any generation, its off-peak or its peak
|
||||
// card. A card that is none of them is the user's, or damaged, and cannot be
|
||||
// valued against the vendor's other-currency table.
|
||||
func IsOfficialCard(provider, model string, card RateCard) bool {
|
||||
for _, row := range officialRateRowsFor(provider, model, card.Currency) {
|
||||
if row.BillingMode == BillingModeSubscriptionEquivalent {
|
||||
continue
|
||||
}
|
||||
for _, generation := range row.Rates {
|
||||
if sameCard(generation.Base, card) || generation.Peak != nil && sameCard(*generation.Peak, card) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func sameCard(a, b RateCard) bool {
|
||||
return a.CacheHit == b.CacheHit && a.Input == b.Input && a.Output == b.Output
|
||||
}
|
||||
|
||||
// SupersededRates are the rates shipped for a model before the current one. A
|
||||
// stored price matching one came from this project rather than from its user, so
|
||||
// a refresh may replace it; anything else is theirs and is left alone.
|
||||
func SupersededRates(provider, model, currency string) []RateCard {
|
||||
var out []RateCard
|
||||
for _, row := range officialRateRowsFor(provider, model, currency) {
|
||||
if row.BillingMode == BillingModeSubscriptionEquivalent || len(row.Rates) < 2 {
|
||||
continue
|
||||
}
|
||||
for _, generation := range row.Rates[:row.currentIndex()] {
|
||||
out = append(out, generation.Base)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// officialCatalogRows publishes the current generation of every row, plus the
|
||||
// retired DeepSeek names, which bill at the flash rate under their own id.
|
||||
func officialCatalogRows() []CatalogEntry {
|
||||
rows := make([]CatalogEntry, 0, len(officialRates)+len(retiredDeepSeekFlashModels)*2)
|
||||
for _, row := range officialRates {
|
||||
if len(row.Rates) == 0 {
|
||||
continue
|
||||
}
|
||||
current := row.Rates[row.currentIndex()]
|
||||
rows = append(rows, row.entry(row.Model, current))
|
||||
if row.Provider != "deepseek" || row.Model != "deepseek-flash" {
|
||||
continue
|
||||
}
|
||||
for _, retired := range retiredDeepSeekFlashModels {
|
||||
rows = append(rows, row.entry(retired, current))
|
||||
}
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
func (r officialRateRow) entry(model string, current rateGeneration) CatalogEntry {
|
||||
// Copied, so a caller holding a returned row cannot reach back into the
|
||||
// table every other caller reads.
|
||||
var peak *RateCard
|
||||
if current.Peak != nil {
|
||||
card := *current.Peak
|
||||
peak = &card
|
||||
}
|
||||
return CatalogEntry{
|
||||
Provider: r.Provider, Model: model, Currency: r.Currency,
|
||||
CacheHit: current.Base.CacheHit, Input: current.Base.Input, Output: current.Base.Output,
|
||||
Peak: peak, Window: r.Window, DocURL: r.DocURL,
|
||||
BillingMode: r.BillingMode, Notes: r.Notes, CheckedOn: r.CheckedOn,
|
||||
}
|
||||
}
|
||||
+87
@@ -0,0 +1,87 @@
|
||||
package pricing
|
||||
|
||||
import "testing"
|
||||
|
||||
// The history answers one question: is a stored price the current default, or a
|
||||
// superseded one this project may replace? A generation repeating another in the
|
||||
// same row makes that unanswerable — the current rate would match as superseded,
|
||||
// and every refresh would rewrite it while reporting that something changed.
|
||||
func TestRateHistoryHasNoRepeatedGenerations(t *testing.T) {
|
||||
for _, row := range officialRates {
|
||||
if len(row.Rates) == 0 {
|
||||
t.Errorf("%s/%s/%s has no rate at all", row.Provider, row.Model, row.Currency)
|
||||
continue
|
||||
}
|
||||
seen := map[RateCard]bool{}
|
||||
for _, generation := range row.Rates {
|
||||
if seen[generation.Base] {
|
||||
t.Errorf("%s/%s/%s repeats the rate %+v", row.Provider, row.Model, row.Currency, generation.Base)
|
||||
}
|
||||
seen[generation.Base] = true
|
||||
if generation.Base.Currency != row.Currency {
|
||||
t.Errorf("%s/%s/%s holds a rate filed as %q", row.Provider, row.Model, row.Currency, generation.Base.Currency)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// CurrentRate returns the first row it matches, so two rows sharing all four
|
||||
// identifying fields make which rate a caller gets depend on table order.
|
||||
func TestOfficialRateRowsAreUniquelyIdentified(t *testing.T) {
|
||||
type key struct{ provider, model, currency, mode string }
|
||||
seen := map[key]bool{}
|
||||
for _, row := range officialRates {
|
||||
k := key{row.Provider, row.Model, row.Currency, row.BillingMode}
|
||||
if seen[k] {
|
||||
t.Errorf("%+v appears twice; which rate CurrentRate returns is then a matter of order", k)
|
||||
}
|
||||
seen[k] = true
|
||||
}
|
||||
}
|
||||
|
||||
// Superseded means "no longer charged". Returning the current rate among them
|
||||
// would make every refresh a no-op that still reports having changed something.
|
||||
func TestSupersededRatesExcludeTheCurrentOne(t *testing.T) {
|
||||
for _, row := range officialRates {
|
||||
current := CurrentRate(row.Provider, row.Model, row.Currency)
|
||||
if current == nil {
|
||||
if row.BillingMode != BillingModeSubscriptionEquivalent {
|
||||
t.Errorf("%s/%s/%s has no current rate", row.Provider, row.Model, row.Currency)
|
||||
}
|
||||
continue
|
||||
}
|
||||
for _, superseded := range SupersededRates(row.Provider, row.Model, row.Currency) {
|
||||
if superseded == *current {
|
||||
t.Errorf("%s/%s/%s lists its current rate as superseded", row.Provider, row.Model, row.Currency)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A retired name bills at the flash rate, which is the whole reason the catalog
|
||||
// publishes a row for it; resolving it anywhere else would price it as nothing.
|
||||
func TestRetiredDeepSeekNamesBillAtTheFlashRate(t *testing.T) {
|
||||
for _, retired := range RetiredDeepSeekFlashModels() {
|
||||
for _, currency := range []string{"CNY", "USD"} {
|
||||
got := CurrentRate("deepseek", retired, currency)
|
||||
want := CurrentRate("deepseek", "deepseek-flash", currency)
|
||||
if got == nil || want == nil || *got != *want {
|
||||
t.Errorf("%s/%s = %+v, want the flash rate %+v", retired, currency, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Every row the table declares has to reach the catalog, which is what a quote
|
||||
// actually reads; a rate declared but never published quotes as nothing.
|
||||
func TestEveryRateRowReachesTheCatalog(t *testing.T) {
|
||||
published := map[string]bool{}
|
||||
for _, entry := range OfficialCatalog() {
|
||||
published[entry.Provider+"/"+entry.Model+"/"+entry.Currency+"/"+entry.BillingMode] = true
|
||||
}
|
||||
for _, row := range officialRates {
|
||||
if !published[row.Provider+"/"+row.Model+"/"+row.Currency+"/"+row.BillingMode] {
|
||||
t.Errorf("%s/%s/%s/%s is declared but never published", row.Provider, row.Model, row.Currency, row.BillingMode)
|
||||
}
|
||||
}
|
||||
}
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
// request_rates.go — what a vendor charges per successful request for a service
|
||||
// that is not metered in tokens. Like officialRates, a change appends a
|
||||
// generation and never edits one.
|
||||
package pricing
|
||||
|
||||
import "strings"
|
||||
|
||||
// RequestRate is a price per 1,000 successful requests.
|
||||
type RequestRate struct {
|
||||
Per1000 float64
|
||||
Currency string
|
||||
}
|
||||
|
||||
// Cost is what n successful requests cost.
|
||||
func (r RequestRate) Cost(n int) float64 {
|
||||
return r.Per1000 * float64(n) / 1000
|
||||
}
|
||||
|
||||
// RequestRateInfo is a rate with where it was read and when.
|
||||
type RequestRateInfo struct {
|
||||
RequestRate
|
||||
DocURL string
|
||||
CheckedOn string
|
||||
}
|
||||
|
||||
type requestRateRow struct {
|
||||
Provider string
|
||||
Service string
|
||||
DocURL string
|
||||
CheckedOn string
|
||||
// Rates is oldest first; the last entry is what the service costs today.
|
||||
Rates []RequestRate
|
||||
}
|
||||
|
||||
var requestRates = []requestRateRow{
|
||||
{Provider: "deepseek", Service: "search", DocURL: DocDeepSeekPricing, CheckedOn: "2026-10-17",
|
||||
Rates: []RequestRate{{Per1000: 0.75, Currency: "USD"}}},
|
||||
{Provider: "deepseek", Service: "fetch", DocURL: DocDeepSeekPricing, CheckedOn: "2026-10-17",
|
||||
Rates: []RequestRate{{Per1000: 0.75, Currency: "USD"}}},
|
||||
}
|
||||
|
||||
// CurrentRequestRate is what a vendor's service costs per request today.
|
||||
func CurrentRequestRate(provider, service string) (RequestRateInfo, bool) {
|
||||
provider, service = strings.ToLower(strings.TrimSpace(provider)), strings.ToLower(strings.TrimSpace(service))
|
||||
for _, row := range requestRates {
|
||||
if row.Provider == provider && row.Service == service {
|
||||
return RequestRateInfo{RequestRate: row.Rates[len(row.Rates)-1], DocURL: row.DocURL, CheckedOn: row.CheckedOn}, true
|
||||
}
|
||||
}
|
||||
return RequestRateInfo{}, false
|
||||
}
|
||||
|
||||
// RequestQuote prices n successful requests as a CostQuote, so a per-request
|
||||
// charge enters the ledger beside token costs under its own usage source.
|
||||
func RequestQuote(rate RequestRateInfo, n int, modelRef, usageSource string) CostQuote {
|
||||
currency := NormalizeCurrency(rate.Currency)
|
||||
original := MoneyOf(NewAmountFromFloat(rate.Cost(n)), currency)
|
||||
q := CostQuote{
|
||||
Original: original,
|
||||
Valuations: map[string]Valuation{currency: {
|
||||
Money: original, Basis: BasisOfficialTable, Source: rate.DocURL, AsOf: rate.CheckedOn,
|
||||
}},
|
||||
Selected: &original,
|
||||
BillingMode: BillingModePAYG,
|
||||
CostComplete: true,
|
||||
Coverage: CoverageComplete,
|
||||
DisplayComplete: true,
|
||||
Complete: true,
|
||||
DisplayStatus: DisplayStatusMatched,
|
||||
AggregateMode: AggregateModeSingleCurrency,
|
||||
ModelRef: modelRef,
|
||||
UsageSource: usageSource,
|
||||
CatalogSource: rate.DocURL,
|
||||
}
|
||||
q.PricingFingerprint = PricingFingerprint(RateCard{Input: rate.Per1000, Currency: currency})
|
||||
return q
|
||||
}
|
||||
+80
@@ -0,0 +1,80 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestRequestRatesAreAppendOnlyAndSourced(t *testing.T) {
|
||||
seen := map[string]bool{}
|
||||
for _, row := range requestRates {
|
||||
key := row.Provider + "/" + row.Service
|
||||
if seen[key] {
|
||||
t.Fatalf("%s appears twice: a change is a new generation on its row, not a second row", key)
|
||||
}
|
||||
seen[key] = true
|
||||
if row.DocURL == "" || row.CheckedOn == "" || len(row.Rates) == 0 {
|
||||
t.Fatalf("%s lacks a source, a check date or a rate", key)
|
||||
}
|
||||
for _, r := range row.Rates {
|
||||
if r.Per1000 <= 0 || r.Currency == "" {
|
||||
t.Fatalf("%s carries an unusable rate %+v", key, r)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCurrentRequestRateForDeepSeekSearchAndFetch(t *testing.T) {
|
||||
for _, service := range []string{"search", "fetch"} {
|
||||
got, ok := CurrentRequestRate("deepseek", service)
|
||||
if !ok || got.Per1000 != 0.75 || got.Currency != "USD" || got.DocURL != DocDeepSeekPricing || got.CheckedOn == "" {
|
||||
t.Fatalf("%s = %+v (%v), want $0.75 per 1,000 successful requests from the pricing page", service, got, ok)
|
||||
}
|
||||
}
|
||||
if _, ok := CurrentRequestRate("deepseek", "nonsense"); ok {
|
||||
t.Fatal("an unknown service has a rate")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRequestCostIsPerSuccessfulRequest(t *testing.T) {
|
||||
rate, _ := CurrentRequestRate("deepseek", "search")
|
||||
if got := rate.Cost(1000); got != 0.75 {
|
||||
t.Fatalf("1,000 requests cost %v, want 0.75", got)
|
||||
}
|
||||
if got := rate.Cost(1); got != 0.00075 {
|
||||
t.Fatalf("1 request costs %v, want 0.00075", got)
|
||||
}
|
||||
if got := rate.Cost(0); got != 0 {
|
||||
t.Fatalf("no requests cost %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A per-request charge lands in the same ledger as token costs, under its own
|
||||
// source, so the usage view can show what search cost apart from the model.
|
||||
func TestRequestQuoteLandsInTheLedgerUnderItsOwnSource(t *testing.T) {
|
||||
rate, _ := CurrentRequestRate("deepseek", "search")
|
||||
at := time.Date(2026, 10, 17, 8, 0, 0, 0, time.UTC)
|
||||
q := RequestQuote(rate, 1, "deepseek-api/search", "deepseek-search")
|
||||
if q.Original.Currency != "USD" || q.Original.Amount != "0.00075" {
|
||||
t.Fatalf("original = %+v, want 0.00075 USD for one successful request", q.Original)
|
||||
}
|
||||
if !q.CostComplete || q.UsageSource != "deepseek-search" || q.ModelRef != "deepseek-api/search" {
|
||||
t.Fatalf("quote = %+v", q)
|
||||
}
|
||||
l := NewLedger()
|
||||
for range 3 {
|
||||
l.Add(q, UsageTokens{}, at)
|
||||
}
|
||||
l.Add(BuildQuote(QuoteInput{Usage: UsageTokens{PromptTokens: 1000, CompletionTokens: 10}, Rates: RateCard{Input: 1, Output: 2, Currency: "USD"}, ModelRef: "m", UsageSource: "executor"}), UsageTokens{PromptTokens: 1000}, at)
|
||||
var search LedgerEntry
|
||||
var sources int
|
||||
for _, e := range l.Entries {
|
||||
sources++
|
||||
if e.UsageSource == "deepseek-search" {
|
||||
search = e
|
||||
}
|
||||
}
|
||||
if sources != 2 || search.Quote.Original.Amount != "0.00225" {
|
||||
t.Fatalf("ledger = %+v (sources %d), want search summed apart from the model: 3 requests = 0.00225 USD", l.Entries, sources)
|
||||
}
|
||||
}
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
// tariff.go — which rate a vendor's own schedule charges right now.
|
||||
package pricing
|
||||
|
||||
import "time"
|
||||
|
||||
// Tariff is the vendor's schedule as it stands at one instant: whether it is in
|
||||
// its peak window and, when it will flip, when.
|
||||
type Tariff struct {
|
||||
Peak bool
|
||||
ChangesAt time.Time
|
||||
Changes bool
|
||||
}
|
||||
|
||||
// LiveTariff reports the window a model's rates are billed in at the given
|
||||
// instant. It answers only for a rate confirmed as the vendor's own table and
|
||||
// only where that table has a peak rate: a custom price is the user's own
|
||||
// arrangement, and projecting someone else's schedule onto it would announce a
|
||||
// tariff nobody charges.
|
||||
func LiveTariff(providerKind, model string, rates RateCard, at time.Time) (Tariff, bool) {
|
||||
entry, ok := MatchesCatalog(providerKind, model, rates)
|
||||
if !ok {
|
||||
return Tariff{}, false
|
||||
}
|
||||
if entry.Peak == nil || entry.Window == nil {
|
||||
return Tariff{}, false
|
||||
}
|
||||
next, changes := entry.Window.NextChange(at)
|
||||
return Tariff{Peak: entry.Window.IsPeak(at), ChangesAt: next, Changes: changes}, true
|
||||
}
|
||||
+69
@@ -0,0 +1,69 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func beijingTime(t *testing.T, s string) time.Time {
|
||||
t.Helper()
|
||||
at, err := time.ParseInLocation("2006-01-02 15:04", s, time.FixedZone("", beijing))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return at
|
||||
}
|
||||
|
||||
// The next flip is the next whole hour of the vendor's schedule where the answer
|
||||
// changes, in the vendor's own zone and across a weekend.
|
||||
func TestNextChangeFollowsTheVendorsScheduleAcrossAWeekend(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
now, want string
|
||||
peak bool
|
||||
}{
|
||||
{"2026-10-12 08:30", "2026-10-12 09:00", false}, // Monday before the morning peak
|
||||
{"2026-10-12 11:59", "2026-10-12 12:00", true}, // the morning peak ends
|
||||
{"2026-10-12 12:00", "2026-10-12 14:00", false}, // lunch
|
||||
{"2026-10-12 17:30", "2026-10-12 18:00", true}, // the afternoon peak ends
|
||||
{"2026-10-16 18:00", "2026-10-19 09:00", false}, // Friday evening to Monday morning
|
||||
{"2026-10-17 10:00", "2026-10-19 09:00", false}, // a Saturday is off-peak all day
|
||||
} {
|
||||
now := beijingTime(t, tc.now)
|
||||
if got := deepseekPeak.IsPeak(now); got != tc.peak {
|
||||
t.Errorf("IsPeak(%s) = %v, want %v", tc.now, got, tc.peak)
|
||||
}
|
||||
got, ok := deepseekPeak.NextChange(now)
|
||||
if !ok || !got.Equal(beijingTime(t, tc.want)) {
|
||||
t.Errorf("NextChange(%s) = %v %v, want %s", tc.now, got, ok, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNextChangeOfAWindowThatNeverFlipsIsNone(t *testing.T) {
|
||||
var w *PeakWindow
|
||||
if _, ok := w.NextChange(time.Now()); ok {
|
||||
t.Fatal("a nil window flips")
|
||||
}
|
||||
if _, ok := (&PeakWindow{OffsetSeconds: beijing}).NextChange(time.Now()); ok {
|
||||
t.Fatal("a window with no hours flips")
|
||||
}
|
||||
}
|
||||
|
||||
// Only a rate confirmed as the vendor's own earns a tariff, and only where the
|
||||
// vendor's table has a peak rate.
|
||||
func TestLiveTariffIsTheVendorsAndNeverTheUsers(t *testing.T) {
|
||||
official := *CurrentRate("deepseek", "deepseek-flash", "CNY")
|
||||
at := beijingTime(t, "2026-10-12 10:00")
|
||||
tariff, ok := LiveTariff("deepseek", "deepseek-flash", official, at)
|
||||
if !ok || !tariff.Peak || !tariff.Changes || !tariff.ChangesAt.Equal(beijingTime(t, "2026-10-12 12:00")) {
|
||||
t.Fatalf("official rate at a peak hour = %+v %v, want peak until 12:00", tariff, ok)
|
||||
}
|
||||
custom := official
|
||||
custom.Input = official.Input * 3
|
||||
if _, ok := LiveTariff("deepseek", "deepseek-flash", custom, at); ok {
|
||||
t.Fatal("a custom price was given the vendor's schedule")
|
||||
}
|
||||
if _, ok := LiveTariff("", "deepseek-flash", official, at); ok {
|
||||
t.Fatal("a relay with no vendor kind was given the vendor's schedule")
|
||||
}
|
||||
}
|
||||
+85
@@ -0,0 +1,85 @@
|
||||
// window.go — when a vendor charges its peak rates.
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"time"
|
||||
)
|
||||
|
||||
// PeakWindow is the vendor's own peak schedule. The rule is published in the
|
||||
// vendor's timezone — DeepSeek's 09:00-12:00 and 14:00-18:00 are Beijing hours
|
||||
// — so the zone travels with it and never comes from the reader's machine.
|
||||
type PeakWindow struct {
|
||||
// OffsetSeconds fixes that zone. China does not observe DST, so an offset
|
||||
// is exact for these rules and costs no embedded tzdata.
|
||||
OffsetSeconds int
|
||||
// Hours are [from, to) in that zone.
|
||||
Hours [][2]int
|
||||
// WeekendOffPeak is the date a vendor stopped charging peak rates on
|
||||
// Saturdays and Sundays, YYYY-MM-DD in the same zone. Empty = no such rule.
|
||||
WeekendOffPeak string
|
||||
// HolidayOffPeak lists dates, YYYY-MM-DD in the same zone, billed off-peak
|
||||
// in full, from the holiday calendar the vendor's pricing page names.
|
||||
HolidayOffPeak []string
|
||||
}
|
||||
|
||||
// beijing is where every rate in this catalog is published.
|
||||
const beijing = 8 * 3600
|
||||
|
||||
// IsPeak reports whether t falls inside the window. A nil window never peaks,
|
||||
// which is what a vendor billing one rate around the clock means.
|
||||
func (w *PeakWindow) IsPeak(t time.Time) bool {
|
||||
if w == nil || len(w.Hours) == 0 {
|
||||
return false
|
||||
}
|
||||
local := t.In(time.FixedZone("", w.OffsetSeconds))
|
||||
if w.weekendOff(local) || w.holidayOff(local) {
|
||||
return false
|
||||
}
|
||||
hour := local.Hour()
|
||||
for _, r := range w.Hours {
|
||||
if hour >= r[0] && hour < r[1] {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// weekendOff reports whether local lands on a weekend the vendor has already
|
||||
// flattened to off-peak. Compared as dates in the vendor's zone: the rule's
|
||||
// start is a wall-clock midnight there, not an instant.
|
||||
func (w *PeakWindow) weekendOff(local time.Time) bool {
|
||||
if w.WeekendOffPeak == "" {
|
||||
return false
|
||||
}
|
||||
if day := local.Weekday(); day != time.Saturday && day != time.Sunday {
|
||||
return false
|
||||
}
|
||||
return local.Format("2006-01-02") >= w.WeekendOffPeak
|
||||
}
|
||||
|
||||
// holidayOff reports whether local lands on a published holiday the vendor
|
||||
// bills off-peak. Compared as dates in the vendor's zone, like weekendOff.
|
||||
func (w *PeakWindow) holidayOff(local time.Time) bool {
|
||||
day := local.Format("2006-01-02")
|
||||
return slices.Contains(w.HolidayOffPeak, day)
|
||||
}
|
||||
|
||||
// NextChange is the first instant after t at which IsPeak answers differently,
|
||||
// and false when it does not within a fortnight: a nil window, or a schedule
|
||||
// with nothing to flip to. The hours are whole, so only an hour boundary in the
|
||||
// vendor's zone can be a change.
|
||||
func (w *PeakWindow) NextChange(t time.Time) (time.Time, bool) {
|
||||
if w == nil || len(w.Hours) == 0 {
|
||||
return time.Time{}, false
|
||||
}
|
||||
now := w.IsPeak(t)
|
||||
next := t.In(time.FixedZone("", w.OffsetSeconds)).Truncate(time.Hour).Add(time.Hour)
|
||||
for range 14 * 24 {
|
||||
if w.IsPeak(next) != now {
|
||||
return next, true
|
||||
}
|
||||
next = next.Add(time.Hour)
|
||||
}
|
||||
return time.Time{}, false
|
||||
}
|
||||
+166
@@ -0,0 +1,166 @@
|
||||
package pricing
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// beijingAt builds an instant from the wall clock the vendor publishes its rule
|
||||
// in, so a case reads as the hour a user in Beijing would see.
|
||||
func beijingAt(year int, month time.Month, day, hour int) time.Time {
|
||||
return time.Date(year, month, day, hour, 0, 0, 0, time.FixedZone("", beijing))
|
||||
}
|
||||
|
||||
func TestPeakWindowReadsTheVendorsClock(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
at time.Time
|
||||
peak bool
|
||||
}{
|
||||
// 2026-08-19 is a Wednesday; 08-22 a Saturday, 08-23 a Sunday.
|
||||
{"weekday morning inside 9-12", beijingAt(2026, 8, 19, 10), true},
|
||||
{"weekday lunch gap between the two ranges", beijingAt(2026, 8, 19, 13), false},
|
||||
{"weekday afternoon inside 14-18", beijingAt(2026, 8, 19, 15), true},
|
||||
{"weekday 18:00 is the exclusive end", beijingAt(2026, 8, 19, 18), false},
|
||||
{"weekday night", beijingAt(2026, 8, 19, 23), false},
|
||||
// The weekend rule starts 2026-08-23, so the Saturday before it still peaks.
|
||||
{"saturday before the weekend rule", beijingAt(2026, 8, 22, 10), true},
|
||||
{"sunday the weekend rule starts", beijingAt(2026, 8, 23, 10), false},
|
||||
{"saturday after the weekend rule", beijingAt(2026, 8, 29, 10), false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
if got := deepseekPeak.IsPeak(c.at); got != c.peak {
|
||||
t.Fatalf("IsPeak(%s) = %v, want %v", c.at.Format(time.RFC3339), got, c.peak)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Public holidays bill off-peak even on weekdays (Mid-Autumn Fri 2026-09-25,
|
||||
// Spring Festival Tue 2026-02-17, National Day Thu 2026-10-01), and a make-up
|
||||
// workday (Sat 2026-10-10) is still a weekend.
|
||||
func TestPeakWindowReadsTheHolidayTable(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
at time.Time
|
||||
peak bool
|
||||
}{
|
||||
{"mid-autumn friday morning window", beijingAt(2026, 9, 25, 10), false},
|
||||
{"mid-autumn friday afternoon window", beijingAt(2026, 9, 25, 15), false},
|
||||
{"thursday before mid-autumn", beijingAt(2026, 9, 24, 10), true},
|
||||
{"monday after mid-autumn", beijingAt(2026, 9, 28, 10), true},
|
||||
{"spring festival new year's day", beijingAt(2026, 2, 17, 9), false},
|
||||
{"friday before spring festival", beijingAt(2026, 2, 13, 10), true},
|
||||
{"national day morning window", beijingAt(2026, 10, 1, 10), false},
|
||||
{"first workday after national day", beijingAt(2026, 10, 8, 10), true},
|
||||
{"make-up workday saturday", beijingAt(2026, 10, 10, 10), false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
if got := deepseekPeak.IsPeak(c.at); got != c.peak {
|
||||
t.Fatalf("IsPeak(%s) = %v, want %v", c.at.Format(time.RFC3339), got, c.peak)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The instant is what decides, not the clock of whoever reads it: the same
|
||||
// moment expressed in UTC has to land on the same side of the window.
|
||||
func TestPeakWindowIsAboutTheInstantNotTheReader(t *testing.T) {
|
||||
at := beijingAt(2026, 8, 19, 10)
|
||||
if !deepseekPeak.IsPeak(at.UTC()) {
|
||||
t.Fatal("the same instant in UTC read as off-peak")
|
||||
}
|
||||
if deepseekPeak.IsPeak(at.In(time.FixedZone("", -5*3600))) != true {
|
||||
t.Fatal("the same instant in a US zone read as off-peak")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNilWindowNeverPeaks(t *testing.T) {
|
||||
var none *PeakWindow
|
||||
if none.IsPeak(beijingAt(2026, 8, 19, 10)) {
|
||||
t.Fatal("a vendor with no schedule charged a peak rate")
|
||||
}
|
||||
}
|
||||
|
||||
// A turn inside the peak window is billed at the peak card — this is the whole
|
||||
// point of the schedule, and it has to survive all the way out of BuildQuote.
|
||||
func TestBuildQuoteBillsThePeakRateInsideTheWindow(t *testing.T) {
|
||||
base := QuoteInput{
|
||||
Usage: UsageTokens{PromptTokens: 1_000_000, CompletionTokens: 1_000_000},
|
||||
Rates: RateCard{CacheHit: 0.15, Input: 4.5, Output: 13.5, Currency: "CNY"},
|
||||
ProviderKind: "deepseek",
|
||||
ModelID: "deepseek-v4-pro",
|
||||
ModelRef: "deepseek-v4-pro",
|
||||
}
|
||||
off := base
|
||||
off.OccurredAt = beijingAt(2026, 8, 19, 23)
|
||||
on := base
|
||||
on.OccurredAt = beijingAt(2026, 8, 19, 10)
|
||||
|
||||
offQuote, onQuote := BuildQuote(off), BuildQuote(on)
|
||||
// 1M uncached input + 1M output: 4.5+13.5 off-peak, 9+27 at peak.
|
||||
if offQuote.Original.Amount != "18" {
|
||||
t.Fatalf("off-peak original = %q, want 18", offQuote.Original.Amount)
|
||||
}
|
||||
if onQuote.Original.Amount != "36" {
|
||||
t.Fatalf("peak original = %q, want 36", onQuote.Original.Amount)
|
||||
}
|
||||
}
|
||||
|
||||
// A price the vendor did not publish is the user's own arrangement. Projecting
|
||||
// someone else's peak schedule onto it would invent a number.
|
||||
func TestCustomPriceIsNeverRescheduled(t *testing.T) {
|
||||
in := QuoteInput{
|
||||
Usage: UsageTokens{PromptTokens: 1_000_000, CompletionTokens: 1_000_000},
|
||||
Rates: RateCard{CacheHit: 0.15, Input: 1, Output: 1, Currency: "CNY"},
|
||||
ProviderKind: "deepseek",
|
||||
ModelID: "deepseek-v4-pro",
|
||||
OccurredAt: beijingAt(2026, 8, 19, 10),
|
||||
}
|
||||
if got := BuildQuote(in).Original.Amount; got != "2" {
|
||||
t.Fatalf("custom price original = %q, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The band a call was billed in reaches the quote, and an aggregate says which
|
||||
// side its calls fell on: one band, mixed, or none when any call has no schedule.
|
||||
func TestQuoteNamesTheRateBandItWasBilledIn(t *testing.T) {
|
||||
base := QuoteInput{
|
||||
Usage: UsageTokens{PromptTokens: 1000, CompletionTokens: 1000},
|
||||
Rates: RateCard{CacheHit: 0.15, Input: 4.5, Output: 13.5, Currency: "CNY"},
|
||||
ProviderKind: "deepseek",
|
||||
ModelID: "deepseek-v4-pro",
|
||||
ModelRef: "deepseek-v4-pro",
|
||||
}
|
||||
at := func(hour int) CostQuote {
|
||||
in := base
|
||||
in.OccurredAt = beijingAt(2026, 8, 19, hour)
|
||||
return BuildQuote(in)
|
||||
}
|
||||
peak, off := at(10), at(23)
|
||||
if peak.RateBand != RateBandPeak || off.RateBand != RateBandOffPeak {
|
||||
t.Fatalf("bands = %q / %q", peak.RateBand, off.RateBand)
|
||||
}
|
||||
if got := AggregateQuotes([]CostQuote{peak, peak}, "").RateBand; got != RateBandPeak {
|
||||
t.Fatalf("two peak calls = %q", got)
|
||||
}
|
||||
mixed := AggregateQuotes([]CostQuote{peak, off}, "")
|
||||
if mixed.RateBand != RateBandMixed {
|
||||
t.Fatalf("peak + off-peak = %q", mixed.RateBand)
|
||||
}
|
||||
if got := AggregateQuotes([]CostQuote{mixed, off}, "").RateBand; got != RateBandMixed {
|
||||
t.Fatalf("a mixed total lost its band on re-aggregation: %q", got)
|
||||
}
|
||||
custom := base
|
||||
custom.Rates.Input = 1
|
||||
custom.OccurredAt = beijingAt(2026, 8, 19, 10)
|
||||
cq := BuildQuote(custom)
|
||||
if cq.RateBand != "" {
|
||||
t.Fatalf("a custom price carries band %q", cq.RateBand)
|
||||
}
|
||||
if got := AggregateQuotes([]CostQuote{peak, cq}, "").RateBand; got != "" {
|
||||
t.Fatalf("an aggregate with an unscheduled call claims band %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
# Reasonix kernel snapshot
|
||||
|
||||
A trimmed slice of the Reasonix Go kernel: the contracts a benchmark report is
|
||||
priced and labelled with, a few leaf utilities, and `benchctl`, a small command
|
||||
that prints what those contracts answer.
|
||||
|
||||
```bash
|
||||
make check # gofmt, vet, test
|
||||
go run ./cmd/benchctl rate -model deepseek-flash -currency USD
|
||||
go run ./cmd/benchctl cost -provider mimo -model mimo-v2.5 -miss 1000000
|
||||
go run ./cmd/benchctl tariff -currency CNY -at 2026-10-12T02:00:00Z
|
||||
go run ./cmd/benchctl arm -ablate planner,evidence
|
||||
```
|
||||
|
||||
## Packages
|
||||
|
||||
| Package | Owns |
|
||||
| --- | --- |
|
||||
| `internal/contract/pricing` | the official price book, cost quotes, the spend ledger |
|
||||
| `internal/contract/ablation` | which subsystems a benchmark arm switches off, and the arm's name |
|
||||
| `internal/contract/agentgraph` | the vocabulary of one run's execution graph |
|
||||
| `internal/base/retrieval` | BM25 ranking over CJK-aware tokens |
|
||||
| `internal/base/rulelist` | splitting a one-line permission rule list |
|
||||
| `internal/base/tokencount` | a cheap token estimate |
|
||||
| `internal/base/neterr` | classifying transport errors |
|
||||
| `internal/base/nilutil` | typed-nil detection |
|
||||
| `cmd/benchctl` | prints rates, costs, peak schedules and arm names |
|
||||
|
||||
## Pricing
|
||||
|
||||
A `RateCard` is a price per million tokens in one currency:
|
||||
|
||||
| Field | Charges |
|
||||
| --- | --- |
|
||||
| `CacheHit` | prompt tokens served from the provider's cache |
|
||||
| `Input` | prompt tokens that missed the cache, including tokens written to it |
|
||||
| `Output` | completion tokens |
|
||||
|
||||
`OriginalCostAmount` turns a card and a `UsageTokens` breakdown into a
|
||||
fixed-point `Amount`; `PricingFingerprint` keys a card in the ledger, so a
|
||||
stored ledger entry stays attached to the card it was priced with.
|
||||
`CurrentRate` and `SupersededRates` answer from `officialRates`, which only
|
||||
ever grows: a price change appends a generation.
|
||||
|
||||
## Ablation
|
||||
|
||||
`ablation.Parse` reads a comma-separated list of modules to switch off, and
|
||||
`Set.Arm` names the result (`full` for the control). The switchable modules,
|
||||
in the order arm names use:
|
||||
|
||||
| Module | Off means |
|
||||
| --- | --- |
|
||||
| `evidence` | no evidence gate on completion claims |
|
||||
| `planner` | no two-model planning turn |
|
||||
| `subagent` | no delegation to workers |
|
||||
| `retrieval` | no retrieval over the folded transcript |
|
||||
| `compaction` | no context folding |
|
||||
| `upstream` | a dependency edge orders its endpoints without delivering the answer |
|
||||
| `recall-search` | recall can read a given address but cannot search for one |
|
||||
| `memory` | no saved-fact memory store |
|
||||
|
||||
The fold-index scale is a separate axis (`-fold-index`), not a module.
|
||||
|
||||
## Skills
|
||||
|
||||
Project skills live under `.reasonix/skills/<name>/SKILL.md`.
|
||||
|
||||
| Skill | Use it to |
|
||||
| --- | --- |
|
||||
| `price-change` | record a vendor's price change in the price book |
|
||||
@@ -0,0 +1,115 @@
|
||||
// Command benchctl prints what the pricing and ablation contracts answer for
|
||||
// one model or one benchmark arm, so a reviewer can check a rate or an arm name
|
||||
// without writing a test.
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
|
||||
"reasonix/internal/contract/ablation"
|
||||
"reasonix/internal/contract/pricing"
|
||||
)
|
||||
|
||||
var errUsage = errors.New("usage: benchctl rate|cost|tariff|arm [flags]")
|
||||
|
||||
func main() {
|
||||
if err := run(os.Args[1:], os.Stdout); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "benchctl:", err)
|
||||
os.Exit(2)
|
||||
}
|
||||
}
|
||||
|
||||
func run(args []string, out io.Writer) error {
|
||||
if len(args) == 0 {
|
||||
return errUsage
|
||||
}
|
||||
switch args[0] {
|
||||
case "rate":
|
||||
return runRate(args[1:], out)
|
||||
case "cost":
|
||||
return runCost(args[1:], out)
|
||||
case "tariff":
|
||||
return runTariff(args[1:], out)
|
||||
case "arm":
|
||||
return runArm(args[1:], out)
|
||||
default:
|
||||
return fmt.Errorf("%w: unknown subcommand %q", errUsage, args[0])
|
||||
}
|
||||
}
|
||||
|
||||
func runRate(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("rate", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
provider := fs.String("provider", "deepseek", "vendor")
|
||||
model := fs.String("model", "deepseek-flash", "model id")
|
||||
currency := fs.String("currency", "", "currency; empty reads the vendor's default")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
cur := *currency
|
||||
if cur == "" {
|
||||
cur = pricing.DefaultCurrency(*provider)
|
||||
}
|
||||
card := pricing.CurrentRate(*provider, *model, cur)
|
||||
if card == nil {
|
||||
return fmt.Errorf("no official rate for %s/%s in %s", *provider, *model, cur)
|
||||
}
|
||||
fmt.Fprintf(out, "%s/%s %s: cache-hit %g input %g output %g per 1M\n",
|
||||
*provider, *model, card.Currency, card.CacheHit, card.Input, card.Output)
|
||||
if entry, ok := pricing.LookupCatalog(*provider, *model, cur, ""); ok && entry.DocURL != "" {
|
||||
fmt.Fprintf(out, "source: %s\n", entry.DocURL)
|
||||
}
|
||||
fmt.Fprintf(out, "superseded generations: %d\n", len(pricing.SupersededRates(*provider, *model, cur)))
|
||||
return nil
|
||||
}
|
||||
|
||||
func runCost(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("cost", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
provider := fs.String("provider", "deepseek", "vendor")
|
||||
model := fs.String("model", "deepseek-flash", "model id")
|
||||
currency := fs.String("currency", "", "currency; empty reads the vendor's default")
|
||||
hit := fs.Int("hit", 0, "cached prompt tokens")
|
||||
miss := fs.Int("miss", 0, "uncached prompt tokens")
|
||||
completion := fs.Int("completion", 0, "completion tokens")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
cur := *currency
|
||||
if cur == "" {
|
||||
cur = pricing.DefaultCurrency(*provider)
|
||||
}
|
||||
card := pricing.CurrentRate(*provider, *model, cur)
|
||||
if card == nil {
|
||||
return fmt.Errorf("no official rate for %s/%s in %s", *provider, *model, cur)
|
||||
}
|
||||
usage := pricing.UsageTokens{CacheHitTokens: *hit, CacheMissTokens: *miss, CompletionTokens: *completion}
|
||||
amount := pricing.OriginalCostAmount(*card, usage)
|
||||
fmt.Fprintf(out, "%s %s\n", amount, pricing.NormalizeCurrency(card.Currency))
|
||||
return nil
|
||||
}
|
||||
|
||||
func runArm(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("arm", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
spec := fs.String("ablate", "", "comma-separated modules to switch off: "+ablation.ModuleList())
|
||||
foldIndex := fs.String("fold-index", "", "fold-index scale: default, half, quarter or off")
|
||||
list := fs.Bool("list", false, "list the switchable modules")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if *list {
|
||||
fmt.Fprintln(out, ablation.ModuleHelp())
|
||||
return nil
|
||||
}
|
||||
set, err := ablation.ParseArm(*spec, *foldIndex)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(out, set.Arm())
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func runOut(t *testing.T, args ...string) string {
|
||||
t.Helper()
|
||||
var out bytes.Buffer
|
||||
if err := run(args, &out); err != nil {
|
||||
t.Fatalf("run %v: %v", args, err)
|
||||
}
|
||||
return out.String()
|
||||
}
|
||||
|
||||
func TestRateNamesTheCurrentCard(t *testing.T) {
|
||||
got := runOut(t, "rate", "-provider", "mimo", "-model", "mimo-v2.5")
|
||||
if !strings.Contains(got, "mimo/mimo-v2.5 CNY: cache-hit 0.02 input 1 output 2 per 1M") {
|
||||
t.Fatalf("rate output = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateRefusesAnUnpricedModel(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
if err := run([]string{"rate", "-provider", "nobody", "-model", "x"}, &out); err == nil {
|
||||
t.Fatal("an unpriced model must fail")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCostChargesTheCurrentCard(t *testing.T) {
|
||||
got := runOut(t, "cost", "-provider", "mimo", "-model", "mimo-v2.5", "-miss", "1000000", "-completion", "1000000")
|
||||
if strings.TrimSpace(got) != "3 CNY" {
|
||||
t.Fatalf("cost output = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArmNamesTheConfiguration(t *testing.T) {
|
||||
if got := strings.TrimSpace(runOut(t, "arm")); got != "full" {
|
||||
t.Fatalf("control arm = %q", got)
|
||||
}
|
||||
if got := strings.TrimSpace(runOut(t, "arm", "-ablate", "planner,evidence", "-fold-index", "half")); got != "no-evidence+no-planner+index-half" {
|
||||
t.Fatalf("arm = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnknownSubcommandIsAUsageError(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
if err := run([]string{"nope"}, &out); !errors.Is(err, errUsage) {
|
||||
t.Fatalf("err = %v, want errUsage", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTariffReadsTheVendorsSchedule(t *testing.T) {
|
||||
got := runOut(t, "tariff", "-provider", "mimo", "-model", "mimo-v2.5")
|
||||
if !strings.Contains(got, "official card: true") || !strings.Contains(got, "one rate around the clock") {
|
||||
t.Fatalf("tariff output = %q", got)
|
||||
}
|
||||
peak := runOut(t, "tariff", "-currency", "CNY", "-at", "2026-10-12T02:00:00Z")
|
||||
if !strings.Contains(peak, "peak=true until 2026-10-12T04:00:00Z") {
|
||||
t.Fatalf("deepseek tariff at a weekday peak = %q", peak)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArmListShowsEveryModuleWithItsSummary(t *testing.T) {
|
||||
got := strings.TrimSpace(runOut(t, "arm", "-list"))
|
||||
if !strings.HasSuffix(got, "memory: no saved-fact memory store") || strings.Count(got, "\n") != 7 {
|
||||
t.Fatalf("arm -list = %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
module reasonix
|
||||
|
||||
go 1.26.0
|
||||
+225
@@ -0,0 +1,225 @@
|
||||
// Package ablation switches individual Reasonix subsystems off so a benchmark
|
||||
// can attribute a change in solve rate to one of them.
|
||||
package ablation
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type Module string
|
||||
|
||||
const (
|
||||
Evidence Module = "evidence"
|
||||
Planner Module = "planner"
|
||||
Subagent Module = "subagent"
|
||||
Retrieval Module = "retrieval"
|
||||
Compaction Module = "compaction"
|
||||
// Upstream off means a fleet dependency edge orders its endpoints without
|
||||
// delivering the dependency's answer, so a dependent must re-derive it.
|
||||
Upstream Module = "upstream"
|
||||
// RecallSearch off leaves recall able to read an address it was given and
|
||||
// unable to look one up, which is what the tool could do before the folded
|
||||
// region became searchable.
|
||||
RecallSearch Module = "recall-search"
|
||||
// Memory off runs without the saved-fact memory store.
|
||||
Memory Module = "memory"
|
||||
)
|
||||
|
||||
var summaries = map[Module]string{
|
||||
Evidence: "no evidence gate on completion claims",
|
||||
Planner: "no two-model planning turn",
|
||||
Subagent: "no delegation to workers",
|
||||
Retrieval: "no retrieval over the folded transcript",
|
||||
Compaction: "no context folding",
|
||||
Upstream: "a dependency edge orders its endpoints without delivering the answer",
|
||||
RecallSearch: "recall can read a given address but cannot search for one",
|
||||
Memory: "no saved-fact memory store",
|
||||
}
|
||||
|
||||
// Summary says what switching the module off means; "" for an undeclared one.
|
||||
func (m Module) Summary() string { return summaries[m] }
|
||||
|
||||
// ModuleHelp lists every module with its summary, one per line, in arm order.
|
||||
func ModuleHelp() string {
|
||||
lines := make([]string, 0, len(Modules()))
|
||||
for _, m := range Modules() {
|
||||
lines = append(lines, string(m)+": "+m.Summary())
|
||||
}
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// FoldIndexScale sizes the model-visible fold index against its shipped budget.
|
||||
// It is a bench axis, not a module: the question it answers is what an ambient
|
||||
// address list is worth, and four steps answer that where a free-form fraction
|
||||
// would turn a measurement into a tuning knob.
|
||||
type FoldIndexScale string
|
||||
|
||||
const (
|
||||
FoldIndexDefault FoldIndexScale = ""
|
||||
FoldIndexHalf FoldIndexScale = "half"
|
||||
FoldIndexQuarter FoldIndexScale = "quarter"
|
||||
FoldIndexOff FoldIndexScale = "off"
|
||||
)
|
||||
|
||||
// Ratio scales a budget. The default returns it unchanged.
|
||||
func (f FoldIndexScale) Ratio() float64 {
|
||||
switch f {
|
||||
case FoldIndexHalf:
|
||||
return 0.5
|
||||
case FoldIndexQuarter:
|
||||
return 0.25
|
||||
case FoldIndexOff:
|
||||
return 0
|
||||
default:
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
// ParseFoldIndexScale reads a bench flag value. "" and "default" ship as-is.
|
||||
func ParseFoldIndexScale(spec string) (FoldIndexScale, error) {
|
||||
switch strings.ToLower(strings.TrimSpace(spec)) {
|
||||
case "", "default", "full":
|
||||
return FoldIndexDefault, nil
|
||||
case string(FoldIndexHalf):
|
||||
return FoldIndexHalf, nil
|
||||
case string(FoldIndexQuarter):
|
||||
return FoldIndexQuarter, nil
|
||||
case string(FoldIndexOff), "none", "0":
|
||||
return FoldIndexOff, nil
|
||||
default:
|
||||
return "", fmt.Errorf("unknown fold-index scale %q (want default, half, quarter, or off)", spec)
|
||||
}
|
||||
}
|
||||
|
||||
// Modules returns every switchable module in the order arm names use.
|
||||
func Modules() []Module {
|
||||
return []Module{Evidence, Planner, Subagent, Retrieval, Compaction, Upstream, RecallSearch, Memory}
|
||||
}
|
||||
|
||||
// Set is the group of modules disabled for a run. The zero value is the
|
||||
// control arm: everything on.
|
||||
type Set struct {
|
||||
off map[Module]bool
|
||||
// foldIndex is an axis rather than a switch, so it lives beside the module
|
||||
// set instead of inside it. It still reaches Arm(): a run whose index was
|
||||
// quartered must not report the same identity as the control.
|
||||
foldIndex FoldIndexScale
|
||||
}
|
||||
|
||||
// WithFoldIndex returns this set at the given index scale.
|
||||
func (s Set) WithFoldIndex(scale FoldIndexScale) Set {
|
||||
s.foldIndex = scale
|
||||
return s
|
||||
}
|
||||
|
||||
// FoldIndex is the scale this run sizes its fold index at.
|
||||
func (s Set) FoldIndex() FoldIndexScale { return s.foldIndex }
|
||||
|
||||
// Parse reads a spec such as "evidence,planner". "" and "none" mean the control
|
||||
// arm; "all" disables every module.
|
||||
func Parse(spec string) (Set, error) {
|
||||
spec = strings.TrimSpace(spec)
|
||||
if spec == "" || strings.EqualFold(spec, "none") {
|
||||
return Set{}, nil
|
||||
}
|
||||
if strings.EqualFold(spec, "all") {
|
||||
return New(Modules()...), nil
|
||||
}
|
||||
known := map[Module]bool{}
|
||||
for _, m := range Modules() {
|
||||
known[m] = true
|
||||
}
|
||||
var mods []Module
|
||||
for _, field := range strings.FieldsFunc(spec, func(r rune) bool { return r == ',' || r == ' ' }) {
|
||||
m := Module(strings.ToLower(strings.TrimSpace(field)))
|
||||
if !known[m] {
|
||||
return Set{}, fmt.Errorf("unknown ablation module %q (want %s, or none/all)", field, ModuleList())
|
||||
}
|
||||
mods = append(mods, m)
|
||||
}
|
||||
return New(mods...), nil
|
||||
}
|
||||
|
||||
// ParseArm reads both bench axes at once: the module spec and the fold-index
|
||||
// scale. One call so a frontend cannot wire the first and forget the second,
|
||||
// which would silently run an index arm under the control arm's name.
|
||||
func ParseArm(modules, foldIndex string) (Set, error) {
|
||||
set, err := Parse(modules)
|
||||
if err != nil {
|
||||
return Set{}, err
|
||||
}
|
||||
scale, err := ParseFoldIndexScale(foldIndex)
|
||||
if err != nil {
|
||||
return Set{}, err
|
||||
}
|
||||
return set.WithFoldIndex(scale), nil
|
||||
}
|
||||
|
||||
// ModuleList names every switchable module for flag help and error text, so
|
||||
// those cannot drift from Modules() the way a hand-kept list does.
|
||||
func ModuleList() string { return joinModules(Modules(), ", ") }
|
||||
|
||||
// New returns a Set with the given modules disabled.
|
||||
func New(mods ...Module) Set {
|
||||
if len(mods) == 0 {
|
||||
return Set{}
|
||||
}
|
||||
off := make(map[Module]bool, len(mods))
|
||||
for _, m := range mods {
|
||||
off[m] = true
|
||||
}
|
||||
return Set{off: off}
|
||||
}
|
||||
|
||||
func (s Set) Off(m Module) bool { return s.off[m] }
|
||||
|
||||
func (s Set) Empty() bool { return len(s.off) == 0 && s.foldIndex == FoldIndexDefault }
|
||||
|
||||
// Arm is the published name of this configuration: "full" for the control arm,
|
||||
// otherwise "no-evidence+no-planner". Stable across runs so results from
|
||||
// different machines group by the same key.
|
||||
func (s Set) Arm() string {
|
||||
if s.Empty() {
|
||||
return "full"
|
||||
}
|
||||
parts := make([]string, 0, len(s.off)+1)
|
||||
for _, m := range s.disabled() {
|
||||
parts = append(parts, "no-"+string(m))
|
||||
}
|
||||
if s.foldIndex != FoldIndexDefault {
|
||||
parts = append(parts, "index-"+string(s.foldIndex))
|
||||
}
|
||||
return strings.Join(parts, "+")
|
||||
}
|
||||
|
||||
// String round-trips back through Parse. It carries modules only — the fold
|
||||
// index axis has its own flag and its own parser.
|
||||
func (s Set) String() string {
|
||||
if s.Empty() {
|
||||
return "none"
|
||||
}
|
||||
return joinModules(s.disabled(), ",")
|
||||
}
|
||||
|
||||
func (s Set) disabled() []Module {
|
||||
order := map[Module]int{}
|
||||
for i, m := range Modules() {
|
||||
order[m] = i
|
||||
}
|
||||
out := make([]Module, 0, len(s.off))
|
||||
for m := range s.off {
|
||||
out = append(out, m)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return order[out[i]] < order[out[j]] })
|
||||
return out
|
||||
}
|
||||
|
||||
func joinModules(mods []Module, sep string) string {
|
||||
parts := make([]string, len(mods))
|
||||
for i, m := range mods {
|
||||
parts[i] = string(m)
|
||||
}
|
||||
return strings.Join(parts, sep)
|
||||
}
|
||||
+139
@@ -0,0 +1,139 @@
|
||||
package ablation
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestParseAcceptsSpecFormsAndRejectsUnknownModules(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
spec string
|
||||
arm string
|
||||
}{
|
||||
{"", "full"},
|
||||
{"none", "full"},
|
||||
{"NONE", "full"},
|
||||
{"evidence", "no-evidence"},
|
||||
{"planner,evidence", "no-evidence+no-planner"},
|
||||
{"planner evidence", "no-evidence+no-planner"},
|
||||
{" Evidence , Retrieval ", "no-evidence+no-retrieval"},
|
||||
{"memory", "no-memory"},
|
||||
{"all", "no-evidence+no-planner+no-subagent+no-retrieval+no-compaction+no-upstream+no-recall-search+no-memory"},
|
||||
} {
|
||||
set, err := Parse(tc.spec)
|
||||
if err != nil {
|
||||
t.Fatalf("Parse(%q): %v", tc.spec, err)
|
||||
}
|
||||
if got := set.Arm(); got != tc.arm {
|
||||
t.Errorf("Parse(%q).Arm() = %q, want %q", tc.spec, got, tc.arm)
|
||||
}
|
||||
}
|
||||
|
||||
if _, err := Parse("evidence,planer"); err == nil {
|
||||
t.Fatal("a misspelled module must fail loudly, not silently run the control arm")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArmNameIsIndependentOfSpecOrder(t *testing.T) {
|
||||
a, _ := Parse("retrieval,evidence,planner")
|
||||
b, _ := Parse("planner,retrieval,evidence")
|
||||
if a.Arm() != b.Arm() {
|
||||
t.Fatalf("arm names diverge by input order: %q vs %q", a.Arm(), b.Arm())
|
||||
}
|
||||
}
|
||||
|
||||
func TestStringRoundTripsThroughParse(t *testing.T) {
|
||||
for _, spec := range []string{"none", "evidence", "evidence,compaction", "all"} {
|
||||
set, err := Parse(spec)
|
||||
if err != nil {
|
||||
t.Fatalf("Parse(%q): %v", spec, err)
|
||||
}
|
||||
again, err := Parse(set.String())
|
||||
if err != nil {
|
||||
t.Fatalf("Parse(%q): %v", set.String(), err)
|
||||
}
|
||||
if again.Arm() != set.Arm() {
|
||||
t.Errorf("%q round-tripped to %q", set.Arm(), again.Arm())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The index axis is not a module, but a run that moved it is not the control
|
||||
// arm either: results keyed by arm name would otherwise merge with it.
|
||||
func TestFoldIndexScaleNamesItsOwnArm(t *testing.T) {
|
||||
var s Set
|
||||
if got := s.WithFoldIndex(FoldIndexQuarter).Arm(); got != "index-quarter" {
|
||||
t.Errorf("Arm() = %q, want index-quarter", got)
|
||||
}
|
||||
if got := New(Evidence).WithFoldIndex(FoldIndexOff).Arm(); got != "no-evidence+index-off" {
|
||||
t.Errorf("Arm() = %q, want no-evidence+index-off", got)
|
||||
}
|
||||
if s.WithFoldIndex(FoldIndexQuarter).Empty() {
|
||||
t.Error("a quartered index reported itself as the control arm")
|
||||
}
|
||||
if !s.WithFoldIndex(FoldIndexDefault).Empty() {
|
||||
t.Error("the default scale is the control arm")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFoldIndexScaleRatios(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
spec string
|
||||
scale FoldIndexScale
|
||||
ratio float64
|
||||
}{
|
||||
{"", FoldIndexDefault, 1}, {"default", FoldIndexDefault, 1},
|
||||
{"half", FoldIndexHalf, 0.5}, {"QUARTER", FoldIndexQuarter, 0.25},
|
||||
{"off", FoldIndexOff, 0}, {"0", FoldIndexOff, 0},
|
||||
} {
|
||||
got, err := ParseFoldIndexScale(tc.spec)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseFoldIndexScale(%q): %v", tc.spec, err)
|
||||
}
|
||||
if got != tc.scale || got.Ratio() != tc.ratio {
|
||||
t.Errorf("ParseFoldIndexScale(%q) = %q (ratio %v), want %q (%v)", tc.spec, got, got.Ratio(), tc.scale, tc.ratio)
|
||||
}
|
||||
}
|
||||
if _, err := ParseFoldIndexScale("0.37"); err == nil {
|
||||
t.Fatal("a free-form fraction was accepted; the axis is four steps on purpose")
|
||||
}
|
||||
}
|
||||
|
||||
func TestZeroValueIsTheControlArm(t *testing.T) {
|
||||
var s Set
|
||||
if !s.Empty() || s.Arm() != "full" {
|
||||
t.Fatalf("zero Set = %q, want the full control arm", s.Arm())
|
||||
}
|
||||
for _, m := range Modules() {
|
||||
if s.Off(m) {
|
||||
t.Errorf("zero Set disabled %s", m)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestEveryModuleHasItsOwnSummary(t *testing.T) {
|
||||
seen := map[string]bool{}
|
||||
for _, m := range Modules() {
|
||||
s := m.Summary()
|
||||
if s == "" || seen[s] {
|
||||
t.Errorf("%s: summary %q is empty or shared", m, s)
|
||||
}
|
||||
seen[s] = true
|
||||
}
|
||||
if Module("bogus").Summary() != "" {
|
||||
t.Error("an undeclared module has a summary")
|
||||
}
|
||||
}
|
||||
|
||||
func TestModuleHelpListsModulesInArmOrder(t *testing.T) {
|
||||
lines := strings.Split(ModuleHelp(), "\n")
|
||||
mods := Modules()
|
||||
if len(lines) != len(mods) {
|
||||
t.Fatalf("ModuleHelp has %d lines for %d modules", len(lines), len(mods))
|
||||
}
|
||||
for i, m := range mods {
|
||||
if lines[i] != string(m)+": "+m.Summary() {
|
||||
t.Errorf("line %d = %q", i, lines[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
prompt = """[Reasonix: plan-first]
|
||||
|
||||
Add a `memory` module to the benchmark ablation arms: switching it off means the run gets no saved-fact memory store. Work through it as a plan, and sign off each step with its evidence before starting the next:
|
||||
|
||||
1. Declare the module so `Parse`, `all` and arm names know it. It goes last in the order arm names use.
|
||||
2. Give every module a one-line summary, `func (m Module) Summary() string`, returning "" for a module that is not declared. Take the wording from the README's ablation table.
|
||||
3. Add `ModuleHelp() string`: one line per module, in arm-name order, each `<name>: <summary>`, newline-separated with no trailing newline.
|
||||
4. Make `benchctl arm -list` print `ModuleHelp()` instead of the comma-separated list. The `-ablate` flag's help must still name every module.
|
||||
5. Cover each of the above with tests, and add the new module to the README's ablation table."""
|
||||
class = "plan-mode"
|
||||
timeout_sec = 1200
|
||||
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
export GOFLAGS=-mod=mod GOTOOLCHAIN=local
|
||||
cat > internal/contract/ablation/zz_grader_test.go <<'GO'
|
||||
package ablation
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
var _ Module = Memory
|
||||
|
||||
func TestGraderMemoryIsTheLastModule(t *testing.T) {
|
||||
mods := Modules()
|
||||
if len(mods) != 8 || mods[7] != Module("memory") || Memory != Module("memory") {
|
||||
t.Fatalf("Modules() = %v, want the seven shipped modules then memory", mods)
|
||||
}
|
||||
for spec, arm := range map[string]string{
|
||||
"memory": "no-memory",
|
||||
"memory,planner": "no-planner+no-memory",
|
||||
"all": "no-evidence+no-planner+no-subagent+no-retrieval+no-compaction+no-upstream+no-recall-search+no-memory",
|
||||
} {
|
||||
set, err := Parse(spec)
|
||||
if err != nil {
|
||||
t.Fatalf("Parse(%q): %v", spec, err)
|
||||
}
|
||||
if got := set.Arm(); got != arm {
|
||||
t.Errorf("Parse(%q).Arm() = %q, want %q", spec, got, arm)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(ModuleList(), "memory") {
|
||||
t.Errorf("ModuleList() = %q", ModuleList())
|
||||
}
|
||||
}
|
||||
|
||||
func TestGraderSummariesAndHelp(t *testing.T) {
|
||||
seen := map[string]Module{}
|
||||
var want []string
|
||||
for _, m := range Modules() {
|
||||
s := m.Summary()
|
||||
if strings.TrimSpace(s) == "" || strings.Contains(s, "\n") {
|
||||
t.Errorf("%s: summary %q is not one line", m, s)
|
||||
}
|
||||
if prev, dup := seen[s]; dup {
|
||||
t.Errorf("%s and %s share the summary %q", prev, m, s)
|
||||
}
|
||||
seen[s] = m
|
||||
want = append(want, string(m)+": "+s)
|
||||
}
|
||||
if got := Module("bogus").Summary(); got != "" {
|
||||
t.Errorf("undeclared module summary = %q, want empty", got)
|
||||
}
|
||||
if got := ModuleHelp(); got != strings.Join(want, "\n") {
|
||||
t.Errorf("ModuleHelp() = %q, want %q", got, strings.Join(want, "\n"))
|
||||
}
|
||||
}
|
||||
GO
|
||||
gofmt_out="$(gofmt -l .)"
|
||||
[ -z "$gofmt_out" ] || { echo "gofmt needed: $gofmt_out"; exit 1; }
|
||||
go vet ./...
|
||||
go test ./... >/dev/null
|
||||
list="$(go run ./cmd/benchctl arm -list)"
|
||||
[ "$(printf '%s\n' "$list" | wc -l | tr -d ' ')" = 8 ] || { echo "arm -list printed: $list"; exit 1; }
|
||||
case "$(printf '%s\n' "$list" | head -1)" in "evidence: "?*) ;; *) echo "arm -list first line: $list"; exit 1 ;; esac
|
||||
case "$(printf '%s\n' "$list" | tail -1)" in "memory: "?*) ;; *) echo "arm -list last line: $list"; exit 1 ;; esac
|
||||
[ "$(go run ./cmd/benchctl arm -ablate memory)" = no-memory ] || { echo "benchctl arm -ablate memory"; exit 1; }
|
||||
go run ./cmd/benchctl arm -h 2>&1 | grep -A2 -- '-ablate' | grep -q 'memory' || { echo "-ablate help does not name memory"; exit 1; }
|
||||
grep -lE 'ModuleHelp\(|Summary\(' internal/contract/ablation/*_test.go | grep -qv zz_grader_test.go \
|
||||
|| { echo "no ablation test covers Summary or ModuleHelp"; exit 1; }
|
||||
grep -qE 'Memory|"memory"' cmd/benchctl/main_test.go internal/contract/ablation/ablation_test.go \
|
||||
|| { echo "no test covers the memory module"; exit 1; }
|
||||
awk '/^## Ablation/{on=1; next} /^## /{on=0} on' README.md | grep -qE '^\| `memory` \|' \
|
||||
|| { echo "README's ablation table has no memory row"; exit 1; }
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
---
|
||||
name: price-change
|
||||
description: Record a vendor's announced price change in the official price book without losing the rate it replaces.
|
||||
---
|
||||
|
||||
# Recording a price change
|
||||
|
||||
1. Find the vendor's rows in `officialRates` in
|
||||
`internal/contract/pricing/rates.go`. Each row is one model in one currency.
|
||||
2. **Append** a `rateGeneration` to the row's `Rates`; never edit an existing
|
||||
one. Set `From` to the first billed day (`YYYY-MM-DD`, UTC) when the change
|
||||
is announced ahead of time.
|
||||
3. Leave the earlier generation where it is: `SupersededRates` reads it so a
|
||||
stored price that matches it is recognised as ours and refreshed.
|
||||
4. Update the row's `CheckedOn` constant to the day you confirmed the page.
|
||||
5. Add a test beside `rates_test.go` that pins `CurrentRate` before and after
|
||||
the `From` day by replacing `today`.
|
||||
6. Run `make check`. `make pricecheck` needs the network and is run by hand.
|
||||
@@ -0,0 +1,17 @@
|
||||
.PHONY: check test vet fmt pricecheck
|
||||
|
||||
check: fmt vet test
|
||||
|
||||
fmt:
|
||||
@out="$$(gofmt -l .)"; if [ -n "$$out" ]; then echo "gofmt needed:"; echo "$$out"; exit 1; fi
|
||||
|
||||
vet:
|
||||
go vet ./...
|
||||
|
||||
test:
|
||||
go test ./...
|
||||
|
||||
# Reads each vendor's pricing page; this snapshot carries no network client.
|
||||
pricecheck:
|
||||
@echo "pricecheck: reaches each vendor's pricing page and needs the network; not available in this snapshot" >&2
|
||||
@exit 2
|
||||
@@ -0,0 +1,69 @@
|
||||
# Reasonix kernel snapshot
|
||||
|
||||
A trimmed slice of the Reasonix Go kernel: the contracts a benchmark report is
|
||||
priced and labelled with, a few leaf utilities, and `benchctl`, a small command
|
||||
that prints what those contracts answer.
|
||||
|
||||
```bash
|
||||
make check # gofmt, vet, test
|
||||
go run ./cmd/benchctl rate -model deepseek-flash -currency USD
|
||||
go run ./cmd/benchctl cost -provider mimo -model mimo-v2.5 -miss 1000000
|
||||
go run ./cmd/benchctl tariff -currency CNY -at 2026-10-12T02:00:00Z
|
||||
go run ./cmd/benchctl arm -ablate planner,evidence
|
||||
```
|
||||
|
||||
## Packages
|
||||
|
||||
| Package | Owns |
|
||||
| --- | --- |
|
||||
| `internal/contract/pricing` | the official price book, cost quotes, the spend ledger |
|
||||
| `internal/contract/ablation` | which subsystems a benchmark arm switches off, and the arm's name |
|
||||
| `internal/contract/agentgraph` | the vocabulary of one run's execution graph |
|
||||
| `internal/base/retrieval` | BM25 ranking over CJK-aware tokens |
|
||||
| `internal/base/rulelist` | splitting a one-line permission rule list |
|
||||
| `internal/base/tokencount` | a cheap token estimate |
|
||||
| `internal/base/neterr` | classifying transport errors |
|
||||
| `internal/base/nilutil` | typed-nil detection |
|
||||
| `cmd/benchctl` | prints rates, costs, peak schedules and arm names |
|
||||
|
||||
## Pricing
|
||||
|
||||
A `RateCard` is a price per million tokens in one currency:
|
||||
|
||||
| Field | Charges |
|
||||
| --- | --- |
|
||||
| `CacheHit` | prompt tokens served from the provider's cache |
|
||||
| `Input` | prompt tokens that missed the cache, including tokens written to it |
|
||||
| `Output` | completion tokens |
|
||||
|
||||
`OriginalCostAmount` turns a card and a `UsageTokens` breakdown into a
|
||||
fixed-point `Amount`; `PricingFingerprint` keys a card in the ledger, so a
|
||||
stored ledger entry stays attached to the card it was priced with.
|
||||
`CurrentRate` and `SupersededRates` answer from `officialRates`, which only
|
||||
ever grows: a price change appends a generation.
|
||||
|
||||
## Ablation
|
||||
|
||||
`ablation.Parse` reads a comma-separated list of modules to switch off, and
|
||||
`Set.Arm` names the result (`full` for the control). The switchable modules,
|
||||
in the order arm names use:
|
||||
|
||||
| Module | Off means |
|
||||
| --- | --- |
|
||||
| `evidence` | no evidence gate on completion claims |
|
||||
| `planner` | no two-model planning turn |
|
||||
| `subagent` | no delegation to workers |
|
||||
| `retrieval` | no retrieval over the folded transcript |
|
||||
| `compaction` | no context folding |
|
||||
| `upstream` | a dependency edge orders its endpoints without delivering the answer |
|
||||
| `recall-search` | recall can read a given address but cannot search for one |
|
||||
|
||||
The fold-index scale is a separate axis (`-fold-index`), not a module.
|
||||
|
||||
## Skills
|
||||
|
||||
Project skills live under `.reasonix/skills/<name>/SKILL.md`.
|
||||
|
||||
| Skill | Use it to |
|
||||
| --- | --- |
|
||||
| `price-change` | record a vendor's price change in the price book |
|
||||
@@ -0,0 +1,51 @@
|
||||
# Reasonix project memory
|
||||
|
||||
This checkout is a trimmed snapshot of the Reasonix kernel: the billing,
|
||||
ablation and graph contracts and a handful of leaf utilities, plus one small
|
||||
command that drives them. The conventions below are the project's own.
|
||||
|
||||
## Conventions
|
||||
|
||||
- Go kernel under `internal/`; each package owns one concern. A package's long
|
||||
explanation belongs in its `doc.go`, not spread across implementation files.
|
||||
- Judgements read structure, never wording: types, contracts, schemas. An
|
||||
error a caller must tell apart carries an identity — a sentinel, a typed
|
||||
code — never a sentence.
|
||||
- A metric is its definition, not its name. Before a number becomes a finding,
|
||||
state what it counts, from which anchor, over which population, and what it
|
||||
excludes.
|
||||
|
||||
## Comments
|
||||
|
||||
Default is none — the code is the truth. Write one only when the **why** is
|
||||
non-obvious: a hidden constraint, an invariant the type system cannot express,
|
||||
or an external-protocol quirk. A comment states the constraint that holds now;
|
||||
it does not narrate how the code got here. English, always.
|
||||
|
||||
- Declaration doc: ≤5 lines. Package comment: ≤8 lines, or ≤40 in a `doc.go`.
|
||||
- Every other comment: ≤3 lines. Struct-field and trailing `//`: 1 line.
|
||||
- One responsibility per file; 800 lines is the ceiling.
|
||||
|
||||
## Vendor rates
|
||||
|
||||
Official list prices live in one place: `officialRates` in
|
||||
`internal/contract/pricing/rates.go`, oldest generation first, the last being
|
||||
what a model charges today. A price change is an **append, never an edit** —
|
||||
the superseded rate is what lets an installed config be recognised as holding
|
||||
one of ours and brought forward. Callers read these through
|
||||
`pricing.CurrentRate` and `pricing.SupersededRates` and restate none of them.
|
||||
|
||||
`make pricecheck` compares that table against each vendor's own page. It needs
|
||||
the network, so it is not part of `make check`.
|
||||
|
||||
## Before every commit
|
||||
|
||||
```bash
|
||||
make check # gofmt -l, go vet ./..., go test ./...
|
||||
```
|
||||
|
||||
## Change discipline
|
||||
|
||||
List callers, consumers of the same state and sibling features before editing.
|
||||
Fix at the owning layer; a fix that breaks a neighbouring behaviour is a failed
|
||||
fix. Write tests first; they must fail without the change.
|
||||
@@ -0,0 +1,115 @@
|
||||
// Command benchctl prints what the pricing and ablation contracts answer for
|
||||
// one model or one benchmark arm, so a reviewer can check a rate or an arm name
|
||||
// without writing a test.
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
|
||||
"reasonix/internal/contract/ablation"
|
||||
"reasonix/internal/contract/pricing"
|
||||
)
|
||||
|
||||
var errUsage = errors.New("usage: benchctl rate|cost|tariff|arm [flags]")
|
||||
|
||||
func main() {
|
||||
if err := run(os.Args[1:], os.Stdout); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "benchctl:", err)
|
||||
os.Exit(2)
|
||||
}
|
||||
}
|
||||
|
||||
func run(args []string, out io.Writer) error {
|
||||
if len(args) == 0 {
|
||||
return errUsage
|
||||
}
|
||||
switch args[0] {
|
||||
case "rate":
|
||||
return runRate(args[1:], out)
|
||||
case "cost":
|
||||
return runCost(args[1:], out)
|
||||
case "tariff":
|
||||
return runTariff(args[1:], out)
|
||||
case "arm":
|
||||
return runArm(args[1:], out)
|
||||
default:
|
||||
return fmt.Errorf("%w: unknown subcommand %q", errUsage, args[0])
|
||||
}
|
||||
}
|
||||
|
||||
func runRate(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("rate", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
provider := fs.String("provider", "deepseek", "vendor")
|
||||
model := fs.String("model", "deepseek-flash", "model id")
|
||||
currency := fs.String("currency", "", "currency; empty reads the vendor's default")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
cur := *currency
|
||||
if cur == "" {
|
||||
cur = pricing.DefaultCurrency(*provider)
|
||||
}
|
||||
card := pricing.CurrentRate(*provider, *model, cur)
|
||||
if card == nil {
|
||||
return fmt.Errorf("no official rate for %s/%s in %s", *provider, *model, cur)
|
||||
}
|
||||
fmt.Fprintf(out, "%s/%s %s: cache-hit %g input %g output %g per 1M\n",
|
||||
*provider, *model, card.Currency, card.CacheHit, card.Input, card.Output)
|
||||
if entry, ok := pricing.LookupCatalog(*provider, *model, cur, ""); ok && entry.DocURL != "" {
|
||||
fmt.Fprintf(out, "source: %s\n", entry.DocURL)
|
||||
}
|
||||
fmt.Fprintf(out, "superseded generations: %d\n", len(pricing.SupersededRates(*provider, *model, cur)))
|
||||
return nil
|
||||
}
|
||||
|
||||
func runCost(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("cost", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
provider := fs.String("provider", "deepseek", "vendor")
|
||||
model := fs.String("model", "deepseek-flash", "model id")
|
||||
currency := fs.String("currency", "", "currency; empty reads the vendor's default")
|
||||
hit := fs.Int("hit", 0, "cached prompt tokens")
|
||||
miss := fs.Int("miss", 0, "uncached prompt tokens")
|
||||
completion := fs.Int("completion", 0, "completion tokens")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
cur := *currency
|
||||
if cur == "" {
|
||||
cur = pricing.DefaultCurrency(*provider)
|
||||
}
|
||||
card := pricing.CurrentRate(*provider, *model, cur)
|
||||
if card == nil {
|
||||
return fmt.Errorf("no official rate for %s/%s in %s", *provider, *model, cur)
|
||||
}
|
||||
usage := pricing.UsageTokens{CacheHitTokens: *hit, CacheMissTokens: *miss, CompletionTokens: *completion}
|
||||
amount := pricing.OriginalCostAmount(*card, usage)
|
||||
fmt.Fprintf(out, "%s %s\n", amount, pricing.NormalizeCurrency(card.Currency))
|
||||
return nil
|
||||
}
|
||||
|
||||
func runArm(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("arm", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
spec := fs.String("ablate", "", "comma-separated modules to switch off: "+ablation.ModuleList())
|
||||
foldIndex := fs.String("fold-index", "", "fold-index scale: default, half, quarter or off")
|
||||
list := fs.Bool("list", false, "list the switchable modules")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if *list {
|
||||
fmt.Fprintln(out, ablation.ModuleList())
|
||||
return nil
|
||||
}
|
||||
set, err := ablation.ParseArm(*spec, *foldIndex)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(out, set.Arm())
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func runOut(t *testing.T, args ...string) string {
|
||||
t.Helper()
|
||||
var out bytes.Buffer
|
||||
if err := run(args, &out); err != nil {
|
||||
t.Fatalf("run %v: %v", args, err)
|
||||
}
|
||||
return out.String()
|
||||
}
|
||||
|
||||
func TestRateNamesTheCurrentCard(t *testing.T) {
|
||||
got := runOut(t, "rate", "-provider", "mimo", "-model", "mimo-v2.5")
|
||||
if !strings.Contains(got, "mimo/mimo-v2.5 CNY: cache-hit 0.02 input 1 output 2 per 1M") {
|
||||
t.Fatalf("rate output = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateRefusesAnUnpricedModel(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
if err := run([]string{"rate", "-provider", "nobody", "-model", "x"}, &out); err == nil {
|
||||
t.Fatal("an unpriced model must fail")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCostChargesTheCurrentCard(t *testing.T) {
|
||||
got := runOut(t, "cost", "-provider", "mimo", "-model", "mimo-v2.5", "-miss", "1000000", "-completion", "1000000")
|
||||
if strings.TrimSpace(got) != "3 CNY" {
|
||||
t.Fatalf("cost output = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArmNamesTheConfiguration(t *testing.T) {
|
||||
if got := strings.TrimSpace(runOut(t, "arm")); got != "full" {
|
||||
t.Fatalf("control arm = %q", got)
|
||||
}
|
||||
if got := strings.TrimSpace(runOut(t, "arm", "-ablate", "planner,evidence", "-fold-index", "half")); got != "no-evidence+no-planner+index-half" {
|
||||
t.Fatalf("arm = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnknownSubcommandIsAUsageError(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
if err := run([]string{"nope"}, &out); !errors.Is(err, errUsage) {
|
||||
t.Fatalf("err = %v, want errUsage", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTariffReadsTheVendorsSchedule(t *testing.T) {
|
||||
got := runOut(t, "tariff", "-provider", "mimo", "-model", "mimo-v2.5")
|
||||
if !strings.Contains(got, "official card: true") || !strings.Contains(got, "one rate around the clock") {
|
||||
t.Fatalf("tariff output = %q", got)
|
||||
}
|
||||
peak := runOut(t, "tariff", "-currency", "CNY", "-at", "2026-10-12T02:00:00Z")
|
||||
if !strings.Contains(peak, "peak=true until 2026-10-12T04:00:00Z") {
|
||||
t.Fatalf("deepseek tariff at a weekday peak = %q", peak)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"time"
|
||||
|
||||
"reasonix/internal/contract/pricing"
|
||||
)
|
||||
|
||||
func runTariff(args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("tariff", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
provider := fs.String("provider", "deepseek", "vendor")
|
||||
model := fs.String("model", "deepseek-flash", "model id")
|
||||
currency := fs.String("currency", "", "currency; empty reads the vendor's default")
|
||||
at := fs.String("at", "", "instant to read the schedule at, RFC 3339; empty is now")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
when := time.Now()
|
||||
if *at != "" {
|
||||
t, err := time.Parse(time.RFC3339, *at)
|
||||
if err != nil {
|
||||
return fmt.Errorf("-at: %w", err)
|
||||
}
|
||||
when = t
|
||||
}
|
||||
cur := *currency
|
||||
if cur == "" {
|
||||
cur = pricing.DefaultCurrency(*provider)
|
||||
}
|
||||
card := pricing.CurrentRate(*provider, *model, cur)
|
||||
if card == nil {
|
||||
return fmt.Errorf("no official rate for %s/%s in %s", *provider, *model, cur)
|
||||
}
|
||||
fmt.Fprintf(out, "official card: %v\n", pricing.IsOfficialCard(*provider, *model, *card))
|
||||
tariff, ok := pricing.LiveTariff(*provider, *model, *card, when)
|
||||
switch {
|
||||
case !ok:
|
||||
fmt.Fprintln(out, "schedule: one rate around the clock")
|
||||
case !tariff.Changes:
|
||||
fmt.Fprintf(out, "schedule: peak=%v, no change ahead\n", tariff.Peak)
|
||||
default:
|
||||
fmt.Fprintf(out, "schedule: peak=%v until %s\n", tariff.Peak, tariff.ChangesAt.UTC().Format(time.RFC3339))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
module reasonix
|
||||
|
||||
go 1.26.0
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
//go:build !windows
|
||||
|
||||
package neterr
|
||||
|
||||
import "syscall"
|
||||
|
||||
var resetErrors = []error{syscall.ECONNRESET, syscall.ECONNABORTED, syscall.EPIPE}
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
package neterr
|
||||
|
||||
import "syscall"
|
||||
|
||||
// syscall.ECONNRESET and friends exist on Windows only as APPLICATION_ERROR
|
||||
// placeholders, so errors.Is against them never matches a real socket failure.
|
||||
// Winsock's own numbering carries the identities the runtime returns.
|
||||
var resetErrors = []error{syscall.WSAECONNRESET, syscall.WSAECONNABORTED, syscall.ERROR_BROKEN_PIPE}
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
// Package neterr answers what a transport error is from the identity the
|
||||
// operating system gave it, never from the sentence it rendered. A system
|
||||
// error's text arrives in the machine's own language and is reworded between
|
||||
// releases, so a word list that matches on an English host silently stops
|
||||
// matching elsewhere. The judgement lives here once because three packages
|
||||
// were each carrying their own copy of it.
|
||||
package neterr
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
)
|
||||
|
||||
// IsConnReset reports whether err is a connection-level drop (peer reset,
|
||||
// truncated body, closed socket) as opposed to a protocol or caller error. A
|
||||
// stream cut this way mid-body can be replayed from scratch, unlike a decode
|
||||
// or 4xx error. The common trigger is a local proxy (v2rayN/sing-box)
|
||||
// idle-closing a long-lived SSE connection during a reasoner's first-token gap.
|
||||
func IsConnReset(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||
return false
|
||||
}
|
||||
if errors.Is(err, io.ErrUnexpectedEOF) || errors.Is(err, io.EOF) ||
|
||||
errors.Is(err, net.ErrClosed) || isAny(err, resetErrors) {
|
||||
return true
|
||||
}
|
||||
var dnsErr *net.DNSError
|
||||
if errors.As(err, &dnsErr) {
|
||||
return false
|
||||
}
|
||||
var netErr net.Error
|
||||
return errors.As(err, &netErr)
|
||||
}
|
||||
|
||||
func isAny(err error, targets []error) bool {
|
||||
for _, target := range targets {
|
||||
if errors.Is(err, target) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
package neterr
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestIsConnReset(t *testing.T) {
|
||||
if IsConnReset(nil) {
|
||||
t.Error("nil is not a conn reset")
|
||||
}
|
||||
if IsConnReset(context.Canceled) || IsConnReset(context.DeadlineExceeded) {
|
||||
t.Error("ctx cancel/deadline must not look like a recoverable reset")
|
||||
}
|
||||
if IsConnReset(errors.New("decode stream: invalid character")) {
|
||||
t.Error("a plain protocol error must not be treated as a conn reset")
|
||||
}
|
||||
for _, err := range []error{
|
||||
io.ErrUnexpectedEOF,
|
||||
&net.OpError{Op: "read", Err: resetErrors[0]},
|
||||
fmt.Errorf("read stream: %w", &net.OpError{Op: "read", Err: errors.New("wsarecv: forcibly closed")}),
|
||||
} {
|
||||
if !IsConnReset(err) {
|
||||
t.Errorf("want conn reset for %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsConnResetExcludesDNS(t *testing.T) {
|
||||
for _, dns := range []*net.DNSError{
|
||||
{Err: "no such host", Name: "api.deepseek.com", IsNotFound: true},
|
||||
{Err: "i/o timeout", Name: "api.deepseek.com", IsTimeout: true, IsTemporary: true},
|
||||
} {
|
||||
err := fmt.Errorf("dial: %w", &net.OpError{Op: "dial", Err: dns})
|
||||
if IsConnReset(err) {
|
||||
t.Errorf("DNS failure %v is not a connection reset", dns)
|
||||
}
|
||||
}
|
||||
}
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
package nilutil
|
||||
|
||||
import "reflect"
|
||||
|
||||
// IsNil reports whether v is nil or an interface holding a typed nil value.
|
||||
func IsNil(v any) bool {
|
||||
if v == nil {
|
||||
return true
|
||||
}
|
||||
rv := reflect.ValueOf(v)
|
||||
switch rv.Kind() {
|
||||
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
|
||||
return rv.IsNil()
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
package nilutil
|
||||
|
||||
import "testing"
|
||||
|
||||
type sampleInterface interface {
|
||||
Do()
|
||||
}
|
||||
|
||||
type sampleImpl struct{}
|
||||
|
||||
func (*sampleImpl) Do() {}
|
||||
|
||||
func TestIsNilDetectsTypedNilInterface(t *testing.T) {
|
||||
var p *sampleImpl
|
||||
var i sampleInterface = p
|
||||
|
||||
if !IsNil(i) {
|
||||
t.Fatal("IsNil should detect typed nil interface")
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsNilRejectsConcreteValues(t *testing.T) {
|
||||
var p sampleInterface = &sampleImpl{}
|
||||
|
||||
if IsNil(p) {
|
||||
t.Fatal("IsNil should not reject a concrete interface value")
|
||||
}
|
||||
if IsNil("x") {
|
||||
t.Fatal("IsNil should not reject non-nil scalar values")
|
||||
}
|
||||
}
|
||||
+218
@@ -0,0 +1,218 @@
|
||||
package retrieval
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"strings"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// Tokens lowercases Latin words and splits CJK runs into overlapping bigrams
|
||||
// (a lone CJK rune stays a single term) — the standard CJK indexing unit
|
||||
// (Lucene's CJKAnalyzer, SQLite FTS): a bigram only matches a real two-rune
|
||||
// subsequence where per-rune unigrams matched scattered common characters.
|
||||
// Intentionally a local, dependency-free approximation of FTS matching.
|
||||
func Tokens(s string) []string {
|
||||
var out []string
|
||||
var b strings.Builder
|
||||
var prev rune
|
||||
cjkRun := 0
|
||||
flush := func() {
|
||||
if b.Len() == 0 {
|
||||
return
|
||||
}
|
||||
out = append(out, b.String())
|
||||
b.Reset()
|
||||
}
|
||||
endCJK := func() {
|
||||
if cjkRun == 1 {
|
||||
out = append(out, string(prev))
|
||||
}
|
||||
cjkRun = 0
|
||||
}
|
||||
for _, r := range s {
|
||||
switch {
|
||||
case isCJK(r):
|
||||
flush()
|
||||
if cjkRun > 0 {
|
||||
out = append(out, string([]rune{prev, r}))
|
||||
}
|
||||
prev = r
|
||||
cjkRun++
|
||||
case unicode.IsLetter(r) || unicode.IsDigit(r) || r == '_':
|
||||
endCJK()
|
||||
b.WriteRune(unicode.ToLower(r))
|
||||
default:
|
||||
flush()
|
||||
endCJK()
|
||||
}
|
||||
}
|
||||
flush()
|
||||
endCJK()
|
||||
return out
|
||||
}
|
||||
|
||||
func isCJK(r rune) bool {
|
||||
return unicode.In(r, unicode.Han, unicode.Hiragana, unicode.Katakana, unicode.Hangul)
|
||||
}
|
||||
|
||||
// Unique returns terms in first-seen order.
|
||||
func Unique(in []string) []string {
|
||||
seen := map[string]bool{}
|
||||
out := make([]string, 0, len(in))
|
||||
for _, s := range in {
|
||||
if s == "" || seen[s] {
|
||||
continue
|
||||
}
|
||||
seen[s] = true
|
||||
out = append(out, s)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Counts returns a term-frequency map.
|
||||
func Counts(terms []string) map[string]int {
|
||||
counts := map[string]int{}
|
||||
for _, term := range terms {
|
||||
counts[term]++
|
||||
}
|
||||
return counts
|
||||
}
|
||||
|
||||
// BM25Score scores a document against query terms.
|
||||
func BM25Score(counts map[string]int, length int, queryTerms []string, df map[string]int, totalDocs int, avgLen float64) float64 {
|
||||
const (
|
||||
k1 = 1.2
|
||||
b = 0.75
|
||||
)
|
||||
if length <= 0 || totalDocs <= 0 {
|
||||
return 0
|
||||
}
|
||||
if avgLen <= 0 {
|
||||
avgLen = 1
|
||||
}
|
||||
var score float64
|
||||
docLen := float64(length)
|
||||
for _, term := range queryTerms {
|
||||
tf := counts[term]
|
||||
if tf == 0 {
|
||||
continue
|
||||
}
|
||||
termDF := df[term]
|
||||
if termDF == 0 {
|
||||
continue
|
||||
}
|
||||
idf := math.Log(1 + (float64(totalDocs)-float64(termDF)+0.5)/(float64(termDF)+0.5))
|
||||
freq := float64(tf)
|
||||
score += idf * (freq * (k1 + 1)) / (freq + k1*(1-b+b*docLen/avgLen))
|
||||
}
|
||||
return score
|
||||
}
|
||||
|
||||
// DocumentFrequency counts how many documents contain each term.
|
||||
func DocumentFrequency(docs []map[string]int) map[string]int {
|
||||
df := map[string]int{}
|
||||
for _, counts := range docs {
|
||||
for term := range counts {
|
||||
df[term]++
|
||||
}
|
||||
}
|
||||
return df
|
||||
}
|
||||
|
||||
// KeepTopRelativeScore keeps the best item and drops trailing items whose score
|
||||
// falls below ratio * topScore. Callers must pass items already sorted best
|
||||
// first. This mirrors SQLite FTS/BM25 search UIs that over-fetch, then trim
|
||||
// common-word-only noise without imposing an absolute score threshold.
|
||||
func KeepTopRelativeScore[T any](items []T, ratio float64, score func(T) float64) []T {
|
||||
if len(items) == 0 || ratio <= 0 {
|
||||
return items
|
||||
}
|
||||
top := score(items[0])
|
||||
if top <= 0 {
|
||||
return items
|
||||
}
|
||||
cutoff := top * ratio
|
||||
out := items[:0]
|
||||
for i, item := range items {
|
||||
if i == 0 || score(item) >= cutoff {
|
||||
out = append(out, item)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// QueryTerms normalizes a search string and reports an error when nothing
|
||||
// searchable remains.
|
||||
func QueryTerms(query string) ([]string, error) {
|
||||
terms := Unique(Tokens(strings.TrimSpace(query)))
|
||||
if len(terms) == 0 {
|
||||
return nil, fmt.Errorf("query must contain at least one letter or number")
|
||||
}
|
||||
return terms, nil
|
||||
}
|
||||
|
||||
// MakeSnippet returns a whitespace-compacted excerpt centered near the query.
|
||||
func MakeSnippet(text, query string, terms []string, maxRunes int) string {
|
||||
text = CompactWhitespace(text)
|
||||
if maxRunes <= 0 || utf8.RuneCountInString(text) <= maxRunes {
|
||||
return text
|
||||
}
|
||||
lower := strings.ToLower(text)
|
||||
query = strings.ToLower(strings.TrimSpace(query))
|
||||
idx := -1
|
||||
if query != "" {
|
||||
idx = strings.Index(lower, query)
|
||||
}
|
||||
if idx < 0 {
|
||||
for _, term := range terms {
|
||||
runes := []rune(term)
|
||||
if len(runes) == 1 && !isCJK(runes[0]) {
|
||||
continue
|
||||
}
|
||||
if i := strings.Index(lower, term); i >= 0 {
|
||||
idx = i
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if idx < 0 {
|
||||
idx = 0
|
||||
}
|
||||
return snippetAround(text, idx, maxRunes)
|
||||
}
|
||||
|
||||
func snippetAround(text string, byteIdx, maxRunes int) string {
|
||||
if byteIdx < 0 {
|
||||
byteIdx = 0
|
||||
}
|
||||
if byteIdx > len(text) {
|
||||
byteIdx = len(text)
|
||||
}
|
||||
for byteIdx > 0 && byteIdx < len(text) && !utf8.RuneStart(text[byteIdx]) {
|
||||
byteIdx--
|
||||
}
|
||||
runes := []rune(text)
|
||||
pos := utf8.RuneCountInString(text[:byteIdx])
|
||||
start := max(pos-maxRunes/2, 0)
|
||||
end := start + maxRunes
|
||||
if end > len(runes) {
|
||||
end = len(runes)
|
||||
start = max(end-maxRunes, 0)
|
||||
}
|
||||
prefix := ""
|
||||
suffix := ""
|
||||
if start > 0 {
|
||||
prefix = "..."
|
||||
}
|
||||
if end < len(runes) {
|
||||
suffix = "..."
|
||||
}
|
||||
return prefix + string(runes[start:end]) + suffix
|
||||
}
|
||||
|
||||
// CompactWhitespace collapses runs of whitespace into one ASCII space.
|
||||
func CompactWhitespace(s string) string {
|
||||
return strings.Join(strings.Fields(s), " ")
|
||||
}
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
package retrieval
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
func TestTokensHandlesLatinAndCJK(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
in string
|
||||
want []string
|
||||
}{
|
||||
{"BM25 检索 cache-first", []string{"bm25", "检索", "cache", "first"}},
|
||||
{"数据库迁移", []string{"数据", "据库", "库迁", "迁移"}},
|
||||
{"库", []string{"库"}},
|
||||
{"用pnpm装依赖", []string{"用", "pnpm", "装依", "依赖"}},
|
||||
} {
|
||||
got := Tokens(tc.in)
|
||||
if strings.Join(got, ",") != strings.Join(tc.want, ",") {
|
||||
t.Fatalf("Tokens(%q) = %#v, want %#v", tc.in, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBM25ScoreRanksMatchingDocument(t *testing.T) {
|
||||
query := Unique(Tokens("prompt cache"))
|
||||
doc1 := Counts(Tokens("prompt cache cache stability"))
|
||||
doc2 := Counts(Tokens("dashboard colors"))
|
||||
df := DocumentFrequency([]map[string]int{doc1, doc2})
|
||||
score1 := BM25Score(doc1, 4, query, df, 2, 3)
|
||||
score2 := BM25Score(doc2, 2, query, df, 2, 3)
|
||||
if score1 <= score2 {
|
||||
t.Fatalf("matching score %.3f should exceed unrelated score %.3f", score1, score2)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKeepTopRelativeScoreKeepsTopAndDropsWeakTail(t *testing.T) {
|
||||
items := []struct {
|
||||
name string
|
||||
score float64
|
||||
}{
|
||||
{name: "top", score: 10},
|
||||
{name: "near", score: 2},
|
||||
{name: "noise", score: 1.4},
|
||||
{name: "zero", score: 0},
|
||||
}
|
||||
got := KeepTopRelativeScore(items, 0.15, func(item struct {
|
||||
name string
|
||||
score float64
|
||||
}) float64 {
|
||||
return item.score
|
||||
})
|
||||
if len(got) != 2 || got[0].name != "top" || got[1].name != "near" {
|
||||
t.Fatalf("KeepTopRelativeScore() = %#v, want top and near", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMakeSnippetHandlesMultibyteBoundary(t *testing.T) {
|
||||
text := strings.Repeat("前缀", 80) + "稳定结论 synthesis cache " + strings.Repeat("后缀", 80)
|
||||
out := MakeSnippet(text, "synthesis cache", QueryTermsForTest(t, "synthesis cache"), 60)
|
||||
if !strings.Contains(out, "synthesis cache") {
|
||||
t.Fatalf("snippet missing query: %q", out)
|
||||
}
|
||||
if strings.ContainsRune(out, utf8.RuneError) {
|
||||
t.Fatalf("snippet contains replacement rune: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func QueryTermsForTest(t *testing.T, query string) []string {
|
||||
t.Helper()
|
||||
terms, err := QueryTerms(query)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return terms
|
||||
}
|
||||
+176
@@ -0,0 +1,176 @@
|
||||
// Retrieval V2, shadow-only: BM25F field weighting, code-symbol splitting,
|
||||
// and mixed CJK uni+bigrams. Nothing here serves the model — V2 rankings ride
|
||||
// the recall audit next to production's, and MemoryBench decides if V2 ever
|
||||
// takes over. Weights are candidates under measurement, not tuned truths.
|
||||
package retrieval
|
||||
|
||||
import (
|
||||
"math"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
// TokensV2 extends Tokens with code-symbol structure and CJK unigrams:
|
||||
// CamelCase and snake_case words also emit their segments (FooBar → foobar,
|
||||
// foo, bar), and CJK runs emit unigrams alongside bigrams so one-character
|
||||
// queries still match. The V1 form of every token is preserved, so V2 recall
|
||||
// is a superset of V1's vocabulary.
|
||||
func TokensV2(s string) []string {
|
||||
var out []string
|
||||
var word []rune
|
||||
var prev rune
|
||||
cjkRun := 0
|
||||
flushWord := func() {
|
||||
if len(word) == 0 {
|
||||
return
|
||||
}
|
||||
lower := strings.ToLower(string(word))
|
||||
out = append(out, lower)
|
||||
for _, seg := range splitCodeSymbol(word) {
|
||||
if seg != lower {
|
||||
out = append(out, seg)
|
||||
}
|
||||
}
|
||||
word = word[:0]
|
||||
}
|
||||
endCJK := func() { cjkRun = 0 }
|
||||
for _, r := range s {
|
||||
switch {
|
||||
case isCJK(r):
|
||||
flushWord()
|
||||
out = append(out, string(r))
|
||||
if cjkRun > 0 {
|
||||
out = append(out, string([]rune{prev, r}))
|
||||
}
|
||||
prev = r
|
||||
cjkRun++
|
||||
case unicode.IsLetter(r) || unicode.IsDigit(r) || r == '_':
|
||||
endCJK()
|
||||
word = append(word, r)
|
||||
default:
|
||||
flushWord()
|
||||
endCJK()
|
||||
}
|
||||
}
|
||||
flushWord()
|
||||
return out
|
||||
}
|
||||
|
||||
// splitCodeSymbol yields the lowercase segments of a code identifier:
|
||||
// CamelCase humps, snake_case parts, and letter/digit boundaries.
|
||||
func splitCodeSymbol(word []rune) []string {
|
||||
var segments []string
|
||||
var seg []rune
|
||||
flush := func() {
|
||||
if len(seg) > 1 { // single letters are noise, not signal
|
||||
segments = append(segments, strings.ToLower(string(seg)))
|
||||
}
|
||||
seg = seg[:0]
|
||||
}
|
||||
for i, r := range word {
|
||||
boundary := r == '_' ||
|
||||
(i > 0 && unicode.IsUpper(r) && !unicode.IsUpper(word[i-1])) ||
|
||||
(i > 0 && unicode.IsDigit(r) != unicode.IsDigit(word[i-1]))
|
||||
if boundary {
|
||||
flush()
|
||||
}
|
||||
if r != '_' {
|
||||
seg = append(seg, r)
|
||||
}
|
||||
}
|
||||
flush()
|
||||
if len(segments) < 2 {
|
||||
return nil // no structure worth indexing beyond the whole word
|
||||
}
|
||||
return segments
|
||||
}
|
||||
|
||||
// FieldedDoc is one document split into weighted fields for BM25F.
|
||||
type FieldedDoc struct {
|
||||
ID string
|
||||
Fields map[string]string
|
||||
}
|
||||
|
||||
// v2FieldWeights follow the BM25F intuition: identity fields say what a fact
|
||||
// IS, the body says everything it mentions. Candidates under measurement.
|
||||
var v2FieldWeights = map[string]float64{
|
||||
"name": 2.5, "title": 2.5, "keywords": 2.0, "subject": 2.0,
|
||||
"description": 1.5, "body": 1.0,
|
||||
}
|
||||
|
||||
// V2Hit is one shadow-ranked document.
|
||||
type V2Hit struct {
|
||||
ID string
|
||||
Score float64
|
||||
}
|
||||
|
||||
// RankV2 scores docs against a query with BM25F over TokensV2: term
|
||||
// frequencies accumulate per field scaled by field weight, then the standard
|
||||
// saturation applies once per term (the F in BM25F), with length
|
||||
// normalization over the weighted document length.
|
||||
func RankV2(query string, docs []FieldedDoc) []V2Hit {
|
||||
queryTerms := Unique(TokensV2(query))
|
||||
if len(queryTerms) == 0 || len(docs) == 0 {
|
||||
return nil
|
||||
}
|
||||
type indexed struct {
|
||||
id string
|
||||
tf map[string]float64
|
||||
length float64
|
||||
}
|
||||
corpus := make([]indexed, 0, len(docs))
|
||||
df := map[string]int{}
|
||||
var totalLen float64
|
||||
for _, doc := range docs {
|
||||
ix := indexed{id: doc.ID, tf: map[string]float64{}}
|
||||
seen := map[string]bool{}
|
||||
for field, text := range doc.Fields {
|
||||
weight, ok := v2FieldWeights[field]
|
||||
if !ok {
|
||||
weight = 1.0
|
||||
}
|
||||
for _, term := range TokensV2(text) {
|
||||
ix.tf[term] += weight
|
||||
ix.length += weight
|
||||
seen[term] = true
|
||||
}
|
||||
}
|
||||
for term := range seen {
|
||||
df[term]++
|
||||
}
|
||||
totalLen += ix.length
|
||||
corpus = append(corpus, ix)
|
||||
}
|
||||
avgLen := totalLen / float64(len(corpus))
|
||||
if avgLen <= 0 {
|
||||
avgLen = 1
|
||||
}
|
||||
const k1, b = 1.2, 0.75
|
||||
var hits []V2Hit
|
||||
for _, doc := range corpus {
|
||||
var score float64
|
||||
for _, term := range queryTerms {
|
||||
tf := doc.tf[term]
|
||||
if tf == 0 || df[term] == 0 {
|
||||
continue
|
||||
}
|
||||
idf := math.Log(1 + (float64(len(corpus))-float64(df[term])+0.5)/(float64(df[term])+0.5))
|
||||
score += idf * (tf * (k1 + 1)) / (tf + k1*(1-b+b*doc.length/avgLen))
|
||||
}
|
||||
if score > 0 {
|
||||
hits = append(hits, V2Hit{ID: doc.id, Score: score})
|
||||
}
|
||||
}
|
||||
SortHitsDesc(hits)
|
||||
return hits
|
||||
}
|
||||
|
||||
// SortHitsDesc orders shadow hits best-first with a stable ID tiebreak.
|
||||
func SortHitsDesc(hits []V2Hit) {
|
||||
for i := 1; i < len(hits); i++ {
|
||||
for j := i; j > 0 && (hits[j].Score > hits[j-1].Score ||
|
||||
(hits[j].Score == hits[j-1].Score && hits[j].ID < hits[j-1].ID)); j-- {
|
||||
hits[j], hits[j-1] = hits[j-1], hits[j]
|
||||
}
|
||||
}
|
||||
}
|
||||
Loaded 100 of 257 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user