zumac
7e92388102
The e2e fixtures finish in 5-15 requests, while a real task in this tree took
deepseek-flash 38. A change aimed at cutting round trips cannot show up on
the short suite, so this adds four long tasks whose workdir is a git archive
of commit 42636ddcb (pricing, ablation, agentgraph and five stdlib-only
leaves) plus a trimmed REASONIX.md, a Makefile, a module README and a small
benchctl consumer:
- rt-cache-write-rate: a CacheWrite rate threaded through cost, fingerprint,
card identity and the catalog, plus an Anthropic row that uses it
- rt-stale-skills: Markdown-only repair of three project skills with no
recognised verifier, the shape behind #13293's lingering stale_verification
- rt-rate-lookup-signatures: five lookups moved to (RateCard, bool),
CatalogKey and ModelRef across about 20 call sites, most in _test.go
- rt-plan-ablation-module: a five-step plan-first task signed off per step
snapshot.sh rebuilds every workdir from the pin and --check detects drift.
One pinned test compares CheckedOn with the wall clock and would fail every
workdir from 2026-11-20; base/ anchors it at the snapshot day instead.
Each task ships a solution/; the corpus is registered with the authoring
guards, which check that every seed fails its grader and every solution
passes. A deepseek-flash baseline (2 trials) solved 8/8 at 13-31 requests
once two grader false negatives in rt-stale-skills were fixed; the README
records it, along with a multi_edit prompt arm and a complete_step
description arm that did not move the request count.
Reasonix kernel snapshot
A trimmed slice of the Reasonix Go kernel: the contracts a benchmark report is
priced and labelled with, a few leaf utilities, and benchctl, a small command
that prints what those contracts answer.
make check # gofmt, vet, test
go run ./cmd/benchctl rate -model deepseek-flash -currency USD
go run ./cmd/benchctl cost -provider mimo -model mimo-v2.5 -miss 1000000
go run ./cmd/benchctl tariff -currency CNY -at 2026-10-12T02:00:00Z
go run ./cmd/benchctl arm -ablate planner,evidence
Packages
| Package | Owns |
|---|---|
internal/contract/pricing |
the official price book, cost quotes, the spend ledger |
internal/contract/ablation |
which subsystems a benchmark arm switches off, and the arm's name |
internal/contract/agentgraph |
the vocabulary of one run's execution graph |
internal/base/retrieval |
BM25 ranking over CJK-aware tokens |
internal/base/rulelist |
splitting a one-line permission rule list |
internal/base/tokencount |
a cheap token estimate |
internal/base/neterr |
classifying transport errors |
internal/base/nilutil |
typed-nil detection |
cmd/benchctl |
prints rates, costs, peak schedules and arm names |
Pricing
A RateCard is a price per million tokens in one currency:
| Field | Charges |
|---|---|
CacheHit |
prompt tokens served from the provider's cache |
Input |
prompt tokens that missed the cache, including tokens written to it |
Output |
completion tokens |
OriginalCostAmount turns a card and a UsageTokens breakdown into a
fixed-point Amount; PricingFingerprint keys a card in the ledger, so a
stored ledger entry stays attached to the card it was priced with.
CurrentRate and SupersededRates answer from officialRates, which only
ever grows: a price change appends a generation.
Ablation
ablation.Parse reads a comma-separated list of modules to switch off, and
Set.Arm names the result (full for the control). The switchable modules,
in the order arm names use:
| Module | Off means |
|---|---|
evidence |
no evidence gate on completion claims |
planner |
no two-model planning turn |
subagent |
no delegation to workers |
retrieval |
no retrieval over the folded transcript |
compaction |
no context folding |
upstream |
a dependency edge orders its endpoints without delivering the answer |
recall-search |
recall can read a given address but cannot search for one |
The fold-index scale is a separate axis (-fold-index), not a module.
Skills
Project skills live under .reasonix/skills/<name>/SKILL.md.
| Skill | Use it to |
|---|---|
price-change |
record a vendor's price change in the price book |