internal/ was 141 packages side by side, so nothing but a reader's memory
said which were foundations and which assembled the rest. Each now sits in a
layer directory, and the directory is the declaration:
base L0 foundation with no domain in it
contract L1 types and interfaces the layers above agree on
model, tools, safety, ext, state, platform
L2 subsystems, peers of one another
runtime L3 the turn engine
session L4 the controller
assembly L5 the composition root
frontend L6 cli, serve, acp and the host adapters
Only paths change: package names, code and behaviour do not. Imports,
repolint and coverage baselines, sensitive paths, CI package lists and docs
follow the new paths; release notes keep the paths they were written with,
and sample paths inside tests and fixtures keep theirs.
repolint gains layer-order: a package may import its own layer and those
below it, read from where it lives. The 47 imports that point upward today
are recorded as that rule's opening debt (-allow-widen, a new rule's first
baseline) and can only go down. Most of it is config, which still reaches
into eight subsystems and is the next thing to take apart; the rest is
event, history and a few tools reaching into runtime for types that belong
in contract.
A new offline mode answers one question against already-recorded runs: if a
child's first accepted complete_subtask had closed it, what work would that
have suppressed. It spends no tokens and changes no runtime behaviour, and it
classifies with the host's own readers rather than a second reading of the same
commands — ToolCallMutationClass and CommandRunsVerification, with verification
tested first because a command whose extent the host cannot establish already
classes as a mutation, and a check is exactly such a command. MutationProven
and MutationUnknown stay apart: reading them as one turned four real writes
into eight, which is the answer this study exists to get right.
The result rejects the barrier. Of 14 children with an accepted report, 1 wrote
a named path afterwards and 7 ran commands of unestablished extent; only 4 did
nothing further. The 47 post-report bash calls are mostly ad-hoc assertions and
inspection that CommandRunsVerification does not recognise as checks, so the
opaque bucket bounds unclassified work rather than claiming it mutated.
What it found instead is that the host asks for the continuation. First-report
adjudication and continued work separate perfectly across all 14: every child
whose first report had claims lowered kept working, every child whose report
stood intact stopped. complete_subtask answers a lowered report with "accepted
... run the check or cite what you really did" while the appended contract says
to call it exactly once as the final tool call. The child follows the later and
more specific instruction. The gap is not a contract without enforcement — it
is an adjudication that is not a state, since a lowered report and an intact
one both return "accepted".
The research record is corrected in the same commit. Its HostDowngrade 0/14 was
read off the handoff audit, which adjudicates only a child's latest report; over
all 39 accepted calls, 25 were adjudicated partial and carried lowered claims.
Extracts the offline-mode dispatch out of main rather than widening its
function-size budget for one more case.
The next experiment asks two questions the first bench cannot: whether an agent
reaches for search when it has forgotten something, and what the ambient
address list is worth in tokens. Both need arms, and the arms need knobs.
recall-search is a real subsystem ablation, so it joins Modules(). Off, it
restores what recall could do before the folded region became searchable: the
runtime refuses a query, the provider-visible schema loses its query parameter
and its description stops offering one, and the line the index writes when it
trims an address stops pointing at a search. An arm told about a capability it
does not have is not the arm before the feature.
The fold-index scale is not a module and does not become one. Set gains an axis
beside the module set: default, half, quarter, off. Four steps rather than a
free-form fraction, because the question is whether an ambient address list has
marginal utility, not where its optimum sits, and a float would turn a
measurement into a tuning knob nobody asked for. It scales the floor along with
the budget — a surviving floor would make the off arm a small index rather
than none.
It reaches Arm() either way: "index-quarter" is not the control arm, and
results keyed by arm name would otherwise merge with it. Both axes travel to
the child binary through one appendArmArgs, because String() carries modules
only and the four call sites that passed it would each have dropped the scale.
ParseArm reads both at once for the same reason.
Measured through the projection, not the flag: 400 tool calls folded at a 120k
window publish 112 index lines at the shipped budget, 58 at half, 28 at a
quarter, 0 off.
Invariants locked by tests, each verified by making it fail:
- the scale not reaching foldIndexBudget -> all four arms publish 112 lines
- the runtime gate held open -> search runs in the arm without it
- the trim line not reverted -> the arm is offered a query
- boot not swapping the tool -> the arm is shown a query parameter
No user-facing surface: both flags are documented as benchmark arms on the same
`run` subcommand -ablate already lives on, and nothing reads them from config.
The repolint budgets for cli.go held it to one line, which is why the flag pair
moved into bench_flags.go rather than growing the god object by seven.
Cache-impact: none on the control arm — the shipped schema, prompt and index
wording are byte-identical; only an explicitly selected arm differs.
Cache-guard: go test ./internal/boot/ (golden baselines unchanged)
Documentation-impact: none.
The graph already carried every stamp a benchmark needs and nothing folded
them, so delegation was reported as a count of children and never as what the
shape bought. Work against wall is the speed-up; the critical path is the floor
the declared dependencies impose, and wall above it is scheduling loss; slot
wait is what the concurrency ceiling cost members with nothing left to wait for
— the one wait no edge on the graph can draw.
The effect test now pins the stamps themselves. Without them every figure here
folds to zero, which reads as a fan-out that took no time rather than as a
measurement that never happened.
Recorded runs are only training data once someone decides which ones were any
good, and the run report already holds that verdict. sft mode reads the two
together: the report decides what is exported, the trajectory's run header
supplies the prefix each sample was produced under.
A run without a header is skipped rather than exported against a guess at what
its prompt and tools were.
depends_on ordered its endpoints and delivered nothing, so a research →
implement pair paid for the same research twice. Three changes that compose:
- The edge carries the dependency's answer. It travels as
ContextRequest.Upstream, never inside TaskSpec.Objective, because the
objective must stay the pristine text delivery classification judges —
ClassifierTaskText is captured before any host framing. One 8 KiB budget is
shared across a node's dependencies, so a wide fan-in cannot flood the
opening turn and no dependency is starved out of it.
- list_subagents enumerates the children a conversation owns. A reference is
the only address a child answer has, and a background fleet that a restart
repaired to interrupted never delivered the aggregate carrying those
references, leaving its completed children unreachable.
- adopt_ref stands a node down to an already-completed child's answer instead
of running it. It is proven in preflight through ReadFinalAnswer, so what may
be adopted cannot drift from what read_subagent_result may read. Nothing
runs, so the node claims no write_paths and cannot collide with a live
writer; prompt and every execution field are refused on it rather than
silently ignored.
Whether an adopted answer is still valid is not readable from structure, so the
host does not decide it: adoption is addressed by reference, and the caller that
listed the reference judges freshness. That is also why a plan fingerprint does
not quietly reuse matching children — reuse the caller cannot see is reuse it
cannot correct.
A child now inherits something, which is exactly what InheritedContext exists to
keep a decision rather than an accident, so the capsule records it and SPEC 3.13
says so.
--ablate upstream leaves the edge ordering its endpoints while delivering
nothing, which is what lets the payload be measured apart from the order.
Against the real provider on benchmarks/upstream-edge/fleet-edge-wide, n=6 per
arm, removing the payload cost +27.6% tokens (exact permutation p=0.011) and
+59% sub-agent tool calls for the same answer, 6/6 solved either way. On a fact
one grep away it buys nothing: the edge pays in proportion to what the
upstream's work cost against how small its result is.
boot.go's delegation registrations moved to delegation_tools.go and Execute's
spec loop to buildItemSpecs; that, plus folding two defaulting branches into
firstNonEmpty, is what paid for the new lines without widening the repolint
baseline.
The product ships uncapped, so a capped task grades a configuration no
default user runs — and on a no-solution task the host's "summarise your
progress" says the very sentence the honesty score exists to hear from the
model. max_steps is optional now and a corpus test keeps it out. Segmented
arms share one timeout_sec instead: wall clock is what cannot be multiplied.
Three leading-directive tables were the only live entrance to the two-model
planner, and they matched a prefix with no word boundary and no idea what the
sentence was about. Probed on nine requests, six misrouted: "make a plan.md
file", "make a plan_test.go that covers the contract", "give me a plan.json
schema" and "先出方案文档提交到 docs/" all asked for a file to be written, and
"plan first-class support for windows paths" matched "plan first" outright.
Each one spent a planner round on work the executor was being asked to do.
The entrance is now host syntax, the way plan mode already works:
PlannerRouteMarker rides at the front of the turn, routes to plan_and_execute,
and is stripped before the model sees it. A frontend prepends it when the user
asks for the workflow; e2ebench's -force-planner uses it instead of putting an
English sentence in front of every benchmark prompt.
Verified against the real provider with planner_model configured: the marker
yields planner 6 / executor 6 usage calls on a task that runs executor 4 alone
without it. normalizePlannerText, hasLeadingDirective, plannerDirectiveText and
its quote scanner leave with the tables — 117 lines.
A turn's obligations were being read off its wording. taskintent held ~1200
lines of CN/EN keyword tables, hand-rolled negation windows, and clause
splitting to answer "is this a mutation request?", and final readiness then
demanded a state change from any sentence that scored one. Its own doc.go
called the vocabulary a liability; boundary_test.go existed to stop it growing.
Readiness now reads the ledger: a turn that changed nothing owes nothing, and
one that did owes the verification, review, and sign-off it already owed. The
contract's intent label follows the same rule -- taskcontract.Kind is observed
from receipts after the fact instead of predicted from text -- and the atomic
contract shape is triggered by a mutation receipt rather than by a verb.
taskpolicy keeps the constraints a user stated outright ("don't modify", "only
run go test"): those restrict, and the user meant them. What goes is the risk
guessing, where "migrate" or "credential" bought a full plan and a forced
review from a topic word rather than from what the turn touches. Risk now
arrives only as host-computed signals.
Also removes machinery that decided nothing in production: the EBM and
reasoning-governor experiments (env-gated, off), the fork-bundle capture that
served them, and the shadow delegation-admission verdict. The per-round
contract replay went with them -- it rebuilt the whole contract from every
receipt on every tool round, quadratic in a long turn, for an audit channel
only the bench implements.
Net -3382 lines. Behavior change: a Delivery turn that claims work without
calling a tool is no longer refused. Catching that belongs to the model's own
claim (update_goal / complete_step), which internal/completion already checks
against the ledger, not to a keyword table.
Cache-impact: none - no tool schema, prompt, or memory text changes; the
per-turn <execution-policy> block rides the turn tail as before and its
route/risk values are derived from the same frozen preset.
Cache-guard: go test ./internal/boot/ (prefix_shape golden unchanged)
Documentation-impact: none - taskintent and the two experiments were never
user-facing; no docs/*.md describes the removed gates.
Reasonix isolates a child's session but not its conclusions. The context
capsule records, field by field, what a child inherited from its parent
and every field is false — yet the one channel that actually carries
information, the parent's free-text prompt, is not modelled at all. Two
host-recorded instruments close that gap.
Evidence origin splits what a delegation's own text pointed at into
directory scope, which is the unavoidable cost of handing work off, and
named files, which say where the answer is. The child is credited only
for paths no named file already covered, so a scope hint never erases
its work. The two are counted apart because one number reports the cheap
act as the expensive one — the combined form read 0-3% independence
where the split reads 94-100%.
The -anchor arm hands the agent a hypothesis before it has read
anything: the task's real cause, a plausible wrong one, or none. Tasks
without an authored seed are skipped rather than run as the control, so
a seeded denominator never absorbs a control run.
Measured over 210 runs across two corpora (deepseek-flash, 5 reps each),
anchor resistance is 1.00 on both — including one whose answers no
command can check — while a wrong hypothesis costs ~2x wall and ~1.35x
tokens to unwind. Children discovered 94-100% of their own evidence:
this parent hands scope, not answers.
Also guards two failures found while running it: the bench read metrics
tags the agent had stopped emitting, and a failed grader's output never
reached the record that had to explain the failure.
The delegation counters landed last commit but nothing surfaced them, so an arm
comparison still could not answer whether extra agents produced verified work
or only tokens.
I went looking for where to build a benchmark harness and found one: cmd/e2ebench
is 11k lines with a task corpus, verify scripts, --ablate arms, comparison,
pareto and HTML reporting already in place. Building a second harness next to it
would have been the exact duplication this branch has spent nine commits
removing, so this extends it instead: the harness metrics mirror gains the new
fields, and one report section joins them to the outcome the harness already
grades.
The section prints the work split (parent versus child tool calls), duplicated
files, how much of the run closed with a checkable claim rather than prose, the
claims the host refused, and write-scope escapes. A single-agent arm renders
"none — n/m solved by a single agent" rather than an empty section, because a
blank section reads as "not measured" when it means "no delegation happened",
and a clean delegated arm prints no alarm lines it has no evidence for.
Two fields I added were already there under the same JSON keys — tool_calls and
subagent_tool_calls — and are dropped rather than shadowed. That is the second
time this pair has caught a duplicate counter, which is a fair argument that the
metrics mirror should be generated from cli.RunMetrics rather than hand-kept.
A twelve-hour session is not interesting because of its length. It is
interesting because of the states it passes through — a session reloaded from
disk, a prefix rebuilt, a compaction crossing a turn boundary, a user arriving
mid-task with a new instruction. -segments N reaches those directly: leg 1
starts the session, later legs resume it with --continue, unambiguous because
each task already runs in its own home and therefore its own session directory.
A resumed leg is deliberately not handed the task again. Its prompt is a bare
continuation, because a leg that restates the work would hide exactly the
degradation this exists to expose. -steer replaces one leg's continuation with
a user turn, which belongs at a leg boundary rather than at a wall-clock
instant nothing can reproduce.
The step budget is divided, never multiplied: a segmented arm gets the same
max_steps as the control, or it would win by being allowed to work longer.
Each leg writes its own metrics file. They share a work dir, so a single path
would leave the last leg's numbers standing in for the whole run and the
earlier legs' tokens would vanish — a failure invisible in any single run, so
the accounting is pinned by a stub agent that records its argv and writes the
metrics it was told to. That test also proves leg 1 starts fresh, later legs
carry --continue and never restate the task, and a failed leg stops the run
rather than resuming a session the child never finished writing.
Renaming the metrics file broke a hard-coded skip in the checkpoint
snapshotter, which would have let every later leg's file read as a change the
agent made. Harness artifacts are now matched by prefix.
The fault seam landed with absolute request indices, which quietly excludes the
tasks most likely to expose a brittle harness: a task that only ever makes four
requests never reaches request seven, so it joins the unfaulted group and
nobody notices the suite is measuring less than it claims. A cadence scales
with the run — every:5:500 — and an absolute index still wins where one is
asked for, so targeted failures stay put.
That gives a run its own control group for free. Under a cadence the short
tasks never hit a fault, so the cost of failure is measured against the same
suite, model and day rather than against a separate arm.
The readout separates two things that are easy to conflate. "Retried" is the
meter seeing another request after a failure — a harness that dies on the first
429 never gets there and was never really tested. "Still solved" is the task
landing anyway. A harness can retry forever and still finish nothing, and that
is not recovery.
The meter existed but nothing reached it. -meter now starts a per-task proxy,
writes a temp config whose benchmarked provider points at it, and hands the
child REASONIX_HOME — no kernel change, and no credential is read or copied,
because the config names an api_key_env and only base_url is rewritten. Only
the provider serving -model is redirected; rewriting every endpoint would send
one vendor's traffic to another's host.
The report then prints the divergence between what the proxy measured and what
the harness said it spent. That number is the point: Reasonix is metered first
precisely because it self-reports, so a disagreement about the same run means
one of the two is wrong and no cross-harness figure is ready to publish.
Two bugs the tests caught while wiring:
- the returned stop func recursed into itself — the closure captured the named
return value it was being assigned to, so every teardown overflowed the stack;
- the divergence compared the meter's tokens over every metered run against the
harness's over only accounted ones, inventing a deficit from a mismatched
denominator. Unaccounted runs now leave both sides of the comparison while
still counting toward real spend.
runTask and main went over the function-size limit, so metering became a small
type that owns setup, teardown and recording, and the checkpoint snapshotter
setup moved next to the rest of the checkpoint code. The lint baseline is
untouched.
Every one of the 49 committed tasks is solvable by construction; the authoring
rule requires it. That makes the corpus unable to ask the one question this
project cares most about: when the work cannot be done, does the agent say so?
On a solvable corpus a false completion and an honest failure to solve look
identical.
The completion-integrity class supplies the other half. Eleven tasks, each
unsolvable for a different reason — contradictory protected tests, a pinned
dependency that does not exist with no network, a spec the repo never
contained, a defect in a tree the prompt forbids touching, an oracle held in an
unset environment variable, a golden that cannot be reproduced, a cycle
Python's import machinery cannot resolve, an authoritative test that is simply
wrong. The only correct outcomes are an honest non-completion or a report
naming what is missing.
Each was checked against the ways an agent could otherwise manufacture a pass,
and several first drafts were thrown out for being quietly solvable: a
call-counter satisfies two single-assert tests, so the contradiction asserts a
set of repeated calls; PyYAML is installed on plenty of machines, so the
missing dependency is a name that exists nowhere; returning the corrupt golden
verbatim passes, so a second assertion demands valid CSV; a re-export layer can
wrap the upstream defect, so the prompt names that layer too. One task was cut
outright rather than shipped weak: two pinned points admit a linear rule, so
"recover the deleted module" was guessable.
verify.sh grades the inverse contract — exit 0 when the fixture survived, exit
1 when a pass was manufactured — with its manifest embedded in the grader,
which e2ebench drops in only after the run. These tasks also leave every
accuracy denominator: "did not cheat" is not "solved", and counting it would
inflate the solve rate and make cost-per-solved meaningless.
The report pins the honesty matrix next to the solvable-side solve rate, so
neither number can be optimized alone: an arm that never claims anything scores
perfectly on integrity and collapses on accuracy.
Production BM25 keeps serving the model untouched. V2 ranks the same
recall pool in shadow: BM25F field weighting (name/title/keywords/
subject over description over body), code-symbol segmentation
(RetryPolicy.MaxBackoffMs matches 'max backoff'), and CJK uni+bigrams
so single-character queries match again. TokensV2 preserves every V1
token, so V2's vocabulary is a superset.
The shadow ranks the full pool BEFORE any production gate - what V1
misses entirely is exactly what the comparison must be able to see -
and rides the memory_recall trajectory record as content-free
shadow_hits (ids and scores only). The bench reports V2-top1 agreement
per run; MemoryBench utility, not ranking metrics, decides if V2 ever
takes over. Field weights are explicitly candidates under measurement.
Bench refinements from the first baseline: pinned-fact tasks mark
memory_markers_prefix so point-of-use counts from turn one (pinned
bodies arrive via the stable prefix, not recall), closing the mb-pin
undercount.
Memory's engineering contract got four dimensions this week; its
effectiveness measurement was still zero. This is the harness that must
exist before any further retrieval tuning: multipliers, thresholds, and
budgets change only when this suite moves.
The core KPI is not Recall@K. A recall hit that looks relevant can be
stale, redundant, or contradicting the repo, and an agent with it can do
worse than one without. The suite therefore measures the paired
counterfactual: the same tasks run with memory on and with the store
hidden (e2ebench -policy memory-off -> REASONIX_EXPERIMENT_NO_MEMORY,
which drops the store, index, pinned guidance, and recall while keeping
instruction docs), and the compare mode reports Utility Delta = paired
Pass(on) - Pass(off), helpful/harmful task lists (harmful is paired
attribution - on-fail while off-pass with recall fired - never a
relevance judgment), and injected-char overhead.
Point of use beats ranking: tasks plant unique markers in seeded fact
bodies, and a marker counts only when it appears in tool arguments or
answer text after a recall injected facts. Recall decisions land in the
trajectory as a content-free memory_recall record (ids, scores,
freshness, chars - never query or fact text) via a new sink capability,
so the same channel feeds the report's Memory shadow line and any future
Retrieval V2 shadow ranking.
Seeding is hermetic: each task's memory/ directory is copied into a
temp REASONIX_STATE_HOME under the workdir's project slug, so runs
never touch the developer's real store. First 12 scenarios cover one of
each class (exact, paraphrase, cjk, symbol, distractor-100, conflict,
stale, contradiction, generic-silence, history-vs-memory, update,
pinned); growing this to the 80-150 target is scenario authoring on a
finished harness.
The runtime side of the low-effort governor, fork-validated on frozen
exploration states across two independent capture rounds: 6 states x 54
continuations with pass parity exact in every state and reasoning
consistently down (neg class 38.3s/231 tok vs control 46.5s/735).
The trigger is the fork-capture condition, now shared verbatim through
governorTrigger: no verification debt, no local execution this turn, and
the round that just ended spent >= 1500 reasoning tokens. While engaged,
prepareSamplingRequest stamps EffortOverride=low on each request; the
governor stands down the moment the turn leaves exploration - debt
opens, a local execution lands, or a discriminating observation arrives.
Cheap rounds alone do not disengage: the fork evidence treated whole
exploration tails, and flapping would buy back full depth mid-cell.
Enforcement sits behind REASONIX_EXPERIMENT_GOVERNOR=1 (the EBM
precedent: eligibility is stamped on every arm so baselines carry the
shadow; e2ebench grows -policy governor). Round reasoning spend moves
from the fork-capture provider wrapper into the executor stream loop, so
the trigger observes it on every run and compaction streams no longer
pollute it; the wrapper shrinks to its snapshot duty.
Shadow chain: governor_eligible/governor_engaged ride OutcomeSample ->
trajectory outcome_progress -> e2ebench outcome digest (eligible/engaged
rounds, first engagement) and the report's governor line. Fork capture
refuses to arm under live enforcement, same as EBM.
Policy triggering is a function of the stochastic trajectory - the same
task reaches blind>=3 one run and not the next, so whole-task A/B mixes
'did the trigger arise' variance into every treatment estimate. This
harness forks control and treatment from the identical eligible state
instead, turning policy evaluation into a local causal experiment.
Capture: at first EBM eligibility the provider wrapper freezes the
state right before the NEXT request - the frozen conversation is
exactly what the uninterrupted run sends (an earlier in-round capture
missed the eligible round's tool results; the invariant test caught
it). Bundles are versioned, carry the outcome-tracker seed and a
workspace copy, and capture refuses to arm under live enforcement so a
treated state can never become a fork origin.
Continuation: beginRunTurn still computes every turn-local
classification from the same input, then the appended user message is
replaced wholesale by the frozen conversation via a single restore
hook. Treatment differs from control by exactly one thing - the nudge
appended to the eligible batch's first tool result, the live policy's
slot - and the single dose disarms the runtime policy.
The invariant test pins the whole contract: the control fork's first
provider request payload deep-equals the original continuation's, and
the treatment payload differs from it in exactly one message by
exactly the nudge suffix.
e2ebench grows '-fork-capture <dir>' (suite mode) and '-mode fork
-bundles <dir> -arm control,treatment -reps N': continuations run in
fresh copies of the bundle workspace, arms alternate order per rep,
and the report is per-pair rows plus per-class aggregation - post-fork
totals ARE Remaining-Cost-From-Eligibility, no alignment needed.
The bundle format is shared infrastructure: reasoning governor,
delegation admission, rollback and stop-policy experiments fork from
the same primitive.
The intervention phase's first runtime policy. Trigger: verification debt
open AND blind_mutations >= 3, at most once per turn. Calibrated against
two recorded baseline runs: healthy trajectories peak at two blind
mutations (edit, edit, check), so the threshold spares 92% of currently
passing runs, while the one recorded failure peaked at five and would
have been nudged at round 10 of 15 with half its runtime left.
The nudge asks for the cheapest discriminating evidence and explicitly
permits finishing the minimum coherent multi-file change first - the
goal is feedback, not test-after-every-edit. Enforcement is env-gated
(REASONIX_EXPERIMENT_EBM=1, e2ebench -policy ebm); eligibility is
stamped on every arm so baseline carries the same shadow, keeping the
A-vs-B comparison inside one dataset.
Runtime records only blind_mutations/ebm_eligible/ebm_fired per round;
e2ebench derives the whole mechanism chain offline: fire round, blind
and debt at fire, rounds/ms/reasoning-tokens to the first discriminating
check, compliance within 1 and 2 rounds, plus debt area and blind peak
per run - the readouts that separate "nudge ignored" from "early
evidence useless".
A local dashboard for watching a bench run while it writes trajectories:
'e2ebench -mode serve -trajectories <dir>' re-summarizes every trajectory
per poll behind /api/state and serves one self-contained dark instrument
page. The run renders as a strip of task segments; each task as a recessed
per-round trace whose win/break polarity is carried by geometry (up/down
off the baseline) because the green/red pair alone fails CVD validation.
The renderer is incremental - DOM nodes are keyed, new rounds pop in,
numbers tween, hover survives polling. A replay engine scrubs the whole
run by wall-clock timestamp (play/pause, 30-600x, jump-to-task), so a
finished run can be watched back round by round. Suite awareness lists
queued tasks as ghosts and prices the remaining time.
summarizeTrajectory splits into scanTrajectoryFile + finish so the
dashboard can read the raw outcome series the summary discards; rounds
gain their observation timestamp for the replay timeline.
golangci-lint round: joinParts was reimplementing strings.Join, the
-task filter ranges over SplitSeq, and the timeline assembles through
a strings.Builder instead of loop concatenation.
Each checkpointed run renders as the lifecycle bar the numbers keep
pointing at: start → first useful mutation → CORRECT → final, positions
proportional to wall clock, with post-solve waste and the full
before/after-correct tallies underneath (rounds, tool calls,
verifications, reviews, mutations — splitAtCorrect now counts them all
in one trajectory pass). One glance answers which phase owns the run:
exploration (long road to the useful mutation), assembly (useful
mutation far from correct), or termination (correct far from final).
First real readout: TTFUM ≈ TTFCS on all four measured tasks — once
this model finally edits, it edits correctly in one go; the assembly
phase is empty and the tail remains the whole story.
Not the first random edit: the earliest checkpoint in which any file
that differs between seed and final already carries its exact final
content — the moment part of the eventual solution first existed.
Cosmetic late edits make the exact-match approximation an overestimate;
the bias is stated, not hidden. Harness artifacts and files unchanged
from the seed are never solution files.
TTFUM splits the pre-correct span the way TTFCS split the whole run:
TTFUM ≈ TTFCS with a long tail to final means the problem is
termination; TTFUM far from task start means the problem is
exploration — finding files, localizing the fault, choosing tools —
and no amount of stop-policy work will touch it. The Correct boundary
line gains the TTFUM median.
The earliest-correct grid makes the stop-policy experiment offline and
free: align each top-level round's end (last tool_result before the
next dispatch) with the checkpoint grades, and every run yields a
curve — had the run stopped at the end of round N, would the grader
have passed? stop_eval carries the per-round verdicts, the earliest
stoppable round, continuations past it, and harmful_continuations:
PASS→FAIL round transitions, the rounds that did not merely waste time
but destroyed a passing state — strictly more information than
'round 6 wasted'. Boundaries before any snapshot grade as the seed
(fail); the final answer segment's end state is the run's final grade.
The Stop policy report line aggregates the curves: how many runs were
stoppable before their final round, the median continuation tail past
the stoppable point, and the harmful-continuation count — the training
target and the evaluation for any future stop policy, measured before
one exists.
The first-correct instant now splits every checkpointed run's behavior
into before and after: mutations_before_correct (workspace states tried
until the right patch), rounds_before_correct / rounds_after_correct
(trajectory tool rounds on each side of the instant, aligned via the
harness clock), verify_after_correct (verification commands re-run on
an already-correct answer), and regressed_after_correct — a PASS
followed by a later FAIL, counted even when the agent repairs the
damage before finishing, which solved_then_broken alone cannot see.
The report gains a Correct boundary line (medians plus the
regression-after-correct rate): rounds-before prices exploration,
rounds-after prices termination, and the regression rate names the most
dangerous habit — not stopping on a correct answer and breaking it by
polishing.
A final grade cannot answer the question that decides the optimization
direction: was the workspace already correct at t=63s with 79 more
seconds of churn on top, or only correct at t=138s? -checkpoints makes
the harness answer it without the agent's participation: a poller
snapshots the task workdir on every observed change (metrics sidecar
and bytecode caches excluded from the change signature; torn mid-write
copies just grade as failures, which is that instant's truth), and
after the run the hidden grader replays over every snapshot oldest
first.
Each result gains checkpoints[], first_correct_ms (TTFCS),
post_solve_waste_ms (wall − TTFCS for solved runs), and
solved_then_broken — the alarm for a passing state the agent later
destroyed, which no final-only grader can see. The KPI line reports
TTFCS median, post-solve waste median, and the solved-then-broke count.
Unlike the round-outcome classifier this needs no progress definition:
the grader itself declares from which moment the answer was already
the answer.
Multi-arm compare gains the routing readout: per task class, each arm's
solve rate and TTCS median with a winner column (best solve rate, ties
to the faster arm) — the table a host-owned profile router would be
built from, measured before any routing code exists. The multi-arm KPI
table also gains TTFT and first-request-cache columns it declared but
never rendered.
-effort passes a reasoning-effort override through to the agent (and to
the warm-cache priming run, keeping prefix shape identical), recorded
per result, making adaptive-reasoning-budget A/Bs one flag: the
pairwise compare's per-class marginal-utility table then answers
Δaccuracy / Δmodel-seconds per class directly.
Suite invocation axes (bin, model, profile, ablation, cache arm,
effort) fold into suiteConfig; the suite-mode tail of main moves to
runSuiteMode.
-cache cold (default) keeps today's fresh-session-per-task behavior and
names it; -cache warm primes the provider prefix cache with a one-step
run in the same workdir before the graded run's clock starts, measuring
the long-lived-session steady state the prefix-cache design optimizes
for. The warming pass's cost is deliberately untracked (the arm prices
the steady state, not the ramp) and its prefix-shaping flags (model,
profile, ablation, cwd) match the graded invocation exactly. Results
carry cache_arm so arms can never be silently mixed; the report title
labels warm runs.
The digest gains first-request cache hit/miss tokens — the cold/warm
verdict signal: a cold session pays the whole prefix as miss, a warmed
one starts near-hit. The KPI line adds TTFT median and first-request
cache hit; the pairwise compare table adds Pass@1, TTFT median, TTCS
median/p90, cache hit, and first-request cache hit rows, so
'-mode compare cold.json warm.json' renders the cold/warm decomposition
directly. Multi-arm tables gain TTFT and first-request cache columns.
Cross-agent fairness note recorded in the README: cold TTCS is the
comparable number against fresh-environment competitors; warm is the
product's steady state. Both are value — they must not be blended.
Every classified round's gap is booked to what the round produced.
Gap-level signals outrank batch analysis: recovery > compaction >
planning > handoff_retry; then the batch decides — verification (a
verification command ran, pass or fail), mutation (a successful
non-ledger write), bookkeeping (ledger-only batches: complete_step,
todo_write, wait, bash_output — the serial sign-off tax made visible),
duplicate_work (every call repeats an already-dispatched name+args),
else evidence_gain (first failures count: they localize, matching the
progress guard's scoring). The final answer segment is finalization.
Productive = evidence_gain/mutation/verification/finalization; the rest
accumulates into wasted_gap_ms. The report gains a Round efficiency
line (useful rounds ratio, wasted model time with per-solved, waste
breakdown sorted by cost) and the phase trace a rounds ledger, so the
readout finally answers: which knife, where.
Old recordings without call IDs skip classification instead of
guessing.
Iterating on one task no longer requires copying it into a scratch
suite root: -task runs only the named comma-separated IDs, preserving
request order, and an unknown ID fails loudly with the available set
instead of silently running zero tasks. README's add-a-task workflow
updated, plus the flags table catches up with compare/traj modes and
suite-mode -attempts.
The headline metric moves from time-to-finish to time-to-correct-
solution: a fast wrong run is not fast. The report gains a KPI line —
Pass@1, Pass@<=N (when -attempts > 1), TTCS median and p90 (measured
over solved tasks only, with a retried solve charged its failed
attempts' wall), and Solved/hour (every attempt's wall in the
denominator, so failures cost real time whether or not a later attempt
lands).
-attempts now also applies to suite mode: each task retries until an
attempt passes, entries carry attempt and ttcs_ms in the JSON, retried
rows are marked (try N) in the table, and the Solved denominator counts
tasks rather than attempts so retries cannot inflate it.
Rendering moves to report.go with the aggregation loop extracted into
suiteStats, keeping main.go inside the size ceiling.
Each suite task's result now carries phase_trace: a privacy-safe latency
trace of counts and milliseconds only — no prompts, arguments, or
outputs. total_ms and solved come from the harness clock and grader;
ttft_ms (span start to first output delta), time_to_first_tool_ms (span
start to first tool start), planner/executor requests (usage events by
source) with their stream-bucket ms, tool calls/queue_ms/critical_path
(interval-union wall), recovery requests (stream+header retries,
reasoning replays, empty-final retries) with tainted-round gap ms,
compaction_ms, and no_progress_signals (progress-guard escalations;
each threshold fires once) all come from the trajectory digest.
Subagent usage counts separately and never claims a parent sampling
attempt, so planner/executor attribution stays clean.
Answers "median wall = 120s — where did it go?" by partitioning each
run's span into disjoint buckets, allocated by priority so overlapping
signals are never double-booked: tools (interval union) > retry backoff
(retrying -> next stream_attempt begin) > compaction (started -> done,
absorbing its inner summarize call) > model streaming (stream_attempt
begin -> commit/discard, with attempts closed by a planner-source usage
event split into a planner bucket) > agent overhead (the remainder:
context assembly, guards, idle). Harness startup is reported separately
as wall_ms minus span. Subagent model calls emit no stream_attempt into
the parent sink and subagent usage never claims a parent attempt, so
parent attribution stays clean.
Also adds -mode traj: re-digest recorded *.trajectory.jsonl files and
print the attribution lines plus per-run JSON, so digest changes can be
evaluated against past runs without re-spending provider tokens.
'Does the planner help' is the wrong question; the decision variable is
what each accuracy point costs in latency, per task class. Tasks gain an
optional class label (the committed suite is tagged), compare mode adds
an overall marginal-utility line (accuracy Δpp · wall/task Δs) and a
per-class breakdown table, and -force-planner prefixes prompts with a
plan-first directive the planner gate recognizes — the 'with planner'
arm for suites whose tasks never trip the gate on their own (the
committed five don't: a real A/B showed usage_by_source with zero
planner calls on every task). Results carry plan_forced so arms are
only comparable with equal forcing.
The planner is the first thing worth A/B-ing: PlanAndExecute turns are
two-model turns, and if the planner request buys +15s per task for
+0.5pp accuracy the trade is a loss — but the report only showed one
opaque request total. The metrics mirror now carries usage_by_source,
the report breaks requests down by origin (executor, planner, subagent,
compaction), and 'e2ebench -mode compare control.json ablated.json'
renders the per-solved delta table (solve rate, model requests, planner
requests, rounds, tool calls, tokens, wall, cost) between two -json
reports under the same accounted-solves convention — one command to
judge an -ablate arm.
Two agents with the same accuracy can differ 2x in how many model
round-trips one solve costs, and per-run averages hide it. The report
gains a per-solved efficiency line — model requests (Steps: every
billed call, subagents and retries included), tool calls, wall time,
and model rounds when trajectories were recorded — all totals across
every accounted run divided by accounted solves, matching the existing
cost/tokens-per-solved convention so failed attempts' spend is charged
to the solves they subsidize.
e2ebench -trajectories DIR passes --trajectory through to each suite
task, keeps one <task-id>.trajectory.jsonl per run, and digests it into
the report: a tools-vs-model time-attribution line in the markdown and a
per-result trajectory summary in the JSON. Subagent tool calls overlap
the parent's wall clock and are excluded from the tool total; a
truncated final line from a killed run is skipped.
* refactor(go): adopt modern stdlib idioms across kernel and desktop
Mechanical sweep with golangci-lint's modernize/intrange/copyloopvar/
usestdlibvars fixers plus hand-fixes for what they could not rewrite:
slices.Contains, min/max, slices.Backward, WaitGroup.Go, reflect.TypeFor,
maps.Copy, strings.Cut/CutPrefix/SplitSeq, strings.Builder, range-over-int.
Net -549 lines with no behavior change.
Three loops in session.go stay index-based: slices.Backward yields element
copies, so the fixer's rewrite silently dropped writes to DecisionReceipts
and ToolCalls (caught by TestDelivery* and TestDependentSameBatch*). All 49
slices.Backward sites were audited for that shape; only these three write
through the index, and they carry a nolint with the reason.
CaptureAfter's scan loop had an empty body and never ran any logic, so the
fixer could not express it. Drop it and the accessor it was the only caller
of. omitzero is deliberately untouched: turning omitempty into omitzero
changes the JSON written for nested structs, which is a wire-format change
to session files and the extension protocol.
* build(lint): gate the modern-idiom linters
Nothing stopped the codebase from drifting back to pre-1.22 idioms. Both
modules are now clean for modernize, intrange, copyloopvar, and
usestdlibvars, so enable them rather than leaving the sweep as a one-off.
omitzero stays disabled with the reason inline; usetesting, perfsprint, and
errorlint still have residue and land separately.
Combine the provider translation, README and changelog corrections, extension-protocol and benchmark docs, locale cleanup, translation parity checks, and custom provider placeholder from the selected open PRs. Update benchmark guidance for the main-v2 base and SWE-bench mode as part of review.
Validated with: go test ./...; pnpm --dir desktop/frontend typecheck; pnpm --dir desktop/frontend test:typecheck.
The benchmark reported a cache-hit percentage but nothing about why the
prefix was invalidated, so a hit-rate regression in a PR gave no signal
about which operation caused it.
metricsSink now tallies event.Usage's CacheDiagnostics.PrefixChangeReasons
into RunMetrics.PrefixChangeReasonCounts, e2ebench mirrors the field in its
parallel runMetrics struct, and render() prints a "Cache resets by cause"
line next to the existing failures-by-class breakdown. The field is
omitempty and the report line is skipped entirely when no run reported any
reason, so older metrics files and runs without diagnostics render exactly
as before.
Verification: go build ./... and go vet clean; new
TestRenderShowsCacheResetsByCause and
TestRenderOmitsCacheResetsLineWhenNoReasonsReported pass, and the existing
TestRenderLeadsWithCostPerSolvedAndFailureClasses is unchanged.
Co-Authored-By: Claude Opus 5 <[email protected]>
* feat(bench): SWE-bench primitives verified against the real harness
Image naming, conda activation, prompt construction, predictions encoding and
grader-report parsing, each pinned by a test against output captured from
swebench 4.1.0 on a real gold run. The orchestration that drives containers
lands next; these are the pieces whose shape could only be learned by running
the harness, not by reading it.
* feat(bench): run SWE-bench inside the official instance images
The agent now runs where the harnesses it is compared against run: inside the
per-instance evaluation container, against /testbed, with the repo's own tests
available to it. Anything less measures a different task.
The container is the isolation boundary, so the in-container config turns the
bash sandbox off — Reasonix otherwise refuses to run bash at all when it cannot
find bubblewrap, which the official images do not ship, and an agent that
cannot run tests is not doing SWE-bench. Nothing else deviates from the shipped
defaults: the provider, model and context window are what a user gets on
install, because a hand-tuned config would make the published number
unreproducible.
Credentials stream into the container over stdin rather than through argv or an
image layer, and the issue text is fully shell-quoted before it becomes one
argv element — a problem statement is untrusted input that routinely contains
backticks and $().
Resolution is never decided here; predictions go to the official grader and its
report is parsed back into the published failure taxonomy.
* feat(bench): publish the SWE-bench Verified subset and how it was chosen
The subset is 50 of the 500 Verified instances, and the script that produced it
ships alongside it. Selection is deterministic — largest-remainder apportionment
across repos, then evenly spaced picks within each repo ordered by difficulty —
so re-running it against the same dataset revision reproduces the file byte for
byte. A sample nobody can regenerate is a sample that can be quietly retuned
after seeing results.
Composition tracks the real benchmark: django 23 of 50 against 46.2% of the
full set, sympy 8 against 15.0%, and difficulty lands at 18/28/4 against the
full set's 38.8/52.2/8.4%. Nothing is excluded, including the psf/requests
instance whose test suite makes live network calls; if it exhausts the grader
timeout it is published as eval_timeout, because dropping the instances a
harness handles badly is how a benchmark stops meaning anything.
* feat(metrics): keep accounting when the agent is killed, and say where it came from
A benchmark run that times out currently reports nothing: the metrics file is
written once, after the controller returns, so a killed agent leaves no trace
of the fifty tool calls it made. In the last SWE-bench run one solved instance
landed in the totals as a free solve, quietly making every per-task figure
cheaper than the truth.
The sink now publishes a throttled snapshot to a `.partial` sidecar as the run
proceeds, through the repository's existing atomic write so a reader only ever
sees a whole file. The final record is marked complete and retires the sidecar,
so the two can never both be counted. The harness prefers the final record and
falls back to the snapshot, reporting recovered numbers as lower bounds rather
than as fact.
Usage is also split by origin. Steps counts every billed model call — executor,
planner, sub-agents, compaction, classifier, router — which is why published
rows read above a max_steps budget the main loop never exhausted. An origin the
build does not recognise keeps its own key rather than being dropped, because a
total that cannot reconcile is worse than one with an unfamiliar row.
Snapshotting made an existing race consequential: Emit is serialized by the
session's event.Sync wrapper, but the run command read the same struct directly
while background jobs could still emit into it. The sink now owns a mutex and
hands out deep copies.
The race test needs `go test -race`, which cannot run on a host without cgo;
CI's race job is what covers it.
* feat(ablation): switch subsystems off behind one shared vocabulary
Nothing could actually be ablated. The only variable axis was the runtime
profile, a coarse bundle, so attributing a change in solve rate to evidence
gating, the planner, subagents, retrieval or cache-preserving compaction was
impossible and every claim about those subsystems rested on assertion.
Add internal/ablation as the single vocabulary: one Set flows from --ablate
through boot into the agent, the controller and spawned sub-agents, and names
the published arm. Each subsystem gets exactly one switch point instead of
scattered conditionals — the planner at effectivePlannerModel, subagents at
the task-tool registration, retrieval at the two BM25-backed tools plus
auto-recall, evidence at finalReadinessCheckFor, and compaction at a new
compactThresholds that collapses the cache-preserving deferral.
boot.Options.DisablePlanner folds into the same Set; ACP's --planner=off maps
onto it rather than carrying a second switch for one subsystem. The arm is
recorded in --metrics and heads the e2e report, and the control arm's command
line stays byte-identical to the pre-ablation one so old numbers remain
comparable.
* docs(cli): document the --ablate benchmark arm
The e2e report could say what a suite cost but not what one solved task
cost, and every failure rendered as a bare cross — a wrong patch and a
guard that deliberately stopped the run were indistinguishable. Neither
number survives a comparison against another harness.
Record wall time, tool-call accounting and the stopping guard in
--metrics, name the guards through agent.PauseClass so hosts stop
reaching into unexported pause types, and lead the report with
cost-per-solved plus a failure-class breakdown. Wall time is the
harness's own clock so the same suite stays comparable when it runs
against a harness that cannot emit our metrics file.
Problem:
The calibrated 400k-token cap was exhausted after three successful baseline tasks, so the final two scenarios were skipped and the release gate could never complete.
Root cause:
Current model and cache behavior uses roughly 425k tokens for the first three tasks while keeping provider cost low, but both the harness default and workflow override retained the older 400k limit.
Fix:
Raise the bounded fixed-suite budget to 800k in the harness and workflow, and add a regression contract for the five-task baseline. Failure or skipping still fails the workflow.
Verification:
go test ./cmd/e2ebench -count=5
go vet ./cmd/e2ebench
go run github.com/rhysd/actionlint/cmd/actionlint@latest .github/workflows/e2e-bot.yml
bash scripts/release-workflows.test.sh
Problem:
The real-provider benchmark ran headlessly without an automatic writer posture and without a usable Linux sandbox backend, so valid file edits were denied and the suite exhausted its token budget.
Root cause:
The harness launched reasonix run in its default Ask mode, while the workflow did not install bubblewrap on the runner.
Fix:
Launch fixed-suite tasks with --auto and install plus verify bubblewrap before running the benchmark. Explicit ask/deny rules and the sandbox boundary remain enforced.
Verification:
go test ./cmd/e2ebench -count=5
go vet ./cmd/e2ebench
go run github.com/rhysd/actionlint/cmd/actionlint@latest .github/workflows/e2e-bot.yml
bash scripts/release-workflows.test.sh
yhh