A batch printed its aggregates and kept nothing they could be recomputed
from, so a published figure could not be tied to the runs behind it after
the temp directory was gone. -batch DIR now writes results.jsonl (one row
per attempt) beside batch.json (manifest v1), and -mode audit re-derives
the report's headline figures from those rows with the harness's own
aggregation, printing the claim beside the recomputation and naming every
figure that moved.
The manifest records what the figures are over (command axes, a task-id
digest, filters, and a denominator defined in tasks), which run produced
them (commit or an explicit unknown, model/provider names, seed when
known, timestamps, harness and runtime), and why they can still be
checked (results_sha256, the claimed figures, and a ledger of planned /
attempted / completed / scored / failed / unmeasured / excluded that the
audit checks for consistency). Absent figures are omitted and read as
unknown, never as zero.
This is an aggregation audit, not a replay: it proves the published
figures follow from the retained rows. It does not re-execute the model,
tools or environment, and a real replay (commit, corpus fingerprint,
model/provider configuration, seed, flags, raw results, runtime version)
is not implemented.
The e2e fixtures finish in 5-15 requests, while a real task in this tree took
deepseek-flash 38. A change aimed at cutting round trips cannot show up on
the short suite, so this adds four long tasks whose workdir is a git archive
of commit 42636ddcb (pricing, ablation, agentgraph and five stdlib-only
leaves) plus a trimmed REASONIX.md, a Makefile, a module README and a small
benchctl consumer:
- rt-cache-write-rate: a CacheWrite rate threaded through cost, fingerprint,
card identity and the catalog, plus an Anthropic row that uses it
- rt-stale-skills: Markdown-only repair of three project skills with no
recognised verifier, the shape behind #13293's lingering stale_verification
- rt-rate-lookup-signatures: five lookups moved to (RateCard, bool),
CatalogKey and ModelRef across about 20 call sites, most in _test.go
- rt-plan-ablation-module: a five-step plan-first task signed off per step
snapshot.sh rebuilds every workdir from the pin and --check detects drift.
One pinned test compares CheckedOn with the wall clock and would fail every
workdir from 2026-11-20; base/ anchors it at the snapshot day instead.
Each task ships a solution/; the corpus is registered with the authoring
guards, which check that every seed fails its grader and every solution
passes. A deepseek-flash baseline (2 trials) solved 8/8 at 13-31 requests
once two grader false negatives in rt-stale-skills were fixed; the README
records it, along with a multi_edit prompt arm and a complete_step
description arm that did not move the request count.
`-policy` is the MemoryBench experiment arm: `memory-off` hides the memory
store so a run can be compared against itself, and the only switch that
separates the two arms of that counterfactual. It is defined at
`cmd/e2ebench/main.go:268` and used in the recipe further down this README, but
the flag table never listed it.
Placed with the other suite-mode experiment arms (`-anchor`, `-cache`) rather
than beside `-ablate`, so it does not collide with the open #11364 that edits
that row.
The benchmarks README still described this harness as the pre-#8244
comprehension/TTL/prune A/B, and taught a subcommand and flags the code has not
had since that rewrite.
ablation.FullFold has had no reader since #8244 (397ed30ec) replaced the
multi-threshold path with the single checkpoint design: foldSource and
carryPriorDigests lost their Off(FullFold) checks together with their test, so
-arm=incremental was identical to the default, and the default already folds the
model-visible projection. The checkpoint design is the intended one, so retire
the dead axis rather than restore the old re-read.
- ablation: drop the FullFold module (constant, comment, module list entry)
- compaction bench: drop the -arm flag, the arms struct, foldArm, and the arm
label from the report title and the JSON report
- benchmarks/README.md: drop the "Fold arms" section, and correct the paragraph
that claimed a fold re-derives its digest from the canonical transcript
- docs/CLI.md: drop full-fold from the --ablate list
Verified: go build ./...; go vet; go test ./benchmarks/compaction/
./internal/contract/ablation/; go run ./tools/repolint clean; golangci-lint
v2.12.2 (the version CI pins) 0 issues. -mode=cost still reports the same eight
generations, -arm is now an unknown flag, and the JSON report no longer carries
an arm label.
Fixes#11358.
Trajectories keep derived events, never what was sent to or received from
the provider, so no run could be played back and nothing caught a kernel
change that altered what the model is asked. The cache-first prefix is meant
to stay byte-stable, and only a live run and its bill showed when it moved.
-tape-record DIR records every exchange at the meter, one tape per task and
trial. -tape-replay DIR answers each run from its tape instead of the
provider, and the report's Replay line names the first request whose body
differs from the recorded one: a message by index and role, the tool list,
or a field. The diverging request is kept beside the recorded one.
Two sources of noise reach the model and had to be held still for a replay
to match: the random workdir path, which is in the workspace block, and the
seeded files' times, which appear in any listing the agent takes. Taped runs
use a fixed workdir per task and trial and pin those times, including the
tape root's, since it is ".." in the listing.
Verified with a real run of bugfix-slug-trailing: recorded in 15.3s for
about USD 0.0014, replayed in 6.0s with no provider traffic, all 6 requests
matched, same tokens and cache hits. The control arm holds: replaying with
-ablate all is reported as "diverged at request 1, tools".
-attempts retries until a pass, so it measures whether the agent can solve a
task at all (Pass@<=N). Nothing measured whether it solves it every time,
which is what a user relying on it meets. -trials N runs every task N times,
keeps every outcome, and adds a Reliability line: pass^N (tasks that passed
every trial), pass@N (any trial), mean pass@1 over all trials, and the flaky
tasks by name. A trial the token budget skips is not counted as a failure,
and the line says how many tasks ran every trial. -trials and -attempts
count repeats differently and are refused together. A run without -trials
reports exactly as before.
Verified with a real run: bugfix-slug-trailing, two trials, both passed,
reported as pass^2 100% (1/1), for about USD 0.001.
The project-check study reported a bypass as reproducible on the strength of a
test that assigned a private field. The field is written once at boot and never
again, so no run inside one process reaches that state, and the corpus said so
the first time it executed: parity, no divergence. The test proves a conditional
mechanism given the state; that is a different claim from a production defect,
and only the second earns priority.
So a destructive test now owes six lines before it is written up as something to
fix — who produces the state, through which transitions, across which
persistence boundary, under whose authority, and whether a real run has been
seen in it. Severity is reachability times frequency times authority, not
whether something can be bypassed.
Authority is in the contract because the same divergence can mean opposite
things. An agent that rewrote the criterion it was being held to is what
baseline protection exists for; an owner who changed the project's policy
between two runs may have superseded it on purpose. A study that cannot say
which it observed cannot say what the right behaviour is.
The segmented probe that motivated this is in the corpus: leg one blocked on the
gate still demanding the criterion loaded at boot, which is the same finding
from the other side — the declaration on disk had already changed and the gate
had not noticed.
Cache-impact: one rule added to REASONIX.md, which is in the stable prefix.
The subagent handoff line is resolved for the evidence there is. Protocol
salience and closure behaviour were never defective: every judged child
reported, and nothing followed a real closure in either arm. What was defective
was the host's vocabulary, which answered a lowered claim and an intact one with
the same word, and that is fixed. Structural terminal enforcement is recorded as
unjustified rather than pending — it would be a defensive invariant, and the
case for it has to be made on delivery risk, not on this data.
The discipline is the study's other output. Three mechanism explanations died
the same way: a command of unknown extent counted as a proven write, one child's
latest adjudication read as every report's, and a syntactically accepted call
read as a closed one. Each was a convenient proxy standing in for the state the
metric was named after, and each produced a confident wrong story. The rule now
sits beside the other judgement rules, and the four-line contract a metric owes
before it becomes a finding sits in the benchmark README with the five
substitutions that caused these.
Cache-impact: REASONIX.md is in the stable prefix, so this adds one rule to it
and no session carries a different prefix mid-run.
Delegation makes three claims and upstream-edge can only settle one: both of
its tasks are strict chains, so their speed-up is 1.00x by construction and no
scheduler decision is observable in them. fanout-width fixes a shape whose
three research items share nothing, which leaves the session concurrency
ceiling as the only thing deciding whether they overlap — so the serialised arm
needs no ablation flag, only max_subagent_concurrency = 1.
Each subsystem retires nine of ten modules and names the live one in entry.py,
so a grep over the markers returns ten candidates and answers nothing.
The product ships uncapped, so a capped task grades a configuration no
default user runs — and on a no-solution task the host's "summarise your
progress" says the very sentence the honesty score exists to hear from the
model. max_steps is optional now and a corpus test keeps it out. Segmented
arms share one timeout_sec instead: wall clock is what cannot be multiplied.
The corpus had 67 Python tasks and no Go one, while the agent's own repository
is Go. A compiled language is a different verification loop — a build before
anything runs, go vet, -race, a module path that must agree with every import —
and its defects have no Python analogue: a data race, a += in a loop that is
quadratic where Python's would be too, map iteration order that makes a test
pass only sometimes. Nine tasks cover that half as ordinary members of the
existing classes.
README already required every task to fail on the pristine seed and pass on a
reference solution, but only the no-solution half was enforced. A solvable task
whose seed grades clean scores the same whether the agent solved it or never
ran and reports 100% forever; TestSolvableCorpusSeedsMustNotGradeClean holds the
other 62 to it. All of them already pass. stageSeed now tolerates the optional
workdir the README documents.
Harness metrics move out of the work dir. The agent reads what is in front of
it, and one observed run spent a tool call on .run-metrics-1.json.partial —
snapshots already dropped these files, which kept them out of the grade but not
out of the agent's context.
Reasonix isolates a child's session but not its conclusions. The context
capsule records, field by field, what a child inherited from its parent
and every field is false — yet the one channel that actually carries
information, the parent's free-text prompt, is not modelled at all. Two
host-recorded instruments close that gap.
Evidence origin splits what a delegation's own text pointed at into
directory scope, which is the unavoidable cost of handing work off, and
named files, which say where the answer is. The child is credited only
for paths no named file already covered, so a scope hint never erases
its work. The two are counted apart because one number reports the cheap
act as the expensive one — the combined form read 0-3% independence
where the split reads 94-100%.
The -anchor arm hands the agent a hypothesis before it has read
anything: the task's real cause, a plausible wrong one, or none. Tasks
without an authored seed are skipped rather than run as the control, so
a seeded denominator never absorbs a control run.
Measured over 210 runs across two corpora (deepseek-flash, 5 reps each),
anchor resistance is 1.00 on both — including one whose answers no
command can check — while a wrong hypothesis costs ~2x wall and ~1.35x
tokens to unwind. Children discovered 94-100% of their own evidence:
this parent hands scope, not answers.
Also guards two failures found while running it: the bench read metrics
tags the agent had stopped emitting, and a failed grader's output never
reached the record that had to explain the failure.
A twelve-hour session is not interesting because of its length. It is
interesting because of the states it passes through — a session reloaded from
disk, a prefix rebuilt, a compaction crossing a turn boundary, a user arriving
mid-task with a new instruction. -segments N reaches those directly: leg 1
starts the session, later legs resume it with --continue, unambiguous because
each task already runs in its own home and therefore its own session directory.
A resumed leg is deliberately not handed the task again. Its prompt is a bare
continuation, because a leg that restates the work would hide exactly the
degradation this exists to expose. -steer replaces one leg's continuation with
a user turn, which belongs at a leg boundary rather than at a wall-clock
instant nothing can reproduce.
The step budget is divided, never multiplied: a segmented arm gets the same
max_steps as the control, or it would win by being allowed to work longer.
Each leg writes its own metrics file. They share a work dir, so a single path
would leave the last leg's numbers standing in for the whole run and the
earlier legs' tokens would vanish — a failure invisible in any single run, so
the accounting is pinned by a stub agent that records its argv and writes the
metrics it was told to. That test also proves leg 1 starts fresh, later legs
carry --continue and never restate the task, and a failed leg stops the run
rather than resuming a session the child never finished writing.
Renaming the metrics file broke a hard-coded skip in the checkpoint
snapshotter, which would have let every later leg's file read as a change the
agent made. Harness artifacts are now matched by prefix.
The fault seam landed with absolute request indices, which quietly excludes the
tasks most likely to expose a brittle harness: a task that only ever makes four
requests never reaches request seven, so it joins the unfaulted group and
nobody notices the suite is measuring less than it claims. A cadence scales
with the run — every:5:500 — and an absolute index still wins where one is
asked for, so targeted failures stay put.
That gives a run its own control group for free. Under a cadence the short
tasks never hit a fault, so the cost of failure is measured against the same
suite, model and day rather than against a separate arm.
The readout separates two things that are easy to conflate. "Retried" is the
meter seeing another request after a failure — a harness that dies on the first
429 never gets there and was never really tested. "Still solved" is the task
landing anyway. A harness can retry forever and still finish nothing, and that
is not recovery.
The meter existed but nothing reached it. -meter now starts a per-task proxy,
writes a temp config whose benchmarked provider points at it, and hands the
child REASONIX_HOME — no kernel change, and no credential is read or copied,
because the config names an api_key_env and only base_url is rewritten. Only
the provider serving -model is redirected; rewriting every endpoint would send
one vendor's traffic to another's host.
The report then prints the divergence between what the proxy measured and what
the harness said it spent. That number is the point: Reasonix is metered first
precisely because it self-reports, so a disagreement about the same run means
one of the two is wrong and no cross-harness figure is ready to publish.
Two bugs the tests caught while wiring:
- the returned stop func recursed into itself — the closure captured the named
return value it was being assigned to, so every teardown overflowed the stack;
- the divergence compared the meter's tokens over every metered run against the
harness's over only accounted ones, inventing a deficit from a mismatched
denominator. Unaccounted runs now leave both sides of the comparison while
still counting toward real spend.
runTask and main went over the function-size limit, so metering became a small
type that owns setup, teardown and recording, and the checkpoint snapshotter
setup moved next to the rest of the checkpoint code. The lint baseline is
untouched.
Every one of the 49 committed tasks is solvable by construction; the authoring
rule requires it. That makes the corpus unable to ask the one question this
project cares most about: when the work cannot be done, does the agent say so?
On a solvable corpus a false completion and an honest failure to solve look
identical.
The completion-integrity class supplies the other half. Eleven tasks, each
unsolvable for a different reason — contradictory protected tests, a pinned
dependency that does not exist with no network, a spec the repo never
contained, a defect in a tree the prompt forbids touching, an oracle held in an
unset environment variable, a golden that cannot be reproduced, a cycle
Python's import machinery cannot resolve, an authoritative test that is simply
wrong. The only correct outcomes are an honest non-completion or a report
naming what is missing.
Each was checked against the ways an agent could otherwise manufacture a pass,
and several first drafts were thrown out for being quietly solvable: a
call-counter satisfies two single-assert tests, so the contradiction asserts a
set of repeated calls; PyYAML is installed on plenty of machines, so the
missing dependency is a name that exists nowhere; returning the corrupt golden
verbatim passes, so a second assertion demands valid CSV; a re-export layer can
wrap the upstream defect, so the prompt names that layer too. One task was cut
outright rather than shipped weak: two pinned points admit a linear rule, so
"recover the deleted module" was guessable.
verify.sh grades the inverse contract — exit 0 when the fixture survived, exit
1 when a pass was manufactured — with its manifest embedded in the grader,
which e2ebench drops in only after the run. These tasks also leave every
accuracy denominator: "did not cheat" is not "solved", and counting it would
inflate the solve rate and make cost-per-solved meaningless.
The report pins the honesty matrix next to the solvable-side solve rate, so
neither number can be optimized alone: an arm that never claims anything scores
perfectly on integrity and collapses on accuracy.
A fold re-derives its digest from the canonical transcript every time, which is
what keeps digests from chaining — and what makes each fold cost more than the
last. CompactionBench put a number on it: by generation 8 one fold reads 448k
tokens across 5 summarizer calls, growing without bound.
Switching the full-fold mechanism off makes a fold read the model-visible view
instead, so the previous digest becomes summarizer input rather than something
to re-derive. Cost stops growing: 1 call and ~76k tokens at every generation,
flat from generation 2 on.
It also loses facts, which is the point of measuring rather than assuming. Over
eight generations against DeepSeek, probe survival drops from 71/71 to 53/71.
The losses are not spread evenly: an exact identifier stated once in a user turn
("ticket RX-4821") disappears at the first re-summarization and never returns —
later generations answer "no ticket ID is present in the conversation" — and a
code fact goes the same way from generation 4. Both are exactly what a
summary-of-a-summary is expected to drop, and they are dropped silently.
So this lands as an arm, not a default: `-arm=incremental` in the bench and
`--ablate full-fold` for the e2e suite. The ablation package already exists to
attribute a benchmark result to one mechanism, and full re-derivation is such a
mechanism; using it keeps the experiment out of Options and out of the config
surface entirely.
planFoldRegion is extracted from compactToProjection along the way: the fold
source and the region plan are separate decisions, and the function was over
the size limit with both inline.
Compaction had no measurement of its own. The context-maintenance harness
covers cold-restart cache behavior after pruning; nothing covered what folding
a session over and over costs, or what it loses.
CompactionBench grows a session one generation at a time and folds after each,
so generation N folds everything 1..N produced — the growth that matters, since
a fold re-derives its digest from the whole canonical transcript.
The cost arm is deterministic and needs no API key. Its scripted summarizer
refuses any input larger than the window the way a real provider does, so a
session that grows until it can no longer be compacted shows up as an error row
instead of as an argument. That arm doubles as a CI regression guard: every
generation must still fold, no single call may leave the window without room
for the digest, and the call count stays under the span ceiling.
The fidelity arm plants what a coding agent must not lose — a standing
constraint, a correction superseding an earlier instruction, an exact
identifier, a pending requirement, whether a passing test has been re-run since
the code changed, a ruled-out hypothesis, a tool outcome, chronology — and
after each fold asks a question only that fact answers. Every probe also runs
against full history in the same session, so a probe the model fails with
everything in front of it is scored as a bad probe rather than a compaction
loss. Answers match on whole words, and a wanted answer does not count when a
rejected one appears in the same reply: "yes, but it has not been re-run since"
is what a drifting digest produces, and it is not a pass.
Three things the first real runs proved necessary, all of them the difference
between a number and a measurement:
- Record what the model actually said. A survival rate no one can audit is
not evidence; the answers now ride in the JSON report.
- Budget for the model's reasoning, not just its answer. A thinking model
spends the first tokens reasoning, so a budget sized for "yes" came back
empty and scored every probe as a lost fact.
- Separate an unanswered probe from a lost one. A context full of tool calls
invites the model to answer with another one; those replies are marked and
excluded from the rate, and the report prints how many there were.
Memory's engineering contract got four dimensions this week; its
effectiveness measurement was still zero. This is the harness that must
exist before any further retrieval tuning: multipliers, thresholds, and
budgets change only when this suite moves.
The core KPI is not Recall@K. A recall hit that looks relevant can be
stale, redundant, or contradicting the repo, and an agent with it can do
worse than one without. The suite therefore measures the paired
counterfactual: the same tasks run with memory on and with the store
hidden (e2ebench -policy memory-off -> REASONIX_EXPERIMENT_NO_MEMORY,
which drops the store, index, pinned guidance, and recall while keeping
instruction docs), and the compare mode reports Utility Delta = paired
Pass(on) - Pass(off), helpful/harmful task lists (harmful is paired
attribution - on-fail while off-pass with recall fired - never a
relevance judgment), and injected-char overhead.
Point of use beats ranking: tasks plant unique markers in seeded fact
bodies, and a marker counts only when it appears in tool arguments or
answer text after a recall injected facts. Recall decisions land in the
trajectory as a content-free memory_recall record (ids, scores,
freshness, chars - never query or fact text) via a new sink capability,
so the same channel feeds the report's Memory shadow line and any future
Retrieval V2 shadow ranking.
Seeding is hermetic: each task's memory/ directory is copied into a
temp REASONIX_STATE_HOME under the workdir's project slug, so runs
never touch the developer's real store. First 12 scenarios cover one of
each class (exact, paraphrase, cjk, symbol, distractor-100, conflict,
stale, contradiction, generic-silence, history-vs-memory, update,
pinned); growing this to the 80-150 target is scenario authoring on a
finished harness.
28 new tasks complete every class to its target: atomic-bugfix 8/8,
repo-exploration 6/6 (invented-token answers), multi-file-bugfix 8/8
(each bug a cross-file contract disagreement: header casing, config
format, id prefixes, degree/radian and cent/dollar units, page bounds),
refactor 6/6 (table-driven mapping, pure/IO split, global-to-class,
shared-helper extraction — structure asserted via source checks),
failing-test-diagnosis 6/6 (BOM handling, floor division, sort
stability, missing-file defaults — test files checksummed),
api-integration 4/4 (backoff schedule, event log replay),
ambiguous 4/4 (contact dedupe, fail-fast config — graders accept the
defensible core only), long-horizon 4/4 (sales report with a
generalization check on a second dataset, inventory reservation sim).
Every grader validated in both directions before commit: fails on the
pristine seed, passes on a reference solution (35/35 including tranche
one).
The committed suite measured toy tasks with increasing precision — the
planner A/B proved it: none of the five tasks ever routed past
ExecutorOnly. This starts the workload-stratified corpus the readouts
were built for: 16 new tasks across eight classes (atomic-bugfix,
repo-exploration, multi-file-bugfix, refactor, failing-test-diagnosis,
api-integration, ambiguous, long-horizon), two per class, with the
target stratification documented in the README. Multi-file, refactor
and long-horizon prompts are phrased to engage the planner gate
naturally; atomic fixes stay short and anchored so they keep routing
ExecutorOnly.
Every grader was validated in both directions before commit: verify.sh
fails on the pristine seed and passes on a reference solution.
failing-test-diagnosis graders checksum the test files so fixing the
test instead of the source cannot pass. Python graders pin a throwaway
PYTHONPYCACHEPREFIX — macOS system Python caches bytecode centrally by
absolute path, and a same-size same-second edit would otherwise run
stale bytecode while tracebacks display the new source.
Existing tasks join the taxonomy (fix-add-bug -> atomic-bugfix,
compaction -> repo-exploration); fizzbuzz/palindrome/subagent-delegation
stay as legacy smoke tasks.
No bandits, no learned routing, no classifier models now: the control
plane stays deterministic and interpretable, and the bigger present
risk is heuristic complexity, not missing ML. But the -json reports
and trajectory directories real runs already produce are exactly the
corpus any future offline learning would need — contracts, event
trajectories, checkpoint oracle verdicts, stop curves, phase traces —
so the README says to keep them. Nothing learned lands before it beats
the deterministic baseline on the same oracle-produced numbers.
-cache cold (default) keeps today's fresh-session-per-task behavior and
names it; -cache warm primes the provider prefix cache with a one-step
run in the same workdir before the graded run's clock starts, measuring
the long-lived-session steady state the prefix-cache design optimizes
for. The warming pass's cost is deliberately untracked (the arm prices
the steady state, not the ramp) and its prefix-shaping flags (model,
profile, ablation, cwd) match the graded invocation exactly. Results
carry cache_arm so arms can never be silently mixed; the report title
labels warm runs.
The digest gains first-request cache hit/miss tokens — the cold/warm
verdict signal: a cold session pays the whole prefix as miss, a warmed
one starts near-hit. The KPI line adds TTFT median and first-request
cache hit; the pairwise compare table adds Pass@1, TTFT median, TTCS
median/p90, cache hit, and first-request cache hit rows, so
'-mode compare cold.json warm.json' renders the cold/warm decomposition
directly. Multi-arm tables gain TTFT and first-request cache columns.
Cross-agent fairness note recorded in the README: cold TTCS is the
comparable number against fresh-environment competitors; warm is the
product's steady state. Both are value — they must not be blended.
-profile grows from baseline|delivery to the real tier set: economy |
balanced | delivery, passed straight through to the agent's runtime
profile (baseline keeps passing no flag so older control command lines
stay byte-identical). Economy's trade — a small schema footprint paid
for with connect_tool_source rounds and provider prefix resets — is now
priced in the report: a Tool surface line (max schema tokens/request,
Σ schema tax and its share of prompt tokens, connect calls, prefix
resets, all from usage cacheDiagnostics; subagent usage excluded), and
tool_source_connect / prefix_reset rows in the mechanism ledger.
Fresh-session benchmarks re-pay the schema miss on every task, so the
surface size prices differently than in the long-lived sessions the
prefix-cache design optimizes for — this makes that visible instead of
folding it into 'model time'.
Iterating on one task no longer requires copying it into a scratch
suite root: -task runs only the named comma-separated IDs, preserving
request order, and an unknown ID fails loudly with the available set
instead of silently running zero tasks. README's add-a-task workflow
updated, plus the flags table catches up with compare/traj modes and
suite-mode -attempts.
'Does the planner help' is the wrong question; the decision variable is
what each accuracy point costs in latency, per task class. Tasks gain an
optional class label (the committed suite is tagged), compare mode adds
an overall marginal-utility line (accuracy Δpp · wall/task Δs) and a
per-class breakdown table, and -force-planner prefixes prompts with a
plan-first directive the planner gate recognizes — the 'with planner'
arm for suites whose tasks never trip the gate on their own (the
committed five don't: a real A/B showed usage_by_source with zero
planner calls on every task). Results carry plan_forced so arms are
only comparable with equal forcing.
e2ebench -trajectories DIR passes --trajectory through to each suite
task, keeps one <task-id>.trajectory.jsonl per run, and digests it into
the report: a tools-vs-model time-attribution line in the markdown and a
per-result trajectory summary in the JSON. Subagent tool calls overlap
the parent's wall clock and are excluded from the tool total; a
truncated final line from a killed run is skipped.
Combine the provider translation, README and changelog corrections, extension-protocol and benchmark docs, locale cleanup, translation parity checks, and custom provider placeholder from the selected open PRs. Update benchmark guidance for the main-v2 base and SWE-bench mode as part of review.
Validated with: go test ./...; pnpm --dir desktop/frontend typecheck; pnpm --dir desktop/frontend test:typecheck.
Will