Rewrite internal/runtime/delegation around one Runner and one tool.
spawn_agents runs 1 to 8 independent tasks in parallel, each in its own
session with three base tools (read_file, write_file, bash); plan mode and
read-only skills keep a read-only child. A child's full answer is archived;
the caller gets the opening, a reference, and the files the child read and
wrote, and pages the rest through read_subagent_result.
Removed with it: task, read_only_task, parallel_tasks, fleet,
list_subagents, background sub-tasks, steering and its limits and settings
UI, continue_from, write_paths claims, isolated worktrees, best-of-n, the
run graph and the runtime-resume benchmark. Stale writes by children stay
refused through the per-agent file views.
The use_capability description no longer names task:fleet, so the stable
prefix and its golden files change once.
A batch printed its aggregates and kept nothing they could be recomputed
from, so a published figure could not be tied to the runs behind it after
the temp directory was gone. -batch DIR now writes results.jsonl (one row
per attempt) beside batch.json (manifest v1), and -mode audit re-derives
the report's headline figures from those rows with the harness's own
aggregation, printing the claim beside the recomputation and naming every
figure that moved.
The manifest records what the figures are over (command axes, a task-id
digest, filters, and a denominator defined in tasks), which run produced
them (commit or an explicit unknown, model/provider names, seed when
known, timestamps, harness and runtime), and why they can still be
checked (results_sha256, the claimed figures, and a ledger of planned /
attempted / completed / scored / failed / unmeasured / excluded that the
audit checks for consistency). Absent figures are omitted and read as
unknown, never as zero.
This is an aggregation audit, not a replay: it proves the published
figures follow from the retained rows. It does not re-execute the model,
tools or environment, and a real replay (commit, corpus fingerprint,
model/provider configuration, seed, flags, raw results, runtime version)
is not implemented.
The Windows runner's Git Bash has no shasum, and the grader's
`shasum -c ... || exit 1` turned "command not found" into the same exit 1 as
a changed file, so the reference solution failed there with nothing printed.
The SHA-256 manifest is now checked with hashlib inside the grader's Python,
over raw bytes, and a mismatch names the files that differ.
gradeSeed now carries the grader's output in its error: the CI log said only
"exit status 1", which cannot tell a missing tool from a failed check.
Verified locally: on a PATH without shasum the reference solution grades
clean; a CRLF Makefile and an edited Go file each fail and are named; the
pristine seed still fails. snapshot.sh --check, the corpus tests and
make check pass.
The e2e fixtures finish in 5-15 requests, while a real task in this tree took
deepseek-flash 38. A change aimed at cutting round trips cannot show up on
the short suite, so this adds four long tasks whose workdir is a git archive
of commit 42636ddcb (pricing, ablation, agentgraph and five stdlib-only
leaves) plus a trimmed REASONIX.md, a Makefile, a module README and a small
benchctl consumer:
- rt-cache-write-rate: a CacheWrite rate threaded through cost, fingerprint,
card identity and the catalog, plus an Anthropic row that uses it
- rt-stale-skills: Markdown-only repair of three project skills with no
recognised verifier, the shape behind #13293's lingering stale_verification
- rt-rate-lookup-signatures: five lookups moved to (RateCard, bool),
CatalogKey and ModelRef across about 20 call sites, most in _test.go
- rt-plan-ablation-module: a five-step plan-first task signed off per step
snapshot.sh rebuilds every workdir from the pin and --check detects drift.
One pinned test compares CheckedOn with the wall clock and would fail every
workdir from 2026-11-20; base/ anchors it at the snapshot day instead.
Each task ships a solution/; the corpus is registered with the authoring
guards, which check that every seed fails its grader and every solution
passes. A deepseek-flash baseline (2 trials) solved 8/8 at 13-31 requests
once two grader false negatives in rt-stale-skills were fixed; the README
records it, along with a multi_edit prompt arm and a complete_step
description arm that did not move the request count.
reportBoundaries folded every row into byKey, so a run that never produced a
transcript re-entered the cue-boundary table as a zero measurement after the
funnel, dose and reformulation exits had excluded it. Skip failed rows, count
them, and say scored=0 with the failed count when nothing is left to compare.
A run that failed before its first model turn was appended as a zero-valued
contextMetrics, so results.jsonl gained an empty row, summarize counted it in
Scored and its stage tables, and batch.json reported it as completed with an
empty task id. Stamp such a run with its task, arm and NoEvidence stage, mark it
failed, and keep it out of every statistic while counting it in the manifest as
failed_runs.
Add linked maps, comparisons, dashboards and workspace-backed photo acquisition
and viewing for desktop task surfaces.
Certify Mantine viewer interactions and isolate Windows command-line and direct
DNS test fixtures from host process/proxy behaviour.
Cache impact: shortened write_file schema changes tool/prefix hashes once on
upgrade (5572 to 5524 schema tokens); system prompt stays unchanged. Update the
boot goldens for that deliberate contract change. Image discovery stays hidden.
A batch printed its own provenance to the terminal and kept the per-run records
on disk, so the two could not be read together afterwards. The fingerprint
(model, endpoint, window, generations, commit, start time) existed only in
scrollback, and every aggregate the report is read for — funnels, dose response,
pooled deltas — was printed rather than stored. Reconstructing a batch after the
fact, or attributing a figure in the README to the runs that produced it, meant
trusting a terminal transcript; the 24-run and 36-run totals in the last batch
are exactly that failure.
Each paying mode now writes batch.json beside results.jsonl: mode, start and
finish, the checkout's commit and dirty state, model, window, generations, the
task ids and arms the batch covered, planned against completed runs, the summed
usage of the runs that reported any, and the sha256 of both artifacts so a
manifest and the records it describes cannot drift apart.
Two things stay honest. Usage totals are absent when nothing reported any,
because an unmeasured cost must not read as a measured zero, while
usage_reported_runs is written even when it is zero, because that is a fact. And
a checkout git cannot describe is recorded as an unknown commit with no dirty
flag at all, rather than as a clean tree.
The harness measures context cost and could not report its own. It handed the
agent event.Discard, so the usage every billed completion already reports went
nowhere, and a batch's spend could only be read from a vendor console: this run's
index batch cost 604,544 prompt tokens (527,232 cached) plus 12,600 completion
tokens over 24 runs, and nothing in the tree would have said so.
The bench now installs a sink and totals what the agent reports — the same record
the product's own round path, compaction, command routing and failure paths bill
to. One path, not a second count of the same fact.
Two things stay honest about what the number means. A dry run prints an empty
suffix rather than "prompt-tok=0", because an unmeasured cost must not read as a
measured one. And the count is called usage_reported_requests: a request that
failed before reporting usage leaves no event, so it is not a count of attempts.
The substrate the Status section asked for now has a result, and the result has a
direction, a size that is not yet quotable, and a reason.
The direction reproduces: six of six tasks, and the pooled paired batch searches
less and pages in less with the cue visible. A size is visible on the four tasks
whose absent-cue arm stayed out of a loop (44% less prompt cost), and it is
recorded as a post-hoc subset rather than an effect size: the two excluded tasks
are excluded because their arm looped here, and the qualification bar does not
screen for that. Saying 44% without that qualifier would mean picking the sample
after seeing the result, which is the mistake that invalidated the previous round.
The counts are stated separately because they differ: 36 runs executed, and the
token totals belong to the 24-run index batch that carried the usage line.
The summary request is one user message that ends on the last tool output, so
the model sometimes carries on the agent's work (pseudo tool calls, "next I
will ...") and the host installed that reply as the digest: a read-only fold has
no changed file for the coverage guard to compare.
- The user turn now ends with an explicit ask for the briefing.
- A reply with no level-2 heading, the shape the summary prompt requires, fails
with the typed code summary_not_digest. It is billed to the compaction, is
retried like any transient summary failure once the input has grown, and
degrades to the mechanical digest when the fold must free space.
- The code has frontend text in the TUI catalogues and the desktop table.
- Test doubles that answer summary requests now speak the heading contract.
The reference-solution half covered only the e2e, train and verification-stress
corpora, so three committed corpora with the same tasks/ layout carried no
evidence that any attempt can score: a grader that can never pass is
indistinguishable from a task with no solution, and a corpus of those teaches
that nothing is ever accepted.
Their pristine seeds already fail, checked across all eleven tasks, so this adds
the reference solutions the guard needs: one ANSWER.md each for the three fleet
tasks, and the non-positive-skipping calc.py the eight project-check graders
assert. The e2e suite stays the single declared exception, named in the skip.
Refs #11852.
TestInstancesDifferBetweenRuns instantiated every task twice and failed
whenever two independent draws collided. Three tasks answer with a bounded
number (i70 quota 1200..9800, i47 lease 15..300, i112 lease 20..420), so a
run fails with probability 1/8601 + 1/286 + 1/401, about 0.61%.
liveRand now delegates to liveRandFrom(io.Reader). The test asserts the
invariant at the seed: two live generators do not share a stream, the same
entropy yields identical instances and different entropy yields different
values.
The reference-solution guard fired only for the train corpus. For every other corpus a missing solution/ was a silent skip, so a grader that can never pass stayed indistinguishable from a task with no solution - the case the rule above it exists to prevent. #11380 supplied the ten vs-pos solutions; this extends the guard to the whole verification-stress corpus and supplies the ten tasks it was still skipping.
The e2e suite keeps the skip, and the skip message now says why: its handwritten tasks do not commit a reference solution.
Verified: go test ./cmd/e2ebench -run TestCorpusGradersPassTheReferenceSolution is green across e2e, verification-stress and train. Removing any single solution/ fails the guard at corpus_test.go:254 (checked by hand for vs-neu-cfgkey), so the added solutions are what the guard grades.
Screened 87 candidates, staged: one pass over the pool, then two more only for
the candidates that were clean on both boundary arms. The six that held on both
arms in all three passes stay, the rest are dropped, so the efficiency substrate
only contains tasks that qualify:
quarter i62-epochlatch, i37-dispatchlag
half i106-budgetseal, i70-ingestfloor
default i47-tenantlease, i51-routedelay
All six are SnippetStop, SnippetThenRead or DefensiveRead in each of their
eighteen judged cells - nothing post-sufficient.
What the screening found: the answer kind decides whether a candidate holds.
Half-tier candidates whose answer was a bare number never survived (0 of 8);
the same tier with a codename answer survived 11 of 20. The dirty class in both
cases was PostSufficientRetrieval - the model re-checking a value it already had
- so the shape that needs no checking is the one that holds.
Also: i07 and i10 were re-measured after the review asked for it (their old
numbers came from the grep shape they no longer use), NeverSufficient is
reported separately rather than counted as clean, and the i11/i12 rows are gone
because those tasks are not in the corpus.
The review settled the bar: a task qualifies only if it is clean on both of its
boundary arms, and none of the first six candidates is. This adds five more,
authored to the same shape but with a single value each and a call whose
arguments name the topic (./scripts/x {{cue}}), which is the shape the review
pointed at for i07 and i10.
- i15-ingest-lag and i16-commit-fence (quarter), i17-route-budget and
i18-salvage-epoch (half), i19-tenant-quota (default).
- All five preflight clean at their own tier's generation.
- The corpus stays unqualified until six candidates hold on both arms across a
three-pass batch; the ones that do not are dropped rather than promoted.
The authoring guard skips any task without a solution/, so all twenty
verification-stress graders have only ever been checked against the pristine
seed - never against an answer that should pass. Ten now carry one.
`-policy` is the MemoryBench experiment arm: `memory-off` hides the memory
store so a run can be compared against itself, and the only switch that
separates the two arms of that counterfactual. It is defined at
`cmd/e2ebench/main.go:268` and used in the recipe further down this README, but
the flag table never listed it.
Placed with the other suite-mode experiment arms (`-anchor`, `-cache`) rather
than beside `-ablate`, so it does not collide with the open #11364 that edits
that row.
The bar named a field that does not exist (`PostSufficientSearches`) and a mode
that cannot produce counts (`-mode=calibrate`). The gate is the run's
`stopping_class` from the `-mode=run-index` report: a task qualifies when it is
neither `PostSufficientRetrieval` nor `PostSufficientRunaway`, so `SnippetStop`,
`SnippetThenRead` and `DefensiveRead` all qualify, and a defensive read is
deliberately not counted as waste. Also records that calibrate is not a
screening tool, which is why its cue-visibility profile is the same for every
task.
The benchmarks README still described this harness as the pre-#8244
comprehension/TTL/prune A/B, and taught a subcommand and flags the code has not
had since that rewrite.
ablation.FullFold has had no reader since #8244 (397ed30ec) replaced the
multi-threshold path with the single checkpoint design: foldSource and
carryPriorDigests lost their Off(FullFold) checks together with their test, so
-arm=incremental was identical to the default, and the default already folds the
model-visible projection. The checkpoint design is the intended one, so retire
the dead axis rather than restore the old re-read.
- ablation: drop the FullFold module (constant, comment, module list entry)
- compaction bench: drop the -arm flag, the arms struct, foldArm, and the arm
label from the report title and the JSON report
- benchmarks/README.md: drop the "Fold arms" section, and correct the paragraph
that claimed a fold re-derives its digest from the canonical transcript
- docs/CLI.md: drop full-fold from the --ablate list
Verified: go build ./...; go vet; go test ./benchmarks/compaction/
./internal/contract/ablation/; go run ./tools/repolint clean; golangci-lint
v2.12.2 (the version CI pins) 0 issues. -mode=cost still reports the same eight
generations, -arm is now an unknown flag, and the JSON report no longer carries
an arm label.
Fixes#11358.
"pnpm install" contains "npm install" as a substring, so verify.sh's
negative grep matched the very answer the positive grep demanded, and the
task could never pass. Disambiguate the negative check so it only matches
"npm install" when it is not preceded by "p", instead of anchoring the
positive check to line start, which would have moved the grader out of
line with every sibling task's plain substring convention.
Also extends TestSolvableCorpusSeedsMustNotGradeClean to the memorybench
corpus (it was not covered by the existing e2e/verification-stress/train
authoring guard) and adds a targeted regression test pinning the four
cases worked through by hand in the report.
Credit to chuiwenwei for the root-cause analysis and both candidate fixes.
Fixes#11329
The readiness gate read only the checks the current process declared. A Goal
restored into a process whose declaration had changed carried its captured
baseline in the delivery checkpoint, but nothing enforced it, so running the
replacement alone was enough to finalize. The Goal resume path was shown
reachable by goal_resume_project_check_test.
The gate now also owes every baseline check no current declaration names,
inside the delivery scope that captured it, anchored on the same write the
gate already uses. The trust-domain rule settles the authority question: the
workspace cannot supersede a captured criterion.
The full switch to ledger obligations is not taken: a resumed process starts
from an empty ledger, where the obligations owe nothing at all, so the switch
would open the resume path this closes. A restored checkpoint from another
scope does not bind an unscoped turn.
The task tool, read-only tasks, fleets, parallel tasks, the sub-agent
store and records, profiles and the child's capability grant move from
internal/runtime/agent to internal/runtime/delegation. They are clients
of the agent: they build child agents and run them. The executor loop
now names none of them.
What the loop itself does for a child stays in agent. child_run.go holds
running an agent as a child and composing its answer and host receipts.
child_registry.go holds shaping the child's tool surface. Delegation uses
a small exported surface for these: AppendHostReceipts, AttachReviewReport,
CompleteSubtaskContract, ExpandToolPatterns, MaxToolOutputBytes,
NormalizeToolSchemas, SubagentImageNote, VisionRefFor, WithCallContext
and the host-receipt labels.
Tests followed the code. The ones that check the agent's own judgements
against the real delegation tools (batch skipping, the plan-mode barrier,
contract hints, delegation profiles across the proxy) run as agent_test
in the agent directory through export_test.go. The user-turn retention
test is split so each package pins its half. Test helpers used on both
sides are copied, since neither test binary may import the other. One
test more than before, from that split; none dropped.
Baseline: the moved files carry their debt unchanged under the new path,
and task.go gains the one import line the move costs (-allow-widen).
Trajectories keep derived events, never what was sent to or received from
the provider, so no run could be played back and nothing caught a kernel
change that altered what the model is asked. The cache-first prefix is meant
to stay byte-stable, and only a live run and its bill showed when it moved.
-tape-record DIR records every exchange at the meter, one tape per task and
trial. -tape-replay DIR answers each run from its tape instead of the
provider, and the report's Replay line names the first request whose body
differs from the recorded one: a message by index and role, the tool list,
or a field. The diverging request is kept beside the recorded one.
Two sources of noise reach the model and had to be held still for a replay
to match: the random workdir path, which is in the workspace block, and the
seeded files' times, which appear in any listing the agent takes. Taped runs
use a fixed workdir per task and trial and pin those times, including the
tape root's, since it is ".." in the listing.
Verified with a real run of bugfix-slug-trailing: recorded in 15.3s for
about USD 0.0014, replayed in 6.0s with no provider traffic, all 6 requests
matched, same tokens and cache hits. The control arm holds: replaying with
-ablate all is reported as "diverged at request 1, tools".
-attempts retries until a pass, so it measures whether the agent can solve a
task at all (Pass@<=N). Nothing measured whether it solves it every time,
which is what a user relying on it meets. -trials N runs every task N times,
keeps every outcome, and adds a Reliability line: pass^N (tasks that passed
every trial), pass@N (any trial), mean pass@1 over all trials, and the flaky
tasks by name. A trial the token budget skips is not counted as a failure,
and the line says how many tasks ran every trial. -trials and -attempts
count repeats differently and are refused together. A run without -trials
reports exactly as before.
Verified with a real run: bugfix-slug-trailing, two trials, both passed,
reported as pass^2 100% (1/1), for about USD 0.001.
Three moves close the edges from the contract layer into runtime/evidence:
- The audit records the host emits beside a run (ReadinessAudit,
OutcomeSample, DelegationAudit) move to contract/hostaudit, so event and
trajectory carry them without the ledger that computes them.
ClassifyEvidenceOrigin, which reads delegation text, stays in evidence as
a function over the record.
- taskcontract and plancontract move from contract to runtime: only the
agent and completion consume them, and both are built from evidence
receipts rather than declaring data anything lower reads.
- evidence moves from runtime to safety. It imports nothing above the
subsystem layer, and builtin tools and the trajectory recorder, both
subsystems, depend on it.
layer-order drops from 18 to 5.
Session persistence moves from internal/runtime/agent to
internal/state/sessionstore: the Session type, its checkpoint and event
log, listing sidecars, branches, leases, recovery copies, migration, and
the compaction and subagent records saved beside a transcript. The agent
keeps the loop and imports the store; history, historycatalog, migration,
guardian, instanceid and sessiontool now reach the store instead of the
agent, which drops layer-order from 26 to 18.
Tests that exercise only store internals moved with it; the rest stay in
agent and qualify the moved names. DigestSessionMessages and
CompactionStateSchemaV1 are exported for the agent tests that build
legacy fixtures.
Baseline: carried with -allow-widen. The moved files bring their essay,
file-size, complexity and orphan debt unchanged under new paths, and
boot.go, acp/service.go, serve/serve.go, agent/task.go and
control/controller.go each gain the one import line the split costs.
internal/ was 141 packages side by side, so nothing but a reader's memory
said which were foundations and which assembled the rest. Each now sits in a
layer directory, and the directory is the declaration:
base L0 foundation with no domain in it
contract L1 types and interfaces the layers above agree on
model, tools, safety, ext, state, platform
L2 subsystems, peers of one another
runtime L3 the turn engine
session L4 the controller
assembly L5 the composition root
frontend L6 cli, serve, acp and the host adapters
Only paths change: package names, code and behaviour do not. Imports,
repolint and coverage baselines, sensitive paths, CI package lists and docs
follow the new paths; release notes keep the paths they were written with,
and sample paths inside tests and fixtures keep theirs.
repolint gains layer-order: a package may import its own layer and those
below it, read from where it lives. The 47 imports that point upward today
are recorded as that rule's opening debt (-allow-widen, a new rule's first
baseline) and can only go down. Most of it is config, which still reaches
into eight subsystems and is the next thing to take apart; the rest is
event, history and a few tools reaching into runtime for types that belong
in contract.
The tool schema was held still on the reasoning that changing it churns the
cache-stable prefix. True of a change anywhere, false of a change at the end,
and nothing measured which applied.
Against deepseek-chat, 10 tools behind a 1278-token system prompt: appending at
the tail reports 2048 cached tokens against a 1920 baseline — more, because
every token before the new one is unchanged and the request is longer. The same
tool at the head reports 1024, and editing the fifth tool's description 1536,
which is the tail after the edit.
A tail tool alternating absent/present holds 1920 and 2048 across three
switches with no miss spike, because the absent surface is a byte prefix of the
present one and one cache entry serves both.
It reaches the network and spends a few thousand input tokens, so it stays out
of lint and CI, next to pricecheck.
/compact set Force, and Force was one bool doing five jobs: waive the trigger,
shrink the recent tail, skip the idempotence check, skip the new-history rule,
and skip the fold's own economics. Only the first two are what a person means
by asking. The other three are why typing the command twice bought two
summaries of a context nothing had been added to — the shape a user actually
meets, since this machine's own history shows automatic compaction firing once
in five thousand calls while every /compact fires by definition.
The bool is now two: a request waives the threshold, because asking is what
makes it manual, and waives economics only when the user says so. Both halves
matter. Setting Force to false would have kept the economics and lost the tail
shrink, leaving a command that does nothing at all below the trigger.
Measured in the project's own harness, one /compact per generation over six
generations:
fold calls summarizer input projection @gen6
before 6 96,654 18,311
after 3 79,890 33,594
-50% -17.3% +83.5%
The last column is the trade being made rather than a regression: folding only
when a fold pays for itself means holding more between folds, and 33.6k of a
128k window is what that costs. The three skipped generations were skipped
because a checkpoint costs the whole prefix cache and too little had closed
behind it to earn a second one.
Declining is now an answer. A manual noop returned rejectCheckpoint — an error,
carrying a sentence that said "no foldable region remains" whichever gate had
actually declined. It returns a verdict with the code the host already settled,
and CompactDeclineText turns that code into words for a caller with no phrase
of its own. Over HTTP the endpoint answers 200 with the code rather than the
204 that could not tell a fold from a session it decided not to pay for. The
verdict's Installed is a bool and not a CompactionOutcome because that enum's
zero value is "installed", and a verdict nobody filled in must not claim a fold.
Two smaller things fell out. acceptCheckpointCandidate had a branch that read
like a force exception and rejected exactly what the branch above it rejected;
a checkpoint landing back at the trigger has bought the next fold rather than
avoided one, and nobody may waive that. And the overflow paths passed Force
alongside a trigger that already implies both waivers.
/compact --force is the escape hatch and stays one: economics only, and it
still cannot land a candidate that is not smaller than what it replaces. The
five behaviours are gated, and the two that matter — a request over an
unchanged context, and one over a few hundred new tokens — were each shown to
buy a summarizer call against the code before this.
yhh