complete_subtask, the host adjudication of its criteria and the 'lowered by the
host' note are gone, with the handoff audit channel and its metrics, trajectory
field and benchmark columns. A delegated child ends the way any turn does and
the parent reads its last reply.
The host's own receipts appended to a child's reply and the write-scope check
stay: they need a record of what the child changed, which is replaced with the
evidence ledger later.
With the final-readiness gate gone, FinalReadinessError, the readiness audit
and the structured readiness on TurnDone have no producer, and neither do the
contract-shadow, completion-report, evidence-bundle, outcome-progress,
protocol-recovery, verification-drift and project-check-probe audits. Removes
the types, the sink capabilities and their forwarding in every wrapper, the
trajectory and metrics fields, the final_readiness outcome and notice code,
and the delivery-recovery submit port.
The ACP status shape and the desktop reader of old sessions keep their
readiness fields: external editors and recorded sessions still carry them.
A batch printed its aggregates and kept nothing they could be recomputed
from, so a published figure could not be tied to the runs behind it after
the temp directory was gone. -batch DIR now writes results.jsonl (one row
per attempt) beside batch.json (manifest v1), and -mode audit re-derives
the report's headline figures from those rows with the harness's own
aggregation, printing the claim beside the recomputation and naming every
figure that moved.
The manifest records what the figures are over (command axes, a task-id
digest, filters, and a denominator defined in tasks), which run produced
them (commit or an explicit unknown, model/provider names, seed when
known, timestamps, harness and runtime), and why they can still be
checked (results_sha256, the claimed figures, and a ledger of planned /
attempted / completed / scored / failed / unmeasured / excluded that the
audit checks for consistency). Absent figures are omitted and read as
unknown, never as zero.
This is an aggregation audit, not a replay: it proves the published
figures follow from the retained rows. It does not re-execute the model,
tools or environment, and a real replay (commit, corpus fingerprint,
model/provider configuration, seed, flags, raw results, runtime version)
is not implemented.
Studio now requires every config key to be declared once with who may change it
and when it takes effect; the two keys this branch adds were not in that table,
so loading a config panicked with `setting.undeclared: network.dns`.
Both are `Guarded` / `OnOpen`, beside `network.proxy_url`: which resolver a
request goes through is a network boundary, so the model must not change it by
itself, and a new value applies on the next start rather than mid-session.
Also rebased onto studio, which the same push asked for.
The Windows runner's Git Bash has no shasum, and the grader's
`shasum -c ... || exit 1` turned "command not found" into the same exit 1 as
a changed file, so the reference solution failed there with nothing printed.
The SHA-256 manifest is now checked with hashlib inside the grader's Python,
over raw bytes, and a mismatch names the files that differ.
gradeSeed now carries the grader's output in its error: the CI log said only
"exit status 1", which cannot tell a missing tool from a failed check.
Verified locally: on a PATH without shasum the reference solution grades
clean; a CRLF Makefile and an edited Go file each fail and are named; the
pristine seed still fails. snapshot.sh --check, the corpus tests and
make check pass.
The e2e fixtures finish in 5-15 requests, while a real task in this tree took
deepseek-flash 38. A change aimed at cutting round trips cannot show up on
the short suite, so this adds four long tasks whose workdir is a git archive
of commit 42636ddcb (pricing, ablation, agentgraph and five stdlib-only
leaves) plus a trimmed REASONIX.md, a Makefile, a module README and a small
benchctl consumer:
- rt-cache-write-rate: a CacheWrite rate threaded through cost, fingerprint,
card identity and the catalog, plus an Anthropic row that uses it
- rt-stale-skills: Markdown-only repair of three project skills with no
recognised verifier, the shape behind #13293's lingering stale_verification
- rt-rate-lookup-signatures: five lookups moved to (RateCard, bool),
CatalogKey and ModelRef across about 20 call sites, most in _test.go
- rt-plan-ablation-module: a five-step plan-first task signed off per step
snapshot.sh rebuilds every workdir from the pin and --check detects drift.
One pinned test compares CheckedOn with the wall clock and would fail every
workdir from 2026-11-20; base/ anchors it at the snapshot day instead.
Each task ships a solution/; the corpus is registered with the authoring
guards, which check that every seed fails its grader and every solution
passes. A deepseek-flash baseline (2 trials) solved 8/8 at 13-31 requests
once two grader false negatives in rt-stale-skills were fixed; the README
records it, along with a multi_edit prompt arm and a complete_step
description arm that did not move the request count.
Windows has a backend now, so Available() is true, but it enforces only after the
user turns on windows_integrity. Tests and the e2e bench that ask whether a default
configuration is confined use ConfinesByDefault (false on Windows), and a caller's
override to enforce (an observed run asks for one) is honoured on Windows only
when the user opted in, since enforcing edits permissions on their folders.
model_rounds and round_outcomes treated every call run_script made as a
top-level dispatch, so each inner result reopened a model gap and each inner
dispatch closed one. One recorded code_mode run with 24 executor requests
reported a median of 87.5 rounds. Nested calls carry no ParentID — only an id
derived from the caller's, inside its dispatch→result window — so the digest
had nothing structural to tell them apart.
The nested id is now declared in internal/contract/tool (NestedCallID /
IsNestedCallOf) and the agent's nestedInvoker builds ids through it, so the
producer and the reader share one definition. e2ebench's callNesting tracks
open top-level calls and treats subagent calls (ParentID) and invoker calls
(derived id) alike: neither cuts a round, joins a batch or adds tool time.
roundEnds and splitAtCorrect use the same tracker. NestedCalls reports how
many there were, and ModelRounds' doc now says what it counts.
With -trials N every trial wrote <task>.trajectory.jsonl, so only the last
survived and every trial's digest read it. Trials now write
<task>.trial-N.trajectory.jsonl (legs: <task>.trial-N.segM...); a single
trial keeps the old name. sft mode resolves the task id through
trajectoryTaskID, which strips trial and leg marks; traj, serve and barrier
modes glob *.trajectory.jsonl and label each file by its stem, so trials
read as distinct entries.
Cause: the digest had no identity for tool-made calls; the trajectory path
had no trial in it.
Blast radius: cmd/e2ebench digest and file naming; nestedInvoker's id string
is unchanged (same "<parent>.<n>" format, now built by the contract helper).
Neighbouring behaviours tested: subagent ParentID exclusion, segment paths,
answer-isolation audit across legs, existing round decomposition tests.
Why this layer: the id convention is owned where nested calls are created;
the digest reads it rather than restating the format.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
aggregateArm dropped an unaccounted run's wall before adding it to
s.WallMs, so compare's Solved/hour and global wall/task divided by a
smaller wall than the suite report uses for the same -json artifact:
report.go adds every run's wall, and a killed run's wall is measured
unconditionally in main.go (only its tokens never landed). The
per-class accumulator ran before the skip, so one section printed two
populations, and the sign of "which arm is cheaper" could flip.
WallMs is now the all-run wall and AccountedWallMs the accounted
subset: Solved/hour and wall/task divide the all-run total, while
"Wall seconds / solved" keeps dividing accounted wall by accounted
solves because that denominator is solved runs. No new persisted
state, and neither how Unaccounted is decided (segmentrun.go) nor the
suite report changes.
This is the residual of #13446 applied to a field that is NOT
unmeasured. That PR decided which runs are unaccounted and kept their
spend out of the token/cost ratios; it never touched wall, and this
change does not revisit that decision. It fixes the wall population
those adjacent rate lines divide, measured for every run including the
ones #13446 excluded elsewhere. Not a re-report of #13446.
Verified: both probes are red on 53a8db913 (compare Solved/hour 120.0
vs the report's 3.9; global wall/task -30.0s vs the class row's
+870.0s) and green after; a no-unaccounted fixture renders compare
byte-identically (goldens); "Wall seconds / solved" divides accounted
wall by accounted solves under either rate line reverted to the
accounted total, while the matching probe turns red; the suite
report's Solved/hour convention is pinned unchanged. gofmt,
GOOS=windows go vet ./cmd/e2ebench/, go test ./cmd/e2ebench (only the
pre-existing environment TestStateHomeIsOutsideWorkdirParent fails,
identically with this change stashed), make check (repolint: clean),
and golangci-lint run ./cmd/e2ebench/... (0 issues) pass.
The host also answers 'schedule exec' by running the CLI's entry, so the
supervisor can start it as the child. One kernel at a time ticks, under the
machine's ticker lock.
renderSolveProfiles folded every run that reached a correct checkpoint into
the boundary medians, including runs whose trajectory was never read. With
-checkpoints and no -trajectories those runs carry zero round and
verification tallies, so the report printed "rounds before 0 · rounds after
0 · verifications after 0" as if they had been measured.
Only runs whose split was actually recorded now fold in, and a suite where no
run recorded one says the boundary was never split instead of printing zeros.
The all-measured medians are unchanged.
Review follow-up: Partial now accumulates across legs (a killed first leg plus a
complete second leg is still a lower bound), and the meter comparison skips a
Partial run the same way it skips an unaccounted one, since its missing tail is
not a divergence. Rebased onto current studio; the stub tests use requireShellStub
so test (windows-latest) can run.
A leg whose final record was missing but whose <metrics>.partial snapshot was
read was marked Unaccounted, which gatherSuiteStats skips outright. That
contradicted Partial's meaning — real numbers that stop at the last snapshot
and belong in the totals — so the recovered tokens, cost and lower-bound hint
were all unreachable. Only a leg with neither record is now Unaccounted; a
recovered snapshot stays accounted and sets Partial, matching the semantics
swebench mode already uses.
renderMeterAccounting also treated PromptTokens+CompletionTokens == 0 as an
absent harness measurement, inferring state from a value. A persisted record
saying zero is a valid zero, so the guard is now the explicit Unaccounted flag
alone.
Tests: the copied audit case that asserted unaccounted=1 for a sidecar-recovered
run encoded the wrong semantics and now asserts Partial=true, Unaccounted=false
and that 1200/80 and the cost reach the aggregates; the divergence fixture that
relied on zero tokens sets Unaccounted explicitly, and a new case pins that a
persisted zero is priced on both sides. The record-less case and the healthy-run
control are unchanged in intent.
runSegments read only the final metrics file. When a child was killed or
timed out it folded the missing record in as an accounted zero, deleted the
<metrics>.partial snapshot the sink had been writing, and left Unaccounted
unset, so every published total omitted what the run spent.
Read the sidecar when the final record is absent, keep the file it came from,
set Partial, and mark the run Unaccounted when neither exists.
renderMeterAccounting no longer prices a run whose harness reported no tokens
against the meter, which invented a -100% divergence.
recallTail had three outcomes and only two reached the audit channel: a
carried memory note and a normal recall each recorded a content-free
decision, while the retrieval ablation switch returned silently and left
lastRecall pointing at an earlier turn. A trajectory could not tell
"retrieval switched off" from "recall found nothing" from "suppressed by a
carried note".
All three now record a RecallResult and emit one memory-recall audit. The
identity travels as a typed memory.SuppressionCode on its own field, from
RecallResult through event.MemoryRecallAudit into the trajectory record's
suppression_code. The prose sentence stays exactly as it was, for readers
only: consumers match the typed field and never split the code out of the
text, so wording cannot become load-bearing again. The new field is additive
and omitempty, so existing records and batches decode unchanged.
The ablated turn also records lastRecall, matching the note path: the
diagnostic surface is defined as the last real turn's decision, and that
turn did decide not to retrieve.
Keeps one /answer handler that serves both question engagement and surface answers, and
carries the computer driver and native-surface options through the workspace switch.
A phone pairs by scanning a QR code, and the credential it keeps rides a
cookie scoped to the origin it scanned. The host held that pairing in memory
only: closing the share or restarting the process threw every device away, so
the next launch asked for a fresh scan.
[serve] remember_paired_devices, false by default and the user's alone (a
project reasonix.toml that sets it is ignored and reported), keeps three things
when it is on: the phones that paired (digests only, written to
<state>/device-trust.json and adopted at startup), the address the share last
listened on, and whether the share was open at all. The persist hooks read the
key when they run, so the panel switch takes effect without a restart, and
turning it off forgets on the spot: everything paired is revoked and both files
are removed. A snapshot taken before a later change is dropped by revision, so
a concurrent pairing write can never overwrite a revoke. Address ordering keeps
its upstream preference. A phone that paired never expires: the point of the
switch is that one's own LAN phone connects again next time, and that is
intentional.
Closing the share still unpairs everything; that is the person withdrawing it.
Only the process going away keeps the phones. The state directory is the
kernel's, outside every workspace, and agent-run commands have no path to it:
sandbox writes are confined to the workspace and ~/.reasonix, and the trust
file never enters them. A restart reopens the listener only when the switch
was on and the share was open when the process left — the state of things the
person chose, not a new one.
Fixes#11834
Refs #11878
Problem: Studio packages need payload and installed-shell acceptance, and a
complete artifact gate before publication. Running the Electron and matrix
contracts on Windows also exposed host-dependent test fixtures.
Root cause: #12914 supplies native Linux ARM64 packaging but checks only Go
payload architectures and selected artifact names. The macOS bundle fixture
used the host OS's temporary path, and the matrix fixture relied on PATH
lookup for a mock git command instead of binding it in the tested shell.
Fix: Keep upstream native runners and system FPM, consolidate ELF checks,
verify both Linux package formats, exercise the installed shell, and require
all twelve nonempty release artifacts. Guard installation acceptance with
GITHUB_ACTIONS=true. Use platform-specific workspace fixtures and a shell
function for the matrix's git fixture, keeping all Windows test coverage.
Rebase on current studio and preserve adjacent release notes.
Verification: Electron 143 tests and Linux/script 39 tests pass, along with
workflow contracts, desktop Go tests, builtin/boot regressions, vet, lint,
repository checks, shell syntax and Actionlint. Previous-head hosted native
Linux package verification and installed acceptance passed for amd64/arm64;
replacement-head Windows CI remains the platform acceptance gate.
Refs #12906
A click on a Windows toast grants the right to take the foreground to the
process it activates, which is the kernel host; the window to raise is the
shell's. The host now grants the right before forwarding the click, and the
shell lifts a window the platform refused to focus. A toast that could not be
delivered is written to the host log instead of being dropped.
esengine