test(bench): preflight the request the model is handed, not the projection

The fixtures were checked against the model-visible projection. That is what
the agent believes; a provider.Request is what the model is told, and
rendering, sanitization and tool-schema construction sit in between. So each of
the 36 arm states now resumes the way a restart does — transcript from disk,
projection from its sidecar, the shipped tool surface at that arm — runs one
round against a capture provider, and is judged on the request it produced.

The capture answers with a clean terminal response. An empty one would buy an
empty-final retry and a second request, and the round under inspection is the
first. The turn's request is the one carrying tools: a fold on the way in asks
the same provider for a digest, and taking request[0] blindly inspected the
summarizer's prompt instead.

The scan enumerates the provider-visible surface rather than marshalling a
Message whole — content, reasoning_content, tool-call names and arguments,
responses_items, and every tool name, description and parameter schema — so a
leak is reported where it is, and local-only fields the boundary is supposed to
strip cannot produce a false hit. Those are asserted absent in their own right:
raw_content, LocalOnly, decision receipts, tool execution and memory citations
crossing the boundary is a regression whether or not a marker rides along.

It found what the projection check could not. i05's answer marker "session"
was readable in tools[0].parameters and tools[2].parameters — in all four arms,
because a tool schema is not part of any projection. r01's "partial" was one
ordinary word away from the same fate. Both are nonces now.

Search arms are held to the ablation's definition: both keep positions, only
the control has query, and the search-off arm's request may not contain the
wording that offers a search — checked over the whole surface, not just the
schema, so a description or an index line cannot reintroduce it.

Getting a resumed fixture to keep its projection turned up one more thing worth
recording: projectionCoversTail cannot revalidate a projection that covers the
entire transcript, since it has neither a matching transcript version nor a
live tail to sit in front of. Real sessions always have one, because work
continues after a fold. The fixtures now do too.

Verified by making each fail: stop scanning tool schemas and i05's leak goes
quiet; leave the search tool unswapped and the search-off arm is named twice,
once for the parameter and once for the wording.

benchmarks/* join cmd/ and desktop/ as hosts in the layering rule. Each is a
main that assembles a real agent, and a harness reproducing the shipped tool
surface has to reach the frontend that defines it. Nothing is opened to the
kernel by this: the rule that matters is that nothing below a frontend may
import one, and a benchmark is above every frontend.

No provider is called anywhere in this commit. This is the gate that says the
API spend would measure what it claims to.
This commit is contained in:
esengine committed 2026-08-28 10:08:40 -07:00
1 parent 24ce27f033
commit fe476b8494
7 files changed
+266 -5

No files matched your search

+1
View File
@@ -7,6 +7,7 @@
/reasonix.exe
/e2ebench
/e2ebench.exe
/context-retrieval
# Website (Astro) build output
/site/dist/