mirror of
https://github.com/esengine/DeepSeek-Reasonix.git
synced 2026-11-07 10:45:59 -10:00
test(bench): preflight the request the model is handed, not the projection
The fixtures were checked against the model-visible projection. That is what the agent believes; a provider.Request is what the model is told, and rendering, sanitization and tool-schema construction sit in between. So each of the 36 arm states now resumes the way a restart does — transcript from disk, projection from its sidecar, the shipped tool surface at that arm — runs one round against a capture provider, and is judged on the request it produced. The capture answers with a clean terminal response. An empty one would buy an empty-final retry and a second request, and the round under inspection is the first. The turn's request is the one carrying tools: a fold on the way in asks the same provider for a digest, and taking request[0] blindly inspected the summarizer's prompt instead. The scan enumerates the provider-visible surface rather than marshalling a Message whole — content, reasoning_content, tool-call names and arguments, responses_items, and every tool name, description and parameter schema — so a leak is reported where it is, and local-only fields the boundary is supposed to strip cannot produce a false hit. Those are asserted absent in their own right: raw_content, LocalOnly, decision receipts, tool execution and memory citations crossing the boundary is a regression whether or not a marker rides along. It found what the projection check could not. i05's answer marker "session" was readable in tools[0].parameters and tools[2].parameters — in all four arms, because a tool schema is not part of any projection. r01's "partial" was one ordinary word away from the same fate. Both are nonces now. Search arms are held to the ablation's definition: both keep positions, only the control has query, and the search-off arm's request may not contain the wording that offers a search — checked over the whole surface, not just the schema, so a description or an index line cannot reintroduce it. Getting a resumed fixture to keep its projection turned up one more thing worth recording: projectionCoversTail cannot revalidate a projection that covers the entire transcript, since it has neither a matching transcript version nor a live tail to sit in front of. Real sessions always have one, because work continues after a fold. The fixtures now do too. Verified by making each fail: stop scanning tool schemas and i05's leak goes quiet; leave the search tool unswapped and the search-off arm is named twice, once for the parameter and once for the wording. benchmarks/* join cmd/ and desktop/ as hosts in the layering rule. Each is a main that assembles a real agent, and a harness reproducing the shipped tool surface has to reach the frontend that defines it. Nothing is opened to the kernel by this: the rule that matters is that nothing below a frontend may import one, and a benchmark is above every frontend. No provider is called anywhere in this commit. This is the gate that says the API spend would measure what it claims to.
This commit is contained in:
1 parent
24ce27f033
commit
fe476b8494
7 files changed
+266
-5
No files matched your search
@@ -7,6 +7,7 @@
|
||||
/reasonix.exe
|
||||
/e2ebench
|
||||
/e2ebench.exe
|
||||
/context-retrieval
|
||||
|
||||
# Website (Astro) build output
|
||||
/site/dist/
|
||||
|
||||
Reference in new issue
Block a user