test(tui): settle the Bash truncation recovery route empirically

`cargo test --workspace` has been red on this branch since the tool-honesty
sweep (33e59b9b9) rewrote SPILLOVER_RECOVERY_HINT to name
`retrieve_tool_result`. Nobody caught it because every routine command in
crates/tui/AGENTS.md is a `--bins` run, and `--bins` never compiles
crates/tui/tests/.

The failing assertion read like an intentional shell-vs-tool-result
distinction. It is not. Git says it came from #5018's "no storage language"
pass: #4619 shipped the footer naming `retrieve_tool_result ref=art_<call>`,
#5018 replaced the entire recovery line with "view full output in the tool
details view" — a view the model cannot open — and froze the removal as a
negative assertion, and #5212 restored the artifact path but left the stale
negative behind.

Decided from behavior rather than names: the scenario now scripts two more
turns. The mock reads the ref out of the receipt text the model was actually
handed and calls `retrieve_tool_result` with it; the run returns the exact
sentinel line the receipt omitted (line 506 of the failure artifact). So the
footer names a route the model can take from that receipt, and the assertion
is inverted with the reasoning recorded at the assertion.

A `File action="read"` probe on the on-disk path was written too and then
dropped: it showed the path is *conditionally* reachable —
ToolContext::resolve_path short-circuits under trust mode, so the read
succeeds in an --auto session and is a path escape otherwise, and even when it
succeeds it pages rather than seeking the omitted range. The truncate.rs doc
comment claiming that route is flatly "refused as a path escape" is corrected
to say what actually happens.

Second, independent failure in the same test, previously masked because the
receipt assertion tripped first: #5212 moved the sentinel to stderr line 100
citing a "22 KB head bound" that does not exist. shell_output bounds each
stream to TRUNCATED_HEAD_BYTES = 30_000/5 = 6_000 bytes of head, so line 100
landed in the stream's own omitted middle and the artifact never carried the
sentinel. Moved to line 50, the middle of the [31, 68] window where the
sentinel survives the stream bound and still falls outside the preview's
32 KiB head, with the arithmetic written down next to the constant.

crates/tui/AGENTS.md: `--bins` and `--tests` are disjoint target sets; say so
and put a `--tests` run in the routine block.
This commit is contained in:
Hmbown committed 2026-08-04 04:09:09 -07:00
1 parent 0963aa6d21
commit 7e13a3cbf6
3 files changed
+142 -50

No files matched your search

+17 -2
View File
@@ -43,13 +43,28 @@ Adding a string is a four-part change: see `locales/AGENTS.md`.
## Verification
```sh
cargo test -p codewhale-tui --bins --locked # full unit suite
cargo test -p codewhale-tui --bins --locked # unit suite (bin targets only)
cargo test -p codewhale-tui --tests --locked # every crates/tui/tests/ target
cargo clippy --workspace --all-targets --locked -- -D warnings
```
Narrower reruns of the slow acceptance targets, once `--tests` has told you
which one moved:
```sh
cargo test -p codewhale-tui --test qa_pty --locked # PTY snapshots
cargo test -p codewhale-tui --test release_runtime_qa --locked
cargo test -p codewhale-tui --test terminal_matrix_qa --locked
cargo clippy --workspace --all-targets --locked -- -D warnings
```
**`--bins` and `--tests` are disjoint target sets.** `crates/tui/tests/` holds
two dozen process-level acceptance targets that a `--bins` run never compiles,
let alone executes, so a green `cargo test -p codewhale-tui --bin codewhale-tui`
says nothing about them. `adaptive_evidence_acceptance` sat red across two
releases for exactly that reason: every routine command anyone ran was a `--bins`
run, and only `cargo test --workspace` reached it. Run both, or run the
workspace gate.
Run clippy with `--all-targets`: `--bin` alone skips test targets and lets lints
reach CI.
+16 -11
View File
@@ -352,13 +352,18 @@ pub const SPILLOVER_RECOVERY_HINT: &str = "omitted range recovery:";
/// Model-facing recovery instruction for a truncated tool result.
///
/// The previous text — "read it back with the read_file tool or with sed line
/// ranges" — offered three routes and **all three were dead**: `read_file` is
/// not model-visible (only `File` is), `File action="read"` on an artifact
/// under `~/.codewhale/sessions/` is refused as a path escape, and `sed`
/// through `Bash` reads outside the workspace too. Meanwhile
/// `retrieve_tool_result` — model-visible, purpose-built, and already named
/// correctly by the web overflow path in `tools/web/overflow.rs` — went
/// unmentioned.
/// ranges" — named `read_file`, which is not model-visible at all (only `File`
/// is), and otherwise leaned on reaching the artifact by path. Reaching it by
/// path is *conditional*: `ToolContext::resolve_path` short-circuits under
/// trust mode, so `File action="read"` on an artifact under
/// `~/.codewhale/sessions/` succeeds in a trusted/auto session and is refused
/// as a path escape otherwise — and even when it succeeds it pages the file
/// rather than seeking the omitted range. Meanwhile `retrieve_tool_result` —
/// model-visible, purpose-built, unconditional, and already named correctly by
/// the web overflow path in `tools/web/overflow.rs` — went unmentioned.
/// `tests/adaptive_evidence_acceptance.rs` proves end to end that a model
/// handed one of these receipts can take the named ref and get the omitted
/// bytes back.
///
/// The distinction that matters is retrievability, not tidiness. An adaptive
/// session artifact carries an `art_<id>` the retrieval tool resolves, so name
@@ -947,10 +952,10 @@ mod tests {
.unwrap_or_else(|e| e.into_inner())
}
/// The old hint named `read_file` (unregistered), `File action="read"`
/// on a path outside the workspace (refused as a path escape), and `sed`
/// through `Bash` (also outside the workspace) — three dead routes — while
/// never naming `retrieve_tool_result`, which is model-visible and exists
/// The old hint named `read_file`, which is not registered for the model
/// at all, and otherwise pointed at routes that only reach the artifact
/// under trust mode (see [`spillover_recovery_instruction`]), while never
/// naming `retrieve_tool_result` — model-visible, unconditional, and built
/// for exactly this.
#[test]
fn truncation_footer_names_a_recovery_route_that_works() {
+109 -37
View File
@@ -24,6 +24,7 @@ use wiremock::{Mock, MockServer, Request, Respond, ResponseTemplate};
const MODEL: &str = "adaptive-evidence-test";
const SUCCESS_CALL_ID: &str = "call_bash_success";
const FAILURE_CALL_ID: &str = "call_bash_failure";
const RETRIEVE_CALL_ID: &str = "call_retrieve_omitted_range";
const SUCCESS_SENTINEL: &str = "DEEP_SUCCESS_EVIDENCE_SENTINEL_4619";
const FAILURE_SENTINEL: &str = "DEEP_FAILURE_EVIDENCE_SENTINEL_4619";
@@ -42,16 +43,10 @@ async fn headless_bash_success_and_failure_are_distinct_bounded_exact_evidence()
);
let requests = server.received_requests().await.expect("recorded requests");
let success_receipt = requests
.iter()
.filter_map(|request| request.body_json::<Value>().ok())
.find_map(|body| tool_result_content_for(&body, SUCCESS_CALL_ID).map(str::to_owned))
.expect("model-visible Bash success receipt");
let failure_receipt = requests
.iter()
.filter_map(|request| request.body_json::<Value>().ok())
.find_map(|body| tool_result_content_for(&body, FAILURE_CALL_ID).map(str::to_owned))
.expect("model-visible Bash failure receipt");
let success_receipt =
receipt_for(&requests, SUCCESS_CALL_ID).expect("model-visible Bash success receipt");
let failure_receipt =
receipt_for(&requests, FAILURE_CALL_ID).expect("model-visible Bash failure receipt");
for (receipt, sentinel) in [
(&success_receipt, SUCCESS_SENTINEL),
(&failure_receipt, FAILURE_SENTINEL),
@@ -66,9 +61,25 @@ async fn headless_bash_success_and_failure_are_distinct_bounded_exact_evidence()
);
assert!(
receipt.contains("/artifacts/"),
"the footer deliberately names the on-disk artifact so the model can read the omitted range back"
"the footer names where the omitted bytes live on disk"
);
// The receipt must name a route the model can take *from this
// receipt*. It previously asserted the opposite — that the footer must
// NOT name `retrieve_tool_result` — which came from #5018's
// "no storage language" pass, not from any shell-vs-tool-result
// distinction: #4619 shipped the footer naming
// `retrieve_tool_result ref=art_<call>`, #5018 replaced the whole
// recovery line with "view full output in the tool details view" (a
// view the model cannot open) and froze that removal as a negative
// assertion, and #5212 restored the artifact path but left the stale
// negative in place. The follow-up probe below settles it empirically
// for *this* receipt: the scripted model reads the ref out of the
// receipt text it was handed, calls `retrieve_tool_result` with it,
// and gets back the exact line the receipt omitted.
assert!(
receipt.contains("retrieve_tool_result"),
"the footer must name the recovery route the model can actually take"
);
assert!(!receipt.contains("retrieve_tool_result"));
assert!(!receipt.contains("[Exact evidence retained"));
assert!(!receipt.contains(sentinel));
assert!(
@@ -79,6 +90,17 @@ async fn headless_bash_success_and_failure_are_distinct_bounded_exact_evidence()
}
assert_ne!(success_receipt, failure_receipt);
// The model followed the `ref=` the failure receipt named and got the byte
// range the receipt omitted. The mock parsed that ref out of the receipt
// text itself, so this only passes when the footer hands over a ref the
// retrieval tool can resolve in the origin session.
let retrieve_receipt = receipt_for(&requests, RETRIEVE_CALL_ID)
.expect("model-visible retrieve_tool_result receipt");
assert!(
retrieve_receipt.contains(FAILURE_SENTINEL),
"retrieve_tool_result must return the range the receipt omitted, got: {retrieve_receipt}"
);
let artifact_dir = find_artifact_dir(home.path()).expect("origin-session artifacts");
let payloads = std::fs::read_dir(&artifact_dir)
.expect("artifact directory")
@@ -160,26 +182,43 @@ struct EvidenceScenario {
requests: Arc<AtomicUsize>,
}
/// Scripted four-turn scenario. Turn 3 is the empirical half of the design
/// question this test settles: rather than asserting from the outside which
/// recovery route a Bash receipt *should* name, the scripted model reads the
/// route out of the receipt it was actually handed and takes it, so the
/// assertion in the test body observes whether that route returns the bytes
/// the receipt omitted.
impl Respond for EvidenceScenario {
fn respond(&self, request: &Request) -> ResponseTemplate {
let sequence = self.requests.fetch_add(1, Ordering::SeqCst);
let body = request.body_json::<Value>().unwrap_or(Value::Null);
let raw = body.to_string();
let response = if raw.contains(FAILURE_CALL_ID) {
final_sse()
} else if raw.contains(SUCCESS_CALL_ID) {
bash_tool_sse(FAILURE_CALL_ID, false)
} else {
bash_tool_sse(SUCCESS_CALL_ID, true)
let response = match sequence {
0 => bash_tool_sse(SUCCESS_CALL_ID, true),
1 => bash_tool_sse(FAILURE_CALL_ID, false),
2 => {
let receipt = tool_result_content_for(&body, FAILURE_CALL_ID)
.expect("Bash failure receipt on the third request");
let reference = quoted_after(receipt, "ref=\"")
.expect("failure receipt must hand over a retrieval ref");
retrieve_tool_sse(reference, FAILURE_SENTINEL)
}
_ => final_sse(),
};
assert!(
sequence < 3,
"unexpected extra model request #{sequence}: {raw}"
sequence < 4,
"unexpected extra model request #{sequence}: {body}"
);
sse_response(response)
}
}
/// Read the recovery ref the truncation footer hands the model, e.g. the
/// `art_call_bash_failure` inside `… call retrieve_tool_result with ref="…"`.
fn quoted_after<'a>(receipt: &'a str, marker: &str) -> Option<&'a str> {
let rest = &receipt[receipt.find(marker)? + marker.len()..];
rest.get(..rest.find('"')?)
}
fn run_exec(workspace: &Path, home: &Path, server: &MockServer) -> std::process::Output {
std::fs::create_dir_all(home.join(".codewhale")).expect("config directory");
std::fs::create_dir_all(home.join(".deepseek")).expect("legacy config directory");
@@ -232,6 +271,13 @@ fn find_artifact_dir(home: &Path) -> Option<PathBuf> {
})
}
fn receipt_for(requests: &[Request], call_id: &str) -> Option<String> {
requests
.iter()
.filter_map(|request| request.body_json::<Value>().ok())
.find_map(|body| tool_result_content_for(&body, call_id).map(str::to_owned))
}
fn tool_result_content_for<'a>(body: &'a Value, call_id: &str) -> Option<&'a str> {
body.get("messages")?
.as_array()?
@@ -251,30 +297,56 @@ fn bash_tool_sse(call_id: &str, success: bool) -> String {
(FAILURE_SENTINEL, "BASH-FAILURE")
};
let command = probe_command(sentinel, prefix, success);
let arguments = serde_json::to_string(&json!({
"action": "run",
"command": command,
"timeout_ms": 30_000
}))
.expect("tool arguments");
tool_call_sse(
call_id,
"Bash",
&json!({"action": "run", "command": command, "timeout_ms": 30_000}),
)
}
/// Take the recovery route the receipt named, asking for the omitted range by
/// the sentinel that rides in it.
fn retrieve_tool_sse(reference: &str, query: &str) -> String {
tool_call_sse(
RETRIEVE_CALL_ID,
"retrieve_tool_result",
&json!({"ref": reference, "mode": "query", "query": query}),
)
}
fn tool_call_sse(call_id: &str, name: &str, arguments: &Value) -> String {
let arguments = serde_json::to_string(arguments).expect("tool arguments");
[
chunk(json!({"id":"tool","object":"chat.completion.chunk","model":MODEL,"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":call_id,"type":"function","function":{"name":"Bash","arguments":arguments}}]},"finish_reason":null}]})),
chunk(json!({"id":"tool","object":"chat.completion.chunk","model":MODEL,"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":call_id,"type":"function","function":{"name":name,"arguments":arguments}}]},"finish_reason":null}]})),
chunk(json!({"id":"tool","object":"chat.completion.chunk","model":MODEL,"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}],"usage":{"prompt_tokens":10,"completion_tokens":2,"total_tokens":12}})),
"data: [DONE]\n\n".to_string(),
].join("")
}
/// Stderr filler line the sentinel rides on, which has to land in a narrow
/// window. `shell_output` bounds each stream to `TRUNCATED_HEAD_BYTES` =
/// 30_000/5 = 6_000 bytes of head plus 24_000 of tail, so anything past ~line
/// 68 of stderr never reaches the artifact at all. Below that, the bounded
/// stdout section (~30.1 KB: head + notice + tail) plus the `STDERR:`
/// separator puts stderr filler line *n* at roughly 30_110 + 86n bytes of the
/// result, so anything before ~line 31 is still inside the preview's 32 KiB
/// head and the receipt would show it. Line 50 sits near the middle of
/// [31, 68] with ~1.6 KB of slack on each side.
///
/// #5212 wrote 100 here against a "22 KB head bound" that does not exist: the
/// sentinel landed in the stream's own omitted middle, so the artifact never
/// carried it and this test's retrievability assertion failed. It went
/// unnoticed because an earlier assertion in the same loop failed first.
const SENTINEL_LINE: usize = 50;
/// Shell fixture that emits enough bytes to force exact-evidence routing under
/// the 32_768-token default threshold. The Bash adapter self-bounds each
/// stream to ~30 KB, so a single stream would now fit inside the hybrid
/// 32 KiB + 8 KiB preview budget; the probe therefore fills stdout AND stderr
/// (~60 KB combined) so the envelope still omits a middle range. The sentinel
/// rides stderr at filler line 100: deep enough to survive the shell tool's
/// own 22 KB head bound (so the artifact retains it) yet beyond the preview's
/// 32 KiB head (so the model receipt omits it). The probe executes through
/// the platform shell — bash on Unix, `cmd /C` on Windows
/// (#1691) — so each platform needs native syntax to exercise the same
/// routing path.
/// (~60 KB combined) so the envelope still omits a middle range, with the
/// sentinel at [`SENTINEL_LINE`] of stderr. The probe executes through the
/// platform shell — bash on Unix, `cmd /C` on Windows (#1691) — so each
/// platform needs native syntax to exercise the same routing path.
#[cfg(not(windows))]
fn probe_command(sentinel: &str, prefix: &str, success: bool) -> String {
let trailer = if success { "" } else { "; exit 7" };
@@ -282,7 +354,7 @@ fn probe_command(sentinel: &str, prefix: &str, success: bool) -> String {
"i=0; while [ \"$i\" -lt 2800 ]; do printf '{prefix}-%04d-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\n' \"$i\"; i=$((i + 1)); done"
);
let stderr_loop = format!(
"j=0; while [ \"$j\" -lt 2800 ]; do if [ \"$j\" -eq 100 ]; then printf '%s\\n' '{sentinel}'; fi; printf '{prefix}-ERR-%04d-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\n' \"$j\"; j=$((j + 1)); done"
"j=0; while [ \"$j\" -lt 2800 ]; do if [ \"$j\" -eq {SENTINEL_LINE} ]; then printf '%s\\n' '{sentinel}'; fi; printf '{prefix}-ERR-%04d-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\n' \"$j\"; j=$((j + 1)); done"
);
format!("{stdout_loop}; {{ {stderr_loop}; }} >&2{trailer}")
}
@@ -304,7 +376,7 @@ fn probe_command(sentinel: &str, prefix: &str, success: bool) -> String {
);
let stdout_loop = format!("0..2799 | ForEach-Object {{ Write-Output ({stdout_line} -f $_) }}");
let stderr_loop = format!(
"0..2799 | ForEach-Object {{ if ($_ -eq 100) {{ [Console]::Error.WriteLine('{sentinel}') }}; [Console]::Error.WriteLine(({stderr_line} -f $_)) }}"
"0..2799 | ForEach-Object {{ if ($_ -eq {SENTINEL_LINE}) {{ [Console]::Error.WriteLine('{sentinel}') }}; [Console]::Error.WriteLine(({stderr_line} -f $_)) }}"
);
let trailer = if success { "" } else { "; exit 7" };
format!("{stdout_loop}; {stderr_loop}{trailer}")