fix(tui): a fork continues when a turn lost its tool call

Forking a compacted conversation and sending the next message could fail on
that first message with `400 No tool output found for tool call …`, and
retrying the fork answered `cannot identify an exact saved-history boundary`.
Both start in a turn whose tool call never produced a result — a failed tool,
or a process that died mid-call. That call is persisted as a `tool_call` item
carrying the call and its failure text, with no `tool_result_for` item, so a
history rebuilt from the records emitted the call alone: the response that
followed was glued into the same assistant message (no result had flushed it),
and the repair answering the call put its answer next to a message that now
held two responses — a shape the Responses API reads as the earlier call never
being answered. The messages that repair wrote also stopped the fork's
document from matching its records, so `/undo`, retry and fork-again refused.

A rebuilt turn now answers that call where the live transcript held its
answer: the item's own failure text for a failure, and the repository's
interrupted-call notice for a call the process never finished — the same
spelling the session repair writes. An answer is paired only with a call the
rebuild actually emits (a snapshot with no tool name is skipped, #5823), and a
call the turn already answers elsewhere is left alone. A call still running is
deliberately untouched: the engine is waiting for its result.

The recovery projection also reads the compaction checkpoint the engine
installs into a synced history as the engine's own message rather than a
prompt no turn record holds. A fork of a compacted source inherits the summary
in its system prompt, so the engine wrote that checkpoint into the fork's own
document on its first load, and the prompt walk then refused the cut.

Tests: cargo test -p codewhale-tui --lib runtime_threads — 240 passed, 1
failed (host_goal_loop_kickoff_arms_one_continuation_and_parks_at_engine_cap,
a load-sensitive goal-continuation test that fails the same way on the
unmodified checkout and passes alone). All three new tests fail without the
fix: a_rebuilt_turn_answers_a_tool_call_whose_outcome_the_store_lost (the
fork's document came out as assistant[thinking, tool_use, thinking, tool_use]
plus a repair receipt), a_compaction_checkpoint_is_not_a_kept_prompt, and
a_lost_call_is_answered_only_where_its_own_outcome_is_known. compaction — 154
passed; session_manager — 107 passed; codewhale-runtime tool_history_repair —
8 passed; rustfmt clean; clippy clean for the touched crates under the CI
allow list.
This commit is contained in:
Ben Gao committed 2026-09-27 07:57:58 +08:00
1 parent 2d9613bac0
commit 9ddf26bcc6
4 files changed
+542 -2

No files matched your search

+4 -1
View File
@@ -10,7 +10,10 @@ use std::collections::{HashMap, HashSet};
use codewhale_models::Role;
use codewhale_models::{ContentBlock, Message};
const CRASH_REPAIR_CONTENT: &str =
/// The repository's one spelling for a tool call the process lost while it was
/// running. The session-facing repair writes it here, and a history rebuilt
/// from turn records makes the same call say the same thing.
pub const CRASH_REPAIR_CONTENT: &str =
"Tool call interrupted by process exit; terminal status: crashed_and_repaired.";
#[derive(Debug, Clone, Default, PartialEq, Eq)]
+5 -1
View File
@@ -184,7 +184,11 @@ from earlier turns.";
/// Preamble for the one conversation-history checkpoint created by compaction.
/// This intentionally follows Codex's `templates/compact/summary_prefix.md`:
/// the checkpoint is a user-history item, never standing system-prompt prose.
const SUMMARY_HEADER: &str = "Another language model started to solve this problem and produced \
///
/// A history rebuilt from turn records has to recognise the checkpoint
/// messages a document carries, so the preamble itself is crate-visible
/// (`runtime_threads`' recovery projection and its tests).
pub(crate) const SUMMARY_HEADER: &str = "Another language model started to solve this problem and produced \
a summary of its thinking process. You also have access to the state of the tools that were used \
by that language model. Use this to build on the work that has already been done and avoid \
duplicating work. Here is the summary produced by the other language model, use the information \
+100
View File
@@ -825,6 +825,17 @@ fn projected_user_text(message: &Message) -> Option<String> {
if message.role.as_str() != "user" {
return None;
}
// An engine-owned compaction checkpoint is not a turn's prompt. No turn
// record can reproduce one — the compaction ran in the engine, between
// records — so a document that carries it would read as drifted from the
// records it was rebuilt from. Every fork of a compacted source inherits
// the summary in its system prompt, and the engine installs the checkpoint
// into the synced history on the fork's first load, so leaving it in took
// the cut away from the fork itself: `/undo`, retry and fork-again each
// refused with "cannot identify an exact saved-history boundary".
if crate::compaction::is_wire_compaction_checkpoint_message(message) {
return None;
}
let text = message
.content
.iter()
@@ -1001,6 +1012,53 @@ fn saved_transcript_is_compacted(
})
}
/// The result a rebuilt history must give a tool call whose outcome the turn
/// store never recorded, or `None` when nothing is missing.
///
/// A call that failed, was interrupted, or was canceled can be persisted as a
/// single `tool_call` item that carries the call alone: the failure text lives
/// on the item's own `detail`, and no `tool_result_for` item follows. Rebuilding
/// only the call leaves it unanswered, which a provider rejects outright —
/// `No tool output found for tool call …`. The missing result is also what
/// stopped the first of two responses from flushing, so the rebuild glued them
/// into one assistant message (`thinking`, `call`, `thinking`, `call`) and the
/// provider read *that* as the first call never being answered, even after the
/// request-time repair had answered it.
///
/// Both are cured by answering the call where its own outcome is known: the
/// item's failure text for a failure, and the repository's interrupted-call
/// notice for a call the process never finished.
///
/// An answer is only ever paired with a call the rebuild actually emits, so the
/// identity required here is the one the caller needs before it emits the call
/// at all (#5823): a snapshot with no tool name is skipped rather than replayed
/// as an empty shell, and answering it would leave the result on its own.
///
/// A call still running is deliberately left alone. A rebuild of a live turn
/// must not invent an outcome the engine is still waiting for — the result is
/// coming — so this answers only the terminal failures.
fn unanswered_call_result(
item: &TurnItemRecord,
call_id: &str,
call_name: &str,
recorded_results: &HashSet<String>,
) -> Option<String> {
if call_id.is_empty() || call_name.is_empty() || recorded_results.contains(call_id) {
return None;
}
match item.status {
TurnItemLifecycleStatus::Failed => {
Some(item.detail.clone().unwrap_or_else(|| item.summary.clone()))
}
TurnItemLifecycleStatus::Interrupted | TurnItemLifecycleStatus::Canceled => {
Some(crate::tool_history_repair::CRASH_REPAIR_CONTENT.to_string())
}
TurnItemLifecycleStatus::Queued
| TurnItemLifecycleStatus::InProgress
| TurnItemLifecycleStatus::Completed => None,
}
}
fn thread_execution_state_matches(left: &ThreadRecord, right: &ThreadRecord) -> bool {
left.schema_version == right.schema_version
&& left.id == right.id
@@ -12575,6 +12633,26 @@ impl RuntimeThreadManager {
ordered
};
// The calls this turn already answers with a result item of their
// own. Only a `tool_call` item is rebuilt into a result — a
// `file_change` item records its own call and result, and the
// rebuild reads neither — so this set names exactly the answers a
// rebuild will emit, and a call in it must not be answered twice.
// A call item may carry its result (the completed live shape) or a
// sibling `tool_result_for` item may (seeded history, and the
// repair's own notice); both are `tool_call` items.
let recorded_results: HashSet<String> = items
.iter()
.filter(|item| item.kind == TurnItemKind::ToolCall)
.filter_map(|item| {
item.metadata
.as_ref()?
.get("tool_result_for")?
.as_str()
.map(str::to_string)
})
.collect();
let mut assistant_blocks: Vec<ContentBlock> = Vec::new();
let mut user_blocks: Vec<ContentBlock> = Vec::new();
let flush_assistant = |blocks: &mut Vec<ContentBlock>, msgs: &mut Vec<Message>| {
@@ -12642,6 +12720,17 @@ impl RuntimeThreadManager {
let tool_use_id = meta_str("tool_use_id");
let tool_name = meta_str("tool_name");
let tool_result_for = meta_str("tool_result_for");
// A call whose own outcome the store never recorded is
// answered with the outcome it does hold, decided while
// the id is still borrowed — see
// `unanswered_call_result`.
let unanswered = unanswered_call_result(
&item,
&tool_use_id,
&tool_name,
&recorded_results,
)
.map(|content| (tool_use_id.clone(), content));
// Completed live turns persist the call and its result
// on one item; seeded history persists them as two.
// Both shapes must rebuild the paired tool_call /
@@ -12684,6 +12773,17 @@ impl RuntimeThreadManager {
is_error: if is_error { Some(true) } else { None },
content_blocks,
});
} else if let Some((call_id, content)) = unanswered {
// The call above had no recorded outcome: pair it
// here, in the position the live transcript held
// its result, so the rebuilt turn is complete.
flush_assistant(&mut assistant_blocks, &mut messages);
user_blocks.push(ContentBlock::ToolResult {
tool_use_id: call_id,
content,
is_error: Some(true),
content_blocks: None,
});
}
}
_ => {}
+433
View File
@@ -1054,6 +1054,304 @@ mod recovery {
Ok(())
}
/// A tool call whose outcome never reached the turn store — the tool
/// failed, so the item carries the failure text and no result item follows
/// — used to rebuild as a call with no answer. Two things then went wrong
/// on a fork of that conversation: the response *after* the lost call was
/// glued into the same assistant message (no result had flushed it), and
/// the request-time repair answered the call by inserting a result next to
/// a message that now held two responses. DeepSeek's Responses API reads
/// that history as `400 No tool output found for tool call …` on every
/// turn the fork ran, and the repaired document no longer matched the
/// records, so retrying the fork refused with "cannot identify an exact
/// saved-history boundary".
#[tokio::test]
async fn a_rebuilt_turn_answers_a_tool_call_whose_outcome_the_store_lost() -> Result<()> {
let _env = crate::test_support::lock_test_env();
let dir = tempfile::tempdir()?;
let _home = crate::test_support::EnvVarGuard::set("CODEWHALE_HOME", dir.path());
let manager = RuntimeThreadManager::open(
config(),
dir.path().to_path_buf(),
test_manager_config(dir.path().join("runtime")),
)?;
let thread = manager
.create_thread(CreateThreadRequest::default())
.await?;
let sessions = crate::session_manager::SessionManager::new(
crate::session_manager::default_sessions_dir()?,
)?;
let base = Utc::now();
let at = |seconds: i64| Some(base + chrono::Duration::seconds(seconds));
let text_item = |id: &str, turn: &str, order: i64, kind: TurnItemKind, text: &str| {
let mut item = sample_item(turn, id, TurnItemLifecycleStatus::Completed);
item.kind = kind;
item.started_at = at(order);
item.ended_at = item.started_at;
item.summary = text.to_string();
item.detail = Some(text.to_string());
item
};
// Two responses in one turn: the first asked for a build that the tool
// reported as failed without recording a result, the second answered
// and ran the tests, which did.
let turn_lost = "turn_lost_the_call";
let mut lost = sample_item(turn_lost, "item_call_lost", TurnItemLifecycleStatus::Failed);
lost.kind = TurnItemKind::ToolCall;
lost.started_at = at(2);
lost.ended_at = lost.started_at;
lost.summary = "bash failed: Failed to execute tool: boom".to_string();
lost.detail = Some("Failed to execute tool: boom".to_string());
lost.metadata = Some(json!({
"tool_use_id": "call_lost",
"tool_name": "bash",
"tool_input": "{\"command\":\"cargo build\"}",
}));
let mut kept = sample_item(
turn_lost,
"item_call_kept",
TurnItemLifecycleStatus::Completed,
);
kept.kind = TurnItemKind::ToolCall;
kept.started_at = at(5);
kept.ended_at = kept.started_at;
kept.summary = "bash: 12 passed".to_string();
kept.detail = Some("12 passed".to_string());
kept.metadata = Some(json!({
"tool_use_id": "call_kept",
"tool_result_for": "call_kept",
"tool_name": "bash",
"tool_input": "{\"command\":\"cargo test\"}",
}));
let turn_after = "turn_of_compaction";
for record in [
text_item(
"item_u1",
turn_lost,
0,
TurnItemKind::UserMessage,
"run the build",
),
text_item(
"item_r1",
turn_lost,
1,
TurnItemKind::AgentReasoning,
"the compiler will tell us what is missing",
),
lost,
text_item(
"item_s1",
turn_lost,
3,
TurnItemKind::Status,
"Continuing — tool results",
),
text_item(
"item_r2",
turn_lost,
4,
TurnItemKind::AgentReasoning,
"the build failed; the tests still say something",
),
kept,
text_item(
"item_a1",
turn_lost,
6,
TurnItemKind::AgentMessage,
"the build failed, the tests passed",
),
text_item(
"item_u2",
turn_after,
7,
TurnItemKind::UserMessage,
"and now the next thing",
),
text_item(
"item_c1",
turn_after,
8,
TurnItemKind::ContextCompaction,
"Made room: 40 → 21 messages",
),
] {
manager.store.save_item(&record)?;
}
for (turn_id, order) in [(turn_lost, 0), (turn_after, 10)] {
let mut turn = sample_turn(&thread.id, turn_id, RuntimeTurnStatus::Completed);
turn.started_at = at(order);
turn.ended_at = turn.started_at;
manager.store.save_turn(&turn)?;
}
let mut stored = manager.get_thread(&thread.id).await?;
stored.latest_turn_id = Some(turn_after.to_string());
manager.store.save_thread(&stored)?;
// The engine's transcript for that conversation once compaction ran:
// the prompts verbatim, nothing that happened between them. A cut has
// to rebuild the exchange from the records — the path this covers.
let compacted: Vec<Message> = serde_json::from_value(json!([
{"role":"user","content":[{"type":"text","text":"run the build"}]},
{"role":"user","content":[{"type":"text","text":"and now the next thing"}]}
]))?;
let saved = crate::session_manager::create_saved_session_with_id_and_mode(
Uuid::new_v4().to_string(),
&compacted,
&thread.model,
dir.path(),
0,
None,
Some("agent"),
);
{
let _admission = manager.session_checkpoint_guard().await;
sessions.save_session(&saved)?;
manager
.set_thread_session_checkpoint(&thread.id, &saved)
.await?;
}
let (fork, _, _, _) = manager.fork_at_user_turn(&thread.id, turn_lost).await?;
let fork_session_id = fork
.session_id
.clone()
.context("a published fork owns a session document")?;
let document = sessions.load_session(&fork_session_id)?;
let shape = |messages: &[Message]| -> Vec<(String, Vec<&'static str>)> {
messages
.iter()
.map(|message| {
let blocks = message
.content
.iter()
.map(|block| match block {
ContentBlock::Text { .. } => "text",
ContentBlock::Thinking { .. } => "thinking",
ContentBlock::ToolUse { .. } => "tool_use",
ContentBlock::ToolResult { .. } => "tool_result",
_ => "other",
})
.collect();
(message.role.as_str().to_string(), blocks)
})
.collect()
};
assert_eq!(
shape(&document.messages),
vec![
("user".to_string(), vec!["text"]),
("assistant".to_string(), vec!["thinking", "tool_use"]),
("user".to_string(), vec!["tool_result"]),
("assistant".to_string(), vec!["thinking", "tool_use"]),
("user".to_string(), vec!["tool_result"]),
("assistant".to_string(), vec!["text"]),
],
"each response keeps its own call and its own answer: {:#?}",
document.messages
);
let calls: Vec<&str> = document
.messages
.iter()
.flat_map(|message| message.content.iter())
.filter_map(|block| match block {
ContentBlock::ToolUse { id, .. } => Some(id.as_str()),
_ => None,
})
.collect();
let answers: Vec<(&str, &str)> = document
.messages
.iter()
.flat_map(|message| message.content.iter())
.filter_map(|block| match block {
ContentBlock::ToolResult {
tool_use_id,
content,
..
} => Some((tool_use_id.as_str(), content.as_str())),
_ => None,
})
.collect();
assert_eq!(calls, ["call_lost", "call_kept"]);
assert_eq!(
answers
.iter()
.map(|(tool_use_id, _)| *tool_use_id)
.collect::<Vec<_>>(),
["call_lost", "call_kept"],
"no call is left unanswered: {:#?}",
document.messages
);
assert!(
answers[0].1.contains("Failed to execute tool: boom"),
"the lost call keeps the failure the turn recorded: {:#?}",
answers
);
// Now the fork's own life: the engine loads it with the source's
// compaction summary as its system prompt, so the synced history grows
// the checkpoint message *before* the fork's first turn, and two turns
// follow it. Retrying the fork — a depth-relative cut of its last turn
// — walks a document whose middle holds a message no record produced.
let mut extended = document.messages.clone();
extended.push(crate::compaction::compaction_checkpoint_message(
&SystemPrompt::Text(format!(
"{} the work so far",
crate::compaction::SUMMARY_HEADER
)),
));
let first_prompt = "and now?";
let second_prompt = "keep going";
let tail: Vec<Message> = serde_json::from_value(json!([
{"role":"user","content":[{"type":"text","text":first_prompt}]},
{"role":"user","content":[{"type":"text","text":second_prompt}]}
]))?;
extended.extend(tail.clone());
let mut rebound = sessions.load_session(&fork_session_id)?;
rebound.messages = extended;
{
let _admission = manager.session_checkpoint_guard().await;
sessions.save_session(&rebound)?;
manager.seed_thread_from_messages(&fork.id, &tail).await?;
manager
.set_thread_session_checkpoint(&fork.id, &rebound)
.await?;
}
let (undo, _, _, _) = manager.fork_at_user_message(&fork.id, 0).await?;
let undo_document = sessions.load_session(
undo.session_id
.as_deref()
.context("a published fork owns a session document")?,
)?;
assert_eq!(
projected_user_texts(&undo_document.messages),
["run the build".to_string(), first_prompt.to_string()],
"the cut keeps the exchange and the first post-fork turn, and reads \
past the engine's checkpoint: {:#?}",
undo_document.messages
);
assert_eq!(
undo_document
.messages
.iter()
.flat_map(|message| message.content.iter())
.filter_map(|block| match block {
ContentBlock::ToolUse { id, .. } => Some(id.as_str()),
_ => None,
})
.collect::<Vec<_>>(),
["call_lost", "call_kept"],
"the successor keeps the answered calls whole: {:#?}",
undo_document.messages
);
Ok(())
}
async fn control_case(interrupt: bool, follow_up: bool) -> Result<()> {
let _env = crate::test_support::lock_test_env();
let dir = tempfile::tempdir()?;
@@ -19229,6 +19527,141 @@ fn saved_history_boundary_refuses_until_every_kept_prompt_is_seen() {
);
}
/// The engine installs a compaction checkpoint into a synced history, and no
/// turn record reproduces one: the compaction ran in the engine, between
/// records. A fork of a compacted source keeps the summary in its system
/// prompt, so the engine writes that checkpoint into the fork's own document
/// on its first load — and `/undo`, retry and fork-again then run these walks
/// over it. It is the engine's message, not a turn's prompt, so it must not
/// read as a kept prompt the records never saw.
#[test]
fn a_compaction_checkpoint_is_not_a_kept_prompt() {
let user = |text: &str| Message {
role: Role::User,
content: vec![ContentBlock::Text {
text: text.to_string(),
cache_control: None,
}],
};
let checkpoint = crate::compaction::compaction_checkpoint_message(&SystemPrompt::Text(
format!("{} the work so far", crate::compaction::SUMMARY_HEADER),
));
assert!(
crate::compaction::is_wire_compaction_checkpoint_message(&checkpoint),
"the fixture is the engine's own checkpoint message"
);
let messages = vec![user("first"), checkpoint, user("undo me")];
assert_eq!(
saved_history_boundary(&messages, &["first".to_string()], "undo me"),
Some(2)
);
assert_eq!(
exact_prefix_boundary(
&messages,
&session_recovery_projection(&[user("first"), user("undo me")])
),
Some(3)
);
}
/// The decision a rebuild makes about a call the turn store left unanswered:
/// which outcome the item itself holds, and which calls it must leave alone.
#[test]
fn a_lost_call_is_answered_only_where_its_own_outcome_is_known() {
let item = |status: TurnItemLifecycleStatus, detail: Option<&str>| {
let mut item = sample_item("turn_lost", "item_call", status);
item.kind = TurnItemKind::ToolCall;
item.summary = "bash failed: no output".to_string();
item.detail = detail.map(str::to_string);
item
};
let none = HashSet::new();
// A failure keeps the text the tool reported, and falls back to the
// item's summary when the record holds no detail of its own.
assert_eq!(
unanswered_call_result(
&item(
TurnItemLifecycleStatus::Failed,
Some("Failed to execute tool: boom")
),
"call-1",
"bash",
&none
),
Some("Failed to execute tool: boom".to_string())
);
assert_eq!(
unanswered_call_result(
&item(TurnItemLifecycleStatus::Failed, None),
"call-1",
"bash",
&none
),
Some("bash failed: no output".to_string())
);
// A call the process never finished says so in the repository's one
// spelling, whichever way it was left behind.
for status in [
TurnItemLifecycleStatus::Interrupted,
TurnItemLifecycleStatus::Canceled,
] {
assert_eq!(
unanswered_call_result(&item(status, None), "call-1", "bash", &none),
Some(crate::tool_history_repair::CRASH_REPAIR_CONTENT.to_string()),
"{status:?} is terminal: no result is coming"
);
}
// The engine is still waiting for a running call; a completed one already
// has its result; a queued one was never delivered. A snapshot without a
// call identity is not rebuilt at all, so answering it would leave the
// result on its own.
for status in [
TurnItemLifecycleStatus::InProgress,
TurnItemLifecycleStatus::Queued,
TurnItemLifecycleStatus::Completed,
] {
assert_eq!(
unanswered_call_result(&item(status, None), "call-1", "bash", &none),
None,
"{status:?} is not a lost outcome"
);
}
assert_eq!(
unanswered_call_result(
&item(TurnItemLifecycleStatus::Failed, None),
"call-1",
"",
&none
),
None,
"a call the rebuild would not emit is not answered"
);
assert_eq!(
unanswered_call_result(
&item(TurnItemLifecycleStatus::Failed, None),
"",
"bash",
&none
),
None
);
// A result the turn records on its own item is that call's answer.
let recorded: HashSet<String> = ["call-1".to_string()].into_iter().collect();
assert_eq!(
unanswered_call_result(
&item(TurnItemLifecycleStatus::Failed, None),
"call-1",
"bash",
&recorded
),
None
);
}
/// #6522 review: `/resume`, `/load` and launch warm the canonical sessions
/// root on a blocking thread, so the confinement predicate that follows on
/// the UI runtime is served from the cache instead of resolving a path.