fix(session): size a resumed session the way the handoff carries it

`codewhale resume` and `codewhale fork` refused any saved document whose
file was over the 8 MiB full-history bound once off-branch history was
archived. That counted the conversation twice: journal entries hold full
messages, and `messages` is a copy rebuilt from the journal on every save
and replaced from it on every load. A conversation could use only about
40% of the bound.

SavedSession::transport_form is now the one definition of what a canonical
handoff carries: the document without its journal-derived `messages` copy
(kept when it differs from the journal). The mounted handoff sends that
form, the receiving owner rebuilds the copy with ensure_journal before it
compares digests, and saved_document_digest sizes that form while its hash
formula, and so every stored digest, is unchanged.

compact_oversized_session measures the whole and the carried compact size
of the projection that archives every off-branch entry (and, only when the
whole document is still over, prunes ended non-durable operations beyond
256). When the whole projection fits it is written as before; when the
carried form already fits as loaded nothing is written; when the projected
carried form fits the projection is written; otherwise nothing is written
and the refusal gives the carried size. A file still over the bound is
read whole and accepted only when its carried form fits; the capped read
stays for files within it, and the post-compaction file-length refusal is
gone.

The in-process mutation request is bounded at the history bound plus a 64
KiB envelope allowance (CANONICAL_REQUEST_ENVELOPE_BYTES), so a carried form
just under 8 MiB is not refused for the key, paths and digests around it.

Known limits: a file left over 8 MiB is parsed whole on every bounded read
(about five per resume). GET /v1/threads/{id}/history, the app-server
bridge, native fork and import merge still apply the whole-document bound;
they can see the archive write and still refuse. A session
whose current conversation and Work state alone exceed 8 MiB is still
refused, untouched. The on-disk format is unchanged: documents still store
`messages`.

Verification: checked once on the final batch tree (parent db89057d84):
cargo fmt --check clean; tui lib nextest across route_budget,
provider_catalog_live, provider_lake, mounted_history_tests,
session_manager, client, core::engine, compaction, tools::subagent,
runtime_api, purge, commands::contract, tui::model_picker and whats_new
2757 passed, 0 failed; tui integration exec_turn_usage 11 passed, 0
failed; feature_registry 2 passed, 0 failed; clippy -p codewhale-tui
--lib --tests -D warnings with the CI allow-list clean; check-blocking-
calls-budget 540 sites, within budget. Not run: hosted CI and the
loopback product captures (those follow on the release build).

Refs #6955
Thanks @7jrxt42BxFZo4iAnN4CX for the measurements across 467 sessions.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

CHANGELOG.md

@@ -98,13 +98,16 @@ guidance.
   wire, including `/cache warmup`. When a level still cannot be sent, the
   error names it and lists the levels the route publishes. Refs #6396.
 
-- An oversized saved session is measured before anything is written. If
-  archiving its off-branch history, and if needed pruning finished
-  non-durable operations beyond the newest 256, would still leave it over
-  the 8 MiB full-history limit, the session file and its archive are left
-  untouched and the refusal gives the projected size and the steps it
-  counted, instead of rewriting the file and then reporting that nothing
-  changed. Refs #6955. Thanks @7jrxt42BxFZo4iAnN4CX.
+- `codewhale resume` and `codewhale fork` now size a saved session the way
+  it is handed over: compact JSON, with the conversation counted once
+  instead of also as its derived message copy. A session that is over 8 MiB
+  on disk only because of that copy, indentation, abandoned branches or
+  finished shell operations opens again. A session whose current
+  conversation and Work state alone exceed 8 MiB is still refused before
+  anything is written: its file and archive are left unchanged and the
+  refusal gives the size it would hand over. Reading the history through the
+  Runtime API or the desktop app still applies the 8 MiB limit to the whole
+  document. Refs #6955. Thanks @7jrxt42BxFZo4iAnN4CX.
 
 - Saved sessions and crash checkpoints are now written as compact JSON
   instead of indented JSON. Files are smaller (16–27% on the two measured
@@ -1326,8 +1329,8 @@ guidance.
   holds a raw sub-agent handoff (#6954).
 - A saved session over 8 MiB can be resumed and forked again. Journal entries
   off the current branch are archived instead of refusing the session (#6955).
-  A session whose current branch alone is over 8 MiB is still refused, and the
-  refusal now gives its size.
+  A session whose current conversation and Work state, counted once, are still
+  over 8 MiB is refused, and the refusal gives the size it would hand over.
 - Snapshot restore and `/undo` work with older git, such as the 2.43 that
   Ubuntu 24.04 ships. They failed there with "only one reference expected".
 - Cancelling your own background shell job no longer reports it as finished

crates/protocol/src/lib.rs

@@ -182,8 +182,9 @@ pub enum CanonicalHistorySource {
         runtime_thread_id: String,
         expected_document_digest: String,
     },
-    /// Local mounted handoff supplies the already protected complete document;
-    /// the actual owner reopens it and compares the same whole-document digest.
+    /// Local mounted handoff supplies the already protected document without
+    /// its journal-derived messages; the actual owner rebuilds them from the
+    /// journal, reopens the file and compares the same whole-document digest.
     SavedSession {
         session: Value,
         expected_document_digest: String,

crates/tui/CHANGELOG.md

@@ -98,13 +98,16 @@ guidance.
   wire, including `/cache warmup`. When a level still cannot be sent, the
   error names it and lists the levels the route publishes. Refs #6396.
 
-- An oversized saved session is measured before anything is written. If
-  archiving its off-branch history, and if needed pruning finished
-  non-durable operations beyond the newest 256, would still leave it over
-  the 8 MiB full-history limit, the session file and its archive are left
-  untouched and the refusal gives the projected size and the steps it
-  counted, instead of rewriting the file and then reporting that nothing
-  changed. Refs #6955. Thanks @7jrxt42BxFZo4iAnN4CX.
+- `codewhale resume` and `codewhale fork` now size a saved session the way
+  it is handed over: compact JSON, with the conversation counted once
+  instead of also as its derived message copy. A session that is over 8 MiB
+  on disk only because of that copy, indentation, abandoned branches or
+  finished shell operations opens again. A session whose current
+  conversation and Work state alone exceed 8 MiB is still refused before
+  anything is written: its file and archive are left unchanged and the
+  refusal gives the size it would hand over. Reading the history through the
+  Runtime API or the desktop app still applies the 8 MiB limit to the whole
+  document. Refs #6955. Thanks @7jrxt42BxFZo4iAnN4CX.
 
 - Saved sessions and crash checkpoints are now written as compact JSON
   instead of indented JSON. Files are smaller (16–27% on the two measured
@@ -1326,8 +1329,8 @@ guidance.
   holds a raw sub-agent handoff (#6954).
 - A saved session over 8 MiB can be resumed and forked again. Journal entries
   off the current branch are archived instead of refusing the session (#6955).
-  A session whose current branch alone is over 8 MiB is still refused, and the
-  refusal now gives its size.
+  A session whose current conversation and Work state, counted once, are still
+  over 8 MiB is refused, and the refusal gives the size it would hand over.
 - Snapshot restore and `/undo` work with older git, such as the 2.43 that
   Ubuntu 24.04 ships. They failed there with "only one reference expected".
 - Cancelling your own background shell job no longer reports it as finished

crates/tui/src/lib.rs

@@ -10176,7 +10176,7 @@ async fn prepare_mounted_session(
             }
         } else {
             let source = CanonicalHistorySource::SavedSession {
-                session: serde_json::to_value(&saved)?,
+                session: serde_json::to_value(saved.transport_form())?,
                 expected_document_digest: source_digest,
             };
             let options = CanonicalHistoryOptions {

crates/tui/src/runtime_api/thread_history.rs

@@ -736,9 +736,11 @@ fn stable_projection_entries(
     Ok(())
 }
 
+const CANONICAL_REQUEST_ENVELOPE_BYTES: usize = 64 * 1024;
+
 pub(crate) fn saved_document_digest(session: &SavedSession) -> Result<String> {
     let mut size = HistorySizeBound(MAX_CANONICAL_HISTORY_BYTES);
-    serde_json::to_writer(&mut size, session)?;
+    serde_json::to_writer(&mut size, &session.transport_form())?;
     Ok(crate::hashing::sha256_hex(
         crate::client::canonical_json(&serde_json::to_value(session)?).as_bytes(),
     ))
@@ -782,7 +784,7 @@ pub(crate) async fn mutate_thread_history_in_runtime(
             && !request.operation_key.chars().any(char::is_control),
         "invalid canonical operation key"
     );
-    let mut size = HistorySizeBound(MAX_CANONICAL_HISTORY_BYTES);
+    let mut size = HistorySizeBound(MAX_CANONICAL_HISTORY_BYTES + CANONICAL_REQUEST_ENVELOPE_BYTES);
     serde_json::to_writer(&mut size, &request)?;
     let admission = runtime.try_history_import_guard()?;
     let runtime = Arc::clone(runtime);
@@ -1177,11 +1179,12 @@ async fn mutate_in_owner(
                 session,
                 expected_document_digest,
             } => {
-                let proposed: SavedSession = serde_json::from_value(session.clone())?;
+                let mut proposed: SavedSession = serde_json::from_value(session.clone())?;
                 ensure!(
                     serde_json::to_value(&proposed)? == *session,
                     "saved history contains unsupported fields; source retained"
                 );
+                proposed.ensure_journal();
                 let id = proposed.metadata.id.clone();
                 let dir = sessions_dir.clone();
                 let expected = expected_document_digest.clone();

crates/tui/src/session_manager.rs

@@ -1203,6 +1203,18 @@ impl SavedSession {
         }
     }
 
+    pub(crate) fn transport_form(&self) -> Self {
+        let mut carried = self.clone();
+        if self
+            .journal
+            .as_ref()
+            .is_some_and(|journal| journal.to_messages() == self.messages)
+        {
+            carried.compact_for_persistence_queue();
+        }
+        carried
+    }
+
     /// Bring `messages` and the journal into the shape the on-disk schema
     /// expects, in place.
     ///
@@ -2631,19 +2643,30 @@ impl SessionManager {
         self.save_session_inner(session, JournalPrune::Excess)
     }
 
-    /// Bring a saved document that exceeds the full-history transport bound
-    /// back under it by archiving every off-branch journal entry (kept in the
-    /// journal archive sidecar, the same durable step the autosave prune
-    /// uses). Only when that is not enough, ended non-durable unpinned Work
-    /// Graph operations beyond the newest `ENDED_OPERATION_CAP` are dropped,
-    /// the bound a live session applies at its next operation. The active
-    /// branch and durable or pinned operations are never touched. The
-    /// compacted document is measured before anything is written: when it
-    /// would still exceed `limit`, nothing is archived or rewritten, and its
-    /// projected size is returned with whether that projection pruned ended
-    /// operations.
+    /// Make a saved document whose file exceeds the full-history transport
+    /// bound readable under it. The bound counts what a canonical handoff
+    /// carries: compact JSON without the journal-derived `messages` copy
+    /// ([`SavedSession::transport_form`]). Every off-branch journal entry is
+    /// projected into the journal archive sidecar (the same durable step the
+    /// autosave prune uses) and, only when the whole projected document is
+    /// still over `limit`, ended non-durable unpinned Work Graph operations
+    /// beyond the newest `ENDED_OPERATION_CAP` are dropped too. Then, in
+    /// order: when the whole projected document fits, it is written; when the
+    /// carried form already fits as loaded, nothing is written; when the
+    /// projected carried form fits, the projection is written; otherwise
+    /// nothing is archived or rewritten and the projected carried size is
+    /// returned with whether that projection pruned ended operations. The
+    /// active branch and durable or pinned operations are never touched.
+    /// Known limits: a file left over `limit` is parsed whole on every bounded
+    /// read, and readers that bound the whole document (Runtime API history,
+    /// native fork, import merge) can see that write and still refuse.
     fn compact_oversized_session(&self, id: &str, limit: u64) -> io::Result<Option<(u64, bool)>> {
-        let project = |session: &SavedSession| -> io::Result<u64> {
+        let size = |session: &SavedSession| -> io::Result<u64> {
+            serde_json::to_vec(session)
+                .map(|bytes| bytes.len() as u64)
+                .map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))
+        };
+        let project = |session: &SavedSession| -> io::Result<(u64, u64)> {
             let mut projected = session.clone();
             if let Some(journal) = projected.journal.as_mut() {
                 let off_branch: std::collections::HashSet<String> =
@@ -2655,13 +2678,18 @@ impl SessionManager {
             }
             self.hydrate_recovered_runtime_binding(&mut projected)?;
             self.hydrate_approval_receipts(&mut projected)?;
-            Ok(serialize_saved_session(projected)?.len() as u64)
+            if let Some(journal) = projected.journal.as_ref() {
+                validate_saved_journal(journal)?;
+            }
+            projected.make_storage_compatible();
+            Ok((size(&projected)?, size(&projected.transport_form())?))
         };
         let mut session = self.load_session_snapshot_with_limit(id, None)?;
-        let mut projected = project(&session)?;
+        let loaded = size(&session.transport_form())?;
+        let (mut whole, mut carried) = project(&session)?;
         let mut pruned_operations = false;
         let session_id = session.metadata.id.clone();
-        if projected > limit
+        if whole > limit
             && let Some(graph) = session
                 .work_state
                 .as_mut()
@@ -2683,7 +2711,7 @@ impl SessionManager {
                 Ok((next, _)) => {
                     *graph = next;
                     pruned_operations = true;
-                    projected = project(&session)?;
+                    (whole, carried) = project(&session)?;
                 }
                 Err(report) => tracing::warn!(
                     %session_id,
@@ -2692,8 +2720,11 @@ impl SessionManager {
                 ),
             }
         }
-        if projected > limit {
-            return Ok(Some((projected, pruned_operations)));
+        if whole > limit && loaded <= limit {
+            return Ok(None);
+        }
+        if whole > limit && carried > limit {
+            return Ok(Some((carried, pruned_operations)));
         }
         self.save_session_inner(session, JournalPrune::AllOffBranch)?;
         Ok(None)
@@ -3497,7 +3528,9 @@ impl SessionManager {
     }
 
     /// The same protected session parser with an explicit transport byte ceiling.
-    /// The file read itself is capped before allocating or parsing its document.
+    /// A file within the ceiling is read capped before allocating or parsing its
+    /// document; a larger one is parsed whole and accepted only when its
+    /// carried form ([`SavedSession::transport_form`]) is within the ceiling.
     pub(crate) fn load_session_snapshot_bounded(
         &self,
         id: &str,
@@ -3513,6 +3546,7 @@ impl SessionManager {
     ) -> io::Result<SavedSession> {
         use std::io::Read;
         let path = self.validated_session_path(id)?;
+        let mut oversized = false;
         let content = match limit {
             None => {
                 let mut content = String::new();
@@ -3543,7 +3577,7 @@ impl SessionManager {
                         return Err(io::Error::new(
                             io::ErrorKind::InvalidData,
                             format!(
-                                "session is {:.1} MiB and the full-history transport limit is {:.1} MiB; {compaction} would leave it at {:.1} MiB, so nothing was changed",
+                                "session is {:.1} MiB and the full-history transport limit is {:.1} MiB; {compaction} would still hand over {:.1} MiB, counting the conversation once, so nothing was changed",
                                 mib(before),
                                 mib(limit as u64),
                                 mib(projected),
@@ -3551,24 +3585,16 @@ impl SessionManager {
                         ));
                     }
                     file = open_private_read_file(&path)?;
-                    let after = file.metadata()?.len();
-                    if after > limit as u64 {
-                        return Err(io::Error::new(
-                            io::ErrorKind::InvalidData,
-                            format!(
-                                "session was {:.1} MiB; compaction rewrote it to {:.1} MiB, still over the {:.1} MiB full-history transport limit; its off-branch journal entries are in {}",
-                                mib(before),
-                                mib(after),
-                                mib(limit as u64),
-                                journal_archive_path(&self.sessions_dir, id)?.display(),
-                            ),
-                        ));
-                    }
+                    oversized = file.metadata()?.len() > limit as u64;
                 }
                 let mut bytes = Vec::new();
-                file.take((limit as u64).saturating_add(1))
-                    .read_to_end(&mut bytes)?;
-                if bytes.len() > limit {
+                if oversized {
+                    file.read_to_end(&mut bytes)?;
+                } else {
+                    file.take((limit as u64).saturating_add(1))
+                        .read_to_end(&mut bytes)?;
+                }
+                if !oversized && bytes.len() > limit {
                     return Err(io::Error::new(
                         io::ErrorKind::InvalidData,
                         "session grew beyond full-history transport bound; source retained",
@@ -3610,6 +3636,17 @@ impl SessionManager {
         session.ensure_journal();
         self.hydrate_approval_receipts(&mut session)?;
         self.apply_late_usage_to_metadata(&mut session.metadata);
+        if let Some(limit) = limit.filter(|_| oversized)
+            && serde_json::to_vec(&session.transport_form())
+                .map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))?
+                .len()
+                > limit
+        {
+            return Err(io::Error::new(
+                io::ErrorKind::InvalidData,
+                "session grew beyond full-history transport bound; source retained",
+            ));
+        }
 
         Ok(session)
     }