serve: keep retained K/V through the cache capacity gate

park_current dropped the retained K/V whenever can_fit failed and fell
back to a full snapshot capture. The gate duplicates what make_room
decides immediately after, with harsher arithmetic: the slot term fires
when make_room enforces the same cap, and the byte term compares the
with-reuse estimate against headroom that ignores the parked image
replacing the retained buffers. make_room already implements the
stated policy - pressure drops the optional retained storage before
evicting any parked conversation (the priority the cache tests pin).
With a full cache the fallback buys nothing: the fresh estimate forces
the same or more eviction, plus a whole-snapshot recapture instead of
the growth - hundreds of ms per park at ~100K tokens where one long
conversation round-trips while short-lived conversations keep the
cache full.

The with-reuse estimate stays uncapped: it counts the retained buffers'
capacity and directories, and put() charges that same true size, so a
capped figure would under-evict and overfill the budget.
This commit is contained in:
Developer authored and Niko1221 committed 2026-10-01 08:15:27 +02:00
1 parent b0711943b2
commit 6cd29d9850
1 file changed
+6 -5
+6 -5
View File
@@ -4006,11 +4006,12 @@ int main(int argc, char** argv) {
estimate = fresh_estimate;
err.clear();
}
if (!reuse.kv.empty() && !conversations.can_fit(estimate, held)) {
// Optional growth capacity must not evict useful conversations.
reuse = {};
estimate = fresh_estimate;
}
// A park carrying its own retained K/V replaces memory the cache
// already held, so capacity is make_room's call - it runs next either
// way, and put()'s accounting still bounds the budget. The with-reuse
// estimate must stay uncapped: it counts the retained buffers'
// capacity and directories, and put() charges that same true size -
// a capped figure would under-evict and overfill the budget.
if (!conversations.make_room(estimate, held)) {
std::fprintf(stderr, "strata serve: conversation cache: skip parking (snapshot %zu MiB exceeds available budget)\n",
estimate >> 20);