mirror of
https://github.com/Niko1221/Strata.git
synced 2026-11-07 18:35:04 -10:00
serve: keep retained K/V through the cache capacity gate
park_current dropped the retained K/V whenever can_fit failed and fell back to a full snapshot capture. The gate duplicates what make_room decides immediately after, with harsher arithmetic: the slot term fires when make_room enforces the same cap, and the byte term compares the with-reuse estimate against headroom that ignores the parked image replacing the retained buffers. make_room already implements the stated policy - pressure drops the optional retained storage before evicting any parked conversation (the priority the cache tests pin). With a full cache the fallback buys nothing: the fresh estimate forces the same or more eviction, plus a whole-snapshot recapture instead of the growth - hundreds of ms per park at ~100K tokens where one long conversation round-trips while short-lived conversations keep the cache full. The with-reuse estimate stays uncapped: it counts the retained buffers' capacity and directories, and put() charges that same true size, so a capped figure would under-evict and overfill the budget.
This commit is contained in:
1 parent
b0711943b2
commit
6cd29d9850
1 file changed
+6
-5
@@ -4006,11 +4006,12 @@ int main(int argc, char** argv) {
|
||||
estimate = fresh_estimate;
|
||||
err.clear();
|
||||
}
|
||||
if (!reuse.kv.empty() && !conversations.can_fit(estimate, held)) {
|
||||
// Optional growth capacity must not evict useful conversations.
|
||||
reuse = {};
|
||||
estimate = fresh_estimate;
|
||||
}
|
||||
// A park carrying its own retained K/V replaces memory the cache
|
||||
// already held, so capacity is make_room's call - it runs next either
|
||||
// way, and put()'s accounting still bounds the budget. The with-reuse
|
||||
// estimate must stay uncapped: it counts the retained buffers'
|
||||
// capacity and directories, and put() charges that same true size -
|
||||
// a capped figure would under-evict and overfill the budget.
|
||||
if (!conversations.make_room(estimate, held)) {
|
||||
std::fprintf(stderr, "strata serve: conversation cache: skip parking (snapshot %zu MiB exceeds available budget)\n",
|
||||
estimate >> 20);
|
||||
|
||||
Reference in new issue
Block a user