Jakub Luwierski 5f8ad8284e serve: the conversation cache keeps its shared prefix; the rest rotates LRU ...

The retained checkpoints are always a prefix chain - the request loop drops
any checkpoint that is not a prefix of the current prompt, because its
positional cells are gone (the KV cache is one arena, one branch of history
at a time).  A radix cache's tree collapsed onto the one branch the session
can hold, and the chain's root is the node a radix cache keeps while its
leaves rotate: in practice the end of the system prompt, which every new
chat of the same client mounts through.

First-in-first-out kept that node only by accident of being first: after
prompt_cache newer checkpoints it was gone, and the next chat re-read the
whole prefix at prefill speed (~30 s per 30K tokens).  Now the root is
pinned while the budget allows root and leaf apart (cap >= 2; below that
the old behaviour stands), the rest rotates by least recent use, and a
mount re-dates the checkpoint it mounted through.

The policy is one pure function (conv_cache.hpp) with its scenarios in
conv_cache_test.cpp; checkpoint_save/restore and the state format are
untouched, so STRATA_CKPT_REREAD's token-for-token guarantee holds.

2026-09-28 16:27:51 +02:00

..

conv_cache.hpp

serve: the conversation cache keeps its shared prefix; the rest rotates LRU

2026-09-28 16:27:51 +02:00

logits_selection.hpp

Strata: Qwen3.8-Flash-Next on a 12-24 GB NVIDIA GPU + 64 GB RAM, one-click install

2026-09-24 18:42:20 +02:00