Problem. #2408 / PR #2409: restoring a large parked conversation on a layer split ended the server with a CUDA out of memory
on a later card. The PR checks the free VRAM of every card against --vram-reserve-mib before a restore. Its own review
points out that a loaded card is always below that reserve, so the check would skip nearly every restore on a split (and
on one GPU). It is not ported.
What a restore allocates. conversation_snapshot_restore writes into the session's existing buffers (pools(st), the GDN
state, the indexer tails); refresh_residency only launches kv_stream_reset or copies into the existing ring. The elastic K/V
growth (kvg_ensure) allocates, but only on one GPU. The one thing a restore can create on a card is the calling thread's
private copy stream for it (conversation_copy.hpp private_stream), which it takes from the pool of finished threads when
there is one (#3350). A failed restore is fatal by contract (the session may be half written). So the need is one stream
and one event, and only for a card whose stream does not exist yet (a snapshot loaded from disk, or a restore on a thread
that never copied on that card).
Change. conv_cache.hpp: kRestoreFirstUseMib (64) and restore_fits(free, stream_ready, n, need): a restore is refused only
for a card that has no copy stream yet and less than 64 MiB free. conversation_copy_stream_ready(device) says whether a
card has one. generate.cpp: before the restore, each card of the run is checked; a snapshot that would not fit goes back to
the cache and the prompt is read again, with a log line. A card that has its stream is never refused, however full.
No sample (a failed cudaMemGetInfo) means the restore goes ahead as it did.
Tests. conv_cache_test: the predicate (cards with a stream are admitted at any free size; the edge at the need; no
sample / no cards / no need). conversation_copy_streams_test: copy_stream_ready before and after the first copy, in the pool
after a thread ends, and the measured cost of a card's first copy stream and event (printed; the test requires under
16 MiB, so 64 MiB leaves a wide margin). conversation_snapshot_test: after the first restore, six more restores of two
images leave the free VRAM where it was.
Not covered. The pure predicate and the generate.cpp code build on z440 (CUDA 13.0); the GPU tests are listed in the stream
notes. No model run: gates/2409-restore-guard.sh (a held-VRAM start on a layer split) shows the skip line and an
unchanged answer on the second turn.
Least recent use keeps the newest checkpoints and every turn adds one, so after a few short turns all 6 sit in the
last 1-2K tokens. An agent client that rewrites its history a few thousand tokens back then resumes from the system
prompt: on 2x CMP 50HX with the pi coding agent a 210,528-token prompt that was the same as the session for 204,349
tokens resumed at 9,525 and read 201,003 again (100 s).
With the flag the victim is the checkpoint whose removal leaves the smallest gap for its distance from the newest, so
the kept points thin out geometrically: after 200 turns of 1,000 tokens they sit 0, 1K, 3K, 8K, 43K and 200K before
the end, and an edit 6K back reads 8K again (least recent use: 200K), 42K back 43K. The same number of checkpoints, no
more RAM; never the root, the newest or a pinned one, the tail rule first, the plain policy without a candidate. The
resume is the usual exact-prefix match.
On the box: an 82,593-token conversation grown turn by turn, a turn ~6 steps back rewritten, resumed at 73,610 tokens
in 10.2 s against 40.0 s read in full; with the expert placement fixed (--adapt-swaps 0, --pcie-frac 0.17) the
greedy answer was the same text as the full read. conv_cache_test: the spread cases (a 200-turn simulation against
least recent use, the fallbacks). docs/DETAILS.md: a paragraph after the prompt cache's options.
The engine's GEN line takes pin=N: the first N prompt tokens are a shared read-only prefix (a long document many
short questions follow). The prompt is read in two parts at N and the checkpoint there is pinned: retention never
evicts it, a parked conversation holding it stays parked, the session file keeps it, and a later request that resumes
from it does not park the branch it leaves, so N queries keep one prefix. One pin at a time. Without pin= nothing
changes.
The server takes "strata_prefix" in a chat, messages or responses request: {"messages": K}, {"message": I,
"chars": N} or {"tokens": N}, resolved to the longest common start of the prompt's ids. A prefix that cannot be
marked is said in the log and ignored; a malformed field is a 400.
Tests: conv_cache_test (pinned eviction), conversation_cache_test (a pinned parked conversation is not evicted),
conversation_file_test (the pinned checkpoint is the one saved), serve.test_server.SharedPrefix.
The retained checkpoints are always a prefix chain - the request loop drops
any checkpoint that is not a prefix of the current prompt, because its
positional cells are gone (the KV cache is one arena, one branch of history
at a time). A radix cache's tree collapsed onto the one branch the session
can hold, and the chain's root is the node a radix cache keeps while its
leaves rotate: in practice the end of the system prompt, which every new
chat of the same client mounts through.
First-in-first-out kept that node only by accident of being first: after
prompt_cache newer checkpoints it was gone, and the next chat re-read the
whole prefix at prefill speed (~30 s per 30K tokens). Now the root is
pinned while the budget allows root and leaf apart (cap >= 2; below that
the old behaviour stands), the rest rotates by least recent use, and a
mount re-dates the checkpoint it mounted through.
The policy is one pure function (conv_cache.hpp) with its scenarios in
conv_cache_test.cpp; checkpoint_save/restore and the state format are
untouched, so STRATA_CKPT_REREAD's token-for-token guarantee holds.
Niko1221