On a card whose experts mostly run on the CPU the slots cost speed per request (RTX 5070, Q2_0: 11-24 %), so setup
now recommends slots only when the expert cache (every card of a split counted) still holds half of the model's
experts beside them (Q2_0: 24 GB and up, or a split); elsewhere it stays at one at a time and says "parallel N
reduces waiting for several users but costs about 10-25% speed per request on this card". --parallel N is
honoured as asked, as before. BATCHING.md: the table with the change against one at a time.
server: a request decoding in a slot whose neighbours finished (no other slot busy, nobody waiting, at least 32
tokens allowed) is BSTOPped and continued on the solo path with its MTP drafts - the engine copies the slot's
sessions back (INFO slot_cache=1; at most twice per request; STRATA_PARALLEL_SOLO=0 keeps it in the slot).
engine: INFO batch_slots=N slot_cache=0|1.
tools/batch_interleave_test.py: the back-to-solo path, and a next turn from a slot's turn checkpoint (a client
that drops the reply's thinking).
docs: "parallel": N, what a slot costs and what setup recommends, the server's handling (interleaved prompt reads,
BYIELD, slot conversation cache, back to solo), exactness and its settings, the RTX 5070 measurements (the slots
buy waiting time, not speed, on a 12 GB card), the protocol; DETAILS / README / AI_SETUP mention the option.
- tools/parking_test.py: a follow-up decodes the same tokens whether its conversation stayed live or came back
from the parking cache (every stage's image with a layer split). 4-GPU split, IQ3_S: 3,786 and 9,276-token
conversations restored in 42 / 51 ms, follow-ups identical; without parking the same follow-ups took 3.4 / 6.8 s.
- tools/batch_test.py: --keys to send sampling keys (sampled rows in batch windows).
- docs/BATCHING.md: parking together with batching, the Testing section; docs/DETAILS.md: parking now works with
--layer-split.
The engine's --serve can now keep up to N conversations open (--batch N) and decode them together, one token
each per verify window; with a layer split, --batch-groups G pipelines the slot groups through the GPUs, and
--trim-stage-weights makes every GPU load only its own layers' dense weights (VRAM back to the expert cache).
The server runs concurrent requests in the slots. See docs/BATCHING.md.
Niko1221