Niko1221/Strata
1
0 Fork 0
mirror of https://github.com/Niko1221/Strata.git synced 2026-11-08 17:35:22 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity
5,940 Commits 2 Branches 73 Tags
main
Commit Graph
3 Commits
Author SHA1 Message Date
Niko1221 8253477bf9 Prefill planner: request-sized chunks. A prompt that fits today's chunk keeps one chunk; a longer one reads in equal chunks of at most the plan's size (no stub tail). Buffers stay sized for today's chunk. 2026-10-18 09:20:24 +02:00
Niko1221 d8471050bd Prefill planner: refit the miss share to two rigs (#2824 SATA 2x3060, P2003 NVMe cgroup sweep); read the drive class from the GGUF path 2026-10-18 05:09:14 +02:00
Niko1221 e77f1dfc3f Prefill planner: price the host (RAM page cache, displaced experts, drive) when choosing chunk and ring (#2824) 
The auto chunk scan only tested the GPU cache.  A lent slot's expert comes back from the RAM copy or the model file for the length of a
prompt, and every chunk streams the whole file set; past the page cache that set is re-read per chunk (28-45 GB per 29K prompt on
2x RTX 3060 / 32 GB / IQ2_XS at 8192-token chunks, ~1 GB at 4096).  prefill_plan.hpp is the pure model (file set, miss share fitted to
#2824, additive stream + compute + drive time); the serve scan and the generate path refine the VRAM pick with it, after the RAM copy
exists.  A set that fits the page cache keeps today's pick.  STRATA_PREFILL_HOST_PLAN=0 turns it off; --prefill N is untouched.
 2026-10-18 03:21:24 +02:00