- setup: a third family, "coder" (IQ1_M), with its own expert profile; its
shard 2 and vision encoder are the original's files and are reused; 32 GB of
RAM is now enough to install (the Coder's experts are 23 GB).
- data/expert-profile-coder.bin: the shipped ranking mapped onto the kept
experts through the release's rco-allocation.txt (72% GPU hits, 12 GB card);
tools/make_profile.py takes --n-expert.
- README/DETAILS: the Coder's speeds (prompts 599-1,350 tok/s, output
43-53 tok/s, 1K-262K) and what it is; bench/results/2026-09-28-coder.
- The experimental speed projection can be turned on for the Coder too.
- The model files (models/, packs/, mtp/: 70-120 GB) live in Strata-data next to
the Strata folder, remembered per user (%APPDATA%\Strata\settings.json,
~/.config/strata/settings.json). Updating by unzipping a new copy used to give
an empty folder and a full download again. Model files of this folder and of
earlier Strata folders on the same drive are moved there (a rename) and their
configs repointed; files on another drive are used where they are. A new copy
with no config sets itself up like the last install, reusing the files.
--data-dir chooses another place.
- The start says, before and while it happens, that the PC can be slow or stop
responding for 1-3 minutes while the experts are loaded and locked: setup's
start box, and the server narrates the engine's start from its log (weights,
experts + the warning, expert cache, ready, a "still starting" line every 20 s).
Prompt path (bench/results/2026-09-28-prefill-speed):
- `--prefill auto`: chunks up to 8192 tokens, sized to the cache slots the prompt path may borrow; a
request borrows only what its prompt needs (Prefill::relayout). setup writes it and switches old
`--prefill 2048` configs once the engine supports it.
- The attention and MoE halves share one scratch region (~260 KB less per token).
- Experts through llama.cpp's MMQ kernels (ggml-cuda mmq.cuh, compiled from the same checkout as ggml
into strata_mmq; Strata Q2_0 blobs converted to GGUF blocks on the gather; IQ1_M layers keep FP16).
- Chunks >= 2048: every non-resident expert streams through a ring in a fixed order across layers, so
the next layer's arrive during its attention half (384 slots when the arena is pinned, else 96).
- The PLE block for the whole chunk at once; the next chunk's PLE rows read on a thread; unpinned
experts copied by stager threads (epoch-tagged claims).
- STRATA_PREFILL_TIMING=1 prints a per-phase GPU timeline.
Q2_0 32K prompt 572 -> 1290 tok/s, IQ3_S 383 -> 1208 (RTX 5070 12 GB). Needles 5/5 1K-262K.
Issue #31 / #29: the serve watchdog (now 60 s) prints a stall report - each expert-pool thread's state,
the verify window's flags, layers served during the stall, memory and page faults - and on Windows
writes a minidump with every thread's stack; Release builds keep a PDB. The engine prints its version.
On a card with most experts in VRAM (RTX 4090) generation could stop for
good after thousands of tokens: the GPU "100%" at low power, one CPU core
busy, no tokens, no error.
- The pool's workers sleep after 20 ms without work (issue #4). With most
experts resident many layers have no CPU work, so they also slept
mid-request, and a worker that woke late was still counted as parked: it
could claim from the next batch while the host was writing it, run a job
twice, or add to `done` after the reset - and `done != n` then never
ended while the GPU waited on the pool. Every claim now carries its
batch: `head` is one 64-bit word (epoch | njobs | index), a claim is a
CAS that only succeeds for the epoch the worker woke for, and the host
waits with `<` instead of `!=`. Same speed (pool_test 47.2 GB/s).
- Every pool wait is bounded (60 s: a message and the engine stops), and
--serve runs a watchdog: a request with no progress (tokens, prompt
chunks, verify windows) for STRATA_WATCHDOG_S (120) seconds ends the
engine with where it was stuck (strata/core/progress.hpp). The server
(issue #27) answers with an error, names the cause from the log, and
starts the engine again on the next request.
- pool_stress: tiny batches with sleeping workers; STRATA_POOL_SPIN_US
shortens the spin to make them sleep constantly.
Tested: pool_test, pool_stress; 25,000-token generation at 262K with KV
streaming and constantly sleeping workers (73.9 tok/s, unchanged); the
watchdog deliberately tripped (2 s): error, log line, restart, next answer.
- An animated preview of the pagoda garden demo under the title (the full
video is a release download: GitHub does not play videos from a
repository), and a screenshot of the app in "Using it".
- "Is my PC enough?" folded into Install; "How fast is it?" and "Which model
should I pick?" come first. The model table shows shard 1 (what gets
loaded) instead of the download, with a plain fit rule: RAM >= shard 1 +
~10 GB, since the experts live in RAM.
- "Something went wrong?" as short questions and answers, including the
first-start freeze and "the engine stopped unexpectedly".
- DETAILS troubleshooting: the same two rows, and the VRAM stall row now
points to engine 0.1.9.
- setup requires engine 0.1.10 (INFO arena_mib, the start-up RAM warning).
The engine takes llama.cpp's control-vector flags (--control-vector-scaled
FILE:SCALE, --control-vector-layer-range A B, --cvec-mode add|project,
--cvec-dir per-layer|single:L) and applies the vector after each steered
layer's FFN write on every hyper-connection stream: project h -= s (h.v) v,
or add h += d - where the experimental-speed-projection package's patched
llama.cpp applies it (build_cvec after the layer's second combine).
- strata/kernels/cvec: one kernel per (stream, token); with a pending fused
write it applies it first with the fused read's own arithmetic, so a
loaded-but-off vector is bitwise the stock engine. cvec_parity tests it.
- Per request: cvec=0|1 on the GEN/GENI line, the API field
"experimental_speed_projection", a switch in the web app's Sampling
drawer; a switch drops the conversation cache once. About and the
Monitor say which way each request ran.
- setup asks (default off; --experimental-speed-projection on|off|GGUF),
original Qwen3.8-Flash-Next only. The vector ships in
data/experimental-speed-projection with a README that says what it is:
its package describes it as a refusal-direction projection.
Measured (bench/results/2026-09-27-esp): off = stock token for token; on
costs 0.2-0.4% per token; mean KL 0.063 on 2,557 teacher-forced tokens.
--host 0.0.0.0 bound fine, but nothing in the normal flow could set it: the
generated run scripts and setup's start always used the default
127.0.0.1, the config had no host, and the server then printed
http://0.0.0.0:8080, which opens nothing on another device.
- "host" in strata-<model>.json (the run scripts need no change); setup
takes --host and --api-key and writes them there.
- With a non-local host the server prints this PC's addresses (the default
route's first), warns when there is no API key, and on Windows says how to
let it through the firewall. The local lines name 127.0.0.1.
- Documented in DETAILS.
- A prompt that leaves no room at all is still the clear 400 with the flag
on (it clamped to 1 token, which the engine then refused as over the
context).
- "fit_max_tokens": true in the run config switches it on too, since the
run scripts are generated by setup.
- Tests: an overshoot is clamped to the room on both APIs, a budget that
fits is unchanged, no room is a 400. Documented in DETAILS.
- serve/web/fonts: Outfit as one variable font (latin + latin-ext, weights
100-900) with its OFL.txt, in place of the design kit's commercial font.
- The page no longer saves the system's theme on first load (which froze
it); only a click on the theme button or toggle is remembered, and the
page follows system changes until then.
- setup: requires engine 0.1.8 (--kv q4_0, the INFO line).
The browser page at http://127.0.0.1:8080 is now the Strata app, in the
Strata design language (serve/web, same origin, no CDN):
- Chat: streaming answers, collapsible thinking, Markdown with code copy,
images when enabled, sampling/thinking settings, export, new chat.
- Monitor: live model state (reading with progress / generating at tok/s),
GPU load, VRAM, temperature, power, PCIe traffic, CPU, RAM, disk, the
context fill and the last requests, each with a minute of history.
- About: the model and engine settings, this PC, the API addresses.
Server: GET /metrics (API key when set), /web/* app files, /fonts/* only
when installed locally (not shipped; the page falls back to the system
font). serve/telemetry.py reads NVML via ctypes and psutil (new setup
dependency) with fallbacks. The engine prints an INFO line before READY
(kv, expert slots, speculation); older servers ignore it and the server
works with older engines.
Based on the dashboard idea in PR #22 by code-martin.
PR #21 (code-martin): a 4-bit K/V format (orthonormal FWHT-256 rotation of K, V and Q, then q4_0 blocks). On merge:
- Q4 is a full third format of the KV-streaming code (host copy, block residency, prompt-path staging, MTP ring,
STATE_HASH): with --kv-resident the PR wrote K/V through the residency map's -1 entries. kv_stream_parity now
checks Q4 streamed vs resident, bitwise.
- The per-token path ran attention twice (once before the blocks were resident); once now.
- The q4 group maximum breaks |x| ties the same way in every lane (one scale per block).
- Dropped: the duplicate path fixes (in 0.1.7), personal .gitignore entries, the re-encoded CMakeLists.
- Measured (bench/results/2026-09-27-kv-q4): int8 matches fp16; q4_0 costs precision on long documents
(perplexity +8-12%), needles 5/5, +3.7% decode at 128K. Opt-in: setup asks above 8K context (int8 recommended),
START-HERE.bat --kv q4_0, or --kv q4_0 in the config.
#23: the start-up warning "the GPU hit path is NOT CORRECT" dated from a long-fixed fault. Measured teacher-forced
on 2,557 tokens (bench/results/2026-09-27-cache-parity): cache on vs off, 95-98% same top-1 and equal perplexity
(-0.005 +- 0.005 nats); the warning is now one accurate line.
Co-Authored-By: code-martin <[email protected]>
- Prompt lookup on by default: the suffix drafter's windows are taken only where a learned policy (measured
acceptance per match length, measured window cost) expects more tokens per ms than the MTP window; lookup windows
up to --spec + 2. Code edits +6-11%, other text unchanged, verified output identical. --suffix-draft 0 = off.
- Per-request sampling (#19) keeps greedy requests on the in-graph sampler (no extra launch per window), and draws
row t of a window with Philox(seed, pos0 + t), so a seed replays the same text whatever the drafts were.
- tools: a shallow install (G:\Strata) no longer crashes the MTP packer and the pack tools (IndexError, #20).
- setup: says at step 1 when the PC has less RAM than the smallest model needs; requires engine 0.1.7.
From 64K context the KV cache lives in pinned RAM and a VRAM block cache holds the part the sparse attention reads
(--kv-resident N; setup turns it on with 32768 when RAM allows). Blocks the indexer selects are made resident on
the device inside the captured graphs (clock eviction, zero-copy reads of misses); writers write through, so the
attention reads exactly the same values. The freed VRAM goes to the expert cache: Q2_0 at 262K decodes 62.6 tok/s
instead of 50.9. The MTP drafter keeps its window in a ring. New ctest kv_stream_parity. Also wires the opt-in
suffix drafter (--suffix-draft) into the engine.
min_p, presence_penalty, repetition_penalty and frequency_penalty move
from "warned and ignored" to the sampled path, and penalty_last_n sets
the window they count over - the server sends the engine's default of
64 when a penalty is set and the client left the window out.
sampling_keys forwards each field only when it is not neutral, so a
client (or a config) that leaves them out costs nothing. The
validator range-checks the new fields and still refuses to start the
server on a value it cannot honour.
Verified: the validator accepts and range-checks min_p/presence/
frequency/repetition/penalty_last_n; the GEN line spells them the
engine's way and omits neutral values; the last-64 default appears
only when a penalty is actually set.
The run config's optional `sampling` block (temperature, top_p, top_k,
seed) sets defaults for the fields a request leaves out, so a client
that sends no sampling keys gets configured sampling instead of greedy.
The request's own fields always win: an explicit temperature=0 still
means greedy, and a field set to null falls back to the default.
Keys the sampled path does not implement yet (min_p, presence_penalty,
repetition_penalty, frequency_penalty) are named at startup and ignored
rather than silently dropped; unknown keys are reported the same way.
An invalid value (top_k outside 1..64, a non-positive seed, a negative
temperature) refuses to start the server - a typo'd config should not
quietly change sampling.
Verified with a spy engine through Service.run: defaults apply when the
request omits the fields, the request's own values win, a null field
falls back to the default, and a server without the block forwards the
request unchanged; the config validator rejects out-of-range values and
warns on the not-yet-implemented keys.
It contradicted the paragraph next to it, and since the sampling keys
now ride the GEN line (and the block-parallel sampler makes a sampled
window cost about what a greedy one does), it was simply false.
- GPU experts: IQ4_XS gate/up (the IQ3_S file keeps layer 47's routed experts in it), transcribed from llama.cpp's
vec_dot_iq4_xs_q8_1; checked on all 48 layers against ggml's float reference (native_expert_parity: 0 failures).
- MTP draft head: gather_rows takes rows of any size (the IQ3_S head is Q6_K, 2100-byte rows, not a multiple of 16).
- setup: IQ3_S for the original model (Swift 1.5 has none), 64 GB RAM, context capped at 128K like IQ3_XXS.
- README / DETAILS: IQ3_S speed (RTX 5070, Ryzen 5 7600, 64 GB): 54 tok/s at 4K, 42 at 128K, 374-397 prompt.
Every request used to start from session_zero and read its whole prompt again,
so a long chat or an agent loop waited minutes per turn. The engine now keeps
the live session and up to 6 checkpoints of the running state in RAM and reads
only the part of a prompt after the longest one it starts with.
- A checkpoint is the state that cannot be rewound - the 36 GDN recurrences and
conv histories, the 12 QSA indexer tails, the PLE history (~118 MB). The KV
cache and the pooled indexer keys are positional (a query reads only cells
below it, and `dead` for its own block), so they stay in place.
- Taken at the last <|im_start|> of each prompt (the prompt is read in two parts
there: the history, then the new turn's header) and every 16K fresh tokens.
- Used only when the prompt starts with exactly its tokens and pictures (images
are compared by a hash of their embeddings and grid); checkpoints the next
request would overwrite are dropped.
- New lines on stdout: RESUME <n> before reading, PP <pos> <n> <ms> <tok/s> per
chunk, REUSED <n> after; DONE gains <drafts accepted> <drafts offered>
<reused>. The server ignores what it does not know; PP lines now count as its
heartbeat, so keep-alives keep flowing during a long prompt.
- --prompt-cache N (default 6, 0 = off), --prompt-cache-every N, --turn-token ID.
- Checks: STRATA_CKPT_REREAD=1 reads a checkpoint's tokens again instead of
restoring it; STRATA_STATE_HASH=1 prints a fingerprint of every state part
after each request.
Streaming
- tool calls stream as they are written (OpenAI tool_calls deltas, Anthropic input_json_delta)
- keep-alives while a long prompt is read, progress in the server window, GET /status
- closing the connection or pressing stop really stops the engine (STOP)
Stalled requests (mostly image turns)
- the expert cache is sized after its slots are written (Windows only counts GPU memory once it is
touched), so the reserve really stays free instead of the card running at 0 MiB and paging
- the prompt chunk shrinks when its buffers do not fit in the cache, instead of allocating on top
- the image encoder warms up before the engine starts; images are encoded in request order
- the log shows the VRAM left once everything is loaded
setup.py now requires engine 0.1.2 (downloaded automatically on the next START-HERE.bat).
Niko1221