#1498 pins the KV host copy with cudaHostRegister under WSL; the switch forces either order. Measurements: 5 interleaved pairs on an RTX 3060, an RTX A4000 and a Tesla P100.
The planar kernel of #2340 is the wide kernel's arithmetic, but it was launched without the 256-GRF mode that #2207 gives the GGUF-layout
wide kernel, so on an Alchemist card it spilled: mmvq_bench 2560 x 248320 on an A750, 4 columns, 1.78 ms (GGUF layout, large GRF) against
15.4 ms planar, 8 columns 4.8 against 48 ms. Same rule as #2207 (STRATA_MMVQ_LARGE_GRF, default on Alchemist only), same arithmetic.
On 2x RTX A4000 behind a host bridge cudaDeviceCanAccessPeer says 1 and cudaDeviceEnablePeerAccess succeeds, and
cudaMemcpyPeerAsync then returns success having moved nothing (1,048,576 of 1,048,576 floats wrong in both directions
in a stand-alone test; right without peer access enabled, the driver stages it). The peer prompt path trusted it: the
peer's input rows were zeros and its results never reached the primary, the first prompt's MoE output was garbage and
the logits NaN (--peer-device 1, --peer-reserve-mib 1500..2100; STRATA_PF_PEER_COMPACT is not the cause).
PeerExperts::open now copies 64 KiB each way once after enabling access; a wrong result turns peer access off again and
takes the host route (the one STRATA_PEER_NO_P2P=1 asks for), and says so. A test that cannot run keeps the old trust.
Reported by @CaptRauti on #2666 (WDDM, 2x RTX 3060: the no-P2P route is the one they ran).
0.1.41 scaled each stage's miss term by its probed link (0391e478);
0.1.44 scales from a 4x gap against the fastest link (9d41e18b, #2007).
On an x16+x4 dual-3090 rig (26.9 vs 6.4 GB/s, 4.2x) either rule moves
auto off the balanced placement (K=27 -> K=19/K=31): decode -4..-8%,
prefill +15..+19% at 10K / -6..-9% at 90K vs 0.1.40.x at matched
calibration. Opt-in restore next to STRATA_SPLIT_LINK_ABS, default off.
On an MI50 ROCm queues are suspended while the kernel reclaims around the HMM-tracked arena; the cached
reads of the new rule churn the page cache. The gfx906 backend keeps the old room >= read_bytes rule;
other HIP cards keep the current rule (the PR applied it to every HIP build).
Co-authored-by: KevinX8 <[email protected]>
The default takes the pipelined decode on two stages only with the draft layer on the first card (mtp_first); asked
for by hand it was taken without a word, and without mtp_first the draft chain shares the last card with its
window (2x R9700: the pipeline lost to the serial loop before 0.1.44). Recommend, never force: it stays on as
asked and a note says what it costs, why mtp_first is not on (no --mtp / --spec 2+ / draft_vocab.bin, or
STRATA_MTP_STAGE0=1 to ask for it) and how to turn it off. No default output changes.
Reported by @dinesh-chander on #2637 (closed by the reporter).
On RTX 4060 Ti 16 GB + RTX 3060 12 GB (48 layers, IQ3_S) the first pick was K=2 and the #1616 retry
(every stage at least 3 layers) landed on its own edge, K=3: the 3060 ran 3 layers, 12 GB idle, 452 tok/s of
prefill against 802 at 28 layers (the prompt flows through the stages as a pipeline). For two GPUs the retry
now first asks for a placement in which each stage owns at least floor(layers x its room / all rooms) layers
(split_rules::share_floors, at least 3, trimmed to fit); the three-layer rule stays the second try and the
first pick the last, and the log says which one was taken. Nothing changes where the first pick has no
one- or two-layer stage (equal cards), and three or more GPUs keep the #1616 rule alone.
Reported by @Tworks25 on #2644, with the sweep of layer_split 12..32 (PR #2643). split_rules_test gets the
reporter's rooms (14.2 GiB and 5.50 GiB) and the identical-card and degenerate cases.
Tesla P100 (sm_60) prompt read, interleaved 5 pairs each, thread vs warp scorer: 30000 tokens 339.29 vs 335.45 tok/s (1.011) and 338.98 vs 330.20 (1.027), 60000 tokens 338.88 vs 330.15 (1.026), 5/5 pairs faster each, below the +3% bar for a new default. Same bits either way. docs line; qsa_scores_parity runs with the switch on.
The z440 A4000 Q2_0 gate with STRATA_PF_FUSED=1 on both arms found the residual dump different from base: gu_covered() took Q2_0 on every CUDA card and d.ok on sm_80+ now required the Q2_0 gate/up setup.
Interleaved A/B, 6 pairs, --pcie-frac 0 vs --pcie-frac 0 + the switch, decode ALL: RTX 3060 IQ3_XXS 53.36 vs 53.16 tok/s (0.996, 3/6 pairs faster); RTX A4000 Q2_0 61.72 vs 62.17 (1.007, 4/6). The author's own +2.2% is below our +3% bar and does not reproduce on Ampere. Declined; the PR author's commit stays in the history only as this revert's parent.
"--ple-io mmap" as ONE element of "args" (or "--ple-io=mmap") reached the engine as a single
argument, which it stops on ("unknown argument: --ple-io mmap"); the usual form is two entries.
The server now splits such an element into the flag and its value and says so at startup
(never one in a flag's value place). A config with a JSON mistake (a missing comma between two
args) gave a traceback; it now names the file, line and column. The slow-PLE-probe warning says
how the two entries are written in a config. --ple-io mmap, ram and direct themselves parse
correctly in the engine (checked on the 0.1.45 build).
Reported by lama7 on #2267.
PleTable::gather_batch matched the Direct-mode prefetch slots by position, so rows
prefetched as A..F and gathered as D,E,F,A,B,C (or A gathered as A,A,A) were read a
second time. Each requested token now looks its key up among the prefetch slots and
decodes from the matching slot into its own output position; duplicates share a slot.
All outstanding tickets are drained on a failed collect (the first error is kept)
instead of dropping the rest. The decoded bytes are unchanged.
Reported by @hotschmoe, who also supplied the fixture and the patch; the patch is
used as proposed. ple_reader_test --selftest gets the reorder / duplicate / subset /
missing / over-the-window cases, compared against the mapped reader.
The SYCL build of the sm_75 series failed with 'out-of-line definition of bf16 does not match any declaration'
(sycl/src/prefill/gemm.dp.cpp); the four gr_* writers would have failed at link. Gemm::bf16_converts_x() answers false.
Ported from c3c's #1581 (the native-IQ and VRAM-loan parts are not in this commit). Default unchanged: Turing keeps returning false for q4_0.
Co-authored-by: c3c <[email protected]>
The v1 kernel keeps its exact previous code (and 37.9 KB of shared memory) unless the flag is set. launch_i8 picks the Turing
single-stage layout by the (emulated) compute capability, so the kernel and the launch agree under STRATA_EMULATE_CC.
No sm_75 card is available here to confirm the series, so the kernels that were on by default for compute capability 7.5
in the submitted PRs now follow one switch, include/strata/core/sm75_series.hpp. Each keeps its own variable, which
wins in both directions. The one-read prompt top-k (#2093) was on for every CUDA card as submitted; it is now Turing only
like the rest. STRATA_EMULATE_CC=75 counts as a Turing card, so a 75-virtual build on an sm_86 GPU runs the series for tests.
block_scores_thread_kernel gave a lane one (query, head) and the whole
128-value query head; ncu on a CMP 50HX showed it bound by the MIO pipe
(mio_throttle first): one shared-memory key load per 4 FMAs. Lane 0's
xor-shuffle tree splits into four independent quarters - c[j] sums groups j,
j+16, j+8, j+24, j+4, j+20, j+12, j+28 - which only meet at the end
(c0 + c2, c1 + c3, then their sum). Now four lanes share a query, lane j
holds quarter j of all four heads, a key load feeds four heads (16 FMAs),
and two xor shuffles finish the same tree. The same pairs in the same order,
so the same bits.
qsa_scores_parity on both cards: all 141 M scores identical to
block_scores_kernel; per call 1,024 queries at 140K 7.76 -> 6.17 ms (-20 %),
at 200K 10.78 -> 8.54 ms (-21 %), 2,048 queries at 40K 12.60 -> 11.57 ms.
agentbench 34K -> 190K: AGENT GATE PASS on the rerun (the first run diverged
at turn 2 - decode's known run-to-run variation; the rerun of the same binary
matched all 36 turns), incremental read -0.6 % median, about -1 % from 150K
up, decode unchanged.
On cards without the TF32 scorer (sm_75) the prompt path scored every
(query, key block) with a whole warp: 20 shuffles and five 512-byte loads per
score. block_scores_thread_kernel gives each lane one (query, head), keeps the
128-float query head in registers, reads each key once from shared memory as
a broadcast, and adds the 32 four-value partials in the tree lane 0 of the
xor-shuffle sum computes. Float addition is commutative, so the same pairs in
the same tree give the same bits; the heads are summed 0..3 from 0.0f as
before; the tail block keeps block_scores_tail_kernel. STRATA_SCORES_THREAD=0
selects the old kernel.
Measured on 2x CMP 50HX (TU102, sm_75, unlocked to RTX 2080 clocks):
- qsa_scores_parity (new): 141 M scores bitwise identical on both cards
(normal values, wide magnitudes with zeros, small integers with ties);
per call 4.6-5.2x faster, e.g. 1,024 queries at 200K: 52.4 -> 10.8 ms,
at 140K: 35.9 -> 7.8 ms. 199 registers, no spills.
- nsys, one +4.4K-token turn at 140K: block_scores_kernel was the top prompt
kernel at 1,045 ms per card (17 %).
- agentbench, IQ3_XXS, layer split, 262144 context, 34K -> 190K (with the
tokenizer cache): engine read per turn at 190K 9.25 -> 6.96 s (476 -> 632
tok/s, +33 %), time to first token 10.49 -> 7.13 s (-32 %); median
incremental read 576 -> 682 tok/s; decode unchanged (73.3 tok/s).
AGENT GATE: all 36 turns word-for-word identical to the base run.
The Turing (v1) prompt attention kernel used 37.9 KB of shared memory per
128-thread block, so one block ran per SM (4 warps, 12.5 % of the warps an
SM can hold), and each 32-cell chunk gathered its scattered K and V rows
after the previous chunk had finished: ncu on a CMP 50HX showed
long_scoreboard (global load latency) as the top stall. For int8 K/V
(--kv int8, mode 1):
- the next chunk's K/V rows and scales are loaded into registers while the
current chunk computes (Turing has no cp.async), then stored where the
gather stored them;
- K is read only by the scores and V only by p.v, so they share one buffer:
V is written over K after the scores' barrier (the softmax's barrier
orders it before p.v). 37.9 -> 29.2 KB: two blocks per SM.
The same bytes in the same places and the same arithmetic, so the same bits.
Other KV modes keep their code. qsa_prompt_attn_parity now prints a checksum
of the new kernel's output, so two builds compare bitwise.
Measured on a CMP 50HX (sm_75), qsa_prompt_attn_parity, 2,048 queries:
int8 21.45 -> 12.91 ms at 40K (-40 %), 21.94 -> 13.31 ms at 140K (-39 %);
all output checksums (int8 and fp16, 4K/40K/140K) identical to the previous
kernel. agentbench 34K -> 190K (IQ3_XXS, layer split, 2x CMP 50HX): AGENT
GATE PASS (36 turns identical), incremental prompt read time -3.5 % (median
per-turn ratio 0.965), decode unchanged.
With --pcie-frac 0 the pool never plans PCIe experts, yet every layer of a
verify window still runs the flag B wait, the staging copy and the grouped
launch over zero groups: six kernels, ~18 us a layer on an RTX 2080 Ti
(kernel-node nsys). STRATA_NO_PCIE_GROUP=1 records the window graph without
them; a plan with PCIe experts then stops the engine with a message.
Engine A/B on an RTX 2080 Ti (Flash-Next Q2_0, 512 tokens, two runs each, with the open sm_75
decode PRs applied): Spanish chat 87.8 -> 89.8 t/s, code 106.1 -> 108.5,
32K 78.4 -> 80.1, 128K 75.0 -> 76.8, 390K 74.7 -> 75.1.
Two kernel pairs of a verify window's layer run as one launch on Turing,
every byte as before:
- the plan's host-flag wait folded into its copy (143-amdk's
wait_copy_i32_from_mapped, gfx12-only until now). STRATA_FUSE_WAIT on
CUDA: 2 = the plan only (Turing default), 1 = also the CPU rows'
copy_rows_from_mapped_after, which makes every row block poll the flag
over PCIe and was -1.5 % on an RTX 2080 Ti, 0 = neither.
- native_quantize_q8_1_il: a projection input's q8_1 blocks and their
interleaved copy (native_q8_1_interleave) in one kernel; an int of the
copy is four codes gathered with two shuffles. Used for the GDN qkv and
out-projection inputs and the QSA input and out-projection. STRATA_QIL=0 / 1
forces it off / on (Turing default on).
RTX 2080 Ti, Flash-Next Q2_0, 512 tokens, two runs each: the plan fold
+0.6..+1.4 % decode, the fused quantize another +0.5..+1 % (code 108.0 ->
109.1 t/s). The fused quantize writes the same bytes as quantize +
interleave at 2560/6144/10240 inputs and 2-4 columns.
The exact layout gives each Q2_0 row four warps, a half block per thread and a
cross-warp reduction. At the engine's 2560 inputs (40 blocks a row) 48 of the
128 threads have no chunk and the reduction dominates: the QSA attn_q
projection (2560x12288) at three columns took 80 us (~110 GB/s) on an
RTX 2080 Ti.
q2_0_warp_rows_kernel: lane l runs the chunks of the exact kernel's threads
l, 32+l, 64+l and 96+l in their loop order, adds the four sums as warp 0 adds
the other warps' partials, then the same warp_sum - every output is bit for
bit native_q2_0_mmvq_kernel's (one column) and launch_multi's exact layout
(two to four). 1/2/3/4 columns: 36.6/60/68/88 -> 24.8/31/31/37 us.
Default on sm_75 only; STRATA_Q2_WARP=0 keeps the old kernels, =1 takes this
one on any CUDA card.
attn_chunk_kernel<1> saturates the SM's shared-memory pipe on Turing: its
time grew linearly with the queries while the SMs were half empty (two
16-byte q reads per head per cell with a two-way bank conflict, and 12
warp_sums = 60 shuffles per cell).
attn_chunk_q8_v3_kernel computes the same operations in the same order:
- the lane's slice of the 12 query heads is read into registers once;
- the 12 lane sums are reduced with reduce12_gfx906 (16 shuffles, every
sum warp_sum's pairs);
- the probabilities are stored cell-major, so the value loop reads a
cell's 12 as three 16-byte broadcasts.
RTX 2080 Ti, with the merge: 2051 selected cells, 1-4 queries
51.6/78.5/110/146 -> 34.5/41.0/73.0/84.9 us; 600 cells
45.9/74.6/105/137 -> 28.6/32.8/33.5/33.3 us. qsa_prompt_attn_parity
reports the same errors with STRATA_ATTN_V3=0 and 1.
Default on sm_75 only; STRATA_ATTN_V3=0 keeps the default kernel, =1 takes
this one on any CUDA card.
The fused prompt-expert kernels (moe_fused.cu for the Strata Q2_0 pack,
moe_fused_iq.cu native_kernel for the native packs) on sm_75, with
STRATA_PF_FUSED=1 STRATA_PF_FUSED_SM75=1:
- cp.async -> plain 16-byte copies (the barriers around them stay),
prefetch.global.L2 kept;
- mma m16n8k32 / m16n8k16 s8 -> four / two m8n8k16 (exact int32 sums,
the same d as the sm_80 instructions), MAGIC as the C operand;
- two activation stages on Turing (64 KB of shared memory);
- native_kernel takes Q2_0 gate/up (64-value blocks, a block per stage),
so the canonical Q2_0 pack (Q2_0 / Q2_0 layers) runs fused; on Turing
only that pair is set up;
- Turing, Q2_0: the two halves of a stage share the weight scale, so a
row's dot is d (dx_0 dot_0 + dx_1 dot_1) - 11 % fewer instructions.
RTX 2080 Ti, one layer, 512 experts top 10 (prefill_fused_iq_test,
Q2_0 / Q2_0 pair added): 2048 tokens MMQ 12.98 ms -> fused 6.59 ms,
16384 tokens 55.08 -> 37.16 ms; fused vs MMQ rel RMS 2e-5, the same
error against the FP32 reference. In the engine (canonical Q2_0 pack,
98K prompt, 16K chunks): prefill 58.3 -> 53.3 s; layer 0 expert rows vs
MMQ: median rel 1.6e-7. Teacher-forced 2048 tokens at P = 16K / 32K /
64K: perplexity vs MMQ +3.4 % / -4.5 % / +0.6 % (FP16 path vs MMQ
+0.7 % / +0.1 % / +0.3 %) - trajectory noise, no systematic shift; the
verify phase (same tokens) 42.36 s MMQ, 42.28 s fused.
A borrow copies the donor's checkpoints up to the match, so a match that reached a pinned shared prefix gave the borrower the pin too. Such a match now takes the conversation as before. The borrow's guard is guard()/unguard()/guard_blocked(), and the refusal before evicting counts the pinned conversations along with the donor.
Rebased by hand onto 0.1.44.1 (evict_oldest() honours the guarded donor).
Co-authored-by: BlueKingMuch <[email protected]>
Sibling subagents with the same system prompt and tools no longer cost each other their history: only the K/V up to the checkpoint and the checkpoint itself are copied into the session (conversation_snapshot_restore_prefix); the parked conversation stays where it is and is pinned while the outgoing one parks. A match on a parked conversation's live state or newest checkpoint is that conversation going on and is taken as before.
Rebased onto 0.1.44.1 by hand: the pinned shared prefix (#879) and evict_oldest() now also skip the borrowed donor, serials follow every erase, and park-gain (#2265) and the pin= sibling rule gate the outgoing park.
Co-authored-by: BlueKingMuch <[email protected]>
A client renders the history again for its next turn, without the reply the slot decoded or the assistant header,
so the slot's held stream (prompt + reply) is no start of the new prompt and pick_slot fell back to the least
recently used slot. With two chats in two slots and two next turns arriving together, whichever was read first
could take the other chat's slot, and the engine's slot copy then overwrote the cache the second turn needed
(0 tokens reused, a full prompt read). pick_slot now also counts the held tokens up to the last <|im_start|>
(where the engine takes its turn checkpoint) when the prompt agrees with them, and a pipelined layer split with one
slot per group (--batch 2 on two cards) no longer ignores what its slots hold.
Least recent use keeps the newest checkpoints and every turn adds one, so after a few short turns all 6 sit in the
last 1-2K tokens. An agent client that rewrites its history a few thousand tokens back then resumes from the system
prompt: on 2x CMP 50HX with the pi coding agent a 210,528-token prompt that was the same as the session for 204,349
tokens resumed at 9,525 and read 201,003 again (100 s).
With the flag the victim is the checkpoint whose removal leaves the smallest gap for its distance from the newest, so
the kept points thin out geometrically: after 200 turns of 1,000 tokens they sit 0, 1K, 3K, 8K, 43K and 200K before
the end, and an edit 6K back reads 8K again (least recent use: 200K), 42K back 43K. The same number of checkpoints, no
more RAM; never the root, the newest or a pinned one, the tail rule first, the plain policy without a candidate. The
resume is the usual exact-prefix match.
On the box: an 82,593-token conversation grown turn by turn, a turn ~6 steps back rewritten, resumed at 73,610 tokens
in 10.2 s against 40.0 s read in full; with the expert placement fixed (--adapt-swaps 0, --pcie-frac 0.17) the
greedy answer was the same text as the full read. conv_cache_test: the spread cases (a 200-turn simulation against
least recent use, the fallbacks). docs/DETAILS.md: a paragraph after the prompt cache's options.
A conversation that only advanced a few hundred tokens since its last park does not need a fresh 3 GB copy: the entry it would supersede still holds this conversation's prefix, and the next turn drops the copy anyway (drop_superseded). Reported in #2264 over 1,418 park events on one long single conversation.
The flag is opt-in and defaults to 0, which parks every turn exactly as before. What this adds is a way to ask for the gate without a rebuild: on the reporter's rig it is 3.26 GB instead of 9.79 GB per benchmark, the superseded class disappears, and the decode median moves -3.4 %, inside that arm's 92.2-124.1 tok/s spread.
parked_prefix_tokens() reports exactly the entries drop_superseded() would remove - it finds them the same way, by the deepest checkpoint the outgoing chain still holds - so the gate cannot name a different entry than the park would drop. conversation_cache_test covers it (7 checks; 4,226 pass).
Fixes#2264
A conversation that was restored and is parked again keeps its unchanged K/V
pages and appends the new ones (92a3d5f). The appended segment reserved room
for later turns, growing geometrically up to 16 MiB - per buffer, and a parked
image has one per K/V layer and part (k, v, their scales, the indexer rows, the
draft layer). With Qwen3.8-Flash-Next (IQ3_S, int8 K/V) a 32K-token main agent
parked again took 1.49 GB instead of 0.97 GB, a 17K-token subagent 1.16 GB
instead of 0.74 GB. The cache counts that capacity against its budget, so with
2816 MiB the reserve decides which conversations are evicted.
The reserve is now at most an eighth of what the buffer held before the append
(at least 64 KiB, and at most 16 MiB as before). Small turns still fill the
reserve instead of adding a transfer each: the 4096-append test keeps its
segment bound, and 256 turns of 1 MiB onto a 16 MiB buffer take 28 segments
(17 before).
The docs said parking falls back to a full capture when the reserve would evict
another conversation; that check went in 6cd29d9. With the reserve bounded the
two captures differ by at most an eighth of the K/V, and the docs now say what
the reserve is.
A standing ban of a whole script (65,929 IDs) was normalized five times per request: in _openai, twice in
validate_request and twice in engine_key. About 200 ms of Python per request, whatever the answer length.
A list is now normalized once. The same object again in a request is answered from the last call; an equal list in
a later request is found by its compact JSON text, which tells False from 0 and "1" from 1 (== does not: [[7, False]]
== [[7, 0]]). Only a list that normalized without an error is stored. Lists under 64 entries skip the cache.
No change to what normalize, engine_key and validate_request return or raise.
PromptEncoder keeps the last prompts with their ids and resume points (one per
conversation-cache slot) and encodes only the text after the longest shared
special-token boundary; the ids equal a full encode. Rebased onto 0.1.44.1
(prepare() keeps the new `cancel` argument; forget_prompt on a context refusal;
count_tokens reads without storing). SECURITY.md notes the in-memory prompts.
Based on gputier's #1852 (8 commits) and blange48's #567.
Co-authored-by: Guillaume PUTIER <[email protected]>
Co-authored-by: benoit lange <[email protected]>
Bit-identical (gate: 10/10 + exact_pp 4K/20K IDENTICAL with STRATA_DEQUANT_WIDE=1 on the RTX 3060 and the RTX A4000), and the time
to the first token of a 150 / 300 / 450-token prompt falls 12-20% on the RTX 3060 (5 of 5 pairs each), the prompt read +2.3%.
STRATA_DEQUANT_WIDE=0 turns it off. Arch-gated to the measured sm_86 and sm_61. Follow-up to #2178 (Techannel), the kernels this switches on;
the same stores are what #1525 (sergqwer) proposes for Q8_0 / IQ4_XS.
A build folder configured while the option's default was OFF keeps
STRATA_Q6K_EXPERTS=OFF in its CMakeCache.txt even after the default
flipped to ON (465cd61d): CMake does not rewrite an option already in
the cache, and source_hash() fingerprints the source files but not the
CMake option values, so setup recompiles the engine with the Q6_K
gate/up expert kernels still disabled. A model whose routed experts are
Q4_K/Q6_K (Unsloth UD-Q4_K_XL) then fails to load with:
strata-q36: blk.34.experts are Q4_K/Q6_K, which the native expert
kernels do not take (build with -DSTRATA_Q6K_EXPERTS=ON for Q6_K)
Passing -DSTRATA_Q6K_EXPERTS=ON explicitly overrides the stale cache
entry, so the engine always has the kernels its default build promises.
Same pin in the Dockerfile build for consistency. Not hardware-specific:
it is a CMake cache-staleness issue, independent of GPU/platform.
The output sum of token t is done after token t + 1's first barrier (sk / sq double-buffered), off the recurrence's chain. Same arithmetic: the
3000-token prompt's logits and 8 greedy steps are bit-identical to the base engine at chunk 512 and 2048 (R9700: 913 -> 885 ms, 607 -> 573 ms).
With many slots on a small card (8 sessions on a 12 GB RTX 3060, UD-Q6_K_XL) the sessions left too little VRAM for the hybrid
mode's staging pair, CPU-output buffer and expert slots, and the start died in a bare cudaMalloc failure. place_experts now adds
up what it is about to allocate and exits 1 with "not enough VRAM left for the hybrid experts' buffers and slots (N MiB needed,
M MiB free ...)" and the way out (fewer slots, a shorter --max-context, fewer --expert-slots). Only a start that would have
crashed changes; the placement and every answer are as before.
tools/qwen35/q36_vram_refusal.py: demanding starts (8 slots, long contexts, every expert as a slot) must reach READY or exit 1
with one of the engine's refusals, never a raw allocation failure.
--pipeline-windows runs its own overlapped decode loop with its own hand-offs, so
STRATA_SPLIT_OVERLAP now switches itself off when it is set (one log line says so).
Rebased onto main 1735d64: the early launch of the next stage keeps #871's graph
variant (the next stage's refresh_ar() runs before its capture and prelaunch, and
not again in its run() when it was prelaunched), resets the plan error word like
run()'s launch, and the chain path checks that word after its sync as well.
Review on #936 (justxiami): the reading stage's wait_flag_ge spun with no bound,
so a lost publish would leave that stream spinning.
- wait_handoff: the reading stage's graph now waits with a bound (default 30 s,
STRATA_SPLIT_WAIT_MS, 0 = unbounded, GPU clock as in gpu_stamp). On timeout it
sets a mapped error word and returns, so the graph completes; the host checks
the word after the window and fails it with an error that names the setting,
instead of using a stale hand-off.
- The host's per-layer wait (20 s, #267) still normally ends a stalled window
first; release_gpu_waits already raises the hand-in flag.
- Test hook STRATA_TEST_HANDOFF_DROP=N: the first stage's N-th window withholds
its flag (a lost publish), to exercise both paths.
Only active with STRATA_SPLIT_OVERLAP=1; the default path is unchanged.
Each verify window of a layer split runs its stages strictly in turn: stage 1 finishes, the host syncs its stream,
then stages and launches stage 2's graph. Under HIP the sync and the graph launch both sit on the critical path of
every window.
With STRATA_SPLIT_OVERLAP=1 the hand-off carries a ready word (mapped, one per hand-off buffer):
- the writing stage's graph ends with handoff_publish: the payload with volatile stores (RDNA keeps plain stores to
mapped memory in L2), a system fence per thread, and the last block raises the flag;
- the reading stage's graph starts with wait_flag_ge(flag, 1);
- Verifier::run captures the next stage's window on this thread, then stages and launches it on a helper thread
(prelaunch) while this thread serves its own layers; it then serves the next stage's layers directly and syncs
the earlier stage once the chain is done;
- release_gpu_waits also raises the hand-in flag; an early return after the prelaunch releases the next stage.
Batch windows and --split-device 0 keep the serial order. Off by default.
Measured on 2x RX 7900 XTX (ROCm, Linux), Swift 1.5 IQ3_XXS, 512-token greedy answers, median of 8:
decode 53 -> 70.1 tok/s (code prompt), 45 -> 55.9 tok/s (prose prompt); greedy output byte-identical with the
flag at 0 and 1; quiz 47/51 both ways. Not measured on NVIDIA.
Split-overlap launch merged with 0.1.44 HostGraphLaunch (STRATA_VERIFY_ASYNC_LAUNCH): the launcher object is
kept in an optional so an early-launched window has none.
Co-authored-by: bsvinay <[email protected]>
Squash of SuperDistance's three commits of PR #2050 (decode between the pieces of a read on the pipelined groups, the
piece lines counting the rows the slots decoded meanwhile, the stall watchdog staying on for a read in pieces),
applied after #2326 here: the BYIELD give-way is #2326's give_way lambda, the concurrent block runs first and the
piece path serves a read that STRATA_PIPED_CONCURRENT leaves to it (pictures, configurations where it is off).
Co-authored-by: SuperDistance <[email protected]>
On Linux a thread inherits its creator's CPU mask, and the host thread is pinned to the one core the pool leaves it
(aux_cpus.hpp). The read's thread and the doorbell thread were created from it, so they shared that core with the host
thread that pumps the windows, and every window waited there for the doorbell of each layer: 6.1 rows/s during a read
on 4x RTX 5080, 4 stages (blange48 on #2326). Windows places the host with CPU Sets, which new threads do not inherit,
so it did not show there. Both threads now move off the host's core and the pool workers' CPUs (on an SMT machine with
the default pool: the idle sibling threads; without such CPUs, every CPU but the host's), and the threads that already
run on the host's CPU (the prompt path's copy threads, started by the first prompt) move the same way at a read.
--aux-cpus places them all when it is on. STRATA_PIPED_CONCURRENT_UNPIN=0 keeps the inherited CPUs (an A/B).
Measured on Linux (WSL2, Ubuntu 24.04, 2x RTX 5090, 4 streams, a cold 32K prompt; same binary, UNPIN=0 / default): the
slots decoded 25.9 and 23.7 rows/s during the read with the threads kept on the host's CPU (76-81 threads there), and
75.6 and 62.8 with this change; with the n-gram table in RAM (WSL's disk reads it 7x slower than Windows) 110.7 rows/s,
as on Windows (107.0). piped_read_join_bench.py (cold32k, followup, two_joiners; table in RAM, 2-3 runs each): 24-37
rows/s during the 30K+ reads kept on the host's CPU, 95-127 with this change, and 925-1,065 stream tokens during the
cold 32K join against 454-484. On Windows, STRATA_TEST_CONC_ON_HOST_CPU=1 (a test) puts both threads on the host
thread's CPU set first: 18.6 rows/s kept there, 105.4 with this change.
The flag works only through a helper cache (--expert-cache-device1..3), and on a plain layer split it is inert
(MULTI_GPU.md, #1447), so the check looks at the helper caches themselves. #2275's rig runs a plain split with the
flag set, and the mode would have stayed off there for a cache it does not have.
The opt-in's section: how the read and the windows share the GPUs, the doorbell thread, the give-way at a chunk, where
the mode is off. Measured against main (a0116b27) on 2x RTX 5090 in one window (main, this, this, main): the stops,
the streams' rate during the reads with the same answers every run, the joining prompts' TTFT, a question during a
read, the same randomised load on both, the doorbell thread's first wait as an A/B; 16 GB cards on each side; a weaker
host as well in #1906's shape; the default path's check; text equality.
tools/piped_read_join_bench.py, the client that measured it, is #2050's, unchanged (the same file as there).
From a review of the concurrent read:
- With external serving batch_poll still finds a window that will not finish (its graph ended or failed with a layer
to ring, or no layer rang for 20 s): the served count is read before the stream query and the ring count after it,
at most one query a millisecond; the last ring's time is an atomic. batch_serve reads a window's row count after its
served count, so a window launched while the doorbell thread was descheduled is not served with the last one's rows.
- #1996's pinned upload rings (0.1.43.1) grew a slot at its first larger upload, a pinned free and allocation on the
read's thread: the drafter's records and the prompt path's routing tables reserve their slots before the read
(PinnedRing::reserve, Prefill::reserve_uploads).
- Off, said once, where serving a doorbell makes a CUDA call (--pcie-mode dma, STRATA_FS_SLOTS, the helper caches) or
the expert source recycles the buffers it hands out (the GGUF in place, unbuffered, STRATA_IO_PREFETCH); a prompt
with pictures is read the usual way. The expert sources' read counters are atomics.
The check is a pure function, concurrent_read_off (batch_read.hpp), with a test in batch_read_test.
- pump's all-idle branch turned the stall watchdog off when the last slot finished mid-read; it stays on while a prompt
is read until the request ends.
- Between windows the doorbell thread slept 50 us, a whole timer tick on Windows (up to 15.6 ms): it now waits on a
condition each launch notifies (STRATA_TEST_BELL_SLEEP_US=N, a test, restores the sleep for an A/B). With nothing
decoding the main thread waits for the read's end or a line instead of sleeping. A pool failure the doorbell thread
set is reported before pump clears the flag for the next window.
- The read's log line ends with the slots' rate and, per stage, its windows and the host time that set each up. The
priority stream's cudaGetLastError runs only under the opt-in.
With a layer split and --batch-groups > 1 a prompt read is one sp.run, so every decoding slot stops for the whole read
(#1906); #2050's pieces bound the stop but drain the pipeline at each piece. With STRATA_PIPED_CONCURRENT=1 the read
runs on a worker thread while the main thread keeps the groups going through the pipeline (pump) and takes BSTOP and
BYIELD meanwhile. The prompt's K/V goes to the main sessions and the slots' to their own; the windows' streams get the
device's highest priority on CUDA (STRATA_PIPED_CONCURRENT_PRIO=0 keeps the default).
- The slots' doorbells are served on a thread of their own that makes no CUDA call (Verifier::batch_serve, now
batch_poll's per-layer service too, and set_external_serve). Two hangs found while building this, read from the
stall dumps against the PDB, had #31's shape: the main thread waited for the driver's lock in cudaSetDevice while its
holder waited for the GPU (a cudaFree in the drafter's prompt pass; cuBLASLt's lazy load of a GEMM module), and a
window on that GPU waited for its doorbell.
- MtpDrafter::reserve_prefill captures the drafter's prompt graphs and sizes its records before the read.
- Prefill::stop_at_chunk: a BYIELD ends the run at its next chunk, also with no slot decoding; the pipeline is drained
there and the part read gives way as #656 does. A read nobody waits for is not cut.
- The reader is joined on every way out (an exception too). Off beside STRATA_PREFILL_HELP and STRATA_SPLIT_MTP_BATCH,
which allocate during a read.
The prompt path gathers a chunk's PLE rows on its own thread and the batch windows gather theirs on the main thread,
from the same table (ss.ple.table). Today they never overlap - a prompt run's read-ahead ends with the run, and the
windows run between runs - but a prompt read beside running windows would race on the reader's tickets and the
prefetch slots. Every public method that touches them now holds a recursive mutex (gather holds it across its issue
and collect). Uncontended it is a few nanoseconds per call; nothing else changes.
Squash of ruibeikaa's six commits of PR #1441 (the first fixed b/a, then runtime-learned b/a, then calibrated b/a, the
segment planned once, --prefill-pipe 1 following STRATA_PREFILL_EQUAL), applied onto 0.1.44.1 with the conflicts in
generate.cpp (Options, BSlot, the lend's chunk with --prefill-head-alias) resolved.
Co-authored-by: ruibeikaa <[email protected]>
A bigger chunk is only taken when the experts, the 0.77 GiB reserve and the chunk's buffers (0.5 MiB per token) still fit without giving the packed
decode copies back: 131072 cells -> 512, 100000 -> 2048, 32768 with --batch 4 -> 1024, --batch 8 -> 512 (hybrid as before).
Prompt chunk (owner decision): with every expert in VRAM the auto rule picks 4096, 2048 or 1024 (the chunk's buffers cost ~0.5 MiB per token,
the sessions' caches 45 KB per cell and session; 1024 at most with --batch N so the other slots' windows are not held for a whole chunk), else 512.
The hybrid mode keeps 2048 and falls back to 1024 / 512 on a card with little room. STRATA_Q36_CHUNK=512 or --prefill 512 gives the old size.
R9700 pp4096: 512 3390, 1024 4495, 2048 5353, 4096 5750 tok/s. Not bit-identical to 512 (other GEMM tiles): see docs/QWEN36.md for the KL.
Decode batches of up to 8 rows take the router and the shared expert's gate in one product (rows of 257), as a single token already did.
k_conv_seq's tile mode switched off the history write for batch rows too (grid y is the group count there): fixed.
k_gdn_seq_cols started every token with a chain of global loads; they are issued one token ahead now. Same arithmetic: the 3000-token
prompt's logits and the next 8 greedy steps are bit-identical to the previous engine at chunk 512 and 2048 (R9700: prompt 931 -> 902 ms
at chunk 512 together with the tiled conv, 616 -> 591 ms at 2048).
q36_kernels.cuh: one warp per row for the shared expert's and attention k/v Q8_0 products (grid capped at 1024 warps: 2048 rows as 2048
warps took 21 us, 5 us as 1024), shared gate+up+SwiGLU+q8_1 in one launch, k_moe_out with 1024 threads and the next layer's q8_1 image, k_add_rms
with the q8_1 image, router top-k and expert grouping in one launch. CUDA: the division / exp the --use_fast_math TUs they mirror use are spelled out.
Engine: prompt GDN conv in tiles of 16 tokens side by side (same arithmetic per token), --draft-vocab N (the MTP draft head over the first N rows only,
a draft proposes and the verify window decides: output unchanged), the packed Q8_0 default on Windows too (it was Linux only), STRATA_Q36_OLD_KERNELS=1
switches all of it off.
R9700, UD-Q4_K_XL: decode 82.5 -> 120.6 tok/s (greedy 10 prompts identical), --mtp 2 160 -> 174.
For that geometry only: the s27 gate/up kernel and the fused SwiGLU/q8_1, then the old down kernel (the new down kernel is slower at n_ff 512).
Flash-Next's shapes (2560/640) take the same paths as before; the path is still STRATA_EXPERT_V2K (default on gfx12).
The same kernel measured on an R9700 (gfx1201): qsa_scores_gfx1100_parity --full compares 223,664,777 scores and 11,895,800 ids bitwise equal
to the warp scorer (35 cases), the scorer stage with 256 queries is 3.6x faster at 131K and 4.0x at 240K, and a fresh prompt reads 8.7% faster at
131K tokens (4 of 4 pairs) and 16.4% faster at 230K (3 of 3), with identical answer tokens and byte-identical exact_pp residual dumps at 70K and 131K
tokens. So on gfx12 it is on for batches of >= 64 queries with >= 16,385 live blocks (65K cells and more) unless STRATA_SCORES_THREAD=0; gfx1100
stays the author's opt-in (STRATA_SCORES_THREAD=1). A one-time stderr line says when it engages by default.
The persistent read's barrier timeout uses __builtin_amdgcn_s_memrealtime, which gfx11 and gfx12 do not have (clang: "needs target feature
s-memrealtime"): the PR as ported failed to build for gfx1100, gfx1151 and gfx1200/gfx1201. On those parts the barrier compiles to an immediate
abort, fused_gr_persist_init refuses the read with a start-up line, and the default read is used. An R9700 (gfx1201) measured the read +0.7% / +1.0%
per window slower than the default in the PR thread, so nothing is lost. The silent, permanent barrier abort at run time that the same thread
describes is noted in docs/AMD_HIP.md.
The verifier reads the hyper-connection twice per layer at 1-8 tokens: three small launches with two global
dependencies, the down projection on 41 blocks. Opt-in STRATA_HC_PERSIST=1 runs it as one kernel of 80 blocks with two
grid barriers on an atomic counter: the down projection split by column slices over all 80 blocks, a distributed
reduction, then the up projection and the same epilogue as gr_up_kernel.
gfx1030 (2x RX 6900 XT, layer split): the read 1.35-2.10x at 1-8 tokens; ms per verify window -4.2 % at 4K and
-2.8 % at 32K (one binary, the variable unset / set, 6 alternating rounds, every round the same way). Another summation order (not bitwise); against a
double-accumulation reference (STRATA_HC_PERSIST=3) as close as the default read.
Residency is decided by running every token count once at start-up with a 50 ms barrier timeout and comparing with
the default read; a timeout keeps the default read for that token count and larger ones, wave64 cards are refused.
One line per card says what was decided. Only the verifier's reads use it (one stream per card); the drafter's reads,
the prompt path, CUDA and SYCL are unchanged (CUDA and SYCL get two empty stubs). With the variable unset both entry
points return in their first line: the output is bitwise the same as main.
Measurements, the simulated smaller cards and a stand-alone timing tool:
bench/results/2026-10-12-hc-persist-gfx1030/.
block_topk_wide_kernel is plain shared-memory / atomicAdd code with no sm_90
feature, but its definition and the dispatch branch sat behind
#if !defined(__HIPCC__), so on AMD a context past the register kernel's reach
(about 135K cells) always fell back to the 4-pass ref radix. The ids are
identical by construction (the same radix rule), which the bench states:
256/256 at 256 queries x 65537 blocks, 1/1 at nq=1.
Measured on a Ryzen AI Max+ 395 (gfx1151, ROCm 10.1), qsa_select_bench:
- nq=1, 131073 blocks (a decode window at 512K context): 2.24 -> 0.77 ms (2.9x)
- nq=256, 65537 blocks (a prompt batch at 262K): 3.16 -> 1.43 ms (2.2x)
End to end this moved the deep-context decode of a 512K-token prompt from
19-22 to 24-26 tok/s and the 512K prefill from 473 to 634 tok/s on that PC
(together with the split scorer of another change). STRATA_TOPK_STREAM=0
still restores the register kernel.
Upstream aa7b0cb gates PcieAuto to CUDA single-GPU (#if !defined(STRATA_USE_HIP)); the
machinery itself is engine-generic (pool ms per missed expert + miss counts vs the probed
link rate), and this build's start-up H2D probe is already best-of-4 with a +-0.7% spread,
so the HIP exclusion was unmeasured-only, the same class as the ROUTE_TAIL_SKIP gate.
Same opt-outs as CUDA: layer splits excluded, STRATA_PCIE_FRAC_DEFAULT=old restores the
fixed start-up rule. Provenance: MistVVK/XeStrata commits 05309096/055cf9b0/c71cac34/
5952e8e7 (their measurement-stabilization cluster) demonstrated the failure mode and the
fix class; their exact subsystem is not portable (it is their own engine code).
(cherry picked from commit acb3deb0522a8fc351a4047d2bd11b8ab7a73dca)
Co-authored-by: Zac Chen <[email protected]>
The engine turns 18 exact (bit-identical) switches on by default on gfx1151
(src/core/arch_defaults.cpp) but keeps gfx1150 out of its arch defaults
(src/core/arch_defaults_test.cpp asserts arch_default_env("gfx1150").empty()).
A gfx1150 install gets the 17 that apply here - the gfx1151 set less
STRATA_HCD_EXACT, which copies the gfx1151 hipBLASLt table's solutions 1176 / 1177
and so does not fit beside a gfx1150 table - written into the config's "env"
beside STRATA_HIPBLASLT_TUNING. The Radeon 890M report (docs/AMD_HIP.md;
bench/results/2026-10-07-community-gfx1150) measured the same greedy answers with
all 17 on and with all 17 off, decode +2% and prompts +3-6%.
The set is setup's own (SETUP_ENV): a re-run refreshes it, and it does not linger
after the opt-in is turned off. A new test (tools/test_setup_strix_halo.py,
Gfx1150Switches) keeps GFX1150_SWITCHES equal to the gfx1151 arch-default table
minus STRATA_HCD_EXACT, keeps the engine's own default path for gfx1150 empty, and
checks the SETUP_ENV wiring. Setup's suites pass (strix_halo 40, amd 36, choices 47).
Co-authored-by: onionbutties <[email protected]>
STRATA_EXPERIMENTAL_GFX1150=1 takes the Radeon 890M / 880M iGPU (Ryzen AI 300,
Strix Point, gfx1150) as a unified-memory card like Strix Halo: it joins AMD_ARCHS
and the UMA handling (new is_uma_apu helper), with notes of its own. The engine
already builds and runs for gfx1150 (0.1.40.2, #1217; tools/hip/gfx1150-hipblaslt-100401.txt
is calibrated on it); no release ships a ready-made Windows engine for it yet, so
a Windows user builds one (STRATA_HIP_ARCHS=gfx1150 tools\hip\build_windows.bat) and
starts setup with --prebuilt dist\. Unset, the default path is unchanged: the test
that asserts gfx1150 is not in AMD_ARCHS (tools/test_setup_strix_halo.py) still passes.
Co-authored-by: onionbutties <[email protected]>
The engine turns 18 exact (bit-identical) switches on by default on gfx1151 (src/core/arch_defaults.cpp) but keeps
gfx1103 out of its arch defaults (src/core/arch_defaults_test.cpp asserts arch_default_env("gfx1103").empty()). A
gfx1103 install gets the 17 that apply here - the gfx1151 set less STRATA_HCD_EXACT, which copies the gfx1151
hipBLASLt table's solutions 1176 / 1177 and so does not fit beside a gfx1103 table - written into the config's
"env" beside STRATA_HIPBLASLT_TUNING. One 780M (Ryzen 7 8745HS, 60 GB, GTT 50 GiB, ROCm 10.1, Q2_0, 4K prompt)
measured the same greedy token ids with the 17 on and with them off, +13% to +15% prompt and decode +3.2% (the
decode half alone); bench/results/2026-10-12-community-gfx1103.
The set is setup's own (SETUP_ENV): a re-run refreshes it, and it does not linger after the opt-in is turned off.
The gfx1103 note now names the native dot4 / WMMA kernels and docs/RADEON_780M.md, which documents the opt-in
build, the table and the measured numbers. Setup's suites pass (strix_halo, rocm_pins, amd, choices: 143 tests).
Refs #1423.
Co-authored-by: AlexisCarbonell <[email protected]>
Calibrated on a Radeon 890M (Strix Point, gfx1150) with
tools/hip/tune_hipblaslt --workspace-mib 32 over the 16 dense GEMM geometries of
the shipped gfx1150-hipblaslt-100401.txt at T = 64, 128, ..., 16384: two passes,
each row keeping the solution with the lowest mean of all six repetitions. The
three rows slower than plain hipBLAS (the tiny bf16 N=1 K=2560 shape at T = 64,
128 and 256) are left out, so those buckets route to the nearest good row. Every
remaining row beats plain hipBLAS: 1.07x to 16.7x per GEMM, geometric mean 5.0x.
The solution IDs are scoped to hipBLASLt 1.5.0 (version 100500, the ROCm
10.2.0a20260930 nightly the engine here was built with).
The Radeon 780M (gfx1103, RDNA3, 12 CUs) builds and runs (issue #1423), but had no hipBLASLt table, so the prompt's
dense GEMMs ran on plain hipBLAS. This adds tools/hip/gfx1103-hipblaslt-100401.txt, calibrated on one 780M with
ROCm 10.1's hipBLASLt 1.4.1 (version number 100401) over the gfx1150 table's 144 GEMM cases at T = 64..16384 with
the engine's 32 MiB workspace.
Two tune_hipblaslt passes back to back; each row keeps the solution with the lowest median of all six repetitions,
and a row is left out when no candidate beats plain hipBLAS. 143 rows, 1.05-23.30x per GEMM, geometric mean 4.95x;
the one shape left out is bf16 N=1 K=2560 T=64 (best 0.039 ms against 0.024 ms on plain hipBLAS, sub-100 us
timings), where the engine falls back to hipBLASEx.
Measured on the 780M (Ryzen 7 8745HS, 60 GB, GTT 50 GiB, Q2_0 + MTP, 32K context): a 4K prompt read 90 tok/s on
plain hipBLAS against 187 with the table (medians of an interleaved A/B; with the gfx1103 exact switches and the
WMMA opt-ins, 277 at 4K and 272 at 32K). hip_prefill_hipblaslt_gemm passes with it (launches=4 fallbacks=0) and
hip_prefill_hcd_exact_parity skips, because the table picks no 1176 / 1177 for the HC down GEMM - the gfx1150 table
gets the same skip, which is why STRATA_HCD_EXACT is not in the gfx1103 switch set. Setup installs for gfx1103 only
with STRATA_EXPERIMENTAL_GFX1103=1 and then picks the table up when the installed hipBLASLt reports 1.4.1.
Docs, the community report (runs.csv, tune-summary.csv, the server config, the engine's start-up lines) and the
setup tables list carry the numbers. Refs #1423.
The switch exists for CUDA (#1372) and is compiled out on HIP. On gfx11 it now runs the WY form of the gated delta rule in
chunks of 16 tokens, every product one 16x16x16 WMMA (FP16 operands, FP32 accumulation, the state in the accumulators), with the
output norm and the z gate fused: a prep kernel for T and P of every chunk, then a scan with a block per value head.
It rounds differently from the FP32 recurrence, so it is opt-in; the default path is unchanged. A gfx1100, gfx1101, gfx1102, gfx1150 or gfx1151 part, 64 tokens and
more, an output row stride divisible by 8 and no STRATA_GDN_REC_HEADS take it; anything else runs the existing kernels.
tests/hip/gdn_chunked.cpp: bounds on the FP16 output and the carried state against the existing recurrence over 0-16,385 tokens
(chunk and stride edges, a split request), a double-precision CPU check, and bit-identical output with the switch off or unset.
tests/hip/gdn_rec_head.cpp unsets the switch, as its variants 0 and 1 are the references there. hip_gdn_chunked --bench and
--bench-default time this path and gfx1151's default kernels, one per process (the switch is read once).
gfx1151 (Ryzen AI Max+ 395, UD-IQ4_XS): a layer's recurrence + norm takes 1.60x less time than the default kernels at 16K and
32K tokens; a whole prompt 1.4-2.1% less at 4K-128K after a warm-up prompt (11 pairs each: 1.8-2.1% with the docs/STRIX_HALO.md section-5 switches,
1.4-1.5% without them, 10 or 11 of 11 faster); first-token KL against the default 0.006-0.007 on average, the same top token in
12 of 12 prompts. Run on gfx1151 only; kernels.cu compiles for gfx1030, 1100, 1151, 1152 and 1201 (docs/AMD_HIP.md).
Co-authored-by: libratechw <[email protected]>
Q8_0 has no minimum term and the F16 kernel's output already differs from the int8 kernels', so the KQ opt-in is
not needed for it. UD-IQ4_XS has five Q8_0-down layers (4 IQ3_S / Q8_0, 1 IQ4_XS / Q8_0): they left the fused path
for MMQ, and since fused_ring() asks every layer, the pack kept MMQ's buffers. d_covered_on() is the one test, for
native_supported() and experts_native(); gfx12 keeps d_covered().
UD-IQ4_XS, gfx1151, 3 interleaved rounds, prompt tok/s (v0.1.42 / 65d809c6 with STRATA_PF_F16=1 / this):
4K 926.6 / 992.2 / 1054.8, 32K 1257.4 / 1364.6 / 1397.1, 128K 1202.0 / 1309.4 / 1333.6.
native_f16_kernel takes the IQ gate/up formats (IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS) and the Q2_0, IQ4_NL
and Q8_0 down: load_unit / convert decode each sub-block once into an int8 weight tile in LDS with its scales as
half2, each wave turns its own rows into F16 (gufo's Q8_0 decode: sign flip, byte permute into 1024's mantissa,
one packed multiply), and the F16 WMMA accumulates all of K. The work item, the activations staged as F16, the WMMA
loop and both epilogues move to moe_f16_common.cuh, shared with the K-quant kernels (moe_f16_kq.cu, now on it; its
output bits are unchanged). The switch becomes STRATA_PF_F16=1 (was STRATA_PF_F16_KQ) and covers both; the K-quants
still need STRATA_PF_FUSED_KQ=1. gfx12 keeps the int8 kernels.
hip_prefill_fused_iq (gfx1151, ROCm 7.14.1 clang 23, top 10 over 512 experts, one 128-expert launch, ms):
- IQ3_S / IQ4_NL at 16384 tokens: gate/up 11.65 -> 9.87, down 6.11 -> 4.71; layer at 8192 37.3 -> 32.2.
- every pair is faster at 4096 and 16384 (IQ2_S gate/up 14.38 -> 9.85, Q8_0 down 6.80 -> 4.79), and at 128-2048
tokens equal or faster.
- FP32 reference rel RMS as the int8 kernels' (IQ3_S / IQ4_NL 1.456e-2 against 1.454e-2); against the int8 model
and MMQ the F16 path sits at 2.4-2.7e-3, so both checks allow 5e-3 under STRATA_PF_F16=1 (f16_on).
- With the weights decoded to F16 in LDS (18 KB a block) gate/up was 9.83 and down 5.71. Neither a VGPR cap nor
dropping the sign table (4 blocks per WGP) nor decoding after the products moved gate/up; without the decode it is
8.98, without the weight loads 9.30.
Off by default (the output differs from the int8 kernels').
moe_f16_kq.cu's bodies are gfx11's WMMA layout (and trap on other targets); v0.1.42's gfx12 native kernels share
experts_native's launch, so the F16 path is taken only off gfx12.
down_kernel takes gate/up's structure for UD-Q4_K_XL's Q5_1 down (47 of its layers): the codes and high-bit words
packed in LDS, w = q d + m per wave in F16, H staged once as F16 from its int8 blocks, FP32 outputs at the routed
rows. The activation staging and the WMMA loop are shared with gate/up (ActStage, mma_tiles). Q8_0 down stays on
native_w11_kernel.
hip_prefill_fused_iq (gfx1151, one layer, 16384 tokens x top 10 over 512 experts): down 7.10 -> 4.83 ms per
128-expert launch; Q4_K / Q5_1 layer 62.7 -> 53.5 ms (85.3 on the int8 kernels); FP32 reference rel RMS 1.304e-2.
UD-Q4_K_XL's routed gate/up was 6/10 of the prompt-time gap to gufo on gfx1151 (128K: 23.1 s against 13.9 s).
moe_f16_kq.cu adapts gufo's paired RoutedF16GEMMKernel (97b8645f, MIT) to Strata's tables, int8 activations and H:
the Q4_K / Q5_K codes stay packed in LDS and each wave turns its own rows into F16 after the read, so the F16 WMMA
accumulates all of K in F32 with no per-block epilogue; the activations are staged once in LDS as F16 for every wave;
the epilogue writes H as int8 per 32 features for the existing down kernels. One block per work item; two tiles of
an expert per item from 64 routed rows per expert on average.
hip_prefill_fused_iq (gfx1151, ROCm 7.14.1 clang 23, one layer, 16384 tokens x top 10 over 512 experts):
- gate/up kernel 14.63 -> 8.46 ms per 128-expert launch; Q4_K / Q5_1 layer 85.3 -> 62.7 ms, Q5_K / Q5_1 89.9 -> 66.2.
- against the FP32 reference the error is the int8 path's (rel RMS 1.305e-2 / 1.302e-2: the int8 activations
dominate); the int8 model check allows 5e-3 under this switch (2.5e-3 measured with Q8_0 down).
- the output bits are the same with 64- and 128-row items.
Off by default; it needs STRATA_PF_FUSED=1 and STRATA_PF_FUSED_KQ=1, whose output already differs from MMQ's.
After a prompt the borrowed expert-cache slots are refilled from the experts' files. With the GGUF read in place an
expert is three tensors, and the refill first assembled each 4 MB blob in a staging buffer (a copy on the calling thread)
and then copied that to the slot. The pieces are contiguous in the mapping, so they can go straight to their offsets in
the slot: blob_parts() hands them out, ExpertCache::fill_slot_queued_parts copies them on the same queue. The bytes in
the slot are the same. Off by default; gfx1151's table turns it on.
STRATA_CPU_IQ_ALL_EXACT1 reproduces ggml-cpu's AVX2 single-token dot bit for bit, which only
holds if the ggml-cpu in this binary is that dot: a STRATA_ISA_FLOOR build runs ggml's AVX1/SSE
dots, and a native build on an AVX-512 CPU run with STRATA_FORCE_AVX2=1 runs its AVX-512 one.
The guard is #2215's (ggml_cpu_has_avx2 / ggml_cpu_has_avx512).
Co-authored-by: Andy Slater <[email protected]>
Three changes for two-card AMD PCs whose faster card does not come first in the
runtime's enumeration (Windows HIP orders the cards itself; HIP_VISIBLE_DEVICES
only filters):
- gfx1032 (RX 6600) joins the unvalidated build list - it shares the gfx1030 /
gfx1031 / gfx1034 dp4a path.
- STRATA_PRIMARY_DEVICE=N picks the visible ordinal the engine's own session
runs on; it is set before any context or allocation exists. The helper
caches take the other visible devices, the HIP arch check and the peer tier's
device queries follow it, and the primary's session is created on the chosen
device. A layer split still assumes device 0 and is refused with this set.
Unset or 0 is upstream's behavior unchanged.
- The helper-GPU caches are allowed beside the resident RAM mode: the experts a
helper holds are left out of the RAM copy as a split stage's are, and the two
adaptive tiers already keep each other's experts out. RemoteExperts::finish
spins on its stream like PeerExperts::finish - on HIP the CUDA branch's
cudaInitDevice(ScheduleSpin) does not exist.
Measured on an RX 6800 (16 GB, gfx1030) + RX 6600 (8 GB, gfx1032), Windows 11,
32 GB of RAM, Swift 1.5 IQ3_XXS at 160K context, the resident RAM mode with the
helper's cache left out of the copy: decode 37.5 -> 44.2 tok/s on 8K-token
requests with a 3800-slot helper; the CPU pool's share of the routed experts
falls from ~2.9 to ~0.4 entries per layer-window.
Rebased onto 296b59d2 (conflicts in generate.cpp, peer_experts.cpp, hip_backend.cmake, docs/AMD_HIP.md resolved by keeping main's code and adding the PR's). Changes: remote_caches is kept (still used by the --batch-mtp check); the spin in RemoteExperts::finish is HIP only (CUDA keeps its cudaInitDevice(ScheduleSpin) + blocking sync). NOT COMPILED YET, not run.
Co-authored-by: tuandat3019 <[email protected]>
The menu never hid NVFP4 on an AVX2 CPU; the engine runs a native pack's rows on AVX2 kernels that are bit-equal to
ggml's. The NVFP4 menu line and the install step now say, on a CPU with AVX2 and no AVX-512, that the CPU's share of
the experts runs on AVX2 rows (measured about as fast when memory-bound, up to ~12% slower on 4-token windows).
A note only: nothing is refused or capped. tools/test_setup_menu.py: 2 tests.
The fork stays off by default on HIP (#816, #826): an RX 6800 on Windows and 4x R9700 decode 12-45% faster without it.
The same card family on another PC goes the other way (#1272: 2x RX 6900 XT on Linux), and the output is identical
either way, so it is a per-PC choice - the open item of #1272.
- tools/calibrate.py: step 6, HIP only. With everything chosen before, the engine is started twice, with
STRATA_SH_STREAM=0 and =1 set explicitly, two rates each; "1" is kept only when it beats "0" by more than MIN_GAIN
(the rule of the other steps). measure() returns the choice as "env"; a failed start drops that candidate (#1337).
start_engine takes an optional env; apply_env() sets the chosen variables and removes the others of ENV_KEYS, so an
older calibration's choice does not linger.
- setup.py: --calibrate writes "env" into the config and the settings file; a reinstall applies a saved calibration's
"env" the same way (a calibration saved before this step leaves "env" as it is).
- docs/AMD_HIP.md: the SH_STREAM note says calibrate measures it.
- tools/test_calibrate.py: the step runs on HIP only, keeps the fork only on a win, falls back on a failed start;
apply_env; setup writes and later removes the variable.
Measured with the real engine (0.1.42 tag, 2x RX 6900 XT gfx1030, Linux, IQ3_S layer split, setup-style config): the
whole calibration took 982 s, the new step chose STRATA_SH_STREAM=1 (off 62.2 / 61.3, on 67.2 / 66.6 tok/s, +8.3%).
Separately, with a fixed protocol, the fork's output is byte-identical to off (4 of 4 requests, one card and the split).
The scorer of 65ce329 is plain FP32 arithmetic, so it runs on any HIP card as it is; without matrix cores the
prompt path otherwise scores on the warp kernel. Its tiling becomes template parameters: every tiling keeps each
score's fmaf chain over k = 0..127 in order and the same relu sum over the heads, so every tiling gives the same
bytes (RX 6900 XT, qsa_select_bench at 32K and 128K: 14 tilings, also on 60 and 72 of its 80 CUs through
HSA_CU_MASK, all byte-identical to the original kernel; the fastest tiling the same at every width). CUDA keeps
the tiling it was written with; gfx103x runs 32/128/16/8: 256 queries at 128K 2.69 ms (warp kernel) -> 0.695 ms
(the original tiling) -> 0.587 ms. FP64 gate passed, selections identical 256/256. On the 2x RX 6900 XT layer
split a 128K-token prompt reads 7.4% faster, 32K 0.8%. A HIP ctest runs it at 131K.
Rebased onto 296b59d2: the docs/AMD_HIP.md conflict (main's SGEMM/rocBLAS-table bullets from the ports before it) is resolved by keeping all bullets.
Co-authored-by: xjc10 <[email protected]>
rocBLAS picks the kernel for each FP16 product from its own table by shape, and on gfx1030 that pick is far
from the best kernel the same library holds for some shapes: N = 10240 K = 2560 (the GDN input projection)
below T = 1152 runs at 5.2 TFLOPS where the best of rocBLAS's 237 solutions for n = 799 is 6x faster (not an
alignment effect: n = 800 is as slow as 799).
tools/hip/tune_rocblas enumerates the library's solutions per shape and token bucket, checks each against the
default kernel's output and that it also runs at smaller odd token counts, and writes the winners (and
`default` rows bounding their reach) as tools/hip/<arch>-rocblas-<version>.txt; Gemm::f16_inplace runs the
table's solution for a shape (STRATA_ROCBLAS_TUNING: a file, or the directory from which the engine takes the
file of its own rocBLAS build; the version string, tweak hash included, is checked). One tiny default product
at startup opens rocBLAS's kernel library (a solution-index call before that fails although the index is
valid); a refused solution falls back to the default kernel for that call, after three refusals for good.
setup points the engine at tools/hip when a table for the card's architecture exists. Shipped: the gfx1030
table for ROCm 10.0.0's rocBLAS 5.6.0.8d1ae90e (112 rows, 76 with a solution).
Measured (RX 6900 XT, 6f32ec0 + #835 + this, temperature 0, the same answers with and without): a 799-token
prompt's GPU time 3,329 -> 2,985 ms (-10.3%), 495 tokens 2,477 -> 2,139 (-13.6%), 8,874 tokens 11,978 ->
11,459 (-4.3%); a PCIe-bound 1,108-token prompt unchanged (its saving reappeared as `wait copy`). ctest 66 of
70 (the 4 failures are outside this change); hip_prefill_rocblas_gemm, the new smoke test, passes with 0
fallbacks. Builds on #835. bench/results/2026-10-05-rdna2-rocblas-table has the per-run lines and the tuner's
summary.
(cherry picked from commit e35e103604)
Rebased onto 296b59d2: conflicts in setup.py (SETUP_ENV gained STRATA_ROCBLAS_TUNING beside the 0.1.44 entries) and docs/AMD_HIP.md (kept both sections) resolved.
Co-authored-by: xjc10 <[email protected]>
The rebased #704 under the name asked for there (upstream's STRATA_HC_Q8 now
reads the GGUF's Q8_0 bytes, a different thing). Opt-in. The four BF16 hc_*
projections per layer (1,200 MiB) are skipped by the loader and uploaded as
int8 codes with one fp32 scale per 32 values (675 MiB); the freed VRAM goes to
the expert cache.
- Q8 is a template parameter of the down/up kernels (single- and multi-token):
the BF16 instantiations compile exactly as before, the launch sites pick the
Q8 ones when w_down / w_up carry scales. The inject rows stay BF16.
- The staged / v3 / upmix / PF_HCDOWN / gfx906 split paths read BF16 rows and
are skipped for it; the prompt path expands the codes into a bf16 scratch for
its GEMM.
- The loader seeks with 64-bit offsets.
- A layer split keeps BF16 with a warning instead of exiting.
Fix-up to the ported #1123 commits. The route is not bit-identical to the native call
(the author measured at most 4.3e-6 of the largest output apart), so unset keeps the native
path on every card, gfx103x included; =1 enables it on gfx103x devices only (a mixed layer
split keeps the native call on the other cards), =all on every AMD device for A/B arms and
tests. The buffers are no longer freed under an in-flight GEMM: the stream is synchronised
before the FP32 copies grow, as the sm_60 FP32 route does. docs/AMD_HIP.md says so.
Port of PR #1838 (commit fb868db01) onto 0.1.44.1. Test file merged with #2134's foresight test (both kept). replace_resident now drops a Foresight slot for the outgoing expert while its RAM place is rewritten (#2134's copies read those places).
Co-authored-by: SanghunYun <[email protected]>
Rebased onto 296b59d2: the docs/AMD_HIP.md conflict (the gfx1031/gfx1150/gfx1152 bullets main gained) is resolved by keeping main's text.
Co-authored-by: vallicgrr <[email protected]>
STRATA_QSA_DUMP_LAYER=i keeps one QSA layer, STRATA_QSA_DUMP_STEPS=1 appends
the steps to each record.
pa_bench: the prompt sparse attention (qsa_prompt_attn_batch) on a
recorded selection - a STRATA_QSA_DUMP record of one layer and chunk with
its steps appended - random q and int8 K/V; median time and an output
hash for A/B of kernel variants (PA_BENCH_OUT=file writes the output).
On Turing the v2 int8 prompt attention (STRATA_PA_V2T=1) stages the next
chunk in registers, so its second shared-memory stage bought nothing - and
at 45.6 KB a block only one fit an SM (64 KB): four warps on four
schedulers, every latency exposed (ncu: 12.5 % warps active, tensor pipe
8 %). Smem2T: one K/V stage, and the q.k partials of cell group nt
(8 cells) written over the 512 bytes of the warp's own K slice that held
those cells (read by then, a __syncwarp between) instead of their own
8.4 KB. 19.7 KB a block: three per SM (168 registers fit three). The
same arithmetic in the same order - bitwise the same output.
RTX 2080 Ti, a recorded 130K selection (QSA layer 5, last chunk: 15871
queries x 2051 cells, tests/cuda/pa_bench.cpp): 113.8 -> 68.5 ms, same
output hash. Engine (Q2_0 pack, 16K chunks, fused experts): 98K prefill
53.3 -> 49.9 s, 4K 3.54 -> 3.42 s; first logits bitwise identical at 98K
and 4K, the same generated text.
On Turing the int8-KV prompt attention ran v1 because v2 stages its K/V
slices with cp.async. Here a warp loads its next slice into registers and
stores it into its other stage after the current chunk (the same bytes in
the same places); the MMAs are the m16n8k8 pairs v1 already uses. Accuracy
as v1 (qsa_prompt_attn_parity vs FP64: 3.7e-6 / 4.9e-6 / 4.5e-6 / 3.8e-6 of
3.5-3.7). RTX 2080 Ti, ms per chunk v1 -> v2: 32K x 1024 queries 13.0 -> 7.6,
64K x 512 6.7 -> 4.8, 1.5K 6.2 -> 5.0, 2.1K x 256 3.2 -> 2.3. Opt-in until
the engine A/B.
With STRATA_BF16_TC the prompt path converted xn16 (T x 10240) from BF16 to
FP16 before each of its two GEMMs (down and inject), in 16M-element slices:
~2.8 s of a 130K prompt read on an RTX 2080 Ti. The four xn16 writers now
store gemm.cu's conversion of the BF16 value (bf_to_hf_conv: the same clamp
and rounding as bf16_to_f16_kernel), and Gemm::bf16(..., x_f16) skips the
conversion on the tensor-core path (same slices, same GEMMs) and converts
back to BF16 on any other. Only when the stage runs BF16 on FP16 tensor
cores, without the BF16X2 low parts or a padded xn16 stride; the MTP draft's
own read keeps BF16. STRATA_XN_F16=0 turns it off.
First logits after a 98K prompt: bitwise identical; prompt read 1431 -> 1473
tok/s; a 4K run writes the same tokens.
The per-group dequant kernel writes 2 bytes every 64 bytes; 0.1.43 coalesced only the Q8_0 path (#1720). On compute
capability 7.5 the BF16 projections of the prompt path take a kernel that writes 16 bytes per thread; the values are
the same (a conversion, no arithmetic), so the output is bit for bit the old kernel's. Other cards are untouched.
The rocBLAS the engine ships has tuned gfx1030 kernels for FP16->FP16, int8 and
FP32->FP32 only; every FP16/BF16-input, FP32-output GEMM (rocBLAS HS/BS) runs a
fallback kernel at ~5 TFLOPS. Widening the inputs to FP32 is exact and SGEMM
accumulates in FP32 like the native call.
Nothing passes through FP16, so unlike STRATA_HIP_PROMPT_F16=1 (#835) the
route cannot overflow: an FP16 GEMM output above 65504 becomes inf and can
spread as NaN, while here it stays a finite FP32 value. It is slower than the
FP16 route, and not bit-identical to the native call (at most 4.3e-6 of the
largest output apart on random inputs).
RX 6800, Coder IQ1_M, fresh 12K-token prompts (4 each, A/B in one binary via
STRATA_RDNA2_SGEMM=0): prompt 312 -> 452 tok/s, GPU timeline 37.2 -> 25.5 s
(gdn 9.4 -> 2.8 s, qsa proj 3.6 -> 1.4 s, hc read 3.85 -> 2.84 s). Decode
unchanged. A 9-task code-repair benchmark: 9/9 first try on both, model time
458 -> 374 s.
Default on for gfx103x only; N < 64 keeps the native call; if the FP32
buffers cannot be allocated the native call runs.
Rebased onto 296b59d2: the conflict in Gemm::f16 and gemm.hpp (the CUDA FP16 algorithm fallback and the sm_60 STRATA_DENSE_F32 block that main gained) is resolved by keeping both; the RDNA2 call sits before the CUDA/HIP GemmEx call.
Co-authored-by: vallicgrr <[email protected]>
block_topk_wide_kernel reads a query's scores six times (four radix passes,
a count, the emit): past ~115K cells that is DRAM traffic. block_topk_sample_kernel
takes a bound from every 32nd block (the KS-th largest of those keys), reads
all blocks once keeping those at or above it, and when they hold at least
`width` cells computes block_topk_kernel's threshold, `above` and tie budget
over them and emits them in block order (bitonic sort): every block at or
above the threshold is a candidate, so the ids are the reference's.
Otherwise the query is flagged and block_topk_wide_kernel<true> runs it.
qsa_topk_parity --selftest and qsa_topk_active_parity: identical ids
(continuous, 64-level, equal scores - the last ones take the fallback).
RTX 2080 Ti, 256 prompt queries under a 512K capacity: 131K 0.51 -> 0.45 ms,
262K 0.95 -> 0.59, 400K 1.48 -> 0.71, 520K 1.84 -> 0.79. Used above 28,672
active blocks; STRATA_TOPK_SAMPLE=0 keeps the wide kernel.
A verify window's 1-8 queries scored and selected on a handful of SMs; at
420K cells on an RTX 2080 Ti the two took ~6 ms of a ~52 ms window.
- block_scores_halve_kernel: the decode scorer's warp sums as one
recursive-halving reduction (each value pairs lane l with l ^ o at offsets
16..1, the butterfly's additions operand for operand), the lane's query
values in registers (shared-memory reads were the limit, ncu mio_throttle),
5-8 queries as two calls of up to 4. Same bits (qsa_select_bench
QSA_BENCH_DECODE=1 hashes, 32K-524K, 1-8 queries). 420K: 1 / 3 / 4 / 6 / 8
queries 117 / 265 / 347 / 510 / 774 -> 101 / 118 / 125 / 238 / 255 us.
STRATA_SCORES_HALVE=0 keeps the old kernel.
- qsa_block_topk_split: the decode top-k over chunks of 2,048 blocks in six
launches (four radix passes into per-query global histograms, a count, the
emit); every CTA takes each digit from the complete histogram with
block_topk_kernel's rule, so the ids are identical (qsa_topk_parity
--selftest: continuous, 64-level and equal scores, 2K-524K). Decode calls
at 420K: 0.21 -> 0.033-0.064 ms; short contexts unchanged within 0.01 ms.
STRATA_TOPK_SPLIT=0 keeps the wide kernel (=1: on any card, for tests).
Both on sm_75 only by default.
block_scores_simt2_kernel: 8 x 8 products per thread, twice the blocks per
CTA and the next K chunk in registers; every dot is the same fmaf chain over
k = 0..127 and the same relu sum, so its scores are bitwise those of
STRATA_SELECT_SIMT=1 (qsa_select_bench QSA_BENCH_HASH=1, 131K-510K).
RTX 2080 Ti, 256 queries: 131K 1.21 -> 0.95 ms, 262K 2.01 -> 1.87,
400K 3.00 -> 2.82, 510K 3.77 -> 3.57 (the warp kernel: 7.7 / 15.3 / 23.2 /
29.6 ms). Opt-in like =1 (another summation order than the warp kernel).
Below stream_all_min() the MMQ path gathered every expert with its own launch
and, for a staged one, its own wait and release event. The staged walk now
groups like the streamed walk: its ring is STAGE_GRP (32) slots, two groups, and
it stages at most 15 entries ahead, so a group's staged slots are not refilled
before the group is gathered (one wait on its last copy, one event releasing
them all). stage_slots() counts those 32 slots wherever the ring is counted or
taken, so a prompt borrows every byte it writes.
STRATA_PREFILL_GROUP_GATHER=0 turns grouping off everywhere, as before.
gr_up75_kernel: two batches of 4 rows of w_up loaded before the dots, the
token activations read once per batch, the 4 xor reductions interleaved and
the row tails held 4 rows to a uint4 - bitwise gr_up_multi_kernel (gr_parity,
gr_multi_parity, fused_gr_bench hashes). Used from 3 tokens on sm_75
(STRATA_GR_UP75). The staged down kernel takes a token group from
blockIdx.y; sm_75 runs T 4 and 6 as pairs (STRATA_GR_DOWN_TG).
fused_gr_bench cold, RTX 2080 Ti, whole read: T3 53.2 -> 50.2 us,
T4 64.1 -> 57.6, T5 76.2 -> 69.3, T6 86.1 -> 70.9; T1-2 unchanged.
native_expert_bench on real Q2_0 rows of Flash-Next, RTX 2080 Ti, bitwise
equal to mode 0 in every case: 10 groups x 1 entry 47.3 -> 41.4 us,
20 x 1 83.7 -> 71.5, 30 x 1 120.2 -> 100.2, 20 x 2 124.2 -> 111.4. The down
projection of mode 4 loses with 3+ entries per group (12 x 3 36.5 -> 40.8,
16 x 4 56.9 -> 68.9), so cap_groups > 30 keeps mode 0 for it.
STRATA_SM75_EXP=0 keeps the CUDA layout; STRATA_EXP_MODE still overrides.
adapt() marks an evicted expert non-resident in host_res at once; d_res follows in apply_pending, once the round's
copies have landed. When the tier does not wait for them (STRATA_ADAPT_NOWAIT=1, or STRATA_ADAPT_LAG > 1) the next
window runs before that. doorbell_publish_res reads d_res, sees the evicted expert as resident, and when every routed
expert of a layer looks resident it does not publish the layer's activation. The pool plans from host_res and
computes the evicted expert on the CPU from the activation published before. Now every verifier launch (run,
run_slot_rows, batch_launch, pl_launch; a layer split's stages too) runs a pre-launch hook. It patches the device
tables from a list of changed entries: res_patch, a small kernel on the verifier's stream reading mapped memory; the
whole table when the list overflows or a device does not map it at the host's address. A failed whole-table upload in
apply_pending leaves its entries to the hook. Only where the tier does not wait. --batch-groups with a layer split
publishes every activation instead: set_always_publish, which now also turns E-6's device plan off.
--pipeline-windows keeps its fence. STRATA_ADAPT_EVICT_SYNC=0 restores the old behaviour. STRATA_ADAPT_CHECKRES=1
reads the tables back before each window, compares them with host_res, and counts the layer-windows that computed a
CPU expert with no activation published.
RTX 5090, IQ2_XS, --expert-cache 12000, 32K explain, 600 tokens: NOWAIT=1 34 of 222 windows with mismatched tables
(2470 entries), 1 such layer-window -> 0 / 0 / 0; LAG=2 55 of 223 (3222 entries: every swap), 5 -> 0 / 0 / 0. LAG=2
is reproducible (two runs of each arm, the same tokens), and the fix changes its answer from token 228. The hook's
cost under NOWAIT (6297 slots, 3 pairs): ms a round -1.1% chat, -1.2% 32K, noise. The default (the tier waits)
installs no hook: tokens and first-token logits identical to d5ea7133 (p2k 200 tokens, 32K explain 600).
Port of PR #1517 (commit eb4796045) onto 0.1.44.1. Conflicts with the 0.1.43 adapt overlap gate and the tail-skip kind -2 resolved; set_adapt_gate and set_always_publish kept side by side. Docs line added.
Co-authored-by: sergiywith <[email protected]>
The real data_folder() moves the model files of this checkout and of the data folder named in the user's settings.json into --data-dir and saves that folder as the new data folder. With a TemporaryDirectory the models are deleted on exit and settings.json is left pointing at a deleted directory (found on the M4 and on the PC: the test wrote %APPDATA%/Strata/settings.json and ~/.config/strata/settings.json). The test is about --source, so data_folder is mocked.
The 2-4 column dense projections of the verify windows ran the plain multi-column kernels on Turing, which are
bound by dp4a throughput there from 2 columns on. mmvq_il_parity --bench --emit-table on an RTX 2080 Ti (SM 1650
MHz) gives a table like the Volta/Pascal ones: IQ4_XS 2560x10240 at 4 columns 81.1 -> 37.4 us, Q4_K 74.5 -> 39.1,
the Q5_K head 1671 -> 1045 us. Bitwise equal (the decode writes the same tokens). No one-column table: the
engine's dedicated single-column kernels sum in another order, for a 0.5% gain. STRATA_SM75_IL=0 turns it off.
Q2_0 512K resident, 4K prompt: one-shot decode 52.6 -> 55.5 tok/s; serve steady state ~69 -> ~77 tok/s.
prompt_attn_i8v3_kernel: QK^T and PV as mma m16n8k32 on the int8 K / V codes as stored; the other operand in three
exact 8-bit parts (q scaled per head and 64-dim group, p * v_scale per row and chunk; X = rint(x * 65536) split into
h, l signed and m unsigned bytes, a u8 x s8 MMA): 24 bits, exact int32 sums, combined in FP32. One K and one V stage
per warp, 12-row partials, launch bounds (128, 3): ~25.5 KB, 3 blocks an SM. sm_80+ (STRATA_PA_SM80); Turing,
Volta and HIP keep their kernels. Unset: v2, byte-identical.
qsa_prompt_attn_parity: against FP64 IMMA 2.15e-6 - 2.64e-6, v2 2.78e-6 - 3.30e-6, the FP32 kernel 1.90e-6 - 2.51e-6.
IQ2_XS, first-token KL to the FP32 kernel (STRATA_PROMPT_ATTN_OLD=1): median 0.0027 (IMMA) against 0.013 (v2);
2K 0.0026 against 0.0248. 1.35-1.53x a launch; a 32K prompt in 8K chunks 358 -> 312 ms of attention.
Fix-up to the ported #1149 commits. The range counters and the saturating to_f16
are compiled for HIP only, so the CUDA kernels (hf_sat in the SwiGLU product, the
embedding image) stay exactly as they were, with no flag read; set_f16_range /
f16_range_read are no-ops there. strata_hip.h (the STRATA_HIP_GFX906 compat layer,
which goes first on that backend's include path) gets the cudaMemcpyFromSymbol
mapping the HIP shim already has, so the counters also compile for gfx906.
The verify window reads pcie_frac of a layer's missed experts over PCIe, a share
the link probe sets; the CPU's speed does not enter it. With a slow CPU the GPU
spins on the pool's rows: IQ2_XS with one AVX2 pool worker (emulated) runs
16.8-17.2 ms a round at the probe's 0.55.
Opt-in, STRATA_PCIE_BALANCE=1 (one GPU, no batch slots, no --pcie-frac):
- wait_flag_ge_stamped: the flag spin kernels also write the GPU clock as the
plan arrives, as the PCIe part starts and as the GPU waits for the CPU's rows
(no extra launch; captured only with the switch on);
- the host times its pool per step; after each window, least squares with
forgetting fit the pool's a + c n_cpu + d n_pcie (d: the link's DRAM reads
slowing the pool - on a DRAM-bound pool moving experts to PCIe saves nothing)
and the GPU's g0 + g n_pcie;
- a layer then reads over PCIe the m of its misses that minimizes the longer
side; every 64th layer that would keep them all on one side moves one, so
both fits stay measured.
IQ2_XS, RTX 5090 + 9950X3D, chat to the end of the turn:
- one AVX2 pool worker: 16.77 / 17.17 -> 12.98 / 12.82 ms a round
(fit: CPU 42-60 us an expert, GPU 28 us a PCIe expert; 0.8 of the misses
move);
- the 9950X3D's 16 workers: 3 pairs 12.19 -> 12.17 (pool DRAM-bound, few move).
Off (the default) the logits and 32 tokens equal v0.1.39 (2K, fixed cache).
Port of PR #1548 (commit a513394c4) onto 0.1.44.1. Conflicts with the 0.1.43 tail skip, PCIE_MIN1 and the fused waits resolved: the fused wait+copy launches stay off while the balance stamps are on. Docs line added.
Co-authored-by: sergiywith <[email protected]>
The up GEMM of each hyper-connection read writes `gated` (T x 10240 FP32, 336 MB at an 8K chunk) and gr_mix_r reads
it back with R at ~96% of the bandwidth - only fewer bytes help. gr_upmix_cuda_kernel (128 tokens x 32 columns x 4
streams a block, K = 320 in cp.async-staged slices of 32, BF16 mma.m16n8k16 with FP32 accumulation, the mix in
registers in gr_mix_r's order writing all its outputs) never writes or reads `gated`. Behind the gfx11 switch;
sm_80+ per device (STRATA_EMULATE_CC honoured), chunks of >= 33 tokens; else the pair. Unset: unchanged.
gr_upmix_parity (new ctest): bitwise the cuBLAS + gr_mix_r pair at T >= 33 on an RTX 5090 (cuBLAS 13.3), <= 1.8e-7 at
T <= 32, both as close to FP64. 1.42x / 1.81x / 2.19x the pair at 2K / 8K / 32K; IQ2_XS: an 8K prompt -4.4%, 32K
-3.9% (the hc read -19% / -24%). Logits identical with it on (600, 2K, 32K); the 32K greedy tokens moved only through
the time-driven suffix-draft policy (identical with --suffix-draft 0).
mmq::quantize(m.mixed, m.src_dev, ...) read and quantized every token's row of `mixed` once per routed expert (K =
10). mmq::quantize_scatter uses llama.cpp's quantize_scatter_mmq_q8_1_cuda (the pinned quantize.cu strata_mmq
already compiles): each token quantized once, its block written to all of its rows through slot_dev, the inverse of
src_dev. The same q8_1 bytes, so the default; STRATA_QUANT_GATHER=1 keeps the gather. No new device code (CUDA and
HIP compile the same kernels); SYCL's prefill.cpp untouched.
prefill_mmq_scatter_test (new ctest): every MMQ q8_1 layout (D4, DS4, D2S6), T = 1..2099: 0 differing bytes. IQ2_XS
600 / 2K / 32K and Q2_0 2K against e8ca9afd: identical first-token logits and 32 greedy tokens. RTX 5090 (96 MB L2):
the quantizer 2x at 2K, even at 8K (the chunk's input fits in L2), 1.4-1.9x at 32K; a 32K prompt with
--prefill auto:32768: 77.5 -> 58.1 ms of quantization.
Co-authored-by: sergiywith <[email protected]>
Windows sends a GPU's interrupts to one logical processor, usually the first,
and the host thread spins there (--host-core first, the default). --host-core
last moves it to the last physical core, but not on a hybrid CPU, whose first
core is its best P-core, so a 12th-14th gen Intel or a Core Ultra keeps the
host on logical processor 0 with the interrupts.
--host-core sibling (or STRATA_HOST_CORE=sibling) keeps the host on that same
first core but on its other hardware thread. The interrupts keep the first
logical processor, which nothing else is pinned to, and the workers keep
every other core, since they take one logical processor per core (with
--pool-affinity auto / p-cores, which also use the P-cores' siblings, the
host's sibling is taken out of their list). Without an SMT sibling it is
--host-core first. It works the same on Linux (the sibling from sysfs), where
the interrupts are usually spread anyway.
detect_cpu_topology records the host core's sibling (CpuTopology::host_sibling)
in each layout; pool_affinity_test checks, in every affinity mode, that
sibling puts the host there with no worker on its core, and that it is first
without SMT. The startup log names the placement.
The README's range check said "none beyond 65504" over 110K tokens without
the numbers. The engine now has the counter: STRATA_F16_RANGE=1 records,
per device, the largest |value| and the count beyond 65504 (or not finite)
where a value enters or leaves FP16 - the activation images (hf_sat, the
image writers' one funnel), the BF16 weights converted to FP16
(bf16_to_f16_rows) and the FP16 GEMM outputs as they are widened
(widen_rows_f16: an output rocBLAS wrote beyond the range is Inf there) -
and prints one line per prompt. Off, the cost is one flag read per value;
on, a 34.4K-token prompt read at 889.8 tok/s against 888.5. to_f16 (the
hc image for the FP16-weight GEMMs, the PLE key image) saturates like the
other image writers now, so every FP16 image is under the same rule. The
HIP shim gains cudaMemcpyFromSymbol.
Measured on one RX 6900 XT over 466K tokens in nine prompts (corpus
slices, two prompts built to push activations - delimiter floods, a token
repeated 4,000 times, 6,000-character runs, base64, five scripts - and
four batches of real requests): outputs peak at 409.5 (160x below 65504),
activations at 112.2, the BF16 weights at 11.56; nothing beyond the range,
nothing non-finite. The prompts built to push activations stayed below
the ordinary corpus. One model (GSQ-RCO IQ3_S); the table is in the
README's range section.
The README also gains the same-card comparison with #1006, which takes
these GEMMs as FP32 SGEMMs: 8.4K / 32.8K / 104K-token prompts 412 / 455 /
456 tok/s on the old path, 622 / 742 / 750 with #1006, 730 / 900 / 904
with this change (the card's FP32 rate against its FP16 rate), with the
phase times and what the SGEMM route covers that this change does not.
(cherry picked from commit e780c8581e)
The test ran only the FP32-out GEMMs, so the gfx103x prompt path's FP16-out
GEMM (f16_inplace, widen_rows_f16) and bf16()'s W conversion through the
scratch were not under ctest. Every shape now runs a second time with
set_f16_io(true), forced on whatever HIP card runs the test: X as the FP16
image for the BF16 products, a scratch of 64 rows so a 96-row weight converts
in two slices (the second written at Y + 64), ldy > N, an odd N (257), T = 1,
and a beta = 1 f16() call, which keeps the FP32-out GEMM. FP16-out results are
checked against the double reference at FP16's rounding (6e-4 relative per
element plus 1e-4, rel L2 < 1e-3); the row padding and the guards around Y
must be untouched, as before. The FP32-out runs keep their tolerance.
Found while adding them: the BF16 inputs were all zero. hip_bfloat16(float)
produced data 0 in this test's build (ROCm 10.0, gfx1030; not traced further),
so the two BF16 cases passed on nothing (rel L2 printed 0). The BF16 bits are
now made by hand (round to nearest even on the float's bits); with real inputs
the BF16 cases pass at rel L2 1.9e-07 (FP32 out) and 2.2e-04 (FP16 out).
On an RX 6900 XT (gfx1030, ROCm 10.0): 11 of 11 pass.
(cherry picked from commit 4c6c9fe374)
On HIP a prompt batch with fewer than 7,168 blocks (28K cells of context) took the original top-k kernel, a line measured on
gfx1201 before the cooperative scan. With the scan the register kernel is faster from about 15.5K cells on gfx1151 (2,048 queries:
original / register kernel 0.57 / 0.82 ms at 13K, 1.01 / 0.87 at 16K, 2.4 / 0.93 at 20K, 3.9 / 1.00 at 24K). The line is now
STRATA_TOPK_REG_MIN_BLOCKS (default 7168, as before) and gfx1151's table sets 4096. The selected cells are the same
(qsa_topk_parity, also run with the register kernel always on: ctest qsa_topk_parity_reg_always).
block_topk_reg_kernel picked each radix digit with thread 0 walking up to 255 bins while the other threads waited.
Threads 0-255 now prefix-sum the bins in descending order, and the one where the count above reaches `width` takes the
digit (bin 0 when none of 255..1 does). The digit and the count above are the same, so the selected ids are the same.
Measured on a Ryzen AI Max+ 395 (gfx1151, ROCm 7.14.1), qsa_topk_parity with 2,048 queries (CTX 2048 10 <ctx> 1),
ms per call:
32K 2.19 -> 1.11 (-49%)
64K 2.54 -> 1.63 (-36%)
128K 4.83 -> 3.79 (-22%)
qsa_topk_parity --selftest: 19 PASS. UD-Q4_K_XL end to end, 3 interleaved rounds: the same text in every round; the
128K prompt moved +0.2%, inside the noise, so the gain is the kernel's and not yet visible end to end.
CUDA (RTX 3060, Windows, CUDA 13.0, sm_86), same call, ms: 4K 2.12 -> 0.975, 8K 2.04 -> 1.03, 16K ~1.9 -> ~1.1, 32K ~2.0 -> 1.27, 64K 2.53 -> 2.15, 128K 7.29 -> 7.29 (no change); qsa_topk_parity --selftest passes, ids identical in every case. Not built: SYCL.
The negative-scale-times-zero workaround was gated to an arch list (gfx1012 on
HIP < 7, gfx1151, gfx1200/gfx1201 since 0ff45cffb); the list only ever chased
the newest report. The workaround writes the scale's sign bit for code 1, which
is value-identical on targets whose compiler never folds: keep it on every HIP
build. Ported from PR #1540 onto the tree that already carries the gfx1200/1201
entries; the comment text is adjusted.
Co-authored-by: nekomario28 <[email protected]>
The pool's default layout puts a worker on every physical core, and a layer's
experts are done when the slowest worker is. On a Core i7-13700K (8 P + 8 E
cores) the E-cores make the verify window wait, by an amount that changes from
run to run.
`--pool-affinity measure` (hybrid CPUs, native packs) times both layouts on
the pack's own experts before the pool is built: two experts of four tokens
per layer through run_split_multi_native, every layer in turn, the host
spinning ~150 us between two layers, three interleaved blocks per layout,
about a second in all. p-cores is taken unless all is more than 10% faster.
One line says what it read and took. It is one reading at start and is not
repeated.
Measured on this branch (0.1.42 + this change), Core i7-13700K, RTX 5090,
64 GB, Windows 11, Swift 1.5 IQ3_XXS, 128K context, one 8K prompt and the
same 1,500 forced tokens (--spec-follow), three interleaved rounds:
default (all, 15 workers) 146.6 / 151.8 / 158.0 tok/s, pool 6.3-7.6 ms per window
--pool-affinity measure 200.0 / 201.5 / 204.8 tok/s, pool 2.1-2.2 ms (read all at 16.4-17.6 ms, took p-cores)
--pool-affinity p-cores 200.3 / 203.9 / 204.9 tok/s, pool 2.1 ms
Over four server starts of one day on 0.1.41 it read all at 14.8, 21.9 and
25.3 ms against 6.1-6.2 ms for p-cores, and once all at 6.6 ms against 8.6 ms,
where it took all.
Without the value nothing new runs: the option parses as before and the
function is not called. The layout changes no result, only where threads run.
Built for CUDA only (sm_120, CUDA 13.0.88, MSVC 14.50 with nvcc's
--allow-unsupported-compiler); HIP and SYCL were not compiled (the change is
in generate.cpp, which both build). Not measured: other hybrid CPUs, Linux.
0.1.43 made the decode window lean under STRATA_VERIFY_NO_HOST with the device
plan (46a77907): the doorbell publish, flag waits and plan copy are dropped
because no host answers them. capture_batch() records batch windows through the
same record_window(), so a batch graph was also recorded lean, but the host batch
path (run_slot_rows / batch_poll) still waits for each layer's doorbell unless the
stage is all-resident. On a stage with mirrored experts the graph finished with
the ring silent: "verify batch: layer 0 never rang (graph finished)", then
UR_RESULT_ERROR_UNINITIALIZED and exit 139 at the first shared window.
Leave lean off while recording a batch window (batch_rec_). The solo window
stays lean. Batch tokens are identical to a run with STRATA_VERIFY_LEAN=00.
With the default --suffix-draft 3 the suffix block above already raises --spec by 2
(4 -> 6) and writes mtp_max_t = 4. The --lookup-chain block then computed
min(mtp_max_t + K, 8) = min(4 + K, 8), so with --spec 4:
K=1 -> max(6, 5) = 6 (unchanged: the flag does nothing)
K=2 -> max(6, 6) = 6 (unchanged: the flag does nothing)
K=3 -> max(6, 7) = 7
Measured on 0.1.40 by reading the cap back from /metrics engine.spec after a restart
per row; K=1 and K=2 were indistinguishable from not passing the flag.
Size from the cap actually in force instead, so K adds up to K of headroom while the
kVerifyMaxT = 8 ceiling and the mtp_max_t bound on the MTP's own drafts are preserved.
Behavior is unchanged when --suffix-draft is 0 or when the cap is already at 8.
Under WSL2 the NVIDIA driver caps what cudaHostAlloc(Mapped) can pin at about
1 GiB in all, so the KV streaming host copy (layer.cpp) cannot be allocated
and setup turns streaming off there: the whole KV cache must sit in VRAM,
beside the expert cache. cudaHostRegister of ordinary memory is not capped the
same way; the expert arena already registers tens of GiB with it on the same
machines (pinned.cu).
kv_host_alloc() maps anonymous memory and registers it Portable|Mapped, before
any page is touched, when /dev/dxg exists (WSL2, detected as generate.cpp's
under_wddm() does). Elsewhere cudaHostAlloc stays first. Either way the other
method is the fallback when the first is refused, and
STRATA_KV_HOST_REGISTER=1/0 forces the order. On native Linux and Windows the
behaviour is unchanged unless cudaHostAlloc is refused (by inspection; not run
there). The failure message no longer says WSL pins only 1 GiB.
Measured on one RTX 5090, Ryzen 9 9950X3D, WSL2 (157 GiB), Windows driver
591.86, IQ3_S, --max-context 1048576 --rope-scaling yarn --rope-scale 4
--kv int8 --kv-resident 32768, with this commit:
- start: "the K/V in 12.38 GiB of pinned RAM"; expert cache 11,609 slots
- tools/needle_bench.py: 4/4 found, 128K and 1M (1,015,070 prompt tokens) at
depths 10 and 90
- decode 99-124 tok/s with 0.5-1M tokens in context (4 runs); 150-191 tok/s
medians from 55-token prompts to 128K (single runs 113-196); prefill about
4,200 tok/s for a cold 1M prompt
kv_stream_parity still passes; it allocates its own host pools, so it checks
the streaming kernels, which this change does not touch.
Not tested: layer split or remote experts under WSL, Docker Desktop.
setup.py still never writes --kv-resident on WSL and removes it there
(upgrade_config); until that changes it has to be added by hand.
Port of PR #1498 (commit 79b43be18).
pin_cache_complement set lend_from_slot = -1 whenever --resident-budget-gib was given, so the experts of the cache
slots the prompt path borrows were in neither VRAM nor RAM during a prompt and were read from the model files on every
chunk, and again when the slots were refilled - even with room left in the budget. The budget branch now appends the
lend region's experts after the ranked ones, from the last slot down (the order the borrowing takes them), while they
fit; the budget stays a hard cap, and the ones past it keep the file fallback. The start-up line reports the count, and
the lend warning names a larger budget as the remedy when one is set.
Measured on an RX 6900 XT 16 GB (HIP gfx1030), Ryzen 9 5950X, 125 GB RAM, UD-Q4_K_XL, --resident-budget-gib 71 (the
copy held all 63.47 GiB outside the cache, 7.5 GiB unused; 1,231 lent slots = 3.60 GiB), engine 0.1.42, one run per
case, greedy: file reads of the lent experts per request 0.5-2.6 GB -> 0; SSD reads per fresh 8K / 16K prompt 2.8 /
1.9 GB -> 0.3 GB; prompt-path host staging 0.8-2.1 s -> 0.06-0.5 s per request; fresh prompts 887 / 7.8K / 15.8K
tokens 180 -> 194, 633 -> 676, 677 -> 698 tok/s. Decode unchanged.
Port of PR #1896 (commit 29063eb2b).
Rotation hands the evicted expert's exchange buffer over as its place in the
RAM copy instead of copying the bytes there. It required one block size for
all layers and fell back to the copy path on any other pack: Swift 1.5 IQ3_S
has seven sizes (1,510,400 to 2,662,400 bytes) and IQ3_XXS nine.
A swap exchanges two experts of one layer, so the slot the promoted expert
leaves has the size the evicted one needs. ExchangeStorage now keeps its
spares per block size (initialize_sized): an exchange takes a spare of its
layer's size and hands back one of the same size. commit() refuses a spare of
another size and an expert of another layer. initialize() keeps the uniform
geometry and its checks. exchange_buffer() takes the layer; reserve_exchanges()
takes the number of trailing buffers that only stage copies (--adapt-async's
second half), kept in the largest size alone. With --adapt-async 1 a rotated
round flips in the tick that sees the copies in: there is nothing for the job
to move, so the round ends one verify window sooner.
Still opt-in (STRATA_EXCHANGE_ROTATE=1); unset, nothing changes. The exchange
allocation grows to one spare of every size per swap of a round: 1.06 GiB of
page-locked RAM on IQ3_S with --adapt-swaps 80 (1.26 GiB with --adapt-async 1)
against 0.20 and 0.40 GiB.
Measured on a Core i7-13700K, RTX 5090 32 GB, 64 GB, Windows 11, Swift 1.5
IQ3_S, 128K, --resident-experts, --adapt-every 1 --adapt-swaps 80, p-cores:
one 8K prompt, the same 1,500 forced tokens, blocking tier, rotation off / on:
161.6-165.1 tok/s (five runs) / 184.9-188.7 (four runs); two runs of each
arm with the same cache size (10,681 slots) wrote identical --window-hashes
for all 434 windows
--serve, 23 coding-agent turns, greedy, three interleaved rounds, blocking tier:
147.8 / 159.2 / 158.4 (the build before) -> 175.2 / 184.9 / 172.8
the same turns with --adapt-async 1, one binary, two rounds:
166.0 / 170.1 -> 185.6 / 182.5
The tier swaps about 30 experts per verify window on this box (44% of the
experts fit in VRAM), so the copy path moved about 50 MiB of RAM per window.
tests/core/exchange_storage_test.cpp gains a three-size case (6,144 exchanges)
and passes built alone with MSVC 14.50. Built for CUDA only (sm_120, CUDA
13.0.88, MSVC 14.50 with nvcc's --allow-unsupported-compiler); HIP and SYCL
were not compiled (expert_source.cpp and generate.cpp are built by both). Not
measured: Linux, AMD cards, a layer split.
Port of PR #2042 (commit 8db375018).
The multi kernel reads the dense IQ3_S matrices (attn_q, attn_gate, ssm_out, attn_k, the shared experts' gate/up)
at 76-114 GB/s on the B70. The new kernel has the shape of the Q4_K..IQ4_XS wide kernels. unitrace, one card,
IQ3_XXS, T=1: worst attn_k (512 rows) 77 -> 123 GB/s, best attn_q (12288 rows) 114 -> 307 GB/s; these matrices take
0.60 ms of kernel time per token instead of 1.36 ms.
Unset or 0: the multi kernel, output identical to main. With the option the greedy output parts from the default at
token 198 of 256 (the sub-blocks are summed in another order).
Since 0.1.40.2 the waves_per_eu(8) variant of native_w11_kernel is uncapped on clang 22 and newer, because clang 22
computes wrong results with it on gfx1100. On gfx1151 with clang 23 (ROCm 7.14.1) the capped kernels give the same
bits and take less time, as alexdns1 measured in #1180. The cap now applies there again, per format
(hip_prefill_fused_iq, one layer, 2048 / 3584 / 8192 tokens, the same output bits hash in every case):
- gate/up IQ2_XXS / IQ3_XXS / IQ3_S / IQ4_XS and down Q2_0 / IQ4_NL: capped, 8-15% less time per layer.
IQ2_S gate/up is capped too, but its LDS leaves room for 3 blocks per WGP and the compiler keeps its 238 VGPRs;
IQ2_S / Q2_0 takes 2-4% less time from the Q2_0 down.
- IQ2_XS gate/up: not capped. Under the cap it spills 104 VGPRs and takes 61-66% more time.
- Q8_0 down: not capped. It spills 18 VGPRs and took -1.9% to +1.9% time (with Q4_K gate/up).
The test is the clang major version: other clang 23 builds are not measured, and clang 24 and newer keep the clang 22
rule. The template argument is the same in the host and every device pass; only the device pass for gfx1151 decides
the attribute (nw_capped). Other architectures, other compilers and -DSTRATA_W_LB=N behave as before: the gfx1100
disassembly is identical to before (kernel names aside). The file builds for gfx1012 / 1030 / 1100 / 1150 / 1151 /
1200 / 1201 and with STRATA_HIP_GFX906 (gfx906); no CUDA build here, and the changed lines are inside
`#if defined(__HIPCC__)`. SYCL has its own copy of this file, not touched. The path is opt-in (STRATA_PF_FUSED=1), so
default output is unchanged.
End to end on a Ryzen AI Max+ 395 (128 GB, gfx1151), UD-IQ4_XS, the section-5 switches of docs/STRIX_HALO.md with
--prefill 16384, 3 interleaved rounds of fresh servers: a 128K prompt read 1,126.1 -> 1,153.4 tok/s (+2.4%; +1.8%,
+2.4%, +2.6% per round). At 4K (868.0 -> 894.9, +3.1%) and 32K (1,182.9 -> 1,207.0, +2.0%) the medians moved too, but
within the spread of 3 runs. Decode unchanged; the same text for all 9 prompts.
prompt_attn_wmma_kernel interleaved a chunk's global reads with its
compute: the V slice was staged first, then each 16-cell tile loaded its K
scale and fragments between the q.k WMMAs, so a wave waited on each read
in turn (a thread trace on gfx1151: 56% of the kernel's time in
s_waitcnt vmcnt, 0.4% in WMMA). The chunk's reads now go out together
(fetch), then the same compute runs on them (chunk). The values and the
order of the arithmetic are unchanged.
gfx1151, ROCm 7.14.1 SDK (clang 23), hip_prompt_attn_wmma with
STRATA_PA_FAST=1, three interleaved runs against the unchanged kernel,
ms per chunk: 32K context / 2,048 queries 12.50 -> 11.29 (-9.7%),
1,500 / 1,500 3.39 -> 3.20, 2,100 / 256 1.30 -> 1.23. The output hashes
(NEWHASH) match with STRATA_PA_FAST=1 and =0. gfx12 compiles the same
change; it was not run there.
The wide kernels (Q6_K a2 and the generic native_mmvq_wide) spill at 2-8 columns with 128
registers per thread on the A770. They now launch with grf_size<256> there. Large GRF halves the
threads resident per Xe core, so it is the default on Alchemist only (strata::intel_gpu_gen);
Battlemage and other devices, and 1 column on every device, launch the same kernels as before.
STRATA_MMVQ_LARGE_GRF=1 forces it on, =0 off.
mmvq_bench now prints a checksum of the output (FNV-1a over the float bits) so the two modes can
be compared bit for bit.
Measured on an Arc A770 16 GB (xe, oneAPI 2026.1 JIT), same day. mmvq_bench Q6_K, 2560x2560 / 2560x10240 /
10240x2560, 1-8 columns: 1 column unchanged; 2 columns 1.3-2.0x, 4 columns 2.3-3.1x, 6 columns 3.0-5.0x, 8 columns
3.7-7.4x faster (e.g. 2560x10240 at 4 columns 289.3 -> 92.9 us, 74 -> 232 GB/s of weights); checksums identical
with STRATA_MMVQ_LARGE_GRF=0 and =1 in all 15 cases. End to end, a0116b27 + #2190 + #2193 + #2203 with and
without this commit, Coder IQ1_M, 128K context, int8 KV, --resident-experts, STRATA_VERIFY_STEPPED=1, 8K prompt,
256 tokens: decode 23.47 / 23.50 / 23.73 / 23.77 -> 25.99 / 27.10 / 27.49 tok/s (+10-17%), prompt unchanged,
output ids identical where the expert placement matched. Not built or run on Battlemage.
With --batch N the verify window ran QSA block scoring, top-k and decode
attention as one launch per row, and the GDN recurrence (window and 1-row
commit) as one launch per slot. At 4-8 live slots these serial launches are
most of the window's cost beyond the expert reads.
Two env switches, both off by default (off = the same launches as before):
STRATA_BATCH_ATTN_ROWS=1 block scores, top-k and decode attention for all
rows of the window in one launch each; every row
reads its own slot's indexer keys and K/V pools
(passed by value as kernel params, so the graph
captured per row layout stays valid). A slot that
streams KV (kv_mode 1) keeps the per-row path.
STRATA_BATCH_GDN_GROUPS=1 GDN recurrence in the window and the 1-row commit
for all slots in one launch (grid.y = slot group).
Conv commits stay per slot.
Same arithmetic per row: 4 concurrent temperature-0 prompts give the same
bytes with and without the switches (--batch 4 and --batch 8).
RTX PRO 6000 Blackwell, IQ3_S, all experts in VRAM, a0116b2, same day,
total tok/s (docs/BATCHING.md has the table):
--batch 4, 4 clients: 268.4 -> 311.7
--batch 8, 8 clients: 308.4 -> 374.8 (reserve 6144 MiB)
Window: 4 rows 14.3 -> 12.2 ms, 8 rows 24.6 -> 19.9 ms.
Built and run with CUDA 13.0 (sm_120) only. The .cu files are also in the
HIP build, which was not built here (no ROCm). The SYCL port has its own
copies of these files and is untouched.
Q4_K, Q5_K, Q6_K e IQ4_XS: il frammento di attivazione di ogni colonna si carica una volta per blocco
di pesi e serve tutte le righe del blocco; le scale Q6_K stanno in registro. Stesso ordine di accumulo e
di riduzione: uscite identiche bit per bit al layout esatto e alla chiamata a colonna singola
(b6_mmvq_bench, memcmp su entrambe le schede). Default spento.
Stacked on #1415. IQ2_S (type 22, 122 gathered) joins STRATA_CPU_IQ_ALL_EXACT1's list: ggml's AVX2
iq2_s dot sums the same int32 lanes as row_dot_ggml_one and ends with 0.125f * hsum_float_8.
The singleton gate now also requires nt < native_gu_mt_min, so with STRATA_IQ_MT_MIN=1 a lone token
keeps the multi-token kernel and native_expert_parity's width invariance passes with the switch on.
- Q6_K wide kernel (and the planar one): the six-bit offset (v - 32) was dpct's per-byte saturating char4 subtract, a per-byte loop on Xe;
no byte can saturate, so a 4-op SWAR form is bit-exact. Q6_K dense matvecs 59.5 -> 30.8 us (1 row), 77.4 -> 45.5 us (4 rows).
- STRATA_Q6_PLANAR (default on, =0 off): the Q6_K LM head and the MTP draft head are converted at load to ql/qh/scales/d planes (same bytes, the
GGUF-layout allocation is freed: no VRAM change) so a lane's loads are single aligned 16-byte loads; the draft head subset is gathered from the planar
head. native_q6_k_mmvq serves every registered matrix at any column count. Bitwise equal outputs.
- ld_i4_a4: the Q8_1 activation qs (4-byte aligned) as one dword-vector load in the Q4_K / Q5_K / IQ4_XS wide kernels (Q5_K 4 rows 81 -> 68 us).
With STRATA_QSA_SPLIT_ATTN=1 the verify window's and the draft layer's QSA attention run through the prompt path's int8
tensor-core kernel with each query's cells split among up to 20 blocks, and the splits are merged with the log-sum-exp
rescale. qsa_decode_attn_split() returns false (nothing launched) unless the KV is int8 on sm_80 or newer, or when its
partials would not fit the caller's scratch, and on HIP; the caller then keeps qsa_decode_attn_batch(). The sums run in
another order, so with the switch on the output is a rounding-level change. Unset, nothing changes.
The idea is from Eddoursul's fork (fd23d91); the kernel is written against this tree's qsa_prompt_attn.cu.
STRATA_SIDE_MIXER=1 runs the GDN gates and the QSA indexer (key projection, tail snapshot, query chain, block scores,
top-k) on a branch of the window's graph beside the attention's projections, joined before the recurrence and the
attention; =2 also runs the GDN gate projection and the QSA queries' chain on a second branch. A branch that projects
reads F4's interleaved copy, written on the main stream before the fork, and qkv is recorded before the branch so its grid
is dispatched first. Every result is bitwise the same as in line; STRATA_DF_BRANCH is off while this is on. Unset, the
window runs in line as before.
The idea is from Eddoursul's fork (34a35db, b063c13).
native_expert_grouped already decodes each weight part once per block, but the apply re-reads every entry's q8_1 block from global memory with 9 scattered 4-byte loads, so an extra row of the window costs ~60% of a whole expert. expdec_gu_kernel stages the group's entries in shared memory (two conflict-free 16-byte loads per call, the fp16 scale as a float), takes rb rows per block and stages the IQ3 codebooks too; the same lane does the same calls in the same order with the same operations (apply_u = apply on the block's words), so the output is bitwise the default kernels' (expdec_bench: identical on every cell, 2 cards). Gate/up IQ2_XXS/IQ2_XS/IQ3_XXS/IQ3_S/IQ2_S at n_embd 2560; the down (Q2_0, IQ4_NL) behind STRATA_EXPDEC_DOWN=1 (in the engine the Q2_0 down was ~10% slower). Knobs STRATA_EXPDEC_R/_DR/_MINB/_RB/_DRB/_MINBLK/_D20; one log line per (role, format) at the first launch. docs/EXPERTDEC.md.
q5_1_dot and q8_0_dot in kq_avx2.cpp held their per-token sums in the same
runtime-indexed __m256 acc[MAXT] that made bf16_rows_dot_multi spill: with the
array live the compiler leaves it in memory, and every token's FMA reads and
writes a stack slot. Both are now template <int NT>, dispatched 1..4 and, past
that, run four tokens per pass with the row's bytes re-read from L1 - the same
idiom the router change above and kq256_gu_rows already use.
Ryzen 7 9800X3D (Zen 5), GCC 15.2.0, build/native_expert_parity --synthetic
(ggml-quantized weights and activations), one thread pinned to core 0, at the
verify window's four tokens:
Q5_1 down rows [2560 x 640]: 190-197 us -> 155-156 us (1.24x)
Q8_0 down rows [2560 x 640]: 135 us -> 104 us (1.30x)
Same sums: 0 of 36 token-sets differ in any bit against ggml's own vec_dot over
1..8 tokens, both before and after; native_expert_parity reports 0 failures.
Left alone: q4k_dot keeps its runtime-nt array. Its token accumulators are not
what spills - the unpatched function carries 8 ymm stack ops for the sums
against 24 (q5_1) and 59 (q8_0); what q4k_dot spills is its weight-side unpack
arrays (q4l/q4h/scale_l/scale_h), which token chunking does not touch and
re-does. The same change measured +5% at four tokens and -4..-7% at six to
eight, so it is a wash at the window and a small regression above it, and q4k is
the gate/up format that runs most. The AVX1 path, the iq_avx2/q2_avx2 rows and
the engine's dispatch are untouched.
Backends: CUDA and HIP do not compile this file. The SYCL port does; not built
on this machine (no icpx here) - the router change in this branch built there.
Tests: native_expert_parity --synthetic q4_K/q5_1 and q4_K/q8_0 (0 failures,
0 bits differ), iq_avx2_parity, iq_avx1_parity, q8k_quant_parity,
router_dot_parity - all exit 0.
bf16_rows_dot_multi held its per-token sums in __m256 acc[MAXT] indexed by a
variable. With eight 256-bit accumulators live GCC also unrolled the column loop
and left the whole array in memory, so every token's FMA read and wrote a stack
slot (vmovaps 0xc0(%rsp) ... vfmadd213ps 0xe0(%rsp)). On a Ryzen 7 5700X3D one
layer's router [512 x 2560] at 4 tokens took 0.41 ms against 0.25 ms for the AVX1
copy in kq_avx1.cpp that kept its accumulators in registers, so every AVX2 CPU ran
the slower of its two kernels (router_dot_parity --cpu 2, median of 3). Pragma
GCC unroll 1 on the column loop changes nothing; the variable-indexed array is
what forces the array into memory.
Four tokens per pass, in four named accumulators. A row above four tokens is read
a second time, from L1: nt=8 measures 0.37 ms, the same as the spilled form. On
the same CPU the AVX2 kernel goes 0.41 -> 0.19 ms at 4 tokens and nt=2 0.36 ->
0.16 ms, and it now runs 1.3x ahead of the AVX1 copy instead of 1.6x behind it.
The loop body carries no stack slot any more (was 21 memory ops per iteration).
Untouched: the arithmetic and the reduction. Each token's FMA chain still runs
over the same columns in the same order and ends in the same hsum_float_8, so
every sum is bit-identical - the output for nt 1/2/4/8 at cols 2560/5120 hashes
the same before and after (FNV-1a over the whole out[]), and router_dot_parity
reports 0 failures with the AVX2 rel error unchanged at 3.16e-07.
Left alone: the expert row kernels (iq_avx2.cpp, q2_avx2.cpp, kq256_gu_rows,
kq256_rows) and the AVX1 path. The iq_avx2_parity bench over four format pairs x
1..8 tokens moved within 1.6% (noise), and the CPU parity tests (8 targets) pass.
The router stays what it was, an estimate for which experts to prefetch. CUDA and
HIP do not compile this file; the SYCL port does, and it builds there under icpx
2026.1 with no new warnings and no spills either.
The AVX1 copy in kq_avx1.cpp kept a comment saying it reads its rows "exactly as the AVX2
kernel does"; with the row loop chunked four tokens at a time that is no longer the
AVX2 kernel's shape, so the clause is dropped there. Comment only - no code change.
attn_chunk_kernel_pre75 (PR #540 by sskver: two cells per step with a 12-shuffle reduction, cell-major
probabilities, V loaded four at a time) is compiled in every CUDA build, not only with STRATA_EXPERIMENTAL_SM60, and
STRATA_ATTN_DAILY=1 selects it on any card for the decode attention (single query and batch: verify window and MTP
drafter) together with attn_merge_v2_kernel (Hardin22), a merge that skips the chunks the chunk kernel marks empty.
Bit for bit with the default kernels by construction: reduce12 adds the same pairs as warp_sum's butterfly at every
level, the V cells fold in the same order with the same fmaf, the masked cells stay out; the merge does the same fmaf
in the same order. Shared memory 15,872 B like the default kernel (spt shares sq's storage). STRATA_ATTN_MERGE_V2=0/1
and STRATA_ATTN_PRE75=0 switch the two halves separately. HIP unchanged.
qsa_merge_daily_parity (tests): memcmp of both kernels against the default ones on four KV formats, widths at the
chunk borders and non-resident pages; --bench times the chunk kernels back to back.
In a 131k decode block_scores_multi_kernel costs ~20 us more per call for every row of the verify window: not the keys (read once for all the queries) but the shared-memory and shuffle pipes, 4 x LDS.128 of the query slices and 20 shuffles per (block, query). block_scores_rows_kernel<NQ> keeps the NQ <= 4 queries' slices in registers and reduces the four heads' lane products with a scatter that keeps warp_sum's pairings (xor 16 and 8 leave one head per lane, then 4, 2, 1): every addition adds the same two operands of the same level, so scores and top-k ids are bitwise the multi kernel's. qsa_rows_bench measures the window's QSA kernels row by row and, with --rows, checks the parity (16 cases per card, 0 differ) and prints both kernels' microseconds: 2080 Ti 47/57/78/101 -> 36/38/49/55 us at T 1-4, 3060 73/109/152/194 -> 63/76/98/130. docs/QSAROWS.md.
New requests took the control lines in whatever order the lock fell, so a 500-token chat question could wait
behind several agent reads. With the env, a prompt of at most that many tokens goes first (arrival order within
each class); a longer one that waited STRATA_ADMIT_MAX_WAIT_S (30 s) counts as short. A request already in
acquire() gives the lines back when a shorter one came. Only the order changes, never what is read.
Ported from havok2-htwo/Strata 08830ddf onto 0.1.44.1: _take_control has no image-request line here, so its
gate is only the ordering; the first admission passes new=True.
The refusal 'this CPU cannot run the expert kernel' belongs to the canonical Q2_0 pack. An NVFP4, i-quant or
Unsloth pack is a native pack (native_experts.txt in the --pack folder) and its rows have AVX2 kernels, but a
config whose --pack points at the wrong folder reads like a CPU that is too old. On a CPU without AVX-512 the
engine now says the file is missing, and where it sits when it is one level down (<pack>/pack).
A layer split hands each prompt chunk's rows (21 MB per 512 tokens) to the next card through a pinned
host buffer: one D2H, a host stream sync, a new std::async thread, then one H2D before the next stage's
first kernel. The 3090s here have no P2P (topo: CNS) and cudaMemcpyPeerAsync is driver-staged and
slower (3.4 ms vs 3.2 ms for D2H + H2D), so the copy stays on the host route; nsys showed 3.8-4.0 ms per
512-row hop, of which ~1-1.4 ms was the thread start alone.
With STRATA_HANDOFF_PIPE=P (2..16, opt-in; unset = unchanged):
- the sending stage records an event when its chunk's work is done and one after each of P row-piece
downloads into the same pinned buffer, and waits on the host only for the first (on_stage_chunk's
checkpoint parts and the error check need the state, not the copy);
- the next stage waits on the GPU for each piece's event (cross-device cudaStreamWaitEvent) before that
piece's upload, so its uploads overlap the remaining downloads (separate PCIe links); a smaller
receiving chunk waits for every piece, then copies as before;
- the next stage's chunks run on one worker thread per sending stage, started and pinned once, whose
packaged_task futures carry the same results and exceptions as std::async's (it clears the CUDA error
state after each job, as a thread's end did). STRATA_HANDOFF_PIPE_WORKER=0 keeps std::async (A/B).
Same bytes in the same rows before the next stage's first kernel: exact. No device memory (events only,
the existing two pinned buffers per stage). Measured (T16-143l5, P=8): 512-row hops 3.8-4.0 -> 1.85-1.89
ms, 143-row hops 1.6 -> 1.0 ms; reads 1.2K about -1.5 %, 150 about -1 %, 9K (agent, --kv-resident) -3 %;
decode rows/s unchanged. batch_test16/32 (greedy and temperature 0.7), batch_interleave_test and
batch_overlap_test (+ the stage fault arm) byte-/token-identical to the reference dumps; an agent-config
run with checkpoint parts byte-identical to the reference binary.
STRATA_SCORES_MULTI=0, the pre-sm_75 attention (STRATA_EXPERIMENTAL_SM60 builds on a
V100/P100) and the gfx906 query swizzle make qsa_block_scores_slots or
qsa_decode_attn_slots decline; the engine keeps its per-row calls there, so a decline
is no failure. A declined case prints SKIP, the test exits 77 when every case was
declined, and STRATA_SCORES_MULTI=0 skips up front.
In a batch window every row reads its own slot's indexer keys and K/V, so each QSA layer ran the
selection and attention once per row: four rows (16 slots) or eight (32 slots) times qsa_block_scores,
qsa_block_topk and qsa_decode_attn_batch, each launch small (the top-k on one CTA, the attention on
cap/64 x 2 blocks). With the switch, per window (up to 8 rows per launch):
- block_scores_slots_kernel: grid (256, rows), blockIdx.y runs the single-query block_scores_multi_kernel
body with that row's slot keys (block_scores_multi_kernel is now a wrapper around the same body)
- qsa_block_topk over the rows in one call (it only reads the rows' scores and steps)
- attn_chunk_slots_kernel: attn_chunk_kernel's body with the row's own QsaAttnPools (an 8-entry table
passed by value, blockIdx.z picks), then the existing batched merge
Each row is computed by the same kernel body with the same reduction order as its own call. The
launchers decline (the per-row calls run) where that call would pick another kernel: STRATA_SCORES_MULTI=0,
the pre-sm_75 / gfx103x attention, the gfx906 query swizzle. A streamed KV cache keeps the per-row
resolve-then-attend loop. Default (switch off): unchanged.
Checked on 4x RTX 3090, T16-143gsc setup (layer split 12,24,37, int8 KV):
- SASS (sm_86) of every existing kernel in qsa_select.cu and of the default attn_chunk_kernel<M> and the
merge is identical before/after; the opt-in lane-cell instantiations differ only in register allocation
- new qsa_slots_parity (ctest, also with STRATA_ATTN_LANECELL=1): fp16, int8, q4_0 and int8-K/q4_0-V
pools, 1/2/4/8 rows of different slots with their own page tables and contexts 37..32768: scores, ids
and attention bitwise identical to one call per row
- batch_test16/32 (greedy and t=0.7), batch_interleave_test and batch_overlap_test (STRATA_PIPE_OVERLAP=1)
token-identical to the build without the change; with the switch off batch_test16 is byte-identical too
- batch decode rows/s (engine level, 30 s, back to back with the reference): 16 slots 683 -> 726 (+6.3 %),
32 slots 750 -> 802 (+6.9 %); STRATA_VERIFY_PROFILE "QSA projections + attention" per stage-window
0.89 -> 0.52 ms (16) and 1.52 -> 0.79 ms (32)
HIP: the new code uses only what the existing kernels use; SYCL has its own sources and only sees the new
declarations. Neither was built here.
Upstream's non-finite flush only marks running slots `tainted`; their checkpoints stayed. With
STRATA_SLOT_CKPT_MOVE a running slot's checkpoints are a resume source (active_source) and a slot's
moved turn checkpoint can be restored into the main session (lent_slot), so a request right after the
flush, logged as "reading this prompt from token 0", still continued from a pre-flush checkpoint.
- the flush drops a running slot's checkpoints too (only with STRATA_SLOT_CKPT_MOVE, the one reader of
them; they go back to the checkpoint pool)
- active_source and the lent_slot search skip tainted slots
Checked with STRATA_NAN_INJECT=600 on the T16-142gsc setup (next turns of chats still running in
slots): before, the two requests after the flush resumed 1628 tokens from slots 0 and 8 ("gave back ...
its turn checkpoint"); now both read 1661 tokens from token 0. A next turn before the flush still
resumes from its running slot. Default path (switch off) unchanged.
On a layer split with more than one group a batch read never gives way to a
BYIELD, so step 4 always failed with 'no such free slot' and steps 5-7 never
ran. --yield auto now expects the give-way only on a non-pipelined batch;
there D is read whole and E admitted beside it, both still equal to solo.
New step 7b: A ends in slot 1, --flush unrelated solo GENs, then F2 can only
start from slot 1's turn checkpoint. --dump writes every token list for an
A/B of two builds.
mmq::whole_tiles walks every visible GPU and every tile width through the
MMQ config table; the sort hook called it twice for each MoE layer of each
prompt chunk, on the host path between the routed-ids download and the
expert launches. Its verdict depends only on the layer's weight formats and
the device set, so MmqPlan now keeps it as a per-layer flag (taken only when
STRATA_MMQ_GROUP_SORT=1 and STRATA_MMQ_RESIDENT_SORT_NE is unset) and logs
how many layers qualify. Same order, same bits; without the switch nothing
new runs.
A pipelined batch window is followed on its stage's stream by the layout's commit graph, and the stage counts
as done only after both. When every row of the window is another slot's one token, that commit is
unconditional (each slot keeps its one row; commit_slot_prefixes allows nothing else), and most of the
graph only repeats what the window wrote: it puts each indexer tail back from its snapshot and appends the
same row again, and copies each PLE history back from the window's own snapshot of it. What it adds is the
GDN conv history and recurrence state, which the window ran without writing.
With the switch, such a window advances the GDN conv history (gdn_conv_l2_multi with commit) and state
(gdn_step_norm_multi with n_keep = 1, the kernel the commit graph runs for a one-row group) itself, as a
one-token solo window does, and takes no tail or history snapshots. No commit graph is captured for the
layout; batch_launch and commit_slot_prefixes launch none, layout_captured and ensure_batch_layouts do not
ask for one. Layouts with proposal rows (--batch-mtp) keep the commit graph. With the switch off every path
is unchanged.
Measured on 4x RTX 3090, UD-IQ4_XS, T16-142fcb (split 12,24,37), decode of 16 / 32 slots, 30 s each,
reference and new build back to back: 653.7 -> 686.0 rows/s (+4.9 %) and 714.1 -> 752.6 rows/s (+5.4 %);
the group step period 24.5 -> 23.3 ms and 44.85 -> 42.6 ms. STRATA_VERIFY_PROFILE: the window's GDN
recurrence takes 0.09 ms (16) / 0.18 ms (32) longer per stage-window with the state written in it.
batch_test16 (16 slots, a 1,200-token prompt) is token-identical to the reference dump; batch_test32
32/32 identical and its dump equal to the switch-off run; batch_overlap_test (STRATA_PIPE_OVERLAP=1) passes
and is token- and RESUME-identical to the switch-off run; batch_interleave_test parts 1-3 identical (its
part 4 fails at the same point with the reference build).
STRATA_MMQ_RESIDENT_SORT_NE=1 puts a fully resident layer's experts in row-count order, so each
16-expert group's max_rows (its tile padding) is close to its experts' own. It also moves the hottest
experts into the last, partial group, whose few tiles take MMQ's stream-k split: the K boundaries move
with the group's tile count and the prompt's bits change.
The new switch sorts only order[0, n - n % 16] (mmq_resident_sort_span) and leaves the partial group's
experts, and so its rows, where they were. A full group's launch has 160 (gate/up) or 320 (down) tiles
per column tile; mmq::whole_tiles checks on every visible GPU that every such launch runs one block per
tile (tile efficiency >= 90 %, NVIDIA) and that all tile widths share the I/K tiles and SRAM layout. Then
each expert's rows get the same bits in any full group whatever its group-mates and max_rows. Where it does
not hold (e.g. 108 or 128 SMs, AMD's stream-k grid) the switch does nothing. RESIDENT_SORT_NE takes
precedence when both are set; with neither the code path is unchanged.
prefill_mmq_tab_test part 4 runs a layer's groups as prefill.cpp does (src-gathered q8_1 rows, absolute
bounds, the group's max_rows) in ID order and in both sorted orders, for IQ3_S / IQ4_XS gate/up and
IQ4_NL / Q8_0 down: with the partial group kept every token's rows are bitwise equal (16 cases); sorting
the partial group too changes 133-1881 rows per case. 890 comparisons, 0 failed (RTX 3090).
Measured on 4x RTX 3090, UD-IQ4_XS, T16-142fcb (split 12,24,37, every expert resident), read time,
medians of 6, back to back: 150-token prompt 145.8 -> 136.2 ms, 1,200-token prompt 428.6 -> 363.4 ms
(RESIDENT_SORT_NE: 134.9 / 358.6 ms). The final residuals of five prompts (300-1,200 tokens,
STRATA_PREFILL_DUMP_R_ALL every position and STRATA_PREFILL_DUMP_R fp32) are byte-identical to the
reference build; with RESIDENT_SORT_NE every row differs. batch_test16 (16 slots, a 1,200-token prompt)
is token-identical to the reference dump.
STRATA_SLOT_CKPT_MOVE: an active slot is a slot source only when the main
drafter's rows come along or are not drafted from (no --mtp, --batch-mtp,
or a batch admission). The slot copy takes the main layers' cells only, so
a solo GEN beside active slots would draft from another conversation's
drafter rows below its start; it reads its prompt as with the switch off.
STRATA_CKPT_THREADS: a checkpoint thread widens its CPU mask before it
goes to the aux CPUs, so the mask the aux placement restores (a request
with aux_cpus=0) is the wide one, not the host thread's single CPU.
STRATA_ADMIT_MERGE_FIRST: a merged first window counts as one row in the
STRATA_DECODE_TIMING window statistics (its prompt rows are not decode rows).
The direct slot copy (STRATA_SLOT_COPY_DIRECT) waited for each of its ~116 copies on the private copy
stream. With the switch, conversation_checkpoint_copy and conversation_kv_copy only queue them (new
`queued` argument, default off), and the slot copy's closing sync_all is the one wait per device. A
streamed K/V state (kv_mode != 0) has its residency refresh on the legacy stream, which is not ordered
after a non-blocking stream: conversation_detail::order_after_queued records an event on the private
stream and makes the legacy stream wait for it first. The copies stay on the private non-blocking stream.
Same copies in the same order on one stream: tokens are identical. Measured on 4x RTX 3090 (T16-142,
24 admissions): the slot copy takes 0.5 ms instead of 1.3 ms; batch_test16 16/16, batch_overlap_test
and batch_interleave_test token-identical to the references; a --kv-resident 20480 run with a 21K
prompt is identical with the switch off and on.
A batch admission copied the main chain's longest checkpoint into the slot (~112 MB of host vectors, 7.6 ms, ~60 ms
on fresh pages). With the switch, the request's own turn checkpoint (or one it took out of the slot it is admitted
to) moves instead; a root, a pinned prefix and any other checkpoint the chain shares are still copied.
The chain then no longer holds it, so the request path also looks in the slots:
- while the main session holds its tokens (live, its cells below it), a slot's checkpoint is restored on the main
session as a chain one would be, with no K/V copy; when that slot is the one the request is admitted to, it moves
back into the chain first (the admission drops the slot's checkpoints), else the slot keeps it and the admission
copies it as it copied the chain's;
- an active slot's checkpoints are a slot source like an idle slot's (its windows write its cells from p on): a
second request with the same history after the main session moved on restores instead of reading it all;
- a slot source that is the admit slot moves its checkpoint into the chain instead of copying it.
The switch also keeps the dropped checkpoints' storage as STRATA_CKPT_POOL=1 does: a moved checkpoint frees
nothing at the next prune, so without it every save faulted in fresh pages (17 -> 38-60 ms, more than the copy saved).
Off by default: nothing changes. A session SAVE or a parked conversation right after an admission lacks the moved
turn checkpoint (the slot holds it).
STRATA_CKPT_POOL=1 (off by default): the serve loop keeps the running-state
vectors of the checkpoints it drops (LRU eviction, the tail one, the prune at
a new request, a batch slot's old one), at most two, and the next checkpoint
or slot copy goes into them. A save then resizes nothing: no zero-fill of
~118 MB of fresh vectors. Every byte is overwritten by the copy.
STRATA_CKPT_THREADS=1 (off by default): a layer split's checkpoint saves the
first stage's part on the calling thread and every later stage's on a
persistent thread of its own (the copies' private streams are per thread, so
short-lived threads would leak them). The threads leave the host thread's one
CPU first (to the aux CPUs with --aux-cpus, else to every CPU the process may
use): on it the copies only took turns.
Measured on a 4-stage split (T16, 24 batch admissions per arm): a turn
checkpoint takes 17.2 ms today, 10.4 ms with the pool and 5.6 ms with both;
the batch tests, the overlap and interleave tests and a driver run give the
same tokens with the switches on as off.
A batch admission read its header through the verify windows (T=4 + T=2 for the
usual 6 tokens) and then its first token in a window of the last prompt token
alone. With the switch the last windowed part leaves its last tokens (up to the
verifier's max_t, 8, the last prompt token included) to the first window: the
part's earlier tokens are read as before, the first window commits every row,
the drafter gets the rows of the prompt tokens in it as the prompt windows give
them, and only row T-1's output is emitted. Row T-1 sits at the first window's
position with the same history and Philox draw, and a row computes what it
computes in any window while every expert is in VRAM, so the token and the
state are the same.
Only for a BGEN admission with every expert of every stage in VRAM (no loan,
no swap pending, the zero-doorbell graphs), and not with --batch-mtp, the
pipelined windows, pictures, the elastic K/V, a peer device, remote caches,
foresight swaps, STRATA_LOGPOS or a checkpoint at the last token. The merged
window takes no drafter q (STRATA_SPEC_PROB), adds nothing to the draft
counts, and the "PP <n-1>" line is printed before it. Verifier::max_t() is new.
Off by default: the same path as before. Tokens (4x RTX 3090, split
12,24,36, 16 slots, STRATA_IQ_MT_MIN=1): 24 admissions of prompts of 3 to 65
tokens and chat headers with and without an assistant prefill (merged T 3 to
8), greedy, seeded and with penalties, on vs off identical; batch_test16 16/16
and equal to the earlier dump; batch_overlap_test token- and RESUME-identical;
batch_interleave_test equal; 47 sampled driver requests equal.
Measured (24 admissions, ~131 and ~1195 tokens): header windows 23.5 plus
first window 11.5 ms become one 7-token window of 27.6 ms, 7.4 ms less per
admission (send to first batch token 246.8 -> 239.4 ms short, 656.1 -> 648.5
long). A window row costs ~2.4 ms up to T=6 and ~4 ms at T=7 and 8 (T=1 11.4,
T=7 28.6, T=8 32.1 ms).
A pipelined read never gives way (batch_read_piece returns 0 when piped), so a
BYIELD that reaches the main loop is for a read that has ended and is dropped.
The main loop still drained the groups for it first, as for a request, unless
STRATA_PIPE_OVERLAP was set. With the switch the BYIELD is dropped as it comes.
A BYIELD the server sends during the read is queued and taken right after BADM,
before any group is restarted, so that drain costs nothing today (0 groups in
flight, 0.0 ms). Only a BYIELD that arrives once the groups run again (the
server reads the control lines late) paid for it: measured with one sent 60 ms
after BADM, drain 26 ms with 4 groups in flight and the first rows 50 ms after
it, a 23.5 ms gap in the slots' rows and about 14 rows lost (at 576 rows/s). With
the switch the longest gap is 7.1 ms, as in plain decode.
Off by default: the same path as before. Tokens: batch_overlap_test (a BYIELD
during B's read) on vs off token- and RESUME-identical, batch_interleave_test
output byte-identical, batch_test16 16/16, and 41 sampled requests of the
admission driver (8 with a BYIELD while 4 groups were in flight) equal.
One stderr line per batch admission ("strata trace: admission of slot N: ...")
splits it into the drain before BGEN, the prompt (batched read, header windows,
the checkpoints' device syncs, copies into RAM and eviction, the rest), the first
window, the slot copy, freeing the slot's old state and the copy of the turn
checkpoint into the slot. Each checkpoint gets its own line, and with pipelined
groups "first rows N ms after the admission" times the refill after BADM. A drain
for another line says how many groups were in flight and what it cost.
Only with STRATA_TRACE set; without it the log and the tokens are unchanged.
Measured (4x RTX 3090, split 12,24,36, 16 slots in 4 groups, 24 admissions):
drain 25-26, header windows 23.5, checkpoint 17.3 (sync 0.1, save 17.2), first
window 11.5, slot copy 1.3, slot freed 0.0, checkpoint copy 7.8 (60 on a slot's
first fill), refill after BADM 23.7 ms.
The >8-slot cap ran only before the slot sessions were carved. When fewer
sessions fit than asked for, the groups were resolved again with no row bound:
auto could pick a group of more than kVerifyMaxT slots (PGroup's rows overrun,
every window refused) and a --batch-groups N that no longer divides the slots
fell back to one serial group whose windows reach only the first 8 slots.
slots_in_windows() picks the most slots whose groups, resolved as
batch_groups::resolve does (a number given, auto, the default's one group for
--adapt-async 1), divide them with at most kVerifyMaxT rows each and at most
one group's worth per stage. It runs before the groups are resolved and again
on the slots that fit (the extra sessions are dropped like the per-stage trim);
the pump refuses a group over kVerifyMaxT rows as a safety net. The hand-off
buffers hold one row per slot only for pipelined groups, so --batch-mtp keeps
upstream's kVerifyMaxT rows. Log and comment wording without the g3 tags.
- The build option is off by default and why (36-38 MiB of VRAM per GPU under EAGER, ~70% of the upstream MMQ
instances' compile time per arch); the nine types with a pointer table.
- The skip-gather hack's 225/577 ms are no bound: it changes the routing and does less work.
- Engine exactness as measured: first-window logits (STRATA_DUMP_FIRST_LOGITS) and greedy tokens byte-identical,
2 vs 1 vs 0, on the 4-GPU split with --prefill 2048 (the streamed walk), with 9 GB less expert cache per GPU
(mixed in-place, copied and mid-group-flushed groups), and on one GPU with --expert-cache-per-layer.
The file mirrors parts of llama.cpp's ggml-cuda mmq.cuh; like the other
ggml-derived kernels (native_moe.cu etc.) it now includes the copyright and
permission notice the MIT license asks for. Comment only.
- ExpertCache::slot_guard() returns the guard open_sized applied (guard_, reset by close), not the request: 0 for
uniform slots, when closed, and on SYCL, whose open_sized applies none.
- tests/core/expert_cache_slot_guard_test: slot_stride, slot_room and slot_tail on a uniform cache (empty off_,
including the last slot) and on sized caches with and without the 256 B guard; no guard gives today's offsets;
close and reopen.
- generate: the "+256 B guard per sized slot" line only for a sized cache; a uniform one says it has no guard.
- prefill: STRATA_MMQ_DIRECT_LOG=1 (with STRATA_MMQ_DIRECT=1) prints each stage's totals of experts read in place,
resident and streamed after every prompt, so an engine run can show which walk read what.
Review fixes for the pointer-table MMQ build:
- STRATA_MMQ_TAB defaults to OFF, like STRATA_MMQ_KQUANTS: the engine loads every kernel at start
(CUDA_MODULE_LOADING=EAGER), and the nine tab instances took 36-38 MiB of VRAM per GPU (sm_86, measured with
a cudaMemGetInfo probe linked with and without the tab objects; equal under LAZY), with STRATA_MMQ_DIRECT unset
too. prep_group_kernel is compiled only with STRATA_MMQ_TAB, so a build without it has the same device code as
3adce85 (the 58 cubins and PTX of build/strata are byte-identical to engine/strata-g3b's).
- The fingerprint block and the tab TUs are skipped on STRATA_HIP_GFX906: that build compiles no .cu from here, so
the define, the uncompiled TUs and an unlinkable prefill_mmq_tab_test no longer appear.
- Tab instances only for Q2_0 IQ2_XXS IQ2_XS IQ2_S IQ3_XXS IQ3_S IQ4_NL IQ4_XS Q8_0, the types prefill_mmq_tab_test
checks bit for bit; Q5_0 and the K-quants (STRATA_ORCA_Q4KS_MMQ, STRATA_MMQ_KQUANTS) keep the gather.
- overread_bytes takes the largest over-read of any config, not the over-read of the largest K tile (the over-read
is not monotonic in the tile length).
- The test fails (instead of skipping) when a tested type has no tab, checks that Q5_0/K-quants report none, and
pins the sm_86 over-reads (0, 72, 136 B); the dead K-quant scale branch is gone.
- SYCL's moe_mmq.dp.cpp gets the STRATA_MMQ_DIRECT stubs (false/0), for a future migrated prefill.cpp.
- moe_mmq.hpp points at docs/DETAILS.md.
MMQ loads a whole tile's ids, and its stream-k fixup one per column tile of max_rows, past an
expert's last row (upstream behaviour on both paths; the values are not used). With the map
padded, compute-sanitizer memcheck runs the whole test clean: 0 errors, 874/874 bitwise equal.
A pointer taken from a dynamically indexed array parameter (mmq_x_tab by value) made nvcc copy
the table to local memory and read the weights through generic LD.E loads instead of
upstream's LDG.E.*.CONSTANT: the products ran 1.2-2.5x slower than upstream's in the unit test
(Q8_0 down 2.2-2.6x, IQ4_NL 1.7-2.1x, IQ3_S 1.2-1.35x), which ate most of the gather saving.
The kernel now takes x0..x15 as `const char * __restrict__` parameters (like upstream's x)
and picks an expert's with a select chain: same SASS load mix as upstream (no local memory),
kernel time within 1% of upstream over every type and J in prefill_mmq_tab_test (69.7 vs
69.3 ms in total), and still 874/874 comparisons bitwise identical.
mmq_plan marks a native layer direct when the pointer-table MMQ covers both products, gate/up
reads nothing past its matrix, the down over-read (72 B IQ4_NL, 136 B Q8_0) fits the 256 B
guard, and the blob is exactly [gate | up | down] in MMQ's matrix sizes; it says once how many
layers qualify.
In the MoE walk (both the staged and the streamed walk) compute() takes the expert's cache slot:
a resident expert whose slot has room for the over-read (slot_tail) and a 16-byte aligned blob
is not gathered - the group's pointer tables point at its gate|up and down in the slot, and the
guard goes into the group's GuardTab. Streamed, peer and unguarded experts are gathered as
before, at compact positions c = 0..ns-1 (gather_native, or gather_native_group with its
mid-group flushes unchanged); the end-of-group flush runs even when the last entry was direct.
At the group end one prep_group launch zeroes the direct experts' guards and the tails after
position ns, replacing the two memsets; both products run with the same n, bounds, ids,
max_rows and buffers plus .tab. STRATA_MMQ_DIRECT=0 (default) and 2 run the existing body.
PrefillStats::experts_direct, said on the generate summary when non-zero.
ExpertCache gets inline set_slot_guard / slot_guard / slot_stride / slot_room / slot_tail; the
4-argument open_sized declaration is unchanged (the SYCL engine defines it out of line against
the same header and ignores the guard). open_sized lays slots out at slot_stride(blob, guard);
blob_ stays the largest blob, so fills, verify_slot and the HIP staging read no further.
slot_tail works on uniform caches too (through slot_offset).
generate.cpp: slot_guard = Prefill::slot_guard_bytes() (mmq::direct_slot_guard(): 256 when
STRATA_MMQ_DIRECT is 1 or 2 on a pointer-table build, else 0) for native packs; the seven
round-256 slot costs (split-skip-if-fits, split placement, per-layer sum, profile sizing,
shrink_to, cache_bytes, stage sizing) use ExpertCache::slot_stride(b, slot_guard), and the
primary and stage caches get set_slot_guard before they open. Guard 0: the same arithmetic as
before, so the default layout is unchanged byte for byte.
A Strata-side copy of llama.cpp's MMQ outer scheduler (mul_mat_q, launch_mul_mat_q,
mul_mat_q_switch_J; fallback=false only) in src/prefill/moe_mmq_tab.cuh: each expert's base
address comes from a 16-entry table passed by value instead of x + z * stride_channel_x.
The launch geometry (J, tiles, stream-k grid and k split, fixup) is computed from the same
mmq_args, and process_tile / load_tiles / vec_dot / write_back / the fixup kernel are
upstream's own templates, so the results are the same bits. Instantiated per type from
mmq_tab_instance.cu.in in strata_mmq (same flags); moe_mmq.cu sees declarations only.
CMake turns it off unless the SHA-256 of mmq.cuh's mirrored region (mul_mat_q_process_tile
through mul_mat_q_case) is a verified one: 3cf03257 (the fetched pin; the vendored tree's
mmq.cuh is byte-identical). HIP and SYCL get nothing; Product::tab is a default-null field.
Also: overread_bytes, tab_fits, direct_mode (STRATA_MMQ_DIRECT, read once, 0 unless the tab
is built), direct_slot_guard, and prep_group (one launch: zero the direct experts' guards and
the two group tails). Nothing calls them yet; the default path is unchanged.
tests/cuda/prefill_mmq_tab_test.cu (STRATA_BUILD_TESTS): 874 comparisons on an RTX 3090, all
bitwise identical - IQ3_S gate/up, IQ4_NL and Q8_0 down (1..16 experts, rows
{0,1,7,8,9,31,64,100,512,1000}, every J 8..128, stream-k fixup on and off, max_rows above the
largest count), every other instanced type, mixed direct/gathered groups (compact c != q,
mid-group flush, last entry direct); negative control: guards left as fp16 NaN give non-finite
down outputs.
The main sessions and the batch slot sessions of a stage live on the same device and are
carved alike, so copy_to_slot / copy_from_slot (without a checkpoint) now copy the running
state and the QSA K/V directly (conversation_checkpoint_copy, conversation_kv_copy:
the snapshot code's private non-blocking copy stream, one device sync before and after) instead of the
checkpoint/K-V image round trip through host memory with a sync per layer. Same bytes into
the same places in the same order; the residency refresh for streamed K/V is shared with
conversation_kv_restore. Opt-in: STRATA_SLOT_COPY_DIRECT=1 (default: the host path,
as upstream).
4x RTX 3090, UD-IQ4_XS, --batch 16, split 12,24,36 + --trim-stage-weights: admission copy
65.6 -> 34.0 ms average (pure copy ~8 ms); tools/batch_test16.py 16/16 and
batch_interleave_test.py (incl. next turn from its slot) identical to solo.
Measured on 4x RTX 3090 (UD-IQ4_XS): 16 and 32 slots decode bit-identical to solo
(tools/batch_test16.py / batch_test32.py); 16 slots 603 tok/s batch decode vs 345 with 8.
Rebased on 0.1.42: the stages' batch row check (batch_rows_error, upstream 48f2c47d) bounds
hbase + S by the hand-off rows; set_handoff_rows passes the per-slot size there (only above 8 slots).
On a Mac the Monitor tab's GPU cards were empty: telemetry had NVML (NVIDIA) and the amdgpu
sysfs files (#301) and nothing else, so the Metal backend read nothing. The Apple GPU's load
and memory are in the IOKit registry's IOAccelerator nodes (ioreg -l -w0 -r -c
IOAccelerator, 22 ms), and the device's memory budget is the same Metal property the engine
reads (metal/src/mtl_rt.mm's recommended_working_set, the engine's "GPU working set" log line
- 26,800,603,136 bytes here, which is 25.0 GiB).
_Metal (a new reader beside _Nvml and _Amd): the index'th accelerator node's PerformanceStatistics
give Device Utilization % (older chips: the renderer or the tiler) and the in-use system
memory; the total is the working set, read through the Metal framework and objc_msgSend (no
PyObjC). The temperature and power sensors are not readable
without root (powermetrics), and an Apple Silicon GPU sits on the die, so there is no PCIe
link: the cards show a dash and static gpu_note says why, the same way #1380's note does for a
Windows AMD card. gpu_reader takes `amd` first, so the AMD trees still read on a Mac for the
tests, and a missing selector is asked for before it is sent (an unanswered one raises an
Objective-C exception no ctypes call can catch).
Service.free_vram_mib now goes through gpu_reader instead of _Nvml directly, so the Mac gets
that reading too (it used to return None there).
Measured on an M1 Max (macOS 26.6.2): through the server, GET /metrics during a request -
gpu_name "Apple M1 Max", gpu_vram_total_mib 25559, gpu_util 74-77 % while decoding (9 % once
idle), gpu_mem_used 10.5 GiB in use of 24.96 GiB, cpu 12.6 %; the plain-text
/prometheus-format metrics carry the same series. serve/test_server.py: MacTelemetry
(fixture registry: two accelerators, index selection, renderer/tiler fallback, a raising
runner, the note and the samples) plus AmdTelemetry, TelemetryNoDisks, Monitor and the full
serve/test_server.py (318 tests) pass on the same box.
What it leaves alone: the NVIDIA and AMD paths (a test-only gpu_reader order change on
macOS), and the web app - the cards read the series it already had.
Co-authored-by: Demetree Kallergis <[email protected]>
setup.py returns from mac_setup() before its own argparse runs, so a Mac never
sees the --source that sets STRATA_SOURCE, and the Mac parser did not define it
either: the flag the download failure recommends landed in parse_known_args'
leftovers and was dropped with "not used on a Mac".
Define it in the Mac parser and set STRATA_SOURCE from it, mirroring setup.py.
Recorded with the hardware they were measured on, per the measured-numbers rule, because
neither can be fixed here: mm_gu3's fault is a threadgroup-count threshold of the
cooperative-tensor path on this GPU (the fix shape would change the M5's register-operand
GEMM), and router_route's race has no acquire atomic to fix it with on this Metal
(device-scope atomics are relaxed-only, macOS 26.6). Both are opt-in and off by default;
the engine's M1 defaults are the v2 expert GEMMs and the separate gemv + moe_route, which
run on this box.
Measured on an M1 Max (macOS 26.6.2, Q2_0): mm_gu3 with 35 threadgroups passes and 40
faults (data-independent; the engine's tile 16x32x32 SG 4 and the v3 test's 190 tiles both
fault). router_route's logits are bitwise gemv's, but two identical runs give different
ids, weights and plan.
Co-authored-by: Demetree Kallergis <[email protected]>
Two stale defaults, both of which kept the test from covering what the engine ships.
The default tile was TM 16 TN 64 TK 16, but the v1 kernel set only instantiates TK 64: the
default run died at pipeline creation on any GPU (measured on an M1 Max:
"mtl: no function mm_gu_42_16_64_16_2"). TK is now 64, which both the v1 and the v2 sets
instantiate, so the default run and --v2 both work; the engine's v2 tile for Q2_0
(16,64,16,2) stays reachable with --tile.
--v3 bound the v1 kernels' argument list ({W, x, H, stok, tiles, ma}) to the v3 kernels:
tiles was read as the sorted-token buffer and MmArgs as raw int2 pairs, so every stride,
n_embd and expert id was garbage. On the M1 Max that is a deterministic GPU
command-buffer error in the GEMM pass; with the right arguments the v3 kernels run, and a
separate M1 limit shows (≥40 threadgroups of mm_gu3 fault, 35 pass, data-independent)
that the engine does not hit: gu_v3 = dn_v3 = coop_k4, which is false on the M1.
v3 now binds {W, XS, HS, tiles, ma} as the engine does, fills XS for v3 too (v3 is v2's
gather plus the register-operand GEMMs), and binds HS - not H - to down3 with DSG
simdgroups.
Measured on an M1 Max (macOS 26.6.2, Q2_0, T 256, 64 experts): default (v1) PASS, rel L2
2.53e-06 vs the fp16-operand reference and 2.37e-04 vs exact; --v2 PASS, 4.42e-06 and
2.37e-04, also with the engine's tile (16,64,16,2).
Leaves alone: no kernel, engine or other test change.
Co-authored-by: Demetree Kallergis <[email protected]>
qsa_attn_mm and qsa_flash (metal/kernels/qsa_prompt.metal) fill their single-simdgroup
matmul2d right inputs with
rk[i] = id >= 0 ? half(float(kq[row * QD + k]) * float(ksc[...])) : half(0.0f);
and that shape - a select with the device load in the arm - loses the threadgroup on an M1
Max: every dispatch of the two kernels fails with
kIOGPUCommandBufferCallbackErrorInnocentVictim, deterministically, and they run only under
MTL_SHADER_VALIDATION=1. qsa_prompt_parity dispatches qsa_attn_mm the way the file intends
(it is the matrix-unit attention of a prompt chunk) and faulted on its first dispatch before
this change. The engine reaches the same fills through qsa_flash, which it dispatches under
STRATA_METAL_PROMPT_QSA=flash; its default on an M1/M4 is the fp32 qsa_attn, which never
touches these kernels - that is why the default prompt path kept running while the parity
test could not, and why this is measured on the test rather than on a request.
The row is now clamped with max(id, 0), the load is unconditional, and the select is on the
value. For an id >= 0 the value is what it was; for an id < 0 it was discarded either way, so
the results do not change.
Measured on an M1 Max (macOS 26.6.2, Metal compiler 32023): qsa_prompt_parity ALL PASS -
qsa_attn_mm vs qsa_attn rel L2 3.09e-04 (pos0 0) and 3.56e-04 (pos0 30000), qsa_flash 3.72e-04
and 4.22e-04, qsa_score_mm worst rel 2.66e-06 (T 96, within budget and with 2051 selected
cells a token); before the change the test faulted on its first dispatch and reported nothing.
The compile failure that started this - qsa_flash's cooperative_tensor copy, "cannot
initialize a parameter of type 'void *' with an rvalue of type 'const void *'" - is already
fixed in main (#2323); this change is the runtime half of the same two kernels' M1 Max
bring-up.
What it leaves alone: the COOP_K4 path (a function constant: compiled out of the M1/M4
pipelines), qsa.metal, and every host file; no CUDA, HIP or SYCL backend builds these kernels.
docs/MAC.md records the M1 Max with the measured tok/s (the hardware's medians, from the
benchmark report: chat decode 10.6, 4K prompt 223.7, 16K prompt 228.6 tok/s).
Co-authored-by: Demetree Kallergis <[email protected]>
stream.cpp: the standalone-mmap fallback (the prompt's buffers as their own allocation) sat inside the chunk loop behind the slot-class check, so it only ran when a chunk was big enough for the prompt's 1.60 GB of buffers. When the arena shrank below that at the start (the 24 GB M2: chunks of 0.8-1.0 GB), every chunk was skipped and the prompt stopped with 'no arena region for the prompt buffers'. Both standalone sites - a fitting chunk whose slots a class cannot spare, and no fitting chunk at all - now share standalone_region(); the fitting-chunk carve and the slot-class rule are unchanged.
This is the half of #2323 that 0.1.44.1 could not see: the hotfix's reporter ran a 96 GB M2 Max, where the arena's chunks are several GB and the prompt region always fits. On a 16-24 GB Mac running --mmap-experts, 0.1.44.1 still stops at the first prompt with 'metal prompt: no arena region for the prompt buffers'; this makes the standalone allocation run there too.
Verified on a 24 GB M2 (macOS 26.6.2, Qwen3.8-Flash-Next GSQ-RCO-Q2_0 streaming): the fallback line appears where the error used to be, prompts run end-to-end (3.7-4.1 tok/s steady in web chat, no crashes). That run predates the standalone_region() refactor, which moves the same statements into one function both call sites use.
docs/MAC.md: the small-arena note says the prompt's buffers also get their own allocation when no chunk of the arena is big enough for them.
strata_bench starts the server on the config's `host` but polled readiness at
a hardcoded 127.0.0.1, so a LAN bind (192.168.1.10 in the report) printed
"still loading" while the server was answering on that address.
Use the configured host, mapping the wildcard binds to loopback and bracketing
IPv6 literals; an explicit --url still wins. Patch from the report, with the
mapping lifted into server_base so a test can reach it.
/metrics/prometheus exposes llamacpp:prompt_tokens_seconds and
llamacpp:predicted_tokens_seconds as averages of the cumulative totals, so an
idle server keeps advertising the speed of its last request.
llama.cpp reads those two names from a bucket it resets on every scrape ("the
gauges are averaged over the window between two scrapes", server_metrics::bucket
in tools/server/server-common.h) and reports 0 when the bucket is empty, so a
llama.cpp dashboard on Strata froze at the last value where llama.cpp decays to
0. Keep a bucket per scrape and expose its delta.
An earlier commit capped the accepted drafts at the end of an answer; that is an exact state, but it moved the hybrid mode's expert-slot
counts, so a sequence of requests with --mtp on a card that does not hold every expert differed from the base engine from the fourth
request on (found by comparing the 10 parity prompts against the base goldens on the 12 GB card). Now the speculative step commits what
the base engine committed; a stop seen before the state moves keeps only x (the server's hand-over of a stopped request to a batch slot then
reuses the held state: its text equals the run alone), and the held tokens list x and every accepted draft, so a conversation that continues
from them matches the state exactly. Setup says what parallel measured on the 12 GB card; docs/QWEN36.md updated.
spec_step takes no draft past an end-of-answer token or past the room max_new leaves, and a stop asked for before the state moves keeps
only x: every token emitted but the last is in the state, so a conversation that continues from the held tokens (the server's solo ->
slot -> solo hand-over, a chat's next turn) neither misses nor reads a token twice. Before, an answer ending inside accepted drafts left
the state ahead of the held tokens (found by tools/qwen35/q36_serve_handoff.py: the first request's text now equals its run alone).
STRATA_Q36_READ_WINDOWS sets the decode windows of the other slots between two chunks of a prompt read (default 1). The mtmd dump takes
several pictures; the engine logs each request's reuse; a 1,024-token test picture.
upload_f32 runs on the host and widens F16 with
__half2float(__ushort_as_half(...)), but __ushort_as_half is a __device__
intrinsic: nvcc 12.1 and 12.4 refuse the call ("calling a __device__ function
from a __host__ function is not allowed") and q36_engine.cu does not compile.
Widen the bits here instead, the way the neighbouring bf16_to_f32 already
does for BF16.
0.1.44 made IQ3_S the recommended Qwen size on a PC with 64 GB or more (IQ3_S: 9 to 15% slower decode than IQ3_XXS, more RAM).
SETUP.bat / --setup on a folder that already holds a model of the family now pre-selects the installed size (and --yes takes it);
the menu marks it "(installed)" next to "(recommended)", every size stays a choice, a first install takes the recommendation.
SETUP_ENV gained STRATA_MMQ_KQ, STRATA_HC_Q8 and STRATA_ROUTE_TAIL_SKIP, which setup writes for Unsloth's files only; a setup
run again dropped a user's own value of them in any other config (a hand-set STRATA_ROUTE_TAIL_SKIP=0). They are setup's only
in an Unsloth config.
/v1/responses (Codex) read only the Chat settings shared with apps, so the config's "reasoning_effort" (#1641) never applied
there (it always thought at xhigh); Service.effort_default() is now the one place that says which level a request that names
none gets, for the chat, Messages and Responses routes. openai_to_messages dropped "reasoning_effort": false (an `or` between
the two fields); effort_kwargs already treats false as thinking off.
The new test renders the prompt of every way a client sets the level (OpenAI reasoning_effort / reasoning.effort /
chat_template_kwargs, Messages API output_config.effort / thinking budgets at the 2048 and 8192 limits / disabled, Responses),
the config key, the shared Chat settings and their precedence, the web app's request shape, tools and a conversation.
Setup: the images question for qwen36 (GPU encoder on an NVIDIA card of 16 GB or more, the CPU encoder otherwise, AMD under Linux on the
CPU; --vision gpu kept as asked with a note), the pinned mmproj (size and SHA-256) beside the GGUF, the vision section and --vision in
the config; --parallel N is written as asked with what a slot costs, and the recommendation is said where every expert still fits.
docs/QWEN36.md: pictures and several requests at once with the measurements.
A slot's window carries its next token and, for the greedy slots that fit beside one row per slot, the proposal of its MTP block;
the proposals for the next window come right after the window from its own hidden states (the first from the state the prompt
left). Grouped GDN rows (a slot's rows in sequence on its own state), verify windows and their commit run per slot; every slot's
tokens equal its solo run. The engine also says what the sessions take and stops with the numbers when they cannot fit.
One conversation state per session; --batch N keeps N of them and decodes one token of each in a single window (rows of
several sessions in one forward, bit-equal to the solo rows), reads a prompt between the other slots' windows and takes a
conversation over from the session that holds its start. GENI / BGENI pictures: the encoder's rows replace the embeddings
of the <|image_pad|> cells, positions as llama.cpp's mtmd (interleaved M-RoPE 11/11/10, text after a picture continues at
p + max(nx, ny)). tools/qwen35/q36_mtmd_dump.cpp is llama.cpp's side of the picture check.
0.1.43's gate already had 'not on a layer split' as its first predicate; it printed the line only when the variable was set. A default 0.1.43 split ran without the overlap, so 0.1.43.1's opt-in default changes nothing on a split. Measured on 2x A4000 (IQ3_XXS, layer split, 5 interleaved pairs): v0.1.43, v0.1.43.1 and v0.1.43.1 + STRATA_ADAPT_OVERLAP=1 decode within 0.6% of each other.
On Windows cudaHostRegister of the whole arena fails (the WDDM shared budget is about half the RAM) and the
sliced fallback registers the first ~30 GiB: with Qwen3.8-Flash-Next IQ3_S layers 0-31, so layers 32-47 never
get a PCIe share, while part of the registered prefix holds GPU-resident experts that DMA never reads.
STRATA_PIN_BY_NEED=1 (the arena on one GPU):
- every blob gets a page-aligned stride (PadMap in pinned.hpp, ArenaExpertSource::arena_off, +0.15 % RAM), so any
set of blobs can be registered exactly; the loaders place each blob at its padded offset
- the expert profile's pairs past the estimated GPU-resident count are merged into ranges by rank and registered
before any page is touched, up to STRATA_ARENA_PIN_GIB (29 GiB by default on Windows) or the first refusal
- pinned() / device_alias() / pcie_layer() work on the ranges (a blob is pinned only if wholly inside one);
lock_resident_ranges() keeps the holes between them resident
- one start-up line: "pin by need: N ranges, X GiB pinned, Y % of the non-resident experts"
STRATA_PCIE_MIN1=1: the PCIe share (nmiss * pcie_num) >> 8 rounds to 0 at 1-2 misses per layer; with two or more
misses one goes over PCIe (when its blob is pinned).
RTX 3090 + Ryzen 5 5600X: 28.96 GiB registered = 92 % of the non-resident experts, all 48 layers with a PCIe
share; prompt read +5-7 % at 56K-117K tokens, decode level alone and +2-3 % with --adapt-async 1 and
--pcie-frac 0.6 (bench/results/2026-10-11-community-rtx3090-5600x-arena-pin-by-need, docs/DETAILS.md).
Both opt-in; without them the layout, the loaders and the sliced pin are those of v0.1.42.
In the arena mode the blocking adaptive tier creates and joins a thread every verify window and the next window
waits for its copies (#1863: the time outside the decode timing breakdown). --adapt-async 1 now also runs with the
arena on one GPU, not only in the resident RAM mode. The arena holds every expert, so a round is a copy-in only:
no copy back, no RAM move, no exchange buffers.
- gates for the arena; ArenaExpertSource::blob_at() reads a blob from the job thread without the read counter
- the evicted slot is marked non-resident and uploaded before the round's copies are posted; an expert becomes
resident only once its copy's event has completed
- STRATA_ADAPT_ASYNC_WAIT=1 (arena default): the next tick waits for the round, so the window an expert turns
GPU-resident does not depend on timing; STRATA_ADAPT_PICK_INLINE=1 (arena default): one round per window;
STRATA_ADAPT_ASYNC_BATCH=N caps the copies per tick (default 0 = all); arena rounds are not capped at 96
- adapt_quiesce(): every K/V grow, kv_quiesce and the VRAM command land the round in flight (they only called
apply_pending(true)); a failed round still drains its copies. This also covers the resident mode's async tier.
- STRATA_DECODE_TIMING: "adapt wait X join Y, Z swaps/window"; --adapt-async help text
RTX 3090 + Ryzen 5 5600X, cold prompts: x1.07-1.11 decode, faster on 12 of 12 prompts; 25.9 -> 21.6 ms per window
(bench/results/2026-10-11-community-rtx3090-5600x-arena-async, docs/DETAILS.md).
Opt-in (--adapt-async 1, default 0); the default path is that of v0.1.42. With the tier on, answers are not
bit-exact run to run (which window first computes a swapped-in expert on the GPU).
The GPU's own wiring of the arena ends seconds after its last command buffer, and a Metal residency set did not hold it either
(wired memory 75 -> 3 GB between requests, measured with vm_stat and a probe). Idle, the arena is ordinary anonymous memory, and a
20 GB file written beside it made macOS compress tens of GB of it: the third panic (wired 2.4 GB, file cache 50 GB, compressor 1.9M pages).
mlock keeps it wired (STRATA_METAL_WIRE_ARENA=0/1, default on from 64 GB; not when the fill stopped early). Same write, arena wired,
full speed: compressor +0.7M pages (the engine's non-arena memory), 408 pages swapped, nothing else.
docs/MAC.md: the footprint cap, the wiring and the new variables.
Both panics (watchdogd starved 92 s) had the streaming engine at 95.3 / 89.9 GiB resident, free memory 13 MB, the compressor at 2-4M pages.
macOS keeps a file-cache floor (vm.vm_page_filecache_min, 37.2 GB on the 128 GB M5 Max) and compresses anonymous memory instead of dropping
file cache below it; the arena is anonymous and the GPU wires it only while a command buffer runs, so during the fill and when idle it is
pageable. Measured with a 1 s memory watchdog: old build + 100 GB arena, or auto arena + 40 GB file write + a 16 GB program: compressor 21k -> 4.07M
pages in 10 s, 29,000 pages swapped, pressure warn. The 67% rule counted only cx.allocated() (7.5 GB) while the process held 11-15 GB more, and
67% was too much anyway.
- arena_budget.h: footprint_cap (anonymous memory: arena + Metal buffers + what the process holds, task_info phys_footprint) = 60% of the RAM on
a Mac of 64 GB or more (and at least the file-cache floor + 12 GB left); resident_cap (a resident pack's mapped file pages, measured: UD-Q4 resident,
wired 91.7 GB, 40 GB write + 16 GB program, nothing compressed) = 70%; STRATA_METAL_LEAVE_SHARE for measuring. The explicit
STRATA_METAL_ARENA_GB is kept and flagged when it is over the cap.
- window.cpp logs the planned total (arena + held + n-gram reserve against the cap) and the footprint after the fill.
- the engine refuses to start while a repack or setup's model preparation holds a lock beside the model (STRATA_METAL_IGNORE_REPACK=1 overrides).
- metal_repack.py: unbuffered write(), F_NOCACHE on the shards and the output, fsync every GiB, a pid lock (file cache stays flat; the old tool filled all 128 GB).
- setup_mac.py: no engine while preparing a model (stops setup), start() waits on a live repack lock, resident rule = the engine's resident_cap.
- tests: arena_budget_test.cpp (47 checks), tools/test_setup_mac.py (repack guard).
- CMake project version 0.1.44; setup MIN_ENGINE (0, 1, 44); NVFP4_ENGINE (0, 1, 44) (UD-Q5/Q6 were already 0.1.44).
- setup: a prebuilt zip without strata_kq leaves no stale library next to the new engine (kept in .previous); the
Dockerfile copies libstrata_kq.so into engine/.
- Model menu: families in the order Qwen (GSQ-RCO), Swift, Coder, Unsloth, NVFP4, Qwen3.6; NVFP4 is not listed on AMD
and Intel (the Unsloth line names UD-IQ4_XS there); on NVIDIA its line says native FP4 (RTX 50) or that UD-IQ4_XS is
faster on this card, with the download and RAM need. The Unsloth family lists its five sizes, images on.
- strata_mcp's built-in tables know the NVFP4 sizes and family. tools/test_setup_menu.py.
Variant 6 of gdn_recurrence_variant (sycl/include/strata/prefill/gdn_xe.hpp): the column kernel's four row groups
and FMA chains in the same order, a 16-lane sub-group holding four columns x four row groups, so the cross-row-group
sums are sub-group shuffles and a token needs one work-group barrier instead of five (256 GRFs a thread).
Arc A770 16 GB (xe, oneAPI 2026.1 JIT), same day, a0116b27 + #2190 + #2193 with and without this commit:
gdn_parity recurrence 51.0 -> 24.5 ms at 8,192 tokens, 204-236 -> 110-113 ms at 32,768 (48 heads), 0 mismatches
(state, FP32, FP16). Coder IQ1_M, 128K context, int8 KV, --resident-experts: prompt 386 -> 396 / 407 tok/s at 8K,
478 / 480 -> 511 / 518 tok/s at 32K (2 runs each), the same output ids in every run.
On by default on Alchemist only (strata::intel_gpu_gen); every other card keeps the column kernels, unchanged.
STRATA_GDN_XE=0 turns it off, =1 turns it on for any Intel GPU with 16-lane sub-groups. gdn_parity compares
variant 6 with variant 0 bit for bit (state, FP32 and FP16 outputs) on any Intel GPU and skips elsewhere;
ctest -R prefill_gdn_parity runs only that part.
On the A770 the multi-token GR read's kernels launch too few work-groups to fill the card. With
STRATA_GR_MORE_GROUPS on:
- sliced down kernel: one work-group per (slice, row block), 80 x 6 = 480 groups instead of 80;
every row's partial sum is computed exactly as before, only by a different group.
- gr_down_reduce: 8 slice loads in flight per step, then added one at a time, so x accumulates
slices 0, 1, ... 79 in the same sequence of float adds as the old loop (-fp-model=precise).
- up kernel: 8 columns per group, 320 groups instead of 160 (the kernel is now templated on the
column count; UPM_COLS stays 16 for every other user), and only group 0 writes the per-token lo
values, which every group computes identically.
Default on Alchemist only (strata::intel_gpu_gen); Battlemage and other devices launch the same
kernels as before. STRATA_GR_MORE_GROUPS=1 forces it on, =0 off; gr_bench runs either way.
Measured on an Arc A770 16 GB (xe, oneAPI 2026.1 JIT), same day, a0116b27 + #2190 + the previous
commit (GR_PICK, large GRF), with and without this one: gr_bench 1-6 tokens 118.9-183.5 -> 94.6-148.4 us,
checksums identical for every token count; gr_multi_parity 32 cases OK with STRATA_GR_MORE_GROUPS=0 and =1.
Decode, Coder IQ1_M, 128K context, int8 KV, --resident-experts, STRATA_VERIFY_STEPPED=1, 8K prompt, 256
tokens: 22.12 / 22.35 / 22.55 / 22.65 -> 23.24 / 23.61 / 23.62 tok/s (+3-5%), output ids identical.
Not built or run on Battlemage.
- GR_PICK: a runtime index into the by-value kernel argument (m.a[k]) made IGC copy the whole
GrMulti (~1.2 KB) into every work-item's private scratch (unitrace on an A770: ~40 KB private
per SIMD32 thread in gr_up_multi / gr_down_reduce / gr_norm_split). GR_PICK picks one field with
compile-time indices only (a chain of selects over the uniform kernel argument). Addressing
only: the same values, read and summed in the same order, so the output is bit-identical. This
part is on for every device; it is not measured on Battlemage.
- The sliced down kernel and the multi-token up kernel in large-GRF mode (256 registers/thread);
at 128 the up kernel still spills ~3.4 KB/thread on the A770 (unitrace). Large GRF halves the
threads resident per Xe core, so it is the default on Alchemist only (strata::intel_gpu_gen);
Battlemage and other devices launch the same kernels as before. STRATA_GR_LARGE_GRF=1 forces it
on, =0 off.
Measured on an Arc A770 16 GB (xe, Linux 7.0, oneAPI 2026.1 JIT), a0116b27 + #2190, same day:
gr_bench 1-6 tokens 205-258 us -> 119-184 us (GR_PICK alone, STRATA_GR_LARGE_GRF=0: 186-239 us),
checksums identical for every token count. Coder IQ1_M, 128K context, int8 KV, --resident-experts,
STRATA_VERIFY_STEPPED=1, 8K prompt, 256 tokens: decode 5.37 tok/s (5.88 / 5.93 with
STRATA_DOORBELL_CHECK=1 / STRATA_LEND_MIRROR=0) -> 22.12 / 22.55 / 15.60 (cold-cache run)
tok/s, 20.61 / 20.93 with STRATA_GR_LARGE_GRF=0; the window's pool phase 183-208 -> 34-35 ms
per round; output ids identical where the expert placement matched. Not built or run on
Battlemage or i915.
On an Arc A770 16 GB on the xe driver, Level Zero has no ext_intel_free_memory and get_memory_info reports 0 free
with the right total. device_free_bytes() kept the 0 (the #1440 fdinfo path only replaces a figure that is too
high), so --expert-cache auto sized the caches from 0 bytes. A 0 with a known total is now read as unknown: on
Linux the card's total less this process's DRM fdinfo figure is used, the same source the A750/i915 path uses.
Anything that reports a non-zero free figure is unchanged; off Linux the 0 stands.
When the main layers' experts fit but the MTP block's do not (UD-Q6_K_XL on
32 GB), the main layers are placed as in the all-in-VRAM mode (same text,
the decode graph kept) and only the MTP block's experts go to slots + the
CPU pool (its drafts). To get there when the fit is close: the packed Q8_0
decode copies are given back, the prompt chunk stays 512, and - said in the
log - the attention scores buffer shrinks to 64 MB and the VRAM reserve to
256 MiB. STRATA_Q36_NO_MTP_CPU=1 keeps the plain hybrid mode.
UD-Q6_K_XL on gfx1201: tg128 73-75 -> 80.6 (llama.cpp HIP 81.2).
Many small copies were slower than the one whole-layer copy on a 12 GB card
(RTX 3060, UD-Q4_K_XL at 37 of 256 resident: pp2048 1540 in runs, 1974
whole). gfx1201 UD-Q6_K_XL (205/256) and Q8_0 (176/256) keep the per-expert
copies: pp4096 3504-3588 and 3015-3079.
The server already keys its encoder cache by the normalized picture bytes (sha256, the .sve file
name); it now sends those keys with the request (img_keys=<hex>,..., in prompt order), and the
engine's conversation cache uses key + grid for a picture when the count matches. Without the key
(an older server) it hashes the embeddings as before. Old engines skip the unknown key.
Measured for #2228 (one 1600x1000 screenshot-like picture, strata-vision, fresh processes and
twice in one process): the CPU encoder gives the same bits on a TR PRO 3975WX at 8/15/16/32 threads
and on a Xeon E5-2643 v3 (e2357d8e...), the GPU encoder on an RTX A4000 is reproducible too
(92cf7818...), but CPU and GPU differ. So a picture's embeddings change when the encoder moves
between CPU and GPU, or after an encoder update, and no longer on its own; with the byte key the
cache's matching does not depend on that at all. CUDA/HIP and SYCL. Test: serve/test_server.py
test_pictures_are_keyed_by_their_bytes_for_the_engine.
The prompt's staging buffer took the whole layer from host memory over PCIe
(~19-28 GB per chunk); now the experts with a VRAM slot are copied device to
device and only the others come over PCIe, in runs of consecutive experts.
Same bytes in the staging buffer (greedy ids identical). UD-Q6_K_XL on
gfx1201 (205 of 256 experts per layer resident): pp4096 1935 -> 3511-3601.
Also: the packed Q8_0 decode copies are given back when that lets every
expert fit (STRATA_Q36_KEEP_PACKED=1 keeps them); STRATA_Q36_STREAM_ALL=1
keeps the whole-layer copy.
A checkpoint (or the live state) is reused only when the prompt starts with its tokens AND its
pictures, and a picture is its grid plus the hash of its embedding floats. In #2228 (screenshot-heavy
agent, CPU image encoder) the cache fell back to checkpoints a long way back although the client's
messages were byte-identical; one explanation is a picture encoded again with other bits. When the
deepest state whose tokens match is rejected for a picture, the engine now logs which picture and
where, once per request, so the next log tells this apart from a rewritten history. Log only; what
is reused does not change.
47Hunter47 measured </tool_response> asked verbatim at 0/15 on 0.1.43.1 (it reached the model as
the real closer). 62185aab already marks it; this pins the cases next to the exception that keeps
a wholly wrapped tool result's outer pair: the bare closer, a request that quotes a whole pair, a
message that only starts with the opener, two closers in one sentence.
Qwen3.6-35B-A3B-Q8_0.gguf stores the GDN alpha / beta projections as Q8_0;
the engine read only F32 / BF16 there and refused the file. They are now
widened to F32 at load (fp16 scale x int8, as ggml). q36_llama_dump:
STRATA_DUMP_CTX sets a smaller context / batch for files whose weights leave
no VRAM for 4096-token buffers.
The engine keeps every expert in RAM when they do not all fit in VRAM, so a
size's RAM need is its file plus room for the rest. The largest size this
PC's RAM holds is the default; any size can still be chosen (a bigger one is
asked about, as before, never refused).
- k_attn_mma (sm_80+): m16n8k16 f16 -> f32, 8 warps x 16 queries, 64-key K/V tiles double-buffered with
cp.async, ldmatrix (.trans for V), fp32 online softmax. Bench on the RTX 3060 at 4096 x 4150 keys x 32
heads: 23.7 TFLOPS vs ggml's flash_attn_ext_f16 15.4 (0.376 vs 0.577 s/step); CPU check 7e-5.
STRATA_QI_GGML_ATTN=1 keeps ggml's.
- q, k and v of a block with one weight type load into one [3C x C] tensor: one GEMM and one activation
quantize instead of three (STRATA_QI_NO_QKV_FUSE=1 keeps three).
- A float4 LayerNorm path with tanh(gate) once per step, off by default (STRATA_QI_FAST_LN=1): no measured gain.
RTX 3060, Q8_0 1024, seed 42, 3 interleaved rounds: steady step 2.09 s vs stock 2.61 (1.25x), denoise
17.93 s vs 22.13 (1.23x).
A start right after another engine exited sometimes fails before the engine writes a line: ROCm aborts with
"terminate called after throwing an instance of 'rocr::AMD::hsa_exception'" (or the process exits with an empty
log) while the previous process's GPU queues are still being released (r9700: 1 of ~10 rapid restarts in the A/Bs,
2 of ~40 for the quants runs). Such a start - within 30 s, no engine line other than the runtime's failure, not the
engine's own error exit 1 - is retried with a message saying why; STRATA_START_RETRIES (default 3, 0 = off).
0.1.41 (0391e478) scaled each stage's miss term by 20 GB/s / its probed link, for 2x 4090 D
with CUDA1 on a physical x1 link (1.6 GB/s: x12, K 22 -> 38, prompt ~350 -> ~1400 tok/s).
On 2x Tesla T10 the same rule moved auto from K=23 to K=14: CUDA1 runs 34 of 48 layers, and as
the prompt path is a pipeline the slower stage sets the rate - 256K prompt 1035 -> 795 tok/s,
decode 46.8 -> 39.5; --layer-split 23 on 0.1.43 restores both (xiake-1 in #2007).
split_rules::link_miss_scale: the scale is the fastest stage's link over this stage's, applied
only from a 4x gap (a riser or chipset x1/x4 slot beside an x16 one), capped at 64. A symmetric
rig keeps the fitted formula whatever its link speed; the x1 case still gets ~x15.
STRATA_SPLIT_LINK_ABS=1 restores the 0.1.41 rule. SYCL has no link term (unchanged).
Test: split_rules_test (the x1 case, equal gen3 cards, a 2x gap, caps, unprobed stages).
Not run on the reporter's T10s: the reply asks for the 'link ... -> miss cost' lines.
The native output head is needed only to generate, the batched prompt path's buffers only to read a prompt. With the flag the head is unloaded while a prompt is read (its VRAM counted into the auto cache and the prompt path's hybrid borrowed/owned layout) and read back in before the first decode window. On an 8 GB RTX PRO 2000 Laptop (Windows, IQ3_XXS, 590 MiB reserve) the prompt chunk went from 512 to 2,048 tokens and prefill from 292.5 to 667.1 tok/s on a 29,854-token prompt; reads of up to 512 fresh tokens cost about 0.6-0.75 s more. Off by default: every new path is behind the flag. docs/PREFILL_HEAD_ALIAS.md has the measurements and limits.
The table is read with F_NOCACHE through a bounded row cache (1M rows), so 2 GB was too much to set aside. M4 32 GB,
IQ3_XXS decode after the 6K document, 5 interleaved runs, auto arena (memory_pressure every 3 s, vm_stat swapouts):
keep 7 + 2 GB (54c0de70) 5.93 tok/s median, arena ~8.5 GB; keep 7 + 0.5 6.59 (+11%), ~10 GB; keep 5 + 0.5 7.47
(+26%), ~12 GB; the least free memory 43-57%, no swap-out in any run. The M5 Max still keeps 15% (20.6 GB).
arena_budget_test: 14 checks pass.
ROCm gives out hardware queues per priority. Sharing one with the window's stream, --pcie-mode dma's expert copies
and the host function that raises flag B queue behind the window graph that spins on that flag: on an R9700 every
dma-mode request timed out at layer 1. With its own queue dma mode runs (copies overlap the VRAM-hit compute).
The figure in "[strata] images: the image encoder started in N s, M MiB of VRAM" read the free VRAM right after the
cache's shrink, before the driver's counter showed it (amdgpu sysfs), so it came out low (286-362 MiB against the 1,339
measured otherwise). A 0.2 s wait before that reading, only on a start that shrank the cache.
Opt-in, with the config's "vram_elastic": the GPU encoder does not start with the server or a load. A picture's turn
makes the expert cache keep "elastic_mib" (default 2048) free on top of the reserve and starts the encoder; after
"elastic_idle_s" (default 120) without a picture a thread stops it between requests and the cache grows back to the
reserve (0 when no POST /v1/vram was made: the start's reserve left it a segment or more smaller). A POST /v1/vram
while the encoder runs stacks on its share; an unload stops it. Without "vram_elastic" the server says so and starts
the encoder as before. serve/test_vision_elastic.py pins each case; docs in VRAM_ELASTIC.md and DETAILS.md.
update_install refreshed the draft subset once per installed config, newest first, so a folder that two configs
name ended with the subset of the config used longest ago. A config without "draft_vocab" (cjk) then put the
default subset over the cyrillic one of the config in use. Now the first config that names a folder decides;
separate folders and a start (which installs its own config's subset) are as before.
Checked: tools/test_setup_draft_vocab.py (3 new tests, 10 pass), test_setup_config and test_setup_configs (18 pass).
Measured on the affected install, two alternating runs each, greedy 600-token Russian story: 54.0 / 54.0 tok/s
with the default subset installed by the update, 68.7 / 68.6 with the cyrillic one (1.39 -> 1.90 tokens per window).
A helper for the large-page arena (#1412): run as Administrator, it adds the account to
SeLockMemoryPrivilege with secedit, says that a new logon is needed, and prints the secpol.msc
steps when it cannot run. Changed from the PR: secedit /configure is limited to
/areas USER_RIGHTS (the exported policy is not re-applied as a whole), and the description no
longer claims a speed-up we have not measured. docs/DETAILS.md points to it. Not run on our side
(no Windows box for it).
Co-authored-by: dvasdekis <[email protected]>
Some amdhip64_7.dll builds (AMD TheRock wheels / ROCm 10.2 / 7.2.4 nightly) do not export
hipKernelNameRefByPtr although their import library has it, so strata.exe failed to load with
STATUS_ENTRYPOINT_NOT_FOUND before main. It is only used to name graph nodes under
STRATA_VERIFY_NODES; on Windows it is now looked up with GetProcAddress and a missing one
leaves the node unnamed.
From #2183, its verify.cpp part only. The PR's other part (Windows HIP --pcie-mode auto as
DMA) changes the default path of every Windows AMD user and was not measured on our side; it
is left out.
Co-authored-by: dvasdekis <[email protected]>
An Arc A770 under xe intermittently never started decoding: the main thread waits in Verifier::run's stepped serve
(queue::wait -> urQueueFinish) for a segment that never drains, xe logs nothing (#2194). The A750's same failure was
fixed by draining the other queues before each window (STRATA_WINDOW_SYNC, default 1 on i915 only). Reproduced on
the A750 by turning that off: STRATA_WINDOW_SYNC=0, STRATA_VERIFY_STEPPED=1, --resident-experts, IQ3_XXS, 4K prompt:
1 of 6 runs hung with #2194's stack; with the default drain 0 of 11 (5 unpinned, 6 with the pipelined helper pinned).
The default is now 1 on i915 or on an Alchemist device (intel_gpu_gen), 0 elsewhere; STRATA_WINDOW_SYNC overrides.
Unchanged on the A750 (i915: already 1) and on Battlemage (0). Not run on an A770 under xe (no such card here).
The first version compared only the extra loan with the RAM available at that point, but the
resident RAM copy is pinned after it and holds the lent slots' experts too: in a 30 GB scope on
an R9700 (IQ3_XXS, --resident-experts) 30 GiB were available before the 23.35 GiB copy, so the
rule never fired. The reporter's PC thrashed with 3.4 GB left at 16384 and ran with ~6.4 GB at
8192, hence 6 GiB.
Both NVFP4 packs keep Qwen's vision tower unquantized, so the image encoder every Flash-Next model uses reads pictures
for them; strata-vision opens the text model vocab-only, which is the pack's dense GGUF here. Setup asks about images as
it does for every model (default no; --vision gpu|cpu), the encoder is the file other installs already share
(mmproj-Qwen3.8-Flash-Next-BF16.gguf at ISTA's pinned revision, 907,543,008 B, sha256 b1a82259...).
Checked end to end through the server on an RTX 5070 (budget mode, GPU encoder, 1024 image tokens): a picture with
"CODE 4217", a red square on the left and a blue circle on the right - NVFP4 answered "4217" and described both shapes,
their colours and sides correctly, as IQ3_XXS did on the same picture. VRAM: the expert cache held 393 slots with
images (--vram-reserve-mib 2100 and the encoder) against 982 without (1500).
The gfx12 default (0.1.43) lets a bare --prefill auto reach 16384, which lends the prompt path
about twice the cache slots of 8192 (R9700, Swift 1.5 IQ3_XXS: 3,907 slots / 6.38 GiB against
2,068 / 3.37 GiB); the lent slots' experts are read from the RAM side during the read. On a
30 GB PC with --resident-experts 38-47K prompts went 20 GB into swap with 1.5M major faults
and the engine was killed; --prefill 8192 ran them at ~95 tok/s (bdwilton in #2090).
At start, when the engine's own 16384 ceiling is in effect, the extra slots of a 16384 loan
over an 8192 one are priced at the largest blob; if the RAM available (the cgroup or
--memory-limit-mib cap included) is below that plus 4 GiB, the ceiling stays at 8192 and a
line says why. STRATA_GFX12_CHUNK=1 keeps 16384; an explicit --prefill N / auto:N is never
touched (the 16384 default only applies to a bare auto).
The 'MiB of VRAM free with everything loaded' line and its LOW warning read CUDA0 alone. In
#2192 (2x RTX 2080 Ti 22 GB, Windows, IQ3_S, --batch 4, --layer-split auto, 0.1.43) CUDA0 had
740 MiB free while CUDA1 began its window captures at 153 MiB and fell to 52; the first batch
window then never finished. Both stall dumps show the host thread inside cudaGraphLaunch called
from Verifier::run_slot_rows (the stage still 'idle': it had not reached its per-layer wait), the
pool asleep and the watchdog firing after 60 s - a page-in under WDDM while the other card's
graph spins on a host flag.
Each later card now gets its own line, and below 512 MiB a LOW warning with the
--vram-reserve-later-mib value that leaves ~768 MiB. Said, never changed.
On Windows with only an AMD card neither NVML nor the amdgpu sysfs files answer, so
hardware_static.gpu_name was None and strata_bench (no nvidia-smi, no rocm-smi on Windows)
labelled the run "CPU only" although the engine ran on the card (ByteJuggler in #1587: RX 6900 XT).
telemetry.windows_display_adapters reads DriverDesc, ProviderName, DriverVersion and the memory
size from HKLM's display class (no subprocess, no vendor library); windows_amd_cards drops a
Ryzen's integrated Radeon (512 MiB here) when a real card sits beside it. hardware_static gets
the name and gpu_vram_total_mib; strata_bench uses the same list when nvidia-smi and rocm-smi
give nothing. Load, VRAM use, temperature and power stay unread on Windows AMD (that is the
reader of #1224 / #1434 / #1871). Checked against this PC's registry (RTX 5070 + the Ryzen 7600's
iGPU: both listed, the iGPU's 512 MiB read); tools/test_windows_amd_gpu.py with a fake registry.
The stock split shape (STRATA_STAGE_TRIM=1, 1024 MiB reserve, NO_HOST=1 from strata-sycl.sh) is
refused at the start on a 2x Arc Pro B60 24 GB pair with IQ3_XXS: the second stage is 2230-3731
experts short and the host mirror covers the first stage only. The same config with NO_HOST=0
runs at 39-50 tok/s with 0 faults (jds1-ai in #1980). When the model's experts take more than
80% of the split cards' VRAM, setup now writes NO_HOST=0 into the config (a value the user set
is kept) and prints why and how to turn it back. IQ2_XS on the same pair (74%) keeps NO_HOST=1.
Test: tools/test_setup_sycl.py test_split_short_rule.
- refuse_exit: after the REFUSED message the engine waits for the queued device work, flushes
and _Exit(2)s instead of returning from main. Returning ran the stages' destructors in an
order the UR loader did not survive: on a 2x Arc Pro B60 pair every REFUSED start segfaulted
in libur_loader, and a trimmed one also left an xe page fault (bcs CAT error) on the second
card that hung the next start (jds1-ai in #1980).
- resident_plan_set_mirror takes the table's length; resident_plan uses the mirror and the
usage counts only for a residency slice inside the table they were set for. A later stage
passes its own residency table, so the slice was taken as an offset across allocations and
read (and counted into) another buffer: the stage-1 xe faults with STRATA_VERIFY_NO_HOST=0
and no trim (10 of 10 requests, 0 tokens). Such a miss is now a plain miss, served by the
host. Patch shape as jds1-ai proposed it.
Compile-checked only (no two-card Intel box here).
Under WDDM a stream's work sits in the driver's command buffer until a driver call flushes it.
With the stream-ordered signal the verify window only spins on a pinned int, so the last
layers' copies and their 4-byte signal copies were never submitted: the first request stopped
with "the adaptive tier's copies for layer 45 never landed" (RTX 4070 SUPER, 0.1.43.1, opt-in
STRATA_ADAPT_OVERLAP=1). STRATA_ADAPT_SIGNAL=host worked.
- Windows: the host function is the default signal; STRATA_ADAPT_SIGNAL=copy asks for the copy.
- Every platform: one cudaStreamQuery on the copy stream after the swaps and their signals are
queued (a no-op for the results; it submits the batch under WDDM). Linux keeps the copy
signal as before.
Needs a Windows run with STRATA_ADAPT_OVERLAP=1 STRATA_ADAPT_SIGNAL=copy to confirm the flush.
bench/results/2026-10-12-p40-sm61-kernels: where a prompt and a decode window spend their time on a P40, the kernel
benches and parity runs, every request of the engine A/B (565-token prompt 205 -> 255 tok/s, decode 34-35 -> 37-38
tok/s; with STRATA_DENSE_F32=1 a 7K prompt 360-432 -> 464-474 tok/s), the byte comparison of the answers, and what
was not tested. docs/OLDER_GPUS.md: the sm_61 section and switches, and two traps when measuring on such a card.
As sm_60 got with STRATA_SM60_TABLE, measured on a Tesla P40 (native dp4a):
- native_expert_bench, 18 groups at 2 tokens on the IQ3_S pack's layers, mode 0 -> mode 8, bitwise: IQ3_S 390 ->
255 us, IQ3_XXS 345 -> 252, IQ2_S 290 -> 238, IQ4_XS 328 -> 261.
- kIlRows_61 / kIl1Rows_61 from three runs of mmvq_il_parity --bench --emit-table (identical but two Q3_K cells;
a cell is non-zero only where every run gained 3%).
For FP16 operands with FP32 sums cuBLAS 12.4 has two Maxwell kernels on a Tesla P40 and picks the slow one for most
chunk sizes: qkv (10240 x 2560) runs at 7.6 TFLOPS for T 7030-7040 and at 4.3 from T 7041 (49 -> 85 ms), so a
7K-token prompt reads in 16.3 or 19.5 s by its token count. cublasSgemm on FP32 copies runs at 8.4-9.8 TFLOPS from
T 512. For the five shapes and the sizes the route takes, the result is bitwise the FP16 call's: gemm_f32_parity,
5,201 chunk sizes through Gemm::f16. That is a property of the cuBLAS build (12.4.5.8 measured), hence opt-in.
dequant_kernel's thread reads its 32-value group a byte at a time and writes 32 two-byte values one by one, 64 bytes
from its neighbour's. On a Tesla P40 that is 10 GB/s for Q6_K, and the IQ3_S pack's dense weights cost every prompt
chunk ~0.5 s of it, whatever the chunk's length (a fifth of a 560-token prompt). The wide kernel takes 8 values a
thread: 16-bit loads, one 16-byte store, the same expression per value.
dequant_wide_parity: 360 cases bitwise (FP16 / BF16 / FP32 out, a padded stride, row slices, sane and raw random
blocks); --bench on the P40: Q6_K 10240 x 2560 7.25 -> 0.33 ms, Q4_K 5.2x, Q5_K 3.8x, IQ4_XS 3.6x.
On by default on compute capability 6.1 only (where it was measured); STRATA_DEQUANT_WIDE=0 / =1 for any card.
The router, the GDN alpha / beta and the shared expert's gate (n_out 1-256,
K 2048) ran on hipBLAS/cuBLAS SGEMM with 8x8 / 16x16 tiles (~10% of a
prompt). k_sgemm_nt: (16 TM) x 64 tiles, TM x 4 outputs per thread, K split
so about 128 blocks run, the partials summed in a fixed order
(deterministic). gfx1201 pp4096: 3290 -> 3455 tok/s. STRATA_Q36_BLAS_F32=1
keeps the BLAS SGEMM.
Warm-up: a throw-away prompt chunk, a decode step and (with drafts) a
speculative step at load make the code objects, BLAS kernels and graphs, so
the first request runs at full speed (pp4096 rep 0: ~1770 -> 3273 tok/s; a
34-token first prompt: 1032 -> 48 ms). --no-warmup skips it.
Hybrid mode with MTP exact by default: the MTP block is loaded whenever the
file has one (so the VRAM layout and the slot count do not depend on --mtp;
the prompt pass through it only runs with drafts); decode and verify rows'
routing is counted only for the rows kept; the slots are rebalanced exactly
at every 32nd position and a verify window never steps over one.
arena_budget.h: the rule as plain arithmetic, tested by metal/tests/arena_budget_test.cpp (13 checks, run on the
M4). The M5 Max froze (2026-10-13 02:08) with an 89 GB arena planned from 'available - 7 GB' beside a 50 GB n-gram
table: macOS now keeps 15% of the RAM (20.6 GB there), the table's mapped bytes (--ple-io ram) or 2 GB (read row by
row) come out too, and just before the arena is committed the free memory is asked again - less than planned: the
arena is laid out again smaller instead of failing. STRATA_METAL_KEEP_FREE_GB / STRATA_METAL_ARENA_GB override.
stream.cpp: arena buffers grow past 8 GB when 4 of 8 GB cannot hold the arena (STRATA_METAL_ARENA_CHUNK_GB).
The coordinator suggested max(12 GB, 15%); on the 32 GB M4 (Q2_0 streamed, 200 tokens, memory_pressure every 3 s)
12 GB kept cut the arena 9.4 -> 4.4 GB and decode 7.8 -> 4.8 tok/s, while 7 GB kept left macOS 55% free the whole
run (12 GB: 70%); same tokens. Hence 7 GB below ~47 GB of RAM, 15% above. Not yet run on the M5 (down).
iq_host.cpp (Metal's copy of the host helpers): Q6_K among the grouped gate/up formats, as mac-kern's moe_gu_14 /
mm_gu3_14 run them. Before, the engine refused UD-Q5_K_XL ('layer 2's experts are Q6_K/Q8_0 ... no GPU kernels').
strata reads the --native file's architecture at the top of main() and hands
a qwen35moe model to strata::qwen35::run (src/qwen35, now the strata_qwen35
library; strata-q36 wraps the same entry point) before any Flash-Next state
exists; every other file runs exactly as before. strata --families lists the
architectures a build runs. The Qwen3.6 path takes strata's flag names where
they mean the same (--native, --max-new, --pool-workers, --no-token-graph,
--gpu, --mtp N) and skips Flash-Next-only ones in serve mode. The SYCL build
keeps refusing the family, now saying CUDA and HIP run it.
setup: Qwen3.6-35B-A3B is a family in the menu (OWN_PATH_FAMILIES,
install_own_path): size, context, the engine (checked with --families), the
pinned GGUF, its tokenizer, a config with --native/--max-context/--mtp 2 and
the start script; no pack, profile, draft layer or tuning. The engine step is
a function shared with the Flash-Next path. strata_inspect says the family
runs on CUDA and HIP.
- dense_bench: the decode window's native_mmvq calls at the model's shapes and formats, 24 matrices per shape in one
graph (weights from DRAM), 1-4 columns, us / GB/s and an output hash.
- STRATA_MMVQ_WIDE_K_RPW=2|4: Q4_K / Q5_K / IQ4_XS wide kernel with 2 or 4 rows per sub-group sharing each column's
activation loads (apply split into act + dot, the same operations); every hash equal. B70, 3 columns: Q4_K
6144x2560 56.6 -> 27.7 us, Q5_K 2560x12288 89.8 -> 75.5 us, Q6_K unaffected.
- STRATA_MMVQ_WIDE_RPW>1 and STRATA_MMVQ_PF now take the aligned Q6_K loads (they were 2-4x slower than the default
without them); still slower than the default except Q6_K 6144x2560 at RPW 2 (50.2 -> 45.3 us).
- fused GR read: the up step's columns per block (STRATA_GR_UP_COLS 4/8/16/32/64, default 16 = before) and its T
butterflies interleaved (STRATA_GR_UP_ILP=1); gr_bench checksums equal for all. 32 columns: 3 tokens 50.7 -> 48.9 us,
1 token 27.5 -> 29.4; ILP slower.
Engine, B70, IQ3_S with data/expert-profile-sycl.bin, 5 interleaved pairs: STRATA_MMVQ_WIDE_K_RPW=2 STRATA_GR_UP_COLS=32
vs default: decode after a 4K prompt 54.31 -> 54.00 tok/s, story 38.41 -> 38.50, outputs identical. Defaults unchanged.
Found by the quant yardstick (--expert-cache 1) on the NVFP4 GPTQ pack: 'pre-filled 0 of 0 slots', and the verify windows
refused to run. Every other native format's blob is rows x row bytes with 1280 / 2560 rows, a multiple of 256, so its
budget is unchanged.
ROCm gives out its hardware queues per priority level; the draft stream shared one with the verifier's, so the
draft round ran only after the commit graph the verifier had just launched. With a queue of its own the commit
runs beside the draft. R9700, UD-Q4_K_XL: draft wall 5.8 -> 3.5 ms a window (STRATA_DRAFT_TIMING).
The per-layer commit was ~98 graph nodes; it runs between a window and the draft round, and on HIP the round
waits for it (R9700 UD-Q4_K_XL: 2.35 -> 0.65 ms GPU). Each layer's arithmetic is the per-layer call's:
gdn_parity (gdn_commit_layers vs the per-layer commit, n_keep 1-8) and qsa_parity (append_steps_layers vs
copy + append_steps, 3 scalings) are bitwise. STRATA_COMMIT_TIMING=1 prints the commit graph's GPU time.
On the RTX 5070 PC (64 GB, Windows) --mmap-experts let the experts it read stay in the engine's working set: free RAM
fell to 0.6-4.9 GB and the memory guard stopped the engine three times (2026-10-13). --resident-budget-gib 22 with the
same pack kept >= 22.8 GB free. The budget mode is what setup already writes for Unsloth's files (resident_budget_gib:
40 GiB on 64 GB); the pack's experts.bin is the file the rest are read from. test_setup_nvfp4: 64 GB -> budget 40.
Verifier::window_logprobs reads the head rows through window_copy_logits under STRATA_GPU_METAL (the CUDA/HIP path
unchanged: verify.cpp's object is byte-IDENTICAL on P1004 and p620). The prompt's windows sample nothing (head
sampling off), so with STRATA_LOGPOS the Metal window computes the head anyway.
tools/quant_quality.py on the M5 Max against the Q8_0 reference (q8_0.tsv 627558d1, set.ids.json 414634dd), 4,995
positions: UD-Q4_K_XL 92.77% top-1, KL 0.0548 (p99 0.855), PPL 4.043 vs 4.052 (CUDA/HIP: 93.09% / 0.056); IQ3_S
87.85% / 0.1422 (CUDA/HIP: 87.7% / 0.144). The Metal window keeps int8 K/V (the yardstick asks fp16).
tools/image/sdcpp_native: a runner that replaces QwenImage21Runner::compute only (sd.cpp keeps the text
encoder, VAE, sampler and RNG), chosen with --model-args qwen_image_2_1_native=1 in CUDA / HIP builds.
- The text prefix runs once per condition (causal, the t = 0 modulation row) and leaves each layer's
fp16 K/V; each step runs the 32 blocks on the image tokens only.
- GEMMs stay ggml graphs on the runner's backend (sd.cpp's MMQ / BLAS kernels and numerics); fused
LayerNorm+modulate, gate+residual(+next LayerNorm), q/k RMSNorm+RoPE+fp16 K/V, SwiGLU and, on gfx11, a
WMMA flash attention (8 waves x 16 queries, 64-key tiles, fp32 softmax) run on the backend's stream.
- apply.py copies the two files into the pinned checkout and makes two small edits (runner choice,
SD_QI_NATIVE for CUDA / HIP builds); anything not covered falls back to the stock runner.
780M, Q8_0, seed 42: 512 2.11 s/step (stock 2.85), 1024 9.58 s/step (stock 14.2), image 96.5 s (131.8).
A parked pool worker watches the epoch's cache line with UMONITOR and sleeps in UMWAIT (C0.1) until the host publishes or the cycles run out. On a laptop whose CPU and GPU share one power budget the spin's power goes back to the GPU. Unset, 0, no WAITPKG or a non-MSVC build: the spin, unchanged. STRATA_POOL_UMWAIT_C02=1 picks the deeper C0.2 state, STRATA_POOL_UMWAIT_ESC=<cycles> C0.1 first and C0.2 after.
Two places upstream builds QsaShapes by hand from the geometry (chunk_bytes, added since the geometry change was
written, and the conversation-state fingerprint, in both trees). They now call qsa_shapes(g) like layer, mtp,
session, verify and prefill; for every file the guard accepts the values are the ones they built before.
generate.cpp reads the geometry before anything else looks at the model, so a model of another architecture was
refused as "missing qwen4exp.block_count" where 0.1.40.2 said "architecture is 'qwen35', this engine requires
'qwen4exp'" (found running Qwen3.8-27B through --native). read_geometry now checks general.architecture first, and
check_architecture is read_geometry plus geometry_unsupported. gguf_geometry_test: a dense qwen35 file (Qwen3.8-27B's
keys) and a file without general.architecture.
Tested on gfx1151 (HIP build of this branch, Windows): the GGUF tests pass; Qwen3.8-27B UD-Q6_K_XL, Q8_0 and
HauhauCS Q4_K_P are refused by architecture in 0.5 s; Qwen3.8-Flash-Next UD-IQ4_XS, greedy, 256 tokens on three
prompts (39, 48 and 5,568 tokens), gives output identical to the installed 0.1.40.2.
The f32 and array members added for gguf_geometry_test left the braced Kv{...} initialisers short of the struct, which
clang's -Wextra reports as -Wmissing-field-initializers in every test that includes the fixture (MSVC /W4 does not).
make_kv() sets the key and type, and each helper sets its own value. Built with the HIP toolchain (gfx1151):
gguf_geometry_test, gguf_split_test, gguf_reader_test and native_dense_ple_key_test pass, with no warning from the
fixture.
read_geometry (gguf_reader.hpp) fills ModelGeometry from the qwen4exp.* keys, under llama.cpp's names: layers, widths,
GDN, attention, indexer (heads, key width, pooling block from compress_ratios, top-k), rope width, hyper-connections,
experts, RMS epsilon, and the vocabulary from tokenizer.ggml.tokens. The six keys the guard always required stay
required; any other absent key keeps the compiled value. Keys that state one thing twice must agree (the interval and
compress_ratios, key and value head widths, routed and shared expert widths, ssm.inner_size and state x v heads).
check_architecture no longer hard-codes 48 / 2560 / 24 / 2: it reads the geometry and refuses a file the kernels are not
compiled for, naming the key ("qwen4exp.block_count = 40, the kernels are compiled for 48"). Only the expert count,
the experts per token (1..64) and the vocabulary may differ. ModelGeometry moves to core/geometry.hpp (no
dependencies), with kCompiledGeometry; the kernels that fix a dimension at compile time static_assert it there, in the
CUDA/HIP tree and the SYCL copies (plus qsa_prompt_attn_xmx).
generate.cpp takes the geometry and K from read_geometry, and the native embedding takes n_embd / n_vocab from it
instead of a literal 248320. The QSA kernel shapes come from one qsa_shapes(g) (layout.hpp) in layer, mtp, session,
verify and prefill, so n_rot, idx_block and idx_top_k follow the geometry too. Mirrored in sycl/.
Tested: tests/core/gguf_geometry_test.cpp (new, synthetic GGUFs) 28/28, gguf_split_test and gguf_reader_test pass,
built with MSVC /W4 /WX on Windows. `gguf_geometry_test --real` on Unsloth's Qwen3.8-Flash-Next UD-IQ4_XS shard 1
reads exactly the compiled shape and the guard accepts it. Not built here: the engine (no CUDA, HIP or oneAPI compiler
on this PC), so generate/layer/mtp/session/verify/prefill and the kernel asserts are reviewed, not compiled.
The Debian fallback loop ran with check=False on Ubuntu too, so a failed 13.0 install (a
network or apt lock error) went on to 13-1..13-4 and then past the loop without a word. Ubuntu
is back to the single checked install of 0.1.43; Debian tries 13-0..13-9 and fails with
guidance when none installs. Checked against NVIDIA's repos today: debian12 and ubuntu2404
carry 13-0..13-3, debian13 starts at 13-1 (up to 13-4). Two tests added.
The CUDA apt repo keyring and cuda-toolkit-13-x packages exist for
debian12/debian13 as well as ubuntu2204/ubuntu2404, so derive the repo
name from /etc/os-release instead of hardcoding ubuntu. Debian 13's
repo carries no cuda-toolkit-13-0, so try 13-0..13-5 in order (Ubuntu
still gets 13-0 first, same as before).
Checked with tools/test_setup_cuda_repo.py (4 tests, no GPU or network
needed): repo mapping for ubuntu/debian/other and the minor-version
fallback.
opencode puts its <env> block (working directory, today's date) near the start of the system message, before ~40K characters that never change. The prefix cache only matches a prompt's start, so each new folder or day read the whole system turn again. With the switch the block goes to the front of the first user message (string or content parts), for OpenAI and Anthropic requests. Off by default; serve/test_env_to_user.py; docs/DETAILS.md.
The server sends BYIELD for a waiting request under half as long only when a PP line (or the 10 s heartbeat) arrives,
and the engine looks for it at its next boundary. Where a boundary's own PP line is the only line for a while - every
chunk of a --batch read on one GPU, and #2050's pipelined pieces - the BYIELD answers that line and reaches the engine
after it has looked: the read gives way one boundary later, or never when that was the last one it could take. It is in
time only where the engine saves a periodic checkpoint after the PP line (every 16,384 tokens by default).
_control() now also calls on_quiet() whenever the engine has been quiet for half a second (its existing poll), and the
solo and admission reads run the same check there as at a PP line. A BYIELD for a read that has ended is ignored by
the engine, as before. serve/test_parallel.py: the fake engine takes --pp-every K (a PP line and a boundary every K
chunks); a solo read and an admission must give way at their first boundary (both fail on main), and the rule (only
to a much shorter request) holds on the poll too.
Measured on one RTX 5090 (Swift 1.5 IQ3_S, --batch 4, the engine built from main), main's server and this one, two
engine starts each (M F F M): a one-line request sent 0.2-8.6 s into a cold ~33K read, with nothing else decoding and
with one stream decoding. Late (a boundary passed after the request waited): 28 of 40 runs on main, 12 of which never
gave way; 4 of 40 with this change, all requests that started waiting 0.02-0.10 s before that boundary's PP line.
Where main was late the short request's TTFT went 3.56 -> 1.70 s (nothing decoding) and 4.40 -> 1.57 s (one stream);
where main never gave way 4.60 -> 1.51 s and 7.59 -> 2.35 s. Where both gave way at the same boundary, both requests'
TTFTs were the same (medians within 0.02 s). On 2x RTX 5090 with #2050's STRATA_PIPED_READ_PIECE=8192: 6 of 12 late without
it, none with it.
On two equal cards every K whose caches hold all the profiled pairs predicts the same window
time, and the search kept the first one tried: the smallest K, which gives the last stage (it
also carries the MTP drafter) the most layers and lets its cache fill the card to the reserve.
2x Arc Pro B70 with IQ3_S: auto K=24, CUDA1 at 31.8 of 31.9 GiB, ~11 tok/s; --layer-split 30
~54 tok/s (agnaylor in #1440). A full xe card does not fail an allocation, it moves buffers to
system memory.
The SYCL search now scores every placement it tries with its smallest spare room (a stage's
room after its session minus every expert of its layers) and, among the placements within
STRATA_SPLIT_HEADROOM_TOL (default 0.2%) of the best predicted time, takes the one with the
most (split_rules::pick_with_headroom). A log line names both picks when it moves the split.
STRATA_SPLIT_HEADROOM=0 keeps the old pick. CUDA/HIP placement is unchanged.
Test: split_rules_test (a 48-layer two-card model of the report: plain K=24 with 664 MiB spare
on the last card, headroom K=25 with 1728 MiB; tolerance and tie rules).
The three are USER_DEFINED (type 4) tokens like </tool_call> (ids 248058, 248066, 248070), so
the tokenizer matched them inside a message's text as the real call / tool-result markers.
They join TOOL_LITERAL_TAGS: inside a user, system or tool message (and the tool list) they are
encoded as ordinary text; an assistant message replaying raw call text keeps its tokens, as
for </tool_call>.
One exception keeps the template's behaviour: a user message whose whole text is
<tool_response>...</tool_response> (a client that formats tool results itself) keeps that outer
pair as the markers, since the template tells such a turn from a real question by exactly that
pair (last_query_index); the text inside is still marked.
Found by 47Hunter47 in #2026. Tests: serve/test_frontend.py QuotedToolCallCloser (every tag
and role, the assistant replay, the outer pair as string or text parts).
M5 Max: ./setup.sh --model UD-Q4_K_XL --vision yes: shards checked by SHA-256, repacked and verified, the pack, the
image encoder; the config it writes answers the test picture ('STRATA 42', 'Red circle / Blue square / Green
triangle').
The routed experts are placed after the engine's buffers: all in VRAM when
they fit (unchanged), else every expert in host memory with the same number
of VRAM slots per layer (--expert-vram GiB, --expert-slots N). Batches of up
to --hybrid-rows rows run the resident experts on native_expert_grouped and
the others on the CPU ExpertPool (ggml-cpu arithmetic) side by side; bigger
prompt chunks stream each layer's experts into two staging buffers for MMQ
(prompt chunk 2048 in this mode). Slots follow the routing counts (after a
prompt, every 32 decode positions). ExpertPool's native phases split rows by
the layer's n_ff / n_embd instead of Flash-Next's constants (same values
there). STRATA_MMQ_KQ=1 by default for this family.
ple_conv_add reads the conv kernel as half: Unsloth's file stores it F32, so it is narrowed (round to nearest even)
into its own buffer; the GSQ packs' F16 tensor is read in place as before. A file whose HC matrices are Q8_0
already makes no copies (the copy buffer was allocated with 0 bytes and the Metal allocation failed).
M5 Max, UD-Q4_K_XL resident (the repacked 82.5 GB file): greedy prompt1 identical to the CUDA reference (z440),
story first differs at token 36; MTP output equal to no MTP; decode no MTP 60.6 / 60.0, MTP --spec 2 93.9 / 74.0
tok/s; prompts 4K 1,374 / 32K 1,288 tok/s, needles found (4K, 32K). llama.cpp Metal on the same shards: 48.4 /
47.3 decode, 821 / 739 prompt.
With STRATA_FS_SLOTS > 0 and the resident RAM mode's adaptive exchanges, a slot copy issued
from an expert's RAM place (note_miss / the router look-ahead's predict) could land after an
exchange rewrote that place with the outgoing expert's bytes; take() then served expert Y's
bytes for expert X (FS_VERIFY: 19 BAD on clean v0.1.43, reported in #2134).
commit_exchanges, commit_copies and commit_flip now hold every staged expert (in and out) in
the swap space while their bytes move: its slot is dropped (never served again) and no copy
of it is issued until the commit is done (commit_copies holds until commit_flip, so the
asynchronous tier's half-done exchange is covered, and the rotation path too). Unlike the
patch in the issue, a dropped slot is not handed out at once: it waits the same `depth`
finished windows as any slot a window may still read, since a window planned before the
exchange can be reading it. "N dropped by exchanges" joins the swap space's report.
Foresight is opt-in; with STRATA_FS_SLOTS unset nothing here runs. Test:
file_expert_source_test --rotation-gpu (both commit paths, the depth rule, the bytes after).
Based on the analysis and patch in #2134.
Co-authored-by: radoslawsz <[email protected]>
The stdin reader thread is detached and captured main's stop flag, mutex, condition variable,
line queue and EOF flag by reference. The server writes QUIT and closes stdin right after it,
so the reader often wakes with EOF while main is returning: it then locked a destroyed mutex
and wrote to a destroyed deque (undefined behaviour on every platform; the Metal port aborted
on it). The state now lives in a shared_ptr that the thread holds a copy of, so it outlives
main's frame for as long as the reader runs. CUDA/HIP and SYCL engines alike; no change to
what the engine reads or answers.
expert_source.cpp's Linux-only STRATA_TEST_UNMAP_GGUF check had a literal 0x00 byte as its
character constant (since 0.1.43; gcc warned "null character(s) preserved in literal"), and the
#2126 buddyinfo parser in pinned.cu got one the same way (a heredoc dropped the backslash).
Both are the escaped character now. Git took pinned.cu for a binary file because of the byte,
so it was committed with CRLF line endings; it is back to LF (no other change).
tools/test_source_nul_bytes.py fails when any tracked source file holds a NUL byte again.
Prompt chunks run the attention scores and P V as FP16 strided-batched GEMMs
with FP32 accumulation (K / V cells and Q converted per chunk, the causal
softmax writing FP16 probabilities), replacing the FP32 SGEMMs. The prompt's
delta rule splits each V head over 4 column groups (16 row groups of 8 rows,
the norm in its own kernel); decode, verify windows and commits keep one
kernel. pp4096 on gfx1201: 2557 -> 3290 tok/s.
The prefill hook hands the prompt's image rows (sp.embd_rows) to the Metal prompt paths (prompt_chain.inc, the
streaming prompt path, the 8-row windows), which take an image row in place of the token embedding.
mrope_table_set keeps the host view of the server's (t, h, w) table; every window, prompt chunk and draft round copies
the rows it reads into the window's buffer, bound at index 20 of qsa_prep with function constant 24 (mac-kern
29724e5e) - the window, both prompt paths and the draft layer. The cached window's QsaArgs are patched by the last
inline argument (the table is bound after it). setup_mac.py --vision yes: the pinned projector (SHA-256) and
strata-vision built with ggml-metal; the config gets --vision and the encoder.
missing.cpp: a crash prints its call stack and fault address.
M5 Max, Q2_0, ISTA's projector, a test picture (shapes and 'STRATA 42'): 'STRATA 42' and 'Red circle / Blue square /
Green triangle', the same answers as the HIP engine on the R9700 (p620) with the same picture and files, also
through the config setup writes. Text greedy unchanged with and without --vision (prompt1, story, the 1,662-token
chain prompt). prefill.cpp: the CUDA (P1004) and HIP (p620) objects are byte-IDENTICAL (preprocessor-only).
The greedy divergence seen with the fused single-token kernels is not a
kernel bug: fused_gdn_conv_l2 and k_conv_seq leave the same history and
differ by 1 ulp in h (FMA contraction of the 4-tap sum); the Q8_1
activation rounding turns that into a flipped near-tie (p01 token 4,
margin 0.43). KL with either is the same. Drop the switch.
Lane tid wrote block tid % 8 of its superblock, so a sub-group's 2-byte stores landed 64 bytes apart. Lane tid now
takes block tid / 4 (the same 32 (block, part) pairs, the same values) and writes its runs with store_run. UD-Q4_K_XL's
down projections are Q5_1 and UD-Q5/Q6/Q8_0's are Q8_0, so this is most of their FP16 prompt-path dequant.
B70, dequant_bench (32 experts, gate/up + down): Q5_1 7.18 -> 2.83 ms, Q8_0 6.77 -> 3.53 ms; Q4_0 3.21, Q4_1 2.76,
Q5_0 3.74 ms after; the FP16 output hash of every type unchanged. dequant_bench lists Q4_0 / Q4_1 / Q5_0 too.
The lane kernels of native_expert_grouped re-read and re-decoded a Q4_K / Q5_K / Q6_K / Q5_1 / Q8_0 expert row once
per entry of its group (the IQ formats already decode once, Multi). Each of these dots is now split into load (the
weight words and scales) and apply (the activations and the same float expression), and the group's entries go
through the existing multi-entry path in chunks of 4 and 2; every entry keeps its own k order and lane sum, so the
outputs are bitwise the per-entry kernels'. STRATA_KQ_SPLIT=0 keeps the old kernels.
B70 (bmg-g31), 2560 x 640 experts, GB/s of expert bytes, 2- / 4-token windows: Q4_K/Q5_1 362 -> 402 / 369 -> 425,
Q5_K/Q8_0 395 -> 440 / 404 -> 464, Q6_K/Q8_0 259 -> 307 / 256 -> 318, Q8_0 430 -> 479 / 429 -> 502; one token -1..2%.
Experts read from pinned host memory (the NO_HOST mirror) go from 18-19 to 23 GB/s on those windows.
Also: STRATA_KQ_GU_LANES (opt-in; the K-quant / Q8_0 gate/up lane split alone, other bits), native_grouped_parity
--kq (split vs per-entry bitwise, and a CPU float reference with ggml's dequantize_row_* for Q4_K/Q5_K/Q6_K/Q5_1/Q8_0)
and --kqbench (per format pair, device or pinned host memory), Q6_K blobs with a finite scale in that harness,
dequant_bench and kq_cpu_bench for the Unsloth formats in the SYCL build.
--mtp D (0..7) loads blk.<n_layers> (eh_proj, enorm/hnorm, attention + MoE
layer, shared_head_norm) and runs greedy requests as speculative steps:
D chained drafts, one verify window, commit of the accepted rows' GDN state
and conv history. Verify windows and plain decode share the per-token GDN
kernels, so the text is the plain greedy text (10 of 10 prompts identical).
Serve reports accepted/offered drafts in DONE.
- moe_mm.metal: W8<Q4_K / Q5_K / Q5_1 / Q6_K> (the lane's runs k and k + 16 share the sub-block's scale, min and
high bits; 4-aligned code runs read as one uint); mm_down3 takes the paired-run decode on the COOP_K4 layout (TK
32). Prompt per layer at T 2048: Q4_K gate/up 10.6 -> 6.0 ms, Q5_K 15.4 -> 9.0, Q6_K 17.2 -> 14.0, Q5_1 down
5.0 -> 3.3; the same values (moe_mm_parity v3 incl. Q2_0 and IQ3_XXS / IQ4_NL)
- qsa.metal: QSA_MROPE (function constant 24) - interleaved M-RoPE (ggml IMROPE, sections 11/11/10): buffer 20 of
qsa_prep = int32 [cells][3] (t, h, w); rotary pair i of the row at pos_base + cell rotates by the cos/sin row
tab[(pos_base + cell) * 3 + i % 3] (pooled indexer keys at pos_base + 4 b too). qsa_parity QSA_MROPE=1 (random
table vs the CPU reference) / QSA_MROPE=id (identity table: the same numbers as the plain kernel)
The HC read takes, per matrix, its load-time Q8_0 copy or the file's own tensor: Unsloth's Q8_0 down/up need no copy,
their F32 inject rows go to gr_down_q8f (mac-kern 945c0d24). router_route_<type> for BF16 and F32 (opt-in). The M4's
v2 prompt kernels: K-quant gate/up TK 128, Q8_0 / IQ4_XS TK 32, Q5_1 / Q8_0 / IQ4_NL down TK 32. GSQ packs unchanged:
Q2_0 prompt1/story, IQ3_XXS prompt1 and the 1,662-token chain prompt give the same tokens as before (M5 Max).
tools/vision/strata_vision.cpp: no posix_fadvise on macOS (it does not have it); strata-vision then builds there with
ggml-metal.
- unsloth_budget / UNSLOTH_RESIDENT_BESIDE_GB 16, ported from sergqwer's fork (sergqwer/strata-nvfp4 3f3546f6, MIT):
no --resident-budget-gib where the experts and 16 GiB fit the RAM (UD-IQ4_XS 71.4, UD-Q4_K_XL 87.7, UD-Q5_K_XL 107.6,
UD-Q6_K_XL 117.7, Q8_0 135.5 GiB), the budget below that, when asked for, and on a unified-memory APU (#1715).
The menu, --check and the install say which; the menu's default for the family is the largest file whose experts
all fit (UD-IQ4_XS where none does).
- UD-Q5_K_XL / UD-Q6_K_XL / Q8_0: six shards each, sizes and SHA-256 pinned at 38bb39ee; "ram_gb" is the all-in-RAM
need, below it setup asks first (default no) and says how many GiB are read from the SSD. Q5/Q6 need engine 0.1.44
(the Q6_K expert kernels in every build); Q8_0 0.1.38.
- UD-Q4_K_XL: ran on an R9700 (docs/QUANTS.md), so #429's AMD question is gone, and it is not experimental.
- tools/strata_mcp.py mirrors the table and the verdicts. Golden configs re-recorded: only UD-Q4_K_XL on 96 GB
(65536, --kv-resident) and 128 GB (131072) changed, as in the fork.
Tests: test_setup_unsloth Resident and BigFiles (the budget tests force the budget mode where their PC now fits every
expert), test_strata_mcp, every tools/test_setup_*.py.
Co-authored-by: sergiywith <[email protected]>
A driver of its own beside Flash-Next's (generate/verify/prefill untouched), built from Strata's kernels:
- weights from the GGUF: dense Q8_0 on the native MMVQ (with the packed Q8_0 copy), the routed experts
repacked at load into native expert blobs (stride in whole blocks for MMQ), F32 norms / projections;
- the plain pre-norm residual layer driver, the residual add fused with the next norm, the MoE end
(combine + shared-expert gate + add + next norm) in one kernel;
- GDN: the fused conv+L2 and SiLU step for a token, one-launch sequential kernels for a prompt chunk;
- attention without an indexer: split-K decode over 256-cell chunks (positions on the device, so the
decode step is one captured graph), and for a prompt chunk scores / values as FP32 GEMMs per KV head
with a causal softmax;
- prompt projections as FP16 GEMMs (prefill::Gemm::native), the F32 router / alpha / beta as SGEMM,
the prompt's experts on llama.cpp's MMQ (entries sorted by expert on the device);
- decode: native router 256/8, native grouped experts (Q5_K / Q6_K downs added to the grouped
kernels' down list for FF 512), router + shared gate in one product, alpha/beta in one launch.
- serve: READY / GEN (sampling keys) / T / PP / DONE / STOP, the held state reused when a request
extends it; --tokens / --force / --dump-full-logits for tools/qwen35/kl.py; --bench-pp / --bench-tg.
- hip_compat cublas_v2.h: cublasSgemm / cublasSgemmStridedBatched.
setup_mac.py: the Unsloth files by name and SHA-256 (setup.py's pinned revision), repacked into one GGUF for the GPU
with every tensor verified, the n-gram table found by name, --native-dense-gguf the repacked file; setup writes
STRATA_METAL_STREAM to the config so the engine follows its choice. UD-* are behind 'missing' (the Metal expert
kernels for Q4_K/Q5_K/Q6_K/Q5_1 are in progress) and only reachable with --model.
stream.cpp: auto streams when the mapped weights + 12 GiB do not fit the GPU working set (was: experts > 60% of
it, which would have streamed UD-Q4_K_XL on the 128 GB M5 Max). M5 Max: Q2_0 stays resident (same tokens and speed);
M4 32 GB: Q2_0 streams as before.
Shard 1's metadata (without split.*) and every tensor but the n-gram table, byte for byte: dense first, then each
layer's routed experts together; --verify hashes every tensor (SHA-256) against its source shard. For Unsloth's
UD-*_K_XL (4-6 shards cutting through layers). Checked on Q2_0 (M5 Max): 35.05 GiB, 1,223 tensors identical,
written and verified in 33 s; the engine on it (--native-dense-gguf the same file, --ple-gguf the original shard 2)
gives the same tokens as on the original shards, prompt1 and story, at the same speed (114.2 / 107.7 tok/s).
From sergqwer/strata-nvfp4 1688a344 (src/prefill/mmq_expert_bench.cpp), with UD-IQ4_XS's gate/up and down formats
added as comparison cases (an f16 d leads their blocks, as Q8_0's).
Co-authored-by: sergiywith <[email protected]>
row_dot_20_sub8: lane t takes k = t (+ k = t + 16 for t < 4, else + 0) and k = t + 8 (+ 0), then xor 4, 2, 1 - the
order warp_sum adds a K = 20 row in (lane i + 16, then + 8), so the result is the same bits; native_down_multi_kernel
routes TD 40 there (as IQ4_NL's K = 40 row_dot_40_sub8) and the down grid takes 32 rows a block for d_type 40.
nvfp4_decode_bench, RTX 3060, 64 distinct experts: down alone 0.260 -> 0.189 ms at 1 token (226 -> 312 GB/s, 87% of
360); the whole expert 0.626 -> 0.554 ms (89%), 2 tokens 0.686 -> 0.596, 3 0.786 -> 0.661, 4 0.910 -> 0.766 ms (from
2.261 before Split<40>). Output hashes unchanged, nvfp4_expert_gpu_parity unchanged.
The ported NVFP4 decode experts had no Split, so they took the per-entry kernels: every weight was decoded again for
every token of a verify window, and the kernel slowed almost linearly with the tokens per expert. Measured with
nvfp4_decode_bench on the RTX 3060 (64 distinct experts, sm_86, CUDA 13.4; best of 50):
| tokens per expert | before | after | |
|---|---:|---:|---:|
| 1 | 0.714 ms (248 GB/s) | 0.626 ms (283 GB/s) | 1.14x |
| 2 | 1.200 ms | 0.686 ms | 1.75x |
| 3 | 1.730 ms | 0.786 ms | 2.20x |
| 4 | 2.261 ms | 0.910 ms | 2.48x |
(UD-IQ4_XS's IQ3_S / IQ4_NL experts, 15% fewer bytes: 0.512 / 0.578 / - / 0.814 ms.)
- Split<40>: vec_dot_nvfp4_q8_1 split into load (the E2M1 codes through kvalues_fp4, the two sub-block scales) and
apply (the dp4a chain and the float expression in the same order), so the multi-entry kernels apply one decode to
up to four tokens. The multi kernels now apply the NVFP4 tail scales on their outputs as the per-entry kernels do
(s_gate / s_up on gate/up, s_down on down; x * 1.0f for every other format, which is exact).
- ue4m3_to_f32_half builds the value from its bits instead of two ldexpf and branches: equal for all 256 bytes
(nvfp4_decode_bench checks them).
- down: NVFP4's rows are K = 20 calls at n_ff 640, as Q2_0's, so they take row_dot_20_sub16 (2 rows a warp, 16 rows a
block) instead of one 20-of-32-lane row a warp: down 0.361 -> 0.260 ms at 1 token. Its addition tree is
warp_sum's (lane i + lane i+16 first), bitwise.
Bit for bit: the output hash of every case above equals the per-entry kernels' (STRATA_OLD_IQ_MMVQ=1);
nvfp4_expert_gpu_parity on real GPTQ blobs unchanged (1.088-1.168%). Other formats: only the store multiplies by a
1.0f scale, and the down launch grid changes for d_type 40 alone.
nvfp4_decode_bench [gu_type d_type] [groups] [tokens per group] [iters]: native_expert_grouped on distinct random
blobs (64 x 2.76 MB, far past L2), best/mean time, weight GB/s against the card's DRAM peak, and an FNV hash of the
output so a kernel change can be checked bit for bit. NVFP4_BENCH_PHASE=1|2 times gate/up or down alone. Also an
exhaustive check of the UE4M3 decode built from bits against the ldexpf form (all 256 bytes).
- fused_gdn_step_norm_silu: the fused GDN step with the SiLU output gate (Qwen3.6), a compile-time
template flag; fused_gdn_step_norm (Flash-Next, sigmoid) is the same kernel instantiated as before.
- native_router_topk(n_expert, k): 512/10 goes to the existing pinned top-10 kernels unchanged; 256/8
is a templated copy of the same topk-moe contract.
- strata-q36ref --kernels strata (default): native MMVQ (q8_1) for every quantized matmul, dense and
per routed expert (Q4_K/Q5_K/Q6_K/Q8_0), the fused GDN conv+L2 and SiLU step, the native router;
--kernels plain keeps the reference kernels.
- kl.py: answer-only statistics beside all steps, llama.cpp's logits cached between runs.
- quant.metal: QUANT_CTAB (function constant 23): the IQ3_XXS / IQ3_S / IQ2_XXS / IQ2_XS / IQ2_S grids and the sign
table read from ggml-common's constant arrays, no threadgroup copy (same values). Decode moe_gu: 3-7% faster
(IQ3_S-pack T2 window 14.96 -> 14.79 ms, 3 of 3); token_chain default (CHAIN_CTAB=0 off). Prompt mm_gu3: slower
(kept off)
- moe_mm.metal: mm_gu4 (decoded operand shared by MS 16-row token tiles): IQ3_S 9.5 -> 8.6 ms, IQ3_XXS equal, Q2_0
slower; mm_gu5 (weight blocks staged in threadgroup memory, coalesced): slower (12.9-20.6 ms); W8 templated on the
address space for IQ3_S / IQ3_XXS / IQ2_S; probes MM_PROBE 5 (loads only: as slow as the full decode)
- moe.metal MOE_PROBE (function constant 22; no table load)
- finding: the prompt i-quant decode is bound by its scattered small device loads (2-7 a 4-value run, 8 rows a load
instruction), not by the grid lookups or the ALU
--family nvfp4 --model NVFP4 | NVFP4-NVIDIA (MODEL-UPDATE Phase 4). The files come from the packs' Hugging Face
repositories at the commits the fork's setup pins (NVFP4_REPOS: Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata
640a1db9, ...-NVIDIA-NVFP4-Strata cca8f7fe), each checked against its size and SHA-256; nothing is converted on the
PC and no file is mirrored. The FP8 n-gram table and the BF16 embedding are the same file in both: a second install
links them. The MTP draft layer is setup's own, as for every model. The engine gets the fork's model flags
(--native and --native-dense-gguf on the dense GGUF, --ple-gguf, --embd-gguf); the rest of setup's choices (context,
KV, the low-RAM mode on the pack's experts.bin) are the usual ones: 96 GB keeps every expert in RAM, 64 GB with a
32 GB card the resident low-RAM mode. NVIDIA only: on AMD setup stops and names UD-IQ4_XS (the HIP engine has no
NVFP4 kernels). Images: off for these packs. A ready-made engine older than NVFP4_ENGINE (0.1.47, to be set at
release) is replaced by one compiled from this source.
Tests: tools/test_setup_nvfp4.py (fetch from a file:// mirror with SHA-256 checks, the shared file linked, a wrong
file refused, the config name read back, mocked installs on 128/96/64 GB, AMD refused); test_setup_pins covers the
new table. Every tools/test_setup_*.py passes.
Co-authored-by: sergiywith <[email protected]>
- sycl: iq_dequant_f16 / iq_dequant_gu_f16 / moe_combine accept the NVFP4 scale arguments the shared headers now
declare (refused if ever given: the SYCL port has no NVFP4 experts), and the no-MMQ stubs match the new mmq
signatures (the calls were ambiguous with the defaulted ones).
- CMake: list(TRANSFORM ... REPLACE) with \2 of an optional group failed on CMake 3.28 for a plain "120"; two
rewrites, one per form.
- src/qwen35/q36ref.cu (target strata-q36ref, CUDA and HIP): Qwen3.6-35B-A3B on one GPU, the whole
model resident, one token at a time with fp32 activations: plain pre-norm residual, GDN with the
SiLU output gate (32 V heads over 16 K heads, conv 8192), dense gated attention (16 Q / 2 KV x 256,
NEOX RoPE over 64 dims), softmax top-8 router over 256 experts with the sigmoid shared-expert gate,
every weight read in its GGUF form (F32/F16/BF16/Q8_0/Q4_K/Q5_K/Q6_K). --act q8 (default) rounds the
activations of quantized matmuls as llama.cpp's GPU path does; --force / --dump-full-logits for
teacher-forced KL; --dump-layers / --dump-logits for parity.
- tools/qwen35/: q36_llama_dump.cpp (llama.cpp's side through the eval callback), parity.py
(greedy answers + per-layer error), compare_dumps.py (two runs, e.g. the GPU-vs-CPU floor), kl.py
(teacher-forced KL against llama.cpp HIP with the CPU backend as the floor), parity_prompts.json.
The fork was built with MSVC only: on Linux its parity tool's F16C rounding helper failed to inline without the
target flag, and the SYCL tree's own list of CPU expert sources lacked nvfp4_avx512.cpp / nvfp4_avx2.cpp.
Port of sergqwer/strata-nvfp4 c83625c3 (the rule for NVFP4 experts in any layer) onto 0.1.43's default, with its
#1884 nearly-all-cached exception kept. On the fork's qwen-nvfp4-gptq pack the skip moved the answers by
teacher-forced KL 0.035 against it off, about the GPTQ quantization's own error against an all-Q8_0 reference
(0.038), for +11.5% decode (RTX 5090, the fork's measurement). Packs without NVFP4 experts keep the default
unchanged. The final default is the owner's, from our own 3060 A/B and KL against Q8_0.
Co-authored-by: sergiywith <[email protected]>
Without AVX-512 (Zen 2/3, Intel 12th-14th gen) the pool's NVFP4 rows went
through ggml-cpu's vec_dot, which re-decodes a row's blocks for every token of
a window. nvfp4_256_* decode a 64-value block once and apply it to up to 8
tokens. Each token's arithmetic is ggml_vec_dot_nvfp4_q8_0's AVX2 path step
for step (vpsignb, vpmaddubsw + vpmaddwd, the same FMA order, hsum_float_8),
so every row is bit-equal to it: an AVX2 CPU computes what it did before.
nvfp4_avx512_parity checks both kernels, whichever this CPU runs (the AVX2
rows must also be bit-equal; 0 differing rows of 1..9 tokens against the
portable and the native ggml-cpu builds), and is now a ctest
(nvfp4_cpu_parity). One expert's gate+up, Zen 5 core: 1.12x ggml at one
token, 2.00x at 4, 2.35x at 8; under DRAM load (--bw) with 6 threads 47.3
against 24.0 GB/s at 4 tokens, 26.6 against 12.0 at 8. Chat decode on the
RTX 5090 with STRATA_FORCE_AVX2=1 and 6 pool workers: 16.5-16.9 ms a round
against 17.3-17.5 (pool 6.6-6.8 against 7.2-7.3 ms).
STRATA_NO_NVFP4_256=1 falls back to ggml-cpu.
(cherry picked from commit 0c859e15f89054a376db7458bfb298e3e6c81f22 of sergqwer/strata-nvfp4)
Upstream's 04dbe561 allocates cpu_rows at its largest before the walk; the fork's allocation of cpu_ones (row_sd 1 for
the CPU's rows on an NVFP4 pack, b59f5dda) sat inside the reallocation that then never ran, so a staged chunk with the
CPU share copied from a null host pointer: "prefill moe_combine: invalid argument" (2K prompt, qwen-nvfp4-gptq).
(cherry picked from commit 52b272ebb92c9ef49029314da1a695d38f570d88 of sergqwer/strata-nvfp4)
Was 640df4f0, the fork's CPU share. Upstream 0.1.40.2 took it (978d3558, with 667f2eca's blobs taken on the
prefill thread and 4322e241's STRATA_DBG_NAN rows; opt-in there), so the port keeps upstream's code and adds only
what an NVFP4 pack needs: the CPU's rows carry s_down already, so their row_sd entries are 1 (cpu_ones, pinned
beside cpu_rows).
Port note: the release loop keeps 0.1.43's steps_up / tok_up / up_ring releases.
(cherry picked from commit a0c2ca233ac8fe8ec9d342bd319c93404791fa15 of sergqwer/strata-nvfp4)
- The blob keeps its {s_gate, s_up, s_down, 0} tail whenever any projection is NVFP4; the other projection's scale is
exactly 1.0, and every path (GPU decode, CPU pool, prompt path) already multiplies by the tail, so 1.0 is exact.
native_expert_layout (GPU) and native_fmt (CPU) no longer require NVFP4 in all three projections.
- tools/iq_pack.py writes such blobs (1.0 for a non-NVFP4 projection, .scale tensors only for the NVFP4 ones).
- native_expert_parity: the synthetic NVFP4 blob gets its tail (it was left zero, so every NVFP4 synthetic expert
computed 0 and failed), non-unit for NVFP4, 1.0 for the other projection, and the reference applies it.
nvfp4/q8_0, q8_0/nvfp4, nvfp4/nvfp4 and q8_0/q8_0 pass: CPU and GPU within 1e-7, ~1.2% from the float reference
(activation rounding).
(cherry picked from commit 473c0959141b36d28e7a2eb16005f8c8a4e3c8d0 of sergqwer/strata-nvfp4)
0.1.38 took the fork's group gather without the NVFP4 parts: each expert's 16-byte scale tail now goes to its
group slot and the group's last flush zeroes the MMQ tail, in the same launch (as #353 on 0.1.38).
(cherry picked from commit 8f2193f028940b9a6d270203b640562435937a7a of sergqwer/strata-nvfp4)
A ModelOpt NVFP4 checkpoint of Qwen3.8-Flash-Next (its routed experts NVFP4, group 16) runs on the native path.
Rebased on 0.1.31: NVFP4 is type 40 in the per-role format lists (STRATA_GU_FMTS / STRATA_D_FMTS / STRATA_MMVQ_FMTS)
beside Q4_K/Q5_K/Q5_1/Q8_0, and 0.1.31's dq_q8_0 serves both.
- tools/nvfp4_convert.py: llama.cpp's converter with the engine's type policy (BF16 control projections, Q8_0 native
tensors and token embedding, no PLE table); tools/nvfp4_verify.py / nvfp4_verify_gguf.py prove the experts bit for
bit.
- tools/iq_pack.py: expert_layout() gives NVFP4 blobs a 16-byte tail {s_gate, s_up, s_down, 0} (the per-expert
weight_scale_2), so every consumer that moves whole blobs moves the scales; an NVFP4 pack always writes experts.bin
(the GGUF has no tails, so the in-place GGUF read does not apply).
- decode: NVFP4 in the native expert kernels (q8_1, UE4M3/E2M1 decoded in software, bit-identical to ggml); each scale
on its own projection's FP32 output (folding s_down into up put the hidden into FP16's subnormals).
- CPU pool: AVX-512 NVFP4 rows (1.8-3.7x ggml-cpu's), ggml-cpu elsewhere.
- prompt path: llama.cpp MMQ - W4A8 by default (int8 MMQ with Blackwell's FP4 MMA hidden, mmq_nvfp4_w4a8.cu), W4A4
and FP16 through STRATA_PREFILL_NVFP4; the scales applied where they are read.
- CMake: the architectures stay as given. Only mmq_nvfp4_w4a4.cu (the FP4 x FP4 kernel and its FP4 activation
quantizer, symbols renamed) is built with 12x -> 12xa in strata_mmq_w4a4; without its image W4A4 falls back to W4A8.
strata.exe: 55 sm_120 cubins + 1 sm_120a.
- 0.25 of the missed experts over PCIe for NVFP4 packs (their blobs are 2.76 MB).
- Parity tools (mmq_nvfp4_parity, nvfp4_avx512_parity, nvfp4_expert_gpu_parity); HIP guards; docs/NVFP4.md.
Checked on this rebase (RTX 5090): the parity tools as before (decode experts 1.09-1.16%, MMQ W4A8 0.53%, W4A4 8.6%
- the same numbers as the 120a-everywhere build); IQ2_XS 128 greedy tokens identical to 0.1.31 main; the NVFP4 pack
against the 0.1.29-based build at the same expert-cache size: identical tokens on a 24-token and an 8K prompt, in W4A8
and in W4A4 (prefill 4,231 / 4,583 tok/s on 8K).
Port note: the PCIe share base for NVFP4 packs (0.25) sits beside 0.1.43's PcieAuto start values.
(cherry picked from commit 1f8a12858cc4ed80a977767afef6b6c3a754ac63 of sergqwer/strata-nvfp4)
- serve/images.py: OpenAI's images API (prompt, size, n, response_format b64_json|url) plus seed and steps; one
sd-cli process per request with the model card's arguments (8 steps, CFG 1, euler, Turbo's sigmas,
--diffusion-fa), the PNGs returned unchanged. The job takes the service's turn (one job at a time on the GPU)
and the config's gpu_policy says what the text engine does meanwhile: keep, shrink (--vram-elastic), unload
(its conversation saved to a session file and restored after the next start), auto.
- setup.py --images qwen-image (--image-quant, --image-backend, --image-gpu-policy): sd.cpp 2988060 + its ggml
as zips, built for CUDA / HIP / Vulkan (LunarG SDK into third_party when the PC has no Vulkan dev files) / CPU;
DiT quant by GPU memory (Q4_K_M <= 8 GB, Q6_K <= 12, Q8_0 above), VAE and the Qwen3-VL-8B UD-Q4_K_XL text
encoder at pinned revisions, each checked against the Hub's SHA-256. tools/image/install.py adds it to an
existing install.
- Web page: an Images tab when the server has image generation.
- docs/IMAGES.md; tests: serve/test_images.py, tools/test_qimage.py.
IQ3_S runs on Metal since mac-kern's IQ4_XS kernels: recommended from 110 GB (Apple's 128 GB Macs), IQ3_XXS where it
fits resident, then Q2_0. A pack that does not fit Metal's share of the memory streams its experts from the SSD (the
engine decides, f-mac-ssd): setup says so with the M4 numbers, writes --suffix-draft 0 and no --mtp (one token per
step, measured faster there) and does not fetch the draft layer.
session_zero and conversation_checkpoint_save/restore go to the Metal window under STRATA_GPU_METAL (the running
state lives there, not in the session's buffers): reset zeroes its GDN recurrences and conv histories, PLE history
and indexer tails (and the draft layer's); a checkpoint holds the same set, after the pending fold. Before this a
new request carried the last conversation's state (M5 Max, Q2_0: an unrelated request after a story answered
differently from a fresh engine; now the same), and on a streaming Mac the session buffers it zeroed or copied had
been given back (the engine stopped).
Checked on the M5 Max (resident) and the M4 (streaming): an unrelated request after another equals a fresh
engine's answer; a three-turn conversation reuses 35 / 73 of its tokens and its second answer equals a fresh
engine's. Shared files: session.cpp and conversation_state.cpp preprocess IDENTICAL and their objects are byte-
IDENTICAL on CUDA (P1004) and HIP (p620). Also: IQ4_XS's v2 prompt tile (_16_64_32_2), and a call stack when a
given-back allocation is touched.
- ModelFamily from general.architecture (qwen4exp, qwen35moe); the architecture guard accepts both, the
engine's guard (native dense/head, --native) refuses Qwen3.6 with "recognised, not run yet".
- ModelGeometry moved to core/geometry.hpp (+ family, shared-expert FFN, in-file MTP layers, vocab);
geometry_from_gguf reads every shape key, checked against the compiled kernels (Flash-Next files give
exactly the defaults; expert count and top-k stay run-time sizes).
- Per-family tensor layout tables (GDN / attention / nextn MTP block, hyper-connections or plain
pre-norm, indexer or none) and a GGUF directory check; MXFP4 block size in the reader.
- setup: --family qwen36 downloads unsloth/Qwen3.6-35B-A3B-MTP-GGUF (pinned 5bc3e238, size + SHA-256)
UD-Q4_K_XL / UD-Q6_K_XL / Q8_0 (+ mmproj) and says the engine does not run it yet; serve's default
model name per family; strata_inspect recognises qwen35moe.
- Tests: model_family_test (synthetic, --real for headers), test_setup_qwen36, serve/test_family_name.
Conflicts in metal/src/engine/window.cpp resolved: the stream's next-layer routing guess beside router_route
(opt-in); the cached window through the stream's segmented run, the kernel profile off while streaming; host reads of
the weights through W_ (the mapped file, or the dense weights in memory when streaming), also in prompt_chain.inc;
the drafter's prompt rows batched, from either prompt path. The prompt: resident -> prompt_chain.inc, streaming ->
the stream's prompt path (it streams each layer's experts for the chunk); STRATA_METAL_PROMPT=window: 8-row windows.
M5 Max: Q2_0 prompt1/story and IQ3_XXS prompt1 with MTP equal to the no-MTP references; the 1,662-token prompt
through prompt_chain.inc equal to before; Q2_0 streamed (STRATA_METAL_STREAM=1) equal to resident on prompt1. The
1,662-token prompt streamed (the stream's prompt kernels: chunks of 1024, flash QSA) first differs at token 21 of 60
from the resident prompt_chain.inc path (different kernels and chunking).
- gemv.metal: gemv_x<F,LPR,NR,SG> = gemv (KS 1) with the window's prepared x staged in threadgroup memory (bitwise
the same arithmetic); GEMV_PROBE 1/2/3 bench probes (skip later tokens' x loads / combine; rows-outer order)
- units.metal: X<F>::load templated on the pointer's address space (device or threadgroup), same arithmetic
- gemv_parity: GEMV_X=1 sweeps gemvx_*, GEMV_PROBE passes the probe
- M4 finding: a window token's cost is the fp32 combine (1 FMA a weight a token), at ~0.85 T FMA/s (near the M4's
peak); the x loads are ~6 us of the ~30-60 us a token, and staging x in threadgroup memory does not help
setup.py hands a darwin run to metal/setup_mac.py (nothing else changes on a PC); setup.sh fetches a pinned
standalone Python 3.12 into .python on a Mac without Python 3.10+ (no sudo, no Homebrew). The Mac setup checks the
chip, unified memory, macOS (15+) and the Command Line Tools; recommends the pack by memory (IQ3_XXS from ~96 GB,
Q2_0 below; IQ3_S only with --model: the Metal expert kernels lack its IQ4_XS experts); builds the engine with pip's
cmake/ninja, Apple's metal-cpp (pinned SHA-256) and llama.cpp's ggml at the pinned commit; prepares the native pack
and the MTP draft layer; writes strata-<model>.json (backend metal, --spec 2) and run-<model>.sh; starts
serve/server.py. tools/test_setup_mac.py: 10 tests with sysctl/sw_vers/xcode-select mocked.
On the M5 Max (m1002): ./setup.sh --model Q2_0 --gguf-dir ... --yes from a fresh copy: Python fetched, engine built,
pack and config written in 29 s (models already downloaded); the server answered /v1/models, the chat page,
streamed /v1/chat/completions and /v1/messages (Q2_0, 112-121 tok/s). python -m pytest serve -q on the Mac: 705
passed, 11 skipped.
The stdin reader (a detached thread) can take in_mu after the serve scope destroyed it, when the server closes the
pipe after QUIT; on macOS that lock throws and the engine ended with an abort. Inside #if STRATA_GPU_METAL: flush
and _Exit(0) after the QUIT profile save. HIP (p620): generate.cpp preprocesses IDENTICAL and its object is
byte-IDENTICAL against the gated tree (P2003 was unreachable for the CUDA compare; the change is preprocessor-only).
The serve loop asks cvec() every request: Metal returns the empty one (no --control-vector-scaled on Metal yet)
instead of the stub that stopped the engine; stubs.cpp regenerated without it. An uncaught exception now prints its
call stack (backtrace_symbols_fd) before the abort.
The BF16 router GEMV and moe_route as one dispatch beside the shared expert's GEMVs. Greedy output unchanged (Q2_0,
IQ3_XXS, both prompts, on and off). Not the default: in this window the router GEMV already runs beside the
shared GEMVs, and the T2 window measured 13.035 ms GPU with it vs 13.027 without (the fused kernel alone 0.615 ms
vs 0.566 for the two); end to end, --spec 2, 3 interleaved pairs, off -> on: Q2_0 113.61 -> 113.28 / 107.50 ->
106.97, IQ3_XXS 103.51 -> 103.07 / 96.19 -> 96.02 tok/s.
- router.metal (new): router_route<F> - the router GEMV rows as gemv's (LPR 32, KS 1) with 512-thread threadgroups,
the last threadgroup to finish (device counter, device-scope fences) runs moe_route's body and resets the counter;
logits / ids / w / plan bitwise gemv + moe_route. token_chain default (CHAIN_ROUTE=0: separate): T2 13.08 -> 12.93 ms
- gemv.metal gemv_cat: two same-format matrices over one x in one dispatch (ssm_alpha + ssm_beta), bitwise; no gain
in graph mode (CHAIN_ALBE opt-in)
- gr2.metal gr_norm_comb: the MoE combine folded into gr_norm, bitwise; +0.6 ms at T2 (CHAIN_COMB opt-in, not kept)
- units.metal Dec<Q3_K>: qs / hmask as packed_ushort4 loads (same values): T2 2560->10240 33.8 -> 32.2 us
- tests/fuse_parity: the three fusions bitwise for 1..4 tokens
With it a verify window of T > 1 commits its first row in place (commit = 0x100 | 1, mac-kern 56526aed): commit(1)
needs no fold, commit(n) folds rows 1..n-1; a prompt window commits all its rows in the step. Greedy output equal
to no MTP (Q2_0, IQ3_XXS, both prompts, --spec 2 and 4, the window prompt path).
Not the default: M5 Max, --spec 2, 3 interleaved pairs, off -> on: Q2_0 113.35 -> 113.24 / 106.98 -> 106.75,
IQ3_XXS 103.53 -> 102.71 / 96.18 -> 95.88 tok/s. The in-place state write (36 x 3 MB) costs every window; the
fold it saves is only the rounds with no accepted draft (24-36% at --spec 2).
- gdn.metal: with commit = 0x100 | k (1 <= k <= GDN_NTOK) the state is written in place and the conv history to
hist_out right after token k - 1, while every token's y is still computed; commit = 1 / 0 unchanged (the full
commit now writes at the last token inside the loop - same values, same speed)
- gdn_parity: the partial commits are bitwise a k-token commit's state and history, and y bitwise the verify window's
(k = 1 .. ntok - 1, ntok 2..4); gr_parity GR_DOWN=SG_KS picks gr_down's variant (sweep: 10.1-11.8 us at T 2, none
better than 8_8 by more than 7%)
Once, after the 20th cached window of T tokens with the head: the window's dispatches grouped by pipeline and weight
role, each group run by itself (7 runs, median). Profiling only: the reruns move the state, so the text after it
is not meant to be read.
Owner decision. Only the Options default under STRATA_GPU_METAL; CUDA/HIP/SYCL keep 0: on P2003 the CUDA
generate.cpp translation unit preprocesses IDENTICAL and its object is byte-IDENTICAL.
M5 Max, MTP, 200 tokens, greedy output equal to no MTP in every run: Q2_0 114.0 / 107.9 tok/s (prompt1 / story,
--suffix-draft 0; was 112.0 / 86.2 at --spec 4), 119.5 / 108.2 with the default suffix drafter; IQ3_XXS 103.5 /
96.9, 108.9 / 97.0 (was 110.2 / 87.4).
The GDN step writes the prepared y planes (no xprep dispatch per GDN layer); a window whose context is within the
2,051-cell budget skips the QSA score and top-k (qids_ holds the identity, restored after a selecting window; the
cached encode is keyed by it). STRATA_METAL_FUSE=0: off.
M5 Max, Q2_0, 200 tokens, 3 interleaved pairs: no MTP 85.1 -> 87.4 (prompt1) and 85.2 -> 87.3 (story); MTP 110.7 ->
112.0 and 85.3 -> 86.2 tok/s; tokens identical on and off and equal to the no-MTP reference.
prompt_chain.inc: tests/prompt_chain.cpp's chain (6f6c555b) into the decode window's own state - fp16 GEMMs on the
matrix units, grouped expert GEMMs, the GDN prompt scan, QSA K/V and indexer keys into the window's caches - plus
the PLE block after layer 0, every row's final residual for the drafter (its prompt rows in one command buffer),
the history parity, RoPE table and pending fold. Chunks of 2048 rows (STRATA_METAL_PROMPT_CHUNK); COOP_K4 picks
the register-operand kernels on the M5 and the older ones elsewhere. The 8-row-window path stays as the fallback
(STRATA_METAL_PROMPT=window, and prompts under STRATA_METAL_PROMPT_MIN = 32 rows).
M5 Max: Q2_0 4K/32K prompts 1,434/1,462 tok/s (window path 190/174), IQ3_XXS 1,260/1,240. Teacher-forced KL of
the chain's prompt vs the window path's, 28 prompts: 0.0004 (top-1 99.4-99.6%); needles 4K-32K x 10/50/90: the
same answers as the window path (12/12, both models).
- gdn_prompt.metal: gdn_cprep (per head and chunk: gates in log space, A, forward substitution for U and W),
gdn_cstate (per head and 32 state columns: the chunks in order), gdn_cout (the intra-chunk output)
- gdn_prompt_parity: CB -100 = the chunked form; y 3e-7, state 2e-7 vs the decode step; 7.4 ms vs 1.9 ms at T 2048
(cprep 1.5, cstate 4.7, cout 1.4) - it needs the matrix units to pay off
- moe_mm.metal: the paired-run (k, k + 16) decode is taken on the COOP_K4 layout unconditionally (coop_probe now
checks that pairing on both tiles used); the runtime per-element check cost ~30%. The MMA-only ceiling of mm_gu3
(MM_PROBE=1: no decode) is 2.45 ms a layer = 55 TFLOP/s useful, so the decode is what is left. Measured and not
kept: fp16-math Q2_0 decode (slower), a W8<Q2_0> sharing the block scale (slower than two W4)
- glue.metal slc_touch + token_chain CHAIN_PREFETCH: the next HC read's weights touched concurrently with the first
projection GEMV after a read; measured +0.83 ms a token (10.46 -> 11.29, 3 of 3 interleaved) - not kept, opt-in
A prompt of 64+ tokens runs in chunks (STRATA_METAL_PROMPT_CHUNK, default 1024 streaming / 2048 resident) through
mac-kern's prompt kernels (fp16 GEMMs, the GDN scan, QSA over the chunk, grouped expert GEMMs v2 on the M4, v3 on the
M5), the PLE block for T rows, on the decode window's GDN / K/V / PLE state; the draft layer's prompt K/V read the
chunk's final rows. Streaming: the expert GEMMs address arena slots (function constant 20 in moe_mm.metal too), a
chunk's handshake puts the layer's experts in place and streams the next layer's non-resident experts into the LRU
tail behind the compute (the decode-hot experts stay); slots in use by the running layer are pinned. The host
code's own device weight copies are given back on a streaming Mac (purge_device_allocations).
STRATA_METAL_PROMPT_GEMM=0 / _MIN, STRATA_METAL_PROMPT_EXPERTS=v2|v3, STRATA_METAL_PROMPT_QSA=v1|flash.
- qsa_attn_mm: the P V matmul in K = 32 cell runs (K = 64 runs computed garbage on the M4), the output through
threadgroup memory (a direct matmul2d store into 258-float rows lost the threadgroup on the M4); also faster on the
M5 (8.1 -> 7.4 ms at pos 0)
- qsa_flash: the same K = 32 runs (was a hang on the M4) and a fresh per-chunk accumulator folded into the running
one (on the M4 a run does not accumulate into elements rescaled in place); now about equal to attn_mm + merge on
the M5 (9.4 vs 9.5 ms at pos 0, 20.1 vs 20.3 at 30000) and the prompt_chain default there (PC_QSA=flash|mm|v1);
the M4 defaults to v1 attention (faster there) with the GEMM score
- qsa_attn_mm bisection probe (function constant 8, QSA_STOP), qsa_prompt_parity QSA_DEBUG / pipeline report;
coop_layout prints the destination layout too
The dense weights are read into one resident buffer; the routed experts live in an arena of slots filled from the
GGUF with F_NOCACHE preads (several pieces of an expert in flight); the window runs in 48 segments with a shared-event
handshake after each layer's router (mtl_rt run_graph_seg), where the host puts the missing experts in place (LRU)
and reads ahead the experts the next layer's router predicts from this layer's FFN input. The expert kernels address
slots through smap (function constant 20); the resident path is unchanged. STRATA_METAL_STREAM=auto|1|0.
- moe_mm.metal mm_gu3/mm_down3 and qsa_prompt.metal qsa_attn_mm/qsa_flash: with COOP_K4 = false every element is
decoded on its own (the M4 holds runs of 2 n at one k); the experts are correct on the M4 that way
- tests/coop_probe.h: probes the layout (COOP_K4=0/1 overrides); moe_mm_parity, qsa_prompt_parity, prompt_chain
set the constant; prompt_chain defaults to experts v2 and QSA v1 without it (faster / correct on the M4)
- qsa_prompt_parity: NaN-fills the partials before the matrix-unit run (a kernel that writes nothing shows),
line-buffered output, QSA_NOFLASH
STRATA_METAL_LOGPOS=<path>: per decode position of one-token windows, the log-probability of the token that came
next (with --spec-follow, the forced text) and the top 64, in phaseA-tests cuda_kl.py's row format.
STRATA_METAL_PROFILE=1: per window size, host time before the GPU, run_graph wall vs GPU time, commit and draft
round times, printed when the window goes away. Both off unless set.
commit(n) after a window of several tokens no longer submits its own command buffer: the fold of the kept prefix is
encoded first in the next window's (cached per pending count). Nothing reads the GDN state in between.
STRATA_METAL_LAZY_FOLD=0 folds at commit as before.
M5 Max, Q2_0, MTP --spec 4, 200 tokens, 3 interleaved pairs: commit 0.87 -> 0.05 ms a round; prompt1 109.25 ->
110.21 tok/s, story 83.99 -> 85.14 (medians); greedy output equal to no MTP, lazy and not.
0.1.42 let --batch-mtp run on a layer split and made the --batch-groups auto default give way to it: one serial group. On
2x RTX A4000 (IQ3_XXS, 3 pairs) four streams aggregated 62.2 tok/s against 92.7 on 0.1.41's pipelined groups (-33%, 0/3 pairs
faster), two streams 55.7 against 71.4 (-22%). With --batch-groups 2 (which turns --batch-mtp off) 88.3 and 71.1. The start-up
line says why --batch-mtp is off and how to keep it. --adapt-async 1 still gives one group. Solo decode and the answers are
untouched.
- qsa_prompt.metal: qsa_score_mm (fp32 MPP, relu-sum epilogue, the tail block scalar; 6.2 -> 1.1 ms at pos 30000,
T 2048), qsa_flash (whole attention a threadgroup with an online softmax, fp16 out; correct but 27.6 ms vs 20.9 for
attn_mm + merge - not used)
- moe_mm.metal: W8<F> decodes a lane's two runs of a 32-group with one read of the group's scale/sign fields;
IQ3_XXS gate/up 9.3 -> 6.7 ms a layer
- prompt_chain: PC_QSA=mm uses qsa_score_mm too; gate/up 16,32,32,4 for every format; breakdown sees the far
chunk's classes
msl/mtp_chain.metal: mtp_pick (the pick through the draft vocabulary and its softmax probability) and mtp_embed_F
(the picked token's embedding row through mac-kern's unit decoders), so every draft step of a round runs in one
command buffer. At load the GPU embedding is checked against the host's dequant (max |diff| 0 on Q3_K and IQ3_S;
any difference falls back to a command buffer per step). STRATA_METAL_MTP_CHAIN=0 keeps the per-step path.
Fix: with accepted drafts (a > 0) draft 0's embedding overwrote the catch-up's row 0 before the GPU read it; draft
0's row now goes to memb_ row a. The dense attention's cell list is per step.
M5 Max, 200 tokens, --spec 4, greedy output equal to no MTP in every run: Q2_0 109.5 / 84.4 tok/s (was 101.4 /
80.0), IQ3_XXS 107.9 / 86.0 (was 101.5 / 82.8).
- MTP (src/core/mtp.cpp hook under STRATA_GPU_METAL: load / bind / prefill / draft / draft_first): the draft layer
of mtp.hpp on mac-kern's kernels, from the MTP GGUF (tools/mtp_pack.py, beside --mtp's rt dir; the F32 norms
folded to 1 + w as tools/mtp_rt.py does): input branches (msl/mtp.metal: rms, strided xprep, stream combine),
the three HC reads, K/V and indexer keys through qsa_prep into its own cache, dense attention over the last
2,051 cells, the MoE (fused gate_up experts), the final mixer and the head over data/draft_vocab.bin's subset
(STRATA_METAL_DRAFT_VOCAB=full: the whole head). One command buffer per draft step (the next step's embedding
is dequantized on the host). Greedy output with MTP equals the output without it, token for token (Q2_0, two
prompts, BF16 and Q8_0 HC); acceptance equals the HIP engine's on the same prompts (124 of 228; 104 of 291 vs 285).
- Prefill (src/prefill/prefill.cpp hook): init brings up the decode window, run reads the prompt through windows
of 8 rows (no head) and feeds on_chunk (the MTP's K/V); draft_kv declines. Prompt windows of 8 rows give the
same tokens as one-token windows. generate.cpp's own Metal prompt loop is gone again (--prefill auto as on the
PC). One window per process (the prompt path may bring it up before the Verifier).
- msl/ple.metal: the PLE block's norms, gate, conv and residual update on the GPU (same tokens as the CPU version).
- The HC read in Q8_0 by default (the owner's decision; STRATA_HC_Q8=0: BF16): the down / up / inject matrices
quantized at load (290 matrices, 680 MB, 70 ms), read by gr_down_q8 / gr_up_q8.
- The encoded window is cached per (T, head, history parity) and only its positions are patched per window.
- f-mac-kern merged up to 4775d233.
M5 Max, Q2_0, greedy, 200 tokens: Q8_0 HC 85 tok/s without MTP; with MTP 101 tok/s (acceptance 0.54) and 79 tok/s
(acceptance 0.36, a prompt the drafter does badly on everywhere: HIP 71 tok/s there).
- moe_mm.metal: mm_gu3 / mm_down3 - weights decoded per lane straight into matmul2d's right-input cooperative tensor
(single-simdgroup scope, no threadgroup staging or barriers in the K loop); W4<F> 4-value decoders for Q2_0,
IQ3_XXS, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_S, IQ4_NL (quant.metal's deq arithmetic)
- qsa_prompt.metal: qsa_attn_mm (S = Q K^T and P V as single-simdgroup matmuls, int8 K/V decoded into the right
input; same partials as qsa_attn, so qsa_merge is unchanged), qsa_q_f16
- tests: qsa_prompt_parity, coop_layout (the cooperative tensors' element layout), moe_mm_parity --v3;
prompt_chain PC_GU / PC_DN / PC_QSA switches (defaults v3 / v3 / mm)
#1240 stopped creating the drafter's branch stream and its two events on HIP: nothing uses them there, and an R9700
lost prompt speed with them. On gfx103x the stream's existence alone makes prompts read faster. RX 6800, Windows,
STRATA_HIP_PROMPT_F16=1, IQ3_XXS, two runs each: a 7.9K-token prompt at 798-808 tok/s with the stream and 584-596
without, a 30.4K-token one at 855-863 against 553-600; all of it is in the prompt path's "wait copy" phase. The two
events change nothing, so they stay out.
gfx103x creates the stream; other HIP cards do not, as before. STRATA_MTP_SIDE_STREAM=1 / =0 forces either. CUDA is
unchanged. Measured on one card under Windows only.
- token_chain: the HC read defaults to the Q8_0 copies through the 3-dispatch read (q8_3); STRATA_HC_Q8=0 selects
the exact BF16 path (--hc still overrides)
- prompt.metal: route_topk, gr_epi_down/up, swiglu_h, moe_out, gr_norm_h (fp16 xn), gemm_hc_up (the HC up GEMM with
the mixing epilogue; w_up rows interleaved by dq_f16's new perm field)
- gdn_prompt.metal: gdn_pre (parallel conv/norms/gates) -> gdn_scan_r (state scan, one barrier a token) -> gdn_post
- dense_mm.metal: DqArgs gains `perm` (4th field)
- moe_mm.metal: mm_scan zero-fills the unused simdgroup slots; indirect grids for gather/gate-up/down in meta;
MM_PROBE bench constant (function constant 5: pipelines of mm_gu2 need it set)
- tests: prompt_chain (whole prompt in chunks, class breakdown), prompt_parity, gdn_prompt_parity; mm_bench tiles
- window.cpp: a window of 2..8 tokens runs the GDN steps without committing; the projections it feeds them are
kept per layer, and commit(n) folds the kept prefix into the state (gdn_step over n tokens with commit, the same
kernel: bitwise the window's trajectory) and turns the conv-history ping-pong. K/V and indexer cells past the
prefix are rewritten when those positions run again, before anything reads them; the PLE history keeps the n
rows. Checked: greedy output with suffix drafts (windows up to 4, 50 of 59 drafts kept) is the same token for
token as one-token windows.
- generate.cpp (Metal only): --mmap-experts by default (the GGUF shards in place; the window reads them itself):
the 31.6 GB expert arena is not loaded any more (RSS 74 -> 40 GB, the rest is the mapped shard)
- stubs.cpp regenerated (256 entry points left as stubs)
IQ3_XXS vs the CUDA engine (P1004, f-mac, same flags, --kv int8): 200 of 200 tokens equal on the capital
prompt, 124 on a story; following CUDA's tokens 3 of 200 picks differ, at margins 0.07-0.23 (near ties).
The Metal build of `strata` is the shared host code (src/, unchanged except preprocessor-only Metal/arm64 branches)
over two Metal-only parts in metal/src/engine:
- window.cpp: the decode window, the one backend hook (Verifier::init/run/commit/copy_logits under
STRATA_GPU_METAL). It is mac-kern's chain on the model GGUF itself (shard 1 mmapped and wrapped no-copy, every
weight read in place: no upload, no expert cache): HC reads, GDN/QSA mixers with their own state (recurrence,
conv history ping-pong, int8 K/V, indexer), router + plan, routed and shared experts, combine, the final mixer,
the LM head and the greedy pick, encoded as one dependency graph. The embedding is dequantized on the CPU into
the shared buffer; the PLE rows come from the shared table reader, their projections run on the GPU and the
block's norms / gate / conv on the CPU between two command buffers. One-token windows (--mtp-max-t 1) for now.
- host helpers copied from the CUDA kernel files the host code reads while it starts (format tables, sizes, path
switches, the RoPE table/config); every other kernel entry point is a generated stub that stops with its name
(stubs.cpp, tools/gen_stubs.py, from clang's AST of our headers).
Shared host code, all inside #if STRATA_GPU_METAL / __APPLE__ / __aarch64__ (the PC objects do not change):
- verify.hpp/.cpp: the hook (an opaque window pointer; init/run/commit/wait_commit/copy_logits hand over)
- generate.cpp: no x86 ISA refusal; a native pack without --prefill reads its prompt through one-token windows;
the model shard handed to the window
- device.cu: the poison fill through the allocation's CPU view; pool_affinity_linux.hpp: a macOS branch (no
pinning); expert_layout.cpp: no xgetbv asm on arm64
metal/include/metal_compat: cpuid.h (all zero on arm64), linux/mman.h (no hugetlb/THP: the fallback taken).
M5 Max, Q2_0, greedy, "What is the capital of France?": "The capital of France is Paris." - the same 36 first
tokens as the HIP engine (R9700, same flags) on that prompt and 39 on a story prompt, then near-tie flips
(margins 0.08-0.41 when following the HIP tokens); decode 74-75 tok/s.
- kernels/mm_bench.metal + tests/mm_bench: M5 Max MetalPerformancePrimitives matmul2d fp16/bf16 62-65 TFLOP/s
(64x64 tiles), fp32 15, simdgroup_matrix half 14.
- kernels/moe_mm.metal: deterministic GPU plan (block histograms, scan, stable placement; padded per-expert tiles),
activation gather to fp16, mm_gu2 (gate+up+SwiGLU) / mm_down2 with weights dequantized (ref_quant arithmetic)
into threadgroup memory and matmul2d in multiply_accumulate mode, mm_combine. tests/moe_mm_parity: Q2_0 and the
IQ3_XXS-pack formats vs a reference with the same fp16 operands (1e-6..4e-5; vs exact float 2e-4..5e-4).
T=2048, 512 experts, per layer: Q2_0 8.0 ms = 25 TFLOP/s = 3.9 us a token (~5,300 tok/s experts only);
IQ3_XXS gate/up + IQ4_NL down 14.5 ms. The tile idea (dequantized weights from threadgroup memory under matmul2d)
follows yibie/strata-mlx's tile_matmul (MIT); the code is ours.
- kernels/dense_mm.metal: dq_f16 (any dense format to fp16 once a chunk), act_f16, gemm_f16 (MPP): 64-65 TFLOP/s
on the GDN/QSA shapes for 2048 tokens, dequantization ~4% of it; tests/dense_mm_parity (12 formats).
metal/CMakeLists.txt holds both: mac-kern's runtime face, kernels and parity programs as they are, and the device
runtime + the engine build (taken from ../src through include/metal_compat) when METAL_CPP_DIR is given.
runtime_hello is kept out of the kernel tree's test glob.
- gr.metal: gr_down_f<SG, KS, F> / gr_up_f<F> for BF16 (the old names) and Q8_0 (gr_down_q8_*, gr_up_q8);
tests/gr_parity GR_Q8=1: vs double on the dequantized Q8_0 weights, 1/3/8 tokens, M5 Max and M4 pass.
- token_chain --hc q8_3. M5 Max, --fuse --graph, interleaved, 2 rounds: Q2_0 1 token 11.62 (BF16) / 10.59 (Q8_0,
2 dispatches) / 10.38 ms (Q8_0, 3); 3 tokens 16.78 / 17.21 / 16.24-16.31. IQ3_XXS 1 token 12.85 / 11.87 / 11.68;
3 tokens 17.88 / 18.59 / 17.47-17.51. The 2-dispatch read's gain was all Q8_0; its kernels are slower than the
3-dispatch ones at windows. M4 HC read: Q8_0 85 us vs BF16 141-144 us.
- gemv.metal: gemv_pair<F1, F2> for the packs' gate/up format pairs (tests/pair_parity: 8 pairs x 1/3/8 tokens vs
double, 1.4e-7..2.6e-7) and GV_SET(T_Q2_0R).
- token_chain CHAIN_PAIR=1: interleaved A/B, 3 pairs, Q8_0 HC --fuse --graph: 10.60-10.61 ms without vs 10.71-10.72
with (3 tokens 17.23-17.25 vs 17.31-17.33): two concurrent GEMVs and the SwiGLU win; opt-in only.
kernels/gridbar.metal + tests/gridbar_bench: phases of a streaming read separated by an atomic grid barrier vs by
dispatch boundaries (M5 Max). Without atomic_thread_fence(device, seq_cst) the phases read other threadgroups'
stale writes; with it 0 stale reads. Speed: 4.5 MiB phases 10.3 us (10 threadgroups) vs 9.9 us a dispatch, 30-34 us
with 20-40 threadgroups; 0.5 MiB phases 1.5 vs 2.2 us. Not pursued.
All under __APPLE__ / STRATA_GPU_METAL, so the Linux and Windows code is unchanged:
- direct_file.cpp: no O_DIRECT on macOS; F_NOCACHE after open keeps the reads out of the buffer cache
- memory.cpp: F_RDADVISE is the read-ahead hint where posix_fadvise(WILLNEED) is used
- expert_source.cpp: available memory from host_statistics64 (free + inactive + purgeable); the posix_fadvise of a
released expert layer is skipped (the madvise before it already released the mapping)
- generate.cpp: <sys/sysmacros.h> is not on macOS (major/minor come from <sys/types.h>); the CUDA toolkit mismatch
check is CUDA's
- vmm.cpp: the cuMem* virtual memory stays CUDA's (Metal takes the HIP branch: elastic K/V off)
- prefill.cpp: std::max<size_t> where size_t and uint64_t meet (the same type on Linux, distinct on macOS)
- gdn.metal: gdn_step_t<IB>; gdn_step8 (1024 threads, 16 state rows a thread) passes and is not faster than
gdn_step (16.3 vs 16.4 us commit, n=1): the state write after the compute is what a commit adds.
- token_chain: CHAIN_GU / CHAIN_DN / CHAIN_DN4 select the routed-expert tiles; measured in the chain (Q8_0 HC,
--fuse, --graph): gate/up 32,1,4 stays; the Q2_0 down 8,1,4 over 8,2,4 (10.63 vs 10.67 ms, 3 tokens 17.23 vs
17.50 ms), now the default.
- kernels/qsa2.metal: qsa_topk2 - the same selection (ids exact at contexts 700..33000): the radix digit found by a
parallel suffix scan of the 256 bins, cells emitted from contiguous block ranges after one scan. 23 -> 8.2 us at
8K, 54 -> 14.5 us at 32K; token_chain (Q8_0 HC, --fuse, --graph, interleaved): 8K 11.40 -> 11.00 ms,
32K 11.90 -> 11.27 ms.
- qsa_attn2 (chunks of 32, head groups) and qsa_attn3 (lane per cell for q.k): both pass, neither beats qsa_attn at
the full budget (21-24 us, nearly independent of the cell count: a latency chain); QSA_ATTN selects, default 1.
- Read yibie/strata-mlx's OPTIMIZATION_RECORD/ROADMAP (MIT) before this round: its finding that merging kernels cut
the kernel count 43% for 15% of the time matches ours.
kernels/mma.metal: 8-row tiles dequantized with ref_quant's arithmetic into threadgroup memory, x loaded transposed
from device memory, simdgroup_multiply_accumulate; tests/mma_parity: 12 formats pass vs double, every token of a
window bitwise a one-token call. Speed (M5 Max): 60-190 GB/s for 1, 3 and 8 tokens alike, against the per-lane
GEMV's 400-595 GB/s at 1 token and 70-230 at 8: the dequantization through threadgroup memory (~8 B a weight) is
the limit. Only Q2_0 at 8 tokens is faster (115 vs 126 us). Not used by token_chain.
- gdn.metal: GdnArgs.plane: the step writes y as the raw plane plus the other planes and the per-16 sums, so the
out projection needs no prep dispatch.
- gr2.metal: gr_down3 can form bo as the MoE combine itself (GrArgs3.comb).
- token_chain --fuse: the GDN prep in the step, no QSA score/top-k dispatch while the context fits the 2051 budget.
Q8_0 HC, --graph: 10.71 -> 10.49 ms (1 token), 17.13 -> 16.86 ms (3); BF16 2-dispatch 11.91 -> 11.65 ms.
--fold-comb (the combine in the next HC read) is slower: 10.49 -> 11.22 ms; off.
- kernels/gr2.metal: gr_down3 (grid row blocks x streams: each threadgroup forms its stream's R', norm and xn in
threadgroup memory, writes per-stream partial dots; R' ping-pong, never in place) and gr_up3 (lo from the
partials, xn on the fly); BF16 or Q8_0 weights. tests/gr2_parity: both formats vs double on their own weights,
prepared planes bitwise = xprep; Q8_0 moves the read's output by 1.2e-3 relative.
- ref_quant: quantize_q8_0 (ggml's reference rounding), used for the load-time Q8_0 copies (STRATA_HC_Q8's form).
- token_chain --hc 3|2|q8. M5 Max, Q2_0, --graph: 1 token 12.00 / 11.91 / 10.71 ms; 3 tokens 17.21 / 18.02 /
17.13 ms.
MAC-PLAN Phase 1, compat design (the alternative to renaming the host calls, f-mac-design S1-S4).
metal/include/metal_compat goes first on the Metal build's include path, as include/strata/hip_compat does on HIP:
its cuda_runtime.h gives the ~70 CUDA runtime names of the Phase 1 inventory (types, constants, calls) on Metal,
each forwarding to metal/src/gpu/runtime.cpp (metal-cpp: gpuAddress device pointers in one residency set, streams =
queue + command buffer + shared-event timeline, graphs = recorded dispatch lists), and immintrin.h gives _mm_pause /
_mm_sfence on arm64. The CUDA / HIP / SYCL builds never see metal/.
Shared files: four preprocessor conditions also exclude STRATA_GPU_METAL from CUDA-only blocks (the cuMem*
segmented expert cache, the STRATA_VERIFY_NODES graph listing, the MTP stream priority, cudaInitDevice of the remote
experts); nothing else in src/ changes, so the PC objects are byte-identical.
On the M5 Max: runtime_hello passes with the unchanged src/core/graph.cpp, and graph, expert_cache, weights,
native_head, native_dense, layer, verify, mtp and session compile for Metal as they are (-fsyntax-only); prefill,
expert_source and generate stop only at macOS platform items (std::max of size_t vs uint64_t, posix_fadvise /
_SC_AVPHYS_PAGES, sys/sysmacros.h).
- gdn.metal: a q/k conv channel is read at the start of three threadgroups (value heads h, h + 16, h + 32) while
its owner wrote it back in place at its end: on the 10-core M4 the threadgroups run in waves and the later ones
read the new history (gdn_parity failed there, passed on the M5 Max where all 48 start together). The committed
history now goes to hist_out (the caller swaps); gdn_parity checks determinism too. Passes on both Macs.
- gemv_parity --gguf: one real tensor per format of a GGUF (experts: expert 0) vs double; Q2_0 file: 11 formats
pass, rel L2 8e-8..2.7e-7.
- src/gguf_lite: a minimal GGUF v3 reader; the file is mmapped and wrapped no-copy as one MTLBuffer.
- moe.metal addresses experts by gate/up/down offsets and strides (a pack blob or the GGUF's three tensors).
- kernels/glue.metal: the shared expert's SwiGLU (prepared activations) and the combine with its sigmoid gate.
- tests/token_chain: every layer of the model with its own formats, all kernels in one command buffer.
M5 Max, Q2_0, context 1000: 12.83 ms a token (serial encoder), 12.01 ms with --graph (a concurrent encoder,
barriers only at true dependencies); a 3-token window 18.42 / 17.23 ms. Bound: 7.27 ms.
- mtl_rt: run_graph (concurrent encoder, Dispatch::barrier).
- kernels/qsa.metal: q/k RMSNorm + NEOX RoPE (host table), int8 KV append (fp16 scale per 64), the indexer's raw
tail / pooled block keys / dead key, ReLU-sum block scores with the +1e9 tail, a cell-level radix top-2051
(ties to the lowest cell, ascending), split-K GQA attention over the selection, log-sum-exp merge * sigmoid(gate)
written as prepared activations; tests/qsa_parity stage by stage (ids exact; contexts 700..9001).
- kernels/sample.metal: two-stage argmax, ties to the lowest id; tests/sample_parity.
kernels/gdn.metal: one threadgroup per value head, the head's state in registers across the window's tokens;
commit writes state + conv history back, verify does not; tests/gdn_parity vs double (1.7e-7), verify leaves the
state untouched, the first m outputs of a longer window are bitwise a window of m (the fold).
- kernels/gr.metal: gr_norm (folds the previous write, per-stream RMSNorm), gr_down (down + inject rows, silu/hc
epilogue, K split), gr_up (gated mean over the streams, writes the prepared activations); tests/gr_parity vs double.
- moe.metal: moe_route = native_router.cu's softmax/top-10/renormalisation and the plan in one threadgroup
(tests/route_parity: ids exact incl. ties, weights 2e-7); moe_gu/moe_down switch once per threadgroup into a body
specialised for the group's entry count (the predicated 8-way loop cost 2x); Q2_0 down reads raw activations
(T_Q2_0R) so no prep dispatch is needed.
- mtl_rt: warm() before timings (the GPU starts at idle clocks).
- kernels/units.metal: per-format unit decoders (Q2_0, Q3_K..Q6_K with the mask trick, IQ4 via a byte->half2
codebook table, BF16, generic for the rest) and one combine for all; xprep writes the pre-scaled activations
and per-16 sums once per input vector.
- kernels/gemv.metal: y = W x for 1..8 tokens in one pass over the weights, NTOK a function constant, optional
K split across simdgroups; tests/gemv_parity: parity vs double for 12 formats x 20 tile shapes, every token of
a window bitwise equal to a one-token call, and the bench on the packs' projection shapes.
- moe.metal on the same decoders; mtl_rt compiles with safe math by default (no reassociation; measured free).
- tests/alu_bench: instruction rates (fp32/fp16 FMA, int, convert, threadgroup/device/constant gathers).
The dpct migration had dropped the cudaMemcpyAsync D2H and the stream sync, so the resident RAM mode with adaptive swaps committed
whatever the exchange buffer held as the evicted expert (lost expert, wrong weights on a later CPU read). STRATA_CHECK_SWAPS=1 is a
test aid: it compares each staged buffer with the model file and audits the RAM copy at the end of a run.
metal/ is its own CMake project (macOS only), nothing in src/ or include/ changes. MSL is compiled at run time
(the Command Line Tools have no offline metal compiler) with ggml-common.h prepended for the i-quant grids.
- src/mtl_rt.*: a C++ face over Metal (buffers, pipelines with function constants, serial/concurrent encoders,
indirect dispatch, indirect command buffers, GPU timing).
- src/ref_quant.*: CPU reference dequantizers for Q2_0, Q4_0, Q5_0, Q8_0, Q3_K..Q6_K, IQ2_XXS/XS/S, IQ3_XXS/S,
IQ4_NL/XS, BF16, F16; checked bitwise against gguf-py (tests/ref_dump + tools/check_ref.py: 17/17 equal).
- kernels/quant.metal: the formats as per-lane units (decode once, dot with every token of the window).
- kernels/moe.metal: moe_plan (top-k -> groups by expert, deterministic, writes the indirect grids), moe_gu
(gate+up+SwiGLU, grouped), moe_down, moe_combine; tests/expert_parity: parity vs double and the bench.
- tests/rt_bench: M5 Max streaming read ~585 GB/s, ~0.85 us per dependent dispatch, ~170 us commit+wait.
Linux CUDA had it on by default since 0.1.43, measured on an RTX A4000 (+5.5..8.3% decode) and a Tesla P100 (+4.4%). An RTX 5090 Laptop with a native IQ3_XXS pack and --expert-cache-per-layer decodes 21% slower with it, and the overlap is the default only there. The switch is now =1 turns it on everywhere, and the startup log names the state (with the reason when it cannot run), so a decode-rate drop is not the only way to see it.
Unchanged: the copy path, the gate predicates and the tokens (same tokens with or without it).
Fixes#2010
On 2x/4x R9700 the drafter's chain and the last stage's window shared a card and slowed each other (window 13 -> 27 ms) in most engine starts; the draft layer now loads on the first card (STRATA_MTP_STAGE0=0 keeps the old place). Exact flags, tokens identical: 2 cards code +21%, 3 cards +19%, 4 cards +22% over the serial loop. A split reads prompts in 8192 chunks (STRATA_SPLIT_CHUNK16K=1 keeps 16384): 16K prompt +36..53%, 32K +18..29%, same tokens. STRATA_PCIE_AUTO_SPLIT=1 now also runs in the pipelined loop (opt-in).
gfx1103 was missing from the gfx11 matrix-core lists, so the 780M ran the portable verify-window and prompt kernels. Added to the
compile-time lists and gfx_arch_is_gfx11_wmma(). Radeon 780M, Q2_0, 5 interleaved pairs: decode (code) 11.96 -> 18.25 tok/s,
same token ids; prompts unchanged. hip_prefill_wmma_gemm_parity passes on gfx1103 (3520 cases). Identity gate (defaults off,
mg_norepeat 10/10 against the portable build, exact_pp 4K/20K residual dumps identical).
The serve path counts every routed expert on the device when a save path and a mirror exist; the saved ranking mixes the counts with the start ranking (prior worth 800k routed entries, STRATA_PROFILE_PRIOR_ENTRIES). Exact: placement is fixed under NO_HOST, the file applies at the next start. B70, IQ3_S: code +13..19% vs the xe default profile, prose -18%.
0.1.43 moved the QSA step records and the batched token ids to a pinned buffer (#1712). A 4-hour multi-slot load on 2x RX 7900 XTX (parallel 4, vision plus long
prompts, layer split) still faulted after 31 minutes, with the picture rows, the draft tokens and bounds and the non-mapped routing tables still going up from
caller memory. PinnedRing (include/strata/core/pinned_ring.hpp) is a ring of four pinned slots with one event each: the bytes are copied into the next slot after
the copy that last read it is done, the DMA runs from the slot, and the caller may change its source on return. No pinned RAM (or STRATA_TEST_PAGEABLE=1): the
copy is made from the caller's memory and the stream is waited for. A failed upload is returned and reported.
Moved onto it: the picture's embedding rows (gathered per run into one slot), the draft K/V step's tokens and bounds, the PLE rows' pageable fallback, the
non-mapped branch's routed-id download and slot/src/bounds uploads, the peer device's src and bounds tables, and the drafter's prompt records (mtp.cpp).
Copies from pinned arenas and cudaHostAlloc buffers are unchanged. Same bytes reach the GPU, so answers are identical.
The analysis of which copies are still pageable, and the bounce-slot-per-event idea, are mantovaniluca91's (patch C on #1712). This is our own form of it.
collect_machine listed every nvidia-smi card. It now takes the bench's --gpu, else the config's gpu (or --gpu in its args); with none given it names the one card the server reports instead of all of them.
ensure_loaded logged 'loading anyway' for a before_load command that failed and co-loaded two models onto one GPU. A non-zero exit or a timeout now leaves the model unloaded and answers 503 with Retry-After (before_load_ignore_failure restores the old behaviour); before_load_timeout_s sets the 120 s limit; POST /unload?wait=1 (or wait=<s>) waits for the running turn instead of answering busy.
The engine answered 'ERR BGEN: no such free slot' when its view of a slot differed from the server's; the server turned the ERR into an HTTP 400 and the conversation, drawn to the same slot by its held prefix, got it again on every turn. Now the refused slot is parked (busy, no owner) for SLOT_PARK_S, the request waits for / takes another slot, and after BGEN_REFUSALS_MAX refusals it ends with a retryable 503.
The host mirror and the unmirrored count cover the first stage only; a later stage whose cache lacks experts has no mirror entry and, under NO_HOST, no host answers for them (device lost at the first verify window on 2x Arc Pro B60). The start now refuses with the existing wording, or logs the count when NO_HOST is off.
The mapped gather_batch took 16 serial demand faults per token from a cold table with no progress mark; the first chunk of a prompt only has layer 0 to hide it behind, so the stall watchdog stopped the engine. Rows are now hinted 4096 at a time (PrefetchVirtualMemory / MADV_WILLNEED). Answers are unchanged. The 0.1.42 auto-switch to mmap on Windows is opt-in until it is measured there.
The per-layer host function made the overlap depend on the driver's callback thread being scheduled: with the pool's
workers spinning on every other CPU it still cost 7 ms per window on a 20-core desktop even with the thread on all CPUs
(RTX 3060, Core Ultra 7 265: 46.5 tok/s with the warmed callbacks against 56.5 with the overlap off). The copy stream now
ends each layer's copies with a 4-byte device-to-host copy of the layer number into a pinned int the gate polls; same stream
order, no host thread. STRATA_ADAPT_SIGNAL=host brings the host functions back. Swaps, residency and answers are unchanged.
On Linux the CUDA (and HIP) runtime creates its host-callback thread lazily, at the first cudaLaunchHostFunc, and a new
thread inherits the creator's CPU mask. The decoding host pins itself to one CPU (SessionLoopScratch::init), so a thread
created after that - which the adaptive overlap's per-layer callbacks did - ran on the host's CPU and fought the host's
spin-wait: ~1000 us instead of ~4 us per callback, decode 2-4x slower with STRATA_ADAPT_OVERLAP on (#1964, #1975).
oal's warm-up (one no-op callback + sync before ExpertPool) is now unconditional on Linux; the answers, swaps and
residency are untouched. Windows CPU Sets are per thread and not inherited, so Windows is unchanged.
Rarely a window's logits become NaN and the reply degenerates into token 0
('!') repeated (#606); the poison sits in the conversation's K/V and
recurrent state, so with the prompt cache every later turn stays poisoned
until it is read again from token 0. The q8_1 clamps (2b726ba, f6601bf,
7a0e302) remove the overflows found so far; this guard catches whatever
cause is left, and drops the poisoned state instead of serving it again.
On by default, as a fix: STRATA_NAN_GUARD=0 turns it off. It only reads
flags, so an answer without a NaN is the one the engine gives with the
guard off (compared byte for byte on a GPU, below).
Engine:
- every verify-window graph checks its head logits beside the greedy pick
(logits_nonfinite_rows: 16 blocks a row, one flag word each, into mapped
memory); Verifier::note_nonfinite reads them after the sync the window
already does (run, run_slot_rows, batch_poll, pl_finish). No extra sync.
With STRATA_NAN_GUARD=0 no kernel is recorded.
- --serve: a flagged window emits none of its tokens, one stderr line names
the position and row, and the request ends with `DONE ... nonfinite` (a
batch slot: `BDONE <slot> <n> nonfinite`; an admission: BADM <slot> 0).
The next request first drops the live session, every checkpoint, every
parked conversation (ConversationCache::clear) and every idle slot, and
reads its prompt from token 0; a slot still running then finishes its
reply and is not kept as a cache after it. A request that ends on it does
not keep its conversation live (#1620's cancelled-read path is skipped
for it).
Covered: the serial decode loop (one GPU or a layer split), the
--pipeline-windows decode loop of a layer split, the prompt's short tail
(read_windows, and read_windows_pl with --pipeline-windows), and both
batch-slot loops (run_slot_rows, and the --batch-groups pipeline's
batch_poll). Not covered: the one-shot command line (no --serve) and the
SYCL port in sycl/, which has its own verify.cpp and generate.cpp.
- STRATA_NAN_INJECT=<n>[,<n>...] (debug): the n-th window the guard checks
reports row 0 as non-finite.
Server: EngineNonFinite on that finish. A request that has sent nothing yet
is sent again once (its usage is the retry's: the failed pass's DONE is
dropped); a reply that had started ends as "length"; a second fire on the
retry ends with an error (503 before an answer, an error event in a
stream). With replicas the router forgets what that replica held.
repeat_stop_tokens stays as the backstop.
Checks: serve tests (python -m unittest discover -s serve -t . -p
"test_*.py": 697 tests, OK, 11 skipped; NonFiniteLogits,
test_nonfinite_in_batch_mode and test_nonfinite_forgets_the_replica are
new). ctest with CUDA_VISIBLE_DEVICES=-1: 24 pass (conversation_cache_test
with clear() among them), 3 skip for want of a GPU, and the 43 GPU parity
tests were not run on a GPU. Built for CUDA 13.0 on Linux (GCC, sm_75 +
sm_86); generate.cpp, verify.cpp and verify_kernels.cu compile with
MSVC 2022 + CUDA 13.0, and conversation_cache_test and conv_cache_test
pass there; built for HIP (ROCm 7.0, gfx1100, compile only) and SYCL
(oneAPI 2026.1.1; it takes only the new declaration in verify_kernels.hpp
and the new inline ConversationCache::clear(), and calls neither).
On a GPU (RTX 3080, RVN IQ3_S, strata.service's one-GPU config):
- Answers, guard on vs STRATA_NAN_GUARD=0, at fixed expert sets
(--pcie-frac 0 --adapt-every 0 --adapt-swaps 0 --mtp-max-t 1
--suffix-draft 0, STRATA_IQ_MT_MIN=1 STRATA_PREFILL_CPU_SHARE=0), a chat
replay of 2 x 4 turns on a 16K-token system prompt: byte-identical in 3
pairs (one with STRATA_ROUTE_TAIL_SKIP=0, one with STRATA_SH_STREAM=0),
and two guard-on runs 8 of 8. One more guard-off run differed from every
other run from its first answer, and a second guard-off run matched the
guard-on runs: a difference between runs of this config, not the guard.
- Cost, Nsight Systems: logits_nonfinite_kernel 424 launches, 4.77 us on
average (7.78 us at most), 2.0 ms against 21.6 s of decode: 0.009%.
- STRATA_NAN_INJECT=1,60: the first window of a new chat flagged, the
server sent the request again from token 0 and the client saw one normal
answer; the 60th, mid-reply in the next turn, ended that reply as
"length" after 58 tokens; the turn after it reused 0 cached tokens and
answered normally. The engine logged exactly the two injected windows.
route_resident_k is one warp per token and the form it replaced walked all 512 experts on a single
thread. Nothing in the tree checked that the two land on the same expert and leave the same ids,
weights and counters, and the option's whole justification is that it moves misses across PCIe and not
the model.
src/kernels/route_resident_parity.cu: 240 random cases over 1 to 8 tokens, a 40-84% resident table,
random rank windows, margins from -0.5 to 1.5, and half the cases rounding the logits onto a coarse
grid so the tie rule (the smallest expert index wins) is exercised. Half the cases get the router's
own top-10 and half get ids with repeats, which is what the already-picked test is for. The reference
is the spec written out plainly - a serial walk with the same comparisons - and not a copy of the
kernel this branch replaced; it runs on the device so that expf, and so the softmax bits, are the same
function on both sides.
ids and the four counters have to match bit for bit. The weights are a softmax ratio and this
reference is a second compilation of it, so they are compared within 8 ulp and the worst seen is
reported; the kernel itself is bit-identical to the serial form it replaced, which was measured with
both kernels side by side in one translation unit. A case that swaps nothing still has to leave the
weights untouched, which the sentinel they start from checks, and the test fails if no case swapped at
all.
route_resident_parity: 240 cases, 205 with swaps (2004 swaps), 0 mismatches, weights within 6 ulp
Registered in ctest next to the other router parity tests, TIMEOUT 120.
The swap kernel is the only work STRATA_ROUTE_RESIDENT adds to a window, and it was written as one
thread per token: with a verify window of 1 to 8 rows that is up to 8 threads of a block, each doing,
per rank in [lo, hi], a 512-expert scan with a ten-way "already picked" test inside it. On this
machine that is about 4 x 512 x 12 scalar operations running on six threads, on the critical stream,
once per layer, and it shows up as its own cost in the window.
STRATA_VERIFY_PROFILE, three legs in one session, same prompts, same expert cache (8083 slots, one
learned profile), medians of the per-window stage table:
configuration hc-read1+router waitB ms/window pcie_e
option off 2.25 9.30 21.85 1.87
option on, one thread per token 5.00 5.01 21.97 1.00
option on, this commit 2.58 4.87 19.75 0.95
The option redirects about a miss per layer-window away from PCIe and saves 4.3 ms of waiting for it;
the serial swap spends 2.75 ms back, which is why the option measured as a wash. In the warp form the
swap costs 0.33 ms, so the waiting it saved stays saved.
End to end, twelve matched pairs of engine restarts, nine decode samples a leg, three greedy 150-token
prompts, the learned expert profile copied back before every leg so both arms start from the same
8083-slot cache. The first six pairs ran the unpatched build first, the last six the patched build
first, so a within-pair order effect would show up as the two halves disagreeing:
pair order decode unpatched patched delta ms/window unpatched patched delta
1 A->B 104.1 115.7 +11.1% 18.64 17.01 -8.7%
2 A->B 103.5 107.4 +3.8% 18.47 18.12 -1.9%
3 A->B 111.4 107.7 -3.3% 17.71 18.26 +3.1%
4 A->B 101.4 113.0 +11.4% 19.11 17.03 -10.9%
5 A->B 105.8 107.6 +1.7% 19.05 18.05 -5.2%
6 A->B 104.8 105.6 +0.8% 18.60 17.83 -4.1%
7 B->A 106.6 113.4 +6.4% 18.70 17.16 -8.2%
8 B->A 103.1 110.2 +6.9% 18.14 17.29 -4.7%
9 B->A 106.9 111.5 +4.3% 17.91 17.26 -3.6%
10 B->A 104.3 114.7 +10.0% 17.81 17.11 -3.9%
11 B->A 108.3 115.9 +7.0% 18.47 16.61 -10.1%
12 B->A 104.5 113.6 +8.7% 18.43 17.76 -3.6%
twelve legs each: unpatched decode median 104.7 (101.4-111.4) ms/window median 18.47 (17.71-19.11)
this commit decode median 112.2 (105.6-115.9) ms/window median 17.27 (16.61-18.26)
decode median +6.6% mean +5.7% faster in 11 of 12 pairs (sign test p = 0.006)
ms/window median -4.4% mean -5.2% shorter in 11 of 12 pairs (sign test p = 0.006)
first six pairs: +2.7% decode, -4.7% window last six: +7.0% decode, -4.3% window
pcie_e, the engine's own count of experts per layer-window that cross PCIe, is the same in both arms
(leg medians 0.63 against 0.64), so the same experts are routed and the difference is the kernel.
The swap's cost, and so its removal, scales with the tail experts the option actually redirects. Four
more pairs with a 4000-slot cache instead of the automatic 8083 (pcie_e 1.83 against 1.78, i.e. about
three times the traffic of the series above), two pairs each way:
pair order decode unpatched patched delta ms/window unpatched patched delta
1 X->Y 83.8 92.0 +9.8% 21.94 20.60 -6.1%
2 Y->X 84.6 92.5 +9.3% 23.97 20.53 -14.4%
3 X->Y 82.2 85.5 +4.0% 23.69 22.65 -4.4%
4 Y->X 81.0 87.7 +8.3% 24.54 22.21 -9.5%
decode median +8.8%, ms/window median -7.8%, 4 of 4 pairs, both orders.
What this changes is the shape, not the choice. One warp per token: lane j reads experts j, j+32, ...
(coalesced instead of a serial walk), the warp reduces to the best logit in a shuffle tree, and the
ranks are still visited in order, because a swap changes the set already picked. The choice is the
same bits: a strictly greater logit wins, the smallest expert index breaks ties, the already-picked
test runs against the ids as they are at that rank, a token with no swap keeps the router's weights
untouched, and the softmax is the same expression.
Parity. 400 random cases - half of them built with many equal logits, random lo/hi, margins from
-0.5 to 1.5, 1 to 8 tokens, about 60% of the experts resident, routed ids with repeats - compare the
two implementations bit for bit on the ids, on the weights (their raw bits) and on the four counters.
The harness carries both kernels side by side; 0 mismatches.
For context, the option's price is the margin's and not the kernel's, and this commit does not touch
it. Teacher-forced log-probabilities over 7677 positions of one 7.6k-token text (technical prose,
source code and Chinese), top-64, against the same text with the option off:
margin argmax same KL (top-64) teacher-forced loss perplexity
off - 0.000 3.2033 -
0.25 84.9% 0.235 3.2238 +0.6%
0.50 83.0% 0.293 3.4360 +7.3%
1.00 63.5% 1.001 4.7751 +49%
Two runs with the option off are bit-identical, so that spread is the option and not the machine.
0.25 keeps nearly all of the waitB saving for a fraction of the distortion; the default 1.0 is not
worth its price.
Tests: the parity harness is a standalone .cu (both kernels, the random cases, the bit comparison).
Happy to wire it into the test suite next to the other kernel tests if you want it in the tree.
The QSA step records of a chunk were uploaded with an unchecked cudaMemcpyAsync from a
std::vector. On gfx1100 the first text prompt after a picture one read garbage records
(5 of 5 faults on a two-request repro, 0 of 5 with a pinned, event-guarded source). The
token id upload has the same shape and now takes the same path; both check the result.
Reported and traced by mantovaniluca91, #1712.
IlQ3K runs q3_q8_dot_impl's arithmetic with the scales and signed values done once per chunk; bitwise native_mmvq's output
for T 1-4 and rows 1/2/4 (mmvq_il_parity). Q2_0 pack: the Q3_K multi-column kernels were 14% of GPU kernel time; the new kernel
takes 2560x10240 at 4 columns from 120 to 70 us on an RTX A4000. The shared table (sm_80+/sm_120) is unchanged.
Bitwise equal to the CUDA layout on all 48 layers at 1-4 tokens a group (native_expert_bench); a 28-group VRAM call 17.6 -> 15.5 ms over the layers at T=1, 24.1 -> 22.6 at T=2, even at T=3, +2.6% at T=4.
Bitwise equal to native_mmvq (mmvq_il_parity, T 2-4, rows 1/2/4); table read off mmvq_il_parity --bench --emit-table on a Tesla P100 (three identical runs).
The pipelined decode loop asks both stages' windows for service millions of times a second while they wait on their
doorbells, and every call switched the device (OnDevice: cudaGetDevice and two cudaSetDevice) before finding nothing
to do. On a 0.1.41 split (RTX 3060 + RTX 5070 Ti, Windows) the host stalled inside one of those cudaSetDevice calls
after 919 million loop passes, with both stages' windows waiting on it (stall dump: the serving thread in cudart from
Verifier::service, every other engine thread idle). Now a call with nothing rung, no WDDM flush due (2 ms) and no
timeout due returns 0 first - the same answer the loop below gives - so the driver is called only to serve a layer,
to flush, or to time out.
The pages had no <link rel="icon"> and serve/web had no icon file, so browsers asked for
/favicon.ico, got a 404 and showed the default blank icon in the tab.
serve/web/favicon.svg is the layer stack the app already uses (serve/web/sprite.svg, i-layers),
redrawn on a 32-unit viewBox so it stays legible at 16 px, in the existing accent token #10b981.
Checked: curl /web/favicon.svg -> 200 image/svg+xml, both pages come back with the icon link,
and the icon rendered at 16/32/48/64 px in Chrome.
_Nvml.read() reports pcie_gen, pcie_gen_max, pcie_width and the two
throughput counters; _Amd.read() (the amdgpu sysfs reader from #301)
reports none of them, so on Linux the Monitor's PCIe tile shows only
dashes while the NVIDIA tile fills in.
This adds the link the card actually gets: the narrowest and slowest hop
between the root port and the card, from each hop's max_link_speed and
max_link_width. A card that negotiates Gen4 on its own hop but sits
behind a Gen3 root port really runs at Gen3, and that is the number a
bandwidth budget needs. pcie_own_gen keeps the card's own hop aside so
the gap is visible, and pcie_path reports how many hops could be read --
the bottleneck is only claimed when all of them could, because a kernel
or a container that hides part of /sys/devices would otherwise report the
card's own hop as the whole path.
Measured on one MI50 32 GB in a Gen3 x16 slot (lspci: the root port at
00:02.0 runs at 8 GT/s x16, the bridge above the card is downgraded, and
the card's own hop negotiates 16 GT/s), /metrics reports pcie_gen 3,
pcie_gen_max 3, pcie_own_gen 4, pcie_width 16 and pcie_path 4/4.
Throughput is deliberately left out: amdgpu's pcie_bw is three counters
whose documented use is a total, not a direction, while _Nvml's
pcie_rx_mb is receive-only, so filling that name from pcie_bw would
label a combined figure as one direction. Happy to follow up separately
if you want the tile to show a rate.
Hardware the numbers come from: one Radeon Instinct MI50 32 GB (gfx906,
PCI 1002:66a1, VBIOS 113-D1631711-100) on a Xeon E5-2696 v4 (Broadwell,
no AVX-512), in a PCIe 3.0 x16 slot, 128 GB RAM, Ubuntu, kernel
7.0.0-34-generic, ROCm 7.14 in a container.
Prefill::arm_cpu_share() asks "is this AMD?" with
#if !defined(STRATA_USE_HIP), but the gfx906 compat build (PR #638)
compiles as HIP *without* STRATA_USE_HIP -- see src/core/expert_cache.cpp
:5-6, which guards the same question with
#if defined(STRATA_USE_HIP) || defined(STRATA_HIP_GFX906).
On gfx906 the CUDA branch is therefore live: the CPU share is armed by
default for chunks below 1,024 tokens and prints its banner, against this
file's own note at :161 ("the layer split, batch slots and HIP stay off").
Seven other sites already carry the || defined(STRATA_HIP_GFX906) form.
Measured on one MI50 32 GB (gfx906), CLI generate, a 4-token prompt, one
binary with one set of weights and one config, only the environment
differing, power saving off:
default experts streamed 309 (295 by DMA, host 174.0 ms), resident 607,
on the CPU 197 (share 0.90) prefill 928.3 ms
=0 experts streamed 506 (492 by DMA, host 177.3 ms), resident 606,
on the CPU 0 (share 0.00) prefill 929.5 ms
Output token ids are identical (271 248068 198 760) and the 0.13%
difference is inside this machine's run-to-run spread (1.2-1.7%), so on
this card the share engages and buys nothing. #1595 and #1729 report the
default costing on CUDA; this is the mechanism that turns it on for AMD.
CUDA and SYCL do not define STRATA_HIP_GFX906, so the added check is a
no-op there and nothing on those backends changes.
Hardware the numbers come from: Xeon E5-2696 v4 (Broadwell, 22C/44T, no
AVX-512), one Radeon Instinct MI50 32 GB (gfx906, PCI 1002:66a1, VBIOS
113-D1631711-100, 34,342,961,152 B of VRAM), PCIe 3.0 x16 effective (the
root port runs 8 GT/s x16 and the bridge above the card is downgraded;
the card's own hop negotiates 16 GT/s, about 15.75 GB/s end to end),
128 GB RAM, Ubuntu, kernel 7.0.0-34-generic, ROCm 7.14 in a container,
built with -DSTRATA_HIP_GFX906=ON and -DCMAKE_HIP_ARCHITECTURES=gfx906.
spark(id, values, max) sets the axis top to max(|max|, the series' own
maximum, 1e-9). gpu_util, temperature, cpu, vram and power pass an
explicit maximum, but the pcie, tok/s, prefill and disk cards do not, so
those axes zoom onto whatever the series happens to hold.
An idle amdgpu pcie counter reads 0.01-0.02 MB/s (50 packets x 256 B per
second), and that becomes a full-amplitude square wave in the Monitor
while the tile's own text says 0.0 MB/s. Measured in the live DOM: the
pcie line's y values are only {4.0, 30.0}, i.e. the baseline and the top.
A floor of 1 only ever lowers the axis - the series maximum still wins -
so a real burst keeps its scale, and the cards that pass a maximum are
untouched.
IQ4XS.test_amd_is_not_asked mocks get_prebuilt_hip, the engine setup.main takes on Windows. On Linux setup.main
compiles the HIP engine instead (build_engine_hip), which the test did not mock. Without a usable system ROCm 7, as
on the tested machine, rocm_root then fetched AMD's ROCm index
(https://rocm.nightlies.amd.com/v2/gfx120X-all/rocm/), wrote Path(sys.prefix)/.strata-rocm.json and tried to run
<python dir>/rocm-sdk. Where sys.prefix is not writable (the system Python) that is a PermissionError. Where it is
(a venv), the test passed while setup.main stopped at the ROCm step with exit code 1, and a stamp with the same
version and index can make a later setup in that venv skip the ROCm install.
The test now mocks build_engine_hip the same way, so on Linux setup.main runs to the end (exit code 0) with no
network request, no subprocess and no write outside the test's temp folder. setup.py is unchanged; the Windows path
never reaches build_engine_hip, so it runs as before.
Tests: python3 -m unittest tools.test_setup_unsloth (WSL2): with the system Python 3.12.3 FAILED (errors=1) before,
38 tests OK after; in venvs on 3.10.22, 3.12.3, 3.13.7 and 3.14.8 OK before and after, and no stamp after. With the
system Python the other 27 tools/test_setup_*.py files give the same results as before.
The `[16.0,18.0)` range is CUDA 13's requirement, but it was applied to every caller: with the
only Visual Studio installed being 2026 (version 18), vswhere finds nothing and find_vcvars()
returns None. That makes the CPU image encoder unbuildable on such a machine - build_vision_cpu()
cannot run - so --vision cpu stays unreachable there even with the gate in hip_vision() open.
#985 relaxed the range for CUDA 13.3+ by adding `cuda_v`, which is right for CUDA callers. A
caller with no CUDA now passes nothing and gets no range, which is what
tools/hip/build_windows.bat already does when it calls vswhere directly.
- `cuda_v is None` -> no -version argument at all. CUDA callers are unchanged: cuda_v=(13, 0)
still refuses VS 18, cuda_v=(13, 3) still takes it.
- STRATA_VCVARS=<vcvars64.bat> picks an install by hand, for a vswhere that cannot be used.
Verified on the RX 6800M / gfx1031 machine of #881 (Windows 11, Chinese locale, the only C++ tools
are Visual Studio 2026 / 18.4.11605):
find_vcvars() -> <VS 18>\VC\Auxiliary\Build\vcvars64.bat
find_vcvars(cuda_v=(13, 0)) -> None
find_vcvars(cuda_v=(13, 3)) -> <VS 18>\VC\Auxiliary\Build\vcvars64.bat
With a draft-vocab head (rt/draft_vocab.bin) the slot drafters of --batch-mtp share the main drafter's head but kept
dhead_type_ at -1, so the first draft_first() of a batch admission threw 'mtp: unsupported native MMVQ GGML type'
and the engine exited (#1121). The CUDA source got this line in #1110; the SYCL port did not.
A periodic loop emitted as digits (a 24k-digit string cycling an 11-digit
pattern) is a single word to the word count and never one token twice
(the tokenizer has no multi-digit tokens), so both #606 and #728 passed
it and the run burned the whole reasoning budget. The reasoning's last
2,048 chars are now checked for a period of at most 64 chars, and a
periodic tail reads as full coverage. The opt-in path only: the default
path and the word count are unchanged.
With the Intel port the configured exe is sycl/serve/strata-sycl.sh, a shell script that runs the repo's
STRATA_SYCL_BIN (default build-sycl-aot/strata) in a container. effort_end_args looked for --tail-role-token in the
script, never found it, and fell back to the top-of-prompt effort with "needs engine 0.1.39 or newer", even on a
0.1.41 engine that has the option. engine_binary() resolves the launcher to the binary it runs; any other exe is read
as before. tools/test_effort_end_args.py covers both paths.
Follow query-swizzle #1661 with the existing reduce-scatter pairing for
12 heads. Keep default paths and softmax/value/merge unchanged. Public
component parity and timing passed; model evidence uses frozen stack.
Record archived-control output parity, verified candidate process, tools, CPU vision and cache restoration. No fresh 200K speed comparison or production promotion is claimed.
Preserve FP operation order and invert a bijective shared-query layout at its vector loads. Limit the path to the current gfx906 wave64 device, INT8 pools and warp scoring. Default remains off.
Add deterministic GPU parity/timing and host-guard probes, plus scoped component and model receipts. The clean upstream build passed both-card byte checks; model gains are recorded separately on the frozen experimental stack.
Pinned memory reads the link 1.7x faster (A750 10.3 -> 17.2 GB/s, B70 13.6 -> 25.6), which moves the derived default pcie_frac
(A750 0.28 -> 0.47) and cost 10% of decode there (median 12.07 -> 10.67 tok/s, 4 of 5 interleaved pairs slower). The B70 is unchanged
(0.37 -> 0.55, 31.4 vs 31.4). The fault the PR fixes is a layer split's.
Co-authored-by: maxious <[email protected]>
server_intel.py replaces StrataEngine._pump to keep LP lines beside their token, but its copy predates --batch: the
slots' BT / BDONE lines went to the main queue, so with "parallel" every request stopped after one token. It also
missed the fatal-line handling and last_err. Both are now as in serve/server.py's _pump.
The host mirror holds the experts missing from VRAM at the start. An adaptive swap moves an expert that was resident
then out of VRAM, so it is in neither; the device plan then waits for the host to serve it, and with
STRATA_VERIFY_NO_HOST (the xe default in strata-sycl.sh) nobody does. Seen after a --batch window, whose host path
counts the misses the tier swaps on: the next single request on one card gave token 0 ("!!!!") at 0.8-1.3 tok/s.
The start already refuses experts in neither under NO_HOST; this keeps that true after the start.
The prompt path dequantizes each dense Q8_0 projection (GDN qkv/gate/out,
QSA q/k/v/out, shared expert, hc) into the GEMM scratch before cuBLAS.
dequant_kernel runs one thread per 32-value block, so a warp's 2-byte
stores land 64 bytes apart: about 70 GB/s of writes on a V100.
dequant_q8_0_kernel gives each block to four threads, 8 values each,
written with 16-byte stores: a warp reads 8 consecutive blocks and writes
512 consecutive bytes. The arithmetic per value is unchanged (d * q), so
the output is the same bits. It is used for Q8_0 when the output rows are
contiguous (ld == cols) and the pointers are aligned (blocks to 2 bytes,
output to 16); everything else takes dequant_kernel as before.
V100-SXM2-16GB, every 2-D Q8_0 tensor of Qwen3.8-Flash-Next UD-Q4_K_XL
(498 tensors), each call timed on its own:
attn_qkv 10240 x 2560, FP16 0.740 -> 0.106 ms (70 -> 495 GB/s written)
attn_q 12288 x 2560, FP16 0.887 -> 0.126 ms
ssm_out 2560 x 6144, FP16 0.451 -> 0.065 ms
all 498 tensors, FP16 138.5 -> 20.4 ms
all 498 tensors, BF16 140.8 -> 20.5 ms
all 498 tensors, FP32 190.9 -> 35.4 ms
An FNV-1a hash over all outputs is equal before and after for FP32, BF16
and FP16. Whole prompts on 4x V100 (where this was first used), 4,096-token
chunks: 18K 4,537 -> 4,708, 65K 6,540 -> 6,766, 123K 6,783 -> 6,985 tok/s.
The whole i-quant family (IQ3_XXS, IQ3_S, IQ2_S, IQ4_XS gate/up, IQ4_NL down) and the
fork's Q2_0 down have x86 dots only under __AVX2__ in ggml-cpu; on an AVX1 floor
(STRATA_ISA_FLOOR=avx, issues #595/#623/#1375: Xeon E5 v1-v2, and AVX-512-only packs)
the CPU-computed experts fall to the scalar _generic dots. iq_avx1.cpp is the older
CPUs' copy of iq_avx2.cpp in the kq_avx1.cpp pattern: a separate TU compiled
-msse4.2 -mavx -mno-avx2 -mno-fma -mno-f16c, gated at runtime behind cpu_avx1_ok() and
only when AVX2 is absent, so no mainstream path changes. 128-bit integer lanes
(pshufb/sign/maddubs are SSSE3), 256-bit float accumulation, software half-to-float,
no gathers (the Fmt32 non-gather path), one IEEE mul per value like the AVX-2 copy.
Plus q8k_quant_avx1, the byte-identical Q8_K activation quantizer in SSE4.2/AVX1.
iq_avx1_parity: all six formats, nt=1..8, bit-exact across widths (#152) and within
1e-5 aggregate of ggml's generic dots; q8k_quant_parity now checks the AVX1 copy too.
gu rows take the AVX-2 mt_min policy (STRATA_IQ_MT_MIN), down rows always; kill switch
STRATA_NO_IQ128=1. E5-2470 v2 (Ivy Bridge): 1.2-3.1x per expert at nt>=2, 2.0x on the
Q8_K quantizer; serve prefill +36-41%, warm decode 22-25 -> 46 tok/s.
Known pre-existing gap found while testing: pool_tasks_test SIGILLs on non-AVX-512
machines even on a clean tree (it predates the ISA floor and is not gated).
q2_rows_any and act_quant_any had two rungs, AVX-512 and AVX2, and the else branch
checked nothing: an AVX-only CPU was handed AVX2 code and died with SIGILL (found on
an E5-2470 v2; pool_tasks_test and q2_bitplane_parity crash on a clean tree, and any
future caller inherits the trap). The third rung: q2_avx1.cpp in the iq_avx1.cpp
containment pattern (-msse4.2 -mavx -mno-avx2 -mno-fma -mno-f16c, a #error guard,
called only behind cpu_avx1_ok() && !cpu_avx2_ok()). 128-bit integer lanes (maddubs,
madd), 256-bit float accumulation, software fp16 scales; the legacy kernel keeps the
AVX-2 kernel's lane pairing and reduction tree (its FMA becomes a mul+add, last bits),
and the bit-plane kernel and its image builder port as-is (pure SSE). The AVX-2 and
AVX-512 paths are untouched. q2_bitplane_parity now runs on the AVX-2 pair where the
CPU has AVX2 and on the AVX1 pair below it, and skips cleanly without AVX.
E5-2470 v2: pool_tasks_test 168 bitwise comparisons pass, q2_bitplane_parity PASS -
both SIGILL'd before.
A helper GPU's expert cache already holds a copy of the blobs the prompt path
streams, and the helpers compute nothing while a prompt is read on a serial
request path. Add RemotePrefillSource, an ExpertSource that wraps the file
source and asks the helpers first: a blob a helper holds is copied
cache-to-host on that helper's own stream, misses are merged into one call to
the fallback's existing batched reads, and every other decision is forwarded
unchanged.
Opt-in through STRATA_PREFILL_HELPER_SOURCE; it applies only where the serve
path is serial (one GPU, no layer split, no batch slots) and the prompt path
streams every expert the main card does not hold. Same-day A/B on an
11-request fixture: 91.5 s -> 65.7 s of prefill (-28.3%) and 92.1 GB (64.8%)
fewer bytes read from the files.
The share is picked from wall-clock timings, so the same short-prompt
request can split the experts differently each run and the answer moves
in its last bits - a resumed conversation's re-read need not match its
first (31 of 1,536 teacher-forced argmax tokens differed in the
reporter's test). A fixed share repeats; 0 gives a run without it
byte-for-byte.
Five simulated agent sessions over the repository's own files (system prompt, four tools, real files as tool
results, at most 2000 lines per read): 23 turns from ~2.5K to ~100K tokens, consecutive turns sharing their prefix.
Arms as ab_engine.py takes them; rounds mirror the arm order on fresh servers; prints the median of the per-turn
decode ratios to the first arm and the turns it won. Nothing personal and nothing downloaded.
A missed expert that every token of the verify window routes at rank R or lower in weight contributes zero
(no renormalisation): less PCIe traffic and CPU expert work. Counted in the request log. Opt-in, changes
the output.
The top-10 is picked again with a logit bonus lambda for the experts resident in VRAM; the weights stay the
router's probabilities renormalised over the 10 picked. After 'Mixture of Cache-Conditional Experts'
(arXiv 2412.00099), without training. Complements STRATA_ROUTE_RESIDENT (a tail swap within a margin):
every rank can move, by an amount the logits decide.
With --resident-experts (setup's low-RAM mode when the experts the GPU does not hold fit the RAM) the start copies
the profile fill of the GPU cache and the RAM copy (pin_cache_complement) through the mapped view of experts.bin.
Every page it touches sits in the process working set beside the page-locked copy until #467's trim after the fill
and the VirtualUnlock after the copy: on a PC short of RAM that pages other memory out, and the copies run at the
page-fault rate (~0.85 GB/s here).
FileExpertSource::begin_startup_unbuffered closes the view, the section and the buffered handle, opens the
unbuffered handles (open_direct) and reads the first blob both ways; if the open fails or the two differ, the view is
mapped again and the start copies through it as before. The fill (prefetch_pairs) and the copy (pin_cache_complement)
then take their existing unbuffered paths (read_direct: requests merged up to 32 MiB). end_startup_unbuffered maps
the view again (map_view, the size checked) and closes the unbuffered handles and the stage buffers, so the file tier
is chosen as before. Same bytes in the same places.
Windows and experts.bin only: the call is under _WIN32, and the GGUF in place, an unbuffered file tier and runs
without a RAM copy are unchanged. STRATA_STARTUP_UNBUFFERED=0 gives the mapped copy back; STRATA_VERIFY_COMPLEMENT=1
prints checksums of the filled cache slots and of the RAM copy. pin_cache_complement no longer VirtualUnlocks the
range of a closed view.
IQ2_XS (33 GiB experts.bin), RTX 5090, Samsung 9100 PRO, --resident-experts --expert-cache auto (~15,200 slots, a
16.4 GiB RAM copy), interleaved against fb58e0db:
- 21 GiB available (a RAM ballast; #467's 32 GB PC had 20.7 GiB), 6 pairs: ready 42.5 -> 9.6 s (chat), 40.7 -> 10.2 s
(32K); the fill 842 -> 6,914 MB/s, the copy 13.7 -> 3.3 s; lowest available 0.25 -> 4.5 GiB; pages written out
~60,000 -> 579-3,340; 3 of 12 base starts held available RAM under 1 GiB for 2 s, none of 12 here.
- 128 GB: peak working set 32.6 -> 18.3 GiB; a restart with the file cached 13.6 -> 8.3 s; the 32K prompt and decode
unchanged.
- Same bits: p2k 200 tokens and first-token logits, 32K 64 / 256 tokens, as fb58e0db; STRATA_VERIFY_COMPLEMENT=1
checksums identical to STRATA_STARTUP_UNBUFFERED=0 (12,595 and 15,438 slots).
Based on RealScaniX's sorted-by-name change; the first request of a set stays byte for byte as the client sent it.
Co-authored-by: RealScaniX <[email protected]>
RulesAgreeWithTheWrittenConfig.test_the_streaming_rule_is_the_flag_setup_writes runs setup.main() through
test_setup_golden's install(), which mocks is_wsl() to False, and then asked setup.kv_streaming_wanted() outside
that mock. Inside WSL the real is_wsl() is True, so the rule answered "no streaming" while the config setup wrote
for the mocked PC has --kv-resident, and the test failed at its first 131072 case (Q2_0, int8, auto). With is_wsl()
the same on both sides, setup and the rule agree in all 54 cases, inside WSL and out.
The test method now runs under the same is_wsl mock as install(), the test-only fix the unsloth tests got for #974
(d1ec32b). setup.py is unchanged; where is_wsl() is already False (Windows, native Linux) the test runs as before.
Tests: python3 tools/test_setup_risk.py (WSL2, Python 3.12.3: was FAILED (failures=1), now 40 tests OK); the other
27 tools/test_setup_*.py files give the same results as before.
A request with "temperature": "0.7", "top_k": 40.0 or "seed": true was
answered 200 and sampled greedy: the value replaced the configured
default in run()'s merge, then sampling_keys dropped it as not a number.
check_request_sampling refuses it by name before anything is sent, as
POST /settings and the run config already do. Only types are checked;
ranges, null fields and in-type values behave as before.
A chat "file" or "input_audio" part, and a messages "document" block
(also inside a tool_result), were dropped without a word: the request got
a 200 and the model answered about a file it never saw. _check_parts
refuses them while the request is converted, naming the type and where
it sits, with /v1/responses' wording. Text, images, thinking, tool_use
and tool_result read as before; redacted_thinking is still dropped; a
chat "refusal" part reads as its text, as on /v1/responses.
The row-count sort of a fully resident layer's experts skipped native-layout packs, though slot/src/off are
built the same way for both layouts. 4x GP100 (sm_60): 1.2K prompt 7.27 -> 5.32 s, 35K 63.3 -> 54.1 s, the
same bits. The switch stays opt-in.
kvg_trim refilled the slots the K/V gave back and then uploaded host_res with a plain cudaMemcpy from pageable memory.
That returns once the table is staged, before its DMA lands, and nothing after it waited for stream 0: the next
request's session_zero syncs main_stream only, and a prompt part of --short-read tokens or fewer goes straight to
ver.run, whose graph reads hits_.d_res on the verifier's cudaStreamNonBlocking stream. res_put (#550) is the upload
every other residency change uses: the same copy, then cudaStreamSynchronize(nullptr).
kvg_ensure's upload needs nothing: qsa_kv_elastic_grow ends in cudaDeviceSynchronize before anything reads the table.
Only --kv-grow (opt-in) runs kvg_trim, so the default path is unchanged.
With stream_tools an array or object argument is held until it is whole,
so a large one (a measured 66 KB edit_file call) left the stream silent
for minutes while tokens were generated - a client's idle timeout then
ended a healthy request (Forge's is 120 s).
The token loop now tracks when anything last went out and sends the
engine's own keep-alive ping after KEEPALIVE_S (10 s) of silence - the
same ping the quiet-engine heartbeat sends, so clients already ignore it.
The reporter's reproduction measured a 27.8 s longest gap on a 7 KB
array (minutes on the 66 KB one); with this change it is 10.0 s and the
call arrives intact.
Change and live measurements from the issue; adds the ping-cadence unit
test (a patched clock over a held-back <tool_call>).
The default router, publish and copy_or_zero kernels are the f-142 ones. The probes set the variable (and
STRATA_DOORBELL_CHECK=1 for publish_res, which copies x only on a miss otherwise).
Port the fork's fast router with a bracketed reciprocal and serial fallback. Add bitwise comparisons against the existing router, including ties, NaN tails and the forced serial path. On the A770's 256-expert/top-10 shape, measured host submission plus execution fell from 395 to 119 us per call.
Pair activation payload stores and mapped result reads into 64-bit uncached transfers, preserving the checksum and unaligned/odd-tail fallback. Register router and payload probes with CTest. The two probes, elementwise_parity and router_top10_parity passed on the A770.
Add the fork's opt-in host-clock stage profiler for further investigation. Profiling remains experimental: enabled runs encountered a payload-arrival timeout, and review flagged concurrent atomic host writes versus non-atomic device clock reads as a synchronization concern. The profiler is not needed for normal inference.
With CPU-computed misses, cache borrowing and an 8.98 GiB VRAM cache, unprofiled Coder IQ1_M runs generated 128 tokens at 11.9-13.5 tok/s and returned final arithmetic answer 41. These runs retained 128K context capacity but used short prompts; the 20 tok/s goal has not been achieved.
The admission checks prompt + max_new <= max_context, but the spec loop
opened every window at S_mtp rows, so a request that fits ended with
"ran out of context" (rc 2) once p + S passed the end, and printed
nothing. The window now takes only the rows the context still holds.
MtpDrafter::draft wrote draft j at cell p + a + j for every step, past
max_cells near the end, where the page table has no entry; it now drafts
only the cells that exist. Far from the end nothing changes.
CUDA 12 refuses a GCC newer than 14, and the docs already tell the user
to run CXX=g++-14 ./setup.sh - but the environment only reaches CMake on
a build folder's FIRST configure, so on a retry (or the encoder's own
folder) the flag never landed: the reporter's run died at the last step.
host_compiler_defs() maps them onto -DCMAKE_C_COMPILER /
-DCMAKE_CXX_COMPILER / -DCMAKE_CUDA_HOST_COMPILER for the engine and the
CUDA image encoder; CUDAHOSTCXX defaults to CXX, as CMake's does. A
C++-only build (the CPU encoder) is not given the CUDA one.
STRATA_VERIFY_STEPPED=1 (Arc A-series): the doorbell window is captured as graph segments ending at each layer's
publish; the host serves the CPU experts between segments, pipelined with the plan copy + VRAM groups on a helper
thread pinned to the host thread's SMT sibling. No kernel then reads a host store while it runs and no host reads a
kernel store while it runs, so the window's mapped buffers are ordinary host USM and the publishes use plain stores
(STRATA_VERIFY_HOST_UNCACHED=1 keeps both uncached). A770, Coder IQ1_M, 8K prompt: GPU wait per window ~70 -> ~43 ms,
decode 22.8-25.0 -> 31.8-32.5 tok/s; 32K 27.2 tok/s.
The pipelined slot groups never run the adaptive tier, so --adapt-async 1 with --batch on a layer split and no
--batch-groups resolves auto to one group, like --batch-mtp (batch_groups.hpp, batch_groups_test); --batch-groups G
or auto given on the command line is honoured and turns --adapt-async off through the existing gate.
batch_step advances the async round one step per window (never between a prompt's chunks), commands drain the round in
flight before they run, a round with nothing to swap ends at its first step, and --adapt-min-gain sets the bar a swap
must clear (default 1.5, as before) for both tiers. --adapt-async is no longer refused with --batch slots (still with
--batch-groups).
batch_step now runs adapt() every --adapt-every windows on a thread beside the commit and the drafts, lands the swaps
(apply_pending) before the window ends, and pauses while a prompt holds a loan of cache slots. The timing line adds the
adaptive tier's rounds/swaps and wait, each stage's GPU-reach wait and pool time, and the VRAM/PCIe/CPU entry split.
The 0.1.41 default --batch-groups auto, the pipelined path's lack of slot drafts and the helper-cache refusal are
written down in docs/BATCHING.md (the limits list and the --batch-mtp paragraph).
--batch-groups auto became the default for a layer split with --batch >= 2, but the --batch-mtp gate ran before
the groups were resolved: it saw 1 group, kept batch-MTP on, and the groups were then raised to 2 - pipelined groups
do not run the slots' MTP drafts (#1413). The resolution is now a pure helper (include/strata/program/batch_groups.hpp,
CPU test src/program/batch_groups_test.cpp): with --batch-mtp and no --batch-groups the default is one group and
the log says why; --batch-groups N (N > 1) or --batch-groups auto given on the command line pipelines the groups and
turns --batch-mtp off, as before.
- generate.cpp: --batch-mtp accepted with a layer split on separate GPUs (refused with --batch-groups); slot drafters
load from the last stage's slot sessions under the last stage's device guard and bind to its weights/head; their
residual buffers, row copy (async on the drafter's stream) and draft-KV copies run on the drafter's device; every
stage verifier gets the 8-row window size and the batch graph limit; slot-drafter logits are not charged to CUDA0.
- verify.cpp: stage_batch's layout rules move to include/strata/core/batch_rows.hpp (CPU test batch_rows_test); grouped
rows are allowed on a stage that hands rows on, the hand-off bound is kVerifyMaxT rows.
- docs/BATCHING.md, --batch-mtp help text.
Step 1 warns below 4 GB of page file (issue #60) and advised "Set it to System managed". In #60 that did not help:
"System managed" and 4096-32768 MB still ran out of commit, and a fixed 64 GB worked - a page file Windows grows on
demand may not grow in time while WDDM charges the card's memory to the commit.
- The advice is a fixed size: Custom size, the same initial and maximum size, 65536 MB (PAGE_FILE_FIXED_MB).
- page_file_setting() reads the Virtual memory setting (PagingFiles; "?:" is the drive in ExistingPageFiles). When a
file is system-managed or its initial size is below its maximum, the warning says that it grows on demand and that
a growing page file may not keep up (#60).
- page_file_gb() still counts only what is there for sure: the commit limit minus RAM, as before, or a file's
configured initial size when that is larger (within its drive's free space). Never the maximum or what a
system-managed file may grow to. The 4 GB threshold is unchanged.
- docs/AI_SETUP.md and docs/DETAILS.md give the same advice.
- tools/test_setup_page_file.py: the parsing, the reading, the advice, and step 1 under --check on the golden
harness's 32 GB PC (registry, drives and GlobalMemoryStatusEx mocked; runs on Linux too).
The MCP server's FALLBACK_MODELS carries its own copy of the model
table, so #1688's setup.py fix leaves the MCP install path rejecting
--family swift --model IQ3_S. Drop the families gate there as well,
flip the validation test, and update the three doc lines that say
IQ3_S is original-model-only.
Measured on an RTX 3090 pair (engine 0.1.40.3, Swift IQ3_S, 200K
context, MTP spec on): 91-137 tok/s decode, expert cache hit rate
96-99%, KV streaming 98.6-99.6% VRAM hits.
UkisAI published a Swift 1.5 IQ3_S tier on 2026-10-08; the
"Swift 1.5 has no IQ3_S" restriction is stale. Both shards match the
family's file pattern exactly (names/sizes/SHA-256 verified against the
Hub listing), the shard count and the shared BF16 mmproj are unchanged,
so it takes the same path the other Swift tiers already use.
The IQ2/IQ3/IQ4_NL/IQ1_M decoders now write values [8 tid, 8 tid + 8) of a superblock (coalesced runs, as Q2_0
already did); iq_xmx_grouped still passed the IQ4_XS-style (sub-run, 32-block) index for them, so the B tile held
the wrong values (xmx_group_parity relerr 7-17 on types 18/20/21/22). All six types back to ~1e-6.
- xmx_moe: joint_matrix FP16 GEMM over a group of up to 16 experts per launch (grf 256).
Standalone vs oneMKL on uneven routing: down 1.75-6x, gate/up 0.8-3x (oneMKL kept above
STRATA_PF_XMX_GU_MAX=250 tokens/expert).
- STRATA_PF_XMX=2: iq_xmx_grouped decodes IQ3_XXS/IQ3_S/IQ2_S/IQ4_XS gate/up and IQ4_NL/Q2_0
down rows in-kernel (no FP16 staging); xmx_group_parity: rel err < 2e-6 vs dequant+oneMKL.
- End to end (8K/32K prompt, A770): +2-5% only; default off.
--pipeline-windows 2 now runs on any layer split with two or more stages. Every
stage but the last is a front stage: the guessed window follows the verified
one through them one card behind (an odd verifier and an odd hand-off on every
boundary, a pair of GDN snapshots on every front stage, commits and the undo
per stage), and only the last stage waits for the verdict. The pipelined prompt
read (--pipeline-windows 1) and --adapt-async stay two-stage.
HIP: the draft layer's per-row gr_read used the last stage's session scratch
(ss.block.gr), which that stage's verifier also uses in lm_head_mix. A chain
launched at a verdict and the next window's last stage can run on the card at
once after a guess that held, and the window's head then read a clobbered mix.
The draft layer now carves its own GrWorkspace.
Measured on 4x RX 7900 XT (Flash-Next GSQ-RCO IQ3_S, split 12,24,36, every
expert in VRAM, greedy, reasoning off): decode 59.4 -> 67.6 tok/s at 4K and
51.8 -> 65.6 at 32K. Greedy text identical to the serial loop on four prompts
with STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0, also with every guess
forced wrong. Debug knob STRATA_PIPELINE_SPEC_DEPTH. Two-stage path not re-run
on two cards.
probe_pcie_h2d_gbps: a pageable 256 MiB source made the copy engine fault
inside the driver's bounce path on a layer split (755x bcs VM_NOT_FOUND at
the fixed VA 0xffffc001ffa10000, L4 = the region never mapped for that VM,
intermittent on fresh boots, wedging the whole load in both the 0.1.40
port and 0.1.41). Pin the source (sycl::malloc_host) and free it with the
matching sycl::free. Validated: two consecutive loads reached the window
without the wedge.
All the split-fault instrumentation (guard watch, stage stamps and the
STRATA_STAGE_WAIT serial gate, terminate dumps, coredump grabbers) lives
on scratch/split-fault-instruments.
dpct turned `cudaStream_t cs_ = nullptr` into `&dpct::get_in_order_queue()`: a drafter that never loads keeps it, and
its destructor destroys the device's default queue (exit 139, sometimes UR_RESULT_ERROR_UNINITIALIZED, after the output).
Arc Pro B70 at fb58e0d without --mtp: 3 of 3 runs crashed before, 0 of 3 after; output unchanged with and without --mtp.
0.1.41 report (RTX 5090, tight RAM): chats failed and the engine restarted until STRATA_PREFILL_CPU_SHARE=0.
- share_rules: default_share_memory_verdict. The default is not armed when the RAM this process can get
(cgroup-aware) is below the share's buffers + 3 GiB, when the Windows commit left is below buffers + 4 GiB,
or when the experts it would read are file pages the free RAM cannot keep. One log line says why and that
STRATA_PREFILL_CPU_SHARE=auto turns it on anyway. An explicit setting is kept and only warns.
- prefill: the share's two pinned buffers are taken once at their largest (an expert takes at most MAXT
tokens' rows); a failed allocation ends the share for the run instead of failing the request.
- unit test, docs/DETAILS.md.
The counter block was one process-wide cudaMalloc on whichever device was current at the first Verifier::init, unreachable from
the other stage's kernels on a layer split without P2P. Each verifier now takes its own device's block in init (outside graph
capture), keeps the pointer, and a launch never picks by the host thread's device; a failed allocation runs without counters.
The exit report prints one block per GPU when there are several (single GPU: the same lines as before). The reporter's rare
fault with --pipeline-windows 2 after this fix is not reproduced here (needs two GPUs).
STRATA_ROUTE_RESIDENT_MTP=1 (experimental, opt-in, changes the output): the MTP draft's router gets the same residency bias,
with the last layer's residency row as its table (single GPU).
The refills of the slots a prompt borrowed (and the VRAM-resize refill) queue hundreds of cudaMemcpyAsync out of blob_stable()
pointers, which for a GGUF source are the recycled stage buffers. With STRATA_STAGE_PIN=1 the 257th assembly overwrote a
buffer whose copy had not run: IQ3_XXS lowmem (budget 12) decoded token id 0 after a 4096-token prompt. Each queued copy now
records the fence of its buffer. exact_pp 4096 and 20000 tokens with STRATA_STAGE_PIN=1: EXACT.
A busy slot flag with no owner kept every later request waiting for a slot nothing
would free, and a request waiting for the fifo (a load, an image encode, /v1/vram)
was counted nowhere. Now: slots have owners (claims / BDONE waits) and an ownerless
busy flag is freed after 3 s; the request waiting for a slot is counted
(admitting) and, past the stall time, named in the log; a parallel request on a
loaded engine no longer takes the fifo, and one that waits for it stays queued.
Tests with the fake engine.
On an RX 7900 XTX the first long prompt at 8192-token chunks charged ~70 GB of
commit and the engine died (0xC0000409); 7500 and below were flat. A bare auto
now stops at 6144 on Windows HIP builds. A chunk the user names stays, with a
warning from 8192 up.
Windows: the serve start prints the commit left and warns when the planned host
allocations plus a 4 GiB floor do not fit; the conversation cache's admission
reads min(available RAM, available commit).
With every expert resident nothing streams, so the share has nothing to take and
only moves the staged-chunk limit (+4.8% at 1,000 tokens on a V100). An explicit
STRATA_PREFILL_CPU_SHARE is honoured as before; the default is unchanged where
experts are not all resident.
A stage with one layer holds a handful of expert slots (a 12 GB 3060: 512), and the
prompt chunk of every card in the pipeline is clamped to what that cache can lend.
When the pick has such a stage, search again among placements that give every
stage at least three layers; keep the new pick and print both and why. With none
startable the first pick stays. Identical cards pick balanced stages and never
reach the second search.
available_host_bytes() = min(MemAvailable, the room under every cgroup limit above the engine, the operator's cap).
Parking and session save/restore read MemAvailable alone: in a container whose real limit sits on a parent cgroup it
cannot see, MemAvailable is stale (about 25 GiB shown when about 1 GiB was left) and the engine was OOM-killed.
conversation_available_memory() is replaced by available_host_bytes() / sample_host_memory() (the SYCL port's
generate.cpp too). The cgroup term covers memory.max and memory.high (v2) and memory.limit_in_bytes (v1) of the
process's group and every visible ancestor, with clean inactive file cache credited; an unparsable cgroup file makes the
sample unknown (fail closed). --memory-limit-mib / STRATA_MEMORY_LIMIT_MIB names the total RAM the container or cgroup
may use. Refusals say how much was available and from which source; one start-up line reports the limit.
tests/core/memory_guard_test.cpp runs on fixture directories (CPU only). Docs: DETAILS.md.
A profitable shorter chained lookup returned before the stale full-size check, so the full chain could stay excluded for good
although the policy re-probes stale sizes in the other selection paths (#1252). Check a confident stale full size before
returning the profitable shorter chain. The 300-round policy, the EMA, the header and plain lookup are unchanged; a regression
test covers the bounded re-probe spacing and the MTP confidence gate.
0.1.39 took this PR's first part with #583 (the xn / grs count, STRATA_RING_BYTES). What is left: bytes_needed
counted the fused layout's smaller MoE buffers for every caller (fused_layout(T, true)), while init takes them only
when it gets an ExpertSource (fused_layout(T, m.src != nullptr)). A prompt path without one was lent fewer bytes than
its unfused buffers take. bytes_needed, bytes_needed_no_ring and (0.1.40) bytes_needed_owned - STRATA_OWNED_PRICE=exact's
sizing - now take the same src the callers pass to init.
When the best split and a different one (boundaries at least 3 layers away, startable) are within 4% predicted decode window
(STRATA_SPLIT_NEAR_TIE_PCT, 0 = off) and the cards differ in speed or link, print the other K and --layer-split K. The model
prices decode only, not the prompt. Print only: the choice is unchanged.
A prompt read on a layer split with no slot decoding was one run, so the BYIELD of the batch_interleave_test step 4 was
dropped at the end of the read; the long prompt kept its slot and the next BGEN failed with 'ERR BGEN: no such free slot'
(pre-existing since 0.1.40.4). That read now goes in pieces of STRATA_BATCH_YIELD_CHUNKS chunks (default 8, 0 = one run), the
same chunks a single run reads, with the yield check between pieces. batch_read_piece holds the arithmetic; batch_read_test covers it.
A page-locked stage buffer is the source of the cache fill's cudaMemcpyAsync, which the DMA reads after the call has
returned; a pageable one is staged by the driver before it returns. The pool recycles a buffer after kStageAge layers or 256
assemblies, so a 4096-token prompt read (up to ~500 distinct experts per layer) could overwrite a buffer whose copy had not run
yet: the corrupted IQ3_S decode of #1237. note_async_read(blob, stream) records an event per pinned buffer after each queued
copy (hit-path fill, peer fill, remote-expert swap); claim_stage waits for it before it hands the buffer out again.
Pageable buffers and every other source are unchanged. STRATA_STAGE_PIN stays opt-in.
The reasoning ended at the first </think> in the output, also when the model only
quotes the tag while it thinks (`</think>`, "</think>\n\n2" from a test). The rest
of the thinking then came back as the answer, and a tool call quoted in it could
be run.
A </think> right after a quote or backtick and followed by anything but a line
break is now reasoning text; the end of the thinking is followed by a line break
and ends it as before. Until the character after such a tag arrives, the tag is
held like a partial one, so streamed and whole outputs agree. If the turn ends
right after it, it was the end of the thinking: a forced tool call still opens
there, and the #1053 retry is not triggered, as before.
Parking happens on a switch and every parked conversation costs one of
--conversation-cache-slots whatever its length. A client that interleaves
short side requests (an agent's tool-result follow-ups, often under 1k tokens)
with one long conversation therefore spends the cache on the side requests and,
once it is full, evicts the long one oldest-first - the reuse it wanted.
Measured on an RTX 4070 SUPER / 32 GB / Q2_0, --conversation-cache-slots 2,
one 19,198-token conversation interleaved with three ~700-token requests, then
the same conversation again:
min 0 skip nothing, side requests park, L is evicted
re-send L: 0 reused + 19198 read in 41,059 ms (wall 41.4 s)
min 12288 skip parking (703 tokens, below the 12288 minimum) x3
re-send L: 19193 reused + 5 read in 135 ms (wall 0.3 s, 304x)
The default 0 keeps the previous behaviour, so nothing changes without the flag:
wants() answers for the length alone and the gate stays enabled() && wants(),
exactly as the existing byte-budget and RAM-floor skips are. The check runs
before make_room(), so a skipped park neither evicts nor stores anything, and it
returns the same non-error path those skips take.
Tests: conversation_cache_test covers the inclusive boundary, 0, a negative that
never reaches the CLI, and that a disabled cache still answers for length; both
faults were reintroduced by hand to confirm the checks fail on them.
AMD_HIP.md, RDNA4 speed switches: STRATA_PF_FUSED=1 now runs the native
IQ packs' prompt experts on the int8-WMMA kernels on gfx1200 / gfx1201.
Says that it is opt-in and why (it rounds differently, so it is not an
arch default), that STRATA_PF_FUSED_NATIVE=0 turns the native path off,
the start-up line it prints, the R9700 measurements (prompt time, kernel
vs MMQ, first-token KL against the FP16 path, retrieval checks), and
what was not run (gfx1200, Windows, K-quant packs).
DETAILS.md: the STRATA_PF_FUSED=1 sentence says where the native kernels
run on HIP. CMakeLists.txt and gfx_arch.hpp: the comments that said the
native prompt-expert kernels are gfx11 only now include gfx12 (comments
only, no build change).
The native IQ / K-quant prompt-expert kernels (moe_fused_iq.cu,
native_w11_kernel) were gfx11-only; on RDNA4 a pack with
STRATA_PF_FUSED=1 silently kept the MMQ path. They now compile and run
on gfx1200 / gfx1201. Still opt-in: STRATA_PF_FUSED=1 stays off by
default (it changes rounding, so it is not an arch_defaults switch), and
STRATA_PF_FUSED_NATIVE=0 still turns the native kernels off.
Compile gating
- STRATA_NAT_W12 for __gfx1200__ / __gfx1201__, STRATA_NAT_WMMA for
either family. The kernel calls the _gfx12 spelling of the iu8 WMMA
builtin (the un-suffixed one does not compile there). Any other
target keeps the trapping body.
Kernel (same arithmetic, other lane layout)
- gfx12 WMMA fragments are 8 int8 per lane and the two lane halves hold
different k (8*hi .. 8*hi+7), not a replica as on gfx11: B and A are
loaded as uint2 at the matching offsets, and every 16-value k step is
still one WMMA, so the per-16 scales of the IQ2_S / IQ2_XS paths are
untouched.
- A lane's 8 accumulator rows are consecutive (8*hi + i), so a feature's
gate and up are in the same lane: SwiGLU needs no __shfl_xor, and the
down tile is stored as two float4 per lane. NW_DROW(i, hi) gives the
row of element i for the per-row scales.
Host side
- gfx_arch_is_gfx12_wmma(): exact gfx1200 / gfx1201 match, never a
prefix, like the gfx11 helper.
- dev_info() takes the gfx11 list or gfx12, and also checks that the
kernel the build loaded is the real one: the trap stub (a
gfx12-generic or SPIR-V build on a listed card) declares no LDS and
still passes hipFuncGetAttributes and the occupancy query.
- native_supported() reads the STRATA_PF_FUSED=1 opt-in itself:
fused::available() / requested() belong to the Q2_0 pack's
expert_w11_kernel, which is not ported and stays false on gfx12. Off
gfx12 the result is what it was. prefill.cpp fused_ring() follows:
a native pack asks native_supported() alone (it implied enabled()
before), so on gfx12 it gets the fused buffer layout and ring.
- Grid factor per format: gfx12 holds 3 blocks per WGP for IQ2_S / IQ2_XS
gate/up (LDS and VGPRs), 4 for the rest. The occupancy API cannot say
(it models 64 KB of LDS per multiprocessor), so it is a small table;
STRATA_PF_OCC_W=N overrides it. gfx11 keeps 4.
- experts_native() refuses a dm that is not 16-byte aligned on gfx12
(the float4 stores); the prompt path's buffers always are.
- moe_fused.cu compares the gfx11 list with the helper instead of the
"gfx11" prefix (no change on any listed part).
Tests (prefill_fused_iq_test)
- Runs on gfx12 (the skip gate also asks native_supported()), covers the
other four pairs of the Coder IQ1_M pack, S20_E sets the expert count,
and the real-shape check of fused against MMQ is now rel RMS <= 1e-3
and worst row <= 2e-2 instead of rel RMS <= 0.05.
Measured on a Radeon AI PRO R9700 (gfx1201), Coder IQ1_M pack:
- parity: all 10 IQ pairs plus Q4_K / Q5_K pass, rel RMS <= 9.1e-5.
- kernels vs MMQ, 48 layers: 1.48x / 1.20x / 1.14x at 4K / 16K / 32K.
- prompt speed in the engine (server-reported, alternating starts per
arm): 4K +12% to +19% depending on the start, 16K +8.3%, 32K +8.5%.
- first-token KL against the FP16 prompt path is 1.10x MMQ's (48 prompts
of 4K), top-1 agreement 48/48; 4/4 long-context retrieval checks.
Not run on gfx11 or CUDA hardware here.
unsloth/Qwen3.8-Flash-Next-GGUF UD-Q5_K_XL shard 1 holds metadata only and
is 10,946,618 bytes; its 32-byte-aligned data start is 10,946,624, so the
engine refused it with 'GGUF: data section starts past EOF'. llama.cpp's
gguf-py reads the same file (0 tensors, data_offset 10,946,624).
A file with tensors is still refused. A file without them gets data_start
clamped to the file size, so file_size() - data_start() stays 0 in the
payload checks downstream.
On one GPU the prompt path computes the draft layer's K/V for each chunk in batches (Prefill::draft_kv, E-9). A layer
split always ran the drafter's own per-group pass instead, because the drafter sits on the last stage. With
STRATA_SPLIT_MTP_BATCH=1 the last stage's prompt path runs E-9 there. Default off: the default path is unchanged.
The drafter's K/V come from Q8_1 x Q8_0 MMQ instead of its own mmvq, so its drafts can differ; the target's tokens are
decided by the verify window.
Prefill::draft_kv says why it declined (optional out-parameter, also in the SYCL port's prefill.cpp), and the split
logs once what the variable did. Which path runs is a pure header (include/strata/program/draft_kv_plan.hpp) with a
CPU test. tools/split_mtp_batch_parity.py compares the greedy texts of the two arms on a live server. Docs: MULTI_GPU.md.
Two synthetic cases: no tensors and the file ends at its header (should
open; data_start() clamped to the file size), and one tensor with the same
truncation (still refused). The first fails on 0.1.40.3: unsloth's
UD-Q5_K_XL shard 1 is that case.
Header-only placement (include/strata/platform/aux_cpus.hpp): the spare CPU set (the host core's SMT siblings, then
CPUs of physical cores with no worker; never a CPU that shares a core with a worker), the list parser, threads the
engine creates pin themselves, threads somebody else created (the CUDA driver's) are moved by a sweep over
/proc/self/task after the pool exists and at every request; pool workers and the host thread are never moved.
--aux-cpus off|auto|LIST / STRATA_AUX_CPUS; per request strata_tune aux_cpus=0|1 (serve/server.py).
Default off: the hooks do nothing. tests/platform/aux_cpus_test covers the derivation and live threads (CPU only).
Builds on #1190. The layer split sized its lend regions (the RAM copies of the experts the prompt path borrows) with
Prefill::bytes_needed before the split's ring override was applied (Prefill::set_ring_override says it must be set
before any bytes_needed), so with STRATA_SPLIT_RING=384 the prompt path borrowed 5396 slots and the copy kept 4662.
The override is now decided first (decide_split_ring); include/strata/program/split_ring.hpp holds the pure helpers
(split_ring_slots, split_lend_slots - the scan's own tail-slot arithmetic, now one function for the region and the
borrow), src/program/split_ring_test.cpp covers them on the CPU. A start line reports borrowed vs kept slots and warns
when kept < borrowed. Docs: MULTI_GPU.md.
Engine 0.1.40 (fe9c10ca, #848) keeps the resident RAM copy on a layer split -
every stage's cache left out of it, the rest ranked by the whole expert
profile - and --resident-budget-gib runs through the same code
(pin_cache_complement), so the old engine refusal of the budget on a split is
gone. setup.py's split_mmap() got the matching exemption (43c8cfa9), but the
budget's three paths still assumed the old engine (#498):
- split_budget(): a start of a budget config on several GPUs deleted
--resident-budget-gib (and asked for ~135 GB of RAM below that);
- offer_together(): a budget config was offered both cards only with ~135 GB
of RAM, and "on both it has no budget";
- unsloth_together() / q4_split at install: --gpus, or "2" when asked, dropped
the budget, and an explicit --resident-budget-gib kept one GPU.
From RESIDENT_SPLIT_ENGINE the budget is kept in all three, at any RAM. The
recommendations stay as they were (the golden configs are unchanged): one GPU
by default at install, "n" by default when a start offers both cards, --yes
alone one GPU; the split is taken only when asked for.
Tests: the #498 tests run with MIN_ENGINE pinned before RESIDENT_SPLIT_ENGINE
(as test_setup_risk's StartOnSeveralGpus does for #642), three new ones run the
kept budget; tools/test_setup_*.py on Linux: 421 tests, the 2 failures of
test_setup_engine_hash are the same without this change (no ready-made 0.1.41
engine to download in the test box).
Measured with engine 0.1.41 on an RTX 3090 (x16) + RTX 3080 Ti (x4), 91 GB of
RAM, UD-Q4_K_XL, budget 64 GiB: the auto split gives the x4 card layers 0-1
(678 slots) and the 3090 layers 2-47 (4,182 slots); 57.6 GiB of experts in RAM,
no file-tier reads per request; decode 45.0 / 45.1 tok/s (two runs) against
38.4 on the 3090 alone, a 20K prompt in ~18 s on both. Answers checked by hand.
A pip launcher whose Python was removed stays on PATH ('ninja --version'
fails) and stopped every build there, though setup had just installed a
working copy in .venv.
UPDATE.bat currently prints a command using `&&` for users to copy and
run manually. `&&` is not supported by Windows PowerShell 5.1, which
is still included with Windows 11.
Print the two Git commands separately instead.
dpct migrated cudaStream_t ext_stream_ = nullptr to a pointer to the
in-order queue, evaluated at construction on whichever device is current.
Two places then misread it: init() compared ext_stream_ against the
*init-time* device's in-order queue, so a stage built on one device and
init'ed on another adopted the constructing device's queue
(cross-context submit failures, UR_RESULT_ERROR_DEVICE_LOST); teardown's
'cs_ != ext_stream_' ownership test drifted the same way.
Restore the CUDA contract: nullptr means 'no stream until set_stream()',
matching include/strata/core/verify.hpp:319 and src/core/verify.cpp:604.
Follow-up to the >4 GiB GEMM staging.
- gemm_stage freed its scratch as soon as a larger operand arrived, but the
in-order queue is asynchronous: a previous staged GEMM could still read the
buffer. Wait for the queue before the free.
- The staging used dpct::get_in_order_queue() while the GEMM runs on the
descriptor's queue (strata::q_of(stream_)); pass the descriptor's queue so
the copy is ordered before the GEMM for a non-default stream, and so two
Gemm objects on different queues do not share the thread_local scratch.
- Staging that could not allocate fell silently back to the direct (wrong)
GEMM; it now says so.
- device_alloc_map had no unregister, so a freed cache base left a stale
entry that device_offset_end could mis-attribute after the VA was reused.
Add device_alloc_unregister and call it where the expert cache frees base_.
setup reads the Intel cards from sysfs but to_sycl() decided the expert
mode from the driver alone, so a card's device id was thrown away and an
A-series on xe (the A770) could not be told from a Battlemage. Carry the
PCI device id through intel_gpus() into to_sycl(), and treat every
Alchemist id (A770/A750/A580/A380/A310/A770M) as an A-series: it has no
FP64 hardware, so its config carries the driver's FP64 emulation (a kernel
on the sampled path declares double and is refused without it).
On i915 (the A750) the experts still load into a CPU-computed RAM arena
(--mmap-experts + STRATA_VERIFY_NO_HOST=0) - a single pinned host
allocation past ~3 GiB fails on that die. On xe (the A770) the A-series
now takes the ordinary streamed path: the mirror is built in chunks (so
the >3 GiB limit is not hit) and the build passes
-ze-opt-greater-than-4GB-buffer-required, so the prompt path is correct
past the cache's 4 GiB. tools/test_setup_sycl.py passes.
On an Arc A770 every prompt answered "!!!!" (token 0). The experts in the
last 0.46 GiB of the 4.46 GiB VRAM expert cache dequantized to NaN in the
prompt path. The slots' bytes were right (a memcpy read back equal to the
GGUF, and a fresh copy dequantized clean); only kernels reading them in place
got wrong bytes. In a JIT build IGC addresses kernel pointer arguments as a
32-bit offset from the allocation's base.
The JIT (non-AOT) build now passes -ze-opt-greater-than-4GB-buffer-required
to the spir64 backend, in the same argument as the divide/sqrt option (CMake
drops a repeated -Xsycl-target-backend=spir64). Measured on the A770, Coder
IQ1_M, --stream-experts: prompts of 73-542 tokens have 0 non-finite values
under STRATA_DBG_NAN=1, and the 542-token prompt answers right. The AOT build
is unchanged. Speed cost not measured.
oneMKL's GEMM reads the last, partial tile of an operand from the wrong
place when that operand lies more than 4 GiB into its device allocation
(measured on an Arc A770, oneMKL 2026.1: a BF16 B past 4 GiB gave its last
21 columns wrong, none below 4 GiB). The prompt path's GEMMs meet this:
their buffers come from the top of the multi-GiB expert cache / prefill
region. This is the leading explanation for the A770's garbage on prompts
longer than a few dozen tokens - it repros with every expert
device-resident, so it is not the CPU handshake.
Register every big device allocation (malloc_device_guarded, and the
segmented expert cache's reserved VA) in a small registry, add
device_offset_end(ptr, bytes), and in Gemm::bf16 / Gemm::f16 copy either
operand into a scratch buffer of the queue's own (a fresh allocation, so
at offset 0) when its end lies past 4 GiB, then run the GEMM on the staged
operand. STRATA_SYCL_GEMM_NO_STAGE=1 is the negative control.
Not validated end to end: the run that showed the garbage was an
all-resident mirror, which no longer fits in this machine's RAM, and the
runnable --mmap-experts config crashes on this A770 with or without the
staging (exit 139, UR_RESULT_ERROR_DEVICE_LOST - verified a separate,
pre-existing fault with the negative control), so the staging has not yet
been shown to change an answer.
The decode handshake publishes a payload (the layer's activations, the
routed expert ids and weights) and then rings a sequence number the host
polls. The fences alone do not order the ring after the payload stores:
measured on an Arc A770, 14 of 2,832 rings in one run became visible
before the last activation store, so the CPU expert pool sometimes
computed on a partly stale payload. This is the fix the port's own
comments call for (elementwise.hpp: "the host can observe the ring and
then read a payload that has not landed").
The publish kernels now write, next to the sequence number, the ring it
belongs to and a checksum of the payload (the wrapping sum of every 32-bit
word mixed with its position, murmur3's finalizer - a plain sum missed a
reordering, [3,7] -> [7,3], of the expert ids) in a per-ring slot modulo 4,
and the payload is stored with the uncached write hint (a plain store to
host USM reaches the host only when the kernel ends on xe). The host waits
for the whole payload (doorbell_wait_payload) before the pool reads it, in
session_loop and both verify-window loops; the checksum is accepted in
either publish shape (with or without the routing weights), so a variant's
choice need not be plumbed to the reader. The sequence word grows from 1 to
9 mapped words (layer.cpp, verify.cpp) and the ring slot is reset with it.
Note this is a robustness fix, not the cure for the A770's garbage on
longer prompts: that one repros with every expert device-resident (no CPU
handshake) and points at oneMKL's >4 GiB operand bug, which needs the BLAS
staging the CUDA/HIP shims have.
At a long context this port borrows expert-cache slots for the prompt
path. Those buffers are carved from the top of the ~10 GiB expert-cache
allocation, and oneMKL's BF16 GEMM reads a wrong tile when an operand lies
more than 4 GiB into its allocation (docs/INTEL.md). The borrowed operands
hit that, so from the first full prefill chunk the router read garbage.
Measured on an Arc A770: a 236-token prompt failed with "prefill: routed
id out of range", and a 452-token one segfaulted the engine (exit 139),
on both the mirrored and mmap configurations. With the prompt path on its
own buffers (its own allocation, reserved exactly by owned_prefill_mib)
and an automatic chunk capped at 2048 tokens, 452 / 1,748 / 3,476-token
prefill chunks ran to completion with no prefill failure (prompt reads
22 (cold) / 676 / 835 tok/s). Whether the *answers* of that configuration
are correct is a separate, still-open issue: the base port has no doorbell
payload checksum and its longer-prompt path garbles on some inputs, with
or without borrowing.
Scoped to Alchemist (PCI device ids, the same table setup_intel.py uses):
Battlemage's long prompts were measured correct with borrowing, so it
keeps the old default. --prefill-borrow still borrows, for testing a
future BLAS staging fix (the fix that lets borrowing return, as the
CUDA/HIP shims already stage).
Two Alchemist (Arc A-series) fixes for the streamed-expert mirror:
- gguf_expert_source: a single sycl::malloc_host past ~3 GiB fails on DG2,
so the mirror (thousands of experts, tens of GiB) could not be built on
an A770 at all. It is now allocated in <=2 GiB chunks; the per-(layer,
expert) address, the layer's first blob and close() use the chunk
pointers. A cleanup guard frees every chunk on an allocation failure,
a failed read, or any exception; the read phase is unchanged.
- strata-sycl.sh: docker's default memlock is 8 MiB, far below what pinning
the mirror or the resident arena needs; pass --ulimit memlock=-1.
Measured on an Arc A770: the Coder IQ1_M (8,474 experts missing VRAM,
16.15 GiB) mirrors and the engine starts; before, it refused with
"8474 experts are neither in VRAM nor mirrored".
The decode handshake publishes a ring sequence the host polls while the
window graph still runs. On an Arc (xe) a system-scope atomic store, even
with a release fence, is served from the GPU's cache and reaches the host
only when the kernel ends, so a host polling mid-window never sees the
ring. That is why the verify window had to wait for the whole graph
(STRATA_VERIFY_NO_HOST) and every expert had to be device-resident or in
the pinned mirror.
STRATA_DOORBELL_ATOMIC_LOAD already fixes the load side the same way; this
does the store side with the Intel uncached L1+L3 write hint. The release
fence stays (ordering only: a later payload store follows the ring).
Measured with sycl/probe/doorbell_store.cpp on an Arc A770: the host sees
the store at 230 ms with the kernel running to 2030 ms; with the old atomic
store it sees it at 2038 ms, exactly when the kernel ends (exit code 3).
-DSTRATA_DOORBELL_ATOMIC_STORE restores the atomic store.
This is a prerequisite for running with the CPU expert pool on xe (the
per-layer handshake); it does not by itself fix longer-prompt prefill.
psutil.disk_io_counters() returns None when it finds no disk (Modal's gVisor
has no /proc/diskstats; psutil's docs say Windows can need `diskperf -y`).
_disk() then raised AttributeError at the first sample and the sampler thread
ended: /metrics' hardware stayed empty, the Monitor tab had nothing to show,
and the frozen-engine check's gpu_busy() read a GPU load of 0. It now returns
no disk reading and the other readings go on.
The READY-read timeout from #1317 kills the engine, but the request that
was loading it ended with no response: the RuntimeError from __init__
matched none of the request path's handlers, and on a lazy first start
restart()'s finally crashed on slot_cv (created only after READY), so a
misleading 400 masked the failure instead.
- raise EngineDied (still RuntimeError) for the start failure, so the
request ends 503 'did not report READY ...; the next request restarts
it' on the load, restart and /v1/load paths alike
- skip the slot_cv wakeup when a lazy first start never created it
serve/test_ready_timeout.py covers both paths with silent fake engines:
the lazy load (unfixed: 400) and a restart that never reports READY
(unfixed: connection closed with no answer).
#1352 orders an auto layer split faster-card-last because the last stage
runs the head, the draft layer and the verify. On a VRAM-asymmetric pair
that rule can put the heaviest stage on the card least able to hold it:
the reporter's 12 GB 4070 Ti ended up with 31 layers + head + verify and
a 3 GiB expert cache (decode ~58 -> ~44 tok/s, the 'card this small'
warning on).
Skip the reorder when it would put the last stage on a card with less
total VRAM than the one the config placed there (NVML / amdgpu sysfs;
unreadable cards keep the reorder). scores/vrams stay injectable for
tests.
The VRAM reuse stays, but bytes_needed_impl counts bo again by default, so the auto chunk and the borrowed cache slots
are those of 0.1.40.3 and the prompt residuals stay byte-identical (on the RTX 3060 the planner otherwise picked a
6656-token chunk instead of 6400). STRATA_EMB_REUSE_ACCOUNT=1 lets the planner use the saved bytes (+3.9% prompt there);
documented in docs/DETAILS.md.
The previous change skipped the DRM-fdinfo subtraction whenever the driver reported usage. On a single Arc Pro B70 that
made the expert cache 12168 slots instead of 11338 (23.09 vs 21.52 GiB), so a one-GPU run was not unchanged. The
subtraction is now skipped only when the process sees more than one GPU (where it double-counts the other card) and the
driver does not report the whole card as free.
The bound of the device's spins on host flags was a build constant: 20,000 reads for a bmg AOT build, 2,000,000
for anything else. A B70 built without AOT (SPIR-V, JIT: the Windows/OpenCL build, or a Linux build with no AOT
target) got the A750-sized bound. The issue reports a 37% drop in decode between 0.1.40.2 and 0.1.40.3. On a Linux B70 JIT build the two bounds decode
alike (--spec 2, 3 pairs: 19.4 vs 19.5 tok/s), so this makes the bound independent of the AOT flag but is not shown to be that drop's cause.
spin_max(queue) decides per device: 20,000 under the xe driver (the B-series on Linux), 2,000,000 elsewhere (the i915
A-series needs it for the host's per-layer expert work; a Windows/OpenCL B-series build measured faster with the long bound).
STRATA_SPIN_MAX=<reads> overrides it at run time; -DSTRATA_SYCL_SPIN_MAX=<reads> still fixes one bound for the
whole build. The three waiting kernels take the bound as an argument.
- generate.cpp: with a layer split the host mirror walked the later stages' layers too (#1054, fixed in #1111 but
lost in the re-migration): 20 GiB of pinned RAM for experts the other card holds, or one 39 GiB pinned allocation
that fails. Both loops stop at the first stage's last layer.
- expert_cache.cpp: device_free_bytes() subtracted the DRM bytes of every card of the process from the card it asked
about, so on a layer split the second card was charged the first one's cache and its expert cache would not open.
The fallback is for drivers that report the whole card as free (Arc A750/i915); it now runs only then.
Reported by crobe201 with both diffs; MaineTim confirmed the second on 2x B60.
Co-authored-by: crobe201 <[email protected]>
The three sites fixed by the previous commit stored a clamped (d, sum) but still
quantized against the unclamped d, so an overflowing block stored int8 values
that no longer matched its stored scale. They now use q8_1_finite(d) and
q8_1_quant like native_quantize_q8_1_kernel, the swiglu quantizer and the CPU
pool. A block whose amax is finite and below 8.3M is stored bit for bit as
before.
Co-authored-by: Matthew Hines <[email protected]>
Three kernels still emit the (d, sum) pair of a q8_1 activation block
with a raw make_half2(d, sum), bypassing the finite helper the rest of
the tree uses (q8_1_finite.hpp, from 19de7f4 / f6601bf):
fused_gr.cu gr_q8_tail - the fused gate-reduce tail (QFUSE)
verify_kernels.cu gdn_q8_1_store - the batched GDN verify store (QFUSE)
iq_kernels.cu s26_swiglu_q8_1_kernel - the S26 swiglu path
A block whose amax overflows (or that reads a non-finite input) emits
d = inf/NaN; fp16 makes the pair NaN (or the pathological (0,0)). Nothing
downstream tests for it: a NaN scale poisons the fp16 dot of the quantized
activation and every sum downstream of it, and a fully non-finite logit
row ends at the sampler's all-NaN fallback - vocabulary id 0, the '!'
face of the degeneration reported in #606. Same defect class already
fixed at the other emit sites; these three were missed.
q8_1_ds() clamps both halves into the fp16-finite range (finite d,
non-negative finite sum - the sum is a mean of |x|), producing for the
GPU paths the same bits the CPU pool and the clamped GPU sites produce
for the same block. A clamp at the producer, content-blind: nothing
downstream of it is model-text-aware.
Criteria and the wider carried-state RCA: docs/Q8_1_FINITE_DS.md and
docs/DEGENERATION_RCA.md.
2e4ddf6 (the #904 port) removed __restrict__ from the activation pointers of
native_quantize_q8_1_kernel, bf16_f32_mmvf_multi_kernel and
native_mmvq_multi_kernel on every CUDA architecture, and turned on the weight
prefetch before pdl_wait(). Both only matter for PDL (sm_90+). On sm_61 the
missing restrict loses the read-only data path (LDG.CI) and decode halves
(36.4 -> 17.7 tok/s on a Tesla P40, 32-36 -> 17.7-18.6 on a GTX 1080 + 1070).
STRATA_PDL_RESTRICT is __restrict__ and kPdlPrefetch false again for
__CUDA_ARCH__ < 700 in the device pass. sm_70 and newer compile to the same
SASS as before (cuobjdump -sass of native_mmvq.cu and native_bf16.cu at sm_70,
86, 90 and 120 is identical); the host pass is unchanged.
Reported and bisected by paulhothersall and lineape.
2e4ddf6 (the #904 port) removed __restrict__ from the activation pointers of
native_quantize_q8_1_kernel, bf16_f32_mmvf_multi_kernel and
native_mmvq_multi_kernel on every CUDA architecture, and turned on the weight
prefetch before pdl_wait(). Both only matter for PDL (sm_90+). On sm_61 the
missing restrict loses the read-only data path (LDG.CI) and decode halves
(36.4 -> 17.7 tok/s on a Tesla P40, 32-36 -> 17.7-18.6 on a GTX 1080 + 1070).
STRATA_PDL_RESTRICT is __restrict__ and kPdlPrefetch false again for
__CUDA_ARCH__ < 700 in the device pass. sm_70 and newer compile to the same
SASS as before (cuobjdump -sass of native_mmvq.cu and native_bf16.cu at sm_70,
86, 90 and 120 is identical); the host pass is unchanged.
Reported and bisected by paulhothersall and lineape.
Found while running it against a live engine: the line said 0.35 as 48.7 on round 3 while the three draws were 57.0, 48.7 and 35.5, because it printed the running median. Reading a sweep's drift is the whole point of the line, so it has to be the draw. The medians and the ranges are still in the line below it.
Phase 1 measures each PCIe share once and takes the maximum, with the three prompts of that rate all at the one
setting. Neighbours cannot be ordered that way: on this box 0.75 through 0.95 come out at 72.7, 69.0, 71.0 and
71.9, which is 0.2% apart, while the same settings moved 8-28% between engine starts. A value that was worth a
real gain can lose its single sample and never be measured again, because phase 1's winner is the share phase 2
then sweeps spec_min_p at, and the confirm step only puts that one resulting pair against the default.
sweep() measures every value SWEEP_ROUNDS times inside the one engine start, one round visiting all of them and
every other round in reverse order, and the winners come from the medians. Drift inside a sweep then lands on
every value the same number of times, and one bad sample cannot decide. The same function runs phase 2, and the
confirm step uses the same round count instead of its own.
Cost: the sweep multiplies by SWEEP_ROUNDS, 3 by default. A rate is three prompts of 128 tokens, so on this box
(six values, 95-100 tok/s) phase 1 goes from about 40 s to about 120 s, on a run that already loads the model more
than once. A machine whose two leaders stay inside each other's noise can raise the constant.
Not in this change: phase 4 needs a restart per worker count and stays sequential, so a count is still taken from
one measurement each.
Tests: python -m unittest tools.test_calibrate (the stand-in engine is a pure function of the tune keys, so the
medians and the winners are the same as before); tools.test_setup_config and tools.test_setup_golden
PCIE_FRACS stops at 0.75, so setup --calibrate never measures the value that takes the CPU pool out of the decode
window completely. 1.0 is a separate operating point rather than 0.75 carried further: the share is applied as
m = (nmiss * pcie_num) >> 8, so at any value below 1.0 at least one miss still goes to the CPU pool, and only at
1.0 does every non-resident expert come over PCIe. A CPU expert sits on the layer's critical path while the GPU
waits for it, and on a slow CPU that is the whole difference.
On a Core Ultra 9 290HX Plus with an RTX 5090 Laptop on PCIe 5.0 x16, 0.55 gives 51.1 tok/s against 71.3 at 1.0,
with cpu_exp falling from 1.06 to 0.00 per layer-window and the window time falling by the 11.3 ms of PCIe waiting
plus the CPU's expert time. A sequential sweep puts 1.0 at 78.8 against 72.7 for 0.75 and 51.5 for 0.35.
0.9 is added beside 1.0 so a machine where 1.0 is too much gets an interior point. pick() keeps the default unless a
candidate beats it by MIN_GAIN, so this cannot make a calibration worse. The cost is one rate per value, about 8 s
here.
Tests: python -m unittest tools.test_calibrate; tools.test_setup_config and tools.test_setup_golden
With a second toolkit beside the distribution's one, CMake compiled with the chosen nvcc but took cudart and cuBLAS from /usr (Ubuntu 24.04, STRATA_NVCC=/opt/cuda-13.0/bin/nvcc: the engine needed libcudart.so.12 while built with the 13.0 headers). Pass CUDAToolkit_ROOT for the engine and the CUDA image encoder; the distribution's own /usr/bin/nvcc keeps CMake's search as before.
Two gfx906 build failures on v0.1.40.2. Both are one-line gates that know
about the wave32 HIP backend but not about STRATA_HIP_GFX906 -- the same
class as #808 and #1083. Neither is fixed on main (e8ca9afd03).
1) src/prefill/prefill.cpp:399 builds an event with
cudaEventDisableTiming | cudaEventBlockingSync, but the gfx906 compat
header include/strata/platform/hip_compat/strata_hip.h maps only
cudaEventDisableTiming and cudaEventDefault. The wave32 header
include/strata/hip_compat/cuda_runtime.h:39 has the mapping.
-> add #define cudaEventBlockingSync hipEventBlockingSync there.
A diff of the whole set of cuda* defines between the two headers lists
one other name the gfx906 header lacks, cudaDevAttrIntegrated, and that
one is already gated upstream (#if defined(__linux__) &&
!defined(STRATA_HIP_GFX906) in src/core/expert_cache.cpp:31), so it is
deliberately not touched here.
2) Link error: undefined symbol
void mul_mat_q_case<(ggml_type)14>(ggml_backend_cuda_context&, mmq_args const&, ihipStream_t*)
(14 = GGML_TYPE_Q6_K).
src/prefill/moe_mmq.cu:219 calls it under
#if defined(__HIPCC__) || defined(STRATA_Q6K_EXPERTS), so any HIP
compilation references it; mmq.cuh declares the case as an extern
template, so the definition must come from
template-instances/mmq-instance-q6_k.cu. The CUDA branch of
CMakeLists.txt adds that file only under STRATA_Q6K_EXPERTS.
STRATA_HIP_GFX906=ON forces STRATA_ENABLE_CUDA=ON, so gfx906 takes the
CUDA branch while its .cu files are compiled as HIP.
-> widen the gate: STRATA_Q6K_EXPERTS OR STRATA_HIP_GFX906.
This adds one MMQ instance to the static library. It does not enable
STRATA_Q6K_EXPERTS (the grouped Q6_K expert kernels that allocate extra
VRAM), and STRATA_HIP_GFX906 is only ever ON in the gfx906 build, so
CUDA and wave32 builds are untouched.
Verified on a single Instinct MI50 32 GB (gfx906, wave64), ROCm/HIP 7.14,
-DSTRATA_HIP_GFX906=ON -DCMAKE_HIP_ARCHITECTURES=gfx906
-DSTRATA_MMQ_KQUANTS=ON:
- the engine builds and serves UD-IQ4_XS (Qwen3.8-Flash-Next) end to end:
/v1/models, chat completions, tool calls, MTP drafts accepted;
- nm -C shows mul_mat_q_case<(ggml_type)14> in the binary (13 instances:
7 8 12 13 14 16 17 18 20 21 22 23 42);
- change 2 is load-bearing: removing mmq-instance-q6_k.cu.o from
libstrata_mmq.a and relinking with the same link command line fails with
exactly the undefined symbol above (the archive holds one U reference
and that one W definition), while the untouched archive relinks cleanly.
Two variants of the same sweep (code-explanation prompts and Russian prose) on
one PC and one server config, engine 0.1.40.3 built from source for sm_89.
Each folder carries the 0.1.40.2 run, the 0.1.40 baseline and an
identical-binary control run so the noise band is in the report.
Result: no measurable change from 0.1.40.2 on this machine.
Same laptop, model files, configuration and scripts as the 0.1.40.1
entry (2026-10-06-community-rtx-5090-laptop, #1263); only the Strata
version changed. Raw data for 0.1.40.3, medians for 0.1.40.2 in the
README. Decode and prompt throughput unchanged within run-to-run
variation, all needles found, coding check 10/10 on 0.1.40.3. A 0/10
coding check on 0.1.40.2 is traced to a greedy reasoning loop that
occurs in 0.1.40.1 as well (re-runs documented). Results only, no
engine changes.
Coder IQ1_M, one RTX 5080 16 GB, the repository's own Dockerfile and entrypoint
on a Windows host (Docker Desktop 4.84.0 / WSL2), two context limits on the same
loaded model: 131,072 and 65,536.
- Speed: three runs at each of 4,096 / 8,192 / 16,384 / 32,768 / 65,536 /
130,560 prompt tokens (128K limit) and at 4,096 / 8,192 / 16,384 / 32,768 /
65,024 (64K limit), greedy, reasoning off, 256-token output cap, zero reused
prompt tokens in all 33 runs.
- Correctness: tools/needle_bench.py 9 of 9 (32k/64k/128k) at the 128K limit and
6 of 6 at the 64K limit, plus a vision check with the encoder on the CPU.
- Provenance: all three model files SHA-256-verified against the repository's
published LFS oids at the pinned revision; MTP manifest and inventory included.
- Environment: Docker Desktop's default WSL2 memory cap makes setup.py read ~31 GB
of a 64 GB host and silently select the low-RAM mode; the report documents the
.wslconfig change used instead. Also documents the 8-token headroom a request
needs (prompt + max_tokens + 8 <= max_context) and the engine's own VRAM warning
at a 128K context on a 16 GB card.
Results-only; no engine changes.
Nine files following docs/COMMUNITY_BENCHMARKS.md. Engine v0.1.40.3
(tag d5ea713) built from source for sm_75, IQ3_S at the model native
262,144-token window, with --vision and --reasoning-budget-tokens 12000.
Three runs each at 4,096 / 32,768 / 128,000 prompt tokens: decode
70.9 / 77.3 / 71.1 tok/s, prompt 868.9 / 1548.9 / 1626.9 tok/s, medians
in the index and full ranges in the report. needle_bench 6/6 at 32k and
128k. Records that NVLink is present but unused by the engine, so a bridge
on consumer boards is not worth buying.
The flag acts only on --expert-cache-device1..3; a plain layer split got a no-op flag that the docs listed as a reason
for the pipeline and the async tier to switch off. Built on f-141-quick's #1352 pipeline-windows rule (same hunk).
MULTI_GPU.md: the two combine (only STRATA_PIPELINE_ADAPT_ASYNC=0 turns async off); adds the 2x3090 table.
Co-authored-by: adambenhassen <[email protected]>
The function name was everything between `<function=` and `>`, untrimmed. A model that writes a space after the `=`
(seen from Qwen3.8-Flash-Next IQ2_XS in Zed's agent panel) got:
- inside the thinking: the name " capture" matched no declared tool, so the #804 rescue kept the call as reasoning
text; the reply ended with finish_reason "stop" and no tool_calls, and the agent stopped mid-task;
- after the thinking: a call named " capture", which the client does not know, its arguments typed without the
tool's schema.
The name is now stripped where the call is parsed (parse_tool_call), where the #804 rescue checks it against the
declared tools, and where a finished call looks up its schema (_finish_call). A name with surrounding whitespace is
never meaningful, so no text becomes a call that was not one: the #804/#1058 rules still decide whether a call
inside the thinking is an act.
serve/test_tool_name_space.py: both places, streamed and not, fed in 3-character pieces; fails 4/4 before, passes
after. test_reasoning_rescue, test_reasoning_tools and test_frontend unchanged and passing.
NTFS serves unbuffered reads of a mapped file one at a time (WD SN580: 2 MiB reads at QD32
2.35 GB/s mapped vs 3.57 GB/s unmapped). With the file tier unbuffered (--resident-budget-gib
or STRATA_UNBUFFERED_LOAD=1) the mapped view is only a fallback after startup, so it is closed
(STRATA_KEEP_MAPPING=1 keeps it). The prompt path's stager claims runs of consecutive jobs and
reads them with one read_direct batch (STRATA_STAGER_BATCH, default 8 for file-backed blobs).
RTX 5070 Ti, 32 GB RAM, IQ3_XXS, 47.6K prompt, --resident-budget-gib 18 --prefill auto:32768:
1260 -> 1370 tok/s (batch 8) -> 1669 tok/s (view closed).
Ported from #833 onto the rewritten main. Two fixes on the way: unmapped_ is declared on every
platform (mapped_blob reads it outside the Windows block, so Linux did not compile), and
copy_blobs asks resident_blob, which follows the exchange rotation (STRATA_EXCHANGE_ROTATE).
The fast kernels build for CUDA; sm_70 defaults to them, other CUDA cards keep
the plain ones. gr_up_fast_kernel ends with gr_q8_tail like gr_up_multi_kernel,
so STRATA_QFUSE gets its q8_1 image from it too. In the engine (staged read)
only the up projection changes: hc-read0's up 1.08 -> 0.96 ms per window on a
V100-SXM2. fused_gr_bench bitwise, gr_multi_parity OK.
Read off mmvq_il_parity --bench on a V100-SXM2-32GB with the table's own rule
(every measured shape of a class at least 3% faster, else native_mmvq). At 3
columns: the Q5_K head 1094 -> 821 us, Q6_K 12288 rows 68.9 -> 47.9 us, Q4_K
10240 rows 43.9 -> 32.5 us. Bitwise native_mmvq's output (mmvq_il_parity OK).
The AMD grouped-expert layouts (STRATA_EXP_MODE) now build for CUDA too. sm_70
defaults to mode 8 - the codebook grid and the group's q8_1 activations in
shared memory, SwiGLU + q8_1 fused into the gate/up epilogue - every other CUDA
card keeps mode 0. On CUDA, IQ2_XXS gate/up joins the shared-memory kernels,
a format without one falls back to the CUDA layout (not R2), and a strided call
(the verify window's PCIe call) keeps the CUDA layout.
V100-SXM2, setup's IQ2_XS, a verify window's VRAM call (28 groups): IQ2_S
gate/up layers 180 -> 150 us at 3 tokens, IQ2_XXS 170 -> 134 us; bitwise mode
0's output on all 48 layers (native_expert_bench), native_grouped_parity 0
failures in modes 0/2/7/8.
A staged blob (the GGUF role slices assembled into one buffer) is handed to
ExpertCache::fill_slot's cudaMemcpyAsync as-is. From a pageable buffer the
driver stages every copy through an internal bounce buffer - an extra copy at
a fraction of the transfer rate, on the calling thread - so the admit path,
the profile prefill and the lent-slot refills all pay it per expert. Allocate
the pool's buffers with cudaHostAlloc instead, falling back to pageable for a
buffer when the driver refuses (pinned memory is scarce on the small-RAM
machines this path serves).
DETAILS.md's STRATA_PREFILL_CPU_SHARE section: chunks below 3,072 tokens are staged when it is on
(STRATA_PREFILL_CPU_SHARE_MAX=1024 keeps the old limit), the experts the CPU may take (the arena, the
page-locked copy, the mapped experts.bin's pages), the layer split, and a table measured on an RTX 5070 Ti
alone and an RTX 3060 + RTX 5070 Ti split (PCIe 3.0 x8, DDR4-3200, --mmap-experts): 512-token prompts
-49% / -41%, 3,000-token -22% / -24%; a recorded 100K-token coding conversation on the two cards 137.4 ->
130.6 s.
STRATA_PREFILL_CPU_SHARE handed the CPU pool its share of a chunk below 1,024 tokens only, only experts in
page-locked RAM, and only on one GPU. With it set:
- the chunks staged after their routing (the only ones the share applies to) go up to
STRATA_PREFILL_CPU_SHARE_MAX tokens (default 3072, at least 1024) instead of 1024; Prefill::arm_cpu_share
sets the limit before any loan is sized, and init sets it too;
- an expert whose blob the CPU reads from RAM without assembling it (the arena's unpinned part, the mapped
experts.bin's pages) can go to the CPU, not only a page-locked one; the blob pointers are taken when the
experts are chosen, and one without a stable blob stays on the GPU;
- on a layer split every stage has the pool (serve, no --batch), and one stage at a time takes it for a
chunk while the other reads its chunk on the GPU alone.
Unset, nothing changes.
auto balanced the CPU's time per expert against the GPU's per streamed expert, both measured with the share on, so
nothing compared a layer with and without it. On jase100k's RX 7900 GRE + Ryzen 7 5700X3D the host's time per
streamed expert rose 107 -> 200 us with auto (0.47): the streaming side saved nothing (0.53 x 1.87 = 0.99) and the
CPU's work came on top, -4.5%. Now every eligible layer is timed with CUDA events from before its routing sync to its
combine, per non-resident expert; adjacent layers alternate no share / the measured share until three ratios are in,
then auto shares while the median of the last five is below 1, with one layer in 29 in the other arm. Layers with a
one-time cost and the first layer are left out. STRATA_DBG_CPU_GATE=1 prints each measurement.
RTX 5090 + 9950X3D, IQ2_XS, 250 / 600 / 900 tokens against off: DMA streaming -9.8/-9.9/-10.4% -> -8.8/-9.1/-10.5%;
an emulated host-copied stream -18.0/-16.7/-14.4% -> -16.1/-15.2/-13.0%; one core + one worker (where the old auto
lost up to +11.5%) +1.4..+2.4% -> +2.0..+2.3%, the cost of the first measurements. Unset: byte-identical.
The parking path has two gates in series: make_room() balances the cache's own budget, then conversation_memory_admit() checks the host's free RAM against the snapshot plus --conversation-cache-min-free-mib. The second was final: a failure dropped the snapshot even though the cache held conversations whose RAM it could have, and the next request read its whole prompt again (270k tokens, 96 s, measured by the reporter in #1347).
Evict the least recently active parked conversations until the gate admits the snapshot, bounded by the slot count; entries that hold a pinned shared prefix are not evicted. The failure line says how many were evicted and how many remain.
Reported with the fix by alanthinker in #1347.
The 16 dense GEMM geometries a Qwen3.8-Flash-Next prompt logs on a Radeon 890M (gfx1150, Strix Point) at
T = 64 .. 16384, 144 rows, calibrated with tools/hip/tune_hipblaslt.cpp against AMD's TheRock ROCm 7.14.1 for
gfx1150 (hipBLASLt 1.4.1 cd957402). Two passes back to back; each row keeps the solution with the lowest median of
all six repetitions (a single repetition stalled now and then, and the tool's mean-of-three picks differed in 26 rows
between the passes).
Every row beats plain hipBLAS, 1.55-22.6x per GEMM. End to end on GSQ-RCO IQ3_XXS, three interleaved rounds: prompts
read 1.5-1.7x faster (3.6K tokens 124 -> 216 tok/s, 7K 130 -> 226 tok/s), decode unchanged; hip_prefill_hipblaslt_gemm
passes with it and a server run logged no fallbacks. Setup does not install for gfx1150, so the table is set by hand
(STRATA_HIPBLASLT_TUNING). Method and data: bench/results/2026-10-07-community-gfx1150.
A verify window's graph waits on the GPU for flags the host raises once
cudaGraphLaunch has returned. A blocking launch returns only when the
graph has finished, so with CUDA_LAUNCH_BLOCKING=1 the first window after
a prompt never ends and the watchdog stops the engine, whatever the prompt
length, GPU count or --pipeline-windows. The engine now says so at start,
and the watchdog's line names it. Nothing changes without the variable.
A pipelined group's window spans its slots up to its last busy one, and the engine keeps no pipelined slot as a
conversation cache, so the least-recently-used pick left leading holes ([_ B]) that cost a whole row each. pick_slot
now fills such a hole first, then follows the spreading order. The engine reports INFO batch_groups so the server
knows the groups really run pipelined. serve/test_slot_alloc.py covers the allocator.
Co-authored-by: blange48 <[email protected]>
The miss cost (190 ms per unit of routed mass, fitted on x16 links) is one
constant for every stage, so --layer-split auto cannot see an asymmetric
pair: on 2x 4090 D with CUDA1 on a physical x1 link (1.6 GB/s measured) it
gives that card 26 layers (K=22) and reads a 29K prompt at ~350 tok/s.
Probe each stage's link inside the search (probe_pcie_h2d_gbps; the
stage-setup probes are skipped when --pcie-frac is given, and the search
needs the reading either way) and scale that stage's miss term by
clamp(20 GB/s / bw, 1, 64). 20 GB/s is pcie_frac_for_gbps' x16 reference;
the lower clamp keeps a symmetric machine on the exact formula the sweep's
K was fitted with (every scale 1.0 -> the objective is unchanged).
With it, auto picks the fit-in-cache boundary on that machine (K=38: the
narrow card's cache holds all of its layers' pairs, zero streaming there)
and reads the same prompt at ~1410 tok/s, ~4x.
The 190 ms mixes the decode CPU-pool cost with the prompt link cost;
scaling the whole per-stage term is the first-order fix (splitting the
constant would need a fresh fit on both a narrow and a wide link).
STRATA_PF_FUSED=1 reaches only chunks of stream_all_min() tokens or more (1,024, measured on a discrete card); a
smaller read runs its experts through MMQ. On Strix Halo every expert is resident and MMQ is ~3x slower on a
~1,000-token read, the size of an agent's turn: 441 / 855-token reads 3.1 / 4.0 s -> 0.9 / 1.2 s, a 12-turn agent
replay 3.4 -> 2.1 s per turn (C2T8, --no-prefill-borrow). It changes bits (the fused path's output, which reads of
1,024+ tokens already get), so section 5: teacher-forced KL against today's path 0.016 on reads under 1,024, the same
as fused vs MMQ on longer reads (0.019); reads of 1,024+ unchanged (KL 0).
Setup installs the ROCm 7.14.0a20260608 wheels for gfx1151 since 0.1.40.2 (#1267); their hipBLASLt is 1.4.0
(100400), which neither gfx1151 table matched, so setup's engine read the prompt through plain hipBLAS.
tune_hipblaslt --workspace-mib 32 on the 100401 table's 90 shapes, on a Ryzen AI Max+ 395 (Radeon 8060S, gfx1151,
128 GB, Fedora 44, kernel 7.2.8) with those wheels: 3.1-14.1x faster than plain hipBLAS per shape (median 6.2x).
hip_prefill_hipblaslt_gemm and hip_prefill_hcd_exact_parity pass with it (fallbacks=0); HIP ctest 61 of 62
(ple_parity needs the Q2_0 fixture). setup picks it up with no code change.
Below N of N lendable slots the uncovered experts are read from the pack during a long prompt (reporter: 18% uncovered = -26% prompt speed on gfx1100). A WARNING line now says so, and docs/AMD_HIP.md documents the headroom knob and the levers from the report. No behaviour change otherwise.
The gated delta rule in the WY form flash-linear-attention uses, all FP32: gdn_chunk_prep_kernel forms k.k / q.k and
T = (I + A)^-1 for every chunk of a 2048-token super-block at once; gdn_chunk_scan_kernel (128 blocks, a key head x
16 value columns each) walks the chunks with its state slice in registers. Every exponent <= 0 (the gates are
negative). The scratch (25.6 MB) is allocated once per device: freed and reallocated per call it cost more host time
than the kernels saved on short prompts. sm_80+, >= 128 SMs, enough shared memory, chunks of >= 128 tokens; else
gdn_rec_kh_kernel. Unset: unchanged.
gdn_rec_parity: against gdn_rec_kh_kernel <= 1.13e-6 (T = 1..4099, 8192, 32768, a non-zero starting state); against
an FP64 recurrence <= 1.05e-6 (the sequential kernel <= 3.2e-7) - FP32 rounding; 199 restarts of 7 shapes bit for
bit (NaN-filled scratch between). RTX 5090: 1.81x the recurrence at 2K-32K; IQ2_XS: the recurrence phase of a 32K
prompt 520-532 -> 306-334 ms (-210 ms, -3.7%), 8K -2.1..-3.6%; first-token KL 0.0016-0.0152, 1.15-1.83x a harmless
control arm (STRATA_PROMPT_ATTN_V1), the same top token.
The MoE input and the routing weights go to host memory on a stream beside the primary's compute stream, so its
own experts start at once instead of behind the copies over its link (mixed_h and w are rewritten only after this
layer's combine, which waits for the peer). STRATA_PF_PEER_XSTREAM=0: the compute stream. With
STRATA_PF_PEER_RING=160, prompt tok/s: 100K 1,854 -> 1,918; 8K and 3.4K unchanged.
Each group's tokens and their row lists (in the experts' order) are built on the host from the rows' pairs and
uploaded once per layer; peer_gather_add is eddoursul/Strata's moe_gather_add with the weight looked up through
the row's pair, so the sums are bitwise those of the per-expert launches. STRATA_PF_PEER_GATHER=0: per expert.
Prompt tok/s against the per-expert launches: 3.4K 825 -> 832, 8K 1,468 -> 1,500, 100K 1,821 -> 1,858.
Co-Authored-By: Eddoursul <[email protected]>
The MoE input goes out as the FFN input's existing FP16 image (mixed_h) and the per-token sums come back as
FP16, saturated at +-65504 (eddoursul/Strata f8de703's FP16 input, 2acfac4's sums_to_f16): half the bytes over
the primary's link both ways. STRATA_PF_PEER_F16=0: FP32.
Peer stream 0.6, prompt tok/s: 3.4K 805 -> 831, 8K 1,284 -> 1,495, 100K 1,578 -> 1,855 (the share off: 1,712).
The 8K and 3.4K answers stay byte-identical to the share off.
Co-Authored-By: Eddoursul <[email protected]>
The design is eddoursul/Strata's second-GPU prompt path (f8de703, 2acfac4, e8ade66): the second GPU adds its
experts' rows into each token's weighted sum and the first GPU's combine adds that to its own experts' rows.
Over the host route the peer scatter-adds its rows into per-token sums (peer_scatter_add, after the fork's
moe_scatter_add: one launch per expert in a fixed order, so the sums are repeatable) and sends T x N floats
back instead of rows_peer x N (up to K = 10 times more); moe_combine_peer adds them to the primary's rows. The
2a row buffers are not allocated (3.4 GB of pinned RAM at 32K-token chunks). STRATA_PF_PEER_SUMS=0: rows.
Same setup, peer stream 0.6, prompt tok/s against the share off: 3.4K 332 -> 805, 8K 762 -> 1,284, 100K
1,712 -> 1,578. The 8K and 3.4K answers are byte-identical to the share off (greedy, 256 / 3,221 tokens).
Co-Authored-By: Eddoursul <[email protected]>
set_peer refused a peer that cannot access the primary directly, so on GPU pairs without P2P (most GeForce
pairs) the prompt path stayed on the primary GPU. Without P2P the share now takes the route the layer split's
helper already uses: the activations and the result rows through mapped pinned host buffers read by copy
kernels, with the compact group buffers (the layer-sized path copies over P2P only). STRATA_PF_PEER_HOST=0
restores the refusal.
RTX 3090 (PCIe 4.0 x4) + RTX 3080 Ti (x16), no P2P, UD-Q4_K_XL, --prefill auto:32768, peer stream 0.35 / 1.0,
prompt tok/s against the share off: 3.4K 332 -> 492 / 551, 8K 763 -> 907 / 776, 100K 1,713 -> 1,072 / 734.
The peer's rows coming back over the primary's x4 link cost more than they save on long prompts; the next
commit sends sums instead.
- tools/check_isa_guard.py disassembles the CPU kernels (objdump + nm on the Linux kernel library, capstone + the /MAP
file of a linked binary on Windows) and fails when a function holding AVX-512 / AVX-VNNI-class instructions is
not a kernel of a wide-ISA file: a weak or COMDAT copy (the linker may pick it for every caller), a std:: /
template instantiation, a probe or helper that every CPU runs, or code in another object. MSVC's own
`__isa_available` vectorizer guard is accepted. ctest isa_guard_check runs it; exit 77 (skip) when the tools
are missing.
- -DSTRATA_SDE=<sde> adds sde_{hsw,adl}_* tests: the CPU parity tests on an emulated Haswell (AVX2 only) and Alder
Lake (AVX-VNNI, no AVX-512); SDE stops at the first instruction the chip lacks.
- tools/test_check_isa_guard.py: unit tests of the policy.
expert.cpp and iq_avx512.cpp are compiled for AVX-512, iq_avx2.cpp and kq_avx2.cpp for AVX2, and a file compiled
that way may use the wide ISA anywhere in its code. Code that every CPU runs was still defined in them:
- cpu_features, CpuFeatures::reason, cpu_require_expert_support (the function that finds out whether the CPU has
AVX-512) and expert_set_oracle_q8_0 / expert_oracle_q8_0_enabled (the first call of every generate, on any CPU)
were in expert.cpp;
- iq512_supported, iq256_supported and kq256_supported (a comparison native_gu_rows asks before it has checked
what the CPU can run) were in the kernel files; iq512_supported was called on every AVX2-only CPU for every
gate/up call.
They now live in expert_layout.cpp and native_expert.cpp, which are built for the baseline. No kernel changes;
the compiled code of the moved functions is identical, so default output is byte-identical.
2026-10-07 20:23:53 +02:00
1303 changed files with 276077 additions and 3411 deletions
# below refuses anything older: silently building for a pre-Ampere arch would produce a binary that runs
# and is wrong about its own performance, which is the worst of both outcomes.
cmake_minimum_required(VERSION3.24)
project(strataVERSION0.1.40.3LANGUAGESCXX)
project(strataVERSION0.1.45LANGUAGESCXX)
# EXPERIMENTAL: the Intel Arc engine, the SYCL port in sycl/ (PR #423; icpx + oneMKL, docs/INTEL_ARC.md). It is its
# own project that takes the unported sources from src/; with this on, nothing below is configured.
option(STRATA_ENABLE_SYCL"EXPERIMENTAL: build the Intel Arc (SYCL) engine from sycl/ - docs/INTEL_ARC.md"OFF)
@@ -70,18 +70,30 @@ if(STRATA_HIP_GFX906)
endif()
option(STRATA_ENABLE_HIP"Build the HIP targets for wave32 AMD GPUs (gfx1100, gfx1201; gfx1012/gfx1030 unvalidated)"OFF)
option(STRATA_PREFILL_MMQ"Build the optional GGML MMQ prefill path for HIP"OFF)
# CUDA: MMQ prompt kernels for Q4_K / Q5_K / Q5_1 (Unsloth's UD-Q4_K_XL experts). Off by default: the engine loads
# every kernel when it starts (CUDA_MODULE_LOADING=EAGER), so these took ~5 MB of VRAM - 1-3 expert slots - from every
# model, and the UD-Q4_K_XL prompt is SSD-bound with or without them (docs/UNSLOTH_Q4.md)
option(STRATA_MMQ_KQUANTS"Build the MMQ prompt kernels for Q4_K / Q5_K / Q5_1 / Q6_K: the CUDA experts of Unsloth's UD-Q4_K_XL, and on HIP the dense GGUF projections (STRATA_DENSE_MMQ)"OFF)
# CUDA: Q6_K gate/up in the grouped expert kernels (Unsloth's UD-Q6_K_XL, docs/UNSLOTH_Q6.md). Off by default: every
# grouped-kernel instance is loaded when the engine starts (CUDA_MODULE_LOADING=EAGER), so one more costs VRAM that
# every other model would lose as expert slots (PR #860 review)
option(STRATA_Q6K_EXPERTS"Build the Q6_K gate/up grouped expert kernels (UD-Q6_K_XL)"OFF)
# MMQ prompt kernels for Q4_K / Q5_K / Q5_1 / Q6_K (Unsloth's UD-Q4/Q5/Q6_K_XL experts; on HIP also the dense GGUF
# projections, STRATA_DENSE_MMQ). On by default since MODEL-UPDATE Phase 1: on CUDA they are their own library
# (strata_kq, loaded only for a model with these formats: the engine loads every kernel of its own binary at start,
# CUDA_MODULE_LOADING=EAGER, and these took ~3.8 MB of VRAM - 3 IQ3_XXS expert slots on a P100 - from every model); on
# HIP a code object loads on first use, so they cost other models nothing there (R9700: the same slots). Used at run
# time with STRATA_MMQ_KQ=1 (opt-in: it changes the prompt's rounding against the FP16 path).
option(STRATA_MMQ_KQUANTS"Build the MMQ prompt kernels for Q4_K / Q5_K / Q5_1 / Q6_K: the experts of Unsloth's K-quant files, and on HIP the dense GGUF projections (STRATA_DENSE_MMQ)"ON)
# Q6_K gate/up in the grouped expert kernels (Unsloth's UD-Q6_K_XL, and layer 2 of UD-Q5_K_XL; docs/QUANTS.md). On by
# default since MODEL-UPDATE Phase 1: one more instance per grouped kernel in iq_kernels.cu, +19 KB of sm_60 code,
# measured to leave every expert-cache slot count as it was (P100, R9700). -DSTRATA_Q6K_EXPERTS=OFF drops it.
option(STRATA_Q6K_EXPERTS"Build the Q6_K gate/up grouped expert kernels (UD-Q6_K_XL, UD-Q5_K_XL)"ON)
if(STRATA_Q6K_EXPERTS)
add_compile_definitions(STRATA_Q6K_EXPERTS=1)
endif()
option(STRATA_ORCA_Q4KS_MMQ"Build the GGML Q5_0 MMQ instance for OrcaRouter Q4_K_S"OFF)
# CUDA: the pointer-table MMQ behind STRATA_MMQ_DIRECT=1 (the prompt path's MMQ reads resident experts in their cache
# slots through a Strata-side copy of llama.cpp's MMQ scheduler, src/prefill/moe_mmq_tab.cuh). Off by default, like
# the K-quant options above: the engine loads every kernel when it starts (CUDA_MODULE_LOADING=EAGER), and these nine
# instances (+15 MB of binary) take 36-38 MiB of VRAM per GPU on sm_86 - expert slots every model would lose, with
# STRATA_MMQ_DIRECT unset too. They also add ~70% of the upstream MMQ instances' compile time per CUDA arch (175 s
# of CPU for the nine at sm_86 alone). Off, the build's device code is byte for byte the one without this feature. CUDA
# only (not HIP, not gfx906); switched off at configure time when the llama.cpp tree's mmq.cuh is not one the copy
# was verified against.
option(STRATA_MMQ_TAB"Build the pointer-table prompt MMQ for STRATA_MMQ_DIRECT=1 (CUDA; ~37 MiB VRAM per GPU)"OFF)
@@ -92,6 +92,8 @@ startet er es und sagt dir, wie du deine Apps verbindest. KI-Tools können Strat
[Lade Strata herunter](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) und entpacke es (oder nutze
`git clone`). **Windows:** Doppelklick auf **`START-HERE.bat`**. **Linux:** Führe im Strata-Ordner **`./setup.sh`** aus.
**Mac (Apple Silicon, M1 bis M5, macOS 15 oder neuer):** Installiere einmal Apples Command Line Tools (`xcode-select --install` im Terminal; Xcode selbst brauchst du nicht). Öffne dann das Terminal im Strata-Ordner und führe **`./setup.sh`** aus. Es baut die Mac-Engine auf deinem Mac (etwa 7 Minuten), stellt dieselben Fragen, lädt das Modell und startet es. Beim nächsten Mal startet `./setup.sh` sofort. Welche Größe zu deinem Mac passt und wie schnell es ist: [docs/MAC.md](docs/MAC.md).
Die Schritte sind für NVIDIA und AMD gleich. Der Installer erkennt deine Karte und richtet die passende Engine
dafür ein. Er stellt dir ein paar Fragen:
@@ -133,9 +135,10 @@ stark komprimiert. Kleinere Größen sind schneller. Größere sind etwas klüge
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** die ~4-Bit-Version von Unsloth, in der Qualität
zwischen IQ3_S und UD-Q4_K_XL. 94 GB Download. Mit weniger als ~80 GB RAM liest Strata einen Teil davon während
der Antwort von der SSD, dort ist es also langsamer (eine NVMe-SSD hilft).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (experimentell): am nächsten am vollen
Modell. Aber Strata liest das meiste davon während der Antwort von der SSD, deshalb schreibt es auf einem PC mit
64 GB nur 7-8.5 Tokens/s.
- **[Unsloth UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL und Q8_0](docs/MODELS.md#unsloths-files-ud-iq4_xs-to-q8_0):** 4 bis 8 Bit, näher am vollen Modell
(93-95 % das gleiche nächste Token wie Q8_0). Schnell, wenn alle Experten in den RAM passen (~88 GB für UD-Q4_K_XL
bis ~136 GB für Q8_0); mit weniger RAM liest Strata einen Teil davon während der Antwort von der SSD, was viel
langsamer ist (7-8.5 Tokens/s für UD-Q4_K_XL auf einem PC mit 64 GB).
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** Das richtest du von Hand
### Optional: mehr Tempo mit einer Grafikkarte, wenn das ganze Modell in den RAM passt
Zwei Opt-in-Einstellungen (von RubenTorney, #1904 und #1905). Sie helfen, wenn du eine Grafikkarte nutzt und dein RAM alle Experten hält (Stratas Arena-Modus, zum Beispiel Q2_0 oder IQ3_XXS auf einem PC mit 64 GB). Öffne deine Konfiguration `strata-<model>.json` im Strata-Datenordner ([wo](docs/INSTALL.md#where-things-are-stored)) und füge hinzu:
- **Schnellere Ausgabe:** `"--adapt-async", "1"` in die Liste `"args"`. Der Experten-Cache wird dann in einem Hilfs-Thread aktualisiert, statt die Antwort anzuhalten. Gemessen auf einer RTX 5070 (Windows, Q2_0): +4-5 % Ausgabe, +7-10 % bei Code und langen Dokumenten. Antworten können leicht vom Standard abweichen (sie bleiben von Lauf zu Lauf gleich).
- **Schnelleres Prompt-Lesen unter Windows:** `"STRATA_PIN_BY_NEED": "1"` im Block `"env"`. Windows kann nur etwa den halben RAM für die Grafikkarte sperren; damit werden zuerst die Experten gesperrt, die die Karte wirklich braucht. Gemessen auf der RTX 5070: +4,4 % Prompt-Lesen.
```json
"args":[...,"--adapt-async","1"],
"env":{"STRATA_PIN_BY_NEED":"1"}
```
Starte Strata danach neu (`START-HERE.bat` oder `./setup.sh`). Zum Rückgängigmachen entferne die beiden Einträge.
## Etwas ist schiefgelaufen?
- **Mein PC ist beim ersten Start von Strata eingefroren.** Das ist normal, während das Modell lädt. Warte und
@@ -90,6 +90,8 @@ cómo conectar tus aplicaciones. Las herramientas de IA también pueden instalar
[Descarga Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) y descomprímelo (o usa `git clone`).
**Windows:** haz doble clic en **`START-HERE.bat`**. **Linux:** ejecuta **`./setup.sh`** en la carpeta de Strata.
**Mac (Apple Silicon, M1 a M5, macOS 15 o posterior):** instala una vez las Command Line Tools de Apple (`xcode-select --install` en Terminal; no hace falta Xcode). Luego abre Terminal en la carpeta de Strata y ejecuta **`./setup.sh`**. Compila el motor de Mac en tu Mac (unos 7 minutos), hace las mismas preguntas, descarga el modelo y lo arranca. La próxima vez, `./setup.sh` arranca enseguida. Qué tamaño cabe en tu Mac y qué velocidad tiene: [docs/MAC.md](docs/MAC.md).
Los pasos son los mismos para NVIDIA y AMD. El instalador detecta tu tarjeta y prepara el motor adecuado para ella.
Te hace unas pocas preguntas:
@@ -130,9 +132,10 @@ tamaños pequeños son más rápidos. Los grandes son algo más inteligentes.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** la versión de ~4 bits de Unsloth, entre IQ3_S y
UD-Q4_K_XL en calidad. Una descarga de 94 GB. Con menos de ~80 GB de RAM, Strata lee una parte desde el SSD
mientras responde, así que ahí es más lento (un SSD NVMe ayuda).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (experimental): el más parecido al modelo
completo. Pero Strata lee la mayor parte desde el SSD mientras responde, así que solo escribe 7-8.5 tokens/s en un
PC con 64 GB.
- **[Unsloth UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL y Q8_0](docs/MODELS.md#unsloths-files-ud-iq4_xs-to-q8_0):** de 4 a 8 bits, más cerca del modelo completo
(93-95 % el mismo token siguiente que Q8_0). Rápidos cuando todos los expertos caben en la RAM (~88 GB para
UD-Q4_K_XL hasta ~136 GB para Q8_0); con menos RAM, Strata lee una parte desde el SSD mientras responde, lo que es
mucho más lento (7-8.5 tokens/s para UD-Q4_K_XL en un PC con 64 GB).
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** se instala a mano. No está en
el menú del instalador.
@@ -164,6 +167,20 @@ Tamaños, descargas y qué cabe dónde: [docs/MODELS.md](docs/MODELS.md). Para a
Más: [dónde se guardan tus chats](docs/INSTALL.md#where-things-are-stored), [la API](docs/DETAILS.md#using-it).
### Opcional: más velocidad con una sola tarjeta cuando todo el modelo cabe en la RAM
Dos ajustes opcionales (de RubenTorney, #1904 y #1905). Ayudan si usas una tarjeta gráfica y tu RAM guarda todos los expertos (el modo arena de Strata, por ejemplo Q2_0 o IQ3_XXS en un PC con 64 GB). Abre tu configuración `strata-<model>.json` en la carpeta de datos de Strata ([dónde](docs/INSTALL.md#where-things-are-stored)) y añade:
- **Generación más rápida:** `"--adapt-async", "1"` en la lista `"args"`. La caché de expertos se actualiza entonces en un hilo auxiliar en vez de pausar la respuesta. Medido en una RTX 5070 (Windows, Q2_0): +4-5 % de generación, +7-10 % en código y documentos largos. Las respuestas pueden diferir un poco de las por defecto (son iguales de una ejecución a otra).
- **Lectura del prompt más rápida en Windows:** `"STRATA_PIN_BY_NEED": "1"` en el bloque `"env"`. Windows solo puede bloquear cerca de la mitad de la RAM para la tarjeta; esto bloquea primero los expertos que la tarjeta necesita de verdad. Medido en la RTX 5070: +4,4 % en la lectura del prompt.
```json
"args":[...,"--adapt-async","1"],
"env":{"STRATA_PIN_BY_NEED":"1"}
```
Reinicia Strata (`START-HERE.bat` o `./setup.sh`) después del cambio. Para deshacerlo, quita las dos entradas.
## ¿Algo salió mal?
- **Mi PC se congeló la primera vez que arrancó Strata.** Es normal mientras carga el modelo. Espera y no cierres la
@@ -90,6 +90,8 @@ arrêter Strata grâce à son [serveur MCP](docs/MCP_SERVER.md).
[Téléchargez Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) et décompressez-le (ou faites un `git clone`).
**Windows :** double-cliquez sur **`START-HERE.bat`**. **Linux :** lancez **`./setup.sh`** dans le dossier Strata.
**Mac (Apple Silicon, M1 à M5, macOS 15 ou plus récent) :** installez une fois les Command Line Tools d'Apple (`xcode-select --install` dans le Terminal ; Xcode lui-même n'est pas nécessaire). Ouvrez ensuite le Terminal dans le dossier Strata et lancez **`./setup.sh`**. Il compile le moteur Mac sur votre Mac (environ 7 minutes), pose les mêmes questions, télécharge le modèle et le démarre. La fois suivante, `./setup.sh` démarre tout de suite. Quelle taille convient à votre Mac et sa vitesse : [docs/MAC.md](docs/MAC.md).
Les étapes sont les mêmes pour NVIDIA et AMD. L'installateur détecte votre carte et installe le bon moteur pour elle.
Il vous pose quelques questions :
@@ -130,9 +132,10 @@ compressées. Les petites tailles sont plus rapides. Les grandes sont un peu plu
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs)** : la version ~4 bits d'Unsloth, entre IQ3_S et
UD-Q4_K_XL en qualité. Un téléchargement de 94 Go. Avec moins de ~80 Go de RAM, Strata en lit une partie depuis
le SSD pendant qu'il répond, il y est donc plus lent (un SSD NVMe aide).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (expérimental) : le plus proche du modèle
complet. Mais Strata en lit la plus grande partie depuis le SSD pendant qu'il répond, donc il n'écrit que
7-8.5 tokens/s sur un PC avec 64 Go.
- **[Unsloth UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL et Q8_0](docs/MODELS.md#unsloths-files-ud-iq4_xs-to-q8_0) :** de 4 à 8 bits, plus proches du modèle
complet (93-95 % le même token suivant que Q8_0). Rapides quand tous les experts tiennent dans la RAM (~88 Go pour
UD-Q4_K_XL jusqu'à ~136 Go pour Q8_0) ; avec moins de RAM, Strata en lit une partie depuis le SSD pendant qu'il
répond, ce qui est bien plus lent (7-8.5 tokens/s pour UD-Q4_K_XL sur un PC avec 64 Go).
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs) :** vous l'installez à la main.
Plus d'infos : [où sont stockées vos discussions](docs/INSTALL.md#where-things-are-stored), [l'API](docs/DETAILS.md#using-it).
### Optionnel : plus de vitesse sur une seule carte quand tout le modèle tient en RAM
Deux réglages optionnels (de RubenTorney, #1904 et #1905). Ils aident si vous utilisez une seule carte graphique et que votre RAM contient tous les experts (le mode arena de Strata, par exemple Q2_0 ou IQ3_XXS sur un PC de 64 Go). Ouvrez votre configuration `strata-<model>.json` dans le dossier de données de Strata ([où](docs/INSTALL.md#where-things-are-stored)) et ajoutez :
- **Génération plus rapide :** `"--adapt-async", "1"` dans la liste `"args"`. Le cache d'experts est alors mis à jour dans un thread auxiliaire au lieu de mettre la réponse en pause. Mesuré sur une RTX 5070 (Windows, Q2_0) : +4-5 % de génération, +7-10 % sur le code et les longs documents. Les réponses peuvent légèrement différer de celles par défaut (elles restent identiques d'une exécution à l'autre).
- **Lecture du prompt plus rapide sous Windows :** `"STRATA_PIN_BY_NEED": "1"` dans le bloc `"env"`. Windows ne peut verrouiller qu'environ la moitié de la RAM pour la carte ; ce réglage verrouille d'abord les experts dont la carte a vraiment besoin. Mesuré sur la RTX 5070 : +4,4 % en lecture du prompt.
```json
"args":[...,"--adapt-async","1"],
"env":{"STRATA_PIN_BY_NEED":"1"}
```
Redémarrez Strata (`START-HERE.bat` ou `./setup.sh`) après la modification. Pour annuler, retirez les deux entrées.
## Un problème ?
- **Mon PC s'est figé au premier démarrage de Strata.** C'est normal pendant le chargement du modèle. Attendez, et ne
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata は無料です。あなたの PC でうまく動いたら、コーヒー 1 杯の支援が開発を続ける力になります。</sub></p>
<sub>Strata は無料で使えます。お使いの PC で快適に動いたら、コーヒー 1 杯分のご支援が開発を続ける励みになります。</sub></p>
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
- **Intel Arc**, built from source on Linux: [Intel Arc](docs/INTEL_ARC.md).
- **Apple Silicon Macs (M1 to M5)**, built from source by `./setup.sh` (the Metal engine): [Mac](docs/MAC.md).
- **AMD Ryzen AI Max (Strix Halo)**, built from source on Linux: [Strix Halo](docs/STRIX_HALO.md).
- **Older processors without AVX2**: they work, but slowly. [Older CPUs](docs/INSTALL.md#older-cpus-experimental).
@@ -90,6 +91,12 @@ you how to connect your apps. AI tools can also install, start and stop Strata t
[Download Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or `git clone` it).
**Windows:** double-click **`START-HERE.bat`**. **Linux:** run **`./setup.sh`** in the Strata folder.
**Mac (Apple Silicon, M1 to M5, macOS 15 or newer):** install Apple's Command Line Tools once
(`xcode-select --install` in Terminal; Xcode itself is not needed). Then open Terminal in the Strata folder and run
**`./setup.sh`**. It builds the Mac engine on your Mac (about 7 minutes), asks the same questions, downloads the model
and starts it. Next time, `./setup.sh` starts it right away. Which size fits your Mac and how fast it is:
[docs/MAC.md](docs/MAC.md).
The steps are the same for NVIDIA and AMD. The installer finds your card and sets up the right engine for it. It
asks you a few questions:
@@ -109,6 +116,8 @@ download stops, run it again: it continues where it left off. Your browser opens
its window to stop the model. `UPDATE.bat` (`./update.sh`) updates Strata without starting it. Updating, Docker,
several cards, where the files go and every option: [docs/INSTALL.md](docs/INSTALL.md).
**Is it fast on your PC?** Run `BENCH.bat` (`./bench.sh`): about 5-10 minutes, it measures chat, 4K/16K prompts and an agent-style session with your own settings and writes a report to share ([how](docs/COMMUNITY_BENCHMARKS.md#run-the-benchmark)).
## Which model should I pick?
The installer recommends one for your RAM. The same model comes in several sizes, compressed more or less. Smaller
@@ -129,8 +138,10 @@ sizes are faster. Larger sizes are a bit smarter.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** Unsloth's ~4-bit version, between IQ3_S and
UD-Q4_K_XL in quality. A 94 GB download. With less than ~80 GB of RAM, Strata reads part of it from the SSD
while it answers, so it is slower there (an NVMe SSD helps).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (experimental): the closest to the full
model. But Strata reads most of it from the SSD while it answers, so it writes only 7-8.5 tokens/s on a 64 GB PC.
- **[Unsloth UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL and Q8_0](docs/MODELS.md#unsloths-files-ud-iq4_xs-to-q8_0):**
4- to 8-bit, closer to the full model (93-95% the same next token as Q8_0). Fast where every expert fits in RAM
(~88 GB for UD-Q4_K_XL up to ~136 GB for Q8_0); with less RAM Strata reads part of them from the SSD while it
answers, which is much slower (7-8.5 tokens/s for UD-Q4_K_XL on a 64 GB PC).
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** you set it up by hand. It is
not in the installer's menu.
@@ -162,6 +173,26 @@ Sizes, downloads and what fits where: [docs/MODELS.md](docs/MODELS.md). To add a
More: [where your chats are stored](docs/INSTALL.md#where-things-are-stored), [the API](docs/DETAILS.md#using-it).
### Optional: more speed on one card when the whole model fits in RAM
Two opt-in settings (by RubenTorney, #1904 and #1905). They help when you run one graphics card and your RAM holds
every expert (Strata's arena mode, for example Q2_0 or IQ3_XXS on a 64 GB PC). Open your config,
`strata-<model>.json` in the Strata data folder ([where](docs/INSTALL.md#where-things-are-stored)), and add:
- **Faster decode:** `"--adapt-async", "1"` to the `"args"` list. The expert cache then updates on a helper thread
instead of pausing the answer. Measured on an RTX 5070 (Windows, Q2_0): +4-5% decode, +7-10% on code and long
documents. Answers can differ slightly from the default (they stay the same from run to run).
- **Faster prompt reading on Windows:** `"STRATA_PIN_BY_NEED": "1"` in the `"env"` block. Windows can lock only about
half the RAM for the graphics card; this locks the experts the card actually needs first. Measured on the RTX 5070:
+4.4% prompt reading.
```json
"args":[...,"--adapt-async","1"],
"env":{"STRATA_PIN_BY_NEED":"1"}
```
Restart Strata (`START-HERE.bat` or `./setup.sh`) after the change. To undo, remove the two entries.
## Something went wrong?
- **My PC froze the first time Strata started.** This is normal while it loads the model. Wait, and don't close the
@@ -90,6 +90,8 @@ conectar seus apps. Ferramentas de IA também podem instalar, iniciar e parar o
[Baixe o Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) e descompacte (ou use `git clone`).
**Windows:** clique duas vezes em **`START-HERE.bat`**. **Linux:** rode **`./setup.sh`** na pasta do Strata.
**Mac (Apple Silicon, M1 a M5, macOS 15 ou mais recente):** instale uma vez as Command Line Tools da Apple (`xcode-select --install` no Terminal; o Xcode em si não é necessário). Depois abra o Terminal na pasta do Strata e rode **`./setup.sh`**. Ele compila o motor do Mac no seu Mac (uns 7 minutos), faz as mesmas perguntas, baixa o modelo e o inicia. Da próxima vez, `./setup.sh` inicia na hora. Qual tamanho cabe no seu Mac e qual a velocidade: [docs/MAC.md](docs/MAC.md).
Os passos são os mesmos para NVIDIA e AMD. O instalador encontra sua placa e configura o engine certo para ela.
Ele faz algumas perguntas:
@@ -130,9 +132,10 @@ Os tamanhos menores são mais rápidos. Os maiores são um pouco mais inteligent
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** a versão de ~4 bits da Unsloth, entre IQ3_S e
UD-Q4_K_XL em qualidade. Um download de 94 GB. Com menos de ~80 GB de RAM, o Strata lê uma parte dele do SSD
enquanto responde, então ali ele fica mais lento (um SSD NVMe ajuda).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (experimental): o mais próximo do modelo
completo. Mas o Strata lê a maior parte dele do SSD enquanto responde, então ele escreve só 7-8.5 tokens/s num PC
com 64 GB.
- **[Unsloth UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL e Q8_0](docs/MODELS.md#unsloths-files-ud-iq4_xs-to-q8_0):** de 4 a 8 bits, mais próximos do modelo
completo (93-95% o mesmo próximo token que o Q8_0). Rápidos quando todos os especialistas cabem na RAM (~88 GB para
o UD-Q4_K_XL até ~136 GB para o Q8_0); com menos RAM, o Strata lê uma parte do SSD enquanto responde, o que é bem
mais lento (7-8.5 tokens/s para o UD-Q4_K_XL num PC com 64 GB).
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** você configura à mão. Ele
não está no menu do instalador.
@@ -165,6 +168,20 @@ Tamanhos, downloads e o que cabe onde: [docs/MODELS.md](docs/MODELS.md). Para ad
Mais: [onde suas conversas ficam salvas](docs/INSTALL.md#where-things-are-stored), [a API](docs/DETAILS.md#using-it).
### Opcional: mais velocidade com uma placa quando o modelo inteiro cabe na RAM
Duas configurações opcionais (de RubenTorney, #1904 e #1905). Elas ajudam quando você usa uma placa de vídeo e sua RAM guarda todos os experts (o modo arena do Strata, por exemplo Q2_0 ou IQ3_XXS num PC com 64 GB). Abra sua configuração `strata-<model>.json` na pasta de dados do Strata ([onde](docs/INSTALL.md#where-things-are-stored)) e adicione:
- **Geração mais rápida:** `"--adapt-async", "1"` na lista `"args"`. O cache de experts passa a ser atualizado numa thread auxiliar em vez de pausar a resposta. Medido numa RTX 5070 (Windows, Q2_0): +4-5% de geração, +7-10% em código e documentos longos. As respostas podem diferir um pouco do padrão (ficam iguais de uma execução para outra).
- **Leitura do prompt mais rápida no Windows:** `"STRATA_PIN_BY_NEED": "1"` no bloco `"env"`. O Windows só consegue travar cerca de metade da RAM para a placa; isto trava primeiro os experts de que a placa realmente precisa. Medido na RTX 5070: +4,4% na leitura do prompt.
```json
"args":[...,"--adapt-async","1"],
"env":{"STRATA_PIN_BY_NEED":"1"}
```
Reinicie o Strata (`START-HERE.bat` ou `./setup.sh`) depois da mudança. Para desfazer, remova as duas entradas.
## Algo deu errado?
- **Meu PC travou na primeira vez que o Strata iniciou.** Isso é normal enquanto ele carrega o modelo. Espere e não
{"d":"code","p":"Write a Python function that merges two sorted lists into one sorted list without using sort(). Include type hints and a short docstring."},
{"d":"code","p":"Implement an LRU cache in Python using OrderedDict with get and put methods, and write three unit tests with pytest."},
{"d":"code","p":"Write a JavaScript function debounce(fn, wait) and explain how it differs from throttle, with a small usage example."},
{"d":"code","p":"Write a C++ class template for a fixed-capacity ring buffer with push, pop and size, using std::array. Mention thread-safety."},
{"d":"code","p":"Show a Rust function that reads a file line by line, counts word frequencies in a HashMap and prints the top 5 words."},
{"d":"code","p":"Write a Go HTTP handler that returns JSON {\"status\":\"ok\"} and a middleware that logs the request duration."},
{"d":"code","p":"Write a SQL query that returns the top 3 customers by total order value per country from tables customers(id,country) and orders(id,customer_id,amount)."},
{"d":"code","p":"Write a bash script that finds all files larger than 100 MB under /var/log, sorted by size, and prints them in human readable form."},
{"d":"code","p":"Explain this Python error and how to fix it:\nTraceback (most recent call last):\n File \"app.py\", line 12, in <module>\n print(data['users'][0]['name'])\nKeyError: 'users'"},
{"d":"code","p":"Refactor this function to be more readable:\ndef f(a):\n r=[]\n for i in range(len(a)):\n if a[i]%2==0:\n r.append(a[i]*a[i])\n return r"},
{"d":"code","p":"Write a regular expression that matches ISO 8601 dates like 2026-10-11 and explain each part."},
{"d":"code","p":"Write a TypeScript interface for a paginated API response and a generic fetch helper that validates the shape at runtime."},
{"d":"math","p":"A train leaves a station at 9:15 and travels at 80 km/h. A second train leaves the same station at 10:00 on the same track at 120 km/h. At what time does the second catch the first? Show the steps."},
{"d":"math","p":"Prove that the sum of the first n odd numbers is n squared, using induction."},
{"d":"math","p":"Solve the system: 3x + 2y = 16 and x - y = 2. Then check the solution."},
{"d":"math","p":"What is the probability of getting exactly two heads in five fair coin flips? Explain the formula you use."},
{"d":"reason","p":"Three friends, Ann, Ben and Cara, each own exactly one of a cat, a dog and a parrot. Ann does not own the dog. Ben owns neither the cat nor the parrot. Who owns what? Explain your reasoning."},
{"d":"reason","p":"I have a 3 liter jug and a 5 liter jug and unlimited water. How can I measure exactly 4 liters?"},
{"d":"chat","p":"I have a job interview tomorrow for a junior data analyst role and I am nervous. Can you give me practical advice?"},
{"d":"chat","p":"What are some good ways to learn a new language as an adult with only 30 minutes a day?"},
{"d":"chat","p":"Recommend a weekly meal prep plan for someone who wants high protein, vegetarian meals on a budget."},
{"d":"chat","p":"My laptop gets very hot and the fan is loud when I open many browser tabs. What can I check?"},
{"d":"creative","p":"Write a short story (about 300 words) about a lighthouse keeper who discovers that the light attracts something other than ships."},
{"d":"creative","p":"Write a poem about autumn rain in the style of a haiku sequence, five stanzas."},
{"d":"creative","p":"Describe a bustling night market in a fantasy city, using all five senses."},
{"d":"science","p":"Explain how a transformer neural network processes a sentence, from tokens to the next-token prediction, for a software engineer who has not studied machine learning."},
{"d":"science","p":"Why is the sky blue and sunsets red? Explain Rayleigh scattering without heavy math."},
{"d":"science","p":"Explain how CRISPR-Cas9 edits a gene, and name two ethical concerns."},
{"d":"history","p":"Summarize the causes and consequences of the fall of the Western Roman Empire in a few paragraphs."},
{"d":"history","p":"Compare the economic policies of the New Deal with those of the Reagan era."},
{"d":"law","p":"Explain the difference between a trademark, a copyright and a patent, with one everyday example of each."},
{"d":"business","p":"Draft a polite email to a customer whose order is delayed by two weeks, offering a 10 percent discount and a new delivery date."},
{"d":"business","p":"Write a one-page project proposal outline for migrating a small company's file server to a cloud storage service."},
{"d":"data","p":"Create a markdown table comparing PostgreSQL, MySQL and SQLite on concurrency, typical use cases, licensing and scaling."},
{"d":"data","p":"Return a JSON object describing a fictional book with fields title, author, year, genres (array) and a nested publisher object. Only output JSON."},
{"d":"tool","p":"You are an assistant with a tool get_weather(city, unit). The user asks: 'Do I need an umbrella in Hamburg tomorrow?' Write the tool call you would make as JSON and then explain what you would tell the user once you get the result."},
{"d":"summ","p":"Summarize in three bullet points: Photosynthesis converts light energy into chemical energy. In the light-dependent reactions, chlorophyll absorbs photons and splits water, releasing oxygen and producing ATP and NADPH. In the Calvin cycle, these carriers drive the fixation of carbon dioxide into sugars. The rate of photosynthesis depends on light intensity, carbon dioxide concentration and temperature."},
{"d":"fr","p":"Explique en français comment fonctionne un moteur à combustion à quatre temps, avec des phrases simples."},
{"d":"de","p":"Schreibe eine kurze, freundliche E-Mail auf Deutsch, in der du einen Termin beim Zahnarzt von Dienstag auf Donnerstag verschiebst."},
{"d":"es","p":"Dame una receta sencilla de tortilla española y explica los errores más comunes al hacerla."},
Measured on 2026-10-07 by [Ryan V](https://github.com/rvannos) on an unprivileged Proxmox LXC container environment. This tests the **125B Mixture-of-Experts model** (`Swift-Qwen3.8-Flash-Next` IQ2_XS) with a 131,072-token context window, layer split across dual NVIDIA Quadro RTX 4000 GPUs, and expert cache tiering between GPU VRAM and host DDR4 RAM.
The system was evaluated both on synthetic generation sweeps and as the core inference engine of a live production pipeline (**Gridiron Gemini Data Orchestrator**), which continuously ingests and synthesizes 45,000 to 62,000-token NFL podcast transcripts.
Prompt reading speeds reached **126.3 to 249.2 tok/s** on 73,000–74,600 token prompts, with sustained decode generation at **18.1 to 23.2 tok/s** across 500–1,100 generated tokens (expert cache hit rates 48–53%).
---
## Hardware and Software
- **GPUs:** 2x NVIDIA Quadro RTX 4000 8 GB GDDR6 (TU104, Turing Tensor Cores, Flash Attention). 125 W power limit per card. Both cards on PCIe 3.0 x16. Persistence mode enabled.
### 2. Real-World Production Pipeline Comparison (Data Orchestrator)
Prior to deploying Strata 125B MoE, the pipeline ran **Gemma 4 12B** (Dense) on a single GPU. Here is the operational delta recorded across production podcast dossier ingestions:
1.**VRAM Efficiency**: Layer splitting across dual 8GB Quadro RTX 4000s allowed Strata to fit dense base weights and 2,473 resident experts directly in VRAM (~6.8 GB per card), while offloading the remaining 33 GB expert arena to host DDR4 RAM with >97% KV VRAM hit rates.
2.**Context Stability**: Processing massive 60k–75k token audio transcripts without JSON truncation or context dropouts eliminated the pipeline's primary reliability bottleneck.
3.**Speculative Decoding on Turing**: Even on older Turing architecture (Quadro RTX 4000), 4-step speculative drafting maintained consistent 18–23 tok/s decode generation.
The unmodified release builds all 199 native targets. Its complete registered
CTest suite reports 30 tests: 26 pass, four fail. These failures were retained
and investigated before model benchmarks:
| Test | Finding |
| --- | --- |
| `iq_parity` | CTest supplies `--selftest`, which this program treats as a fixture directory. A separate run against the available actual IQ fixtures passes. Native GGUF expert oracles across eight layers also pass. |
| `ple_parity` | The standalone Q2_0 fixture is unavailable. This case remains unqualified. The measured profile uses original IQ2_XS/native model artifacts. |
| `gr_parity` | Ordinary/native GR subchecks pass. Opt-in QFUSE and multi-QFUSE checks fail because the SYCL implementations explicitly return false. `STRATA_QFUSE` is absent in these measurements. |
| `s2_expert_grouped_parity` | Old/new grouped kernels differ bitwise in four cases. All CPU-oracle comparisons report zero rows outside tolerance, with worst normalized error no greater than 1.03e-7. This path is used by the drafter, so independent exact target-only/one-draft/MTP4 model-stream checks were required. |
Those model-stream checks pass independently for both prefill profiles. No
engine code was patched or numeric tolerance relaxed. This does not qualify
other models, formats or optional fused paths.
An earlier CTest attempt refused to run because its log directory was owned by
root. Only `Testing/Temporary` ownership was repaired; the build and binary
remained root-owned. That pre-execution refusal and the classified full-suite
failures are excluded from performance summaries. No hardware fault occurred
in either attempt; recovery was checked before continuing.
A controller progress-file read race also stopped a queued phase before its
payload started. Atomic progress-file replacement fixed it and passed a
concurrent-read check. The failed controller receipts were preserved; no
native test or benchmark had to be repeated because of that failure.
A subsequent pre-install audit encountered the restored UI's health listener
before it was ready. The controller stopped before installation; a bounded
transport-readiness check fixed that sequencing issue. Full idle recovery
passed before continuing. This did not invalidate or repeat any benchmark.
Each 8K profile passes 21 native checks and 24 API checks, including cancellation,
restart, tools, queued isolation and near-8K capacity. Reference checks compare
actual token streams, not only successful loading or fluent text. Benchmark
repeats within each profile match exactly. Different prefill sizes can change
long continuations, so their decode-rate difference is not a pure kernel
comparison. Arithmetic accuracy is recorded separately from runtime validity.
The 262K profile separately passes five native reference checks and 12 API
checks at progressive and full context, including exact speculative/reference
agreement and overflow rejection. Full-window timings are individual capacity
checks, separate from the three-run matched-input benchmarks.
Independent guards enforce exclusive GPU ownership, Gen4 x16, zero watched
Measured on 2026-10-07 by timnevits. **Official upstream release**, commit
`e8ca9afd03d839d4f8dbbe82dffce7f8a3bafd7a`, with **no local engine patches**.
The measurements come from my B65 hardware.
Both memory profiles pass their bounded correctness checks. With the same 7K inputs, the 8K/prefill-512 profile delivers **43.49 decode tok/s**, versus **42.38** with the 262K/prefill-512 profile (-2.6%). The 8K/prefill-4096 profile reduces median 7K TTFT from **22.23 to 8.47 seconds**, with decode tradeoffs below.
## Hardware and configuration
One Intel Arc Pro B65, 32 GiB VRAM (`8086:e222`), **Gen4 x16**, 200 W cap;
i5-12600K, 128 GB DDR4-3200, Samsung 980 NVMe. Other model owners were stopped.
Ubuntu 26.04.1, kernel 7.0.0-38, xe/NEO 26.22.38646.7, Level Zero 1.28.6.
Native Release/JIT build with existing oneAPI 2026.1.0 and oneMKL, precise FP,
correctly rounded divide/sqrt and 32-lane subgroups. The tagged engine labels
itself `0.1.40-sycl`; the commit and binary hash identify this build.
Generated content is discarded; only numeric/hash receipts persist.
Prefill 4096 cuts the 7K TTFT median by 61.9%, but all five long continuations differ from batch 512 and the largest measured 7K decode-cell regression is 16.2%. It is an optional prefill-latency tradeoff, not an overall decode improvement. We keep batch 512 for normal serving. With batch 512, the larger memory profile's 7K TTFT is 24.36s versus 22.23s; 10/10 matched main benchmark cells have identical output hashes across the two memory profiles. See raw per-workload data before generalizing the aggregate.
## Validation and limits
Both 8K profiles pass **21 native reference checks** and **24 API/lifecycle
checks**, including exact target-only/one-draft/MTP4 agreement on the tested
have no resets, PCIe errors, swap-out or OOM; peaks are recorded in qualification.json.
Failed qualification attempts are excluded from throughput. Both previously
local mirror safety fixes are upstream: incomplete coverage refuses startup,
and pinned mirror memory is freed before QUIT exit.
The qualified installed profile uses 262144 usable API tokens, prefill 512 and a 4096 MiB reserve. Routed pre/post-reboot full-window, tool, authorization and queued-isolation checks, graceful unload/restart, retained-model recovery and actual Pi chat/read-tool checks pass. Fresh full-window prefill remains about 15–16 minutes; capacity does not imply low latency or broad long-context quality. This report is pinned to v0.1.40.2; v0.1.40.3 was released during testing and is not measured here.
[BUILD.md](BUILD.md) gives exact preparation and portable reproduction steps.
[Portable fixture audit](portable-fixture-audit.json) verifies that the published
drivers produce the measured main inputs and public token fixtures, without
Ported from PR #1432 for 0.1.41. Raw dumps over 100 KB and binary files were left out to keep the repository small; the README, summaries, configs and per-run results are unchanged. The files below were in the original PR.
1 files, 0.3 MB:
| file | bytes | why |
|---|---:|---|
| `results.json` | 282,662 | raw dump over 100 KB |
RAM logs must live on tmpfs. No generated content is written by this script.
"""
importargparse
importhashlib
importjson
importos
frompathlibimportPath
importre
importsignal
importsys
importthreading
importtime
TASKS=[
'Write a complete Python standard-library LRU cache module with expiry and locking, validation, iteration, detailed docstrings, demonstrations and boundary-case discussion. Write at least 1500 words of code and explanation.',
'Write a complete Python standard-library CSV import module with typed validation, error collection, transaction staging, provenance, test examples and extensive documentation. Write at least 1500 words of code and explanation.',
'Write a detailed story of at least 1500 words about a lighthouse keeper and marine biologist deciphering a research notebook. Develop their distinct voices, observations, uncertainty, and discoveries.',
'Explain accurate inventory audits, discrepancy investigation, provenance, reconciliation, and periodic checks in at least 1500 words with concrete examples and worked scenarios.',
'Act as an incident coordinator. Develop a detailed operational response plan for a failed local service: evidence collection, hypotheses, bounded tests, rollback, recovery verification and communication. Include worked scenarios and at least 1500 words.'
]
WARMUP='Write a detailed 1500-word guide to maintaining a community garden, with seasonal plans, worked examples and practical checklists.'
FILLER='\n'.join(f'Archive record {i}: shelf {i%17}, item {(i*37)%997}, status reviewed, source verified, date unknown. Preserve provenance and record uncertainty.'foriinrange(900))
RAM logs must live on tmpfs. No generated content is written by this script.
"""
importargparse
importhashlib
importjson
importos
frompathlibimportPath
importre
importsignal
importsys
importthreading
importtime
TASKS=[
'Write a complete Python standard-library LRU cache module with expiry and locking, validation, iteration, detailed docstrings, demonstrations and boundary-case discussion. Write at least 1500 words of code and explanation.',
'Write a complete Python standard-library CSV import module with typed validation, error collection, transaction staging, provenance, test examples and extensive documentation. Write at least 1500 words of code and explanation.',
'Write a detailed story of at least 1500 words about a lighthouse keeper and marine biologist deciphering a research notebook. Develop their distinct voices, observations, uncertainty, and discoveries.',
'Explain accurate inventory audits, discrepancy investigation, provenance, reconciliation, and periodic checks in at least 1500 words with concrete examples and worked scenarios.',
'Act as an incident coordinator. Develop a detailed operational response plan for a failed local service: evidence collection, hypotheses, bounded tests, rollback, recovery verification and communication. Include worked scenarios and at least 1500 words.'
]
WARMUP='Write a detailed 1500-word guide to maintaining a community garden, with seasonal plans, worked examples and practical checklists.'
FILLER='\n'.join(f'Archive record {i}: shelf {i%17}, item {(i*37)%997}, status reviewed, source verified, date unknown. Preserve provenance and record uncertainty.'foriinrange(900))
"iq_parity":"CTest treats --selftest as a fixture directory. Subsequent actual IQ fixtures and native GGUF expert oracles across eight layers passed.",
"ple_parity":"Q2_0 PLE fixture missing; unqualified. Actual profile uses original IQ2_XS PLE.",
"gr_parity":"Only unsupported opt-in QFUSE / multi-QFUSE checks fail. Port explicitly returns false and ordinary/native GR subchecks pass; STRATA_QFUSE remains absent.",
"s2_expert_grouped_parity":"Old/new grouped S2 results differ bitwise in four cases. All numerical CPU oracle checks show0outside tolerance, worst<=1.03e-7 normalized. S2 grouped path used by drafter; require exact target-only vs speculative model streams before throughput."
},
"native_source_patches":[],
"continue_scope":"Only original native IQ2_XS, default paths; no QFUSE, no other model/new format/general qualification. Preserve all failures; independent IQ fixture/native8layers then model reference/API gates before speed.",
[image-check.png](data-131072/image-check.png)). The encoder falling back to the CPU is
visible in the log as `ggml_cuda_init: failed to initialize CUDA` followed by
`strata-vision: on the CPU, 16 threads`, which is what `VISION=cpu` asks for.
## Notes for users of this environment
1.**Docker Desktop's default RAM makes the installer choose the low-RAM mode.** With no
`%UserProfile%\.wslconfig`, Docker Desktop caps the WSL2 VM at 50% of host RAM, so the
container saw ~31 GB of the host's 64 GB. `setup.py` reads RAM from `/proc/meminfo`, so it
did not error — it selected the low-RAM mode ("the GPU holds ~38% of IQ1_M's experts, the
other ~15 GB stay in RAM, read once from a copy in the model folder"). A 32 GB PC is a
documented low-RAM case, but on a 64 GB PC it is silent and unwanted. Adding
`[wsl2]` / `memory=56GB` ([wslconfig.txt](wslconfig.txt)) and restarting
(`wsl --shutdown`) gives the container 54.9 GiB and the normal mode. Both measured runs
used the normal mode.
2.**A request needs 8 tokens of headroom.**`prompt + max_tokens + 8 ≤ max_context`, and
the engine does not truncate. A 65,280-token prompt with `max_tokens: 256` on the
65,536-token limit was refused with HTTP 400: *"prompt (65280 tokens) + max tokens (256)
exceeds the context (65536); requests are never truncated. Send a smaller max_tokens (at
most 248 here)"*. [probe-request.py](probe-request.py) reproduces it. The largest prompt
measured here therefore leaves 512 tokens: 130,560 of 131,072 and 65,024 of 65,536.
3.**The port is published on `127.0.0.1` only**, so no API key was set
(`"api_key": false` in `/health`). Other containers reach the same server over a
user-defined Docker network at `http://strata:8080`; nothing was exposed to the LAN.
## Limitations
- **One session per configuration.** The warm-up state differs between them, and the
128K-vs-64K comparison in the results is a comparison of two sessions, not of two
interleaved settings. The engine's own log shows CUDA-graph capture
(`strata verify: capturing the N-token window`) happening during the first requests of a
session, which is one candidate explanation for the ordering; it was not tested. Treat the
difference as an observation to reproduce, not a result.
- **One machine, one quantization, one GPU, one synthetic workload.** Long output, sampled
decoding, thinking, multi-request concurrency (`parallel`), tool use and a sustained
thermal run were not evaluated. The needle and image checks measure recall and reading on
those specific inputs, not general model quality.
- **The PCIe link ran at x8 of a possible x16**, and the card reports a maximum width of
x16. The engine measured 28.9 GB/s host-to-device; the RTX 5090 report on a Gen5 x16 link
measured 50.0 GB/s. Whether the narrower link costs anything here was not tested — the
link is a property of this machine's slot wiring, so absolute numbers may move on a host
that runs the card at x16.
- **Windows and WSL2 are between Strata and the hardware.** File I/O and page-locked memory
go through the WSL2 VM; `--ulimit memlock=-1` was passed and the engine reported
`MAP_HUGETLB unavailable` with transparent huge pages requested instead. Numbers from a
bare-metal Linux host with the same card are not guaranteed to match.
- **Timings include the whole client path** (HTTP, front-end tokenization, loopback) and are
measured at the client, not inside the engine.
- The 131,072-token configuration logged the engine's own VRAM warning quoted above. It did
not cause a failure in any measured run, but it is a real margin of 152 MiB and a reason
to prefer `--vram-reserve-mib 1060` (or a smaller context) for unattended use.
- One telemetry file, `data-131072/telemetry-speed-overlapped.jsonl`, was written by two
samplers that overlapped for about eleven minutes (819 of 820 lines parse; the timestamps
contain one discontinuity). Only peaks and ranges are taken from it. The clean runs are
`data-131072/telemetry-checks.jsonl` and `data-65536/telemetry-speed.jsonl`.
## Files
| File | What it is |
| --- | --- |
| `benchmark.py`, `monitor.py`, `summarize-telemetry.py`, `image-check.py`, `probe-request.py` | the scripts, runnable as documented above |
| `data-131072/`, `data-65536/` | `results.json` (per-run engine timings, draft statistics, output text, request hashes) and `summary.json`, the telemetry and its memory summary, `needles.json`, the image check, and the engine's status at startup |
| `config-131072.json`, `config-65536.json`, `engine-131072.log`, `engine-65536.log` | the recorded configuration and the unabridged engine log of each session |
| `environment.txt`, `python-packages.txt`, `wslconfig.txt` | host and container environment |
| `model-provenance.json`, `sha256.txt`, `mtp-manifest.json`, `mtp-inventory.md` | what was downloaded and its hashes |
Ported from PR #1460 for 0.1.41. Raw dumps over 100 KB and binary files were left out to keep the repository small; the README, summaries, configs and per-run results are unchanged. The files below were in the original PR.
"text":"The provided Python module consists of a series of functions, each named with a specific pattern (`task_XXXXX`), that perform a deterministic integer transformation. Each function takes a single integer argument `value` and returns an integer result computed using a linear congruential formula modulo a fixed prime number, 100003. The general form of the transformation for a given task index `n` is:\n\n```python\nresult = (value * multiplier + addend) % 100003\n```\n\nwhere `multiplier` and `addend` are derived from the task index. For most tasks (0 through 96), the multiplier is `n + 1` and the addend is `n`. For example, `task_00000` uses multiplier 1 and addend 0, while `task_00096` uses multiplier 97 and addend 96. However, starting from `task_00097`, the pattern changes: the multiplier resets to 1, and the addend continues to increment. Thus, `task_00097` uses multiplier 1 and addend 97, `task_00"
"text":"The provided Python module consists of a series of functions, each named with a specific pattern (`task_XXXXX`), that perform a deterministic integer transformation. Each function takes a single integer argument `value` and returns an integer result computed using a linear transformation followed by a modulo operation. The general form of each function is:\n\n```python\ndef task_XXXXX(value: int) -> int:\n return (value * A + B) % 100003\n```\n\nwhere `A` and `B` are constants specific to each function. The modulus `100003` is a prime number, which is significant in modular arithmetic because it ensures that the multiplicative group modulo 100003 is cyclic and that every non-zero element has a multiplicative inverse. This property is often exploited in cryptographic algorithms, hash functions, and pseudo-random number generators.\n\n### Deterministic Integer Transforms\n\nEach function implements a linear congruential transformation, a common technique in number theory and computer science. The transformation `value * A + B` is an affine transformation over the integers. When combined with the modulo operation, it becomes a function from the finite field (or ring) of integers modulo 10"
"text":"The provided Python module presents a series of deterministic integer transforms, each encapsulated within a uniquely named function. These functions, labeled from `task_00000` to `task_00112` (and potentially beyond), follow a consistent mathematical pattern: they take an integer input, apply a linear transformation of the form `(value * multiplier + offset) % modulus`, and return the result. This structure serves as a synthetic benchmark for evaluating code generation, pattern recognition, and algorithmic consistency in automated systems.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of each function is a linear congruential transformation. For any given task index `n`, the function computes `(value * (n + 1) + n) % 100003`. The modulus used is 100003, which is a prime number. This choice of modulus is significant because it ensures that the multiplicative inverse exists for all non-zero multipliers, making the transformation bijective (one-to-one) over the finite field defined by the modulus. This property is critical in cryptographic contexts, hash functions, and pseudo-random number generators, where invertibility and uniform distribution are desired.\n\nThe multiplier increases linearly"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance and correctness of a system under a specific nonce (`series-8192-trial-1`). The code consists of a series of function definitions, each named `task_XXXXX`, where `XXXXX` is a zero-padded integer index. Each function accepts a single integer argument `value` and returns an integer result computed via a linear transformation followed by a modulo operation.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of each function is the expression:\n```python\n(value * multiplier + offset) % 100003\n```\nThis is a classic linear congruential generator (LCG) step, often used in pseudo-random number generation or cryptographic hashing contexts, though here it serves as a deterministic arithmetic transform. The modulus `100003` is a prime number, which ensures that the multiplicative group modulo 100003 is cyclic and that multiplication by any non-zero integer less than the modulus is invertible. This property is crucial for ensuring that the transform is bijective (one-to-one) within the range `[0, 100002]`, provided the multiplier is not"
"text":"The provided Python module consists of a series of functions, each named with a specific pattern (`task_XXXXX`), that perform a deterministic integer transformation. Each function takes a single integer argument `value` and returns an integer result computed using a linear transformation followed by a modulo operation. The general form of each function is:\n\n```python\ndef task_XXXXX(value: int) -> int:\n return (value * A + B) % 100003\n```\n\nwhere `A` and `B` are constants specific to each function. The modulus `100003` is a prime number, which is significant in modular arithmetic because it ensures that the multiplicative group modulo 100003 is cyclic and that every non-zero element has a multiplicative inverse. This property is often exploited in cryptographic algorithms, hash functions, and pseudo-random number generators.\n\n### Deterministic Integer Transforms\n\nEach function implements a linear congruential transformation, a common technique in number theory and computer science. The transformation `f(x) = (a*x + b) mod m` is a linear congruential generator (LCG) when used iteratively, but here it is applied once per function call."
"text":"The provided Python module consists of a series of functions, each named with a specific pattern (`task_XXXXX`), that perform a deterministic integer transformation. Each function takes a single integer argument `value` and returns an integer result computed using a linear congruential formula modulo a fixed prime number, 100003. This structure is typical of benchmarking suites, cryptographic primitives, or hash function implementations where deterministic, fast, and collision-resistant (or at least well-distributed) outputs are required.\n\n### Deterministic Integer Transforms\n\nEach function implements a linear transformation of the form:\n\n```\nf(value) = (value * a + b) % m\n```\n\nwhere `a` is the multiplier, `b` is the additive constant, and `m = 100003` is the modulus. The modulus 100003 is a prime number, which ensures that the multiplicative group modulo `m` is cyclic and that multiplication by any non-zero `a` (where `1 \u2264 a < m`) is invertible. This property is critical for ensuring that the transformation is bijective (one-to-one) when `a` is coprime to `m`, which it always is since"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance and correctness of a system's ability to handle a large number of deterministic integer transforms. The code consists of a series of functions, each named with a specific pattern (`task_XXXXX`), that perform a linear transformation on an input integer and return the result modulo a large prime number.\n\n### Deterministic Integer Transforms\n\nAt the core of each function is a linear transformation of the form $f(x) = (ax + b) \\pmod m$. In this specific module, the modulus $m$ is consistently set to `100003`, which is a prime number. The coefficients $a$ and $b$ vary for each function. For example, `task_00000` implements $f(x) = (1x + 0) \\pmod{100003}$, while `task_00001` implements $f(x) = (2x + 1) \\pmod{100003}$.\n\nThese transforms are deterministic, meaning that for a given input `value`, the output will always be the same. This property is crucial for benchmarking, as it allows for"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance and correctness of a system's ability to handle a large number of deterministic integer transforms. The code consists of a series of functions, each named with a specific pattern (`task_XXXXX`), that perform a linear transformation on an input integer and return the result modulo a large prime number.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of each function is a linear transformation of the form:\n`result = (value * multiplier + constant) % MODULUS`\n\nHere, `MODULUS` is consistently set to `100003`. This number is a prime, which is a common choice in competitive programming and cryptographic contexts to ensure that the modulo operation behaves predictably and avoids certain edge cases associated with composite moduli (such as zero divisors).\n\nThe `multiplier` and `constant` vary for each function. For example:\n- `task_00000`: `(value * 1 + 0) % 100003`\n- `task_00001`: `(value * 2 + 1) % 100003`\n- `task"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance and correctness of a system's ability to handle a large number of deterministic integer transforms. The code consists of a series of functions, each named with a specific pattern (`task_XXXXX`), that perform a linear transformation on an input integer and return the result modulo a large prime number.\n\n### Deterministic Integer Transforms\n\nAt the core of each function is a linear transformation of the form $f(x) = (ax + b) \\pmod m$. In this specific module, the modulus $m$ is consistently set to `100003`, which is a prime number. The coefficients $a$ and $b$ vary for each function. For example, `task_00000` implements $f(x) = (1x + 0) \\pmod{100003}$, which is effectively an identity function within the modular space. As the task index increases, the coefficients change in a predictable, albeit somewhat irregular, pattern.\n\nFor instance, `task_00001` uses $a=2, b=1$, while `task_00002` uses $a=3, b"
"text":"This Python module presents a highly repetitive, synthetic collection of functions that serve as a benchmark for deterministic integer transforms. The code consists of a series of functions named `task_00000` through `task_00888` (and potentially beyond), each performing a specific linear transformation on an input integer `value` and returning the result modulo 100003.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of each function is a linear congruential generator (LCG) style operation: `(value * multiplier + addend) % 100003`. The modulus, 100003, is a prime number. This choice of modulus is significant because it ensures that the multiplicative group of integers modulo 100003 is cyclic, which is a desirable property in cryptographic and pseudo-random number generation contexts. However, in this specific benchmark context, the modulus likely serves to keep the output within a fixed range (0 to 100002) and to prevent integer overflow issues in languages with fixed-width integers, although Python handles arbitrary-precision integers natively.\n\nThe transformation is deterministic: for a given `value`, `multiplier"
"text":"The provided Python module consists of a series of 889 distinct functions, named sequentially from `task_00000` to `task_00888`. Each function accepts a single integer argument `value` and returns an integer result. The core logic of every function is identical in structure: it performs a linear transformation on the input value and then applies a modulo operation. Specifically, the formula used is `(value * multiplier + offset) % 100003`.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nThe fundamental operation here is an affine transformation over a finite field. The modulus, 100003, is a prime number. This choice of modulus is significant because it ensures that the set of integers modulo 100003 forms a finite field ($\\mathbb{Z}_{100003}$). In this context, the multiplication by the `multiplier` and the addition of the `offset` are well-defined operations that preserve the properties of the field.\n\nThe `multiplier` and `offset` parameters vary for each function. By analyzing the pattern, we can deduce how these parameters are generated. For the first 97 functions (`"
"text":"This synthetic Python module presents a highly repetitive, programmatic structure that serves as a benchmark for evaluating code generation, static analysis, and execution performance. The module consists of a series of functions named `task_00000` through `task_00888` (and potentially beyond, as the snippet cuts off). Each function accepts a single integer argument `value` and returns an integer. The core logic of every function is identical in structure: `(value * multiplier + offset) % 100003`.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nThe fundamental operation here is a linear congruential transform. For any given function `task_NNNNN`, the output is determined by the formula $f(x) = (ax + b) \\pmod m$, where $m = 100003$. The modulus $100003$ is a prime number, which is significant in number theory. In modular arithmetic, if $m$ is prime, then for any $a$ not divisible by $m$, the multiplication by $a$ is a bijection (a one-to-one mapping) over the set of integers modulo $m$. This means that for a fixed $"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance, parsing, and execution capabilities of a Python interpreter or compiler. It consists of a massive sequence of function definitions, each named `task_XXXXX`, where `XXXXX` is a zero-padded integer index. Each function accepts a single integer argument `value` and returns a transformed integer result based on a specific linear formula modulo a large prime number.\n\n### 1. Structure and Naming Convention\n\nThe code follows a strict, repetitive pattern. The function names are generated sequentially from `task_00000` to `task_01756` (and potentially beyond, as the snippet cuts off). This naming convention suggests that these functions are intended to be accessed programmatically, perhaps via `globals()` or a dictionary mapping, rather than being called directly by name in source code. The zero-padding ensures lexicographical sorting matches numerical sorting, which is a common practice in benchmarking suites to ensure deterministic ordering of tasks.\n\n### 2. The Mathematical Transform\n\nEach function implements a linear congruential generator (LCG) step, a fundamental component in pseudo-random number generation. The general form of the transformation is:\n\n$$ f(x) = (a \\cdot"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance and correctness of a system's ability to handle a large volume of deterministic integer transforms. The code consists of a series of functions, each named with a specific pattern (`task_XXXXX`), that perform a linear transformation on an input integer and return the result modulo a large prime number.\n\n### 1. The Mathematical Transform\n\nEach function in the module follows the general form:\n```python\ndef task_XXXXX(value: int) -> int:\n return (value * A + B) % M\n```\nWhere:\n- `value` is the input integer.\n- `A` is a multiplier coefficient.\n- `B` is an additive constant.\n- `M` is the modulus, which is consistently set to `100003` throughout the module.\n\nThis is a classic **Linear Congruential Transform**. In cryptography and random number generation, such transforms are often used to map inputs to a specific range. Here, the modulus `100003` is a prime number, which ensures that the mapping is bijective (one-to-one) if the multiplier `A` is not a multiple of the modulus. Since"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance and correctness of a system's ability to handle a large volume of deterministic integer transforms. The code consists of a series of functions, named sequentially from `task_00000` to `task_01756`, each performing a specific linear transformation on an input integer `value`.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of each function is the expression:\n```python\nreturn (value * multiplier + offset) % 100003\n```\nThis is a classic linear congruential generator (LCG) step, often used in pseudo-random number generation or cryptographic hashing. Here, `multiplier` and `offset` are constants specific to each function. The modulus `100003` is a prime number, which ensures that the arithmetic operations remain within a finite field, preventing integer overflow issues in languages with fixed-width integers (though Python handles arbitrary precision integers, the modulo operation keeps the result bounded).\n\nThe pattern of `multiplier` and `offset` follows a specific sequence. For the first 97 functions (`task_00000` to `task_0009"
"text":"This Python module represents a synthetic benchmark designed to test the performance and correctness of integer arithmetic operations, specifically focusing on linear transformations combined with modular arithmetic. The code consists of a series of functions, named `task_00000` through `task_03471`, each performing a specific calculation on an input integer `value`.\n\n### 1. Structure and Naming Convention\n\nThe module follows a strict naming convention: `task_XXXXX`, where `XXXXX` is a zero-padded integer representing the task index. This structure suggests that the module is generated programmatically rather than written by hand. The functions are uniform in signature: they accept a single integer argument `value` and return an integer.\n\nThe core logic of each function is identical in form:\n```python\nreturn (value * A + B) % 100003\n```\nHere, `A` is the multiplier and `B` is the additive constant. The modulus is fixed at `100003`, which is a prime number. This choice of modulus is significant because it ensures that the transformation is a bijection (a one-to-one mapping) over the set of integers modulo 100003, provided"
"text":"This Python module represents a synthetic benchmark designed to test the performance and correctness of a system's ability to handle a large volume of deterministic integer transforms. The code consists of a series of functions, named `task_00000` through `task_03471`, each performing a specific linear transformation on an input integer `value`.\n\n### 1. Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of every function in this module is the formula:\n`return (value * multiplier + offset) % 100003`\n\nThis is a classic **Linear Congruential Generator (LCG)** step, often used in pseudo-random number generation or hashing. Here, however, it is used as a deterministic transform.\n\n* **Multiplier (`a`)**: The multiplier cycles through values from 1 to 97. For example, `task_00000` uses `1`, `task_00001` uses `2`, and so on. When the multiplier reaches 97, it resets to 1 for the next block of tasks (e.g., `task_00097` uses `1` again).\n* **Offset"
"text":"This Python module represents a synthetic benchmark designed to test the performance and correctness of integer arithmetic operations, specifically focusing on linear congruential transforms and modular arithmetic. The code consists of a series of functions, each named `task_XXXXX`, where `XXXXX` is a zero-padded integer index. Each function accepts a single integer argument `value` and returns a transformed integer result.\n\n### 1. Deterministic Integer Transforms\n\nThe core logic of every function in this module follows the pattern:\n```python\nreturn (value * A + B) % M\n```\nWhere:\n- `A` is the multiplier.\n- `B` is the additive constant.\n- `M` is the modulus, which is consistently `100003` across all functions.\n\nThis structure is known as a **Linear Congruential Generator (LCG)** step or a simple affine transformation. The transformation is deterministic, meaning that for a given input `value`, the output will always be the same. This property is crucial for benchmarking, as it allows for predictable verification of results without relying on random number generation, which can introduce non-determinism and complicate testing.\n\n### 2. Modulo Arithmetic and the Mod"
"text":"The provided Python module consists of a series of functions, each named with a specific pattern (`task_XXXXX`), that perform a deterministic integer transformation. Each function takes a single integer argument `value` and returns an integer result computed using a linear congruential formula modulo a fixed prime number, 100003. This structure is typical of benchmarking suites, cryptographic primitives, or hash function implementations where deterministic, fast, and reproducible outputs are required.\n\n### Deterministic Integer Transforms\n\nEach function implements a linear transformation of the form:\n\n```\nresult = (value * multiplier + addend) % MOD\n```\n\nwhere `MOD = 100003`, a prime number. The multiplier and addend vary per function. For example:\n- `task_00000`: `(value * 1 + 0) % 100003`\n- `task_00001`: `(value * 2 + 1) % 100003`\n- `task_00096`: `(value * 97 + 96) % 100003`\n- `task_00"
"text":"The provided Python module consists of a series of functions, each named with a specific pattern (`task_XXXXX`), that perform a deterministic integer transformation. Each function takes a single integer argument `value` and returns an integer result computed using a linear transformation followed by a modulo operation. The general form of each function is:\n\n```python\ndef task_XXXXX(value: int) -> int:\n return (value * A + B) % 100003\n```\n\nwhere `A` and `B` are constants specific to each function. The modulus `100003` is a prime number, which is significant in modular arithmetic because it ensures that the multiplicative group modulo 100003 is cyclic and that every non-zero element has a multiplicative inverse. This property is often exploited in cryptographic algorithms, hash functions, and pseudo-random number generators.\n\n### Deterministic Integer Transforms\n\nEach function implements a linear congruential transformation, a common technique in number theory and computer science. The transformation `value * A + B` is an affine transformation over the integers. When combined with the modulo operation, it becomes a linear congruential generator (LCG) step, though here it is"
"text":"The provided Python module presents a series of deterministic integer transforms, each encapsulated within a uniquely named function. These functions, labeled from `task_00000` to `task_00112` (and potentially beyond), follow a consistent mathematical pattern: they take an integer input, apply a linear transformation, and return the result modulo a large prime number, specifically 100003. This structure serves as a synthetic benchmark for evaluating code generation, pattern recognition, and algorithmic consistency in automated systems.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of each function is the expression `(value * multiplier + offset) % 100003`. This is a classic linear congruential generator (LCG) step, often used in pseudo-random number generation or cryptographic hashing, though here it is applied deterministically to a single input. The multiplier and offset vary per function, creating a distinct mapping for each task. For instance, `task_00000` uses a multiplier of 1 and an offset of 0, effectively returning `value % 100003`. In contrast, `task_00001` uses a multiplier of 2"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance and correctness of deterministic integer transforms under modulo arithmetic. The code consists of a series of functions, each named with a unique identifier (`task_00000` through `task_00222` and beyond), that perform a linear transformation on an input integer `value`. The core operation in each function is of the form `(value * multiplier + offset) % 100003`. This structure allows for a detailed analysis of computational complexity, arithmetic properties, and software engineering practices.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nThe fundamental operation in this module is a linear congruential function, often seen in pseudo-random number generators or cryptographic primitives, though here it is used for benchmarking. The formula $f(x) = (ax + b) \\pmod m$ defines a deterministic mapping from the input integer to an output integer within the range $[0, m-1]$. In this specific module, the modulus $m$ is fixed at 100,003. This number is significant because 100,003 is a prime number. Using a prime modulus ensures that the multiplicative group of integers"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance and correctness of a system handling a large number of deterministic integer transforms. The code consists of a series of functions, named sequentially from `task_00000` to `task_00222` (and potentially beyond, as the snippet cuts off), each performing a specific linear transformation on an input integer `value`.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of each function is a linear congruential generator (LCG) style transformation: `(value * multiplier + addend) % 100003`. This formula is a fundamental component in many cryptographic and pseudo-random number generation algorithms. The modulus `100003` is a prime number, which is a critical property for ensuring that the transformation is bijective (one-to-one) within the range of integers modulo 100003, provided the multiplier is not a multiple of the modulus. Since the multipliers used here range from 1 to 97, and the modulus is 100003, all multipliers are coprime to the modulus. This guarantees that for any input `value`, the output is uniquely determined"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance and correctness of a system under a specific nonce (`series-8192-trial-3`). The code consists of a series of function definitions, each named `task_XXXXX`, where `XXXXX` is a zero-padded integer index. Each function accepts a single integer argument `value` and returns an integer result. The core logic within each function is a linear transformation followed by a modulo operation.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nThe fundamental operation in each function is of the form:\n`return (value * multiplier + offset) % 100003`\n\nHere, `100003` is a prime number. This choice of modulus is significant in computer science and cryptography because it ensures that the multiplicative group of integers modulo 100003 is cyclic, and every non-zero element has a multiplicative inverse. This property is often desirable in hashing functions or pseudo-random number generators to avoid certain degenerate cycles or collisions that might occur with composite moduli.\n\nThe transformation is an affine transformation over the finite field $\\mathbb{Z}_{100003}$. Specifically, for a"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance and correctness of a system's ability to handle a large number of deterministic integer transforms. The code consists of a series of functions, each named with a specific pattern (`task_XXXXX`), that perform a linear transformation on an input integer and return the result modulo a large prime number.\n\n### Deterministic Integer Transforms\n\nAt the core of this module is the concept of a deterministic integer transform. Each function takes a single integer argument, `value`, and applies a linear transformation of the form $f(x) = (ax + b) \\pmod m$. Here, $a$ is the multiplier, $b$ is the additive constant, and $m$ is the modulus, which is consistently set to `100003` across all functions.\n\nThe multiplier $a$ and the additive constant $b$ follow a specific pattern based on the function's index. For the first 97 functions (indices 0 to 96), the multiplier $a$ is equal to the index plus one ($i + 1$), and the additive constant $b$ is equal to the index ($i$). For example, `task_00000"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance and correctness of a system's ability to handle a large number of deterministic integer transforms. The code consists of a series of functions, each named with a specific pattern (`task_XXXXX`), that perform a linear transformation on an input integer and return the result modulo a large prime number.\n\n### Deterministic Integer Transforms\n\nAt the core of each function is a linear transformation of the form $f(x) = (ax + b) \\pmod m$. In this specific module, the modulus $m$ is consistently set to `100003`, which is a prime number. The coefficients $a$ and $b$ vary for each function. For example, `task_00000` implements $f(x) = (1x + 0) \\pmod{100003}$, while `task_00001` implements $f(x) = (2x + 1) \\pmod{100003}$.\n\nThese transforms are deterministic, meaning that for a given input `value`, the output will always be the same. This property is crucial for benchmarking, as it allows for"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance and correctness of a system's ability to handle a large volume of deterministic integer transforms. The code consists of a series of functions, named sequentially from `task_00000` to `task_00444` (and potentially beyond, though the snippet cuts off), each performing a specific linear transformation on an input integer `value`.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of each function is a linear congruential generator (LCG) style operation: `(value * multiplier + addend) % 100003`. This structure is fundamental in computer science, particularly in pseudo-random number generation and cryptographic hashing, though here it is used for deterministic testing.\n\nThe modulus `100003` is a prime number. Choosing a prime modulus ensures that the multiplicative group of integers modulo $p$ is cyclic, which has implications for the periodicity of the sequence if these functions were chained. However, in this context, the modulus serves primarily to bound the output range to $[0, 100002]$, preventing integer overflow issues in languages with fixed-width integers (though Python handles arbitrary"
"text":"This synthetic Python module presents a highly repetitive, programmatic structure that serves as a benchmark for evaluating code generation, static analysis, and execution performance. The module consists of a series of functions, each named with a specific pattern (`task_XXXXX`), that perform a deterministic integer transform on an input value. Below is a detailed analysis of the code, focusing on its mathematical properties, structural design, and implications for software engineering practices.\n\n### 1. Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of every function in this module is the expression:\n`return (value * multiplier + offset) % 100003`\n\nThis represents a linear congruential generator (LCG) step, a fundamental component in pseudo-random number generation and cryptographic hashing. The modulus `100003` is a prime number. Choosing a prime modulus is critical in modular arithmetic because it ensures that the multiplicative group of integers modulo $p$ is cyclic. This property guarantees that for any non-zero multiplier, there exists a modular inverse, allowing the transformation to be bijective (reversible) if the multiplier is not a multiple of the modulus.\n\nThe parameters `multiplier` and `offset` vary across the functions."
"text":"The provided Python module represents a synthetic benchmark designed to test the performance and correctness of a system's ability to handle a large volume of deterministic integer transforms. The code consists of a series of functions, named sequentially from `task_00000` to `task_00888` (and potentially beyond, though the snippet cuts off), each performing a specific linear transformation on an input integer.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of each function is a linear congruential generator (LCG) style transformation: `(value * multiplier + addend) % 100003`. This formula is a fundamental component of many pseudorandom number generators and cryptographic hash functions. The constant `100003` is a prime number, which is significant in modular arithmetic. When the modulus is prime, the multiplicative group of integers modulo $p$ is cyclic, meaning that for any non-zero multiplier, the operation is invertible. This ensures that the transformation is a bijection (a one-to-one mapping) over the set of integers modulo 100003, provided the multiplier is not a multiple of the modulus.\n\nThe multipliers and addends follow a specific"
"text":"This Python module presents a highly repetitive, synthetic collection of functions that perform deterministic integer transforms. Each function follows a strict pattern: it accepts an integer `value`, applies a linear transformation of the form `(value * multiplier + offset) % modulus`, and returns the result. The modulus is consistently set to `100003`, which is a prime number. This choice of modulus is significant because it ensures that the multiplicative group of integers modulo 100003 is cyclic, meaning that for any non-zero multiplier, the transformation is bijective (a permutation) over the set of integers modulo 100003. This property is often desirable in cryptographic contexts or pseudo-random number generation, although here it appears to be used for benchmarking or synthetic data generation.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nThe core operation in each function is a linear congruential generator (LCG) step, defined by the formula $f(x) = (ax + c) \\pmod m$. In this module, $m = 100003$. The multiplier $a$ and the increment $c$ vary per function. For example, `task_00000` uses $a"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance, parsing, and execution capabilities of a Python interpreter or a code analysis tool. It consists of a massive sequence of function definitions, each named `task_XXXXX`, where `XXXXX` is a zero-padded integer index. Each function accepts a single integer argument `value` and returns a transformed integer based on a specific linear formula modulo a large prime number.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of every function in this module is the expression:\n`return (value * multiplier + offset) % 100003`\n\nThis is a classic **Linear Congruential Generator (LCG)** step, often used in pseudo-random number generation or cryptographic hashing. The operation is deterministic: for a given `value`, `multiplier`, and `offset`, the output is always the same.\n\n1. **Multiplier (`m`)**: The multiplier cycles through values from 1 to 97. For example, `task_00000` uses `1`, `task_00001` uses `2`, ..., `task_00096` uses `97`. Then `task"
"text":"The provided Python module represents a synthetic benchmark designed to test the performance, parsing, and execution capabilities of a Python interpreter or a code analysis tool. It consists of a massive sequence of function definitions, each named `task_XXXXX`, where `XXXXX` is a zero-padded integer index. Each function accepts a single integer argument `value` and returns a transformed integer. The transformation follows a consistent mathematical pattern: `(value * multiplier + offset) % 100003`.\n\n### Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of each function is a linear congruential transformation. The formula `y = (ax + b) mod m` is a fundamental operation in computer science, often used in pseudo-random number generation (specifically, the Linear Congruential Generator method) and cryptographic hashing. Here, `m` is fixed at `100003`, which is a prime number. The choice of a prime modulus ensures that the multiplicative group of integers modulo `m` is cyclic, meaning that for any non-zero `a`, there exists a multiplicative inverse. This property is crucial if one were to reverse the transformation, though in this benchmark, the functions are strictly one-way mappings.\n\nThe"
"text":"The provided Python module consists of a large, repetitive collection of functions, each named with a specific pattern (`task_XXXXX`) and containing a single return statement. This code serves as a synthetic benchmark or test harness, likely generated programmatically to evaluate the performance, parsing capabilities, or memory usage of a code interpreter or compiler. Below is a detailed analysis of the code's structure, mathematical logic, and software engineering implications.\n\n### 1. Deterministic Integer Transforms and Modulo Arithmetic\n\nAt the core of every function in this module is a linear transformation followed by a modulo operation. The general form of the transformation for a given function `task_NNNNN` is:\n\n$$ f(x) = (x \\cdot A + B) \\pmod M $$\n\nWhere:\n* $x$ is the input integer (`value`).\n* $A$ is a multiplier specific to the function.\n* $B$ is an additive constant specific to the function.\n* $M$ is the modulus, which is consistently `100003` across all functions.\n\n**The Modulus (100003):**\nThe number 100003 is a prime number. In modular"
strata generate: 1406 MiB of weights loaded from /data/packs/coder-iq1_m in 2.4 s (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2018.88 MiB of weights, in 2.7 s
strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
strata generate: profile /opt/strata/data/expert-profile-coder.bin: 12288 ranked pairs, built for 12288 slots
strata generate: PLE on, table 320001536 rows of /data/models/coder-IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf
strata mtp: draft layer loaded, 950 MiB of VRAM (experts 675, dense 111), files read in 1.34 s (586 MiB/s)
strata generate: experimental native Q5_K head, 521472000 bytes, in 0.9 s
strata generate: CPU pool tasks/phase: 48 (automatic), participating threads: 16
strata generate: 15 pool workers on logical processors 2,4,6,8,10,12,14,16,18,20,22,24,26,28,30, the host thread on 0 (draining too) (--host-core first)
strata generate: expert cache auto: 7.67 GiB free, 700 MiB reserved (+218 MiB for the draft head) -> 2733 slots
strata generate: expert cache 3554 slots, 6.78 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
so a reply can differ slightly from a run without the cache (same quality:
bench/results/2026-09-27-cache-parity).
PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 3554 of 3554 slots from the profile in 0.4 s (18700 MB/s); slot 0 verified
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 15 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.40.3)
strata generate: token graph hit path: 3554 resident experts, decided on the device
strata serve: prompt chunk auto: 8192 tokens, a 384-slot ring
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata verify: window up to 6 tokens, 75.1 MiB of device buffers
strata mtp: draft head over 106299 tokens (212.9 MiB)
strata serve: 179 MiB of VRAM free with everything loaded - LOW: requests may stall; add --vram-reserve-mib 1033 to the config's args (or lower --max-context)
strata verify: capturing the 6-token window (39 MiB of VRAM free)
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: capturing the 1-token window (124 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: capturing the 4-token window (63 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata serve: prompt 32343 tokens = 0 reused + 32343 read in 13363 ms (2420.4 tok/s), 9 generated in 281 ms (32.0 tok/s), drafts accepted 6 of 6, 2 checkpoints
strata serve: decode expert cache hit rate: 51.8% (1654 hits / 3193 lookups); 1127 more read by the GPU over PCIe or from another GPU (26.1% of all 4320 routed)
strata serve: prompt 32343 tokens = 0 reused + 32343 read in 10796 ms (2995.8 tok/s), 9 generated in 145 ms (61.9 tok/s), drafts accepted 6 of 6, 2 checkpoints
strata serve: decode expert cache hit rate: 57.3% (1855 hits / 3240 lookups); 1080 more read by the GPU over PCIe or from another GPU (25.0% of all 4320 routed)
strata serve: prompt 32344 tokens = 16384 reused + 15960 read in 5118 ms (3118.5 tok/s), 10 generated in 203 ms (49.3 tok/s), drafts accepted 9 of 9, 2 checkpoints
strata serve: decode expert cache hit rate: 62.8% (3047 hits / 4850 lookups); 1390 more read by the GPU over PCIe or from another GPU (22.3% of all 6240 routed)
strata serve: prompt 62627 tokens = 0 reused + 62627 read in 19687 ms (3181.1 tok/s), 10 generated in 203 ms (49.2 tok/s), drafts accepted 6 of 9, 4 checkpoints
strata serve: decode expert cache hit rate: 65.9% (3273 hits / 4970 lookups); 1270 more read by the GPU over PCIe or from another GPU (20.4% of all 6240 routed)
strata verify: capturing the 3-token window (108 MiB of VRAM free)
strata verify: captured the 3-token window (upload no error, sync no error)
strata serve: prompt 62625 tokens = 0 reused + 62625 read in 19539 ms (3205.2 tok/s), 9 generated in 192 ms (46.8 tok/s), drafts accepted 8 of 8, 4 checkpoints
strata serve: decode expert cache hit rate: 74.9% (3569 hits / 4763 lookups); 997 more read by the GPU over PCIe or from another GPU (17.3% of all 5760 routed)
strata verify: capturing the 2-token window (100 MiB of VRAM free)
strata verify: captured the 2-token window (upload no error, sync no error)
strata serve: prompt 62625 tokens = 16384 reused + 46241 read in 15015 ms (3079.7 tok/s), 9 generated in 224 ms (40.2 tok/s), drafts accepted 7 of 7, 4 checkpoints
strata serve: decode expert cache hit rate: 80.2% (3961 hits / 4939 lookups); 821 more read by the GPU over PCIe or from another GPU (14.3% of all 5760 routed)
strata serve: prompt 125919 tokens = 0 reused + 125919 read in 42208 ms (2983.3 tok/s), 9 generated in 172 ms (52.4 tok/s), drafts accepted 4 of 6, 6 checkpoints
strata serve: decode expert cache hit rate: 80.2% (3655 hits / 4559 lookups); 721 more read by the GPU over PCIe or from another GPU (13.7% of all 5280 routed)
strata serve: prompt 125919 tokens = 0 reused + 125919 read in 41025 ms (3069.3 tok/s), 9 generated in 282 ms (31.9 tok/s), drafts accepted 4 of 14, 6 checkpoints
strata serve: decode expert cache hit rate: 82.9% (6635 hits / 8001 lookups); 1119 more read by the GPU over PCIe or from another GPU (12.3% of all 9120 routed)
strata serve: prompt 125920 tokens = 16384 reused + 109536 read in 38581 ms (2839.1 tok/s), 10 generated in 154 ms (65.0 tok/s), drafts accepted 9 of 9, 6 checkpoints
strata serve: decode expert cache hit rate: 88.0% (5018 hits / 5702 lookups); 538 more read by the GPU over PCIe or from another GPU (8.6% of all 6240 routed)
strata serve: prompt 261 tokens = 0 reused + 261 read in 920 ms (283.6 tok/s), 10 generated in 153 ms (65.4 tok/s), drafts accepted 7 of 9, 1 checkpoints
strata serve: decode expert cache hit rate: 71.6% (3828 hits / 5343 lookups); 897 more read by the GPU over PCIe or from another GPU (14.4% of all 6240 routed)
strata generate: 1406 MiB of weights loaded from /data/packs/coder-iq1_m in 1.6 s (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2018.88 MiB of weights, in 3.9 s
strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
strata generate: profile /opt/strata/data/expert-profile-coder.bin: 12288 ranked pairs, built for 12288 slots
strata generate: PLE on, table 320001536 rows of /data/models/coder-IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf
strata mtp: draft layer loaded, 876 MiB of VRAM (experts 675, dense 111), files read in 1.29 s (609 MiB/s)
strata generate: experimental native Q5_K head, 521472000 bytes, in 0.5 s
strata generate: CPU pool tasks/phase: 48 (automatic), participating threads: 16
strata generate: 15 pool workers on logical processors 2,4,6,8,10,12,14,16,18,20,22,24,26,28,30, the host thread on 0 (draining too) (--host-core first)
strata generate: expert cache auto: 8.63 GiB free, 700 MiB reserved (+218 MiB for the draft head) -> 3119 slots
strata generate: expert cache 4060 slots, 7.73 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
so a reply can differ slightly from a run without the cache (same quality:
bench/results/2026-09-27-cache-parity).
PROFILE, ranked by routing frequency, no eviction.
strata generate: pre-filled 4060 of 4060 slots from the profile in 0.5 s (17790 MB/s); slot 0 verified
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 15 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.40.3)
strata generate: token graph hit path: 4060 resident experts, decided on the device
strata serve: prompt chunk auto: 8192 tokens, a 384-slot ring
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata verify: window up to 6 tokens, 74.7 MiB of device buffers
strata mtp: draft head over 106299 tokens (212.9 MiB)
strata serve: 178 MiB of VRAM free with everything loaded - LOW: requests may stall; add --vram-reserve-mib 1034 to the config's args (or lower --max-context)
strata verify: capturing the 6-token window (178 MiB of VRAM free)
strata verify: captured the 6-token window (upload no error, sync no error)
strata verify: capturing the 1-token window (144 MiB of VRAM free)
strata verify: captured the 1-token window (upload no error, sync no error)
strata verify: capturing the 4-token window (112 MiB of VRAM free)
strata verify: captured the 4-token window (upload no error, sync no error)
strata serve: prompt 19 tokens = 0 reused + 19 read in 636 ms (29.9 tok/s), 2 generated in 242 ms (8.3 tok/s), drafts accepted 3 of 3, 1 checkpoints
strata serve: decode expert cache hit rate: 42.8% (706 hits / 1648 lookups); 752 more read by the GPU over PCIe or from another GPU (31.3% of all 2400 routed)
strata verify: capturing the 2-token window (73 MiB of VRAM free)
strata verify: captured the 2-token window (upload no error, sync no error)
strata verify: capturing the 3-token window (39 MiB of VRAM free)
strata verify: captured the 3-token window (upload no error, sync no error)
strata serve: prompt 4096 tokens = 0 reused + 4096 read in 2301 ms (1779.7 tok/s), 256 generated in 4280 ms (59.8 tok/s), drafts accepted 155 of 212, 1 checkpoints
strata serve: decode expert cache hit rate: 80.3% (105981 hits / 131930 lookups); 19270 more read by the GPU over PCIe or from another GPU (12.7% of all 151200 routed)
strata serve: prompt 4096 tokens = 0 reused + 4096 read in 1887 ms (2170.6 tok/s), 256 generated in 3686 ms (69.5 tok/s), drafts accepted 148 of 212, 1 checkpoints
strata serve: decode expert cache hit rate: 85.6% (119145 hits / 139204 lookups); 14876 more read by the GPU over PCIe or from another GPU (9.7% of all 154080 routed)
strata serve: prompt 4096 tokens = 0 reused + 4096 read in 1876 ms (2183.4 tok/s), 256 generated in 3799 ms (67.4 tok/s), drafts accepted 140 of 200, 1 checkpoints
strata serve: decode expert cache hit rate: 86.2% (119197 hits / 138209 lookups); 13471 more read by the GPU over PCIe or from another GPU (8.9% of all 151680 routed)
strata serve: prompt 8192 tokens = 0 reused + 8192 read in 3357 ms (2439.9 tok/s), 256 generated in 3336 ms (76.7 tok/s), drafts accepted 154 of 218, 1 checkpoints
strata serve: decode expert cache hit rate: 86.6% (121460 hits / 140328 lookups); 13272 more read by the GPU over PCIe or from another GPU (8.6% of all 153600 routed)
strata serve: prompt 8192 tokens = 0 reused + 8192 read in 2975 ms (2753.9 tok/s), 256 generated in 3387 ms (75.6 tok/s), drafts accepted 143 of 202, 1 checkpoints
strata serve: decode expert cache hit rate: 86.4% (119670 hits / 138552 lookups); 13128 more read by the GPU over PCIe or from another GPU (8.7% of all 151680 routed)
strata serve: prompt 8192 tokens = 0 reused + 8192 read in 2903 ms (2821.9 tok/s), 256 generated in 3386 ms (75.6 tok/s), drafts accepted 140 of 204, 1 checkpoints
strata serve: decode expert cache hit rate: 87.5% (123895 hits / 141613 lookups); 11987 more read by the GPU over PCIe or from another GPU (7.8% of all 153600 routed)
strata serve: prompt 16384 tokens = 0 reused + 16384 read in 5499 ms (2979.5 tok/s), 256 generated in 3313 ms (77.3 tok/s), drafts accepted 154 of 212, 1 checkpoints
strata serve: decode expert cache hit rate: 85.7% (117192 hits / 136810 lookups); 13910 more read by the GPU over PCIe or from another GPU (9.2% of all 150720 routed)
strata serve: prompt 16384 tokens = 0 reused + 16384 read in 5559 ms (2947.4 tok/s), 256 generated in 3259 ms (78.6 tok/s), drafts accepted 159 of 209, 1 checkpoints
strata serve: decode expert cache hit rate: 85.3% (113216 hits / 132736 lookups); 14144 more read by the GPU over PCIe or from another GPU (9.6% of all 146880 routed)
strata serve: prompt 16384 tokens = 0 reused + 16384 read in 5498 ms (2979.7 tok/s), 256 generated in 3502 ms (73.1 tok/s), drafts accepted 143 of 208, 1 checkpoints
strata serve: decode expert cache hit rate: 85.1% (118676 hits / 139465 lookups); 14615 more read by the GPU over PCIe or from another GPU (9.5% of all 154080 routed)
strata serve: prompt 32768 tokens = 0 reused + 32768 read in 10864 ms (3016.3 tok/s), 256 generated in 3313 ms (77.3 tok/s), drafts accepted 158 of 219, 2 checkpoints
strata serve: decode expert cache hit rate: 85.8% (118583 hits / 138239 lookups); 13921 more read by the GPU over PCIe or from another GPU (9.1% of all 152160 routed)
strata serve: prompt 32768 tokens = 0 reused + 32768 read in 10809 ms (3031.6 tok/s), 256 generated in 3467 ms (73.8 tok/s), drafts accepted 155 of 230, 2 checkpoints
strata serve: decode expert cache hit rate: 86.7% (125563 hits / 144825 lookups); 14055 more read by the GPU over PCIe or from another GPU (8.8% of all 158880 routed)
strata serve: prompt 32768 tokens = 0 reused + 32768 read in 10800 ms (3034.1 tok/s), 256 generated in 3624 ms (70.6 tok/s), drafts accepted 149 of 232, 2 checkpoints
strata serve: decode expert cache hit rate: 86.7% (130031 hits / 150051 lookups); 14109 more read by the GPU over PCIe or from another GPU (8.6% of all 164160 routed)
strata serve: prompt 19 tokens = 0 reused + 19 read in 442 ms (43.0 tok/s), 2 generated in 97 ms (20.6 tok/s), drafts accepted 3 of 3, 1 checkpoints
strata serve: decode expert cache hit rate: 43.6% (697 hits / 1597 lookups); 803 more read by the GPU over PCIe or from another GPU (33.5% of all 2400 routed)
strata verify: capturing the 5-token window (0 MiB of VRAM free)
strata verify: captured the 5-token window (upload no error, sync no error)
strata serve: prompt 4096 tokens = 0 reused + 4096 read in 1867 ms (2194.2 tok/s), 256 generated in 3504 ms (73.1 tok/s), drafts accepted 157 of 215, 1 checkpoints
strata serve: decode expert cache hit rate: 83.6% (112080 hits / 133989 lookups); 16731 more read by the GPU over PCIe or from another GPU (11.1% of all 150720 routed)
strata serve: prompt 4096 tokens = 0 reused + 4096 read in 1844 ms (2221.6 tok/s), 256 generated in 3276 ms (78.2 tok/s), drafts accepted 153 of 209, 1 checkpoints
strata serve: decode expert cache hit rate: 87.0% (119435 hits / 137329 lookups); 12911 more read by the GPU over PCIe or from another GPU (8.6% of all 150240 routed)
strata serve: prompt 4096 tokens = 0 reused + 4096 read in 1938 ms (2113.8 tok/s), 256 generated in 3549 ms (72.1 tok/s), drafts accepted 149 of 219, 1 checkpoints
strata serve: decode expert cache hit rate: 84.6% (120167 hits / 142031 lookups); 15889 more read by the GPU over PCIe or from another GPU (10.1% of all 157920 routed)
strata serve: prompt 8192 tokens = 0 reused + 8192 read in 2921 ms (2804.9 tok/s), 256 generated in 3403 ms (75.2 tok/s), drafts accepted 142 of 208, 1 checkpoints
strata serve: decode expert cache hit rate: 85.9% (121080 hits / 140950 lookups); 13610 more read by the GPU over PCIe or from another GPU (8.8% of all 154560 routed)
strata serve: prompt 8192 tokens = 0 reused + 8192 read in 2945 ms (2781.6 tok/s), 256 generated in 3332 ms (76.8 tok/s), drafts accepted 147 of 210, 1 checkpoints
strata serve: decode expert cache hit rate: 86.6% (121419 hits / 140283 lookups); 13317 more read by the GPU over PCIe or from another GPU (8.7% of all 153600 routed)
strata serve: prompt 8192 tokens = 0 reused + 8192 read in 2968 ms (2760.3 tok/s), 256 generated in 3352 ms (76.4 tok/s), drafts accepted 148 of 208, 1 checkpoints
strata serve: decode expert cache hit rate: 86.7% (120139 hits / 138571 lookups); 13109 more read by the GPU over PCIe or from another GPU (8.6% of all 151680 routed)
strata serve: prompt 16384 tokens = 0 reused + 16384 read in 5459 ms (3001.4 tok/s), 256 generated in 3259 ms (78.5 tok/s), drafts accepted 158 of 217, 1 checkpoints
strata serve: decode expert cache hit rate: 86.8% (120440 hits / 138688 lookups); 12992 more read by the GPU over PCIe or from another GPU (8.6% of all 151680 routed)
strata serve: prompt 16384 tokens = 0 reused + 16384 read in 5431 ms (3016.8 tok/s), 256 generated in 3256 ms (78.6 tok/s), drafts accepted 160 of 204, 1 checkpoints
strata serve: decode expert cache hit rate: 85.3% (110944 hits / 129998 lookups); 14002 more read by the GPU over PCIe or from another GPU (9.7% of all 144000 routed)
strata serve: prompt 16384 tokens = 0 reused + 16384 read in 5466 ms (2997.2 tok/s), 256 generated in 3741 ms (68.4 tok/s), drafts accepted 143 of 232, 1 checkpoints
strata serve: decode expert cache hit rate: 85.5% (128762 hits / 150603 lookups); 15477 more read by the GPU over PCIe or from another GPU (9.3% of all 166080 routed)
strata serve: prompt 32768 tokens = 0 reused + 32768 read in 10623 ms (3084.7 tok/s), 256 generated in 3480 ms (73.6 tok/s), drafts accepted 148 of 214, 2 checkpoints
strata serve: decode expert cache hit rate: 86.6% (122712 hits / 141748 lookups); 13292 more read by the GPU over PCIe or from another GPU (8.6% of all 155040 routed)
strata serve: prompt 32768 tokens = 0 reused + 32768 read in 10797 ms (3034.9 tok/s), 256 generated in 3578 ms (71.5 tok/s), drafts accepted 146 of 221, 2 checkpoints
strata serve: decode expert cache hit rate: 86.6% (125996 hits / 145459 lookups); 13421 more read by the GPU over PCIe or from another GPU (8.4% of all 158880 routed)
strata serve: prompt 32768 tokens = 0 reused + 32768 read in 10862 ms (3016.7 tok/s), 256 generated in 3643 ms (70.3 tok/s), drafts accepted 149 of 237, 2 checkpoints
strata serve: decode expert cache hit rate: 87.0% (132574 hits / 152458 lookups); 14102 more read by the GPU over PCIe or from another GPU (8.5% of all 166560 routed)
strata serve: prompt 65024 tokens = 0 reused + 65024 read in 21649 ms (3003.6 tok/s), 256 generated in 3470 ms (73.8 tok/s), drafts accepted 159 of 216, 4 checkpoints
strata serve: decode expert cache hit rate: 84.7% (115333 hits / 136235 lookups); 15445 more read by the GPU over PCIe or from another GPU (10.2% of all 151680 routed)
strata serve: prompt 65024 tokens = 0 reused + 65024 read in 21476 ms (3027.7 tok/s), 256 generated in 3473 ms (73.7 tok/s), drafts accepted 148 of 219, 4 checkpoints
strata serve: decode expert cache hit rate: 86.5% (124564 hits / 143936 lookups); 13504 more read by the GPU over PCIe or from another GPU (8.6% of all 157440 routed)
strata serve: prompt 65024 tokens = 0 reused + 65024 read in 20339 ms (3197.0 tok/s), 256 generated in 3202 ms (79.9 tok/s), drafts accepted 151 of 210, 4 checkpoints
strata serve: decode expert cache hit rate: 84.9% (115788 hits / 136448 lookups); 14752 more read by the GPU over PCIe or from another GPU (9.8% of all 151200 routed)
"note":"The manifest carries a SHA-256 per tensor, which the fetcher verifies against the checkpoint's own hashes; the full BF16 shards were not downloaded. The installer then built mtp-q2_0.gguf and the runtime pack under /data/mtp/rt (6.5 GB on disk including the tensors)."
},
"prepared_pack":{
"path":"/data/packs/coder-iq1_m",
"bytes_on_disk":"1.4 GB (the installer's own pack; the experts themselves are served natively from shard 1)"
},
"notes":[
"All three model files matched the published hashes byte for byte, so the numbers below are for the published IQ1_M, not for a re-quantized or truncated copy.",
"The download and the pack preparation were done by the unmodified installer in the official image; nothing was quantized or repacked by hand."
# Community benchmark: RTX 5090 + RTX PRO 4000 Blackwell, UD-Q4_K_XL on the helper-cache config, engine 0.1.40.3
Measured 2026-10-07 on a daily-driver coding-agent box. The config is the two-GPU helper cache: all 48 layers on the 5090, the PRO 4000 holding 5000 extra experts (`--expert-cache-device1` + `--remote-expert-opt`). Standard ladder: 4k / 32k / 128k fresh prompts (nonce first, nothing reused), one warm-up then 3 measured runs per length, `temperature 0`, 256 tokens out (reasoning counts toward the 256). Limits at the end.
- CPU: AMD Ryzen 7 9800X3D (8 cores / 16 threads). RAM: 128 GB. OS: Windows 11 Pro
- Engine: ready-made CUDA 13.0 build (`engine-BUILD.json`), repo at `d5ea713` (v0.1.40.3)
- Background workload: a vLLM embedding server holds about 12 GiB of GPU 1 permanently, which caps the helper cache at 5000 slots; GPU 0 is Strata-only. This is the machine's real state, not a clean bench rig
- No `STRATA_*` environment overrides; no calibration; reasoning at the server default
## Method
-`bench_ladder.py`: prompt = Strata source and docs cut to the target length plus an "explain and propose three refactorings" task, with a random nonce first so nothing is reused from the cache. One warm-up per length, then 3 measured runs; speeds are the engine's own `timings` from the response (`prompt_per_second`, `predicted_per_second`)
- Daily-use context: the same box runs a coding agent (Kilo) at 15K-160K-token depths between the ladder runs; the ladder numbers are cold-prompt, single-stream
## Results
### Standard ladder (3 measured runs per length after one warm-up)
| Context | Actual prompt tokens | Prompt tok/s median (range) | Decode tok/s median (range) | Draft accept |
### Daily use (warm, coding-agent sessions, depth 15K-160K)
Warm decode (prompt checkpoint-reused, second and later requests in a session): 96.5-121.4 tok/s, median ~101, expert cache hit 92.5-95.3 %. Cold prompt reading of 147K-160K-token prompts with 0 reused: 2,090-2,131 tok/s. KV streaming hit VRAM 99.4-99.7 % up to about 47K depth, 95-96 % at 140-160K.
## Limits
- The vLLM server on GPU 1 caps the helper cache at 5000 slots; a clean rig could size it larger
- One model, one quant, one workload family; the daily-use numbers are production sessions (varying prompts and depths), not repeated identical runs
- The helper cache is chosen over the layer split because the pair is lopsided (the PRO 4000 in every verify window slows decode); the split was not benched in the ladder format here
~90K (1 run — marked as single-run in the table). One warm-up request
(not counted) precedes the battery; the server had already been running with
a warmed 100% expert cache, and every measured run is fully cold on prompt
tokens (no reuse).
- TTFT = seconds from request send to the first content delta (streaming);
reasoning was off, so the first token is answer text.
- Timings are the engine's own `timings` block (prompt_ms / predicted_ms);
wall time from the client. No model loading is included (server pre-loaded).
- Memory: VRAM free after load **492 MiB** (reported by the engine);
host RAM ~46 GB resident for the engine + page cache.
## Results
| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range |
strata serve: prompt 809 tokens = 0 reused + 809 read in 1916 ms (422.3 tok/s), 256 generated in 3615 ms (70.8 tok/s), drafts accepted 131 of 223, 1 checkpoints
strata serve: prompt 809 tokens = 0 reused + 809 read in 1906 ms (424.5 tok/s), 256 generated in 3138 ms (81.6 tok/s), drafts accepted 149 of 217, 1 checkpoints
strata serve: prompt 809 tokens = 0 reused + 809 read in 1903 ms (425.1 tok/s), 256 generated in 3323 ms (77.0 tok/s), drafts accepted 145 of 234, 1 checkpoints
strata serve: prompt 17971 tokens = 0 reused + 17971 read in 12507 ms (1436.9 tok/s), 256 generated in 3514 ms (72.8 tok/s), drafts accepted 142 of 226, 1 checkpoints
strata serve: prompt 17971 tokens = 0 reused + 17971 read in 12496 ms (1438.2 tok/s), 256 generated in 3492 ms (73.3 tok/s), drafts accepted 143 of 228, 1 checkpoints
strata serve: prompt 17971 tokens = 0 reused + 17971 read in 12510 ms (1436.5 tok/s), 256 generated in 3517 ms (72.8 tok/s), drafts accepted 142 of 220, 1 checkpoints
strata serve: prompt 44011 tokens = 0 reused + 44011 read in 30422 ms (1446.7 tok/s), 256 generated in 3449 ms (74.2 tok/s), drafts accepted 145 of 204, 2 checkpoints
strata serve: prompt 44011 tokens = 0 reused + 44011 read in 30428 ms (1446.4 tok/s), 256 generated in 3493 ms (73.3 tok/s), drafts accepted 142 of 209, 2 checkpoints
strata serve: prompt 44011 tokens = 0 reused + 44011 read in 30439 ms (1445.9 tok/s), 256 generated in 3366 ms (76.0 tok/s), drafts accepted 150 of 209, 2 checkpoints
strata serve: prompt 119731 tokens = 0 reused + 119731 read in 84886 ms (1410.5 tok/s), 256 generated in 3683 ms (69.5 tok/s), drafts accepted 142 of 220, 4 checkpoints
Loaded 100 of 1303 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Niko1221