0.1.43's gate already had 'not on a layer split' as its first predicate; it printed the line only when the variable was set. A default 0.1.43 split ran without the overlap, so 0.1.43.1's opt-in default changes nothing on a split. Measured on 2x A4000 (IQ3_XXS, layer split, 5 interleaved pairs): v0.1.43, v0.1.43.1 and v0.1.43.1 + STRATA_ADAPT_OVERLAP=1 decode within 0.6% of each other.
On Windows cudaHostRegister of the whole arena fails (the WDDM shared budget is about half the RAM) and the
sliced fallback registers the first ~30 GiB: with Qwen3.8-Flash-Next IQ3_S layers 0-31, so layers 32-47 never
get a PCIe share, while part of the registered prefix holds GPU-resident experts that DMA never reads.
STRATA_PIN_BY_NEED=1 (the arena on one GPU):
- every blob gets a page-aligned stride (PadMap in pinned.hpp, ArenaExpertSource::arena_off, +0.15 % RAM), so any
set of blobs can be registered exactly; the loaders place each blob at its padded offset
- the expert profile's pairs past the estimated GPU-resident count are merged into ranges by rank and registered
before any page is touched, up to STRATA_ARENA_PIN_GIB (29 GiB by default on Windows) or the first refusal
- pinned() / device_alias() / pcie_layer() work on the ranges (a blob is pinned only if wholly inside one);
lock_resident_ranges() keeps the holes between them resident
- one start-up line: "pin by need: N ranges, X GiB pinned, Y % of the non-resident experts"
STRATA_PCIE_MIN1=1: the PCIe share (nmiss * pcie_num) >> 8 rounds to 0 at 1-2 misses per layer; with two or more
misses one goes over PCIe (when its blob is pinned).
RTX 3090 + Ryzen 5 5600X: 28.96 GiB registered = 92 % of the non-resident experts, all 48 layers with a PCIe
share; prompt read +5-7 % at 56K-117K tokens, decode level alone and +2-3 % with --adapt-async 1 and
--pcie-frac 0.6 (bench/results/2026-10-11-community-rtx3090-5600x-arena-pin-by-need, docs/DETAILS.md).
Both opt-in; without them the layout, the loaders and the sliced pin are those of v0.1.42.
In the arena mode the blocking adaptive tier creates and joins a thread every verify window and the next window
waits for its copies (#1863: the time outside the decode timing breakdown). --adapt-async 1 now also runs with the
arena on one GPU, not only in the resident RAM mode. The arena holds every expert, so a round is a copy-in only:
no copy back, no RAM move, no exchange buffers.
- gates for the arena; ArenaExpertSource::blob_at() reads a blob from the job thread without the read counter
- the evicted slot is marked non-resident and uploaded before the round's copies are posted; an expert becomes
resident only once its copy's event has completed
- STRATA_ADAPT_ASYNC_WAIT=1 (arena default): the next tick waits for the round, so the window an expert turns
GPU-resident does not depend on timing; STRATA_ADAPT_PICK_INLINE=1 (arena default): one round per window;
STRATA_ADAPT_ASYNC_BATCH=N caps the copies per tick (default 0 = all); arena rounds are not capped at 96
- adapt_quiesce(): every K/V grow, kv_quiesce and the VRAM command land the round in flight (they only called
apply_pending(true)); a failed round still drains its copies. This also covers the resident mode's async tier.
- STRATA_DECODE_TIMING: "adapt wait X join Y, Z swaps/window"; --adapt-async help text
RTX 3090 + Ryzen 5 5600X, cold prompts: x1.07-1.11 decode, faster on 12 of 12 prompts; 25.9 -> 21.6 ms per window
(bench/results/2026-10-11-community-rtx3090-5600x-arena-async, docs/DETAILS.md).
Opt-in (--adapt-async 1, default 0); the default path is that of v0.1.42. With the tier on, answers are not
bit-exact run to run (which window first computes a swapped-in expert on the GPU).
The GPU's own wiring of the arena ends seconds after its last command buffer, and a Metal residency set did not hold it either
(wired memory 75 -> 3 GB between requests, measured with vm_stat and a probe). Idle, the arena is ordinary anonymous memory, and a
20 GB file written beside it made macOS compress tens of GB of it: the third panic (wired 2.4 GB, file cache 50 GB, compressor 1.9M pages).
mlock keeps it wired (STRATA_METAL_WIRE_ARENA=0/1, default on from 64 GB; not when the fill stopped early). Same write, arena wired,
full speed: compressor +0.7M pages (the engine's non-arena memory), 408 pages swapped, nothing else.
docs/MAC.md: the footprint cap, the wiring and the new variables.
Both panics (watchdogd starved 92 s) had the streaming engine at 95.3 / 89.9 GiB resident, free memory 13 MB, the compressor at 2-4M pages.
macOS keeps a file-cache floor (vm.vm_page_filecache_min, 37.2 GB on the 128 GB M5 Max) and compresses anonymous memory instead of dropping
file cache below it; the arena is anonymous and the GPU wires it only while a command buffer runs, so during the fill and when idle it is
pageable. Measured with a 1 s memory watchdog: old build + 100 GB arena, or auto arena + 40 GB file write + a 16 GB program: compressor 21k -> 4.07M
pages in 10 s, 29,000 pages swapped, pressure warn. The 67% rule counted only cx.allocated() (7.5 GB) while the process held 11-15 GB more, and
67% was too much anyway.
- arena_budget.h: footprint_cap (anonymous memory: arena + Metal buffers + what the process holds, task_info phys_footprint) = 60% of the RAM on
a Mac of 64 GB or more (and at least the file-cache floor + 12 GB left); resident_cap (a resident pack's mapped file pages, measured: UD-Q4 resident,
wired 91.7 GB, 40 GB write + 16 GB program, nothing compressed) = 70%; STRATA_METAL_LEAVE_SHARE for measuring. The explicit
STRATA_METAL_ARENA_GB is kept and flagged when it is over the cap.
- window.cpp logs the planned total (arena + held + n-gram reserve against the cap) and the footprint after the fill.
- the engine refuses to start while a repack or setup's model preparation holds a lock beside the model (STRATA_METAL_IGNORE_REPACK=1 overrides).
- metal_repack.py: unbuffered write(), F_NOCACHE on the shards and the output, fsync every GiB, a pid lock (file cache stays flat; the old tool filled all 128 GB).
- setup_mac.py: no engine while preparing a model (stops setup), start() waits on a live repack lock, resident rule = the engine's resident_cap.
- tests: arena_budget_test.cpp (47 checks), tools/test_setup_mac.py (repack guard).
- CMake project version 0.1.44; setup MIN_ENGINE (0, 1, 44); NVFP4_ENGINE (0, 1, 44) (UD-Q5/Q6 were already 0.1.44).
- setup: a prebuilt zip without strata_kq leaves no stale library next to the new engine (kept in .previous); the
Dockerfile copies libstrata_kq.so into engine/.
- Model menu: families in the order Qwen (GSQ-RCO), Swift, Coder, Unsloth, NVFP4, Qwen3.6; NVFP4 is not listed on AMD
and Intel (the Unsloth line names UD-IQ4_XS there); on NVIDIA its line says native FP4 (RTX 50) or that UD-IQ4_XS is
faster on this card, with the download and RAM need. The Unsloth family lists its five sizes, images on.
- strata_mcp's built-in tables know the NVFP4 sizes and family. tools/test_setup_menu.py.
Variant 6 of gdn_recurrence_variant (sycl/include/strata/prefill/gdn_xe.hpp): the column kernel's four row groups
and FMA chains in the same order, a 16-lane sub-group holding four columns x four row groups, so the cross-row-group
sums are sub-group shuffles and a token needs one work-group barrier instead of five (256 GRFs a thread).
Arc A770 16 GB (xe, oneAPI 2026.1 JIT), same day, a0116b27 + #2190 + #2193 with and without this commit:
gdn_parity recurrence 51.0 -> 24.5 ms at 8,192 tokens, 204-236 -> 110-113 ms at 32,768 (48 heads), 0 mismatches
(state, FP32, FP16). Coder IQ1_M, 128K context, int8 KV, --resident-experts: prompt 386 -> 396 / 407 tok/s at 8K,
478 / 480 -> 511 / 518 tok/s at 32K (2 runs each), the same output ids in every run.
On by default on Alchemist only (strata::intel_gpu_gen); every other card keeps the column kernels, unchanged.
STRATA_GDN_XE=0 turns it off, =1 turns it on for any Intel GPU with 16-lane sub-groups. gdn_parity compares
variant 6 with variant 0 bit for bit (state, FP32 and FP16 outputs) on any Intel GPU and skips elsewhere;
ctest -R prefill_gdn_parity runs only that part.
On the A770 the multi-token GR read's kernels launch too few work-groups to fill the card. With
STRATA_GR_MORE_GROUPS on:
- sliced down kernel: one work-group per (slice, row block), 80 x 6 = 480 groups instead of 80;
every row's partial sum is computed exactly as before, only by a different group.
- gr_down_reduce: 8 slice loads in flight per step, then added one at a time, so x accumulates
slices 0, 1, ... 79 in the same sequence of float adds as the old loop (-fp-model=precise).
- up kernel: 8 columns per group, 320 groups instead of 160 (the kernel is now templated on the
column count; UPM_COLS stays 16 for every other user), and only group 0 writes the per-token lo
values, which every group computes identically.
Default on Alchemist only (strata::intel_gpu_gen); Battlemage and other devices launch the same
kernels as before. STRATA_GR_MORE_GROUPS=1 forces it on, =0 off; gr_bench runs either way.
Measured on an Arc A770 16 GB (xe, oneAPI 2026.1 JIT), same day, a0116b27 + #2190 + the previous
commit (GR_PICK, large GRF), with and without this one: gr_bench 1-6 tokens 118.9-183.5 -> 94.6-148.4 us,
checksums identical for every token count; gr_multi_parity 32 cases OK with STRATA_GR_MORE_GROUPS=0 and =1.
Decode, Coder IQ1_M, 128K context, int8 KV, --resident-experts, STRATA_VERIFY_STEPPED=1, 8K prompt, 256
tokens: 22.12 / 22.35 / 22.55 / 22.65 -> 23.24 / 23.61 / 23.62 tok/s (+3-5%), output ids identical.
Not built or run on Battlemage.
- GR_PICK: a runtime index into the by-value kernel argument (m.a[k]) made IGC copy the whole
GrMulti (~1.2 KB) into every work-item's private scratch (unitrace on an A770: ~40 KB private
per SIMD32 thread in gr_up_multi / gr_down_reduce / gr_norm_split). GR_PICK picks one field with
compile-time indices only (a chain of selects over the uniform kernel argument). Addressing
only: the same values, read and summed in the same order, so the output is bit-identical. This
part is on for every device; it is not measured on Battlemage.
- The sliced down kernel and the multi-token up kernel in large-GRF mode (256 registers/thread);
at 128 the up kernel still spills ~3.4 KB/thread on the A770 (unitrace). Large GRF halves the
threads resident per Xe core, so it is the default on Alchemist only (strata::intel_gpu_gen);
Battlemage and other devices launch the same kernels as before. STRATA_GR_LARGE_GRF=1 forces it
on, =0 off.
Measured on an Arc A770 16 GB (xe, Linux 7.0, oneAPI 2026.1 JIT), a0116b27 + #2190, same day:
gr_bench 1-6 tokens 205-258 us -> 119-184 us (GR_PICK alone, STRATA_GR_LARGE_GRF=0: 186-239 us),
checksums identical for every token count. Coder IQ1_M, 128K context, int8 KV, --resident-experts,
STRATA_VERIFY_STEPPED=1, 8K prompt, 256 tokens: decode 5.37 tok/s (5.88 / 5.93 with
STRATA_DOORBELL_CHECK=1 / STRATA_LEND_MIRROR=0) -> 22.12 / 22.55 / 15.60 (cold-cache run)
tok/s, 20.61 / 20.93 with STRATA_GR_LARGE_GRF=0; the window's pool phase 183-208 -> 34-35 ms
per round; output ids identical where the expert placement matched. Not built or run on
Battlemage or i915.
On an Arc A770 16 GB on the xe driver, Level Zero has no ext_intel_free_memory and get_memory_info reports 0 free
with the right total. device_free_bytes() kept the 0 (the #1440 fdinfo path only replaces a figure that is too
high), so --expert-cache auto sized the caches from 0 bytes. A 0 with a known total is now read as unknown: on
Linux the card's total less this process's DRM fdinfo figure is used, the same source the A750/i915 path uses.
Anything that reports a non-zero free figure is unchanged; off Linux the 0 stands.
When the main layers' experts fit but the MTP block's do not (UD-Q6_K_XL on
32 GB), the main layers are placed as in the all-in-VRAM mode (same text,
the decode graph kept) and only the MTP block's experts go to slots + the
CPU pool (its drafts). To get there when the fit is close: the packed Q8_0
decode copies are given back, the prompt chunk stays 512, and - said in the
log - the attention scores buffer shrinks to 64 MB and the VRAM reserve to
256 MiB. STRATA_Q36_NO_MTP_CPU=1 keeps the plain hybrid mode.
UD-Q6_K_XL on gfx1201: tg128 73-75 -> 80.6 (llama.cpp HIP 81.2).
Many small copies were slower than the one whole-layer copy on a 12 GB card
(RTX 3060, UD-Q4_K_XL at 37 of 256 resident: pp2048 1540 in runs, 1974
whole). gfx1201 UD-Q6_K_XL (205/256) and Q8_0 (176/256) keep the per-expert
copies: pp4096 3504-3588 and 3015-3079.
The server already keys its encoder cache by the normalized picture bytes (sha256, the .sve file
name); it now sends those keys with the request (img_keys=<hex>,..., in prompt order), and the
engine's conversation cache uses key + grid for a picture when the count matches. Without the key
(an older server) it hashes the embeddings as before. Old engines skip the unknown key.
Measured for #2228 (one 1600x1000 screenshot-like picture, strata-vision, fresh processes and
twice in one process): the CPU encoder gives the same bits on a TR PRO 3975WX at 8/15/16/32 threads
and on a Xeon E5-2643 v3 (e2357d8e...), the GPU encoder on an RTX A4000 is reproducible too
(92cf7818...), but CPU and GPU differ. So a picture's embeddings change when the encoder moves
between CPU and GPU, or after an encoder update, and no longer on its own; with the byte key the
cache's matching does not depend on that at all. CUDA/HIP and SYCL. Test: serve/test_server.py
test_pictures_are_keyed_by_their_bytes_for_the_engine.
The prompt's staging buffer took the whole layer from host memory over PCIe
(~19-28 GB per chunk); now the experts with a VRAM slot are copied device to
device and only the others come over PCIe, in runs of consecutive experts.
Same bytes in the staging buffer (greedy ids identical). UD-Q6_K_XL on
gfx1201 (205 of 256 experts per layer resident): pp4096 1935 -> 3511-3601.
Also: the packed Q8_0 decode copies are given back when that lets every
expert fit (STRATA_Q36_KEEP_PACKED=1 keeps them); STRATA_Q36_STREAM_ALL=1
keeps the whole-layer copy.
A checkpoint (or the live state) is reused only when the prompt starts with its tokens AND its
pictures, and a picture is its grid plus the hash of its embedding floats. In #2228 (screenshot-heavy
agent, CPU image encoder) the cache fell back to checkpoints a long way back although the client's
messages were byte-identical; one explanation is a picture encoded again with other bits. When the
deepest state whose tokens match is rejected for a picture, the engine now logs which picture and
where, once per request, so the next log tells this apart from a rewritten history. Log only; what
is reused does not change.
47Hunter47 measured </tool_response> asked verbatim at 0/15 on 0.1.43.1 (it reached the model as
the real closer). 62185aab already marks it; this pins the cases next to the exception that keeps
a wholly wrapped tool result's outer pair: the bare closer, a request that quotes a whole pair, a
message that only starts with the opener, two closers in one sentence.
Qwen3.6-35B-A3B-Q8_0.gguf stores the GDN alpha / beta projections as Q8_0;
the engine read only F32 / BF16 there and refused the file. They are now
widened to F32 at load (fp16 scale x int8, as ggml). q36_llama_dump:
STRATA_DUMP_CTX sets a smaller context / batch for files whose weights leave
no VRAM for 4096-token buffers.
The engine keeps every expert in RAM when they do not all fit in VRAM, so a
size's RAM need is its file plus room for the rest. The largest size this
PC's RAM holds is the default; any size can still be chosen (a bigger one is
asked about, as before, never refused).
- k_attn_mma (sm_80+): m16n8k16 f16 -> f32, 8 warps x 16 queries, 64-key K/V tiles double-buffered with
cp.async, ldmatrix (.trans for V), fp32 online softmax. Bench on the RTX 3060 at 4096 x 4150 keys x 32
heads: 23.7 TFLOPS vs ggml's flash_attn_ext_f16 15.4 (0.376 vs 0.577 s/step); CPU check 7e-5.
STRATA_QI_GGML_ATTN=1 keeps ggml's.
- q, k and v of a block with one weight type load into one [3C x C] tensor: one GEMM and one activation
quantize instead of three (STRATA_QI_NO_QKV_FUSE=1 keeps three).
- A float4 LayerNorm path with tanh(gate) once per step, off by default (STRATA_QI_FAST_LN=1): no measured gain.
RTX 3060, Q8_0 1024, seed 42, 3 interleaved rounds: steady step 2.09 s vs stock 2.61 (1.25x), denoise
17.93 s vs 22.13 (1.23x).
A start right after another engine exited sometimes fails before the engine writes a line: ROCm aborts with
"terminate called after throwing an instance of 'rocr::AMD::hsa_exception'" (or the process exits with an empty
log) while the previous process's GPU queues are still being released (r9700: 1 of ~10 rapid restarts in the A/Bs,
2 of ~40 for the quants runs). Such a start - within 30 s, no engine line other than the runtime's failure, not the
engine's own error exit 1 - is retried with a message saying why; STRATA_START_RETRIES (default 3, 0 = off).
0.1.41 (0391e478) scaled each stage's miss term by 20 GB/s / its probed link, for 2x 4090 D
with CUDA1 on a physical x1 link (1.6 GB/s: x12, K 22 -> 38, prompt ~350 -> ~1400 tok/s).
On 2x Tesla T10 the same rule moved auto from K=23 to K=14: CUDA1 runs 34 of 48 layers, and as
the prompt path is a pipeline the slower stage sets the rate - 256K prompt 1035 -> 795 tok/s,
decode 46.8 -> 39.5; --layer-split 23 on 0.1.43 restores both (xiake-1 in #2007).
split_rules::link_miss_scale: the scale is the fastest stage's link over this stage's, applied
only from a 4x gap (a riser or chipset x1/x4 slot beside an x16 one), capped at 64. A symmetric
rig keeps the fitted formula whatever its link speed; the x1 case still gets ~x15.
STRATA_SPLIT_LINK_ABS=1 restores the 0.1.41 rule. SYCL has no link term (unchanged).
Test: split_rules_test (the x1 case, equal gen3 cards, a 2x gap, caps, unprobed stages).
Not run on the reporter's T10s: the reply asks for the 'link ... -> miss cost' lines.
The native output head is needed only to generate, the batched prompt path's buffers only to read a prompt. With the flag the head is unloaded while a prompt is read (its VRAM counted into the auto cache and the prompt path's hybrid borrowed/owned layout) and read back in before the first decode window. On an 8 GB RTX PRO 2000 Laptop (Windows, IQ3_XXS, 590 MiB reserve) the prompt chunk went from 512 to 2,048 tokens and prefill from 292.5 to 667.1 tok/s on a 29,854-token prompt; reads of up to 512 fresh tokens cost about 0.6-0.75 s more. Off by default: every new path is behind the flag. docs/PREFILL_HEAD_ALIAS.md has the measurements and limits.
The table is read with F_NOCACHE through a bounded row cache (1M rows), so 2 GB was too much to set aside. M4 32 GB,
IQ3_XXS decode after the 6K document, 5 interleaved runs, auto arena (memory_pressure every 3 s, vm_stat swapouts):
keep 7 + 2 GB (54c0de70) 5.93 tok/s median, arena ~8.5 GB; keep 7 + 0.5 6.59 (+11%), ~10 GB; keep 5 + 0.5 7.47
(+26%), ~12 GB; the least free memory 43-57%, no swap-out in any run. The M5 Max still keeps 15% (20.6 GB).
arena_budget_test: 14 checks pass.
ROCm gives out hardware queues per priority. Sharing one with the window's stream, --pcie-mode dma's expert copies
and the host function that raises flag B queue behind the window graph that spins on that flag: on an R9700 every
dma-mode request timed out at layer 1. With its own queue dma mode runs (copies overlap the VRAM-hit compute).
The figure in "[strata] images: the image encoder started in N s, M MiB of VRAM" read the free VRAM right after the
cache's shrink, before the driver's counter showed it (amdgpu sysfs), so it came out low (286-362 MiB against the 1,339
measured otherwise). A 0.2 s wait before that reading, only on a start that shrank the cache.
Opt-in, with the config's "vram_elastic": the GPU encoder does not start with the server or a load. A picture's turn
makes the expert cache keep "elastic_mib" (default 2048) free on top of the reserve and starts the encoder; after
"elastic_idle_s" (default 120) without a picture a thread stops it between requests and the cache grows back to the
reserve (0 when no POST /v1/vram was made: the start's reserve left it a segment or more smaller). A POST /v1/vram
while the encoder runs stacks on its share; an unload stops it. Without "vram_elastic" the server says so and starts
the encoder as before. serve/test_vision_elastic.py pins each case; docs in VRAM_ELASTIC.md and DETAILS.md.
update_install refreshed the draft subset once per installed config, newest first, so a folder that two configs
name ended with the subset of the config used longest ago. A config without "draft_vocab" (cjk) then put the
default subset over the cyrillic one of the config in use. Now the first config that names a folder decides;
separate folders and a start (which installs its own config's subset) are as before.
Checked: tools/test_setup_draft_vocab.py (3 new tests, 10 pass), test_setup_config and test_setup_configs (18 pass).
Measured on the affected install, two alternating runs each, greedy 600-token Russian story: 54.0 / 54.0 tok/s
with the default subset installed by the update, 68.7 / 68.6 with the cyrillic one (1.39 -> 1.90 tokens per window).
A helper for the large-page arena (#1412): run as Administrator, it adds the account to
SeLockMemoryPrivilege with secedit, says that a new logon is needed, and prints the secpol.msc
steps when it cannot run. Changed from the PR: secedit /configure is limited to
/areas USER_RIGHTS (the exported policy is not re-applied as a whole), and the description no
longer claims a speed-up we have not measured. docs/DETAILS.md points to it. Not run on our side
(no Windows box for it).
Co-authored-by: dvasdekis <[email protected]>
Some amdhip64_7.dll builds (AMD TheRock wheels / ROCm 10.2 / 7.2.4 nightly) do not export
hipKernelNameRefByPtr although their import library has it, so strata.exe failed to load with
STATUS_ENTRYPOINT_NOT_FOUND before main. It is only used to name graph nodes under
STRATA_VERIFY_NODES; on Windows it is now looked up with GetProcAddress and a missing one
leaves the node unnamed.
From #2183, its verify.cpp part only. The PR's other part (Windows HIP --pcie-mode auto as
DMA) changes the default path of every Windows AMD user and was not measured on our side; it
is left out.
Co-authored-by: dvasdekis <[email protected]>
An Arc A770 under xe intermittently never started decoding: the main thread waits in Verifier::run's stepped serve
(queue::wait -> urQueueFinish) for a segment that never drains, xe logs nothing (#2194). The A750's same failure was
fixed by draining the other queues before each window (STRATA_WINDOW_SYNC, default 1 on i915 only). Reproduced on
the A750 by turning that off: STRATA_WINDOW_SYNC=0, STRATA_VERIFY_STEPPED=1, --resident-experts, IQ3_XXS, 4K prompt:
1 of 6 runs hung with #2194's stack; with the default drain 0 of 11 (5 unpinned, 6 with the pipelined helper pinned).
The default is now 1 on i915 or on an Alchemist device (intel_gpu_gen), 0 elsewhere; STRATA_WINDOW_SYNC overrides.
Unchanged on the A750 (i915: already 1) and on Battlemage (0). Not run on an A770 under xe (no such card here).
The first version compared only the extra loan with the RAM available at that point, but the
resident RAM copy is pinned after it and holds the lent slots' experts too: in a 30 GB scope on
an R9700 (IQ3_XXS, --resident-experts) 30 GiB were available before the 23.35 GiB copy, so the
rule never fired. The reporter's PC thrashed with 3.4 GB left at 16384 and ran with ~6.4 GB at
8192, hence 6 GiB.
Both NVFP4 packs keep Qwen's vision tower unquantized, so the image encoder every Flash-Next model uses reads pictures
for them; strata-vision opens the text model vocab-only, which is the pack's dense GGUF here. Setup asks about images as
it does for every model (default no; --vision gpu|cpu), the encoder is the file other installs already share
(mmproj-Qwen3.8-Flash-Next-BF16.gguf at ISTA's pinned revision, 907,543,008 B, sha256 b1a82259...).
Checked end to end through the server on an RTX 5070 (budget mode, GPU encoder, 1024 image tokens): a picture with
"CODE 4217", a red square on the left and a blue circle on the right - NVFP4 answered "4217" and described both shapes,
their colours and sides correctly, as IQ3_XXS did on the same picture. VRAM: the expert cache held 393 slots with
images (--vram-reserve-mib 2100 and the encoder) against 982 without (1500).
The gfx12 default (0.1.43) lets a bare --prefill auto reach 16384, which lends the prompt path
about twice the cache slots of 8192 (R9700, Swift 1.5 IQ3_XXS: 3,907 slots / 6.38 GiB against
2,068 / 3.37 GiB); the lent slots' experts are read from the RAM side during the read. On a
30 GB PC with --resident-experts 38-47K prompts went 20 GB into swap with 1.5M major faults
and the engine was killed; --prefill 8192 ran them at ~95 tok/s (bdwilton in #2090).
At start, when the engine's own 16384 ceiling is in effect, the extra slots of a 16384 loan
over an 8192 one are priced at the largest blob; if the RAM available (the cgroup or
--memory-limit-mib cap included) is below that plus 4 GiB, the ceiling stays at 8192 and a
line says why. STRATA_GFX12_CHUNK=1 keeps 16384; an explicit --prefill N / auto:N is never
touched (the 16384 default only applies to a bare auto).
The 'MiB of VRAM free with everything loaded' line and its LOW warning read CUDA0 alone. In
#2192 (2x RTX 2080 Ti 22 GB, Windows, IQ3_S, --batch 4, --layer-split auto, 0.1.43) CUDA0 had
740 MiB free while CUDA1 began its window captures at 153 MiB and fell to 52; the first batch
window then never finished. Both stall dumps show the host thread inside cudaGraphLaunch called
from Verifier::run_slot_rows (the stage still 'idle': it had not reached its per-layer wait), the
pool asleep and the watchdog firing after 60 s - a page-in under WDDM while the other card's
graph spins on a host flag.
Each later card now gets its own line, and below 512 MiB a LOW warning with the
--vram-reserve-later-mib value that leaves ~768 MiB. Said, never changed.
On Windows with only an AMD card neither NVML nor the amdgpu sysfs files answer, so
hardware_static.gpu_name was None and strata_bench (no nvidia-smi, no rocm-smi on Windows)
labelled the run "CPU only" although the engine ran on the card (ByteJuggler in #1587: RX 6900 XT).
telemetry.windows_display_adapters reads DriverDesc, ProviderName, DriverVersion and the memory
size from HKLM's display class (no subprocess, no vendor library); windows_amd_cards drops a
Ryzen's integrated Radeon (512 MiB here) when a real card sits beside it. hardware_static gets
the name and gpu_vram_total_mib; strata_bench uses the same list when nvidia-smi and rocm-smi
give nothing. Load, VRAM use, temperature and power stay unread on Windows AMD (that is the
reader of #1224 / #1434 / #1871). Checked against this PC's registry (RTX 5070 + the Ryzen 7600's
iGPU: both listed, the iGPU's 512 MiB read); tools/test_windows_amd_gpu.py with a fake registry.
The stock split shape (STRATA_STAGE_TRIM=1, 1024 MiB reserve, NO_HOST=1 from strata-sycl.sh) is
refused at the start on a 2x Arc Pro B60 24 GB pair with IQ3_XXS: the second stage is 2230-3731
experts short and the host mirror covers the first stage only. The same config with NO_HOST=0
runs at 39-50 tok/s with 0 faults (jds1-ai in #1980). When the model's experts take more than
80% of the split cards' VRAM, setup now writes NO_HOST=0 into the config (a value the user set
is kept) and prints why and how to turn it back. IQ2_XS on the same pair (74%) keeps NO_HOST=1.
Test: tools/test_setup_sycl.py test_split_short_rule.
- refuse_exit: after the REFUSED message the engine waits for the queued device work, flushes
and _Exit(2)s instead of returning from main. Returning ran the stages' destructors in an
order the UR loader did not survive: on a 2x Arc Pro B60 pair every REFUSED start segfaulted
in libur_loader, and a trimmed one also left an xe page fault (bcs CAT error) on the second
card that hung the next start (jds1-ai in #1980).
- resident_plan_set_mirror takes the table's length; resident_plan uses the mirror and the
usage counts only for a residency slice inside the table they were set for. A later stage
passes its own residency table, so the slice was taken as an offset across allocations and
read (and counted into) another buffer: the stage-1 xe faults with STRATA_VERIFY_NO_HOST=0
and no trim (10 of 10 requests, 0 tokens). Such a miss is now a plain miss, served by the
host. Patch shape as jds1-ai proposed it.
Compile-checked only (no two-card Intel box here).
Under WDDM a stream's work sits in the driver's command buffer until a driver call flushes it.
With the stream-ordered signal the verify window only spins on a pinned int, so the last
layers' copies and their 4-byte signal copies were never submitted: the first request stopped
with "the adaptive tier's copies for layer 45 never landed" (RTX 4070 SUPER, 0.1.43.1, opt-in
STRATA_ADAPT_OVERLAP=1). STRATA_ADAPT_SIGNAL=host worked.
- Windows: the host function is the default signal; STRATA_ADAPT_SIGNAL=copy asks for the copy.
- Every platform: one cudaStreamQuery on the copy stream after the swaps and their signals are
queued (a no-op for the results; it submits the batch under WDDM). Linux keeps the copy
signal as before.
Needs a Windows run with STRATA_ADAPT_OVERLAP=1 STRATA_ADAPT_SIGNAL=copy to confirm the flush.
bench/results/2026-10-12-p40-sm61-kernels: where a prompt and a decode window spend their time on a P40, the kernel
benches and parity runs, every request of the engine A/B (565-token prompt 205 -> 255 tok/s, decode 34-35 -> 37-38
tok/s; with STRATA_DENSE_F32=1 a 7K prompt 360-432 -> 464-474 tok/s), the byte comparison of the answers, and what
was not tested. docs/OLDER_GPUS.md: the sm_61 section and switches, and two traps when measuring on such a card.
As sm_60 got with STRATA_SM60_TABLE, measured on a Tesla P40 (native dp4a):
- native_expert_bench, 18 groups at 2 tokens on the IQ3_S pack's layers, mode 0 -> mode 8, bitwise: IQ3_S 390 ->
255 us, IQ3_XXS 345 -> 252, IQ2_S 290 -> 238, IQ4_XS 328 -> 261.
- kIlRows_61 / kIl1Rows_61 from three runs of mmvq_il_parity --bench --emit-table (identical but two Q3_K cells;
a cell is non-zero only where every run gained 3%).
For FP16 operands with FP32 sums cuBLAS 12.4 has two Maxwell kernels on a Tesla P40 and picks the slow one for most
chunk sizes: qkv (10240 x 2560) runs at 7.6 TFLOPS for T 7030-7040 and at 4.3 from T 7041 (49 -> 85 ms), so a
7K-token prompt reads in 16.3 or 19.5 s by its token count. cublasSgemm on FP32 copies runs at 8.4-9.8 TFLOPS from
T 512. For the five shapes and the sizes the route takes, the result is bitwise the FP16 call's: gemm_f32_parity,
5,201 chunk sizes through Gemm::f16. That is a property of the cuBLAS build (12.4.5.8 measured), hence opt-in.
dequant_kernel's thread reads its 32-value group a byte at a time and writes 32 two-byte values one by one, 64 bytes
from its neighbour's. On a Tesla P40 that is 10 GB/s for Q6_K, and the IQ3_S pack's dense weights cost every prompt
chunk ~0.5 s of it, whatever the chunk's length (a fifth of a 560-token prompt). The wide kernel takes 8 values a
thread: 16-bit loads, one 16-byte store, the same expression per value.
dequant_wide_parity: 360 cases bitwise (FP16 / BF16 / FP32 out, a padded stride, row slices, sane and raw random
blocks); --bench on the P40: Q6_K 10240 x 2560 7.25 -> 0.33 ms, Q4_K 5.2x, Q5_K 3.8x, IQ4_XS 3.6x.
On by default on compute capability 6.1 only (where it was measured); STRATA_DEQUANT_WIDE=0 / =1 for any card.
The router, the GDN alpha / beta and the shared expert's gate (n_out 1-256,
K 2048) ran on hipBLAS/cuBLAS SGEMM with 8x8 / 16x16 tiles (~10% of a
prompt). k_sgemm_nt: (16 TM) x 64 tiles, TM x 4 outputs per thread, K split
so about 128 blocks run, the partials summed in a fixed order
(deterministic). gfx1201 pp4096: 3290 -> 3455 tok/s. STRATA_Q36_BLAS_F32=1
keeps the BLAS SGEMM.
Warm-up: a throw-away prompt chunk, a decode step and (with drafts) a
speculative step at load make the code objects, BLAS kernels and graphs, so
the first request runs at full speed (pp4096 rep 0: ~1770 -> 3273 tok/s; a
34-token first prompt: 1032 -> 48 ms). --no-warmup skips it.
Hybrid mode with MTP exact by default: the MTP block is loaded whenever the
file has one (so the VRAM layout and the slot count do not depend on --mtp;
the prompt pass through it only runs with drafts); decode and verify rows'
routing is counted only for the rows kept; the slots are rebalanced exactly
at every 32nd position and a verify window never steps over one.
arena_budget.h: the rule as plain arithmetic, tested by metal/tests/arena_budget_test.cpp (13 checks, run on the
M4). The M5 Max froze (2026-10-13 02:08) with an 89 GB arena planned from 'available - 7 GB' beside a 50 GB n-gram
table: macOS now keeps 15% of the RAM (20.6 GB there), the table's mapped bytes (--ple-io ram) or 2 GB (read row by
row) come out too, and just before the arena is committed the free memory is asked again - less than planned: the
arena is laid out again smaller instead of failing. STRATA_METAL_KEEP_FREE_GB / STRATA_METAL_ARENA_GB override.
stream.cpp: arena buffers grow past 8 GB when 4 of 8 GB cannot hold the arena (STRATA_METAL_ARENA_CHUNK_GB).
The coordinator suggested max(12 GB, 15%); on the 32 GB M4 (Q2_0 streamed, 200 tokens, memory_pressure every 3 s)
12 GB kept cut the arena 9.4 -> 4.4 GB and decode 7.8 -> 4.8 tok/s, while 7 GB kept left macOS 55% free the whole
run (12 GB: 70%); same tokens. Hence 7 GB below ~47 GB of RAM, 15% above. Not yet run on the M5 (down).
iq_host.cpp (Metal's copy of the host helpers): Q6_K among the grouped gate/up formats, as mac-kern's moe_gu_14 /
mm_gu3_14 run them. Before, the engine refused UD-Q5_K_XL ('layer 2's experts are Q6_K/Q8_0 ... no GPU kernels').
strata reads the --native file's architecture at the top of main() and hands
a qwen35moe model to strata::qwen35::run (src/qwen35, now the strata_qwen35
library; strata-q36 wraps the same entry point) before any Flash-Next state
exists; every other file runs exactly as before. strata --families lists the
architectures a build runs. The Qwen3.6 path takes strata's flag names where
they mean the same (--native, --max-new, --pool-workers, --no-token-graph,
--gpu, --mtp N) and skips Flash-Next-only ones in serve mode. The SYCL build
keeps refusing the family, now saying CUDA and HIP run it.
setup: Qwen3.6-35B-A3B is a family in the menu (OWN_PATH_FAMILIES,
install_own_path): size, context, the engine (checked with --families), the
pinned GGUF, its tokenizer, a config with --native/--max-context/--mtp 2 and
the start script; no pack, profile, draft layer or tuning. The engine step is
a function shared with the Flash-Next path. strata_inspect says the family
runs on CUDA and HIP.
- dense_bench: the decode window's native_mmvq calls at the model's shapes and formats, 24 matrices per shape in one
graph (weights from DRAM), 1-4 columns, us / GB/s and an output hash.
- STRATA_MMVQ_WIDE_K_RPW=2|4: Q4_K / Q5_K / IQ4_XS wide kernel with 2 or 4 rows per sub-group sharing each column's
activation loads (apply split into act + dot, the same operations); every hash equal. B70, 3 columns: Q4_K
6144x2560 56.6 -> 27.7 us, Q5_K 2560x12288 89.8 -> 75.5 us, Q6_K unaffected.
- STRATA_MMVQ_WIDE_RPW>1 and STRATA_MMVQ_PF now take the aligned Q6_K loads (they were 2-4x slower than the default
without them); still slower than the default except Q6_K 6144x2560 at RPW 2 (50.2 -> 45.3 us).
- fused GR read: the up step's columns per block (STRATA_GR_UP_COLS 4/8/16/32/64, default 16 = before) and its T
butterflies interleaved (STRATA_GR_UP_ILP=1); gr_bench checksums equal for all. 32 columns: 3 tokens 50.7 -> 48.9 us,
1 token 27.5 -> 29.4; ILP slower.
Engine, B70, IQ3_S with data/expert-profile-sycl.bin, 5 interleaved pairs: STRATA_MMVQ_WIDE_K_RPW=2 STRATA_GR_UP_COLS=32
vs default: decode after a 4K prompt 54.31 -> 54.00 tok/s, story 38.41 -> 38.50, outputs identical. Defaults unchanged.
Found by the quant yardstick (--expert-cache 1) on the NVFP4 GPTQ pack: 'pre-filled 0 of 0 slots', and the verify windows
refused to run. Every other native format's blob is rows x row bytes with 1280 / 2560 rows, a multiple of 256, so its
budget is unchanged.
ROCm gives out its hardware queues per priority level; the draft stream shared one with the verifier's, so the
draft round ran only after the commit graph the verifier had just launched. With a queue of its own the commit
runs beside the draft. R9700, UD-Q4_K_XL: draft wall 5.8 -> 3.5 ms a window (STRATA_DRAFT_TIMING).
The per-layer commit was ~98 graph nodes; it runs between a window and the draft round, and on HIP the round
waits for it (R9700 UD-Q4_K_XL: 2.35 -> 0.65 ms GPU). Each layer's arithmetic is the per-layer call's:
gdn_parity (gdn_commit_layers vs the per-layer commit, n_keep 1-8) and qsa_parity (append_steps_layers vs
copy + append_steps, 3 scalings) are bitwise. STRATA_COMMIT_TIMING=1 prints the commit graph's GPU time.
On the RTX 5070 PC (64 GB, Windows) --mmap-experts let the experts it read stay in the engine's working set: free RAM
fell to 0.6-4.9 GB and the memory guard stopped the engine three times (2026-10-13). --resident-budget-gib 22 with the
same pack kept >= 22.8 GB free. The budget mode is what setup already writes for Unsloth's files (resident_budget_gib:
40 GiB on 64 GB); the pack's experts.bin is the file the rest are read from. test_setup_nvfp4: 64 GB -> budget 40.
Verifier::window_logprobs reads the head rows through window_copy_logits under STRATA_GPU_METAL (the CUDA/HIP path
unchanged: verify.cpp's object is byte-IDENTICAL on P1004 and p620). The prompt's windows sample nothing (head
sampling off), so with STRATA_LOGPOS the Metal window computes the head anyway.
tools/quant_quality.py on the M5 Max against the Q8_0 reference (q8_0.tsv 627558d1, set.ids.json 414634dd), 4,995
positions: UD-Q4_K_XL 92.77% top-1, KL 0.0548 (p99 0.855), PPL 4.043 vs 4.052 (CUDA/HIP: 93.09% / 0.056); IQ3_S
87.85% / 0.1422 (CUDA/HIP: 87.7% / 0.144). The Metal window keeps int8 K/V (the yardstick asks fp16).
tools/image/sdcpp_native: a runner that replaces QwenImage21Runner::compute only (sd.cpp keeps the text
encoder, VAE, sampler and RNG), chosen with --model-args qwen_image_2_1_native=1 in CUDA / HIP builds.
- The text prefix runs once per condition (causal, the t = 0 modulation row) and leaves each layer's
fp16 K/V; each step runs the 32 blocks on the image tokens only.
- GEMMs stay ggml graphs on the runner's backend (sd.cpp's MMQ / BLAS kernels and numerics); fused
LayerNorm+modulate, gate+residual(+next LayerNorm), q/k RMSNorm+RoPE+fp16 K/V, SwiGLU and, on gfx11, a
WMMA flash attention (8 waves x 16 queries, 64-key tiles, fp32 softmax) run on the backend's stream.
- apply.py copies the two files into the pinned checkout and makes two small edits (runner choice,
SD_QI_NATIVE for CUDA / HIP builds); anything not covered falls back to the stock runner.
780M, Q8_0, seed 42: 512 2.11 s/step (stock 2.85), 1024 9.58 s/step (stock 14.2), image 96.5 s (131.8).
A parked pool worker watches the epoch's cache line with UMONITOR and sleeps in UMWAIT (C0.1) until the host publishes or the cycles run out. On a laptop whose CPU and GPU share one power budget the spin's power goes back to the GPU. Unset, 0, no WAITPKG or a non-MSVC build: the spin, unchanged. STRATA_POOL_UMWAIT_C02=1 picks the deeper C0.2 state, STRATA_POOL_UMWAIT_ESC=<cycles> C0.1 first and C0.2 after.
Two places upstream builds QsaShapes by hand from the geometry (chunk_bytes, added since the geometry change was
written, and the conversation-state fingerprint, in both trees). They now call qsa_shapes(g) like layer, mtp,
session, verify and prefill; for every file the guard accepts the values are the ones they built before.
generate.cpp reads the geometry before anything else looks at the model, so a model of another architecture was
refused as "missing qwen4exp.block_count" where 0.1.40.2 said "architecture is 'qwen35', this engine requires
'qwen4exp'" (found running Qwen3.8-27B through --native). read_geometry now checks general.architecture first, and
check_architecture is read_geometry plus geometry_unsupported. gguf_geometry_test: a dense qwen35 file (Qwen3.8-27B's
keys) and a file without general.architecture.
Tested on gfx1151 (HIP build of this branch, Windows): the GGUF tests pass; Qwen3.8-27B UD-Q6_K_XL, Q8_0 and
HauhauCS Q4_K_P are refused by architecture in 0.5 s; Qwen3.8-Flash-Next UD-IQ4_XS, greedy, 256 tokens on three
prompts (39, 48 and 5,568 tokens), gives output identical to the installed 0.1.40.2.
The f32 and array members added for gguf_geometry_test left the braced Kv{...} initialisers short of the struct, which
clang's -Wextra reports as -Wmissing-field-initializers in every test that includes the fixture (MSVC /W4 does not).
make_kv() sets the key and type, and each helper sets its own value. Built with the HIP toolchain (gfx1151):
gguf_geometry_test, gguf_split_test, gguf_reader_test and native_dense_ple_key_test pass, with no warning from the
fixture.
read_geometry (gguf_reader.hpp) fills ModelGeometry from the qwen4exp.* keys, under llama.cpp's names: layers, widths,
GDN, attention, indexer (heads, key width, pooling block from compress_ratios, top-k), rope width, hyper-connections,
experts, RMS epsilon, and the vocabulary from tokenizer.ggml.tokens. The six keys the guard always required stay
required; any other absent key keeps the compiled value. Keys that state one thing twice must agree (the interval and
compress_ratios, key and value head widths, routed and shared expert widths, ssm.inner_size and state x v heads).
check_architecture no longer hard-codes 48 / 2560 / 24 / 2: it reads the geometry and refuses a file the kernels are not
compiled for, naming the key ("qwen4exp.block_count = 40, the kernels are compiled for 48"). Only the expert count,
the experts per token (1..64) and the vocabulary may differ. ModelGeometry moves to core/geometry.hpp (no
dependencies), with kCompiledGeometry; the kernels that fix a dimension at compile time static_assert it there, in the
CUDA/HIP tree and the SYCL copies (plus qsa_prompt_attn_xmx).
generate.cpp takes the geometry and K from read_geometry, and the native embedding takes n_embd / n_vocab from it
instead of a literal 248320. The QSA kernel shapes come from one qsa_shapes(g) (layout.hpp) in layer, mtp, session,
verify and prefill, so n_rot, idx_block and idx_top_k follow the geometry too. Mirrored in sycl/.
Tested: tests/core/gguf_geometry_test.cpp (new, synthetic GGUFs) 28/28, gguf_split_test and gguf_reader_test pass,
built with MSVC /W4 /WX on Windows. `gguf_geometry_test --real` on Unsloth's Qwen3.8-Flash-Next UD-IQ4_XS shard 1
reads exactly the compiled shape and the guard accepts it. Not built here: the engine (no CUDA, HIP or oneAPI compiler
on this PC), so generate/layer/mtp/session/verify/prefill and the kernel asserts are reviewed, not compiled.
The Debian fallback loop ran with check=False on Ubuntu too, so a failed 13.0 install (a
network or apt lock error) went on to 13-1..13-4 and then past the loop without a word. Ubuntu
is back to the single checked install of 0.1.43; Debian tries 13-0..13-9 and fails with
guidance when none installs. Checked against NVIDIA's repos today: debian12 and ubuntu2404
carry 13-0..13-3, debian13 starts at 13-1 (up to 13-4). Two tests added.
The CUDA apt repo keyring and cuda-toolkit-13-x packages exist for
debian12/debian13 as well as ubuntu2204/ubuntu2404, so derive the repo
name from /etc/os-release instead of hardcoding ubuntu. Debian 13's
repo carries no cuda-toolkit-13-0, so try 13-0..13-5 in order (Ubuntu
still gets 13-0 first, same as before).
Checked with tools/test_setup_cuda_repo.py (4 tests, no GPU or network
needed): repo mapping for ubuntu/debian/other and the minor-version
fallback.
opencode puts its <env> block (working directory, today's date) near the start of the system message, before ~40K characters that never change. The prefix cache only matches a prompt's start, so each new folder or day read the whole system turn again. With the switch the block goes to the front of the first user message (string or content parts), for OpenAI and Anthropic requests. Off by default; serve/test_env_to_user.py; docs/DETAILS.md.
The server sends BYIELD for a waiting request under half as long only when a PP line (or the 10 s heartbeat) arrives,
and the engine looks for it at its next boundary. Where a boundary's own PP line is the only line for a while - every
chunk of a --batch read on one GPU, and #2050's pipelined pieces - the BYIELD answers that line and reaches the engine
after it has looked: the read gives way one boundary later, or never when that was the last one it could take. It is in
time only where the engine saves a periodic checkpoint after the PP line (every 16,384 tokens by default).
_control() now also calls on_quiet() whenever the engine has been quiet for half a second (its existing poll), and the
solo and admission reads run the same check there as at a PP line. A BYIELD for a read that has ended is ignored by
the engine, as before. serve/test_parallel.py: the fake engine takes --pp-every K (a PP line and a boundary every K
chunks); a solo read and an admission must give way at their first boundary (both fail on main), and the rule (only
to a much shorter request) holds on the poll too.
Measured on one RTX 5090 (Swift 1.5 IQ3_S, --batch 4, the engine built from main), main's server and this one, two
engine starts each (M F F M): a one-line request sent 0.2-8.6 s into a cold ~33K read, with nothing else decoding and
with one stream decoding. Late (a boundary passed after the request waited): 28 of 40 runs on main, 12 of which never
gave way; 4 of 40 with this change, all requests that started waiting 0.02-0.10 s before that boundary's PP line.
Where main was late the short request's TTFT went 3.56 -> 1.70 s (nothing decoding) and 4.40 -> 1.57 s (one stream);
where main never gave way 4.60 -> 1.51 s and 7.59 -> 2.35 s. Where both gave way at the same boundary, both requests'
TTFTs were the same (medians within 0.02 s). On 2x RTX 5090 with #2050's STRATA_PIPED_READ_PIECE=8192: 6 of 12 late without
it, none with it.
On two equal cards every K whose caches hold all the profiled pairs predicts the same window
time, and the search kept the first one tried: the smallest K, which gives the last stage (it
also carries the MTP drafter) the most layers and lets its cache fill the card to the reserve.
2x Arc Pro B70 with IQ3_S: auto K=24, CUDA1 at 31.8 of 31.9 GiB, ~11 tok/s; --layer-split 30
~54 tok/s (agnaylor in #1440). A full xe card does not fail an allocation, it moves buffers to
system memory.
The SYCL search now scores every placement it tries with its smallest spare room (a stage's
room after its session minus every expert of its layers) and, among the placements within
STRATA_SPLIT_HEADROOM_TOL (default 0.2%) of the best predicted time, takes the one with the
most (split_rules::pick_with_headroom). A log line names both picks when it moves the split.
STRATA_SPLIT_HEADROOM=0 keeps the old pick. CUDA/HIP placement is unchanged.
Test: split_rules_test (a 48-layer two-card model of the report: plain K=24 with 664 MiB spare
on the last card, headroom K=25 with 1728 MiB; tolerance and tie rules).
The three are USER_DEFINED (type 4) tokens like </tool_call> (ids 248058, 248066, 248070), so
the tokenizer matched them inside a message's text as the real call / tool-result markers.
They join TOOL_LITERAL_TAGS: inside a user, system or tool message (and the tool list) they are
encoded as ordinary text; an assistant message replaying raw call text keeps its tokens, as
for </tool_call>.
One exception keeps the template's behaviour: a user message whose whole text is
<tool_response>...</tool_response> (a client that formats tool results itself) keeps that outer
pair as the markers, since the template tells such a turn from a real question by exactly that
pair (last_query_index); the text inside is still marked.
Found by 47Hunter47 in #2026. Tests: serve/test_frontend.py QuotedToolCallCloser (every tag
and role, the assistant replay, the outer pair as string or text parts).
M5 Max: ./setup.sh --model UD-Q4_K_XL --vision yes: shards checked by SHA-256, repacked and verified, the pack, the
image encoder; the config it writes answers the test picture ('STRATA 42', 'Red circle / Blue square / Green
triangle').
The routed experts are placed after the engine's buffers: all in VRAM when
they fit (unchanged), else every expert in host memory with the same number
of VRAM slots per layer (--expert-vram GiB, --expert-slots N). Batches of up
to --hybrid-rows rows run the resident experts on native_expert_grouped and
the others on the CPU ExpertPool (ggml-cpu arithmetic) side by side; bigger
prompt chunks stream each layer's experts into two staging buffers for MMQ
(prompt chunk 2048 in this mode). Slots follow the routing counts (after a
prompt, every 32 decode positions). ExpertPool's native phases split rows by
the layer's n_ff / n_embd instead of Flash-Next's constants (same values
there). STRATA_MMQ_KQ=1 by default for this family.
ple_conv_add reads the conv kernel as half: Unsloth's file stores it F32, so it is narrowed (round to nearest even)
into its own buffer; the GSQ packs' F16 tensor is read in place as before. A file whose HC matrices are Q8_0
already makes no copies (the copy buffer was allocated with 0 bytes and the Metal allocation failed).
M5 Max, UD-Q4_K_XL resident (the repacked 82.5 GB file): greedy prompt1 identical to the CUDA reference (z440),
story first differs at token 36; MTP output equal to no MTP; decode no MTP 60.6 / 60.0, MTP --spec 2 93.9 / 74.0
tok/s; prompts 4K 1,374 / 32K 1,288 tok/s, needles found (4K, 32K). llama.cpp Metal on the same shards: 48.4 /
47.3 decode, 821 / 739 prompt.
With STRATA_FS_SLOTS > 0 and the resident RAM mode's adaptive exchanges, a slot copy issued
from an expert's RAM place (note_miss / the router look-ahead's predict) could land after an
exchange rewrote that place with the outgoing expert's bytes; take() then served expert Y's
bytes for expert X (FS_VERIFY: 19 BAD on clean v0.1.43, reported in #2134).
commit_exchanges, commit_copies and commit_flip now hold every staged expert (in and out) in
the swap space while their bytes move: its slot is dropped (never served again) and no copy
of it is issued until the commit is done (commit_copies holds until commit_flip, so the
asynchronous tier's half-done exchange is covered, and the rotation path too). Unlike the
patch in the issue, a dropped slot is not handed out at once: it waits the same `depth`
finished windows as any slot a window may still read, since a window planned before the
exchange can be reading it. "N dropped by exchanges" joins the swap space's report.
Foresight is opt-in; with STRATA_FS_SLOTS unset nothing here runs. Test:
file_expert_source_test --rotation-gpu (both commit paths, the depth rule, the bytes after).
Based on the analysis and patch in #2134.
Co-authored-by: radoslawsz <[email protected]>
The stdin reader thread is detached and captured main's stop flag, mutex, condition variable,
line queue and EOF flag by reference. The server writes QUIT and closes stdin right after it,
so the reader often wakes with EOF while main is returning: it then locked a destroyed mutex
and wrote to a destroyed deque (undefined behaviour on every platform; the Metal port aborted
on it). The state now lives in a shared_ptr that the thread holds a copy of, so it outlives
main's frame for as long as the reader runs. CUDA/HIP and SYCL engines alike; no change to
what the engine reads or answers.
expert_source.cpp's Linux-only STRATA_TEST_UNMAP_GGUF check had a literal 0x00 byte as its
character constant (since 0.1.43; gcc warned "null character(s) preserved in literal"), and the
#2126 buddyinfo parser in pinned.cu got one the same way (a heredoc dropped the backslash).
Both are the escaped character now. Git took pinned.cu for a binary file because of the byte,
so it was committed with CRLF line endings; it is back to LF (no other change).
tools/test_source_nul_bytes.py fails when any tracked source file holds a NUL byte again.
Prompt chunks run the attention scores and P V as FP16 strided-batched GEMMs
with FP32 accumulation (K / V cells and Q converted per chunk, the causal
softmax writing FP16 probabilities), replacing the FP32 SGEMMs. The prompt's
delta rule splits each V head over 4 column groups (16 row groups of 8 rows,
the norm in its own kernel); decode, verify windows and commits keep one
kernel. pp4096 on gfx1201: 2557 -> 3290 tok/s.
The prefill hook hands the prompt's image rows (sp.embd_rows) to the Metal prompt paths (prompt_chain.inc, the
streaming prompt path, the 8-row windows), which take an image row in place of the token embedding.
mrope_table_set keeps the host view of the server's (t, h, w) table; every window, prompt chunk and draft round copies
the rows it reads into the window's buffer, bound at index 20 of qsa_prep with function constant 24 (mac-kern
29724e5e) - the window, both prompt paths and the draft layer. The cached window's QsaArgs are patched by the last
inline argument (the table is bound after it). setup_mac.py --vision yes: the pinned projector (SHA-256) and
strata-vision built with ggml-metal; the config gets --vision and the encoder.
missing.cpp: a crash prints its call stack and fault address.
M5 Max, Q2_0, ISTA's projector, a test picture (shapes and 'STRATA 42'): 'STRATA 42' and 'Red circle / Blue square /
Green triangle', the same answers as the HIP engine on the R9700 (p620) with the same picture and files, also
through the config setup writes. Text greedy unchanged with and without --vision (prompt1, story, the 1,662-token
chain prompt). prefill.cpp: the CUDA (P1004) and HIP (p620) objects are byte-IDENTICAL (preprocessor-only).
The greedy divergence seen with the fused single-token kernels is not a
kernel bug: fused_gdn_conv_l2 and k_conv_seq leave the same history and
differ by 1 ulp in h (FMA contraction of the 4-tap sum); the Q8_1
activation rounding turns that into a flipped near-tie (p01 token 4,
margin 0.43). KL with either is the same. Drop the switch.
Lane tid wrote block tid % 8 of its superblock, so a sub-group's 2-byte stores landed 64 bytes apart. Lane tid now
takes block tid / 4 (the same 32 (block, part) pairs, the same values) and writes its runs with store_run. UD-Q4_K_XL's
down projections are Q5_1 and UD-Q5/Q6/Q8_0's are Q8_0, so this is most of their FP16 prompt-path dequant.
B70, dequant_bench (32 experts, gate/up + down): Q5_1 7.18 -> 2.83 ms, Q8_0 6.77 -> 3.53 ms; Q4_0 3.21, Q4_1 2.76,
Q5_0 3.74 ms after; the FP16 output hash of every type unchanged. dequant_bench lists Q4_0 / Q4_1 / Q5_0 too.
The lane kernels of native_expert_grouped re-read and re-decoded a Q4_K / Q5_K / Q6_K / Q5_1 / Q8_0 expert row once
per entry of its group (the IQ formats already decode once, Multi). Each of these dots is now split into load (the
weight words and scales) and apply (the activations and the same float expression), and the group's entries go
through the existing multi-entry path in chunks of 4 and 2; every entry keeps its own k order and lane sum, so the
outputs are bitwise the per-entry kernels'. STRATA_KQ_SPLIT=0 keeps the old kernels.
B70 (bmg-g31), 2560 x 640 experts, GB/s of expert bytes, 2- / 4-token windows: Q4_K/Q5_1 362 -> 402 / 369 -> 425,
Q5_K/Q8_0 395 -> 440 / 404 -> 464, Q6_K/Q8_0 259 -> 307 / 256 -> 318, Q8_0 430 -> 479 / 429 -> 502; one token -1..2%.
Experts read from pinned host memory (the NO_HOST mirror) go from 18-19 to 23 GB/s on those windows.
Also: STRATA_KQ_GU_LANES (opt-in; the K-quant / Q8_0 gate/up lane split alone, other bits), native_grouped_parity
--kq (split vs per-entry bitwise, and a CPU float reference with ggml's dequantize_row_* for Q4_K/Q5_K/Q6_K/Q5_1/Q8_0)
and --kqbench (per format pair, device or pinned host memory), Q6_K blobs with a finite scale in that harness,
dequant_bench and kq_cpu_bench for the Unsloth formats in the SYCL build.
--mtp D (0..7) loads blk.<n_layers> (eh_proj, enorm/hnorm, attention + MoE
layer, shared_head_norm) and runs greedy requests as speculative steps:
D chained drafts, one verify window, commit of the accepted rows' GDN state
and conv history. Verify windows and plain decode share the per-token GDN
kernels, so the text is the plain greedy text (10 of 10 prompts identical).
Serve reports accepted/offered drafts in DONE.
- moe_mm.metal: W8<Q4_K / Q5_K / Q5_1 / Q6_K> (the lane's runs k and k + 16 share the sub-block's scale, min and
high bits; 4-aligned code runs read as one uint); mm_down3 takes the paired-run decode on the COOP_K4 layout (TK
32). Prompt per layer at T 2048: Q4_K gate/up 10.6 -> 6.0 ms, Q5_K 15.4 -> 9.0, Q6_K 17.2 -> 14.0, Q5_1 down
5.0 -> 3.3; the same values (moe_mm_parity v3 incl. Q2_0 and IQ3_XXS / IQ4_NL)
- qsa.metal: QSA_MROPE (function constant 24) - interleaved M-RoPE (ggml IMROPE, sections 11/11/10): buffer 20 of
qsa_prep = int32 [cells][3] (t, h, w); rotary pair i of the row at pos_base + cell rotates by the cos/sin row
tab[(pos_base + cell) * 3 + i % 3] (pooled indexer keys at pos_base + 4 b too). qsa_parity QSA_MROPE=1 (random
table vs the CPU reference) / QSA_MROPE=id (identity table: the same numbers as the plain kernel)
The HC read takes, per matrix, its load-time Q8_0 copy or the file's own tensor: Unsloth's Q8_0 down/up need no copy,
their F32 inject rows go to gr_down_q8f (mac-kern 945c0d24). router_route_<type> for BF16 and F32 (opt-in). The M4's
v2 prompt kernels: K-quant gate/up TK 128, Q8_0 / IQ4_XS TK 32, Q5_1 / Q8_0 / IQ4_NL down TK 32. GSQ packs unchanged:
Q2_0 prompt1/story, IQ3_XXS prompt1 and the 1,662-token chain prompt give the same tokens as before (M5 Max).
tools/vision/strata_vision.cpp: no posix_fadvise on macOS (it does not have it); strata-vision then builds there with
ggml-metal.
- unsloth_budget / UNSLOTH_RESIDENT_BESIDE_GB 16, ported from sergqwer's fork (sergqwer/strata-nvfp4 3f3546f6, MIT):
no --resident-budget-gib where the experts and 16 GiB fit the RAM (UD-IQ4_XS 71.4, UD-Q4_K_XL 87.7, UD-Q5_K_XL 107.6,
UD-Q6_K_XL 117.7, Q8_0 135.5 GiB), the budget below that, when asked for, and on a unified-memory APU (#1715).
The menu, --check and the install say which; the menu's default for the family is the largest file whose experts
all fit (UD-IQ4_XS where none does).
- UD-Q5_K_XL / UD-Q6_K_XL / Q8_0: six shards each, sizes and SHA-256 pinned at 38bb39ee; "ram_gb" is the all-in-RAM
need, below it setup asks first (default no) and says how many GiB are read from the SSD. Q5/Q6 need engine 0.1.44
(the Q6_K expert kernels in every build); Q8_0 0.1.38.
- UD-Q4_K_XL: ran on an R9700 (docs/QUANTS.md), so #429's AMD question is gone, and it is not experimental.
- tools/strata_mcp.py mirrors the table and the verdicts. Golden configs re-recorded: only UD-Q4_K_XL on 96 GB
(65536, --kv-resident) and 128 GB (131072) changed, as in the fork.
Tests: test_setup_unsloth Resident and BigFiles (the budget tests force the budget mode where their PC now fits every
expert), test_strata_mcp, every tools/test_setup_*.py.
Co-authored-by: sergiywith <[email protected]>
A driver of its own beside Flash-Next's (generate/verify/prefill untouched), built from Strata's kernels:
- weights from the GGUF: dense Q8_0 on the native MMVQ (with the packed Q8_0 copy), the routed experts
repacked at load into native expert blobs (stride in whole blocks for MMQ), F32 norms / projections;
- the plain pre-norm residual layer driver, the residual add fused with the next norm, the MoE end
(combine + shared-expert gate + add + next norm) in one kernel;
- GDN: the fused conv+L2 and SiLU step for a token, one-launch sequential kernels for a prompt chunk;
- attention without an indexer: split-K decode over 256-cell chunks (positions on the device, so the
decode step is one captured graph), and for a prompt chunk scores / values as FP32 GEMMs per KV head
with a causal softmax;
- prompt projections as FP16 GEMMs (prefill::Gemm::native), the F32 router / alpha / beta as SGEMM,
the prompt's experts on llama.cpp's MMQ (entries sorted by expert on the device);
- decode: native router 256/8, native grouped experts (Q5_K / Q6_K downs added to the grouped
kernels' down list for FF 512), router + shared gate in one product, alpha/beta in one launch.
- serve: READY / GEN (sampling keys) / T / PP / DONE / STOP, the held state reused when a request
extends it; --tokens / --force / --dump-full-logits for tools/qwen35/kl.py; --bench-pp / --bench-tg.
- hip_compat cublas_v2.h: cublasSgemm / cublasSgemmStridedBatched.
setup_mac.py: the Unsloth files by name and SHA-256 (setup.py's pinned revision), repacked into one GGUF for the GPU
with every tensor verified, the n-gram table found by name, --native-dense-gguf the repacked file; setup writes
STRATA_METAL_STREAM to the config so the engine follows its choice. UD-* are behind 'missing' (the Metal expert
kernels for Q4_K/Q5_K/Q6_K/Q5_1 are in progress) and only reachable with --model.
stream.cpp: auto streams when the mapped weights + 12 GiB do not fit the GPU working set (was: experts > 60% of
it, which would have streamed UD-Q4_K_XL on the 128 GB M5 Max). M5 Max: Q2_0 stays resident (same tokens and speed);
M4 32 GB: Q2_0 streams as before.
Shard 1's metadata (without split.*) and every tensor but the n-gram table, byte for byte: dense first, then each
layer's routed experts together; --verify hashes every tensor (SHA-256) against its source shard. For Unsloth's
UD-*_K_XL (4-6 shards cutting through layers). Checked on Q2_0 (M5 Max): 35.05 GiB, 1,223 tensors identical,
written and verified in 33 s; the engine on it (--native-dense-gguf the same file, --ple-gguf the original shard 2)
gives the same tokens as on the original shards, prompt1 and story, at the same speed (114.2 / 107.7 tok/s).
From sergqwer/strata-nvfp4 1688a344 (src/prefill/mmq_expert_bench.cpp), with UD-IQ4_XS's gate/up and down formats
added as comparison cases (an f16 d leads their blocks, as Q8_0's).
Co-authored-by: sergiywith <[email protected]>
row_dot_20_sub8: lane t takes k = t (+ k = t + 16 for t < 4, else + 0) and k = t + 8 (+ 0), then xor 4, 2, 1 - the
order warp_sum adds a K = 20 row in (lane i + 16, then + 8), so the result is the same bits; native_down_multi_kernel
routes TD 40 there (as IQ4_NL's K = 40 row_dot_40_sub8) and the down grid takes 32 rows a block for d_type 40.
nvfp4_decode_bench, RTX 3060, 64 distinct experts: down alone 0.260 -> 0.189 ms at 1 token (226 -> 312 GB/s, 87% of
360); the whole expert 0.626 -> 0.554 ms (89%), 2 tokens 0.686 -> 0.596, 3 0.786 -> 0.661, 4 0.910 -> 0.766 ms (from
2.261 before Split<40>). Output hashes unchanged, nvfp4_expert_gpu_parity unchanged.
The ported NVFP4 decode experts had no Split, so they took the per-entry kernels: every weight was decoded again for
every token of a verify window, and the kernel slowed almost linearly with the tokens per expert. Measured with
nvfp4_decode_bench on the RTX 3060 (64 distinct experts, sm_86, CUDA 13.4; best of 50):
| tokens per expert | before | after | |
|---|---:|---:|---:|
| 1 | 0.714 ms (248 GB/s) | 0.626 ms (283 GB/s) | 1.14x |
| 2 | 1.200 ms | 0.686 ms | 1.75x |
| 3 | 1.730 ms | 0.786 ms | 2.20x |
| 4 | 2.261 ms | 0.910 ms | 2.48x |
(UD-IQ4_XS's IQ3_S / IQ4_NL experts, 15% fewer bytes: 0.512 / 0.578 / - / 0.814 ms.)
- Split<40>: vec_dot_nvfp4_q8_1 split into load (the E2M1 codes through kvalues_fp4, the two sub-block scales) and
apply (the dp4a chain and the float expression in the same order), so the multi-entry kernels apply one decode to
up to four tokens. The multi kernels now apply the NVFP4 tail scales on their outputs as the per-entry kernels do
(s_gate / s_up on gate/up, s_down on down; x * 1.0f for every other format, which is exact).
- ue4m3_to_f32_half builds the value from its bits instead of two ldexpf and branches: equal for all 256 bytes
(nvfp4_decode_bench checks them).
- down: NVFP4's rows are K = 20 calls at n_ff 640, as Q2_0's, so they take row_dot_20_sub16 (2 rows a warp, 16 rows a
block) instead of one 20-of-32-lane row a warp: down 0.361 -> 0.260 ms at 1 token. Its addition tree is
warp_sum's (lane i + lane i+16 first), bitwise.
Bit for bit: the output hash of every case above equals the per-entry kernels' (STRATA_OLD_IQ_MMVQ=1);
nvfp4_expert_gpu_parity on real GPTQ blobs unchanged (1.088-1.168%). Other formats: only the store multiplies by a
1.0f scale, and the down launch grid changes for d_type 40 alone.
nvfp4_decode_bench [gu_type d_type] [groups] [tokens per group] [iters]: native_expert_grouped on distinct random
blobs (64 x 2.76 MB, far past L2), best/mean time, weight GB/s against the card's DRAM peak, and an FNV hash of the
output so a kernel change can be checked bit for bit. NVFP4_BENCH_PHASE=1|2 times gate/up or down alone. Also an
exhaustive check of the UE4M3 decode built from bits against the ldexpf form (all 256 bytes).
- fused_gdn_step_norm_silu: the fused GDN step with the SiLU output gate (Qwen3.6), a compile-time
template flag; fused_gdn_step_norm (Flash-Next, sigmoid) is the same kernel instantiated as before.
- native_router_topk(n_expert, k): 512/10 goes to the existing pinned top-10 kernels unchanged; 256/8
is a templated copy of the same topk-moe contract.
- strata-q36ref --kernels strata (default): native MMVQ (q8_1) for every quantized matmul, dense and
per routed expert (Q4_K/Q5_K/Q6_K/Q8_0), the fused GDN conv+L2 and SiLU step, the native router;
--kernels plain keeps the reference kernels.
- kl.py: answer-only statistics beside all steps, llama.cpp's logits cached between runs.
- quant.metal: QUANT_CTAB (function constant 23): the IQ3_XXS / IQ3_S / IQ2_XXS / IQ2_XS / IQ2_S grids and the sign
table read from ggml-common's constant arrays, no threadgroup copy (same values). Decode moe_gu: 3-7% faster
(IQ3_S-pack T2 window 14.96 -> 14.79 ms, 3 of 3); token_chain default (CHAIN_CTAB=0 off). Prompt mm_gu3: slower
(kept off)
- moe_mm.metal: mm_gu4 (decoded operand shared by MS 16-row token tiles): IQ3_S 9.5 -> 8.6 ms, IQ3_XXS equal, Q2_0
slower; mm_gu5 (weight blocks staged in threadgroup memory, coalesced): slower (12.9-20.6 ms); W8 templated on the
address space for IQ3_S / IQ3_XXS / IQ2_S; probes MM_PROBE 5 (loads only: as slow as the full decode)
- moe.metal MOE_PROBE (function constant 22; no table load)
- finding: the prompt i-quant decode is bound by its scattered small device loads (2-7 a 4-value run, 8 rows a load
instruction), not by the grid lookups or the ALU
--family nvfp4 --model NVFP4 | NVFP4-NVIDIA (MODEL-UPDATE Phase 4). The files come from the packs' Hugging Face
repositories at the commits the fork's setup pins (NVFP4_REPOS: Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata
640a1db9, ...-NVIDIA-NVFP4-Strata cca8f7fe), each checked against its size and SHA-256; nothing is converted on the
PC and no file is mirrored. The FP8 n-gram table and the BF16 embedding are the same file in both: a second install
links them. The MTP draft layer is setup's own, as for every model. The engine gets the fork's model flags
(--native and --native-dense-gguf on the dense GGUF, --ple-gguf, --embd-gguf); the rest of setup's choices (context,
KV, the low-RAM mode on the pack's experts.bin) are the usual ones: 96 GB keeps every expert in RAM, 64 GB with a
32 GB card the resident low-RAM mode. NVIDIA only: on AMD setup stops and names UD-IQ4_XS (the HIP engine has no
NVFP4 kernels). Images: off for these packs. A ready-made engine older than NVFP4_ENGINE (0.1.47, to be set at
release) is replaced by one compiled from this source.
Tests: tools/test_setup_nvfp4.py (fetch from a file:// mirror with SHA-256 checks, the shared file linked, a wrong
file refused, the config name read back, mocked installs on 128/96/64 GB, AMD refused); test_setup_pins covers the
new table. Every tools/test_setup_*.py passes.
Co-authored-by: sergiywith <[email protected]>
- sycl: iq_dequant_f16 / iq_dequant_gu_f16 / moe_combine accept the NVFP4 scale arguments the shared headers now
declare (refused if ever given: the SYCL port has no NVFP4 experts), and the no-MMQ stubs match the new mmq
signatures (the calls were ambiguous with the defaulted ones).
- CMake: list(TRANSFORM ... REPLACE) with \2 of an optional group failed on CMake 3.28 for a plain "120"; two
rewrites, one per form.
- src/qwen35/q36ref.cu (target strata-q36ref, CUDA and HIP): Qwen3.6-35B-A3B on one GPU, the whole
model resident, one token at a time with fp32 activations: plain pre-norm residual, GDN with the
SiLU output gate (32 V heads over 16 K heads, conv 8192), dense gated attention (16 Q / 2 KV x 256,
NEOX RoPE over 64 dims), softmax top-8 router over 256 experts with the sigmoid shared-expert gate,
every weight read in its GGUF form (F32/F16/BF16/Q8_0/Q4_K/Q5_K/Q6_K). --act q8 (default) rounds the
activations of quantized matmuls as llama.cpp's GPU path does; --force / --dump-full-logits for
teacher-forced KL; --dump-layers / --dump-logits for parity.
- tools/qwen35/: q36_llama_dump.cpp (llama.cpp's side through the eval callback), parity.py
(greedy answers + per-layer error), compare_dumps.py (two runs, e.g. the GPU-vs-CPU floor), kl.py
(teacher-forced KL against llama.cpp HIP with the CPU backend as the floor), parity_prompts.json.
The fork was built with MSVC only: on Linux its parity tool's F16C rounding helper failed to inline without the
target flag, and the SYCL tree's own list of CPU expert sources lacked nvfp4_avx512.cpp / nvfp4_avx2.cpp.
Port of sergqwer/strata-nvfp4 c83625c3 (the rule for NVFP4 experts in any layer) onto 0.1.43's default, with its
#1884 nearly-all-cached exception kept. On the fork's qwen-nvfp4-gptq pack the skip moved the answers by
teacher-forced KL 0.035 against it off, about the GPTQ quantization's own error against an all-Q8_0 reference
(0.038), for +11.5% decode (RTX 5090, the fork's measurement). Packs without NVFP4 experts keep the default
unchanged. The final default is the owner's, from our own 3060 A/B and KL against Q8_0.
Co-authored-by: sergiywith <[email protected]>
Without AVX-512 (Zen 2/3, Intel 12th-14th gen) the pool's NVFP4 rows went
through ggml-cpu's vec_dot, which re-decodes a row's blocks for every token of
a window. nvfp4_256_* decode a 64-value block once and apply it to up to 8
tokens. Each token's arithmetic is ggml_vec_dot_nvfp4_q8_0's AVX2 path step
for step (vpsignb, vpmaddubsw + vpmaddwd, the same FMA order, hsum_float_8),
so every row is bit-equal to it: an AVX2 CPU computes what it did before.
nvfp4_avx512_parity checks both kernels, whichever this CPU runs (the AVX2
rows must also be bit-equal; 0 differing rows of 1..9 tokens against the
portable and the native ggml-cpu builds), and is now a ctest
(nvfp4_cpu_parity). One expert's gate+up, Zen 5 core: 1.12x ggml at one
token, 2.00x at 4, 2.35x at 8; under DRAM load (--bw) with 6 threads 47.3
against 24.0 GB/s at 4 tokens, 26.6 against 12.0 at 8. Chat decode on the
RTX 5090 with STRATA_FORCE_AVX2=1 and 6 pool workers: 16.5-16.9 ms a round
against 17.3-17.5 (pool 6.6-6.8 against 7.2-7.3 ms).
STRATA_NO_NVFP4_256=1 falls back to ggml-cpu.
(cherry picked from commit 0c859e15f89054a376db7458bfb298e3e6c81f22 of sergqwer/strata-nvfp4)
Upstream's 04dbe561 allocates cpu_rows at its largest before the walk; the fork's allocation of cpu_ones (row_sd 1 for
the CPU's rows on an NVFP4 pack, b59f5dda) sat inside the reallocation that then never ran, so a staged chunk with the
CPU share copied from a null host pointer: "prefill moe_combine: invalid argument" (2K prompt, qwen-nvfp4-gptq).
(cherry picked from commit 52b272ebb92c9ef49029314da1a695d38f570d88 of sergqwer/strata-nvfp4)
Was 640df4f0, the fork's CPU share. Upstream 0.1.40.2 took it (978d3558, with 667f2eca's blobs taken on the
prefill thread and 4322e241's STRATA_DBG_NAN rows; opt-in there), so the port keeps upstream's code and adds only
what an NVFP4 pack needs: the CPU's rows carry s_down already, so their row_sd entries are 1 (cpu_ones, pinned
beside cpu_rows).
Port note: the release loop keeps 0.1.43's steps_up / tok_up / up_ring releases.
(cherry picked from commit a0c2ca233ac8fe8ec9d342bd319c93404791fa15 of sergqwer/strata-nvfp4)
- The blob keeps its {s_gate, s_up, s_down, 0} tail whenever any projection is NVFP4; the other projection's scale is
exactly 1.0, and every path (GPU decode, CPU pool, prompt path) already multiplies by the tail, so 1.0 is exact.
native_expert_layout (GPU) and native_fmt (CPU) no longer require NVFP4 in all three projections.
- tools/iq_pack.py writes such blobs (1.0 for a non-NVFP4 projection, .scale tensors only for the NVFP4 ones).
- native_expert_parity: the synthetic NVFP4 blob gets its tail (it was left zero, so every NVFP4 synthetic expert
computed 0 and failed), non-unit for NVFP4, 1.0 for the other projection, and the reference applies it.
nvfp4/q8_0, q8_0/nvfp4, nvfp4/nvfp4 and q8_0/q8_0 pass: CPU and GPU within 1e-7, ~1.2% from the float reference
(activation rounding).
(cherry picked from commit 473c0959141b36d28e7a2eb16005f8c8a4e3c8d0 of sergqwer/strata-nvfp4)
0.1.38 took the fork's group gather without the NVFP4 parts: each expert's 16-byte scale tail now goes to its
group slot and the group's last flush zeroes the MMQ tail, in the same launch (as #353 on 0.1.38).
(cherry picked from commit 8f2193f028940b9a6d270203b640562435937a7a of sergqwer/strata-nvfp4)
A ModelOpt NVFP4 checkpoint of Qwen3.8-Flash-Next (its routed experts NVFP4, group 16) runs on the native path.
Rebased on 0.1.31: NVFP4 is type 40 in the per-role format lists (STRATA_GU_FMTS / STRATA_D_FMTS / STRATA_MMVQ_FMTS)
beside Q4_K/Q5_K/Q5_1/Q8_0, and 0.1.31's dq_q8_0 serves both.
- tools/nvfp4_convert.py: llama.cpp's converter with the engine's type policy (BF16 control projections, Q8_0 native
tensors and token embedding, no PLE table); tools/nvfp4_verify.py / nvfp4_verify_gguf.py prove the experts bit for
bit.
- tools/iq_pack.py: expert_layout() gives NVFP4 blobs a 16-byte tail {s_gate, s_up, s_down, 0} (the per-expert
weight_scale_2), so every consumer that moves whole blobs moves the scales; an NVFP4 pack always writes experts.bin
(the GGUF has no tails, so the in-place GGUF read does not apply).
- decode: NVFP4 in the native expert kernels (q8_1, UE4M3/E2M1 decoded in software, bit-identical to ggml); each scale
on its own projection's FP32 output (folding s_down into up put the hidden into FP16's subnormals).
- CPU pool: AVX-512 NVFP4 rows (1.8-3.7x ggml-cpu's), ggml-cpu elsewhere.
- prompt path: llama.cpp MMQ - W4A8 by default (int8 MMQ with Blackwell's FP4 MMA hidden, mmq_nvfp4_w4a8.cu), W4A4
and FP16 through STRATA_PREFILL_NVFP4; the scales applied where they are read.
- CMake: the architectures stay as given. Only mmq_nvfp4_w4a4.cu (the FP4 x FP4 kernel and its FP4 activation
quantizer, symbols renamed) is built with 12x -> 12xa in strata_mmq_w4a4; without its image W4A4 falls back to W4A8.
strata.exe: 55 sm_120 cubins + 1 sm_120a.
- 0.25 of the missed experts over PCIe for NVFP4 packs (their blobs are 2.76 MB).
- Parity tools (mmq_nvfp4_parity, nvfp4_avx512_parity, nvfp4_expert_gpu_parity); HIP guards; docs/NVFP4.md.
Checked on this rebase (RTX 5090): the parity tools as before (decode experts 1.09-1.16%, MMQ W4A8 0.53%, W4A4 8.6%
- the same numbers as the 120a-everywhere build); IQ2_XS 128 greedy tokens identical to 0.1.31 main; the NVFP4 pack
against the 0.1.29-based build at the same expert-cache size: identical tokens on a 24-token and an 8K prompt, in W4A8
and in W4A4 (prefill 4,231 / 4,583 tok/s on 8K).
Port note: the PCIe share base for NVFP4 packs (0.25) sits beside 0.1.43's PcieAuto start values.
(cherry picked from commit 1f8a12858cc4ed80a977767afef6b6c3a754ac63 of sergqwer/strata-nvfp4)
- serve/images.py: OpenAI's images API (prompt, size, n, response_format b64_json|url) plus seed and steps; one
sd-cli process per request with the model card's arguments (8 steps, CFG 1, euler, Turbo's sigmas,
--diffusion-fa), the PNGs returned unchanged. The job takes the service's turn (one job at a time on the GPU)
and the config's gpu_policy says what the text engine does meanwhile: keep, shrink (--vram-elastic), unload
(its conversation saved to a session file and restored after the next start), auto.
- setup.py --images qwen-image (--image-quant, --image-backend, --image-gpu-policy): sd.cpp 2988060 + its ggml
as zips, built for CUDA / HIP / Vulkan (LunarG SDK into third_party when the PC has no Vulkan dev files) / CPU;
DiT quant by GPU memory (Q4_K_M <= 8 GB, Q6_K <= 12, Q8_0 above), VAE and the Qwen3-VL-8B UD-Q4_K_XL text
encoder at pinned revisions, each checked against the Hub's SHA-256. tools/image/install.py adds it to an
existing install.
- Web page: an Images tab when the server has image generation.
- docs/IMAGES.md; tests: serve/test_images.py, tools/test_qimage.py.
IQ3_S runs on Metal since mac-kern's IQ4_XS kernels: recommended from 110 GB (Apple's 128 GB Macs), IQ3_XXS where it
fits resident, then Q2_0. A pack that does not fit Metal's share of the memory streams its experts from the SSD (the
engine decides, f-mac-ssd): setup says so with the M4 numbers, writes --suffix-draft 0 and no --mtp (one token per
step, measured faster there) and does not fetch the draft layer.
session_zero and conversation_checkpoint_save/restore go to the Metal window under STRATA_GPU_METAL (the running
state lives there, not in the session's buffers): reset zeroes its GDN recurrences and conv histories, PLE history
and indexer tails (and the draft layer's); a checkpoint holds the same set, after the pending fold. Before this a
new request carried the last conversation's state (M5 Max, Q2_0: an unrelated request after a story answered
differently from a fresh engine; now the same), and on a streaming Mac the session buffers it zeroed or copied had
been given back (the engine stopped).
Checked on the M5 Max (resident) and the M4 (streaming): an unrelated request after another equals a fresh
engine's answer; a three-turn conversation reuses 35 / 73 of its tokens and its second answer equals a fresh
engine's. Shared files: session.cpp and conversation_state.cpp preprocess IDENTICAL and their objects are byte-
IDENTICAL on CUDA (P1004) and HIP (p620). Also: IQ4_XS's v2 prompt tile (_16_64_32_2), and a call stack when a
given-back allocation is touched.
- ModelFamily from general.architecture (qwen4exp, qwen35moe); the architecture guard accepts both, the
engine's guard (native dense/head, --native) refuses Qwen3.6 with "recognised, not run yet".
- ModelGeometry moved to core/geometry.hpp (+ family, shared-expert FFN, in-file MTP layers, vocab);
geometry_from_gguf reads every shape key, checked against the compiled kernels (Flash-Next files give
exactly the defaults; expert count and top-k stay run-time sizes).
- Per-family tensor layout tables (GDN / attention / nextn MTP block, hyper-connections or plain
pre-norm, indexer or none) and a GGUF directory check; MXFP4 block size in the reader.
- setup: --family qwen36 downloads unsloth/Qwen3.6-35B-A3B-MTP-GGUF (pinned 5bc3e238, size + SHA-256)
UD-Q4_K_XL / UD-Q6_K_XL / Q8_0 (+ mmproj) and says the engine does not run it yet; serve's default
model name per family; strata_inspect recognises qwen35moe.
- Tests: model_family_test (synthetic, --real for headers), test_setup_qwen36, serve/test_family_name.
Conflicts in metal/src/engine/window.cpp resolved: the stream's next-layer routing guess beside router_route
(opt-in); the cached window through the stream's segmented run, the kernel profile off while streaming; host reads of
the weights through W_ (the mapped file, or the dense weights in memory when streaming), also in prompt_chain.inc;
the drafter's prompt rows batched, from either prompt path. The prompt: resident -> prompt_chain.inc, streaming ->
the stream's prompt path (it streams each layer's experts for the chunk); STRATA_METAL_PROMPT=window: 8-row windows.
M5 Max: Q2_0 prompt1/story and IQ3_XXS prompt1 with MTP equal to the no-MTP references; the 1,662-token prompt
through prompt_chain.inc equal to before; Q2_0 streamed (STRATA_METAL_STREAM=1) equal to resident on prompt1. The
1,662-token prompt streamed (the stream's prompt kernels: chunks of 1024, flash QSA) first differs at token 21 of 60
from the resident prompt_chain.inc path (different kernels and chunking).
- gemv.metal: gemv_x<F,LPR,NR,SG> = gemv (KS 1) with the window's prepared x staged in threadgroup memory (bitwise
the same arithmetic); GEMV_PROBE 1/2/3 bench probes (skip later tokens' x loads / combine; rows-outer order)
- units.metal: X<F>::load templated on the pointer's address space (device or threadgroup), same arithmetic
- gemv_parity: GEMV_X=1 sweeps gemvx_*, GEMV_PROBE passes the probe
- M4 finding: a window token's cost is the fp32 combine (1 FMA a weight a token), at ~0.85 T FMA/s (near the M4's
peak); the x loads are ~6 us of the ~30-60 us a token, and staging x in threadgroup memory does not help
setup.py hands a darwin run to metal/setup_mac.py (nothing else changes on a PC); setup.sh fetches a pinned
standalone Python 3.12 into .python on a Mac without Python 3.10+ (no sudo, no Homebrew). The Mac setup checks the
chip, unified memory, macOS (15+) and the Command Line Tools; recommends the pack by memory (IQ3_XXS from ~96 GB,
Q2_0 below; IQ3_S only with --model: the Metal expert kernels lack its IQ4_XS experts); builds the engine with pip's
cmake/ninja, Apple's metal-cpp (pinned SHA-256) and llama.cpp's ggml at the pinned commit; prepares the native pack
and the MTP draft layer; writes strata-<model>.json (backend metal, --spec 2) and run-<model>.sh; starts
serve/server.py. tools/test_setup_mac.py: 10 tests with sysctl/sw_vers/xcode-select mocked.
On the M5 Max (m1002): ./setup.sh --model Q2_0 --gguf-dir ... --yes from a fresh copy: Python fetched, engine built,
pack and config written in 29 s (models already downloaded); the server answered /v1/models, the chat page,
streamed /v1/chat/completions and /v1/messages (Q2_0, 112-121 tok/s). python -m pytest serve -q on the Mac: 705
passed, 11 skipped.
The stdin reader (a detached thread) can take in_mu after the serve scope destroyed it, when the server closes the
pipe after QUIT; on macOS that lock throws and the engine ended with an abort. Inside #if STRATA_GPU_METAL: flush
and _Exit(0) after the QUIT profile save. HIP (p620): generate.cpp preprocesses IDENTICAL and its object is
byte-IDENTICAL against the gated tree (P2003 was unreachable for the CUDA compare; the change is preprocessor-only).
The serve loop asks cvec() every request: Metal returns the empty one (no --control-vector-scaled on Metal yet)
instead of the stub that stopped the engine; stubs.cpp regenerated without it. An uncaught exception now prints its
call stack (backtrace_symbols_fd) before the abort.
The BF16 router GEMV and moe_route as one dispatch beside the shared expert's GEMVs. Greedy output unchanged (Q2_0,
IQ3_XXS, both prompts, on and off). Not the default: in this window the router GEMV already runs beside the
shared GEMVs, and the T2 window measured 13.035 ms GPU with it vs 13.027 without (the fused kernel alone 0.615 ms
vs 0.566 for the two); end to end, --spec 2, 3 interleaved pairs, off -> on: Q2_0 113.61 -> 113.28 / 107.50 ->
106.97, IQ3_XXS 103.51 -> 103.07 / 96.19 -> 96.02 tok/s.
- router.metal (new): router_route<F> - the router GEMV rows as gemv's (LPR 32, KS 1) with 512-thread threadgroups,
the last threadgroup to finish (device counter, device-scope fences) runs moe_route's body and resets the counter;
logits / ids / w / plan bitwise gemv + moe_route. token_chain default (CHAIN_ROUTE=0: separate): T2 13.08 -> 12.93 ms
- gemv.metal gemv_cat: two same-format matrices over one x in one dispatch (ssm_alpha + ssm_beta), bitwise; no gain
in graph mode (CHAIN_ALBE opt-in)
- gr2.metal gr_norm_comb: the MoE combine folded into gr_norm, bitwise; +0.6 ms at T2 (CHAIN_COMB opt-in, not kept)
- units.metal Dec<Q3_K>: qs / hmask as packed_ushort4 loads (same values): T2 2560->10240 33.8 -> 32.2 us
- tests/fuse_parity: the three fusions bitwise for 1..4 tokens
With it a verify window of T > 1 commits its first row in place (commit = 0x100 | 1, mac-kern 56526aed): commit(1)
needs no fold, commit(n) folds rows 1..n-1; a prompt window commits all its rows in the step. Greedy output equal
to no MTP (Q2_0, IQ3_XXS, both prompts, --spec 2 and 4, the window prompt path).
Not the default: M5 Max, --spec 2, 3 interleaved pairs, off -> on: Q2_0 113.35 -> 113.24 / 106.98 -> 106.75,
IQ3_XXS 103.53 -> 102.71 / 96.18 -> 95.88 tok/s. The in-place state write (36 x 3 MB) costs every window; the
fold it saves is only the rounds with no accepted draft (24-36% at --spec 2).
- gdn.metal: with commit = 0x100 | k (1 <= k <= GDN_NTOK) the state is written in place and the conv history to
hist_out right after token k - 1, while every token's y is still computed; commit = 1 / 0 unchanged (the full
commit now writes at the last token inside the loop - same values, same speed)
- gdn_parity: the partial commits are bitwise a k-token commit's state and history, and y bitwise the verify window's
(k = 1 .. ntok - 1, ntok 2..4); gr_parity GR_DOWN=SG_KS picks gr_down's variant (sweep: 10.1-11.8 us at T 2, none
better than 8_8 by more than 7%)
Once, after the 20th cached window of T tokens with the head: the window's dispatches grouped by pipeline and weight
role, each group run by itself (7 runs, median). Profiling only: the reruns move the state, so the text after it
is not meant to be read.
Owner decision. Only the Options default under STRATA_GPU_METAL; CUDA/HIP/SYCL keep 0: on P2003 the CUDA
generate.cpp translation unit preprocesses IDENTICAL and its object is byte-IDENTICAL.
M5 Max, MTP, 200 tokens, greedy output equal to no MTP in every run: Q2_0 114.0 / 107.9 tok/s (prompt1 / story,
--suffix-draft 0; was 112.0 / 86.2 at --spec 4), 119.5 / 108.2 with the default suffix drafter; IQ3_XXS 103.5 /
96.9, 108.9 / 97.0 (was 110.2 / 87.4).
The GDN step writes the prepared y planes (no xprep dispatch per GDN layer); a window whose context is within the
2,051-cell budget skips the QSA score and top-k (qids_ holds the identity, restored after a selecting window; the
cached encode is keyed by it). STRATA_METAL_FUSE=0: off.
M5 Max, Q2_0, 200 tokens, 3 interleaved pairs: no MTP 85.1 -> 87.4 (prompt1) and 85.2 -> 87.3 (story); MTP 110.7 ->
112.0 and 85.3 -> 86.2 tok/s; tokens identical on and off and equal to the no-MTP reference.
prompt_chain.inc: tests/prompt_chain.cpp's chain (6f6c555b) into the decode window's own state - fp16 GEMMs on the
matrix units, grouped expert GEMMs, the GDN prompt scan, QSA K/V and indexer keys into the window's caches - plus
the PLE block after layer 0, every row's final residual for the drafter (its prompt rows in one command buffer),
the history parity, RoPE table and pending fold. Chunks of 2048 rows (STRATA_METAL_PROMPT_CHUNK); COOP_K4 picks
the register-operand kernels on the M5 and the older ones elsewhere. The 8-row-window path stays as the fallback
(STRATA_METAL_PROMPT=window, and prompts under STRATA_METAL_PROMPT_MIN = 32 rows).
M5 Max: Q2_0 4K/32K prompts 1,434/1,462 tok/s (window path 190/174), IQ3_XXS 1,260/1,240. Teacher-forced KL of
the chain's prompt vs the window path's, 28 prompts: 0.0004 (top-1 99.4-99.6%); needles 4K-32K x 10/50/90: the
same answers as the window path (12/12, both models).
- gdn_prompt.metal: gdn_cprep (per head and chunk: gates in log space, A, forward substitution for U and W),
gdn_cstate (per head and 32 state columns: the chunks in order), gdn_cout (the intra-chunk output)
- gdn_prompt_parity: CB -100 = the chunked form; y 3e-7, state 2e-7 vs the decode step; 7.4 ms vs 1.9 ms at T 2048
(cprep 1.5, cstate 4.7, cout 1.4) - it needs the matrix units to pay off
- moe_mm.metal: the paired-run (k, k + 16) decode is taken on the COOP_K4 layout unconditionally (coop_probe now
checks that pairing on both tiles used); the runtime per-element check cost ~30%. The MMA-only ceiling of mm_gu3
(MM_PROBE=1: no decode) is 2.45 ms a layer = 55 TFLOP/s useful, so the decode is what is left. Measured and not
kept: fp16-math Q2_0 decode (slower), a W8<Q2_0> sharing the block scale (slower than two W4)
- glue.metal slc_touch + token_chain CHAIN_PREFETCH: the next HC read's weights touched concurrently with the first
projection GEMV after a read; measured +0.83 ms a token (10.46 -> 11.29, 3 of 3 interleaved) - not kept, opt-in
A prompt of 64+ tokens runs in chunks (STRATA_METAL_PROMPT_CHUNK, default 1024 streaming / 2048 resident) through
mac-kern's prompt kernels (fp16 GEMMs, the GDN scan, QSA over the chunk, grouped expert GEMMs v2 on the M4, v3 on the
M5), the PLE block for T rows, on the decode window's GDN / K/V / PLE state; the draft layer's prompt K/V read the
chunk's final rows. Streaming: the expert GEMMs address arena slots (function constant 20 in moe_mm.metal too), a
chunk's handshake puts the layer's experts in place and streams the next layer's non-resident experts into the LRU
tail behind the compute (the decode-hot experts stay); slots in use by the running layer are pinned. The host
code's own device weight copies are given back on a streaming Mac (purge_device_allocations).
STRATA_METAL_PROMPT_GEMM=0 / _MIN, STRATA_METAL_PROMPT_EXPERTS=v2|v3, STRATA_METAL_PROMPT_QSA=v1|flash.
- qsa_attn_mm: the P V matmul in K = 32 cell runs (K = 64 runs computed garbage on the M4), the output through
threadgroup memory (a direct matmul2d store into 258-float rows lost the threadgroup on the M4); also faster on the
M5 (8.1 -> 7.4 ms at pos 0)
- qsa_flash: the same K = 32 runs (was a hang on the M4) and a fresh per-chunk accumulator folded into the running
one (on the M4 a run does not accumulate into elements rescaled in place); now about equal to attn_mm + merge on
the M5 (9.4 vs 9.5 ms at pos 0, 20.1 vs 20.3 at 30000) and the prompt_chain default there (PC_QSA=flash|mm|v1);
the M4 defaults to v1 attention (faster there) with the GEMM score
- qsa_attn_mm bisection probe (function constant 8, QSA_STOP), qsa_prompt_parity QSA_DEBUG / pipeline report;
coop_layout prints the destination layout too
The dense weights are read into one resident buffer; the routed experts live in an arena of slots filled from the
GGUF with F_NOCACHE preads (several pieces of an expert in flight); the window runs in 48 segments with a shared-event
handshake after each layer's router (mtl_rt run_graph_seg), where the host puts the missing experts in place (LRU)
and reads ahead the experts the next layer's router predicts from this layer's FFN input. The expert kernels address
slots through smap (function constant 20); the resident path is unchanged. STRATA_METAL_STREAM=auto|1|0.
- moe_mm.metal mm_gu3/mm_down3 and qsa_prompt.metal qsa_attn_mm/qsa_flash: with COOP_K4 = false every element is
decoded on its own (the M4 holds runs of 2 n at one k); the experts are correct on the M4 that way
- tests/coop_probe.h: probes the layout (COOP_K4=0/1 overrides); moe_mm_parity, qsa_prompt_parity, prompt_chain
set the constant; prompt_chain defaults to experts v2 and QSA v1 without it (faster / correct on the M4)
- qsa_prompt_parity: NaN-fills the partials before the matrix-unit run (a kernel that writes nothing shows),
line-buffered output, QSA_NOFLASH
STRATA_METAL_LOGPOS=<path>: per decode position of one-token windows, the log-probability of the token that came
next (with --spec-follow, the forced text) and the top 64, in phaseA-tests cuda_kl.py's row format.
STRATA_METAL_PROFILE=1: per window size, host time before the GPU, run_graph wall vs GPU time, commit and draft
round times, printed when the window goes away. Both off unless set.
commit(n) after a window of several tokens no longer submits its own command buffer: the fold of the kept prefix is
encoded first in the next window's (cached per pending count). Nothing reads the GDN state in between.
STRATA_METAL_LAZY_FOLD=0 folds at commit as before.
M5 Max, Q2_0, MTP --spec 4, 200 tokens, 3 interleaved pairs: commit 0.87 -> 0.05 ms a round; prompt1 109.25 ->
110.21 tok/s, story 83.99 -> 85.14 (medians); greedy output equal to no MTP, lazy and not.
0.1.42 let --batch-mtp run on a layer split and made the --batch-groups auto default give way to it: one serial group. On
2x RTX A4000 (IQ3_XXS, 3 pairs) four streams aggregated 62.2 tok/s against 92.7 on 0.1.41's pipelined groups (-33%, 0/3 pairs
faster), two streams 55.7 against 71.4 (-22%). With --batch-groups 2 (which turns --batch-mtp off) 88.3 and 71.1. The start-up
line says why --batch-mtp is off and how to keep it. --adapt-async 1 still gives one group. Solo decode and the answers are
untouched.
- qsa_prompt.metal: qsa_score_mm (fp32 MPP, relu-sum epilogue, the tail block scalar; 6.2 -> 1.1 ms at pos 30000,
T 2048), qsa_flash (whole attention a threadgroup with an online softmax, fp16 out; correct but 27.6 ms vs 20.9 for
attn_mm + merge - not used)
- moe_mm.metal: W8<F> decodes a lane's two runs of a 32-group with one read of the group's scale/sign fields;
IQ3_XXS gate/up 9.3 -> 6.7 ms a layer
- prompt_chain: PC_QSA=mm uses qsa_score_mm too; gate/up 16,32,32,4 for every format; breakdown sees the far
chunk's classes
msl/mtp_chain.metal: mtp_pick (the pick through the draft vocabulary and its softmax probability) and mtp_embed_F
(the picked token's embedding row through mac-kern's unit decoders), so every draft step of a round runs in one
command buffer. At load the GPU embedding is checked against the host's dequant (max |diff| 0 on Q3_K and IQ3_S;
any difference falls back to a command buffer per step). STRATA_METAL_MTP_CHAIN=0 keeps the per-step path.
Fix: with accepted drafts (a > 0) draft 0's embedding overwrote the catch-up's row 0 before the GPU read it; draft
0's row now goes to memb_ row a. The dense attention's cell list is per step.
M5 Max, 200 tokens, --spec 4, greedy output equal to no MTP in every run: Q2_0 109.5 / 84.4 tok/s (was 101.4 /
80.0), IQ3_XXS 107.9 / 86.0 (was 101.5 / 82.8).
- MTP (src/core/mtp.cpp hook under STRATA_GPU_METAL: load / bind / prefill / draft / draft_first): the draft layer
of mtp.hpp on mac-kern's kernels, from the MTP GGUF (tools/mtp_pack.py, beside --mtp's rt dir; the F32 norms
folded to 1 + w as tools/mtp_rt.py does): input branches (msl/mtp.metal: rms, strided xprep, stream combine),
the three HC reads, K/V and indexer keys through qsa_prep into its own cache, dense attention over the last
2,051 cells, the MoE (fused gate_up experts), the final mixer and the head over data/draft_vocab.bin's subset
(STRATA_METAL_DRAFT_VOCAB=full: the whole head). One command buffer per draft step (the next step's embedding
is dequantized on the host). Greedy output with MTP equals the output without it, token for token (Q2_0, two
prompts, BF16 and Q8_0 HC); acceptance equals the HIP engine's on the same prompts (124 of 228; 104 of 291 vs 285).
- Prefill (src/prefill/prefill.cpp hook): init brings up the decode window, run reads the prompt through windows
of 8 rows (no head) and feeds on_chunk (the MTP's K/V); draft_kv declines. Prompt windows of 8 rows give the
same tokens as one-token windows. generate.cpp's own Metal prompt loop is gone again (--prefill auto as on the
PC). One window per process (the prompt path may bring it up before the Verifier).
- msl/ple.metal: the PLE block's norms, gate, conv and residual update on the GPU (same tokens as the CPU version).
- The HC read in Q8_0 by default (the owner's decision; STRATA_HC_Q8=0: BF16): the down / up / inject matrices
quantized at load (290 matrices, 680 MB, 70 ms), read by gr_down_q8 / gr_up_q8.
- The encoded window is cached per (T, head, history parity) and only its positions are patched per window.
- f-mac-kern merged up to 4775d233.
M5 Max, Q2_0, greedy, 200 tokens: Q8_0 HC 85 tok/s without MTP; with MTP 101 tok/s (acceptance 0.54) and 79 tok/s
(acceptance 0.36, a prompt the drafter does badly on everywhere: HIP 71 tok/s there).
- moe_mm.metal: mm_gu3 / mm_down3 - weights decoded per lane straight into matmul2d's right-input cooperative tensor
(single-simdgroup scope, no threadgroup staging or barriers in the K loop); W4<F> 4-value decoders for Q2_0,
IQ3_XXS, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_S, IQ4_NL (quant.metal's deq arithmetic)
- qsa_prompt.metal: qsa_attn_mm (S = Q K^T and P V as single-simdgroup matmuls, int8 K/V decoded into the right
input; same partials as qsa_attn, so qsa_merge is unchanged), qsa_q_f16
- tests: qsa_prompt_parity, coop_layout (the cooperative tensors' element layout), moe_mm_parity --v3;
prompt_chain PC_GU / PC_DN / PC_QSA switches (defaults v3 / v3 / mm)
#1240 stopped creating the drafter's branch stream and its two events on HIP: nothing uses them there, and an R9700
lost prompt speed with them. On gfx103x the stream's existence alone makes prompts read faster. RX 6800, Windows,
STRATA_HIP_PROMPT_F16=1, IQ3_XXS, two runs each: a 7.9K-token prompt at 798-808 tok/s with the stream and 584-596
without, a 30.4K-token one at 855-863 against 553-600; all of it is in the prompt path's "wait copy" phase. The two
events change nothing, so they stay out.
gfx103x creates the stream; other HIP cards do not, as before. STRATA_MTP_SIDE_STREAM=1 / =0 forces either. CUDA is
unchanged. Measured on one card under Windows only.
- token_chain: the HC read defaults to the Q8_0 copies through the 3-dispatch read (q8_3); STRATA_HC_Q8=0 selects
the exact BF16 path (--hc still overrides)
- prompt.metal: route_topk, gr_epi_down/up, swiglu_h, moe_out, gr_norm_h (fp16 xn), gemm_hc_up (the HC up GEMM with
the mixing epilogue; w_up rows interleaved by dq_f16's new perm field)
- gdn_prompt.metal: gdn_pre (parallel conv/norms/gates) -> gdn_scan_r (state scan, one barrier a token) -> gdn_post
- dense_mm.metal: DqArgs gains `perm` (4th field)
- moe_mm.metal: mm_scan zero-fills the unused simdgroup slots; indirect grids for gather/gate-up/down in meta;
MM_PROBE bench constant (function constant 5: pipelines of mm_gu2 need it set)
- tests: prompt_chain (whole prompt in chunks, class breakdown), prompt_parity, gdn_prompt_parity; mm_bench tiles
- window.cpp: a window of 2..8 tokens runs the GDN steps without committing; the projections it feeds them are
kept per layer, and commit(n) folds the kept prefix into the state (gdn_step over n tokens with commit, the same
kernel: bitwise the window's trajectory) and turns the conv-history ping-pong. K/V and indexer cells past the
prefix are rewritten when those positions run again, before anything reads them; the PLE history keeps the n
rows. Checked: greedy output with suffix drafts (windows up to 4, 50 of 59 drafts kept) is the same token for
token as one-token windows.
- generate.cpp (Metal only): --mmap-experts by default (the GGUF shards in place; the window reads them itself):
the 31.6 GB expert arena is not loaded any more (RSS 74 -> 40 GB, the rest is the mapped shard)
- stubs.cpp regenerated (256 entry points left as stubs)
IQ3_XXS vs the CUDA engine (P1004, f-mac, same flags, --kv int8): 200 of 200 tokens equal on the capital
prompt, 124 on a story; following CUDA's tokens 3 of 200 picks differ, at margins 0.07-0.23 (near ties).
The Metal build of `strata` is the shared host code (src/, unchanged except preprocessor-only Metal/arm64 branches)
over two Metal-only parts in metal/src/engine:
- window.cpp: the decode window, the one backend hook (Verifier::init/run/commit/copy_logits under
STRATA_GPU_METAL). It is mac-kern's chain on the model GGUF itself (shard 1 mmapped and wrapped no-copy, every
weight read in place: no upload, no expert cache): HC reads, GDN/QSA mixers with their own state (recurrence,
conv history ping-pong, int8 K/V, indexer), router + plan, routed and shared experts, combine, the final mixer,
the LM head and the greedy pick, encoded as one dependency graph. The embedding is dequantized on the CPU into
the shared buffer; the PLE rows come from the shared table reader, their projections run on the GPU and the
block's norms / gate / conv on the CPU between two command buffers. One-token windows (--mtp-max-t 1) for now.
- host helpers copied from the CUDA kernel files the host code reads while it starts (format tables, sizes, path
switches, the RoPE table/config); every other kernel entry point is a generated stub that stops with its name
(stubs.cpp, tools/gen_stubs.py, from clang's AST of our headers).
Shared host code, all inside #if STRATA_GPU_METAL / __APPLE__ / __aarch64__ (the PC objects do not change):
- verify.hpp/.cpp: the hook (an opaque window pointer; init/run/commit/wait_commit/copy_logits hand over)
- generate.cpp: no x86 ISA refusal; a native pack without --prefill reads its prompt through one-token windows;
the model shard handed to the window
- device.cu: the poison fill through the allocation's CPU view; pool_affinity_linux.hpp: a macOS branch (no
pinning); expert_layout.cpp: no xgetbv asm on arm64
metal/include/metal_compat: cpuid.h (all zero on arm64), linux/mman.h (no hugetlb/THP: the fallback taken).
M5 Max, Q2_0, greedy, "What is the capital of France?": "The capital of France is Paris." - the same 36 first
tokens as the HIP engine (R9700, same flags) on that prompt and 39 on a story prompt, then near-tie flips
(margins 0.08-0.41 when following the HIP tokens); decode 74-75 tok/s.
- kernels/mm_bench.metal + tests/mm_bench: M5 Max MetalPerformancePrimitives matmul2d fp16/bf16 62-65 TFLOP/s
(64x64 tiles), fp32 15, simdgroup_matrix half 14.
- kernels/moe_mm.metal: deterministic GPU plan (block histograms, scan, stable placement; padded per-expert tiles),
activation gather to fp16, mm_gu2 (gate+up+SwiGLU) / mm_down2 with weights dequantized (ref_quant arithmetic)
into threadgroup memory and matmul2d in multiply_accumulate mode, mm_combine. tests/moe_mm_parity: Q2_0 and the
IQ3_XXS-pack formats vs a reference with the same fp16 operands (1e-6..4e-5; vs exact float 2e-4..5e-4).
T=2048, 512 experts, per layer: Q2_0 8.0 ms = 25 TFLOP/s = 3.9 us a token (~5,300 tok/s experts only);
IQ3_XXS gate/up + IQ4_NL down 14.5 ms. The tile idea (dequantized weights from threadgroup memory under matmul2d)
follows yibie/strata-mlx's tile_matmul (MIT); the code is ours.
- kernels/dense_mm.metal: dq_f16 (any dense format to fp16 once a chunk), act_f16, gemm_f16 (MPP): 64-65 TFLOP/s
on the GDN/QSA shapes for 2048 tokens, dequantization ~4% of it; tests/dense_mm_parity (12 formats).
metal/CMakeLists.txt holds both: mac-kern's runtime face, kernels and parity programs as they are, and the device
runtime + the engine build (taken from ../src through include/metal_compat) when METAL_CPP_DIR is given.
runtime_hello is kept out of the kernel tree's test glob.
- gr.metal: gr_down_f<SG, KS, F> / gr_up_f<F> for BF16 (the old names) and Q8_0 (gr_down_q8_*, gr_up_q8);
tests/gr_parity GR_Q8=1: vs double on the dequantized Q8_0 weights, 1/3/8 tokens, M5 Max and M4 pass.
- token_chain --hc q8_3. M5 Max, --fuse --graph, interleaved, 2 rounds: Q2_0 1 token 11.62 (BF16) / 10.59 (Q8_0,
2 dispatches) / 10.38 ms (Q8_0, 3); 3 tokens 16.78 / 17.21 / 16.24-16.31. IQ3_XXS 1 token 12.85 / 11.87 / 11.68;
3 tokens 17.88 / 18.59 / 17.47-17.51. The 2-dispatch read's gain was all Q8_0; its kernels are slower than the
3-dispatch ones at windows. M4 HC read: Q8_0 85 us vs BF16 141-144 us.
- gemv.metal: gemv_pair<F1, F2> for the packs' gate/up format pairs (tests/pair_parity: 8 pairs x 1/3/8 tokens vs
double, 1.4e-7..2.6e-7) and GV_SET(T_Q2_0R).
- token_chain CHAIN_PAIR=1: interleaved A/B, 3 pairs, Q8_0 HC --fuse --graph: 10.60-10.61 ms without vs 10.71-10.72
with (3 tokens 17.23-17.25 vs 17.31-17.33): two concurrent GEMVs and the SwiGLU win; opt-in only.
kernels/gridbar.metal + tests/gridbar_bench: phases of a streaming read separated by an atomic grid barrier vs by
dispatch boundaries (M5 Max). Without atomic_thread_fence(device, seq_cst) the phases read other threadgroups'
stale writes; with it 0 stale reads. Speed: 4.5 MiB phases 10.3 us (10 threadgroups) vs 9.9 us a dispatch, 30-34 us
with 20-40 threadgroups; 0.5 MiB phases 1.5 vs 2.2 us. Not pursued.
All under __APPLE__ / STRATA_GPU_METAL, so the Linux and Windows code is unchanged:
- direct_file.cpp: no O_DIRECT on macOS; F_NOCACHE after open keeps the reads out of the buffer cache
- memory.cpp: F_RDADVISE is the read-ahead hint where posix_fadvise(WILLNEED) is used
- expert_source.cpp: available memory from host_statistics64 (free + inactive + purgeable); the posix_fadvise of a
released expert layer is skipped (the madvise before it already released the mapping)
- generate.cpp: <sys/sysmacros.h> is not on macOS (major/minor come from <sys/types.h>); the CUDA toolkit mismatch
check is CUDA's
- vmm.cpp: the cuMem* virtual memory stays CUDA's (Metal takes the HIP branch: elastic K/V off)
- prefill.cpp: std::max<size_t> where size_t and uint64_t meet (the same type on Linux, distinct on macOS)
- gdn.metal: gdn_step_t<IB>; gdn_step8 (1024 threads, 16 state rows a thread) passes and is not faster than
gdn_step (16.3 vs 16.4 us commit, n=1): the state write after the compute is what a commit adds.
- token_chain: CHAIN_GU / CHAIN_DN / CHAIN_DN4 select the routed-expert tiles; measured in the chain (Q8_0 HC,
--fuse, --graph): gate/up 32,1,4 stays; the Q2_0 down 8,1,4 over 8,2,4 (10.63 vs 10.67 ms, 3 tokens 17.23 vs
17.50 ms), now the default.
- kernels/qsa2.metal: qsa_topk2 - the same selection (ids exact at contexts 700..33000): the radix digit found by a
parallel suffix scan of the 256 bins, cells emitted from contiguous block ranges after one scan. 23 -> 8.2 us at
8K, 54 -> 14.5 us at 32K; token_chain (Q8_0 HC, --fuse, --graph, interleaved): 8K 11.40 -> 11.00 ms,
32K 11.90 -> 11.27 ms.
- qsa_attn2 (chunks of 32, head groups) and qsa_attn3 (lane per cell for q.k): both pass, neither beats qsa_attn at
the full budget (21-24 us, nearly independent of the cell count: a latency chain); QSA_ATTN selects, default 1.
- Read yibie/strata-mlx's OPTIMIZATION_RECORD/ROADMAP (MIT) before this round: its finding that merging kernels cut
the kernel count 43% for 15% of the time matches ours.
kernels/mma.metal: 8-row tiles dequantized with ref_quant's arithmetic into threadgroup memory, x loaded transposed
from device memory, simdgroup_multiply_accumulate; tests/mma_parity: 12 formats pass vs double, every token of a
window bitwise a one-token call. Speed (M5 Max): 60-190 GB/s for 1, 3 and 8 tokens alike, against the per-lane
GEMV's 400-595 GB/s at 1 token and 70-230 at 8: the dequantization through threadgroup memory (~8 B a weight) is
the limit. Only Q2_0 at 8 tokens is faster (115 vs 126 us). Not used by token_chain.
- gdn.metal: GdnArgs.plane: the step writes y as the raw plane plus the other planes and the per-16 sums, so the
out projection needs no prep dispatch.
- gr2.metal: gr_down3 can form bo as the MoE combine itself (GrArgs3.comb).
- token_chain --fuse: the GDN prep in the step, no QSA score/top-k dispatch while the context fits the 2051 budget.
Q8_0 HC, --graph: 10.71 -> 10.49 ms (1 token), 17.13 -> 16.86 ms (3); BF16 2-dispatch 11.91 -> 11.65 ms.
--fold-comb (the combine in the next HC read) is slower: 10.49 -> 11.22 ms; off.
- kernels/gr2.metal: gr_down3 (grid row blocks x streams: each threadgroup forms its stream's R', norm and xn in
threadgroup memory, writes per-stream partial dots; R' ping-pong, never in place) and gr_up3 (lo from the
partials, xn on the fly); BF16 or Q8_0 weights. tests/gr2_parity: both formats vs double on their own weights,
prepared planes bitwise = xprep; Q8_0 moves the read's output by 1.2e-3 relative.
- ref_quant: quantize_q8_0 (ggml's reference rounding), used for the load-time Q8_0 copies (STRATA_HC_Q8's form).
- token_chain --hc 3|2|q8. M5 Max, Q2_0, --graph: 1 token 12.00 / 11.91 / 10.71 ms; 3 tokens 17.21 / 18.02 /
17.13 ms.
MAC-PLAN Phase 1, compat design (the alternative to renaming the host calls, f-mac-design S1-S4).
metal/include/metal_compat goes first on the Metal build's include path, as include/strata/hip_compat does on HIP:
its cuda_runtime.h gives the ~70 CUDA runtime names of the Phase 1 inventory (types, constants, calls) on Metal,
each forwarding to metal/src/gpu/runtime.cpp (metal-cpp: gpuAddress device pointers in one residency set, streams =
queue + command buffer + shared-event timeline, graphs = recorded dispatch lists), and immintrin.h gives _mm_pause /
_mm_sfence on arm64. The CUDA / HIP / SYCL builds never see metal/.
Shared files: four preprocessor conditions also exclude STRATA_GPU_METAL from CUDA-only blocks (the cuMem*
segmented expert cache, the STRATA_VERIFY_NODES graph listing, the MTP stream priority, cudaInitDevice of the remote
experts); nothing else in src/ changes, so the PC objects are byte-identical.
On the M5 Max: runtime_hello passes with the unchanged src/core/graph.cpp, and graph, expert_cache, weights,
native_head, native_dense, layer, verify, mtp and session compile for Metal as they are (-fsyntax-only); prefill,
expert_source and generate stop only at macOS platform items (std::max of size_t vs uint64_t, posix_fadvise /
_SC_AVPHYS_PAGES, sys/sysmacros.h).
- gdn.metal: a q/k conv channel is read at the start of three threadgroups (value heads h, h + 16, h + 32) while
its owner wrote it back in place at its end: on the 10-core M4 the threadgroups run in waves and the later ones
read the new history (gdn_parity failed there, passed on the M5 Max where all 48 start together). The committed
history now goes to hist_out (the caller swaps); gdn_parity checks determinism too. Passes on both Macs.
- gemv_parity --gguf: one real tensor per format of a GGUF (experts: expert 0) vs double; Q2_0 file: 11 formats
pass, rel L2 8e-8..2.7e-7.
- src/gguf_lite: a minimal GGUF v3 reader; the file is mmapped and wrapped no-copy as one MTLBuffer.
- moe.metal addresses experts by gate/up/down offsets and strides (a pack blob or the GGUF's three tensors).
- kernels/glue.metal: the shared expert's SwiGLU (prepared activations) and the combine with its sigmoid gate.
- tests/token_chain: every layer of the model with its own formats, all kernels in one command buffer.
M5 Max, Q2_0, context 1000: 12.83 ms a token (serial encoder), 12.01 ms with --graph (a concurrent encoder,
barriers only at true dependencies); a 3-token window 18.42 / 17.23 ms. Bound: 7.27 ms.
- mtl_rt: run_graph (concurrent encoder, Dispatch::barrier).
- kernels/qsa.metal: q/k RMSNorm + NEOX RoPE (host table), int8 KV append (fp16 scale per 64), the indexer's raw
tail / pooled block keys / dead key, ReLU-sum block scores with the +1e9 tail, a cell-level radix top-2051
(ties to the lowest cell, ascending), split-K GQA attention over the selection, log-sum-exp merge * sigmoid(gate)
written as prepared activations; tests/qsa_parity stage by stage (ids exact; contexts 700..9001).
- kernels/sample.metal: two-stage argmax, ties to the lowest id; tests/sample_parity.
kernels/gdn.metal: one threadgroup per value head, the head's state in registers across the window's tokens;
commit writes state + conv history back, verify does not; tests/gdn_parity vs double (1.7e-7), verify leaves the
state untouched, the first m outputs of a longer window are bitwise a window of m (the fold).
- kernels/gr.metal: gr_norm (folds the previous write, per-stream RMSNorm), gr_down (down + inject rows, silu/hc
epilogue, K split), gr_up (gated mean over the streams, writes the prepared activations); tests/gr_parity vs double.
- moe.metal: moe_route = native_router.cu's softmax/top-10/renormalisation and the plan in one threadgroup
(tests/route_parity: ids exact incl. ties, weights 2e-7); moe_gu/moe_down switch once per threadgroup into a body
specialised for the group's entry count (the predicated 8-way loop cost 2x); Q2_0 down reads raw activations
(T_Q2_0R) so no prep dispatch is needed.
- mtl_rt: warm() before timings (the GPU starts at idle clocks).
- kernels/units.metal: per-format unit decoders (Q2_0, Q3_K..Q6_K with the mask trick, IQ4 via a byte->half2
codebook table, BF16, generic for the rest) and one combine for all; xprep writes the pre-scaled activations
and per-16 sums once per input vector.
- kernels/gemv.metal: y = W x for 1..8 tokens in one pass over the weights, NTOK a function constant, optional
K split across simdgroups; tests/gemv_parity: parity vs double for 12 formats x 20 tile shapes, every token of
a window bitwise equal to a one-token call, and the bench on the packs' projection shapes.
- moe.metal on the same decoders; mtl_rt compiles with safe math by default (no reassociation; measured free).
- tests/alu_bench: instruction rates (fp32/fp16 FMA, int, convert, threadgroup/device/constant gathers).
The dpct migration had dropped the cudaMemcpyAsync D2H and the stream sync, so the resident RAM mode with adaptive swaps committed
whatever the exchange buffer held as the evicted expert (lost expert, wrong weights on a later CPU read). STRATA_CHECK_SWAPS=1 is a
test aid: it compares each staged buffer with the model file and audits the RAM copy at the end of a run.
metal/ is its own CMake project (macOS only), nothing in src/ or include/ changes. MSL is compiled at run time
(the Command Line Tools have no offline metal compiler) with ggml-common.h prepended for the i-quant grids.
- src/mtl_rt.*: a C++ face over Metal (buffers, pipelines with function constants, serial/concurrent encoders,
indirect dispatch, indirect command buffers, GPU timing).
- src/ref_quant.*: CPU reference dequantizers for Q2_0, Q4_0, Q5_0, Q8_0, Q3_K..Q6_K, IQ2_XXS/XS/S, IQ3_XXS/S,
IQ4_NL/XS, BF16, F16; checked bitwise against gguf-py (tests/ref_dump + tools/check_ref.py: 17/17 equal).
- kernels/quant.metal: the formats as per-lane units (decode once, dot with every token of the window).
- kernels/moe.metal: moe_plan (top-k -> groups by expert, deterministic, writes the indirect grids), moe_gu
(gate+up+SwiGLU, grouped), moe_down, moe_combine; tests/expert_parity: parity vs double and the bench.
- tests/rt_bench: M5 Max streaming read ~585 GB/s, ~0.85 us per dependent dispatch, ~170 us commit+wait.
Linux CUDA had it on by default since 0.1.43, measured on an RTX A4000 (+5.5..8.3% decode) and a Tesla P100 (+4.4%). An RTX 5090 Laptop with a native IQ3_XXS pack and --expert-cache-per-layer decodes 21% slower with it, and the overlap is the default only there. The switch is now =1 turns it on everywhere, and the startup log names the state (with the reason when it cannot run), so a decode-rate drop is not the only way to see it.
Unchanged: the copy path, the gate predicates and the tokens (same tokens with or without it).
Fixes#2010
On 2x/4x R9700 the drafter's chain and the last stage's window shared a card and slowed each other (window 13 -> 27 ms) in most engine starts; the draft layer now loads on the first card (STRATA_MTP_STAGE0=0 keeps the old place). Exact flags, tokens identical: 2 cards code +21%, 3 cards +19%, 4 cards +22% over the serial loop. A split reads prompts in 8192 chunks (STRATA_SPLIT_CHUNK16K=1 keeps 16384): 16K prompt +36..53%, 32K +18..29%, same tokens. STRATA_PCIE_AUTO_SPLIT=1 now also runs in the pipelined loop (opt-in).
gfx1103 was missing from the gfx11 matrix-core lists, so the 780M ran the portable verify-window and prompt kernels. Added to the
compile-time lists and gfx_arch_is_gfx11_wmma(). Radeon 780M, Q2_0, 5 interleaved pairs: decode (code) 11.96 -> 18.25 tok/s,
same token ids; prompts unchanged. hip_prefill_wmma_gemm_parity passes on gfx1103 (3520 cases). Identity gate (defaults off,
mg_norepeat 10/10 against the portable build, exact_pp 4K/20K residual dumps identical).
The serve path counts every routed expert on the device when a save path and a mirror exist; the saved ranking mixes the counts with the start ranking (prior worth 800k routed entries, STRATA_PROFILE_PRIOR_ENTRIES). Exact: placement is fixed under NO_HOST, the file applies at the next start. B70, IQ3_S: code +13..19% vs the xe default profile, prose -18%.
0.1.43 moved the QSA step records and the batched token ids to a pinned buffer (#1712). A 4-hour multi-slot load on 2x RX 7900 XTX (parallel 4, vision plus long
prompts, layer split) still faulted after 31 minutes, with the picture rows, the draft tokens and bounds and the non-mapped routing tables still going up from
caller memory. PinnedRing (include/strata/core/pinned_ring.hpp) is a ring of four pinned slots with one event each: the bytes are copied into the next slot after
the copy that last read it is done, the DMA runs from the slot, and the caller may change its source on return. No pinned RAM (or STRATA_TEST_PAGEABLE=1): the
copy is made from the caller's memory and the stream is waited for. A failed upload is returned and reported.
Moved onto it: the picture's embedding rows (gathered per run into one slot), the draft K/V step's tokens and bounds, the PLE rows' pageable fallback, the
non-mapped branch's routed-id download and slot/src/bounds uploads, the peer device's src and bounds tables, and the drafter's prompt records (mtp.cpp).
Copies from pinned arenas and cudaHostAlloc buffers are unchanged. Same bytes reach the GPU, so answers are identical.
The analysis of which copies are still pageable, and the bounce-slot-per-event idea, are mantovaniluca91's (patch C on #1712). This is our own form of it.
collect_machine listed every nvidia-smi card. It now takes the bench's --gpu, else the config's gpu (or --gpu in its args); with none given it names the one card the server reports instead of all of them.
ensure_loaded logged 'loading anyway' for a before_load command that failed and co-loaded two models onto one GPU. A non-zero exit or a timeout now leaves the model unloaded and answers 503 with Retry-After (before_load_ignore_failure restores the old behaviour); before_load_timeout_s sets the 120 s limit; POST /unload?wait=1 (or wait=<s>) waits for the running turn instead of answering busy.
The engine answered 'ERR BGEN: no such free slot' when its view of a slot differed from the server's; the server turned the ERR into an HTTP 400 and the conversation, drawn to the same slot by its held prefix, got it again on every turn. Now the refused slot is parked (busy, no owner) for SLOT_PARK_S, the request waits for / takes another slot, and after BGEN_REFUSALS_MAX refusals it ends with a retryable 503.
The host mirror and the unmirrored count cover the first stage only; a later stage whose cache lacks experts has no mirror entry and, under NO_HOST, no host answers for them (device lost at the first verify window on 2x Arc Pro B60). The start now refuses with the existing wording, or logs the count when NO_HOST is off.
The mapped gather_batch took 16 serial demand faults per token from a cold table with no progress mark; the first chunk of a prompt only has layer 0 to hide it behind, so the stall watchdog stopped the engine. Rows are now hinted 4096 at a time (PrefetchVirtualMemory / MADV_WILLNEED). Answers are unchanged. The 0.1.42 auto-switch to mmap on Windows is opt-in until it is measured there.
The per-layer host function made the overlap depend on the driver's callback thread being scheduled: with the pool's
workers spinning on every other CPU it still cost 7 ms per window on a 20-core desktop even with the thread on all CPUs
(RTX 3060, Core Ultra 7 265: 46.5 tok/s with the warmed callbacks against 56.5 with the overlap off). The copy stream now
ends each layer's copies with a 4-byte device-to-host copy of the layer number into a pinned int the gate polls; same stream
order, no host thread. STRATA_ADAPT_SIGNAL=host brings the host functions back. Swaps, residency and answers are unchanged.
On Linux the CUDA (and HIP) runtime creates its host-callback thread lazily, at the first cudaLaunchHostFunc, and a new
thread inherits the creator's CPU mask. The decoding host pins itself to one CPU (SessionLoopScratch::init), so a thread
created after that - which the adaptive overlap's per-layer callbacks did - ran on the host's CPU and fought the host's
spin-wait: ~1000 us instead of ~4 us per callback, decode 2-4x slower with STRATA_ADAPT_OVERLAP on (#1964, #1975).
oal's warm-up (one no-op callback + sync before ExpertPool) is now unconditional on Linux; the answers, swaps and
residency are untouched. Windows CPU Sets are per thread and not inherited, so Windows is unchanged.
Rarely a window's logits become NaN and the reply degenerates into token 0
('!') repeated (#606); the poison sits in the conversation's K/V and
recurrent state, so with the prompt cache every later turn stays poisoned
until it is read again from token 0. The q8_1 clamps (2b726ba, f6601bf,
7a0e302) remove the overflows found so far; this guard catches whatever
cause is left, and drops the poisoned state instead of serving it again.
On by default, as a fix: STRATA_NAN_GUARD=0 turns it off. It only reads
flags, so an answer without a NaN is the one the engine gives with the
guard off (compared byte for byte on a GPU, below).
Engine:
- every verify-window graph checks its head logits beside the greedy pick
(logits_nonfinite_rows: 16 blocks a row, one flag word each, into mapped
memory); Verifier::note_nonfinite reads them after the sync the window
already does (run, run_slot_rows, batch_poll, pl_finish). No extra sync.
With STRATA_NAN_GUARD=0 no kernel is recorded.
- --serve: a flagged window emits none of its tokens, one stderr line names
the position and row, and the request ends with `DONE ... nonfinite` (a
batch slot: `BDONE <slot> <n> nonfinite`; an admission: BADM <slot> 0).
The next request first drops the live session, every checkpoint, every
parked conversation (ConversationCache::clear) and every idle slot, and
reads its prompt from token 0; a slot still running then finishes its
reply and is not kept as a cache after it. A request that ends on it does
not keep its conversation live (#1620's cancelled-read path is skipped
for it).
Covered: the serial decode loop (one GPU or a layer split), the
--pipeline-windows decode loop of a layer split, the prompt's short tail
(read_windows, and read_windows_pl with --pipeline-windows), and both
batch-slot loops (run_slot_rows, and the --batch-groups pipeline's
batch_poll). Not covered: the one-shot command line (no --serve) and the
SYCL port in sycl/, which has its own verify.cpp and generate.cpp.
- STRATA_NAN_INJECT=<n>[,<n>...] (debug): the n-th window the guard checks
reports row 0 as non-finite.
Server: EngineNonFinite on that finish. A request that has sent nothing yet
is sent again once (its usage is the retry's: the failed pass's DONE is
dropped); a reply that had started ends as "length"; a second fire on the
retry ends with an error (503 before an answer, an error event in a
stream). With replicas the router forgets what that replica held.
repeat_stop_tokens stays as the backstop.
Checks: serve tests (python -m unittest discover -s serve -t . -p
"test_*.py": 697 tests, OK, 11 skipped; NonFiniteLogits,
test_nonfinite_in_batch_mode and test_nonfinite_forgets_the_replica are
new). ctest with CUDA_VISIBLE_DEVICES=-1: 24 pass (conversation_cache_test
with clear() among them), 3 skip for want of a GPU, and the 43 GPU parity
tests were not run on a GPU. Built for CUDA 13.0 on Linux (GCC, sm_75 +
sm_86); generate.cpp, verify.cpp and verify_kernels.cu compile with
MSVC 2022 + CUDA 13.0, and conversation_cache_test and conv_cache_test
pass there; built for HIP (ROCm 7.0, gfx1100, compile only) and SYCL
(oneAPI 2026.1.1; it takes only the new declaration in verify_kernels.hpp
and the new inline ConversationCache::clear(), and calls neither).
On a GPU (RTX 3080, RVN IQ3_S, strata.service's one-GPU config):
- Answers, guard on vs STRATA_NAN_GUARD=0, at fixed expert sets
(--pcie-frac 0 --adapt-every 0 --adapt-swaps 0 --mtp-max-t 1
--suffix-draft 0, STRATA_IQ_MT_MIN=1 STRATA_PREFILL_CPU_SHARE=0), a chat
replay of 2 x 4 turns on a 16K-token system prompt: byte-identical in 3
pairs (one with STRATA_ROUTE_TAIL_SKIP=0, one with STRATA_SH_STREAM=0),
and two guard-on runs 8 of 8. One more guard-off run differed from every
other run from its first answer, and a second guard-off run matched the
guard-on runs: a difference between runs of this config, not the guard.
- Cost, Nsight Systems: logits_nonfinite_kernel 424 launches, 4.77 us on
average (7.78 us at most), 2.0 ms against 21.6 s of decode: 0.009%.
- STRATA_NAN_INJECT=1,60: the first window of a new chat flagged, the
server sent the request again from token 0 and the client saw one normal
answer; the 60th, mid-reply in the next turn, ended that reply as
"length" after 58 tokens; the turn after it reused 0 cached tokens and
answered normally. The engine logged exactly the two injected windows.
route_resident_k is one warp per token and the form it replaced walked all 512 experts on a single
thread. Nothing in the tree checked that the two land on the same expert and leave the same ids,
weights and counters, and the option's whole justification is that it moves misses across PCIe and not
the model.
src/kernels/route_resident_parity.cu: 240 random cases over 1 to 8 tokens, a 40-84% resident table,
random rank windows, margins from -0.5 to 1.5, and half the cases rounding the logits onto a coarse
grid so the tie rule (the smallest expert index wins) is exercised. Half the cases get the router's
own top-10 and half get ids with repeats, which is what the already-picked test is for. The reference
is the spec written out plainly - a serial walk with the same comparisons - and not a copy of the
kernel this branch replaced; it runs on the device so that expf, and so the softmax bits, are the same
function on both sides.
ids and the four counters have to match bit for bit. The weights are a softmax ratio and this
reference is a second compilation of it, so they are compared within 8 ulp and the worst seen is
reported; the kernel itself is bit-identical to the serial form it replaced, which was measured with
both kernels side by side in one translation unit. A case that swaps nothing still has to leave the
weights untouched, which the sentinel they start from checks, and the test fails if no case swapped at
all.
route_resident_parity: 240 cases, 205 with swaps (2004 swaps), 0 mismatches, weights within 6 ulp
Registered in ctest next to the other router parity tests, TIMEOUT 120.
The swap kernel is the only work STRATA_ROUTE_RESIDENT adds to a window, and it was written as one
thread per token: with a verify window of 1 to 8 rows that is up to 8 threads of a block, each doing,
per rank in [lo, hi], a 512-expert scan with a ten-way "already picked" test inside it. On this
machine that is about 4 x 512 x 12 scalar operations running on six threads, on the critical stream,
once per layer, and it shows up as its own cost in the window.
STRATA_VERIFY_PROFILE, three legs in one session, same prompts, same expert cache (8083 slots, one
learned profile), medians of the per-window stage table:
configuration hc-read1+router waitB ms/window pcie_e
option off 2.25 9.30 21.85 1.87
option on, one thread per token 5.00 5.01 21.97 1.00
option on, this commit 2.58 4.87 19.75 0.95
The option redirects about a miss per layer-window away from PCIe and saves 4.3 ms of waiting for it;
the serial swap spends 2.75 ms back, which is why the option measured as a wash. In the warp form the
swap costs 0.33 ms, so the waiting it saved stays saved.
End to end, twelve matched pairs of engine restarts, nine decode samples a leg, three greedy 150-token
prompts, the learned expert profile copied back before every leg so both arms start from the same
8083-slot cache. The first six pairs ran the unpatched build first, the last six the patched build
first, so a within-pair order effect would show up as the two halves disagreeing:
pair order decode unpatched patched delta ms/window unpatched patched delta
1 A->B 104.1 115.7 +11.1% 18.64 17.01 -8.7%
2 A->B 103.5 107.4 +3.8% 18.47 18.12 -1.9%
3 A->B 111.4 107.7 -3.3% 17.71 18.26 +3.1%
4 A->B 101.4 113.0 +11.4% 19.11 17.03 -10.9%
5 A->B 105.8 107.6 +1.7% 19.05 18.05 -5.2%
6 A->B 104.8 105.6 +0.8% 18.60 17.83 -4.1%
7 B->A 106.6 113.4 +6.4% 18.70 17.16 -8.2%
8 B->A 103.1 110.2 +6.9% 18.14 17.29 -4.7%
9 B->A 106.9 111.5 +4.3% 17.91 17.26 -3.6%
10 B->A 104.3 114.7 +10.0% 17.81 17.11 -3.9%
11 B->A 108.3 115.9 +7.0% 18.47 16.61 -10.1%
12 B->A 104.5 113.6 +8.7% 18.43 17.76 -3.6%
twelve legs each: unpatched decode median 104.7 (101.4-111.4) ms/window median 18.47 (17.71-19.11)
this commit decode median 112.2 (105.6-115.9) ms/window median 17.27 (16.61-18.26)
decode median +6.6% mean +5.7% faster in 11 of 12 pairs (sign test p = 0.006)
ms/window median -4.4% mean -5.2% shorter in 11 of 12 pairs (sign test p = 0.006)
first six pairs: +2.7% decode, -4.7% window last six: +7.0% decode, -4.3% window
pcie_e, the engine's own count of experts per layer-window that cross PCIe, is the same in both arms
(leg medians 0.63 against 0.64), so the same experts are routed and the difference is the kernel.
The swap's cost, and so its removal, scales with the tail experts the option actually redirects. Four
more pairs with a 4000-slot cache instead of the automatic 8083 (pcie_e 1.83 against 1.78, i.e. about
three times the traffic of the series above), two pairs each way:
pair order decode unpatched patched delta ms/window unpatched patched delta
1 X->Y 83.8 92.0 +9.8% 21.94 20.60 -6.1%
2 Y->X 84.6 92.5 +9.3% 23.97 20.53 -14.4%
3 X->Y 82.2 85.5 +4.0% 23.69 22.65 -4.4%
4 Y->X 81.0 87.7 +8.3% 24.54 22.21 -9.5%
decode median +8.8%, ms/window median -7.8%, 4 of 4 pairs, both orders.
What this changes is the shape, not the choice. One warp per token: lane j reads experts j, j+32, ...
(coalesced instead of a serial walk), the warp reduces to the best logit in a shuffle tree, and the
ranks are still visited in order, because a swap changes the set already picked. The choice is the
same bits: a strictly greater logit wins, the smallest expert index breaks ties, the already-picked
test runs against the ids as they are at that rank, a token with no swap keeps the router's weights
untouched, and the softmax is the same expression.
Parity. 400 random cases - half of them built with many equal logits, random lo/hi, margins from
-0.5 to 1.5, 1 to 8 tokens, about 60% of the experts resident, routed ids with repeats - compare the
two implementations bit for bit on the ids, on the weights (their raw bits) and on the four counters.
The harness carries both kernels side by side; 0 mismatches.
For context, the option's price is the margin's and not the kernel's, and this commit does not touch
it. Teacher-forced log-probabilities over 7677 positions of one 7.6k-token text (technical prose,
source code and Chinese), top-64, against the same text with the option off:
margin argmax same KL (top-64) teacher-forced loss perplexity
off - 0.000 3.2033 -
0.25 84.9% 0.235 3.2238 +0.6%
0.50 83.0% 0.293 3.4360 +7.3%
1.00 63.5% 1.001 4.7751 +49%
Two runs with the option off are bit-identical, so that spread is the option and not the machine.
0.25 keeps nearly all of the waitB saving for a fraction of the distortion; the default 1.0 is not
worth its price.
Tests: the parity harness is a standalone .cu (both kernels, the random cases, the bit comparison).
Happy to wire it into the test suite next to the other kernel tests if you want it in the tree.
The QSA step records of a chunk were uploaded with an unchecked cudaMemcpyAsync from a
std::vector. On gfx1100 the first text prompt after a picture one read garbage records
(5 of 5 faults on a two-request repro, 0 of 5 with a pinned, event-guarded source). The
token id upload has the same shape and now takes the same path; both check the result.
Reported and traced by mantovaniluca91, #1712.
IlQ3K runs q3_q8_dot_impl's arithmetic with the scales and signed values done once per chunk; bitwise native_mmvq's output
for T 1-4 and rows 1/2/4 (mmvq_il_parity). Q2_0 pack: the Q3_K multi-column kernels were 14% of GPU kernel time; the new kernel
takes 2560x10240 at 4 columns from 120 to 70 us on an RTX A4000. The shared table (sm_80+/sm_120) is unchanged.
Bitwise equal to the CUDA layout on all 48 layers at 1-4 tokens a group (native_expert_bench); a 28-group VRAM call 17.6 -> 15.5 ms over the layers at T=1, 24.1 -> 22.6 at T=2, even at T=3, +2.6% at T=4.
Bitwise equal to native_mmvq (mmvq_il_parity, T 2-4, rows 1/2/4); table read off mmvq_il_parity --bench --emit-table on a Tesla P100 (three identical runs).
The pipelined decode loop asks both stages' windows for service millions of times a second while they wait on their
doorbells, and every call switched the device (OnDevice: cudaGetDevice and two cudaSetDevice) before finding nothing
to do. On a 0.1.41 split (RTX 3060 + RTX 5070 Ti, Windows) the host stalled inside one of those cudaSetDevice calls
after 919 million loop passes, with both stages' windows waiting on it (stall dump: the serving thread in cudart from
Verifier::service, every other engine thread idle). Now a call with nothing rung, no WDDM flush due (2 ms) and no
timeout due returns 0 first - the same answer the loop below gives - so the driver is called only to serve a layer,
to flush, or to time out.
The pages had no <link rel="icon"> and serve/web had no icon file, so browsers asked for
/favicon.ico, got a 404 and showed the default blank icon in the tab.
serve/web/favicon.svg is the layer stack the app already uses (serve/web/sprite.svg, i-layers),
redrawn on a 32-unit viewBox so it stays legible at 16 px, in the existing accent token #10b981.
Checked: curl /web/favicon.svg -> 200 image/svg+xml, both pages come back with the icon link,
and the icon rendered at 16/32/48/64 px in Chrome.
_Nvml.read() reports pcie_gen, pcie_gen_max, pcie_width and the two
throughput counters; _Amd.read() (the amdgpu sysfs reader from #301)
reports none of them, so on Linux the Monitor's PCIe tile shows only
dashes while the NVIDIA tile fills in.
This adds the link the card actually gets: the narrowest and slowest hop
between the root port and the card, from each hop's max_link_speed and
max_link_width. A card that negotiates Gen4 on its own hop but sits
behind a Gen3 root port really runs at Gen3, and that is the number a
bandwidth budget needs. pcie_own_gen keeps the card's own hop aside so
the gap is visible, and pcie_path reports how many hops could be read --
the bottleneck is only claimed when all of them could, because a kernel
or a container that hides part of /sys/devices would otherwise report the
card's own hop as the whole path.
Measured on one MI50 32 GB in a Gen3 x16 slot (lspci: the root port at
00:02.0 runs at 8 GT/s x16, the bridge above the card is downgraded, and
the card's own hop negotiates 16 GT/s), /metrics reports pcie_gen 3,
pcie_gen_max 3, pcie_own_gen 4, pcie_width 16 and pcie_path 4/4.
Throughput is deliberately left out: amdgpu's pcie_bw is three counters
whose documented use is a total, not a direction, while _Nvml's
pcie_rx_mb is receive-only, so filling that name from pcie_bw would
label a combined figure as one direction. Happy to follow up separately
if you want the tile to show a rate.
Hardware the numbers come from: one Radeon Instinct MI50 32 GB (gfx906,
PCI 1002:66a1, VBIOS 113-D1631711-100) on a Xeon E5-2696 v4 (Broadwell,
no AVX-512), in a PCIe 3.0 x16 slot, 128 GB RAM, Ubuntu, kernel
7.0.0-34-generic, ROCm 7.14 in a container.
Prefill::arm_cpu_share() asks "is this AMD?" with
#if !defined(STRATA_USE_HIP), but the gfx906 compat build (PR #638)
compiles as HIP *without* STRATA_USE_HIP -- see src/core/expert_cache.cpp
:5-6, which guards the same question with
#if defined(STRATA_USE_HIP) || defined(STRATA_HIP_GFX906).
On gfx906 the CUDA branch is therefore live: the CPU share is armed by
default for chunks below 1,024 tokens and prints its banner, against this
file's own note at :161 ("the layer split, batch slots and HIP stay off").
Seven other sites already carry the || defined(STRATA_HIP_GFX906) form.
Measured on one MI50 32 GB (gfx906), CLI generate, a 4-token prompt, one
binary with one set of weights and one config, only the environment
differing, power saving off:
default experts streamed 309 (295 by DMA, host 174.0 ms), resident 607,
on the CPU 197 (share 0.90) prefill 928.3 ms
=0 experts streamed 506 (492 by DMA, host 177.3 ms), resident 606,
on the CPU 0 (share 0.00) prefill 929.5 ms
Output token ids are identical (271 248068 198 760) and the 0.13%
difference is inside this machine's run-to-run spread (1.2-1.7%), so on
this card the share engages and buys nothing. #1595 and #1729 report the
default costing on CUDA; this is the mechanism that turns it on for AMD.
CUDA and SYCL do not define STRATA_HIP_GFX906, so the added check is a
no-op there and nothing on those backends changes.
Hardware the numbers come from: Xeon E5-2696 v4 (Broadwell, 22C/44T, no
AVX-512), one Radeon Instinct MI50 32 GB (gfx906, PCI 1002:66a1, VBIOS
113-D1631711-100, 34,342,961,152 B of VRAM), PCIe 3.0 x16 effective (the
root port runs 8 GT/s x16 and the bridge above the card is downgraded;
the card's own hop negotiates 16 GT/s, about 15.75 GB/s end to end),
128 GB RAM, Ubuntu, kernel 7.0.0-34-generic, ROCm 7.14 in a container,
built with -DSTRATA_HIP_GFX906=ON and -DCMAKE_HIP_ARCHITECTURES=gfx906.
spark(id, values, max) sets the axis top to max(|max|, the series' own
maximum, 1e-9). gpu_util, temperature, cpu, vram and power pass an
explicit maximum, but the pcie, tok/s, prefill and disk cards do not, so
those axes zoom onto whatever the series happens to hold.
An idle amdgpu pcie counter reads 0.01-0.02 MB/s (50 packets x 256 B per
second), and that becomes a full-amplitude square wave in the Monitor
while the tile's own text says 0.0 MB/s. Measured in the live DOM: the
pcie line's y values are only {4.0, 30.0}, i.e. the baseline and the top.
A floor of 1 only ever lowers the axis - the series maximum still wins -
so a real burst keeps its scale, and the cards that pass a maximum are
untouched.
IQ4XS.test_amd_is_not_asked mocks get_prebuilt_hip, the engine setup.main takes on Windows. On Linux setup.main
compiles the HIP engine instead (build_engine_hip), which the test did not mock. Without a usable system ROCm 7, as
on the tested machine, rocm_root then fetched AMD's ROCm index
(https://rocm.nightlies.amd.com/v2/gfx120X-all/rocm/), wrote Path(sys.prefix)/.strata-rocm.json and tried to run
<python dir>/rocm-sdk. Where sys.prefix is not writable (the system Python) that is a PermissionError. Where it is
(a venv), the test passed while setup.main stopped at the ROCm step with exit code 1, and a stamp with the same
version and index can make a later setup in that venv skip the ROCm install.
The test now mocks build_engine_hip the same way, so on Linux setup.main runs to the end (exit code 0) with no
network request, no subprocess and no write outside the test's temp folder. setup.py is unchanged; the Windows path
never reaches build_engine_hip, so it runs as before.
Tests: python3 -m unittest tools.test_setup_unsloth (WSL2): with the system Python 3.12.3 FAILED (errors=1) before,
38 tests OK after; in venvs on 3.10.22, 3.12.3, 3.13.7 and 3.14.8 OK before and after, and no stamp after. With the
system Python the other 27 tools/test_setup_*.py files give the same results as before.
The `[16.0,18.0)` range is CUDA 13's requirement, but it was applied to every caller: with the
only Visual Studio installed being 2026 (version 18), vswhere finds nothing and find_vcvars()
returns None. That makes the CPU image encoder unbuildable on such a machine - build_vision_cpu()
cannot run - so --vision cpu stays unreachable there even with the gate in hip_vision() open.
#985 relaxed the range for CUDA 13.3+ by adding `cuda_v`, which is right for CUDA callers. A
caller with no CUDA now passes nothing and gets no range, which is what
tools/hip/build_windows.bat already does when it calls vswhere directly.
- `cuda_v is None` -> no -version argument at all. CUDA callers are unchanged: cuda_v=(13, 0)
still refuses VS 18, cuda_v=(13, 3) still takes it.
- STRATA_VCVARS=<vcvars64.bat> picks an install by hand, for a vswhere that cannot be used.
Verified on the RX 6800M / gfx1031 machine of #881 (Windows 11, Chinese locale, the only C++ tools
are Visual Studio 2026 / 18.4.11605):
find_vcvars() -> <VS 18>\VC\Auxiliary\Build\vcvars64.bat
find_vcvars(cuda_v=(13, 0)) -> None
find_vcvars(cuda_v=(13, 3)) -> <VS 18>\VC\Auxiliary\Build\vcvars64.bat
With a draft-vocab head (rt/draft_vocab.bin) the slot drafters of --batch-mtp share the main drafter's head but kept
dhead_type_ at -1, so the first draft_first() of a batch admission threw 'mtp: unsupported native MMVQ GGML type'
and the engine exited (#1121). The CUDA source got this line in #1110; the SYCL port did not.
A periodic loop emitted as digits (a 24k-digit string cycling an 11-digit
pattern) is a single word to the word count and never one token twice
(the tokenizer has no multi-digit tokens), so both #606 and #728 passed
it and the run burned the whole reasoning budget. The reasoning's last
2,048 chars are now checked for a period of at most 64 chars, and a
periodic tail reads as full coverage. The opt-in path only: the default
path and the word count are unchanged.
With the Intel port the configured exe is sycl/serve/strata-sycl.sh, a shell script that runs the repo's
STRATA_SYCL_BIN (default build-sycl-aot/strata) in a container. effort_end_args looked for --tail-role-token in the
script, never found it, and fell back to the top-of-prompt effort with "needs engine 0.1.39 or newer", even on a
0.1.41 engine that has the option. engine_binary() resolves the launcher to the binary it runs; any other exe is read
as before. tools/test_effort_end_args.py covers both paths.
Follow query-swizzle #1661 with the existing reduce-scatter pairing for
12 heads. Keep default paths and softmax/value/merge unchanged. Public
component parity and timing passed; model evidence uses frozen stack.
Record archived-control output parity, verified candidate process, tools, CPU vision and cache restoration. No fresh 200K speed comparison or production promotion is claimed.
Preserve FP operation order and invert a bijective shared-query layout at its vector loads. Limit the path to the current gfx906 wave64 device, INT8 pools and warp scoring. Default remains off.
Add deterministic GPU parity/timing and host-guard probes, plus scoped component and model receipts. The clean upstream build passed both-card byte checks; model gains are recorded separately on the frozen experimental stack.
Pinned memory reads the link 1.7x faster (A750 10.3 -> 17.2 GB/s, B70 13.6 -> 25.6), which moves the derived default pcie_frac
(A750 0.28 -> 0.47) and cost 10% of decode there (median 12.07 -> 10.67 tok/s, 4 of 5 interleaved pairs slower). The B70 is unchanged
(0.37 -> 0.55, 31.4 vs 31.4). The fault the PR fixes is a layer split's.
Co-authored-by: maxious <[email protected]>
server_intel.py replaces StrataEngine._pump to keep LP lines beside their token, but its copy predates --batch: the
slots' BT / BDONE lines went to the main queue, so with "parallel" every request stopped after one token. It also
missed the fatal-line handling and last_err. Both are now as in serve/server.py's _pump.
The host mirror holds the experts missing from VRAM at the start. An adaptive swap moves an expert that was resident
then out of VRAM, so it is in neither; the device plan then waits for the host to serve it, and with
STRATA_VERIFY_NO_HOST (the xe default in strata-sycl.sh) nobody does. Seen after a --batch window, whose host path
counts the misses the tier swaps on: the next single request on one card gave token 0 ("!!!!") at 0.8-1.3 tok/s.
The start already refuses experts in neither under NO_HOST; this keeps that true after the start.
The prompt path dequantizes each dense Q8_0 projection (GDN qkv/gate/out,
QSA q/k/v/out, shared expert, hc) into the GEMM scratch before cuBLAS.
dequant_kernel runs one thread per 32-value block, so a warp's 2-byte
stores land 64 bytes apart: about 70 GB/s of writes on a V100.
dequant_q8_0_kernel gives each block to four threads, 8 values each,
written with 16-byte stores: a warp reads 8 consecutive blocks and writes
512 consecutive bytes. The arithmetic per value is unchanged (d * q), so
the output is the same bits. It is used for Q8_0 when the output rows are
contiguous (ld == cols) and the pointers are aligned (blocks to 2 bytes,
output to 16); everything else takes dequant_kernel as before.
V100-SXM2-16GB, every 2-D Q8_0 tensor of Qwen3.8-Flash-Next UD-Q4_K_XL
(498 tensors), each call timed on its own:
attn_qkv 10240 x 2560, FP16 0.740 -> 0.106 ms (70 -> 495 GB/s written)
attn_q 12288 x 2560, FP16 0.887 -> 0.126 ms
ssm_out 2560 x 6144, FP16 0.451 -> 0.065 ms
all 498 tensors, FP16 138.5 -> 20.4 ms
all 498 tensors, BF16 140.8 -> 20.5 ms
all 498 tensors, FP32 190.9 -> 35.4 ms
An FNV-1a hash over all outputs is equal before and after for FP32, BF16
and FP16. Whole prompts on 4x V100 (where this was first used), 4,096-token
chunks: 18K 4,537 -> 4,708, 65K 6,540 -> 6,766, 123K 6,783 -> 6,985 tok/s.
The whole i-quant family (IQ3_XXS, IQ3_S, IQ2_S, IQ4_XS gate/up, IQ4_NL down) and the
fork's Q2_0 down have x86 dots only under __AVX2__ in ggml-cpu; on an AVX1 floor
(STRATA_ISA_FLOOR=avx, issues #595/#623/#1375: Xeon E5 v1-v2, and AVX-512-only packs)
the CPU-computed experts fall to the scalar _generic dots. iq_avx1.cpp is the older
CPUs' copy of iq_avx2.cpp in the kq_avx1.cpp pattern: a separate TU compiled
-msse4.2 -mavx -mno-avx2 -mno-fma -mno-f16c, gated at runtime behind cpu_avx1_ok() and
only when AVX2 is absent, so no mainstream path changes. 128-bit integer lanes
(pshufb/sign/maddubs are SSSE3), 256-bit float accumulation, software half-to-float,
no gathers (the Fmt32 non-gather path), one IEEE mul per value like the AVX-2 copy.
Plus q8k_quant_avx1, the byte-identical Q8_K activation quantizer in SSE4.2/AVX1.
iq_avx1_parity: all six formats, nt=1..8, bit-exact across widths (#152) and within
1e-5 aggregate of ggml's generic dots; q8k_quant_parity now checks the AVX1 copy too.
gu rows take the AVX-2 mt_min policy (STRATA_IQ_MT_MIN), down rows always; kill switch
STRATA_NO_IQ128=1. E5-2470 v2 (Ivy Bridge): 1.2-3.1x per expert at nt>=2, 2.0x on the
Q8_K quantizer; serve prefill +36-41%, warm decode 22-25 -> 46 tok/s.
Known pre-existing gap found while testing: pool_tasks_test SIGILLs on non-AVX-512
machines even on a clean tree (it predates the ISA floor and is not gated).
q2_rows_any and act_quant_any had two rungs, AVX-512 and AVX2, and the else branch
checked nothing: an AVX-only CPU was handed AVX2 code and died with SIGILL (found on
an E5-2470 v2; pool_tasks_test and q2_bitplane_parity crash on a clean tree, and any
future caller inherits the trap). The third rung: q2_avx1.cpp in the iq_avx1.cpp
containment pattern (-msse4.2 -mavx -mno-avx2 -mno-fma -mno-f16c, a #error guard,
called only behind cpu_avx1_ok() && !cpu_avx2_ok()). 128-bit integer lanes (maddubs,
madd), 256-bit float accumulation, software fp16 scales; the legacy kernel keeps the
AVX-2 kernel's lane pairing and reduction tree (its FMA becomes a mul+add, last bits),
and the bit-plane kernel and its image builder port as-is (pure SSE). The AVX-2 and
AVX-512 paths are untouched. q2_bitplane_parity now runs on the AVX-2 pair where the
CPU has AVX2 and on the AVX1 pair below it, and skips cleanly without AVX.
E5-2470 v2: pool_tasks_test 168 bitwise comparisons pass, q2_bitplane_parity PASS -
both SIGILL'd before.
A helper GPU's expert cache already holds a copy of the blobs the prompt path
streams, and the helpers compute nothing while a prompt is read on a serial
request path. Add RemotePrefillSource, an ExpertSource that wraps the file
source and asks the helpers first: a blob a helper holds is copied
cache-to-host on that helper's own stream, misses are merged into one call to
the fallback's existing batched reads, and every other decision is forwarded
unchanged.
Opt-in through STRATA_PREFILL_HELPER_SOURCE; it applies only where the serve
path is serial (one GPU, no layer split, no batch slots) and the prompt path
streams every expert the main card does not hold. Same-day A/B on an
11-request fixture: 91.5 s -> 65.7 s of prefill (-28.3%) and 92.1 GB (64.8%)
fewer bytes read from the files.
The share is picked from wall-clock timings, so the same short-prompt
request can split the experts differently each run and the answer moves
in its last bits - a resumed conversation's re-read need not match its
first (31 of 1,536 teacher-forced argmax tokens differed in the
reporter's test). A fixed share repeats; 0 gives a run without it
byte-for-byte.
Five simulated agent sessions over the repository's own files (system prompt, four tools, real files as tool
results, at most 2000 lines per read): 23 turns from ~2.5K to ~100K tokens, consecutive turns sharing their prefix.
Arms as ab_engine.py takes them; rounds mirror the arm order on fresh servers; prints the median of the per-turn
decode ratios to the first arm and the turns it won. Nothing personal and nothing downloaded.
A missed expert that every token of the verify window routes at rank R or lower in weight contributes zero
(no renormalisation): less PCIe traffic and CPU expert work. Counted in the request log. Opt-in, changes
the output.
The top-10 is picked again with a logit bonus lambda for the experts resident in VRAM; the weights stay the
router's probabilities renormalised over the 10 picked. After 'Mixture of Cache-Conditional Experts'
(arXiv 2412.00099), without training. Complements STRATA_ROUTE_RESIDENT (a tail swap within a margin):
every rank can move, by an amount the logits decide.
With --resident-experts (setup's low-RAM mode when the experts the GPU does not hold fit the RAM) the start copies
the profile fill of the GPU cache and the RAM copy (pin_cache_complement) through the mapped view of experts.bin.
Every page it touches sits in the process working set beside the page-locked copy until #467's trim after the fill
and the VirtualUnlock after the copy: on a PC short of RAM that pages other memory out, and the copies run at the
page-fault rate (~0.85 GB/s here).
FileExpertSource::begin_startup_unbuffered closes the view, the section and the buffered handle, opens the
unbuffered handles (open_direct) and reads the first blob both ways; if the open fails or the two differ, the view is
mapped again and the start copies through it as before. The fill (prefetch_pairs) and the copy (pin_cache_complement)
then take their existing unbuffered paths (read_direct: requests merged up to 32 MiB). end_startup_unbuffered maps
the view again (map_view, the size checked) and closes the unbuffered handles and the stage buffers, so the file tier
is chosen as before. Same bytes in the same places.
Windows and experts.bin only: the call is under _WIN32, and the GGUF in place, an unbuffered file tier and runs
without a RAM copy are unchanged. STRATA_STARTUP_UNBUFFERED=0 gives the mapped copy back; STRATA_VERIFY_COMPLEMENT=1
prints checksums of the filled cache slots and of the RAM copy. pin_cache_complement no longer VirtualUnlocks the
range of a closed view.
IQ2_XS (33 GiB experts.bin), RTX 5090, Samsung 9100 PRO, --resident-experts --expert-cache auto (~15,200 slots, a
16.4 GiB RAM copy), interleaved against fb58e0db:
- 21 GiB available (a RAM ballast; #467's 32 GB PC had 20.7 GiB), 6 pairs: ready 42.5 -> 9.6 s (chat), 40.7 -> 10.2 s
(32K); the fill 842 -> 6,914 MB/s, the copy 13.7 -> 3.3 s; lowest available 0.25 -> 4.5 GiB; pages written out
~60,000 -> 579-3,340; 3 of 12 base starts held available RAM under 1 GiB for 2 s, none of 12 here.
- 128 GB: peak working set 32.6 -> 18.3 GiB; a restart with the file cached 13.6 -> 8.3 s; the 32K prompt and decode
unchanged.
- Same bits: p2k 200 tokens and first-token logits, 32K 64 / 256 tokens, as fb58e0db; STRATA_VERIFY_COMPLEMENT=1
checksums identical to STRATA_STARTUP_UNBUFFERED=0 (12,595 and 15,438 slots).
Based on RealScaniX's sorted-by-name change; the first request of a set stays byte for byte as the client sent it.
Co-authored-by: RealScaniX <[email protected]>
RulesAgreeWithTheWrittenConfig.test_the_streaming_rule_is_the_flag_setup_writes runs setup.main() through
test_setup_golden's install(), which mocks is_wsl() to False, and then asked setup.kv_streaming_wanted() outside
that mock. Inside WSL the real is_wsl() is True, so the rule answered "no streaming" while the config setup wrote
for the mocked PC has --kv-resident, and the test failed at its first 131072 case (Q2_0, int8, auto). With is_wsl()
the same on both sides, setup and the rule agree in all 54 cases, inside WSL and out.
The test method now runs under the same is_wsl mock as install(), the test-only fix the unsloth tests got for #974
(d1ec32b). setup.py is unchanged; where is_wsl() is already False (Windows, native Linux) the test runs as before.
Tests: python3 tools/test_setup_risk.py (WSL2, Python 3.12.3: was FAILED (failures=1), now 40 tests OK); the other
27 tools/test_setup_*.py files give the same results as before.
A request with "temperature": "0.7", "top_k": 40.0 or "seed": true was
answered 200 and sampled greedy: the value replaced the configured
default in run()'s merge, then sampling_keys dropped it as not a number.
check_request_sampling refuses it by name before anything is sent, as
POST /settings and the run config already do. Only types are checked;
ranges, null fields and in-type values behave as before.
A chat "file" or "input_audio" part, and a messages "document" block
(also inside a tool_result), were dropped without a word: the request got
a 200 and the model answered about a file it never saw. _check_parts
refuses them while the request is converted, naming the type and where
it sits, with /v1/responses' wording. Text, images, thinking, tool_use
and tool_result read as before; redacted_thinking is still dropped; a
chat "refusal" part reads as its text, as on /v1/responses.
The row-count sort of a fully resident layer's experts skipped native-layout packs, though slot/src/off are
built the same way for both layouts. 4x GP100 (sm_60): 1.2K prompt 7.27 -> 5.32 s, 35K 63.3 -> 54.1 s, the
same bits. The switch stays opt-in.
kvg_trim refilled the slots the K/V gave back and then uploaded host_res with a plain cudaMemcpy from pageable memory.
That returns once the table is staged, before its DMA lands, and nothing after it waited for stream 0: the next
request's session_zero syncs main_stream only, and a prompt part of --short-read tokens or fewer goes straight to
ver.run, whose graph reads hits_.d_res on the verifier's cudaStreamNonBlocking stream. res_put (#550) is the upload
every other residency change uses: the same copy, then cudaStreamSynchronize(nullptr).
kvg_ensure's upload needs nothing: qsa_kv_elastic_grow ends in cudaDeviceSynchronize before anything reads the table.
Only --kv-grow (opt-in) runs kvg_trim, so the default path is unchanged.
With stream_tools an array or object argument is held until it is whole,
so a large one (a measured 66 KB edit_file call) left the stream silent
for minutes while tokens were generated - a client's idle timeout then
ended a healthy request (Forge's is 120 s).
The token loop now tracks when anything last went out and sends the
engine's own keep-alive ping after KEEPALIVE_S (10 s) of silence - the
same ping the quiet-engine heartbeat sends, so clients already ignore it.
The reporter's reproduction measured a 27.8 s longest gap on a 7 KB
array (minutes on the 66 KB one); with this change it is 10.0 s and the
call arrives intact.
Change and live measurements from the issue; adds the ping-cadence unit
test (a patched clock over a held-back <tool_call>).
The default router, publish and copy_or_zero kernels are the f-142 ones. The probes set the variable (and
STRATA_DOORBELL_CHECK=1 for publish_res, which copies x only on a miss otherwise).
Port the fork's fast router with a bracketed reciprocal and serial fallback. Add bitwise comparisons against the existing router, including ties, NaN tails and the forced serial path. On the A770's 256-expert/top-10 shape, measured host submission plus execution fell from 395 to 119 us per call.
Pair activation payload stores and mapped result reads into 64-bit uncached transfers, preserving the checksum and unaligned/odd-tail fallback. Register router and payload probes with CTest. The two probes, elementwise_parity and router_top10_parity passed on the A770.
Add the fork's opt-in host-clock stage profiler for further investigation. Profiling remains experimental: enabled runs encountered a payload-arrival timeout, and review flagged concurrent atomic host writes versus non-atomic device clock reads as a synchronization concern. The profiler is not needed for normal inference.
With CPU-computed misses, cache borrowing and an 8.98 GiB VRAM cache, unprofiled Coder IQ1_M runs generated 128 tokens at 11.9-13.5 tok/s and returned final arithmetic answer 41. These runs retained 128K context capacity but used short prompts; the 20 tok/s goal has not been achieved.
The admission checks prompt + max_new <= max_context, but the spec loop
opened every window at S_mtp rows, so a request that fits ended with
"ran out of context" (rc 2) once p + S passed the end, and printed
nothing. The window now takes only the rows the context still holds.
MtpDrafter::draft wrote draft j at cell p + a + j for every step, past
max_cells near the end, where the page table has no entry; it now drafts
only the cells that exist. Far from the end nothing changes.
CUDA 12 refuses a GCC newer than 14, and the docs already tell the user
to run CXX=g++-14 ./setup.sh - but the environment only reaches CMake on
a build folder's FIRST configure, so on a retry (or the encoder's own
folder) the flag never landed: the reporter's run died at the last step.
host_compiler_defs() maps them onto -DCMAKE_C_COMPILER /
-DCMAKE_CXX_COMPILER / -DCMAKE_CUDA_HOST_COMPILER for the engine and the
CUDA image encoder; CUDAHOSTCXX defaults to CXX, as CMake's does. A
C++-only build (the CPU encoder) is not given the CUDA one.
STRATA_VERIFY_STEPPED=1 (Arc A-series): the doorbell window is captured as graph segments ending at each layer's
publish; the host serves the CPU experts between segments, pipelined with the plan copy + VRAM groups on a helper
thread pinned to the host thread's SMT sibling. No kernel then reads a host store while it runs and no host reads a
kernel store while it runs, so the window's mapped buffers are ordinary host USM and the publishes use plain stores
(STRATA_VERIFY_HOST_UNCACHED=1 keeps both uncached). A770, Coder IQ1_M, 8K prompt: GPU wait per window ~70 -> ~43 ms,
decode 22.8-25.0 -> 31.8-32.5 tok/s; 32K 27.2 tok/s.
The pipelined slot groups never run the adaptive tier, so --adapt-async 1 with --batch on a layer split and no
--batch-groups resolves auto to one group, like --batch-mtp (batch_groups.hpp, batch_groups_test); --batch-groups G
or auto given on the command line is honoured and turns --adapt-async off through the existing gate.
batch_step advances the async round one step per window (never between a prompt's chunks), commands drain the round in
flight before they run, a round with nothing to swap ends at its first step, and --adapt-min-gain sets the bar a swap
must clear (default 1.5, as before) for both tiers. --adapt-async is no longer refused with --batch slots (still with
--batch-groups).
batch_step now runs adapt() every --adapt-every windows on a thread beside the commit and the drafts, lands the swaps
(apply_pending) before the window ends, and pauses while a prompt holds a loan of cache slots. The timing line adds the
adaptive tier's rounds/swaps and wait, each stage's GPU-reach wait and pool time, and the VRAM/PCIe/CPU entry split.
The 0.1.41 default --batch-groups auto, the pipelined path's lack of slot drafts and the helper-cache refusal are
written down in docs/BATCHING.md (the limits list and the --batch-mtp paragraph).
--batch-groups auto became the default for a layer split with --batch >= 2, but the --batch-mtp gate ran before
the groups were resolved: it saw 1 group, kept batch-MTP on, and the groups were then raised to 2 - pipelined groups
do not run the slots' MTP drafts (#1413). The resolution is now a pure helper (include/strata/program/batch_groups.hpp,
CPU test src/program/batch_groups_test.cpp): with --batch-mtp and no --batch-groups the default is one group and
the log says why; --batch-groups N (N > 1) or --batch-groups auto given on the command line pipelines the groups and
turns --batch-mtp off, as before.
- generate.cpp: --batch-mtp accepted with a layer split on separate GPUs (refused with --batch-groups); slot drafters
load from the last stage's slot sessions under the last stage's device guard and bind to its weights/head; their
residual buffers, row copy (async on the drafter's stream) and draft-KV copies run on the drafter's device; every
stage verifier gets the 8-row window size and the batch graph limit; slot-drafter logits are not charged to CUDA0.
- verify.cpp: stage_batch's layout rules move to include/strata/core/batch_rows.hpp (CPU test batch_rows_test); grouped
rows are allowed on a stage that hands rows on, the hand-off bound is kVerifyMaxT rows.
- docs/BATCHING.md, --batch-mtp help text.
Step 1 warns below 4 GB of page file (issue #60) and advised "Set it to System managed". In #60 that did not help:
"System managed" and 4096-32768 MB still ran out of commit, and a fixed 64 GB worked - a page file Windows grows on
demand may not grow in time while WDDM charges the card's memory to the commit.
- The advice is a fixed size: Custom size, the same initial and maximum size, 65536 MB (PAGE_FILE_FIXED_MB).
- page_file_setting() reads the Virtual memory setting (PagingFiles; "?:" is the drive in ExistingPageFiles). When a
file is system-managed or its initial size is below its maximum, the warning says that it grows on demand and that
a growing page file may not keep up (#60).
- page_file_gb() still counts only what is there for sure: the commit limit minus RAM, as before, or a file's
configured initial size when that is larger (within its drive's free space). Never the maximum or what a
system-managed file may grow to. The 4 GB threshold is unchanged.
- docs/AI_SETUP.md and docs/DETAILS.md give the same advice.
- tools/test_setup_page_file.py: the parsing, the reading, the advice, and step 1 under --check on the golden
harness's 32 GB PC (registry, drives and GlobalMemoryStatusEx mocked; runs on Linux too).
The MCP server's FALLBACK_MODELS carries its own copy of the model
table, so #1688's setup.py fix leaves the MCP install path rejecting
--family swift --model IQ3_S. Drop the families gate there as well,
flip the validation test, and update the three doc lines that say
IQ3_S is original-model-only.
Measured on an RTX 3090 pair (engine 0.1.40.3, Swift IQ3_S, 200K
context, MTP spec on): 91-137 tok/s decode, expert cache hit rate
96-99%, KV streaming 98.6-99.6% VRAM hits.
UkisAI published a Swift 1.5 IQ3_S tier on 2026-10-08; the
"Swift 1.5 has no IQ3_S" restriction is stale. Both shards match the
family's file pattern exactly (names/sizes/SHA-256 verified against the
Hub listing), the shard count and the shared BF16 mmproj are unchanged,
so it takes the same path the other Swift tiers already use.
The IQ2/IQ3/IQ4_NL/IQ1_M decoders now write values [8 tid, 8 tid + 8) of a superblock (coalesced runs, as Q2_0
already did); iq_xmx_grouped still passed the IQ4_XS-style (sub-run, 32-block) index for them, so the B tile held
the wrong values (xmx_group_parity relerr 7-17 on types 18/20/21/22). All six types back to ~1e-6.
- xmx_moe: joint_matrix FP16 GEMM over a group of up to 16 experts per launch (grf 256).
Standalone vs oneMKL on uneven routing: down 1.75-6x, gate/up 0.8-3x (oneMKL kept above
STRATA_PF_XMX_GU_MAX=250 tokens/expert).
- STRATA_PF_XMX=2: iq_xmx_grouped decodes IQ3_XXS/IQ3_S/IQ2_S/IQ4_XS gate/up and IQ4_NL/Q2_0
down rows in-kernel (no FP16 staging); xmx_group_parity: rel err < 2e-6 vs dequant+oneMKL.
- End to end (8K/32K prompt, A770): +2-5% only; default off.
--pipeline-windows 2 now runs on any layer split with two or more stages. Every
stage but the last is a front stage: the guessed window follows the verified
one through them one card behind (an odd verifier and an odd hand-off on every
boundary, a pair of GDN snapshots on every front stage, commits and the undo
per stage), and only the last stage waits for the verdict. The pipelined prompt
read (--pipeline-windows 1) and --adapt-async stay two-stage.
HIP: the draft layer's per-row gr_read used the last stage's session scratch
(ss.block.gr), which that stage's verifier also uses in lm_head_mix. A chain
launched at a verdict and the next window's last stage can run on the card at
once after a guess that held, and the window's head then read a clobbered mix.
The draft layer now carves its own GrWorkspace.
Measured on 4x RX 7900 XT (Flash-Next GSQ-RCO IQ3_S, split 12,24,36, every
expert in VRAM, greedy, reasoning off): decode 59.4 -> 67.6 tok/s at 4K and
51.8 -> 65.6 at 32K. Greedy text identical to the serial loop on four prompts
with STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0, also with every guess
forced wrong. Debug knob STRATA_PIPELINE_SPEC_DEPTH. Two-stage path not re-run
on two cards.
probe_pcie_h2d_gbps: a pageable 256 MiB source made the copy engine fault
inside the driver's bounce path on a layer split (755x bcs VM_NOT_FOUND at
the fixed VA 0xffffc001ffa10000, L4 = the region never mapped for that VM,
intermittent on fresh boots, wedging the whole load in both the 0.1.40
port and 0.1.41). Pin the source (sycl::malloc_host) and free it with the
matching sycl::free. Validated: two consecutive loads reached the window
without the wedge.
All the split-fault instrumentation (guard watch, stage stamps and the
STRATA_STAGE_WAIT serial gate, terminate dumps, coredump grabbers) lives
on scratch/split-fault-instruments.
dpct turned `cudaStream_t cs_ = nullptr` into `&dpct::get_in_order_queue()`: a drafter that never loads keeps it, and
its destructor destroys the device's default queue (exit 139, sometimes UR_RESULT_ERROR_UNINITIALIZED, after the output).
Arc Pro B70 at fb58e0d without --mtp: 3 of 3 runs crashed before, 0 of 3 after; output unchanged with and without --mtp.
0.1.41 report (RTX 5090, tight RAM): chats failed and the engine restarted until STRATA_PREFILL_CPU_SHARE=0.
- share_rules: default_share_memory_verdict. The default is not armed when the RAM this process can get
(cgroup-aware) is below the share's buffers + 3 GiB, when the Windows commit left is below buffers + 4 GiB,
or when the experts it would read are file pages the free RAM cannot keep. One log line says why and that
STRATA_PREFILL_CPU_SHARE=auto turns it on anyway. An explicit setting is kept and only warns.
- prefill: the share's two pinned buffers are taken once at their largest (an expert takes at most MAXT
tokens' rows); a failed allocation ends the share for the run instead of failing the request.
- unit test, docs/DETAILS.md.
The counter block was one process-wide cudaMalloc on whichever device was current at the first Verifier::init, unreachable from
the other stage's kernels on a layer split without P2P. Each verifier now takes its own device's block in init (outside graph
capture), keeps the pointer, and a launch never picks by the host thread's device; a failed allocation runs without counters.
The exit report prints one block per GPU when there are several (single GPU: the same lines as before). The reporter's rare
fault with --pipeline-windows 2 after this fix is not reproduced here (needs two GPUs).
STRATA_ROUTE_RESIDENT_MTP=1 (experimental, opt-in, changes the output): the MTP draft's router gets the same residency bias,
with the last layer's residency row as its table (single GPU).
The refills of the slots a prompt borrowed (and the VRAM-resize refill) queue hundreds of cudaMemcpyAsync out of blob_stable()
pointers, which for a GGUF source are the recycled stage buffers. With STRATA_STAGE_PIN=1 the 257th assembly overwrote a
buffer whose copy had not run: IQ3_XXS lowmem (budget 12) decoded token id 0 after a 4096-token prompt. Each queued copy now
records the fence of its buffer. exact_pp 4096 and 20000 tokens with STRATA_STAGE_PIN=1: EXACT.
A busy slot flag with no owner kept every later request waiting for a slot nothing
would free, and a request waiting for the fifo (a load, an image encode, /v1/vram)
was counted nowhere. Now: slots have owners (claims / BDONE waits) and an ownerless
busy flag is freed after 3 s; the request waiting for a slot is counted
(admitting) and, past the stall time, named in the log; a parallel request on a
loaded engine no longer takes the fifo, and one that waits for it stays queued.
Tests with the fake engine.
On an RX 7900 XTX the first long prompt at 8192-token chunks charged ~70 GB of
commit and the engine died (0xC0000409); 7500 and below were flat. A bare auto
now stops at 6144 on Windows HIP builds. A chunk the user names stays, with a
warning from 8192 up.
Windows: the serve start prints the commit left and warns when the planned host
allocations plus a 4 GiB floor do not fit; the conversation cache's admission
reads min(available RAM, available commit).
With every expert resident nothing streams, so the share has nothing to take and
only moves the staged-chunk limit (+4.8% at 1,000 tokens on a V100). An explicit
STRATA_PREFILL_CPU_SHARE is honoured as before; the default is unchanged where
experts are not all resident.
A stage with one layer holds a handful of expert slots (a 12 GB 3060: 512), and the
prompt chunk of every card in the pipeline is clamped to what that cache can lend.
When the pick has such a stage, search again among placements that give every
stage at least three layers; keep the new pick and print both and why. With none
startable the first pick stays. Identical cards pick balanced stages and never
reach the second search.
available_host_bytes() = min(MemAvailable, the room under every cgroup limit above the engine, the operator's cap).
Parking and session save/restore read MemAvailable alone: in a container whose real limit sits on a parent cgroup it
cannot see, MemAvailable is stale (about 25 GiB shown when about 1 GiB was left) and the engine was OOM-killed.
conversation_available_memory() is replaced by available_host_bytes() / sample_host_memory() (the SYCL port's
generate.cpp too). The cgroup term covers memory.max and memory.high (v2) and memory.limit_in_bytes (v1) of the
process's group and every visible ancestor, with clean inactive file cache credited; an unparsable cgroup file makes the
sample unknown (fail closed). --memory-limit-mib / STRATA_MEMORY_LIMIT_MIB names the total RAM the container or cgroup
may use. Refusals say how much was available and from which source; one start-up line reports the limit.
tests/core/memory_guard_test.cpp runs on fixture directories (CPU only). Docs: DETAILS.md.
A profitable shorter chained lookup returned before the stale full-size check, so the full chain could stay excluded for good
although the policy re-probes stale sizes in the other selection paths (#1252). Check a confident stale full size before
returning the profitable shorter chain. The 300-round policy, the EMA, the header and plain lookup are unchanged; a regression
test covers the bounded re-probe spacing and the MTP confidence gate.
0.1.39 took this PR's first part with #583 (the xn / grs count, STRATA_RING_BYTES). What is left: bytes_needed
counted the fused layout's smaller MoE buffers for every caller (fused_layout(T, true)), while init takes them only
when it gets an ExpertSource (fused_layout(T, m.src != nullptr)). A prompt path without one was lent fewer bytes than
its unfused buffers take. bytes_needed, bytes_needed_no_ring and (0.1.40) bytes_needed_owned - STRATA_OWNED_PRICE=exact's
sizing - now take the same src the callers pass to init.
When the best split and a different one (boundaries at least 3 layers away, startable) are within 4% predicted decode window
(STRATA_SPLIT_NEAR_TIE_PCT, 0 = off) and the cards differ in speed or link, print the other K and --layer-split K. The model
prices decode only, not the prompt. Print only: the choice is unchanged.
A prompt read on a layer split with no slot decoding was one run, so the BYIELD of the batch_interleave_test step 4 was
dropped at the end of the read; the long prompt kept its slot and the next BGEN failed with 'ERR BGEN: no such free slot'
(pre-existing since 0.1.40.4). That read now goes in pieces of STRATA_BATCH_YIELD_CHUNKS chunks (default 8, 0 = one run), the
same chunks a single run reads, with the yield check between pieces. batch_read_piece holds the arithmetic; batch_read_test covers it.
A page-locked stage buffer is the source of the cache fill's cudaMemcpyAsync, which the DMA reads after the call has
returned; a pageable one is staged by the driver before it returns. The pool recycles a buffer after kStageAge layers or 256
assemblies, so a 4096-token prompt read (up to ~500 distinct experts per layer) could overwrite a buffer whose copy had not run
yet: the corrupted IQ3_S decode of #1237. note_async_read(blob, stream) records an event per pinned buffer after each queued
copy (hit-path fill, peer fill, remote-expert swap); claim_stage waits for it before it hands the buffer out again.
Pageable buffers and every other source are unchanged. STRATA_STAGE_PIN stays opt-in.
The reasoning ended at the first </think> in the output, also when the model only
quotes the tag while it thinks (`</think>`, "</think>\n\n2" from a test). The rest
of the thinking then came back as the answer, and a tool call quoted in it could
be run.
A </think> right after a quote or backtick and followed by anything but a line
break is now reasoning text; the end of the thinking is followed by a line break
and ends it as before. Until the character after such a tag arrives, the tag is
held like a partial one, so streamed and whole outputs agree. If the turn ends
right after it, it was the end of the thinking: a forced tool call still opens
there, and the #1053 retry is not triggered, as before.
Parking happens on a switch and every parked conversation costs one of
--conversation-cache-slots whatever its length. A client that interleaves
short side requests (an agent's tool-result follow-ups, often under 1k tokens)
with one long conversation therefore spends the cache on the side requests and,
once it is full, evicts the long one oldest-first - the reuse it wanted.
Measured on an RTX 4070 SUPER / 32 GB / Q2_0, --conversation-cache-slots 2,
one 19,198-token conversation interleaved with three ~700-token requests, then
the same conversation again:
min 0 skip nothing, side requests park, L is evicted
re-send L: 0 reused + 19198 read in 41,059 ms (wall 41.4 s)
min 12288 skip parking (703 tokens, below the 12288 minimum) x3
re-send L: 19193 reused + 5 read in 135 ms (wall 0.3 s, 304x)
The default 0 keeps the previous behaviour, so nothing changes without the flag:
wants() answers for the length alone and the gate stays enabled() && wants(),
exactly as the existing byte-budget and RAM-floor skips are. The check runs
before make_room(), so a skipped park neither evicts nor stores anything, and it
returns the same non-error path those skips take.
Tests: conversation_cache_test covers the inclusive boundary, 0, a negative that
never reaches the CLI, and that a disabled cache still answers for length; both
faults were reintroduced by hand to confirm the checks fail on them.
AMD_HIP.md, RDNA4 speed switches: STRATA_PF_FUSED=1 now runs the native
IQ packs' prompt experts on the int8-WMMA kernels on gfx1200 / gfx1201.
Says that it is opt-in and why (it rounds differently, so it is not an
arch default), that STRATA_PF_FUSED_NATIVE=0 turns the native path off,
the start-up line it prints, the R9700 measurements (prompt time, kernel
vs MMQ, first-token KL against the FP16 path, retrieval checks), and
what was not run (gfx1200, Windows, K-quant packs).
DETAILS.md: the STRATA_PF_FUSED=1 sentence says where the native kernels
run on HIP. CMakeLists.txt and gfx_arch.hpp: the comments that said the
native prompt-expert kernels are gfx11 only now include gfx12 (comments
only, no build change).
The native IQ / K-quant prompt-expert kernels (moe_fused_iq.cu,
native_w11_kernel) were gfx11-only; on RDNA4 a pack with
STRATA_PF_FUSED=1 silently kept the MMQ path. They now compile and run
on gfx1200 / gfx1201. Still opt-in: STRATA_PF_FUSED=1 stays off by
default (it changes rounding, so it is not an arch_defaults switch), and
STRATA_PF_FUSED_NATIVE=0 still turns the native kernels off.
Compile gating
- STRATA_NAT_W12 for __gfx1200__ / __gfx1201__, STRATA_NAT_WMMA for
either family. The kernel calls the _gfx12 spelling of the iu8 WMMA
builtin (the un-suffixed one does not compile there). Any other
target keeps the trapping body.
Kernel (same arithmetic, other lane layout)
- gfx12 WMMA fragments are 8 int8 per lane and the two lane halves hold
different k (8*hi .. 8*hi+7), not a replica as on gfx11: B and A are
loaded as uint2 at the matching offsets, and every 16-value k step is
still one WMMA, so the per-16 scales of the IQ2_S / IQ2_XS paths are
untouched.
- A lane's 8 accumulator rows are consecutive (8*hi + i), so a feature's
gate and up are in the same lane: SwiGLU needs no __shfl_xor, and the
down tile is stored as two float4 per lane. NW_DROW(i, hi) gives the
row of element i for the per-row scales.
Host side
- gfx_arch_is_gfx12_wmma(): exact gfx1200 / gfx1201 match, never a
prefix, like the gfx11 helper.
- dev_info() takes the gfx11 list or gfx12, and also checks that the
kernel the build loaded is the real one: the trap stub (a
gfx12-generic or SPIR-V build on a listed card) declares no LDS and
still passes hipFuncGetAttributes and the occupancy query.
- native_supported() reads the STRATA_PF_FUSED=1 opt-in itself:
fused::available() / requested() belong to the Q2_0 pack's
expert_w11_kernel, which is not ported and stays false on gfx12. Off
gfx12 the result is what it was. prefill.cpp fused_ring() follows:
a native pack asks native_supported() alone (it implied enabled()
before), so on gfx12 it gets the fused buffer layout and ring.
- Grid factor per format: gfx12 holds 3 blocks per WGP for IQ2_S / IQ2_XS
gate/up (LDS and VGPRs), 4 for the rest. The occupancy API cannot say
(it models 64 KB of LDS per multiprocessor), so it is a small table;
STRATA_PF_OCC_W=N overrides it. gfx11 keeps 4.
- experts_native() refuses a dm that is not 16-byte aligned on gfx12
(the float4 stores); the prompt path's buffers always are.
- moe_fused.cu compares the gfx11 list with the helper instead of the
"gfx11" prefix (no change on any listed part).
Tests (prefill_fused_iq_test)
- Runs on gfx12 (the skip gate also asks native_supported()), covers the
other four pairs of the Coder IQ1_M pack, S20_E sets the expert count,
and the real-shape check of fused against MMQ is now rel RMS <= 1e-3
and worst row <= 2e-2 instead of rel RMS <= 0.05.
Measured on a Radeon AI PRO R9700 (gfx1201), Coder IQ1_M pack:
- parity: all 10 IQ pairs plus Q4_K / Q5_K pass, rel RMS <= 9.1e-5.
- kernels vs MMQ, 48 layers: 1.48x / 1.20x / 1.14x at 4K / 16K / 32K.
- prompt speed in the engine (server-reported, alternating starts per
arm): 4K +12% to +19% depending on the start, 16K +8.3%, 32K +8.5%.
- first-token KL against the FP16 prompt path is 1.10x MMQ's (48 prompts
of 4K), top-1 agreement 48/48; 4/4 long-context retrieval checks.
Not run on gfx11 or CUDA hardware here.
unsloth/Qwen3.8-Flash-Next-GGUF UD-Q5_K_XL shard 1 holds metadata only and
is 10,946,618 bytes; its 32-byte-aligned data start is 10,946,624, so the
engine refused it with 'GGUF: data section starts past EOF'. llama.cpp's
gguf-py reads the same file (0 tensors, data_offset 10,946,624).
A file with tensors is still refused. A file without them gets data_start
clamped to the file size, so file_size() - data_start() stays 0 in the
payload checks downstream.
On one GPU the prompt path computes the draft layer's K/V for each chunk in batches (Prefill::draft_kv, E-9). A layer
split always ran the drafter's own per-group pass instead, because the drafter sits on the last stage. With
STRATA_SPLIT_MTP_BATCH=1 the last stage's prompt path runs E-9 there. Default off: the default path is unchanged.
The drafter's K/V come from Q8_1 x Q8_0 MMQ instead of its own mmvq, so its drafts can differ; the target's tokens are
decided by the verify window.
Prefill::draft_kv says why it declined (optional out-parameter, also in the SYCL port's prefill.cpp), and the split
logs once what the variable did. Which path runs is a pure header (include/strata/program/draft_kv_plan.hpp) with a
CPU test. tools/split_mtp_batch_parity.py compares the greedy texts of the two arms on a live server. Docs: MULTI_GPU.md.
Two synthetic cases: no tensors and the file ends at its header (should
open; data_start() clamped to the file size), and one tensor with the same
truncation (still refused). The first fails on 0.1.40.3: unsloth's
UD-Q5_K_XL shard 1 is that case.
Header-only placement (include/strata/platform/aux_cpus.hpp): the spare CPU set (the host core's SMT siblings, then
CPUs of physical cores with no worker; never a CPU that shares a core with a worker), the list parser, threads the
engine creates pin themselves, threads somebody else created (the CUDA driver's) are moved by a sweep over
/proc/self/task after the pool exists and at every request; pool workers and the host thread are never moved.
--aux-cpus off|auto|LIST / STRATA_AUX_CPUS; per request strata_tune aux_cpus=0|1 (serve/server.py).
Default off: the hooks do nothing. tests/platform/aux_cpus_test covers the derivation and live threads (CPU only).
Builds on #1190. The layer split sized its lend regions (the RAM copies of the experts the prompt path borrows) with
Prefill::bytes_needed before the split's ring override was applied (Prefill::set_ring_override says it must be set
before any bytes_needed), so with STRATA_SPLIT_RING=384 the prompt path borrowed 5396 slots and the copy kept 4662.
The override is now decided first (decide_split_ring); include/strata/program/split_ring.hpp holds the pure helpers
(split_ring_slots, split_lend_slots - the scan's own tail-slot arithmetic, now one function for the region and the
borrow), src/program/split_ring_test.cpp covers them on the CPU. A start line reports borrowed vs kept slots and warns
when kept < borrowed. Docs: MULTI_GPU.md.
Engine 0.1.40 (fe9c10ca, #848) keeps the resident RAM copy on a layer split -
every stage's cache left out of it, the rest ranked by the whole expert
profile - and --resident-budget-gib runs through the same code
(pin_cache_complement), so the old engine refusal of the budget on a split is
gone. setup.py's split_mmap() got the matching exemption (43c8cfa9), but the
budget's three paths still assumed the old engine (#498):
- split_budget(): a start of a budget config on several GPUs deleted
--resident-budget-gib (and asked for ~135 GB of RAM below that);
- offer_together(): a budget config was offered both cards only with ~135 GB
of RAM, and "on both it has no budget";
- unsloth_together() / q4_split at install: --gpus, or "2" when asked, dropped
the budget, and an explicit --resident-budget-gib kept one GPU.
From RESIDENT_SPLIT_ENGINE the budget is kept in all three, at any RAM. The
recommendations stay as they were (the golden configs are unchanged): one GPU
by default at install, "n" by default when a start offers both cards, --yes
alone one GPU; the split is taken only when asked for.
Tests: the #498 tests run with MIN_ENGINE pinned before RESIDENT_SPLIT_ENGINE
(as test_setup_risk's StartOnSeveralGpus does for #642), three new ones run the
kept budget; tools/test_setup_*.py on Linux: 421 tests, the 2 failures of
test_setup_engine_hash are the same without this change (no ready-made 0.1.41
engine to download in the test box).
Measured with engine 0.1.41 on an RTX 3090 (x16) + RTX 3080 Ti (x4), 91 GB of
RAM, UD-Q4_K_XL, budget 64 GiB: the auto split gives the x4 card layers 0-1
(678 slots) and the 3090 layers 2-47 (4,182 slots); 57.6 GiB of experts in RAM,
no file-tier reads per request; decode 45.0 / 45.1 tok/s (two runs) against
38.4 on the 3090 alone, a 20K prompt in ~18 s on both. Answers checked by hand.
A pip launcher whose Python was removed stays on PATH ('ninja --version'
fails) and stopped every build there, though setup had just installed a
working copy in .venv.
UPDATE.bat currently prints a command using `&&` for users to copy and
run manually. `&&` is not supported by Windows PowerShell 5.1, which
is still included with Windows 11.
Print the two Git commands separately instead.
dpct migrated cudaStream_t ext_stream_ = nullptr to a pointer to the
in-order queue, evaluated at construction on whichever device is current.
Two places then misread it: init() compared ext_stream_ against the
*init-time* device's in-order queue, so a stage built on one device and
init'ed on another adopted the constructing device's queue
(cross-context submit failures, UR_RESULT_ERROR_DEVICE_LOST); teardown's
'cs_ != ext_stream_' ownership test drifted the same way.
Restore the CUDA contract: nullptr means 'no stream until set_stream()',
matching include/strata/core/verify.hpp:319 and src/core/verify.cpp:604.
Follow-up to the >4 GiB GEMM staging.
- gemm_stage freed its scratch as soon as a larger operand arrived, but the
in-order queue is asynchronous: a previous staged GEMM could still read the
buffer. Wait for the queue before the free.
- The staging used dpct::get_in_order_queue() while the GEMM runs on the
descriptor's queue (strata::q_of(stream_)); pass the descriptor's queue so
the copy is ordered before the GEMM for a non-default stream, and so two
Gemm objects on different queues do not share the thread_local scratch.
- Staging that could not allocate fell silently back to the direct (wrong)
GEMM; it now says so.
- device_alloc_map had no unregister, so a freed cache base left a stale
entry that device_offset_end could mis-attribute after the VA was reused.
Add device_alloc_unregister and call it where the expert cache frees base_.
setup reads the Intel cards from sysfs but to_sycl() decided the expert
mode from the driver alone, so a card's device id was thrown away and an
A-series on xe (the A770) could not be told from a Battlemage. Carry the
PCI device id through intel_gpus() into to_sycl(), and treat every
Alchemist id (A770/A750/A580/A380/A310/A770M) as an A-series: it has no
FP64 hardware, so its config carries the driver's FP64 emulation (a kernel
on the sampled path declares double and is refused without it).
On i915 (the A750) the experts still load into a CPU-computed RAM arena
(--mmap-experts + STRATA_VERIFY_NO_HOST=0) - a single pinned host
allocation past ~3 GiB fails on that die. On xe (the A770) the A-series
now takes the ordinary streamed path: the mirror is built in chunks (so
the >3 GiB limit is not hit) and the build passes
-ze-opt-greater-than-4GB-buffer-required, so the prompt path is correct
past the cache's 4 GiB. tools/test_setup_sycl.py passes.
On an Arc A770 every prompt answered "!!!!" (token 0). The experts in the
last 0.46 GiB of the 4.46 GiB VRAM expert cache dequantized to NaN in the
prompt path. The slots' bytes were right (a memcpy read back equal to the
GGUF, and a fresh copy dequantized clean); only kernels reading them in place
got wrong bytes. In a JIT build IGC addresses kernel pointer arguments as a
32-bit offset from the allocation's base.
The JIT (non-AOT) build now passes -ze-opt-greater-than-4GB-buffer-required
to the spir64 backend, in the same argument as the divide/sqrt option (CMake
drops a repeated -Xsycl-target-backend=spir64). Measured on the A770, Coder
IQ1_M, --stream-experts: prompts of 73-542 tokens have 0 non-finite values
under STRATA_DBG_NAN=1, and the 542-token prompt answers right. The AOT build
is unchanged. Speed cost not measured.
oneMKL's GEMM reads the last, partial tile of an operand from the wrong
place when that operand lies more than 4 GiB into its device allocation
(measured on an Arc A770, oneMKL 2026.1: a BF16 B past 4 GiB gave its last
21 columns wrong, none below 4 GiB). The prompt path's GEMMs meet this:
their buffers come from the top of the multi-GiB expert cache / prefill
region. This is the leading explanation for the A770's garbage on prompts
longer than a few dozen tokens - it repros with every expert
device-resident, so it is not the CPU handshake.
Register every big device allocation (malloc_device_guarded, and the
segmented expert cache's reserved VA) in a small registry, add
device_offset_end(ptr, bytes), and in Gemm::bf16 / Gemm::f16 copy either
operand into a scratch buffer of the queue's own (a fresh allocation, so
at offset 0) when its end lies past 4 GiB, then run the GEMM on the staged
operand. STRATA_SYCL_GEMM_NO_STAGE=1 is the negative control.
Not validated end to end: the run that showed the garbage was an
all-resident mirror, which no longer fits in this machine's RAM, and the
runnable --mmap-experts config crashes on this A770 with or without the
staging (exit 139, UR_RESULT_ERROR_DEVICE_LOST - verified a separate,
pre-existing fault with the negative control), so the staging has not yet
been shown to change an answer.
The decode handshake publishes a payload (the layer's activations, the
routed expert ids and weights) and then rings a sequence number the host
polls. The fences alone do not order the ring after the payload stores:
measured on an Arc A770, 14 of 2,832 rings in one run became visible
before the last activation store, so the CPU expert pool sometimes
computed on a partly stale payload. This is the fix the port's own
comments call for (elementwise.hpp: "the host can observe the ring and
then read a payload that has not landed").
The publish kernels now write, next to the sequence number, the ring it
belongs to and a checksum of the payload (the wrapping sum of every 32-bit
word mixed with its position, murmur3's finalizer - a plain sum missed a
reordering, [3,7] -> [7,3], of the expert ids) in a per-ring slot modulo 4,
and the payload is stored with the uncached write hint (a plain store to
host USM reaches the host only when the kernel ends on xe). The host waits
for the whole payload (doorbell_wait_payload) before the pool reads it, in
session_loop and both verify-window loops; the checksum is accepted in
either publish shape (with or without the routing weights), so a variant's
choice need not be plumbed to the reader. The sequence word grows from 1 to
9 mapped words (layer.cpp, verify.cpp) and the ring slot is reset with it.
Note this is a robustness fix, not the cure for the A770's garbage on
longer prompts: that one repros with every expert device-resident (no CPU
handshake) and points at oneMKL's >4 GiB operand bug, which needs the BLAS
staging the CUDA/HIP shims have.
At a long context this port borrows expert-cache slots for the prompt
path. Those buffers are carved from the top of the ~10 GiB expert-cache
allocation, and oneMKL's BF16 GEMM reads a wrong tile when an operand lies
more than 4 GiB into its allocation (docs/INTEL.md). The borrowed operands
hit that, so from the first full prefill chunk the router read garbage.
Measured on an Arc A770: a 236-token prompt failed with "prefill: routed
id out of range", and a 452-token one segfaulted the engine (exit 139),
on both the mirrored and mmap configurations. With the prompt path on its
own buffers (its own allocation, reserved exactly by owned_prefill_mib)
and an automatic chunk capped at 2048 tokens, 452 / 1,748 / 3,476-token
prefill chunks ran to completion with no prefill failure (prompt reads
22 (cold) / 676 / 835 tok/s). Whether the *answers* of that configuration
are correct is a separate, still-open issue: the base port has no doorbell
payload checksum and its longer-prompt path garbles on some inputs, with
or without borrowing.
Scoped to Alchemist (PCI device ids, the same table setup_intel.py uses):
Battlemage's long prompts were measured correct with borrowing, so it
keeps the old default. --prefill-borrow still borrows, for testing a
future BLAS staging fix (the fix that lets borrowing return, as the
CUDA/HIP shims already stage).
Two Alchemist (Arc A-series) fixes for the streamed-expert mirror:
- gguf_expert_source: a single sycl::malloc_host past ~3 GiB fails on DG2,
so the mirror (thousands of experts, tens of GiB) could not be built on
an A770 at all. It is now allocated in <=2 GiB chunks; the per-(layer,
expert) address, the layer's first blob and close() use the chunk
pointers. A cleanup guard frees every chunk on an allocation failure,
a failed read, or any exception; the read phase is unchanged.
- strata-sycl.sh: docker's default memlock is 8 MiB, far below what pinning
the mirror or the resident arena needs; pass --ulimit memlock=-1.
Measured on an Arc A770: the Coder IQ1_M (8,474 experts missing VRAM,
16.15 GiB) mirrors and the engine starts; before, it refused with
"8474 experts are neither in VRAM nor mirrored".
The decode handshake publishes a ring sequence the host polls while the
window graph still runs. On an Arc (xe) a system-scope atomic store, even
with a release fence, is served from the GPU's cache and reaches the host
only when the kernel ends, so a host polling mid-window never sees the
ring. That is why the verify window had to wait for the whole graph
(STRATA_VERIFY_NO_HOST) and every expert had to be device-resident or in
the pinned mirror.
STRATA_DOORBELL_ATOMIC_LOAD already fixes the load side the same way; this
does the store side with the Intel uncached L1+L3 write hint. The release
fence stays (ordering only: a later payload store follows the ring).
Measured with sycl/probe/doorbell_store.cpp on an Arc A770: the host sees
the store at 230 ms with the kernel running to 2030 ms; with the old atomic
store it sees it at 2038 ms, exactly when the kernel ends (exit code 3).
-DSTRATA_DOORBELL_ATOMIC_STORE restores the atomic store.
This is a prerequisite for running with the CPU expert pool on xe (the
per-layer handshake); it does not by itself fix longer-prompt prefill.
psutil.disk_io_counters() returns None when it finds no disk (Modal's gVisor
has no /proc/diskstats; psutil's docs say Windows can need `diskperf -y`).
_disk() then raised AttributeError at the first sample and the sampler thread
ended: /metrics' hardware stayed empty, the Monitor tab had nothing to show,
and the frozen-engine check's gpu_busy() read a GPU load of 0. It now returns
no disk reading and the other readings go on.
The READY-read timeout from #1317 kills the engine, but the request that
was loading it ended with no response: the RuntimeError from __init__
matched none of the request path's handlers, and on a lazy first start
restart()'s finally crashed on slot_cv (created only after READY), so a
misleading 400 masked the failure instead.
- raise EngineDied (still RuntimeError) for the start failure, so the
request ends 503 'did not report READY ...; the next request restarts
it' on the load, restart and /v1/load paths alike
- skip the slot_cv wakeup when a lazy first start never created it
serve/test_ready_timeout.py covers both paths with silent fake engines:
the lazy load (unfixed: 400) and a restart that never reports READY
(unfixed: connection closed with no answer).
#1352 orders an auto layer split faster-card-last because the last stage
runs the head, the draft layer and the verify. On a VRAM-asymmetric pair
that rule can put the heaviest stage on the card least able to hold it:
the reporter's 12 GB 4070 Ti ended up with 31 layers + head + verify and
a 3 GiB expert cache (decode ~58 -> ~44 tok/s, the 'card this small'
warning on).
Skip the reorder when it would put the last stage on a card with less
total VRAM than the one the config placed there (NVML / amdgpu sysfs;
unreadable cards keep the reorder). scores/vrams stay injectable for
tests.
The VRAM reuse stays, but bytes_needed_impl counts bo again by default, so the auto chunk and the borrowed cache slots
are those of 0.1.40.3 and the prompt residuals stay byte-identical (on the RTX 3060 the planner otherwise picked a
6656-token chunk instead of 6400). STRATA_EMB_REUSE_ACCOUNT=1 lets the planner use the saved bytes (+3.9% prompt there);
documented in docs/DETAILS.md.
The previous change skipped the DRM-fdinfo subtraction whenever the driver reported usage. On a single Arc Pro B70 that
made the expert cache 12168 slots instead of 11338 (23.09 vs 21.52 GiB), so a one-GPU run was not unchanged. The
subtraction is now skipped only when the process sees more than one GPU (where it double-counts the other card) and the
driver does not report the whole card as free.
The bound of the device's spins on host flags was a build constant: 20,000 reads for a bmg AOT build, 2,000,000
for anything else. A B70 built without AOT (SPIR-V, JIT: the Windows/OpenCL build, or a Linux build with no AOT
target) got the A750-sized bound. The issue reports a 37% drop in decode between 0.1.40.2 and 0.1.40.3. On a Linux B70 JIT build the two bounds decode
alike (--spec 2, 3 pairs: 19.4 vs 19.5 tok/s), so this makes the bound independent of the AOT flag but is not shown to be that drop's cause.
spin_max(queue) decides per device: 20,000 under the xe driver (the B-series on Linux), 2,000,000 elsewhere (the i915
A-series needs it for the host's per-layer expert work; a Windows/OpenCL B-series build measured faster with the long bound).
STRATA_SPIN_MAX=<reads> overrides it at run time; -DSTRATA_SYCL_SPIN_MAX=<reads> still fixes one bound for the
whole build. The three waiting kernels take the bound as an argument.
- generate.cpp: with a layer split the host mirror walked the later stages' layers too (#1054, fixed in #1111 but
lost in the re-migration): 20 GiB of pinned RAM for experts the other card holds, or one 39 GiB pinned allocation
that fails. Both loops stop at the first stage's last layer.
- expert_cache.cpp: device_free_bytes() subtracted the DRM bytes of every card of the process from the card it asked
about, so on a layer split the second card was charged the first one's cache and its expert cache would not open.
The fallback is for drivers that report the whole card as free (Arc A750/i915); it now runs only then.
Reported by crobe201 with both diffs; MaineTim confirmed the second on 2x B60.
Co-authored-by: crobe201 <[email protected]>
The three sites fixed by the previous commit stored a clamped (d, sum) but still
quantized against the unclamped d, so an overflowing block stored int8 values
that no longer matched its stored scale. They now use q8_1_finite(d) and
q8_1_quant like native_quantize_q8_1_kernel, the swiglu quantizer and the CPU
pool. A block whose amax is finite and below 8.3M is stored bit for bit as
before.
Co-authored-by: Matthew Hines <[email protected]>
Three kernels still emit the (d, sum) pair of a q8_1 activation block
with a raw make_half2(d, sum), bypassing the finite helper the rest of
the tree uses (q8_1_finite.hpp, from 19de7f4 / f6601bf):
fused_gr.cu gr_q8_tail - the fused gate-reduce tail (QFUSE)
verify_kernels.cu gdn_q8_1_store - the batched GDN verify store (QFUSE)
iq_kernels.cu s26_swiglu_q8_1_kernel - the S26 swiglu path
A block whose amax overflows (or that reads a non-finite input) emits
d = inf/NaN; fp16 makes the pair NaN (or the pathological (0,0)). Nothing
downstream tests for it: a NaN scale poisons the fp16 dot of the quantized
activation and every sum downstream of it, and a fully non-finite logit
row ends at the sampler's all-NaN fallback - vocabulary id 0, the '!'
face of the degeneration reported in #606. Same defect class already
fixed at the other emit sites; these three were missed.
q8_1_ds() clamps both halves into the fp16-finite range (finite d,
non-negative finite sum - the sum is a mean of |x|), producing for the
GPU paths the same bits the CPU pool and the clamped GPU sites produce
for the same block. A clamp at the producer, content-blind: nothing
downstream of it is model-text-aware.
Criteria and the wider carried-state RCA: docs/Q8_1_FINITE_DS.md and
docs/DEGENERATION_RCA.md.
2e4ddf6 (the #904 port) removed __restrict__ from the activation pointers of
native_quantize_q8_1_kernel, bf16_f32_mmvf_multi_kernel and
native_mmvq_multi_kernel on every CUDA architecture, and turned on the weight
prefetch before pdl_wait(). Both only matter for PDL (sm_90+). On sm_61 the
missing restrict loses the read-only data path (LDG.CI) and decode halves
(36.4 -> 17.7 tok/s on a Tesla P40, 32-36 -> 17.7-18.6 on a GTX 1080 + 1070).
STRATA_PDL_RESTRICT is __restrict__ and kPdlPrefetch false again for
__CUDA_ARCH__ < 700 in the device pass. sm_70 and newer compile to the same
SASS as before (cuobjdump -sass of native_mmvq.cu and native_bf16.cu at sm_70,
86, 90 and 120 is identical); the host pass is unchanged.
Reported and bisected by paulhothersall and lineape.
2e4ddf6 (the #904 port) removed __restrict__ from the activation pointers of
native_quantize_q8_1_kernel, bf16_f32_mmvf_multi_kernel and
native_mmvq_multi_kernel on every CUDA architecture, and turned on the weight
prefetch before pdl_wait(). Both only matter for PDL (sm_90+). On sm_61 the
missing restrict loses the read-only data path (LDG.CI) and decode halves
(36.4 -> 17.7 tok/s on a Tesla P40, 32-36 -> 17.7-18.6 on a GTX 1080 + 1070).
STRATA_PDL_RESTRICT is __restrict__ and kPdlPrefetch false again for
__CUDA_ARCH__ < 700 in the device pass. sm_70 and newer compile to the same
SASS as before (cuobjdump -sass of native_mmvq.cu and native_bf16.cu at sm_70,
86, 90 and 120 is identical); the host pass is unchanged.
Reported and bisected by paulhothersall and lineape.
Found while running it against a live engine: the line said 0.35 as 48.7 on round 3 while the three draws were 57.0, 48.7 and 35.5, because it printed the running median. Reading a sweep's drift is the whole point of the line, so it has to be the draw. The medians and the ranges are still in the line below it.
Phase 1 measures each PCIe share once and takes the maximum, with the three prompts of that rate all at the one
setting. Neighbours cannot be ordered that way: on this box 0.75 through 0.95 come out at 72.7, 69.0, 71.0 and
71.9, which is 0.2% apart, while the same settings moved 8-28% between engine starts. A value that was worth a
real gain can lose its single sample and never be measured again, because phase 1's winner is the share phase 2
then sweeps spec_min_p at, and the confirm step only puts that one resulting pair against the default.
sweep() measures every value SWEEP_ROUNDS times inside the one engine start, one round visiting all of them and
every other round in reverse order, and the winners come from the medians. Drift inside a sweep then lands on
every value the same number of times, and one bad sample cannot decide. The same function runs phase 2, and the
confirm step uses the same round count instead of its own.
Cost: the sweep multiplies by SWEEP_ROUNDS, 3 by default. A rate is three prompts of 128 tokens, so on this box
(six values, 95-100 tok/s) phase 1 goes from about 40 s to about 120 s, on a run that already loads the model more
than once. A machine whose two leaders stay inside each other's noise can raise the constant.
Not in this change: phase 4 needs a restart per worker count and stays sequential, so a count is still taken from
one measurement each.
Tests: python -m unittest tools.test_calibrate (the stand-in engine is a pure function of the tune keys, so the
medians and the winners are the same as before); tools.test_setup_config and tools.test_setup_golden
PCIE_FRACS stops at 0.75, so setup --calibrate never measures the value that takes the CPU pool out of the decode
window completely. 1.0 is a separate operating point rather than 0.75 carried further: the share is applied as
m = (nmiss * pcie_num) >> 8, so at any value below 1.0 at least one miss still goes to the CPU pool, and only at
1.0 does every non-resident expert come over PCIe. A CPU expert sits on the layer's critical path while the GPU
waits for it, and on a slow CPU that is the whole difference.
On a Core Ultra 9 290HX Plus with an RTX 5090 Laptop on PCIe 5.0 x16, 0.55 gives 51.1 tok/s against 71.3 at 1.0,
with cpu_exp falling from 1.06 to 0.00 per layer-window and the window time falling by the 11.3 ms of PCIe waiting
plus the CPU's expert time. A sequential sweep puts 1.0 at 78.8 against 72.7 for 0.75 and 51.5 for 0.35.
0.9 is added beside 1.0 so a machine where 1.0 is too much gets an interior point. pick() keeps the default unless a
candidate beats it by MIN_GAIN, so this cannot make a calibration worse. The cost is one rate per value, about 8 s
here.
Tests: python -m unittest tools.test_calibrate; tools.test_setup_config and tools.test_setup_golden
With a second toolkit beside the distribution's one, CMake compiled with the chosen nvcc but took cudart and cuBLAS from /usr (Ubuntu 24.04, STRATA_NVCC=/opt/cuda-13.0/bin/nvcc: the engine needed libcudart.so.12 while built with the 13.0 headers). Pass CUDAToolkit_ROOT for the engine and the CUDA image encoder; the distribution's own /usr/bin/nvcc keeps CMake's search as before.
Two gfx906 build failures on v0.1.40.2. Both are one-line gates that know
about the wave32 HIP backend but not about STRATA_HIP_GFX906 -- the same
class as #808 and #1083. Neither is fixed on main (e8ca9afd03).
1) src/prefill/prefill.cpp:399 builds an event with
cudaEventDisableTiming | cudaEventBlockingSync, but the gfx906 compat
header include/strata/platform/hip_compat/strata_hip.h maps only
cudaEventDisableTiming and cudaEventDefault. The wave32 header
include/strata/hip_compat/cuda_runtime.h:39 has the mapping.
-> add #define cudaEventBlockingSync hipEventBlockingSync there.
A diff of the whole set of cuda* defines between the two headers lists
one other name the gfx906 header lacks, cudaDevAttrIntegrated, and that
one is already gated upstream (#if defined(__linux__) &&
!defined(STRATA_HIP_GFX906) in src/core/expert_cache.cpp:31), so it is
deliberately not touched here.
2) Link error: undefined symbol
void mul_mat_q_case<(ggml_type)14>(ggml_backend_cuda_context&, mmq_args const&, ihipStream_t*)
(14 = GGML_TYPE_Q6_K).
src/prefill/moe_mmq.cu:219 calls it under
#if defined(__HIPCC__) || defined(STRATA_Q6K_EXPERTS), so any HIP
compilation references it; mmq.cuh declares the case as an extern
template, so the definition must come from
template-instances/mmq-instance-q6_k.cu. The CUDA branch of
CMakeLists.txt adds that file only under STRATA_Q6K_EXPERTS.
STRATA_HIP_GFX906=ON forces STRATA_ENABLE_CUDA=ON, so gfx906 takes the
CUDA branch while its .cu files are compiled as HIP.
-> widen the gate: STRATA_Q6K_EXPERTS OR STRATA_HIP_GFX906.
This adds one MMQ instance to the static library. It does not enable
STRATA_Q6K_EXPERTS (the grouped Q6_K expert kernels that allocate extra
VRAM), and STRATA_HIP_GFX906 is only ever ON in the gfx906 build, so
CUDA and wave32 builds are untouched.
Verified on a single Instinct MI50 32 GB (gfx906, wave64), ROCm/HIP 7.14,
-DSTRATA_HIP_GFX906=ON -DCMAKE_HIP_ARCHITECTURES=gfx906
-DSTRATA_MMQ_KQUANTS=ON:
- the engine builds and serves UD-IQ4_XS (Qwen3.8-Flash-Next) end to end:
/v1/models, chat completions, tool calls, MTP drafts accepted;
- nm -C shows mul_mat_q_case<(ggml_type)14> in the binary (13 instances:
7 8 12 13 14 16 17 18 20 21 22 23 42);
- change 2 is load-bearing: removing mmq-instance-q6_k.cu.o from
libstrata_mmq.a and relinking with the same link command line fails with
exactly the undefined symbol above (the archive holds one U reference
and that one W definition), while the untouched archive relinks cleanly.
Two variants of the same sweep (code-explanation prompts and Russian prose) on
one PC and one server config, engine 0.1.40.3 built from source for sm_89.
Each folder carries the 0.1.40.2 run, the 0.1.40 baseline and an
identical-binary control run so the noise band is in the report.
Result: no measurable change from 0.1.40.2 on this machine.
Same laptop, model files, configuration and scripts as the 0.1.40.1
entry (2026-10-06-community-rtx-5090-laptop, #1263); only the Strata
version changed. Raw data for 0.1.40.3, medians for 0.1.40.2 in the
README. Decode and prompt throughput unchanged within run-to-run
variation, all needles found, coding check 10/10 on 0.1.40.3. A 0/10
coding check on 0.1.40.2 is traced to a greedy reasoning loop that
occurs in 0.1.40.1 as well (re-runs documented). Results only, no
engine changes.
Coder IQ1_M, one RTX 5080 16 GB, the repository's own Dockerfile and entrypoint
on a Windows host (Docker Desktop 4.84.0 / WSL2), two context limits on the same
loaded model: 131,072 and 65,536.
- Speed: three runs at each of 4,096 / 8,192 / 16,384 / 32,768 / 65,536 /
130,560 prompt tokens (128K limit) and at 4,096 / 8,192 / 16,384 / 32,768 /
65,024 (64K limit), greedy, reasoning off, 256-token output cap, zero reused
prompt tokens in all 33 runs.
- Correctness: tools/needle_bench.py 9 of 9 (32k/64k/128k) at the 128K limit and
6 of 6 at the 64K limit, plus a vision check with the encoder on the CPU.
- Provenance: all three model files SHA-256-verified against the repository's
published LFS oids at the pinned revision; MTP manifest and inventory included.
- Environment: Docker Desktop's default WSL2 memory cap makes setup.py read ~31 GB
of a 64 GB host and silently select the low-RAM mode; the report documents the
.wslconfig change used instead. Also documents the 8-token headroom a request
needs (prompt + max_tokens + 8 <= max_context) and the engine's own VRAM warning
at a 128K context on a 16 GB card.
Results-only; no engine changes.
Nine files following docs/COMMUNITY_BENCHMARKS.md. Engine v0.1.40.3
(tag d5ea713) built from source for sm_75, IQ3_S at the model native
262,144-token window, with --vision and --reasoning-budget-tokens 12000.
Three runs each at 4,096 / 32,768 / 128,000 prompt tokens: decode
70.9 / 77.3 / 71.1 tok/s, prompt 868.9 / 1548.9 / 1626.9 tok/s, medians
in the index and full ranges in the report. needle_bench 6/6 at 32k and
128k. Records that NVLink is present but unused by the engine, so a bridge
on consumer boards is not worth buying.
The flag acts only on --expert-cache-device1..3; a plain layer split got a no-op flag that the docs listed as a reason
for the pipeline and the async tier to switch off. Built on f-141-quick's #1352 pipeline-windows rule (same hunk).
MULTI_GPU.md: the two combine (only STRATA_PIPELINE_ADAPT_ASYNC=0 turns async off); adds the 2x3090 table.
Co-authored-by: adambenhassen <[email protected]>
The function name was everything between `<function=` and `>`, untrimmed. A model that writes a space after the `=`
(seen from Qwen3.8-Flash-Next IQ2_XS in Zed's agent panel) got:
- inside the thinking: the name " capture" matched no declared tool, so the #804 rescue kept the call as reasoning
text; the reply ended with finish_reason "stop" and no tool_calls, and the agent stopped mid-task;
- after the thinking: a call named " capture", which the client does not know, its arguments typed without the
tool's schema.
The name is now stripped where the call is parsed (parse_tool_call), where the #804 rescue checks it against the
declared tools, and where a finished call looks up its schema (_finish_call). A name with surrounding whitespace is
never meaningful, so no text becomes a call that was not one: the #804/#1058 rules still decide whether a call
inside the thinking is an act.
serve/test_tool_name_space.py: both places, streamed and not, fed in 3-character pieces; fails 4/4 before, passes
after. test_reasoning_rescue, test_reasoning_tools and test_frontend unchanged and passing.
NTFS serves unbuffered reads of a mapped file one at a time (WD SN580: 2 MiB reads at QD32
2.35 GB/s mapped vs 3.57 GB/s unmapped). With the file tier unbuffered (--resident-budget-gib
or STRATA_UNBUFFERED_LOAD=1) the mapped view is only a fallback after startup, so it is closed
(STRATA_KEEP_MAPPING=1 keeps it). The prompt path's stager claims runs of consecutive jobs and
reads them with one read_direct batch (STRATA_STAGER_BATCH, default 8 for file-backed blobs).
RTX 5070 Ti, 32 GB RAM, IQ3_XXS, 47.6K prompt, --resident-budget-gib 18 --prefill auto:32768:
1260 -> 1370 tok/s (batch 8) -> 1669 tok/s (view closed).
Ported from #833 onto the rewritten main. Two fixes on the way: unmapped_ is declared on every
platform (mapped_blob reads it outside the Windows block, so Linux did not compile), and
copy_blobs asks resident_blob, which follows the exchange rotation (STRATA_EXCHANGE_ROTATE).
The fast kernels build for CUDA; sm_70 defaults to them, other CUDA cards keep
the plain ones. gr_up_fast_kernel ends with gr_q8_tail like gr_up_multi_kernel,
so STRATA_QFUSE gets its q8_1 image from it too. In the engine (staged read)
only the up projection changes: hc-read0's up 1.08 -> 0.96 ms per window on a
V100-SXM2. fused_gr_bench bitwise, gr_multi_parity OK.
Read off mmvq_il_parity --bench on a V100-SXM2-32GB with the table's own rule
(every measured shape of a class at least 3% faster, else native_mmvq). At 3
columns: the Q5_K head 1094 -> 821 us, Q6_K 12288 rows 68.9 -> 47.9 us, Q4_K
10240 rows 43.9 -> 32.5 us. Bitwise native_mmvq's output (mmvq_il_parity OK).
The AMD grouped-expert layouts (STRATA_EXP_MODE) now build for CUDA too. sm_70
defaults to mode 8 - the codebook grid and the group's q8_1 activations in
shared memory, SwiGLU + q8_1 fused into the gate/up epilogue - every other CUDA
card keeps mode 0. On CUDA, IQ2_XXS gate/up joins the shared-memory kernels,
a format without one falls back to the CUDA layout (not R2), and a strided call
(the verify window's PCIe call) keeps the CUDA layout.
V100-SXM2, setup's IQ2_XS, a verify window's VRAM call (28 groups): IQ2_S
gate/up layers 180 -> 150 us at 3 tokens, IQ2_XXS 170 -> 134 us; bitwise mode
0's output on all 48 layers (native_expert_bench), native_grouped_parity 0
failures in modes 0/2/7/8.
A staged blob (the GGUF role slices assembled into one buffer) is handed to
ExpertCache::fill_slot's cudaMemcpyAsync as-is. From a pageable buffer the
driver stages every copy through an internal bounce buffer - an extra copy at
a fraction of the transfer rate, on the calling thread - so the admit path,
the profile prefill and the lent-slot refills all pay it per expert. Allocate
the pool's buffers with cudaHostAlloc instead, falling back to pageable for a
buffer when the driver refuses (pinned memory is scarce on the small-RAM
machines this path serves).
DETAILS.md's STRATA_PREFILL_CPU_SHARE section: chunks below 3,072 tokens are staged when it is on
(STRATA_PREFILL_CPU_SHARE_MAX=1024 keeps the old limit), the experts the CPU may take (the arena, the
page-locked copy, the mapped experts.bin's pages), the layer split, and a table measured on an RTX 5070 Ti
alone and an RTX 3060 + RTX 5070 Ti split (PCIe 3.0 x8, DDR4-3200, --mmap-experts): 512-token prompts
-49% / -41%, 3,000-token -22% / -24%; a recorded 100K-token coding conversation on the two cards 137.4 ->
130.6 s.
STRATA_PREFILL_CPU_SHARE handed the CPU pool its share of a chunk below 1,024 tokens only, only experts in
page-locked RAM, and only on one GPU. With it set:
- the chunks staged after their routing (the only ones the share applies to) go up to
STRATA_PREFILL_CPU_SHARE_MAX tokens (default 3072, at least 1024) instead of 1024; Prefill::arm_cpu_share
sets the limit before any loan is sized, and init sets it too;
- an expert whose blob the CPU reads from RAM without assembling it (the arena's unpinned part, the mapped
experts.bin's pages) can go to the CPU, not only a page-locked one; the blob pointers are taken when the
experts are chosen, and one without a stable blob stays on the GPU;
- on a layer split every stage has the pool (serve, no --batch), and one stage at a time takes it for a
chunk while the other reads its chunk on the GPU alone.
Unset, nothing changes.
auto balanced the CPU's time per expert against the GPU's per streamed expert, both measured with the share on, so
nothing compared a layer with and without it. On jase100k's RX 7900 GRE + Ryzen 7 5700X3D the host's time per
streamed expert rose 107 -> 200 us with auto (0.47): the streaming side saved nothing (0.53 x 1.87 = 0.99) and the
CPU's work came on top, -4.5%. Now every eligible layer is timed with CUDA events from before its routing sync to its
combine, per non-resident expert; adjacent layers alternate no share / the measured share until three ratios are in,
then auto shares while the median of the last five is below 1, with one layer in 29 in the other arm. Layers with a
one-time cost and the first layer are left out. STRATA_DBG_CPU_GATE=1 prints each measurement.
RTX 5090 + 9950X3D, IQ2_XS, 250 / 600 / 900 tokens against off: DMA streaming -9.8/-9.9/-10.4% -> -8.8/-9.1/-10.5%;
an emulated host-copied stream -18.0/-16.7/-14.4% -> -16.1/-15.2/-13.0%; one core + one worker (where the old auto
lost up to +11.5%) +1.4..+2.4% -> +2.0..+2.3%, the cost of the first measurements. Unset: byte-identical.
The parking path has two gates in series: make_room() balances the cache's own budget, then conversation_memory_admit() checks the host's free RAM against the snapshot plus --conversation-cache-min-free-mib. The second was final: a failure dropped the snapshot even though the cache held conversations whose RAM it could have, and the next request read its whole prompt again (270k tokens, 96 s, measured by the reporter in #1347).
Evict the least recently active parked conversations until the gate admits the snapshot, bounded by the slot count; entries that hold a pinned shared prefix are not evicted. The failure line says how many were evicted and how many remain.
Reported with the fix by alanthinker in #1347.
The 16 dense GEMM geometries a Qwen3.8-Flash-Next prompt logs on a Radeon 890M (gfx1150, Strix Point) at
T = 64 .. 16384, 144 rows, calibrated with tools/hip/tune_hipblaslt.cpp against AMD's TheRock ROCm 7.14.1 for
gfx1150 (hipBLASLt 1.4.1 cd957402). Two passes back to back; each row keeps the solution with the lowest median of
all six repetitions (a single repetition stalled now and then, and the tool's mean-of-three picks differed in 26 rows
between the passes).
Every row beats plain hipBLAS, 1.55-22.6x per GEMM. End to end on GSQ-RCO IQ3_XXS, three interleaved rounds: prompts
read 1.5-1.7x faster (3.6K tokens 124 -> 216 tok/s, 7K 130 -> 226 tok/s), decode unchanged; hip_prefill_hipblaslt_gemm
passes with it and a server run logged no fallbacks. Setup does not install for gfx1150, so the table is set by hand
(STRATA_HIPBLASLT_TUNING). Method and data: bench/results/2026-10-07-community-gfx1150.
A verify window's graph waits on the GPU for flags the host raises once
cudaGraphLaunch has returned. A blocking launch returns only when the
graph has finished, so with CUDA_LAUNCH_BLOCKING=1 the first window after
a prompt never ends and the watchdog stops the engine, whatever the prompt
length, GPU count or --pipeline-windows. The engine now says so at start,
and the watchdog's line names it. Nothing changes without the variable.
A pipelined group's window spans its slots up to its last busy one, and the engine keeps no pipelined slot as a
conversation cache, so the least-recently-used pick left leading holes ([_ B]) that cost a whole row each. pick_slot
now fills such a hole first, then follows the spreading order. The engine reports INFO batch_groups so the server
knows the groups really run pipelined. serve/test_slot_alloc.py covers the allocator.
Co-authored-by: blange48 <[email protected]>
The miss cost (190 ms per unit of routed mass, fitted on x16 links) is one
constant for every stage, so --layer-split auto cannot see an asymmetric
pair: on 2x 4090 D with CUDA1 on a physical x1 link (1.6 GB/s measured) it
gives that card 26 layers (K=22) and reads a 29K prompt at ~350 tok/s.
Probe each stage's link inside the search (probe_pcie_h2d_gbps; the
stage-setup probes are skipped when --pcie-frac is given, and the search
needs the reading either way) and scale that stage's miss term by
clamp(20 GB/s / bw, 1, 64). 20 GB/s is pcie_frac_for_gbps' x16 reference;
the lower clamp keeps a symmetric machine on the exact formula the sweep's
K was fitted with (every scale 1.0 -> the objective is unchanged).
With it, auto picks the fit-in-cache boundary on that machine (K=38: the
narrow card's cache holds all of its layers' pairs, zero streaming there)
and reads the same prompt at ~1410 tok/s, ~4x.
The 190 ms mixes the decode CPU-pool cost with the prompt link cost;
scaling the whole per-stage term is the first-order fix (splitting the
constant would need a fresh fit on both a narrow and a wide link).
STRATA_PF_FUSED=1 reaches only chunks of stream_all_min() tokens or more (1,024, measured on a discrete card); a
smaller read runs its experts through MMQ. On Strix Halo every expert is resident and MMQ is ~3x slower on a
~1,000-token read, the size of an agent's turn: 441 / 855-token reads 3.1 / 4.0 s -> 0.9 / 1.2 s, a 12-turn agent
replay 3.4 -> 2.1 s per turn (C2T8, --no-prefill-borrow). It changes bits (the fused path's output, which reads of
1,024+ tokens already get), so section 5: teacher-forced KL against today's path 0.016 on reads under 1,024, the same
as fused vs MMQ on longer reads (0.019); reads of 1,024+ unchanged (KL 0).
Setup installs the ROCm 7.14.0a20260608 wheels for gfx1151 since 0.1.40.2 (#1267); their hipBLASLt is 1.4.0
(100400), which neither gfx1151 table matched, so setup's engine read the prompt through plain hipBLAS.
tune_hipblaslt --workspace-mib 32 on the 100401 table's 90 shapes, on a Ryzen AI Max+ 395 (Radeon 8060S, gfx1151,
128 GB, Fedora 44, kernel 7.2.8) with those wheels: 3.1-14.1x faster than plain hipBLAS per shape (median 6.2x).
hip_prefill_hipblaslt_gemm and hip_prefill_hcd_exact_parity pass with it (fallbacks=0); HIP ctest 61 of 62
(ple_parity needs the Q2_0 fixture). setup picks it up with no code change.
Below N of N lendable slots the uncovered experts are read from the pack during a long prompt (reporter: 18% uncovered = -26% prompt speed on gfx1100). A WARNING line now says so, and docs/AMD_HIP.md documents the headroom knob and the levers from the report. No behaviour change otherwise.
The gated delta rule in the WY form flash-linear-attention uses, all FP32: gdn_chunk_prep_kernel forms k.k / q.k and
T = (I + A)^-1 for every chunk of a 2048-token super-block at once; gdn_chunk_scan_kernel (128 blocks, a key head x
16 value columns each) walks the chunks with its state slice in registers. Every exponent <= 0 (the gates are
negative). The scratch (25.6 MB) is allocated once per device: freed and reallocated per call it cost more host time
than the kernels saved on short prompts. sm_80+, >= 128 SMs, enough shared memory, chunks of >= 128 tokens; else
gdn_rec_kh_kernel. Unset: unchanged.
gdn_rec_parity: against gdn_rec_kh_kernel <= 1.13e-6 (T = 1..4099, 8192, 32768, a non-zero starting state); against
an FP64 recurrence <= 1.05e-6 (the sequential kernel <= 3.2e-7) - FP32 rounding; 199 restarts of 7 shapes bit for
bit (NaN-filled scratch between). RTX 5090: 1.81x the recurrence at 2K-32K; IQ2_XS: the recurrence phase of a 32K
prompt 520-532 -> 306-334 ms (-210 ms, -3.7%), 8K -2.1..-3.6%; first-token KL 0.0016-0.0152, 1.15-1.83x a harmless
control arm (STRATA_PROMPT_ATTN_V1), the same top token.
The MoE input and the routing weights go to host memory on a stream beside the primary's compute stream, so its
own experts start at once instead of behind the copies over its link (mixed_h and w are rewritten only after this
layer's combine, which waits for the peer). STRATA_PF_PEER_XSTREAM=0: the compute stream. With
STRATA_PF_PEER_RING=160, prompt tok/s: 100K 1,854 -> 1,918; 8K and 3.4K unchanged.
Each group's tokens and their row lists (in the experts' order) are built on the host from the rows' pairs and
uploaded once per layer; peer_gather_add is eddoursul/Strata's moe_gather_add with the weight looked up through
the row's pair, so the sums are bitwise those of the per-expert launches. STRATA_PF_PEER_GATHER=0: per expert.
Prompt tok/s against the per-expert launches: 3.4K 825 -> 832, 8K 1,468 -> 1,500, 100K 1,821 -> 1,858.
Co-Authored-By: Eddoursul <[email protected]>
The MoE input goes out as the FFN input's existing FP16 image (mixed_h) and the per-token sums come back as
FP16, saturated at +-65504 (eddoursul/Strata f8de703's FP16 input, 2acfac4's sums_to_f16): half the bytes over
the primary's link both ways. STRATA_PF_PEER_F16=0: FP32.
Peer stream 0.6, prompt tok/s: 3.4K 805 -> 831, 8K 1,284 -> 1,495, 100K 1,578 -> 1,855 (the share off: 1,712).
The 8K and 3.4K answers stay byte-identical to the share off.
Co-Authored-By: Eddoursul <[email protected]>
The design is eddoursul/Strata's second-GPU prompt path (f8de703, 2acfac4, e8ade66): the second GPU adds its
experts' rows into each token's weighted sum and the first GPU's combine adds that to its own experts' rows.
Over the host route the peer scatter-adds its rows into per-token sums (peer_scatter_add, after the fork's
moe_scatter_add: one launch per expert in a fixed order, so the sums are repeatable) and sends T x N floats
back instead of rows_peer x N (up to K = 10 times more); moe_combine_peer adds them to the primary's rows. The
2a row buffers are not allocated (3.4 GB of pinned RAM at 32K-token chunks). STRATA_PF_PEER_SUMS=0: rows.
Same setup, peer stream 0.6, prompt tok/s against the share off: 3.4K 332 -> 805, 8K 762 -> 1,284, 100K
1,712 -> 1,578. The 8K and 3.4K answers are byte-identical to the share off (greedy, 256 / 3,221 tokens).
Co-Authored-By: Eddoursul <[email protected]>
set_peer refused a peer that cannot access the primary directly, so on GPU pairs without P2P (most GeForce
pairs) the prompt path stayed on the primary GPU. Without P2P the share now takes the route the layer split's
helper already uses: the activations and the result rows through mapped pinned host buffers read by copy
kernels, with the compact group buffers (the layer-sized path copies over P2P only). STRATA_PF_PEER_HOST=0
restores the refusal.
RTX 3090 (PCIe 4.0 x4) + RTX 3080 Ti (x16), no P2P, UD-Q4_K_XL, --prefill auto:32768, peer stream 0.35 / 1.0,
prompt tok/s against the share off: 3.4K 332 -> 492 / 551, 8K 763 -> 907 / 776, 100K 1,713 -> 1,072 / 734.
The peer's rows coming back over the primary's x4 link cost more than they save on long prompts; the next
commit sends sums instead.
- tools/check_isa_guard.py disassembles the CPU kernels (objdump + nm on the Linux kernel library, capstone + the /MAP
file of a linked binary on Windows) and fails when a function holding AVX-512 / AVX-VNNI-class instructions is
not a kernel of a wide-ISA file: a weak or COMDAT copy (the linker may pick it for every caller), a std:: /
template instantiation, a probe or helper that every CPU runs, or code in another object. MSVC's own
`__isa_available` vectorizer guard is accepted. ctest isa_guard_check runs it; exit 77 (skip) when the tools
are missing.
- -DSTRATA_SDE=<sde> adds sde_{hsw,adl}_* tests: the CPU parity tests on an emulated Haswell (AVX2 only) and Alder
Lake (AVX-VNNI, no AVX-512); SDE stops at the first instruction the chip lacks.
- tools/test_check_isa_guard.py: unit tests of the policy.
expert.cpp and iq_avx512.cpp are compiled for AVX-512, iq_avx2.cpp and kq_avx2.cpp for AVX2, and a file compiled
that way may use the wide ISA anywhere in its code. Code that every CPU runs was still defined in them:
- cpu_features, CpuFeatures::reason, cpu_require_expert_support (the function that finds out whether the CPU has
AVX-512) and expert_set_oracle_q8_0 / expert_oracle_q8_0_enabled (the first call of every generate, on any CPU)
were in expert.cpp;
- iq512_supported, iq256_supported and kq256_supported (a comparison native_gu_rows asks before it has checked
what the CPU can run) were in the kernel files; iq512_supported was called on every AVX2-only CPU for every
gate/up call.
They now live in expert_layout.cpp and native_expert.cpp, which are built for the baseline. No kernel changes;
the compiled code of the moved functions is identical, so default output is byte-identical.
RX 7900 XT 20 GB, IQ3_S, Windows HIP, auto cache, default reserve 700: 0.1.40 ended with 4,255 slots (8.12 GiB) and 2,170 MiB free, 0.1.40.2
with 5,512 slots (10.48 GiB) and 323 MiB free; WDDM paged and decode fell from 51.7 to 21-25 tok/s. With --vram-reserve-mib 2600: 4,299 slots
(8.20 GiB), 2,224 MiB free, long prompt 1,015.8 -> 1,355.9 tok/s (+33.5%), decode 24.3 -> 55.5 tok/s (+128%), back inside 0.1.40's ranges.
A reserve given on the command line is kept; a smaller card keeps its own sizing. The cache size changes no output bit.
amdhip64_7.dll imports rocm_kpack.dll, which imports the MSVC runtime. Only the two runtime DLLs were copied beside the exe, so a
full-path load of the bundled one failed with error 126, Windows fell back to System32's copy, and big prompts died with
hipErrorInvalidDeviceFunction (RX 7800 XT, gfx1101). The closure is now read from the PE import tables of the files in the zip
(tools/hip/dll_closure.py, no pefile), by setup.hip_runtime_beside_exe and tools/hip/package_windows.py. On the 0.1.40.2 HIP zip it is
amdhip64_7, amd_comgr, rocm_kpack, msvcp140, vcruntime140, vcruntime140_1. The engine, when the loaded runtime is not the one beside it,
also tries LoadLibraryEx on the bundled one and prints GetLastError. Diagnostic only. Not tested on a Windows AMD card.
The native router reads 512 logits a row; the window path two lines above already tested n_expert == 512 and K == 10, the
per-token fallback tested only native_router_enabled(), so a model with fewer experts read past each row. It now takes the
generic router_top10 like its siblings. native_multi_parity checks the generic router on 256-wide rows.
Reported-by: rwkeyes (#1357)
0.1.40.2 turned fork F4 on by default after single-GPU measurements (+4.2% on an sm_86, +1.3 to +2.2% on an sm_120). A
two-GPU Windows layer split (RTX 4080 SUPER + RTX 5070 Ti) was reported 15% slower with it on (#1373). The outputs are
bitwise the same either way, so only speed changes. STRATA_MMVQ_IL=1 turns it on; docs/DETAILS.md says why.
samp_acc_t is float by default (-DSTRATA_SYCL_SAMPLER_FP64=1 keeps the reference's double). On an Arc A750 one row of the
tail cost 451 us in double and 20 us in float; a sampled decode ran at 7.62 -> 8.10 tok/s (medians of 5 interleaved
pairs, runs 7.1-9.2). sampler_parity: 0 failures on the A750 (JIT build) and on the B70; both cards' 10-prompt gates pass
10 of 10, and the B70 gate's sampled cases pick the same tokens as before. Of the kernels that use double, only the
sampler runs on a native pack (the router, combine, gate and norm kernels there are float).
Probes: sycl/probe/fp64_cost.cpp (FP64 against float in the small reduction kernels), gemm_vary.cpp.
docs/INTEL.md: the aliased-pages bug, NO_HOST, the prompt-chunking numbers, draft-length sweep, FP64 and XMX findings;
docs/INTEL_ARC.md: a row for the B70/A750 runs.
With STRATA_PREFILL_CPU_SHARE on, the last rows_cpu rows of GU and H hold whatever the buffer held before; the check read
them and reported them as non-finite (about 1 in 256 garbage floats), with Dm and the layer output clean.
blob() bumps a plain reads_ counter and, for a file source, its mapped/staged paths; calling it from the share's
thread raced with the prefill thread's own blob_stable()/pinned() calls (the streamed experts). The blobs of the CPU's
experts are now taken before the thread starts, so the thread reads only the arena bytes, the activations and the
grouping tables, none of which the prefill thread writes until the thread is joined.
A chunk below stream_all_min() (1,024 tokens: an agent's tool result, a test's
output) streams every routed expert outside VRAM over PCIe while the decode CPU
pool sits idle and the arena holds the same experts in RAM. Opt-in,
STRATA_PREFILL_CPU_SHARE=auto|x:
- the non-resident experts routed by at most MAXT of the chunk's tokens,
fewest first, run on the pool on a thread while the GPU streams and runs the
rest; their unweighted rows go to Dm's tail (as a peer GPU's do) and the
combine weights them as any other row;
- auto: the share measured, each layer's CPU time per expert and the GPU's per
streamed expert (CUDA events around its expert work, read after the next
layer's routing sync), and the next layers hand the CPU g / (c + g), where
both end together - a small CPU takes a small share;
- Q2_0 layers take the pool's Q2_0 kernels' activations (ActQ, as decode's),
i-quant layers the native vec_dot type.
Serve without batch slots and generate, one GPU. Unset (the default): no
expert moves, byte-identical to v0.1.39 (IQ2_XS 600 and 2,100 tokens, fixed
cache, logits and 32 tokens).
RTX 5090 + 9950X3D, 600-token prompts, ms (off / auto):
- UD-Q4_K_XL: 1,548 / 1,369, 1,528 / 1,296 (share 0.65; KL between 0.006);
- Q2_0: 472 / 445, 491 / 448;
- IQ2_XS: 493 / 481, 502 / 460, 527 / 529; 250 tokens 424 / 355, 402 / 382,
379 / 367 (KL to STRATA_PREFILL_MMQ=0 0.0053 off, 0.0080 auto);
- IQ2_XS on 2 AVX2 workers (emulated): 512 / 487 (share 0.49).
Every prompt chunk streams the experts the VRAM cache does not hold, over PCIe, and a 4K prompt touches nearly all 512
experts of a layer. The short first chunk (STRATA_PREFILL_FIRST, 256 tokens, there to start the GPU while PLE rows are
read) therefore cost a whole extra pass of that stream whenever the cache holds only part of the model. Its default is
now 256 only when the cache holds every expert, else 0 (an explicit STRATA_PREFILL_FIRST still decides).
Arc Pro B70 32 GB, IQ3_S (11.5k of 24.6k experts in VRAM, the rest in the pinned mirror at 13 GB/s), a 4,095-token
prompt, MTP on, interleaved runs, medians of 3: 2048-token chunks 618 -> 730 tok/s, 4096-token chunks 784 -> 980.
sycl/setup_intel.py now writes --prefill 4096 for a card with 24 GB or more (it costs ~700 expert cache slots, about 3%
of decode on the 200-token story).
An Arc Pro B70 (xe) returned the 22 GiB expert-cache arena with two of its 2 MiB pages mapped onto the same memory:
a write to the page at +2 MiB showed at +1022 MiB, so cache slots there held other experts' bytes and the prompt
came out NaN at the first layer that routed to one of them (layer 4, expert 249). Which pages alias depends on the
allocation history (the context size, the MTP window), so it comes and goes between runs.
strata::arena_alias_check tags every 64 KiB of a fresh range and reads the tags back in a second kernel;
strata::malloc_device_guarded allocates again behind a growing spacer until the range is clean (and warns if it
never is). It guards the expert cache arena and every device allocation of 32 MiB or more (session arena, weights,
MTP, verify, head, prefill). STRATA_ARENA_ALIAS_CHECK=0 skips it. Two debug aids: STRATA_VERIFY_ALL_SLOTS=1 (with
STRATA_VERIFY_FIND=1, STRATA_VERIFY_DUMP=dir) reads every filled slot back and compares it with the GGUF, and the
probes sycl/probe/arena_rw.cpp and alias_scan.cpp.
A blocking read of the engine's READY line or of the vision encoder's pipe that
never returned held the request turn for good: the engine sat idle in its command
loop, so no watchdog could fire and every later request queued behind it (the lost
step class of #481). The READY read now has a 900 s timeout, the encoder's READY
and ENC reads 300 s each. A process that says nothing for that long is killed, the
read raises, and the next request starts a fresh one. Slow loads and slow encodes
stay far inside the limits.
Only the read timeouts of the reporter's patch are taken. The watchdog that ends
an engine after 90 s without output and CPU progress is left out for 0.1.41.
Tests: a silent engine, a silent encoder at READY and at ENC, and an in-time answer.
Reported-by: btc1000w
Co-authored-by: btc1000w <[email protected]>
A hint only: it never enables the kernel and says the output bits change. Silent when the variable is set to anything. Follows the R9700 numbers of PR #1318 (+31% prompt at 8.8K).
I calibrated a table for hipBLASLt 1.4.1 with tune_hipblaslt over the 32 geometries of
gfx1201-hipblaslt-100500.txt (R9700, system ROCm 10.1.0, hipBLASLt 1.4.1): 1.15x geometric
mean per GEMM over hipBLAS, but only 1,606 -> 1,613 tok/s median prompt over three matched
trials (a 1.008x geometric mean) and decode unchanged, so within noise - which is what the
R9700 paragraph above already reported, now from a second machine. The cold first prompt of
each start is left out of those numbers.
I dropped two rows from it: bf16 N=10240 K=320 ldy=10240 at T=4096 and at T=8192. hipBLAS
won T=8192 in five separate runs (0.609-0.619 ms against 0.648-0.677 ms for the best of 16
candidates), and dropping only the T=8192 row would send that call to the T=4096 row rather
than back to hipBLAS, because closest() takes the nearest T bucket.
Two calibration traps I ran into are recorded too: the first case of a fresh tune_hipblaslt
process times its hipBLAS baseline while the clocks are still ramping, and the first fresh
prompt after a model load is not a measurement (949 vs 1,423 tok/s in the two arms there).
On my R9700 (engine 0.1.40, IQ3_S at 431072 ctx, --kv int8, the hipBLASLt 1.4.1 table of
the previous commit, GPU idle): fresh prompts of 4,210 and 8,830 tokens, three matched
trials each and the cold first prompt of each start left out - 1,571 -> 2,064 and
1,568 -> 2,065 tok/s at 8,830 tokens (+31.4 and +31.7%) and 1,699 -> 1,990 at 4,210
(+17.1%), medians 1,571 -> 2,064, a 1.27x geometric mean over those three - with decode
unchanged (89.8 -> 86.9 tok/s). hip_prompt_attn_wmma passes on this card.
STRATA_SELECT_WMMA=1 on top of it changed nothing at 4-9K prompts (2,026 tok/s), which is
consistent with its +1.5% being a 16K/262K-ctx number.
The shipped gfx1100-hipblaslt-100401.txt calibrates "the 26 dense GEMM
geometries of the shipped gfx1100 table", all at T=4096/8192. A prompt on
Qwen3.8-Flash-Next reads a whole other geometry through the prompt path, and
it is issued once per expert group at whatever T the group has - so it never
matches a row:
STRATA_HIPBLASLT_VERBOSE=1, GSQ-RCO IQ2_XS, 192000 ctx, 3151-token prompt
hipBLASLt tuning enabled (26 rows, gfx1100, version 100401)
642 fallbacks, all f16 N=1280 K=2560 ldy=1280, over 642 distinct
T values from 1 to 1873
That is the small-T expert GEMM, and it is nearly every launch in the prompt.
TuningTable::closest() matches on (n, k, ldy) and takes the nearest t_bucket,
so the rows below are a spread of buckets over the small-T range rather than
one row per T; bf16 10240 2560 10240 (the hyper-connection projection, one
fallback) is covered the same way.
tools/hip/gfx1100-hipblaslt-100401.txt: 17 rows added with tune_hipblaslt
from this tree, against the engine's own 32 MiB workspace (gemm.cu:441), on
the same RX 7900 XTX and packaged ROCm 10.0.0 the existing rows were
calibrated on. No existing row is changed. 26 rows -> 43.
Same binary, same config, the table as the only difference, 3 interleaved
pairs at a 3151-token prompt:
691.6 -> 940.7 tok/s (+36.0%)
712.3 -> 943.2 tok/s (+32.4%)
727.7 -> 928.7 tok/s (+27.6%)
mean 710.5 -> 937.5 tok/s (+31.9%)
Decode unchanged (58.6 -> 60.6 tok/s, inside this box's 2.3% run drift), and
STRATA_HIPBLASLT_VERBOSE=1 reports fallbacks=0 with the 43 rows, which the
26-row table does not.
The pick becomes argmax_i p_i / E_i with E_i = -log(u_i) and u_i a hash of (seed, counter, token id), an exact
sample of p. The noise a token gets does not depend on which other tokens survived the draft's or the target's
cut, so a coupled draft (STRATA_SPEC_COUPLED=1) agrees with the target on the tokens they share. Off by default:
nothing changes without the variable. sampler_parity: kernel vs host reference under three chains, frequencies
vs softmax, the flag is read and greedy is unchanged.
Ported from #904 onto the 0.1.40 verify window. Every switch is OFF by default, so
default output and the default launch sequence are unchanged:
- STRATA_DF_PDL=1: programmatic dependent launch (pdl.hpp) for the window's
quantizations and dense projections (multi-column MMVQ, multi-row BF16 GEMV);
sm_90+ only, CUDA 12.3+; HIP and older cards keep ordinary launches.
- STRATA_DF_PLE=1: the window's hyper-connection write and PLE post-ops in one
pass, which also writes the commit's history snapshots (native_gr_post_multi
with strides, native_ple_postops_batch_snap).
- STRATA_DF_QB_Q4=1: a Q4_0 KV cache takes the batched QSA query path too.
- STRATA_DF_BRANCH=1: the mixer's side branches inside the window graph.
Left opt-in because the author saw a stall on Linux with NVML queries.
The q/gate split now uses copy_rows_strided (a kernel) instead of a
cudaMemcpy2DAsync so a PDL chain stays kernel to kernel; the bytes are the same.
Left out: STRATA_DF_KVAPP (kv_append_q4_step_multi). 0.1.40 already appends a
window's K/V cells in one launch (kv_append_q4_steps, #783 PR-d).
Merged by hand with the 0.1.40 forms of the same code: the exact-T BF16
multi-token kernel, the PAIR/TS MMVQ variants, QFUSE (q8_attn), the fused
norm+rope and the batched residual write.
Co-authored-by: Francesco Albano <[email protected]>
(PR #904)
The high index bits of a 32-value half are spread one per byte by a table and
punpcklbw puts them next to the low index bytes, so each grid lookup is a
16-bit load plus the table load. The sign vector is one broadcast load plus one
pshufb, and the (2s+1) scale vectors are table loads. Arithmetic is untouched
(same words in the same lanes). The decode is ported onto the 0.1.40 layout of
iq_avx2.cpp: the gathered IQ3_S/IQ2_S forms (Fmt32<121>, <122>) and the VNNI
rows are left as they are.
Co-authored-by: Stefanos Anastasiou <[email protected]>
(PR #930)
Opt-in (only with --expert-cache-per-layer). The same VRAM holds more experts
(7,008 -> 9,312 slots at equal VRAM per the author). --expert-cache N stays the
budget of N largest blobs; shrink_to keeps whole per-layer lists so a blob never
lands in another layer's range. The free-VRAM reading is device_free_bytes(), as
in the shared-cache branch. The PR's png is left out.
(PR #934)
Opt-in (default 0 keeps three tasks per participating thread, so default
output and timing are unchanged). Adds pool_tasks_test, which compares the
split against the default partition bitwise for 0..4096 tasks.
(PR #949)
The prefill Stager waited by spinning: a yield loop on `issued`, and
cudaEventSynchronize on events created without cudaEventBlockingSync
(which spins under the default scheduler). With --mmap-experts there
are 32 stagers. On Linux they inherit the host thread's single-core
pin and starve it: on a DGX Spark (GB10) prompts were read ~10x slower
(IQ3_S 8K 91 -> 1,222 tok/s, 30K 88 -> 1,388 tok/s with this change).
Unpinned instead, the 32 spinners took every core for the whole prompt
(process CPU ~1,800%); with the waits sleeping it is ~100%.
- ring wait: std::atomic::wait, with notify_all in issued_one() and
finish()
- dma_done events: cudaEventDisableTiming | cudaEventBlockingSync
- HIP shim: cudaEventBlockingSync -> hipEventBlockingSync (untested)
Decode speed and output tokens are unchanged.
(PR #1057)
ExpertCache::admit never evicts because eviction policy is a measured
question (R4.1's LFU-decay vs LRU sweep). This is the harness for that
measurement: a policy's hit rate depends only on the order (layer,
expert) pairs were routed, which --dump-routing records, so replaying a
trace through simulated slot pools needs no GPU. fill-only (the
engine's current behaviour), lru, lfu-decay and an oracle (Belady's
MIN, the upper bound); global or R4.2g per-layer scope matching
ExpertCache::layer_slot_range; any slot count; optional profile prefill;
per-window rates, so a topic shift shows up as a per-window dip.
tools/test_bench_eviction.py runs without a GPU (14 cases): each
policy's victim choice, the per-layer quota arithmetic, the windows
split, and profile prefill order.
(PR #991)
A one-int-a-column interleaved copy of the window's q8_1 blocks lets a warp decode each weight block once and apply it
to every token with the single-column dot's own expression (the same *_q8_dot_impl calls), walking the chunks of the
exact multi-column layout and adding the four virtual warps' sums in warp order, so every output is bitwise the
multi-column kernel's. IQ4_XS, Q4_K, Q5_K and Q6_K projections (and the head) of 2-4 tokens use it; the copy is made
from xq_ by one small kernel, so the fused HC-read producer works too. STRATA_MMVQ_IL=0 keeps native_mmvq's kernels.
CUDA only (HIP keeps its kernels). mmvq_il_parity checks every case bitwise for rows 1/2/4 and the table; --bench times it.
Ported by hand from Eddoursul's fork (738e166, b67a291), with the IQ3_S and Q8_0 parts and the 5-8 column widths left out.
Co-authored-by: Eddoursul <[email protected]>
kvg_trim took only the short request's cell count, so every short request in
a long conversation's session shrank the K/V to 8192 cells and gave the
expert slots back, and the conversation's next turn grew it again and moved
the experts again (a parked conversation is restored through kvg_ensure).
With STRATA_KV_GROW_HOLD=1 the trim keeps the longest parked conversation's
cells (+256) while it is parked. Opt-in: the default trims to the request
alone as in 0.1.40, so the K/V gives back its VRAM as before. Output is
unchanged; only --kv-grow is affected.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
With --layer-split auto and STRATA_STAGE_TRIM=1 a later stage's weights load after the search, and the search priced them
as the full arena with no projections, so a small card was over-credited: P40 24 GB + RTX 3070 8 GB picked K=2 (prefill
107 / 112 tok/s at 4K / 32K) where K=16 gives 343 / 654 in the report. The carve is now priced for each candidate range:
the trimmed arena (WeightTable::pool_bytes) plus NativeDense::weight_bytes_for, a walk of load over the GGUF headers that
rounds each matrix to cudaMalloc's 2 MiB granule (+0.8% on the iq3_s pack, high on purpose). After each late stage loads, the
bytes priced are compared with the VRAM it really took and a warning says when they drift.
Only the late-weights arm changes (auto with the trim asked, opt-in); every other run has its weights loaded before the
search, carve_bytes is 0 there, and placements do not move. Ported from 572b607 of #880 onto this tree's loader (layer ranges
as arguments, STRATA_Q8_PACKED, STRATA_HC_Q8 untouched); not run on a mixed pair here.
Co-authored-by: gopinath87607 <[email protected]>
The report shows the bundled amdhip64_7.dll next to strata.exe and the System32 copy (a different build) in use, with the
prompt GEMMs failing. The import is bound before main, so the engine cannot swap it; it now prints a WARNING naming the
bundled file when the loaded one is another, so the log carries the fact. Nothing else changes. The fix itself (loading the
bundled runtime by full path, or a delay-loaded import) needs a Windows AMD machine to test and is not in this commit.
On Windows an engine with 313 MiB of VRAM free after a 15,104-token borrow exited silently inside the verify-window
capture. The cause is not reproduced here (no Windows AMD card), so this only leaves a trace and a hint, and changes
nothing: the capture prints how much VRAM is free before it starts, and a Windows start with less than 512 MiB free
and --prefill auto names --prefill auto:8192 (the reporter's workaround) and --vram-reserve-mib. The chunk choice is
untouched.
The build stopped with a FATAL_ERROR because gfx1150 was in none of the architecture lists, although the device code
(the WMMA guards, sudot4) already covers it. It is now in the unvalidated list: the build warns and goes on. The
report's machine (96 GB, TheRock 7.14.1, IQ3_XXS: decode 17-19 tok/s, prompt about 125-135 tok/s) is in docs/AMD_HIP.md.
Setup's own AMD path is unchanged.
Co-authored-by: zeriyoshi <[email protected]>
Phones save portrait photos as landscape JPEGs with an Orientation tag, and the encoder's decoder ignores it, so the
model saw them sideways. Such a JPEG now takes the Pillow route (exif_transpose, then PNG); JPEGs without the tag, PNG,
BMP and GIF still pass through byte for byte, and without Pillow nothing changes. Other formats that carry the tag
are turned too.
Co-authored-by: crazyaimachine <[email protected]>
On a layer split with --batch-groups, two concurrent requests ran at about half speed: the slot choice could put both in
one group, and a group's window carried a row for every slot, active or not. Slot choice now prefers the group with the
fewest busy slots (a held prefix of 512 tokens or more still wins), and a window covers the slots up to its last active
one. Without --batch-groups the slot choice is exactly as before.
Co-authored-by: crazyaimachine <[email protected]>
The search priced placements by decode time only, so at 512K context it could pick a split where a stage could neither
lend a 512-token prompt chunk nor afford its own buffers, and the engine ran out of VRAM at start while an explicit
--layer-split worked. A placement is now taken only if every stage can lend the last-resort chunk or afford its own
buffers (the serve path's own tests, priced from the same figures); the tie-break for balanced stages stays. A
start-time probe on the live free figures says which card is short.
Changed from the PR: nothing is refused. If no placement can start on paper, the search runs again without the gate
(the choice is then what it was before) and prints what is short and how to make room; the probe only warns.
Co-authored-by: fukc-gihtub <[email protected]>
Every layout of active slots keeps its own graph pair, and with the expert cache sized to the reserve the next window
graph can find no VRAM left; the engine then exited (#997, `parallel` 4 to 8). cudaGraphInstantiate that runs out of
memory now evicts the least recently used batch layouts, other than the one being captured, until it fits; they are
captured again when needed. The start prints how many were freed. Nothing changes while everything fits.
With the key on, a call of a declared tool is the call when the model writes the tool's name as a parameter tag,
a JSON object inside <tool_call>, or a bare <function=NAME> at the start of a line; two calls in one <tool_call>
are two calls. Anything else stays text. Without the key every reply is exactly as before. On the author's corpus of
1,462 agent turns: 92.9% read as intended before, 98.3% with the key. Re-files #966.
Codex's review request sends tools, an additional_tools item and text.format json_schema at once, and got a 400.
Now a tool call passes through as it does without the format (words written before the call go out as a message
ahead of it), and the schema is checked on the turn that ends in text, as json_schema works without tools. With
tools the system directive says the schema is for the final answer only. The chat route's rule is unchanged.
Tests in serve/test_responses.py reproduce the request shape: a call, a valid text answer, and (with jsonschema
installed) an answer that breaks the schema.
The wait for the engine's next line was 10 s, so a disconnect during a quiet prefill reached the engine only at the
next heartbeat. Both waits now poll the cancel event every 0.5 s; the HTTP heartbeat stays at 10 s, and an expired
silence deadline is enforced before another heartbeat. serve/test_quiet_cancel.py drives a fake engine that says
nothing until STOP. This is the Python half of #962; the engine half (stop between prompt chunks) is open.
--force now removes the old pack's markers (index.txt, manifest.json) before the build writes a byte. A forced build
that stops part-way leaves experts.bin without a manifest, which the next build or setup makes again, instead of the
old markers over partly new experts.
A string or list as a tool's schema passed the tools check and raised later, in parse_tool_call and the stream parser,
after the 200 and part of the reply. It is now a 400 naming the tool and the field. A missing or null schema still
passes.
A window size's cost was updated only when that size was chosen, and the re-probe covered only sizes seen fewer than
three times. A size priced high during one slow stretch (a prompt's cache refill) could stay priced out for the life
of the process. A size not measured for 300 rounds is now tried again when a confident lookup could use it, one
round per stretch. Output is unchanged (the policy only picks which drafts to verify); draft_policy_test gains a case
that fails without the change.
HIP keeps the shared expert on the draft's own stream; only CUDA branches. The stream and events were created and
never used, and the report's R9700 lost prompt speed after the first request.
STRATA_PF_FUSED=1 on gfx1100 gave prompts that answered only "!". Two causes, from the report:
- the stage loop wrote the next weight buffer with no barrier after it; one __syncthreads() now follows the write.
- the waves_per_eu(8) variant of native_w11_kernel computes wrong results with clang 22 (ROCm 7.10): there the cap is 1
(NW_LB; -DSTRATA_W_LB=N overrides it). Earlier compilers keep 8.
Default output is unchanged (the path is opt-in on gfx11).
An Arc Pro B70 (xe) page-faults when a kernel reads ordinary pageable host memory (devcoredump: ccs0, FaultType 0,
faulted address in the CPU's mmap range, then Timedout job and device lost). Without --stream-experts the experts
outside the VRAM cache sit in a pageable arena that the PCIe share hands to the GPU. The default share is 0 there
(the CPU computes those misses) with a one-line warning naming --stream-experts, as setup configures it. An explicit
--pcie-frac is kept and warned about. i915 (Arc Alchemist), CUDA and HIP are unchanged.
strata::intel_gpu_driver() reads the driver from sysfs; verify.cpp's i915 test uses it.
The mirror table (host-mirror pointers for the device plan) is filled only from pinned USM aliases under
--stream-experts, so it cannot hand out pageable pointers.
STRATA_QFUSE=1 (the gfx1151 arch default) has gdn_step_norm_multi write the
out-projection's q8_1 image itself and skips native_quantize_q8_1. The batched
recurrence (batch_rec_) calls gdn_step_norm_multi once per slot with no q8_1
destination, so with --batch the projection read stale bytes and every
concurrent answer was corrupt. The batched case now quantizes as without
QFUSE; the fused image holds the same bytes (gr_parity), so batched output
equals solo. Single-request paths are unchanged.
A stage that was 100% resident at init takes the doorbell graph while
a prompt loan/shrink/swap is in flight (ar_off_), and that graph waits
on host doorbells the skipped service never raises: one GPU spins at
100% while the host sits in cudaStreamSynchronize, until the watchdog
fires with the previous stage's stale label. Key the all-resident
service skip in run_slot_rows, batch_poll and service off ar_on(),
as run() already does.
PR #1102 (win10ogod) is the same fix for run_slot_rows and batch_poll;
this one also covers service().
Co-authored-by: win10ogod <[email protected]>
#783's batched append of a verify window's cells passed no host copy for the
hybrid K8V4 format (int8 and q4_0 pass st.host, and the per-token path passes
K8V4's two halves). With --kv-resident, a block that was not resident when its
cells were appended came back from the host copy without them, so the tokens a
window wrote were lost: free generation degenerated into repetition (#1188,
#1135) and parked conversations lost their decode-written turns (#1169). The
batched path now passes the same host halves as the per-token path.
wait_step and the chain wait polled ext_oneapi_empty() through DPCT_CHECK_ERROR, which is always 0, so the
loop left after 1024 spins and read draft tokens the GPU had not written (-1). Same migration bug as the
cudaStreamQuery fix in verify.cpp, missed in mtp.cpp. IQ3_XXS on the A750: 61 of 99 drafts accepted on the
story prompt (the 5070 gets 106 of 166); copy_'s readiness test in verify.cpp had the same flaw.
STRATA_DBG_DRAFT=1 prints each window's drafts and the target's tokens.
On an Arc A750 (i915) a verify window launched behind other GPU work in flight (the previous window's commit,
a prefill) often never started: no GPU breadcrumb, the 20 s ring timeout, the engine ends. queues_wait_and_throw
before the launch removes it (9 of 9 three-request runs pass, almost all failed before). Default on for i915,
off elsewhere (STRATA_WINDOW_SYNC=1/0/2 overrides). STRATA_RING_TIMEOUT_S raises the 20 s doorbell timeout.
Level Zero's free-memory query returns the constant total on an Arc A750 (i915); the expert cache and draft head
were sized from the whole card and the first launch failed with out-of-host-memory. device_free_bytes now lowers
the figure by what this process holds (drm-total-local0 / drm-total-vram0 in /proc/self/fdinfo) and says so once.
On the Arc A750 a kernel never saw the host's doorbell and result-row stores to ordinary host USM
(sycl/probe/doorbell.cpp: only the uncached allocation completes the handshake), so every ring wait
hit its bound and the window read stale rows: NaN logits. verify's mapped staging and the pool's
y_miss rows now use host_malloc_polled (sycl_queue.hpp; needs libze-dev's header, else sycl::malloc_host;
STRATA_HOST_UNCACHED=0 turns it off). IQ3_XXS on the A750 now answers 12*12 = 144.
Also: the expert cache and the MTP draft head check Level Zero's null return for a too-large
allocation (and an Arc A750 needs UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1 for a single allocation over
4 GiB); gather_rows no longer asks for a cooperative launch it does not use; the expert-cache
zeroing in generate.cpp is chunked too. Based on the probe in PR #423 (maxfridbe).
One memset of the 25 GiB expert-cache arena runs on the blitter engine and times out on the
Arc Pro B70 (xe: engine memory CAT error, GT reset). big_fill_zero zeroes in 256 MiB kernel
launches; used for the expert cache, the MTP and verify arenas and the KV reset.
The sycl/ copies were at 0.1.39 and did not build against the 0.1.40.x shared
headers (load_experts_gguf ready argument, fused_gr_read_multi bool, Gemm::bf16/native ldx,
13 symbols missing at link). Re-migrated the 5ffcce43 and 0.1.40.1 trees with migrate.sh,
merged with merge_upstream.py, hand-resolved the conflicts (the port's tuned expert and
MMVQ kernels kept; verify, mtp, generate adopt 0.1.40.1 with the port's queue handling).
Stubs: vmm (no driver VMM on Level Zero), --remote-expert-opt. Dropped upstream's
opt-in AMD S26/S27 grouped-expert kernels and STRATA_QFUSE/HC_Q8 reads on SYCL.
build.sh and migrate.sh source setvars.sh outside set -u (they exited silently).
The first version staged every uncached blob and made the layer wait for all of them before any expert computed, which
lost the overlap the kernels' own page faults had (-19% on a cold 16 GB lane). Now prefetch() hands the layer's uncached
blobs to the I/O threads and returns; the kernels that reach a blob before it lands wait on its pages, the others
compute. STRATA_IO_PF_STAGE=1 keeps the staging variant. Residency is probed on four sampled pages, not the whole blob.
The mapped file tier reads experts through page faults. With STRATA_IO_PREFETCH=1 (Linux, experts.bin or the GGUF
in place) a blob whose pages are not cached is read with one pread of the whole blob into a staging buffer, the
predicted experts of the next layer(s) are read ahead on I/O threads (STRATA_IO_PF_THREADS, STRATA_IO_PREFETCH_DEPTH,
STRATA_IO_PF_AHEAD=0 for demand reads only), and a cached blob is still handed out as the mapping. Same bytes, so the
same output; off by default.
Counters: each request now logs the OS's own view of the file tier (/proc/self/io read_bytes and major faults) beside
the engine's expert reads, and with STRATA_IO_STATS=1 or io prefetch a mincore split of cached and uncached bytes,
read-ahead used / unused / skipped, and the decode's wait in the fetches (--stats prints the same).
blob_stable(): a pointer that is not recycled, for the copies that run asynchronously (adaptive swaps, the GPU cache
fills, the prompt stager).
Asked with Accept: text/plain or ?format=prometheus; every other request keeps the JSON.
Latency histograms, a Grafana dashboard and scrape configs under docs/monitoring.
Only the Prometheus part of #793; the pipelined-window slot allocator needs a 2-GPU check and is left out.
Reads the routing ids first and queues the shared expert afterwards; staging
counts real transfers. STRATA_PREFILL_STREAM_AHEAD=0 restores the old schedule.
--adapt-async 1 no longer drops to the blocking tier with --pipeline-windows 2: the steps that overwrite
something a window in flight may read wait for those windows. STRATA_PIPELINE_ADAPT_ASYNC=0 keeps the
blocking tier there. Opt-in; default output unchanged.
When every placement holds the whole expert profile the predicted decode window is the same for all of them and the winner was floating-point noise: --trim-stage-weights on four GPUs chose layers 6/30/5/7 and read a 32K prompt at 1,750 tok/s, against 2,790 for 12/12/12/12 with the same decode speed and tokens (4x Radeon AI PRO R9700, IQ3_S). Among placements whose window time agrees to 1e-6 ms the smallest slowest stage is kept.
The engine's GEN line takes pin=N: the first N prompt tokens are a shared read-only prefix (a long document many
short questions follow). The prompt is read in two parts at N and the checkpoint there is pinned: retention never
evicts it, a parked conversation holding it stays parked, the session file keeps it, and a later request that resumes
from it does not park the branch it leaves, so N queries keep one prefix. One pin at a time. Without pin= nothing
changes.
The server takes "strata_prefix" in a chat, messages or responses request: {"messages": K}, {"message": I,
"chars": N} or {"tokens": N}, resolved to the longest common start of the prompt's ids. A prefix that cannot be
marked is said in the log and ignored; a malformed field is a 400.
Tests: conv_cache_test (pinned eviction), conversation_cache_test (a pinned parked conversation is not evicted),
conversation_file_test (the pinned checkpoint is the one saved), serve.test_server.SharedPrefix.
Speculative rejection sampling for the MTP drafts of a sampled request. The draft layer samples
its draft from the chain's distribution q (its own Philox stream, counter | 1<<61) and writes
q's (id, probability) list; the verify head computes the target's post-chain p per row from the
split sampler's block lists (penalties, top_k, top_p, min_p, temperature; at most 64 entries)
and accepts draft d with min(1, p(d)/q(d)), else samples the residual max(0, p - q) with its
own uniform. Row t writes the draft on an accept and the residual sample (never d) on a reject,
so the host's accept loop and the commit are unchanged. Rows past the MTP drafts (a lookup
chain's tail) and suffix windows are point masses: accept with p(d), residual p without d - no
q never means q = 0 (the llama.cpp #29924 class). Greedy requests never reach the new kernels;
the nine existing sampler kernels keep identical SASS.
- spec_prob.hpp: the rule, host reference; sample_tokens_spec / spec_draft_sample kernels
- STRATA_SPEC_PROB_GATE=pick, STRATA_SPEC_PROB_DT=x, STRATA_SPEC_MIN_TEMP=x: tunables, off
- spec_prob_test (CPU, 2M trials per case, chi-square 99.9%, controls that must fail) and
spec_verify_parity (GPU verifier vs host reference, token for token)
The per-request totals (drafts accepted / offered) hide where a window ends. With
STRATA_SPEC_DEPTH=1 the serve loop counts, for the windows drafted by the MTP layer, how many
times draft 0, 1, 2 ... was tested (every earlier draft was kept) and how many of those were
kept, and prints one stderr line per request. Off by default; nothing else changes.
Built on our RTX 5070 with nvcc 13.2.51 and 13.0.88 from the same source: with 13.2 the IQ3_S MMVQ product
(iq_multi_parity) and the IQ2_S / IQ3_S MMQ prompt path (prefill_fused_iq_test) are wrong, relative error 0.5 to 1.0; the
other formats are bitwise equal. setup now takes an installed 13.0 / 13.1 over 13.2 for sm_120 (STRATA_NVCC still wins) and
otherwise warns, saying what to install. Recommends, never refuses.
With the fused layout (STRATA_PF_FUSED=1, a chunk of 1024 tokens or more) layer 0 is treated as one the fused path does not
take, so a prompt must stop with the backstop's error instead of overflowing the shrunk buffers. The check that
the guard works at a 16,384-token chunk without needing a pack that really has such a layer.
Streams an answer, stops reading, resends or edits the prompt after 0-0.4 s (or with another request already running)
and checks every answer for real text, not a repeated token. Run the engine with STRATA_DBG_NAN=1 beside it.
--prefill auto:32768 lends 6,000 cache slots to the prompt path; on 31 GB of RAM only 3,700 of their experts stay in
RAM and the rest are read from the SSD on every chunk (prompts ~3x slower than --prefill auto). The engine now says
so at start, with the counts. A warning, never a cap: with the RAM for them the big chunk is 21-35% faster.
The LOW free-VRAM line at READY now also names the explicit --expert-cache (its slot count) and suggests a smaller
one or 'auto', which re-checks the free VRAM after the slots are written. Said, never changed.
gdn_step_commit_kernel reads n_keep from device memory. With STRATA_DBG_GDN=1 a count past the window's rows is
printed and the commit skipped, where a gfx1030 reporter sees a memory-aperture violation. A template parameter
keeps the default kernel unchanged.
fused_layout() and fused_ring() already keep the shrunk GU/H/Xq buffers and the per-layer expert path in step. If
they ever drift (a layer that takes MMQ or the FP16 path at a chunk of stream_all_min() tokens or more while the
buffers are the fused ones), the prompt now stops with 'the fused layout's MoE buffers are too small for layer N'
instead of writing past the buffer: an illegal access, or a kernel that never returns and shows up as
'prefill mmq: iota' one prompt later. Same check in the SYCL copy.
Co-authored-by: Krypto-Whitehat <[email protected]>
A deleted ROCm build named by HIP_PATH / HIP_DEVICE_LIB_PATH / LLVM_PATH, and a TEMP the comgr JIT cannot
create files in, both crash amdhip64 before the engine prints a line. Only a missing path and an unwritable
folder are touched; a healthy environment passes through unchanged.
uv's own managed Python loads pip wheels' libraries where Nix's Python cannot. Changed from the PR as the plan asked: uv is used only on NixOS or when STRATA_UV=1 is set (no silent toolchain change on other systems), and the setup.py hunk is left out. Not run on NixOS here. (PR #662)
"reasoning_close_retry": true in the config: when a reply ends on its end-of-turn token still inside <think>, with no answer and no tool call, the thinking is closed once and generation continues (the engine holds the prefix). Off by default; tested both ways and for a reply that already answered.
PleTable's Mmap+lock path now calls strata::platform::lock_resident (working-set minimum + VirtualLock, chunked per GiB) instead of refusing to start. A lock failure still falls back to touching the pages with a warning, same as the POSIX path.
Changed as the plan asked: only a full VirtualLock counts (a partial one is unlocked again and said so), a warning when the table is more than half of the RAM, the usage text indent fixed. Compiled on Windows here (MSVC, CUDA 13.0); not run with --ple-io ram yet.
(PR #813)
A reinstall put the model on the old drive again because setup keeps the last data folder in its settings file. The troubleshooting table now says where, and that --data-dir, --models-dir, --gguf-dir or deleting the data_dir entry change it.
An explicit budget bigger than the RAM the PC has free (or than its total less ~6 GiB) pushes experts to swap or the file tier; one report went from dead slow to fast by deleting the number. The server now says so at start and goes on (recommend, never force).
find_vcvars looked for versions [16.0,18.0) only, because CUDA 13.0-13.2 refuse a newer Visual Studio. The range now goes up to 19.0 when the toolkit found is 13.3 or newer (the reporter built with VS 18 and CUDA 13.3); older toolkits stay on 2019 and 2022.
Codex's compaction request has tools: [], so its prompt shared almost nothing with the cached conversation and was read again from the start. With "codex_compaction_cache": true the tools of that conversation's last prompt are kept (one entry, keyed by session and thread) and used for the compaction prompt only; the parser and the response keep the request's own tools, and a request that would not fit with them is rendered as sent. Changed from the PR as the plan asked: off by default, the mismatch documented, ported on top of the opt-in #923 and tested over HTTP with a tool call emitted in the compaction. (PR #924)
Codex sends a thread-title request beside each user turn (another session, no tools); running it on the engine replaces the one prefix cache. With "codex_thread_titles": true in the config it is answered at once with the user's line cut to 36 characters (usage 0). Changed from the PR as the plan asked: off by default (the title is not model-written), documented, tested both ways. (PR #923)
Rebased onto the 0.1.40.1 Vision (work_dir, #480 absolute paths). Added tests for the lazy encoder (nothing starts, restart starts it, a failed start is killed); the --lazy help no longer says text-only. A live smoke with a vision config and --lazy is still to run.
(PR #673)
Cut down to reporting only, as the plan asked: max_context is 0 until READY, and /v1/status, /v1/models, /props, /health, /slots and /metrics told a client the model had no context for the whole load. They now use the last known context (reported_ctx). Requests are still checked in prepare() as before.
Co-authored-by: updalla-apshir <[email protected]>
A reasoning-budget continuation (and the reasoning-loop recovery pass) is another native generation, not another API request: each pass keeps its own DONE, the prompt reuse is the first pass's (the generated reasoning is not this request's cached input), and times, tokens and draft counts (offloaded included) are summed. Rebased onto the 0.1.40.1 Service.run. (PR #615)
Cut down from the PR as the plan asked (recommend, never force): an empty --api-key or STRATA_API_KEY is already refused (#213) and a blank config key is refused with #956; an api_key that is present but empty now prints that the server has no key and starts. The setup-side refusal is left out (the final line already says no API key is set).
Co-authored-by: Guillaume PUTIER <[email protected]>
The 0.1.32 image encoder was configured without -DSTRATA_PORTABLE, so the old OFF default built it native
(MSVC + FindSIMD: /arch:AVX512 on the build machine) and it stopped with 0xC000001D on AVX2-only CPUs. A build
without flags is now the AVX2 baseline; the builds for the PC they run on (setup.py's two encoder builds, the
Dockerfile) pass -DSTRATA_PORTABLE=OFF and stay native.
- portable also forces GGML_AVX512*, GGML_AVX_VNNI and GGML_AMX_* OFF, so a -D cannot put AVX-512 back in
(with the old file, -DSTRATA_PORTABLE=ON -DGGML_AVX512=ON gave 10522 zmm instructions; now 0)
- native restores a fresh native configure's GGML_OPENMP / GGML_AVX* when a folder was portable before
- the configure log says which of the two it is
- test: build_engine's encoder build (gpu and cpu) passes -DSTRATA_PORTABLE=OFF
Rebased by hand onto the 0.1.40 setup.py (the Windows MSVC environment of #881 kept). The build script of the ready-made engine zips is not in this repository: it must pass -DSTRATA_PORTABLE=ON for the encoder (the default was OFF until now).
(PR #427)
Short forms with the required fields, a link to the troubleshooting table, and the pull request template. Taken without CODEOWNERS and without the template's checklist (the changes the plan asked for); blank issues stay allowed.
Prompt and output rates for the AMD RX 7900 XTX (gfx1100) beside the NVIDIA rows, with the measurement note. Only the DETAILS tables are taken: the PR's branch also deleted about 190 lines of current text, and the README tables are two fixed machines (RTX 5070 and RX 9070 XT).
tools/quantscope.py (from 1872183316/quantscope, MIT OR Apache-2.0) reads only the headers (a file, a folder, a URL, ms: or hf: repositories); tools/strata_inspect.py compares them with data/gguf_fingerprints.json, the files setup installs. Changed from the PR: the size menu is untouched (its bits per weight line would change the golden output; left out), and the untested verdict says that setup still refuses the GGUF names it lists as unsupported, so the two do not contradict each other. (PR #911)
Setup never switches hosts by itself: auto is Hugging Face, and a download that fails says --source modelscope exists (no probe at start). With it, each file is checked against the SHA-256 ModelScope publishes (a wrong one is deleted; a file with no published hash is warned about), the MTP tensors against the pinned checkpoint's own hashes, and setup says that ModelScope serves current files, not our pinned revision. Changed from the PR: the automatic switch and its 10 s probe are gone.
Co-authored-by: 1872183316 <[email protected]>
A new step 5 tries two tier settings (every window, 80 or 160 swaps, decay 0.97) against the engine's own, after the worker count, one restart each; a setting is kept only when it beats the default by 3%. The tier follows a text over some windows, so this step measures 512-token answers. The author measured +7.9% on a DDR3 Xeon with an RTX 4060 Ti. Applied by a 3-way merge onto the 0.1.40 calibrate.py; the check that it keeps the engine default on a fast-RAM PC is still to run on a GPU.
The generation card and board both run, the lanes now, and the copy rate that implies for a slow link. Changed from the PR: the fewer-lanes note is worded as a hint (a card or laptop can narrow the link while idle), not a verdict.
Co-authored-by: 1872183316 <[email protected]>
The model is the same base model and the same image encoder as UD-IQ4_XS; a user ran it by hand (#967, #971). Setup asks about images for it and warns that we have not run this file with images (no smoke test here: the 111 GB file is not on this PC). MCP's plan and size table follow.
Codex v0.160 sends an input item of type additional_tools, which was refused with a 400. The item is no message; a tools array in it joins the request's own tools, and one without a usable array adds nothing. The item's exact shape is not documented here: only the tools-array form is read.
Both tests upload and memset on the legacy default stream, then launch on
a cudaStreamNonBlocking stream, which does not wait for it. cudaMemset is
asynchronous to the host and a pageable cudaMemcpy may return before the
DMA lands, so the kernels can read buffers still being written. A device
sync before each launch closes it, the same fix 0.1.36 made in the fused
tests. Not reproduced on an RTX 5090: 0 failures in 210 runs of gr_parity
before and after.
Only the first commit of the PR (the stream syncs); the second one (a rewrite of ple_parity without the fixtures) is not taken. The same syncs for the HIP tests were not added (no AMD test machine). (PR #568)
With several GPUs in the config's "gpu" the server appended --layer-split auto, which the engine refuses beside
--peer-device (#531), so the peer tier could not be started through the server at all:
strata generate: --peer-device cannot be combined with --layer-split (use one or the other)
The split (and --split-skip-if-fits) is now added only when --peer-device is not among the args; the start line
names the peer tier instead of a layer split. Tests in serve/test_server.py (PeerDevice), docs/SECOND_GPU.md says
how to configure it.
Ported by hand onto the 0.1.40 engine_args (layer_split_value); the start banner now says the peer tier is in use when the split is not.
(PR #665)
--expert-cache N is N times the largest blob, compared only with the free VRAM
read before the slots are written; the check that runs once they are written is
auto-only. Under WDDM the free figure can be about a gigabyte high and an
over-committed allocation goes to system memory instead of failing, so only the
speed shows it. Measured at 1,048,576 on a 32 GB 4080 SUPER: 11,631 slots with
0 MiB free ran 13.7 tok/s, 11,178 slots (auto) with 217 MiB free ran 102-122,
and raising --vram-reserve-mib moves the free figure because it is deducted
before the cache is sized.
Also records that the GPU process Shared/Dedicated counters did not separate the
slow run from the fast ones on that machine, and adds a Troubleshooting row.
Read against generate.cpp: the second size check after the slots are written is auto-only (the loop breaks on !auto_cache), and the LOW line is printed at 256 MiB.
(PR #799)
The entry (serve/server.py with llama-swap's port, checkEndpoint /health, healthCheckTimeout), the DNS-rebinding
'Host ... is not allowed' trap and its three fixes, /upstream/<model>/ for Strata's own pages, and a note on the
Read against the 0.1.40 server: allowed_hosts and STRATA_ALLOWED_HOSTS, /health, the effort handling and the 403 text match.
(PR #829)
ggml's tiled CPU flash-attention kernel needs the head size (72 in this
encoder) to be a multiple of the vector width: 8 floats with AVX2, 16 with
AVX-512. With AVX-512, which a build from source gets on a CPU that has it,
ggml falls back to a kernel that is several times slower and accumulates in
FP16. The release builds are AVX2, where flash attention is the faster one,
so auto stays the default there.
1024x1024 picture, Ryzen 7 7700X, 8 threads, F16 mmproj: AVX2 8.3-9.9 s
with it, 14.0-15.2 s without; AVX-512 43.5 s with it (26% off the FP32
output), 13.3 s without. --gpu keeps auto, and an explicit --flash-attn
still wins.
Rebased onto the 0.1.40 strata_vision.cpp (the --min-tokens usage line). Only a source build on an AVX-512 CPU changes; the release builds (AVX2) do not. The Zen4 timing is not run here.
(PR #660)
- GGUF_DIR passes setup's --gguf-dir, so a container can use model files the
host already has instead of downloading them again (#252/#444 cover the
native setup only)
- RESIDENT_BUDGET_GIB and KV_STREAMING pass --resident-budget-gib and
--kv-streaming; unset keeps setup's own defaults
- INSTALL.md: a mounted GGUF folder must be writable on the first start
(the Unsloth SHA-256 result is kept as <shard>.done next to each shard)
- INSTALL.md: releasing tens of GB of page-locked RAM on SIGTERM takes longer
than Docker's 10 s default; without --stop-timeout the container is killed
(exit 137)
The INSTALL.md paragraph was added by hand (the PR's numbered list no longer exists); KV_STREAMING maps to setup's current --kv-streaming auto|on|off.
(PR #491)
The bare message cost a full debugging session: on Linux the failure was a
missing libcuda.so.1 (a host driver library the nix loader cannot see) in a
launch environment, not memory pressure. cudaGetErrorString names the real
cause, e.g. "CUDA driver version is insufficient for CUDA runtime version".
Ported by hand onto the 0.1.40 native_head.cpp (the PR's hunk context had moved).
(PR #476)
The PR guarded the mark of a whole shard; mark() itself now warns and goes on, so the SHA-256 result of a verified Unsloth shard and the shared-shard link on a --gguf-dir share do not stop setup either. The step is repeated on the next run.
A whole shard without its .done mark gets one written next to it. On a
read-only --gguf-dir (a network share) that write raised OSError and setup
stopped. It now warns that the mark cannot be written and goes on.
(PR #570)
A splitter may copy general.architecture into EVERY shard of a split while the
qwen4exp.* hyperparameters live only in shard 1 (seen with the Huihui
abliterated re-split of the UD-Q4_K_XL GGUF: general.architecture in shards
2-4, no qwen4exp.* there). NativeDense::load then ran check_architecture on
shard 2 and failed with 'missing qwen4exp.block_count'.
Run the full guard only when the shard is the first of the split (no split.no,
or split.no == 0); later shards take the existing split-metadata consistency
check whether or not they carry the architecture key. Verified end to end:
with this change the engine loads the re-split GGUF (previously: refused),
pack built with iq_pack.py --compat-bf16, coherent greedy output.
(PR #648)
read_expert_profile read the header's version field and then ignored it
((void) version), so a profile of a later format would have been read as a
version 1 layout: the fields after the version are only known to be
n_layers, n_expert, slots, n_ranked in a version 1 file. Refuse any other
version right after the magic, before the geometry check, with the version
found, the version this engine reads, and the file path in the message.
tests/core/expert_profile_version_test.cpp (registered like
expert_profile_save_test, CPU only): a version 1 file loads; the same bytes
with the version field at 2 or 0 are refused naming the version and the
file; a version 2 file of another geometry is refused by its version, not
its geometry; back at 1 it loads again.
(PR #632)
The inputs and the NaN sentinel go up with cudaMemcpy / cudaMemset on the legacy stream, and the kernels run on a
cudaStreamNonBlocking stream, which does not wait for it: a pageable copy or the memset can land after the kernels
started. On an RTX 4090 (sm_89, 128 SMs) that failed the screen ("unwritten or non-finite output") in 3 of 3 runs,
in a different product each time; with the sync before the kernels, 5 of 5 runs passed. The same fix as the fused
tests' (prefill_fused_moe_test.cpp, prefill_fused_iq_test.cpp).
(PR #532)
A request stopped on the solo path and continued in a slot is sent again as its prompt plus what it generated, and the engine holds all of that: the usage split counted the continuation. The first read's reuse is kept, and Anthropic's input_tokens is at least 1 (Claude Code took 0 as not given and counted the prompt twice: Prompt is too long at about half the context). Rebased onto the 0.1.40.1 server and the stop_sequence field.
strata_install refused every vision value but no with backend=hip, although setup's own --vision cpu reads images on the CPU beside the HIP engine. It now follows setup.hip_vision: cpu on Linux is planned, a GPU encoder (yes, gpu) and cpu on Windows get a message that says what works. The Windows AMD engine in MCP's hardware detection is left as it was (it still reports AMD cards on Linux only).
The configured key loses the spaces and line ends a header cannot carry; a key outside ASCII matches as UTF-8 and as Latin-1 (constant-time). Server side only: the web app part of the PR (the UTF-8 header and the key-rejected note) is left out.
reasoning_budget_tokens joins the shared Chat settings (0 means no budget, a whole number of tokens otherwise) and is read after the shared and config defaults are merged, as max_tokens is. Rebased onto the 0.1.40.1 Service.run (force argument).
req.get("defaults") is None both for an absent key and for an explicit null, and null means clients use their own settings again. A body the endpoint does not recognise cleared them all. {"defaults": null} still clears.
A context meter that divides n_prompt_tokens by n_ctx read 0 % against Strata. The key is the running request's prompt size, kept after it ends. Added to the PR: with --batch /slots lists every slot (the view /metrics has) instead of one entry.
Co-authored-by: Zerschranzer <[email protected]>
With "parallel": N the engine runs N requests at once, and /v1/status already says so. /props still said total_slots 1, so llama.cpp clients that read it refused or serialized concurrent runs. It now reports the same count, at least 1.
The caps (supports_tools, supports_tool_calls, supports_parallel_tool_calls, supports_system_role, supports_preserve_reasoning) are rendered once at template load. Changed from the PR: every probe error is swallowed (logged at debug) and turns that feature off, so a custom template that fails in an odd way cannot crash the start.
Co-authored-by: Ivan Tuzhilkin <[email protected]>
A POST behind a relay that sends Transfer-Encoding: chunked had no Content-Length and read as empty (400 No messages provided). _body(), the control-body read and the unread-body drain now decode it (it wins over a Content-Length, as RFC 9112 says). A chunked body is capped at 256 MiB (413; the announced size is checked before a chunk is read) and a malformed one is a 400. Content-Length requests are unchanged. Left out of the PR: nothing; added: the cap, the other body readers, tests.
Co-authored-by: Oscar Rojas <[email protected]>
The vision cache unlinked the oldest .sve on every insert past 64, including a file the same request had not read yet (FileNotFoundError). Vision.encode_all() protects the request's earlier files; the cache shrinks back on the next insert. A cache hit now counts as a recent use.
The solo and admission paths read the engine's ERR line, then sent STOP and drained for a DONE that never comes (it holds the control lines meanwhile). The ERR now ends the exchange.
Rows for #927 (2x RX 6900 XT), #917 (Radeon 8060S, gfx1151) and #955 (Arc Pro B65) in the new COMMUNITY.md index, with notes on what each
build was; the bullets I had added to docs/COMMUNITY_BENCHMARKS.md are removed so the index lives in one place. TRIMMED.md lists the
seven telemetry files left out of the 6900 XT folder.
The gfx1031 entry now lists the RX 6700 XT run (#1027) and the RX 6800M run on Windows (#915, #1078), and says the Windows zip
needs gfx1031 in the STRATA_HIP_ARCHS build list, which the default list has from 0.1.40.2 on.
NVIDIA_V100.md said sm_60 was not measured. It now points to the two-P40 layer split (#1028), the P100 alone and with a helper card
(#1069) and the V100 + P100 pair on 0.1.40 (#1079), each one reporter's run. OLDER_GPUS.md notes the extra host flag a GCC 12.3 +
nvcc build needed in #1074 (one report).
Three fresh engine launches, five fixed tasks at 512 and 7,000 prompt tokens, 640 output tokens: decode medians 40.4-41.3
and 39.8-40.9 tok/s, reference hashes recorded. The six local patches the report built with are kept as data; patches 3, 5 and 6
are ported as source changes in this release, the others were already in 0.1.40. Index line added to COMMUNITY_BENCHMARKS.md.
Co-authored-by: Tim Nevits <[email protected]>
Interactive Q2_0 and UD-IQ4_XS chats against llama.cpp Vulkan, then controlled A-E runs of the gfx1151 hipBLASLt tuning table
(250.9 to 528.9 prompt tok/s), dense MMQ (+0.2%) and the shared-expert decode stream (+2.5%, 8 requests). The build is
0.1.39 plus then-unmerged PRs (#895, #820, a tuning table adapted from #755), not the 0.1.40 release; the index line in
COMMUNITY_BENCHMARKS.md says so and notes that 0.1.40 turned the shared-expert stream off on HIP (#826).
One card, a layer split and the expert-helper mode, Strata 0.1.39 stock and with then-unmerged #835, #849 and #854, original
Flash-Next IQ3_S at 131,072 context; three runs at 4K, 32K and 128K prompt tokens and six recall checks per configuration.
Index line added to COMMUNITY_BENCHMARKS.md with a note that #835 is opt-in in 0.1.40, so the second set is not the
release default. The one-second telemetry.jsonl files (5 MB) are left out, and the README says so.
Co-authored-by: xjc10 <[email protected]>
The ready-made Windows HIP zip had no gfx1031 code, so an RX 6700 XT or 6800M user had to build from source with
STRATA_HIP_ARCHS=gfx1031 (#915, PR #1078; #1027 ran a 6700 XT as gfx1030). setup already knows gfx1031 (AMD_ARCHS, the
wheels index, the PCI id) and the arch gate builds it. This only adds it to the default list the zip is built from.
Prepared, not decided: building the release zip with this list makes it larger (one more arch of rocBLAS kernels and
engine code), and the owner decides whether the 0.1.40.2 zip carries it.
Two pieces of PR #895 that 0.1.40 did not have; the rest of it is in 0.1.40 (gfx1151 in the arch gate, sudot4, setup
and the Strix Halo defaults) or is not taken (the ROCm pin bump for every AMD arch).
- dq_q2_0: the compiler on gfx1151 (ROCm 7.13) folds the half path's negative scale times +0 to +0, as HIP 5.7 did on
gfx1012. The sign-preserving path now covers gfx1151 too. Changes bits on gfx1151 only (the same value the FP32 and CPU
paths give); nothing else is touched.
- Linux HIP, --expert-cache auto: an APU's "VRAM" is system RAM, and the device-free figure counts the whole GPU-addressable
pool, so a cache sized from it can ask the OOM killer for nearly all RAM. The PR capped the cache at the host's free
memory less 4 GiB; here it only prints a warning with both numbers and leaves the cache as asked
(recommend, never force). --expert-cache N sets it by hand.
Compile check only on HIP; the run needs a gfx1151 box.
Co-authored-by: mini <[email protected]>
A plain hipMemset runs on the null stream, which a non-blocking stream does not wait for; on a Radeon 8060S (gfx1151,
Windows) the memset landed after the MMQ kernel and left only the sentinel. hipMemsetAsync on the same stream orders it.
Taken from PR #919's Windows gfx1151 port; its other pieces are not taken here (the Windows carve-out accounting needs
a Windows gfx1151 run).
Co-authored-by: Stormace <[email protected]>
Merges two community reports of the same mechanism (GPU queues suspended while the kernel reclaims KFD userptr pages)
into one short section in AMD_HIP.md and one line in TROUBLESHOOTING.md: HSA_USERPTR_FOR_PAGED_MEM=0 (two R9700, ROCm
7.2) and GPU_PINNED_MIN_XFER_SIZE=1048576 with --mmap-experts (RX 6800), each labelled as one machine and not
confirmed, with the MemoryMax versus MemoryHigh note. Condensed from the two PRs.
Co-authored-by: zengyishou <[email protected]>
Co-authored-by: YamineRL <[email protected]>
setup listed the RX 7600 XT as "not supported" although the arch gate and the Windows zip already build it, and one
RX 7600 XT ran the engine (Swift IQ3_XXS, 146 requests up to 51K context, about 43 tok/s decode, 250-300 tok/s prompt,
unvalidated; docs/AMD_HIP.md has the report. The two setup tests that expected "not supported" now expect it listed.
- conversation_snapshot_test: the pinned host buffers are freed with sycl::free (they come from USM, not malloc).
- STRATA_VERIFY_NO_HOST with experts that are neither in VRAM nor mirrored now refuses to start, instead of a warning
followed by a generation with no safe host fallback.
- On QUIT the direct-access USM host mirror is released (gguf_src.close()) before the process leaves; a failure to
release is reported and exits 1.
From the Arc Pro B65 bench report (timnevits), where they are shipped as patch files; the other patches there were
already in 0.1.40. SYCL compile check only.
Co-authored-by: Tim Nevits <[email protected]>
Only subgroup row zero of the 32x8 block reaches the router's synchronization in route(); the other seven rows return
before it. A work-group barrier there is divergent, and on an Arc A770 the kernel hung. The surviving row now meets a
subgroup barrier. Also recorded in sycl/tools/fixups.py so a regenerated tree keeps it.
Salvaged from PR #797 (the rest of that draft is not taken: its build fixes were already in 0.1.40). An A770 on i915
reported decode only with this change (23.0 tok/s with MTP, 8.65 without). Needs an Arc owner to test on the current
tree; SYCL compile check only.
Co-authored-by: Txgom2 <[email protected]>
tune_hipblaslt on the gfx1201-100500 table's 32 shapes, on a Ryzen AI Max+ 395 (Radeon 8060S, gfx1151) with ROCm
10.2.0a20260930 (whl-next, hipBLASLt 1.5.0 efafc1c1). 3.9-15x faster than plain hipBLAS per shape; prefill with
STRATA_HIPBLASLT_VERBOSE=1: launches=3138 fallbacks=0.
Only the table is taken from PR #944. Its gfx11 WMMA attention and scorer duplicate the kernels already in 0.1.40, and
its "gfx11" prefix gate would enable gfx1103 and gfx1152, which 0.1.40 leaves out on purpose. To be re-measured on the
gfx1151 box.
Co-authored-by: Nathan Bradley <[email protected]>
On an Arc Pro B60 a system-scope atomic load of host USM is still served
from the GPU's cache, so a device spin never sees the host's store and
runs to kSpinMax every time. Read the flag through an annotated_ptr with
an uncached L1+L3 read_hint after an acquire fence instead. Bounded
ping-pong: 263 ms -> 2.4 us per round trip; per-layer waits on a B60
with experts in the host mirror: 8.7-10.5 -> 19.3 tok/s decode.
-DSTRATA_DOORBELL_ATOMIC_LOAD keeps the old load.
Based on konijiwa110's PR #742. Below sm_80 there is no TF32 tensor-core scorer, so the prompt path scores the QSA
blocks on the warp kernel: 5 shuffles per 4 products, ~1 TFLOPS on an RTX 2080 Ti, 24% of a 128K prompt. With
STRATA_SELECT_SIMT=1, qsa_block_scores_tc runs an FP32 tiled GEMM there (rows = query x indexer head, K = 128 through
shared memory, 2 queries x 4 heads x 4 blocks per thread, relu per head summed in registers), ~6.9 TFLOPS by the
author's measurement (128K prompt 1,016 to 1,242 tok/s, 248K 715 to 952 on one 2080 Ti).
The tiled kernel sums in another order than the warp kernel, so its scores are not bitwise equal. The PR made it the
default below sm_80; here it is opt-in, so default output is unchanged until an interleaved A/B of at least 10 pairs
supports a default. The tail block keeps the warp kernel's arithmetic. STRATA_SELECT_OLD=1 still forces the warp kernel.
qsa_select_bench is now also a CUDA test: at 32K, and at 131K with STRATA_QSA_WARP=1 and STRATA_SELECT_SIMT=1 so any
card runs the FP32 tiled scorer (FP64 gate, selections compared with the warp kernel's).
Co-authored-by: konijiwa110 <[email protected]>
On HIP the default attention kernel's score loop is bound by the LDS pipe on gfx1030: its 12 x 5 __shfl_xor_sync per
cell compile to ds_bpermute_b32 and share that pipe with the shared-memory q reads. attn_chunk_kernel_pre75 (PR #540,
bit for bit the default kernel) is now built for HIP wave32 (not the gfx906 backend) and run on gfx103x by default:
- NC_ template parameter, cells per step: CUDA 2 as before, HIP 8 (on gfx1030 every step up in NC was faster although
occupancy fell from NC 1: the q reads bound it, not occupancy)
- DPP_ template parameter: the butterfly's lane exchanges (reduce12, the softmax max / sum) through DPP row_xmask within
a row and v_permlanex16 across rows instead of ds_bpermute; same values, same pairing order, unchanged bit for bit
- pre75_attn() on HIP: gfx103x per device; STRATA_ATTN_PRE75=0 off, =1 also another wave32 card
CUDA is unchanged: the non-experimental build does not build the kernel, and the template defaults equal the old
constants with shfl_x<false> = __shfl_xor_sync.
RX 6900 XT, IQ3_S, prompt tok/s at 9.4K / 34.7K / 105.8K: 441 / 467 / 460 -> 459 / 494 / 488; with #835's FP16 GEMMs
746 / 917 / 928 -> 801 / 1,022 / 1,041. Teacher-forced log-probabilities against the default kernel: KL 0, argmax 100%.
bench, docs: the pre75 attention kernel on RX 6900 XT - measurements, exactness, decode A/B; OLDER_GPUS.md notes
bench/results/2026-10-04-rdna2-pre75-attention: the rig, the compiler output (registers, occupancy, ds_bpermute counts)
that motivated NC 8 and DPP, prompt speed on this branch (+4.2 / +5.7 / +6.1% at 9.4K / 34.7K / 105.8K tokens), the
step-by-step NC / DPP series on the FP16 prompt path, the teacher-forced exactness check (KL 0 over 1,578 positions),
a decode A/B (0.3-0.6 ms shorter windows, two rounds), the limits, and the configuration used.
docs/OLDER_GPUS.md: the RX 6800 / 6900 row says this kernel is the default on gfx1030 and names the switch; the A/B
switches line says what STRATA_ATTN_PRE75 does there.
Follow-up to #600 (fks's mma.m8n8k4 prompt attention for sm_70).
The kernel splits q (for q.k) and p (for p.v) into FP16 hi + lo halves
and runs both products into the same accumulator. Volta's tensor cores
don't keep full FP32 precision while they accumulate, so once the hi
products are in the accumulator most of the much smaller lo
contribution gets truncated away, and the split buys less than it
should. Giving the lo products their own chain and adding the two in
FP32 at the end fixes that.
qsa_prompt_attn_parity, one V100-SXM2-16GB, 2,048 queries, error vs the
FP64 reference (the FP32 kernel in brackets):
ctx 8,192 5.88e-6 -> 3.02e-6 (2.14e-6) 11.94 -> 12.20 ms
ctx 32,768 5.08e-6 -> 2.33e-6 (2.37e-6) 12.20 -> 12.38 ms
ctx 131,072 5.35e-6 -> 2.75e-6 (1.81e-6) 12.34 -> 12.51 ms
Registers unchanged (166 / 226 / 163 for modes 0 / 1 / 3), no spills.
docs/NVIDIA_V100.md updated for the new error level.
gfx1010 (Navi 10) is measured on an RX 5700 XT: 8 GB card, Coder IQ1_M, 32K
context, prompt 115 / 144 / 149 tok/s and decode 18.0 / 20.3 / 23.3 tok/s at 4K /
16K / 30K prompt tokens, 6 of 6 needle checks at 8K and 30K, 61 of 61 ctest on the
card. Nothing beyond the arch list is needed: the portable scalar dp4a that main
falls to measured 162.7 / 24.3 against 158.3 / 24.0 with the __gfx1012__ SDWA
control (#442) - inside the run-to-run spread - and 16 of 16 HIP tests pass on it.
gfx1011 (Navi 12) is untested on hardware: same RDNA1 ISA, builds with 0 errors on
ROCm 7.14's gfx101X-dgpu wheels.
Two box facts, for whoever picks this up: hipBLASLt 1.4.0 refuses all 26 of the
engine's GEMM shapes on gfx1010 (heuristic_count = 0), so there is no tuning table
and prompts run on plain hipBLAS; and in a PC that also has an AMD iGPU, ROCm's HSA
runtime segfaults in GpuAgent::InitDma while enumerating the pair - below HIP, so
HIP_VISIBLE_DEVICES cannot reach it - and the engine must start with
ROCR_VISIBLE_DEVICES naming the one KFD node (rocminfo for the index).
Co-authored-by: AcckiyGerman <[email protected]>
CMake caches CMAKE_HIP_FLAGS_<CONFIG> while it enables HIP, from the
values its platform files set once the compiler is identified. When
HIP cannot compile CMake's identification source, that configure
caches them empty and then stops; every later configure of the same
build folder keeps the cached values, so the build that finally works
compiles every HIP source without -O, i.e. at -O0, while C and C++ get
-O3 -DNDEBUG. setup configures build-hip every time, so one failed
attempt is enough.
That happened on an RX 6800 (Windows 11, HIP SDK 7.2, CMake 4.0.1):
the first attempt failed on Visual Studio 2026's <cmath>, the next one
passed with the include fix, and the engine (0.1.30 with the Windows
build of #247, --expert-cache 2048) decoded at 0.24-0.27 tok/s with
7.1-7.5 s per decode window. STRATA_VERIFY_PROFILE put the time in
the GPU's own stages (per window: q8+qkv GEMVs 1468 ms and out-proj
849 ms in the DeltaNet layers, the LM head 540 ms); the waits for flags
the CPU raises were 41 ms of its 8.4 s per window. With the flags back:
59-62 ms per window, 28.1-28.3 tok/s.
After enable_language(HIP), hip_backend.cmake gives each empty
per-configuration HIP flag the value CMake computed for the compiler
(CMAKE_HIP_FLAGS_<CONFIG>_INIT), which is what a fresh build folder
gets. Flags with a value, from CMake or the command line, are left
alone; one set empty on purpose is filled too.
Reproduced on Windows (HIP SDK 7.2, CMake 4.0.1) with a first
configure that failed on --rocm-path flags split at the space in
C:/Program Files, and on Linux (CMake 3.28, TheRock's ROCm 7.10 clang)
with one that failed on a --rocm-device-lib-path that does not exist.
After a working second configure in the same folder,
CMAKE_HIP_FLAGS_RELEASE stays empty and native_mmvq.cu compiles
without -O; with this change it gets -O3 -DNDEBUG. A fresh folder gets
-O3 -DNDEBUG either way.
setup.py's engine fingerprint now covers cmake/ as well, so a change
there alone (this one) recompiles a locally built engine on its next
start; that includes NVIDIA engines compiled here.
The batch slots' drafters share the draft-vocab head but never copied its ggml type; it stayed -1 and the first
admission of --batch-mtp exited with "unsupported native MMVQ GGML type".
cudaGetDriverEntryPointByVersion arrived in CUDA 12.5; the CUDA 12.2-12.4 engine build stopped in vmm.cpp. Same
fallback expert_cache.cpp already carries, and the sticky error of a failed lookup is cleared.
Co-authored-by: zhouxihong1 <[email protected]>
The STRATA_HIP_GFX906 build defines STRATA_HIP_GFX906, not STRATA_USE_HIP, so three
sites that gate on STRATA_USE_HIP still take the CUDA branch and fail to compile:
1. fused_gr.cu: a bare 'return;' in the bool fused_gr_read_multi -> 'return false;'.
2. mtp.cpp: the MTP stream-priority block (cudaDeviceGetStreamPriorityRange) is gated
on !defined(STRATA_HIP_GFX906) too.
3. vmm.cpp: the <cuda.h> include and the CUDA driver VMM code get the same gate.
All three are inside #if blocks that only the gfx906 build reaches.
vmm.hpp only includes <cstdint> and <vector>, so the unqualified size_t is not
declared on toolchains that do not expose ::size_t through those headers
(GCC 12.2, Debian 12). Qualify it as std::size_t, as expert_cache.hpp does.
Bench folder ported from PR #1078 by Hugua700, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #1062 by fioruccione, authorship kept.
Trimmed for 0.1.40.2: 7 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #1028 by sarge18, authorship kept.
Trimmed for 0.1.40.2: 8 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #1027 by GPF, authorship kept.
Trimmed for 0.1.40.2: 6 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #1016 by avarakin, authorship kept.
Trimmed for 0.1.40.2: 21 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #995 by lfontanez, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: bench/results/2026-09-29-speed-0126/README.md, bench/results/2026-09-29-speed-0126/matrix.json, docs/COMMUNITY_BENCHMARKS.md, tools/integrate_speed_0126_shin_blackmamba.py.
Bench folder ported from PR #832 by Zauberio, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #466 by gravitomagnetic, authorship kept.
Trimmed for 0.1.40.2: 1 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #993 by obiscr, authorship kept.
Trimmed for 0.1.40.2: 1 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #913 by 1872183316, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #902 by Mitsuasa513, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #882 by T-Crypt, authorship kept.
Trimmed for 0.1.40.2: 1 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #850 by christopherrobertbrooks-tech, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #834 by T-Crypt, authorship kept.
Trimmed for 0.1.40.2: 4 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #823 by christopherrobertbrooks-tech, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #815 by did-technomancer, authorship kept.
Trimmed for 0.1.40.2: 3 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #780 by 1314521gjy, authorship kept.
Trimmed for 0.1.40.2: 2 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #757 by 30crows, authorship kept.
Trimmed for 0.1.40.2: 4 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #745 by xyzzing, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #740 by jase100k, authorship kept.
Trimmed for 0.1.40.2: 3 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #721 by zimuhuan-code, authorship kept.
Trimmed for 0.1.40.2: 8 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #674 by pertain99, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #624 by Dmitry-B, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #618 by HarukiOtaku, authorship kept.
Trimmed for 0.1.40.2: 2 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #521 by lijieming, authorship kept.
Trimmed for 0.1.40.2: 2 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #483 by Efs-O, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #440 by Ambolio, authorship kept.
Trimmed for 0.1.40.2: 4 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #418 by qni-live, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #416 by pipeob0, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #404 by rwkeyes, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #811 by steve8697, authorship kept.
Trimmed for 0.1.40.2: 27 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #777 by steve8697, authorship kept.
Trimmed for 0.1.40.2: 21 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #758 by xxDoman, authorship kept.
Trimmed for 0.1.40.2: 21 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #698 by CYoung83, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #628 by Yasei-no-otoko, authorship kept.
Trimmed for 0.1.40.2: 30 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/AMD_HIP.md.
Bench folder ported from PR #508 by R6DJO, authorship kept.
Trimmed for 0.1.40.2: 40 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #433 by brenoperucchi, authorship kept.
Trimmed for 0.1.40.2: 2 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #417 by mlfather, authorship kept.
Trimmed for 0.1.40.2: 162 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #707 by noahark, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #499 by homeofe, authorship kept.
Trimmed for 0.1.40.2: 2 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #469 by homeofe, authorship kept.
Trimmed for 0.1.40.2: 4 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #389 by shrisha108, authorship kept.
Trimmed for 0.1.40.2: 1 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
The live smoke showed the same quoted-example problem in the answer: a model that wrote the call format in a
fenced example after </think> got a real tool call. The content parser now uses the fence / inline-code
tracking of the reasoning rescue: an opener inside a ``` or ~~~ fence (tracked across stream chunks) or inside
inline code stays text, the decision depending only on the text before it, so streamed and whole outputs agree.
Openers at the top level fire as before, including one after a sentence on the same line (none of the 37
corpus specimens needed that, but the old tests do and nothing argues against it). A closed call at the end of
an answer cut by max tokens stays a call.
Tests: quoted calls stay text and real ones fire at every chunk split; OpenAI (whole and streamed), Anthropic
and Responses over HTTP.
Found in the live smoke: an old request that learns its engine died while restart() has already taken the
process (self.proc is None) failed in death_note() with an AttributeError, and its client got no reply at
all. death_note(), alive() and _send() now read the process once and treat None as a dead engine.
Test: after close() the death note is a sentence, alive() is False and a send raises EngineDied.
Found in the live smoke of #1012: a long prompt being admitted when the engine is killed read its own
end-of-engine line (EngineDied), and its cleanup then drained the control lines for up to 300 s for an
end line that had been read already; the client got its 503 only after that. The cleanup now skips the
STOP and the drain when the engine process the request was sent to is gone (the server counts its engine
processes in gen, and a request notes the one it has the control lines of), and the drain polls once a
second and gives up as soon as that engine is gone.
Tests: a long read killed alone, and with short requests arriving meanwhile (it gives way and is admitted):
every request ends in a 503 within seconds, and the next request restarts the engine.
Versions are read as their first three numbers everywhere; the one place that was not (the version compiled
into an engine binary, `engine=0.1.40.1`) now accepts an optional fourth number. A test pins that a hotfix
tag keeps finding the engine zips: the checkout's v<project version> release first, the latest release second.
StrataEngine.restart() runs __init__ again, which made a new condition variable, control lock, wait counter
and wait list while requests were still waiting on the old ones: they hung for good, or failed with
`list.remove(x): x not in list`. The admission state is now made once and kept; restart() wakes the waiters
when it ends. A request that waits for the control lines now ends with a clean EngineDied (503, "not sent")
when the engine is gone and nobody is starting it, keeps waiting while a restart is under way, and goes on
with the new engine when it is up. The lock is checked again after it is taken, a slot that frees itself
after a restart keeps the lists of its own process, and the wait counters cannot go below zero.
Tests (fake batch engine behind the real StrataEngine and Service): waiters during a restart are served by
the new engine (2 and 4 slots, same condition variable and lock), get a clean 503 when the engine stays down,
and when the restart itself fails; the next request restarts and is answered; the solo engine is unchanged.
The #804 rescue turned any declared <tool_call> in the reasoning into a real call, so a call that was only
quoted (in a code fence, mid-sentence, in inline code, followed by more prose) ran, and so did one in a reply
cut by max tokens (47Hunter47's `rm -rf build` case). Now it must start a line (after optional whitespace),
be outside a code fence (``` or ~~~, tracked across chunks) and inline code, close with </tool_call>, and have
only whitespace or more calls after it before the end of the turn or </think>. Such calls wait until then,
and become reasoning text again when the turn ends by length, cancel or error. The decision is the same
streamed or whole. Declared calls beside an undeclared wrapper are still delivered, the wrapper stays
reasoning.
The server tells the parser how the turn ended, logs how many calls were read or kept as text, and counts
the read ones in the metrics totals (tool_calls_from_reasoning).
Tests: 37 recorded and constructed specimens (PR #525's corpus and 47Hunter47's 12 live strandings; 16 act,
21 do not) at chunk sizes 1, 2, 3, 7, random and whole; fence and opener split at every position; OpenAI
(whole and streamed), Anthropic and Responses over HTTP.
With "parallel": N, a batch window that fails prints its reason as an ERR
line on stdout and the engine exits with code 1. At that point every
request reads its own slot's lines, so nobody reads that ERR line, and the
server's note fell back to the log's last "strata" line (a stats line in
#997), or to "the usual cause is running out of RAM" without a log.
The pump now keeps an ERR line that was the engine's last stdout line, and
death_note gives it first. test_parallel's fake engine can fail its first
window over two slots: the note names the error.
(cherry picked from commit 9c5066303e)
dma_batch_parity.cu failed to compile on Linux/WSL (nvcc 13.0 with g++): 'template argument 2 is invalid' at the
std::array<T, cap> locals of copy_expert_blobs, whatever the form of the size (int, size_t, a literal). The engine
objects and the Windows build were not affected; the Linux tests build was.
vmm_test failed on the RTX 5070 / driver 610.47: the pageable host-to-device copy was still in flight when
a.unmap() pulled the chunks out from under it, an illegal memory access that stuck to every later call (the last
two checks, the 8-chunk free-VRAM gain and cudaGetLastError, failed as a result). It was not a VRAM leak:
released memory comes back at once (18 MiB gained for 9 chunks of 2 MiB, polled 10 s). generate.cpp already
synchronizes before it unmaps.
gfx1151 is now a supported AMD arch (TheRock gfx1151 index; HIP build on Linux, docs/STRIX_HALO.md). The check is exact (gfx_arch_is, as the
engine's gfx_arch.hpp): gfx1150 / gfx1152 / gfx1103 are named as Strix Point / Krackan / Phoenix-Hawk Point and stay unsupported. PCI 1002:1586
(one id for the 8060S / 8050S / 8040S) and the Windows adapter names map to gfx1151; the KFD simd_count names the 8060S / 8050S / 8040S.
Unified memory: the BIOS carve-out + the shared pool (GTT; Windows half the RAM) size the context; only the carve-out counts as extra
room in the low-RAM mode. A discrete card is picked before an APU. A Windows zip without gfx1151 code is refused with the reason.
UD-IQ4_XS is the recommended model from ~80 GB of memory; setup changes no host setting. tools/test_setup_strix_halo.py uses Aurora's real sysfs.
The default STRATA_HIP_ARCHS of tools/hip/build_windows.bat gains gfx1151. The ROCm 10.2.0a20260930 Windows SDK compiles the strata target for
gfx1151 (stream 140-j built it on this PC) and its rocm-sdk-device-gfx1151 wheel carries the rocBLAS kpack and the hipBLASLt kernels
package_windows.py copies, as for gfx1201. Unvalidated on a Windows Strix Halo.
Bitwise on the 5070 (identity gate: Q2_0 and IQ3_XXS identical 10/10 with it on against off, same exe), but the 10-pair
interleaved decode A/B (Q2_0, default config) shows no win: B/A medians 0.97 / 0.97 / 0.99 on the three greedy requests.
On the HIP build the fused kernel is off until check 6 passes there, so check 6 reports SKIPPED unless STRATA_NORM_ROPE=1 asks for it. (gfx1151 on Aurora, 0.1.40: with STRATA_NORM_ROPE=1 it does not pass - about 7% of the rotated elements differ - which is why the kernel stays off on HIP.)
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
On gfx1151 the shuffle reduction of the sum of squares inside the divergent-looking branch rounded differently for ~3% of the outputs than the single-token kernel (gdn_parity section 5, Aurora); the base kernel had the shuffles outside it, and with them back out the multi kernel is bitwise the single one for T=1..8 on gfx1151. No change for CUDA.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
Found by native_multi_parity on gfx1151 (Aurora): the float4 combine differed from the scalar kernel there, because the first pair's other product was contracted into the FMA. Written as fmaf per expert it is the pinned contract (first product rounded, FMA per expert in order) on every compiler.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
For a window of more than one token the drafter routes with one multi-column router GEMV and one multi-row top-10 and combines all rows in one launch (verify already did; each row is bitwise the single-token call, native_multi_parity). Needs the native router (512 experts, k = 10) and the native combine; otherwise the per-token calls stay.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
quantize_q8_0_scaled_kernel reduces the block's max with a warp shuffle (one warp per 32-value block) instead of one thread walking the block; fmax ignores NaN the same way in both. moe_grouped_s2 fuses SwiGLU with that quantize and writes the activation back to gate_up, so the grouped parity checks see the same buffer. STRATA_S2_SWIGLU_Q8=0 keeps the two launches; a static_assert pins FF % 32 == 0 for the fused kernel.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
STRATA_SH_FORK_LATE=0 forks at the top as before. The other two commit-4 changes of the PR are not taken: the shared-expert
Q8_1 reuse is already in f-140 as the opt-in STRATA_VERIFY_QDEDUP (and the fused q8_1 store), and the PLE native-key batch is
already in f-140 as the opt-in STRATA_PLE_BATCH=1; both would have changed output on the native packs by default.
Dropped on purpose: STRATA_VERIFY_PRELAUNCH and the dead copy_rows_from_mapped<ZERO_HITS>.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
MtpDrafter::draft now runs the round for T = a + 1: the catch-up writes the K/V of the accepted cells [0, a) and the full layer covers row a, instead of writing all T cells and rewriting cell a. No later attention reaches a cell past the one being drafted and the next round writes it first, so the draft tokens and probabilities are unchanged. draft_first (a = 0) no longer writes its repeated rows at all.
STRATA_MTP_CATCHUP_ALL=1 restores the whole-window catch-up (and the old staging sizes).
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
iq_kernels.cu: row_dot_80_sub16 (two rows per warp, all lanes active) for the n_embd == 2560 gate/up of every split format, exact-N instantiations of the multi-column MMVQ (launch_mmvq_multi_nc), the 3-entry tail in the grouped gate/up and down kernels. native_mmvq.cu: SmallTraits for Q4_0, Q5_0, Q8_0 and IQ4_NL decode the weight chunks and block scale once per (row, block) and apply them to every activation column with the arithmetic of small_q8_dot, and the multi-column launch runs two rows per block for any K.
Kill switches: STRATA_NO_SUB16_GU=1 (sub-warp gate/up/down), STRATA_OLD_IQ_MMVQ=1 (per-column kernels, existing), STRATA_NO_MMVQ_ROWS2=1 (one row per block at large K). mmvq_multi_parity, iq_multi_parity, native_grouped_parity and native_expert_parity pin the columns to the single-column results.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
gr_read_multi / gr_write_multi (and native_gr_rms_norm_weighted_multi, native_gr_pre_gated_multi, native_gr_post_multi under them) run T tokens in one launch per step; the verify window's PLE writes, last-layer write and final mixer, and the MTP drafter's write and final mixer use them. The BF16 MMVF multi kernel and the GR up / staged down kernels are instantiated per exact T (1..6, generic above), the BF16 weight tile is unpacked once for all tokens, with the same FMA order as before. STRATA_NO_MULTI_GR=1 restores the per-token calls and the generic kernels.
STRATA_GR_DOWN_MAX4 keeps its meaning from the previous commit (sm_120 default on, explicit setting decides); this commit does not change that default.
gr_multi_parity (new, memcmp): gr_read_multi (with and without injection) and gr_write_multi (in place and out of place) against the single-token calls at the real geometry, T=1..8.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
The 4-token accumulator variant of the fused GR down read keeps the same tile, block size, accumulation order and
variant as the kFusedGrMaxT one, so its bits are the same; #832 measured +2.8-3.5% decode with it on an RTX 5070.
Unset, the engine now turns it on for compute capability 12.0 only (decided once per device), and off on every
other card and on the HIP build, where it was not measured. STRATA_GR_DOWN_MAX4=0 or =1 still decides explicitly.
native_qsa_rms_norm_rope does norm<256> and the rope in one launch with the same FMA order as the pair (register barriers keep nvcc from contracting across the norm/rope boundary); rope_parity check 6 compares it with native_qsa_rms_norm_weighted + native_rope_apply by memcmp for head_dim 128/256, strided q|gate and contiguous inputs, unscaled and YaRN. The in-place rope also launches one 32-thread block per row.
Verify's norm_rope and q split, and the MTP drafter's q path (all T tokens at once; K for a one-token window) use it. native_norm_rope_usable() gates it: on the CUDA build it is on unless STRATA_NO_NORM_ROPE=1; on the HIP build it is off until rope_parity check 6 passes on AMD hardware, and STRATA_NORM_ROPE=1 turns it on there. The MTP q path's split, FWHT and gate-apply now run over all T tokens in one call each (row-independent).
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
The plan kernel grouped duplicate experts and numbered the groups with thread 0 walking all entries; every entry now finds its group in parallel and a packed (entries << 16 | groups) warp prefix scan numbers them. The residency check, the empty plan and *plan_err of #871 are kept as they are. static_asserts pin the 4-warp / packing limits.
resident_plan_parity (new, memcmp): n=1..80 entries over pools of 3..512 experts with and without a slot table, with and without a skip word, plus the non-resident case, against a host replay of the plan's contract.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
kv_append_q8_steps / kv_append_q4_steps append n cells in one 3D-grid launch per pool, and native_qsa_indexer_append_steps walks the n indexer cells in order inside one CTA (one barrier between cells). Verify record_window, the commit graph and MtpDrafter::record_forward use them; a multi-slot batch keeps the per-token loop (each row has its own K/V). STRATA_NO_BATCH_KV_STEP=1 restores the per-token calls.
The kv single-cell calls are now the n=1 case of the same kernel. kv_steps_parity (new, memcmp) covers INT8, Q4_0 and the hybrid's folded single-plane calls for n=1..8; qsa_parity gains append_steps against the sequential indexer append at strides 1 and 5 under none/linear/YaRN.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
native_router_top10_multi runs one warp per token (8 per CTA) instead of one CTA per token; native_moe_combine(_multi) takes a float4 kernel for k=10 with a shared row on aligned buffers; the shared expert's sigmoid-scale is vectorised. Same arithmetic order, so same bits.
New native_multi_parity (memcmp): the multi router against the single call for n=1..19 on random, tied and NaN logits, and the combine against a host replay of its contract for the float4 and scalar paths.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
gdn_ab_multi is instantiated per T=1..6 (generic above) with the weight unpack hoisted out of the token loop; gdn_step_norm_multi prefetches the next token's q/k while the current one runs, and when no state is written and every token is an output (the verify half) it skips the per-token bookkeeping. gdn_step_norm_kernel splits its reduction buffer so one barrier goes.
gdn_parity gains section 5: for every T = 1..8, gdn_ab_multi against fused_gdn_ab, and gdn_step_norm_multi (verify half, commit half, t_out_begin, state-only commit) against the single-token kernel chained, compared with memcmp.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
A CTA of the fixed 256-CTA grid whose first block is past the batch's largest n_bid returns before staging the queries. Same scores, same bits.
STRATA_QSA_EARLY_EXIT=0 restores the old staging. kv_stream resolve_kernel: only threads that looked something up add to the shared counter.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
On the RTX 5070 (Windows, 4 KB pages, the whole-arena registration refused) the deferred per-layer registration made
greedy decode 4-10% slower than registering the sliced arena after the load (0.1.39's order), with identical output
(Q2_0: STRATA_DEFERRED_REGISTER=0 gave +10% on the same build; 12-pair A/B against 0.1.39 flat after).
#826 turned the fork off on HIP (it costs discrete RDNA cards). On Strix Halo's unified memory it pays, measured on the
merged 0.1.40 code (stream 140-m). The switch is read on first use, after the arch table ran.
(b & 0x08080808) * 0x1E reaches 0xF0F0F0F0 and overflowed a signed int; on sm_60 (strata_dp4a
fallback) the compiler turned that into garbage: native_expert_parity q4_0/q4_0 and q4_0/q4_1
gpu rel 1.7e4 and 6.8e2. Unsigned wrap is defined and gives the same bits.
Claude Code starts its system prompt with
`x-anthropic-billing-header: cc_version=2.1.170.bf4; cc_entrypoint=sdk-cli; cch=b145e;`.
cch changes on every request and the version's 4th part on every session. The
template renders the tool list first, so every turn's prompt differed ~22K
tokens in and only the last 16K checkpoint was reused. Both stamps are pinned to
f's, as llama.cpp does (ggml-org/llama.cpp#21793); only a header at the very
start of the system text is touched.
Port of PR #931 onto the #554 vision-tag marks: literal_tags() adds the tokenizer's
control tokens to LITERAL_TAGS, longest first; a prompt with none of them is
rendered and encoded exactly as before.
Co-authored-by: Chanyeong Lim <[email protected]>
Step 5 and step 7 decided these two settings inline. A front-end that has to say what a model would change needs the
same answer, so the rules are named and setup calls them itself: low_ram_wanted is the old `budget is None and (...)`
(budget is set only for a size that has a RAM budget), and the streaming branches keep their messages and say they
are kv_streaming_wanted.
Behaviour unchanged, and checked: test_setup_risk's new RulesAgreeWithTheWrittenConfig runs 54 mocked installs over
three sizes, two contexts, three --kv and three --kv-streaming choices, and 24 more over two PCs and four --low-ram
choices, and compares each written config's --kv-resident / --resident-experts / --mmap-experts with the function.
Co-Authored-By: Strata qwen3.8-flash-next-iq3_s <strata@localhost>
AI-developed: written by an agent (pi) running on Strata's own local model, qwen3.8-flash-next-iq3_s; tested by hand on an RTX 5060 Ti 16 GB / Ryzen 7 5700X3D / 64 GB PC with engine 0.1.39
An existing pack experts.bin (low-RAM copy or the AVX-512 Q2_0 conversion) and an
existing MTP draft layer need no new room. A re-run of setup on an installed model
used to demand ~arena_gb + 9 GB free and fail on a nearly full models volume.
The RAM line was [ok] and the verdict 'This PC can run Strata' although every size
said 'does not fit'. Now the line is [!], the verdict says no size fits, and the exit
code is 1 for scripts. Check only: --model NAME --yes still installs one anyway.
MIN_ENGINE >= RESIDENT_SPLIT_ENGINE now turns on the resident low-RAM mode on a layer split: the golden
baseline records the new 2-GPU configs (both cards, --resident-experts, --remote-expert-opt), and the
tests of the pre-0.1.40 behaviour pin MIN_ENGINE to 0.1.39. CMakeLists' project version is what
source_version() reads, so a compiled engine must say 0.1.40 to satisfy MIN_ENGINE.
- bench_tips(): a warning for a fixed --expert-cache on Windows (#780#781), --prefill auto:32768 only as a tip
with 96+ GB RAM and a warning below 64 GB, STRATA_RESIDENT_HEADROOM_GIB=6 on resident configs with <= 48 GB,
--conversation-cache-mib 8192 for agent clients when 24+ GB stay free, --vision cpu on <= 12 GB cards.
- a 2-socket tip for --pool-workers (one socket's cores - 1).
- calibrate also tries a quarter of the default, P-cores - 1 and one socket's cores - 1.
No default, config or engine argument changes; --mtp-window left out on purpose.
--host-core: the host thread's core, first (the default) or last, the workers on the others.
Windows sends a GPU's interrupts to one logical processor, usually the first, and every copy that lands raises one
(~13 us in the ISR and a DPC): a host that spins on the GPU's flags there waits for them (eddoursul's fork, F12:
the host saw a layer's ring up to 30-110 us late, p99). --host-core last (or STRATA_HOST_CORE=last) puts the host
on the last physical core and lets the workers have the first. The default is the layout the pool has always used;
a hybrid CPU keeps it. The startup log names the processors the pool workers and the host thread run on. The
placement changes no result, only where threads run. The fork's other part (a per-second re-pinning from the
processors' interrupt time) is not ported.
Ported from eddoursul's fork, commit 5137b33 (MIT).
Co-authored-by: Eddoursul <[email protected]>
Builds on midhatn's #807 (cudaMemcpyBatchAsync for the verifier's DMA group). dma_batch.hpp now holds one
helper, copy_blobs, for independent host-to-device copies on a stream, which the verifier's PCIe share
(copy_expert_blobs) and the adaptive tier's refills (both loops of generate.cpp: --serve's main device, the CLI's)
use. Changes against the pull request, from the review: mode 0 is the old loop untouched (it never stops the
engine: the PR aborted on any copy error, which 0.1.39 ignored); the Windows host alias (cudaHostGetDevicePointer)
is looked up once per blob; a batch the runtime refuses is redone as the loop with one warning; HIP, CUDA before 13
and groups over 128 keep the loop; the batch order is the stream's. eddoursul's fork batches the same copies
(F12: under WDDM each copy raises an interrupt where it lands, on the processor the GPU's interrupts go to).
STRATA_DMA_BATCH is 0 (off) unless set; 1 batches, 2 adds CUDA 13.4's overlap hint.
Co-authored-by: midhatn <[email protected]>
Co-authored-by: Eddoursul <[email protected]>
Cherry-picked unchanged from the pull request, for F12; the follow-up commit turns its helper into the single
copy_blobs helper and fixes what the review found.
The final mixer in one fused read.
The hyper-connection read before the head ran gr_read's kernels once per token of the window (lm_head_mix), and once
per draft pass in the draft layer. Both now go through fused_gr_read_multi, the layers' read (no injection weight:
the final mixer), the window's tokens in one launch. Its sums are gr_read's (fused_gr_read_multi is bitwise
fused_gr_read, which gr_parity checks against the plain read). CUDA only; HIP keeps gr_read per token
(STRATA_HEAD_MIX_MULTI=0 does too). This tree's opt-in STRATA_FUSE_HEAD_GR path is unchanged.
Ported from eddoursul's fork, commit 2ec7a80 (MIT).
Co-authored-by: Eddoursul <[email protected]>
The draft pass's kernels in the main layer's forms.
- quantize_q8_0_scaled runs a warp a 32-value block (it ran a thread a block: 80 threads for the draft layer's
input): the block's |max| over the same values, then each value's bytes as before. CUDA only.
- The draft graphs' inputs from mapped memory in one copy kernel (copy_from_mapped_multi): 7 launches in the round
graph and 2 in a step become one each (CUDA; HIP keeps its separate copies).
- The shared expert on a branch of the draft graphs beside the router and the routed experts, joined before the
combine (CUDA; HIP keeps one stream; STRATA_MTP_SHARED_BRANCH=0 turns it off).
The fork's verify_router in the draft pass and its native-format draft experts are not part of this commit (they
belong to F1 and F15). Ported from eddoursul's fork, commit d3d48b1 (MIT).
Co-authored-by: Eddoursul <[email protected]>
A one-token window commits itself.
A one-token window always keeps its token, so its graph now advances the sequence state itself: the GDN step kernel
writes the state (n_keep = 1, the arithmetic the commit's replay used), the conv kernel writes the history it read
(gdn_conv_l2_multi's commit flag), and its indexer append and PLE history were final already. Verifier::commit
launches no commit graph after it: the draft round graph then runs alone instead of beside the commit's kernels,
which read and write each GDN layer's 3 MB state.
CUDA only (HIP keeps the commit graph after every window; STRATA_ONE_TOKEN_COMMIT=0 does too); batch windows keep
theirs. Ported from eddoursul's fork, commit 235784c (MIT).
Co-authored-by: Eddoursul <[email protected]>
The draft round graph runs the draft layer's front once.
MtpDrafter::capture_round ran the draft layer's front (the embedding, the fc projections, the attention
hyper-connection read, the K/V projections and appends) over the window's T cells, then the whole layer again for
row a, repeating row a's front. record_forward is now record_front (T rows) and record_rest (one row: the attention
from the q projection, the MLP, the final mixer, the head); the round graph runs the front once, copies row a's
residual, injection and mixed input to row 0 (copy_row_to_first, the row read on the device) and runs the rest
there. The multi-row front is per row the one-row front's arithmetic (checked by the window hashes on fixed text).
The coupled-sampling and rotated-KV paths of this tree stay in the same functions.
Ported from eddoursul's fork, commit 22b46b6 (MIT).
Co-authored-by: Eddoursul <[email protected]>
The verify head's argmax and the draft's top probability over many blocks a row.
The draft head's top probability ran row_top_prob, one 1024-thread block a row over the 40,525-token draft head
(and the head's greedy pick, sampler_greedy_kernel, one block a row on sm_80 to sm_89). argmax_rows scans a row
in up to 128 blocks (4,096 logits each), the row's last block merging their picks (the larger value, on equality
the lower index: the one-block kernel's order); row_top_prob_split computes row_top_prob's 32 warps' sums in 8
blocks, the row's last block adding them in order. Both keep their counters and partials in per-row slots of one
scratch. Bitwise: the new verify_parity test checks them against sample_tokens' greedy pick and row_top_prob (ties,
NaN, -inf, 1-8 rows on one scratch).
CUDA only: argmax_rows is used on sm_80 to sm_89 (sm_90+ keeps sample_tokens' cluster kernel); row_top_prob_split on
every CUDA card; HIP and the cards below sm_80 keep the old kernels (STRATA_MULTI_BLOCK_ARGMAX=0 turns both off).
Ported from eddoursul's fork, commit fbb8bfa (MIT).
Co-authored-by: Eddoursul <[email protected]>
--spec-follow PATH makes a generate run emit a given continuation (token ids)
instead of the window's argmax, and accept the drafts that match it, so runs
with different speculation settings (or two builds) process the same text.
--window-hashes PATH writes one line per verify window: its index, position,
size, an FNV-1a hash of its final residual rows and its argmaxes; two builds
doing the same arithmetic write the same file. Both are off by default and
change nothing when unused.
Co-authored-by: Eddoursul <[email protected]>
A verify window takes the previous adapt round's swaps once they are STRATA_ADAPT_LAG windows old. The default is 1,
exactly 0.1.39; lag 2 is opt-in. pending_age resets after every pending.clear(). The batch paths, --pipeline-windows
and the peer tier are not lagged.
Co-authored-by: Benjamin Larsson <[email protected]>
- carve: with owned buffers the streamed ring is one cudaMalloc instead of up to 1,024 pieces each rounded to 2 MiB
(~0.5 GiB). A borrowed region keeps its per-slot layout and price, so the loans' slot counts do not move.
- Prefill::bytes_needed_owned prices the real allocation (2 MiB pages, the ring in one piece). The cache sizing uses it
only with STRATA_OWNED_PRICE=exact; the default is still 0.1.39's 160 + chunk * 680 / 1024 rule.
- the "device buffers do not fit" error names the free VRAM, the bytes taken and the buffer that failed.
No refusal was added: parts B (borrow_residency) and C (the advisory planner) stay out.
Co-authored-by: Jakub Luwierski <[email protected]>
Items 1-2 of #863 (gathered decode per core, AVX-VNNI rows) are bitwise and stay. Item 3 changes a lone token's IQ3_S
rounding on Intel hybrid CPUs, so it needs the switch. The #706 bit-plane path stays in front of the VNNI/plain rows.
#152's rule gives a lone token's gate/up rows to ggml's dot, because the multi-token kernels were no faster at one
token. With the gathered decode that no longer holds for IQ3_S on the Intel cores that gather it: an expert's gate/up
rows for one token on an i9-14900KF P-core take 0.34 ms with ggml's dot and 0.19 ms with the AVX-2 kernel. And in
decode most CPU experts serve one token of the window.
So on a CPU without AVX-512 where cpu_gather_fast() holds (Intel, Alder Lake or later, see the gather commit) and
STRATA_IQ256_GATHER is not 0, IQ3_S takes the multi-token kernel from one token on. This only pays together with the
gathered decode: on the same CPU's E-cores, which keep the scalar decode, it is 0.71 -> 0.68 ms, about even. That is
why it follows the gather's machine-wide rule and not the per-thread one. Measured in the engine with Swift 1.5
IQ3_XXS (whose IQ3_S layers this reaches): the CPU expert time per verify window -4%.
This changes results. A lone token's IQ3_S gate/up rows now round like the multi-token kernel's (the order of the
float additions), no longer like ggml's dot. They are the same rows the token gets in a group, so on these CPUs IQ3_S
experts no longer depend on how many drafts a window held (#152), and every core rounds them alike. Down rows and
the other formats keep the default rule. STRATA_IQ_MT_MIN set (2 is the old default) keeps its rule for every format.
The decision is native_gu_mt_min(): every probe is in expert_layout.cpp and cpu_avx2_ok() comes first. Nothing
compiled for AVX2 runs before that check, including on an older-CPU build (STRATA_ISA_FLOOR) on an AVX-only CPU.
iq_avx2_parity checks native_gu_rows wherever native_gu_mt_min is 1: each token's rows must match alone and in groups
of 2..8. docs/DETAILS.md notes the new default next to STRATA_IQ_MT_MIN.
Alder Lake, Sapphire Rapids and later have AVX-VNNI on every core: vpdpwssd does the madd and the add of an i-quant
row in one uop, and vpdpbusd the maddubs + madd(ones) of the IQ4_NL and Q2_0 down rows. The sums are integer, so the
results are bit for bit those of the AVX2 forms: vpdpwssd wraps like madd + add, and the maddubs it replaces in the
down rows cannot saturate (IQ4_NL |w| <= 127, Q2_0 codes 0..3).
The row kernels move to iq_avx2_rows.inl and q2_avx2_rows.inl, which iq_avx2.cpp and q2_avx2.cpp include twice: once
as they were (namespace plain) and once with AVX-VNNI (namespace vnni). In the second copy every function is declared
with target("avxvnni") under GCC/Clang, so the intrinsics compile there and nowhere else: neither file gets
-mavxvnni, which would let the compiler use the extension in the plain path. MSVC takes the intrinsics under
/arch:AVX2. CMake checks that the compiler has them (GCC 11, Clang 12) and otherwise builds without the copies
(STRATA_AVXVNNI=0).
cpu_avxvnni_ok() in expert_layout.cpp, which is not compiled for AVX2: cpu_avx2_ok() and CPUID 7.1:EAX[4], off under
STRATA_FORCE_ISA (a CPU that stops at AVX2), off in a STRATA_ISA_FLOOR build, and STRATA_NO_AVXVNNI=1 turns it off.
The VNNI copies are reached only through the AVX2 kernels' entry points, which the engine already calls only behind
cpu_avx2_ok().
Measured on an i9-14900KF on top of the gathered decode: another 4-10% on a P-core's i-quant gate/up rows, 15-25% on
an E-core's Q2_0 down rows.
iq_avx2_parity runs the VNNI variants as well (iq4nl256_down_rows_v, q2_0_gguf_rows_multi_avx2_v): every combination
bit for bit against the scalar one-token rows, 1..8 tokens.
0736f05 reads the IQ3 grid words with one gather under STRATA_IQ256_GATHER=1, on every core, and keeps it off by
default because AVX2 gather throughput depends on the core. This makes the gathered decode its own Fmt32 (118, 121,
122), chosen once per call instead of a branch in every decode, and takes it further:
- IQ3_XXS: the four even-sign masks by one 64-bit gather as well;
- IQ3_S: the 9th index bit by one sllv and one and; the sign bytes by one broadcast and one shuffle;
- IQ2_S: the four 10-bit grid indices built in a register and read by one 64-bit gather, the signs as IQ3_S's.
Each is its format's scalar decode bit for bit: the same words land in the same lanes. IQ2_XXS measured 4% slower
gathered and IQ2_XS has its own sign decode, so both keep the scalar one.
STRATA_IQ256_GATHER: unset = auto, 0 = the scalar decode everywhere (as before), 1 = the gathered one everywhere.
Auto gathers on a thread whose core is an Intel performance core of Alder Lake / Sapphire Rapids or newer:
- GenuineIntel with AVX-VNNI (CPUID 7.1:EAX[4]) as the generation marker. That leaves out Haswell/Broadwell (slow
gathers) and Skylake through Ice Lake / Tiger Lake (gathers slowed by the Downfall/GDS microcode);
- not the E-core-only parts, which are not hybrid: Alder Lake-N / Twin Lake, Grand Ridge, Sierra Forest and
Clearwater Forest, by model;
- on a hybrid CPU only a thread on a P-core (CPUID 1Ah core type 40h): the E-cores gather slower than they assemble.
AMD keeps the scalar decode (Zen 2/3 gather slowly; Zen 4/5 take the AVX-512 kernels). STRATA_FORCE_ISA turns auto
off. The probes are in expert_layout.cpp, which is not compiled for AVX2, and the core type is asked once per thread
(the pool pins its workers).
Measured on an i9-14900KF, gate/up rows of one expert at three tokens on one P-core, scalar -> gathered: IQ3_S 0.40
-> 0.27 ms, IQ2_S 0.31 -> 0.27, IQ3_XXS 0.26 -> 0.24. On its E-cores the gathered decode was slower.
iq_avx2.cpp has no runtime initializer again (d072203): the switch and STRATA_IQ_PREFETCH are read on first use.
iq_avx2_parity, a CPU-only CTest: random blocks (every grid index and sign pattern occurs) for IQ2_XXS, IQ2_XS, IQ2_S,
IQ3_XXS, IQ3_S and IQ4_XS gate/up rows and IQ4_NL / Q2_0 down rows; every variant, 1..8 tokens, each token's rows bit
for bit the scalar variant's rows for that token alone; the scalar rows against ggml-cpu's vec_dot (Q2_0: a double
reference). --bench: ms per expert on one thread, each variant, ggml's per-token dot and the engine's dispatch, over
more expert blobs than an L3 holds.
The PLE Q8_0 reader duplicates #651 and is dropped here. The grouped Q6_K instance is built only on request
because every instance is loaded at engine start (EAGER) and costs slots for every other model.
Co-authored-by: Thomas Weaver <[email protected]>
Gate/up is Q6_K on 47 of 48 layers and Q8_0 on layer 2; down is Q8_0 on
every layer. The page had those roles swapped. native_expert_parity
q6_K/q8_0 is now a normal test: the old one expected the pair to be
refused, and native_expert_supported accepts it.
The measured server runs are on this machine (RTX PRO 4500 Blackwell,
engine 0.1.39 after the rebase, and one cold prefill on 0.1.38). Quality
against llama.cpp is still unmeasured, and the three Q6_K parity
programs have not been run on a GPU.
PleTable accepts per_layer_token_embd.weight as Q8_0 (170-B rows): the
UD-Q6_K_XL file ships its PLE table in Q8_0, not IQ4_NL. The row reader
is already parameterized by bytes-per-row; only the type check, a
dequantize_q8_0 block loop (5 x 34 B) and the PLE_ROW_BYTES_MAX bump
(160 -> 170) were new. No GPU kernel changes: gather outputs floats.
- ple_reader_test --selftest also runs 170-B rows (90/110 were already run)
- ple_q8_parity: real-artifact Q8_0 PLE rows against ggml's reference
dequantizer, through Mmap and Direct (bit-exact on the synthetic
fixture from tools/ple_q8_fixture.py)
- docs/UNSLOTH_Q6.md: the manual workflow, verified shard inventory
(169.2 GB, 6 shards, table alone in shard 3), pending real-run numbers
A four-GPU run picked its split from a formula that reads only `layer_ms`
(SMs x clock) and shares the layers in proportion to speed; the free VRAM
that decides how many experts a stage can actually hold never entered it,
so the search could hand the most layers to the card with the least room
and never notice. All C(L-3,3) placements are now scored - 15,180 at 48
layers - and 2,068 of them cost nothing extra, because a stage's fill
depends only on its own layer range and its own free VRAM: `predict`
already hands each ranked pair to the one stage that owns its layer and
stops each stage at its first pair that does not fit, so a range is walked
once and memoised.
The cost model's mass curve changes with it. `mass[r]` was (r+1)^-1.2,
fitted to the 5080+3090 sweep - still right on that rig (99.35% predicted
at 20,000 held, 99.4% measured) and wrong on IQ3_S, where it claims the
top 5,805 pairs hold 94.9% of the routed mass and the rig measures 69.2%
over four boots and 14.9M decode lookups. No exponent reconciles them,
because the curve is a property of the model's routing and the profile
carries only a ranking, with no counts. The mass is now the increment of
H(f) = 1 - (1 - f)^b over the held fraction, whose prefixes sum to exactly
H(N/total); b = 3 (STRATA_SPLIT_COVER_B overrides it) is chosen to leave
the validated regime where it was - at f = 0.8138 it gives the old fit's
99.35% - while at f = 0.59 it gives 93.2% against 92.2% measured.
This is the placement search from #584, carried on v0.1.39 and without the
weight carve it rode in on: v0.1.39 ships its own carve (#559 / #639), and
the search only runs under `--layer-split auto`, where nothing is carved -
so `cap[i]` is already net of the dense weights and a stage's room is what
`predict` already subtracts, its session.
Measured on the 4-way IQ3_S rig (2x RTX 3060 + 2x RTX 5060), same config and
same flags in both arms, the binary the only difference: the four-way split
moves from K=10,19,34 to K=14,31,41, the prompt chunk the starved cards allow
grows 768 -> 1280 tokens, prefill goes 425 -> 597 tok/s (+40%) and decode
44 -> 51 tok/s (+16%), with the decode expert-cache hit 74.7% -> 83.3%. Two
boots per arm, and the split is deterministic. The curve and the search are
one commit, so that measures the pair, not the search alone.
The duplicate block_topk_stream_kernel is dropped: 0.1.39's block_topk_wide_kernel gives the same ids.
Only the 90-135K Turing band is new (counted bound, sm_75). Parity cases from the PR are kept.
A segment reads in ceil(tokens / max_chunk) chunks of equal size, kept only where the last chunk would stream
every expert anyway. Used only at the segment call sites; request_chunk is unchanged.
Co-authored-by: architectds <[email protected]>
Down rows are only 10 blocks, so the per-row reduction was a large share; four rows now share one hadd tree. The single-row reduction uses the same tree, so a row's bits do not depend on how rows are split into tasks. Down rows +6-7% per core.
The activation is pre-permuted once per token into the 2-bit codes' plane order (ActQ::qp/psum/pscale), so rows need no unpack interleave: four planes summed in int16, one madd, the -1 code offset subtracted exactly in the integers, one FMA per two blocks per token. 2.3-2.8x faster per core than the b256 kernel, closer to exact (rms rel 1.0e-7 vs 2.7e-7), width-invariant. STRATA_Q2_LEGACY=1 selects the previous kernel.
A conversation the engine holds (what a parked snapshot holds) goes to one file and comes back, also after a
restart, so a long prompt is not read again:
- engine: `SAVE <path>` / `RESTORE <path>` between requests, answered `SAVED|RESTORED <tokens> <bytes> <ms>` or
`ERR <reason>`; single GPU only (refused with --layer-split) and needs --prompt-cache > 0.
- format v1 (src/core/conversation_file.cpp): 64-byte header bound to a model fingerprint (name, size, first and
last MiB of each weight file) and a config fingerprint (engine version, --kv, STRATA_KV_ROT, --kv-resident,
--max-context, --mtp-window, control vectors), the payload, a payload hash and an end marker; written to
`<path>.tmp` and renamed over `path` (MoveFileExW on Windows).
- fail-closed restore: size, header, both fingerprints, every count against the bytes left, the payload hash and
the usual snapshot validation all run before any device write; a refused file leaves the session untouched.
- only the deepest checkpoint is saved (the next turn's resume point) and the K/V is streamed from the pools
straight into the file's 16 MiB blocks, without a full host capture; Linux uses O_DIRECT when the filesystem
takes it (buffered otherwise or with STRATA_SESSION_BUFFERED=1), Windows buffered Win32 I/O.
- serve/server.py: llama-server's `POST /slots/0?action=save|restore {"filename": NAME}` with --slot-save-path DIR
(or "slot_save_path" in the config); NAME must be a plain file name inside DIR.
RTX 4070 Ti, IQ3_XXS, a 63,025-token conversation: file 1.20 GB, save 0.72 s, restore 0.89 s, the next 32-token
turn after a restore 1.15 s (first token 0.55 s) with the same output tokens as without saving.
Tests: conversation_file_test (CPU, no model; also built for Windows and run under Wine),
conversation_snapshot_test, serve/test_slots.py, serve/test_security.py.
From an audit of this PR, each finding checked against the code:
- VmmRange: a run whose cuMemSetAccess failed stayed counted as mapped (a later map_range skipped it, so the chunks
were never made accessible); it is unmapped and freed again (commit_run).
- a failed growth or trim could leave host_res, d_res and the slot range half-changed while serve went on to the
next request; such a failure now stops the engine, as other CUDA failures do. The residency uploads are checked.
- the trim maps the chunks it takes back as one run (one cuMemSetAccess instead of one per 2 MiB chunk) and refills
the slots with async copies and one sync instead of a blocking cudaMemcpy per expert.
IQ2_XS: the served scenario (grow at a request and mid-decode, trim and refill, regrow, a parked conversation's
restore) runs as before; vmm_test passes.
At --max-context 262144 the K/V is allocated whole at start (3.35 GiB at int8 with the drafter's) and the expert
cache is sized around it. With --kv-grow (or STRATA_KV_GROW=1) the K/V pools and the cache arena are CUDA VMM ranges
(core/vmm.hpp; the driver calls through the runtime's entry-point query, nothing links the driver library). The K/V
maps 2 MiB chunks for the cells the requests reach (16384 at start); when a request needs more, slots just below the
prompt path's loan give theirs up, a hotter expert in them first moving into the slot of the coldest one in the loan
region. A short request after a long one hands the chunks back and the slots refill from the profile. No address
moves, so no graph is captured again, and --max-context cells always fit.
- serve: grows at a request (its prompt, and a parked conversation before its restore) and in the decode rounds;
trims once the outgoing session is parked
- generate: grows once for the prompt and max_new, before the loan
- one GPU with a profile, the whole K/V in VRAM (no --kv-resident), every expert in RAM (not the low-RAM tier);
STRATA_KV_GROW_INIT / _STEP / _FLOOR for tests. Off by default: more slots move experts between GPU and CPU.
- tests/core/vmm_test: map, a chunk moved between ranges keeps its bytes, release
IQ2_XS, RTX 5090, 262K, fixed cache: first-token logits and 64 tokens bit-identical to 0.1.31 off, mapped whole,
grown into new VRAM, and grown by giving up 266 slots. Cache auto: 15,119-15,207 -> 17,300-17,503 slots, short chat
15.06-15.33 -> 14.71-14.83 ms a round; an emulated 16 GB card: 3,396 -> 5,719 slots, short chat 100 -> 115 tok/s,
a 32K prompt 7.44 -> 6.36 s and 24.6 -> 22.2 ms a round after it.
Setup no longer excludes K8V4 from KV streaming (KV_CELL_BYTES prices 816 B/cell), and docs/DETAILS.md says
it streams. This needs the engine of the PR #711 commit: MIN_ENGINE must be raised to 0.1.40 at the release,
or an older engine refuses the pair.
Co-authored-by: Trevin <[email protected]>
K8V4 (INT8 K + rotated Q4_0 V, 816 B/cell) was refused with --kv-resident. It now streams like
the other formats:
- kv_stream: a fourth block format, kKvHybrid (= 3, the number conversation snapshots already
use for K8V4), moved as three runs per block - K codes, K scales, V q4_0 - by the same resolve,
copy, ring-restore and stage paths; kv_block_bytes knows its size.
- kv_hybrid_k_half / kv_hybrid_v_half: each half of the host copy (and of the prompt path's
staging pool) as the single-format append kernels see it, so a hybrid layer's host copy is
written by the same folded calls that write its VRAM pools. Both fields of a half point at the
same array: the kernels test the K-side pointer for presence and pick K or V per lane.
- The decode (layer.cpp), verify-window (verify.cpp) and prompt (prefill.cpp) appends pass those
halves; the prompt path's staging pool gets a K8V4 layout; the host copy is allocated in it.
- Conversation snapshots read a streamed K8V4 state's host copy, as they do for the others.
- qsa_kv_format returns kKvHybrid instead of exiting; the two refusals are gone.
setup.py still keeps K8V4 unstreamed: enabling it there needs a MIN_ENGINE that has this.
Tests: kv_stream_parity runs K8V4 too (streamed vs resident attention, bitwise): 306 batches,
70,449 misses, 0 failures, ring restore identical; a negative control (the V half's K-side
pointer left null) fails it from the first batch. kv_hybrid_parity: all pass.
The attribute-after-declarator form from #778 does not compile with nvcc and MSVC as host (the CUDA build failed in mmvq_multi_kernel). The macro uses it under __HIPCC__ only, so CUDA is as it was.
Review changes on #835: off unless STRATA_HIP_PROMPT_F16=1 (a tip on gfx103x); act16/g_act_f16 and set_act_f16 are HIP only so CUDA kernels are unchanged; Prefill::init decides the mode once per stage on the stage's own device and bf16x2 takes it as an argument; widen_rows_f16 counts non-finite outputs under STRATA_DBG_NAN; new hip_gemm_f16_io_parity test.
Co-authored-by: xjc10 <[email protected]>
A bullet in the RDNA2 section: rocBLAS on gfx1030 only has tuned FP16-in/FP16-out kernels, what #835 changes,
STRATA_HIP_PROMPT_F16=0 for the old path, the measured prompt speeds on a second community machine (2x RX 6900 XT,
ROCm 10.0, engine 0.1.39), that the output is not bit-identical (link to the distribution check), and the ctest result
with -DSTRATA_BUILD_TESTS=ON: 60 of 65 pass with and without the change, the 2 skipped and 3 failing tests and why.
Per-value rounding bounds of both paths (BF16-weight products: the BF16 activation rounding goes, an FP16 output
rounding comes; quantized-weight products: an FP16 output rounding comes), why the two are not compared by argument,
how the BF16X2 reference relates to them, and the open follow-up (an fp32 forward pass as reference).
rocBLAS on gfx103x has tuned GEMMs for FP16 in / FP16 out only; FP16 in / FP32 out (Gemm::f16, Gemm::native) and BF16 in
(Gemm::bf16) fall back to generic kernels, 5.6 / ~5.3 TFLOPS against 37.7 on gfx1030 (N=10240, T=7313, K=2560).
On gfx103x only (prompt_f16(), per device; STRATA_HIP_PROMPT_F16=0/1 overrides):
- Gemm::f16 / native run FP16 out into the start of each FP32 row of Y (ldc = 2 ldy halves); widen_rows_f16 widens each
row in place from its end - no extra buffer, no host sync
- Gemm::bf16 converts W in row slices to FP16 through the dequantization scratch and takes X as the FP16 image the
gr_* / to_bf16 kernels now write (set_act_f16; saturated, a NaN stays a NaN); BF16X2 is off there
- Gemm::set_f16_io is an explicit per-instance switch that only Prefill::init turns on, so every other caller
(gemm_bf16_parity) runs what it ran; CUDA builds are unchanged
2x RX 6900 XT, IQ3_S: one card 439 / 466 / 461 -> 744 / 915 / 926 tok/s at 9.4K / 34.7K / 105.8K tokens; layer split
444 / 707 -> 833 / 1,356 at 8.3K / 33.6K. Needle 15/15 on both paths; teacher-forced, the FP16 path is closer to a
higher-precision reference (BF16X2) than the old path. Details: bench/results/2026-10-04-rdna2-fp16-prompt/README.md.
Workaround for the gfx1030 hang that needs MMQ prompt + adaptive swaps + SDMA together. Off by default; a source outside the registered arena, or not 16-byte aligned, takes the normal cudaMemcpyAsync. Untested on the reporter's 2x RX 6900 XT.
Fixes 1 and 4 of #870, plus a test that every function setup_intel.py swaps into setup.py accepts all of the original's parameters. Fixes 2 and 3 are PR #784's session/native_dense ports.
Co-authored-by: Magh97 <[email protected]>
Reworked from #873: no 8 x 250 ms retry at every failing call site, no #if in device_info (cudaMemGetInfo already maps to this function), no retry of real failures (only hipErrorInvalidValue, once), before/after lines in the log. The strata-device 5x retry loop is dropped.
Co-authored-by: unimo-chan <[email protected]>
HIP clang for gfx1030 (ROCm 7.1) rejects 'alignas(16)' placed between
__shared__ and the declaration ('alignas attribute cannot be applied to
types'). Use __attribute__((aligned(16))) after the declarator instead —
same layout, compiles everywhere.
A function-like macro splits its arguments on the commas inside a template
argument list, so cudaFuncSetAttribute(kernel<1, 6, true>, ...) failed to
compile (fused_gr.cu:1089-1093, 'too many arguments provided to function-like
macro invocation'). Restore the template function 0.1.38's hip_compat/
cuda_runtime.h had, and add the two carveout definitions used beside it.
gfx906-only: include/strata/platform/hip_compat/ is on the include path only
inside if(STRATA_HIP_GFX906); CUDA and the wave32 RDNA backend are untouched.
Two findings from the Copilot review on #695.
The five-character prefix match (replaced by the named sibling sets in the
parent commit) also grouped gfx1010 (RDNA1) with gfx1030 (RDNA2), so the gate
would have passed a table between two architectures that share no solution
IDs. The test now pins that pair in both directions, and the third RDNA3
member (gfx1102) alongside the pair already covered. Measured against the two
implementations with MSVC, same test binary source: the prefix-match gate
fails exactly these two cases (13 cases, 2 failures), the sibling-set gate
passes all 13.
<unistd.h>/getpid() and a hard-coded /tmp broke the test on Windows/MSVC with
STRATA_BUILD_TESTS=1. It now uses std::filesystem::temp_directory_path with a
steady-clock token and a per-case file name, the shape
tests/core/gguf_split_test.cpp already uses, and it reports a scratch-file
write failure instead of letting it read as a gate rejection and pass for the
wrong reason.
Built and run with MSVC 14.4x (/std:c++20, /W4): 13 cases, 0 failures.
The header gate compared the table's architecture to the runtime's with
`arch != expected_arch`. A card that reports its RDNA4 sibling - a Radeon
AI PRO R9700 (gfx1201) under HSA_OVERRIDE_GFX_VERSION=12.0.0 reports
gfx1200 - rejected its own calibration at startup and lost the tuned
prefill route entirely:
prefill gemm: tuning architecture mismatch: file=gfx1201 runtime=gfx1200; using hipBLASEx
Solution IDs are scoped to the architecture FAMILY (gfx1200/gfx1201 share
them, as gfx1100/gfx1101/gfx1102 do), and a solution ID that does not exist
on the running card already fails the per-solution gate in prefill/gemm.cu
(getAlgosFromIndex / matmulIsAlgoSupported) and falls back to hipBLASEx per
call. The header gate can therefore be family-wide: the version gate stays
strict, cross-family pairs stay rejected, and the per-row gate decides which
rows actually run.
Adds tests/prefill/tuning_header_arch_family_test.cpp (synthetic tables, no
model, no GPU): 10 cases, RED on the strict gate (3 sibling failures),
GREEN on the family gate.
Calibrated with tools/hip/tune_hipblaslt.cpp on a Radeon PRO W7800 (gfx1100) under ROCm 7.2.4. setup.py:hipblaslt_table() resolves tools/hip/<arch>-hipblaslt-<version>.txt, and no 100202 table exists for gfx1100, so prefill silently falls back to plain hipBLAS. Requested in #505.
tools/hip/gfx1100-hipblaslt-100500.txt: the same 16 dense GEMM geometries as
the other gfx1100 tables (bf16 and f16, T=4096 and T=8192, 32 rows), calibrated
on an RX 7900 XTX with tune_hipblaslt (relative L2 <= 1e-4, max abs <= 1e-2, no
padding writes) against a ROCm 10.2.0a20261003 nightly SDK
(libamdhip64.so.7.17.26392), hipBLASLt 1.5.0 (version number 100500).
The engine refuses a table of another hipBLASLt version, so on this ROCm the
shipped gfx1100 tables (100100, 100200) are refused and prefill runs on plain
hipBLAS - the same gap the gfx1201 1.5.0 table (#481) closed for the R9700.
setup picks this one by name once the installed hipBLASLt reports 100500.
Measured on a 0.1.38-lineage nightly build of this tree, Qwen3.8-Flash-Next
IQ3_S greedy, arms differing only in STRATA_HIPBLASLT_TUNING (median of 3
clean runs each):
| prompt tokens | with table | plain hipBLAS | gain |
| 131071 | 1687 tok/s | 926 tok/s | 1.82x |
decode unchanged (62.3 vs 60.7 tok/s, overlapping ranges). Per-run data,
the calibration command and the tuner log: #745
(bench/results/2026-10-04-community-rx7900xtx-hipblaslt-100500/).
docs/AMD_HIP.md: the shipped-tables list carries the entry.
ROCm 10.0.0's hipBLASLt is 1.4.1, whose solution ids are version 100401, and no
gfx1100 table shipped for it - so on an RX 7900 XTX with a packaged ROCm 10 the
prompt path's dense GEMMs fell back to plain hipBLAS. This is the same version
mismatch PR #745 measured on ROCm 10.2 (their 100500), reproduced on 10.0.0.
tools/hip/gfx1100-hipblaslt-100401.txt: calibrated with tools/hip/tune_hipblaslt
over the 26 dense geometries of the shipped gfx1100 table, on a Ryzen 7 9800X3D.
Swift 1.5 IQ3_XXS, 262144 ctx, --kv int8, fresh prompts (a nonce per request, or
the server's prompt checkpoint reads nothing), medians of 3:
1.7K tokens 682 -> 1,038 tok/s (+52%)
6.5K tokens 857 -> 1,524 tok/s (+78%)
Decode unchanged (79.0 vs 80.3 tok/s, inside this machine's 2.3% run drift), and
hip_prefill_hipblaslt_gemm with STRATA_HIPBLASLT_VERBOSE=1 reports fallbacks=0.
Gemm::native_mmq, its call, the f16_to_f32/set_bounds kernels and the Q6_K case are under __HIPCC__; the CUDA type list loses q6_k. The K % 256 != 0 out-of-bounds read is already refused in native_mmq (kept). Still opt-in: STRATA_DENSE_MMQ=1 on a STRATA_MMQ_KQUANTS=ON HIP build.
The prompt path's dense projections (attention qkv/gate, the PLE key, the shared
experts, the router, the head) run dequantize-to-FP16 + cuBLAS: on the RDNA2
cards that is 68% of the prefill GPU time at ~25% of the card's fp32-accumulate
rate, while the same shapes through llama.cpp's int8 MMQ run ~7x faster
(measured with bench_dense_gemm). This wires Gemm::native's beta = 0 products
through the MMQ path the MoE experts already take: quantize the FP16 activations
to q8_1 (they are converted to FP32 in the dequant scratch, which the old path
only fills afterwards) and run one mul_mat_q product per 1024-row chunk. It is
opt-in (STRATA_DENSE_MMQ=1; unset, nothing changes) and falls back silently to
the old path on any shape MMQ does not take.
- STRATA_MMQ_KQUANTS also compiles the Q4_K / Q5_K / Q5_1 / Q6_K instances on
the HIP branch (the mixed-quant packs' dense tensors are K-quants: this
model's attention/PLE matrices are IQ4_XS/IQ3_S/Q4_K/Q5_K/Q6_K), adds Q6_K to
the CUDA list, and defines STRATA_MMQ_KQUANTS for the HIP target.
- Gemm::native_mmq: one MMQ context, xq/ident/bounds buffer (~9.5 MB) and a
failed-flag per Gemm; fp16->fp32 and the {0, rows} bounds helper join
moe_mmq's kernel set.
- K must be a full 256-value multiple. MMQ loads the weights in 256-value K
chunks, and past a partial row the chunk reads bytes past the row - the next
row's, or for the last weight row bytes past the tensor, which no caller
guarantees to be zeros (the MoE path is safe because its gather buffers end
in a zeroed tail). On this pack the 640-value IQ4_NL shared-expert down
picked up a NaN scale from the next tensor's blocks and zeroed the output;
the odd shapes stay on the dequantize + cuBLAS path.
Measured on the RX 6900 XT (gfx1030), swift-1.5-iq3_xxs, WORKERS=15 ADAPT=1
PREFILL=8192, bench_prefill.py fresh medians: 4K prefill 436.1 -> 757.2 tok/s,
8K fresh 447.7 -> 773.9 tok/s, 8K followup 163.9 -> 206.1 tok/s; decode within
noise (40.6 vs 40.3 tok/s at --pcie-frac 0, 52.7 vs 49.9 at the 0.55 default).
Greedy output: bit-identical for the first 200 tokens of a short prompt; a 4K
prompt's greedy text diverges at token ~26 (the q8_1 activation rounding) and
stays coherent.
HIP only (__HIPCC__); the CUDA kernels are unchanged. A volatile store to mapped host memory can stay in L2 on RDNA4 until the stream syncs; __threadfence_system() after the store flushes it. Reporter's one-line diagnostic fix, applied to doorbell_ring/publish/publish_res (publish_value already had it). For #579 and #613 to test.
The fork overlaps the shared expert with the routed experts on a second stream.
It pays off on CUDA, where a36be1d added it, but on HIP the cross-stream event
dependency costs more than the shared expert takes to run, so there the fork
starts off and STRATA_SH_STREAM=1 turns it back on. CUDA's default is unchanged.
Same shape as mtp.cpp's STRATA_MTP_PREFILL_SYNC (#382): the env var always wins,
only the default differs per backend. The profiler takes the fork off to time the
stages, so a profiled run never showed this - which is why STRATA_SH_STREAM=0 was
the only way to see it.
Radeon AI PRO R9700 (gfx1201, ROCm 6.4.3), Qwen3.8-Flash-Next IQ3_XXS, --kv int8
--kv-resident 32768 --resident-experts, temp 0, 3 runs each, same binary and
prompt, 48,067-token prompt so the KV stream path runs:
stock 0.1.39 (fork on, default) 43.40 t/s (38.90 - 44.60)
this change (fork off, default) 62.90 t/s (55.40 - 63.80)
STRATA_SH_STREAM=1 43.50 t/s (39.10 - 45.20)
STRATA_SH_STREAM=0 63.00 t/s (55.90 - 64.10)
The fork is a scheduling change only: the same four prompts answer byte for byte
the same with it on and off. #816 reports the same fall on a gfx1030 (RX 6800,
Windows, 42.5 -> 56.4 t/s), and a comment on #646 sees it on a gfx1100.
Short prompts are barely affected here (65.4 against 62.7 t/s at 256 tokens), so
this only shows once the KV stream path is in play.
- #831: an explicit --expert-cache above the profile's length was cut to it with no word on the native-pack path; a
warning now says so (the allocation is unchanged).
- #841: a layer-split stage whose expert cache fails to open (out of memory with the VRAM free) is retried with 90% of
the slots, up to 3 times, each try logged. Nothing changes when the first open works.
- #779: a large-page arena cannot be paged out, so the three working-set lock sites skip it (the note says so).
- #771: with /sys/kernel/mm/transparent_hugepage/enabled = [always] the kernel already backs the arena with huge
pages where it can; the extra MADV_HUGEPAGE only added direct reclaim and compaction on every fault (a 25 s start
took 432 s on a fragmented machine). It is not requested there, and STRATA_NO_ARENA_THP=1 skips it anywhere.
STRATA_ARENA_RELEASE already exists and means the opposite (=0 keeps the experts a GPU holds, generate.cpp), so the
opt-in is renamed in the engine, the tests, DETAILS.md and the bench folder. The working-set assert in
file_expert_source_test reports, not fails, a trimmed page that Windows maps straight back with the switch on. The
bench folder keeps the README and the two scripts (the raw export and the harness self-tests are dropped).
Co-authored-by: jordicor <[email protected]>
0.1.31 registers the whole arena with CUDA before the first byte is read. With 4 KB pages, which every account
without the Lock Pages privilege gets, that registration is ~1.5 s of the start that nothing hides (IQ2_XS:
33 GiB).
- PinnedArena(bytes, bounds, Deferred) reserves only. register_slices() registers the per-layer slices in order on
a thread and publishes how many are done.
- Every reader (load_experts_gguf buffered and unbuffered, load_experts_ranges, load_experts_direct) writes layer L
only once slices L and L+1 are registered. Writing a page while cudaHostRegister runs on it corrupted the arena
under WDDM, even on large pages. A layer boundary inside a page puts that page in both registrations, so the next
slice must be done too (page-aligned in effect, without moving the slice bounds an expert must not straddle).
- A slice CUDA refuses ends the registration, and the rest is locked resident as in the sliced fallback.
- Only where the whole arena would be registered anyway: no cap (STRATA_ARENA_PIN_GIB, multi-GPU under WDDM) and no
shared arena file. Linux keeps #253's whole-arena pin. STRATA_DEFERRED_REGISTER=0 restores the old order.
- STRATA_VERIFY_ARENA=1 prints the loaded arena's FNV-1a checksum (tests).
Checksums: the same with the old order and the deferred one, 4 KB and large pages, the GGUF in place and experts.bin
(IQ2_XS; 13 deferred loads).
IQ2_XS without experts.bin, RTX 5090, 128 GB. Each cell is the time until the arena is loaded / until a 16-token
answer is done. Cold: the files pushed out of the file cache first. Two to three runs each:
| | cold | restart |
|---|---|---|
| 4 KB pages (STRATA_NO_LARGEPAGES=1), 0.1.31 | 12.4-13.0 / 16.3-16.8 s | 7.8-8.5 / 10.9-11.7 s |
| 4 KB pages, parts 1+2 | 11.1-11.5 / 14.9-15.4 s | 5.5-5.7 / 8.5-8.9 s |
| large pages, 0.1.31 | 17.6-18.7 / 21.1-22.4 s | 4.6-5.1 / 7.0-7.6 s |
| large pages, parts 1+2 | 17.6-19.9 / 21.3-23.6 s | 4.1-4.5 / 6.7-7.1 s |
The cold spread with large pages comes from the drive: the same build gave 17.6 and 19.9 s with the deferred
registration and the cache probe switched off as well.
Review of #773 (2 x RTX 5060 Ti 16 GB, 32 GB RAM, --mmap-experts, no --resident-budget-gib): with no RAM copy the
file cache is the expert tier, and forcing unbuffered reads there re-read 163-629 GB from the drive and halved the
speed. The automatic decision never armed in that mode; now the forced one does not either, and the log says so.
through the page cache, as page faults of the mapped GGUF - small requests on the critical path, and pages that a
32 GB PC cannot keep beside a 16 GiB RAM copy anyway, so they were read again and again.
- open_direct opens the files with O_DIRECT; read_direct queues every 4 KiB-aligned window of a batch with one
io_submit (linux/aio_abi.h, no library; allowed by Docker's default seccomp profile, unlike io_uring) and waits for
all of them, each thread with its own context and aligned buffer. Windows merges windows up to 1 MiB apart for
NTFS; here only windows that touch are merged. A refused submit or a failed read falls back to the mapping, as on
Windows.
- The same decision as Windows' `cache_counts = false` arm: unbuffered when the file cache could not keep the expert
bytes outside the RAM copy (platform::file_cache_keeps, from MemAvailable and the tightest cgroup limit), re-checked
once the copy is built (#577). STRATA_UNBUFFERED_LOAD=1 / 0 forces it either way. A pack's experts.bin works too
(its Linux open now records the path).
- direct_fallbacks() counts the blobs an unbuffered read could not deliver.
- Tests: file_expert_source_test reads a synthetic pack unbuffered - one blob at a time, a batch of adjacent experts,
copy_blob - and requires the file's bytes and no fallback (a mutation that fails every read is caught). ctest: 63
of 66 pass; the three others need a Q2_0 pack, a higher `ulimit -l` and AVX-512, as before.
Measured on an RTX 3080 Ti Laptop 16 GB, i9-12900H, 30 GB RAM, KIOXIA PCIe 4 NVMe, Linux 7.0, ext4: Qwen3.8-Flash-Next
Abliterated Q4_K_M in place (--native), --resident-budget-gib 16, --expert-cache auto, 8K context, greedy; three
prompts (code, Russian prose, a 7.8K-token summary), each with and without MTP, two rounds of buffered
(STRATA_UNBUFFERED_LOAD=0) against unbuffered, the same binary. Decode 4.38 -> 7.90 tok/s on average (x1.80;
x1.49-x2.01 per prompt and mode), the 7.8K prompt read at 154/172 -> 184/290 tok/s, startup 36 -> 18-21 s. The
code prompt's answer is the same text in all 16 runs; the other two differ between runs of either arm (the
adaptive GPU cache changes which experts the CPU computes), as before.
The native output head (437 MB) and the token embedding (260 MiB) were copied
out of the mapped GGUF one page fault at a time, and strata-vision's projector
(907 MB) went through llama.cpp's sequential reads, all before the server was
ready. Each now asks for its bytes ahead (the vision encoder honours
STRATA_READ_AHEAD=0 too), and the head, the embedding, the projector load, the
encoder warm-up and the pinned cache complement log their times.
On Linux every load phase read its files one page fault (one 128 KB readahead)
at a time, so a cold start spent most of its time waiting on the drive at
queue depth ~2. A new platform::advise_willneed asks the kernel for a range in
128 KiB steps (WILLNEED reads at most one readahead window per call), and is
called before the weight load, the native dense upload, the profile fill (256
pairs ahead, STRATA_FILL_AHEAD), the peer tier's fill (--peer-device), the
resident RAM copy and the MTP draft files. STRATA_READ_AHEAD=0 turns it off.
Each phase logs its time.
Gen3 NVMe, RTX 5090, 32 GB, Q2_0 resident at 262K: ready in 70 s, was ~920 s.
- #657: check_shards refuses a one-tensor shard that is longer than its tensor (the PLE table), naming both sizes.
- #737: UD-Q4_K_XL on several GPUs below the 135 GB estimate is a warning and a question (explicit --gpus --yes is
the consent), not a refusal.
- #670: an engine update keeps the engine it replaces in engine/.previous (one generation, all-or-nothing swap);
setup.py --rollback-engine puts it back.
- #798: setup counts P/E cores from the hybrid PMU lists, else by 90% of the largest capacity (tests on a fake /sys).
- #775: a learned --expert-profile / --expert-profile-save added by hand survives a setup run.
- #779: a tip (--vram-reserve-mib 1500) when nvidia-smi says the card drives a display.
- #841: typed-ahead keys are flushed before a setup question on a terminal.
- #881: read_text() with encoding utf-8 in setup.py and package_windows.py; the CPU image encoder build finds vcvars on
Windows; PCI 73DF (RX 6700 XT / 6800M) is gfx1031 with a self-build hint.
Linux counted a core as a P-core only when cpu_capacity equalled the maximum. Turbo Boost Max 3.0 gives the "favored"
P-cores a slightly higher capacity, so a Core Ultra 7 270K Plus (8P + 16E) counted 2 P-cores and started one pool
worker. Intel's /sys/devices/cpu_core/cpus and cpu_atom/cpus decide when they exist; without them an E-core is one
under 90% of the largest capacity. A CPU whose capacities differ by a few percent is no longer hybrid.
- #480: the image encoder's directory has no space in its path (a TEMP under a user name with a space broke every
image: the ENC and GENI lines are split at spaces); the ENC line names its files relative to the encoder's cwd.
- #874: the per-request combined embeddings file goes to /dev/shm on Linux when it has room, and is opened
short-lived (FILE_ATTRIBUTE_TEMPORARY) on Windows, instead of heavy steady disk writes.
- #735: WinError 4551 / 1260 on starting the engine or the encoder says Windows blocked the (unsigned) program
(Smart App Control) and what to do. A message only; no signing.
- #779: a timestamped "engine started: exe args" line in the engine log.
- #769: only EADDRINUSE says "already in use"; any other listen error says what the OS said.
Co-authored-by: ziphao2010 <[email protected]>
A minimized console window made Windows 11 treat the server and its children as background work and move their
threads to the E-cores (about 20% slower on a hybrid CPU). contain() now sets ProcessPowerThrottling off for each
child, and the server does it for itself. Windows 10 knows only the EXECUTION_SPEED bit, so it is tried on its own
when both bits are refused.
Reported with the patch by mozophe.
NativeDense serves these formats; the packer sent them down the BF16 path, so such packs failed at start.
Co-authored-by: Wesleyvdk <[email protected]>
build did mkdir(exist_ok=True) and then wrote experts.bin, dense.bin,
embd.bin and manifest.json over whatever pack the directory held, with
nothing said. Now a manifest.json, index.txt or native_experts.txt in
--out (the files the engine reads a pack's identity from) stops build
before it opens the GGUF or writes a byte, naming the directory and the
file; it returns 1 like the other refusals. An empty or absent directory
proceeds as before, and so does experts.bin without manifest.json: build
writes the manifest last, so that is a build that did not finish, not a
pack.
tools/test_shards.py: each marker is refused with the path and the file
named and nothing in the directory touched; an empty and an absent
directory go on into the build; an unfinished build is rebuilt.
model_config_files() leaves strata-<model>.shared-settings.json out in the start menu, previous_config, the
data-folder move and strata_mcp.configs().
Co-authored-by: Christoph Wübbels <[email protected]>
#406 lets a PC below setup's 256K estimate keep 256K with a note, but the list
jumped 128K -> 256K: there was no step between the 128K rule and a size that PC
may not hold. 204800 is inside the trained 262144, so it needs no rope scaling,
and with IQ3_S it costs ~2.8 GB of 8-bit KV against ~3.6 GB at 256K.
Measured on an RTX 4080 16 GB, Ryzen 9 9950X3D, 62 GB usable, IQ3_XXS, engine
0.1.37, 8-bit KV (setup turns KV streaming off at this size, so the KV stays in
VRAM and the expert cache loses the difference):
context 128K 200K
decode, shallow 94.1 85.1 t/s (median of 10, warm)
decode, 30K deep 87.2 85.3 t/s
decode, 124K deep 94.1 79.2 t/s
prefill 3257-3284 3189-3318 t/s
experts cached 3661 2951 (-710)
a 203259-token prompt: 3093 t/s prefill, 68.6 t/s decode
tools/strata_mcp.py's fallback list mirrors CONTEXTS and moves with it, so a PC
that cannot import setup sees the same menu. tools/test_setup_risk.py picks 256K
at "6" now and a new test covers a 200K pick: kept, no rope scaling, and the RAM
note a 64 GB PC gets.
A .part holding every byte (setup stopped between the last byte and the
rename) was resumed with a range past its end; the server answers 416,
which download() retried 30 times 10 s apart before renaming the file.
get_prebuilt kept an engine archive that is not a zip (or a damaged one)
together with its .done mark. download() trusts the mark, so every later run
failed on the same file, even after the right archive was published; on the
update path the engine was never updated. The archive and its mark are now
dropped before the error propagates, as 0.1.32 does for a refused archive.
Compile check on ROCm 7.14.1 (Aurora): f-140 itself stops in fused_gr.cu (12 errors) and mtp.cpp (2) for gfx906 there; with this
commit the port adds no error of its own (fused_gr.cu's pre-existing ones remain).
Without --mtp, serve drafts with the suffix/prompt-lookup drafter only (or one
token per round), as the plain generate command already does; every token is
still verified against the model. The draft layer's VRAM goes to the expert
cache instead (1,000 -> 1,678 slots in an A/B on an RTX 2060 SUPER 8 GB). The
bind, the prompt path's draft prefill (batched and per token), the draft-KV
restore, prompt length, max drafts, draft sampling, coupled-draft history and
the draft call itself are skipped without it. The conversation cache is off
without it: a parked conversation always carries the draft layer's K/V.
With a split the two cards take turns on every verify window: stage 0 runs its
layers and hands off, then waits while stage 1 runs the rest, the head and the
draft. The batch pipeline (--batch-groups) overlaps different conversations;
one conversation's decode stayed serial.
--pipeline-windows 2 runs window K+1 on stage 0 while stage 1 verifies window
K. The guess for K+1 comes from the MTP chain, launched as soon as K's verdict
is known and teacher-forced through K+1's drafts: if K is accepted as guessed,
K+1 is already half done; if not, stage 0 restores its GDN state from a
snapshot, replays K's commit with the real count, and K+1 is rebuilt from the
real tokens. --pipeline-windows 1 overlaps only the prompt's short reads
through the windows (--short-read). Default 0: nothing changes.
What it adds:
- Verifier: a non-blocking form of run()/commit() (capture_all, pl_launch,
prestage, service, done, pl_finish, pl_commit_async), the batch pipeline's
batch_launch/batch_poll pattern for one sequence. The window, its staging,
the PLE gather at layer 0, the zero-doorbell graph's flag, the #267 release
on a timeout and the commit graph are run()'s. Two verifiers per stage (one
per window parity) share one stream of their card (set_stream) and each
pair has its own hand-off.
- set_always_publish: the pipelined verifiers copy the token rows to the host
at every layer, because the adaptive tier now marks experts evicted on the
host while windows are in flight, before the device residency table follows.
- MtpDrafter: chain_launch (the round and its steps back to back, an event
after each output, the first steps fed the in-flight window's drafts by a
one-thread forcing kernel), prefill without a host sync, 8 rows with 2, the
drafter's stream at the highest priority (CUDA only).
- the adaptive tier beside the pipeline: evictions are marked first, then the
refill copies wait (on the tier's thread) for every window in flight; the
RAM side of a swap (exchange, release, unpin) waits for the windows that may
still pull the incoming experts over PCIe.
- an agreement gate: a speculative window is launched only when the chain's
estimate says it is likely kept (STRATA_PIPELINE_THETA, default 0.2); a
window known to be wrong gets empty plans for its remaining layers; a lookup
window's continuation becomes the speculative window behind it.
- VRAM: 160 MiB more kept out of each card's cache for the second verifier,
and the two GDN snapshots on the first card.
- off, with one log line saying why, for anything it does not support: --batch
slots, --peer-device, the helper caches, three or more stages, a split on
one GPU, no drafter. Requests with repetition penalties or coupled draft
sampling decode serially. Nothing is in flight outside the two pipelined
loops, so checkpoints, parking and the serial paths are unchanged.
Measured: Swift 1.5 IQ3_XXS at 160K (q4_0 KV, the stock draft layer), RTX 4060
Ti (layers 0-19) + RTX 5080 (20-47), i9-14900KF, 32 GB of RAM, the resident
RAM mode on the split (#848); greedy, 500 tokens, two interleaved pairs of
three rounds, decode tok/s, serial -> pipelined: Python code 90.4 -> 103.1,
C code 76.6 -> 81.6, English prose 63.0 -> 71.1, Italian prose 41.6 -> 46.3,
a copy-heavy edit (a 5 KB file back with a rename) 90.5 -> 117.9; mean
72.4 -> 84.0 (+16%). With the experts read through the OS file cache
(--mmap-experts on that PC) the file reads dominate and it measured no faster.
Correctness: with STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0 the
greedy text is identical to the serial loop's when both have the same expert
caches (4/4 prompts, 500 tokens; the serial run given the pipeline's reserve
through --vram-reserve-mib). The same text with STRATA_PIPELINE_THETA=2 (no
speculative window), STRATA_PIPELINE_FORCE_MISS=1 (every speculative window
wrong, rolled back) with and without DOOM_SKIP, PRESTAGE=0 and
SNAP_OVERLAP=0. Against a run without the flag a few experts move from a card
to the CPU (the pipeline's VRAM), which rounds them differently, so a near-tie
can flip. With the adaptive tier on: 22 varied requests at temperature 0.6
(thinking on and off, five languages, code), and 11 more with
STRATA_PIPELINE_FORCE_MISS=2, gave no degenerate answer, no stall.
The watchdog prints the pipelined loop's state when it fires;
STRATA_PIPELINE_TRACE=<file> writes a timeline.
For the pipelined verify windows of a layer split (next commit). Nothing calls
it yet, and nothing changes unless set_force_capture is set before load.
- chain_launch: the catch-up round and its steps launched back to back with no
host wait, an event after each output (chain_outputs_ready reads them as they
land, chain_poll once the chain is done). Its first steps can take given
tokens as their input instead of their own picks (teacher forcing: the drafts
of a window already in flight), through a one-thread kernel at the end of the
round and step graphs that does nothing while its mapped token is -1; the
outputs stay the chain's own picks.
- prepare_chain / prepare_prefill: capture every graph and size the prefill's
input records up front, so neither a later chain nor prefill(sync = false)
waits for the device.
- set_source_R: the rows a round reads, copied into the bound buffer first (the
pipelined loop has one stage-1 verifier per window parity).
- with forcing on, the drafter's stream gets the highest priority (CUDA;
STRATA_MTP_PRIORITY=0: the default).
draft() resets the forced tokens and copies the source rows the same way, so
the serial loop's drafts are unchanged.
Extract rotation and atomic ownership publication from aa6e0e3 and 1a50d91 onto upstream a1641e9. Preserve current host-memory tests. Include one historical token-identical A/B with explicit provenance and document fresh CPU/CUDA validation.
In the resident RAM mode (--resident-experts) an adaptive round runs on a
thread the decode loop joins after the draft: choose, copy the evicted slots
back with a stream sync, copy the new experts in from the RAM copy (through
the driver's staging when it is pageable). With ~90 swaps that was ~24 ms
every 4th window (measured on our fork). --adapt-async 1 splits a round into
four steps that advance between windows on a helper thread, so no window
waits for one:
1. choose (adapt()'s rule, over snapshots of the counts and the residency)
and copy the evicted slots back (D2H), each from the cache of the card
that owns its layer
2. stage the exchanges, mark the evicted experts non-resident (the CPU reads
them from the exchange buffers), copy the new experts in: straight from
a page-locked RAM copy, else through a pinned bounce buffer
3. mark the new experts resident, move the evicted blobs into the RAM places
the new ones left
4. flip the copy's offsets
A slot is overwritten only after its old expert stopped being planned for the
GPU, and a RAM place is reused only once its expert is resident. The residency
tables on the devices are uploaded at steps 2 and 3, as apply_pending does
(the verify graph's doorbell reads them). A request starts with the round
drained, before its prompt lends any slot. --expert-profile-save counts the
routing at the start of each round, as the blocking tier does. Swaps that
need no exchange (the evicted expert is in the RAM copy already, or the new
one is not: a RAM budget) take the same steps without a copy back; a new
expert outside the RAM copy is read from the files on the helper thread.
Default off: which window first computes a swapped-in expert on the GPU
depends on when its copy lands, so the tier is not bit-exact from run to run.
Without the flag nothing changes. With it, the blocking tier still runs
outside --serve, without the resident RAM copy, and with --batch slots,
--peer-device or --remote-expert-opt (said once at startup). The exchange
buffers are twice as many with it (the bounce half): 96 more blob-sized
pinned buffers, ~130 MiB with 1.38 MB blobs.
Layer split (with the resident mode on a split, #848): each swap's copies run
on the owning card's refill stream under that card's device; the homes are
looked up the way #848's resident_stage_swaps does.
Measured: Swift 1.5 IQ3_XXS at 160K (q4_0 KV), i9-14900KF, 32 GB of RAM,
Windows 11, greedy, decode tok/s, blocking -> --adapt-async 1:
- RTX 4060 Ti + RTX 5080 layer split with the resident RAM mode (#848),
--trim-stage-weights, the stock draft layer, five prompts x two rounds,
two interleaved pairs: 79.7 / 78.8 -> 87.6 / 86.2 (+10%).
- the same split with a fine-tuned draft layer and --spec-min-p 0.65, four
prompts x three rounds:
81.0 / 81.8 -> 108.1 / 107.0 (+32%).
- RTX 5080 alone with --resident-experts: 26.7 / 26.6 -> 29.5 / 28.5 (noisy).
Without the flag the greedy output is the same as main's, byte for byte (4/4
prompts, 500 tokens, STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0). With
it, 22 varied requests at temperature 0.6 on the split (43,825 experts
swapped in over 622 rounds) gave no degenerate answer.
Drops the dead stage_pool trim block (the search only runs for auto, where stage_trim is false), keeps the search's number
for each card unchanged (the full untrimmed arena), and fixes the 'needs an explicit --layer-split' warning, which now
fires only when there is no GPU split at all.
--trim-stage-weights (PR #559) / STRATA_STAGE_TRIM=1 (PR #639) gives a stage only its own layers' dense
weights, but it refuses to run unless --layer-split is written out: `auto` chooses the split points 590
lines after the stages' weights are loaded, so by the time the split is known there is nothing left to trim.
Those weights now load after the search. The search still prices each card's arena from the pack index
(`WeightTable::pool_bytes` reads the index and allocates nothing), and takes it off the card's measured
free VRAM, so the number it sees is the number it has always seen - the split it picks is the split it has
always picked. This buys cache, not a different placement. The expert caches are sized later, from live
free VRAM, so they collect the freed bytes.
On the 4-way IQ3_S rig, `--layer-split auto` (which picks K=10,19,34):
CUDA1 3650 -> 4608 expert slots, CUDA2 1241 -> 2485, CUDA3 533 -> 1772
prompt chunk 768 -> 6144 tokens
prefill 469.7 -> 1929.7 tok/s on the same 107,082-token prompt
CUDA0 still trims only with an explicit --layer-split. Its arena is read out to build the prompt path's PLE
tables, 590 lines before the split exists, so moving it is the larger half of this change. It is also the
cheap half to leave: measured, it costs 1% (1929.7 against 1950.8 for an explicit `10,19,34
--trim-stage-weights`, where CUDA0's cache is 4598 slots instead of 3017).
bench/results/2026-10-04-rdna2-helper-pcie-share: the rig (two cards on PCIe 4.0 x8, probe 14.1 GB/s -> pcie_frac 0.39),
the method (STRATA_DECODE_TIMING per request), and the per-window breakdown for the baseline on a1641e9 (with and
without #835 on top: 59.6-60.4 ms per window, 4 PCIe entries per layer-window, the GPU-reach wait at 43 ms), for the fix
with the probed share (two runs: 37.2 / 33.8 and 35.2 / 33.8 ms, the share down to 0.06-0.26 entries) and with
--pcie-frac 0, and for the --pcie-frac 0 workaround on the baseline; the limits and the configuration used.
docs/SECOND_GPU.md: the helper section said HIP had not been validated; it now gives the two-card HIP measurement, what
the share cost before this change and the workaround, and keeps three/four cards as not validated.
The verify window's GPU plan counted a layer's misses and moved the last pcie_num/256 of them to the primary over PCIe,
leaving out the primary's resident experts and the peer tier's but not those a helper GPU (RemoteExperts) holds. The
plan runs before RemoteExperts::begin(), which only claims rows the plan left at kind -1, so a PCIe share above 0 read
part of the helper's experts over the primary's link while the helper idled for them.
RemoteExperts::holds(layer, expert) (the helper cache's slot_of), and the plan treats such an expert as it treats the
peer tier's: not a miss, never in the PCIe share; its rows stay at kind -1 for begin(). No default changes.
2x RX 6900 XT, IQ3_S, --remote-expert-opt with the probed share 0.39: 59.6 / 58.5 -> 37.2 / 33.8 ms per window,
PCIe entries per layer-window 4.00 -> 0.26 / 0.06, decode 40.6 / 39.3 -> 65.6 / 67.7 tok/s (--pcie-frac 0: 66.9 / 67.1).
Maintainer: separate stop and recover modes (reasoning_loop_recovery: false|stop|recover), token-level splice of the effort sentence (EFFORT_TEXT), a client's top_p/top_k never touched, checks at the first clean boundary after each 512 tokens, bounded history, tests incl. a literal think tag (#728, #869)
A request with "strata_checkpoint": false sends ckpt=0 to the engine. That request reads its prompt in one part and
saves no checkpoint at its last turn boundary, nor the periodic ones of a long prompt. It still resumes from a
checkpoint it matches and still saves the system-prompt root. Absent or true: today's behavior; an engine without the
key skips it.
prepare() wrote the request's combined embeddings file (req-<id>.sve in the vision directory, ~10 MB a picture) right
after encoding, and only run()'s finally deleted it. But prepare() could still raise after writing it - EngineStarting
(the 503 while the engine starts, #344), the "no room to answer" and "exceeds the context" 400s - and a request can end
between prepare() and run(): its client gone before the first chunk was written, anything raised in between. run() is
a generator, so one that never started has no finally to run. Each of these left its file behind for good, and
clients that retry 503s added one per retry.
prepare() now writes the file only after those checks, and records its path before writing it. run() takes the path
over as it starts and deletes the file as before; do_POST's finally calls the new Service.drop_embeddings(), which
deletes a file that no run() took over.
Tests: EngineStarting, no room and exceeds-the-context leave no req-*.sve; over HTTP, on both APIs, a 503, a 400, a
client gone after prepare() and answered requests leave none.
The markers are special tokens only where the template writes them for an image item. The same strings in a
message's text (an agent reading chat_template.jinja, a tool result quoting it) became the same special ids, so text
with <|vision_start|><|image_pad|> before a picture took that picture's embeddings, every later picture moved up one
and the last real marker became text - every count still matched.
On 0.1.39 this takes #537's path: the vision markers join <think> / </think> as literal tags (frontend.VISION_TAGS),
swapped for private-use marks before the template and encoded as text through the tokenizer's plain spans. A pair
of markers cut apart across text parts is refused ("the prompt and its images do not match") instead of shifting.
With tool_result pictures reaching the encoder, a server without one (every lazy-start config is text-only) or an
encoder that refuses a picture (WebP without Pillow, an error from strata-vision) answered 400. Clients resend the
whole history every turn, so one such picture failed every later request of the conversation; before, the picture
was dropped silently. Now a tool's picture that cannot be read becomes "[image omitted: <reason>]" in its place; a
picture the user sends keeps the 400.
Maintainer: log a left-out tool picture, a tool picture's URL is downloaded once (bytes reused by prepare), tests (#529; #582 and #819 are duplicates)
Claude Code returns a picture it opens with Read as an image block inside a tool_result. The Anthropic
path turned every tool_result into its text, so the image never reached the encoder and the model
described a picture it had not seen. The tool message now carries the image item, as the OpenAI path's
tool messages already did; the chat template renders it inside <tool_response>. An image sent beside
tool results in the same user turn is kept too. A text-only tool result is still one string, so text
prompts are unchanged.
jsonschema is optional (serve/structured.py): without it a json_schema answer is checked to be one JSON object, so
the answer the test expects to fail passes, and the test failed on every install without it (0.1.39's own as well).
Codex sends its MCP tools as namespace tools (mcp__websearch with web_search
inside). The Responses adapter shows the model mcp__websearch.web_search, but
Qwen3.8-Flash-Next writes the call as mcp__websearch__web_search, the flat
form Codex used for MCP tools. That name had no entry in the reverse map, so
the call came back without a namespace and Codex rejected it as an
unsupported call. Register the double-underscore spelling too (setdefault, so
a real tool of that name still wins).
prepare_format refused every schema whose root was not literally "type":"object",
so a root anyOf/oneOf of object shapes - what apps written for llama.cpp's
json_schema send - got a 400, although every answer it allows is one JSON object.
_only_objects() accepts a root that can only be an object: "type":"object", an
anyOf/oneOf whose branches all qualify, an allOf with a qualifying member, or a
local $ref to one (a $ref cycle never qualifies). The "return exactly one JSON
object" directive and the object check without jsonschema stay true. A root that
also allows an array, string, number, boolean or null (including
"type":["object","null"] and a schema without a type) is still refused before the
engine loads.
Tests: serve/test_structured.py, unions accepted (a non-object answer is still
refused) and non-object roots refused: 10 OK with and without jsonschema
installed. test_server/security/lifecycle/detok/mcp/monitor/winjob: 214 OK.
docs/DETAILS.md: the root rule.
The server is HTTP/1.0 and closes the connection after each response, but
never said so. A client that pools connections (.NET's SocketsHttpHandler)
then sometimes put its next request on the closing socket and got
'response ended prematurely' without the server ever reading it: 25 of
~1000 requests in one night from an agent app, each 1-24 ms after the
previous answer.
A reply such as "I will use the `<tool_call>` format now." followed by the call raised
ValueError("malformed tool call") in OutputParser.feed and ended the request: everything after the
first <tool_call> was taken as the call body. A call is the tag followed (after whitespace) by
<function=, as the chat template writes it; the tag with anything else after it is now content, and
it is held while streaming until its follower arrives, so streamed and whole output agree.
Found while porting tolerant tool-call recovery from a Qwen engine (ninfer --tolerant-tool-calls):
of its three recovery cases, Strata already handled values that quote the markup (#210) and prose
after the calls; this was the one that failed. Tests: ToolCallTagInProse (fails on main with 21
subtest errors, passes here); python -m unittest serve.test_server: 120 tests OK.
Without a grammar decoder a forced tool_choice was ignored, so clients
that need a call (an agent's first step, its "done" call) got text.
The server now writes the call's opening itself - <tool_call> and
<function= (with the name for a named choice) - and the model can only
go on with a call:
- without thinking, the opening ends the prompt;
- with thinking, it is written where the thinking ends, at a clean
point, the way the budget's wrap-up is (continuing from the prefix
the engine already holds);
- when reasoning_budget_tokens wraps up the thinking, it follows the
wrap-up.
"auto" and no tool_choice are unchanged. "none" offers no tools, as on
the Responses route. An unknown name, a forced choice without tools
and other values are a 400 instead of being ignored; a forced choice
with MCP tools too. The arguments are not constrained; a reply cut by
max tokens stays "length" without tool_calls, as before.
Maintainer: unknown tool_choice values are logged and act as auto (no 400), the flat {type:function,name} shape, Anthropic any/tool/none mapped and passed to /v1/messages, tests (#790). Note: tool_choice none now offers no tools on both routes.
A body over 1 MiB was left unread, and its answer was reset on Windows. The limit is now 5 s for the whole body, read one socket read at a time: a body of any size gets its answer, and a client that sends a byte at a time is let go after the 5 s (read() waited for the whole piece, with a new timeout for each byte).
The server answers before reading the body in some cases (401, 403, /load, /unload, a method with no handler) and then closes. Closing on unread bytes makes Windows send a reset, and a client that sends its body after the headers (http.client, urllib, requests) gets WinError 10054 instead of the answer.
After an answer, the body no handler took is now read and dropped, up to 1 MiB and with a 5 s timeout. A larger or chunked body is left unread, as before.
Filter earlier assistant messages with no text, images or tool calls
before rendering the prompt. Keep the final message.
Co-authored-by: jramirez-uab <[email protected]>
Signed-off-by: Arthur031221 <[email protected]>
With tools declared, a <tool_call> in the reasoning is held whole; a declared name becomes a tool call,
anything else (undeclared name, malformed, unfinished, cut by </think>) stays reasoning text. finish() never
makes a call from it. Without tools the parser is byte-identical. Tests: parser (split tags, undeclared,
no tools, unfinished), OpenAI and Anthropic finish reasons, Anthropic block order.
Co-authored-by: kossum <[email protected]>
Maintainer: fatal lines checked before batch routing, 'an earlier window never finished' is fatal, alive() check in generate_batched, tests run on Windows (#748).
Vision.load read whatever an image source named. A network path (\\host\share\x.png, //host/share/x.png,
\\?\UNC\host\..., \??\UNC\host\..., file:////host/share/x.png) made Windows connect to that host and sign in with the
user's NTLM credentials - os.path.isfile alone does it. Such paths are now refused before anything looks at them, and
so is file://host/share/x.png (that names \\host\share; the old code read it as the relative path host/share/x.png).
An http(s) image was read whole, with no cap, inside prepare()'s `with self.fifo`: a huge one filled the memory and a
slow one held every other request for as long as urlopen waited (60 s per read). Now it is read up to 32 MiB
(IMAGE_URL_MAX; a larger Content-Length is refused unread), downloaded before the FIFO is taken, and a URL that cannot
be read is a 400 instead of a dropped connection.
Other sites' pages: with no api_key (the default) _authorized() lets every request in, and /v1/chat/completions and
/v1/messages read the body as JSON whatever its Content-Type; only /settings, /v1/load, /v1/unload and the MCP tools
check the Content-Type and Origin (_own_page). So any web page open in the user's browser could POST a chat request as
text/plain - a CORS "simple" request, no preflight - that names a file on this computer (or a network path) as an
image, and the server read it. The page gets no answer to read (no CORS headers unless cors_origins lists it or "*"),
but the read, and a network path's sign-in, happened. Reproduced over HTTP: answered 200, the file handed to the
encoder. Now a request whose Origin is neither the server's own address nor in trusted_origins may send data: and
http(s) images only; a file path or file:// URL is a 400 that says so. Clients that send no Origin (CLI tools, SDKs,
agents), Strata's own page (it sends data: URLs) and trusted origins are unchanged. As for /settings and MCP, a
DNS-rebinding page, whose Origin matches its Host header, still counts as the server's own.
Tests: each network form is refused with os.path.isfile never called, and local files, file:// and data: URLs still
load; the cap with and without a Content-Length, and a 404; the URL is downloaded with the FIFO free; a foreign-Origin
text/plain request naming a file is a 400 on both APIs, while data: images and no-Origin, own-origin and
trusted-origin requests are answered.
Row error on 4,000 random rows against the checkpoint's BF16 table (Q8_0 0.53%, FP8 2.64%, IQ4_NL 7.60%, matching the
numbers reported in #651), and teacher-forced KL on the Q2_0 model with each table.
The 0.1.39 paragraph said nobody had run the 0.1.39 port on an Arc. Two Arc Pro B60 reports (#867, #870) did, with
the two sycl/ fixes now in 0.1.40 (#784 the compile fix, #866 the ring wait). Say that, and that an unpatched 0.1.39
port did not compile against its own engine sources.
INTEL_ARC.md: "What has been run" rows for 2x and 1x Arc Pro B60, plus notes for the card: PCI id 8086:e211 (bmg-g21), setvars.sh
stopping at OCL_ICD_FILENAMES under `set -u`, running natively without Docker, and the layer-split startup warning that counts the
other card's layers.
sycl/setup_intel.py: the B60 table entry was e221, which the kernel's xe id list files under the BMG-G31 cards (0xE220-0xE223). Both
Arc Pro B60 cards measured here are 8086:e211, so setup named them "Intel GPU e211 (xe)" with a guessed VRAM size. e211 is added;
e221 stays, since there is no card here to say it is wrong. tools/test_setup_sycl.py checks the table.
COMMUNITY_BENCHMARKS.md and bench/results/2026-10-04-community-2x-arc-pro-b60/: the report, in the template's format, with the
server configs, engine logs and per-request timings.
The migration turned `q = cudaStreamQuery(cs)` into `DPCT_CHECK_ERROR(cs->ext_oneapi_empty())`. That macro yields 0 whenever
the call does not throw, whatever the queue holds. The guard `q != 1 && *seq < want` (verify.cpp, session.cpp) therefore
fired on any ring that took more than 2 ms and reported "never rang (graph finished)" while the graph was still running.
With every expert resident the host does not wait per layer (STRATA_VERIFY_NO_HOST), so the path measured on the B70 never
reached it. With host-mirrored experts the first layer whose GPU time passes 2 ms does: layer 15 of the Coder on one Arc Pro
B60, every time.
q is now 0 when the queue is empty and 1 while it is running, as the guard and the trace comment ("1 = still running") expect.
fixups.py gets the same rewrite (entry 8), so a fresh dpct run of session.cpp produces it too.
Measured on one Arc Pro B60 (4,042 of 12,288 Coder IQ1_M experts mirrored in RAM, 128 greedy tokens, AOT bmg-g21): before,
exit 1 at layer 15 in 5 of 5 tries; after, exit 0 in 7 of 7 runs, at 11.2-11.9 tok/s with coherent text.
The sycl port went in as 5ffcce4 ("sycl: port 0.1.39"). Two engine-side changes
were merged into the release branch after it (4f5e6c8, f5410ae) and nothing in
sycl/ was updated for them, so the v0.1.39 tag does not compile:
- #626 (5a8b292) made the thread affinity a value: pin_current_thread()
returns strata::kernels::cpu::ThreadAffinity and restore_thread_affinity()
takes it. sycl/include/strata/core/session.hpp still stored a `long long`
(no conversion exists, and it did not include the header that names the
type); sycl/src/core/session.cpp still assigned the returned struct to it
and passed the `long long` back.
- #559 (9461f5e) added the two layer-range arguments to NativeDense::load.
The sycl out-of-line definition was still the 4-argument one and no longer
matched the declaration in include/strata/core/native_dense.hpp.
Both are fixed by mirroring the CUDA sources: the same member type, the same
`outside()` predicate, and the same place it is applied in the tensor loop.
`pinned` now follows CUDA as well: it is pinned_core.valid, so a pin that
failed is not "restored". Nothing changes when the pin succeeds.
No caller passes layer_lo/layer_hi yet, so `outside()` is never true today:
this restores the build and does not change what the engine computes.
Measured here: Arch Linux, Intel oneAPI DPC++ 2026.0.0 + oneMKL 2026.0,
SPIR-V (JIT), cmake -S sycl -B build-sycl -G Ninja -DCMAKE_C_COMPILER=icx
-DCMAKE_CXX_COMPILER=icpx, then
`cmake --build build-sycl --target strata -- -k 0` -> 0 errors, links. The two
errors are a type and a declaration mismatch, so the compiler version does not
matter.
Not built here: the other targets (kernel parity tests, benches) and the AOT
path. Not run on an Arc in this change.
The full checkpoint stores the table as BF16 (320-byte rows, 102.4 GB), the source of record; the engine gets it as
IQ4_NL (about 8% off per row) or FP8. A bfloat16 is the top half of a float32, so widening is a shift: exact for
every value, no rounding and no scale. One entry in the PLE format list (type BF16, self-describing: no scale, no
metadata) and bf16_dequant_row; the direct reader takes the 320-byte row like any other width, and
PLE_ROW_BYTES_MAX is 320.
ple_reader_test checks the widening is exact over the patterns a lossy path breaks on (subnormals, +-inf, both NaNs);
the generic every-format test covers a synthetic BF16 GGUF through both readers.
Not in this commit: the PR's rename of tools/ple_fp8_pack.py (nothing needs it) and any setup option. Whether a
table format other than IQ4_NL is worth offering is a quality question (see the measurement in the 0.1.40 notes).
Ported by the maintainer from constantindjonkam's PR #586.
A plain llama-quantize Q4_0 file of this model (bartowski's Qwen3.8-Flash-Next-Q4_0 and
Swift-1.5 Q4_0: gate/up Q4_0, down Q4_0 on 42 layers and Q4_1 on 6, a Q4_0 PLE table and
token embedding) was refused at start: no GPU kernels for Q4_0/Q4_1 experts, and the PLE
reader took only IQ4_NL, Q5_0 and FP8. Follows #222 (Q4 to Q6 quantizations).
- Q4_0: Fmt<2> with the codes centred before dp4a (q - 8 as signed bytes), the exact
integer form ggml-cpu's q4_0 x q8_0 dot uses, rather than llama.cpp's CUDA chain
(uncentred codes minus 8 x the q8_1 float sum): on a real Q4_0 expert the CUDA chain
measured 1.4e-2 relative error against the float product, the centred form 5.3e-3.
- Q4_1: Fmt<3>, the Q5_1 chain without the fifth bit and the same min-term choice.
- dq_q4_0 / dq_q4_1 for the prompt path's dequant fallback; row bytes, is_iq and the
GU/D/MMVQ format lists.
- PLE: Q4_0 rows (90 B, as IQ4_NL) through the same reader: one entry in the PLE format list (ple_format_test and
ple_reader_test cover it).
- docs/UNSLOTH_Q4.md: a Q4_0 section with the measured run.
native_expert_parity --synthetic (now in the CMake matrix): q4_0/q4_0 CPU 1.29e-2, GPU
1.29e-2, CPU-GPU 1.1e-7; q4_0/q4_1 CPU-GPU 3.2e-4; GPU dequant bit-exact against
to_float. The existing pairs are unchanged. On bartowski's file (revision 928589f), layers
0 and 5 (Q4_1 down) and 20 and 47: CPU and GPU 1.1-1.3e-2, CPU-GPU 1.1e-7 (Q4_0 down) and
5e-4 to 1.1e-3 (Q4_1 down). Served on one RTX 3060 (all experts in RAM, 131K context):
24,461-token prompt 787 tok/s read, 24.0 tok/s out; 67,934 tokens 737 / 26.4, the hidden
fact found.
Ported onto the format table by the maintainer (the PLE part of the PR is the Q4_0 table entry).
mlock alone brings a cold file in one page at a time from one thread, ~0.47 GiB/s (about two minutes for a 54 GB
Q8_0 table); 16 threads reach ~2 GiB/s on anon761's machine. The pages are then resident and mlock only pins them.
When mlock fails, the pages stay faulted in (the message says they may be reclaimed) instead of being touched again
one by one.
ple_reader_test gets --make-table PATH FORMAT (a big synthetic table, with --rows) and --ram-time PATH (open mapped
and locked, print the seconds) to time it cold.
Ported by the maintainer from anon761's PR #651.
Co-authored-by: ksanislo <[email protected]>
PleTable::open compared rows x row_bytes against the file after the 64-bit product could already have wrapped, and
mapped the tensor before checking it fits. Now: refuse a row count that overflows rows x row_bytes and a data
section that starts past the end of the file, check offset and size without an addition that can wrap
(offset > have, need > have - offset), require a single-tensor shard to start at offset 0 and fill the file, and map
the table only after all of that. close() also resets the FP8 scale and the byte counter.
ple_reader_test --selftest writes tables that claim one row too many, one too few and a wrapping row count, for every
format, and checks both readers refuse each with a message and stay closed.
Checks taken from CC-David-CC's #865 (its opt-in Q8_0 table is #651's; the direct reader reads Q8_0, so the
experimental switch and the mapped-only limit are not needed).
Co-authored-by: CC-David-CC <[email protected]>
Ordinary GGUFs ship per_layer_token_embd.weight in other formats than IQ4_NL: Unsloth's UD-Q6_K_XL and
Swift-1.5 Q4_K_L as Q8_0 (170-byte rows, ~54 GB), a Q5_K_M finetune as Q5_1 (120-byte rows). The engine refused
both at start.
- dequantize_q5_1 in strata/artifact/dequant.hpp (ggml's dequantize_row_q5_1: unsigned 5-bit codes times d plus the
block minimum m); Q8_0 rows use the existing dequantize_q8_0
- two entries in the PLE format list (the table-driven commit before this one): Q5_1 and Q8_0, on both readers;
PLE_ROW_BYTES_MAX grows to 170
- tests: dequant_q5_1_test (synthetic blocks against ggml, no model, no GPU), dequant_bf16_test knows Q5_1,
ple_q5_parity takes Q5_0, Q5_1 or Q8_0 tables and checks both the mapped and the direct reader
Ported onto the format table by the maintainer from anon761's PR #651 (the --ple-io ram fault-in is the next
commit). Quality of the Q8_0 table: ksanislo measured 0.53% error against the BF16 table, FP8 2.66%, IQ4_NL 7.59%.
Co-authored-by: ksanislo <[email protected]>
PleTable had one bool per format (q5_0, fp8) and an if-chain in decode(), open() and format(). Now one list,
ple_formats(), holds one entry per table type: its GGUF type name, the bytes of a 160-value row, whether it
carries a scale and its dequantizer. open(), format(), the row size, the error string and the row buffers all
read it, so the next formats (Q8_0, Q5_1, Q4_0, BF16) are one entry each.
No behaviour change: IQ4_NL, Q5_0 and FP8 decode with the same functions, and the "not IQ4_NL, Q5_0 or FP8 (I8)"
error text is built from the list and reads the same.
ple_reader_test --selftest now writes a PLE-only GGUF for every format in the list and checks both readers
(direct and mapped) against the format's own dequantizer, for single rows and a 16-row token gather.
The kernel reproduces one hipBLASLt solution's summation order; the test compares both on the same inputs for T from 64
to 16384 (bitwise) and SKIPs without STRATA_HIPBLASLT_TUNING.
16 switches whose output was byte-identical to the switch off on Aurora (GDN head / PP / CONVL2 / NOY / SPLIT, QFUSE,
CVEC_FUSE, HCD_EXACT, PF_PAD, Q8 / Q6 packed, MMVF_ROWS, ATTN_LANECELL, EXPERT_V2, TSUM, LFUSE, PLE_BATCH) are set at
start-up when device 0 is gfx1151 and the user has not set them; STRATA_GFX1151_DEFAULTS=0 turns the table off. The
GDN head kernels are AMD-only now (HIP builds; CUDA keeps its key-head kernel), and verify.cpp's namespace-scope env
reads became lazy so the table reaches them.
WmmaTraits::mma's #else returned its accumulator (zeros) for a gfx11 part outside the __gfx1100__..__gfx1151__
list; it traps now, and the runtime gates of the WMMA GEMM, the prompt attention, the block scorer and the HC
up-projection take exactly gfx1100 / 1101 / 1102 / 1150 / 1151 (strata/kernels/gfx_arch.hpp) so a gfx1103 or
gfx1152 falls back to hipBLASLt instead of launching code that is not there. The gate is cached per device
(a process-wide static answered for the first GPU only; hipGetDeviceProperties on every call was the review's other
complaint).
0.1.39 already skips the host round trip when every expert is in VRAM (all_resident_, on by default, #871's
refresh_ar), so the port keeps that graph and adds only what halo built on top of it: the LFUSE gated combine.
A prompt loan, a VRAM shrink or an adaptive swap marks experts -1 in the residency table. The zero-doorbell graph
planned from the device table and returned silently on a -1, so the window ran a stale plan (!!!! at 100% residency).
refresh_ar() checks the host table before each window and runs the doorbell graph (its own captured variant) while any
expert is out; resident_plan also writes a mapped error flag the host checks after the window.
Every upload of the expert residency table (host_res -> d_res, and each stage's) was a plain cudaMemcpy: a pageable
copy on the legacy stream, which may return before its DMA has landed, while the kernels that read the table run on
streams created cudaStreamNonBlocking, which do not wait for the legacy stream. A kernel enqueued right after could
read a table half old, half new: an expert computed by both the GPU and the CPU, or by neither. The ~100 KB copy
lands in microseconds, so this rarely fires, but nothing ordered it. The uploads now go through res_put(), which
waits for the copy on the current device's legacy stream only. Same values, same output.
With an engine that keeps --resident-experts on a layer split, setup no longer
steers a low-RAM PC with two cards to one of them:
- at install, the low-RAM mode on several GPUs takes the resident variant by
the same rule as on one card (what no card holds fits the RAM with
LOW_RAM_HEADROOM_GB beside it), without the one-card question;
- at a start, a resident config on several GPUs keeps --resident-experts
(split_mmap leaves it alone);
- a resident config started on a PC with two cards is offered both, "y" by
default, as any other config.
It is gated on MIN_ENGINE >= RESIDENT_SPLIT_ENGINE (0.1.40 here, a guess at
the release that would carry the engine change), so until then setup behaves
as before and the golden configs are unchanged. Two tests run the new paths
with MIN_ENGINE patched; tools/test_setup_*.py pass (260 tests).
--resident-experts (the low-RAM mode's resident variant) ran as plain
--mmap-experts as soon as a layer split was set. It now keeps its RAM copy on
a split:
- the experts every stage's cache holds are left out of the RAM copy, not only
CUDA0's;
- when the rest does not fit whole (#467), the copy keeps the hottest by the
whole expert profile. Each stage only keeps its own layers' share of the
profile, and ranked by CUDA0's share alone the copy held layers 0..K-1's
experts and read every later layer's from the files (4.0 GiB in RAM instead
of 21.2 on the PC below);
- an adaptive swap copies the evicted expert back into RAM from the cache of
the card that owns its layer, on that card's stream. host_res holds slot
numbers of whichever cache owns the layer: reading a later stage's slot out
of CUDA0's cache put another expert's bytes in RAM, and after the first
swaps the answers turned to garbage. (Latent until now: the guard kept the
mode off a split.)
- the decode profile prints every stage's GPU profile, not only CUDA0's.
Remote expert caches (--expert-cache-remote) are still refused with it.
Swift 1.5 IQ3_XXS at 160K, q4_0 KV, RTX 4060 Ti (layers 0-19) + RTX 5080
(20-47), i9-14900KF, 32 GB of RAM, Windows 11, four greedy prompts, decode
tok/s (first round / fourth):
- 5080 alone, --resident-experts: 25 / 29
- split, --mmap-experts (what --resident-experts became): 32 / 64
- split, --resident-experts (22 GiB of experts locked in RAM): 71 / 69
With --pcie-frac 0 --adapt-every 0 the split's greedy output is the same as
--mmap-experts' (4/4 prompts). With adaptive swaps on, 33 varied requests at
temperature 0.6 (thinking on and off, five languages, code, each checked for
degenerate output) exchanged 81,783 experts with the two caches and gave no
degenerate answer.
--vram-reserve-mib is every card's reserve, but the card that drives the
monitors needs more headroom than one that drives none: one value has to fit
the display card, and the other cards then keep that headroom for nothing.
--vram-reserve-later-mib N is the reserve of every card after the first
(default: --vram-reserve-mib's value), in the split search and in each later
card's cache sizing.
RTX 4060 Ti (no display, layers 0-19) + RTX 5080 (display, 20-47), Swift 1.5
IQ3_XXS at 160K: --vram-reserve-mib 1000 gives the 4060 Ti's cache 4551 slots;
--vram-reserve-mib 300 --vram-reserve-later-mib 1800 gives it 4867, and the
5080 keeps 1.3 GB free with everything loaded.
Bitwise-identical to e58d7fc (generated ids identical on UD-IQ4_XS, UD-Q4_K_XL, IQ3_S, IQ3_XXS at 8K/4K/64K; harness fused-output hash unchanged):
- table16: IQ4_NL / IQ4_XS decode through the direct-selector v_perm form (X5; exhaustive 2^32 check). -DSTRATA_W_NO_T16 restores the old form.
- moe_combine: four columns per thread with float4 loads, same fma order (X5); falls back to the scalar kernel if Dm / shared / bo are not 16-byte aligned. -DSTRATA_W_NO_COMB.
- native_w11_kernel: the H tile aliased onto the weight buffers (gate/up LDS 46.5 -> 30.2 KB), amdgpu_waves_per_eu(8), 4 blocks per WGP (X1), only for the types that do not spill under the 192-VGPR cap (gate/up IQ2_*, IQ3_*, IQ4_XS; down Q2_0, Q8_0, IQ4_NL). -DSTRATA_W_NO_OCC.
- X3 on1 (skip the second 16-row n tile of the down kernel) NOT included: no gain alone (-0.1%), and with the occupancy cap it makes IQ4_NL down spill 62 VGPRs and the whole bundle 2.3% slower.
0.1.39 clamped two q8_1 activation quantizers (#606: an fp16 sum or scale past 65504 rounds to inf, the dot products
then read inf * 0 = NaN, and a NaN in the residual ends as one token repeated). Two more store the block unclamped:
- native_swiglu_quantize_q8_1_kernel (native_mmvq.cu), the shared expert's fused SwiGLU + q8_1, on by default
(STRATA_FUSED_SWIGLU_Q81) in decode and the multi-token pass;
- native_gu_fused_kernel (iq_kernels.cu), the routed experts' fused SwiGLU + q8_1 (gfx906 builds, STRATA_EXP_MODE=8).
Both now go through q8_1_finite / q8_1_quant / q8_1_ds as their siblings do: every finite block is stored bit for bit
as before. iq_multi_parity's #606 check covers the shared expert's one (gate = 32, up = x / 32: its activations are x
exactly). rwkeyes found the first one (#606).
gdn_rec_quad_kernel's limit was shared memory: every wave re-read the 64 k / q floats of its row group per token
and the shuffles went through the same unit (probe: no k / q reads 9.8 ms, no shuffles 16.9 ms, of 21 ms). A lane
now carries two columns (8 lanes x 2 columns x 4 row groups = 16 columns a wave, 64 a block, 96 blocks, 384 waves
in one round at 230 VGPRs): the two columns share the k / q reads, and each wave has two independent dependency
chains. Per column the same arithmetic in the same order. One 16K chunk-layer 25.4 -> 16.9 ms (gdn_probe14.hip,
bitwise vs the old kernel); 64K prompt 1232.4 -> 1267.6 tok/s (-1.47 s, interleaved, 2 runs), ids identical at 8K
and 64K. STRATA_GDN_PP=1 is the one-column kernel with the register fixes only (-0.66 s).
gdn_rec_quad_kernel needed 196 VGPRs = 7 waves a SIMD, but its 192 blocks of 4 waves need 9.6: a second round of
blocks ran after the first (half the heads took exactly half the time). Cause: the 32 state-row addresses of the
first loads were CSE'd with the final stores and kept live across the whole walk (64 VGPRs); the stores' lane offset
now goes through an empty asm, so the addresses are made again - 133 VGPRs. A full chunk's four tokens also run
with the state alternating between two register sets (the update writes the other one), which drops the 32
register copies the compiler made after each update (131 VALU a token instead of ~178). The same arithmetic in
the same order. One 16K chunk-layer 25.4 -> 20.5 ms (gdn_probe12.hip, bitwise vs the old kernel); 64K prompt
1233.7 -> 1249.0 tok/s, -0.66 s; engine output ids identical at 8K and 64K.
STRATA_GDN_CONVL2: gdn_conv_tiled_kernel with gdn_l2_kernel folded in (a block's 128 threads are one head's channels):
the tile's conv loop verbatim, then the sum of squares spelled out as the compiler built the l2 kernel (first xor step
an fma on the partner's rounded square, then adds), the four partials in the same order, the same v * rsqrtf(ss+eps).
16K chunk, per layer: 9.13 ms -> 6.95 ms. A first version keeping the values in registers differed by 1 ulp in the
silu (different code shape); this one keeps the tiled loop as is.
STRATA_GDN_NOY: the norm kernel does not store the FP32 y nothing reads (403 MB per 16K chunk and layer); the FP16
value is made with v_fma_mixlo_f16, the instruction the compiler used next to the store (otherwise a mul and a
convert round twice and 1 ulp of the FP16 results differ).
Engine ids match the unchanged binary at 8K (acd9c8328408).
native_w11_kernel: the sign bytes of the IQ2 / IQ3 grids come from a shared-memory table of byte masks
((g ^ m) + c per 4 bytes; no zero byte in any grid) instead of the compare / subtract emulation; the decoded
weight stages are one flat shared buffer addressed through a pointer, which changes the code the compiler makes
for the down kernel (7.5 -> 6.2 ms per 128-expert launch, IQ3_S / IQ4_NL, 16384 tokens). One layer of 16384
tokens, fused experts: IQ2_S 83.8 -> 76.3 ms, IQ2_XXS 74.4 -> 68.9, IQ3_S 78.5 -> 73.8, IQ3_XXS 77.6 -> 71.8.
Outputs are bit for bit the same (the test now prints a hash of the fused output in pair order).
The formats carry a minimum, so the activation blocks' spare bytes now hold the int8 code sums per 32 values (written by
every path, read by none else). Layers MMQ does not cover on HIP (Q4_K) join the fused layout through MmqPlan::fo.
Opt-in (default output bit-identical). Test pairs plus an int8-rounded model check; QA_SAME probe in the attention harness.
With the image encoder on the GPU, the 700 MiB reserve can leave ~200 MiB free on a 12 GB card (UD-IQ4_XS
end-to-end check on an RTX 5070). Setup prints a one-line recommendation; the written config is unchanged.
- "experimental" is per model now: MODELS["UD-Q4_K_XL"] keeps it, the unsloth family and UD-IQ4_XS drop it. The
first menu lists the family without [experimental], with a one-line about (UD-IQ4_XS: a 94 GB download; under
~80 GB of RAM part of its experts are read from the SSD). Its size menu lists UD-IQ4_XS first (the default) and
UD-Q4_K_XL second; an experimental size always sorts last. --model names are unchanged.
- UD-IQ4_XS: no EXPERIMENTAL warning, --check verdict without it. Images are asked (off by default), per model: the
engine's image path has no restriction for this pack, which uses the original model's image encoder; not yet run
with images. UD-Q4_K_XL keeps images off.
- MCP: the fallback tables mirror setup's (per-model experimental/vision), sizes in setup's order, per-size
"experimental" and "images", the vision check per model.
- Docs: README and its six translations, MODELS, UNSLOTH_Q4, AI_SETUP, MCP_SERVER.
- Tests: the menus, the default, --model by name, images, --check, the pack/tokenizer paths; the golden configs
are unchanged.
- a prompt that fits 0.1.39's auto chunk keeps 0.1.39's ring (Prefill::set_ring_budget's small_max): 4K prompts
were 1-17% slower with the smaller ring (Coder -15%, IQ3_S -5%, IQ3_XXS -2%; 5 interleaved pairs each);
- ring slots are given up for a bigger chunk only where 0.1.39's rule held the chunk under 6144 (ring_cap_for);
from 6144 on the ring keeps its 0.1.39 size (the Coder 6144/384 -> 7936/199 was -12% at 32K);
- the unpinned arm's cap stays 96 (the byte budget made it ~49 on IQ3_S);
- the scan sets its own ring budget instead of set_ring_override, so a layer split's 96-slot override (#340) is no
longer clobbered by the scan.
RTX 5070 vs STRATA_RING_BYTES=0: 4K -0.7..+1.1% (bit-identical); 32K IQ3_XXS +18.5% (cache 1500) / +9.4%, Coder
+3.1%, IQ3_S +0.2%.
The padded activation of ssm_out / attn_output was handed to Gemm::native, whose padded-weight path needs a
strided dequant; the GSQ-RCO IQ3_XXS model has IQ3_S ssm_out tensors (no strided dequant for the i-quants), so
the engine exited at the first 64K chunk ("a padded X needs STRATA_PF_PAD's path"). Now such a weight is
dequantized unpadded and strata_pf_gemm_f16_ld reads X at its stride: the same kernel and products as with
PF_PAD off, the same bits.
Added where a config goes to several GPUs (setup --gpus / a multi-GPU install, start --gpus, "use both" at
start). Single-GPU configs are not touched. Opt-out: setup --no-remote-expert-opt, or "remote_expert_opt": false
in the config (a user key, kept on re-runs). The engine uses it only with a helper cache (--expert-cache-device1..3),
so a layer split runs as before. tools/test_setup_remote_opt.py; the 2x16 GB golden configs gain the flag.
The hyper-connection read ran two hipBLASLt BF16 GEMMs over the same T x 10240 xn16: down (N 320, 6.78 ms per
16K-token half, 15.8 TFLOPS - the 4 KB-multiple row stride) and inject (N 4, 1.81 ms). strata_pf_hcdown_bf16 does
both in one 128 x 128 BK-64 WMMA GEMM: weight rows [w_down; w_inject] read from the two tensors, the epilogue
splitting lo (T x 320) and inj (T x 4), so xn16 is read once; with the flag, xn16 is written with token stride
10240 + 64 (gr_norm_rs / gr_write_norm_rs / gr_write_cvec_norm_rs take an ldx; unchanged and bitwise when off).
s23/hcdown_probe.hip, 16K tokens: 3.60 / 3.77 ms for both (unpadded 5.75).
Opt-in, on chunks of STRATA_PF_SWITCH_MIN_T+ tokens. FP32 accumulation in another k order than hipBLASLt:
rounding-level. Aurora, c-best3 v3.2: 64K timing hc read 7234 / 7253 -> 6439 / 6175 ms; prompt 8K 1048 -> 1075 /
1036, 64K 1053 / 1056 -> 1067 / 1068 tok/s. Teacher-forced KL (8192-token prefix, 2001 positions) vs the default,
next to the accepted MMQ-vs-fused pair: offset 0 0.162 (mmq 0.158), 30000 0.110 (0.126), 70000 0.031 (0.033);
ledger {32,17,130}, 64K retrieval 3/3.
With the speed-projection control vector on, every steered layer's FFN write-back took three passes over the
T x 10240 FP32 residual (gr_write, cvec_apply, the next half's gr_norm_rs) instead of the one gr_write_norm_rs of
the other layers. gr_write_cvec_norm_rs does the three in one: gr_write's arithmetic, cvec_kernel's dot order and
two-level reduction, gr_norm_rs's sum order and block_sum - the same bits. STRATA_CVEC_FUSE_CHECK=N runs the
three-kernel path on a copy of R on the first N fused writes and compares R, the row scales and the BF16 image
(Aurora, 8K prompt, layers 4-11: all identical). cvec_tables() hands out the current device's vector tables.
c-best3 v3.1 on Aurora, one 64K prompt with STRATA_PREFILL_TIMING (x2): hc read + combine 13.0 / 13.1 ->
10.8 / 11.2 s, GPU timeline 64.6 / 64.9 -> 60.6 / 63.0 s; benches interleaved, same ids: 8K prompt
971.5 / 1019.9 -> 998.8 / 1045.1, 64K 1025.3 / 1012.8 -> 1039.2 / 1014.4 tok/s.
`--prefill auto` walked a fixed list of chunk sizes - 32768, 16384, 8192,
6144, ... - and took the first that fitted, with the ring a fixed 384 slots
taken off the top before the chunk was ever considered. Both halves of
that were wrong on a rig like this one.
The list is coarse exactly where it matters. `bytes_needed` is a sum of
(T x positive constant) terms plus a max of such sums, so it rises
monotonically with T, and so does every test the scan applies - which makes
the largest chunk that fits a bisection on the 256-token grid the prompt
path already works on. Seven probes against the list's ten, at one
`bytes_needed` each. It matters because this rig affords ~8,700 tokens and
was handed 8192; 8704 is not on the list.
The order was wrong because chunk and ring spend the same borrowed VRAM,
and a ring slot is worth far more than a chunk token. Measured here (2x
RTX 3060 + 2x RTX 5060, CUDA3 lending at its 90% cap, 120K prompt): 8960
tokens with the 17-slot ring that leaves reads at 963 tok/s, 8192/130 at
1,008, 7168 with the ring full at its 199-slot byte budget at 1,000, 5632/199
at 915. A ring slot is worth ~0.53 tok/s and a chunk token ~0.05, so the 69
slots between a full ring and 8192's 130 are worth more than the 512 chunk
tokens they cost - and once the ring IS full, a smaller chunk buys nothing.
So: the largest chunk that still leaves the ring full, and only a rig where
no chunk can afford one falls back to the old rule with a kRingMin floor.
The room is counted in BYTES, not slots. A ring slot is max_blob, but a
cache slot holds its own layer's blob - 2.15 MiB against a 2.54 MiB max_blob
here. In slots the ring looked 18% cheaper than it is, a chunk whose ring
fitted only after that discount was refused outright, and the scan stopped a
step short. `bytes_from_slots` is the inverse of the existing
`slots_from_bytes`, and `cache_slots_for` factors the latter out for the
serve path's per-participant version. Both inverses clamp at k <= 0: the
lend budget is `min(slots - 128, pct * slots / 100)` and goes negative on a
cache under 128 slots, where `slots - k` is off_[slots] - the end of the
prefix sum, so that cache lends 0 bytes rather than reading past its offsets.
Upstream's per-participant `fits`/`fits_one` and the #448 diagnostic are kept:
the ring room is taken over the smallest participant, `alone` is the chunk
CUDA0 alone would read in, and the small-card warning is unchanged.
Verified on the rig: the scan picks 7680/199 against the old rule's 8960/17,
1018.8 and 1021.1 tok/s on two cold 120K prompts against 963.1 - +5.9%.
Q8_0 picks 5632/101, its 506 MiB budget over a 5,222,400 B blob.
0.1.39b (Niko1221): behind STRATA_RING_BYTES=1 with the byte-budget ring; the default keeps 0.1.39's chunk list
(plan_lend and the serve scan), its log line gains the ring. The INFO fields (the Monitor commit) are not taken.
With global loads removed, the BK 32 double-buffered kernel still stopped at 37-39 TFLOPS of the ~50 WMMA peak:
the K loop itself (fragment reads, one barrier per 32 k) was the limiter. pfg::kernel_bk64 stages a 64-k tile in
one LDS buffer (55 KB) from 12 named registers with 2 barriers per tile, half the loop trips. Every accumulator
takes the same WMMAs in the same k order: the same bits (s23/gemm_probe11.hip, memcmp vs the BK 32 code).
gfx1151, M 16384, both operands padded (2 runs): N 2560 K 6144 38.0 -> 41.3 TFLOPS, N 12288 K 2560 38.9 -> 40.9,
N 10240 K 2560 40.3 -> 40.4. Also probed and rejected (same bits, slower): A rows read under the previous row's
WMMAs (36.7-37.9; 8-16 B spills), half-wave 16 B reads + v_permlanex16 (31.7-32.4), both (32.8-33.5).
Used for N >= 1024 and K % 64 == 0. c-best3 v3 on Aurora, BK 32 -> BK 64, interleaved, same ids: prompt
8K 959.8 / 947.8 -> 968.0 / 1001.9, 64K 1007.9 / 1004.0 -> 1028.0 / 1025.4 tok/s.
Default: 0.1.39's 384/1024-slot ring and bytes_needed count, so the default loan and output stay as they were.
The auto chunk scan (the PR's second commit) is not taken.
On a card whose experts mostly run on the CPU the slots cost speed per request (RTX 5070, Q2_0: 11-24 %), so setup
now recommends slots only when the expert cache (every card of a split counted) still holds half of the model's
experts beside them (Q2_0: 24 GB and up, or a split); elsewhere it stays at one at a time and says "parallel N
reduces waiting for several users but costs about 10-25% speed per request on this card". --parallel N is
honoured as asked, as before. BATCHING.md: the table with the change against one at a time.
The ring was 384 slots - a number measured on Q2_0, where a slot is one
expert blob of 1,382,400 B, so 506 MiB. A slot is a whole blob, so on a
pack with bigger blobs the same 384 slots are a different amount of
memory: 1,912 MiB on Q8_0, more than half of an 8 GB card's expert cache,
and what kept that card at a 1024-token chunk when its own buffers would
have fitted 6144. Measured on the 4-way rig, one env var and nothing
else: chunk 1024 -> 6144, prefill 87 -> 402 tok/s.
So the budget is the constant and the slot count is derived from the
pack's max blob: 384 on Q2_0 (pinned) or 96 (not), exactly as before, and
the pack it was tuned on does not move. The fused rule (1024 slots, #136)
is preserved the same way, and `ring_cap()` still bounds the result. This
is also what lets the auto chunk scan treat "the ring is full" as a plain
byte comparison.
Also exports the three accessors the scan needs: bytes_needed_no_ring
(the chunk's own buffers, without the ring), ring_max_slots (the cap in
slots) and ring_slots_for (what the ring resolves to for a chunk, after
the override, the env var and the pinned-share rule - what `init` lays out
and what the INFO line reports).
And one fix on the way: `bytes_needed` counted `carve`'s buffers from
memory and got two wrong - it took `xn` unconditionally, though `carve`
takes it only under STRATA_GR_UNFUSED, and it never took `grs`. Net
T*(D-HC)*4 bytes too many: 42 MB at a 1024-token chunk. The direction is
safe - the prompt path was told it had less room than it did - but it
under-sizes every loan.
A 4 KB-multiple row stride (K 2048 / 4096 / 6144 halves) camps on the memory channels: the gfx1151 WMMA prompt
GEMM ran ssm_out (N 2560, K 6144) at 19 TFLOPS against 31-32 for K 2560. Padding the rows by 64 halves (gemm_probe8,
M 16384): K 6144 19.1 -> 35.0 TFLOPS (W alone 29.0, X alone 23.7), K 2560 31.7 -> 36.6.
With STRATA_PF_GEMM=1 on chunks of STRATA_PF_SWITCH_MIN_T+ tokens: Gemm::native dequantizes the weight into the
scratch with stride K + 64 (dequant_f16_ld, non-i-quant types) and runs strata_pf_gemm_f16_ld; the FP16
activations of the two K 6144 projections (gdn_recurrence's y16 for ssm_out, gate_attn's out16 for attn_output)
are written with stride 6144 + 64. The same products in the same order: the same bits.
c-best3 v2 + STRATA_PF_PAD=1 (Aurora, interleaved, same generated ids): prompt 8K 954.2 / 930.7 -> 983.7 / 940.8,
64K 947.0 / 946.2 -> 974.8 / 1014.4 tok/s.
Per MoE layer of a verify window (2-8 rows, resident combine path):
- the shared expert's scalar gate (a 1-row BF16 GEMV, one latency-bound
workgroup) rides as one more block of the router GEMV launch
(bf16_gemv_fp32_mmvf_multi_aux; the rows kernel's per-output code is the
multi kernel's - checked bitwise for 2-8 rows, with and without TSUM)
- its sigmoid + row scale move into the combine (combine<true, true>: the same
sigmoid expression, the product rounded on its own with contraction off in
that block only, then added as before)
- shared gate and up projections in one launch (native_mmvq_pair: blocks
n_out.. take the second matrix; Q8_0, exact multi layout, not packed)
Per attention layer: the window's int8 KV appends in one launch
(kv_append_q8_steps, blockIdx.z = token x K/V; distinct cells).
-4 launches per MoE layer, -(n-1) per attention layer; STRATA_LFUSE_GATE=0 /
STRATA_LFUSE_PAIR=0 turn the gate / pair parts off.
Engine (gfx1151, UD-IQ4_XS, c-best3 + EXPERT_V2 + Q8_PACKED + TSUM, S25's 18
prompts, same binary): 51.29 -> 50.31 ms per verify window, 54.33 -> 55.12
tok/s; 59K prompts 54.19 -> 53.17 ms, 42.97 -> 43.79 tok/s; all outputs identical.
server: a request decoding in a slot whose neighbours finished (no other slot busy, nobody waiting, at least 32
tokens allowed) is BSTOPped and continued on the solo path with its MTP drafts - the engine copies the slot's
sessions back (INFO slot_cache=1; at most twice per request; STRATA_PARALLEL_SOLO=0 keeps it in the slot).
engine: INFO batch_slots=N slot_cache=0|1.
tools/batch_interleave_test.py: the back-to-solo path, and a next turn from a slot's turn checkpoint (a client
that drops the reply's thinking).
docs: "parallel": N, what a slot costs and what setup recommends, the server's handling (interleaved prompt reads,
BYIELD, slot conversation cache, back to solo), exactness and its settings, the RTX 5070 measurements (the slots
buy waiting time, not speed, on a 12 GB card), the protocol; DETAILS / README / AI_SETUP mention the option.
INSTALL.md gets the section (the build setup compiles, what differs, the forced-path numbers from a
Ryzen 5 7600 + RTX 5070, what is untested); the CPU rows of INSTALL.md / DETAILS.md and AI_SETUP.md
point to it.
STRATA_HC_UPMIX, STRATA_PA_FAST and STRATA_PF_GEMM change the prompt path's rounding; on short prompts their time
gain is negligible but they change the generated text. With N set, chunks shorter than N keep the default
arithmetic (and outputs) while long prompts keep the gain. Unset or 0: every chunk, as before.
gfx1151, c-best3 + STRATA_PF_SWITCH_MIN_T=4096: S25's 18 prompts (1.0-1.9K tokens) give outputs identical to
c-best2 v2 on 18 of 18 (49.28 tok/s; c-best3 without the gate 48.83 / 48.67, other texts).
batch_test.py: the engine log in the temp folder, the CUDA DLLs on PATH, ASCII output, --mt-min (empty: the
engine's default kernels, for the invariance check without the exact settings). server: /metrics live.waiting
and live.outside_slots instead of a guessed count.
Windows: pool workers pin with SetThreadGroupAffinity (the processor group was dropped: a 3990X's 63 workers
collapsed to 32 group-local masks); the host thread selects a CPU Set and restores the previous selection.
Linux: the complete, dynamically sized affinity mask is kept and restored (CPU IDs past 63).
pool_affinity_test (Windows) and pool_affinity_linux_test. Squashed from the PR's 8 commits (net diff).
The slots' sessions may take a fifth of the expert cache (~VRAM less 5 GB), up to 4: 2 on 12 GB, 3 on 16 GB, 4
from 24 GB, none below. Without --parallel setup only says it once; with it the config gets "parallel": N and a
count above the recommendation is kept with a note.
- "parallel": N in the run config passes --batch N (a value the engine cannot use is said and passed on: the
engine warns and adjusts); the server uses the slot count the engine reports (INFO batch_slots)
- every running request keeps its own status, rate window and engine figures (history rows are their own);
/metrics live.slots shows each slot (idle with the tokens it holds, reading, decoding with tok/s),
/v1/status concurrency.serving says N
- a conversation's next turn goes to the slot that holds its start, a new one to an empty or the least recently
used slot; a slot left at a stop token keeps what it holds
- a long prompt read (solo or an admission) gives way (BYIELD) to a waiting request with a prompt under half as
long, at most twice; the part read waits in a reserved slot and the request goes on once the other is in
- the engine start and POST /v1/vram are serialized against parallel requests
- serve/test_parallel.py: a scripted engine behind the real StrataEngine and Service
- --slots N is the same as --batch N; a count the engine cannot run (no --serve, 1, more than 8, a split on one
GPU, sessions that do not fit, --batch-groups that do not divide) is a warning and fewer slots or none, never a
refusal; INFO says batch_slots=N when slots run
- a batch window runs over the active slots only (Verifier::run_slot_rows, graphs per row list): an idle slot is
not written, so it keeps the conversation it served
- a request whose prompt starts with what an idle slot holds (all of it, or its turn checkpoint) continues from
the slot's sessions instead of reading the history again
- with slots, a prompt is read one prompt chunk at a time (the same chunks as one run) and the slots decode
between two chunks for half as long as the chunk took (STRATA_BATCH_DECODE_SHARE)
- BYIELD <slot>: the prompt being read stops at its next chunk boundary, the part read is copied into the slot,
YIELDED <slot> <tokens> + DONE cancel; the same request sent again goes on from there with the same chunks
and checkpoints (#656's cooperative preemption, a slot as the park)
Default (no --batch) unchanged.
PR #638 was cut from 0.1.38. 0.1.39's segmented expert cache (#533) keeps its cuMem* code out of HIP builds through
STRATA_USE_HIP, which the gfx906 compat build does not define, so it included <cuda.h>. Its virtual-memory blocks are
now behind STRATA_EC_NO_VMM (STRATA_USE_HIP or STRATA_HIP_GFX906); the CUDA and RDNA preprocessor output is unchanged.
(std::max) keeps windows.h's max macro out of two Windows-only lines (found by a Windows gfx906 compile check).
s26_down_l_kernel (STRATA_EXPERT_V2): a warp's 4 rows have 40 (block, half)
items each; lane-strided per row, lanes 0-7 took 2 items of every row (8
loads) and lanes 8-31 one (4). Under STRATA_TSUM=1 the 160 items are spread
5 per lane (item p = 32 i + lane is row p / 40), all 5 loads issued before the
dots. A lane's items of a row are then exactly those of the per-row layout's
lane (l - 40 q) mod 32, in the same order, and an xor butterfly gives the same
bits under any rotation of the lanes (each level's partner differs by +-2^j,
which is +2^j modulo 2^(j+1)): every output is bitwise the same (ws3 harness,
memcmp, T 1-3: 1.09 / 1.11 / 1.12x for gate/up + down with TSUM, vs 1.03 / 1.05
/ 1.06x with the butterfly alone). The same spread for gate/up (80 items x 4
rows, 10 per lane in 1-3 load groups) was 0.6-0.8x and is not used.
A warp that ends with P independent warp sums (P tokens of a row, or rows x
tokens) used P xor butterflies of 5 shuffles each. s26_tsum.cuh does them in
one transposed butterfly: at offset 16 a lane keeps half of its values and adds
the partner's copy of that half, at 8 half again, ... then plain xor steps.
Every level adds the same pair of values the butterfly adds (fp add commutes),
so each sum is bitwise the same; P - 1 + (5 - log2 P) moves instead of 5 P, as
DPP row_xmask / permlanex16 on AMD (__shfl_xor_sync elsewhere).
Applied under STRATA_TSUM=1 to the decode kernels that end in several sums:
- HC read (STRATA_HC_Q8): gr_down_q8 / gr_up_q8 (ws4 harness, every output
equal, T 1-4: 1.00 / 1.03 / 1.03 / 1.15x per read)
- STRATA_EXPERT_V2 gate/up and down (ws3 harness, memcmp, T 1-3:
1.03 / 1.05 / 1.06x)
- native_mmvq_multi_kernel (exact layout) and the Q8_0 / Q6_K packed kernels
- bf16_f32_mmvf_rows_kernel (STRATA_MMVF_ROWS), both reduction stages
Engine (gfx1151, UD-IQ4_XS, c-best3 + EXPERT_V2 + Q8_PACKED, S25's 18
prompts): 54.21 -> 52.10 ms per verify window, 51.31 -> 53.73 tok/s, all 18
outputs identical.
- layer split: commit stays synchronous on a split (set_commit_async(!multi_gpu), the `|| next_ != nullptr`),
no pre-staging of the next stage's inputs, kDrafterMib 1000 - the split's sizing and order as in 0.1.39
(untested here: one GPU);
- the CPU expert kernels keep cpu_avx512_ok() (F/BW/VL/VNNI/VBMI): #646's AVX-512BW-only check would send a
CPU without VNNI/VBMI to kernels that use them;
- resident_plan_kernel: one thread per window entry up to 128 (64 lost a 7- or 8-token window's last entries
on the all-resident path);
- the host keeps zeroing the GPU's rows under STRATA_VERIFY_DEVICE_PLAN=1 (copy_or_zero_from_mapped copies
every row there).
dpct left the DPCT1106 note but dropped cudaMemGetInfo in generate.cpp's
stage_room, so every later stage of --layer-split sized its expert cache from
0 bytes free and the second card crashed with 'expert cache: no room'.
Reported with the fix by tmking01 in #423 (2x Arc Pro B70); also a fixups.py
entry so a re-migration keeps it.
The SYCL port of PR #423 tracked 0.1.38; 13 of its migrated copies were behind
0.1.39's sources. Re-migrated 0.1.38 and 0.1.39 with sycl/tools/migrate.sh,
normalized, merged with sycl/tools/merge_upstream.py:
- clean merges: expert_cache, expert_source, native_dense, pinned, elementwise,
kv_stream, iq_multi_parity, prefill, verify.hpp; q8_1_finite.hpp new (#606)
- verify.cpp: the #649 trace kept with the port's queue handling
(STRATA_VERIFY_TRACE; the HIP-only coherent/doorbell switches stay HIP-only)
- generate.cpp: the port's --prefill-borrow beside --vram-elastic/--vram-segment-mib
- iq_kernels / native_mmvq: the port's kernels kept, #606's finite q8_1 scale
and sum applied by hand (dpct's conflicts there were kernel-name hashes)
- expert_cache: --vram-elastic (#533) is CUDA-only on the port too (no driver
VMM on Level Zero through dpct); it reports so instead of failing to build
- version 0.1.39
Compiled here with icpx 2026.1.1 (SPIR-V, no Intel GPU on this PC): strata
builds. Not run on an Arc.
docs/OLDER_GPUS.md: Pascal / Volta through the CUDA 12 engine (#395#600#540#655), RTX 20's STRATA_BF16_TC opt-in,
AMD gfx1031 (#524), gfx1012 (#442) and gfx906 (#638, #677) with the reporters' numbers; README, AI_SETUP, AMD_HIP and
NVIDIA_V100 point to it.
Random BF16 rows of the router's shape [512 x 2560], 1..8 tokens: both within 3.3e-7 of the double
sums on a Ryzen 5 7600 (0.53 ms and 0.50 ms per layer for 4 tokens). Two ctest entries, the second with
STRATA_FORCE_ISA=avx (the AVX-only CPU's path). Test program only.
CUDA 13 dropped Pascal and Volta. A model whose GPUs include one (its oldest card decides: one engine runs per model)
now runs the experimental CUDA 12 engine in engine-cuda12/ - the ready-made strata-windows-x64-cuda12.zip with NVIDIA's
CUDA 12 pip libraries, or compiled here with a CUDA 12.x toolkit (Linux; STRATA_NVCC picks one, #601) - and setup says
why. engine/ and every model on it stay as they were.
- the cards are an opt-in: naming one (--gpu / --gpus), --cuda 12, STRATA_EXPERIMENTAL_SM60=1, or a PC with no newer
NVIDIA card and no supported AMD card; a PC with a newer card keeps recommending it
- --cuda 12|13 (or STRATA_CUDA) overrides the choice and is kept, with a warning where it cannot work (recommend,
never force); --cuda 12 also runs with a driver older than 580 (525+), which the stop now mentions
- at a start, a Pascal / Volta card added to a CUDA 13 model moves that model's config to the CUDA 12 engine
- updates look at both engine folders; a compiled CUDA 12 engine builds in build-cuda12/
- an RTX 50 card in a CUDA 12 engine is a warning (CUDA 12.8+ builds it; #220 crashed with 12.8), no longer a stop
The setup golden test (every recommended config on five PCs) is unchanged; tools/test_setup_older_gpus.py covers the rest.
- CMake: STRATA_ISA_FLOOR (empty by default) builds ggml-cpu for AVX (Sandy/Ivy Bridge) or SSE4.2
(Nehalem/Westmere) instead of the build host or AVX2; every other ggml ISA option off. Strata's own
AVX2/AVX-512 kernels stay in the binary behind their CPU checks. Excludes STRATA_PORTABLE. The ggml
block follows the Strata_Dirigo fork's (rwkeyes).
- STRATA_FORCE_ISA=avx2|avx|sse caps the engine's CPU probes, so the older-CPU paths run on any PC
(how this was tested here: no old CPU). cpu_sse42_ok(), isa_floor_build().
- The engine: a normal build still refuses a CPU without AVX2, now pointing at the experimental build;
a floor build checks its own floor (AVX, or SSE4.2 + POPCNT) and warns that it is experimental and
slow (or, on an AVX2 CPU, that the normal build is faster).
- setup: a CPU without AVX2 warns ("experimental, slow") and compiles the engine on the PC with the
floor it has, instead of failing (#595's idea); the prebuilt AVX2 engine is not used there. Recorded
in BUILD.json; a build folder for another floor is configured afresh. STRATA_ISA_FLOOR=avx|none asks
for it on any PC. Windows + AMD has no source build: a clear stop. The MCP server's recommendation
notes it instead of refusing.
- tools/test_setup_oldcpu.py.
Default build: nothing changes (no new CMake definitions, the probes answer as before without the env).
The two IQ3 gate/up formats decode each 32-value half with eight scalar loads
assembled by _mm256_set_epi32. A scratch build that substitutes a constant for
the grid vector shows that assembly is ~half the single-core kernel (IQ3_XXS
248 -> 130 us, IQ3_S 357 -> 140 us), and the two formats are ~58% of the
gate/up lookup work on the Swift 1.5 IQ3_XXS pack.
Add an opt-in AVX2 gather for IQ3_XXS (iq3xxs_grid) and IQ3_S (iq3s_grid),
the mirror of the AVX-512 kernels' STRATA_IQ_GATHER. On the i7-12850HX
(Alder Lake, no AVX-512) it is bit-identical - a direct scalar-vs-gather output
dump matches for all five gate/up formats, native_expert_parity stays green -
and 1.32-1.34x on the two formats isolated, +1.0% serve end to end (on >= off in
5 of 5 interleaved pairs). The pool is flat in the deterministic --stats (the
15-worker pool is not decode-bound), so it is a per-core win that compounds.
Default off, like the AVX-512 one: AVX2 gather throughput is microarchitecture-
dependent. IQ2_XXS / IQ2_XS / IQ2_S use 64-bit grids, where an AVX2 i64 gather
measured a wash, and are unchanged. The AVX-512 path is untouched.
Without a hugetlb pool the Linux arena fell back to a plain anonymous
mapping with 4 KB pages. The CPU expert pool streams whole experts (about
3 MB each for the 4-5 bit GGUFs) out of that arena, so every expert costs
~750 TLB misses.
Map the fallback 2 MiB-aligned and madvise(MADV_HUGEPAGE), so THP in
"madvise" mode (the common distro default) backs it. STRATA_NO_LARGEPAGES=1
keeps 4 KB pages, as on the hugetlb path; the log note says which one it got.
A prompt that fits one chunk runs a layer split's stages one after the other,
so while one card reads its layers the other idles. --peer-device already has
the mechanism for this (PeerPrefill's peer streaming: a share of the experts
the primary streams goes over the peer's own PCIe link and is computed there),
but it needs P2P and is refused with --layer-split.
This makes PeerPrefill usable as a stream-only peer between the stages:
- set_stage_helper: each stage's prompt path gets the next stage's (the last
one the first's); a stream-only PeerPrefill holds no experts (peer == null),
so every use of the peer tier is guarded
- per prompt, bind_stage_helper points it at the idle stage's own lent prompt
buffers (mixed, Xq, GU, H, Hq, Dm, ring, MMQ context): no VRAM of its own
- without P2P the activations, the row table, the group bounds and the result
rows go through mapped host memory, read by copy kernels on the device that
needs them. Copy-engine uploads queued behind the ring's expert blobs (the
grouping tables' reason for mapped memory), which cost ~20 ms per layer and
ate the whole gain; cudaMemcpyPeerAsync without P2P is staged by the driver
- only for one-chunk prompts on the MMQ path (not the fused kernels, not with
--peer-device), with a share that falls with the chunk: min(0.45,
0.5 - T/16384), off below 0.3 (~3.3K tokens) - measured, below
- STRATA_PREFILL_HELP=0 turns it off, STRATA_PREFILL_HELP_FRAC fixes the share
2x RTX 3090 (no P2P), UD-Q4_K_XL, layer split auto, 262K context,
--adapt-swaps 0, prompt tok/s without / with: 1.1K 496 / 723, 1.5K 674 / 833,
2K 878 / 1128, 2.5K 1017 / 1296, 3K 1260 / 1440, 4K 1605 / 1607, 8K 2215 /
2221; decode unchanged. The output is repeatable (same prompt, same answer)
but not bit-identical to STRATA_PREFILL_HELP=0's: the helped rows are
computed in another MMQ grouping.
0.1.39b (Niko1221): opt-in (STRATA_PREFILL_HELP=1), since the helped rows round differently from the default;
merged with #598 (run_impl / next_run_).
Add default-off cache coordination, helper-side weighted reduction and CPU activation quantization on demand. Allow helper cache capacities to be sized automatically at startup.
Port the saved v2.2 implementation onto upstream 0.1.38 without changing the separate peer tier. Document the helper-only scope and current validation limits.
The Coder's router is 256 experts, which the native 512 x 10 router does not take, so a verify window routed each
token with its own projection and top-k. moe_route_window does the window in one multi-column BF16 projection
(bf16_gemv_fp32_mmvf_cols over the existing multi-row kernel) and one n-token top-k; each token's route is bitwise
what moe_route gives it (route_window_parity, new ctest). gfx906 build only, so the CUDA and RDNA paths are
unchanged; STRATA_ROUTE_PER_TOKEN=1 restores the per-token calls.
2x MI50, Coder IQ1_M, 32K, greedy 1,500-token request: verify window 55.5 -> 52.5 ms, generated text identical.
- native_expert_grouped: the AMD layouts handle the IQ packs' gate/up and down types only; Unsloth's Q4_K / Q5_K
gate/up and Q5_1 / Q8_0 down now take the CUDA layout instead of exiting (native_grouped_parity found it).
- fused_gr_read_multi: the K-split gr_down (fixed-order finish) sums in another order than the single-token
fused_gr_read, so it broke gr_parity's multi == single contract; it is now opt-in (STRATA_GR_SPLIT=1). The
latency-hidden norm/up (same sums, same order) run in the default read (STRATA_GR_FAST=0: off).
- mmvq_multi_parity: on gfx906 the wave64 MMVQ is the same exact kernel for every column count, so the negative
control has nothing to find; a second ctest entry runs it on the CUDA layout (STRATA_MMVQ_WAVE=0), where it has
power.
- kv_hybrid_parity: the tensor-core prompt attention refuses on gfx906 as on the RDNA backend.
ctest on 2x MI50 (gfx906): 34 of 34 pass (iq_parity and ple_parity need fixture files and are not counted).
The HIP backend (STRATA_ENABLE_HIP) is wave32 RDNA only, and the engine refuses a wave64 card. This adds a
separate opt-in build for gfx906: the CUDA sources compiled as HIP through a small compat layer, with a CUDA warp
as a logical half of the 64-lane wavefront. With the option off nothing changes for CUDA or the RDNA backend;
every new code path is under STRATA_HIP_GFX906.
Build: -DSTRATA_HIP_GFX906=ON -DCMAKE_HIP_ARCHITECTURES=gfx906 (implies STRATA_ENABLE_CUDA's targets, refuses
STRATA_ENABLE_HIP). The CUDA runtime/cuBLAS names come from include/strata/platform/hip_compat/ (first on the
include path), CUDA::cudart / CUDA::cublas become aliases of hip::host / roc::hipblas, every .cu (and the ggml MMQ
sources, with GGML_USE_HIP) is compiled as HIP. The native kernels' --use_fast_math (an nvcc flag) is not passed.
Correctness fixes found on the way (each one changed results or hung, not only speed):
- __fmul_rn / __fadd_rn / __fsub_rn contracted into FMA under HIP; they no longer do (as on NVIDIA) - the
elementwise and quantize_act parity tests pass bitwise;
- the device check of the gfx906 build reads the card's gcnArchName (the wave32 check is the RDNA backend's);
- ggml MMQ got NVIDIA compute capability 9.0 for gfx906 and hung the prompt path; it gets the AMD arch code;
- __byte_perm is a scratch-memory array in HIP: the IQ4 table lookup uses llama.cpp's v_perm_b32 sequence
(agent-request decode 29.9 -> 39.2 tok/s on 2x MI50);
- the stage profiler's wall clock runs at 25 MHz on gfx906, not 100 MHz;
- a CUDA toolkit-mismatch check that reads CUDART_VERSION is skipped (no CUDART_VERSION under HIP);
- gdn_rec_parity (cp.async, CUDA occupancy API) is not built for gfx906.
wave64 kernels (gfx906 only; the CUDA layout stays one env switch away for A/B, every one bitwise-checked):
- grouped native experts, STRATA_EXP_MODE=7 (default): gfx906 has no packed byte SIMD, so __vcmpne4/__vsub4
became ~12 word ops each; signs are now dp4a(g ^ m, u) - dp4a(m, u) (exact in integers), grids and the group's
activations sit in LDS, one weight load serves every token of the group. Same k-to-lane order, so the output is
bitwise the old layout's (native_expert_bench, real GGUF rows). Verify window 60.0 -> 51.3 ms.
STRATA_EXP_MODE=2 is the previous layout;
- MMVQ: one wavefront per row with a 64-lane butterfly, no LDS/barrier, same kernel for 1..8 columns
(+2.3% decode; STRATA_MMVQ_WAVE=0: CUDA layout);
- router top-10 in one wavefront (register argmax, no block barriers), bit-identical (router_top10_parity);
- gr_down_multi split along K (8 rows x 5 slices of 2048 per block, fixed-order finish) (+2.9%;
STRATA_GR_SPLIT=0: old);
- hyper-connection read: norm with its loads in flight and xn in registers, up with 8 lanes per row
(51.3 -> 50.2 ms per window; STRATA_GR_FAST=0: old);
- the QSA prompt-attention tensor-core/WMMA kernels are not used on gfx906 (the FP32 kernel runs).
Benches (not ctest; they need a model shard): native_expert_bench (one expert layout against another, bitwise,
with timings) and fused_gr_bench (the fast hyper-connection kernels against the old ones).
The routing-aware prefetch's router dot (bf16_rows_dot_multi) lives in kq_avx2.cpp, so an AVX-only
CPU started fine and died on the first request (SIGILL in vpmovzxwd on a Xeon E5-2687W). It now
takes the AVX2 kernel where the CPU has AVX2, kq_avx1.cpp's AVX dot (mul + add, 128-bit integer
widening) where it has AVX, and the same sums in plain C++ below that. cpu_avx1_ok() probes AVX +
OSXSAVE + the YMM state. An estimate only (which experts to prefetch). A CPU with AVX2 runs exactly
what it ran before.
From rwkeyes' Strata_Dirigo fork (MIT; 0d00a66, e0754d3), with MSVC's /arch:AVX added and
__builtin_memcpy replaced by std::memcpy for MSVC.
#540's attention kernel (bit-exact, +7% prompt on a V100) uses about twice the registers and also runs decode, so it
is not taken for every card: the kernel the ready-made engine runs is restored as it was, and #540's version lives
beside it as attn_chunk_kernel_pre75, compiled only with -DSTRATA_EXPERIMENTAL_SM60=ON and launched on a card below
sm_75 (per device; STRATA_ATTN_PRE75=0 is the A/B arm). The launches pass cap/stride explicitly (the same 0, 0 as
the defaults) so either kernel can be picked through one pointer.
Conflicts with 0.1.39: --trim-stage-weights is now the same switch as STRATA_STAGE_TRIM=1 (PR #639's
implementation, already in 0.1.39); the #533 VRAM command is taken between batch windows only when no slot
is decoding.
STRATA_EXPERIMENTAL_SM60 now reaches every target (add_compile_definitions). prompt_attn_v70_kernel with its launcher,
and Gemm::bf16's fp32 path for Pascal, exist only under it: the ready-made engine (sm_75+) loads no extra kernels for
them, while the experimental build's sm_60/61/70 cards get them.
A native pack's Q2_0 layers (most IQ packs' down projections) ran on Strata's own Q2_0 kernels, which
are AVX2 at least. q2_native_kernels() keeps them there when the CPU has AVX2 and otherwise hands the
layer to ggml-cpu's Q2_0 vec_dot like every other native type (the pool, the dispatcher's activation
quantizer and the parity test). STRATA_KQ256's gate/up kernel needs AVX2 too. native_expert_parity
skips the AVX2 checks on a CPU without AVX2. No change on a CPU with AVX2.
(The engine part of #595; its setup and run-script parts are not in this commit.)
Old path against new kernel on 8.5K / 33K / 99K-token chats, 617 scored positions each, with two controls that change only the rounding
order (BF16 GEMMs without the FP16 detour, another prefill chunk), reruns of both paths (bit-identical), and the kernel's own error on the
model's data (relative 4.4e-7 mean, 4.8e-6 worst row over ~5 M rows).
The experimental sm_70 build (STRATA_EXPERIMENTAL_SM60, #236) had no page. NVIDIA_V100.md says how to build it, which kernel each
part of the prompt path uses on Volta and what the prompt attention kernel was measured at.
bench/results/2026-10-03-v100-prompt-attn: prompt speed with and without the kernel (7K / 28K / 114K tokens), the nsys share of the
attention before and after, the parity numbers, needle recall (15 of 15 with either path) and the greedy-output comparison.
On sm_70 qsa_prompt_attn_batch refused the device and the prompt path fell
back to the decode kernel (FP32 FMAs, one query at a time): 19-25% of the GPU
time of a 32K prompt on a V100.
prompt_attn_v70_kernel is the v1 kernel's structure with the two matrix
products on m8n8k4 (FP16 in, FP32 accumulate): q.k with one dim group (64 dims,
= one int8 scale group) per warp and 8 cells per quad-pair, p.v with 16 dims per
quad-pair. q and p keep their hi + lo FP16 split, the four dim groups' partials
are added in a fixed order (deterministic). Int8, FP16 and K8V4 KV are handled;
Q4_0 KV (mode 4, sm_80+ only upstream) keeps the old kernel. The fragment layout
was measured on a V100 and matches turbomind's SM70_MMA_884.
qsa_prompt_attn_parity (V100): error against FP64 ~5e-6 (output scale ~3.6),
int8 3.0x and fp16 1.3x faster than the FP32 kernel per 2048-query chunk. The
test now skips its Q4_0 cases below sm_80 instead of exiting 2 there.
End to end, 32K prompt, one V100, greedy output identical:
26 s -> 22 s on current main (STRATA_PROMPT_ATTN_OLD=1 is the old path).
PR #655's FP16 tensor-core path for Gemm::bf16, adapted so the ready-made engine stays byte-identical:
- default on for compute capability 7.0-7.4 only (Volta; the experimental STRATA_EXPERIMENTAL_SM60 build), off for
7.5 (RTX 20): STRATA_BF16_TC=1 opts in, =0 turns it off on any 7.x card
- from #540: finite values beyond FP16's range clamp to +-65504; a beta = 1 product (the BF16X2 remainder, in
FP16's subnormal band) keeps cuBLAS; the compute capability is cached per device (a layer split can mix cards)
- from #395: Pascal (6.x; cuBLAS has no BF16 GEMM there) widens both operands to fp32 by an exact shift and runs
cublasSgemm with the same fp32 accumulator, tiled over T in the instance's own buffers
sm_80 and newer, and HIP, run the same cuBLAS / hipBLAS call as before.
Co-Authored-By: Károly Sárdi <[email protected]>
Co-Authored-By: Rafal R@V Prasal <[email protected]>
native_gu_rows() tested cpu_avx512_ok() before the AVX-512 kernel but not cpu_avx2_ok() before the
AVX2 one, and native_down_rows() had the same asymmetry for its two multi-token kernels (kq256 and
the IQ4_NL one). On a CPU without AVX2 a verify window of two or more tokens took an /arch:AVX2 path
and died with 0xC000001D. With the gates it falls through to ggml-cpu's vec_dot, which covers every
native type for any nt. A CPU with AVX2 takes exactly the paths it took before.
(Cherry-picked from #394's 0083bb7 without its generate.cpp part; the startup gate is reworked for the
opt-in STRATA_ISA_FLOOR build in a later commit.)
#391 removed MSVC's BMI2 shlx from the table's constructor; GCC still vectorised the same loop into
AVX2 (vpbroadcastb) and ran it before main() on an AVX-only Xeon E5 (found by rwkeyes' Strata_Dirigo
fork, 6dd37a2, which made it constexpr). constexpr here as well, deriving ksigns_iq2xs[i] (the 7 bits
plus an even-parity bit) so the constant expression does not read a non-constexpr array. Same 128
values; the file now has no runtime initializer.
A namespace-scope static initializer shifted a byte into place with << (8 * k), which is a
variable shift. MSVC compiling this file with /arch:AVX2 emits BMI2 shlx for that - it assumes
anything with AVX2 also has BMI2, true from Haswell on and false of everything before it.
Because the table is static, the constructor runs before main(), so this faulted at PROCESS START:
strata.exe exited 0xC000001D with no output at all, not even for --help. One shlx in the whole
linked binary was enough to make the engine unstartable on any pre-Haswell CPU under MSVC.
Storing the byte directly is the same 64-bit value on the little-endian targets this file is for,
and leaves no variable shift for the compiler to widen.
Found by building the engine for sm_61 on a Sandy Bridge-E (Xeon E5-2680) with
STRATA_EXPERIMENTAL_SM60=ON, which 0.1.31 added (#236). Verified by disassembly: no shlx/shrx/sarx/
mulx/andn/bzhi/pext/pdep anywhere in strata.exe afterwards, and --help now runs.
This is independent of any AVX1 work - it is a plain startup crash on the CPU this was built on.
(cherry picked from commit 3fcaf03840)
Volta has FP16 tensor cores and no BF16 ones as well, so cuBLAS takes the same SIMT path for BF16 products there.
Suggested in review. Not measured on sm_70; STRATA_BF16_TC=0 turns it off.
Turing has no BF16 tensor cores, so cublasGemmEx on CUDA_R_16BF inputs falls back to a SIMT fp32 kernel
(magma_sgemmEx). On an RTX 2080 Ti that was the largest kernel of the prompt path, 20 % of a 4K prompt:
the hyper-connection down / up projections, the router and indexer rows, PLE key / value.
Gemm::bf16 now converts the weight and the activations (in row slices, into the instance's own scratch)
BF16 -> FP16, which is exact for every value inside FP16's range, and runs the existing FP16 GEMM (fp32
accumulate) on the tensor cores. Single-row products keep cuBLAS's GEMV. STRATA_BF16_TC=0/1 overrides;
the default is on for sm_75 only, so other cards run exactly what they ran.
New test gemm_bf16_parity: Gemm::bf16 against cuBLAS's BF16 product over the prompt path's shapes,
within fp32 rounding (worst relative difference 1.7e-5 on sm_75; 0 with the path off).
Qwen3.8-Flash-Next Q2_0, RTX 2080 Ti, i7-6700: prefill 877 / 1134 / 1153 / 1116 / 1009 t/s at
4K / 16K / 32K / 64K / 128K -> 1020 / 1337 / 1366 / 1312 / 1166.
It hung on the B70 until the stager/PLE event-wait fixes; it runs since and reads
Coder prompts faster (40K 1,106 -> 1,160, 128K 897 -> 920, 128K k8v4 1,006 ->
1,041, 256K 760 -> 780 tok/s; same output). It copies every non-resident expert,
so with a big host tier it loses (IQ2_XS, a quarter in the RAM mirror: 2,184
tokens ~560 -> 254 tok/s): generate.cpp hands the non-resident share to the
prompt path, which walks only below 10%. STRATA_PREFILL_STREAM_ALL=1/0 forces it.
Docs: the 40K prompt breakdown, and why the cluster kernels and Q4_0 XMX prompt
attention are not worth porting here; the 2K "regression" was session variance
(old and new builds A/B'd: 784-799 tok/s both).
Tested on the B70: Coder 2,184-token and 40K, IQ2_XS 19- and 2,184-token
outputs identical; IQ2_XS 562 tok/s (stream-all off by its share), Coder 40K
1,160 tok/s.
On CUDA the register kernel holds 33,792 blocks (~135K cells). Past it qsa_block_topk took the original kernel:
256 threads, each reading its own stretch of the scores on all six passes, one shared histogram updated with
atomics. The dispatch follows the capacity (--max-context), so with a context past ~135K that was every prompt
batch and every decode window on a card without the sm_75 active-bound dispatch (#512), whatever the prompt's
length, and on sm_75 every prompt batch past 33,792 blocks and every decode window.
block_topk_wide_kernel is the register kernel's layout (1,024 threads, a histogram per warp, warp scans) with the
keys read from memory:
- the four radix passes: thread t reads blocks t, t + 1024, ... - a histogram has no order, so a warp reads 32
neighbours at a time;
- the count and the output: warp w holds `per` rows of 32 consecutive blocks, and the cells are written in the
order (warp, row, lane); a row without a selected block is skipped.
The selection rule is the original kernel's (radix threshold, ties to the lowest index, cells ascending): identical
ids. One launch per call, no scratch memory, no capacity limit.
Dispatch (CUDA; HIP is unchanged and does not compile the kernel): where the blocks do not fit the registers the
new kernel, except a call that names fewer than 4,608 blocks (a prompt batch below ~18K cells), which keeps the
original kernel: its fixed cost is lower there. A decode window (no block count) takes the new kernel at any
context. The register kernel's range, the sm_75 active-bound dispatch and the sm_90+ cluster path are unchanged.
STRATA_TOPK_OLD=1 keeps the original kernel, as before.
qsa_topk_parity (new, with the kernel parity tests): the dispatched top-k against qsa_block_topk_ref on both sides
of the register kernel's reach, at short contexts under a long capacity and up to 524,288 cells, called as decode
and as the prompt path call it, with continuous scores, 64 score levels and equal scores.
RTX 3060 12 GB (sm_86), 256 queries with their block count under a 262,144-cell capacity (524,288 for the last
two), original -> dispatched:
16K cells 0.199 -> 0.199 ms (the original kernel)
32K 0.845 -> 0.328
131K 4.70 -> 0.97
262K 16.09 -> 1.71
393K 26.02 -> 2.49
524K 37.53 -> 3.24
4 queries without a count (a decode window): 0.031 -> 0.028 ms at 8K, 0.176 -> 0.083 at 131K, 0.309 -> 0.127 at
262K.
IQ3_XXS on that card (--resident-experts, --max-context 262144, --kv int8 --kv-resident 32768),
tools/needle_bench.py, STRATA_TOPK_OLD=1 against the default: tok/s of the prompt, and the prompt path's selection
phase (STRATA_PREFILL_TIMING):
16,471 tokens 1004.5, 1016.2 -> 989.0, 1013.6 195 -> 195 ms
31,970 1043.3, 1055.5 -> 1060.1, 1060.5 900 -> 731
61,333 932.9 -> 1013.8 4,061 -> 2,428
122,755 904.7, 904.2 -> 992.7, 989.6 20,019 -> 9,034
185,729 819.7 -> 954.7 50,055 -> 20,761
243,176 743.3 -> 933.7 96,326 -> 35,544
The needles are found in every run.
3-way merge from the port's "main + PRs" migration to dpct of upstream 0.1.38
(prefill.cpp re-merged with the histogram diff: upstream's peer-device and fused
restructure kept, the port's edits carried over - the stager's GGUF job, the
stream-all gate, the fp16 NaN checks; generate.cpp: the pipelined profile fill
beside upstream's prefetching serial fill; mtp/verify: the SYCL host mapping).
- the PRs' header forks dropped: upstream has them now
- stubs: --peer-device (peer_experts.cpp; open refuses) and the fused int8 prompt
kernels (moe_fused_stub.cpp; MMQ is not built here)
- off on SYCL: the sm_90 cluster greedy sampler and QSA top-k (fixups.py)
- fused_gr: dpct's undeduced kernel names filled in (fixups.py)
- STRATA_GDN_KEYHEAD stays opt-in on the port (upstream's gate is NVIDIA SM counts)
- version 0.1.38
Tested on the B70: build clean; native_grouped, iq_multi, gr, qsa, kv_hybrid,
quantize_act parity pass, qsa_prompt_attn's int8/fp16 pass (its new Q4_0
tensor-core cases are refused: not in the port); Coder short/long, IQ2_XS, 40K
outputs identical; 128K int8 KV streaming 892 / 66.8 tok/s; GR_DOWN_MAX4 neutral,
GDN_KEYHEAD 8% slower (stays off).
The trim was on by default in the HIP builds. It now waits, as on CUDA, for the PR author's re-run of the release
branch on his cards; the default flips after that. --layer-split auto and one GPU were never affected.
"saving a conversation checkpoint failed" was the only text when the checkpoint's
device sync failed - a GPU hang that Windows then resets (VIDEO_ENGINE_TIMEOUT), not a
checkpoint problem. The error now names it: "...: the GPU stopped responding before it
(<error>) - a GPU hang; on Windows the driver is then reset". Error text only.
Two private engines, the same conversation: the baseline keeps it live, the candidate
parks it (an unrelated request in between, a re-rendering client, or two long
conversations taking turns) and restores it. Each turn's greedy tokens must match and
its decode speed must be within --tolerance (15%) of the live turn's. Pure logic
tested without a GPU (tools/test_conversation_restore_speed.py).
#528 did not reproduce on the RTX 5070 with it: Q2_0 37K tokens (--kv-resident 16384)
and IQ3_XXS 50-75K tokens (--kv-resident 32768) decoded at the live turns' speed after
a restore, with the same tokens.
- The per-layer wait for the router ids names itself in a stall report ("waiting for
the GPU (attention, router) at layer"): a GPU stall reads apart from a host one.
- STRATA_PF_STEP_SYNC=1 (opt-in): the compute and copy streams are waited for after the
K/V staging, the K/V append and each layer; a step over 250 ms is logged with its
layer and chunk, and a stall names the step.
- STRATA_KV_HOST_DMA=1 (HIP only, opt-in): with KV streaming the prompt path's append
writes the staging pool and the resident slots only, and one DMA copies the chunk's
blocks to the host copy - no kernel writes host memory over PCIe. Same bytes for
every reader (the staged first block is complete; past the chunk nothing is read
before a later append writes it).
Q2_0 on the 5070: the same tokens with STRATA_PF_STEP_SYNC=1 as without (and as 0.1.38).
STRATA_VERIFY_TRACE=1 records each verify window's handshake on the host (window
start, launch, each layer's ring and service, the final sync, a timeout, the release
and how long the GPU took to drain after it) with the seq / flag / A / B words, and
GPU breadcrumbs: the profiler's stage stamps per layer and token group, written to
mapped memory so they read while the GPU hangs. A timeout prints them before and
after the release; the serve watchdog's stall report (Verifier::diag) prints them too.
HIP only, opt-in, for the reporter's A/B: STRATA_VERIFY_COHERENT=1 allocates the
handshake words and rows with hipHostMallocCoherent; STRATA_DOORBELL_STORE=1 has the
GPU store each step's ring (known at capture) instead of read-modify-writing it over
PCIe. CUDA builds and the default path are unchanged (Q2_0 on the 5070: the same 96
tokens with and without the trace as 0.1.38; a forced stall dumps the breadcrumbs up
to the withheld flag).
docs/AMD_HIP.md's gfx1030 report measured an i7-13700KF (8P + 8E) at 38-42 tok/s with
all 15 workers against 36 with 8, so a CPU with no more E-cores than P-cores keeps the
engine's own count; the E-heavy chips the fork measured (i9-14900KF: 15 of 23 faster)
get the recommendation.
--vram-elastic (the config's "vram_elastic": true) allocates the expert cache as one
reserved address range backed by physical segments (CUDA virtual memory management,
512 MiB each, --vram-segment-mib), so every slot keeps its address. Between
requests, the engine command `VRAM <reserve_mib>` gives the last segments back to
the driver until that much VRAM is free (their experts become CPU misses, the prompt
path's loan moves down and its chunk shrinks to fit), or maps them again and puts
the same experts back when more is free. `VRAM` alone: the start's reserve. The
server's POST /v1/vram {"reserve_mib": N | null} sends it after a running request,
keeps it for an engine that is unloaded, and /v1/status shows the cache's size.
Never automatic. One NVIDIA GPU only (no layer split, helper caches, --peer-device
or the resident low-RAM mode); HIP refuses the flag.
RTX 5070, Q2_0 (4.8 GiB cache): reserve 6000 -> 4.3 GiB freed in 78 ms, decode
44 -> 33 tok/s; grown back in 92 ms, the answers token for token the ones before.
Without the flag the cache is the one cudaMalloc it was (same answers with and
without the flag until a VRAM command).
IQ3_XXS on the RTX 5070, temperature 0, 256 tokens, 8 French prompts (topics outside the corpus), 2 English and 2 code
prompts, arms alternating, 2 passes each: drafts accepted in French 0.505 (en) / 0.507 (cjk) -> 0.597 (fr), higher on
all 8 prompts and in each pass; English 0.610 -> 0.629, code 0.772 -> 0.779. Decode speed was not measurable here
(builds ran beside it); the reporter measured 141 -> 158 tok/s in French on an RTX 5090.
UD-IQ4_XS sits between IQ3_S and UD-Q4_K_XL: IQ3_S gate/up experts (IQ4_XS in one layer) with IQ4_NL downs (Q8_0 in
five layers), 59.5 GB of them, and UD-Q4_K_XL's dense side. The engine's kernels already take these formats; it was
packed (iq_pack.py --compat-bf16) and run on the Strix Halo, not yet on NVIDIA.
- MODELS["UD-IQ4_XS"] in the unsloth family: three shards pinned at the same revision 38bb39e with their sizes and
SHA-256 (the Hub's LFS pointers), the RAM budget like UD-Q4_K_XL (budget, one GPU, no images, --compat-bf16 pack,
no experts.bin), engine 0.1.38 or newer (the fused-layout fix for packs not every layer of which is covered).
- A model can have its own shard pattern and count (model_file / model_shards); budget_model() tells which Unsloth
model a config runs, for the #498 multi-GPU messages and RAM need.
- The AMD question (#429) stays for UD-Q4_K_XL only (nvidia_only): UD-IQ4_XS's formats have AMD prompt kernels.
- --gguf-dir accepts UD-IQ4_XS files (#444's list says so).
- UNSLOTH_Q4.md: its files, what setup does, what is known (Strix Halo), and the BF16 rounding of the hyper-connection
projections that every --compat-bf16 pack has (not changed here). MODELS.md: a paragraph.
- test_setup_unsloth: pins, budget, a mocked install, the engine check, no AMD question, the config read back.
UD-Q4_K_XL's install, its tests and every other model's config are unchanged.
Stateless, on the chat path: a Responses request becomes the chat template's
messages, tools and kwargs, runs through Service.run, and its reasoning, text
and tool calls come back as output items and the official typed SSE events
(response.created ... response.completed / incomplete / failed).
- input as a string or items (messages, reasoning, function_call,
function_call_output, custom tool calls); instructions and leading developer
messages make one system message; tool results are put in call order
- function, namespace (Codex's MCP/agent tools) and custom tools; hosted tools
are left out with a log line
- reasoning.effort -> reasoning_effort, max_output_tokens, text.format
json_schema/json_object through the structured output path
- reasoning returned as reasoning_text (and as opaque encrypted_content when
asked) and replayed into the prompt, so Codex's next turn reuses the cache
- errors in the Responses format; same API key, Host and Origin checks as /v1
- a response.in_progress event every 15 s while a long prompt is read
(Codex's idle timeout ignores SSE comments)
- docs: DETAILS.md section with a Codex config.toml
Based on #630 by CC-David-CC (the event assembler, namespace tools, call-id
ordering of results), without its opt-in flag, replay key and crypto
dependency, strict-function and JSON-format refusals, or the second
summary generation pass. Tested with Codex CLI 0.160.0 and openai 2.15.0.
Co-authored-by: David <[email protected]>
The About tab's Model settings card reads GET /config and saves through
POST /config: the sampling defaults, reasoning_budget_tokens,
fit_max_tokens, anthropic_thinking, effort_position, aliases,
idle_unload_s, lazy_load, engine_silence_s, api_monitor, open_browser
and --vram-reserve-mib (serve/runconfig.py). Every value is checked
first (one bad value changes nothing), only the named keys change, every
other key of strata-<model>.json stays as it is (as setup run again
keeps them, #629), the earlier file is kept as .bak, and the change is
used from the next start. POST takes JSON from Strata's own page only,
like /settings, and the API key when one is set; the network, key, MCP
and program keys are not editable there.
- MODELS.md: the RAM rule is the experts + ~10 GB (setup's arena_gb), not shard 1 + 10 GB; the sizes per model.
- DETAILS.md: prompts are read in chunks of up to 8,192 tokens (32,768 opt-in), not 2,048; Q2_0's AVX-512 pack is
~40 GB on disk = 34 GB of experts plus the dense weights.
- README: the NVIDIA speed table says which engine measured which row (Q2_0 0.1.36, the others 0.1.26).
- expert_cache.hpp: the comment that the cache "does not yet compute anything" is gone.
Private reports through GitHub's private vulnerability reporting; the
server's protections as of 0.1.38 (127.0.0.1 by default, the API key,
the Host and Origin checks, CORS, MCP tools and the request monitor
opt-in), with DETAILS.md for every setting. README links it.
/metrics gains "conversation_cache": the parked conversations, their
bytes, parks/restores/evictions and the last switch, read from the
engine's own log lines ("conversation cache: parked/restored ...") from
the start of its current run on; the budget and slots from its INFO line;
and the prompt tokens reused in the last request and since the start.
The card shows the parked slots and memory when --conversation-cache-mib
is on, the reuse lines always, and how to turn parking on when it is off.
--gpu-stages replays per-layer graphs, which a native (IQ) pack does not capture, and refused with "not captured
with the split". The serve path already times a native window's stages: STRATA_DECODE_TIMING=1 prints the
per-request decode split and STRATA_VERIFY_PROFILE=1 adds the GPU stages. DETAILS.md now says how to read them, and
--gpu-stages on a native pack says to use them.
The decode hit rate is the VRAM share of the experts the pool looked up; the routed experts the GPU reads over PCIe
(--pcie-frac) or another GPU computes are in neither count, so raising --pcie-frac raised the "hit rate" while the
decode got slower. Its meaning stays; now that share is said beside it:
- engine: DriveState::offload_entries counts those entries; the serve line adds "; N more read by the GPU over
PCIe or from another GPU (X% of all M routed)", and the DONE line a 16th field (an older server ignores it).
- server: /metrics' requests get "pcie_share" (None from an older engine), the console's done line "(+X% of the
routed experts over PCIe)", and the Monitor's column is "VRAM hit rate" with "+X% PCIe" after it.
- DETAILS.md says what each means.
No number changes; tokens are unchanged.
CMake links CUDA code with cudart_static by default and the targets also link CUDA::cudart (cudart.lib); MSVC with
CUDA 12.x (the STRATA_EXPERIMENTAL_SM60 build for sm_60/sm_70) then fails with LNK1169. That build (MSVC +
STRATA_EXPERIMENTAL_SM60, no CMAKE_CUDA_RUNTIME_LIBRARY given) now sets CMAKE_CUDA_RUNTIME_LIBRARY Shared, the
reporter's verified fix. The ready-made engine and every other build are unchanged. INSTALL.md: where github.com
cannot be reached, third_party/llama.cpp at the pinned commit can be put there by hand.
The effort's instruction is the first thing the template renders, so a
request that only changes reasoning_effort (or turns thinking off) read
the whole prompt again. With "effort_position": "end" in the run config
the server renders every prompt's start as the default effort's and puts
a low/medium effort in a short system turn right before the answer
(thinking off: just the template's empty thinking block); the rendering
is template-agnostic (history + turn + the generation prompt).
Engine: --tail-role-token ID (default -1, off) moves the turn checkpoint
in front of such a trailing turn, so it holds the conversation and not the
effort text; INFO reports tail_role_token. The server passes the id of
"system" only when the engine binary knows the flag, and says so when it
does not. Without the key nothing changes: the same prompt ids, no flag.
The first mapped start writes the whole arena (25 GB for the Coder) next to the pack. A drive without that much free
space filled up before the write failed. Now the start checks the free space first (the arena plus 2 GiB to spare);
without it, it says so and keeps the arena in RAM for this start. Linux opt-in only; nothing changes without the env.
The trim was measured on 2x MI50 (gfx906) only; nobody has run it on NVIDIA multi-GPU, where CUDA-only paths could
still read another stage's dense weights on a card that no longer holds them. The HIP builds trim with explicit split
points as the PR does (STRATA_STAGE_TRIM=0 keeps the full copies); the CUDA build keeps the full copies unless
STRATA_STAGE_TRIM=1. --layer-split auto and one GPU are unchanged everywhere.
find_nvcc takes the newest toolkit it finds, so with CUDA 12.8 and 12.9 installed, 12.9 was always used - and 12.9's
math headers do not compile with glibc 2.43 (Ubuntu 26.04). STRATA_NVCC=<nvcc> is now the only one considered (a
warning when the file is missing, or when it is too new for the Pascal/Volta build). CUDA_HOME is a candidate like
CUDA_PATH. TROUBLESHOOTING.md: the two build errors and what to do (g++-14 for gcc > 14; CUDA 12.8 or 13.x for glibc
2.43). Without STRATA_NVCC nothing changes.
The pinned expert arena (the default tier) was allocated without looking at the RAM: on Linux the anonymous mapping
succeeds whatever the host has, and a container whose memory limit is below the arena was killed by the OOM killer
part-way through the load, without a message.
- detail::host_available_memory: the file tier's probe (MemAvailable lowered to the tightest cgroup v2 limit's room),
now with its files as parameters (tested on fake trees), cgroup v1's memory controller (memory.limit_in_bytes -
memory.usage_in_bytes), and no cgroup line at all meaning MemAvailable alone (it was "cannot determine", which
refused a --resident-budget-gib on a v1 host). It also returns the tightest hard limit itself.
- ArenaExpertSource::open, after the size check and before the allocation: a hard cgroup limit below the arena is
refused with both numbers (a certain OOM kill); less RAM available than the arena is a warning with the numbers
and what runs with less (--mmap-experts with --resident-budget-gib), and the load goes on. Not for a shared arena.
- file_expert_source_test: v2 limit / max / malformed / missing, v1 limit / unlimited / not mounted, no cgroup.
On a 7200 RPM disk the default --ple-io direct's random reads of the n-gram table stalled prompts past the 60 s
watchdog, and the stall report looked like any other engine stall.
- Engine (Linux): at start with --ple-io direct, the table's disk is looked up in sysfs (queue/rotational); on a
rotational one it warns and recommends --ple-io ram or an SSD (not on WSL, whose virtual disk says rotational).
- Stall report (Linux): the engine's threads in state D and their wchan ("threads waiting on the disk (state D): 16
of 35 (blk_io_schedule x16)"), with what that means.
- Setup (Linux): a model on a rotational disk gets --ple-io ram when the RAM holds the table beside the model,
otherwise a warning that recommends an SSD. Nothing changes on an SSD.
- DETAILS.md: --ple-io direct is for SSDs (it reads unbuffered, not through the OS cache), and a troubleshooting row.
The tokenizer matches <think> and </think> as their special tokens even
in plain text (type 4, as llama.cpp does), so a user quoting "</think>"
handed the model a real end-of-reasoning token and the answer came back
empty. Before the template is rendered the tags in message text (user,
tool, system, assistant history, tool call arguments, tool descriptions)
are swapped for private-use marks; after it, those spans are encoded as
ordinary text (Tokenizer.encode's new plain= spans), the template's own
markers stay special. An assistant turn that opens with its own
<think>...</think> block keeps that block. A prompt without the tags is
encoded exactly as before; count_tokens uses the same path.
hardware_key, the settings file's key for a saved calibration, read the cards with nvidia-smi's list: on an AMD PC
every card was "?|0GB" (two different cards shared one calibration), and on a PC with an NVIDIA card too a HIP index
named that NVIDIA card or stopped setup ("there is no GPU N"). A HIP config's cards now come from amd_gpus, in HIP's
numbering, with the architecture in the name (an RX 7900 XTX and an RX 7900 XT are both gfx1100; a card without a
product name is known by its arch). Setup reuses a saved calibration on Linux HIP as on NVIDIA (setup_calibration);
Windows HIP keeps the defaults for now. The tuning is still offered on NVIDIA only - a `./setup.sh --calibrate` on
HIP runs it by hand, and each control is verified on HIP before setup offers it.
The NVIDIA key and the settings file's format are unchanged. Tests: tools/test_setup_amd.py CalibrationKey (7).
French answers drafted poorly with the shipped subsets: 23.6% of the token occurrences of French text fall outside
draft_vocab_en.bin, and the draft head cannot propose those (no subset adds the accented words and word pieces).
- tools/draft_vocab.py: a coverage mode. --corpus FILE... adds to the base the most frequent tokens of the corpus that
together cover --coverage (default 0.99) of its occurrences, and prints the share outside the base and the new list.
- data/draft_vocab_fr.bin: draft_vocab_en.bin plus 5,686 tokens (46,211 ids), from 10 French Wikipedia articles
(France, Paris, Napoleon Ier, Langue francaise, Union europeenne, Litterature francaise, Cuisine francaise,
Economie de la France, Revolution francaise, Seconde Guerre mondiale; 1.24 M characters, 326,926 tokens) at 0.99:
python tools/draft_vocab.py --gguf <IQ3_XXS shard 1> --base data/draft_vocab_en.bin --corpus <the articles> \
--coverage 0.99 --out data/draft_vocab_fr.bin
Outside the list on that corpus: 0.74% (23.65% with en).
- setup.py: "fr" in DRAFT_VOCABS / --draft-vocab (~151 MiB head on IQ3_S, between en and cyrillic). The sha256 check
knows it as a shipped subset, so a later --draft-vocab cjk/en replaces it and a subset made by hand stays.
- tests: tools/test_draft_vocab.py (the coverage mode), tools/test_setup_draft_vocab.py (fr replaces a shipped subset
and back, keeps the en ids first in their order, its MiB estimate).
The default subset (cjk) is unchanged.
A broken state (or a model in a loop) can answer one token forever: #606 streamed 36,689 tokens of "!" for four
minutes, because a request without max_tokens gets the whole remaining context. The server now ends such a reply
at 256 identical tokens in a row with finish "length" and a line in its window saying what to do if it repeats on
a new request. "repeat_stop_tokens": N in strata-<model>.json sets the run (0 = off). Nothing changes for a reply
without such a run.
Both q8_1 activation quantizers (iq_kernels.cu q8_1_store, native_mmvq.cu native_quantize_q8_1_kernel) stored the
sum of a block's 32 raw activations as fp16: past 65504 (one massive-activation dimension is enough) it rounded to
inf, and d = amax / 127 did past amax = 8.3M. The dot products then read inf * 0 = NaN, and a NaN in the residual
answers one token forever (ggml-org/llama.cpp#23606 is the same defect).
Now both are clamped to the largest finite half (and the values to the int8 range they then exceed). A block that
was finite before is stored bit for bit as before - values in (65504, 65520) round to 65504 anyway, and a finite
block's |x / d| is at most 127 - so the default output only changes where a block overflowed (and was NaN). A NaN
input stays NaN, so STRATA_DBG_NAN still finds it.
iq_multi_parity checks both quantizers against the 0.1.38 formula on a host transcription: every finite block as
before, and three overflowing blocks (sum 96,000, sum -69,984, amax 2e7) stored finite.
On a hybrid CPU (Intel 12th gen+, AMD Zen 5 + Zen 5c) setup writes --pool-workers
N into the run config: the performance cores but the host loop's one, plus half
of the efficiency cores. An E-core runs the expert kernels ~2.2x slower and each
layer waits for its slowest part; Hardin22 measured 15 workers at 165 / 116 tok/s
against 106 / 84 with all 23 on an i9-14900KF (#642). A recommendation only: the
line is in the config, a saved calibration's measured count wins, and a CPU whose
cores are all alike gets the same config as before.
The engine's pool, with every core allowed (the default affinity), now orders the
P-cores first on a hybrid CPU, so a smaller pool runs on the P-cores and the first
E-cores rather than on whatever the OS numbered first (from Hardin22's fork).
Non-hybrid CPUs: unchanged.
Co-Authored-By: Hardin22 <[email protected]>
Kept in the run config's vision.max_tokens (the server already passes it
to the encoder); a setup run again keeps it for the same encoder device.
More than the default is allowed with a note on the encode time. Without
the flag the config is as before. DETAILS.md: vision.mmproj can point to
a Q8_0 mmproj of this model.
- The draft layer's K/V lives on the last stage's GPU (the drafter is loaded there); its save, ring restore and
verification now run with that stage's image under that device, instead of with stage 0's image on GPU 0, where
the ring restore kernel ran against another GPU's memory (seen in Niko1221/Strata#653, which places it the same way).
- Parking with a layer split on one GPU (--split-device 0) is refused at start: untested.
- The whole-session validate refuses an image with stage images; clearer message for stage parts in live.
- --trim-stage-weights says which layers each GPU loads, and fails when index.txt cannot be read.
- conversation_snapshot_test: the whole-session refusal; parking_test.py runs with STRATA_SNAPSHOT_VERIFY=1.
tools/make_profile.py builds the ranking as the base's order, then the
trace's pairs, then the fill. The base is only appended to, so --base can
never move a pair the base already ranks - and the shipped
data/expert-profile.bin ranks all 24,576 pairs, which makes a trace a
no-op:
wrote ...: 24576 ranked pairs (24576 from the base, 0 from the traces, ...)
So the documented way to seed a profile from --dump-routing (rank the
pairs your workload routes most) silently did nothing with the shipped
base; only --no-base changed anything, and that drops the base's ranking
entirely.
Add --reorder: rank the trace's pairs first (most frequent first), then
let the base fill the rest, then the fill. The default is unchanged. The
ranking moves into a rank_profile() so it can be tested without a model;
tools/test_make_profile.py covers the no-op, --reorder, --no-base, the
frequency/tie order and an end-to-end run over a complete base. On this
host's coding trace the same command now puts 5,808 trace pairs first.
--no-browser (on a start or at setup) saves "open_browser": false in the
model's run config; start() and run-<model>.bat/.sh then leave --open
out, and the server ignores --open when the config says so (a start
script written earlier still has it). --browser undoes it. Without the
flag the config, the command and the script are as before.
The adaptive swap loops called srcp->blob() for every swap to collect the spans pin_blobs locks, on every build -
pin_blobs is a no-op except on Linux AMD with STRATA_ARENA_MMAP=1. blob() is not free: it counts reads and the file
tier (--mmap-experts) may assemble the blob, a second time for the copy. pin_blobs_on() now guards the gathering, so
CUDA and the runs without STRATA_ARENA_MMAP do exactly what 0.1.38 did.
"layer_split" is the first layer of each later GPU ("24,36,42" for 4 GPUs), not a count of layers per card. A
value such as "24,16,12,12" failed the engine's check after the start began, and a JSON list reached the engine as
"[24, 16, 12, 12]".
- server.py layer_split_value: "auto" (default), a comma string or a JSON list of one rising number per GPU after the
first, from 2; anything else stops the start with the format and an example for that many GPUs.
- The engine's message says the same, with the value it got.
- MULTI_GPU.md and setup's --layer-split help say it.
The host expert arena is normally read into pinned (locked) RAM. On a machine whose GPUs already hold most
experts, that locks memory the CPU never reads: with the Coder (25 GB arena) on 2x 16 GB GPUs and 32 GB of RAM,
~1 GB stayed available at 128K context, the kernel swapped, and a multi-turn checkpoint took 25-80 s.
With STRATA_ARENA_MMAP=1 (opt-in, Linux; off by default, nothing changes without it):
- the first start writes the arena to the pack's experts.bin (padded by one blob so a whole-slot copy may start at
any expert); later starts map it read-only, not locked, not pinned. The page cache keeps what the CPU pool and
the prompt path read and gives the rest back under pressure; a dropped page is re-read from the file, so no
result depends on what is resident. The GPUs get no mapped alias of it: run with --pcie-frac 0;
- after the VRAM caches are filled, the pages of experts a GPU holds are handed back (MADV_DONTNEED,
STRATA_ARENA_RELEASE=0 keeps them) and the rest are prefetched (MADV_WILLNEED); an adaptive swap releases the
expert that went to VRAM and prefetches the one that came back;
- HIP: ROCclr copies a pageable source of 1 MiB or more by locking its pages in place and keeps them locked, which
undid every release. The engine raises GPU_PINNED_MIN_XFER_SIZE (an explicit setting wins) so such copies
stage through ROCclr's own buffers, and locks an adaptive swap's source pages itself only until the copy lands
(the page ranges are merged first: a copy from a range only part of which is registered fails);
- STRATA_RSS_TRACE=1 prints the process's RssFile at the startup steps (madvise reports success on locked pages
and frees nothing; RssFile is the honest check).
The existing --mmap-experts refuses native packs (the Coder); this path is for them.
Measured on 2x AMD Instinct MI50 16 GB + 32 GB DDR4 (gfx906 port, Coder IQ1_M, 128K context, layer split):
RAM available while serving ~1 GB -> 25 GB; multi-turn re-read of a checkpoint 25-80 s -> <= 3.2 s; decode
unchanged (49.06 vs 49.16 ms per verify window). Not measured on NVIDIA (pin_blobs is a no-op there).
The file tier's routing-aware prefetch (RouterLookahead) copies all layers' ffn_gate_inp.weight from CUDA0's arena.
With CUDA0 trimmed to its own layers it found them missing and turned itself off ("the routers are not BF16 in the
arena"). CUDA0 now keeps the routers (~2.5 MiB a layer) when --mmap-experts is on; the other stages and the runs
without --mmap-experts trim as before.
Without this every stage of a layer split keeps a full copy of the dense weights (~3.4 GB for the Coder),
although it only ever reads its own layers. That VRAM goes to the expert cache instead.
- the canonical arena of each GPU skips the `blk.N.*` tensors of layers outside its range (PLE tensors stay
everywhere), read from the pack's index.txt;
- NativeDense::set_layer_range limits the native projections the next `load` uploads the same way;
- only with explicit split points (`--layer-split 27`); `auto` and a single GPU are unchanged;
STRATA_STAGE_TRIM=0 keeps the full copies;
- one log line per GPU says what it loads, and a later stage logs its free VRAM before the cache is sized.
Measured on 2x AMD Instinct MI50 16 GB (gfx906 port, Coder IQ1_M, 128K context, --layer-split 27):
experts held in VRAM 8,819 -> 10,626 of 12,288, VRAM hit rate 95% -> 98%, decode on a 17K-token agent prompt
39.2 -> 41.7 tok/s. The trimmed tensors are never read by that GPU.
A re-run for an installed model (another --context, say) rewrote
strata-<model>.json from scratch and dropped the user's sampling,
mcp_servers, mcp and other blocks. Setup now carries over every key it
does not write (and env/vision entries of the user's own, such as a Q8_0
mmproj), keeps the earlier file as strata-<model>.json.bak and says what
it kept and which hand-added engine options it did not. A copy set up
like an earlier install carries them over from that install's config.
A first install writes the same config as before.
The native head's cudaMalloc ran after the expert arena had registered ~45 GiB of host pages. WDDM then refused it
on a 16 GB card with the desktop on the iGPU ("native head upload: out of memory" with GiBs free) - the drafter is
loaded before the arena for the same reason. Now the head and the logits buffer load right after the drafter, still
before the expert cache is sized, so the slots are the same. The head's, the drafter's and a split stage's head
failures print "(N MiB of M MiB VRAM free on this GPU)".
setup: a context of 64K or more whose KV streaming does not fit the RAM test now says so (it was silent, and a
regenerated config lost --kv-resident without a word).
The server replied to /load and /unload without reading the request body; on
Windows, closing a socket with unread data sends a TCP reset, so the client saw
a connection reset instead of the answer. Read the (bounded) body first.
Taken from #630 by CC-David-CC.
0.1.38 made the choice before the resident set existed, from the budget asked for, against every shard's bytes
(dense weights and the PLE table included). On a 96 GB PC with UD-Q4_K_XL (budget 72 GiB, 48.94 GiB really
resident) that read 83.0 - 72 - 4 = 7 GiB of room against 103.7 GiB of files, so every refill after a prompt read
the drive: prompts 15-40% slower than 0.1.34.
- platform::file_cache_keeps(avail, arena, read): the decision, unit-tested in platform_memory_test with the
reporter's numbers and the 32/64 GB PCs of #357/#362.
- set_unbuffered: the arena is at most every expert's bytes, and the cache has to keep only the experts outside it.
- recheck_unbuffered, after pin_cache_complement: the same decision from the RAM the copy really holds and the
expert bytes outside it; switches either way (startup, nothing reading) and logs the numbers.
- The 32 and 64 GB PCs still read unbuffered (the profile fill and the copy too, as before).
Same tokens either way: only how the same bytes are read changes.
openai_to_messages / anthropic_to_messages check that tools is a list of
named tool objects (a JSON-string list is decoded, as for messages) and
raise ValueError otherwise; the MCP name set skips non-objects too.
0.1.38's close() already ends the old engine and waits for it (QUIT, terminate, kill, 20 s each) and raises
EngineStuck when it cannot. The PR's second kill and 120 s wait, after which a new engine was started next to one
still alive, is left out; the start retry and the last known context stay.
Seen in long agent sessions (Claude Code, many turns at ~20K tokens): when the engine was ended mid-request (the
60 s watchdog), restart() started the new engine while the old process still held its VRAM. The new one ran out
of memory and exited before READY, `max_context` stayed 0, and from then on every request failed with
"400 ... exceeds the context (0)" until the server was restarted by hand.
- restart() ends the old process first (close(); kill and wait if it is stuck), so its GPU memory is free;
- a start that exits before READY is tried again, 3 times, RESTART_RETRY_S (15 s) apart; `starting` is set
meanwhile, so a request gets the 503 "starting" of #344;
- `known_ctx` keeps the last context the engine reported: after a restart that failed for good, a request is
checked against it and reaches run(), which starts the engine again, instead of a 400 about context 0.
Two tests in RestartWindow (both fail without this change): a start that fails once is retried and the engine
comes up; a request after a failed restart gets 200, not a 400. `python -m unittest serve.test_server`: 120 OK.
native_expert_grouped for the UD-IQ4_XS shapes (n_embd 2560, n_ff 640): s26_gu_l_kernel / s26_down_l_kernel keep
every row's lane-strided k order, Split<TY>::apply expression and warp_sum, so every output is bitwise the old one's;
a warp interleaves 4 rows (more loads in flight), the chunk's activation q8_1 rows are read from LDS, and for IQ3_S
the 512-entry grid table too (the same values); SwiGLU and its q8_1 run as one launch (contraction off in the
quantizer). Other formats / shapes keep native_gu_multi_kernel / native_down_multi_kernel.
gfx1151 harness (S26 ws3: real expert blobs, 32 experts x 4 layers = 298 MB, decode-style windows planned as
resident_plan does, memcmp of every output, paired HIP events): one layer's experts T=1 140.3 -> 133.3 us, T=2
225.1 -> 204.1, T=3 300.2 -> 265.9 (166 -> 175-185 GB/s). Tried and rejected: non-temporal loads (no gain), whole
rows staged in LDS (0.6-0.8x: occupancy), 8 rows per warp for down.
Engine (UD-IQ4_XS, c-best3 env + the switch, same binary, S25's 18 prompts): outputs 18/18 identical; 56.6 / 57.0
-> 55.65 / 55.27 (and 55.61 with the fused SwiGLU q8_1) ms per verify window, 48.6-48.9 -> 50.3 tok/s;
59K prompts 2/2 identical, 58.65 -> 56.77 / 56.66 ms per window, 39.70 -> 41.01 / 41.09 tok/s.
- conversation_snapshot_*(..., const QsaState* draft): nullptr saves, validates and restores an image without the
draft layer's K/V. The later stages park that way; the draft ring is saved once, with the first stage's image.
- ConversationKvReuse::stages / ConversationCache::retain(.., stage_kv): every stage's restored K/V is retained, so
its next park copies only what changed, as the first stage's already did.
- conversation_checkpoints_split / _merge: the checkpoints are moved into their stage parts for the capture and
back afterwards (and on restore), instead of copied.
- Tests: conversation_cache_test (split/merge round trip without copies, refused misalignments, per-stage reuse),
conversation_snapshot_test (draft-less images: estimate, validation both ways, A/B/A exactness, incremental reuse).
- tools/parking_test.py: a follow-up decodes the same tokens whether its conversation stayed live or came back
from the parking cache (every stage's image with a layer split). 4-GPU split, IQ3_S: 3,786 and 9,276-token
conversations restored in 42 / 51 ms, follow-ups identical; without parking the same follow-ups took 3.4 / 6.8 s.
- tools/batch_test.py: --keys to send sampling keys (sampled rows in batch windows).
- docs/BATCHING.md: parking together with batching, the Testing section; docs/DETAILS.md: parking now works with
--layer-split.
park_current snapshotted stage 0 only, so a split rejected parking. A parked conversation now holds one image per
later stage (SavedConversation::stage_images, captured from each stage's session on its own GPU); a split
checkpoint is cut into its stage-0 part and its stage parts while parking and put back together on restore; every
stage image is validated before anything is overwritten; the guard is removed.
Patch posted on #559 by crazyaimachine (written by their coding agent), tested on 2 GPUs there and on a 4-GPU
split here.
Co-authored-by: benoit lange <[email protected]>
GDN step (verify_kernels.cu, S25's STRATA_GDN_SPLIT=1 split, handed to S26):
- the commit's replay writes no outputs (t_out_begin = MT): y_dummy_ was never read; the state is unchanged;
- the commit runs gdn_state_replay_kernel: S25's column split (4 blocks of 32 columns per head) with gate, beta and the
v column of every row prefetched and the next row's k / q loaded while the current one computes;
- the window keeps the old kernel for one row (the split did not pay there) and runs the split for 2+ rows with
gdn_out_norm_kernel rewritten in the old kernel's code shape: S25's 128-thread norm differed from the old kernel by
1 ulp in ~7% of the outputs (405 of 6144 at T=1); the S x RG version is bitwise equal.
gfx1151 harness (h_v 48, 36 layers rotating, paired, every layer memcmp-equal): commit 44.1 -> 39.1 us at 1 row,
50.7 -> 41.8 at 3, 60.8 -> 46.1 at 6; window 23.8 -> 23.2 us at 2 rows, 30.1 -> 26.7 at 4.
STRATA_QFUSE=1: q8_1 images written by their producers instead of a separate native_quantize_q8_1 launch:
- the GDN output norm (both window kernels) quantizes its outputs (warp = one 32-column q8_1 block), contraction
off in the quantizer so fma(a, b, shfl) cannot round the first sum differently (that made ds.sum differ by 1 ulp);
- the hyper-connection read (gr_up_multi_kernel, gr_up_q8_kernel): the second of the two 16-column blocks of each
32-column group (a per-caller counter, reset by that block) reads the 32 `mixed` values back and quantizes them -
attention side into xq_, FFN side into nat_xq_; fused_gr_read_multi returns whether it wrote them (the v3 read
does not). gr_parity: new check, T 1..8, direct and graph replays, q8_1 bytes equal to native_quantize_q8_1's.
Harness: GDN step + quantize 23.1 -> 20.5 us (1 row), 26.4 -> 23.3 (2), 32.6 -> 27.0 (4), q8_1 bytes equal.
Engine (UD-IQ4_XS, c-best2 + the switch, S25's 18 prompts, 256 new tokens): 18/18 identical in every run;
c-best2 49.46 / 49.44 tok/s, + STRATA_GDN_SPLIT=1 50.13 / 50.23 / 49.99, + STRATA_QFUSE=1 50.10 / 49.98 / 49.95.
strata_pf_gemm_f16 (wmma_gemm.cu), tried in Gemm::f16 before hipBLASLt for T >= 64, N >= 512, K % 32 == 0, beta
0 / 1: block 128 x 256 (128 x 128 below N 1024), BK 32, 8 waves of 64 x 64, LDS double buffer with the next stage
prefetched into named registers, grouped tile order (8 row tiles sweep all column tiles, so the weights stream
from DRAM once per group). FP32 accumulation, another K order than hipBLASLt: rounding-level.
gfx1151, M 16384 (TFLOPS, tuned hipBLASLt in brackets): 12288 x 2560 33.0 (27.4), 10240 x 2560 31.9 (26.8), 6144 x
2560 31.1 (25.9), 2560 x 2560 30.1 (24.9), 640 x 2560 30.2 (25.3), 2560 x 6144 19.4 (19.1); the register-only WMMA
ceiling is ~50. Engine 8K: gdn 1122 / 1112 -> 1083 / 1071 ms, qsa proj 813 / 716 -> 692 / 676, gdn out proj
753 -> 645 / 665. Teacher-forced KL 0.099 / 0.028 on two text offsets (accepted pair: 0.117 / 0.030). Off by
default. With STRATA_HC_UPMIX + STRATA_PA_FAST, UD-IQ4_XS 64K prompt 872.8 / 880.6 -> 942.4 / 937.8 tok/s.
prompt_attn_wmma_kernel carries q and p as FP16 hi + lo pairs (two WMMAs each, FP32-level). SPLIT = false drops
the lo halves: FP16 attention as the usual flash-attention kernels, half the matrix work.
gfx1151, real 64K selections (16383 queries x 2051 cells, one QSA layer): 138.7 -> 117.8 ms; rel RMS vs FP64
4.8e-7 -> 2.7e-4. Engine 8K "qsa attn" 718 / 712 -> 589 / 590 ms. Teacher-forced KL on three text offsets 0.153 /
0.087 / 0.027 (the accepted MMQ-vs-fused pair: 0.153 / 0.117 / 0.030). Off by default.
The read's up GEMM wrote `gated` (T x 10240 FP32: 671 MB per 16K chunk) for gr_mix_r to read back. gr_upmix
computes a tile of 128 tokens x 16 columns for the 4 streams on the matrix cores (BF16 in, FP32 accumulate, the
block's 64 w_up rows staged in LDS once) and runs the mix in registers, in gr_mix_r's order. The grid is ordered
column-tile fastest: the blocks in flight share one token tile, so the R reads of the epilogue stay on few pages
(token-tile fastest touched 512 rows x 40 KB per block: 28 ms instead of 3.7 at 8K - address translation on this
box goes through the IOMMU). STRATA_HC_UPMIX_CHECK=N runs the default pair on the first N reads and prints the
difference.
gfx1151, UD-IQ4_XS: `mixed` vs the default rel RMS 1.2-3.6e-7 on every read of an 8K prompt; prompt-timing "hc
read" 1352 / 1313 -> 1080 / 1212 ms at 8K; 64K prompt 874.3 / 876.1 -> 902.7 / 903.0 tok/s. Rounding-level (the K
order of the GEMM): teacher-forced KL on three text offsets 0.170 / 0.085 / 0.032 vs the accepted MMQ-vs-fused
pair's 0.153 / 0.117 / 0.030; ledger and 8K / 64K retrieval pass. Off by default.
With --batch, a consumer that left mid-answer (a client disconnect, a tool call or a stop string ending the
response) left the engine writing the abandoned answer's tokens, and the next request read them as its own:
it answered the previous question again. Now the solo request is STOPped and read to its DONE, an admission
to its BADM, and a slot is BSTOPped and freed only at its BDONE. STRATA_BATCH_TRACE=1 logs the control lines.
tools/early_close_test.py reproduces it (solo and with a second request running).
The engine's --serve can now keep up to N conversations open (--batch N) and decode them together, one token
each per verify window; with a layer split, --batch-groups G pipelines the slot groups through the GPUs, and
--trim-stage-weights makes every GPU load only its own layers' dense weights (VRAM back to the expert cache).
The server runs concurrent requests in the slots. See docs/BATCHING.md.
Appends [n][hc*n_embd] BF16 rows (round-to-nearest-even) in position order for every batched prompt chunk: the
input the MTP draft layer reads, for training it against this trunk. Off unless the variable is set.
Layer 1's PLE block ran its key (10240 x 2560) and value (2560 x 2560) GEMVs once per window row, reading ~40 MB of
weights per row. Both depend only on the row's n-gram embedding, so with the switch the group's rows go through
the multi-row forms of the same kernels (native MMVQ over n columns / bf16_gemv_fp32_mmvf_multi, each row's
arithmetic the single-row call's) and ple_block copies its row's results instead of recomputing them. The
history-dependent rest of the block stays row by row.
The GGUF's F32 inject rows hold BF16-exact values (conversions.json: "the values are BF16's"), so quantizing them
to Q8_0 only added error: the Q8_0 read now takes the pack's BF16 inject rows (STRATA_HC_Q8_INJECT=1 keeps the
Q8_0 copy). The final mixer before the head (output_hc_down/up, also BF16-rounded in the pack) now reads the
GGUF's Q8_0 values through the same multi read.
Conflicts: mtp.cpp (the draft head: S25 converts it with --mtp-q4 first, then STRATA_Q6_PACKED packs it only
while it is still Q6_K), native_dense.cpp (includes).
gdn_step_norm_multi_kernel ran one 512-thread block per head - 32 blocks for a layer's 2 MB state, ~36 us per
launch at T=1 on gfx1151 (~60 GB/s) and +4 us per window row; it runs twice per GDN layer and window (the window
and the commit's replay). With the switch the recurrence runs on (head, 32-column) blocks with the same
(column, row group) threads and state rows, so every column's arithmetic is unchanged; the output norm, which
needs a head's 128 columns, follows in gdn_out_norm_kernel with the same warps and the same sum order.
The main output head (248320 x 2560) and the MTP draft head (its 106299-row subset) are Q6_K: 210-byte blocks, so
every int load is 2-byte aligned. STRATA_Q6_PACKED=1 builds a packed copy of each at load - the same bytes as ql /
qh / scales / d planes (+497 MiB, +213 MiB) - and native_q6_k_mmvq on a registered pointer runs
native_q6_k_packed_kernel: aligned non-temporal dword loads, 640 persistent workgroups, and the thread ownership,
q6_q8_dot_impl expression, accumulation order and reduction of native_q6_k_mmvq_kernel<false> /
native_mmvq_multi_kernel<Q6KTraits, NC, 4, 1>, so every output is bitwise equal (exact multi-column layout only).
STRATA_Q6P_SELFTEST=1 checks it at load (1-4 columns, memcmp) and times it paired; =2 also sweeps the grid.
gfx1151 load-time self-test (11 paired samples, medians): main head 1 column 2420 -> 2300 us, 2 columns 2563 ->
2455, 4 columns 2973 -> 2975 (no gain: not bandwidth-bound there); draft head 1 column 1319-1351 -> 946-961 us,
2 columns 1420-1459 -> 1064-1090, 4 columns 1629-1678 -> 1320-1365. Outputs bitwise equal. Grid 320-20480 flat.
STRATA_VERIFY_RESIDENT=1: when every expert of every layer is resident in the VRAM/UMA tier (checked once at init;
a full cache never evicts; one stage only), the window graph is recorded without the per-layer host step: no
doorbell_publish, no wait_flag_ge (3 per layer), no plan copy from mapped memory, no PCIe/CPU share (fetch_blobs,
rebase_ptrs, its empty grouped launches, copy_rows_from_mapped) and no moe_hit_add: the device plans each group
(resident_plan, the host's exact loop) and the combine reads the hit rows as 0.0f + hit (combine<true>), which is
what the zeroed part rows plus parts += hit gave. run() then skips the per-layer spin. Same tokens.
resident_plan: the plan in one block of 128 threads (entry i per thread; groups in first-occurrence order, entries
in increasing i, the same arrays) instead of one thread - 44 us -> a few us per call on gfx1151, 48 per window.
Used by STRATA_VERIFY_DEVICE_PLAN=1 too.
STRATA_VERIFY_QDEDUP=1: the experts' q8_1 image of the MoE input is made before the shared expert, whose gate/up
read it instead of quantizing the same rows again (quantize_q8_1_rows and native_quantize_q8_1 write the same
bytes); the shared expert's SwiGLU and its q8_1 run as one launch (native_swiglu_q8_1_kernel: the same expression,
then native_quantize_q8_1_kernel's block quantizer); the BF16 copy of the input is skipped when the shared expert's
native BF16 gate (which never reads it) is on. Same bytes, same tokens.
Aurora (gfx1151, UD-IQ4_XS, S23's c-u4 config, one binary, env-only arms): output ids identical in every run.
Per verify window (rocprofv3 trace, 8K prompt + 300 outputs, 114 windows): see aurora_s23.md (S26, overhead).
The router (512 x 2560) and the indexer projections run bf16_f32_mmvf_multi_kernel: one block per output row, so
the window's T activation rows were read again by every block (+3.3 us per row per launch on gfx1151, ~0.2 ms per
window row). With the switch a block computes 4 output rows from one read of the activations; per output each
thread keeps the same pairs, FMA order and reductions.
attn_chunk_kernel scored a chunk's 64 cells warp by warp: each lane 8 of the 256 dimensions, then one warp_sum
per query head - 12 shuffle reductions per cell, which is what made the kernel ~53 us per layer at T=1 and
+32 us per window row on gfx1151 (it scales linearly with T). With the switch, thread t scores cell t % 64 for
3 of the 12 heads alone: the same 8-dimension partial expression per former lane, then lane 0's butterfly order
(a[l] += a[l + o], o = 16 .. 1), so each score has the same operations in the same order. Softmax, values and the
merge are unchanged.
The read takes the Q8_0 kernels only when the inject rows have a Q8_0 copy as well, and the GGUF stores them as
F32: with only down/up loaded (0.62 GiB) every window still ran the BF16 read (Aurora: 18/18 outputs identical
to the default, verify 61.4 vs 60.9 ms). The inject rows (4 x 10240 per projection) are now quantized at load
with ggml's Q8_0 reference rounding.
NativeDense keeps the six hc_* projections per layer as stored (Q8_0, 0.65 GiB) beside the pack's BF16 copy
(1.20 GiB, --compat-bf16's rounding); the verify window's fused_gr_read_multi then reads them with split-K
kernels: gr_down_q8 (10 row groups + the inject rows x 16 chunks of 640 columns, the norm folded in as
unscaled partials + per-chunk sums of squares) and gr_up_q8. Every other path keeps BF16. Another summation
order and the GGUF's own values: output-changing, so opt-in and quality-checked separately. Also the
STRATA_GR_V3=2 knob (the two-half v3 split where only 64 KiB of LDS is available). Taken over from S23
(aurora_s23/hc_q8_grv3s2.diff) unchanged.
On Strix Halo (UD-IQ4_XS, head Q6_K) the 4-bit head costs acceptance (tokens per step 2.81 -> 2.72 with both)
while the smaller 40.5K-token subset (--mtp-draft-vocab data/draft_vocab_en.bin) costs none, so the two 4-bit
copies are now separate choices.
Measured on Strix Halo (UD-IQ4_XS, 1-2K contexts) a verify row costs ~9 ms on a ~61 ms window, so chaining
every 3+-token match (49% of chained drafts accepted) gave +6.4% tokens per step for +7% window time. The
policy now picks the number of chained tokens k with the best expected tokens per ms: the MTP window's
measured tokens + p_mtp (c + c^2 + .. + c^k) at the measured cost of the longer window, where p_mtp is the
product of the draft layer's own draft probabilities and c the chained drafts' acceptance by match length
(their own decayed counts, learned only from rounds that reached the chain). Unmeasured sizes are probed a
few times when p_mtp * c >= 0.6. STRATA_LOOKUP_CHAIN_FIXED=1 takes every proposal as before.
Both opt-in; drafts only, the verify window decides every token (18/18 prompts identical on IQ3_XXS).
- --mtp-q4: the draft layer's ten Q8_0 projections (fc_embedding/fc_hidden, q/k/v/o, the shared expert) and
the draft head's subset rows are requantized to Q4_0 at load/bind with ggml's reference quantizer
(native-experts builds). The prompt path's batched K/V pass keeps the Q8_0 originals.
- --mtp-draft-vocab FILE: the draft head's token subset from FILE instead of <mtp>/draft_vocab.bin
(data/draft_vocab_en.bin: 40,525 tokens instead of 106,299).
RTX 5070, IQ3_XXS (head Q5_K), 18 prompts: draft ms per window 2.55 -> 2.23 (q4), 2.13 (en), 1.91 (both);
tokens per step 2.64 -> 2.63 / 2.67 / 2.57.
All opt-in; with none of them given the engine runs exactly as before.
- --lookup-chain K [--lookup-chain-min M]: after the MTP's drafts, the suffix drafter matches the context
plus those drafts and appends up to K of the tokens that followed the match to the same verify window
(the window grows to at most 8). Greedy verification keeps the output. Own counts: a stderr line per
request and five more DONE fields when on.
- include/strata/spec/draft_source.hpp: a DraftSource interface (the request's own text is
PromptLookupSource; other sources register in extra_draft_sources() and the longest match wins).
- --mtp-hnorm stream: the draft layer's pre_fc_norm_hidden as one RMS per hyper-connection stream
(llama.cpp's qwen4exp MTP graph) instead of one over all four (vLLM's qwen4_exp MTP, the default).
New kernel native_qsa_rms_norm_grouped; the batched prompt K/V pass follows the switch.
- Diagnostics: STRATA_MTP_FULL_HEAD=1 drafts over the whole vocabulary; STRATA_MTP_TOP2=1 counts how often
the draft layer's runner-up is the token verify picked where a draft was rejected.
A lossless load-time repack of the native dense Q8_0 matrices with 1024 <= n_in <= 4096 and n_out >= 2048 (on
UD-IQ4_XS: attn_qkv, attn_gate, attn_q; 84 matrices, +1.87 GiB): the same int8 values in a qs plane and the same
fp16 scales in a d plane, 34 bytes per 32 values as before. native_q8_0_mmvq finds a registered matrix by its
GGUF-layout pointer (that copy stays: the prompt GEMMs read it) and runs native_q8_0_packed_kernel: 640 persistent
workgroups; each thread owns the bytes of the exact layout (block tid/4 + 32 i, ints 2 (tid%4) and +1), reads them as
one aligned 8-byte run with streaming loads, and accumulates and reduces in the same order, so every output is
bitwise equal to native_small_mmvq_kernel / native_mmvq_multi_kernel (exact layout; with the upstream multi-column
layout the GGUF kernels run).
gfx1151, S26 standalone harness (8 real UD-IQ4_XS layers per shape rotating, paired HIP events, 15 samples, every
layer x 1-5 columns memcmp-equal): attn_qkv +10-12%, attn_gate +11-12%, attn_q +10-11% per call. 6144-wide rows
(ssm_out, attn_output) and rows < 2048 are not packed (slower in the engine / standalone).
Engine (UD-IQ4_XS, S23's c-u4 config, same binary, env only): output ids identical in all 10 runs; steady output
8K 40.06 vs 39.77 tok/s (3 runs each), 64K 30.94 vs 30.34 (2 each).
HIP counts a gfx11 WGP (two CUs, which the WMMA prompt-expert kernels use as one in
the default WGP mode) as one multiprocessor, and the occupancy query answers 1 block
for these kernels where 2 run side by side - so every fused launch (native IQ and
Q2_0) ran a 20-block grid on gfx1151, 2 waves per SIMD. wgp_blocks() doubles the
query's answer (STRATA_PF_OCC=N sets it, an experiment knob).
The same work items and arithmetic: parity passes, fused-vs-MMQ rel RMS unchanged,
and the engine's outputs are the same tokens.
gfx1151, one layer, 8192 tokens x top 10, IQ3_S / IQ4_NL: blocks per WGP 1: 60.7 ms,
2: 44.7, 3: 49.9, 4: 45.2. Q2_0 (2048 tokens): 13.3 -> 11.4 ms.
Engine, UD-IQ4_XS (MTP + vision + speed projection): 64K prompt 819.8 / 823.9 ->
892.4 / 878.8 tok/s, 8K 825.5 / 820.8 -> 874.5 (the other 8K run overlapped a
download: 758); outputs identical (drafts 292/516 at 64K, 343/468 at 8K); ledger
{32,17,130}, 8K retrieval 3/3.
A rebinding page cannot send the key, so with one configured the check only
broke tunnels and proxies that pass their own name on (cloudflared does).
Keyless setups keep it, with allowed_hosts; the 403 now also says that
setting an API key turns the check off.
It keeps 256 of the 512 experts, chosen on code data, so it is weaker outside
code and in other languages; for general use, or when RAM allows, MODELS.md
now points to the full models.
/unload and /load took a plain form POST from any site: they now take JSON
from Strata's own page only, like /settings (_own_page). Without an API key, a
POST to /v1/* that carries an Origin must come from Strata's own page, a page
on localhost or an allowed host name, an origin in trusted_origins or
cors_origins, or a non-web scheme (browser extensions, desktop apps), and then
send JSON; others (and Origin: null) get 403, so a web page can no longer burn
GPU time with a text/plain POST. Requests without an Origin (curl, the SDKs)
are unchanged; with an API key, the key decides.
A web page of another site could point its own name at 127.0.0.1 and reach
the server as the same origin: without an API key it could read /status,
/metrics and /api/requests, change settings and run MCP tools. The server now
answers only Host names it knows: localhost (and *.localhost), any IP address
(rebinding needs a name), the bind address and, bound beyond this PC, the PC's
name, name.local, its LAN addresses and host.docker.internal; any port. No
Host header (HTTP/1.0) passes. "allowed_hosts" in the config and
STRATA_ALLOWED_HOSTS add names (".example.com" for a domain, "*" turns the
check off); trusted_origins' hosts count too. A refusal is a 403 naming the
setting and one log line.
The weight arena is the first large VRAM allocation and its size does not depend on the context, so when it fails
something else holds the card (another program, an engine still exiting). The error now names the CUDA error and the
card's free and total VRAM at that moment; a layer split's stage says the same for its card, with the MiB it needed.
Error paths only.
One line under the start banner: the config's engine options without the model's file paths, and the
server's fields (host:port, an API key set or not, GPUs, draft subset, fit_max_tokens, thinking budget).
The key itself is never printed.
The provider, the context and output limits, the thinking level as a default and as variants, and how
they relate to the context check, fit_max_tokens and reasoning_budget_tokens.
A CUDA 13 build that linked the system's CUDA 12 libcudart.so (a kit with a dangling libcudart.so symlink) read
cudaDeviceProp shifted: sharedMemPerBlockOptin came back as 1, #420's fits() then refused MMQ for every i-quant and
the prompts ran ~4x slower, with a log line printing "1 bytes of shared memory per block".
- ggml_cuda_info() (CUDA) reads cc, smpbo, smpb, nsm, warp size, integrated and cooperative launch with
cudaDeviceGetAttribute, which is ABI-stable. A matched build reads the same values: the same MMQ choices, the
same output.
- The engine warns at start when the loaded CUDA runtime's major version differs from the headers' it was compiled
with, and says to rebuild against one toolkit (-DCUDAToolkit_ROOT).
HIP is unchanged.
Dev::put uploads with a pageable cudaMemcpy, which can return before its DMA lands, and the graph replays and the
timing cases run on a cudaStreamNonBlocking stream that does not wait for the legacy stream: the graph could read
the previous replay's logits (one argmax token off; 5 of 6 replays on an RTX 5060 Ti, now and then on a 5090).
Dev::put now synchronizes the device after its copy, which covers run_graph_case and bench(). Test only; no engine
code.
A reply that ends with finish "length" inside the reasoning has no answer; the server log now says so
and names reasoning_budget_tokens (per request, or in the config for every request). Nothing the client
receives changes.
With the default 700 MiB reserve the expert cache fills an AMD card that also drives a KDE/Wayland desktop. When the
desktop then needs VRAM (the browser --open starts, a new app), amdgpu moves GPU memory to system RAM (GTT), which
the experts already fill: the OOM killer ends kwin_wayland / plasmashell (#560, R9700), or the compositor fails to pin
its framebuffer (#516, 7900 XTX). --vram-reserve-mib 3072 / 4000 fixed it in both reports.
A recommendation, nothing is changed: setup prints the tip on a Linux desktop session (WAYLAND_DISPLAY or DISPLAY) with
the HIP backend and no reserve given, and the server prints it after the start when the engine reports less than
2048 MiB of VRAM free (INFO vram_free_mib) and the reserve is under 3072 - before --open starts the browser.
INSTALL.md and AMD_HIP.md describe the symptom and the fix. No engine code.
installed_configs() and update_install() keep only JSON objects with "exe" and "args"; any other
strata-*.json in the folder gets a one-line warning naming it.
The verify window waits for the adaptive tier's copies since #463; STRATA_ADAPT_NOWAIT=1 starts it as 0.1.37 did (the
non-blocking query), for timing that wait. Default unchanged.
fused_layout() (STRATA_PF_FUSED=1) shrinks GU / H / Xq to the fused kernels' needs whenever
fused_ring() is true, and fused_ring() was true when ANY layer of a native pack was covered. A pack
with an uncovered layer - Unsloth UD-IQ4_XS: 5 layers with Q8_0 down projections - then ran MMQ for
those layers at the full chunk in buffers sized for stream_all_min() - 1 tokens: on gfx1151 an 8K
prompt answered "!!!!..." and a 64K prompt hung (--prefill 2048: an illegal memory access). Such a
pack now keeps MMQ's buffers; its covered layers still take the fused kernels. UD-IQ4_XS 8K
retrieval after the fix: 3/3 correct, 837.9 prompt tok/s. (The CUDA path shares this code.)
fused_layout() (STRATA_PF_FUSED=1) shrinks GU / H / Xq to the fused kernels' needs whenever
fused_ring() is true, and fused_ring() was true when ANY layer of a native pack was covered. A pack
with an uncovered layer - Unsloth UD-IQ4_XS: 5 layers with Q8_0 down projections - then ran MMQ for
those layers at the full chunk in buffers sized for stream_all_min() - 1 tokens: on gfx1151 an 8K
prompt answered "!!!!..." and a 64K prompt hung (--prefill 2048: an illegal memory access). Such a
pack now keeps MMQ's buffers; its covered layers still take the fused kernels. UD-IQ4_XS 8K
retrieval after the fix: 3/3 correct, 837.9 prompt tok/s. (The CUDA path shares this code.)
#531's peer-aware lookup indexed the residency table with the 512-expert constant NE instead of the model's own
count. The Coder has 256 experts per layer, so from its second layer a resident expert was looked up in the wrong row,
and a prompt long enough for the streamed walk stopped with "an expert is neither resident, streamed nor on the peer"
(found by the 57K real-use check). The 512-expert models index the same row as before.
30 more rows from tools/hip/tune_hipblaslt.cpp on Aurora (90 in all). IQ3_XXS, 64K prompt, the
gfx11 fused experts: --prefill 16384 with them 849.2 / 847.0 tok/s vs 8192 803.1 / 824.0 / 829.3.
Setup no longer writes a fixed 300 MiB reserve for cards under 8 GB: the engine's own rule (lower the reserve only as
far as the smallest working cache needs, down to 300) chooses better, and a fixed 300 left an emulated small card with
5 MiB free. Setup keeps the tip (an 8K context, --draft-vocab en). When the engine lowered the reserve and the card ends
below the 256 MiB LOW line, it prints one warning (requests may stall; close other programs that use the GPU, or lower
--max-context) instead of advising a larger reserve, which would leave the cache no room.
peer_experts.cpp calls cudaDeviceCanAccessPeer, cudaDeviceEnablePeerAccess and cudaMemcpyPeerAsync and tests
cudaErrorPeerAccessAlreadyEnabled; the HIP shim had none of them, so the Windows HIP build stopped there. Mapped to
their hip* names.
On an RTX 3060 Laptop (6 GB) the weights, the KV cache and the draft layer left 0.29 GiB, the default 700 MiB reserve
and the draft head took it, the expert cache got 0 slots and the verify window refused to start.
- setup: a single card under 8 GB (nvidia-smi below 7.5 GB; an 8 GB card lists 7.99) gets --vram-reserve-mib 300
unless one is given (images on the GPU keep theirs), marked in the config as setup's own choice so a later setup
decides again. It prints what frees more VRAM if the start still stops: an 8K context, --draft-vocab en, other
programs closed. The draft layer stays (the server needs --mtp and --spec). 8 GB and larger cards: the same config
(golden test).
- engine: a cache too small to lend the prompt path a 256-token chunk (its buffers plus the 128-slot floor; one slot
without --prefill) makes the prompt path allocate its own on top. When the default reserve leaves less than that,
the reserve shrinks to what leaves exactly that cache, down to 300 MiB (what comes after the cache is paid from
the reserve: at 300 an emulated small card ended with 5 MiB free, LOW by the serve check). With no slot at all the start stops with how many
MiB are short, what the session and the draft head take, and the options. A start whose cache could lend that
chunk is sized as before. Checked with a VRAM ballast on the 5070 (IQ3_XXS, 32K): 258 slots needed, 0.71 GiB free
-> "about 333 MiB more", and with --vram-reserve-mib 700 given, "733 MiB more" plus the reserve tip.
- verify's refusal points at that line.
From PR #409 (af404df), only device_free_bytes(): on an integrated GPU (cudaDevAttrIntegrated, on HIP
hipDeviceAttributeIntegrated; the Ryzen AI Max 8060S) the expert cache sizing and checks count
/proc/meminfo MemAvailable less STRATA_UMA_HEADROOM_GIB (6) when that is more than cudaMemGetInfo:
the page cache of experts read from the GGUF in place gives way to a device allocation. The PR's
aarch64 / DGX Spark CPU parts are not taken.
From PR #313 (1e42cf3): src/prefill/wmma_gemm.cu/.h, the Gemm::f16/bf16 hooks before hipBLASLt, the
build macro STRATA_WMMA_GFX11 and its parity test; not its prompt attention (STRATA_HIP_WMMA covers it).
Fix: the intrinsics guard named only gfx1100/1101/1102 while the runtime gate takes any gfx11, so on
gfx1150/gfx1151 mma() compiled to "return c" (silently wrong); it now names gfx1150/gfx1151 too.
hip_prefill_wmma_gemm_parity on gfx1151: 3520 cases pass (f16 max_abs 2.4e-7, bf16 0).
In the streamed walk every expert put a cross-stream wait (its copy) and an event record (its ring slot back) on the
compute stream around a ~3 us gather. Under WDDM each pair left ~10 us of GPU idle even when the copy had long
landed: an nsys trace of a 32K prompt showed 226 ms of idle between back-to-back gathers while the host was 3-11 ms
ahead of the GPU. Now a group of 16 is gathered by one launch (copy16_group_kernel) after one wait on its last
streamed copy - the copy stream is in order - and one event releases all of its ring slots: the issuer waits on
used[used_of[slot]]. A ring entry the routing skipped inside an open group first gathers what the group holds so
far, so no more than a group's entries are ever held back from the issuer. The same bytes in the same group slots.
STRATA_PREFILL_GROUP_GATHER=0: one at a time, as before.
IQ2_XS, RTX 5090, 262K context: 32K prompt -8.6%, 8K -10%, 2K -12.6%; first-token logits bit-identical at 2K, 8K,
16K and 32K and with a 96-slot ring.
The switch was read with getenv on every launch_multi call, which runs per layer when decode is not captured in a
graph. It is now read once, like the engine's other STRATA_* switches.
STRATA_GR_DOWN_MAX4=1: launches of at most four tokens run the GR down-projection kernels (plain, split and staged)
with per-thread storage for four tokens instead of kFusedGrMaxT, keeping the block dimensions, the tile width, the
accumulation order and the plain/split/staged choice: the same bits. Off by default. RTX PRO 6000 Blackwell, IQ3_S,
MTP 4, 64K: decode +2.1% (code) / +2.4% (prose) with identical output tokens.
From #443; its one-warp MMVQ experiment (STRATA_MMVQ_WARP1, not bitwise), the roofline benches and the benchmark
evidence files are left out.
fused_ring() turns the fused ring sizing off for any --peer-device session (core::peer_portable), but the layer's
fused choice tested the peer's prompt-path state (m.pp), which is null when set_peer declines (no P2P, the buffers do
not fit, --peer-prefill-rows 0). Both now follow the flag, so a peer session's prompt path is MMQ with the MMQ ring
and buffers it was sized for. Without --peer-device nothing changes.
A second visible CUDA GPU (N >= 1) holds an adaptive expert cache beside
CUDA0's, filled with the ranked pairs the primary does not hold. Activations
and result rows cross NVLink or another P2P path; the peer also computes its
experts' rows for prompt chunks (MMQ; the fused path is not combined yet).
Knobs: --peer-device N, --peer-reserve-mib, --peer-slots, --peer-adapt-swaps,
--peer-prefill-rows; sizing stays manual until #216's buffer lending lands.
Default output unchanged: without the flag byte-identical to v0.1.36, with it
byte-identical under the exactness gate. cudaHostAllocPortable only with a
peer. docs/SECOND_GPU.md documents the flags and refused combinations.
Measured on 2x RTX 3090 + NVLink, IQ3_S: prefill +15/+25/+25 %, decode +19/+14/+16 %.
E-9 (Prefill::draft_kv) declines when the drafter's KV is a ring, which it is
with KV streaming (--kv-resident), so every chunk falls back to the E-4 pass in
MtpDrafter::prefill. On gfx1201 / ROCm 7.2.4 that queue (a graph launch and
four device copies per group, ~2,000 groups per 8192-token chunk, one sync)
sometimes never completes: the prompt hangs in hipGraphLaunch or the final sync
until the watchdog ends the engine.
HIP builds now use the per-group loop unless STRATA_MTP_PREFILL_SYNC=0; CUDA is
unchanged. The variable now reads its value (any value, 0 included, used to
select the old loop).
R9700, full IQ3_XXS, --kv-resident 32768, 131K context, mixed load: 0.1.31
hung in round 1 on 2 of 2 tries; this change ran 30 of 30 rounds clean with no
measurable prompt cost.
Calibrated on a Radeon AI PRO R9700 with tune_hipblaslt (24 of the gfx1100
table's 26 shapes). Setup only picks it when the card is gfx1201 and the
installed hipBLASLt is exactly 100202.
With that hipBLASLt the plain hipBLAS path is far off: on the full GSQ-RCO
IQ3_XXS (KV streaming, engine 0.1.31) a fresh 32K prompt reads at 1,177 tok/s
with the table against 638 without it, and a 7K prompt at 1,164 against 656
(median of 4 warm runs per arm, arms alternated); decode is unchanged.
hip_prefill_hipblaslt_gemm passes with it. docs/AMD_HIP.md notes that the
1.4.1 result (0-3%) does not carry over to 1.2.2.
STRATA_NO_LARGEPAGES is documented as a same-run A/B switch for the large-page path, but reserve() only read it in the Windows branch; on Linux the MAP_HUGETLB path ignored it, so the switch silently did nothing. Honor it there too.
The failure message also said "no hugetlb pool configured?" even when a pool exists but is smaller than the mapping: MAP_HUGETLB is all-or-nothing, so a 34 GiB pool cannot serve a 40 GiB arena and the whole mapping falls back to 4 KiB pages. Print the needed 2 MiB page count against vm.nr_hugepages (the Windows branch already names its byte count and error codes).
setup's low-RAM packs have an experts.bin, and with it 0.1.31's tier mapped it: on an emulated 32 GB PC the mapped
reads took the RAM the budget was sized for, and available RAM fell below 1 GB (#286, measured yesterday).
- set_unbuffered also takes experts.bin (paths_ is the one file); read_direct reads one window per blob (a blob is
contiguous there) and merges near ones as for the GGUF.
- blob() hands out the stage buffers for any unbuffered source (staged()), so prefetch, begin_layer and transient
work as with the GGUF in place; the mapped experts.bin path is unchanged otherwise.
IQ2_XS pack with experts.bin, emulated 32 GB PC + 16 GB GPU, 16 GiB budget, 8K prompt + 256 tokens, two runs:
decode 100.2 / 100.6 tok/s, done at 17.2 / 15.5 s, the prompt path's host copies 2.1 s (the GGUF in place: 84-89
tok/s, 21.5-22.1 s, 4.3 s). Greedy tokens identical unbuffered, through the cache and on 0.1.31.
With --resident-budget-gib on a PC whose RAM cannot hold the expert files beside the budget, every expert outside
the budget comes from the drive anyway. Through the mapped GGUF that costs twice:
- the page faults are small requests on the critical path (the prompt path's stager, the profile fill);
- the pages they bring in take the RAM the budget was sized for. On an emulated 32 GB PC, the 16 GiB budget was
clamped to 11.65 GiB after the profile fill.
- FileExpertSource::set_unbuffered: chosen by experts_unbuffered (#357) with the budget as the RAM it keeps: whenever
the files could not be kept beside it, cached now or not - a partly cached file's mapped pages would still land in
the working set (seen on an emulated 64 GB PC: the budget shrank, then RAM ran out). Each shard is opened
FILE_FLAG_NO_BUFFERING | OVERLAPPED.
- copy_from_files, the decode prefetch and the complement copy read every role window of a batch at once, then
copy it into place.
- Windows that are less than 1 MiB apart merge into requests of up to 32 MiB. NTFS runs a file's unbuffered reads
one at a time while the file is mapped anywhere, and the engine always maps the shards (token embedding, PLE).
Measured on a PCIe 5 drive at queue depth 48: 512 KiB reads give 10.4 GB/s on an unmapped file and 3.4 GB/s on a
mapped one; 8 MiB reads give 8.6 GB/s.
- The profile fill reads its pairs in batches of 64, the next batch while this one is copied into the slots.
- warms() is false when unbuffered. A PrefetchVirtualMemory of the mapping would read the experts through the cache,
a second time.
- Otherwise nothing changes: the mapped path stays for a keepable set, for no budget, and off Windows.
STRATA_UNBUFFERED_LOAD=1/0 forces the choice.
Greedy tokens are identical with the reads unbuffered, through the cache, and on 0.1.31.
The definition gained a defaulted parameter while the header kept the four-parameter declaration. The engine's
caller is in the same file, so it built, but tests/core/expert_layout_test.cpp calls it through the header and
failed to link (LNK2019, unresolved load_experts_gguf). The default now lives in the declaration;
expert_layout_test passes.
A buffered read copies every expert through the file cache. That pays off only when the cache can keep the files
for the next start. On a PC whose RAM cannot hold them beside the arena (a 64 GB PC with IQ2_XS: 24 GiB left for
36.5 GiB of files), every start reads the drive anyway, and the copy is pure cost.
- experts_unbuffered() (pinned.cu): timed random 64 KiB reads tell whether the files are in the cache already
(a cached read takes ~10 us, the drive ~80). Unbuffered is chosen only when they are not and the RAM left beside
the arena could not keep them either. A warm restart that 0.1.31 serves from the cache stays on its FILE* path.
STRATA_UNBUFFERED_LOAD=1/0 forces the choice; the log names the reader and why.
- load_experts_gguf: each chunk's 4 KiB-aligned window is read with FILE_FLAG_NO_BUFFERING into an aligned buffer and
scattered into the blobs, with a handle per role file as expert_gguf_file() resolves it (split shards). 16 readers.
- load_experts_direct (pinned.cu): a pack's experts.bin ranges are read unbuffered straight into the arena. When the
ranges are not 4 KiB-aligned, it falls back to load_experts_ranges.
IQ2_XS without experts.bin, RTX 5090, emulated 64 GB PC (a locked-pages RAM ballast leaves 58 GiB), two rounds:
| | first start: arena / exit | restart: arena / exit |
|---|---|---|
| 0.1.31 | 22.1-25.2 s / 28.2-31.0 s | 12.5-16.0 s / 17.5-20.8 s |
| this | 18.3-20.2 s / 23.1-25.4 s | 6.0-6.1 s / 9.8-9.9 s |
With 128 GB the probe finds the files cached (15 of 16 reads) and the buffered path runs as before. Greedy tokens are
identical unbuffered, buffered and on 0.1.31 main.
--adapt-decay F sets the factor the routing counts are multiplied by after each adaptation (0.7, as before; the same
float, so the default is bit for bit 0.1.37's). #477's routing heat is added before the decay, as before.
From #407; its --adapt-tuned (opt-in re-ranking every 2 rounds, 192 swaps, x0.92 for caches holding 20-60% of the
experts) is left out of this release.
- The engine's `avx2` switch is on for AVX-512 CPUs too, so the widened gate sent an AVX-512 CPU's IQ4_XS gate/up rows
to the new AVX-2 kernel instead of ggml-cpu: other rounding on CPUs that #415 did not mean to change. A format with
only an AVX-2 kernel now takes it on CPUs without AVX-512; everything else is dispatched as before.
- native_expert_parity ran its AVX-512 check for IQ4_XS on an AVX-512 CPU: iq512_rows has no case for it, so the
rows stayed zero and the check failed. Each check runs only for the formats its kernel implements; the AVX-2 one
runs on AVX-512 CPUs too, so this PC tests the new kernel.
IQ4_XS (23) was missing from iq256_supported, so on an AVX-2-only CPU its gate/up expert rows
fell back to ggml's per-token vec_dot while every other i-quant format got the multi-token
kernel that decodes the weights once for the whole verify window.
Fmt32<23> reuses the 16-value codebook IQ4_NL already has. Each of the eight 32-value sub-blocks
carries its own 6-bit scale (two nibbles of scales_l plus two bits of scales_h), used signed as
(ls - 32): the scale folds into the int16 operand of madd_epi16 instead of becoming a float
multiply per sub-block, and the codebook sign is carried the way iq4nl_rows does it (|w| as the
unsigned maddubs operand, w's sign applied to the activation).
The gate in native_expert.cpp and in native_expert_parity cannot be iq512_supported alone:
IQ4_XS has an AVX-2 kernel and no AVX-512 one, so on an AVX-512 CPU the widened gate must still
pick each kernel only for the formats it implements - falling through an empty switch would
leave ff unwritten rather than falling back to ggml-cpu. Widening the parity gate is what makes
the format tested at all: with iq512_supported alone the tool skipped it on every CPU.
The cherry-picked change kept the old `if (wait) cudaEventSynchronize` above its own (a second, no-op wait), and its
serve-mode call site sat four columns left of its block. The same code otherwise.
A native pack leaves the per-token loop at `if (native_pack) { spec_pos = pos;
break; }`, before the site that dumps a row per position. The header was
written at open anyway, so the file claimed 64 or 66 rows and contained none -
and a reader that trusted it reported "0 differing positions" from a file with
no positions in it, which is the one answer that looks like a pass.
The header is now written with the FIRST row. A run that dumps nothing leaves
an empty file, which is what actually happened, and a reader can treat size
zero as "this mode never dumps" instead of as agreement.
Verified on the native pack: the file is now 0 bytes where it was 8. The
non-native path is unchanged apart from WHEN the header lands; it was not
exercised here because no non-native model is set up on this machine.
(cherry picked from commit b87d7dcfe9a0754c0a2b8b9776b7f2ec8980b6a1)
Greedy decode was not reproducible on this box. Repeated identical requests
diverged from about the 28th generated character, and the drafter, the expert
routing and the first verify window were all byte-stable, which left the
adaptive tier as the only thing left that could move a result.
It is. `apply_pending(false)` polls the swap's event with a NON-BLOCKING
cudaEventQuery, so whether the experts an adapt round just copied in were
resident for the next window depended on whether their PCIe copies had landed.
Resident runs the GPU kernel; non-resident is computed on the CPU, and on a
machine without AVX-512 those two kernels round differently enough to move a
top logit by 0.26 - measured at the window right after the swap was issued.
From there the trajectory is a different one, permanently.
Waiting is the fix. The copies were issued during the previous window's
drafting, so the wait is short, and in exchange the residency table is a
function of what was actually copied rather than of how fast the disk was.
With the default --adapt-every 4 left in place, five runs are now byte-identical
across all 24 verify windows and emit the identical 40-token stream; before the
fix three of five agreed and two forked at window 16. --adapt-every 0 (frozen
placement) was already deterministic, which is what identified the tier.
The STRATA_TRACE_ADAPT lines added with the probe are env-gated and off by
default: they report whether a round considered, skipped, or swapped, and
whether the previous round's copies had landed.
No cost measurement is claimed here: this says the trajectory is a function of
the inputs again, not what the wait does to decode throughput.
(cherry picked from commit 1e3888a73b69a332585056af5b86643ff8367242)
native_expert_grouped launched a block row per possible group (cap = window tokens x 10) for gate/up and down,
and SwiGLU and the q8_1 quantization as two kernels over all cap entries. The verify window's PCIe call has no
group at all at pcie_frac 0 and usually none otherwise, yet every layer paid ~17k blocks that started only to
return plus two passes over entries that are not its own.
- gate/up and down stride over the groups (rows y, y + gridDim.y, ...); the PCIe call runs 4 block rows
(kPcieGroupRows), the VRAM call keeps one per possible group. Each group's rows are computed by the same warp
code in the same order, so the results do not depend on the grid.
- SwiGLU + q8_1 in one kernel over the call's own entries [grp_start[0], grp_start[n]); the product is rounded on
its own (__fmul_rn; on HIP a plain product under contract(off)) before the block sum, as when it went through
memory, and the quantizer body is quantize_q8_1_kernel's (factored out, its PTX unchanged).
- STRATA_GROUPED_V1=1 keeps the old launches for A/B timing; native_grouped_parity checks old vs new bitwise
(every gate/up x down format, 0..cap groups, entries from 0 and past 0, grid_groups 1..4 and > cap).
The engine takes the bound on Turing only, so on other cards the test compared the capacity rule with itself.
STRATA_TOPK_ACTIVE_ANY=1 (set by the test) takes the Turing dispatch on any card; the kernels are the same on every
architecture. The CMake block moves above the qsa_select_bench comment it had split.
With --max-context 262144 the QSA score rows have a stride of 65538 blocks; a prefill batch at 131K actual context
fits the register top-k kernel's 33792-block limit, but CUDA dispatch tested the allocated capacity and chose the
reference kernel. The supplied active_blocks bound is now used on sm_75 only; other CUDA cards keep the capacity
rule, HIP its own policy, and decode (no bound) is unchanged. STRATA_TOPK_CAPACITY_GUARD restores the capacity rule.
qsa_topk_active_parity: 16 cases at stride 65538, selected ids bitwise.
Squashed from #512 (its commit and its merge of upstream 0.1.36).
Co-Authored-By: Codex <[email protected]>
PleReader's row_bytes has been a runtime parameter since the FP8 table
(160 B rows) needed it -- not 90-specific anywhere in the actual read/
cache/job-building code (RowCache, Job/Use, the straddle-length formula
in issue()). ngram.cpp's "Q5_0 PLE requires --ple-io mmap" refusal
predates that generalization (or was just never revisited): FP8's 160 B
rows already go through PleIo::Direct with no such gate.
ple_reader_test --selftest now runs its full synthetic suite (decode-
shaped tickets, page-straddling rows, dedup/edges, bulk, two in-flight
tickets, fault injection, keep-alive -- both threaded and caller-thread
modes) at row_bytes=110 as well as the production default 90, before
touching the refusal. Both: OK.
ple_reader_test --gguf against the real OrcaRouter Q4_K_M shard (the
same one from the Q5_0 kernel PR): direct vs mmap over 312 real PLE
tickets -- bit-identical, and direct is ~8.6x faster (461 vs 3959 us/
token) on this SSD, matching the ~2-2.6ms/token mmap cost the original
"sixteen serial page faults" comment in gather() measured for the
IQ4_NL table.
End to end on the running model (engine 0.1.32, RTX 3090): same text/
math/vision outputs as --ple-io mmap (deterministic sampling), with a
consistent decode speedup (~9-12% across three real generations) and
image-prompt PLE throughput up from not practical to measure to 185
tok/s -- the full per-token win is smaller than the isolated 8.6x since
PLE is one part of a token's cost, not all of it.
is_iq()/the GU/D/MMVQ format lists had Q4_K, Q5_K, Q5_1 and Q8_0 already,
but not Q5_0 -- a plain `llama-quantize` "_M"/"_S" tier GGUF (not Unsloth's
dynamic method) commonly puts Q5_0 on down-projections, sometimes gate/up
too, and native_expert_supported() refused those layers outright ("this
engine has no GPU kernels for").
vec_dot_q5_0_q8_1/dq_q5_0 port vecdotq.cuh's integer chain and the
dequantizer verbatim (symmetric format, so -- unlike Q5_1 -- there's no
min-term rounding choice to make). Fmt<6> plus the row-bytes/is_iq/
dq_dispatch entries follow the existing Q5_1 (type 7) pattern.
native_expert_parity --synthetic q4_K/q5_0 and q5_0/q8_0 (both now in the
CMake test matrix): 0 failures, GPU dequant bit-exact against to_float,
gate/up and down rows bit-exact against ggml's own vec_dot. q5_0/q5_0
(Q5_0 in both roles) is left out of the matrix -- stacking the same
lower-precision format on both roles pushes the GPU path to ~3.6% rel
error against the float reference, just over this test's 3% threshold;
that's compounded quantization noise (CPU rel error is the same ~1.25%
baseline as q5_0/q8_0), not a kernel correctness issue, and no quantizer
I've seen actually picks Q5_0 for both roles of the same expert.
Verified end to end on a real model: a community Q4_K_M GGUF (gate/up
Q4_K, down Q5_0 on several layers) that previously failed at layer 6
now loads and serves correctly on an RTX 3090 (engine 0.1.32).
gdn_rec_cols_pipe_kernel is bound by shared memory, not by arithmetic: every thread of a warp needs the same
32 q and k values per token (the k twice), and a warp receives one such broadcast value per clock. The new
kernel runs column c of the three value heads that share a key head (head % 16) in one thread, so every q/k
value it loads feeds three heads and the token's k row is loaded into registers once for both of its uses.
Its inputs come in blocks of 8 tokens, copied to shared memory by cp.async while the block before computes,
and a token needs 2 __syncthreads instead of 5. 64 blocks instead of 192. Per value head and column the same
arithmetic in the same order: gdn_rec_parity checks output, state and FP16 output bit for bit (T from 1 to
4099) and times it: 18.8 -> 13.0 ms per layer at 32768 tokens.
The output norm stored the normalized value in FP32 as well, which nothing reads (the out projection takes
the FP16 copy): 4.65 -> 3.05 ms per layer at 32768 tokens.
Together 845 -> 579 ms per 32768-token block over the 36 DeltaNet layers; in the engine +2.3 % prefill
tokens/s on 32000-token prompts and +2.2 % on 64000-token prompts, the same greedy answers. Measured on 0.1.32
on a 4080 SUPER (80 SMs); on 0.1.31 the same change gave 1.42x, 1.51x and +2.1 % / +2.0 %. CUDA sm_80+
(cp.async) on a card that holds all 64 blocks at once; otherwise, on Turing and with STRATA_GDN_KEYHEAD=0 the
kernel as before.
Squashed from #413 (5 commits: the kernel, gdn_rec_parity and its --bench SM bands; the 48-63 SM exception
added and taken back).
On Turing mode 4 would run as pairs of m16n8k8 MMAs, which no parity run has checked yet; RTX 20 cards keep the old
kernel for --kv q4_0 until one does.
The tensor-core prompt attention (D-1) refused Q4_0 pools, so --kv q4_0 read every prompt through the FP32 per-query
kernel. Mode 4 is mode 1 (int8) with a scale per 32 values: each q4_0 block's 16 code bytes become 32 exact int8
codes (code - 8) at gather, its fp16 scale goes to the per-group scales, which are applied in FP32 as int8's are
(q.k per 32-dim group; p.v folds each group's scale into p, two groups per warp). The p.v fold is bounded by the
chunk's largest |scale|: q4_0 scales are signed (ggml's d = max / -8). The caller's rotation of q and of the output
is unchanged. STRATA_PROMPT_ATTN_Q4=0: the old kernel (A/B).
qsa_prompt_attn_parity gets Q4_0 cases with signed scales spread over two decades, as a pool holds them (32K and
1.5K contexts), against FP64 and the old kernel.
With KV streaming (--kv-resident) the MTP drafter's K/V is a ring of its window (kv_mode 2: page p in slot p % n_slots
over a host copy), and draft_kv refused it, so every prompt chunk fell back to the drafter's own pass - one graph per
few rows. The batched path takes the ring with its own page table and host copy (as a streamed main layer appends);
the cells it writes are those the window can still reach, which the ring holds, and one batch never spans more than
the ring's slots, so no two cells of a batch share a slot. STRATA_MTP_BATCH_RING=0: the drafter's own pass (A/B).
STRATA_DRAFT_TIMING=1 (debug): the pass's time, synced.
Stager::work waited for a buffer's DMA only from the generation's kRing-th job on (j >= kRing); a generation's first
kRing jobs wrote their buffers straight away. The previous generation's last DMAs from those buffers are only
known to be done where something else waits for them: the routing sync before a staged layer's start(), or the
sync at the end of a chunk with an MTP drafter or stages. A stream_all chunk that ends without one (no --mtp), or a
ring entry the routing skipped (its copy is never waited for), leaves a window where the next chunk's first jobs
overwrite a pinned buffer a DMA may still read. The event is now waited for on every job; an event never recorded,
or already complete, returns at once.
IQ2_XS with --mmap-experts --resident-budget-gib 16 (12,144 of 48,668 streamed experts through the stager): the
same tokens on a 32K prompt, prefill 6,491 vs 6,546 ms.
From an audit: the upload of a chunk's rows, the event recorded after it and the wait on the other buffer's event
before the SSD thread refills it were unchecked, so a failed record or wait let the thread write a buffer an
upload might still read. A failure now ends the chunk with the CUDA error.
Layer 1 adds the per-layer embedding to every token of a chunk, so its rows must be in. A later chunk's rows were
already read on a thread during the chunk before; the first chunk's were read synchronously before layer 0, so the
GPU idled while the SSD read them (an nsys trace: ~245 ms of a single-chunk 32K prompt). Now the first chunk's
gather starts on a thread too and lands just before layer 1 (ple_land), which also starts the next chunk's.
The reads themselves went 64 at a time: a 32K prompt is 512,720 rows in 213,358 4 KiB reads, ~620K IOPS at queue
depth 64. --ple-inflight now defaults to 256, where the drive saturates (1024 is no faster). Decode reads 16 rows a
token and is unchanged. The same rows: the same results.
IQ2_XS, RTX 5090: a 32K prompt 5986 / 6013 -> 5843 / 5808 ms, 8K 1714 -> 1624 ms; first-token logits bit-identical.
In the streamed walk every expert put a cross-stream wait (its copy) and an event record (its ring slot back) on the
compute stream around a ~3 us gather. Under WDDM each pair left ~10 us of GPU idle even when the copy had long
landed: an nsys trace of a 32K prompt showed 226 ms of idle between back-to-back gathers while the host was 3-11 ms
ahead of the GPU. Now a group of 16 is gathered by one launch (copy16_group_kernel) after one wait on its last
streamed copy - the copy stream is in order - and one event releases all of its ring slots: the issuer waits on
used[used_of[slot]]. A ring entry the routing skipped inside an open group first gathers what the group holds so
far, so no more than a group's entries are ever held back from the issuer. The same bytes in the same group slots.
STRATA_PREFILL_GROUP_GATHER=0: one at a time, as before.
IQ2_XS, RTX 5090, 262K context: 32K prompt -8.6%, 8K -10%, 2K -12.6%; first-token logits bit-identical at 2K, 8K,
16K and 32K and with a 96-slot ring.
- engine (serve mode): GEN key logprobs=K (0..20) writes "LP logprob id:lp ..."
after each "T id", from the verify window's head logits (Verifier::copy_logits;
before temperature, penalties and sampling); absent key, no LP lines
- server_intel.py: pairs LP lines with their tokens (StrataEngine._pump), passes
logprobs/top_logprobs to the engine, and fills OpenAI's choices[].logprobs
(content / reasoning_content) in streamed chunks and the collected reply; the
stop token is not listed. serve/ itself is unchanged.
Tested: a fake engine on hardac (streamed, not streamed, without logprobs,
stop token excluded); on the B70 with the IQ2_XS: same answer with and without,
"Paris" -0.0019 with top-1 equal to the chosen token, no measurable time cost.
expert_w11_kernel: moe_fused.cu's design (GPU grouping, experts read in place, gate/up + SwiGLU +
int8 H, then down) on v_wmma_i32_16x16x16_iu8. A work item = 128 weight rows x a 64-row tile, 8
waves of 2x2 16x16 tiles; operands straight from global one block ahead, weight scales in LDS.
The activations take gfx11's K order (act_pos); per 32-value half the integer dot is exact and the
epilogue is the CUDA kernel's. hip_prefill_fused_moe (tests/cuda/prefill_fused_moe_test.cpp) on
gfx1151: vs FP32 reference rel RMS 1.070e-2 (= MMQ's), vs MMQ 3.0e-5; one 2048-token layer 23.2 ms
(MMQ path) -> 11.3 ms. The native packs' fused kernels stay CUDA-only (stubs on HIP).
Replaces the per-head kernel of the previous commit (VGPR-bound, spilled). Same column-split grid
as gdn_rec_cols_pipe_kernel (4 x 32-column blocks per head), but lane g of a column's four holds
exactly the old thread's row group g; kv / o partials meet through wave shuffles and are added in
the old order, so no __syncthreads per token; q/k/v/gate/beta staged 4 tokens at a time (~10 KB
LDS: all 192 blocks resident). The output norm runs the old kernel's body over a grid-stride walk
(T x 48 blocks of 128 threads before). tests/hip/gdn_rec_head.cpp: y, y16 and state bitwise equal
to the old kernels for T 1..2051 (variant 3 = new recurrence + old norm, a diagnostic); 2048 tokens
9.8 -> 4.1 ms on gfx1151 at the test's 600-880 MHz.
One block per value head; each column's 128 state rows in the registers of two lanes of one wave
(the 32-row group partials swapped within the wave and added in the old order); q/k/v/gate/beta
staged in LDS 8 tokens at a time with the next chunk prefetched; the RMS norm + gate of
gdn_out_norm_kernel fused per chunk. tests/hip/gdn_rec_head.cpp: y, y16 and the state bitwise
equal to the column-split + norm kernels for T 1..2051; 2048 tokens 9.8 -> 6.7 ms on gfx1151
(at the 600-880 MHz the GPU's DPM runs this test at). STRATA_GDN_HEAD=0 keeps the old kernels.
gfx11's bf16 WMMA layout: 16 k per lane, C rows 2i + lane/16. On gfx1151 (qsa_select_bench, 131624
capacity): ctx 32768 x 2048 queries scores 14.2 -> 3.5 ms (4.1x), selections identical 2047/2048;
ctx 65536 x 8192 queries 110.1 -> 25.9 ms (4.3x), identical 8192/8192. Still opt-in, as on gfx12.
All 15 prompt GEMM shapes of Qwen3.8-Flash-Next at chunks 2048 and 8192 (T = 2047/2048/8191/8192),
tuned on Aurora (Ryzen AI Max+ 395) with tools/hip/tune_hipblaslt.cpp.
One dpct run of upstream main + the seven PRs, 3-way merged into the files they
touch (sycl/tools/merge_upstream.py); their header changes are forked into
sycl/include until upstream merges them.
- #385 stager waits for a buffer's previous DMA on every job (port: polled seq)
- #463 decode waits for an adaptive swap before reading residency
- #453 drafter's batched K/V on the KV-streaming ring
- #374 first chunk's PLE rows read beside layer 0; port fixes: the next read
starts at c0 + T (the port's first chunk is shorter), and the upload is
marked by a polled sequence number instead of a host event wait, which
deadlocked prompts past ~4K tokens under the Level Zero v2 adapter
- #363 group stride + fused SwiGLU/q8_1 for the PCIe expert call, re-applied
to the port's lane kernels (fp contract off, as upstream's HIP form)
- #407 --adapt-tuned (opt-in), #413 key-head DeltaNet recurrence (opt-in via
STRATA_GDN_KEYHEAD=1: 8% slower on the B70; its NVIDIA gate and SM-holding
parity bench not ported)
Tested on the B70: native_grouped_parity, iq_multi, gr, qsa, kv_hybrid,
quantize_act, qsa_prompt_attn parity pass; Coder 19/2,184-token, IQ2_XS and
40K outputs identical to 0.1.33; Coder 2,184-token prompt 784 -> 825 tok/s;
128K int8 KV streaming 895 / 67.0 tok/s; IQ2_XS decode +1.7% (#363).
Co-authored-by: sergqwer, constantindjonkam, architectds, BlueKingMuch (upstream PR authors)
Port of the Aurora experiment (mmq-resident-rowcount-sorting-v1, +13% 8K prompt on gfx1151 over
0.1.30 + Lt, identical output ids there): each 16-expert MMQ group's max_rows is close to its
experts' own row counts, so less tile padding. Host-only; slot/src/off rebuilt before the upload.
gfx11's v_wmma_f32_16x16x16_f16 takes all 16 k per lane (lanes 16..31 repeat 0..15) and
returns C rows 2i + lane/16; the kernel loads its fragments in that layout. Opt-in as on gfx12.
The probe timed one 1 GiB burst; an x16 PCIe 4 laptop (RTX A3000) read 18.5, 6.9 and 5.8 GB/s
in three starts, and the low reading set pcie_frac (0.12-0.15 where calibrate.py found 0.35).
The same 1 GiB is now four bursts of 256 MiB, each timed on its own (events between them on
the stream; on Windows HIP the host clock, #377), and the fastest is the link's figure: a
disturbance only ever slows a copy, while the median would still follow one lasting through
half the bursts. The probe copies no more than before.
The share was base above 20 GB/s and base * bw / 26 below (19.9 GB/s -> 0.42, 20.0 -> 0.55;
4.0 -> 0.08, 3.9 -> 0). It is now base * min(1, bw / 20): continuous, unchanged from 20 GB/s
up (an x16 PCIe 4/5 link reads ~26-28 GB/s, so the default stays 0.55 there), in proportion to
the link below, so the link's time on its share stays about what the x16 share costs. The rule
is one function for CUDA0 and the layer split's stages.
The log names every burst: "PCIe probe: 28.9 GB/s host->device (best of 28.9 28.9 28.9 28.9)"
(the RTX 5070 here; pcie_frac 0.55 as before).
Noted, not changed: a given --pcie-frac is every stage's share and skips the stages' probes.
The engine already leaves --vram-reserve-mib MiB of VRAM free (default 700). INSTALL.md now says what it does, its
default, and how to set it (the flag, or "--vram-reserve-mib", "2048" in the config's args). Setup's new flag writes
it into the config only when given (one value: it replaces the images' own 700, with a note when it is less than the
GPU image encoder needs); on a start of an installed model it is saved into that model's args, and a new copy set up
from an earlier install keeps it. The default config is unchanged (golden test). Tests in test_setup_choices.py.
The engine refuses --resident-budget-gib with a layer split, so setup always dropped UD-Q4_K_XL to one GPU. With RAM
>= the GGUF size + UNSLOTH_RAM_LEFT_GB (~135 GB) and 2+ usable cards, setup now asks (one GPU stays the default and
--yes keeps it; an explicit --gpus is honoured; an explicit --resident-budget-gib keeps one GPU) and the split writes
"gpu": [..], "layer_split": "auto" without the budget: all the experts loaded into RAM at start, the files through the
OS file cache (the config #498 measured: 2x RTX 3090, 31 -> 64-78 tok/s). Latent bug: START-HERE --gpus 0,1 (or the
start-time offer, or a hand-edited config) on a UD-Q4 config kept the budget and the engine exited with code 2; the
split now drops it, or setup stops before saving when the RAM is too small. docs/UNSLOTH_Q4.md says so. Tests in
tools/test_setup_unsloth.py; the golden configs are unchanged.
hf() takes the host from HF_ENDPOINT (default huggingface.co, a trailing slash dropped); the pinned revisions, the
fallback to main and the SHA-256 checks are unchanged. tools/mtp_fetch.py reads the MTP tensors from the same host.
Step 5 prints the folder setup expects the model files in and how to use files downloaded by hand (that folder with
their original names, or --gguf-dir), and the mirror when one is set. INSTALL.md says the same. Tests: the endpoint
in hf()/hf_unpinned()/mtp_fetch, and step 5's words over a mocked install. The golden configs are unchanged.
The server window said "loading the experts into RAM (about N GB)" whatever the flags. Now: --mmap-experts maps them
from the model files (the OS file cache reads them, nothing is loaded up front; its "experts via mmap" line triggers
the message when the model has no PLE line), --resident-experts copies the ones the GPU does not hold, and
--resident-budget-gib N copies up to N GiB. setup's start banner says "maps" for the plain mapped mode too. Tests:
the words per flag set, and the narrator over a mapped and an arena log.
Every start failure, not only the draft-head case (#474), now carries the engine's own reason: the last 20 lines
this start wrote to the engine log (never an earlier start's). The failed start also closes its pipes and its
handle on the log. Tests: the tail itself, and a fake engine that exits with a reason in its log.
A request whose engine prints nothing for engine_silence_s (default 300 s; while a prompt is read, a chunk may take
3x the previous one's time, the first up to its tokens at 50 tok/s more) ends with EngineSilent (an EngineDied): the
engine is killed and the next request starts it again. The STOP drain, under the request FIFO, is no longer an
untimed wait. "engine_silence_s": 0 keeps the old behaviour. Tests: unit (silence, PP chunks, first chunk, STOP
never acknowledged, 0, config) and over HTTP with a fake engine process (non-stream, stream, stop token).
On Windows, hipMemGetInfo (ROCclr's PAL backend) reports the card's size
minus this process's own allocations; what the desktop and other
programs hold on the card is not subtracted. On a card that also drives
the desktop the engine counts the ~930 MiB those hold as free, and
--expert-cache auto fills the card past what Windows lets the process
keep in VRAM: WDDM moves part of it out to system RAM. On an RX 6800
(HIP SDK 7.2) with setup's configuration: a 9.83 GiB cache, the card at
16132 of its 16368 MiB, 929 MiB more of the engine in system RAM than
with this change, 30.1-31.0 tok/s decode. On a 0.1.30 build,
--vram-reserve-mib 1500 left the card as full (16037 MiB), since the
reserve is taken from the same figure.
Windows gives every process a video memory budget that does account
for the others (DXGI QueryVideoMemoryInfo). On that card it is 799 MiB
below the card's size while the process is small and about 1.8 GiB
below once it holds 11 GiB; allocations past it still succeed.
cudaMemGetInfo, which the HIP shim maps to hipMemGetInfo, now lowers
hipMemGetInfo's free figure by what the budget withholds (the card's
size minus the budget), on Windows only. What is sized from it then
stays within the budget: --expert-cache auto sizes the cache from it,
and the existing check after the slots are written shrinks it ("only
8 MiB free once the slots are written ... shrinking"): an 8.05 GiB
cache, the card at 15230 MiB, 40.6-41.5 tok/s.
Lowering HIP's own figure, instead of taking the budget minus DXGI's
usage, keeps memory the HIP runtime has freed but still caches counted
as free, as HIP counts it.
dxgi.dll is loaded at run time, so nothing new is linked; the adapter
is found by the device's LUID. Without dxgi.dll, the adapter or a
budget below the card's size, hipMemGetInfo's figure stands. The first
query logs the budget once. STRATA_WDDM_BUDGET=0 turns it off. CUDA
and Linux builds are unchanged.
The PCIe probe sets the share of the missed experts the GPU reads over
the link (--pcie-frac): the native default 0.55 from 20 GB/s up, less
on a slower link (#44). On Windows HIP it timed its copies with events
that do not bracket them there: an RX 6800 read between 3,348 and
25,999 GB/s in 24 engine starts, so every link kept the x16 share. A
standalone copy of the probe on that card: its events measure 0.13 ms
for the 1 GiB, the host clock 43 ms (25.1 GB/s).
On Windows HIP the probe now times the same copies on the host, between
two synchronizes (1 GiB, tens of milliseconds): the same RX 6800 on
PCIe 4.0 x16 reads 27.2 GB/s in every run, which keeps 0.55 there. On
a PCIe 4.0 x8 or 3.0 x16 link the share now shrinks as #44 intended.
CUDA and Linux keep the events.
device, expert_source, native_head, prefill gemm, prefill.cpp, generate.cpp
merged from dpct of 8554b6b and 5b734e4 with sycl/tools/merge_upstream.py (the
3-way merge restricted to files upstream changed, dpct kernel-name hashes
canonicalized to the port's). Version 0.1.35.
The ring rule: 1024 slots for the Q2_0 pack's fused experts (P3), 512 for native packs with fused layers.
Quality (teacher-forced vs the FP16 path, Q2_0): fused as close as MMQ at 8K, closer at 32K (phaseA-tests s18-tf).
src/prefill/moe_fused_iq.cu: the Q2_0 path's design (moe_fused.cu: the grouping on the GPU, a pointer per expert -
cache slot or ring slot, no gather -, SwiGLU and H's int8 rounding in the gate/up epilogue) for the raw GGUF blobs of
the native packs. The load stage decodes the i-quant blocks to int8 in shared memory (llama.cpp's codebooks and sign
tables, held in shared memory; a 32-value sub-block per thread per 64 values of K, read one stage ahead with 16-bit
loads and two 256-value blocks ahead into L2), with each sub-block's scale (per 16 values for IQ2_XS / IQ2_S, which
multiply at m16n8k16). The activations are int8 per 32 values in natural order (quantize_act_native); fragments by
ldmatrix; the magic bias 1.5 * 2^23 enters as the mma's C, so a dot becomes a float with one add. A work item is 256
weight rows x 64 routed rows, or 128 x 128 when the experts have 56-112 rows each (STRATA_PF_FUSED_TILE=64|128).
Wiring (prefill.cpp): a native layer takes these kernels when both its formats are covered, else MMQ (IQ1_M keeps the
FP16 path); the 512-slot ring of the fused path now also for a native pack with such layers. sm_75, HIP and the
default (no variable) are unchanged: on IQ2_XS, IQ3_XXS and IQ3_S the default output is 0.1.35's (256/256 greedy
tokens, 4K and 32K).
tests/cuda/prefill_fused_iq_test (exit 77 below sm_80), RTX 5070, 256 tokens x 64 experts against a double-precision
reference from ggml's to_float, rel RMS / worst row - MMQ and fused equal to four digits: IQ2_S/Q2_0 1.489e-2 /
2.93e-2 (fused vs MMQ 2.6e-4), IQ2_XXS/Q2_0 1.464e-2 / 2.96e-2 (1.9e-4), IQ2_XS/IQ4_NL 1.465e-2 / 2.41e-2 (3.5e-5),
IQ3_XXS/IQ4_NL 1.458e-2 / 2.73e-2 (1.6e-4), IQ3_S/IQ4_NL 1.470e-2 / 2.65e-2 (6.5e-5), IQ4_XS/Q2_0 1.446e-2 /
2.72e-2 (4.4e-5); both tile shapes. One layer (512 experts, top 10), ms, MMQ path -> fused at 2048 / 3584 / 8192
tokens: IQ2_S 10.5 -> 5.2, 12.5 -> 9.3, 21.4 -> 17.5; IQ2_XXS 8.8 -> 4.2, 10.7 -> 7.6, 17.9 -> 13.6; IQ3_S 10.6 -> 5.6,
12.3 -> 8.8, 19.8 -> 18.4; IQ3_XXS 10.2 -> 5.1, 11.9 -> 8.3, 19.1 -> 16.5. MMQ's products alone (no gathers) at
3584 / 8192: IQ2_S 8.9 / 17.9, IQ3_S 8.1 / 15.6: the kernels are about even with MMQ's, the gain is the path around them.
(The test syncs the device after its uploads: a cudaMemcpy from pageable memory on the legacy stream can still be
queued on Windows while a non-blocking stream runs - the grouping then read zeroed ids, every pair on expert 0.
prefill_fused_moe_test's timing part has the same pattern.)
End to end, RTX 5070, speed_135.py's settings, greedy, A (MMQ) / B (STRATA_PF_FUSED=1) alternating, 2 pairs, prompt
tok/s: IQ2_XS 4K 1,228 -> 1,378 (+12%; a cold first A run of 590 left out), 32K 2,107 -> 2,170 (+3.0%); IQ3_XXS 4K
1,001 -> 1,020 (+1.8%), 32K 1,745 -> 1,739 (-0.4%); IQ3_S 4K 927 -> 925 (-0.2%), 32K 1,382 -> 1,362 (-1.4%). The IQ3
packs' prompts here wait on the host copies of their unpinned experts (IQ3_XXS 4K: 5,062 of 22,573 streamed, ~3.0 s
of 3.4 s), so faster kernels change little. Each arm repeats itself (256/256; IQ3_XXS 32K B and IQ3_S 32K A split at
162 / 223, the decode's timing); A vs B greedy agreement, identical prefix of 256: IQ2_XS 204 / 88, IQ3_XXS 182 / 89,
IQ3_S 31 / 256.
- qsa_block_topk: calls of up to 16 queries (decode windows) run on a cluster of 8 CTAs per query
(block_topk_cluster_kernel): keys in shared memory, digit histograms pushed through distributed shared memory,
one cluster barrier per radix pass, a parallel digit search, cluster-wide offsets for the ordered emit. The
threshold and the cut at it are the one-CTA kernels' (same tie rule, NaN keys 0). STRATA_QSA_CLUSTER=0: off.
RTX 5070 per call: 21.9 -> 15.6 us at 32K, 58 -> 18 at 128K, 200 -> 22 at 262K.
- sample_tokens greedy without penalties: a cluster of 8 CTAs per row (sampler_greedy_cluster_kernel), four loads
in flight per thread, results merged in CTA 0 (lowest index on ties, NaN never taken, 0 when none).
STRATA_ARGMAX_MULTI=0: off. 39.6 -> 5.9 us per call at 248,320 logits.
- Both launch through cudaLaunchKernelEx with the cluster attribute (captured in the decode graphs) only where
they run: CUDA, compute capability 9+ (STRATA_EMULATE_CC respected), the kernel's own PTX/SASS at sm_90+ (an
older build's JIT copy is a trap), room for one cluster - checked once per device. HIP and sm_75/86/89 keep the
one-CTA kernels.
- decode_cluster_parity (ctest; skips below sm_90): bitwise vs the one-CTA dispatch and the reference kernel over
434 top-k cases (contexts 7..262,144, every tail weight, 1-16 queries, ties / NaN / +-inf / -0 / denormal / all
NaN) and 216 argmax cases, plus a CUDA graph capturing both (first calls inside the capture); --bench timings.
With P3's smaller buffers the 1024-slot ring fits beside an 8192-token chunk at 32K (borrowed 3,153 of 3,690 slots,
under the 90% cap). RTX 5070, Q2_0 (cfg-q2rel), --prefill auto, prompt tok/s at 512 / 1024 slots (2 alternating
pairs): 4K 1,512 / 1,605, 32K 2,485 / 2,660, 128K (KV streaming) 2,346 / 2,392 - there the chunk falls from 8192 to
6144 and is still faster. Against MMQ (one more pair on this build): 4K 1,299 -> 1,609 (+23.9%), 32K 2,187 -> 2,656
(+21.5%), 128K 2,128 -> 2,454 (+15.3%). The ring does not change the numbers (the same output as at 512 slots).
Without STRATA_PF_FUSED nothing changes (384 / 96 slots, cap 512).
In a layout whose chunks run the fused experts (STRATA_PF_FUSED=1, the Q2_0 pack, a streamed chunk) GU holds the
grouping tables, H the int8 H rows and Xq the per-token int8 activations, and Hq nothing - instead of MMQ's FP32 GU / H
and per-slot q8_1 rows: ~100 KB a token less in the MoE set, which then no longer sets the shared region's size (the
GDN set does). A last chunk below stream_all_min() still runs MMQ in the same buffers, so each keeps MMQ's size for
stream_all_min() - 1 tokens. The ring may grow to 1024 slots with the fused experts (STRATA_PREFILL_RING; the default
stays 512). Without STRATA_PF_FUSED nothing changes: the same buffers, ring cap (512) and events.
The fused walk now counts the experts it reads from the cache in experts_resident (it stayed 0): every resident expert
of the layer, routed or not - at a streamed chunk's size nearly all are.
RTX 5070, Q2_0 (cfg-q2rel), --prefill auto, 2 alternating pairs, prompt tok/s (borrowed slots):
4K: MMQ 1,305 (1621), fused 1,517 (1749), fused + P3 1,512 (1534), + 1024-slot ring 1,605 (2046)
32K: MMQ 2,177 (3071), fused 2,453 (3199), fused + P3 2,485 (2641), + 1024-slot ring 2,660 (3153)
128K (--max-context 131072 --kv-resident 32768): MMQ 2,108 (chunk 8192), fused 2,140 (6144), fused + P3 2,346
(8192), + 1024-slot ring 2,392 (6144)
A 16K chunk still does not fit the 12 GB card: at 8192 tokens the loan is 2,641 of 3,690 slots, ~2.1K of them per
8K tokens.
The fused path launches on batches of a layer's streamed experts, so with the 384-slot ring a layer's last batch
waited for slots its own first batch frees. With STRATA_PF_FUSED=1 (and the canonical Q2_0 layout) the pinned-RAM
ring is 512 slots; without it nothing changes (384 / 96 as before; STRATA_PREFILL_RING still overrides).
RTX 5070, Q2_0 (cfg-q2rel), greedy, 3 alternating pairs, prompt tok/s: 4K 1,275 -> 1,516 (+18.9%), 32K 2,176 ->
2,422 (+11.3%); at 384 slots the fused path gave +4-7% / +3%. 1024 slots: 4K +24-27%, but the 32K chunk falls
from 8192 to 6144 (+6%). The default path's output is identical to 0.1.35's on both prompts.
Strata's own kernels (src/prefill/moe_fused.cu): the layer's input rounded to int8 once per token (a scale per 32
values), the (token, k) pairs grouped by expert on the GPU (counts, offsets, a 64-row tile list - no host sync),
then per batch of up to 128 experts one gate/up launch (mma.sync m16n8k32 s8, the 2-bit codes unpacked in
registers, Q2_0's offset folded into the activation blocks' sums, SwiGLU and H's int8 rounding in the epilogue)
and one down launch into the per-slot rows the existing combine reads. Experts are read where they are (cache
slot or streamed ring slot, a pointer per expert): no gather.
Used only on the streamed walk (chunks from stream_all_min()) of the canonical Q2_0 layout on sm_80+ CUDA, with
STRATA_PF_FUSED=1; it reuses the MMQ buffers (Xq, H, GU). Everything else - the default, native/IQ packs, small
chunks, sm_75, HIP - runs the MMQ path unchanged (the old block is only wrapped in the else branch).
tests/cuda/prefill_fused_moe_test (exit 77 below sm_80), RTX 5070: vs a double-precision reference, MMQ rel RMS
1.144e-2 (worst row 4.15e-2), fused 1.144e-2 (4.15e-2); fused vs MMQ 9.0e-5. One layer at 2048 tokens x top 10
over 512 experts: MMQ path 11.12 ms, fused 3.13 ms.
`--expert-profile-save P` (--serve) writes the adaptive tier's learning as a profile in tools/make_profile.py's
format, byte for byte (read_expert_profile reads it back): every pair ranked - the experts resident in VRAM now
first, then the routing counted since the start (the decayed usage summed at each adapt step, before its decay),
then the profile the engine started from. On QUIT / stdin EOF and every --expert-profile-save-every minutes
(default 10, 0 = exit only) between requests, through <P>.tmp and a rename. Without the flag nothing is counted or
written: `heat` stays empty and every added loop is over it.
Server: "expert_profile_save" (and "expert_profile_save_every") in the config pass the flags, and the next start
begins from the saved file instead of the config's --expert-profile when its header matches that profile's
layers x experts. tests/core/expert_profile_save_test.cpp (CPU only) checks the order and the bytes.
In a git clone they run `git pull --ff-only` (a zip copy is told to download the new zip), then `setup.py --update`:
what a plain start does to an install before it starts the model - the Python packages, the ready-made engine when
MIN_ENGINE is newer (a compiled one when its source changed), each model's config upgrades and its draft subset, the
Windows AMD runtime - and nothing else: no question, no model files, no start. Both scripts run from one parsed
block, since the pull can rewrite the running script. Documented in docs/INSTALL.md (Updating) and the README.
Engine: when the MTP draft head's subset does not fit, it prints what the head needs, the VRAM free, and the smaller
shipped subsets (--draft-vocab cyrillic / en, their MiB from the head's real row size) with how to pick one
(START-HERE.bat / ./setup.sh --draft-vocab en, kept in the model's config). Text only, on the failure path.
Server: a start that ended that way carries the engine's hint (or the same advice for an older engine) in its
"the engine exited before it was ready" error. Setup: a card under 14 GB with the default subset gets a note
recommending --draft-vocab en; nothing is changed, an explicit or saved choice gets no note, and the golden
configs are unchanged.
The engine's `strata serve: prompt ...` line now says "= 0 reused + 12288 of 98179 read in ... (cancelled)" with the
rate from the tokens read, and DONE gets a 15th field: the prompt tokens actually read. The server's /metrics history
and totals count reused + read for a cancelled read (prompt_total keeps the whole prompt); an older engine's DONE line
(no such field) is read as before.
The fill from the GGUF (--stream-experts) read each expert's three slices,
copied the blob and waited for the copy before the next read: ~12,000-18,500
experts with neither the SSD nor PCIe kept busy. Now the slots are admitted
in profile order first (same placement), reads go in file order by up to 8
threads into page-locked batches of 64, and a batch's copies overlap the next
batch's reads. STRATA_FILL_SERIAL=1 keeps the serial fill.
Tested on the B70: cold page cache, Coder fill 23.4 GiB 76.2 s -> 18.8 s;
launch to first token Coder 82 s -> 26 s, IQ2_XS 120 s -> 41 s; outputs
identical to the serial fill.
README and docs/MODELS.md said only "Weaker outside coding". #438 measured Chinese answers wrong or looping on the
Coder where English was fine; both now say plainly that this includes Chinese and other CJK text and point to the
sizes that keep every expert (Q2_0, IQ2_XS, IQ3_S).
With Unsloth's UD-IQ3_XXS or UD-Q2_K_XL in the folder, setup either said only "has no IQ3_XXS model file"
(--family unsloth --model IQ3_XXS) or took the UD-IQ3_XXS file for the GSQ-RCO IQ3_XXS by the size in its name.
Now:
- a chosen shard whose name is a quantization setup has no size for (UD-IQ3_XXS, UD-Q2_K_XL, a K-quant) stops
with "<file> is UD-IQ3_XXS, a GGUF Strata cannot run" and what Strata runs: ISTA-DASLab's GSQ-RCO files (and
Swift 1.5's and the Coder's) and Unsloth's UD-Q4_K_XL only;
- a folder without the chosen file but with other GGUFs lists the ones Strata cannot use and the --family/--model
of the ones it can (an empty folder keeps check_shards' "missing");
- a size the family does not have names the families that do (--family qwen --model IQ3_XXS), and with
--gguf-dir also what Strata runs.
The server already read drafts_offered / drafts_accepted from the engine's DONE line but only returned them in a
response's timings (draft_n / draft_n_accepted). Each /metrics request entry now carries both (null when the
engine did not report them) and the totals keep their running sums since the server started, so a dashboard such
as llm-visuals can show the acceptance. Additive: no existing field changes.
- engine (layer splits only): when a stage's cache cannot fund the chunk CUDA0 alone would choose, a WARNING
names it (CUDA index, name, cache slots, the slots the chunk borrows) and points at the helper cache
(--expert-cache-device1, docs/SECOND_GPU.md). The chunk choice itself is unchanged: fits() is split into
fits_one() with the same tests; one GPU never reaches the new log.
- setup: a pair whose second card is under 12 GB beside a first card of 12 GB or more is still offered, but
the first card alone is recommended (the default with --yes and Enter; offer_together asks "n" by default),
with a note. A named --gpus pair is kept. Equal pairs (2x 8/16/24 GB) are unchanged.
- The automatic drop of the small card from an auto split is not done: stages load their weights before the
boundary search and the exact prompt-buffer bytes need the sessions made after it.
tools/calibrate.py read any "gpu" list as a layer split and added --layer-split auto, so a one-card config with a
helper card for the expert tier ("gpu": [0] + --expert-cache-device1) was refused by the engine before it read a
weight, a "0,2" string never got its split, and split_skip_if_fits was ignored. The tuner now builds the arguments
with serve.server.engine_args, the same as a normal start.
When the tuning fails, setup prints the engine's own last error line from the log (only lines written during the
tuning), and says the failure again where it is not lost: before --calibrate starts the model, and in the
"All set." summary of an install that tuned.
rocm_root took /opt/rocm when bin/hipcc and lib/libhipblas.so* were there and the version was 7.0 or newer. A
runtime-only ROCm 7 (no -dev packages) has both, and the engine's cmake configure then failed on the missing
hip-lang package. The system ROCm now also needs lib/cmake/hip-lang/hip-lang-config.cmake (lib, lib64 or the
multiarch lib folder, where CMake looks) and include/hip/hip_runtime.h; without them setup warns, naming what is
missing, and takes the TheRock wheels path, as it already does for a ROCm older than 7.0.
A fresh copy of Strata set itself up like the newest strata-*.json of another Strata folder and crashed with
JSONDecodeError when that file was empty or cut off. previous_config now takes the newest one that parses as a JSON
object; each one that does not gets a one-line warning naming it, and with none the copy is set up as a fresh
install. Every place setup writes a run config now goes through write_config (a .tmp file, then os.replace), so a
setup stopped half-way no longer leaves an empty config behind.
- Windows: the process working set is trimmed before the resident complement's RAM reading, so the experts
the GPU cache pre-fill touched through the mapping count as available again (standby list); one line
prints the reading before and after.
- --resident-experts (soft): when the whole complement does not fit, a second try keeps what fits, by
profile rank, through the #403 budget path (RAM room minus the 256 MiB margin) instead of falling back
to mmap entirely; a WARNING names it. Same file bytes either way, so the answers are unchanged.
- Only the resident mode calls pin_cache_complement: the default and mmap paths are untouched.
Windows looks for an imported DLL in the exe's folder, then System32, then PATH (where the zip's rocm/bin is): a
driver that installs its own amdhip64_7.dll in System32 won, and the bundled rocBLAS/hipBLAS ran on that runtime - an
access violation on a W7900, hipErrorInvalidDeviceFunction on a 7900 XTX, on the first prompt. setup copies
amdhip64_7.dll and amd_comgr.dll from engine/rocm/bin next to the exes after the unzip, before the device probe and
on every start (so a 0.1.34 install is fixed too); package_windows.py ships them there; the engine and strata-device
log which runtime they loaded. rocBLAS/hipBLASLt stay in rocm/bin (their kernel libraries and .kpack are found from
there).
openai_to_messages and anthropic_to_messages iterated req["messages"] directly, so a client that double-encodes
the array as a JSON string had its characters iterated and the request crashed on m.get. The new _object_list
decodes such a string and requires a list of objects; anything else raises ValueError("messages must be a list
of objects"), which the server answers with a 400. A message's tool_calls get the same guard.
- gguf_expert_source: ExpertLayout::gguf_file is per layer and role
(3 * layer + role) since upstream 0.1.31; the port indexed it by layer, so
experts in shard 2 were read from shard 1 at shard-2 offsets (Swift 1.5:
layers 13-47, NaN logits from layer 13). One fd per layer and role now.
- iq_kernels: vec_dot_iq2_xs and Split<17> took their 16-bit codes through
a uint16_t* to an int2 (type punning the device compiler does not honour);
by shifts now. iq_multi_parity checks IQ2_XS against a CPU decode from
ggml's tables (iq2xs_host_tables.inc) and passes.
- prefill: STRATA_DBG_NAN=1 also reports the experts' fp16 GEMM inputs.
Tested on the B70: Swift 1.5 IQ2_XS answers correctly (48 tok/s raw, short
thinking); the original IQ2_XS output is unchanged (65109a, 58 tok/s).
expert_cache.cpp, expert_source.cpp, fused_gr, qsa_prompt_attn, gr_parity and
generate.cpp merged from a fresh dpct of 2ca1b66 and 8554b6b; files that only
differed by dpct run-to-run noise left alone. Conflicts: --resident-cpu-experts
(upstream's resident_cpu_explicit kept with --stream-experts) and the prompt
attention's capability check (the port's XMX dispatch kept). Version 0.1.33.
Tested on the B70: build clean; gr_parity and qsa_prompt_attn_parity pass;
Coder short/long, IQ2_XS and the 40K prompt (borrowing) outputs identical,
78.3 / 75.7 / 58.0 / 68.9 tok/s. setup_intel.py --check and an install run, and
server_intel.py with the mock engine, against the new setup.py / server.py.
The hang was in the streamed-expert prompt path, which borrowing is the only
way to reach on the Coder: the stager threads waited on the copy queue's
events to reuse their 16 page-locked buffers, and under the Level Zero v2
adapter that left another queue's barrier on the same event waiting forever
once the ring wrapped. The copy queue now writes a sequence number into
page-locked memory after each DMA and the stager polls it.
- prefill.cpp: Stager::issued_one / buffer_free; STRATA_PREFILL_SYNC=1 debug
switch (wait and log after every phase)
- generate.cpp: borrowing is the default above a 32K context again
Tested on the B70: 40K prompt with borrowing 1,117-1,144 tok/s, decode 69
(output identical to the run without borrowing); 128K int8/q4_0/k8v4 decode
66.6/62.8/66.8 (from 59.5/53.4/54.4), 256K 55.9/51.7/52.2 (from
48.7/43.2/42.3); Coder and IQ2_XS 32K outputs unchanged.
llama.cpp's tile choice aborts the process ("J_best=0") when no tile size of a type fits the card's shared memory.
The plan now asks the same question for every visible GPU (the experts' gate+up and down shapes, the draft layer's
Q8_0 projections) and keeps a product with no tile on the existing FP16 path, saying which type, card and limits.
Where every product fits (all cards tested here), the plan is the same as before.
#429 (jkuepker): the HIP engine has no prompt kernels for its Q4_K/Q5_K experts and it has not been run on AMD -
--check says so, and setup asks (--model UD-Q4_K_XL --yes tries it). #425 (jctaborda): the finished shards and the
.part files already on the disk count against the download's size.
A non-streamed request wrote nothing until its end, and a streamed one only a keep-alive per prompt chunk (the
first write after a hang-up usually still succeeds), so a dropped request kept the engine reading the whole prompt
or generating to max_tokens while the next request waited. A watcher per request checks the client's socket for EOF
every 0.5 s and cancels; the engine then stops at its next prompt chunk.
dpct of 6ac1a2c and 2ca1b66, normalized, kernel-name hashes canonicalized to
the port's, then git merge-file into sycl/: 49 files, 10 with real conflicts.
- verify: upstream's set_commit_async/wait_commit on the port's deferred
commit (wait_commit = commit_finish); copy_logits on the queue
- fused_gr: upstream's split/staged read variants merged but unused by the
port's read; their per-card self-test segfaults on the B70, so on SYCL it
runs only with STRATA_HC_CHECK=1. The port's split-norm kernel renamed
(upstream has one of the same name); two colliding kernel names renamed
- fixups.py: fused_gr's per-block local-memory query
- hand files: mtp.cpp, verify.cpp, ple_reader.cpp (keep-alive reads + trace)
- qsa_prompt_attn_parity turns on the XMX kernel it checks
- version 0.1.32
Tested on the B70: build clean; Coder (19 / 2,184-token prompts) and IQ2_XS
outputs identical to 0.1.31 at the same speed (78.1 / 75.7, 58.5 tok/s);
40K prompt 1,201 tok/s, decode 65.1; kv_hybrid, qsa_prompt_attn, qsa, gr,
kv_q4, kv_q8, rope, router, sampler, quantize_act parity pass.
Every file outside sycl/ (and docs/INTEL.md) is now upstream's, byte for byte.
The Intel pieces that used to edit shared files move under sycl/ and wrap
upstream from outside:
- sycl/setup_intel.py: runs setup.py's main() with the GPU check, engine
step, RAM rule and config writer swapped for the SYCL port's
- sycl/serve/server_intel.py: serve/server.py plus the Arc's Monitor
readings (sycl/serve/xe_telemetry.py) and the optional model menu
(sycl/serve/web/switcher.js)
- sycl/tools/gpustat.py (was tools/intel/), sycl/.gitignore; migrate.sh
writes its compilation database outside the checkout
- removed: the llama.cpp engine path (serve/engine_llama.py), the README box
Engine sources in sycl/ still track 0.1.31; the 0.1.32 kernels are not
migrated yet.
- fixups.py: default-queue memcpy/memset in the parity tests first waits for
all device queues (cudaMemcpy's legacy-stream semantics), applied to every
src/kernels/*_parity.cpp
- generate.cpp: the post-prompt refill loops sync the queued copies every 256
entries so the ring cannot wrap over in-flight slots
- kv_hybrid_parity: enable STRATA_PROMPT_ATTN_XMX (step 5 needs the XMX path)
Tested on the B70: build clean; qsa_parity, gr_parity OK.
The recommended answers stay the defaults, and --yes / Enter write the same config as before
(tools/test_setup_golden.py, five PCs x five sizes). What a user asks for beyond the recommendation is kept and
its risk said:
- #406: the RAM rule for IQ3_XXS / IQ3_S (experts + KV + 24 GB) is part of the recommended context (the smaller of
it and the GPU's rule), no longer a cap: an explicit --context, a pick in the list or the earlier install's
context is kept with a note ("256K with IQ3_S needs ~78 GB of RAM by setup's estimate; this PC has 64. Kept as
you chose ..."), and the list marks the sizes past it. The low-RAM mode keeps its KV cache in VRAM, so there
the context does not count against RAM.
- #364#384: the low-RAM mode recommends one GPU (the resident variant has no layer split) and asks: 2 = all the
chosen GPUs with the experts read through the OS file cache (--mmap-experts, layer_split auto; faster in both
reports, RAM can fill up during long prompts). An explicit --gpus (or the earlier install's cards) is kept;
--yes keeps one GPU. The GPUs' share counts every card. The one-GPU text no longer sends users to --low-ram off.
At start, --gpus on a --resident-experts config (and offer_together's "y", whose default is now n for such a
config) switches it to --mmap-experts with a note instead of an engine refusal.
- S4: --resident-budget-gib N for UD-Q4_K_XL (default: the RAM less 24 GB, as before); more is kept with a note,
and a given N is not lowered for KV streaming.
- confirm_risk(msg, explicit, yes): one helper for the stops a user may take on - asked with default n, and with
--yes an explicit flag is the consent (--yes alone keeps the stop): UD-Q4_K_XL under 48 GB (--model), the RAM
floor (--model), the paging question (--model, besides --low-ram off), a card under 8 GB named with --gpus.
- --kv-streaming on|off|auto (auto: the RAM test, as before); on past the RAM test with a note, never with k8v4 or
under WSL.
The refusals that stay are the impossible ones: rope none past 262144, disk space, engine version, driver < 580,
no AVX2, a size outside the family.
The low-RAM mode's resident variant has no layer split yet, and the engine refused the pair. When the residency
comes only from setup's soft --resident-experts (no --resident-cpu-experts, no --resident-budget-gib, no remote
caches), it now prints a WARNING - the experts the GPUs do not hold are read through the OS file cache and RAM may
fill up during long prompts - and continues exactly as --mmap-experts: the resident flags are cleared before
anything reads them, so the source, the placement and the per-stage caches are the mapped mode's (#364's hand-edited
config). An explicit --resident-cpu-experts, a budget, or remote caches are still refused.
The explicit --expert-cache reduced under a layer split is now a WARNING that names --vram-reserve-mib.
FileExpertSource::pin_cache_complement read the available RAM twice: the budget was clamped to the first reading
(less the 4 GiB headroom) and the resident-memory check then failed it on a second reading a few MB lower, so a
64 GB PC with setup's --resident-budget-gib 40 could exit at start. With a budget, the check now reuses the reading
the budget was sized from, and a clamped budget stops 256 MiB under the room. An unclamped budget holds the same
experts as before (the lend region is off with a budget, so `budget` there is unused).
If the budget still cannot be kept (no RAM reading, an allocation that fails), generate.cpp warns and serves every
expert the GPU does not hold from the files through the OS file cache (--mmap-experts) instead of exiting; the
complement is left unbuilt on every failure path.
setup.main() with --yes, and with Enter for every question, on 64 GB + 32 GB, 32 GB + 2x24 GB, 47 GB + 2x16 GB,
96 GB + 16 GB and 128 GB + 24 GB, for the recommended size, Q2_0, IQ3_XXS, IQ3_S and UD-Q4_K_XL: the run config
recorded by this setup (tools/test_setup_golden.json), so the choices that follow cannot change the default.
With STRATA_GR_V3=1 "fused GR multi max-T=8 LDS launch and changing graph replay" failed: the test compares the
multi-token read with the single-token kernel bit for bit, and v3 sums `down` in another order (row group x stream
blocks). It also compared `lo`, the default kernels' workspace between down and up, which v3 keeps in shared
memory and never writes. Under v3 the test now checks R_out, rs, inject and mixed within 2e-6 of the largest
reference value and leaves `lo` out; without it, bit for bit as before. Passes both ways.
open_sized seeded every layer's cursor at slot 0, so all layers wrote the same slots and the startup check failed;
the profile fill ended at the first full layer. Now each layer starts in its own range, a full layer skips only
its own pairs, and per-layer mode keeps slots of the largest blob (its ranges ignore the rank a sized slot was cut
for). The default (shared) cache is unchanged.
The host check compared the major version only (>= 7) while the device code traps below sm_75, so an sm_70 card
took the m16n8k8 kernel and hit __trap(). sm_75 and newer: the same choice as before.
START-HERE.bat on a Windows PC whose AMD Radeon is the card Strata can use now
sets it up like an NVIDIA one: no compiler, no ROCm install, only the AMD driver.
- Detection without any AMD software: the display adapters present
(Win32_VideoController: name + PCI id) with each one's 64-bit VRAM size from
the display-class registry (readable without admin; #247/#325's source),
the arch by PCI device id, else by name. An integrated or older Radeon is
listed and refused, as on Linux. The NVIDIA-or-AMD choice is the Linux one:
AMD by itself when no NVIDIA card can be used, asked when both can.
- The engine: strata-windows-x64-hip.zip from the release (this version's,
then the latest; --prebuilt for a local folder), unpacked into engine/ and
checked for this card's arch and version >= 0.1.33; an older one is
replaced on a later start like the NVIDIA zip. Without a published zip setup
says how to make one (tools\hip\build_windows.bat + --prebuilt).
- Before the model download, the engine's own strata-device --list-devices
(with its rocm/bin on PATH) must see the card - otherwise it stops and
points to the AMD driver - and gives the card's HIP ordinal (#325: an
integrated Radeon is HIP's device 0). From then on setup lists the AMD
cards in HIP's order, so "gpu" in the config and --gpu N are HIP numbers.
- The config: backend hip, lib_dirs = engine\rocm\bin, the hipBLASLt table
for the zip's hipBLASLt version (BUILD.json). Images on the CPU and
several AMD cards (--gpus) stay Linux-only for now (said, not silent).
- Texts: setup's description and step 1 name NVIDIA or AMD; START-HERE.bat
needs "an NVIDIA or AMD graphics driver"; the "AMD backend runs on Linux
only" stop is gone.
- An installed HIP engine's BUILD.json ("gfx1201" archs) is no longer read as
an NVIDIA one by get_prebuilt.
Linux is unchanged (KFD detection, TheRock wheels, compiled here).
Tests: tools/test_setup_amd.py WindowsDetection (mocked WMI/registry rows,
the list-devices parser on this PC's real output, the iGPU-first ordinal
match, the driver-missing stop, the zip install/refusals, the dispatch);
test_setup_choices' Linux CPU-encoder case pins WIN=False.
Co-authored-by: Dimitri Vasdekis <[email protected]>
Co-authored-by: isshin <[email protected]>
build_windows.bat builds strata.exe + strata-device.exe (and with `tests` every
HIP test) for gfx1100;gfx1101;gfx1102;gfx1200;gfx1201;gfx1030 with no admin
rights and no AMD GPU: ROCm 10.2.0a20260930 from AMD's TheRock wheels
(whl-next: rocm[libraries,devel,device-<arch>...], the version #325 ran on an
RX 9070 XT) in a private venv (.rocm-win, ~7 GB), ROCm's clang as host and HIP
compiler with MSVC's headers/linker from vcvars, STRATA_PORTABLE (AVX2 ggml).
package_windows.py makes dist/strata-windows-x64-hip.zip (524 MiB, 810 MiB
unpacked): strata.exe (61 MiB), strata-device.exe, BUILD.json (backend hip,
archs, ROCm and hipBLASLt versions, lib_dirs rocm/bin), rocm/bin with the DLLs
the programs import followed through ROCm (amdhip64_7, hipblas, rocblas,
rocsolver, libhipblaslt, libtensilelite-host, origami, rocm_kpack) + amd_comgr
(loaded by name) + the MSVC runtime, rocBLAS/hipBLASLt kernels for these archs,
rocBLAS's kpack device code (found at ../.kpack from rocblas.dll), licenses and
a NOTICE. The user needs only the AMD driver (the HIP runtime uses its
user-mode components).
Checked on this PC's Ryzen iGPU (gfx1036, a test build of the same tree): with
only the zip's rocm/bin on PATH, strata-device lists the card and
hipblasGemmEx BF16 -> FP32 (T=100) matches the CPU (max err 4.7e-7); ctest
52 of 56 pass, the 4 failures the same as #325's RX 9070 XT run (hip_handoff,
and ple_parity / expert_parity / pool_test without the pack fixture).
- prefill gemm: on Windows hipBLAS can return success with the right BF16/FP16
product and still leave hipErrorInvalidValue set (#247 on gfx1201: hc up once
T >= 96, the router), which the next kernel check turns into an exit. That
one stale error is cleared after a GEMM that succeeded; any other error
still stops the engine. Only under __HIPCC__ && _WIN32: on Linux a stale
hipErrorInvalidValue is a real error and keeps being reported (our review
of #247). CUDA compiles a ((void) 0).
- native embedding (HIP builds): when the runtime refuses a device alias for
the mapped table, the table goes into VRAM (as when it cannot be pinned)
instead of failing the start; the pinned copy is freed so the destructor's
one-of-two rule holds (#325's version leaked it and freed dev_owned_ twice).
CUDA keeps the old code path.
- tests/hip/mapped_alias (#325): what a hipHostGetDevicePointer alias is on
this stack. On Windows it is the host pointer itself, and a D2D copy into it
does not land - hip_handoff's timeout there; kernels reading it work (the
engine serves on Windows). Rewritten to read the kernel's output back with
hipMemcpy (the PR's version dereferenced device memory on the host) and to
try the kernel read in both cases. Diagnostic: exits 0, 77 without a device.
Co-authored-by: Dimitri Vasdekis <[email protected]>
Co-authored-by: isshin <[email protected]>
On Windows the AMD cards are found in the display-adapter order, but
HIP_VISIBLE_DEVICES counts the devices the HIP runtime enumerates, and an
integrated Radeon takes HIP's device 0 and pushes the discrete card to 1 - so
the engine was pointed at the iGPU, which the arch check then refuses (found
by dvasdekis on an RX 9070 XT + Ryzen iGPU).
- device_count() and device_summary(): enumerate without device_info's arch
check (the cards this binary has no code for are part of the answer).
- strata-device --list-devices: every device the runtime enumerates, as
"device N: <name>", then "arch gfx1201, 15.9 GiB, wave32" (CUDA: compute
capability and VRAM), then "cannot run: <why>" for a card this binary
cannot run. setup.py parses it (next commit).
- server.py hip_visible(): a one-card config's "hip_ordinal" (written by
#325's branch) wins over "gpu"; a layer split keeps its list.
Checked on this PC's own Ryzen iGPU through TheRock's Windows HIP runtime:
"device 0: AMD Radeon(TM) Graphics / arch gfx1036, 28.1 GiB, wave32 / cannot
run: ... not an architecture this Strata engine was compiled for (...)".
Co-authored-by: Dimitri Vasdekis <[email protected]>
CMake refuses to mix cl.exe with Clang for HIP, so on Windows ROCm's clang++
compiles the host code too (targeting the MSVC ABI, with MSVC's headers and
linker from vcvars). The CUDA->HIP shim is force-included with /FI when the
host front end is MSVC-style (cl / clang-cl) and with -include otherwise, its
path in forward slashes; CMAKE_HIP_ARCHITECTURES may also be space-separated.
On Windows the shim keeps <windows.h>'s min/max macros out (NOMINMAX,
WIN32_LEAN_AND_MEAN).
Taken from jagsan-cyber's #247 as ported by dvasdekis in #325 (2373c75); the
arch lists and the arch check stay 0.1.32's. Built on this PC with ROCm
10.2.0a20260930 (TheRock wheels, the version #325 ran on an RX 9070 XT) for
gfx1100;gfx1101;gfx1102;gfx1200;gfx1201;gfx1030: strata.exe and all 253
targets with the tests. CUDA builds do not read these files.
Co-authored-by: Dimitri Vasdekis <[email protected]>
Claude Code, Claude Desktop, Cursor, VS Code, Codex ... add it over stdio (JSON-RPC 2.0, MCP 2025-06-18, also
2025-03-26 / 2024-11-05); standard library only, so it runs before setup made .venv. Eight tools: strata_status
(running server via /health, /v1/status, /status, /metrics; installed configs, engine, data folder; hardware and
setup's recommendation), strata_models, strata_install (setup --yes --no-start in the background, plan first and
confirm=true to start, progress / cancel on later calls), strata_start (the run script's command, waits for
/health), strata_stop (POST /unload, then ends the server it started), strata_logs, strata_benchmark,
strata_connect_info.
Every argument is checked against a schema and setup's own tables; the only path (data_dir) must be the data
folder, inside the Strata folder, or a new Strata* folder outside system folders. Processes are tracked by pid and
start time in .strata-mcp/, so only processes it started are ever ended; a Strata started elsewhere is only
unloaded. docs/MCP_SERVER.md has every client's config; docs/DETAILS.md links it. Tests use a fake setup and
server (Windows and Linux).
README.md (279 -> ~180 lines): what it is, speed on an NVIDIA and an AMD baseline PC side by side (the AMD
numbers are TODO-AMD placeholders until the RX 9070 XT baseline is measured), what you need with NVIDIA and AMD
as equals, install by asking an AI assistant (docs/AI_SETUP.md, MCP server) or by hand, pick-by-RAM table,
using it, the four common problems, how it works with the two diagrams, Buy Me a Coffee button below the speed
table and at the end. Its anchors other pages use (#which-model-should-i-pick, #install, #using-it,
#something-went-wrong, #how-does-it-work) are kept.
Everything the README had moves, nothing dropped: docs/INSTALL.md (requirements, Windows, Linux, AMD cards,
several cards, Docker, updating, where things are stored, setup's questions incl. rope scaling / images /
speed projection, calibrate, all options), docs/MODELS.md (speed per size, sizes, will it fit, low-RAM, Coder,
Swift 1.5, Unsloth UD-Q4_K_XL, Orca), docs/TROUBLESHOOTING.md (the README's problems plus AMD ones),
docs/HOW_IT_WORKS.md (the longer explanation, credits, license). New docs/AI_SETUP.md (instructions for an AI
agent installing Strata for a user) and AGENTS.md. .github/FUNDING.yml (buy_me_a_coffee: strataengine).
AMD_HIP.md loses "Experimental" in its title and intro; DETAILS.md lists the AMD cards and points to the new pages.
--backend's help, the NVIDIA-or-AMD question and the GPU line drop "experimental" (the Unsloth model, the
speed projection and the Pascal/Volta build keep it); the AMD choice says images run on the CPU (0.1.32)
instead of "no images". setup.sh's header comment likewise.
Re-migrated (dpct of 0.1.29 and 0.1.31, 3-way merge per file): 45 files
changed upstream, 38 conflicts resolved; verify.cpp, mtp.cpp and the new
conversation_state.cpp ported by hand; 7 new upstream files.
Fixed on the way:
- the port's grouped expert kernels stay the default: upstream's new multi
kernels (8 rows a group) ran on the port's 32-row grid and wrote a quarter
of each expert (end-of-text output); STRATA_EXPERT_SPLIT=1 runs them with
their own grid. Lanes per row now selectable (STRATA_GU/DOWN_LANES)
- dpct's unparenthesised __fadd_rn: the native QSA indexer's per-token
pooled key (decode), the native QSA scores and the s2 activation rounding
replaced instead of added - from the first migration; fixups.py fixed
- doorbell waits: fixups.py bounds upstream's new spin spelling again
- prompt-slot borrowing hangs since upstream's rework: off by default
- the Turing PTX branch in qsa_prompt_attn, dp4a.hpp's __nanosleep, the
coupled-draft capture signature, a duplicate kernel name
- gr_parity: the single-token GR read runs the multi kernels (bitwise)
- qsa_parity: a cross-queue copy race in the test itself
Decode 78.2 / 75.7 tok/s (Coder), 58.6 / 64.2 (IQ2_XS); 128K-256K complete.
Two changes to the prompt path's QSA selection (qsa_select.cu), measured on a Radeon AI PRO R9700 (gfx1201) with
--max-context 262144:
1. qsa_block_topk picked its register kernel by max_blocks, a CAPACITY (--max-context / 4 + 2). With a 262144-cell
capacity every prompt, even a 4K one, took the slow reference top-k (up to ~10x slower at 128K+). The prompt path
now passes the batch's active block count (the argument qsa_block_scores already takes); decode passes none and
keeps the capacity rule and the original register width, so it is unchanged. On HIP the register kernel is
instantiated for 66 blocks per thread (contexts up to 270,336 cells, the 262144 --max-context) and, below
~28K cells, where it measured 0.2-0.9x of the reference on gfx1201, the reference is kept. Same ids as the reference
(checked 256/256 at every size).
2. qsa_block_scores_tc on gfx1200/gfx1201: block_scores_wmma_kernel on v_wmma_f32_16x16x16_bf16 (wave32). Each operand
is split into three bf16 slices (hi / mid / lo are exact 8-bit mantissa slices of the fp32, so no scaling is
needed) and the six products of order <= 2 are summed; the hi*hi chain keeps its own accumulator and the five
corrections another, added once at the end (a single shared accumulator rounded the corrections at the big
accumulator's ulp: 8e-7 of the score scale, failing the harness's gate at short context; split: 1.6e-7, the warp
kernel's own error is of the same order). The tail block n_bid is the warp kernel's arithmetic. Like the TF32
kernel it is not bitwise with the warp kernel (a near-tie can select differently). Gated at compile time
(__gfx1200__/__gfx1201__) and run time (gcnArchName); other HIP targets keep the warp kernel; CUDA is unchanged.
qsa_select_bench now builds on HIP, takes the engine's capacity as a 4th argument, checks the scorer against an FP64
reference (gate: no worse than 4x the warp kernel's error, floored at 1e-6 of the score scale), exits non-zero on a
failed gate, and is registered with ctest (32K context under a 262144-cell capacity).
R9700, synthetic bench, 256 queries, capacity 262144: scores 0.81 -> 0.18 ms at 32K, 3.5 -> 0.65 ms at 131K, 6.8 ->
1.5 ms at 262K; top-k 3.9 -> 0.51 ms at 131K, 12.9 -> 1.3 ms at 262K. End to end (Swift 1.5 IQ3_XXS, --kv int8,
prefill timing, same binary with STRATA_SELECT_OLD=1 STRATA_TOPK_OLD=1 as the old arm): 107.7K-token prompt 79.5 s ->
59.5 s (1.34x; qsa select 23.3 s -> 3.4 s); 30.9K 1.02x; below 10K about 1.01x. ctest 44/44.
dp4a through the plain signed v_dot4_i32_i8 (RDNA2 has no sudot4).
setup knows the card and the gfx103X-all TheRock index.
docs carry a community report (RX 6900 XT 16 GB: complete build, 28/28
ctest, 38-42 tok/s decode at 131,072 context, 330-339 tok/s prefill)
docs/AMD_HIP.md: the RDNA4 validation note still said there was no gfx1201 table and none was shipped; it now says
that holds for hipBLASLt 1.4.1 and points to the 1.5.0 table and its measured gain. The "Tuning table" text no longer
says the parity test "exits non-zero when the table does not fit". It describes what the test checks (architecture and
version through the loader, two of the 32 rows against hipBLASEx) and what it does not: a solution id the library
rejects falls back to hipBLASEx and the test still passes, so it says to read STRATA_HIPBLASLT_VERBOSE's
"fallbacks=" summary and to recalibrate for another 1.5.0 build.
tests/hip/prefill_hipblaslt_gemm.cpp: the header comment, the SKIP and missing-row messages and the case comment now
say it is a smoke test over two rows, and it prints "smoke test: 4 cases on 2 of the table's N rows" at the end.
Test behaviour and exit codes are unchanged.
Clarify usage of hip_prefill_hipblaslt_gemm and tuning process for different 1.5.0 builds.
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Both helpers built their BF16 inputs with hip_bfloat16(x).data. Compiled as host C++ (no __HIPCC__) the SDK's
hip_bfloat16 is a struct holding a uint16_t, and the conversion from a float is not the engine's round-to-nearest-even
BF16 encoding (kernels/bf16_bits.hpp has that one).
Observed effect on gfx1201 with ROCm 10.2.0a, parity test with the 100500 table: the T=37 beta=1 BF16 case printed
relative_l2=0 max_abs=0 with the old encoding and relative_l2=3.6e-07 max_abs=2.6e-06 with this one; the T=4096 BF16
case prints 0 either way, and the f16 cases are unchanged. I did not inspect the old encoder's bit patterns directly,
so the exact fault is inferred from that difference, not shown.
Use strata::kernels::bf16_from_f32() in both, so the BF16 rows of gfx1201-hipblaslt-100500.txt were calibrated on
intended inputs and anyone regenerating a table gets the same. No change to the engine or to the f16 path.
tools/hip/gfx1201-hipblaslt-100500.txt: 16 dense GEMM geometries (bf16 and f16) at T=4096 and T=8192, 32 rows,
calibrated on an R9700 with tune_hipblaslt (16 candidates per row, relative L2 <= 1e-4, max abs <= 1e-2, no padding
writes) against rocm-nightly-gfx120x-all 10.2.0a20260914, hipBLASLt 1.5.0 (version number 100500).
The engine refuses a table of another hipBLASLt version, so the existing gfx1201 table in #262 (100401, hipBLASLt
1.4.1) is not used on this ROCm; setup picks tools/hip/<arch>-hipblaslt-<version>.txt by the installed version, so both
can be shipped side by side. Compared with #262's 24 rows this table has the same 24 shapes (with solution ids chosen
for hipBLASLt 1.5.0: 13 of the 24 ids are the same) and 8 more, all bf16 (N=1/K=2560 at T=8192, N=4/K=10240, N=512/K=2560
at T=4096, N=2560/K=2560 at T=4096, and N=10240 with K=2560 and K=320 at T=4096/8192).
docs/AMD_HIP.md lists the shipped tables and how to check one.
With STRATA_ROPE_TABLE unset the table was skipped at run time, yet the compiled default kernels changed their
results: the model-level parity run (Swift IQ2_XS, tools/conversation_cache_parity.py, STRATA_STATE_HASH) gave the
same tokens but other GDN / indexer / K/V state hashes than 0.1.31's (bisected to #280). native_rope's apply,
the indexer's append / append_first / append_blocks and the prompt path's rope_kernel are templates on TAB now;
the host launches <false> whenever no table applies, and that instantiation has no table read at all. Checked:
the same tokens and every state hash as round 1 with the whole branch.
- sycl/include/dpct/LICENSE.TXT: the SYCLomatic helper headers' license
(Apache-2.0 WITH LLVM-exception; their SPDX notices already name it)
- sycl/third_party/ggml/LICENSE: ggml's MIT notice beside the migrated copy
- tools/intel/gpustat.py + gpustat.service: the root sampler the Monitor
tab reads load and VRAM from on xe (finds the card, sizes VRAM from its
BAR; GPUSTAT_PCI / GPUSTAT_VRAM_MB override)
- dpct reports and .yaml sidecars removed and ignored
- docs/INTEL.md: no references to a private test rig; the sampler and the
model_switcher setting documented
Conflicts: setup.py (upstream's reworked low-RAM/paging/rope logic kept, the
Intel backends guarded out of the experts-in-RAM rules as before; meta, the
CUDA/HIP build stamp, only read where the resident variant uses it) and
serve/server.py (upstream's /load, /unload and idle-unload beside the port's
/switcher and request logging). The SYCL copies are refreshed next.
- telemetry: on an Intel Arc (xe) the GPU readings come from a root
sampler's /run/gpustat.json (load, VRAM: xe exposes them to root only)
and sysfs (temperature, power, cap, the PCIe link the card trained at,
capped by the slot). NVML stays first; nothing changes on NVIDIA.
- a run config may name a model_switcher RPC (one model at a time on a
shared GPU): GET/POST /switcher report and request a swap, and the web
app shows a Model menu with the models served on this port; it waits
for /health to name the new model, then reloads. setup.py keeps the
setting when it rewrites a config.
k8v4 (KV in VRAM, no streaming): 995 tok/s prompt / 65.8 decode at 128K,
804 / 51.2 at 256K with 994 experts served from the host mirror.
API: 48-72 tok/s on a short chat answer, prompt included.
The engine sets CUDA_MODULE_LOADING=EAGER (a kernel loaded mid-prompt found no VRAM), so the three extra mmq
instances (+24 MB of binary) were resident for every model: IQ3_S 2261 -> 2260 slots, Q2_0 3678 -> 3675 (S1's A/B,
IQ3_S story decode ~1.5% lower). Measured with the IQ3_S setup config against 0.1.31 (wt-131-r9 build-release):
- slots 2261 / free 5.25 GiB, the same as 0.1.31 again (2260 / 5.24 with the instances);
- engine threads the same (23 at READY, 25 after a request, 0.1.31 likewise): nothing new runs for a normal pack -
the stager's 32 threads are only for a source with file-read blobs, the verify hook only with STRATA_LOGPOS;
- 5 alternating pairs, greedy story / code / third prompt: 0.1.31 39.5 / 44.7 / 32.0, this 40.3 / 45.2 / 33.3
tok/s (story B >= A in 5 of 5).
The UD-Q4_K_XL prompt is SSD-bound with or without MMQ (29 s per 4K chunk either way), so the released engine keeps
FP16 products for it; prefill_mmq_kquant_test is built with the option.
Summary 77.9 tok/s (19-token prompt) / 74.2 after 2,184 tokens; long
contexts with setup's KV streaming: 66.1 tok/s decode after 128K, 54.1
after 256K (all experts in VRAM). The 'not re-measured' caveat is gone.
- the GR norm ran one work-group per token; one per (token, stream) visits
every element in the same order, so rs and xn are bitwise unchanged
(gr_bench checksums equal), GR read 76.7 -> 71.4 us at 6 tokens
- the gate/up expert dots read their 2-byte-aligned weights as aligned
4-byte loads (page-safe) instead of 16-bit pairs; same ints
- grids in local memory (STRATA_GRID_SLM=1) measured slower: off
Decode 77.9 tok/s (19-token prompt), output tokens unchanged.
The gather wrote 97 rows into a 61-row device buffer (and read them back into a 61-row vector); the next launch
reported the overrun as "iq_dequant_f32: invalid argument". The buffer is max(gathered, table) rows now.
The expert down projection (IQ4_NL in 39 of 48 layers) was the largest
expert kernel: ggml reads four uint32 of the codebook from global memory
per lookup and each weight int as two 16-bit loads. Register constants
and aligned 4-byte loads (the third only when unaligned, so page-safe)
produce the same ints: 80.8 -> 43.1 us per call, output tokens identical.
-DSTRATA_IQ4NL_FAST=0 restores ggml's path.
The summary still described llama.cpp as the Intel engine at 23-25 tok/s.
It now compares NVIDIA, the SYCL port (setup's default once built) and
llama.cpp; the stale 'not ported'/'not done' items, setup's KV choice and
the served speed are corrected, and the long-context table is marked as
measured before the decode work.
S5's branch (f-132-q4) compiles the Q4_K, Q5_K and Q5_1 MMQ instances on CUDA and enables them with the private
define STRATA_MMQ_KQUANTS; with both, moe_mmq.cu would have duplicate case labels. STRATA_ORCA_Q4KS_MMQ now adds only
q5_0 (OrcaRouter's Q5_0 expert downs; HIP included) - supported() and run() take Q5_0 under it.
The engine read the 51.2e9-value n-gram table only as IQ4_NL (ISTA-DASLab's shard 2, 90 B/row), which is 8.1% off the
checkpoint's own values per row (correlation 0.996-0.997 on rows from all 128 shards: same table, same order). The
checkpoint stores it as 128 shards of [2500012, 160] F8_E4M3 plus one weight_scale.
- tools/ple_fp8_pack.py copies those bytes, unchanged, into a one-tensor GGUF: per_layer_token_embd.weight
[160, 320001536] as I8, with strata.ple.format = f8_e4m3 and strata.ple.scale. 51.2 GB, streamed.
- PleTable opens either format; the row size (90 / 160) and the decode (IQ4_NL codebook / an E4M3 table times the
scale) follow the file. PleReader takes the row size at open; its cache sizes itself by it. GGUF reader: type I8.
- A PLE-only table (architecture strata-ple) is not added to the dense-projection sources.
- The start line names the table's format.
- ple_fp8_parity: rows through Direct, Mmap and gather_batch match torch's decode of the checkpoint bit for bit.
First-token KL of the IQ4_NL table against the FP8 one: 0.0009 (1K), 0.0013 (2K), 0.0002 (4K), 0.022 (32K); mean 0.0061, top-1 4/4.
Rebased onto 0.1.31 + #317 + #296 (Niko1221): one row size (`rb`) and one decoder in PleTable for the three table
formats - IQ4_NL, #296's Q5_0 (110 B rows, the mapped reader only, as #296 has it) and FP8 (160 B) - and #317's
SSD keep-alive picks its page inside the table at the table's own row size; 0.1.31's shard list for the dense
sources keeps the PLE-only exception; ple_fp8_parity builds with STRATA_BUILD_TESTS beside #296's checks. The
"PLE on" start line stays 0.1.31's.
- The prompt path streams every expert once per 4K chunk; with the GGUF read in place ~35 GB of that comes from
the SSD as page faults, and 4 copy threads with 16 blobs in flight read it at ~0.6 GB/s. When the expert source
has transient (file-read) blobs the stager now defaults to 32 threads and a ring of 4 x threads (the env
overrides stay); every other source keeps 4 / 16. Measured (4K chunk): 71 s -> 34 s (16/64) -> 29 s (32/128);
16K prompt 273-282 s -> 99 s.
- moe_mmq: Q4_K, Q5_K and Q5_1 in supported() and the dispatcher, their mmq instances compiled for CUDA
(STRATA_MMQ_KQUANTS; HIP unchanged). The dispatch follows eddoursul/Strata 7706c1b; main's prompt path already
picks MMQ per layer. GPU time of the experts' products 4x lower (10 s -> 2.4 s per 4K chunk), not visible in
the wall time on this PC (SSD-bound either way: 29 s with and without); after a 16K prompt the continuation
agrees with llama.cpp at 92.2% (KL 0.029) against 90.2-90.6% (0.038) with the FP16 products. The other
models' formats are not affected (their layers' types decide; none uses these three).
- tests/cuda/prefill_mmq_kquant_test: the four expert formats through MMQ against ggml's dequantized weights
(synthetic, several experts per group, permuted rows, a zero row; rel. L2 <= 0.34%), and Q6_K stays uncovered.
A disconnect in the log now names the client that closed it. The
strata-sycl-dev image inherits a health check on port 8080 (llama.cpp's)
that marked the serving container unhealthy forever.
- FAMILIES["unsloth"] (marked [experimental], not the default) and MODELS["UD-Q4_K_XL"] ("budget": True,
48 GB of RAM or more); HF_REVISIONS pins unsloth/Qwen3.8-Flash-Next-GGUF; families may have more than two
shards ("shards": 4).
- The download resumes as before; then verify_sha256 checks each shard's size and SHA-256 once (remembered in its
finish mark; a wrong file is deleted so the next run downloads it again).
- resident_budget_gib: round(RAM) - 24 GiB (40 on 64 GB, the measured setting), at most the 71 GiB of experts, at
least 8; a KV cache streamed to RAM (64K+) comes out of it. --resident-budget-gib replaces the low-RAM mode for
this model (no experts.bin; --low-ram is ignored with a warning); one GPU, no images, no speed projection; 8K
context recommended on GPUs under 14 GB; no --ple-gguf for a 4-shard file (the engine finds the PLE shard).
- The engine version is checked right after the engine step, before the 111 GB download.
- choices_from_config reads a size with a dash (strata-unsloth-ud-q4_k_xl.json).
- tools/test_setup_unsloth.py: pins, budget, SHA-256 handling, config round trip, and setup.main() end to end with
every outside effect mocked (URLs, pack flags, engine args, old engine, too little RAM, two GPUs + images).
- README and docs/UNSLOTH_Q4.md / DETAILS.md describe it.
The model support itself follows eddoursul/Strata (PR #245).
Same token sequences through Strata (40 GiB budget, greedy, adaptation off, STRATA_LOGPOS_TOPK=20 on the verify
windows) and llama.cpp 3cf0325 (CPU, mmap, top 20 from llama_get_logits_ith): argmax agreement, top-5/10 overlap,
KL over the top 20, perplexity. Code 99.0% / thinking 97.5% (the fork, eddoursul/Strata: 98.2-99.3% / 95.6-95.8%);
after a 16K prompt 90.2% on Strata's greedy continuation and 91.2% on real text, the same real-text positions with
512 tokens of context 89.2% - so not a long-context loss: all positions with a top-1/top-2 gap of 0.5 or more agree
(and every code/thinking position from 0.2), perplexities within 3%.
The two real-weight checks were added to every CUDA build with native experts (a full release build compiled them);
they are test programs. docs/ORCA_Q4_K_S.md: the build flag for them, and that 0.1.31's GPU expert kernels take
Q4_K gate/up with Q5_1 or Q8_0 downs (Q5_0 is not among them yet: the engine refuses such a pack at start, naming the layer).
Load three-shard Uncensored weights while preserving Q5_0 PLE rows
and Q4_K/Q5 expert bytes.
Add native CUDA and MMQ paths with real-weight parity checks.
Rebased onto 0.1.31 (Niko1221): 0.1.31 already loads each expert role from its own shard (the per-role shard column
of native_experts.txt, iq_pack.py, expert_source.cpp, native_expert_parity's multi-shard mode) and has its own
Q4_K/Q5_K/Q5_1/Q8_0 GPU expert decoders, so this PR's gguf_role_file / v4 manifest, its iq_pack.py and
test_iq_pack.py changes and its iq_kernels.cu decoders are dropped. Kept: Q5_1 in native_mmvq (type 7 through
0.1.31's iq_mmvq) and in the BF16 dequantizer, the Q5_0 PLE table (mapped reader), the Q4_K/Q5_0/Q5_1 MMQ
instances under the STRATA_ORCA_Q4KS_MMQ option (OFF), ple_q5_parity / q5_projection_parity and the doc.
#272 made Auto the default, which on a hybrid CPU moves the workers (P-core primaries, then SMT siblings, then
E-cores) and cuts their number to P-cores - 1: a speed change nobody has measured here, on the machines most
users have. The default is All - the same cores in the same order as before on every CPU (Windows: the first
logical processor of each core; Linux: one allowed CPU per (package, core)), and the same worker count - in
Options, ExpertPool, physical_cores() and detect_cpu_topology(). --pool-affinity auto | p-cores opts in; the
hybrid line is printed only then. Usage text says so.
- Add detect_cpu_topology() for Windows (GetLogicalProcessorInformationEx) and Linux (/sys/devices/system/cpu/cpu*/cpu_capacity)
- Identify hybrid P/E-core topology and SMT sibling structures
- Prioritize primary P-cores for worker allocation, reserving core 0 for host thread
- Default pool_workers to (p_cores - 1) on hybrid CPUs when unspecified
- Add --pool-affinity auto|p-cores|all CLI option
- Maintain 100% backward compatibility on non-hybrid CPUs and default configs
iq_embed_rows (97 rows gathered out of order, with repeats) and iq_dequant_f32 on a random BF16 table, every value
compared bit for bit with the exact widening (bits << 16); embed_type_supported(30) / iq_supported(30) /
iq_row_bytes(30). ctest native_expert_parity_bf16_embd.
- tools/embd_bf16_pack.py copies embed_tokens (BF16, as the checkpoint ships it) into a one-tensor GGUF; only the
safetensors shard that holds it is needed (3 GB for Qwen/Qwen3.8-Flash-Next).
- --embd-gguf PATH: NativeEmbed reads token_embd from that file; iq_embed_rows and iq_dequant_f32 decode BF16
(embed_type_supported). Mapped host memory as before (1.2 GiB instead of 0.3 for IQ4_XS), no VRAM.
First-token KL of the GGUF's IQ4_XS embedding against the BF16 one: 0.0015 (1K), 0.0089 (2K; the top token changes), 0.0010 (4K), 0.037 (32K); mean 0.012, ~100x the run-to-run noise. On the NVFP4 fork, whose GGUF embedding is Q8_0, the same comparison gave 0.0025. Decode and prompt speed are unchanged (fork); the start takes ~0.2 s more.
Rebased onto 0.1.31 (Niko1221): NativeEmbed::load takes the shard list (one file for --embd-gguf) and checks the
model's architecture since 0.1.31 - the one-tensor file says "strata-embd", so only its tensor is checked there;
the Q4_K/Q5_K/Q5_1/Q8_0 decoders 0.1.31 added stay beside the BF16 one.
- mtp.cpp record_forward: the `if (st_.kv_rot)` branch rotated K and V and then always appended with
kv_append_q4_step - for INT8 + STRATA_KV_ROT=1 into Q4_0 pools the state does not have. The rotation and the
append are separate now: rotate when kv_rot, then append in the state's own format (Q4_0, INT8, FP16).
- conversation_snapshot.cpp: rotated INT8 K/V is its own snapshot format (+16), so a conversation snapshot or prompt
checkpoint taken with the rotation is refused by a state without it and the reverse ("incompatible K/V
geometry"); without the rotation the format is unchanged. conversation_snapshot_test checks both directions.
- generate: says when STRATA_KV_ROT=1 applies; the q4_0 line's comment back.
INT8 K and V can go through the Walsh-Hadamard rotation --kv q4_0 already uses (kv_q4.hpp): it spreads outlier
channels over the 64-value scale groups before quantization; the queries are rotated too (<Hq, Hk> = <q, k>) and the
output rotated back.
- QsaState::kv_rot: Q4_0 always, INT8 with STRATA_KV_ROT=1 (off by default: the same tokens as main).
- Every writer and reader follows it: decode, the verify window, the prompt path, the MTP drafter, and the drafter's
batched prompt pass (Prefill::draft_kv) - which rotated only for Q4_0.
- verify.cpp: the batched window path (qb) never rotated its queries; it did not matter while Q4_0 was kept off it.
First-token KL to fp16 K/V, 4 prompts incl. 32K: NVFP4 pack 0.0066 -> 0.0051; IQ2_XS 0.0022 -> 0.0054. Hence opt-in.
- The table's angles differ from the fast-math cosf/sinf ones in the last bits, so every output moves: off by
default (the kernels compute the angle exactly as 0.1.31), STRATA_ROPE_TABLE=1 turns it on (replaces the PR's
STRATA_ROPE_LEGACY A/B switch).
- The per-device registration was a raw pointer into the session's arena that outlived it: QsaState::owns_rope marks
the state that built the table (not one that borrowed it with share_rope, like the MTP drafter's), and
session_release() - to be called before a session's memory is freed - clears every device's entry pointing at
it (rope_table_release). The latest session of a device now wins, instead of the longest table.
The native decode/verify rope and the indexer's pooled keys computed pos * powf(...) with cosf/sinf under
--use_fast_math (0.0014 rad off at 32K, ~0.02 at 262K), the prompt path with precise float libm, and the float64
table the session builds was read only by the non-native path. rope_table_set registers that table per device
(mrope.hpp); rope_cs reads it, or computes in float64 past its end.
Measured on main with IQ2_XS: first-token KL of the old angles against the table 0.0045 on a 32K prompt (two runs of
main: 0 to 0.00013 on short prompts). On an NVFP4 pack: 0.0063 at 32K and at 125K. Speed unchanged.
STRATA_ROPE_LEGACY=1 brings the old float angles back for A/B.
Rebased onto 0.1.31 (Niko1221): 0.1.31's rope scaling (#84: linear / YaRN, rope_scaled_angle) is in every rope
kernel and in the session's table (build_rope_table with the RopeScaling), so a kernel reads the table only when it
was built with the scaling it was given, and otherwise - or past the table's end - computes the angle exactly as
0.1.31 does (no separate float64 device path); rope_table_set registers the scaling with the table.
For a per-position comparison with another engine on the same tokens (argmax agreement, top-k overlap, KL on the
top K): docs/UNSLOTH_Q4.md compares UD-Q4_K_XL with llama.cpp this way. Off unless the variable is set; the
eight existing columns are unchanged.
Every request that does not continue the live session parks it, and put() never removed older copies of the
same conversation while make_room evicted oldest-first: a subagent's 9 successive turns (each re-rendering the
previous reply) parked 9 copies and pushed the parent conversation out of 4 slots. drop_superseded() now runs
before make_room (park_current) and in put(): an entry whose DEEPEST checkpoint the outgoing chain (or live
state) still holds - the same conversation a turn back, differing only in the stale reply tail - is dropped
(counted apart from evictions, logged). An entry without checkpoints, of the other steering mode, or whose
deepest checkpoint the outgoing chain lacks (another conversation sharing only the system-prompt root) is kept;
an oversized put drops nothing. The cache is opt-in (#189): with it off nothing runs. Tests in
conversation_cache_test (MAIN -> SUB x9 -> MAIN with 4 slots; the kept cases).
With the AMD backend (config backend "hip") telemetry reads the amdgpu driver's sysfs files instead of NVML:
gpu_busy_percent, mem_info_vram_used/_total, and hwmon temp1_input, power1_average (or power1_input) and
power1_cap. The card is found as setup's amd_gpus() numbers it (KFD topology GPU nodes in order, CPU nodes
skipped, drm_render_minor -> /sys/class/drm/renderD<N>/device), which is HIP_VISIBLE_DEVICES' numbering; a
layer split sums/averages as on NVIDIA. Service.free_vram_mib (min_free_vram_mib) uses it on HIP. NVIDIA
paths unchanged. Test: AmdTelemetry (a fake sysfs tree, two cards and a CPU node). Not yet checked on a real
AMD box.
The second (remainder) GEMM changes what the prompt path computes, so it is opt-in: 2 = all but the
hyper-connection's projections, 1 = all. With 0 every lo pointer is null: no extra buffers in carve()/bytes_needed(),
the PLE batch size as before, and every kernel writes exactly the BF16 images it wrote before.
The prompt path fed BF16-rounded activations (2^-9 relative) to the BF16-weight projections, where decode feeds FP32:
the router (a discrete top-10), the indexer (a discrete block selection), SSM alpha/beta, the shared-expert gate, the
PLE key/value and the hyper-connection. Each now takes x as hi + lo, both BF16 (lo = bf16(x - hi)), in two GEMMs
summed in FP32: ~16 mantissa bits, decode's FP32 x to within ~1e-5.
STRATA_PREFILL_BF16X2: 2 (default) all but the hyper-connection, 1 all, 0 off.
IQ2_XS, first-token KL to mode 1: short prompts main 0.0028/0.0069/0.0009, mode 2 0.0014/0.0024/0.0012; a 32K prompt
0.074 and 0.072 (the hyper-connection is what matters there).
A bigger chunk reads a long prompt in a different order of chunks (other expert loans, other rounding of the
batched path), so it stays out of the default: plain --prefill auto picks from 8192 down exactly as 0.1.31 did (the
PR's "no chunk past the context" rule, which also moved the choice below 8192 for small contexts, now applies only
to the new sizes). auto:N (16384 or 32768) or STRATA_PREFILL_AUTO_MAX with auto sets the ceiling; o.prefill_chunk
starts at that ceiling for the buffer estimates. Both kAutoChunks lists - plan_lend and the serve path's per-stage
fits() - take the same sizes under the same rule.
Every chunk streams again each expert it routes to, so a long prompt read in 8192-token chunks crossed PCIe with
nearly all experts of every layer four times per 32K. The auto list now starts at 32768 and 16384, still bounded by
the lend rule and by --max-context; a request lends only what its prompt needs, so short prompts are unaffected.
IQ2_XS, RTX 5090, 64K context, a 32K prompt (3 runs each): 5,624 -> 6,465 tok/s, 40K -> 18K experts streamed.
NVFP4 pack at 262K: 3,535 -> 5,201 tok/s.
fmaxf(NaN, -65504) is -65504, so #281's clamp turned a NaN product into -65504 and the prompt path went on with a
plausible value where it used to show the NaN (STRATA_DBG_NAN and the NaN checks find it there). isnan(x) ? x :
clamp(x): only +-inf (and finite values from 65520 up, which rounded to inf) saturate; everything else is unchanged.
- prefill/kernels.cu: the SwiGLU products that feed the prompt path's FP16 down projection are clamped to FP16's
range (+-65504) before rounding. A token with a massive activation otherwise becomes inf there and NaN after the
GEMM, where decode's q8_1 path (range ~8e6) stays finite.
- native_gdn_preprocess.cu: the unfused gate softplus used logf(1 + expf(v)), which returns 0 for v below ~-16
(1 + e^v rounds to 1); the fused kernels, gdn.cu and verify_kernels.cu already use log1pf(expf(v)). The two paths
now agree.
On ordinary prompts nothing moves: first-token KL to main 0, 0.00013, 0 (the run-to-run noise).
Some SSDs (here a WD_BLACK SN7100 on Windows 11) drop into a power state
after ~250 ms without a command and stall the next reads 50-150 ms; the
NVMe idle timeouts of Windows' power plan do not change it. In decode that
hits the first read after a pause: at the start of a request, or after a
few rounds whose rows all came from the row cache. --ple-io ram, which
avoids the reads altogether, is not available on Windows.
The reader's worker now reads one page of the table (a different one each
time) when no read has gone out for 100 ms, until 60 s after the last
issue; then the SSD may sleep until the next request re-arms it. A
keep-alive read counts when it completes and stays out of the row-read
counters and latencies; the "ple io" line adds "SSD kept awake by N reads
(slowest X ms)". A keep-alive read that fails, or cannot be submitted,
turns the keep-alive off instead of failing the reader.
STRATA_SSD_KEEPALIVE=0 turns it off, =50 shortens the period;
STRATA_SSD_KEEPALIVE_WINDOW sets the window in seconds. Only with the I/O
worker (the default), not with --ple-sync-submit or --ple-io mmap/ram.
The reader's statistics are now copied and reset under the worker's lock
(snapshot()), since the worker may be counting a keep-alive read. The
selftest checks the period, the window, the re-arm by a ticket the row
cache serves, that rows and read counters are untouched, and that the
caller-thread mode has none.
A scan on that PC (reads after pauses of every length): 0 of 211 bursts
stalled after pauses under 200 ms, 35 of 73 after 200 ms and more, 0 of 68
of those with the keep-alive on.
main already has an opt-in "v3" hyper-connection read (#186, STRATA_GR_V3=1, another summation order); #315's
"v2"/"v3" and STRATA_HC_V2 read as the same thing. Now: the plain read (0.1.31's default), split (the norm per
token and stream) and staged (split + the down projection's activations staged by cp.async), kHcPlain/kHcSplit/
kHcStaged, gr_norm_split_kernel / gr_down_staged_kernel, and STRATA_HC_SPLIT: unset or 2 = the newest the card's
check accepts, 1 = at most split, 0 = the plain read. The check compares against the plain read (launch_multi with
kHcPlain) and the single-token read, never against what STRATA_GR_V3 selects; with STRATA_GR_V3=1 that opt-in read
still runs first, unchanged.
`"aliases": [...]` (or a comma-separated string) in strata-<model>.json: /v1/models lists the model with an
`aliases` field and each alias as its own entry (`alias_of`), /props?model= accepts them, and a request naming
an alias is answered under that name (OpenAI, streamed or not, and Anthropic). Any other name is still served
under the model's own name, as before; without aliases the responses are unchanged. Documented in DETAILS.md.
Test: ModelAliases (serve/test_server.py).
The multi-token read (verify windows, the MTP drafter) keeps v1's
arithmetic exactly - every sum over the same elements in the same order -
and only splits the work differently:
- the norm (v2, v3): one block per token and stream instead of one per
token. Each thread visits the elements of its stream in the order it did
in v1, so every sum of squares and rs is v1's; R' * w_norm is recomputed
and scaled once (the same two roundings as v1's store and in-place
scale). On an RTX 4080 SUPER: 0.58 -> 0.23 ms per 4-token round.
- the down projection (v3): v1's rows per block and lane order, but the
activations arrive by cp.async in half-stream tiles, two in flight, in a
bank-conflict-free two-plane layout, and the next tile's weights load
while the current one is used. A tile is 160 = 5 x 32 chunks, so a lane
still accumulates its chunks lane + 32 q in ascending order. Before sm_80
and on HIP the staging is a plain copy (the same bits).
Verifier::init runs fused_gr_check once per card: v1, v2, v3 and the
single-token read on random bf16 weights and inputs (1..8 tokens, with and
without the pending write), every output compared bit for bit; the newest
variant that agrees runs on that card and the log says which. A card that
has not been checked runs v1. STRATA_HC_V2=0 keeps v1, =2 stops at v2.
The down kernels carry as many tokens per launch as the card's shared
memory allows (v3: 10 KB per token), as v1 did.
Rebased onto 0.1.31 (Niko1221): v1 is 0.1.31's default read unchanged - the TILEV 1280/2560 down kernel (sm_75's
small tile), the pre-Volta capacity rule, STRATA_EMULATE_CC - moved into down_chunk()/launch_multi(); main's opt-in
STRATA_GR_V3 path (#186, another summation order) still comes first in fused_gr_read_multi, and the card's check
calls launch_multi(1) directly, so it compares against the default read whatever STRATA_GR_V3 says.
The commit graph writes the GDN state and conv tail, the QSA indexer's tail/dead/pooled/block_pos and the PLE
history; with #284 it may still run when commit() returns. Verifier::commit records commit_done_ after it, and
Verifier::wait_commit() synchronizes that event (not the device; a no-op when nothing is pending, recursing into a
layer split's later stages). A layer split's first stage keeps the synchronous commit (next_ set).
Audited, and where the wait now is:
- the serve decode loop's end (end of turn, max tokens, STOP/cancel): wait before the live session, the state
fingerprint (STRATA_STATE_HASH) and the DONE line - so the next request's park (conversation snapshot capture),
restore, checkpoint restore and zeroing never see a running commit;
- the prompt read by verify windows (serve): wait at its end - the batched prompt path (other streams), prompt
checkpoints (cudaDeviceSynchronize already) and snapshots may follow;
- the start of each serve request: an event wait replaces #284's cudaDeviceSynchronize (covers error paths);
- the end of a generate run: wait right after the decode loop, before the stats and the dumps (#284 synchronized
only later, after the routing dump);
- unchanged and safe: the next window (same stream), the MTP draft (its own stream; reads the window's final rows
and its own K/V, which commit does not write), the adaptive tier thread (expert slots and residency tables only),
STRATA_LOGPOS / STRATA_DUMP_FIRST_LOGITS (read run()'s head logits after run() synchronized), QUIT and /unload
(the Verifier destructor synchronizes its streams before freeing).
Setup forced images off on the AMD backend. hip_vision() now takes --vision cpu: build_engine_hip builds
tools/vision without CUDA beside the HIP engine (build_vision_cpu, again when its source changes) and records
vision/vision_src in BUILD.json; the start config is the CPU encoder's, as on NVIDIA. --vision yes/gpu stay off
with a hint (no HIP encoder build yet); without --vision nothing changes. Test: tools/test_setup_choices.py.
The commit's cudaStreamSynchronize held the host ~0.45 ms a round. The next window follows it on the same stream and
the drafter (its own stream) reads only this window's final rows and its own K/V, so a single-GPU session
(Verifier::set_commit_async) returns at once. Whatever touches the session from another stream or the host afterwards
synchronizes the device: the start of each --serve request (checkpoint restore, zeroing, the prompt path) and the end
of a generate run; checkpoint saves already did. A layer split keeps the wait, as does STRATA_COMMIT_SYNC=1.
IQ2_XS, 400-token answers, 3 runs each: 128.6 -> 135.8 tok/s decode (+5.6%), the same tokens. NVFP4 pack +0.8%.
The engine/CMake part shipped in 0.1.31. Setup now admits compute capability 6.x and 7.0 only when
STRATA_EXPERIMENTAL_SM60=1 is set (gpu_problem; without it the message says so), compiles the engine with
-DSTRATA_EXPERIMENTAL_SM60=ON for cards below sm_75 (engine_defs), and takes the newest CUDA 12.x toolkit
(find_nvcc(below=(13, 0)); CUDA 13 cannot build sm_60/sm_70) - stopping with a hint when there is none, or
when an RTX 50 card (CUDA 13) shares the model with such a card. Without the variable nothing changes.
Test: tools/test_setup_choices.py (mocked toolkits and cards).
It keeps the last 100 requests' prompts, answers and reasoning in memory. Off (the default): begin_request() keeps
nothing, the streams are not wrapped, GET /api/requests and /api-monitor answer 404. Every record write goes through
_note(), which does nothing without a record (the PR's direct self.record[...] writes in the stream error paths
would have failed with the monitor off). A client disconnect in do_POST is marked and then re-raised, as the server
handled it before. The start line says when it is on (and that anyone can read the history without an API key).
test_monitor: turns it on; MonitorOff checks the default; Anthropic requests ask for thinking with "thinking" (#278).
--gguf-dir built exactly <model>-00001-of-00002.gguf and -00002-of-00002.gguf, so a model split or named
differently (-00001-of-00003, Unsloth's ...-00001-of-00004.gguf) was not found. gguf_dir_shards() takes the
published name when it is there (unchanged for the existing models), else the one first shard in the folder
whose name has the size in it, and names shards 1..N from it as the engine and iq_pack do. Help text updated.
Test: tools/test_setup_choices.py (2- and 4-shard names).
The release zip's strata-vision has no sm_75 code (BUILD.json vision_archs 86/89/120), so on an RTX 20 card
setup fell back to compiling the encoder, which fails on most Windows PCs. prebuilt_vision() now chooses the
CPU encoder (the same program without --gpu) with a warning; --build still compiles a GPU one. The next release
zip should build strata-vision with 75-real as well. Test: tools/test_setup_choices.py.
Since 0.1.25 NativeDense treats an IQ3_XXS/IQ4_XS/Q8_0 GGUF key as native, so for a native pack the loader
skipped the pack's key row and then refused the BF16 row iq_pack --compat-bf16 writes for OrcaRouter's IQ3_XXS
key ("native dense: incompatible matrix blk.1.ple_key.weight"). NativeDense::keep_unquantized_ple_key reads the
key row's code_bits from index.txt (WeightTable::index_code_bits) and keeps an unquantized row in the arena;
NativeDense::load does not upload the GGUF key over it, and the PLE uses the BF16 key as before 0.1.25. A
quantized key row (Q2_0, UD-Q4_K_XL packs) is unchanged; the GSQ packs' GGUF keys are BF16 (not eligible), so
they never reach this path. The reporter's fallback (Gen4536). Test: native_dense_ple_key_test (synthetic GGUF
+ pack, CUDA).
A mirror that ignores the Range header answers 200 with the shard's start (its JSON header); cut to the asked
length it passed the size check and the manifest hashed the wrong bytes, so the drafter accepted nothing. get()
now requires HTTP 206 with the requested Content-Range. The 31 MTP tensors' sha256 at the pinned revision
(de4b8e4) ship in SHA256 (two independent fetches agree; 29 re-read from the revision, the two expert tensors'
first/last 16 MiB): a wrong tensor is fetched again once, then refused. An inventory read from another
repository/revision is read again and its tensors are kept only when they hash right. `mtp_fetch.py verify`
(offline, cached by size/mtime) finds a corrupt install; setup then fetches the bad tensors again and rebuilds
the draft layer. From PR #324: a refused engine archive no longer keeps its zip and .done mark.
Tests: tools/test_mtp_fetch.py (fake checkpoint behind a mocked urlopen, a mirror that ignores Range),
test_setup_pins (refused archive).
- The rebase keeps 0.1.31's requirements.txt install in setup.py and the Dockerfile: jsonschema is not added.
- serve/structured.py imports jsonschema on the first json_schema request, not at server start (a server without it
started fine before #334 and still does). json_object never needs it; json_schema without it is checked to be one
JSON object, and the server window says once how to get full schema validation.
- test_structured: the two schema-violation tests skip without jsonschema; new tests for the object-only fallback
and for "import serve.server" not importing jsonschema. docs/DETAILS.md says it is optional.
- StrataEngine.close() (now also the unload and the restart): QUIT and EOF, then terminate, then kill, each with 20 s
as the unload had (#333 went from QUIT straight to kill after 10 s; an engine unpinning tens of GB can need more).
A process that outlives all three raises EngineStuck and stays owned.
- do_POST caught every RuntimeError as a 503 - EngineDied, GpuBusy and ModelBusy are RuntimeErrors too, and any other
failure was turned into a vague 503. Only EngineStuck is caught now (also around POST /unload, and the idle-unload
thread survives it); EngineDied keeps its own answer.
- test_lifecycle: terminate before kill, EngineStuck, an EngineDied on load answered as such.
- The Origin check of /settings and the MCP tools let any page through when the request carried X-Forwarded-For,
CF-Ray or CF-Connecting-IP (headers every reverse proxy or tunnel adds), and compared hosts without their ports
(a page on another local port passed): any web page could then change the shared settings or run MCP tools with the
user's rights. Removed. The Origin must be this server's host:port, or an origin in the config's new
"trusted_origins" (the web app behind a proxy or tunnel); a proxy header alone opens nothing.
- CORS is opt-in: "cors_origins" in the config (an origin, a list, or "*"). Only /v1/* answers with
Access-Control-Allow-Origin (the origin itself, or "*"), never /settings, /unload or the app's files; the
preflight echoes the headers asked for ("*" does not cover Authorization). OPTIONS is answered 204 either way.
- X-Accel-Buffering: no on SSE stays. origins_of() checks both lists at start (scheme://host[:port], no paths;
"*" only for cors_origins). docs/DETAILS.md; tests for the bypasses, the trusted origin, CORS off/on, the header.
- Implement do_OPTIONS (204 No Content) and add Access-Control-Allow-* headers to allow web-based clients (Open WebUI, LibreChat, browser extensions) to query endpoints.
- Update _own_page to recognize reverse proxies and tunnels (Cloudflare Tunnel, Tailscale Funnel, Nginx) via X-Forwarded-Host, CF-Ray, and forwarded headers, preventing 403 CSRF false-positives when serving remotely.
- Add X-Accel-Buffering: no to SSE streaming responses to prevent reverse proxies from buffering real-time token generation.
- Add unit tests in serve/test_server.py covering CORS preflight, CORS headers, and reverse proxy origin checks.
Co-authored-by: Gemini <[email protected]>
"thinking" absent meant no thinking, also for a request that gives reasoning_budget_tokens (#123), which asks for
thinking with a cap (serve.test_server's test_anthropic_stream failed: no thinking at all). Such a request renders
as before. test_anthropic_thinks_only_when_asked pins the five cases.
restart() set ended=False before __init__, which zeroes max_context and blocks until READY, so a request in that
window skipped load() and failed with a 400 "leaves no room to answer in the context (0)". The engine is now
marked alive by __init__ once READY; prepare() answers 503 "the engine is starting" when max_context <= 0. The
output thread is bound to its own process and queue, so a killed engine's thread no longer marks its successor
dead or writes into its queue. Tests: RestartWindow (a fake strata --serve with a held READY).
park_current dropped the retained K/V whenever can_fit failed and fell
back to a full snapshot capture. The gate duplicates what make_room
decides immediately after, with harsher arithmetic: the slot term fires
when make_room enforces the same cap, and the byte term compares the
with-reuse estimate against headroom that ignores the parked image
replacing the retained buffers. make_room already implements the
stated policy - pressure drops the optional retained storage before
evicting any parked conversation (the priority the cache tests pin).
With a full cache the fallback buys nothing: the fresh estimate forces
the same or more eviction, plus a whole-snapshot recapture instead of
the growth - hundreds of ms per park at ~100K tokens where one long
conversation round-trips while short-lived conversations keep the
cache full.
The with-reuse estimate stays uncapped: it counts the retained buffers'
capacity and directories, and put() charges that same true size, so a
capped figure would under-evict and overfill the budget.
- Add set_coupled_draft() and coupled_draft_state() in include/strata/core/coupled_draft.hpp
- Add --coupled-draft and --no-coupled-draft CLI options to src/program/generate.cpp
- Default option to STRATA_SPEC_COUPLED environment variable for full backward compatibility
- Update usage() help text with option descriptions
Isolated benchmark (no inference-path changes) comparing the production
bf16_gemv warp kernel against an mma.sync.m16n8k8 tensor-core variant with
BF16->FP16 conversion, on the real decode GEMV shapes (n_in = 2560). Three
TC variants (W preconverted / inline conversion / fused output) and a full
k-split occupancy sweep; parity against an fp64 reference.
Result: the TC variant is 1.3-2.2x slower on every real shape even at its
best config - the warp kernel is already bandwidth-bound (374 GB/s) and a
single-token GEMV wastes 15/16 of the m16 MMA tile. Keeping the warp
kernel; the benchmark is checked in so this does not have to be measured
twice.
kProfPer 32 -> 33: fused_gr_read_multi writes three stamps per half and the second
half's base was 30, so its up-stamp landed at slot 32 = (l+1)*32 + 0, the next layer's
stage-0 base. Derived columns (gap/head/hc0) also wrapped to ~1.8e19 ns whenever a
slot was unstamped, printing ~1e14 ms per window. A delta now counts only when both
stamps are nonzero and monotonic; the hc0 split replaces the single hc-read0 column
only when the split chain is complete. Metrics-only: no computed model value changes.
Reason recorded in PORTING.md 19.
An existing experts.bin was kept whenever its size matched the layout. A different checkpoint with the same geometry
(a re-quantization or a fine-tune packed into the same folder) has the same size, so the engine silently served the
old experts with the new model's dense weights.
Before keeping the file, the first and last blobs of the first, middle and last layers are now compared with the
GGUF; a mismatch rewrites it. The check reads six blobs, so an unchanged pack is still reused at once.
Test: test_experts_bin_is_kept_only_with_this_ggufs_blobs packs two GGUFs of the same geometry into one folder (it
fails on the previous code).
Rebased onto 0.1.31 (Niko1221): 0.1.31 already refuses an experts.bin whose sidecar (experts.bin.src.json: shard
names and sizes, the hash of native_experts.txt) does not match, so the six-blob comparison runs only when the
sidecar says "reuse" - it catches the case the sidecar cannot (a re-quantized or fine-tuned checkpoint under the same
file names and sizes). It reads each of the six expert blobs on its own instead of concatenating the whole layer.
The test packs the second model under the first one's file name, so it reaches the new check.
The full setup rewrites the model's config and refreshed the MTP folder's subset with `a.draft_vocab or "cjk"`, so
running it again without the flag put the CJK subset back over a cyrillic (or en) choice, and dropped draft_vocab from
the config. saved_draft_vocab() reads the choice the config holds; the flag still wins. tools/test_setup_draft_vocab.py.
The shipped draft subsets hold 142 of the vocabulary's 18,580 Cyrillic tokens, so an answer in Ukrainian, Russian,
Bulgarian or Serbian drafted almost nothing. data/draft_vocab_cyrillic.bin is draft_vocab_en.bin plus every token
whose text is Cyrillic (58,963 ids), built with the repository's own tool:
python tools/draft_vocab.py --gguf <IQ2_XS shard 1> --base data/draft_vocab_en.bin --add cyrillic
- tools/draft_vocab.py: a "cyrillic" script (U+0400-052F, 1C80-1C8F, 2DE0-2DFF, A640-A69F).
- setup.py: --draft-vocab cyrillic; docs/DETAILS.md.
On the NVFP4 fork: 1.4 -> 2.1 tokens per round, 83 -> 109 tok/s in Ukrainian answers, English unchanged.
- --flash-attn on|off|auto (mtmd's flash_attn_type): flash attention keeps K and V in FP16. On the NVFP4 fork the
vision tower's output was 11.2% off with it on the GPU; FP32 attention (CPU) 0.1%.
- On the CPU a CUDA build keeps the GPU unseen (CUDA_VISIBLE_DEVICES=-1 before the runtime starts): it otherwise
opened a context there, 0.4-0.7 GB of VRAM the expert cache loses.
- On the CPU without --threads: one thread per core (mtmd's default is 4); no warm-up encode (~6 s of start).
- serve/server.py: relative exe/mmproj/model paths in the vision config are the config's cwd's, as the engine's are.
- StrataEngine.restart() runs __init__ again (after a crash, an idle unload's reload), which started one more
STRATA_REQUEST_LINES follower each time and printed every request line once per restart: a follower is started
once per log path (the log is opened for append, so the first follower keeps reading the same file).
- The field the follower printed as "ttft" is the engine's prompt read time (the new tokens only, without the queue,
the template, the first decode step): it is named prompt_read.
Found by running Claude Code against the server (claude -p, three tool turns) and by starting a config with a
relative engine path:
- POST /v1/messages/count_tokens answered 404; Claude Code asks it for its context figures. It now renders and
tokenizes the request exactly as /v1/messages would, without running the model.
- A /v1/messages request with no "thinking" (and no effort) got the template's default, which thinks: Claude Code's
helper calls (a ~60-token session title) came back with a thinking block and no text. Anthropic's thinking is
opt-in, so such a request now renders with enable_thinking false. A config's reasoning_effort still applies;
"thinking" set, or an effort, behave as before.
- STRATA_REQUEST_LINES=1: a thread follows the engine log and re-states each request's summary line on stdout, for
supervisors that only see the server's stdout (a tray, llama-swap). Off by default.
- A relative "exe" in the config failed in Windows' CreateProcess (WinError 2): it is made absolute against the
config's cwd, as the engine's cwd already is.
Tests: count_tokens equals the input_tokens /v1/messages reports for the same request; the request line pattern
parses the engine's summary. python -m unittest serve.test_server: 44 OK.
- generate.cpp: the startup line names "(an unnamed GPU)" when cudaGetDevice/cudaGetDeviceProperties fail (the
error is cleared), and the "no code for" message uses the same name; a HIP build prints the gfx architecture
instead of a CUDA compute capability.
- device.cu: device_code_error() returns "" on HIP - cudaFuncAttributes/cudaFuncGetAttributes have no mapping in
hip_compat, and gpu_arch_problem() already stops a HIP run on an architecture the build lacks.
- fused_gr.cu: STRATA_EMULATE_CC also answers the two compute-capability queries there (small tile, the sm_70
capacity rule), as the PR's description promises; with the variable unset nothing changes.
- generate.cpp: before the expert arena loads, the engine names the card and its compute capability and checks that
the build has code for it (device_code_error: cudaFuncGetAttributes on a kernel). A binary built for other GPUs
used to fail at the first kernel, after the whole arena had loaded; now it stops in ~0.3 s and says why.
- include/strata/core/emulate.hpp: STRATA_EMULATE_CC=75|80|86|89 makes the host's device queries (compute
capability, shared memory per block) answer as that generation, for a PTX build of it JIT-compiled on a newer
card - ggml's MMQ config, the QSA kernels' choice, fused_gr's shared memory. Tests only.
- STRATA_QSA_WARP=1|select|attn: the pre-sm_80 QSA kernels on any card (A/B).
STRATA_DUMP_FIRST_LOGITS=<file>: the logits of the first window's first position (the token the prompt predicts),
n_vocab float32, written once per run. Two engine variants run on the same prompt then give a first-token KL
divergence: how far a prompt-path change moves the model's distribution, not only whether its top token changed.
On a layer split the last stage's verifier writes it.
On RDNA4 the router took 39 us per call in decode (16% of the decode GPU time): 512 dependent FP64 adds, ten
selection passes with 20 block barriers, a renormalisation re-reading its weights from global memory. The sum's only
use is inv = (float)(1.0 / sum): all terms are positive with the largest exactly 1, so the serial sum and a tree sum
are within 2^-43 of each other; when the bracket tree*(1 -+ 2^-40) maps to one float, that float is the serial sum's
(1/x and the cast are monotone), otherwise thread 0 runs the serial sum as before. The selection runs on wave 0 from
registers (the argmax with the lowest index on ties is unique), the renormalisation from registers in rank order.
hip_router_fast checks ids and weights bitwise against the portable kernel (65,536 rows with ties, near-ties, NaN
rows, the forced serial path; and 256 experts / k 8). CUDA builds unchanged (HIP-only code).
The v2 tensor-core design with gfx12's v_wmma_f32_16x16x16_f16: one block per (query, KV head), wave w owns
dims [64w, 64w+64) (int8 scale group w), q and p split into FP16 hi + lo parts, int8 codes exact as FP16, scales in
FP32, the four groups' q.k partials added in a fixed order, online softmax over 32-cell chunks. Masked
(non-resident) cells get no weight, as in the decode kernel. FP32-level accuracy, not bitwise equal to the default
kernel, hence opt-in. hip_prompt_attn_wmma = qsa_prompt_attn_parity built for HIP (skips off gfx12).
NVIDIA builds unchanged (all new code is under __HIPCC__ / the HIP test section).
Each work-group owns a 128-column slice of all 324 rows, stages that
slice of xn once (<= 4 KB SLM) and streams the rows through it, 4 lanes
per row; a reduce kernel sums the 80 slice partials in a fixed order.
GR read 108.6 -> 76.5 us at 6 tokens. STRATA_GR_DOWN_SLICED=0 restores
the direct kernel.
Q8_0 and IQ4_NL blocks (34/18 bytes) are 2-byte aligned: the wide kernels
load them through load16_a2, now page-safe (the second chunk only when the
address is unaligned). Q8_0 4-6x, IQ4_NL 640->2560 4x. Q6_K's Q8_1 activation
int4 loads were 4-byte aligned: four int loads instead.
Output identical for 146 of 256 tokens, then a near-tie flips (summation
order); STRATA_MMVQ_WIDE_32=0 restores the old kernels.
- resident_plan groups the entries with one thread per entry (thread 0
alone took 73 us per layer); the plan is the host loop's exactly
- the GR down kernel reads xn from global memory instead of a 60 KB SLM
tile per work-group (occupancy); outputs bitwise equal (gr_bench)
- 16-byte-load wide kernels for Q4_K, Q5_K and IQ4_XS (IQ4 codebook in
registers): IQ4_XS 107 -> 323 GB/s, Q4_K/Q5_K 1.3-1.6x
Output tokens identical in every engine run.
Q6_K blocks are 210 bytes, so the wide kernel's 16-byte ql/qh loads were
2-byte aligned and the B70 split them: 145 GB/s against 640 GB/s for the
same blocks at a 224-byte stride. load16_a2 does two aligned loads and a
shift; with it the wide kernel takes every Q6_K shape and up to 8 columns
(2.6-3.4x in q6k_align_bench). Output tokens identical.
Also a port copy of ple_reader.cpp with STRATA_PLE_TRACE=1, which showed
the 45/39 tok/s two-speed runs are one 226 ms PLE stall landing in either
the prompt or the first decode round (TTFT + decode is the same).
- docs/UNSLOTH_Q4.md: what UD-Q4_K_XL is (Hugging Face revision 38bb39e, the four shards with sizes and SHA-256),
disk/RAM/GPU needs (111.3 GB download, 1.4 GB pack, no experts.bin), the iq_pack.py command and why
--compat-bf16, the numerical policy (native experts/embedding/head/projections/PLE; the converted small tensors
listed in conversions.json), the MTP draft layer, a server config (--native shard 1, --ple-gguf found by name,
--resident-budget-gib N with N = RAM minus 20-24 GB), measured speed on an RTX 5070 / 64 GB PC (7-8.5 tok/s at
40 GiB; ~60 s to the first answer), the switches (STRATA_LOOKAHEAD=0, STRATA_KQ256=1, STRATA_PARTIAL_PIN=1,
STRATA_FETCH_THREADS), scope and validation, credits (eddoursul #245, gopinath87607 #255, jagsan-cyber #247).
Experimental, not in setup.
- docs/DETAILS.md: the mapped low-RAM mode reads a native pack's experts from the GGUF when it has no experts.bin
(checked, identical answers on the Coder); --resident-budget-gib and the routing-aware prefetch; where the tier
counters are (--stats, the server log, /metrics' ram_blobs / file_blobs / file_mb); a pointer from "Which model?".
On the RTX 5070 / 64 GB PC the GPU computing a share of the misses over PCIe measured no faster than the CPU:
7.30 and 7.44 tok/s with 24 and 16 GiB of a 40 GiB budget registered, against 7.05-7.74 unpinned, and
registering costs startup time and driver memory. The default budget mode is now locked-only; the routing-aware
prefetch (multi-run A/B: 6.93 -> 7.91 tok/s mean, identical answers) stays on with the GGUF in place.
Add preliminary native v0.1.26 results, raw run JSON, scrubbed guest hardware facts, and the local API harness. Prompt and memory topology limitations are documented.
- RouterLookahead: when the CPU pool starts layer l, a worker thread applies layer l+1's router (a BF16 host copy
of ffn_gate_inp, 126 MB for 48 layers) to layer l's MoE input - the router's own input - for each token of the
window, takes the top STRATA_LOOKAHEAD_K (default 10), drops the experts the GPU cache (host_res) or the RAM
copy holds, and asks the source to warm the rest. It never waits: a prediction still running makes the next
one skip. STRATA_LOOKAHEAD=0 is the A/B arm.
- ExpertSource::warm / warms(): FileExpertSource with the GGUF in place issues one PrefetchVirtualMemory for the
predicted slices (madvise(WILLNEED) elsewhere). Pages only: blob(), the experts computed and their bytes do not
change (the same answers in every A/B run). Every other source: no-op, and the predictor is not started.
- bf16_rows_dot_multi (kq_avx2.cpp, AVX2): the router read once for all tokens of the window.
- Stats: experts warmed, how many of the decode's file-tier reads had been warmed, the predictor's time and skips.
UD-Q4_K_XL at a 40 GiB budget (single-token-router version): ~48-54% of the file tier's reads had been warmed;
reading time 850-968 -> 630-730 ms per verify round; decode 5.31/6.60 tok/s off vs 7.08/6.79/6.91 on.
- Locking the whole 40 GiB arena in the working set and then registering its prefix made the next device
allocation fail (the prompt path's token id buffer); the registered prefix is page-locked by the driver
already, so only the rest is locked now. 24 GiB registered out of a 40 GiB budget then starts and runs.
- The registered prefix ends at an expert boundary: a blob that began inside it and ran past it was taken as
page-locked by cudaMemcpyAsync and refused (an adaptive refill failed, with 16 GiB registered).
- The prompt path's token id buffer error names the CUDA error.
- kq_avx2 multi-token kernels: bit-exact, but no faster in the engine (the rows phases stayed at 72 ms per
verify round at a 40 GiB budget; a window's expert groups hold ~1.4 tokens and ggml's per-token calls find the
weights in L1), so they are opt-in with STRATA_KQ256=1; native_expert_parity still checks them bit for bit.
Experts without a VRAM slot are read once into pinned host memory and the
device-built verify plan points the expert kernels at them over PCIe. The
plan used to drop non-resident experts, so output with misses diverged.
Forced 1,879-expert miss: decode 2.6 -> 40.9 tok/s, output identical.
Before any later stage loads, split-auto checks whether CUDA0 alone holds every profiled expert pair plus the whole
session (KV of every layer), the drafter + head, the verify windows and the reserve; if so it clears the split (one
GPU) and logs why, else it logs how much of the profile CUDA0 would hold and keeps the split. Default behaviour is
unchanged: the engine flag is opt-in, and the server passes it only for a config with "split_skip_if_fits": true
(and a GPU list). Not with remote expert caches.
Measured on amdbox (Coder IQ1_M, --expert-cache auto --prefill auto, 128 greedy tokens, gpu [1, 0] = R9700 first,
2 starts per arm, warm = the second request of each size):
| arm | 4K warm | 16K warm | decode |
| split (flag off) | 1,244 | 1,897 | 50.5-53.0 |
| --split-skip-if-fits (on) | 1,776 | 1,791 | 59.7-61.8 |
| R9700 alone | 1,787 | 1,801 | 59.9-61.3 |
The flag's run is the R9700-alone run (same tokens, all four requests); with the 9070 XT first it keeps the split
("would hold only 9.84 of the profile's 23.42 GiB"). On this pair the split still reads 16K prompts ~5% faster.
- The expert arena was pinned at most 8 GiB whenever a layer split (or remote experts) ran, on every OS - a
Windows/WDDM workaround that cost a Linux 4090 + 3060 split two thirds of its prompt speed (#253,
bettercallcaleb). The cap now applies only under WDDM (_WIN32, or WSL2 detected by /dev/dxg), for multi_gpu and
expert_cache_remote alike. STRATA_ARENA_PIN_GIB overrides it on any OS: "0" pins the whole arena, N > 0 at most
N GiB (arena_pin_cap_gib(), -1 when unset: the same name and meaning as the S2 branch's helper, file-local here).
- The split's prompt chunk no longer stops the start when its buffers do not fit: a prompt path without a loan
first gets the largest chunk whose buffers (Prefill::bytes_needed) leave 512 MiB free on its device, and an init
that still fails steps down one size (8192 -> 6144 -> ... -> 2048 -> 1536 -> 1024 -> 512): Prefill::reset() frees
what the failed init made, the loans are re-priced, and the failed allocation's sticky error is consumed (else the
next launch check reported "out of memory").
- pinned.cu: the capped note no longer says "for CUDA1".
- docs: MULTI_GPU.md, SECOND_GPU.md.
Measured on amdbox (Linux, R9700 + RX 9070 XT, gpu [1, 0], Coder IQ1_M, 128 greedy tokens; tok/s):
| cache | arm | 4K cold | 4K warm | 16K cold | 16K warm | decode |
| auto (holds ~all) | 8 GiB cap | 678 | 1,355 | 1,709 | 1,880 | 44.5-51.5 |
| | whole arena | 703 | 1,234 | 1,729 | 1,864 | 44.6-51.4 |
| fixed 1500 on CUDA0 | 8 GiB cap | 608 | 1,101 | 1,287 | | 34.5-36.1 |
| | whole arena | 716 | 1,160 | 1,803 | | 34.8-36.0 |
(2 starts per arm for auto, 5 for fixed.) With most experts streamed the whole arena reads a 16K prompt 40%
faster; with every expert already on the cards the two are within noise except 4K warm (-9%: the pinned share
raises the auto lend to 90% and the streamed ring to 384 slots). Identity: the fixed-cache arms gave the same
tokens in all 5 + 5 starts (4K, 16K, 4K again), and the auto arms 2 + 2. The step-down was exercised on the
9070 XT with --expert-cache 3000 --no-prefill-borrow --prefill 8192 ("need 4528 MiB, 2910 MiB free: 2048-token
chunks"; with the init retry alone it went 8192 -> ... -> 2048, and 1536 ran all four requests). Known, also on
main: that own-buffer configuration at 2048 makes hipBLAS pick hipBLASLt kernels whose code objects this ROCm 7.14
lacks (TensileLibrary_..._gfx1201-xnack+.co not found, "launchKernel failed") and the prompt hangs. The WDDM side (Windows, WSL2) is unchanged and not re-measured here.
The copies' page faults read a few clusters each; one PrefetchVirtualMemory call for every role slice about to
be copied lets the memory manager queue large reads at once, and the parallel copies then find the pages
resident or in flight. Only the GGUF-in-place source's prefetch uses it; STRATA_FETCH_PVM=0 is the A/B arm.
- Registering 30 GiB of a 40 GiB RAM tier left the driver unable to page-lock the prompt path's token id buffer
afterwards (RTX 5070, WDDM: "prefill: the token id buffer"); the prefix is now capped at 24 GiB, the size a
whole 24 GiB budget page-locks with, and STRATA_PARTIAL_PIN_GIB sets it.
- A verify window's expert groups hold ~1.4 tokens on average; one token goes to ggml's own dot (the same bits,
without the multi-token kernel's per-token arrays).
Where the 288 ms/token at a 40 GiB budget went (UD-Q4_K_XL, --stats): of the 945 ms CPU pool call per verify
round (3.46 tokens), 871 ms were the dispatch thread assembling missed experts from the mapped GGUF one after
another (~391 blobs, 1.3 GB per round, ~1.5 GB/s: page faults from the SSD); the kernels were 42 ms gate/up and
30 ms down. At 24 GiB, 771 ms/round more went into the PCIe planning, which asked blob() - a file read with the
GGUF in place - of every miss just to learn it was not pinned.
- ExpertSource::prefetch: the dispatch hands the source the layer's missed experts before its jobs loop;
FileExpertSource with the GGUF in place assembles them on STRATA_FETCH_THREADS threads (default 8) so the
SSD sees a queue instead of one fault at a time. A no-op for every other source. Staging buffers being
filled are never reused, and a blob asked while its fill runs waits for it.
- The PCIe planning asks pinned() before blob(): the same decisions and bytes for every source.
- --resident-budget-gib: when the driver refuses to page-lock the whole budget, the locked arena's hottest
prefix (profile order) is registered with cudaHostRegister, as much as it takes (from the whole down in 2 GiB
steps), so the GPU can read those misses over PCIe and the prompt path can DMA them; pinned() answers for
that prefix only. STRATA_PARTIAL_PIN=0 is the A/B arm.
- src/kernels/cpu/kq_avx2.cpp: ggml-cpu's AVX2 vec_dot for Q4_K x Q8_K, Q5_1 x Q8_1 and Q8_0 x Q8_0 with the
token loop inside the block loop - the weight side decoded once per verify window, each token's integer chain
and float operations exactly ggml's, so the results are ggml's to the bit for any group size (no #152 rule).
native_gu_rows / native_down_rows use them for those types (only UD-Q4_K_XL has them); STRATA_NO_KQ256 is the
A/B arm. native_expert_parity checks them bit for bit against vec_dot for 1..8 tokens and times 4 tokens.
- Stats: the decode's file tier alone (blobs, MB, MB/round, reading ms/round and GB/s per thread) and the
prompt path's copies separately.
--kv-resident 32768 keeps the whole KV in pinned host memory and only the attended window in VRAM, so every
expert stays in VRAM at long contexts. B70, decode after the prompt: 128K int8 8.5 -> 38.7 tok/s, 128K q4_0
32.7 -> 35.3, 256K int8 3.7 -> 31.3, 256K q4_0 5.0 -> 31.5; the prompt pays 5-13%. setup.py now turns it on
from 64K for the SYCL engine (its RAM check counts only the KV copy and the engine, the experts being in VRAM)
and keeps INT8 KV, which is the faster of the two with streaming.
setup refused AMD GPU lists at install and at start ("NVIDIA only for now"), although the engine's layer split runs
on two AMD cards (0.1.30, from a hand-written config).
- amd_parse_gpus(): --gpus "1,0" (setup's HIP numbering, the first card the main one) or "all" (every supported
card, the most VRAM first); every card must be in AMD_ARCHS, else a plain message with what can be used.
- install: --backend hip --gpus writes "gpu": [..] + "layer_split" like the NVIDIA path; build_engine_hip compiles
for the set of the chosen cards' archs (CMAKE_HIP_ARCHITECTURES "gfx1100;gfx1201" already worked) and records
it in BUILD.json; an engine whose archs cover the cards is not rebuilt; update_installed_engine keeps every arch.
- rocm_root(archs): cards of two families need a system ROCm (TheRock's wheels hold one family) - said plainly.
- start: --gpus on an AMD install is saved like on NVIDIA (refused when the engine has no code for a card);
--gpus all finds the AMD cards on a PC without NVIDIA cards that can share.
- docs/AMD_HIP.md, README; tools/test_setup_amd.py: lists, "all", refusals, the one-family wheel rule, the arch set
in the build and BUILD.json.
Cut down from #254 and #256 onto 0.1.30's AMD tables:
- setup.py: AMD_ARCHS / AMD_NAMES / ROCM_INDEXES / AMD_CARDS list gfx1101 (TheRock gfx110X-dgpu) and gfx1200
(gfx120X-all); amd_gpus() takes the sysfs root so the detection can be tested.
- cmake/hip_backend.cmake: gfx1101 and gfx1200 build without the "not validated" warning (a status line names
them community-validated); gfx1102 (#192: ctest only) keeps the warning.
- docs/AMD_HIP.md (Community-validated cards: the owners' setups, ctest and numbers) and README.
- tools/test_setup_amd.py: detection on a mocked KFD topology (every family, the CPU node, an iGPU and gfx1102
listed as not supported, product-name fallbacks, the ROCm index per family).
Left out: the PRs' own cmake/device.cu arch checks (0.1.30's compiled_for covers them), the dp4a change (main has
it), math_constants.h, the bench JSONs, the launcher, and both hipBLASLt tables (neither PR measured the prompt speed
without its table).
Co-Authored-By: Joseph Hohertz <[email protected]>
Co-Authored-By: efeisot <[email protected]>
Cut down from #262 (ttio2tech): only the intrinsics and their test.
- HIP's __byte_perm indexes a byte array in private memory (scratch traffic in the IQ4_XS/Q2_0 table
lookups); byte_perm() is the same selection (nibble & 7 of the pair {y:x}) in one v_perm_b32.
- __vsub4, __vsubss4 and __vcmpne4 work on all four byte lanes at once (Hacker's Delight 2-18) instead
of a per-lane loop.
- tests/hip/intrinsics checks them on 1,024 random operands against host references (passes on gfx1201).
Left out: the arch/setup/device.cu parts (0.1.30 already builds and checks gfx1201; main's dp4a covers
gfx1100-1102/1200/1201) and the gfx1201 hipBLASLt table (none is shipped for gfx1201).
A/B vs main on amdbox (Coder IQ1_M, 128 greedy tokens): identity with a fixed cache (--expert-cache 1500
--prefill 2048), 5 + 5 starts per card, 4K / 16K / 4K again: the modal outputs are identical on the R9700
and on the RX 9070 XT (one start of 20 differed from its arm's mode, the known HIP start drift). Speed with
setup's settings (auto cache/chunk), 2 + 2 starts per card, tok/s:
| card | arm | 4K cold | 4K warm | 16K cold | 16K warm | decode |
| R9700 | main | 1,027 | 1,792 | 1,695 | 1,812 | 46.0-52.2 |
| | #262 | 1,023 | 1,786 | 1,681 | 1,808 | 53.0-61.0 |
| RX 9070 XT | main | 781 | 1,018 | 1,421 | 1,517 | 32.7-41.4 |
| | #262 | 780 | 1,017 | 1,424 | 1,517 | 34.9-44.0 |
Decode +15% on the R9700, +5-7% on the 9070 XT; prompts unchanged.
Co-Authored-By: ttio2tech <[email protected]>
setup.py gets a third backend, sycl: on an Intel Arc with build-sycl-aot/strata and the strata-sycl-dev image
present (or --intel-engine strata; --intel-engine llama keeps llama.cpp), it builds the native pack and the
MTP draft layer (reusing an mtp-q2_0.gguf that is already there), writes the engine's arguments with the
container's paths through sycl/serve/strata-sycl.sh, --stream-experts with a reserve that keeps every expert in
VRAM (1,024 MiB up to 32K, 2,048 MiB and 4,096-token chunks above), Q4_0 KV above 64K, and keeps the sampling
block of an earlier config. start() checks the model files through the config's sycl_root; upgrade_config
leaves the SYCL config alone (the wrapper has no version, which read as 0.1.12 and downgraded --prefill).
Tested on the B70: the Coder's config regenerated by setup serves at 38.9 / 30.3 tok/s (first turn / follow-up)
with all 12,288 experts in VRAM.
The serve loop used to abort at exit (139): the Verifier's destructor waited on its queue after the Level
Zero context had begun tearing down. The destructor is now exception-safe and serve mode exits directly once
its requests are done: exit 0.
The Coder on the B70 now runs Strata's own engine behind serve/server.py --engine strata: the wrapper starts
build-sycl-aot/strata inside strata-sycl-dev with the serve pipes attached and the port's run-time switches set.
Through the OpenAI API with temperature 0.6 / top-p 0.95 / top-k 20 / repetition penalty 1.05: 36 tok/s decode
on a first turn, 33 on a follow-up (conversation cache reused), vs 24-26 for llama.cpp on the same card. The
expert reserve has to leave every expert in VRAM (--stream-experts has no host copy): 1,024 MiB at 32K context.
The selection dump measured how much neighbouring prompt positions' QSA selections overlap at 80K tokens: the
union of 8 is 3.3x one selection (16: 5.1x), 12% shared by all 8, so a grouped-gather attention kernel would buy
at most 5-8 s of an 80K prompt. XMX v2 across the 128K/256K matrix: 23-33% slower prompts than the FP32 attention.
- FileExpertSource: a native pack without experts.bin, given the --native shard, maps every GGUF file
native_experts.txt names (after check_experts_gguf) and assembles a blob [gate | up | down] from the
three role slices when asked: into a small pool of buffers for blob() (a buffer is reused once its blob
has not been asked for during 3 layers (begin_layer) or 256 assemblies), or straight into the caller's
memory for copy_blob(). No 30-77 GB experts.bin on the disk. With experts.bin present nothing changes.
- ExpertSource::transient / copy_blob: the prompt path's stager copies a transient blob itself (its queue
holds a whole chunk's experts), everything else keeps using blob().
- --resident-budget-gib N (the resident mode with a budget): the experts the GPU cache does not hold are
copied into RAM in expert-profile order while they fit N GiB (clamped to available RAM minus the
headroom), the rest are read from the files; no lend region then. Without the flag the resident mode
and --mmap-experts are unchanged.
- Counters: blobs served from the RAM copy, from the files, and bytes read from the files; in the stats
table ("expert tiers"), the serve log, and appended to DONE (RAM blobs, file blobs, file MB for the
request), which serve/server.py puts into /metrics' request history (older DONE lines still parse).
- expert_layout_test: the GGUF-in-place source gives the loaded arena's bytes through blob() and
copy_blob(), an assembled blob stays put while in use, and experts.bin still wins when present.
- Dots: llama.cpp's vec_dot_q4_K / q5_K / q8_0_q8_1 transcribed (VDR 2), via eddoursul/Strata
8029fa9 and #255 (gopinath87607, Q8_0), which agree. Q5_1's min term multiplies the sum of the
QUANTIZED activations (ggml-cpu's convention, as the fork does), not llama.cpp CUDA's ds.y.
- Dequantizers dq_q4_k / dq_q5_k / dq_q5_1 / dq_q8_0 (llama.cpp's formulas) for the prompt path's FP16
fallback, f32 dumps and the Q8_0 embedding; iq_row_bytes / is_iq know the four types;
iq_dequant_gu_f16 now refuses an unknown type instead of leaving its output unwritten.
- The grouped gate/up, down and iq_mmvq switches are generated from STRATA_GU_FMTS /
STRATA_D_FMTS / STRATA_MMVQ_FMTS; the existing entries launch the same kernels as before.
- native_expert_supported(gu, d, n_embd, n_ff): generate checks every layer of a native pack right
after expert_layout_load and refuses an unsupported pair by name before anything is allocated
(it used to exit from inside the first verify window).
- Native PLE key: Q8_0 accepted (NativeDense, generate, ple.cu); native_mmvq already runs it.
- native_expert_parity: --synthetic GU/DOWN (random ggml-quantized blobs at 2560/640; port of the
fork's mode), GPU dequant f32/f16 and embedding rows vs ggml to_float bit for bit, split models via
GgufModel, --q5_1-min. CTests: q4_K/q5_1, q4_K/q8_0, q5_K/q8_0, q8_0/q8_0, the Q5_1 min term, and
q6_K refused. RTX 5070: GPU and CPU within 1.25-1.37e-2 of the float reference for all three real
pairs; every dequantized value identical; Q5_1 GPU vs ggml-cpu 7.5e-5 (vs 1.95e-2 for the ds.y
variant); IQ3_XXS/IQ4_NL and IQ2_XS/Q2_0 unchanged.
- Not yet: #242's decode-once traits for the new types (they take the per-entry kernel; see
phaseA-tests/s5_cs3_design.md), prompt MMQ for Q4_K/Q5_K/Q5_1, identity gates.
`gguf_split_test --real SHARD` opens a real model's shards (GgufModel), checks the architecture on the
metadata shard and prints where output / token_embd / the PLE table and key / layer 11's experts are.
`expert_layout_test --real PACK NATIVE_SHARD` checks a real pack's native_experts.txt against its model
(check_experts_gguf; no expert byte is read). Both pass on the Q2_0, IQ3_XXS (v2), IQ3_S, Coder IQ1_M
and Swift IQ2_XS packs and models on the reference PC.
- FORM (eddoursul/Strata's list of which tensor the engine reads as BF16, F16 or F32) decides the
pack form; the conversion itself is upstream's bf16_bytes (gguf-py, finite check, round-to-nearest-
even). By default only lossless conversions happen: F32 whose values are exactly BF16 (low 16 bits
zero; before, only the routers) and F16/BF16 widened to F32. Rounding (inexact F32, Q8_0 hyper-
connection / PLE value / output_hc, F16 -> BF16) still needs --compat-bf16. NaN/Inf are refused
even where the low bits are zero.
- A Q8_0 PLE key stays native (served from the GGUF, like Q2_0); other quantized keys take the BF16
path as before. (The engine's native PLE key learns Q8_0 in the formats change.)
- blk.1.ple_conv1d stored F32 (Q8_0 / UD-Q4_K_XL files) goes into dense.bin as index kind 3 and the
loader narrows it to F16 (#255, gopinath87607), with or without --compat-bf16; a value beyond
F16's range is refused. generate refuses a pack whose conv1d is not F16 (#255's check, which also
accepts the canonical Q2_0 pack's verbatim-F16 kind 0 - #255 as written refuses that pack).
- conversions.json (schema 1): the source shards' names and sizes, and per converted tensor its
shard, source/destination type, method, exact flag, largest absolute error and source sha256.
compat-bf16.json is written as before. Refusals happen before anything is published.
- Byte identity: index.txt, native_experts.txt and dense.bin of Q2_0, IQ3_XXS, IQ3_S, Coder IQ1_M and
Swift IQ2_XS are identical to main's packer (Swift's 96 exact F32 routers are now listed in
conversions.json).
- Tests: exact F32->BF16 by default, inexact only with --compat-bf16 (error bound), NaN/+-Inf
refused with and without the flag (previous pack untouched), conv1d kind 3 (bytes, record, F16
kept, overflow refused), Q8_0 projections need the flag and are recorded, Q8_0 key native.
- Each expert role is resolved in the shard that holds it and its offset is absolute in THAT shard
(the first role's data_start was used for all three; #255, gopinath87607). A layer whose roles
are in different shards gets the per-role column `gate,up,down` and the file becomes v4; every
other pack keeps the v3 header and bytes (checked on Q2_0, IQ3_XXS, IQ3_S, Coder IQ1_M, Swift
IQ2_XS: index.txt, native_experts.txt and dense.bin identical to main's packer).
- The split keys are checked as the engine does (shard 1 has the metadata; split.count / split.no /
split.tensors.count consistent). The expert table is computed and checked before any file of
the pack changes.
- experts.bin is written under a temporary name and gets experts.bin.src.json (the shards' names
and sizes, the sha256 of native_experts.txt); it is reused only when that matches, never by size.
An experts.bin of another source is refused without --experts-bin (the engine would read it).
- dense.bin and index.txt are published together after the old completion marker
(native_experts.txt) is removed, so a stop part-way never mixes files of two runs.
- .absolute() stays (not .resolve()): symlinked HF snapshot names keep the split naming.
- Tests: a synthetic 4-shard GGUF (shard 1 metadata only, layer 1 split per role; offsets per
shard, experts.bin byte for byte), v3 kept when nothing is split, missing shard, duplicate tensor,
a shard of another split, truncated shard, wrong type (pack untouched), interrupted pack (experts
and index), stale same-size experts.bin. Idea of per-role names also in #247 (jagsan-cyber).
Groundwork for Unsloth's UD-Q4_K_XL (4 shards: shard 1 metadata only, output/token_embd/PLE table in
shard 2, layer 11's down in shard 2 and its gate/up in shard 3). Existing packs load unchanged.
- include/strata/artifact/gguf_split.hpp: gguf_split_paths (from eddoursul/Strata 8029fa9) finds the
shards from any shard's name; a missing shard is an error naming it. It replaces generate.cpp's
model_shards(), which skipped a shard it could not open.
- gguf_reader.hpp: GgufModel (8029fa9) looks a tensor up across the shards and refuses a name in two
shards; the split keys are validated as #255 (gopinath87607) does for the head: shard 1 carries the
architecture, later shards declare split.count / split.no / split.tensors.count consistently, and the
directories add up to split.tensors.count. tensor_payload_bytes / in_bounds for payload checks.
- NativeHead::load / NativeEmbed::load take the shard list: the architecture from the metadata shard,
the tensor from the shard that holds it; the embedding also checks its payload is in the file.
- generate: --native resolves all shards once (native_shards), the head from --native-head-gguf's
family, the model keys from the metadata shard; --ple-gguf defaults to the shard that holds
per_layer_token_embd.weight.
- native_experts.txt: the shard column may be per role, `gate,up,down` (#255's comma form; empty =
the --native shard). The reader accepts v2-v4 and refuses a higher version; an engine older than
this one fails to open a "gate,up,down" file loudly. Fixes the parse's own aliasing hazard
(parts.assign(3, parts[0])).
- load_experts_gguf reads each role from its file, after check_experts_gguf proved every (file,
offset) is that tensor, of the pack's type and dimensions, and inside its file.
- Tests (synthetic, no model, no GPU): gguf_split_test (4 shards, missing shard, duplicate, split keys,
count, truncation); expert_layout_test (copies of the Q2_0/IQ2_XS/IQ3_XXS/IQ3_S/Coder/Swift packs'
native_experts.txt parse field for field as before; v4 column; v5 / 4 / 2 names refused; the arena
from split synthetic shards at 2560/640 with Q4_K/Q5_1 and Q5_K/Q8_0, every byte compared, and
wrong offset / file / type / shape / truncated / missing shard refused).
v2: 64-cell chunks, V packed two cells at a time as 32-byte stores, K^T as 4-byte pairs with INT8 converted on
the load, hi+lo Q in one accumulator per scale group, one accumulator update per chunk. qsa_prompt_attn_parity
passes in every mode and kv_hybrid_parity passes completely with it (its last step needs a tensor-core-class
prompt kernel). 13.4 ms per chunk vs v1's 18.7 and the FP32 fallback's 5.5: the attention is gather-bound, so
it stays opt-in (STRATA_PROMPT_ATTN_XMX=1, or =32 for 32-cell chunks).
INTEL.md: 128K and 256K prompts with int8, q4_0 and k8v4 KV on the B70. All complete; q4_0 keeps every expert
in VRAM at 128K (decode 32.7 tok/s after the prompt); at 256K all three evict experts and decode drops to 4-5.
Without borrowing, the VRAM reserve for the KV state and the chunk buffers evicts ~1,900 experts, the prompt
streams them (790 tok/s at 80,000 tokens) and decode after it runs at 2 tok/s. Borrowing lends ~940 slots and
refills them in ~1 s: 1,062 tok/s and 35-40 tok/s decode. It costs ~1 s per prompt at short contexts (2,184
tokens 610 vs 792 tok/s), so the default is by context: borrow above 32K. --prefill-borrow / --no-prefill-borrow
decide explicitly; layer splits keep their own buffers as before.
-DSTRATA_EXPERIMENTAL_SM60=ON now names Volta (sm_70) beside Pascal (sm_60)
and is a declared option; below sm_60 is refused even then. The runtime check
in device_info refused anything below compute capability 7.5 even in that
build, so the flag produced an engine that could not start on the cards it was
built for: strata_core now gets STRATA_EXPERIMENTAL_SM60=1 with the flag and
the check admits 6.0 and up there. The release engine is unchanged (7.5).
Checked here: configuring for 70-real without the flag still fails with the
7.5 message; with it (CUDA 12.6 - CUDA 13 no longer compiles compute_70, noted
in the comment) it configures and strata-device builds with the define.
- Re-migration merge of upstream 99b4097..0ea2fc3 (GDN recurrence pipelining, block scores read once per window,
the GPU-wide sampler, the draft head's VRAM reserve, correctness fixes); verify.cpp/mtp.cpp ported by hand; two
new fixups (a cudaStream_t cast in the sampler, a const sycl::free in native_head). Parity unchanged (19 of 21).
B70: 2,184-token prompt 765 -> 792 tok/s, decode at that context 43.1 -> 45.2 tok/s, outputs identical.
- The prompt path's stream plan kept GgufExpertSource::blob() pointers for ~1,900 experts per chunk against a
512-slot ring, so blobs were overwritten before they were copied (the first 80k measurement was partly wrong
experts). The stager threads now read (layer, expert) themselves via GgufExpertSource::read_into.
- The stream-all walk hangs in its first large chunk on this card (copy engine stuck on a barrier, any ring size,
either issuer; the old read path too). The port defaults to the per-layer routed-only walk (upstream e258e6e's
STRATA_PREFILL_RING=8, included); STRATA_PREFILL_STREAM_ALL=1 restores it for debugging.
- The PLE reader copy keeps its 64-deep queue. 80,000 tokens now complete: 101 s = 790 tok/s.
Every CPU expert kernel is AVX2 at least (q2_avx2.cpp, iq_avx2.cpp; the
AVX-512 ones are chosen above them) and so is ggml-cpu in the release build,
but nothing checked for AVX2: a pre-Haswell CPU died of an illegal instruction.
cpu_avx2_ok() (AVX, AVX2, FMA, F16C and the OS's XSAVE state) is checked
before the pack's layout load, and the message names the CPU (cpu_name, the
CPUID brand string). STRATA_FORCE_AVX2 is unaffected (it only answers
cpu_avx512_ok).
The #251 report stalled in "reading the prompt, from token 0" and printed the
previous request's verify window as if it were the stuck one.
- Verifier::diag says "(last window, not the current stage)" when the stage
is not a verify window.
- The batched prompt read sets its stage per chunk (preparing, finishing) and
per layer, with the chunk's first position (Progress::chunk, a third
progress_at argument; every other call leaves it -1), so the stall report and
the watchdog line read e.g. "reading the prompt (batched): layer 17 of the
prompt chunk from token 8192".
Text only: nothing computed changes.
A verify window the GPU never finished held the host in an unbounded
cudaStreamSynchronize, and the serve watchdog then aborted the process with
the window's spin kernel (wait_flag_ge / wait_flag_ge_or on a host-mapped
flag) still resident - on Windows the GPU was left "lost" until a power cycle.
- The final wait of Verifier::run polls cudaStreamQuery (the same wait while
the window runs) and gives up after 20 s like the per-layer wait, through
the existing error path.
- Verifier::release_gpu_waits writes UINT32_MAX into the mapped flags M/A/B
(past every ring, so each spin kernel returns; wait_flag_ge_or returns on its
flag too), then waits for the streams up to 5 s; the verifier refuses later
windows. Both timeouts run it before returning their error, and the serve
watchdog and the expert pool's stall aborts run it (progress.hpp
release_gpu_fn, every live verifier) before std::abort.
- Test hook: STRATA_TEST_VERIFY_STALL=N withholds the last layer's flag in the
N-th window; unset it is never taken.
The normal path runs the same kernels in the same order.
When cudaHostRegister refuses the whole expert arena, the sliced fallback
pinned layer after layer until the driver refused one (28 GiB on a 63 GB
single-GPU PC), and WDDM then refused every later device allocation
("native head upload: out of memory"). On Windows the sliced pass now stops
at the adapter's DXGI non-local budget - the process's current usage - 4 GiB
(IDXGIAdapter3::QueryVideoMemoryInfo for the CUDA device's LUID; dxgi.dll is
loaded at run time), or RAM/2 - 8 GiB when that query fails. The arena line
names the cap and where it came from.
STRATA_ARENA_PIN_GIB: unset = the above; 0 = no cap (the old behaviour);
N > 0 = at most N GiB from the start. arena_pin_cap_gib() (pinned.hpp)
reads it, -1 when unset, for the multi-GPU cap in generate.cpp as well.
A machine whose whole-arena registration succeeds is unchanged, and a
caller's cap (multi-GPU) still takes precedence.
This PC's whole registration fails too: IQ3_XXS now pins 24 GiB (32 slices;
budget 30.8 GiB - 2.0 in use - 4) instead of 28 GiB (36).
The GGUF branch of the arena load (a native pack without experts.bin, the
normal install) read through std::ifstream, which MSVC splits into 4095-byte
freads - 0.02 GiB/s on the reporter's PC. It now uses the same fopen/fread
with a 64-bit seek as load_experts_ranges (#89), with an RAII close. Same
chunks, same bytes, same scatter.
Measured here (IQ3_XXS pack without experts.bin, RTX 5070, 64 GB): 1.49-1.60
GiB/s before, 3.17-3.25 GiB/s after (start 45-48 s -> 29-34 s); the generated
tokens are the same.
Chats (and the Chat tab's settings and API key) only in the browser's local storage (strata.* keys), the start
config strata-<model>.json and its siblings in the Strata folder, the model files in Strata-data next to it (or
--data-dir), and where that folder is remembered (%APPDATA%\Strata\settings.json, ~/.config/strata/settings.json).
A request (OpenAI or Anthropic) may set "reasoning_budget_tokens": N, and "reasoning_budget_tokens" in
strata-<model>.json sets a default (a request's own value wins; 0 means no budget). When the thinking reaches N
tokens - at a clean point, no tag held back and no character split across tokens - Service.run stops the engine,
feeds a short wrap-up line and </think> through the detokenizer and parser (the client sees it as the end of the
thinking; it counts as output), and generates again from prompt + what was generated + the wrap-up: a prefix of
what the engine's session already holds, so it continues from it. With no room left to answer it ends with
"length" as before. A bad value is a 400 before anything is sent (a bad config value refuses to start). Off
by default: without the field and the key every request runs exactly as before (one engine pass). Anthropic's
thinking.budget_tokens still only picks the level. Documented in docs/DETAILS.md.
Tests (serve/test_server.py ThinkingBudget, a fake engine that continues after </think>): off by default, the
wrap-up and the continued prompt (the first prompt + the kept thinking + the wrap-up), the Anthropic stream, a
budget the thinking stays under, the config default and a request turning it off, thinking off, no room left,
bad values.
- The ready-made engine: with the default --prebuilt, setup looks at releases/download/v<version>/ first (the
version CMakeLists.txt says, source_version()) and falls back to releases/latest with a message when that
release is not published (yet). An explicit --prebuilt / STRATA_PREBUILT_URL is used as before; an
installed engine new enough is kept without asking the network.
- Hugging Face: every model family's shards and vision encoder come from a fixed commit (HF_REVISIONS, the repos'
`sha` on 2026-09-30), tools/mtp_fetch.py reads the BF16 checkpoint at its commit (STRATA_MTP_REVISION
overrides). A revision a repository no longer has falls back to its current files, with a message. Files
already downloaded keep their names and .done marks, so nothing is downloaded again.
- Python: requirements.txt pins the packages and their dependencies (numpy per Python range, 3.10+), used by
setup's pip step and the Dockerfile. A venv set up before this (bare names in .strata-pip.json) is left as it
is; only packages it never had are installed, pinned.
Tests: tools/test_setup_pins.py (mocked network: the tag URL first, latest as the fallback, an installed engine
kept, a gone revision falling back to main in download() and mtp_fetch, the pip step for fresh, old and bumped
installs).
m16n8k16 needs sm_80, but sm_75 has m16n8k8 with the same A/B/C register mapping, so
each k=16 step becomes two k=8 steps on the fragments as they are already laid out
(a[0]/a[1] with b[0], a[2]/a[3] with b[1]). The sum now rounds twice, so the two paths
do not agree bit for bit.
cp.async does not exist before sm_80, so Turing takes the v1 kernel (launch<1>, same
accuracy, another summation order) instead of launch_i8; the capability probe now
accepts cc >= 7 and a device without cp.async sets g_turing.
Measured on a Quadro RTX 8000, CUDA 12.9, arch 75: qsa_prompt_attn_parity PASS
(FAILURES: 0; vs FP64 new 6.99e-06, old 2.37e-06, output scale 3.62); the int8 kernel
62.3 -> 19.6 ms per chunk (3.19x), fp16 65.0 -> 25.1 ms (2.59x). Engine prefill on
three Turing cards: 9.7K-token prompt 1,050 -> 1,177 tok/s, 42K 1,124 -> 1,251 tok/s;
an 80K needle test answers correctly.
iq_mmvq, native_gu_kernel and native_down_kernel called the format's dot once
per column (or group entry), re-reading the weight words and redoing the grid
lookups and sign unpacking each time. Each dot is now split into load (weight
side) and apply (activation side, same integer chain and float expression), as
native_mmvq.cu's multi-column traits are: the dense kernel takes 1, 2, 4 or 8
columns per pass (8 at a time past 8), the grouped ones 4 entries per pass
(~48 registers; the default verify window, --spec 4 with the split off, fits
in one pass).
Every column keeps the same calls, lane stride, summation order and warp_sum,
so results are bitwise equal to the old kernels, which STRATA_OLD_IQ_MMVQ=1 (or
iq_set_old_kernels) keeps for A/B timing.
Tests: iq_multi_parity (ctest, GPU, synthetic, no model) checks old vs new
bitwise for every format, ncols 1..8 and 11, groups of 0..11 entries; --bench
times both. iq_parity (real rows from tools/iq_fixture.py) now runs the dot for
1 to 8 columns and requires every column of a multi-column call to equal a
one-column call on it bitwise (exact layout, the default). native_expert_parity
runs the GPU expert with both the decode-once and the per-entry kernels on real
blobs and requires the same bits.
The grouped kernels (verify window, MTP layer) staged each entry's activations
as xs_q[k][c * 8 + j], an 8-way bank conflict on every read, and recomputed the
chunk sum hx with eight dp4a for every row. The per-hit kernels (token graph)
read each chunk with 44 byte loads.
New kernels, on by default:
- activations regrouped once per chunk so the codes expand with shift+mask
and hx is summed once per chunk;
- grouped: word-major shared layout xs_w[j][k * NC + c] plus (dx, hx) pairs,
conflict-free on the staging writes and the reads, two rows per warp;
- per hit: nine aligned word loads per activation chunk, uint2 code loads,
16-bit scale loads, one warp per (gate, up) pair or pair of down rows.
s, hx and the float expression dw * dx * (float) (s - hx) are unchanged, so
every output matches the previous kernels bit for bit. STRATA_OLD_GROUPED=1
(or moe_grouped_select_old(1)) selects the previous kernels, and so does any
activation row, scratch or blob arena the wide loads cannot take.
STRATA_GROUPED_PAIR_MIN_HITS=N keeps the previous per-hit kernels below N hits
of capacity. moe_grouped_last_path() reports which kernels the last call ran.
The funnel shift that realigns an activation chunk is written as a 64-bit
shift instead of __funnelshift_r, which nothing else in the tree uses.
tests: s2_expert_grouped_parity runs every entry point with the previous
kernels then the new ones on synthetic Q2_0 blobs and Q8_0 activations, checks
the path taken, and requires output and scratch to match byte for byte, plus a
double-precision host reference. It covers the stride and arena fallbacks and
a 2-byte aligned activation row. --bench times previous against new. It
needs a GPU and is registered in ctest.
Low-RAM mode on a multi-GPU pick dropped to one GPU with only "the low-RAM mode runs on one GPU". It now says
why: the experts plus ~10 GB for the rest (e.g. 43 + 10 = 53 GB) against this PC's RAM, or that --low-ram was
chosen; that the engine does not split layers in that mode (it refuses --resident-experts with a layer split);
and what keeps all the GPUs (more RAM, a smaller size, --low-ram off).
--low-ram off on a PC short of RAM asked "Install it anyway?" with the default n, so an unattended --yes
install always stopped although the user had turned the mode off explicitly. With --low-ram off the default is
y (--yes installs, with a warning); with auto the question and its n default are as before (#125), and the stop
message names --low-ram off --yes. The two blocks are functions now (low_ram_one_gpu_why, confirm_paging);
tests in tools/test_setup_lowram.py with mocked input.
The follow-up to #231: openai_collect kept a call the output ended inside, with its truncated arguments, in
message.tool_calls (finish_reason "stop" or "length"). It now leaves out every call whose arguments are not
JSON when the finish reason is not "tool_calls" - what anthropic_collect does with such a tool_use block. The
streamed answers are unchanged. Tests: the cut call over both whole APIs (at the end of the turn and at the
token limit) and openai_collect directly.
When the model's output ended inside a tool call whose name was already
streamed, finish() closed its JSON and emitted a tool_call event, and the
answer ended with finish_reason "tool_calls" / stop_reason "tool_use", so
clients ran the truncated arguments; an MCP call ran with empty arguments.
The call now stays unfinished: its JSON is not closed and no tool_call
follows its tool_start. Both APIs report "tool_calls" / "tool_use" only when
every streamed call completed, and the non-streamed Anthropic answer leaves
out a call whose input never closed.
Fixes#211
_bpe rescanned every adjacent pair after each merge: O(n^2) on one long pre-token piece (a CJK run, a minified
blob). With the pack's vocabulary a 2,000-character CJK run took 1.6 s and 8K characters tens of seconds;
_bpe_heap takes about 5 ms and 20 ms. It pops the lowest rank, then the lowest position - the order the scan's
strict `<` picks - and skips pairs whose symbols have merged since they were pushed. Words up to HEAP_MIN (64)
symbols run the existing scan unchanged.
Tests (serve/test_detok.py HeapBpe): the heap against the scan word by word and through encode (both
parse_special modes, heap forced on for every word and at the default threshold) over random text with CJK and
Hangul runs, conjoining jamo, combining marks, URLs, minified code and specials - with a synthetic vocabulary
(always) and the pack's (when there is one); an 8K-character CJK run round-trips in well under a second.
Service.run() recorded the finished request's history row and totals, set busy=False and dropped `tail` in a
finally that ran after `with self.fifo:` had released the lock. A request waiting for the fifo could start in
that gap: the old request then recorded the NEW request's status as its own, cleared its busy flag and popped
its tail, and the new request crashed in _note (KeyError 'tail'). The block now runs inside the fifo (an
inner try/finally, which also sees the disconnect), and _note reads the tail defensively. Test: a stream
aborted mid-answer while another request waits, with a fifo that lets the waiter run first on release.
Rows that serve 1 or 2 activations (the near-universal shape of routed
decode) read each weight block twice less often: one 256-bit shuffle
ladder covers two consecutive blocks (lanes are per-128-bit, so the
arithmetic is the SSE ladder run twice and the re-homed halves keep the
original byte order) -> the same madd lane mapping, bit-identical
results. The SSE unpack stays for 3+ tokens: at four accumulators the
extra live unpack vectors spill registers and it measurably loses.
Measured on i7-8700K AVX2, qwen3.8 125B-MoE Q2_0 decode: +10% gate/up and
+5-7% down at 1 token/expert in the standalone harness
(ML-Experiments/bench5/q2_cpu_bench.cpp, bit-exact verified), and -7
ms/verify-window end to end (CPU pool waitCPU ~19 -> ~13 ms), GPU stages
and generation unchanged.
Turing (sm_75) opt-in shared memory is 64 KiB. With the old fixed
TILE=2560 staging the down kernel could hold at most 6 tokens
(2560*6*4 = 60 KiB), so 7+ token windows split into two launches and,
more importantly, a single 40 KiB tile at T=4 filled the SM so only
one block per SM fit next to the resident expert/attention work.
Specialize gr_down_multi_kernel on the per-token tile size:
- sm_75: TILE=1280 -> up to 8 tokens staged in one 40 KiB launch
(kFusedGrMaxT=8 covered), and T-token tiles fit 1-3 blocks/SM
depending on T, so concurrent hc lanes can overlap.
- every other arch keeps TILE=2560 exactly as before.
Per-lane weight chunk order within a tile stays strictly increasing
in (tile, chunk), so the accumulation order for every (expert, row)
is identical to the 2560 path and results are bitwise unchanged.
Measured on RTX 2070 (8700K host, engine 0.1.27, Qwen3.6 125B Q2_0,
29K context, temperature 0, identical benchmark): GDN hc0 down
2.73-2.80 -> 1.88-1.98 ms/window (-30%), QSA hc0 down 0.90-0.93 ->
0.62-0.65, up/norm kernels unchanged, GPU util 91.7 -> 93.5%,
decode tok/s slightly higher on average (within per-run noise;
window is CPU expert-scheduling bound).
The i-quant fixtures were never published, so iq_parity reported ten
"missing fixture" failures on any fresh tree. tools/iq_fixture.py now
generates them deterministically (seeded rows/cols, RandomState) into
the BUILD directory: for each of the ten types it writes the raw block
bytes (sane fp16 scales at each layout's scale offsets, seeded index and
sub-scale bytes) plus the gguf-py dequantized reference - gguf-py is
dequant-only for the i-quants, so no quantizer is needed, and Q2_0 (the
repository's type 42) uses tools/gguf_writer.py's codec. The kernel
type ids in the headers match the vendored gguf-py's enum one for one.
Dependencies: python3 with numpy and the vendored gguf-py - nothing is
pip-installed, and CMake probes the configured interpreter and .venv,
registering the tests only when a usable interpreter is found and
saying so otherwise.
CTest now runs iq_parity (previously built but never registered) with
the fixtures as an explicit CTest fixture: generation completes before
the test even under parallel execution, and a generation failure fails
the run. The missing-fixture diagnostic names both candidate files and
the generating command; the nonzero exit is preserved.
`Gemm::init` collapsed five distinct failures into three bare strings ("cublasCreate
failed", "workspace", "dequant scratch of N MiB"), so an init failure on a tight machine
could not be told apart from a driver or install problem without a debugger.
Each call is now checked through two small helpers that report the operation and the real
status - `cuBLAS status N` or `cudaGetErrorString` - and optionally set
`resource_failure` when the failure means the machine is out of room
(`CUBLAS_STATUS_ALLOC_FAILED`, `cudaErrorMemoryAllocation`). A caller can then retry with
a smaller cache or prompt instead of treating a shortage as fatal.
The parameter is defaulted to nullptr, so every existing caller is source-compatible; the
upstream HIP/hipblaslt block is untouched.
Verifier::window_logprobs writes one line per row of the last verify window's
head: position, target, its log-probability, the top token and its
log-probability, a hit flag, the log-probability of STRATA_LOGPOS_EXTRA
(default 248046, <|im_end|>) and the target's log-probability renormalized
without that token. read_windows calls it before commit when
STRATA_LOGPOS=<path> is set; otherwise the only cost is one null check.
A layer split's earlier stage forwards to the stage that holds the head,
under OnDevice as commit() does.
AMD_HIP.md: the per-arch ROCm wheels, the system-ROCm rule, the table rule, the
arch list and the startup check; an RDNA4 section (ctest 42 pass / 1 skip /
2 non-GPU failures, the arch check, end-to-end 4K/16K numbers on both cards and
with setup's TheRock 7.10 wheels, the 9/9 server smoke on each card, why no
gfx1201 table is shipped, what is not validated); a Tuning table section with
the tune_hipblaslt command at the engine's shapes. setup's no-table message
points there.
On gfx1201 a plain store to mapped pinned memory stays in the GPU's L2 until the
stream is synchronized, so hip_handoff failed ("separate copy/ring timeout") on
both RX 9070 XT and R9700. A probe of the ring forms on both cards, graph and
stream launches, 100 rounds each: plain store 0/100 seen without a sync;
volatile store, plain store + trailing system fence, and a system-scope atomic
release store 100/100 (41-157 us), no stale payloads. The ring now uses the
volatile form doorbell_publish_kernel already uses. The ring kernel is the
--no-publish-kernel fallback; the default fused publish was already volatile.
qsa_prompt_attn_parity linked CUDA::cudart unconditionally, so
-DSTRATA_ENABLE_HIP=ON -DSTRATA_BUILD_TESTS=ON stopped at CMake's generate step
("CUDA::cudart ... target was not found"). The tool compares the tensor-core
kernel that HIP compiles out, so it is built with CUDA only.
- CMake: CMAKE_HIP_ARCHITECTURES may list gfx1100 and gfx1201 (validated) and
gfx1101 / gfx1102 / gfx1200 (build with a warning: same wave32, 64 KiB LDS and
dot4, not run by us; #192, #176). The list reaches the code as STRATA_HIP_ARCHS.
- Runtime: the engine (every GPU a run uses) and strata-device compare the card's
gcnArchName (up to the ':' feature suffix) with the compiled archs and require
wave32, with a message naming the card, its arch and the build's archs, instead
of a later "invalid device function".
- __dp4a uses __builtin_amdgcn_sudot4 (v_dot4_i32_iu8) on all five archs; the
portable fallback stays for other compilers/targets.
- setup.py: AMD_ARCHS gfx1100 + gfx1201; TheRock's wheel index per GPU family
(gfx110X-dgpu, gfx120X-all; the pinned 7.10.0a20251120 is on both), only that
family's libraries on the path, reinstall when the family changes; a system
ROCm older than 7.0 is passed over for the wheels; the engine is rebuilt when
the chosen card's arch is not in its BUILD.json. The hipBLASLt table is
tools/hip/<arch>-hipblaslt-<version>.txt for the card's arch AND the installed
hipBLASLt (read from hipblaslt-version.h, 1.4.1 -> 100401), else none with a
message; before, the highest-named table of the arch was taken whatever the
installed version, which the engine then refused.
conversation_session_sizes() gives one session's running-state sizes: the GDN bytes of its gdn_alloc
rows, the indexer tail/dead/block_pos per owned QSA state; an inconsistent carve is rejected.
Checkpoint save/validate/restore, the snapshot size estimate, capture, validation and restore iterate
the owned QSA ordinals [qsa_ord0, qsa_ord0 + qsa_alloc) instead of every model layer (the spare
pooled row is written back per owned layer only), and a parked snapshot records the carve it was
taken from (layer_lo, layer_hi): a restore into another range is refused before any write.
One GPU (the default full range: gdn_alloc = n_gdn_layers, qsa_ord0 = 0, qsa_alloc = n_qsa_layers)
saves and restores exactly the bytes it did before, in the same order. generate.cpp's
conv_state_sizes uses the same function. The tests' hand-built sessions carry the full-range carve,
and the host-only validation test checks the carved sizes and the rejections.
Conflicts in generate.cpp:
- against #216's session carve: conv_state_sizes takes the session (the carve's signature) and scales
the PR's whole-model GDN size to ss.gdn_alloc rows; checkpoint_save/checkpoint_restore are the PR's
(conversation_state.cpp), which the next commit makes carve-aware.
- STATE_HASH: the per-GDN-layer line over ss.gdn_alloc (guarded for zero rows), the QSA loop over the
owned ordinals [qsa_ord0, qsa_ord0 + qsa_alloc) with the PR's dead= and pooled_full= in it, and
end_cell from ss.max_cells (the primary state's max_cells, and valid with no QSA layer).
- INFO: f-resram's arena_mib (resident bytes under --mmap-experts) and the PR's three
conversation_cache_* fields.
The --conversation-cache-* options sit in the option parser's second chain (6fb466b); their value is
`number`, not `parsed`, which is that chain's flag.
doplxyz's minimal patch from #178 (their commit 0c59582): CMake, the
runtime device check, the signed dot4 intrinsic and setup's AMD_ARCHS
accept gfx1201 beside gfx1100.
On a PC whose VRAM cache plus RAM can hold all experts but whose RAM alone cannot (e.g. 32 GB + a 24 GB GPU with
Q2_0 / IQ2_XS / the Coder), the plain low-RAM mode (--mmap-experts) reads the non-VRAM experts through the OS file
cache, which such a PC keeps giving back to the SSD. The resident variant copies exactly those experts from
experts.bin into RAM once at start:
- engine: --resident-experts = --mmap-experts --resident-cpu-experts, the copy page-locked when the driver allows
(else locked in the working set), 4 GiB headroom, plain mmap with a warning when it does not fit. Built after the
prompt path's lend plan: the lendable slots' experts are kept in RAM too, from the last slot down, as far as RAM
allows. Adaptive swaps stay on: an evicted expert is copied back from its VRAM slot into the RAM place of the one
that replaces it (exchange buffers, committed when the swap lands), so no swap reads the file.
- --resident-cpu-experts (the ROCm arm) no longer requires --adapt-every 0 and gets the lend region too; its copy
stays pageable with 8 GiB headroom as before.
- the PCIe share is gated per source (pcie_layer): the arena answers as before; a page-locked resident copy answers
for every layer. Same bits as --mmap-experts for the same placement with --pcie-frac 0 or STRATA_RESIDENT_PIN=0.
- INFO arena_mib reports the RAM the experts take in the mmap modes (the resident copy, or 0); the serve log and the
CLI report the resident GiB, exchanges and blob reads from the file.
- setup: in the low-RAM mode, resident when RAM >= (experts minus the GPU's estimated share at this context, less the
image encoder's room) + 10 GB; --low-ram resident|mmap forces a variant; needs engine 0.1.30 (else mmap).
- tests: lend-region choice and exchange bookkeeping (CPU only). Default (arena) path unchanged.
Under sampling the draft layer proposed its argmax. With STRATA_SPEC_COUPLED=1 a sampled request's drafts are
sampled with the target's own chain (top_k / top_p / min_p / temperature, penalties over the same history) and the
same Philox uniform the verify window draws for the row that checks the draft: counter = MTP cell + 1 (a draft at
cell p + a + j is checked by row j of the next window, counter p' + j). Verification is still exact-match against
the target's sample, so the output does not depend on the drafts; acceptance should rise.
- sampler.cu: coupled_stage / coupled_penalize / coupled_merge kernels (params, seed, counter and penalty ring from
device memory, so they live in the drafter's CUDA graphs); the split part kernel reused unchanged; the tail
(sampled_tail_warp) templated to also return the pick's probability. The five existing sampler kernels compile to
identical SASS (sm_120).
- mtp.cpp: separate coupled round/step graphs (the argmax graphs are untouched and still used for greedy requests);
subset-aware (sample over the draft head's subset, id <-> subset maps for the penalties); --spec-min-p gates on the
draft's probability of its sampled token under the chain.
- generate.cpp: set_draft_sampling per request (serve and CLI), the penalty history base before each draft (serve).
- coupled_draft.hpp + coupled_draft_test.cpp: the counter / ring arithmetic and a host-only test (907k checks).
Switch off: no allocations, no new launches, the same graphs.
DETAILS.md: what it does, that no flag means no change, the contributors' paired per-position NLL (yarn 2 -0.18 nats vs none and linear 2 at 293K, no separation at 2.7K/32K, needles not discriminating), the explicit-factor pass-through, fixed per run. The stale 'engine 0.1.18' references go; the setup's 384K/512K menu entries and the README say experimental.
rope_scaling_invalid() checks every value with std::isfinite first (a NaN passes every range comparison), factor >= 1, original context > 0, a finite positive freq_scale and mscale; the engine applies it to the resolved config (CLI + model file) and fails at startup with the reason, and native_rope, both native indexer appends and the prefill rope use the same rule instead of three hand-copied lists. RopeScaling::kernel_args() gives the analytic kernels their float constants and returns the identity (1, 0, 0, 0, 1) for none whatever the other knobs hold, as the table builder already ignored them: an unscaled run cannot pick up a stray --yarn-attn-factor on the native path only. none resets the knobs (with a note if some were given). The magnitude-correction note prints whenever mscale != 1, and only then (not for plain linear).
indexer_key_append_kernel skipped the rotation at pos 0, which is only an identity while row 0 of the table is (1, 0). Under YaRN row 0 is (mscale, 0), so dead and pooled[0] stayed the only unscaled keys of a scaled cache and disagreed with the native indexer's spare. The rotation at angle 0 reduces to v * cos_tab[pair] on the rotated dims (the sine is 0 in every scaling), so no partner read and no barrier; unscaled it is v * 1.0f, bit for bit the old value. qsa_parity checks the table spare against the unscaled spare times row 0 (bit-exact), pooled[0] == dead, the native spare's factor, and that YaRN's mscale is observable there.
pooled= hashes rows [0, L/4) of every QSA layer's idx_pooled again, as in 0.1.29 and earlier, so a
state hash from this engine compares with an older one field by field. The PR's extent (those rows plus
the spare row at L/4, which a conversation restore writes back) is pooled_full=; dead= stays. The
cache gates compare pooled and pooled_full; the cache-off gate against an older engine compares pooled
directly and needs pooled_full only as proof that the candidate is this build.
A failed checkpoint save/restore prints its reason to stderr (the ERR line has none). The cache
harness reads its files as UTF-8 (Windows' default code page stopped it at the tokenizer), and the
isolation gate passes absolute fixture paths (the engine runs in the config's cwd).
Two things a layer split was paying for and not using.
* THE SESSION CARVE. A stage carved state for ALL 48 layers whatever slice it ran - every QSA KV pool at
max_cells, every GDN recurrence - the whole-model-sized-buffer disease the chunked-QSA-prefill PR cured
upstream. `session_bytes`/`session_init` take [layer_lo, layer_hi), and a session now owns only that range:
`qsa_states`/`gdn_state` keep GLOBAL ordinal indexing and simply hold fewer rows (`qsa_ord0`/`qsa_alloc`,
`gdn_ord0`/`gdn_alloc`), per-layer consumers subtract `gdn_ord0`, and the model-level uses - the shared RoPE
table, the prefill staging identity, the MTP rope borrow - read `qsa_states[qsa_primary()]`. A range with no
QSA layer still carves ONE primary state so those uses stay valid, and the default full range is
byte-identical to the old carve. On the 2x3060 + 2x5060 rig, CUDA1's [30, 48) session costs 0.93 GiB:
7.83 GiB free before the carve, 6.90 GiB after (`session [30, 48) and the head`).
* PER-STAGE PROMPT LOANS. `no_prefill_borrow = true` for split stages said "each stage's prompt path has its
own buffers" - true, but that is a reason to give each stage its own LOAN, not to make every stage withhold a
chunk-sized reserve from its cache for the whole session, and forced on it also collapsed `--prefill auto` to
2048. Each prompt path now borrows the tail of ITS OWN stage's cache (`PfPart`), so outside the prompt the
whole cache is expert cache: `--prefill auto` is the largest chunk every participant can lend (8192 tokens
here), a participant refills its loan into its own slots and marks only its own layers (a slot refilled into
another cache, or another stage's rows marked resident, is silent and produces plausible tokens), and the
chunk is the largest one EVERY participant can lend rather than one card's answer imposed on the others.
A participant that cannot lend the buffers falls back to the prompt path's own, as a lone one always could;
with a single participant the plan reduces to `plan_lend` exactly, so the single-GPU loan is unchanged from
main - the percentage cap stays an AUTO-chunk rule (an explicit `--prefill` is the operator's number and a
loan of it only has to leave the 128-slot floor). The flat 1 GiB withheld from every stage after the first
becomes what is actually held back after the weights, session and drafter exist: 96 MiB of verify windows,
plus 1000 MiB of drafter and head on the ONE stage that carries them. CUDA0 lends 2162 slots (3.98 GiB),
CUDA1 1993 of its 2653 (3.97 GiB).
* REPORTING. `expert_slots`/`expert_cache_mib` sum every tier: a 4-GPU 256K run reported CUDA0's cache alone
(3327 experts) while the four cards held 13310, so the Monitor tab was wrong by 4x for every multi-GPU
config. `expert_slots_primary`/`expert_cache_primary_mib` keep the old figure. The helper ranks'
ms_begin/ms_wait are cumulative since boot and were printed next to per-request deltas; they take deltas now.
KV-streaming stats iterate the session's OWNED ordinals.
* REFUSALS. `--gpu-only-full` and `--gpu-stages` replay every layer through CUDA0's session, which a split
carves to CUDA0's range, so they are refused with a layer split rather than answering about another stage's
state; the token-graph hit path is disabled under a split (such a graph cannot span stages); and the sessions
are allocated AFTER the split search, because the search prices each placement by what that range's session
will cost on that device (`session_bytes` is pure arithmetic).
Two gates stand between this engine and a 6.0 card, plus one attribute the card does
not honour. None of them needs the kernels rewritten:
* `CMakeLists.txt` refuses to configure below sm_75;
* `__dp4a` (6.1+) appears 29 times in the i-quant kernels (`native_mmvq.cu`,
`s2_expert_grouped.cu`, `iq_kernels.cu`) and `__nanosleep` (7.0+) in the three
doorbell waits (`elementwise.cu`, `verify_kernels.cu`);
* the Turing port sizes `fused_gr_read_multi`'s slice from
`cudaDevAttrMaxSharedMemoryPerBlockOptin`, which on an sm_60 answers 65536 - a card
whose launch validator enforces the 49152 B per-block limit instead. The capacity
comes out 6 tokens and every 6- or 8-token window fails the launch with
"invalid argument".
## The changes
* **`include/strata/kernels/dp4a.hpp` (new)** - `STRATA_DP4A` and `strata_spin_pause()`.
The dp4a fallback is llama.cpp's own (`ggml/src/ggml-cuda/common.cuh`): both operands
read as SIGNED bytes, accumulated into int32 - which is what `__dp4a` does for exactly
these call sites, the kernels here being transcribed from `vecdotq.cuh`/`mmvq.cu`. The
fallback is bit-exact, not approximate; the nanosleep sites only pace single-thread
doorbell waits, so on sm_6x the loop spins. Both are `__CUDA_ARCH__` selections, so a
HIP build takes neither: its `__dp4a` and `__nanosleep` are `hip_compat`'s own.
* **`CMakeLists.txt`** - `-DSTRATA_EXPERIMENTAL_SM60=ON` relaxes the architecture gate,
and the refusal message names it. The UTF-8 BOM stays.
* **`fused_gr.cu`** - below sm_70 the slice capacity comes from
`cudaDevAttrMaxSharedMemoryPerBlock` (49152 B -> 4 tokens) instead of the opt-in. The
Turing port's slicing, its no-opt-in fallback and the HIP expression are unchanged.
On sm_70 and newer this is a no-op: the opt-in is honoured there, so the capacity is the
value it already was, and `STRATA_DP4A` and `strata_spin_pause()` compile to `__dp4a` and
`__nanosleep` exactly as before. Building for a 6.0 card needs a CUDA release that still
targets Pascal (CUDA 13 removed it; 12.x has compute_60), and `CMAKE_CUDA_STANDARD` has
to stay 17 for it: nvcc 12.9 refuses `-std=c++20` when the host compiler is VS2019's `cl`
and silently falls back to C++14, which rejects the nested
`namespace strata::kernels {` declarations.
Rebased onto 0.1.27 (0.1.21 -> 0.1.24 -> 0.1.27). The 0.1.21 form of this patch clamped
the MTP and verify windows instead, because the fused GR read could not launch a 6-token
window at all. Now that `fused_gr_read_multi` slices the batch itself, the clamp, its
`fused_gr_max_window` helper and the two `generate.cpp` call sites are gone, and the full
window runs:
strata verify: window up to 6 tokens, 69.1 MiB of device buffers
strata verify: captured the 6-token window (upload no error, sync no error)
Measured on a 6.0 card (GP100-class, four dies through the layer split, the model's
IQ3_XXS quant, an 883-token prompt, 256 generated, the shipped `--spec 4`): every one of
the 24576 experts resident, 100% decode cache hit rate, 110 tok/s prefill and 21.3 tok/s
decode (warm median). This port on the 0.1.21 base measured 110.6 and 17.9 in the same
session with the same prompt, so the full window is worth about 19% of decode here. One
RTX 3090 Ti with the release engine measures 305 and 79 tok/s for the same model. The
port is about fitting, not speed: decode is bounded by Pascal's per-layer arithmetic, not
by capacity or memory. Only the 6.0 path was exercised on hardware.
Merged by hand onto 0.1.29: SIGTERM takes #153's Ctrl+C path (the timed sleep loop and the second-Ctrl+C kill stay);
SIGINT keeps Python's handler; the image builds RTX 20 (75) too.
Reserve accounted space for later appends without moving prior payloads. Prefer full capture when optional capacity would evict useful conversations, and report the actual additional RAM required. Exercise 4096 small appends and non-mutating admission checks under sanitizers.
Grow snapshot buffers in segments, track the first rewritten token, and charge retained active storage to the existing RAM budget. Refresh running state, partial pages and draft boundary state. Add full-capture versus incremental growth/rewind checks and report snapshot sizes with usable pressure-test budget ranges.
Replay identical requests and require token, known-answer, reuse, resolved-setting and comparable main-state parity. Retain the complete pooled-state diagnostic and add a completed-row fingerprint matching upstream. Add negative controls that remain active under optimized Python.
Move core diagnostics from the dependent NVMe work (4d73bb3) into the shared-core review branch. Read back authoritative bytes and resident ring pages without rewriting state; remove a redundant GPU fixture refill that could mask restore failures. Bound hash workspace and check synchronization/copy errors. Opt-in diagnostics record batched versus token draft prefill.
Validation: current core translation units pass C++ syntax checks; 1188 host transfer checks pass with ASan/UBSan. Real GPU and full-model validation remain pending the benchmark window.
The note fired for every scaled run inside the trained 262144 and its text
named YaRN's magnitude correction even for linear, whose resolved mscale
is exactly 1. Split in two: the in-window warning now only says that no
position needs the extension while the resolved scaling still applies to
every angle, and the magnitude explanation became its own note, printed
only when YaRN is resolved and its magnitude multiplier differs from 1
(yarn factor 1 inside the window stays quiet; linear never sees it). The
main rope scaling line is unchanged, the numbers it prints already carry
the resolved mscale.
Diagnostics only. Checked on the rebased tree: none prints nothing,
linear 2 prints the in-window note without the magnitude line, yarn 2
prints both (mscale 1.069315), yarn factor 1 prints the in-window note
only, and yarn 2 past the trained window prints the magnitude note
without the in-window one.
v0.1.29's new draft_kv site (the 0.1.26 batched draft prompt pass) kept
upstream's float freq_base form through the rebase, which both misses the
scaling on the draft K/V and does not compile against the RopeScaling
signature the other four sites carry. It passes rope_scaling() now, like
the prompt K, Q and indexer-query sites.
Item 8 verification. The parity section now runs all three scaling
configs - none, linear 2 (angle shrink only), yarn 2 (ramp + mscale) -
and "bit-identical" is asserted on the BIT representations (memcmp of
the state words), not float !=, which would pass a NaN-payload or -0.0
flip. Each scaled variant must also differ from the none baseline in
pooled-key bits, so the parity cannot pass because both sides ignore
the scaling. Green on this tree: 0 of 33409 state words differ in
every config; 128 (linear) and 192 (yarn) pooled words move.
The implementation landed with the derived-factor change; per the item's
own scope note this is verification and tests, not a second one. The
table from the item is now pinned case by case - 131072 and 262144 with
yarn and no factor resolve to 1, an explicit --rope-scale 2 stays 2 inside
the window, 393216/524288 keep 1.5/2 - and linear is checked under the
same rule the old fallback applied to it too. On the real engine the
resolved combo runs as "rope scaling yarn, factor 1 (freq_scale 1, mscale
1.000000)" and a greedy 12-token run is token-identical to the stock
baseline - evidence on one path, not a bit-identity claim.
The factor ladder (smallest of 1.5/2/3/4 covering the chosen context) is
gone. The automatic factor is now the final context over the trained
262144, clamped at 1 - max(1.0, final / trained) - computed AFTER the
RAM reductions, so it always matches the context the engine will serve:
512K + IQ3_S under 90 GB of RAM + yarn, no factor: final 128K, factor 1
no reduction: 384K -> 1.5, 512K -> 2
a chosen method inside the trained range: factor 1, not 2
Factor 1 removes the automatic expansion - the trained angles stand - but
it is not a switch for rope as a whole: an explicitly chosen method's
settings keep their defined behavior. A user-supplied --rope-scale is
still kept verbatim when a reduction changes the context; the ok() line
now says where the factor came from (final context / trained, or as
requested), and help, docstring and docs/DETAILS.md drop the
"smallest covering factor" wording.
The triple-check pass found what the menu could not: --context is not
limited to CONTEXTS, and the factor ladder (1.5, 2.0) ended at 512K, so
`--context 600000` died with a bare traceback instead of a setup message.
The ladder extends (3, 4) and a context past its top refuses cleanly,
naming --rope-scale as the way out. The interactive explanation block
also stays quiet under --yes, whose ok() line announces the choice
already.
The comment said the setup adds the flag when the chosen context needs it;
the code called fail(). `START-HERE.bat --setup --context 393216` - the
command the docs give - stopped instead of configuring. Now an omitted
--rope-scaling with a past-trained context resolves the config itself:
yarn (llama.cpp's extension method, recall-tested here at 512K; one
question when interactive, the silent default with --yes) and the smallest
covering factor (1.5 at 384K, 2 at 512K). An explicit none is still
refused for such a context, with the reason and the way out spelled out
instead of a bare stop; explicit linear/yarn/scale pass verbatim; inside
the trained 262144 nothing turns on by itself.
The resolution moved AFTER the RAM reduction (the §4 ordering): the factor
must cover the context the engine will serve, so a 384K choice reduced to
128K by the RAM limit gets no scaling at all, and an explicit selection
that survives the reduction says so in a warning.
The rope config is now two pure functions (covering_factor, resolve_rope)
with its own unit tests (tools/test_setup_rope.py); the engine args take
the resolved values; help, module docstring, the context menu note, README
and docs/DETAILS.md describe the implemented behavior.
The rebase onto 0.1.24 had brought the QSA indexer's C-2 chunk append,
whose kernels rotated the pooled keys inline - cosf/sinf of the unscaled
angle, no mscale - so a scaled prefill would have written indexer keys
under a different rotation than the per-token append the decode path uses
(rope_scaling.hpp's one-rotation rule, broken silently). The batch path
now resolves freq_scale/corr_dims/mscale from the same RopeScaling the
single append reads and rotates through the shared rope_scaled_angle; the
raw freq_base parameter is gone from its signature.
qsa_parity pins the contract the C-2 rework claimed: the batch append's
whole end state (pooled, spare, tail ring, block_pos) is BIT-IDENTICAL to
the sequential single appends', under none and yarn 2 alike, and the
scaling is observable in the batched keys.
The first end-to-end check that a scaled run recalls past the trained end:
8 of 8 probes found the needle at mid-depth, temperature 0 - 1k/32k/128k on
the stock model (arm A), 262k and 512k prompts on linear factor 2 (arm B)
and yarn factor 2 (arm C), 512k being 2.01x the trained 262,144. The yarn
262k probe was repeated green after the rebase onto 0.1.20, next to the
per-arm engine and server logs and the launch recipes.
What it does not establish is written down in the README: one depth, one
needle per length, and no long-document perplexity with and without
scaling at 262K yet - the one open measurement.
The defaults collided with the opt-outs: "none" was both the untouched
--rope-scaling default and the explicit switch-off, and 1.0 the same for
--rope-scale, so qwen4exp.rope.scaling.type/factor from the model file won
even against an explicit flag - the opposite of the stated precedence (and
of docs/DETAILS.md). Now the empty string and 0 mean the flag is absent
(the same sentinel idiom as --rope-freq-base 0 and --yarn-orig-ctx 0), the
model file fills only what no explicit flag named, and `none` / `1` are
the opt-outs they read as.
The resolution moves above the weight pipeline: the config needs only the
CLI and the model file's metadata, its one hard constraint (set before
session_init builds the table and the graphs capture the constants) is
unchanged, and a bad rope key now dies before the shards are scanned.
Checked on the bench machine against a synthetic GGUF carrying
rope.scaling.type yarn / factor 4: the flags absent -> yarn 4; --rope-scaling
none -> no scaling; --rope-scale 1 -> yarn, factor 1, mscale 1; explicit
linear 2 -> linear 2; a GGUF type other than none/linear/yarn dies before
the weight scan. The real artifact (no rope keys) starts and generates
unchanged, and rope_parity stays green.
DETAILS.md gains the 'Context extension' subsection - what linear and yarn do,
the setup path (383K/512K in the context menu pick the factor themselves), the
engine flags with llama.cpp's names and their defaults, why the scaling is fixed
for the whole run (K sits in the cache post-RoPE), and the one honest caveat:
inside the trained range a scaled run is a slightly different model, and the
recall numbers are text-only at one depth. Troubleshooting grows the row for
the refusal a context past 262,144 now raises without the flag; README points
the context question at it.
CONTEXTS grows 383K/512K, labelled in the menu, and the setup refuses a context
past the trained 262,144 without --rope-scaling (the engine would refuse it
too). The factor defaults to the smallest that covers the chosen context
(1.5 at 393K, 2 at 524K) and lands in the engine args like every other flag.
Nine flags, llama.cpp's names: --rope-scaling none|linear|yarn, --rope-scale,
--rope-freq-base, --rope-freq-scale, --yarn-orig-ctx, --yarn-ext-factor,
--yarn-attn-factor, --yarn-beta-fast, --yarn-beta-slow. Ranges die in the
validation block at second zero; the config itself resolves after the model file
has had its say and before session_init builds the table or captures a graph
(rope_scaling.hpp's one-writer rule). Precedence: CLI over the model file's
qwen4exp.rope.* keys over the struct defaults - today's artifact ships no rope
keys, so that channel is a no-op defaults path for future fine-tunes.
A scaled run prints its resolved summary (type, factor, freq_scale, base,
mscale, context vs trained) to stderr, which is what the bench logs quote, and
warns when scaling is on with the context still inside the trained range -
YaRN's magnitude correction applies everywhere, so that run is not the stock
model.
The table path got scaling for free (it lives in the table contents); the three
ANALYTIC paths computed the angle in-kernel and so need the config explicitly.
Each wrapper now takes the RopeScaling and passes the resolved constants -
freq_scale, the corr_dims pair, ext_factor, attn_factor - as kernel arguments.
They are process constants set before session_init, so baking them at CUDA-graph
capture is correct by the same argument as every other kernel argument: only
per-token values must ride device memory, and positions still do (st.pos_dev).
The shared float32 arithmetic is rope_scaled_angle (rope_scaling.hpp), a
line-for-line rope_yarn: one definition for native_rope, the native indexer's
pooled-key rotation and prefill's rope, instead of three transcriptions that
could drift. The spare indexer key keeps its zero angle; under YaRN the mscale
rides in through cos(0) on both pooled keys and indexer queries, so top-k is
unmoved. rope_parity adds the check this whole wiring answers to: native vs
table on the same input, yarn and none alike - which caught the first draft
passing scaling.mscale() into the helper that applies the log term itself, a
double magnitude correction visible as ~7% on every rotated dim.
The artifact pins theta 1e7 with no rope.scaling keys, so the engine had exactly
one rope: inv[i] = theta**(-2i/n_rot), no freq_scale, no correction. Contexts
past the trained 262,144 had no coherent path.
rope_scaling.hpp adds the one process config - none | linear | YaRN, llama.cpp's
types - and build_rope_table grows an overload that folds the whole scaled angle
(ggml's rope_yarn: interpolation, the corr_dims ramp along the pairs, and the
mscale magnitude correction) into the float64 host table. The rotation kernels
cannot tell a scaled table from an unscaled one, so nothing downstream changes:
for the default decode path and the indexer's pooled key, scaling IS the table.
The math is transcribed from the vendored ggml (ggml.c corr_dims, ggml-cuda/
rope.cu rope_yarn/ramp, MIT headers kept), not re-derived; the one deliberate
deviation is bookkeeping - the ramp takes the PAIR index ggml's i0/2 would
produce, because every Strata rope kernel speaks in pairs.
For type None the new builder calls the original loop verbatim, and the parity
test holds it bit-identical; linear(4) is checked bit-exact against a float64
reference and equal row-for-row to none at p/4; YaRN against a float64
transcription of the ggml spec, plus the structural checks a tolerance cannot
fake: cos_tab[0] == mscale (position 0 is angle 0 for every pair), first pair
extrapolates / last interpolates at a far position (cos for the first, sin for
the last - near zero sin IS the angle, which is where interpolation happens),
and an observability assertion that the fixtures see scaling at all.
The rows of a 2,184-token prompt are 27k random 4 KB O_DIRECT reads (114 MB for ~7 MB of rows), all before the
first kernel: 466 ms with the reader's 16 blocking threads. 64 threads (the port's default now, a copy of
src/platform/direct_file.cpp; STRATA_IO_THREADS still overrides) give 330 ms and 128 / 256 the same - the drive
tops out near 85k IOPS. A 256-token first chunk (STRATA_PREFILL_FIRST, 0: off) lets the GPU start while the
rest of the rows are read behind it: 2,184 tokens 3,075 -> 2,860 ms, time to first token ~150 ms less; 8,000
tokens 986 tok/s. The chunk boundary moves rounding in the expert GEMMs, so a long greedy continuation can
diverge late (token 45 of 64 on the test prompt); short prompts are bit-identical.
Measured on the B70, outputs identical before and after each change (docs/INTEL.md "Speed work"):
- every window graph and the drafter's graphs captured at load (STRATA_WARM_GRAPHS=0 restores first-use capture)
- the commit graph is left running while the drafter's round runs on its own queue: decode 38.3 -> 43.1 tok/s at
the 2,184-token context, 46.5 -> 49.7 on a short prompt
- the draft layer's batched prompt pass takes prompts under 64 rows (106 -> 16 ms on 19 tokens)
- one device module per kernel: the first launch in the prompt path 245 -> 1 ms
- the expert dequant writes each thread's run as one vector store: 0.085 -> 0.030 ms per expert; the 2,184-token
prompt 566 -> 682 tok/s, 8,000 tokens 720 -> 841
- SWAR sign compare/subtract in the expert dots, a local-memory resident-plan kernel, a split-K fused down kernel
(bit-identical to the single-token kernel, gr_parity): parity-clean, no measurable decode change
XMX: oneMKL's FP16 GEMMs already run on the XMX units (30-60 TFLOP/s). Two joint_matrix kernels are in the tree,
correct and opt-in because both lose to the paths they would replace on this card: xmx_gemm_iq (fused dequant +
GEMM from the quantized rows; xmx_gemm_bench) and qsa_prompt_attn_xmx (the mma.sync prompt attention's port;
qsa_prompt_attn_parity passes at 1e-6 of scale; STRATA_PROMPT_ATTN_XMX=1).
Profiling: unitrace -q needs Intel's metrics libraries (Dockerfile.metrics) and dev.xe.observation_paranoid=0;
the per-kernel table is in INTEL.md. A trap recorded there: --prefill-until with a native pack drops the prompt
tail (the token loop is skipped), so a "windows for short prompts" routing was invalid and is not in this commit.
The canonical path zeroes session state from put_input at position 0, but a
native pack breaks out of that loop before the first put_input and runs verify
windows instead, leaving cudaMalloc'd state uninitialized. On a reused
allocation that produced NaN/overflow and degenerate identical-token output.
Zero the state explicitly when native_pack is set.
Also removes an unused argmax() helper that emitted a compiler warning, and
ignores the local .pi/ harness directory.
IQ3_S and IQ3_XXS asked for 262K were cut to 128K on any machine under
90 GB of RAM. The limit is now counted: the experts' arena, plus the
context's 8-bit KV (in RAM with KV streaming, ~13.7 KB a token), plus
24 GB of room.
- A 64 GB PC still gets 128K for both models, as before.
- A box with ~84 GB keeps the 262K it asked for. A Colab A100-40G
(83.5 GiB) runs IQ3_S at 262K, with images, in 56 GiB.
Display both rates in t/s with independent graph scales and a darker
prefill curve. Expose prefill_tok_s_mean using the engine's existing
rate, excluding cached tokens.
Co-authored-by: Codex (gpt-6-astra-medium) <[email protected]>
Under WSL2 the NVIDIA driver can pin/map only about 1 GiB of host memory in
total (dmesg: "dxgk: create_existing_sysmem: establish_gpadl failed: -122").
With --expert-cache-device1/2 the startup ran out of it twice:
- CUDA2's context is created after CUDA0's weights and MTP, whose mapped host
buffers have spent the budget: "CUDA2 experts: cudaSetDevice(2) failed: out
of memory". STRATA_EARLY_REMOTE_CONTEXTS=1 now creates every secondary
context up front, like CUDA1's (opt-in: the default order is unchanged).
- With three contexts up, the 260 MiB mapped native embedding no longer fits:
"native embedding: cannot pin 260 MiB". It now falls back to a device copy
on the current GPU instead of failing the start.
Measured in Docker Desktop (WSL2), RTX 5070 Ti + 2x RTX 5060 Ti, Swift 1.5
IQ3_XXS: starts and serves with both helper caches (5000 experts each).
Closing the console window (or a crash, or Task Manager) ends the server without its cleanup, and strata --serve,
strata-vision and the MCP servers kept running with their VRAM and RAM. Each child now goes into a job object that
kills everything in it when its only handle - the server's - closes; what the children start joins it too. The
server itself stays out, so a browser --open starts is not tied to it. A no-op elsewhere.
serve/test_winjob.py: a contained child and its grandchild end when the process that contained them is killed.
The sampled path (penalties, top_k, top_p, min_p, temperature, draw) kept one
row on one SM: sampler_kernel excluded the logits already kept by sweeping the
kept list for every logit of every round, O(k^2 x n_vocab) shared-memory
compares per row, and ran the double-precision tail on all 1,024 threads.
Three implementations now pick the same token:
- Split (new default): stage 1 cuts each row into 4,096-logit blocks (61 per
row at 248,320 logits), each warp keeps the top_k of its 1,024 penalised
logits in registers, and the block merges its four warp lists. Stage 2, one
warp per row, merges the block lists and runs the tail on one warp.
- sampler_one_block_kernel: one block per row, top_k rounds that each take the
logits strictly after the previous pick in the selection order (value
descending, id ascending). It is the fallback when the split cannot run
(stream under graph capture, vocabulary above 262,144, more than 64 rows,
no scratch) and is selectable with STRATA_SAMPLER_ONE_BLOCK=1.
- sampler_kernel, the previous kernel, kept unchanged as the reference and
selectable with STRATA_OLD_SAMPLER=1.
The list, its order and the pick are unchanged: the first k of a union lie
within the first k of each part, and a merge of ordered lists is ordered. The
tail keeps the same arithmetic, with the exps and quotients shared across one
warp and the ordered sums and scans run by lane 0.
The block lists live in a per-stream scratch buffer grown on demand. A
grown-out buffer is retired rather than freed, and a failed grow is
remembered so later calls of that size do not retry cudaMalloc.
sampler_parity is the parity test. Fixtures 16 to 18 pin the whole top_k list
position by position under ties (+inf, -inf, NaN, +0/-0, sentinel id 0,
penalised tokens landing on other tokens' values), tied draws through the whole
chain, and the automatic fallbacks (graph capture, row cap). Each pick is an
integer token id compared for equality with a host reference. ctest runs the
binary once per path: sampler_parity (split), sampler_parity_one_block and
sampler_parity_old, so all three paths are held to the same expected ids.
hip_compat/cuda_runtime.h: map cudaStreamIsCapturing, cudaStreamCaptureStatus
and cudaStreamCaptureStatusNone to their hip equivalents, which the new
capture check needs on HIP builds.
24d4013 put the software prefetch in iq_avx512.cpp, so it only reaches CPUs with AVX-512; the AVX-2 kernels, which is what a Zen 3 / Haswell / Skylake-client box actually runs the i-quant rows on, had none. Same env, same default, same hint: STRATA_IQ_PREFETCH is the distance in bytes, 0 = off, default 2048.
Measured on a Ryzen 7 5700X3D (AVX-2 only, 64 GB DDR4-2666, 4 KB pages - large pages refused) with IQ3_S decode, 4 rounds interleaved, outputs identical:
gate/up phase 13.571 -> 13.024 ms/round (-4.0%)
rows 25.1 -> 26.1 GB/s (+1.0)
whole round 45.54 -> 44.94 ms/round (-1.3%)
and with the pool cold (first round after the arena is built) the round is -2.4%. The same 4 rounds with STRATA_IQ_PREFETCH=0 land within 0.1% of the unpatched binary, so the patch itself costs nothing. native_expert_parity: 0 failures with the prefetch on and off.
The non-temporal hint was measured worse than T0 at the same distance (-0.7% vs -1.3%, +0.5 vs +1.0 GB/s), so this keeps T0. 4096 is a little worse than 1024/2048, which is why the default stays 2048 as in the AVX-512 file.
When a request is cancelled during prompt reading (e.g. client disconnect
or timeout), err is set to 'cancelled'. Because err is declared in main()
and was never cleared between requests, any subsequent request in v0.1.26+
evaluated 'if (!e.empty())' in sp.on_chunk on its first chunk, immediately
failing with 'strata serve: cancelled' and terminating the engine with code 1.
Clearing err at the start of each request, before prompt segment runs,
and on entry to Prefill::run prevents state pollution across requests.
The merge of upstream main (99b4097) left 28 migrated copies behind. They are refreshed by re-migration:
dpct over the old and the merged tree, sycl/tools/normalize.sh (dpct's names, unchanged files, message
serials, fixups.py) on both, then a 3-way merge per file. Copies dpct never produced (verify.cpp, mtp.cpp)
take upstream's K8V4 diff by hand. The PTX gate fixup follows upstream's new __HIPCC__ guard (#elif).
The AOT build never received -cl-fp32-correctly-rounded-divide-sqrt (it went to the spir64 backend, which
an AOT build does not have): quantize_act_parity had its 303k mismatches back. The option now reaches
ocloc through -device "... -options ...". 19 of 22 parity tests pass again; kv_hybrid_parity (new) passes
all but its tensor-core step and is built, not registered.
B70, merged engine, AOT, --stream-experts --no-prefill-borrow --spec 4 --mtp: 2,184-token prompt 566.5
tok/s (was 475), decode 38.3 tok/s at that context (was 42.7; 77% draft acceptance on this text), MTP
prompt pass 38.6 ms (E-9 batched).
bw probe (incompressible data): 4-byte loads stream at 395 GB/s, 16-byte at 596. The Q6_K matvec sits at
130-160 GB/s in every variant tried (lanes per row, rows per warp, unroll, 16-byte loads, software
pipelining): latency-bound on a dependent load->unpack->dp4a chain, not ALU or bandwidth. The 16-byte
kernel is kept for n_out >= 4096 (+13% there, slower on small shapes); rel 5e-8 against the shared one.
Decode 20.80 tok/s (spec 2). The drafter is the lever now: MTP fetch + pack running.
The prompt path collects hundreds of blob pointers per layer and copies from them on other threads later;
the 8-slot ring handed back pointers that were overwritten by then, and its index was not thread-safe.
Slots are now keyed by (layer, expert), reused oldest-first among 512 (~1.4 GB), under a mutex.
Long prompts also hung in the prompt path's streamed-expert handshake (lent cache slots make experts
non-resident, the host supplies them behind a flag the GPU wait kernel does not see). --no-prefill-borrow
keeps every expert resident and removes the handshake: 2,000 tokens 693 tok/s, 2,185 tokens 560 tok/s,
300 tokens 180 tok/s (llama.cpp on the same card: 150 short, 424 at 105k).
Rig: watchdog reports and kills a hung run at the expected time (EXPECT_S/QUIET_S); --spec 2 default
(20.75 tok/s decode vs 16.95 at spec 4: the suffix drafter accepts 9%).
- exit segfault: dpct's static global_memory/constant_memory objects destructed after the default queue
(queue::get_device in the core); heap-allocated, never freed. Exit code 0; unitrace can write its table.
- Dockerfile.unitrace (pti-gpu unitrace in the dev image) + sycl/rank_kernels.py: per-kernel device time.
- Q6KTraits: the column-independent unpack (shifts, masks, saturating byte subtract) moved into load();
acts()/dot() split; a row-blocked kernel (native_mmvq_rowwarp_kernel, STRATA_MMVQ_RPW) kept but off:
with a 300 ms warm-up the shared kernel already does 2560x2560x4 in 48 us (110 GB/s) and the variants tie.
- mmvq_bench: the dense kernel timing harness (warm-up + 400 iterations; a 5 ms run measures the clock ramp).
route<NE> templated on the expert count; native_router_top10_multi_ne dispatches 256/512. Both call sites
(the verify window and the per-token moe_route) take it for the Coder. 64 tokens: 16.25 -> 16.95 tok/s,
same output. Also: Dockerfile.unitrace (per-kernel Level Zero profiling image).
A window routes the same expert for up to 6 tokens; the grouped kernels recomputed the dequantised weights
(grid lookups, sign unpacking, the byte-wise compare/xor/subtract emulations) once per token. Multi<TY>
splits each of the five formats this model uses (iq2_s, iq3_xxs, iq3_s gate/up; iq4_nl, q2_0 down) into
prep (once) and acts + finish (per token); the kernels take entries four at a time. Lanes per row is a
compile-time constant (STRATA_EXPERT_LANES, 8): 32 lanes on an 80-call row was mostly reduction.
Measured with the new NATIVE_BENCH=1 mode of native_expert_parity (10 experts x 4 entries, one layer):
iq3_xxs/iq4_nl 1.467 -> 0.649 ms, iq2_s/q2_0 ~0.68 -> 0.579 ms, iq3_s/iq4_nl 0.750 ms; parity unchanged.
strata::dp4a wraps dpct::dp4a (sycl::ext::oneapi::dot_acc is the same emulation with a header that breaks
the link across translation units).
The -Xs device flag was mis-quoted; fixed. Measured on the B70 with the same 19-token prompt, 32 tokens:
decode 15.17 tok/s, prefill 28.79 tok/s, a 4-token window 73 ms on the card. The persistent JIT cache
(SYCL_CACHE_PERSISTENT=1) is the alternative for the JIT build: 15 MB, second start 15.6 tok/s.
Measured on the B70: 64 tokens in 52.0 s, of which ~47 s is the first window (the runtime JIT-compiles
every kernel on first use: 6 tokens take 47.5 s, 64 take 52.0 s). The output is the model's real answer.
Debug tooling kept in the port: STRATA_VERIFY_EAGER=1 runs the window on the queue instead of as a graph
and turns the stage stamps into host clocks, so STRATA_VERIFY_PROFILE=1 prints ms per stage without a
device timer; STRATA_VERIFY_DEBUG=1 prints the residual after every layer and the round's phase times.
build.sh takes BUILD_DIR and AOT (an ahead-of-time bmg-g31 build removes the JIT cost).
--stream-experts (GgufExpertSource): every expert goes straight from the GGUF into the VRAM cache through a
small ring of staging buffers; no resident host arena, so the engine runs on a 23 GiB machine.
What it took to get from "loads" to "<think>\nThe user wants a Python function":
- the doorbell handshake through host-mapped memory: volatile device loads do not bypass the caches on
Intel; system-scope atomics (strata::sys_load/sys_store) do. A device-side spin is bounded (kSpinMax): an
unbounded one wedged the GT twice when its process died.
- STRATA_VERIFY_NO_HOST=1 + STRATA_VERIFY_DEVICE_PLAN=1: with every expert resident the GPU plans each
layer itself and the host waits for the whole window graph; per-layer ring visibility inside a graph is
not reliable on this platform (measured with sycl/probe).
- -fsycl-default-sub-group-size=32: the kernels are written for 32-lane warps; dpct pinned only 134 of 289.
- cudaMemcpy is synchronous; dpct's default-queue memcpy had no wait, and the streaming ring reuses its
buffers - the expert cache filled from overwritten memory (non-deterministic residuals, NaN by layer 5).
- native_expert_parity hand-ported (dpct cannot parse it without ggml-cpu.h): the GPU native expert
kernel matches the float reference on real IQ1_M rows.
- verify.cpp debug readbacks (STRATA_VERIFY_DEBUG): the per-layer residual ladder that found all of it.
Not yet: speed (0.17 tok/s - the spin budgets), the segfault at exit, the tensor-core kernels.
Rebased onto main (0.1.22): the portable pre-sm_80 QSA scorer and the sm_80
guard on the tensor-core prompt attention are already in main, so this is
the rest of the port.
- the runtime floor: device_info refuses anything older than compute
capability 7 (main's runtime check still requires sm_120, so a binary
built for sm_75 would never start),
- fused_gr: the down kernel carries the tokens in slices that fit the
card's shared-memory opt-in (Turing, 64 KB: 6-token slices; 4 tokens when
the card reports no opt-in, under the 48 KB default), merged on top of
main's per-device opt-in - full 8-token windows no longer fail on Turing
with "invalid argument",
- the setup/CMake floor is 7.5 (RTX 20 / 30 / 40 / 50) without an
experimental flag: CMake's arch refusals, setup.py's gpu_problem and the
GPU hints and error texts,
- docs/MULTI_GPU.md says what a Turing card actually runs.
Verified end-to-end on RTX 2070 (sm_75, 0.1.20 base): model loads, OpenAI/
Anthropic API serving works, greedy output identical to the sm_120 build.
Merged (PR #87) with two changes: the runtime floor is 7.5 exactly (major*10+minor < 75; the PR let Volta 7.0/7.2
through), kept beside main's HIP check; and a card reporting no shared-memory opt-in slices the tokens to what fits the
48 KB default for its own tile (4 tokens on CUDA, all 8 with HIP's smaller tile, so gfx1100 keeps one launch).
The tensor-core prompt-attention kernel is already refused at runtime on AMD
(line 695), but the file unconditionally included <math_constants.h> for
CUDART_INF_F, which is not present on a ROCm-only build. Replace it with the
portable INFINITY macro from <cmath> so the HIP build succeeds without a CUDA
toolkit installed.
The port lives in sycl/ and changes no upstream file: sycl/src and sycl/include hold only the files the
migration touched, sycl/CMakeLists.txt takes everything else from the original tree.
How it was made, reproducibly (sycl/tools/): a dev image (the llama.cpp SYCL image + SYCLomatic 2025.3 +
CUDA 12.8 headers from the pip wheels), migrate.sh (compilation database + dpct over 86 translation
units), fixups.py (every hand fix with its reason, idempotent), build.sh (icpx, ninja).
What dpct got wrong and the script repairs: CUDA's null stream became a null sycl::queue* (strata::q_of
maps it to the default queue); __ldg((const float*) p) lost its cast and read one byte; __fadd_rn lost
parentheses; ggml's lookup tables were threaded through kernel parameters with the wrong table per
template; helper headers renamed since 2025.3; cudaGraphUpload, graph introspection and %globaltimer
have no equivalents. Two compiler flags are load-bearing: -fp-model=precise, and the device compiler's
-cl-fp32-correctly-rounded-divide-sqrt - the Arc's fp32 divide is not correctly rounded by default, and
quantize_act_parity went from 303k mismatches to byte-exact with it.
Not ported: the three kernels with inline PTX (mma.sync, ldmatrix, cp.async) take their existing
pre-sm_80 fallback; the launchers refuse the device. The XMX joint_matrix versions are next.
The engine itself starts on the card, loads the pack, the projections, the PLE table and the 23.4 GiB
expert arena, and is then stopped by the test rig's memory guard: Strata keeps the experts as a resident
host copy, and the rig has 23 GiB of RAM (upstream asks for 32). A streaming fill of the VRAM cache
from the GGUF is the next piece; the whole model fits in the 32 GB card.
Prompt reading scales up with prompt size (150 tok/s at 2.7k, 424 tok/s at 105k), so the full
window costs about five minutes, not fifteen. 30.3 GB in use with 1.5 GB free at 131,072.
About 20 KiB of KV per token (only 12 of 48 layers are full attention). 32k, 64k, 96k and 128k all
load with the whole model on the card, 128k with 1.6 GB to spare; 160k is under the safety margin
and 256k does not fit - asking for it spilled VRAM into host RAM and took the machine down until its
watchdog reset it, so the doc says so in bold.
The generic prompt kernel now templates K and V element types separately, so
mode 3 is mode 1's INT8 K path (codes, scales, i8x2 MMA conversion) composed
with mode 0's fp16 V path: the row's q4_0 blocks dequantize straight into the
fp16 V row at gather. The caller's output Hadamard is unchanged. Pure Q4_0
pools still refuse (whole-format q4 stays on the old kernel).
The v2 cp.async int8 kernel (launch_i8) is untouched; mode 3 uses the generic
launch<>, as fp16 does. Parity's [5/5] flips from a refusal check to a numeric
check against the dequant reference (accuracy-level, not bitwise, like the
kernel's own int8 variant).
K stays INT8 (unrotated - the attention scores stay exactly int8's), V uses the
rotated q4_0 of kv_q4.hpp: 816 B per cell vs int8's 1,056. The K/V quantization
asymmetry is the point: K enters through the softmax scores, V only through a
softmax-weighted average (llama.cpp's q8_K/q4_V hybrids reason the same way).
No new kernels: the append/gather entry points already select K/V by blockIdx.z
and take separate K/V pointers, so the hybrid composes the existing validated
q8/q4 kernels with the unused half's lanes folded onto the used pool (a
bit-identical duplicate write). The fused attention gets KV_MODE 3 (K-side int8
loader, V-side q4); 0.1.22's tensor-core prompt path (qsa_prompt_attn) refuses
the hybrid pools through its !v_q check and the old kernel runs - covered by
the parity test. The MTP drafter stays INT8 (mtp.cpp toggles the format globals
around its state creation, any ring shape); qsa_kv_format exits on a hybrid
state rather than misreporting a layout (the kernels' own unsupported-geometry convention); the hybrid appends pass nullptr for
the host mirror (mode 0 has none). setup.py excludes k8v4 from the whole
auto-streaming decision (the engine refuses --kv-resident with it).
kv_hybrid_parity: bitwise append/gather through a non-identity page table,
step and batched mode-3 attention vs host references (<= 1.1e-07), and the
qsa_prompt_attn refusal - all at machine precision; kv_q8/kv_q4 parity unchanged.
Measured on a RTX 3090, Coder IQ1_M, 118.75k-token prompts, temp 0: decode
82-97 t/s (int8: 75; the band is draft-acceptance variance), prefill unchanged
~1,130-1,170 t/s, 26-needle seed 42 identical to int8's tokens, KV 1.39 GB at
131k (int8: 1.80). At --max-context 204800 on 24 GB: 1,024 t/s prefill,
82-91 t/s decode, 25/26 needles at 197.9k.
Without nvidia-smi, setup now looks for an Intel discrete GPU in sysfs (vendor 0x8086 under xe or
i915, named by PCI id - lspci's database does not know an Arc Pro B70 yet) and takes a path with no
compiler and no CUDA toolkit: download the model, export the tokenizer, find a llama-server
(--llama-server PATH, one on PATH, or the B70 container image via docker), write a config with a
"llama" block and a start-llama.sh, and a run script that uses --engine llama.
The CUDA engine's rules that do not apply are stepped around rather than deleted: the compute
capability and driver-version checks, the experts-in-RAM gates (on llama.cpp the fit is VRAM,
shard 1 = the download minus the 28.8 GB lookup table), the MTP draft download (5 GB the llama
engine cannot use), and the engine's own argument list.
engine_from_config runs the config's start script when nothing answers at the url, so the run
script is one click and closing the server stops the container.
Verified on an Arc Pro B70: setup runs steps 1-7 clean against the downloaded Coder IQ1_M.
docs/INTEL.md has the whole picture and the measurements.
The Strata engine is CUDA. On a GPU without it - written for and measured on an Intel Arc Pro
B70 - the same GGUF weights run through llama.cpp's SYCL backend. serve/engine_llama.py puts
llama-server behind Engine.generate(ids, max_new, sampling, cancel), so the OpenAI and
Anthropic endpoints, streaming, tool calls, MCP and the web app are unchanged above it.
It sends the prompt as token ids (the server's own tokenizer, from the same GGUF) with
return_tokens, streams ids back with llama-server's timings mapped onto Strata's `last`,
heartbeats while a long prompt is read, and stops the server by closing the connection when a
client goes away. Attach to a running server ("llama": {"url"}) or spawn one ("llama": {"exe"}).
Not yet: images (Strata's Vision produces embeddings for the CUDA engine; llama-server takes
the image itself) - start without "vision" and image requests are refused cleanly.
Tests: a scripted llama-server (the wire format observed on b29c606), plus a live round trip
when STRATA_LLAMA_URL is set. Measured on the B70 with Coder IQ1_M: 23-25 tok/s decode,
~150 tok/s prefill, correct code, GPU-bound.
The prompt path's selection at long context was 18% of a 128K prompt. Two changes:
* block scores as a GEMM on tensor cores: rows (query, indexer head) x blocks, K = 128, relu per head summed
in the warp kernel's order; 3xTF32 (hi*hi + hi*lo + lo*hi) keeps FP32-level accuracy; the tail block (the
`dead` key, +1e9) keeps the warp kernel's arithmetic. sm_80+ (TF32 MMA), shared-memory opt-in per device;
STRATA_SELECT_OLD=1 (or an older card): the warp kernel. Not bitwise (another summation order).
* top-k with each query's keys read once into registers (1,024 threads, up to 33 blocks each: contexts up to
~135K), per-warp histograms, block scans - the same selection rule, identical ids (exact; decode uses it
too). STRATA_TOPK_OLD=1: the original kernel.
5070, 256 queries (src/kernels/qsa_select_bench.cpp): 32K 1.18 -> 0.36 ms, 128K 4.68 -> 1.33 ms; synthetic selections
identical 256/256, the register top-k identical to the reference.
Quality gate (Q2_0, vs STRATA_SELECT_OLD=1 + STRATA_TOPK_OLD=1): needles 8K/16K/32K x 5 depths 15/15 with both;
16K continuation 32/32 identical, KL 0.005; 32K top-1 same, KL 0.03 (the size of any FP32-level change, cf. D-1).
Prompt: 32K 1,742 -> 1,797 tok/s (select 897 -> 365 ms), 128K 1,608 -> 1,843 tok/s (select 14.6 -> 4.2 s),
time to first token 82.7 -> 72.3 s.
It says what happens (the experts must stay in RAM, so the OS pages part of them: much slower, maybe no start) and
asks, "no" by default, so an unattended --yes install still stops. Thanks @FatheredPuma81.
The opt-in was set once per process, but it is a per-device setting. The int8 kernel fits the default 48 KB; the
fp16 one (KV of contexts up to 8K) needs 54 KB, so on the second card of a split its launch failed.
Resolved against 0.1.22: E-6's device-plan waits and copies kept (the row copy of the CPU share is used when the
device plan is off - bit-identical, +0.0 on the GPU rows as the pool leaves them); the multi-GPU per-device
shared-memory opt-in and stage pool callback kept. Left out: the session-wide --pcie-frac 0 skip of the PCIe path,
because a request can ask for a PCIe share (pcie_frac=, used by --calibrate) and the window graph is captured once.
--prefill auto sized the chunk only when there were cache slots to lend; with none the buffers stayed at 8,192
tokens and a full card stopped at start with "device buffers ... do not fit". Thanks @sinand99.
The server sent only cvec= with GENI and the engine skipped every other key, so a request with an image decoded
greedily whatever the client asked. Both now pass the text path's keys. Defaults unchanged (greedy unless set).
Thanks @jleaders for pinning it down.
Its MMA and cp.async need sm_80: an sm_75 build stopped with ptxas errors. They now compile to a trap below
sm_80, and qsa_prompt_attn_batch refuses such a device at run time, so the old kernel runs there. RTX 30/40/50:
the same code (parity unchanged: vs FP64 3.17e-6).
The expert-arena load runs at ~0.05 GiB/s (13-14 min to ready) when the serve
is started from Task Scheduler with its registration defaults (Priority 7 /
Below normal, least-privilege token), versus 1.4-1.5 GiB/s (~35 s) from a
terminal or SSH - same binary, args and cache state.
- docs/DETAILS.md: measured table + the two registration settings that fix it
- src/program/generate.cpp: when load_gib_per_second() < 0.2 GiB/s, print a
hint naming this cause under the load line
Expose loaded-model metadata through /models and /v1/models.
Add /props and single-slot /slots responses with existing API-key
enforcement and regression coverage.
Validation: all 30 server tests pass. Pi 0.87.1 discovered the loaded
model with 262144-token context and thinking support, and completed
a chat smoke test successfully.
Fixes#117.
Co-authored-by: Codex (gpt-6-astra-medium) <[email protected]>
The GPU, VRAM, temperature and power cards read all the cards the model runs on: their total (VRAM, power, PCIe
traffic), mean (load) or hottest (temperature), with each card's own reading underneath; the GPU name lists them.
One card: unchanged.
The same copies on the same (legacy) stream, but queued instead of waited on one by one: the same bytes, ordered the
same against earlier work. Q2_0 on a 5070 (1,782 slots): 162 -> 96 ms per prompt, 4K and 32K outputs identical.
The rest (publishing slots in the background while decode starts) would hide the last ~2% of the time to first
token on a 4K prompt; not done. STRATA_REFILL_BLOCKING=1 keeps the old loop.
D-3 measurement: Q2_0, 128K prompt with KV streaming, 5070: staging 1,331 ms of 81.3 s (1.6%); staging only the
selected union could save part of that, so D-3 is not done. The same prompt: 1,144 tok/s on 0.1.21, 1,596 on this
branch (+39.5%), time to first token 115.7 -> 83.4 s.
vswhere -latest picked Visual Studio 2026 (version 18) on a PC that also has 2022, and CUDA 13.1's nvcc stops with
"unsupported Microsoft Visual Studio version". Now only 2019/2022 are looked for; with only a newer one, the 2022
Build Tools are installed as when none is there.
Several users could pick one GPU but not two. Now START-HERE.bat / setup.sh:
* lists every NVIDIA card with "can be used" or why not (older than the RTX 30 series; under 8 GB to share a model);
* asks, when two or more cards can share the model, which to use - the two best together recommended (newest
generation first), every combination and each single card offered; --yes takes the recommendation;
* asks once, at the next start of a model installed on one card, whether to use both from now on (kept);
* --gpus 0,2 / --gpus all / --gpu 0,2 at a start saves the cards for that model; --gpu N is for that start only;
* a card that cannot be used (or is gone) stops with its reason and the command that works instead.
The engine gets code for every chosen card (a local build compiles all their generations; the ready-made one is
checked against each), the context is recommended from the smallest card, and --calibrate measures the split.
One block per (query, KV head) walks its selected cells in chunks of 32 with an online softmax (no split-K
scratch, no merge kernel). q.k and p.v are FP16 MMAs with FP32 accumulation; int8 codes enter exactly, their
scales in FP32; q and p are split into hi + lo halves after an exact power-of-two prescale, so the result keeps
FP32-level accuracy. Int8 KV: each warp owns one 64-dim scale group and gathers its own K/V slices with
double-buffered cp.async. FP16 KV: the first version. Q4_0 KV, or STRATA_PROMPT_ATTN_OLD=1: the old kernel.
qsa_prompt_attn_parity (5070, synthetic): vs FP64 3.2e-6 (old kernel 2.5e-6); int8 4.0x faster (26.6 -> 6.6 ms per
2,048-query chunk at 32K), fp16 1.45x. Q2_0, 32K prompt: attention 5,216 -> 1,318 ms, prompt 1,386 -> 1,646 tok/s
(+18.8%). Not bitwise: the model amplifies any FP32-level change (v1 vs v2 of this kernel, same accuracy, another
summation order, differ as much as old vs new: residual rel L2 mean 0.060 vs 0.061). Needles 8K/16K/32K x 5 depths:
15/15 with both kernels.
Also STRATA_SEL_OVERLAP (debug): how many of the cells 16-query tiles read compared to one query at a time.
A user reported that he could pick a GPU but not two. With v0.1.21 (reproduced on a 5080 + 2080 Ti + 3090 PC):
* START-HERE.bat --gpus 0,2 on an installed model started it on GPU 0 alone: --gpus was saved only with --setup,
and the start passed its first card as --gpu, which overrides the saved choice;
* START-HERE.bat --gpu 0,2 stopped with "argument --gpu: invalid int value";
* an update unzipped next to a layer-split install stopped at step 1: "there is no GPU [0, 2]".
Now the first two start across both cards (the server gets --gpu 0,2 and splits the layers, auto), and an update
sets itself up across the same cards with the saved layer split. --gpu N still runs one card.
Two gaps kept the routing trace one-shot-only:
- drive_pool_multi, the verify-window dispatch the serve path uses for every
decode call, had no trace hook; --dump-routing under --serve opened the file
and wrote nothing. Records now go out per token in drive_pool's format
(layer, k, ids, weights). The multi dispatch fuses the router weights into
the kernel and does not surface them, so records carry unit weights:
make_profile.py ranks pairs by routed frequency, which is the signal that
matters; a one-shot run still records true weights.
- the trace sat in a stdio buffer until teardown, so a crash lost the whole
session's data. It now flushes at every request's DONE.
Verified live: a serve-mode trace grows per request, parses cleanly in
make_profile.py's format (all 48 layers, records per decode position), and
captures real multi-session traffic for expert-profile building.
One 512-thread block per head (48 blocks) walked the chunk. The value columns' recurrences are
independent; only the per-token output norm couples a head's columns. Now 4 blocks of 32 columns per
head (192 blocks, 128 threads per __syncthreads), and a parallel norm kernel with the same warp sums.
Per column the same arithmetic in the same order: residual dumps and continuations byte-identical to
0.1.21 (4K, 20K). Q2_0 32K, back to back: GDN 5118 -> 4264 ms, prompt 1258 -> 1308 tok/s.
STRATA_GDN_REC_HEADS=1: the per-head kernel. (A chunked delta-rule kernel - not bitwise - is the
larger step left.)
After routing, one thread checks the group's entries against the device residency table; when all
are resident it writes the plan the host pool would (same layout, same order) and marks the ring.
The layer's waits on the host (plan, PCIe share, CPU share) then return at once and the CPU rows
are zeros; the host still runs its loop, off the GPU's critical path. STRATA_VERIFY_DEVICE_PLAN=0: off.
A group of <= --spec rows was staged in mapped memory and synced before the next: ~5,500 host round
trips on a 32K prompt. The rows' token / step / position records are now uploaded once per call and
each group is device copies plus a graph on the one stream, one sync at the end (a graph variant
without the mapped staging). The same work in the same order: continuations byte-identical to 0.1.21
(4K, 20K). Q2_0 32K: the draft layer's share 1091 -> 905 ms; prompt 1213 (Phase B) -> 1387 tok/s with
Phase C and D-5. STRATA_MTP_PREFILL_SYNC=1 keeps the old loop.
Agent clients put screenshots in tool results; flattening non-user messages to text made the model blind to them. The template and images_of already handle them; only the conversion was too strict.
The thread that launches a chunk's kernels also issued the expert copies, and waited there for
the host copies of the unpinned blobs (the arena part Windows would not pin): the GPU idled - the
'wait copy' and 'dequant' time of the i-quant prompts. A per-chunk issuer thread now walks the
stream: the same copies in the same order into the same slots, a slot refilled only after the
compute stream recorded it free. STRATA_PREFILL_ISSUER=0 keeps the inline issue.
IQ3_S 32K prompt: 1143 -> 1213 tok/s, 'wait copy' 15.2% -> 7.5%. Exact: residual dumps and
continuations byte-identical to 0.1.21 at 20K (IQ3_S, IQ3_XXS: +16% prompt tok/s with Phase C).
The stager's ring and threads are tunable (STRATA_STAGER_RING / _THREADS); the defaults stay
(more copy threads were slower: 816 tok/s at 6).
Measured on a Ryzen 5 7600 (IQ3_S, 4K context, CLI decode timeline, outputs identical):
gate/up 37.1 -> 36.5 ms/round with a 2 KB prefetch; the gather decode (same results) is 3-5%
slower than the scalar lookups, so it is opt-in (STRATA_IQ_GATHER=1). More pool threads (SMT)
do not help: the pool reads its rows at ~30 GB/s, near DDR5's practical rate through 4 KB pages
- memory-bound, so decode speed now comes from VRAM hits (E-1) rather than CPU kernels.
The 4-tap conv reads inputs, not its own outputs, so 64-token tiles are independent: 160x the
threads of the per-channel walk. Same per-element expression (identical bits: residual dumps and
continuations byte-identical to 0.1.21, also with 1001-token chunks). STRATA_GDN_CONV_SERIAL=1: the walk.
- C-1: the block-score grid spans the batch's largest n_bid + 1 blocks instead of the context's
capacity (the blocks past a query's n_bid returned at once). The graphed decode paths keep the
capacity grid.
- C-2: the indexer's appends of a chunk in three launches (the first cell's spare, one CTA per
completed block, the tail) instead of one launch per token and QSA layer (24K per 2048-token chunk).
The end state is the sequential one's: the same arithmetic per block, only the batch's last block
writes the spare after it. STRATA_INDEXER_PER_TOKEN=1 keeps the old loop.
Exact: residual dumps and continuations byte-identical to 0.1.21 (4K, 20K, and 5K read in chunks of
1001 so blocks straddle chunks). Coder prompt tok/s with C-4: 4K 1063 -> 1255 (+18%), 20K 1216 -> 1373 (+13%).
The conversation cache's reuse is kept per request (GET /metrics shows it),
but the APIs did not report it, so clients showed a cached prompt as 0%:
- OpenAI: usage.prompt_tokens_details.cached_tokens, streamed and not.
- Anthropic: the final usage splits input_tokens from
cache_read_input_tokens (message_start can only know the whole prompt).
- llama.cpp's `timings` block on the last chunk and on a whole reply
(prompt_n, cache_n, prompt/predicted ms and per second), from the
engine's own clock: what llama.cpp's clients already show as speed.
- GET /v1/status (with the API key when one is set): the model and its
window, images, the APIs, whether a request is running, the last
request's timings and the GPU and RAM from the telemetry, for a
front-end that polls its server rather than guessing.
An engine without a clock (MockEngine) reports no timings and 0 cached.
Tests: serve/test_server.py UsageAndStatus.
live.tok_s returned `generated / (now - first_token)`: a cumulative mean whose
first sample is 1/elapsed. Against an engine paced at 25 tokens/s the Speed
card, the header pill and the tok_s sparkline read 27,235.7 tok/s at that first
sample - about 1090x the real rate - and then undershot for the first second of
every answer (one token counted over up to a second of elapsed time). Because
the sparkline scales to the maximum of its 60-sample ring, one such sample also
flattened the Speed graph for a minute.
tok_s is now the rate over the last RATE_WINDOW_S (2 s), with the mean so far
reported only while a request is younger than RATE_MIN_SPAN_S (0.25 s) and its
span floored there, so no sample can diverge. The old formula is kept as
_tok_s_mean() and still reported (live.tok_s_mean, status.tokens_per_s_mean).
The same request fixes a second mis-attribution: a request that ended without a
DONE line (engine death, error, client disconnect) recorded the PREVIOUS
request's generated/decode_ms as its own decode_tok_s. The engine's counters
are now trusted only when its DONE replaced the object captured at request
start, and the row carries the engine's own token count as engine_generated.
Tests: serve/test_server.py::LiveRate paces a mock engine and polls GET /metrics
at 10 ms. Both cases fail without the server.py change (990.0 tok/s recorded
for a request that never finished) and pass with it; the rest of the suite is
unchanged at 28 tests, OK. The engine's own per-request counters - what the
Speed card shows for "last request" - are untouched, and the server's progress
and summary log lines keep their whole-request mean on purpose.
- One batched gather per chunk instead of a launch per token (same arithmetic: identical bits).
- The prompt path's 'embed+steps' time was mostly the host reading the n-gram (PLE) rows from the SSD
with the GPU idle: one thread issued every 90-byte read (~12 us of kernel time each, ~80K reads/s).
DirectFile now only queues a read; a pool issues them (4 threads on Windows, overlapped; 16 blocking
O_DIRECT preads on Linux, which read them one at a time before - perf-review F-2). STRATA_IO_THREADS.
Q2_0 4K prompt: PLE 591 -> 291 ms, 941 -> 1010 tok/s.
- STRATA_PREFILL_TIMING also prints the host's share (chunk setup, the per-chunk callback).
Exactness: the prompt path's final residuals (STRATA_PREFILL_DUMP_R) and the continuation are
byte-identical to 0.1.21 at 4K and 20K tokens, IQ and S2 embedding tables.
Each stage saves its own part of a checkpoint when it has read the chunk that reaches the
position (Prefill::on_stage_chunk, its state synchronized at that chunk's end), by the same
rule the last stage uses; the last stage puts the parts together in on_chunk. They were off
across GPUs because the first stage is a chunk ahead when the last one reports.
- the image-position (mrope) table and the control vector's tables are per device: every
stage's rope and steering kernels read their own device's copy; picture rows go through
the split prompt path (stage 0 embeds them, the later stages take the hand-off)
- KV streaming: each stage's session already streams its own pinned KV copy; no longer forced off
- --mmap-experts: allowed (no pinned arena: every miss the CPU computes, prompts copy on the host)
- --expert-cache-remote with a split: the helpers run on the visible GPUs no stage runs on and
hold only experts no stage's cache holds
It re-decoded every generated id per token: 2 ms per token after 8K tokens, 4 ms after 16K.
Now 0.0005 ms at any length; the text is the same (serve/test_detok.py checks it against the
old algorithm on real and random token streams), and a complete character is no longer held
back behind an incomplete one.
Fitted on the 5080 + 3090 K sweep (18-32): it picks K=26, the measured best (prompt +18-20%
over the 5080 alone), and gives a slow third card (2080 Ti) a single layer.
The server sets CUDA_VISIBLE_DEVICES to the list and adds --layer-split ("layer_split" in the
config, auto by default); the Monitor reads the first (main) card.
build writes source.shard1_sha256 / shard2_sha256 into manifest.json (unless --skip-hash, which is how
setup builds), and nothing read them back: src/, serve/, setup.py, tools/ have 0 readers. A pack
verified against a different shard than it was built from reported tensor MISMATCHes, or passed under
--limit.
A full verify (no --limit) now hashes the shard it is handed and fails naming both digests when they
differ; a manifest without the field, or a --limit run, is unchanged. Shard 2's hash is recorded but not
checked here: verify is not handed that file. tools/test_shards.py: a matching hash passes, a wrong one
fails naming both, an absent hash or --limit skips. On main the wrong-hash case returns 0.
find() is first-match over the tensor directory, so a GGUF that carries one name twice resolves it to
whichever entry came first; the other tensor's bytes are never read and nothing says so. GGUF has no
index that could arbitrate between the two, so refusal is the whole fix: parse() keeps the names it has
seen and throws on the second sighting ("GGUF: duplicate tensor name 'blk.0.attn_q.weight' in <path>"),
as llama.cpp's gguf_init_from_file does.
gguf_reader_test (STRATA_BUILD_TESTS; no model, no GPU) writes a two-tensor GGUF v3 by hand: distinct
names open and find() sees both; the same name twice is refused and the error names the tensor and the
file. On main the second file opens: 3 of 4 checks fail.
setup checked a shard by presence only, and a download by Content-Length only: a shard copied short into
--gguf-dir, a .part renamed by hand or a download an older setup finished without a mark passed as a
model file, and the failure came later from the packer or the engine, at the first tensor past the end,
naming neither the file nor the shortfall.
check_shards() reads each shard's header (not its data) and compares the file's length with what its own
tensor directory needs: "m-00001-of-00002.gguf is short: 1,234 of 1,274 bytes (40 missing)"; a header
that does not parse is refused by name; a missing file stays "missing <path>". The download's "could not
finish" line now says the bytes on disk and the bytes the server announced.
tools/test_shards.py (unittest over a minimal GGUF written by hand; no download): whole shards pass; a
missing, a short and a header-truncated shard each stop setup with the file named.
--layer-split K1[,K2..]|auto with --split-device D1[,D2..]: each later stage runs its layer
range on its own GPU (weights, session, cache, verify window, prompt path). The prompt path
hands each chunk's rows to the next stage, which reads it on a thread while the previous
stage reads the next chunk. Checkpoints carry every stage's session state; adaptive swaps
copy into the layer's own cache; each GPU probes its own PCIe share. "auto" scores every
placement (up to 3 GPUs) by the profile-weighted pairs each cache would hold.
STRATA_EXPERIMENTAL_SM75: lets CMake build for a Turing card (the 2080 Ti A/B).
--split-device D (default: the second visible GPU) runs layers [K, n) on device D with
its own copy of the dense weights, session, head, drafter and an expert cache filled
from the profile's pairs of its layers. Verifier and MtpDrafter switch to the device
they were made on (OnDevice), the pool wrapper swaps the stage's cache pointers, the
penalty history lives with the head, and the s2 LUT is per device.
v1 switches off what does not cross devices yet: checkpoints, adaptive swaps, the PCIe
share and the batched prompt path (prompts read through the verify windows).
--layer-split K runs layers [0, K) and [K, n_layers) as two verify stages on the same device (sharing weights,
session and expert cache). The hand-off carries R, the last layer's pending write and inject, so the next stage
folds it in exactly as the unsplit window does. The GPU shared-memory opt-ins are now per device.
Each layer waits for the helper GPU on the CPU pool's critical path. Measured on RIBPC (Windows): ~0.3 ms
per round trip, more than the fast CPU needs for the same experts. The helper's context is created with
cudaDeviceScheduleSpin (STRATA_REMOTE_SPIN=0: the default), and its input/compact-output staging is mapped:
the kernels read and write it directly, two copies fewer per layer (STRATA_REMOTE_ZEROCOPY=0: copies).
The drafter loader had the same defect as the expert arena: read_file() pulled the whole
dense.bin through std::ifstream, and the 512-expert loop read 64 MiB chunks through the
same 4095-byte-per-call path. Same fix (fopen/fread, _fseeki64/_ftelli64, short-read
check), so the drafter stage stops paying per-read overhead.
On MSVC, std::ifstream::read does not issue one fread per call: basic_filebuf::xsgetn
loops over fread(_Ptr, 1, 4095, _Myfile) (__msvc_filebuf.hpp, _Read_size = 4095).
A 44.7 GB expert arena therefore costs ~11.4 million 4095-byte reads instead of a few
thousand bulk reads, and the load is dominated by per-call overhead.
Read the arena with fopen/fread, an explicit 64-bit seek (_fseeki64) and a short-read
check that fails the load (LoadStats.ok/error) instead of silently returning partial
data. Behaviour on Linux/glibc is unchanged in substance: same bytes, same call sites.
The page referenced its stylesheet, script, sprite, fonts and every API call
with absolute paths (/web/app.css, fetch("/health"), url("/fonts/...")).
Behind a reverse proxy that mounts the app under a sub-path - llama-swap's
/upstream/<model>/, an nginx subpath - those resolve to the proxy root and
404: an unstyled page stuck at Connecting, or a Monitor that polls /metrics
at the proxy root and reports 'Server not reachable'.
Make every same-origin reference relative:
- index.html: (href|src)="/web/... -> web/...
- app.js: fetch("/... -> fetch("..., plus the endpoint strings used outside
a plain fetch(" call - the Monitor's ternary fetch(reqShowAll ?
"/metrics?requests=all" : "/metrics") and the SPRITE constant - because
the quote does not always sit right after 'fetch('
- tokens.css: url("/fonts/... -> url("../fonts/... (fonts have their own
top-level /fonts/ route, one level above the CSS)
Standalone at / resolves the same, so nothing changes there.
Verified through llama-swap's /upstream/<model-id>/ prefix in a browser:
page, stylesheets, fonts, sprite icons, /health, /settings, /metrics, /mcp
and a streamed /v1/chat/completions POST all 200; the Monitor tab connects
and streams live telemetry. Same checks pass at / standalone.
The CMake arch guard already lets RTX 30 (sm_86) / 40 (sm_89) build - its own
comment says the kernels need sm_80 or newer (tf32 mma, bf16 math) - and the
README lists RTX 30/40/50 as supported. But device_info() still refused
anything with cc_major != 12, so a working Ampere/Ada build died at startup
with 'Strata targets sm_120 only'. Relax the runtime check to the kernels'
real floor (compute capability 8.0) and align the stale 120-only comments.
Verified on RTX A2000 12GB + RTX 3080 10GB (both sm_86), CUDA 12.0, gcc-13,
engine 0.1.20 serving daily: TG 21-24 t/s flat 1k-128k, 29.6 t/s code-gen at
81% draft acceptance, PP 375/445 @8k/32k, 8/8 needle retrieval at 113k.
A prompt read from token 0 is also checkpointed at its first turn boundary when that is 2,048 tokens or
more (--prompt-cache-root N, 0 = off). It becomes the chain's root, which #62's retention policy pins,
so a new chat of the same client reads only what follows the system prompt and tools. Without it, #62's
root existed only for system prompts past --prompt-cache-every (16K).
Co-Authored-By: code-martin <[email protected]>
The gate/up row and IQ4_NL down comparisons printed their distance from ggml's vec_dot without affecting the exit code, so a kernel that silently diverged still reported 0 failures. Gate both at 1e-5 (measured 3e-8 and 1e-7, so 100x headroom). Verified: 0 failures at 1e-5, and with the threshold dropped to 1e-9 the gate fires on every layer.
Also corrects the comment above the gate/up dispatch: STRATA_NO_IQ512 drops an AVX-512 CPU to the AVX-2 kernel, not to ggml; ggml's vec_dot needs both switches set.
- Reset cache hits/lookups baseline at start of autoregressive decode to exclude prefill distortion
- Append req_hits and req_look to engine DONE line and log decode hit rate to stderr
- Parse decode hit metrics in server.py and record hit_rate per request in history
- Add Hit rate column to Recent requests table in web monitor
A small MCP client (JSON-RPC 2.0, stdio and Streamable HTTP, standard library
only): servers from the run config ("mcp_servers", or Claude Desktop's
"mcpServers" block) or --mcp-config start with the server; their tools are
offered as <server>__<tool>. A request that sends "strata_mcp": true (the web
app does) gets them: the server runs the calls, feeds the results back and lets
the model continue, up to mcp.max_rounds; the stream carries strata_mcp events.
Other clients are unchanged. Errors, crashes and timeouts become "error: ..."
results; long results are cut at mcp.max_result_chars; Stop cancels a running
tool. GET /mcp lists servers and tools; the web app shows each call as a
collapsible block, a switch in the Sampling drawer and a Monitor card.
The mock engine takes --script more than once (answers in turn).
tools/calibrate.py starts the engine through serve's StrataEngine and measures decode speed with the
three settings that depend on the machine: --pcie-frac and --spec-min-p through new per-request GEN
keys (pcie_frac=, spec_min_p=; the server forwards a request's "strata_tune"), then 2-3 worker counts
(a restart each). A setting is kept only when it beats the product default by more than 3% in an
interleaved re-measurement. Setup offers it after an interactive install and runs it with --calibrate;
the result is saved per PC + model + context in the settings file and re-applied when a config is
written, so updates keep it. The engine's INFO line now says pool_workers, pcie_frac and spec_min_p.
tools/test_calibrate.py tests it with a stand-in engine (no GPU).
On Windows every allocation on the graphics card is also charged to the system commit (RAM + page file).
With the page file off or tiny, the expert cache's one big cudaMalloc failed while the VRAM was free and
the engine stopped. An auto-sized cache now retries at three quarters of the size (up to 8 times) and
says how much commit Windows has left; a failure that remains names the page-file setting. Setup warns
when the page file is under 4 GB; DETAILS has a troubleshooting row. STRATA_TEST_CACHE_FAIL=N fakes N
failed opens to test the retry.
- serve: a verify window samples T rows, but only row 0's penalty history was staged; rows 1..T-1 read
unwritten slots (past the allocation at penalty_last_n=4096). Every row now gets its own history
(strata::kernels::penalty_rows: the consumed tail, the fed-back token, the drafts before that row).
- prompt windows (their picks are discarded) skip the head sampling: Verifier::set_head_sampling.
- sampler: history ids outside [0, n_vocab) are ignored (they set bits past the shared bitmap); sampled
top_k 0 or > 64 means 64 and every row is written (0 used to leave the row unwritten).
- server: a client's top_k 0 or > 64 is sent as 64 (it fell back to the default 20).
- CLI: the speculative loop gets the CLI's sampling (it was always greedy), the per-token path draws
Philox(seed, position) like a verify window, and the effective sampling is printed.
- sampler_parity: fixtures 12-15 (per-row histories T=1..8 x H=1..4096, out-of-vocab ids, the top_k
contract, penalty_rows); outputs are pre-filled so an unwritten row fails.
- setup requires engine 0.1.19.
Found by the engineering performance review (C1, C2).
Bring in 45 upstream commits through f32577e. Update the split-GGUF test fixture with router geometry required by upstream's dynamic expert counts. Preserve the Orca BF16 conversion and native PLE compatibility paths.
Validation: Release CUDA build for sm_120; 8 packing tests; 23 server tests; sampler, gated residual, and KV streaming GPU parity checks.
The retained checkpoints are always a prefix chain - the request loop drops
any checkpoint that is not a prefix of the current prompt, because its
positional cells are gone (the KV cache is one arena, one branch of history
at a time). A radix cache's tree collapsed onto the one branch the session
can hold, and the chain's root is the node a radix cache keeps while its
leaves rotate: in practice the end of the system prompt, which every new
chat of the same client mounts through.
First-in-first-out kept that node only by accident of being first: after
prompt_cache newer checkpoints it was gone, and the next chat re-read the
whole prefix at prefill speed (~30 s per 30K tokens). Now the root is
pinned while the budget allows root and leaf apart (cap >= 2; below that
the old behaviour stands), the rest rotates by least recent use, and a
mount re-dates the checkpoint it mounted through.
The policy is one pure function (conv_cache.hpp) with its scenarios in
conv_cache_test.cpp; checkpoint_save/restore and the state format are
untouched, so STRATA_CKPT_REREAD's token-for-token guarantee holds.
`use_bits` required only a non-null history, but the launch sizes the bitmap
for `penalty_last_n > 0` alone - a caller handing over a stale history buffer
with the penalties disabled made the kernel zero `bits_words` words of a
ZERO-BYTE dynamic shared allocation. Unreachable from the serve path
(set_history passes nullptr when the window is empty) and invisible to
memcheck (the write lands in the unallocated SMEM window), but undefined per
the contract of the public `sample_tokens`. The gate now also requires a
non-empty window (`hlen > 0`, both kernels), and fixture 10 pins a stale
history with last_n = 0 to the no-history result in both kernels.
- A config's "gpu" (or the server's --gpu N) runs the engine and the image
encoder on that card: CUDA_VISIBLE_DEVICES=N with CUDA_DEVICE_ORDER=PCI_BUS_ID,
so N means what nvidia-smi and the Monitor's NVML mean (CUDA's own default,
fastest first, can number the cards differently). The Monitor reads that card.
- setup lists every NVIDIA GPU, picks the one with the most VRAM (or --gpu N),
says so when there is more than one, and saves the choice; START-HERE.bat
--gpu N also overrides it when starting. A number that does not exist stops
with the list of cards. Configs without "gpu" run as before.
- Tests: the child environment; the choice on a simulated 4060 + 5070 PC; end to
end on the real engine with --gpu 0.
The sampled chain applied the repetition/frequency/presence penalties a second
time after the temperature, and ran min_p before top_p: both came from the
order of the case labels in llama.cpp's common/sampling.cpp, not the order its
chain runs (params.samplers: PENALTIES ... TOP_K ... TOP_P, MIN_P ...
TEMPERATURE). With presence_penalty 1.5 at T 0.7 a repeated token was drawn at
2.6% instead of 10.5% (@hendrikp's example). Greedy decoding is unchanged.
sampler_parity: the host reference follows the same order; two new fixtures
check against an independently computed distribution (#53's example over 8,000
draws: 0.1040 vs 0.1050) and that top_p-before-min_p is observable. Both fail on
the old kernel (642 and 91 draws differ).
Builds its long texts from this repository's own files at run time, hides a
code word at the chosen depths and asks through the API. Checked on the Coder:
1K, 32K (10/50/90%) and 128K found.
- POST routes ignore the query string (Claude Code calls /v1/messages?beta=true
and got a 404, which it reports as a model problem); /settings too.
- A system (or late developer) message after the first one becomes a user
message in place: the chat template only takes a system message at the start
and raised "System message must be at the beginning" (the client saw a dropped
connection). Kept in place, not merged into the first, so the conversation
cache still reuses the prompt.
- Tests with the issues' own requests (they fail on the old code); DETAILS:
how to point Claude Code at Strata.
pcie_frac's defaults (0.55 native / 0.2 canonical) were measured on a PCIe 4.0 x16 link.
A x8 card in a x8 slot carries about half of that, and the copy engine's share then waits
for DMA that arrived late. Probe the H2D bandwidth once at startup the way the engine uses
it (DMA reads from pinned host memory) and scale the default by it: an RTX 5060 Ti at x8
probes 14.1 GB/s -> pcie_frac 0.29 instead of 0.55. --pcie-frac still wins when given, and
a machine that cannot run the probe keeps the current default.
description:Something in Strata doesn't work as documented
title:"[Bug]: "
labels:["bug"]
body:
- type:markdown
value:|
Thanks for the report.
Please search the [open issues](https://github.com/Niko1221/Strata/issues?q=is%3Aissue+state%3Aopen) first — this may already be tracked.
- type:textarea
id:what
attributes:
label:What happened
description:What you did, what you got, and what you expected instead. Paste the exact error text if there is one. If there's a set of steps or commands that reproduces it, put them here too.
validations:
required:true
- type:input
id:hardware
attributes:
label:Strata version, GPU, OS
description:The engine version is on the About tab, or in `GET /v1/status` under `engine`.
placeholder:"0.1.31 · RTX 5070 12GB · Windows 11"
validations:
required:true
- type:textarea
id:log
attributes:
label:Engine log
description:Drag in `strata-<model>.log` from your Strata folder. Drag and drop works directly into this box.
validations:
required:false
- type:markdown
value:|
A couple of extras that help a lot:
- Check the [troubleshooting table](https://github.com/Niko1221/Strata/blob/main/docs/DETAILS.md#troubleshooting) first, many reports are already solved there.
description:Suggest something new or an improvement
title:"[Feature]: "
labels:["enhancement"]
body:
- type:markdown
value:|
Explain what you want in plain language.
- type:textarea
id:idea
attributes:
label:What do you want to do?
description:Describe the goal rather than the implementation. If there's a workaround you use today, mention it.
validations:
required:true
- type:markdown
value:|
All of this counts: model and architecture support, the server and API, the web app, multi-GPU and layer split, images and vision, MCP servers, setup and installation, and performance.
<p align="center"><b>Ein KI-Modell mit 125 Milliarden Parametern auf deinem eigenen Gaming-PC</b><br>
NVIDIA- oder AMD-Grafikkarte (ab 12 GB) · Windows oder Linux · kostenlos und Open Source</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Ein Voxel-Pagodengarten, den das Modell in Strata geschrieben hat, läuft im Browser"></a><br>
<sub>Ein Voxel-Pagodengarten aus einem einzigen Prompt, auf einer RTX 5070 mit Strata (IQ3_S, 128K Kontext) ·
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">ganzes Video (49 s)</a></sub></p>
Strata lässt **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** auf einem normalen PC laufen.
Das ist ein großes, kluges KI-Modell, das sonst meist einen Server braucht. Es chattet, schreibt Code, versteht
Bilder und arbeitet mit deinen Apps und Coding-Agents zusammen. Nichts verlässt deinen PC.
## Wie schnell ist es?
Wir haben es auf zwei ganz normalen Gaming-PCs gemessen. Ein Token ist etwa ¾ eines Wortes.
- **Schreibt Antworten:** wie schnell die Antwort in einem kurzen Chat erscheint. 60 Tokens pro Sekunde sind
schneller, als du lesen kannst.
- **Liest deinen Prompt:** wie schnell es aufnimmt, was du schickst (hier ein Dokument, Code oder Chatverlauf mit
NVIDIA: Q2_0 mit Engine 0.1.36, die anderen Zeilen mit 0.1.26 (4K Antworten, 32K Prompts). Die vollständigen
Tabellen stehen in [DETAILS.md](docs/DETAILS.md#speed-measured). Eine Karte mit mehr VRAM ist schneller: Eine
RTX 3090 (24 GB) sollte etwa 100-140 Tokens pro Sekunde schreiben. Lange Chats und andere Karten:
[Tempo jedes Modells](docs/MODELS.md#how-fast-is-each-size), [Ergebnisse aus der Community](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata ist kostenlos. Wenn es auf deinem PC gut läuft, hält ein Kaffee die Arbeit daran am Laufen.</sub></p>
## Was du brauchst
| | |
| --- | --- |
| **Grafikkarte** | **NVIDIA** GeForce RTX 20, 30, 40 oder 50 Serie, oder **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 oder RX 6800 / 6900 Serie. Sie braucht **12 GB VRAM oder mehr**. |
| **RAM** | 32 GB oder mehr. Dein RAM entscheidet, [welches Modell](#welches-modell-soll-ich-nehmen) passt. Mit 64 GB laufen alle Größen. |
| **Festplatte** | Etwa 80 GB frei. Nimm wenn möglich eine SSD: Der erste Start geht dann viel schneller. |
| **System** | Windows 10 / 11 oder Linux und ein aktueller Grafiktreiber von NVIDIA oder AMD. |
Alles andere richtet der Installer ein. Zwei oder drei Karten können sich das Modell teilen ([Multi-GPU](docs/MULTI_GPU.md)).
Experimentell, von Community-Mitgliedern auf ihren eigenen Rechnern geschrieben und getestet:
- **Intel Arc**, unter Linux aus dem Quellcode gebaut: [Intel Arc](docs/INTEL_ARC.md).
- **AMD Ryzen AI Max (Strix Halo)**, unter Linux aus dem Quellcode gebaut: [Strix Halo](docs/STRIX_HALO.md).
- **Ältere Prozessoren ohne AVX2**: Sie funktionieren, aber langsam. [Ältere CPUs](docs/INSTALL.md#older-cpus-experimental).
Die vollständige Liste: [docs/INSTALL.md](docs/INSTALL.md#what-you-need).
## Installieren
### Lass deine KI es einrichten
Nutzt du einen KI-Coding-Assistenten (Claude Code, Cursor, Codex, GitHub Copilot, ...)? Füge dort diesen Text ein:
```text
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
```
Er prüft deine Grafikkarte, deinen RAM und deine Festplatte und wählt das passende Modell. Dann installiert und
startet er es und sagt dir, wie du deine Apps verbindest. KI-Tools können Strata auch über seinen
[MCP-Server](docs/MCP_SERVER.md) installieren, starten und stoppen.
### Oder mach es selbst
[Lade Strata herunter](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) und entpacke es (oder nutze
`git clone`). **Windows:** Doppelklick auf **`START-HERE.bat`**. **Linux:** Führe im Strata-Ordner **`./setup.sh`** aus.
**Mac (Apple Silicon, M1 bis M5, macOS 15 oder neuer):** Installiere einmal Apples Command Line Tools (`xcode-select --install` im Terminal; Xcode selbst brauchst du nicht). Öffne dann das Terminal im Strata-Ordner und führe **`./setup.sh`** aus. Es baut die Mac-Engine auf deinem Mac (etwa 7 Minuten), stellt dieselben Fragen, lädt das Modell und startet es. Beim nächsten Mal startet `./setup.sh` sofort. Welche Größe zu deinem Mac passt und wie schnell es ist: [docs/MAC.md](docs/MAC.md).
Die Schritte sind für NVIDIA und AMD gleich. Der Installer erkennt deine Karte und richtet die passende Engine
dafür ein. Er stellt dir ein paar Fragen:
- welches Modell und welche Größe,
- wie viel Kontext (wie viel Text sich das Modell merkt),
- ob es Bilder lesen soll.
Drück jedes Mal Enter, um die empfohlene Antwort zu nehmen. Dann lädt er das Modell herunter (etwa 70 GB) und
startet es. Bricht der Download ab, starte ihn einfach neu: Er macht dort weiter, wo er aufgehört hat. Dein Browser
öffnet die Strata-App unter `http://127.0.0.1:8080`.
> **Während das Modell startet, kann dein PC 1-3 Minuten langsam sein oder nicht reagieren** (beim ersten Mal am
> längsten). Strata lädt 35-55 GB in deinen RAM und reserviert einen Teil davon für die Grafikkarte. Das ist normal.
> Warte und schließ das Fenster nicht. Im Fenster siehst du, was Strata gerade tut.
**Beim nächsten Mal** startest du wieder `START-HERE.bat` (oder `./setup.sh`). Es startet sofort und lädt nichts
doppelt herunter. Schließ das Fenster, um das Modell zu stoppen. `UPDATE.bat` (`./update.sh`) aktualisiert Strata,
ohne es zu starten. Updates, Docker, mehrere Karten, wo die Dateien landen und alle Optionen:
[docs/INSTALL.md](docs/INSTALL.md).
## Welches Modell soll ich nehmen?
Der Installer empfiehlt eins passend zu deinem RAM. Dasselbe Modell gibt es in mehreren Größen, mehr oder weniger
stark komprimiert. Kleinere Größen sind schneller. Größere sind etwas klüger.
| Dein RAM | Nimm | Warum |
| --- | --- | --- |
| **32 GB** | **Coder** | passt in 32 GB und ist für Code gemacht (mit einer 24-GB-Karte laufen auch Q2_0 und IQ2_XS) |
| **48 GB** | **IQ2_XS** (oder Q2_0, das schnellste) | die größeren Größen passen nicht |
| **64 GB** | **IQ2_XS** (empfohlen) oder IQ3_XXS / IQ3_S | alle Größen passen; IQ3_S ist das beste und das langsamste |
| **96 GB oder mehr** | **IQ3_S** oder Unsloths UD-IQ4_XS (~4 Bit) | Platz für die größten Größen, auch wenn alles andere offen ist |
- **[Coder](docs/MODELS.md#coder):** eine Version für Code, bei der die Hälfte der Experten entfernt wurde. Sie
erreicht 91 % des SWE-bench-Verified-Werts des vollen Modells (von ihren Autoren gemessen) und passt in 32 GB RAM.
Außerhalb von Code ist sie schwächer, auch bei chinesischem und anderem CJK-Text (#438). Nimm dafür Q2_0, IQ2_XS
oder IQ3_S, die alle Experten behalten.
- **[Swift 1.5](docs/MODELS.md#swift-15):** ein Fine-Tune, das viel kürzer nachdenkt, bevor es antwortet. Du
bekommst die Antwort früher, bei etwa gleicher Qualität.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** die ~4-Bit-Version von Unsloth, in der Qualität
zwischen IQ3_S und UD-Q4_K_XL. 94 GB Download. Mit weniger als ~80 GB RAM liest Strata einen Teil davon während
der Antwort von der SSD, dort ist es also langsamer (eine NVMe-SSD hilft).
- **[Unsloth UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL und Q8_0](docs/MODELS.md#unsloths-files-ud-iq4_xs-to-q8_0):** 4 bis 8 Bit, näher am vollen Modell
(93-95 % das gleiche nächste Token wie Q8_0). Schnell, wenn alle Experten in den RAM passen (~88 GB für UD-Q4_K_XL
bis ~136 GB für Q8_0); mit weniger RAM liest Strata einen Teil davon während der Antwort von der SSD, was viel
langsamer ist (7-8.5 Tokens/s für UD-Q4_K_XL auf einem PC mit 64 GB).
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** Das richtest du von Hand
ein. Es steht nicht im Menü des Installers.
Größen, Downloads und was wohin passt: [docs/MODELS.md](docs/MODELS.md). Um später ein weiteres Modell
### Optional: mehr Tempo mit einer Grafikkarte, wenn das ganze Modell in den RAM passt
Zwei Opt-in-Einstellungen (von RubenTorney, #1904 und #1905). Sie helfen, wenn du eine Grafikkarte nutzt und dein RAM alle Experten hält (Stratas Arena-Modus, zum Beispiel Q2_0 oder IQ3_XXS auf einem PC mit 64 GB). Öffne deine Konfiguration `strata-<model>.json` im Strata-Datenordner ([wo](docs/INSTALL.md#where-things-are-stored)) und füge hinzu:
- **Schnellere Ausgabe:** `"--adapt-async", "1"` in die Liste `"args"`. Der Experten-Cache wird dann in einem Hilfs-Thread aktualisiert, statt die Antwort anzuhalten. Gemessen auf einer RTX 5070 (Windows, Q2_0): +4-5 % Ausgabe, +7-10 % bei Code und langen Dokumenten. Antworten können leicht vom Standard abweichen (sie bleiben von Lauf zu Lauf gleich).
- **Schnelleres Prompt-Lesen unter Windows:** `"STRATA_PIN_BY_NEED": "1"` im Block `"env"`. Windows kann nur etwa den halben RAM für die Grafikkarte sperren; damit werden zuerst die Experten gesperrt, die die Karte wirklich braucht. Gemessen auf der RTX 5070: +4,4 % Prompt-Lesen.
```json
"args":[...,"--adapt-async","1"],
"env":{"STRATA_PIN_BY_NEED":"1"}
```
Starte Strata danach neu (`START-HERE.bat` oder `./setup.sh`). Zum Rückgängigmachen entferne die beiden Einträge.
## Etwas ist schiefgelaufen?
- **Mein PC ist beim ersten Start von Strata eingefroren.** Das ist normal, während das Modell lädt. Warte und
schließ das Fenster nicht. Nach 10 Minuten immer noch eingefroren? Starte den PC neu, schließ andere Programme und
versuch es noch einmal, oder nimm eine kleinere Größe.
- **Es hat beim Herunterladen oder Installieren aufgehört.** Starte `START-HERE.bat` (oder `./setup.sh`) noch
einmal. Es macht dort weiter, wo es aufgehört hat.
- **Es ist sehr langsam und die Festplattenlampe blinkt ständig, oder es meldet „the engine stopped
unexpectedly“.** Dein PC hat nicht genug freien RAM. Schließ andere Programme (Browser brauchen viel) oder nimm
eine kleinere Größe (Q2_0 oder IQ2_XS).
- **Es meldet, dass Port 8080 schon belegt ist.** Strata läuft bereits. Such nach seinem Fenster.
Weitere Probleme und ihre Lösungen: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Kommst du nicht weiter?
Öffne ein [Issue](https://github.com/Niko1221/Strata/issues) und hänge `strata-<model>.log` aus dem Strata-Ordner
an. Du hast ein Sicherheitsproblem gefunden? Melde es vertraulich: [SECURITY.md](SECURITY.md).
## Wie funktioniert es?
Modelle wie dieses laufen normalerweise auf Servern mit Hunderten Gigabyte Grafikspeicher. Deine Grafikkarte hat
12-24 GB. Strata bringt das Modell trotzdem unter, indem es **die Arbeit auf deinen ganzen PC verteilt**. Stell dir
eine Küche vor: Was du ständig brauchst, bleibt auf der Arbeitsfläche, der Rest wartet in der Vorratskammer.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Die 24,576 Experten des Modells: die meistgenutzten auf der Grafikkarte, alle im RAM, eine Nachschlagetabelle auf der SSD"></p>
- **Das Modell ist ein Team aus 24,576 kleinen Spezialisten („Experten“).** Jedes Wort braucht nur 10 davon.
- **Deine Grafikkarte** hält die paar tausend Experten, die am häufigsten gebraucht werden. **Dein RAM** hält alle,
und **dein Prozessor** rechnet gleichzeitig am Rest. **Deine SSD** hält eine große Nachschlagetabelle.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Ein kleiner Helfer rät die nächsten Wörter; das große Modell prüft alle auf einmal und behält die richtigen"></p>
- **Raten, dann prüfen:** Ein kleiner Helfer rät die nächsten paar Wörter. Das große Modell prüft sie alle auf
einmal. Du bekommst dieselbe Antwort, nur 1.6-1.8x schneller.
- **Lange Texte werden in großen Stücken gelesen** (bis zu 8,192 Tokens auf einmal), mit über 1,000 Tokens pro
Sekunde.
Die längere Erklärung: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Jeder Teil und seine Zahlen:
[die Details](docs/DETAILS.md#how-it-works) und das [Paper](docs/paper/Strata-Paper.pdf).
## Danksagung und Lizenz
Das Modell ist [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) vom Qwen-Team. Komprimiert
wurde es von [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
und Unsloth. Strata nutzt Teile von [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Alle Danksagungen:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata ist Open Source unter der [MIT-Lizenz](LICENSE). Ein
paar Teile und alle Modelle haben eigene Lizenzen ([welche](docs/HOW_IT_WORKS.md#license)).
## Strata unterstützen
Strata ist kostenlos und Open Source. Wenn es dir nützt, kannst du die Entwicklung unterstützen:
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><b>Ejecuta un modelo de IA de 125 mil millones de parámetros en tu propio PC gaming</b><br>
Tarjeta gráfica NVIDIA o AMD (12 GB o más) · Windows o Linux · libre y de código abierto</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Un jardín con una pagoda de vóxeles que escribió el modelo de Strata, funcionando en el navegador"></a><br>
<sub>Un jardín con una pagoda de vóxeles, hecho con un solo prompt en una RTX 5070 con Strata (IQ3_S, contexto de 128K) ·
NVIDIA: Q2_0 con el motor 0.1.36, las demás filas con 0.1.26 (respuestas de 4K, prompts de 32K). Las tablas
completas están en [DETAILS.md](docs/DETAILS.md#speed-measured). Una tarjeta con más VRAM es más rápida: una
RTX 3090 (24 GB) debería escribir unos 100-140 tokens por segundo. Chats largos y otras tarjetas:
[velocidad de cada modelo](docs/MODELS.md#how-fast-is-each-size), [resultados de la comunidad](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata es gratis. Si funciona bien en tu PC, un café ayuda a seguir trabajando en él.</sub></p>
## Qué necesitas
| | |
| --- | --- |
| **Tarjeta gráfica** | **NVIDIA** GeForce serie RTX 20, 30, 40 o 50, o **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 o serie RX 6800 / 6900. Necesita **12 GB de VRAM o más**. |
| **RAM** | 32 GB o más. Tu RAM decide [qué modelo](#qué-modelo-elijo) cabe. Con 64 GB funcionan todos los tamaños. |
| **Disco** | Unos 80 GB libres. Usa un SSD si puedes: el primer arranque es mucho más rápido. |
| **Sistema** | Windows 10 / 11 o Linux, y un controlador gráfico actual de NVIDIA o AMD. |
El instalador prepara todo lo demás. Dos o tres tarjetas pueden repartirse el modelo ([multi-GPU](docs/MULTI_GPU.md)).
Experimental, escrito y probado por miembros de la comunidad en sus propios equipos:
- **Intel Arc**, compilado desde el código fuente en Linux: [Intel Arc](docs/INTEL_ARC.md).
- **AMD Ryzen AI Max (Strix Halo)**, compilado desde el código fuente en Linux: [Strix Halo](docs/STRIX_HALO.md).
- **Procesadores antiguos sin AVX2**: funcionan, pero despacio. [Older CPUs](docs/INSTALL.md#older-cpus-experimental).
La lista completa: [docs/INSTALL.md](docs/INSTALL.md#what-you-need).
## Instalación
### Deja que tu IA lo instale
¿Usas un asistente de programación con IA (Claude Code, Cursor, Codex, GitHub Copilot, ...)? Pega esto en él:
```text
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
```
Revisa tu tarjeta gráfica, tu RAM y tu disco, y elige el modelo que cabe. Luego lo instala, lo arranca y te explica
cómo conectar tus aplicaciones. Las herramientas de IA también pueden instalar, arrancar y detener Strata con su
[servidor MCP](docs/MCP_SERVER.md).
### O hazlo tú mismo
[Descarga Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) y descomprímelo (o usa `git clone`).
**Windows:** haz doble clic en **`START-HERE.bat`**. **Linux:** ejecuta **`./setup.sh`** en la carpeta de Strata.
**Mac (Apple Silicon, M1 a M5, macOS 15 o posterior):** instala una vez las Command Line Tools de Apple (`xcode-select --install` en Terminal; no hace falta Xcode). Luego abre Terminal en la carpeta de Strata y ejecuta **`./setup.sh`**. Compila el motor de Mac en tu Mac (unos 7 minutos), hace las mismas preguntas, descarga el modelo y lo arranca. La próxima vez, `./setup.sh` arranca enseguida. Qué tamaño cabe en tu Mac y qué velocidad tiene: [docs/MAC.md](docs/MAC.md).
Los pasos son los mismos para NVIDIA y AMD. El instalador detecta tu tarjeta y prepara el motor adecuado para ella.
Te hace unas pocas preguntas:
- qué modelo y qué tamaño,
- cuánto contexto (cuánto texto tiene presente el modelo),
- si debe entender imágenes.
Pulsa Enter cada vez para usar la respuesta recomendada. Después descarga el modelo (unos 70 GB) y lo arranca. Si la
descarga se detiene, vuelve a ejecutarlo: sigue donde lo dejó. Tu navegador abre la aplicación de Strata en
`http://127.0.0.1:8080`.
> **Mientras el modelo arranca, tu PC puede ir lento o dejar de responder durante 1-3 minutos** (más la primera vez).
> Strata carga 35-55 GB en tu RAM y reserva una parte para la tarjeta gráfica. Es normal. Espera y no cierres la
> ventana. La ventana muestra lo que está haciendo Strata.
**La próxima vez**, vuelve a ejecutar `START-HERE.bat` (o `./setup.sh`). Arranca enseguida y no descarga nada dos
veces. Cierra su ventana para detener el modelo. `UPDATE.bat` (`./update.sh`) actualiza Strata sin arrancarlo.
Actualizaciones, Docker, varias tarjetas, dónde se guardan los archivos y todas las opciones: [docs/INSTALL.md](docs/INSTALL.md).
## ¿Qué modelo elijo?
El instalador te recomienda uno según tu RAM. El mismo modelo viene en varios tamaños, más o menos comprimidos. Los
tamaños pequeños son más rápidos. Los grandes son algo más inteligentes.
| Tu RAM | Elige | Por qué |
| --- | --- | --- |
| **32 GB** | **Coder** | cabe en 32 GB y está hecho para programar (con una tarjeta de 24 GB también funcionan Q2_0 e IQ2_XS) |
| **48 GB** | **IQ2_XS** (o Q2_0, el más rápido) | los tamaños más grandes no caben |
| **64 GB** | **IQ2_XS** (recomendado), o IQ3_XXS / IQ3_S | caben todos los tamaños; IQ3_S es el mejor y el más lento |
| **96 GB o más** | **IQ3_S**, o el UD-IQ4_XS de Unsloth (~4 bits) | espacio para los tamaños más grandes con todo lo demás abierto |
- **[Coder](docs/MODELS.md#coder):** una versión para programar a la que se le quitó la mitad de los expertos. Alcanza
el 91% de la puntuación SWE-bench Verified del modelo completo (medido por sus autores) y cabe en 32 GB de RAM. Es
más flojo fuera del código, también con chino y otros textos CJK (#438). Para eso, elige Q2_0, IQ2_XS o IQ3_S, que
conservan todos los expertos.
- **[Swift 1.5](docs/MODELS.md#swift-15):** un ajuste fino que piensa mucho menos tiempo antes de responder. Recibes
la respuesta antes, con más o menos la misma calidad.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** la versión de ~4 bits de Unsloth, entre IQ3_S y
UD-Q4_K_XL en calidad. Una descarga de 94 GB. Con menos de ~80 GB de RAM, Strata lee una parte desde el SSD
mientras responde, así que ahí es más lento (un SSD NVMe ayuda).
- **[Unsloth UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL y Q8_0](docs/MODELS.md#unsloths-files-ud-iq4_xs-to-q8_0):** de 4 a 8 bits, más cerca del modelo completo
(93-95 % el mismo token siguiente que Q8_0). Rápidos cuando todos los expertos caben en la RAM (~88 GB para
UD-Q4_K_XL hasta ~136 GB para Q8_0); con menos RAM, Strata lee una parte desde el SSD mientras responde, lo que es
mucho más lento (7-8.5 tokens/s para UD-Q4_K_XL en un PC con 64 GB).
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** se instala a mano. No está en
el menú del instalador.
Tamaños, descargas y qué cabe dónde: [docs/MODELS.md](docs/MODELS.md). Para añadir otro modelo más adelante, ejecuta
`SETUP.bat` (Linux: `./setup.sh --setup`).
## Cómo usarlo
<p align="center"><img src="docs/media/runpagoda.png" width="900" alt="La pestaña Monitor de la aplicación de Strata junto a un agente de programación"><br>
<sub>El <b>Monitor</b> de la aplicación de Strata (izquierda) mientras un agente de programación escribe el jardín de la pagoda del vídeo (derecha)</sub></p>
- **En el navegador:** abre `http://127.0.0.1:8080`. Tiene **Chat**, un **Monitor** en vivo del modelo y de tu
GPU/CPU/RAM, y **About** con los ajustes y las direcciones.
- **Tus aplicaciones y agentes de programación:** añade un proveedor "compatible con OpenAI" con la URL base
**`http://127.0.0.1:8080/v1`**. Sirve cualquier clave de API y cualquier nombre de modelo.
- Aplicaciones que usan la API de Anthropic: `http://127.0.0.1:8080/v1/messages` (Claude Code:
`ANTHROPIC_BASE_URL=http://127.0.0.1:8080`).
- Codex CLI y otras aplicaciones que usan la API Responses de OpenAI: `/v1/responses`
- **Razonamiento:** elige **off, low, medium o high** en el menú del chat o en el "reasoning effort" de tu aplicación.
Off es lo más rápido. High es lo mejor para preguntas difíciles.
- **Imágenes:** responde que sí a "Images?" en la instalación. Luego haz clic en **Picture** en el chat, o adjunta
imágenes en tu aplicación. Las tarjetas AMD leen imágenes en Linux a través del procesador; en Windows todavía no.
- **Desde tu móvil u otro PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Pon siempre una clave.
- **Una petición cada vez:** por defecto Strata responde una petición y las demás esperan. Para responder varias a la
vez, pon `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). En una tarjeta de 12 GB, cada respuesta va más lenta.
- **Prompts largos:** Strata lee entero el primer mensaje de un chat, más o menos 1 minuto por cada 30,000 tokens. Los
mensajes siguientes empiezan en segundos.
Más: [dónde se guardan tus chats](docs/INSTALL.md#where-things-are-stored), [la API](docs/DETAILS.md#using-it).
### Opcional: más velocidad con una sola tarjeta cuando todo el modelo cabe en la RAM
Dos ajustes opcionales (de RubenTorney, #1904 y #1905). Ayudan si usas una tarjeta gráfica y tu RAM guarda todos los expertos (el modo arena de Strata, por ejemplo Q2_0 o IQ3_XXS en un PC con 64 GB). Abre tu configuración `strata-<model>.json` en la carpeta de datos de Strata ([dónde](docs/INSTALL.md#where-things-are-stored)) y añade:
- **Generación más rápida:** `"--adapt-async", "1"` en la lista `"args"`. La caché de expertos se actualiza entonces en un hilo auxiliar en vez de pausar la respuesta. Medido en una RTX 5070 (Windows, Q2_0): +4-5 % de generación, +7-10 % en código y documentos largos. Las respuestas pueden diferir un poco de las por defecto (son iguales de una ejecución a otra).
- **Lectura del prompt más rápida en Windows:** `"STRATA_PIN_BY_NEED": "1"` en el bloque `"env"`. Windows solo puede bloquear cerca de la mitad de la RAM para la tarjeta; esto bloquea primero los expertos que la tarjeta necesita de verdad. Medido en la RTX 5070: +4,4 % en la lectura del prompt.
```json
"args":[...,"--adapt-async","1"],
"env":{"STRATA_PIN_BY_NEED":"1"}
```
Reinicia Strata (`START-HERE.bat` o `./setup.sh`) después del cambio. Para deshacerlo, quita las dos entradas.
## ¿Algo salió mal?
- **Mi PC se congeló la primera vez que arrancó Strata.** Es normal mientras carga el modelo. Espera y no cierres la
ventana. ¿Sigue congelado después de 10 minutos? Reinicia el PC, cierra otros programas y vuelve a intentarlo, o
elige un tamaño más pequeño.
- **Se detuvo durante la descarga o la instalación.** Vuelve a ejecutar `START-HERE.bat` (o `./setup.sh`). Sigue
donde se detuvo.
- **Va muy lento y la luz del disco no deja de parpadear, o dice "the engine stopped unexpectedly".** Tu PC no tiene
suficiente RAM libre. Cierra otros programas (los navegadores usan mucha), o elige un tamaño más pequeño (Q2_0 o
IQ2_XS).
- **Dice que el puerto 8080 ya está en uso.** Strata ya está en marcha. Busca su ventana.
Más problemas y sus soluciones: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). ¿Sigues atascado? Abre un
[issue](https://github.com/Niko1221/Strata/issues) y adjunta `strata-<model>.log` de la carpeta de Strata. ¿Has
encontrado un problema de seguridad? Infórmalo en privado: [SECURITY.md](SECURITY.md).
## ¿Cómo funciona?
Los modelos como este suelen funcionar en servidores con cientos de gigabytes de memoria gráfica. Tu tarjeta gráfica
tiene 12-24 GB. Strata hace que el modelo quepa **repartiendo el trabajo por todo tu PC**. Piensa en una cocina: lo
que usas todo el rato se queda en la encimera, y el resto espera en la despensa.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Los 24,576 expertos del modelo: los más usados en la tarjeta gráfica, todos en la RAM y una tabla de consulta en el SSD"></p>
- **El modelo es un equipo de 24,576 pequeños especialistas ("expertos").** Cada palabra solo necesita 10 de ellos.
- **Tu tarjeta gráfica** guarda los pocos miles de expertos que más se usan. **Tu RAM** los tiene todos, y **tu
procesador** trabaja con el resto al mismo tiempo. **Tu SSD** guarda una gran tabla de consulta.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Un pequeño ayudante adivina las siguientes palabras; el modelo grande las comprueba todas a la vez y se queda con las correctas"></p>
- **Adivinar y comprobar:** un pequeño ayudante adivina las siguientes palabras. El modelo grande las comprueba todas
a la vez. Recibes la misma respuesta, 1.6-1.8x antes.
- **Los textos largos se leen en trozos grandes** (hasta 8,192 tokens cada vez), a más de 1,000 tokens por segundo.
La explicación larga: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Cada parte y sus números:
[los detalles](docs/DETAILS.md#how-it-works) y el [artículo](docs/paper/Strata-Paper.pdf).
## Créditos y licencia
El modelo es [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), del equipo de Qwen. Lo
y Unsloth. Strata usa partes de [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Todos los créditos:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata es de código abierto con la [licencia MIT](LICENSE).
Algunas partes y cada modelo tienen sus propias licencias ([cuáles](docs/HOW_IT_WORKS.md#license)).
## Apoya a Strata
Strata es libre y de código abierto. Si te resulta útil, puedes apoyar su desarrollo:
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><b>Faites tourner un modèle d'IA de 125 milliards de paramètres sur votre propre PC de jeu</b><br>
Carte graphique NVIDIA ou AMD (12 Go ou plus) · Windows ou Linux · gratuit et open source</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Un jardin de pagode en voxels écrit par le modèle de Strata, qui tourne dans le navigateur"></a><br>
<sub>Un jardin de pagode en voxels, généré en une seule requête sur une RTX 5070 avec Strata (IQ3_S, contexte de 128K) ·
Strata fait tourner **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** sur un PC normal. C'est
un grand modèle d'IA, très capable, qui demande d'habitude un serveur. Il discute, écrit du code, lit des images et
fonctionne avec vos applications et vos agents de code. Rien ne quitte votre PC.
## Quelle est sa vitesse ?
Nous l'avons mesurée sur deux PC de jeu ordinaires. Un token correspond à environ ¾ d'un mot.
- **Écrit les réponses :** la vitesse à laquelle la réponse s'affiche dans une courte discussion. 60 tokens par seconde, c'est plus rapide que votre lecture.
- **Lit votre prompt :** la vitesse à laquelle il absorbe ce que vous envoyez (ici un document, du code ou un historique de discussion de 32K tokens).
<table>
<tr><th>NVIDIA : RTX 5070 (12 Go), Ryzen 5 7600, 64 Go de RAM</th><th>AMD : RX 9070 XT (16 Go), Ryzen 9 3900X, 47 Go de RAM</th></tr>
<tr><td>
| Taille | Écrit les réponses | Lit votre prompt |
| --- | ---: | ---: |
| **Q2_0** | 94 tokens/s | 2,650 tokens/s |
| **IQ2_XS** | 79 tokens/s | 2,090 tokens/s |
| **IQ3_XXS** | 62 tokens/s | 1,750 tokens/s |
| **IQ3_S** | 53 tokens/s | 1,620 tokens/s |
| **Coder** | 55 tokens/s | 2,180 tokens/s |
</td><td>
| Taille | Écrit les réponses | Lit votre prompt |
| --- | ---: | ---: |
| **Q2_0** | 60 tokens/s | 1,160 tokens/s |
| **IQ2_XS** | 52 tokens/s | 1,110 tokens/s |
| **Coder** | 44 tokens/s | 1,420 tokens/s |
</td></tr>
</table>
NVIDIA : Q2_0 avec le moteur 0.1.36, les autres lignes avec 0.1.26 (réponses de 4K, prompts de 32K). Les tableaux
complets sont dans [DETAILS.md](docs/DETAILS.md#speed-measured). Une carte avec plus de VRAM va plus vite : une
RTX 3090 (24 Go) devrait écrire environ 100-140 tokens par seconde. Longues discussions et autres cartes :
[vitesse de chaque modèle](docs/MODELS.md#how-fast-is-each-size), [résultats de la communauté](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata est gratuit. S'il tourne bien sur votre PC, un café aide à poursuivre le travail.</sub></p>
## Ce qu'il vous faut
| | |
| --- | --- |
| **Carte graphique** | **NVIDIA** GeForce RTX série 20, 30, 40 ou 50, ou **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 ou série RX 6800 / 6900. Il faut **12 Go de VRAM ou plus**. |
| **RAM** | 32 Go ou plus. Votre RAM décide [quel modèle](#quel-modèle-choisir-) tient. Avec 64 Go, toutes les tailles tournent. |
| **Disque** | Environ 80 Go libres. Prenez un SSD si possible : le premier démarrage est bien plus rapide. |
| **Système** | Windows 10 / 11 ou Linux, avec un pilote graphique NVIDIA ou AMD à jour. |
L'installateur s'occupe de tout le reste. Deux ou trois cartes peuvent se partager le modèle ([multi-GPU](docs/MULTI_GPU.md)).
Expérimental, écrit et testé par des membres de la communauté sur leurs propres machines :
- **Intel Arc**, compilé depuis les sources sous Linux : [Intel Arc](docs/INTEL_ARC.md).
- **AMD Ryzen AI Max (Strix Halo)**, compilé depuis les sources sous Linux : [Strix Halo](docs/STRIX_HALO.md).
- **Processeurs plus anciens sans AVX2** : ça marche, mais lentement. [Older CPUs](docs/INSTALL.md#older-cpus-experimental).
La liste complète : [docs/INSTALL.md](docs/INSTALL.md#what-you-need).
## Installation
### Laissez votre IA l'installer
Vous utilisez un assistant de code IA (Claude Code, Cursor, Codex, GitHub Copilot, ...) ? Collez-lui ceci :
```text
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
```
Il vérifie votre carte graphique, votre RAM et votre disque, et choisit le modèle qui convient. Ensuite il l'installe,
le démarre et vous explique comment connecter vos applications. Les outils d'IA peuvent aussi installer, démarrer et
arrêter Strata grâce à son [serveur MCP](docs/MCP_SERVER.md).
### Ou faites-le vous-même
[Téléchargez Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) et décompressez-le (ou faites un `git clone`).
**Windows :** double-cliquez sur **`START-HERE.bat`**. **Linux :** lancez **`./setup.sh`** dans le dossier Strata.
**Mac (Apple Silicon, M1 à M5, macOS 15 ou plus récent) :** installez une fois les Command Line Tools d'Apple (`xcode-select --install` dans le Terminal ; Xcode lui-même n'est pas nécessaire). Ouvrez ensuite le Terminal dans le dossier Strata et lancez **`./setup.sh`**. Il compile le moteur Mac sur votre Mac (environ 7 minutes), pose les mêmes questions, télécharge le modèle et le démarre. La fois suivante, `./setup.sh` démarre tout de suite. Quelle taille convient à votre Mac et sa vitesse : [docs/MAC.md](docs/MAC.md).
Les étapes sont les mêmes pour NVIDIA et AMD. L'installateur détecte votre carte et installe le bon moteur pour elle.
Il vous pose quelques questions :
- quel modèle et quelle taille,
- combien de contexte (la quantité de texte que le modèle garde en tête),
- s'il doit lire les images.
Appuyez sur Entrée à chaque fois pour garder la réponse recommandée. Ensuite il télécharge le modèle (environ 70 Go)
et le démarre. Si le téléchargement s'arrête, relancez-le : il reprend là où il s'était arrêté. Votre navigateur
ouvre l'application Strata à l'adresse `http://127.0.0.1:8080`.
> **Pendant le démarrage du modèle, votre PC peut ralentir ou ne plus répondre pendant 1-3 minutes** (plus longtemps la première fois).
> Strata charge 35-55 Go dans votre RAM et en réserve une partie pour la carte graphique. C'est normal. Attendez, et
> ne fermez pas la fenêtre. La fenêtre montre ce que fait Strata.
**La fois suivante**, relancez `START-HERE.bat` (ou `./setup.sh`). Il démarre tout de suite et ne télécharge rien
deux fois. Fermez sa fenêtre pour arrêter le modèle. `UPDATE.bat` (`./update.sh`) met Strata à jour sans le démarrer.
Mises à jour, Docker, plusieurs cartes, emplacement des fichiers et toutes les options : [docs/INSTALL.md](docs/INSTALL.md).
## Quel modèle choisir ?
L'installateur en recommande un selon votre RAM. Le même modèle existe en plusieurs tailles, plus ou moins
compressées. Les petites tailles sont plus rapides. Les grandes sont un peu plus intelligentes.
| Votre RAM | Prenez | Pourquoi |
| --- | --- | --- |
| **32 Go** | **Coder** | il tient dans 32 Go et il est fait pour le code (avec une carte de 24 Go, Q2_0 et IQ2_XS tournent aussi) |
| **48 Go** | **IQ2_XS** (ou Q2_0, le plus rapide) | les plus grandes tailles ne tiennent pas |
| **64 Go** | **IQ2_XS** (recommandé), ou IQ3_XXS / IQ3_S | toutes les tailles tiennent ; IQ3_S est le meilleur et le plus lent |
| **96 Go ou plus** | **IQ3_S**, ou l'UD-IQ4_XS d'Unsloth (~4 bits) | de la place pour les plus grandes tailles, avec tout le reste ouvert |
- **[Coder](docs/MODELS.md#coder) :** une version pour le code, avec la moitié des experts en moins. Elle atteint 91 %
du score SWE-bench Verified du modèle complet (mesuré par ses auteurs) et tient dans 32 Go de RAM. Elle est moins
bonne en dehors du code, y compris pour le chinois et les autres textes CJK (#438). Pour ces cas, prenez Q2_0,
IQ2_XS ou IQ3_S, qui gardent tous les experts.
- **[Swift 1.5](docs/MODELS.md#swift-15) :** un fine-tune qui réfléchit bien moins longtemps avant de répondre. Vous
avez la réponse plus tôt, avec à peu près la même qualité.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs)** : la version ~4 bits d'Unsloth, entre IQ3_S et
UD-Q4_K_XL en qualité. Un téléchargement de 94 Go. Avec moins de ~80 Go de RAM, Strata en lit une partie depuis
le SSD pendant qu'il répond, il y est donc plus lent (un SSD NVMe aide).
- **[Unsloth UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL et Q8_0](docs/MODELS.md#unsloths-files-ud-iq4_xs-to-q8_0) :** de 4 à 8 bits, plus proches du modèle
complet (93-95 % le même token suivant que Q8_0). Rapides quand tous les experts tiennent dans la RAM (~88 Go pour
UD-Q4_K_XL jusqu'à ~136 Go pour Q8_0) ; avec moins de RAM, Strata en lit une partie depuis le SSD pendant qu'il
répond, ce qui est bien plus lent (7-8.5 tokens/s pour UD-Q4_K_XL sur un PC avec 64 Go).
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs) :** vous l'installez à la main.
Il n'est pas dans le menu de l'installateur.
Tailles, téléchargements et ce qui tient où : [docs/MODELS.md](docs/MODELS.md). Pour ajouter un autre modèle plus
- **Réflexion :** choisissez **off, low, medium ou high** dans le menu du chat ou dans le « reasoning effort » de
votre application. Off est le plus rapide. High est le meilleur pour les questions difficiles.
- **Images :** répondez oui à « Images? » pendant l'installation. Ensuite cliquez sur **Picture** dans le chat, ou
joignez des images dans votre application. Les cartes AMD lisent les images sous Linux via le processeur ; sous
Windows, pas encore.
- **Depuis votre téléphone ou un autre PC :** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Définissez
toujours une clé.
- **Une requête à la fois :** par défaut, Strata répond à une requête et les autres attendent. Pour répondre à
plusieurs en même temps, mettez `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). Sur une carte de 12 Go, chaque
réponse devient alors plus lente.
- **Longs prompts :** Strata lit en entier le premier message d'une discussion, environ 1 minute pour 30,000 tokens.
Les messages suivants démarrent en quelques secondes.
Plus d'infos : [où sont stockées vos discussions](docs/INSTALL.md#where-things-are-stored), [l'API](docs/DETAILS.md#using-it).
### Optionnel : plus de vitesse sur une seule carte quand tout le modèle tient en RAM
Deux réglages optionnels (de RubenTorney, #1904 et #1905). Ils aident si vous utilisez une seule carte graphique et que votre RAM contient tous les experts (le mode arena de Strata, par exemple Q2_0 ou IQ3_XXS sur un PC de 64 Go). Ouvrez votre configuration `strata-<model>.json` dans le dossier de données de Strata ([où](docs/INSTALL.md#where-things-are-stored)) et ajoutez :
- **Génération plus rapide :** `"--adapt-async", "1"` dans la liste `"args"`. Le cache d'experts est alors mis à jour dans un thread auxiliaire au lieu de mettre la réponse en pause. Mesuré sur une RTX 5070 (Windows, Q2_0) : +4-5 % de génération, +7-10 % sur le code et les longs documents. Les réponses peuvent légèrement différer de celles par défaut (elles restent identiques d'une exécution à l'autre).
- **Lecture du prompt plus rapide sous Windows :** `"STRATA_PIN_BY_NEED": "1"` dans le bloc `"env"`. Windows ne peut verrouiller qu'environ la moitié de la RAM pour la carte ; ce réglage verrouille d'abord les experts dont la carte a vraiment besoin. Mesuré sur la RTX 5070 : +4,4 % en lecture du prompt.
```json
"args":[...,"--adapt-async","1"],
"env":{"STRATA_PIN_BY_NEED":"1"}
```
Redémarrez Strata (`START-HERE.bat` ou `./setup.sh`) après la modification. Pour annuler, retirez les deux entrées.
## Un problème ?
- **Mon PC s'est figé au premier démarrage de Strata.** C'est normal pendant le chargement du modèle. Attendez, et ne
fermez pas la fenêtre. Toujours figé après 10 minutes ? Redémarrez le PC, fermez les autres programmes et
réessayez, ou choisissez une taille plus petite.
- **Il s'est arrêté pendant le téléchargement ou l'installation.** Relancez `START-HERE.bat` (ou `./setup.sh`). Il
reprend là où il s'était arrêté.
- **C'est très lent et le voyant du disque clignote sans arrêt, ou il affiche « the engine stopped unexpectedly ».**
Votre PC n'a pas assez de RAM libre. Fermez les autres programmes (les navigateurs en utilisent beaucoup), ou
choisissez une taille plus petite (Q2_0 ou IQ2_XS).
- **Il dit que le port 8080 est déjà utilisé.** Strata tourne déjà. Cherchez sa fenêtre.
Autres problèmes et leurs solutions : [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Toujours bloqué ? Ouvrez une
[issue](https://github.com/Niko1221/Strata/issues) et joignez `strata-<model>.log` depuis le dossier Strata. Vous
avez trouvé un problème de sécurité ? Signalez-le en privé : [SECURITY.md](SECURITY.md).
## Comment ça marche ?
Les modèles comme celui-ci tournent d'habitude sur des serveurs avec des centaines de gigaoctets de mémoire
graphique. Votre carte graphique a 12-24 Go. Strata fait tenir le modèle en **répartissant le travail sur tout votre
PC**. Pensez à une cuisine : ce que vous utilisez tout le temps reste sur le plan de travail, et le reste attend dans
le placard.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Les 24,576 experts du modèle : les plus sollicités sur la carte graphique, tous en RAM, une table de correspondance sur le SSD"></p>
- **Le modèle est une équipe de 24,576 petits spécialistes (les « experts »).** Chaque mot n'en demande que 10.
- **Votre carte graphique** garde les quelques milliers d'experts les plus utilisés. **Votre RAM** les contient tous,
et **votre processeur** travaille sur les autres en même temps. **Votre SSD** contient une grande table de
correspondance.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Un petit assistant devine les mots suivants ; le grand modèle les vérifie tous d'un coup et garde les bons"></p>
- **Deviner, puis vérifier :** un petit assistant devine les quelques mots suivants. Le grand modèle les vérifie
tous d'un coup. Vous obtenez la même réponse, 1.6-1.8x plus tôt.
- **Les longs textes sont lus par gros morceaux** (jusqu'à 8,192 tokens à la fois), à plus de 1,000 tokens par seconde.
L'explication plus longue : [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Chaque partie et ses chiffres :
[les détails](docs/DETAILS.md#how-it-works) et l'[article](docs/paper/Strata-Paper.pdf).
## Crédits et licence
Le modèle est [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), de l'équipe Qwen. Il a été
compressé par [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
et Unsloth. Strata utilise des parties de [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Tous les
crédits : [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata est open source sous [licence MIT](LICENSE).
Quelques parties et chaque modèle ont leur propre licence ([lesquelles](docs/HOW_IT_WORKS.md#license)).
## Soutenir Strata
Strata est gratuit et open source. S'il vous est utile, vous pouvez soutenir son développement :
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata は無料で使えます。お使いの PC で快適に動いたら、コーヒー 1 杯分のご支援が開発を続ける励みになります。</sub></p>
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><b>Run a 125-billion-parameter AI model on your own gaming PC</b><br>
NVIDIA or AMD graphics card (12 GB or more) · Windows or Linux · free and open source</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="A voxel pagoda garden that Strata's model wrote, running in the browser"></a><br>
<sub>A voxel pagoda garden, 1 shot prompt running on an RTX 5070 with Strata (IQ3_S, 128K context) ·
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">full video (49 s)</a></sub></p>
Strata runs **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** - a large, smart AI model that
normally needs a server - on your own PC. It writes its answers at **60-95 tokens per second** (a token is about ¾
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata is free. If it runs well on your PC, a coffee keeps the work on it going.</sub></p>
- **Intel Arc**, built from source on Linux: [Intel Arc](docs/INTEL_ARC.md).
- **Apple Silicon Macs (M1 to M5)**, built from source by `./setup.sh` (the Metal engine): [Mac](docs/MAC.md).
- **AMD Ryzen AI Max (Strix Halo)**, built from source on Linux: [Strix Halo](docs/STRIX_HALO.md).
- **Older processors without AVX2**: they work, but slowly. [Older CPUs](docs/INSTALL.md#older-cpus-experimental).
The full list: [docs/INSTALL.md](docs/INSTALL.md#what-you-need).
## Install
**You need:** an NVIDIA RTX 30, 40 or 50 card with 12 GB of VRAM or more, enough RAM for the size you pick (above),
~80 GB of free disk space (an SSD makes the first start much faster), and Windows 10/11 or Linux. The only thing you
install yourself is a current **NVIDIA driver** ([nvidia.com/drivers](https://www.nvidia.com/drivers) or the NVIDIA
App). Everything else - Python, the engine, the model - is set up for you.
### Let your AI set it up
**Windows**
Do you use an AI coding assistant (Claude Code, Cursor, Codex, GitHub Copilot, ...)? Paste this into it:
1. [Download this project](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or `git clone` it).
2. Double-click **`START-HERE.bat`**.
3. Answer a few questions - or just press Enter each time for the recommended choice:
- **Which model and size?** The original or Swift 1.5, and Q2_0, IQ2_XS, IQ3_XXS or IQ3_S - see [above](#which-model-should-i-pick)
- **How much context?** How much text it can keep in mind at once (it suggests one for your card)
- **Images?** Whether it should also read pictures
- **Experimental speed projection?** Off unless you say yes - [read what it does](docs/DETAILS.md#experimental-speed-projection-experimental-off-by-default) first
```text
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
```
Then it downloads everything (the model is ~70 GB, so the first time takes a while - you can stop and it picks up
where it left off) and **starts the model**. Your browser opens the Strata app at `http://127.0.0.1:8080`.
It checks your graphics card, RAM and disk and picks the model that fits. Then it installs and starts it and tells
you how to connect your apps. AI tools can also install, start and stop Strata through its
[MCP server](docs/MCP_SERVER.md).
> **While the model starts, your PC can be slow or stop responding for 1-3 minutes** (longest the first time): Strata
> loads 35-55 GB into your RAM and locks part of it for the graphics card. That's normal - wait, and don't close the
> window. The window tells you what it is doing.
### Or do it yourself
**Next time**, just double-click `START-HERE.bat` again: it starts right away, nothing is downloaded twice. Close its
window to stop the model.
[Download Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or `git clone` it).
**Windows:** double-click **`START-HERE.bat`**. **Linux:** run **`./setup.sh`** in the Strata folder.
**Updating:** download the new version and unzip it anywhere (or `git pull`), then run `START-HERE.bat` in it. The
model files are kept in a `Strata-data` folder next to your Strata folder, so a new copy finds them and sets itself up
the same way - nothing big is downloaded again.
**Mac (Apple Silicon, M1 to M5, macOS 15 or newer):** install Apple's Command Line Tools once
(`xcode-select --install` in Terminal; Xcode itself is not needed). Then open Terminal in the Strata folder and run
**`./setup.sh`**. It builds the Mac engine on your Mac (about 7 minutes), asks the same questions, downloads the model
and starts it. Next time, `./setup.sh` starts it right away. Which size fits your Mac and how fast it is:
[docs/MAC.md](docs/MAC.md).
**Linux:** run `./setup.sh` - same questions, same result.
The steps are the same for NVIDIA and AMD. The installer finds your card and sets up the right engine for it. It
asks you a few questions:
- which model and which size,
- how much context (how much text the model keeps in mind),
- whether it should read pictures.
Press Enter each time for the recommended answer. Then it downloads the model (about 70 GB) and starts it. If the
download stops, run it again: it continues where it left off. Your browser opens the Strata app at
`http://127.0.0.1:8080`.
> **While the model starts, your PC can be slow or stop responding for 1-3 minutes** (longest the first time).
> Strata loads 35-55 GB into your RAM and locks part of it for the graphics card. This is normal. Wait, and don't
> close the window. The window shows what Strata is doing.
**Next time**, run `START-HERE.bat` (or `./setup.sh`) again. It starts right away and downloads nothing twice. Close
its window to stop the model. `UPDATE.bat` (`./update.sh`) updates Strata without starting it. Updating, Docker,
several cards, where the files go and every option: [docs/INSTALL.md](docs/INSTALL.md).
**Is it fast on your PC?** Run `BENCH.bat` (`./bench.sh`): about 5-10 minutes, it measures chat, 4K/16K prompts and an agent-style session with your own settings and writes a report to share ([how](docs/COMMUNITY_BENCHMARKS.md#run-the-benchmark)).
## Which model should I pick?
The installer recommends one for your RAM. The same model comes in several sizes, compressed more or less. Smaller
sizes are faster. Larger sizes are a bit smarter.
| Your RAM | Take | Why |
| --- | --- | --- |
| **32 GB** | **Coder** | it fits 32 GB, and it is made for code (with a 24 GB card, Q2_0 and IQ2_XS run too) |
| **48 GB** | **IQ2_XS** (or Q2_0, the fastest) | the larger sizes do not fit |
| **64 GB** | **IQ2_XS** (recommended), or IQ3_XXS / IQ3_S | every size fits; IQ3_S is the best and the slowest |
| **96 GB or more** | **IQ3_S**, or Unsloth's UD-IQ4_XS (~4-bit) | room for the largest sizes with everything else open |
- **[Coder](docs/MODELS.md#coder):** a coding version with half of the experts removed. It reaches 91% of the full
model's SWE-bench Verified score (measured by its authors) and fits 32 GB of RAM. It is weaker outside code,
including Chinese and other CJK text (#438). For those, take Q2_0, IQ2_XS or IQ3_S, which keep every expert.
- **[Swift 1.5](docs/MODELS.md#swift-15):** a fine-tune that thinks for a much shorter time before it answers. You
get the answer sooner, at about the same quality.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** Unsloth's ~4-bit version, between IQ3_S and
UD-Q4_K_XL in quality. A 94 GB download. With less than ~80 GB of RAM, Strata reads part of it from the SSD
while it answers, so it is slower there (an NVMe SSD helps).
- **[Unsloth UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL and Q8_0](docs/MODELS.md#unsloths-files-ud-iq4_xs-to-q8_0):**
4- to 8-bit, closer to the full model (93-95% the same next token as Q8_0). Fast where every expert fits in RAM
(~88 GB for UD-Q4_K_XL up to ~136 GB for Q8_0); with less RAM Strata reads part of them from the SSD while it
answers, which is much slower (7-8.5 tokens/s for UD-Q4_K_XL on a 64 GB PC).
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** you set it up by hand. It is
not in the installer's menu.
Sizes, downloads and what fits where: [docs/MODELS.md](docs/MODELS.md). To add another model later, run
`SETUP.bat` (Linux: `./setup.sh --setup`).
## Using it
<p align="center"><img src="docs/media/runpagoda.png" width="900" alt="The Strata app's Monitor tab next to a coding agent"><br>
<sub>The Strata app's <b>Monitor</b> (left) while a coding agent writes the pagoda garden from the video (right)</sub></p>
- **In the browser:** `http://127.0.0.1:8080` - the Strata app (it opens by itself when the model starts): **Chat**, a
live **Monitor** of the model and your GPU/CPU/RAM, and **About** with the settings and addresses.
- **Chat in the terminal:** `.venv\Scripts\python chat.py`
- **Your apps and coding agents:** add it as an "OpenAI-compatible" provider with base URL
**`http://127.0.0.1:8080/v1`**, any API key and any model name. Apps that use Anthropic's API: `http://127.0.0.1:8080/v1/messages`.
- **Thinking:** the model thinks before it answers. Choose **off, low, medium or high** - in the chat page menu, with
`/think low` in `chat.py`, or with your app's "reasoning effort" setting. Off is fastest; high is best for hard questions.
- **Pictures:** in the chat page click **Picture**; in `chat.py` type `/image <path>`; in apps just attach them.
- **From your phone or another PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`, then open the
address the server window prints; see the [details](docs/DETAILS.md#using-it).
- **Experimental speed projection (off by default):** an experimental control vector that setup can turn on; it
changes how the model answers - read [what it does](docs/DETAILS.md#experimental-speed-projection-experimental-off-by-default) first.
- **In the browser:** open`http://127.0.0.1:8080`. It has **Chat**, a live **Monitor** of the model and your
GPU/CPU/RAM, and **About** with the settings and addresses.
- **Your apps and coding agents:** add an "OpenAI-compatible" provider with the base URL
**`http://127.0.0.1:8080/v1`**. Any API key and any model name work.
- Apps that use Anthropic's API: `http://127.0.0.1:8080/v1/messages` (Claude Code:
`ANTHROPIC_BASE_URL=http://127.0.0.1:8080`).
- Codex CLI and other apps that use the OpenAI Responses API: `/v1/responses`
- **Thinking:** choose **off, low, medium or high** in the chat menu or in your app's "reasoning effort". Off is the
fastest. High is best for hard questions.
- **Pictures:** say yes to "Images?" in setup. Then click **Picture** in the chat, or attach pictures in your app.
AMD cards read pictures on Linux through the processor; on Windows they can't yet.
- **From your phone or another PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Always set a key.
- **One request at a time:** by default Strata answers one request, and the others wait. To answer several at once,
set `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). On a 12 GB card this makes each answer slower.
- **Long prompts:** Strata reads the first message of a chat in full, about 1 minute per 30,000 tokens. Follow-up
messages start in seconds.
**Good to know:** it answers one request at a time. The first message of a chat is read in full (about 1 minute per
30,000 tokens); after that it keeps the conversation and reads only what is new, so follow-ups start in seconds.
More: [where your chats are stored](docs/INSTALL.md#where-things-are-stored), [the API](docs/DETAILS.md#using-it).
### Optional: more speed on one card when the whole model fits in RAM
Two opt-in settings (by RubenTorney, #1904 and #1905). They help when you run one graphics card and your RAM holds
every expert (Strata's arena mode, for example Q2_0 or IQ3_XXS on a 64 GB PC). Open your config,
`strata-<model>.json` in the Strata data folder ([where](docs/INSTALL.md#where-things-are-stored)), and add:
- **Faster decode:** `"--adapt-async", "1"` to the `"args"` list. The expert cache then updates on a helper thread
instead of pausing the answer. Measured on an RTX 5070 (Windows, Q2_0): +4-5% decode, +7-10% on code and long
documents. Answers can differ slightly from the default (they stay the same from run to run).
- **Faster prompt reading on Windows:** `"STRATA_PIN_BY_NEED": "1"` in the `"env"` block. Windows can lock only about
half the RAM for the graphics card; this locks the experts the card actually needs first. Measured on the RTX 5070:
+4.4% prompt reading.
```json
"args":[...,"--adapt-async","1"],
"env":{"STRATA_PIN_BY_NEED":"1"}
```
Restart Strata (`START-HERE.bat` or `./setup.sh`) after the change. To undo, remove the two entries.
## Something went wrong?
**My PC froze, or got very slow, the first time Strata started.**
That's normal while it starts, most of all the first time. Strata loads 35-55 GB into your RAM, locks part of it for
the graphics card, and works out how much of the model fits on your GPU. The mouse can freeze for a few minutes. **Wait, and don't close the
window.** The next starts are much faster. Still frozen after 10 minutes? Restart the PC, close other programs
(browsers use a lot of RAM) and try again. If it keeps happening, pick a smaller size (Q2_0 or IQ2_XS).
-**My PC froze the first time Strata started.** This is normal while it loads the model. Wait, and don't close the
window. Still frozen after 10 minutes? Restart the PC, close other programs and try again, or pick a smaller size.
- **It stopped while downloading or installing.** Run `START-HERE.bat` (or `./setup.sh`) again. It continues where
it stopped.
- **It's very slow and the disk light keeps blinking, or it says "the engine stopped unexpectedly".** Your PC does
not have enough free RAM. Close other programs (browsers use a lot), or pick a smaller size (Q2_0 or IQ2_XS).
- **It says port 8080 is already in use.** Strata is already running. Look for its window.
**It stopped while downloading or installing.**
Run `START-HERE.bat` again. It continues where it stopped.
**It says the NVIDIA driver is too old.**
Update it (NVIDIA App or [nvidia.com/drivers](https://www.nvidia.com/drivers)), restart the PC, and run
`START-HERE.bat` again.
**It says port 8080 is already in use.**
Strata is already running. Look for its window.
**It's very slow and the disk light keeps blinking.**
Your PC is out of free RAM. Close other programs, or pick a smaller size (Q2_0 or IQ2_XS).
**An answer stopped with "the engine stopped unexpectedly".**
Usually not enough RAM (on Linux the system then stops the engine). Just send your message again: Strata starts the
engine by itself. If it keeps happening, close other programs or pick a smaller size.
**It says the prompt exceeds the context.**
The conversation is longer than the context you chose. Start a new chat, or run `START-HERE.bat --setup` and pick more
context.
**Still stuck?** Look in the [full troubleshooting table](docs/DETAILS.md#troubleshooting), or open an issue and
attach `strata-<model>.log` from the Strata folder.
More problems and their fixes: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Still stuck? Open an
[issue](https://github.com/Niko1221/Strata/issues) and attach `strata-<model>.log` from the Strata folder. Found a
security problem? Report it privately: [SECURITY.md](SECURITY.md).
## How does it work?
A model this big doesn't fit on a gaming graphics card. Strata splits the work between the parts of your PC:
Models like this one usually run on servers with hundreds of gigabytes of graphics memory. Your graphics card has
12-24 GB. Strata makes the model fit by **sharing the work across your whole PC**. Think of a kitchen: the things
you use all the time stay on the counter, and the rest waits in the pantry.
<p align="center"><img src="docs/paper/tiers.svg" width="700" alt="how Strata splits the model between GPU, RAM and SSD"></p>
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="The model's 24,576 experts: the busiest on the graphics card, all of them in RAM, a lookup table on the SSD"></p>
- **The GPU** runs the part of the model that is used for every word, plus the "experts" it needs most often.
- **The RAM** holds all 24,576 experts, and **the CPU** computes the few the GPU doesn't have - at the same time as the GPU.
- **The SSD** holds a big lookup table; the model reads a few rows of it per word.
- **A small helper inside the model guesses the next words**, and Strata checks several guesses at once. That makes
it 1.6-1.8x faster than going word by word - and the answer is exactly the same.
- **The model is a team of 24,576 small specialists ("experts").** Each word needs only 10 of them.
- **Your graphics card** keeps the few thousand experts that are used most often. **Your RAM** holds all of them,
and **your processor** works on the rest at the same time. **Your SSD** holds a big lookup table.
The full story is in the [paper](docs/paper/Strata-Paper.pdf) and the [details](docs/DETAILS.md).
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="A small helper guesses the next words; the big model checks them all at once and keeps the right ones"></p>
## Credits
- **Guess, then check:** a small helper guesses the next few words. The big model checks them all at once. You get
the same answer, 1.6-1.8x sooner.
- **Long texts are read in big pieces** (up to 8,192 tokens at a time), at over 1,000 tokens per second.
- Model: [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team; compressed versions by
[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF) by UkisAI. Their licenses apply
to the model files.
- Built with parts of [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp) (MIT). Ideas from
[Splash](https://github.com/incoai/splash), [ninfer](https://github.com/Neroued/ninfer) and
[HyperQwen](https://github.com/syv-ai/HyperQwen). More in the [details](docs/DETAILS.md#credits-and-licenses).
The longer explanation: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Every part and its numbers:
[the details](docs/DETAILS.md#how-it-works) and the [paper](docs/paper/Strata-Paper.pdf).
## License
## Credits and license
Strata is open source under the [MIT License](LICENSE). A few parts carry their own licenses: `third_party/ggml`
(MIT, llama.cpp / ggml), the web app's font (SIL Open Font License 1.1) and the experimental speed projection's
vector in `data/experimental-speed-projection` (Qwen Community License 1.0, from the model's activations). The
models are not part of this repository; each model's own license applies to its files.
The model is [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team. It was
compressed by [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
and Unsloth. Strata uses parts of [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). All credits:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata is open source under the [MIT License](LICENSE). A few
parts and every model have their own licenses ([which ones](docs/HOW_IT_WORKS.md#license)).
## Support Strata
Strata is free and open source. If it is useful to you, you can support its development:
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><b>Rode um modelo de IA de 125 bilhões de parâmetros no seu próprio PC gamer</b><br>
Placa de vídeo NVIDIA ou AMD (12 GB ou mais) · Windows ou Linux · gratuito e de código aberto</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Um jardim com pagode em voxels que o modelo do Strata escreveu, rodando no navegador"></a><br>
<sub>Um jardim com pagode em voxels, feito com um único prompt numa RTX 5070 com o Strata (IQ3_S, contexto de 128K) ·
NVIDIA: Q2_0 com o engine 0.1.36, as outras linhas com 0.1.26 (respostas de 4K, prompts de 32K). As tabelas
completas estão em [DETAILS.md](docs/DETAILS.md#speed-measured). Uma placa com mais VRAM é mais rápida: uma
RTX 3090 (24 GB) deve escrever cerca de 100-140 tokens por segundo. Conversas longas e outras placas:
[velocidade de cada modelo](docs/MODELS.md#how-fast-is-each-size), [resultados da comunidade](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>O Strata é gratuito. Se ele roda bem no seu PC, um café ajuda a manter o trabalho nele.</sub></p>
## Do que você precisa
| | |
| --- | --- |
| **Placa de vídeo** | **NVIDIA** GeForce RTX séries 20, 30, 40 ou 50, ou **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 ou séries RX 6800 / 6900. Ela precisa de **12 GB de VRAM ou mais**. |
| **RAM** | 32 GB ou mais. Sua RAM decide [qual modelo](#qual-modelo-devo-escolher) cabe. Com 64 GB, roda qualquer tamanho. |
| **Disco** | Cerca de 80 GB livres. Use um SSD se puder: a primeira inicialização fica bem mais rápida. |
| **Sistema** | Windows 10 / 11 ou Linux, e um driver de vídeo atualizado da NVIDIA ou da AMD. |
O instalador cuida de todo o resto. Duas ou três placas podem dividir o modelo ([multi-GPU](docs/MULTI_GPU.md)).
Experimental, escrito e testado por membros da comunidade nas próprias máquinas:
- **Placas de vídeo mais antigas** (Tesla P40 / V100, GTX 10, Radeon VII / MI50, RX 6700 XT, RX 5500 XT): [Older GPUs](docs/OLDER_GPUS.md).
- **Intel Arc**, compilado a partir do código-fonte no Linux: [Intel Arc](docs/INTEL_ARC.md).
- **AMD Ryzen AI Max (Strix Halo)**, compilado a partir do código-fonte no Linux: [Strix Halo](docs/STRIX_HALO.md).
- **Processadores mais antigos sem AVX2**: funcionam, mas devagar. [Older CPUs](docs/INSTALL.md#older-cpus-experimental).
A lista completa: [docs/INSTALL.md](docs/INSTALL.md#what-you-need).
## Instalação
### Deixe sua IA instalar
Você usa um assistente de programação com IA (Claude Code, Cursor, Codex, GitHub Copilot, ...)? Cole isto nele:
```text
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
```
Ele verifica sua placa de vídeo, RAM e disco e escolhe o modelo que cabe. Depois instala, inicia e explica como
conectar seus apps. Ferramentas de IA também podem instalar, iniciar e parar o Strata pelo
[servidor MCP](docs/MCP_SERVER.md) dele.
### Ou faça você mesmo
[Baixe o Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) e descompacte (ou use `git clone`).
**Windows:** clique duas vezes em **`START-HERE.bat`**. **Linux:** rode **`./setup.sh`** na pasta do Strata.
**Mac (Apple Silicon, M1 a M5, macOS 15 ou mais recente):** instale uma vez as Command Line Tools da Apple (`xcode-select --install` no Terminal; o Xcode em si não é necessário). Depois abra o Terminal na pasta do Strata e rode **`./setup.sh`**. Ele compila o motor do Mac no seu Mac (uns 7 minutos), faz as mesmas perguntas, baixa o modelo e o inicia. Da próxima vez, `./setup.sh` inicia na hora. Qual tamanho cabe no seu Mac e qual a velocidade: [docs/MAC.md](docs/MAC.md).
Os passos são os mesmos para NVIDIA e AMD. O instalador encontra sua placa e configura o engine certo para ela.
Ele faz algumas perguntas:
- qual modelo e qual tamanho,
- quanto contexto (quanto texto o modelo guarda na memória),
- se ele deve entender imagens.
Aperte Enter em cada uma para usar a resposta recomendada. Depois ele baixa o modelo (cerca de 70 GB) e o inicia.
Se o download parar, rode de novo: ele continua de onde parou. Seu navegador abre o app do Strata em
`http://127.0.0.1:8080`.
> **Enquanto o modelo inicia, seu PC pode ficar lento ou parar de responder por 1-3 minutos** (mais na primeira vez).
> O Strata carrega 35-55 GB na sua RAM e reserva parte dela para a placa de vídeo. Isso é normal. Espere e não
> feche a janela. A janela mostra o que o Strata está fazendo.
**Da próxima vez**, rode `START-HERE.bat` (ou `./setup.sh`) de novo. Ele inicia na hora e não baixa nada duas
vezes. Feche a janela para parar o modelo. `UPDATE.bat` (`./update.sh`) atualiza o Strata sem iniciá-lo.
Atualização, Docker, várias placas, onde ficam os arquivos e todas as opções: [docs/INSTALL.md](docs/INSTALL.md).
## Qual modelo devo escolher?
O instalador recomenda um de acordo com sua RAM. O mesmo modelo vem em vários tamanhos, mais ou menos comprimidos.
Os tamanhos menores são mais rápidos. Os maiores são um pouco mais inteligentes.
| Sua RAM | Escolha | Por quê |
| --- | --- | --- |
| **32 GB** | **Coder** | cabe em 32 GB e é feito para código (com uma placa de 24 GB, Q2_0 e IQ2_XS também rodam) |
| **48 GB** | **IQ2_XS** (ou Q2_0, o mais rápido) | os tamanhos maiores não cabem |
| **64 GB** | **IQ2_XS** (recomendado), ou IQ3_XXS / IQ3_S | todos os tamanhos cabem; IQ3_S é o melhor e o mais lento |
| **96 GB ou mais** | **IQ3_S**, ou o UD-IQ4_XS da Unsloth (~4-bit) | espaço para os maiores tamanhos com todo o resto aberto |
- **[Coder](docs/MODELS.md#coder):** uma versão para programação com metade dos experts removida. Ela alcança 91%
da pontuação do modelo completo no SWE-bench Verified (medido pelos autores) e cabe em 32 GB de RAM. É mais fraca
fora do código, inclusive em chinês e outros textos CJK (#438). Para isso, escolha Q2_0, IQ2_XS ou IQ3_S, que
mantêm todos os experts.
- **[Swift 1.5](docs/MODELS.md#swift-15):** um fine-tune que pensa por bem menos tempo antes de responder. Você
recebe a resposta mais cedo, com quase a mesma qualidade.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** a versão de ~4 bits da Unsloth, entre IQ3_S e
UD-Q4_K_XL em qualidade. Um download de 94 GB. Com menos de ~80 GB de RAM, o Strata lê uma parte dele do SSD
enquanto responde, então ali ele fica mais lento (um SSD NVMe ajuda).
- **[Unsloth UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL e Q8_0](docs/MODELS.md#unsloths-files-ud-iq4_xs-to-q8_0):** de 4 a 8 bits, mais próximos do modelo
completo (93-95% o mesmo próximo token que o Q8_0). Rápidos quando todos os especialistas cabem na RAM (~88 GB para
o UD-Q4_K_XL até ~136 GB para o Q8_0); com menos RAM, o Strata lê uma parte do SSD enquanto responde, o que é bem
mais lento (7-8.5 tokens/s para o UD-Q4_K_XL num PC com 64 GB).
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** você configura à mão. Ele
não está no menu do instalador.
Tamanhos, downloads e o que cabe onde: [docs/MODELS.md](docs/MODELS.md). Para adicionar outro modelo depois, rode
`SETUP.bat` (Linux: `./setup.sh --setup`).
## Como usar
<p align="center"><img src="docs/media/runpagoda.png" width="900" alt="A aba Monitor do app Strata ao lado de um agente de programação"><br>
<sub>O <b>Monitor</b> do app Strata (à esquerda) enquanto um agente de programação escreve o jardim com pagode do vídeo (à direita)</sub></p>
- **No navegador:** abra `http://127.0.0.1:8080`. Lá tem o **Chat**, um **Monitor** ao vivo do modelo e da sua
GPU/CPU/RAM, e o **About** com as configurações e os endereços.
- **Seus apps e agentes de programação:** adicione um provedor "OpenAI-compatible" com a URL base
**`http://127.0.0.1:8080/v1`**. Qualquer chave de API e qualquer nome de modelo funcionam.
- Apps que usam a API da Anthropic: `http://127.0.0.1:8080/v1/messages` (Claude Code:
`ANTHROPIC_BASE_URL=http://127.0.0.1:8080`).
- Codex CLI e outros apps que usam a OpenAI Responses API: `/v1/responses`
- **Raciocínio:** escolha **off, low, medium ou high** no menu do chat ou no "reasoning effort" do seu app. Off é o
mais rápido. High é o melhor para perguntas difíceis.
- **Imagens:** responda sim para "Images?" na instalação. Depois clique em **Picture** no chat, ou anexe imagens no
seu app. Placas AMD entendem imagens no Linux usando o processador; no Windows, ainda não.
- **Do seu celular ou de outro PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Sempre defina uma chave.
- **Uma requisição por vez:** por padrão, o Strata responde uma requisição e as outras esperam. Para responder
várias ao mesmo tempo, defina `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). Numa placa de 12 GB, isso deixa
cada resposta mais lenta.
- **Prompts longos:** o Strata lê a primeira mensagem de uma conversa inteira, cerca de 1 minuto a cada 30,000
tokens. As mensagens seguintes começam em segundos.
Mais: [onde suas conversas ficam salvas](docs/INSTALL.md#where-things-are-stored), [a API](docs/DETAILS.md#using-it).
### Opcional: mais velocidade com uma placa quando o modelo inteiro cabe na RAM
Duas configurações opcionais (de RubenTorney, #1904 e #1905). Elas ajudam quando você usa uma placa de vídeo e sua RAM guarda todos os experts (o modo arena do Strata, por exemplo Q2_0 ou IQ3_XXS num PC com 64 GB). Abra sua configuração `strata-<model>.json` na pasta de dados do Strata ([onde](docs/INSTALL.md#where-things-are-stored)) e adicione:
- **Geração mais rápida:** `"--adapt-async", "1"` na lista `"args"`. O cache de experts passa a ser atualizado numa thread auxiliar em vez de pausar a resposta. Medido numa RTX 5070 (Windows, Q2_0): +4-5% de geração, +7-10% em código e documentos longos. As respostas podem diferir um pouco do padrão (ficam iguais de uma execução para outra).
- **Leitura do prompt mais rápida no Windows:** `"STRATA_PIN_BY_NEED": "1"` no bloco `"env"`. O Windows só consegue travar cerca de metade da RAM para a placa; isto trava primeiro os experts de que a placa realmente precisa. Medido na RTX 5070: +4,4% na leitura do prompt.
```json
"args":[...,"--adapt-async","1"],
"env":{"STRATA_PIN_BY_NEED":"1"}
```
Reinicie o Strata (`START-HERE.bat` ou `./setup.sh`) depois da mudança. Para desfazer, remova as duas entradas.
## Algo deu errado?
- **Meu PC travou na primeira vez que o Strata iniciou.** Isso é normal enquanto ele carrega o modelo. Espere e não
feche a janela. Ainda travado depois de 10 minutos? Reinicie o PC, feche outros programas e tente de novo, ou
escolha um tamanho menor.
- **Parou durante o download ou a instalação.** Rode `START-HERE.bat` (ou `./setup.sh`) de novo. Ele continua de
onde parou.
- **Está muito lento e a luz do disco não para de piscar, ou aparece "the engine stopped unexpectedly".** Seu PC
não tem RAM livre suficiente. Feche outros programas (navegadores usam muita), ou escolha um tamanho menor (Q2_0
ou IQ2_XS).
- **Diz que a porta 8080 já está em uso.** O Strata já está rodando. Procure a janela dele.
Mais problemas e como resolver: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Ainda sem solução? Abra uma
[issue](https://github.com/Niko1221/Strata/issues) e anexe o `strata-<model>.log` da pasta do Strata. Encontrou um
problema de segurança? Informe em particular: [SECURITY.md](SECURITY.md).
## Como funciona?
Modelos como este normalmente rodam em servidores com centenas de gigabytes de memória de vídeo. Sua placa de vídeo
tem 12-24 GB. O Strata faz o modelo caber **dividindo o trabalho pelo PC inteiro**. Pense numa cozinha: o que você
usa o tempo todo fica na bancada, e o resto espera na despensa.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Os 24,576 experts do modelo: os mais usados na placa de vídeo, todos na RAM, uma tabela de consulta no SSD"></p>
- **O modelo é uma equipe de 24,576 pequenos especialistas ("experts").** Cada palavra precisa de só 10 deles.
- **Sua placa de vídeo** guarda os poucos milhares de experts usados com mais frequência. **Sua RAM** guarda todos
eles, e **seu processador** trabalha no resto ao mesmo tempo. **Seu SSD** guarda uma grande tabela de consulta.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Um pequeno ajudante adivinha as próximas palavras; o modelo grande confere todas de uma vez e fica com as certas"></p>
- **Adivinhar e conferir:** um pequeno ajudante adivinha as próximas palavras. O modelo grande confere todas de
uma vez. Você recebe a mesma resposta, 1.6-1.8x mais cedo.
- **Textos longos são lidos em grandes pedaços** (até 8,192 tokens por vez), a mais de 1,000 tokens por segundo.
A explicação mais longa: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Cada parte e seus números:
[os detalhes](docs/DETAILS.md#how-it-works) e o [artigo](docs/paper/Strata-Paper.pdf).
## Créditos e licença
O modelo é o [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), da equipe Qwen. Ele foi
comprimido por [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
e Unsloth. O Strata usa partes do [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Todos os créditos:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). O Strata é de código aberto sob a [MIT License](LICENSE).
Algumas partes e todos os modelos têm licenças próprias ([quais](docs/HOW_IT_WORKS.md#license)).
## Apoie o Strata
O Strata é gratuito e de código aberto. Se ele é útil para você, você pode apoiar o desenvolvimento:
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
{"d":"code","p":"Write a Python function that merges two sorted lists into one sorted list without using sort(). Include type hints and a short docstring."},
{"d":"code","p":"Implement an LRU cache in Python using OrderedDict with get and put methods, and write three unit tests with pytest."},
{"d":"code","p":"Write a JavaScript function debounce(fn, wait) and explain how it differs from throttle, with a small usage example."},
{"d":"code","p":"Write a C++ class template for a fixed-capacity ring buffer with push, pop and size, using std::array. Mention thread-safety."},
{"d":"code","p":"Show a Rust function that reads a file line by line, counts word frequencies in a HashMap and prints the top 5 words."},
{"d":"code","p":"Write a Go HTTP handler that returns JSON {\"status\":\"ok\"} and a middleware that logs the request duration."},
{"d":"code","p":"Write a SQL query that returns the top 3 customers by total order value per country from tables customers(id,country) and orders(id,customer_id,amount)."},
{"d":"code","p":"Write a bash script that finds all files larger than 100 MB under /var/log, sorted by size, and prints them in human readable form."},
{"d":"code","p":"Explain this Python error and how to fix it:\nTraceback (most recent call last):\n File \"app.py\", line 12, in <module>\n print(data['users'][0]['name'])\nKeyError: 'users'"},
{"d":"code","p":"Refactor this function to be more readable:\ndef f(a):\n r=[]\n for i in range(len(a)):\n if a[i]%2==0:\n r.append(a[i]*a[i])\n return r"},
{"d":"code","p":"Write a regular expression that matches ISO 8601 dates like 2026-10-11 and explain each part."},
{"d":"code","p":"Write a TypeScript interface for a paginated API response and a generic fetch helper that validates the shape at runtime."},
{"d":"math","p":"A train leaves a station at 9:15 and travels at 80 km/h. A second train leaves the same station at 10:00 on the same track at 120 km/h. At what time does the second catch the first? Show the steps."},
{"d":"math","p":"Prove that the sum of the first n odd numbers is n squared, using induction."},
{"d":"math","p":"Solve the system: 3x + 2y = 16 and x - y = 2. Then check the solution."},
{"d":"math","p":"What is the probability of getting exactly two heads in five fair coin flips? Explain the formula you use."},
{"d":"reason","p":"Three friends, Ann, Ben and Cara, each own exactly one of a cat, a dog and a parrot. Ann does not own the dog. Ben owns neither the cat nor the parrot. Who owns what? Explain your reasoning."},
{"d":"reason","p":"I have a 3 liter jug and a 5 liter jug and unlimited water. How can I measure exactly 4 liters?"},
{"d":"chat","p":"I have a job interview tomorrow for a junior data analyst role and I am nervous. Can you give me practical advice?"},
{"d":"chat","p":"What are some good ways to learn a new language as an adult with only 30 minutes a day?"},
{"d":"chat","p":"Recommend a weekly meal prep plan for someone who wants high protein, vegetarian meals on a budget."},
{"d":"chat","p":"My laptop gets very hot and the fan is loud when I open many browser tabs. What can I check?"},
{"d":"creative","p":"Write a short story (about 300 words) about a lighthouse keeper who discovers that the light attracts something other than ships."},
{"d":"creative","p":"Write a poem about autumn rain in the style of a haiku sequence, five stanzas."},
{"d":"creative","p":"Describe a bustling night market in a fantasy city, using all five senses."},
{"d":"science","p":"Explain how a transformer neural network processes a sentence, from tokens to the next-token prediction, for a software engineer who has not studied machine learning."},
{"d":"science","p":"Why is the sky blue and sunsets red? Explain Rayleigh scattering without heavy math."},
{"d":"science","p":"Explain how CRISPR-Cas9 edits a gene, and name two ethical concerns."},
{"d":"history","p":"Summarize the causes and consequences of the fall of the Western Roman Empire in a few paragraphs."},
{"d":"history","p":"Compare the economic policies of the New Deal with those of the Reagan era."},
{"d":"law","p":"Explain the difference between a trademark, a copyright and a patent, with one everyday example of each."},
{"d":"business","p":"Draft a polite email to a customer whose order is delayed by two weeks, offering a 10 percent discount and a new delivery date."},
{"d":"business","p":"Write a one-page project proposal outline for migrating a small company's file server to a cloud storage service."},
{"d":"data","p":"Create a markdown table comparing PostgreSQL, MySQL and SQLite on concurrency, typical use cases, licensing and scaling."},
{"d":"data","p":"Return a JSON object describing a fictional book with fields title, author, year, genres (array) and a nested publisher object. Only output JSON."},
{"d":"tool","p":"You are an assistant with a tool get_weather(city, unit). The user asks: 'Do I need an umbrella in Hamburg tomorrow?' Write the tool call you would make as JSON and then explain what you would tell the user once you get the result."},
{"d":"summ","p":"Summarize in three bullet points: Photosynthesis converts light energy into chemical energy. In the light-dependent reactions, chlorophyll absorbs photons and splits water, releasing oxygen and producing ATP and NADPH. In the Calvin cycle, these carriers drive the fixation of carbon dioxide into sugars. The rate of photosynthesis depends on light intensity, carbon dioxide concentration and temperature."},
{"d":"fr","p":"Explique en français comment fonctionne un moteur à combustion à quatre temps, avec des phrases simples."},
{"d":"de","p":"Schreibe eine kurze, freundliche E-Mail auf Deutsch, in der du einen Termin beim Zahnarzt von Dienstag auf Donnerstag verschiebst."},
{"d":"es","p":"Dame una receta sencilla de tortilla española y explica los errores más comunes al hacerla."},
- **Correctness runs:** `rib_norepeat.py`, with a fixed cache, no swaps, no PCIe share and no suffix drafts where two
runs are compared byte for byte.
- **Speed runs:** `rib_pp.py`.
- Raw results are in `data/`.
## Correctness
| Check | Result |
|---|---|
| One GPU, this branch vs the 0.1.21-equivalent run (fixed cache) | 10/10 byte-identical |
| Speed projection (control vector), one GPU vs the split on the same GPU (K=24) | 10/10 byte-identical |
| KV streaming (128K context, 32K resident), one GPU vs the split on the same GPU | 10/10 byte-identical |
| Images, one GPU vs the split on the same GPU (three questions about a test picture) | identical answers |
| Speed projection / KV streaming / plain, 5080 + 3090 (K=24): no-repeat under presence 100, the 144 sanity case | all pass, 0 repeats |
| The vector acts across GPUs: 5080 + 3090 with vs without the vector | 8 of 10 answers differ (as on one GPU: 7 of 10) |
| Images, 5080 + 3090 (K=24): "What color is the circle?" / "What number?" / describe | "Red" / "42" / "A red circle, the number 42, and a blue square are displayed on a white background" |
| Mid-prompt checkpoint (every 16,384 tokens): a 19.5K prompt, then another sharing its first 18,600 tokens | one GPU and 5080 + 3090: reused 16,384, answer right |
| 5080 + 3090 (K=30) + a 2080 Ti helper holding 3,000 experts | all pass |
| `--mmap-experts` with the Coder (a native pack) | a clear start-up error: it needs a canonical pack's `experts.bin` (with or without a split) |
Across GPUs the text is not byte-identical to one GPU: experts that one card leaves to the CPU run on a GPU in the
split, and the two round differently.
## Speed
| Run | Prompt 16K / 28K / 60K tok/s | Decode story / code tok/s | Hits |
"text":"The wind didn\u2019t just howl; it screamed. It tore through the iron gallery of the Blackwood Point Light, rattling the glass panes like a prisoner testing the bars of a cell.\n\nElias Thorne was not a man who feared the dark. He was a man who feared the silence.\n\nFor forty years, he had lived in the shadow of the Fresnel lens, a rotating eye of glass and brass that swept the Atlantic with a beam of pure, unyielding light. His job was simple: keep the light burning. Keep the logbook current. Keep the foghorn sounding when the mist rolled in, thick as wool.\n\nBut Elias knew something else. He knew that the light was not just for ships. It was for the lost.\n\nIt was November 14th, 1982. The storm was a Category 5, a monster that had swallowed the horizon. The waves were mountains of black water, crashing against the granite"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":134,
"drafts_offered":180,
"tok_s":80.8,
"text":"```python\nimport csv\nfrom typing import List, Dict\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, str]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False"
"text":"The wind didn\u2019t just howl; it screamed against the glass of Lantern House 42, demanding entry into every crack and crevice that had been sealed with tar in previous years but now wept like old wounds on an open sore skin\u2014like a wound left unhealed by time. Elias knew this rhythm better than he remembered his own name<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":26,
"drafts_offered":76,
"tok_s":49.3,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in my teeth, searching for something to break its rhythm against stone that refused be broken by anything less than God himself had forged into being willing and able withstand eternity\u2019s cruelty without complaint or mercy from start finish until now forevermore if any such concept even existed within human comprehension let alone divine authority which we all knew did not exist but wished it would because wishful thinking is always easier to imagine hopelessly impossible truths as probable lies than vice versa.\n\nElias Thorne was eighty-three years old today, exactly one hundred and forty-seven days since his last heart attack had been logged by Dr Armitage who himself retired seven months ago after receiving a subpoena from the county clerk indicating he might have overstayed professional courtesy boundaries"
"text":"The phrase \"the History Of The Sea\" is inherently ambiguous, as it could refer to several distinct historical narratives: geological formation (how oceans came into existence), maritime navigation and exploration by humans throughout human civilization; or perhaps more specifically in modern times refers often used metaphorically for economic trends like stock market fluctuations (\"sea change\"). However given your request likely intends an academic-style comprehensive overview covering both natural science aspects AND cultural significance through time periods\u2014especially since this kind question appears frequently on exams asking students write essays discussing evolution from primordial origins up present-day impacts including environmental changes due climate crisis etc.\u2014I will craft such broad yet structured piece combining geology + anthropology/ecological impact together while maintaining coherent flow suitable high-level educational context without overly technical jargon but still accurate enough scientifically grounded facts where appropriate within reasonable scope limits typical school assignments expect (~80\u201312% word count range implied implicitly though not stated explicitly here we aim moderate length ~65 words? Wait no user said \u201clong\u201d so probably >3k+ characters minimum safe bet around 7-9 paragraphs total approximating maybe close-to-full-page standard double-spaced equivalent which translates roughly \u22654K chars safely acceptable). Let me proceed accordingly ensuring clarity structure intro body conclusion style formal tone neutral informative balanced perspective acknowledging uncertainties if any minor ones exist e.g exact dates vary depending sources hence use hedging language sparingly only when necessary avoid overclaiming certainty especially regarding speculative theories vs established consensus knowledge base current mainstream understanding circa early-mid"
"text":"The wind didn\u2019t just howl; it screamed. It tore through the iron gallery of the Blackwood Point Light, rattling the glass panes like a prisoner testing the bars of a cell.\n\nElias Thorne was not a man who feared the dark. He was a man who feared the silence.\n\nFor forty years, he had lived in the shadow of the Fresnel lens, a rotating eye of glass and brass that swept the Atlantic with a beam of pure, unyielding light. His job was simple: keep the light burning. Keep the logbook current. Keep the foghorn sounding when the mist rolled in, thick as wool.\n\nBut Elias knew something else. He knew that the light was not just for ships. It was for the lost.\n\nIt was November 14th, 1982. The storm was a Category 5, a monster that had swallowed the horizon. The waves were mountains of black water, crashing against the granite"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":134,
"drafts_offered":180,
"tok_s":81.0,
"text":"```python\nimport csv\nfrom typing import List, Dict\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, str]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False"
"text":"The wind didn\u2019t just howl; it screamed against the glass of Lantern House 42, demanding entry into every crack and crevice that had been sealed with tar in previous years but now wept like old wounds on an open sore skin\u2014like a wound left unhealed by time. Elias knew this rhythm better than he remembered his own name<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":26,
"drafts_offered":76,
"tok_s":48.9,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in my teeth, searching for something to break its rhythm against stone that refused be broken by anything less than God himself had forged into being willing and able withstand eternity\u2019s cruelty without complaint or mercy from start finish until now forevermore if any such concept even existed within human comprehension let alone divine authority which we all knew did not exist but wished it would because wishful thinking is always easier to imagine hopelessly impossible truths as probable lies than vice versa.\n\nElias Thorne was eighty-three years old today, exactly one hundred and forty-seven days since his last heart attack had been logged by Dr Armitage who himself retired seven months ago after receiving a subpoena from the county clerk indicating he might have overstayed professional courtesy boundaries"
"text":"The phrase \"the History Of The Sea\" is inherently ambiguous, as it could refer to several distinct historical narratives: geological formation (how oceans came into existence), maritime navigation and exploration by humans throughout human civilization; or perhaps more specifically in modern times refers often used metaphorically for economic trends like stock market fluctuations (\"sea change\"). However given your request likely intends an academic-style comprehensive overview covering both natural science aspects AND cultural significance through time periods\u2014especially since this kind question appears frequently on exams asking students write essays discussing evolution from primordial origins up present-day impacts including environmental changes due climate crisis etc.\u2014I will craft such broad yet structured piece combining geology + anthropology/ecological impact together while maintaining coherent flow suitable high-level educational context without overly technical jargon but still accurate enough scientifically grounded facts where appropriate within reasonable scope limits typical school assignments expect (~80\u201312% word count range implied implicitly though not stated explicitly here we aim moderate length ~65 words? Wait no user said \u201clong\u201d so probably >3k+ characters minimum safe bet around 7-9 paragraphs total approximating maybe close-to-full-page standard double-spaced equivalent which translates roughly \u22654K chars safely acceptable). Let me proceed accordingly ensuring clarity structure intro body conclusion style formal tone neutral informative balanced perspective acknowledging uncertainties if any minor ones exist e.g exact dates vary depending sources hence use hedging language sparingly only when necessary avoid overclaiming certainty especially regarding speculative theories vs established consensus knowledge base current mainstream understanding circa early-mid"
"text":"The wind didn\u2019t just howl; it screamed. It tore through the gaps in the iron railing of the Lantern Room, carrying the scent of brine and rotting kelp. Elias pulled his woolen collar up to his chin, his fingers numb and clumsy inside thick leather gloves.\n\nHe was the last man on the island. The automated systems had failed three winters ago, a glitch in the solar-powered backup that no engineer had been able to fix before the supply boat stopped coming. Now, it was just him, the light, and the sea.\n\nElias climbed the spiral stairs, his boots heavy with salt spray. The iron steps groaned under his weight, a sound that seemed to mock the storm outside. He reached the top, the glass walls vibrating with the force of the gale. Outside, the world was a blur of black and white. The ocean was invisible; it was only a presence, a weight pressing against the stone foundation of the tower.\n\nHe"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":173,
"tok_s":96.8,
"text":"```python\nimport csv\nfrom typing import List, Dict\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, str]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False"
"text":"The wind didn\u2019t just howl; it screamed against the glass of Lantern House 42, demanding entry into every crack and crevice that had been sealed with tar in previous years but now wept like old wounds at this one\u2019s age: forty-three winters since its construction. My name was Elias Thorne\u2014no middle initial\u2014and I kept no logbook for visitors because there were none; only ships passed by, their hulls cutting through the black water as if they too feared what lay beneath it more than above or beside them in fog so thick that even my own reflection vanished from sight when looked upon directly.\n\nI climbed each morning not to greet dawn but check whether light still lived within me\u2014the bulb\u2019s heart had failed twice before, once during a storm named *Harrow*,"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":29,
"drafts_offered":110,
"tok_s":52.2,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in my teeth, searching for something to hold onto until I died.\u201d said Elias Thorne as he adjusted his collar against November\u2019s bite.** He was not merely complaining of cold weather but articulating a poetic metaphor: that even after death (metaphorically), there would be no relief from this relentless gale.\n\nElias had been the last human keeper on any significant stretch coast since automation became standard practice, yet here he stood , stubbornly refusing to let technology replace his vigilance . His cottage perched atop a crumbling granite foundation was more than just shelter; it housed centuries of handwritten logs detailing every ship\u2019s name and cargo load that passed within sight.\n\nOn most nights Elias tended the light mechanically\u2014"
"text":"The phrase \"the History Of The Sea\" is inherently ambiguous, as it could refer to several distinct historical narratives: geological formation (how oceans came into existence), maritime navigation and exploration histories (\"history at/over/across/on/in/of/to/from/toward/off by means around above below through under upon alongside against among between within without throughout despite during after before since until while when where who whom whose which what how why whether\"), or perhaps even metaphorical interpretations involving human interaction with water bodies over time periods like ages eras epochs generations centuries decades years months days hours minutes seconds moments instants times occasions instances events episodes incidents occurrences situations circumstances conditions states phases stages steps processes procedures sequences series chains cycles loops repetitions iterations continuations progressions developments evolutions transformations changes alterations modifications adjustments corrections revisions amendments updates upgrades improvements enhancements refinements optimizations perfecting completing finishing concluding ending terminating ceasing stopping halting pausing resuming continuing proceeding advancing progressing moving shifting translating rotating reflecting mirroring flipping turning twisting bending curving arches spirals helix coils twists turns rotations reflections mirrors flips inversions reversings reversion backtracking retracing reversing undo redo do again repeat repeats repeated repeatedly repetition repetitive repetitiveness repete repeater repeatingly recurrent recurrence recurring recurs recur recurred recursion recursive recursively recurse recessed recessionary regress regression regressed retrograde reverse reversed reversal revert reverted reverberate resonated echo echoed echoing echoes echolocation sonar radar radio telecommunication communication communications communicate communicated communicating communicative comms comm chatter chat chats chatting chatted talk talks talked"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and shadow.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined his life for thirty years.\n\nElias was not a man of many words. He was a man of many tasks. His hands, mapped with scars from broken glass and rusted iron, moved with a rhythm that predated his memory. He checked the clockwork mechanism of the Fresnel lens, ensuring the rotation was smooth, silent, and precise. *Tick. Tick. Tick.* The sound was the heartbeat of the coast."
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":135,
"drafts_offered":184,
"tok_s":85.2,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral vaults below Elias\u2019s boots smelled faintly brine and old coffee as he climbed spiral stairs worn smooth by decades feet his hands were calloused from gripping iron railings that had been polished to a dull shine\n\nElias was not merely keeping light alive; He kept order intact. Every night, at precisely 18:07 hours local time\u2014adjusted for leap seconds and magnetic drift\u2014he wound the clockwork mechanism inside brass housing behind Fresnel lens This wasn\u2019t vanity It\u2019s necessity The rotation speed must remain constant or ships lose their bearing in fog where compasses fail\n\nHe spoke softly to himself while working, murmuring numbers aloud because silence made him"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":67,
"repeats":0,
"drafts_accepted":14,
"drafts_offered":68,
"tok_s":41.2,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in his teeth, like he had no mouth to hold back its fury anymore that night.\u201d Elara pulled her wool coat tighter around herself as she climbed spiral staircase of **Blackrock Lighthouse**. Her boots were slick with salt spray from previous days.<|im_end|>"
"text":"The ocean is not merely water; it has been, and remains to this day one giant storybook that weaves together geology with biology as wellas human culture into an epic narrative spanning billions years from its earliest beginnings in primordial times through modern-day ecological crises The vast expanse covers more than seventy percent Earth\u2019s surface area making up nearly all life support systems on our planet yet despite being so large humans have only explored less five thousand square kilometers out total 361 million km\u00b2 which means over ninety-nine point nine eight six per cent unexplored territory This makes oceans arguably larger unknown territories compared space exploration where NASA estimates they\u2019ve mapped roughly ten billion stars versus current marine surveys covering just two hundred fifty-six species documented globally against estimated millions undiscovered organisms including deep-sea creatures like anglerfish or vampire squid whose existence was first recorded centuries ago but still remain poorly understood due extreme pressures depths exceeding eleven miles below Pacific Trenches Mariana located near Guam island chain reaching maximum depth known today at approximately thirty-seven feet under pressure equivalent atmospheric levels causing structural collapse if unprotected by specialized vessels capable handling such conditions without failure mechanisms designed specifically for submersible operations operating within these zones require titanium hulls rated withstand forces greater gravity acting upon objects falling vertically downward acceleration rates calculated using Newtonian physics principles applied universally across gravitational fields regardless medium density variations present throughout different layers stratosphere troposphere mesospheres thermoclines etceterally speaking however when discussing aquatic environments instead air masses replace gases dissolved oxygen nitrogen arg"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy-three years old, and he had been the keeper of the Blackwood Point Light for forty of those years. The other thirty-three had been spent in a small, drafty cottage that smelled perpetually of kerosene and damp wood.\n\nFor decades, Elias had tended the great Fresnel lens, polishing the glass prisms until they gleamed like diamonds, winding the clockwork mechanism that turned the beam, and trimming the wick of the oil lamp. It was a job of patience, of precision, of silence. The light was his charge. The sea was his neighbor.\n\nBut this morning, the sea was not his neighbor. It was his enemy.\n\nThe storm had been forecasted for three days, but it had arrived in one. The barometric pressure had dropped so sharply that Elias\u2019s ears popped."
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":178,
"tok_s":63.7,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral vaults below Elias\u2019s boots smelled faintly brine and old coffee as he climbed spiral stairs worn smooth by decades feet his hands were calloused from gripping iron railings that had been polished to a dull shine.\n\nElias wasn\u2019t just keeping light; was guarding silence the beam swept across black water every seven seconds cutting through fog so thick it swallowed sound entirely for three nights straight no ships answered its signal not even fishing trawlers seeking shelter they\u2019d long abandoned this stretch of coast where currents shifted unpredictably pulling debris from sunken wrecks into tangled heaps that snagged anchors and tore hulls open like wounds.\n\nHe remembered the last time someone had"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":59,
"repeats":0,
"drafts_accepted":11,
"drafts_offered":46,
"tok_s":33.7,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in his teeth, trying to pry them apart like loose planks of weathered oak outside your own mouth again and then you are going away from home is fine but for now let me write a proper story.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 11",
"tokens":60,
"repeats":0,
"drafts_accepted":17,
"drafts_offered":48,
"tok_s":43.2,
"text":"Aardvark \nAntelope allonax (Oryx) \u2013 *Note: Using standard names for clarity* \n\nLet me provide a clean list of exactly **1** animal name on each single physical text file/terminal output stream with no other characters except the newline (\\n).<|im_end|>"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element\u2014it became civilization itself through its capacity as both barrier (to land) but also conduit between lands\u2014and thus gave rise directly from being part nature\u2019s own evolutionary engine into becoming humanity's greatest historical actor across all time scales since first emergence until now present-day future projections still ongoing today within our current era context window limits allowed here by system constraints set above threshold levels required before output generation begins below line break marker inserted immediately after final punctuation mark ending entire preceding paragraph block structure defined earlier during initial drafting phase prior execution stage completed successfully without errors detected internally via internal validation routines executed automatically upon completion cycle termination signal received end-of-file condition triggered correctly according standard protocol specifications followed strictly adhered-to throughout process flow control logic implemented using structured programming constructs compatible with target runtime environment configuration parameters specified herein explicitly declared beforehand at top-level scope declaration section located near beginning source code file header comments included inline documentation style guidelines observed consistently applied uniformly everywhere else similarly formatted sections appearing later down document body text area scrolling vertically downward directionally aligned left margin justified right edge wrapped around fixed-width column boundaries determined dynamically based available screen real estate measured pixel-per-unit scaling factor adjusted responsively depending viewport dimensions reported back client-side rendering pipeline components responsible"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the fabric stiff with salt and ice. Outside the thick glass of the lantern room, the Atlantic was a churning cauldron of black and white, the waves crashing against the granite base with a violence that shook the very foundations of the tower. It was November, 1924, and the storm had been raging for three days.\n\nElias was not a man who feared the dark. He was a man who feared the silence.\n\nHe checked the clock on the wall. 11:42 PM. The rotation of the great Fresnel lens was steady, a rhythmic *whirrr-click, whirrr-click* that seemed to hold the chaos of the ocean at bay. The beam swept out, a blade of pure, concentrated light cutting through the rain and sleet, illuminating the jagged teeth of the reef below.\n\n"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":179,
"tok_s":81.3,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral vaults below Elias\u2019s boots were heavy with salt and exhaustion as he climbed one hundred twelve steps to reach his post at dawn every morning for thirty years. His name was written on a brass plaque by door: **Elias Thorne**.\n\nHe wasn\u2019t just keeping light; He kept time, weathered hands gripping the cold iron railing until knuckles turned white against frostbite and windburn that never healed fully because they were always there waiting to be felt again when he stepped outside into storm or calm alike. The lens above him was a Fresnel giant\u2019s eye made of hand-ground crystal prisms each one polished daily with chamois cloth soaked in whale"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":65,
"repeats":0,
"drafts_accepted":11,
"drafts_offered":58,
"tok_s":42.6,
"text":"The wind didn\u2019t just blow at Cape Hollow; it screamed through the gaps in my cottage\u2019s shutters, rattling them like loose teeth until they slammed against their hinges every night for eleven months of winter and summer alike.\u201d I had never seen calm water here\u2014only gray churning seas that swallowed ships smaller than themselves<|im_end|>"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element within our planet\u2019s geography but rather its most fundamental biological engine\u2014a living system whose existence predates humanity by hundreds millions years yet continues shaping every aspect from climate regulation through evolutionary biology down even into modern industrial logistics today's global supply chains depend upon them directly indirectly via atmospheric pressure systems weather patterns migration routes food webs carbon cycles oxygen generation thermal buffering tidal forces salinity gradients depth stratification light attenuation currents circulation gyres upwelling zones thermoclines halothermics bathymetric features continental shelves abyssal plains trenching ridges volcanic arcs island formations atoll structures seamounts submarine canons canyon walls cliff faces overhang ledge platforms reef flats fore-reef back-lagoon inner-shelf outer-slope drop-offs pinnacle pinnacles spires columns arches tunnels caves grottos swim-through passages chimney vents blowholes fissures cracks crevices joints faults fractures breaks gaps holes openings void spaces hollow chambers cavernous interiors subterranean networks underground rivers lakes pools springs wells shaft pits sinks drains sewers pipes conduits tubes channels gutters ditches furrows grooves rills runnels streams brooks rivulets creeks tributaries branches forks splits divergences con"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the fabric stiff with salt and ice. Outside the thick glass of the lantern room, the Atlantic was a churning cauldron of black and white, the waves crashing against the granite base with a violence that shook the very foundations of the tower. It was November, 1924, and the storm had been raging for three days.\n\nElias was not a man who feared the dark. He was a man who feared the silence.\n\nHe checked the clock on the wall. 11:42 PM. The rotation of the great Fresnel lens was steady, a rhythmic *whirrr-click, whirrr-click* that seemed to hold the chaos of the ocean at bay. The beam swept out, a blade of pure, concentrated light cutting through the rain and sleet, illuminating the jagged teeth of the reef below.\n\n"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":179,
"tok_s":76.2,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral vaults below Elias\u2019s boots were heavy with salt and exhaustion as he climbed one hundred twelve steps to reach his post at dawn every morning for thirty years. His name was written on a brass plaque by door: **Elias Thorne**.\n\nHe wasn\u2019t just keeping light; He kept time, weathered hands gripping the cold iron railing until knuckles turned white against frostbite and windburn that never healed fully because they were always there waiting to be felt again when he stepped outside into storm or calm alike. The lens above him was a Fresnel giant\u2019s eye made of hand-ground crystal prisms each one polished daily with chamois cloth soaked in whale"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":65,
"repeats":0,
"drafts_accepted":11,
"drafts_offered":58,
"tok_s":42.3,
"text":"The wind didn\u2019t just blow at Cape Hollow; it screamed through the gaps in my cottage\u2019s shutters, rattling them like loose teeth until they slammed against their hinges every night for eleven months of winter and summer alike.\u201d I had never seen calm water here\u2014only gray churning seas that swallowed ships smaller than themselves<|im_end|>"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element within our planet\u2019s geography but rather its most fundamental biological engine\u2014a living system whose existence predates humanity by hundreds millions years yet continues shaping every aspect from climate regulation through evolutionary biology down even into modern industrial logistics today's global supply chains depend upon them directly indirectly via atmospheric pressure systems weather patterns migration routes food webs carbon cycles oxygen generation thermal buffering tidal forces salinity gradients depth stratification light attenuation currents circulation gyres upwelling zones thermoclines halothermics bathymetric features continental shelves abyssal plains trenching ridges volcanic arcs island formations atoll structures seamounts submarine canons canyon walls cliff faces overhang ledge platforms reef flats fore-reef back-lagoon inner-shelf outer-slope drop-offs pinnacle pinnacles spires columns arches tunnels caves grottos swim-through passages chimney vents blowholes fissures cracks crevices joints faults fractures breaks gaps holes openings void spaces hollow chambers cavernous interiors subterranean networks underground rivers lakes pools springs wells shaft pits sinks drains sewers pipes conduits tubes channels gutters ditches furrows grooves rills runnels streams brooks rivulets creeks tributaries branches forks splits divergences con"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was the last of the line, the only man left on the jagged spine of Widow\u2019s Rock. For forty years, he had tended the light. For forty years, he had listened to the ocean\u2019s endless, hungry whisper.\n\nTonight was different. The storm wasn\u2019t just a storm; it was a *tempest*. The waves didn\u2019t crash; they *shattered*. They threw themselves against the granite base of the lighthouse with the force of a battering ram, sending tremors up through the soles of Elias\u2019s boots.\n\nHe climbed the spiral stairs, one hundred and twelve steps of iron and rust. His hand, calloused and scarred from decades of polishing the brass railing, found the familiar groove in the metal. He didn\u2019t look up. He didn\u2019t look down. He looked *forward*."
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":135,
"drafts_offered":176,
"tok_s":102.5,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty jail cell below sea level three hundred feet down to where Elias kept his boots dry and ready for emergencies when he had no time even think about changing them.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":140,
"repeats":0,
"drafts_accepted":22,
"drafts_offered":90,
"tok_s":50.1,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in his teeth, demanding entry into every crack and crevice of Stone Harbor Lighthous eKeepers have always been solitary figures against isolation\u2019s greatest adversary: time itself Elias Thorne knew this intimately. For thirty years he had kept watch over one hundred thousand ships each night scanning for lights that might signal disaster or hope\n\nHis job was simple but deadly serious He checked weather logs every day without fail The foghorn echoed through the tower three times before dawn Each morning at 4:02 AM Elias would climb eighty-two steps to reach his office where coffee cooled beside a single candle flickering in drafts from below<|im_end|>"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth\u2019s surface.\u201d \u2014 *This quote by Rachel Carson serves as an apt introduction.* The story we tell today begins with that first drop falling into primordial pools billions years ago\u2014but what follows next? What happens when those drops become oceans themselves\u2014when they begin moving across continents carrying sediment from mountains down rivers until finally reaching their final resting place at last\u2014the deep abyssal plains where sunlight cannot reach nor pressure crushes everything else alive there below depths greater than six kilometers beneath waves crashing against shores above them all around world forever changing shape over time due plate tectonics shifting landmass slowly apart together again repeatedly throughout geological epochs spanning hundreds millions or even thousands million+ yrs since formation early Hadean era through Archeozoic eons onward present Holocene epoch still ongoing now right here under current conditions observed via satellites monitoring global climate patterns affecting currents temperatures salinity levels biodiversity distribution human impacts altering ecosystems directly indirectly causing extinction events locally globally recently historically prehistorically anciently mythologically culturally spiritually philosoph scientifically mathematically physically chem biol geo astro cosmological sociopolitical economic environmental ecological evolutionary genetic linguistic literary artistic musical cinematic theatrical photographic journalistic digital analog virtual real tangible intangible metaphysical ontosemantic semiotic symbolic allegorical narrative descriptive analytical interpretive evaluative critical reflective introspective contemplatory ponderous thoughtful considerate attentive observant perceptive insightful intelligent knowledgeable informed educated literati learned skilled capable"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and green.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined his life for thirty years.\n\nElias was not a man of many words. He was a man of many tasks. His hands, mapped with scars from rope burns and broken glass, moved with a rhythm that predated his memory. He checked the clockwork mechanism of the Fresnel lens, listening for the faint, rhythmic *tick-tick-tick* that signaled the rotation was true. If the rhythm faltered, ships"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":178,
"tok_s":104.1,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty jail cell below sea level to above cloud height at once\u2014no matter how high you climbed or deep into its foundations down through rock and saltwater. It was November again when Elias first noticed that his logbook had begun recording entries he hadn\u2019t written yet: *Storm surge expected Tuesday, but the sky is clear.*\n\nHe stopped writing for a moment to listen\u2014not with ears alone\u2014but by pressing both palms flat against cold iron walls where condensation formed in slow spirals like fingerprints left behind after touching something invisible. The metal hummed faintly beneath his skin as if remembering old storms long past their dates on paper calendars torn from newspapers dated three years ahead of today"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":17,
"drafts_offered":63,
"tok_s":52.6,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in his teeth, trying to get inside where he had spent thirty years refusing entry except for family and friends like him who were old enough too know better than young men ever could be again about what safety meant. It was November 2014 when Elias Thorne first stepped onto that spiral staircase with only one boot on because his other foot hurt after walking from the harbor office all morning long before anyone else thought he needed shoes off there for work even if nobody saw them do it but everyone knew why so they did not ask him anymore once this happened already twice last winter near same season now here again today too soon then tomorrow maybe never will come back unless someone sends message tonight via radio signal code Morse dot dash pattern"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\u201d \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \u201cthe Sea\u201d or \"Ocean\" (terms often used interchangeably but distinct geographically) was never just an element within our planet\u2019s geography\u2014it became its engine itself\u2014a living system whose evolution shaped everything else around us from climate patterns down through evolutionary biology right up into modern civilization's economic structures cultural identities political boundaries ecological systems scientific understanding technological capabilities philosophical frameworks psychological experiences artistic expressions literary metaphors cinematic narratives musical compositions architectural designs urban planning strategies agricultural practices industrial processes medical discoveries legal precedents educational curricula religious traditions spiritual beliefs mythological stories folkloric tales poetic verses dramatic scripts visual arts photographic images digital media social networks behavioral norms emotional responses cognitive models decision-making algorithms predictive analytics statistical probabilities probabilistic reasoning causal inference counterfactual thinking hypothetical scenarios normative ethics deontal logic modal semantics temporal dynamics spatial relations topographical features hydrological cycles meteor phenomena atmospheric chemistry geological formations tectonic shifts volcanic eruptions seismic events tsunamis hurricanes cyclones monsoons typhoons droughts floods storms winds currents waves swells breakers surf foam spray mist fog rain snow ice glaciers berg floes drift pack-ice brine salinity pH alkalines acidity hardness softness temperature pressure depth light darkness color hue saturation brightness contrast luminance chroma tone value shade tint reflection"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, mapped with veins like the roots of an ancient oak, gripped the cold iron railing.\n\nFor forty years, Elias had tended the light at the end of the world. Not the end of the map, but the end of the land. Here, the Atlantic threw itself against the granite cliffs with such violence that the spray reached the gallery, thirty feet above the rocks.\n\nHe checked the clock. 11:42 PM. The fog was thickening, a gray soup that swallowed the horizon. In the old days, he would have climbed the stairs to the lantern room, polished the Fresnel lens, and wound the clockwork mechanism that turned the great beam. But the automation had changed"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":135,
"drafts_offered":179,
"tok_s":80.7,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral hall outside my window at midnight on November third this year\u2014so I pulled up to check what was happening with that storm and why they were calling me out there when all we had left for tonight\u2019s shift is one last pass through our logs before dawn breaks.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":17,
"drafts_offered":83,
"tok_s":43.9,
"text":"The wind didn\u2019t just blow at Cape Harrow; it screamed through the gaps in its teeth, hungry and relentless for three centuries of stone salt-spray history that had long since forgotten to be polite or gentle with itself against anything smaller than God Himself was willing acknowledge as divine protection from drowning souls who were already lost before they ever touched water.\n\nElias knew this rhythm better he himself could hear it sometimes; his boots worn smooth by pacing the spiral stair\u2019s center-post groove where thousands upon countless steps had rubbed away texture until all that remained beneath sole leather now felt like polished bone carved out slowly but surely over time into something more hollow than wood ever really should be allowed become when used so often enough within such confined circular spaces meant only for vertical ascent down always descending again back upward never truly leaving"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element within our planet\u2019s geography but rather its most fundamental biological engine\u2014a living system whose existence predates humanity by billions years yet continues shaping every aspect from climate regulation through evolutionary biology all way down modern industrial supply chains today's global shipping lanes carry cargo across vast distances because they were once crossed only slowly via coastal hopping routes used since antiquity until steamships revolutionized travel during late nineteenth century when diesel engines replaced wind power entirely allowing container ships carrying thousands TEUs (Twenty-foot Equivalent Units) per voyage crossing entire Pacific basins without stopping at intermediate ports anymore unlike earlier sailing vessels needing favorable winds currents seasonal patterns dictating departure windows based upon monsoon calendars established centuries ago before technological advances changed those constraints fundamentally altering logistics networks worldwide thereby transforming economic structures globally while simultaneously creating ecological disruptions due invasive species transported unintentionally alongside legitimate goods causing cascading ecosystem collapses locally where non-native organisms outcompete native fauna disrupting food webs leading population declines affecting biodiversity indices measured scientifically using standardized metrics like Shannon-Wiener diversity index calculations applied consistently over decades revealing trends showing significant decreases post-introduction events correl strongly with introduction dates matching historical records documenting first sightings confirmed taxonomically classified formally according International Code Nomenclature rules"
"text":"The wind didn\u2019t just howl; it screamed. It tore through the gaps in the iron railing of the Lantern Room, carrying the scent of brine and rotting kelp. Elias pulled his woolen collar up to his chin, his fingers numb and clumsy inside thick leather gloves.\n\nHe was the last man on the island. The automated systems had failed three winters ago, a glitch in the solar panel array that no engineer had been able to fix before the supply boat stopped running. Now, it was just him, the light, and the sea.\n\nElias climbed the spiral stairs, his boots heavy with salt spray. The iron steps groaned under his weight, a sound that seemed to mock the storm outside. He reached the top, the glass walls vibrating with the force of the gale.\n\nInside the lantern room, the great Fresnel lens sat like a dormant eye. It was cold, dark, and waiting.\n\nElias moved to the control panel. His"
},
{
"name":"code",
"tok_s":103.6,
"reps":[
103.6
],
"hit_rate":0.957731667226356,
"text":"```python\nimport csv\nfrom typing import List, Dict\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, str]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False"
}
],
"stage_timing":[],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"strata generate: layer split: CUDA1 holds its weights, session and head; 18.13 GiB free",
"strata generate: layer split auto: CUDA0 84 SMs at 2.81 GHz -> 0.31 ms per layer, 8.30 GiB for experts",
"strata generate: layer split auto: CUDA1 82 SMs at 1.86 GHz -> 0.48 ms per layer, 14.14 GiB for experts",
"strata generate: layer split auto: K=22 - predicted 19.7 ms per decode window; the caches hold 11298 of 12288 profiled pairs (~99.7% of the routed mass)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the fabric stiff with salt and ice. Outside the thick glass of the lantern room, the Atlantic was a churning cauldron of black and white, the waves crashing against the granite base with a violence that shook the very foundations of the tower. It was November, 1924, and the storm had been raging for three days.\n\nElias was not a man who feared the dark. He was a man who feared the silence.\n\nHe checked the clock on the wall. 11:42 PM. The rotation of the great Fresnel lens was steady, a rhythmic *whirrr-click, whirrr-click* that seemed to sync with his own heartbeat. He had been doing this for forty years. His hands were mapped with scars from rope burns and broken glass, his eyes permanently squinted against the glare of the bulb."
},
{
"name":"code",
"tok_s":74.0,
"reps":[
74.0
],
"hit_rate":0.5873783774988882,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and shadow.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined Elias\u2019s life for thirty years. He stood before the great Fresnel lens, a cathedral of glass and brass that caught the light of the single, powerful bulb and fractured it into a thousand beams.\n\nHe checked the clock. 11:42 PM. The rotation was smooth, silent, a hypnotic sweep that painted the darkness with a slow, rhythmic pulse. *Flash. Dark. Flash. Dark.*\n\nE"
},
{
"name":"code",
"tok_s":97.0,
"reps":[
97.0
],
"hit_rate":0.9562207324057541,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"strata generate: layer split: CUDA1 holds its weights, session and head; 17.96 GiB free",
"strata generate: layer split auto: CUDA0 84 SMs at 2.81 GHz -> 0.31 ms per layer, 8.13 GiB for experts",
"strata generate: layer split auto: CUDA1 82 SMs at 1.86 GHz -> 0.48 ms per layer, 13.98 GiB for experts",
"strata generate: layer split auto: K=22 - predicted 19.7 ms per decode window; the caches hold 11199 of 12288 profiled pairs (~99.7% of the routed mass)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, rough as sandpaper, gripped the iron railing as he ascended the final flight of steps to the lantern room.\n\nBelow him, the Atlantic churned in a frenzy, whitecaps exploding against the black rocks like fireworks. Above, the sky was a bruised purple, heavy with rain that hadn\u2019t yet fallen.\n\nElias checked the clock. 11:42 PM. The light was due.\n\nHe didn\u2019t flip a switch. He didn\u2019t press a button. He reached for the heavy brass lever that had worn smooth by his palm over the years. With a grunt that matched the strain of his aging muscles, he pulled it down.\n\n*Clunk.*\n\nThe"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":181,
"tok_s":105.5,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral hall outside my window at midnight on November third this year\u2014so I pulled up to that date and time for context because you asked me about a lighthouse keeper but gave no specifics beyond \u201cshort story,\u201d so let\u2019s set it: isolated coastal station off the Oregon coast during winter storm season, 1987.\n\nMy name is Elias Vane (I made this fictional; real keepers often kept journals signed with initials only), and I\u2019m writing from memory because my logbook got waterlogged when a rogue wave breached lower storage room at dawn\u2014so here\u2019s what happened:<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":109,
"repeats":0,
"drafts_accepted":13,
"drafts_offered":56,
"tok_s":54.1,
"text":"The wind didn\u2019t just blow at Cape Hollow; it screamed through the gaps in my cottage\u2019s shutters, rattling them like loose teeth until they settled into silence with every storm break out of three hundred days here being storms that broke something else again but mostly kept me warm because I burned coal and wood whenever we had any left over from last winter when even then if you\u2019re lucky enough to get through it all without freezing yourself so hard your own thoughts become ice too\n\nBut Elias knew better than anyone: fear was useful only until dawn<|im_end|>"
"text":"The ocean is not merely water; it has been, and continues to be one with humanity\u2019s most profound narrative arc in human civilization: **the story we tell ourselves through our relationship** between landmasses separated by vast expansiveness that defines both isolation from other worlds (and thus their own unique evolutionary paths) as well than connection across them via currents wind patterns tides depth pressure temperature salinity biodiversity geology climate change economic trade routes ecological systems cultural exchange political boundaries scientific discovery technological advancement environmental degradation conservation efforts maritime archaeography historical cartographic evolution legal frameworks international law enforcement mechanisms insurance liability regimes regulatory compliance standards safety protocols navigation aids communication networks emergency response strategies search-and-rescue operations salvage rights ownership disputes heritage protection zones sustainable harvesting practices aquaculture development infrastructure engineering projects coastal management policies urban planning regulations zoning ordinances building codes architectural design guidelines aesthetic considerations functional requirements accessibility provisions energy efficiency measures thermal insulation techniques moisture control methods ventilation optimization lighting solutions acoustic soundproofing structural integrity assessments seismic resistance evaluations flood mitigation plans storm surge barriers levee construction seawall reinforcement breakwater installation harbor dredging channel maintenance buoy placement lighthouse automation radar monitoring sonar mapping bathymetric surveys hydrographic charts nautical almanacs tide tables current predictions weather forecasts marine meteorological models satellite imagery remote sensing data assimilation numerical simulations computational fluid dynamics finite element analysis partial differential equations tensor calculus linear algebra probability theory statistics stochastic processes Markov chains Bayesian inference machine learning artificial neural network deep convolutional recurrent transformer attention heads generative adversarial diffusion transformers large language model tokenization"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, rough as sandpaper, gripped the iron railing as he ascended the final flight of steps to the lantern room.\n\nBelow him, the Atlantic churned in a frenzy, whitecaps exploding against the black rocks like fireworks. Above, the sky was a bruised purple, heavy with rain that hadn\u2019t yet fallen.\n\nElias checked the clock. 11:42 PM. The light was due.\n\nHe didn\u2019t flip a switch. He didn\u2019t press a button. He reached for the heavy brass lever that had worn smooth by his palm over the years. With a grunt that matched the strain of his aging muscles, he pulled it down.\n\n*Clunk.*\n\nThe"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":136,
"drafts_offered":181,
"tok_s":99.4,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral hall outside my window at midnight on November third this year\u2014so I pulled up to that date and time for context because you asked me about a lighthouse keeper but gave no specifics beyond \u201cshort story,\u201d so let\u2019s set it: isolated coastal station off the Oregon coast during winter storm season, 1987.\n\nMy name is Elias Vane (I made this fictional; real keepers often kept journals signed with initials only), and I\u2019m writing from memory because my logbook got waterlogged when a rogue wave breached lower storage room at dawn\u2014so here\u2019s what happened:<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":109,
"repeats":0,
"drafts_accepted":13,
"drafts_offered":56,
"tok_s":42.1,
"text":"The wind didn\u2019t just blow at Cape Hollow; it screamed through the gaps in my cottage\u2019s shutters, rattling them like loose teeth until they settled into silence with every storm break out of three hundred days here being storms that broke something else again but mostly kept me warm because I burned coal and wood whenever we had any left over from last winter when even then if you\u2019re lucky enough to get through it all without freezing yourself so hard your own thoughts become ice too\n\nBut Elias knew better than anyone: fear was useful only until dawn<|im_end|>"
"text":"The ocean is not merely water; it has been, and continues to be one with humanity\u2019s most profound narrative arc in human civilization: **the story we tell ourselves through our relationship** between landmasses separated by vast expansiveness that defines both isolation from other worlds (and thus their own unique evolutionary paths) as well than connection across them via currents wind patterns tides depth pressure temperature salinity biodiversity geology climate change economic trade routes ecological systems cultural exchange political boundaries scientific discovery technological advancement environmental degradation conservation efforts maritime archaeography historical cartographic evolution legal frameworks international law enforcement mechanisms insurance liability regimes regulatory compliance standards safety protocols navigation aids communication networks emergency response strategies search-and-rescue operations salvage rights ownership disputes heritage protection zones sustainable harvesting practices aquaculture development infrastructure engineering projects coastal management policies urban planning regulations zoning ordinances building codes architectural design guidelines aesthetic considerations functional requirements accessibility provisions energy efficiency measures thermal insulation techniques moisture control methods ventilation optimization lighting solutions acoustic soundproofing structural integrity assessments seismic resistance evaluations flood mitigation plans storm surge barriers levee construction seawall reinforcement breakwater installation harbor dredging channel maintenance buoy placement lighthouse automation radar monitoring sonar mapping bathymetric surveys hydrographic charts nautical almanacs tide tables current predictions weather forecasts marine meteorological models satellite imagery remote sensing data assimilation numerical simulations computational fluid dynamics finite element analysis partial differential equations tensor calculus linear algebra probability theory statistics stochastic processes Markov chains Bayesian inference machine learning artificial neural network deep convolutional recurrent transformer attention heads generative adversarial diffusion transformers large language model tokenization"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, rough as sandpaper, knew every bolt, every lever, and every creak in the iron bones of the tower.\n\nFor forty years, Elias had been the guardian of the Point. Not just of the light, but of the silence between the waves.\n\nIt was November. The sky was a bruised purple, heavy with rain that hadn\u2019t yet fallen. The sea was a churning cauldron of grey and white, throwing itself against the granite base with a violence that shook the floorboards beneath Elias\u2019s boots.\n\nHe climbed the stairs. One hundred and twelve steps. He counted them by feel, his left hand trailing along the cold iron railing. At the top, the air was"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":137,
"drafts_offered":178,
"tok_s":103.4,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral vaults below Elias\u2019s boots were heavy with salt and exhaustion as he climbed one hundred twelve steps to reach top room where his world began again.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":58,
"repeats":0,
"drafts_accepted":14,
"drafts_offered":54,
"tok_s":52.9,
"text":"The wind didn\u2019t just blow at Cape Harrow; it screamed through the gaps in stone and steel, carrying with them spray from three miles out into churning black water that no longer resembled an ocean so much as abyssal chaos waiting for its prey to surface beneath waves of salt.<|im_end|>"
"text":"The ocean is not merely water; it has been, and continues to be one with humanity\u2019s most profound narrative arc in human civilization: **the story told by waves**. It serves as both witness (through sedimentary layers)and participant( through currents that carry messages across continents). Its \"history\" spans geological epochs where life first emerged from primordial pools into complex ecosystems today\u2014and cultural histories shaped entirely around its roleas connector between distant lands or barrier separating them forever depending on perspective taken at different times throughout millennia since ancient civilizations began building boats outof logs carved using stone tools until modern container ships carrying billions tons cargo daily cross Atlantic routes once navigated onlyby stars alone under moonlight during nights when sailors feared monsters lurking beneath hulls made fragile against storms they could barely predict even now despite advanced meteorology systems developed over past century after industrial revolution transformed maritime trade patterns globally due economic pressures driving demand for faster logistics chains linking factories producing electronics manufactured inside cities located far inland near rivers feeding estuaries opening directly onto open seas connected via canals engineered centuries ago like Suez Canal built originallyfor military purposes later adapted commercial use post-World War II era reshaping global supply networks dramatically because geopolitical shifts altered which ports remained accessible politically versus physically blocked temporarily then reopened permanently following treaty agreements signed among nations agreeing upon shared responsibilities regarding environmental stewardship including coral reef restoration efforts funded jointly international organizations coordinating scientific research conducted aboard vessels specifically designed deep-sea exploration equipped robotic submersibles capable descending trenches deeper than any previous technology allowed"
"text":"The wind didn\u2019t just blow; it screamed. It tore through the iron gallery of the Blackwood Point Lighthouse, rattling the glass panes like a prisoner shaking its bars.\n\nElias Thorne was seventy years old, and he had been the keeper for forty of them. He knew every creak in the spiral staircase, every draft that slipped under the door, and every shift in the weather that the barometer couldn\u2019t quite predict. He knew that when the fog rolled in thick enough to swallow the horizon, the world ceased to exist. There was only the light, the stone, and him.\n\nIt was November, the month of storms. The sea was a churning cauldron of grey and white, throwing itself against the granite base with a violence that made the floorboards tremble. Elias stood by the lantern room, his hands wrapped around a mug of coffee that had gone cold hours ago. He wasn\u2019t drinking it. He was holding it for warmth."
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":137,
"drafts_offered":163,
"tok_s":70.7,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv(file_path: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys being the header values\n and values being the corresponding cell values.\n \n Args:\n file_path: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(file_path, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_basic():\n \"\"\"Test basic CSV parsing with simple data.\"\"\"\n import tempfile\n import os\n \n # Create a temporary CSV"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral vaults below Elias\u2019s boots were heavy with salt and exhaustion as he climbed one hundred twelve steps to reach topmost gallery where his world began.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":30,
"drafts_offered":111,
"tok_s":34.7,
"text":"The wind didn\u2019t just blow at Blackwood Rock; it screamed through the gaps in his stone cottage like an angry ghost trying to get inside, yet finding no door left unlocked because Elias never let one stay that way for more than three seconds after entering or leaving during storm season. He had been doing this since 1987\u2014since before he even knew why\u2014and now it was muscle memory so deep into his bones and fingers\u2019 calloused skin texture patterns worn thin by years of gripping the same iron railing at midnight watch that if you asked him what time exactly, down to quarter-hour increments when fog rolled in thick enough where light beams bent around itself like old rope coiling tight on a winch drum spooling out slowly from below deck floor level storage locker number seven under stairs leading"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\u201d \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \u201cthe Sea\u201d or \"Ocean\" (terms often used interchangeably but distinct geographically) was never just an element within our planet\u2019s geography\u2014it became its engine itself\u2014a living system whose existence predates humanity by billions years yet continues shaping us today through climate regulation oxygen production food supply trade routes cultural identity ecological resilience economic development political boundaries scientific discovery technological innovation environmental crisis conservation efforts historical memory philosophical reflection poetic metaphor musical composition cinematic storytelling literary narrative visual art architectural inspiration culinary tradition spiritual symbolism psychological influence sociological transformation educational foundation medical necessity agricultural dependency industrial revolution urban planning infrastructural engineering legal framework regulatory policy international cooperation global governance planetary stewardship existential risk mitigation sustainable practice ethical consideration moral imperative social justice democratic participation civic engagement public health community welfare individual responsibility personal accountability collective action shared experience common understanding mutual respect reciprocal trust collaborative effort cooperative endeavor joint venture unified mission integrated strategy coordinated plan synchronized schedule aligned timeline concurrent execution simultaneous operation parallel processing sequential progression iterative refinement recursive improvement continuous adaptation dynamic adjustment flexible response adaptive behavior responsive conduct proactive initiative anticipatory foresight predictive modeling probabilistic assessment statistical analysis empirical observation experimental verification theoretical hypothesis conceptual abstraction logical deduction rational inference critical evaluation analytical reasoning systematic investigation methodical inquiry structured exploration comprehensive examination detailed scrutiny meticulous inspection thorough review exhaustive audit complete survey full census total enumeration absolute"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, mapped with veins like the roots of an ancient oak, gripped the cold iron railing.\n\nFor forty years, Elias had tended the light at the end of the world. Not the end of the map, but the end of the land. Here, the Atlantic threw itself against the granite cliffs with such violence that the spray reached the gallery, thirty feet above the rocks.\n\nHe checked the clock. 11:42 PM. The fog was thickening, a gray soup that swallowed the horizon. In the old days, he would have climbed the stairs to the lantern room, polished the Fresnel lens, and wound the clockwork mechanism that turned the great beam. But the automation had changed"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":135,
"drafts_offered":179,
"tok_s":81.6,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral hall outside my window at midnight on November third this year\u2014so I pulled up to check what was happening with that storm and why they were calling me out there when all we had left for tonight\u2019s shift is one last pass through our logs before dawn breaks.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":17,
"drafts_offered":83,
"tok_s":44.1,
"text":"The wind didn\u2019t just blow at Cape Harrow; it screamed through the gaps in its teeth, hungry and relentless for three centuries of stone salt-spray history that had long since forgotten to be polite or gentle with itself against anything smaller than God Himself was willing acknowledge as divine protection from drowning souls who were already lost before they ever touched water.\n\nElias knew this rhythm better he himself could hear it sometimes; his boots worn smooth by pacing the spiral stair\u2019s center-post groove where thousands upon countless steps had rubbed away texture until all that remained beneath sole leather now felt like polished bone carved out slowly but surely over time into something more hollow than wood ever really should be allowed become when used so often enough within such confined circular spaces meant only for vertical ascent down always descending again back upward never truly leaving"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element within our planet\u2019s geography but rather its most fundamental biological engine\u2014a living system whose existence predates humanity by billions years yet continues shaping every aspect from climate regulation through evolutionary biology all way down modern industrial supply chains today's global shipping lanes carry cargo across vast distances because they were once crossed only slowly via coastal hopping routes used since antiquity until steamships revolutionized travel during late nineteenth century when diesel engines replaced wind power entirely allowing container ships carrying thousands TEUs (Twenty-foot Equivalent Units) per voyage crossing entire Pacific basins without stopping at intermediate ports anymore unlike earlier sailing vessels needing favorable winds currents seasonal patterns dictating departure windows based upon monsoon calendars established centuries ago before technological advances changed those constraints fundamentally altering logistics networks worldwide thereby transforming economic structures globally while simultaneously creating ecological disruptions due invasive species transported unintentionally alongside legitimate goods causing cascading ecosystem collapses locally where non-native organisms outcompete native fauna disrupting food webs leading population declines affecting biodiversity indices measured scientifically using standardized metrics like Shannon-Wiener diversity index calculations applied consistently over decades revealing trends showing significant decreases post-introduction events correl strongly with introduction dates matching historical records documenting first sightings confirmed taxonomically classified formally according International Code Nomenclature rules"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, mapped with veins like the roots of an ancient oak, gripped the cold iron railing.\n\nFor forty years, Elias had tended the light at the end of the world. Not the end of the map, but the end of the land. Here, the Atlantic threw itself against the granite cliffs with such violence that the spray reached the gallery, thirty feet above the rocks.\n\nHe checked the clock. 11:42 PM. The fog was thickening, a gray soup that swallowed the horizon. In the old days, he would have climbed the stairs to the lantern room, polished the Fresnel lens, and wound the clockwork mechanism that turned the great beam. But the automation had changed"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":135,
"drafts_offered":179,
"tok_s":80.6,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral hall outside my window at midnight on November third this year\u2014so I pulled up to check what was happening with that storm and why they were calling me out there when all we had left for tonight\u2019s shift is one last pass through our logs before dawn breaks.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":17,
"drafts_offered":83,
"tok_s":43.3,
"text":"The wind didn\u2019t just blow at Cape Harrow; it screamed through the gaps in its teeth, hungry and relentless for three centuries of stone salt-spray history that had long since forgotten to be polite or gentle with itself against anything smaller than God Himself was willing acknowledge as divine protection from drowning souls who were already lost before they ever touched water.\n\nElias knew this rhythm better he himself could hear it sometimes; his boots worn smooth by pacing the spiral stair\u2019s center-post groove where thousands upon countless steps had rubbed away texture until all that remained beneath sole leather now felt like polished bone carved out slowly but surely over time into something more hollow than wood ever really should be allowed become when used so often enough within such confined circular spaces meant only for vertical ascent down always descending again back upward never truly leaving"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element within our planet\u2019s geography but rather its most fundamental biological engine\u2014a living system whose existence predates humanity by billions years yet continues shaping every aspect from climate regulation through evolutionary biology all way down modern industrial supply chains today's global shipping lanes carry cargo across vast distances because they were once crossed only slowly via coastal hopping routes used since antiquity until steamships revolutionized travel during late nineteenth century when diesel engines replaced wind power entirely allowing container ships carrying thousands TEUs (Twenty-foot Equivalent Units) per voyage crossing entire Pacific basins without stopping at intermediate ports anymore unlike earlier sailing vessels needing favorable winds currents seasonal patterns dictating departure windows based upon monsoon calendars established centuries ago before technological advances changed those constraints fundamentally altering logistics networks worldwide thereby transforming economic structures globally while simultaneously creating ecological disruptions due invasive species transported unintentionally alongside legitimate goods causing cascading ecosystem collapses locally where non-native organisms outcompete native fauna disrupting food webs leading population declines affecting biodiversity indices measured scientifically using standardized metrics like Shannon-Wiener diversity index calculations applied consistently over decades revealing trends showing significant decreases post-introduction events correl strongly with introduction dates matching historical records documenting first sightings confirmed taxonomically classified formally according International Code Nomenclature rules"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, mapped with veins like the roots of an ancient oak, gripped the cold iron railing.\n\nFor forty years, Elias had tended the light at the end of the world. Not the end of the map, but the end of the land. Here, the Atlantic threw itself against the granite cliffs with such violence that the spray reached the gallery, thirty feet above the rocks.\n\nHe checked the clock. 11:42 PM. The fog was thickening, a gray soup that swallowed the horizon. In the old days, he would have climbed the stairs to the lantern room, polished the Fresnel lens, and wound the clockwork mechanism that turned the great beam. But the automation had changed"
},
{
"name":"greedy code 200",
"tokens":200,
"repeats":0,
"drafts_accepted":135,
"drafts_offered":179,
"tok_s":80.5,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed against the glass of Lantern House 42, rattling like chains in an empty cathedral hall outside my window at midnight on November third this year\u2014so I pulled up to check what was happening with that storm and why they were calling me out there when all we had left for tonight\u2019s shift is one last pass through our logs before dawn breaks.<|im_end|>"
},
{
"name":"sampled presence=100 last_n=64 seed 3",
"tokens":160,
"repeats":0,
"drafts_accepted":17,
"drafts_offered":83,
"tok_s":44.3,
"text":"The wind didn\u2019t just blow at Cape Harrow; it screamed through the gaps in its teeth, hungry and relentless for three centuries of stone salt-spray history that had long since forgotten to be polite or gentle with itself against anything smaller than God Himself was willing acknowledge as divine protection from drowning souls who were already lost before they ever touched water.\n\nElias knew this rhythm better he himself could hear it sometimes; his boots worn smooth by pacing the spiral stair\u2019s center-post groove where thousands upon countless steps had rubbed away texture until all that remained beneath sole leather now felt like polished bone carved out slowly but surely over time into something more hollow than wood ever really should be allowed become when used so often enough within such confined circular spaces meant only for vertical ascent down always descending again back upward never truly leaving"
"text":"The ocean is not merely water; it has been, and remains to this day one giant machine for making life possible on Earth.\" \u2014 *David Attenborough* \n\nThis quote captures perhaps more than any other single sentence in human language: that which we call \"the Sea\" or \u201cOcean\u201d was never just an element within our planet\u2019s geography but rather its most fundamental biological engine\u2014a living system whose existence predates humanity by billions years yet continues shaping every aspect from climate regulation through evolutionary biology all way down modern industrial supply chains today's global shipping lanes carry cargo across vast distances because they were once crossed only slowly via coastal hopping routes used since antiquity until steamships revolutionized travel during late nineteenth century when diesel engines replaced wind power entirely allowing container ships carrying thousands TEUs (Twenty-foot Equivalent Units) per voyage crossing entire Pacific basins without stopping at intermediate ports anymore unlike earlier sailing vessels needing favorable winds currents seasonal patterns dictating departure windows based upon monsoon calendars established centuries ago before technological advances changed those constraints fundamentally altering logistics networks worldwide thereby transforming economic structures globally while simultaneously creating ecological disruptions due invasive species transported unintentionally alongside legitimate goods causing cascading ecosystem collapses locally where non-native organisms outcompete native fauna disrupting food webs leading population declines affecting biodiversity indices measured scientifically using standardized metrics like Shannon-Wiener diversity index calculations applied consistently over decades revealing trends showing significant decreases post-introduction events correl strongly with introduction dates matching historical records documenting first sightings confirmed taxonomically classified formally according International Code Nomenclature rules"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning black void, punctuated by whitecaps that looked like teeth.\n\nHe climbed the spiral stairs, his knees protesting with every step. The iron rungs were cold, slick with condensation that had leaked through the hairline cracks in the masonry. At the top, the lantern room hummed with the low, steady thrum of the rotating lens. It was a sound Elias had learned to sleep with, a heartbeat of the house.\n\nHe checked the clock. 11:42 PM. The light swept across the water, a slow, hypnotic arc. *Flash. Darkness. Flash. Darkness.*\n\nFor a moment, the beam caught something that wasn\u2019t water. A shape"
},
{
"name":"code",
"tok_s":92.9,
"hit_rate":0.8636411400016346,
"text":"```python\nimport csv\nfrom typing import List, Dict\n\n\ndef parse_csv(filepath: str) -> List[Dict[str, str]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header row\n and values from the corresponding row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(filepath, 'r', newline='') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n```<|im_end|>"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning cauldron of black ink and white foam, throwing itself against the granite base with the force of a battering ram.\n\nInside the lantern room, the air was thick with the smell of kerosene and old brass. Elias wiped his hands on his woolen trousers, leaving faint, oily smears on the fabric. He checked the clock. 11:42 PM. The rotation was due.\n\nHe reached for the crank handle, a heavy iron wheel worn smooth by decades of his own palms. With a grunt that matched the strain of his shoulders, he began to turn it. *Click. Click. Click.* The gears engaged, and the massive Fresnel lens began its slow"
},
{
"name":"code",
"tok_s":88.0,
"hit_rate":0.7011385968111186,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, the glass was vibrating like a plucked cello string. The sea, usually a rhythmic pulse against the granite base, was now a chaotic, frothing beast, throwing itself against the rocks with the force of a battering ram.\n\nElias climbed the spiral stairs, his hand gripping the cold iron railing. His knees ached, not from age, but from the constant vibration of the floor beneath his feet. He reached the lantern room, the heart of the lighthouse, and checked the clockwork mechanism. The great lens, a prism of Fresnel glass, rotated with a steady, hypnotic *whirrr-click, whirrr-click*. It was the only sound that mattered. The light swept out into the black void, cutting through"
},
{
"name":"code",
"tok_s":103.8,
"reps":[
97.6,
106.4,
107.3
],
"hit_rate":0.9984863945578232,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 524 windows; per window: wait for the GPU 6.456 ms, pool + plan 0.181 ms, host staging 0.581 ms, commit 0.206 ms",
"strata serve: stage 1: 524 windows; per window: wait for the GPU 14.872 ms, pool + plan 0.501 ms, host staging 0.000 ms, commit 0.387 ms",
"strata serve: stage 0: 593 windows; per window: wait for the GPU 6.479 ms, pool + plan 0.185 ms, host staging 0.515 ms, commit 0.207 ms",
"strata serve: stage 1: 593 windows; per window: wait for the GPU 15.053 ms, pool + plan 0.496 ms, host staging 0.000 ms, commit 0.390 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 18)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, the glass was vibrating like a plucked cello string. The sea, usually a rhythmic pulse against the granite base, was now a chaotic, frothing beast, throwing itself against the rocks with the force of a battering ram.\n\nElias stood at the top of the spiral staircase, his hand resting on the cold iron railing. He was not afraid. Fear was for the tourists who came in summer, wide-eyed and clutching their cameras. Elias was a keeper. He was a guardian.\n\nHe checked the logbook. The entry for the night was simple: *Wind, 60 knots. Visibility, zero. Light, operational.*\n\nHe walked down the stairs, his boots heavy on the metal grating. The smell of salt"
},
{
"name":"code",
"tok_s":107.4,
"reps":[
98.9,
111.3,
112.1
],
"hit_rate":0.9986286196529353,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header row\n and values from the corresponding row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with"
}
],
"stage_timing":[
"strata serve: stage 0: 533 windows; per window: wait for the GPU 6.226 ms, pool + plan 0.294 ms, host staging 0.561 ms, commit 0.225 ms",
"strata serve: stage 1: 533 windows; per window: wait for the GPU 14.156 ms, pool + plan 0.373 ms, host staging 0.000 ms, commit 0.371 ms",
"strata serve: stage 0: 599 windows; per window: wait for the GPU 6.314 ms, pool + plan 0.292 ms, host staging 0.501 ms, commit 0.225 ms",
"strata serve: stage 1: 599 windows; per window: wait for the GPU 14.366 ms, pool + plan 0.375 ms, host staging 0.000 ms, commit 0.374 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 19)",
"text":"The wind didn\u2019t just blow; it screamed. It clawed at the granite base of the tower, trying to pry it from the bedrock, and it howled through the narrow slits of the lantern room glass.\n\nElias Thorne was seventy years old, but he moved with the grace of a man who had spent fifty years listening to the sea. He knew the rhythm of the waves against the rocks below\u2014the *thud, thud, crash*\u2014and he knew when that rhythm changed. When the sea grew too quiet, or too violent, he knew a storm was coming, or a ship was in trouble.\n\nTonight, the storm was a Category 4. The rain lashed against the windows like gravel thrown by a giant. Elias stood in the center of the lantern room, his hand resting on the cold brass railing. He wasn\u2019t afraid. He was a guardian.\n\nHe checked the clockwork mechanism. It was old, a relic from 1"
},
{
"name":"code",
"tok_s":105.9,
"reps":[
103.2,
106.8,
107.7
],
"hit_rate":0.9981849807154201,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 537 windows; per window: wait for the GPU 6.685 ms, pool + plan 0.403 ms, host staging 0.593 ms, commit 0.215 ms",
"strata serve: stage 1: 537 windows; per window: wait for the GPU 13.928 ms, pool + plan 0.282 ms, host staging 0.000 ms, commit 0.359 ms",
"strata serve: stage 0: 606 windows; per window: wait for the GPU 6.764 ms, pool + plan 0.399 ms, host staging 0.528 ms, commit 0.216 ms",
"strata serve: stage 1: 606 windows; per window: wait for the GPU 14.108 ms, pool + plan 0.288 ms, host staging 0.000 ms, commit 0.362 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 20)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and shadow.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined his life for thirty years.\n\nElias was not a man of many words. He was a man of many tasks. His hands, mapped with scars from rope burns and broken glass, moved with a rhythm that predated his memory. He checked the clockwork mechanism of the Fresnel lens, ensuring the rotation was smooth, silent, and steady. *Tick. Tick. Tick.* The sound was the heartbeat of the coast.\n\n"
},
{
"name":"code",
"tok_s":106.7,
"reps":[
100.6,
110.5,
109.0
],
"hit_rate":0.998218711927068,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 527 windows; per window: wait for the GPU 6.934 ms, pool + plan 0.620 ms, host staging 0.530 ms, commit 0.245 ms",
"strata serve: stage 1: 527 windows; per window: wait for the GPU 13.912 ms, pool + plan 0.365 ms, host staging 0.000 ms, commit 0.359 ms",
"strata serve: stage 0: 595 windows; per window: wait for the GPU 7.031 ms, pool + plan 0.595 ms, host staging 0.472 ms, commit 0.245 ms",
"strata serve: stage 1: 595 windows; per window: wait for the GPU 14.051 ms, pool + plan 0.361 ms, host staging 0.000 ms, commit 0.361 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 21)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy-three years old, and he had been the keeper of the Blackwood Point Light for forty of those years. The other thirty-three had been spent in a small, drafty cottage on the mainland, waiting for the day when the sea would finally claim him.\n\nTonight, the storm was a Category 4. The waves weren\u2019t just crashing; they were climbing. They reached the gallery railing, ten feet above the base of the tower, spraying a fine mist of brine that coated Elias\u2019s glasses.\n\nHe climbed the spiral stairs, his knees protesting with every step. The iron steps were cold, slick with condensation. At the top, in the lantern room, the air was thin and sharp. The great Fresnel lens, a cathedral of glass and brass, stood silent. It was time to wake it.\n\n"
},
{
"name":"code",
"tok_s":106.4,
"reps":[
100.7,
110.3,
108.1
],
"hit_rate":0.9971798370756316,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 529 windows; per window: wait for the GPU 7.229 ms, pool + plan 0.561 ms, host staging 0.529 ms, commit 0.233 ms",
"strata serve: stage 1: 529 windows; per window: wait for the GPU 13.058 ms, pool + plan 0.253 ms, host staging 0.000 ms, commit 0.334 ms",
"strata serve: stage 0: 598 windows; per window: wait for the GPU 7.325 ms, pool + plan 0.552 ms, host staging 0.469 ms, commit 0.235 ms",
"strata serve: stage 1: 598 windows; per window: wait for the GPU 13.238 ms, pool + plan 0.260 ms, host staging 0.000 ms, commit 0.337 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 22)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning black void, swallowing the horizon, throwing white foam against the granite base like a tantrum.\n\nInside the lantern room, the air was thick with the smell of kerosene and old brass. Elias wiped his hands on his wool trousers, the fabric stiff with salt and grease. He checked the clockwork mechanism that turned the lens. It was a beast of a thing, a Fresnel lens of eight feet in diameter, a cathedral of glass and crystal that focused the beam of the lamp into a single, piercing needle of light.\n\nHe wound the weights. *Click, click, click.* The gears engaged. The massive lens began its slow, hypnotic rotation.\n\nFor a moment, the"
},
{
"name":"code",
"tok_s":108.4,
"reps":[
105.1,
109.0,
111.1
],
"hit_rate":0.9961698810290479,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 537 windows; per window: wait for the GPU 7.423 ms, pool + plan 0.953 ms, host staging 0.542 ms, commit 0.237 ms",
"strata serve: stage 1: 537 windows; per window: wait for the GPU 12.223 ms, pool + plan 0.236 ms, host staging 0.000 ms, commit 0.324 ms",
"strata serve: stage 0: 606 windows; per window: wait for the GPU 7.519 ms, pool + plan 0.904 ms, host staging 0.483 ms, commit 0.239 ms",
"strata serve: stage 1: 606 windows; per window: wait for the GPU 12.395 ms, pool + plan 0.242 ms, host staging 0.000 ms, commit 0.326 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 23)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and shadow.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined Elias\u2019s life for thirty years. He was not merely a keeper; he was a guardian of silence.\n\nHe checked the clock. 11:42 PM. The rotation of the great Fresnel lens was due. With a practiced hand, he engaged the brake on the clockwork mechanism. The heavy brass gears groaned, then began their slow, rhythmic turn. *Tick. Tick. Tick.* It was a heartbeat"
},
{
"name":"code",
"tok_s":109.5,
"reps":[
104.9,
111.0,
112.6
],
"hit_rate":0.9953796671318199,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 519 windows; per window: wait for the GPU 7.835 ms, pool + plan 0.767 ms, host staging 0.557 ms, commit 0.244 ms",
"strata serve: stage 1: 519 windows; per window: wait for the GPU 11.922 ms, pool + plan 0.232 ms, host staging 0.000 ms, commit 0.325 ms",
"strata serve: stage 0: 587 windows; per window: wait for the GPU 7.943 ms, pool + plan 0.744 ms, host staging 0.497 ms, commit 0.246 ms",
"strata serve: stage 1: 587 windows; per window: wait for the GPU 12.073 ms, pool + plan 0.238 ms, host staging 0.000 ms, commit 0.326 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 24)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and green.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined Elias\u2019s life for thirty years. He was a keeper of the light, but more importantly, he was a keeper of time.\n\nHe checked his pocket watch. 11:42 PM. The rotation of the great Fresnel lens was due. With a practiced, rhythmic *clunk-clunk-clunk*, he wound the clockwork mechanism. The massive lens began its slow, hypnotic turn, sweeping its beam across"
},
{
"name":"code",
"tok_s":109.7,
"reps":[
104.4,
112.7,
111.9
],
"hit_rate":0.9938670271352759,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 519 windows; per window: wait for the GPU 8.376 ms, pool + plan 1.030 ms, host staging 0.566 ms, commit 0.273 ms",
"strata serve: stage 1: 519 windows; per window: wait for the GPU 10.886 ms, pool + plan 0.214 ms, host staging 0.000 ms, commit 0.297 ms",
"strata serve: stage 0: 588 windows; per window: wait for the GPU 8.506 ms, pool + plan 0.985 ms, host staging 0.505 ms, commit 0.274 ms",
"strata serve: stage 1: 588 windows; per window: wait for the GPU 11.036 ms, pool + plan 0.219 ms, host staging 0.000 ms, commit 0.298 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 26)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, the glass was vibrating like a plucked cello string. The sea, usually a rhythmic pulse against the granite base, was now a chaotic, frothing beast, throwing itself against the rocks with the force of a battering ram.\n\nElias climbed the spiral stairs, his hand gripping the cold iron railing. His knees ached, not from age, but from the constant vibration of the floor beneath his feet. He reached the lantern room, the heart of the lighthouse, and checked the clockwork mechanism. The great lens, a prism of Fresnel glass, rotated with a steady, hypnotic *whirrr-click, whirrr-click*. It was the only sound that mattered. The light swept out into the black void, cutting through"
},
{
"name":"code",
"tok_s":112.5,
"reps":[
101.4,
112.1,
124.1
],
"hit_rate":0.9896636587366694,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv(file_path: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys being the header values\n and values being the corresponding cell values.\n \n Args:\n file_path: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(file_path, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n```<|im_end|>"
}
],
"stage_timing":[
"strata serve: stage 0: 527 windows; per window: wait for the GPU 8.765 ms, pool + plan 1.401 ms, host staging 0.593 ms, commit 0.281 ms",
"strata serve: stage 1: 527 windows; per window: wait for the GPU 9.893 ms, pool + plan 0.195 ms, host staging 0.000 ms, commit 0.276 ms",
"strata serve: stage 0: 578 windows; per window: wait for the GPU 8.908 ms, pool + plan 1.359 ms, host staging 0.559 ms, commit 0.284 ms",
"strata serve: stage 1: 578 windows; per window: wait for the GPU 10.025 ms, pool + plan 0.200 ms, host staging 0.000 ms, commit 0.278 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 28)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, the glass was vibrating like a plucked cello string. The sea, usually a rhythmic pulse against the granite base, was now a chaotic, frothing beast, throwing itself against the rocks with the force of a battering ram.\n\nElias stood at the top of the spiral staircase, his hand resting on the cold iron railing. He was not afraid. Fear was for the tourists who came in summer, wide-eyed and clutching their cameras. Elias was a keeper. He was a guardian.\n\nHe checked the logbook. The entry for the night was simple: *Wind, 60 knots. Visibility, zero. Light, operational.*\n\nHe walked down the stairs, his boots heavy on the metal grating. The smell of salt"
},
{
"name":"code",
"tok_s":109.6,
"reps":[
100.7,
114.1,
114.1
],
"hit_rate":0.9825370343977666,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 540 windows; per window: wait for the GPU 9.206 ms, pool + plan 1.794 ms, host staging 0.525 ms, commit 0.298 ms",
"strata serve: stage 1: 540 windows; per window: wait for the GPU 8.767 ms, pool + plan 0.173 ms, host staging 0.000 ms, commit 0.242 ms",
"strata serve: stage 0: 608 windows; per window: wait for the GPU 9.390 ms, pool + plan 1.733 ms, host staging 0.471 ms, commit 0.301 ms",
"strata serve: stage 1: 608 windows; per window: wait for the GPU 8.910 ms, pool + plan 0.178 ms, host staging 0.000 ms, commit 0.244 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 30)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning cauldron of black ink and white foam, throwing itself against the granite base with the force of a battering ram.\n\nInside the lantern room, the air was thick with the smell of kerosene and old brass. Elias wiped his hands on his woolen trousers, leaving faint, oily smears on the fabric. He checked the clock. 11:42 PM. The rotation was due.\n\nHe reached for the crank handle, a heavy iron wheel worn smooth by decades of his own palms. With a grunt that matched the strain of his shoulders, he began to turn it. *Click. Click. Click.* The gears engaged, and the massive Fresnel lens began its slow"
},
{
"name":"code",
"tok_s":108.7,
"reps":[
98.8,
112.2,
115.1
],
"hit_rate":0.9777186846317358,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 533 windows; per window: wait for the GPU 9.839 ms, pool + plan 2.192 ms, host staging 0.585 ms, commit 0.318 ms",
"strata serve: stage 1: 533 windows; per window: wait for the GPU 8.014 ms, pool + plan 0.156 ms, host staging 0.000 ms, commit 0.231 ms",
"strata serve: stage 0: 602 windows; per window: wait for the GPU 10.014 ms, pool + plan 2.111 ms, host staging 0.520 ms, commit 0.321 ms",
"strata serve: stage 1: 602 windows; per window: wait for the GPU 8.094 ms, pool + plan 0.160 ms, host staging 0.000 ms, commit 0.232 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 32)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy-three years old, and his hands were a map of the coastline he had guarded for fifty years. The knuckles were swollen, the skin cracked like dry earth, but they moved with a precision that no machine could replicate.\n\nHe stood on the gallery of the *Harrow Point* lighthouse, the iron railing cold against his palms. Below, the Atlantic churned in a frenzy, whitecaps exploding against the black rocks like fireworks. Above, the sky was a bruised purple, heavy with rain that hadn\u2019t yet fallen.\n\nElias checked the clock. 11:42 PM. The light was due.\n\nHe didn\u2019t look at the gauge. He didn\u2019t check the fuel levels. He knew. He always knew. The rhythm of the ocean, the pressure of the air, the subtle vibration"
},
{
"name":"code",
"tok_s":105.2,
"reps":[
87.8,
113.5,
114.2
],
"hit_rate":0.9109811738648947,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 382 windows; per window: wait for the GPU 15.155 ms, pool + plan 4.331 ms, host staging 0.656 ms, commit 0.458 ms",
"strata serve: stage 0: 455 windows; per window: wait for the GPU 15.790 ms, pool + plan 5.552 ms, host staging 0.688 ms, commit 0.467 ms",
"strata serve: stage 0: 524 windows; per window: wait for the GPU 16.064 ms, pool + plan 5.529 ms, host staging 0.606 ms, commit 0.472 ms",
"strata serve: stage 0: 593 windows; per window: wait for the GPU 16.305 ms, pool + plan 5.462 ms, host staging 0.540 ms, commit 0.475 ms"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning cauldron of black ink and white foam, throwing itself against the granite base with the force of a battering ram.\n\nInside the lantern room, the air was thick with the smell of kerosene and old brass. Elias wiped his hands on his woolen trousers, leaving faint, oily smears on the fabric. He checked the clock. 11:42 PM. The rotation was due.\n\nHe reached for the crank handle, a heavy iron wheel worn smooth by decades of his own palms. With a grunt that matched the strain of his shoulders, he began to turn it. *Click. Click. Click.* The gears engaged, and the massive Fresnel lens began its slow"
},
{
"name":"code",
"tok_s":88.2,
"hit_rate":0.7011385968111186,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning black void, punctuated by whitecaps that looked like teeth.\n\nHe climbed the spiral stairs, his boots heavy with salt and exhaustion. The iron steps groaned under his weight, a sound that had become so familiar to him that he could sleep to it. But tonight, he wasn\u2019t sleeping. He was working.\n\nThe lens was the heart of the lighthouse. A massive, rotating Fresnel lens, a cathedral of glass and brass that turned with a mechanical hum. Elias checked the clockwork mechanism. It was smooth, rhythmic, a heartbeat in the dark. But the light itself was failing.\n\nHe climbed higher, into the lantern room. The wind battered the glass panes,"
},
{
"name":"code",
"tok_s":87.5,
"reps":[
87.5
],
"hit_rate":0.7075461363025227,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 39 windows; per window: wait for the GPU 17.252 ms, pool + plan 8.473 ms, host staging 0.611 ms, commit 0.525 ms",
"strata serve: stage 0: 47 windows; per window: wait for the GPU 18.018 ms, pool + plan 8.616 ms, host staging 0.607 ms, commit 0.534 ms",
"strata serve: stage 0: 169 windows; per window: wait for the GPU 15.223 ms, pool + plan 5.717 ms, host staging 0.688 ms, commit 0.467 ms",
"strata serve: stage 0: 243 windows; per window: wait for the GPU 16.500 ms, pool + plan 7.316 ms, host staging 0.742 ms, commit 0.481 ms"
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the rough fabric scratching against his stubble. Outside the thick, salt-crusted glass of the lantern room, the Atlantic was a churning cauldron of black and white foam. Rain lashed against the panes in horizontal sheets, blurring the world into a smear of gray and shadow.\n\nInside, however, it was warm. The smell of kerosene and old wood hung heavy in the air, a scent that had defined his life for thirty years.\n\nElias was not a man of many words. He was a man of many tasks. His hands, mapped with scars from rope burns and broken glass, moved with a rhythm that predated his memory. He checked the clockwork mechanism of the Fresnel lens, listening for the faint, rhythmic *tick-tick-tick* that signaled the rotation was true. If the rhythm faltered, ships"
},
{
"name":"code",
"tok_s":108.9,
"reps":[
103.3,
113.7,
109.5
],
"hit_rate":0.9971795386830036,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 542 windows; per window: wait for the GPU 7.167 ms, pool + plan 0.572 ms, host staging 0.550 ms, commit 0.232 ms",
"strata serve: stage 1: 542 windows; per window: wait for the GPU 12.855 ms, pool + plan 0.250 ms, host staging 0.000 ms, commit 0.334 ms",
"strata serve: stage 0: 610 windows; per window: wait for the GPU 7.276 ms, pool + plan 0.559 ms, host staging 0.493 ms, commit 0.234 ms",
"strata serve: stage 1: 610 windows; per window: wait for the GPU 13.048 ms, pool + plan 0.258 ms, host staging 0.000 ms, commit 0.337 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"strata generate: layer split: CUDA1 holds its weights, session and head; 18.13 GiB free",
"strata generate: layer split auto: CUDA0 84 SMs at 2.81 GHz -> 0.31 ms per layer, 8.30 GiB for experts",
"strata generate: layer split auto: CUDA1 82 SMs at 1.86 GHz -> 0.48 ms per layer, 14.14 GiB for experts",
"strata generate: layer split auto: K=22 - predicted 19.7 ms per decode window; the caches hold 11298 of 12288 profiled pairs (~99.7% of the routed mass)",
"text":"The wind didn\u2019t just blow; it screamed. It tore through the iron gallery of the Blackwood Point Lighthouse, rattling the glass panes like a prisoner shaking its bars.\n\nElias Thorne was seventy years old, and he had been the keeper for forty of them. He knew every creak of the spiral staircase, every draft that slipped under the door, and every shift in the barometric pressure that preceded a storm. He was not merely a worker; he was the heartbeat of the coast.\n\nIt was November, the month of grey seas and shorter days. The fog had rolled in thick, a wall of white so dense that even the beam of the Fresnel lens seemed to struggle against it. Elias stood in the lantern room, his hands roughened by salt and rope, adjusting the rotation mechanism. The great lens, a cathedral of glass and brass, began its slow, rhythmic sweep. *Flash. Dark. Flash. Dark.*\n\nHe checked the log"
},
{
"name":"code",
"tok_s":105.1,
"reps":[
98.4,
109.9,
107.0
],
"hit_rate":0.9949900793650793,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 492 windows; per window: wait for the GPU 15.016 ms, pool + plan 0.297 ms, host staging 0.517 ms, commit 0.404 ms",
"strata serve: stage 1: 492 windows; per window: wait for the GPU 5.775 ms, pool + plan 0.854 ms, host staging 0.001 ms, commit 0.210 ms",
"strata serve: stage 0: 561 windows; per window: wait for the GPU 15.257 ms, pool + plan 0.306 ms, host staging 0.455 ms, commit 0.408 ms",
"strata serve: stage 1: 561 windows; per window: wait for the GPU 5.852 ms, pool + plan 0.820 ms, host staging 0.001 ms, commit 0.210 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"strata generate: layer split: CUDA1 holds its weights, session and head; 9.98 GiB free",
"strata generate: layer split auto: CUDA0 82 SMs at 1.86 GHz -> 0.48 ms per layer, 16.45 GiB for experts",
"strata generate: layer split auto: CUDA1 84 SMs at 2.81 GHz -> 0.31 ms per layer, 5.99 GiB for experts",
"strata generate: layer split auto: K=30 - predicted 20.8 ms per decode window; the caches hold 10684 of 12288 profiled pairs (~99.5% of the routed mass)",
"text":"The wind didn\u2019t just blow; it screamed. It tore through the iron gallery of the Blackwood Point Lighthouse, rattling the glass panes like a prisoner shaking its bars.\n\nElias Thorne was seventy years old, and he had been the keeper for forty of them. He knew every creak in the spiral staircase, every draft that slipped under the door, and every shift in the weather that the barometer couldn\u2019t quite predict. He knew that when the fog rolled in thick enough to swallow the horizon, the world ceased to exist. There was only the light, the stone, and him.\n\nIt was November, the month of storms. The sea was a churning cauldron of grey and white, throwing itself against the granite base with the force of a battering ram. Elias stood in the lantern room, his hands rough as sandpaper, adjusting the massive Fresnel lens. The rotation mechanism groaned, a sound like a ship\u2019s anchor chain being h"
},
{
"name":"code",
"tok_s":89.6,
"reps":[
81.0,
104.8,
83.0
],
"hit_rate":0.9975041387598007,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 2: 522 windows; per window: wait for the GPU 12.405 ms, pool + plan 0.242 ms, host staging 0.006 ms, commit 0.375 ms",
"strata serve: stage 0: 590 windows; per window: wait for the GPU 7.387 ms, pool + plan 0.549 ms, host staging 0.517 ms, commit 0.248 ms",
"strata serve: stage 1: 590 windows; per window: wait for the GPU 3.071 ms, pool + plan 0.011 ms, host staging 0.000 ms, commit 0.606 ms",
"strata serve: stage 2: 590 windows; per window: wait for the GPU 12.592 ms, pool + plan 0.250 ms, host staging 0.006 ms, commit 0.377 ms"
],
"engine_lines":[
"strata generate: layer split across 3 GPUs: CUDA0, then CUDA1,2 (split auto)",
"strata generate: layer split: CUDA2 holds its weights, session and head; 18.13 GiB free",
"strata generate: layer split auto: CUDA0 84 SMs at 2.81 GHz -> 0.31 ms per layer, 8.30 GiB for experts",
"strata generate: layer split auto: CUDA1 68 SMs at 1.54 GHz -> 0.69 ms per layer, 13.50 GiB for experts",
"strata generate: layer split auto: CUDA2 82 SMs at 1.86 GHz -> 0.48 ms per layer, 14.14 GiB for experts",
"strata generate: layer split auto: K=22,23 - predicted 19.9 ms per decode window; the caches hold 11298 of 12288 profiled pairs (~99.7% of the routed mass)",
"text":"The wind didn\u2019t just blow; it screamed. It tore through the iron gallery of the Blackwood Point Lighthouse, rattling the glass panes like a prisoner shaking its bars.\n\nElias Thorne was seventy years old, and he had been the keeper for forty of them. He knew every creak in the spiral staircase, every draft that slipped under the door, and every shift in the weather that the barometer couldn\u2019t quite predict. He knew that when the fog rolled in thick enough to swallow the horizon, the world ceased to exist. There was only the light, the stone, and him.\n\nIt was November, the month of storms. The sea was a churning cauldron of grey and white, throwing itself against the granite base with a violence that made the floorboards tremble. Elias stood by the lantern room, his hands wrapped around a mug of coffee that had gone cold hours ago. He wasn\u2019t drinking it. He was holding it for warmth."
},
{
"name":"code",
"tok_s":111.1,
"hit_rate":1.0,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv(file_path: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys being the header values\n and values being the corresponding cell values.\n \n Args:\n file_path: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(file_path, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_basic():\n \"\"\"Test basic CSV parsing with simple data.\"\"\"\n import tempfile\n import os\n \n # Create a temporary CSV"
}
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias had lived in the tower for thirty years, and he knew the difference between a gale and a hurricane by the way the glass rattled in its frame. Tonight, it was the latter. The sea was a churning cauldron of black ink and white foam, throwing itself against the granite base with the force of a battering ram.\n\nInside the lantern room, the air was thick with the smell of kerosene and old brass. Elias wiped his hands on his woolen trousers, leaving faint, oily smears on the fabric. He checked the clock. 11:42 PM. The rotation was due.\n\nHe reached for the crank handle, a heavy iron wheel worn smooth by decades of his own palms. With a grunt that matched the strain of his shoulders, he began to turn it. The massive Fresnel lens, a cathedral of glass and crystal, began its slow, hypnotic sweep"
},
{
"name":"code",
"tok_s":101.8,
"hit_rate":0.991767530342175,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"text":"The wind didn\u2019t just blow; it screamed. It clawed at the granite base of the tower, trying to pry the lighthouse from its bedrock. Inside, however, the air was still, smelling faintly of kerosene and old paper.\n\nElias had been the keeper of the Point for thirty-two years. He knew every creak in the floorboards, every draft that slipped through the window seals, and exactly how much oil the great Fresnel lens needed to turn its slow, hypnotic rotation.\n\nIt was 2:00 AM. The storm was at its peak. The sea was a chaotic black mirror, whipped into whitecaps that looked like jagged teeth.\n\nElias climbed the spiral stairs, his hand gripping the cold iron railing. His boots, heavy with salt and wear, thumped a steady rhythm against the stone. *Thump. Thump. Thump.*\n\nAt the top, the lantern room hummed with the vibration of"
},
{
"name":"code",
"tok_s":86.4,
"hit_rate":0.9798276439089693,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 16)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy-three years old, and he had been the keeper of the Blackwood Point Light for forty of those years. The other thirty-three had been spent in a hospital bed, waiting for the end.\n\nEvery night, Elias climbed the one hundred and twelve steps of the spiral staircase. His knees, worn smooth by the friction of his own boots over decades, protested with a sharp ache. He counted them. *One, two, three\u2026* The rhythm was his heartbeat. If he lost the count, he lost the light.\n\nAt the top, the lantern room hummed with the low, electric thrum of the Fresnel lens. It was a cathedral of glass and brass, a machine that had never failed him. Not once. Not in four decades.\n\nHe polished the brass railing with a rag that smelled of lemon and"
},
{
"name":"code",
"tok_s":104.3,
"hit_rate":0.9842543225870746,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split 22)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his woolen collar up to his chin, the fabric stiff with salt and ice. Outside the thick glass of the lantern room, the Atlantic was a churning cauldron of black and white, the waves crashing against the granite base with a violence that shook the very foundations of the tower. It was November, 1924, and the storm had been raging for three days.\n\nElias was not a man who feared the dark. He was a man who feared the silence.\n\nHe checked the clock on the wall. 11:42 PM. The rotation of the great Fresnel lens was steady, a rhythmic *whirrr-click, whirrr-click* that seemed to sync with his own heartbeat. He had been doing this for forty years. His hands were mapped with scars from rope burns and broken glass, his eyes permanently squinted against the glare of the bulb."
},
{
"name":"code",
"tok_s":93.4,
"hit_rate":0.990286044973545,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"text":"The wind didn\u2019t just blow; it screamed.\n\nElias pulled his wool coat tighter around his shoulders, the fabric stiff with salt and age. He was seventy years old, but he moved with the grace of a man who had spent decades on a narrow spiral staircase. His hands, rough as sandpaper, knew every bolt, every lever, and every creak in the iron bones of the tower.\n\nFor forty years, Elias had been the guardian of the Point. Not just of the light, but of the silence between the waves.\n\nIt was November. The sky was a bruised purple, heavy with rain that hadn\u2019t yet fallen. The sea was a churning cauldron of grey and white, throwing itself against the granite base with a violence that shook the floorboards beneath Elias\u2019s boots.\n\nHe climbed the stairs. One hundred and twelve steps. He counted them by feel, his left hand trailing along the cold iron railing. At the top, the air was"
},
{
"name":"code",
"tok_s":58.0,
"reps":[
58.0
],
"hit_rate":0.9700542735219787,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv_to_dicts(filepath: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys from the header and values from the row.\n \n Args:\n filepath: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result: List[Dict[str, Any]] = []\n with open(filepath, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_to_dicts():\n import tempfile\n import os\n \n # Create a temporary CSV file\n with tempfile.NamedTemporaryFile"
}
],
"stage_timing":[
"strata serve: stage 0: 168 windows; per window: wait for the GPU 9.683 ms, pool + plan 0.509 ms, host staging 0.714 ms, commit 0.320 ms",
"strata serve: stage 1: 168 windows; per window: wait for the GPU 25.253 ms, pool + plan 0.881 ms, host staging 0.000 ms, commit 0.559 ms",
"strata serve: stage 0: 242 windows; per window: wait for the GPU 9.860 ms, pool + plan 0.550 ms, host staging 0.750 ms, commit 0.328 ms",
"strata serve: stage 1: 242 windows; per window: wait for the GPU 26.029 ms, pool + plan 0.980 ms, host staging 0.000 ms, commit 0.571 ms"
],
"engine_lines":[
"strata generate: layer split across 2 GPUs: CUDA0, then CUDA1 (split auto)",
"text":"The wind didn\u2019t just blow; it screamed. It tore through the iron gallery of the Blackwood Point Lighthouse, rattling the glass panes like a prisoner shaking its bars.\n\nElias Thorne was seventy years old, and he had been the keeper for forty of them. He knew every creak in the spiral staircase, every draft that slipped under the door, and every shift in the weather that the barometer couldn\u2019t quite predict. He knew that when the fog rolled in thick enough to swallow the horizon, the world ceased to exist. There was only the light, the stone, and him.\n\nIt was November, the month of storms. The sea was a churning cauldron of grey and white, throwing itself against the granite base with a violence that made the floorboards tremble. Elias stood by the lantern room, his hands wrapped around a mug of coffee that had gone cold hours ago. He wasn\u2019t drinking it. He was holding it for warmth."
},
{
"name":"code",
"tok_s":69.7,
"reps":[
69.7
],
"hit_rate":1.0,
"text":"```python\nimport csv\nfrom typing import List, Dict, Any\n\n\ndef parse_csv(file_path: str) -> List[Dict[str, Any]]:\n \"\"\"\n Parse a CSV file into a list of dictionaries.\n \n Each dictionary represents a row, with keys being the header values\n and values being the corresponding cell values.\n \n Args:\n file_path: Path to the CSV file.\n \n Returns:\n A list of dictionaries, one per row (excluding the header).\n \"\"\"\n result = []\n with open(file_path, 'r', newline='', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n result.append(dict(row))\n return result\n\n\n# Tests\ndef test_parse_csv_basic():\n \"\"\"Test basic CSV parsing with simple data.\"\"\"\n import tempfile\n import os\n \n # Create a temporary CSV"
}
],
"stage_timing":[
"strata serve: stage 2: 165 windows; per window: wait for the GPU 13.452 ms, pool + plan 0.162 ms, host staging 0.000 ms, commit 0.328 ms",
"strata serve: stage 0: 234 windows; per window: wait for the GPU 6.235 ms, pool + plan 0.145 ms, host staging 0.704 ms, commit 0.214 ms",
"strata serve: stage 1: 234 windows; per window: wait for the GPU 15.592 ms, pool + plan 0.246 ms, host staging 0.000 ms, commit 0.657 ms",
"strata serve: stage 2: 234 windows; per window: wait for the GPU 13.916 ms, pool + plan 0.177 ms, host staging 0.000 ms, commit 0.335 ms"
],
"engine_lines":[
"strata generate: layer split across 3 GPUs: CUDA0, then CUDA1,2 (split auto)",
Status: preliminary; prompt set and harness guidance requested upstream.
## Hardware
| Component | Guest-visible details |
|---|---|
| CPU | AMD EPYC 7453, 20 vCPU, AVX2; AVX-512 not exposed |
| RAM | 165 GiB visible to Ubuntu 24.04.5 KVM guest; physical DIMM layout and bandwidth not verified |
| GPUs | 2x RTX 3090 24 GB; PCIe Gen4 x16 maximum reported per card; driver 595.91.07 |
| Power limit | 280 W configured on each GPU; defaults are 350 W / 420 W, max-adjustable limits 366 W / 450 W |
| Storage | 541 GiB ext4 guest data disk; host storage layout not verified |
The power values above are configured caps, not measured workload draw. Idle draw during the hardware capture was approximately 35 W / 29 W.
## Method
Native Linux setup, local build from v0.1.26, `--prefill auto`, shipped expert profile, automatic expert cache, int8 KV, MTP on, images off, experimental speed projection off. The GPU power limit was 280 W per card. Native runs used a systemd service with `LimitMEMLOCK=infinity`; other llama-swap model containers were stopped.
The local API harness used the public Strata source tree as its prompt corpus, with generated tasks, `temperature: 0`, `reasoning_effort: none`, and `max_tokens: 256`. It is not the upstream code-agent prompt set or harness. Runs use unique request IDs and no conversation reuse in speed mode. Three runs per speed cell are summarized by medians; actual `usage.prompt_tokens` are recorded because nominal context labels can drift. The first decode run is also shown. Several requests ended early on EOS, so actual generated counts are in the raw JSON.
No calibration comparison, upstream prompt comparison, `STRATA_SPLIT_TIMING` stage timing, or under-load power-draw capture has been done. Treat all prompt/decode comparisons with upstream as indicative only until the requested upstream method is available.
## Results
See [`matrix.md`](matrix.md) for summary tables and [`matrix.json`](matrix.json) for all successful run records. Per-run source JSON, logs, the local harness, and scrubbed hardware facts are in [`data/`](data/).
Highlights:
- IQ3_XXS on one 3090: 1K–262K cells, three runs each; 262K prompts were 232,183 / 259,589 / 261,068 tokens and fit within the 262,144 context.
- Coder IQ1_M at 32K: single-GPU median decode 66.8 tok/s; dual-GPU auto (K=19) median 98.6 tok/s.
- Fixed Coder K sweep: K=22 was the highest observed three-run median (103.3 tok/s); independently generated prompts mean this is not a paired ranking.
- IQ3_XXS and IQ3_S dual-GPU auto both selected K=24 at 32K.
- IQ3_XXS and IQ3_S needles passed 3/3 at 8K / 16K / 28K on the tested single/dual configurations; two-turn checkpoint reuse passed for both.
## Limits and follow-up
- Upstream prompts/harness are not yet available; all current speed data uses a local source-tree corpus.
- The guest exposes one NUMA node and 165 GiB RAM, but physical DIMM topology and memory bandwidth are unknown.
- PCIe x16 is the reported maximum link width; the engine probes were about 23–26 GB/s. Verify PCIe generation/width under sustained load before claiming a direct x16-vs-x4 comparison.
- Coder dual-GPU needle/two-turn checks and dual-GPU timing breakdowns are not run yet.
- K=18 has one successful run at 83.1 tok/s; a repeated startup failed while uploading a dense weight with an out-of-memory error, so it is not a three-run median.
- Calibration/default comparisons and IQ3_S single-GPU results remain outstanding.
"answer_head":"Looking at the last file above, `docs/AMD_HIP_PERFORMANCE.md`, it documents the performance evidence and correctness limits for the experimental AMD HIP backend targeting gfx1100 (RX 7900 XTX). It records:\n\n- The exact build/runtime configuration used for the benchmark (env vars, CLI flags, model pa",
Loaded 100 of 3076 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Niko1221