I/O path only: the same bytes are read, only HOW the experts outside the RAM copy are read (unbuffered or through the file cache) is chosen differently.
file_tier_unbuffered took its `avail` from cgroup_available_bytes, which credits the group's whole `file` counter as reclaimable. On a RAM-budget
config that credit includes pages the engine holds itself: P2003 (RTX 3060, 46 GB, 36 GiB group), UD-IQ4_XS-v31, 22 GiB RAM copy: the re-check read 18.7 GiB
available when the group had 11 GiB left (limit 36, current 24.8). With held = 0.52 >= 0.5 the timed drive probe was skipped and the file tier switched to the file
cache (direct reads 7.9 GB/s vs 2.5 GB/s cached): prompts 2.5-3x slower (pp2k 85 vs 273 tok/s, pp8k 195 vs 580), decode 10-14% slower. The one start that read
12.0 GiB kept the fast mode, which is why the first start of a session was sometimes fast.
Now the smaller of that figure and the conversation cache's sampler (only the clean inactive file cache is credited, the group's own limit, no operator
cap) decides. The --memory-limit-mib cap is not applied: it bounds the engine's own allocations, not the page cache (counting it sent IQ3_S unbuffered and cut its decode
2-3x). A flip now prints once: "strata: expert reads now go through the file cache (X GiB of RAM free for them, Y GiB limit)"; the technical line is behind STRATA_TRACE.
STRATA_UNBUFFERED_LOAD still forces either mode.
Measured on P2003 (build wip11, n >= 3 starts per arm, interleaved fresh engines, I/O-quiet before each start, memory watch on, floor 14336 MiB):
v31 unforced: main flipped in 5 of 6 starts, this change 0 of 9 (6 + 3): pp2k 85-94 -> 263-273 tok/s, pp8k 187-198 -> 580-590, code_py decode +14 to +20%;
v31 DFlash2 vs MTP in the kept state (3 rounds): DFlash2 +2 to +14% per kind (json_extract -3.5%);
IQ3_S: the file cache stays (9 of 9 starts), decode level with main (code_py 17.0 vs 15.6 tok/s, json_extract 14.2 vs 13.4, page cache evicted before every start);
low RAM (6-7 GiB ballast, budget 16 GiB): no regression, MemAvailable minimum 14911 MiB;
memory minimum with the unbuffered state 15.6-15.8 GB (floor 14.3 GB), no arm stopped.
Not P1.8: main and the P1.8 build flip alike.
(cherry picked from commit be3831e9d944fb44ac36d176f4764e30df0f9f6b)
native_w11_kernel takes a Q5_0 down block (d, qh, qs[16]; 22 bytes): the 5-bit codes are rebuilt with the qh bits and kept as the signed int8 q - 16 (no minimum
term), the scale is d. d_covered_on() says yes for T_Q5_0 only on gfx12 with the switch; the layer otherwise leaves the fused path for the gather + FP16
dequantize + hipBLASLt route as before. hip_prefill_fused_iq gets the pairs IQ3_XXS / Q5_0 and IQ4_XS / Q5_0 (reference-checked: the build has no Q5_0 MMQ).
R9700 (p620, v31, DFlash2, same executable with and without the switch, 3 rounds x 2 reps): prompt 2K +44%, 8K +40%, 32K +13%; decode unchanged.
First-token KL against the FP16 route (self-variation 0): 2K mean 0.016, max 0.068, top-1 24/24. Off by default in this commit.
(cherry picked from commit 97c3033f566d27ea1d741d7b893ac13e0f4e8526)
"strata: RDNA4 (gfx12): prompt chunks of 30,000+ tokens use the chunked GDN scan (STRATA_GDN_CHUNKED=0 turns it off)" is printed once at start with the
other gfx12 default lines (owner's wording, 11-06): only on gfx12, only while STRATA_GDN_CHUNKED is unset and STRATA_GFX12_DEFAULTS is not 0.
The gate follows the gfx12 defaults' one off switch, so the line is true whenever it is printed.
(cherry picked from commit b2d1680a864195383697559edb9b6cf8180b9a19)
STRATA_GDN_CHUNKED unset on a gfx12 card (RDNA4): the FP16 matrix-core chunked recurrence runs for prompt chunks of 30,000 tokens or more
(--prefill auto takes chunks up to 32,768 there, so a 32K prompt is one chunk of about 31,996 rows). Shorter chunks keep the kernels they
ran, so every prompt up to 20K is bit for bit what it was. STRATA_GDN_CHUNKED=0 turns it off, =1 is the old opt-in (every chunk of 64
tokens or more), STRATA_GDN_CHUNKED_MIN=N moves the line. Owner OK 11-06.
R9700 (p620, IQ3_S, DFlash2, interleaved pairs against the same tree without the change, 5 rounds x 2 reps): prompt 32K +6.1% (5/5),
64K +6.7% (5/5). First-token KL against the unchanged build (self-variation 0.00000): all 24 prompts of 2K, 8K, 20K and the 8K agent turn
identical; 32K 0.00005, 32K+ 0.00004, 64K 0.00086; top-1 equal on all 30 prompts. No log line yet (wording is with the coordinator).
(cherry picked from commit 7b579367aaa56978232cb60ef405987a199d736f)
In peer mode the adaptive overlap was refused, so every decode window first waited for the previous round's copies. With the overlap the peer
tier issues its copies in layer order, signals each layer's landing the way the primary does (a stream-ordered D2H copy of the layer number into
a pinned int, or host functions), its residency table takes the swaps at the window's start (flip_pending), and a layer waits for a tier's copies
only when that tier swapped in it (AdaptGate: peer_landed, peer_has, per-tier last). The wait before a window clears the peer's marks with the
primary's. Without the variable nothing changes.
Ported from #2720 (bulhenvaster), rebased by hand onto the current gate (the watchdog label stays). Not ported: STRATA_PEER_LAUNCH_IN_POOL
(the author measured no gain on his build), the new decode timing line and the documentation paragraph.
(cherry picked from commit e78b64a7348d685dfd11799fdb395661f2120436)
With --peer-device the prompt path never gave the CPU pool a share (cpu_maybe required !m.pp), so a short read at long context streamed every
expert neither card holds over the primary's link while the CPU sat idle. With the switch the share runs beside the peer: it takes only experts
neither card computes, its rows go between the primary's and the peer's block (the peer's stays the tail its combine reads from back_at), and the
staged chunks go up to cpu_share_max()'s 3072 tokens. Without the switch nothing changes.
Ported from #2721 (bulhenvaster), rebased by hand onto the current prompt path (the default-armed share's limit and the routed-stream gate).
(cherry picked from commit c8b6c43c1f2653636d15fe14abf53112357570aa)
Decode reads missed expert blobs from host RAM three ways; the PCIe share's
blobs cross into VRAM anyway and were dropped after the window. With the
switch set, the share takes each layer's most-routed misses and a fetched
expert routed enough more than the layer's least-routed resident expert that
the window does not route is copied straight into that expert's slot
(fetch_blobs gets an optional dst2 target per blob). =2 leaves the tier's own
swaps out of a layer where an admission ran. STRATA_ADAPT_FETCH_VERIFY=1 reads
every admitted slot back. Off, with a log line saying why, where it cannot run.
Unset changes nothing.
Ported from PR 2964 (sergqwer) onto f-0216-core; the strata generate verify
loop was merged by hand (main's speculative-q lines kept) and the DETAILS.md
note became a row of the opt-in switch table.
l620 (RTX PRO 5000, IQ3_S, 3,896-slot cache, --pcie-frac 0.25): decode =2 +10.2%,
=1 +5.0% (4 rounds); token forcing over 14,167 tokens: mismatch 2.73% vs 2.53% /
2.53% for two unset arms (+0.19 pt, 90% CI -0.02..+0.42), PASS.
(cherry picked from commit b2bfaef906acee4f86bdf9f052aee6a4d295fb4e)
Prompt-path block scores in single-pass TF32 with a per-block error bound,
3xTF32 rescoring only near each query's top-k cut-off, so qsa_block_topk
returns the same cells in the same order, ties included. Off unless the
variable is set; says once where it cannot run (HIP, Metal, SYCL, layer-major
path). Adds qsa_two_pass_test, qsa_two_pass_status_test and the GPU parity
test qsa_two_pass_parity.
Ported from PR 4119 (bitaccesscomau); the log line of the per-prompt counters
is shortened without dropping a field.
l620 (sm_120, IQ3_S), greedy ids identical off vs on in every pair:
prompt 128K +1.65% (n=12), 256K +4.10% (n=8), 32K -0.14% (noise).
(cherry picked from commit bbf7c55f1351d913233250fe7869642f7c2b98a1)
drafting.CLASS_DEFAULTS: one removable entry per set, read from the signals the confidence already reads (a CUDA card whose engine slots are under the last-but-one CONF_STEPS step).
IQ3_S: on P2003 the file cache makes the MTP faster than DFlash2 by 12-44% (RTX2 t4b). UD-IQ4_XS-v31: DFlash2 beats the MTP by about 6.5% (t8, 3 starts; DF-VAL forced-fast tuned 0.8 x1.037).
The v31 entry DEPENDS ON RTX2's FILE-TIER FIX (unbuffered reads) BEING IN THE SAME RELEASE; without it remove that entry.
Anything chosen by name wins. Resident CUDA, Strix, gfx1201, Arc and Metal entries and the conf tables are unchanged. The sm_86 support line reads the set: UD-IQ4_XS-v31 gets the existing
'faster on some kinds of text and slower on others' line, every other model keeps 'slower'.
(cherry picked from commit 42412d42ff835d6a66a78fbf6d119911908eda1b)
One more SET_DEFAULTS line for the measured cell (DF-VAL, setup's config, budget 38, 3 rounds: tuned 0.35 against stock 0.35 x1.046, 3 of 3 rounds, every kind 3 of 3).
(cherry picked from commit c2b801157e71a503da859b66baff12fff28cd299)
The SYCL 0.2.11 engine reads only bf16 selector codebooks, so Arc gets its own tuned file (qwen3.8-flash-next-dflash2.strata1.q4_0-bf16sel.pack, 522,458,112 bytes, own SHA-256 pin) through
SetDefault.pack. Only IQ3_XXS, the set measured on the A750 (tuned 0.6 x1.054, 3 of 3 rounds); other Arc sets keep the stock pack. SYCL never gets the selq4 file, CUDA / HIP / Metal never the
bf16sel one. NOTICE names the second file and no longer overclaims about the text.
(cherry picked from commit 6111c583b70e3c88a3d54039e13d76b19f15742d)
11 of 50 engines on 0.2.11 never got past the prompt on this card class. Only Auto changes: DFlash2 chosen by name, a saved choice and the person's own options still win; gfx1151, other AMD, CUDA, Metal and SYCL and the confidence tables are unchanged. The support line under the Drafting dropdown says it in plain words (app catalog entry in all nine languages).
(cherry picked from commit 8b4398743aa67368d72277310bbe8093bbe4923c)
SET_DEFAULTS entries carry a card condition (tuned_applies: the CONF_STEPS first-step slots or the Spark's shared memory for CUDA, gfx1151 for HIP); no entry names a conf, so
conf_for / CONF_STEPS / BACKEND_CONF run as before and only the pack changes. Enabled only for the cells measured at the shipped conf (CUDA-resident IQ3_S and UD-IQ4_XS-v31, Strix IQ3_S, UD-Q4_K_XL, UD-IQ4_XS).
12 GB and 24 GB cards, several cards, gfx1201, other AMD and SYCL never fetch or use it. Fetch keeps tuned, then stock, then MTP; silent offline; feed backends are asked per start's backend.
(cherry picked from commit f096dcd750b5928ace6f9eca1e2936b9e35578d4)
EXPERT_WINDOW_SLOTS (2 * MAXT * TOPK) is the slots a size pool must hold for one window's picks to have a place; ExpertCache's check, the residency
planner's floor and the arena's keep all read it. expert_residency_test now also covers the GSQ-RCO IQ3_S set's real pools (7, two of them a single
layer), a sweep of RAM targets against a copy of the old global ranking as the negative control, a refusal beyond what the floors allow, and that
the card's sliced 16-byte-padded strides partition the layers like the whole-expert bytes do. Class E (placement only).
Mixed-quant models can fail to load in the split engine after cold-expert offload: the global residency ranking can leave one size pool below the
2 * MAXT * TOPK slots Up::expert_cache() needs even while other pools can still give up experts. Group experts by both the gate/up and the
down byte strides, keep the cold-first ranking but skip eviction from a pool that has reached the window floor, count only the extra evictions
that really happen, and say so before the model is uploaded when the request cannot be placed.
Ported from #5158 (midhatn) unchanged; the tests and the engine check follow in the next commit.
With --trim-stage-weights every later stage's HC_Q8 loop over the GGUF copied the rows of all 48 layers and read its own. The loop now skips a
tensor outside the stage's layer range, the filter the upload loop above it and the bf16-pack path below it already use. The Q8_0 rows copied
and the bytes not copied are recorded in the run manifest (STRATA_MANIFEST). STRATA_HC_Q8_ALL_STAGES=1 copies every layer as before.
No value changes: a stage never reads another stage's layers; without a trim the range is unset and nothing is skipped. Class E.
Co-authored-by: havok2-htwo <[email protected]>
SET_DEFAULTS is the one per-(backend, set) table (Metal only; IQ3_S stays on the MTP). Setup, strata launch and strata drafting fetch fetch the tuned pack through the signed feed
(pinned size and SHA-256, resumable, silent offline, the standard drafter and then the MTP as fallbacks). Apache-2.0 licence and the modified-by-Strata notice ship in
data/dflash2-tuned and are copied beside the file. The Metal 'not tested' line is replaced for these sets.
Ported from #4871 (cctan): src/prefill/moe_iq4_w4a8.cu, tried first by mmq::Context::run; unset, nothing changes.
Changes on top of the PR: the "would MMQ split this launch along K" test is now mmq::launch_splits_k, which reads the
device's real MMQ configs (tile width, stream_k flag, tile efficiency) instead of an Ampere table, so the kernel declines
exactly the launches whose sums MMQ adds in another order, on any architecture; sm_120 joins sm_80..sm_88 (bit-for-bit
against MMQ on an RTX PRO 5000 and an RTX 3060); the kernel counts what it ran and why it declined and prints that at exit;
prefill_iq4_w4a8_test takes --tokens N (tails), --wide (scale extremes) and --bench (MMQ against the kernel per product set).
Class E: 0 differing output words in every case (tokens 97 / 2049 / 4096 / 8191 / 15360 / 32768, uniform and log-normal routing,
group buffers and pointer table). Kernel time against MMQ on the down products (IQ4_NL): +9..+70% on sm_86, +13..+50% on sm_120.
cpu_vnni_ok() = AVX-512 F + DQ + BW + VL + VNNI + OS state, cpu_vbmi_ok() = F + VL + VBMI; the Q2_0 row dot and the q8 quantizer take the 512-bit form
on VNNI without VBMI (the one VBMI op, vpmultishiftqb, has a bit-for-bit substitute unpack), and the i-quant / Q5_0 rows of iq_avx512.cpp, whose object holds
no VBMI instruction, gate on cpu_vnni_ok() instead of cpu_avx512_ok(). Adjustments over the PR: DQ is asked for (the file is compiled with it), the i-quant rows
gate on VNNI (the rows use vpdpwssd/vpdpbusd), and STRATA_NO_VBMI=1 runs the substitute unpack on a CPU that has VBMI (a test knob).
IQ4_XS gate/up: ggml-cpu's dot order (the integer lanes of its AVX2 dot, f16 scale times y.d, eight float lanes, its hsum) for every token count;
before, an AVX-512 CPU ran ggml's one-token dot per token. Q5_0 down: ggml's dot order, weights decoded once per block pair for all tokens.
IQ4_NL down: iq4nl256_down_rows' order, 512-bit. Each is checked once against the code it replaces; STRATA_NO_IQ4XS512=1 / STRATA_NO_QD512=1 switch
them off. The IQ4_XS nibble-plane decode follows the layout of PR 5243.
Co-authored-by: bsorensen110 <[email protected]>
Ported from #4972 (Phlimosx), extended: the residency is checked again at every request's start, so a
smaller cache (VRAM resize) or an eviction turns io prefetch, its decode gate and the look-ahead back on,
and a regrown cache turns them off again. Only what the defaults switched on is touched: an explicit
STRATA_IO_PREFETCH=1 keeps io prefetch and the look-ahead, STRATA_LOOKAHEAD=1 keeps the look-ahead,
STRATA_FILE_TIER_IDLE=0 turns the policy off. set_io_prefetch clears the gate flag with the prefetch.
Class E: the answers are identical (45 of 45 decode answers on l620, --mmap-experts, 24576 of 24576 resident).
A diagnostic for testers and for the performance work, off by default. With STRATA_MANIFEST unset every call returns at once
and nothing is printed or written. With it set, the engine writes <file> (whole file, replaced by rename) when it is ready and
again after a request if something new was recorded; the launcher (strata_control/split_engine.py) writes the all-cards split
decision to <file>.launch.json. Sorted keys, no timestamps, process ids or addresses, so two matched runs give the same text.
Content:
- engine: version, engine kind (single GPU or layer split), stages, KV format, drafter, spec window, pipeline windows, PLE I/O.
- budgets: context, KV resident cells, expert slots and cache MiB (all tiers and primary), VRAM reserve, prefill chunk cap and
stages, pool workers, memory limit, resident budget, batch slots, conversation cache.
- paths: one entry per decision with selected/declined and the reason in plain words: prompt experts per format class (MMQ,
FP16 WMMA tiles, native fused kernels, FP16 GEMM, with the gate/up and down formats and the layer count), MMQ direct and group
sort, HC_Q8, the output head (VRAM or pinned host, repacked or not), route tail skip, DMA batch mode.
- tensors: dense and head formats as held (count, bytes), the packed and HC_Q8 copies made at load.
- launch: split-engine mode, engine chosen, reasons, card architecture and count, routed-expert types, and the AMD auto rule
condition by condition (architecture, card count, expert types), so four gfx1201 with the IQ3_S set is recognised and two or
three cards are not promoted by the four-card rule.
The only text it prints is "strata: STRATA_MANIFEST: cannot write <file>" on stderr, once, when the variable is set and the file
cannot be written.
Tests: tests/core/run_manifest_test.cpp (off is free and writes nothing, matched runs in any recording order give the same text,
replace-on-rerecord, escaping, dirty tracking, unwritable target), tests/split/test_split_engine.py::LaunchManifest.
Class E: the queued jobs (page offsets, read lengths, Use order inside a job) and the dedup_rows count are the same as
before; the bytes delivered are the same.
PleReader::issue() built a std::unordered_map by page and a std::vector<Job> on the heap on every call, then sorted the
jobs by offset. A decode step asks for 1..16 rows (the prefetch for PLE_N_HEADS = 16), so those allocations and the
hashing were most of the work. A batch of up to 64 rows (PR #4910) now collects its misses into a fixed stack array,
sorts them by (page offset, request order), and merges equal pages in one pass straight into the queue. Nothing is shared
between calls, so concurrent issuers (they hold the reader's mutex when threaded) have no scratch to race on.
Batches above 64 rows (a prompt chunk: thousands of rows) keep the previous hash path. The PR also sorted every miss in
that case with a reusable vector; measured, that is slower there (see below), so it was not taken. A batch of more than
2^32 rows is refused like a failed reader, as the 32-bit order/ticket counters cannot hold it (the old code could not
queue it either).
Measured with tests/ngram/ple_issue_bench.cpp (this thread's heap allocations and thread CPU time inside issue(), threaded
reader, 3000 calls per cell, 2 rounds that agree within 3%; Ryzen 5 5600X, g++ -O2, synthetic 300k-row table, rows are a
4000-row window / uniform / skewed):
rows allocations/issue before -> after CPU us/issue before -> after
1 5.1 -> 2.1 1.69 -> 1.54
4 13.4 -> 5.4 2.00 -> 1.82
16 40.3 -> 17.4 3.44 -> 2.53
64 125.2 -> 51.8 8.07 -> 5.01
above 64: unchanged (the PR's sort-everything variant took 159 vs 66 us at 3072 rows and 2.9 vs 0.55 ms at 49152).
The decode impact is not measured here (see the round's results file).
Tests: ple_reader_test --selftest now also checks, for batch sizes 0, 1, 2, 15, 16, 63, 64, 65, 66, 127..129, 300, 1000,
4096 (random, clustered, one row n times, descending, out-of-range mixed in, page-straddling rows twice), that the bytes
are right, requests grew by the rows asked for and dedup_rows by (rows read - distinct pages). With the existing
selftest and the multi-client stress test (rows across pages, injected delays and read errors, late writers, failed
reader): pass plain, under ASan+UBSan, and the stress test under TSan.
Co-authored-by: martifaj <[email protected]>
Correctness (class E): the same windows are served in the same order, only a little earlier; the tokens do not change.
With --pipeline-windows 2 on a two-stage layer split the drafter sits on the first stage's card (mtp_first).
read_windows_pl called mtp.prefill(...) right after a stage-1 window finished, while stage 0 still had a window in
flight on that same card. MtpDrafter::prefill blocks the host until its stream is idle (on HIP it synchronizes once
per group whatever its `sync` argument says: prefill_per_group_sync), a stage-0 window only advances when this same
thread serves its doorbells, so the host sat in hipStreamSynchronize and the window it waited for could never
finish: the first request of a fresh engine read its prompt, produced 0 tokens, and the watchdog ended the engine
60 s later (gfx1201 + gfx1030, issue #5147). Neither 20 s guard in Verifier::service / done can fire, because the
loop never runs again.
serve_stage0_to_end() serves stage 0's windows to their end before that prefill, when the drafter shares stage 0's
card. The HIP per-group synchronization (which exists because the unsynchronized E-4 queue hangs on gfx1201 /
ROCm 7.2.4) is left as it is: the caller is fixed, not the mitigation. Pipeline windows 0 and 1 take no new path
(mtp_first is false there). The overlap lost around each drafter prefill has not been measured.
Not testable on the boxes of this round: it needs a mixed pair of HIP cards (gfx1201 + gfx1030) and a two-stage split;
p620 has one R9700. Compiles on CUDA (z440, sm_86).
Co-authored-by: cloud4t0r <[email protected]>
Correctness (class E, not a speedup): the product values do not change; the rows past T*K are never routed to.
llama.cpp's mul_mat_q_stream_k_fixup loads ids_dst[col_low + jt*J + j] for a whole J-column tile before it compares
the tile with the expert's rows (the main kernel skips an empty tile first; the fixup only skips its writes), and its
tiles run to ceil(max_rows / J) * J for every expert of the launch. A table of exactly T*K entries is read past its
end by up to roundup(max_rows, J), and the main kernel reads up to J - 1 past its last row. Strata's `ids_identity`
was an exact-sized allocation: when it ended a 2 MiB page the over-read faulted on the next (Xid 31 FAULT_PDE,
"prefill copy_i32: an illegal memory access" on a long prompt on a CUDA layer split; issue #1468 and others).
PR #5108 added T + 256 rows by hand. Here the margin is derived: J is bounded by ggml's mul_mat_q_switch_J
(J = 8 .. 128 in steps of 8), the tile loops of moe_mmq.cu now use that one constant (kMmqTileColsMax /
kMmqTileColsStep), and the table is T*K + roundup(T, J) + J rows (an expert takes at most one row per token), with
checked 64-bit arithmetic: a size that does not fit fails the carve instead of wrapping to a small table. The same
size is used by carve, bytes_needed_impl and the three places that write the identity (helper rebind, normal prompt,
layer-major), so the estimate and the carve stay identical. The peer prefill's row table, an exact cudaMalloc used by
the same MMQ launches, gets the same margin.
Tests: tests/prefill/mmq_ids_margin_test.cpp enumerates every index the fixup and the main kernel can read for 2000
random layouts (late experts, empty tiles, every J), checks the exact-sized table of before is overrun by the late
expert layout, and the overflow refusals. Builds on CUDA sm_86 (z440).
Co-authored-by: quantforge-br <[email protected]>
Correctness (class E): kept even without a speedup.
Flag B (PR #5096): the graph waits for "flag B >= layer + 1" and flag B is one counter that only rises. A non-empty
plan raises it from a host function on the copy stream, after its uploads. An empty plan (n <= 0) raised it on the
spot, even when the copy stream still held an earlier layer's, or the other token group's, uploads: the counter
jumped past a layer whose bytes had not landed. Verifier::raise_flag_b_ordered() now raises it through the copy
stream whenever copies were queued in this window, and directly only when none were. The same ordering is used at
the four places that publish an empty plan when the pool did not (run, run_slot_rows, batch_serve, service), which
had the same hazard. A failed host-function launch used to be ignored (flag B never rose and the window hung); it
now stops the engine like a failed upload does, after releasing the GPU waits.
Adaptive-copy bookkeeping (PR #4844): a round planned on the pipelined loop whose loop drained before step 3 is
marked resident on the serial rule, which used the lo/hi/bi of an earlier round, so only that range of swaps was
marked and the others were lost (the slot held the new expert, the table still named the old one). The pipelined plan
now names the whole round. The range arithmetic moved into AdaptiveCopyRange (adaptive_range.hpp) so a host test can
walk the rounds.
Tests: tests/core/flag_order_test.cu (delayed host-function copy standing in for a slow DMA, a consumer kernel that
reads staging once flag B allows it, a negative control that shows the old direct raise releases it early, flag only
rises, nothing queued, graph replay twice, every device; passes on both A4000s of z440), tests/program/
adaptive_range_test.cpp (serial batches for 0, 1, many swaps and several caps, the stale-range loss, empty round).
Co-authored-by: AlanJager <[email protected]>
Co-authored-by: uvlad23 <[email protected]>
Reporting only (class E): no engine speedup, no change to what the engine computes.
serve/server.py
- A prompt read that gives way (BYIELD) and resumes keeps the request's external prefix boundary, and the prompt
time and tokens of every leg are summed exactly once (prefill_leg_stats; PR #5318).
- The admission's DONE times one token and BDONE counts that token but times only the slot: the slot time now
includes the admission time, and the output token is not counted twice. A figure the engine never reported
(a close at the admission token, in the slot, a carry back to the solo path) stays unknown (None) instead of
turning into 0 or into the API's output count (PR #5307).
- The two patches overlap in the request finalization; ended_on_yield is taken before the unknown-figures step so
the identity check still sees the yield's own record.
PfTimer::mark(): a mark in the phase already running is not recorded (PR #4006). The totals are the same, a
fraction of the event records; under WDDM every record is a submission of its own and the launch gaps they made
were charged to "dequant". The rule is a pure function in prefill/timer_marks.hpp with a host test.
Tests: serve/test_yield_accounting.py (new: two yields, solo and admission yields, cancel after one and two
yields), serve/test_parallel.py BatchLegAccounting (EOS and length finishes, zero slot time, close at the
admission token and in the slot, slot to solo carry), serve/test_request_accounting.py (unknown stays unknown,
explicit zero), tests/prefill/timer_marks_test.cpp (same totals with 18 of 23 records skipped).
Co-authored-by: YusenZhang0601 <[email protected]>
Co-authored-by: shoucandanghehe <[email protected]>
Co-authored-by: Vop3p <[email protected]>
vec_dot_q5_0_q8_1 used the q8_1 block's ds.y (the sum of the original activations). With the SwiGLU outputs (not zero-mean) the whole-expert error with Q5_0 down was 2.1e-2 .. 3.1e-2 against 1.2e-2 for every other down type. Now 1.2e-2, and the GPU agrees with the CPU path. The Q5_0 parity limit is tightened to 1.8e-2.
native_expert_parity: --bench decode micro-number, +0/-0 is no fp16 difference.
Registers every gate/up x down pair of the new sets (CUDA and HIP, SYCL), a dense type test
(native_dense_types_parity), Q5_0 down in native_grouped_parity, new-set types in the MMQ screen,
and a --synthetic mode in the SYCL native_expert_parity.
- native_expert_parity: ctest entries for every gate/up x down pair the new sets use
(IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ4_XS gate/up; Q2_0, IQ4_NL, Q5_0, Q8_0 down)
at hidden 2560 / FF 640.
- native_dense_parity: dense Q2_0..Q6_K, IQ4_NL, IQ4_XS decode (MMVQ, 1-8 columns) and the
prompt dequantizers (f32, f16, bf16) against ggml.
- prefill_mmq_v31_test: the new expert formats through MMQ (Q5_0 only in a
STRATA_ORCA_Q4KS_MMQ build), plus a --bench of Q5_0 MMQ against dequantize + cuBLAS.
The checker treated every ET_DYN without PT_INTERP as a static executable, so the RUNPATH $ORIGIN that Pillow, numpy and sharp libraries carry was
reported as 'static PIE with a damaged dynamic section' (35 files in 0.2.11, 18 Pillow libraries in the published 0.2.10 Control; false positives).
A static PIE is now an ET_DYN with no interpreter, a non-zero entry point and DF_1_PIE in DT_FLAGS_1 and no DT_SONAME; a library (no entry point, or a
SONAME and no PIE flag) is never checked. A candidate whose dynamic section cannot be read is reported as damaged, because the overwrite that crashes
rg also erases the flags, so the test for 'executable' uses the entry point, which the damage does not touch. Tests (tools/test_appimage_elf.py, 16, run
on p1006): shared libraries with and without a SONAME, entry point or RUNPATH pass; a static PIE passes; a static PIE with a 0x5A .dynamic or an added
RUNPATH fails; a real gcc shared library with RUNPATH $ORIGIN passes; a real gcc static PIE passes and its overwritten copy fails. Build tool only.
The sidebar now shows a file's current content, so a CRLF file reaches the Markdown renderer. Every line then ends in a carriage return, which '.'
does not match: a list line passed the list-start test but not the item pattern, the list loop took no line, and the outer loop never advanced - an
endless loop pushing nodes (150 MB to 4.4 GB in 20 s on a 60 byte file; QAGUI final pass, the agent's Add-Content writes CRLF). 0.2.10 never showed
the changed text so it never met it; the bug is older and any CRLF Markdown list opened in the preview would hit it.
Line ends (CRLF, CR, LF) are read as one in the renderer and in the preview slicer, and the list loop always moves on (a line the item pattern rejects is
shown as text). Test: tests/markdown-crlf.test.mjs renders in a child process with a 200 MB cap and a 20 s limit; on 68bead108 all 4 cases abort out of
memory, with the fix all pass. Edge 156, seeded live turns (stream, finish with the card, file rewritten with CRLF, cards opened 6 times): heap stays
near 19 MB with the fix. No visible text.
The Select popup (every dropdown in Settings, Models and the tune dialogs) was capped at 320 px whatever the window and opened upward through a
transform, so in a short window or at a large Interface size it scrolled although the window had room, or started above the top of the window
(Edge 156 matrix: Select popups 4 of 72 and 3 of 72 cases right on 548fdf061, 69 of 72 and 61 of 72 after; the rest are harness not-open rows).
It now uses lib/overlay-place.ts like the menus, measures its real height once it is on the screen, and writes maxHeight. The Mode menu and the
sidebar More menu already passed on 548fdf061 (77 of 77, 84 of 84; 43 of 77 and 49 of 84 on v0.2.10). No visible text.
The preview kept a file's first content: openPreview dropped the freshly read item when its id was already a tab, the panel's loading effect had no
content version, and the effect that re-adds a message's artifacts put the stored first copy back. All three change together, through pure rules in
lib/preview-merge.ts: opening a changed file replaces its tab in place; a message's artifact only adds a missing tab or replaces an older, different
copy (each copy carries the time it was read or made) and never overwrites a newer one; both return the same array when nothing changed, so no update
and no loop follows. The panel reloads when (id, content version) changes: a hash of the text, or the host's mtime and size for pictures and PDFs
(/api/files/preview now sends mtime). The reading view is reset by the file, not by its content. Tests: stale case, 2000 alternations without a
change, one load per real change. Real Edge 156 with the real panel: the old rules keep Version One, the new ones show Version Two through 300
sync/render alternations with 0 Markdown renders. No visible text.
A reload does not stop an agent turn, but the page that came back showed "Network error. Try sending the message again."
with no Stop button, and a new message was refused with "This chat is still working". The host now answers GET /api/chat/active
(the chats with a turn running) and POST /api/chat/attach: a running turn sends one catch-up (the start, the text and thinking
so far as a resync frame, every tool card as it is now with the approvals waiting for a yes, the open questions) and then its
live frames; a turn that ended while the page was away is shown whole once, named by the request id of the cut-off reply.
At start the page asks for the running turns, takes each over in the reply it already has (running, Stop button, the
failure it wrote is cleared), and asks once about cut-off replies of the other recent chats; nothing to take over leaves the
reply as it was. Existing states and texts only.
The things that used to run at bootOnce from the chats in memory (follow time-skip runs, end loops the last window left running,
mark tool cards of vanished turns) now run once the chats have been read from the host (resumeSessions), because the chats are
no longer in the page at that moment. The save retry waits a little longer each time (up to a minute) and never keeps a closing
page alive. Also: tests for a second window's changes (applyChatChange) in the parallel-chats test.
Conflicts in strata_cli/bench_facts.py (the redactor), three hunks: kept every side. The patterns stay QATUI's (the key and the names are matched as whole tokens, a hyphen next to the key does not hide it, the names are not touched inside a label the function wrote), each entry carries SECURE's third field (is a login or host name) for redact_document, and the host names come from socket.gethostname() and sysinfo.node() (LIFE), both marked as names.
On a PC whose login is an ordinary word ("user", "test", "admin") the whole-document pass turned status reasons and field names into <user>, and the report failed its own validation. Paths, addresses and the key are still replaced everywhere; the name rules apply to hardware, model and configuration text, error messages, engine options and request checks.
main() and mac_control_setup() read fam['hf'] to build the pin lookup URL; the NVFP4 families download a file list and have no such key, so every NVFP4 install raised KeyError.
The helper takes None and then keeps the old mark. Found by the INT3 merge run (tools/test_setup_nvfp4.py, tools/test_setup_dl_fix.py).
The app host saves each chat as its own file (data/chats/<id>.json, an index, a version and a revision in
every file), written to a temporary file, flushed and renamed over the chat, so a crash or a full disk in
the middle keeps the older version whole. The folder and files are owner-only. The page talks to the host
through /api/chats (the page must name itself with x-strata-client; other sites are refused), sends only the
chats that changed, never cuts anything, and keeps a chat the host cannot write whole in memory, retries it,
and says so with the notice and the "Copy unsaved chats" button in all nine languages. A save from an older
copy (a second window) keeps the version it displaces in chats/conflicts. The browser's two older layouts
(the 0.2.10 list and the per-chat entries) are handed to the host once at start and left untouched as a
fallback for one release; a last-moment copy covers a page that is closed before the host confirmed a save.
Other windows are told by an event and take over what changed.
Conflict in serve/server.py main(): LIFECYCLE wraps _main in the shared activity lease (#4463), OWN wraps it in the decoupled console (#4479 / #2481). Kept both: the lease is taken first, and _main runs inside decoupled_console() unless STRATA_BLOCKING_CONSOLE=1; the lease is closed in the same finally.
Conflict in strata_cli/bench_facts.py (the redaction helper): kept both sides' intent. The host name comes from sysinfo.node() (LIFE) and the key is redacted with usable_key(key), word=True, exact=True (SECURE).
DRAIN_TIMEOUT_S (300 s) and QUARANTINE_S (600 s) apply only after a request has left and the server has sent STOP / BSTOP:
_drain_control (the finally of generate_batched), _quarantine_control and the BDONE wait of _release_slot_when_done. They
never cover a generation a client is still reading. But they were counted from the start of the drain, so an engine that
was told to stop and kept producing lines (an older engine runs on to max_new; a long prompt read ends at its next chunk)
could be ended after 900 s, with every other request in it. Now every line the engine writes starts the limit again: only
an engine that says nothing at all for 300 s + 600 s after the stop is ended.
Tests (fake engine, limits scaled to 0.3 s + 0.4 s): an engine that keeps talking for ~1.8 s after the stop is not ended and
the request in another slot is untouched (fails with the previous commit's absolute limits); a silent one is ended; the
drain-timeout leak test now really closes the stream mid-read (the old version was ended by a BYIELD before the drain);
long replies, far over the limits, complete. Also: a client that is idle while the engine thinks is not cut by the 60 s
write limit (0.3 s between tokens, 0.1 s limit, the whole reply arrives).
On a batch engine the control lines carry one exchange at a time and every request takes `ctl` before it sends or reads
there. VRAM and SAVE / RESTORE read the same lines for their answer without it (POST /v1/vram took it itself, the image
encoder's cache resize, the load-time reserve and the session handler did not): run beside a request they took its
T / PP / DONE lines, and the request lost its reply (or session_file ended the engine as out of step).
StrataEngine.vram / session_file now run under _exclusive_control(): ctl on a batch engine (a thread that already holds it,
as Service.vram does, is marked and does not wait for itself), nothing to take for one request at a time. Waiting longer
than CONTROL_WAIT_S raises the existing "a request is still running; try again when it has finished". No output changes.
Test: the fake engine answers VRAM / SAVE / RESTORE; a request streams while the call is made. On the base the request
ends with 212 of 421 tokens; with the fix it is complete and the call is answered.
Each client puts its own random canary in its prompt and asks the model to repeat it; some leave before the first byte,
during the prompt read or after a few chunks. Every text a client receives (content and reasoning) is searched for the
canaries issued in the run: a canary that is not the client's own is a cross-delivery (exit 1). Used for the real-engine
A/B of the round 0213 ownership fixes.
print() in the server blocks when the console stops taking output (a Windows console in selection mode, a paused
terminal, a launcher that stopped reading the pipe), in whatever thread called it - including those that own the
engine's turn, a cancel or an admission. While serve main() runs, stdout and stderr (when they are the process's own)
are replaced by serve/nonblocking_out.NonBlockingOut: write() queues and returns, one daemon thread writes to the real
stream, the oldest text is dropped (and counted) when the queue is over 1 MiB / 4000 chunks. Restored and drained at the
end of main(). Protocol output (engine pipe, HTTP) is not touched. STRATA_BLOCKING_CONSOLE=1 turns it off.
No visible text is added; a drop is counted, not announced.
Tests: a plain print to an unread pipe blocks (control); the wrapper never blocks, keeps the newest lines in order, and
drains. Approach credit: the proposal in #4479 (one process-owned output consumer).
The thread that writes the SSE stream to a client also holds the engine's turn (the one-request path). A client that
stops reading without closing (a hung process, a dropped network) filled the socket buffers and the write blocked until
the OS gave the connection up: the engine idle, every other request waiting. A write that makes no progress for
STREAM_WRITE_TIMEOUT_S (60 s; STRATA_STREAM_WRITE_TIMEOUT_S, 0 = off) now ends as a disconnect: the engine is
stopped and the turn given back. No text changes.
Test: a client that never reads a 400k-token stream; on the base the turn is still held after 20 s, with the fix it is
given back at once. Part 2 of #4271 (arrival order of the one-request path: the turn is a plain Lock) is not done here.
_take_control read its turn, gave up the lock and then waited: a notification sent in between was lost and the waiter
slept up to 0.5 s although it could go. It now reads the turn (and the short-first arbitration) again under the lock
before waiting, and a waiter that leaves (cancelled, closed) notifies the others.
Approach credit: PR #4738. Tests are new (a hooked Condition forces the schedule; on the base the waiter waits).
- GpuBusy before any segment and with zero tokens: the history row says "error" with no prompt tokens and takes no part
in the totals, latencies or last timings (#3040); a continuation after native work keeps the old accounting;
- the engine's and the image encoder's stdout are decoded with errors="replace": on Windows a byte such as 0x8e ended
the reader thread (UnicodeDecodeError) and the server lost the engine's output. Protocol lines stay ASCII.
Not taken from PR #5044: the q36 engine OOM messages (engine C++, and the wording goes to the coordinator first).
Approach credit: PR #5044 (Python parts). Tests: test_parallel (refused admission accounting), test_serve_robustness
(a 0x8e byte before READY and in a reply).
StrataEngine.__init__ cleaned up a start that ended before READY through self.proc; restart()'s close() on another
thread had set it to None (AttributeError: NoneType has no attribute stdin, a recoverable engine abort became a 400).
The cleanup now uses the process object of THIS start, captured once right after it was spawned.
Approach credit: PR #4792 (same idea; the test is new: the closing thread is forced inside proc.wait()).
Polls come from several request and monitor threads. A second poll moved `pos` between the first one's size check and
its read: lines were read twice (parks counted double) and a read length could go negative (ValueError). Found by
HWSTALL while triaging #3443.
serve/test_serve_robustness.py: the overlap is forced with a hooked getsize; on the base every parked line is counted
twice (8 != 4).
A --batch request read its slot's lines from `self.slot_q[slot]`, the control lines from `self.lines`, and sent its
BSTOP / BYIELD / STOP with the slot number, all looked up when used. After the engine was restarted the same names meant
the NEW process: a request of the old one read (and could stop) the new one's requests. And a request that left before
the engine had ended its exchange (a drain that timed out after 300 s) gave the slot and the control lines back, so the
engine's late T / PP / DONE / BADM / BT frames reached the next request as its own.
- the request notes the queues of the engine process it was admitted on and reads only those; a send meant for that
process (BSTOP, BYIELD, STOP) is not sent to a later one (EngineDied instead); slot state is only given back on the
same process;
- an exchange that is not drained in DRAIN_TIMEOUT_S is quarantined: the control lines and the admission's slot stay with
it, in the background, until the engine gives DONE / BADM (then a continued admission is BSTOPped and freed at its
BDONE); if the engine never does within QUARANTINE_S it is ended, as a silent engine is (#481). A slot's BDONE wait that
times out ends the engine too, instead of freeing a slot whose lines may still come;
- the pump drops BT / BDONE frames for a slot no request has claimed (counted in stale_frames);
- _ctl_mode, _ctl_result, _yielded, _last_done, _ctl_erred are per request thread on a batch engine;
- serve/test_stream_ownership.py (BatchOwnership): interleaved clients with canaries, a cancel in the middle of the prompt
read followed by reuse, a drain that times out (DEAFSTOP), the engine ending during a drain and restarting. On the base
the restart case fails: requests of the old process stay attached to the new process's slots.
The other cases of the register (mid-prefill cancel, drain timeout, slot not handed on) pass on the base with the fake
engine; they are kept as regression tests. The real-engine check follows.
ReplicaEngine.generate released the per-replica run lock in its finally without closing the child generator. The
child's own STOP and drain-to-DONE ran later, when the wrapper frame was collected, after the next request had sent
its GEN: the two read each other's tokens and DONE (an empty reply, or another request's text), and a cleanup error
was lost as an ignored generator exception. The unbatched engine's shared `last` was also read by Service after the
lock was free, so a reply could carry the next request's figures.
- close the child before the lock is given back; a close that fails ends the replica (heal() starts it again) and the
error still reaches Service (EngineSilent handling is unchanged);
- copy the new DONE figures into a per-caller object while the replica is owned; with no new DONE the caller's earlier
object stays, so an old result is never recorded as new;
- serve/test_stream_ownership.py: canary replies from a fake engine, a hooked run lock that forces the legal schedule
(the successor takes the replica right after release), 30 forced hand-offs and 60 unforced back-to-back requests.
On the base: empty reply for the successor, the predecessor's canary in its text, 30 of 30 forced hand-offs crossed,
figures of the other request.
Approach credit: PR #2886 (scheme A: explicit child close, caller-local DONE snapshot). Written in this tree.
linuxdeploy rewrites the RPATH of every ELF in the AppDir; for a static PIE (the bundled rg) that leaves .dynamic overwritten and the program does not
start (shown on a real gcc static PIE in tools/test_appimage_elf.py). repack-appimage.sh already put the staged bytes back, but --check covered only
engine-bundle and node, nothing ran the rg, and nothing stopped a pack with a damaged one. appimage_elf.py scans for a damaged .dynamic (garbage tags,
RUNPATH on a static PIE, filled relocation table, filler runs) and compares to the staged files; --check now covers control, harness, node and engine-bundle,
scans, and runs the bundled rg; the pack path runs the same check before it squashes. Whether the released AppImages were repacked is outside this repo.
shortTitle() cuts at a grapheme (Intl.Segmenter) or a code point, so a title never ends in half an emoji, a lone surrogate or a ZWJ. lib/store.ts still
cuts with prompt.slice(0, 47) in three places (not edited here: the GUIDATA owner's file); they should call shortTitle. Tests cover ordinary emoji,
ZWJ families, flags, skin tones, keycaps, combining marks, Arabic, Hebrew, CJK, and a JSON save and restore.
Interface size scales the page with CSS zoom on the root. In WebView2 (Edge 156) getBoundingClientRect and innerWidth are screen pixels, but left/top/
width/max-height written on an element are zoomed pixels, so a menu placed from its anchor landed (scale - 1) too far right and down: at 125% on a
1080p window the More menu was entirely off the screen (14 of 18 size and scale cases broken, measured in Edge 156). lib/overlay-place.ts places in
screen pixels (below, else above, else the roomier side shortened, kept inside the window) and divides by a scale measured from the page at the end.
Used by FloatingMenu (More, chat actions, preview file actions), Popover, the Select popup and the two tooltips. No layout or text change at 100%.
'N settings could not be tried: not enough free memory' is the result message (and the Terminal's tuning line) only when some candidates were skipped for memory
in a result that otherwise keeps the current settings; the structured outcome is unchanged and values.not_tried carries the count. Owner decision 11-04.
The window process (only it) is pointed at a fontconfig file that includes the system configuration, rejects *COLRv1* font files inside
<selectfont> and prefers the outline Noto Emoji. Written once under the data folder, only when a COLRv1 font is installed; FONTCONFIG_FILE already
set, STRATA_FONT_CONFIG=off or =<file> leave it alone. fontguard.rs is plain std with unit tests that run on every platform. The three variables
join the crash report. Also: the desktoplog crash-file test asserted the literal 0.2.0 and failed on any other version; it now follows the
crate version. Linux-only wiring (run_window_process) is not compiled on this PC; no WebKitGTK here to show the crash or its absence.
limit.output, CLAUDE_CODE_MAX_OUTPUT_TOKENS and maxTokens are max(4096, context/8): 16384 at 128K, so OpenCode's compaction summary has room (the tested 32K server keeps
4096). The input limit and compaction point stay as tested. Rewriting a config keeps an output cap other than the 4096 older versions wrote. #4528: the documented
provider block is still accepted by OpenCode's schema (deprecated in favour of 'providers'); no change. Idea as in PR #4230, written here.
The engine's flag schema picks which options are valueless: on DeepSeek --mmap-experts on|off|auto keeps its word through derive_config, the one argument
precedence (arg_layers) and a second launch. Approach as in PR #5028, written here.
The pre-pass that gave a hand-copied full-length shard the 'whole' mark also marked a shard left by a run that stopped between the rename and the hash check,
so download() skipped it and the pinned hash was never looked at. A file the pin table names is left unmarked now, and download() hashes and repairs it;
--gguf-dir files and files without a pin keep the mark. Both setup flows call one helper. Approach as in PR #4318.
WKWebView sends compositionend before the committing keydown, with isComposing false and keyCode 229; WebView2 sends the keydown first. The
composer asks a small guard (lib/ime-enter.ts): a key that says it is composing or has keyCode 229 belongs to the IME, and the first Enter in the
few milliseconds after a compositionend is the commit; the flag is used up by the next key and expires, so the next Enter sends exactly once.
The guard runs before the command menu too. Replay tests for WebView2, WKWebView, WebKitGTK, Latin input, Shift+Enter, a cancelled conversion
and shortcuts. No visible text.
Windows: the shell subscribes to WebView2 ProcessFailed; macOS: to the web content process termination. The page's own renderer dying is
written to the run log (kind, reason, exit code, the address it showed) and the page is loaded again after a short wait: the same screen at
/?recovery=1 first, home at /?recovery=2 second, and nothing after two recoveries in two minutes (the window is left as it is). The decisions
are in src/recovery.rs (unit tested). The page reads ?recovery=n before it renders (lib/recovery.ts), removes it from the address, and starts
with the sidebar previews closed until the person opens one or another chat. The chats are untouched. No visible text.
#5056 the worker is started without STRATA_API_KEY, so it answers to the key the tuner made for it. #5122 a candidate that adds no memory needs what the base run
started with (min of the old peak-RSS rule and base free memory plus the candidate's delta). #5078 a 5xx from the worker adds the engine log's last error lines;
an allocation failure is an OOM, other aborts fail that candidate alone.
pick_slot also recognises an earlier message boundary of a slot's held stream (the last question replaced), so the request no longer overwrites another chat's idle slot. A boundary every held conversation shares (one client's system prompt) names no owner; busy slots are never picked. Tests: branch affinity, busy owner, shared root, interleaved branches (retained prefix 96000 -> 115200 of 115200 tokens).
Two chats in one workspace folder: the end-of-turn sweep of chat A listed the files chat B had just written as
chat A's deliverables, and the other way round. A registry shared by the turns of the host (FolderClaims) now
records which chat's file tool wrote which file; the sweep skips files another chat claimed, and for turns that
overlapped in the same folder it also leaves out files nobody claimed (a script's output cannot be told apart),
rather than give them to the wrong chat. A turn that ran alone behaves as before. Chat records, tool cards and
approvals were already keyed by chat id and session id; a store-level test with two interleaved live chats and
the same call id in both pins that down. The per-folder bucket, the shared model cache and the approval-policy
notice described in the issue are Harness and engine behaviour, not changed here.
Every SYCL start died with TypeError: unexpected keyword 'announce' because install_two_card_spec wrapped engine_args(cfg) while serve/server.py calls it with exe
and, for the memory budget check, announce=False. The quiet call prints no line. Tests run the real base function under the wrapper.
Approach from PR #5075. --hf failed on every repository with AttributeError: 'Header' object has no attribute '_size'; a cut or huge-length header is now a ValueError that names the file.
Approach from PR #5040: _close_verdict holds while </tool_call> is still arriving, and the streaming scan of a body whose end is known runs as final. #4701 (final call) and #4702 (longer code fence) are already fixed on the base; the corpus gets the four 4701 specimens (PR #4762 and #5040 fixtures) and runs them whole, in pieces and cut at every byte, with streamed arguments on and off.
_drm_vendor_backend skips a make whose every card reports a small own memory (the 512 MiB carve-out of an APU) while another make has a card;
intel_arc_cards reads each entry on its own, takes card10 and up, and counts an unlisted Battlemage/Alchemist id as an Arc of the family's smallest memory.
The launcher's merge of the Arc beside other makes was already in place. Backend half as in PR #5184, written here.
The sidebar parsed and rebuilt the whole file on every render of the chat. It now renders through PreviewMarkdown: memoised by the text, cut into slices at block boundaries (each memoised, keyed by place and content), two slices at first and two more per waiting step (one timer, cancelled by a newer text), a block above 40000 characters and a file above 200000 shown as plain text. No visible text, no layout change.
Explicit launch values, then persisted overrides (user_args and hand edits to the launch config), then calibrated values (recorded by
engine calibrate --apply-result with the model and context they were measured for), then Auto. derive_config builds the argv once through
strata_control/arg_layers.py; explicit values go to user_args so the DFlash2 policy keeps them, and are not remembered when left out next time.
Failed stops are counted apart from stopped ones; the empty-list line is for zero targets only. No wording changes: the existing lines are reused.
Approach as in PR #4568.
running_launches binds prof before the try so a failed load uses this row's defaults; kept_key skips list rows that carry only an error
and keeps looking for a valid key. Approach as in PR #4562 and #4563; tests use real temporary profiles in both orders.
Approach from PR #4204. sd-cli at the pinned stable-diffusion.cpp commit reads --prompt-file as raw bytes (checked in examples/common/common.cpp at 2988060). The job folder is the 0700 mkdtemp one; the file is 0600, created exclusively and removed with the job's other files. The stand-in sd-cli reads the file; a canary test checks the recorded argument list.
Approach from PRs #4500 (one redact_object call before validation) and #4599 (none is no key, whole word, own case). tools/strata_bench.py gets the same none rule and key matching.
The 8-character floor of #4599 is not taken: a short key is still removed where it stands alone.
requirements.txt and the four hash-locked terminal locks (windows, linux x86_64, linux aarch64, macOS arm64) move to 13.0.0; Python 3.10 stays on
12.3.0. Image.open() in the vision path, the EXIF probe and the DeepSeek picture reader now only opens JPEG, PNG, GIF, BMP, WebP, TIFF and AVIF, so a
JPEG 2000 file never reaches OpenJPEG on any Pillow version. Approach from PR #4220 (version split by Python range).
The saved chat list cut long fields, dropped the oldest messages of a long chat and left old chats out of the
list when it passed 3.5M characters, and every restart trimmed the chats again (12 messages of 140,000
characters came back as 6). Now each chat has its own storage entry holding the chat exactly as it is, the index
only keeps the order, and a chat the browser refuses keeps its last good copy, stays whole in memory, is tried
again at every save and is reported (state.sessionSave, unsavedSessionsJson). The old single-entry list is moved
over chat by chat and a chat leaves it only after its own entry was written. Approach from PR #5192 (never remove
chats) kept; the field clipping it still did is gone.
- gpu_roles.describe(card_numbers=True) prints "GPU <n+1> (--gpu <n>)"; the two test_launch.py assertions are back on the old digits plus the hint. The Terminal has no translation catalog for this line.
- split_engine: the refusal sentence ends "has 7 GiB free)" when the free memory was read (the size, as before, when it was not).
- the desktop app asked `drafting plan` / `drafting fetch` before it unloads the running model, so its own engine counted as busy memory on a model switch and the plan could pick DFlash2 and fetch the 1.2 GB drafter. New flag --engines-unloading (hidden from --help), sent by native-control.mjs for a load that unloads the running instances (not for a model added beside them); same effect as Runner.card_probe. Logic only, no text.
- tests: python (drafting plan with and without the flag), node split-app.test.mjs (flag on the argv, a 2-card model switch, a model added beside).
- discovery.prepared_beside(gguf, data_dirs): after the models-folder walk it also looks in --data-dir and the remembered data folder for mtp/rt, the pack and the picture reader (setup writes them there); plan, download_plan and the own-files preview pass them. A model folder outside the data folder no longer asks for the 5.2 GB layer a second time.
- split_engine: Card.free_mib from the same nvidia-smi call (memory.free; AMD: the render node's vram_used), fit = min(size, free) - 700 MiB; busy cards get the layer split with DFlash2 and no draft-layer part. Runner.card_probe leaves free memory out while a server of this launch (same profile or port) is about to be stopped. Free cards decide as before.
- plan: the cards line prints the numbers --gpu takes and leaves out cards CUDA_VISIBLE_DEVICES (AMD: ROCR/HIP_VISIBLE_DEVICES) hides; the split probe respects the same variables when no card is named. Digits only.
- tests/terminal/test_fix3_0212.py (41 tests); launch_support.Home clears the visibility variables.
Logic only, no text changed: bench_tips takes the GPU and leaves the tip out for those configs; older cards, AMD, a layer split and no GPU information keep it. The explicit auto:32768 warning is unchanged. test_setup_bench_tips covers it (10 passed on l620).
The gfx12 rule in generate.cpp now serves both: nv_chunk.hpp (STRATA_NV_CHUNK=0 or STRATA_DEFAULTS=026 keep 8192, =16 keeps 16384, =1 skips the RAM guard), the fit scan still picks a smaller chunk when the cache cannot lend the buffers, the #2090 RAM guard steps 32768 -> 16384 -> 8192, and a layer split takes 8192 back (STRATA_SPLIT_CHUNK16K=1 keeps the ceiling). An explicit --prefill N, auto:N or STRATA_PREFILL_AUTO_MAX is never touched. Volta / Turing / Pascal keep 8192.
A6000 UD-Q4_K_XL, 13.1K of 24.6K experts in VRAM, MMQ_KQ on: prompt 20K +43.4%, 32K +22.5%, 64K +21.3%, 8K 0.0%; cards that hold every expert: flat.
nv_defaults_test covers the ceiling, the switches, the card rule and the step-down.
An integrated GPU cannot report how much room is left (the pool reads 0 MiB free on a start that works). With a chunk above the 16384 default that the person or the environment names (STRATA_PREFILL_AUTO_MAX=32768, --prefill auto:32768) and a UD-Q4_K_XL cache that fills the pool, the driver could not create a stream (PAL failed to submit CMD) and the first launch after the prompt buffers died with HIP 719. Once the prompt path's own buffers are in place the engine now tries one stream, fill and wait; if that fails it prints one line naming the chunk, the cache and the 16384 alternative and ends with exit code 2, before any prompt is read (HIP builds, integrated GPU, own buffers, chunk above 16384 only; the 16384 default, CUDA and Metal are untouched; STRATA_PF_OWN_PROBE_OFF=1 skips it). The failed context cannot be reused, so the engine cannot step down in the same process.
The #1630 warning about a named chunk of 8192 or more no longer prints on gfx1151 (it said a bare --prefill auto stops at 6144, stale since 0.2.8); it is printed once the device is known. STRATA_TRACE marks around the prompt path and the buffer size of the chunk.
setup: the bench tip for --prefill auto:32768 is not said on an APU (apu_hides_chunk_tip). The tuner's prefill_32k arm needs a discrete GPU. The app's saferStart does not retry a start that ended with the chunk refusal (the same chunk would fail again, and a memory setting is not the cause). Tests for all three.
- drafting.blocked_by(named=True): only --pipeline-windows still blocks; Metal keeps the old rule (no per-request drafters there)
- apply/launcher/plan pass the choice; the all-cards split engine runs --draft dflash2 --draft-model PACK only for DFlash2 by name AND --split-engine on
- the split engine's own DFlash2 start line counts as "DFlash2 on"; a split run config with --draft dflash2 is left as it is
- app Drafting hint and Terminal --drafting help say it (owner-approved wording, English only)
- tests: terminal drafting/single/split-parallel, split engine, serve parallel, node parallel-default
kNvKqDefaultOn is true; STRATA_MMQ_KQ=0, STRATA_NV_KQ_DEFAULT=0 and STRATA_DEFAULTS=026 turn it off, a STRATA_MMQ_KQ the user sets is kept, Volta / Turing / Pascal keep the FP16 prompt path. A pack without K-quant experts never reads the variable.
Start check (RTX PRO 6000, UD-Q4_K_XL, no STRATA variable set): the start prints the K-quant MMQ line and loads libstrata_kq; 4K prompt GPU timeline 1,047 ms (dequant 7.3%) against 1,500 ms (dequant 25.1%) with STRATA_MMQ_KQ=0, prompt 2,854 vs 2,233 tok/s (one request each).
Owner 11-03 ("3 Yes"). Since 0.1.44 setup wrote STRATA_ROUTE_TAIL_SKIP=0 into the run config of every Unsloth file whose experts are all in RAM. On Linux gfx1200 / gfx1201 it no longer
does: the engine's own default (rank 7, applied only while the variable is unset, so =0 still turns it off) runs. UD-Q4_K_XL on one R9700: decode +6.2% (2 rounds x 8 classes, interleaved) and a
median +15.6% over the 61 suite tasks; the suite scores 56/61 against 55, 57 and 56 for the default arms; start line "STRATA_ROUTE_TAIL_SKIP=7 is ON (default)". Teacher-forced decode-window
KL against no skip 0.0773 (noise floor 0.00000, top-1 91.1%; agent segment 0.20) on a static, low-hit-rate cache, an upper bound for real use. Windows and other cards keep the 0.
unsloth_default_env takes the card's arch (default "": the old behaviour); test_tail_skip_stays_on_for_linux_gfx12 pins the rule.
gemm_dequant_side_parity failed with the cache because its synthetic scratch (10.5 M elements) is smaller than four of its weights, so uncached they take the sliced path and cached one full-N GEMM. The engine's scratch is 32 M elements and every dense projection fits it. gemm_w16_parity runs the engine's scratch, shapes and types, cache off against on, bitwise. STRATA_PF_W16_CACHE_MIB=0 turns it off.
UD-Q4_K_XL's Q4_K gate/up and Q5_1 down experts read the prompt through the FP16 dequant path plus cuBLAS on CUDA (STRATA_PREFILL_TIMING, 8K prompt on an A6000: dequant 21%, gemm gate/up 24%, gemm down 13% of the GPU timeline).
With STRATA_MMQ_KQ=1 (libstrata_kq): RTX PRO 6000 (sm_120) prompt 4K +37.7%, 8K +25.8%, 20K +34.3%, 32K +32.6% (n 4 per cell, 2 rounds, all won); A6000 (sm_86) 4K +1.7%, 8K +9.0%, 20K +8.1%, 32K +11.2%. First-token KL against the FP16 path over 24 prompts of 2K: 0.0243 (PRO 6000) and 0.0351 (A6000), top-1 23/24 and 22/24.
nv_kq_default_env / apply_nv_kq_default: compute capability 8.0 and up (lowest card decides), off switches STRATA_MMQ_KQ=0, STRATA_NV_KQ_DEFAULT=0, STRATA_DEFAULTS=026; kNvKqDefaultOn stays false, so nothing changes by default.
Conflict in src/split/src/prefill.cu (pf_ring and the ring wiring): kept both sides. The HIP all-to-all small-chunk path stays, guarded by !own_out (the own-rows output is the NVIDIA hyper-connection split's), and the ring runs with nvdual's own_out argument; in the next-card wiring the HIP a2a box/flag copies and the hyper-connection split's input pointers are both kept.
Owner 11-03 ("do all 5"), two prompt defaults for Linux gfx12 (R9700), both changing rounding slightly and both with their off switch.
STRATA_MMQ_KQ=1 joins the gfx12 value-changing defaults (gfx12_extra.hpp, applied by apply_arch_defaults, so a raw engine gets it too; Linux only like the
other approved gfx12 prompt defaults; STRATA_MMQ_KQ=0, STRATA_GFX12_DEFAULTS=0 or STRATA_DEFAULTS=026 turn it off, a value the user sets wins). The experts of
Unsloth's UD-Q4/Q5/Q6_K_XL files (Q4_K, Q5_K, Q5_1, Q6_K) then run their prompt products on llama.cpp's MMQ kernels instead of being dequantized to FP16 and
multiplied by hipBLASLt (UD-Q4_K_XL ran 25K dequant launches and ~50K small Tensile GEMMs per 8K prompt). R9700, same job, interleaved: prompt 4K +10.3%,
8K +14.3%, 32K +16.8%; first-token KL against the FP16 path 0.031 over 24 prompts of ~2K (agent turns 0.065), top-1 23/24. A pack without K-quant experts never
reads the variable. The fused variant (STRATA_PF_FUSED_KQ) stays a flag.
The gfx12 ceiling of a bare --prefill auto goes from 16384 to 32768 (gfx12_chunk.hpp, pure functions with tests). A 20K prompt is one chunk instead of
16384 + a 3.6K tail, and every chunk of 1,024 tokens or more streams all the experts the card does not hold. R9700, IQ3_S, 3 rounds interleaved: 20K +12.1%,
64K +2.6%, 32K +1.3%, 8K +0.9%; UD-Q4_K_XL 20K +26%; first-token KL against 16384 at most 5e-5. The lent slots are taken only for what a prompt needs and are
refilled after it; the RAM guard (#2090) now steps 32768 -> 16384 -> 8192 until the extra loan fits. STRATA_GFX12_CHUNK=16 keeps 16384, =0 keeps 8192.
gfx12_extra_test covers both (gfx only, Linux, off switches, user value wins, the ceiling and its step-down); docs/DETAILS.md lists them.
The Linux gfx12 matrix-core prompt path hard-set the streamed-expert ring to 96 slots (measured before the fused IQ3_S kernels). A layer's streamed
experts want a deeper ring at small and middle chunks. R9700 (gfx1201), IQ3_S, 3 rounds x 2 reps interleaved, prompt tok/s against ring 96: ring 256
4K +12.5%, 8K +2.4%, 2K +1.4%, 32K -0.4%; ring 192 4K +11.8%, 8K +4.4%, 32K -1.3%. New default build against the old one: 2K +2.3%, 4K +16.1%,
8K +2.7% (3/3 rounds each), 20K and 32K +0.1%, decode unchanged. UD-Q4_K_XL (MMQ): ring 192 4K +6.9%, 8K +9.5%, 20K +4.0%.
The ring is staging depth only: first-token full-vocabulary logits are bit-identical (md5) for ring 96 / 192 / 384 on 6 prompts of 8K to 64K; greedy
128-token hashes with static residency are identical on UD-Q4 (12/12) and, on IQ3_S, the few divergent requests occur in ring-96 engines as often
as in ring-192 engines (the known HIP timing divergence of an engine's first requests).
Chunks of 12,288 tokens and more keep 96. The ring is capped at ~800 MiB of blobs. STRATA_PREFILL_RING=N and STRATA_GFX12_RING=0 restore the old value.
Only the Linux gfx12 branch (matrix-core prompt attention active) changes; gfx1151 and Windows keep their rule.
Host: draftingPlan/draftingFetch no longer pass --no-split (only when told noSplit, after the draft layer for the split could not be downloaded); the start asks Control
(splitEngine) to turn the run config into the split engine's before the profile is saved, the second try after a failed start is never split, the load waits at least 10 minutes
for a split start, and the Details line says what runs (or that the server started the layer split). A failed MTP layer fetch ends in one plain note and the 0.2.10 layer split
plan. Screen: the split word counts with two or more cards; the download note and the Drafting hint say Auto is the split with the MTP draft layer (text only, two new strings in nine languages).
Tests: tests/split-app.test.mjs (fake Control, 13 tests); drafting.test.mjs and ui-r3.test.mjs updated.
`strata split-engine apply --config IN --out OUT` turns the run config the app built into the split engine's (same steps as strata launch: rule, strata-split-N beside the
engine, draft layer on disk, lanes 1..8 from "parallel"); nothing is written on a no. Shared helpers in split_engine.py (exe_finder, draft_layer_present, apply_run_config,
split_plan_line), used by the launcher too. `strata drafting plan` asks the rule with the choice in hand (a saved DFlash2 under an Auto on the screen is Auto), says split for
an explicit MTP as the Terminal does, and uses the Terminal's plan line; expected() refuses a conversation memory other than int8 as the start does. --no-split stays.
Tests: AppPlanEqualsTerminalPlan, AppStartsTheSplit replace AppDoesNotSplit; docs/SPLIT_ENGINE.md describes the app.
Strata's gdn_tc.cuh scan (mma.m16n8k16 f16 hi/lo split, state in registers, 2048-token super-blocks, prep kernel per value head) ported
into the split engine's prompt GDN (gdn_tc_prompt.cuh, part 4, CUDA sm_80+; HIP keeps the sub-chunk kernels). FNX_PFX_GDN_TC=1 runs it in
place of k_gdn_ch_prep / k_gdn_ch_rec when the card has the scratch for it. split_gdn_tc_test checks outputs and end state against the
sub-chunk kernels on random inputs (one card slice, C 33..32768) and times both. FNX_GDN_TIME=1 prints the prompt GDN's GPU time per card.
App: `strata drafting plan/fetch` gain --no-split, and every drafting call the host makes (control-runtime.mjs draftingPlan, draftingFetch) carries it, so Auto in the app on
2 NVIDIA / 4 R9700 cards is the layer split with DFlash2 alone, no draft layer is planned or fetched for the split, and the Models screen label and fit say DFlash2 again
(capabilities.dflash2Split stays false). The Terminal's plan is unchanged (split + MTP). The gguf_dir fix of installMtpLayer and the explicit-MTP path stay.
Lanes: split_config cuts the lanes to 1..8 (the engine refuses "--batch must be 1..8"): the AMD four-card default PARALLEL 32, a layer config's own --batch (IQ3_S on 4 R9700
gave 32 and fell back to the layer split) and a profile parallel above 8; serve/server.py parallel_args' split branch cuts the same.
Tests: tests/terminal/test_split_parallel.py (AppDoesNotSplit, LaneClamp), apps/host/tests/drafting.test.mjs.
Found live on the AppImage (srv118): after an app install from the person's own GGUF folder, a plain Auto load on two cards (the plan says split + MTP, the layer is missing) ran installMtpLayer, which asked the installer for the model by name without gguf_dir; the installer started downloading the 111 GB model into the app's data folder (stopped after 3.5 GB). installMtpLayer now passes gguf_dir = the folder of the run config's --native. Test: apps/host/tests/drafting.test.mjs.
The Qwen family's picture file is also looked for in the folder above --gguf-dir and in that folder's models/ (pictures.places() already did);
Qwen3.6 links another size's copy of its picture file instead of fetching it again; on a Mac, when the installed DeepSeek engine reads no pictures,
the download total printed before the engine step is corrected (nothing is fetched before that point); the early Coder builds' temporary-repository
picture file (972,207,968 B, the same tensors) is accepted as the shared one (DSV41_ALT_FILES). test_dsv41_mac_vision's step(5) search starts at
the engine line (it found an earlier family's step 5).
App (apps/host): a plain Auto load (no choice carried, none saved but Auto) fetches the draft layer when `strata drafting plan` says mtp_needed and split,
the same as the Terminal; a Load that carries a choice is as before. probeDrafting reads the plan's split word (capabilities.dflash2Split); the install's drafter
(resolveDrafter) takes it too, so a new install on 2 cards leaves DFlash2 out and gets the MTP layer. drafting.ts / native-settings.mjs
resolveDrafter(choice, family, served, split): Auto is the MTP where the split takes the start, a DFlash2 the person chose stays; NativeModels.tsx passes it, so the
label and the fit use the MTP. `strata drafting plan` gains "split".
Launcher: when the server starts the normal layer split because the all-cards split engine would not come up (metrics engine.split_fallback), the final summary
replaces "Using the all-cards split engine on N cards" with "The all-cards split engine could not start (<the engine's own reason>); running the normal layer split."
Tests: apps/host/tests/drafting.test.mjs; tests/terminal/test_split_parallel.py (PlanTellsTheApp, SummaryTellsWhatRuns).
The split runs the model's draft layer (--mtp) and exits at start without it ("sfnx: --native and --mtp are required"). The plan and the fetch for Auto
resolved to DFlash2 (drafter 1.2 GB, no draft layer), so on 2 x sm_86/sm_120 or 4 x gfx1201 the split was chosen at start, failed, and the server fell back
to the layer split after "Using the all-cards split engine on 2 cards".
(B) strata_control/split_engine.expected(): the rule apply_to_config uses, asked before anything is on disk (model assumed when its file is not there yet).
Runner.split_expected / resolved_drafting and drafting_inputs: Auto where the split will take the start is the MTP, so download_plan and the fetch include the
draft layer (5.2 GB) and not DFlash2's drafter, setup gets --mtp on without --drafter dflash2, and the plan says "Drafting: MTP (the all-cards split engine uses
the MTP draft layer, 5.2 GB)". The app's `strata drafting plan/fetch` (drafting_cmd._inputs) follows the same rule. A person's own DFlash2 / off, --split-engine
off, one card, other cards, three cards keep what they had.
(A) Runner.apply_split_engine: when the --mtp folder is missing or empty at start the split is not picked (Auto: silently, the layer split with DFlash2 starts as
the plan said; "on": one line saying to add --drafting mtp). _auto_gates() is the Auto-only part of the rule, shared by apply_to_config and expected().
Tests: tests/terminal/test_split_parallel.py (Terminal and app plan paths, AMD 4 x gfx1201 and 2, one card, sm_89, own DFlash2, off, missing draft layer).
The split runs the model's draft layer (--mtp) and exits at start without it ("sfnx: --native and --mtp are required"). The plan and the fetch for Auto
resolved to DFlash2 (drafter 1.2 GB, no draft layer), so on 2 x sm_86/sm_120 or 4 x gfx1201 the split was chosen at start, failed, and the server fell back
to the layer split after "Using the all-cards split engine on 2 cards".
(B) strata_control/split_engine.expected(): the rule apply_to_config uses, asked before anything is on disk (model assumed when its file is not there yet).
Runner.split_expected / resolved_drafting and drafting_inputs: Auto where the split will take the start is the MTP, so download_plan and the fetch include the
draft layer (5.2 GB) and not DFlash2's drafter, setup gets --mtp on without --drafter dflash2, and the plan says "Drafting: MTP (the all-cards split engine uses
the MTP draft layer, 5.2 GB)". The app's `strata drafting plan/fetch` (drafting_cmd._inputs) follows the same rule. A person's own DFlash2 / off, --split-engine
off, one card, other cards, three cards keep what they had.
(A) Runner.apply_split_engine: when the --mtp folder is missing or empty at start the split is not picked (Auto: silently, the layer split with DFlash2 starts as
the plan said; "on": one line saying to add --drafting mtp). _auto_gates() is the Auto-only part of the rule, shared by apply_to_config and expected().
Tests: tests/terminal/test_split_parallel.py (Terminal and app plan paths, AMD 4 x gfx1201 and 2, one card, sm_89, own DFlash2, off, missing draft layer).
The split's run config always carries --batch N. ensure_profile still wrote the launch profile's parallel=<card fit> (4 on 32 GB or more) beside
it, and strata serve --profile (cli._effective) refuses a parallel override next to --batch: every 2-card NVIDIA start with the split on Auto
stopped with "parallel override conflicts with --batch/--slots in run config args" before the engine ran. Same for the AMD four-card default.
- runner.ensure_profile: no parallel override when the run config already names its slots (run_config_owns_slots)
- runner.apply_split_engine: the person's --parallel N is the split's lane count (cfg["parallel"] for the rule; taken out again on a refusal)
- cli._effective: a parallel override on the split's run config (split_engine_note) is not a conflict (the app's Concurrent requests); other configs keep the check
- server.parallel_args: "parallel" on the split's run config sets its --batch
- tests/terminal/test_split_parallel.py: Terminal path, app path, AMD four-card, lanes from the flag, the check kept for other configs
fit_start treats a discrete AMD card (backend hip; an APU gives no card memory and is left alone) like CUDA: the KV window walk, the working-cache threshold and
single requests priced from a measured base. The HIP base is 3900 MiB, measured on an R9700 (gfx1201) held by a memory hog: the engine's free VRAM before the expert cache is
the squeeze minus 5045 MiB from 7000 to 5200 MiB (CUDA: minus 6324), so a HIP start holds about 1.28 GiB less. DFlash2 alone at 128K with setup's window of 32768:
7942 MiB 787 slots 864 MiB free 48-52 tok/s; 7000 439 slots; 6400 273 slots (loan keeps 16); 5800 head on the host (31 tok/s); 5200 and 4600 refused with the plain line.
An 8 GB AMD card therefore starts as set up; a 6 GB one walks the window.
The draft-layer drop, the 16-slot prompt-path loan, the own-buffer fallback and the head-to-host move run only on a discrete GPU (strata::spill_allowed:
not Metal, not an integrated GPU such as Strix Halo), with a CHECK for the gate in arch_defaults_test. qsa_kv_resident_min is 4096 (20480 on a Metal build;
STRATA_KV_RESIDENT_MIN overrides): windows of 4096 and 8192 gave the same tokens as 20480 with the cache fixed. fit_start walks the window 20480 / 8192 / 4096 only as
far as the card needs. When every step leaves the cache no slot, the output head (Q6_K, 0.41 GiB) moves to mapped pinned host memory (about 3x slower decode).
Measured on the 8 GB mimic at 128K: DFlash2 alone 391 MiB free, 63-70 tok/s, 32K prompt 19.8 s; MTP asked: layer dropped, 555 MiB free, 61-71 tok/s, 32K prompt 13.4 s;
2 requests 277 MiB free, 46 tok/s per user; 4 requests 171 MiB free, 15 tok/s per user. 12 GB cells unchanged.
A card whose expert cache stays below the 128 slots a prompt-path loan keeps behind used to be squeezed (reserve lowered to 100 MiB free, then the first long
prompt aborted in an mmq allocation) or gave the prompt path 600 MiB of its own buffers and stopped. The loan now leaves 16 slots, the auto lend cap is lifted
for such a card, and every 128-slot floor in the lend, request-time lend and VRAM-shrink paths follows it (g_lend_keep). When the draft layer, the head and a
working cache with room left do not fit in the free VRAM before the layer loads, the start runs without the layer and says so (STRATA_MTP_KEEP=1 keeps it).
An out-of-memory in the layer or head load with --batch names --batch and --max-context. Measured on an 8 GB mimic at 128K with --kv-resident 20480: DFlash2
alone 391 MiB free, 69-73 tok/s, a 32K-token prompt in 19.8 s; MTP asked: layer dropped, 557 MiB free, 62-70 tok/s, 32K prompt in 13.4 s. 12 GB unchanged.
A short section next to the Linux setup: what Strata's own downloader does, the measured speeds (anonymous, 16-24 MB/s whatever the method), and the manual route
with `hf download` and `--gguf-dir`, which setup already supports (checks the SHA-256, never deletes a file in your own folder). No code change.
The context ladder, the fewer-requests step and the one-request-at-a-time fallback of the earlier fit are gone. A start short of VRAM keeps its
context and its number of requests and is moved off the GPU by the backend's knobs (CUDA: the KV window down to the engine's 20480; Arc: the prompt
chunk, then the reserve, then the window), with one plain note. CUDA single requests are priced from the measured base per drafter (DFlash2 alone,
DFlash2 with the MTP layer, MTP, none). Measured on the 8 GB mimic: DFlash2 alone starts at 128K with --kv-resident 20480 and not with 32768.
BASE_SLOTTED_MIB gains (cuda, flash-next) 4990 from the 12 GB and 8 GB CUDA cells; a CUDA start with request slots needs a working cache (263 slots) not the one-slot refusal limit; the one-request fallback keeps the largest context a single request runs (16K on 8 GB); plan() tries fewer requests (down to 2) before one at a time; the draft-layer and head loads name --batch/--max-context when they run out of memory.
drafting.DRAFTER_MIB gains dflash2_selq4 (715); drafting.vram_gb(data_dir, engine) gives the true figure (an engine program that carries 'selector codebooks: Q4_0 rows' and a Q4_0 pack that setup / Load converts), strata drafting plan reports vram_gb and selq4 and its text says the same number, Control passes capabilities.dflash2Selq4 to the page and the fit (apps/host/src/lib/native-fit.ts DFLASH2_SELQ4_VRAM_GB, apps/host/server/native-parallel.mjs) uses it; apps/host/tests/drafting.test.mjs reads DRAFTER_MIB from drafting.py and fails when a copy differs.
Marker per backend: CUDA and HIP builds carry the text (src/core/dflash2.cpp); the SYCL drafter (sycl/src/core/dflash2.cpp, bf16 select kernel) does not read Q4_0 rows, its built binary (P1006 rc1sycl) has no such text so it stays on the bf16 pack, and its loader now refuses a Q4_0 codebook with a plain message instead of tiling it as a matrix. Metal: DF2-METAL's f-0291-metal-selq4. The idea of the Q4_0 selector is nazeshinjite's (nazeshinjite/Qwen3.8-Flash-Next-DFlash2-GGUF).
IQ3_S on 2 RTX A6000, 32K and 62.9K prompts, four pairs against the same build without it: 3.3 to 5.6% faster. Only the memory path changes: the merge reads
device memory instead of the mapped host buffer, so the logits are unchanged (checked on the 53-prompt set).
The selector's two codebooks as Q4_0 rows take 174 MiB (about 88 expert slots) less graphics memory than bf16, with the same drafts: the committed text was byte-identical to the bf16 pack's in 40 of 40 requests (greedy and sampled, everything resident), draft time and acceptance equal, and ahead 1.7-2.2% where the cache is short (RESULTS-DF2MEM.md section 15).
The idea and the bytes are nazeshinjite's: the community GGUF nazeshinjite/Qwen3.8-Flash-Next-DFlash2-GGUF (Apache-2.0) stores the two tables as Q4_0, and tools/dflash2_pack.py --q4_0-selector / --convert-selector writes exactly those bytes (selector_predecessor sha256 55f70049...ffd28, selector_successor 3b596669...a72d2 over the tensor bytes, equal to the GGUF's).
drafting_fetch.convert_selector (called by setup, strata drafting fetch and Load, never by a start or a serve; no network; about 120 MB of memory) checks the Q4_0 pack against PACK_SHA256, converts it in-process and checks the result against CONVERTED_BYTES / CONVERTED_SHA256 (339694592 bytes, 4a42b982...3ecd); anything else is deleted and the bf16-selector pack keeps running. drafting.pack_path returns the converted pack only when it verifies (a marker beside it keeps the hash check off the start path) and the engine program carries the text 'selector codebooks: Q4_0 rows'; a missing, short or corrupt file, an engine without the text or an engine that is not known gets the Q4_0 pack. Includes f-0291-df2-comm (Q4_0 rows in the select kernel, STRATA_DF2_HEAD=native) merged into rc1.
DF2-METAL found that SUPPORTED_FAMILIES and the model-name pattern left Unsloth out, so UD-Q4_K_XL and UD-IQ4_XS (the same Qwen3.8-Flash-Next weights, and the case the
DFlash2 numbers on Strix, R9700 and M5 Max were measured on) started the MTP everywhere.
- strata_control/drafting.py: "unsloth" is a supported family; the name pattern also matches qwen3.8-flash-next-unsloth-<model> (what setup.py and metal/setup_mac.py write:
f"{family name}-{model.lower()}"). Swift (UkisAI's fine-tune), Coder (ISTA-DASLab's coding version) and NVFP4 stay on the MTP. The stale "Unsloth" in the "another model"
comment is gone.
- The app's two copies of the rule follow (apps/host/server/native-settings.mjs, src/lib/drafting.ts): the Drafting dropdown and the memory estimate treat Unsloth like Qwen.
- Tests: Unsloth names on cuda, hip, metal and sycl resolve to DFlash2 alone with the vocabulary; Swift, Coder and NVFP4 names stay MTP with their arguments untouched; the names
are read from setup.py and metal/setup_mac.py; the terminal flow gets the Drafting step for Unsloth; tests/drafting.test.mjs.
On a Strix Halo the prompt path keeps its own buffers at the 16,384-token chunk (prefill_own_keep_chunk) and the cache sizing already reserved them. The
step-down before the prompt path init read cudaMemGetInfo after the pinned RAM complement (which the unified pool counts twice): 860 MiB "free" in the MTP
engine (no chunk fits, the chunk stays, prompts read at 955 tok/s) but 1.7 GiB in the DFlash2-alone engine (no MTP layer, a slightly larger cache), where
1,536 fit: chunk cut to 1,536, prompts at 303 tok/s (UD-Q4_K_XL, 30K prompt 98.9 s against 31.4 s; 8K 28.3 s against 10.8 s). A real allocation failure still
steps the chunk down (the init loop). Only reached when prefill_own_keep_chunk is set: Windows HIP gfx1151 with a bare --prefill auto.
The Monitor tab (serve/web/app.js), the API monitor page (monitor.js, monitor.html) and the Terminal Overview (strata_cli/overview.py) show the token-weighted
average of the run instead of the last request's speed when the server is idle; live values are unchanged. Ported by hand onto rc1 from the community patch
by keinebenutzer: https://github.com/keinebenutzer/STRATA-average-patch (MIT, Copyright (c) 2026 keinebenutzer), made against v0.1.55 c62d5e3a. Read in full
(no network, telemetry, eval or markup in what it adds); none of its files was run.
Tests: serve/test_speed_average.py (the page code run in node, display-only guard), tests/terminal/test_overview.py (edge cases, fallback).
split_config writes "drafting": "mtp" (what the split engine runs), so serve/server.py's drafter step, which resolves Auto to DFlash2 where the
engine has it, cannot add DFlash2 flags to the split engine's arguments; the fallback keeps the person's own drafting word (or none: Auto), so the
layer split that starts instead gets DFlash2 as usual. The rule refuses (quietly in auto) for an explicit dflash2 or off, and for Auto on a card
where the split has not been measured to decode faster than the layer split with DFlash2. Tests: all four cases, the card rule, the server's
drafter step on a split config.
1 The version dropdown no longer shows the "Recommended" tag (the order and the Downloaded tag stay).
2 Gyro-S with Auto sent --mtp-draft-vocab FILE and no --mtp; engine 0.1.55.2 only takes --mtp DIR for the vocabulary and ended with exit code 2.
(a) strata_control/drafting.py reads the engine program (never starts it) for the start message that says what it takes: the vocabulary file alone, --mtp DIR
(the MTP layer still not loaded), or no --draft-fallback at all (then no DFlash2 flag is sent). serve/server.py hands the program to the rule.
(b) An Auto start that picked DFlash2 and ended before it was ready is started once more with the MTP, in one quiet line ("Drafting: MTP (DFlash2 didn't start)",
nine languages); a DFlash2 the person named fails with the engine's reason.
(c) tests: tests/terminal/test_drafting.py (new class), serve/test_drafting_retry.py (a fake engine process), version-picker test.
strata launch --split-engine auto|on|off (settings field, saved in the preset and written to the launch config, so an off survives updates);
STRATA_SPLIT_ENGINE=0/1 for testers. auto (the default) uses the split engine where the rule says yes and says nothing where it says no; on says
why it cannot run; off never. The rule also leaves the layer split for choices the split engine does not honour (DFlash2, custom engine arguments,
replicas, a conversation memory format other than int8). The engine is looked up per card count and architecture (strata-split-N-smCC, then
strata-split-N); sm_120 joins sm_86 as a tested architecture. Tests: modes, auto/on/off, silent refusals, the flag, a saved off.
serve/server.py: when the split engine fails to start, the one line now says why in the engine's words (its 'sfnx: failed:' line) and the
split engine's log is kept beside the normal engine's (it wrote the same file). src/split/src/session.cpp: FNX_DUMP_FIRST_LOGITS=PATH writes
each request's first-token full-vocabulary logits to PATH.<n>, for comparing the prompt path with another engine's.
dflash2_gemm_bench q4 on an R9700 (gfx1201, p620, 8 columns, cold weights, bitwise vs variant 1 in every cell): gate+up 56 -> 44.5 us, down 39 -> 23.4, qkv 26.8 -> 21.4, o 23.6 -> 14.9, fc 61.5 -> 35.2, kv 15.2 -> 10.0, akp/mkp 12.1 -> 6.3, selector 11.1 -> 5.2.
Serve A/B (128K, 8 classes, 3 rounds x 2 reps): drafter 2.5-3.3 -> 2.2-2.8 ms a round, +1.0% tok/s over the old variant (n 6 per class). CLI greedy ids identical across old / new. Other cards keep 17. gfx1151 and gfx1100 compile.
Before this, a start with such a file (UD-Q6_K_XL) read 55 GB of embeddings and spent more than 25 minutes loading before it gave up, so the
server's fallback to the normal layer split came very late. The check runs when the file is opened, before anything is read or placed.
EXPERT_TYPES gains IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, IQ4_NL and Q2_0; a set with such experts may have embeddings, shared experts and an output
head in any type the loader converts to Q8_0; the memory estimate counts the Q2_0 to Q4_0 repack twice; the per-layer embedding table type is
checked. Tests for the supported sets (the IQ3_S set, UD-IQ4_XS), the plain refusal lines and the scan.
On 2x RTX A6000 (IQ3_S, MTP, three residency levels) a 32K prompt reads 37-55% sooner chunk-major and an 8K prompt is level (DF2-MGPU, boston); the default had been on for a layer split since 0.2.7 from a 2x A4000 measurement with streamed experts only. Owner's yes 2026-11-02. STRATA_PREFILL_LAYER_MAJOR set by hand still wins both ways; a single card, HIP and the small-card rule (which sets the variable itself) are unchanged.
Setup writes absolute paths into the run config (cwd and --expert-profile into the Control bundle that ran it, the picture reader and library folders into
an engine package, the DeepSeek package). An update keeps the new Control/engine and the one before, so the second update after setup removed the folder
the config names: the model showed as not installed and a start failed ("Qwen breaks each time Strata is updated").
strata_control/cfg_paths.py is the one resolver: a string that names a file in components/control/<ver>, components/engine/<ver>-<kind> or
engine-dsv41/<ver>-<kind> whose version folder is gone is mapped to the same file in the active Control / the newest engine of its own kind. Paths the
person typed, packs, models, output paths (--expert-profile-save, log) and folders that still exist are left alone.
- profiles.read_run_config applies it in memory, so the doctor, the model list, the launch plan and the host see the current folders.
- runtime.start writes it once into every saved run config, one line in the host log, keeps a caller's expected source digest valid, and stops with the
file named when it exists in no current version.
Tests: tests/terminal/test_cfg_paths.py (real update_control twice, real runtime.start for the Terminal and the app, DeepSeek); 9 of 11 fail without the wiring.
strata_control/drafting.py: "sycl" joins DFLASH2_BACKENDS, so Auto picks DFlash2 (Q4_0 pack, no MTP layer, --draft-fallback none) on an Intel engine.
BACKEND_CONF = {"sycl": 0.6} replaces the CUDA slot steps for Intel (A750, 298 expert slots, serve sweeps: 0.6 was the best of 0.35 / 0.6 / 0.8); a --draft-conf
of the person's own still wins. An Arc A-series card (A3xx / A5xx / A7xx, told by its Windows adapter name or its Linux PCI id) counts as tested; Arc B-series,
Arc Pro and integrated graphics keep "has not been tested on your graphics card yet".
sycl engine: --mtp-draft-vocab is accepted in place of --mtp for the draft vocabulary (the app passes no --mtp with DFlash2); STRATA_DF2_TIMING also prints the
request-end context update and each prompt context update, STRATA_DF2_NO_PROMPT_CTX (measurement only) skips the prompt's context update.
dflash2_bench: a ctx mode that times the prompt's context update.
Measured on the A750 (IQ3_XXS, --prompt-cache 6, 5 rounds): DFlash2 alone is 2.6% behind MTP in decode tokens per second (95% interval -4.8 to -0.4) and 1.1%
behind over time; a fresh 4K prompt reads 4.6 s slower. Tests: tests/terminal/test_drafting.py, test_drafting_hardware.py, test_issues_022.py (117 pass).
On l620 a 49 GiB cgroup limit and the 46.84 GiB arena: the engine's check (arena + blob <= limit) passed, the arena is pinned and never given back, the
engine's own memory (about 3 GiB) came on top, and the cgroup OOM killer ended the engine at the start (journal: "A process of this unit has been
killed by the OOM killer", twice). With the fallback allowed the arena now needs arena + blob + max(RAM to keep, the fallback headroom) under the
hard limit (cgroup, container, job object, --memory-limit-mib), else the RAM copy runs. Without a fallback (hand-set --ple-io ram, shared arena file,
STRATA_ARENA_RAM_FALLBACK=0) the comparison is unchanged. arena_hard_limit_need in ple_residency.hpp, unit cases for a limit just above the arena.
An engine built with CUDA 12.8.61 faults in the prompt path on an RTX PRO 6000 (compute capability 12.0, driver 580) with the batched PLE post-ops on, which is the
default: "an illegal memory access" at every prompt length from 94 tokens, first error in the native PLE postops launch (CUDA_LAUNCH_BLOCKING), and the isolated
verify_batch_parity test faults on native_ple_postops_batch_snap with 4 tokens. STRATA_PLE_BATCH=0 runs. The same source built with CUDA 13.0 (the released
cuda13 package) runs the default at 94 to 20K tokens, where the batch is ~23% faster prompts, so only the 12.8 toolchain changes: ple_batch_default(cudart_version, hip,
major, minor) is false only for CUDART_VERSION 12080..12089 on cc 12.0 (not HIP, not another card, not another toolkit), ple_batch_resolve lets STRATA_PLE_BATCH win in both
directions, and prefill.cpp says once when it turned the batch off. Known issue #220 / #224; CMake already warned for sm_120 builds with CUDA < 13.0. Unit test in
arch_defaults_test; docs/OLDER_GPUS.md.
"Leave 20% for my work" reads the version of the engine the Engine tab selected (snapshot engine version, or the folder named in the run config's exe).
An engine of RAM_KEEP_MIN_ENGINE (0.1.56) or newer gets the env STRATA_RAM_KEEP_GIB = clamp(ceil(0.2 x RAM GB), 5, 8). An older engine, or one whose
version cannot be read, gets the rule of 0.2.9 unchanged: --resident-budget-gib from the 80% rule and a STRATA_RESIDENT_HEADROOM_GIB of 20%; neither
is ever given an argument it does not know. Switching engines between loads swaps the rule; full performance removes either. Tests at 24, 32, 64 and 128 GB
for both kinds of engine, with the preference on and off.
HF discussion 30 ("the model's pack is not complete ()", 7900 XTX, files converted by ISTA's script):
- dsv41_pack_info: an engine that prints nothing is reported by how it ended (signal 11, exit code N), never as an empty string.
- The engine's file check is repeated in Python (dsv41_pack_problems); a pack that does not fit the model files (missing or short file, a file replaced
since the pack was made: pack-source.json size and time, unreadable description) is rebuilt once with one line; when only the engine's own check gave
no answer the pack is kept, one warning carries the reason and the test question decides. The failure line is one line and names the file (the
launcher shows only the first [X] line).
- Both layouts whole (what ISTA's converter leaves) use the six-shard files; without --gguf-dir six-shard files in setup's own folder are seen when all five are there.
- The 0.2.9 Linux DeepSeek package holds code for gfx1200/gfx1201 only: hip_archs_from_pci is a third probe for the card; an unnamed card gets one plain line,
a known card without code is told so (install, install_package), setup warns for its own card pick and reads BUILD.json archs beside a hand-made engine.
Tests: tools/test_setup_dsv41.py (PackCheck, CardsAndLayout, CardGate), tests/terminal/test_dsv41_engine.py (HipCards).
1 Read pictures is a plain checkbox; no hint, no download button (Load does it); a failed picture download keeps one line under the box.
2 A checkbox tick is white in light mode (dark mode unchanged).
3 The progress card shows the step, the numbers and a thin bar; Stop is a text button; no "buttons wait" sentence; reduced motion honoured.
4 The Read pictures choice is kept per version and no longer re-derived from host reports while a job runs; the reader probe keeps its last answer.
5 No separator under Read pictures.
6 Running chat: an SVG ring before the chat's name.
7 The chat's time and token count are a wrapping flex row without fixed widths.
8 A run with many files is one block: delivered-to folder, working folder, and a count that opens the list.
Also: "Leave 20% for my work" is now "Leave room for my work" (hint: keeps 5-8 GB free), nine languages (gui-<lang>.json).
lm_prepare no longer declines for the DFlash2 taps on one stage. lm_run takes a layer's completed output when the next layer starts (the hyper-connection write of its FFN half), puts each sub-chunk's rows at positions >= tap_first_ in the staging, and hands the super-chunk's rows to on_taps in position order when all its layers are done. A layer split and a multi-prompt read still decline. Untested on a GPU at this commit: the equivalence test and the A/B are running.
Measured with no MTP layer loaded against the MTP at equal VRAM (l620 mimic, IQ3_S, chat with a rewind plus four cold prompts, 3 paired rounds): best confidence 0.8 at 1.4K and 3.3K engine slots, 0.65 at 6.6K, 0.5 at 9.2K, 0.5 and 0.35 level at 15.7K; 0.35 loses to the MTP up to 6.6K slots. An explicit --expert-cache N is a budget of largest blobs, about 1.3 N engine slots on IQ3_S, now converted before the steps apply. The slot estimate no longer deducts the drafter from a baseline that already holds the MTP layer, because the DFlash2 is the only drafter.
An engine older than --ram-keep-gib (0.1.52, 0.1.54, 0.1.55, which a person can choose in the Engine tab) stops with "unknown argument";
it ignores an environment variable. The engine still takes the flag for CLI users (the flag wins over the env). Full performance removes the env;
a STRATA_RAM_KEEP_GIB, --ram-keep-gib or RAM mode of the person wins. New host test: an older engine gets no argument it does not know.
gr_down_pf_launch and gr_down_rb_launch set cudaFuncAttributeMaxDynamicSharedMemorySize inside a function-local `static const bool once` lambda. The attribute belongs to
the device: only the first card to launch got it, and on the second card of a layer split the launch of a window of 5 or more rows (51-82 KB of dynamic shared memory
against the 48 KB default) was rejected with "invalid argument", which inside the graph capture surfaced as "fused_gr_read_multi v3: operation failed due to a previous error
during capture" and an engine that answered every request with no tokens. Reproduced on 2 x RTX PRO 6000 (CUDA 12.8): the int3 head with the default mask fails at the
6-token window capture; STRATA_GR_V3=0 and one card work. set_attr_once_per_device (one flag per device, under a mutex) replaces both lambdas; no kernel, launch geometry or
arithmetic changes. Every other cudaFuncSetAttribute site of the tree was already per device (listed in phaseA-tests/blackwell/RESULTS-DF2NV.md section 9).
tools/test_func_attr_scope.py (pytest, CPU) fails if cudaFuncSetAttribute / hipFuncSetAttribute appears inside a function-local static lambda or a std::call_once.
Engine: --ram-keep-gib W (default 0; STRATA_RAM_KEEP_GIB the same). The expert arena is used only if W GiB stay available after it;
otherwise the RAM copy of the hottest experts leaves max(W, the safety headroom) free, and STRATA_RESIDENT_HEADROOM_GIB still wins when set.
The safety headroom is 8 GiB on Windows (the page trim of the copy needs the room) and 4 GiB elsewhere; with W = 0 nothing else changes.
The decision lives in ple_residency.hpp (ram_keep_plan and its parts) and is covered by ple_residency_test over free RAM x W x arena size.
The explicit modes (--resident-budget-gib, --resident-experts) leave max(W, their headroom).
Host: profile 'work' passes --ram-keep-gib W, W = clamp(ceil(0.2 x RAM GB), 5, 8), instead of the 80% budget and the 20% env headroom; the engine
decides from the RAM free at its start. A flag or env of the person wins; full performance passes nothing and takes the earlier version's settings out.
resource-ram.test.mjs covers 24, 32, 64 and 128 GB with the preference on and off. Docs in DETAILS.md.
The default path on a small expert cache streams every expert again for every 512-5,888-token chunk. On an RTX PRO 5000 held to 1.3K / 1.6K / 1.9K engine slots the unfused experts plus layer-major read a 20K / 32K prompt -44% / -50%, -41% / -42%, -14% / -12% sooner (a real 12 GB 5070: -26% at 20K); from a default chunk of 8,192 (2.3K slots) the default path wins and the change costs +1..+25%, so the rule keys on that chunk (strata::small_card_prompt_path, unit-tested at its boundaries). Output changes at rounding level (first-token KL against the FP16 prompt path 0.012-0.018 for every path including the default, top-1 47 of 48), owner's yes 2026-11-02. STRATA_PF_FUSED / STRATA_PREFILL_LAYER_MAJOR set by hand win, STRATA_PF_SMALL=0 turns the rule off; a layer split, HIP, a non-native pack and STRATA_DEFAULTS=026 keep the default.
split_moe_tile_test runs the prompt kernel's weight tiles on a GPU for one expert of each routed-expert type and tests/split/tile_check.py compares them with gguf-py. split_fnx_run is sFNX's logits tool on the same engine objects.
log_on_ defaulted to true and alloc() allocated the pinned staging; a drafter that never ran alloc() (the --parallel slots' clone_alloc) therefore ran the
context update's two D2H copies into a null pointer: an illegal memory access on its first draft. The log is now off by default, init_ring_log() (called
by alloc(), callable by a clone's own allocation on its device) allocates the staging and turns it on, and a drafter without it keeps the pre-log rule.
A request that ends on EOS or max_new skips the draft that would have made the last accepted window's context rows, so every turn left a hole of a + 1
positions (5 a turn, accumulating) that a later turn resuming after it found blank and the log could not rebuild. The serve loop now updates those rows
(one synchronous context update, ~0.4 ms) before it finishes the request: a four-turn agent loop plus a parked return now rebuilds with 0 rows missing.
The cold line prints only when 64 or more rows of the window are missing. copy_log_state_from(src) moves a lineage's log, tag shadow and epoch with a request
between two drafters (solo <-> a --parallel slot). STRATA_DF2_RING_STATS=1 prints the live log, the stash and each rebuild per request.
The draft ring has 2,064 slots (position mod 2064). A request that resumes below the highest position written (a repeated or rewound prefix, a
parked conversation brought back) found the part of its first draft's window [n - 2047, resume) overwritten by later positions or never written,
and the drafter drafted poorly until it refilled: on an RTX PRO 6000 with the prompt cache on, DFlash2 alone was +14% over MTP on cold and live
turns but +3% on restored prompts and -2% on an exact repeat (acceptance 0.63 and 0.59 against 0.79 and 0.70 with the cache off). The repeat case
printed no warning at all, because the token history was equal.
The K/V of the five layers is a function of one row per position, the quantized hidden_norm output (2560 int8 codes and 80 block scales, 2.9 KB;
the K/V is 40 KB and the taps 51 KB). Every context update now copies that row into a chunked, copy-on-write host log of the lineage (a
pinned staging copy inside the captured graph, moved to the log after the stream sync). begin_request rebuilds, with the same kernels on the
same bytes, every slot of the window below `resume` whose tag does not match (a host shadow of the tags says which): the rebuilt K/V is bitwise
the original update's (dflash2_ring_test: rewind, parked, branch, evicted; 578 rows in 4.0 ms, a whole window of 2042 rows in 12.7 ms).
A lineage replaced by another is moved, not copied, into a stash (STRATA_DF2_LOG_MIB, default 256, oldest evicted); a request whose prefix matches a
stashed lineage swaps it back in; a rewind or branch keeps the old branch as a copy-on-write stash entry. STRATA_DF2_RING_LOG=0 restores the old
behaviour. Rows never logged are reported (cold) instead of silently blank.
With DFlash2 as the drafter the MTP stays loaded as the fallback (use_mtp stays true), and three decisions tested use_mtp where they meant
"the MTP chain drew this round's candidates":
- STRATA_SPEC_COST=1 (a GB10 default): T = clamp(chain_rows, 1, S_mtp), but chain_rows is only written by the MTP chain callback, so with DFlash2
and no --draft-conf the window stayed at one row (CLI and serve; code_py 175 vs 258 tok/s on an RTX PRO 6000, serve 166-230 vs 283-397).
- STRATA_SPEC_PROB=1, sampled requests: ver.set_spec_q(mtp.spec_q()) judged DFlash2's deterministic candidates against the MTP's stale q lists
(CLI: 107 of 137 windows), and serve's gated path was not declined for DFlash2.
- the lookup choice read mtp.prob() for the same reason.
spec/window_rule.hpp holds the window rule and the q rule, both loops call it. The default path is unchanged: window_rule_test compares the new
rule with a copy of the old code over 9216 inputs (only the DFlash2 + cost rule + no --draft-conf cases differ). The CLI stats also print the
window histogram for any rule that sets the width and, when STRATA_SPEC_PROB judged windows, their count and source.
The rule asked ram_gb + KV + 1 (64.8 GB for IQ3_S, a 64 GB PC reads 63.1), which left a small card with the KV in VRAM and 720 expert slots at 128K. It is now the smaller of that and arena_gb + LOW_RAM_HEADROOM_GB + KV (62.1 GB for IQ3_S; the KV's RAM comes out of the engine's RAM copy or file cache, not on top). It never asks more than before; --kv-streaming on|off and WSL are unchanged; the size stays 32768.
Measured (phaseA-tests/gh-sweep-1031/KV-128.md): IQ3_S, 12 GB mimic on l620 under a 45 GiB cap, 756 -> 1,267 slots, 32K prompt 21-34 s -> 12-17 s, 118K prompt 76-122 s -> 45-63 s, decode flat to +10%, engine RAM unchanged; on the owner's RTX 5070 (single runs) 924 -> 1,585 slots, 32K prompt 130 -> 79 s, RAM copy 30.8 -> 28.9 GiB, no paging. Decode there is within noise.
Tests: need numbers, never-more-than-before over every model/context/KV, the rule at the edges and for WSL, setup writing --kv-resident on a 63.1 GB PC with a 12 GB card and not at 61.9 GB or at 32K. The golden baseline gets --kv-resident 32768 for the 64 GB / 32 GB card IQ3_S and recommended entries. Setup and tuning tests on boston: same 24 failures as f-0291-int3 (not touched), 7 more pass.
The fallback's RAM copy (previous commit) now uses the --resident-budget-gib form: the budget is what is available minus the headroom
(default 8 GiB, STRATA_RESIDENT_HEADROOM_GIB changes it), so the experts outside the copy are read unbuffered instead of through the mapped
view whose pages grow the working set until available RAM is near zero. On Windows the complement copy releases the mapped pages to the
standby list every 4 layers (the VirtualUnlock call that already ran once after the copy): available RAM fell to 0.9 GiB for ~10 s during the copy
on a 64 GB PC with 44.6 GiB available. No data changes.
Measured on the RTX 5070 PC (128K, 44.6 GiB available): lowest available RAM 4.05 GiB, 2,000-token answer 25.2 tok/s (9.4-10.1 without the fix).
Auto and the default pick DFlash2 with the Q4_0 pack for Qwen3.8-Flash-Next and Gyro; Intel SYCL stays MTP (one line, DFLASH2_BACKENDS). The GUI dropdown, TUI, strata launch, setup (--drafter), serve/server.py, the host's Load and the drafting command agree. A new install fetches only the selected drafter, a switch fetches the other once (never at start), the draft vocabulary comes with the DFlash2 pack. A layer split or several requests load only the MTP and say so in one line; no drafter on disk says so too. Fit, plan and slot estimates use the chosen drafter (0.89 GB DFlash2, 1.08 GB MTP).
Decode: ggml's vec_dot for IQ4_XS, IQ3_S, IQ3_XXS and IQ2_S added to the mmvq expert launchers beside IQ4_NL and Q4_0.
Prompt: weight tiles for Q4_0, IQ4_NL, IQ4_XS, IQ3_S, IQ3_XXS and IQ2_S in moe_mma_iq.cuh (ggml's dequantization rounded once to f16).
Loader: Strata's Q2_0 down experts are repacked to Q4_0 blocks (lossless, twice the bytes).
When the expert arena does not fit the available RAM (#4846), the run read every expert through the plain file source with no RAM copy:
9-10 tok/s on the owner's 5070 PC (44.6 GiB available for a 46.84 GiB arena). The same engine with a RAM copy of the hottest experts
(--resident-budget-gib 40 / --resident-experts) gave 24-31 tok/s at the same RAM. The fallback now sets the --resident-experts flags
(resident_cpu_experts, resident_pin, resident_soft, 4 GiB headroom) when a static expert profile is set; if the file cannot be opened the
flags are cleared with the rest of the fallback. STRATA_ARENA_FALLBACK_RESIDENT=0 keeps the plain file source.
Not built, not run: the patch is untested.
Adds the IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS and Q2_0 types to the GGUF type table and a
loader step that, only when a file has routed experts in IQ2_S, IQ3_XXS, IQ3_S or Q2_0,
dequantizes its dense matrices, shared experts, projections and embeddings (K-quants,
IQ4_XS, IQ4_NL, BF16, F16) and writes them again as Q8_0, and the routers, injections
and SSM gates as F32. Other files are not touched. The routed IQ expert kernels are not
there yet, so the set still falls back.
Checked against gguf-py on the ISTA IQ3_S file: 762 tensors converted, the sampled ones
all within the Q8_0 bound (F32 targets exact), every source type covered.
Per-run source choice, no mid-run fallback. With the MTP loaded the DFlash2 gathered a second copy of the 106299-row draft head (212.9 MiB on Flash-Next); it now uses the MTP's when the type, the vocabulary ids and the layout match, and the expert cache is sized without the copy. The cache sizing also prices the DFlash2 head from the vocabulary file when there is no MTP.
With DFlash2 as the drafter the MTP stays loaded as the fallback (use_mtp stays true), and three decisions tested use_mtp where they meant
"the MTP chain drew this round's candidates":
- STRATA_SPEC_COST=1 (a GB10 default): T = clamp(chain_rows, 1, S_mtp), but chain_rows is only written by the MTP chain callback, so with DFlash2
and no --draft-conf the window stayed at one row (CLI and serve; code_py 175 vs 258 tok/s on an RTX PRO 6000, serve 166-230 vs 283-397).
- STRATA_SPEC_PROB=1, sampled requests: ver.set_spec_q(mtp.spec_q()) judged DFlash2's deterministic candidates against the MTP's stale q lists
(CLI: 107 of 137 windows), and serve's gated path was not declined for DFlash2.
- the lookup choice read mtp.prob() for the same reason.
spec/window_rule.hpp holds the window rule and the q rule, both loops call it. The default path is unchanged: window_rule_test compares the new
rule with a copy of the old code over 9216 inputs (only the DFlash2 + cost rule + no --draft-conf cases differ). The CLI stats also print the
window histogram for any rule that sets the width and, when STRATA_SPEC_PROB judged windows, their count and source.
Per-run source choice, no mid-run fallback. With the MTP loaded the DFlash2 gathered a second copy of the 106299-row draft head (212.9 MiB on Flash-Next); it now uses the MTP's when the type, the vocabulary ids and the layout match, and the expert cache is sized without the copy. The cache sizing also prices the DFlash2 head from the vocabulary file when there is no MTP.
With DFlash2 as the drafter the MTP stays loaded as the fallback (use_mtp stays true), and three decisions tested use_mtp where they meant
"the MTP chain drew this round's candidates":
- STRATA_SPEC_COST=1 (a GB10 default): T = clamp(chain_rows, 1, S_mtp), but chain_rows is only written by the MTP chain callback, so with DFlash2
and no --draft-conf the window stayed at one row (CLI and serve; code_py 175 vs 258 tok/s on an RTX PRO 6000, serve 166-230 vs 283-397).
- STRATA_SPEC_PROB=1, sampled requests: ver.set_spec_q(mtp.spec_q()) judged DFlash2's deterministic candidates against the MTP's stale q lists
(CLI: 107 of 137 windows), and serve's gated path was not declined for DFlash2.
- the lookup choice read mtp.prob() for the same reason.
spec/window_rule.hpp holds the window rule and the q rule, both loops call it. The default path is unchanged: window_rule_test compares the new
rule with a copy of the old code over 9216 inputs (only the DFlash2 + cost rule + no --draft-conf cases differ). The CLI stats also print the
window histogram for any rule that sets the width and, when STRATA_SPEC_PROB judged windows, their count and source.
Per-run source choice, no mid-run fallback. With the MTP loaded the DFlash2 gathered a second copy of the 106299-row draft head (212.9 MiB on Flash-Next); it now uses the MTP's when the type, the vocabulary ids and the layout match, and the expert cache is sized without the copy. The cache sizing also prices the DFlash2 head from the vocabulary file when there is no MTP.
With DFlash2 as the drafter the MTP stays loaded as the fallback (use_mtp stays true), and three decisions tested use_mtp where they meant
"the MTP chain drew this round's candidates":
- STRATA_SPEC_COST=1 (a GB10 default): T = clamp(chain_rows, 1, S_mtp), but chain_rows is only written by the MTP chain callback, so with DFlash2
and no --draft-conf the window stayed at one row (CLI and serve; code_py 175 vs 258 tok/s on an RTX PRO 6000, serve 166-230 vs 283-397).
- STRATA_SPEC_PROB=1, sampled requests: ver.set_spec_q(mtp.spec_q()) judged DFlash2's deterministic candidates against the MTP's stale q lists
(CLI: 107 of 137 windows), and serve's gated path was not declined for DFlash2.
- the lookup choice read mtp.prob() for the same reason.
spec/window_rule.hpp holds the window rule and the q rule, both loops call it. The default path is unchanged: window_rule_test compares the new
rule with a copy of the old code over 9216 inputs (only the DFlash2 + cost rule + no --draft-conf cases differ). The CLI stats also print the
window histogram for any rule that sets the width and, when STRATA_SPEC_PROB judged windows, their count and source.
- With a loaded model that runs without pictures (no internet, no room), the existing "Picture reading needs a few more files" offer says the one short line
instead of a second warning beside it; the offer is hidden while a download or load runs.
- "Download picture-reading files" ends with a toast; a model that is loaded without pictures is told to load again.
- Nine-language catalog: three more sentences. UI preview fixture: context 128K with its marker; the Chrome driver reports dialogs that are new after the click.
A model that is downloaded, in its own GGUF folder, with Read pictures on (the default) and no picture file started an install job on Load:
`--data-dir <app models folder>\Strata-data --gguf-dir <own folder> --vision yes`. That re-checked the model files (finish marks written in the
person's own folder), prepared a second pack and downloaded the draft layer into the wrong data folder, and rewrote data_dir in app-settings.json.
- strata_control/pictures.py (new): finds the pinned picture file (size and SHA-256; beside the model, one folder up, in the model's data folder) or
downloads it once into <data folder>/models, shared; the data folder is the one the model's run config names (packs/<tag>, mtp/rt). Works out the run
config's vision section for the engine in use with the same rule as setup (CPU reader under 24 GB, GPU reader from 24 GB, AMD/Intel Arc/Mac readers).
No internet is one plain sentence after a short probe; a start never downloads (acquisition guard).
- native-control.mjs: Load (kind load, vision on) fetches the files through Control, writes the vision section, shows plain-word progress with a bar and
loads. No internet, no room on the disk or a failed download: the model loads without pictures, nothing saved about pictures, one short line on the
page. A restart or a tuning run never downloads. kind pictures does the same without a load (the "Download picture-reading files" button).
A model whose only gap is a deleted picture file is no longer "missing files". installArgs: an installed model is prepared again in the folder its
own run config names, never the app's models folder, and that folder is never written to the settings; the Drafting plan looks there too.
- Models page: the Load shortcut no longer needs the picture files prepared for the families Load can fetch for; the button and the line use the new kind.
- Tests: tests/terminal/test_pictures.py, apps/host/tests/load-pictures.test.mjs, UI preview fixture scenario, Chrome driver script, nine-language catalog.
Coder: expert-pruned 256-expert Coder, M5 Max, 5 interleaved pairs, identical greedy tokens in all 40 runs per class, over --spec 2: story -2.5%
(0/5), code +7.1%, JSON +9.3%, agent turn +12.5% (5/5 each). It has no Mac setup entry yet; the name is what a config for it would pass.
fp16 decode (PR 4807): setup writes STRATA_METAL_F16=1 into the config's env for a resident IQ3_S or Q2_0 pack (STRATA_METAL_F16=0 there turns
it off; streaming and the other sets are untouched). The owner's rule was an option unless it passes the 61-task quality suite: off / on / off
again on IQ3_S 54/61 each, on Q2_0 54/61 each, the same 7 tasks failing in all arms, no family drop, no new hallucination flag, 0 pass/fail
changes between two runs of the same arm; decode +14% (IQ3_S) and +15% (Q2_0) in the suite runs, top-1 96-97% and mean KL 0.009-0.016 against fp32
(phaseA-tests/mac-q/RESULTS.md sections 20, 22, 30).
On a Strix Halo under Windows the first cudaGraphLaunch of a doorbell window sometimes never returned: the GPU had already rung layer 0
("GPU rang 1, flags served 0") while the main thread, which serves the doorbells, was still inside the launch call. HostGraphLaunch (the
CUDA instrumented-launch helper, STRATA_VERIFY_ASYNC_LAUNCH=1) is now compiled for the Windows HIP build too, and arch_defaults sets the
switch for Windows gfx1151 (launch_default_env; STRATA_GFX1151_DEFAULTS=0 or STRATA_VERIFY_ASYNC_LAUNCH=0 turns it off, a user setting is kept).
Measured on the strix box, UD-Q4_K_XL, default queues, 19,853 slots, interleaved within one job: old launch 14 hangs in 75 starts
(t2 4/20, t5 3/25, release build 7/30), helper-thread launch 0 in 75 (+ 8/8 on the final build); decode 48.2 vs 48.4 tok/s; output tokens
identical; IQ3_S 8K prompt 1,413-1,427 vs 1,419-1,423 tok/s with identical output; quality quick subset PASS, agent loops 2/2.
Also STRATA_LAUNCH_PROBE=1 (diagnostic): launch-call time, GPU doorbell state during the call, window graph node count.
(cherry picked from commit 93b9264e30054996d1cb9e89d27d211ed1156b15)
The default changes the answers and gained +0.2% to +0.6% on an Arc A750 at 16K, 32K, 64K and 128K (3 runs per arm up to 64K, one pair at 128K; first-token KL 1e-9, same top-1 and tokens): its prompts are bound by expert streaming over PCIe. Under the 5% bar, so Alchemist and other devices keep the warp kernel unless STRATA_SELECT_GEMM=1. Battlemage unchanged (+19% to +28% at 100K to 128K on 2x Arc Pro B65).
After the 20 s wait and the give-back of the PLE lock, a shortage no longer registers the arena anyway (a 60 GiB host stalled in reclaim until it was power-cycled). The run uses the --mmap-experts expert source, says so in one line with one next step, and the start never refuses. A hand-set --ple-io ram and --shared-expert-arena are kept as asked (warn only); a hard cgroup limit below the arena falls back instead of refusing; STRATA_ARENA_RAM_FALLBACK=0 is the old behaviour. If the file cannot be opened either, the arena loads the old way. Decision rule and tests in ple_residency.hpp / ple_residency_test.
The engine gives 5 experts per word on two or more RDNA4 cards that together hold at least half of the experts (4 x R9700), but strata launch --plan, the Speed screen and the app's Speed label called Full quality the default there.
dsv41_fit.rdna4_faster() applies the engine's rule with the plan's own numbers: one card the run uses (on Windows one AMD card): yes; several: yes when the plan puts at least half of the set's experts on them (3 or more R9700 for every official set; two hold 37-47%, so they keep 6). strata launch uses it with the chosen model; the model catalog carries rdna4_faster per DeepSeek size for the app (NativeSize, Speed label); --speed help and docs/MODELS.md say it. Other makes and older Radeons are unchanged.
One R9700 holds under a fifth of a 2.25 bpw set, so the CPU lane and the SSD set the speed, as on the 16 GiB cards and the multi-card RDNA4 rule that already take 5.
1 x R9700 on p620 (125 GB RAM), 2.25 bpw, frozen expert profile, 5 interleaved pairs, 5 against 6 experts: decode 15.24 against 13.08 tok/s (+16%, 5 of 5 pairs), a scripted agent loop -24% wall time (5 of 5), the long-read loop -12% (5 of 5). KL against the 6-expert run 0.030 (6 against 6: 0.008); against the original model 0.1106 against 0.1067-0.1071. quality027 quick: 18 of 19 with 5 experts and with 6, the same task flagged in both.
ds4_experts_rule.hpp: experts_per_word() takes single_rdna4 (the engine passes rdna4() with one device); DS4_DEFAULTS=026 gives the model's 6 back. The Terminal's Speed words (strata launch --plan and screens), the app's Speed label (arch added to RunCard) and docs/MODELS.md say the same. Several RDNA4 cards keep the existing rule (they must hold half of the experts); the screens still do not mirror that one.
In the single-request speculative paths the accepted rows of a verify window were committed first and the output loop cut to max_new or EOS afterwards, so a request's last window could park a snapshot with tokens the client never received. conversation_prefix() needs the whole snapshot to be a prefix of the next prompt, so the snapshot was refused and the older checkpoint was used (about 6% of a 12K-token prompt read again).
serve_stop_bound() caps the accepted rows at the rows the request can still emit and ends at the first EOS, before pl_commit_async. Merged prompt windows keep committing every row. HIP only (STRATA_USE_HIP), where it was measured.
Patch and policy test (2,736 cases against the old loop) by hanchengme, from the issue's attachment. Two changes on our side: it is on by default, STRATA_SERVE_STOP_BOUND=0 turns it off; the test is registered in CMake.
Measured on 4 x R9700, Qwen3.8-Flash-Next IQ3_S, 2 replicas x 2 cards, --batch 8 --batch-groups 2, four 11-15K-token conversations with 768-token answers, conversations opened one at a time: 4 of 16 continuations fell back to the older checkpoint without it, 0 of 16 with it (reuse 98.36% against 99.74%). Opened together as batch rows both builds miss the same continuations (slot overwritten before it is parked); not changed here.
GuidedSetup filled its family and size once at mount. When the catalog had not arrived yet it fell back to the first size in the list (Q2_0, a 66 GB download) even with an installed IQ3_S on the computer. Only the person's own choice is state now; the start is worked out from the catalog on every render (startingChoice in native-catalog.ts): a ready installed version first (the recommended one if it is among them), else the recommendation. No design change.
The owner's audit now scans every app sheet, and f-0291-ui's appended blocks broke
its rules (flat, tokens only, shadow only on surfaces that overlap content):
- .native-progress (#4802): gradient + --st-sh-pop shadow -> flat --st-accent-tint
fill, the accent border stays (it still stands out; no shadow).
- .approvaldock > .askcard/.toolcard (#4725) and .approvaldock > .askq (#4244): the
--st-sh-bar shadow dropped; the accent border that marks them stays.
- .st-code .st-btn--icon[data-copied] (#4748): the literal rgba(255,255,255,.08)
is now color-mix(--st-code-ink 8%, transparent), the same recipe liquid.css uses
for the code block edge.
- .quote-selection (#4785) floats beside selected words, so it is added to the
audit's ELEVATION_ALLOWED list instead of losing its shadow.
Merged f-gui-look-snap (682c21d64): the owner's uncommitted wt-gui-look work as one
commit on 4cf0d8cd3, 32 files +343/-216 plus the two untracked files
apps/host/src/styles/liquid.css and apps/host/scripts/browser.mjs. Line endings were
normalised from CRLF to LF when the snapshot was taken (the raw working-tree diff of
~14,800 lines is the CRLF flip; nothing of it is carried here). The approved UI list
(f-0291-ui, 14 items) is already in f-0291-int3.
Three-way merge result: no textual conflicts. Files both sides changed, and how each
resolved:
- components/NativeModels.tsx (int3: progress card, context memory + KV hint; owner:
Manage models ghost/sm + compact dialog, "Other local models" class, Expert scroll
waits for layout): disjoint hunks, both kept.
- lib/plain-errors.ts (int3: out-of-VRAM rule; owner: unwrap {"error":..} / HTTP 400
bodies before the rules run): both kept; the unwrap runs first, so a wrapped
out-of-VRAM message still reaches its rule.
- screens/Serving.tsx (int3: two addresses, CopyBtn; owner: empty-state "Connect an
app" is the primary button): disjoint, both kept.
- screens/Settings.tsx (int3: While-an-answer-is-written row, Select import; owner:
Engine panel gets the "Engine" heading): disjoint, both kept.
- styles/app.css (int3: appended blocks + the reading-comfort rule; owner: edits in
place): auto-merged, both kept.
- styles/onboarding.css (int3: language Select rules appended; owner: welcome
headline, badge ink, glow removed): disjoint, both kept.
- components/TuneRunView.tsx also auto-merged (int3 +1 line, owner: progress bar uses
transform).
M5 Max, DeepSeek 2.25 bpw, 3 interleaved pairs: with the lent room a 639-token prompt took 6.2-7.6 s against 4.1 s with DS4_BORROW=0 (same bytes
read, the compute waited for reads equally long), 8K prompts -3.9%, 32K -20% (borrow also selects the 32,768 chunk); and the engine died
("mtl: wrap 7763574784 failed", exit 4) in 8 of 9 runs of a mixed 165/600-token/8K sequence that the build with DS4_BORROW=0 finished 3 of 3.
Only Metal changes (one_gpu_or is under STRATA_METAL); CUDA and HIP keep devs.size() == 1. DS4_BORROW=1 turns it back on.
The plan said 2,947 experts for one R9700 and 13,180 for four; the engine held 2,366-2,566 and 12,147-12,192. It took 4.5 GB of dense weights and 1.15 GB of reserve per card and forgot the driver context, the prompt buffers and the dense weights every card repeats. Constants now come from the engine's startup lines (f-042-maya-rc d93547aed): 2.25 bpw one R9700 2,543, four 12,260; boston 2 x A6000 within 1%, A4000 and RTX 5070 within 5%.
strata launch --model-path on files that stay where they are printed 'tight: needs more disk space' with nothing to download: the menu row counted the whole download against Strata's folder. With own files the verdict now comes from the memory words alone.
Router: ds4_route and ds4_route_publish left an all-NaN gate row's pick at 0x7fffffff and read sc[0x7fffffff] (threadgroup memory) and
vram[0x7fffffff]; the test process dies with SIGBUS on the old kernel. ds4_route_fallback_pick takes the lowest expert not picked yet
(the CUDA/HIP guard's rule) and raises a flag in a buffer of the Metal host; router_nonfinite_take synchronises and reads it
(metal_host::router_bad_take), so ds4_main ends the request with the error it already has. The warning names the GPU on Metal; the
CUDA/HIP text is unchanged. Test: metal/tests/ds4_route_guard_parity.cpp (finite, all NaN, +-inf, half NaN, fewer than k finite).
Dense Q4_K: ds4_gguf.cu rows() sends a dense matrix of a type metal_host::dequant_rows does not know to iq::dequant_rows, whose CUDA
launch has no Metal dispatch and aborts the engine. ds4_dq_q4k (ggml's dequantize_row_q4_K, the Q5_K kernel without the fifth bit) is
added; any other dense type now fails with a plain message naming it. Test: metal/tests/ds4_dq_parity.cpp (Q4_K, Q5_K, Q6_K, Q8_0, BF16
against ref_quant, padded row stride).
Keeps the wide lookup sizes for copy-heavy text (a rewrite matches long) and leaves lists and prose, which match short, at the 8-row cap. Off by default. Tried and reverted in the next-but-one commit: it did not remove the prose cost of 16 rows and cost rewrite gain.
Port of pull request 4737 by InB4DevOps (commit c14ee15d9 on v0.2.8). The mixer's input-only QSA/GDN work runs on graph branches; scheduling only, no arithmetic change.
STRATA_SM121_PERF=0, STRATA_DEFAULTS=026 or STRATA_SIDE_MIXER=0 turn it off. Measured here on a Spark, IQ3_S, greedy: ids identical 48 of 48; alone +1.0..+2.9% per class, with mode 8 +3.6% pooled.
Not yet measured at parallel 4.
Port of pull request 4699 by InB4DevOps (commit 76c0aa601 on v0.2.8). The existing mode 8 kernels are selected for compute capability 12.1 when the sm_121 table is on;
STRATA_SM121_PERF=0, STRATA_DEFAULTS=026 and an explicit STRATA_EXP_MODE keep the old mode 0.
Measured here on a Spark, Qwen IQ3_S, serve, greedy: bit-exact (ids identical 48 of 48), with the side mixer +3.6% decode pooled over 8 prompt classes (4 rounds), mode 8 alone +0.3..+2.4%.
prompt_attn_wmma_kernel was limited by its LDS (21,648 B a block, 5 waves a SIMD), not by VGPRs (amdgpu_waves_per_eu
and amdgpu_num_vgpr change nothing here). Two exact changes, on by default:
- STRATA_PA_TRIM (default 2; 0 off, 1 first level): the score and probability buffers hold the 12 real query rows
instead of 16 (rows 12..15 of the WMMA are zero queries nothing reads; the 4th wave skips the softmax of them),
and at level 2 the staged V rows are 72 bytes instead of 80. LDS 21,648 -> 19,008 -> 17,984 B, 7 waves a SIMD.
- STRATA_PA_RPF (default on; 0 off): chunk ci + 1's selected-cell id and page-table reads are issued before chunk
ci's compute.
The arithmetic and its order do not change. hip_prompt_attn_wmma checksums are identical across all variants (32K
context 8f3a85a9cb885e7e, 8K context de4684fb3f370000), and engine first-token logits with every exact lever of this
branch on against off are byte-identical on a 4.5K and an 8K prompt (IQ3_S pack). Kernel time, gfx1151, int8 KV,
STRATA_PA_FAST: 32K context / 2,048 queries 11.05 -> 9.95 ms, 8K context / 4,096 queries 21.76 -> 18.90 ms.
The idea of raising the kernel's occupancy is the qsa3 change of halo-box/strix-llama.cpp PR 213 (MIT, The ggml authors);
the means here differ and the code is ours.
Needs the owner's yes: the existing opt-in STRATA_SELECT_WMMA=1 with STRATA_SELECT_ALT_MIN_TOKENS=28672 becomes the gfx12 default in
gfx12_extra.hpp, with its own log line. A chunk whose context is below the floor keeps the warp kernel, so a prompt that stays under about
28K tokens is bit-identical (first-token KL 0.00000 on twelve 8K-24K prompts). Off: STRATA_SELECT_WMMA=0, STRATA_GFX12_DEFAULTS=0 or
STRATA_DEFAULTS=026; a variable the user sets is kept. Chosen over PR 3073's simt2 (author Vop3p) on the data.
R9700 (p620, IQ3_S, serve, interleaved pairs, prompt tok/s against unset, 3 pairs): 32K +1.83% (3/3), 98K +5.74% (3/3); first-token KL on seven
30K-96K prompts mean 0.0018, max 0.0055, top-1 7/7; needle recall 25/25 (8K to 96K, 5 depths). Without a floor: 24K +3.34%, 32K +4.5..6.0%,
98K +6.9..7.3% but values change from 8K up (KL 0.0056); simt2 is slower below 32K. arch_defaults_test now clears the variables this
default sets before its integrated-GPU checks; gfx12_extra_test covers the default and its off switches.
warm_flow() runs cublasSgemm / cublasGemmEx on tiny scratch inputs at load. The Metal compatibility layer (metal/include/metal_compat/cublas_v2.h) only has the dispatches the real flow uses: its sgemm_tn found no host view for the scratch buffers and the call ended in strata_metal_dense_gemm_missing(), an abort (exit code -6) on every start, restarted in a loop by the server. start_warm() now returns at once under STRATA_METAL. Found on an M5 Max with the 2.25 bpw set: the v0.2.9 engine answered the four checks; this tree aborted until DS4_WARM=0 was set.
The final hyper-connection mixer of the verify window and of the draft layer as one fused_gr_read_multi instead of one gr_read per
token (PR 3264, author bsorensen110). Kept apart from the main gfx12 table in gfx12_extra.hpp so it merges with the table on
another branch without touching it; STRATA_GFX12_DEFAULTS=0, STRATA_DEFAULTS=026 and a variable the user sets decide first.
R9700 (p620, IQ3_S, serve, static residency, 8 interleaved pairs with an A/A arm): json_extract +0.82% (6/8), agent +0.63% (6/8),
code_py -0.96% (3/8, that prompt's A/A moves 3.5% a pair); greedy ids identical in 16/16 and 15/16 request sets.
On gfx1151 the compiler folded x * x into the first add of block_sum (fma of this lane's x with the other lane's
square), a different rounding than the sum of two rounded squares, so about 4% of rows came out with a scale one ulp
off. An empty asm on the square stops the fold (__fmul_rn compiles to the same fma). Checked in the assembly
(v_mul then v_add) and on the device: hip_qsa_nr_parity 44 of 44 OK, every output element bit for bit, T 1..8191;
engine first-token logits with STRATA_QSA_FUSE_NR / STRATA_QSA_FUSE_GATE on and off are byte-identical on a 4.5K and
an 8K prompt (IQ3_S pack).
The exact-column instantiations 13..16 of the wide kernels (native MMVQ, the wave variant, the packed Q8_0 and Q6_K launches,
the BF16 multi GEMV) are built only above 12, MAX_NCOLS follows the cap, and kVerifyLongT follows a wide cap (a cap of 12 keeps
--lookup-max-t at 12). Measured for the narrow/wide comparison in phaseA-tests/pr-priorities/RESULTS-NV2.md.
Mode 2 of STRATA_SELECT_SIMT was added for HIP in the previous commit as an opt-in. For the RX 6000 cards its contributor measured it
on (gfx103x) it now joins the one community switch: on by default with STRATA_AMD_COMMUNITY_TUNING on, any STRATA_SELECT_SIMT value
the user sets wins, and no other card changes. Values change in the last bits (another summation order); the contributor's numbers are
labelled as theirs in docs/AMD_HIP.md and have not been reproduced (no gfx103x card here). amd_community_test checks the default and
that the switch turns it off.
data/quant_quality/v1/set.ids.json is tokenized for Qwen (ids up to 248,069); fed to the DeepSeek engine it gave nonsense text and, for the
runs that dump logits, a late "token out of range". read_score_ids now takes the model's vocabulary and refuses the file at once, naming the
segment, the id, its position and the fix; quant_quality.py run does the same against its config's tokenizer folder (or the vocab_size that
`ids` now records in the ids file it writes).
The engine's #879 check only sees logits that come back to the host; the Metal engine picks tokens on the GPU, so a half-precision overflow (STRATA_METAL_F16) would show as wrong text, not as an event. With the variable set, each head window's logits are scanned: a line for the first five windows that held a non-finite value, a count every 500 head windows. Off by default, no cost unset.
Controls of glorysdj's pull request #4185: unset means on for a Battlemage device and off for every other device (strata::bm_default_on); a set value still decides. Measured results are in phaseA-tests/pr-priorities/RESULTS-INTEL.md.
Unset, the output is unchanged (the build with the change, unset, gives the same greedy ids as the build without it in 12 of 12 runs, gemv_parity / gr_parity / expert_parity pass). With STRATA_METAL_F16=1 the decode windows of a resident model use half-precision GEMVs (Q2_0, Q3_K-Q6_K, IQ4_XS) and the Q2_0 routed experts.
M5 Max, spec 3 + min-p 0.5, 3 interleaved pairs, decode tok/s unset -> on: Q2_0 story 105.0 -> 114.9 (+9.4%), code 125.6 -> 143.7 (+15.5%), json 130.8 -> 135.1 (+3.3%, one noisy pair), agent 116.1 -> 131.6 (+13.4%); IQ3_S (dense GEMVs only) story 93.1 -> 104.3 (+8.8%), code 103.6 -> 118.5 (+12.5%), json 122.5 -> 135.4 (+10.6%), agent 103.8 -> 117.8 (+12.3%).
It changes arithmetic: teacher-forced over 4,995 positions against the same build unset, Q2_0 top-1 97.12%, mean KL 0.0094 (p99 0.119), perplexity 4.456 against 4.442; IQ3_S top-1 96.16%, mean KL 0.0160 (p99 0.211), perplexity 4.119 against 4.135. The half range is not guarded (see the PR). Off by default.
After an app or Control update the engine is never touched, and the app said nothing. Strata Control keeps what the last update or check found newer
(update_available of 'strata components engine', Qwen engine and DeepSeek engine; read without the network); the host passes it to the app as
engine.updateAvailable and the Models screen shows one card in plain words with the existing Update engine job as its button (disabled while a job
runs or a model is loaded). A DeepSeek engine update is reported as an engine update, not as 'Strata is updated'. JavaScript only; the Control side is
on f-0291-keepcurrent.
c9e0d6688 uses it in a __global__ function; nvcc refuses a constexpr host function there (error: calling a constexpr __host__ function from a __global__ function is not allowed). HIP accepted it, CUDA did not: the f-042-maya-dq line did not build for NVIDIA.
The old kernel (the gfx11 default since 0.2.7, picked for speed) gave different dense weights on every run on a Strix Halo: layer 0 of a
200-token prompt differed by 27% between two runs of the same engine on the 2.25 pack (Q5_K) and by 26% on UD-IQ1_S, while dq_rows_kernel2
is bit-identical run to run and 1.2-1.3% from the window path (GEMV, no dequantizer). A gfx11 device now defaults to kernel2 whatever
DS4_DEFAULTS says; DS4_DQ_OLD=1 forces the old kernel there and elsewhere, but only for a type whose staging fits (Q4_K, Q5_K, Q2_K:
dq_old_ok); Q6_K (840 bytes against the 704-byte buffer) and the rest always use kernel2. 2.25 prompt speed on the Strix Halo: 4K 166.1 -> 164.9
tok/s, 8K 297.9 -> 294.2 (3 runs each, -0.7% / -1.2%). UD-IQ1_S's long prompts and agent loop work.
Port of PR 3307 by gopinath87607 (rebased onto f-0291-int-trial, three commits squashed; the mtp.cpp signatures keep the
gated-proposal DraftResult argument and the prompt-helper arming keeps the failed-allocation cudaGetLastError).
STRATA_MTP_ROOM_MIN=TOKENS (--serve with --mtp): a request that reads at least TOKENS prompt tokens points the prompt
path's ring at the drafter's 512 routed experts while it is read. The block is lent in place: never freed, never allocated
again, never moves, so no prompt path is laid out again and the decode's captured graphs stay valid; the decode's own bytes
come back (a DMA from a pinned copy) before the first window. Unset, nothing of it runs.
Follow-up to the port of PR 4039 by merbanan. The switches stay opt-in and the prompt path is unchanged when they are off;
the lines are the activation evidence the test plan asks for (chunk, rows held).
Port of PR 4040 by merbanan (rebased onto f-0291-int-trial, bitwise by the author's checks).
--lookup-max-t N (up to 16) lifts only the prompt-lookup windows to N tokens; the MTP drafts keep --spec. A window
past kVerifyMaxT tokens runs as two token groups of at most kVerifyMaxT (the --spec-split layout), the head runs in
column chunks, the batched GDN commit takes up to 16 kept rows, MtpDrafter::draft_long catches the drafter up past its
window, and every lookup size is its own window graph (--lookup-sizes A,B picks them; one that finds no VRAM is dropped).
A build whose windows already hold 16 rows (STRATA_WINDOW_MAX_T=16, the HIP default) runs the 16 as one group.
Batch, pipelined and layer-split windows stay at kVerifyMaxT. Without the flag nothing changes.
Port notes: kVerifyLongT = 16 for both window widths, the per-window-size graph arrays keep the trunk's
[sel_long_][T] layout, the head chunk loop keeps fused_gr_read_multi's own slicing, the penalty history is sized for
the widest window only when --lookup-max-t is past 8, and draft_long forwards the gated-proposal view.
Port of pull request 3382 by Vop3p (commit 2b65710f9 on v0.2.0-beta), rebased by hand on the 0.2.9 trial tree.
Conflict in generate.cpp: the share gate goes first in run(), before the co-read branch, and the kva_floors / set_cell_floor bookkeeping after the read is
kept as it is. Addition to the original: a read run on its own thread beside decoding slots (STRATA_BATCH_CONCURRENT_READ=1, run_conc) keeps the share
blocked for the whole read and does not touch the slots' state from the reader thread. Off unless the variable is set.
Ports the two RDNA4 pull requests onto this tree by hand (three-way merge of the rejected hunks). Where the contributor made a
default depend on a device query, the default is the gfx12 table or the env variable instead, so STRATA_GFX12_DEFAULTS=0 still
gives the old behaviour; the LDS-only barriers and the S2 and head changes that were only run on gfx1201 apply to gfx12 only,
every other target keeps what it ran before.
- iq_kernels.cu (3259): LDS-only barriers in the grid / activation staging, first-pass barrier skipped, vectorised ed_stage_entry,
expdec_gu_kernel launch bounds in waves per EU on HIP, EXPDEC down read at first use; expdec_bench builds on HIP.
- s2_expert_grouped.cu (3264): single-entry gu/down specialisation for the draft layer, fused swiglu + q8_0 on the hit paths,
LDS-only barrier on gfx12.
- verify_kernels.cu, quantize_act.cu, elementwise.cu, shared_expert.cu, fused_gr.cu, mtp.cpp, verify.cpp (3264): multi-block argmax
and top-prob on gfx12, warp-per-block quantize_q8_0_scaled, mapped copies in one launch on HIP, shared expert reuses the q8_1
image, counter-free q8_mixed in the hyper-connection read, the draft layer and the head write the q8_1 image themselves.
- parity tests of both pull requests (verify_parity on HIP, s2/shared/gr parity cases).
Ports three community changes that we have no card to run (no gfx1010, no gfx1030 here), on by default under the one
setting STRATA_AMD_COMMUNITY_TUNING (=0 turns the tables off; cmake -DSTRATA_AMD_COMMUNITY_TUNING=OFF compiles the portable
dot for gfx1010 again). Each is limited to the card or exact library build it was made for.
- gfx1010 (RX 5700 / 5700 XT) signed dot4 takes the SDWA byte-select sequence gfx1012 already used: PR 4855, author Keoian.
Same arithmetic, four signed byte products accumulated modulo 2^32.
- rocBLAS table gfx1030 / 5.4.0.ad84daa81c (TheRock gfx103X-all 7.13 wheels): PR 3343, author elightcap.
- rocBLAS table gfx1030 / 5.2.0.dabb6df2b98-dirty (Arch Linux ROCm 7.2.4): PR 3924, author schmittcn.
include/strata/core/amd_community.hpp holds the switch, the list of community tables and the engine log line (printed only
on gfx1010 and gfx1030); gemm.cu refuses a community table with the switch off, the existing loader still refuses any table
whose architecture or full rocBLAS version string differs. amd_community_test checks that the switch turns each one off.
The contributors' numbers are labelled as theirs in docs/AMD_HIP.md and have not been reproduced.
Port of PR 4039 by merbanan (rebased onto f-0291-int-trial; same bits).
The hyper-connection read's scratch (xn16, lo, lo16, gated and the BF16x2 low halves) is taken as a fourth set of the
region the attention half and the MoE half share: a half's read is done with it before that half's set is written, and
the fused write at a half's end (gr_write_norm_rs: the next half's xn16) runs after the half's last region reader on the
same stream. The same kernels on the same values: the same bits, ~63 KB a token less outside the region, so --prefill
auto finds a bigger chunk. Off by default, and off with the unfused hyper-connection path.
bytes_needed also counts STRATA_PF_GROUP_SCRATCH's larger bounds table (bounds_ints), as carve takes it.
Port of PR 4039 by merbanan (rebased onto f-0291-int-trial; the group-scratch path stays off for weight-stationary layers, whose launches write the whole chunk's rows into GU; the WS offsets follow bounds_ints).
A streamed prompt chunk sends every non-resident expert of every layer over PCIe once, so the chunk size sets
how often an expert crosses the link per token. The chunk is capped by its buffers, which live in the cache
slots the prompt path borrows; on an 8 GB card (RTX 2060 SUPER, IQ3_XXS, 1453 slots) a 4096-token chunk took
1198 of the 1325 lendable slots, and --prefill 8192 fell back to 4096.
The MMQ path's gate/up output, its SwiGLU and its q8_1 copy were sized for the whole chunk (T*K rows, ~84 KB
a token), but a group's products read them and are done before the next group starts. With the switch they
hold one group's rows: gate/up writes its absolute rows at GU - ra, SwiGLU / quantize / down read them from 0.
The cap is T (one expert's most: a token routes to an expert once) plus three times a group's average; a
group that passes it runs as sub-ranges of its experts, each with its own relative bounds appended to the
bounds table. Only on a layout whose every layer takes MMQ (no fused experts, no FP16 fallback); a layer
that would take another path stops the prompt with a message instead of overrunning the smaller buffers.
STRATA_PF_GROUP_CAP=<rows> forces a smaller cap (test knob).
Measured (RTX 2060 SUPER, Ryzen 9 3900X, Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS, resident RAM, k8v4 + KV
streaming; v11 at --prefill 4096 vs this at 5120, 3 pairs, 3 requests per load):
- --prefill auto: 4096 tokens (1197 slots) -> 5120 tokens (1219 slots); 6144 and 8192 still do not fit
- prefill 4K 445.4 -> 449.3, 16K 456.8 -> 493.5 (+8.1%), 32K 456.2 -> 504.5 (+10.6%); decode +0.1 / +0.4 / -0.9
- first-token logits after a 15.6K-token prompt at the same chunk size: bitwise equal to the full buffers;
with STRATA_PF_GROUP_CAP forcing sub-ranges on most groups KL 0.001 (MMQ's stream-k order per launch)
Ported from glorysdj's draft pull request #4185 (commits 961f62963 and 0726ef5ad of that branch; the documentation commit is not taken).
A compile option STRATA_SYCL_IQ_DECODE_FAST (default OFF: packed Q2 expansion, IQ4 bit-select lookup, fast IQ3_S sign arithmetic) and fourteen runtime controls, all default off:
STRATA_IQ3S_GU_SLM, STRATA_IQ4NL_DOWN_SLM, STRATA_RESIDENT_PLAN_COMPACT, STRATA_IQ2S_GU_ROW_REUSE, STRATA_IQ3XXS_GU_ROW_REUSE, STRATA_GR_CACHED_NORM, STRATA_GR_LO_REUSE, STRATA_MMVQ_SHAPE_RPW, STRATA_GDN_QK_REG, STRATA_EXPERT_ROW_E3, STRATA_Q6_SHAPE_GRF, STRATA_IQ3S_GU_ROW_REUSE, STRATA_GR_REDUCE_ILP, STRATA_ATTN_MERGE_ACTIVE.
Conflicts resolved against this tree: the kprof launch wrappers kept (new launches in the conflicted hunks use strata::kprof::kp), the resident-plan launch keeps the kprof wrapper with the compact thread count, native_expert_bench.cpp keeps this tree's fixed bench (the PR's older version of the same hunks is dropped), qsa_prompt_attn_parity.cpp keeps the per-cell check and adds the PR's FP32HASH line and QA_FP32_GRAPH timing.
Also in the diff: docs/INTEL.md section from the PR.
Port of pull request 4380 by moses-sun (commit 5d5e380c4 on v0.2.7), applied without conflict. Refs issue 3577. Homogeneous rigs answer what the single cached bool used to;
not run on a heterogeneous split (no such box here).
Port of pull request 4715 by cedrik-fuoco (commits bd1410701 and d5a0d8294 on 0.2.7.1). The three bf16_proj calls of the layer-major walk (now in Prefill::lm_run)
omitted the stream argument; under STRATA_HC_REQ8=1 the int8 hyper-connection codes were expanded on the default stream while the GEMMs read them on another.
Port of pull request 3382 by Vop3p (commit 2b65710f9 on v0.2.0-beta), rebased by hand on the 0.2.9 trial tree.
Conflict in generate.cpp: the share gate goes first in run(), before the co-read branch, and the kva_floors / set_cell_floor bookkeeping after the read is
kept as it is. Addition to the original: a read run on its own thread beside decoding slots (STRATA_BATCH_CONCURRENT_READ=1, run_conc) keeps the share
blocked for the whole read and does not touch the slots' state from the reader thread. Off unless the variable is set.
Port of pull request 4283 by bitaccesscomau (two commits 3956b94a7 and fcb0e399c on v0.2.8), applied without conflict on the 0.2.9 trial tree.
Off unless the variable is set; STRATA_PREFILL_ROUTED_STREAM_DEPTH sets the copies queued ahead (default 16).
Ported from maxfridbe's pull request #3248 (commits e55a11c08 and 8cc654f23 of that branch).
- STRATA_ATTN_PERCELL (default on): qsa_prompt_attn_percell scores one work-item per cell in the prompt path's fallback attention (reached when the Battlemage ESIMD kernel refuses the pools, e.g. FP16 or Q4_0 KV, or STRATA_PROMPT_ATTN_ESIMD=0). =0 keeps the warp kernel. Not bitwise: FP32 in another summation order.
- STRATA_SELECT_GEMM (default: from 2048 blocks): QSA block scores as oneMKL FP32 GEMM tiles plus qsa_block_scores_reduce; =0 the warp kernel, =1 always. Not bitwise.
- qsa_prompt_attn_parity also checks the per-cell kernel and is registered with ctest.
Conflict resolution against the tree: qsa_select.dp.cpp keeps qsa_block_scores_rows beside the new reduce kernel; docs/INTEL.md keeps this tree's text (the PR's table is not copied; measurements are in the Intel notes).
Ported from maxfridbe's pull request #3248 (commits d95a9c4b0, 5fbef34f9, 344e43e1d, c44fb63be of that branch), cherry-picked without the prompt-kernel commits.
- conversation_snapshot: a copy with neither end in device memory (pinned host <-> pageable host) is a memcpy on the CPU; through the queue it runs on the Arc's copy engine, which the PR reports hanging (Engine reset: engine_class=bcs).
- s2_expert_grouped: #pragma clang fp contract(off) in the two transposed grouped kernels (fixups.py 9d keeps it across re-migrations; 9c the snapshot test's sycl::free).
- gr_parity: the STRATA_QFUSE read check is skipped when the SYCL fused GR read writes no q8_1.
- strata-sycl.sh: STRATA_CACHE_DIR defaults to the mounted data root so the int8 crossover cache survives the --rm container.
Port of PR 3095 by Vop3p (rebased onto f-0291-int-trial; the slot drafter load and bind
conflicts with STRATA_PIPED_MTP resolved by keeping both).
With --batch slots a request that is alone runs the solo path, and that path can keep
--pipeline-windows: both parities' verifiers share the stage's session, so copy_to_slot reads a
whole state after the pipeline drains. With --batch-mtp the slot drafters follow the solo drafter
to the first stage's card when mtp_first. Opt-in; off it is unchanged.
The previous commit stopped whenever the dense weights did not fit in free memory minus --reserve-mib. They would load in the margin, so that case is now a warning (the load goes on) and only need > free memory, where the cudaMalloc cannot succeed, ends the start.
The router's stand-in pick (previous commit) keeps a NaN from becoming a memory fault, but the layer's output is garbage and the request would stream nonsense. route / route_publish now set a device flag when no pick came out; ds4_main polls it at the first token and every 8 (one small async copy and a stream wait, not a per-token sync), and the model says once in the log which device met it. The request ends with 'ERR the model's numbers went non-finite ...' and the conversation is dropped (reset), the same ERR-then-DONE shape as a failed prompt. The --parallel loop is not covered; the host-routed window path already errors on a non-finite weight.
Starting never reaches the feed again (test_no_acquisition passes unchanged). What a start does now, locally: a saved profile whose DeepSeek engine
folder an update removed is pointed at the newest installed package of its own kind (strata_control/dsv41_paths.py, called from runtime.start), which
fixes the 'engine could not start: file not found' after a strata launch updated the engine beside the app.
The explicit update (strata update, the app's Update engine, updater.run_update with the engine) now also updates an installed DeepSeek engine
(updater.update_dsv41): the feed's newer package of the same kind is downloaded, checked (signature, every file), unpacked next to the old one,
started once (--version), and only then every saved run config is pointed at it; older folders go, except one a running engine uses. update --check
reports it; the check kept after any update or check also feeds engine_component.cached_offers / status()['update_available'], which the app reads
without the network to show one card. strata launch's own upgrade (dsv41_engine.py) is left as it was.
Tests: tests/terminal/test_dsv41_update_path.py (the old keep_current tests, adapted, plus a start that makes no request).
qsa_prompt_attn_batch_gated writes half(attn * sigmoid(gate)) straight from the epilogue of
prompt_attn_wmma_kernel, with the expressions of gate_attn_kernel (a * (1 / (1 + expf(-g))), round to
FP16), so the image the o-projection reads is the same bits and the FP32 attention buffer is neither
written nor read back (about 0.4 GB a QSA layer at 8K tokens). Used only where that kernel runs (HIP,
int8 KV, STRATA_HIP_WMMA) and the output is not rotated back (kv_rot / hybrid K8V4 keep gate_attn);
anything else, and STRATA_QSA_FUSE_GATE=0, runs the two-step path. Both prompt-layer loops use it.
Exactness is checked end to end (first-token logits and greedy ids with the switch on and off); there is no
kernel-level test for it yet. Idea from halo-box/strix-llama.cpp PR 213 (attention gate fused into the QSA
prefill epilogue, MIT, The ggml authors); the code is ours.
When every score left is NaN (the gate's input went non-finite), nothing beats -FLT_MAX in route_kernel / route_publish_kernel, the pick stays at its 0x7fffffff start, and the next lines read sc[0x7fffffff] (shared memory), vram[0x7fffffff] and, in the prompt path, vs[0x7fffffff] in moe_sort_kernel: a memory fault in a kernel far from the cause. The thread that picks now takes the lowest expert not picked yet when no pick came out (one compare a pick on the finite path); the weights stay NaN as before. The host-routed window path, which has the picks on the host anyway, ends the request with a clear error on a non-finite weight.
tests/dsv41/ds4_route_guard_test.cu (target ds4_route_guard_test, needs a GPU, no model): valid and distinct picks for NaN / inf / partly NaN logits, the host's top-k when finite, and launch times (DS4_ROUTE_TIMING=1).
rms_rope_rows replaces split_q + rms_rows + rope for the query and rms_rows + rope for K and the indexer query.
It uses the same block_sum, the same (s * x) * w product and rope_kernel's own angle and rotation expressions, so
the outputs are the same bits; the traffic drops from about 1.1 GB to 0.4 GB a QSA layer at 8K tokens. On by
default, STRATA_QSA_FUSE_NR=0 runs the old chain.
hip_qsa_nr_parity compares every output element bit for bit against the old chain (T 1..8191, four start
positions, NaN-poisoned outputs); hip_pf_cover_test poisons every output of the gfx11 prompt GEMMs (wmma_gemm.cu)
and checks that each element is written and that sampled rows match an FP64 reference. Neither has run on a GPU yet.
Idea from halo-box/strix-llama.cpp PR 213 (fused RMS norm + RoPE, MIT, The ggml authors); the code is ours.
Before runtime start, a saved profile that runs strata-dsv41 from an engine-dsv41 folder is pointed at the newest installed package of its kind
(every saved profile's run config, not only the one started), and the signed feed is asked for a newer package at most once in six hours (half an hour
after a failed check). The package is fetched by a separate background process the start waits for at most 90 seconds; after that the start goes on
with the installed engine and the fetch ends in the background. Nothing in it can fail a start. A package folder a running engine uses is never removed
(also in upgrade() and install(), so a strata launch beside the app is safe), and two fetches never run at once.
Known: tests/terminal/test_no_acquisition.py::test_the_deepseek_engine_check_is_reached_only_from_launch_and_setup fails by design, because a start now
reaches the feed (owner rule: starting never acquires). It needs the owner's decision before this branch merges.
Port of Project Maya 8b56730ad (MIT): the dense weights (attention, shared experts, head) always stay in VRAM, so a device whose free memory cannot hold its whole layers' dense weights is told so before anything loads. Devices with tensor-parallel members are not checked.
Random legal blocks against a CPU reference: iq::dequant_rows through both dequantizers (DS4_DQ_OLD=0 / 1 / the default), iq::mv with 1..16 columns, f16_rows, ds4_gguf.cu's rows and mv on Q8_0 / BF16 / ggml types, the library GEMM and the WMMA kernel after the dequantizer, and a repeat path that runs one call many times. Shapes include k % 1024 == 0, which ds4_iq_test never ran.
one_gpu_plan may turn ln_one / ln_land off after the lane timing, but recent_n and the slots' fills stay (recent_n is set at the load, before the plan). borrow_begin then skipped rc_drain: a fill done but not yet mapped, or still writing, met the run's buffers in the tail, and the tick after the hand-back mapped its expert over what the run wrote (NaN routing, then an illegal memory access on a long prompt after shorter ones on one GPU). rc_in_tail() (recent_n > 0 and the fill stream exists) now also triggers rc_drain.
Port of the idea of Project Maya v1.1.45 (c968b9486, MIT): our rc_drain has no for_lend argument and this tree has no x3 / copy_flush, so only the condition is taken.
dq_rows_kernel, the default dense dequantizer on gfx11 (Strix Halo), stages a warp's 32 sub-blocks in a 44-uint4 (704 byte) shared buffer.
Four Q6_K blocks are 840 bytes, so the staging ran into the next warp's buffer and the dense rows came out wrong: NaN in layer 0 of every
chunked prompt (96 tokens and more) of a UD-IQ1_S or UD-IQ1_M set, whose attention is Q6_K. Q3_K (440 bytes, not a multiple of 16) read 8
bytes past what was staged. Q5_K (704 bytes, the ISTA-DAS sets' dense type) fits and keeps the staged path. Stage only when the span fits
and is a multiple of 16 bytes.
(from d6be7ff6a on f-042-strix-ds)
dq_rows_kernel (the default on gfx11, e.g. Strix Halo) stages a warp's 32 sub-blocks in a 44-uint4 (704 byte) shared buffer. Four Q6_K
blocks are 840 bytes, so the staging ran into the next warp's buffer and the dense rows came out wrong (NaN in layer 0 of every chunked
prompt of an Unsloth UD-IQ1_S/UD-IQ1_M set, whose attention is all Q6_K; windows of 16 use the GEMV and were fine). Q5_K (704 bytes, the
IST-DAS sets' dense type) fits and keeps the staged path. Stage only when the span fits and is a multiple of 16 bytes.
M5 Max, MTP, 5 interleaved pairs, fixed harness (each run's spec, min-p and context read from the engine's own log), greedy ids identical in every run, over --spec 2:
Q2_0 story 104.4 -> 104.7 (+0.4%), code 114.4 -> 127.3 (+11.5%), JSON 124.6 -> 132.5 (+6.3%), agent turn 115.9 -> 117.9 (+1.9%, 3/5 pairs).
IQ3_S story 94.2 -> 93.3 (-0.9%), code 103.8 -> 106.1 (+2.5%), JSON 105.1 -> 122.6 (+16.7%), agent turn 102.1 -> 107.1 (+4.9%).
The earlier Q2_0 'flat' result and IQ3_S '+1.4..3.6%' came from a harness that gave the spec flags to the first pair of an arm only. Other sets keep --spec 2.
Window::init accepts 512 or 256 experts a layer; the decode router (router_route / moe_route), the prompt router (route_topk) and the plan buffers use the model's count, the SSD stream reads it from the GGUF. The route kernels are templated on the logits a lane holds (16 for 512, 8 for 256); the 512 path is the old code token for token, the logits rows keep their 512-float stride, the predictor's unused columns are -inf. The MTP layer keeps its own 512 experts.
Tests: prompt_parity route_topk and fuse_parity router_route run both counts (all pass on the M5 Max). IQ3_S 512 experts: greedy ids identical to the build before in every class and prompt (3 interleaved pairs), speed equal (decode +/-0.5% median, prompts -0.5..+5.9%, inside the spread). Not run: an end-to-end 256-expert model (58 GB download).
Since the conversation-cache fix gave sample_host_memory a macOS branch, the 0.2.7 policy (table in RAM when it fits) started firing on the Mac: the 26.8 GiB n-gram table was locked in RAM, the footprint went from 8 to 35-42 GB and free memory dipped to 19 GB. M5 Max, IQ3_S, 3 interleaved pairs, same binary, same tokens: a cold 4K prompt after decode work is 10.6% slower with the table in RAM (-10.7 -10.6 -10.0% against the v0.2.9 binary, +2.4 +15.2 +12.2% for direct reads against RAM), decode is the same inside the spread. Direct reads are what the Mac had before; an explicit --ple-io still wins.
The Qwen3.6 engine (src/qwen35/q36_engine.cu) started its penalty history from the whole prompt (`hist = ids`), so its model-card presence penalty 1.5 counted the prompt's last tokens as in the other engines. The history now starts at the last prompt token and grows with the tokens written (penalty_history_start in include/strata/qwen35/sampling.hpp), the same window as src/program/generate.cpp; batch slots copy that history. STRATA_PENALTY_PROMPT=1 keeps the whole prompt.
Tested: qwen35_sampling_test (276 checks, 9 new for the default and the opt-out), and a CUDA sm_86 build on z440. Replay on z440, one A4000, UD-Q6_K_XL, the reporter's conversation cut to 250 records, 400 tokens: 16 sampled seeds and 2 greedy requests per arm. This model did not return the old JSON in either arm (0/16 with the default, 0/16 with STRATA_PENALTY_PROMPT=1; 0/2 greedy in both), so this replay does not show a difference between the two windows; no answer in either arm started with the old JSON or a code fence.
docs/DETAILS.md: the sentence saying the Qwen3.6 path still counts the prompt is replaced.
Same FMAs in the same order as the runtime-bounded loops (bitwise); nsys: gdn_ab 352 -> 290 us/round, router mmvf 533 -> 492 us/round. Engine A/B pending.
sycl/src/program/generate.cpp is its own copy of the serve loop, so the change in ed76b12f9 did not reach Intel cards. Same window: the fed-back last prompt token and the reply's tokens, built from the request's prompt ids on every request (a restored conversation gets the same window as a cold read). The helpers are added to sycl/include/strata/kernels/sampler.hpp, which that build includes first.
Measured on two Arc Pro B65 (SYCL AOT bmg-g31), Qwen3.8-Flash-Next IQ3_S, the reporter's 4-message conversation (32,951 tokens, cached), --spec 2, 16 sampled seeds, 500 tokens: old JSON 0/16 and numbered plan 16/16 with the new default; old JSON 8/16, plan 7/16, other 1/16 with STRATA_PENALTY_PROMPT=1. Greedy, 2 requests per arm: numbered plan 2/2 in both arms on this model.
M5 Max, MTP, IQ3_S, 5 interleaved pairs (2 + 3), greedy tokens identical in every run: --spec 3 --spec-min-p 0.5 against --spec 2:
story 93.8 vs 92.4 tok/s (+1.4%), code 103.4 vs 100.4 (+3.0%), JSON 105.3 vs 101.6 (+3.6%), agent turn 100.8 vs 98.0 (+2.8%).
Q2_0 measured flat (-1.0% to 0.0%, 5 pairs; 800-token story -0.7%), so it keeps --spec 2; --spec 3/4 without min-p lose on both.
mac_fit.spec_args picks the flags by model; engine_args takes the model name (setup.py and metal/setup_mac.py pass it); a streaming
Mac keeps STREAM_SPEC, a Mac without the draft layer keeps --spec 2. The engine's own default (--spec 2) and an explicit --spec in a
config are unchanged. Test: tools/test_setup_mac.py.
With the model-card presence penalty 1.5 over the last 64 tokens, the prompt's own tokens (the digits of a new task, a quoted JSON answer) were penalised, and a conversation whose last turn asks for something new could get its previous answer back. The window now starts at the end of the prompt: the fed-back last prompt token and the reply's tokens. Earlier assistant turns inside the prompt are not counted. The history is built from the request's prompt ids on every request, so a conversation restored from the conversation cache or a saved slot gets the same window as a cold read. The STRATA_ADMIT_MERGE_FIRST merged first window is not used for a request with a penalty window, since its prompt rows would join the reply window.
Measured with the opt-in switch (618cd7a4b) on a P100, IQ2_XS, the reporter's saved 4-message conversation, 32 seeds per arm: old JSON returned 13/32 as shipped, 4/32 reply-only, 3/32 with presence 0. Qwen IQ3_S on an R9700, 16 seeds: old JSON 8/16 as shipped, 0/16 reply-only. 61-task quality027, greedy, one run per arm: 56 and 55 passed as shipped (two runs), 55 reply-only, 54 with presence 0.
Added penalty_window_test (header-only, CPU) and a line in docs/DETAILS.md.
Same FMAs in the same order as the runtime-bounded loops (bitwise); nsys: gdn_ab 352 -> 290 us/round, router mmvf 533 -> 492 us/round. Engine A/B pending.
DeepSeek: an illegal memory access, launch failure or hardware exception in a prompt read or a decode round used to turn into an ERR/cancel while the engine kept running on a dead GPU context, so every later request failed with HTTP 400 until someone restarted it. strata-dsv41 now prints 'ERR fatal GPU error: ...' and exits (code 3). Qwen: a failed decode window, commit or final commit wait that leaves a sticky fatal runtime error does the same (the prompt-read path already did, #224), by the runtime's error text so CUDA, HIP and SYCL builds agree.
Server: that ERR line fails the request in flight as an engine death (503, retryable) and marks the engine ended, so the next request starts it again at once, with one plain log line and a plain death note. Restart-loop guard: a third fatal death within ten minutes stops the restarts and answers 503 with the reason until the window has passed; one death counts once.
Test hooks (off unless set): DS4_INJECT_FATAL=N / STRATA_INJECT_FATAL=N fail the Nth request / decode window with a real fatal error's text; *_ONCE=<file> fires it once across restarts.
Tested: serve/test_fatal_gpu_restart.py (7 tests); through the real server, request 2 injected, DeepSeek on boston (IQ1_M pack, 2x A6000): request 2 got a 503, the restart took 44 s, requests 1, 3, 4, 5, 6 answered 42; Qwen on a P100 (IQ2_XS), fault injected mid-decode in request 4: 503, restart, requests 1-3, 5, 6 answered 42. The guard counted '1 of 3' there and '2 of 3' in an earlier run where the injection fired on every restart. NOT tested: a real GPU fault (none reproduces), the Windows and Metal builds, the third-death refusal through a running server (unit-tested only), conversations coming back from saved slots after the restart (no check made).
Bitwise the plain v3 read (gr_v3_screen, T 1-8, 0 differing); 4-8% faster per read on cold weights. Weight prefetch and 2/4 rows a warp measured slower and stay off. Engine A/B pending.
hcf.metal (gr_norm folded into the producer of bo with a last-arrival threadgroup: bit-exact, +1.2 ms slower), gemvu.metal and
moeu.metal (unit loops unrolled x2: +2.3 / +1.5 ms), qsab.metal (batched qsa_attn: +0.07 ms). None is used by the engine; they
stay with their numbers (phaseA-tests/mac-q/RESULTS.md section 7). token_chain gets the CHAIN_* switches that measured them
(NORM2, COMB2, NONORM, NOOPS, SKIPK, HCF, KEEPNORM, GEMVU, MOEU, QB, DEFS).
Window::prompt ran each chunk's MTP-layer K/V rows (9% of an 8K prompt) after the chunk, with the GPU otherwise idle for them.
They now go to a second command queue (Context::run_graph_async / wait_async) while the next chunk's main work runs on the
first. The chunk's final residual rows are double-buffered (pb_.R / pb_.R2); the rows of chunk c+1 wait for chunk c's (shared
drafter cache and scratch); every way out of prompt() joins. Not used with M-RoPE (images). STRATA_METAL_MTP_OVERLAP=0
restores the sequential rows.
mtp_rms_b: mtp_rms with the row loaded up front (same sum, same products, same order); under STRATA_METAL_LDB=0 the original.
Engine, Q2_0, MTP spec 2, 5 interleaved pairs, same binary: p8k 1936 vs 1910 tok/s (+1.4%, 3/5), p32k 1747 vs 1711 (+2.0%, 4/5),
greedy ids identical in every run. The overlap is partial: the two queues share the GPU.
With the model-card default presence penalty 1.5 over the last 64 tokens, the prompt's own tokens (digits of a new task, a quoted JSON) were penalised, and a conversation whose last turn asks for something new could get its previous answer back. Measured on a P100, IQ2_XS, the reporter's saved 4-message conversation, 32 seeds per arm, 400-500 tokens: as shipped 13/32 returned the old JSON and 16/32 a numbered plan; reply-only penalty 4/32 and 25/32; presence 0 3/32 and 26/32. quality027 --quick (19 tasks, greedy, one run each, no noise arm): shipped 15, reply-only 16, presence 0 17 passed; the 1-2 task differences are single tasks and inside run-to-run noise. Default unchanged (switch off). The old JSON still came back in about 1 of 10 in the best arms, so the penalty is a large part of the cause, not all of it.
bit 0-1 down (batched staging loads, weight prefetch), bits 2-3 up, bits 4-5 rows per warp 2/4 for T<=4. Screen bit-for-bit against the plain v3 read; speed not established yet.
gr_norm stored R in place inside its loop, so each of its ten iterations (and the ten of the second loop) waited a full
memory latency for the next load: 98 times a token, ~6 us of a 7.7 us dispatch. gr_norm2 loads R, bo and w_norm first, then
sums, reduces and stores; moe_combine_sh2 loads the ten plan entries, then the ten rows, then sums. The arithmetic and its
order are the originals', so the outputs are bitwise the same. STRATA_METAL_LDB=0 selects the original kernels.
token_chain (Q2_0, 1 token, graph): 10.65 -> 10.14 ms. Engine, MTP spec 2, 4 interleaved pairs: story 100.97 -> 103.90 tok/s
(+2.9%, 4/4), code 108.63 -> 110.84 (+2.0%, 4/4), greedy ids identical in every run.
Off by default. With STRATA_STALL_STACKS=1 the stall report suspends each other thread in turn, unwinds it with the unwind tables
(no heap, no loader lock, no stdio) and prints the frames, named from the PDB when it lies beside strata.exe. The minidump flags now
include process thread data, handle data and memory info. Used to locate the Windows gfx1151 UD-Q4_K_XL stall (host thread inside
the HIP runtime after a graph launch).
A systemd service without LimitMEMLOCK may lock 8 MiB, so the 26.8 GiB mlock of the PLE table was refused and the engine ran with a reclaimable table (the reporter measured up to 6% slower decode) behind a two-line message. Before mlock the engine now raises RLIMIT_MEMLOCK's soft value to the hard one (needs no privilege) and says so; if the table still does not fit, the message gives the allowed size and the remedy (LimitMEMLOCK=infinity, or ulimit -l unlimited). The residency policy itself is unchanged. Tested here: the CUDA engine builds (sm_60). The run under 8 MiB soft/hard, 8 MiB soft with a higher hard limit, and unlimited is queued, not yet seen.
The Mac now reads its free RAM for the conversation-cache guard (arena_avail's rule), the window's checkpoint uses the shared
contract (3 tail rows a layer, a scratch block_pos), and the session's QsaStates point at the window's K/V buffers (and the
drafter's), which the generic snapshot copies as cell-major byte prefixes. Before, the session's own pools were never written on
Metal. Two conversations taking turns (11K and 6.5K, 24K and 3K tokens): the second request of each re-read its prompt (6.3 s,
14.9 s); now it reuses it (46-65 ms), with the same answer ids.
Heretic v2 IQ4_NL keeps blk.0.attn_gate.weight as BF16 (GGUF type 30) while the other 35 gates are Q8_0. NativeDense has no kernel for type 30 and skipped it, so the prompt path and the verify window, which read the gate through the native route, stopped with 'blk.0.attn_gate.weight has no native GGUF blocks (run with --native)' on the first request. NativeDense::load now makes such a matrix (BF16 in the file, BF16 row in the pack, 2-D, a multiple of 32 wide) into Q8_0 on the host (the same rounding the engine already uses for STRATA_HC_Q8) and binds it, so every dispatcher finds a native projection; weight_bytes_for counts it. The pack's BF16 copy stays. This is not bit-exact against reading the BF16 values (a Q8_0 rounding on one 2560x6144 matrix, the same format its siblings have); reading the BF16 directly would need changes in the prompt path, the verify window (with its side-stream branches) and gdn_layer. Tested here: the CUDA engine builds for sm_60 (p1004). NOT tested: any run, because the 109 GB Heretic IQ4_NL model is not on a test box; the reporter's own run is the first test.
A Windows process starts with the I/O priority of the program that started it. Run from a Task Scheduler task, a service, or a terminal that an IDE or coding agent keeps at low priority, the engine inherited Low I/O priority: the reporter measured PLE direct reads about 15x slower, cold prompt reads 3-5x slower and decode 14% slower on the same drive, and the engine blamed the drive. Both engines (strata and strata-dsv41) now read ProcessIoPriority at start and set it to Normal when it is lower (no privilege needed to go from Low to Normal). STRATA_KEEP_IO_PRIORITY=1 leaves it as inherited. The CPU priority class is not touched. New io_priority_test (Windows): started at Low it ends at Normal, started at Normal it is untouched, KEEP leaves Low alone. Tested here: that test passes on Windows; the HIP engine builds. Not tested: the reporter's slowdown itself (needs their drive and a low-priority launcher with an engine run).
A worker going to sleep at the moment a batch started could miss both the new generation and the wake-up (the caller then runs the batch alone, slower, not wrong). Taken from Project Maya v1.1.36 (#618, Cody Harrison), MIT, credited in third_party/project-maya/NOTICE.md.
On Windows the HIP runtime gives every cudaHostRegister'ed range a second address for the GPU and takes any host pointer inside it for the registered memory, but it does not reserve that range in the process's address space. In about one start in ten the stack of the thread that pins the expert arena lay there: small cudaMemcpy calls from local variables then read (and wrote) the arena, and every long prompt was read wrong until a restart.
- pinned.cu: each registered range's alias is reserved (MEM_RESERVE); a range whose alias already covers memory in use is unregistered and treated as refused by the driver, so it stays resident but unpinned (logged).
- generate.cpp: main keeps the free address space below 1 TiB reserved for the life of the process and runs the engine on a new thread, so its stack and later allocations lie above the aliases (STRATA_STACK_ISOLATE=0 turns it off).
- pinned_alias_test, docs/AMD_HIP.md.
Windows HIP only; other backends keep their behaviour. Author's measurements: 0 bad starts in 110 (main + change 0 in 70, main alone 5 in 70) on an RX 7900 XTX. Here: the HIP objects, pinned_alias_test and pin_budget_test build for gfx1100 with ROCm 10.2 (no gfx1100 card to run them on). The hunk in pinned.hpp was applied by hand because v0.2.9 has the shared-file fields between the lines the PR touched.
The screen looked for strata-vision in the engine folder for every model; DeepSeek carries its own picture tower (--vision on|off), so the check always failed for it and Read pictures stayed off and locked.
A programmatic-tool-calling harness shows the model its tools as tools.write(...) inside code, and the model then wrote the native call with that spelling; the call came back as text and the run stopped. The prefix is read like the other unambiguous aliases (a declared name always wins, an undeclared name stays text). Also: a tool_reference with a non-string name no longer fails the request. Tests for the prefixes and for the history rule on the Anthropic and Responses shapes.
The 0.2.9 rule read only assistant messages' tool_calls. It now also reads tool_use blocks (Anthropic) and function_call / custom_tool_call input items (Responses, with the namespace in front of the name). Test on the /v1/messages route added.
Claude Code declares ToolSearch and a defer_loading placeholder and names its MCP tools only in the conversation; when the model
calls one of them straight away the call came back as text and the session stopped. Ported from PR #4700 by xrip onto v0.2.9
(the server.py hunk for the parser call moved); the PR's own tests are included.
Co-authored-by: Niko1221 <[email protected]>
A read from token 0 that reaches the shared start (shared=, pin=) no longer stops at it: the start is a restore point inside
the one run of chunks (feed_marks), and the state at it is saved apart for new conversations after the read, from that mark
(Model::slot_save(out, err, at): the window rings and partial groups from the mark, the compressed KV and index keys of rows
0 .. at / ratio, the tokens up to it). The stop cost a whole extra pass over the streamed experts: a fresh 6000-token prompt
with the shared start at 4500 and a tail point, DGX Spark first token 21.0 s -> 16.4 s (14.35 s with no keys at all).
STRATA_DS4_PREFIX_STOP=1 stops at the start as before. A new chat behind the start still takes it back in 0.11 s and reads
the same tokens as a cold read; the agent loop total is unchanged (188.8 s against 187.9 s, 3 runs each).
Credit: Project Maya #607 (mw00), MIT, third_party/project-maya/NOTICE.md.
serve/server.py (0.2.9) sets the nine qualified switches only on Linux; Windows had none. Measured on a Strix Halo Windows PC
(Qwen3.8-Flash-Next IQ3_S, serve/server.py --parallel 4 over the engine, tools/par_load.py w2 at 4 clients, 3 rounds, interleaved,
download paused): 51.0 / 51.2 / 51.7 / 50.7 tok/s without, 55.0 / 55.6 / 55.2 / 55.1 with (+8%), TTFT p50 2.57 -> 2.43 s,
max chunk gap p95 1.97 / 1.37 -> 1.27 s, no errors. Not STRATA_BATCH_CONCURRENT_READ.
parallel_default_env() in arch_defaults.cpp: CKPT_POOL, CKPT_POOL_READY=4, SLOT_CKPT_MOVE, SLOT_COPY_DIRECT, SLOT_COPY_BATCHSYNC,
BATCH_GDN_GROUPS, BATCH_ATTN_ROWS, BATCH_SELF_COMMIT, QSA_SLOTS, set from generate.cpp with --batch / --slots of 2 or more on a
Windows gfx1151; a switch the user set is kept; STRATA_GFX1151_DEFAULTS=0 turns it off. Unit test in arch_defaults_test.cpp.
verify.cpp read STRATA_BATCH_SELF_COMMIT and STRATA_QSA_SLOTS into namespace-scope constants, before main() sets the defaults, so
two of the nine would never have seen them (the first engine build measured 52.2 / 52.6 tok/s, the base). They are read at the
first use now. Also logs the time of each verify window graph capture.
Strata 0.2.9 (engines and the SYCL package built from 99866a47a2b8a21b131aad162a584ee9da92f8f0; the Windows and Linux x86 app, Terminal, Control and installers from c62d5e3a15b40ea51c96aae9e8d36928965f53ab; the Mac set from 4cf0d8cd3 and the Spark Control from 38f56619c, which the later card fix does not touch; the later commits change the Python Control, server, setup, tests, tools, docs, the app host and the Mac Metal engine only)
Strata 0.2.9 (engines and the SYCL package built from 99866a47a2b8a21b131aad162a584ee9da92f8f0; the Windows and Linux x86 app, Terminal, Control and installers from c62d5e3a15b40ea51c96aae9e8d36928965f53ab; the Mac set from 4cf0d8cd3 and the Spark Control from 38f56619c, which the later card fix does not touch; the later commits change the Python Control, server, setup, tests, tools, docs, the app host and the Mac Metal engine only)
Measured on Boston (one A6000 as a 12 GB card, ballast leaving 64 GB of RAM, UD-IQ1_M): the first 1000-token prompt read took 17.0-17.5 s with the
B1 default against 11.1 s before B1 and 11.2 s with DS4_FLOW_BUFFERED=0. The setting is Project Maya #312's (Peasant Smith / mw00); it stays available
as DS4_FLOW_BUFFERED=<tokens> until it is measured to win here.
With a layer's slots partly in the lent tail region (B1, Project Maya #312 tail lending), a swap that hands its outgoing expert to the RAM store
(DS4_STORE_PUT, on by default) read vram_base + slot * unit past the layer's own allocation: on HIP (p620, short prompt, windows path, cache on)
the first swap raised an illegal memory access and poisoned the context. The DBG slot dump had the same address. Adds tests/dsv41/short_prompt_check.py
(a 40-token prompt twice with the default sampler, then greedy; fails on ERR or illegal-access / router-picked lines).
Project Maya #691 (needmorevram, c7d08c5), the "a chunk streams only the experts it picks" part, onto src/dsv41's batches
path (Streamer / copy_ahead; DS4_FLOW=0 or a pack the flow does not take): with DS4_SKIP_UNPICKED_ROWS=N (or
DS4_SKIP_UNPICKED=1 for every chunk) a chunk of at most N rows waits for its layer's routing and then reads and copies only
the experts some row of it picked, into the same slots and batches, so the summing order is the same. Off by default.
Strata's flow path already streams only routed experts, so the default path gains nothing from it.
DS4_UNPICKED_STAT=1 prints, per chunk size, the share of a layer's experts no row of the chunk picked.
Credit: Project Maya #691 (needmorevram), MIT, third_party/project-maya/NOTICE.md.
Project Maya #691 (needmorevram, c7d08c5), the restore-point part, onto src/dsv41: Model::feed_marks() keeps the restore
points inside a run of chunks from snapshot sets the layers fill on the way (kSnapN = 8 sets; feed_mark's one set became
an array), each the state a read up to that point leaves, instead of a run up to each point; Model::marks_inside() says
whether that is worth it (the long-prompt skip's extra rows at most a chunk, inner_cost); the serve loop reads the rest
of a prompt that way when the points left are inside it (not past a shared start a read from token 0 stops at, #607).
STRATA_DS4_MARKS_INSIDE=0 stops at each point as before. feed()'s `after` (the rest of the prompt still to come) moves
the long-prompt skip's kept rows to the prompt's end. Not ported from #691: the chunk streaming only the experts it
picks, the EXL3 lend, DMA read-back of host sums.
Credit: Project Maya #691 (needmorevram), MIT, third_party/project-maya/NOTICE.md.
Project Maya #607 (mw00, b2d4c30) onto src/dsv41: when a prompt read from token 0 reaches its shared start (shared=,
pin=) the state the read keeps there is also copied apart from the conversation, in RAM (slot_save). A prompt that does
not continue the conversation in the model takes back the longest kept start it begins with, when that covers more than
the model's own restore points (a kept conversation slot only when it covers more still), and reads only the rest. With
or without STRATA_DS4_SLOT_KEEP. STRATA_DS4_PREFIX_KEEP (2 starts, LRU), STRATA_DS4_PREFIX_GB (2), STRATA_DS4_PREFIX_MIN
(1024); 0 keeps none. Starts with pictures are not kept.
Credit: Project Maya #607 (mw00), MIT, third_party/project-maya/NOTICE.md.
With the lane on (DS4_STREAM=1) and one discrete NVIDIA GPU, the CPU lane's RAM copies are pinned and the fetch kernel pulls a streamed
expert where it lies; a verify window's experts stream too and land in a recent slot, so the link's bytes stay in VRAM.
Gating and follow-ups: one-GPU path only where experts stream (1b60249, 482e46c #496, 7c0ceb1 #551, mw00), no landing while the swap is pending
and a table-update wait (32f3c8c handmade0octopus), a failed fill is never published (#646 handmade0octopus), Windows with pin by need (#567 TheSmokeDev),
HIP pull clock on gfx11/gfx12 and DS4_IQ1S_ROWS (#563 Cass, mw00). Everything stays behind DS4_STREAM=1; DS4_RAM_SKIP_W (changes answers) is opt-in.
Not ported: the helper-GPU condition (#415). Built on CUDA sm_120; model identity/A-B not run yet.
The layer keeps DS4_RECENT=<n> VRAM slots (taken from its coldest experts) for CPU-lane experts it used twice in its last 8 ticks.
A thread copies each fill on its own stream, the next tick maps it by one tab_set launch a device; a swap leaves recent slots and
experts being filled alone, and a run of chunks that lends the tail waits for the fills first. Opt-in (default 0), values unchanged.
Not taken from #344: all-reduce over DMA, segmented top-k, DS4_INDEX_TC, the cuBLAS workspace pin.
ds4_gguf_test: its Q8_K default now follows the engine (opt-in, DS4_IQ_Q8K=1).
A worker going to sleep at the moment a batch started could miss both the new generation and the wake-up (the caller then runs the batch alone, slower, not wrong). Taken from Project Maya v1.1.36 (#618, Cody Harrison), MIT, credited in third_party/project-maya/NOTICE.md.
Taken from Project Maya (MIT), staged against v1.1.42:
- A2: the tail restore point is left out where its read stop costs more than
reading its tokens again (TailCost wired into the serve loop; one in 32 still
kept to measure again; STRATA_DS4_RESTORE_TAIL_AUTO=0 keeps it always). Maya
#668 (mw00, 8721412 and 66fb17b).
- A5: the flow timing correction of the opt-in timing pick stays in the prompt
size class that measured it (ds4_path_policy.hpp + test). Maya #612 (Will
Hampson, 6e0d27d). Our default pick is by length already.
- A sampler CUDA error ends the request instead of sending token -1; logits()
copies on the head stream. Maya #619 (Cody Harrison, 2040174).
- A CPU-lane layer that cannot compute its experts stops the engine instead of
answering zeros; a layer with no GPU kernel is refused at load, any other
warns. Maya #620 (Cody Harrison, e598716).
- bf16_rn by arithmetic on CUDA too. Maya #676 (rilken, 79628d7).
- DS4_ASYNC_T default 8 -> 16: windows of 9-16 rows take the same device path
as 1-8. Maya #608 (mw00, b9c07e7). DS4_ASYNC_T=8 gives the old path.
Project Maya #639 (needmorevram) and v1.1.37 / v1.1.39 (mw00), written onto Strata's server: deepseek_restore_points()
reads the last user turn, the first user turn (the end of the system prompt and tools) and the tail from the prompt's ids
as the template wrote them; Service.prepare puts them in the request's _restore (never a client's own), StrataEngine
sampling_keys turns them into marks=<pos>,... tail=<pos> shared=<pos>. The shared start stays Strata's pin=N where the
request already has one (the system messages); shared= is asked only without it. STRATA_DS4_RESTORE_POINTS (4, 0 = off)
and STRATA_DS4_RESTORE_TAIL (512, 0 = none). serve/test_ds4_restore_points.py.
Credit: Project Maya #639 (needmorevram), mw00; third_party/project-maya/NOTICE.md.
Ports Project Maya's restore points (#639, needmorevram, 50a3313; ds4_marks.hpp v1.1.42 by mw00 and needmorevram) onto
src/dsv41 as a per-file merge, not a commit apply:
- ds4_marks.hpp as in Maya v1.1.42 (latest_point, mark_plan, mark_victim, TailCost; TailCost is not wired in yet, #668).
- Model::mark(shared) / marks_cap() / check_bytes(): up to STRATA_DS4_RESTORE_POINTS (4, 0 = none, at most 16) restore
points besides checkpoints 0 and 1 and the anchor, the least recently used giving way, the shared start last.
restore() and best_check() go back to a mark; a slot keeps the marks with their roles (3 mark, 4 shared start), slot
identity strata-ds4-session-4; the .meta file takes up to 3 + 16 checkpoint positions.
- Replaces Strata's fourth checkpoint (ck[3], the pin=N shared start) by a shared mark: pin=N is now shared=N.
- GEN keys marks=<pos>,... tail=<pos> shared=<pos> (pin=N kept): the prompt read stops at each point mark_plan keeps and
keeps the state there.
- Tests: tests/dsv41/marks_test.cpp (Maya's ds4_marks_test), slot_test for the longer checkpoint list.
Credit: Project Maya #639 (needmorevram), #668 / ds4_marks.hpp (mw00), MIT, third_party/project-maya/NOTICE.md.
Owner rule: a consistent Strix Halo speed gain goes on by default. Windows gfx1151, engine 0.1.54 builds, interleaved, idle box
(ab_f16def, 2 rounds, no environment on either side): IQ3_S prompts 4K 1,180 -> 1,279, 8K 1,289 -> 1,398, 32K 1,299 -> 1,404 tok/s
(x1.08); UD-IQ4_XS 8K 1,092 -> 1,223, 32K 1,135 -> 1,248 (x1.10-1.12). Decode over 200 tokens unchanged (58.6 -> 58.2, 48.8 -> 48.5),
first greedy ids identical, 38 of 38 runs exit 0. The rounding differs from the int8 kernels; KL is information only.
STRATA_PF_F16=0 turns it off. arch_defaults_test expects six switches on Windows gfx1151.
With --parallel/--batch >= 2 on a CUDA or HIP Qwen engine the server sets CKPT_POOL=1, CKPT_POOL_READY=4, SLOT_CKPT_MOVE, SLOT_COPY_DIRECT, SLOT_COPY_BATCHSYNC, BATCH_GDN_GROUPS, BATCH_ATTN_ROWS, BATCH_SELF_COMMIT and QSA_SLOTS unless the environment or the config's env already names them (=0 turns one off). DeepSeek, SYCL, Metal and Vulkan engines are untouched.
Unbuffered reads of a file that has a view in the process run at half the drive's rate on Windows (probe on Strix Halo: 6.5 GB/s with no view, 3.2 with one). Default on for an integrated GPU, where the CPU lane's misses read through the store's slots. Decode 9.1 -> 9.4-9.7 tok/s (code, drafter off, 2 interleaved pairs); the 32K prompt is unchanged; no failed reads in any run.
UD-IQ4_XS and UD-Q4_K_XL keep five layers with Q8_0 down experts; without STRATA_PF_F16 they left the fused path for MMQ.
On those layers only the F16 kernel now runs (STRATA_PF_Q8D_F16=0 turns it off); every other format keeps its kernel.
UD-IQ4_XS on Windows gfx1151, interleaved x3: 4K prompt 1,071 -> 1,120 tok/s, 8K 1,136 -> 1,180, same output ids;
IQ3_S unchanged (1,190 -> 1,186, 1,286 -> 1,284).
A file with a view (or a buffered handle with cached reads) in the process halves unbuffered read speed on Windows (probe: 6.5 GB/s with no view, 3.2 with a view of the whole file or of 1 GB of it, 3.3 with a buffered handle held open, 6.7 on a copy while the original is mapped). The CPU lane reads misses through the store's slots, so the views are not needed after the load: Pack::drop_views() unmaps them, reopens never-read fallback handles and Pack::read_at goes unbuffered. StoreHold re-reads an expert that has no slot instead of falling back to the view.
Also: DS4_IO_STATS (demand read queue/read/copy times), DS4_IO_PRIO, DS4_READ_SPLIT (default 1, off; deeper queues did not help). Decode 8.9 -> 9.6 tok/s (code, drafter off).
The service thread spun 2 ms after its last answer; a Strix Halo decode layer takes ~2.8 ms, so it slept (Windows sleep_for(20 us) wakes after ~1 ms+) before almost every request.
Decode 8.2 -> 9.3 tok/s on pack 2.25 (2 interleaved pairs). DS4_PROF in decode and in the prompt hangs the HIP driver on Windows; DS4_PROF_SYNC drains the stream at each decode stage mark instead.
uma_expert_cache counted only the RAM the OS sees, so UD-IQ4_XS (55 GiB of experts) and UD-Q4_K_XL (72 GiB) on a PC with a
64 GB carve-out and 64 GB seen by Windows got no cache slots beside the 40 GiB budget and read the rest from the SSD.
The carve-out (less 14 GB for the dense weights and buffers) now adds its slots. Tests added.
- the pdl capability queries answer false instead of aborting the engine (#3981, #2715)
- each batch slot keeps its own running state in the Metal window (GDN state and conv history, QSA K/V, indexer tail and pooled keys,
PLE history), side by side in bigger buffers; the slot copy, the checkpoints and session_zero follow the session they are for
- a batch window holds one token of each active slot: the GDN step, the QSA kernels and the PLE conv take a per-row slot, cell and
history parity (function constants, off for every other caller), so the dense and expert weights are read once for all rows
- STRATA_METAL_BATCH_ROWS=0 runs the slots one window each instead (diagnosis)
The download default stays on the four-shard pin cb02422c; nothing is fetched under the new names. The six-shard files (Hub commit a419d74c,
27 October: five shards a set, the Engram as the sixth) and the four-shard files with the first shard ISTA re-published on 26 October (what the
Hub's last four-shard commit 653a9874 serves; #4237 was refused by the app because only the pinned size of shard 1 was known) are taken from
files already here, by size and SHA-256.
- setup.py: DSV41_ISTA6_FILES (15 files), shards6 on the three sets, dsv41_layout_of / dsv41_parts / dsv41_pack_layout; install_dsv41 picks the
layout of a --gguf-dir folder (both whole: the pinned one), packs the five files with the Engram as --engram, and builds the pack again when it
was made from the other layout's files. The converted Engram (same name and size, another hash) is accepted beside the pinned one.
- strata_control/dsv41_sets.py, models.py: the app's check and `strata models import` know the six-shard names, the re-published shard 1, and
that a folder with both whole layouts of one set is that set once.
- strata_cli/launch: --model-path and the guided screen name what is missing in the layout that is there.
- tools/dsv41/pack_gguf.py: --engram as the sixth shard of a six-shard split; the pinned release's Engram serves the six-shard shards too.
- Tests for each; docs/DEEPSEEK41.md.
Builds on the cherry-picks of #4348 (integer tensors in the GGUF reader) and #4632 (StackArc3's six-shard work, option A), which this
commit reworks so the four-shard pin and files keep working unchanged.
Related: #4237, #4256, #4632.
On 27 October 2026 ISTA-DASLab replaced the four-shard IST-DAS builds on
the Hub's main with six-shard builds (llama.cpp b11870+ layout), and
--model-path with those files failed with "...-00001-of-00004.gguf is
missing". The v0.2.7.1 pin still serves the old files.
- Pin a419d74c: five shards per build named of-00006 and the shared
Engram as the sixth. The Engram was converted in place (same name and
size, new hash).
- pack_gguf.py counts an --engram file as the sixth shard only if it is
one file, not already among the shards, number count-1 of the same
split, and holds the single tensor per_layer_token_embd.weight.
- An Engram whose finish mark names an old hash is fetched again in the
same run. One whose mark has no hash is hashed: a match is kept and
marked, a mismatch is fetched again. A user's own old one is refused
with the reason.
- A pack from an earlier run is rebuilt unless it names this run's files.
- Six-shard only; four-shard files are ignored and stay on disk. The
first-release name tables are now empty. No engine code changes.
Related: #4237, #4256.
BLOCK_GEOMETRY had no row for I8/I16/I32/I64, which GGML_TYPES names and which IST-DASLab's DeepSeek-V4.1
first shard uses for the Engram hash parameters (tools/dsv41/pack_gguf.py). The preflight in front of setup
and the app reads "no geometry" as "unsupported file" (strata_control/gguf_preflight.py:165), so a complete
file was refused as "cut short or another version" or as "a format Strata does not read" (#4256).
setup.check_shards sized such a tensor at 0 bytes (setup.py:3324), so a shard cut inside one passed as whole.
The rows are the ones tools/dsv41/ista_intake.py:112 already uses for these types.
The scratch of the FP32 prompt attention is batch x chunks x n_head x 258 floats (about 0.41 GB at 32 and 1.65 GB at 128 on
Flash-Next), independent of the prompt chunk. Prefill::init allocated it from STRATA_ATTN_BATCH (128 since 09524eb47) while the
estimate that sizes the expert cache and the prompt path's loan (bytes_needed_impl) still counted 32, so about 1.2 GB was never
reserved and a single 32 GB Arc Pro B65 with the config setup writes failed its first request ("device buffers for a chunk of 512
tokens do not fit (2098 of 32656 MiB free)"). Both now use one function (attn_batch_for), and the batch is capped at the chunk's
token count, since a chunk never runs more queries than it has.
The default goes back to 32: with the budget honest, 128 would take 1.2 GB from the expert cache on every Intel card for a
1.1-1.3% faster prompt on two B65s. STRATA_ATTN_BATCH still sets it (1..1024), and is now counted.
INTEL_ARC.md: one bullet with the numbers (2x Arc Pro B65: decode +17% to +23% over the serial --spec 4 setup, 5 pairs of 5),
what stays as before, and the two switches; one row in the tested cards table. MULTI_GPU.md: the Default line of
"Pipeline windows" now names the Intel Arc case beside AMD RDNA4 and NVIDIA.
On two Battlemage cards the engine pipelines the two cards' verify windows by default, and then answers as a 2-token window
whatever --spec says (pipelined --spec 4 equals serial --spec 2 in 100 of 100 agent requests; serial --spec 4 differs in 3).
So the generated config says --spec 2 there: the same answers, and about 20% faster than the old --spec 4.
strata_control/intel_spec.py holds the rule: the cards are found by PCI device id (the xe driver and an id in 0xE200-0xE2FF,
the id the engine's architecture enum comes from), never by name. It changes only setup's own default "--spec 4", only
with the draft layer on, and only for exactly two such cards. A --spec in "user_args", any other value, one card, an
A-series card in the pair, three cards and "mtp": false are left alone. Applied in three places: sycl/setup_intel.py when it
writes the run config of a layer split (the copy in "args_written" follows, so a rerun does not take it for a hand edit),
the launcher's derive_config (before the person's --set options, which win), and the Intel server's engine_args at every
start (a config an older setup wrote). Tests in tests/sycl/test_intel_two_card_spec.py.
When --pipeline-windows is not given, the engine turns it on for --serve with a layer split of exactly two stages whose
GPUs are both Battlemage (Xe2) cards by architecture id (strata::intel_gpu_gen), with the MTP drafter (--spec 2 or more),
no --batch, no --peer-device and no helper caches. A single card, an Arc A-series card, a mix, three or more stages
and any other vendor stay serial. --pipeline-windows N and STRATA_PIPELINE_AUTO=0 override it, and one log line says it
is on. The condition is a host-only function (strata/pipeline_auto.hpp) with a test that needs no GPU.
Measured on 2x Arc Pro B65: a 1 h mixed serve (1,082 requests) with 0 errors, 0 token differences against the serial run
of the same requests and flat VRAM; decode +17% to +28% over the serial --spec 4 default. The pipelined windows answer as
a 2-token window, so setup and the launcher write --spec 2 for such a config (next commit).
Which builds take 16 rows and why, how the 8-row build stays main's code, what was tested (real 16-row windows,
identity on seven boxes), the four-card profile with its measured effect per setting, and the opt-out.
For exactly four Radeon AI PRO R9700 cards (at least 28 GB each, hip, no --peer-device) running Qwen3.8-Flash-Next
IQ3_S, setup.py and the launcher fill in what the person has not chosen: parallel 32 with --batch-groups 4, the layer
split 13,26,39 (the output head costs about four layers on the last card), --prefill 2048 (in place of the "auto" setup
writes), and STRATA_QSA_SLOTS, STRATA_BATCH_SELF_COMMIT and STRATA_SLOT_COPY_DIRECT in the run config's env. A setting
of the person's own always wins; STRATA_FOUR_CARD_PROFILE=0 or "four_card_profile": false turns it off. The prompt cache
stays on (agent follow-up turns wait 0.8 s for the first token instead of 1.5 s).
Through the launcher's configuration on the 4 x R9700 box (total tok/s, warm round, parallel 4 before): 1 request 66.6 ->
67.9, 8 at once 156.6 -> 261.2, 32 at once 155.4 -> 369.6, four 7.9K prompts out 24.3 -> 35.0 and in 1892 -> 2706.
batch_test with the profile's flags: 32 of 32 slots identical to solo; a 25 minute soak of 32 clients: 0 errors.
serve/server.py no longer says "at most 8 requests" for a layer split (each pipelined group holds its own slots).
Tests: tests/terminal/test_amd_four_card.py.
A pipelined group of a layer split is its own window, so 32 slots on four cards were four groups of 8 and nothing
larger could run. The window limit is now a build option (STRATA_WINDOW_MAX_T = auto | 8 | 16; auto is 16 for HIP,
8 for CUDA; SYCL and Metal never set it): include/strata/kernels/window_limits.hpp.
A window of up to 8 rows runs exactly the kernels it ran before (HIP gfx1201 window 8 against main: 3096 of 3097
kernels identical, the other a debug variant; CUDA sm_86 and sm_60: identical or renamed but for a few prefill kernels
NVVM compiles differently, with identical results). The code for 9 to 16 rows is only reached by a wider window:
- per-row arrays of the verifier, the expert source tables (128 / 256 entries) and the remote / peer expert tables follow
the window; CPU expert jobs stay at 8 tokens (a 9..16-row group is two jobs over the same expert);
- the hyper-connection read is made in slices of 8 rows (its Q8_0 launch would otherwise skip rows);
- the grouped expert kernels run a second pass (SKIP template argument) instead of a 16-entry shared array (51 KB,
over the 48 KB static limit of CUDA);
- resident_plan runs on 128 threads (up to 8 rows, as it was) or 256; MMVQ, bf16 GEMV, shared expert and MoE combine
take 9..16 columns only in a wide build; the sm_60 packed path runs wide calls in passes of 8;
- STRATA_P4_TIMING and STRATA_P4_TRACE (include/strata/program/pipe_probe.hpp): opt-in timing and trace of the
pipelined windows, off unless set.
Tests: src/core/window_limits_test.cpp (target window_limits_test). Real 16-row windows (--batch 64 on four stages,
--batch 16 --batch-groups 1) decode the tokens they decode alone.
The engine's host threads have no autorelease pool, and Metal parks one extra
reference per new MTLBuffer, command buffer and encoder in the calling thread's
pool. Nothing drained it, so every buffer Context::alloc made (a 16 KB job
array per expert matvec, ~90K a request) and every stream command buffer and
encoder stayed alive after free(): +1.3 GB, +90K Metal regions and a slower
queue per request, until a big Mac slowed to a crawl.
- Context::alloc and the fallback wrap: create the buffer inside an
autoreleasepool (DS4 runtime only; the Qwen build is untouched).
- strata::gpu runtime: a pool around command-buffer, compute-encoder and blit
creation; the compute encoder is retained and released in end_encoder.
- Event marks: an event recorded again, or destroyed, before it was waited on
now drops its command buffer token (ds4_ctx_drop / Context::drop_mark).
- route, combine and sparse_attn free the stand-in buffers they allocate for
absent inputs.
On an Arc A750 the lane-cell score phase read a 4,106-token prompt 0.7% slower than the unchanged kernel (three
interleaved runs against the published 0.1.52 engine; with STRATA_ATTN_LANECELL=0 the branch matched it), while it is
+5% on the Arc Pro B65. The default now follows strata::intel_gpu_gen(): the runtime's architecture enum first, the
device name only as a fallback. STRATA_ATTN_LANECELL=1/0 still forces either way on any device.
Problem: the timing lambda measured milliseconds per iteration and returned them, while the printout labels the value
"us" and derives its GB/s as `bytes / us / 1e3` - so every number this bench has printed was 1000x too small in time
and 1000x too large in bandwidth. A launch that moves 19.7 MB in ~140 us was reported as 0.1 us and 148,713 GB/s, which
is why the tool looked broken rather than mislabelled. The wall clock always agreed with the real work (200 iterations
618 ms, 2,000 iterations 3,566 ms).
Result, 2x Arc Pro B65, shard 1 (Qwen3.8-Flash-Next IQ3_S experts, 10 groups), before and after the one-line fix:
| case | before | after |
| --- | ---: | ---: |
| 2 tokens | 0.1 "us", 148,713 "GB/s" | **143.4 us: gate/up 84.5 us (148 GB/s), down 59.4 us (78 GB/s)** |
| 8 tokens | 0.3 "us" | **340.8 us: gate/up 217.4 us (58 GB/s), down 126.0 us (37 GB/s)** |
and the model's own layers at two tokens: layer 30 iq3_xxs/q2_0 143.6 us, 31 iq3_xxs/iq4_nl 140.4 us, 32 iq2_s/iq4_nl
135.5 us, 33 iq3_xxs/iq4_nl 140.2 us - gate/up 131-149 GB/s, down 78-168 GB/s, results bitwise equal to the reference in
every case.
Approach: return microseconds (`ms * 1e3 / iters`), which is what the printout, the minimum-of-three logic and the
bandwidth formula already assume. No kernel, no timing path and no measurement logic changed.
Validation: the same run before and after; the wall-clock scaling that exposed the bug; bitwise equality retained in
all four model layers and both token counts. Not run: CUDA/HIP (this file is the port's own), other shards.
Limits: this fixes the units only. The numbers are per-call costs with the bench's own synthetic groups, so a rate taken
from it still has to be matched to the real window (48 calls per layer step).
Problem: the bench fills its output with 0xff and then compares the mode against the reference, so a launch that
returns without enqueueing leaves both sides bitwise equal on an untouched buffer and the tool reports "0 differing"
and a 0.1 us timing - a measurement of nothing that looks like a very fast kernel. The 0xff fill is NaN as a float, so
"no finite value in the reference window" is exactly that case.
Result, 2x Arc Pro B65, shard 1 layer 20 (`iq3_xxs/q2_0`, 10 groups, 2 tokens): the guard does *not* fire - the window
does write finite values, so this defect is not what makes the timing wrong. The timing is still 0.1 us per launch
against a 19.7 MB working set (0.3 us at 8 tokens), which scales with work but is 20-60x too fast for the launch to be
waited for; the remaining cause is in the timing loop's wait, not in the launch. That stays an open port item.
Approach: the guard only refuses and reports; it changes no timing path and no kernel.
Validation: builds and runs on the box; the run above is the only one taken. Not run: CUDA/HIP (this file is the port's
own), and the timing itself is still unproven - do not use its numbers.
Limits: this is a diagnostic, not the fix. `native_expert_bench` numbers remain unusable until the wait is found.
Problem: `--mtp-q4 head` builds the Q4_0 draft head by gathering the subset and dequantizing it row by row, but with
`STRATA_Q6_PLANAR` (the SYCL port's default for Q6_K heads) the source is stored as 16-byte-aligned planes and the
gather copies planes as if they were rows. The result is a draft head that proposes nothing at all, with no error:
decode falls from 73.8-75.6 tok/s to 35.3 tok/s (the non-speculative rate) and the DONE line reports "0 of 0" drafts.
Result, 2x Arc Pro B65, Qwen3.8-Flash-Next IQ3_S, 128 tokens of a 483-token chat prompt, one warm request each:
| configuration | decode | drafts |
| --- | ---: | --- |
| `--mtp-q4 head`, planar head (before) | 35.3 tok/s | 0 of 0 |
| `--mtp-q4 head`, planar head (after) | refused with "needs the packed Q6_K head layout (STRATA_Q6_PLANAR=0)" | - |
| `--mtp-q4 head` with `STRATA_Q6_PLANAR=0` | 69.5 tok/s | 54 of 63 |
| default head (no `--mtp-q4`) | 74.9 / 75.6 tok/s | 55 of 63 |
Approach: a guard in `make_q4_head` that refuses the option when the head really is planar, so the engine says why
instead of running with speculation silently disabled. The packed-layout path is unchanged and still works, which is
what the measurement above shows.
Validation: the guard message appears in the engine's own output; `--mtp-q4 head` with the packed layout still produces
54 of 63 drafts; the default path's tokens are identical to the pre-change reference (`compare_tokens.py`, no
difference) and its rate is unchanged. Not run: CUDA/HIP (the file is the port's own copy and the guard is behind
`STRATA_NATIVE_EXPERTS`), a non-Q6_K head, `--batch-mtp`.
Limits: this makes a broken opt-in fail loudly; it does not make the 4-bit head fast. With the packed layout it is
*slower* than the Q6_K head (69.5 against 74.9), so the option is still not worth enabling.
Problem: the prompt path's per-layer grouping tables - `[ids | slot | src]`, T x K int32s each, ~245 KB per layer at a
2,048-token chunk - live in mapped pinned host memory, and on the mapped arm the GPU reads them there, once per layer.
On a PCIe-attached pair that is 48 rounds of host-memory reads per chunk, and the port already carried the A/B arm to
copy them instead (`STRATA_GROUP_COPY=1`) without a measurement deciding between them.
Result, 2x Arc Pro B65, Qwen3.8-Flash-Next IQ3_S, `--pipeline-windows 2`, three warm repetitions, same build, back to
back:
| case | mapped arm | copy arm | gain |
| --- | ---: | ---: | ---: |
| prefill 4,106 tokens (AOT) | 1,303.2-1,303.5 tok/s | **1,332.8-1,337.5** | +2.5% |
| prefill 16,380 tokens (AOT) | 1,617.8-1,622.7 | **1,657.2-1,665.2** | +2.5% |
| prefill 10,938 chat prompt (AOT) | 1,575.6-1,578.4 | **1,613.8-1,616.8** | +2.5% |
| decode, chat prompt, 128 tokens | 74.9 / 75.6 | 74.9 / 75.6 | unchanged |
Approach: the default flips to the copy arm for this port; `STRATA_GROUP_COPY=0` restores the mapped one. Both arms
allocate nothing new when the buffer is already big enough, and the copy arm reuses the host vectors the code already
keeps for exactly this.
Validation: the token stream is identical between the two arms at both prompt sizes and on the decode run
(`scripts/compare_tokens.py`: first difference none), and identical to the pre-change reference (the rc-p16k run of
round 42). Not run: CUDA/HIP (the file is the port's own copy), a host-shared-memory device (an iGPU, where the mapped
arm may be the better one - hence the switch), a 32K prompt.
Limits: +2.5% on prefill only; the decode path does not use these tables.
Problem: `--pipeline-windows 2` (the two-card window overlap) ran but produced different tokens from
`--pipeline-windows 1`. Ten rounds of switches, splits and modes narrowed it to the two alternating verifiers, and a
state dump (STRATA_PIPELINE_DUMP: a hash of the first stage's GDN recurrence and conv history after every window)
found the moment: identical after the prompt, different after the first decode window.
Root cause: the serial path lets a one-token window advance the GDN state inside the window and skips the commit
graph for it (`record_window` + `commit`). That is safe there because a serial window never runs before its
predecessor's verdict. A *pipelined* window may: stage 0 runs window K+1 speculatively while window K's verdict is
still open, so a self-commit lands on state that is not final yet - and the first decode window of every request is a
single token.
Fix: a per-verifier switch (`set_self_commit_off`) that the pipelined verifiers set. With it `record_window` does not
self-commit and `pl_commit_async` always queues the commit graph - the configuration that already measured
token-identical (`STRATA_ONE_TOKEN_COMMIT=0`) but kept a global switch.
Result on 2x Arc Pro B65, Qwen3.8-Flash-Next IQ3_S, same build, back to back:
| case | pw 1 | pw 2 | tokens |
| --- | ---: | ---: | --- |
| short prompt, 256 tokens, spec 4 | 56.3 / 56.1 tok/s | **59.6 / 61.4** | identical |
| 4K context, 128 tokens | 55.6 | **60.3** | identical |
| `--spec 2 --spec-min-p 0`, 64 tokens | 49.2 / 50.3 | **59.6 / 63.2** | identical |
| 512 tokens, one run | - | **60.6** | - |
| prefill 4,106 tokens | 1,244.8 / 1,242.0 | 1,253.8 / 1,251.3 | identical |
| prefill 16,380 tokens | 1,548.1 | 1,561.1 | - |
So decode gains 7-9% at the default window and 21-26% at two-token windows, and the prompt path is unchanged or
marginally faster.
Validation: tokens identical in all four decode cases above (`scripts/compare_tokens.py`); the serial path's tokens are
unchanged against the pre-change reference. The state dump still reads one window behind in the pipelined arm (an
ordering artifact of where it copies), so the token comparisons are the evidence, not the dump.
Not run: CUDA/HIP (the files are the port's own), a 32K-context prompt, an overnight soak.
Limits: `--pipeline-windows 2` stays opt-in. It costs 278 MiB of card 0's expert cache (the second verifier and the
GDN snapshots) and its gain depends on the window size, so the default remains `--pipeline-windows 1`.
The pipelined state dump showed the first multi-token window leaving the GDN state one row ahead of the serial path,
so the next question was whether the commit count was wrong. With STRATA_PIPELINE_DUMP set, `pl_commit_async` now
prints `n_keep`, `T`, `pos0` and whether the one-token skip applies, which answers it: the counts are right.
```
commit n_keep=1 T=1 pos0=18 self_skip=1 (stage 0)
commit n_keep=1 T=1 pos0=19 self_skip=1
verdict p=19 T=1 a=0 B none
commit n_keep=1 T=1 pos0=19 self_skip=1 (stage 1)
commit n_keep=2 T=2 pos0=20 self_skip=0 (stage 0)
commit n_keep=2 T=2 pos0=20 self_skip=0 (stage 1)
verdict p=20 T=2 a=1 B launched ON
```
So the multi-token window commits two rows in both stages, exactly as the verdict says, and the extra row must come
from the *window's own* self-commit: `record_window` self-commits when `T == 1 && !batch_rec_ && !g_qfuse() &&
one_token_self_commit()` (verify.cpp:844), and the transition from a self-committed single-token window to the first
graph-committed window is where the state gains the row. That is the next thing to diff - the dump currently hashes
the recurrence and the conv history together.
Debug only: the print needs a state dump to be requested and changes no behaviour.
Problem: the serial commit knows that a one-token window advances the GDN state itself (`record_window`) and skips the
commit graph for it; `pl_commit_async`, which the pipelined windows and the pipelined prompt reader use, ran the graph
anyway. A one-token window therefore advanced the state twice. The first decode window of a request is always one
token, which is why `--pipeline-windows 2` produced different tokens from `--pipeline-windows 1`.
Evidence: the state dump added in the previous commit (STRATA_PIPELINE_DUMP, hash of the first stage's GDN state after
every window) was identical for both arms after the prompt (`pos 18 sum 1952.13774 hash b97305876cf73d2c`) and
different after the first decode window (`pos 19` 3222.063 in the pipelined arm against 2815.93065 serial). With
`STRATA_ONE_TOKEN_COMMIT=0` - which makes the serial path take the graph as well - the two arms produced **identical
tokens**, which pinned the asymmetry to this one branch.
Result: with the skip, the two paths' states agree after the first and second windows (`pos 19` and `pos 20` hashes
equal) and the tokens are identical up to and including token 5; the first difference moved from token 4 to token 6.
Validation: `--pipeline-windows 1` tokens are unchanged (identical to the reference), and the prompt path, which also
commits through `pl_commit_async`, did not move (4,106 tokens 1,239.7-1,248.9 tok/s, 16,380 tokens 1,552.1, the same
as before the change). Not run: CUDA/HIP (the file is the port's own copy), a 32K-context prompt.
Limits: this does not make `--pipeline-windows 2` token-exact. A second defect remains: after the first *multi-token*
window the pipelined state is one row ahead of the serial one (`pos 22` in the pipelined dump equals the serial
`pos 23`), so the commit count and the window's own advance still disagree somewhere for T > 1. The path stays
opt-in and the shipped default is `--pipeline-windows 1`.
`--pipeline-windows 2` runs, is faster (57.7 tok/s against 51.3; 68.1 against 45.9 with a forced window size) and
produces different tokens. Rounds 16-21 eliminated every switch and mode: the speculation (THETA), the prestaging,
the agreement chain, the snapshot overlap, a fixed window size, the split point (a one-layer stage 1 still differs)
and the elastic KV (off by default). What is left is the two alternating verifiers themselves.
This adds the instrument for the next step: STRATA_PIPELINE_DUMP=<file> appends a hash and the sum of the first
stage's GDN state (recurrence state plus conv history) after every window. It is debug-only - it waits for the
stage's stream - and off unless the variable is set.
It already shows one defect on its own: in the pipelined path the same position of a second request does not hold the
same state as the first request's (`pos 18 windows 2 sum 3669.18189 hash 5f718405585185d9` against
`pos 18 windows 1 sum 1952.13774 hash b97305876cf73d2c`), so a request does not start from a reset sequence state.
That is separate from the token divergence, which reproduces on the very first request of a fresh process
(`compare_tokens.py`: first difference at token 4).
The serial path needs the same dump before the two can be diffed element by element; that is the next step, and
`--pipeline-windows 1` (the shipped path) is unaffected by this commit.
Problem: `attn_batch` was 32, a value chosen for the shuffle-reduction score phase's vector-matrix kernel. The
lane-cell phase is now the prompt default, and it wants more queries per launch.
Result, same build, back to back, 3 warm repetitions each (2x Arc Pro B65, Qwen3.8-Flash-Next IQ3_S, 2,048-token
chunks):
| prompt | batch 32 | batch 128 | gain |
| ---: | ---: | ---: | ---: |
| 4,106 tokens | 1,227.9-1,235.1 tok/s | 1,240.9-1,247.6 | +0.9% |
| 16,380 tokens | 1,582.2-1,584.2 | 1,604.9-1,605.8 | +1.3% |
Batch 512 is another 0.2% on the 16K prompt but takes four times the scratch (400 MB per stage instead of 100 MB),
so 128 is the trade. The expert cache keeps its slots: 10,253 either way at 4K.
Approach: the default of the `STRATA_ATTN_BATCH` switch (added earlier this branch) moves from 32 to 128; the
variable still overrides it in 1..1024.
Validation: 64 generated tokens are identical to the serial reference (`scripts/compare_tokens.py`, first
difference: none). Not run: CUDA/HIP (untouched - the switch is the SYCL port's), a card small enough for the
100 MB of extra scratch to cost expert-cache slots.
Limits: the gain is attention-bound and small; it is the tuning of the phase that landed in the previous commit, not
a new path.
Problem: the prompt path runs `qsa_decode_attn_batch` (the tensor-core kernel is opt-in and 2.9x slower here), and
attention is 21% of a 4,106-token prompt's GPU timeline and 24% of a 16,380-token one. The port already had the
`STRATA_ATTN_LANECELL` score phase - one cell per thread, no shuffle reductions, scores bit-identical to the
shuffle reduction - but only behind an environment variable, so nobody got it.
Result, same build, same box, back to back, 3 warm repetitions each (2x Arc Pro B65, Qwen3.8-Flash-Next IQ3_S,
`--spec 4`, 2,048-token chunks, nothing else on the GPUs):
| prompt | shuffle reduction | lane-cell | gain |
| ---: | ---: | ---: | ---: |
| 4,106 tokens | 1,167.4-1,171.9 tok/s | 1,226.3-1,235.3 | +5.2% |
| 16,380 tokens | 1,456.3-1,457.4 tok/s | 1,524.3-1,526.2 | +4.7% |
| decode (256 tokens, 18-token prompt) | 56.5 / 57.0 | 56.6 / 57.0 | unchanged |
Approach: the choice follows the query count. The prompt path hands the kernel a batch of queries (`attn_batch`, 32
by default), the decode window hands it the window's few rows (at most eight per call), and the few-query case is
1.4% slower with the lane-cell phase (55.9-56.3 against 56.6-57.0 tok/s), so `n_q >= 16` takes it.
`STRATA_ATTN_LANECELL=1/0` forces either way, so the old path is one variable away.
Validation: 64 generated tokens from the gated build are identical to the same prompt's tokens from the serial
reference (`scripts/compare_tokens.py`, first difference: none); the lane-cell phase's scores are the shuffle
reduction's bits by construction (one cell per thread, same order). Not run: CUDA/HIP (untouched - the change is in
`sycl/src/kernels/cuda/qsa_decode_attn.dp.cpp`, the port's own copy), the parity bench on the gated build (it forces
its own arms), a 32K-context prompt.
Limits: measured on two Arc Pro B65 with a fully resident expert cache; the gain is attention-bound, so a prompt
short enough to be dominated by the expert GEMMs will show less.
`--batch N` (the concurrent serving slots) loads the OpenCL adapter as well, so on a two-card `xe` box
dpct::device_count() then reports five devices: the two Level-Zero GPUs, the CPU, and the OpenCL view of the same
two cards. An automatic split took every index above 0, so it tried to put a stage on the CPU and the engine died
before serving:
layer split across 5 GPUs: CUDA0, then CUDA1,2,3,4 (split auto)
layer split, CUDA2 weights: the weight arena does not fit (1466 MiB needed, 0 MiB of 160876 MiB free on that card)
The auto-filled split now only takes GPUs of the primary device's backend, so `--batch 2` with `--layer-split auto`
starts on the same 25/23 split as a plain run. An explicit `--layer-split K` / `--split-device` is unchanged.
Measured on 2x Arc Pro B65, Qwen3.8-Flash-Next IQ3_S, 64 new tokens per slot:
| slots | window | rows/window | aggregate |
| ---: | ---: | ---: | ---: |
| 1 (speculative stream) | 44.4 ms | 2.35 | 57 tok/s |
| 2 | 40.6 ms | 2.00 | 49 rows/s |
| 4 | 58.1 ms | 4.00 | 69 rows/s |
| 8 | 88.7-89.3 ms | 7.00 | 78-79 rows/s |
So a batch window costs about 18 ms + 10 ms per row, and eight slots (the most that fit the VRAM) reach ~79
tokens/s of aggregate decode - above the single stream's 57, below the 100 the two-card budget would need.
The prompt path runs the decode-shaped FP32 attention kernel `attn_batch` queries per launch (the tensor-core
kernel is opt-in and 2.9x slower here, see qsa_prompt_attn_parity). 32 was chosen for a vector-matrix kernel;
on 2x Arc Pro B65 more queries per launch give it more to overlap.
Measured on Qwen3.8-Flash-Next IQ3_S, 16,380-token prompt, second request in the process (STRATA_PREFILL_TIMING):
| attn_batch | qsa attn | prefill |
| ---: | ---: | ---: |
| 32 (default) | 2,572 ms (23.0%) | 1,294 tok/s |
| 64 | 2,492 ms | 1,309 |
| 128 | 2,448 ms | 1,316 |
| 256 | 2,422 ms | 1,315 |
| 512 | 2,409 ms (22.0%) | 1,319 |
So the switch is worth ~2% and the default stays 32 (the difference is inside the run-to-run spread on one pair of
runs); STRATA_ATTN_BATCH=512 is the measured best for a long prompt. The kernel itself is the problem: it is
1-4% of the card's fp16 peak at long context, and the XMX port is correct (2e-6 against FP64 in
qsa_prompt_attn_parity) but 3x slower, using 122,880 of 131,072 bytes of local memory for its 64-cell layout
(mode 32 halves that to 86,784 and is slower still).
The bench's queue is created with an exception handler, which swallows a failing kernel: the timing then reads
"0.3 us per group of 10 experts" because nothing ran, and the bit-for-bit comparison passes trivially (both
buffers stay at the 0xff fill). Every wait now goes through wait_and_throw() and prints the what() of a failure.
Verified on the Arc box: a run with real geometry prints dn=10 and non-zero outputs, so the kernel does run; the
per-mode timing of a single small group is still under one expert's memory traffic and needs the mode/phase
semantics checked before its GB/s numbers are quoted.
sycl/src/kernels/native_expert_bench.cpp was in the tree without a target, so it never compiled - and it still
carried raw CUDA calls (cudaMalloc, cudaMemcpy, cudaMemsetAsync, cudaFree, cudaEventRecord/ElapsedTime). It is now
a target under STRATA_NATIVE_EXPERTS and uses USM allocations, queue copies and a host clock on its own in-order
queue, so the grouped native expert modes can be timed and compared bit for bit on real GGUF rows at the decode
window's shape (G experts x T tokens).
Run: native_expert_bench <shard1.gguf> <layer> <G> <T> <ref mode> <mode> [iters]
Measured on the Arc box (layer 0 of the IQ3_S pack = iq3_xxs gate/up, iq4_nl down): every mode is bitwise equal to
the reference, G=480 x T=6 reports gu 7.7 us and down 5.3 us for the whole group - the absolute numbers still need
the mode's own semantics checked (a single group's 0.3 us is below one expert's memory traffic, so that path is
returning early).
STRATA_VERIFY_PROFILE collects GPU timestamps per layer step into device memory, but the readback ran on the
device's default queue: neither ordered after the window's work on cs_ nor necessarily this stage's device. That
is why the stage table was only ever printed in the eager mode, which stamps from the host after draining the
device - so its numbers include launch overhead and gaps and cannot attribute kernel time.
* collect_profile() reads the stamps on cs_ (under OnDevice(device_)) and waits there.
* The window prints the table from the GPU's own clock every N windows:
STRATA_VERIFY_PROFILE=1 STRATA_VERIFY_PROFILE_EVERY=32, without the eager drain.
Measured on 2x Arc Pro B65, Qwen3.8-Flash-Next IQ3_S, decode window T=4, per stage (24 layers) and per window:
total 20-24 ms; of that the routed experts ("VRAM hits") 5.0-6.6 ms in the GDN layers and 1.3-3.0 ms in the QSA
layers, the dense q8/qkv gemv 2.0, hc-read1+router 1.9, shared+quant 0.96, out-proj 1.1, z 0.97. The eager table
used to overstate hc-read0 and the experts by 2-4x because it timed the host's waits.
The two-card window overlap never ran on the Intel engine: the first poll of
every pipelined window returned "verify: error".
* Verifier::done() lost CUDA's cudaErrorNotReady branch. The port's
sycl_event_query() returns 1 for "still running"; the migrated code treated
every non-zero value as an error. A pending window is not an error now:
flush the queue every 2 ms and time out after 20 s, as the CUDA original does.
* A drafter completion event was used as a barrier on the other card's queue.
SYCL rejects a cross-context dependency ("Cannot submit to a queue with a
dependency from a graph that is associated with a different context"). On two
cards the host waits for the event; the same card keeps the barrier.
* The first card's second verifier (ver_b) took whatever device was current at
init; it is created under OnDevice(0) now.
Diagnostics: queue_wait_diag() names the queue, its device and the SYCL message
instead of "error"; an unqueryable completion event says so; and with
STRATA_VERIFY_DEBUG=1 every verifier prints its device and streams.
Verified on 2x Arc Pro B65, Qwen3.8-Flash-Next IQ3_S:
* --pipeline-windows 1: tokens identical to the serial path (64 tokens on a
short prompt, 32 after a 4K prompt, two runs each); a short prompt read drops
from 206 ms to 118 ms (-43%).
* --pipeline-windows 2: runs now (it crashed before), ~8-20% faster decode, but
it does NOT produce the serial path's tokens yet - the decode overlap and its
GDN rollback still need work on this backend. Not a default.
Service took the ids of "<|im_end|>" and "<|endoftext|>" as its stop ids whatever the tokenizer made of them. DeepSeek-V4.1's tokenizer
has neither as one token: it cuts them into '<', '|', 'im', '_end', '|', '>' and 'end', 'of', 'text'. A server run without the family
named (a hand-written config, a gateway) therefore ended the reply in front of the first "im", "|", ">", "end", "of" or "text" it
wrote, and the real end token (id 1) did not end it. Nerazim is N, er, az, im: the reply stopped at 'im' and the client got "Neraz".
The engine and its --eos 1 were fine; nothing was cut by the stop-string hold-back (StopMatcher gives its held tail back at the end).
end_token_ids() keeps a marker only where it is a single token, and falls back to the tokenizer's own end token (special_ids "eos").
Qwen's tokenizers are unchanged. The family guess from the previous commit still names the right parser and DSML for DeepSeek.
serve/test_stop_holdback.py: the end ids of both tokenizers; StopMatcher over every split of the text and one character at a time
(Qwen and DeepSeek stop lists, words ending like the start of a stop string); the Detokenizer plus StopMatcher byte by byte; and the
OpenAI and /v1/messages routes, streamed and whole, with and without the family named: no stop string (whole reply), a stop string that
never comes (the held tail is given back), a stop string (cut exactly there).
The tool-call forms the model drifts into (<parameter=NAME> as the opener, JSON inside <tool_call>, a bare <function=NAME>, a second call in one
wrapper) are read as the call when the tool is declared; anything else stays text. On the 1,462-turn drift corpus 98.3% of turns read as intended
with it and 92.9% without; the specimens of the tool corpus read the same with it on or off.
- Service.tool_call_recovery starts True; tool_call_recovery_setting(cfg) is the config rule (absent = on, a non-bool still stops the start).
- Tests: a fresh Service reads the drifted call without being told; the config rule; the old on/off switch test is unchanged.
- docs/DETAILS.md: the config row says on since 0.2.8.
0.2.7 turned STRATA_ROUTE_PRIOR=1.5 on by itself on a discrete NVIDIA card of 13 GiB or less. Reports and our own 0.2.7 agent check (4.5 of 10
tasks with it on, 5.6 with it off) tie it to repetition loops and weaker results in long agent sessions, so the owner took the default back
(2026-10-29). With nothing set the engine now routes exactly as 0.2.6 and as STRATA_ROUTE_PRIOR=0 did; STRATA_DEFAULTS=026 is unchanged.
- sm120_series.hpp: route_prior_default() answers 0 on every card; route_prior_resolve() is the pure rule the engine reads (a set value wins on every
card); kRoutePriorSmallCard stays the 1.5 opt-in value.
- verify.cpp: g_route_prior() uses it; the "on by default on this card" start line is gone.
- Expert settings / Terminal: "Faster answers on 12 GB NVIDIA cards" is now an Off-by-default row that writes STRATA_ROUTE_PRIOR=1.5; the words
say it is faster on 12 GB cards but can repeat itself in long agent sessions (nine languages). A 0.2.7 file with STRATA_ROUTE_PRIOR=0 reads as Off.
- Tests: arch_defaults_test (default table over 12 cards x hip x integrated x 026, explicit opt-in values), speed-switches.test.mjs and
tests/terminal/test_speed_switches.py.
- docs/DETAILS.md: both places that described the 0.2.7 default.
serve/server.py run on its own with a hand-written config (no "arch": "deepseek41") gave DeepSeek-V4.1 the Qwen defaults: the stop ids
of <|im_end|> and <|endoftext|>. The DeepSeek tokenizer splits those strings into pieces, so a "|" or ">" in a reply ended it
(finish_reason stop at token 19 in front of a Markdown table, or right after "## 3. Recom"), and the real end token (id 1) was not a
stop id, so it came out as the text <｜end▁of▁sentence｜>. Reproduced on 2 x RTX A6000 with the reporter's request: the config with the
arch answers 730-1161 tokens six times out of six; without it, stops at 192-436 tokens, one of them right after "## 3. Recom".
The tokenizer's own pre-tokenizer name ("deepseek3") now picks the family when the config names none; a named arch is never changed.
(cherry picked from commit b18292d702)
--expert-cache auto sized the cache from the VRAM free after the weights, less --vram-reserve-mib, the draft head and the other priced
buffers. What the load allocates after the cache and no setting prices - the CUDA libraries the prompt path and the verifier warm up
(cuBLAS handles and workspaces, lazily loaded modules) and the verifier's small buffers - came out of the reserve, so the card ended
about 240 MiB under it: 459 MiB free at --vram-reserve-mib 700 on an RTX A4000 16 GB, 457 MiB on an RTX A6000 48 GB, under the 512 MiB line
the serve check warns about at every request (#2065). Measured with STRATA_TRACE=1: -164 MiB at the prompt path and -80 MiB at the verifier
beyond the draft head, the same with --prefill 512, 2048 and auto, on both cards, and the same on v0.2.6 (so not new in 0.2.7).
auto_cache_slots() (include/strata/core/auto_cache.hpp, tested in src/core/auto_cache_test.cpp) now leaves kLateAllocMib = 256 for it,
unless the card could not keep min_slots (#496) after it, and not under STRATA_DEFAULTS=026. An explicit --expert-cache N and
--vram-reserve-mib N are untouched; the cache is ~100 slots (about 1.8% on a 16 GB card) smaller.
On a two-GPU layer split, every prompt on the chunked path stalled: a device's wait kernel spun for a
hand-off while that device's thread blocked in its first cuBLAS call, and the other device's thread too.
host_wait_at_least polls the same host-mapped flag (the kernels' signed flag - seq test) before the four
cross-group waits of the chunk path, so the kernel finds the flag up. The GPU work and its order do not
change.
The server ends an engine an earlier run lost (strata_control/procs.py) before it
starts the next one, and waited 10 s after the kill. An engine with tens of GB
of pinned memory takes longer to give it back, so the new engine's memory guard
saw 50 GiB still in use and loaded a 40 GiB arena into 10 GiB. The wait is now
up to 120 s, with a progress line every 10 s; one still there after that is said
in plain words (what it is, why starting now is risky, what to do) and the start
goes on: nothing is refused, the engine waits for RAM itself and warns.
Also documents what the parser repairs (#4078#4427#4077) and the arena RAM wait
in docs/DETAILS.md. Tests: serve/test_leftover_engines.py and a fake-clock case
in tests/terminal/test_procs.py.
An arena load with 10 GiB available for a 40 GiB arena sent a 60 GiB host into
memory reclaim until it was reset. The check that is there warns only (a
cgroup limit below the arena still stops the start), and its own example is an
engine that is still exiting. Before the warning, the load now waits for the
RAM to come back, re-reading it every 0.25 s for up to 20 s, and says so in the
log every 5 s (STRATA_ARENA_RAM_WAIT_S changes the seconds, 0 loads at once).
If it is still short the load goes on as before, with the numbers; below half
of the arena the warning says the whole PC can stall in reclaim and names a
systemd MemoryHigh= as the way to keep one program from taking all the RAM.
Nothing is refused or capped.
detail::wait_for_ram is the loop with its sampler and sleeper passed in, tested
in file_expert_source_test (test_wait_for_ram). expert_source.cpp passes
g++ -std=c++20 -fsyntax-only against the CUDA 13.1 headers; the engine itself
was not built or run for this change.
serve/server.py run on its own with a hand-written config (no "arch": "deepseek41") gave DeepSeek-V4.1 the Qwen defaults: the stop ids
of <|im_end|> and <|endoftext|>. The DeepSeek tokenizer splits those strings into pieces, so a "|" or ">" in a reply ended it
(finish_reason stop at token 19 in front of a Markdown table, or right after "## 3. Recom"), and the real end token (id 1) was not a
stop id, so it came out as the text <｜end▁of▁sentence｜>. Reproduced on 2 x RTX A6000 with the reporter's request: the config with the
arch answers 730-1161 tokens six times out of six; without it, stops at 192-436 tokens, one of them right after "## 3. Recom".
The tokenizer's own pre-tokenizer name ("deepseek3") now picks the family when the config names none; a named arch is never changed.
#4427: tool calls came back as text or with a wrong argument.
- A JSON array or object argument with a slip in it (a trailing comma, single
quotes, a bare string between two members, brackets left open) is read as
written instead of reaching the client as a string that fails its schema; the
reporter's todo_write array with "pending", "in_progress" in one object is the
test. Valid JSON is untouched; text that is no array or object stays text.
The repair is counted (/metrics strata:tool_args_repaired_total) and logged.
- A parameter's </parameter> followed by </tool_call> ends the value: the call
whose </function> was left out is a call.
- The same holds for a call written inside the thinking, and one whose output
ends on </function> inside the thinking; a following opener confirms an
omitted </tool_call> there too, as in the answer.
- The call announced while streaming and finished by the end of the output
carried empty arguments in its final event (read by MCP and the Responses
assembler): it now carries the parsed ones.
#4077: after the reasoning ended at a closing think tag, one more at the start
of a line in the answer is the stray: the model quoted the tag while thinking.
It is dropped and the answer's code tracking starts over, so a call that follows
is not taken for a quote by a fence or backtick the reasoning left open. The
reasoning after the stray is still sent as the answer (a stream cannot take it
back); the call arrives. Logged and counted.
Adds serve/test_tool_corpus.py (49 specimens: the reporters' examples, the
Qwen3.8-Flash-Next form, the drift shapes, DeepSeek DSML; each whole, in pieces
of 1, 3 and 7 bytes and cut in two at every byte, streaming on and off) and
serve/test_tool_routes.py (Chat, Messages and Responses, whole and streamed).
setup.py said the CUDA 12 engine 'works with driver 528 or newer' (525 on Linux), NVIDIA's minor-version line, while the signed feed's cuda12
packages (built with CUDA 12.9) carry min_driver 575.51 / 576.02 and strata update refuses them below that: a V100 PC on driver 556-560 was told
the engine would work and then that the feed had no engine for it.
strata_control/driver_floor.py reads the floor from the installed cuda12 package, else the signed feed (once, 10 s at most, only in a setup run
that needs the CUDA 12 engine), else the figure the published packages carry. Setup warns with that number (recommend, never force: a locally
built engine may still run); the only stop left is NVIDIA's own line, under which no CUDA 12 program starts. docs/OLDER_GPUS.md says the same.
source_version() read CMakeLists.txt, which the packaged Terminal does not carry: prebuilt_bases() raised FileNotFoundError and ended the setup.
It answers '0' now when there is no source tree, prebuilt_bases() then looks at the latest release only, and an installed engine is not called
stale for want of a version to compare with.
strata install passed its tail to setup.py through argparse.REMAINDER, which does not take a tail that starts with an option the parser does
not know (--source, --family): 'unrecognized arguments', or the value split from its option. The options of setup.py now go behind a --
before the parse; --update and --help stay strata's own.
When the runtime refused the start because another host still owned the profile ('already has a live managed runtime'), no engine existed
and no log was written, but the error ended with 'The engine's log has the reason: <path>' and the screen showed an empty folder. A busy
profile now gets its own sentence (the other server, and strata launch --stop), and a log that is not on disk is no longer named, here or
in the sentence after an engine that stopped while loading.
The 0.2.7 policy locks the 26.8 GiB PLE table in RAM at start when table + expert arena + 8 GiB fit. What the rest of the load takes
afterwards (KV pools, pinned staging, host buffers of two GPUs) is not known then: on a 128 GB two-GPU PC it left under 5 GiB and the
automatic conversation cache turned itself off, a regression against 0.2.6, which kept 8192 MiB.
When the cache is automatic and the lock came from the policy (not --ple-io ram), the cache is sized first: if unlocking the table
would give the cache more room, PleTable::unlock() hands the lock back (the pages stay cached, the OS may reclaim them), the log says
so, and the cache is sized from the RAM then free. --conversation-cache-mib N and an explicit --ple-io ram are untouched.
ple_lock_blocks_cache() is the decision, unit-tested in src/core/ple_residency_test.cpp.
slot_load() selected the lead layer's device once and then copied into every member of a tensor-parallel group; the members' buffers
sit on their own GPUs, so on two identical cards (tp 2) the second member's copy ran with the wrong device current. A failed copy
makes slot_load() fail, and slot_take() then deletes the saved conversation. Every other per-member loop (restore, reset) already
selects the member's device; slot_load() now does too.
A batch slot's chain holds one checkpoint (the one its admission copied in). park_slot() went through park_source(), whose
drop_superseded() (#342) removes a parked entry when the outgoing chain holds that entry's deepest checkpoint; the main session's
copy of the same conversation qualifies, and the root checkpoint at the end of an agent's system prompt went with it. The next
overlapping request then read the whole 5 K-token prefix again (323 ms in 0.2.6, 4130 ms in 0.2.7).
ConversationCache::supersedes_richer() says when the park would drop an entry that has a checkpoint the slot's chain lacks; park_slot()
then leaves the slot unparked (it is overwritten, as in 0.2.6) and says so. A slot with no such parked copy is parked as before.
The predicate and its test come from the report (shenmintao, #4420).
store_gb_now() took any drop of 4 GB or more in the free RAM between the first estimate and the end of the weight upload for a
GPU that shares the PC's RAM, and subtracted a context-sized reserve (27 GB at 256K). On a discrete pair (2 x RTX 3090) the
pinned staging buffers and the page cache cause the same drop, so a 256K context shrank the store from 48 GB to 15-18 GB and
decode fell from 9-12 to 3-4 tokens a second (a regression since 0.2.4.2).
The decision is now store_gpu_shares_ram() (ds4_experts_rule.hpp): the RAM must have dropped AND a device must report
cudaDevAttrIntegrated. Strix Halo, the DGX Spark's GB10 and the Apple path keep the old rule; a discrete card never counts.
Unit test in tests/dsv41/experts_rule_test.cpp.
The Qwen engine's --vision stands alone, the DeepSeek engine's is a pair (--vision on|off). Taking out only the name left the value behind
(--eos 1 off --dspark auto: unknown option off, exit 2). A DeepSeek model keeps its pair whole now.
The Settings field 'GPU memory for other apps' wrote --vram-reserve-mib and the resource preference wrote --pool-workers into a DeepSeek
model's arguments; strata-dsv41 takes --reserve-mib and has no --pool-workers. Before a load, the arguments and the person's saved options
of a DeepSeek model are built from the engine's option list (src/dsv41/ds4_main.cpp, checked by a test): the reserve is renamed, anything
else is left out with its value and said in the load's log. The tune operation no longer writes the resource preference into a DeepSeek file.
The app's ceiling probe read one file. The second symptom of this branch - `direct runtime intent has an invalid timeout`,
on a build whose CLI validator was already fixed - is the reason it now reads the operation record too: a start wait is
bounded by management, by the record written before the start runs, and by the runtime, and a Control can raise any of
the three on its own. The app asks for more than 600 s only when a Control's own sources hand both of the first two to
the runtime's ceiling, so a half-raised Control keeps the wait it will accept instead of failing the load. Anything the
app cannot read, or a line that still spells out 600, keeps today's 600 s.
Checked against the Control folders on this PC and against the released sources in git: v0.2.5 and v0.2.7 both carry
`MAX_START_WAIT = 1800` with both gates at 600, so the app asks them for 600 s - which is what they take; the patched
tree and the staged 0.1.52 component ask 1720 s for a 180 GB model; the installed 0.1.49.2 component asks 600 s. The
marker for the record is `"wait_s": (` rather than `wait_s`, because the record names that key in its field lists long
before its range table, and the first match there would have pinned the app to 600 s forever.
The packaged test drives the four Control shapes (nothing, runtime only, CLI only, all three) so each half-raised
combination is a case, not a guess.
A local build of the previous commit got further and said `direct runtime intent has an invalid timeout`. A start wait is
checked three times before the engine is reached, and 27cbbfe6 raised only the runtime's: management.runtime_action (fixed in
the previous commit), journal._validate_direct_intent, and runtime.start itself. The record is written before the start runs,
so its own (0, 600) for runtime.start and (1, 600) for startup_wait_s stopped a long wait that the layer above had just
accepted.
The record now reads both ceilings from the runtime that runs the wait, the way the CLI validator does. A wait above them is
still refused, and stop and verify keep their own 600 s, which is what runtime.stop and runtime.verify take.
The tests are the reason this was found twice by hand instead of once here: the CLI test from the previous commit faked
runtime.start, so it stopped at the first gate and passed while the record still refused the wait. One test now walks
management -> journal -> runtime.start with only the engine itself faked, at 300, 600, 601 and MAX_START_WAIT, and says which
wait failed to arrive; another pins each kind's ceiling on the record directly. Reverting the journal change fails three of
them with the quoted message; with it, all four pass. StartWaitLimit keeps its records in a scratch folder.
The version gate of the first commit was wrong, and a local build of it said so on the first load of a big model:
`Start wait must be finite seconds in 0..600`. Control validates a start wait twice and the two ceilings disagree.
`runtime.start` takes `MAX_START_WAIT` (1800, 27cbbfe6), but `strata runtime start --wait` - the command the app uses -
checks the wait in `management.runtime_action`, where 27cbbfe6 left it at 600. It refuses the start before the engine
is reached, so no released Control (v0.2.4 through v0.2.7, Control 0.1.45 to 0.1.52) takes a wait above 600 s however
it is numbered: a version gate could not tell which Control would take it.
`management.runtime_action` now takes the ceiling from the runtime it hands the wait to, for both start and restart,
which is what 27cbbfe6 had already done for the direct `runtime.restart` path. The default stays 300 s and every wait
at or below 600 s is unchanged. The app reads the ceiling out of the Control it found - the same source-read it does
for the settings fields and /v1/responses - rather than guessing from a version, so it keeps 600 s on the Control
people have today and takes the longer one as soon as that Control is updated. A Control it cannot read falls back to
600 s, which is what the app asks for now.
Tests: the CLI path gets the ceiling test that only existed for `runtime.start` before, and the packaged load test
drives the three Control shapes (no ceiling, runtime 1800 with a CLI stuck at 600, CLI following the runtime) instead
of a version string.
The app waited a flat 300 s for any model, so a big model on a slow disk ended with `startup_wait_expired` while the engine
went on reading: #4191 reports it at about 180 GB and says editing 300 to 600 in this file fixed it on 0.2.1 but needs
several files now, and #3459 is the same stop on Windows after the tuning stage. Main solved the Terminal's side for
itself (27cbbfe6, #3949) and left this file with 300 s, or the config's engine_ready_s clamped to 600 s.
That clamp was written the same day as `MAX_START_WAIT = 1800`, on a branch whose Control still refused more than 600 s —
its message says "at most the 600 s Control allows". Control now takes 1800 s, so the clamp's reason has gone: it is the
only reason the app cannot use the size figure the Terminal already uses.
- startWaitFor(config,entry,model) takes the model's own size where the app has it (the installed model's bytes on disk,
else the catalog's download figure) and applies main's shape: ten minutes, about eight more seconds per GB above 40 GB.
A model nobody can size keeps the 300 s it always had, and a config's engine_ready_s is a floor, never shortened.
- The ceiling comes from the Control this app talks to, not from a guess: 600 s before Control 0.1.50, 1800 s from there.
Control rejects a wait above its own ceiling before it starts anything, so an older Control keeps today's behaviour.
- Where the start now waits longer than the server's own engine wait (900 s), the run config gets `engine_ready_s` above
it, as `strata launch` does: the server ends the engine when its wait runs out, so a start that waits longer than the
server can never win. It is only ever raised, and only in the launch's own copy of the config.
No new variable, no change to Control or the Terminal, and nothing to set: the wait follows the model, as main chose for
the Terminal. The app's outer five-minute health loop is not the load wait in packaged mode (Control has already reported
ready), so it is left alone.
Validation: Linux, Node v22.23.1, against upstream/main 5e8a4cca on the same machine. `npm test` in apps/host: 1227 tests,
1120 pass, the same 2 failures as on upstream/main (component-updates "an older host left running" and timeskip-real
"Stop ends the job"). The two new tests cover 180 GB at 600 s on Control 0.1.45, 1720 s on 0.1.50 with the server readying
after it, a small model keeping 300 s, the 1800 s ceiling, and setup's 3600 s for DeepSeek being honoured and never
shortened. The 11 version and size cases were checked against the formula directly, including an unknown Control version,
which falls back to the conservative 600 s. Not run: a real load on a real PC, Windows, macOS.
A bare `<function=` in prose or code (no name, or a name not followed by a
parameter or close) counted as a quoted call, so every later </parameter>
was skipped: the call came out as content, or never finished when streaming.
Adds the review's two values to test_quoted_call_values.
A parameter close inside a call that the value itself opened (more
<function= than </function> before it in the value), or one followed by the
same parameter name again, is part of the value. Prose between two calls
still ends the first call. Test: serve/test_quoted_call_values.py.
discover_run_config picked the first installed model by id. With a DeepSeek and a Qwen model installed, "deepseek-..." sorted first, and the
test start ran the engine under test (strata.exe) with DeepSeek's options: "unknown argument: --eos", server_exited, engine 0.1.50 refused
("did not start correctly on this computer"). A DeepSeek model runs on strata-dsv41, which has its own package, so it is left out; with only a
DeepSeek model installed the update has no model to test with, as on a first run.
strata_cli/launch/settings.py saves the model into a preset/choices file lower-case ("--model iq3_xxs"),
and LaunchRunner.setup_args() forwarded it unchanged. setup.py's --model choices are the MODELS table's own
keys ("IQ3_XXS", "Gyro-S"), so argparse refused the argument: starting a saved preset with any choice that
differs from the installed config (--vision, --context, --kv) reran setup and stopped there. Typing the same
name on the command line worked, because cmd.py normalises it through Catalog.resolve().
setup_args() now resolves the pair through the same Catalog.resolve() and keeps the old value when it
cannot. The except is deliberately broad: the resolver reads the model table, and a table that cannot be
read here must not turn a launch that worked into a traceback - the values then go on to setup unchanged,
exactly as before. The new test drives that case with a resolver that raises.
The prompt-side settle is 800 ms because it sits inside the prompt's own time, and for a run with no next request
(the CLI, a one-shot tool call) that is the right price. In a server it is not: an agent's turn ends, it runs its
tool for a second or ten, and that time is free for the swap rounds the *next* answer needs.
Measured why it matters (job d18-attrib, 2026-10-28): the same candidate engine with the settle at 6,000 ms
decodes a 256-token code answer at 10.92 / 11.14 tok/s, at 800 ms at 9.82 / 9.62 - 12% paid for the prompt.
The idle settle is meant to give that back without returning the 6 s to the prompt.
How: Model::settle_idle(cancel) runs cache_settle's swap rounds on its own thread, without the RAM store preload,
for at most DS4_SETTLE_IDLE_MS (6,000) and stopping between rounds when cancel is set. The serve loop starts it
at the top of each iteration (nothing in flight), and cancels and joins it the moment a line arrives, before any
request work touches the cache. DS4_SETTLE_IDLE=0 turns it off; the prompt-side budget is unchanged.
Not measured yet: the benefit needs a serve-path harness (two GEN requests with an idle between them). On the CLI
the thread is never started, so no CLI number moves.
The switch was on for gfx12 only, and ds4_kernels.cu said why: "gfx11 and gfx1151 (Strix Halo) are untested
and keep the scalar kernels". Measured on Strix Halo 2026-10-28 (2.00 bpw, 32K prompt, the synchronised
chunk profile): the sparse-attention stage is 43.3% of the prompt's GPU time and runs at 2.3 TFLOPS
effective; with DS4_FAST_ATTN=1 the stage falls from 38,340 ms to 6,976 ms and the prompt from 101.6 s to
70.8 s - +43.6%, 322.5 -> 463.0 tok/s - with the same greedy first token (27408,).
The plan's D-P3a had closed this on +1.6 / -0.2 / +1.2 / -2.8% end to end, which is what 4K-8K shows; at
32K the term is five times its 4K size and the switch is the whole difference.
fast_attn_default() takes the gfx1151-on-Windows flag as its own argument (the existing three-argument calls
in the test are updated), ds4_kernels.cu decides it from the device name once, and experts_rule_test covers
the new cell: gfx1151 Windows on, 0.2.6 arithmetic off, DS4_FAST_ATTN=0 off, =1 on anywhere.
The settle sits inside the prompt's own time: prompt: N tokens in X ms includes it, so a 6 s budget was
6 s of every DeepSeek first token. Measured on Strix Halo 2026-10-28 (p4k fixture, three interleaved rounds
each, --dspark off): 6,000 ms -> 153.1 tok/s, 800 ms -> 190.0 (+24.1%), 0 -> 197.4 (+29.0%); the decode of
the 256-token answer that follows pays 2.9% at 800 ms and 3.8% at 0. 800 ms keeps 1,412-1,436 swaps against
the 6,000's 987-1,046, so it is still moving the hot experts in, it just stops paying for the last 5 s.
An agent turn's answer is 50-300 tokens (where the swaps buy nothing) and its first token is the 800 ms, so
this is the plan's WP-8 option (2) without the server-loop work of the idle settle (option 1, still the better
answer and not done here).
DS4_SETTLE_MS still overrides it (0 = off, any value in ms).
- src/program/generate.cpp: the gfx1151 Windows prompt chunk goes from 8,192 to 16,384. The prompt reads every expert once per
chunk, so a bigger chunk reads them fewer times: 32K 1,250.6-1,260.0 -> 1,286.3-1,307.9 tok/s, 64K 1,188.4-1,191.0 ->
1,234.3-1,238.2 (+3.0% / +3.9%, interleaved pairs); at 4K and 8K the chunk is the prompt and nothing changes. The buffers'
reserve grows to ~10.9 GiB and the cache keeps all 24,576 slots (84.7 GiB free at a 64K context). STRATA_GFX1151_CHUNK=0
keeps 6,144; auto:32768 is another ~2% at 64K and stays opt-in.
- docs/STRIX_HALO_WINDOWS.md section 4: what 0.2.8 turns on for gfx1151 on Windows (the five-switch matrix-core set and the
prompt keeping its own buffers), the measured prompt table with the opt-outs, the chunk.
- docs/STRIX_HALO.md section 5: the Windows gfx1151 exception to 'the set is opt-in on an integrated Radeon'.
Numbers: Qwen3.8-Flash-Next IQ3_S, engine 0.1.52, prompts 4K to 64K, x1.59 to x1.72 over 0.2.7; first-token KL 0.0014 mean
(bar 0.0228); the own-buffer path is bit-exact (KL 0.00000 against the lending engine with the same switches).
Two prompt defaults for Strix Halo (gfx1151) on Windows, measured on the strix box with the
0.1.52 engine and Qwen3.8-Flash-Next IQ3_S (ids prompts of 4,095 / 8,190 / 11,999 tokens,
interleaved pairs, one engine at a time):
1. The matrix-core prompt set (HIP_WMMA, PF_GEMM, PA_FAST, HC_UPMIX, PF_FUSED) is on by
default, as it has been on the Radeon 780M since 021-minipc: 4K 688.8 -> 1,192.8 tok/s,
8K 787.0 -> 1,283.5, 12K 750.4 -> 1,256.5 (x1.63 to x1.72). SELECT_WMMA stays out
(x1.003 at 8K on the f-024-strix-pp branch); PF_F16 was measured and adds nothing on
this pack, so the bit-changing surface is the five switches the branch gated.
First-token KL against the 0.2.7 default 0.00210 / 0.00039 / 0.00214 at 4K / 8K / 12K,
top-1 the same, noise floor 0.00000 (the accepted HC_Q8 bar is 0.0228 mean).
2. The prompt path keeps its own buffers instead of lending expert-cache slots. A lent
slot's expert is re-read from the host arena during every chunk at ~0.5 GB/s here (the
arena is unregistered in UMA mode): 823 slots = 2.75 s of a 3.3 s 4K prompt in the
issuer thread. With own buffers every expert stays resident (24,576 slots, 'experts
streamed 0') and the prompt is 8.5% (4K) to 8.9% (8K) faster. STRATA_PREFILL_BORROW=1
keeps 0.2.7's lending, STRATA_GFX1151_DEFAULTS=0 the whole block; --prefill auto keeps
the 8,192-token chunk the gfx1151 rule chose instead of falling back to 2,048.
arch_defaults_test: the Windows gfx1151 expectations and the new opt-outs.
Only the engine / Control number: the app, the Terminal and the Metal engine keep their versions here; the release bump of the other files
is the release's own commit.
nvidia-smi gives a GB10's memory.total as [N/A]. tools/strata_mcp.py's own NVIDIA probe (the fallback of the launch hardware page)
dropped the card; it now counts it as setup.py's gpus() does (arm64 only). strata doctor --live accepted no card with that figure; on arm64
it lists the card with no memory figure (x86 still refuses it). The DeepSeek fit plan for a unified-memory GPU with no BIOS carve-out
(GB10) no longer speaks of Windows, a BIOS and a Radeon, and quotes the numbers measured on the Spark. The Windows Strix Halo plan text
is unchanged (tests).
tools/dgx_bench.py, dgx_warm.py, dgx_compare.py, dgx_quality.py, dgx_quality24.py, dgx_needle.py, dgx_turns.py, dgx_longctx.py, dgx_128k.py,
dgx_kvresident*.py, dgx_concurrent.py and dgx_smoke.py (the acceptance gate in one command; its data folder is now an option and its scratch
files live in a folder of their own) and docs/DGX_SPARK_LINUX.md.
- strata_control/targets.py: host_platform() names Linux ARM64 'linux-aarch64' (as uname and Tauri do; a Mac stays macos-arm64) and the
cuda backend is committed to there (nothing else is: no HIP, SYCL or CPU engine for arm64 Linux)
- strata_control/feed.py: linux-aarch64 is a feed platform for engine, dsv41 and control packages (and a Tauri target name, no app is built)
- packaging/package_terminal.py: the Terminal for linux-aarch64 with python-build-standalone 3.12.15 aarch64-unknown-linux-gnu (SHA-256 from
the release's SHA256SUMS); packaging/lock_wheels.py + locks/runtime-linux-aarch64-cp312.txt: its hash-pinned wheels (the same 23 packages
as the x86_64 lock; platformdirs resolved to 4.13.6 against 4.13.0 there)
- packaging/make_local_feed.py: --dsv41 for the DeepSeek packages, linux-aarch64 terminals; packaging/install.sh installs on Linux ARM64
and says the engine is for the DGX Spark (GB10) only, instead of refusing
- packaging/build_cuda_linux_aarch64.sh: the release recipe (strata + strata-dsv41 + the reader, the engine package and the separate
DeepSeek package, sm_121 + compute_121, driver 580.65, build id naming the commit)
- tests: test_linux_aarch64.py, the Spark rows in test_release_matrix.py
Nothing changes for another platform: every edit is a new table row, a new branch for aarch64 Linux or a new file.
- gpus(): a GPU whose memory nvidia-smi does not report (GB10) on arm64 is counted as Strix Halo's is: the RAM less the OS share
(UMA_OS_LEFT_GB), uma True, dedicated_gb 0, so the low-RAM and fit rules do not count the same memory twice. #3622 listed the whole RAM
as VRAM beside the RAM.
- cpu_info(): an arm64 CPU is named from lscpu (Cortex-X925 + Cortex-A725), not 'unknown CPU'. The PCIe probe is skipped for a
unified-memory GPU (nvidia-smi's 1.0 x1 there means nothing).
- get_prebuilt(): on arm64 Linux the ready-made engine is the linux-aarch64 package of the signed feed (engine_from_feed); without one,
setup compiles as #3622 had it do always. setup no longer forces --build on arm64.
- Step 7 on a GB10 (is_gb10: uma, sm_121, arm64 Linux, nothing else): --mmap-experts (one copy of the experts, all 24,576 cache
slots), --prefill auto:32768, --mtp-q4 all, --mtp-hnorm stream, each measured on one GB10 with the answers unchanged; --draft-vocab
en is printed as a tip, not set.
- The CUDA repository key URL is sbsa on arm64; the DeepSeek-engine-missing message names the Spark build script on arm64.
Tests: test_setup_arm64.py (unified-memory shape, is_gb10, lscpu name); test_setup_pins.py skips its x86-zip Engine tests on arm64 Linux.
nv_defaults.cpp gets an sm_121 table beside sm_86 and sm_60: STRATA_PF_FUSED=1 (fused IQ prompt kernels), STRATA_PREFILL_NVFP4=w4a4
(NVFP4 prompt MMQ with FP4 activations), STRATA_HC_UPMIX=1 (fused hyper-connection read), STRATA_SPEC_COST=1 (cost-aware MTP window).
Measured on one GB10 with paired A/Bs (docs/DGX_SPARK_LINUX.md); a switch the user set is kept. STRATA_SM121_PERF=0 turns the table
off, and so does the 0.2.7 master switch STRATA_DEFAULTS=026. Unlike the sm_86 / sm_60 tables these are not bit-exact (the fused prompt
path and the FP4 activations round differently; 24 of 24 greedy answers were identical), so the start line does not call them exact.
moe_fused_iq.cu: on a GB10 the work-item rule never reached its 128-row tile (the launch average is above 112 rows an expert at every
prompt size), so pick_ww takes the 128-row tile from 65,536 routed rows a launch up (+3% prompt at 16K and 32K; 4K keeps 64).
STRATA_PF_TILE_DEBUG=1 prints the choice. Other parts take the line they took before.
- ds4_gg_cpu.cpp: amd_family() on a non-x86 CPU read no variable the rest of the function uses; it now reports no AMD family
- exl3_cpu.cpp: kGemmK moved above the AVX-512 section, as exl3_cpu_gemm_scratch() sizes by it on every CPU
- ds4_model.cu: uma_free(): on an integrated NVIDIA GPU on Linux cudaMemGetInfo counts only MemFree, while the model files' page cache
holds most of what the kernel can still give back. The placement, the CPU-lane warm-up and the vision tower size their room from
MemAvailable less DS4_UMA_HEADROOM_GIB (default 6) when that is larger. A discrete GPU, HIP and Windows are not touched.
On a GB10: 12,163-12,226 of 15,744 experts in graphics memory where cudaMemGetInfo's free placed 11,961.
#3622 made the strata target compile on aarch64. This adds what a Spark build needs beyond that:
- the trellis dispatcher (tq_cpu.cpp, its scalar rows) is part of the arm64 kernel library, as it is in the Metal build, so the
DeepSeek engine's CPU lane links; ds4_tq_cpu.cpp names the AVX units on x86-64 only
- a STRATA_PORTABLE build on arm64 sets ggml-cpu's -march to armv8.2-a+dotprod+fp16+i8mm (STRATA_ARM64_ARCH) instead of the build
host's; GGML_NATIVE OFF alone would leave plain armv8-a and no NEON dot products
- the x86 kernels' parity programs and benches are not configured on arm64; the test suite defaults off there (-DSTRATA_BUILD_TESTS=ON
still builds what builds)
- spark/build_engine.sh: the reproducible build of strata, strata-dsv41 and the picture reader into an engine folder, with the CUDA
libraries beside them (RUNPATH $ORIGIN)
Every change is behind STRATA_CPU_X86 being OFF or inside a new file; the x86 and Metal builds do not take it.
The 0.2.7 DeepSeek changes (persistent-grid IQ kernels, int8 rounding batches, hc_mix_split with a block counter, head columns,
chunk-path knobs) are kept; the Metal engine's hooks sit on top of them: every kernel launch is strata_ds4_launch(), the rounding
batch ends in the Metal multi-quantiser dispatch, the head stays at 4 columns a pass on Metal (8 moves the KL), and the macOS
parallel expert prefetch is macOS-only.
Same 15 files as the 0.2.6 bump (issue template, AGENTS.md, CMakeLists.txt 0.1.52, desktop and host package files, Cargo.toml and the strata-desktop entry in Cargo.lock, SettingsUpdates comment, qa1-storage test, metal 0.1.52-metal, build_hip_linux.sh PKGVER 0.1.52, strata_cli PRODUCT_VERSION, the two version tests), plus the Strata-0.2.7-<file> names in the md files.
The engine's own tower (319 MiB) through strata launch and setup (--vision off still turns it off); setup fetches
the pinned 972 MB mmproj; the server sends pictures to the tower. Checked on 4x R9700, 1x R9700 and a real
launch on 2x A6000 (download, pack, picture answer over HTTP); 0/10 stalls, KL unchanged. Also: a newer shard 1
from --model-path (ISTA's 2026-10-26 chat template) is accepted after its own size and SHA-256 check.
Pipelined groups beat the serial default at 2 and 4 streams on 2x A6000 (+17/+22%) and 2x A4000 (+7..12/+17..20%);
one stream equal, -4% on A4000 IQ3_XXS. STRATA_BATCH_GROUPS_NV_SERIAL=1 or STRATA_DEFAULTS=026 gives serial.
ISTA re-published (and patch_chat_template.py rewrites) the first shard of each set with a larger chat template, so a shard 1 on disk can be 11 KB
larger than the pinned revision's. setup refused it (size mismatch; the GGUF preflight does not read DeepSeek tensors, so it was not taken as a
whole own file). The pinned revision stays the download; the newer shard 1 is accepted by its own size and SHA-256 with --gguf-dir.
native_arithmetic_parity failed on an Intel i9-13900K (AVX2 + AVX-VNNI, ggml built native)
in five of its eight configurations: "gu=16 down=20 ff=96 nt=2 token=0". Not the VNNI row
path: STRATA_NO_AVXVNNI=1 changes nothing, STRATA_NO_IQ4NL=1 passes. The IQ4_NL down
reference (iq4nl_reference_rows) differs from that ggml's one-token dot on the scalar tail
block of a row with an odd number of blocks (ff=96 is three): that ggml does not contract
`sumf += d * sumi` into an FMA, the reference lets the compiler do it. On the other
machines tested the two agree, so it is a property of the ggml build, not of the CPU.
With an unfused tail in the reference all eight configurations pass on that machine.
The invariant path now checks each reference once against the linked ggml-cpu on synthetic
blocks before using it (iq4nl_reference_ok for IQ4_NL with 2, 3, 5 and 20 blocks;
iq512_reference_ok for types 18, 21 and 22 with SwiGLU, on AVX-512 CPUs). On a mismatch
it prints one line and the invariant path uses ggml's own one-token dot per token: slower
on such a build, never different. The Default path is not touched; only the opt-in
STRATA_SPEC_PROB=1 decode reaches this code.
Tests: native_arithmetic_parity, eight configurations, on the i9-13900K (4,866,048
comparisons, with the fallback line), z440 and the owner PC (see the series notes).
0.2.1 made a layer split over exactly two NVIDIA cards with --batch and no --batch-groups run one serial group (measured
then on 2x RTX A4000, IQ3_XXS: +12..16%). With the 0.2.7 code the pipelined groups are faster at 2 and 4 streams on
every box and quant measured, so the default goes back to the pipeline.
Measured (tools/serve_load.py, --batch 4, 256 tokens per request, interleaved pairs, total tok/s, pipelined / serial):
- 2x RTX A6000, IQ3_S, all experts in VRAM: 1 stream 101.7 / 101.7; 2 streams 125.7 / 107.4 (+17%); 4 streams 171.4 / 141.0
(+22%). The v0.2.6 binary on the same box: serial default 104.9 and 137.2, --batch-groups auto 124.6 and 169.6.
- 2x RTX A4000, Q2_0 pack (17037 of 24576 pairs cached, 4 clean pairs): 2 streams 80.2-82.5 / 75.0-75.5 (+7..9%, 4 of 4);
4 streams 110.8-111.5 / 94.8-95.3 (+17%, 4 of 4); 1 stream 63.6-68.7 / 68.2-70.3.
- 2x RTX A4000, IQ3_XXS (13056 of 24576 pairs cached, 3 pairs): 2 streams 73.2-73.7 / 65.8-66.5 (+12%, 3 of 3); 4 streams
100.5-100.7 / 82.9-83.6 (+20%, 3 of 3); 1 stream 58.4-60.5 / 60.7-62.6 (serial about 4% faster, 3 of 3).
The single stream runs on the solo path in both; only the A4000s show the 4%.
What changed: batch_groups.hpp has a pure rule nv_serial_wanted(env value, STRATA_DEFAULTS=026). STRATA_BATCH_GROUPS_NV_SERIAL
set to anything but 0 asks for the serial group, an explicit 0 wins over STRATA_DEFAULTS=026, and with the variable unset or
empty STRATA_DEFAULTS=026 gives v0.2.6's serial group. An explicit --batch-groups (a number or auto) still wins. Three or more
cards, AMD builds and --batch-mtp/--adapt-async handling are unchanged. The serial message names the opt-in.
Tests: batch_groups_test (the rule over the env values and 026, two cards pipeline, the existing serial cases). Real engine on
2x A6000 (IQ3_S), no --batch-groups: the engine says "2 groups of 2 slots" / "2 groups of 1 slot"; with 4 and 2 slots, greedy,
greedy+seeded sampled, slots finishing at different lengths and a cancel in the middle of a draft, every slot's tokens equal its
solo decode and the pipelined build's. A long-prompt slot differs from the pre-merge base build in the same way with the flag
and without it (it is the parent's prompt path, not this change).
Not covered: no run on AMD or on three cards; the 4% single-stream cost on the A4000s is measured on one box.
THIS COMMIT ALONE MAKES THE WALK A DEFAULT. Drop it and the layer-major prefill stays
an opt-in (STRATA_PREFILL_LAYER_MAJOR=1) with the chunk-major path as the default.
Rule (include/strata/core/sm120_series.hpp, pure, tested in src/core/arch_defaults_test.cpp):
- prefill_layer_major_default(hip, stages, v026): on for two or more stages on CUDA,
off on one GPU, in a HIP build, and under STRATA_DEFAULTS=026 (qwen_defaults_026());
- prefill_layer_major_resolve(env, ...): STRATA_PREFILL_LAYER_MAJOR=0 / 1 (any text
atoi reads) wins in both directions, over the rule and over STRATA_DEFAULTS=026.
Every decline of the earlier commits stays and runs in lm_prepare as before (HIP,
image prompts, DFlash2 taps, KVA, stop_at_chunk, a pending force_chunk, a peer GPU,
APR / NVFP4 / WMMA / fused layers, STRATA_EXPERT_NIBBLE, no VRAM, no pinned RAM):
such a prompt reads chunk-major with nothing but a log line. When the walk is on by
default (nothing set) the VRAM margin is 512 MiB instead of 96 and the super-chunk
target 32768 tokens (the size measured) instead of 8192; "a short prompt" is not logged.
STRATA_PREFILL_LM_TOKENS and STRATA_PREFILL_LM_MARGIN_MIB still win.
Measured (z440, 2x RTX A4000 16 GB, Qwen3.8-Flash-Next IQ3_XXS, experts streamed, 5
interleaved pairs, time to first token, medians): 8K 2.8x, 20K 4.8-5.7x, 32K 5.1-5.5x
faster (141 s to 26-28 s); decode 38.6 vs 38.4 tok/s; needles 12 of 12 found in both
arms. The speed-up is larger than the report's 2x because --prefill auto picks 512-token
chunks there. The first-token KL numbers are in the gate notes (3976-kl.sh).
Not covered: one GPU (stays opt-in), HIP, SYCL, Metal, three or more stages, a card
with more VRAM than the pack needs (where the experts are VRAM-resident and the
chunk-major path has little to re-read).
Problem. #3948: with STRATA_QFUSE=1 the verified text changes and MTP acceptance drops, on CUDA. The QFUSE kernels are bit-exact
(gr_parity, gdn_parity, 10,500 random windows each on sm_86 and sm_120, after 91a4b8a67), yet the engine's greedy text still differed
from QFUSE off on every prompt tried. Model run, l620 (RTX PRO 5000, sm_120), Qwen3.8-Flash-Next IQ3_S, MTP --spec 6, identity-gate
flags, 4 prompts of 4K and 12K tokens, 96 greedy tokens: QFUSE on differed from off on 4 of 4 (the off-vs-off control was identical).
Cause. A window of one token advances the GDN state and the indexer tail itself ("self-commit") so that commit() needs no commit
graph. Verifier::record_window decides this with
`T == 1 && !always_publish_ && !batch_rec_ && !g_qfuse() && one_token_self_commit()`: with STRATA_QFUSE the fused GDN output
kernel takes the q8_1 destination and the window does not self-commit. Verifier::commit() skipped the commit graph with
`last_t_ == 1 && !always_publish_ && one_token_self_commit()`, without the QFUSE term. So with QFUSE on, every accepted one-token
window left the recurrent state (and the indexer tail) one token behind. Windows of one token are the ordinary case when the drafter
offers nothing, which matches the report: fewer drafts offered, lower acceptance, different text, and an unchanged text where
no window of one token occurs. On HIP (where QFUSE is the default on gfx1151) one_token_self_commit() is off unless
STRATA_ONE_TOKEN_COMMIT=1, so the two sides agreed there.
Change. commit() takes the same condition: `... && one_token_self_commit() && !g_qfuse()`. Without STRATA_QFUSE nothing changes.
Result (same l620 run). QFUSE on with this change against QFUSE off, same build: logits md5 and the 96 greedy tokens byte-identical
on all 4 prompts (IDENTITY PASS); the build without the change differs on all 4. The default path (QFUSE unset) of the merged head
against the f-027-int2 head (2a2264c21): first-window logits and greedy tokens byte-identical on the same 4 prompts.
Not covered. Serve answers and draft counters (3948-qfuse.sh step 2, SERVE=1), sm_86, and the reporter's 40-prompt set.
The report asks for the first data mismatch between the GGUF row bytes and what the direct reader delivered, and for an early
check of the gathered values; neither exists. Its trace stops at "the embedding has 96 non-finite values" without saying whether
the reader gave a wrong row or the table holds one.
With STRATA_PLE_CHECK=1 (a tester switch, off by default; one static flag read per gather call when off) every direct-mode decode (read_row, collect,
gather_batch, also from the prefetched slots) checks the row's 160 values. For a value that is not finite or beyond 1e5 (an IQ4_NL
row cannot decode beyond 8.3e6; real PLE values are orders smaller) it re-reads that row from the file with a buffered read and prints
one line to stderr (at most 8 per table): the row, its file offset and page offset, the value, and either "the delivered bytes ARE the
file's bytes" or "THE DELIVERED BYTES DIFFER FROM THE FILE from byte N" with the first 8 bytes of each. That separates a reader
fault (the bug) from a damaged table. gates/3602-ple-direct.sh sets it in its direct arms.
Checked: ple_reader_test --selftest on z440 with the switch on prints the "ARE the file's bytes" line for the synthetic table's random
rows (which decode to huge values); with it off the output and the bytes are unchanged (ple_reader_selftest passes).
Not checked: the "DIFFER" branch has no natural trigger (that is the point); it was read, not run.
Problem. #3948: with STRATA_QFUSE=1 the verified text changes and MTP acceptance drops (RTX 3080, sm_86). The report suspected the
group counter or the nat_xq_ offsets of fused_gr.cu; the audit and the new two-group test (previous commit of this stream)
rule those out. What the new test found instead, on an RTX 5070 (sm_120, CUDA 13.0, MSVC), over 10,500 random windows of two groups: in
about 1 run of 400 one q8_1 code of the fused image differs by exactly one from native_quantize_q8_1's, in every mode (group-relative,
window-indexed, direct, graph replay). The scale/sum pair and the other codes agree.
Cause. native_mmvq.cu, which holds native_quantize_q8_1 (the quantiser every consumer of a q8_1 image was written for), is
compiled with --use_fast_math (CMakeLists.txt, the native_* list), so its `amax / 127` and `x / d` are the fast full-range division
(div.full.f32, up to 2 ulp off). fused_gr.cu (the hyper-connection read's tail, gr_up_q32_kernel) and verify_kernels.cu (the GDN
output norm, gdn_q8_1_store) are not in that list and divide with IEEE rounding. roundf(x / d) then lands on the other side of a .5
at the rare element whose quotient is next to a boundary. One changed activation code per ~10^6 values is about one per few
hundred tokens at 80 hyper-connection reads of 2560 values per token: enough to move a near-tie in the verify, and the
draft (which quantises with the separate kernel) disagrees with the verifier a little more often, which lowers acceptance. That
matches the report (acceptance 0.755 against 0.79, different text, and an unchanged text where no draft is made).
Change. include/strata/kernels/q8_1_finite.hpp: q81_div(a, b) is `a / b`, except in a translation unit that defines
STRATA_Q81_FASTMATH_DIV (CUDA device code only, never HIP), where it is the same PTX division the fast-math file gets (div.full.f32).
q8_1_quant and the scale use it. fused_gr.cu and verify_kernels.cu define the macro; nothing else does, so every other kernel,
the separate quantiser and every HIP build compile to what they did. STRATA_QFUSE is off by default on CUDA (it is on by default on
gfx1151, which is HIP and unchanged), so the default path is bit-identical.
Result (RTX 5070, sm_120): gr_parity's two-group window test: 10,500 runs with 1,500 random inputs for each T, all modes, zero
differences (before: dozens); the whole of gr_parity, gdn_parity and gdn_parity with STRATA_GDN_SPLIT=1 pass.
Tests. gr_parity: the two-group test prints the first differing bytes (block, field), and STRATA_GRP_REPS=N / STRATA_GRP_KEEP_GOING=1
repeat it longer and count every failure. gdn_parity: the q8_1 destination check uses its own stream (native_quantize_q8_1 refuses the
null stream, which crashed the test on Windows), unbuffered stdout and STRATA_GDN_PARITY_TRACE=1.
Not covered. sm_86 (the report's card) has not run the new test yet (z440 was reserved for other jobs); the cause is not
architecture specific. The model run (gates/3948-qfuse.sh) is still owed: QFUSE on against off must now be byte-identical, and the
draft counters equal.
Problem. #3844: --spec 5 measured +6.1% decode on an RTX 5060 Ti, but a saved preset that carries it was left out entirely
("outside the safe values [3, 4]"), and an imported one was refused. The tuner's BOUNDS are the values it tries; they are
not a finding that a value is harmful. The never list (--spec 6 and wider: 4% to 16% slower, --expert-cache 0, ...) is.
Change (Terminal / Control; touches strata_control/tune_knobs.py, strata_control/presets.py, strata_cli/tune_commands.py and
serve/server.py).
- tune_knobs.check_setting_parts(args, env, ctx) returns (never, bounds). check_setting is the two lists together, in the
order it had, so the tuner still skips --spec 5 and its tests are unchanged.
- presets.overlay applies a preset whose only problems are outside BOUNDS and returns info["warnings"], one short line each
("--spec 5: not tried by the tuner, kept because it is your setting"). A never-list setting still means "not applied".
presets.create and import_preset refuse only never-list settings.
- serve/server.py prints each warning as "[strata] warning: ..." when the preset is applied; strata_cli tune_commands
_effective adds "warnings" to its payload.
Tests. tests/tuning/test_presets.py: a preset with --spec 5 is saved, applied (the flag becomes 5, the input config is not
modified), warns once, and an in-range preset has no warnings; the never list still refuses --spec 6; import of --spec 5 works.
tests/tuning/test_autotune.py: --spec 5 stays in the tuner's refused list. python -m pytest tests/tuning: 274 passed;
tests/terminal -k "tun or preset or overlay": passed.
Not covered. The app's Details text does not show the warning yet (it reads the payload key above).
Problem. #2408 / PR #2409: restoring a large parked conversation on a layer split ended the server with a CUDA out of memory
on a later card. The PR checks the free VRAM of every card against --vram-reserve-mib before a restore. Its own review
points out that a loaded card is always below that reserve, so the check would skip nearly every restore on a split (and
on one GPU). It is not ported.
What a restore allocates. conversation_snapshot_restore writes into the session's existing buffers (pools(st), the GDN
state, the indexer tails); refresh_residency only launches kv_stream_reset or copies into the existing ring. The elastic K/V
growth (kvg_ensure) allocates, but only on one GPU. The one thing a restore can create on a card is the calling thread's
private copy stream for it (conversation_copy.hpp private_stream), which it takes from the pool of finished threads when
there is one (#3350). A failed restore is fatal by contract (the session may be half written). So the need is one stream
and one event, and only for a card whose stream does not exist yet (a snapshot loaded from disk, or a restore on a thread
that never copied on that card).
Change. conv_cache.hpp: kRestoreFirstUseMib (64) and restore_fits(free, stream_ready, n, need): a restore is refused only
for a card that has no copy stream yet and less than 64 MiB free. conversation_copy_stream_ready(device) says whether a
card has one. generate.cpp: before the restore, each card of the run is checked; a snapshot that would not fit goes back to
the cache and the prompt is read again, with a log line. A card that has its stream is never refused, however full.
No sample (a failed cudaMemGetInfo) means the restore goes ahead as it did.
Tests. conv_cache_test: the predicate (cards with a stream are admitted at any free size; the edge at the need; no
sample / no cards / no need). conversation_copy_streams_test: copy_stream_ready before and after the first copy, in the pool
after a thread ends, and the measured cost of a card's first copy stream and event (printed; the test requires under
16 MiB, so 64 MiB leaves a wide margin). conversation_snapshot_test: after the first restore, six more restores of two
images leave the free VRAM where it was.
Not covered. The pure predicate and the generate.cpp code build on z440 (CUDA 13.0); the GPU tests are listed in the stream
notes. No model run: gates/2409-restore-guard.sh (a held-VRAM start on a layer split) shows the skip line and an
unchanged answer on the second turn.
Problem. On a layer split the later cards are left with very little VRAM after loading (#3702: 68 MiB on the last card; #3638:
268 and 666 MiB, then every batched prompt read of more than ~400 tokens ends non-finite after hours; #3797: the weight arena
fails with 320 MiB free and the message does not say who holds the rest). The window-graph eviction (12b0bce95) and the copy
stream pool (#3350) are in already; this is the rest of the code audit of what is allocated after load.
Audit. The table of every lazy allocation after load (what, when, size, how a failure is handled) is in
phaseA-tests/notes-027-gh-eng-d.md. Bugs found and fixed here, each small:
- verify.cpp capture / capture_batch and mtp.cpp finish_capture: the cudaGraphUpload and its sync were only printed. The
upload is a warm-up, but a failure (no VRAM) stayed as the thread's pending error and the next kernel wrapper's
cudaGetLastError ended the engine. The error is now cleared and the first launch uploads the graph.
- verify.cpp capture_commit and mtp.cpp finish_capture: the commit graph called cudaGraphInstantiate directly (no
eviction of older layouts, unlike the window graphs) and reported the status of the end-of-capture call, which is success,
so the message read "no error". capture_commit now goes through instantiate_evicting and reports the instantiate's status.
- instantiate_evicting instantiates into a local and sets the member only on success (what a failed instantiate leaves in
its output is not specified, and a non-null member means "captured").
- prefill.cpp carve: after a failed take of the shared region the sub-allocators had no base and each cudaMalloc'd its
buffer (hundreds of MiB in all) on the card that had just run out. carve now returns at once; the caller reports the
failure as before.
- generate.cpp: a failed set_stage_helper / set_peer is reported and the run goes on, but its CUDA error stayed pending;
cleared.
Found and not changed: relend() re-plans only the first participant (#3638 comment). Read: it runs only from the VRAM
command, which is off on a layer split, so it cannot touch a later stage's loan. Other candidates (ignored cudaEventCreate for
the pipelined fences, unchecked expert copies, cudaFuncSetAttribute flags set before success) are in the notes.
Changes for the report of #3797 and the reserve (#3702).
- include/strata/program/gpu_report.hpp: on a failed weight-arena allocation (first card and each stage) the engine now
also prints, for every card of the run, the free and total memory and the largest other programs that hold memory on it
(NVML, loaded at run time; where the driver does not list them it says so). Text only; the failure is the same.
tests/program/gpu_report_test.cpp pins the text.
- include/strata/program/stage_reserve.hpp: the arithmetic of stage_room (what the split keeps free on a card) moved out of
generate.cpp unchanged; tests/program/stage_reserve_test.cpp compares it with the old expression over a grid. The file
also holds a PROPOSAL for a larger window term on a later card (96 MiB + 6 MiB per further window size + 48 MiB, 174 MiB
for --spec 4) which nothing calls: the default placement is not changed. gates/3702-later-reserve.sh measures it.
- The low-VRAM warning of a later card now adds how many MiB the plan meant to keep free there and how far off that was.
Tests: gpu_report_test, stage_reserve_test (host only, built on z440).
Not covered: no model was run. What the ~700 MiB the plan does not count on the last card is (window graph sets, kernel code
loads) needs the run in gates/3702-later-reserve.sh.
Problem. #3948 reports that STRATA_QFUSE=1 changes the verified text and lowers MTP acceptance on an RTX 3080 (sm_86); the
control (QFUSE off against off) was noisy too. The reporter read the code and suspects the per-group counter in
fused_gr.cu gr_q8_tail (not reset when a graph replays with another T) or the nat_xq_ offsets of a window of two groups.
Audit (no model run). The counter is reset by the block that finishes the group (atomicExch before it reads the 32 values),
so a launch leaves it at zero whatever T is; two blocks own a 32-column group in every kernel that calls gr_q8_tail, the
32-column kernel (gr_up_q32_kernel) quantises on its own without a counter; the fence pattern is the usual
threadFenceReduction one. The two destinations are consistent with their consumers in verify.cpp: the attention half's
xq_ is indexed from the group's first token (the projections read it from row 0), the expert half's nat_xq_ by the window's
token (the experts read it from tb). The GDN side (verify_kernels.cu gdn_q8_1_store, rows from t_out_begin) matches its
consumer too. I found no defect by reading. One thing the tests below will show if it matters: nvcc ignores the
`#pragma clang fp contract(off)` in gdn_q8_1_store, so on CUDA the ds.sum field could differ by one ulp from the separate
quantiser (the q8 bytes would not).
Tests (synthetic data, no model).
- gr_parity: the engine's G = 2 call pattern - the window split at (T + 1) / 2, two fused_gr_read_multi calls in a row on
one stream with the same counters, group-relative destinations (xq_) and window-indexed ones (nat_xq_), direct and as a
graph replay; fresh random inputs for every one of 200 repetitions for each T = 2..8; after every run both groups' q8_1
bytes equal native_quantize_q8_1 of `mixed`, nothing outside the written ranges changed, and all counters are back at zero.
- gdn_parity: gdn_step_norm_multi with a q8_1 destination for every T and t_out_begin (verify and commit calls): the bytes
equal native_quantize_q8_1 of the output rows, the outputs and the state are the ones of the call without it, and
nothing is written past the rows. A second ctest, gdn_parity_split, runs all of gdn_parity with STRATA_GDN_SPLIT=1 (the
4-blocks-a-head step, which has its own output-norm kernel).
Not covered. The reporter's model run (QFUSE on/off, MTP, 40 prompts): gates/3948-qfuse.sh. The tests were built on z440
(CUDA 13.0, sm_86); their GPU run is listed in the stream notes.
Problem. #3602 reports a deterministic all-logits NaN with --ple-io direct on IQ3_S (Windows, two GGUF shards) where
--ple-io mmap passes; a diagnostic build found non-finite values already in the gathered PLE embedding of one token. The
bytes the model got were not the file's (the largest finite value, 7.9e6, is what a random fp16 scale gives an IQ4_NL
row). This commit audits the reader and builds the test the report asks for; it does not claim to have found that
fault.
Audit (no model run). src/ngram/ple_reader.cpp, src/platform/direct_file.cpp and the consumers (PleTable gather_batch /
prefetch_rows / collect, prefill's double-buffered ple_gather, verify's windows): rows are copied and cached under the
reader's mutex after a complete read, short reads are checked per row, straddling rows get a two-page read, the offset
arithmetic is 64-bit, the set-associative cache and the slot buffers are only touched under the lock, and prefill's two
host buffers are ordered by the ple_copied events. I found no way to hand out a row before its read completed or from a
stale buffer.
One real bug, in the error path. After a failed read the I/O worker parks for good (worker_loop), but issue() went on
queueing jobs, so the next collect() waited for them for ever. issue() now returns a ticket with no jobs once the reader
has failed, and collect() returns the error at once (and drops the ticket). Same change in the SYCL copy
(sycl/src/ngram/ple_reader.cpp).
Tests.
- src/ngram/ple_reader_stress_test.cpp (ctest ple_reader_stress): 7 table shapes (header 0/192/4000/4051, rows of 1, 90,
110, 160 and 4096 bytes, with and without bytes after the table) x 15 reader configurations (io thread or caller,
in-flight 1/2/3/64, row cache of 0, 16 and 100000 rows, injected completion delays, keep-alive), four client threads
with one to three tickets outstanding collected in any order; every delivered row is compared byte for byte with a
buffered read of the file; collected buffers are compared again later (a late write would show); rows on every page
phase, the last rows, out-of-range rows; the reader after a failed read (the file is cut short under it); with --big a
5 GiB sparse table with rows around 2^31 and 2^32 and at its end.
- table_is_consistent_under_concurrent_use in src/ngram/ple_reader_test.cpp: one PleTable used by a prompt thread
(big gather_batch), a decode window (prefetch_rows, gather_batch) and single gathers at once, against the mapped reader.
Results (z440, Linux, g++ 13, reader + direct_file + test built alone): plain build with --big OK; ASan+UBSan OK. The
error-path test against the unchanged reader hangs (the test exits 3 after 20 s); with the change it passes. ThreadSanitizer
flagged a data race that was a mistake in the test (the delay changed from a second thread in the caller-thread mode,
which has no lock); fixed in the test.
Not covered. The NaN itself: it needs a run on the reporter's configuration (gate script gates/3602-ple-direct.sh). The
Windows run of the new test is listed in the stream notes.
Gyro-L (agentionai's 50 GiB file) stores its experts as a pair trellis: 128
values per block, a fp16 scale and 64 steps of K bits (K = 5, 6, 7), two
weights per step taken from the first pair of the TQ2_T codebook entry the
step's state selects. The engine refused the file because it did not know the
three type ids. This adds them next to the existing trellis types 144-146
without changing those.
What changed:
- gguf_reader.hpp / tools/gguf_reader.py: names and block sizes (42 / 50 / 58 bytes).
- tq_cpu.*, tq_states.hpp, tq_avx2.cpp: tq_supported / tq_kbits / tq_block_bytes
for the new ids (the bytes per block are 2 + 8K for the pair layout, 2 + 4K
for the others), a scalar pair kernel (the reference) and an AVX2 pair kernel
that is bit for bit equal to it; AVX-512 uses the AVX2 pair kernel.
- iq_kernels.cu: TqInfo, the lane decode (tqp_states8, a byte window), the
dot product, the decode-once Split, the fp16 dequantizer and the format lists
for 147-149. They are reached only by a pack that holds these types.
- moe_mmq.cu, strata_ggml_tq_overlay.cmake, third_party/ggml-tq/tq.cuh,
ggml-common-tq.h: the MMQ prompt path for the three types (overlay
enumerators, traits, tile loader branch, instances).
- native_expert_parity: names tqp5-7 and skips its synthetic check, with a
message, on a ggml that has no trellis traits.
- docs/GYRO.md: the "not supported" sentence now says how Gyro-L is read.
Changes to the contributor's patch:
- tq_cpu.cpp called the AVX2 pair kernel without the x86 guard the rest of the
file has, which breaks the Metal build (it compiles tq_cpu.cpp without
tq_avx2.cpp): the pair rows go through one helper that falls back to the scalar kernel.
- moe_mmq.cu named GGML_TYPE_TQP5-7 under STRATA_MMQ_TQ, but the public
agentionai llama.cpp has the older trellis types and not these: a new
STRATA_MMQ_TQP (set for the overlay and for a tree that has the tqp instances)
guards them, so that tree still builds. The HIP overlay lists get the three
instances too, and ggml_tq_overlay_test checks them (and sets the CMake
policy its IN_LIST needs).
- native_expert_parity: the skip test compared a type-name pointer to null, but
ggml_type_name of an id past the table reads past it (the test crashed on
an upstream ggml); it compares against GGML_TYPE_COUNT.
- docs/GYRO.md keeps the plain description and leaves out the contributor's
measured speeds (we have not repeated them).
Tests added: src/kernels/cpu/tqp_cpu_test.cpp (no ggml, no model: the scalar
kernel against a bit-by-bit reference from the format definition, every ISA
equal to scalar for 1-8 tokens and for gate/up rows, the type predicates for ids
0-299 against the old formulas), the pair types in tq_pascal_bench (fp16
dequantizer bit for bit against the definition, decode kernels within 3e-2,
decode-once == per-column), tools/test_gguf_tqp.py, and native_expert_parity
tqp5/6/7 registered when the ggml tree has the tqp instances.
Checked (no model): tqp_cpu_test passes on x86 (AVX2 on z440; AVX2 and
AVX-512 on the owner PC with MSVC); tq_pascal_bench on an RTX 5070 (sm_120,
MSVC build): 0 failures, the TQP dequantizer matches the definition in all
bits, and the checksum lines of TQ2_T/TQK6/TQK7 are byte-identical to the
build without this change. Builds: CUDA sm_86 (gcc), sm_60 (CUDA 12.9),
sm_120 (MSVC), SYCL, and the CPU files for aarch64.
Not covered: the contributor's own parity runs need a ggml tree that carries the
types (not public here), and the 50 GiB model run (load check, coherent
answer, 32K prompt, MTP, Gyro-S/M identity) is still to be done. Metal, SYCL and
HIP have no kernels for the types; HIP was read, not built.
Problem (KeyanHu-git): the gated proposal needs the choice between the MTP window and a
lookup window to be fixed by the history, not by the draws of the rounds it will verify.
Freezing the policy for the whole request (the first commit) does that, but also freezes
the cost table, so a repeat-heavy request kept the window sizes it inherited. In the
author's diagnostic workload the frozen policy chose mean lookup windows of 5.87 and 3.38
on two repeat requests; with this change 4.31 and 4.05 (from the PR text; not
reproduced here).
Change: the request's copy of the DraftPolicy (source_policy) gets three calls, used only
when gated_proposal is on:
- observe_committed(): the round's accept and probe counts, applied at once. A lookup
window counts as a probe of its size (tried_), so a size whose cost is only guessed is
tried kProbes times, not every round.
- defer_cost(end, t, ms): the round's time is held. A time covers the window and the next
MTP draft, so it depends on every Philox counter up to
max(p + T - 1, p + a + 1 + S_mtp).
- apply_costs(committed): applies, in order, the held times whose end is behind the
committed position, with the same update observe() makes (including the STRATA_SPEC_COST
clamp), in the context bucket the round ran in; delayed_n_ keeps a held time from
counting as a second probe.
choose() counts the probes as cost_n_ - delayed_n_ + tried_.
Port note: the DraftPolicy on the current tree keeps its costs per context bucket
(STRATA_SPEC_COST), so tried_, delayed_n_ and the held times carry the bucket. A gated
request does not run with STRATA_SPEC_COST (it falls back to the older rule), so its
copy stays in bucket 0.
Default path: unchanged. choose() reads two new terms; on any path that never calls the new
functions tried_ and delayed_n_ are +0.0 and cost_n_ - 0.0 + 0.0 == cost_n_ exactly.
Checked by driving the old DraftPolicy (namespace-renamed copy of f-027-gh-eng) and the
new one with the same random call sequences (observe, observe_chain, observe_probs,
set_context, seed_costs, set_robust, choose, chain, keep_draft; 400 sequences of 3000
steps): 25,983,538 bitwise comparisons of picks, costs and rates, 0 mismatches; one
observe_committed call makes the same harness report 60,528 mismatches. generate.cpp:
nothing calls the new functions when gated_proposal is false.
Tests: draft_policy_test (the PR's added assertions, on the bucketed class) and
spec_admission_test pass on z440 and on the owner PC.
Not covered: the sampling argument for the delay boundary (that a held time cannot steer
a window through a counter that is drawn again) is the author's; a unit test cannot show
it, only the distribution run (gate script 3416-spec-prob-dist.sh, arm NEWLK) can.
Problem (KeyanHu-git): the rejection rule of the previous commit judges each row against
the target's probabilities p. For a native (IQ) pack the CPU experts round a token's rows
differently depending on how many tokens share its expert in the window (the grouped
kernels against ggml's one-token dot, #152). So p of one row moved with the window width,
and the same sampled text could depend on the draft length. On random blocks a grouped
Default call differs from the token alone in 700 to 12000 of about 18000 output floats
for the IQ gate/up and IQ4_NL down formats.
Change: NativeArithmetic { Default, WindowInvariant }, fixed for one pool batch.
native_gu_rows / native_down_rows become thin wrappers of one template with the arithmetic
as a parameter. The Default instance keeps every old branch (the new conditions are
compile-time true there). WindowInvariant gives each token the bits of the one-token call
on this build and CPU: kernels that already match it stay grouped (NVFP4, Trellis and
Hadamard rows, KQ256 when ggml runs its AVX2/FMA dot); the IQ rows that differ use
iq512_reference_gu_rows (types 18, 21, 22: the same decode, ggml AVX2's accumulation
order, one weight decode per group) or iq4nl_reference_rows (IQ4_NL down), and anything
else falls to the one-token dot per token. ExpertPool::run_split_multi_native takes the
arithmetic and expert_pool_dispatch_multi passes it; generate.cpp uses the invariant
pool only when the MTP drafts probabilistically (mtp.prob(), i.e. only with
STRATA_SPEC_PROB=1). Prefill, greedy decode and every other decode run Default.
The pool picks the entry point with one compare and a direct call per chunk.
Default path (STRATA_SPEC_PROB unset): unchanged, shown by running the old and the new
native_gu_rows / native_down_rows (Default) on the same random blocks: 61 format, width
and amplitude lines (IQ2_XXS/XS/S, IQ3_XXS/S, IQ4_XS, NVFP4, Q4_K, Trellis with and
without folded Hadamard, IQ4_NL / Q2_0 / Q5_1 / Q8_0 down; widths 1 to 8; row sub-ranges),
output hashes identical in 10 settings of the environment switches on z440 (AVX2) and
in 13 on the owner PC (Zen4, AVX-512, MSVC), including STRATA_NO_IQ512, STRATA_IQ_GATHER
and STRATA_IQ_MT_MIN=1. g++ codegen of the AVX-512 row kernels is not byte-identical for
5 of 87 functions (inlining and register allocation of the same source); the AVX-512
kernels were run only under MSVC.
Tests: native_arithmetic_parity, eight ctest configurations (default, AVX, KQ256,
IQ_MT_MIN=1, no512, ggml only, NVFP4 AVX2, NVFP4 ggml), registered in CMakeLists.txt and
sycl/CMakeLists.txt. It compares the invariant entry points at every group width 1..8
with the one-token calls, bitwise, for the IQ, NVFP4, Q4_K/Q5_1/Q8_0, Trellis and
Hadamard formats. All eight pass on z440 and on the owner PC; the existing CPU expert
tests (iq_avx2_parity, iq_avx1_parity, iq512_gather_parity, pool_tasks_test,
nvfp4_cpu_parity, router_dot_parity, q2_bitplane_parity) pass.
Not covered: an ARM CPU (not built here), an AVX-512 machine with fast gathers (Sapphire
Rapids), the layer-split call site, and any model run.
Problem (KeyanHu-git, refs #2028, #2591): with STRATA_SPEC_PROB=1 the default
--spec-min-p gate tests the token drawn from the draft head's distribution q, but the
verifier still judged the draft against the unfiltered q and treated a failed gate as a
plain sample from p. That shifts the output distribution: with p = (0.5, 0.5),
q = (0.875, 0.125) and threshold 0.5 it emits (0.5625, 0.4375).
Change: the gate is part of the proposal. Keep q's mass on the tokens that pass the gate
without renormalising (mu); a draw that fails the gate becomes NULL. An offered token D
is accepted with min(1, p(D)/mu(D)); a rejection and NULL draw from the normalised
positive part of p - mu; a position with no attempted proposal is a plain sample from p.
Serve and the CLI build the same packet (core/gated_proposal.hpp) and give it to the
verifier explicitly (Verifier::set_spec_proposal, sample_tokens_proposal). The explicit
call has no plain-p fallback: an invalid packet row is reported, not sampled around.
STRATA_SPEC_PROB_GATE=top keeps its normalised-q rule.
Default path (STRATA_SPEC_PROB unset): unchanged. mtp.prob() is false, so gated_proposal
is false, the new draft-result argument is null, GatedDraftLimit does nothing,
Verifier::run takes the old branch (set_spec_q clears the measure flag), and the
sampler kernel's old instance is the old code with the new branches compiled out.
Read hunk by hunk; no hunk runs a new branch or a new allocation per token. The
default-identity model run is written as a gate script and not run yet.
Port notes against the PR as submitted:
- Unsupported combinations (parallel or multi-device decode, --lookup-chain,
--adapt-swaps, PCIe share, --spec-min-p not finite, STRATA_SPEC_COST, DFlash2,
STRATA_SPEC_T_CYCLE, --spec-corrupt, Metal) used to be an error that ended the
request or the server. They now warn once and keep the older rule, so setting the
variable never stops a run. --adapt-swaps 96 and the PCIe share are on by default, so
the exact rule needs --lookup-chain 0 --adapt-swaps 0 --pcie-frac 0.
- The lookup source of a probabilistic window is chosen by a copy of the policy fixed at
the request start, without asking whether the MTP's first guess agrees. Learning from
committed rounds inside a request is the policy commit of this series.
- The CLI loop on the current tree has the out-of-memory window retry, DFlash2 and
STRATA_SPEC_T_CYCLE; the gated path does not use the retry (a narrower window would
change the room the packet was built for). Serve keeps merged first windows
(STRATA_ADMIT_MERGE_FIRST) at their length.
- docs/DETAILS.md: two short bullets instead of the PR's long paragraph (that paragraph
no longer exists on this tree). bench/results and the PNGs are not ported.
Tests: gated_proposal_test and spec_admission_test (default, pick, top, exact-top-only)
pass on z440 (CPU, g++ 13) and on the owner PC (MSVC). On the owner PC's RTX 5070
(sm_120, MSVC build, synthetic data, no model) spec_proposal_parity, spec_verify_parity and
sampler_parity (default, one-block and old-kernel configurations) pass. Also built:
CUDA sm_86 on z440 and boston, SYCL on P1006 (engine and host tests). The Metal path is
not built here.
Not covered: a model-level distribution test (chi-square of the first tokens against
plain sampling) and the identity run with the variable unset. Both are written as gate
scripts (3416-spec-prob-dist.sh, 3416-default-identity.sh) and not run.
Prefill on one card, decode on another, session handed over through RAM; exact token for token (state hash match at 4K
and 16K on 2x A4000 and 2x A6000). Nothing changes unless split_serve / --split-serve is set. On two alike cards it is
slower than the layer split on every workload measured (docs say so); it can only pay on dissimilar cards. Idea and
manual setup by MarcoTool (#4000).
Measured on 2x RTX A6000 (IQ3_S): slower than the layer split on every workload;
exactness checks pass (Q2_0 on 2x A4000, IQ3_S on 2x A6000).
Idea and manual setup by @MarcoTool in #4000.
No copy, code-constant, JSON-spelling, retrieval or unanswerable defects (retrieval 12/12, JSON 3/3, unanswerable 9/9, extraction equal to off); two long-winded answers, which the runs with the prior off also show two or three of per run. Story +15%, code +37% against off.
f-027-gh-eng gained STRATA_EXPERT_NIBBLE (4-bit expert slots, the sm_60 experimental
build): only run_impl gathers such a layer for MMQ. The layer-major walk would read
the slot as GGUF blocks, so lm_prepare now declines when any layer is converted and
the prompt takes the chunk-major path. Compiled out of every build without
STRATA_EXPERT_NIBBLE_BUILT.
The engine did not build on aarch64: the CPU expert path assumed the x86 AVX
kernels, the CPUID probes and the x86 spin/fence intrinsics. After this change
the strata target compiles on aarch64 (CUDA host code, native GGUF packs).
On x86 nothing is selected differently.
What changed:
- CMake: STRATA_CPU_X86 is decided from CMAKE_SYSTEM_PROCESSOR (an unset
processor stays the x86 build). On aarch64 src/kernels/cpu/portable.cpp
stands in for the AVX kernel sources; native experts run on ggml-cpu's NEON
dot products. -ffp-contract=off on aarch64. STRATA_ISA_FLOOR is ignored with
a warning there.
- include/strata/platform/cpu_relax.hpp: strata_cpu_pause / strata_store_fence
map to _mm_pause / _mm_sfence on x86 and to yield / dmb oshst on aarch64.
verify.cpp, pool.cpp, session.cpp, mtp.cpp and graph.cpp use them.
- expert_layout.cpp: the CPUID/XCR0 probes compile on x86 only and report no
AVX on other CPUs.
- generate.cpp: --cpu-oracle-q8-0 is rejected on a non-x86 CPU.
- setup.py: nvidia-smi's "[N/A]" memory on GB10 (unified memory) takes the
pool size from /proc/meminfo on ARM64 only; the CPU probe no longer needs
the x86 "flags" line; setup warns that the ARM64 engine is compiled on the
PC and which packs work.
Changes to the contributor's patch, found while moving it onto the 0.2.7 line:
- portable.cpp had no iq512x_rows / iq512x_supported / iq512_gather_on (added
to the x86 kernels after the PR was written): the aarch64 link would fail.
- The Metal build (Apple silicon) shares these files. It keeps its barrier
(dmb ishst), its plain router loop (its bf16_rows_dot_multi is a stub), its
Metal start-up block in generate.cpp and its "unknown" CPU name.
- The CPU side of the GPU doorbell: strata_load_fence() after the spins that
wait for a flag the GPU raised (dmb ishld on ARM, nothing on x86 and on
Apple silicon), so the data written before the flag is not read early.
- A card whose memory nvidia-smi does not report stays unlisted on x86 (the
new fallback is for ARM64 only).
- tests/terminal/test_setup_arm64.py.
Checked: with the x86 CUDA build (sm_86, gcc 13 -O3, z440) the objects of
verify, session, mtp, graph, expert_source, pool, expert_layout and generate
are byte-identical before and after (the x86 wrappers are macros for the same
intrinsics). Every C++ file of the strata target compiles for aarch64 with
clang, and the symbols of the CPU kernel library link up (syntax and symbols
only: no ARM machine, no ARM run). The Metal-shared files compile for
aarch64-macos with STRATA_GPU_METAL.
Not covered: no ARM hardware here. The tests that include x86 headers
(iq_avx2_parity and the other AVX parity tools) and the DeepSeek target do
not build on aarch64; the strata target does not need them. Memory ordering
on ARM is reasoned, not run. The Q2_0 and Gyro packs need the x86 kernels.
DETAILS.md gets two sections: the decode switches (GRAPH_PRIO, MMVQ_IL_PF) with the
two sm_80-only defaults named, and the prompt switches (PF_FUSED_NT / BATCH / F16 /
WS, GDN_TC, PLE_COMPACT / PLE_COPY_STREAM, EMBED_VRAM, DEQUANT_SIDE) with what is
bitwise and what is rounding-level. MMVQ_IL_TABLE.md gets the sm_80 row of the table
list and a paragraph on the hoisted-load tables (kIlPf_80 / kIlPf58_80, Q6_K only) and
on measuring a card with mmvq_il_parity --bench --emit-table --pf 0,3,4.
Rewritten from the PR's text: every figure is labelled as the reporter's (one CMP
170HX; we have no sm_80 card and measured none of them), and the prompt-switch
section is new (the PR documented only the decode switches and the conversation
cache).
Original docs by linxuhao (PR #3779, commit 6d6f0f39).
Problem: the interleaved small-batch matvec (native_mmvq_il) has one weight block
in flight per warp: its four `if (kbx < bpr)` sections each sit behind their own
branch, so on a GA100 (ncu: 432 GB/s on the qkv Q6_K matrix) the loads of a group
are not issued together.
Change, opt-in (every variant is bitwise native_mmvq): native_mmvq_il_pf_kernel
issues every weight load of a group (4 virtual warps x R rows) before any of the
group's dots, with the addresses clamped to the row's last block for a tail, and
with the ring variants the next group's loads before this group's dots. The
per-lane accumulation order is the plain kernel's. P2's forms replace
conversions by exact ones where the value is a small integer: (float)p becomes
p + 1.5 * 2^23 minus 1.5 * 2^23 for |p| < 2^22, and the saturating byte subtract
of Q6_K's codes (0..63, so nothing saturates) becomes two instructions. Q4_K,
Q5_K, IQ4_XS and Q6_K have the variants; Q3_K (the Q2_0 pack's type) keeps the
plain kernel.
- STRATA_MMVQ_IL_PF=1: the measured table (kIlPf_80 / kIlPf58_80: Q6_K only, read
only when the card is sm_80; on every other card 1 does nothing);
=2..6: one variant for every type and shape (A/B only, any CUDA card).
- Unset or 0: the plain kernels through the same launch code (the variant argument
is 0, and the added code is one cached environment read and a switch).
- CUDA only; HIP has no interleaved path.
Tests: mmvq_il_parity --pf a,b,.. bit-checks each listed variant against
native_mmvq for every type and shape, T = 1..8 and rows 0/1/2/4, and with --bench
times them interleaved (PFBENCH lines) and prints the joint (variant, rows) table
in the source form of kIlPf_<cc>. New ctest entry mmvq_il_parity_pf runs it with
--pf 0,2,3,4,5,6 on whatever card is present.
Reporter's numbers (linxuhao, CMP 170HX): decode +1.4 % (two reference / on / reference /
on sets) over the plain kernels; with STRATA_GRAPH_PRIO decode +1.3 % in their
per-feature pair (4K prompts +3.9 %, 16K +1.2 %, 52K +0.2 % in the same pair, which
the reporter reads as noise). Q4_K, Q5_K and IQ4_XS variants tie the plain kernels
there. Theirs, not measured here.
Original change by linxuhao (PR #3779, commit 116992ab).
Harness only; no engine code. The bench part of mmvq_il_parity times the
interleaved matvec cold (24 copies of the weights rotated past the L2), hot (one
copy), hot with other weights' reads in between (--gap MB, what a window's GEMVs do
between a prefetch and its consumer), and with the timed weights in a persisting L2
set-aside (--persist); --ab runs the five arms per case in one process, because
the clocks ramp over the first ~100 ms of a run. A 150 ms warm-up of launches runs
before the first timed case, and MMVQ_IL_TMIN bounds T from below.
The parity part is unchanged: without --bench the same checks run in the same
order (every rows choice against native_mmvq, bitwise). The bench code is CUDA only
(the test is built with STRATA_ENABLE_CUDA); the persisting-L2 calls are left out
under STRATA_HIP_GFX906.
Original change by linxuhao (PR #3779, commit 1f4b8139).
Problem: the decode window graph runs every kernel node at the launch stream's
priority, so the one-block indexer kernels of the side branch wait behind the
queued blocks of the big q GEMV, and the shared expert competes with the routed
experts.
Change, opt-in (bitwise by construction: no kernel changes, the same launches on
the same inputs; only the dispatch order moves): with STRATA_GRAPH_PRIO=1 the
side mixer's branch stream (GDN gates, QSA indexer chain) is created at the
device's highest priority, cs_ and side2_ in the middle, the shared expert's
stream at the lowest, and the window graph is instantiated with
cudaGraphInstantiateFlagUseNodePriority. If a driver did not copy the stream
priority into the nodes, the capture-time notes (the branch forks and the
priority streams' tails) set the attribute per node. One log line per captured
window prints the node priority histogram. A card whose priority range is 0..0
changes nothing.
CUDA only: the members and the three methods sit under !STRATA_USE_HIP &&
!STRATA_HIP_GFX906 && !STRATA_GPU_METAL; graph_prio_on() is false there.
Merged by hand with #3702 (the window-graph eviction): instantiate_evicting now
takes both keep_t and the flags, and the retry after freeing other window
sizes keeps the flags (the merged code would otherwise have dropped the node
priorities on that retry).
With the variable unset: every stream is created as before and the capture-time
notes return at once.
Tests: none of its own (a priority is a scheduling hint with no value to compare).
The window graphs are exercised by the engine; see the checklist for the
reporter's greedy-answer identity check.
Reporter's numbers (linxuhao, CMP 170HX, IQ3_S, --spec 3, 3K context, interleaved in
one session, each arm with STRATA_QIL=1): reference 125.3 / 126.6, GRAPH_PRIO
127.8 / 129.0, reference 125.0 / 126.8 tok/s, +1.8 %. Greedy 11 of 11 identical. In
the later per-feature pair together with STRATA_MMVQ_IL_PF: decode +1.3 %. Theirs, not
measured here.
Original change by linxuhao (PR #3779, commit 68c487c3).
One section in DETAILS.md for the switches the four conversation-cache commits
add: what each copies where, its VRAM and host memory cost, that they stay off
on HIP, gfx906 and Metal, and the measured times.
Rewritten from the PR's section: the times are labelled as the reporter's
(linxuhao, one CMP 170HX on PCIe Gen2 x4; we have no sm_80 card and measured
none of them), the independent RTX 3090 data of bitaccesscomau is added, and the
pool row describes STRATA_CKPT_POOL_READY=N instead of the PR's change to the
meaning of STRATA_CKPT_POOL=N (STRATA_CKPT_POOL=1 is unchanged).
Original section by linxuhao (PR #3779, commit a04f4399).
Problem: restoring a parked conversation copies its K/V from the host over the
link: 27-107 ms over a PCIe Gen2 x4 link.
Change, opt-in and only with STRATA_PARK_ASYNC=1 (CUDA only): a park also keeps
the whole K/V of its image in a VRAM slot of its own (64 MiB granules, up to N MiB
in all, only while 1.5 GB of VRAM stays free beside it); the background copy to
the host reads from the slot, and a restore of that image copies device to device
from the slot (about 1 ms) without waiting for the background copy. Each K/V
layer carries its own join-and-release token, so any order of destruction is safe
(the layers outlive their image when they are retained for the next park). Under
VRAM pressure the elastic K/V's growth releases every slot and the stage, then
retries once (both growth sites in generate.cpp).
With the variable unset the slot code is not reached: the budget is 0, a restore
finds dev_tag 0 and takes the host path, and the growth sites see
conversation_park_vram_release_all() return 0 and behave as before.
Tests: none of its own (it needs a card with the VRAM for the slot and a parked
conversation); see the checklist for the reporters' A/B/A conversation check with
STRATA_PARK_VRAM_MIB set.
Reporter's numbers (linxuhao, CMP 170HX): restore of a 3K conversation 28-47 ->
7-12 ms; outputs identical. The first restore of a run is slower (41.6 / 56.9 ms)
than later ones (6.5-12.6 ms) and they have not found why. Theirs, not measured
here.
Original change by linxuhao (PR #3779, commit bafca410).
Problem (reporter, CMP 170HX, PCIe Gen2 x4): a conversation switch's park still
paid 38-52 ms for the live state, because the STRATA_CKPT_POOL pool was empty at
every park (a switch takes two checkpoints), so the 118 MB GDN vector was
allocated fresh and page-faulted; and a checkpoint's 37 small synchronous
device-to-host copies queue behind the 4 MiB DMA pieces of STRATA_CKPT_ASYNC.
Change:
- STRATA_CKPT_POOL_READY=N (opt-in, 1..16, with STRATA_CKPT_POOL on): the pool
keeps N ready checkpoints, pre-sized to the session's state sizes with their
pages in, topped up on a background thread after any take (not with a layer
split).
- STRATA_CKPT_GATHER=1 (CUDA, opt-in): the small running-state vectors (ple, and
each layer's indexer tail, dead row and block position) are gathered device to
device into one buffer and fetched with one copy into a pinned buffer.
- The park stage is allocated rounded up to 64 MiB (a growing conversation does not
cudaFree/cudaMalloc it every time) and a VRAM-resident K/V skips the pointer
attribute query.
- The background helpers (pool top-up, the checkpoint DMA, the park DMA) leave the
host's pinned core (aux_cpus.hpp); no output depends on where a thread runs.
- A parked image whose live GDN state is still being copied joins before its
buffers are freed (SavedConversation::live_pending).
- STRATA_PARK_TRACE prints the joins, the live-state vectors' readiness and the
K/V sub-timers.
Differs from the PR as submitted: the PR made STRATA_CKPT_POOL=1 allocate two
ready checkpoints at startup (+236 MB of host RSS) and read STRATA_CKPT_POOL=N as
a count. That changes what an existing switch does, so here STRATA_CKPT_POOL=1
keeps its meaning (the pool holds dropped checkpoints only, nothing is allocated
ahead of time) and the ready pool has its own variable, STRATA_CKPT_POOL_READY.
A user of the PR's STRATA_CKPT_POOL=2 sets STRATA_CKPT_POOL=1
STRATA_CKPT_POOL_READY=2. The PR's one-checkpoint refill after a park with
STRATA_PARK_ASYNC is replaced by the top-up, as in the PR, and so is only active
with the new variable. Merged by hand into #2526's park_source.
With none of the new variables set: the stage size is rounded up only when the
park stage is on, the extra code is trace output behind STRATA_PARK_TRACE and
the join calls.
Reporter's numbers (linxuhao, CMP 170HX): park at a 3K context 64-75 -> 21-40 ms
(all park switches together, A/B/A conversation check identical, greedy 11 of 11
identical). Theirs, not measured here.
Original change by linxuhao (PR #3779, commit 56088e71).
Problem: parking the outgoing conversation copies its K/V device to host and
deep-copies its checkpoints before the next prompt can start: 213 ms at a 3K
context, 334 ms at 13K on a PCIe Gen2 x4 card (the reporter's CMP 170HX).
Change, opt-in and CUDA only (off on HIP, gfx906 and Metal):
- STRATA_PARK_ASYNC=1: conversation_kv_save copies the device K/V device to
device into a VRAM stage (only while 1.5 GB stay free beside it; otherwise the
save copies as before), and a background thread moves the stage into the host
buffers in 4 MiB pieces through two pinned buffers. conversation_park_join
waits for it: every reader of a parked image's K/V, the cache before it frees an
entry, and the image itself (SavedConversation::pending) before its buffers go.
ConversationBuffer::resize_for_overwrite leaves the added bytes unfilled for
that copy; the buffer's segments use an allocator that value-initializes only
when a fill value is given, and resize(n, value) is unchanged for every other
caller.
- With it, the checkpoints this request drops right after the park (all of them
when another state comes in, else the ones off this prompt's path) move into
the parked image instead of being copied, and a background allocation refills
STRATA_CKPT_POOL for the next save. Only the main session's checkpoints move,
not an idle batch slot's (the park_source of #2526 serves both).
- conversation_snapshot_save takes dropped checkpoints' storage (the existing
STRATA_CKPT_POOL) for its copies; with the pool off, nothing is taken.
- Unset: park_drops is null, no storage is taken, the K/V copies are the
synchronous ones; the added work is the K/V size sum for the stage request and
a few join calls.
Differs from the PR as submitted: the PR's join in conversation_file.cpp
(put_payload) is not ported; the SAVE/RESTORE handler of generate.cpp joins the
park first (see the previous commit, which carries the PR's 33a8146a fix).
Hand-merged into the park_source / ParkSource refactor of #2526.
Not covered: if the background copy fails after resize_for_overwrite, the
parked image keeps unfilled bytes; the failure is logged. A restore of such an
image would read them. No test of the park path with a failing copy.
Tests: the PARK_ASYNC path has no unit test in this commit (it needs a card with 1.5 GB free for the
stage); the reporter's A/B/A conversation check covers it, see the checklist.
Reporter's numbers (linxuhao, CMP 170HX): park 213 -> 64-75 ms (3K), 334 -> 68 ms
(13K); greedy answers identical on their 11 prompts and the A/B/A conversation
check. Theirs, not measured here.
Original change by linxuhao (PR #3779, commit 3e0c141c).
Problem: on a slow host link a turn checkpoint copies the whole GDN state to
the host on the request path (118 MB at 1.64 GB/s on a PCIe Gen2 x4 card, 72 ms;
an independent RTX 3090 on PCIe 3.0 x16 showed 73 ms to 62 ms with the switch).
Restoring a parked conversation copies it back.
Change, both opt-in and CUDA only (off on HIP, gfx906 and Metal):
- STRATA_CKPT_ASYNC=1: the GDN state is copied device to device into a VRAM stage
(one buffer the size of the state, allocated at the first checkpoint save, plus
a pinned host buffer of the same size; if either allocation fails the save falls
back to the synchronous copy), and a background thread moves it in 4 MiB pieces
into the checkpoint's host vector. Every reader joins first and only for the copy
it needs (restore, snapshot save and restore, the parked copies, the serve loop at
each request start, the SAVE/RESTORE session handler).
- STRATA_CKPT_VRAM_SLOTS=N (with ASYNC): the last N saved GDN states stay in VRAM,
tag-checked, only allocated while 1.5 GB stays free; a restore of a checkpoint
whose tag a slot holds copies device to device.
- STRATA_TRACE lines carry a steady-clock timestamp; STRATA_PARK_TRACE prints the
timing of the save steps.
- Unset: the code adds a mutex-guarded no-op join per checkpoint save, restore and
snapshot, a tag field that stays 0 and a few clock reads; the copies are the
synchronous ones as before.
Differs from the PR as submitted: the PR put a join into conversation_file.cpp
(put_checkpoint), which broke the conversation_file_test and
session_save_reclaim_test builds without CUDA and matched only the copied vector,
not the one the copy in flight writes, so a SAVE within one checkpoint copy after a
turn could write an incomplete GDN state (found by bitaccesscomau; fixed in the
PR's last commit 33a8146a). This commit carries that fix: conversation_file.cpp
stays as it is and the join runs first in the SAVE/RESTORE handler of
generate.cpp. The commit 33a8146a is therefore not a separate commit here.
One more join, found by running the new ctest entries: conversation_snapshot_sources (the SAVE
path of a session file) saves the running state with the background copy and the caller writes
it to the file at once, after the handler's join has already run. It now joins the copy of
that checkpoint before it returns; without it a session file written with
STRATA_CKPT_ASYNC=1 could hold an incomplete GDN state (conversation_snapshot_test failed
at "both files read" before the join).
Tests: conversation_snapshot_test now joins after each save and, with
STRATA_TEST_BIG_GDN=1 (a GDN state over 1 MiB, so the async path is taken), runs as
the new ctest entries conversation_snapshot_test_ckpt_async and
conversation_snapshot_test_ckpt_vram_slots: every checkpoint, capture and restore
must give the same bytes as the plain run.
Not covered: no test destroys a checkpoint while its copy is in flight (readers
join; the cache's eviction paths do not); the reporter saw 3 of about 35 engine
starts stop answering after a park with evictions, on the paths with the
switches off, and could not reproduce it in 15 reruns. The VRAM stage has no
headroom check of its own (an RTX 3090 with 147 MiB minimum free showed no failure
in 42 of 42 saves; the allocation failure falls back silently).
Reporter's numbers (linxuhao, CMP 170HX): turn checkpoint 71.4 -> 0.9 ms,
restore 114-126 -> 28-47 ms. Independent data (bitaccesscomau, RTX 3090 24 GB,
PCIe 3.0 x16, release defaults, 4 pairs): checkpoint save 73.2 -> 62.4 ms, tiny-prompt
time to first token median ratio 1.038, outputs byte-identical. Theirs, not measured
here.
Original change by linxuhao (PR #3779, commits 0d512925 and 33a8146a).
Problem: the prompt path's DeltaNet recurrence is a large part of the prompt time
of a GA100 card.
Change, opt-in and rounding-level (the output is not the FP32 kernels' bits):
src/prefill/gdn_tc.cuh holds the chunked WY-form recurrence with every product
on the tensor cores (f16x3 splits with power-of-two scaling, one block per value
head) and its prep kernel; kernels.cu dispatches it before the STRATA_GDN_CHUNKED
branch for prompt chunks of 128+ tokens when STRATA_GDN_TC is set.
Where it runs and where it does not:
- gdn_rec_tc returns "not supported" and the existing kernels run unchanged on a
card below sm_80, with fewer than 48 SMs, or with less than 122 KB of
shared memory a block (the kernel needs 124,944 B). That excludes sm_86, sm_89
and sm_120 (99 KB opt-in): the code is compiled into those builds and never
launched there. It runs on sm_80 and sm_90. A failed scratch allocation
(60.3 MB a device) also falls back.
- Device code for cards below sm_80 compiles to trapping stubs behind
STRATA_GDN_TC_SM80; the whole header sits under !__HIPCC__.
- Unset, the dispatch costs one cached flag test per call.
Tests: gdn_rec_parity gains the tc variant (T = 1..4099, 8192, 32768, six stress
inputs against gdn_rec_kh_kernel, gate 1e-4; "not on this card" where it cannot
run), --selftest-frag (each tensor-core helper on one warp against a double
reference) and --bench-tc. The new ctest entry gdn_rec_parity_selftest_frag runs
the helper self-test, which any sm_80+ card can run (77 below sm_80). The full
tc variant can only run on an sm_80 or sm_90 card.
Reporter's numbers (linxuhao, CMP 170HX): recurrence 105 -> 34 ms at 3.3K tokens
and 1,705 -> 547 ms at 52K; prompts +3.9 % (4K), +6.0 % (16K), +6.7 % (52K) over
the same base; within 2.3e-6 of the FP32 kernel in gdn_rec_parity; teacher-forced
KL against the bitwise base mean 0.0157, median 0.0011, max 0.098, top-1 100 %;
1 of 5 conversation checks identical and decode -2.6 % in their pair (they
think draft acceptance moves with the rounding; not checked). Theirs, not
measured here.
Original change by linxuhao (PR #3779, commit f1cabc99).
Problem: the NT = 256 kernels still decode every 32 values to int8 and scale
the int32 products in FP32.
Change, both opt-in and both rounding-level (the output is not bit-identical to the
int8 kernels; they are the PR's "rounding-level" switches):
- STRATA_PF_FUSED_F16=1 (implies the NT = 256 set): the decoded weights times
their block scales and the activations times theirs as F16, m16n8k16 F16
products into FP32 accumulators. The k order inside a 16-value group is
permuted the same way for both operands.
- STRATA_PF_FUSED_WS=1: native_ws_kernel, producer warps decode into an F16 ring
in shared memory and consumer warps run the same F16 products (named barriers),
so it has the F16 path's rounding. One block per multiprocessor; about 101 KB of
shared memory at its defaults, so it is set up on sm_80 / sm_90 cards and
says "not available" and keeps the 512-thread kernels on an sm_86 / sm_89 /
sm_120 card (99 KB). Compile-time knobs STRATA_WS_PROD / _SLOTS / _NB / _KS / _HTAB.
- Both take a WideBatch like the NT = 256 kernels, so STRATA_PF_BATCH above 128
reaches them. CUDA only (under !__HIPCC__); the Turing pass compiles the F16
instances against __trap() stubs and never runs them (dev_info stops before the
NT = 256 setup on Turing).
- With neither variable set the kernels, their launches and their shared-memory
sizes are as before (the F16 template parameter defaults to false).
Differs from the PR as submitted: when STRATA_PF_FUSED_Q2N=1 is on as well
(#3809, after the PR was written) the WS setup also sets up the Q2_0 gate/up
instance.
Tests: prefill_fused_iq_test_f16 and prefill_fused_iq_test_ws (ctest) run the
fused-experts test (double-precision reference and MMQ) with the switch set. The
test now skips (77) when a requested NT / F16 / WS kernel set is not set up on the
device, so a card that cannot run WS does not report a pass for the 512-thread
kernels.
Limits of the test for these two arms only: with STRATA_PF_FUSED_F16=1 or _WS=1 the
test uses the limits it already has for STRATA_PF_F16 (F16 products): fused against the
int8-rounded model rel RMS 5e-3 (2e-3 otherwise) and against MMQ 5e-3 (1e-3 otherwise).
Measured with F16 on an RTX A4000 (sm_86, z440), over the 11 pairs: against the int8-rounded
model rel RMS 2.85e-3 to 2.90e-3 and worst row 5.9e-3 to 8.1e-3; against MMQ the same;
against the FP32 reference 1.438e-2 to 1.480e-2 with MMQ at 1.435e-2 to 1.478e-2 (the
fused path stays within 1.003x of MMQ's rel RMS and 1.03x of its worst row). The rounding is the F16 rounding of the
weights times their scales and of the activations times theirs (an 11-bit mantissa, about
2^-11 relative per operand), which the int8-rounded model does not have; it is not a
different summation order. The same arm on an RTX PRO 5000 (sm_120): up to 2.90e-3 (rel RMS), worst row up to 8.1e-3.
The default path and the NT=256 arm keep the old limits and their printed errors are
unchanged: with no switch set the test prints identical output (52 lines) from the base
tree and this tree, on an RTX A4000 (sm_86) and on an RTX PRO 5000 (sm_120).
Reporter's numbers (linxuhao, CMP 170HX): one MoE layer of 3,256 tokens 8.27 ms
(NT = 256) to 7.90 ms (F16) and 7.65 ms (WS); prompts +2..+6 % over the
256-thread kernels, 1 of 11 greedy answers changes. Teacher-forced KL against the
bitwise base on their 20-prompt set: F16 / WS mean 0.0062, median 0.00006, max
0.076, top-1 100 %. Theirs, not measured here.
Original change by linxuhao (PR #3779, commit 579cbfbb).
Problem: a layer split with --batch pipelines the slots in groups, and the pipelined groups run no MTP drafts.
--batch-mtp with serial windows leaves all but one card waiting. A multi-stream deployment wants both.
What changed: STRATA_PIPED_MTP=N (with --batch-mtp) keeps the pipeline and drafts in it. A group's window carries
each of its slots' rows: the slot's token and up to N MTP proposals (at most --spec - 1; the first always, each
further one while the draft layer was at least --spec-min-p sure of the one before). The last stage judges each
slot's proposals on their own. The number of rows to keep is known only then, and by then the earlier stages run
other groups' windows, which reuse the scratch the commit replays from. So a small save graph copies those rows
right behind each window (Verifier::batch_launch_rows), and every stage queues the commit of the kept rows after
the verdict (Verifier::batch_commit_rows). The slot drafter catches up on its own rows and drafts the next
proposals as an asynchronous chain (MtpDrafter::chain_launch). Unset, nothing changes.
Changes on top of the PR as sent (review):
- The verifier window size is worked out where the verifiers are built, from the slots that fit and the final
groups, not earlier from --batch as asked. Unset it is the old expression.
- STRATA_PIPED_MTP is decided again once the slots that fit are known. It says it is off, and why, when fewer than
2 groups are left, when STRATA_BATCH_SELF_COMMIT is on, or when a group's window would hold more than
kVerifyMaxT rows. The start line names groups and slots correctly.
- batch_groups::resolve keeps its old mtp_dropped rule; only mtp_piped is new, and it also applies with
--batch-groups given (the PR's fix 1). The "--batch-groups 1" test case now expects "not dropped".
- --remote-expert-opt alone no longer blocks --batch-mtp on a layer split: the object it guards is only made with
a helper expert cache (the PR's fix 2).
- docs/BATCHING.md rewritten with the numbers measured here.
Tests: batch_groups_test (the PR's cases, four "unset is as before" cases); builds on CUDA sm_86 (Linux), SYCL
(the sycl tree has its own verify.cpp and generate.cpp and does not include this code) and MSVC. Model runs:
see docs/BATCHING.md and the gate scripts 4121-*.sh.
Not covered: AMD (HIP) was not built or run; the draft layer on the first stage; helper expert caches.
2.0 showed copy and code defects on the RTX 3060 (IQ3_XXS); 1.5 showed none in 19 tasks and keeps the speed
(story +15%, code +37% vs off). The 61-task check decides 1.5 vs a 1.0 fallback.
+6.9% Qwen IQ3_S decode on Strix Halo Windows, prompts unchanged; token ids identical in 4 starts x 5 prompts.
Linux gfx1151 and other cards unchanged; STRATA_DEFAULTS=026 restores it. Docs label the DeepSeek Strix numbers by build.
The Windows guide says what 0.2.7 does and how to turn the switch back on, backed by five fixed greedy prompts
(895 ids) run four times with the switch on, off, on, off: identical ids. STRIX_HALO.md and the 0.2.7 release notes
get a line. The DeepSeek 4K prompt figures of 2026-10-25 (78.7, 87.2) stay and are labelled with their build; a table
of the 0.2.7 candidate changes one thing at a time (chunk size, DS4_DEFAULTS=026).
Interleaved A/B on a Strix Halo Windows PC (Qwen IQ3_S, engine 0.1.51): chat decode 43.6 tok/s with the fork,
46.6 without (+6.9%), prompts unchanged. Linux gfx1151 and every other architecture keep their table.
STRATA_SH_STREAM=1 turns it back on, STRATA_DEFAULTS=026 keeps v0.2.6's table.
arch_default_env(arch, defaults026, windows) is a pure rule; arch_defaults_test checks both platforms.
Ported PRs (authors kept) and our fixes, plus the owner's part-2 items: the restored TROUBLESHOOTING entries with a
link check, #3895 (no shared-expert fork calibration on Windows HIP with --kv-resident; a warning on a hand-set
STRATA_SH_STREAM=1), #3930 (docs), #3421 (Docker stops when models are present and MODEL_PATH is unset), #3570
(launch --show-key reads setup's config), #3311 (engine paragraph only) and #3029 (Strix Halo Windows guide,
re-measured on 0.2.7).
Structure from the guide in PR #3029 by HaiYunChuang (what the PC reaches, memory and the BIOS, the configuration, what is on by default, traps),
rewritten for the 0.2.7 install and with every number measured again: the Terminal and the gfx1151 engines built from f-027-int2 (3e468f5de), a
local signed test feed, strata launch as a standard test user on the NucBox EVO-X2 (Adrenalin 26.10.1), tools/strata_bench.py over a tunnel.
Qwen IQ3_S: 44 tok/s chat, prompts 755 / 821 / 792 tok/s at 4K / 16K / 32K. DeepSeek 2.00 bpw: 12 to 14 tok/s chat, prompts 214 to 342 tok/s.
An interleaved A/B of STRATA_SH_STREAM on Windows gfx1151 (three pairs): off is 6.9% faster in chat decode, prompts unchanged.
What launch chose and logged is quoted instead of a hand-made config. The first guide's tuning rows (engines 0.1.41 to 0.1.47) are not repeated.
Co-authored-by: HaiYunChuang <[email protected]>
Part 1 fixes and 12 ported PRs (authors kept), and the owner's part-2 items: the Monitor readings strip with
local currency (#3628), approval notification + opt-in minimise to tray (#3477), opt-in done sound (#3586),
app port setting with a busy warning (#3402) and interface size up to 200% (#3970). Host tests: only the two
known base failures on Linux; connector-setup and the new tests pass on Windows.
Problem: Gemm::native dequantizes each dense weight to FP16 in front of its
GEMM on the compute stream, so the dequant never overlaps the kernels queued
ahead of it.
Change, opt-in (CUDA only, always off on HIP): with STRATA_DEQUANT_SIDE=1 the
prompt path takes a second GEMM scratch (64 MiB more of its region, counted in
bytes_needed and carried through relayout) and Gemm::set_dequant_side hands it
to the Gemm. native() then dequantizes the next weight on a non-blocking side
stream into the scratch the compute stream read last but one. Four events order
every write and read of each scratch (free -> dequant -> ready -> GEMM -> free),
so each product sees the same bytes as in the serial form. The row-sliced path
(a weight larger than the scratch) stays serial. Metal answers false and keeps
the serial form.
Differs from the PR as submitted: if the side stream, its events or the
setup fail, the PR failed the prompt-path init. Here one line is printed and the
dense weights are dequantized in front of their GEMMs as before. Unset, nothing
changes (gemm_dequant_side() is one cached flag; the scratch, stream and events
are not created).
Test: new gemm_dequant_side_parity (ctest, GPU, synthetic, 77 without a device):
eight Q4_K / Q5_K / Q6_K weights of the engine's shapes (one larger than the
scratch, so it takes the sliced path) run three rounds through Gemm::native
serially and with the side stream, with and without a spin kernel queued ahead of
each GEMM; every output must be identical bit for bit.
Not covered: dense weights read by the side stream are the resident ones; a weight
that arrived by an upload on another stream would need its own event (the prompt
path has none that do).
Reporter's numbers (linxuhao, CMP 170HX): measured with STRATA_PLE_COPY_STREAM and
STRATA_EMBED_VRAM, the four PLE / embedding / dequant switches together give
+2.2..+2.6 % prompts. Theirs, not measured here.
Original change by linxuhao (PR #3779, commit 741fc6ab).
DS4_DEFAULTS=026 behind the DeepSeek speed-ups Off toggle, the 12 GB NVIDIA route-prior row, the Advanced-only
Qwen '0.2.6 behaviour' row (STRATA_DEFAULTS=026), and the app-ui Quit test fix. Terminal suites pass; host suite
failures on Linux also fail on the base.
Owner decision for 0.2.7: the tables of PR #3453 (the cells where an RTX 4070 and an RTX 4080 community run gave the same rows count, and
the one-column Q3_K row) ship on for sm_89 cards, untested on Ada by us. The output is bitwise the same with them off (a table only
changes how many rows a warp takes); no end-to-end gain was seen on a 4090 or 4080 in the report, so the reporter tests 0.2.7 and reports.
The rule is a pure function in sm120_series.hpp: mmvq_il_89_default(hip, cc, v026) is true for cc 89 only, false in HIP builds and under
STRATA_DEFAULTS=026 (defaults_026.hpp, as 38c9080db did for the other defaults); mmvq_il_89_resolve() adds the variable: "1" and "0" win,
unset or other text is the rule. native_mmvq.cu reads it once. arch_defaults_test checks cc 89 on, every other cc off, HIP off, 026 off,
"0" off, "1" wins over 026, the empty text. mmvq_il_parity expects the on-tables by default and the old ones with =0 or 026.
Docs: MMVQ_IL_TABLE.md, DETAILS.md and the 0.2.7 release notes (a line that says it is not measured on an RTX 40 card, with the off switch).
French catalogs for the app (3,528 strings + fix/tune/tools catalogs) and the served web page (247 keys),
README.pl.md from the current README (#3355, Jakub Luwierski), Polski in every README language row, and the
README.md language row's double-encoded names fixed. Tests: 21 host files 180 pass on p1006, Terminal and web
i18n checks on l620.
serve/web/locales/fr.json (247 keys, plurals one/other, same terms as the app catalog) and the French line in serve/web/i18n.js. tools/web_i18n.py and serve/test_web_i18n.py (node and python) pass.
Ports the Polish translation from #3355 and the Polski entry in the language row of every README. The pull request translated the older long README, so README.pl.md is rewritten from the current README.md (logo, centred language row, DeepSeek speed tables including Strix Halo, Gyro, coffee button). README.md's own language row had garbled names (double-encoded text); it is written out correctly.
Co-authored-by: Jakub Luwierski <[email protected]>
fr.json, fix-fr.json, tune-fr.json and tools-fr.json (3,654 strings, vous, plain short words), the Francais entry in the language picker, every locale test and gap check now covers fr, a french-catalog test (spacing before ? ! : ;, apostrophes, short labels, no leftover English), scripts/fr_sync.py to list and merge strings added later, and scripts/language-shots.mjs for screenshots of the app in one language.
Wide/narrow CSS (#3176), page translations with a language choice, opt-in KaTeX math (#2972, #2974),
sandboxed svg/html preview (#1628), Load/Unload in the header (#3550), and one chat-history design
(#1787, #2615, #3823, #4004). Tests: browser suite on the owner PC, node + pytest serve suites on boston
(457 passed, 2 skipped).
Problem: on a GP100 die the verify window's gate/up kernels for the IQ2_S,
IQ3_XXS and IQ3_S expert formats are not held by memory bandwidth but by each
block's serial chain (unpack the codebook indices, the emulated dp4a, the
16-lane reduction); the same kernel with every dot product removed still takes
about two thirds of its time.
Change: with STRATA_EXPERT_NIBBLE=1 the engine, after the expert cache is
filled and every expert is resident, transcodes the gate/up blocks of every
fully resident IQ2_S / IQ3_XXS / IQ3_S layer (n_embd 2560, n_ff 640) into slots
of 4-bit codes, 16 bytes per 32 weights, with the integer scales and fp16 d kept
as they are; the down rows stay GGUF. The decode gate/up reads those slots with
kernels that run the production integer chain and float tail in the production
order (R4S / R4P / R4SV / R4PX by format and group size); the prompt path's MMQ
gather rebuilds the GGUF blocks from a slot on the fly. At load every item is
round-tripped and every slot's rebuilt GGUF blocks are compared with the staged
blob before the cache is sealed (ExpertCache:🦭 a later fill_slot stops the
engine). New: include/strata/core/expert_nibble.hpp (slot contract, switches,
per-layer table), nib_gu_proto.cuh (STRATA_NIB_GU_PROTO=1 variant), the
transcoder, hash and rebuild kernels and the 4-bit gate/up kernels in
iq_kernels.cu, generate.cpp (the fill, the VRAM accounting, STRATA_EXPERT_NIBBLE_
RECHECK), iq_nibble_parity and native_expert_bench --nibble.
Review changes against the PR:
- An option the switch cannot work with (--peer-device, --kv-grow, --vram-elastic,
remote caches, STRATA_MMQ_DIRECT, ..., a card that is not cc 6.0, a build
without the kernels) now prints a warning and the start goes on with the usual
GGUF slots, where the PR refused the start. What is known only later (an expert
out of VRAM, a loan, the prompt chunk) still refuses, since slots are
converted by then.
- The checks this switch adds to expert_source.cpp and prefill.cpp are compiled
out of every build that has no 4-bit kernels, so the default builds add nothing
to the hit split or the prompt loop.
- iq_nibble_parity gets --synthetic-only (every gate/up format x both down
types on random experts, no model file), skips with 77 on a card that is not a
GP100, and the synthetic and --header runs are ctests.
- The PR's docs section is rewritten for these changes and its measurements are
labeled as the author's.
Default path (switch unset): the table is empty, nibble::slot_bytes(l) is
expert_layout().blob_bytes(l) and nibble_max_slot_bytes() is max_blob, every
sizing the PR touched returns what it returned before, and the new branches in
generate.cpp (guard_raw before the adaptive tier's copies, the profile fill's
test), expert_cache.cpp (one bool in the fill calls and admit) and verify.cpp
(a capture-time layout choice, compiled out without the kernels) are not taken.
Proof on the kernels: in the sm_60 build (CUDA 12.9) the SASS of all 446
kernels iq_kernels.cu had before is identical with and without this change,
as is that of the 603 in native_mmvq.cu, 29 in qsa_select.cu and 10 in
moe_mmq.cu; 21 kernels are new.
Tests: iq_nibble_parity --header passes on z440 and boston (the host half of the
contract; also a ctest). The GPU parts (--synthetic-only, a model shard) refuse a
card that is not a GP100, so they were only compiled here (sm_60, CUDA 12.9,
P1004: iq_nibble_parity and native_expert_bench build; the whole tree builds).
sm_86 build without the SM60 option (z440): exit 0, the 4-bit paths compiled out.
Not covered: no GP100 run. The model gate is
phaseA-tests/gh027/gates/3455-expert-nibble.sh: iq_nibble_parity --synthetic-only
and on a model's layers on a P100, then identical replies with the switch on and
off (a layer is converted only when every expert of every layer is resident, so
the rig needs the pack to fit whole), then the decode A/B. The contributor's
numbers (four GP100 dies, Swift 1.5 IQ3_S, 47 of 48 layers converted, +12.7 GiB
VRAM, decode 47.3 -> 51.9 tok/s, +9.7%; prompt reading 2.3-3.0% slower; replies
of 15 requests equal) were not re-measured, and the enabled path was written
against an older base than this branch's.
Problem: on a GP100 die (compute capability 6.0: no dp4a, no tensor cores) the
prompt pass's routed experts run as MMQ launches per 16-expert group: a
gate/up, a swiglu, a quantize and a down launch per group, each group's experts
gathered and padded to the group's largest row count, every MMQ tile decoding
its weight blocks again. With every expert resident the per-group overhead, not
the weight bytes, is a large share of a prompt read on this card.
Change: with STRATA_MMQ_WS=1, a layer whose routed experts are all resident on
this GPU runs quantize_scatter, then gate/up as ONE weight-stationary launch
over all its experts (read where they lie in the expert cache), ONE
swiglu_quant over all its rows, then the down as ONE launch. New
src/prefill/moe_ws*.cu(h): a CTA holds 128 (or 256) weight rows of one expert
and runs them against up to 64 of that expert's activation rows, a thread per
weight row, so each weight block is decoded once per pass; gate/up for
IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS and IQ3_S, down for Q2_0 and IQ4_NL. Every
product keeps MMQ's int8 values, scales and per-32-value epilogue in its K
order, so GU, H and the down rows are MMQ's bits. ws::faster carries the
author's measured ws / MMQ time ratios per format and picks a layer only where
both products win at its rows per routed expert. prefill.cpp decides per layer
after quantize_scatter (all experts resident, this GPU alone, no peer or CPU
share, not the fused layout, not STRATA_MMQ_DIRECT, no Hadamard, no NVFP4 tail,
no FP4 activations, aligned, not STRATA_DBG_NAN); every other layer keeps its
path. STRATA_WS_FAST=1 (second opt-in, same bits) selects a down kernel for the
IQ4_NL 32-token tier with fewer instructions per pass.
Gating: the kernels and STRATA_PREFILL_WS exist only in the CUDA build with
STRATA_EXPERIMENTAL_SM60 (not HIP, not Metal; prefill.cpp links stubs that
answer mode 0 elsewhere). At run time ws::mode() is 1 only with STRATA_MMQ_WS=1
and every visible GPU at compute capability 6.0. Unset, nothing is allocated
(the offset and chunk buffers, under 1 MiB per die, exist only when the mode is
not 0), the arena layout and the mapped table are the ones before, and the
per-layer test short-circuits on the cached mode.
Not ported: moe_ws_down_bench (needs a model shard) and its CMake target.
Added: ws::set_mode (tests only) and tests/cuda/prefill_ws_parity_test.cpp, a
synthetic layer-level check: random experts of all 5 gate/up x 2 down formats,
random routing, the per-group MMQ (mmq::Context per expert) against ws::gate_up
+ swiglu_quant + ws::down, GU, Hq and Dm compared byte for byte, the down also
alone on the reference H and with STRATA_WS_FAST on. It gives its verdict on a
compute capability 6.0 card only (the kernels follow MMQ's sm_60 SASS); other
cards run it for information with --any.
Tests: prefill_ws_parity_test on boston (2x RTX A6000, sm_86, CUDA 13.1, gpu lock,
SM60 build, --any because it is not a GP100): 40 cases (all 5 gate/up x 2 down
formats, four routings each), 0 declined; the largest GU and down-alone error
against that card's MMQ is 5.2e-07 of the largest value (float rounding: MMQ's
sm_86 epilogue is not the sm_60 one the kernels follow), the full chain's Dm
differs up to 1e-3 where int8 rounding of H flips on those rounding
differences, and STRATA_WS_FAST output equals the default kernel's in all
cases, 0 differing bytes. The byte-for-byte verdict against MMQ is the P100's.
sm_60 compile of the whole tree (CUDA 12.9, P1004): exit 0; sm_86 build without
the SM60 option (z440): exit 0. SASS of the 10 kernels moe_mmq.cu had before
this change is identical in the sm_60 build.
Not covered: no GP100 run here. The contributor's numbers (prompt reads 31-37%
shorter on four GP100 dies with every expert resident: 240 tokens 2,156 -> 1,364
ms, a fresh 26K read 41,231 -> 28,555 ms; one run per arm) were not re-measured.
The model gate is phaseA-tests/gh027/gates/3752-mmq-ws.sh.
Problem: the token-embedding table of a native pack lives in mapped pinned host
memory, so every prompt's scattered row gather crosses the PCIe link. On a Gen2
x4 link that costs more than the table's size in VRAM is worth.
Change, opt-in: with STRATA_EMBED_VRAM=1 the table is copied into the current
device's VRAM when the table plus 512 MiB stay free (322 MiB for an IQ4_XS
table). The gather kernels read the same bytes through dev_ as before. If the
VRAM is not there or the copy fails, the table stays in mapped host memory as
before and one line says so. The start-up line now says where the table is
("in VRAM" also for the two existing VRAM fallbacks, which it called mapped
host memory).
With the variable unset nothing changes (one getenv at load).
Not covered: a layer split across several devices places the table on the
device that is current when it loads, as the existing VRAM fallback does. No
unit test (it moves the same bytes to a different memory; the gather kernel is
unchanged).
Reporter's numbers (linxuhao, CMP 170HX): measured together with
STRATA_PLE_COPY_STREAM and STRATA_DEQUANT_SIDE on a 51.8K-token prompt, the
embedding step went from 191-201 to 70 ms. Theirs, not measured here.
Original change by linxuhao (PR #3779, commit 952dd784).
Problem: with STRATA_PLE_COMPACT the raw-row upload sat on the compute stream
before layer 1, which idles for the DMA (4.7 MB per 3K-token chunk, 2.9 ms on a
Gen2 x4 link).
Change, opt-in, only with STRATA_PLE_COMPACT=1: the raw rows go up on a
non-blocking stream of their own once the previous chunk's dequant has read the
device buffer (ple_raw_free event); the compute stream waits for the upload
(ple_copied) only at the dequant. Same bytes, same kernel. The stream is drained
when a run ends so no upload is left in flight from a buffer the next run
refills.
Differs from the PR as submitted: if the stream or the event cannot be created,
the PR failed Prefill::init. Here it prints one line and the rows go up on the
compute stream as with STRATA_PLE_COMPACT alone. With the variable unset
nothing changes (the stream and the events stay null).
Reporter's numbers (linxuhao, CMP 170HX): PLE time of a 51.8K-token prompt 95-103
to 47 ms, measured together with STRATA_EMBED_VRAM and STRATA_DEQUANT_SIDE
(+2.2..+2.6 % prompts for the four PLE/embedding/dequant switches together).
Theirs, not measured here.
Original change by linxuhao (PR #3779, commit 0ba47e61).
Problem: the prompt path reads 16 PLE rows a token, dequantizes them on the
host and uploads 10,240 floats a token. On a slow host link (PCIe Gen2 x4) that
upload is on the critical path of the first layer.
Change, opt-in with STRATA_PLE_COMPACT=1: for an IQ4_NL table held in memory
(--ple-io ram / mmap) the prompt path copies the raw 90-byte rows (1,440 bytes
a token) into the same pinned buffer, uploads them and runs
ple_iq4nl_dequant on the device (d times the code's value, one multiply, the
split-half nibble order of iq4nl_dequant_row). Any other table (Direct I/O,
pageable fallback, another format) and a failed allocation take the host path
as before. With the variable unset nothing changes: the new code is behind one
cached flag, and the extra members of the Prefill state stay null.
Two things differ from the PR as submitted:
- PleTable::gather_raw_iq4nl writes a row past the table as scale 0 and code 8
in every nibble. The PR wrote zero bytes, which the device decodes to
-0.0f (0 times code 0's value -127), while the host path writes +0.0f for such
a row.
- New test ple_iq4nl_dequant_test (ctest, synthetic, no model, 77 without a
device): the device decode against iq4nl_dequant_row, bit for bit, over every
fp16 scale pattern with random codes, random rows, an odd row count and the
out-of-table row.
Reporter's numbers (linxuhao, CMP 170HX, PCIe Gen2 x4): prompts +1.0..+1.7 %
(4K, 16K) with the switch on. Theirs, not measured here. On a Gen4 x16 card
the gain will be smaller.
Metal gets a stub (the Metal prompt runs through its window and never calls
it).
Original change by linxuhao (PR #3779, commit da818261).
Problem: the prompt path stores H (swiglu of the gate/up rows) as floats and
then quantizes it to the q8_1 rows the down product reads: two kernels and an
H round trip per MMQ group.
Change: swiglu_quant_kernel, a block per row (128 threads, 4 values each,
blockIdx.y the row's 512-value half), computes swiglu_kernel's expression for
each value and quantize_mmq_q8_1<D4>'s result for each 32-value block, so hq
holds the same bytes and H is never stored. swiglu_quant_ok(type) says where
it applies (a q8_1 D4 down type, no FP4 activations). Built only in the CUDA
build with STRATA_EXPERIMENTAL_SM60; HIP and the other builds get
swiglu_quant_ok() == false and a no-op. Nothing calls it in this commit; the
weight-stationary path of the next commit does. prefill.cpp gets the matching
stubs for builds without the MMQ sources.
Tests: prefill_swiglu_quant_test (new, registered under STRATA_BUILD_TESTS in
the SM60 build, skips with 77 without a device) compares the fused kernel with
swiglu + quantize byte for byte for Q2_0, IQ4_NL and Q8_0 down types, split and
interleaved gate/up rows, a zero row and a row with one huge value.
Run on z440 (RTX A4000, sm_86, CUDA 13.0, SM60 build, gpu lock): all 18 cases print
"0 of N bytes differ" (1, 37 and 4113 rows) and the test ends "prefill_swiglu_quant_test:
ok". sm_60 compile of the tree (CUDA 12.9, P1004): exit 0.
This is a partial port of another contributor's prompt-path fusion change;
only the kernel is taken, as in PR 3752.
Problem: on a GA100 card (70 SMs) the 512-thread fused prompt-expert kernels
leave the card idle at every barrier and at every grid tail.
Change, all opt-in:
- STRATA_PF_FUSED_NT=256 sets up a second set of native_kernel instances with
256 threads and two blocks a multiprocessor (WW 2 only), with two-pass weight
loads (the loads of the next k step are issued before the current step's
decode; load_unit_l gives the same words as load_unit). Every output is the
same thread's sum in the same order.
- STRATA_PF_BATCH=N launches up to N experts at once (WideBatch holds 512 blob
pointers, 4 KB of kernel parameters: CUDA 12.1+, Volta and newer; HIP, gfx906
and the Pascal build keep kMaxBatch). Above 128 it needs the NT = 256 kernels.
Below 128 it only changes the launch grouping.
- With neither variable set the 512-thread kernels, their launches and their
load sequence are unchanged: tile_rows(ww, 512) equals the old 256 / ww, the
two-pass code is behind a constant-false TWO_PASS, and the one added test in
experts_native is occ_nt > 0, which only STRATA_PF_FUSED_NT=256 sets.
- If the 256-thread instances cannot be set up (shared memory, occupancy) the
engine prints one line and keeps the 512-thread kernels.
Cost: the NT = 256 instances are new device code in every CUDA build (the
reporter measured 365,632 bytes for sm_80).
Tests: new ctest entry prefill_fused_iq_test_nt256 runs prefill_fused_iq_test
(the fused kernels against the double-precision reference and the MMQ path)
with STRATA_PF_FUSED_NT=256; it exits 77 below sm_80.
Reporter's numbers (linxuhao, CMP 170HX): one MoE layer of 3,256 tokens
8.27 ms with NT = 256; prompts +7.6 % (4K), +4.9 % (16K), +4.3 % (52K) for
NT=256 with BATCH=512 on the PR's own build. Greedy answers identical on their
11 prompts. These are theirs.
Original change by linxuhao (PR #3779, commit 056d36a9). Merged onto the Q2_0
fused path of #3809 (STRATA_PF_FUSED_Q2N=1) without textual conflicts; the PR
predates it, so one line is mine: the NT = 256 setup also sets up the Q2_0
gate/up instance when Q2N is on (without it that instance would have launched
without its shared-memory attribute).
moe_shared_init and moe_scatter_add came with the layer-major walk and had no test.
lm_kernels_parity runs them on synthetic data of the model's width (N = 2560, no
model, a few MiB) against a host reference:
- moe_shared_init: acc = shared * sigmoid(sg), within 1e-5 relative (the device
sigmoid uses __expf);
- moe_scatter_add: the per-token fma chain in row order, bitwise equal to the host
chain and bitwise equal between two launches (no atomics), over groups of one
row, one sub-chunk, several sub-chunks, one row per token and eight per token.
Registered in CMakeLists.txt next to gr_upmix_parity (CUDA builds, not gfx906),
SKIP_RETURN_CODE 77 when no device is visible.
Not covered: the walk itself needs a model (see the 3976 gate scripts).
Problem: an sm_80 card took the shared interleaved-MMVQ row tables, which
were measured on other cards (#3252 reports A100 slowdowns).
Change: kIlRows_80 (2-4 columns), kIlRows58_80 (5-8 columns) and kIl1Rows_80
(one column), looked up under the key 80 only. They were read off
mmvq_il_parity --bench --emit-table with the same 3 % rule as the other tables.
STRATA_MMVQ_IL_SM80=0 maps an sm_80 card to the key 81, which has no table, so it
gets the shared table again (and no one-column table, as before).
A table only picks the rows a warp (0, 1, 2 or 4); 0 means native_mmvq's kernels.
mmvq_il_parity already checks every rows choice against native_mmvq bitwise for
every type and shape, T = 1..8, so any entry of the new tables is covered.
Reporter's numbers (linxuhao, one CMP 170HX): decode +2.7..+3.2 % over the shared
table, Qwen3.8-Flash-Next IQ3_S. The measured card is not an A100; the rows may
want re-measuring there.
Original change by linxuhao (PR #3779, commit eaedca84); the commit message here
is rewritten. The sm_89 table of #3453 is a separate change in the same list.
Problem: sm_80 (GA100: A100, A30, CMP 170HX) ran the per-group dequant kernels
that sm_61 and sm_86 had already left for the wide ones, because
dequant_wide_on() had no sm_80 entry.
Change: dequant_wide_on() answers yes for compute capability 8.0 as well. The
test is the raw (major, minor) of the device, so it is false on every other
card (sm_60, 70, 75, 86 keep their answer, sm_89, sm_90 and sm_120 stay off as
before). STRATA_DEQUANT_WIDE=0 still turns the wide kernels off.
Reporter's numbers (linxuhao, one CMP 170HX, GA100 sm_80 with 70 SMs): the GPU
time of a 3,259-token prompt went from 857 to 792 ms, prompts +6.3..+7.2 % at
3.3K tokens and +3.9 % at 52K. We have no sm_80 card; these are theirs.
Bit identity: the wide kernels are the ones sm_61 and sm_86 already run by
default. dequant_wide_parity compares them with the per-group kernels byte for
byte for FP16, BF16, FP32, the padded stride and a row slice. The kernels have
no sm_80-only code, so the result does not depend on the card; what only an sm_80
card can show is that ptxas for sm_80 keeps the same arithmetic (see the
checklist in phaseA-tests/gh027/gates/3779-sm80-checklist.md).
Original change by linxuhao (PR #3779, commit 48dc535e); the commit message
here is rewritten.
Problem: a request declares tools; the model writes a complete call for a name that is not among them (bash for shell, a dropped
prefix, an MCP name with every "__" written as "_"). Since #3075 such a call stays text, which is the rule we keep, but the client
saw an ordinary answer with finish_reason stop and nothing else, so the agent turn ended with raw XML and the model never learned
of its mistake. Reporters on OpenCode, Pithagoras, OpenClaw, LibreChat, Open WebUI and Pi hit it.
What changed (declared-only stays: nothing undeclared is ever returned as a call, finish_reason is unchanged):
- serve/frontend.py OutputParser: a declared name that contains "__" is also read when written with every "__" as "_"
(mcp_playwright_browser_navigate for mcp__playwright__browser_navigate); the call goes to the client under the declared name.
Same rules as the other aliases: only when unambiguous, and a tool declared under that spelling wins. A reordered or renamed
tool (server1_label_values for label_values_mcp_server1, bash for shell) is not guessed.
- serve/server.py: when a block went back as text for an undeclared name, the reply carries
"strata_rejected_tool_calls": [{"name": ..., "reason": "undeclared"}] (the names the parser already collected, up to five): at the
top level of a Chat Completions reply and in its last streamed chunk, and in the message_delta event and the message of
/v1/messages. It is absent when nothing was rejected, so every other reply is unchanged. The text itself is untouched.
- The server window line says the response carries the names; the count goes to totals["tool_calls_unread"] and to
/metrics/prometheus as strata:tool_calls_unread_total.
- docs/DETAILS.md: the alias, the marker, the counter.
Tests: serve/test_tool_name_alias.py (collapsed spelling at several chunk widths, a declared spelling and a plain tool unchanged,
a tool declared in the collapsed spelling wins, two tools with one collapsed spelling are not guessed, reordered/renamed names stay
text and are named), serve/test_server.py ForcedToolChoice (the field in Chat Completions whole and streamed and in /v1/messages,
absent for an ordinary answer and a declared call, the collapsed name is a call; the counter). The whole serve/ suite: 1055 tests,
two failures that also fail on the base tree (test_fatal_recovery on this PC, RestartWindow.test_not_alive_until_ready under load).
Not covered: returning an undeclared call as a real tool call (the owner keeps declared-only); a block for a DECLARED name that came
back as text because it was malformed (counted, no name to list); the /v1/messages field with a real client; a stderr or log-file copy
of the warning.
Problem: a decode window's QSA select runs one CTA per query for the top-k
(block_topk_wide_kernel past ~135K cells) and the warp-shuffle block scorer
(block_scores_multi_kernel), in each QSA layer. Both grow linearly with the
context: on one GP100 die at 300K cells a top-k call takes 0.39 ms with 1-5 SMs
busy and a scores call 0.22 ms (two queries). At short contexts the same work is
small and an extra kernel is pure overhead.
Change: qsa_select.cu (inside !__HIPCC__ && STRATA_EXPERIMENTAL_SM60) gets
block_topk_multi_* (six kernels over several CTAs per query, one per 2,048
blocks of capacity, at most 32, exchanging through caller scratch: no clusters,
no spin waits, capturable) and block_scores_dx_kernel (the multi scorer's
per-head sums built in registers as the same 32-leaf xor-butterfly tree, no
__shfl_xor_sync, the tail block stays block_scores_tail_kernel's). New public
qsa_topk_multi_scratch_ints, qsa_block_topk_long and qsa_block_scores_long
forward to qsa_block_topk / qsa_block_scores outside the build gate or their
conditions. verify.cpp keeps a second set of window graphs (exec_[sel_long_][T])
for windows that reach STRATA_SELECT_LONG_AT cells (default 65536); the choice
is made before a capture and handed to the next stage of an overlapped layer
split, capture_all captures both, pl_launch picks by the window's position.
The ids and the score bits are the replaced kernels'. STRATA_TOPK_MULTI=0 and
STRATA_SCORES_DX=0 turn either half off inside the long graphs.
Review changes against the PR: in a build without the multi-CTA kernels
STRATA_SELECT_LONG=1 now says so and is ignored (the PR would have kept two
sets of graphs for nothing there); Metal gets stubs for the three new calls
(the SYCL tree has its own verify.cpp and qsa_select copy).
Default path: unset, select_long_on() is a cached false, topk_scratch_ is not
allocated, sel_long_ stays 0, and every window uses exec_[0][T], the graph it
used before; the index change is the only difference in verify.cpp.
Merge note for f-027-gh-eng (#3702 evict_window_graph, not in this branch's
base): its exec_ / exec_nr_ / launched_ accesses need the same [sel_long_]
first index.
Tests: src/kernels/qsa_select_long_parity.cpp (new; registered with
STRATA_SCORES_DX=1 STRATA_TOPK_MULTI=1 in the SM60 test block, and a second
case with STRATA_TOPK_MULTI_N=3; skips with 77 without the kernels) compares
qsa_block_scores_long with qsa_block_scores (nq 1-8, n_bid 8 .. 131072, tail
blocks of 0-3 cells, every block 0..n_bid memcmp) and qsa_block_topk_long with
qsa_block_topk (nq 1-16, contexts 1,000 .. 524,288, random scores, all-equal
scores and 7 distinct values for ties across CTAs, selected ids of every query;
the multi scratch is junk-filled and must have been written). z440 (RTX A4000,
sm_86, CUDA 13.0, gpu lock, build with -DSTRATA_EXPERIMENTAL_SM60=ON): "172
cases, 0 differ -> PASS" with one CTA per 2,048 blocks and with N=3. sm_60
compile of the whole tree (CUDA 12.9, P1004): exit 0.
Not covered: no GP100 run here (the kernels' last word is the P100, and the
speed claim). The contributor's numbers (four GP100 dies, every expert resident,
--pipeline-windows 3: 400 tokens at 115K context 51.0 -> 52.5 tok/s, +2.9%;
prompts below 64K unchanged; replies and state hashes equal on 15 requests) were
not re-measured. The model gate is phaseA-tests/gh027/gates/3820-select-long.sh.
Everything the Chat view shows (header, tabs, message box, settings, toasts,
tool blocks, the sidebar, the history and Load/Unload texts) now goes through the
translation catalog: serve/web/locales/{de,es,ja,ko,pt,ru,zh}.json beside en.json
(written from the English text in the code by tools/web_i18n.py). Plurals follow
the language (Russian has few and many, Japanese and Korean only one form).
serve/test_web_i18n.py fails when a catalog lacks a key, has a stale one or
changes a {placeholder}. A ninth language is one file plus one line in
serve/web/i18n.js. The Monitor, Images and About views are still English (the
language setting says so); they are the next step.
Browser tests: a language pick translates and is kept, /?lang= is for one visit,
an unknown code stays English, all seven catalogs load without an error.
docs/DETAILS.md: a section "The web page" (every switch, what the preview can and
cannot reach, the languages, the tests), the Sampling drawer's new name, and the
credits (KaTeX and the contributors of the PRs ported into the page).
Five overlapping proposals, one set of controls (Chat settings > Chats, all off
by default; off, the page is the page from before: one chat, New chat clears it
with an Undo, Send is off while an answer is written, no new request field).
- Saved chats list (#1787): the chats kept in this browser (IndexedDB, one record
per chat, names of attachments only), a list with day groups, search over
titles and text, rename, delete with Undo, collapse; a private window or full
store keeps the list in memory and says so once. Nothing goes to the server.
The list has the New chat button, so the composer's one is hidden while it is on.
- Edit and rewind messages (#2615 + #3823): under each message, Regenerate (a new
answer to the same question), Edit answer (its text only: thinking and tool
calls go, no request) and Rewind to here. Rewind asks, as #3823 proposes: "Edit
and send again here" hides this message and the ones after it, puts the
question and its attachments in the box and keeps one Undo bar ("2 messages
hidden"); "Branch from here" starts a new saved chat with the messages before
it and leaves this one alone. A branch keeps a title that says where it came
from ("Branch of ...", also at the very first message) and links back. One
control does what #2615 had as Edit on a question and as Remove from here.
- Send while answering (#4004, for the web page): "Wait" (default, as before),
"Send after" (the message waits and goes when the answer is done) or
"Interrupt" (the answer stops where it is, what is written stays and is marked
interrupted, the message goes at once; a running tool finishes first). One
message waits; Take back returns it to the box; Stop keeps it for you to send.
The pure logic is serve/web/history.js, the store serve/web/chatstore.js, the
page's part serve/web/chats-ui.js. Tests: serve/test_web_history.js (Node, with
the IndexedDB stub and cases of #1787), serve/test_web_history.py, and 17 browser
tests (tools/webtest/browser_test.mjs). Screenshots: tools/webtest/shots.mjs.
Also: the Sampling drawer is now Chat settings (its button says so), a wide
window lets a message use the column (#3176 widened only the column).
Ported by hand and merged: #1787 by Scotty Hudson, #2615 by Joyer Huang (uppet),
issues #3823 and #4004 by xvcezrqpbxtpzblrt (their proposals, for the desktop app,
carried to the web page).
Co-authored-by: Scotty Hudson <[email protected]>
Co-authored-by: Joyer Huang <[email protected]>
Co-authored-by: xvcezrqpbxtpzblrt <[email protected]>
The layer-major walk is a copy of run_impl's attention half with the experts moved
out. It does not carry four things run_impl does, and when one of them is asked for
the walk now declines (the prompt is read chunk-major, and the one-time "layer-major
off for this prompt" line names the reason):
- the DFlash2 drafter's prompt taps (tap_layers_): the walk would leave the taps
stale and the drafter would read old rows;
- STRATA_KVA (approximate late layers): the walk would silently read exactly;
- a stop_at_chunk callback (the pipelined batch reader that yields at chunk
boundaries): the walk reads every token, and stopped_at() must stay -1;
- a pending force_chunk (a --prefill-pipe plan): it is left set for run_impl.
The walk also resets stopped_at_ like run_impl does at the start of a run.
With STRATA_PREFILL_LAYER_MAJOR unset none of this is reached.
Not covered: no model run in this commit.
Only NVIDIA CUDA has run the layer-major walk. On a HIP build (wave32 or the
gfx906 compat layer) lm_prepare now declines with a reason, so a prompt takes the
chunk-major path and the one-time "layer-major off for this prompt" line names it.
The code still compiles everywhere the shared prefill.cpp does; nothing else
changes. With STRATA_PREFILL_LAYER_MAJOR unset nothing is reached.
Not covered: HIP and SYCL were not built or run (SYCL has its own prefill.cpp).
run_impl never clears a later stage's hand_in_, so after one short prompt (read
chunk-major) every later prompt was declined with "a layer split's chunk hand-off
is in use". lm_prepare_chain now declines only when a hand-off is really in flight
(a stage's next_run_ is valid) and otherwise clears the stale hand-in pointers.
A prompt longer than the common super-chunk is split into the same number of
super-chunks of near-equal size (32K at a 28K fit: 2 x 16K, not 28K + 4K). That
keeps a stage's residual rows in VRAM and lets a split's stages overlap on every
super-chunk. STRATA_PREFILL_LM_EVEN=0 gives the old largest-first split. It only
acts when STRATA_PREFILL_LAYER_MAJOR is set.
The split was declined whenever the model has PLE layers. The PLE block (layer 1)
reads the token history, which only the first stage has, and a later stage never
reaches it. So the split is declined only when the first stage does not hold both
layer 0 and layer 1.
Problem: layer-major prefill declined every layer split, so a split model kept
re-reading each layer's streamed experts once per 8192-token chunk.
Changed:
- lm_prepare_chain sizes every stage with the same super-chunk (the smallest any
stage fits, up to four rounds until they agree) and gives them back together;
- a stage runs only its layers [LB, LE); a later stage's rows arrive in its pinned
R_host;
- a stage before the last writes its final rows into the next stage's R_host on
the copy stream, then runs the next stage on a thread (std::async) while it reads
its own next super-chunk. There is one hand-off buffer: a stage's final pass waits
until the next stage has finished the previous super-chunk;
- layer_major_mid() asks the last stage, the one that reports the rows;
- run() drains the pipeline after the walk and clears the mid flag on every stage.
Not covered: only two stages were tried by the reporter (RTX 3060 + RTX 5060 Ti);
the peer-GPU tier and the stage helper have no layer-major form and decline.
run_layer_major called mmq::iota on ids_identity, which is null when no layer is on
MMQ; it now skips that. The QSA shapes come from core::qsa_shapes(g) as run_impl
does, instead of being filled in by hand.
lm_prepare required ids_identity and tok_dev even when no layer is on MMQ. Only a
model with an MMQ layer needs the identity table, so the check now asks for it only
then. Without this a model whose layers all take the FP16 path was declined.
Problem: lm_prepare declined every model whose experts are not all on MMQ. All 48
layers of UD-Q4_K_XL take run_impl's FP16 path, so layer-major never ran on them.
Changed: a layer outside the MMQ plan dequantizes each routed expert to FP16 once
per super-chunk and runs two FP16 GEMMs per entry on that entry's FP16 input rows
(a new per-super-chunk buffer mixed_h, the bits run_impl's gather reads), then
adds the group's rows per token with moe_scatter_add as the MMQ path does. The
buffers each path needs are checked separately, and a missing one gives a decline
reason that names it. The summary line counts the groups on the FP16 path.
Not covered: no model run in this commit.
Problem: the layer-major walk (#1670) was written for an older base. It read the
hyper-connection matrices as bf16 only, accepted expert formats its MMQ pass does
not carry, and called the QSA indexer with the older argument lists.
Changed (Norhiro's port of vizdatom's walk, kept as one step):
- the HC projections are dequantized per half when STRATA_HC_Q8 holds them as Q8_0
with no bf16 copy, and released after the GEMMs that read them are enqueued;
- lm_prepare declines (the prompt then runs chunk-major) for a layer that needs the
WMMA tiles, an APR (Hadamard-folded) layer, and an NVFP4 layer;
- STRATA_PREFILL_DIRECT is not used by the walk.
Not covered: no model run in this commit.
A super-chunk of S tokens is read layer by layer: each 512-token sub-chunk runs the attention half,
the FFN half hyper-connection read, the router and the shared expert in order (GDN / QSA / PLE state
advance as in the default path); then the layer experts run once for the whole super-chunk - each
routed expert staged once, its rows of every sub-chunk in one MMQ group, the results added per token.
Expert traffic drops by about S / 512 (a 12.9K prompt: ~509 GB -> ~79 GB on ryzen).
- STRATA_PREFILL_LM_TOKENS: the super-chunk target (default 8192), shrunk to fit free VRAM minus
STRATA_PREFILL_LM_MARGIN_MIB (default 96).
- STRATA_PREFILL_LM_R=vram|host|auto: the residual rows of the super-chunk in VRAM, or in pinned RAM
with the copies overlapped on the copy stream (auto: VRAM only when the whole target fits there).
- STRATA_LM_PREFETCH=0: no prefetch (default on: every streamed expert of a layer goes to the stager
when its attention halves start; the ones the routing skips are skipped if not started).
- Not for a layer split, a peer GPU, image prompts, the fused layout or a prompt under two chunks.
- on_chunk still reports every sub-chunk in order; layer_major_mid() tells serve not to checkpoint
before a super-chunk end (the state is already at that end).
Port to 0.1.43.1: the copied run_impl body follows the new base. Every blob the stager or the copy stream reads later
comes from blob_stable() (blob() may hand out a recycled STRATA_IO_PREFETCH buffer); the token ids go up through tok_up,
the QSA step records and the PLE rows' pageable fallback through up_ring (#1712, #1996); the PLE gather threads pin to
--aux-cpus; bf16x2 takes m.f16_io; the K8V4 append is upstream's, which writes the host copy and the staging pool by
halves. The indexer passes the INT8 pooled keys as the original does, in the QSA port's order (pooled_i8 and
pooled_scale after block_pos, the int8 qsa_block_scores(_tc) overloads); STRATA_IDX_INT8 stays opt-in and unsupported on
the prompt path, as in the original. stage_one calls the stager's pass() before wait(): with batched claims a routed job
after about a ring's depth of skipped prefetch jobs waited forever. lm_prepare reads the fused layout from m.fused_bufs
and declines while STRATA_DBG_FEAT, STRATA_DBG_ROUTE_OVERRIDE or STRATA_PREFILL_DUMP_R_ALL is set. serve's near-tail
checkpoint (--prompt-cache-tail) is not taken mid-super-chunk, so under layer-major it is taken only when the last
super-chunk is a single sub-chunk.
From vizdatom's #1670, original commit .
moe_shared_init writes the shared expert part of moe_combine before the routed experts are known;
moe_scatter_add adds one expert group rows into a per-token accumulator, per token in a fixed order
(deterministic, no atomics).
From vizdatom's #1670, original commit f0173382d0646907917edf99e59617c1412d898e.
Chunks below stream_all_min() stage each layer's unpinned experts on the stager's few threads, and with a mapped
file tier each of those is a chain of 128 KiB page faults, a shallow queue for the drive. With STRATA_PREFILL_WARM=1
(and a source that warms: the GGUF in place, or experts.bin with STRATA_BIN_WARM=1) a background thread hands the
source the layer's whole list at once before the stager starts, and the next layer's list from the previous chunk
(less what is on the GPU or pinned) while this layer computes. It only warms pages: the stager copies the same
bytes. At most four lists queued; a thread that falls behind drops the oldest. Off by default: the A/B arm.
Port to 0.1.43.1: the warmer thread pins itself with aux_cpus::pin_current_thread() first, like the other helper threads since --aux-cpus (it would otherwise inherit the host thread's CPU).
From vizdatom's #1670, original commit cabe33c4f585d4c9dd0af1fc9f014d00e2920042.
issued is a watermark, so a job nobody waits for could be passed while its thread was still writing
the pinned buffer the next job on that buffer was filling - a corrupt expert blob. A job now also
waits until the job kRing before it has marked itself ready. skip(j) lets a job the routing turned
out not to need be marked ready without reading it (the layer-major prefetch). The default walk
consumes every job in order, so neither changes it.
Port to 0.1.43.1: the stager now claims batched runs and sleeps in issued.wait, so the ready[i - kRing] wait
sits in the run's per-job wait loop after the issued wait, and a skipped job is left out of the run's gather
(no memcpy, prefill_copy or copy_blobs entry); its ready flag and `active` stay with the run's single store and
decrement. New pass(j): skipped jobs are never issued, so the layer-major walk moves the watermark to the
routed job it is about to wait for (store plus notify_all, which wakes sleeping threads); without it a routed
job after about a ring's depth of skipped ones waited forever. The default walks never call it.
From vizdatom's #1670, original commit 149b899548fbc98977219e6fd4ee7866384e61f6.
One button in the header of every tab: Load while the model is unloaded, Unload
otherwise. It calls POST /load and POST /unload (the server already had both,
for its own page only). Unload asks first and never interrupts a turn: the
button is off while an answer is being written, an image is made or a request is
queued. The pill and the Monitor say Unloaded / Loading / Unloading, a message or
an image on an unloaded model shows Loading, and a poll that started before the
POST cannot flip the button back (the POST's state is held until a later poll
agrees). On a phone the pill's text is the first thing the header row gives up
(it measures the row, not a fixed width), so Load/Unload and the theme button
stay on the screen from 320 px.
Browser tests (tools/webtest/browser_test.mjs): unload with its question, load
back, declining, not while answering, a message that loads, the header at 320 to
640 px.
Ported by hand from PR #3550 by Sakurai Momoka (sakuumomo); the texts now go
through the page's translation catalog.
Co-authored-by: Sakurai Momoka <[email protected]>
A complete ```svg or ```html block in an answer gets a Preview button. The
block's own code is drawn in an iframe (srcdoc) that is cut off from the page:
sandbox="" (an opaque origin: no access to the page's DOM, storage, cookies or
server, no scripts, forms, popups, downloads or navigation of the page), a CSP
in the frame's document (default-src 'none': no network, fonts, frames or media;
styles inline, pictures as data: only; no <base>, no form target), no referrer
and no permissions. A well-formed svg is shown as an <img> with a data: URL: an
image runs no script, loads nothing and animates in every browser. The frame
takes the page's light or dark colours and is built again when the theme
changes. A page (html) can also be run WITH scripts, one block at a time, after
a warning that is shown every time: sandbox="allow-scripts" only (still an
opaque origin), the CSP allows the block's inline script and no address; closing
the preview ends it.
What the sandbox flags alone do not stop is a link or <meta refresh> inside the
model's html loading a web page into the frame. The browser test shows it does
(tools/webtest/browser_test.mjs, with and without the header). So the page is
served with Content-Security-Policy: frame-src 'none' (nothing else: a srcdoc
frame inherits the page's policy, and the script preview needs its inline
script), and a <meta http-equiv=refresh> in a previewed page is dropped.
tools/webtest holds the mock server (the page's own files, scripted answers,
no model) and the browser tests; serve/test_web_preview.js checks the documents,
flags and frames, serve/test_web_preview.py the header and that the page never
listens for a frame's postMessage.
Ported by hand from PR #1628 by thoives.
Co-authored-by: Thomas Ivesdal-Tronstad <[email protected]>
A PC whose CPU lacks the AVX-512 set that setup checks for gets the native Q2_0 pack. On a CUDA card of sm_80 or newer its
Q2_0 / Q2_0 prompt experts run through MMQ, although the fused int8 kernel for that format (#2145) is in the binary and set
up only on a Turing card (b6443f3ae).
STRATA_PF_FUSED_Q2N=1 sets that kernel up on any sm_80+ card and takes it for the Q2_0 / Q2_0 layers, under its own variable
so that STRATA_PF_FUSED=1 keeps its bits. STRATA_PF_FUSED=0 and STRATA_PF_FUSED_NATIVE=0 still turn it off. A failed setup
leaves the occupancy as it was and the layers on MMQ.
- moe_fused_iq.cu: DevInfo.q2n; dev_info() sets up Q2_0 gate/up when the switch is set; gu_covered() and native_supported()
take the pair only when q2n is set (a Q2_0 gate with another down type is refused, as on Turing).
- tests/cuda/prefill_fused_iq_test.cpp: --unset-pf-fused leaves STRATA_PF_FUSED unset (the engine's default) and skips below
sm_80; CMakeLists.txt: the ctest entry prefill_fused_iq_test_q2n runs the Q2_0 / Q2_0 pair with the switch alone (exit 77 =
skipped without an sm_80+ card; CUDA branch only).
- docs/DETAILS.md: one section in the author's measured numbers (RTX 5080) and the A4000 caveat below.
Unset (the default), nothing changes: dev_info() reads the variable once and sets nothing up, gu_covered() and
native_supported() return what they returned. Read line by line against base: the Turing path (t75) and the HIP path are
untouched. Set, the prompt arithmetic changes (first-token KL against FP16 0.0110, MMQ 0.0112, the author's figure) and the
fused path's smaller buffers let --prefill auto pick a larger chunk, which is most of the author's 16K gain; the generated
text can change.
Not verified: an RTX A4000 (sm_86). The kernel was taken on every CUDA card in #2145 and then kept to Turing after an A4000
no longer matched the old output and read 4K prompts 4.5% slower; the PR does not repeat that run. Merge note: #3277 and #3779
change dev_info() and native_supported() in the same file.
Tests: prefill_fused_iq_test_q2n and prefill_fused_iq_test pass on an RTX A4000 (sm_86, synthetic data); on the Q2_0 / Q2_0 layer the
fused kernel is 1.76x / 1.30x / 1.24x the MMQ path at 2048 / 4864 / 8192 tokens with the same relative RMS against FP32. Builds on CUDA
sm_86 (Linux) and sm_120 (MSVC). Not covered: no model run; the engine-level A4000 check is the gate.
Chat settings > Render math (off by default). When it is on, \( \), \[ \], $x$
and $$ $$ in an answer are drawn as formulas by KaTeX 0.19.0, which is vendored
in serve/web/vendor/katex (script, style, 20 woff2 fonts, MIT licence and a
PROVENANCE.json with the npm integrity and the sha256 of every file, so the page
needs no network). Nothing of KaTeX loads until an answer holds a formula.
The shared core (math.js, from #2972) keeps the text of every message exactly as
the model wrote it: formulas are swapped for placeholders before the Markdown
runs and drawn over them afterwards, so history, copying, the Markdown export
and the requests never see the drawing. Code, link targets, prices ($5 and $10)
and a formula still being streamed stay text. A formula that fails, is over
16 KiB or times out stays as its source text.
Safety: KaTeX runs with trust off (no \href, \url, \includegraphics, \htmlClass:
they are drawn as their own name), maxExpand 1000, maxSize 20 and a fresh macro
table per formula; it builds DOM nodes itself, so no HTML string from a model
reaches the page. A formula already drawn is placed again at once while an
answer streams, so it does not flicker back to source.
serve/test_web_math.js runs the real vendored KaTeX against hostile formulas;
serve/test_web_math.py checks the vendored files against PROVENANCE.json and
that nothing outside this server is addressed. serve/web/vendor is -text in
.gitattributes so the hashes hold on a Windows checkout.
Ported by hand from PRs #2972 (core) and #2974 (KaTeX) by CC-David-CC.
Co-authored-by: David <[email protected]>
The served page had no way to translate its text: every string was English in
the HTML and the script. Now the English text stays in the code next to a key
(tr("key", "text", vars), data-i18n="key" in the HTML), a language is a catalog
serve/web/locales/<lang>.json, and a catalog that is missing or has no entry for
a key leaves the English text. The language is the user's choice (About >
Settings), kept in this browser; nothing is chosen from the browser's language,
because a page translated in part would mix two languages. /?lang=de shows one
for a visit.
The server hands out /web/locales/<lang>.json and /web/vendor/katex/** from
serve/webfiles.py: whitelisted names and types, no backslash, colon, percent or
dot-dot, and the resolved path must stay inside the folder (also against a link
out of it). tools/web_i18n.py reads the English text out of the code, writes
en.json and checks every catalog for the same keys and {placeholders}; the
check runs once the catalogs exist.
Only the part of PR #3311 by gopinath87607 that is about this engine: a periodic mark is saved at a chunk boundary of the batched prefill and the read carries on
(src/program/generate.cpp, periodic_checkpoint), so the interval is not a piece-size dial; what cuts a read into runs is the re-read, the root, a message
boundary and the start of the last turn; --prompt-cache-every 0 skips the periodic and the near-tail marks. The wall-time numbers and the comparison with the Maya
checkout belong to another engine and are left out. New section in docs/PROMPT_CACHE_TAIL.md, linked from the option's row in docs/DETAILS.md.
--show-key only knew a running launch or a preset. A model set up by setup.py (START-HERE.bat, ./setup.sh) keeps its generated key in strata-<model>.json, so the command
said there was no key and people read the JSON by hand. With no launch and no --preset, it now looks at the run configs of the installed models: one model with a key
prints that key (stdout is the key alone; a hint line with the model and the file goes to stderr), several models ask for --model NAME (any case, the size or the full
id), a name that matches nothing says so. --preset never falls back to a setup config, and no key is printed while the choice is open. Tests:
tests/terminal/test_launch_show_key_setup.py.
A volume or bind mount from the 0.1.x image keeps its models in /data/models. The entry point moves a folder that is part of the volume, but a folder the person
mounted there is left alone, and a failed move leaves the files too; with no MODEL_PATH the start then did not see them and downloaded the model again
(tens of GB) without a word. Now, when .gguf files are still in /data/models and MODEL_PATH is not set, the container stops with code 2 and names the folder,
says what to do (MODEL_PATH to the folder with the shards, or the files into /data/Strata-data/models) and that nothing was downloaded.
ALLOW_DOWNLOAD=1 goes on anyway with the same text as a warning. docs/DOCKER.md lists the setting. The other points of the issue (the key, the removed
variables, the status loop) are the owner's design questions and are not changed here. Tested by running the guard function with sh (tests/terminal/test_docker_files.py);
the whole container was not run (no docker on this PC).
Prefill on one card, decode on another, the session handed over in RAM through the engine's
own session file (SAVE/RESTORE). The engine's new request form GEN 0 prefill=1 reads a prompt
as a request does, up to where the first verify window starts, and stops with the session at
n - 1 tokens, which is the place a continuing request resumes from. Opt-in; without
split_serve nothing changes.
Idea and manual setup by @MarcoTool in #4000.
Problem: the strata-dsv41 packages were built for sm_86 and sm_120 (+ PTX 120). An RTX 4090 user was stopped at setup on 0.2.4.1; since
b750d757f an sm_86 package is accepted on an sm_89 card (cubin same-major rule), but the engine had no sm_89 image of its own, and the docs said
nothing about RTX 40 or about building the engine for a card the packages do not cover. The reporter built strata-dsv41 for 86-real;89-real
themselves and ran the 2.00 set at 40.2 tok/s on an RTX 4090 48 GB (theirs, --dspark on).
What changed:
- packaging/build_dsv41_cuda.sh and build_dsv41_windows.bat: the default architectures are now 86-real;89-real;120, so the DeepSeek CUDA 13
packages list sm_86, sm_89, sm_120, compute_120 (Windows and Linux alike). The CUDA 12.9 line in the script header says ARCH="86-real;89".
- CMakeLists.txt: a build of strata-dsv41 with no -DCMAKE_CUDA_ARCHITECTURES defaults to 86-real;89-real;120 (the Qwen engine's default
stays 120, and the warning now names the architectures it picked).
- strata_control/engine_component.py arch_native(): true when a package holds machine code for the card itself. resolve() and
feed.select_engine() use it as the last tie-breaker, so with otherwise equal packages the one with the sm_89 image is taken on an RTX 40;
an sm_86-only package still fits (no refusal).
- setup.py: the "engine not available" stop now points at the build recipe (docs/DEEPSEEK41.md); the developer line names the
architecture to pass. The card list already said RTX 30, 40 and 50.
- docs/DEEPSEEK41.md, docs/MODELS.md: RTX 40 coverage, the reporter's measurement quoted as theirs, no claim of our own (we have no Ada card),
and a section "Building the engine for your card".
Tests: tests/terminal/test_engine_manifest.py (arch_native; the build scripts and the CMake default name sm_89 and both OSes agree; the
package list they give is sm_86 sm_89 sm_120 compute_120; an sm_89 card is accepted and prefers the package with sm_89). tools/test_setup_dsv41.py
unchanged and passing.
Not covered: no run on an Ada card (gate script in phaseA-tests/gh027/gates/3935-ada-dsv41.sh).
One bullet from PR #3930 by NorwinYu: on two identical RTX 2080 Ti over a dual-socket Xeon E5-2696 v4 a plain split moved decode from 35.2 to 51.0 tok/s and an
11.6K prompt from 672 to 908 tok/s, because the CPU pool per window fell from 9.52 to 0.79 ms (STRATA_DECODE_TIMING=1). The page's own 5080 + 3090
figures, where the second card is on par, came from a fast CPU; the bullet says so and that it is one measurement. No code changes: setup already splits
across two cards. Shipped without a hardware run of ours; the reply asks owners of two older cards on slow CPUs to compare and report.
The chat, About and Monitor columns grow with the window (the chat and the
composer to 1100 px, About to 1100 px, Monitor to 1920 px), the metric cards
wrap with auto-fit, and under 640 / 560 / 480 px the controls get bigger
touch targets, the composer bar wraps, and long words, code and tables stay
inside the viewport.
Ported by hand from PR #3176 by Matthias Nott (mnott); the CSS is his.
Two paragraphs from PR #3895 by Vop3p, put under the table row they explain: the fork pays only when it has a CPU-pool wait to hide (4,859 slots on one
RX 6900 XT, GPU-reach wait 20.7 of 39.1 ms; with every expert resident it is 43.4 against 62.9 tok/s on an R9700), and the Windows RX 6900 XT fault with
the fork plus --kv-resident, its bisection and its limits (one PC, kernel not isolated). The text now says what the code does about it and asks RX 6000 owners on Windows to test.
On one Windows RX 6900 XT STRATA_SH_STREAM=1 together with --kv-resident failed the engine on its first request (verify: unspecified launch failure).
setup --calibrate (tools/calibrate.py) no longer measures the fork on Windows when KV streaming is on (the fork is off by default on AMD, so
nothing changes unless the person sets it), strata tune's never-pick list holds the combination (strata_control/tune_knobs.py), and a config that
sets STRATA_SH_STREAM=1 with --kv-resident on Windows HIP gets one warning line when the engine starts (serve/server.py sh_stream_warning). Nothing is refused or
removed: the variable still reaches the engine. Shipped without a hardware run, from one report; the reply to #3895 asks RX 6000 owners on Windows to test
and report. Tests: tests/terminal/test_sh_stream_windows.py.
The 0.2.5 rewrite dropped about 25 entries: the Mac quarantine steps, strata doctor, sending logs, the first-start freeze, the graphics memory the app keeps free,
Smart App Control, UPDATE.bat and git pull (#1276), the build tools and the gcc/glibc compile problems, the engine checksum, the engine stopped unexpectedly,
slower than the tables, the empty-reply and Turing switches, --expert-cache and --resident-budget-gib, the page-locked OOM, the NVIDIA 0x141 reset, CUDA 13.2 on
RTX 50, the AMD section (KFD, the build list, 0xC0000005, the driver reset, pinned allocations, verify timeouts) and the body of the verify-timeout entry, which
the rewrite had left empty. Each is back with the 0.2.7 facts: the app's reserve defaults (native-resources.mjs), the Drafting setting, --vision on|off|gpu|cpu
in the Terminal and --vision no in a source checkout, Intel Arc's CPU reader, and the Terminal route next to the source-checkout one.
Every pointer resolves: setup.py's driver-reset pointer, the server's Smart App Control message, DETAILS.md's STRATA_KEEP_EMPTY_TURNS / STRATA_TOPK_STREAM line,
OLDER_GPUS.md and MAC.md. tests/terminal/test_troubleshooting_page.py checks the entries, those pointers and the page's own links and anchors.
Problem: below sm_70 the prompt reads each QSA layer's attention through the
decode kernels (qsa_prompt_attn_batch returns false without tensor cores): 32
queries a call, every chunk's partial results (817 KB a query) written to
scratch and merged by a second kernel.
Change: src/kernels/cuda/qsa_pattn.cu keeps the contributor's default variant
only, ps_score_x_kernel (q in registers, the chunk's K staged with 16-byte
loads, two cells at a time through a select-free reduce-scatter) and
ps_value_d4_kernel (four dims per thread, merged into the output as the merge
kernel does). Every float operation of attn_chunk_kernel_pre75 and
attn_merge_kernel is kept in its order, so the output is the decode kernels'
bit for bit. The PR's other variants (per-chunk, fused persistent, bench split
codes), pattn_bench and the variant argument are not ported; pattn_bench needs
a STRATA_QSA_DUMP file from a model run.
Gating: compiled only in the CUDA build with STRATA_EXPERIMENTAL_SM60 (HIP and
Metal link a stub that answers false; prefill.cpp keeps the call out of the
Metal build). At run time only with STRATA_PATTN_SM60=1, on compute capability
6.x, int8 K/V, head_dim 256, 12 query heads per KV head, at most 33 chunks.
Unset, the only cost on the default path is one cached int test per QSA layer,
and only when qsa_prompt_attn_batch has already declined. Any refusal or a
failed launch returns false and the usual 32-query loop runs (one warning, no
exit; the PR called exit(1) on a launch failure).
Tests: src/kernels/qsa_pattn_parity.cpp (new, registered in CMakeLists.txt in
the experimental sm_60 test block, skips with 77 when the kernels are absent)
compares the kernels with the Daily's chunk kernel plus attn_merge_kernel on
random int8 K/V: a shuffled page table with ~3% unresident pages and a run of
64 unresident pages (whole chunks masked, some queries empty), three selection
shapes, caps 64 / 130 / 2051 / 2112, widths 0..cap, n_q up to 40 with a
scratch that holds only 2 decode-layout queries (several batches), junk in the
scratch and output. On a Pascal card it also requires the checked entry to take
the shapes and qsa_decode_attn_batch to match. The kernel entry that skips the
compute-capability test (qsa_pattn_batch_any_arch) exists for this test only.
Results: z440 (RTX A4000, sm_86, CUDA 13.0, build with
-DSTRATA_EXPERIMENTAL_SM60=ON, gpu lock): "qsa_pattn_parity: 468 cases, 0
differ, 29724792 nonzero output values in all -> PASS". sm_60 compile of the
whole tree (CUDA 12.9, P1004): exit 0.
Not covered: no P100 run here (the Pascal-only checks of the test and the
speed claims wait for the gate, phaseA-tests/gh027/gates/3943-pattn.sh). The
contributor's numbers (43.3 -> 30.8 ms a 1,536-query chunk per QSA layer at
26K context, 2-3% less time on 8K-150K token prompts on four dies) were not
re-measured; compute capability 6.1 is covered by the same switch and was not
measured by anyone.
The Speed group gets a row for Qwen on a discrete CUDA card of 13 GiB or less: "Prefers the parts of the model already on the graphics card, which makes answers
much faster on 12 GB cards. Turn off if answers look worse than before." On (recommended) by default; Off writes STRATA_ROUTE_PRIOR=0, On removes it. The
DeepSeek speed-ups row, when Off, writes DS4_DEFAULTS=026 and the six DS4_*=0; On removes all of them, and a file that holds only the six (an earlier test
build) shows as Off. The Qwen "0.2.6 behaviour" switch is Terminal-only and is not in the app. New strings fx82 and fx83 in the eight fix catalogs.
- DeepSeek speed-ups Off now writes DS4_DEFAULTS=026 (the engine's master switch back to the 0.2.6 defaults) plus the six DS4_*=0 for older engines; On removes
all seven. A file with only the six (an earlier test build) still reads as Off and On removes them (`legacy` entries).
- New switch "Faster answers on 12 GB NVIDIA cards" (the residency-biased routing default): on by default, Off writes STRATA_ROUTE_PRIOR=0. Applies to Qwen
Flash-Next (no arch) on a CUDA build with a discrete NVIDIA card of 13 GiB or less (not a chip that shares system memory), the engine's own rule.
- New Terminal-only switch "Qwen: 0.2.6 behaviour" (STRATA_DEFAULTS=026, off by default), a row of the Advanced table for every Qwen model, for
troubleshooting; the app never shows it. DeepSeek has DS4_DEFAULTS instead.
tests/terminal/test_speed_switches.py covers the new defaults, applicability, the seven DeepSeek entries and the legacy six, the Advanced rows and the
comparison with the app's table (the Terminal-only switch is left out of it).
/v1/chat/completions on the CUDA and HIP engine accepted logprobs and
returned an empty choices[0].logprobs.content. Now "logprobs": true with
"top_logprobs" 0 to 20 (and the legacy "logprobs": N) returns one entry
per token with its alternatives, streamed or not, and "allowed_token_ids"
returns a distribution renormalised over exactly the ids the caller named.
Engine (src/program/generate.cpp): the GEN line takes logprobs=K and
allowed=a,b,c. After each "T id" the engine writes one "LP <logprob>
[id:logprob ...]" line from the verify window's head logits before
sampling, penalties and temperature. The SYCL engine already wrote LP
lines for logprobs=K; it now reads allowed= as well, with the same line
format. A request without logprobs reaches none of this: both call sites
are behind "req_logprobs >= 0", a local int of the request, so the default
path prints what it printed before and does no extra work per token.
Changes from the PR: the call sites are guarded instead of calling the
function for every token, the function is static, the token index is
checked against the row, and the Chinese comment is in English.
Server (serve/server.py): LP lines are read before the token is yielded
and belong to the request that asked (tagged with its cancel Event), the
entries spell the reply, json_schema replies return no logprobs rather
than a mismatching set, an allowed id past the vocabulary is a 400, and
logprobs on a --parallel server is answered for max_tokens 1 only (a
slot's BT lines carry no LP line). serve/structured.py: _extract_json is
now extract_json (the logprobs alignment uses it); the two uses added by
the #2893 retry follow the rename.
Left out: the PR's bench and result files (none in the diff). #3332 (/v1/completions) and #4056
(logprobs on it) build on this change; this one needs nothing from them.
Tests: serve.test_logprobs and serve.test_logprobs_parallel (30 tests with
the PR's), serve.test_structured, serve.test_responses: 94 tests OK.
Not covered: no model ran here. The PR author reports 28 of 28 on an RTX
5080 with MTP and on a GTX 1080 Ti; the layer-split pipelined decode path
(the first call site) was not run by anyone, and the LP values were not
compared with a CPU softmax of dumped logits.
Co-authored-by: Leonardo Murillo <[email protected]>
Review of the first port: it registered kIlRows_89 / kIl1Rows_89 for every
compute capability 8.9 card by default. Bit-identity of an interleaved-kernel
choice to native_mmvq is a property of the compiled code, and the release
engine builds its own sm_89 cubin that no card of ours can run; the issue also
saw no end-to-end decode change on a 4090 or a 4080, and the 4070 and 4080
runs agree in 53 of the 75 cells of the 2-4 column table (33 of them non-zero). So the default
change has no measured benefit and an unproven bitwise claim on the card it
changes.
Change: sm_89 answers the shared table, with no one-column row, exactly as
before, unless STRATA_MMVQ_IL_89=1 is set (read once, like the neighbouring
switches). With it set the cells from the two community runs apply. Nothing
else in the lookup changes; sm_86, sm_120 and the others are untouched.
Why it is bitwise: native_mmvq_il_kernel keeps a separate accumulator set per
row (acc[v][i][c]) and sums each row over the same chunk order and the same
warp_sum, so the rows-a-warp count R only changes how many rows a warp walks,
not the order of any row's float additions; the plain-versus-interleaved
equality is what mmvq_il_parity measures (memcmp against native_mmvq for rows
1/2/4, T 1-8, odd row counts). That test passes on sm_86; it has not run on an
Ada card.
Tests: mmvq_il_parity now checks the unset behaviour (sm_89 equals the shared
table, Q3_K and the one-column row answer 0) or, with STRATA_MMVQ_IL_89=1, the
4070/4080 agreement. A second ctest, mmvq_il_parity_89table, runs the table
check with the switch set and no kernel case. On z440 (2x RTX A4000, sm_86,
CUDA 13.0, gpu lock): mmvq_il_parity and mmvq_il_parity_89table both pass, and
the binary prints "mmvq_il_parity: OK (every case, T 1-8, rows 1/2/4 and the
table, bitwise)" with the switch unset and set.
Model gate owed on an sm_89 card: phaseA-tests/gh027/gates/3453-sm89-il.sh
(mmvq_il_parity on the card, then identity of the answers with the switch on,
then a speed A/B that decides whether the default flips).
Co-authored-by: rodi4eg <[email protected]>
With --batch-groups above 1 on a layer split the pipelined batch loop never starts a round of the adaptive VRAM tier (the
serial batch windows start one every --adapt-every windows). While the slots are busy the tier only moves when they go back
to the solo path, so a fresh engine keeps its start-up profile for minutes.
STRATA_PIPE_ADAPT_EVERY=N (opt-in; unset, 0 or not a number is off): after N group windows pump starts no new group on the
first stage, lets the groups in flight finish, then runs one round as batch_step does between serial windows, adapt() and
apply_pending(true), so nothing in flight reads a cache slot while experts swap and the new experts have landed before the
next launch. Not while a prompt is read beside the slots (in_prompt_read, its loan of cache slots), not with --adapt-async,
not while the last round's copies are in flight, not with the tier off. The rule is two pure functions in
include/strata/program/batch_groups.hpp (pipe_adapt_every, pipe_adapt_due) with 12 new cases in batch_groups_test. With the
switch set the engine says so once at start and the "strata batch (pipelined ...)" line adds the rounds, the experts
swapped in and the time spent. docs/BATCHING.md documents the switch with the author's table (2x RTX 5090, six streams,
309 to 410 rows/s over eight bursts after a fresh start, two engines per setting).
Reviewed: unset costs one counter increment per group window and one false check where a group starts, and prints no new
line. A due round cannot wait forever: no new group starts, the groups in flight finish, then the round runs. A prompt read
sets in_prompt_read before the reader thread starts (including STRATA_PIPED_CONCURRENT) and clears it after the reader
joined; the drain after the read runs the round if it is due.
Tests: batch_groups_test (the new cases). Not covered and still the author's to show (the PR is a draft): a soak with
prompts read beside the slots, a Linux run, another GPU pair, more than two stages, and exactness settings with the switch
set. The model gate is written down.
STRATA_ENGINE_BACKEND=hip-rdna2 worked for strata update (updater.detect_hardware), but strata update --backend hip-rdna2 offered no package and
strata launch stopped with "STRATA_ENGINE_BACKEND='hip-rdna2' is not one of cpu, cuda, hip, metal, sycl". engine_component.split_backend reads
hip-<variant> as the hip family and its package; detect_backend (the start path), detect_hardware and update --backend all use it. Any other name is
checked as before. Tests in tests/terminal/test_gfx_selection.py.
STRATA_RAM_ADAPT (opt-in, --serve, one GPU, needs --adapt-every) lets the resident RAM copy follow the conversation:
experts the windows keep routing to take the RAM places of unused experts of their layer. Each move re-read the incoming
expert from the file and a layer moved one pair per window, although the incoming expert was usually read for the window
before and its blob still sits in that read's stage buffer.
What changed:
- FileExpertSource::replace_resident_staged(layer, in, out): the same move as replace_resident, with the bytes copied out
of the finished stage buffer of `in` (no file read). The copy runs under stage_mu_, so claim_stage cannot give the buffer
to another blob meanwhile; it needs a non-busy buffer whose key is `in`, and returns false (nothing changed) otherwise,
and the pair then goes the old way. The Foresight hold/unhold around the copy is the one install_resident uses.
- generate.cpp: a layer pairs up to 4 of its most routed outside experts with up to 4 of its least routed ones in the copy
(outgoing count below 0.5, incoming 2 or more), largest difference first. A pair whose incoming expert is staged moves by
copy; `n` now caps only the pairs that need a file read. The log line counts the moves and how many came out of a stage
buffer (ram_staged()).
- docs/DETAILS.md: the RAM_ADAPT bullet in the low-RAM list in plain words and one row of the measured table (the author's
numbers: RTX 3060, IQ2_XS and IQ3_XXS, 120 ShareGPT requests, 3 interleaved runs each).
Unset (the default) nothing runs: the block is guarded by ram_adapt_n > 0 as before. The weights moved are byte copies of the
same blob; which experts sit in RAM changes, as it already does with RAM_ADAPT, so greedy token ids may differ from run to
run once it is on. Changed from the PR: the DETAILS.md hunk was rewritten against the current, shorter bullet.
Tests: file_expert_source_test (new staged cases; the positive case needs unbuffered load and is skipped without it).
Not covered: no model run. Model gate written down: RTX 3060 ShareGPT replay =8 against =1.
The Build section said nothing about it. gemm.cu (create_hipblaslt_state, own_tuning_dir) takes STRATA_HIPBLASLT_TUNING, else the file
<engine folder>/tuning/<arch>-hipblaslt-<version>.txt, else plain hipBLAS. The report measured 826 against 1,043 tok/s at 8K prompts on a
gfx1151 (one run per arm, stated as such) and the log line that tells whether a table was taken. The extra log line the issue suggests for the
no-table case is a small engine change and is not part of this commit.
The first #3218 commit made setup.py admit a Maxwell (sm_52) card only
when a tester set STRATA_EXPERIMENTAL_SM52=1, and then routed it to the
CUDA 12 engine. That engine holds no sm_52 code, and users never set
environment variables to get a fix.
sm60_card() is back to Pascal and Volta (6.0 to 7.0). A PC whose only
NVIDIA cards are Maxwell now gets a warning (maxwell_note) that says no
ready-made engine holds code for it, gives the source-build recipe
(-DSTRATA_EXPERIMENTAL_SM60=ON -DCMAKE_CUDA_ARCHITECTURES=52, CUDA 12.x)
and the one reporter's result, and points to docs/OLDER_GPUS.md. The card
table line says Maxwell runs only from a source build. A Maxwell card
beside a newer card prints nothing new. Naming the card (--gpu, --cuda
12) or any env var does not admit it. The CMake floor from the first
commit is unchanged (opt-in build only).
Tests: tools.test_setup_older_gpus 23 tests OK (a Maxwell-only PC through
setup.main() with and without the old switches installs nothing and
prints the recipe; a Maxwell card beside an RTX prints no note).
Not covered: no engine was built for sm_52 by this commit; the P1004
compile is recorded in the notes.
The report written by BENCH.bat and bench.sh had fields missing or wrong: nvidia-smi's 'CUDA UMD Version:' (current drivers) was not read, the value
was the newest CUDA the driver supports but was labelled like the engine's toolkit, storage always said 'not measured', and Windows 11 showed as
Windows 10. Now the report says 'CUDA (driver supports up to): X; engine built for CUDA Y' (BUILD.json beside the engine), names the storage device
(Get-PhysicalDisk, system_profiler, lsblk -e 7 to skip snap loop devices), adds a 'GPU detail (after the run, near idle)' line with power limit and draw,
PCIe generation and width, pstate and temperature, and recognises Windows 11. The standard report (strata bench run, strata_cli/bench_facts.py) already
read the UMD version, storage and power limit; this brings the legacy tool to the same. Ported from PR #3250 by JoShMiQueL with its tests.
Two 0.1.44.1 changes no page told. TROUBLESHOOTING.md gets the start error 'the engine process ended: ...' (serve/server.py start_exit_note: the signal or
Windows exit code, and for an illegal instruction the AVX, AVX2, FMA and F16C flags the CPU lacks). SOURCE_INSTALL.md says a setup run again
pre-selects the installed size, marked '(installed)', and --yes takes it (setup.py size_tag, installed_size). Written by hand from PR #3243 by
Buddychek, whose hunks no longer fit the rewritten pages; each sentence checked against the two functions.
Review fix on the #4005 port. STRATA_PREFILL_COPY_WAIT_DEDUPE=1 skipped the copy stream's wait on a slot's release
event when the previous entry had waited on the same event. Two cases break that:
- At idx == ring the previous entry (ring - 1) is still in the first round, so its wait was on the event of an earlier
walk. The same slot id can be this walk's first flush, which re-records the event after that wait; skipping the wait for
slot 0 then lets the next copy overwrite a slot that the flush's gather has not finished reading.
- With a ring of one MMQ group (16, the smallest ring) a flush of the whole ring re-records the event right after the
previous slot's wait.
The rule is now copy_wait_dedupe_ok() in include/strata/prefill/copy_events.hpp: idx > ring, ring > group, same event.
copy_events_test models two to three walks with random group sizes and random issuer/consumer interleaving and checks that
a refill always waits on a record at or after the flush that released its slot. The same model with the PR's original rule
(idx >= ring) finds the stale-event case, so the test can fail. The default path is unchanged (dedupe off).
Tests: copy_events_test passes (z440, g++ -std=c++20). Not covered: a GPU run.
Review fix on the #2526 port. At an admission from a slot the port copies every checkpoint of the slot that is not
longer than the copied-back length into the main session's chain. Such a checkpoint is valid there only when its tokens
are a prefix of what was copied back (its K/V cells are then the ones the main session holds) and it has no pictures
(the main session's pictures are cleared at a give-back). The PR did not check either; a checkpoint left over from a
branch the slot no longer holds would sit in the main chain with cells that are not its own.
Both trees (CUDA/HIP and SYCL) now test the prefix and the empty picture list per checkpoint. The checkpoint the
request continues from is copied as before. Nothing changes when every checkpoint is a prefix, which is the normal case.
Tests: compiles on CUDA and SYCL (see the build lines of the branch). Not covered: no model run; the gate is written down.
docs/INTEL.md and docs/INTEL_ARC.md described an SSD expert tier that strata-sycl.sh's default (STRATA_VERIFY_NO_HOST=1) does not have: every
expert must be in VRAM or in the pinned RAM mirror, and the engine refuses to start otherwise (sycl/src/program/generate.cpp, 'REFUSED: ... are
neither in VRAM nor mirrored'). A native (IQ) pack refuses --spec 0 and 1 (sycl/src/program/generate.cpp and src/program/generate.cpp). Ported by hand
from PR #2153 by nekomario28 onto the rewritten pages, each sentence checked against those two messages. The hunk for docs/TROUBLESHOOTING.md is left
out: the entry it changed is no longer on that page.
Adds a minimal handoff configuration (--prompt-cache 1, --prompt-cache-every 0, --prompt-cache-root 0, --conversation-cache-mib 0), what each
option does, that it costs RAM, that it does not make cold prompts prefill in parallel, and how to look for the handoff in the engine log, /metrics
and /v1/status. Ported from PR #3334 by oscarlius (continuing #3035). The four options exist in src/program/generate.cpp and the o.prompt_cache > 0
gate matches the text; the log evidence was not reproduced here (no model run).
metal/src/mtl_rt.mm names MTLGPUFamilyMetal4 (macOS 26 SDK) and MTLCompileOptions.mathMode (macOS 15 SDK) behind @available, a run-time check;
the names must still exist in the SDK the engine is compiled against, so the metal-link job failed on the macOS 14 SDK on every pull request that
touched src/, include/ or metal/. The job now runs on macos-26 (the runner-images repository lists a macos-26 image). Ported from PR #3870
by stuchapin909. A GitHub Actions run cannot be made here: the author ran this workflow on his fork; the first run after the merge proves it.
#3824 changes the same line with macos-15 and an Xcode 26.0.1 path pin, and is not taken.
make_blob quantized its random test rows with ggml-cpu's type traits from_float, which the i-quants do not have: an IQ3_S pair called a null
pointer (an access violation on Windows, SIGSEGV on Linux) before the first timing. It now uses ggml_quantize_chunk, with a flat importance
matrix where the type needs one. A Q2_0 gate/up or down is timed as pool.cpp runs it (activations as an ActQ, Strata's own Q2_0 rows, SwiGLU
in between) instead of with ggml-cpu's Q2_0 dot, and the pair's header line says so. Other formats are timed as before. Ported from PR #3871 by
Licenser.
Built on this PC (MSVC, CPU only) and run: iq3_s/q2_0 0.531, iq2_xs/q2_0 0.336 and q2_0/q2_0 0.214 ms per expert per thread (--mb 512, 1 thread).
Checkpoint.raw mapped every shard for the life of the process. On Windows each faulted page of the ~64 GB of expert bytes is charged to the
process working set, so the pack step looked like a 64 GB RAM demand and was killed by free-RAM watchers on a PC with a model loaded (nothing
leaked: it all came back at once). raw() now reads the tensor's byte range into a fresh buffer through one cached file handle per shard; the
pages go to the system file cache, which counts as available. The author measured the same bytes, 1.2x faster, flat working set, and a
byte-identical pack. raw() is no longer thread-safe (this file uses no threads). Ported from PR #3378 by jerry78424 (the fix for his issue #3377)
with its two tests; I ran the tests (13 pass); the full 102 GB pack was not run.
A GGUF header with an unknown metadata type or an impossible length raised KeyError or OverflowError inside the reader, and setup, which expects
a ValueError, printed a traceback instead of its diagnostic naming the broken shard. Strings, scalars, arrays and tensor dimensions now share
one bounded read; counts are checked against the bytes left in the file before anything is constructed; type ids are checked before their values
are decoded; nested arrays are reported as unsupported; general.alignment must be a positive multiple of 8. The file name is added once to the
message. Ported from PR #2288 by KeyanHu-git with its tests (tools/test_gguf_reader_malformed.py).
Checked against the old reader on the 18 GGUF v3 vocabulary files of the pinned llama.cpp: same metadata, tensors and data start.
When one assistant message made several tool calls and the client returned the results in another order, the call ids were dropped before the
prompt was rendered, so the model saw anonymous results in the wrong order. The ids are now kept while the request is normalized; a run of
results is reordered to the call order only when the call ids and result ids are non-empty unique strings with matching sets. Anything else
(missing, duplicate, unknown or incomplete ids) keeps the received order, and the ids never reach the template. Ported from PR #3395 by
j-luwierski with its seven tests (serve/test_chat_tool_results.py); the hunk was merged by hand beside require_messages.
Problem: an Ada card took the shared MMVQ-IL table, which has no Q3_K row, so
Q3_K shapes ran the plain kernel and the other types used a table measured on
sm_86 and sm_120.
Change: kIlRows_89 (2-4 columns) and kIl1Rows_89 (one column) are registered
for compute capability 8.9. The 2-4 column table keeps only the cells where the
RTX 4070 run (rodi4eg) and the RTX 4080 run (kzhv) gave the same rows count and
sets the rest to 0, the plain kernel; the 4070 and 4080 disagree in 13 of 15
cells of the unmerged tables. The one-column row is Q3_K only, which both cards
gave. The issue's 5-8 column table is not taken: the 4080's changed in 10-14 of
20 entries between two runs of the same card, so there is no kIlRows58_89. The
bench output files of the PR are not ported.
Every choice in a table is bitwise native_mmvq's output (kernel row split only),
so answers do not change; this is speed only. No end-to-end decode change was
seen on a 4090 or a 4080 in the issue, so the gain is the microbenchmark's.
This does change which kernel an sm_89 card runs by default, so the model gate
(identity on sm_89) is written down in phaseA-tests/gh027/gates/3453-sm89-il.sh.
Tests: mmvq_il_parity (registered in CMakeLists.txt) now checks sm_89's table
against the two source runs (check_tables) instead of expecting the shared
table, and still checks bitwise parity of every case and rows choice on the GPU
it runs on (sm_86 on z440). docs/MMVQ_IL_TABLE.md updated.
Not covered: no sm_89 card here, so the table speed is the contributors'
numbers, not ours.
Co-authored-by: kzhv <[email protected]>
When the elastic image encoder failed to restart, its exception left request preparation and the client got a closed connection. It now goes
through the EngineDied path: HTTP 503 with the reason, a later picture tries again, text requests and a later good start are unaffected. An idle
encoder stop is logged even when the cache shrink was refused, and a layer split gets a line saying its encoder VRAM has to be kept free with
--vram-reserve-mib. Ported from PR #2435 by i-Vladimir-i with its tests (chat, messages and responses, streamed and not, plus a real encoder
that exits with code 71 on Linux) and the docs/VRAM_ELASTIC.md text.
On a layer split the later card is left with a few dozen MiB after loading, and each single-request window size is
captured the first time it is used (68, 64, 58, 52 and then 10 MiB free in the report); when one capture's
cudaGraphInstantiate ran out of memory, instantiate_evicting could only free graphs of batch slot layouts, a
single-request run has none, and "verify: instantiate: out of memory" ended the engine.
Now, after the batch layouts, a single-request window's capture frees the graph of the largest other window size (never the
one being captured or a prelaunched one; pick_window_victim) and instantiates again, and says so once: the freed size is
captured again when it is next used. capture_all keeps every size at once and is excluded. Only a failed instantiate does
anything new; with enough VRAM nothing changes.
window_evict_test (CPU, header-only) covers the choice. The real gate needs a model on a 2-GPU split with a thin later card
(listed in the triage file); not changed here: the later-stage reserve of the split search (kWindowMib), which is a default
placement change.
Under WDDM every cudaEventRecord and cross-stream wait is a submission of its own (the author measured ~80 / ~45 us each
on an RX 6900 XT under Windows 11), and the streamed walk paid one record and one wait per expert although a group
gathers after one wait on its last copy. Two tester switches, threaded issuer only:
- STRATA_PREFILL_COPY_EVENT_EVERY=N (2..64): the copied event is recorded every N-th entry, before the issuer waits and on
the last; the compute stream waits on the first recorded entry at or after the one it needs (the copy stream is in order).
- STRATA_PREFILL_COPY_WAIT_DEDUPE=1: the copy stream skips its wait on a slot's release event when the previous slot was
released by the same event.
The same copies into the same slots: the same bytes.
Default path (both unset): read line by line. The issuer records copied[slot] and waits on used[used_of[slot]] exactly as
before (the counters are bookkeeping, cover() is a no-op). Added to the PR so this holds by construction: wait_copied takes
the slot the old code used and, without the sparse switch, waits on copied[slot] at all three consumer sites (the PR derived
the slot from the entry number). The record/wait selection is now three pure functions in
include/strata/prefill/copy_events.hpp, with a host test (copy_events_test, registered in CMakeLists.txt) that models the
issuer and consumer with a full ring and checks that every wait names a live recorded entry at or after the one needed.
Tests: copy_events_test passes (z440, g++ -std=c++20). Not yet: a CUDA build of prefill.cpp in this tree (queued), no GPU
run. Not covered: the Windows HIP / WDDM speed claim and NVIDIA (the gate is written down). Needs the author's numbers
repeated before any default talk.
--host :: failed before the model loaded because Server always opened an IPv4 socket. Server now selects AF_INET6 when the bind address contains a
colon; the ready line, the 'from other devices' lines and the browser address bracket an IPv6 address. The bind is still the person's choice and
no default changes. Ported from PR #3212 by purrsec with its tests (serve/test_security.py, including a real :: bind). setup.py's --host help and
docs/DETAILS.md say :: is all IPv6 interfaces; the README hunks of the PR (they name strata launch --host, not checked) are left out.
A reporter built Strata for two Quadro M6000 (sm_52) by lowering two
floors and saw about 22 tokens a second with Qwen3.8-Flash-Next IQ1.
CMakeLists.txt: the experimental build (-DSTRATA_EXPERIMENTAL_SM60=ON)
now accepts sm_52 and prints a warning for it (Maxwell, one reporter, no
native half precision, CUDA 12.x). Without the flag the build still
stops at sm_75 as before, and the architectures a build uses are not
changed.
setup.py: sm60_card() admits a 5.2 card only when the tester sets
STRATA_EXPERIMENTAL_SM52=1, because the ready-made CUDA 12 engine holds
no sm_52 code and a Maxwell card must not be routed to it by default.
With the variable set, setup warns about the Maxwell card instead of
refusing. Default: a Maxwell card is "not supported" exactly as before.
docs/OLDER_GPUS.md: a Maxwell row and a short section, with the build
command and a request for an answer-text check.
Tests: tools/test_setup_older_gpus.py (20 tests, new Maxwell cases for
default, opt-in and a newer card present). Not covered: no model run, and
the answer text on a Maxwell card is unchecked.
When HEAD and the one-byte range probe both gave no length, download() took an early EOF as a finished file even though the repository pin (or
ModelScope's published size) names the size. The published size now stands in for the length, so the existing resume, retry and parallel code
publishes the file only after it has all its bytes; a size that disagrees with the pin is still refused before the transfer and the SHA-256
check is unchanged. Ported from PR #2287 by KeyanHu-git with its tests (tools/test_setup_download_length.py).
install_own_path treated a file as present only when <file>.done existed, and that mark is written by setup's own download or by verify_sha256,
which a file from another tool (LM Studio) never reached: setup said it was missing and asked for the full download size in free disk. With
--gguf-dir an existing file and its mmproj go to verify_sha256(own=True), which checks size and SHA-256, leaves a mismatching file alone and
writes the mark. A file that is not in the folder is still reported missing. Ported from PR #3606 by ai-bits with its two tests.
Under WDDM cudaMemGetInfo answers for the calling process: memory another process holds on the card (the image encoder,
a browser, another engine) is not in its free figure. The reporter measured, on an RTX 5090, 30,994 MiB free from CUDA
where NVML read 24,348 with 6 GiB held elsewhere, so --expert-cache auto sized the cache too large and shrank it by
guesses.
device_free_bytes() now takes min(CUDA's figure, NVML's figure for the whole card) when STRATA_VRAM_FREE=device is set.
nvml.dll is loaded from the system directory at first use (LoadLibraryExW with LOAD_LIBRARY_SEARCH_SYSTEM32), the card
is matched by its PCI bus id, and a missing nvml.dll, a missing export or a failed nvmlInit leaves CUDA's figure with
one line on stderr. The serve check of "VRAM free with everything loaded" uses the same function. The value can only get
smaller, so the switch can only shrink the expert cache.
Unset (the default), nothing changes: the NVML code is compiled only for _WIN32 and not for HIP, Metal or the no-VMM
build, and the env variable is read once. The serve check's min() is the same value elsewhere (on a unified-memory
system device_free_bytes() is never below cudaMemGetInfo's figure). Added to the PR: the NVML block is also left out
of a SYCL build (SYCL_LANGUAGE_VERSION), whose CUDA names are a compatibility layer. docs/DETAILS.md: one row in the
measured-and-left-off table instead of the PR's long paragraph.
Tests: builds on Linux CUDA (the Windows-only block is skipped), SYCL (P1006) and on Windows MSVC (owner PC build of
the final tree). Not covered: no run with a model; the Windows gate is written down.
resolve_rope accepted nan, inf, zero and values below 1 and handed them to the engine. It now stops with a message that names --rope-scale and the
valid range, before any configuration is written; valid factors and the automatic scaling are unchanged. Ported from PR #3393 by j-luwierski
(follow-up to #2950) with its tests; the --help text gets the range.
tools/check_cuda_spills.py reads the resource usage the compiler wrote into the kernel libraries (cuobjdump
--dump-resource-usage: STACK and LOCAL per entry function and per sm_) and fails on any Strata kernel outside its ALLOWED
table, which carries the measured allowance of the kernels that hold such an array on purpose; ggml's vendored kernels
(no namespace) are skipped and counted. ctest cuda_spill_guard runs it on eight libraries, skips (77) without cuobjdump or
a demangler, CUDA builds only (HIP has no cuobjdump); check_cuda_spills_unit runs tools/test_check_cuda_spills.py
(stand-in demanglers are sh scripts, so not on Windows).
Changes against the PR: (1) a demangler that exits 0 but prints a blank name used to make the kernel's key empty, which
reads as vendored, so a spilling Strata kernel was skipped; that is now "no verdict" (77) with a test. (2) The unit test is
registered. (3) The ALLOWED table was re-measured on f-027-int: sm_120 with CUDA 13.0 on Windows (MSVC) and sm_86 with the
CUDA 13.0 objects built on Linux (cuobjdump 12.9); allowances are only raised (new DFlash2 gemm8_kernel and
rope_table_kernel, native_gu_swiglu_kernel on sm_86, dequant_kernel / expdec_gu_kernel / prompt_attn_i8v3_kernel /
native_gu_swiglu_kernel on sm_120), none lowered or dropped. Result: 0 violations on both architectures; a test shows the
same object fails once one allowance is lowered.
Ported from PR #2683 (the tracking issue is #2682).
A json_object or json_schema answer that holds LaTeX such as \alpha is
rejected (502) because \a, \g and the like are not JSON escapes. New
opt-in: "repair_latex_escapes": true in response_format (and in
text.format on /v1/responses). Only after the strict parse failed, the
server doubles each backslash that does not start a valid JSON escape and
parses once more with the same duplicate-key and NaN rules, then checks
the schema as before, and prints one log line. Without the option, and
for any answer that parses on the first try, nothing changes.
Escapes that are valid JSON (\b \f \n \r \t \/ \" \ and a well-formed
\uXXXX) are left alone, because they cannot be told from LaTeX: a \beta
next to a \alpha comes back with a backspace. Truncated output, arrays,
NaN and duplicate keys still fail.
Tests: three new tests in serve/test_structured.py (default still 502,
opt-in 200 and stream, failures stay failures, helper cases, Responses
format passthrough). serve/test_structured.py 13 tests OK.
With STRATA_DECODE_TIMING=1 the serial line listed `verify + commit/emit + draft` after `ms/window =`, while the adaptive
wait before verify and the join after the draft were timed but printed after the sum, so the named parts did not add up
to ms/window. The serial line now reads `adapt wait + verify + commit/emit + draft + adapt join + other`, where `other` is
the decode time minus the named parts (not a timed step). The pipelined decode keeps its old line, byte for byte, and
nothing is printed with the timing off or with no windows. Diagnostic only: no decode path changes.
docs/DETAILS.md: a short paragraph on the new line (the hunk was rebased by hand onto the current file).
Tests: compiles on CUDA (the format string was counted against its 22 arguments); no parser in the repo reads
`ms/window = verify`. Not covered: no run of the line itself (it needs a decode).
The quick speed check has its own GPU probe, separate from strata_cli/bench_facts.py, and tried nvidia-smi then rocm-smi only. On a Radeon with
amd-smi alone it wrote the server's vendor stub and 'GPU driver: not read'. The same amd-smi probe as #3519 is added in the shape this file
uses, and a bare vendor name from the server no longer replaces a card the probe named. Ported from PR #3630 by l33tm4st3r.
The three new tests are POSIX-only (symlinks, /sys); run on z440 (Linux, Python 3.12): OK. tools/test_strata_bench.py on Windows: OK.
A batch slot holds the reply it decoded. The main session was parked with the conversation's prompt only (the
admission's copy), so when the next admission or a yielding read overwrote the slot, the reply lived nowhere: the
conversation's next turn found its prompt-read state and read the whole reply again (the author measured 255 of a
256-token reply re-read on four 200K chats over two slots).
What changed:
- park_current is split into park_source (one budget / RAM-admission / snapshot path) and two callers: park_current
(the main session, output line and byte accounting as before) and the new park_slot (an idle slot's ids, checkpoints,
sessions and draft K/V). The ParkSource passes what a park captures; the main session's retained K/V (#1503
take_reuse_for) is taken only for the main session, never for a slot.
- park_slot runs before copy_to_slot at a slot admission and before a BYIELD read overwrites a slot. It does nothing
when the live main session or a parked conversation already holds that state (ConversationCache::holds /
conversation_covers, new, with unit tests), for a slot with pictures, or without --batch-mtp when the drafter is on.
- When a slot is admitted into the main session its checkpoints come along (the shorter ones too), so the chain keeps its
branch points and the parked copies a turn back are superseded (#342) instead of taking an entry each.
- Same refactor in sycl/src/program/generate.cpp.
Changed from the PR: parking is an optimization like a checkpoint (#3350). The PR ended the request (ERR) or the batch
(batch_fatal) when park_slot's snapshot failed; here a failed snapshot, a failed device sync and a host allocation
that runs out are logged, the slot is overwritten as before and the request goes on. The slot's extra checkpoints that
come along at an admission are copied under a bad_alloc guard (the one the request continues from is copied as before).
Tests: conversation_cache_test (new holds() cases and the four-conversation replay at policy level) and
conversation_split_failure_test. Not covered: no model run (the batch path itself needs one); the gate is written down.
The Responses validator still refused anything below 1 after the chat
path made max_tokens of -1 and 0 mean no cap (documented in
docs/DETAILS.md). It now accepts -1 and 0 as no cap; values below -1,
floats, booleans and strings stay a 400. Nothing else changes: the
server already passes max_output_tokens or 0 to prepare(), which gives
the answer the room left in the context.
Tests: OverHttp.test_max_output_tokens_zero_and_minus_one_mean_no_cap
(stream and non-stream, status completed) and
test_max_output_tokens_bad_values_are_400. The first fails without the
change. serve/test_responses.py 43 tests OK.
Problem: with STRATA_Q8_SM60=1 the main model's Q8_0 matrices run on the GP100
Q8_0 kernel, but the MTP drafter's own Q8_0 draft head and projections kept the
generic kernels.
Change: with STRATA_MTP_Q8_SM60=1 (on top of STRATA_Q8_SM60=1) the drafter packs
its Q8_0 projections as packed copies, as the main model does (the prompt path's
draft_kv still reads the GGUF layout, so a copy is kept), and packs the draft
head's subset in place when every column count fits the kernel's shared-memory
planes and the data is 16-byte aligned (otherwise a packed copy). The in-place
pack writes the GGUF blocks back if the device write fails. Release unregisters
the entry without a cudaFree when the data was packed in place. New
native_q8_0_sm60_pack_quiet and native_q8_0_sm60_pack_in_place; HIP returns
false; metal/src/engine/stubs_0145.cpp gets matching stubs that answer
"nothing packed".
Replies do not depend on the drafts (they are verified); only the draft rounding
changes, because the Pascal kernel sums in another order.
Default path: mtp_q8_sm60() is false unless the env is set, so the packing calls
are skipped. The destructor now walks the tensor list once at teardown, not in a
hot loop. The SYCL tree has its own sycl/src/core/mtp.cpp, so it is not touched.
Tests: compile-only on sm_60 (CUDA 12.9) on P1004. Model gate (identical replies
with the flag on and off, decode speed on a P100) is written down in
phaseA-tests/gh027/gates/3751-mtp-q8-sm60.sh.
Not covered: no GPU run here; the contributor measured +4.4% on four GP100 dies
with all experts resident, one run per arm, not on a single streamed P100.
#1337 made a failed engine start in a later calibration step a candidate
that loses. An engine that started and then exited during its own rate
measurement still failed the whole run. measure_candidate() now treats
both the same: the candidate is dropped and recorded in
report["failed_starts"], the earlier results stay. It is used for the CPU
worker step, the adaptive expert tier step and the AMD shared-expert
fork. An operator cancel (CalibrationCancelled) is still raised. The first
start (steps 1 to 3) still fails the run.
src/core/verify.cpp: when adapt_gate_wait times out after 20 s, the
message now says which layer was reached (atomic and copy signal), that
the copy stream is not drained by the release, and dumps the
STRATA_VERIFY_TRACE breadcrumbs when that is on. Diagnostics only, #2188
stays open. The watchdog label is set only once a wait actually starts,
so a gate that is already open costs nothing extra (changed from the
PR, which set it on every call).
Tests: tools/test_calibrate.py 33 tests OK (3 new death tests).
Not covered: engine compile of verify.cpp is checked in the build step.
Problem: on a GP100 die (cc 6.0, no dp4a, no tensor cores) the verify window's
routed down projection for the native packs' IQ4_NL and Q2_0 rows (n_ff 640)
is latency bound. native_down_multi_kernel reads each weight as 2-byte loads,
one item at a time, straight from global memory.
Change: native_down_ws_kernel copies a block's rows (IQ4_NL 32, Q2_0 64, 11,520
bytes) into shared memory with 16-byte loads, then runs the same row dots on the
copy: same sub-warp, lane, items, float expressions and order. Only the integer
sum inside an item changes form (vmad byte selectors, prmt table lookups), the
same int8 x int8 products, so every output is native_down_multi_kernel's (expert
mode 0) bit for bit.
Gating: compiled only in the CUDA build with STRATA_EXPERIMENTAL_SM60 (not HIP,
not Metal). At run time only with STRATA_DOWN_WS=1 on a device of compute
capability exactly 6.0, IQ4_NL / Q2_0 down rows at n_ff 640, and not with
STRATA_OLD_IQ_MMVQ or STRATA_NO_SUB16_GU. Unset, the only change on the default
path is one cached boolean test per native_expert_grouped call.
docs/OLDER_GPUS.md lists the switch. The contributor's microbench numbers (one
PH402 die, Swift 1.5 IQ3_S, one-token window: IQ4_NL 72.6 us mode 8 and 54.5 us
mode 0 become 42.3 us; Q2_0 53.5 / 65.4 us become 38.6 us) were measured on a
four-GP100 rig, not here.
Tests: compile-only on sm_60 (CUDA 12.9) on P1004. The contributor's bitwise
check (native_expert_bench hash against modes 0 and 8) is not run here: it needs
a GP100 GPU run and the model's layers. The gate script is
phaseA-tests/gh027/gates/3867-down-ws.sh.
Not covered: no end-to-end measurement on a single P100 (16 GB cannot hold the
47 GB pack resident).
ROCm 7 and newer ship amd-smi and no rocm-smi, and the amdgpu product_name is empty on gfx1201, so a Radeon AI PRO R9700 run was filed
under 'unknown' with 'runtime: not measured'. bench_facts reads the market name from amd-smi static --asic, the VRAM from the amdgpu sysfs
file of the PCI address amd-smi list reports, the ROCm version from the install (or amd-smi -h), and keeps the probe's card name when the
server only says 'AMD Radeon'. Ported from PR #3519 by l33tm4st3r with its tests and the BENCHMARK_PROTOCOL.md text.
Changed while porting: the amd-smi branch also took macOS (os.name is posix there), which would have hidden the Apple chip; it now skips
darwin, and a test keeps the chip.
The "This computer" card said only "Fits fully in memory." for DeepSeek, with no word of the ~265 GB of disk. For a DeepSeek version it now says "Needs about {N} GB
of free disk space. What does not fit in memory is read from your SSD." (the spill line stays beside it when part of the model is read from the SSD), and the
selected version's line reads "{N} GB RAM + VRAM, plus about {disk} GB of disk". {disk} is not hard-coded: the catalog (tools/strata_mcp.py) gives each set
its `disk_gb`, the set's whole download (model files, n-gram table, drafter, metadata) plus the pack and room to work (dsv41_fit.DISK_HEADROOM_GB), so 2.00 and
2.25 differ. Same card and row markup, no design change; every other model keeps its card. Both sentences are fx80 and fx81 in the eight fix catalogs.
The Terminal's DeepSeek description makes no "fits fully in memory" claim (its lines already state the download and the free disk), so it is unchanged.
Older converters write the Engram scalars as engram.key_length and
engram.pad_id, and omit the sparse-attention geometry. The packer now
reads the old names only when the canonical engram.head_dim and
engram.pad_token_id are absent, and a new --config-json option fills the
omitted geometry from the source model's config.json. A value that
disagrees with metadata already in the GGUF stops the pack with a clear
message. The weights are never rewritten and a set that already has the
canonical names packs exactly as before.
Tests: LegacyModelMetadata (2 new, 5 in the file) in
tests/terminal/test_pack_gguf_engram.py.
Not covered: the author reports the older Q2_K set still needs dense
Q2/Q3 and Engram Q2_K tensor support in the engine; this does not add it.
The final response.completed / response.incomplete event and a
non-streaming /v1/responses reply now carry the same top-level timings
object that /v1/chat/completions has. The server already put timings in
the done dict; the Responses assembler dropped them.
Tests: UsageAndStatus.test_responses (stream and non-stream).
Unset timings leave the event unchanged; response.failed has none.
test_iq_pack imports without llama.cpp's gguf-py and skips the classes that build GGUF fixtures with a named reason; the MCP discovery tests use
a free port, not whatever Strata serves on 8080; the six conversation-cache tests add the repo root to sys.path so they run from the repo root
as the suite does. Ported from PR #3236 by Buddychek (tests only, no shipped code).
The 0.2.5 rewrite of the page dropped the 'Windows AMD: the driver resets' entry (v0.2.4.2 had it), although setup.py still tells a Windows gfx12
card owner to read it and tools/test_setup_amd.py (TdrPointer) checks for it, so that test failed on main. Restored from v0.2.4.2 with the
log and config locations worded for a source checkout.
Three 0.1.42 changes no page told. Ported by hand from PR #3244 by Buddychek (the PR conflicted with the rewritten pages), each sentence
checked against the code: /metrics/prometheus needs the API key like /metrics (serve/server.py _authorized), the prompt-speed row is
(prompt_tokens - reused) / prompt_ms (apps/host/server/monitor-dashboard.mjs), STRATA_VCVARS is read by setup.find_vcvars. The compiler
text now says Visual Studio 2022 (2019 is refused since 0.2.5, #3566) and went to docs/SOURCE_INSTALL.md.
conversation_detail::private_stream() and order_after_queued() kept one non-blocking stream and one event per device in
thread_local storage and never destroyed them. The layer split's later stages run each prompt chunk on a fresh
std::async thread (STRATA_HANDOFF_PIPE off, the default), and a checkpoint part is saved on that thread, so every
periodic checkpoint of every long cold prompt left a stream and an event behind on each later card: a few MiB of VRAM
per long prompt that never came back (#3470: the later card's free VRAM drops 2-6 MiB per long cold prompt, no plateau),
until cudaStreamCreate failed with "out of memory" (#3350: "conversation snapshot running-state copy: out of memory",
engine exit) on a card that was already nearly full.
A thread now hands its handles to a process-wide pool when it ends and the next thread takes them, so the number of
streams and events is the number of threads copying at once. Same stream, same flags, same contract; only the lifetime
changed. copy_streams_created() / copy_events_created() count what the process made.
conversation_copy_streams_test (CUDA, no model, skips without a device): 200 short-lived threads in a row create at most
one stream and one event; ten rounds of six concurrent threads at most six more; the same on a second device.
A conversation checkpoint is an optimization, like parking, but the writer treated a failed copy as fatal: "conversation
snapshot running-state copy: out of memory" (cudaErrorMemoryAllocation from the snapshot copy: the private copy stream's
first creation or the driver's staging on a nearly full card, not host RAM) made the prefill fail and the engine exit,
21 times in 4 h for one reporter and 16 in 34 h for another, each exit a reload and the in-flight requests lost.
Now, after a failed save, checkpoint_failure_is_survivable() clears the non-sticky allocation error and synchronizes the
device: when that is clean the checkpoint is skipped with a line on stderr and the request goes on (a layer split's stage
part is left empty, which already means "no checkpoint at this length"; the message-boundary / turn / root checkpoints
after a prompt chunk do the same). A device that is unwell (a sticky error, a GPU that stopped answering) still ends the
request exactly as before. Same edits in sycl/src/program/generate.cpp.
STRATA_TEST_COPY_FAIL_AT=N (tests only, like STRATA_TEST_SLOW_COPY_US): the N-th snapshot copy of the process fails with
cudaErrorMemoryAllocation, so the case can be made without a full card. Unset, nothing changes.
Not changed: "verify: instantiate: out of memory" (graph capture), which the reporter saw once.
CMakeLists.txt tested "NOT DEFINED CMAKE_CUDA_ARCHITECTURES" after enable_language(CUDA). Since CMake 3.18 that call
fills an unset variable with nvcc's default (sm_75 for CUDA 13), so the sm_120 default never applied: a hand build with
no option was an sm_75 build. The default is now set before enable_language, unless the user gave
-DCMAKE_CUDA_ARCHITECTURES or the CUDAARCHS environment variable, and the configure step prints a warning that says what
to pass for another card (a recommendation, not a refusal).
With -DCMAKE_CUDA_ARCHITECTURES=native, the one NVFP4 W4A4 unit (mmq_nvfp4_w4a4.cu) stayed on plain `native`, where
ptxas refuses the FP4 MMA on an RTX 50. `native` is now replaced by CMAKE_CUDA_ARCHITECTURES_NATIVE (the detected
architectures) before the 12x -> 12xa rewrite, as a list, so a mixed PC works too. docs/SOURCE_INSTALL.md says what
`native` and the other values build.
Release and setup builds pass their architectures explicitly and are unchanged. Configure matrix, CUDA 12.9 on Linux:
none -> 120 (+ the warning), CUDAARCHS=86 -> 86, 89, "75;86;120" (w4a4 120a only for the 12x entry), native -> detected.
Ported from PR #4029.
--adapt-decay took any number. Above 1 the adaptive tier's usage counts grow without end (inf after a few hundred
adaptations) and the swap gains turn NaN, which breaks the sort that orders the swaps; a negative value flips the counts.
The value is now checked where it is parsed (include/strata/program/adapt_decay.hpp, also in sycl/): a finite value in 0..1
is used exactly as before (0 and 1 included, the range the calibration profile already allows); anything else, NaN and inf
too, runs with the default 0.7 and says so on stderr. A warning instead of the PR's exit 2: a risky pick is reported,
the engine still starts.
adapt_decay_test (25 checks) covers the range, the fallback and finite counts after 100000 adaptations.
Ported from PR #1284 (the check; the refusal became a fallback).
The sandbox of a test start was deleted at once, so the PID records that show why a start failed were lost; only the last 64 KiB of the engine
log stayed. A failed start now also keeps run.json (without keys), the host, child and ready JSON records of the runtime (tokens cut to their
first 8 characters) and the host and server logs in .updates/logs/test-start-<version>-failed. The last three failed starts are kept.
Best effort, never hides the failure. Tests: tests/terminal/test_updater.py (FailedStartRecords).
The manual build in docs/INSTALL.md uses CMake's default generator (Unix Makefiles on Linux) in build/, and setup --build always asks for
Ninja, so cmake stopped with 'Does not match the generator used previously'. cmake_build now reads CMAKE_GENERATOR from the folder's
CMakeCache.txt; when it is not Ninja it says so in one line and removes the cache and CMakeFiles (no other file). A Ninja folder or a new
folder is untouched. Tests: tools/test_setup_build_dir.py.
A cmake -P script has no cmake_minimum_required, so CMake 3.x used the OLD behaviour and stopped at 'if given arguments ... IN_LIST: Unknown
arguments' (ctest ggml_tq_overlay with CMake 3.31). Checked with pip cmake 3.31.6 and 4.4.3 against the pinned llama.cpp: both print
'ggml_tq_overlay_test: ok'. A static test keeps the policy before the first IN_LIST in every cmake/*_test.cmake.
The script ended with a grep pipeline, so a failed compile exited 0 (and, before the ocloc check was added at the end, a good build exited 1).
A build.sh && next-step chain then ran after a failure. The status of cmake --build is kept and the script exits with it. The test runs the real
script with a stub cmake on PATH (good build: 0, failed compile: 1; the old script exits 0 on the failed compile).
tools/test_setup_golden normalized strata.exe but not strata-vision.exe, so all 25 baseline entries failed on Linux (48 subtests). VEXE
and HVEXE are mapped to <VEXE> and <HVEXE> like EXE, and the baseline records <VEXE>. test_nvfp4_local_pack imports pytest, which is not in
requirements.txt; under python -m unittest discover the module is now skipped with a reason instead of failing to import.
Substitutions were ordered by the length of their regular expression, and a user name wrapped in word bounds outranked the Strata or home
folder that contains it, so a report kept /home/<user>/Strata instead of <strata>. They are now ordered by the text they replace. Ported from
PR #3619 by ctr-137 with its test (Linux and Windows layouts).
The page said that starting a model never downloads and that nothing checks for updates in the background. Since 0.2.4.2 strata launch with a
DeepSeek model checks the signed feed (10 s at most, silent without a connection) and replaces an older installed DeepSeek engine package.
Ported from PR #4028 by pschaub (wording from issue #4022), checked against strata_cli/launch/runner.py upgrade_dsv41_engine.
engine_env put the ROCm library folders first on PATH, which only Windows' loader reads. On AMD Linux strata --families exited 127 on
libhipblas.so.3, so setup falsely said the engine does not run Qwen3.6. On Linux and elsewhere the folders now go first on LD_LIBRARY_PATH
(an existing value is kept); Windows keeps PATH. Ported from PR #3305 by xxxliqu1dxxx (resubmission of #2914), with its tests.
metal/setup_mac.py returned right after the engine step on --update while the model questions had been asked, so choosing a model the Mac did
not have ended with no download and no word. The update now names the missing model (or image encoder) and points at ./setup.sh --setup; an
update for a model that is installed prints what it printed before. Ported from PR #3848 by demetree, with its two tests (mocked; not run on a Mac).
adopt_user_defaults() set the token's default DACL with FA (FILE_ALL_ACCESS). That default also applies to new process objects, and the file
mask has no process query or set-information right, so a child could not be reopened by PID: the HighQoS helper timed out and
procs.start_token could not record the child. GA lets Windows map the mask to the object being created; the principals are unchanged.
Ported from PR #3428 by kossum (QoS reopen test). Test from PR #3399 by midhatn (a live child is identified and recorded) added; both fail
without the change and pass with it.
Co-authored-by: MIDHAT NASHAR <[email protected]>
_start() checked cfg['exe'] with Path.exists(), so a run config whose engine is component:strata-engine was refused as a missing file
before the server could resolve and verify it. Ported from PR #4027 by jubeater. A component reference is no longer tested as a file; model
files and literal engine paths are checked as before. ensure_engine_for, the Windows HIP runtime repair and the HIP architecture lookup
skip the source-build engine folder for a component. Eight tests (tools/test_setup_component_start.py).
Ported from PR #4011 by SuperDistance. project_version() read CMakeLists.txt with Path.read_text() and no encoding, which raised
UnicodeDecodeError on Windows code pages such as cp936. The test makes read_text default to each of 14 Windows code pages. The same crash was
reported in #3235 and #3414.
.git is not in the build context, so package_terminal.py had no commit to put in the archive and the documented source build stopped.
Ported from PR #3644 by robertoetcheverryr: ARG STRATA_COMMIT / ENV STRATA_BUILD_COMMIT as in docker/Dockerfile, and the two 'From source'
commands in docs/DOCKER.md pass --build-arg STRATA_COMMIT=$(git rev-parse HEAD).
Added: .dockerignore leaves out the folders docker/entrypoint.sh creates in a mounted data/ (install, state, home, Strata-data). A dangling
data/install/third_party/llama.cpp link in a clone that had been mounted made the Terminal stage fail with a copytree error (seen in the
review thread of #3644). Not built with docker here (no docker on this PC).
docker/build.sh staged only top-level tools/*.py, so the Terminal stage stopped at 'installer tool missing: tools/dsv41/tune.py'.
Ported from PR #3974 by hurui200320: the pathspec is tools/**/*.py, the no-.git fallback finds nested files, and the context test reads
INSTALLER_TOOLS and INSTALLER_SCRIPTS from packaging/package_terminal.py. Fixes the failure that images built from v0.2.4.1 and v0.2.4.2 had.
The portable lifeline watcher built a child's identity from `ps -o lstart=,stat=` and kept the whole line, so the token
changed with the process state (R while it starts, S asleep). The watcher compared its own reading with the token taken
at start, decided it was not the child it was told about and exited without ending it (test_lifeline "the child outlived
a killed server (watcher)", #3407). The identity is now the start time alone: field 22 of /proc/<pid>/stat on Linux
(a zombie has none), `ps -o lstart=` elsewhere; strata_control/procs.start_token drops the state column the same way.
A launch that is not a real Popen is not watched.
Telemetry._loop no longer ends on the first failing sample (a machine whose GPU or psutil source is unreadable): it
prints one line and keeps sampling, and NoLeakedSampler checks the thread itself instead of counting threads.
New test: the token is the same while the child runs (R) and sleeps (S); on the old code it fails on Linux
('...R' != '...S').
Ported from PR #3590.
A declared namespace.member tool (Codex MCP: mcp__filesystem.write_file) was accepted only as namespace__member (#3556).
Qwen3.8-Flash-Next also writes namespace=member (the <function= tag's equals sign ends up in the name), namespace_member
(one underscore dropped) and the bare member. Each is now an alias of the declared tool when exactly one declared tool
produces it and no declared name is the same; a name that matches nothing stays text (#3075), two namespaces that share
a member are not guessed, and a declared flat tool always wins. Requests without namespaced tools are unchanged.
serve/test_tool_name_alias.py (10 tests, 5 of the first 9 fail without the change; the tenth pins that a flat tool named
like the underscore alias wins), docs/DETAILS.md names the spellings.
Ported from PR #3972.
With --batch and conversation parking, a restore keeps the restored K/V (ConversationCache::retain) so the next park of
the main session copies only what changed. The reuse tracked a token count (unchanged_tokens), not whose tokens, and
copy_from_slot (a batch slot giving its conversation back to the main session) replaced the main session's K/V without
dropping the retained image. The next park reused the earlier conversation's K/V for the new one's prefix: a restore of
the new conversation then answered from the other one's history.
retain() now records the ids of the conversation the K/V belongs to; a park calls take_reuse_for(live), which reuses only
the prefix live shares with those ids (nothing when they share none or the ids are unknown), and one stderr line is
printed whenever a reuse is cut. copy_from_slot drops the retained image before it replaces the main session's K/V.
Same edits in sycl/src/program/generate.cpp. The guard only cuts reuse: at most one full copy of a parked conversation's
K/V, the saved bytes are otherwise identical; no flag, no default change.
Ported from PR #1503 onto the current tree (the first-stage ids sit after the stages field so the existing aggregate
initialisers keep their meaning). conversation_cache_test gains the cross-conversation cases of the PR and the logged
sequence (restore P, a slot hands Q to the main session, park Q): another conversation never gets the K/V, a diverging
one only the shared prefix on every stage, a slot hand-back drops it, unknown identity is never reused. Removing the
ids limit makes the first case fail.
A command on the allowlist used to run without a card with any option except -delete/-exec/-execdir/-fprint/-fls/-ok. Write modes (sort -o,
tree -o, uniq IN OUT), program-execution modes (sort --compress-program, rg --pre, git grep -O), date -s and tail -f got through, and so did
git config -e, git branch --set-upstream-to and git remote ... forms.
Each command now has a spec of the options and operands that only look (short and long, spelled in full); every other option, abbreviation
or attached form asks. git subcommands have the same specs, a branch/tag name is taken only with a list option, git config only reads with
--get. date takes only +FORMAT operands, uniq at most one operand, hostname none. less and more leave the list. Quoted texts are checked as
operands too. `du -sh` no longer trips the 'sh' word check.
Tests: apps/host/tests/gate-readonly.test.mjs (deny and allow cases for options, operands, spellings, quoting; every listed command asks for
an unknown option).
sample() applied the repeat / frequency / presence penalties to lg[id] for every token id of the last penalty_last_n
tokens of the history. A row shorter than the vocabulary (the MTP draft head picks among the first N token ids,
--draft-vocab N) made a history token with id >= N an out-of-bounds read and write on the host. A token that is not in
the row takes no penalty; the loop skips ids < 0 or >= lg.size(). A full-vocabulary row is bit-identical.
The loop and Sampling move to include/strata/qwen35/sampling.hpp (header-only) so a CPU unit test can run them:
src/qwen35/sampling_test.cpp (qwen35_sampling_test) checks nothing is written past a short row, that in-row tokens
take the penalty, and that a full row matches the unguarded loop bit for bit over 108 penalty/window mixes. Without the
guard the test dies with an access violation.
Co-authored-by: yehya <[email protected]>
cache_settle()'s store preload (DS4_SETTLE_STORE) read 2,243 hot experts into free store slots after every chunked prompt: 4,713 ms of a 4K prompt. Measured on the box (code fixture, prompt + 256 tokens, interleaved, 3 runs a side): with it on the decode is 13.9/14.4 s, with it off 14.2 s - no benefit - while the 4K prompt goes 31.3 -> 26.6 s (130.89 -> 153.78 tok/s). The swap loop above it is what the decode uses (it holds the CPU lane at ~5-7 k experts a run against ~14 k without), so that keeps its budget; DS4_SETTLE_STORE=1 restores the preload.
On the RTX 5070 (--agent-long, 22 tasks) split 36 kept agents at 6/10 (exact 4 and 7) and extraction at 585/606 facts (exact
585) for +11% prompt speed at 14K and +18% at 30K; the split-24 default passed 0/10 agent runs and found 556 facts, a tail of
8192 with split 24 found 574. KVA stays off unless STRATA_KVA=1. docs/KVA.md gives the numbers and the reasons; kva_test
checks the opt-in's split 36 / tail 2048 and the overrides. The geometry gate is unchanged.
The app toggle will write STRATA_KVA=1 for Qwen models in general; the split layer, the tail and the measured quality apply to
Qwen3.8-Flash-Next's shape only (48 layers, 512 experts). Any other shape (Qwen3.6-35B-A3B, ...) now logs "KVA configured but
NOT applied: this model's shape (L layers, E experts) is not one KVA was measured on" and reads exactly; the routine reasons
(short prompt, continuation) still come first. STRATA_KVA_ANY_MODEL=1 allows it, with one warning. kva_test and docs/KVA.md
cover both.
KVA was measured on the Flash-Next shape (48 layers, 512 experts) and the engine declines any other shape with a log reason, so the row would do nothing
for Qwen3.6-35B-A3B. Setup writes "arch" in a run config only for the families with an engine path of their own (Qwen3.6-35B-A3B: qwen35moe; DeepSeek:
deepseek41); every Qwen3.8-Flash-Next file (the original, Swift, Coder, Unsloth, Gyro, NVFP4) has none. The row is therefore shown for a model without an
"arch" on CUDA or HIP, in the app's Native engine settings and in the Terminal's Advanced table. The Radeon RX 9000 and DeepSeek rows are unchanged.
Tests: a Qwen3.6 config gets no KVA row (CUDA and HIP), the six Flash-Next family names do, and through the host fixture a model with an arch has no switch row.
The quality runs found lost records and invented rows in long extraction tasks and defects in agent loops with a ~10K
system prompt when most of the prompt was approximated. Opt-in now: unset (or 0) nothing runs, the draft layer included.
Switched on, the tail is 8192 tokens (STRATA_KVA_SUFFIX overrides), the safest configuration measured. A model other than
Qwen3.8-Flash-Next's geometry gets a one-line warning instead of a refusal. Kept: the untouched draft layer (cell floor
behind STRATA_KVA_DRAFT_FLOOR=1), the decline reasons, STRATA_KVA_SKIP=moe, the tests. docs/KVA.md says why it is off and
how to turn it on; kva_test checks unset = off and STRATA_KVA=1 = on with the 8192 tail.
The Speed row now follows the shared table's default: Faster long prompts reads "Off (recommended)" / "On" and its new sentence is "Reads very long prompts
faster by approximating part of the work. Off by default: answers about long documents can miss or mix up details." (replaces fx73; "Off (recommended)"
is fx79; translated in zh, ja, de, es, ru, pt, ko). The other two rows are unchanged ("On (recommended)" / "Off"). A STRATA_KVA=0 stored by an earlier
0.2.7 test build shows as Off. Still shown only where KVA applies (Qwen on CUDA or HIP). tests/speed-switches.test.mjs updated: the default launch has
no KVA entry, On writes STRATA_KVA=1, Off removes it, the old 0 shows as Off, and the other two switches are tested as before.
Owner decision (26 October, conditional on the last KVA variant results): KVA ships off. strata_control/speed_switches.py now gives every switch a default
and the environment it writes outside that default: KVA is default off and writes STRATA_KVA=1; the Radeon RX 9000 and DeepSeek switches stay on by
default and write their off entries as before. A launch in the default carries no entry. A file that holds STRATA_KVA=0 (the off switch of an earlier
0.2.7 test build) reads as off, which is what it is now; choosing Off cleans it up.
The Advanced table row says "off by default; set 1 to turn on" with the new sentence (`--set STRATA_KVA=1`; `off` or Del puts the default back).
tests/terminal/test_speed_switches.py covers the defaults, KVA on/off/old-zero, the rows and the comparison with the app's table.
The default stays the state-only form (byte-identical). With STRATA_KVA_SKIP=moe the GDN recurrence (gate projection, output
normalization, output projection) and the QSA attention (query side, selection, attention, output projection) run in full
for layers >= split, their outputs go into the residual, and the loop leaves before the MoE half; the expert transfers of
those layers are still skipped. The KVA line says "mode moe-skip". Motivation: the extraction losses of the state-only form
come from late-layer K/V and state built from a stale residual; this keeps the attention halves' contribution in it.
kva_test covers the switch.
fast_attn_enabled() was opt-in (DS4_FAST_ATTN=1); with the prompt chunk at 1024 it is worth far more than when
it was gated: measured on 4x R9700, 2.25 bpw, 8K code prompts, three warm prompts a load, the same ids
(5705e1164a) --
without it 1159 / 1180 / 1195 ppts (median 1180)
with it 1327 / 1354 / 1360 ppts (median 1354, +14.7%)
and after this commit the un-enved default reads 1314 / 1357 / 1369 (median 1357), i.e. identical to setting the
flag, so a deployment no longer needs any environment at all. The QK/PV sums are the matrix cores' order, which
the owner already approved for the prefill; the decode path keeps the scalar kernels. DS4_FAST_ATTN=0 turns it
off (it is a no-op on CUDA, whose dispatch is not compiled).
ring_mb_auto() returned a flat 400 MB on any box whose experts do not fit in VRAM, but on such a box the ring is exactly what lets a prompt queue its likely experts early. 400 -> 1200 MB measured on the box (settle off): the queue 3,200 -> 8,087 experts, the disk's effective rate 1.49 -> 2.44 GB/s, the compute's exposed read wait 11.4 -> 3.5 s, the 4K prompt 148.98 -> 190.98 tok/s; 16K 290.97 -> 315.01, 32K 316.92 -> 325.89. Decode unmoved in an interleaved A/B (17.54/16.39 against 17.61 tok/s). It is now a sixteenth of the VRAM room clamped to 400-1200 MB, so a smaller pool still loads; DS4_RING_MB still overrides.
c.A: the attention slab decides how many times each dense matrix is re-read a prompt. 8 -> 32 MB (STRATA_DS4_A_MB): 32K 325.89 -> 357.78 tok/s, 16K 315.01 -> 328.04, 8K +1.6%, 4K flat, decode unmoved (16.81 against 17.60 tok/s interleaved, noise-limited) for ~0.4 GB of VRAM.
Best measured prompt set with the two new defaults (DS4_SETTLE_MS=0;DS4_SETTLE_STORE=0, which is a prompt-only trade): 4K 195.18, 8K 258.47, 16K 328.04, 32K 357.78 tok/s. With the settle work at its default a 4K prompt is 31.3 s = 130.89 tok/s (was 88.30).
The dense GEMM was 11.1 s of a 4K prompt at hipBLAS's measured 4.97 TFLOPS (M 4096 N 2304 K 5120, the engine's call convention). A blocked WMMA kernel reaches 23.96 TFLOPS at that shape (128x128 tile, BK 32, 2-stage LDS, rocwmma; scripts/wmma2.cu) and the engine's own tuned kernel (BM 128 x BN 256, grouped tile order, raw wmma intrinsics) 20.8-20.9 at the wq_b shape and 38-41 at M 16384. gemm() now calls strata_pf_gemm_f16_ld with hipBLAS as the fallback (rows < 64, N < 512, K % 32, unaligned X: unchanged); STRATA_DENSE_PFGEMM=0 restores hipBLAS for an A/B. Same operands, the kernel's own k order (rounding-level).
Measured on the box (eng-028, 4E224E7F -> 467D7320, 4K prompt fixture p4k, chunk 16384): 46.4 -> 38.1 s, 88.30 -> 107.63 tok/s; with DS4_SETTLE_MS=0;DS4_SETTLE_STORE=0 27.5 s = 148.98 tok/s. 8K 134.66 -> 218.31, 16K 158.6 -> 290.97, 32K 178.9 -> 316.92 (settle off). Decode unchanged in an interleaved A/B (old 15.17/16.73, new 17.38/18.48 tok/s) with acceptance 151/164 = 92% (93/98/100/100/90 per position).
c.A (the attention slab, the reason each dense matrix is re-read 10-16x a prompt) takes STRATA_DS4_A_MB: 32 gives 107.63 -> 110.67, small because the expert reads now dominate.
The Models screen listed the shared sizes for DeepSeek but installed and loaded it at 8K whatever was chosen. Now the choice counts (8K stays the
default for a version that is not installed) and the list is DeepSeek's own: 8K to the limit the installed pack states in ds4.txt (max_position_embeddings,
read by the host as `contextMax`); a pack without it behaves as before (up to 256K). Past 256K the existing note slot under the model shows: "Not tested yet.
A context this long takes graphics memory from the model and reads long prompts slowly." (fx78 in the eight fix catalogs). Qwen's choices are unchanged.
src/lib/deepseek-context.ts is the app's copy of strata_control/dsv41_context.py; tests/terminal/test_dsv41_context.py compares the two.
The model reads 1,048,576 positions natively. Setup and `strata launch` offered DeepSeek the shared list that stops at 512K and dressed anything past
256K in Qwen's "rope scaling" words. Now:
- strata_control/dsv41_context.py: the choices are 8K, 32K, 64K, 128K, 256K, 512K and 1M (labelled "512K" and "1M"), limited to the pack's own limit
(max_position_embeddings in its ds4.txt) when it has one; a size past it (2M, or 1M on a pack that says 256K) is the one refusal; past 256K a plain
note says it is not tested yet and what it costs.
- setup.py: --context is checked before any download and again against the pack once it is built; the install tool refuses the same sizes.
- Terminal: the context step lists DeepSeek's own sizes (and the note beside them), a typed size is checked, `--context 512k` / `1m` and 524288 / 1048576
are accepted, plan and launch carry the note as a warning; Qwen's list and wording are unchanged.
- Engine: read, no change needed. strata-dsv41 takes --max-context as given (ds4_main.cpp) and sizes its buffers from it (ds4_model.cu); nothing clamps it
below the pack's limit, and the rope settings come from the pack. A test keeps it that way.
- docs/DEEPSEEK41.md: the cost (graphics memory per position taken from the expert cache, slow reading of very long prompts) and that 512K / 1M are untested.
Measured on a 5070 (IQ3_S) the cell floor cut decode after a 14K / 30K approximated prompt by 22 / 34 percent (drafts
accepted 100 -> 83 / 73 percent): a draft layer that reads only the exact tail loses its context. The approximated
chunks' draft cells are usable, so by default nothing about the draft layer changes: not the window, not the cells
built, not the attention. The floor machinery (MtpDrafter::set_cell_floor, window_ids_floor, KvaFloorRegistry) stays
behind STRATA_KVA_DRAFT_FLOOR=1 for research. kva_test now really covers window_start with floor 0, the registry and
wants_cell_floor.
The first default-on build set --mtp-window 2048 once per process whenever KVA was on, so short prompts (where KVA never
applies) lost draft context and decode speed. Now the draft layer keeps its window (32768 by default) and
MtpDrafter::set_cell_floor() names the first draft cell built from a real row: a KVA read raises it to the end of its
approximated chunks (they are neither built nor read), the draft attention starts at max(floor, pos - window)
(window_ids_floor, window_start.hpp), and KvaFloorRegistry gives every state a request continues from (live, checkpoint,
parked) the floor of its own tokens. A session KVA did not approximate has floor 0 and attends exactly as with
STRATA_KVA=0; with STRATA_KVA=0 or a process KVA is ruled out for the draft layer's code is 0.2.6's. The tail is 2048
whatever the window. Unit tests: window_start with floor 0 equals the plain window, the registry, wants_cell_floor.
- locale-gaps: the sentence added in 72c37ea2d (RTX 30/40/50, R9700, Strix Halo; "the line below says what to expect") had no key. fx77 in the eight fix catalogs.
- packaged-control: findControl was given an env without STRATA_COMPONENTS_DIR, so it fell back to the real per-user components folder and found the
updated Control installed on the PC (read only). The test now passes a scratch components folder.
Three rows under "Speed" in the native settings, in the existing row markup with the existing Select (On (recommended) / Off): Faster long prompts
(Qwen on CUDA or HIP; off writes STRATA_KVA=0), Faster answers on Radeon RX 9000 / AI PRO R9700 (Qwen on a gfx1200/gfx1201 card; off writes
STRATA_PCIE_FRAC_DEFAULT=old) and DeepSeek speed-ups (DeepSeek; off writes the six DS4_* = 0 of the contract). Each has a plain sentence saying what
turning it off does. A row is shown only where it applies, like the DeepSeek Speed row; the host decides from the model's file and the hardware report.
The switch is the model's file "env": off adds exactly its entries, on removes them, so a default launch is byte for byte what 0.2.6 started and the
Terminal, Control and the app reach the engine the same way (serve/server.py child_env). The host now shows and edits STRATA_* and DS4_* names (the
Strix Halo async entry is visible and kept); other names stay untouched. New strings are fx70..fx76 in the fix catalogs, translated in zh, ja, de, es,
ru, pt, ko. tests/speed-switches.test.mjs covers the default, each switch off and on again (same file), where each applies, and the catalogs.
strata_control/speed_switches.py holds the engine switch contract (Faster long prompts: STRATA_KVA=0; Radeon RX 9000 / R9700: STRATA_PCIE_FRAC_DEFAULT=old;
DeepSeek speed-ups: six DS4_* = 0), where each applies (Qwen on CUDA/HIP; Qwen on a gfx1200/gfx1201 card; DeepSeek) and the one-sentence effect text.
`strata launch` shows the environment names of the switches that apply as rows of the Advanced table (default "on", the sentence beside them); typing the
off value (`--set STRATA_KVA=0`) writes it into the run config's env, `off` takes it out again. A default launch carries none of them.
tests/terminal/test_speed_switches.py checks the names, the applicability, the config a launch writes, the rows, and that the app's table
(apps/host/server/native-settings.mjs) says the same.
The tail follows the draft layer: with KVA on and no --mtp-window given, the draft window becomes 2048 cells and the
fully computed tail 2112 (a given --mtp-window N sets the tail N + 64; a window of every cell, or STRATA_KVA_SUFFIX, is
handled and named). Layer splits, batch slots, --peer-device, pictures, the DFlash2 drafter, other model geometries and
the Metal build read exactly, with the existing "NOT applied: reason" line (quiet for continuation and short reads unless
STRATA_KVA=1). The reasons live in KvaConfig::decline and are unit tested (kva_test). Later chat turns continue from the
live session or a checkpoint and read their new tokens exactly over the cells the first read left; docs/KVA.md says so.
The slab rule (8 MB of q/o per device, 256 rows here) was set to keep the dense dequant's VRAM small; with a
1024-token chunk it dequantizes q_b/wo once per 256 rows - four times a chunk, 32 times an 8K prompt.
Measured on 4x R9700, 2.25 bpw, share 0.77, chunk 1024, 8K code prompt, ids 5705e1164a:
256-row rule 1204 / 1236 / 1250 ppts, decode 77.7 tok/s
chunk's 1024 1324 / 1353 / 1363 decode 78.1 (+9% prefill, decode unchanged)
The 1024 rows cost 268 MB a device (1% of the card). So high-share runs take min(chunk, 1024) rows and the
old rule stays elsewhere; DS4_ATTN_SLAB still overrides. The un-enved default verified at 1335 ppts /
77.8 tok/s with the ring at 253 MB.
With the 1024-token prompt chunk (the default where the cards hold most of the experts, abfe7d813) a big
staging ring is no longer the trade it was: every MB it takes is a tenth of an expert the decode keeps.
Measured on the same box, 8K code prompt + six 128-token decodes a load, ids 5705e1164a / 81d895366c:
2000 MB 1243 ppts / 73.2 tok/s
1000 MB 1287 / 77.5
500 MB 1314 / 78.5
256 MB 1324 / 78.4
128 MB 1338 / 78.3
So high-share runs take clamp(room/device * 0.008, 128, 512) MB (248 MB here) and the old deeper ring stays
for the cases it was measured for (share < 0.5, where the cards stream most of the experts). The un-enved
default verified at 1339 ppts / 78.1 tok/s, 42 s of summed GPU time in a 36 s pass.
prefill_chunk_default() returned 1024 only for a single-card run (own_default); a multi-card run whose cards
hold most of the experts fell through to 2048 (HIP skips the two 4096 branches) - and the rule's own datapoint
for 1024 was a 4 x R9700 run ('1024 370 tok/s, 8192 214'). Measured again here, 2.25 bpw, 4 cards, 8K code
prompt, --dspark auto, ids 5705e1164a, 4 prompts a load:
512 1110 / 1137 / 1142 ppts (median 1123)
1024 1229 / 1257 / 1270 (median 1243) <- this commit's default
2048 689 / 695 / 696 (median 692) <- what the engine picked before
8192 690 / 690 / 695 (median 690) <- what the 0.2.7 configs passed
Decode unchanged (73.2 against 73.2 tok/s medians, --dspark auto). So the owner's default 8K prompt goes from
692 to 1243 ppts (+80%) with no flag; --prefill-chunk still overrides.
DeepInfer's rule: verify a draft while the product of the confidences so far stays above the cut, instead of
only each draft's own floor. The 0.2.7 notes left it off ('not measured faster on code: H100 / owner PC'),
Project Maya runs 0.5 on a Radeon 8065S (+6.5% decode, the same tokens).
Measured on 4x R9700, DeepSeek 2.25 bpw, six 128-token decodes an arm in one load, same ids
(81d895366c): 71.7 -> 76.9 tok/s (medians) with STRATA_DS4_DSPARK_CUM=0.5, and a second pair 74.1/73.6;
the default with the rule on (no env) verified at 74.5 tok/s (6 decodes). CUDA keeps 0 (the rule was flat
there). STRATA_DS4_DSPARK_CUM still overrides either way.
The prompt path copies every streamed expert from the RAM copy into the pinned staging before its DMA (the
file's own note: 6.8 GB/s through the staging against 24-28 GB/s probes). Registering the copy lets it DMA
in place. Three interleaved pairs on 4x R9700, 2.25 bpw (share 0.72): the second 8K prompt of a process
12.0 / 11.9 / 11.9 s against 28.8 / 13.8 / 12.3 s unpinned; decode 35.42 / 35.50 / 34.9-36.0 against
35.43 / 35.45 / 34.9-36.0 tok/s, i.e. no cost; ids unchanged. Costs 23 s of load (registering a 50 GB
copy) - once a process, which for a server is the right trade. Auto only when the expert share is >= 0.5
(the CPU lane is then a minority of the decode, so the pinned pages' TLB cost cannot bite); DS4_COPY_PIN=0/1
forces it, and it stays off where the 12 GB cards' decode is CPU-lane-bound (the -15% measured there).
The RAM->staging memcpy runs on the IoQueue's threads; on a warm 8K prompt (stream stage 14.3 s, 12.8 GB a
device) they are busy 4.1 s, i.e. ~3 GB/s a device - a real share, and the thing to check in the 50-95 s
first-prompt runs (where the same counters are what to read first). DS4_PROF only.
The first prompt after a load is 2-9x slower than the second on the 4x R9700 2.25 bpw cell and it is not
the disk (0 MB read during it), the RAM store (removing it is worse), the dense GEMM shapes (q_b+rope
1.46 s) or the compute waiting (0 ms). The lead is the RAM->pinned-staging->DMA path (the file's own note:
6.8 GB/s through the staging against 24-28 GB/s probes); DS4_COPY_PIN=1 measured 47.9 -> 31.4 s on the
first prompt and 28.8 -> 12.0 s on the second, decode unchanged. This commit only adds the counters
(staging / DMA issue / slot wait) to the existing prompt stream line under DS4_PROF; no behaviour change.
No behaviour change on the 4x R9700 cell (126.8 GB of room / 4 = 31.7 GB a device, 8% -> 2000 MB, the value
measured in 76177ec34); on other card counts it is the per-device share that was meant.
The prompt path streams the experts the cards cannot hold. ring_mb_auto() gave 16 MB whenever the cards
held >= 1/1.3 of the experts, which is exactly this 2.25 bpw cell (126.8 GB of room, 156.3 GB of experts,
share 0.769): the stream then had no depth and the expert tiles waited 1.7 s inside a 5.2 s stage. A share
of the VRAM room (5%, 400-2000 MB) when any expert spills, 16 MB when they all fit (the 0.2.6 rule, kept
for the decode it was chosen for).
Measured, 4x R9700, 2.25 plain pack, 6 experts, --dspark off, 8192-token code prompt, ids 5705e1164a:
8K warm 632 -> 675-678 ppts (+6.8%, 5 prompts a run, 3 runs), agent decode 35.9 -> 35.3 tok/s (-1.7%,
4 runs). The profile of the warm pass: expert stage 5185 -> 4034 ms (kernels 3509 -> 2645 ms, the same
kernels now that the weights arrive in VRAM ahead of them, waits 1676 -> 1389 ms), everything else equal.
Also in this commit (opt-in, defaults byte-identical except the F16 activation path which is bit-identical):
- SetArgs::x16: gate/up take the layer's F16 copy of the activations (rows_f16 rounds exactly as the tile's
own smem store did) - half the activation bytes, 26% faster gate/up in the standalone bench
(box/expert_bench.cpp), +0.3% at the pass level (the stage is stream-bound).
- HIPK_CFG 6..11: tile variants for the expert GEMMs (64x128, 32x128, 128x256, 64x128/4 waves,
128/128+down 256, 64x256). 64x128 (cfg 6) is 21% faster than 128x128 in the bench and +0.9% in the pass.
- HIPK_DBG 1/2/3: switch the MMA / the decode off in the tiles (the breakdown in 09-marlin-kernel-diary.md).
- ds4_iq_hip.inc: a UTF-8 BOM on a source file removed.
It now ships on at 2.0 for a card of 13 GiB or less: +37.6% story / +35.2% code
decode over five interleaved pairs on an RTX 3060 12 GB, and it stays off on
bigger cards where it measured neutral to negative (l620 -1.9%/-0.5%, z440
-4.2%/+11.0%). It changes which experts run - the owner approved that trade
for 12 GB cards - and STRATA_ROUTE_PRIOR=0 turns it off.
The design was HIP-only: 80 resident blocks, a column slice of D each, two
grid barriers on an atomic counter, a barrier timeout so a card that cannot
hold all 80 blocks resident turns a hang into a refused start-up check. Ported
by giving grid_sync a CUDA branch (volatile generation counter + clock64() for
the timeout, scaled to SM clocks), guarding the gfx-arch refusal to HIP and
adding a masked-shuffle helper.
On the RTX PRO 5000 Blackwell 72 GB it passes its own check - '80 blocks on 110
multiprocessors, 0 token counts timed out, max relative difference 1.8e-07
against the default read' - and measurably does NOT win: story 201.4 -> 190.6
tok/s (-5.4%), code 245.6 -> 249.6 (+1.6%) over 3 interleaved pairs. That
matches the AMD note in the same file (an R9700 measured it neutral to 1%
slower), so it stays opt-in (STRATA_HC_PERSIST=1) and is not a default. The
port is kept as the starting point for a CUDA-tuned block mapping.
The router's top-10 is picked again with a logit bonus for the experts already
in VRAM (route_prior.cu); the weights stay the router's. Measured on an RTX
3060 12 GB, IQ3_XXS, 5 interleaved pairs of fresh engines: story 44.2 -> 60.9
tok/s (+37.6%), code 47.3 -> 64.0 (+35.2%); =1.0 gives +22.6% / +24.6%. Three
more pairs against the switch itself read +23.6% / +36.9%.
It changes which experts run, so it is a default only where the expert cache
cannot hold the model, and the owner approved that trade for 12 GB cards. It
is NOT a default on a bigger card: on an RTX PRO 5000 72 GB it measures
-1.9% / -0.5% and on 2x RTX A4000 -4.2% / +11.0% (3 pairs each); both were
re-run after this change and are unchanged (201.7 / 246.9 and 87.1 / 107.1
tok/s). STRATA_ROUTE_PRIOR=0 turns it off and any explicit value wins.
A draft for the owner: the one engine default that changed (the wide
hyper-connection up read on a 12.0 card, +1.9% story / +0.7% code decode on
the RTX PRO 5000, 5 pairs, bitwise), the four settings that are faster where
the PC supports them (PLE in RAM, the English draft vocabulary, --prefill-pipe
on a two-card split, q4_0 KV on 12 GB with its precision cost), and the list
of what was measured and rejected so it is not retried. No version number is
claimed: the owner decides what else 0.2.7 carries.
STRATA_GR_UP_WIDE (one warp a row, N/2 = 1,280 blocks instead of 160)
measured on the RTX PRO 5000 Blackwell 72 GB, IQ3_S, every expert in VRAM,
5 interleaved pairs of fresh engines: story 197.6 -> 201.3 tok/s (+1.9%),
code 243.5 -> 245.1 (+0.7%). A third run against the switch (default on vs
STRATA_GR_UP_WIDE=0) reads +1.7% / +0.8%. The produced token ids were
identical in all 6 pairs, and fused_gr_bench's per-T hashes are the plain
read's. The switch stays: =0 off, =1 on any card, STRATA_SM120_PERF=0 the
whole 12.0 set. Other cards are untouched.
setup writes DS4_ASYNC_T=4 into the config env for an integrated Radeon: the engine's mailbox path is sized for cards that hold
most of the experts and costs this box 31% on code (15.9-16.3 -> 20.1-21.3 tok/s), 16% on agent with identical accepted tokens,
9% on chat; the 2.25 set (no drafter) is unchanged. The same tokens come out. Discrete cards keep the engine default.
The engine's windowed decode uses the mailbox ("async") path for windows up to DS4_ASYNC_T (8 by default). That default was sized
for cards that hold most of the experts; on the 128 GB gfx1151 box the 6-row verify window is faster on the synchronous path.
Measured 2026-10-25 with the 2.00 bpw set and the shipped 0.1.51 engine (greedy, 256-token sets, three interleaved reps):
- code 15.87-16.33 -> 20.14-21.28 tok/s (+31%);
- agent, with identical accepted tokens a round (5.07): 15.93 -> 18.48 (+16%);
- chat 10.33 -> 11.30 (+9%);
- 2.25 code unchanged (11.2 -> 10.9): the drafter is unloaded there, so the extra syncs have nothing to hide.
The same tokens come out; only the scheduling changes, and the drafter's own single-token windows (T=1) stay on the async path.
dsv41_config_env(cards) writes DS4_ASYNC_T=4 into the generated config for an integrated Radeon only (a discrete card keeps the
engine default); STRATA_DSV41_ASYNC_T=n overrides the value and =8 restores the default. Tests:
tests/terminal/test_dsv41_async_t.py (5 cases). No engine rebuild; the signed dsv41 package is unchanged.
Docs: the STRIX_HALO and DEEPSEEK41 measured tables and the README row now read 19-21 tok/s code, 18 agent, 11 chat.
The trace origin was thread-local, and the prompt's later stages run on their
own thread, so their timestamps could not be compared with the first stage's.
One process-wide origin makes the hand-off visible.
Found with it on z440 (Q2_0, split 24, 16,383-token prompt, 8K chunks): stage
0 starts at 0 ms and 3,267 ms, stage 1 at 3,268 ms and 5,996 ms, read ends at
8,775 ms - the two stages do overlap chunk-for-chunk, and each takes ~2.7 s
per 8K chunk for its 24 layers. A split 30 run leaves stage 1 at 3.09 s a
chunk with 18 layers, so the per-chunk cost is not its layer count.
Same diagnostic as the request-step timestamps: with a layer split the chunk
line now says which stage is starting a chunk and when, so a prompt's stage
overlap can be read off the log. STRATA_TRACE only; no behaviour change.
Measured with it on z440 (Q2_0, split 24, 16,383-token prompt): serial reads
run stage 0's two 8,192-token chunks 3,332 ms apart and the whole read takes
8,788 ms; --prefill-pipe 1500 reads four 4,096-token chunks 1,582 ms apart and
the read takes 8,375 ms, so the stages overlap only partially at best.
The trace lines named the step a request reached but not how long it took to
get there, so a stall between two marks was invisible. Each line now carries
the time since the first traced step (thread-local). A diagnostic only: no
step, no order and no output changes without STRATA_TRACE set.
Found with it on z440 (Q2_0, layer split 24, 16,383-token prompt): the batched
read takes 8,758 ms while stage 0's own GPU timeline is 5,374 ms - stage 1 is
the slow one - and the expert-cache refill after the prompt costs a further
592 ms of the request.
Measured on the 128 GB box with the 2.00 bpw set: 4K prompt 78.7 -> 87.2 tok/s, 8K 108.9 -> 125.1, 32K 155.2 -> 176.9;
decode unchanged (16.53 -> 16.59). setup writes --prefill-chunk 8192 only for an integrated Radeon whose whole pool is at
least 64 GB; the escape hatch is STRATA_DSV41_NO_BIG_CHUNK=1. Also repairs UTF-8 damage in three docs from an earlier
PowerShell edit.
The engine's prompt-chunk rule returns 1,024 when a single card holds half or more of the experts; it was calibrated on
cards that hold most of them (4 x R9700). On the 128 GB Strix Halo (share 0.62, one shared pool) the bigger chunk wins:
measured with the 2.00 bpw set, engine 0.1.51, 2026-10-25: 4K prompt 78.7 -> 87.2 tok/s, 8K 108.9 -> 125.1, 32K 155.2 ->
176.9. Decode is unchanged (16.53 -> 16.59): the chunk buffers take ~5 GB from the GPU expert tier (81.9 -> 76.6 GB) and the
RAM store's hit rate rises (68.6% -> 76.8%) to compensate.
dsv41_prefill_chunk_args(cards, ram) writes --prefill-chunk 8192 for an integrated Radeon only, and only when the whole pool
(the RAM Windows sees plus the BIOS carve-out) is at least 64 GB; a discrete card or a smaller unified PC keeps the engine's
choice, and STRATA_DSV41_NO_BIG_CHUNK=1 turns it off. The chunk changes the prompt GEMMs' summation order, not the model: the
repo's chunk-size KL check reads 1024 0.1705 / 2048 0.1692 / 4096 0.1688 / 8192 0.1695. Test:
tests/terminal/test_dsv41_prefill_chunk.py (5 cases).
Also repairs UTF-8 damage in README.md, docs/STRIX_HALO.md and docs/MODELS.md from an earlier PowerShell edit (the middots,
em dashes and arrows had been double-encoded; no text changed). No engine rebuild is needed: the existing signed dsv41 package
accepts the flag.
The up projection of the hyper-connection read runs 160 blocks of 8 warps
(each warp takes 8 rows) on a 12.0 card, so 160 blocks of 192 SMs and ~6.7
warps an SM: the read is latency-bound at about 0.5 TB/s. This kernel takes
one row per warp and 2 columns x the 4 streams a block, so the grid is
N/2 = 1280 blocks (53 warps an SM). Every row's dot, xor tree, epilogue and
the cross-stream sum are gr_up_multi_kernel's, in its order, so the outputs
are bitwise the same; the token streams of both arms were identical in all
6 pairs measured.
Measured on l620 (RTX PRO 5000 72 GB, all experts in VRAM, IQ3_S, 3
interleaved pairs, fresh engine each): story 197.4 -> 201.3 tok/s (+1.9%),
code 243.0 -> 244.7 (+0.7%). Off by default (STRATA_GR_UP_WIDE unset); the
q8_1 tail of S26 needs the 16-column blocks, so the wide kernel is only used
when no q8_1 image is written.
The Windows AMD DeepSeek engine (feed member dsv41, windows-x86_64-hip-rdna3, gfx1151) ships with these
changes: the 0.2.6 copy-all rule is off on an integrated GPU (it copied 53 GB into RAM beside the GPU's 82 GB
in one 128 GB pool and an 8K prompt stalled; the in-place path is the 0.2.5 soak path), a Windows HIP package's
rocm/bin is on the engine's library path (setup, run config, app), the unified-memory fit plan counts the BIOS
carve-out, and --feed reaches the DeepSeek engine download. Docs, app text and tests included.
Measured on the Strix box (2.00 bpw, Faster, DSpark auto): 4K prompt 78.7 tok/s, 8K 108.9, code 17.6 tok/s,
chat 11.0. Clean-user end-to-end from a signed feed: install, download, pack, smoke, serve, chat, tool call,
agent round trip: 5/5.
strata launch --feed passed the feed to setup, but setup's parser had no --feed (getattr(a, 'feed', None)
was always None), so the flag was refused with 'Unrecognized arguments' and a local or non-default feed could
never serve the DeepSeek engine (found by the clean-user test on strix, 2026-10-25). It is now the same
option strata update has: a URL or a folder.
Found by the clean-user test on strix: 'strata launch --feed DIR' read the local feed for its own engine
check and then setup went to GitHub for the DeepSeek package, so a local (or any non-default) feed never
served the DeepSeek engine. setup's get_dsv41_engine/upgrade_dsv41_engine take the feed, install_dsv41
passes --feed, and runner.setup_args hands --feed to setup (a test in test_launch_dsv41).
The unified-memory plan counted only the RAM Windows sees (64 GB on the owner's box) and ignored the 64 GB
BIOS carve-out the GPU owns: 128 GB in one pool. uma_total_gb() adds the carve-out (or the GPU figure less
its shared part), so the plan now says about 109 GB of the 2.00 set's 132 GB stay in memory and 23 GB are read
from the SSD; the docs and the test fixtures use the 128 GB box. The 'measured' text says 128 GB unified too.
The Windows AMD DeepSeek engine package (feed member dsv41, windows-x86_64-hip-rdna3, arch gfx1151) is built from this
branch and installed as its own package; these are the code changes it needs.
- ds4_model.cu: on an integrated GPU the CPU lane's experts are no longer copied into RAM by the auto rule. The GPU's
expert tier and that copy are the same shared memory: on the 64 GB Strix Halo the GPU held 82 GB of experts, the rule
saw 62 GB 'free' and copied 53 GB, and an 8K prompt did not finish in 11 minutes. Found 2026-10-25 on strix; the
corrected build does the same prompt at 108.9 tok/s (4K 78.7), code answers 17.6 tok/s, chat 11.0. DS4_COPY_ALL=1
still forces the copy for a test.
- dsv41_engine.py: lib_dirs()/start_env() put the package's BUILD.json lib_dirs on PATH/LD_LIBRARY_PATH (a Windows HIP
package keeps hipBLAS/rocBLAS in rocm/bin, or a clean PC fails with 0xC0000135); the upgrade test-start uses it.
- setup.py: the dsv41 engine's library dirs come from the package (so the run config carries rocm/bin), the unsupported-card
text names Ryzen AI Max on Windows, and the manual package hint names the .tar.gz the release page actually has.
- runner.py: the same library dirs for the app/Terminal path.
- dsv41_fit.py: a unified-memory plan for the official IST-DAS sets (one pool, what stays in RAM, what is read from the SSD,
measured 14-16 tok/s on code, 10-11 on chat, 4K 78 / 8K 109), never a refusal; tools/strata_mcp.py uses it for the app
catalog's on-this-PC line.
- Docs: STRIX_HALO.md (DeepSeek section with speeds, driver 26.10.1, memory behaviour, limits), DEEPSEEK41.md, MODELS.md,
AI_REFERENCE.md, README.md, packaging/README.md (the Windows AMD package recipe).
- Tests: tests/terminal/test_dsv41_fit.py (7) and 4 new tests in test_dsv41_engine.py (lib dirs, feed selection for gfx1151).
The Windows commit limit decides how many experts an integrated GPU may hold (place_experts). With AMD driver 26.10.1 the limit
went from 74 GB to 133 GB on a Strix Halo, the engine put 9506 experts (86 GB) on the GPU instead of 8105, every byte of the
GPU's address space was gone, and the 32K prompt's later allocations spilled into RAM: free RAM fell to 0 and the engine died
("rocBLAS: No devices found"). The experts now leave 4 GiB + 0.2 MB a token of context above 16384 (DS4_UMA_GPU_RESERVE_GB sets
it, tests); only an integrated GPU on Windows is affected.
Strix Halo (gfx1151, 2.00 pack, shipped profile), driver 26.10.1:
32K prompt: killed (min free RAM 0) before, 113.2 tok/s with min free 13.6 GB now; 64K prompt (70000 context): 122.2 tok/s, min free 14.4 GB
4K / 8K prompt: 62.8 / 83.9 tok/s (64.9 / 86.2 with the 9506 experts: -3%), min free 13.4-13.8 GB
KL, code segment, 999 positions: 0.1020 (0.1028 before)
A smaller reserve (3 GiB + 0.1 MB a token) gained 0.0% on 4K / 8K / code decode and left 0.9 GB less free at 64K, so it was not taken.
R9700 1 card 2.00: 21.30 -> 21.82 tok/s (+2.4%), 4 cards 35.39 -> 37.00 (+4.5%), 3 pairs each, same generated ids. DS4_QUANT_SINGLE=1 / DS4_QUANT_NODEDUP=1 go back to one launch an item / no sharing.
The 512-bit i-quant rows (IQ2_XXS/XS/S, IQ3_XXS/S) assembled their grid words from scalar loads (~30 uops a 64-value chunk). Where cpu_gather_fast()
(Intel Sapphire Rapids and later) they now load them by one gather, 32-bit for IQ3 and 64-bit for IQ2 (STRATA_IQ_GATHER=1/0 force it; Zen 4 stays scalar:
its gathers are microcoded). The same words are loaded, so every output bit is unchanged. One token of an IQ3_XXS / IQ3_S expert takes the exact 512-bit rows (ggml-cpu's
single-token AVX2 dot bit for bit) with the gather, after a first-use bit check against this ggml; STRATA_NO_IQ512X1=1 turns it off.
Xeon 8468, native_gu_rows (gate+up of an expert), us: IQ3_XXS 1 token 295 -> 219, 2 tokens 462 -> 300, 4 tokens 558 -> 389; IQ3_S 415 -> 305, 603 -> 342, 562 -> 296;
IQ2_S 2 tokens 377 -> 235, 4 tokens 437 -> 327 (one IQ2_S token stays on ggml: 239 against 234). Output checksums identical with and without.
Test iq512_gather_parity (both lookup modes): exact rows == ggml vec_dot at 1-4 tokens, engine rows within float rounding.
After 'Update Strata' installed a new app version, the host that asked for it was ended by the installer, so the job stayed saved as running and the next start turned it into a red 'Interrupted operation' alert in Settings > Update and Diagnostics. The job now records the app version it started from, and a start under a different version marks it done ('Strata is updated to X').
strata update and the app's check reused the feed used last, including a loopback address or a folder from a test of the update feed; every later check on that PC then said 'No internet connection'. Only an https address is remembered now.
Without --adapt-async on the command line, the expert arena on one CUDA GPU (--serve, adaptive tier on, no --batch, no layer
split, no pipelined windows, no peer or remote tier) starts on the blocking tier, ignores 64 steady decode windows, measures 64,
and when the tier's wait is at least 5% of a window runs 64 windows on the asynchronous tier, going back (for the session) only if
that was more than 2% slower per token. One log line at each switch. --adapt-async 0|1 and STRATA_ADAPT_ASYNC=0 keep their meaning;
the identity gates run with --adapt-every 100000 and never reach it. Decision logic and its unit test are header-only
(adapt_async_default.hpp, adapt_async_default_test).
Window 2's residual stream was window 1's stale tail: stream copies and mtl_rt's deferred command buffer had no
ordering at sync points (memcpy, stream sync). Now a full barrier over all streams and the context, and the context
takes a lock (helper threads flush too). engram_dequant_rows wrapped n*18/32*hd bytes but strides rows by hd: rows
past 192 of 288 read unmapped memory (inf scales, router picked INT_MAX). The CPU lane (ggml-cpu rows) was compiled
out, so every non-resident expert contributed zero. Debug: DS4_DBG_ALL (every layer), DS4_DBG_EMB, DS4_DBG_EG.
init_pre writes T*8 floats - 32 bytes for T=4 - and its wrapper wrapped the enclosing 16 384-byte page to bind them. That aliasing
MTLBuffer overlaps the arena pages that hold h, and with it present h read back as 'valid at both ends, NaN in the middle' with
nan_range starting at exactly 4 096 - one wrapped page of floats. The work is trivial, so it now runs on the CPU through host_view:
no wrap, no alias, no dispatch.
Result with the fix: h at layer 0 is clean (zeros=40960, ninf=0, max=0) instead of ninf=4096 max=7.6e+07. The output is still
0,0,0,... so embed's write still does not reach h - but the contamination that had been confusing every probe of that buffer is
gone, and the two remaining readings (host_view vs cudaMemcpy on h between embed and layer 0) can now be compared cleanly.
Fixes this session, all measured: the 1.32 GB all-zero embedding file (my converter read the sparse trial shards), my own
20000-dispatch flush cap (it blocked every prompt of 256+ tokens), and now the init_pre page alias.
Targets, unmet: decode 8.94 tok/s vs 100, prompt 4000 tokens ~27 tok/s vs 1000, and no valid token has ever been produced.
Measured this session, in order of how much they narrow the problem:
* the pack's embedding table is real, and the engine hands embed correct rows (rows_in: first=4759 4759 4759 4759)
* ds4_embed's kernel receives the right arguments (args probe: dim=5120 hc=4)
* one Dev, one h: DBG devh prints id=0 h=0x100011a8000, the same address layer() reads
* the memory model is sound: writing through host_view round-trips through cudaMemcpy at 4 KiB and at 64 MiB (42 -> 42)
* both dispatch routes work: a constant written via Arg::B reads back 3 3 3 3, via a device pointer passed by value 7 7 7 7
* the Metal embed path is load-bearing: breaking its pipeline changes the exit code 0 -> 3
What is still wrong: h is partly NaN at layer 0 (nan_range=4096..40703, max=7.6e+07) with valid data at both ends, and
wholly NaN from layer 1, so the logits never resolve and every run prints 0,0,0,... Nine hypotheses were tested and
disproved today: the zeroed table (a real bug, fixed, not the cause), two queues, host_view vs device pointer, memory
coherence, the binding, a dead Metal branch, a wrong dim, a short write, and two Dev records.
Targets, measured, unmet: decode 8.94 tok/s vs 100; prompt 4000 tokens ~27 tok/s vs 1000. The 51.8 and 176 tok/s reported
earlier were dispatch-ENCODING rates and are withdrawn here as they were in the notes.
Next: locate the real d.h allocation (several probes landed on paths the engine does not take) and read that buffer at
three points in one run - at allocation, after embed, and at layer 0.
embed writes h = 0x144f24000
layer 0 reads h = 0x100011a8000
Both report device 0, and forcing --gpus 0 does not change it, so it is not a tensor-parallel artefact: the engine holds two Dev
records with two h allocations, embed fills one and layer() reads the other, and the bounce in ds4_model.cu (if (l.dev != cur)) is
meant to reconcile them but does not on this path.
That single pair of addresses explains every symptom: layer 0's input buffer was never written (hence zeros=40960), the first block
then produces NaN, and the logits never resolve. It also retires the last four hypotheses: the embedding table (real), the embed
kernel (verified: writes as_type<float>(0x4759 << 16)), two-queue ordering (shared Context now), and stale host reads (the probe
flushes).
Correcting the record: the 51.8 tok/s decode and 176 tok/s prefill reported earlier were dispatch-ENCODING rates and are withdrawn.
The real measured rates are 8.76 tok/s decode and ~27 tok/s at 4000 tokens, and no valid token has ever been produced. Targets remain
100 tok/s and 1000 tok/s: both unmet.
Next: print cur, l.dev and the two h addresses inside layer() - whichever differs is the buffer the embedding never reached.
Measured this stretch:
* the pack's embedding table is real (embed_pinned = 3d6b 3e6e 3e65 3dd7)
* embed itself is correct: immediately after the call h[0..3] reads 55296, which is exactly as_type<float>(0x4759 << 16),
the BF16 embedding value converted the way both ds4_embed and embed_kernel convert it
* yet at the top of layer 0 h is all zeros (zeros=40960 max=0), and block 1 turns that zero into 100% NaN
Nothing between the two probes writes h except init_pre (which writes pre). The remaining suspect is which Dev's h the layer reads:
the engine has a cross-device bounce (cudaMemcpyAsync through a host staging buffer) that ds4_metal_flush() does not cover, and a
stale l.dev under 'tensor parallel 2 -> 1' would produce exactly this.
Three hypotheses about the embedding are now retired with evidence: the zeroed table (a real bug, fixed, did not change h), two-queue
ordering between init_pre/embed and the rest (they now share one Context, no change), and the kernel writing nowhere (it writes
correctly). Targets remain unmet: decode 8.76 tok/s vs 100, prompt 4000 ~27 tok/s vs 1000, and no valid token has been produced.
State of the port, honestly: the engine runs, 40+ kernels dispatch, the pack and the 110 GB Engram are real, and it has NEVER
produced a valid token. Both targets are unmet: decode 8.76 tok/s (target 100), prompt 4000 tokens ~27 tok/s (target 1000).
What this commit adds:
* the drafter path: ds4_mx_gu/ds4_mx_dn (MXFP4 prompt gate_up/down, one runtime-ne kernel), ds4_markov_add, ds4_confidence, with
hosts and guards. --dspark on now runs end to end, but the confidence is NaN (its taps come from the trunk, which is NaN), so no
drafts are offered and there is no drafter speedup.
* token-boundary commits: seven host-read sites in ds4_model.cu flush the deferred command buffer before reading a device result.
This is what exposed that the earlier 51.8 / 176 tok/s figures were dispatch-ENCODING rates, not GPU rates: with the work inside
the timed window the real numbers are 8.76 tok/s decode and ~27 tok/s at 4K. Those earlier numbers are withdrawn.
* embed/init_pre now dispatch through the shared Context (they had their own Metal queue, so no ordering against the copies that
feed them) - measured, and it did NOT fix the blocker.
* the embedding blocker, measured at both ends: the pack's table is real (embed_pinned reads 3d6b 3e6e 3e65 3dd7), ds4_embed is
dispatched at ds4_model.cu:7490 and returns, and h[0..3] is 0 0 0 0 immediately afterwards, so the kernel writes nothing into the
buffer the engine reads. Behind it, block 1 turns a zero input into 100% NaN (a 0/0 in the first norm or router).
* env-gated debug probes (DS4_DBG_H, DS4_DBG_EMB, DS4_DBG_L) left in for the next session; they are inert without the variables.
Known edit hazard, learned three times: patching by a generic statement pattern lands in the wrong function. Patch by unique
context or by function name.
Removed the per-dispatch histogram from encode() (it cost a mutex and a string key per dispatch; measurement showed no gain, so the
~10 us a dispatch is genuine encode + engine host work). Reverted the two experiments that did not hold up: the q8_quant dedupe
(the repeated calls all differ - 0 dispatches saved) and skipping host_view() in cpu_addr (breaks the engine, 15 dispatches).
Prefill chunk sweep, 4000 tokens, pack-2.25-real: --prefill-chunk 16 -> 165.90 tok/s, 32 -> 175.89, 64 -> 176.26. The engine runs
16-token windows regardless, so the curve flattens: 176 tok/s is the 4K number and the target is 1000.
Measured state: decode 51.8 tok/s (target 100), prompt 4K 176.3 tok/s (target 1000). Neither met; the remaining levers are kernel
fusion for decode and a key-tiled attention kernel for prefill, both rewrites.
Two measurements tonight that matter:
1. Every prompt of 256 tokens or more died at exactly '20073 dispatches, 2 command buffers'. That is the deferred command buffer's 20,000
dispatch cap - my own constant from when I introduced batching - flushing mid-token. Raising it to 2,000,000 makes the long prompts run:
64 tokens 535.8 tok/s
128 tokens 556.3 tok/s
256 tokens 542.6 tok/s (31,508 dispatches, 1 command buffer)
1024 tokens 451.0 tok/s (126,020 dispatches, 1 command buffer)
4000 tokens 170.9 tok/s (492,254 dispatches, 1 command buffer)
The curve falls with length, so the 4K figure is attention scaling, not fixed overhead.
2. The per-kernel dispatch histogram now works (Pipeline carries its name; the earlier '<unnamed>' was a lookup bug):
ds4_q8_quant 4122 (36%)
ds4_iq_mv_k 1218
ds4_finish 1038
ds4_hc_pre_norm 486 / hc_split 480 / hc_mix 480 / hc_post_v 480
ds4_gemv_f32 474
~1,890 dispatches a token at 8.8 us each = the 16.7 ms a token, so decode speed is 1/(launches x 8.8us). 100 tok/s needs
~1,130 launches: dedupe q8_quant per layer (-380) and fuse the four hc_* passes (-320) gets there arithmetically.
Decode 51.8 tok/s (target 100, measured on pack-2.25-real), prompt 4K 170.9 tok/s (target 1000). Neither target met.
Measured (real 110 GB Engram pack, engine's own lines):
prompt 64 tokens -> 535.76 tok/s
prompt 128 tokens -> 556.31 tok/s
decode -> 48.3-50.0 tok/s
prompt 256+ -> aborts, router sentinel 2147483647 (no pps figure)
Targets are 100 tok/s decode and 1000 tok/s prompt: both roughly 2x away, neither met.
Added:
* sparse_attn_packed<G> (G=8/4/2/1) and the BF16 row variant, with the host wrapper that mirrors AttnRows and picks G - the 4K
abort at ds4_kernels.cu:2161 is gone.
* ds4_iq_mv widened to 16 accumulators; host chunks any job with ncol > 16 into 16-column jobs (this prompt needs ncol=96), and
Q5_K/Q6_K dense matrices get ds4_iq_mv_k instead of being fed to the IQ2-only decoder.
* route, gather_rows, combine kernels.
* Wrap extents made exact across 7 wrappers (moe_sort, swiglu, hc_split scale/base, sparse_attn rows, unpack_sel, engram_dequant);
the old 'bytes + 65536' form could overrun an arena chunk's edge.
Known bug, not fixed: for prompts of 256 tokens or more the attention SINK logits read NaN (sink[57] at the first failing window),
which poisons the softmax denominator, then q, then the router. 64 and 128 tokens are clean, so it looks like a path switch at the
256 boundary (few = T <= kAttnScratchT in sparse_attn) or an overrun into the sink buffer. That is what stands between the current
556 tok/s and a 4K figure.
Five conflicts, resolved keeping every f-024-rel behaviour (slots pin, long-prompt skip, HIPK, ZC fix) and maya2's:
- ds4_main.cpp, request struct: both fields kept (pin=N from f-024-slots and dspark_cum from maya2).
- ds4_main.cpp, request parser: both keys kept (pin, dspark_cum/spec_cum).
- ds4_main.cpp, prompt feed loop: slots' pin_at (the read stops at N for the shared-start checkpoint 3) and maya2's Stop handling (checkpoint 1 before the prompt, set_cancel, the stop check at the top of each feed) combined: the loop checks Stop first, then sizes the feed with the pin cut (n is no longer const).
- ds4_model.cu, forward chunk loop (two places): the skip's 'run.runs(k, il)' (an early chunk skipped past the yoco layer) kept, and maya2's halt_now() / halt_end(di, err) check added after it, so a skipped layer does no halt work.
win_submit (ds4_kernels.cu) is compiled to a no-op outside _WIN32 (#if defined(_WIN32) ... else (void) s), so it is already out of the Linux CUDA and Linux HIP builds.
Owner PC (RTX 5070 12 GB, Ryzen 5 7600, official 2.00 pack, shipped profile-2.00.bin, 3 alternating pairs, medians, off = every switch off):
4K prompt, off 92.64 tok/s: win_submit 99.12 (+7.0%, 3/3 pairs), flow pump 97.37 (+5.1%, 3/3), 64 Engram readers 96.64 (+4.3%, 3/3),
unbuffered Engram 85.56 (-7.6%, 0/3), all on 86.84 (-6.3%).
code decode, off 4.51: timer 4.51 (1/3), 64 readers 4.41 (2/3), unbuffered 4.67 (2/3, one pair 3.83), all 4.57.
mmap decode, off 1.95: cold-page prefetch 1.71 (-12%, 0/3).
Cold (no profile) agreed on win_submit and flow pump (+4.5%, +2.7%, 2/2) and showed the timer +12%, readers + unbuffered +22% on decode (2/2, off arm noisy).
Identity: --vram-experts 0, off vs all IDENTICAL (4K prompt, code decode); text equal to off in every warm arm.
Stop during a long prompt (H100, 8K and 16K): STOP answered in 0.12 / 0.16 s, the conversation before the prompt kept, the next request identical.
DSpark cum cut: H100 code 20.13 tok/s off vs 19.60 at 0.5, no gain.
ON: DS4_WIN_SUBMIT, DS4_FLOW_PUMP_WAIT, Stop, STRATA_DS4_LANE_FREE (a no-op on the physical-core lane).
OFF, switch turns it on: DS4_TIMER_MS=1, DS4_ENGRAM_THREADS=64, DS4_EG_DIRECT=1, DS4_IQ_PF_COLD=1, STRATA_DS4_DSPARK_CUM=0.5.
One conflict, docs/DEEPSEEK41.md (both sides edited the three-set paragraph and the file table). Resolution: f-024-rel's text kept (IQ1_M first, the model-card sentence, 533 GB for all three, the 'not a model to pick' Engram row), with the setup worker's sentence that setup recommends 1.75 where 2.00 is tight in memory or disk added to the 1.75 entry.
Context::run() committed a command buffer and blocked in waitUntilCompleted for EVERY dispatch. At ~60 dispatches a layer over 40
layers that is 3.3 us x 2400 plus a 133 us commit+wait each, and it showed up as exactly the 330 ms a token the engine reported.
Now a single dispatch is appended to a persistent command buffer and returned from without waiting; the buffer is committed when it
fills (20000 dispatches) or when the engine synchronises (strata::gpu's stream/device synchronise call ds4_metal_flush()).
Measured on the same fast harness (--vram-experts 0.05 --cpu-threads 2, ids 1,2, n=6):
before: prompt 5.43 tok/s, decode 3.00 tok/s, layers 330.1 ms/token
after: prompt 77.0 tok/s, decode 48.7 tok/s, layers 19.1 ms/token
with the experts resident in VRAM instead of streamed: decode 42.6 tok/s (layers 19.5 ms) - so residency is no longer the limit.
The remaining gap to 100 tok/s is the 1-token IQ2 matvec itself: with ncol == 1 the eight accumulators are idle and every weight is
decoded for a single dot, which is the 87.7 GB/s case measured earlier (versus 437 GB/s at an 8-token window). Fixing that is kernel
work, not runtime work.
One conflict, src/dsv41/ds4_model.cu (forward_chunks): f-024-rel has the long-prompt skip (kv_only layers), f-024-hipk added DS4_PROF stage marks around the same lines.
Resolution: the f-024-rel code kept as it is, and hipk's three prof_mark(d, 5/6/7) calls added at the same places (before the q/kv gemms, before the snapshot, after the kv_only return). They are no-ops with DS4_PROF unset.
Official harness completes with no abort: /tmp/r.out = '0,0,0,0,', zero 'no Metal dispatch', zero metal errors. Prompt 4 tokens in
5112 ms, decode 4 tokens in 18876 ms. The ids look degenerate and numerical parity is NOT established - the Engram in this pack is
zeros, and a wrong kernel would produce the same output - so what is proven is that the engine walks the entire path and emits ids,
nothing more.
Kernels added (each verified by the engine's next abort naming a later or different launch):
prompt expert GEMM: gate_up_kernel<T,NE>, down_kernel<T,NE> (ds4_iq.metal, with IQ3_XXS type 18 joining the decoders - this
pack mixes gate types 16/17 with down types 18/22)
mx_combine, engram_dequant (fmt 2), engram_gate, bf16_mv (engram wkv), compress, index_score, topk (radix with simd_ballot),
finish_kernel<true> (KV pack - the call site had no guard at all), finish rounding branch, gemv_f16_rows, block_max, hc_mean,
iota_rows, sampler_greedy (ds4_init.metal, 1521 lines)
mx_wait skipped under STRATA_METAL, like the pre-existing wait_seq no-op
Also fixed a real pre-existing bug: the 'payload + 65536 from the page base' wrap convention walked off the mapping when a small
buffer sat at an arena chunk's edge (flaky 'mtl: wrap 81920/98304 failed', exit 4, 2 runs in 4). ds4_finish.cpp now wraps exactly
the pages a buffer spans.
Not ported: sampler_kernel<0> (the sampled path; the harness is greedy), engram fmt 0/1 (this pack is fmt 2), bf16_mv's non-vector
variant. Expert types beyond 16/17/18/22 make the IQ kernels return false, so the CUDA launch names the line instead of a silent pass.
Same default as Maya #363. For the deepseek41 family only: Service.reasoning_budget() answers 32768 when the request has
no reasoning_budget_tokens / thinking_budget_tokens and the config has no reasoning_budget_tokens at all (a config value,
0 included, wins; a request's 0 turns it off). The existing wrap-up (#123) closes the thinking at the budget, and the
answer-room clamp still cuts it to leave the answer its share of max_tokens. Thinking off and other families unchanged.
Tests in serve/test_dsv41.py: the default, a request's 0 and own value, the config's value and 0, another family, and a
5000-token cap that is wrapped up and answered. serve suite: 1027 tests OK (15 skipped).
/v1/messages and /v1/chat/completions with max_tokens 200-600 and the thinking on spent all of max_tokens on the
thinking: the 0.1.45 answer room starts at max_tokens 1024 and a client that gave no budget of its own got no text block.
For the deepseek41 family only, with no budget from the client or the config, the thinking is now capped at max_tokens / 2
for max_tokens from 48 up to 1023 (the existing wrap-up, </think> and the answer follow in the other half). A client's
own reasoning_budget_tokens / budget_tokens wins, "answer_room": false turns it off, thinking off is unchanged, other
families (Qwen) are untouched. Maya #363 closes the thinking in the engine instead; the server's #123 budget already does
that for every engine, so only the short-cap gap needed closing.
Tests (serve/test_dsv41.py SmallCap, fake engine that thinks without end): max_tokens 200, 400 and 600 on both APIs return a
non-empty text block with the thinking wrapped up at half; thinking off answers directly; a client budget wins; a tool
call under a short cap still parses; another family keeps the old behaviour. serve suite: 1025 tests OK (15 skipped).
Layers 20-39 of DeepSeek-V4.1-Flash read a prompt's earlier tokens only through layer 20's compressed KV and index
keys, and each looks back one 128-token window more than the layer before. So for the answer and for the rings the
decode reads they need their own work only on the last (40 - 20) x 127 + 128 = 2,668 rows. A prompt of ~3,700 or more
new tokens on one GPU now runs its earlier chunks through layers 0-19 plus layer 20's kv_only part (the hyper-connection
pre-norm, the compressor latent, the index keys, the compressed KV) and nothing of the later layers' attention or
experts; the chunk the cut falls in runs layers 20-39 from the cut row on; later chunks run every layer. The expert flow
plans nothing for the skipped (chunk, layer)s. STRATA_DS4_PROMPT_SKIP: 1 (default) the skip, 2 only the last 128 rows
(DeepSeek's own serving; answers shift a little), 0 off. Same approach as Maya #318.
H100 emulation (64 GB cgroup, 12 GiB VRAM, store 41 GB, 3.6 GB/s SSD cap): 16K prompt 69.3 s -> 51.5 s (236 -> 318
tok/s, two alternating pairs), 4K 32.5 s -> 30.7 s; 800 teacher-forced positions after a 4,200-token skipped prompt: KL
0.00029 against the full work, top-1 against the text 88.11% against 87.98%, ppl 1.5110 against 1.5111.
--score gets DS4_SCORE_PREFIX=N (the segments joined, the first N tokens read as one prompt with the last row's logits,
the rest scored) so the skip can be scored; tools/dsv41/test_prompt_skip.py runs it against a pack when one is named.
The Maya notice file now says later Maya changes are covered by the same notice.
A prompt that does not continue the conversation in the model first sets that one aside (position, window rings,
compressor partial groups, each KV source's compressed KV and index keys, its checkpoints), then takes back the kept
conversation it does continue, in RAM or from the slot folder, also across restarts. The latest checkpoint the prompt
still agrees with is used (end of the last prompt, before it, the anchor of the first prompt of a run, and the shared
start the server marks with pin=N). The .meta format and the prefix matching are in ds4_slotmeta.hpp with a GPU-free
test (ds4_slot_test). STRATA_DS4_REUSE=0 reads every request from token 0 (reference for tests). Off unless
STRATA_DS4_SLOT_KEEP=1.
f-dsv41-rel was based on the 0.2.2-era tree (merge-base 1f727f4b5), so the merge brings it onto 7ddc17a38 without dropping 0.2.3 / 0.2.3.1.
Conflict resolutions (0.2.3.1's behaviour kept everywhere that is not DeepSeek):
- packaging/package_engine.py: build() keeps 0.2.3.1's check_reader / allow_stale_vision / allow_no_reader and adds require_dsv41; both the
picture-reader check and the strata-dsv41 presence check run; the CLI call passes both sets of arguments.
- packaging/ENGINE-COMPONENT.md: 0.2.3.1's cuda13 / cuda12 rows (vision built per variant), with strata-dsv41.exe added to the cuda13 contents.
- serve/server.py: the parser is self.parser_cls (OutputParser, or the DeepSeek DSML parser) and keeps 0.2.3.1's declared_only argument.
- strata_cli/launch/cmd.py: the --plan output prints 0.2.3.1's drafting line and then the DeepSeek placement lines.
- strata_cli/launch/runner.py: plan() returns 0.2.3.1's dict with "drafting", plus "placement" for multi-file (DeepSeek) models;
drafting_plan_line() kept; dsv41_engine() and placement() added.
ds4_bounds_scan, ds4_moe_sort_count and ds4_moe_sort_fill (ds4_kernels.cu:1938-1943) with the host wrapper that launches them
in order on one command buffer (serial dispatch, so call order is execution order). kPicks = 8: idx is [T][8], q -> (q/8,
q%8); one threadgroup a virtual slot, a block-wide prefix over 8 warps, row_tok/pos written only by the filling lanes.
One Metal fact paid for: simd_ballot returns a simd_vote that does not cast to an integer here. The counts come instead from
simd_prefix_exclusive_sum (a lane's offset inside its simdgroup) and simd_sum (the group's total) - which is a cleaner
formulation than CUDA's ballot+popc anyway. AOT: 20,160-byte AIR, rc=0.
Correction worth recording: the engine's current gate is NOT moe_sort. It still stops at ds4_kernels.cu:1915, i.e. the Engram
row decoder (engram_dequant_tok) - I attributed :1915 to moe_sort two briefs ago and that was wrong. The routing trio is ported
and wired but not yet reached; its turn comes after the Engram decode.
ds4_hc_split is the SCALAR path of ds4_kernels.cu:268: lane 0 of a 32-thread group does the whole token in a local 8x8 matrix -
sigmoid pre/post, row softmax with max shift, +eps, column normalise, then iters-1 alternating row/column normalisations. That is
the same mathematics as the CUDA warp-lane version for hc == 4, without the shuffles: correctness first, the lane-parallel form can
follow if it ever shows in a profile.
The first attempt segfaulted (rc 139) and the cause is worth remembering: scale and base were memcpy'd as if they were plain host
pointers, but every operand the engine hands a kernel is a device buffer - reach it through strata::gpu::host_view() and pass it as
a wrapped Buffer with its offset. With that fixed the engine advanced from :1908 to :1915.
Queue, nine items, each verified by the engine advancing: init_pre, embed, cublasSgemm, finish, hc_post, hc_mix_scale, hc_mix,
hc_split -> next is engram_dequant_tok (:1915), the Engram row decoder whose maths was verified standalone earlier.
Three more kernels, each confirmed the only way that matters: the engine moved past the launch line it had been stopping at.
* hc_post_inplace (ds4_kernels.cu:1677) - h[t][k][e] = post[t][k]*y[t][e] + sum_j comb[t][j][k]*h_old[t][j][e]
* hc_mix_scale (:1692) - mixes[t][i] *= rsqrt(sum(h[t])^2 / hcdim + eps)
* hc_mix (:1901) - mixes[t][m] = (h[t] . fn[m]) * rsqrt(mean(h[t]^2) + eps)
all in ds4_init.metal with host glue in ds4_finish.cpp (which is now the home of the small kernels: it owns the Context, the
pipeline cache and the wrap/offset helpers). Queue so far, seven items, every one verified by the engine advancing:
init_pre, embed, cublasSgemm, finish, hc_post, hc_mix_scale, hc_mix -> next is hc_split (:1908).
hc_split is NOT a transcription: its softmax/sinkhorn is expressed in warp-lane geometry (shfl_xor masks 1,2 for rows and 4,8
across columns over lanes 0..15, for hc == 4) and it loops 'iters' times. A wrong lane mapping yields a plausible but skewed
combination matrix, so it needs a parity test against the CUDA version on synthetic mixes before it is trusted.
ds4_finish.cpp binds the engine's x as a wrapped buffer with an offset, passes Ds4Finish as a constant struct whose pointer fields
are the engine's gpuAddresses, and dispatches one launch with grid = n_vec (the kernel's blockIdx.x is the threadgroup index).
It declines the mrope (rd > 0) and rounding (quant != 0) branches and the KV pack, so those still abort with the launch line
rather than being skipped.
Verified the only way that matters: the engine no longer stops at :1894, it stops at ds4_kernels.cu:1921 (hc_post_inplace_kernel).
Queue so far, each step confirmed by the engine advancing: init_pre, embed, cublasSgemm, finish.
Two plumbing lessons: ds4_finish.cpp had to be added to STRATA_DS4_SOURCES (strata_ds4 does not use the engine-source glob), and
scp with a relative remote path put it nowhere - absolute paths from now on.
ds4_finish (ds4_kernels.cu:1894) with its Ds4Finish struct and the block reduction: warp_sum over the 256-thread group
(simd_shuffle_xor 16,8,4,2,1 into red[8], barrier, second warp_sum, barrier), the RMS path (inv = rsqrt(tot/len + eps), then
r *= inv * w[e]) and the store (BF16 round to nearest even into dst, or back into x).
AOT gate: xcrun metal -c -> 10,032-byte AIR, rc=0. Two Metal constraints cost a round each and are worth remembering:
'long long' is not supported (use 'long'), and a kernel parameter must be a buffer/constant reference, not a plain value.
Not wired yet: the host side still has to bind the Ds4Finish struct (pointers as 64-bit gpuAddresses, strides as long), and the
mrope (rd > 0) and rounding (quant != 0) branches are not written - the host wrapper will decline those until they are, so the
engine names the line rather than skipping a step.
ds4_model.cu:2737 is the engine's first dense call: cublasSgemm(OP_T, OP_N, n_mix, T, hcd, &one, fn, hcd, h, hcd, &zero,
mixes, 32) - mixes[t][m] = h[t] . fn[m], both operands F32, alpha 1, beta 0. ds4_sgemm_tn (one thread an output element) plus
metal_host::sgemm_tn wrap the engine's buffers and dispatch it; cublas_v2.h calls it for exactly that case and keeps aborting
for everything else, so an unported shape fails loudly rather than computing something wrong.
With that the engine passes the dense product and names its next need: finish_kernel<false> (the plain variant - RMS norm,
mrope, optional in-place rounding, no KV pack) at ds4_kernels.cu:1898.
Queue so far, each step verified by the engine advancing: init_pre, embed, cublasSgemm, then finish_kernel<false>.
The DeepSeek engine now loads a real 48,786-tensor pack on the Mac, plans the staging (102.26 GB of experts in unified memory,
5,604 on the CPU lane), and executes until it needs a kernel that has no dispatch. Two are ported:
* init_pre (ds4_kernels.cu:1865) - a three-line initialiser
* embed (ds4_kernels.cu:1873) - the BF16 gather with hc copies
each as MSL in metal/src/engine/msl/ds4_init.metal plus host glue in metal/src/engine/ds4_host.cpp (wrap the engine's own device
buffer through strata::gpu::host_view + mtl::Context::wrap, then Dispatch), wired into the shared call site under
#if defined(STRATA_METAL). The engine advanced past both and now asks for finish_kernel at :1896.
Two infrastructure fixes were needed and are in here:
* runtime.cpp: StreamImpl's encoder/command-buffer state is touched from the loader and the compute path, so two encoders could be
open on one command buffer and Metal asserted ('A command encoder is already encoding'). A std::recursive_mutex around
encoder()/flush()/encode_copy()/encode_fill()/encode_dispatch() fixes it - recursive because encode_copy calls flush (a plain
mutex deadlocked, which cost one round).
* ds4_host.cpp compiles its MSL relative to the context's kernel_dir (metal/kernels), so the engine's own kernels are
../src/engine/msl/<file>.
Known: this build has STRATA_NATIVE_EXPERTS=OFF, so the engine routes all experts to VRAM ('this build has no ggml-cpu'); the
CPU-lane path is untested on the Mac.
metal/src/engine/ds4_ctx.cpp owns one mtl::Context, allocates through Context::alloc and remembers gpuAddress -> Buffer, so the
engine's device memory can live in the same residency set as the dispatches. Off unless STRATA_DS4_METAL_CTX=1, which keeps the
current behaviour and makes the two paths comparable the moment a pack exists. Build rc=0.
It exists because strata::gpu (what metal_compat's cudaMalloc maps to) is a CUDA-shaped facade with its own device state: a dsv41
launch through mtl::Context would otherwise read a foreign residency set. Flipping cudaMalloc/cudaFree to this under the env var
is the two-line change, and it is deliberately left for a session that can test it against a loaded pack.
metal/src/engine/ds4_host.cpp compiles ds4_quant.metal once through mtl::Context::compile, wraps the engine's buffers with
cx.wrap(strata::gpu::host_view(...)), builds a Dispatch (args = buffer, inline ldx, q, d; grid = n/32; tg = 32) and runs it with
cx.run(). Build rc=0 - the port's integration path is now code, not a plan.
It also exposed the last architectural question, which the comments record: the Qwen engine allocates and dispatches entirely
inside mtl::Context (cx.alloc / cx.wrap / cx.run) and uses strata::gpu only for a purge helper, so a dsv41 kernel must be given
the gpuAddress of a Buffer allocated by the SAME Context that encodes its dispatch. Until metal_compat's allocations go through
that Context, the wrapper compiles but would dispatch into a foreign residency set - the plan for that is written up in
RIGS/m1002.md rather than guessed at here.
metal/tests/dsv41/iq2s_t8_parity.mm: block 82 B (f16 d, 32 index bytes, 32 sign bytes, 8 qh, 8 scale), grid index assembled
from qs[4ib+l] plus two bits of qh[ib] (<< (8-2l) & 0x300), signs read at qs+32 in the block, scale alternating db[l/2] -
transcribed from ggml's dequantize_row_iq2_s. Verified: 65,536 rows x 5,120 k x 8-token window, rel L2 2.86e-07, PASS.
Rate 515.71 GB/s effective (64.46 GB/s of raw weight traffic) - the fastest of the three expert formats and 89 % of the
579 GB/s streaming ceiling. With IQ2_XXS (437.5) and IQ2_XS (467.8) the 89 GB of IQ2 weight in the 2.25bpw set now has
verified Metal kernels, which is 165-195 tok/s of headroom on expert traffic alone.
metal/tests/dsv41/iq2xs_t8_parity.mm: block 74 B (f16 d, 32 uint16 qs, 8 scale bytes), grid = iq2xs_grid[qs & 511],
signs = ksigns_iq2xs[qs >> 9], two scales a group (sc & 0xf, sc >> 4) - transcribed from ggml's dequantize_row_iq2_xs and
verified against the CPU reference: rel L2 2.83e-07 at 24 MB and 2.88e-07 at 97 MB, PASS.
Rate: 117.4 GB/s effective on 24 MB (launch-dominated) and 467.8 GB/s effective on 97 MB, which is in line with IQ2_XXS's
437.5. IQ2_XS is 24 GB of the 2.25bpw set and IQ2_XXS 49 GB, so the two largest expert formats are now ported.
metal/tests/dsv41/iq2xxs_t8_parity.mm decodes the weights ONCE per 8-token window and applies them to eight accumulators, which is
how the engine runs decode (DSpark's draft/verify rounds are 4-5 tokens, the prompt window is 8). Same tables, same decode as
the verified one-token kernel, and the reference checks it: relative L2 2.91e-07 on sampled rows, PASS.
The number that matters: 437.52 GB/s effective (54.69 GB/s of raw weight traffic) against 87.71 GB/s for the warp-per-row form.
At 2.33 GB of expert reads a token that is ~5.3 ms of expert time, i.e. ~188 tok/s of headroom - the first measurement that says
the 50 tok/s target is reachable, and it says the expert path must be the T-window form from the start, as the repo's moe_mm is.
metal/tests/dsv41/iq2xxs_gemv_parity.mm decodes IQ2_XXS exactly as ggml does (block 66 B, db = d*(0.5 + (aux32[1] >> 28))*0.25,
grid = iq2xxs_grid[aux8[l]], signs = ksigns_iq2xs[(aux32[1] >> 7l) & 127]) and dots it with the activations. Measured against a
CPU reference that uses the same tables: 65,536 rows x 5,120 k, relative L2 2.88e-07, PASS, 87.71 GB/s.
Two findings that shape the expert path:
* the group headers advance 4 bytes and read 8, so an 8-byte uint2 load is misaligned on odd groups. Both vector-load variants
failed (L2 1.38 and 0.985) and were slower; byte assembly is the correct access pattern, as in CUDA's own decoder.
* 87.7 GB/s is 4.8x slower than the BF16 GEMV's 425 GB/s for the same traffic, because the grid decode costs per byte. At 2.33 GB
of expert reads a token that caps decode near 38 tok/s - below the 50 target - so the expert path must be tiled (one weight read
feeding several tokens, as the repo's moe_mm does) rather than one row per warp.
That is a measured design constraint now, not an assumption.
metal/src/engine/msl/ggml_grids.h carries kmask_iq2xs[8], ksigns_iq2xs[128], iq2xxs_grid[256], iq2xs_grid[512], iq2s_grid[1024],
iq3xxs_grid[256] and iq3s_grid[512] as MSL constant arrays. metal/tools/gen_grids.sh produces them from
third_party/ggml/ggml-common.h (3cf03257f, GGML_TABLE_BEGIN/END blocks), so the Metal expert kernels use bit-identical tables
to CUDA instead of a hand copy - the same discipline that caught the half-rounded-scale error in the Q8 kernel.
Every expert format in the 2.25bpw set (IQ2_XXS 49 GB, IQ2_XS 24 GB, IQ2_S 16 GB, IQ3_XXS) is grid-based, so nothing in the
expert path could be written without these.
metal/tests/dsv41/q8_gemv_parity.mm: Q8_0 blocks (32 int8 + one f16 scale) times F32 activations, one warp a row, lane-strided
blocks. 16,384 rows x 5,120 k, relative L2 8.11e-08, PASS, 173.52 GB/s and 326.62 GFLOP/s (0.514 ms for 89 MB).
The first run measured 1.85e-04 and looked like a kernel bug; it was the CPU reference using the exact float scale while the
kernel reads the half-rounded scale from the file. Matching the rounding took it to 8e-08 - the same class of drift that would
quietly cost quality if the port did not reproduce the file's arithmetic exactly.
Three kernels are now verified with reproducible parity tests: the Q8 activation quantiser, the BF16 GEMV and this one.
metal/tests/dsv41/bf16_mv_parity.mm compiles the MSL version of the BF16 matrix-vector product (ds4_gguf.cu:1043's
mx_mv_kernel, single job: one warp a row, lane-strided k, warp reduction), dispatches it on the GPU and compares against a
double-precision reference: 2048 rows x 5120 k, relative L2 1.80e-07, PASS.
It also prints throughput: 83.19 GB/s and 83.2 GFLOP/s, 0.252 ms for 21 MB of BF16 weights. That is a naive kernel (no
threadgroup tiling) against a GPU that streams 579 GB/s in rt_bench, so 7x of headroom remains - and the number matters:
2.33 GB of expert reads a token at 83 GB/s caps decode near 36 tok/s, which is exactly why the expert path has to use the
tiled multi-token kernels (the repo's gemv/moe_mm family) rather than a warp-per-row matvec.
A second measurement lesson: per-row relative error of ~1e-3 on a 5120-term float dot is expected cancellation, not a bug;
the standard measure is relative L2 over the result, which is what both parity tests now use.
metal/src/engine/msl/ds4_quant.metal carries the Q8 quantiser that metal/tests/dsv41/q8_quant_parity.mm proved bit-exact (4096
blocks, 131,072 values, 0 differences). It is the first dsv41 kernel in the engine's own MSL directory, next to the Qwen
engine's mtp/ple kernels, so the runtime loader will find it by name when the launch is wired.
The comment records the driver quirk that cost the debugging: threadgroup_position_in_grid arrived as 0 in every threadgroup
on this Mac, so the block index comes from thread_position_in_grid / 32.
metal/tests/dsv41/q8_quant_parity.mm compiles the MSL version of q8_quant_kernel (ds4_gguf.cu:201) at run time, dispatches it
on the GPU and compares every int8 against the same arithmetic on the CPU. 4096 blocks, 131,072 values: 0 differ, worst
scale error 4.6e-04 (half rounding). PASS.
Two bugs found on the way, both in the test harness rather than the kernel: indexing with thread_position_in_grid where the
threadgroup id was meant, and passing ldx = 1 where the stride between blocks of 32 values is 32 (which made block b read a
window starting at b instead of 32b - the giveaway was block 1 reproducing block 0's values shifted by one).
This is the pattern every remaining kernel follows: an MSL function, a pipeline from it, the args as buffers, one dispatch,
and a CPU reference in the same program.
strata_ds4_launch is a macro that passes __FILE__ and __LINE__ to strata_ds4_launch_at, so the abort message reads
'src/dsv41/ds4_iq.cu:1234' rather than a generic line. Each run of the engine is then an exact work item instead of a guess
about which of the 146 launches is next, and the message is machine-readable from a log. Build rc=0, strata_ds4 = 1,412,984 B.
Two things made this possible:
* the launch syntax. 146 kernel launches (119 in the sources, 27 in the vendor tree) are now
strata_ds4_launch(kernel, grid, block, shared, stream, args...), a call a C++ compiler can parse, with a macro that forces
the global name past argument-dependent lookup and a body that stops the run naming the kernel until a Metal dispatch exists.
Every source then compiles as C++ through include/metal_compat - .cu files included, since the Metal project has no CUDA
language and was silently skipping them.
* the last CUDA names the kernel files needed: __grid_constant__, cudaDataType, __umulhi/__mulhi, the bitwise atomics,
__halves2half2 / __half2half2 / __low2half / __high2half / __low2float / __high2float / __hadd2 / __hmul2 / __hfma2 /
__hneg2, the occupancy query and the stream-capture status.
The EXL3 legacy path (v1.1.8's kernels, NVIDIA before sm_80) is defined in metal/src/engine/exl3_legacy_stub.cpp: four of its
seven entry points abort with their name, three answer as the declarations require. The 2.25bpw text model does not use EXL3.
Result: build rc=0, 0 undefined symbols, strata_ds4 = 1,410,552 bytes, Mach-O arm64, and it runs (its own usage text). Next
is the run-time loop: a pack reports the first kernel it needs, implement it in MSL, repeat.
The DeepSeek host sources now build inside the Metal project as their own executable, with the .cu files compiled as C++ through
include/metal_compat. ninja reaches step 23/35 and ds4_model.cu (8,232 lines, host only) builds; the first failure is a <<<>>>
launch in ds4_gguf.cu, which is the port's real boundary.
The measured size of the remaining work: 101 kernel launches to convert to Metal dispatches and 78 kernel bodies to write in MSL
(text only; the vision tower adds 21 and is optional). ds4_model.cu has zero of both.
Two gotchas from building it: run_locked.sh returns exit code 69 for builds that are fine (run ninja directly), and the Xcode
licence line on stderr is noise - the toolchain compiles without accepting it. Run with -j8 directly, the host sources compile in
about a minute.
A full-list compile (-ferror-limit=0) found 350 errors behind the 20-error display cap, and they collapsed into twelve missing
families. With dim3, __threadfence_system, __funnelshift_r/l, rsqrtf, __expf/__logf/__sinf/__cosf, sincosf, __ldg, __ldcv,
__float2int_* and __shfl_up_sync in cuda_device_builtins.h, plus the C math names in scope, the counts are
ds4_iq 52, ds4_gguf 42, ds4_kernels 132, ds4_vision 27 (was 66/67/185/32) AND the distinct-names list is now EMPTY: what is
left is not a missing name but device construct - the <<<grid, block>>> launch syntax and shared-memory declarations that a
C++ compiler cannot parse as written.
That is the boundary of the host-compile audit, which was its purpose: the API surface is complete, and the remaining work is
the MSL implementation itself.
Twenty rounds of 'compile ds4_iq.cu as C++ and fix the first name it cannot find' produced the complete list, and it is small:
__byte_perm and the bit/int intrinsics, the SIMD video instructions with their exact per-lane semantics (__vsub4,
__vsubss4, __vadd4, __vaddss4, __vcmpeq4, __vcmpgtu4, __dp4a), the rn float helpers, integer min/max, __align__, the
vector types beyond float2/float4, the CUDA device built-ins for the host compile (threadIdx/blockIdx/__syncthreads/__shfl_*
- labelled as host-only, nothing there is a kernel), and wmma in CUDA 12's tag style whose every call aborts naming the
simdgroup-matrix equivalent.
ds4_model.cu stays at 0 errors. The four kernel sources are at the compiler's 20-error display cap; what is left behind it
is mma_sync's signature and single scalars (__float2int_rn), not structure. Next is the MSL work itself.
ds4_numerics.cuh names __NV_E4M3/__NV_SATFINITE and __half_raw, ds4_q6k.cuh names __fmul_rn/__ushort_as_half, and both kernel
files include mma.h: cuda_bf16.h now carries the fp8 enums and includes cuda_fp16.h (for __half_raw), cuda_fp16.h carries the
five rn helpers and __ushort_as_half, and mma.h says what it is (the wmma surface does not exist on Metal; the token path
does not use it, and an instantiation fails loudly).
State after this: ds4_model.cu 0 errors; the four kernel files are at the compiler's 20-error limit with a shorter list each -
the remaining class is the device code itself (block decoders, shuffles, shared memory), which is the MSL work.
The kernel sources' single remaining error was cuda_fp16.h not existing; with it, vector_types.h and cuda_fp8.h in place they
get past the wall and show their real errors - ds4_iq 7, ds4_gguf 7, ds4_kernels 20, ds4_vision 20 - which is now the work
queue for the Metal port. ds4_model.cu stays at 0.
Both copies (this branch and the box) carry the box's version of cuda_fp16.h; an earlier local edit had failed on a path and
this commit syncs them.
The last layer of the compat surface: the CUDA function-space qualifiers (__host__/__device__/__global__/__shared__/
__constant__/__forceinline__/__launch_bounds__), cpu_set_t's CPU_ZERO/CPU_SET for the macOS branch of ds4_os.hpp,
CUBLAS_GEMM_DEFAULT_TENSOR_OP, a cudaMemcpy2D that reports failure rather than copying nothing, and a cublasGemmEx that takes
any argument list (the engine passes cuBLAS's full 19-argument spelling) and aborts naming kernels/dense_mm.metal.
With that, src/dsv41/ds4_model.cu - 8,232 lines, host only - compiles against metal/include/metal_compat on the M5 Max. The
four kernel sources are one error each, and the kernel work itself (MSL bodies for 27 + 76 + 33 device functions, plus the
dense GEMM and a pitched copy at the runtime) is what remains before a first token.
The include had landed between '#ifndef NOMINMAX' and '#include <windows.h>', so on a Mac it was never reached - that is why
the audit still showed every Linux name undeclared. Moved after '#pragma once': ds4_model.cu goes from 20 to 15 errors.
The 15 left are four CPU_ZERO/CPU_SET uses (macOS has no cpu_set_t macros) and the __host__/__device__ qualifiers that
cuda_runtime.h does not define yet; both are a few lines and are the next step, then the Linux surface of the engine's
storage helpers is done.
ds4_os_apple.h maps O_DIRECT (-> 0, F_NOCACHE does the work), posix_fadvise (-> 0), mincore (-> ENOSYS, 'unknown' is the
answer the header's contract already allows), MADV_HUGEPAGE and SYS_mbind (no-ops) and the CPU-affinity calls (no-ops on a
kernel that schedules on its own), and ds4_os.hpp includes it at the top under __APPLE__.
State: included, but ds4_model.cu still reports the same 20 errors at the same uses, so something in the include order is
not taking effect yet - the next session's first job is one -E on ds4_os.hpp to see which definitions win. Not claiming it
works.
Compiling src/dsv41/*.cu against metal/include/metal_compat as C++ is how the DeepSeek-on-Metal port is scoped, and it is
much smaller than assumed: ds4_model.cu is 8,232 lines of *host* code with zero device markers, and the six non-kernel
sources already compile. What was missing at the compat layer:
* cuda_bf16.h - __nv_bfloat16/__nv_bfloat162/__nv_fp8_storage_t and the four conversions the engine uses
(__float2bfloat16_rn, __bfloat16_as_ushort, __nv_cvt_float_to_fp8, __nv_cvt_fp8_to_halfraw), E4M3 with round-to-nearest.
* cublas_v2.h - the nine GEMM calls (7 cublasGemmEx, 2 cublasSgemm) and their enums. The dense GEMM itself is NOT wired:
it aborts with a message naming kernels/dense_mm.metal (which exists and passes its parity test) rather than returning a
silently wrong result.
After these, ds4_model.cu's remaining 20 errors are all in ds4_os.hpp - the Linux-only OS surface (O_DIRECT,
posix_fadvise, mincore, MADV_HUGEPAGE, SYS_mbind, cpu_set_t/sched_setaffinity) - which needs a __APPLE__ branch.
T tokens in tiles of 16 and rows in groups of 8 (activation slice and accumulators in L1): 7.7-9 ns per block and token on IQ2/IQ3 rows against
28-60 for ggml's dot (x3.6-6.6), IQ1_M 16 ns (x1.8); bit-identical to ggml (ds4_gg_bench --exact at 7, 8, 16, 33, 64 tokens, all six types).
gg_cpu_gate_up takes up to 64 tokens a call (and any number in pieces); ds4_gg_bench --expert times the whole expert pipeline.
strata_kernels_cpu compiled expert.cpp / q2_avx2.cpp / iq_avx*.cpp with hard-coded -mavx512f / -mavx2, so on an arm64 host the
build died with 'unsupported option -mavx512f for target arm64-apple-darwin27.2.0'. STRATA_CPU_KERNELS_X86 now decides:
on x86 the same sources and ISA flags as before, elsewhere the portable files plus src/kernels/cpu/iq256_portable.cpp, which
defines iq256_supported() = false and iq256_rows() = abort (callers check the first), so ds4_gg_cpu.cpp keeps ggml-cpu's
rows - which have their own NEON kernels on Apple Silicon.
Verified on the M5 Max: configure rc=0, build rc=0, ds4_gg_bench runs IQ3_XXS at 83.9 GB/s through ggml-cpu's rows (the
x86 EPYC does 60.5), 0 of 1,567,344 values differ in bits against ggml and 7.14e-09 against a double reference. The x86 path
is untouched: the guard only selects which sources join the library.
The layer-major chunked prompt path only ran with --threads on / --tp > 1. --prefill-chunk N (N > 0) now enables it on one card too (kChunkedPromptDefault stays false until the decode gate is signed off). A chunk's routing bumps the layers' heat and CPU-lane hits, and cache_settle() runs the swap rule in rounds after the prompt so the decode that follows finds the hot experts in VRAM (8K prompt: 1699 swaps, 5.6 s). If the chunk buffers do not fit, the chunk is halved down to 256 rows, then windows of 16 with one line saying so. ds4_main: --window 0 (default) feeds a whole prompt when the chunk path is on.
The rows are ggml-cpu's vec_dot bit for bit at any token count, so the default cannot change an answer; a one-time per-type check against the
ggml-cpu build in use turns them off if they ever differ (DS4_GG_IQ256=0 never, =1 always). EX rows: 5 and 6 tokens in one decode pass
(were 4+rest), independent integer chains for 1-2 tokens, IQ1_M: delta dot from per-call pair sums of the activation, group signs from one
shifted load. The main engine's iq256_rows hash is unchlanged (phaseA-tests/cpul/hash_iq256.cpp).
The main engine's iq256_* rows keep their float order (EX=false, the default; hash of their outputs on all types and 1..8 tokens is
identical before/after, phaseA-tests/cpul/hash_iq256.cpp). The new iq256x_rows (EX=true) sums as ggml-cpu does - block scale f16(d)*y.d, the
format constant after the sum, ggml's hsum - so a token's dot no longer depends on how many tokens share the pass (the greedy ids of a
DSpark run changed with DS4_GG_IQ256=1 because the old order differed in the last bits). IQ1_M (29), single-token in ggml, gets a
multi-token AVX2 row. ds4_gg_bench --exact: 0 differing bits for IQ1_M, IQ2_XXS/XS/S, IQ3_XXS/S at 1..8 tokens.
Conflict decisions (win side = f-dsv41-win, port side = f-dsv41-port):
- ds4_iq.cu: kept port. Its IQ1_S/IQ1_M/IQ4_NL kernels are verified bit-exact on HIP gfx1201, sm_86 and sm_120; win's
IQ1_M table load and supported() list are a subset (IQ1_S, IQ4_NL added).
- ds4_kernels.cu/.cuh: kept port's engram_dequant fmt int (0 MXFP8, 1 Q6_K, 2 IQ4_NL) and its iq4nl_value, replacing
win's q6k flag and iq4nl_value; same arithmetic.
- ds4_model.cu: kept port's eg_fmt vector and its Engram type check (unknown type exits 2 instead of segfaulting);
win's eg_q6k is gone, the duplicate IQ4_NL row branch from the textual merge is removed.
- ds4_packinfo.cpp: kept port (device types derived from iq::supported()).
- iq_expert_test.cpp: kept port (hip_or_cuda.h); win's tests/dsv41/hip_as_cuda.h removed (nothing else used it).
The IQ1_M test-d placement is the same bytes in both versions.
- CMakeLists.txt: kept port's test-language block (HIP, CUDA or CXX for the CPU-only build); win's
_strata_dsv41_test_lang variable dropped.
- pack_gguf.py: kept port's type table (IQ1_S/IQ1_M comment) and combined both missing-Engram checks
(has_part from win's split Engram layout AND port's dense-override names).
Deriving their engram.gate from their k*q and transposing their wkv scored 0.409 then 2.996 KL - both my arithmetic, not
their data (their k/q/wkv are stored transposed and the gate is not k*q in the engine's layout). --gate-kv-from <pack>
copies those two tensors from a pack that is known good, so a pack can carry their ple table and nothing else of theirs
differ. That is how the comparison the plan asks for was finally made (numbers in EACH-PC-PLAN/R9700-BOX.md, step 2):
their 110.6 GB IQ4_NL table scores KL 0.16787 / ppl 3.627 against 0.16502 / 3.588 for our 203 GB MXFP8 one - +0.003 KL,
inside this engine's run-to-run placement noise - while saving 92 GB of disk.
Also fixes the fold's reads: Gguf's tensor offsets are relative to the shard's data section (pack_gguf adds data_start),
so the fold must add it too; without that every tensor came out as NaN.
tests/dsv41/{iq,tq}_expert_test.cpp included <hip/hip_runtime.h> unconditionally, which no CUDA toolkit ships - every CUDA
box had to pass a hand-written shim on -I (found on l620 and z440, 2026-10-23, and reported as the one thing worth
upstreaming). include/strata/dsv41/hip_or_cuda.h now provides the handful of HIP names those tests use on a HIP build and
maps them onto the CUDA runtime otherwise (hipMalloc/Free/Memcpy, device sync, the error type and the events).
Verified on r9700 that the HIP build is untouched: incremental build rc=0, ds4_iq_test --type 29 ALL OK, ds4_tq_test ALL OK.
Their engram_k/q/wkv are transposed against the engine's layout (wkv [6144, 25600] against [25600, 6144], k/q [5120, 4]
against a [4, 5120] gate), so the fold now transposes and writes the engine's shape. The pack builds (their IQ4_NL table,
all files present) and runs without crashing, but it scores KL 0.40915 - and that is this tool's bug, not their data: a
direct read of their BF16 engram_k through Gguf's offsets already comes out as NaN, while the engine reads the same tensors
through a pack fine, so the fault is in how this code turns an offset into bytes (the Q5_K dequant inherits it).
Do not quote 0.409 as the IQ4_NL-vs-MXFP8 comparison; that measurement is still open. Next step is to read the tensors the
way pack_gguf writes them into tensors.txt and check one row against the engine's own dump.
Their sets keep the Engram as three per-layer tensors (blk.N.engram_k/q/wkv) while the engine wants engram.gate (k*q
folded) and engram.wkv; their engram-IQ4_NL.gguf carries only the ple table, so a pack built around their Engram stopped at
'blk.1.engram_kv.weight: missing'. engram_fold.py folds the gate and renames the kv into a dense_requant-style folder, and
pack_gguf now treats a tensor a --dense-override supplies as present (it checked the quant's shards and the Engram set only).
Result: packs/ista_2.00_native - 48784 tensors, their IQ4_NL table, Engram layers 1,14, 2896 Engram reads for a 32-token
prompt, no crash.
NOT yet a measure of their table: it scored KL 0.40915 / ppl 4.769 against 0.16502 / 3.588 for the same set with our MXFP8
table, because their engram_k/q/wkv are stored TRANSPOSED relative to the engine's layout (their engram_wkv is [6144, 25600]
against [25600, 6144]; their k/q [5120, 4] against the [4, 5120] gate). Copying the bytes as they are yields a valid pack
with the wrong data. engram_fold.py still needs the transpose (gate in float; the Q5_K kv dequantised, transposed, written
BF16) - Phase 1.1's 'load their layout as published'.
It was a hand-kept copy of the supported() switch and fell behind when IQ1_S, IQ1_M and IQ4_NL got kernels: the 2.00 set's
60 IQ1_M expert tensors (59.5 GB) were reported as 'no GPU kernel ... the CPU lane takes them' while the kernels were in the
same binary and ds4_iq_test passed them. That is exactly the line an operator trusts before a download, so it now walks
ids 0..63, asks iq::name()/iq::supported(), and adds MXFP4 and the trellis types by hand (they are not in that switch).
Verified: packs/ista_2.00bpw_9e2528f and packs/h225 both report 'no GPU kernel: none'. Also fixes the intake's eval call
(session16.sh takes the pack directory name, not a path - the 2.00 score had failed with an empty runs/s16_<path>.tsv).
Found and fixed on p620, the only box with their engram-IQ4_NL.gguf (110.6 GB). Before this, a pack built around their
file loaded for 38 s and then died in Model::Impl::eg_issue at the first token (3 of 3 runs, addr2line 0x6d3450).
- Model::Impl::load: one guard line and exit(2) for an Engram tensor whose type the reader does not handle, naming the
layer and the type (verified with a pack retyped to IQ4_XS, rc 2; a missing embed tensor also exits 2). The guard stays
for the next unknown type.
- eg_q6k becomes eg_fmt per layer: 0 MXFP8 ([scale, 32 codes] blocks, 264 B a row), 1 ggml Q6_K (210 B a 256), 2 ggml
IQ4_NL (18 B a 32: an fp16 d and 16 nibble bytes), 3 an old pack's split FP8. The load loop sniffs the type from the
pack's tensor table, eg_issue builds the row descriptor from it (144 B a row, off + id*144, no scale tensor), and
EgReader::read() needed no new branch: its contiguous, scale-free path already served Q6_K.
- ds4_kernels.cu: engram_dequant_kernel and engram_dequant_tok_kernel take the format and add the IQ4_NL branch, a literal
port of ggml's dequantize_row_iq4_nl (fp16 d per 18-byte block, low nibbles 0-15, high 16-31, kvalues_iq4nl).
- The Engram row slot stays cols*(hd+hd/32) for every format, so existing packs are byte-identical: --pack-info on the
MXFP8 h225 pack and on a Q6_K zero pack is unchanged, and a Q6_K pack's 32-token ids are identical old against new.
Verified with their real table: the pack that segfaulted runs 32 tokens in 1692-1711 ms (18.1-18.3 tok/s) and produces
coherent Python (GGML_TYPE_Q2_0 = 42, GGML_TYPE_NAMES: dict[int, str]); 64 tokens on the code prompt, DSpark off 17.16
and on 23.36 tok/s (44 of 53 drafts accepted). IQ4_NL decode is bit-exact against ggml on 8 of 8 sampled rows, and the row
addresses match the pack's tensor table (data_start 320, 144 B a row).
Not covered: no non-zero MXFP8 table existed on the box to re-check that path in-engine (the Q6_K and synthetic-zero MXFP8
packs were), there is no in-engine float dump against the Python reference (the proof is formula- and address-level), and
the legacy split-FP8 format (3) is kept but untested.
1. CMakeLists: -DSTRATA_BUILD_DSV41=ON was silently dropped when no GPU backend was on, because the whole DSV41 block
sits inside if(STRATA_ENABLE_CUDA OR STRATA_ENABLE_HIP). Found on p1004 (2026-10-23), whose P100 is not a DeepSeek
target but which is exactly the box the CPU-lane A/B belongs on. A new branch builds ds4_gg_bench with no GPU
runtime at all. Verified on r9700 with both backends off: configure rc=0, build rc=0, the benchmark runs IQ3_XXS at
60.5 GB/s (ggml-cpu) against 64.1 (AVX2 rows, x1.06) and 8.02e-09 against a double reference.
2. tools/dsv41/ista_intake.py: the pack step now uses the recipe that actually builds a loadable pack for these sets -
--engram-meta (the UD set's first shard carries the Engram hash parameters, without it pack_gguf sees no Engram layers
and refuses per_layer_token_embd), --mmproj, the combined --skip-tensors regex, and the DSpark noise token and markov
rank (pack_gguf fills the rest of the geometry from the drafter but not those two, so a pack came out with
dspark_noise_token_id 0). Verified end to end on the 2.25 set: the intake rebuilds packs/h225's content exactly -
same 48794 tensor lines, --pack-info 'files: 10, all present', 'no GPU kernel: none', dspark geometry 5/128799/256/128/3.
Found and implemented on z440's CUDA bring-up, applied here and re-verified on HIP.
ds4_iq.cu: decode<T>/deq8<T> for kIQ1_S, kIQ4_NL, kIQ1_M, plus iq1_codes() (iq1s_grid_gpu's 4-bit nibbles -> the int8 codes
8*(n-1)+-1 through two __vsub4) and iq1m_d() (IQ1_M carries its F16 d in the top nibbles of its four u16 scales - there is no d
field). kQK<T> is 32 for IQ4_NL as well as MXFP4; load_tab uses iq1s_grid_gpu / kvalues_iq4nl; DS4_IQ_SWITCH and
supported() cover the three types, so they leave the CPU lane and join the device path.
tests/dsv41/iq_expert_test.cpp: make_mat now writes IQ1_M's scale where the format keeps it (bytes 49/51/53/55, high nibbles).
Without that the derived d is random and roughly 1/32 of the blocks come out NaN, which std::max then hides.
Parity (z440, one A4000, ds4_iq_test --type 19,20,29): token gate_up 6.70e-08 / 2.04e-08 / 5.97e-08, token down 3.62e-08 /
4.71e-08 / 3.71e-08 against ggml's to_float and a double reference; CPU lane 3.4e-8..8.2e-8; F16 prompt tiles inside their
own tolerances. Bandwidth: IQ4_NL 376 GB/s (84% of the A4000's 448, bandwidth-bound), IQ1_S 214, IQ1_M 202 (both
compute-bound at 0.92-1.10e12 weight-values/s against IQ2_XXS's 1.22e12).
Not covered: no end-to-end run with a pack that uses these expert types (none on z440); a combined-seed run put IQ1_M's
light-path gate_up at 1.10e-2 against its 1e-2 tolerance, which is the int8 activation rounding of that path (it does not
move with tensor cores off), not the dequant.
1. ds4_tq.cu:262 used the two-argument __shfl_xor, which is HIP-only; the rest of the tree already uses
__shfl_xor_sync(0xffffffffu, v, lane), which both backends have.
2. tests/dsv41/{tq,iq}_expert_test.cpp were forced to LANGUAGE HIP, so a CUDA configure failed at generate time
('The language HIP was requested for compilation but was not enabled'). The language now follows the enabled
backend: HIP, CUDA, else CXX.
Verified on r9700 that the HIP build is untouched: incremental rebuild rc=0, ds4_iq_test --type 17 ALL OK, the engine
--pack-info clean. Verified on z440 (CUDA 12.9, sm_86) by the bring-up agent: HEAD builds, ds4_iq_test ALL OK for
IQ2_XXS/XXS/IQ2_XS/IQ2_S/IQ3_XXS/IQ3_S/IQ4_XS at 5e-8 against ggml, 272-369 GB/s on one A4000. IQ1_M, IQ1_S and
IQ4_NL still have no CUDA kernel (CPU lane only) - that is the Phase 5 kernel list.
X01: TestStartFailed has one text builder (TestStartFailed.plain): what did not work in plain words, that nothing was installed and the current engine is unchanged, and one next step (`strata update --force`, or a plain retry when the card was busy). The engine's own words stay in .reason, the state file and the JSON. The app's message matcher also accepts the new wording. Hints that a caller passes, such as the one for a busy launch, are used as they are, so the launch-holds-the-card line of f-023-tui1 composes with it.
#18: `components rollback` when an earlier rollback is still in force says the engine is already the previous one and what to run, instead of "0.1.45.3 -> 0.1.45.3".
#15 (K05): max_tokens, max_completion_tokens and max_output_tokens below 1 are a 400 naming the field (they used to mean "no cap"); an unknown model id is still answered by the loaded model, now documented in DETAILS.md.
#11: the refusal of a feed older than the one already accepted now says both sequence numbers, the folder that holds the newer one (.updates) and one next step (use the newest release folder, or delete state.json to take the older one). install.ps1 and install.sh print the whole check output, not its first 6 lines, and no longer blame a missing signature when the check gave a reason.
#12: tar.exe got the path through the ANSI/OEM code page and failed on accented letters. The installer now runs Windows' own bsdtar (System32 first, so a GNU tar from Git for Windows cannot be picked up for a zip) with ASCII relative arguments from inside the data folder and renames the result, so no non-ASCII path ever crosses that boundary. Tested on C:\...\<U-umlaut>n<i-umlaut> test with a stub release folder; the old script fails there with "could not chdir".
plan_import now returns missing_shards (the file names of the split set that are not in the folder). problem_words and plain_issues say "<name>-00002-of-00002.gguf is missing" instead of "some parts of the model are missing"; more than two are counted. The three call sites (models import, strata launch --model-path, the Model screen's own-files field) pass the list.
#6 (A07/B03): a model that is not installed now counts the draft layer (5.2 GB) and the picture reader (0.9 GB) in the one place (modelstate.assess), so the Folder step, the confirm and --plan agree (Gyro-S 58.5 + 5.2 + 0.9 = 64.6 GB). Gyro joins the families that get the helpers. --mtp off and --drafting off drop the draft layer from the count and from what setup is told. setup.py's disk check counts the draft layer it is about to fetch.
B03: --plan keeps "engine_missing" for no engine at all; an installed engine that fails the check says so in plain words and adds engine_usable.
B10/E04: with --drafting off (or an installed config that says off) the written run config carries "mtp": false and no --mtp / --spec-min-p arguments.
The Terminals before 0.2.1.3 (0.2.0-beta, 0.2.1, 0.2.1.1, 0.2.1.2) install a newer Control and report "updated", but their launcher has no hand-over, so the old Control keeps running. Their updater test-starts the staged Control with --version (STRATA_NO_ACQUIRE=1, no STRATA_HANDOVER, from the .s staging folder) and shows the first 120 characters of the answer when that fails. Control now answers that start with one plain sentence and exit 7, so the old updater keeps the old Control and prints the sentence instead of "updated". Every updater that can hand over sets STRATA_HANDOVER and is not touched. Release note added.
Cargo.toml and Cargo.lock, desktop and host package.json and lock, PRODUCT_VERSION, CMake project and Metal STRATA_VERSION, build_hip_linux PKGVER default, About comments, tests that pin the number, Docker image tags and docs (0.2.3-<flavor>), download names in the READMEs, AMD_HIP.md and UPDATES.md, AGENTS.md, the bug report placeholder. The engine moves too: it shares the CMake number and its source changed since 0.2.2 (DFlash2, verify and prefill paths). A new test checks every place agrees and that a 0.2.2 install (Control 0.1.47) is offered Control 0.1.48.
The release runbook promises the pack/requant builder checks free space before it writes (DEEPSEEK-RELEASE.MD 9.6 item 5);
it did not. --dry-run now prints 'space: N GB planned, M GB free on <fs>' and returns 3 when 98% of the free space does
not cover the plan, so 'requant_dry && requant' cannot start a run that will fill the disk; a real run raises instead.
With --resume only the bytes still missing are counted against the free space.
Verified on the box with two recipes that do not fit /data today: rq3 (224.3 GB planned against 128.6 GB free) and rq_gu3
(251.2 GB against 128.6 GB) both refuse with the numbers printed, rc 3 in dry-run. pack_gguf's own output is a few MB
(tensors.txt, engram.bin, tokenizer) and needs no guard.
The rule from DEEPSEEK-RELEASE.MD 6.5.2 - every ggml type works, a tensor whose type has no GPU kernel stays on the CPU
lane with one warning instead of stopping the load - is now implemented for IQ1_S (19), IQ1_M (29) and IQ4_NL (20):
- iq::known(t) (a layout exists: block_values/block_bytes/name) is separated from iq::supported(t) (a device kernel
exists). type_row_bytes/type_block use known(), so a pack lays out such a tensor instead of refusing it;
gg::type_on_device(t) is what the VRAM tier tests.
- the CPU lane takes them: gg_cpu_type() covers 10-14, 16-23 and 29 (ggml-cpu's row dots).
- a layer whose experts are such a type is flagged cpu_only at load, printed once ('no GPU kernel for it, its experts
stay on the CPU lane') and excluded from the VRAM budget in place_experts().
- iq::block_values(IQ4_NL) = 32 (QK4_NL): the only block here that is not 256 values. Without it the CPU lane read
IQ4_NL at the wrong row stride (2.4e-02 and 1.4e+05 error against the reference).
Verified: ds4_iq_test --type 29,19,20 runs the CPU lane against ggml's own decode and a double reference - IQ1_M
1.6e-08 / 4.6e-08, IQ1_S 5.4e-08 / 4.0e-08, IQ4_NL 3.1e-08 / 3.2e-08, ALL OK; the device types still pass; --pack-info
reports the layer and role for each such tensor. A full model with real IQ1_M weights has not been run (none on the
box): the arithmetic of the lane is what is verified.
Owner decision 2026-10-22. setup_intel no longer switches pictures off for a B-series card; it prints the line instead (known risk: the first-picture xe reset on the B70, 2026-10-19). strata launch gets the same line as a warn-level note (Computer.new_card_note) and its picture default for an Intel engine now follows the CPU-reader rule at any VRAM size (it used the GPU reader for a 32 GB B70). Gyro adoption on Arc was already allowed in this tree (adopt.FAMILIES). Tests that pinned the opt-in are updated; docs/INTEL.md, packaging/README.md and docs/GYRO.md say B-series is supported and untested by us.
Same failure on v0.2.2 (011c9e600): the failed new start is refused for its port as intended, the old config is relaunched, but it was still loading when the 5 s startup wait ended, so rolled_back was False and the left-over process held the temp folder at teardown.
Today a pack built by another tool had dspark_block_size 0: DSpark ran one token a round and was worth +0.8% instead of
+28-77%. The report now shows block_size / noise_token_id / markov_rank / the drafter's expert counts, the mtp tensor
count, and the Engram layer ids, with a <- marker when the block size is 0 or there is no drafter. Verified on h225,
c0q3hQ6_K, c0q2d5 and mxfp4 (all: block 5, noise 128799, rank 256, 1242 mtp tensors, engram layers 1,14).
The pack is pointers: ds4.txt + tensors.txt + the weights where model_dir says. Today /home/niko/dsv41-c0/t3 was empty
and the c0q3 pack (all IQ3_XXS) described itself perfectly while being unable to load. --pack-info now stats every file,
compares the length with the largest tensor end, prints MISSING/SHORT lines and exits 2 when any is bad; a healthy pack
prints 'files: N, all present and long enough'. Verified: h225 10/10, c0q3hQ6_K 19/19, c0q2d5 12/12, c0q3 fails with
six MISSING lines.
The IST-DAS 2.25bpw GGUF differs from ours in five ways that stopped pack_gguf, each found by building its pack:
- geometry(): ALIASES for the keys they spell differently (kv_source_layer_ids, index_source_layer_ids, the indexer's
candidate_*, dspark_target_layer_ids, engram_layer_ids). Without the source ids the engine refused the pack
('layer 2: compress ratio 2 without a KV source of that ratio before it').
- --engram-meta FILE: their GGUF carries no Engram hash parameters (no ple.*/engram.layers, no token map, no
multipliers/primes/offsets - those are tensors of integer type in their file); the parameters come from a GGUF of the
same model that still has them, the tables from --engram.
- integer tensors (I8/I16/I32/I64/F64) are left out with a line instead of stopping the build (engram_token_map.weight,
engram_primes.weight, ...).
- --skip-tensors REGEX: their attn_output_a is one interleaved 3-D tensor ([8,1024,4096] ne) that this engine's
o_groups slices cannot take; our Q5_K requant of the same BF16 weights supplies it through --dense-override, which
now also *adds* a tensor the quant leaves out (it could only replace before).
- the DSpark geometry (dspark_block_size / n_routed_experts / num_experts_per_tok / target_layer_ids / markov_rank) is
filled from the drafter's dflash.* keys and the markov tensor's shape, and --dspark-noise-token sets the model's noise
token (128799). Without it d_block stayed 0 and DSpark ran one token a round: h225 measured 16.77 tok/s off vs
25.93 on (3.16 tokens a round) after the fix, 1.01 before.
- The Models screen gets one more column in the context row, the existing Select, with the four choices; DFlash2 is offered for Qwen3.8-Flash-Next
(the qwen and gyro families) only. The choice is saved with the model (a setting the app owns, like Draft layer) and shown again next time.
- Before a load the host asks Control (strata drafting plan / fetch --if-needed): when the choice would use the DFlash2 drafter and its file is not
here, it is downloaded pinned and checked and packed, with plain progress in the load. A drafter that cannot be had never stops the load: the MTP
starts and the job says so. The download plan dialog lists the drafter's size when it is chosen.
- What ran is told in the engine's words under Details and below the row: "Drafting: MTP (DFlash2 didn't fit)" and the other three reasons, all in
the eight languages. Off is the draft layer's own switch.
- The app never reads the suggestion table: Control decides (strata_control/drafting.py), so the Terminal and the app cannot disagree.
Logic and tests ported from #3578 by medking82. The alert stays red and in place; only its words change for the restart interruption of an update that a later update of the same component finished. Sentence-case text in all eight languages; the PR's screenshots and validation files are left out.
Co-authored-by: medking82 <[email protected]>
The overflow was found and reported in #3601 by G0-0000. Its 75% column made the inputs narrower at every width, so this keeps the current layout when it fits: the label column may shrink, long names wrap, the input keeps 360 px.
Co-authored-by: G0-0000 <[email protected]>
- strata_control/drafting.py is the one place: the choice (saved in the model's run config as "drafting"), the table card class -> suggested
drafter with the measured evidence (Auto picks DFlash2 only where it was faster on every tested prompt; today no class qualifies, so Auto is
the MTP everywhere), the pinned drafter (incoai/Qwen3.8-Flash-Next-DFlash2 rev a22a1744, sizes and SHA-256 of the repository), what each choice
puts on the engine arguments, and the plain "Drafting: MTP (DFlash2 didn't fit)" words read from the engine's own log.
- serve/server.py applies the choice at every start and reports "drafting" / "drafting_note" in /metrics.
- strata launch --drafting writes it to the launch config and presets, shows it in --plan, adds the drafter to the download plan when it would
be used, fetches it before the start (a failure starts the MTP and says so), and prints the engine's fallback line.
- strata drafting plan|fetch (JSON/JSONL for the app). tools/dflash2_pack.py now reads safetensors with numpy alone (no torch on the person's
PC), byte for byte the same file as before (checked against the earlier version), and ships in the Terminal bundle.
- DFlash2 is for Qwen3.8-Flash-Next (qwen, gyro) only. The engine in this release runs it only in its CLI run, not with --serve, so while
drafting.ENGINE_SERVES_DFLASH2 is False the choice starts the MTP, says "DFlash2 isn't available in this mode yet" and downloads nothing.
strata-dsv41 --pack-info --pack <dir> reads only ds4.txt and tensors.txt: no device, no weights, no lock. It prints the
layer x role type map with an expert's bytes and a token's MB (top-k picks), the type totals, all-experts GB at the mean
bpw, the dense side split into embed / head / mtp / vision / engram, and every tensor whose type has no device kernel
(IQ1_S, IQ1_M, IQ4_NL today) with the layer and role - the H1 rule's report, and the first line of every benchmark log.
Verified on c0q3 (208.1 GB at 3.06 bpw, 3.251 GB a token - the document's figures) and c0q3d5.
- ds4_iq.cu: IQ2_XS (type 17) through the same kernels as IQ2_XXS: Ty<17> (74 B, 1024-word grid),
decode<17> and deq8<17>, block_bytes/name/dispatch. ds4_iq_test compares the token path, the window
path, the prompt sets and the CPU lane against ggml's own to_float + a double reference: worst
4.6e-08 (token gate_up), 4.6e-08 (down), 8.9e-03 (F16 tiles), CPU lane 3.5e-08. IQ2_XXS / IQ2_S /
IQ3_XXS / IQ3_S / IQ4_XS pass the same test unchanged.
- ds4_model.cu: gg_type_of takes 17; embed.weight may be any ggml block type (Q5_K / Q4_K / Q6_K ...)
through gg_row_to_bf16 (ggml's to_float, the same BF16 rounding as the Q8_0 path) - the IST-DAS sets'
Q5_K token_embd needs no dense override now.
- gg_dot_bench: the A/B's pass criterion was |x-y|/|x| on single outputs, which fails on cancellation
while the two paths agree to a few ulps; it now also compares 64 sampled rows against a double
reference scaled by the L1 norm (worst 1.1e-08) and requires |x-y| <= 1e-6 of the output scale.
A/B at 32 threads: single token x1.03 (IQ2_XXS) / x1.13 (IQ2_S); 4 rows x2.4 / x2.3.
- CMake: ds4_iq_test target; ds4_tq_test and ds4_tq_test_portable link ds4_gg_cpu.cpp (the ggml CPU lane
ds4_iq_cpu.cpp refers to since 49e82263a, missing from those targets).
Python's OpenSSL checks against certifi's roots and does not fetch a missing intermediate; Chrome and curl.exe use the Windows store
(schannel), which also holds the roots antivirus HTTPS scanning, a company proxy or a preview build adds. strata_control/tls.py
install() puts truststore (pinned in requirements.txt and the three runtime locks with hashes, CPython 3.12 in every archive) under
every HTTPS connection of setup, Control's transfer and component download, serve and the fetch tools, with certifi as the fallback.
A certificate failure is no longer the raw urlopen text and is not retried: it names the likely causes (antivirus web scanning, a
company or school proxy), says what to try, and never offers to skip the check.
The host: server/system-ca.mjs adds the system roots to Node's bundled list at start (Node 24.5+), and the agent process gets
NODE_USE_SYSTEM_CA=1.
- /compact was typed into the chat and sent to the model as text (it answered "nothing to compact, this conversation is still short" at 90%
full). The host now runs the Harness's /compact and says how full the chat was and is, measured.
- Before a message goes out the model server counts it (its own tokenizer). Fits: sent. Fits once the older messages are a summary: summarised
first, with a line in the chat. Can never fit (a 103 KB paste is 70K tokens in a 64K window): not sent, and the card offers "Attach as a file"
(saved in the chat folder, the assistant reads it in parts) or "Send a shortened preview".
- A refusal that remains after the Harness's own compact-and-retry is a plain sentence with the engine's text under Details, and the card
offers "Summarize and continue" or "Start a new chat" instead of "Try sending the message again".
- Compaction shows: a status line while it runs and one plain line afterwards. The context meter follows the running turn (the host reports
what the context holds after every model request) so it no longer reads 1% between two tool steps.
- A Harness error "lost host child is still present" reads as "the last Strata session is still closing down".
- All new sentences are in the catalog in the eight languages. The real-engine run is scripts/context-e2e.mjs.
The child marker without a creation time could not tell the recorded engine from another Python or Strata program that had reused the
number, so every load said "lost host child is still present" until the marker was deleted by hand. The child is always started before
its marker is written, so a live process created after the marker's file time (with a two-second slack) is not the child. Unreadable
times still keep refusing; a recorded creation time still decides by itself. New procs.started_epoch reads a creation time on Windows,
Linux and macOS.
The read-only check refused any command containing & ; ( ) { } before it understood quoting, so listing "C:\...\GUI Q&A" showed an
approval card. Quoted text is now set aside first (maskQuotes): an expansion inside double quotes ($ or a backtick), an unterminated
quote, a quote glued to other characters and a quoted text that starts with a dash are still not understood, so those and everything
outside quotes ask as before.
The Harness prices a conversation at 4 characters a token; code, logs and JSON are about twice as dense. A new Harness plugin
(plugins/strata-context.mjs, arithmetic in context-budget.mjs) wraps the token meter so pressure is the last measured prompt plus what
came after, priced at the density that measurement showed. It also wraps the compaction summary request: when it would not fit the
window it is asked for in pieces (a message bigger than a piece is cut for the request, each piece builds on the summary before it, a piece
the server still refuses is halved). A chat the engine refused once (a 70K-token message in a 64K window) is no longer stuck: the next
message compacts and goes through. Tests run the real Harness against a scripted server that counts and refuses like the engine.
recommend() (Terminal, app and shared vectors) now turns a card with 8 GB or less Off only when a bigger usable card is there or it has under 4 GB. The hardware report (tools/strata_mcp.py) applies setup's own opt-in rule for a PC with only Pascal/Volta NVIDIA cards and no usable AMD card, so those cards are no longer flagged 'older than the RTX 20 series' (engine_variant cuda12); the feed already picks the cuda12 package for sm_61 and refuses cuda13. Tests: shared gpu-roles vectors, strata_mcp hardware, feed selection for sm_61, and Tune keeping a two-card role map.
The load used to stop with 'Some model files are missing' and nothing else, so Retry looked the same every time. Control's model list now carries the required parts it cannot find (part, file, status); the host joins them with the installer's missing GGUF files and shards, puts the names in the error (four shown, all in Details and in the load log), and the app shows 'These model files are missing: ...' with 'Press Download & load again to fetch them. Files that are already there are kept.' in all eight languages.
While the callback is pending the browser holds rendering, so the frame callback never ran and every click waited for the browser's own guard: about 4 s in Chromium (and so the WebView2 app window, reproduced in Edge 155: callback 4023 ms) and about 5 s in Firefox. The callback now resolves on the first DOM change after the hash flipped, or after 150 ms. Measured in Edge 155: callback 9-19 ms, full transition about 350 ms, with and without a 3.6 MB chat history.
On an AMD R9700 PC a leftover nvidia-smi made detection say cuda, so the update staged cuda13 and its test start failed. detect_backend (Control) and gpuBackendHere (app) now let the Windows display adapters decide first and ignore a tool for a make with no adapter. The update decision keeps the installed engine's make when the PC has a card of it, and otherwise stages nothing and says so in plain words (status wrong-make, shown by the app). update_engine refuses a package of another make before any download. --backend and STRATA_ENGINE_BACKEND are the explicit way to change make.
The reader was clamped only when the config carried vision.max_tokens; an old, hand-edited or migrated config left it unclamped and
one picture became 93,700 image tokens. Vision.token_limit gives every reader a limit (GPU 1024, CPU 300, the numbers setup writes)
unless the config names one (a value the person set is kept, 0 still means none). strata-vision refuses a picture that comes back
over the limit plus an eighth with a plain ERR instead of passing 90 times the budget on, and prints a WARNING when its own warm-up
picture is over it. The server logs a picture that is more than 4 times the limit with the numbers to send with a report.
shell: true meant cmd.exe on Windows (Get-Content failed), the host's own folder (python tools/foo.py found nothing) and a fixed two
minutes with silently cut output. custom-command-run.mjs: a Windows command that reads as PowerShell runs in pwsh or powershell.exe,
any other in cmd.exe as before, sh elsewhere (or the tool's named shell); the folder is the call's optional cwd, then the tool's cwd,
then the home folder, and a missing folder is a plain refusal; the time is the tool's timeoutSec (1 to 3600, default 120) and ends the
programs the command started; a cut output says so. Settings stores cwd, shell and timeoutSec for a command tool and an edit that
does not send them keeps them. No new field in the Settings screen: that needs a design decision.
A launch command that printed nothing was reported as 'The program, folder or file was opened.', and the system prompt told the model
to say it is open. A browser given a file:// address also returns without an error, and the model then reported that it had previewed
the page. The card now says what is known (the command finished without an error, so the program was started; Strata cannot confirm
that a window or page opened). The prompt (and the plan view step) send local images, SVG, HTML and PDF to the present tool and
forbid claiming a window opened or a page rendered unless a tool confirmed it.
T5: the agent's tool servers came from local[0], the first model loaded, not the model the chat runs on. buildProfile takes
defaultLocalId (the native default instance, agent-bridge option defaultLocalId); the agent's default route follows it too.
T6: the editor and the save check accepted 64 characters while the profile dropped everything over 32 or with a dot, silently.
server/mcp-names.mjs holds the one rule (32, letters, digits, hyphen, underscore; src/lib/mcp-names.ts is its copy, a test
keeps them equal). A name saved before the rule stays saveable. mcpRows reports what it leaves out (name, no program or address,
a name that collapses to another's id); the agent status carries skippedServers and the chat shows one notice per reason before
the turn. The sentences are in all eight languages (src/locales/tools-<lang>.json).
A session whose preset lookup threw ran the call unasked (return next()). It is treated as Manual now: a call that is not provably
read-only asks, a read goes through, and the reason is logged.
Ported from #3556 (Amit Tsur): a declared namespaced tool written with its flat alias (mcp__server__tool for mcp__server.tool) keeps
its schema while streaming and is reported under the declared name; an ambiguous alias is no alias; a quoted incomplete wrapper
(`Format: <tool_call>... </function> and then I explain it.`) stays text; a following call opener separates concatenated calls;
streamed JSON is closed only once the wrapper or the next call confirms the boundary; a call cut by the output limit stays
incomplete. The Responses-only alias map is gone (the parser resolves aliases now).
Reconciled with the 0.2.2 envelope rule: with tools declared, a call for an undeclared name stays text on every route (the PR's
tests that expected a native call for an undeclared name now expect text, or a native call only where no tools were declared and
declared_only is off: the Responses/Codex compaction route keeps that). The aliases attribute replaces the envelope work's _alias
set, so the lenient opener (#3441) and the declared-only gate use the PR's unambiguous alias table.
New rail: a call that named a tool the request did not declare is counted (parser.unread) with the names the model wrote
(parser.unread_names), and the server's existing log line for unread calls now prints them.
Co-authored-by: Amit Tsur <[email protected]>
 used to come out as a link with a stray '!', and a data: URI as literal text. Allowlist (src/lib/markdown-images.ts):
data: PNG/JPEG/WebP/GIF (never SVG), the host's /api/files/raw address, and a scheme-less path read through that address for the
active chat. http(s) pictures are not fetched: they show as a link with the alt text. file:, javascript:, blob:, share paths and
every other scheme show as text. One .md-image rule keeps the picture inside the bubble.
build_cuda_linux.sh builds strata-vision for the same architectures as the engine (cuda12 includes sm_60/61/70) and records
vision_archs in BUILD.json; the Dockerfile records it too. packaging/build_vision_windows.bat does the same for the Windows
cuda12 and cuda13 zips (they shared one reader built before Pascal support, #3669). package_engine.py refuses a CUDA package whose
reader's vision_archs do not cover its archs and a macOS package without strata-vision (#3697); --allow-stale-vision and
--no-picture-reader are the explicit exits; --stamp-vision records a hand-built reader. ENGINE-COMPONENT.md describes the release steps.
A local engine reading a long prompt sends the Harness nothing for minutes, so the 4 minute stall limit ended the turn mid-read. The host now hands the bridge the engine's /metrics progress (live.prompt_read, generated, slots); a moving mark keeps the turn alive for at most 30 minutes after the Harness last spoke. An engine that stops reporting still ends at the usual limit. API routes are unchanged.
Every streamed token stringified every chat and wrote the lot to localStorage, which grew the WebView2 renderer to gigabytes. Writes from the streaming path are now coalesced (1 s), flushed when a turn ends, is cancelled or fails, and on pagehide/hidden. Unchanged chats reuse their JSON by object identity; a field over 200k characters, a chat over 1.5M and a list over 3.5M are shortened in the saved copy only. The stored format is unchanged.
A picture a tool returned wrote to an elastic encoder that was not running (AttributeError, then a 400 that every later
turn repeated). _note_unreadable_tool_images now calls _vision_up and turns a start failure into the usual
[image omitted: why] note; Vision.encode raises a clear ValueError when no reader runs.
vision_start_error no longer returns a blank sentence: last stderr lines, the exit status in words, the seconds of a
timeout, and a kernel-image error (sm_60 on the Windows cuda12 reader, #3669) names the card/architecture and the CPU way out.
No release feed ever carried a linux-x86_64-sycl engine package, because no recipe built one (the 0.1.45 package of stream split-s1 was made by
hand). build_sycl_linux.sh builds the JIT engine with oneAPI, bundles the oneAPI runtime and the Unified Runtime adapters with collect_libs.py,
sets RUNPATH $ORIGIN on every ELF file (Intel's libraries have none or a wrong one), packages it with archs intel-arc and checks the package
with --verify, binary_deps.py and an env -i strata --version. _drm_vendor_backend takes the sysfs root so the Arc detection can be tested with a
fake /sys/class/drm; tests/terminal/test_sycl_package.py covers the choice (an Arc PC with no oneAPI tools gets the package, nobody else does,
Windows gets a reason) and the recipe; the release-matrix test has the sycl package.
The migrated generate.cpp had no --version (the CUDA and HIP engines answer it first thing), and three things in dpct's helper headers
built a SYCL device at program start: blas::descriptor's inline _saved_queue_ptr, the dpct::device_memory constructor (pointer table,
mem_mgr, dev_mgr) and its default-built sycl::queue member. On a PC whose user cannot open the GPU the engine therefore aborted with
'No device of requested type available' before it could print anything, which also failed the engine smoke test of strata update.
The headers are edited and sycl/tools/fixups.py reproduces each edit after a re-migration. A frozen first-token-logits run on an Arc A750
(4095-token prompt, IQ3_XXS) is byte-identical before and after.
F5: a start that did not reach ready names the stage, the cause line from the engine log, a next step (render group on Linux, smaller
context, strata update) and the log path; --no-tui and runs without a keyboard never ask 'Keep or move?' and print what was kept.
F3: a context or picture change forwards the installed cards and Main/Helper roles to setup.
N3: --mtp on|off reaches setup, and setup no longer fetches the draft layer for --mtp off.
N4/N2: --plan counts a picture reader unless one sits beside the files, names the --vision off and --mtp off switches, and counts
the ready-made pack (NVFP4) setup would fetch; setup's disk hint is --data-dir under launch.
N5/N6: STRATA_DATA_DIR/install is looked at first, --plan says when an earlier install is the one in use, and an installed engine
older than the model needs is updated or named.
N8: --plan warns when the file headers say another family or model than the one named.
N9: update --check names the package it selected.
N11: the Arc adopt path takes Gyro and the sentence has no double full stop.
setup.py imports tools/calibrate.py after a tune, but INSTALLER_TOOLS did not list it, so every launch after a tune died with
ModuleNotFoundError and only 'The model could not be installed'. The list now names it, and add_installer also reads the
tools/ modules setup.py imports so the list cannot go stale again (stage-control.py uses the same list).
A new test stages the bundle the way package_terminal.py does and imports setup.py and its tools with tools/ limited to the bundle.
strata launch keeps the last 400 lines of setup's output in launch/<name>.setup.log, shows the Python error line in plain words
when setup crashed without an [X] line, and names the file.
A source checkout's .venv\Scripts\python.exe starts the real interpreter as its child, so the pid in the server's ready file
was one below the pid the runtime host started, the ready file was never matched, no probe was sent and the update ended
with startup_wait_expired. The host now accepts a ready pid within three parents of its child (Windows only).
The test start also shows progress (loading, loaded and waiting for one token) and a wait that runs out says what the engine was
doing, with the kept log, instead of blaming a busy graphics card because the log mentions VRAM.
+8.4% on json_extract at fixed windows (53.55 -> 58.03 tok/s), 94 of 250
output tokens differ from the BF16 read, cold HC read 93.8 -> 20.3 us
(142 -> 352 GB/s). Experimental, opt-in only, never a default.
gr_read launched six kernels per hyper-connection read: norm, down,
gate, mean, inject, write. The inject projection is now the down grid's
last block (one warp per stream, the same lane stride and warp_sumf as
gr_inject_kernel), and gate + mean are one kernel (one block owns WARPS
columns for every stream, the per-stream gated values meet in shared and
are added in the old c order). Both keep the old per-row arithmetic and
reduction order, so lo/lq, mixed and inject are bitwise unchanged; gated
is no longer written to global because nothing outside gr_read reads it.
Six launches per read become four.
This is a no-op for the default IQ3_S path on the Strix Halo: that engine
already takes the fused (staged) hyper-connection read, not gr_read. The
change is kept for the unfused gr_read path (CUDA, other HIP
architectures and --no-fused-gr).
Verified on gfx1151: json_extract at fixed windows (--spec 4
--spec-min-p 0 --suffix-draft 0) gives an identical output line before
and after.
Cargo.toml and Cargo.lock, desktop and host package.json, PRODUCT_VERSION, CMake project and Metal STRATA_VERSION, build_hip_linux PKGVER default, About comment, tests that pin the number, Docker image tags and docs, download names in the READMEs, AMD_HIP.md and UPDATES.md.
iq_kernels.cu is f-022-int's (staged codebook, row chunks and staged activations on cc 6.0; STRATA_TQ_STAGE). On top of it, tq_sub8_on()
(CUDA, not cc 6.0, STRATA_TQ_STAGE / STRATA_TQ_SUB8 not 0) turns on the staged codebook and the staged 8-lane down rows for the other cards
and adds the 8-lane 160-dot gate/up rows (row_dot_160_sub8). RTX 3060, Gyro-M, 24 groups, 1 token: gate/up 142 us (the cc 6.0 path forced:
158, off: 225), down 77 (off: 184); the hashes of the outputs are equal in all three modes.
The reporter's prompts ended in non-finite logits in the window that reads the prompt's last few tokens (after the last turn boundary), every time and on every retry, and passed with --short-read 0. The cause on their machine (a Windows VM, sm_89, a Broadwell AVX2 CPU, IQ3_S) was not reproduced on a P100 with a Broadwell Xeon, an RTX 3060 or a Core Ultra.
The engine now reads the request that follows such a window with --short-read 0 for that one request, so the server's transparent retry succeeds instead of failing the same way. Nothing changes until a prompt window reports non-finite logits. STRATA_NAN_INJECT=1 exercises it.
With the runtime capped to two hardware queues the two primed queues are all of them, and the high-priority draft stream and the draft side stream share them: a 4K prompt on an RX 9060 XT read 681-753 tok/s primed and 1188-1238 unprimed. Primed only when the cap is 3 or more or unset; STRATA_HIP_QUEUE_PRIME=nn still forces it. HIP code only (gfx12, Linux).
GYRO.md gets the measured RTX 3060 numbers of the shared-memory codebook / 8-lane row kernels (Gyro-S 53.6 -> 62.8 tok/s, Gyro-M 50.6 -> 56.1, 5 interleaved pairs, identity gate identical), the release notes one line, and the places that said Gyro-M decodes about 20% slower than Q2_0 now say 16-21% (RTX 3060 12 GB after the kernels: 16%, R9700: 17-21%). The translated READMEs keep the old figure.
packaging/build_cuda_linux.sh is the recipe for the Linux cuda13 and cuda12 packages (cuda12: CUDA 12.9, 60;61;70;75;80;86;89, STRATA_EXPERIMENTAL_SM60=ON, driver 575.51). package_engine.py refuses a cuda12 package without sm_60, sm_61 or sm_70, so the 0.2.1 mistake (sm_75/86/89 only, a P100 container in a restart loop) cannot ship again.
The root Dockerfile takes the CUDA image from CUDA_IMAGE: the CUDA 12 case turns on the Pascal/Volta build when an architecture below 75 is listed, names the package cuda12 with the matching driver, and the terminal stage has git (package_terminal.py calls it).
A <tool_call> whose opener drops the word function (<=NAME>, <function:NAME>, <NAME>) is read as <function=NAME> when NAME is a declared tool.
The answer follows the rule the reasoning already had (#804): a call counts only for a declared tool. Chat Completions and /v1/messages requests that declared no tools get the text back; the Responses route keeps its Codex compaction behaviour.
Blocks that open like a call but do not read as one are counted in the server window.
After #3484 the process-wide STRATA_PF_PAD can be on for a gfx12 stage, which then dequantized every weight padded, found no product and dequantized it
again unpadded before the copy path. Ask strata_pf_padded_gemm_available() for the current device first; same products, same bits.
A 39.8 GB shard came out with the wrong SHA-256 on 0.2.1 and was deleted, 40 GB of download lost.
- every ranged answer must be a 206 with exactly the Content-Range asked for (start, end and total), a fitting
Content-Length and the same ETag as the first answer; a changed file part-way stops the download and drops the .part
- the raw unbuffered write is looped until every byte is down (its return value was ignored)
- a .part records the address and ETag it came from (.part.src); one from another mirror or revision is dropped, not mixed
- a parallel download always has a piece list before the first byte is written; if none can be kept it uses one stream,
so holes in the sparse file are never read as a finished prefix
- the one-stream resume accepts a 206 only at the byte asked for; another range restarts from zero
- a finished file with a pinned hash but no finish mark is checked, not marked; the rename waits for a virus scanner
- a SHA-256 mismatch re-reads the file from the server piece by piece, rewrites only the blocks that differ and checks
again ("Some parts of the file were damaged; downloading only those parts again"); deleted only if still wrong
- update archives (strata_control/transfer.py): a resumed answer is appended only when its Content-Range starts at the byte asked for
- tests: tests/terminal/test_download_integrity.py (a local server that answers 200 for ranges, wrong ranges and totals,
cut bodies, another ETag, 503s, flipped bytes)
Python only (setup.py, strata_control/transfer.py, tests).
A Strix Halo's free VRAM is the GPU pool (carve-out + GTT, nearly all RAM with ttm.pages_limit raised) while the cache copies experts into that
same RAM on top of the arena, so a cache sized from the pool alone could ask for more than the OS had left (OOM killer). The auto size is now
min(GPU free, MemAvailable - max(8 GiB, 8% of RAM)); --expert-cache N is kept as asked, STRATA_UMA_RESERVE_GIB=N sets the reserve (0 = old sizing).
STRATA_PF_PAD=1 STRATA_PF_GEMM=1 lets the two K 6144 producers write row stride 6,208 into the prompt GEMM, but the only
product that reads a padded stride is strata_pf_gemm_f16_ld, which is built into wmma_gemm.cu's gfx11 matrix-core block
and returns false everywhere else. On an RDNA4 card Gemm::native then reached its "a padded X (ldx 6208, K 6144) needs
STRATA_PF_PAD's path" branch and called std::exit(1): under serve/ that is one engine restart per request. Reproduced
on an RX 9070 (gfx1201, ROCm 7.2.4, kernel 7.2.11) at --vram-reserve-mib 700 and 1,536, so it was not the VRAM guard.
- wmma_gemm.cu/.h: strata_pf_padded_gemm_available() answers whether the padded product exists on the current device
(the same gfx11 list the kernels are compiled for; false in a build with no gfx11 target).
- prefill.cpp: pf_pad() asks it before padding, and prints one line when the env asks for a product this card has not.
- gemm.cu/.hpp: a padded X that no kernel took is copied to a contiguous FP16 buffer (one cudaMemcpy2DAsync, one
grow-only buffer per Gemm) and the ordinary paths run - the same products, the same bits. exit(1) remains only for a
failed allocation, with a message that says so.
Checked on the RX 9070, same 1,199-token prompt and 8 greedy tokens in every arm:
- before (0.1.45, pre-fix binary): `prefill gemm: a padded X (ldx 6208, K 6144) needs STRATA_PF_PAD's path`, exit 1.
- after, STRATA_PF_PAD=1 STRATA_PF_GEMM=1: `prefill gemm: STRATA_PF_PAD needs the padded gfx11 product, which this card
does not run; not padding`, prompt and decode complete, exit 0, output identical to the unpadded run.
- after, with the gate forced open so that the producers really pad and Gemm::native takes the copy path: no exit, no
"needs STRATA_PF_PAD's path", output identical to the unpadded run (13 271 248046 198 248045 74455 198 248068).
With the gate in place STRATA_PF_GEMM=1 alone is inert on this card (4K 1,247 against 1,243 tok/s, 8K 1,330 against
1,321, medians of 3 starts).
docs/AMD_HIP.md carries the RDNA4 bullet for it. Built for gfx1201 (RX 9070, ROCm 7.2.4, Release): 338 of 338; ctest
77 of 79, the two failures (ple_parity without the Q2_0 GGUF here, hip_prefill_mmq_parity's APR block reading
ggml_type_name(145) with GGML_TYPE_COUNT 43) fail identically on origin/main. The changed files are shared with the
CUDA build and were not compiled for CUDA here (no nvcc on this machine); SYCL is untouched.
(cherry picked from commit 341625013d)
The #1576 memory guard compared the iGPU's BIOS carve-out with the dGPU's VRAM and could keep the iGPU as the last stage (head, draft layer, verify).
The picture reader inherited the engine's first visible card, the iGPU. setup --gpus all took every usable card including the APU.
All three are recommendations: a number list, gpu_order as_given, a manual layer split and vision.cuda_device are kept as asked.
The int8 codebook lookups of tq_load went through the global table: a warp's 32 random 2-byte reads touched ~20 L1 sectors each
(native_expert_bench, ncu: 51M sectors for 2.8M requests, L1 64% busy). The grouped gate/up and down kernels now stage the 4 KB
codebook per block (stage_iq_grid, STRATA_IQ_STAGE_GRID=0 reads it in place) and take 4 rows per warp: gate/up K=160 (new
row_dot_160_sub8) and the K=40 down (row_dot_40_sub8, as IQ4_NL), 32 rows per block so the staging is amortised and no lane idles.
The sums keep row_dot_multi's tree (lane sums of t, t+8, t+16, t+24, then the same xor steps), so the outputs are bit for bit the
old ones: native_expert_bench prints an FNV hash of the outputs now, equal for base and new on Gyro-S, Gyro-M and Q2_0 blobs.
RTX 3060, 24 groups, 1 token: Gyro-M gate/up 221 -> 140 us, down 182 -> 86 us; Gyro-S 228 -> 157 and 173 -> 89.
strata_control/vram_fit.py estimates the VRAM left for the expert cache (calibrated on 12 A750 starts) and walks the ladder 32K -> 16K -> 8K, prompt chunk 1024, reserve down to 100 MiB. Used by sycl/setup_intel.py (setup), strata launch's recommended context and the adopt path. A config that already fits is returned unchanged; values the user chose are kept; it never refuses.
The start-up line and the --pool-cores check used the layout as seen by whichever thread asked, and on Linux a thread that was
already pinned (the host, after the session pins it) sees only its own CPU. The layout is now read in the constructor.
- setup.py: the nine pinned shards of unsloth/DeepSeek-V4.1-Flash-1017-GGUF UD-Q8_K_XL (revision f924b94f, sizes and SHA-256 from the reference run; the DSpark drafter and picture reader pinned but not fetched), MULTI_FILE_FAMILIES, install_dsv41: plain warnings for the 508 GB download, RAM and disk (a refusal only where the disk cannot hold the files), the strata-dsv41 check before any download, --gguf-dir for files you already have, pack_gguf.py, config with --pack --max-context 8192 --eos 1 --tp 2 and the cards; menu entry and a --check line.
- strata_control/dsv41_fit.py: where the model goes on the cards and in RAM, from the measured run (6,240 experts in graphics memory predicted on 4 x 32 GB, 6,197 measured), the n-gram table's SSD note.
- tools/strata_mcp.py and strata_cli/launch: the model is in the list; strata launch --plan prints the placement (layer blocks per card, experts on the cards and on the CPU, RAM, disk, the SSD note); --model-path takes the nine shards where they are; parallel, pictures and tuning are said, not passed to an engine that does not take them.
- tests: tools/test_setup_dsv41.py (22), tests/terminal/test_launch_dsv41.py (13).
thread_name.hpp pulled <windows.h> (and WIN32_LEAN_AND_MEAN) into every file that includes aux_cpus.hpp; on Windows the function is now
defined in pool.cpp, which already includes it. The PLE reader and the direct file readers are not named (they do not link the pool).
Both pass --pool-cores to the engine (experimental, off by default). A list that cannot be read is said and left out.
strata launch also shows the notices that the flags themselves produce (they were dropped before).
The start-up now has one plain line: CPU workers on logical processors 2,4,...,30 (15 workers, one per physical core; host on 0).
It says so when two workers share a core, when the host's core is taken, and when the OS did not say which processors are
siblings. --pool-cores LIST / STRATA_POOL_CORES pins the workers to exactly the listed logical processors (off by default;
siblings, the host's processor and processors the process may not use are warned about, never refused). The default placement
is unchanged. The pool workers, the host thread and the long-lived helper threads get names (strata-pool-N, strata-host,
strata-stage, strata-adapt, ...), so ps -L, /proc and Windows thread tools show which is which. Windows without core
information no longer counts every logical processor as a core when it picks the default worker count.
- tools/strata_tokenizer.py: the deepseek-v3 pre-tokenizer (llama.cpp's DEEPSEEK3_LLM list as a sequence of splits) chosen by the pack's tokenizer.json 'pre'; special tokens decode to their literal text (the DSML token is not in the byte alphabet). 20 of 20 reference prompts tokenize to the ids llama.cpp made.
- serve/dsml.py: DSML calls (the GGUF template's spelling) translated as they stream into the form OutputParser reads, so tool streaming, tool_choice and calls in the thinking work as for the other families; string=true values stay strings.
- serve/server.py: Service.set_family('deepseek41'): end token from the pack, DsmlOutputParser, forced-call opening in DSML, effort words low/medium/high/max/1..100, model name and card sampling.
- serve/test_dsv41.py (39 tests, fake engine) and fixtures from the GGUF's own template and the llama.cpp reference prompts.
A 39.8 GB shard came out with the wrong SHA-256 on 0.2.1 and was deleted, 40 GB of download lost.
- every ranged answer must be a 206 with exactly the Content-Range asked for (start, end and total), a fitting
Content-Length and the same ETag as the first answer; a changed file part-way stops the download and drops the .part
- the raw unbuffered write is looped until every byte is down (its return value was ignored)
- a .part records the address and ETag it came from (.part.src); one from another mirror or revision is dropped, not mixed
- a parallel download always has a piece list before the first byte is written; if none can be kept it uses one stream,
so holes in the sparse file are never read as a finished prefix
- the one-stream resume accepts a 206 only at the byte asked for; another range restarts from zero
- a finished file with a pinned hash but no finish mark is checked, not marked; the rename waits for a virus scanner
- a SHA-256 mismatch re-reads the file from the server piece by piece, rewrites only the blocks that differ and checks
again ("Some parts of the file were damaged; downloading only those parts again"); deleted only if still wrong
- update archives (strata_control/transfer.py): a resumed answer is appended only when its Content-Range starts at the byte asked for
- tests: tests/terminal/test_download_integrity.py (a local server that answers 200 for ranges, wrong ranges and totals,
cut bodies, another ETag, 503s, flipped bytes)
Python only (setup.py, strata_control/transfer.py, tests).
- strata launch with no --gpu replaced the installed config's own card choice (setup --gpu 1, the app's pick, "gpus_asked") by the
fit rule's pick, which is card 0 on equal cards. The installed choice and its order are kept; the Cards step preselects it.
- The server prints one line tying the engine's own numbering ("GPU 0 / CUDA0" in its log) to nvidia-smi's, when they differ.
- A card named by the app or by strata launch drops an old AMD config's hip_ordinal, which would have sent the engine to the old card.
- A list "gpu" in an image config is a device list ("1,0"), not "[1, 0]".
stream.cpp: load_dense skipped nothing of the n-gram table because it was always another shard; in a Gyro
file it is the first 28.8 GB of the one file (the 32 GB allocation that failed on the Mac mini).
mac_fit / setup_mac / setup.py (app path) / strata_mcp: Gyro-S and Gyro-M as the 'gyro' family of the Mac
(one file, --compat-bf16 pack, --native and --ple-gguf the same file).
expert_parity --decode-check compares ref_quant's trellis decoder with the format's definition written out bit by bit.
The decode (moe_gu / moe_down) and prompt (mm_gu2 / mm_down2) expert kernels read the three trellis types
through a unit decoder in quant.metal (the codebook from threadgroup or constant memory); had128 kernels
rotate the gate/up and down inputs of Hadamard-folded layers in the decode window and both prompt paths.
ref_quant decodes the types with the CPU engine's own state extraction; expert_parity takes real experts of a
GGUF (--gguf --layer), the pack's hadamard.txt (--had-pack) and has a bitwise rotation test (--had-test).
The server and strata serve start on 0.0.0.0 without a key and print one plain
warning. strata launch gets --no-api-key; the Access screen makes a key by
default and t turns it off. api_key "none" in a config means no key on
purpose. Cloudflare without a key gets a stronger warning. Docs updated.
The Terminal on PATH ignored components/control/active.json, so after
strata update the old Control kept running. strata.py now checks the
pointer first (stdlib only) and runs the newer, whole Control with its
own interpreter; never an older one, never outside the control folder,
no loop (STRATA_HANDOVER), one warning and a fallback if it is broken.
strata update says the next command already runs the new Control.
Maya's tools/ds4/chat_template.jinja lost tool associations and later system instructions; the GGUF's embedded
template matched DeepSeek's encoding.py byte for byte on 9 cases (notes-dsv41-ref.md).
Co-authored-by: mw00 <[email protected]>
CMake option STRATA_BUILD_DSV41 (default OFF, adds the target strata-dsv41 and a private static kernel lib; no
existing target changes). The vendored exl3 / sampler / window-grouping kernels live in namespace strata::dsv41k so
no symbol is shared with strata_kernels. Missing CUDA names for HIP come from a force-included private header
(include/strata/dsv41/hip_extra.h), the IQ CPU lane stays the stub branch, MAYA_* env aliases are gone.
New --score IDS.json --score-out RUN.tsv: teacher-forced top-64 log-probabilities in tools/quant_quality.py's
row format (the gate against the llama.cpp reference). MIT notice added to every ported file.
Co-authored-by: mw00 <[email protected]>
Co-authored-by: needmorevram <[email protected]>
Verbatim copy of mw00/project-maya src/ds4, tests/ds4_*, tools/ds4 and the exl3 / sampler
kernel sources it needs, MIT, with their LICENSE and a NOTICE. Not built yet; the next
commits adapt it to Strata's tree.
Co-authored-by: needmorevram <[email protected]>
tools/trellis/tqenc.{hpp,cpp}: shift-trellis Viterbi (group-min ACS), tail-biting by a rotated free pass or the exact
search over all start groups, fp16 scale refinement, importance weights. tqenc_test.cpp checks the streams, the
Viterbi against a naive DP, planted solutions, bit-exact decode against tq_states.hpp and the CPU kernels.
The CPU decoder gets K = 5 and 4 (ids 224, 225, experimental v1); existing types and layouts are unchanged.
The agent footer reads the local engine's /metrics for the turn (no empty number on short replies, prompt reading speed kept apart), model APIs are timed from the first token, the monitor average and the request list use answer speeds, the footer shows the last reply's own speed, and the server's done line no longer prints a million tok/s for a one-token reply.
The server and strata serve start on 0.0.0.0 without a key and print one plain
warning. strata launch gets --no-api-key; the Access screen makes a key by
default and t turns it off. api_key "none" in a config means no key on
purpose. Cloudflare without a key gets a stronger warning. Docs updated.
The Terminal on PATH ignored components/control/active.json, so after
strata update the old Control kept running. strata.py now checks the
pointer first (stdlib only) and runs the newer, whole Control with its
own interpreter; never an older one, never outside the control folder,
no loop (STRATA_HANDOVER), one warning and a fallback if it is broken.
strata update says the next command already runs the new Control.
tools/idle_gpu_ab.py is the 0.2.1 split A/B harness with a per-card nvidia-smi sampler. Result on the 0.2.1 engine: the two-card
split is 47-49 % idle in decode (cards take turns), the single card 5 %; STRATA_SPLIT_OVERLAP, --pipeline-windows 1/2 (with and
without the draft layer on card 0) and --batch-groups 2 give no gain worth a default. docs/MULTI_GPU.md: the overlap flag is now
measured on NVIDIA.
- the plain summary and the README get a Capped column (requests that reached the 512-token cap) and leave out the
short case's prompt rate (an 80-token prompt measures the fixed cost of a request); results.json keeps it
- BENCHMARK_PROTOCOL.md, COMMUNITY_BENCHMARKS.md, BENCH.bat and bench.sh: the standard suite took 2 to 4 minutes after
loading on an RTX 3060 and on 2x RTX A4000, not 20 to 30
On a fresh Windows PC (long paths off) the AMD engine packages could not be unpacked: staging under .updates/staging/<12 hex> pushed the
longest hipBLASLt file past 259 characters, and the failure surfaced as 'local file operation failed (FileNotFoundError)'.
- staging is now <components>/.s/<4 hex>/engine/<folder>; Control stages the same way
- unpack, delete and verify go through extended-length paths (strata_control/longpath.py)
- OSError messages name the file and say so when the path is too long
- a plain warning (shown as a warn line by launch) when the install folder is too deep for the engine's own file loading
- tests: real 0.2.1 package path lengths for a 20 character user name
The first runs of strata bench run on the 0.1.46 release engine (RTX 3060, 2x RTX A4000) showed:
- the release terminal archive carried no COMMIT file, so every run was nonconforming; package_terminal.py now takes
--commit / STRATA_COMMIT and refuses to build a distributable archive when the commit is unknown
- the model drive on an LVM volume was not found (lsblk -s nests lvm -> part -> disk)
- nvidia-smi 615 prints CUDA UMD Version, so no runtime was recorded; the runtime is now the engine's build
runtime (engine-component.json) plus the driver's supported CUDA version
- engine.log was empty (serve --quiet); it now holds the run config's log from the start of the run
- a fresh server (0 prior requests) was reported as 'could not tell whether freshly loaded'
package_terminal.py reads STRATA_BUILD_COMMIT first and only then asks git, and no longer crashes when git is missing. build.sh passes the commit as a build arg, the Dockerfile turns it into that env.
Cargo.toml and Cargo.lock, desktop and host package.json, PRODUCT_VERSION, CMake project and Metal STRATA_VERSION, build_hip_linux PKGVER default, About comment, tests that pin the number. tauri.conf.json has no version key (Tauri takes it from Cargo.toml). Download names in the READMEs, AMD_HIP.md and UPDATES.md follow.
When the room rule picks the file cache but the cache holds under half of the read set, the engine now
times O_DIRECT preads (8 threads) against first touches of a mapping (3 threads) on random 1 MiB
pieces of the model file, once, and reads unbuffered when the direct path runs at 4 GB/s or more and
is 1.3x (under 30% held) / 1.5x (under 50%) faster than the cached path. The reporter's PCIe 5 NVMe
(8.18 vs 4.93 GB/s, 13% held) lands on unbuffered (2x prompts, 3-4x decode in their A/B). A 1.7-2.5
GB/s NVMe (P1004) stays on the cache, where it decodes 1.1-1.6x faster; a ratio alone could not
separate the two, hence the absolute floor. STRATA_FILE_TIER_PROBE=0 keeps the room rule alone.
Default change: proposed, not validated on a fast drive here. Decision in platform::file_tier_prefers_direct
(platform_memory_test).
Zero-copy does not apply to experts.bin (one contiguous blob the kernels already read from the mapping), but the io_fill_ prefetch path does. RTX 3060, IQ3_XXS 43 GB pack, 32 GB cgroup: decode 1.38/1.21/1.33x (5/5 pairs), prompts 1.04/1.06x. At 16 GB the cold 4K prompt was 4-6% slower, so it stays off below the room test and the log says how to force it. Identity gates 10/10, exact_pp 4K/20K identical.
On gfx1200/gfx1201 under Linux the compute stream and the expert staging stream
landed on the same compute pipe in about half of the starts, which made 4K prompts
13% slower (p620, 1,980 vs 2,290 tok/s). Five queues (four normal, one highest) are
now opened one after the other on the main thread before any other thread starts.
STRATA_HIP_QUEUE_PRIME=0 turns it off. Answers are unchanged.
The app and setup rewrote the run config's "args" on every Load: the resource preference forced
--vram-reserve-mib to 1536 (2048 with pictures, a fifth of the card on "work") whatever the file said, and
a setup run (started by the Load when the context or pictures differ) dropped every option it did not
choose itself, the control vector options included.
- run config: "user_args" holds the options set by hand; "args_written" is the args as setup/the app last
wrote them, so a hand edit of "args" is noticed and moved into user_args (strata_control/user_args.py,
apps/host/server/native-user-args.mjs: the same rule)
- every start (serve/server.py engine_args), every setup run (setup.carry_user_args), every host write and load
and a saved tuning put the person's options over the generated ones; the same option: theirs wins
- old files without args_written: setup keeps the options the old args had, the new ones lack and setup.py never names
(plus the control vector group); a reserve that is not setup's own 700
- --vram-reserve-mib: any number the person set (Expert setting, user_args, the key, a value in args) is used as it
is, 0 included; the safety default (1536/1024, 2048 with pictures, 20% on "work") stays for a card that drives a
display or is not known to be headless; a card the probe says drives no display keeps 512 (700 with pictures);
a 0.2.0-beta file where the app wrote the number itself is told from a changed one
- hardware: drives_display per card (nvidia-smi display_active; the Windows adapter with a desktop resolution for AMD)
- setup --vram-reserve-mib, the web settings page and `start --vram-reserve-mib` record the value as the person's
- docs/TROUBLESHOOTING.md: what is kept and what the app still chooses
f3be9743 warms the first BF16 and FP16 products of every device at start. On a layer split across a
gfx1151 and a gfx1201 card it slowed prompts about 25% (hc read of the gfx1151 stage 2.7 -> 9.0 s per
32K block; STRATA_PREFILL_WARM=0 restores it). Two cards of one architecture lose nothing (2x R9700,
interleaved pairs). The warm-up now runs only when every visible card has the same gcnArchName;
STRATA_PREFILL_WARM=1 asks for it anyway. Policy in warm_policy.hpp with warm_policy_test.
On Meteor Lake and Arrow Lake-H the LP-E cores are in neither the cpu_core nor the cpu_atom
list, so the pool and setup counted them as P-cores: a Core Ultra 7 255H read as 8P+8E, the
more-E-than-P default plan did not apply and 15 workers ran (-35%). A CPU in no list now falls
back to its capacity. Tests: pool_plan_test (helper), tools/test_setup_hybrid.py (fake /sys).
Each AMD engine package gets tuning/<arch>-hipblaslt-<version>.txt for the architectures it covers and the
hipBLASLt version it bundles, and records hipblaslt_version in BUILD.json (packaging/stage_tuning.py for the
Linux build scripts, tools/hip/package_windows.py for the Windows zip). package_engine.py lists the tables in
the manifest and refuses an AMD stage without a version or with a table for another version or architecture.
serve/server.py with_component_engine sets STRATA_HIPBLASLT_TUNING from the package (strata_control/
hipblaslt_tuning.py): a table only for the exact arch and version, a log line for a mismatch, an unreadable
card or a missing version, STRATA_HIPBLASLT_TABLE=on|off (or "hipblaslt_table" in the run config) as the one
switch, and a STRATA_HIPBLASLT_TUNING the person set is kept. setup.py looks in the engine folder's tuning/
first.
A table changes the rounding of the prompt's GEMMs (R9700: residual differs, first-token KL 0.016), so the
automatic choice is off (AUTO_USES_TABLE = False) until the owner decides; "on" uses the table. No gfx1201
table exists for hipBLASLt 1.2.0 and one calibrated for it was not worth shipping (+1% prompt, -2% decode).
sign-and-notarize.sh starts strata-vision --version in the signed engine folders (it already signs every Mach-O).
MAC.md carries the M4 numbers (0.45 s on the GPU, 50.7 s on the CPU for a 260-token picture) and the rule.
- The EXIT trap of build.sh (--ctx-only or --ctx given) and publish.sh (--work given) ended with a failing test, so dash reported exit 1 after a
successful run. Both traps now use if/fi. Test: build.sh --ctx-only exits 0 and its context holds docs/TERMINAL_LAUNCH.md.
- docs/DOCKER.md: image sizes, the 2026-10-19 test of the images built from the 0.2.0-beta packages (cuda13 and cuda12 on an RTX A4000, ROCm images
built and checked, wrong-card message).
tools/vision builds one program for the Mac: ggml's Metal backend inside, --gpu reads on the GPU, without it on the CPU. The reader
choice (strata_control/vision_reader.py) gives a Mac the GPU reader whatever its memory (the CPU reader needs ~50 s per picture:
ggml has no BF16 vector kernel on ARM). setup.py's Mac path downloads the mmproj and writes the vision section; strata launch,
the MCP server and metal/setup_mac.py default pictures on. packaging/build_vision_mac.sh builds the program for the engine package.
- .github/workflows/docker-images.yml: manual only (workflow_dispatch) with the release tag as input; a plan job lists the flavors the
release has an engine package for; one job per flavor builds the image and, only with push on, pushes ghcr.io/niko1221/strata:<version>-<flavor>
and latest-<flavor> with GITHUB_TOKEN (packages: write).
- docker/fetch-engines.py: downloads the engine package from the release and checks it against the signed update feed (feed signature, archive
size and SHA-256, manifest hash, package signature, every file), the same code as `strata update`. Flavors cuda13, cuda12, rocm-gfx120x,
rocm-gfx110x, rocm-gfx1151 (the halo package), sycl (only if the feed lists one).
- docker/publish.sh: the same steps by hand (fetch, build, start check, push only with --push).
- Fix: docker/build.sh did not copy docs/TERMINAL_LAUNCH.md, which package_terminal.py needs; no image could be built from a clean tree.
- Dockerfile: OCI source/url/documentation labels (ghcr links the package to the repository).
- Compose files and docs/DOCKER.md name the 0.2.1 tags on ghcr.io; DOCKER.md has a maintainers section. Tests for the scripts, the workflow and the fetch.
Not a hang: 42 tests against a fake engine that paces real time took 127 s in one module and passed. The yield/failure tests move to serve/test_parallel_yield.py; the shared fake-engine helpers are FakeEngineCase.
get_prebuilt has probed the release with a HEAD request since 0.2.0; the refusal tests reached GitHub, got a 404 and compiled instead of refusing. The code is right, the test was stale.
A --data-dir that names a new folder used to move Models, packs and mtp into it and rewrite the saved data_dir. Now the terminal asks (keep or move), --yes and a missing keyboard keep the old library in place (still found as a root), --move-data moves, and the saved default changes only on a move, --set-default-data-dir, or a PC with no models. Tests use temp folders.
A single connection to Hugging Face's CDN stalls to 1-3 MB/s for 10-30 s again and again on the
owner's route; eight connections hide it, but a request per 32 MiB piece paid TCP slow start every
time. A request now covers up to four neighbouring pieces (the piece list and resumes are unchanged),
a try that moved bytes forward does not count as a failed one, and the default is 16 connections.
A server whose HEAD gives no length (ModelScope) is asked for bytes 0-0 so it gets the connections
too. mtp_fetch reuses the CDN address, reads the 28 shard headers side by side and lets the
connections run ahead of a slow range.
The installed app listed every model's pictures as "off", so each Load of a picture-capable model ran
setup again, which writes the arguments (--mtp, --spec-min-p, --vision, --max-context) and the vision
section from its own choices. Hand edits and the Read pictures tick came back each time.
- host: images comes from the run config's vision section, so a prepared model loads straight away
- host: Read pictures off no longer downloads/keeps the reader; --vision is left out of the engine arguments
- host: Draft layer (MTP) setting; "MTP per request" and it are the app's own and no longer frozen
- host: a context size set by hand shows in the Context list
- "mtp": false in the run config (strata_control/draft_layer.py) is applied at every start by
serve/server.py, kept by setup, written by `setup --mtp off` and `strata launch --mtp off`
- strata_install accepts any context size, not only the listed ones
- docs: draft layer off, and what the app owns in the run config
Reported crash (0.2.0-beta, Windows 11 26300): strata-desktop.exe exits with 0xc0000409 at 0x31a315 and no log. That offset is the
fast-fail in Rust's abort after a panic (panic = "abort"); a Tauri setup error is a panic ("Failed to setup app"), and the shell had no
log on Windows, no console (windows_subsystem), so the reason was lost.
- the run log opens first in main (%LOCALAPPDATA%\Strata\logs, the folder Save logs already reads) and a panic hook writes the
message and place to it and to desktop-crash-<time>.log, and shows a plain message box before the abort
- setup no longer returns an error to Tauri: a host that does not start or a window that cannot be made is reported the same way
and ends the app with exit code 1; the build error is no longer an expect
- the host's output goes to the run log (it was lost without a console)
- plain next step in the box for a WebView2 problem and for a blocked host
- STRATA_NO_DIALOG (scripted runs) and STRATA_DESKTOP_TEST_PANIC (tests) for proving the net
download() reads a file of 256 MiB or more in 32 MiB ranged pieces over 8 connections
(STRATA_DL_CONNECTIONS=1 restores one stream), writes each piece at its own offset,
and keeps the finished pieces in <name>.part.chunks so a stopped download continues
without fetching them again. The redirect to the signed CDN address is resolved once
and again before it expires or after a 401/403/410. A server without Range support
or without a length is read in one stream as before; an older single-stream .part
continues after its last whole piece. The progress line and the .part size the app
reads are unchanged, the SHA-256 pin is still checked on the whole file, and the
.done mark is the same. tools/mtp_fetch.py reads its ranges in parallel too.
tauri build makes Strata.app.tar.gz and its .sig before sign-and-notarize.sh re-signs and notarises the app, so the published Mac update would be the unsigned app. The script now writes Strata.app.tar.gz from the stapled app on request; the Tauri key stays on the PC (npx tauri signer sign), and set-app-platform checks the .sig against tauri.conf.json's pubkey over the exact bytes and records url, signature, sha256 and size.
Qwen3.8-Flash-Next (ISTA), Swift 1.5, the Coder and the picture reader get fixed
hashes (REPO_PINS, checked in download()). A 404 on the pinned commit stops instead
of taking the current file; STRATA_ALLOW_UNPINNED=1 is the announced tester opt-out
(setup.py and tools/mtp_fetch.py). Swift moves to the commit that has its IQ3_S.
Tests for pins, mismatch, 404, opt-out and offline use; SAFETY.md updated.
Header uses the existing brand lockup. Performance tables get bars and plain labels, with the engine versions left to DETAILS. What you need is built around 32 GB of RAM plus VRAM and links the platform docs. Troubleshooting moves below How does it work.
DETAILS: pictures on by default with the 24 GB reader rule, the engine label of the prompt-chunk planner (before 0.1.45.3, not 0.1.46), the per-architecture sets measured on 3060, A4000 and P100, the N7 and ROUTE_PRIOR decisions, a pointer to the app and Terminal troubleshooting. Links to private phaseA-tests notes removed from DETAILS and BATCHING. Tester readmes: AMD pictures are on (CPU under 24 GB, GPU on a 24 GB 7900 XTX), the RX 7900 text reconciled with the community runs, Mac needs macOS 15 and has no picture reader yet.
MAC: the .dmg and the Terminal bring the Metal engine; pictures only with the source build. AMD_HIP: the 24 GB
picture rule and the measured R9700 GPU reader. INTEL: the grouped XMX label is engine 0.1.45.3 (not 0.1.48), plus
the A750 prompt-hang fix (fca3c5958; STRATA_PF_DMA_PAGEABLE=1 restores the old copy). INTEL_ARC: maintainers now test
on a B70 and an A750; where the Intel engine comes from. OLDER_GPUS: the P100 sm_60 defaults (STRATA_SM60_PERF).
MULTI_GPU: two models on two cards, one make per engine. MODELS and TROUBLESHOOTING: the Mac and Intel Arc have no
picture reader yet.
Agents install the Terminal with the one line and run strata launch --model NAME --yes --no-tui (flags checked against
strata launch --help). Pictures follow the 24 GB rule on every make; the Mac has no picture reader yet. INSTALL gains
The app and The Terminal sections, update paths for both, and a Mac source section. AGENTS points to CONTRIBUTING.md.
Pictures are on by default (CPU reader under 24 GB, GPU reader from 24 GB, every make; no Mac reader yet).
Gyro and NVFP4-QAD read pictures; Gyro needs engine 0.1.45.1+ and is correct on RTX 50 since the SwiGLU guard.
The three NVFP4 versions get their app names. A Qwen3.6-35B-A3B section, the app and Terminal ways to add a model,
and engine versions on the older speed tables. Troubleshooting: no-engine warning, needs a newer engine / Update
engine, Mac xattr steps, strata doctor, Save logs. Private test-notes references removed.
Pictures on by default (CPU reader under 24 GB, GPU reader from 24 GB, every make; none on the Mac yet), the Mac app,
the Terminal in a short section, the Anthropic base URL without /v1, two models on two cards, one make per engine.
The speed tables stay and say which engine measured them. A Contributing line links CONTRIBUTING.md.
setup.py control_engine_step stops only for strata launch (STRATA_NEED_ENGINE=1); the app warns and continues. The Models card and the first-run
download card show the owner's plain warning before and during the download; the load refusal says the same with Settings > Engine.
A /plan on a small model searched again after every refused search and rewrote its plan after each one until the context reached 100%,
then the turn ended with no answer and no message. The bridge now counts what the turn shows (turn-guard.mjs): 3 failed searches in a
row, 10 searches in a plan turn (25 otherwise), a plan written from 1. again 4 times, a local model's context at 93%. The turn is stopped the
way Stop stops it but ends as a normal turn: the plan it had is still proposed (the plan card says why it stopped early), any other turn
ends with one plain translated sentence with a next step (frames carry the English sentence as `key`, the page translates it, 8 languages),
and the Harness's own /compact is asked for so the next message has room. A turn that ends with no words at all always gets a message:
"I ran out of room in this chat..." at 85% of the window or more, otherwise "I stopped without writing an answer...". Settings -> Tools says
results may come from Bing when DuckDuckGo is busy. Tests: counting (pure) and three end-to-end runs through the real Harness with the scripted
model (searchloop / bloat) and the real search code on a pretend web that refuses (bridge option searchMock), no network.
- tools/strata_mcp.py: disk usage from the files a model really uses (every part of a multi-part GGUF, the picture reader, packs, draft layer),
a file named by several flags or shared by several models counted once; Delete's plan stays separate (removable_gb)
- Settings > Storage: one list and one total (optimized models and other local models), shared files as a line of their own
- Check folder on the same folder no longer unloads the loaded models (O01); an actual change still asks first
- Settings shows the app version (0.2.0-beta) as Strata with the engine beside it; Control's 0.1.45 is no longer shown as Strata
and no longer goes into the update check as the app version
launch_gu_swiglu (STRATA_GU_SWIGLU, on by default on compute capability 12.0 cards since 147-verify) runs
16-lane gate/up rows that assume 80 dots a row, but its guard rejected only the K-quants. The APR trellis
types (TQ2_T / TQK6 gate/up: qk 128, ipb 8, 160 dots at n_embd 2560) went through it: half of each row,
Gyro-S answered with loops, garbage or nothing on every RTX 50 / RTX PRO Blackwell card. They now take
launch_gu's full-warp kernel, as launch_gu itself always did for them; 80-dot formats are unchanged.
l620 (RTX PRO 5000, Gyro-S through serve/server.py, the owner's QA prompts, greedy and the app's sampling):
before 0/8 correct, after 8/8 ("STRATA_OK 42", "4", the follow-up sentence, "PING"), the same as
STRATA_SM120_PERF=0 / STRATA_GU_SWIGLU=0; Gyro-S fix vs STRATA_SM120_PERF=0: identity gate 10/10, exact_pp
4K / 20K IDENTICAL (KL 0).
The default DuckDuckGo search failed with "limiting searches from this computer" after two or three quick searches (reproduced on a
Linux box: two searches 1.2 s apart answered, the third got HTTP 202 and every DuckDuckGo page, html and lite, refused for many minutes
while it was polled). Now: browser-shaped requests (Referer, Origin, Sec-Fetch-*), one request at a time per source 2.5 to 4 s apart with
random jitter and at most 8 a minute, a 30 minute session cache per question, a per-source cooldown after a refusal that doubles from
1 to 10 minutes (Retry-After honoured) during which that source is not asked, no lite-page follow-up after a refusal, one retry for a
timeout or 5xx. When DuckDuckGo refuses, Bing's result page is used (links decoded from its redirects). When every source refuses the
tool says, in one plain sentence, that web search is limited, to try again in a minute or add a key (Exa, Perplexity or DeepSeek) in
Settings -> Tools -> Web search, and tells the model not to retry. Tests use a mocked rate-limited search and a real Bing page fixture.
- native state: the host's 'native' events are applied at once; the light poll (1 s while a job runs, none while hidden) is no longer
held by a slow forced read; a read that never answers is dropped after 20 s
- install progress: every file of the model is counted (parts, picture reader beside the model folders, NVFP4 packs by their pinned sizes);
the bar no longer sits at 99 % while the picture reader downloads; the screen names the part (part 2 of 4) and lists the parts
- a config is 'ready' only when every part of a multi-part GGUF is whole
- Models page keeps version, context, computer use, picture reading and expert choices across tabs; Read pictures is on for a version that
is not downloaded yet; the running download is named
- version list: smaller rows, a Downloaded tag, NVFP4 / NVFP4-NVIDIA / NVFP4-QAD named by who made them with one line each
- first-run page: the four recommended versions, then a More versions list (the Models page list)
"/loop 1m say hi, stop after 3 answers" kept firing and the model answered "stopped" every minute until /loop stop. The count
("stop after 3 answers", "three replies", "5 times"), the time ("for the next 2 hours", "stop after 30 minutes") and a stop
condition ("stop when it passes", "until the file exists") are now parsed out of the line, left out of the message the model gets,
and enforced by a new scheduler (lib/loop-runner.ts): after the last allowed answer nothing is sent again, and the person is told in
one plain sentence why it ended. The scheduler also ends the loop when an answer says it is done or stopped (a bare "Stopped."
always; "Done ..." and similar when a limit was stated; DONE as the first word when a stop condition was given in words), and after
three failed answers in a row. A busy chat waits without using up the count. Tests drive the scheduler with a scripted model and a
fake clock. Strings in 8 languages.
Messages-API clients (coding CLIs, the SDKs) add /v1/messages to their base URL themselves. The Connect dialog showed the OpenAI-style
base that ends in /v1, so those apps posted /v1/v1/messages and got a 404. The third tab now shows the root address with a
one-line note (the base-URL variable for coding CLIs), the other tabs keep /v1. strata connect cuts a pasted /v1 back to the root for
every app and says so for coding CLIs, and strata launch and the monitor's Connect view list the Messages-API address. The server (serve
and the app's host, plus the shared listener) reads /v1/v1/messages and /v1/v1/messages/count_tokens as the real paths; nothing else
is aliased. Tested with the real Python SDK client against the mock-engine server.
The catalog only walked setup.FAMILIES; Qwen3.6 lives in OWN_PATH_FAMILIES, so the app's Qwen3.6 page had an empty list.
strata_models now lists its sizes (download, memory, minimum engine 0.1.45, pictures) and strata_install plans them.
strata update exits 0 when the feed has no engine for this PC (or only one for another card family), so launch went on and downloaded the model before failing with the generic engine line. ensure_engine now re-checks the engine after the update step and refuses at once with the update's own plain sentence plus one next step; setup.py in Control mode stops at the engine step for a build of another card family, an old driver, or when launch asks for an engine (STRATA_NEED_ENGINE=1), before the download step. The app without any engine keeps its model-first order.
An experts.bin pack (the NVFP4 packs) keeps STRATA_COLD_OVERLAP, STRATA_IO_PREFETCH and STRATA_IO_PF_GATE
off unless they are set: the defaults were measured on the GGUF read in place only. The startup line and
docs/DETAILS.md say so.
- STRATA_COLD_OVERLAP defaults to 4 on Linux (0 elsewhere), STRATA_IO_PREFETCH to 1 on Linux for the
file tier (mapped GGUF in place or experts.bin), STRATA_IO_PF_GATE to 1 with io prefetch. Off switches:
STRATA_COLD_OVERLAP=0, STRATA_IO_PREFETCH=0, STRATA_IO_PF_GATE=0. One startup line names them.
- cgroup_available_bytes: every clean file page of the group (memory.stat file, active + inactive; else
inactive_file) is reclaimable, less min(half of it, limit / 64), as MemAvailable counts page cache on a
bare host. A container whose own reads left active cache was judged short of RAM and its file tier
switched to unbuffered reads (an 18 GiB RAM tier in a 32 GiB group: 1.2 GiB available instead of ~11).
Unit tests with mock memory.stat / memory.max values.
- The per-request cold-reads line only with STRATA_COLD_OVERLAP or STRATA_IO_STATS set.
- docs/DETAILS.md: the 0.2.0 Linux file-tier defaults.
With STRATA_IO_PREFETCH=1, the decode's read-ahead (pread on the I/O threads, the look-ahead two layers
deep) and its pread demand reads run only while the gate is open; the prompt path is never gated.
Checked every 16 decode windows:
- room: (the tightest cgroup v2 limit - the group's non-file memory, else MemAvailable) / the experts'
bytes; closes below STRATA_IO_PF_GATE_LO 0.55, opens above STRATA_IO_PF_GATE_HI 0.65;
- pays: closes when fewer than 128 MB were read ahead in 16 windows, or under STRATA_IO_PF_GATE_USEFUL
(0.10) of those bytes were used; then a probe every 256 windows.
Closed, the look-ahead predicts one layer and warms with the madvise hint, as without io prefetch. The
decode thread (and the fetch threads it starts) is marked by FileExpertSource::DecodeScope. The serve
log's file-tier line says the room, the share used, the state and the flips. A request's wait for the
adaptive round in flight at its start is logged when >= 1 ms.
5 pairs (Q2_0 GGUF in place, 12 GB card emulated, cold SSD, cgroup RAM caps), mode 4 + IO_PREFETCH +
gate vs mode 4 alone: 32 GB 1.16 / 1.21 / 1.70 decode, 1.86 / 1.98 prompts; 16 GB (closed) 1.00 / 1.01 /
1.10, 1.54 / 1.78 prompts; an 18 GiB RAM tier 0.91 / 0.84 / 0.86 (its IO_PREFETCH start re-checks the
RAM inside the cgroup and switches the file tier to unbuffered - not the gate's doing).
Identity gates vs b829a3189 (fixed flags, Q2_0 and IQ3_S, default and mode 4 + IO_PREFETCH + gate with
--mmap-experts): 10/10 and exact_pp 4K / 20K IDENTICAL in all four.
Owner-approved for 0.2.0. The card list says (GPU n) with the same numbers as Monitor and the picker; a loaded
model is named from the catalog (Qwen3.8-Flash-Next Q2_0) in the picker, the Loaded list, the sidebar card and the
Monitor, the raw file id only as a fallback; with Run multiple models on, This computer and the card rows show what is
left after the models that stay loaded (a short muted line says so), in 8 languages. The picker rows no longer
print the raw id beside the name.
ready_needed() drops the checkpoint's tensors/* once the model is prepared (prepared_done): the engine reads the prepared files only. have_model, ready_fetch, the already-downloaded search in other Strata folders and the disk need now use it, so a prepared model whose 95 GB of safetensors were deleted is ready instead of failing with 'not enough free disk' and a 102 GB download.
NVFP4-QAD (FP4-Local) reads pictures: vision True in setup.py and the MCP catalog; the config's vision section uses prepared/dense.gguf as the text model with the shared mmproj, like the GPTQ pack.
Tests for both; the launch test for a model without pictures now stands one in.
The coordinator's rule: the owner PC and a tester's system service run connectors for unrelated tunnels, and
quick tunnels must keep working. Running connectors are listed (pid; the command line is only matched,
never shown). A refusal only when one is clearly Strata's own: --url at Strata's port, or the same token.
Terminal (strata launch) follows the same rule. Checked with a real cloudflared on r9700.
Found running the AMD AppImage on r9700 (Q2_0 on card 1, IQ3_XXS on card 2, Run multiple models):
- the card list ignored loaded models (same text, same pre-selected card); a second model on a used card died
after two slow engine starts with 'server_exited'. Now the row says 'Already used by <model>', a free card is
pre-selected while Run multiple models is on, and a failed start there is one plain sentence (no useless retry).
- Unload refused because a chat still used the model (409) showed nothing near the button; it now shows the
sentence and the next step in the Loaded models card.
- a model that failed to start was listed in the chat picker (disabled); it is hidden.
Inside the AppImage LD_LIBRARY_PATH holds the bundled libraries, and on Ubuntu 26.04 (uutils coreutils)
dirname fails there: 'libsystemd.so.0: version LIBSYSTEMD_254 not found'. The python/python3 launchers then
computed an empty folder and the app said Control stopped / no engine. Found with the AMD AppImage on r9700.
Gyro's expert files are native_experts.txt v5 (Hadamard-folded). The engine reads v5 since the APR port
(f47b0385f); the released 0.1.45 stops at v4 (its binary and source have no v5 reader, every 0.1.45.1+ package
has). GYRO_ENGINE in setup.py and the catalog the app reads (tools/strata_mcp.py: engine in the fallback table,
engine_min on every size) now say 0.1.45.1. The host marks sizes the installed engine cannot read
(needsNewerEngine; an engine built from this checkout or with no readable version is never called too old),
refuses the download and the load with a plain sentence, and Models / first-run setup show the existing
'This model needs a newer engine' notice with Update engine and hold the buttons back.
The per-model settings already live in one text file (strata-<model>.json: args, env, sampling and the
named keys). Expert settings gets an Open settings file button (Notepad, open -t, or the system's
plain-text program) with the note that changes apply on the next start. The file is checked when the
panel opens, when the window comes back into focus, and before a load: bad JSON, a wrong value, an
unknown key (the nearest real name is offered) are said in plain words with the line, in 8 languages.
A mistake holds the start back instead of an engine that dies on it; an unknown key only warns.
The app only looked in Strata's own connector folders, so a cloudflared installed by the system
(/usr/bin, brew, winget, a .deb) was reported as not installed. Now: own folder, component folder,
PATH, then /usr/bin, /usr/local/bin, ~/.local/bin, Homebrew, Program Files. The first copy that runs
--version and is 2022.1.0 or newer is used; the Connector step names it and its place. A cloudflared
already running (system service or started by the person) is said so plainly and not doubled.
Terminal (strata launch) gets the same search and check. Plain messages in 8 languages.
SYCL has no cudaHostRegister, so the expert arena is pageable memory even where the engine calls it pinned. A copy->memcpy of a 2.3 MB blob
from it goes through the Level Zero driver's staging path on the calling thread, which never returned on some prompts (A750, default
flags, a 2,744-token docs excerpt: the thread spun in sched_yield inside libze_intel_gpu with every engine idle, 6 of 6 runs; also seen
in a 5.3K-token needle prompt of the task suite). Only USM host allocations (the --stream-experts pinned mirror) are DMA'd directly now
(answer cached per 2 MiB bucket); every other expert goes through the stager's pinned buffers like an unpinned one.
STRATA_PF_DMA_PAGEABLE=1 restores the old direct copy.
Checks: the repro completes 3 of 3 (0 of 6 before); identical first-token logits and tokens, old vs new: A750 (frozen share, 1.4K, 2.5K and
4K prompts), B70 stock config and B70 without --stream-experts (there the new path is also faster: prefill 680-740 vs 77-456 tok/s); A750 4K
prefill 79.4 vs 79.3 tok/s, B70 stock 1246 vs 1240.
Adds sycl/tools/test_prompt_deadlock.sh with the 2,744-token prompt (passes on the fix, times out on the old build).
llama.cpp refuses a GGUF whose tensor directory has a type it does not know, so the picture reader could not open a Gyro model (TQK6 / TQK7, ids 145 and 146) and Gyro was left without pictures. strata-vision now copies the key/value header into a small tensor-less GGUF and loads that vocab-only; the weights are never read or type-checked. Falls back to the file itself when the header holds a value type the copier does not handle.
Gyro-S and Gyro-M read pictures (setup.py, MCP catalog). Adds a C++ test (strata-vision-gguf-test) with a GGUF that has unknown tensor types.
The NVIDIA worker's non-bitwise result came from a cherry-pick onto a tree without the dot-count guard in launch_gu (K-quant rows have 160 dots, the sub-16 rows assume 80); the merged tree has it and the Split path is bitwise on every route.
core/gfx12_platform_defaults.hpp holds the per-platform rule (checked in arch_defaults_test); the engine log names it at the first prompt; docs/AMD_HIP.md.
Conflicts with 148-amd's gfx12 WMMA default resolved: STRATA_HIP_WMMA unset = gfx12 on every call (148-amd), 0 = off, 1 = forced (also gfx11), auto = gfx12 with n_q >= STRATA_HIP_WMMA_MIN_Q; failed launch falls back to the FP32 kernel; the 96-slot ring follows the same state; the opt-in hint is gone.
The zero-copy pre-pass ran mincore for every CPU expert of a layer, the RAM copy's too: -2..-9% decode
with an 18 GiB RAM tier. It now skips the RAM-resident and override experts (b32: 0.98/0.97/0.91 ->
1.05/1.10/1.08 vs mode 0, 3 pairs). A file without a residency query (not Linux) never goes zero-copy.
- STRATA_IO_PF_SCHED=1 (with STRATA_IO_PREFETCH=1): confidence-scored read-ahead from the router
lookahead, per-window speculative byte budget, deadlines (late jobs dropped), useful-share feedback;
new per-request counters (read ahead MB, used before demand, late, not admitted).
- STRATA_RAM_ADAPT2=1 (with STRATA_RAM_ADAPT=n): RAM-tier promotions ranked by expected stall saved per
byte (drive vs page cache), read on the source's I/O thread, installed at window boundaries; the
prompt's routing seeds it with a limited weight (Prefill::on_routes).
- STRATA_COLD_OVERLAP: 1 reads a layer's staged blobs in the background while the pool computes the
ready experts, each landed group computed as it lands; 2 measures R / C_r / C_c per layer; 3 adds
zero-copy and 4 is zero-copy alone: an expert the page cache holds whole (GGUF in place, Linux mincore)
is computed from the mapping through a per-job NativeFmt (ExpertJobMulti::fmt) instead of being
assembled into a stage buffer. Same bytes, same rows, same bits.
Default (all unset) is the old code path: identity gates vs b829a3189 IDENTICAL (Q2_0 and IQ3_S,
fixed flags, 10/10 + exact_pp 4K/20K); mode 3 with --mmap-experts also 10/10 + 4K/20K IDENTICAL.
Teacher-forced KL against the old kernel on an R9700 (Q2_0, int8 KV, fixed cache so the floor is 0.0000): 0.0003 over 6,024
positions (p99 0.006, top-1 99.8%), 0.0014 over 1,760 positions of 8 long documents (top-1 98.9%); HumanEval, reasoning, agent,
needles, CJK and tool calls equal. gfx11 stays opt-in. The 96-slot prompt ring follows the kernel (qsa_hip_wmma_active()).
The start-up hint about the switch is gone.
The auto chunk scan only tested the GPU cache. A lent slot's expert comes back from the RAM copy or the model file for the length of a
prompt, and every chunk streams the whole file set; past the page cache that set is re-read per chunk (28-45 GB per 29K prompt on
2x RTX 3060 / 32 GB / IQ2_XS at 8192-token chunks, ~1 GB at 4096). prefill_plan.hpp is the pure model (file set, miss share fitted to
#2824, additive stream + compute + drive time); the serve scan and the generate path refine the VRAM pick with it, after the RAM copy
exists. A set that fits the page cache keeps today's pick. STRATA_PREFILL_HOST_PLAN=0 turns it off; --prefill N is untouched.
The catalog no longer forces images:false in the app; a model reads pictures when it is picture-capable and the engine component carries strata-vision.
API models: a model reads pictures when the person ticks 'This model reads pictures' or the server says so (model list modalities, llama-server /props); the chat sends image_url to any OpenAI-style API that does, and says in plain words how to fix every case where it cannot.
Importing a model folder recognises the picture reader next to it and says plainly when a model type is not ours (use a Custom model API).
native_w11_kernel's gate/up path walked 256-value super-blocks only; a Q2_0 row is 64-value blocks, so gate/up of that
format now takes the same one-block-a-stage walk its down path has. R9700 Q2_0: prompt +16.7% (4K) / +19.0% (20K),
5/5 pairs; hip_prefill_fused_iq Q2_0 / Q2_0: fused vs MMQ rel RMS 1.6e-5.
The RTX 5070 check (48 SMs, IQ2_XS pack, experts partly in RAM): the
fused int8 prompt experts were 4.7% slower on 20K prompts (0/6 pairs) with
no decode gain, and one token a block row could not be told apart from
token pairs in the hyper-connection down projection (an earlier run leaned
to pairs). So on a 12.0 card with fewer than 80 SMs the prompt experts
keep MMQ (STRATA_PF_FUSED=1 still takes the fused kernels) and the down
projection takes token pairs at every even launch (STRATA_GR_DOWN_TG=2);
12.0 cards with 80+ SMs (the RTX PRO 5000 measured) are unchanged.
A new prompt's first chunk waits for its n-gram rows (65,536 random SSD
reads at 4K tokens) unless the table is in RAM. RTX PRO 5000 (Linux, NVMe,
251 GB, a fresh engine per prompt, 3 rounds, the same bits): prompt 4K
1298 -> 1059 ms (-18%), 20K 4644 -> 4223 (-9%), 98K 20447 -> 20027 (-2%),
decode +0.8%. On an R9700 (slow prompt) the read hides behind layer 0 and
nothing changes; under 64-200 GB RAM caps (cgroup, page cache included)
ram mode equalled direct there and cost ~6 s of a cold start.
The rule: not low-RAM mode, Linux (not WSL), and RAM >= the model's own
need + the table + 16 GB (Q2_0: 93 GB, IQ3_S: 107 GB); a smaller PC keeps
the SSD reads as before. Setup says what it did and how to undo it.
Tests: RotationalDisk gets three SSD cases; the golden configs keep the
Windows behaviour (ple_ram_platform_ok mocked).
Owner decision, the AMD gfx12 rule: on a compute capability 12.0 card the
native packs' prompt experts run on the fused int8 kernels (moe_fused_iq)
for the gate/up formats measured there - the IQ3_S pack's IQ2_S, IQ3_XXS,
IQ3_S and IQ4_XS layers with IQ4_NL / Q2_0 down. IQ2_XXS / IQ2_XS gate/up
keep MMQ until measured. This rounds differently from MMQ:
RTX PRO 5000, IQ3_S: prompt 4K +7.0%, 20K +5.2%, 98K +3.4% (3/3 pairs);
teacher-forced KL against MMQ 0.0002 (5998 positions, noise floor 0.0000)
and 0.0010 (1760 positions after ~6K-token documents); first-token KL
against the FP16 prompt path 0.01181 vs MMQ's 0.01236 (48 prompts of
1.5K-24K tokens); HumanEval 156/164, reasoning 26/30, agent 6/6, needles
6/6, CJK 8/8, tool calls 4/5 - the same as MMQ.
STRATA_PF_FUSED=0 keeps MMQ, =1 takes every covered format (as before),
STRATA_SM120_PERF=0 drops the 12.0 default. Other cards: unchanged.
native_gu_swiglu_kernel: a 512-thread block owns 32 hidden columns of a
group; each 16-lane sub-warp computes gate row r and up row r together
(row_dot2_80_sub16: both rows' loads before the applies, each row's sums in
row_dot_80_sub16's order), the block forms h = silu(gate) * up in shared
memory, and warp k writes entry k's q8_1 block with q8_1_store. The values
are the separate kernels' bit for bit (gate/up are the same floats, the
product is rounded on its own as in swiglu_q8_1_entries_kernel); it saves
the gate/up round trip and one launch per layer.
RTX PRO 5000 (IQ3_S, all experts resident): a layer's gate/up + SwiGLU
35.5 + 2.4 us -> 36.3 us; fixed 4-token rounds 11.92 -> 11.80 ms; default
flags, 6 interleaved pairs: decode +0.6% story, +0.9% code, +0.5% 8K
document, 6/6 pairs each, the same tokens in every case.
On by default for compute capability 12.0 (sm120_series.hpp); elsewhere
opt-in with STRATA_GU_SWIGLU=1; STRATA_GU_SWIGLU=0 turns it off.
Measured on the RTX PRO 5000 against the token pairs of the previous commit,
fixed verify rounds, same text: 2 tokens 9.44 -> 9.24 ms, 3 tokens 10.81 ->
10.56, 4 tokens 11.94 -> 11.90, 5 tokens 14.04 -> 13.70. Each (row, token)
is still summed by one warp in the same order. STRATA_GR_DOWN_TG=2 / 0 picks
pairs / no split as before.
The shared table was measured on an RTX 5070 (48 SMs) together with an RTX
3060, so a small 12.0 card keeps it; the RTX PRO 5000 table (110 SMs) now
applies to 12.0 cards with 80 or more SMs (il_cc answers 1201 for them).
The 5-8 column table stays for every 12.0 card. docs/DETAILS.md: the
measured decode gain of the sm_120 defaults, docs/MMVQ_IL_TABLE.md: the
new tables and switches.
Measured on an RTX PRO 5000 Blackwell 72 GB (IQ3_S, every expert in VRAM),
fixed 4-token verify rounds, story / code, the same text as before:
STRATA_SIDE_MIXER=2 13.67 -> 13.15 ms a round
STRATA_ATTN_DAILY=1 -> 13.31
STRATA_GR_DOWN_TG=2 -> 13.34
STRATA_GR_UP75=1 -> 13.44
all four 13.70 -> 12.23 ms (2-token rounds 10.37 -> 9.59)
Each is bitwise equal to the kernels it replaces (their parity tests; the
identity gate). include/strata/core/sm120_series.hpp holds the rule: on for
compute capability 12.0 unless STRATA_SM120_PERF=0; each switch's own
variable still wins in both directions on any card. The down projection's
token pairs run at every even launch there (Turing keeps 4 and 6 only).
STRATA_ATTN_DAILY / STRATA_ATTN_MERGE_V2 and STRATA_SIDE_MIXER are now
decided per device (a layer split can mix cards). Other cards: unchanged.
A verify window of 5+ rows fell from the interleaved small-batch kernels (1-4
columns) back to native_mmvq_multi: on an RTX PRO 5000 Blackwell the dense
projections of a round jumped +2.0 ms from T=4 to T=5 (the main Q5_K head
521 -> 1011 us). native_mmvq_il_kernel now also runs 5-8 columns from the
cp = 8 interleaved copy (1-2 rows a warp), every value bitwise native_mmvq's
(mmvq_il_parity, T 1-8, every rows choice).
Tables for compute capability 12.0, read off mmvq_il_parity --bench
--emit-table on the RTX PRO 5000 (3% rule): kIlRows_120 for 2-4 columns (it
takes the interleaved kernels in more cells than the shared table, e.g. the
head at 2 columns 507 -> 367 us) and kIlRows58_120 for 5-8 columns (the head
at 5 / 8 columns 899 -> 491 / 1178 -> 667 us). Other cards keep their
tables and native_mmvq at 5+ columns.
STRATA_MMVQ_IL_WIDE=0 turns the 5-8 column path off; STRATA_MMVQ_IL_120=0
gives a 12.0 card the shared 2-4 column table again.
Conflict in src/prefill/prefill.cpp (step 2b): the split rotates mixed into Xr (APR, had_gu)
before 0.1.45's quantize_scatter (NVFP4 activations keep the gather); 0.1.46 adds the pf_wmma
FP16 gather branch. Both kept: wmma_l gathers FP16 rows (and rotates Xs / the swiglu output Hh
in fp16 when the layer is Hadamard-folded), otherwise the split's rotate then quantize_scatter
or the NVFP4 gather.
Merge resolution as in f-145-perfprs-withvolta: iq_kernels.cu (IQ3_XXS / IQ3_S staged sign-mask tables over the EXPDEC grid staging), prefill/kernels.cu and prefill/prefill.cpp keep both sides; CMakeLists keeps both test blocks.
The switch made every card answer as sm_75, so a Tesla P100 (sm_60) launched the m16n8k8 prompt-attention kernels, whose sm_60 code is a __trap(): the engine died on the first long prompt with "prefill copy_i32: unspecified launch failure". A card below sm_75 now ignores the switch and runs the kernel it runs without it (the same tokens).
Claude Code asks for max_tokens 32000 on every request and the Harness for 16384; on a 32K context the
prompt fits but prompt + max_tokens does not, and the server answered 400 "request exceeds the context
window ... requests are never truncated" before the first token.
Service.prepare() (the one place chat/completions, messages and responses all pass through) now clamps
max_tokens to ctx - CTX_SLACK - prompt when the prompt fits, logs one line, and the answer_room thinking
budget (3/4 of max_tokens) is taken from the clamped value. A prompt that alone does not fit is still the
plain 400 ("no room to answer"). --no-fit-max-tokens / "fit_max_tokens": false brings back the strict
llama.cpp 400; --fit-max-tokens stays accepted.
Tests: serve/test_server.py MaxTokens (shortened by default in the three dialects, exact fit and one over,
prompt-too-long 400, thinking budget follows, strict mode), serve/test_context_overflow.py (default
service), serve/test_monitor.py (error record uses the strict mode).
tools/nvfp4_local_pack.py convert|ple|mtp|verify reads NVIDIA's ModelOpt checkpoint and writes what the existing NVFP4 path takes:
the CSF block scales are decoded to plain E4M3 (experts then repack bit for bit), MXFP8 weights are dequantized exactly and take the
Q8_0/BF16 policy of nvfp4_convert, the n-gram table becomes a one-tensor GGUF of 90-byte rows, the MTP head the folder mtp_pack reads.
ngram.cpp reads that table as a new PLE format (I8 bytes, strata.ple.format = nvfp4x16).
An adopted model reports the parent of the person's own GGUF folder as its data folder; the next preparation sent that as the folder for the packs and Control refused it ('a new data folder must be named Strata'). A preparation from a GGUF folder now uses Strata's own download folder. Found by the real-box check on P2003; test fails without the fix.
Control answers a refused folder with exit 5 and the plan (ok:false, issues); the host threw that away and the page said only 'The files could not be checked'. The plan with its reasons now reaches the page, which says 'Strata can't run these files yet' and keeps the checker's codes under Details. Found by the real-box check (a Gemma vocab GGUF on P2003).
- Models: the family switch (Qwen3.8-Flash-Next, Qwen3.6-35B-A3B) is back in the navbar without the word Model; version, context and computer use, the tuning line and Load/Tune sit in one card beside This computer, so the tab fits the first view; image generation moves into Expert mode
- Tune: one result card for faster, already the fastest, couldn't measure (busy PC, with plain copy and Try again), failed and stopped, driven by the host's code, reason_code and values; the same Tune dialog frame for all of them
- Tune running: the step icon glows and a small dot beeps instead of a ring spinning around a square tile; reduced motion respected
- Expert mode: Context memory and Low-RAM mode show the active tuning's values, the engine settings list what the tuning sets, and loading does not re-prepare the model because of them
- the tuning line (Using your tuning / Switch off) is a calm line with an icon
- strings in 8 languages; tests in tune-outcome.test.mjs
- Settings > Storage: downloaded optimized models show family + quantization with the full name as a tooltip, size aligned, Delete at the end
- Engine > Diagnostics: one View logs button; Save logs lives in the logs view
- About > Privacy: six short lines across the box instead of one long paragraph
- Monitor: the request summary is a plain sentence ('In 18 requests your average output was 47.4 tok/s and prompt reading 1,332 tok/s, reusing 91K tokens.'); the screenshot leaves out Model state and Conversation cache
- Tasks: Bypass permissions replaces 'Allowed to make changes' and is the default for a new task; an opened task has an empty left navbar and no Model section, its name in the page, and a readable Markdown last answer; no deleted-tasks list, restore or suggestions; + icon opens the editor modal (Draft it with AI inside it); removing says 'Task removed'
- strings in 8 languages; tests in ui4-fixes.test.mjs
- while a real run works, an animated A/B card shows what the assistant is doing (host activity), elapsed time and Stop (TimeSkipWorking); the sidebar no longer opens at the start and opens only when the finished run changed files; a run that changed nothing says so in the chat and has no tab
- the finished A versus B (TimeSkipCompare) keeps one window size whatever was made: text and Markdown rendered with changed words marked; HTML as two sandboxed previews (no permissions, no scripts, no outside requests) with a View code toggle; pictures side by side, click to enlarge; code and other files as a folded line diff; several files through a compact file switcher
- host: changed pictures are listed as pictures and served from the run's own folders by GET /api/timeskip/file (images only, inside the run, size-capped)
- strings in 8 languages; tests for the file kinds, word and line diffs, the HTML policy, the activity words and the picture route
- status line under the newest message: pulse, a rotating playful word only while it thinks, the real action once a tool runs, elapsed time and tokens; only the newest status frame is ever shown and it is cleared on idle, done, error and stream end (src/lib/live-status.ts)
- reasoning is one quiet line (what it is thinking about, then 'Thought for 38s'); opened it follows the newest line unless the person scrolls up
- goal bar above the input: small, icon-only pause/resume, edit, remove, from the goal frame, the SSE goal event and /api/goal (src/lib/goal.ts); resume sends goalAction:'resume' with no user bubble
- all strings in 8 languages; tests in tests/live-chat.test.mjs
Locales: both sides added keys from s2676, so ui2's 88 keys are renumbered s2849-s2936 in all 8 files. events.mjs: ui2's verbsFor target plus agent2's ask card. index.mjs: the node:path import takes both sides' names (relative, sep). The workspace default is ui2's (Documents/Strata/Workspace).
- approval cards: one plain question per tool call (server/agent/ask.mjs maps commands and files to sentences; src/lib/ask.ts holds the translated templates), a compact calm card, the exact command under Details
- Tasks: the opened task shows schedule, what it may do, last result, instructions and last answer; back link before the title like the chat's navbar; pause/resume and delete icons, Edit, Run now; pause and play icons added to the sprite
- Models: 'This computer' shows graphics memory and RAM bars with what the chosen version takes, and one plain line about the SSD (src/lib/native-fit.ts, from the catalog's memory_gb and the hardware report; the n-gram table is not counted)
- Models: one-line version list: Recommended (the fixed four), Swift, GSQ-RCO, NVFP4, Gyro, Unsloth groups, Use your own GGUF; RAM+VRAM in bold, download size muted, 'Uses SSD N GB' only where it applies
- Tune: five steps with their own animated icons and states, a live speed readout and sparkline, a result card with the gain and what it was saved as; reduced motion respected; the tuner now reports each measurement's speed
- catalog: memory_gb and mem_total per version (tools/strata_mcp.py)
- tests: ask, native-fit, tune-phase, autotune speeds, memory figures; headless checks (ui-flows ui2/ui2d, scripts/approval-flow.mjs), light and dark
- agent context: the Harness reports a cached prompt as cacheReadTokens, apart from inputTokens; the context now counts input + cache read + cache write + reply of the last request (a long agent chat showed 0-3%; checked against a real Swift IQ2_XS 64K run: 8,152 tokens, matching the engine's own 8,042-token prompt + 110 reply)
- first workspace: Documents\Strata\Workspace (display name Strata) unless STRATA_WORKSPACE names one, never the app folder; a saved first workspace that is the app folder moves there and its chats follow through an alias
- workspaces can be renamed (display name only, path kept): PATCH /api/workspaces/:id, pencil in the sidebar, 8 languages
- tests: workspaces.test.mjs, cache-aware context in functional-fixes.test.mjs
#1498 pins the KV host copy with cudaHostRegister under WSL; the switch forces either order. Measurements: 5 interleaved pairs on an RTX 3060, an RTX A4000 and a Tesla P100.
- agent path: live stats frames and a final tok/s from the stream timing; the context is the last request (one source)
- tool rows say what happened in words (Wrote a file, Ran a command, Couldn't write the file) with the file or command; approval cards show it too
- a launch that returned without error reports 'Started.' instead of 'nothing to show'
- agent system prompt replaced by a short plain one; goal, skill, workflow and fork tools left out (first message about 7.5k -> about 4.3k tokens)
- Models: the running model reports the profile it was started with and the page shows it; Reasoning popover names the loaded native model
- installed model with other settings: 'Apply settings and load', no size, no download wording
- plain chat: agent:false or tools:false answers directly and never runs a tool
- tests: stats/context/verbs/launch/prompt size, server-side enable_thinking=false with the agent's request shape, plain chat over HTTP, ui-flows profile scenario
cmake/strata_ggml_tq_overlay.cmake writes, at configure time, a copy of the pinned ggml-cuda headers (and ggml.h, quantize.cu) into
the build folder with TQ2_T / TQK6 / TQK7 routed the way IQ2_XS is: same Q8_0_16 tile layout, dot kernels and tile configs (the
fork's own configs for them are copies of IQ2_XS's), three template instances, the tile loader and tq.cuh from third_party/ggml-tq.
Only the strata_mmq target sees the copy; every other type is compiled from the untouched files. Anchors are checked: another
llama.cpp turns the overlay off with a message (FP16 prompt path). moe_mmq.cu knows the trellis block geometry itself.
The tile loader and the trellis tables are agentionai/llama.cpp's (MIT, ggml authors; Laurent Zuijdwijk), ported from
agentionai/llama.cpp 93a24e5 ggml/src/ggml-cuda/{tq.cuh,mmq-load-tiles.cuh,tq-lut-i8.cuh}.
Checked: Gyro-S and Gyro-M exact_pp 4096/20000 residual dumps IDENTICAL to rc2 (R9700 gfx1201); Q2_0 and IQ3_XXS IDENTICAL to f-split
on gfx1201 and sm_86; ctest ggml_tq_overlay.
index.mjs: createNativeControl gets control, appUpdater and hardwareProbe; the agent bridge gets componentsDirectory() and the bundled Harness folder.
native-control.mjs: this tree's file kept, s4b's parallel choice, fit_max_tokens, instance parallel/mcpServers re-applied after buildConfig; its own retry around control.start is folded into load-host's: ONE second try in all, which carries the memory change and/or the drop of several-at-once, with load-host's plain sentence (FALLBACK_NOTE only when no memory change applies). Test added.
locales: both sides had added keys s2571.., so s4b's seven task keys are renumbered s2615-s2621 in all 8 files.
native-control.mjs: this tree's file kept (S6 updates, load retry, remove), s5's buildConfig split out of loadEngine, auto-tuner, tuneConfig/tuneStatus/autotune, autotune and recheck operations and presets re-applied.
NativeModels.tsx: s5's Tune model button (the real tuner) and NativeTune dialog; the old short-benchmark Tune button stays out. native.ts: job progress field.
The planar kernel of #2340 is the wide kernel's arithmetic, but it was launched without the 256-GRF mode that #2207 gives the GGUF-layout
wide kernel, so on an Alchemist card it spilled: mmvq_bench 2560 x 248320 on an A750, 4 columns, 1.78 ms (GGUF layout, large GRF) against
15.4 ms planar, 8 columns 4.8 against 48 ms. Same rule as #2207 (STRATA_MMVQ_LARGE_GRF, default on Alchemist only), same arithmetic.
On 2x RTX A4000 behind a host bridge cudaDeviceCanAccessPeer says 1 and cudaDeviceEnablePeerAccess succeeds, and
cudaMemcpyPeerAsync then returns success having moved nothing (1,048,576 of 1,048,576 floats wrong in both directions
in a stand-alone test; right without peer access enabled, the driver stages it). The peer prompt path trusted it: the
peer's input rows were zeros and its results never reached the primary, the first prompt's MoE output was garbage and
the logits NaN (--peer-device 1, --peer-reserve-mib 1500..2100; STRATA_PF_PEER_COMPACT is not the cause).
PeerExperts::open now copies 64 KiB each way once after enabling access; a wrong result turns peer access off again and
takes the host route (the one STRATA_PEER_NO_P2P=1 asks for), and says so. A test that cannot run keeps the old trust.
Reported by @CaptRauti on #2666 (WDDM, 2x RTX 3060: the no-P2P route is the one they ran).
Built on P2006 from the f-split tree and run through the Terminal archive: resolve (kfd gfx1103), tune, serve and client, TUI, Ctrl+C/SIGHUP, saved preset, memory back to baseline.
Both files are one GGUF with the n-gram table inside: 58.5 GB and 92.0 GB downloads for 27.6 GB and 35 GB of VRAM + RAM together.
Neither is a default; the MCP recommendation names them only where the 2-3-bit sizes and the Coder do not fit. Pinned to the Hub
revision 1ada08422e40 (sizes and SHA-256 from its LFS pointers). Model license: Qwen Community License 1.0.
For tq2_t / tqk6 / tqk7 experts (--synthetic, --synthetic-hadamard), the
gate and down rows on every ISA the CPU has against the scalar kernel bit
for bit, and the scalar kernel against ggml-cpu's vec_dot on the same
Q8_0 activations. The CPU path of check (b) already runs through the new
kernels via native_gu_rows / native_down_rows.
Ported from agentionai/Strata rc2 7d0ecd0 (rc2 = 9ff02c9), author Laurent Zuijdwijk.
Expert-cache misses of the Gyro files run on the CPU workers. Until now those
rows went through ggml-cpu's vec_dot one token at a time. Strata now has its
own kernels for the three trellis types (tq_cpu.hpp), chosen at run time like
the i-quant ones:
- the 32768 x 4 int16 state table (one 8-byte lookup per four weights, built
from the codebook in third_party/ggml/ggml-common-tq.h, now also visible to
host code), scalar state extraction, four table rows per YMM by broadcast
loads + VPBLENDD, VPMADDWD (AVX2) or VPDPWSSD (AVX-512 VNNI);
- the weights of a block are decoded once for all the tokens routed to the
expert (a verify window), then each token is a few instructions;
- one fixed arithmetic for every ISA: exact int32 sums per Q8_0 sub-block,
four float FMA chains, a fixed final sum. AVX2, AVX-512 and the scalar
reference give the same bits, and a token gives the same bits alone or in a
group of any size (no group-size rule needed, cf. #152).
native_gu_rows / native_down_rows take them for types 144-146 on AVX2 and
AVX-512 CPUs. STRATA_NO_TQ_KERNELS=1 restores ggml-cpu's vec_dot,
STRATA_TQ_ISA=scalar|avx2|avx512 caps the ISA. A CPU without AVX2 keeps
ggml-cpu.
tq_cpu_parity (CPU only, ctest tq_cpu_parity and tq_cpu_parity_avx2):
AVX2 / AVX-512 vs the scalar kernel bit for bit for groups of 1-8 tokens,
vs ggml-cpu's vec_dot (rel ~1e-7, float order) and vs to_float in double
(rel ~1e-5, the int16 table), plain and Hadamard-rotated, for Gyro-S
(tqk6/tqk7) and Gyro-M (tq2_t) experts through the engine entry points.
`tq_cpu_parity --bench` is the micro-benchmark (clock-calibrated cycles).
Zen 5, one thread, weights streamed from DRAM, core cycles per 128-weight
block (one token / per token in a group of three):
ggml-cpu vec_dot, AVX2 build: 108-117 / 108-116
ggml-cpu vec_dot, AVX-512 build: 64-66 / 61-62
Strata AVX2: 47-49 / 22-23
Strata AVX-512: 47-49 / 22
(Q2_0's AVX2 kernel: ~20; the repacked AVX-512 Q2_0 kernel: ~7)
Ported from agentionai/Strata rc2 7c85f1d (rc2 = 9ff02c9), author Laurent Zuijdwijk.
hadamard_rows / hadamard_rows_f16 still exit on an invalid block, like
check() does on a failed launch, but now through one helper whose message
names the entry point, the block, the row width and the requirement (a power
of two in [2, 1024] dividing the width), and where the mismatch comes from.
Ported from agentionai/Strata rc2 48ff8b9 (rc2 = 9ff02c9), author Laurent Zuijdwijk.
Not changed to an fp32 rotation: hadamard_rows_f16 already does its
arithmetic in fp32 and rounds once on the store, so the only extra error is
the fp16 rounding of the SwiGLU output before the rotation, while an fp32
copy would cost a T*K*640 float buffer or a fused kernel on the non-MMQ
fallback. The comment states this, the precision cost and the parity bound
(relative L1 < 2e-3 against the float rotation); the gate/up input refers to it.
Ported from agentionai/Strata rc2 50a1972 (rc2 = 9ff02c9), author Laurent Zuijdwijk.
The v5 rewrite for Hadamard-folded experts was a re.sub on "v[34]": on any
other header it changed nothing and still wrote hadamard.txt, an
inconsistent pack the engine refuses at load. native_experts_v5() now
rewrites v3 and v4 only and otherwise returns an error naming the version
found and the supported ones; main() prints it and exits 1 before any file
of the pack is written. Tests: both rewrites, the refusal for v2/v5/v6/v10
and a missing header, and a full pack run that must leave no
native_experts.txt or hadamard.txt behind.
Ported from agentionai/Strata rc2 235787c (rc2 = 9ff02c9), author Laurent Zuijdwijk.
run_impl called hadamard_device_signs for every folded layer of every chunk,
which takes a process-wide mutex and may cudaMalloc. Prefill::init now
resolves the per-layer gate/up and down sign pointers on its device into
Impl::had_sx / had_sh, as peer_had_signs does for the peer and the layer
split's helper (P.had_sh), and run_impl only reads them. carve no longer
warms the device table on every relayout (init's lookup does that now, once);
the comments on peer_had_signs and the new members say what is resolved when.
Ported from agentionai/Strata rc2 3d191fa (rc2 = 9ff02c9), author Laurent Zuijdwijk.
run_impl asked hadamard_device_signs for the down projection's signs at a
literal 640 (and gate/up's at N), while carve and the peer path resolve them
at lay.fmt[l].n_ff / n_embd. The widths are part of the device copy's key, so
the two only shared an entry because the widths happen to match. All lookups
now use the layout's widths.
The other 640s next to the rotations are the row width of the prompt path's
expert buffers and products (swiglu, quantize, GEMM), fixed for this model
like N; Prefill::init now refuses a native layout whose experts are not
N x 640 instead of assuming it. No other sign or rotation width in the tree
is hardcoded (verify, peer and remote experts use the geometry).
Ported from agentionai/Strata rc2 ff463c8 (rc2 = 9ff02c9), author Laurent Zuijdwijk.
Decode reads the int8 codebook tq_lut_i8 (one global scale, 0.85% relative
RMS); the FP16 prompt path reads the fp16 codebook and the MMQ path is the
GGML fork's tile loader. The comment over tq_load / tq_apply now says that
only the decode-once and per-entry kernels are bitwise equal to each other,
and that native_expert_parity bounds the decode path against ggml's float
dequantizer at relative L1 error < 3e-2 (the prompt dequantizers at < 1e-6).
The same goes into a short "Numerics" section of docs/GYRO.md and the parity
test's header.
Ported from agentionai/Strata rc2 457047e (rc2 = 9ff02c9), author Laurent Zuijdwijk.
nat_xr_ (T x N floats) was taken from the verify arena for every pack, though
only a layer whose gate/up is Hadamard-folded reads it. It is now carved only
when some layer of the native layout has had_gu, as the prompt path does with
its rotated copy; the one reader is under that layer's f.had_gu, so it never
sees the nullptr.
Ported from agentionai/Strata rc2 687b7e6 (rc2 = 9ff02c9), author Laurent Zuijdwijk.
The GPU path for Hadamard-folded experts is now validated end to end on
CUDA sm_86, sm_89 (also a two-GPU layer split and peer decode) and sm_120,
and on HIP gfx1151, so `strata generate` no longer exits without
STRATA_APR=1 (the variable is ignored if set). It logs a warning instead for
the two setups that have not been run end to end yet: a HIP GPU other than
gfx1151, and the P2P prompt peer (Prefill::set_peer).
docs/GYRO.md and tools/gyro_setup.sh no longer set STRATA_APR=1; the setup
script's AMD note now says what was validated on gfx1151.
Ported from agentionai/Strata rc2 0fabd20 (rc2 = 9ff02c9), author Laurent Zuijdwijk.
Gyro-S is 58.5 GB (54.5 GiB). Gyro-M and AMD have now run end to end;
long-prompt prefill is measured. The measured section now has the int8
decode numbers (rc1 includes them and --prefill auto), the server
protocol, Gyro-M and the 24 GB RTX 3090; the cache-size rows stay,
labelled as the 2026-10-02 build.
Ported from agentionai/Strata rc2 c494121 (rc2 = 9ff02c9), author Laurent.
Upstream 0.1.39 calls the decode-once loaders as
S::template load<STAGE_GRID>(row, kbx, iqs, s_grid) (the IQ types can stage
their grid table in shared memory). The trellis loader has no grid to stage;
it takes the template parameter and the table pointer and ignores them.
Found by the first CUDA build of rc1 (CUDA 12.4, sm_89).
Ported from agentionai/Strata rc2 07e9afd (rc2 = 9ff02c9), author Laurent Zuijdwijk.
set_peer and set_stage_helper no longer decline an APR pack. Their MMQ
share now takes the same rotations as the primary's MMQ path:
- gate/up input: the primary sends its rotated copy (m.Xr = H (s * x), made
once for its own MMQ quantize) instead of `mixed` to the peer / helper
card, over P2P or the mapped host route. One rotation serves both cards,
bit for bit the same input; `mixed` still feeds the shared expert.
- down input: on the peer / helper card, hadamard_rows on H between
mmq::swiglu and mmq::quantize, in the compact (per group) and the
layer-sized buffers, with that card's own copy of the layer's sign
vector (PeerPrefill::had_sh, made at setup on the peer's device).
- The fused int8 prompt kernels never take a folded layer (fused_nat), and
bind_stage_helper already refuses layers on the fused kernels, so the
helper's folded layers are always on MMQ with the separate
swiglu -> rotate -> quantize passes.
- The trellis types need nothing more there: the peer gathers the blobs
with mmq::gather_native and runs the same MMQ instances (STRATA_MMQ_TQ).
Not run on a GPU yet. Needs a 2-GPU box: Gyro-S with --peer-device (and
separately a layer split with STRATA_PREFILL_HELP=1), the first token's
logits and a greedy generation against the single-GPU run.
Ported from agentionai/Strata rc2 c64e1ec (rc2 = 9ff02c9), author Laurent Zuijdwijk.
Port of the agention llama.cpp CUDA change (perf/tq-cuda-decode 0005): the 2048-point trellis
codebook as int8 pairs with one global scale (max |point| 3.984 -> 127; relative RMS error 0.85%
of the codebook std). A trellis step's 4 weights are two 2-byte entries = one int, so a decoded
lane (TqLane) is 4 ints instead of 16 floats (5 registers instead of 17) and tq_apply is 4 dp4a
instead of 16 int8->float conversions + 16 FMAs. load/apply stay split: decode-once and per-entry
kernels remain bitwise equal. The dequantizers (prompt path, embedding) keep the fp16 codebook.
native_expert_parity --synthetic tqk6/tqk7 tq2_t/tq2_t and --synthetic-hadamard: 0 failures (RTX
3090, CUDA 12.4); GPU rel vs reference 1.18e-2 -> 2.13e-2 (tqk6/tqk7), 2.20e-2 (tq2_t), within the
3e-2 bound; gpu decode-once vs per-entry: bitwise equal.
Ported from agentionai/Strata rc2 87bad56 (rc2 = 9ff02c9), author Laurent.
The Gyro config used --prefill 512, so the prompt path borrowed only 364 cache slots. Measured on
an RTX 3090 (24 GB, cache auto = 15,756 slots, ctx 131k), same prompts: 4K-token prompt 767 ->
1,325 tok/s, 16K 817 -> 1,870 tok/s, on par with ISTA IQ2_XS under setup.py's config (1,401 /
2,058); decode unchanged within noise.
Ported from agentionai/Strata rc2 a4d70dd (rc2 = 9ff02c9), author Laurent.
docs/GYRO.md: build against the agentionai llama.cpp fork, pack the published GGUF unchanged,
Strata's MTP runtime, server config, RTX 5090 measurements (full and 16/8 GB-sized expert caches)
and known limits. tools/gyro_setup.sh does every step on Linux (CUDA or HIP).
Ported from agentionai/Strata rc2 759675e (rc2 = 9ff02c9), author Laurent Zuijdwijk.
carve() takes m.Xr (T x 2560 floats, the Hadamard-rotated copy of `mixed` for
the MMQ path) from the MoE set of the shared region, but moe_set_bytes, which
sizes that region (and bytes_needed / the borrowed slots), did not count it.
With an APR pack and MMQ on, the MoE set is the region's largest, so its last
take overflowed and init failed with "device buffers for a chunk of N tokens
do not fit" at every chunk and cache size, borrowed or own buffers alike
(reported on an RTX 5090 with the published Gyro-S). Workaround on an older
build: STRATA_PREFILL_MMQ=0 (the FP16 prompt path takes no Xr).
Ported from agentionai/Strata rc2 09042dd (rc2 = 9ff02c9), author Laurent Zuijdwijk.
- iq_kernels.cu: dq_q6_k (llama.cpp's dequantize_block_q6_K, two halves per
thread) in dq_dispatch; embed_type_supported and iq_row_bytes take Q6_K
(embedding only) - Gyro-S's token_embd.
- ngram.cpp: PleTable reads Q8_0 rows (170 B; PLE_ROW_BYTES_MAX raised), mapped
and direct SSD I/O - Gyro-M's per_layer_token_embd.
- native_expert_parity: --gguf-embd (random token_embd rows via iq_embed_rows
vs ggml to_float), --gguf-ple (random PLE rows via PleTable, both I/O modes),
--load-check (the engine's start-up checks for a pack + GGUF, no GPU load).
gfx1151: Gyro-S token_embd (Q6_K) and Gyro-M token_embd (Q8_0): 300 rows,
0 values differ; Gyro-M PLE (Q8_0) and Gyro-S PLE (IQ4_NL): 320 rows, 0
differ, mmap and direct. --load-check on packs of both published files:
0 failures. (Measured before the rebase onto 0.1.39.)
Ported from agentionai/Strata rc2 cc287a6 (rc2 = 9ff02c9), author Laurent Zuijdwijk.
- iq_kernels.cu: the TQ lane dot split into tq_load (16 codebook values) and
tq_apply; vec_dot_tq_q8_1 = apply(load), Split<144..146> + kSplit, so
verify windows decode each weight lane once for all entries (bitwise
equal to the per-entry kernels).
- MMQ: with STRATA_GGML_DIR = the agention fork, the fork's
mmq-instance-{tq2_t,tqk6,tqk7} are built and moe_mmq takes the types
(STRATA_MMQ_TQ). prefill.cpp rotates a copy of `mixed` (Xr) before
mmq::quantize and H between swiglu and quantize; device sign vectors are
made at prefill setup.
- hip_prefill_mmq_parity: TQ gate/up (1280x2560) and down (2560x640, odd
blocks per row) passes; MMQ_PARITY_REPEAT; the output sentinel is now set
with hipMemsetAsync on the test stream (a null-stream hipMemset could land
after the MMQ kernel and make every output look unwritten).
- Rebased onto upstream 0.1.39: the fused int8 prompt kernels (STRATA_PF_FUSED)
never take a folded layer. The multi-GPU prompt paths (set_peer,
set_stage_helper) decline a Hadamard-folded pack here; a later commit
gives their MMQ share the rotations.
gfx1151 (ROCm 7.2.1, STRATA_PREFILL_MMQ=ON): MMQ parity passed 3 runs x 10
repeats (rel_l2 1.4e-3..3.2e-3); native_expert_parity --synthetic and
--synthetic-hadamard tqk6/tqk7 tq2_t/tq2_t: 0 failures, decode-once bitwise
equal. CUDA 12.8 sm_120 full build compiled (incl. TQ MMQ instances) before the
rebase onto 0.1.39.
Ported from agentionai/Strata rc2 dc19786 (rc2 = 9ff02c9), author Laurent Zuijdwijk.
native_expert_layout takes gu_row / d_row / up_off / down_off from
iq_row_bytes, which had no cases for the trellis types and returned 0:
every gate, up and down row read row 0 of the blob, and up/down read the
gate matrix. The dequantizers (flat superblock indexing) were unaffected,
which is why they matched ggml bit for bit while the expert matmul did not.
native_expert_supported now also refuses a type without row bytes.
gfx1151 (ROCm 7.2.1): native_expert_parity --synthetic and
--synthetic-hadamard tqk6/tqk7 tq2_t/tq2_t: 0 failures (gpu rel = cpu rel,
cpu-gpu 5e-5..1e-3).
Ported from agentionai/Strata rc2 dae1f66 (rc2 = 9ff02c9), author Laurent Zuijdwijk.
- third_party/ggml/ggml-common-tq.h: the fork's TQ2_T/TQK6/TQK7 block
structs and 2048-point codebook (device table).
- iq_kernels.cu: Fmt<144..146> (the fork's tq.cuh lane decode against q8_1,
4 q8_1 blocks per 128-block), dq_tq in dq_dispatch (prompt FP16 path,
embedding), is_iq / row bytes; hadamard_rows / hadamard_rows_f16
(shared-memory FWHT in the CPU's butterfly order) and
hadamard_device_signs; native_expert_grouped rotates h when
NativeExpertLayout::had_block is set.
- verify.cpp (local GPU), remote_experts.cpp (helper GPUs), prefill.cpp
(FP16 fallback; folded layers never take MMQ) rotate x / h into their
own buffers; the shared expert and the CPU doorbell copy keep x.
- native_expert_parity --synthetic-hadamard; ctests for tqk6/tqk7 and
tq2_t when ggml is the agention fork.
- generate.cpp: APR packs run only with STRATA_APR=1 until validated end
to end on a GPU.
- hip_backend.cmake: gfx1151 as an unvalidated architecture.
Rebased onto upstream 0.1.39: native_expert_grouped takes the separate
SwiGLU / rotate / q8_1 passes when the down weights are folded (the one-pass
swiglu_q8_1 kernel has no h to rotate); the new decode peer GPU
(peer_experts.cpp) rotates its private input copy and passes the down signs,
as remote_experts.cpp does.
Ported from agentionai/Strata rc2 6c7abde (rc2 = 9ff02c9), author Laurent Zuijdwijk.
- hadamard.{hpp,cpp}: hadamard.txt parser, process-wide spec, in-place
signs + normalized FWHT per block.
- expert_layout_load reads native_experts.txt v5 + hadamard.txt and sets
NativeFmt::{had_block, had_gu, had_d, gu_signs, d_signs}; inconsistent
packs are refused.
- native_quant_act / native_quant_h rotate before quantizing, which covers
every CPU consumer (expert_source's doorbell copy, the pool's h).
- generate.cpp refuses Hadamard packs until the GPU paths rotate too.
- native_hadamard_test: Strata's CPU expert vs the fork's ggml graph
(ggml_mul(signs) + llama_mul_mat_hadamard + ggml_mul_mat): bit-identical
on real Gyro-S (TQK6/TQK7) and Gyro-M (TQ2_T) experts, also through
ExpertPool::run_split_multi_native; synthetic Q8_0 reproduces the
unrotated float expert to 1.2%.
Built with -DSTRATA_GGML_DIR=<agention llama.cpp> (ggml-cpu supplies the
trellis vec_dot kernels).
Ported from agentionai/Strata rc2 1587897 (rc2 = 9ff02c9), author Laurent Zuijdwijk.
- Type tables: PTQ1_0/TQ2_T/TQK6/TQK7 (143-146, 128-value blocks) in the C++
and Python GGUF readers.
- iq_pack.py: validates prism.hadamard.* and writes hadamard.txt (block,
sign vectors, folded roles per layer); native_experts.txt becomes v5 for
such models so an engine that cannot rotate refuses the pack. Anything
it cannot apply (dense folded weights, inverse tables, gate without up,
Q2_0 experts, missing sign widths) is refused. 3 new unit tests.
- embd_bf16_pack.py --gguf: a GGUF's token_embd dequantized to BF16, for
embedding types without a GPU dequantizer (Gyro-S: Q6_K).
Ported from agentionai/Strata rc2 d864b6a (rc2 = 9ff02c9), author Laurent Zuijdwijk.
- runtime start/restart/verify that fail print the run config's engine log path and its last lines (the line was only 'server_exited')
- doctor: on Linux with a SYCL or HIP run config, a /dev/dri/renderD* (and /dev/kfd) the user cannot open is GPU_DEVICE_ACCESS (warning) naming the render group;
seen on P1006 where the engine only said 'No device of requested type available'
P2006 (Radeon 780M, gfx1103) resolved the gfx1200/gfx1201 package as usable: HIP detection looked at the PCI vendor only.
The Linux kfd topology (gfx_target_version) names each card's gfx target; a package whose archs list none of them is skipped with the reason.
Unknown targets (Windows, no kfd) are not a refusal, as with CUDA.
- strata-gate plugin: Manual asks before a change (no blocked attempt first); children of a Manual chat reach the same card
- preset.mjs: the Harness standard preset minus scheduler, search without a key, unanswerable question tool, plan mode
- helper agents projected into the existing agent rows; Stop stops the helpers; the run ends when they have reported
- /goal through the Harness goal Remotes; goal rounds stream into the chat
- native-parallel.mjs: request slots by default in the installed app (fit policy, one-line switch), batch-mtp, one-slot retry
- plain wording for agent failures and tool errors, original text kept as details
- stage-harness.mjs: pruned production Harness for the installer (not run yet)
0.1.41 scaled each stage's miss term by its probed link (0391e478);
0.1.44 scales from a 4x gap against the fastest link (9d41e18b, #2007).
On an x16+x4 dual-3090 rig (26.9 vs 6.4 GB/s, 4.2x) either rule moves
auto off the balanced placement (K=27 -> K=19/K=31): decode -4..-8%,
prefill +15..+19% at 10K / -6..-9% at 90K vs 0.1.40.x at matched
calibration. Opt-in restore next to STRATA_SPLIT_LINK_ABS, default off.
Found qualifying the Terminal archive on real boxes:
- engine calibrate crashed with ModuleNotFoundError (tools.calibrate / setup are not in the archive): refuse with exit 5 naming STRATA_INSTALL_DIR and tune run --mode api; --plan reports tool_available
- models import --plan failed with "fingerprint catalog is unavailable": data/gguf_fingerprints.json is now in the archive
- tune plan/run --engine <package exe> could not load the package's libraries: engine_env puts the package folder first; --engine component:strata-engine accepted
- a boolean candidate (--kv-grow) was passed as "--kv-grow 0" ("unknown argument: 0"), and the protocol harness turned "0" into a bare flag (on): flag_argv
- an unmeasured candidate's reason showed the engine usage dump; the reason line comes first now
ErrorNotice and toasts map known failures (server_exited, engine missing, network, integrity, Harness missing, Control) to a sentence and a next step with the original text under Details. The install progress sums a list of parts (model files, engine) into one count with a time left. The Read pictures block is hidden when the catalog has no pictures. Control reports the last server log lines of a failed start. Reading size scales headings, code, tables and reasoning. All eight locales.
On an MI50 ROCm queues are suspended while the kernel reclaims around the HMM-tracked arena; the cached
reads of the new rule churn the page cache. The gfx906 backend keeps the old room >= read_bytes rule;
other HIP cards keep the current rule (the PR applied it to every HIP build).
Co-authored-by: KevinX8 <[email protected]>
native-control runs Control's tools/strata_mcp.py on Control's Python (installerEnvironment: STRATA_NO_ACQUIRE stays for the
process; only the install job the user starts may fetch model files), in a writable per-user install folder. The catalog,
hardware (nvidia-smi) and recommendation come from it, the install plan and job go through it, and a failed catalog is an
empty one. Calibration stays off in the packaged app. packaged-control tests adjusted and extended.
In the app's mode the install downloads the model the user chose, prepares it with the bundled tools, installs no Python
package, fetches no engine and compiles nothing, and writes a run config naming component:strata-engine into a per-user
folder (STRATA_INSTALL_DIR). Only bundled tools run (run() refuses pip, git, cmake and anything else). The engine
component is looked up if present; a model can be downloaded before an engine is installed.
stage-control.py adds setup.py, the preparation tools (static import closure), ref/load.py and data/ to the staged Control.
GuidedSetup read catalog.recommendation.family without checking that a catalog has a
recommendation; the packaged host's catalog had none and the app showed 'couldn't show this
screen'. GuidedSetup and NativeModels now tolerate no catalog, no recommendation, no families
and families without sizes. A test renders both components with each of those shapes.
The default takes the pipelined decode on two stages only with the draft layer on the first card (mtp_first); asked
for by hand it was taken without a word, and without mtp_first the draft chain shares the last card with its
window (2x R9700: the pipeline lost to the serial loop before 0.1.44). Recommend, never force: it stays on as
asked and a note says what it costs, why mtp_first is not on (no --mtp / --spec 2+ / draft_vocab.bin, or
STRATA_MTP_STAGE0=1 to ask for it) and how to turn it off. No default output changes.
Reported by @dinesh-chander on #2637 (closed by the reporter).
On RTX 4060 Ti 16 GB + RTX 3060 12 GB (48 layers, IQ3_S) the first pick was K=2 and the #1616 retry
(every stage at least 3 layers) landed on its own edge, K=3: the 3060 ran 3 layers, 12 GB idle, 452 tok/s of
prefill against 802 at 28 layers (the prompt flows through the stages as a pipeline). For two GPUs the retry
now first asks for a placement in which each stage owns at least floor(layers x its room / all rooms) layers
(split_rules::share_floors, at least 3, trimmed to fit); the three-layer rule stays the second try and the
first pick the last, and the log says which one was taken. Nothing changes where the first pick has no
one- or two-layer stage (equal cards), and three or more GPUs keep the #1616 rule alone.
Reported by @Tworks25 on #2644, with the sweep of layer_split 12..32 (PR #2643). split_rules_test gets the
reporter's rooms (14.2 GiB and 5.50 GiB) and the identical-card and degenerate cases.
Tauri single-instance plugin: a second launch focuses the open window and starts nothing. On Windows the shell puts itself in a
kill-on-close Job Object, so killing it, a crash, logoff or shutdown ends the host, Control, the owner host, the server and the
engine. The host follows its parent (STRATA_EXIT_WITH_PARENT) and takes SIGHUP and SIGBREAK down the clean shutdown; every engine
it starts is tied to the host with runtime start --die-with-pid (drain 15 s, then ended); it runs runtime cleanup once at start.
The shell waits 60 s for the host's clean stop (the host's own deadline is 45 s) before ending it.
lifeline.popen ties every engine, image encoder, MCP server and sd-cli to the server that started them (kill-on-close job on Windows,
PR_SET_PDEATHSIG from one anchor thread on Linux, a pipe watcher elsewhere). Engines and encoders get their own process group so
Ctrl+C, a closing terminal and a console Break reach the server only, which stops the engine the clean way (STOP, QUIT, wait 20 s,
terminate 5 s, kill). SIGHUP, SIGBREAK and the Windows console close/logoff/shutdown events take Ctrl+C's path; the handlers are
installed before the model loads and the browser launch no longer holds the main thread (#1872). The owner guard escalates SIGTERM
to SIGKILL after a grace period. strata_control/procs.py records every launch and finds leftovers; the next engine start, runtime
start, runtime stop and the new runtime cleanup end them after re-checking identity. runtime start --die-with-pid for app-owned
instances. tests/terminal/lifecycle_matrix.py is the close-path matrix.
Tesla P100 (sm_60) prompt read, interleaved 5 pairs each, thread vs warp scorer: 30000 tokens 339.29 vs 335.45 tok/s (1.011) and 338.98 vs 330.20 (1.027), 60000 tokens 338.88 vs 330.15 (1.026), 5/5 pairs faster each, below the +3% bar for a new default. Same bits either way. docs line; qsa_scores_parity runs with the switch on.
The z440 A4000 Q2_0 gate with STRATA_PF_FUSED=1 on both arms found the residual dump different from base: gu_covered() took Q2_0 on every CUDA card and d.ok on sm_80+ now required the Q2_0 gate/up setup.
Interleaved A/B, 6 pairs, --pcie-frac 0 vs --pcie-frac 0 + the switch, decode ALL: RTX 3060 IQ3_XXS 53.36 vs 53.16 tok/s (0.996, 3/6 pairs faster); RTX A4000 Q2_0 61.72 vs 62.17 (1.007, 4/6). The author's own +2.2% is below our +3% bar and does not reproduce on Ampere. Declined; the PR author's commit stays in the history only as this revert's parent.
"--ple-io mmap" as ONE element of "args" (or "--ple-io=mmap") reached the engine as a single
argument, which it stops on ("unknown argument: --ple-io mmap"); the usual form is two entries.
The server now splits such an element into the flag and its value and says so at startup
(never one in a flag's value place). A config with a JSON mistake (a missing comma between two
args) gave a traceback; it now names the file, line and column. The slow-PLE-probe warning says
how the two entries are written in a config. --ple-io mmap, ram and direct themselves parse
correctly in the engine (checked on the 0.1.45 build).
Reported by lama7 on #2267.
PleTable::gather_batch matched the Direct-mode prefetch slots by position, so rows
prefetched as A..F and gathered as D,E,F,A,B,C (or A gathered as A,A,A) were read a
second time. Each requested token now looks its key up among the prefetch slots and
decodes from the matching slot into its own output position; duplicates share a slot.
All outstanding tickets are drained on a failed collect (the first error is kept)
instead of dropping the rest. The decoded bytes are unchanged.
Reported by @hotschmoe, who also supplied the fixture and the patch; the patch is
used as proposed. ple_reader_test --selftest gets the reorder / duplicate / subset /
missing / over-the-window cases, compared against the mapped reader.
Approval label is Needs your approval on every card; example-card results lose the Demo prefix. A finished agent tool shows Approved only when the person approved it. Other Demo labels (simulated-model toasts, estimates, preview test, time-skip, context panel, tasks, serving) become truthful Example/Illustrative/Simulated wording in every locale.
The page sends the chat's mode with each turn. The bridge switches the session to our preset with the Harness's own /permission command, verifies it through the permissions projection, and fails the turn closed if it cannot. Manual is read-only with ask (a change needs a one-shot approval), Plan is read-only with never (refused, no card), Bypass is workspace-write with never. Tests run the three modes against the real Harness with the scripted worker.
setup.start passes it to the server it spawns (subprocess.call env, or execve for Docker); the runtime host sets it in
the owned server's environment. The test checks the start's child environment.
- engine-install.mjs extracts with tarBinary(): System32\tar.exe on Windows (bsdtar reads .zip and .tar.*), /usr/bin/tar
elsewhere.
- util.mjs which() asks System32\where.exe on Windows (first match), /usr/bin/env which elsewhere.
- the tuning and Time-skip state default to %LOCALAPPDATA%\Strata\tuning and \time-skip on Windows
(apps/host/server/platform-paths.mjs; POSIX keeps $XDG_STATE_HOME/strata). The Python side resolves the same folders
(strata_control/paths.py, used by the runtime directory and the tune and timeskip commands), so the host and the
terminal agree.
- tests: apps/host/tests/platform-paths.test.mjs (both platforms from either host) and tests/terminal/test_state_paths.py.
The terminal archive is a new folder with no run configs. models.resolve_install() now looks at the install folders
setup.py remembered in settings.json (installs, next to data_dir) when the archive's own folder has none, read-only,
and reports how it found them (explicit, here, remembered, default). The run configs name their engine and model
files by absolute path, so the old install and its data folder keep serving: nothing is copied, moved or created.
tests/terminal/test_migration.py builds today's layout in a temp dir (strata-<model>.json, engine/, run scripts,
.venv, a separate Strata-data with GGUF, packs and MTP, settings.json) and checks the listing, inspect, a profile and
serve --plan, a plain start and the explicit setup path leave every file, size, mtime and hash as they were.
A box with no DNS (p620) can build the Linux archive from a folder of wheels fetched and hash-checked elsewhere:
pip still runs --require-hashes against the lock, with --no-index --find-links.
strata runtime start/status/stop/verify/restart now work on Windows. The owner transport (loopback-only TCP, 256-bit
token, state file in a private directory) was already there; the supervisor around it was fcntl/getuid/killpg only.
- strata_control/winproc.py: LockFileEx byte-range leases (shared and exclusive, on a byte beyond the data so the file
stays readable) and process identity (OpenProcess + GetProcessTimes). Liveness never uses os.kill(pid, 0): on
Windows that is TerminateProcess.
- runtime.py: state directory %LOCALAPPDATA%\Strata\runtime with a protected one-entry ACL (existing ones must be
unshared); owner-only checks through private_files.is_unshared; the state file's address must be a loopback port;
the owner host is detached with CREATE_NEW_PROCESS_GROUP|CREATE_NO_WINDOW; the server runs in a kill-on-close job
object (serve/winjob.py, terminate() added), so the whole tree ends with the host however the host ends; a graceful
stop closes the server's stdin pipe (--owner-fd 0, serve/owner_guard.py turns the EOF into the Ctrl+C drain path);
the deadline kill ends the job; the ownership marker records the child's creation time, so a PID that was reused
by another process is not mistaken for the recorded child. A PID from a state file is only compared, never signalled.
- A connect failure to a lost owner is a typed ProfileError (exit 3) instead of a raw OSError.
- `strata serve` and calibration take the activity lease on Windows too.
- tests/terminal/test_runtime.py runs on Windows (32 tests were skipped): a .cmd stands in for the fake engine, the
signal tests have Windows variants; new tests: a second client attaches from another process, wrong/stale tokens
cannot stop the owner, a recorded PID of an unrelated live process is never touched, a reused PID with another
creation time is not the recorded child, the liveness check does not end its target, the directory ACL, shared and
exclusive leases.
The shell now builds and tests on Windows. Three fixes found by running the installed NSIS build:
Tauri's resource_dir() is a verbatim path (\?\C:\...) that node.exe cannot take as its entry script, so the host died at once and the window hung; the path is made plain.
The self-test only knew the macOS Resources layout; it now finds resources next to the exe (Windows) and under lib/Strata (Linux).
A host that exits early is reported at once instead of after the 60 s health wait.
@tauri-apps/cli 2.12.3 is a pinned devDependency with a lockfile (npm ci), and the npm scripts call it instead of cargo tauri.
package_terminal.py builds the terminal archive for windows-x86_64 (.zip), linux-x86_64 and darwin-arm64 (.tar.gz)
from the pinned python-build-standalone CPython 3.12.15 (SHA-256 per target, from the release's SHA256SUMS) and a
per-target lock, packaging/locks/runtime-<target>-cp312.txt, installed with pip --require-hashes --no-deps
--only-binary=:all: at BUILD time only (pip is the interpreter's own, no upgrade; the result must equal the lock).
packaging/lock_wheels.py makes a lock. Console-script launchers that name the build path are removed; a VERSION
file and the read-only gguf_reader tool join the payload.
- bin/strata.cmd and bin/strata.ps1 (Windows), bin/strata (POSIX): the bundled runtime first, PYTHONNOUSERSITE=1,
no PYTHONHOME/PYTHONPATH; a path with spaces works.
- START-HERE.bat, SETUP.bat, UPDATE.bat, setup.sh, update.sh use a python/ folder beside them when present (no
Python lookup or install).
- strata --version (one line: version and interpreter), strata serve --engine mock, a typed exit-5 message when the
server's packages are missing (strata.py preflight).
- packaging/smoke_terminal.py extracts an archive in a clean env (space in the path, PATH without Python, dead proxy)
and runs --version, doctor, profile list, install, serve --engine mock (health, models, a chat answer).
- tests/terminal/test_packaging_pins.py: digests, lock shape and agreement with requirements.txt, launcher order.
setup.py mixed acquisition (pip, downloads, engine fetch and compile) with configuration and start. A start of an
installed model used to check the engine version against the release and download a newer one, and moved model
files into the data folder.
- strata_control/acquisition.py: forbidden() / require() / AcquisitionRefused (exit 5, code acquisition_refused,
names SETUP / UPDATE / `strata install`). STRATA_NO_ACQUIRE=1 forbids a whole process.
- setup.py: start() runs inside forbidden(); download, pip_install, run, every urlopen (now _urlopen), the
engine fetch/compile and CUDA-library installs call require() first. A plain start only says when the engine is
older than expected (engine_update_note) and no longer touches the network; UPDATE / setup keep
update_installed_engine. A plain start reads the data folder from settings.json and creates, moves and copies
nothing (data_folder_in_place); setup, check, calibrate and update still gather files as before.
- strata install [--update] [setup options]: the explicit installer entry; a folder without setup.py says so.
- tests/terminal/test_no_acquisition.py: the start/serve import graph reaches no installer code, no serving module
runs pip/git/cmake/npm or opens a URL (listed exceptions are request-time or loopback), and a real start with the
network and every subprocess mocked makes no request and runs no installer.
Tuning, Time-skip, the revise command, the native MCP client, setup and
calibration jobs and the native server all go through server/python-resolver.mjs;
the Microsoft Store alias is skipped because it must start.
A Windows checkout (autocrlf) has CRLF in LICENSE, and the bundle verify
required the file to start with b'MIT License\n'. Normalize line endings
before the check, pin the file to LF, and test a CRLF member.
14 test files took the path of `new URL(rel, import.meta.url).pathname`, which on
Windows is "/C:/..." and cannot be resolved (esbuild "Could not resolve"). Strip the
leading slash before the drive letter. No behavior change on macOS/Linux. Went from 14
whole-file failures to 6 real test failures (listed in the S0 notes). Easy to drop if the
owner would rather have a different fix.
Follow-up to the owner's rebuild branch strata/rebuild-2026-10 (small, test-only).
IMPLEMENTATION_STATUS.md, ACCEPTANCE-EVIDENCE.md, QUALIFICATION-KIT.md and the
evidence logs (text only, about 140 KB). The 170 MB of screenshots under
apps/host/docs/evidence and checkpoints are not in git; they are in
StrataSplit\evidence-archive. The Electron mentions in IMPLEMENTATION_STATUS.md are
history (Electron was replaced by Tauri); the superseded top-level desktop/ folder is
not carried over.
From the owner's rebuild branch strata/rebuild-2026-10.
packaging/package_terminal.py builds the self-contained terminal archive with its own
Python runtime, monitor_bundle.py the portable monitor, acceptance_matrix.py maps the 67
acceptance rows to evidence, README.md documents the target matrix. The built archive
(release/artifacts/*.tar.gz, 42 MB) is not in git.
From the owner's rebuild branch strata/rebuild-2026-10.
strata_control/tuning.py (the job contract and the HyperCore-Tune decision), tune_engine
(measures through the real API and records memory), tune_history, tune_apply (explicit,
reversible apply and rollback of a recommendation) and tune_api, with tests/tuning
against a stub engine. Review-only: nothing here starts a sweep from the UI, and the
sweep has the known gaps listed in the S0 notes (e.g. --kv-grow 0 / --expert-cache 0
command lines).
From the owner's rebuild branch strata/rebuild-2026-10.
The Time-skip state machine: checkpoint, revise (a bounded Harness turn), check and
decide, with a durable job record, cancellation, and the "inconclusive means ask the
person" rule. The host routes are in apps/host; the CLI entry points are in strata_cli.
From the owner's rebuild branch strata/rebuild-2026-10.
A Harness profile for Strata, a bridge that drives a Harness agent session against a
Strata server (_bridge.py, bridge_e2e.py, bridge_tests.py), the tool-execution /
approval / cancel end-to-end test (tool_flow_e2e.py) and the bounded revision turn the
Time-skip job uses (revise_once.py). Tested only against a scripted mock worker; the
Harness is not wired into the app's chat yet.
From the owner's rebuild branch strata/rebuild-2026-10.
The Tauri shell (src-tauri Rust, 400-odd lines) that starts the Node App Host and
opens the owner's UI, with the macOS entitlements, icons and generated schemas, and
the build scripts in apps/desktop/scripts that package the app around a private Node
sidecar (digest-verified fetch). The Node binary, Rust target/ and DMGs are not in git;
src-tauri/resources/node/ keeps only its .gitignore placeholder.
Windows and Linux packaging is not working yet (the sidecar lookup expects "node" and the
scripts are bash-only); see the S0 notes.
From the owner's rebuild branch strata/rebuild-2026-10.
The UI source from the 14 October handoff (src/, public/, index.html, vite and
tsconfig, tests, audit-fixes, design docs) and the App Host (server/*.mjs): model
import and monitor routes, API-provider catalog and streaming (OpenAI chat, OpenAI Responses,
the Messages API), serving, tunnel, tasks, plus the host routes for Time-skip
(server/timeskip.mjs) and the tuning plan/reports. src/lib/time-skip.ts renders a real
job's own record.
The UI design is unchanged. The rebuild routed #/tuning to a new TunePanel; that is
reverted: src/screens/ModelRun.tsx and src/main.tsx are the handoff's again (#/tuning
goes where the handoff sends it), and src/components/TunePanel.tsx and
src/styles/tuning.css, which only that route used, are not carried over. Left out of
git on purpose: dist/ (build output), node_modules, and docs/evidence + docs/checkpoints
(170 MB of screenshots, kept in StrataSplit\evidence-archive).
From the owner's rebuild branch strata/rebuild-2026-10.
requirements.txt keeps the runtime packages only; requirements-build.txt pins
cmake and ninja. setup.cmake_build installs only the helper that is missing, right
before a local engine or image-encoder build, so the ready-made engine path (and
serving) never runs pip for build tools. ram_gb() reads hw.memsize on macOS and no
longer crashes without /proc/meminfo. docs/INSTALL.md notes the split; two tests in
tools/test_setup_pins.py cover it.
From the owner's rebuild branch strata/rebuild-2026-10. setup.py merged onto 0.1.45
without conflicts (0.1.45's MIN_ENGINE and pins are unchanged).
New serve/ready.py, serve/owner_guard.py and serve/launch_identity.py; server.py
gains the queue wait deadline (QueueWaitExpired/QueueCancelled), the opt-in
STRATA_BATCH_TELEMETRY plumbing (BSTAT/PSTAT parsed off the engine's token stream,
dropped from request output), the engine stage-plan reader, a main(argv, ...)
entry the terminal can call, and per-replica selection hooks; telemetry.py reports
GPU names; the tests for all of it.
server.py merged by hand onto 0.1.45: one conflict, in StrataEngine._take_control, where
0.1.45's STRATA_ADMIT_SHORT_FIRST ordering and the rebuild's queue deadline both edit the
acquire step; both are kept (ordering check first, then the deadline-bounded acquire).
Every line 0.1.45 added to server.py (the #2518 pick_slot fix, the havok multi-user
switches, the Qwen3.6 parallel/vision work) is still present.
From the owner's rebuild branch strata/rebuild-2026-10.
include/strata/core/batch_telemetry.hpp (BatchWindowSampler, StageInFlightSampler),
its unit test, and the hooks in the generate.cpp batch loops and the --batch-groups
pipelined loop. Off unless STRATA_BATCH_TELEMETRY is exactly "1": no new output, no
new state beyond two disabled samplers, so the default output is unchanged.
Host-side C++ only (no CUDA/HIP/SYCL code), so it builds on every backend.
From the owner's rebuild branch strata/rebuild-2026-10. generate.cpp and
CMakeLists.txt merged onto 0.1.45 without conflicts.
The command-line and TUI client with profiles, doctor, monitor, serve, calibrate,
bench, the runtime owner, component registry and offline policy, plus tests/terminal,
the component/offline tests, the smoke tools, the terminal CI workflows and the
terminal docs. Tuning and Time-skip modules of strata_control follow in their own
commits (strata_cli/cli.py imports them lazily).
From the owner's rebuild branch strata/rebuild-2026-10 (1e4171e2, based on 296b59d2).
README.md and tools/calibrate.py applied as small 3-way patches.
Keep node_modules, Tauri target/, DMGs, the Node sidecar, release archives, the
built App Host dist/ and the Mac evidence screenshots out of git. The two
.terminal-*venv entries are from the owner's rebuild branch strata/rebuild-2026-10;
the rest are new for the clean integration branch.
The SYCL build of the sm_75 series failed with 'out-of-line definition of bf16 does not match any declaration'
(sycl/src/prefill/gemm.dp.cpp); the four gr_* writers would have failed at link. Gemm::bf16_converts_x() answers false.
Ported from c3c's #1581 (the native-IQ and VRAM-loan parts are not in this commit). Default unchanged: Turing keeps returning false for q4_0.
Co-authored-by: c3c <[email protected]>
The v1 kernel keeps its exact previous code (and 37.9 KB of shared memory) unless the flag is set. launch_i8 picks the Turing
single-stage layout by the (emulated) compute capability, so the kernel and the launch agree under STRATA_EMULATE_CC.
No sm_75 card is available here to confirm the series, so the kernels that were on by default for compute capability 7.5
in the submitted PRs now follow one switch, include/strata/core/sm75_series.hpp. Each keeps its own variable, which
wins in both directions. The one-read prompt top-k (#2093) was on for every CUDA card as submitted; it is now Turing only
like the rest. STRATA_EMULATE_CC=75 counts as a Turing card, so a 75-virtual build on an sm_86 GPU runs the series for tests.
block_scores_thread_kernel gave a lane one (query, head) and the whole
128-value query head; ncu on a CMP 50HX showed it bound by the MIO pipe
(mio_throttle first): one shared-memory key load per 4 FMAs. Lane 0's
xor-shuffle tree splits into four independent quarters - c[j] sums groups j,
j+16, j+8, j+24, j+4, j+20, j+12, j+28 - which only meet at the end
(c0 + c2, c1 + c3, then their sum). Now four lanes share a query, lane j
holds quarter j of all four heads, a key load feeds four heads (16 FMAs),
and two xor shuffles finish the same tree. The same pairs in the same order,
so the same bits.
qsa_scores_parity on both cards: all 141 M scores identical to
block_scores_kernel; per call 1,024 queries at 140K 7.76 -> 6.17 ms (-20 %),
at 200K 10.78 -> 8.54 ms (-21 %), 2,048 queries at 40K 12.60 -> 11.57 ms.
agentbench 34K -> 190K: AGENT GATE PASS on the rerun (the first run diverged
at turn 2 - decode's known run-to-run variation; the rerun of the same binary
matched all 36 turns), incremental read -0.6 % median, about -1 % from 150K
up, decode unchanged.
On cards without the TF32 scorer (sm_75) the prompt path scored every
(query, key block) with a whole warp: 20 shuffles and five 512-byte loads per
score. block_scores_thread_kernel gives each lane one (query, head), keeps the
128-float query head in registers, reads each key once from shared memory as
a broadcast, and adds the 32 four-value partials in the tree lane 0 of the
xor-shuffle sum computes. Float addition is commutative, so the same pairs in
the same tree give the same bits; the heads are summed 0..3 from 0.0f as
before; the tail block keeps block_scores_tail_kernel. STRATA_SCORES_THREAD=0
selects the old kernel.
Measured on 2x CMP 50HX (TU102, sm_75, unlocked to RTX 2080 clocks):
- qsa_scores_parity (new): 141 M scores bitwise identical on both cards
(normal values, wide magnitudes with zeros, small integers with ties);
per call 4.6-5.2x faster, e.g. 1,024 queries at 200K: 52.4 -> 10.8 ms,
at 140K: 35.9 -> 7.8 ms. 199 registers, no spills.
- nsys, one +4.4K-token turn at 140K: block_scores_kernel was the top prompt
kernel at 1,045 ms per card (17 %).
- agentbench, IQ3_XXS, layer split, 262144 context, 34K -> 190K (with the
tokenizer cache): engine read per turn at 190K 9.25 -> 6.96 s (476 -> 632
tok/s, +33 %), time to first token 10.49 -> 7.13 s (-32 %); median
incremental read 576 -> 682 tok/s; decode unchanged (73.3 tok/s).
AGENT GATE: all 36 turns word-for-word identical to the base run.
The Turing (v1) prompt attention kernel used 37.9 KB of shared memory per
128-thread block, so one block ran per SM (4 warps, 12.5 % of the warps an
SM can hold), and each 32-cell chunk gathered its scattered K and V rows
after the previous chunk had finished: ncu on a CMP 50HX showed
long_scoreboard (global load latency) as the top stall. For int8 K/V
(--kv int8, mode 1):
- the next chunk's K/V rows and scales are loaded into registers while the
current chunk computes (Turing has no cp.async), then stored where the
gather stored them;
- K is read only by the scores and V only by p.v, so they share one buffer:
V is written over K after the scores' barrier (the softmax's barrier
orders it before p.v). 37.9 -> 29.2 KB: two blocks per SM.
The same bytes in the same places and the same arithmetic, so the same bits.
Other KV modes keep their code. qsa_prompt_attn_parity now prints a checksum
of the new kernel's output, so two builds compare bitwise.
Measured on a CMP 50HX (sm_75), qsa_prompt_attn_parity, 2,048 queries:
int8 21.45 -> 12.91 ms at 40K (-40 %), 21.94 -> 13.31 ms at 140K (-39 %);
all output checksums (int8 and fp16, 4K/40K/140K) identical to the previous
kernel. agentbench 34K -> 190K (IQ3_XXS, layer split, 2x CMP 50HX): AGENT
GATE PASS (36 turns identical), incremental prompt read time -3.5 % (median
per-turn ratio 0.965), decode unchanged.
With --pcie-frac 0 the pool never plans PCIe experts, yet every layer of a
verify window still runs the flag B wait, the staging copy and the grouped
launch over zero groups: six kernels, ~18 us a layer on an RTX 2080 Ti
(kernel-node nsys). STRATA_NO_PCIE_GROUP=1 records the window graph without
them; a plan with PCIe experts then stops the engine with a message.
Engine A/B on an RTX 2080 Ti (Flash-Next Q2_0, 512 tokens, two runs each, with the open sm_75
decode PRs applied): Spanish chat 87.8 -> 89.8 t/s, code 106.1 -> 108.5,
32K 78.4 -> 80.1, 128K 75.0 -> 76.8, 390K 74.7 -> 75.1.
Two kernel pairs of a verify window's layer run as one launch on Turing,
every byte as before:
- the plan's host-flag wait folded into its copy (143-amdk's
wait_copy_i32_from_mapped, gfx12-only until now). STRATA_FUSE_WAIT on
CUDA: 2 = the plan only (Turing default), 1 = also the CPU rows'
copy_rows_from_mapped_after, which makes every row block poll the flag
over PCIe and was -1.5 % on an RTX 2080 Ti, 0 = neither.
- native_quantize_q8_1_il: a projection input's q8_1 blocks and their
interleaved copy (native_q8_1_interleave) in one kernel; an int of the
copy is four codes gathered with two shuffles. Used for the GDN qkv and
out-projection inputs and the QSA input and out-projection. STRATA_QIL=0 / 1
forces it off / on (Turing default on).
RTX 2080 Ti, Flash-Next Q2_0, 512 tokens, two runs each: the plan fold
+0.6..+1.4 % decode, the fused quantize another +0.5..+1 % (code 108.0 ->
109.1 t/s). The fused quantize writes the same bytes as quantize +
interleave at 2560/6144/10240 inputs and 2-4 columns.
The exact layout gives each Q2_0 row four warps, a half block per thread and a
cross-warp reduction. At the engine's 2560 inputs (40 blocks a row) 48 of the
128 threads have no chunk and the reduction dominates: the QSA attn_q
projection (2560x12288) at three columns took 80 us (~110 GB/s) on an
RTX 2080 Ti.
q2_0_warp_rows_kernel: lane l runs the chunks of the exact kernel's threads
l, 32+l, 64+l and 96+l in their loop order, adds the four sums as warp 0 adds
the other warps' partials, then the same warp_sum - every output is bit for
bit native_q2_0_mmvq_kernel's (one column) and launch_multi's exact layout
(two to four). 1/2/3/4 columns: 36.6/60/68/88 -> 24.8/31/31/37 us.
Default on sm_75 only; STRATA_Q2_WARP=0 keeps the old kernels, =1 takes this
one on any CUDA card.
attn_chunk_kernel<1> saturates the SM's shared-memory pipe on Turing: its
time grew linearly with the queries while the SMs were half empty (two
16-byte q reads per head per cell with a two-way bank conflict, and 12
warp_sums = 60 shuffles per cell).
attn_chunk_q8_v3_kernel computes the same operations in the same order:
- the lane's slice of the 12 query heads is read into registers once;
- the 12 lane sums are reduced with reduce12_gfx906 (16 shuffles, every
sum warp_sum's pairs);
- the probabilities are stored cell-major, so the value loop reads a
cell's 12 as three 16-byte broadcasts.
RTX 2080 Ti, with the merge: 2051 selected cells, 1-4 queries
51.6/78.5/110/146 -> 34.5/41.0/73.0/84.9 us; 600 cells
45.9/74.6/105/137 -> 28.6/32.8/33.5/33.3 us. qsa_prompt_attn_parity
reports the same errors with STRATA_ATTN_V3=0 and 1.
Default on sm_75 only; STRATA_ATTN_V3=0 keeps the default kernel, =1 takes
this one on any CUDA card.
The fused prompt-expert kernels (moe_fused.cu for the Strata Q2_0 pack,
moe_fused_iq.cu native_kernel for the native packs) on sm_75, with
STRATA_PF_FUSED=1 STRATA_PF_FUSED_SM75=1:
- cp.async -> plain 16-byte copies (the barriers around them stay),
prefetch.global.L2 kept;
- mma m16n8k32 / m16n8k16 s8 -> four / two m8n8k16 (exact int32 sums,
the same d as the sm_80 instructions), MAGIC as the C operand;
- two activation stages on Turing (64 KB of shared memory);
- native_kernel takes Q2_0 gate/up (64-value blocks, a block per stage),
so the canonical Q2_0 pack (Q2_0 / Q2_0 layers) runs fused; on Turing
only that pair is set up;
- Turing, Q2_0: the two halves of a stage share the weight scale, so a
row's dot is d (dx_0 dot_0 + dx_1 dot_1) - 11 % fewer instructions.
RTX 2080 Ti, one layer, 512 experts top 10 (prefill_fused_iq_test,
Q2_0 / Q2_0 pair added): 2048 tokens MMQ 12.98 ms -> fused 6.59 ms,
16384 tokens 55.08 -> 37.16 ms; fused vs MMQ rel RMS 2e-5, the same
error against the FP32 reference. In the engine (canonical Q2_0 pack,
98K prompt, 16K chunks): prefill 58.3 -> 53.3 s; layer 0 expert rows vs
MMQ: median rel 1.6e-7. Teacher-forced 2048 tokens at P = 16K / 32K /
64K: perplexity vs MMQ +3.4 % / -4.5 % / +0.6 % (FP16 path vs MMQ
+0.7 % / +0.1 % / +0.3 %) - trajectory noise, no systematic shift; the
verify phase (same tokens) 42.36 s MMQ, 42.28 s fused.
A borrow copies the donor's checkpoints up to the match, so a match that reached a pinned shared prefix gave the borrower the pin too. Such a match now takes the conversation as before. The borrow's guard is guard()/unguard()/guard_blocked(), and the refusal before evicting counts the pinned conversations along with the donor.
Rebased by hand onto 0.1.44.1 (evict_oldest() honours the guarded donor).
Co-authored-by: BlueKingMuch <[email protected]>
Sibling subagents with the same system prompt and tools no longer cost each other their history: only the K/V up to the checkpoint and the checkpoint itself are copied into the session (conversation_snapshot_restore_prefix); the parked conversation stays where it is and is pinned while the outgoing one parks. A match on a parked conversation's live state or newest checkpoint is that conversation going on and is taken as before.
Rebased onto 0.1.44.1 by hand: the pinned shared prefix (#879) and evict_oldest() now also skip the borrowed donor, serials follow every erase, and park-gain (#2265) and the pin= sibling rule gate the outgoing park.
Co-authored-by: BlueKingMuch <[email protected]>
A client renders the history again for its next turn, without the reply the slot decoded or the assistant header,
so the slot's held stream (prompt + reply) is no start of the new prompt and pick_slot fell back to the least
recently used slot. With two chats in two slots and two next turns arriving together, whichever was read first
could take the other chat's slot, and the engine's slot copy then overwrote the cache the second turn needed
(0 tokens reused, a full prompt read). pick_slot now also counts the held tokens up to the last <|im_start|>
(where the engine takes its turn checkpoint) when the prompt agrees with them, and a pipelined layer split with one
slot per group (--batch 2 on two cards) no longer ignores what its slots hold.
Least recent use keeps the newest checkpoints and every turn adds one, so after a few short turns all 6 sit in the
last 1-2K tokens. An agent client that rewrites its history a few thousand tokens back then resumes from the system
prompt: on 2x CMP 50HX with the pi coding agent a 210,528-token prompt that was the same as the session for 204,349
tokens resumed at 9,525 and read 201,003 again (100 s).
With the flag the victim is the checkpoint whose removal leaves the smallest gap for its distance from the newest, so
the kept points thin out geometrically: after 200 turns of 1,000 tokens they sit 0, 1K, 3K, 8K, 43K and 200K before
the end, and an edit 6K back reads 8K again (least recent use: 200K), 42K back 43K. The same number of checkpoints, no
more RAM; never the root, the newest or a pinned one, the tail rule first, the plain policy without a candidate. The
resume is the usual exact-prefix match.
On the box: an 82,593-token conversation grown turn by turn, a turn ~6 steps back rewritten, resumed at 73,610 tokens
in 10.2 s against 40.0 s read in full; with the expert placement fixed (--adapt-swaps 0, --pcie-frac 0.17) the
greedy answer was the same text as the full read. conv_cache_test: the spread cases (a 200-turn simulation against
least recent use, the fallbacks). docs/DETAILS.md: a paragraph after the prompt cache's options.
A conversation that only advanced a few hundred tokens since its last park does not need a fresh 3 GB copy: the entry it would supersede still holds this conversation's prefix, and the next turn drops the copy anyway (drop_superseded). Reported in #2264 over 1,418 park events on one long single conversation.
The flag is opt-in and defaults to 0, which parks every turn exactly as before. What this adds is a way to ask for the gate without a rebuild: on the reporter's rig it is 3.26 GB instead of 9.79 GB per benchmark, the superseded class disappears, and the decode median moves -3.4 %, inside that arm's 92.2-124.1 tok/s spread.
parked_prefix_tokens() reports exactly the entries drop_superseded() would remove - it finds them the same way, by the deepest checkpoint the outgoing chain still holds - so the gate cannot name a different entry than the park would drop. conversation_cache_test covers it (7 checks; 4,226 pass).
Fixes#2264
A conversation that was restored and is parked again keeps its unchanged K/V
pages and appends the new ones (92a3d5f). The appended segment reserved room
for later turns, growing geometrically up to 16 MiB - per buffer, and a parked
image has one per K/V layer and part (k, v, their scales, the indexer rows, the
draft layer). With Qwen3.8-Flash-Next (IQ3_S, int8 K/V) a 32K-token main agent
parked again took 1.49 GB instead of 0.97 GB, a 17K-token subagent 1.16 GB
instead of 0.74 GB. The cache counts that capacity against its budget, so with
2816 MiB the reserve decides which conversations are evicted.
The reserve is now at most an eighth of what the buffer held before the append
(at least 64 KiB, and at most 16 MiB as before). Small turns still fill the
reserve instead of adding a transfer each: the 4096-append test keeps its
segment bound, and 256 turns of 1 MiB onto a 16 MiB buffer take 28 segments
(17 before).
The docs said parking falls back to a full capture when the reserve would evict
another conversation; that check went in 6cd29d9. With the reserve bounded the
two captures differ by at most an eighth of the K/V, and the docs now say what
the reserve is.
A standing ban of a whole script (65,929 IDs) was normalized five times per request: in _openai, twice in
validate_request and twice in engine_key. About 200 ms of Python per request, whatever the answer length.
A list is now normalized once. The same object again in a request is answered from the last call; an equal list in
a later request is found by its compact JSON text, which tells False from 0 and "1" from 1 (== does not: [[7, False]]
== [[7, 0]]). Only a list that normalized without an error is stored. Lists under 64 entries skip the cache.
No change to what normalize, engine_key and validate_request return or raise.
PromptEncoder keeps the last prompts with their ids and resume points (one per
conversation-cache slot) and encodes only the text after the longest shared
special-token boundary; the ids equal a full encode. Rebased onto 0.1.44.1
(prepare() keeps the new `cancel` argument; forget_prompt on a context refusal;
count_tokens reads without storing). SECURITY.md notes the in-memory prompts.
Based on gputier's #1852 (8 commits) and blange48's #567.
Co-authored-by: Guillaume PUTIER <[email protected]>
Co-authored-by: benoit lange <[email protected]>
Bit-identical (gate: 10/10 + exact_pp 4K/20K IDENTICAL with STRATA_DEQUANT_WIDE=1 on the RTX 3060 and the RTX A4000), and the time
to the first token of a 150 / 300 / 450-token prompt falls 12-20% on the RTX 3060 (5 of 5 pairs each), the prompt read +2.3%.
STRATA_DEQUANT_WIDE=0 turns it off. Arch-gated to the measured sm_86 and sm_61. Follow-up to #2178 (Techannel), the kernels this switches on;
the same stores are what #1525 (sergqwer) proposes for Q8_0 / IQ4_XS.
A build folder configured while the option's default was OFF keeps
STRATA_Q6K_EXPERTS=OFF in its CMakeCache.txt even after the default
flipped to ON (465cd61d): CMake does not rewrite an option already in
the cache, and source_hash() fingerprints the source files but not the
CMake option values, so setup recompiles the engine with the Q6_K
gate/up expert kernels still disabled. A model whose routed experts are
Q4_K/Q6_K (Unsloth UD-Q4_K_XL) then fails to load with:
strata-q36: blk.34.experts are Q4_K/Q6_K, which the native expert
kernels do not take (build with -DSTRATA_Q6K_EXPERTS=ON for Q6_K)
Passing -DSTRATA_Q6K_EXPERTS=ON explicitly overrides the stale cache
entry, so the engine always has the kernels its default build promises.
Same pin in the Dockerfile build for consistency. Not hardware-specific:
it is a CMake cache-staleness issue, independent of GPU/platform.
The output sum of token t is done after token t + 1's first barrier (sk / sq double-buffered), off the recurrence's chain. Same arithmetic: the
3000-token prompt's logits and 8 greedy steps are bit-identical to the base engine at chunk 512 and 2048 (R9700: 913 -> 885 ms, 607 -> 573 ms).
With many slots on a small card (8 sessions on a 12 GB RTX 3060, UD-Q6_K_XL) the sessions left too little VRAM for the hybrid
mode's staging pair, CPU-output buffer and expert slots, and the start died in a bare cudaMalloc failure. place_experts now adds
up what it is about to allocate and exits 1 with "not enough VRAM left for the hybrid experts' buffers and slots (N MiB needed,
M MiB free ...)" and the way out (fewer slots, a shorter --max-context, fewer --expert-slots). Only a start that would have
crashed changes; the placement and every answer are as before.
tools/qwen35/q36_vram_refusal.py: demanding starts (8 slots, long contexts, every expert as a slot) must reach READY or exit 1
with one of the engine's refusals, never a raw allocation failure.
--pipeline-windows runs its own overlapped decode loop with its own hand-offs, so
STRATA_SPLIT_OVERLAP now switches itself off when it is set (one log line says so).
Rebased onto main 1735d64: the early launch of the next stage keeps #871's graph
variant (the next stage's refresh_ar() runs before its capture and prelaunch, and
not again in its run() when it was prelaunched), resets the plan error word like
run()'s launch, and the chain path checks that word after its sync as well.
Review on #936 (justxiami): the reading stage's wait_flag_ge spun with no bound,
so a lost publish would leave that stream spinning.
- wait_handoff: the reading stage's graph now waits with a bound (default 30 s,
STRATA_SPLIT_WAIT_MS, 0 = unbounded, GPU clock as in gpu_stamp). On timeout it
sets a mapped error word and returns, so the graph completes; the host checks
the word after the window and fails it with an error that names the setting,
instead of using a stale hand-off.
- The host's per-layer wait (20 s, #267) still normally ends a stalled window
first; release_gpu_waits already raises the hand-in flag.
- Test hook STRATA_TEST_HANDOFF_DROP=N: the first stage's N-th window withholds
its flag (a lost publish), to exercise both paths.
Only active with STRATA_SPLIT_OVERLAP=1; the default path is unchanged.
Each verify window of a layer split runs its stages strictly in turn: stage 1 finishes, the host syncs its stream,
then stages and launches stage 2's graph. Under HIP the sync and the graph launch both sit on the critical path of
every window.
With STRATA_SPLIT_OVERLAP=1 the hand-off carries a ready word (mapped, one per hand-off buffer):
- the writing stage's graph ends with handoff_publish: the payload with volatile stores (RDNA keeps plain stores to
mapped memory in L2), a system fence per thread, and the last block raises the flag;
- the reading stage's graph starts with wait_flag_ge(flag, 1);
- Verifier::run captures the next stage's window on this thread, then stages and launches it on a helper thread
(prelaunch) while this thread serves its own layers; it then serves the next stage's layers directly and syncs
the earlier stage once the chain is done;
- release_gpu_waits also raises the hand-in flag; an early return after the prelaunch releases the next stage.
Batch windows and --split-device 0 keep the serial order. Off by default.
Measured on 2x RX 7900 XTX (ROCm, Linux), Swift 1.5 IQ3_XXS, 512-token greedy answers, median of 8:
decode 53 -> 70.1 tok/s (code prompt), 45 -> 55.9 tok/s (prose prompt); greedy output byte-identical with the
flag at 0 and 1; quiz 47/51 both ways. Not measured on NVIDIA.
Split-overlap launch merged with 0.1.44 HostGraphLaunch (STRATA_VERIFY_ASYNC_LAUNCH): the launcher object is
kept in an optional so an early-launched window has none.
Co-authored-by: bsvinay <[email protected]>
Squash of SuperDistance's three commits of PR #2050 (decode between the pieces of a read on the pipelined groups, the
piece lines counting the rows the slots decoded meanwhile, the stall watchdog staying on for a read in pieces),
applied after #2326 here: the BYIELD give-way is #2326's give_way lambda, the concurrent block runs first and the
piece path serves a read that STRATA_PIPED_CONCURRENT leaves to it (pictures, configurations where it is off).
Co-authored-by: SuperDistance <[email protected]>
On Linux a thread inherits its creator's CPU mask, and the host thread is pinned to the one core the pool leaves it
(aux_cpus.hpp). The read's thread and the doorbell thread were created from it, so they shared that core with the host
thread that pumps the windows, and every window waited there for the doorbell of each layer: 6.1 rows/s during a read
on 4x RTX 5080, 4 stages (blange48 on #2326). Windows places the host with CPU Sets, which new threads do not inherit,
so it did not show there. Both threads now move off the host's core and the pool workers' CPUs (on an SMT machine with
the default pool: the idle sibling threads; without such CPUs, every CPU but the host's), and the threads that already
run on the host's CPU (the prompt path's copy threads, started by the first prompt) move the same way at a read.
--aux-cpus places them all when it is on. STRATA_PIPED_CONCURRENT_UNPIN=0 keeps the inherited CPUs (an A/B).
Measured on Linux (WSL2, Ubuntu 24.04, 2x RTX 5090, 4 streams, a cold 32K prompt; same binary, UNPIN=0 / default): the
slots decoded 25.9 and 23.7 rows/s during the read with the threads kept on the host's CPU (76-81 threads there), and
75.6 and 62.8 with this change; with the n-gram table in RAM (WSL's disk reads it 7x slower than Windows) 110.7 rows/s,
as on Windows (107.0). piped_read_join_bench.py (cold32k, followup, two_joiners; table in RAM, 2-3 runs each): 24-37
rows/s during the 30K+ reads kept on the host's CPU, 95-127 with this change, and 925-1,065 stream tokens during the
cold 32K join against 454-484. On Windows, STRATA_TEST_CONC_ON_HOST_CPU=1 (a test) puts both threads on the host
thread's CPU set first: 18.6 rows/s kept there, 105.4 with this change.
The flag works only through a helper cache (--expert-cache-device1..3), and on a plain layer split it is inert
(MULTI_GPU.md, #1447), so the check looks at the helper caches themselves. #2275's rig runs a plain split with the
flag set, and the mode would have stayed off there for a cache it does not have.
The opt-in's section: how the read and the windows share the GPUs, the doorbell thread, the give-way at a chunk, where
the mode is off. Measured against main (a0116b27) on 2x RTX 5090 in one window (main, this, this, main): the stops,
the streams' rate during the reads with the same answers every run, the joining prompts' TTFT, a question during a
read, the same randomised load on both, the doorbell thread's first wait as an A/B; 16 GB cards on each side; a weaker
host as well in #1906's shape; the default path's check; text equality.
tools/piped_read_join_bench.py, the client that measured it, is #2050's, unchanged (the same file as there).
From a review of the concurrent read:
- With external serving batch_poll still finds a window that will not finish (its graph ended or failed with a layer
to ring, or no layer rang for 20 s): the served count is read before the stream query and the ring count after it,
at most one query a millisecond; the last ring's time is an atomic. batch_serve reads a window's row count after its
served count, so a window launched while the doorbell thread was descheduled is not served with the last one's rows.
- #1996's pinned upload rings (0.1.43.1) grew a slot at its first larger upload, a pinned free and allocation on the
read's thread: the drafter's records and the prompt path's routing tables reserve their slots before the read
(PinnedRing::reserve, Prefill::reserve_uploads).
- Off, said once, where serving a doorbell makes a CUDA call (--pcie-mode dma, STRATA_FS_SLOTS, the helper caches) or
the expert source recycles the buffers it hands out (the GGUF in place, unbuffered, STRATA_IO_PREFETCH); a prompt
with pictures is read the usual way. The expert sources' read counters are atomics.
The check is a pure function, concurrent_read_off (batch_read.hpp), with a test in batch_read_test.
- pump's all-idle branch turned the stall watchdog off when the last slot finished mid-read; it stays on while a prompt
is read until the request ends.
- Between windows the doorbell thread slept 50 us, a whole timer tick on Windows (up to 15.6 ms): it now waits on a
condition each launch notifies (STRATA_TEST_BELL_SLEEP_US=N, a test, restores the sleep for an A/B). With nothing
decoding the main thread waits for the read's end or a line instead of sleeping. A pool failure the doorbell thread
set is reported before pump clears the flag for the next window.
- The read's log line ends with the slots' rate and, per stage, its windows and the host time that set each up. The
priority stream's cudaGetLastError runs only under the opt-in.
With a layer split and --batch-groups > 1 a prompt read is one sp.run, so every decoding slot stops for the whole read
(#1906); #2050's pieces bound the stop but drain the pipeline at each piece. With STRATA_PIPED_CONCURRENT=1 the read
runs on a worker thread while the main thread keeps the groups going through the pipeline (pump) and takes BSTOP and
BYIELD meanwhile. The prompt's K/V goes to the main sessions and the slots' to their own; the windows' streams get the
device's highest priority on CUDA (STRATA_PIPED_CONCURRENT_PRIO=0 keeps the default).
- The slots' doorbells are served on a thread of their own that makes no CUDA call (Verifier::batch_serve, now
batch_poll's per-layer service too, and set_external_serve). Two hangs found while building this, read from the
stall dumps against the PDB, had #31's shape: the main thread waited for the driver's lock in cudaSetDevice while its
holder waited for the GPU (a cudaFree in the drafter's prompt pass; cuBLASLt's lazy load of a GEMM module), and a
window on that GPU waited for its doorbell.
- MtpDrafter::reserve_prefill captures the drafter's prompt graphs and sizes its records before the read.
- Prefill::stop_at_chunk: a BYIELD ends the run at its next chunk, also with no slot decoding; the pipeline is drained
there and the part read gives way as #656 does. A read nobody waits for is not cut.
- The reader is joined on every way out (an exception too). Off beside STRATA_PREFILL_HELP and STRATA_SPLIT_MTP_BATCH,
which allocate during a read.
The prompt path gathers a chunk's PLE rows on its own thread and the batch windows gather theirs on the main thread,
from the same table (ss.ple.table). Today they never overlap - a prompt run's read-ahead ends with the run, and the
windows run between runs - but a prompt read beside running windows would race on the reader's tickets and the
prefetch slots. Every public method that touches them now holds a recursive mutex (gather holds it across its issue
and collect). Uncontended it is a few nanoseconds per call; nothing else changes.
Squash of ruibeikaa's six commits of PR #1441 (the first fixed b/a, then runtime-learned b/a, then calibrated b/a, the
segment planned once, --prefill-pipe 1 following STRATA_PREFILL_EQUAL), applied onto 0.1.44.1 with the conflicts in
generate.cpp (Options, BSlot, the lend's chunk with --prefill-head-alias) resolved.
Co-authored-by: ruibeikaa <[email protected]>
A bigger chunk is only taken when the experts, the 0.77 GiB reserve and the chunk's buffers (0.5 MiB per token) still fit without giving the packed
decode copies back: 131072 cells -> 512, 100000 -> 2048, 32768 with --batch 4 -> 1024, --batch 8 -> 512 (hybrid as before).
Prompt chunk (owner decision): with every expert in VRAM the auto rule picks 4096, 2048 or 1024 (the chunk's buffers cost ~0.5 MiB per token,
the sessions' caches 45 KB per cell and session; 1024 at most with --batch N so the other slots' windows are not held for a whole chunk), else 512.
The hybrid mode keeps 2048 and falls back to 1024 / 512 on a card with little room. STRATA_Q36_CHUNK=512 or --prefill 512 gives the old size.
R9700 pp4096: 512 3390, 1024 4495, 2048 5353, 4096 5750 tok/s. Not bit-identical to 512 (other GEMM tiles): see docs/QWEN36.md for the KL.
Decode batches of up to 8 rows take the router and the shared expert's gate in one product (rows of 257), as a single token already did.
k_conv_seq's tile mode switched off the history write for batch rows too (grid y is the group count there): fixed.
k_gdn_seq_cols started every token with a chain of global loads; they are issued one token ahead now. Same arithmetic: the 3000-token
prompt's logits and the next 8 greedy steps are bit-identical to the previous engine at chunk 512 and 2048 (R9700: prompt 931 -> 902 ms
at chunk 512 together with the tiled conv, 616 -> 591 ms at 2048).
q36_kernels.cuh: one warp per row for the shared expert's and attention k/v Q8_0 products (grid capped at 1024 warps: 2048 rows as 2048
warps took 21 us, 5 us as 1024), shared gate+up+SwiGLU+q8_1 in one launch, k_moe_out with 1024 threads and the next layer's q8_1 image, k_add_rms
with the q8_1 image, router top-k and expert grouping in one launch. CUDA: the division / exp the --use_fast_math TUs they mirror use are spelled out.
Engine: prompt GDN conv in tiles of 16 tokens side by side (same arithmetic per token), --draft-vocab N (the MTP draft head over the first N rows only,
a draft proposes and the verify window decides: output unchanged), the packed Q8_0 default on Windows too (it was Linux only), STRATA_Q36_OLD_KERNELS=1
switches all of it off.
R9700, UD-Q4_K_XL: decode 82.5 -> 120.6 tok/s (greedy 10 prompts identical), --mtp 2 160 -> 174.
For that geometry only: the s27 gate/up kernel and the fused SwiGLU/q8_1, then the old down kernel (the new down kernel is slower at n_ff 512).
Flash-Next's shapes (2560/640) take the same paths as before; the path is still STRATA_EXPERT_V2K (default on gfx12).
The same kernel measured on an R9700 (gfx1201): qsa_scores_gfx1100_parity --full compares 223,664,777 scores and 11,895,800 ids bitwise equal
to the warp scorer (35 cases), the scorer stage with 256 queries is 3.6x faster at 131K and 4.0x at 240K, and a fresh prompt reads 8.7% faster at
131K tokens (4 of 4 pairs) and 16.4% faster at 230K (3 of 3), with identical answer tokens and byte-identical exact_pp residual dumps at 70K and 131K
tokens. So on gfx12 it is on for batches of >= 64 queries with >= 16,385 live blocks (65K cells and more) unless STRATA_SCORES_THREAD=0; gfx1100
stays the author's opt-in (STRATA_SCORES_THREAD=1). A one-time stderr line says when it engages by default.
The persistent read's barrier timeout uses __builtin_amdgcn_s_memrealtime, which gfx11 and gfx12 do not have (clang: "needs target feature
s-memrealtime"): the PR as ported failed to build for gfx1100, gfx1151 and gfx1200/gfx1201. On those parts the barrier compiles to an immediate
abort, fused_gr_persist_init refuses the read with a start-up line, and the default read is used. An R9700 (gfx1201) measured the read +0.7% / +1.0%
per window slower than the default in the PR thread, so nothing is lost. The silent, permanent barrier abort at run time that the same thread
describes is noted in docs/AMD_HIP.md.
The verifier reads the hyper-connection twice per layer at 1-8 tokens: three small launches with two global
dependencies, the down projection on 41 blocks. Opt-in STRATA_HC_PERSIST=1 runs it as one kernel of 80 blocks with two
grid barriers on an atomic counter: the down projection split by column slices over all 80 blocks, a distributed
reduction, then the up projection and the same epilogue as gr_up_kernel.
gfx1030 (2x RX 6900 XT, layer split): the read 1.35-2.10x at 1-8 tokens; ms per verify window -4.2 % at 4K and
-2.8 % at 32K (one binary, the variable unset / set, 6 alternating rounds, every round the same way). Another summation order (not bitwise); against a
double-accumulation reference (STRATA_HC_PERSIST=3) as close as the default read.
Residency is decided by running every token count once at start-up with a 50 ms barrier timeout and comparing with
the default read; a timeout keeps the default read for that token count and larger ones, wave64 cards are refused.
One line per card says what was decided. Only the verifier's reads use it (one stream per card); the drafter's reads,
the prompt path, CUDA and SYCL are unchanged (CUDA and SYCL get two empty stubs). With the variable unset both entry
points return in their first line: the output is bitwise the same as main.
Measurements, the simulated smaller cards and a stand-alone timing tool:
bench/results/2026-10-12-hc-persist-gfx1030/.
block_topk_wide_kernel is plain shared-memory / atomicAdd code with no sm_90
feature, but its definition and the dispatch branch sat behind
#if !defined(__HIPCC__), so on AMD a context past the register kernel's reach
(about 135K cells) always fell back to the 4-pass ref radix. The ids are
identical by construction (the same radix rule), which the bench states:
256/256 at 256 queries x 65537 blocks, 1/1 at nq=1.
Measured on a Ryzen AI Max+ 395 (gfx1151, ROCm 10.1), qsa_select_bench:
- nq=1, 131073 blocks (a decode window at 512K context): 2.24 -> 0.77 ms (2.9x)
- nq=256, 65537 blocks (a prompt batch at 262K): 3.16 -> 1.43 ms (2.2x)
End to end this moved the deep-context decode of a 512K-token prompt from
19-22 to 24-26 tok/s and the 512K prefill from 473 to 634 tok/s on that PC
(together with the split scorer of another change). STRATA_TOPK_STREAM=0
still restores the register kernel.
Upstream aa7b0cb gates PcieAuto to CUDA single-GPU (#if !defined(STRATA_USE_HIP)); the
machinery itself is engine-generic (pool ms per missed expert + miss counts vs the probed
link rate), and this build's start-up H2D probe is already best-of-4 with a +-0.7% spread,
so the HIP exclusion was unmeasured-only, the same class as the ROUTE_TAIL_SKIP gate.
Same opt-outs as CUDA: layer splits excluded, STRATA_PCIE_FRAC_DEFAULT=old restores the
fixed start-up rule. Provenance: MistVVK/XeStrata commits 05309096/055cf9b0/c71cac34/
5952e8e7 (their measurement-stabilization cluster) demonstrated the failure mode and the
fix class; their exact subsystem is not portable (it is their own engine code).
(cherry picked from commit acb3deb0522a8fc351a4047d2bd11b8ab7a73dca)
Co-authored-by: Zac Chen <[email protected]>
The engine turns 18 exact (bit-identical) switches on by default on gfx1151
(src/core/arch_defaults.cpp) but keeps gfx1150 out of its arch defaults
(src/core/arch_defaults_test.cpp asserts arch_default_env("gfx1150").empty()).
A gfx1150 install gets the 17 that apply here - the gfx1151 set less
STRATA_HCD_EXACT, which copies the gfx1151 hipBLASLt table's solutions 1176 / 1177
and so does not fit beside a gfx1150 table - written into the config's "env"
beside STRATA_HIPBLASLT_TUNING. The Radeon 890M report (docs/AMD_HIP.md;
bench/results/2026-10-07-community-gfx1150) measured the same greedy answers with
all 17 on and with all 17 off, decode +2% and prompts +3-6%.
The set is setup's own (SETUP_ENV): a re-run refreshes it, and it does not linger
after the opt-in is turned off. A new test (tools/test_setup_strix_halo.py,
Gfx1150Switches) keeps GFX1150_SWITCHES equal to the gfx1151 arch-default table
minus STRATA_HCD_EXACT, keeps the engine's own default path for gfx1150 empty, and
checks the SETUP_ENV wiring. Setup's suites pass (strix_halo 40, amd 36, choices 47).
Co-authored-by: onionbutties <[email protected]>
STRATA_EXPERIMENTAL_GFX1150=1 takes the Radeon 890M / 880M iGPU (Ryzen AI 300,
Strix Point, gfx1150) as a unified-memory card like Strix Halo: it joins AMD_ARCHS
and the UMA handling (new is_uma_apu helper), with notes of its own. The engine
already builds and runs for gfx1150 (0.1.40.2, #1217; tools/hip/gfx1150-hipblaslt-100401.txt
is calibrated on it); no release ships a ready-made Windows engine for it yet, so
a Windows user builds one (STRATA_HIP_ARCHS=gfx1150 tools\hip\build_windows.bat) and
starts setup with --prebuilt dist\. Unset, the default path is unchanged: the test
that asserts gfx1150 is not in AMD_ARCHS (tools/test_setup_strix_halo.py) still passes.
Co-authored-by: onionbutties <[email protected]>
The engine turns 18 exact (bit-identical) switches on by default on gfx1151 (src/core/arch_defaults.cpp) but keeps
gfx1103 out of its arch defaults (src/core/arch_defaults_test.cpp asserts arch_default_env("gfx1103").empty()). A
gfx1103 install gets the 17 that apply here - the gfx1151 set less STRATA_HCD_EXACT, which copies the gfx1151
hipBLASLt table's solutions 1176 / 1177 and so does not fit beside a gfx1103 table - written into the config's
"env" beside STRATA_HIPBLASLT_TUNING. One 780M (Ryzen 7 8745HS, 60 GB, GTT 50 GiB, ROCm 10.1, Q2_0, 4K prompt)
measured the same greedy token ids with the 17 on and with them off, +13% to +15% prompt and decode +3.2% (the
decode half alone); bench/results/2026-10-12-community-gfx1103.
The set is setup's own (SETUP_ENV): a re-run refreshes it, and it does not linger after the opt-in is turned off.
The gfx1103 note now names the native dot4 / WMMA kernels and docs/RADEON_780M.md, which documents the opt-in
build, the table and the measured numbers. Setup's suites pass (strix_halo, rocm_pins, amd, choices: 143 tests).
Refs #1423.
Co-authored-by: AlexisCarbonell <[email protected]>
Calibrated on a Radeon 890M (Strix Point, gfx1150) with
tools/hip/tune_hipblaslt --workspace-mib 32 over the 16 dense GEMM geometries of
the shipped gfx1150-hipblaslt-100401.txt at T = 64, 128, ..., 16384: two passes,
each row keeping the solution with the lowest mean of all six repetitions. The
three rows slower than plain hipBLAS (the tiny bf16 N=1 K=2560 shape at T = 64,
128 and 256) are left out, so those buckets route to the nearest good row. Every
remaining row beats plain hipBLAS: 1.07x to 16.7x per GEMM, geometric mean 5.0x.
The solution IDs are scoped to hipBLASLt 1.5.0 (version 100500, the ROCm
10.2.0a20260930 nightly the engine here was built with).
The Radeon 780M (gfx1103, RDNA3, 12 CUs) builds and runs (issue #1423), but had no hipBLASLt table, so the prompt's
dense GEMMs ran on plain hipBLAS. This adds tools/hip/gfx1103-hipblaslt-100401.txt, calibrated on one 780M with
ROCm 10.1's hipBLASLt 1.4.1 (version number 100401) over the gfx1150 table's 144 GEMM cases at T = 64..16384 with
the engine's 32 MiB workspace.
Two tune_hipblaslt passes back to back; each row keeps the solution with the lowest median of all six repetitions,
and a row is left out when no candidate beats plain hipBLAS. 143 rows, 1.05-23.30x per GEMM, geometric mean 4.95x;
the one shape left out is bf16 N=1 K=2560 T=64 (best 0.039 ms against 0.024 ms on plain hipBLAS, sub-100 us
timings), where the engine falls back to hipBLASEx.
Measured on the 780M (Ryzen 7 8745HS, 60 GB, GTT 50 GiB, Q2_0 + MTP, 32K context): a 4K prompt read 90 tok/s on
plain hipBLAS against 187 with the table (medians of an interleaved A/B; with the gfx1103 exact switches and the
WMMA opt-ins, 277 at 4K and 272 at 32K). hip_prefill_hipblaslt_gemm passes with it (launches=4 fallbacks=0) and
hip_prefill_hcd_exact_parity skips, because the table picks no 1176 / 1177 for the HC down GEMM - the gfx1150 table
gets the same skip, which is why STRATA_HCD_EXACT is not in the gfx1103 switch set. Setup installs for gfx1103 only
with STRATA_EXPERIMENTAL_GFX1103=1 and then picks the table up when the installed hipBLASLt reports 1.4.1.
Docs, the community report (runs.csv, tune-summary.csv, the server config, the engine's start-up lines) and the
setup tables list carry the numbers. Refs #1423.
The switch exists for CUDA (#1372) and is compiled out on HIP. On gfx11 it now runs the WY form of the gated delta rule in
chunks of 16 tokens, every product one 16x16x16 WMMA (FP16 operands, FP32 accumulation, the state in the accumulators), with the
output norm and the z gate fused: a prep kernel for T and P of every chunk, then a scan with a block per value head.
It rounds differently from the FP32 recurrence, so it is opt-in; the default path is unchanged. A gfx1100, gfx1101, gfx1102, gfx1150 or gfx1151 part, 64 tokens and
more, an output row stride divisible by 8 and no STRATA_GDN_REC_HEADS take it; anything else runs the existing kernels.
tests/hip/gdn_chunked.cpp: bounds on the FP16 output and the carried state against the existing recurrence over 0-16,385 tokens
(chunk and stride edges, a split request), a double-precision CPU check, and bit-identical output with the switch off or unset.
tests/hip/gdn_rec_head.cpp unsets the switch, as its variants 0 and 1 are the references there. hip_gdn_chunked --bench and
--bench-default time this path and gfx1151's default kernels, one per process (the switch is read once).
gfx1151 (Ryzen AI Max+ 395, UD-IQ4_XS): a layer's recurrence + norm takes 1.60x less time than the default kernels at 16K and
32K tokens; a whole prompt 1.4-2.1% less at 4K-128K after a warm-up prompt (11 pairs each: 1.8-2.1% with the docs/STRIX_HALO.md section-5 switches,
1.4-1.5% without them, 10 or 11 of 11 faster); first-token KL against the default 0.006-0.007 on average, the same top token in
12 of 12 prompts. Run on gfx1151 only; kernels.cu compiles for gfx1030, 1100, 1151, 1152 and 1201 (docs/AMD_HIP.md).
Co-authored-by: libratechw <[email protected]>
Q8_0 has no minimum term and the F16 kernel's output already differs from the int8 kernels', so the KQ opt-in is
not needed for it. UD-IQ4_XS has five Q8_0-down layers (4 IQ3_S / Q8_0, 1 IQ4_XS / Q8_0): they left the fused path
for MMQ, and since fused_ring() asks every layer, the pack kept MMQ's buffers. d_covered_on() is the one test, for
native_supported() and experts_native(); gfx12 keeps d_covered().
UD-IQ4_XS, gfx1151, 3 interleaved rounds, prompt tok/s (v0.1.42 / 65d809c6 with STRATA_PF_F16=1 / this):
4K 926.6 / 992.2 / 1054.8, 32K 1257.4 / 1364.6 / 1397.1, 128K 1202.0 / 1309.4 / 1333.6.
native_f16_kernel takes the IQ gate/up formats (IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS) and the Q2_0, IQ4_NL
and Q8_0 down: load_unit / convert decode each sub-block once into an int8 weight tile in LDS with its scales as
half2, each wave turns its own rows into F16 (gufo's Q8_0 decode: sign flip, byte permute into 1024's mantissa,
one packed multiply), and the F16 WMMA accumulates all of K. The work item, the activations staged as F16, the WMMA
loop and both epilogues move to moe_f16_common.cuh, shared with the K-quant kernels (moe_f16_kq.cu, now on it; its
output bits are unchanged). The switch becomes STRATA_PF_F16=1 (was STRATA_PF_F16_KQ) and covers both; the K-quants
still need STRATA_PF_FUSED_KQ=1. gfx12 keeps the int8 kernels.
hip_prefill_fused_iq (gfx1151, ROCm 7.14.1 clang 23, top 10 over 512 experts, one 128-expert launch, ms):
- IQ3_S / IQ4_NL at 16384 tokens: gate/up 11.65 -> 9.87, down 6.11 -> 4.71; layer at 8192 37.3 -> 32.2.
- every pair is faster at 4096 and 16384 (IQ2_S gate/up 14.38 -> 9.85, Q8_0 down 6.80 -> 4.79), and at 128-2048
tokens equal or faster.
- FP32 reference rel RMS as the int8 kernels' (IQ3_S / IQ4_NL 1.456e-2 against 1.454e-2); against the int8 model
and MMQ the F16 path sits at 2.4-2.7e-3, so both checks allow 5e-3 under STRATA_PF_F16=1 (f16_on).
- With the weights decoded to F16 in LDS (18 KB a block) gate/up was 9.83 and down 5.71. Neither a VGPR cap nor
dropping the sign table (4 blocks per WGP) nor decoding after the products moved gate/up; without the decode it is
8.98, without the weight loads 9.30.
Off by default (the output differs from the int8 kernels').
moe_f16_kq.cu's bodies are gfx11's WMMA layout (and trap on other targets); v0.1.42's gfx12 native kernels share
experts_native's launch, so the F16 path is taken only off gfx12.
down_kernel takes gate/up's structure for UD-Q4_K_XL's Q5_1 down (47 of its layers): the codes and high-bit words
packed in LDS, w = q d + m per wave in F16, H staged once as F16 from its int8 blocks, FP32 outputs at the routed
rows. The activation staging and the WMMA loop are shared with gate/up (ActStage, mma_tiles). Q8_0 down stays on
native_w11_kernel.
hip_prefill_fused_iq (gfx1151, one layer, 16384 tokens x top 10 over 512 experts): down 7.10 -> 4.83 ms per
128-expert launch; Q4_K / Q5_1 layer 62.7 -> 53.5 ms (85.3 on the int8 kernels); FP32 reference rel RMS 1.304e-2.
UD-Q4_K_XL's routed gate/up was 6/10 of the prompt-time gap to gufo on gfx1151 (128K: 23.1 s against 13.9 s).
moe_f16_kq.cu adapts gufo's paired RoutedF16GEMMKernel (97b8645f, MIT) to Strata's tables, int8 activations and H:
the Q4_K / Q5_K codes stay packed in LDS and each wave turns its own rows into F16 after the read, so the F16 WMMA
accumulates all of K in F32 with no per-block epilogue; the activations are staged once in LDS as F16 for every wave;
the epilogue writes H as int8 per 32 features for the existing down kernels. One block per work item; two tiles of
an expert per item from 64 routed rows per expert on average.
hip_prefill_fused_iq (gfx1151, ROCm 7.14.1 clang 23, one layer, 16384 tokens x top 10 over 512 experts):
- gate/up kernel 14.63 -> 8.46 ms per 128-expert launch; Q4_K / Q5_1 layer 85.3 -> 62.7 ms, Q5_K / Q5_1 89.9 -> 66.2.
- against the FP32 reference the error is the int8 path's (rel RMS 1.305e-2 / 1.302e-2: the int8 activations
dominate); the int8 model check allows 5e-3 under this switch (2.5e-3 measured with Q8_0 down).
- the output bits are the same with 64- and 128-row items.
Off by default; it needs STRATA_PF_FUSED=1 and STRATA_PF_FUSED_KQ=1, whose output already differs from MMQ's.
After a prompt the borrowed expert-cache slots are refilled from the experts' files. With the GGUF read in place an
expert is three tensors, and the refill first assembled each 4 MB blob in a staging buffer (a copy on the calling thread)
and then copied that to the slot. The pieces are contiguous in the mapping, so they can go straight to their offsets in
the slot: blob_parts() hands them out, ExpertCache::fill_slot_queued_parts copies them on the same queue. The bytes in
the slot are the same. Off by default; gfx1151's table turns it on.
STRATA_CPU_IQ_ALL_EXACT1 reproduces ggml-cpu's AVX2 single-token dot bit for bit, which only
holds if the ggml-cpu in this binary is that dot: a STRATA_ISA_FLOOR build runs ggml's AVX1/SSE
dots, and a native build on an AVX-512 CPU run with STRATA_FORCE_AVX2=1 runs its AVX-512 one.
The guard is #2215's (ggml_cpu_has_avx2 / ggml_cpu_has_avx512).
Co-authored-by: Andy Slater <[email protected]>
Three changes for two-card AMD PCs whose faster card does not come first in the
runtime's enumeration (Windows HIP orders the cards itself; HIP_VISIBLE_DEVICES
only filters):
- gfx1032 (RX 6600) joins the unvalidated build list - it shares the gfx1030 /
gfx1031 / gfx1034 dp4a path.
- STRATA_PRIMARY_DEVICE=N picks the visible ordinal the engine's own session
runs on; it is set before any context or allocation exists. The helper
caches take the other visible devices, the HIP arch check and the peer tier's
device queries follow it, and the primary's session is created on the chosen
device. A layer split still assumes device 0 and is refused with this set.
Unset or 0 is upstream's behavior unchanged.
- The helper-GPU caches are allowed beside the resident RAM mode: the experts a
helper holds are left out of the RAM copy as a split stage's are, and the two
adaptive tiers already keep each other's experts out. RemoteExperts::finish
spins on its stream like PeerExperts::finish - on HIP the CUDA branch's
cudaInitDevice(ScheduleSpin) does not exist.
Measured on an RX 6800 (16 GB, gfx1030) + RX 6600 (8 GB, gfx1032), Windows 11,
32 GB of RAM, Swift 1.5 IQ3_XXS at 160K context, the resident RAM mode with the
helper's cache left out of the copy: decode 37.5 -> 44.2 tok/s on 8K-token
requests with a 3800-slot helper; the CPU pool's share of the routed experts
falls from ~2.9 to ~0.4 entries per layer-window.
Rebased onto 296b59d2 (conflicts in generate.cpp, peer_experts.cpp, hip_backend.cmake, docs/AMD_HIP.md resolved by keeping main's code and adding the PR's). Changes: remote_caches is kept (still used by the --batch-mtp check); the spin in RemoteExperts::finish is HIP only (CUDA keeps its cudaInitDevice(ScheduleSpin) + blocking sync). NOT COMPILED YET, not run.
Co-authored-by: tuandat3019 <[email protected]>
The menu never hid NVFP4 on an AVX2 CPU; the engine runs a native pack's rows on AVX2 kernels that are bit-equal to
ggml's. The NVFP4 menu line and the install step now say, on a CPU with AVX2 and no AVX-512, that the CPU's share of
the experts runs on AVX2 rows (measured about as fast when memory-bound, up to ~12% slower on 4-token windows).
A note only: nothing is refused or capped. tools/test_setup_menu.py: 2 tests.
The fork stays off by default on HIP (#816, #826): an RX 6800 on Windows and 4x R9700 decode 12-45% faster without it.
The same card family on another PC goes the other way (#1272: 2x RX 6900 XT on Linux), and the output is identical
either way, so it is a per-PC choice - the open item of #1272.
- tools/calibrate.py: step 6, HIP only. With everything chosen before, the engine is started twice, with
STRATA_SH_STREAM=0 and =1 set explicitly, two rates each; "1" is kept only when it beats "0" by more than MIN_GAIN
(the rule of the other steps). measure() returns the choice as "env"; a failed start drops that candidate (#1337).
start_engine takes an optional env; apply_env() sets the chosen variables and removes the others of ENV_KEYS, so an
older calibration's choice does not linger.
- setup.py: --calibrate writes "env" into the config and the settings file; a reinstall applies a saved calibration's
"env" the same way (a calibration saved before this step leaves "env" as it is).
- docs/AMD_HIP.md: the SH_STREAM note says calibrate measures it.
- tools/test_calibrate.py: the step runs on HIP only, keeps the fork only on a win, falls back on a failed start;
apply_env; setup writes and later removes the variable.
Measured with the real engine (0.1.42 tag, 2x RX 6900 XT gfx1030, Linux, IQ3_S layer split, setup-style config): the
whole calibration took 982 s, the new step chose STRATA_SH_STREAM=1 (off 62.2 / 61.3, on 67.2 / 66.6 tok/s, +8.3%).
Separately, with a fixed protocol, the fork's output is byte-identical to off (4 of 4 requests, one card and the split).
The scorer of 65ce329 is plain FP32 arithmetic, so it runs on any HIP card as it is; without matrix cores the
prompt path otherwise scores on the warp kernel. Its tiling becomes template parameters: every tiling keeps each
score's fmaf chain over k = 0..127 in order and the same relu sum over the heads, so every tiling gives the same
bytes (RX 6900 XT, qsa_select_bench at 32K and 128K: 14 tilings, also on 60 and 72 of its 80 CUs through
HSA_CU_MASK, all byte-identical to the original kernel; the fastest tiling the same at every width). CUDA keeps
the tiling it was written with; gfx103x runs 32/128/16/8: 256 queries at 128K 2.69 ms (warp kernel) -> 0.695 ms
(the original tiling) -> 0.587 ms. FP64 gate passed, selections identical 256/256. On the 2x RX 6900 XT layer
split a 128K-token prompt reads 7.4% faster, 32K 0.8%. A HIP ctest runs it at 131K.
Rebased onto 296b59d2: the docs/AMD_HIP.md conflict (main's SGEMM/rocBLAS-table bullets from the ports before it) is resolved by keeping all bullets.
Co-authored-by: xjc10 <[email protected]>
rocBLAS picks the kernel for each FP16 product from its own table by shape, and on gfx1030 that pick is far
from the best kernel the same library holds for some shapes: N = 10240 K = 2560 (the GDN input projection)
below T = 1152 runs at 5.2 TFLOPS where the best of rocBLAS's 237 solutions for n = 799 is 6x faster (not an
alignment effect: n = 800 is as slow as 799).
tools/hip/tune_rocblas enumerates the library's solutions per shape and token bucket, checks each against the
default kernel's output and that it also runs at smaller odd token counts, and writes the winners (and
`default` rows bounding their reach) as tools/hip/<arch>-rocblas-<version>.txt; Gemm::f16_inplace runs the
table's solution for a shape (STRATA_ROCBLAS_TUNING: a file, or the directory from which the engine takes the
file of its own rocBLAS build; the version string, tweak hash included, is checked). One tiny default product
at startup opens rocBLAS's kernel library (a solution-index call before that fails although the index is
valid); a refused solution falls back to the default kernel for that call, after three refusals for good.
setup points the engine at tools/hip when a table for the card's architecture exists. Shipped: the gfx1030
table for ROCm 10.0.0's rocBLAS 5.6.0.8d1ae90e (112 rows, 76 with a solution).
Measured (RX 6900 XT, 6f32ec0 + #835 + this, temperature 0, the same answers with and without): a 799-token
prompt's GPU time 3,329 -> 2,985 ms (-10.3%), 495 tokens 2,477 -> 2,139 (-13.6%), 8,874 tokens 11,978 ->
11,459 (-4.3%); a PCIe-bound 1,108-token prompt unchanged (its saving reappeared as `wait copy`). ctest 66 of
70 (the 4 failures are outside this change); hip_prefill_rocblas_gemm, the new smoke test, passes with 0
fallbacks. Builds on #835. bench/results/2026-10-05-rdna2-rocblas-table has the per-run lines and the tuner's
summary.
(cherry picked from commit e35e103604)
Rebased onto 296b59d2: conflicts in setup.py (SETUP_ENV gained STRATA_ROCBLAS_TUNING beside the 0.1.44 entries) and docs/AMD_HIP.md (kept both sections) resolved.
Co-authored-by: xjc10 <[email protected]>
The rebased #704 under the name asked for there (upstream's STRATA_HC_Q8 now
reads the GGUF's Q8_0 bytes, a different thing). Opt-in. The four BF16 hc_*
projections per layer (1,200 MiB) are skipped by the loader and uploaded as
int8 codes with one fp32 scale per 32 values (675 MiB); the freed VRAM goes to
the expert cache.
- Q8 is a template parameter of the down/up kernels (single- and multi-token):
the BF16 instantiations compile exactly as before, the launch sites pick the
Q8 ones when w_down / w_up carry scales. The inject rows stay BF16.
- The staged / v3 / upmix / PF_HCDOWN / gfx906 split paths read BF16 rows and
are skipped for it; the prompt path expands the codes into a bf16 scratch for
its GEMM.
- The loader seeks with 64-bit offsets.
- A layer split keeps BF16 with a warning instead of exiting.
Fix-up to the ported #1123 commits. The route is not bit-identical to the native call
(the author measured at most 4.3e-6 of the largest output apart), so unset keeps the native
path on every card, gfx103x included; =1 enables it on gfx103x devices only (a mixed layer
split keeps the native call on the other cards), =all on every AMD device for A/B arms and
tests. The buffers are no longer freed under an in-flight GEMM: the stream is synchronised
before the FP32 copies grow, as the sm_60 FP32 route does. docs/AMD_HIP.md says so.
Port of PR #1838 (commit fb868db01) onto 0.1.44.1. Test file merged with #2134's foresight test (both kept). replace_resident now drops a Foresight slot for the outgoing expert while its RAM place is rewritten (#2134's copies read those places).
Co-authored-by: SanghunYun <[email protected]>
Rebased onto 296b59d2: the docs/AMD_HIP.md conflict (the gfx1031/gfx1150/gfx1152 bullets main gained) is resolved by keeping main's text.
Co-authored-by: vallicgrr <[email protected]>
STRATA_QSA_DUMP_LAYER=i keeps one QSA layer, STRATA_QSA_DUMP_STEPS=1 appends
the steps to each record.
pa_bench: the prompt sparse attention (qsa_prompt_attn_batch) on a
recorded selection - a STRATA_QSA_DUMP record of one layer and chunk with
its steps appended - random q and int8 K/V; median time and an output
hash for A/B of kernel variants (PA_BENCH_OUT=file writes the output).
On Turing the v2 int8 prompt attention (STRATA_PA_V2T=1) stages the next
chunk in registers, so its second shared-memory stage bought nothing - and
at 45.6 KB a block only one fit an SM (64 KB): four warps on four
schedulers, every latency exposed (ncu: 12.5 % warps active, tensor pipe
8 %). Smem2T: one K/V stage, and the q.k partials of cell group nt
(8 cells) written over the 512 bytes of the warp's own K slice that held
those cells (read by then, a __syncwarp between) instead of their own
8.4 KB. 19.7 KB a block: three per SM (168 registers fit three). The
same arithmetic in the same order - bitwise the same output.
RTX 2080 Ti, a recorded 130K selection (QSA layer 5, last chunk: 15871
queries x 2051 cells, tests/cuda/pa_bench.cpp): 113.8 -> 68.5 ms, same
output hash. Engine (Q2_0 pack, 16K chunks, fused experts): 98K prefill
53.3 -> 49.9 s, 4K 3.54 -> 3.42 s; first logits bitwise identical at 98K
and 4K, the same generated text.
On Turing the int8-KV prompt attention ran v1 because v2 stages its K/V
slices with cp.async. Here a warp loads its next slice into registers and
stores it into its other stage after the current chunk (the same bytes in
the same places); the MMAs are the m16n8k8 pairs v1 already uses. Accuracy
as v1 (qsa_prompt_attn_parity vs FP64: 3.7e-6 / 4.9e-6 / 4.5e-6 / 3.8e-6 of
3.5-3.7). RTX 2080 Ti, ms per chunk v1 -> v2: 32K x 1024 queries 13.0 -> 7.6,
64K x 512 6.7 -> 4.8, 1.5K 6.2 -> 5.0, 2.1K x 256 3.2 -> 2.3. Opt-in until
the engine A/B.
With STRATA_BF16_TC the prompt path converted xn16 (T x 10240) from BF16 to
FP16 before each of its two GEMMs (down and inject), in 16M-element slices:
~2.8 s of a 130K prompt read on an RTX 2080 Ti. The four xn16 writers now
store gemm.cu's conversion of the BF16 value (bf_to_hf_conv: the same clamp
and rounding as bf16_to_f16_kernel), and Gemm::bf16(..., x_f16) skips the
conversion on the tensor-core path (same slices, same GEMMs) and converts
back to BF16 on any other. Only when the stage runs BF16 on FP16 tensor
cores, without the BF16X2 low parts or a padded xn16 stride; the MTP draft's
own read keeps BF16. STRATA_XN_F16=0 turns it off.
First logits after a 98K prompt: bitwise identical; prompt read 1431 -> 1473
tok/s; a 4K run writes the same tokens.
The per-group dequant kernel writes 2 bytes every 64 bytes; 0.1.43 coalesced only the Q8_0 path (#1720). On compute
capability 7.5 the BF16 projections of the prompt path take a kernel that writes 16 bytes per thread; the values are
the same (a conversion, no arithmetic), so the output is bit for bit the old kernel's. Other cards are untouched.
The rocBLAS the engine ships has tuned gfx1030 kernels for FP16->FP16, int8 and
FP32->FP32 only; every FP16/BF16-input, FP32-output GEMM (rocBLAS HS/BS) runs a
fallback kernel at ~5 TFLOPS. Widening the inputs to FP32 is exact and SGEMM
accumulates in FP32 like the native call.
Nothing passes through FP16, so unlike STRATA_HIP_PROMPT_F16=1 (#835) the
route cannot overflow: an FP16 GEMM output above 65504 becomes inf and can
spread as NaN, while here it stays a finite FP32 value. It is slower than the
FP16 route, and not bit-identical to the native call (at most 4.3e-6 of the
largest output apart on random inputs).
RX 6800, Coder IQ1_M, fresh 12K-token prompts (4 each, A/B in one binary via
STRATA_RDNA2_SGEMM=0): prompt 312 -> 452 tok/s, GPU timeline 37.2 -> 25.5 s
(gdn 9.4 -> 2.8 s, qsa proj 3.6 -> 1.4 s, hc read 3.85 -> 2.84 s). Decode
unchanged. A 9-task code-repair benchmark: 9/9 first try on both, model time
458 -> 374 s.
Default on for gfx103x only; N < 64 keeps the native call; if the FP32
buffers cannot be allocated the native call runs.
Rebased onto 296b59d2: the conflict in Gemm::f16 and gemm.hpp (the CUDA FP16 algorithm fallback and the sm_60 STRATA_DENSE_F32 block that main gained) is resolved by keeping both; the RDNA2 call sits before the CUDA/HIP GemmEx call.
Co-authored-by: vallicgrr <[email protected]>
block_topk_wide_kernel reads a query's scores six times (four radix passes,
a count, the emit): past ~115K cells that is DRAM traffic. block_topk_sample_kernel
takes a bound from every 32nd block (the KS-th largest of those keys), reads
all blocks once keeping those at or above it, and when they hold at least
`width` cells computes block_topk_kernel's threshold, `above` and tie budget
over them and emits them in block order (bitonic sort): every block at or
above the threshold is a candidate, so the ids are the reference's.
Otherwise the query is flagged and block_topk_wide_kernel<true> runs it.
qsa_topk_parity --selftest and qsa_topk_active_parity: identical ids
(continuous, 64-level, equal scores - the last ones take the fallback).
RTX 2080 Ti, 256 prompt queries under a 512K capacity: 131K 0.51 -> 0.45 ms,
262K 0.95 -> 0.59, 400K 1.48 -> 0.71, 520K 1.84 -> 0.79. Used above 28,672
active blocks; STRATA_TOPK_SAMPLE=0 keeps the wide kernel.
A verify window's 1-8 queries scored and selected on a handful of SMs; at
420K cells on an RTX 2080 Ti the two took ~6 ms of a ~52 ms window.
- block_scores_halve_kernel: the decode scorer's warp sums as one
recursive-halving reduction (each value pairs lane l with l ^ o at offsets
16..1, the butterfly's additions operand for operand), the lane's query
values in registers (shared-memory reads were the limit, ncu mio_throttle),
5-8 queries as two calls of up to 4. Same bits (qsa_select_bench
QSA_BENCH_DECODE=1 hashes, 32K-524K, 1-8 queries). 420K: 1 / 3 / 4 / 6 / 8
queries 117 / 265 / 347 / 510 / 774 -> 101 / 118 / 125 / 238 / 255 us.
STRATA_SCORES_HALVE=0 keeps the old kernel.
- qsa_block_topk_split: the decode top-k over chunks of 2,048 blocks in six
launches (four radix passes into per-query global histograms, a count, the
emit); every CTA takes each digit from the complete histogram with
block_topk_kernel's rule, so the ids are identical (qsa_topk_parity
--selftest: continuous, 64-level and equal scores, 2K-524K). Decode calls
at 420K: 0.21 -> 0.033-0.064 ms; short contexts unchanged within 0.01 ms.
STRATA_TOPK_SPLIT=0 keeps the wide kernel (=1: on any card, for tests).
Both on sm_75 only by default.
block_scores_simt2_kernel: 8 x 8 products per thread, twice the blocks per
CTA and the next K chunk in registers; every dot is the same fmaf chain over
k = 0..127 and the same relu sum, so its scores are bitwise those of
STRATA_SELECT_SIMT=1 (qsa_select_bench QSA_BENCH_HASH=1, 131K-510K).
RTX 2080 Ti, 256 queries: 131K 1.21 -> 0.95 ms, 262K 2.01 -> 1.87,
400K 3.00 -> 2.82, 510K 3.77 -> 3.57 (the warp kernel: 7.7 / 15.3 / 23.2 /
29.6 ms). Opt-in like =1 (another summation order than the warp kernel).
Below stream_all_min() the MMQ path gathered every expert with its own launch
and, for a staged one, its own wait and release event. The staged walk now
groups like the streamed walk: its ring is STAGE_GRP (32) slots, two groups, and
it stages at most 15 entries ahead, so a group's staged slots are not refilled
before the group is gathered (one wait on its last copy, one event releasing
them all). stage_slots() counts those 32 slots wherever the ring is counted or
taken, so a prompt borrows every byte it writes.
STRATA_PREFILL_GROUP_GATHER=0 turns grouping off everywhere, as before.
gr_up75_kernel: two batches of 4 rows of w_up loaded before the dots, the
token activations read once per batch, the 4 xor reductions interleaved and
the row tails held 4 rows to a uint4 - bitwise gr_up_multi_kernel (gr_parity,
gr_multi_parity, fused_gr_bench hashes). Used from 3 tokens on sm_75
(STRATA_GR_UP75). The staged down kernel takes a token group from
blockIdx.y; sm_75 runs T 4 and 6 as pairs (STRATA_GR_DOWN_TG).
fused_gr_bench cold, RTX 2080 Ti, whole read: T3 53.2 -> 50.2 us,
T4 64.1 -> 57.6, T5 76.2 -> 69.3, T6 86.1 -> 70.9; T1-2 unchanged.
native_expert_bench on real Q2_0 rows of Flash-Next, RTX 2080 Ti, bitwise
equal to mode 0 in every case: 10 groups x 1 entry 47.3 -> 41.4 us,
20 x 1 83.7 -> 71.5, 30 x 1 120.2 -> 100.2, 20 x 2 124.2 -> 111.4. The down
projection of mode 4 loses with 3+ entries per group (12 x 3 36.5 -> 40.8,
16 x 4 56.9 -> 68.9), so cap_groups > 30 keeps mode 0 for it.
STRATA_SM75_EXP=0 keeps the CUDA layout; STRATA_EXP_MODE still overrides.
adapt() marks an evicted expert non-resident in host_res at once; d_res follows in apply_pending, once the round's
copies have landed. When the tier does not wait for them (STRATA_ADAPT_NOWAIT=1, or STRATA_ADAPT_LAG > 1) the next
window runs before that. doorbell_publish_res reads d_res, sees the evicted expert as resident, and when every routed
expert of a layer looks resident it does not publish the layer's activation. The pool plans from host_res and
computes the evicted expert on the CPU from the activation published before. Now every verifier launch (run,
run_slot_rows, batch_launch, pl_launch; a layer split's stages too) runs a pre-launch hook. It patches the device
tables from a list of changed entries: res_patch, a small kernel on the verifier's stream reading mapped memory; the
whole table when the list overflows or a device does not map it at the host's address. A failed whole-table upload in
apply_pending leaves its entries to the hook. Only where the tier does not wait. --batch-groups with a layer split
publishes every activation instead: set_always_publish, which now also turns E-6's device plan off.
--pipeline-windows keeps its fence. STRATA_ADAPT_EVICT_SYNC=0 restores the old behaviour. STRATA_ADAPT_CHECKRES=1
reads the tables back before each window, compares them with host_res, and counts the layer-windows that computed a
CPU expert with no activation published.
RTX 5090, IQ2_XS, --expert-cache 12000, 32K explain, 600 tokens: NOWAIT=1 34 of 222 windows with mismatched tables
(2470 entries), 1 such layer-window -> 0 / 0 / 0; LAG=2 55 of 223 (3222 entries: every swap), 5 -> 0 / 0 / 0. LAG=2
is reproducible (two runs of each arm, the same tokens), and the fix changes its answer from token 228. The hook's
cost under NOWAIT (6297 slots, 3 pairs): ms a round -1.1% chat, -1.2% 32K, noise. The default (the tier waits)
installs no hook: tokens and first-token logits identical to d5ea7133 (p2k 200 tokens, 32K explain 600).
Port of PR #1517 (commit eb4796045) onto 0.1.44.1. Conflicts with the 0.1.43 adapt overlap gate and the tail-skip kind -2 resolved; set_adapt_gate and set_always_publish kept side by side. Docs line added.
Co-authored-by: sergiywith <[email protected]>
The real data_folder() moves the model files of this checkout and of the data folder named in the user's settings.json into --data-dir and saves that folder as the new data folder. With a TemporaryDirectory the models are deleted on exit and settings.json is left pointing at a deleted directory (found on the M4 and on the PC: the test wrote %APPDATA%/Strata/settings.json and ~/.config/strata/settings.json). The test is about --source, so data_folder is mocked.
The 2-4 column dense projections of the verify windows ran the plain multi-column kernels on Turing, which are
bound by dp4a throughput there from 2 columns on. mmvq_il_parity --bench --emit-table on an RTX 2080 Ti (SM 1650
MHz) gives a table like the Volta/Pascal ones: IQ4_XS 2560x10240 at 4 columns 81.1 -> 37.4 us, Q4_K 74.5 -> 39.1,
the Q5_K head 1671 -> 1045 us. Bitwise equal (the decode writes the same tokens). No one-column table: the
engine's dedicated single-column kernels sum in another order, for a 0.5% gain. STRATA_SM75_IL=0 turns it off.
Q2_0 512K resident, 4K prompt: one-shot decode 52.6 -> 55.5 tok/s; serve steady state ~69 -> ~77 tok/s.
prompt_attn_i8v3_kernel: QK^T and PV as mma m16n8k32 on the int8 K / V codes as stored; the other operand in three
exact 8-bit parts (q scaled per head and 64-dim group, p * v_scale per row and chunk; X = rint(x * 65536) split into
h, l signed and m unsigned bytes, a u8 x s8 MMA): 24 bits, exact int32 sums, combined in FP32. One K and one V stage
per warp, 12-row partials, launch bounds (128, 3): ~25.5 KB, 3 blocks an SM. sm_80+ (STRATA_PA_SM80); Turing,
Volta and HIP keep their kernels. Unset: v2, byte-identical.
qsa_prompt_attn_parity: against FP64 IMMA 2.15e-6 - 2.64e-6, v2 2.78e-6 - 3.30e-6, the FP32 kernel 1.90e-6 - 2.51e-6.
IQ2_XS, first-token KL to the FP32 kernel (STRATA_PROMPT_ATTN_OLD=1): median 0.0027 (IMMA) against 0.013 (v2);
2K 0.0026 against 0.0248. 1.35-1.53x a launch; a 32K prompt in 8K chunks 358 -> 312 ms of attention.
Fix-up to the ported #1149 commits. The range counters and the saturating to_f16
are compiled for HIP only, so the CUDA kernels (hf_sat in the SwiGLU product, the
embedding image) stay exactly as they were, with no flag read; set_f16_range /
f16_range_read are no-ops there. strata_hip.h (the STRATA_HIP_GFX906 compat layer,
which goes first on that backend's include path) gets the cudaMemcpyFromSymbol
mapping the HIP shim already has, so the counters also compile for gfx906.
The verify window reads pcie_frac of a layer's missed experts over PCIe, a share
the link probe sets; the CPU's speed does not enter it. With a slow CPU the GPU
spins on the pool's rows: IQ2_XS with one AVX2 pool worker (emulated) runs
16.8-17.2 ms a round at the probe's 0.55.
Opt-in, STRATA_PCIE_BALANCE=1 (one GPU, no batch slots, no --pcie-frac):
- wait_flag_ge_stamped: the flag spin kernels also write the GPU clock as the
plan arrives, as the PCIe part starts and as the GPU waits for the CPU's rows
(no extra launch; captured only with the switch on);
- the host times its pool per step; after each window, least squares with
forgetting fit the pool's a + c n_cpu + d n_pcie (d: the link's DRAM reads
slowing the pool - on a DRAM-bound pool moving experts to PCIe saves nothing)
and the GPU's g0 + g n_pcie;
- a layer then reads over PCIe the m of its misses that minimizes the longer
side; every 64th layer that would keep them all on one side moves one, so
both fits stay measured.
IQ2_XS, RTX 5090 + 9950X3D, chat to the end of the turn:
- one AVX2 pool worker: 16.77 / 17.17 -> 12.98 / 12.82 ms a round
(fit: CPU 42-60 us an expert, GPU 28 us a PCIe expert; 0.8 of the misses
move);
- the 9950X3D's 16 workers: 3 pairs 12.19 -> 12.17 (pool DRAM-bound, few move).
Off (the default) the logits and 32 tokens equal v0.1.39 (2K, fixed cache).
Port of PR #1548 (commit a513394c4) onto 0.1.44.1. Conflicts with the 0.1.43 tail skip, PCIE_MIN1 and the fused waits resolved: the fused wait+copy launches stay off while the balance stamps are on. Docs line added.
Co-authored-by: sergiywith <[email protected]>
The up GEMM of each hyper-connection read writes `gated` (T x 10240 FP32, 336 MB at an 8K chunk) and gr_mix_r reads
it back with R at ~96% of the bandwidth - only fewer bytes help. gr_upmix_cuda_kernel (128 tokens x 32 columns x 4
streams a block, K = 320 in cp.async-staged slices of 32, BF16 mma.m16n8k16 with FP32 accumulation, the mix in
registers in gr_mix_r's order writing all its outputs) never writes or reads `gated`. Behind the gfx11 switch;
sm_80+ per device (STRATA_EMULATE_CC honoured), chunks of >= 33 tokens; else the pair. Unset: unchanged.
gr_upmix_parity (new ctest): bitwise the cuBLAS + gr_mix_r pair at T >= 33 on an RTX 5090 (cuBLAS 13.3), <= 1.8e-7 at
T <= 32, both as close to FP64. 1.42x / 1.81x / 2.19x the pair at 2K / 8K / 32K; IQ2_XS: an 8K prompt -4.4%, 32K
-3.9% (the hc read -19% / -24%). Logits identical with it on (600, 2K, 32K); the 32K greedy tokens moved only through
the time-driven suffix-draft policy (identical with --suffix-draft 0).
mmq::quantize(m.mixed, m.src_dev, ...) read and quantized every token's row of `mixed` once per routed expert (K =
10). mmq::quantize_scatter uses llama.cpp's quantize_scatter_mmq_q8_1_cuda (the pinned quantize.cu strata_mmq
already compiles): each token quantized once, its block written to all of its rows through slot_dev, the inverse of
src_dev. The same q8_1 bytes, so the default; STRATA_QUANT_GATHER=1 keeps the gather. No new device code (CUDA and
HIP compile the same kernels); SYCL's prefill.cpp untouched.
prefill_mmq_scatter_test (new ctest): every MMQ q8_1 layout (D4, DS4, D2S6), T = 1..2099: 0 differing bytes. IQ2_XS
600 / 2K / 32K and Q2_0 2K against e8ca9afd: identical first-token logits and 32 greedy tokens. RTX 5090 (96 MB L2):
the quantizer 2x at 2K, even at 8K (the chunk's input fits in L2), 1.4-1.9x at 32K; a 32K prompt with
--prefill auto:32768: 77.5 -> 58.1 ms of quantization.
Co-authored-by: sergiywith <[email protected]>
The staged IQ3_XXS / IQ3_S tables trade a per-block copy and barrier for
cheaper lookups, which pays only where the grouped expert kernels are
decode-bound. On GP100 (6.0: no __dp4a, 70-140 GB/s) it does; on an RTX 3090
Ti (sm_86), forced on with STRATA_IQ_STAGE_TABLES=1, the same real Swift 1.5
IQ3_S layers (20 experts, 24 entries) took 3-13% longer (71-76 -> 75-84 us;
those kernels read at 490-665 GB/s there) and IQ3_XXS was unchanged, the
output bitwise the same. 6.1 cards (P40, GTX 10) have __dp4a and are
unmeasured, so the default now asks for 6.0 rather than any 6.x;
STRATA_IQ_STAGE_TABLES=1 still turns it on anywhere.
docs/OLDER_GPUS.md says so, and that neither STRATA_Q8_SM60 nor the tables
is measured on 6.1: both build for sm_61, and tools/q8_sm60_check.cu passes
on an RTX 3090 Ti, where the kernel's dot is the hardware __dp4a.
IQ3_S was the one i-quant gate/up format whose codebook the grouped expert
kernels still read from global memory (IQ2_XXS / IQ2_XS / IQ2_S stage theirs
everywhere, IQ3_XXS on 6.x since the first commit of this branch). Its 512
codebook words are now staged per block with two 256-word tables of byte
masks - IQ3_S keeps one explicit sign byte per 8 values, so the tables hold
the masks of a sign byte's bits 0-3 and 4-7 - applied as
(g ^ mask) + (mask & 0x01010101): every iq3s_grid byte is odd, so a negated
byte never carries into the next. Bitwise the same output (iq_multi_parity
and native_grouped_parity on a GP100 die, with and without
STRATA_IQ_STAGE_GRID_MMVQ=1).
One GP100 die, a verify window's layer of real Swift 1.5 IQ3_S experts
(20 experts, 24 entries): 517 -> 393 us with Q2_0 down, 492 -> 369 us with
IQ4_NL down. Swift 1.5 IQ3_XXS (13 of 48 layers IQ3_S), four GP100 dies,
1M context, decode tok/s, medians of three interleaved engine starts:
36.0 -> 37.2 (10.7K document), 36.3 -> 37.4 (cached), 43.1 -> 44.5
(reasoning), 41.6 -> 43.0 (code), with the same drafts accepted.
The switch covers both formats and is renamed accordingly:
stage_tables_on(), STRATA_IQ_STAGE_TABLES=1|0 (was STRATA_IQ_STAGE_GRID18).
Still the default on compute capability 6.x only.
The tool converted only the shard it was given and hard-linked the others, which is enough for Flash-Next (every
natively served projection and the head are in shard 1) but not for Swift 1.5 IQ3_XXS, which keeps 78 of them in
shard 1 and 223 in shard 2: one run left three quarters of them in their K- / i-quants, silently, and the kernel
only runs Q8_0 matrices. Now any shard of the model may be named; every shard that holds such a matrix is written
as before (same tensor-info length, Q8_0 blocks appended, offsets checked) and the others are hard-linked.
Checked: Swift 1.5 IQ3_XXS in one run gives both shards byte-identical to two single-shard runs (301 tensors);
Flash-Next IQ3_XXS converts 301 tensors in shard 1 and links shard 2.
GP100 has no __dp4a, and on a GP100 die the verify window's dense projections read their weights at 40-80 GB/s
of ~730. With STRATA_Q8_SM60=1 every Q8_0 dense matrix (any shape) and the head get the packed planes of
STRATA_Q8_PACKED and run src/kernels/cuda/q8_sm60.cuh: a warp (half / quarter warp for narrow rows) per row,
16-byte weight loads through the read-only cache, the window's q8_1 activations copied once per workgroup into
shared memory as an int8 plane plus float scales, STRATA_DP4A (vmad on sm_60), a shuffle sum with no barrier per
row, persistent workgroups (4 per SM, STRATA_Q8_SM60_BPS). Shapes it declines (> 48 KiB of shared memory: a
6144-wide row with 7-8 columns) keep the usual kernel. Not bitwise the exact kernels (summation order); the
packed copies cost their size in VRAM (Flash-Next: 2.9 GiB of dense projections over the stages, 0.6 GiB head).
The GSQ-RCO packs have no Q8_0 dense matrices: tools/q8_dense_gguf.py writes a shard 1 whose natively served
projections and head are Q8_0, keeping the tensor-info block's length so every expert keeps its offset (the native
pack is reused) and hard-linking the other shards beside it.
PH402 SKU 200 (four GP100 dies), clocks locked at 1050 MHz, Flash-Next IQ3_XXS, 32K, layer split 12,24,36,
--prefill 2048, greedy, decode tok/s (medians, interleaved starts): GSQ-RCO shard 28.3, Q8_0 shard with
STRATA_Q8_PACKED 31.1, Q8_0 shard with STRATA_Q8_SM60 38.9 (+37%); per window dense 19.2 -> 11.2 ms, head
4.25 -> 2.1 ms; prompt reads unchanged. tools/q8_sm60_check.cu: 57 shape x column cases against a CPU reference,
max relative error < 1e-5. docs/OLDER_GPUS.md has the table, the conditions and how to use it.
Co-authored-by: ruibeikaa <[email protected]>
Windows sends a GPU's interrupts to one logical processor, usually the first,
and the host thread spins there (--host-core first, the default). --host-core
last moves it to the last physical core, but not on a hybrid CPU, whose first
core is its best P-core, so a 12th-14th gen Intel or a Core Ultra keeps the
host on logical processor 0 with the interrupts.
--host-core sibling (or STRATA_HOST_CORE=sibling) keeps the host on that same
first core but on its other hardware thread. The interrupts keep the first
logical processor, which nothing else is pinned to, and the workers keep
every other core, since they take one logical processor per core (with
--pool-affinity auto / p-cores, which also use the P-cores' siblings, the
host's sibling is taken out of their list). Without an SMT sibling it is
--host-core first. It works the same on Linux (the sibling from sysfs), where
the interrupts are usually spread anyway.
detect_cpu_topology records the host core's sibling (CpuTopology::host_sibling)
in each layout; pool_affinity_test checks, in every affinity mode, that
sibling puts the host there with no worker on its core, and that it is first
without SMT. The startup log names the placement.
The README's range check said "none beyond 65504" over 110K tokens without
the numbers. The engine now has the counter: STRATA_F16_RANGE=1 records,
per device, the largest |value| and the count beyond 65504 (or not finite)
where a value enters or leaves FP16 - the activation images (hf_sat, the
image writers' one funnel), the BF16 weights converted to FP16
(bf16_to_f16_rows) and the FP16 GEMM outputs as they are widened
(widen_rows_f16: an output rocBLAS wrote beyond the range is Inf there) -
and prints one line per prompt. Off, the cost is one flag read per value;
on, a 34.4K-token prompt read at 889.8 tok/s against 888.5. to_f16 (the
hc image for the FP16-weight GEMMs, the PLE key image) saturates like the
other image writers now, so every FP16 image is under the same rule. The
HIP shim gains cudaMemcpyFromSymbol.
Measured on one RX 6900 XT over 466K tokens in nine prompts (corpus
slices, two prompts built to push activations - delimiter floods, a token
repeated 4,000 times, 6,000-character runs, base64, five scripts - and
four batches of real requests): outputs peak at 409.5 (160x below 65504),
activations at 112.2, the BF16 weights at 11.56; nothing beyond the range,
nothing non-finite. The prompts built to push activations stayed below
the ordinary corpus. One model (GSQ-RCO IQ3_S); the table is in the
README's range section.
The README also gains the same-card comparison with #1006, which takes
these GEMMs as FP32 SGEMMs: 8.4K / 32.8K / 104K-token prompts 412 / 455 /
456 tok/s on the old path, 622 / 742 / 750 with #1006, 730 / 900 / 904
with this change (the card's FP32 rate against its FP16 rate), with the
phase times and what the SGEMM route covers that this change does not.
(cherry picked from commit e780c8581e)
The test ran only the FP32-out GEMMs, so the gfx103x prompt path's FP16-out
GEMM (f16_inplace, widen_rows_f16) and bf16()'s W conversion through the
scratch were not under ctest. Every shape now runs a second time with
set_f16_io(true), forced on whatever HIP card runs the test: X as the FP16
image for the BF16 products, a scratch of 64 rows so a 96-row weight converts
in two slices (the second written at Y + 64), ldy > N, an odd N (257), T = 1,
and a beta = 1 f16() call, which keeps the FP32-out GEMM. FP16-out results are
checked against the double reference at FP16's rounding (6e-4 relative per
element plus 1e-4, rel L2 < 1e-3); the row padding and the guards around Y
must be untouched, as before. The FP32-out runs keep their tolerance.
Found while adding them: the BF16 inputs were all zero. hip_bfloat16(float)
produced data 0 in this test's build (ROCm 10.0, gfx1030; not traced further),
so the two BF16 cases passed on nothing (rel L2 printed 0). The BF16 bits are
now made by hand (round to nearest even on the float's bits); with real inputs
the BF16 cases pass at rel L2 1.9e-07 (FP32 out) and 2.2e-04 (FP16 out).
On an RX 6900 XT (gfx1030, ROCm 10.0): 11 of 11 pass.
(cherry picked from commit 4c6c9fe374)
On HIP a prompt batch with fewer than 7,168 blocks (28K cells of context) took the original top-k kernel, a line measured on
gfx1201 before the cooperative scan. With the scan the register kernel is faster from about 15.5K cells on gfx1151 (2,048 queries:
original / register kernel 0.57 / 0.82 ms at 13K, 1.01 / 0.87 at 16K, 2.4 / 0.93 at 20K, 3.9 / 1.00 at 24K). The line is now
STRATA_TOPK_REG_MIN_BLOCKS (default 7168, as before) and gfx1151's table sets 4096. The selected cells are the same
(qsa_topk_parity, also run with the register kernel always on: ctest qsa_topk_parity_reg_always).
Ported from shinbunbun/llama-cpp-p100-patches (MIT), patches 29 `mmvq-iq3xxs-grid-smem` and 30
`mmvq-ksigns-smem`: the grouped expert kernels stage the 256-word iq3xxs_grid and two 128-word tables of per-byte
sign masks once per block, and apply a mask as (g ^ mask) + (mask & 0x01010101) instead of the per-lookup
unpack_ksigns / __vcmpne4 / __vsub4 chain (iq3xxs_grid's bytes are all >= 4, so no carry crosses a byte).
Bitwise the same output: checked end to end on a PH402 (GP100), greedy, three prompts x three interleaved rounds,
identical replies with and without. Measured only on GP100, where it is within noise (+0-4% decode), so it is the
default on compute capability 6.x only; STRATA_IQ_STAGE_GRID18=1 / 0 forces it on / off on any card.
block_topk_reg_kernel picked each radix digit with thread 0 walking up to 255 bins while the other threads waited.
Threads 0-255 now prefix-sum the bins in descending order, and the one where the count above reaches `width` takes the
digit (bin 0 when none of 255..1 does). The digit and the count above are the same, so the selected ids are the same.
Measured on a Ryzen AI Max+ 395 (gfx1151, ROCm 7.14.1), qsa_topk_parity with 2,048 queries (CTX 2048 10 <ctx> 1),
ms per call:
32K 2.19 -> 1.11 (-49%)
64K 2.54 -> 1.63 (-36%)
128K 4.83 -> 3.79 (-22%)
qsa_topk_parity --selftest: 19 PASS. UD-Q4_K_XL end to end, 3 interleaved rounds: the same text in every round; the
128K prompt moved +0.2%, inside the noise, so the gain is the kernel's and not yet visible end to end.
CUDA (RTX 3060, Windows, CUDA 13.0, sm_86), same call, ms: 4K 2.12 -> 0.975, 8K 2.04 -> 1.03, 16K ~1.9 -> ~1.1, 32K ~2.0 -> 1.27, 64K 2.53 -> 2.15, 128K 7.29 -> 7.29 (no change); qsa_topk_parity --selftest passes, ids identical in every case. Not built: SYCL.
The negative-scale-times-zero workaround was gated to an arch list (gfx1012 on
HIP < 7, gfx1151, gfx1200/gfx1201 since 0ff45cffb); the list only ever chased
the newest report. The workaround writes the scale's sign bit for code 1, which
is value-identical on targets whose compiler never folds: keep it on every HIP
build. Ported from PR #1540 onto the tree that already carries the gfx1200/1201
entries; the comment text is adjusted.
Co-authored-by: nekomario28 <[email protected]>
gemm_iq_f16_grouped: one launch per MMQ expert group, a block = 128 weight rows of one
expert x up to 256 of its token rows, each weight superblock dequantized once into shared
memory by dq_dispatch (the FP16 path's formulas), FP16 activations, WMMA with FP32 sums,
the next activation tile prefetched into registers. sm_70 defaults to it for the layers
MMQ takes and for IQ1_M (which rides the MMQ grouping); STRATA_PF_WMMA=0 keeps MMQ.
Co-authored-by: NoxestNoz <[email protected]>
(merge-resolved onto 0.1.44.1: the MMQ gather now zeroes the group tail itself and the quantizer takes a scale buffer)
The pool's default layout puts a worker on every physical core, and a layer's
experts are done when the slowest worker is. On a Core i7-13700K (8 P + 8 E
cores) the E-cores make the verify window wait, by an amount that changes from
run to run.
`--pool-affinity measure` (hybrid CPUs, native packs) times both layouts on
the pack's own experts before the pool is built: two experts of four tokens
per layer through run_split_multi_native, every layer in turn, the host
spinning ~150 us between two layers, three interleaved blocks per layout,
about a second in all. p-cores is taken unless all is more than 10% faster.
One line says what it read and took. It is one reading at start and is not
repeated.
Measured on this branch (0.1.42 + this change), Core i7-13700K, RTX 5090,
64 GB, Windows 11, Swift 1.5 IQ3_XXS, 128K context, one 8K prompt and the
same 1,500 forced tokens (--spec-follow), three interleaved rounds:
default (all, 15 workers) 146.6 / 151.8 / 158.0 tok/s, pool 6.3-7.6 ms per window
--pool-affinity measure 200.0 / 201.5 / 204.8 tok/s, pool 2.1-2.2 ms (read all at 16.4-17.6 ms, took p-cores)
--pool-affinity p-cores 200.3 / 203.9 / 204.9 tok/s, pool 2.1 ms
Over four server starts of one day on 0.1.41 it read all at 14.8, 21.9 and
25.3 ms against 6.1-6.2 ms for p-cores, and once all at 6.6 ms against 8.6 ms,
where it took all.
Without the value nothing new runs: the option parses as before and the
function is not called. The layout changes no result, only where threads run.
Built for CUDA only (sm_120, CUDA 13.0.88, MSVC 14.50 with nvcc's
--allow-unsupported-compiler); HIP and SYCL were not compiled (the change is
in generate.cpp, which both build). Not measured: other hybrid CPUs, Linux.
0.1.43 made the decode window lean under STRATA_VERIFY_NO_HOST with the device
plan (46a77907): the doorbell publish, flag waits and plan copy are dropped
because no host answers them. capture_batch() records batch windows through the
same record_window(), so a batch graph was also recorded lean, but the host batch
path (run_slot_rows / batch_poll) still waits for each layer's doorbell unless the
stage is all-resident. On a stage with mirrored experts the graph finished with
the ring silent: "verify batch: layer 0 never rang (graph finished)", then
UR_RESULT_ERROR_UNINITIALIZED and exit 139 at the first shared window.
Leave lean off while recording a batch window (batch_rec_). The solo window
stays lean. Batch tokens are identical to a run with STRATA_VERIFY_LEAN=00.
With the default --suffix-draft 3 the suffix block above already raises --spec by 2
(4 -> 6) and writes mtp_max_t = 4. The --lookup-chain block then computed
min(mtp_max_t + K, 8) = min(4 + K, 8), so with --spec 4:
K=1 -> max(6, 5) = 6 (unchanged: the flag does nothing)
K=2 -> max(6, 6) = 6 (unchanged: the flag does nothing)
K=3 -> max(6, 7) = 7
Measured on 0.1.40 by reading the cap back from /metrics engine.spec after a restart
per row; K=1 and K=2 were indistinguishable from not passing the flag.
Size from the cap actually in force instead, so K adds up to K of headroom while the
kVerifyMaxT = 8 ceiling and the mtp_max_t bound on the MTP's own drafts are preserved.
Behavior is unchanged when --suffix-draft is 0 or when the cap is already at 8.
Under WSL2 the NVIDIA driver caps what cudaHostAlloc(Mapped) can pin at about
1 GiB in all, so the KV streaming host copy (layer.cpp) cannot be allocated
and setup turns streaming off there: the whole KV cache must sit in VRAM,
beside the expert cache. cudaHostRegister of ordinary memory is not capped the
same way; the expert arena already registers tens of GiB with it on the same
machines (pinned.cu).
kv_host_alloc() maps anonymous memory and registers it Portable|Mapped, before
any page is touched, when /dev/dxg exists (WSL2, detected as generate.cpp's
under_wddm() does). Elsewhere cudaHostAlloc stays first. Either way the other
method is the fallback when the first is refused, and
STRATA_KV_HOST_REGISTER=1/0 forces the order. On native Linux and Windows the
behaviour is unchanged unless cudaHostAlloc is refused (by inspection; not run
there). The failure message no longer says WSL pins only 1 GiB.
Measured on one RTX 5090, Ryzen 9 9950X3D, WSL2 (157 GiB), Windows driver
591.86, IQ3_S, --max-context 1048576 --rope-scaling yarn --rope-scale 4
--kv int8 --kv-resident 32768, with this commit:
- start: "the K/V in 12.38 GiB of pinned RAM"; expert cache 11,609 slots
- tools/needle_bench.py: 4/4 found, 128K and 1M (1,015,070 prompt tokens) at
depths 10 and 90
- decode 99-124 tok/s with 0.5-1M tokens in context (4 runs); 150-191 tok/s
medians from 55-token prompts to 128K (single runs 113-196); prefill about
4,200 tok/s for a cold 1M prompt
kv_stream_parity still passes; it allocates its own host pools, so it checks
the streaming kernels, which this change does not touch.
Not tested: layer split or remote experts under WSL, Docker Desktop.
setup.py still never writes --kv-resident on WSL and removes it there
(upgrade_config); until that changes it has to be added by hand.
Port of PR #1498 (commit 79b43be18).
pin_cache_complement set lend_from_slot = -1 whenever --resident-budget-gib was given, so the experts of the cache
slots the prompt path borrows were in neither VRAM nor RAM during a prompt and were read from the model files on every
chunk, and again when the slots were refilled - even with room left in the budget. The budget branch now appends the
lend region's experts after the ranked ones, from the last slot down (the order the borrowing takes them), while they
fit; the budget stays a hard cap, and the ones past it keep the file fallback. The start-up line reports the count, and
the lend warning names a larger budget as the remedy when one is set.
Measured on an RX 6900 XT 16 GB (HIP gfx1030), Ryzen 9 5950X, 125 GB RAM, UD-Q4_K_XL, --resident-budget-gib 71 (the
copy held all 63.47 GiB outside the cache, 7.5 GiB unused; 1,231 lent slots = 3.60 GiB), engine 0.1.42, one run per
case, greedy: file reads of the lent experts per request 0.5-2.6 GB -> 0; SSD reads per fresh 8K / 16K prompt 2.8 /
1.9 GB -> 0.3 GB; prompt-path host staging 0.8-2.1 s -> 0.06-0.5 s per request; fresh prompts 887 / 7.8K / 15.8K
tokens 180 -> 194, 633 -> 676, 677 -> 698 tok/s. Decode unchanged.
Port of PR #1896 (commit 29063eb2b).
Rotation hands the evicted expert's exchange buffer over as its place in the
RAM copy instead of copying the bytes there. It required one block size for
all layers and fell back to the copy path on any other pack: Swift 1.5 IQ3_S
has seven sizes (1,510,400 to 2,662,400 bytes) and IQ3_XXS nine.
A swap exchanges two experts of one layer, so the slot the promoted expert
leaves has the size the evicted one needs. ExchangeStorage now keeps its
spares per block size (initialize_sized): an exchange takes a spare of its
layer's size and hands back one of the same size. commit() refuses a spare of
another size and an expert of another layer. initialize() keeps the uniform
geometry and its checks. exchange_buffer() takes the layer; reserve_exchanges()
takes the number of trailing buffers that only stage copies (--adapt-async's
second half), kept in the largest size alone. With --adapt-async 1 a rotated
round flips in the tick that sees the copies in: there is nothing for the job
to move, so the round ends one verify window sooner.
Still opt-in (STRATA_EXCHANGE_ROTATE=1); unset, nothing changes. The exchange
allocation grows to one spare of every size per swap of a round: 1.06 GiB of
page-locked RAM on IQ3_S with --adapt-swaps 80 (1.26 GiB with --adapt-async 1)
against 0.20 and 0.40 GiB.
Measured on a Core i7-13700K, RTX 5090 32 GB, 64 GB, Windows 11, Swift 1.5
IQ3_S, 128K, --resident-experts, --adapt-every 1 --adapt-swaps 80, p-cores:
one 8K prompt, the same 1,500 forced tokens, blocking tier, rotation off / on:
161.6-165.1 tok/s (five runs) / 184.9-188.7 (four runs); two runs of each
arm with the same cache size (10,681 slots) wrote identical --window-hashes
for all 434 windows
--serve, 23 coding-agent turns, greedy, three interleaved rounds, blocking tier:
147.8 / 159.2 / 158.4 (the build before) -> 175.2 / 184.9 / 172.8
the same turns with --adapt-async 1, one binary, two rounds:
166.0 / 170.1 -> 185.6 / 182.5
The tier swaps about 30 experts per verify window on this box (44% of the
experts fit in VRAM), so the copy path moved about 50 MiB of RAM per window.
tests/core/exchange_storage_test.cpp gains a three-size case (6,144 exchanges)
and passes built alone with MSVC 14.50. Built for CUDA only (sm_120, CUDA
13.0.88, MSVC 14.50 with nvcc's --allow-unsupported-compiler); HIP and SYCL
were not compiled (expert_source.cpp and generate.cpp are built by both). Not
measured: Linux, AMD cards, a layer split.
Port of PR #2042 (commit 8db375018).
The multi kernel reads the dense IQ3_S matrices (attn_q, attn_gate, ssm_out, attn_k, the shared experts' gate/up)
at 76-114 GB/s on the B70. The new kernel has the shape of the Q4_K..IQ4_XS wide kernels. unitrace, one card,
IQ3_XXS, T=1: worst attn_k (512 rows) 77 -> 123 GB/s, best attn_q (12288 rows) 114 -> 307 GB/s; these matrices take
0.60 ms of kernel time per token instead of 1.36 ms.
Unset or 0: the multi kernel, output identical to main. With the option the greedy output parts from the default at
token 198 of 256 (the sub-blocks are summed in another order).
Since 0.1.40.2 the waves_per_eu(8) variant of native_w11_kernel is uncapped on clang 22 and newer, because clang 22
computes wrong results with it on gfx1100. On gfx1151 with clang 23 (ROCm 7.14.1) the capped kernels give the same
bits and take less time, as alexdns1 measured in #1180. The cap now applies there again, per format
(hip_prefill_fused_iq, one layer, 2048 / 3584 / 8192 tokens, the same output bits hash in every case):
- gate/up IQ2_XXS / IQ3_XXS / IQ3_S / IQ4_XS and down Q2_0 / IQ4_NL: capped, 8-15% less time per layer.
IQ2_S gate/up is capped too, but its LDS leaves room for 3 blocks per WGP and the compiler keeps its 238 VGPRs;
IQ2_S / Q2_0 takes 2-4% less time from the Q2_0 down.
- IQ2_XS gate/up: not capped. Under the cap it spills 104 VGPRs and takes 61-66% more time.
- Q8_0 down: not capped. It spills 18 VGPRs and took -1.9% to +1.9% time (with Q4_K gate/up).
The test is the clang major version: other clang 23 builds are not measured, and clang 24 and newer keep the clang 22
rule. The template argument is the same in the host and every device pass; only the device pass for gfx1151 decides
the attribute (nw_capped). Other architectures, other compilers and -DSTRATA_W_LB=N behave as before: the gfx1100
disassembly is identical to before (kernel names aside). The file builds for gfx1012 / 1030 / 1100 / 1150 / 1151 /
1200 / 1201 and with STRATA_HIP_GFX906 (gfx906); no CUDA build here, and the changed lines are inside
`#if defined(__HIPCC__)`. SYCL has its own copy of this file, not touched. The path is opt-in (STRATA_PF_FUSED=1), so
default output is unchanged.
End to end on a Ryzen AI Max+ 395 (128 GB, gfx1151), UD-IQ4_XS, the section-5 switches of docs/STRIX_HALO.md with
--prefill 16384, 3 interleaved rounds of fresh servers: a 128K prompt read 1,126.1 -> 1,153.4 tok/s (+2.4%; +1.8%,
+2.4%, +2.6% per round). At 4K (868.0 -> 894.9, +3.1%) and 32K (1,182.9 -> 1,207.0, +2.0%) the medians moved too, but
within the spread of 3 runs. Decode unchanged; the same text for all 9 prompts.
prompt_attn_wmma_kernel interleaved a chunk's global reads with its
compute: the V slice was staged first, then each 16-cell tile loaded its K
scale and fragments between the q.k WMMAs, so a wave waited on each read
in turn (a thread trace on gfx1151: 56% of the kernel's time in
s_waitcnt vmcnt, 0.4% in WMMA). The chunk's reads now go out together
(fetch), then the same compute runs on them (chunk). The values and the
order of the arithmetic are unchanged.
gfx1151, ROCm 7.14.1 SDK (clang 23), hip_prompt_attn_wmma with
STRATA_PA_FAST=1, three interleaved runs against the unchanged kernel,
ms per chunk: 32K context / 2,048 queries 12.50 -> 11.29 (-9.7%),
1,500 / 1,500 3.39 -> 3.20, 2,100 / 256 1.30 -> 1.23. The output hashes
(NEWHASH) match with STRATA_PA_FAST=1 and =0. gfx12 compiles the same
change; it was not run there.
The wide kernels (Q6_K a2 and the generic native_mmvq_wide) spill at 2-8 columns with 128
registers per thread on the A770. They now launch with grf_size<256> there. Large GRF halves the
threads resident per Xe core, so it is the default on Alchemist only (strata::intel_gpu_gen);
Battlemage and other devices, and 1 column on every device, launch the same kernels as before.
STRATA_MMVQ_LARGE_GRF=1 forces it on, =0 off.
mmvq_bench now prints a checksum of the output (FNV-1a over the float bits) so the two modes can
be compared bit for bit.
Measured on an Arc A770 16 GB (xe, oneAPI 2026.1 JIT), same day. mmvq_bench Q6_K, 2560x2560 / 2560x10240 /
10240x2560, 1-8 columns: 1 column unchanged; 2 columns 1.3-2.0x, 4 columns 2.3-3.1x, 6 columns 3.0-5.0x, 8 columns
3.7-7.4x faster (e.g. 2560x10240 at 4 columns 289.3 -> 92.9 us, 74 -> 232 GB/s of weights); checksums identical
with STRATA_MMVQ_LARGE_GRF=0 and =1 in all 15 cases. End to end, a0116b27 + #2190 + #2193 + #2203 with and
without this commit, Coder IQ1_M, 128K context, int8 KV, --resident-experts, STRATA_VERIFY_STEPPED=1, 8K prompt,
256 tokens: decode 23.47 / 23.50 / 23.73 / 23.77 -> 25.99 / 27.10 / 27.49 tok/s (+10-17%), prompt unchanged,
output ids identical where the expert placement matched. Not built or run on Battlemage.
With --batch N the verify window ran QSA block scoring, top-k and decode
attention as one launch per row, and the GDN recurrence (window and 1-row
commit) as one launch per slot. At 4-8 live slots these serial launches are
most of the window's cost beyond the expert reads.
Two env switches, both off by default (off = the same launches as before):
STRATA_BATCH_ATTN_ROWS=1 block scores, top-k and decode attention for all
rows of the window in one launch each; every row
reads its own slot's indexer keys and K/V pools
(passed by value as kernel params, so the graph
captured per row layout stays valid). A slot that
streams KV (kv_mode 1) keeps the per-row path.
STRATA_BATCH_GDN_GROUPS=1 GDN recurrence in the window and the 1-row commit
for all slots in one launch (grid.y = slot group).
Conv commits stay per slot.
Same arithmetic per row: 4 concurrent temperature-0 prompts give the same
bytes with and without the switches (--batch 4 and --batch 8).
RTX PRO 6000 Blackwell, IQ3_S, all experts in VRAM, a0116b2, same day,
total tok/s (docs/BATCHING.md has the table):
--batch 4, 4 clients: 268.4 -> 311.7
--batch 8, 8 clients: 308.4 -> 374.8 (reserve 6144 MiB)
Window: 4 rows 14.3 -> 12.2 ms, 8 rows 24.6 -> 19.9 ms.
Built and run with CUDA 13.0 (sm_120) only. The .cu files are also in the
HIP build, which was not built here (no ROCm). The SYCL port has its own
copies of these files and is untouched.
Q4_K, Q5_K, Q6_K e IQ4_XS: il frammento di attivazione di ogni colonna si carica una volta per blocco
di pesi e serve tutte le righe del blocco; le scale Q6_K stanno in registro. Stesso ordine di accumulo e
di riduzione: uscite identiche bit per bit al layout esatto e alla chiamata a colonna singola
(b6_mmvq_bench, memcmp su entrambe le schede). Default spento.
Stacked on #1415. IQ2_S (type 22, 122 gathered) joins STRATA_CPU_IQ_ALL_EXACT1's list: ggml's AVX2
iq2_s dot sums the same int32 lanes as row_dot_ggml_one and ends with 0.125f * hsum_float_8.
The singleton gate now also requires nt < native_gu_mt_min, so with STRATA_IQ_MT_MIN=1 a lone token
keeps the multi-token kernel and native_expert_parity's width invariance passes with the switch on.
- Q6_K wide kernel (and the planar one): the six-bit offset (v - 32) was dpct's per-byte saturating char4 subtract, a per-byte loop on Xe;
no byte can saturate, so a 4-op SWAR form is bit-exact. Q6_K dense matvecs 59.5 -> 30.8 us (1 row), 77.4 -> 45.5 us (4 rows).
- STRATA_Q6_PLANAR (default on, =0 off): the Q6_K LM head and the MTP draft head are converted at load to ql/qh/scales/d planes (same bytes, the
GGUF-layout allocation is freed: no VRAM change) so a lane's loads are single aligned 16-byte loads; the draft head subset is gathered from the planar
head. native_q6_k_mmvq serves every registered matrix at any column count. Bitwise equal outputs.
- ld_i4_a4: the Q8_1 activation qs (4-byte aligned) as one dword-vector load in the Q4_K / Q5_K / IQ4_XS wide kernels (Q5_K 4 rows 81 -> 68 us).
With STRATA_QSA_SPLIT_ATTN=1 the verify window's and the draft layer's QSA attention run through the prompt path's int8
tensor-core kernel with each query's cells split among up to 20 blocks, and the splits are merged with the log-sum-exp
rescale. qsa_decode_attn_split() returns false (nothing launched) unless the KV is int8 on sm_80 or newer, or when its
partials would not fit the caller's scratch, and on HIP; the caller then keeps qsa_decode_attn_batch(). The sums run in
another order, so with the switch on the output is a rounding-level change. Unset, nothing changes.
The idea is from Eddoursul's fork (fd23d91); the kernel is written against this tree's qsa_prompt_attn.cu.
STRATA_SIDE_MIXER=1 runs the GDN gates and the QSA indexer (key projection, tail snapshot, query chain, block scores,
top-k) on a branch of the window's graph beside the attention's projections, joined before the recurrence and the
attention; =2 also runs the GDN gate projection and the QSA queries' chain on a second branch. A branch that projects
reads F4's interleaved copy, written on the main stream before the fork, and qkv is recorded before the branch so its grid
is dispatched first. Every result is bitwise the same as in line; STRATA_DF_BRANCH is off while this is on. Unset, the
window runs in line as before.
The idea is from Eddoursul's fork (34a35db, b063c13).
native_expert_grouped already decodes each weight part once per block, but the apply re-reads every entry's q8_1 block from global memory with 9 scattered 4-byte loads, so an extra row of the window costs ~60% of a whole expert. expdec_gu_kernel stages the group's entries in shared memory (two conflict-free 16-byte loads per call, the fp16 scale as a float), takes rb rows per block and stages the IQ3 codebooks too; the same lane does the same calls in the same order with the same operations (apply_u = apply on the block's words), so the output is bitwise the default kernels' (expdec_bench: identical on every cell, 2 cards). Gate/up IQ2_XXS/IQ2_XS/IQ3_XXS/IQ3_S/IQ2_S at n_embd 2560; the down (Q2_0, IQ4_NL) behind STRATA_EXPDEC_DOWN=1 (in the engine the Q2_0 down was ~10% slower). Knobs STRATA_EXPDEC_R/_DR/_MINB/_RB/_DRB/_MINBLK/_D20; one log line per (role, format) at the first launch. docs/EXPERTDEC.md.
q5_1_dot and q8_0_dot in kq_avx2.cpp held their per-token sums in the same
runtime-indexed __m256 acc[MAXT] that made bf16_rows_dot_multi spill: with the
array live the compiler leaves it in memory, and every token's FMA reads and
writes a stack slot. Both are now template <int NT>, dispatched 1..4 and, past
that, run four tokens per pass with the row's bytes re-read from L1 - the same
idiom the router change above and kq256_gu_rows already use.
Ryzen 7 9800X3D (Zen 5), GCC 15.2.0, build/native_expert_parity --synthetic
(ggml-quantized weights and activations), one thread pinned to core 0, at the
verify window's four tokens:
Q5_1 down rows [2560 x 640]: 190-197 us -> 155-156 us (1.24x)
Q8_0 down rows [2560 x 640]: 135 us -> 104 us (1.30x)
Same sums: 0 of 36 token-sets differ in any bit against ggml's own vec_dot over
1..8 tokens, both before and after; native_expert_parity reports 0 failures.
Left alone: q4k_dot keeps its runtime-nt array. Its token accumulators are not
what spills - the unpatched function carries 8 ymm stack ops for the sums
against 24 (q5_1) and 59 (q8_0); what q4k_dot spills is its weight-side unpack
arrays (q4l/q4h/scale_l/scale_h), which token chunking does not touch and
re-does. The same change measured +5% at four tokens and -4..-7% at six to
eight, so it is a wash at the window and a small regression above it, and q4k is
the gate/up format that runs most. The AVX1 path, the iq_avx2/q2_avx2 rows and
the engine's dispatch are untouched.
Backends: CUDA and HIP do not compile this file. The SYCL port does; not built
on this machine (no icpx here) - the router change in this branch built there.
Tests: native_expert_parity --synthetic q4_K/q5_1 and q4_K/q8_0 (0 failures,
0 bits differ), iq_avx2_parity, iq_avx1_parity, q8k_quant_parity,
router_dot_parity - all exit 0.
bf16_rows_dot_multi held its per-token sums in __m256 acc[MAXT] indexed by a
variable. With eight 256-bit accumulators live GCC also unrolled the column loop
and left the whole array in memory, so every token's FMA read and wrote a stack
slot (vmovaps 0xc0(%rsp) ... vfmadd213ps 0xe0(%rsp)). On a Ryzen 7 5700X3D one
layer's router [512 x 2560] at 4 tokens took 0.41 ms against 0.25 ms for the AVX1
copy in kq_avx1.cpp that kept its accumulators in registers, so every AVX2 CPU ran
the slower of its two kernels (router_dot_parity --cpu 2, median of 3). Pragma
GCC unroll 1 on the column loop changes nothing; the variable-indexed array is
what forces the array into memory.
Four tokens per pass, in four named accumulators. A row above four tokens is read
a second time, from L1: nt=8 measures 0.37 ms, the same as the spilled form. On
the same CPU the AVX2 kernel goes 0.41 -> 0.19 ms at 4 tokens and nt=2 0.36 ->
0.16 ms, and it now runs 1.3x ahead of the AVX1 copy instead of 1.6x behind it.
The loop body carries no stack slot any more (was 21 memory ops per iteration).
Untouched: the arithmetic and the reduction. Each token's FMA chain still runs
over the same columns in the same order and ends in the same hsum_float_8, so
every sum is bit-identical - the output for nt 1/2/4/8 at cols 2560/5120 hashes
the same before and after (FNV-1a over the whole out[]), and router_dot_parity
reports 0 failures with the AVX2 rel error unchanged at 3.16e-07.
Left alone: the expert row kernels (iq_avx2.cpp, q2_avx2.cpp, kq256_gu_rows,
kq256_rows) and the AVX1 path. The iq_avx2_parity bench over four format pairs x
1..8 tokens moved within 1.6% (noise), and the CPU parity tests (8 targets) pass.
The router stays what it was, an estimate for which experts to prefetch. CUDA and
HIP do not compile this file; the SYCL port does, and it builds there under icpx
2026.1 with no new warnings and no spills either.
The AVX1 copy in kq_avx1.cpp kept a comment saying it reads its rows "exactly as the AVX2
kernel does"; with the row loop chunked four tokens at a time that is no longer the
AVX2 kernel's shape, so the clause is dropped there. Comment only - no code change.
attn_chunk_kernel_pre75 (PR #540 by sskver: two cells per step with a 12-shuffle reduction, cell-major
probabilities, V loaded four at a time) is compiled in every CUDA build, not only with STRATA_EXPERIMENTAL_SM60, and
STRATA_ATTN_DAILY=1 selects it on any card for the decode attention (single query and batch: verify window and MTP
drafter) together with attn_merge_v2_kernel (Hardin22), a merge that skips the chunks the chunk kernel marks empty.
Bit for bit with the default kernels by construction: reduce12 adds the same pairs as warp_sum's butterfly at every
level, the V cells fold in the same order with the same fmaf, the masked cells stay out; the merge does the same fmaf
in the same order. Shared memory 15,872 B like the default kernel (spt shares sq's storage). STRATA_ATTN_MERGE_V2=0/1
and STRATA_ATTN_PRE75=0 switch the two halves separately. HIP unchanged.
qsa_merge_daily_parity (tests): memcmp of both kernels against the default ones on four KV formats, widths at the
chunk borders and non-resident pages; --bench times the chunk kernels back to back.
In a 131k decode block_scores_multi_kernel costs ~20 us more per call for every row of the verify window: not the keys (read once for all the queries) but the shared-memory and shuffle pipes, 4 x LDS.128 of the query slices and 20 shuffles per (block, query). block_scores_rows_kernel<NQ> keeps the NQ <= 4 queries' slices in registers and reduces the four heads' lane products with a scatter that keeps warp_sum's pairings (xor 16 and 8 leave one head per lane, then 4, 2, 1): every addition adds the same two operands of the same level, so scores and top-k ids are bitwise the multi kernel's. qsa_rows_bench measures the window's QSA kernels row by row and, with --rows, checks the parity (16 cases per card, 0 differ) and prints both kernels' microseconds: 2080 Ti 47/57/78/101 -> 36/38/49/55 us at T 1-4, 3060 73/109/152/194 -> 63/76/98/130. docs/QSAROWS.md.
justxiami reported (#1098) a 4xV100 layer split dying with
STRATA_GDN_CHUNK=1. Reproduced on 2xV100: the second card launched the
chunked kernels at the 48 KB default (a process-global std::call_once set
the opt-in on the first card only) and the engine exited with
"gdn_chunk wmma: invalid argument".
Two more per-process assumptions in the same path:
- gdn_chunk_cc() cached the first card's capability for the whole process
- the chunked workspace was thread_local with no destructor: the per-request
stage threads leaked ~50 MB per card per request (the second card ran out
of memory on the next prompt) and one card's pointers could reach another
Now: the opt-in and the capability are recorded per device (each attempt
clears the runtime error, and a refused wmma opt-in falls back to the FMA
path), and the workspace is one per (thread, device) with a destructor.
Verified on 2xV100 32K: STRATA_GDN_CHUNK=1/2 run to completion over repeated
rounds (1: 1900 tok/s prefill, 2: 1766), gdn_chunk_parity 0 failures.
CUDA events over one gdn_recurrence call per side (S=128, 16 q/k heads, 48 v heads, C=10240,
T=1024/4096/8192), all four paths and the max abs diff between the chunked wmma output and the
pipelined recurrence. RESULTS.md records the table and the error bounds for
bench/results/2026-10-02-gdn-chunk (measured on the V100-SXM2-16GB, CUDA 12.8, sm_70 build).
GPU, synthetic, no model. Compares through the real dispatch (STRATA_GDN_CHUNK /
STRATA_GDN_REC_WARP A/B) at T=64/128/200/320 - 200 exercises the < 64-token tail - and asserts
measured bounds, not equality: the recurrence is the FP32 floor (y abs 1.1e-6, state 3.0e-8), the
chain-split and the chunked FMA path are FP32-level (~3e-6), the wmma chunked path is the
f32->f16 staging level (y abs 2.3e-3, state 8.9e-5). The sync-free warp recurrence is asserted
BIT-EXACT against gdn_rec_cols_pipe (memcmp of y and the final state). A NaN or inf anywhere
fails the check outright - an unordered comparison would otherwise leave the max at zero.
On main's 0.1.38 recurrence the out norm keeps its raw FP32 output in place and writes only the
FP16 copy, so the kernel comparisons use that raw output (ref_full's yraw_out) and the norm is
checked once through its FP16 store.
The prompt GDN recurrence is a per-token chain; on a V100 that chain is what the 36 GDN layers
cost. This adds the chunked form (cumsum / kkt / fwd over 64-token chunks, chunk-serial scan,
the state in registers - the gdn-chunk-sm70 structure of llama.cpp-v100, hand-written CUDA):
a chunk is a triangular solve and a batch of matmuls instead of 64 sequential rank-1 steps.
Two variants: an FP32-only FMA scan and an m16n16k16 wmma one with the reference's f32->f16
staging. Both are FP32-level, NOT bit-exact (the sums are reordered; the wmma path stages
q/k/v and the state through f16) - gdn_chunk_parity bounds them.
Also two Volta variants of the recurrence itself: a sync-free warp-shuffle reduction (BIT-EXACT
vs gdn_rec_cols_pipe, but measured 2.6x slower - kept as the bitwise control) and a 4-accumulator
chain split of the pipelined kernel (FP32-level, measured +6-7% at T=4096/8192, the V100 default
now). All of this is compiled only into the STRATA_EXPERIMENTAL_SM60 build and dispatched only
at cc==7.0; sm_80 and up keep the trunk's kernel bit-for-bit. The env switches
(STRATA_GDN_CHUNK=1/2 chunked, STRATA_GDN_REC_WARP=0/1/2) still override.
Measured on the V100 (bench/gdn_chunk_bench, real shapes, one gdn_recurrence call): the chunked
path is correct but 4-6x SLOWER than the recurrence as implemented (4-warp CTAs underfeed the
tensor cores - see bench/results/2026-10-02-gdn-chunk/RESULTS.md), so it stays opt-in.
Fusions (all bit-exact vs the pipelines they replace; bench/micro/fusions_parity.cu
checks every one bitwise and times the before/after at decode shapes):
- gate/up dual GEMV + SwiGLU in ONE kernel (llama.cpp's mul_mat_gated):
s_gemv_pair_silu, one read of the activation for both projections and the
SwiGLU in the epilogue - the up intermediate never lands in memory. Rows are
accumulated by the shared row functions (s_rowdev.cuh) exactly as the
standalone kernels accumulate them. shared_expert uses it whenever both
projections are S-family (bitwise checked for all three silu kinds).
- silu(gate)*up + the intermediate quantizer fused (was 2 kernels):
swilu_quantize_q8_0/_scaled/_q8_K, warp-parallel swilu then the verbatim
quantizer body. Wired into the S2 expert entry points and shared_expert.
- every activation image of x in one pass (was 2-3 kernels): quantize_act_images
produces block_q8_0 + block_q8_K + bf16 from one read of x. Wired into
gdn_layer, qsa_layer and moe_shared.
- multi-weight bf16 GEMV (llama.cpp's mul_mat_multi): bf16_gemv_pair computes
two matrices from one read of x; used for ssm_alpha/ssm_beta and for the
indexer's q/k projections (q and k now one launch).
The call sites are gated on STRATA_EXPERIMENTAL_SM60 (the Pascal/Volta build),
so the ready-made engine runs exactly the pipelines it did before; the fused
kernels are the same code either way.
Measured (V100, CUDA events, n_embd 2560 / FF 640, 200 reps):
expert intermediate (swilu+q8_0) 15.7 -> 11.6 us M=1, 18.8 -> 11.4 M=8
activation images (q8_0+q8_K+bf16) 46.2 -> 30.4 us
dual GEMV + swilu (gate,up) 30.4 -> 14.8 us
bf16 pair (alpha,beta) 24.2 -> 14.4 us M=1, 23.7 -> 14.1 M=8
Parity: fusions_parity 0 failures (bitwise), plus s_gemv/s_gemv_q8k/s2_gemv_q8/
s2_expert_grouped/shared_expert/quantize_act/bf16_gemv/elementwise/mmvq_multi/
iq_multi/native_expert parity suites all pass unchanged.
Port of oldlocal commit 4672cf9 ("kernels (sm_70): the prompt attention's
latency pipeline - 2.2x over the FP32 kernel at 32K") onto the #600
prompt_attn_v70_kernel, the m8n8k4 Volta kernel now inline in
qsa_prompt_attn.cu. Still gated on STRATA_EXPERIMENTAL_SM60, so the
ready-made engine keeps exactly the code it had.
Two of the commit's three changes port; the third (splitting the dependent
WMMA chains into f0/f1 and fh/fl) is already inherent to the mma884 kernel -
its scores run two tg[rb] chains and its p.v four tmp[rb][nt] chains - so
porting it would only change the summation order. The two ported, measured
on a V100-SXM2-16GB (CUDA 12.8):
* the staging loop issues every global load of a thread before any of its
stores (2-4 in flight), instead of paying the HBM latency per piece;
* the next chunk's ids/page-table lookups double-buffer (row[2][CH]) and run
just before the previous p.v, where their latency hides.
The original commit measured, on the #371 kernel, 2,048 queries of a 32K
selection: 34.4 -> 15.8 ms (2.18x) int8, 33.5 -> 15.4 ms (2.17x) fp16 KV,
35.6 -> 20.3 ms (1.76x) K8V4 (its q4_0 V dequantizes to FP16 at staging, the
sm_75/80 kernel's same trade).
This port, same 2,048 queries of a 32K selection (qsa_prompt_attn_sm70_bench,
which also gained the K8V4 pools): the Volta kernel goes int8 12.11 -> 10.25 ms,
fp16 KV 25.70 -> 17.39 ms, K8V4 25.71 -> 24.12 ms, so against the FP32 decode
kernel int8 rises 1.73x -> 2.05x and fp16 0.80x -> 1.18x (K8V4, 0.85x -> 0.91x,
keeps main's #600 dequant-to-FP16 trade).
qsa_prompt_attn_parity passes all four cases. An A/B against the unpatched
kernel shows the kernel's output is unchanged - every FP64 and new-vs-old
figure is identical before and after (int8 5.08e-6, fp16 5.29e-6 vs FP64,
within the 4x gate).
The experimental build (-DSTRATA_EXPERIMENTAL_SM60=ON, Pascal/Volta) gains a
kv_gather_q8_wide_kernel: one uint2 (8 B) load of codes, one uint4 (16 B) store of
the dequantized fp16 and one scale per thread instead of two. The 8 values lie in
one 64-value group and the arithmetic is unchanged ((float) code * scale ->
f16_from_f32), so the output is byte-identical. The ready-made engine keeps exactly
the 4-wide kv_gather_q8_kernel and launches it unchanged, so its gather is bit for
bit the release engine's. On a device below sm_75 the wide kernel is selected at
run time inside the experimental build (STRATA_Q8_GATHER_WIDE=0 opts out).
Measured on 2x Tesla V100-SXM2-16GB (SM 7.0), CUDA 12.8, one card: +11.4% on the
32k gather microbench (15.85 -> 14.05 us/call, 401 -> 452 GB/s). kv_q8_parity
stays OK at worst INT8-vs-FP16 error 0.590 quantization steps.
New requests took the control lines in whatever order the lock fell, so a 500-token chat question could wait
behind several agent reads. With the env, a prompt of at most that many tokens goes first (arrival order within
each class); a longer one that waited STRATA_ADMIT_MAX_WAIT_S (30 s) counts as short. A request already in
acquire() gives the lines back when a shorter one came. Only the order changes, never what is read.
Ported from havok2-htwo/Strata 08830ddf onto 0.1.44.1: _take_control has no image-request line here, so its
gate is only the ordering; the first admission passes new=True.
The refusal 'this CPU cannot run the expert kernel' belongs to the canonical Q2_0 pack. An NVFP4, i-quant or
Unsloth pack is a native pack (native_experts.txt in the --pack folder) and its rows have AVX2 kernels, but a
config whose --pack points at the wrong folder reads like a CPU that is too old. On a CPU without AVX-512 the
engine now says the file is missing, and where it sits when it is one level down (<pack>/pack).
A layer split hands each prompt chunk's rows (21 MB per 512 tokens) to the next card through a pinned
host buffer: one D2H, a host stream sync, a new std::async thread, then one H2D before the next stage's
first kernel. The 3090s here have no P2P (topo: CNS) and cudaMemcpyPeerAsync is driver-staged and
slower (3.4 ms vs 3.2 ms for D2H + H2D), so the copy stays on the host route; nsys showed 3.8-4.0 ms per
512-row hop, of which ~1-1.4 ms was the thread start alone.
With STRATA_HANDOFF_PIPE=P (2..16, opt-in; unset = unchanged):
- the sending stage records an event when its chunk's work is done and one after each of P row-piece
downloads into the same pinned buffer, and waits on the host only for the first (on_stage_chunk's
checkpoint parts and the error check need the state, not the copy);
- the next stage waits on the GPU for each piece's event (cross-device cudaStreamWaitEvent) before that
piece's upload, so its uploads overlap the remaining downloads (separate PCIe links); a smaller
receiving chunk waits for every piece, then copies as before;
- the next stage's chunks run on one worker thread per sending stage, started and pinned once, whose
packaged_task futures carry the same results and exceptions as std::async's (it clears the CUDA error
state after each job, as a thread's end did). STRATA_HANDOFF_PIPE_WORKER=0 keeps std::async (A/B).
Same bytes in the same rows before the next stage's first kernel: exact. No device memory (events only,
the existing two pinned buffers per stage). Measured (T16-143l5, P=8): 512-row hops 3.8-4.0 -> 1.85-1.89
ms, 143-row hops 1.6 -> 1.0 ms; reads 1.2K about -1.5 %, 150 about -1 %, 9K (agent, --kv-resident) -3 %;
decode rows/s unchanged. batch_test16/32 (greedy and temperature 0.7), batch_interleave_test and
batch_overlap_test (+ the stage fault arm) byte-/token-identical to the reference dumps; an agent-config
run with checkpoint parts byte-identical to the reference binary.
STRATA_SCORES_MULTI=0, the pre-sm_75 attention (STRATA_EXPERIMENTAL_SM60 builds on a
V100/P100) and the gfx906 query swizzle make qsa_block_scores_slots or
qsa_decode_attn_slots decline; the engine keeps its per-row calls there, so a decline
is no failure. A declined case prints SKIP, the test exits 77 when every case was
declined, and STRATA_SCORES_MULTI=0 skips up front.
In a batch window every row reads its own slot's indexer keys and K/V, so each QSA layer ran the
selection and attention once per row: four rows (16 slots) or eight (32 slots) times qsa_block_scores,
qsa_block_topk and qsa_decode_attn_batch, each launch small (the top-k on one CTA, the attention on
cap/64 x 2 blocks). With the switch, per window (up to 8 rows per launch):
- block_scores_slots_kernel: grid (256, rows), blockIdx.y runs the single-query block_scores_multi_kernel
body with that row's slot keys (block_scores_multi_kernel is now a wrapper around the same body)
- qsa_block_topk over the rows in one call (it only reads the rows' scores and steps)
- attn_chunk_slots_kernel: attn_chunk_kernel's body with the row's own QsaAttnPools (an 8-entry table
passed by value, blockIdx.z picks), then the existing batched merge
Each row is computed by the same kernel body with the same reduction order as its own call. The
launchers decline (the per-row calls run) where that call would pick another kernel: STRATA_SCORES_MULTI=0,
the pre-sm_75 / gfx103x attention, the gfx906 query swizzle. A streamed KV cache keeps the per-row
resolve-then-attend loop. Default (switch off): unchanged.
Checked on 4x RTX 3090, T16-143gsc setup (layer split 12,24,37, int8 KV):
- SASS (sm_86) of every existing kernel in qsa_select.cu and of the default attn_chunk_kernel<M> and the
merge is identical before/after; the opt-in lane-cell instantiations differ only in register allocation
- new qsa_slots_parity (ctest, also with STRATA_ATTN_LANECELL=1): fp16, int8, q4_0 and int8-K/q4_0-V
pools, 1/2/4/8 rows of different slots with their own page tables and contexts 37..32768: scores, ids
and attention bitwise identical to one call per row
- batch_test16/32 (greedy and t=0.7), batch_interleave_test and batch_overlap_test (STRATA_PIPE_OVERLAP=1)
token-identical to the build without the change; with the switch off batch_test16 is byte-identical too
- batch decode rows/s (engine level, 30 s, back to back with the reference): 16 slots 683 -> 726 (+6.3 %),
32 slots 750 -> 802 (+6.9 %); STRATA_VERIFY_PROFILE "QSA projections + attention" per stage-window
0.89 -> 0.52 ms (16) and 1.52 -> 0.79 ms (32)
HIP: the new code uses only what the existing kernels use; SYCL has its own sources and only sees the new
declarations. Neither was built here.
Upstream's non-finite flush only marks running slots `tainted`; their checkpoints stayed. With
STRATA_SLOT_CKPT_MOVE a running slot's checkpoints are a resume source (active_source) and a slot's
moved turn checkpoint can be restored into the main session (lent_slot), so a request right after the
flush, logged as "reading this prompt from token 0", still continued from a pre-flush checkpoint.
- the flush drops a running slot's checkpoints too (only with STRATA_SLOT_CKPT_MOVE, the one reader of
them; they go back to the checkpoint pool)
- active_source and the lent_slot search skip tainted slots
Checked with STRATA_NAN_INJECT=600 on the T16-142gsc setup (next turns of chats still running in
slots): before, the two requests after the flush resumed 1628 tokens from slots 0 and 8 ("gave back ...
its turn checkpoint"); now both read 1661 tokens from token 0. A next turn before the flush still
resumes from its running slot. Default path (switch off) unchanged.
On a layer split with more than one group a batch read never gives way to a
BYIELD, so step 4 always failed with 'no such free slot' and steps 5-7 never
ran. --yield auto now expects the give-way only on a non-pipelined batch;
there D is read whole and E admitted beside it, both still equal to solo.
New step 7b: A ends in slot 1, --flush unrelated solo GENs, then F2 can only
start from slot 1's turn checkpoint. --dump writes every token list for an
A/B of two builds.
mmq::whole_tiles walks every visible GPU and every tile width through the
MMQ config table; the sort hook called it twice for each MoE layer of each
prompt chunk, on the host path between the routed-ids download and the
expert launches. Its verdict depends only on the layer's weight formats and
the device set, so MmqPlan now keeps it as a per-layer flag (taken only when
STRATA_MMQ_GROUP_SORT=1 and STRATA_MMQ_RESIDENT_SORT_NE is unset) and logs
how many layers qualify. Same order, same bits; without the switch nothing
new runs.
A pipelined batch window is followed on its stage's stream by the layout's commit graph, and the stage counts
as done only after both. When every row of the window is another slot's one token, that commit is
unconditional (each slot keeps its one row; commit_slot_prefixes allows nothing else), and most of the
graph only repeats what the window wrote: it puts each indexer tail back from its snapshot and appends the
same row again, and copies each PLE history back from the window's own snapshot of it. What it adds is the
GDN conv history and recurrence state, which the window ran without writing.
With the switch, such a window advances the GDN conv history (gdn_conv_l2_multi with commit) and state
(gdn_step_norm_multi with n_keep = 1, the kernel the commit graph runs for a one-row group) itself, as a
one-token solo window does, and takes no tail or history snapshots. No commit graph is captured for the
layout; batch_launch and commit_slot_prefixes launch none, layout_captured and ensure_batch_layouts do not
ask for one. Layouts with proposal rows (--batch-mtp) keep the commit graph. With the switch off every path
is unchanged.
Measured on 4x RTX 3090, UD-IQ4_XS, T16-142fcb (split 12,24,37), decode of 16 / 32 slots, 30 s each,
reference and new build back to back: 653.7 -> 686.0 rows/s (+4.9 %) and 714.1 -> 752.6 rows/s (+5.4 %);
the group step period 24.5 -> 23.3 ms and 44.85 -> 42.6 ms. STRATA_VERIFY_PROFILE: the window's GDN
recurrence takes 0.09 ms (16) / 0.18 ms (32) longer per stage-window with the state written in it.
batch_test16 (16 slots, a 1,200-token prompt) is token-identical to the reference dump; batch_test32
32/32 identical and its dump equal to the switch-off run; batch_overlap_test (STRATA_PIPE_OVERLAP=1) passes
and is token- and RESUME-identical to the switch-off run; batch_interleave_test parts 1-3 identical (its
part 4 fails at the same point with the reference build).
STRATA_MMQ_RESIDENT_SORT_NE=1 puts a fully resident layer's experts in row-count order, so each
16-expert group's max_rows (its tile padding) is close to its experts' own. It also moves the hottest
experts into the last, partial group, whose few tiles take MMQ's stream-k split: the K boundaries move
with the group's tile count and the prompt's bits change.
The new switch sorts only order[0, n - n % 16] (mmq_resident_sort_span) and leaves the partial group's
experts, and so its rows, where they were. A full group's launch has 160 (gate/up) or 320 (down) tiles
per column tile; mmq::whole_tiles checks on every visible GPU that every such launch runs one block per
tile (tile efficiency >= 90 %, NVIDIA) and that all tile widths share the I/K tiles and SRAM layout. Then
each expert's rows get the same bits in any full group whatever its group-mates and max_rows. Where it does
not hold (e.g. 108 or 128 SMs, AMD's stream-k grid) the switch does nothing. RESIDENT_SORT_NE takes
precedence when both are set; with neither the code path is unchanged.
prefill_mmq_tab_test part 4 runs a layer's groups as prefill.cpp does (src-gathered q8_1 rows, absolute
bounds, the group's max_rows) in ID order and in both sorted orders, for IQ3_S / IQ4_XS gate/up and
IQ4_NL / Q8_0 down: with the partial group kept every token's rows are bitwise equal (16 cases); sorting
the partial group too changes 133-1881 rows per case. 890 comparisons, 0 failed (RTX 3090).
Measured on 4x RTX 3090, UD-IQ4_XS, T16-142fcb (split 12,24,37, every expert resident), read time,
medians of 6, back to back: 150-token prompt 145.8 -> 136.2 ms, 1,200-token prompt 428.6 -> 363.4 ms
(RESIDENT_SORT_NE: 134.9 / 358.6 ms). The final residuals of five prompts (300-1,200 tokens,
STRATA_PREFILL_DUMP_R_ALL every position and STRATA_PREFILL_DUMP_R fp32) are byte-identical to the
reference build; with RESIDENT_SORT_NE every row differs. batch_test16 (16 slots, a 1,200-token prompt)
is token-identical to the reference dump.
STRATA_SLOT_CKPT_MOVE: an active slot is a slot source only when the main
drafter's rows come along or are not drafted from (no --mtp, --batch-mtp,
or a batch admission). The slot copy takes the main layers' cells only, so
a solo GEN beside active slots would draft from another conversation's
drafter rows below its start; it reads its prompt as with the switch off.
STRATA_CKPT_THREADS: a checkpoint thread widens its CPU mask before it
goes to the aux CPUs, so the mask the aux placement restores (a request
with aux_cpus=0) is the wide one, not the host thread's single CPU.
STRATA_ADMIT_MERGE_FIRST: a merged first window counts as one row in the
STRATA_DECODE_TIMING window statistics (its prompt rows are not decode rows).
The direct slot copy (STRATA_SLOT_COPY_DIRECT) waited for each of its ~116 copies on the private copy
stream. With the switch, conversation_checkpoint_copy and conversation_kv_copy only queue them (new
`queued` argument, default off), and the slot copy's closing sync_all is the one wait per device. A
streamed K/V state (kv_mode != 0) has its residency refresh on the legacy stream, which is not ordered
after a non-blocking stream: conversation_detail::order_after_queued records an event on the private
stream and makes the legacy stream wait for it first. The copies stay on the private non-blocking stream.
Same copies in the same order on one stream: tokens are identical. Measured on 4x RTX 3090 (T16-142,
24 admissions): the slot copy takes 0.5 ms instead of 1.3 ms; batch_test16 16/16, batch_overlap_test
and batch_interleave_test token-identical to the references; a --kv-resident 20480 run with a 21K
prompt is identical with the switch off and on.
A batch admission copied the main chain's longest checkpoint into the slot (~112 MB of host vectors, 7.6 ms, ~60 ms
on fresh pages). With the switch, the request's own turn checkpoint (or one it took out of the slot it is admitted
to) moves instead; a root, a pinned prefix and any other checkpoint the chain shares are still copied.
The chain then no longer holds it, so the request path also looks in the slots:
- while the main session holds its tokens (live, its cells below it), a slot's checkpoint is restored on the main
session as a chain one would be, with no K/V copy; when that slot is the one the request is admitted to, it moves
back into the chain first (the admission drops the slot's checkpoints), else the slot keeps it and the admission
copies it as it copied the chain's;
- an active slot's checkpoints are a slot source like an idle slot's (its windows write its cells from p on): a
second request with the same history after the main session moved on restores instead of reading it all;
- a slot source that is the admit slot moves its checkpoint into the chain instead of copying it.
The switch also keeps the dropped checkpoints' storage as STRATA_CKPT_POOL=1 does: a moved checkpoint frees
nothing at the next prune, so without it every save faulted in fresh pages (17 -> 38-60 ms, more than the copy saved).
Off by default: nothing changes. A session SAVE or a parked conversation right after an admission lacks the moved
turn checkpoint (the slot holds it).
STRATA_CKPT_POOL=1 (off by default): the serve loop keeps the running-state
vectors of the checkpoints it drops (LRU eviction, the tail one, the prune at
a new request, a batch slot's old one), at most two, and the next checkpoint
or slot copy goes into them. A save then resizes nothing: no zero-fill of
~118 MB of fresh vectors. Every byte is overwritten by the copy.
STRATA_CKPT_THREADS=1 (off by default): a layer split's checkpoint saves the
first stage's part on the calling thread and every later stage's on a
persistent thread of its own (the copies' private streams are per thread, so
short-lived threads would leak them). The threads leave the host thread's one
CPU first (to the aux CPUs with --aux-cpus, else to every CPU the process may
use): on it the copies only took turns.
Measured on a 4-stage split (T16, 24 batch admissions per arm): a turn
checkpoint takes 17.2 ms today, 10.4 ms with the pool and 5.6 ms with both;
the batch tests, the overlap and interleave tests and a driver run give the
same tokens with the switches on as off.
A batch admission read its header through the verify windows (T=4 + T=2 for the
usual 6 tokens) and then its first token in a window of the last prompt token
alone. With the switch the last windowed part leaves its last tokens (up to the
verifier's max_t, 8, the last prompt token included) to the first window: the
part's earlier tokens are read as before, the first window commits every row,
the drafter gets the rows of the prompt tokens in it as the prompt windows give
them, and only row T-1's output is emitted. Row T-1 sits at the first window's
position with the same history and Philox draw, and a row computes what it
computes in any window while every expert is in VRAM, so the token and the
state are the same.
Only for a BGEN admission with every expert of every stage in VRAM (no loan,
no swap pending, the zero-doorbell graphs), and not with --batch-mtp, the
pipelined windows, pictures, the elastic K/V, a peer device, remote caches,
foresight swaps, STRATA_LOGPOS or a checkpoint at the last token. The merged
window takes no drafter q (STRATA_SPEC_PROB), adds nothing to the draft
counts, and the "PP <n-1>" line is printed before it. Verifier::max_t() is new.
Off by default: the same path as before. Tokens (4x RTX 3090, split
12,24,36, 16 slots, STRATA_IQ_MT_MIN=1): 24 admissions of prompts of 3 to 65
tokens and chat headers with and without an assistant prefill (merged T 3 to
8), greedy, seeded and with penalties, on vs off identical; batch_test16 16/16
and equal to the earlier dump; batch_overlap_test token- and RESUME-identical;
batch_interleave_test equal; 47 sampled driver requests equal.
Measured (24 admissions, ~131 and ~1195 tokens): header windows 23.5 plus
first window 11.5 ms become one 7-token window of 27.6 ms, 7.4 ms less per
admission (send to first batch token 246.8 -> 239.4 ms short, 656.1 -> 648.5
long). A window row costs ~2.4 ms up to T=6 and ~4 ms at T=7 and 8 (T=1 11.4,
T=7 28.6, T=8 32.1 ms).
A pipelined read never gives way (batch_read_piece returns 0 when piped), so a
BYIELD that reaches the main loop is for a read that has ended and is dropped.
The main loop still drained the groups for it first, as for a request, unless
STRATA_PIPE_OVERLAP was set. With the switch the BYIELD is dropped as it comes.
A BYIELD the server sends during the read is queued and taken right after BADM,
before any group is restarted, so that drain costs nothing today (0 groups in
flight, 0.0 ms). Only a BYIELD that arrives once the groups run again (the
server reads the control lines late) paid for it: measured with one sent 60 ms
after BADM, drain 26 ms with 4 groups in flight and the first rows 50 ms after
it, a 23.5 ms gap in the slots' rows and about 14 rows lost (at 576 rows/s). With
the switch the longest gap is 7.1 ms, as in plain decode.
Off by default: the same path as before. Tokens: batch_overlap_test (a BYIELD
during B's read) on vs off token- and RESUME-identical, batch_interleave_test
output byte-identical, batch_test16 16/16, and 41 sampled requests of the
admission driver (8 with a BYIELD while 4 groups were in flight) equal.
One stderr line per batch admission ("strata trace: admission of slot N: ...")
splits it into the drain before BGEN, the prompt (batched read, header windows,
the checkpoints' device syncs, copies into RAM and eviction, the rest), the first
window, the slot copy, freeing the slot's old state and the copy of the turn
checkpoint into the slot. Each checkpoint gets its own line, and with pipelined
groups "first rows N ms after the admission" times the refill after BADM. A drain
for another line says how many groups were in flight and what it cost.
Only with STRATA_TRACE set; without it the log and the tokens are unchanged.
Measured (4x RTX 3090, split 12,24,36, 16 slots in 4 groups, 24 admissions):
drain 25-26, header windows 23.5, checkpoint 17.3 (sync 0.1, save 17.2), first
window 11.5, slot copy 1.3, slot freed 0.0, checkpoint copy 7.8 (60 on a slot's
first fill), refill after BADM 23.7 ms.
The >8-slot cap ran only before the slot sessions were carved. When fewer
sessions fit than asked for, the groups were resolved again with no row bound:
auto could pick a group of more than kVerifyMaxT slots (PGroup's rows overrun,
every window refused) and a --batch-groups N that no longer divides the slots
fell back to one serial group whose windows reach only the first 8 slots.
slots_in_windows() picks the most slots whose groups, resolved as
batch_groups::resolve does (a number given, auto, the default's one group for
--adapt-async 1), divide them with at most kVerifyMaxT rows each and at most
one group's worth per stage. It runs before the groups are resolved and again
on the slots that fit (the extra sessions are dropped like the per-stage trim);
the pump refuses a group over kVerifyMaxT rows as a safety net. The hand-off
buffers hold one row per slot only for pipelined groups, so --batch-mtp keeps
upstream's kVerifyMaxT rows. Log and comment wording without the g3 tags.
- The build option is off by default and why (36-38 MiB of VRAM per GPU under EAGER, ~70% of the upstream MMQ
instances' compile time per arch); the nine types with a pointer table.
- The skip-gather hack's 225/577 ms are no bound: it changes the routing and does less work.
- Engine exactness as measured: first-window logits (STRATA_DUMP_FIRST_LOGITS) and greedy tokens byte-identical,
2 vs 1 vs 0, on the 4-GPU split with --prefill 2048 (the streamed walk), with 9 GB less expert cache per GPU
(mixed in-place, copied and mid-group-flushed groups), and on one GPU with --expert-cache-per-layer.
The file mirrors parts of llama.cpp's ggml-cuda mmq.cuh; like the other
ggml-derived kernels (native_moe.cu etc.) it now includes the copyright and
permission notice the MIT license asks for. Comment only.
- ExpertCache::slot_guard() returns the guard open_sized applied (guard_, reset by close), not the request: 0 for
uniform slots, when closed, and on SYCL, whose open_sized applies none.
- tests/core/expert_cache_slot_guard_test: slot_stride, slot_room and slot_tail on a uniform cache (empty off_,
including the last slot) and on sized caches with and without the 256 B guard; no guard gives today's offsets;
close and reopen.
- generate: the "+256 B guard per sized slot" line only for a sized cache; a uniform one says it has no guard.
- prefill: STRATA_MMQ_DIRECT_LOG=1 (with STRATA_MMQ_DIRECT=1) prints each stage's totals of experts read in place,
resident and streamed after every prompt, so an engine run can show which walk read what.
Review fixes for the pointer-table MMQ build:
- STRATA_MMQ_TAB defaults to OFF, like STRATA_MMQ_KQUANTS: the engine loads every kernel at start
(CUDA_MODULE_LOADING=EAGER), and the nine tab instances took 36-38 MiB of VRAM per GPU (sm_86, measured with
a cudaMemGetInfo probe linked with and without the tab objects; equal under LAZY), with STRATA_MMQ_DIRECT unset
too. prep_group_kernel is compiled only with STRATA_MMQ_TAB, so a build without it has the same device code as
3adce85 (the 58 cubins and PTX of build/strata are byte-identical to engine/strata-g3b's).
- The fingerprint block and the tab TUs are skipped on STRATA_HIP_GFX906: that build compiles no .cu from here, so
the define, the uncompiled TUs and an unlinkable prefill_mmq_tab_test no longer appear.
- Tab instances only for Q2_0 IQ2_XXS IQ2_XS IQ2_S IQ3_XXS IQ3_S IQ4_NL IQ4_XS Q8_0, the types prefill_mmq_tab_test
checks bit for bit; Q5_0 and the K-quants (STRATA_ORCA_Q4KS_MMQ, STRATA_MMQ_KQUANTS) keep the gather.
- overread_bytes takes the largest over-read of any config, not the over-read of the largest K tile (the over-read
is not monotonic in the tile length).
- The test fails (instead of skipping) when a tested type has no tab, checks that Q5_0/K-quants report none, and
pins the sm_86 over-reads (0, 72, 136 B); the dead K-quant scale branch is gone.
- SYCL's moe_mmq.dp.cpp gets the STRATA_MMQ_DIRECT stubs (false/0), for a future migrated prefill.cpp.
- moe_mmq.hpp points at docs/DETAILS.md.
MMQ loads a whole tile's ids, and its stream-k fixup one per column tile of max_rows, past an
expert's last row (upstream behaviour on both paths; the values are not used). With the map
padded, compute-sanitizer memcheck runs the whole test clean: 0 errors, 874/874 bitwise equal.
A pointer taken from a dynamically indexed array parameter (mmq_x_tab by value) made nvcc copy
the table to local memory and read the weights through generic LD.E loads instead of
upstream's LDG.E.*.CONSTANT: the products ran 1.2-2.5x slower than upstream's in the unit test
(Q8_0 down 2.2-2.6x, IQ4_NL 1.7-2.1x, IQ3_S 1.2-1.35x), which ate most of the gather saving.
The kernel now takes x0..x15 as `const char * __restrict__` parameters (like upstream's x)
and picks an expert's with a select chain: same SASS load mix as upstream (no local memory),
kernel time within 1% of upstream over every type and J in prefill_mmq_tab_test (69.7 vs
69.3 ms in total), and still 874/874 comparisons bitwise identical.
mmq_plan marks a native layer direct when the pointer-table MMQ covers both products, gate/up
reads nothing past its matrix, the down over-read (72 B IQ4_NL, 136 B Q8_0) fits the 256 B
guard, and the blob is exactly [gate | up | down] in MMQ's matrix sizes; it says once how many
layers qualify.
In the MoE walk (both the staged and the streamed walk) compute() takes the expert's cache slot:
a resident expert whose slot has room for the over-read (slot_tail) and a 16-byte aligned blob
is not gathered - the group's pointer tables point at its gate|up and down in the slot, and the
guard goes into the group's GuardTab. Streamed, peer and unguarded experts are gathered as
before, at compact positions c = 0..ns-1 (gather_native, or gather_native_group with its
mid-group flushes unchanged); the end-of-group flush runs even when the last entry was direct.
At the group end one prep_group launch zeroes the direct experts' guards and the tails after
position ns, replacing the two memsets; both products run with the same n, bounds, ids,
max_rows and buffers plus .tab. STRATA_MMQ_DIRECT=0 (default) and 2 run the existing body.
PrefillStats::experts_direct, said on the generate summary when non-zero.
ExpertCache gets inline set_slot_guard / slot_guard / slot_stride / slot_room / slot_tail; the
4-argument open_sized declaration is unchanged (the SYCL engine defines it out of line against
the same header and ignores the guard). open_sized lays slots out at slot_stride(blob, guard);
blob_ stays the largest blob, so fills, verify_slot and the HIP staging read no further.
slot_tail works on uniform caches too (through slot_offset).
generate.cpp: slot_guard = Prefill::slot_guard_bytes() (mmq::direct_slot_guard(): 256 when
STRATA_MMQ_DIRECT is 1 or 2 on a pointer-table build, else 0) for native packs; the seven
round-256 slot costs (split-skip-if-fits, split placement, per-layer sum, profile sizing,
shrink_to, cache_bytes, stage sizing) use ExpertCache::slot_stride(b, slot_guard), and the
primary and stage caches get set_slot_guard before they open. Guard 0: the same arithmetic as
before, so the default layout is unchanged byte for byte.
A Strata-side copy of llama.cpp's MMQ outer scheduler (mul_mat_q, launch_mul_mat_q,
mul_mat_q_switch_J; fallback=false only) in src/prefill/moe_mmq_tab.cuh: each expert's base
address comes from a 16-entry table passed by value instead of x + z * stride_channel_x.
The launch geometry (J, tiles, stream-k grid and k split, fixup) is computed from the same
mmq_args, and process_tile / load_tiles / vec_dot / write_back / the fixup kernel are
upstream's own templates, so the results are the same bits. Instantiated per type from
mmq_tab_instance.cu.in in strata_mmq (same flags); moe_mmq.cu sees declarations only.
CMake turns it off unless the SHA-256 of mmq.cuh's mirrored region (mul_mat_q_process_tile
through mul_mat_q_case) is a verified one: 3cf03257 (the fetched pin; the vendored tree's
mmq.cuh is byte-identical). HIP and SYCL get nothing; Product::tab is a default-null field.
Also: overread_bytes, tab_fits, direct_mode (STRATA_MMQ_DIRECT, read once, 0 unless the tab
is built), direct_slot_guard, and prep_group (one launch: zero the direct experts' guards and
the two group tails). Nothing calls them yet; the default path is unchanged.
tests/cuda/prefill_mmq_tab_test.cu (STRATA_BUILD_TESTS): 874 comparisons on an RTX 3090, all
bitwise identical - IQ3_S gate/up, IQ4_NL and Q8_0 down (1..16 experts, rows
{0,1,7,8,9,31,64,100,512,1000}, every J 8..128, stream-k fixup on and off, max_rows above the
largest count), every other instanced type, mixed direct/gathered groups (compact c != q,
mid-group flush, last entry direct); negative control: guards left as fp16 NaN give non-finite
down outputs.
The main sessions and the batch slot sessions of a stage live on the same device and are
carved alike, so copy_to_slot / copy_from_slot (without a checkpoint) now copy the running
state and the QSA K/V directly (conversation_checkpoint_copy, conversation_kv_copy:
the snapshot code's private non-blocking copy stream, one device sync before and after) instead of the
checkpoint/K-V image round trip through host memory with a sync per layer. Same bytes into
the same places in the same order; the residency refresh for streamed K/V is shared with
conversation_kv_restore. Opt-in: STRATA_SLOT_COPY_DIRECT=1 (default: the host path,
as upstream).
4x RTX 3090, UD-IQ4_XS, --batch 16, split 12,24,36 + --trim-stage-weights: admission copy
65.6 -> 34.0 ms average (pure copy ~8 ms); tools/batch_test16.py 16/16 and
batch_interleave_test.py (incl. next turn from its slot) identical to solo.
Measured on 4x RTX 3090 (UD-IQ4_XS): 16 and 32 slots decode bit-identical to solo
(tools/batch_test16.py / batch_test32.py); 16 slots 603 tok/s batch decode vs 345 with 8.
Rebased on 0.1.42: the stages' batch row check (batch_rows_error, upstream 48f2c47d) bounds
hbase + S by the hand-off rows; set_handoff_rows passes the per-slot size there (only above 8 slots).
On a Mac the Monitor tab's GPU cards were empty: telemetry had NVML (NVIDIA) and the amdgpu
sysfs files (#301) and nothing else, so the Metal backend read nothing. The Apple GPU's load
and memory are in the IOKit registry's IOAccelerator nodes (ioreg -l -w0 -r -c
IOAccelerator, 22 ms), and the device's memory budget is the same Metal property the engine
reads (metal/src/mtl_rt.mm's recommended_working_set, the engine's "GPU working set" log line
- 26,800,603,136 bytes here, which is 25.0 GiB).
_Metal (a new reader beside _Nvml and _Amd): the index'th accelerator node's PerformanceStatistics
give Device Utilization % (older chips: the renderer or the tiler) and the in-use system
memory; the total is the working set, read through the Metal framework and objc_msgSend (no
PyObjC). The temperature and power sensors are not readable
without root (powermetrics), and an Apple Silicon GPU sits on the die, so there is no PCIe
link: the cards show a dash and static gpu_note says why, the same way #1380's note does for a
Windows AMD card. gpu_reader takes `amd` first, so the AMD trees still read on a Mac for the
tests, and a missing selector is asked for before it is sent (an unanswered one raises an
Objective-C exception no ctypes call can catch).
Service.free_vram_mib now goes through gpu_reader instead of _Nvml directly, so the Mac gets
that reading too (it used to return None there).
Measured on an M1 Max (macOS 26.6.2): through the server, GET /metrics during a request -
gpu_name "Apple M1 Max", gpu_vram_total_mib 25559, gpu_util 74-77 % while decoding (9 % once
idle), gpu_mem_used 10.5 GiB in use of 24.96 GiB, cpu 12.6 %; the plain-text
/prometheus-format metrics carry the same series. serve/test_server.py: MacTelemetry
(fixture registry: two accelerators, index selection, renderer/tiler fallback, a raising
runner, the note and the samples) plus AmdTelemetry, TelemetryNoDisks, Monitor and the full
serve/test_server.py (318 tests) pass on the same box.
What it leaves alone: the NVIDIA and AMD paths (a test-only gpu_reader order change on
macOS), and the web app - the cards read the series it already had.
Co-authored-by: Demetree Kallergis <[email protected]>
setup.py returns from mac_setup() before its own argparse runs, so a Mac never
sees the --source that sets STRATA_SOURCE, and the Mac parser did not define it
either: the flag the download failure recommends landed in parse_known_args'
leftovers and was dropped with "not used on a Mac".
Define it in the Mac parser and set STRATA_SOURCE from it, mirroring setup.py.
Recorded with the hardware they were measured on, per the measured-numbers rule, because
neither can be fixed here: mm_gu3's fault is a threadgroup-count threshold of the
cooperative-tensor path on this GPU (the fix shape would change the M5's register-operand
GEMM), and router_route's race has no acquire atomic to fix it with on this Metal
(device-scope atomics are relaxed-only, macOS 26.6). Both are opt-in and off by default;
the engine's M1 defaults are the v2 expert GEMMs and the separate gemv + moe_route, which
run on this box.
Measured on an M1 Max (macOS 26.6.2, Q2_0): mm_gu3 with 35 threadgroups passes and 40
faults (data-independent; the engine's tile 16x32x32 SG 4 and the v3 test's 190 tiles both
fault). router_route's logits are bitwise gemv's, but two identical runs give different
ids, weights and plan.
Co-authored-by: Demetree Kallergis <[email protected]>
Two stale defaults, both of which kept the test from covering what the engine ships.
The default tile was TM 16 TN 64 TK 16, but the v1 kernel set only instantiates TK 64: the
default run died at pipeline creation on any GPU (measured on an M1 Max:
"mtl: no function mm_gu_42_16_64_16_2"). TK is now 64, which both the v1 and the v2 sets
instantiate, so the default run and --v2 both work; the engine's v2 tile for Q2_0
(16,64,16,2) stays reachable with --tile.
--v3 bound the v1 kernels' argument list ({W, x, H, stok, tiles, ma}) to the v3 kernels:
tiles was read as the sorted-token buffer and MmArgs as raw int2 pairs, so every stride,
n_embd and expert id was garbage. On the M1 Max that is a deterministic GPU
command-buffer error in the GEMM pass; with the right arguments the v3 kernels run, and a
separate M1 limit shows (≥40 threadgroups of mm_gu3 fault, 35 pass, data-independent)
that the engine does not hit: gu_v3 = dn_v3 = coop_k4, which is false on the M1.
v3 now binds {W, XS, HS, tiles, ma} as the engine does, fills XS for v3 too (v3 is v2's
gather plus the register-operand GEMMs), and binds HS - not H - to down3 with DSG
simdgroups.
Measured on an M1 Max (macOS 26.6.2, Q2_0, T 256, 64 experts): default (v1) PASS, rel L2
2.53e-06 vs the fp16-operand reference and 2.37e-04 vs exact; --v2 PASS, 4.42e-06 and
2.37e-04, also with the engine's tile (16,64,16,2).
Leaves alone: no kernel, engine or other test change.
Co-authored-by: Demetree Kallergis <[email protected]>
qsa_attn_mm and qsa_flash (metal/kernels/qsa_prompt.metal) fill their single-simdgroup
matmul2d right inputs with
rk[i] = id >= 0 ? half(float(kq[row * QD + k]) * float(ksc[...])) : half(0.0f);
and that shape - a select with the device load in the arm - loses the threadgroup on an M1
Max: every dispatch of the two kernels fails with
kIOGPUCommandBufferCallbackErrorInnocentVictim, deterministically, and they run only under
MTL_SHADER_VALIDATION=1. qsa_prompt_parity dispatches qsa_attn_mm the way the file intends
(it is the matrix-unit attention of a prompt chunk) and faulted on its first dispatch before
this change. The engine reaches the same fills through qsa_flash, which it dispatches under
STRATA_METAL_PROMPT_QSA=flash; its default on an M1/M4 is the fp32 qsa_attn, which never
touches these kernels - that is why the default prompt path kept running while the parity
test could not, and why this is measured on the test rather than on a request.
The row is now clamped with max(id, 0), the load is unconditional, and the select is on the
value. For an id >= 0 the value is what it was; for an id < 0 it was discarded either way, so
the results do not change.
Measured on an M1 Max (macOS 26.6.2, Metal compiler 32023): qsa_prompt_parity ALL PASS -
qsa_attn_mm vs qsa_attn rel L2 3.09e-04 (pos0 0) and 3.56e-04 (pos0 30000), qsa_flash 3.72e-04
and 4.22e-04, qsa_score_mm worst rel 2.66e-06 (T 96, within budget and with 2051 selected
cells a token); before the change the test faulted on its first dispatch and reported nothing.
The compile failure that started this - qsa_flash's cooperative_tensor copy, "cannot
initialize a parameter of type 'void *' with an rvalue of type 'const void *'" - is already
fixed in main (#2323); this change is the runtime half of the same two kernels' M1 Max
bring-up.
What it leaves alone: the COOP_K4 path (a function constant: compiled out of the M1/M4
pipelines), qsa.metal, and every host file; no CUDA, HIP or SYCL backend builds these kernels.
docs/MAC.md records the M1 Max with the measured tok/s (the hardware's medians, from the
benchmark report: chat decode 10.6, 4K prompt 223.7, 16K prompt 228.6 tok/s).
Co-authored-by: Demetree Kallergis <[email protected]>
stream.cpp: the standalone-mmap fallback (the prompt's buffers as their own allocation) sat inside the chunk loop behind the slot-class check, so it only ran when a chunk was big enough for the prompt's 1.60 GB of buffers. When the arena shrank below that at the start (the 24 GB M2: chunks of 0.8-1.0 GB), every chunk was skipped and the prompt stopped with 'no arena region for the prompt buffers'. Both standalone sites - a fitting chunk whose slots a class cannot spare, and no fitting chunk at all - now share standalone_region(); the fitting-chunk carve and the slot-class rule are unchanged.
This is the half of #2323 that 0.1.44.1 could not see: the hotfix's reporter ran a 96 GB M2 Max, where the arena's chunks are several GB and the prompt region always fits. On a 16-24 GB Mac running --mmap-experts, 0.1.44.1 still stops at the first prompt with 'metal prompt: no arena region for the prompt buffers'; this makes the standalone allocation run there too.
Verified on a 24 GB M2 (macOS 26.6.2, Qwen3.8-Flash-Next GSQ-RCO-Q2_0 streaming): the fallback line appears where the error used to be, prompts run end-to-end (3.7-4.1 tok/s steady in web chat, no crashes). That run predates the standalone_region() refactor, which moves the same statements into one function both call sites use.
docs/MAC.md: the small-arena note says the prompt's buffers also get their own allocation when no chunk of the arena is big enough for them.
strata_bench starts the server on the config's `host` but polled readiness at
a hardcoded 127.0.0.1, so a LAN bind (192.168.1.10 in the report) printed
"still loading" while the server was answering on that address.
Use the configured host, mapping the wildcard binds to loopback and bracketing
IPv6 literals; an explicit --url still wins. Patch from the report, with the
mapping lifted into server_base so a test can reach it.
/metrics/prometheus exposes llamacpp:prompt_tokens_seconds and
llamacpp:predicted_tokens_seconds as averages of the cumulative totals, so an
idle server keeps advertising the speed of its last request.
llama.cpp reads those two names from a bucket it resets on every scrape ("the
gauges are averaged over the window between two scrapes", server_metrics::bucket
in tools/server/server-common.h) and reports 0 when the bucket is empty, so a
llama.cpp dashboard on Strata froze at the last value where llama.cpp decays to
0. Keep a bucket per scrape and expose its delta.
An earlier commit capped the accepted drafts at the end of an answer; that is an exact state, but it moved the hybrid mode's expert-slot
counts, so a sequence of requests with --mtp on a card that does not hold every expert differed from the base engine from the fourth
request on (found by comparing the 10 parity prompts against the base goldens on the 12 GB card). Now the speculative step commits what
the base engine committed; a stop seen before the state moves keeps only x (the server's hand-over of a stopped request to a batch slot then
reuses the held state: its text equals the run alone), and the held tokens list x and every accepted draft, so a conversation that continues
from them matches the state exactly. Setup says what parallel measured on the 12 GB card; docs/QWEN36.md updated.
spec_step takes no draft past an end-of-answer token or past the room max_new leaves, and a stop asked for before the state moves keeps
only x: every token emitted but the last is in the state, so a conversation that continues from the held tokens (the server's solo ->
slot -> solo hand-over, a chat's next turn) neither misses nor reads a token twice. Before, an answer ending inside accepted drafts left
the state ahead of the held tokens (found by tools/qwen35/q36_serve_handoff.py: the first request's text now equals its run alone).
STRATA_Q36_READ_WINDOWS sets the decode windows of the other slots between two chunks of a prompt read (default 1). The mtmd dump takes
several pictures; the engine logs each request's reuse; a 1,024-token test picture.
upload_f32 runs on the host and widens F16 with
__half2float(__ushort_as_half(...)), but __ushort_as_half is a __device__
intrinsic: nvcc 12.1 and 12.4 refuse the call ("calling a __device__ function
from a __host__ function is not allowed") and q36_engine.cu does not compile.
Widen the bits here instead, the way the neighbouring bf16_to_f32 already
does for BF16.
0.1.44 made IQ3_S the recommended Qwen size on a PC with 64 GB or more (IQ3_S: 9 to 15% slower decode than IQ3_XXS, more RAM).
SETUP.bat / --setup on a folder that already holds a model of the family now pre-selects the installed size (and --yes takes it);
the menu marks it "(installed)" next to "(recommended)", every size stays a choice, a first install takes the recommendation.
SETUP_ENV gained STRATA_MMQ_KQ, STRATA_HC_Q8 and STRATA_ROUTE_TAIL_SKIP, which setup writes for Unsloth's files only; a setup
run again dropped a user's own value of them in any other config (a hand-set STRATA_ROUTE_TAIL_SKIP=0). They are setup's only
in an Unsloth config.
/v1/responses (Codex) read only the Chat settings shared with apps, so the config's "reasoning_effort" (#1641) never applied
there (it always thought at xhigh); Service.effort_default() is now the one place that says which level a request that names
none gets, for the chat, Messages and Responses routes. openai_to_messages dropped "reasoning_effort": false (an `or` between
the two fields); effort_kwargs already treats false as thinking off.
The new test renders the prompt of every way a client sets the level (OpenAI reasoning_effort / reasoning.effort /
chat_template_kwargs, Messages API output_config.effort / thinking budgets at the 2048 and 8192 limits / disabled, Responses),
the config key, the shared Chat settings and their precedence, the web app's request shape, tools and a conversation.
Setup: the images question for qwen36 (GPU encoder on an NVIDIA card of 16 GB or more, the CPU encoder otherwise, AMD under Linux on the
CPU; --vision gpu kept as asked with a note), the pinned mmproj (size and SHA-256) beside the GGUF, the vision section and --vision in
the config; --parallel N is written as asked with what a slot costs, and the recommendation is said where every expert still fits.
docs/QWEN36.md: pictures and several requests at once with the measurements.
A slot's window carries its next token and, for the greedy slots that fit beside one row per slot, the proposal of its MTP block;
the proposals for the next window come right after the window from its own hidden states (the first from the state the prompt
left). Grouped GDN rows (a slot's rows in sequence on its own state), verify windows and their commit run per slot; every slot's
tokens equal its solo run. The engine also says what the sessions take and stops with the numbers when they cannot fit.
One conversation state per session; --batch N keeps N of them and decodes one token of each in a single window (rows of
several sessions in one forward, bit-equal to the solo rows), reads a prompt between the other slots' windows and takes a
conversation over from the session that holds its start. GENI / BGENI pictures: the encoder's rows replace the embeddings
of the <|image_pad|> cells, positions as llama.cpp's mtmd (interleaved M-RoPE 11/11/10, text after a picture continues at
p + max(nx, ny)). tools/qwen35/q36_mtmd_dump.cpp is llama.cpp's side of the picture check.
0.1.43's gate already had 'not on a layer split' as its first predicate; it printed the line only when the variable was set. A default 0.1.43 split ran without the overlap, so 0.1.43.1's opt-in default changes nothing on a split. Measured on 2x A4000 (IQ3_XXS, layer split, 5 interleaved pairs): v0.1.43, v0.1.43.1 and v0.1.43.1 + STRATA_ADAPT_OVERLAP=1 decode within 0.6% of each other.
On Windows cudaHostRegister of the whole arena fails (the WDDM shared budget is about half the RAM) and the
sliced fallback registers the first ~30 GiB: with Qwen3.8-Flash-Next IQ3_S layers 0-31, so layers 32-47 never
get a PCIe share, while part of the registered prefix holds GPU-resident experts that DMA never reads.
STRATA_PIN_BY_NEED=1 (the arena on one GPU):
- every blob gets a page-aligned stride (PadMap in pinned.hpp, ArenaExpertSource::arena_off, +0.15 % RAM), so any
set of blobs can be registered exactly; the loaders place each blob at its padded offset
- the expert profile's pairs past the estimated GPU-resident count are merged into ranges by rank and registered
before any page is touched, up to STRATA_ARENA_PIN_GIB (29 GiB by default on Windows) or the first refusal
- pinned() / device_alias() / pcie_layer() work on the ranges (a blob is pinned only if wholly inside one);
lock_resident_ranges() keeps the holes between them resident
- one start-up line: "pin by need: N ranges, X GiB pinned, Y % of the non-resident experts"
STRATA_PCIE_MIN1=1: the PCIe share (nmiss * pcie_num) >> 8 rounds to 0 at 1-2 misses per layer; with two or more
misses one goes over PCIe (when its blob is pinned).
RTX 3090 + Ryzen 5 5600X: 28.96 GiB registered = 92 % of the non-resident experts, all 48 layers with a PCIe
share; prompt read +5-7 % at 56K-117K tokens, decode level alone and +2-3 % with --adapt-async 1 and
--pcie-frac 0.6 (bench/results/2026-10-11-community-rtx3090-5600x-arena-pin-by-need, docs/DETAILS.md).
Both opt-in; without them the layout, the loaders and the sliced pin are those of v0.1.42.
In the arena mode the blocking adaptive tier creates and joins a thread every verify window and the next window
waits for its copies (#1863: the time outside the decode timing breakdown). --adapt-async 1 now also runs with the
arena on one GPU, not only in the resident RAM mode. The arena holds every expert, so a round is a copy-in only:
no copy back, no RAM move, no exchange buffers.
- gates for the arena; ArenaExpertSource::blob_at() reads a blob from the job thread without the read counter
- the evicted slot is marked non-resident and uploaded before the round's copies are posted; an expert becomes
resident only once its copy's event has completed
- STRATA_ADAPT_ASYNC_WAIT=1 (arena default): the next tick waits for the round, so the window an expert turns
GPU-resident does not depend on timing; STRATA_ADAPT_PICK_INLINE=1 (arena default): one round per window;
STRATA_ADAPT_ASYNC_BATCH=N caps the copies per tick (default 0 = all); arena rounds are not capped at 96
- adapt_quiesce(): every K/V grow, kv_quiesce and the VRAM command land the round in flight (they only called
apply_pending(true)); a failed round still drains its copies. This also covers the resident mode's async tier.
- STRATA_DECODE_TIMING: "adapt wait X join Y, Z swaps/window"; --adapt-async help text
RTX 3090 + Ryzen 5 5600X, cold prompts: x1.07-1.11 decode, faster on 12 of 12 prompts; 25.9 -> 21.6 ms per window
(bench/results/2026-10-11-community-rtx3090-5600x-arena-async, docs/DETAILS.md).
Opt-in (--adapt-async 1, default 0); the default path is that of v0.1.42. With the tier on, answers are not
bit-exact run to run (which window first computes a swapped-in expert on the GPU).
The GPU's own wiring of the arena ends seconds after its last command buffer, and a Metal residency set did not hold it either
(wired memory 75 -> 3 GB between requests, measured with vm_stat and a probe). Idle, the arena is ordinary anonymous memory, and a
20 GB file written beside it made macOS compress tens of GB of it: the third panic (wired 2.4 GB, file cache 50 GB, compressor 1.9M pages).
mlock keeps it wired (STRATA_METAL_WIRE_ARENA=0/1, default on from 64 GB; not when the fill stopped early). Same write, arena wired,
full speed: compressor +0.7M pages (the engine's non-arena memory), 408 pages swapped, nothing else.
docs/MAC.md: the footprint cap, the wiring and the new variables.
Both panics (watchdogd starved 92 s) had the streaming engine at 95.3 / 89.9 GiB resident, free memory 13 MB, the compressor at 2-4M pages.
macOS keeps a file-cache floor (vm.vm_page_filecache_min, 37.2 GB on the 128 GB M5 Max) and compresses anonymous memory instead of dropping
file cache below it; the arena is anonymous and the GPU wires it only while a command buffer runs, so during the fill and when idle it is
pageable. Measured with a 1 s memory watchdog: old build + 100 GB arena, or auto arena + 40 GB file write + a 16 GB program: compressor 21k -> 4.07M
pages in 10 s, 29,000 pages swapped, pressure warn. The 67% rule counted only cx.allocated() (7.5 GB) while the process held 11-15 GB more, and
67% was too much anyway.
- arena_budget.h: footprint_cap (anonymous memory: arena + Metal buffers + what the process holds, task_info phys_footprint) = 60% of the RAM on
a Mac of 64 GB or more (and at least the file-cache floor + 12 GB left); resident_cap (a resident pack's mapped file pages, measured: UD-Q4 resident,
wired 91.7 GB, 40 GB write + 16 GB program, nothing compressed) = 70%; STRATA_METAL_LEAVE_SHARE for measuring. The explicit
STRATA_METAL_ARENA_GB is kept and flagged when it is over the cap.
- window.cpp logs the planned total (arena + held + n-gram reserve against the cap) and the footprint after the fill.
- the engine refuses to start while a repack or setup's model preparation holds a lock beside the model (STRATA_METAL_IGNORE_REPACK=1 overrides).
- metal_repack.py: unbuffered write(), F_NOCACHE on the shards and the output, fsync every GiB, a pid lock (file cache stays flat; the old tool filled all 128 GB).
- setup_mac.py: no engine while preparing a model (stops setup), start() waits on a live repack lock, resident rule = the engine's resident_cap.
- tests: arena_budget_test.cpp (47 checks), tools/test_setup_mac.py (repack guard).
- CMake project version 0.1.44; setup MIN_ENGINE (0, 1, 44); NVFP4_ENGINE (0, 1, 44) (UD-Q5/Q6 were already 0.1.44).
- setup: a prebuilt zip without strata_kq leaves no stale library next to the new engine (kept in .previous); the
Dockerfile copies libstrata_kq.so into engine/.
- Model menu: families in the order Qwen (GSQ-RCO), Swift, Coder, Unsloth, NVFP4, Qwen3.6; NVFP4 is not listed on AMD
and Intel (the Unsloth line names UD-IQ4_XS there); on NVIDIA its line says native FP4 (RTX 50) or that UD-IQ4_XS is
faster on this card, with the download and RAM need. The Unsloth family lists its five sizes, images on.
- strata_mcp's built-in tables know the NVFP4 sizes and family. tools/test_setup_menu.py.
Variant 6 of gdn_recurrence_variant (sycl/include/strata/prefill/gdn_xe.hpp): the column kernel's four row groups
and FMA chains in the same order, a 16-lane sub-group holding four columns x four row groups, so the cross-row-group
sums are sub-group shuffles and a token needs one work-group barrier instead of five (256 GRFs a thread).
Arc A770 16 GB (xe, oneAPI 2026.1 JIT), same day, a0116b27 + #2190 + #2193 with and without this commit:
gdn_parity recurrence 51.0 -> 24.5 ms at 8,192 tokens, 204-236 -> 110-113 ms at 32,768 (48 heads), 0 mismatches
(state, FP32, FP16). Coder IQ1_M, 128K context, int8 KV, --resident-experts: prompt 386 -> 396 / 407 tok/s at 8K,
478 / 480 -> 511 / 518 tok/s at 32K (2 runs each), the same output ids in every run.
On by default on Alchemist only (strata::intel_gpu_gen); every other card keeps the column kernels, unchanged.
STRATA_GDN_XE=0 turns it off, =1 turns it on for any Intel GPU with 16-lane sub-groups. gdn_parity compares
variant 6 with variant 0 bit for bit (state, FP32 and FP16 outputs) on any Intel GPU and skips elsewhere;
ctest -R prefill_gdn_parity runs only that part.
On the A770 the multi-token GR read's kernels launch too few work-groups to fill the card. With
STRATA_GR_MORE_GROUPS on:
- sliced down kernel: one work-group per (slice, row block), 80 x 6 = 480 groups instead of 80;
every row's partial sum is computed exactly as before, only by a different group.
- gr_down_reduce: 8 slice loads in flight per step, then added one at a time, so x accumulates
slices 0, 1, ... 79 in the same sequence of float adds as the old loop (-fp-model=precise).
- up kernel: 8 columns per group, 320 groups instead of 160 (the kernel is now templated on the
column count; UPM_COLS stays 16 for every other user), and only group 0 writes the per-token lo
values, which every group computes identically.
Default on Alchemist only (strata::intel_gpu_gen); Battlemage and other devices launch the same
kernels as before. STRATA_GR_MORE_GROUPS=1 forces it on, =0 off; gr_bench runs either way.
Measured on an Arc A770 16 GB (xe, oneAPI 2026.1 JIT), same day, a0116b27 + #2190 + the previous
commit (GR_PICK, large GRF), with and without this one: gr_bench 1-6 tokens 118.9-183.5 -> 94.6-148.4 us,
checksums identical for every token count; gr_multi_parity 32 cases OK with STRATA_GR_MORE_GROUPS=0 and =1.
Decode, Coder IQ1_M, 128K context, int8 KV, --resident-experts, STRATA_VERIFY_STEPPED=1, 8K prompt, 256
tokens: 22.12 / 22.35 / 22.55 / 22.65 -> 23.24 / 23.61 / 23.62 tok/s (+3-5%), output ids identical.
Not built or run on Battlemage.
- GR_PICK: a runtime index into the by-value kernel argument (m.a[k]) made IGC copy the whole
GrMulti (~1.2 KB) into every work-item's private scratch (unitrace on an A770: ~40 KB private
per SIMD32 thread in gr_up_multi / gr_down_reduce / gr_norm_split). GR_PICK picks one field with
compile-time indices only (a chain of selects over the uniform kernel argument). Addressing
only: the same values, read and summed in the same order, so the output is bit-identical. This
part is on for every device; it is not measured on Battlemage.
- The sliced down kernel and the multi-token up kernel in large-GRF mode (256 registers/thread);
at 128 the up kernel still spills ~3.4 KB/thread on the A770 (unitrace). Large GRF halves the
threads resident per Xe core, so it is the default on Alchemist only (strata::intel_gpu_gen);
Battlemage and other devices launch the same kernels as before. STRATA_GR_LARGE_GRF=1 forces it
on, =0 off.
Measured on an Arc A770 16 GB (xe, Linux 7.0, oneAPI 2026.1 JIT), a0116b27 + #2190, same day:
gr_bench 1-6 tokens 205-258 us -> 119-184 us (GR_PICK alone, STRATA_GR_LARGE_GRF=0: 186-239 us),
checksums identical for every token count. Coder IQ1_M, 128K context, int8 KV, --resident-experts,
STRATA_VERIFY_STEPPED=1, 8K prompt, 256 tokens: decode 5.37 tok/s (5.88 / 5.93 with
STRATA_DOORBELL_CHECK=1 / STRATA_LEND_MIRROR=0) -> 22.12 / 22.55 / 15.60 (cold-cache run)
tok/s, 20.61 / 20.93 with STRATA_GR_LARGE_GRF=0; the window's pool phase 183-208 -> 34-35 ms
per round; output ids identical where the expert placement matched. Not built or run on
Battlemage or i915.
On an Arc A770 16 GB on the xe driver, Level Zero has no ext_intel_free_memory and get_memory_info reports 0 free
with the right total. device_free_bytes() kept the 0 (the #1440 fdinfo path only replaces a figure that is too
high), so --expert-cache auto sized the caches from 0 bytes. A 0 with a known total is now read as unknown: on
Linux the card's total less this process's DRM fdinfo figure is used, the same source the A750/i915 path uses.
Anything that reports a non-zero free figure is unchanged; off Linux the 0 stands.
When the main layers' experts fit but the MTP block's do not (UD-Q6_K_XL on
32 GB), the main layers are placed as in the all-in-VRAM mode (same text,
the decode graph kept) and only the MTP block's experts go to slots + the
CPU pool (its drafts). To get there when the fit is close: the packed Q8_0
decode copies are given back, the prompt chunk stays 512, and - said in the
log - the attention scores buffer shrinks to 64 MB and the VRAM reserve to
256 MiB. STRATA_Q36_NO_MTP_CPU=1 keeps the plain hybrid mode.
UD-Q6_K_XL on gfx1201: tg128 73-75 -> 80.6 (llama.cpp HIP 81.2).
Many small copies were slower than the one whole-layer copy on a 12 GB card
(RTX 3060, UD-Q4_K_XL at 37 of 256 resident: pp2048 1540 in runs, 1974
whole). gfx1201 UD-Q6_K_XL (205/256) and Q8_0 (176/256) keep the per-expert
copies: pp4096 3504-3588 and 3015-3079.
The server already keys its encoder cache by the normalized picture bytes (sha256, the .sve file
name); it now sends those keys with the request (img_keys=<hex>,..., in prompt order), and the
engine's conversation cache uses key + grid for a picture when the count matches. Without the key
(an older server) it hashes the embeddings as before. Old engines skip the unknown key.
Measured for #2228 (one 1600x1000 screenshot-like picture, strata-vision, fresh processes and
twice in one process): the CPU encoder gives the same bits on a TR PRO 3975WX at 8/15/16/32 threads
and on a Xeon E5-2643 v3 (e2357d8e...), the GPU encoder on an RTX A4000 is reproducible too
(92cf7818...), but CPU and GPU differ. So a picture's embeddings change when the encoder moves
between CPU and GPU, or after an encoder update, and no longer on its own; with the byte key the
cache's matching does not depend on that at all. CUDA/HIP and SYCL. Test: serve/test_server.py
test_pictures_are_keyed_by_their_bytes_for_the_engine.
The prompt's staging buffer took the whole layer from host memory over PCIe
(~19-28 GB per chunk); now the experts with a VRAM slot are copied device to
device and only the others come over PCIe, in runs of consecutive experts.
Same bytes in the staging buffer (greedy ids identical). UD-Q6_K_XL on
gfx1201 (205 of 256 experts per layer resident): pp4096 1935 -> 3511-3601.
Also: the packed Q8_0 decode copies are given back when that lets every
expert fit (STRATA_Q36_KEEP_PACKED=1 keeps them); STRATA_Q36_STREAM_ALL=1
keeps the whole-layer copy.
A checkpoint (or the live state) is reused only when the prompt starts with its tokens AND its
pictures, and a picture is its grid plus the hash of its embedding floats. In #2228 (screenshot-heavy
agent, CPU image encoder) the cache fell back to checkpoints a long way back although the client's
messages were byte-identical; one explanation is a picture encoded again with other bits. When the
deepest state whose tokens match is rejected for a picture, the engine now logs which picture and
where, once per request, so the next log tells this apart from a rewritten history. Log only; what
is reused does not change.
47Hunter47 measured </tool_response> asked verbatim at 0/15 on 0.1.43.1 (it reached the model as
the real closer). 62185aab already marks it; this pins the cases next to the exception that keeps
a wholly wrapped tool result's outer pair: the bare closer, a request that quotes a whole pair, a
message that only starts with the opener, two closers in one sentence.
Qwen3.6-35B-A3B-Q8_0.gguf stores the GDN alpha / beta projections as Q8_0;
the engine read only F32 / BF16 there and refused the file. They are now
widened to F32 at load (fp16 scale x int8, as ggml). q36_llama_dump:
STRATA_DUMP_CTX sets a smaller context / batch for files whose weights leave
no VRAM for 4096-token buffers.
The engine keeps every expert in RAM when they do not all fit in VRAM, so a
size's RAM need is its file plus room for the rest. The largest size this
PC's RAM holds is the default; any size can still be chosen (a bigger one is
asked about, as before, never refused).
- k_attn_mma (sm_80+): m16n8k16 f16 -> f32, 8 warps x 16 queries, 64-key K/V tiles double-buffered with
cp.async, ldmatrix (.trans for V), fp32 online softmax. Bench on the RTX 3060 at 4096 x 4150 keys x 32
heads: 23.7 TFLOPS vs ggml's flash_attn_ext_f16 15.4 (0.376 vs 0.577 s/step); CPU check 7e-5.
STRATA_QI_GGML_ATTN=1 keeps ggml's.
- q, k and v of a block with one weight type load into one [3C x C] tensor: one GEMM and one activation
quantize instead of three (STRATA_QI_NO_QKV_FUSE=1 keeps three).
- A float4 LayerNorm path with tanh(gate) once per step, off by default (STRATA_QI_FAST_LN=1): no measured gain.
RTX 3060, Q8_0 1024, seed 42, 3 interleaved rounds: steady step 2.09 s vs stock 2.61 (1.25x), denoise
17.93 s vs 22.13 (1.23x).
A start right after another engine exited sometimes fails before the engine writes a line: ROCm aborts with
"terminate called after throwing an instance of 'rocr::AMD::hsa_exception'" (or the process exits with an empty
log) while the previous process's GPU queues are still being released (r9700: 1 of ~10 rapid restarts in the A/Bs,
2 of ~40 for the quants runs). Such a start - within 30 s, no engine line other than the runtime's failure, not the
engine's own error exit 1 - is retried with a message saying why; STRATA_START_RETRIES (default 3, 0 = off).
0.1.41 (0391e478) scaled each stage's miss term by 20 GB/s / its probed link, for 2x 4090 D
with CUDA1 on a physical x1 link (1.6 GB/s: x12, K 22 -> 38, prompt ~350 -> ~1400 tok/s).
On 2x Tesla T10 the same rule moved auto from K=23 to K=14: CUDA1 runs 34 of 48 layers, and as
the prompt path is a pipeline the slower stage sets the rate - 256K prompt 1035 -> 795 tok/s,
decode 46.8 -> 39.5; --layer-split 23 on 0.1.43 restores both (xiake-1 in #2007).
split_rules::link_miss_scale: the scale is the fastest stage's link over this stage's, applied
only from a 4x gap (a riser or chipset x1/x4 slot beside an x16 one), capped at 64. A symmetric
rig keeps the fitted formula whatever its link speed; the x1 case still gets ~x15.
STRATA_SPLIT_LINK_ABS=1 restores the 0.1.41 rule. SYCL has no link term (unchanged).
Test: split_rules_test (the x1 case, equal gen3 cards, a 2x gap, caps, unprobed stages).
Not run on the reporter's T10s: the reply asks for the 'link ... -> miss cost' lines.
The native output head is needed only to generate, the batched prompt path's buffers only to read a prompt. With the flag the head is unloaded while a prompt is read (its VRAM counted into the auto cache and the prompt path's hybrid borrowed/owned layout) and read back in before the first decode window. On an 8 GB RTX PRO 2000 Laptop (Windows, IQ3_XXS, 590 MiB reserve) the prompt chunk went from 512 to 2,048 tokens and prefill from 292.5 to 667.1 tok/s on a 29,854-token prompt; reads of up to 512 fresh tokens cost about 0.6-0.75 s more. Off by default: every new path is behind the flag. docs/PREFILL_HEAD_ALIAS.md has the measurements and limits.
The table is read with F_NOCACHE through a bounded row cache (1M rows), so 2 GB was too much to set aside. M4 32 GB,
IQ3_XXS decode after the 6K document, 5 interleaved runs, auto arena (memory_pressure every 3 s, vm_stat swapouts):
keep 7 + 2 GB (54c0de70) 5.93 tok/s median, arena ~8.5 GB; keep 7 + 0.5 6.59 (+11%), ~10 GB; keep 5 + 0.5 7.47
(+26%), ~12 GB; the least free memory 43-57%, no swap-out in any run. The M5 Max still keeps 15% (20.6 GB).
arena_budget_test: 14 checks pass.
ROCm gives out hardware queues per priority. Sharing one with the window's stream, --pcie-mode dma's expert copies
and the host function that raises flag B queue behind the window graph that spins on that flag: on an R9700 every
dma-mode request timed out at layer 1. With its own queue dma mode runs (copies overlap the VRAM-hit compute).
The figure in "[strata] images: the image encoder started in N s, M MiB of VRAM" read the free VRAM right after the
cache's shrink, before the driver's counter showed it (amdgpu sysfs), so it came out low (286-362 MiB against the 1,339
measured otherwise). A 0.2 s wait before that reading, only on a start that shrank the cache.
Opt-in, with the config's "vram_elastic": the GPU encoder does not start with the server or a load. A picture's turn
makes the expert cache keep "elastic_mib" (default 2048) free on top of the reserve and starts the encoder; after
"elastic_idle_s" (default 120) without a picture a thread stops it between requests and the cache grows back to the
reserve (0 when no POST /v1/vram was made: the start's reserve left it a segment or more smaller). A POST /v1/vram
while the encoder runs stacks on its share; an unload stops it. Without "vram_elastic" the server says so and starts
the encoder as before. serve/test_vision_elastic.py pins each case; docs in VRAM_ELASTIC.md and DETAILS.md.
update_install refreshed the draft subset once per installed config, newest first, so a folder that two configs
name ended with the subset of the config used longest ago. A config without "draft_vocab" (cjk) then put the
default subset over the cyrillic one of the config in use. Now the first config that names a folder decides;
separate folders and a start (which installs its own config's subset) are as before.
Checked: tools/test_setup_draft_vocab.py (3 new tests, 10 pass), test_setup_config and test_setup_configs (18 pass).
Measured on the affected install, two alternating runs each, greedy 600-token Russian story: 54.0 / 54.0 tok/s
with the default subset installed by the update, 68.7 / 68.6 with the cyrillic one (1.39 -> 1.90 tokens per window).
A helper for the large-page arena (#1412): run as Administrator, it adds the account to
SeLockMemoryPrivilege with secedit, says that a new logon is needed, and prints the secpol.msc
steps when it cannot run. Changed from the PR: secedit /configure is limited to
/areas USER_RIGHTS (the exported policy is not re-applied as a whole), and the description no
longer claims a speed-up we have not measured. docs/DETAILS.md points to it. Not run on our side
(no Windows box for it).
Co-authored-by: dvasdekis <[email protected]>
Some amdhip64_7.dll builds (AMD TheRock wheels / ROCm 10.2 / 7.2.4 nightly) do not export
hipKernelNameRefByPtr although their import library has it, so strata.exe failed to load with
STATUS_ENTRYPOINT_NOT_FOUND before main. It is only used to name graph nodes under
STRATA_VERIFY_NODES; on Windows it is now looked up with GetProcAddress and a missing one
leaves the node unnamed.
From #2183, its verify.cpp part only. The PR's other part (Windows HIP --pcie-mode auto as
DMA) changes the default path of every Windows AMD user and was not measured on our side; it
is left out.
Co-authored-by: dvasdekis <[email protected]>
An Arc A770 under xe intermittently never started decoding: the main thread waits in Verifier::run's stepped serve
(queue::wait -> urQueueFinish) for a segment that never drains, xe logs nothing (#2194). The A750's same failure was
fixed by draining the other queues before each window (STRATA_WINDOW_SYNC, default 1 on i915 only). Reproduced on
the A750 by turning that off: STRATA_WINDOW_SYNC=0, STRATA_VERIFY_STEPPED=1, --resident-experts, IQ3_XXS, 4K prompt:
1 of 6 runs hung with #2194's stack; with the default drain 0 of 11 (5 unpinned, 6 with the pipelined helper pinned).
The default is now 1 on i915 or on an Alchemist device (intel_gpu_gen), 0 elsewhere; STRATA_WINDOW_SYNC overrides.
Unchanged on the A750 (i915: already 1) and on Battlemage (0). Not run on an A770 under xe (no such card here).
The first version compared only the extra loan with the RAM available at that point, but the
resident RAM copy is pinned after it and holds the lent slots' experts too: in a 30 GB scope on
an R9700 (IQ3_XXS, --resident-experts) 30 GiB were available before the 23.35 GiB copy, so the
rule never fired. The reporter's PC thrashed with 3.4 GB left at 16384 and ran with ~6.4 GB at
8192, hence 6 GiB.
Both NVFP4 packs keep Qwen's vision tower unquantized, so the image encoder every Flash-Next model uses reads pictures
for them; strata-vision opens the text model vocab-only, which is the pack's dense GGUF here. Setup asks about images as
it does for every model (default no; --vision gpu|cpu), the encoder is the file other installs already share
(mmproj-Qwen3.8-Flash-Next-BF16.gguf at ISTA's pinned revision, 907,543,008 B, sha256 b1a82259...).
Checked end to end through the server on an RTX 5070 (budget mode, GPU encoder, 1024 image tokens): a picture with
"CODE 4217", a red square on the left and a blue circle on the right - NVFP4 answered "4217" and described both shapes,
their colours and sides correctly, as IQ3_XXS did on the same picture. VRAM: the expert cache held 393 slots with
images (--vram-reserve-mib 2100 and the encoder) against 982 without (1500).
The gfx12 default (0.1.43) lets a bare --prefill auto reach 16384, which lends the prompt path
about twice the cache slots of 8192 (R9700, Swift 1.5 IQ3_XXS: 3,907 slots / 6.38 GiB against
2,068 / 3.37 GiB); the lent slots' experts are read from the RAM side during the read. On a
30 GB PC with --resident-experts 38-47K prompts went 20 GB into swap with 1.5M major faults
and the engine was killed; --prefill 8192 ran them at ~95 tok/s (bdwilton in #2090).
At start, when the engine's own 16384 ceiling is in effect, the extra slots of a 16384 loan
over an 8192 one are priced at the largest blob; if the RAM available (the cgroup or
--memory-limit-mib cap included) is below that plus 4 GiB, the ceiling stays at 8192 and a
line says why. STRATA_GFX12_CHUNK=1 keeps 16384; an explicit --prefill N / auto:N is never
touched (the 16384 default only applies to a bare auto).
The 'MiB of VRAM free with everything loaded' line and its LOW warning read CUDA0 alone. In
#2192 (2x RTX 2080 Ti 22 GB, Windows, IQ3_S, --batch 4, --layer-split auto, 0.1.43) CUDA0 had
740 MiB free while CUDA1 began its window captures at 153 MiB and fell to 52; the first batch
window then never finished. Both stall dumps show the host thread inside cudaGraphLaunch called
from Verifier::run_slot_rows (the stage still 'idle': it had not reached its per-layer wait), the
pool asleep and the watchdog firing after 60 s - a page-in under WDDM while the other card's
graph spins on a host flag.
Each later card now gets its own line, and below 512 MiB a LOW warning with the
--vram-reserve-later-mib value that leaves ~768 MiB. Said, never changed.
On Windows with only an AMD card neither NVML nor the amdgpu sysfs files answer, so
hardware_static.gpu_name was None and strata_bench (no nvidia-smi, no rocm-smi on Windows)
labelled the run "CPU only" although the engine ran on the card (ByteJuggler in #1587: RX 6900 XT).
telemetry.windows_display_adapters reads DriverDesc, ProviderName, DriverVersion and the memory
size from HKLM's display class (no subprocess, no vendor library); windows_amd_cards drops a
Ryzen's integrated Radeon (512 MiB here) when a real card sits beside it. hardware_static gets
the name and gpu_vram_total_mib; strata_bench uses the same list when nvidia-smi and rocm-smi
give nothing. Load, VRAM use, temperature and power stay unread on Windows AMD (that is the
reader of #1224 / #1434 / #1871). Checked against this PC's registry (RTX 5070 + the Ryzen 7600's
iGPU: both listed, the iGPU's 512 MiB read); tools/test_windows_amd_gpu.py with a fake registry.
The stock split shape (STRATA_STAGE_TRIM=1, 1024 MiB reserve, NO_HOST=1 from strata-sycl.sh) is
refused at the start on a 2x Arc Pro B60 24 GB pair with IQ3_XXS: the second stage is 2230-3731
experts short and the host mirror covers the first stage only. The same config with NO_HOST=0
runs at 39-50 tok/s with 0 faults (jds1-ai in #1980). When the model's experts take more than
80% of the split cards' VRAM, setup now writes NO_HOST=0 into the config (a value the user set
is kept) and prints why and how to turn it back. IQ2_XS on the same pair (74%) keeps NO_HOST=1.
Test: tools/test_setup_sycl.py test_split_short_rule.
- refuse_exit: after the REFUSED message the engine waits for the queued device work, flushes
and _Exit(2)s instead of returning from main. Returning ran the stages' destructors in an
order the UR loader did not survive: on a 2x Arc Pro B60 pair every REFUSED start segfaulted
in libur_loader, and a trimmed one also left an xe page fault (bcs CAT error) on the second
card that hung the next start (jds1-ai in #1980).
- resident_plan_set_mirror takes the table's length; resident_plan uses the mirror and the
usage counts only for a residency slice inside the table they were set for. A later stage
passes its own residency table, so the slice was taken as an offset across allocations and
read (and counted into) another buffer: the stage-1 xe faults with STRATA_VERIFY_NO_HOST=0
and no trim (10 of 10 requests, 0 tokens). Such a miss is now a plain miss, served by the
host. Patch shape as jds1-ai proposed it.
Compile-checked only (no two-card Intel box here).
Under WDDM a stream's work sits in the driver's command buffer until a driver call flushes it.
With the stream-ordered signal the verify window only spins on a pinned int, so the last
layers' copies and their 4-byte signal copies were never submitted: the first request stopped
with "the adaptive tier's copies for layer 45 never landed" (RTX 4070 SUPER, 0.1.43.1, opt-in
STRATA_ADAPT_OVERLAP=1). STRATA_ADAPT_SIGNAL=host worked.
- Windows: the host function is the default signal; STRATA_ADAPT_SIGNAL=copy asks for the copy.
- Every platform: one cudaStreamQuery on the copy stream after the swaps and their signals are
queued (a no-op for the results; it submits the batch under WDDM). Linux keeps the copy
signal as before.
Needs a Windows run with STRATA_ADAPT_OVERLAP=1 STRATA_ADAPT_SIGNAL=copy to confirm the flush.
bench/results/2026-10-12-p40-sm61-kernels: where a prompt and a decode window spend their time on a P40, the kernel
benches and parity runs, every request of the engine A/B (565-token prompt 205 -> 255 tok/s, decode 34-35 -> 37-38
tok/s; with STRATA_DENSE_F32=1 a 7K prompt 360-432 -> 464-474 tok/s), the byte comparison of the answers, and what
was not tested. docs/OLDER_GPUS.md: the sm_61 section and switches, and two traps when measuring on such a card.
As sm_60 got with STRATA_SM60_TABLE, measured on a Tesla P40 (native dp4a):
- native_expert_bench, 18 groups at 2 tokens on the IQ3_S pack's layers, mode 0 -> mode 8, bitwise: IQ3_S 390 ->
255 us, IQ3_XXS 345 -> 252, IQ2_S 290 -> 238, IQ4_XS 328 -> 261.
- kIlRows_61 / kIl1Rows_61 from three runs of mmvq_il_parity --bench --emit-table (identical but two Q3_K cells;
a cell is non-zero only where every run gained 3%).
For FP16 operands with FP32 sums cuBLAS 12.4 has two Maxwell kernels on a Tesla P40 and picks the slow one for most
chunk sizes: qkv (10240 x 2560) runs at 7.6 TFLOPS for T 7030-7040 and at 4.3 from T 7041 (49 -> 85 ms), so a
7K-token prompt reads in 16.3 or 19.5 s by its token count. cublasSgemm on FP32 copies runs at 8.4-9.8 TFLOPS from
T 512. For the five shapes and the sizes the route takes, the result is bitwise the FP16 call's: gemm_f32_parity,
5,201 chunk sizes through Gemm::f16. That is a property of the cuBLAS build (12.4.5.8 measured), hence opt-in.
dequant_kernel's thread reads its 32-value group a byte at a time and writes 32 two-byte values one by one, 64 bytes
from its neighbour's. On a Tesla P40 that is 10 GB/s for Q6_K, and the IQ3_S pack's dense weights cost every prompt
chunk ~0.5 s of it, whatever the chunk's length (a fifth of a 560-token prompt). The wide kernel takes 8 values a
thread: 16-bit loads, one 16-byte store, the same expression per value.
dequant_wide_parity: 360 cases bitwise (FP16 / BF16 / FP32 out, a padded stride, row slices, sane and raw random
blocks); --bench on the P40: Q6_K 10240 x 2560 7.25 -> 0.33 ms, Q4_K 5.2x, Q5_K 3.8x, IQ4_XS 3.6x.
On by default on compute capability 6.1 only (where it was measured); STRATA_DEQUANT_WIDE=0 / =1 for any card.
The router, the GDN alpha / beta and the shared expert's gate (n_out 1-256,
K 2048) ran on hipBLAS/cuBLAS SGEMM with 8x8 / 16x16 tiles (~10% of a
prompt). k_sgemm_nt: (16 TM) x 64 tiles, TM x 4 outputs per thread, K split
so about 128 blocks run, the partials summed in a fixed order
(deterministic). gfx1201 pp4096: 3290 -> 3455 tok/s. STRATA_Q36_BLAS_F32=1
keeps the BLAS SGEMM.
Warm-up: a throw-away prompt chunk, a decode step and (with drafts) a
speculative step at load make the code objects, BLAS kernels and graphs, so
the first request runs at full speed (pp4096 rep 0: ~1770 -> 3273 tok/s; a
34-token first prompt: 1032 -> 48 ms). --no-warmup skips it.
Hybrid mode with MTP exact by default: the MTP block is loaded whenever the
file has one (so the VRAM layout and the slot count do not depend on --mtp;
the prompt pass through it only runs with drafts); decode and verify rows'
routing is counted only for the rows kept; the slots are rebalanced exactly
at every 32nd position and a verify window never steps over one.
arena_budget.h: the rule as plain arithmetic, tested by metal/tests/arena_budget_test.cpp (13 checks, run on the
M4). The M5 Max froze (2026-10-13 02:08) with an 89 GB arena planned from 'available - 7 GB' beside a 50 GB n-gram
table: macOS now keeps 15% of the RAM (20.6 GB there), the table's mapped bytes (--ple-io ram) or 2 GB (read row by
row) come out too, and just before the arena is committed the free memory is asked again - less than planned: the
arena is laid out again smaller instead of failing. STRATA_METAL_KEEP_FREE_GB / STRATA_METAL_ARENA_GB override.
stream.cpp: arena buffers grow past 8 GB when 4 of 8 GB cannot hold the arena (STRATA_METAL_ARENA_CHUNK_GB).
The coordinator suggested max(12 GB, 15%); on the 32 GB M4 (Q2_0 streamed, 200 tokens, memory_pressure every 3 s)
12 GB kept cut the arena 9.4 -> 4.4 GB and decode 7.8 -> 4.8 tok/s, while 7 GB kept left macOS 55% free the whole
run (12 GB: 70%); same tokens. Hence 7 GB below ~47 GB of RAM, 15% above. Not yet run on the M5 (down).
iq_host.cpp (Metal's copy of the host helpers): Q6_K among the grouped gate/up formats, as mac-kern's moe_gu_14 /
mm_gu3_14 run them. Before, the engine refused UD-Q5_K_XL ('layer 2's experts are Q6_K/Q8_0 ... no GPU kernels').
strata reads the --native file's architecture at the top of main() and hands
a qwen35moe model to strata::qwen35::run (src/qwen35, now the strata_qwen35
library; strata-q36 wraps the same entry point) before any Flash-Next state
exists; every other file runs exactly as before. strata --families lists the
architectures a build runs. The Qwen3.6 path takes strata's flag names where
they mean the same (--native, --max-new, --pool-workers, --no-token-graph,
--gpu, --mtp N) and skips Flash-Next-only ones in serve mode. The SYCL build
keeps refusing the family, now saying CUDA and HIP run it.
setup: Qwen3.6-35B-A3B is a family in the menu (OWN_PATH_FAMILIES,
install_own_path): size, context, the engine (checked with --families), the
pinned GGUF, its tokenizer, a config with --native/--max-context/--mtp 2 and
the start script; no pack, profile, draft layer or tuning. The engine step is
a function shared with the Flash-Next path. strata_inspect says the family
runs on CUDA and HIP.
- dense_bench: the decode window's native_mmvq calls at the model's shapes and formats, 24 matrices per shape in one
graph (weights from DRAM), 1-4 columns, us / GB/s and an output hash.
- STRATA_MMVQ_WIDE_K_RPW=2|4: Q4_K / Q5_K / IQ4_XS wide kernel with 2 or 4 rows per sub-group sharing each column's
activation loads (apply split into act + dot, the same operations); every hash equal. B70, 3 columns: Q4_K
6144x2560 56.6 -> 27.7 us, Q5_K 2560x12288 89.8 -> 75.5 us, Q6_K unaffected.
- STRATA_MMVQ_WIDE_RPW>1 and STRATA_MMVQ_PF now take the aligned Q6_K loads (they were 2-4x slower than the default
without them); still slower than the default except Q6_K 6144x2560 at RPW 2 (50.2 -> 45.3 us).
- fused GR read: the up step's columns per block (STRATA_GR_UP_COLS 4/8/16/32/64, default 16 = before) and its T
butterflies interleaved (STRATA_GR_UP_ILP=1); gr_bench checksums equal for all. 32 columns: 3 tokens 50.7 -> 48.9 us,
1 token 27.5 -> 29.4; ILP slower.
Engine, B70, IQ3_S with data/expert-profile-sycl.bin, 5 interleaved pairs: STRATA_MMVQ_WIDE_K_RPW=2 STRATA_GR_UP_COLS=32
vs default: decode after a 4K prompt 54.31 -> 54.00 tok/s, story 38.41 -> 38.50, outputs identical. Defaults unchanged.
Found by the quant yardstick (--expert-cache 1) on the NVFP4 GPTQ pack: 'pre-filled 0 of 0 slots', and the verify windows
refused to run. Every other native format's blob is rows x row bytes with 1280 / 2560 rows, a multiple of 256, so its
budget is unchanged.
ROCm gives out its hardware queues per priority level; the draft stream shared one with the verifier's, so the
draft round ran only after the commit graph the verifier had just launched. With a queue of its own the commit
runs beside the draft. R9700, UD-Q4_K_XL: draft wall 5.8 -> 3.5 ms a window (STRATA_DRAFT_TIMING).
The per-layer commit was ~98 graph nodes; it runs between a window and the draft round, and on HIP the round
waits for it (R9700 UD-Q4_K_XL: 2.35 -> 0.65 ms GPU). Each layer's arithmetic is the per-layer call's:
gdn_parity (gdn_commit_layers vs the per-layer commit, n_keep 1-8) and qsa_parity (append_steps_layers vs
copy + append_steps, 3 scalings) are bitwise. STRATA_COMMIT_TIMING=1 prints the commit graph's GPU time.
On the RTX 5070 PC (64 GB, Windows) --mmap-experts let the experts it read stay in the engine's working set: free RAM
fell to 0.6-4.9 GB and the memory guard stopped the engine three times (2026-10-13). --resident-budget-gib 22 with the
same pack kept >= 22.8 GB free. The budget mode is what setup already writes for Unsloth's files (resident_budget_gib:
40 GiB on 64 GB); the pack's experts.bin is the file the rest are read from. test_setup_nvfp4: 64 GB -> budget 40.
Verifier::window_logprobs reads the head rows through window_copy_logits under STRATA_GPU_METAL (the CUDA/HIP path
unchanged: verify.cpp's object is byte-IDENTICAL on P1004 and p620). The prompt's windows sample nothing (head
sampling off), so with STRATA_LOGPOS the Metal window computes the head anyway.
tools/quant_quality.py on the M5 Max against the Q8_0 reference (q8_0.tsv 627558d1, set.ids.json 414634dd), 4,995
positions: UD-Q4_K_XL 92.77% top-1, KL 0.0548 (p99 0.855), PPL 4.043 vs 4.052 (CUDA/HIP: 93.09% / 0.056); IQ3_S
87.85% / 0.1422 (CUDA/HIP: 87.7% / 0.144). The Metal window keeps int8 K/V (the yardstick asks fp16).
tools/image/sdcpp_native: a runner that replaces QwenImage21Runner::compute only (sd.cpp keeps the text
encoder, VAE, sampler and RNG), chosen with --model-args qwen_image_2_1_native=1 in CUDA / HIP builds.
- The text prefix runs once per condition (causal, the t = 0 modulation row) and leaves each layer's
fp16 K/V; each step runs the 32 blocks on the image tokens only.
- GEMMs stay ggml graphs on the runner's backend (sd.cpp's MMQ / BLAS kernels and numerics); fused
LayerNorm+modulate, gate+residual(+next LayerNorm), q/k RMSNorm+RoPE+fp16 K/V, SwiGLU and, on gfx11, a
WMMA flash attention (8 waves x 16 queries, 64-key tiles, fp32 softmax) run on the backend's stream.
- apply.py copies the two files into the pinned checkout and makes two small edits (runner choice,
SD_QI_NATIVE for CUDA / HIP builds); anything not covered falls back to the stock runner.
780M, Q8_0, seed 42: 512 2.11 s/step (stock 2.85), 1024 9.58 s/step (stock 14.2), image 96.5 s (131.8).
A parked pool worker watches the epoch's cache line with UMONITOR and sleeps in UMWAIT (C0.1) until the host publishes or the cycles run out. On a laptop whose CPU and GPU share one power budget the spin's power goes back to the GPU. Unset, 0, no WAITPKG or a non-MSVC build: the spin, unchanged. STRATA_POOL_UMWAIT_C02=1 picks the deeper C0.2 state, STRATA_POOL_UMWAIT_ESC=<cycles> C0.1 first and C0.2 after.
Two places upstream builds QsaShapes by hand from the geometry (chunk_bytes, added since the geometry change was
written, and the conversation-state fingerprint, in both trees). They now call qsa_shapes(g) like layer, mtp,
session, verify and prefill; for every file the guard accepts the values are the ones they built before.
generate.cpp reads the geometry before anything else looks at the model, so a model of another architecture was
refused as "missing qwen4exp.block_count" where 0.1.40.2 said "architecture is 'qwen35', this engine requires
'qwen4exp'" (found running Qwen3.8-27B through --native). read_geometry now checks general.architecture first, and
check_architecture is read_geometry plus geometry_unsupported. gguf_geometry_test: a dense qwen35 file (Qwen3.8-27B's
keys) and a file without general.architecture.
Tested on gfx1151 (HIP build of this branch, Windows): the GGUF tests pass; Qwen3.8-27B UD-Q6_K_XL, Q8_0 and
HauhauCS Q4_K_P are refused by architecture in 0.5 s; Qwen3.8-Flash-Next UD-IQ4_XS, greedy, 256 tokens on three
prompts (39, 48 and 5,568 tokens), gives output identical to the installed 0.1.40.2.
The f32 and array members added for gguf_geometry_test left the braced Kv{...} initialisers short of the struct, which
clang's -Wextra reports as -Wmissing-field-initializers in every test that includes the fixture (MSVC /W4 does not).
make_kv() sets the key and type, and each helper sets its own value. Built with the HIP toolchain (gfx1151):
gguf_geometry_test, gguf_split_test, gguf_reader_test and native_dense_ple_key_test pass, with no warning from the
fixture.
read_geometry (gguf_reader.hpp) fills ModelGeometry from the qwen4exp.* keys, under llama.cpp's names: layers, widths,
GDN, attention, indexer (heads, key width, pooling block from compress_ratios, top-k), rope width, hyper-connections,
experts, RMS epsilon, and the vocabulary from tokenizer.ggml.tokens. The six keys the guard always required stay
required; any other absent key keeps the compiled value. Keys that state one thing twice must agree (the interval and
compress_ratios, key and value head widths, routed and shared expert widths, ssm.inner_size and state x v heads).
check_architecture no longer hard-codes 48 / 2560 / 24 / 2: it reads the geometry and refuses a file the kernels are not
compiled for, naming the key ("qwen4exp.block_count = 40, the kernels are compiled for 48"). Only the expert count,
the experts per token (1..64) and the vocabulary may differ. ModelGeometry moves to core/geometry.hpp (no
dependencies), with kCompiledGeometry; the kernels that fix a dimension at compile time static_assert it there, in the
CUDA/HIP tree and the SYCL copies (plus qsa_prompt_attn_xmx).
generate.cpp takes the geometry and K from read_geometry, and the native embedding takes n_embd / n_vocab from it
instead of a literal 248320. The QSA kernel shapes come from one qsa_shapes(g) (layout.hpp) in layer, mtp, session,
verify and prefill, so n_rot, idx_block and idx_top_k follow the geometry too. Mirrored in sycl/.
Tested: tests/core/gguf_geometry_test.cpp (new, synthetic GGUFs) 28/28, gguf_split_test and gguf_reader_test pass,
built with MSVC /W4 /WX on Windows. `gguf_geometry_test --real` on Unsloth's Qwen3.8-Flash-Next UD-IQ4_XS shard 1
reads exactly the compiled shape and the guard accepts it. Not built here: the engine (no CUDA, HIP or oneAPI compiler
on this PC), so generate/layer/mtp/session/verify/prefill and the kernel asserts are reviewed, not compiled.
The Debian fallback loop ran with check=False on Ubuntu too, so a failed 13.0 install (a
network or apt lock error) went on to 13-1..13-4 and then past the loop without a word. Ubuntu
is back to the single checked install of 0.1.43; Debian tries 13-0..13-9 and fails with
guidance when none installs. Checked against NVIDIA's repos today: debian12 and ubuntu2404
carry 13-0..13-3, debian13 starts at 13-1 (up to 13-4). Two tests added.
The CUDA apt repo keyring and cuda-toolkit-13-x packages exist for
debian12/debian13 as well as ubuntu2204/ubuntu2404, so derive the repo
name from /etc/os-release instead of hardcoding ubuntu. Debian 13's
repo carries no cuda-toolkit-13-0, so try 13-0..13-5 in order (Ubuntu
still gets 13-0 first, same as before).
Checked with tools/test_setup_cuda_repo.py (4 tests, no GPU or network
needed): repo mapping for ubuntu/debian/other and the minor-version
fallback.
opencode puts its <env> block (working directory, today's date) near the start of the system message, before ~40K characters that never change. The prefix cache only matches a prompt's start, so each new folder or day read the whole system turn again. With the switch the block goes to the front of the first user message (string or content parts), for OpenAI and Anthropic requests. Off by default; serve/test_env_to_user.py; docs/DETAILS.md.
The server sends BYIELD for a waiting request under half as long only when a PP line (or the 10 s heartbeat) arrives,
and the engine looks for it at its next boundary. Where a boundary's own PP line is the only line for a while - every
chunk of a --batch read on one GPU, and #2050's pipelined pieces - the BYIELD answers that line and reaches the engine
after it has looked: the read gives way one boundary later, or never when that was the last one it could take. It is in
time only where the engine saves a periodic checkpoint after the PP line (every 16,384 tokens by default).
_control() now also calls on_quiet() whenever the engine has been quiet for half a second (its existing poll), and the
solo and admission reads run the same check there as at a PP line. A BYIELD for a read that has ended is ignored by
the engine, as before. serve/test_parallel.py: the fake engine takes --pp-every K (a PP line and a boundary every K
chunks); a solo read and an admission must give way at their first boundary (both fail on main), and the rule (only
to a much shorter request) holds on the poll too.
Measured on one RTX 5090 (Swift 1.5 IQ3_S, --batch 4, the engine built from main), main's server and this one, two
engine starts each (M F F M): a one-line request sent 0.2-8.6 s into a cold ~33K read, with nothing else decoding and
with one stream decoding. Late (a boundary passed after the request waited): 28 of 40 runs on main, 12 of which never
gave way; 4 of 40 with this change, all requests that started waiting 0.02-0.10 s before that boundary's PP line.
Where main was late the short request's TTFT went 3.56 -> 1.70 s (nothing decoding) and 4.40 -> 1.57 s (one stream);
where main never gave way 4.60 -> 1.51 s and 7.59 -> 2.35 s. Where both gave way at the same boundary, both requests'
TTFTs were the same (medians within 0.02 s). On 2x RTX 5090 with #2050's STRATA_PIPED_READ_PIECE=8192: 6 of 12 late without
it, none with it.
On two equal cards every K whose caches hold all the profiled pairs predicts the same window
time, and the search kept the first one tried: the smallest K, which gives the last stage (it
also carries the MTP drafter) the most layers and lets its cache fill the card to the reserve.
2x Arc Pro B70 with IQ3_S: auto K=24, CUDA1 at 31.8 of 31.9 GiB, ~11 tok/s; --layer-split 30
~54 tok/s (agnaylor in #1440). A full xe card does not fail an allocation, it moves buffers to
system memory.
The SYCL search now scores every placement it tries with its smallest spare room (a stage's
room after its session minus every expert of its layers) and, among the placements within
STRATA_SPLIT_HEADROOM_TOL (default 0.2%) of the best predicted time, takes the one with the
most (split_rules::pick_with_headroom). A log line names both picks when it moves the split.
STRATA_SPLIT_HEADROOM=0 keeps the old pick. CUDA/HIP placement is unchanged.
Test: split_rules_test (a 48-layer two-card model of the report: plain K=24 with 664 MiB spare
on the last card, headroom K=25 with 1728 MiB; tolerance and tie rules).
The three are USER_DEFINED (type 4) tokens like </tool_call> (ids 248058, 248066, 248070), so
the tokenizer matched them inside a message's text as the real call / tool-result markers.
They join TOOL_LITERAL_TAGS: inside a user, system or tool message (and the tool list) they are
encoded as ordinary text; an assistant message replaying raw call text keeps its tokens, as
for </tool_call>.
One exception keeps the template's behaviour: a user message whose whole text is
<tool_response>...</tool_response> (a client that formats tool results itself) keeps that outer
pair as the markers, since the template tells such a turn from a real question by exactly that
pair (last_query_index); the text inside is still marked.
Found by 47Hunter47 in #2026. Tests: serve/test_frontend.py QuotedToolCallCloser (every tag
and role, the assistant replay, the outer pair as string or text parts).
M5 Max: ./setup.sh --model UD-Q4_K_XL --vision yes: shards checked by SHA-256, repacked and verified, the pack, the
image encoder; the config it writes answers the test picture ('STRATA 42', 'Red circle / Blue square / Green
triangle').
The routed experts are placed after the engine's buffers: all in VRAM when
they fit (unchanged), else every expert in host memory with the same number
of VRAM slots per layer (--expert-vram GiB, --expert-slots N). Batches of up
to --hybrid-rows rows run the resident experts on native_expert_grouped and
the others on the CPU ExpertPool (ggml-cpu arithmetic) side by side; bigger
prompt chunks stream each layer's experts into two staging buffers for MMQ
(prompt chunk 2048 in this mode). Slots follow the routing counts (after a
prompt, every 32 decode positions). ExpertPool's native phases split rows by
the layer's n_ff / n_embd instead of Flash-Next's constants (same values
there). STRATA_MMQ_KQ=1 by default for this family.
ple_conv_add reads the conv kernel as half: Unsloth's file stores it F32, so it is narrowed (round to nearest even)
into its own buffer; the GSQ packs' F16 tensor is read in place as before. A file whose HC matrices are Q8_0
already makes no copies (the copy buffer was allocated with 0 bytes and the Metal allocation failed).
M5 Max, UD-Q4_K_XL resident (the repacked 82.5 GB file): greedy prompt1 identical to the CUDA reference (z440),
story first differs at token 36; MTP output equal to no MTP; decode no MTP 60.6 / 60.0, MTP --spec 2 93.9 / 74.0
tok/s; prompts 4K 1,374 / 32K 1,288 tok/s, needles found (4K, 32K). llama.cpp Metal on the same shards: 48.4 /
47.3 decode, 821 / 739 prompt.
With STRATA_FS_SLOTS > 0 and the resident RAM mode's adaptive exchanges, a slot copy issued
from an expert's RAM place (note_miss / the router look-ahead's predict) could land after an
exchange rewrote that place with the outgoing expert's bytes; take() then served expert Y's
bytes for expert X (FS_VERIFY: 19 BAD on clean v0.1.43, reported in #2134).
commit_exchanges, commit_copies and commit_flip now hold every staged expert (in and out) in
the swap space while their bytes move: its slot is dropped (never served again) and no copy
of it is issued until the commit is done (commit_copies holds until commit_flip, so the
asynchronous tier's half-done exchange is covered, and the rotation path too). Unlike the
patch in the issue, a dropped slot is not handed out at once: it waits the same `depth`
finished windows as any slot a window may still read, since a window planned before the
exchange can be reading it. "N dropped by exchanges" joins the swap space's report.
Foresight is opt-in; with STRATA_FS_SLOTS unset nothing here runs. Test:
file_expert_source_test --rotation-gpu (both commit paths, the depth rule, the bytes after).
Based on the analysis and patch in #2134.
Co-authored-by: radoslawsz <[email protected]>
The stdin reader thread is detached and captured main's stop flag, mutex, condition variable,
line queue and EOF flag by reference. The server writes QUIT and closes stdin right after it,
so the reader often wakes with EOF while main is returning: it then locked a destroyed mutex
and wrote to a destroyed deque (undefined behaviour on every platform; the Metal port aborted
on it). The state now lives in a shared_ptr that the thread holds a copy of, so it outlives
main's frame for as long as the reader runs. CUDA/HIP and SYCL engines alike; no change to
what the engine reads or answers.
expert_source.cpp's Linux-only STRATA_TEST_UNMAP_GGUF check had a literal 0x00 byte as its
character constant (since 0.1.43; gcc warned "null character(s) preserved in literal"), and the
#2126 buddyinfo parser in pinned.cu got one the same way (a heredoc dropped the backslash).
Both are the escaped character now. Git took pinned.cu for a binary file because of the byte,
so it was committed with CRLF line endings; it is back to LF (no other change).
tools/test_source_nul_bytes.py fails when any tracked source file holds a NUL byte again.
Prompt chunks run the attention scores and P V as FP16 strided-batched GEMMs
with FP32 accumulation (K / V cells and Q converted per chunk, the causal
softmax writing FP16 probabilities), replacing the FP32 SGEMMs. The prompt's
delta rule splits each V head over 4 column groups (16 row groups of 8 rows,
the norm in its own kernel); decode, verify windows and commits keep one
kernel. pp4096 on gfx1201: 2557 -> 3290 tok/s.
The prefill hook hands the prompt's image rows (sp.embd_rows) to the Metal prompt paths (prompt_chain.inc, the
streaming prompt path, the 8-row windows), which take an image row in place of the token embedding.
mrope_table_set keeps the host view of the server's (t, h, w) table; every window, prompt chunk and draft round copies
the rows it reads into the window's buffer, bound at index 20 of qsa_prep with function constant 24 (mac-kern
29724e5e) - the window, both prompt paths and the draft layer. The cached window's QsaArgs are patched by the last
inline argument (the table is bound after it). setup_mac.py --vision yes: the pinned projector (SHA-256) and
strata-vision built with ggml-metal; the config gets --vision and the encoder.
missing.cpp: a crash prints its call stack and fault address.
M5 Max, Q2_0, ISTA's projector, a test picture (shapes and 'STRATA 42'): 'STRATA 42' and 'Red circle / Blue square /
Green triangle', the same answers as the HIP engine on the R9700 (p620) with the same picture and files, also
through the config setup writes. Text greedy unchanged with and without --vision (prompt1, story, the 1,662-token
chain prompt). prefill.cpp: the CUDA (P1004) and HIP (p620) objects are byte-IDENTICAL (preprocessor-only).
The greedy divergence seen with the fused single-token kernels is not a
kernel bug: fused_gdn_conv_l2 and k_conv_seq leave the same history and
differ by 1 ulp in h (FMA contraction of the 4-tap sum); the Q8_1
activation rounding turns that into a flipped near-tie (p01 token 4,
margin 0.43). KL with either is the same. Drop the switch.
Lane tid wrote block tid % 8 of its superblock, so a sub-group's 2-byte stores landed 64 bytes apart. Lane tid now
takes block tid / 4 (the same 32 (block, part) pairs, the same values) and writes its runs with store_run. UD-Q4_K_XL's
down projections are Q5_1 and UD-Q5/Q6/Q8_0's are Q8_0, so this is most of their FP16 prompt-path dequant.
B70, dequant_bench (32 experts, gate/up + down): Q5_1 7.18 -> 2.83 ms, Q8_0 6.77 -> 3.53 ms; Q4_0 3.21, Q4_1 2.76,
Q5_0 3.74 ms after; the FP16 output hash of every type unchanged. dequant_bench lists Q4_0 / Q4_1 / Q5_0 too.
The lane kernels of native_expert_grouped re-read and re-decoded a Q4_K / Q5_K / Q6_K / Q5_1 / Q8_0 expert row once
per entry of its group (the IQ formats already decode once, Multi). Each of these dots is now split into load (the
weight words and scales) and apply (the activations and the same float expression), and the group's entries go
through the existing multi-entry path in chunks of 4 and 2; every entry keeps its own k order and lane sum, so the
outputs are bitwise the per-entry kernels'. STRATA_KQ_SPLIT=0 keeps the old kernels.
B70 (bmg-g31), 2560 x 640 experts, GB/s of expert bytes, 2- / 4-token windows: Q4_K/Q5_1 362 -> 402 / 369 -> 425,
Q5_K/Q8_0 395 -> 440 / 404 -> 464, Q6_K/Q8_0 259 -> 307 / 256 -> 318, Q8_0 430 -> 479 / 429 -> 502; one token -1..2%.
Experts read from pinned host memory (the NO_HOST mirror) go from 18-19 to 23 GB/s on those windows.
Also: STRATA_KQ_GU_LANES (opt-in; the K-quant / Q8_0 gate/up lane split alone, other bits), native_grouped_parity
--kq (split vs per-entry bitwise, and a CPU float reference with ggml's dequantize_row_* for Q4_K/Q5_K/Q6_K/Q5_1/Q8_0)
and --kqbench (per format pair, device or pinned host memory), Q6_K blobs with a finite scale in that harness,
dequant_bench and kq_cpu_bench for the Unsloth formats in the SYCL build.
--mtp D (0..7) loads blk.<n_layers> (eh_proj, enorm/hnorm, attention + MoE
layer, shared_head_norm) and runs greedy requests as speculative steps:
D chained drafts, one verify window, commit of the accepted rows' GDN state
and conv history. Verify windows and plain decode share the per-token GDN
kernels, so the text is the plain greedy text (10 of 10 prompts identical).
Serve reports accepted/offered drafts in DONE.
- moe_mm.metal: W8<Q4_K / Q5_K / Q5_1 / Q6_K> (the lane's runs k and k + 16 share the sub-block's scale, min and
high bits; 4-aligned code runs read as one uint); mm_down3 takes the paired-run decode on the COOP_K4 layout (TK
32). Prompt per layer at T 2048: Q4_K gate/up 10.6 -> 6.0 ms, Q5_K 15.4 -> 9.0, Q6_K 17.2 -> 14.0, Q5_1 down
5.0 -> 3.3; the same values (moe_mm_parity v3 incl. Q2_0 and IQ3_XXS / IQ4_NL)
- qsa.metal: QSA_MROPE (function constant 24) - interleaved M-RoPE (ggml IMROPE, sections 11/11/10): buffer 20 of
qsa_prep = int32 [cells][3] (t, h, w); rotary pair i of the row at pos_base + cell rotates by the cos/sin row
tab[(pos_base + cell) * 3 + i % 3] (pooled indexer keys at pos_base + 4 b too). qsa_parity QSA_MROPE=1 (random
table vs the CPU reference) / QSA_MROPE=id (identity table: the same numbers as the plain kernel)
The HC read takes, per matrix, its load-time Q8_0 copy or the file's own tensor: Unsloth's Q8_0 down/up need no copy,
their F32 inject rows go to gr_down_q8f (mac-kern 945c0d24). router_route_<type> for BF16 and F32 (opt-in). The M4's
v2 prompt kernels: K-quant gate/up TK 128, Q8_0 / IQ4_XS TK 32, Q5_1 / Q8_0 / IQ4_NL down TK 32. GSQ packs unchanged:
Q2_0 prompt1/story, IQ3_XXS prompt1 and the 1,662-token chain prompt give the same tokens as before (M5 Max).
tools/vision/strata_vision.cpp: no posix_fadvise on macOS (it does not have it); strata-vision then builds there with
ggml-metal.
- unsloth_budget / UNSLOTH_RESIDENT_BESIDE_GB 16, ported from sergqwer's fork (sergqwer/strata-nvfp4 3f3546f6, MIT):
no --resident-budget-gib where the experts and 16 GiB fit the RAM (UD-IQ4_XS 71.4, UD-Q4_K_XL 87.7, UD-Q5_K_XL 107.6,
UD-Q6_K_XL 117.7, Q8_0 135.5 GiB), the budget below that, when asked for, and on a unified-memory APU (#1715).
The menu, --check and the install say which; the menu's default for the family is the largest file whose experts
all fit (UD-IQ4_XS where none does).
- UD-Q5_K_XL / UD-Q6_K_XL / Q8_0: six shards each, sizes and SHA-256 pinned at 38bb39ee; "ram_gb" is the all-in-RAM
need, below it setup asks first (default no) and says how many GiB are read from the SSD. Q5/Q6 need engine 0.1.44
(the Q6_K expert kernels in every build); Q8_0 0.1.38.
- UD-Q4_K_XL: ran on an R9700 (docs/QUANTS.md), so #429's AMD question is gone, and it is not experimental.
- tools/strata_mcp.py mirrors the table and the verdicts. Golden configs re-recorded: only UD-Q4_K_XL on 96 GB
(65536, --kv-resident) and 128 GB (131072) changed, as in the fork.
Tests: test_setup_unsloth Resident and BigFiles (the budget tests force the budget mode where their PC now fits every
expert), test_strata_mcp, every tools/test_setup_*.py.
Co-authored-by: sergiywith <[email protected]>
A driver of its own beside Flash-Next's (generate/verify/prefill untouched), built from Strata's kernels:
- weights from the GGUF: dense Q8_0 on the native MMVQ (with the packed Q8_0 copy), the routed experts
repacked at load into native expert blobs (stride in whole blocks for MMQ), F32 norms / projections;
- the plain pre-norm residual layer driver, the residual add fused with the next norm, the MoE end
(combine + shared-expert gate + add + next norm) in one kernel;
- GDN: the fused conv+L2 and SiLU step for a token, one-launch sequential kernels for a prompt chunk;
- attention without an indexer: split-K decode over 256-cell chunks (positions on the device, so the
decode step is one captured graph), and for a prompt chunk scores / values as FP32 GEMMs per KV head
with a causal softmax;
- prompt projections as FP16 GEMMs (prefill::Gemm::native), the F32 router / alpha / beta as SGEMM,
the prompt's experts on llama.cpp's MMQ (entries sorted by expert on the device);
- decode: native router 256/8, native grouped experts (Q5_K / Q6_K downs added to the grouped
kernels' down list for FF 512), router + shared gate in one product, alpha/beta in one launch.
- serve: READY / GEN (sampling keys) / T / PP / DONE / STOP, the held state reused when a request
extends it; --tokens / --force / --dump-full-logits for tools/qwen35/kl.py; --bench-pp / --bench-tg.
- hip_compat cublas_v2.h: cublasSgemm / cublasSgemmStridedBatched.
setup_mac.py: the Unsloth files by name and SHA-256 (setup.py's pinned revision), repacked into one GGUF for the GPU
with every tensor verified, the n-gram table found by name, --native-dense-gguf the repacked file; setup writes
STRATA_METAL_STREAM to the config so the engine follows its choice. UD-* are behind 'missing' (the Metal expert
kernels for Q4_K/Q5_K/Q6_K/Q5_1 are in progress) and only reachable with --model.
stream.cpp: auto streams when the mapped weights + 12 GiB do not fit the GPU working set (was: experts > 60% of
it, which would have streamed UD-Q4_K_XL on the 128 GB M5 Max). M5 Max: Q2_0 stays resident (same tokens and speed);
M4 32 GB: Q2_0 streams as before.
Shard 1's metadata (without split.*) and every tensor but the n-gram table, byte for byte: dense first, then each
layer's routed experts together; --verify hashes every tensor (SHA-256) against its source shard. For Unsloth's
UD-*_K_XL (4-6 shards cutting through layers). Checked on Q2_0 (M5 Max): 35.05 GiB, 1,223 tensors identical,
written and verified in 33 s; the engine on it (--native-dense-gguf the same file, --ple-gguf the original shard 2)
gives the same tokens as on the original shards, prompt1 and story, at the same speed (114.2 / 107.7 tok/s).
From sergqwer/strata-nvfp4 1688a344 (src/prefill/mmq_expert_bench.cpp), with UD-IQ4_XS's gate/up and down formats
added as comparison cases (an f16 d leads their blocks, as Q8_0's).
Co-authored-by: sergiywith <[email protected]>
row_dot_20_sub8: lane t takes k = t (+ k = t + 16 for t < 4, else + 0) and k = t + 8 (+ 0), then xor 4, 2, 1 - the
order warp_sum adds a K = 20 row in (lane i + 16, then + 8), so the result is the same bits; native_down_multi_kernel
routes TD 40 there (as IQ4_NL's K = 40 row_dot_40_sub8) and the down grid takes 32 rows a block for d_type 40.
nvfp4_decode_bench, RTX 3060, 64 distinct experts: down alone 0.260 -> 0.189 ms at 1 token (226 -> 312 GB/s, 87% of
360); the whole expert 0.626 -> 0.554 ms (89%), 2 tokens 0.686 -> 0.596, 3 0.786 -> 0.661, 4 0.910 -> 0.766 ms (from
2.261 before Split<40>). Output hashes unchanged, nvfp4_expert_gpu_parity unchanged.
The ported NVFP4 decode experts had no Split, so they took the per-entry kernels: every weight was decoded again for
every token of a verify window, and the kernel slowed almost linearly with the tokens per expert. Measured with
nvfp4_decode_bench on the RTX 3060 (64 distinct experts, sm_86, CUDA 13.4; best of 50):
| tokens per expert | before | after | |
|---|---:|---:|---:|
| 1 | 0.714 ms (248 GB/s) | 0.626 ms (283 GB/s) | 1.14x |
| 2 | 1.200 ms | 0.686 ms | 1.75x |
| 3 | 1.730 ms | 0.786 ms | 2.20x |
| 4 | 2.261 ms | 0.910 ms | 2.48x |
(UD-IQ4_XS's IQ3_S / IQ4_NL experts, 15% fewer bytes: 0.512 / 0.578 / - / 0.814 ms.)
- Split<40>: vec_dot_nvfp4_q8_1 split into load (the E2M1 codes through kvalues_fp4, the two sub-block scales) and
apply (the dp4a chain and the float expression in the same order), so the multi-entry kernels apply one decode to
up to four tokens. The multi kernels now apply the NVFP4 tail scales on their outputs as the per-entry kernels do
(s_gate / s_up on gate/up, s_down on down; x * 1.0f for every other format, which is exact).
- ue4m3_to_f32_half builds the value from its bits instead of two ldexpf and branches: equal for all 256 bytes
(nvfp4_decode_bench checks them).
- down: NVFP4's rows are K = 20 calls at n_ff 640, as Q2_0's, so they take row_dot_20_sub16 (2 rows a warp, 16 rows a
block) instead of one 20-of-32-lane row a warp: down 0.361 -> 0.260 ms at 1 token. Its addition tree is
warp_sum's (lane i + lane i+16 first), bitwise.
Bit for bit: the output hash of every case above equals the per-entry kernels' (STRATA_OLD_IQ_MMVQ=1);
nvfp4_expert_gpu_parity on real GPTQ blobs unchanged (1.088-1.168%). Other formats: only the store multiplies by a
1.0f scale, and the down launch grid changes for d_type 40 alone.
nvfp4_decode_bench [gu_type d_type] [groups] [tokens per group] [iters]: native_expert_grouped on distinct random
blobs (64 x 2.76 MB, far past L2), best/mean time, weight GB/s against the card's DRAM peak, and an FNV hash of the
output so a kernel change can be checked bit for bit. NVFP4_BENCH_PHASE=1|2 times gate/up or down alone. Also an
exhaustive check of the UE4M3 decode built from bits against the ldexpf form (all 256 bytes).
- fused_gdn_step_norm_silu: the fused GDN step with the SiLU output gate (Qwen3.6), a compile-time
template flag; fused_gdn_step_norm (Flash-Next, sigmoid) is the same kernel instantiated as before.
- native_router_topk(n_expert, k): 512/10 goes to the existing pinned top-10 kernels unchanged; 256/8
is a templated copy of the same topk-moe contract.
- strata-q36ref --kernels strata (default): native MMVQ (q8_1) for every quantized matmul, dense and
per routed expert (Q4_K/Q5_K/Q6_K/Q8_0), the fused GDN conv+L2 and SiLU step, the native router;
--kernels plain keeps the reference kernels.
- kl.py: answer-only statistics beside all steps, llama.cpp's logits cached between runs.
- quant.metal: QUANT_CTAB (function constant 23): the IQ3_XXS / IQ3_S / IQ2_XXS / IQ2_XS / IQ2_S grids and the sign
table read from ggml-common's constant arrays, no threadgroup copy (same values). Decode moe_gu: 3-7% faster
(IQ3_S-pack T2 window 14.96 -> 14.79 ms, 3 of 3); token_chain default (CHAIN_CTAB=0 off). Prompt mm_gu3: slower
(kept off)
- moe_mm.metal: mm_gu4 (decoded operand shared by MS 16-row token tiles): IQ3_S 9.5 -> 8.6 ms, IQ3_XXS equal, Q2_0
slower; mm_gu5 (weight blocks staged in threadgroup memory, coalesced): slower (12.9-20.6 ms); W8 templated on the
address space for IQ3_S / IQ3_XXS / IQ2_S; probes MM_PROBE 5 (loads only: as slow as the full decode)
- moe.metal MOE_PROBE (function constant 22; no table load)
- finding: the prompt i-quant decode is bound by its scattered small device loads (2-7 a 4-value run, 8 rows a load
instruction), not by the grid lookups or the ALU
--family nvfp4 --model NVFP4 | NVFP4-NVIDIA (MODEL-UPDATE Phase 4). The files come from the packs' Hugging Face
repositories at the commits the fork's setup pins (NVFP4_REPOS: Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata
640a1db9, ...-NVIDIA-NVFP4-Strata cca8f7fe), each checked against its size and SHA-256; nothing is converted on the
PC and no file is mirrored. The FP8 n-gram table and the BF16 embedding are the same file in both: a second install
links them. The MTP draft layer is setup's own, as for every model. The engine gets the fork's model flags
(--native and --native-dense-gguf on the dense GGUF, --ple-gguf, --embd-gguf); the rest of setup's choices (context,
KV, the low-RAM mode on the pack's experts.bin) are the usual ones: 96 GB keeps every expert in RAM, 64 GB with a
32 GB card the resident low-RAM mode. NVIDIA only: on AMD setup stops and names UD-IQ4_XS (the HIP engine has no
NVFP4 kernels). Images: off for these packs. A ready-made engine older than NVFP4_ENGINE (0.1.47, to be set at
release) is replaced by one compiled from this source.
Tests: tools/test_setup_nvfp4.py (fetch from a file:// mirror with SHA-256 checks, the shared file linked, a wrong
file refused, the config name read back, mocked installs on 128/96/64 GB, AMD refused); test_setup_pins covers the
new table. Every tools/test_setup_*.py passes.
Co-authored-by: sergiywith <[email protected]>
- sycl: iq_dequant_f16 / iq_dequant_gu_f16 / moe_combine accept the NVFP4 scale arguments the shared headers now
declare (refused if ever given: the SYCL port has no NVFP4 experts), and the no-MMQ stubs match the new mmq
signatures (the calls were ambiguous with the defaulted ones).
- CMake: list(TRANSFORM ... REPLACE) with \2 of an optional group failed on CMake 3.28 for a plain "120"; two
rewrites, one per form.
- src/qwen35/q36ref.cu (target strata-q36ref, CUDA and HIP): Qwen3.6-35B-A3B on one GPU, the whole
model resident, one token at a time with fp32 activations: plain pre-norm residual, GDN with the
SiLU output gate (32 V heads over 16 K heads, conv 8192), dense gated attention (16 Q / 2 KV x 256,
NEOX RoPE over 64 dims), softmax top-8 router over 256 experts with the sigmoid shared-expert gate,
every weight read in its GGUF form (F32/F16/BF16/Q8_0/Q4_K/Q5_K/Q6_K). --act q8 (default) rounds the
activations of quantized matmuls as llama.cpp's GPU path does; --force / --dump-full-logits for
teacher-forced KL; --dump-layers / --dump-logits for parity.
- tools/qwen35/: q36_llama_dump.cpp (llama.cpp's side through the eval callback), parity.py
(greedy answers + per-layer error), compare_dumps.py (two runs, e.g. the GPU-vs-CPU floor), kl.py
(teacher-forced KL against llama.cpp HIP with the CPU backend as the floor), parity_prompts.json.
The fork was built with MSVC only: on Linux its parity tool's F16C rounding helper failed to inline without the
target flag, and the SYCL tree's own list of CPU expert sources lacked nvfp4_avx512.cpp / nvfp4_avx2.cpp.
Port of sergqwer/strata-nvfp4 c83625c3 (the rule for NVFP4 experts in any layer) onto 0.1.43's default, with its
#1884 nearly-all-cached exception kept. On the fork's qwen-nvfp4-gptq pack the skip moved the answers by
teacher-forced KL 0.035 against it off, about the GPTQ quantization's own error against an all-Q8_0 reference
(0.038), for +11.5% decode (RTX 5090, the fork's measurement). Packs without NVFP4 experts keep the default
unchanged. The final default is the owner's, from our own 3060 A/B and KL against Q8_0.
Co-authored-by: sergiywith <[email protected]>
Without AVX-512 (Zen 2/3, Intel 12th-14th gen) the pool's NVFP4 rows went
through ggml-cpu's vec_dot, which re-decodes a row's blocks for every token of
a window. nvfp4_256_* decode a 64-value block once and apply it to up to 8
tokens. Each token's arithmetic is ggml_vec_dot_nvfp4_q8_0's AVX2 path step
for step (vpsignb, vpmaddubsw + vpmaddwd, the same FMA order, hsum_float_8),
so every row is bit-equal to it: an AVX2 CPU computes what it did before.
nvfp4_avx512_parity checks both kernels, whichever this CPU runs (the AVX2
rows must also be bit-equal; 0 differing rows of 1..9 tokens against the
portable and the native ggml-cpu builds), and is now a ctest
(nvfp4_cpu_parity). One expert's gate+up, Zen 5 core: 1.12x ggml at one
token, 2.00x at 4, 2.35x at 8; under DRAM load (--bw) with 6 threads 47.3
against 24.0 GB/s at 4 tokens, 26.6 against 12.0 at 8. Chat decode on the
RTX 5090 with STRATA_FORCE_AVX2=1 and 6 pool workers: 16.5-16.9 ms a round
against 17.3-17.5 (pool 6.6-6.8 against 7.2-7.3 ms).
STRATA_NO_NVFP4_256=1 falls back to ggml-cpu.
(cherry picked from commit 0c859e15f89054a376db7458bfb298e3e6c81f22 of sergqwer/strata-nvfp4)
Upstream's 04dbe561 allocates cpu_rows at its largest before the walk; the fork's allocation of cpu_ones (row_sd 1 for
the CPU's rows on an NVFP4 pack, b59f5dda) sat inside the reallocation that then never ran, so a staged chunk with the
CPU share copied from a null host pointer: "prefill moe_combine: invalid argument" (2K prompt, qwen-nvfp4-gptq).
(cherry picked from commit 52b272ebb92c9ef49029314da1a695d38f570d88 of sergqwer/strata-nvfp4)
Was 640df4f0, the fork's CPU share. Upstream 0.1.40.2 took it (978d3558, with 667f2eca's blobs taken on the
prefill thread and 4322e241's STRATA_DBG_NAN rows; opt-in there), so the port keeps upstream's code and adds only
what an NVFP4 pack needs: the CPU's rows carry s_down already, so their row_sd entries are 1 (cpu_ones, pinned
beside cpu_rows).
Port note: the release loop keeps 0.1.43's steps_up / tok_up / up_ring releases.
(cherry picked from commit a0c2ca233ac8fe8ec9d342bd319c93404791fa15 of sergqwer/strata-nvfp4)
- The blob keeps its {s_gate, s_up, s_down, 0} tail whenever any projection is NVFP4; the other projection's scale is
exactly 1.0, and every path (GPU decode, CPU pool, prompt path) already multiplies by the tail, so 1.0 is exact.
native_expert_layout (GPU) and native_fmt (CPU) no longer require NVFP4 in all three projections.
- tools/iq_pack.py writes such blobs (1.0 for a non-NVFP4 projection, .scale tensors only for the NVFP4 ones).
- native_expert_parity: the synthetic NVFP4 blob gets its tail (it was left zero, so every NVFP4 synthetic expert
computed 0 and failed), non-unit for NVFP4, 1.0 for the other projection, and the reference applies it.
nvfp4/q8_0, q8_0/nvfp4, nvfp4/nvfp4 and q8_0/q8_0 pass: CPU and GPU within 1e-7, ~1.2% from the float reference
(activation rounding).
(cherry picked from commit 473c0959141b36d28e7a2eb16005f8c8a4e3c8d0 of sergqwer/strata-nvfp4)
0.1.38 took the fork's group gather without the NVFP4 parts: each expert's 16-byte scale tail now goes to its
group slot and the group's last flush zeroes the MMQ tail, in the same launch (as #353 on 0.1.38).
(cherry picked from commit 8f2193f028940b9a6d270203b640562435937a7a of sergqwer/strata-nvfp4)
A ModelOpt NVFP4 checkpoint of Qwen3.8-Flash-Next (its routed experts NVFP4, group 16) runs on the native path.
Rebased on 0.1.31: NVFP4 is type 40 in the per-role format lists (STRATA_GU_FMTS / STRATA_D_FMTS / STRATA_MMVQ_FMTS)
beside Q4_K/Q5_K/Q5_1/Q8_0, and 0.1.31's dq_q8_0 serves both.
- tools/nvfp4_convert.py: llama.cpp's converter with the engine's type policy (BF16 control projections, Q8_0 native
tensors and token embedding, no PLE table); tools/nvfp4_verify.py / nvfp4_verify_gguf.py prove the experts bit for
bit.
- tools/iq_pack.py: expert_layout() gives NVFP4 blobs a 16-byte tail {s_gate, s_up, s_down, 0} (the per-expert
weight_scale_2), so every consumer that moves whole blobs moves the scales; an NVFP4 pack always writes experts.bin
(the GGUF has no tails, so the in-place GGUF read does not apply).
- decode: NVFP4 in the native expert kernels (q8_1, UE4M3/E2M1 decoded in software, bit-identical to ggml); each scale
on its own projection's FP32 output (folding s_down into up put the hidden into FP16's subnormals).
- CPU pool: AVX-512 NVFP4 rows (1.8-3.7x ggml-cpu's), ggml-cpu elsewhere.
- prompt path: llama.cpp MMQ - W4A8 by default (int8 MMQ with Blackwell's FP4 MMA hidden, mmq_nvfp4_w4a8.cu), W4A4
and FP16 through STRATA_PREFILL_NVFP4; the scales applied where they are read.
- CMake: the architectures stay as given. Only mmq_nvfp4_w4a4.cu (the FP4 x FP4 kernel and its FP4 activation
quantizer, symbols renamed) is built with 12x -> 12xa in strata_mmq_w4a4; without its image W4A4 falls back to W4A8.
strata.exe: 55 sm_120 cubins + 1 sm_120a.
- 0.25 of the missed experts over PCIe for NVFP4 packs (their blobs are 2.76 MB).
- Parity tools (mmq_nvfp4_parity, nvfp4_avx512_parity, nvfp4_expert_gpu_parity); HIP guards; docs/NVFP4.md.
Checked on this rebase (RTX 5090): the parity tools as before (decode experts 1.09-1.16%, MMQ W4A8 0.53%, W4A4 8.6%
- the same numbers as the 120a-everywhere build); IQ2_XS 128 greedy tokens identical to 0.1.31 main; the NVFP4 pack
against the 0.1.29-based build at the same expert-cache size: identical tokens on a 24-token and an 8K prompt, in W4A8
and in W4A4 (prefill 4,231 / 4,583 tok/s on 8K).
Port note: the PCIe share base for NVFP4 packs (0.25) sits beside 0.1.43's PcieAuto start values.
(cherry picked from commit 1f8a12858cc4ed80a977767afef6b6c3a754ac63 of sergqwer/strata-nvfp4)
- serve/images.py: OpenAI's images API (prompt, size, n, response_format b64_json|url) plus seed and steps; one
sd-cli process per request with the model card's arguments (8 steps, CFG 1, euler, Turbo's sigmas,
--diffusion-fa), the PNGs returned unchanged. The job takes the service's turn (one job at a time on the GPU)
and the config's gpu_policy says what the text engine does meanwhile: keep, shrink (--vram-elastic), unload
(its conversation saved to a session file and restored after the next start), auto.
- setup.py --images qwen-image (--image-quant, --image-backend, --image-gpu-policy): sd.cpp 2988060 + its ggml
as zips, built for CUDA / HIP / Vulkan (LunarG SDK into third_party when the PC has no Vulkan dev files) / CPU;
DiT quant by GPU memory (Q4_K_M <= 8 GB, Q6_K <= 12, Q8_0 above), VAE and the Qwen3-VL-8B UD-Q4_K_XL text
encoder at pinned revisions, each checked against the Hub's SHA-256. tools/image/install.py adds it to an
existing install.
- Web page: an Images tab when the server has image generation.
- docs/IMAGES.md; tests: serve/test_images.py, tools/test_qimage.py.
IQ3_S runs on Metal since mac-kern's IQ4_XS kernels: recommended from 110 GB (Apple's 128 GB Macs), IQ3_XXS where it
fits resident, then Q2_0. A pack that does not fit Metal's share of the memory streams its experts from the SSD (the
engine decides, f-mac-ssd): setup says so with the M4 numbers, writes --suffix-draft 0 and no --mtp (one token per
step, measured faster there) and does not fetch the draft layer.
session_zero and conversation_checkpoint_save/restore go to the Metal window under STRATA_GPU_METAL (the running
state lives there, not in the session's buffers): reset zeroes its GDN recurrences and conv histories, PLE history
and indexer tails (and the draft layer's); a checkpoint holds the same set, after the pending fold. Before this a
new request carried the last conversation's state (M5 Max, Q2_0: an unrelated request after a story answered
differently from a fresh engine; now the same), and on a streaming Mac the session buffers it zeroed or copied had
been given back (the engine stopped).
Checked on the M5 Max (resident) and the M4 (streaming): an unrelated request after another equals a fresh
engine's answer; a three-turn conversation reuses 35 / 73 of its tokens and its second answer equals a fresh
engine's. Shared files: session.cpp and conversation_state.cpp preprocess IDENTICAL and their objects are byte-
IDENTICAL on CUDA (P1004) and HIP (p620). Also: IQ4_XS's v2 prompt tile (_16_64_32_2), and a call stack when a
given-back allocation is touched.
- ModelFamily from general.architecture (qwen4exp, qwen35moe); the architecture guard accepts both, the
engine's guard (native dense/head, --native) refuses Qwen3.6 with "recognised, not run yet".
- ModelGeometry moved to core/geometry.hpp (+ family, shared-expert FFN, in-file MTP layers, vocab);
geometry_from_gguf reads every shape key, checked against the compiled kernels (Flash-Next files give
exactly the defaults; expert count and top-k stay run-time sizes).
- Per-family tensor layout tables (GDN / attention / nextn MTP block, hyper-connections or plain
pre-norm, indexer or none) and a GGUF directory check; MXFP4 block size in the reader.
- setup: --family qwen36 downloads unsloth/Qwen3.6-35B-A3B-MTP-GGUF (pinned 5bc3e238, size + SHA-256)
UD-Q4_K_XL / UD-Q6_K_XL / Q8_0 (+ mmproj) and says the engine does not run it yet; serve's default
model name per family; strata_inspect recognises qwen35moe.
- Tests: model_family_test (synthetic, --real for headers), test_setup_qwen36, serve/test_family_name.
Conflicts in metal/src/engine/window.cpp resolved: the stream's next-layer routing guess beside router_route
(opt-in); the cached window through the stream's segmented run, the kernel profile off while streaming; host reads of
the weights through W_ (the mapped file, or the dense weights in memory when streaming), also in prompt_chain.inc;
the drafter's prompt rows batched, from either prompt path. The prompt: resident -> prompt_chain.inc, streaming ->
the stream's prompt path (it streams each layer's experts for the chunk); STRATA_METAL_PROMPT=window: 8-row windows.
M5 Max: Q2_0 prompt1/story and IQ3_XXS prompt1 with MTP equal to the no-MTP references; the 1,662-token prompt
through prompt_chain.inc equal to before; Q2_0 streamed (STRATA_METAL_STREAM=1) equal to resident on prompt1. The
1,662-token prompt streamed (the stream's prompt kernels: chunks of 1024, flash QSA) first differs at token 21 of 60
from the resident prompt_chain.inc path (different kernels and chunking).
- gemv.metal: gemv_x<F,LPR,NR,SG> = gemv (KS 1) with the window's prepared x staged in threadgroup memory (bitwise
the same arithmetic); GEMV_PROBE 1/2/3 bench probes (skip later tokens' x loads / combine; rows-outer order)
- units.metal: X<F>::load templated on the pointer's address space (device or threadgroup), same arithmetic
- gemv_parity: GEMV_X=1 sweeps gemvx_*, GEMV_PROBE passes the probe
- M4 finding: a window token's cost is the fp32 combine (1 FMA a weight a token), at ~0.85 T FMA/s (near the M4's
peak); the x loads are ~6 us of the ~30-60 us a token, and staging x in threadgroup memory does not help
setup.py hands a darwin run to metal/setup_mac.py (nothing else changes on a PC); setup.sh fetches a pinned
standalone Python 3.12 into .python on a Mac without Python 3.10+ (no sudo, no Homebrew). The Mac setup checks the
chip, unified memory, macOS (15+) and the Command Line Tools; recommends the pack by memory (IQ3_XXS from ~96 GB,
Q2_0 below; IQ3_S only with --model: the Metal expert kernels lack its IQ4_XS experts); builds the engine with pip's
cmake/ninja, Apple's metal-cpp (pinned SHA-256) and llama.cpp's ggml at the pinned commit; prepares the native pack
and the MTP draft layer; writes strata-<model>.json (backend metal, --spec 2) and run-<model>.sh; starts
serve/server.py. tools/test_setup_mac.py: 10 tests with sysctl/sw_vers/xcode-select mocked.
On the M5 Max (m1002): ./setup.sh --model Q2_0 --gguf-dir ... --yes from a fresh copy: Python fetched, engine built,
pack and config written in 29 s (models already downloaded); the server answered /v1/models, the chat page,
streamed /v1/chat/completions and /v1/messages (Q2_0, 112-121 tok/s). python -m pytest serve -q on the Mac: 705
passed, 11 skipped.
The stdin reader (a detached thread) can take in_mu after the serve scope destroyed it, when the server closes the
pipe after QUIT; on macOS that lock throws and the engine ended with an abort. Inside #if STRATA_GPU_METAL: flush
and _Exit(0) after the QUIT profile save. HIP (p620): generate.cpp preprocesses IDENTICAL and its object is
byte-IDENTICAL against the gated tree (P2003 was unreachable for the CUDA compare; the change is preprocessor-only).
The serve loop asks cvec() every request: Metal returns the empty one (no --control-vector-scaled on Metal yet)
instead of the stub that stopped the engine; stubs.cpp regenerated without it. An uncaught exception now prints its
call stack (backtrace_symbols_fd) before the abort.
The BF16 router GEMV and moe_route as one dispatch beside the shared expert's GEMVs. Greedy output unchanged (Q2_0,
IQ3_XXS, both prompts, on and off). Not the default: in this window the router GEMV already runs beside the
shared GEMVs, and the T2 window measured 13.035 ms GPU with it vs 13.027 without (the fused kernel alone 0.615 ms
vs 0.566 for the two); end to end, --spec 2, 3 interleaved pairs, off -> on: Q2_0 113.61 -> 113.28 / 107.50 ->
106.97, IQ3_XXS 103.51 -> 103.07 / 96.19 -> 96.02 tok/s.
- router.metal (new): router_route<F> - the router GEMV rows as gemv's (LPR 32, KS 1) with 512-thread threadgroups,
the last threadgroup to finish (device counter, device-scope fences) runs moe_route's body and resets the counter;
logits / ids / w / plan bitwise gemv + moe_route. token_chain default (CHAIN_ROUTE=0: separate): T2 13.08 -> 12.93 ms
- gemv.metal gemv_cat: two same-format matrices over one x in one dispatch (ssm_alpha + ssm_beta), bitwise; no gain
in graph mode (CHAIN_ALBE opt-in)
- gr2.metal gr_norm_comb: the MoE combine folded into gr_norm, bitwise; +0.6 ms at T2 (CHAIN_COMB opt-in, not kept)
- units.metal Dec<Q3_K>: qs / hmask as packed_ushort4 loads (same values): T2 2560->10240 33.8 -> 32.2 us
- tests/fuse_parity: the three fusions bitwise for 1..4 tokens
With it a verify window of T > 1 commits its first row in place (commit = 0x100 | 1, mac-kern 56526aed): commit(1)
needs no fold, commit(n) folds rows 1..n-1; a prompt window commits all its rows in the step. Greedy output equal
to no MTP (Q2_0, IQ3_XXS, both prompts, --spec 2 and 4, the window prompt path).
Not the default: M5 Max, --spec 2, 3 interleaved pairs, off -> on: Q2_0 113.35 -> 113.24 / 106.98 -> 106.75,
IQ3_XXS 103.53 -> 102.71 / 96.18 -> 95.88 tok/s. The in-place state write (36 x 3 MB) costs every window; the
fold it saves is only the rounds with no accepted draft (24-36% at --spec 2).
- gdn.metal: with commit = 0x100 | k (1 <= k <= GDN_NTOK) the state is written in place and the conv history to
hist_out right after token k - 1, while every token's y is still computed; commit = 1 / 0 unchanged (the full
commit now writes at the last token inside the loop - same values, same speed)
- gdn_parity: the partial commits are bitwise a k-token commit's state and history, and y bitwise the verify window's
(k = 1 .. ntok - 1, ntok 2..4); gr_parity GR_DOWN=SG_KS picks gr_down's variant (sweep: 10.1-11.8 us at T 2, none
better than 8_8 by more than 7%)
Once, after the 20th cached window of T tokens with the head: the window's dispatches grouped by pipeline and weight
role, each group run by itself (7 runs, median). Profiling only: the reruns move the state, so the text after it
is not meant to be read.
Owner decision. Only the Options default under STRATA_GPU_METAL; CUDA/HIP/SYCL keep 0: on P2003 the CUDA
generate.cpp translation unit preprocesses IDENTICAL and its object is byte-IDENTICAL.
M5 Max, MTP, 200 tokens, greedy output equal to no MTP in every run: Q2_0 114.0 / 107.9 tok/s (prompt1 / story,
--suffix-draft 0; was 112.0 / 86.2 at --spec 4), 119.5 / 108.2 with the default suffix drafter; IQ3_XXS 103.5 /
96.9, 108.9 / 97.0 (was 110.2 / 87.4).
The GDN step writes the prepared y planes (no xprep dispatch per GDN layer); a window whose context is within the
2,051-cell budget skips the QSA score and top-k (qids_ holds the identity, restored after a selecting window; the
cached encode is keyed by it). STRATA_METAL_FUSE=0: off.
M5 Max, Q2_0, 200 tokens, 3 interleaved pairs: no MTP 85.1 -> 87.4 (prompt1) and 85.2 -> 87.3 (story); MTP 110.7 ->
112.0 and 85.3 -> 86.2 tok/s; tokens identical on and off and equal to the no-MTP reference.
prompt_chain.inc: tests/prompt_chain.cpp's chain (6f6c555b) into the decode window's own state - fp16 GEMMs on the
matrix units, grouped expert GEMMs, the GDN prompt scan, QSA K/V and indexer keys into the window's caches - plus
the PLE block after layer 0, every row's final residual for the drafter (its prompt rows in one command buffer),
the history parity, RoPE table and pending fold. Chunks of 2048 rows (STRATA_METAL_PROMPT_CHUNK); COOP_K4 picks
the register-operand kernels on the M5 and the older ones elsewhere. The 8-row-window path stays as the fallback
(STRATA_METAL_PROMPT=window, and prompts under STRATA_METAL_PROMPT_MIN = 32 rows).
M5 Max: Q2_0 4K/32K prompts 1,434/1,462 tok/s (window path 190/174), IQ3_XXS 1,260/1,240. Teacher-forced KL of
the chain's prompt vs the window path's, 28 prompts: 0.0004 (top-1 99.4-99.6%); needles 4K-32K x 10/50/90: the
same answers as the window path (12/12, both models).
- gdn_prompt.metal: gdn_cprep (per head and chunk: gates in log space, A, forward substitution for U and W),
gdn_cstate (per head and 32 state columns: the chunks in order), gdn_cout (the intra-chunk output)
- gdn_prompt_parity: CB -100 = the chunked form; y 3e-7, state 2e-7 vs the decode step; 7.4 ms vs 1.9 ms at T 2048
(cprep 1.5, cstate 4.7, cout 1.4) - it needs the matrix units to pay off
- moe_mm.metal: the paired-run (k, k + 16) decode is taken on the COOP_K4 layout unconditionally (coop_probe now
checks that pairing on both tiles used); the runtime per-element check cost ~30%. The MMA-only ceiling of mm_gu3
(MM_PROBE=1: no decode) is 2.45 ms a layer = 55 TFLOP/s useful, so the decode is what is left. Measured and not
kept: fp16-math Q2_0 decode (slower), a W8<Q2_0> sharing the block scale (slower than two W4)
- glue.metal slc_touch + token_chain CHAIN_PREFETCH: the next HC read's weights touched concurrently with the first
projection GEMV after a read; measured +0.83 ms a token (10.46 -> 11.29, 3 of 3 interleaved) - not kept, opt-in
A prompt of 64+ tokens runs in chunks (STRATA_METAL_PROMPT_CHUNK, default 1024 streaming / 2048 resident) through
mac-kern's prompt kernels (fp16 GEMMs, the GDN scan, QSA over the chunk, grouped expert GEMMs v2 on the M4, v3 on the
M5), the PLE block for T rows, on the decode window's GDN / K/V / PLE state; the draft layer's prompt K/V read the
chunk's final rows. Streaming: the expert GEMMs address arena slots (function constant 20 in moe_mm.metal too), a
chunk's handshake puts the layer's experts in place and streams the next layer's non-resident experts into the LRU
tail behind the compute (the decode-hot experts stay); slots in use by the running layer are pinned. The host
code's own device weight copies are given back on a streaming Mac (purge_device_allocations).
STRATA_METAL_PROMPT_GEMM=0 / _MIN, STRATA_METAL_PROMPT_EXPERTS=v2|v3, STRATA_METAL_PROMPT_QSA=v1|flash.
- qsa_attn_mm: the P V matmul in K = 32 cell runs (K = 64 runs computed garbage on the M4), the output through
threadgroup memory (a direct matmul2d store into 258-float rows lost the threadgroup on the M4); also faster on the
M5 (8.1 -> 7.4 ms at pos 0)
- qsa_flash: the same K = 32 runs (was a hang on the M4) and a fresh per-chunk accumulator folded into the running
one (on the M4 a run does not accumulate into elements rescaled in place); now about equal to attn_mm + merge on
the M5 (9.4 vs 9.5 ms at pos 0, 20.1 vs 20.3 at 30000) and the prompt_chain default there (PC_QSA=flash|mm|v1);
the M4 defaults to v1 attention (faster there) with the GEMM score
- qsa_attn_mm bisection probe (function constant 8, QSA_STOP), qsa_prompt_parity QSA_DEBUG / pipeline report;
coop_layout prints the destination layout too
The dense weights are read into one resident buffer; the routed experts live in an arena of slots filled from the
GGUF with F_NOCACHE preads (several pieces of an expert in flight); the window runs in 48 segments with a shared-event
handshake after each layer's router (mtl_rt run_graph_seg), where the host puts the missing experts in place (LRU)
and reads ahead the experts the next layer's router predicts from this layer's FFN input. The expert kernels address
slots through smap (function constant 20); the resident path is unchanged. STRATA_METAL_STREAM=auto|1|0.
- moe_mm.metal mm_gu3/mm_down3 and qsa_prompt.metal qsa_attn_mm/qsa_flash: with COOP_K4 = false every element is
decoded on its own (the M4 holds runs of 2 n at one k); the experts are correct on the M4 that way
- tests/coop_probe.h: probes the layout (COOP_K4=0/1 overrides); moe_mm_parity, qsa_prompt_parity, prompt_chain
set the constant; prompt_chain defaults to experts v2 and QSA v1 without it (faster / correct on the M4)
- qsa_prompt_parity: NaN-fills the partials before the matrix-unit run (a kernel that writes nothing shows),
line-buffered output, QSA_NOFLASH
STRATA_METAL_LOGPOS=<path>: per decode position of one-token windows, the log-probability of the token that came
next (with --spec-follow, the forced text) and the top 64, in phaseA-tests cuda_kl.py's row format.
STRATA_METAL_PROFILE=1: per window size, host time before the GPU, run_graph wall vs GPU time, commit and draft
round times, printed when the window goes away. Both off unless set.
commit(n) after a window of several tokens no longer submits its own command buffer: the fold of the kept prefix is
encoded first in the next window's (cached per pending count). Nothing reads the GDN state in between.
STRATA_METAL_LAZY_FOLD=0 folds at commit as before.
M5 Max, Q2_0, MTP --spec 4, 200 tokens, 3 interleaved pairs: commit 0.87 -> 0.05 ms a round; prompt1 109.25 ->
110.21 tok/s, story 83.99 -> 85.14 (medians); greedy output equal to no MTP, lazy and not.
0.1.42 let --batch-mtp run on a layer split and made the --batch-groups auto default give way to it: one serial group. On
2x RTX A4000 (IQ3_XXS, 3 pairs) four streams aggregated 62.2 tok/s against 92.7 on 0.1.41's pipelined groups (-33%, 0/3 pairs
faster), two streams 55.7 against 71.4 (-22%). With --batch-groups 2 (which turns --batch-mtp off) 88.3 and 71.1. The start-up
line says why --batch-mtp is off and how to keep it. --adapt-async 1 still gives one group. Solo decode and the answers are
untouched.
- qsa_prompt.metal: qsa_score_mm (fp32 MPP, relu-sum epilogue, the tail block scalar; 6.2 -> 1.1 ms at pos 30000,
T 2048), qsa_flash (whole attention a threadgroup with an online softmax, fp16 out; correct but 27.6 ms vs 20.9 for
attn_mm + merge - not used)
- moe_mm.metal: W8<F> decodes a lane's two runs of a 32-group with one read of the group's scale/sign fields;
IQ3_XXS gate/up 9.3 -> 6.7 ms a layer
- prompt_chain: PC_QSA=mm uses qsa_score_mm too; gate/up 16,32,32,4 for every format; breakdown sees the far
chunk's classes
msl/mtp_chain.metal: mtp_pick (the pick through the draft vocabulary and its softmax probability) and mtp_embed_F
(the picked token's embedding row through mac-kern's unit decoders), so every draft step of a round runs in one
command buffer. At load the GPU embedding is checked against the host's dequant (max |diff| 0 on Q3_K and IQ3_S;
any difference falls back to a command buffer per step). STRATA_METAL_MTP_CHAIN=0 keeps the per-step path.
Fix: with accepted drafts (a > 0) draft 0's embedding overwrote the catch-up's row 0 before the GPU read it; draft
0's row now goes to memb_ row a. The dense attention's cell list is per step.
M5 Max, 200 tokens, --spec 4, greedy output equal to no MTP in every run: Q2_0 109.5 / 84.4 tok/s (was 101.4 /
80.0), IQ3_XXS 107.9 / 86.0 (was 101.5 / 82.8).
- MTP (src/core/mtp.cpp hook under STRATA_GPU_METAL: load / bind / prefill / draft / draft_first): the draft layer
of mtp.hpp on mac-kern's kernels, from the MTP GGUF (tools/mtp_pack.py, beside --mtp's rt dir; the F32 norms
folded to 1 + w as tools/mtp_rt.py does): input branches (msl/mtp.metal: rms, strided xprep, stream combine),
the three HC reads, K/V and indexer keys through qsa_prep into its own cache, dense attention over the last
2,051 cells, the MoE (fused gate_up experts), the final mixer and the head over data/draft_vocab.bin's subset
(STRATA_METAL_DRAFT_VOCAB=full: the whole head). One command buffer per draft step (the next step's embedding
is dequantized on the host). Greedy output with MTP equals the output without it, token for token (Q2_0, two
prompts, BF16 and Q8_0 HC); acceptance equals the HIP engine's on the same prompts (124 of 228; 104 of 291 vs 285).
- Prefill (src/prefill/prefill.cpp hook): init brings up the decode window, run reads the prompt through windows
of 8 rows (no head) and feeds on_chunk (the MTP's K/V); draft_kv declines. Prompt windows of 8 rows give the
same tokens as one-token windows. generate.cpp's own Metal prompt loop is gone again (--prefill auto as on the
PC). One window per process (the prompt path may bring it up before the Verifier).
- msl/ple.metal: the PLE block's norms, gate, conv and residual update on the GPU (same tokens as the CPU version).
- The HC read in Q8_0 by default (the owner's decision; STRATA_HC_Q8=0: BF16): the down / up / inject matrices
quantized at load (290 matrices, 680 MB, 70 ms), read by gr_down_q8 / gr_up_q8.
- The encoded window is cached per (T, head, history parity) and only its positions are patched per window.
- f-mac-kern merged up to 4775d233.
M5 Max, Q2_0, greedy, 200 tokens: Q8_0 HC 85 tok/s without MTP; with MTP 101 tok/s (acceptance 0.54) and 79 tok/s
(acceptance 0.36, a prompt the drafter does badly on everywhere: HIP 71 tok/s there).
- moe_mm.metal: mm_gu3 / mm_down3 - weights decoded per lane straight into matmul2d's right-input cooperative tensor
(single-simdgroup scope, no threadgroup staging or barriers in the K loop); W4<F> 4-value decoders for Q2_0,
IQ3_XXS, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_S, IQ4_NL (quant.metal's deq arithmetic)
- qsa_prompt.metal: qsa_attn_mm (S = Q K^T and P V as single-simdgroup matmuls, int8 K/V decoded into the right
input; same partials as qsa_attn, so qsa_merge is unchanged), qsa_q_f16
- tests: qsa_prompt_parity, coop_layout (the cooperative tensors' element layout), moe_mm_parity --v3;
prompt_chain PC_GU / PC_DN / PC_QSA switches (defaults v3 / v3 / mm)
#1240 stopped creating the drafter's branch stream and its two events on HIP: nothing uses them there, and an R9700
lost prompt speed with them. On gfx103x the stream's existence alone makes prompts read faster. RX 6800, Windows,
STRATA_HIP_PROMPT_F16=1, IQ3_XXS, two runs each: a 7.9K-token prompt at 798-808 tok/s with the stream and 584-596
without, a 30.4K-token one at 855-863 against 553-600; all of it is in the prompt path's "wait copy" phase. The two
events change nothing, so they stay out.
gfx103x creates the stream; other HIP cards do not, as before. STRATA_MTP_SIDE_STREAM=1 / =0 forces either. CUDA is
unchanged. Measured on one card under Windows only.
- token_chain: the HC read defaults to the Q8_0 copies through the 3-dispatch read (q8_3); STRATA_HC_Q8=0 selects
the exact BF16 path (--hc still overrides)
- prompt.metal: route_topk, gr_epi_down/up, swiglu_h, moe_out, gr_norm_h (fp16 xn), gemm_hc_up (the HC up GEMM with
the mixing epilogue; w_up rows interleaved by dq_f16's new perm field)
- gdn_prompt.metal: gdn_pre (parallel conv/norms/gates) -> gdn_scan_r (state scan, one barrier a token) -> gdn_post
- dense_mm.metal: DqArgs gains `perm` (4th field)
- moe_mm.metal: mm_scan zero-fills the unused simdgroup slots; indirect grids for gather/gate-up/down in meta;
MM_PROBE bench constant (function constant 5: pipelines of mm_gu2 need it set)
- tests: prompt_chain (whole prompt in chunks, class breakdown), prompt_parity, gdn_prompt_parity; mm_bench tiles
- window.cpp: a window of 2..8 tokens runs the GDN steps without committing; the projections it feeds them are
kept per layer, and commit(n) folds the kept prefix into the state (gdn_step over n tokens with commit, the same
kernel: bitwise the window's trajectory) and turns the conv-history ping-pong. K/V and indexer cells past the
prefix are rewritten when those positions run again, before anything reads them; the PLE history keeps the n
rows. Checked: greedy output with suffix drafts (windows up to 4, 50 of 59 drafts kept) is the same token for
token as one-token windows.
- generate.cpp (Metal only): --mmap-experts by default (the GGUF shards in place; the window reads them itself):
the 31.6 GB expert arena is not loaded any more (RSS 74 -> 40 GB, the rest is the mapped shard)
- stubs.cpp regenerated (256 entry points left as stubs)
IQ3_XXS vs the CUDA engine (P1004, f-mac, same flags, --kv int8): 200 of 200 tokens equal on the capital
prompt, 124 on a story; following CUDA's tokens 3 of 200 picks differ, at margins 0.07-0.23 (near ties).
The Metal build of `strata` is the shared host code (src/, unchanged except preprocessor-only Metal/arm64 branches)
over two Metal-only parts in metal/src/engine:
- window.cpp: the decode window, the one backend hook (Verifier::init/run/commit/copy_logits under
STRATA_GPU_METAL). It is mac-kern's chain on the model GGUF itself (shard 1 mmapped and wrapped no-copy, every
weight read in place: no upload, no expert cache): HC reads, GDN/QSA mixers with their own state (recurrence,
conv history ping-pong, int8 K/V, indexer), router + plan, routed and shared experts, combine, the final mixer,
the LM head and the greedy pick, encoded as one dependency graph. The embedding is dequantized on the CPU into
the shared buffer; the PLE rows come from the shared table reader, their projections run on the GPU and the
block's norms / gate / conv on the CPU between two command buffers. One-token windows (--mtp-max-t 1) for now.
- host helpers copied from the CUDA kernel files the host code reads while it starts (format tables, sizes, path
switches, the RoPE table/config); every other kernel entry point is a generated stub that stops with its name
(stubs.cpp, tools/gen_stubs.py, from clang's AST of our headers).
Shared host code, all inside #if STRATA_GPU_METAL / __APPLE__ / __aarch64__ (the PC objects do not change):
- verify.hpp/.cpp: the hook (an opaque window pointer; init/run/commit/wait_commit/copy_logits hand over)
- generate.cpp: no x86 ISA refusal; a native pack without --prefill reads its prompt through one-token windows;
the model shard handed to the window
- device.cu: the poison fill through the allocation's CPU view; pool_affinity_linux.hpp: a macOS branch (no
pinning); expert_layout.cpp: no xgetbv asm on arm64
metal/include/metal_compat: cpuid.h (all zero on arm64), linux/mman.h (no hugetlb/THP: the fallback taken).
M5 Max, Q2_0, greedy, "What is the capital of France?": "The capital of France is Paris." - the same 36 first
tokens as the HIP engine (R9700, same flags) on that prompt and 39 on a story prompt, then near-tie flips
(margins 0.08-0.41 when following the HIP tokens); decode 74-75 tok/s.
- kernels/mm_bench.metal + tests/mm_bench: M5 Max MetalPerformancePrimitives matmul2d fp16/bf16 62-65 TFLOP/s
(64x64 tiles), fp32 15, simdgroup_matrix half 14.
- kernels/moe_mm.metal: deterministic GPU plan (block histograms, scan, stable placement; padded per-expert tiles),
activation gather to fp16, mm_gu2 (gate+up+SwiGLU) / mm_down2 with weights dequantized (ref_quant arithmetic)
into threadgroup memory and matmul2d in multiply_accumulate mode, mm_combine. tests/moe_mm_parity: Q2_0 and the
IQ3_XXS-pack formats vs a reference with the same fp16 operands (1e-6..4e-5; vs exact float 2e-4..5e-4).
T=2048, 512 experts, per layer: Q2_0 8.0 ms = 25 TFLOP/s = 3.9 us a token (~5,300 tok/s experts only);
IQ3_XXS gate/up + IQ4_NL down 14.5 ms. The tile idea (dequantized weights from threadgroup memory under matmul2d)
follows yibie/strata-mlx's tile_matmul (MIT); the code is ours.
- kernels/dense_mm.metal: dq_f16 (any dense format to fp16 once a chunk), act_f16, gemm_f16 (MPP): 64-65 TFLOP/s
on the GDN/QSA shapes for 2048 tokens, dequantization ~4% of it; tests/dense_mm_parity (12 formats).
metal/CMakeLists.txt holds both: mac-kern's runtime face, kernels and parity programs as they are, and the device
runtime + the engine build (taken from ../src through include/metal_compat) when METAL_CPP_DIR is given.
runtime_hello is kept out of the kernel tree's test glob.
- gr.metal: gr_down_f<SG, KS, F> / gr_up_f<F> for BF16 (the old names) and Q8_0 (gr_down_q8_*, gr_up_q8);
tests/gr_parity GR_Q8=1: vs double on the dequantized Q8_0 weights, 1/3/8 tokens, M5 Max and M4 pass.
- token_chain --hc q8_3. M5 Max, --fuse --graph, interleaved, 2 rounds: Q2_0 1 token 11.62 (BF16) / 10.59 (Q8_0,
2 dispatches) / 10.38 ms (Q8_0, 3); 3 tokens 16.78 / 17.21 / 16.24-16.31. IQ3_XXS 1 token 12.85 / 11.87 / 11.68;
3 tokens 17.88 / 18.59 / 17.47-17.51. The 2-dispatch read's gain was all Q8_0; its kernels are slower than the
3-dispatch ones at windows. M4 HC read: Q8_0 85 us vs BF16 141-144 us.
- gemv.metal: gemv_pair<F1, F2> for the packs' gate/up format pairs (tests/pair_parity: 8 pairs x 1/3/8 tokens vs
double, 1.4e-7..2.6e-7) and GV_SET(T_Q2_0R).
- token_chain CHAIN_PAIR=1: interleaved A/B, 3 pairs, Q8_0 HC --fuse --graph: 10.60-10.61 ms without vs 10.71-10.72
with (3 tokens 17.23-17.25 vs 17.31-17.33): two concurrent GEMVs and the SwiGLU win; opt-in only.
kernels/gridbar.metal + tests/gridbar_bench: phases of a streaming read separated by an atomic grid barrier vs by
dispatch boundaries (M5 Max). Without atomic_thread_fence(device, seq_cst) the phases read other threadgroups'
stale writes; with it 0 stale reads. Speed: 4.5 MiB phases 10.3 us (10 threadgroups) vs 9.9 us a dispatch, 30-34 us
with 20-40 threadgroups; 0.5 MiB phases 1.5 vs 2.2 us. Not pursued.
All under __APPLE__ / STRATA_GPU_METAL, so the Linux and Windows code is unchanged:
- direct_file.cpp: no O_DIRECT on macOS; F_NOCACHE after open keeps the reads out of the buffer cache
- memory.cpp: F_RDADVISE is the read-ahead hint where posix_fadvise(WILLNEED) is used
- expert_source.cpp: available memory from host_statistics64 (free + inactive + purgeable); the posix_fadvise of a
released expert layer is skipped (the madvise before it already released the mapping)
- generate.cpp: <sys/sysmacros.h> is not on macOS (major/minor come from <sys/types.h>); the CUDA toolkit mismatch
check is CUDA's
- vmm.cpp: the cuMem* virtual memory stays CUDA's (Metal takes the HIP branch: elastic K/V off)
- prefill.cpp: std::max<size_t> where size_t and uint64_t meet (the same type on Linux, distinct on macOS)
- gdn.metal: gdn_step_t<IB>; gdn_step8 (1024 threads, 16 state rows a thread) passes and is not faster than
gdn_step (16.3 vs 16.4 us commit, n=1): the state write after the compute is what a commit adds.
- token_chain: CHAIN_GU / CHAIN_DN / CHAIN_DN4 select the routed-expert tiles; measured in the chain (Q8_0 HC,
--fuse, --graph): gate/up 32,1,4 stays; the Q2_0 down 8,1,4 over 8,2,4 (10.63 vs 10.67 ms, 3 tokens 17.23 vs
17.50 ms), now the default.
- kernels/qsa2.metal: qsa_topk2 - the same selection (ids exact at contexts 700..33000): the radix digit found by a
parallel suffix scan of the 256 bins, cells emitted from contiguous block ranges after one scan. 23 -> 8.2 us at
8K, 54 -> 14.5 us at 32K; token_chain (Q8_0 HC, --fuse, --graph, interleaved): 8K 11.40 -> 11.00 ms,
32K 11.90 -> 11.27 ms.
- qsa_attn2 (chunks of 32, head groups) and qsa_attn3 (lane per cell for q.k): both pass, neither beats qsa_attn at
the full budget (21-24 us, nearly independent of the cell count: a latency chain); QSA_ATTN selects, default 1.
- Read yibie/strata-mlx's OPTIMIZATION_RECORD/ROADMAP (MIT) before this round: its finding that merging kernels cut
the kernel count 43% for 15% of the time matches ours.
kernels/mma.metal: 8-row tiles dequantized with ref_quant's arithmetic into threadgroup memory, x loaded transposed
from device memory, simdgroup_multiply_accumulate; tests/mma_parity: 12 formats pass vs double, every token of a
window bitwise a one-token call. Speed (M5 Max): 60-190 GB/s for 1, 3 and 8 tokens alike, against the per-lane
GEMV's 400-595 GB/s at 1 token and 70-230 at 8: the dequantization through threadgroup memory (~8 B a weight) is
the limit. Only Q2_0 at 8 tokens is faster (115 vs 126 us). Not used by token_chain.
- gdn.metal: GdnArgs.plane: the step writes y as the raw plane plus the other planes and the per-16 sums, so the
out projection needs no prep dispatch.
- gr2.metal: gr_down3 can form bo as the MoE combine itself (GrArgs3.comb).
- token_chain --fuse: the GDN prep in the step, no QSA score/top-k dispatch while the context fits the 2051 budget.
Q8_0 HC, --graph: 10.71 -> 10.49 ms (1 token), 17.13 -> 16.86 ms (3); BF16 2-dispatch 11.91 -> 11.65 ms.
--fold-comb (the combine in the next HC read) is slower: 10.49 -> 11.22 ms; off.
- kernels/gr2.metal: gr_down3 (grid row blocks x streams: each threadgroup forms its stream's R', norm and xn in
threadgroup memory, writes per-stream partial dots; R' ping-pong, never in place) and gr_up3 (lo from the
partials, xn on the fly); BF16 or Q8_0 weights. tests/gr2_parity: both formats vs double on their own weights,
prepared planes bitwise = xprep; Q8_0 moves the read's output by 1.2e-3 relative.
- ref_quant: quantize_q8_0 (ggml's reference rounding), used for the load-time Q8_0 copies (STRATA_HC_Q8's form).
- token_chain --hc 3|2|q8. M5 Max, Q2_0, --graph: 1 token 12.00 / 11.91 / 10.71 ms; 3 tokens 17.21 / 18.02 /
17.13 ms.
MAC-PLAN Phase 1, compat design (the alternative to renaming the host calls, f-mac-design S1-S4).
metal/include/metal_compat goes first on the Metal build's include path, as include/strata/hip_compat does on HIP:
its cuda_runtime.h gives the ~70 CUDA runtime names of the Phase 1 inventory (types, constants, calls) on Metal,
each forwarding to metal/src/gpu/runtime.cpp (metal-cpp: gpuAddress device pointers in one residency set, streams =
queue + command buffer + shared-event timeline, graphs = recorded dispatch lists), and immintrin.h gives _mm_pause /
_mm_sfence on arm64. The CUDA / HIP / SYCL builds never see metal/.
Shared files: four preprocessor conditions also exclude STRATA_GPU_METAL from CUDA-only blocks (the cuMem*
segmented expert cache, the STRATA_VERIFY_NODES graph listing, the MTP stream priority, cudaInitDevice of the remote
experts); nothing else in src/ changes, so the PC objects are byte-identical.
On the M5 Max: runtime_hello passes with the unchanged src/core/graph.cpp, and graph, expert_cache, weights,
native_head, native_dense, layer, verify, mtp and session compile for Metal as they are (-fsyntax-only); prefill,
expert_source and generate stop only at macOS platform items (std::max of size_t vs uint64_t, posix_fadvise /
_SC_AVPHYS_PAGES, sys/sysmacros.h).
- gdn.metal: a q/k conv channel is read at the start of three threadgroups (value heads h, h + 16, h + 32) while
its owner wrote it back in place at its end: on the 10-core M4 the threadgroups run in waves and the later ones
read the new history (gdn_parity failed there, passed on the M5 Max where all 48 start together). The committed
history now goes to hist_out (the caller swaps); gdn_parity checks determinism too. Passes on both Macs.
- gemv_parity --gguf: one real tensor per format of a GGUF (experts: expert 0) vs double; Q2_0 file: 11 formats
pass, rel L2 8e-8..2.7e-7.
- src/gguf_lite: a minimal GGUF v3 reader; the file is mmapped and wrapped no-copy as one MTLBuffer.
- moe.metal addresses experts by gate/up/down offsets and strides (a pack blob or the GGUF's three tensors).
- kernels/glue.metal: the shared expert's SwiGLU (prepared activations) and the combine with its sigmoid gate.
- tests/token_chain: every layer of the model with its own formats, all kernels in one command buffer.
M5 Max, Q2_0, context 1000: 12.83 ms a token (serial encoder), 12.01 ms with --graph (a concurrent encoder,
barriers only at true dependencies); a 3-token window 18.42 / 17.23 ms. Bound: 7.27 ms.
- mtl_rt: run_graph (concurrent encoder, Dispatch::barrier).
- kernels/qsa.metal: q/k RMSNorm + NEOX RoPE (host table), int8 KV append (fp16 scale per 64), the indexer's raw
tail / pooled block keys / dead key, ReLU-sum block scores with the +1e9 tail, a cell-level radix top-2051
(ties to the lowest cell, ascending), split-K GQA attention over the selection, log-sum-exp merge * sigmoid(gate)
written as prepared activations; tests/qsa_parity stage by stage (ids exact; contexts 700..9001).
- kernels/sample.metal: two-stage argmax, ties to the lowest id; tests/sample_parity.
kernels/gdn.metal: one threadgroup per value head, the head's state in registers across the window's tokens;
commit writes state + conv history back, verify does not; tests/gdn_parity vs double (1.7e-7), verify leaves the
state untouched, the first m outputs of a longer window are bitwise a window of m (the fold).
- kernels/gr.metal: gr_norm (folds the previous write, per-stream RMSNorm), gr_down (down + inject rows, silu/hc
epilogue, K split), gr_up (gated mean over the streams, writes the prepared activations); tests/gr_parity vs double.
- moe.metal: moe_route = native_router.cu's softmax/top-10/renormalisation and the plan in one threadgroup
(tests/route_parity: ids exact incl. ties, weights 2e-7); moe_gu/moe_down switch once per threadgroup into a body
specialised for the group's entry count (the predicated 8-way loop cost 2x); Q2_0 down reads raw activations
(T_Q2_0R) so no prep dispatch is needed.
- mtl_rt: warm() before timings (the GPU starts at idle clocks).
- kernels/units.metal: per-format unit decoders (Q2_0, Q3_K..Q6_K with the mask trick, IQ4 via a byte->half2
codebook table, BF16, generic for the rest) and one combine for all; xprep writes the pre-scaled activations
and per-16 sums once per input vector.
- kernels/gemv.metal: y = W x for 1..8 tokens in one pass over the weights, NTOK a function constant, optional
K split across simdgroups; tests/gemv_parity: parity vs double for 12 formats x 20 tile shapes, every token of
a window bitwise equal to a one-token call, and the bench on the packs' projection shapes.
- moe.metal on the same decoders; mtl_rt compiles with safe math by default (no reassociation; measured free).
- tests/alu_bench: instruction rates (fp32/fp16 FMA, int, convert, threadgroup/device/constant gathers).
The dpct migration had dropped the cudaMemcpyAsync D2H and the stream sync, so the resident RAM mode with adaptive swaps committed
whatever the exchange buffer held as the evicted expert (lost expert, wrong weights on a later CPU read). STRATA_CHECK_SWAPS=1 is a
test aid: it compares each staged buffer with the model file and audits the RAM copy at the end of a run.
metal/ is its own CMake project (macOS only), nothing in src/ or include/ changes. MSL is compiled at run time
(the Command Line Tools have no offline metal compiler) with ggml-common.h prepended for the i-quant grids.
- src/mtl_rt.*: a C++ face over Metal (buffers, pipelines with function constants, serial/concurrent encoders,
indirect dispatch, indirect command buffers, GPU timing).
- src/ref_quant.*: CPU reference dequantizers for Q2_0, Q4_0, Q5_0, Q8_0, Q3_K..Q6_K, IQ2_XXS/XS/S, IQ3_XXS/S,
IQ4_NL/XS, BF16, F16; checked bitwise against gguf-py (tests/ref_dump + tools/check_ref.py: 17/17 equal).
- kernels/quant.metal: the formats as per-lane units (decode once, dot with every token of the window).
- kernels/moe.metal: moe_plan (top-k -> groups by expert, deterministic, writes the indirect grids), moe_gu
(gate+up+SwiGLU, grouped), moe_down, moe_combine; tests/expert_parity: parity vs double and the bench.
- tests/rt_bench: M5 Max streaming read ~585 GB/s, ~0.85 us per dependent dispatch, ~170 us commit+wait.
Linux CUDA had it on by default since 0.1.43, measured on an RTX A4000 (+5.5..8.3% decode) and a Tesla P100 (+4.4%). An RTX 5090 Laptop with a native IQ3_XXS pack and --expert-cache-per-layer decodes 21% slower with it, and the overlap is the default only there. The switch is now =1 turns it on everywhere, and the startup log names the state (with the reason when it cannot run), so a decode-rate drop is not the only way to see it.
Unchanged: the copy path, the gate predicates and the tokens (same tokens with or without it).
Fixes#2010
On 2x/4x R9700 the drafter's chain and the last stage's window shared a card and slowed each other (window 13 -> 27 ms) in most engine starts; the draft layer now loads on the first card (STRATA_MTP_STAGE0=0 keeps the old place). Exact flags, tokens identical: 2 cards code +21%, 3 cards +19%, 4 cards +22% over the serial loop. A split reads prompts in 8192 chunks (STRATA_SPLIT_CHUNK16K=1 keeps 16384): 16K prompt +36..53%, 32K +18..29%, same tokens. STRATA_PCIE_AUTO_SPLIT=1 now also runs in the pipelined loop (opt-in).
gfx1103 was missing from the gfx11 matrix-core lists, so the 780M ran the portable verify-window and prompt kernels. Added to the
compile-time lists and gfx_arch_is_gfx11_wmma(). Radeon 780M, Q2_0, 5 interleaved pairs: decode (code) 11.96 -> 18.25 tok/s,
same token ids; prompts unchanged. hip_prefill_wmma_gemm_parity passes on gfx1103 (3520 cases). Identity gate (defaults off,
mg_norepeat 10/10 against the portable build, exact_pp 4K/20K residual dumps identical).
The serve path counts every routed expert on the device when a save path and a mirror exist; the saved ranking mixes the counts with the start ranking (prior worth 800k routed entries, STRATA_PROFILE_PRIOR_ENTRIES). Exact: placement is fixed under NO_HOST, the file applies at the next start. B70, IQ3_S: code +13..19% vs the xe default profile, prose -18%.
0.1.43 moved the QSA step records and the batched token ids to a pinned buffer (#1712). A 4-hour multi-slot load on 2x RX 7900 XTX (parallel 4, vision plus long
prompts, layer split) still faulted after 31 minutes, with the picture rows, the draft tokens and bounds and the non-mapped routing tables still going up from
caller memory. PinnedRing (include/strata/core/pinned_ring.hpp) is a ring of four pinned slots with one event each: the bytes are copied into the next slot after
the copy that last read it is done, the DMA runs from the slot, and the caller may change its source on return. No pinned RAM (or STRATA_TEST_PAGEABLE=1): the
copy is made from the caller's memory and the stream is waited for. A failed upload is returned and reported.
Moved onto it: the picture's embedding rows (gathered per run into one slot), the draft K/V step's tokens and bounds, the PLE rows' pageable fallback, the
non-mapped branch's routed-id download and slot/src/bounds uploads, the peer device's src and bounds tables, and the drafter's prompt records (mtp.cpp).
Copies from pinned arenas and cudaHostAlloc buffers are unchanged. Same bytes reach the GPU, so answers are identical.
The analysis of which copies are still pageable, and the bounce-slot-per-event idea, are mantovaniluca91's (patch C on #1712). This is our own form of it.
collect_machine listed every nvidia-smi card. It now takes the bench's --gpu, else the config's gpu (or --gpu in its args); with none given it names the one card the server reports instead of all of them.
ensure_loaded logged 'loading anyway' for a before_load command that failed and co-loaded two models onto one GPU. A non-zero exit or a timeout now leaves the model unloaded and answers 503 with Retry-After (before_load_ignore_failure restores the old behaviour); before_load_timeout_s sets the 120 s limit; POST /unload?wait=1 (or wait=<s>) waits for the running turn instead of answering busy.
The engine answered 'ERR BGEN: no such free slot' when its view of a slot differed from the server's; the server turned the ERR into an HTTP 400 and the conversation, drawn to the same slot by its held prefix, got it again on every turn. Now the refused slot is parked (busy, no owner) for SLOT_PARK_S, the request waits for / takes another slot, and after BGEN_REFUSALS_MAX refusals it ends with a retryable 503.
The host mirror and the unmirrored count cover the first stage only; a later stage whose cache lacks experts has no mirror entry and, under NO_HOST, no host answers for them (device lost at the first verify window on 2x Arc Pro B60). The start now refuses with the existing wording, or logs the count when NO_HOST is off.
The mapped gather_batch took 16 serial demand faults per token from a cold table with no progress mark; the first chunk of a prompt only has layer 0 to hide it behind, so the stall watchdog stopped the engine. Rows are now hinted 4096 at a time (PrefetchVirtualMemory / MADV_WILLNEED). Answers are unchanged. The 0.1.42 auto-switch to mmap on Windows is opt-in until it is measured there.
The per-layer host function made the overlap depend on the driver's callback thread being scheduled: with the pool's
workers spinning on every other CPU it still cost 7 ms per window on a 20-core desktop even with the thread on all CPUs
(RTX 3060, Core Ultra 7 265: 46.5 tok/s with the warmed callbacks against 56.5 with the overlap off). The copy stream now
ends each layer's copies with a 4-byte device-to-host copy of the layer number into a pinned int the gate polls; same stream
order, no host thread. STRATA_ADAPT_SIGNAL=host brings the host functions back. Swaps, residency and answers are unchanged.
On Linux the CUDA (and HIP) runtime creates its host-callback thread lazily, at the first cudaLaunchHostFunc, and a new
thread inherits the creator's CPU mask. The decoding host pins itself to one CPU (SessionLoopScratch::init), so a thread
created after that - which the adaptive overlap's per-layer callbacks did - ran on the host's CPU and fought the host's
spin-wait: ~1000 us instead of ~4 us per callback, decode 2-4x slower with STRATA_ADAPT_OVERLAP on (#1964, #1975).
oal's warm-up (one no-op callback + sync before ExpertPool) is now unconditional on Linux; the answers, swaps and
residency are untouched. Windows CPU Sets are per thread and not inherited, so Windows is unchanged.
Rarely a window's logits become NaN and the reply degenerates into token 0
('!') repeated (#606); the poison sits in the conversation's K/V and
recurrent state, so with the prompt cache every later turn stays poisoned
until it is read again from token 0. The q8_1 clamps (2b726ba, f6601bf,
7a0e302) remove the overflows found so far; this guard catches whatever
cause is left, and drops the poisoned state instead of serving it again.
On by default, as a fix: STRATA_NAN_GUARD=0 turns it off. It only reads
flags, so an answer without a NaN is the one the engine gives with the
guard off (compared byte for byte on a GPU, below).
Engine:
- every verify-window graph checks its head logits beside the greedy pick
(logits_nonfinite_rows: 16 blocks a row, one flag word each, into mapped
memory); Verifier::note_nonfinite reads them after the sync the window
already does (run, run_slot_rows, batch_poll, pl_finish). No extra sync.
With STRATA_NAN_GUARD=0 no kernel is recorded.
- --serve: a flagged window emits none of its tokens, one stderr line names
the position and row, and the request ends with `DONE ... nonfinite` (a
batch slot: `BDONE <slot> <n> nonfinite`; an admission: BADM <slot> 0).
The next request first drops the live session, every checkpoint, every
parked conversation (ConversationCache::clear) and every idle slot, and
reads its prompt from token 0; a slot still running then finishes its
reply and is not kept as a cache after it. A request that ends on it does
not keep its conversation live (#1620's cancelled-read path is skipped
for it).
Covered: the serial decode loop (one GPU or a layer split), the
--pipeline-windows decode loop of a layer split, the prompt's short tail
(read_windows, and read_windows_pl with --pipeline-windows), and both
batch-slot loops (run_slot_rows, and the --batch-groups pipeline's
batch_poll). Not covered: the one-shot command line (no --serve) and the
SYCL port in sycl/, which has its own verify.cpp and generate.cpp.
- STRATA_NAN_INJECT=<n>[,<n>...] (debug): the n-th window the guard checks
reports row 0 as non-finite.
Server: EngineNonFinite on that finish. A request that has sent nothing yet
is sent again once (its usage is the retry's: the failed pass's DONE is
dropped); a reply that had started ends as "length"; a second fire on the
retry ends with an error (503 before an answer, an error event in a
stream). With replicas the router forgets what that replica held.
repeat_stop_tokens stays as the backstop.
Checks: serve tests (python -m unittest discover -s serve -t . -p
"test_*.py": 697 tests, OK, 11 skipped; NonFiniteLogits,
test_nonfinite_in_batch_mode and test_nonfinite_forgets_the_replica are
new). ctest with CUDA_VISIBLE_DEVICES=-1: 24 pass (conversation_cache_test
with clear() among them), 3 skip for want of a GPU, and the 43 GPU parity
tests were not run on a GPU. Built for CUDA 13.0 on Linux (GCC, sm_75 +
sm_86); generate.cpp, verify.cpp and verify_kernels.cu compile with
MSVC 2022 + CUDA 13.0, and conversation_cache_test and conv_cache_test
pass there; built for HIP (ROCm 7.0, gfx1100, compile only) and SYCL
(oneAPI 2026.1.1; it takes only the new declaration in verify_kernels.hpp
and the new inline ConversationCache::clear(), and calls neither).
On a GPU (RTX 3080, RVN IQ3_S, strata.service's one-GPU config):
- Answers, guard on vs STRATA_NAN_GUARD=0, at fixed expert sets
(--pcie-frac 0 --adapt-every 0 --adapt-swaps 0 --mtp-max-t 1
--suffix-draft 0, STRATA_IQ_MT_MIN=1 STRATA_PREFILL_CPU_SHARE=0), a chat
replay of 2 x 4 turns on a 16K-token system prompt: byte-identical in 3
pairs (one with STRATA_ROUTE_TAIL_SKIP=0, one with STRATA_SH_STREAM=0),
and two guard-on runs 8 of 8. One more guard-off run differed from every
other run from its first answer, and a second guard-off run matched the
guard-on runs: a difference between runs of this config, not the guard.
- Cost, Nsight Systems: logits_nonfinite_kernel 424 launches, 4.77 us on
average (7.78 us at most), 2.0 ms against 21.6 s of decode: 0.009%.
- STRATA_NAN_INJECT=1,60: the first window of a new chat flagged, the
server sent the request again from token 0 and the client saw one normal
answer; the 60th, mid-reply in the next turn, ended that reply as
"length" after 58 tokens; the turn after it reused 0 cached tokens and
answered normally. The engine logged exactly the two injected windows.
route_resident_k is one warp per token and the form it replaced walked all 512 experts on a single
thread. Nothing in the tree checked that the two land on the same expert and leave the same ids,
weights and counters, and the option's whole justification is that it moves misses across PCIe and not
the model.
src/kernels/route_resident_parity.cu: 240 random cases over 1 to 8 tokens, a 40-84% resident table,
random rank windows, margins from -0.5 to 1.5, and half the cases rounding the logits onto a coarse
grid so the tie rule (the smallest expert index wins) is exercised. Half the cases get the router's
own top-10 and half get ids with repeats, which is what the already-picked test is for. The reference
is the spec written out plainly - a serial walk with the same comparisons - and not a copy of the
kernel this branch replaced; it runs on the device so that expf, and so the softmax bits, are the same
function on both sides.
ids and the four counters have to match bit for bit. The weights are a softmax ratio and this
reference is a second compilation of it, so they are compared within 8 ulp and the worst seen is
reported; the kernel itself is bit-identical to the serial form it replaced, which was measured with
both kernels side by side in one translation unit. A case that swaps nothing still has to leave the
weights untouched, which the sentinel they start from checks, and the test fails if no case swapped at
all.
route_resident_parity: 240 cases, 205 with swaps (2004 swaps), 0 mismatches, weights within 6 ulp
Registered in ctest next to the other router parity tests, TIMEOUT 120.
The swap kernel is the only work STRATA_ROUTE_RESIDENT adds to a window, and it was written as one
thread per token: with a verify window of 1 to 8 rows that is up to 8 threads of a block, each doing,
per rank in [lo, hi], a 512-expert scan with a ten-way "already picked" test inside it. On this
machine that is about 4 x 512 x 12 scalar operations running on six threads, on the critical stream,
once per layer, and it shows up as its own cost in the window.
STRATA_VERIFY_PROFILE, three legs in one session, same prompts, same expert cache (8083 slots, one
learned profile), medians of the per-window stage table:
configuration hc-read1+router waitB ms/window pcie_e
option off 2.25 9.30 21.85 1.87
option on, one thread per token 5.00 5.01 21.97 1.00
option on, this commit 2.58 4.87 19.75 0.95
The option redirects about a miss per layer-window away from PCIe and saves 4.3 ms of waiting for it;
the serial swap spends 2.75 ms back, which is why the option measured as a wash. In the warp form the
swap costs 0.33 ms, so the waiting it saved stays saved.
End to end, twelve matched pairs of engine restarts, nine decode samples a leg, three greedy 150-token
prompts, the learned expert profile copied back before every leg so both arms start from the same
8083-slot cache. The first six pairs ran the unpatched build first, the last six the patched build
first, so a within-pair order effect would show up as the two halves disagreeing:
pair order decode unpatched patched delta ms/window unpatched patched delta
1 A->B 104.1 115.7 +11.1% 18.64 17.01 -8.7%
2 A->B 103.5 107.4 +3.8% 18.47 18.12 -1.9%
3 A->B 111.4 107.7 -3.3% 17.71 18.26 +3.1%
4 A->B 101.4 113.0 +11.4% 19.11 17.03 -10.9%
5 A->B 105.8 107.6 +1.7% 19.05 18.05 -5.2%
6 A->B 104.8 105.6 +0.8% 18.60 17.83 -4.1%
7 B->A 106.6 113.4 +6.4% 18.70 17.16 -8.2%
8 B->A 103.1 110.2 +6.9% 18.14 17.29 -4.7%
9 B->A 106.9 111.5 +4.3% 17.91 17.26 -3.6%
10 B->A 104.3 114.7 +10.0% 17.81 17.11 -3.9%
11 B->A 108.3 115.9 +7.0% 18.47 16.61 -10.1%
12 B->A 104.5 113.6 +8.7% 18.43 17.76 -3.6%
twelve legs each: unpatched decode median 104.7 (101.4-111.4) ms/window median 18.47 (17.71-19.11)
this commit decode median 112.2 (105.6-115.9) ms/window median 17.27 (16.61-18.26)
decode median +6.6% mean +5.7% faster in 11 of 12 pairs (sign test p = 0.006)
ms/window median -4.4% mean -5.2% shorter in 11 of 12 pairs (sign test p = 0.006)
first six pairs: +2.7% decode, -4.7% window last six: +7.0% decode, -4.3% window
pcie_e, the engine's own count of experts per layer-window that cross PCIe, is the same in both arms
(leg medians 0.63 against 0.64), so the same experts are routed and the difference is the kernel.
The swap's cost, and so its removal, scales with the tail experts the option actually redirects. Four
more pairs with a 4000-slot cache instead of the automatic 8083 (pcie_e 1.83 against 1.78, i.e. about
three times the traffic of the series above), two pairs each way:
pair order decode unpatched patched delta ms/window unpatched patched delta
1 X->Y 83.8 92.0 +9.8% 21.94 20.60 -6.1%
2 Y->X 84.6 92.5 +9.3% 23.97 20.53 -14.4%
3 X->Y 82.2 85.5 +4.0% 23.69 22.65 -4.4%
4 Y->X 81.0 87.7 +8.3% 24.54 22.21 -9.5%
decode median +8.8%, ms/window median -7.8%, 4 of 4 pairs, both orders.
What this changes is the shape, not the choice. One warp per token: lane j reads experts j, j+32, ...
(coalesced instead of a serial walk), the warp reduces to the best logit in a shuffle tree, and the
ranks are still visited in order, because a swap changes the set already picked. The choice is the
same bits: a strictly greater logit wins, the smallest expert index breaks ties, the already-picked
test runs against the ids as they are at that rank, a token with no swap keeps the router's weights
untouched, and the softmax is the same expression.
Parity. 400 random cases - half of them built with many equal logits, random lo/hi, margins from
-0.5 to 1.5, 1 to 8 tokens, about 60% of the experts resident, routed ids with repeats - compare the
two implementations bit for bit on the ids, on the weights (their raw bits) and on the four counters.
The harness carries both kernels side by side; 0 mismatches.
For context, the option's price is the margin's and not the kernel's, and this commit does not touch
it. Teacher-forced log-probabilities over 7677 positions of one 7.6k-token text (technical prose,
source code and Chinese), top-64, against the same text with the option off:
margin argmax same KL (top-64) teacher-forced loss perplexity
off - 0.000 3.2033 -
0.25 84.9% 0.235 3.2238 +0.6%
0.50 83.0% 0.293 3.4360 +7.3%
1.00 63.5% 1.001 4.7751 +49%
Two runs with the option off are bit-identical, so that spread is the option and not the machine.
0.25 keeps nearly all of the waitB saving for a fraction of the distortion; the default 1.0 is not
worth its price.
Tests: the parity harness is a standalone .cu (both kernels, the random cases, the bit comparison).
Happy to wire it into the test suite next to the other kernel tests if you want it in the tree.
The QSA step records of a chunk were uploaded with an unchecked cudaMemcpyAsync from a
std::vector. On gfx1100 the first text prompt after a picture one read garbage records
(5 of 5 faults on a two-request repro, 0 of 5 with a pinned, event-guarded source). The
token id upload has the same shape and now takes the same path; both check the result.
Reported and traced by mantovaniluca91, #1712.
IlQ3K runs q3_q8_dot_impl's arithmetic with the scales and signed values done once per chunk; bitwise native_mmvq's output
for T 1-4 and rows 1/2/4 (mmvq_il_parity). Q2_0 pack: the Q3_K multi-column kernels were 14% of GPU kernel time; the new kernel
takes 2560x10240 at 4 columns from 120 to 70 us on an RTX A4000. The shared table (sm_80+/sm_120) is unchanged.
Bitwise equal to the CUDA layout on all 48 layers at 1-4 tokens a group (native_expert_bench); a 28-group VRAM call 17.6 -> 15.5 ms over the layers at T=1, 24.1 -> 22.6 at T=2, even at T=3, +2.6% at T=4.
Bitwise equal to native_mmvq (mmvq_il_parity, T 2-4, rows 1/2/4); table read off mmvq_il_parity --bench --emit-table on a Tesla P100 (three identical runs).
The pipelined decode loop asks both stages' windows for service millions of times a second while they wait on their
doorbells, and every call switched the device (OnDevice: cudaGetDevice and two cudaSetDevice) before finding nothing
to do. On a 0.1.41 split (RTX 3060 + RTX 5070 Ti, Windows) the host stalled inside one of those cudaSetDevice calls
after 919 million loop passes, with both stages' windows waiting on it (stall dump: the serving thread in cudart from
Verifier::service, every other engine thread idle). Now a call with nothing rung, no WDDM flush due (2 ms) and no
timeout due returns 0 first - the same answer the loop below gives - so the driver is called only to serve a layer,
to flush, or to time out.
The pages had no <link rel="icon"> and serve/web had no icon file, so browsers asked for
/favicon.ico, got a 404 and showed the default blank icon in the tab.
serve/web/favicon.svg is the layer stack the app already uses (serve/web/sprite.svg, i-layers),
redrawn on a 32-unit viewBox so it stays legible at 16 px, in the existing accent token #10b981.
Checked: curl /web/favicon.svg -> 200 image/svg+xml, both pages come back with the icon link,
and the icon rendered at 16/32/48/64 px in Chrome.
_Nvml.read() reports pcie_gen, pcie_gen_max, pcie_width and the two
throughput counters; _Amd.read() (the amdgpu sysfs reader from #301)
reports none of them, so on Linux the Monitor's PCIe tile shows only
dashes while the NVIDIA tile fills in.
This adds the link the card actually gets: the narrowest and slowest hop
between the root port and the card, from each hop's max_link_speed and
max_link_width. A card that negotiates Gen4 on its own hop but sits
behind a Gen3 root port really runs at Gen3, and that is the number a
bandwidth budget needs. pcie_own_gen keeps the card's own hop aside so
the gap is visible, and pcie_path reports how many hops could be read --
the bottleneck is only claimed when all of them could, because a kernel
or a container that hides part of /sys/devices would otherwise report the
card's own hop as the whole path.
Measured on one MI50 32 GB in a Gen3 x16 slot (lspci: the root port at
00:02.0 runs at 8 GT/s x16, the bridge above the card is downgraded, and
the card's own hop negotiates 16 GT/s), /metrics reports pcie_gen 3,
pcie_gen_max 3, pcie_own_gen 4, pcie_width 16 and pcie_path 4/4.
Throughput is deliberately left out: amdgpu's pcie_bw is three counters
whose documented use is a total, not a direction, while _Nvml's
pcie_rx_mb is receive-only, so filling that name from pcie_bw would
label a combined figure as one direction. Happy to follow up separately
if you want the tile to show a rate.
Hardware the numbers come from: one Radeon Instinct MI50 32 GB (gfx906,
PCI 1002:66a1, VBIOS 113-D1631711-100) on a Xeon E5-2696 v4 (Broadwell,
no AVX-512), in a PCIe 3.0 x16 slot, 128 GB RAM, Ubuntu, kernel
7.0.0-34-generic, ROCm 7.14 in a container.
Prefill::arm_cpu_share() asks "is this AMD?" with
#if !defined(STRATA_USE_HIP), but the gfx906 compat build (PR #638)
compiles as HIP *without* STRATA_USE_HIP -- see src/core/expert_cache.cpp
:5-6, which guards the same question with
#if defined(STRATA_USE_HIP) || defined(STRATA_HIP_GFX906).
On gfx906 the CUDA branch is therefore live: the CPU share is armed by
default for chunks below 1,024 tokens and prints its banner, against this
file's own note at :161 ("the layer split, batch slots and HIP stay off").
Seven other sites already carry the || defined(STRATA_HIP_GFX906) form.
Measured on one MI50 32 GB (gfx906), CLI generate, a 4-token prompt, one
binary with one set of weights and one config, only the environment
differing, power saving off:
default experts streamed 309 (295 by DMA, host 174.0 ms), resident 607,
on the CPU 197 (share 0.90) prefill 928.3 ms
=0 experts streamed 506 (492 by DMA, host 177.3 ms), resident 606,
on the CPU 0 (share 0.00) prefill 929.5 ms
Output token ids are identical (271 248068 198 760) and the 0.13%
difference is inside this machine's run-to-run spread (1.2-1.7%), so on
this card the share engages and buys nothing. #1595 and #1729 report the
default costing on CUDA; this is the mechanism that turns it on for AMD.
CUDA and SYCL do not define STRATA_HIP_GFX906, so the added check is a
no-op there and nothing on those backends changes.
Hardware the numbers come from: Xeon E5-2696 v4 (Broadwell, 22C/44T, no
AVX-512), one Radeon Instinct MI50 32 GB (gfx906, PCI 1002:66a1, VBIOS
113-D1631711-100, 34,342,961,152 B of VRAM), PCIe 3.0 x16 effective (the
root port runs 8 GT/s x16 and the bridge above the card is downgraded;
the card's own hop negotiates 16 GT/s, about 15.75 GB/s end to end),
128 GB RAM, Ubuntu, kernel 7.0.0-34-generic, ROCm 7.14 in a container,
built with -DSTRATA_HIP_GFX906=ON and -DCMAKE_HIP_ARCHITECTURES=gfx906.
spark(id, values, max) sets the axis top to max(|max|, the series' own
maximum, 1e-9). gpu_util, temperature, cpu, vram and power pass an
explicit maximum, but the pcie, tok/s, prefill and disk cards do not, so
those axes zoom onto whatever the series happens to hold.
An idle amdgpu pcie counter reads 0.01-0.02 MB/s (50 packets x 256 B per
second), and that becomes a full-amplitude square wave in the Monitor
while the tile's own text says 0.0 MB/s. Measured in the live DOM: the
pcie line's y values are only {4.0, 30.0}, i.e. the baseline and the top.
A floor of 1 only ever lowers the axis - the series maximum still wins -
so a real burst keeps its scale, and the cards that pass a maximum are
untouched.
IQ4XS.test_amd_is_not_asked mocks get_prebuilt_hip, the engine setup.main takes on Windows. On Linux setup.main
compiles the HIP engine instead (build_engine_hip), which the test did not mock. Without a usable system ROCm 7, as
on the tested machine, rocm_root then fetched AMD's ROCm index
(https://rocm.nightlies.amd.com/v2/gfx120X-all/rocm/), wrote Path(sys.prefix)/.strata-rocm.json and tried to run
<python dir>/rocm-sdk. Where sys.prefix is not writable (the system Python) that is a PermissionError. Where it is
(a venv), the test passed while setup.main stopped at the ROCm step with exit code 1, and a stamp with the same
version and index can make a later setup in that venv skip the ROCm install.
The test now mocks build_engine_hip the same way, so on Linux setup.main runs to the end (exit code 0) with no
network request, no subprocess and no write outside the test's temp folder. setup.py is unchanged; the Windows path
never reaches build_engine_hip, so it runs as before.
Tests: python3 -m unittest tools.test_setup_unsloth (WSL2): with the system Python 3.12.3 FAILED (errors=1) before,
38 tests OK after; in venvs on 3.10.22, 3.12.3, 3.13.7 and 3.14.8 OK before and after, and no stamp after. With the
system Python the other 27 tools/test_setup_*.py files give the same results as before.
The `[16.0,18.0)` range is CUDA 13's requirement, but it was applied to every caller: with the
only Visual Studio installed being 2026 (version 18), vswhere finds nothing and find_vcvars()
returns None. That makes the CPU image encoder unbuildable on such a machine - build_vision_cpu()
cannot run - so --vision cpu stays unreachable there even with the gate in hip_vision() open.
#985 relaxed the range for CUDA 13.3+ by adding `cuda_v`, which is right for CUDA callers. A
caller with no CUDA now passes nothing and gets no range, which is what
tools/hip/build_windows.bat already does when it calls vswhere directly.
- `cuda_v is None` -> no -version argument at all. CUDA callers are unchanged: cuda_v=(13, 0)
still refuses VS 18, cuda_v=(13, 3) still takes it.
- STRATA_VCVARS=<vcvars64.bat> picks an install by hand, for a vswhere that cannot be used.
Verified on the RX 6800M / gfx1031 machine of #881 (Windows 11, Chinese locale, the only C++ tools
are Visual Studio 2026 / 18.4.11605):
find_vcvars() -> <VS 18>\VC\Auxiliary\Build\vcvars64.bat
find_vcvars(cuda_v=(13, 0)) -> None
find_vcvars(cuda_v=(13, 3)) -> <VS 18>\VC\Auxiliary\Build\vcvars64.bat
With a draft-vocab head (rt/draft_vocab.bin) the slot drafters of --batch-mtp share the main drafter's head but kept
dhead_type_ at -1, so the first draft_first() of a batch admission threw 'mtp: unsupported native MMVQ GGML type'
and the engine exited (#1121). The CUDA source got this line in #1110; the SYCL port did not.
A periodic loop emitted as digits (a 24k-digit string cycling an 11-digit
pattern) is a single word to the word count and never one token twice
(the tokenizer has no multi-digit tokens), so both #606 and #728 passed
it and the run burned the whole reasoning budget. The reasoning's last
2,048 chars are now checked for a period of at most 64 chars, and a
periodic tail reads as full coverage. The opt-in path only: the default
path and the word count are unchanged.
With the Intel port the configured exe is sycl/serve/strata-sycl.sh, a shell script that runs the repo's
STRATA_SYCL_BIN (default build-sycl-aot/strata) in a container. effort_end_args looked for --tail-role-token in the
script, never found it, and fell back to the top-of-prompt effort with "needs engine 0.1.39 or newer", even on a
0.1.41 engine that has the option. engine_binary() resolves the launcher to the binary it runs; any other exe is read
as before. tools/test_effort_end_args.py covers both paths.
Follow query-swizzle #1661 with the existing reduce-scatter pairing for
12 heads. Keep default paths and softmax/value/merge unchanged. Public
component parity and timing passed; model evidence uses frozen stack.
Record archived-control output parity, verified candidate process, tools, CPU vision and cache restoration. No fresh 200K speed comparison or production promotion is claimed.
Preserve FP operation order and invert a bijective shared-query layout at its vector loads. Limit the path to the current gfx906 wave64 device, INT8 pools and warp scoring. Default remains off.
Add deterministic GPU parity/timing and host-guard probes, plus scoped component and model receipts. The clean upstream build passed both-card byte checks; model gains are recorded separately on the frozen experimental stack.
Pinned memory reads the link 1.7x faster (A750 10.3 -> 17.2 GB/s, B70 13.6 -> 25.6), which moves the derived default pcie_frac
(A750 0.28 -> 0.47) and cost 10% of decode there (median 12.07 -> 10.67 tok/s, 4 of 5 interleaved pairs slower). The B70 is unchanged
(0.37 -> 0.55, 31.4 vs 31.4). The fault the PR fixes is a layer split's.
Co-authored-by: maxious <[email protected]>
server_intel.py replaces StrataEngine._pump to keep LP lines beside their token, but its copy predates --batch: the
slots' BT / BDONE lines went to the main queue, so with "parallel" every request stopped after one token. It also
missed the fatal-line handling and last_err. Both are now as in serve/server.py's _pump.
The host mirror holds the experts missing from VRAM at the start. An adaptive swap moves an expert that was resident
then out of VRAM, so it is in neither; the device plan then waits for the host to serve it, and with
STRATA_VERIFY_NO_HOST (the xe default in strata-sycl.sh) nobody does. Seen after a --batch window, whose host path
counts the misses the tier swaps on: the next single request on one card gave token 0 ("!!!!") at 0.8-1.3 tok/s.
The start already refuses experts in neither under NO_HOST; this keeps that true after the start.
The prompt path dequantizes each dense Q8_0 projection (GDN qkv/gate/out,
QSA q/k/v/out, shared expert, hc) into the GEMM scratch before cuBLAS.
dequant_kernel runs one thread per 32-value block, so a warp's 2-byte
stores land 64 bytes apart: about 70 GB/s of writes on a V100.
dequant_q8_0_kernel gives each block to four threads, 8 values each,
written with 16-byte stores: a warp reads 8 consecutive blocks and writes
512 consecutive bytes. The arithmetic per value is unchanged (d * q), so
the output is the same bits. It is used for Q8_0 when the output rows are
contiguous (ld == cols) and the pointers are aligned (blocks to 2 bytes,
output to 16); everything else takes dequant_kernel as before.
V100-SXM2-16GB, every 2-D Q8_0 tensor of Qwen3.8-Flash-Next UD-Q4_K_XL
(498 tensors), each call timed on its own:
attn_qkv 10240 x 2560, FP16 0.740 -> 0.106 ms (70 -> 495 GB/s written)
attn_q 12288 x 2560, FP16 0.887 -> 0.126 ms
ssm_out 2560 x 6144, FP16 0.451 -> 0.065 ms
all 498 tensors, FP16 138.5 -> 20.4 ms
all 498 tensors, BF16 140.8 -> 20.5 ms
all 498 tensors, FP32 190.9 -> 35.4 ms
An FNV-1a hash over all outputs is equal before and after for FP32, BF16
and FP16. Whole prompts on 4x V100 (where this was first used), 4,096-token
chunks: 18K 4,537 -> 4,708, 65K 6,540 -> 6,766, 123K 6,783 -> 6,985 tok/s.
The whole i-quant family (IQ3_XXS, IQ3_S, IQ2_S, IQ4_XS gate/up, IQ4_NL down) and the
fork's Q2_0 down have x86 dots only under __AVX2__ in ggml-cpu; on an AVX1 floor
(STRATA_ISA_FLOOR=avx, issues #595/#623/#1375: Xeon E5 v1-v2, and AVX-512-only packs)
the CPU-computed experts fall to the scalar _generic dots. iq_avx1.cpp is the older
CPUs' copy of iq_avx2.cpp in the kq_avx1.cpp pattern: a separate TU compiled
-msse4.2 -mavx -mno-avx2 -mno-fma -mno-f16c, gated at runtime behind cpu_avx1_ok() and
only when AVX2 is absent, so no mainstream path changes. 128-bit integer lanes
(pshufb/sign/maddubs are SSSE3), 256-bit float accumulation, software half-to-float,
no gathers (the Fmt32 non-gather path), one IEEE mul per value like the AVX-2 copy.
Plus q8k_quant_avx1, the byte-identical Q8_K activation quantizer in SSE4.2/AVX1.
iq_avx1_parity: all six formats, nt=1..8, bit-exact across widths (#152) and within
1e-5 aggregate of ggml's generic dots; q8k_quant_parity now checks the AVX1 copy too.
gu rows take the AVX-2 mt_min policy (STRATA_IQ_MT_MIN), down rows always; kill switch
STRATA_NO_IQ128=1. E5-2470 v2 (Ivy Bridge): 1.2-3.1x per expert at nt>=2, 2.0x on the
Q8_K quantizer; serve prefill +36-41%, warm decode 22-25 -> 46 tok/s.
Known pre-existing gap found while testing: pool_tasks_test SIGILLs on non-AVX-512
machines even on a clean tree (it predates the ISA floor and is not gated).
q2_rows_any and act_quant_any had two rungs, AVX-512 and AVX2, and the else branch
checked nothing: an AVX-only CPU was handed AVX2 code and died with SIGILL (found on
an E5-2470 v2; pool_tasks_test and q2_bitplane_parity crash on a clean tree, and any
future caller inherits the trap). The third rung: q2_avx1.cpp in the iq_avx1.cpp
containment pattern (-msse4.2 -mavx -mno-avx2 -mno-fma -mno-f16c, a #error guard,
called only behind cpu_avx1_ok() && !cpu_avx2_ok()). 128-bit integer lanes (maddubs,
madd), 256-bit float accumulation, software fp16 scales; the legacy kernel keeps the
AVX-2 kernel's lane pairing and reduction tree (its FMA becomes a mul+add, last bits),
and the bit-plane kernel and its image builder port as-is (pure SSE). The AVX-2 and
AVX-512 paths are untouched. q2_bitplane_parity now runs on the AVX-2 pair where the
CPU has AVX2 and on the AVX1 pair below it, and skips cleanly without AVX.
E5-2470 v2: pool_tasks_test 168 bitwise comparisons pass, q2_bitplane_parity PASS -
both SIGILL'd before.
A helper GPU's expert cache already holds a copy of the blobs the prompt path
streams, and the helpers compute nothing while a prompt is read on a serial
request path. Add RemotePrefillSource, an ExpertSource that wraps the file
source and asks the helpers first: a blob a helper holds is copied
cache-to-host on that helper's own stream, misses are merged into one call to
the fallback's existing batched reads, and every other decision is forwarded
unchanged.
Opt-in through STRATA_PREFILL_HELPER_SOURCE; it applies only where the serve
path is serial (one GPU, no layer split, no batch slots) and the prompt path
streams every expert the main card does not hold. Same-day A/B on an
11-request fixture: 91.5 s -> 65.7 s of prefill (-28.3%) and 92.1 GB (64.8%)
fewer bytes read from the files.
The share is picked from wall-clock timings, so the same short-prompt
request can split the experts differently each run and the answer moves
in its last bits - a resumed conversation's re-read need not match its
first (31 of 1,536 teacher-forced argmax tokens differed in the
reporter's test). A fixed share repeats; 0 gives a run without it
byte-for-byte.
Five simulated agent sessions over the repository's own files (system prompt, four tools, real files as tool
results, at most 2000 lines per read): 23 turns from ~2.5K to ~100K tokens, consecutive turns sharing their prefix.
Arms as ab_engine.py takes them; rounds mirror the arm order on fresh servers; prints the median of the per-turn
decode ratios to the first arm and the turns it won. Nothing personal and nothing downloaded.
A missed expert that every token of the verify window routes at rank R or lower in weight contributes zero
(no renormalisation): less PCIe traffic and CPU expert work. Counted in the request log. Opt-in, changes
the output.
The top-10 is picked again with a logit bonus lambda for the experts resident in VRAM; the weights stay the
router's probabilities renormalised over the 10 picked. After 'Mixture of Cache-Conditional Experts'
(arXiv 2412.00099), without training. Complements STRATA_ROUTE_RESIDENT (a tail swap within a margin):
every rank can move, by an amount the logits decide.
With --resident-experts (setup's low-RAM mode when the experts the GPU does not hold fit the RAM) the start copies
the profile fill of the GPU cache and the RAM copy (pin_cache_complement) through the mapped view of experts.bin.
Every page it touches sits in the process working set beside the page-locked copy until #467's trim after the fill
and the VirtualUnlock after the copy: on a PC short of RAM that pages other memory out, and the copies run at the
page-fault rate (~0.85 GB/s here).
FileExpertSource::begin_startup_unbuffered closes the view, the section and the buffered handle, opens the
unbuffered handles (open_direct) and reads the first blob both ways; if the open fails or the two differ, the view is
mapped again and the start copies through it as before. The fill (prefetch_pairs) and the copy (pin_cache_complement)
then take their existing unbuffered paths (read_direct: requests merged up to 32 MiB). end_startup_unbuffered maps
the view again (map_view, the size checked) and closes the unbuffered handles and the stage buffers, so the file tier
is chosen as before. Same bytes in the same places.
Windows and experts.bin only: the call is under _WIN32, and the GGUF in place, an unbuffered file tier and runs
without a RAM copy are unchanged. STRATA_STARTUP_UNBUFFERED=0 gives the mapped copy back; STRATA_VERIFY_COMPLEMENT=1
prints checksums of the filled cache slots and of the RAM copy. pin_cache_complement no longer VirtualUnlocks the
range of a closed view.
IQ2_XS (33 GiB experts.bin), RTX 5090, Samsung 9100 PRO, --resident-experts --expert-cache auto (~15,200 slots, a
16.4 GiB RAM copy), interleaved against fb58e0db:
- 21 GiB available (a RAM ballast; #467's 32 GB PC had 20.7 GiB), 6 pairs: ready 42.5 -> 9.6 s (chat), 40.7 -> 10.2 s
(32K); the fill 842 -> 6,914 MB/s, the copy 13.7 -> 3.3 s; lowest available 0.25 -> 4.5 GiB; pages written out
~60,000 -> 579-3,340; 3 of 12 base starts held available RAM under 1 GiB for 2 s, none of 12 here.
- 128 GB: peak working set 32.6 -> 18.3 GiB; a restart with the file cached 13.6 -> 8.3 s; the 32K prompt and decode
unchanged.
- Same bits: p2k 200 tokens and first-token logits, 32K 64 / 256 tokens, as fb58e0db; STRATA_VERIFY_COMPLEMENT=1
checksums identical to STRATA_STARTUP_UNBUFFERED=0 (12,595 and 15,438 slots).
Based on RealScaniX's sorted-by-name change; the first request of a set stays byte for byte as the client sent it.
Co-authored-by: RealScaniX <[email protected]>
RulesAgreeWithTheWrittenConfig.test_the_streaming_rule_is_the_flag_setup_writes runs setup.main() through
test_setup_golden's install(), which mocks is_wsl() to False, and then asked setup.kv_streaming_wanted() outside
that mock. Inside WSL the real is_wsl() is True, so the rule answered "no streaming" while the config setup wrote
for the mocked PC has --kv-resident, and the test failed at its first 131072 case (Q2_0, int8, auto). With is_wsl()
the same on both sides, setup and the rule agree in all 54 cases, inside WSL and out.
The test method now runs under the same is_wsl mock as install(), the test-only fix the unsloth tests got for #974
(d1ec32b). setup.py is unchanged; where is_wsl() is already False (Windows, native Linux) the test runs as before.
Tests: python3 tools/test_setup_risk.py (WSL2, Python 3.12.3: was FAILED (failures=1), now 40 tests OK); the other
27 tools/test_setup_*.py files give the same results as before.
A request with "temperature": "0.7", "top_k": 40.0 or "seed": true was
answered 200 and sampled greedy: the value replaced the configured
default in run()'s merge, then sampling_keys dropped it as not a number.
check_request_sampling refuses it by name before anything is sent, as
POST /settings and the run config already do. Only types are checked;
ranges, null fields and in-type values behave as before.
A chat "file" or "input_audio" part, and a messages "document" block
(also inside a tool_result), were dropped without a word: the request got
a 200 and the model answered about a file it never saw. _check_parts
refuses them while the request is converted, naming the type and where
it sits, with /v1/responses' wording. Text, images, thinking, tool_use
and tool_result read as before; redacted_thinking is still dropped; a
chat "refusal" part reads as its text, as on /v1/responses.
The row-count sort of a fully resident layer's experts skipped native-layout packs, though slot/src/off are
built the same way for both layouts. 4x GP100 (sm_60): 1.2K prompt 7.27 -> 5.32 s, 35K 63.3 -> 54.1 s, the
same bits. The switch stays opt-in.
kvg_trim refilled the slots the K/V gave back and then uploaded host_res with a plain cudaMemcpy from pageable memory.
That returns once the table is staged, before its DMA lands, and nothing after it waited for stream 0: the next
request's session_zero syncs main_stream only, and a prompt part of --short-read tokens or fewer goes straight to
ver.run, whose graph reads hits_.d_res on the verifier's cudaStreamNonBlocking stream. res_put (#550) is the upload
every other residency change uses: the same copy, then cudaStreamSynchronize(nullptr).
kvg_ensure's upload needs nothing: qsa_kv_elastic_grow ends in cudaDeviceSynchronize before anything reads the table.
Only --kv-grow (opt-in) runs kvg_trim, so the default path is unchanged.
With stream_tools an array or object argument is held until it is whole,
so a large one (a measured 66 KB edit_file call) left the stream silent
for minutes while tokens were generated - a client's idle timeout then
ended a healthy request (Forge's is 120 s).
The token loop now tracks when anything last went out and sends the
engine's own keep-alive ping after KEEPALIVE_S (10 s) of silence - the
same ping the quiet-engine heartbeat sends, so clients already ignore it.
The reporter's reproduction measured a 27.8 s longest gap on a 7 KB
array (minutes on the 66 KB one); with this change it is 10.0 s and the
call arrives intact.
Change and live measurements from the issue; adds the ping-cadence unit
test (a patched clock over a held-back <tool_call>).
The default router, publish and copy_or_zero kernels are the f-142 ones. The probes set the variable (and
STRATA_DOORBELL_CHECK=1 for publish_res, which copies x only on a miss otherwise).
Port the fork's fast router with a bracketed reciprocal and serial fallback. Add bitwise comparisons against the existing router, including ties, NaN tails and the forced serial path. On the A770's 256-expert/top-10 shape, measured host submission plus execution fell from 395 to 119 us per call.
Pair activation payload stores and mapped result reads into 64-bit uncached transfers, preserving the checksum and unaligned/odd-tail fallback. Register router and payload probes with CTest. The two probes, elementwise_parity and router_top10_parity passed on the A770.
Add the fork's opt-in host-clock stage profiler for further investigation. Profiling remains experimental: enabled runs encountered a payload-arrival timeout, and review flagged concurrent atomic host writes versus non-atomic device clock reads as a synchronization concern. The profiler is not needed for normal inference.
With CPU-computed misses, cache borrowing and an 8.98 GiB VRAM cache, unprofiled Coder IQ1_M runs generated 128 tokens at 11.9-13.5 tok/s and returned final arithmetic answer 41. These runs retained 128K context capacity but used short prompts; the 20 tok/s goal has not been achieved.
The admission checks prompt + max_new <= max_context, but the spec loop
opened every window at S_mtp rows, so a request that fits ended with
"ran out of context" (rc 2) once p + S passed the end, and printed
nothing. The window now takes only the rows the context still holds.
MtpDrafter::draft wrote draft j at cell p + a + j for every step, past
max_cells near the end, where the page table has no entry; it now drafts
only the cells that exist. Far from the end nothing changes.
CUDA 12 refuses a GCC newer than 14, and the docs already tell the user
to run CXX=g++-14 ./setup.sh - but the environment only reaches CMake on
a build folder's FIRST configure, so on a retry (or the encoder's own
folder) the flag never landed: the reporter's run died at the last step.
host_compiler_defs() maps them onto -DCMAKE_C_COMPILER /
-DCMAKE_CXX_COMPILER / -DCMAKE_CUDA_HOST_COMPILER for the engine and the
CUDA image encoder; CUDAHOSTCXX defaults to CXX, as CMake's does. A
C++-only build (the CPU encoder) is not given the CUDA one.
STRATA_VERIFY_STEPPED=1 (Arc A-series): the doorbell window is captured as graph segments ending at each layer's
publish; the host serves the CPU experts between segments, pipelined with the plan copy + VRAM groups on a helper
thread pinned to the host thread's SMT sibling. No kernel then reads a host store while it runs and no host reads a
kernel store while it runs, so the window's mapped buffers are ordinary host USM and the publishes use plain stores
(STRATA_VERIFY_HOST_UNCACHED=1 keeps both uncached). A770, Coder IQ1_M, 8K prompt: GPU wait per window ~70 -> ~43 ms,
decode 22.8-25.0 -> 31.8-32.5 tok/s; 32K 27.2 tok/s.
The pipelined slot groups never run the adaptive tier, so --adapt-async 1 with --batch on a layer split and no
--batch-groups resolves auto to one group, like --batch-mtp (batch_groups.hpp, batch_groups_test); --batch-groups G
or auto given on the command line is honoured and turns --adapt-async off through the existing gate.
batch_step advances the async round one step per window (never between a prompt's chunks), commands drain the round in
flight before they run, a round with nothing to swap ends at its first step, and --adapt-min-gain sets the bar a swap
must clear (default 1.5, as before) for both tiers. --adapt-async is no longer refused with --batch slots (still with
--batch-groups).
batch_step now runs adapt() every --adapt-every windows on a thread beside the commit and the drafts, lands the swaps
(apply_pending) before the window ends, and pauses while a prompt holds a loan of cache slots. The timing line adds the
adaptive tier's rounds/swaps and wait, each stage's GPU-reach wait and pool time, and the VRAM/PCIe/CPU entry split.
The 0.1.41 default --batch-groups auto, the pipelined path's lack of slot drafts and the helper-cache refusal are
written down in docs/BATCHING.md (the limits list and the --batch-mtp paragraph).
--batch-groups auto became the default for a layer split with --batch >= 2, but the --batch-mtp gate ran before
the groups were resolved: it saw 1 group, kept batch-MTP on, and the groups were then raised to 2 - pipelined groups
do not run the slots' MTP drafts (#1413). The resolution is now a pure helper (include/strata/program/batch_groups.hpp,
CPU test src/program/batch_groups_test.cpp): with --batch-mtp and no --batch-groups the default is one group and
the log says why; --batch-groups N (N > 1) or --batch-groups auto given on the command line pipelines the groups and
turns --batch-mtp off, as before.
- generate.cpp: --batch-mtp accepted with a layer split on separate GPUs (refused with --batch-groups); slot drafters
load from the last stage's slot sessions under the last stage's device guard and bind to its weights/head; their
residual buffers, row copy (async on the drafter's stream) and draft-KV copies run on the drafter's device; every
stage verifier gets the 8-row window size and the batch graph limit; slot-drafter logits are not charged to CUDA0.
- verify.cpp: stage_batch's layout rules move to include/strata/core/batch_rows.hpp (CPU test batch_rows_test); grouped
rows are allowed on a stage that hands rows on, the hand-off bound is kVerifyMaxT rows.
- docs/BATCHING.md, --batch-mtp help text.
Step 1 warns below 4 GB of page file (issue #60) and advised "Set it to System managed". In #60 that did not help:
"System managed" and 4096-32768 MB still ran out of commit, and a fixed 64 GB worked - a page file Windows grows on
demand may not grow in time while WDDM charges the card's memory to the commit.
- The advice is a fixed size: Custom size, the same initial and maximum size, 65536 MB (PAGE_FILE_FIXED_MB).
- page_file_setting() reads the Virtual memory setting (PagingFiles; "?:" is the drive in ExistingPageFiles). When a
file is system-managed or its initial size is below its maximum, the warning says that it grows on demand and that
a growing page file may not keep up (#60).
- page_file_gb() still counts only what is there for sure: the commit limit minus RAM, as before, or a file's
configured initial size when that is larger (within its drive's free space). Never the maximum or what a
system-managed file may grow to. The 4 GB threshold is unchanged.
- docs/AI_SETUP.md and docs/DETAILS.md give the same advice.
- tools/test_setup_page_file.py: the parsing, the reading, the advice, and step 1 under --check on the golden
harness's 32 GB PC (registry, drives and GlobalMemoryStatusEx mocked; runs on Linux too).
The MCP server's FALLBACK_MODELS carries its own copy of the model
table, so #1688's setup.py fix leaves the MCP install path rejecting
--family swift --model IQ3_S. Drop the families gate there as well,
flip the validation test, and update the three doc lines that say
IQ3_S is original-model-only.
Measured on an RTX 3090 pair (engine 0.1.40.3, Swift IQ3_S, 200K
context, MTP spec on): 91-137 tok/s decode, expert cache hit rate
96-99%, KV streaming 98.6-99.6% VRAM hits.
UkisAI published a Swift 1.5 IQ3_S tier on 2026-10-08; the
"Swift 1.5 has no IQ3_S" restriction is stale. Both shards match the
family's file pattern exactly (names/sizes/SHA-256 verified against the
Hub listing), the shard count and the shared BF16 mmproj are unchanged,
so it takes the same path the other Swift tiers already use.
The IQ2/IQ3/IQ4_NL/IQ1_M decoders now write values [8 tid, 8 tid + 8) of a superblock (coalesced runs, as Q2_0
already did); iq_xmx_grouped still passed the IQ4_XS-style (sub-run, 32-block) index for them, so the B tile held
the wrong values (xmx_group_parity relerr 7-17 on types 18/20/21/22). All six types back to ~1e-6.
- xmx_moe: joint_matrix FP16 GEMM over a group of up to 16 experts per launch (grf 256).
Standalone vs oneMKL on uneven routing: down 1.75-6x, gate/up 0.8-3x (oneMKL kept above
STRATA_PF_XMX_GU_MAX=250 tokens/expert).
- STRATA_PF_XMX=2: iq_xmx_grouped decodes IQ3_XXS/IQ3_S/IQ2_S/IQ4_XS gate/up and IQ4_NL/Q2_0
down rows in-kernel (no FP16 staging); xmx_group_parity: rel err < 2e-6 vs dequant+oneMKL.
- End to end (8K/32K prompt, A770): +2-5% only; default off.
--pipeline-windows 2 now runs on any layer split with two or more stages. Every
stage but the last is a front stage: the guessed window follows the verified
one through them one card behind (an odd verifier and an odd hand-off on every
boundary, a pair of GDN snapshots on every front stage, commits and the undo
per stage), and only the last stage waits for the verdict. The pipelined prompt
read (--pipeline-windows 1) and --adapt-async stay two-stage.
HIP: the draft layer's per-row gr_read used the last stage's session scratch
(ss.block.gr), which that stage's verifier also uses in lm_head_mix. A chain
launched at a verdict and the next window's last stage can run on the card at
once after a guess that held, and the window's head then read a clobbered mix.
The draft layer now carves its own GrWorkspace.
Measured on 4x RX 7900 XT (Flash-Next GSQ-RCO IQ3_S, split 12,24,36, every
expert in VRAM, greedy, reasoning off): decode 59.4 -> 67.6 tok/s at 4K and
51.8 -> 65.6 at 32K. Greedy text identical to the serial loop on four prompts
with STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0, also with every guess
forced wrong. Debug knob STRATA_PIPELINE_SPEC_DEPTH. Two-stage path not re-run
on two cards.
probe_pcie_h2d_gbps: a pageable 256 MiB source made the copy engine fault
inside the driver's bounce path on a layer split (755x bcs VM_NOT_FOUND at
the fixed VA 0xffffc001ffa10000, L4 = the region never mapped for that VM,
intermittent on fresh boots, wedging the whole load in both the 0.1.40
port and 0.1.41). Pin the source (sycl::malloc_host) and free it with the
matching sycl::free. Validated: two consecutive loads reached the window
without the wedge.
All the split-fault instrumentation (guard watch, stage stamps and the
STRATA_STAGE_WAIT serial gate, terminate dumps, coredump grabbers) lives
on scratch/split-fault-instruments.
dpct turned `cudaStream_t cs_ = nullptr` into `&dpct::get_in_order_queue()`: a drafter that never loads keeps it, and
its destructor destroys the device's default queue (exit 139, sometimes UR_RESULT_ERROR_UNINITIALIZED, after the output).
Arc Pro B70 at fb58e0d without --mtp: 3 of 3 runs crashed before, 0 of 3 after; output unchanged with and without --mtp.
0.1.41 report (RTX 5090, tight RAM): chats failed and the engine restarted until STRATA_PREFILL_CPU_SHARE=0.
- share_rules: default_share_memory_verdict. The default is not armed when the RAM this process can get
(cgroup-aware) is below the share's buffers + 3 GiB, when the Windows commit left is below buffers + 4 GiB,
or when the experts it would read are file pages the free RAM cannot keep. One log line says why and that
STRATA_PREFILL_CPU_SHARE=auto turns it on anyway. An explicit setting is kept and only warns.
- prefill: the share's two pinned buffers are taken once at their largest (an expert takes at most MAXT
tokens' rows); a failed allocation ends the share for the run instead of failing the request.
- unit test, docs/DETAILS.md.
The counter block was one process-wide cudaMalloc on whichever device was current at the first Verifier::init, unreachable from
the other stage's kernels on a layer split without P2P. Each verifier now takes its own device's block in init (outside graph
capture), keeps the pointer, and a launch never picks by the host thread's device; a failed allocation runs without counters.
The exit report prints one block per GPU when there are several (single GPU: the same lines as before). The reporter's rare
fault with --pipeline-windows 2 after this fix is not reproduced here (needs two GPUs).
STRATA_ROUTE_RESIDENT_MTP=1 (experimental, opt-in, changes the output): the MTP draft's router gets the same residency bias,
with the last layer's residency row as its table (single GPU).
The refills of the slots a prompt borrowed (and the VRAM-resize refill) queue hundreds of cudaMemcpyAsync out of blob_stable()
pointers, which for a GGUF source are the recycled stage buffers. With STRATA_STAGE_PIN=1 the 257th assembly overwrote a
buffer whose copy had not run: IQ3_XXS lowmem (budget 12) decoded token id 0 after a 4096-token prompt. Each queued copy now
records the fence of its buffer. exact_pp 4096 and 20000 tokens with STRATA_STAGE_PIN=1: EXACT.
A busy slot flag with no owner kept every later request waiting for a slot nothing
would free, and a request waiting for the fifo (a load, an image encode, /v1/vram)
was counted nowhere. Now: slots have owners (claims / BDONE waits) and an ownerless
busy flag is freed after 3 s; the request waiting for a slot is counted
(admitting) and, past the stall time, named in the log; a parallel request on a
loaded engine no longer takes the fifo, and one that waits for it stays queued.
Tests with the fake engine.
On an RX 7900 XTX the first long prompt at 8192-token chunks charged ~70 GB of
commit and the engine died (0xC0000409); 7500 and below were flat. A bare auto
now stops at 6144 on Windows HIP builds. A chunk the user names stays, with a
warning from 8192 up.
Windows: the serve start prints the commit left and warns when the planned host
allocations plus a 4 GiB floor do not fit; the conversation cache's admission
reads min(available RAM, available commit).
With every expert resident nothing streams, so the share has nothing to take and
only moves the staged-chunk limit (+4.8% at 1,000 tokens on a V100). An explicit
STRATA_PREFILL_CPU_SHARE is honoured as before; the default is unchanged where
experts are not all resident.
A stage with one layer holds a handful of expert slots (a 12 GB 3060: 512), and the
prompt chunk of every card in the pipeline is clamped to what that cache can lend.
When the pick has such a stage, search again among placements that give every
stage at least three layers; keep the new pick and print both and why. With none
startable the first pick stays. Identical cards pick balanced stages and never
reach the second search.
available_host_bytes() = min(MemAvailable, the room under every cgroup limit above the engine, the operator's cap).
Parking and session save/restore read MemAvailable alone: in a container whose real limit sits on a parent cgroup it
cannot see, MemAvailable is stale (about 25 GiB shown when about 1 GiB was left) and the engine was OOM-killed.
conversation_available_memory() is replaced by available_host_bytes() / sample_host_memory() (the SYCL port's
generate.cpp too). The cgroup term covers memory.max and memory.high (v2) and memory.limit_in_bytes (v1) of the
process's group and every visible ancestor, with clean inactive file cache credited; an unparsable cgroup file makes the
sample unknown (fail closed). --memory-limit-mib / STRATA_MEMORY_LIMIT_MIB names the total RAM the container or cgroup
may use. Refusals say how much was available and from which source; one start-up line reports the limit.
tests/core/memory_guard_test.cpp runs on fixture directories (CPU only). Docs: DETAILS.md.
A profitable shorter chained lookup returned before the stale full-size check, so the full chain could stay excluded for good
although the policy re-probes stale sizes in the other selection paths (#1252). Check a confident stale full size before
returning the profitable shorter chain. The 300-round policy, the EMA, the header and plain lookup are unchanged; a regression
test covers the bounded re-probe spacing and the MTP confidence gate.
0.1.39 took this PR's first part with #583 (the xn / grs count, STRATA_RING_BYTES). What is left: bytes_needed
counted the fused layout's smaller MoE buffers for every caller (fused_layout(T, true)), while init takes them only
when it gets an ExpertSource (fused_layout(T, m.src != nullptr)). A prompt path without one was lent fewer bytes than
its unfused buffers take. bytes_needed, bytes_needed_no_ring and (0.1.40) bytes_needed_owned - STRATA_OWNED_PRICE=exact's
sizing - now take the same src the callers pass to init.
When the best split and a different one (boundaries at least 3 layers away, startable) are within 4% predicted decode window
(STRATA_SPLIT_NEAR_TIE_PCT, 0 = off) and the cards differ in speed or link, print the other K and --layer-split K. The model
prices decode only, not the prompt. Print only: the choice is unchanged.
A prompt read on a layer split with no slot decoding was one run, so the BYIELD of the batch_interleave_test step 4 was
dropped at the end of the read; the long prompt kept its slot and the next BGEN failed with 'ERR BGEN: no such free slot'
(pre-existing since 0.1.40.4). That read now goes in pieces of STRATA_BATCH_YIELD_CHUNKS chunks (default 8, 0 = one run), the
same chunks a single run reads, with the yield check between pieces. batch_read_piece holds the arithmetic; batch_read_test covers it.
A page-locked stage buffer is the source of the cache fill's cudaMemcpyAsync, which the DMA reads after the call has
returned; a pageable one is staged by the driver before it returns. The pool recycles a buffer after kStageAge layers or 256
assemblies, so a 4096-token prompt read (up to ~500 distinct experts per layer) could overwrite a buffer whose copy had not run
yet: the corrupted IQ3_S decode of #1237. note_async_read(blob, stream) records an event per pinned buffer after each queued
copy (hit-path fill, peer fill, remote-expert swap); claim_stage waits for it before it hands the buffer out again.
Pageable buffers and every other source are unchanged. STRATA_STAGE_PIN stays opt-in.
The reasoning ended at the first </think> in the output, also when the model only
quotes the tag while it thinks (`</think>`, "</think>\n\n2" from a test). The rest
of the thinking then came back as the answer, and a tool call quoted in it could
be run.
A </think> right after a quote or backtick and followed by anything but a line
break is now reasoning text; the end of the thinking is followed by a line break
and ends it as before. Until the character after such a tag arrives, the tag is
held like a partial one, so streamed and whole outputs agree. If the turn ends
right after it, it was the end of the thinking: a forced tool call still opens
there, and the #1053 retry is not triggered, as before.
Parking happens on a switch and every parked conversation costs one of
--conversation-cache-slots whatever its length. A client that interleaves
short side requests (an agent's tool-result follow-ups, often under 1k tokens)
with one long conversation therefore spends the cache on the side requests and,
once it is full, evicts the long one oldest-first - the reuse it wanted.
Measured on an RTX 4070 SUPER / 32 GB / Q2_0, --conversation-cache-slots 2,
one 19,198-token conversation interleaved with three ~700-token requests, then
the same conversation again:
min 0 skip nothing, side requests park, L is evicted
re-send L: 0 reused + 19198 read in 41,059 ms (wall 41.4 s)
min 12288 skip parking (703 tokens, below the 12288 minimum) x3
re-send L: 19193 reused + 5 read in 135 ms (wall 0.3 s, 304x)
The default 0 keeps the previous behaviour, so nothing changes without the flag:
wants() answers for the length alone and the gate stays enabled() && wants(),
exactly as the existing byte-budget and RAM-floor skips are. The check runs
before make_room(), so a skipped park neither evicts nor stores anything, and it
returns the same non-error path those skips take.
Tests: conversation_cache_test covers the inclusive boundary, 0, a negative that
never reaches the CLI, and that a disabled cache still answers for length; both
faults were reintroduced by hand to confirm the checks fail on them.
AMD_HIP.md, RDNA4 speed switches: STRATA_PF_FUSED=1 now runs the native
IQ packs' prompt experts on the int8-WMMA kernels on gfx1200 / gfx1201.
Says that it is opt-in and why (it rounds differently, so it is not an
arch default), that STRATA_PF_FUSED_NATIVE=0 turns the native path off,
the start-up line it prints, the R9700 measurements (prompt time, kernel
vs MMQ, first-token KL against the FP16 path, retrieval checks), and
what was not run (gfx1200, Windows, K-quant packs).
DETAILS.md: the STRATA_PF_FUSED=1 sentence says where the native kernels
run on HIP. CMakeLists.txt and gfx_arch.hpp: the comments that said the
native prompt-expert kernels are gfx11 only now include gfx12 (comments
only, no build change).
The native IQ / K-quant prompt-expert kernels (moe_fused_iq.cu,
native_w11_kernel) were gfx11-only; on RDNA4 a pack with
STRATA_PF_FUSED=1 silently kept the MMQ path. They now compile and run
on gfx1200 / gfx1201. Still opt-in: STRATA_PF_FUSED=1 stays off by
default (it changes rounding, so it is not an arch_defaults switch), and
STRATA_PF_FUSED_NATIVE=0 still turns the native kernels off.
Compile gating
- STRATA_NAT_W12 for __gfx1200__ / __gfx1201__, STRATA_NAT_WMMA for
either family. The kernel calls the _gfx12 spelling of the iu8 WMMA
builtin (the un-suffixed one does not compile there). Any other
target keeps the trapping body.
Kernel (same arithmetic, other lane layout)
- gfx12 WMMA fragments are 8 int8 per lane and the two lane halves hold
different k (8*hi .. 8*hi+7), not a replica as on gfx11: B and A are
loaded as uint2 at the matching offsets, and every 16-value k step is
still one WMMA, so the per-16 scales of the IQ2_S / IQ2_XS paths are
untouched.
- A lane's 8 accumulator rows are consecutive (8*hi + i), so a feature's
gate and up are in the same lane: SwiGLU needs no __shfl_xor, and the
down tile is stored as two float4 per lane. NW_DROW(i, hi) gives the
row of element i for the per-row scales.
Host side
- gfx_arch_is_gfx12_wmma(): exact gfx1200 / gfx1201 match, never a
prefix, like the gfx11 helper.
- dev_info() takes the gfx11 list or gfx12, and also checks that the
kernel the build loaded is the real one: the trap stub (a
gfx12-generic or SPIR-V build on a listed card) declares no LDS and
still passes hipFuncGetAttributes and the occupancy query.
- native_supported() reads the STRATA_PF_FUSED=1 opt-in itself:
fused::available() / requested() belong to the Q2_0 pack's
expert_w11_kernel, which is not ported and stays false on gfx12. Off
gfx12 the result is what it was. prefill.cpp fused_ring() follows:
a native pack asks native_supported() alone (it implied enabled()
before), so on gfx12 it gets the fused buffer layout and ring.
- Grid factor per format: gfx12 holds 3 blocks per WGP for IQ2_S / IQ2_XS
gate/up (LDS and VGPRs), 4 for the rest. The occupancy API cannot say
(it models 64 KB of LDS per multiprocessor), so it is a small table;
STRATA_PF_OCC_W=N overrides it. gfx11 keeps 4.
- experts_native() refuses a dm that is not 16-byte aligned on gfx12
(the float4 stores); the prompt path's buffers always are.
- moe_fused.cu compares the gfx11 list with the helper instead of the
"gfx11" prefix (no change on any listed part).
Tests (prefill_fused_iq_test)
- Runs on gfx12 (the skip gate also asks native_supported()), covers the
other four pairs of the Coder IQ1_M pack, S20_E sets the expert count,
and the real-shape check of fused against MMQ is now rel RMS <= 1e-3
and worst row <= 2e-2 instead of rel RMS <= 0.05.
Measured on a Radeon AI PRO R9700 (gfx1201), Coder IQ1_M pack:
- parity: all 10 IQ pairs plus Q4_K / Q5_K pass, rel RMS <= 9.1e-5.
- kernels vs MMQ, 48 layers: 1.48x / 1.20x / 1.14x at 4K / 16K / 32K.
- prompt speed in the engine (server-reported, alternating starts per
arm): 4K +12% to +19% depending on the start, 16K +8.3%, 32K +8.5%.
- first-token KL against the FP16 prompt path is 1.10x MMQ's (48 prompts
of 4K), top-1 agreement 48/48; 4/4 long-context retrieval checks.
Not run on gfx11 or CUDA hardware here.
unsloth/Qwen3.8-Flash-Next-GGUF UD-Q5_K_XL shard 1 holds metadata only and
is 10,946,618 bytes; its 32-byte-aligned data start is 10,946,624, so the
engine refused it with 'GGUF: data section starts past EOF'. llama.cpp's
gguf-py reads the same file (0 tensors, data_offset 10,946,624).
A file with tensors is still refused. A file without them gets data_start
clamped to the file size, so file_size() - data_start() stays 0 in the
payload checks downstream.
On one GPU the prompt path computes the draft layer's K/V for each chunk in batches (Prefill::draft_kv, E-9). A layer
split always ran the drafter's own per-group pass instead, because the drafter sits on the last stage. With
STRATA_SPLIT_MTP_BATCH=1 the last stage's prompt path runs E-9 there. Default off: the default path is unchanged.
The drafter's K/V come from Q8_1 x Q8_0 MMQ instead of its own mmvq, so its drafts can differ; the target's tokens are
decided by the verify window.
Prefill::draft_kv says why it declined (optional out-parameter, also in the SYCL port's prefill.cpp), and the split
logs once what the variable did. Which path runs is a pure header (include/strata/program/draft_kv_plan.hpp) with a
CPU test. tools/split_mtp_batch_parity.py compares the greedy texts of the two arms on a live server. Docs: MULTI_GPU.md.
Two synthetic cases: no tensors and the file ends at its header (should
open; data_start() clamped to the file size), and one tensor with the same
truncation (still refused). The first fails on 0.1.40.3: unsloth's
UD-Q5_K_XL shard 1 is that case.
Header-only placement (include/strata/platform/aux_cpus.hpp): the spare CPU set (the host core's SMT siblings, then
CPUs of physical cores with no worker; never a CPU that shares a core with a worker), the list parser, threads the
engine creates pin themselves, threads somebody else created (the CUDA driver's) are moved by a sweep over
/proc/self/task after the pool exists and at every request; pool workers and the host thread are never moved.
--aux-cpus off|auto|LIST / STRATA_AUX_CPUS; per request strata_tune aux_cpus=0|1 (serve/server.py).
Default off: the hooks do nothing. tests/platform/aux_cpus_test covers the derivation and live threads (CPU only).
Builds on #1190. The layer split sized its lend regions (the RAM copies of the experts the prompt path borrows) with
Prefill::bytes_needed before the split's ring override was applied (Prefill::set_ring_override says it must be set
before any bytes_needed), so with STRATA_SPLIT_RING=384 the prompt path borrowed 5396 slots and the copy kept 4662.
The override is now decided first (decide_split_ring); include/strata/program/split_ring.hpp holds the pure helpers
(split_ring_slots, split_lend_slots - the scan's own tail-slot arithmetic, now one function for the region and the
borrow), src/program/split_ring_test.cpp covers them on the CPU. A start line reports borrowed vs kept slots and warns
when kept < borrowed. Docs: MULTI_GPU.md.
Engine 0.1.40 (fe9c10ca, #848) keeps the resident RAM copy on a layer split -
every stage's cache left out of it, the rest ranked by the whole expert
profile - and --resident-budget-gib runs through the same code
(pin_cache_complement), so the old engine refusal of the budget on a split is
gone. setup.py's split_mmap() got the matching exemption (43c8cfa9), but the
budget's three paths still assumed the old engine (#498):
- split_budget(): a start of a budget config on several GPUs deleted
--resident-budget-gib (and asked for ~135 GB of RAM below that);
- offer_together(): a budget config was offered both cards only with ~135 GB
of RAM, and "on both it has no budget";
- unsloth_together() / q4_split at install: --gpus, or "2" when asked, dropped
the budget, and an explicit --resident-budget-gib kept one GPU.
From RESIDENT_SPLIT_ENGINE the budget is kept in all three, at any RAM. The
recommendations stay as they were (the golden configs are unchanged): one GPU
by default at install, "n" by default when a start offers both cards, --yes
alone one GPU; the split is taken only when asked for.
Tests: the #498 tests run with MIN_ENGINE pinned before RESIDENT_SPLIT_ENGINE
(as test_setup_risk's StartOnSeveralGpus does for #642), three new ones run the
kept budget; tools/test_setup_*.py on Linux: 421 tests, the 2 failures of
test_setup_engine_hash are the same without this change (no ready-made 0.1.41
engine to download in the test box).
Measured with engine 0.1.41 on an RTX 3090 (x16) + RTX 3080 Ti (x4), 91 GB of
RAM, UD-Q4_K_XL, budget 64 GiB: the auto split gives the x4 card layers 0-1
(678 slots) and the 3090 layers 2-47 (4,182 slots); 57.6 GiB of experts in RAM,
no file-tier reads per request; decode 45.0 / 45.1 tok/s (two runs) against
38.4 on the 3090 alone, a 20K prompt in ~18 s on both. Answers checked by hand.
A pip launcher whose Python was removed stays on PATH ('ninja --version'
fails) and stopped every build there, though setup had just installed a
working copy in .venv.
UPDATE.bat currently prints a command using `&&` for users to copy and
run manually. `&&` is not supported by Windows PowerShell 5.1, which
is still included with Windows 11.
Print the two Git commands separately instead.
dpct migrated cudaStream_t ext_stream_ = nullptr to a pointer to the
in-order queue, evaluated at construction on whichever device is current.
Two places then misread it: init() compared ext_stream_ against the
*init-time* device's in-order queue, so a stage built on one device and
init'ed on another adopted the constructing device's queue
(cross-context submit failures, UR_RESULT_ERROR_DEVICE_LOST); teardown's
'cs_ != ext_stream_' ownership test drifted the same way.
Restore the CUDA contract: nullptr means 'no stream until set_stream()',
matching include/strata/core/verify.hpp:319 and src/core/verify.cpp:604.
Follow-up to the >4 GiB GEMM staging.
- gemm_stage freed its scratch as soon as a larger operand arrived, but the
in-order queue is asynchronous: a previous staged GEMM could still read the
buffer. Wait for the queue before the free.
- The staging used dpct::get_in_order_queue() while the GEMM runs on the
descriptor's queue (strata::q_of(stream_)); pass the descriptor's queue so
the copy is ordered before the GEMM for a non-default stream, and so two
Gemm objects on different queues do not share the thread_local scratch.
- Staging that could not allocate fell silently back to the direct (wrong)
GEMM; it now says so.
- device_alloc_map had no unregister, so a freed cache base left a stale
entry that device_offset_end could mis-attribute after the VA was reused.
Add device_alloc_unregister and call it where the expert cache frees base_.
setup reads the Intel cards from sysfs but to_sycl() decided the expert
mode from the driver alone, so a card's device id was thrown away and an
A-series on xe (the A770) could not be told from a Battlemage. Carry the
PCI device id through intel_gpus() into to_sycl(), and treat every
Alchemist id (A770/A750/A580/A380/A310/A770M) as an A-series: it has no
FP64 hardware, so its config carries the driver's FP64 emulation (a kernel
on the sampled path declares double and is refused without it).
On i915 (the A750) the experts still load into a CPU-computed RAM arena
(--mmap-experts + STRATA_VERIFY_NO_HOST=0) - a single pinned host
allocation past ~3 GiB fails on that die. On xe (the A770) the A-series
now takes the ordinary streamed path: the mirror is built in chunks (so
the >3 GiB limit is not hit) and the build passes
-ze-opt-greater-than-4GB-buffer-required, so the prompt path is correct
past the cache's 4 GiB. tools/test_setup_sycl.py passes.
On an Arc A770 every prompt answered "!!!!" (token 0). The experts in the
last 0.46 GiB of the 4.46 GiB VRAM expert cache dequantized to NaN in the
prompt path. The slots' bytes were right (a memcpy read back equal to the
GGUF, and a fresh copy dequantized clean); only kernels reading them in place
got wrong bytes. In a JIT build IGC addresses kernel pointer arguments as a
32-bit offset from the allocation's base.
The JIT (non-AOT) build now passes -ze-opt-greater-than-4GB-buffer-required
to the spir64 backend, in the same argument as the divide/sqrt option (CMake
drops a repeated -Xsycl-target-backend=spir64). Measured on the A770, Coder
IQ1_M, --stream-experts: prompts of 73-542 tokens have 0 non-finite values
under STRATA_DBG_NAN=1, and the 542-token prompt answers right. The AOT build
is unchanged. Speed cost not measured.
oneMKL's GEMM reads the last, partial tile of an operand from the wrong
place when that operand lies more than 4 GiB into its device allocation
(measured on an Arc A770, oneMKL 2026.1: a BF16 B past 4 GiB gave its last
21 columns wrong, none below 4 GiB). The prompt path's GEMMs meet this:
their buffers come from the top of the multi-GiB expert cache / prefill
region. This is the leading explanation for the A770's garbage on prompts
longer than a few dozen tokens - it repros with every expert
device-resident, so it is not the CPU handshake.
Register every big device allocation (malloc_device_guarded, and the
segmented expert cache's reserved VA) in a small registry, add
device_offset_end(ptr, bytes), and in Gemm::bf16 / Gemm::f16 copy either
operand into a scratch buffer of the queue's own (a fresh allocation, so
at offset 0) when its end lies past 4 GiB, then run the GEMM on the staged
operand. STRATA_SYCL_GEMM_NO_STAGE=1 is the negative control.
Not validated end to end: the run that showed the garbage was an
all-resident mirror, which no longer fits in this machine's RAM, and the
runnable --mmap-experts config crashes on this A770 with or without the
staging (exit 139, UR_RESULT_ERROR_DEVICE_LOST - verified a separate,
pre-existing fault with the negative control), so the staging has not yet
been shown to change an answer.
The decode handshake publishes a payload (the layer's activations, the
routed expert ids and weights) and then rings a sequence number the host
polls. The fences alone do not order the ring after the payload stores:
measured on an Arc A770, 14 of 2,832 rings in one run became visible
before the last activation store, so the CPU expert pool sometimes
computed on a partly stale payload. This is the fix the port's own
comments call for (elementwise.hpp: "the host can observe the ring and
then read a payload that has not landed").
The publish kernels now write, next to the sequence number, the ring it
belongs to and a checksum of the payload (the wrapping sum of every 32-bit
word mixed with its position, murmur3's finalizer - a plain sum missed a
reordering, [3,7] -> [7,3], of the expert ids) in a per-ring slot modulo 4,
and the payload is stored with the uncached write hint (a plain store to
host USM reaches the host only when the kernel ends on xe). The host waits
for the whole payload (doorbell_wait_payload) before the pool reads it, in
session_loop and both verify-window loops; the checksum is accepted in
either publish shape (with or without the routing weights), so a variant's
choice need not be plumbed to the reader. The sequence word grows from 1 to
9 mapped words (layer.cpp, verify.cpp) and the ring slot is reset with it.
Note this is a robustness fix, not the cure for the A770's garbage on
longer prompts: that one repros with every expert device-resident (no CPU
handshake) and points at oneMKL's >4 GiB operand bug, which needs the BLAS
staging the CUDA/HIP shims have.
At a long context this port borrows expert-cache slots for the prompt
path. Those buffers are carved from the top of the ~10 GiB expert-cache
allocation, and oneMKL's BF16 GEMM reads a wrong tile when an operand lies
more than 4 GiB into its allocation (docs/INTEL.md). The borrowed operands
hit that, so from the first full prefill chunk the router read garbage.
Measured on an Arc A770: a 236-token prompt failed with "prefill: routed
id out of range", and a 452-token one segfaulted the engine (exit 139),
on both the mirrored and mmap configurations. With the prompt path on its
own buffers (its own allocation, reserved exactly by owned_prefill_mib)
and an automatic chunk capped at 2048 tokens, 452 / 1,748 / 3,476-token
prefill chunks ran to completion with no prefill failure (prompt reads
22 (cold) / 676 / 835 tok/s). Whether the *answers* of that configuration
are correct is a separate, still-open issue: the base port has no doorbell
payload checksum and its longer-prompt path garbles on some inputs, with
or without borrowing.
Scoped to Alchemist (PCI device ids, the same table setup_intel.py uses):
Battlemage's long prompts were measured correct with borrowing, so it
keeps the old default. --prefill-borrow still borrows, for testing a
future BLAS staging fix (the fix that lets borrowing return, as the
CUDA/HIP shims already stage).
Two Alchemist (Arc A-series) fixes for the streamed-expert mirror:
- gguf_expert_source: a single sycl::malloc_host past ~3 GiB fails on DG2,
so the mirror (thousands of experts, tens of GiB) could not be built on
an A770 at all. It is now allocated in <=2 GiB chunks; the per-(layer,
expert) address, the layer's first blob and close() use the chunk
pointers. A cleanup guard frees every chunk on an allocation failure,
a failed read, or any exception; the read phase is unchanged.
- strata-sycl.sh: docker's default memlock is 8 MiB, far below what pinning
the mirror or the resident arena needs; pass --ulimit memlock=-1.
Measured on an Arc A770: the Coder IQ1_M (8,474 experts missing VRAM,
16.15 GiB) mirrors and the engine starts; before, it refused with
"8474 experts are neither in VRAM nor mirrored".
The decode handshake publishes a ring sequence the host polls while the
window graph still runs. On an Arc (xe) a system-scope atomic store, even
with a release fence, is served from the GPU's cache and reaches the host
only when the kernel ends, so a host polling mid-window never sees the
ring. That is why the verify window had to wait for the whole graph
(STRATA_VERIFY_NO_HOST) and every expert had to be device-resident or in
the pinned mirror.
STRATA_DOORBELL_ATOMIC_LOAD already fixes the load side the same way; this
does the store side with the Intel uncached L1+L3 write hint. The release
fence stays (ordering only: a later payload store follows the ring).
Measured with sycl/probe/doorbell_store.cpp on an Arc A770: the host sees
the store at 230 ms with the kernel running to 2030 ms; with the old atomic
store it sees it at 2038 ms, exactly when the kernel ends (exit code 3).
-DSTRATA_DOORBELL_ATOMIC_STORE restores the atomic store.
This is a prerequisite for running with the CPU expert pool on xe (the
per-layer handshake); it does not by itself fix longer-prompt prefill.
psutil.disk_io_counters() returns None when it finds no disk (Modal's gVisor
has no /proc/diskstats; psutil's docs say Windows can need `diskperf -y`).
_disk() then raised AttributeError at the first sample and the sampler thread
ended: /metrics' hardware stayed empty, the Monitor tab had nothing to show,
and the frozen-engine check's gpu_busy() read a GPU load of 0. It now returns
no disk reading and the other readings go on.
The READY-read timeout from #1317 kills the engine, but the request that
was loading it ended with no response: the RuntimeError from __init__
matched none of the request path's handlers, and on a lazy first start
restart()'s finally crashed on slot_cv (created only after READY), so a
misleading 400 masked the failure instead.
- raise EngineDied (still RuntimeError) for the start failure, so the
request ends 503 'did not report READY ...; the next request restarts
it' on the load, restart and /v1/load paths alike
- skip the slot_cv wakeup when a lazy first start never created it
serve/test_ready_timeout.py covers both paths with silent fake engines:
the lazy load (unfixed: 400) and a restart that never reports READY
(unfixed: connection closed with no answer).
#1352 orders an auto layer split faster-card-last because the last stage
runs the head, the draft layer and the verify. On a VRAM-asymmetric pair
that rule can put the heaviest stage on the card least able to hold it:
the reporter's 12 GB 4070 Ti ended up with 31 layers + head + verify and
a 3 GiB expert cache (decode ~58 -> ~44 tok/s, the 'card this small'
warning on).
Skip the reorder when it would put the last stage on a card with less
total VRAM than the one the config placed there (NVML / amdgpu sysfs;
unreadable cards keep the reorder). scores/vrams stay injectable for
tests.
The VRAM reuse stays, but bytes_needed_impl counts bo again by default, so the auto chunk and the borrowed cache slots
are those of 0.1.40.3 and the prompt residuals stay byte-identical (on the RTX 3060 the planner otherwise picked a
6656-token chunk instead of 6400). STRATA_EMB_REUSE_ACCOUNT=1 lets the planner use the saved bytes (+3.9% prompt there);
documented in docs/DETAILS.md.
The previous change skipped the DRM-fdinfo subtraction whenever the driver reported usage. On a single Arc Pro B70 that
made the expert cache 12168 slots instead of 11338 (23.09 vs 21.52 GiB), so a one-GPU run was not unchanged. The
subtraction is now skipped only when the process sees more than one GPU (where it double-counts the other card) and the
driver does not report the whole card as free.
The bound of the device's spins on host flags was a build constant: 20,000 reads for a bmg AOT build, 2,000,000
for anything else. A B70 built without AOT (SPIR-V, JIT: the Windows/OpenCL build, or a Linux build with no AOT
target) got the A750-sized bound. The issue reports a 37% drop in decode between 0.1.40.2 and 0.1.40.3. On a Linux B70 JIT build the two bounds decode
alike (--spec 2, 3 pairs: 19.4 vs 19.5 tok/s), so this makes the bound independent of the AOT flag but is not shown to be that drop's cause.
spin_max(queue) decides per device: 20,000 under the xe driver (the B-series on Linux), 2,000,000 elsewhere (the i915
A-series needs it for the host's per-layer expert work; a Windows/OpenCL B-series build measured faster with the long bound).
STRATA_SPIN_MAX=<reads> overrides it at run time; -DSTRATA_SYCL_SPIN_MAX=<reads> still fixes one bound for the
whole build. The three waiting kernels take the bound as an argument.
- generate.cpp: with a layer split the host mirror walked the later stages' layers too (#1054, fixed in #1111 but
lost in the re-migration): 20 GiB of pinned RAM for experts the other card holds, or one 39 GiB pinned allocation
that fails. Both loops stop at the first stage's last layer.
- expert_cache.cpp: device_free_bytes() subtracted the DRM bytes of every card of the process from the card it asked
about, so on a layer split the second card was charged the first one's cache and its expert cache would not open.
The fallback is for drivers that report the whole card as free (Arc A750/i915); it now runs only then.
Reported by crobe201 with both diffs; MaineTim confirmed the second on 2x B60.
Co-authored-by: crobe201 <[email protected]>
The three sites fixed by the previous commit stored a clamped (d, sum) but still
quantized against the unclamped d, so an overflowing block stored int8 values
that no longer matched its stored scale. They now use q8_1_finite(d) and
q8_1_quant like native_quantize_q8_1_kernel, the swiglu quantizer and the CPU
pool. A block whose amax is finite and below 8.3M is stored bit for bit as
before.
Co-authored-by: Matthew Hines <[email protected]>
Three kernels still emit the (d, sum) pair of a q8_1 activation block
with a raw make_half2(d, sum), bypassing the finite helper the rest of
the tree uses (q8_1_finite.hpp, from 19de7f4 / f6601bf):
fused_gr.cu gr_q8_tail - the fused gate-reduce tail (QFUSE)
verify_kernels.cu gdn_q8_1_store - the batched GDN verify store (QFUSE)
iq_kernels.cu s26_swiglu_q8_1_kernel - the S26 swiglu path
A block whose amax overflows (or that reads a non-finite input) emits
d = inf/NaN; fp16 makes the pair NaN (or the pathological (0,0)). Nothing
downstream tests for it: a NaN scale poisons the fp16 dot of the quantized
activation and every sum downstream of it, and a fully non-finite logit
row ends at the sampler's all-NaN fallback - vocabulary id 0, the '!'
face of the degeneration reported in #606. Same defect class already
fixed at the other emit sites; these three were missed.
q8_1_ds() clamps both halves into the fp16-finite range (finite d,
non-negative finite sum - the sum is a mean of |x|), producing for the
GPU paths the same bits the CPU pool and the clamped GPU sites produce
for the same block. A clamp at the producer, content-blind: nothing
downstream of it is model-text-aware.
Criteria and the wider carried-state RCA: docs/Q8_1_FINITE_DS.md and
docs/DEGENERATION_RCA.md.
2e4ddf6 (the #904 port) removed __restrict__ from the activation pointers of
native_quantize_q8_1_kernel, bf16_f32_mmvf_multi_kernel and
native_mmvq_multi_kernel on every CUDA architecture, and turned on the weight
prefetch before pdl_wait(). Both only matter for PDL (sm_90+). On sm_61 the
missing restrict loses the read-only data path (LDG.CI) and decode halves
(36.4 -> 17.7 tok/s on a Tesla P40, 32-36 -> 17.7-18.6 on a GTX 1080 + 1070).
STRATA_PDL_RESTRICT is __restrict__ and kPdlPrefetch false again for
__CUDA_ARCH__ < 700 in the device pass. sm_70 and newer compile to the same
SASS as before (cuobjdump -sass of native_mmvq.cu and native_bf16.cu at sm_70,
86, 90 and 120 is identical); the host pass is unchanged.
Reported and bisected by paulhothersall and lineape.
2e4ddf6 (the #904 port) removed __restrict__ from the activation pointers of
native_quantize_q8_1_kernel, bf16_f32_mmvf_multi_kernel and
native_mmvq_multi_kernel on every CUDA architecture, and turned on the weight
prefetch before pdl_wait(). Both only matter for PDL (sm_90+). On sm_61 the
missing restrict loses the read-only data path (LDG.CI) and decode halves
(36.4 -> 17.7 tok/s on a Tesla P40, 32-36 -> 17.7-18.6 on a GTX 1080 + 1070).
STRATA_PDL_RESTRICT is __restrict__ and kPdlPrefetch false again for
__CUDA_ARCH__ < 700 in the device pass. sm_70 and newer compile to the same
SASS as before (cuobjdump -sass of native_mmvq.cu and native_bf16.cu at sm_70,
86, 90 and 120 is identical); the host pass is unchanged.
Reported and bisected by paulhothersall and lineape.
Found while running it against a live engine: the line said 0.35 as 48.7 on round 3 while the three draws were 57.0, 48.7 and 35.5, because it printed the running median. Reading a sweep's drift is the whole point of the line, so it has to be the draw. The medians and the ranges are still in the line below it.
Phase 1 measures each PCIe share once and takes the maximum, with the three prompts of that rate all at the one
setting. Neighbours cannot be ordered that way: on this box 0.75 through 0.95 come out at 72.7, 69.0, 71.0 and
71.9, which is 0.2% apart, while the same settings moved 8-28% between engine starts. A value that was worth a
real gain can lose its single sample and never be measured again, because phase 1's winner is the share phase 2
then sweeps spec_min_p at, and the confirm step only puts that one resulting pair against the default.
sweep() measures every value SWEEP_ROUNDS times inside the one engine start, one round visiting all of them and
every other round in reverse order, and the winners come from the medians. Drift inside a sweep then lands on
every value the same number of times, and one bad sample cannot decide. The same function runs phase 2, and the
confirm step uses the same round count instead of its own.
Cost: the sweep multiplies by SWEEP_ROUNDS, 3 by default. A rate is three prompts of 128 tokens, so on this box
(six values, 95-100 tok/s) phase 1 goes from about 40 s to about 120 s, on a run that already loads the model more
than once. A machine whose two leaders stay inside each other's noise can raise the constant.
Not in this change: phase 4 needs a restart per worker count and stays sequential, so a count is still taken from
one measurement each.
Tests: python -m unittest tools.test_calibrate (the stand-in engine is a pure function of the tune keys, so the
medians and the winners are the same as before); tools.test_setup_config and tools.test_setup_golden
PCIE_FRACS stops at 0.75, so setup --calibrate never measures the value that takes the CPU pool out of the decode
window completely. 1.0 is a separate operating point rather than 0.75 carried further: the share is applied as
m = (nmiss * pcie_num) >> 8, so at any value below 1.0 at least one miss still goes to the CPU pool, and only at
1.0 does every non-resident expert come over PCIe. A CPU expert sits on the layer's critical path while the GPU
waits for it, and on a slow CPU that is the whole difference.
On a Core Ultra 9 290HX Plus with an RTX 5090 Laptop on PCIe 5.0 x16, 0.55 gives 51.1 tok/s against 71.3 at 1.0,
with cpu_exp falling from 1.06 to 0.00 per layer-window and the window time falling by the 11.3 ms of PCIe waiting
plus the CPU's expert time. A sequential sweep puts 1.0 at 78.8 against 72.7 for 0.75 and 51.5 for 0.35.
0.9 is added beside 1.0 so a machine where 1.0 is too much gets an interior point. pick() keeps the default unless a
candidate beats it by MIN_GAIN, so this cannot make a calibration worse. The cost is one rate per value, about 8 s
here.
Tests: python -m unittest tools.test_calibrate; tools.test_setup_config and tools.test_setup_golden
With a second toolkit beside the distribution's one, CMake compiled with the chosen nvcc but took cudart and cuBLAS from /usr (Ubuntu 24.04, STRATA_NVCC=/opt/cuda-13.0/bin/nvcc: the engine needed libcudart.so.12 while built with the 13.0 headers). Pass CUDAToolkit_ROOT for the engine and the CUDA image encoder; the distribution's own /usr/bin/nvcc keeps CMake's search as before.
Two gfx906 build failures on v0.1.40.2. Both are one-line gates that know
about the wave32 HIP backend but not about STRATA_HIP_GFX906 -- the same
class as #808 and #1083. Neither is fixed on main (e8ca9afd03).
1) src/prefill/prefill.cpp:399 builds an event with
cudaEventDisableTiming | cudaEventBlockingSync, but the gfx906 compat
header include/strata/platform/hip_compat/strata_hip.h maps only
cudaEventDisableTiming and cudaEventDefault. The wave32 header
include/strata/hip_compat/cuda_runtime.h:39 has the mapping.
-> add #define cudaEventBlockingSync hipEventBlockingSync there.
A diff of the whole set of cuda* defines between the two headers lists
one other name the gfx906 header lacks, cudaDevAttrIntegrated, and that
one is already gated upstream (#if defined(__linux__) &&
!defined(STRATA_HIP_GFX906) in src/core/expert_cache.cpp:31), so it is
deliberately not touched here.
2) Link error: undefined symbol
void mul_mat_q_case<(ggml_type)14>(ggml_backend_cuda_context&, mmq_args const&, ihipStream_t*)
(14 = GGML_TYPE_Q6_K).
src/prefill/moe_mmq.cu:219 calls it under
#if defined(__HIPCC__) || defined(STRATA_Q6K_EXPERTS), so any HIP
compilation references it; mmq.cuh declares the case as an extern
template, so the definition must come from
template-instances/mmq-instance-q6_k.cu. The CUDA branch of
CMakeLists.txt adds that file only under STRATA_Q6K_EXPERTS.
STRATA_HIP_GFX906=ON forces STRATA_ENABLE_CUDA=ON, so gfx906 takes the
CUDA branch while its .cu files are compiled as HIP.
-> widen the gate: STRATA_Q6K_EXPERTS OR STRATA_HIP_GFX906.
This adds one MMQ instance to the static library. It does not enable
STRATA_Q6K_EXPERTS (the grouped Q6_K expert kernels that allocate extra
VRAM), and STRATA_HIP_GFX906 is only ever ON in the gfx906 build, so
CUDA and wave32 builds are untouched.
Verified on a single Instinct MI50 32 GB (gfx906, wave64), ROCm/HIP 7.14,
-DSTRATA_HIP_GFX906=ON -DCMAKE_HIP_ARCHITECTURES=gfx906
-DSTRATA_MMQ_KQUANTS=ON:
- the engine builds and serves UD-IQ4_XS (Qwen3.8-Flash-Next) end to end:
/v1/models, chat completions, tool calls, MTP drafts accepted;
- nm -C shows mul_mat_q_case<(ggml_type)14> in the binary (13 instances:
7 8 12 13 14 16 17 18 20 21 22 23 42);
- change 2 is load-bearing: removing mmq-instance-q6_k.cu.o from
libstrata_mmq.a and relinking with the same link command line fails with
exactly the undefined symbol above (the archive holds one U reference
and that one W definition), while the untouched archive relinks cleanly.
Two variants of the same sweep (code-explanation prompts and Russian prose) on
one PC and one server config, engine 0.1.40.3 built from source for sm_89.
Each folder carries the 0.1.40.2 run, the 0.1.40 baseline and an
identical-binary control run so the noise band is in the report.
Result: no measurable change from 0.1.40.2 on this machine.
Same laptop, model files, configuration and scripts as the 0.1.40.1
entry (2026-10-06-community-rtx-5090-laptop, #1263); only the Strata
version changed. Raw data for 0.1.40.3, medians for 0.1.40.2 in the
README. Decode and prompt throughput unchanged within run-to-run
variation, all needles found, coding check 10/10 on 0.1.40.3. A 0/10
coding check on 0.1.40.2 is traced to a greedy reasoning loop that
occurs in 0.1.40.1 as well (re-runs documented). Results only, no
engine changes.
Coder IQ1_M, one RTX 5080 16 GB, the repository's own Dockerfile and entrypoint
on a Windows host (Docker Desktop 4.84.0 / WSL2), two context limits on the same
loaded model: 131,072 and 65,536.
- Speed: three runs at each of 4,096 / 8,192 / 16,384 / 32,768 / 65,536 /
130,560 prompt tokens (128K limit) and at 4,096 / 8,192 / 16,384 / 32,768 /
65,024 (64K limit), greedy, reasoning off, 256-token output cap, zero reused
prompt tokens in all 33 runs.
- Correctness: tools/needle_bench.py 9 of 9 (32k/64k/128k) at the 128K limit and
6 of 6 at the 64K limit, plus a vision check with the encoder on the CPU.
- Provenance: all three model files SHA-256-verified against the repository's
published LFS oids at the pinned revision; MTP manifest and inventory included.
- Environment: Docker Desktop's default WSL2 memory cap makes setup.py read ~31 GB
of a 64 GB host and silently select the low-RAM mode; the report documents the
.wslconfig change used instead. Also documents the 8-token headroom a request
needs (prompt + max_tokens + 8 <= max_context) and the engine's own VRAM warning
at a 128K context on a 16 GB card.
Results-only; no engine changes.
Nine files following docs/COMMUNITY_BENCHMARKS.md. Engine v0.1.40.3
(tag d5ea713) built from source for sm_75, IQ3_S at the model native
262,144-token window, with --vision and --reasoning-budget-tokens 12000.
Three runs each at 4,096 / 32,768 / 128,000 prompt tokens: decode
70.9 / 77.3 / 71.1 tok/s, prompt 868.9 / 1548.9 / 1626.9 tok/s, medians
in the index and full ranges in the report. needle_bench 6/6 at 32k and
128k. Records that NVLink is present but unused by the engine, so a bridge
on consumer boards is not worth buying.
The flag acts only on --expert-cache-device1..3; a plain layer split got a no-op flag that the docs listed as a reason
for the pipeline and the async tier to switch off. Built on f-141-quick's #1352 pipeline-windows rule (same hunk).
MULTI_GPU.md: the two combine (only STRATA_PIPELINE_ADAPT_ASYNC=0 turns async off); adds the 2x3090 table.
Co-authored-by: adambenhassen <[email protected]>
The function name was everything between `<function=` and `>`, untrimmed. A model that writes a space after the `=`
(seen from Qwen3.8-Flash-Next IQ2_XS in Zed's agent panel) got:
- inside the thinking: the name " capture" matched no declared tool, so the #804 rescue kept the call as reasoning
text; the reply ended with finish_reason "stop" and no tool_calls, and the agent stopped mid-task;
- after the thinking: a call named " capture", which the client does not know, its arguments typed without the
tool's schema.
The name is now stripped where the call is parsed (parse_tool_call), where the #804 rescue checks it against the
declared tools, and where a finished call looks up its schema (_finish_call). A name with surrounding whitespace is
never meaningful, so no text becomes a call that was not one: the #804/#1058 rules still decide whether a call
inside the thinking is an act.
serve/test_tool_name_space.py: both places, streamed and not, fed in 3-character pieces; fails 4/4 before, passes
after. test_reasoning_rescue, test_reasoning_tools and test_frontend unchanged and passing.
NTFS serves unbuffered reads of a mapped file one at a time (WD SN580: 2 MiB reads at QD32
2.35 GB/s mapped vs 3.57 GB/s unmapped). With the file tier unbuffered (--resident-budget-gib
or STRATA_UNBUFFERED_LOAD=1) the mapped view is only a fallback after startup, so it is closed
(STRATA_KEEP_MAPPING=1 keeps it). The prompt path's stager claims runs of consecutive jobs and
reads them with one read_direct batch (STRATA_STAGER_BATCH, default 8 for file-backed blobs).
RTX 5070 Ti, 32 GB RAM, IQ3_XXS, 47.6K prompt, --resident-budget-gib 18 --prefill auto:32768:
1260 -> 1370 tok/s (batch 8) -> 1669 tok/s (view closed).
Ported from #833 onto the rewritten main. Two fixes on the way: unmapped_ is declared on every
platform (mapped_blob reads it outside the Windows block, so Linux did not compile), and
copy_blobs asks resident_blob, which follows the exchange rotation (STRATA_EXCHANGE_ROTATE).
The fast kernels build for CUDA; sm_70 defaults to them, other CUDA cards keep
the plain ones. gr_up_fast_kernel ends with gr_q8_tail like gr_up_multi_kernel,
so STRATA_QFUSE gets its q8_1 image from it too. In the engine (staged read)
only the up projection changes: hc-read0's up 1.08 -> 0.96 ms per window on a
V100-SXM2. fused_gr_bench bitwise, gr_multi_parity OK.
Read off mmvq_il_parity --bench on a V100-SXM2-32GB with the table's own rule
(every measured shape of a class at least 3% faster, else native_mmvq). At 3
columns: the Q5_K head 1094 -> 821 us, Q6_K 12288 rows 68.9 -> 47.9 us, Q4_K
10240 rows 43.9 -> 32.5 us. Bitwise native_mmvq's output (mmvq_il_parity OK).
The AMD grouped-expert layouts (STRATA_EXP_MODE) now build for CUDA too. sm_70
defaults to mode 8 - the codebook grid and the group's q8_1 activations in
shared memory, SwiGLU + q8_1 fused into the gate/up epilogue - every other CUDA
card keeps mode 0. On CUDA, IQ2_XXS gate/up joins the shared-memory kernels,
a format without one falls back to the CUDA layout (not R2), and a strided call
(the verify window's PCIe call) keeps the CUDA layout.
V100-SXM2, setup's IQ2_XS, a verify window's VRAM call (28 groups): IQ2_S
gate/up layers 180 -> 150 us at 3 tokens, IQ2_XXS 170 -> 134 us; bitwise mode
0's output on all 48 layers (native_expert_bench), native_grouped_parity 0
failures in modes 0/2/7/8.
A staged blob (the GGUF role slices assembled into one buffer) is handed to
ExpertCache::fill_slot's cudaMemcpyAsync as-is. From a pageable buffer the
driver stages every copy through an internal bounce buffer - an extra copy at
a fraction of the transfer rate, on the calling thread - so the admit path,
the profile prefill and the lent-slot refills all pay it per expert. Allocate
the pool's buffers with cudaHostAlloc instead, falling back to pageable for a
buffer when the driver refuses (pinned memory is scarce on the small-RAM
machines this path serves).
DETAILS.md's STRATA_PREFILL_CPU_SHARE section: chunks below 3,072 tokens are staged when it is on
(STRATA_PREFILL_CPU_SHARE_MAX=1024 keeps the old limit), the experts the CPU may take (the arena, the
page-locked copy, the mapped experts.bin's pages), the layer split, and a table measured on an RTX 5070 Ti
alone and an RTX 3060 + RTX 5070 Ti split (PCIe 3.0 x8, DDR4-3200, --mmap-experts): 512-token prompts
-49% / -41%, 3,000-token -22% / -24%; a recorded 100K-token coding conversation on the two cards 137.4 ->
130.6 s.
STRATA_PREFILL_CPU_SHARE handed the CPU pool its share of a chunk below 1,024 tokens only, only experts in
page-locked RAM, and only on one GPU. With it set:
- the chunks staged after their routing (the only ones the share applies to) go up to
STRATA_PREFILL_CPU_SHARE_MAX tokens (default 3072, at least 1024) instead of 1024; Prefill::arm_cpu_share
sets the limit before any loan is sized, and init sets it too;
- an expert whose blob the CPU reads from RAM without assembling it (the arena's unpinned part, the mapped
experts.bin's pages) can go to the CPU, not only a page-locked one; the blob pointers are taken when the
experts are chosen, and one without a stable blob stays on the GPU;
- on a layer split every stage has the pool (serve, no --batch), and one stage at a time takes it for a
chunk while the other reads its chunk on the GPU alone.
Unset, nothing changes.
auto balanced the CPU's time per expert against the GPU's per streamed expert, both measured with the share on, so
nothing compared a layer with and without it. On jase100k's RX 7900 GRE + Ryzen 7 5700X3D the host's time per
streamed expert rose 107 -> 200 us with auto (0.47): the streaming side saved nothing (0.53 x 1.87 = 0.99) and the
CPU's work came on top, -4.5%. Now every eligible layer is timed with CUDA events from before its routing sync to its
combine, per non-resident expert; adjacent layers alternate no share / the measured share until three ratios are in,
then auto shares while the median of the last five is below 1, with one layer in 29 in the other arm. Layers with a
one-time cost and the first layer are left out. STRATA_DBG_CPU_GATE=1 prints each measurement.
RTX 5090 + 9950X3D, IQ2_XS, 250 / 600 / 900 tokens against off: DMA streaming -9.8/-9.9/-10.4% -> -8.8/-9.1/-10.5%;
an emulated host-copied stream -18.0/-16.7/-14.4% -> -16.1/-15.2/-13.0%; one core + one worker (where the old auto
lost up to +11.5%) +1.4..+2.4% -> +2.0..+2.3%, the cost of the first measurements. Unset: byte-identical.
The parking path has two gates in series: make_room() balances the cache's own budget, then conversation_memory_admit() checks the host's free RAM against the snapshot plus --conversation-cache-min-free-mib. The second was final: a failure dropped the snapshot even though the cache held conversations whose RAM it could have, and the next request read its whole prompt again (270k tokens, 96 s, measured by the reporter in #1347).
Evict the least recently active parked conversations until the gate admits the snapshot, bounded by the slot count; entries that hold a pinned shared prefix are not evicted. The failure line says how many were evicted and how many remain.
Reported with the fix by alanthinker in #1347.
The 16 dense GEMM geometries a Qwen3.8-Flash-Next prompt logs on a Radeon 890M (gfx1150, Strix Point) at
T = 64 .. 16384, 144 rows, calibrated with tools/hip/tune_hipblaslt.cpp against AMD's TheRock ROCm 7.14.1 for
gfx1150 (hipBLASLt 1.4.1 cd957402). Two passes back to back; each row keeps the solution with the lowest median of
all six repetitions (a single repetition stalled now and then, and the tool's mean-of-three picks differed in 26 rows
between the passes).
Every row beats plain hipBLAS, 1.55-22.6x per GEMM. End to end on GSQ-RCO IQ3_XXS, three interleaved rounds: prompts
read 1.5-1.7x faster (3.6K tokens 124 -> 216 tok/s, 7K 130 -> 226 tok/s), decode unchanged; hip_prefill_hipblaslt_gemm
passes with it and a server run logged no fallbacks. Setup does not install for gfx1150, so the table is set by hand
(STRATA_HIPBLASLT_TUNING). Method and data: bench/results/2026-10-07-community-gfx1150.
A verify window's graph waits on the GPU for flags the host raises once
cudaGraphLaunch has returned. A blocking launch returns only when the
graph has finished, so with CUDA_LAUNCH_BLOCKING=1 the first window after
a prompt never ends and the watchdog stops the engine, whatever the prompt
length, GPU count or --pipeline-windows. The engine now says so at start,
and the watchdog's line names it. Nothing changes without the variable.
A pipelined group's window spans its slots up to its last busy one, and the engine keeps no pipelined slot as a
conversation cache, so the least-recently-used pick left leading holes ([_ B]) that cost a whole row each. pick_slot
now fills such a hole first, then follows the spreading order. The engine reports INFO batch_groups so the server
knows the groups really run pipelined. serve/test_slot_alloc.py covers the allocator.
Co-authored-by: blange48 <[email protected]>
The miss cost (190 ms per unit of routed mass, fitted on x16 links) is one
constant for every stage, so --layer-split auto cannot see an asymmetric
pair: on 2x 4090 D with CUDA1 on a physical x1 link (1.6 GB/s measured) it
gives that card 26 layers (K=22) and reads a 29K prompt at ~350 tok/s.
Probe each stage's link inside the search (probe_pcie_h2d_gbps; the
stage-setup probes are skipped when --pcie-frac is given, and the search
needs the reading either way) and scale that stage's miss term by
clamp(20 GB/s / bw, 1, 64). 20 GB/s is pcie_frac_for_gbps' x16 reference;
the lower clamp keeps a symmetric machine on the exact formula the sweep's
K was fitted with (every scale 1.0 -> the objective is unchanged).
With it, auto picks the fit-in-cache boundary on that machine (K=38: the
narrow card's cache holds all of its layers' pairs, zero streaming there)
and reads the same prompt at ~1410 tok/s, ~4x.
The 190 ms mixes the decode CPU-pool cost with the prompt link cost;
scaling the whole per-stage term is the first-order fix (splitting the
constant would need a fresh fit on both a narrow and a wide link).
STRATA_PF_FUSED=1 reaches only chunks of stream_all_min() tokens or more (1,024, measured on a discrete card); a
smaller read runs its experts through MMQ. On Strix Halo every expert is resident and MMQ is ~3x slower on a
~1,000-token read, the size of an agent's turn: 441 / 855-token reads 3.1 / 4.0 s -> 0.9 / 1.2 s, a 12-turn agent
replay 3.4 -> 2.1 s per turn (C2T8, --no-prefill-borrow). It changes bits (the fused path's output, which reads of
1,024+ tokens already get), so section 5: teacher-forced KL against today's path 0.016 on reads under 1,024, the same
as fused vs MMQ on longer reads (0.019); reads of 1,024+ unchanged (KL 0).
Setup installs the ROCm 7.14.0a20260608 wheels for gfx1151 since 0.1.40.2 (#1267); their hipBLASLt is 1.4.0
(100400), which neither gfx1151 table matched, so setup's engine read the prompt through plain hipBLAS.
tune_hipblaslt --workspace-mib 32 on the 100401 table's 90 shapes, on a Ryzen AI Max+ 395 (Radeon 8060S, gfx1151,
128 GB, Fedora 44, kernel 7.2.8) with those wheels: 3.1-14.1x faster than plain hipBLAS per shape (median 6.2x).
hip_prefill_hipblaslt_gemm and hip_prefill_hcd_exact_parity pass with it (fallbacks=0); HIP ctest 61 of 62
(ple_parity needs the Q2_0 fixture). setup picks it up with no code change.
Below N of N lendable slots the uncovered experts are read from the pack during a long prompt (reporter: 18% uncovered = -26% prompt speed on gfx1100). A WARNING line now says so, and docs/AMD_HIP.md documents the headroom knob and the levers from the report. No behaviour change otherwise.
The gated delta rule in the WY form flash-linear-attention uses, all FP32: gdn_chunk_prep_kernel forms k.k / q.k and
T = (I + A)^-1 for every chunk of a 2048-token super-block at once; gdn_chunk_scan_kernel (128 blocks, a key head x
16 value columns each) walks the chunks with its state slice in registers. Every exponent <= 0 (the gates are
negative). The scratch (25.6 MB) is allocated once per device: freed and reallocated per call it cost more host time
than the kernels saved on short prompts. sm_80+, >= 128 SMs, enough shared memory, chunks of >= 128 tokens; else
gdn_rec_kh_kernel. Unset: unchanged.
gdn_rec_parity: against gdn_rec_kh_kernel <= 1.13e-6 (T = 1..4099, 8192, 32768, a non-zero starting state); against
an FP64 recurrence <= 1.05e-6 (the sequential kernel <= 3.2e-7) - FP32 rounding; 199 restarts of 7 shapes bit for
bit (NaN-filled scratch between). RTX 5090: 1.81x the recurrence at 2K-32K; IQ2_XS: the recurrence phase of a 32K
prompt 520-532 -> 306-334 ms (-210 ms, -3.7%), 8K -2.1..-3.6%; first-token KL 0.0016-0.0152, 1.15-1.83x a harmless
control arm (STRATA_PROMPT_ATTN_V1), the same top token.
The MoE input and the routing weights go to host memory on a stream beside the primary's compute stream, so its
own experts start at once instead of behind the copies over its link (mixed_h and w are rewritten only after this
layer's combine, which waits for the peer). STRATA_PF_PEER_XSTREAM=0: the compute stream. With
STRATA_PF_PEER_RING=160, prompt tok/s: 100K 1,854 -> 1,918; 8K and 3.4K unchanged.
Each group's tokens and their row lists (in the experts' order) are built on the host from the rows' pairs and
uploaded once per layer; peer_gather_add is eddoursul/Strata's moe_gather_add with the weight looked up through
the row's pair, so the sums are bitwise those of the per-expert launches. STRATA_PF_PEER_GATHER=0: per expert.
Prompt tok/s against the per-expert launches: 3.4K 825 -> 832, 8K 1,468 -> 1,500, 100K 1,821 -> 1,858.
Co-Authored-By: Eddoursul <[email protected]>
The MoE input goes out as the FFN input's existing FP16 image (mixed_h) and the per-token sums come back as
FP16, saturated at +-65504 (eddoursul/Strata f8de703's FP16 input, 2acfac4's sums_to_f16): half the bytes over
the primary's link both ways. STRATA_PF_PEER_F16=0: FP32.
Peer stream 0.6, prompt tok/s: 3.4K 805 -> 831, 8K 1,284 -> 1,495, 100K 1,578 -> 1,855 (the share off: 1,712).
The 8K and 3.4K answers stay byte-identical to the share off.
Co-Authored-By: Eddoursul <[email protected]>
The design is eddoursul/Strata's second-GPU prompt path (f8de703, 2acfac4, e8ade66): the second GPU adds its
experts' rows into each token's weighted sum and the first GPU's combine adds that to its own experts' rows.
Over the host route the peer scatter-adds its rows into per-token sums (peer_scatter_add, after the fork's
moe_scatter_add: one launch per expert in a fixed order, so the sums are repeatable) and sends T x N floats
back instead of rows_peer x N (up to K = 10 times more); moe_combine_peer adds them to the primary's rows. The
2a row buffers are not allocated (3.4 GB of pinned RAM at 32K-token chunks). STRATA_PF_PEER_SUMS=0: rows.
Same setup, peer stream 0.6, prompt tok/s against the share off: 3.4K 332 -> 805, 8K 762 -> 1,284, 100K
1,712 -> 1,578. The 8K and 3.4K answers are byte-identical to the share off (greedy, 256 / 3,221 tokens).
Co-Authored-By: Eddoursul <[email protected]>
set_peer refused a peer that cannot access the primary directly, so on GPU pairs without P2P (most GeForce
pairs) the prompt path stayed on the primary GPU. Without P2P the share now takes the route the layer split's
helper already uses: the activations and the result rows through mapped pinned host buffers read by copy
kernels, with the compact group buffers (the layer-sized path copies over P2P only). STRATA_PF_PEER_HOST=0
restores the refusal.
RTX 3090 (PCIe 4.0 x4) + RTX 3080 Ti (x16), no P2P, UD-Q4_K_XL, --prefill auto:32768, peer stream 0.35 / 1.0,
prompt tok/s against the share off: 3.4K 332 -> 492 / 551, 8K 763 -> 907 / 776, 100K 1,713 -> 1,072 / 734.
The peer's rows coming back over the primary's x4 link cost more than they save on long prompts; the next
commit sends sums instead.
- tools/check_isa_guard.py disassembles the CPU kernels (objdump + nm on the Linux kernel library, capstone + the /MAP
file of a linked binary on Windows) and fails when a function holding AVX-512 / AVX-VNNI-class instructions is
not a kernel of a wide-ISA file: a weak or COMDAT copy (the linker may pick it for every caller), a std:: /
template instantiation, a probe or helper that every CPU runs, or code in another object. MSVC's own
`__isa_available` vectorizer guard is accepted. ctest isa_guard_check runs it; exit 77 (skip) when the tools
are missing.
- -DSTRATA_SDE=<sde> adds sde_{hsw,adl}_* tests: the CPU parity tests on an emulated Haswell (AVX2 only) and Alder
Lake (AVX-VNNI, no AVX-512); SDE stops at the first instruction the chip lacks.
- tools/test_check_isa_guard.py: unit tests of the policy.
expert.cpp and iq_avx512.cpp are compiled for AVX-512, iq_avx2.cpp and kq_avx2.cpp for AVX2, and a file compiled
that way may use the wide ISA anywhere in its code. Code that every CPU runs was still defined in them:
- cpu_features, CpuFeatures::reason, cpu_require_expert_support (the function that finds out whether the CPU has
AVX-512) and expert_set_oracle_q8_0 / expert_oracle_q8_0_enabled (the first call of every generate, on any CPU)
were in expert.cpp;
- iq512_supported, iq256_supported and kq256_supported (a comparison native_gu_rows asks before it has checked
what the CPU can run) were in the kernel files; iq512_supported was called on every AVX2-only CPU for every
gate/up call.
They now live in expert_layout.cpp and native_expert.cpp, which are built for the baseline. No kernel changes;
the compiled code of the moved functions is identical, so default output is byte-identical.
2026-10-07 20:23:53 +02:00
3967 changed files with 816960 additions and 12970 deletions
description:Tell us whether the all-cards split engine works on your AMD cards (it prints a line asking for this on cards Strata has not run it on)
title:"[Engine] AMD split engine: <your card> x <count>"
labels:["engine","amd-split-report"]
body:
- type:markdown
value:|
Strata runs the all-cards split engine by itself on Radeon AI PRO R9700 / RX 9070 (gfx1201). On other AMD cards (RX 9060, RX 7000, the 780M and similar) it only starts when you ask for it with `strata launch --split-engine on`, and it prints one line asking you to report whether it works. We cannot test those cards, so your report is how we learn.
Write only what you saw. A report that says "it did not start" is as useful as one that says "it works".
- type:input
id:card
attributes:
label:Card model and count
description:For example "2 x RX 7900 XTX 24 GB" or "3 x RX 9060 XT 16 GB"
validations:
required:true
- type:input
id:arch
attributes:
label:Architecture (the gfx name)
description:From the engine's line, or `rocminfo | grep gfx`. For example gfx1100.
validations:
required:true
- type:input
id:rocm
attributes:
label:ROCm / driver version
description:Strata's engine package brings its own ROCm libraries; the kernel driver is yours (`uname -r`, `amd-smi version`).
validations:
required:true
- type:input
id:model
attributes:
label:Model and quantization
description:The model Strata loaded, for example Qwen3.8-Flash-Next UD-Q4_K_XL or the IQ3_S set.
validations:
required:true
- type:textarea
id:selftest
attributes:
label:The self-test line
description:Paste the line from the engine's log that starts "The split engine's self-test" (passed or failed). The log is in the run folder shown at launch; `strata launch` also prints it.
render:text
validations:
required:true
- type:dropdown
id:result
attributes:
label:What happened
options:
- It started and the answers look right
- It started and the answers look wrong
- The self-test failed, so the normal layer split started
- It failed to start (not the self-test)
- It started and crashed or hung later
validations:
required:true
- type:input
id:speed
attributes:
label:Speed, tok/s (the split engine, then the layer split with `--split-engine off` if you tried both)
description:For example "decode 61 tok/s, layer split 48 tok/s, one user".
- type:textarea
id:more
attributes:
label:Anything else
description:Wrong answers (the prompt and what you got), crashes (the last lines of the engine log), or what you changed.
description:Share measurements from your machine, or report a problem with the benchmark tools
title:"[Benchmark] "
labels:["benchmark"]
body:
- type:markdown
value:|
The title must start with **[Benchmark]**. Keep it, and add the GPU and model after it.
To share a full report, follow the [community benchmark guide](https://github.com/Niko1221/Strata/blob/main/docs/COMMUNITY_BENCHMARKS.md) and open a results-only pull request. The standard protocol is a [draft](https://github.com/Niko1221/Strata/blob/main/docs/BENCHMARK_PROTOCOL.md); existing tools do not implement it yet. Include poor results and failed requests.
- type:textarea
id:what
attributes:
label:What did you measure, or what went wrong
description:Hardware (GPU and VRAM, CPU, RAM, OS), Strata version, model and quantization, the exact command, and the numbers. Paste the report or link the pull request.
description:Something in Strata doesn't work as documented
title:"[Bug]: "
title:"[Area] "
labels:["bug"]
body:
- type:markdown
value:|
Thanks for the report.
Thanks for the report. Please read [CONTRIBUTING.md](https://github.com/Niko1221/Strata/blob/main/CONTRIBUTING.md) first.
Please search the [open issues](https://github.com/Niko1221/Strata/issues?q=is%3Aissue+state%3Aopen) first — this may already be tracked.
Search the [open and closed issues](https://github.com/Niko1221/Strata/issues?q=is%3Aissue) first: this may already be tracked, and new evidence belongs in that thread.
Write only what you saw. Do not guess at steps, errors, hardware or severity.
**The title must start with its area:** [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other]. Choose the same one below.
- type:dropdown
id:area
attributes:
label:Area
description:Start the issue title with the same area in square brackets, for example "[Engine] decode stops after 2 minutes". Pick Other if none fits.
options:
- "[Engine] (speed, kernels, models, the server and API)"
- "[GUI] (the desktop app)"
- "[TUI] (the Terminal: strata launch, connect, monitor)"
- "[Benchmark] (measurements and reports)"
- "[Docs] (documentation)"
- "[Other] (anything else, including setup and installation)"
validations:
required:true
- type:dropdown
id:how
attributes:
label:Desktop app, Terminal or source?
options:
- Desktop app (Windows, Linux or Mac)
- Terminal (the `strata` command)
- Source (START-HERE.bat, setup.sh or a build)
- Docker
- Other
validations:
required:true
- type:textarea
id:what
attributes:
label:What happened
description:What you did, what you got, and what you expected instead. Paste the exact error text if there is one. If there's a set of steps or commands that reproduces it, put them here too.
description:What you did, what you got, and what you expected instead. Paste the exact error text if there is one.
validations:
required:true
- type:textarea
id:reproduce
attributes:
label:Smallest steps or command that reproduce it
description:The shortest list of steps that shows the problem. If you cannot make it happen again, say so here.
validations:
required:false
- type:dropdown
id:reproduction-status
attributes:
label:Reproduction status
options:
- Reproduced locally
- Unconfirmed static analysis (I read code or ran a scanner; I did not see it fail)
validations:
required:true
- type:input
id:version
attributes:
label:Strata version
description:"In the app: Settings > About. In the Terminal: `strata --version`. From source: the About tab of the web page, or `engine` in `GET /v1/status`, or the git commit."
placeholder:"Strata 0.2.12 · engine 0.1.58"
validations:
required:true
- type:input
id:hardware
attributes:
label:Strata version, GPU, OS
description:The engine version is on the About tab, or in `GET /v1/status` under `engine`.
placeholder:"0.1.31 · RTX 5070 12GB · Windows 11"
label:Graphics card, memory and system
description:The card and its VRAM, the RAM of the PC, and the operating system.
placeholder:"RTX 5070 12GB · 64 GB RAM · Windows 11"
validations:
required:true
- type:input
id:model
attributes:
label:Model, size and settings
description:The model and size you started, the context, and whether pictures were on. Write what is relevant.
description:Drag in `strata-<model>.log` from your Strata folder. Drag and drop works directly into this box.
label:Logs
description:|
The app: Settings > Engine > Diagnostics > View logs > Save logs, then drag the file in. The Terminal: paste the output of `strata doctor`. From source: drag in `strata-<model>.log` from your Strata folder.
Remove API keys, passwords and private text first.
validations:
required:false
- type:markdown
value:|
A couple of extras that help a lot:
- Check the [troubleshooting table](https://github.com/Niko1221/Strata/blob/main/docs/DETAILS.md#troubleshooting) first, many reports are already solved there.
- Check the [troubleshooting page](https://github.com/Niko1221/Strata/blob/main/docs/TROUBLESHOOTING.md) first, many reports are already solved there.
description:Suggest something new or an improvement
title:"[Feature]: "
title:"[Area] "
labels:["enhancement"]
body:
- type:markdown
value:|
Explain what you want in plain language.
Explain what you want in plain language. For a large feature, a change to a default or a refactor, please agree the scope here first. Read [CONTRIBUTING.md](https://github.com/Niko1221/Strata/blob/main/CONTRIBUTING.md).
**The title must start with its area:** [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other]. Choose the same one below.
- type:dropdown
id:area
attributes:
label:Area
description:Start the issue title with the same area in square brackets, for example "[Engine] decode stops after 2 minutes". Pick Other if none fits.
options:
- "[Engine] (speed, kernels, models, the server and API)"
- "[GUI] (the desktop app)"
- "[TUI] (the Terminal: strata launch, connect, monitor)"
- "[Benchmark] (measurements and reports)"
- "[Docs] (documentation)"
- "[Other] (anything else, including setup and installation)"
validations:
required:true
- type:textarea
id:idea
attributes:
label:What do you want to do?
description:Describe the goal rather than the implementation. If there's a workaround you use today, mention it.
description:Describe the goal rather than the implementation. Say who needs it and why.
validations:
required:true
- type:textarea
id:workaround
attributes:
label:What do you do today instead?
description:The workaround you use now, if there is one.
validations:
required:false
- type:markdown
value:|
All of this counts: model and architecture support, the server and API, the web app, multi-GPU and layer split, images and vision, MCP servers, setup and installation, and performance.
All of this counts: model and architecture support, the server and API, the app and the Terminal, multi-GPU and layer split, images and vision, MCP servers, setup and installation, and performance.
description:Ask how to use Strata, or share a result
title:"Question: "
title:"[Area] "
labels:["question"]
body:
- type:markdown
value:|
Not sure how to do something, or want to share numbers from your machine? Ask here.
Not sure how to do something, or want to share numbers from your machine? Ask here. To share a full benchmark, see [CONTRIBUTING.md](https://github.com/Niko1221/Strata/blob/main/CONTRIBUTING.md#share-a-benchmark).
**The title must start with its area:** [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other]. Choose the same one below.
- type:dropdown
id:area
attributes:
label:Area
description:Start the issue title with the same area in square brackets, for example "[Engine] decode stops after 2 minutes". Pick Other if none fits.
options:
- "[Engine] (speed, kernels, models, the server and API)"
- "[GUI] (the desktop app)"
- "[TUI] (the Terminal: strata launch, connect, monitor)"
- "[Benchmark] (measurements and reports)"
- "[Docs] (documentation)"
- "[Other] (anything else, including setup and installation)"
validations:
required:true
- type:textarea
id:question
attributes:
@@ -13,10 +29,17 @@ body:
description:What you're trying to do. Include your version, GPU, and OS if it affects the answer.
validations:
required:true
- type:textarea
id:tried
attributes:
label:What you tried
description:What you already did and where you got stuck.
validations:
required:false
- type:textarea
id:numbers
attributes:
label:Results (optional)
description:Only if you ran a benchmark or measured speed. Model, quant, context, and the tok/s you get.
<!-- If Applicable, reference the GitHub issue -->
Issue: Resolves #
**The title must start with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other], for example "[Engine] fix decode stop on long prompts" or "[GUI] keep the model list after a tab switch".**
## Summary
<!-- Quick Summary of changes -->
<!--
Read CONTRIBUTING.md first. Lead with the problem and the result, not a file-by-file list.
Placeholders below are not test results: replace them with what you observed, or write "none".
-->
## What changed
<!-- Specifics on files changed, and what changes were made there -->
## Problem and result
<!-- What triggered this change. How it behaves before and after. -->
## Extra Notes
<!-- Any extra notes, delete if there are none -->
## Related discussion
<!-- The issue URL and the maintainer's agreement on scope, when the change needs one (features, refactors, defaults). Write "none" if not applicable. -->
## Approach
<!-- Why this fix. Compatibility effects. Keep unrelated cleanup out of the PR. -->
## Validation
<!-- Exact commands, the revision and the environment tested, what you saw. Keep the three lists apart. -->
missing.push('Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other], then a short description. Example: "[Engine] decode stops after 2 minutes".');
missing.push('Re-test on Strata 0.2.0-beta and put that version in "'+form.needsCurrentVersion+'". If it no longer happens, say so in a comment instead.');
missing.push('Describe it in the issue body: what happened, what you expected, and the exact error if there is one. The issue forms (New issue > Bug report) list what to include.');
}
if(!lenient){
if(!ANY_VERSION_RE.test(text))missing.push('Add the Strata version (Settings > About, or `strata --version`).');
elseif(needCurrent&&!CURRENT_VERSION_RE.test(text))missing.push('Re-test on Strata 0.2.0-beta and name that version in the issue. If it no longer happens, say so in a comment instead.');
if(!HARDWARE_RE.test(text))missing.push('Add your graphics card with its VRAM, your RAM and your operating system.');
missing.push('Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other], then a short description. Example: "[Engine] fix decode stop on long prompts".');
if(!problem||realText(problem.value).length<20)missing.push('Fill in "Problem and result": what triggered the change, and how it behaves before and after.');
missing.push('Fill in "Validation": the exact commands, the revision and environment tested, what you saw. Keep Passed, Failed and Not run apart.');
}else{
valText=validation.value;
}
}else{
constplain=realText(text);
if(plain.length<150){
missing.push('Describe the pull request: the problem and the result. The pull request template shows the parts (Problem and result, Related discussion, Approach, Validation, Limits).');
}
if(!/(problem|result|before|after|fix|because|bug)/i.test(plain))missing.push('Add "Problem and result": what triggered the change, and how it behaves before and after.');
if(!/(validat|tested|passed|ran |built|build|not run)/i.test(plain))missing.push('Add "Validation": the exact commands, the revision and environment tested, and what you saw.');
'Thanks for the '+(isPull?'pull request':'issue')+'. To keep the tracker readable, every item needs an area prefix in the title and the parts of the '+(isPull?'pull request template':'issue form')+' filled in. This one is missing:',
'',
checklist(missing),
'',
'Edit the '+(isPull?'pull request':'issue')+' (title and description) and this note goes away on its own. If it is still missing parts in 5 days it will be closed, which is not a rejection: fix it and comment `/reopen`.',
'',
'Details: [CONTRIBUTING.md]('+CONTRIB+')',
].join('\n');
}
functionreopenMissingComment(isPull,missing){
return[
'Not reopened yet. Still missing:',
'',
checklist(missing),
'',
'Edit the '+(isPull?'pull request':'issue')+' and comment `/reopen` again. Details: [CONTRIBUTING.md]('+CONTRIB+')',
].join('\n');
}
functionreopenOkComment(isPull){
return'Reopened, thank you. A maintainer will pick it up from here.'+(isPull?' If it conflicts with main, rebase it first; that speeds up review.':'');
}
functionstaleComment(isPull){
return[
'Closing this '+(isPull?'pull request':'issue')+' because it still did not follow the format after 5 days. That is housekeeping, not a rejection.',
'',
'To bring it back: fix the title (area prefix) and the missing parts, then comment `/reopen`. A check runs and reopens it, or tells you what is still missing. See [CONTRIBUTING.md]('+CONTRIB+').',
assert.ok(!c.checkIssue({title:'[GUI] pin models',body:'### Area\n\n[GUI]\n\n### What do you want to do?\n\n_No response_'}).ok);
constq='### Area\n\n[Other]\n\n### Your question\n\nHow do I move the model folder to another drive?';
assert.ok(c.checkIssue({title:'[Other] model folder',body:q}).ok);
});
test('free-form issue',()=>{
constgood='Strata 0.2.0-beta on an RTX 3060 12 GB with 32 GB RAM on Linux: the server hangs when I send two requests at once. Expected both to finish.';
assert.ok(c.checkIssue({title:'[Engine] hang on two requests',body:good,mode:'reopen'}).ok);
assert.ok(c.checkIssue({title:'[Other] idea',body:'It would help if the app remembered my last model between runs, today I pick it again.',labels:['enhancement'],mode:'reopen'}).ok);
});
constprBody=(over={})=>{
constp={
problem:'Decode stopped on prompts above 8k because the stop check read the wrong length. Now it continues.',
validation:'**Passed:** `ctest -R decode`, commit abcdef1, RTX 3060, 12 of 12.\n\n**Failed:** none observed\n\n**Not run:** HIP build',
...over,
};
return`## Problem and result\n<!-- hint -->\n${p.problem}\n\n## Related discussion\nnone\n\n## Approach\nFix the length.\n\n## Validation\n${p.validation}\n\n## Limits\nnone`;
constbody=prBody({validation:'**Passed:** I tried it and it worked for me on my machine, looked fine.'});
assert.ok(c.checkPull({title:'[Engine] x y',body,mode:'intake'}).ok);
assert.ok(!c.checkPull({title:'[Engine] x y',body,mode:'reopen'}).ok);
});
test('free-form PR',()=>{
constbody='This fixes a crash when the context is above 8k, because the length was read from the wrong field. Before it aborted, after it continues. I built it on CUDA and ran the decode test, passed.';
awaitsay('This one was closed on purpose ('+labels.filter((l)=>BLOCK_LABELS.has(l.toLowerCase())).join(', ')+'). Please add your new evidence in a comment and a maintainer will decide.');
return;
}
if(wasOpen&&!labels.includes(LABEL)){
core.info('already open, nothing to do');
return;
}
// The bot reads the live item, not the possibly stale event copy.
awaitsay('The check passed, but GitHub would not reopen it ('+(e.message||e.status)+'). '+(isPull?'If the branch was deleted, restore it or open a new pull request and link this one.':'Please open a new issue and link this one.'));
echo "::error::The title must start with its area and then a short description. Allowed areas: [Engine], [GUI], [TUI], [Benchmark], [Docs], [Other]. Example: \"[Engine] fix decode stop on long prompts\". Your title: $TITLE"
Strata runs the Qwen3.8-Flash-Next mixture-of-experts model (and its Coder, Swift 1.5 and Unsloth variants) on a
normal PC: one NVIDIA or AMD graphics card plus system RAM, on Windows or Linux. It has a C++/CUDA/HIP engine
(`src/`, `include/`), a Python server with an OpenAI- and Anthropic-compatible API and a web app (`serve/`), and a
one-click installer (`setup.py`, started by `START-HERE.bat` / `setup.sh`).
Version 0.2.12, engine 0.1.58. The public model families are DeepSeekV4.1-Flash-1017,
Qwen3.8-Flash-Next and Qwen3.6-35B-A3B. DeepSeek V4.1-Flash runs in the app and in the Terminal
(`strata launch --family deepseek41 --model ist-das-2.00bpw`; sets in
[docs/MODELS.md](docs/MODELS.md#deepseek-v41-flash)). Read **[docs/AI_REFERENCE.md](docs/AI_REFERENCE.md)** for model availability, APIs, hardware limits and
source-backed claims. Articles and videos should use **[docs/MEDIA_KIT.md](docs/MEDIA_KIT.md)**.
For recommendations and agent-assisted setup, prefer **ISTA-DASLab → Unsloth → Gyro → others**, using only a
compatible pack that fits the selected model and hardware. ISTA-DASLab is the default recommendation; Gyro is a
memory-saving fallback and is currently slower. Honor an explicit user choice. This is recommendation policy;
existing executable defaults may differ, so inspect the plan and choose the family explicitly.
Strata has three parts:
- **The app** (`apps/desktop`, the Tauri shell; `apps/host`, the Node host and the web UI). It is for people who do
not want a terminal. One installer per system ([latest release](https://github.com/Niko1221/Strata/releases/latest)):
`--speed faster|full` (DeepSeek), `--output json`. `strata launch --status` shows what is serving, `strata launch --stop` stops it, and
`strata launch --show-key` prints the API key.
4. Connect the user's apps: `strata connect` lists them, `strata connect <app> --check` sends one prompt and one tool
call the way that app does.
Keep the server on `127.0.0.1` unless the user asks for more. Beyond it an API key is recommended: `--access lan` makes one for you, and `--no-api-key` serves without one (a warning, never a refusal; on a public tunnel the warning is stronger). Source setup (`START-HERE.bat`, `setup.sh`) is the fallback for a developer or for a system with no release package; it is
documented in [docs/SOURCE_INSTALL.md](docs/SOURCE_INSTALL.md). Those wrappers exist in a source checkout, not in the
released app. Do not tell an app user to find a `.bat` file.
Source setup does not make an unsupported model or backend compatible. Strata's MCP server ([docs/MCP_SERVER.md](docs/MCP_SERVER.md)) offers the same steps as tools.
## Working on the code
- How the engine works, every measured number, the API and all settings: [docs/DETAILS.md](docs/DETAILS.md) and
the [paper](docs/paper/Strata-Paper.pdf).
- AMD (HIP) build and validation: [docs/AMD_HIP.md](docs/AMD_HIP.md); multi-GPU: [docs/MULTI_GPU.md](docs/MULTI_GPU.md).
- Setup's own tests run without a GPU or downloads: `python tools/test_setup_<name>.py` (for example
rem Benchmark this Strata install the way it is used and write a report you can share (docs/COMMUNITY_BENCHMARKS.md).
rem It starts the model with your settings (or uses the one already running), takes about 10 minutes and changes nothing.
rem Options: --clients 4 (several requests at once), --long (a 32K prompt), --compare <older results folder>.
rem This is the quick speed check. The standard benchmark (docs/BENCHMARK_PROTOCOL.md), whose results other people can compare with, is `strata bench run`.
setlocal
title Strata - benchmark
cd /d "%~dp0"
ifnotexist".venv\Scripts\python.exe"(
echo Strata is not set up here yet: run START-HERE.bat first.
pause
exit /b 1
)
echo Quick speed check. For results other people can compare with (the standard benchmark, a few minutes), run: .venv\Scripts\python.exe strata.py bench run
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Ein Voxel-Pagodengarten, den das Modell in Strata geschrieben hat, läuft im Browser"></a><br>
<sub>Ein Voxel-Pagodengarten aus einem einzigen Prompt, auf einer RTX 5070 mit Strata (IQ3_S, 128K Kontext) ·
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">ganzes Video (49 s)</a></sub></p>
<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="Strata über GitHub Sponsors unterstützen"></a>
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Strata auf Buy Me a Coffee unterstützen"></a>
</p>
Strata lässt **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** auf einem normalen PC laufen.
Das ist ein großes, kluges KI-Modell, das sonst meist einen Server braucht. Es chattet, schreibt Code, versteht
Bilder und arbeitet mit deinen Apps und Coding-Agents zusammen. Nichts verlässt deinen PC.
NVIDIA: Q2_0 mit Engine 0.1.36, die anderen Zeilen mit 0.1.26 (4K Antworten, 32K Prompts). Die vollständigen
Tabellen stehen in [DETAILS.md](docs/DETAILS.md#speed-measured). Eine Karte mit mehr VRAM ist schneller: Eine
RTX 3090 (24 GB) sollte etwa 100-140 Tokens pro Sekunde schreiben. Lange Chats und andere Karten:
[Tempo jedes Modells](docs/MODELS.md#how-fast-is-each-size), [Ergebnisse aus der Community](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata ist kostenlos. Wenn es auf deinem PC gut läuft, hält ein Kaffee die Arbeit daran am Laufen.</sub></p>
## Was du brauchst
| | |
| Anbieter | Auswahl |
| --- | --- |
| **Grafikkarte** | **NVIDIA** GeForce RTX 20, 30, 40 oder 50 Serie, oder **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 oder RX 6800 / 6900 Serie. Sie braucht **12 GB VRAM oder mehr**. |
| **RAM** | 32 GB oder mehr. Dein RAM entscheidet, [welches Modell](#welches-modell-soll-ich-nehmen) passt. Mit 64 GB laufen alle Größen. |
| **Festplatte** | Etwa 80 GB frei. Nimm wenn möglich eine SSD: Der erste Start geht dann viel schneller. |
| **System** | Windows 10 / 11 oder Linux und ein aktueller Grafiktreiber von NVIDIA oder AMD. |
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
In der **Desktop-App** und im **Terminal**. Was nicht in den Arbeitsspeicher passt, wird von deiner SSD gelesen. Darum brauchst du viel freien Speicherplatz.
| Set (ISTA-DAS) | Download | Freier Speicherplatz nötig |
| --- | ---: | ---: |
| 1.75 bpw | 239 GB | etwa 245 GB |
| 2.00 bpw | 257 GB | etwa 265 GB |
| 2.25 bpw | 274 GB | etwa 280 GB |
Läuft auf NVIDIA RTX 30, 40 und 50 (Windows und Linux) und auf AMD Radeon AI PRO R9700 (Linux). Das Setup lädt die Dateien und die DeepSeek-Engine herunter und startet dann das Modell.
Er prüft deine Grafikkarte, deinen RAM und deine Festplatte und wählt das passende Modell. Dann installiert und
startet er es und sagt dir, wie du deine Apps verbindest. KI-Tools können Strata auch über seinen
[MCP-Server](docs/MCP_SERVER.md) installieren, starten und stoppen.
Gemessen mit dem 2.00-bpw-Set auf Strata 0.2.5. „Ausgabe“ ist das Tempo der Antwort, „Prompt“ zeigt, wie schnell Strata deine Eingabe liest.
### Oder mach es selbst
| Hardware | Ausgabe | Prompt |
| --- | ---: | ---: |
| RTX 5070 12 GB, 64 GB RAM | 4 bis 6 Token/s | etwa 110 Token/s (4K) |
| RTX 3060 12 GB, 46 GB RAM | 9 bis 11 Token/s | etwa 155 Token/s (4K) |
| 1x Radeon AI PRO R9700, 251 GB RAM | 21 bis 25 Token/s | 320 bis 360 Token/s (8K) |
| 4x Radeon AI PRO R9700 | 74 bis 77 Token/s | 400 bis 450 Token/s (8K) |
[Lade Strata herunter](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) und entpacke es (oder nutze
`git clone`). **Windows:** Doppelklick auf **`START-HERE.bat`**. **Linux:** Führe im Strata-Ordner **`./setup.sh`** aus.
[Speicher, Festplatte und Details](docs/DEEPSEEK41.md) · [Versionshinweise](https://github.com/Niko1221/Strata/releases)
Die Schritte sind für NVIDIA und AMD gleich. Der Installer erkennt deine Karte und richtet die passende Engine
dafür ein. Er stellt dir ein paar Fragen:
</details>
- welches Modell und welche Größe,
- wie viel Kontext (wie viel Text sich das Modell merkt),
- ob es Bilder lesen soll.
<a id="the-other-sizes"></a>
Drück jedes Mal Enter, um die empfohlene Antwort zu nehmen. Dann lädt er das Modell herunter (etwa 70 GB) und
startet es. Bricht der Download ab, starte ihn einfach neu: Er macht dort weiter, wo er aufgehört hat. Dein Browser
öffnet die Strata-App unter `http://127.0.0.1:8080`.
Unsicher, was passt? [Wähle ein Modell für deinen Computer](docs/MODELS.md#pick-by-memory).
> **Während das Modell startet, kann dein PC 1-3 Minuten langsam sein oder nicht reagieren** (beim ersten Mal am
> längsten). Strata lädt 35-55 GB in deinen RAM und reserviert einen Teil davon für die Grafikkarte. Das ist normal.
> Warte und schließ das Fenster nicht. Im Fenster siehst du, was Strata gerade tut.
## Installation
**Beim nächsten Mal** startest du wieder `START-HERE.bat` (oder `./setup.sh`). Es startet sofort und lädt nichts
doppelt herunter. Schließ das Fenster, um das Modell zu stoppen. `UPDATE.bat` (`./update.sh`) aktualisiert Strata,
ohne es zu starten. Updates, Docker, mehrere Karten, wo die Dateien landen und alle Optionen:
[docs/INSTALL.md](docs/INSTALL.md).
<a id="option-a-the-strata-app"></a>
## Welches Modell soll ich nehmen?
### Desktop-App
Der Installer empfiehlt eins passend zu deinem RAM. Dasselbe Modell gibt es in mehreren Größen, mehr oder weniger
stark komprimiert. Kleinere Größen sind schneller. Größere sind etwas klüger.
**[Strata für Windows, Linux oder Mac herunterladen →](https://github.com/Niko1221/Strata/releases/latest)**
| Dein RAM | Nimm | Warum |
| --- | --- | --- |
| **32 GB** | **Coder** | passt in 32 GB und ist für Code gemacht (mit einer 24-GB-Karte laufen auch Q2_0 und IQ2_XS) |
| **48 GB** | **IQ2_XS** (oder Q2_0, das schnellste) | die größeren Größen passen nicht |
| **64 GB** | **IQ2_XS** (empfohlen) oder IQ3_XXS / IQ3_S | alle Größen passen; IQ3_S ist das beste und das langsamste |
| **96 GB oder mehr** | **IQ3_S** oder Unsloths UD-IQ4_XS (~4 Bit) | Platz für die größten Größen, auch wenn alles andere offen ist |
Installiere die App, wähle ein Modell und leg mit dem Chat los. Es gibt einen Installer pro System. Beim ersten Start lädt die App die Engine für deine Grafikkarte herunter. [Hilfe zum Installer](docs/INSTALL.md#the-app).
- **[Coder](docs/MODELS.md#coder):** eine Version für Code, bei der die Hälfte der Experten entfernt wurde. Sie
erreicht 91 % des SWE-bench-Verified-Werts des vollen Modells (von ihren Autoren gemessen) und passt in 32 GB RAM.
Außerhalb von Code ist sie schwächer, auch bei chinesischem und anderem CJK-Text (#438). Nimm dafür Q2_0, IQ2_XS
oder IQ3_S, die alle Experten behalten.
- **[Swift 1.5](docs/MODELS.md#swift-15):** ein Fine-Tune, das viel kürzer nachdenkt, bevor es antwortet. Du
bekommst die Antwort früher, bei etwa gleicher Qualität.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** die ~4-Bit-Version von Unsloth, in der Qualität
zwischen IQ3_S und UD-Q4_K_XL. 94 GB Download. Mit weniger als ~80 GB RAM liest Strata einen Teil davon während
der Antwort von der SSD, dort ist es also langsamer (eine NVMe-SSD hilft).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (experimentell): am nächsten am vollen
Modell. Aber Strata liest das meiste davon während der Antwort von der SSD, deshalb schreibt es auf einem PC mit
64 GB nur 7-8.5 Tokens/s.
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** Das richtest du von Hand
Modelle wie dieses laufen normalerweise auf Servern mit Hunderten Gigabyte Grafikspeicher. Deine Grafikkarte hat
12-24 GB. Strata bringt das Modell trotzdem unter, indem es **die Arbeit auf deinen ganzen PC verteilt**. Stell dir
eine Küche vor: Was du ständig brauchst, bleibt auf der Arbeitsfläche, der Rest wartet in der Vorratskammer.
Öffne unter Windows danach ein neues Terminal. Dann:
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Die 24,576 Experten des Modells: die meistgenutzten auf der Grafikkarte, alle im RAM, eine Nachschlagetabelle auf der SSD"></p>
```sh
strata launch
```
- **Das Modell ist ein Team aus 24,576 kleinen Spezialisten („Experten“).** Jedes Wort braucht nur 10 davon.
- **Deine Grafikkarte** hält die paar tausend Experten, die am häufigsten gebraucht werden. **Dein RAM** hält alle,
und **dein Prozessor** rechnet gleichzeitig am Rest. **Deine SSD** hält eine große Nachschlagetabelle.
Folge den Anweisungen, um ein Modell herunterzuladen und zu starten. [Terminal-Anleitung](docs/TERMINAL_LAUNCH.md).
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Ein kleiner Helfer rät die nächsten Wörter; das große Modell prüft alle auf einmal und behält die richtigen"></p>
</details>
- **Raten, dann prüfen:** Ein kleiner Helfer rät die nächsten paar Wörter. Das große Modell prüft sie alle auf
einmal. Du bekommst dieselbe Antwort, nur 1.6-1.8x schneller.
- **Lange Texte werden in großen Stücken gelesen** (bis zu 8,192 Tokens auf einmal), mit über 1,000 Tokens pro
Sekunde.
<p align="center">
<img src="docs/media/terminal-monitor.png" width="900" alt="Strata-Terminal-App mit Modellaktivität, Tempo, Speicher und letzten Anfragen">
<br><sub>Behalte Modell und Hardware im Blick, während es läuft.</sub>
</p>
Die längere Erklärung: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Jeder Teil und seine Zahlen:
[die Details](docs/DETAILS.md#how-it-works) und das [Paper](docs/paper/Strata-Paper.pdf).
<a id="build-from-source"></a>
## Danksagung und Lizenz
[Aus dem Quellcode bauen](docs/SOURCE_INSTALL.md) · [Docker](docs/DOCKER.md) · [Vollständige Installationsanleitung](docs/INSTALL.md)
Das Modell ist [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) vom Qwen-Team. Komprimiert
wurde es von [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
und Unsloth. Strata nutzt Teile von [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Alle Danksagungen:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata ist Open Source unter der [MIT-Lizenz](LICENSE). Ein
paar Teile und alle Modelle haben eigene Lizenzen ([welche](docs/HOW_IT_WORKS.md#license)).
| **Apple Silicon · M1 oder neuer** | [Mac-Setup](docs/MAC.md) |
| **AMD Ryzen AI Max · Strix Halo** | [Strix-Halo-Setup](docs/STRIX_HALO.md) |
<a id="using-it"></a>
## Tools
**Claude Code · Codex CLI · OpenCode · oh-my-pi · pi.dev · Hermes Agent · OpenClaw**, und weitere über die OpenAI- und Anthropic-kompatiblen APIs.
Führe **`strata connect`** aus, um die Einrichtung für dein Tool zu sehen. [Verbindungsanleitung](docs/TERMINAL_LAUNCH.md#connect-your-apps-strata-connect).
Strata ist kostenlos und Open Source. Wenn es dir nützt, kannst du die Entwicklung unterstützen:
**Hilf mit, lokale KI für alle offen zu halten.** Ein GitHub-Sponsoring oder ein Kaffee hilft, Entwicklung und Tests auf mehr Hardware zu finanzieren.
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: Strata unterstützen" height="48"></a>
### Treue Unterstützer
Danke an alle, die Strata unterstützen.
<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Un jardín con una pagoda de vóxeles que escribió el modelo de Strata, funcionando en el navegador"></a><br>
<sub>Un jardín con una pagoda de vóxeles, hecho con un solo prompt en una RTX 5070 con Strata (IQ3_S, contexto de 128K) ·
<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="Patrocina Strata en GitHub Sponsors"></a>
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Apoya Strata en Buy Me a Coffee"></a>
</p>
Strata ejecuta **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** en un PC normal. Es un
modelo de IA grande e inteligente que normalmente necesita un servidor. Conversa, escribe código, entiende imágenes
y funciona con tus aplicaciones y agentes de programación. Nada sale de tu PC.
NVIDIA: Q2_0 con el motor 0.1.36, las demás filas con 0.1.26 (respuestas de 4K, prompts de 32K). Las tablas
completas están en [DETAILS.md](docs/DETAILS.md#speed-measured). Una tarjeta con más VRAM es más rápida: una
RTX 3090 (24 GB) debería escribir unos 100-140 tokens por segundo. Chats largos y otras tarjetas:
[velocidad de cada modelo](docs/MODELS.md#how-fast-is-each-size), [resultados de la comunidad](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata es gratis. Si funciona bien en tu PC, un café ayuda a seguir trabajando en él.</sub></p>
## Qué necesitas
| | |
| Proveedor | Opciones |
| --- | --- |
| **Tarjeta gráfica** | **NVIDIA** GeForce serie RTX 20, 30, 40 o 50, o **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 o serie RX 6800 / 6900. Necesita **12 GB de VRAM o más**. |
| **RAM** | 32 GB o más. Tu RAM decide [qué modelo](#qué-modelo-elijo) cabe. Con 64 GB funcionan todos los tamaños. |
| **Disco** | Unos 80 GB libres. Usa un SSD si puedes: el primer arranque es mucho más rápido. |
| **Sistema** | Windows 10 / 11 o Linux, y un controlador gráfico actual de NVIDIA o AMD. |
En la **aplicación de escritorio** y en el **Terminal**. Lo que no cabe en la memoria se lee desde tu SSD, así que hace falta mucho espacio libre en disco.
| Conjunto (ISTA-DAS) | Descarga | Espacio libre necesario |
| --- | ---: | ---: |
| 1.75 bpw | 239 GB | unos 245 GB |
| 2.00 bpw | 257 GB | unos 265 GB |
| 2.25 bpw | 274 GB | unos 280 GB |
Funciona en NVIDIA RTX 30, 40 y 50 (Windows y Linux) y en AMD Radeon AI PRO R9700 (Linux). La instalación descarga los archivos y el motor de DeepSeek, y después inicia el modelo.
Medido con el conjunto de 2.00 bpw en Strata 0.2.5. «Salida» es la velocidad de la respuesta; «Prompt» es la rapidez con la que Strata lee tu texto.
| Hardware | Salida | Prompt |
| --- | ---: | ---: |
| RTX 5070 12 GB, 64 GB de RAM | 4 a 6 tokens/s | unos 110 tokens/s (4K) |
| RTX 3060 12 GB, 46 GB de RAM | 9 a 11 tokens/s | unos 155 tokens/s (4K) |
| 1x Radeon AI PRO R9700, 251 GB de RAM | 21 a 25 tokens/s | 320 a 360 tokens/s (8K) |
| 4x Radeon AI PRO R9700 | 74 a 77 tokens/s | 400 a 450 tokens/s (8K) |
[Memoria, disco y detalles](docs/DEEPSEEK41.md) · [Notas de la versión](https://github.com/Niko1221/Strata/releases)
</details>
<a id="the-other-sizes"></a>
¿No sabes qué cabe? [Elige un modelo para tu ordenador](docs/MODELS.md#pick-by-memory).
## Instalación
### Deja que tu IA lo instale
<a id="option-a-the-strata-app"></a>
¿Usas un asistente de programación con IA (Claude Code, Cursor, Codex, GitHub Copilot, ...)? Pega esto en él:
### Aplicación de escritorio
```text
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
**[Descargar Strata para Windows, Linux o Mac →](https://github.com/Niko1221/Strata/releases/latest)**
Instala la aplicación, elige un modelo y empieza a chatear. Hay un instalador por sistema. En el primer arranque, la aplicación descarga el motor para tu tarjeta gráfica. [Ayuda con el instalador](docs/INSTALL.md#the-app).
<img src="docs/media/app-chat.png" width="900" alt="Aplicación de escritorio de Strata con un chat y una vista previa del código en vivo, una junto a otra">
<br><sub>Chat, archivos y vista previa en vivo en una sola ventana.</sub>
</p>
<a id="option-b-the-terminal-one-line"></a>
### Aplicación de Terminal
¿Prefieres la terminal? Instala Strata y ejecuta **`strata launch`**.
<details open>
<summary><b>Comandos de instalación</b></summary>
**Linux / Mac**
```sh
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
```
Revisa tu tarjeta gráfica, tu RAM y tu disco, y elige el modelo que cabe. Luego lo instala, lo arranca y te explica
cómo conectar tus aplicaciones. Las herramientas de IA también pueden instalar, arrancar y detener Strata con su
[Descarga Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) y descomprímelo (o usa `git clone`).
**Windows:** haz doble clic en **`START-HERE.bat`**. **Linux:** ejecuta **`./setup.sh`** en la carpeta de Strata.
En Windows, abre una terminal nueva después. Luego:
Los pasos son los mismos para NVIDIA y AMD. El instalador detecta tu tarjeta y prepara el motor adecuado para ella.
Te hace unas pocas preguntas:
```sh
strata launch
```
- qué modelo y qué tamaño,
- cuánto contexto (cuánto texto tiene presente el modelo),
- si debe entender imágenes.
Sigue las indicaciones para descargar e iniciar un modelo. [Guía del Terminal](docs/TERMINAL_LAUNCH.md).
Pulsa Enter cada vez para usar la respuesta recomendada. Después descarga el modelo (unos 70 GB) y lo arranca. Si la
descarga se detiene, vuelve a ejecutarlo: sigue donde lo dejó. Tu navegador abre la aplicación de Strata en
`http://127.0.0.1:8080`.
</details>
> **Mientras el modelo arranca, tu PC puede ir lento o dejar de responder durante 1-3 minutos** (más la primera vez).
> Strata carga 35-55 GB en tu RAM y reserva una parte para la tarjeta gráfica. Es normal. Espera y no cierres la
> ventana. La ventana muestra lo que está haciendo Strata.
<p align="center">
<img src="docs/media/terminal-monitor.png" width="900" alt="Aplicación de Terminal de Strata con la actividad del modelo, la velocidad, la memoria y las peticiones recientes">
<br><sub>Observa tu modelo y tu hardware mientras se ejecuta.</sub>
</p>
**La próxima vez**, vuelve a ejecutar `START-HERE.bat` (o `./setup.sh`). Arranca enseguida y no descarga nada dos
veces. Cierra su ventana para detener el modelo. `UPDATE.bat` (`./update.sh`) actualiza Strata sin arrancarlo.
Actualizaciones, Docker, varias tarjetas, dónde se guardan los archivos y todas las opciones: [docs/INSTALL.md](docs/INSTALL.md).
<a id="build-from-source"></a>
## ¿Qué modelo elijo?
[Compilar desde el código fuente](docs/SOURCE_INSTALL.md) · [Docker](docs/DOCKER.md) · [Guía de instalación completa](docs/INSTALL.md)
El instalador te recomienda uno según tu RAM. El mismo modelo viene en varios tamaños, más o menos comprimidos. Los
tamaños pequeños son más rápidos. Los grandes son algo más inteligentes.
<a id="what-you-need"></a>
| Tu RAM | Elige | Por qué |
| --- | --- | --- |
| **32 GB** | **Coder** | cabe en 32 GB y está hecho para programar (con una tarjeta de 24 GB también funcionan Q2_0 e IQ2_XS) |
| **48 GB** | **IQ2_XS** (o Q2_0, el más rápido) | los tamaños más grandes no caben |
| **64 GB** | **IQ2_XS** (recomendado), o IQ3_XXS / IQ3_S | caben todos los tamaños; IQ3_S es el mejor y el más lento |
| **96 GB o más** | **IQ3_S**, o el UD-IQ4_XS de Unsloth (~4 bits) | espacio para los tamaños más grandes con todo lo demás abierto |
## Hardware
- **[Coder](docs/MODELS.md#coder):** una versión para programar a la que se le quitó la mitad de los expertos. Alcanza
el 91% de la puntuación SWE-bench Verified del modelo completo (medido por sus autores) y cabe en 32 GB de RAM. Es
más flojo fuera del código, también con chino y otros textos CJK (#438). Para eso, elige Q2_0, IQ2_XS o IQ3_S, que
conservan todos los expertos.
- **[Swift 1.5](docs/MODELS.md#swift-15):** un ajuste fino que piensa mucho menos tiempo antes de responder. Recibes
la respuesta antes, con más o menos la misma calidad.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** la versión de ~4 bits de Unsloth, entre IQ3_S y
UD-Q4_K_XL en calidad. Una descarga de 94 GB. Con menos de ~80 GB de RAM, Strata lee una parte desde el SSD
mientras responde, así que ahí es más lento (un SSD NVMe ayuda).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (experimental): el más parecido al modelo
completo. Pero Strata lee la mayor parte desde el SSD mientras responde, así que solo escribe 7-8.5 tokens/s en un
PC con 64 GB.
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** se instala a mano. No está en
| **Intel Arc · Linux** | [Instalación en Intel](docs/INTEL_ARC.md) |
| **Apple Silicon · M1 o posterior** | [Instalación en Mac](docs/MAC.md) |
| **AMD Ryzen AI Max · Strix Halo** | [Instalación en Strix Halo](docs/STRIX_HALO.md) |
Tamaños, descargas y qué cabe dónde: [docs/MODELS.md](docs/MODELS.md). Para añadir otro modelo más adelante, ejecuta
`SETUP.bat` (Linux: `./setup.sh --setup`).
<a id="using-it"></a>
## Cómo usarlo
## Herramientas
<p align="center"><img src="docs/media/runpagoda.png" width="900" alt="La pestaña Monitor de la aplicación de Strata junto a un agente de programación"><br>
<sub>El <b>Monitor</b> de la aplicación de Strata (izquierda) mientras un agente de programación escribe el jardín de la pagoda del vídeo (derecha)</sub></p>
**Claude Code · Codex CLI · OpenCode · oh-my-pi · pi.dev · Hermes Agent · OpenClaw**, y más, mediante las API compatibles con OpenAI y Anthropic.
- **En el navegador:** abre `http://127.0.0.1:8080`. Tiene **Chat**, un **Monitor** en vivo del modelo y de tu
GPU/CPU/RAM, y **About** con los ajustes y las direcciones.
- **Tus aplicaciones y agentes de programación:** añade un proveedor "compatible con OpenAI" con la URL base
**`http://127.0.0.1:8080/v1`**. Sirve cualquier clave de API y cualquier nombre de modelo.
- Aplicaciones que usan la API de Anthropic: `http://127.0.0.1:8080/v1/messages` (Claude Code:
`ANTHROPIC_BASE_URL=http://127.0.0.1:8080`).
- Codex CLI y otras aplicaciones que usan la API Responses de OpenAI: `/v1/responses`
- **Razonamiento:** elige **off, low, medium o high** en el menú del chat o en el "reasoning effort" de tu aplicación.
Off es lo más rápido. High es lo mejor para preguntas difíciles.
- **Imágenes:** responde que sí a "Images?" en la instalación. Luego haz clic en **Picture** en el chat, o adjunta
imágenes en tu aplicación. Las tarjetas AMD leen imágenes en Linux a través del procesador; en Windows todavía no.
- **Desde tu móvil u otro PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Pon siempre una clave.
- **Una petición cada vez:** por defecto Strata responde una petición y las demás esperan. Para responder varias a la
vez, pon `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). En una tarjeta de 12 GB, cada respuesta va más lenta.
- **Prompts largos:** Strata lee entero el primer mensaje de un chat, más o menos 1 minuto por cada 30,000 tokens. Los
mensajes siguientes empiezan en segundos.
Ejecuta **`strata connect`** para ver las instrucciones de tu herramienta. [Guía de conexión](docs/TERMINAL_LAUNCH.md#connect-your-apps-strata-connect).
Más: [dónde se guardan tus chats](docs/INSTALL.md#where-things-are-stored), [la API](docs/DETAILS.md#using-it).
<a id="guides"></a>
<a id="how-fast-is-it"></a>
<a id="how-does-it-work"></a>
<a id="something-went-wrong"></a>
## ¿Algo salió mal?
## Ayuda
- **Mi PC se congeló la primera vez que arrancó Strata.** Es normal mientras carga el modelo. Espera y no cierres la
ventana. ¿Sigue congelado después de 10 minutos? Reinicia el PC, cierra otros programas y vuelve a intentarlo, o
elige un tamaño más pequeño.
- **Se detuvo durante la descarga o la instalación.** Vuelve a ejecutar `START-HERE.bat` (o `./setup.sh`). Sigue
donde se detuvo.
- **Va muy lento y la luz del disco no deja de parpadear, o dice "the engine stopped unexpectedly".** Tu PC no tiene
suficiente RAM libre. Cierra otros programas (los navegadores usan mucha), o elige un tamaño más pequeño (Q2_0 o
IQ2_XS).
- **Dice que el puerto 8080 ya está en uso.** Strata ya está en marcha. Busca su ventana.
[Todas las guías](docs) · [Solución de problemas](docs/TROUBLESHOOTING.md) · [Informar de un error](https://github.com/Niko1221/Strata/issues/new/choose) ·
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: apoya Strata" height="48"></a>
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Los 24,576 expertos del modelo: los más usados en la tarjeta gráfica, todos en la RAM y una tabla de consulta en el SSD"></p>
### Apoyos fieles
- **El modelo es un equipo de 24,576 pequeños especialistas ("expertos").** Cada palabra solo necesita 10 de ellos.
- **Tu tarjeta gráfica** guarda los pocos miles de expertos que más se usan. **Tu RAM** los tiene todos, y **tu
procesador** trabaja con el resto al mismo tiempo. **Tu SSD** guarda una gran tabla de consulta.
Gracias a todas las personas que apoyan Strata.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Un pequeño ayudante adivina las siguientes palabras; el modelo grande las comprueba todas a la vez y se queda con las correctas"></p>
<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
- **Adivinar y comprobar:** un pequeño ayudante adivina las siguientes palabras. El modelo grande las comprueba todas
a la vez. Recibes la misma respuesta, 1.6-1.8x antes.
- **Los textos largos se leen en trozos grandes** (hasta 8,192 tokens cada vez), a más de 1,000 tokens por segundo.
[Licencia MIT](LICENSE) · [Créditos y licencias de los modelos](docs/HOW_IT_WORKS.md#credits) · [Privacidad](docs/SAFETY.md)
La explicación larga: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Cada parte y sus números:
[los detalles](docs/DETAILS.md#how-it-works) y el [artículo](docs/paper/Strata-Paper.pdf).
<a id="option-c-let-your-ai-set-it-up"></a>
## Créditos y licencia
<sub>Para herramientas de IA y lectores automáticos: <a href="AGENTS.md">referencia técnica</a> · <a href="docs/AI_SETUP.md">instrucciones de instalación</a>.</sub>
El modelo es [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), del equipo de Qwen. Lo
y Unsloth. Strata usa partes de [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Todos los créditos:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata es de código abierto con la [licencia MIT](LICENSE).
Algunas partes y cada modelo tienen sus propias licencias ([cuáles](docs/HOW_IT_WORKS.md#license)).
## Apoya a Strata
Strata es libre y de código abierto. Si te resulta útil, puedes apoyar su desarrollo:
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Un jardin de pagode en voxels écrit par le modèle de Strata, qui tourne dans le navigateur"></a><br>
<sub>Un jardin de pagode en voxels, généré en une seule requête sur une RTX 5070 avec Strata (IQ3_S, contexte de 128K) ·
<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="Soutenir Strata sur GitHub Sponsors"></a>
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Soutenir Strata sur Buy Me a Coffee"></a>
</p>
Strata fait tourner **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** sur un PC normal. C'est
un grand modèle d'IA, très capable, qui demande d'habitude un serveur. Il discute, écrit du code, lit des images et
fonctionne avec vos applications et vos agents de code. Rien ne quitte votre PC.
**Gratuit, open source, et à vous de l'exécuter.** Utilisez Strata sur votre bureau ou avec vos outils de programmation préférés. Aucun compte nécessaire.
Nous l'avons mesurée sur deux PC de jeu ordinaires. Un token correspond à environ ¾ d'un mot.
## Modèles
- **Écrit les réponses :** la vitesse à laquelle la réponse s'affiche dans une courte discussion. 60 tokens par seconde, c'est plus rapide que votre lecture.
- **Lit votre prompt :** la vitesse à laquelle il absorbe ce que vous envoyez (ici un document, du code ou un historique de discussion de 32K tokens).
Nous recommandons **d'abord ISTA-DASLab**, puis **Unsloth** et d'autres fournisseurs, lorsqu'un modèle compatible tient en mémoire.
<table>
<tr><th>NVIDIA : RTX 5070 (12 Go), Ryzen 5 7600, 64 Go de RAM</th><th>AMD : RX 9070 XT (16 Go), Ryzen 9 3900X, 47 Go de RAM</th></tr>
| Taille | Écrit les réponses | Lit votre prompt |
| --- | ---: | ---: |
| **Q2_0** | 94 tokens/s | 2,650 tokens/s |
| **IQ2_XS** | 79 tokens/s | 2,090 tokens/s |
| **IQ3_XXS** | 62 tokens/s | 1,750 tokens/s |
| **IQ3_S** | 53 tokens/s | 1,620 tokens/s |
| **Coder** | 55 tokens/s | 2,180 tokens/s |
</td><td>
| Taille | Écrit les réponses | Lit votre prompt |
| --- | ---: | ---: |
| **Q2_0** | 60 tokens/s | 1,160 tokens/s |
| **IQ2_XS** | 52 tokens/s | 1,110 tokens/s |
| **Coder** | 44 tokens/s | 1,420 tokens/s |
</td></tr>
</table>
NVIDIA : Q2_0 avec le moteur 0.1.36, les autres lignes avec 0.1.26 (réponses de 4K, prompts de 32K). Les tableaux
complets sont dans [DETAILS.md](docs/DETAILS.md#speed-measured). Une carte avec plus de VRAM va plus vite : une
RTX 3090 (24 Go) devrait écrire environ 100-140 tokens par seconde. Longues discussions et autres cartes :
[vitesse de chaque modèle](docs/MODELS.md#how-fast-is-each-size), [résultats de la communauté](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata est gratuit. S'il tourne bien sur votre PC, un café aide à poursuivre le travail.</sub></p>
## Ce qu'il vous faut
| | |
| Fournisseur | Choix |
| --- | --- |
| **Carte graphique** | **NVIDIA** GeForce RTX série 20, 30, 40 ou 50, ou **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 ou série RX 6800 / 6900. Il faut **12 Go de VRAM ou plus**. |
| **RAM** | 32 Go ou plus. Votre RAM décide [quel modèle](#quel-modèle-choisir-) tient. Avec 64 Go, toutes les tailles tournent. |
| **Disque** | Environ 80 Go libres. Prenez un SSD si possible : le premier démarrage est bien plus rapide. |
| **Système** | Windows 10 / 11 ou Linux, avec un pilote graphique NVIDIA ou AMD à jour. |
| **Gyro · agentionai** | [Gyro-S, Gyro-M](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF). Utilise moins de mémoire, mais plus lent. |
Dans l'**application de bureau** et dans le **Terminal**. Ce qui ne tient pas en mémoire est lu depuis votre SSD : il faut donc beaucoup d'espace disque libre.
Fonctionne sur les NVIDIA RTX 30, 40 et 50 (Windows et Linux) et sur l'AMD Radeon AI PRO R9700 (Linux). L'installation télécharge les fichiers et le moteur DeepSeek, puis démarre le modèle.
Mesuré avec le jeu 2.00 bpw sur Strata 0.2.5. « Sortie » est la vitesse de la réponse, « Prompt » la vitesse à laquelle Strata lit votre texte.
| Matériel | Sortie | Prompt |
| --- | ---: | ---: |
| RTX 5070 12 GB, 64 GB de RAM | 4 à 6 tokens/s | environ 110 tokens/s (4K) |
| RTX 3060 12 GB, 46 GB de RAM | 9 à 11 tokens/s | environ 155 tokens/s (4K) |
| 1x Radeon AI PRO R9700, 251 GB de RAM | 21 à 25 tokens/s | 320 à 360 tokens/s (8K) |
| 4x Radeon AI PRO R9700 | 74 à 77 tokens/s | 400 à 450 tokens/s (8K) |
[Mémoire, disque et détails](docs/DEEPSEEK41.md) · [Notes de version](https://github.com/Niko1221/Strata/releases)
</details>
<a id="the-other-sizes"></a>
Vous ne savez pas ce qui tient ? [Choisir un modèle pour votre ordinateur](docs/MODELS.md#pick-by-memory).
## Installation
### Laissez votre IA l'installer
<a id="option-a-the-strata-app"></a>
Vous utilisez un assistant de code IA (Claude Code, Cursor, Codex, GitHub Copilot, ...) ? Collez-lui ceci :
### Application de bureau
```text
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
**[Télécharger Strata pour Windows, Linux ou Mac →](https://github.com/Niko1221/Strata/releases/latest)**
Installez l'application, choisissez un modèle et commencez à discuter. Il y a un seul installateur par système. Au premier lancement, l'application télécharge le moteur de votre carte graphique. [Aide pour l'installateur](docs/INSTALL.md#the-app).
[Téléchargez Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) et décompressez-le (ou faites un `git clone`).
**Windows :** double-cliquez sur **`START-HERE.bat`**. **Linux :** lancez **`./setup.sh`** dans le dossier Strata.
Sous Windows, ouvrez ensuite un nouveau terminal. Puis :
Les étapes sont les mêmes pour NVIDIA et AMD. L'installateur détecte votre carte et installe le bon moteur pour elle.
Il vous pose quelques questions :
```sh
strata launch
```
- quel modèle et quelle taille,
- combien de contexte (la quantité de texte que le modèle garde en tête),
- s'il doit lire les images.
Suivez les invites pour télécharger et démarrer un modèle. [Guide du Terminal](docs/TERMINAL_LAUNCH.md).
Appuyez sur Entrée à chaque fois pour garder la réponse recommandée. Ensuite il télécharge le modèle (environ 70 Go)
et le démarre. Si le téléchargement s'arrête, relancez-le : il reprend là où il s'était arrêté. Votre navigateur
ouvre l'application Strata à l'adresse `http://127.0.0.1:8080`.
</details>
> **Pendant le démarrage du modèle, votre PC peut ralentir ou ne plus répondre pendant 1-3 minutes** (plus longtemps la première fois).
> Strata charge 35-55 Go dans votre RAM et en réserve une partie pour la carte graphique. C'est normal. Attendez, et
> ne fermez pas la fenêtre. La fenêtre montre ce que fait Strata.
<p align="center">
<img src="docs/media/terminal-monitor.png" width="900" alt="Application Terminal de Strata affichant l'activité du modèle, la vitesse, la mémoire et les requêtes récentes">
<br><sub>Suivez votre modèle et votre matériel pendant qu'il tourne.</sub>
</p>
**La fois suivante**, relancez `START-HERE.bat` (ou `./setup.sh`). Il démarre tout de suite et ne télécharge rien
deux fois. Fermez sa fenêtre pour arrêter le modèle. `UPDATE.bat` (`./update.sh`) met Strata à jour sans le démarrer.
Mises à jour, Docker, plusieurs cartes, emplacement des fichiers et toutes les options : [docs/INSTALL.md](docs/INSTALL.md).
<a id="build-from-source"></a>
## Quel modèle choisir ?
[Compiler depuis les sources](docs/SOURCE_INSTALL.md) · [Docker](docs/DOCKER.md) · [Guide d'installation complet](docs/INSTALL.md)
L'installateur en recommande un selon votre RAM. Le même modèle existe en plusieurs tailles, plus ou moins
compressées. Les petites tailles sont plus rapides. Les grandes sont un peu plus intelligentes.
<a id="what-you-need"></a>
| Votre RAM | Prenez | Pourquoi |
| --- | --- | --- |
| **32 Go** | **Coder** | il tient dans 32 Go et il est fait pour le code (avec une carte de 24 Go, Q2_0 et IQ2_XS tournent aussi) |
| **48 Go** | **IQ2_XS** (ou Q2_0, le plus rapide) | les plus grandes tailles ne tiennent pas |
| **64 Go** | **IQ2_XS** (recommandé), ou IQ3_XXS / IQ3_S | toutes les tailles tiennent ; IQ3_S est le meilleur et le plus lent |
| **96 Go ou plus** | **IQ3_S**, ou l'UD-IQ4_XS d'Unsloth (~4 bits) | de la place pour les plus grandes tailles, avec tout le reste ouvert |
## Matériel
- **[Coder](docs/MODELS.md#coder) :** une version pour le code, avec la moitié des experts en moins. Elle atteint 91 %
du score SWE-bench Verified du modèle complet (mesuré par ses auteurs) et tient dans 32 Go de RAM. Elle est moins
bonne en dehors du code, y compris pour le chinois et les autres textes CJK (#438). Pour ces cas, prenez Q2_0,
IQ2_XS ou IQ3_S, qui gardent tous les experts.
- **[Swift 1.5](docs/MODELS.md#swift-15) :** un fine-tune qui réfléchit bien moins longtemps avant de répondre. Vous
avez la réponse plus tôt, avec à peu près la même qualité.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs)** : la version ~4 bits d'Unsloth, entre IQ3_S et
UD-Q4_K_XL en qualité. Un téléchargement de 94 Go. Avec moins de ~80 Go de RAM, Strata en lit une partie depuis
le SSD pendant qu'il répond, il y est donc plus lent (un SSD NVMe aide).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (expérimental) : le plus proche du modèle
complet. Mais Strata en lit la plus grande partie depuis le SSD pendant qu'il répond, donc il n'écrit que
7-8.5 tokens/s sur un PC avec 64 Go.
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs) :** vous l'installez à la main.
- **Réflexion :** choisissez **off, low, medium ou high** dans le menu du chat ou dans le « reasoning effort » de
votre application. Off est le plus rapide. High est le meilleur pour les questions difficiles.
- **Images :** répondez oui à « Images? » pendant l'installation. Ensuite cliquez sur **Picture** dans le chat, ou
joignez des images dans votre application. Les cartes AMD lisent les images sous Linux via le processeur ; sous
Windows, pas encore.
- **Depuis votre téléphone ou un autre PC :** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Définissez
toujours une clé.
- **Une requête à la fois :** par défaut, Strata répond à une requête et les autres attendent. Pour répondre à
plusieurs en même temps, mettez `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). Sur une carte de 12 Go, chaque
réponse devient alors plus lente.
- **Longs prompts :** Strata lit en entier le premier message d'une discussion, environ 1 minute pour 30,000 tokens.
Les messages suivants démarrent en quelques secondes.
Lancez **`strata connect`** pour voir la configuration de votre outil. [Guide de connexion](docs/TERMINAL_LAUNCH.md#connect-your-apps-strata-connect).
Plus d'infos : [où sont stockées vos discussions](docs/INSTALL.md#where-things-are-stored), [l'API](docs/DETAILS.md#using-it).
<a id="guides"></a>
<a id="how-fast-is-it"></a>
<a id="how-does-it-work"></a>
<a id="something-went-wrong"></a>
## Un problème ?
## Aide
- **Mon PC s'est figé au premier démarrage de Strata.** C'est normal pendant le chargement du modèle. Attendez, et ne
fermez pas la fenêtre. Toujours figé après 10 minutes ? Redémarrez le PC, fermez les autres programmes et
réessayez, ou choisissez une taille plus petite.
- **Il s'est arrêté pendant le téléchargement ou l'installation.** Relancez `START-HERE.bat` (ou `./setup.sh`). Il
reprend là où il s'était arrêté.
- **C'est très lent et le voyant du disque clignote sans arrêt, ou il affiche « the engine stopped unexpectedly ».**
Votre PC n'a pas assez de RAM libre. Fermez les autres programmes (les navigateurs en utilisent beaucoup), ou
choisissez une taille plus petite (Q2_0 ou IQ2_XS).
- **Il dit que le port 8080 est déjà utilisé.** Strata tourne déjà. Cherchez sa fenêtre.
Autres problèmes et leurs solutions : [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Toujours bloqué ? Ouvrez une
[issue](https://github.com/Niko1221/Strata/issues) et joignez `strata-<model>.log` depuis le dossier Strata. Vous
avez trouvé un problème de sécurité ? Signalez-le en privé : [SECURITY.md](SECURITY.md).
## Comment ça marche ?
Les modèles comme celui-ci tournent d'habitude sur des serveurs avec des centaines de gigaoctets de mémoire
graphique. Votre carte graphique a 12-24 Go. Strata fait tenir le modèle en **répartissant le travail sur tout votre
PC**. Pensez à une cuisine : ce que vous utilisez tout le temps reste sur le plan de travail, et le reste attend dans
le placard.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Les 24,576 experts du modèle : les plus sollicités sur la carte graphique, tous en RAM, une table de correspondance sur le SSD"></p>
- **Le modèle est une équipe de 24,576 petits spécialistes (les « experts »).** Chaque mot n'en demande que 10.
- **Votre carte graphique** garde les quelques milliers d'experts les plus utilisés. **Votre RAM** les contient tous,
et **votre processeur** travaille sur les autres en même temps. **Votre SSD** contient une grande table de
correspondance.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Un petit assistant devine les mots suivants ; le grand modèle les vérifie tous d'un coup et garde les bons"></p>
- **Deviner, puis vérifier :** un petit assistant devine les quelques mots suivants. Le grand modèle les vérifie
tous d'un coup. Vous obtenez la même réponse, 1.6-1.8x plus tôt.
- **Les longs textes sont lus par gros morceaux** (jusqu'à 8,192 tokens à la fois), à plus de 1,000 tokens par seconde.
L'explication plus longue : [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Chaque partie et ses chiffres :
[les détails](docs/DETAILS.md#how-it-works) et l'[article](docs/paper/Strata-Paper.pdf).
## Crédits et licence
Le modèle est [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), de l'équipe Qwen. Il a été
compressé par [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
et Unsloth. Strata utilise des parties de [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Tous les
crédits : [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata est open source sous [licence MIT](LICENSE).
Quelques parties et chaque modèle ont leur propre licence ([lesquelles](docs/HOW_IT_WORKS.md#license)).
[Tous les guides](docs) · [Dépannage](docs/TROUBLESHOOTING.md) · [Signaler un bogue](https://github.com/Niko1221/Strata/issues/new/choose) ·
[Benchmarks](docs/COMMUNITY_BENCHMARKS.md) · [Comment ça marche](docs/HOW_IT_WORKS.md) · [Contribuer](CONTRIBUTING.md)
## Soutenir Strata
Strata est gratuit et open source. S'il vous est utile, vous pouvez soutenir son développement :
**Aidez à garder l'IA locale ouverte à tous.** Un parrainage sur GitHub ou un café aide à financer le développement et les tests sur davantage de matériel.
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Buy Me a Coffee で Strata を支援する"></a>
</p>
Strata は **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** を普通の PC で動かします。
これは大きくて賢い AI モデルで、ふつうはサーバーが必要です。チャットをしたり、コードを書いたり、画像を読んだりできます。
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata は無料です。あなたの PC でうまく動いたら、コーヒー 1 杯の支援が開発を続ける力になります。</sub></p>
**ローカル AI をみんなに開かれたものに。** GitHub でのスポンサーやコーヒー1杯が、開発と、より多くのハードウェアでのテストの支えになります。
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: Strata を支援する" height="48"></a>
### いつも支えてくださる方々
Strata を支えてくださるすべての方に感謝します。
<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="A voxel pagoda garden that Strata's model wrote, running in the browser"></a><br>
<sub>A voxel pagoda garden, 1 shot prompt running on an RTX 5070 with Strata (IQ3_S, 128K context) ·
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">full video (49 s)</a></sub></p>
<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="Sponsor Strata on GitHub Sponsors"></a>
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Support Strata on Buy Me a Coffee"></a>
</p>
Strata runs **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** on a normal PC. This is a
large, smart AI model that usually needs a server. It chats, writes code, reads pictures and works with your apps
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata is free. If it runs well on your PC, a coffee keeps the work on it going.</sub></p>
## What you need
| | |
| Provider | Choices |
| --- | --- |
| **Graphics card** | **NVIDIA** GeForce RTX 20, 30, 40 or 50 series, or **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 or RX 6800 / 6900 series. It needs **12 GB of VRAM or more**. |
| **RAM** | 32 GB or more. Your RAM decides [which model](#which-model-should-i-pick) fits. 64 GB runs every size. |
| **Disk** | About 80 GB free. Use an SSD if you can: the first start is much faster. |
| **System** | Windows 10 / 11 or Linux, and a current graphics driver from NVIDIA or AMD. |
In the **desktop app** and the **Terminal**. What does not fit in memory is read from your SSD, so it needs a lot of free
disk space.
| Set (ISTA-DAS) | Download | Free disk needed |
| --- | ---: | ---: |
| 1.75 bpw | 239 GB | about 245 GB |
| 2.00 bpw | 257 GB | about 265 GB |
| 2.25 bpw | 274 GB | about 280 GB |
| ISTA-DAS Coder (half of the experts removed) | 191 GB | about 197 GB |
Runs on NVIDIA RTX 30, 40 and 50 series (Windows and Linux), AMD Radeon AI PRO R9700 (Linux), AMD Ryzen AI Max — Strix Halo
(Windows), and Apple Silicon Macs with 64 GB or more, text only ([Mac](docs/DEEPSEEK_MAC.md)). Setup downloads the files and the DeepSeek engine, then starts the model.
- **Thinking:** choose **off, low, medium or high** in the chat menu or in your app's "reasoning effort". Off is the
fastest. High is best for hard questions.
- **Pictures:** say yes to "Images?" in setup. Then click **Picture** in the chat, or attach pictures in your app.
AMD cards read pictures on Linux through the processor; on Windows they can't yet.
- **From your phone or another PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Always set a key.
- **One request at a time:** by default Strata answers one request, and the others wait. To answer several at once,
set `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). On a 12 GB card this makes each answer slower.
- **Long prompts:** Strata reads the first message of a chat in full, about 1 minute per 30,000 tokens. Follow-up
messages start in seconds.
Run **`strata connect`** to see setup instructions for your tool. [Connection guide](docs/TERMINAL_LAUNCH.md#connect-your-apps-strata-connect).
More: [where your chats are stored](docs/INSTALL.md#where-things-are-stored), [the API](docs/DETAILS.md#using-it).
<a id="guides"></a>
<a id="how-fast-is-it"></a>
<a id="how-does-it-work"></a>
<a id="something-went-wrong"></a>
## Something went wrong?
## Help
- **My PC froze the first time Strata started.** This is normal while it loads the model. Wait, and don't close the
window. Still frozen after 10 minutes? Restart the PC, close other programs and try again, or pick a smaller size.
- **It stopped while downloading or installing.** Run `START-HERE.bat` (or `./setup.sh`) again. It continues where
it stopped.
- **It's very slow and the disk light keeps blinking, or it says "the engine stopped unexpectedly".** Your PC does
not have enough free RAM. Close other programs (browsers use a lot), or pick a smaller size (Q2_0 or IQ2_XS).
- **It says port 8080 is already in use.** Strata is already running. Look for its window.
More problems and their fixes: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Still stuck? Open an
[issue](https://github.com/Niko1221/Strata/issues) and attach `strata-<model>.log` from the Strata folder. Found a
security problem? Report it privately: [SECURITY.md](SECURITY.md).
## How does it work?
Models like this one usually run on servers with hundreds of gigabytes of graphics memory. Your graphics card has
12-24 GB. Strata makes the model fit by **sharing the work across your whole PC**. Think of a kitchen: the things
you use all the time stay on the counter, and the rest waits in the pantry.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="The model's 24,576 experts: the busiest on the graphics card, all of them in RAM, a lookup table on the SSD"></p>
- **The model is a team of 24,576 small specialists ("experts").** Each word needs only 10 of them.
- **Your graphics card** keeps the few thousand experts that are used most often. **Your RAM** holds all of them,
and **your processor** works on the rest at the same time. **Your SSD** holds a big lookup table.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="A small helper guesses the next words; the big model checks them all at once and keeps the right ones"></p>
- **Guess, then check:** a small helper guesses the next few words. The big model checks them all at once. You get
the same answer, 1.6-1.8x sooner.
- **Long texts are read in big pieces** (up to 8,192 tokens at a time), at over 1,000 tokens per second.
The longer explanation: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Every part and its numbers:
[the details](docs/DETAILS.md#how-it-works) and the [paper](docs/paper/Strata-Paper.pdf).
## Credits and license
The model is [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team. It was
compressed by [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
and Unsloth. Strata uses parts of [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). All credits:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata is open source under the [MIT License](LICENSE). A few
parts and every model have their own licenses ([which ones](docs/HOW_IT_WORKS.md#license)).
[All guides](docs) · [Troubleshooting](docs/TROUBLESHOOTING.md) · [Report a bug](https://github.com/Niko1221/Strata/issues/new/choose) ·
[Benchmarks](docs/COMMUNITY_BENCHMARKS.md) · [How it works](docs/HOW_IT_WORKS.md) · [Contribute](CONTRIBUTING.md)
## Support Strata
Strata is free and open source. If it is useful to you, you can support its development:
**Help keep local AI open to everyone.** A GitHub sponsorship or a coffee helps fund development and testing on more hardware.
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: support Strata" height="48"></a>
### Loyal supporters
Thank you to everyone who supports Strata.
<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="Wesprzyj Stratę na GitHub Sponsors"></a>
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Wesprzyj Stratę na Buy Me a Coffee"></a>
**Bezpłatna, z otwartym kodem źródłowym i do uruchomienia u siebie.** Używaj Straty na swoim komputerze albo razem z ulubionymi narzędziami do programowania. Konto nie jest potrzebne.
## Modele
Najpierw polecamy **ISTA-DASLab**, potem **Unsloth** i innych dostawców — o ile zgodny model mieści się w pamięci.
| **Gyro · agentionai** | [Gyro-S, Gyro-M](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF). Zużywa mniej pamięci, ale działa wolniej. |
W Windows otwórz potem nowe okno terminala. Następnie:
```sh
strata launch
```
Postępuj zgodnie z podpowiedziami, aby pobrać i uruchomić model. [Przewodnik po Terminalu](docs/TERMINAL_LAUNCH.md).
</details>
<p align="center">
<img src="docs/media/terminal-monitor.png" width="900" alt="Aplikacja Terminal Straty pokazująca aktywność modelu, szybkość, pamięć i ostatnie zapytania">
<br><sub>Obserwuj model i sprzęt w trakcie pracy.</sub>
</p>
<a id="build-from-source"></a>
[Budowanie ze źródeł](docs/SOURCE_INSTALL.md) · [Docker](docs/DOCKER.md) · [Pełny przewodnik instalacji](docs/INSTALL.md)
| **AMD Ryzen AI Max · Strix Halo** | [Konfiguracja Strix Halo](docs/STRIX_HALO.md) |
<a id="using-it"></a>
## Narzędzia
**Claude Code · Codex CLI · OpenCode · oh-my-pi · pi.dev · Hermes Agent · OpenClaw** i inne — przez API zgodne z OpenAI i Anthropic.
Uruchom **`strata connect`**, aby zobaczyć instrukcję konfiguracji dla swojego narzędzia. [Przewodnik po połączeniach](docs/TERMINAL_LAUNCH.md#connect-your-apps-strata-connect).
[Testy wydajności](docs/COMMUNITY_BENCHMARKS.md) · [Jak to działa](docs/HOW_IT_WORKS.md) · [Współtworzenie](CONTRIBUTING.md)
## Wesprzyj Stratę
**Pomóż zachować lokalną sztuczną inteligencję otwartą dla wszystkich.** Sponsoring na GitHubie lub kawa pomaga finansować rozwój i testy na większej liczbie sprzętu.
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: wesprzyj Stratę" height="48"></a>
### Wierni wspierający
Dziękujemy wszystkim, którzy wspierają Stratę.
<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Um jardim com pagode em voxels que o modelo do Strata escreveu, rodando no navegador"></a><br>
<sub>Um jardim com pagode em voxels, feito com um único prompt numa RTX 5070 com o Strata (IQ3_S, contexto de 128K) ·
<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="Apoie o Strata no GitHub Sponsors"></a>
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Apoie o Strata no Buy Me a Coffee"></a>
</p>
O Strata roda o **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** num PC comum. É um modelo
de IA grande e inteligente que normalmente precisa de um servidor. Ele conversa, escreve código, entende imagens e
funciona com seus apps e agentes de programação. Nada sai do seu PC.
NVIDIA: Q2_0 com o engine 0.1.36, as outras linhas com 0.1.26 (respostas de 4K, prompts de 32K). As tabelas
completas estão em [DETAILS.md](docs/DETAILS.md#speed-measured). Uma placa com mais VRAM é mais rápida: uma
RTX 3090 (24 GB) deve escrever cerca de 100-140 tokens por segundo. Conversas longas e outras placas:
[velocidade de cada modelo](docs/MODELS.md#how-fast-is-each-size), [resultados da comunidade](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>O Strata é gratuito. Se ele roda bem no seu PC, um café ajuda a manter o trabalho nele.</sub></p>
## Do que você precisa
| | |
| Provedor | Opções |
| --- | --- |
| **Placa de vídeo** | **NVIDIA** GeForce RTX séries 20, 30, 40 ou 50, ou **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 ou séries RX 6800 / 6900. Ela precisa de **12 GB de VRAM ou mais**. |
| **RAM** | 32 GB ou mais. Sua RAM decide [qual modelo](#qual-modelo-devo-escolher) cabe. Com 64 GB, roda qualquer tamanho. |
| **Disco** | Cerca de 80 GB livres. Use um SSD se puder: a primeira inicialização fica bem mais rápida. |
| **Sistema** | Windows 10 / 11 ou Linux, e um driver de vídeo atualizado da NVIDIA ou da AMD. |
No **aplicativo de desktop** e no **Terminal**. O que não cabe na memória é lido do seu SSD, por isso é preciso muito espaço livre em disco.
| Conjunto (ISTA-DAS) | Download | Espaço livre necessário |
| --- | ---: | ---: |
| 1.75 bpw | 239 GB | cerca de 245 GB |
| 2.00 bpw | 257 GB | cerca de 265 GB |
| 2.25 bpw | 274 GB | cerca de 280 GB |
Roda em NVIDIA RTX 30, 40 e 50 (Windows e Linux) e em AMD Radeon AI PRO R9700 (Linux). A instalação baixa os arquivos e o motor do DeepSeek e depois inicia o modelo.
Medido com o conjunto de 2.00 bpw no Strata 0.2.5. “Saída” é a velocidade da resposta; “Prompt” é a rapidez com que o Strata lê o seu texto.
| Hardware | Saída | Prompt |
| --- | ---: | ---: |
| RTX 5070 12 GB, 64 GB de RAM | 4 a 6 tokens/s | cerca de 110 tokens/s (4K) |
| RTX 3060 12 GB, 46 GB de RAM | 9 a 11 tokens/s | cerca de 155 tokens/s (4K) |
| 1x Radeon AI PRO R9700, 251 GB de RAM | 21 a 25 tokens/s | 320 a 360 tokens/s (8K) |
| 4x Radeon AI PRO R9700 | 74 a 77 tokens/s | 400 a 450 tokens/s (8K) |
[Memória, disco e detalhes](docs/DEEPSEEK41.md) · [Notas da versão](https://github.com/Niko1221/Strata/releases)
</details>
<a id="the-other-sizes"></a>
Não sabe o que cabe? [Escolha um modelo para o seu computador](docs/MODELS.md#pick-by-memory).
## Instalação
### Deixe sua IA instalar
<a id="option-a-the-strata-app"></a>
Você usa um assistente de programação com IA (Claude Code, Cursor, Codex, GitHub Copilot, ...)? Cole isto nele:
### Aplicativo de desktop
```text
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
**[Baixar o Strata para Windows, Linux ou Mac →](https://github.com/Niko1221/Strata/releases/latest)**
Instale o aplicativo, escolha um modelo e comece a conversar. Há um instalador por sistema. Na primeira vez que abre, o aplicativo baixa o motor da sua placa de vídeo. [Ajuda com o instalador](docs/INSTALL.md#the-app).
[Baixe o Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) e descompacte (ou use `git clone`).
**Windows:** clique duas vezes em **`START-HERE.bat`**. **Linux:** rode **`./setup.sh`** na pasta do Strata.
No Windows, abra um novo terminal depois. Em seguida:
Os passos são os mesmos para NVIDIA e AMD. O instalador encontra sua placa e configura o engine certo para ela.
Ele faz algumas perguntas:
```sh
strata launch
```
- qual modelo e qual tamanho,
- quanto contexto (quanto texto o modelo guarda na memória),
- se ele deve entender imagens.
Siga as instruções para baixar e iniciar um modelo. [Guia do Terminal](docs/TERMINAL_LAUNCH.md).
Aperte Enter em cada uma para usar a resposta recomendada. Depois ele baixa o modelo (cerca de 70 GB) e o inicia.
Se o download parar, rode de novo: ele continua de onde parou. Seu navegador abre o app do Strata em
`http://127.0.0.1:8080`.
</details>
> **Enquanto o modelo inicia, seu PC pode ficar lento ou parar de responder por 1-3 minutos** (mais na primeira vez).
> O Strata carrega 35-55 GB na sua RAM e reserva parte dela para a placa de vídeo. Isso é normal. Espere e não
> feche a janela. A janela mostra o que o Strata está fazendo.
<p align="center">
<img src="docs/media/terminal-monitor.png" width="900" alt="Aplicativo de Terminal do Strata mostrando a atividade do modelo, a velocidade, a memória e as solicitações recentes">
<br><sub>Acompanhe seu modelo e seu hardware enquanto ele roda.</sub>
</p>
**Da próxima vez**, rode `START-HERE.bat` (ou `./setup.sh`) de novo. Ele inicia na hora e não baixa nada duas
vezes. Feche a janela para parar o modelo. `UPDATE.bat` (`./update.sh`) atualiza o Strata sem iniciá-lo.
Atualização, Docker, várias placas, onde ficam os arquivos e todas as opções: [docs/INSTALL.md](docs/INSTALL.md).
<a id="build-from-source"></a>
## Qual modelo devo escolher?
[Compilar a partir do código-fonte](docs/SOURCE_INSTALL.md) · [Docker](docs/DOCKER.md) · [Guia de instalação completo](docs/INSTALL.md)
O instalador recomenda um de acordo com sua RAM. O mesmo modelo vem em vários tamanhos, mais ou menos comprimidos.
Os tamanhos menores são mais rápidos. Os maiores são um pouco mais inteligentes.
<a id="what-you-need"></a>
| Sua RAM | Escolha | Por quê |
| --- | --- | --- |
| **32 GB** | **Coder** | cabe em 32 GB e é feito para código (com uma placa de 24 GB, Q2_0 e IQ2_XS também rodam) |
| **48 GB** | **IQ2_XS** (ou Q2_0, o mais rápido) | os tamanhos maiores não cabem |
| **64 GB** | **IQ2_XS** (recomendado), ou IQ3_XXS / IQ3_S | todos os tamanhos cabem; IQ3_S é o melhor e o mais lento |
| **96 GB ou mais** | **IQ3_S**, ou o UD-IQ4_XS da Unsloth (~4-bit) | espaço para os maiores tamanhos com todo o resto aberto |
## Hardware
- **[Coder](docs/MODELS.md#coder):** uma versão para programação com metade dos experts removida. Ela alcança 91%
da pontuação do modelo completo no SWE-bench Verified (medido pelos autores) e cabe em 32 GB de RAM. É mais fraca
fora do código, inclusive em chinês e outros textos CJK (#438). Para isso, escolha Q2_0, IQ2_XS ou IQ3_S, que
mantêm todos os experts.
- **[Swift 1.5](docs/MODELS.md#swift-15):** um fine-tune que pensa por bem menos tempo antes de responder. Você
recebe a resposta mais cedo, com quase a mesma qualidade.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** a versão de ~4 bits da Unsloth, entre IQ3_S e
UD-Q4_K_XL em qualidade. Um download de 94 GB. Com menos de ~80 GB de RAM, o Strata lê uma parte dele do SSD
enquanto responde, então ali ele fica mais lento (um SSD NVMe ajuda).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (experimental): o mais próximo do modelo
completo. Mas o Strata lê a maior parte dele do SSD enquanto responde, então ele escreve só 7-8.5 tokens/s num PC
com 64 GB.
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** você configura à mão. Ele
- **Raciocínio:** escolha **off, low, medium ou high** no menu do chat ou no "reasoning effort" do seu app. Off é o
mais rápido. High é o melhor para perguntas difíceis.
- **Imagens:** responda sim para "Images?" na instalação. Depois clique em **Picture** no chat, ou anexe imagens no
seu app. Placas AMD entendem imagens no Linux usando o processador; no Windows, ainda não.
- **Do seu celular ou de outro PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Sempre defina uma chave.
- **Uma requisição por vez:** por padrão, o Strata responde uma requisição e as outras esperam. Para responder
várias ao mesmo tempo, defina `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). Numa placa de 12 GB, isso deixa
cada resposta mais lenta.
- **Prompts longos:** o Strata lê a primeira mensagem de uma conversa inteira, cerca de 1 minuto a cada 30,000
tokens. As mensagens seguintes começam em segundos.
Execute **`strata connect`** para ver as instruções da sua ferramenta. [Guia de conexão](docs/TERMINAL_LAUNCH.md#connect-your-apps-strata-connect).
Mais: [onde suas conversas ficam salvas](docs/INSTALL.md#where-things-are-stored), [a API](docs/DETAILS.md#using-it).
<a id="guides"></a>
<a id="how-fast-is-it"></a>
<a id="how-does-it-work"></a>
<a id="something-went-wrong"></a>
## Algo deu errado?
## Ajuda
- **Meu PC travou na primeira vez que o Strata iniciou.** Isso é normal enquanto ele carrega o modelo. Espere e não
feche a janela. Ainda travado depois de 10 minutos? Reinicie o PC, feche outros programas e tente de novo, ou
escolha um tamanho menor.
- **Parou durante o download ou a instalação.** Rode `START-HERE.bat` (ou `./setup.sh`) de novo. Ele continua de
onde parou.
- **Está muito lento e a luz do disco não para de piscar, ou aparece "the engine stopped unexpectedly".** Seu PC
não tem RAM livre suficiente. Feche outros programas (navegadores usam muita), ou escolha um tamanho menor (Q2_0
ou IQ2_XS).
- **Diz que a porta 8080 já está em uso.** O Strata já está rodando. Procure a janela dele.
Mais problemas e como resolver: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Ainda sem solução? Abra uma
[issue](https://github.com/Niko1221/Strata/issues) e anexe o `strata-<model>.log` da pasta do Strata. Encontrou um
problema de segurança? Informe em particular: [SECURITY.md](SECURITY.md).
## Como funciona?
Modelos como este normalmente rodam em servidores com centenas de gigabytes de memória de vídeo. Sua placa de vídeo
tem 12-24 GB. O Strata faz o modelo caber **dividindo o trabalho pelo PC inteiro**. Pense numa cozinha: o que você
usa o tempo todo fica na bancada, e o resto espera na despensa.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Os 24,576 experts do modelo: os mais usados na placa de vídeo, todos na RAM, uma tabela de consulta no SSD"></p>
- **O modelo é uma equipe de 24,576 pequenos especialistas ("experts").** Cada palavra precisa de só 10 deles.
- **Sua placa de vídeo** guarda os poucos milhares de experts usados com mais frequência. **Sua RAM** guarda todos
eles, e **seu processador** trabalha no resto ao mesmo tempo. **Seu SSD** guarda uma grande tabela de consulta.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Um pequeno ajudante adivinha as próximas palavras; o modelo grande confere todas de uma vez e fica com as certas"></p>
- **Adivinhar e conferir:** um pequeno ajudante adivinha as próximas palavras. O modelo grande confere todas de
uma vez. Você recebe a mesma resposta, 1.6-1.8x mais cedo.
- **Textos longos são lidos em grandes pedaços** (até 8,192 tokens por vez), a mais de 1,000 tokens por segundo.
A explicação mais longa: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Cada parte e seus números:
[os detalhes](docs/DETAILS.md#how-it-works) e o [artigo](docs/paper/Strata-Paper.pdf).
## Créditos e licença
O modelo é o [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), da equipe Qwen. Ele foi
comprimido por [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
e Unsloth. O Strata usa partes do [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Todos os créditos:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). O Strata é de código aberto sob a [MIT License](LICENSE).
Algumas partes e todos os modelos têm licenças próprias ([quais](docs/HOW_IT_WORKS.md#license)).
[Todos os guias](docs) · [Solução de problemas](docs/TROUBLESHOOTING.md) · [Relatar um bug](https://github.com/Niko1221/Strata/issues/new/choose) ·
O Strata é gratuito e de código aberto. Se ele é útil para você, você pode apoiar o desenvolvimento:
**Ajude a manter a IA local aberta para todos.** Um patrocínio no GitHub ou um café ajuda a financiar o desenvolvimento e os testes em mais hardware.
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor_on_GitHub-EA4AAA?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="GitHub Sponsors: apoie o Strata" height="48"></a>
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: apoie o Strata" height="48"></a>
### Apoiadores fiéis
Obrigado a todas as pessoas que apoiam o Strata.
<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
[Licença MIT](LICENSE) · [Créditos e licenças dos modelos](docs/HOW_IT_WORKS.md#credits) · [Privacidade](docs/SAFETY.md)
<a id="option-c-let-your-ai-set-it-up"></a>
<sub>Para ferramentas de IA e leitores automáticos: <a href="AGENTS.md">referência técnica</a> · <a href="docs/AI_SETUP.md">instruções de instalação</a>.</sub>
<a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="在 Buy Me a Coffee 上支持 Strata"></a>
<a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee：支持 Strata" height="48"></a>
### 忠实支持者
感谢每一位支持 Strata 的人。
<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
@@ -23,8 +23,9 @@ this PC only. The details and every setting are in [docs/DETAILS.md](docs/DETAIL
- **Where it listens.** `127.0.0.1` by default. `--host 0.0.0.0` (or `"host"` in `strata-<model>.json`) opens it to
your network, and the server then warns when no API key is set.
- **API key.** `"api_key"` in the run config (or `STRATA_API_KEY`) is required on `/v1/*` and on every endpoint
that shows the model's state, requests or answers (`/status`, `/metrics`, `/settings`, `/mcp`, `/props`, `/slots`,
`/api/requests`, `/config`) and on every `POST`. It is compared in constant time. Set one before you open the
that shows the model's state, requests or answers (`/status`, `/metrics`, `/metrics/prometheus`, `/settings`,
`/mcp`, `/props`, `/slots`, `/api/requests`, `/config`) and on every `POST`. It is compared in constant time.
Set one before you open the
server to your network or put a tunnel in front of it.
- **Host check (DNS rebinding, 0.1.38).** Without an API key the server answers only requests whose `Host` is a name
it knows (`localhost`, an IP address, the address it listens on and, when it listens beyond this PC, this PC's
@@ -39,6 +40,57 @@ this PC only. The details and every setting are in [docs/DETAILS.md](docs/DETAIL
- **MCP tools** are opt-in: only the servers you put in the run config, only for requests from Strata's own page
that ask for them. They run with your user's rights, and the model decides when to call them.
- **The request monitor** (`/api-monitor`, which keeps the last prompts and answers in memory) is off unless
`"api_monitor": true` is set.
`"api_monitor": true` is set. Apart from it, the server keeps in its process memory the text and tokens of its
last few prompts, whatever `"api_monitor"` says, to encode the next turn faster; no endpoint shows them, and they
are gone when the server stops.
## The desktop app and the Terminal
Strata comes in three parts that share one engine: the **desktop app**, the **Terminal** (`strata`) and the **engine** that runs
the model. The section above is about the engine's HTTP server. This is what the other two add.
- **The app's host listens on this PC only.** The app serves its page from a local host process on `127.0.0.1` (port 8791 unless
changed). A request to its `/api/` endpoints that carries a browser `Origin` from any other site gets 403. The app shell holds a
random secret (32 characters or more, compared in constant time) that is needed to quit the host or tell it which window is open,
and the host gives its own helper processes a second random secret. By default the models the app starts are served on this PC only.
- **Sharing is a separate, key-protected listener.** "Share with other devices" (and the Cloudflare tunnel) will not start until an
API key is set. It opens a different listener that answers only `/v1/models`, `/v1/chat/completions` and `/v1/messages`, never the
app's page or its host endpoints.
- **App and Terminal talk through an owner channel.** One owner process per installation starts and ends the engine, and a second
client (the Terminal) attaches to it. On macOS and Linux that is a private Unix socket in a folder only you can open. On Windows it
is a TCP listener on `127.0.0.1` that answers only requests carrying a 256-bit token kept in an owner-only folder. That is a weaker
guarantee than a socket other users cannot open: on Windows the token is what protects it, as with any loopback service that has a
token (`strata_control/owner_transport.py`). The bridge between the app shell and the host (used for the app update) is also loopback and protected by a token.
- **Starting a model and serving never download anything.** Downloads and updates are one explicit step: `strata update`, or "Update
engine" and "Update Strata" in the app's Settings. Models and saved tuning are never replaced by an update.
- **Updates come from a signed feed.** The release publishes `strata-feed.json` and `strata-feed.json.sig` (Ed25519). An installed
Strata checks the signature against the public key it carries (`strata_control/update_keys.json`), refuses a feed older than the last
one it accepted (a replayed old feed) or expired, and checks every engine package against its SHA-256, its signed manifest and the
card in your PC before it test-starts it and switches to it. Any failure keeps the engine you had. The app installer itself is signed
with a separate key (Tauri's minisign key) that the app and the Terminal both check. The private keys are kept offline by the owner.
See [packaging/UPDATES.md](packaging/UPDATES.md).
### How far to trust `curl | sh` and `irm | iex`
The Terminal installs with one line:
```sh
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
Piping a script into a shell runs whatever the server sends, so it deserves the same care as running any downloaded program.
What the script does is short and written at its top: it fetches the Terminal archive named in the release's `install-manifest.txt`,
checks its SHA-256, unpacks it into your home folder (no administrator rights, nothing outside your home), checks the release's signed feed
with the Terminal it just unpacked and removes it again if that fails, and puts `strata` on your PATH.
Those checks have a limit. The checksum comes from the same release as the archive, and the public key that checks the feed arrives inside
that archive, so for the first install all trust rests on HTTPS and on the `Niko1221/Strata` GitHub release itself. The checks catch a
damaged or half-finished download and a feed that was not signed by the release key; they do not protect you if the release has been replaced
by someone who controls it. Later updates are checked against the key already on your PC, which is a stronger position.
If that is not enough: save the script (`curl -fsSLO https://github.com/Niko1221/Strata/releases/latest/download/install.sh`), read it, and
run it from the file; or install the desktop app or build from source instead and check the code you build.
Strata has not had an outside security audit yet.
What Strata downloads, when it uses the internet, what stays on your PC and how downloads are checked is written up in [docs/SAFETY.md](docs/SAFETY.md).
rem Strata for Windows: the first run installs everything and starts the model; later runs just start it.
rem Needs only an NVIDIA or AMD graphics driver. Python is installed for your user account if it is missing (no admin needed).
setlocal
rem Python reads and writes text as UTF-8, not in the Windows code page (cp1251, cp866, ...): a folder or user name with Cyrillic or other letters stays intact (#3202).
set"PYTHONUTF8=1"
title Strata
cd /d "%~dp0"
rem A bundled Python runtime (the python\ folder of the terminal archive) is used when it is there: nothing to find,
rem install or download. SETUP.bat and UPDATE.bat run this file, so they use it too.
ifexist"python\python.exe"gotobundled
ifexist".venv\Scripts\python.exe"gotorun
call:findpy
@@ -36,6 +41,14 @@ echo Could not create the Python environment in .venv
"""Is every static-PIE program inside an AppImage still the program it was staged as? (#4337)
Why: the bundled ripgrep (`harness/node_modules/@vscode/ripgrep-linux-x64/bin/rg`) is a statically linked PIE. Such a program relocates itself at
start-up by reading its own `.dynamic` section through a fixed offset in its code. linuxdeploy, which Tauri runs over the AppDir, treats every ELF file
as a library to fix up and rewrites its RPATH; for a static PIE that moves `.dynamic` to a new segment and leaves the old place filled with a filler
byte, so the start-up code reads garbage. Reporters on #4337 see the same crash offset in every run (`rg + 0x3ffcf1`) and filler bytes (0x5A) where
`.rela.dyn` and `.dynamic` were. A static PIE whose `.dynamic` is filled does not start: tools/test_appimage_elf.py shows that on a real one.
apps/desktop/scripts/repack-appimage.sh puts the staged bytes back; this file is the check that says whether it did.
appimage_elf.py scan <folder> every static PIE under <folder> must have a sound .dynamic (exit 1 if not)
appimage_elf.py compare <staged-folder> <folder> the same files, byte for byte, plus what differs in their ELF headers
What is checked: only static PIE programs. A shared library (.so, Python's and Node's native modules, Pillow, numpy, sharp) is not one, however it is
named: it has no entry point, or a DT_SONAME and no DF_1_PIE flag, and its RPATH/RUNPATH is normal. A static PIE is an ET_DYN with no interpreter, an entry
point, and DF_1_PIE in DT_FLAGS_1; one whose .dynamic cannot be read at all (overwritten) is reported as damaged, since the overwrite takes the flags too.
Standard library only; reads ELF64 little endian (x86-64 and arm64), anything else is ignored.
out.append(f"dynamic section: {len(da)} -> {len(db)} entries"+("; an RPATH/RUNPATH was added"ifany(tin(DT_RPATH,DT_RUNPATH)fort,_indb)andnotany(tin(DT_RPATH,DT_RUNPATH)fort,_inda)else""))
returnout
def_files(folder):
forbase,dirs,namesinos.walk(folder):
dirs.sort()
fornameinsorted(names):
path=os.path.join(base,name)
ifos.path.islink(path)ornotos.path.isfile(path):
continue
yieldpath
defscan(folder):
"""{relative path: problems} for every static PIE under `folder` that has any."""
found={}
forpathin_files(folder):
try:
withopen(path,"rb")asf:
head=f.read(4)
ifhead!=b"\x7fELF":
continue
data=head+f.read()
exceptOSError:
continue
problems=check_static_pie(data)
ifproblems:
found[os.path.relpath(path,folder)]=problems
returnfound
defcompare(staged_folder,folder):
"""{relative path: sentences} for every staged ELF file that is missing or differs in `folder`."""
* --family qwen --model IQ3_XXS --data <data folder that has the model files> [--smi "nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits"]
constchat=awaitcall('/v1/chat/completions','POST',{model:instance?.apiModelId??modelId.toLowerCase(),max_tokens:400,stream:false,temperature:0,messages:[{role:'user',content:'Reply with exactly one word: pong'}]});
record('the host sees two or more graphics cards',gpus.filter(g=>g.usable!==false).length>=2,JSON.stringify(gpus.map(g=>({i:g.index,n:g.name,gb:g.vram_gb}))));
record('the model list offers the cards choice for this model',!!opt&&opt.options.length>=2,JSON.stringify(opt?.options?.map(o=>({key:o.key,cards:o.cards,verdict:o.verdict,rec:o.recommended}))));
record('a model that fits one card recommends the single largest card (the split is offered, not forced)',opt?.recommended==='single'&&opt.options.find(o=>o.key==='split'),`recommended=${opt?.recommended}`);
record(`load with ${label} starts the engine`,loaded.job.state==='done'&&instance?.status==='ready',`${loaded.job.state}${loaded.job.error??''}; instance=${instance?.status}; gpus=${JSON.stringify(instance?.gpus)}`);
record(`the instance runs on exactly the chosen card(s)`,JSON.stringify([...(instance?.gpus??[])].sort())===JSON.stringify([...cards].sort()),`want ${JSON.stringify(cards)} got ${JSON.stringify(instance?.gpus)}`);
record(`only the chosen card(s) hold the model (graphics memory in use: ${used.join(', ')} MiB)`,JSON.stringify(holding)===JSON.stringify([...cards].sort()),`holding ${JSON.stringify(holding)}`);
for(constneedof[nodeExe,join(control,'strata.py'),join(repoHost,'dist','index.html')])if(!existsSync(need)){console.error(`missing ${need} (run stage-control.py, fetch-node and "npm run build" in apps/host first)`);process.exit(2);}
check((state?.catalog?.families?.length??0)>0&&sizes>=3,'the catalog has real families and sizes',`${state?.catalog?.families?.length} families, ${sizes} sizes`);
check(!!state?.catalog?.recommendation,'the catalog has a recommendation',JSON.stringify(state?.catalog?.recommendation??null).slice(0,120));
check(Array.isArray(state?.hardware?.gpus),'the hardware lists GPUs',(state?.hardware?.gpus??[]).map(g=>`${g.name}${g.vram_gb} GB`).join('; ')||'none on this PC');
check(state?.installDir&&realpathSync.native(state.installDir).toLowerCase().startsWith(realpathSync.native(work).toLowerCase()),'models go to a writable per-user folder, not the bundle',state?.installDir);
check(plan.status===200&&!!plan.body?.plan,'the install plan for the recommended model is a plan (nothing downloaded)',plan.body?.summary?.slice(0,100)??JSON.stringify(plan.body).slice(0,150));
check(!existsSync(join(work,'Strata-data','models')),'a plan wrote no model files');
constcontrolJson=(...argv)=>{constr=control(...argv);letjson=null;try{json=JSON.parse(r.stdout.slice(r.stdout.indexOf('{')));}catch{/* not json */}return{code:r.status,json,text:r.stdout+r.stderr};};
record('fixtures',packages.length>0&&existsSync(join(dirs.models,`strata-${modelId}.json`)),`engine package(s) ${packages.join(', ')} in ${dirs.components}; run config strata-${modelId}.json in ${dirs.models}`);
}else{
installFakeEngine();
makeTokenizerAndConfig();
record('fixtures',true,`stub engine package in ${dirs.components}; run config strata-stub.json`);
record('models come from Control',native.available&&model?.ready===true&&native.packaged===true,`available=${native.available}; model ready=${model?.ready}; ${native.summary??native.error}`);
record('load through the host starts the engine under Control',loaded.job.state==='done'&&instance?.status==='ready',`job=${loaded.job.state}${loaded.job.error??''}; instance=${instance?.status}`);
record('the engine process running is the binary from the component folder',!!ran,listed[0]?listed[0].trim().slice(0,160):'no engine process under the component folder');
}else{
record('the engine that ran is the one in the component folder',ran&&ran.script.toLowerCase().startsWith(dirs.components.toLowerCase()),ran?.script??'engine never ran');
record('a second client sees the same owner and instance (ownership)',runtime.json?.state==='ready'&&!!runtime.json.owner_id&&!!runtime.json.instance_id&&runtime.json.host_pid!==host.pid,
`state=${runtime.json?.state}; owner=${String(runtime.json?.owner_id).slice(0,8)}; instance=${String(runtime.json?.instance_id).slice(0,8)}; owner host pid ${runtime.json?.host_pid} is not the app host (${host.pid})`);
?{model:instance?.apiModelId??modelId,messages:[{role:'user',content:'Reply with exactly one word: pong'}],max_tokens:200,temperature:0,reasoning_effort:'none',stream:false}
constanswer=`${message.content??''}${message.reasoning_content??''}`;// the stub emits token 97 ("a"); the server files it as reasoning
record(real?'the real model answers through the host (the answer text is checked)':'the engine answers through the host (the stub token, not an error)',
record('no engine for this PC is a notice and nothing is started',missing.job.state==='error'&&/No engine for this PC/.test(missing.job.error)&&/never downloads/.test(missing.job.error)&&!existsSync(mark),
record('clean shutdown stops the engine the app started and exits',stopping.status===202&&code===0&&final.json?.state==='stopped'&&(!ran2||!alive(ran2.pid)),`status=${stopping.status}; host exit=${code}; runtime=${final.json?.state}; engine alive=${ran2?alive(ran2.pid):'n/a'}`);
// ---- hard kill: an engine whose app host dies ends with it (the owner follows the host: runtime start --die-with-pid)
record('nothing is left to stop afterwards',/stopped|not managed/.test(JSON.stringify(cleanup.json))&&controlJson('runtime','status','--profile','app-1').json?.state==='stopped',JSON.stringify(cleanup.json).slice(0,160));
// ---- the shell itself is killed (what the desktop shell's crash or kill looks like to the host): the host leaves, and
// its engine with it, with nothing announced. Windows also has the job object; this is the path that works everywhere.
conststarted2=awaitstartHostViaShell();
if(started2.failed){record('host starts under a stand-in shell',false,'');try{started2.shell.kill('SIGKILL');}catch{/* gone */}}
try{consthid=join(dirs.components,'engine.hidden-by-flow');if(real&&existsSync(hid)&&!existsSync(join(dirs.components,'engine')))renameSync(hid,join(dirs.components,'engine'));}catch{/* reported by the run */}
<div class="onboard-welcome-copy"><h1>State of the art. At home.</h1><p>Strata runs AI on your own computer.<br>Setting it up takes a few minutes.</p>
<div class="onboard-benefits"><span>${check}Private</span><span>${check}Fast</span><span>${check}Easy to use</span></div></div>
<footer class="onboard-footer onboard-footer--welcome"><button class="st-btn st-btn--primary">Install Strata</button><small>Installs for you only. No administrator needed.</small></footer>`,
<div class="mock-body"><h2 class="mock-title">Installing Strata…</h2><p class="mock-sub">Copying the app and its engine room. You can keep working meanwhile.</p>
<footer class="onboard-footer"><span></span><span class="mock-hint">Next: choose your first model inside Strata.</span><button class="st-btn st-btn--primary" disabled>Installing…</button></footer>`,
<div class="onboard-welcome-copy"><h1>Strata is ready.</h1><p>Open it to pick your first model. Strata downloads it for you and gets it ready.</p></div>
<footer class="onboard-footer onboard-footer--welcome"><button class="st-btn st-btn--primary">Open Strata</button><small>A shortcut was added to your Start menu and desktop.</small></footer>`,
* (the app folder is <root>/app: node/, host/, control/ ...; <root>/components gets the engine, from <root>/app/engine-bundle when the app carries one)
for(leti=0;i<180&&status?.engineSetup;i++){awaitdelay(1000);status=(awaitcall('/api/control/status')).body;}// the engine that came with the app is copied on the first start
console.log(' good preview:',JSON.stringify(goodPreview.body).slice(0,500));
record('a folder of a supported model is recognised',goodPreview.status===200&&goodPreview.body?.preview?.file_set_status==='plausible'&&!!selection,`family=${selection?.family}; model=${selection?.model}; files=${goodPreview.body?.preview?.file_count}`);
if(!selection||args.only==='preview')return;
// ---- prepare (what the page's button does): install from the folder, nothing copied or downloaded
record('the catalog lists the recognised version',!!catalogSize,catalogSize?`id=${catalogSize.id}`:`no ${selection.family}/${selection.model} in the catalog`);
record('the model is ready and points at the folder, not a copy',entry?.ready===true,`model=${modelId}; ready=${entry?.ready}; dataFolder=${entry?.dataFolder}`);
record('load starts the engine from the component folder',loaded.job.state==='done'&&instance?.status==='ready',`job=${loaded.job.state}${loaded.job.error??''}; instance=${instance?.status}`);
if(instance?.status!=='ready')return;
record('one engine process is running',engineProcs().length===1,engineProcs().map(l=>l.trim().slice(0,140)).join(' | '));
// ---- a chat answer, checked
constchat=awaitfetch(base+'/api/chat/stream',{method:'POST',headers:{'content-type':'application/json'},body:JSON.stringify({sessionId:'e2e-chat',message:'Reply with exactly one word: pong',history:[],nativeModelId:instance.id,thinking:'Off',agent:false,workspaceId:null})});
try{constf=JSON.parse(line);if(f.type==='delta')text+=f.text??'';if(f.type==='done')done=f;if(f.type==='error')text+=` [error: ${f.message}]`;}catch{/* partial line */}
}
}
record('the model answers through the app (the answer text is checked)',chat.status===200&&/pong/i.test(text),`status=${chat.status}; answer=${JSON.stringify(text.slice(0,120))}; tokensOut=${done?.tokensOut}; tps=${done?.tps}`);
# every checksum in the README must be the checksum of a file that is here, and a line that names a file and carries a checksum must carry that file's checksum
hashes="";bad=0;seen=0
for d in "${DIRS[@]}";dofor f in "$d"/*;do[ -f "$f"]&& grep -qF "$(basename "$f")""$VERIFY"&&hashes="$hashes$(sha_of "$f")";done;done# only files the README names are hashed (a folder may hold big unrelated files)
f="$(find_file "$name")"||{echo"make-tester-readme: no file named $name in: ${DIRS[*]} (a README must not carry a hash it did not compute)" >&2;exit 1;}
* Also: --silent <url> (a server whose model list says nothing about pictures: ON by default), --agent (the same picture through the agent; needs a Harness)
record('the engine component carries the picture reader and the host knows it (visionReader)',reader&&native.visionReader===true,`component=${reader} host=${native.visionReader}`);
console.log(` answer from the API model: ${JSON.stringify((sent.text??sent.message??'').slice(0,300))}`);
record('the picture goes to the API model and is answered',sent.status===200&&!sent.error&&(sent.text??'').trim().length>0&&names((sent.text??'').toLowerCase()),`status=${sent.status} err=${sent.error??''}`);
if(args['text-only']){
awaitsaveApi('Text only server',args['text-only'],[{id:'plain',name:'Plain model'}]);
constt=awaitreadsAfter('Text only server','plain',false);
record('a server without a picture reader is known not to read pictures',t.m?.reads===false,JSON.stringify(t.m));
record('a picture for it is refused in plain words that say how to fix it',refused.status===409&&/Read pictures with this model/.test(refused.message??'')&&/Settings/.test(refused.message??''),`${refused.status}`);
constticked=awaitsaveApi('Text only server',args['text-only'],[{id:'plain',name:'Plain model',vision:true}]);
constafter=(awaitcall('/api/model-providers')).body.connections.find(x=>x.name==='Text only server');
record('ticking "Read pictures with this model" is saved and makes the model read pictures',after?.models[0]?.vision===true&&after?.models[0]?.reads===true,JSON.stringify(after?.models[0]));
record('with the tick the picture is sent on (the server itself decides what to do with it)',through.status===200&&(through.error??'')!==''?/turn off/.test(through.error):through.status===200,`${through.status}${through.message??through.error??''}`);
record('a model switched OFF refuses pictures and says where to turn them on again',notRead.status===409&&/not to read pictures/.test(notRead.message??'')&&/Settings/.test(notRead.message??''),`${notRead.status}${notRead.message??''}`);
console.log(` other style: ${JSON.stringify(style.message)}`);
record('an API of another style says plainly why pictures stay in the preview',style.status===409&&/OpenAI/.test(style.message??''),`${style.status}`);
record('the model is prepared with its picture reader',!!model?.visionPrepared,JSON.stringify({ready:model?.ready,visionPrepared:model?.visionPrepared}));
record('the model loads with the picture reader',loaded.job.state==='done'&&instance?.status==='ready'&&instance?.vision===true,`${loaded.job.state}${loaded.job.error??''} vision=${instance?.vision}`);
record('the model loaded with pictures ON without being asked per load (default)',instance?.vision===true,'');
console.log(` answer from the local model (${Math.round((Date.now()-t0)/1000)} s): ${JSON.stringify((local.text??local.message??'').slice(0,300))}`);
constlow=(local.text??'').toLowerCase();
record('the local model answers about the picture (names the shapes)',local.status===200&&/red|circle/.test(low)&&/blue|square/.test(low)&&/green|triangle/.test(low),`status=${local.status} err=${local.error??''}`);
record('the local model reads STRATA 42',low.includes('42'));
// the same picture through the agent (the Harness route must declare image input: the "does not declare image input" error of the tester)
if(args.agent){// needs a Harness beside the host (the installed layout has one; a bare test box does not)
record('a picture for the OFF model is refused with a next step',refusedLocal.status===409&&/Read pictures/.test(refusedLocal.message??'')&&/Models/.test(refusedLocal.message??''),`${refusedLocal.status}`);
# What was put back must be sound before it is packed: a damaged static PIE (the bundled ripgrep) stops the pack here, not on a person's computer.
elf_and_rg_check ||{echo"the restored AppDir failed the static PIE / ripgrep check; not packing" >&2;exit 1;}
# The launcher records the environment the AppImage was started from, before its own hooks add the AppImage's libraries and GTK/GIO
# settings (LD_LIBRARY_PATH, GTK_*, XDG_DATA_DIRS ...). The host gives it back to every program that is not Strata's own (zenity,
# kdialog, xdg-open, the assistant's shell): apps/host/server/external-env.mjs. Without it a system program loads the AppImage's libssl
# and dies ("OPENSSL_3.2.0 not found"). The line goes right after `this_dir=...`, ahead of every hook.
if ! grep -q STRATA_ORIGINAL_ENV "$WORK/AppDir/AppRun";then
sed -i '/^this_dir=/a # Strata: the environment this AppImage was started from (given back to programs that are not Strata'"'"'s own)\nexport STRATA_ORIGINAL_ENV="$(env -0 | base64 -w0)"'"$WORK/AppDir/AppRun"
grep -q STRATA_ORIGINAL_ENV "$WORK/AppDir/AppRun"||{echo"could not add the environment record to AppRun" >&2;exit 1;}
check(order.length===2&&order[0].startsWith('SPAN')&&order[1].startsWith('BUTTON')&&Number(order[0].split(':')[1])<Number(order[1].split(':')[1]),'the time left sits left of Stop',order.join(' '));
check(samples.length>=3&&samples.every((v,i)=>i===0||v>=samples[i-1]),'one combined count that only goes up',samples.map(v=>v.toFixed(1)).join(', '));
check(overflow<=2,'the Models tab fits the first view without scrolling',String(overflow)+' px over');
check(awaitpage.locator('.family-switch').count()===1,'the model family switch is in the navbar');
check(awaitpage.locator('h1').filter({hasText:/^Model$/}).count()===0,'and the navbar does not say "Model"');
check(/IQ3_S|High quality/.test(summary??''),'Models selects the model that is on this PC, not the recommendation',(summary??'').replace(/\s+/g,' ').trim());
check(!/IQ3_XXS|Balanced/.test(summary??''),'the recommendation (IQ3_XXS) is not what is selected');
check(inst?.resourceProfile==='work','the running model reports the profile it was started with',String(inst?.resourceProfile));
awaitdelay(1500);
check(awaitwork().getAttribute('aria-pressed')==='true'&&awaitfull().getAttribute('aria-pressed')==='false','after the load the page still shows Leave 20%, not Full performance');
check(!/Download and prepare|Download . load|approximately|[0-9] GB/i.test(text),'for a downloaded model the pop-up has no download wording or size',text.replace(/\s+/g,' ').slice(0,600));
check(/Apply/.test(text),'it says what happens: apply settings and load');
}elseconsole.log('info the settings were the saved ones, so loading needed no pop-up at all');
// ------------------------------------------------------------------------------------------------ version list, This computer, Tune, a task
for(constschemeof['light','dark'])awaitscenario(scheme==='light'?'ui2':'ui2d',{installed:true,env:{FAKE_TOKEN_DELAY:'0.02',FAKE_REPLY:'I am thinking about it. </think> '+'Here is the answer, word by word. '.repeat(12),STRATA_TUNE_NO_QUIET_CHECK:'1'}},async({base,api,browser,shot})=>{
check(/This computer/.test(pcText)&&/(Fits fully in memory|read from the SSD)/.test(pcText),`${scheme}: This computer says in one line whether it fits`,pcText.slice(0,200));
check(awaitpc.locator('.pc-meter').count()>=1,`${scheme}: it shows the memory bars`);
check(awaitpc.locator('.chip, .st-pill, [class*="badge"]').count()===0,`${scheme}: no chips or badges in it`);
for(constjobofjson?.jobs??[])if(job.kind==='autotune'&&fakePhase==='done'){job.state='done';job.progress=null;job.result={outcome:'improved',gain_pct:18,base:{decode_tok_s:41.2},chosen:{decode_tok_s:48.6},preset:{id:'fast',name:'Fast · 64K'},answers:{class:'exact',fields:[]}};}elseif(job.kind==='autotune'&&job.state==='running')job.progress={stage:'rounds',text:null,minutesLeft:9,runsDone:3,runsPlanned:6,elapsedS:300,step:'Trying: more experts kept in memory (round 1 of 2)',speeds:[{label:'base',tok_s:41.2},{label:'Fewer helper threads',tok_s:40.1},{label:'More experts kept in memory',tok_s:46.8}]};
check(sawRun,`${scheme}: a step is shown as running`);
check(sawSpeed,`${scheme}: the live speed shows a number while it runs`);
if(sawSpeed){fakePhase='done';awaitpage.locator('.tune-outcome').waitFor({timeout:15000}).catch(()=>{});check(/18% faster/.test((awaitpage.locator('.tune-outcome__body h3').textContent().catch(()=>''))??''),`${scheme}: the result card shows the gain as its headline`);check(/Fast/.test((awaitpage.locator('.tune-outcome__facts').textContent().catch(()=>''))??''),`${scheme}: and what it was saved as`);}
// a task: the opened task is skimmable, with pause / delete / Edit / Run now
constmade=awaitapi('/api/tasks',{method:'POST',headers:{'content-type':'application/json'},body:JSON.stringify({title:'Open Downloads folder',prompt:'Open my Downloads folder and tell me what is new.',frequency:'weekdays',at:'08:00',weekday:1,timeZone:'Europe/Ljubljana',mode:'manual',enabled:true})});
constid=made.body?.id??made.body?.task?.id;
check(!!id,`${scheme}: a task is created`,JSON.stringify(made.body).slice(0,120));
check(awaitpage.getByRole('button',{name:/^Pause /}).count()===1&&awaitpage.getByRole('button',{name:/^Delete /}).count()>=1&&awaitpage.getByRole('button',{name:/Run now/}).count()>=1,`${scheme}: pause, delete, Edit and Run now are in the header`);
check(awaitpage.locator('.main__back').count()===0&&awaitpage.locator('.main__title-line .task-state').count()===0,`${scheme}: nothing sits on the left of an opened task's navbar`);
check(/Weekdays/.test(factsText)&&/Schedule/i.test(factsText)&&/Last result/i.test(factsText),`${scheme}: schedule, what it may do and the last result are at the top`,factsText.slice(0,160));
awaitscenario('tune',{installed:true,env:{FAKE_TOKEN_DELAY:'0.02',FAKE_REPLY:['I am thinking about it.','</think>','','Here is a list:','','- one','- two','','## Heading','','Some `code` and **bold** text.','','| a | b |','|---|---|','| 1 | 2 |','','```','code line','```',''].join(String.fromCharCode(10))}},async({base,api,browser,dirs,shot})=>{
constpage=awaitnewPage(browser,base);
awaitclickText(page,/get started/i);
awaitpage.getByText(/Choose a version/i).waitFor({timeout:30000});
check(tps>15&&tps<120,'the speed shown is the decode speed (the engine paced at ~50 tokens a second)',speedText.replace(/\s+/g,' ').slice(0,160));
check(/32\.8K|32K|32,768|32768/.test(contextText)&&!/128K/.test(contextText),'the context shown is the one the model runs with (32K), not 128K',contextText.replace(/\s+/g,' ').slice(0,160));
check(sizes[0][1]===12&&sizes[1][1]===14&&sizes[2][1]===16&&sizes[3][1]===22&&sizes[0][2]===12&&sizes[3][2]===22,'the reading size changes the text size of the conversation',JSON.stringify(sizes));
check(/Delete this version\?/.test(dialog)&&/Remove 83\.6 GB of downloaded files\. You can download this version again\./.test(dialog),'the confirm dialog uses the existing words and the real size',dialog.replace(/\s+/g,' ').slice(0,600));
check(/running model will be unloaded/i.test(dialog),'and says the loaded model is unloaded first');
console.log('info chat opened in',open,'ms; long tasks',JSON.stringify(long),'requests',JSON.stringify(requests.filter(r=>r[1]>100)),'all',requests.length);
check(open<1500&&Math.max(0,...long)<300,'Models to Chat opens at once, with no long block',open+' ms, longest task '+Math.max(0,...long)+' ms');
check(awaitpage.getByRole('button',{name:/^Tune model$/}).count()===1,'the Models tab has the Tune model button (the real tuner, not the old short benchmark)');
awaitpage.getByRole('button',{name:/Other local models/i}).click().catch(async()=>{awaitclickText(page,/Manage models/i);awaitclickText(page,/Use a local GGUF/i);});
awaitpage.getByText(/Folder with the GGUF files/i).first().waitFor({timeout:15000}).catch(()=>{});
awaitshot(page,'2-local-gguf');
check(awaitpage.getByText(/Folder with the GGUF files/i).count()>0,'the local GGUF page opens in the same design as Models');
awaitpage.getByPlaceholder(/Full path to the folder/i).fill(join(dirs.install,'does-not-exist'));
check(/°C/.test(mon),'the Monitor shows GPU data with no model loaded',(mon.match(/.{0,30}°C.{0,20}/)?.[0]??'no temperature text').replace(/\s+/g,' '));
for(leti=0;i<40;i++){messages.push({id:'u'+i,role:'user',content:'question '+i,at:now},{id:'a'+i,role:'assistant',content:'answer number '+i+' with some text to fill the line and more of it.',at:now,tokens:20,inputTokens:5});}
check(/server_exited/.test(details)&&/exited before/.test(details),'Details holds the code and the reason the server logged',details.replace(/\s+/g,' ').slice(0,120));
constbadOpen=awaitnoScroll(page);
check(badOpen.length===0,'and the pop-up still does not scroll with Details open',badOpen.join('; '));
check(saved.prefs.contentFontSize===19&&saved.prefs.language==='de','text size and language survive a restart on the same port',JSON.stringify(saved.prefs));
record('the model is installed with the picture reader prepared (visionPrepared)',!!model?.visionPrepared,JSON.stringify({ready:model?.ready,visionPrepared:model?.visionPrepared,images:model?.images}));
record('load starts the engine with the picture reader',loaded.job.state==='done'&&instance?.status==='ready',`${loaded.job.state}${loaded.job.error??''}; instance=${instance?.status}; vision=${instance?.vision}`);
record('the host says the loaded model reads pictures',!!instance?.vision,`vision=${instance?.vision}`);
// ---- a real picture through the host's own /v1 endpoint
description="Strata Desktop — local AI, tuned to one model."
edition="2021"
rust-version="1.77"
license="MIT"
# Linux: the drop of a file on the window is read straight from the GTK drag signals (see hook_linux_drops in main.rs); the versions are the ones Tauri itself uses.
[target.'cfg(target_os = "linux")'.dependencies]
gtk="0.18"
webkit2gtk="2.0"
# Windows: the shell listens for the page's renderer failing (WebView2 ProcessFailed, see hook_renderer_failures in main.rs); these are the versions wry itself uses.
{"core":{"default_permission":{"identifier":"default","description":"Default core plugins set.","permissions":["core:path:default","core:event:default","core:window:default","core:webview:default","core:app:default","core:image:default","core:resources:default","core:menu:default","core:tray:default"]},"permissions":{},"permission_sets":{},"global_scope_schema":null},"core:app":{"default_permission":{"identifier":"default","description":"Default permissions for the plugin.","permissions":["allow-version","allow-name","allow-tauri-version","allow-identifier","allow-bundle-type","allow-register-listener","allow-remove-listener","allow-supports-multiple-windows"]},"permissions":{"allow-app-hide":{"identifier":"allow-app-hide","description":"Enables the app_hide command without any pre-configured scope.","commands":{"allow":["app_hide"],"deny":[]}},"allow-app-show":{"identifier":"allow-app-show","description":"Enables the app_show command without any pre-configured scope.","commands":{"allow":["app_show"],"deny":[]}},"allow-bundle-type":{"identifier":"allow-bundle-type","description":"Enables the bundle_type command without any pre-configured scope.","commands":{"allow":["bundle_type"],"deny":[]}},"allow-default-window-icon":{"identifier":"allow-default-window-icon","description":"Enables the default_window_icon command without any pre-configured scope.","commands":{"allow":["default_window_icon"],"deny":[]}},"allow-exit":{"identifier":"allow-exit","description":"Enables the exit command without any pre-configured scope.","commands":{"allow":["exit"],"deny":[]}},"allow-fetch-data-store-identifiers":{"identifier":"allow-fetch-data-store-identifiers","description":"Enables the fetch_data_store_identifiers command without any pre-configured scope.","commands":{"allow":["fetch_data_store_identifiers"],"deny":[]}},"allow-identifier":{"identifier":"allow-identifier","description":"Enables the identifier command without any pre-configured scope.","commands":{"allow":["identifier"],"deny":[]}},"allow-name":{"identifier":"allow-name","description":"Enables the name command without any pre-configured scope.","commands":{"allow":["name"],"deny":[]}},"allow-register-listener":{"identifier":"allow-register-listener","description":"Enables the register_listener command without any pre-configured scope.","commands":{"allow":["register_listener"],"deny":[]}},"allow-remove-data-store":{"identifier":"allow-remove-data-store","description":"Enables the remove_data_store command without any pre-configured scope.","commands":{"allow":["remove_data_store"],"deny":[]}},"allow-remove-listener":{"identifier":"allow-remove-listener","description":"Enables the remove_listener command without any pre-configured scope.","commands":{"allow":["remove_listener"],"deny":[]}},"allow-set-app-theme":{"identifier":"allow-set-app-theme","description":"Enables the set_app_theme command without any pre-configured scope.","commands":{"allow":["set_app_theme"],"deny":[]}},"allow-set-dock-visibility":{"identifier":"allow-set-dock-visibility","description":"Enables the set_dock_visibility command without any pre-configured scope.","commands":{"allow":["set_dock_visibility"],"deny":[]}},"allow-supports-multiple-windows":{"identifier":"allow-supports-multiple-windows","description":"Enables the supports_multiple_windows command without any pre-configured scope.","commands":{"allow":["supports_multiple_windows"],"deny":[]}},"allow-tauri-version":{"identifier":"allow-tauri-version","description":"Enables the tauri_version command without any pre-configured scope.","commands":{"allow":["tauri_version"],"deny":[]}},"allow-version":{"identifier":"allow-version","description":"Enables the version command without any pre-configured scope.","commands":{"allow":["version"],"deny":[]}},"deny-app-hide":{"identifier":"deny-app-hide","description":"Denies the app_hide command without any pre-configured scope.","commands":{"allow":[],"deny":["app_hide"]}},"deny-app-show":{"identifier":"deny-app-show","description":"Denies the app_show command without any pre-configured scope.","commands":{"allow":[],"deny":["app_show"]}},"deny-bundle-type":{"identifier":"deny-bundle-type","description":"Denies the bundle_type command without any pre-configured scope.","commands":{"allow":[],"deny":["bundle_type"]}},"deny-default-window-icon":{"identifier":"deny-default-window-icon","description":"Denies the default_window_icon command without any pre-configured scope.","commands":{"allow":[],"deny":["default_window_icon"]}},"deny-exit":{"identifier":"deny-exit","description":"Denies the exit command without any pre-configured scope.","commands":{"allow":[],"deny":["exit"]}},"deny-fetch-data-store-identifiers":{"identifier":"deny-fetch-data-store-identifiers","description":"Denies the fetch_data_store_identifiers command without any pre-configured scope.","commands":{"allow":[],"deny":["fetch_data_store_identifiers"]}},"deny-identifier":{"identifier":"deny-identifier","description":"DenLine truncated
Loaded 100 of 3967 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Niko1221
