The VRAM reuse stays, but bytes_needed_impl counts bo again by default, so the auto chunk and the borrowed cache slots
are those of 0.1.40.3 and the prompt residuals stay byte-identical (on the RTX 3060 the planner otherwise picked a
6656-token chunk instead of 6400). STRATA_EMB_REUSE_ACCOUNT=1 lets the planner use the saved bytes (+3.9% prompt there);
documented in docs/DETAILS.md.
The previous change skipped the DRM-fdinfo subtraction whenever the driver reported usage. On a single Arc Pro B70 that
made the expert cache 12168 slots instead of 11338 (23.09 vs 21.52 GiB), so a one-GPU run was not unchanged. The
subtraction is now skipped only when the process sees more than one GPU (where it double-counts the other card) and the
driver does not report the whole card as free.
The bound of the device's spins on host flags was a build constant: 20,000 reads for a bmg AOT build, 2,000,000
for anything else. A B70 built without AOT (SPIR-V, JIT: the Windows/OpenCL build, or a Linux build with no AOT
target) got the A750-sized bound. The issue reports a 37% drop in decode between 0.1.40.2 and 0.1.40.3. On a Linux B70 JIT build the two bounds decode
alike (--spec 2, 3 pairs: 19.4 vs 19.5 tok/s), so this makes the bound independent of the AOT flag but is not shown to be that drop's cause.
spin_max(queue) decides per device: 20,000 under the xe driver (the B-series on Linux), 2,000,000 elsewhere (the i915
A-series needs it for the host's per-layer expert work; a Windows/OpenCL B-series build measured faster with the long bound).
STRATA_SPIN_MAX=<reads> overrides it at run time; -DSTRATA_SYCL_SPIN_MAX=<reads> still fixes one bound for the
whole build. The three waiting kernels take the bound as an argument.
- generate.cpp: with a layer split the host mirror walked the later stages' layers too (#1054, fixed in #1111 but
lost in the re-migration): 20 GiB of pinned RAM for experts the other card holds, or one 39 GiB pinned allocation
that fails. Both loops stop at the first stage's last layer.
- expert_cache.cpp: device_free_bytes() subtracted the DRM bytes of every card of the process from the card it asked
about, so on a layer split the second card was charged the first one's cache and its expert cache would not open.
The fallback is for drivers that report the whole card as free (Arc A750/i915); it now runs only then.
Reported by crobe201 with both diffs; MaineTim confirmed the second on 2x B60.
Co-authored-by: crobe201 <[email protected]>
The three sites fixed by the previous commit stored a clamped (d, sum) but still
quantized against the unclamped d, so an overflowing block stored int8 values
that no longer matched its stored scale. They now use q8_1_finite(d) and
q8_1_quant like native_quantize_q8_1_kernel, the swiglu quantizer and the CPU
pool. A block whose amax is finite and below 8.3M is stored bit for bit as
before.
Co-authored-by: Matthew Hines <[email protected]>
Three kernels still emit the (d, sum) pair of a q8_1 activation block
with a raw make_half2(d, sum), bypassing the finite helper the rest of
the tree uses (q8_1_finite.hpp, from 19de7f4 / f6601bf):
fused_gr.cu gr_q8_tail - the fused gate-reduce tail (QFUSE)
verify_kernels.cu gdn_q8_1_store - the batched GDN verify store (QFUSE)
iq_kernels.cu s26_swiglu_q8_1_kernel - the S26 swiglu path
A block whose amax overflows (or that reads a non-finite input) emits
d = inf/NaN; fp16 makes the pair NaN (or the pathological (0,0)). Nothing
downstream tests for it: a NaN scale poisons the fp16 dot of the quantized
activation and every sum downstream of it, and a fully non-finite logit
row ends at the sampler's all-NaN fallback - vocabulary id 0, the '!'
face of the degeneration reported in #606. Same defect class already
fixed at the other emit sites; these three were missed.
q8_1_ds() clamps both halves into the fp16-finite range (finite d,
non-negative finite sum - the sum is a mean of |x|), producing for the
GPU paths the same bits the CPU pool and the clamped GPU sites produce
for the same block. A clamp at the producer, content-blind: nothing
downstream of it is model-text-aware.
Criteria and the wider carried-state RCA: docs/Q8_1_FINITE_DS.md and
docs/DEGENERATION_RCA.md.
2e4ddf6 (the #904 port) removed __restrict__ from the activation pointers of
native_quantize_q8_1_kernel, bf16_f32_mmvf_multi_kernel and
native_mmvq_multi_kernel on every CUDA architecture, and turned on the weight
prefetch before pdl_wait(). Both only matter for PDL (sm_90+). On sm_61 the
missing restrict loses the read-only data path (LDG.CI) and decode halves
(36.4 -> 17.7 tok/s on a Tesla P40, 32-36 -> 17.7-18.6 on a GTX 1080 + 1070).
STRATA_PDL_RESTRICT is __restrict__ and kPdlPrefetch false again for
__CUDA_ARCH__ < 700 in the device pass. sm_70 and newer compile to the same
SASS as before (cuobjdump -sass of native_mmvq.cu and native_bf16.cu at sm_70,
86, 90 and 120 is identical); the host pass is unchanged.
Reported and bisected by paulhothersall and lineape.
2e4ddf6 (the #904 port) removed __restrict__ from the activation pointers of
native_quantize_q8_1_kernel, bf16_f32_mmvf_multi_kernel and
native_mmvq_multi_kernel on every CUDA architecture, and turned on the weight
prefetch before pdl_wait(). Both only matter for PDL (sm_90+). On sm_61 the
missing restrict loses the read-only data path (LDG.CI) and decode halves
(36.4 -> 17.7 tok/s on a Tesla P40, 32-36 -> 17.7-18.6 on a GTX 1080 + 1070).
STRATA_PDL_RESTRICT is __restrict__ and kPdlPrefetch false again for
__CUDA_ARCH__ < 700 in the device pass. sm_70 and newer compile to the same
SASS as before (cuobjdump -sass of native_mmvq.cu and native_bf16.cu at sm_70,
86, 90 and 120 is identical); the host pass is unchanged.
Reported and bisected by paulhothersall and lineape.
Found while running it against a live engine: the line said 0.35 as 48.7 on round 3 while the three draws were 57.0, 48.7 and 35.5, because it printed the running median. Reading a sweep's drift is the whole point of the line, so it has to be the draw. The medians and the ranges are still in the line below it.
Phase 1 measures each PCIe share once and takes the maximum, with the three prompts of that rate all at the one
setting. Neighbours cannot be ordered that way: on this box 0.75 through 0.95 come out at 72.7, 69.0, 71.0 and
71.9, which is 0.2% apart, while the same settings moved 8-28% between engine starts. A value that was worth a
real gain can lose its single sample and never be measured again, because phase 1's winner is the share phase 2
then sweeps spec_min_p at, and the confirm step only puts that one resulting pair against the default.
sweep() measures every value SWEEP_ROUNDS times inside the one engine start, one round visiting all of them and
every other round in reverse order, and the winners come from the medians. Drift inside a sweep then lands on
every value the same number of times, and one bad sample cannot decide. The same function runs phase 2, and the
confirm step uses the same round count instead of its own.
Cost: the sweep multiplies by SWEEP_ROUNDS, 3 by default. A rate is three prompts of 128 tokens, so on this box
(six values, 95-100 tok/s) phase 1 goes from about 40 s to about 120 s, on a run that already loads the model more
than once. A machine whose two leaders stay inside each other's noise can raise the constant.
Not in this change: phase 4 needs a restart per worker count and stays sequential, so a count is still taken from
one measurement each.
Tests: python -m unittest tools.test_calibrate (the stand-in engine is a pure function of the tune keys, so the
medians and the winners are the same as before); tools.test_setup_config and tools.test_setup_golden
PCIE_FRACS stops at 0.75, so setup --calibrate never measures the value that takes the CPU pool out of the decode
window completely. 1.0 is a separate operating point rather than 0.75 carried further: the share is applied as
m = (nmiss * pcie_num) >> 8, so at any value below 1.0 at least one miss still goes to the CPU pool, and only at
1.0 does every non-resident expert come over PCIe. A CPU expert sits on the layer's critical path while the GPU
waits for it, and on a slow CPU that is the whole difference.
On a Core Ultra 9 290HX Plus with an RTX 5090 Laptop on PCIe 5.0 x16, 0.55 gives 51.1 tok/s against 71.3 at 1.0,
with cpu_exp falling from 1.06 to 0.00 per layer-window and the window time falling by the 11.3 ms of PCIe waiting
plus the CPU's expert time. A sequential sweep puts 1.0 at 78.8 against 72.7 for 0.75 and 51.5 for 0.35.
0.9 is added beside 1.0 so a machine where 1.0 is too much gets an interior point. pick() keeps the default unless a
candidate beats it by MIN_GAIN, so this cannot make a calibration worse. The cost is one rate per value, about 8 s
here.
Tests: python -m unittest tools.test_calibrate; tools.test_setup_config and tools.test_setup_golden
With a second toolkit beside the distribution's one, CMake compiled with the chosen nvcc but took cudart and cuBLAS from /usr (Ubuntu 24.04, STRATA_NVCC=/opt/cuda-13.0/bin/nvcc: the engine needed libcudart.so.12 while built with the 13.0 headers). Pass CUDAToolkit_ROOT for the engine and the CUDA image encoder; the distribution's own /usr/bin/nvcc keeps CMake's search as before.
Two gfx906 build failures on v0.1.40.2. Both are one-line gates that know
about the wave32 HIP backend but not about STRATA_HIP_GFX906 -- the same
class as #808 and #1083. Neither is fixed on main (e8ca9afd03).
1) src/prefill/prefill.cpp:399 builds an event with
cudaEventDisableTiming | cudaEventBlockingSync, but the gfx906 compat
header include/strata/platform/hip_compat/strata_hip.h maps only
cudaEventDisableTiming and cudaEventDefault. The wave32 header
include/strata/hip_compat/cuda_runtime.h:39 has the mapping.
-> add #define cudaEventBlockingSync hipEventBlockingSync there.
A diff of the whole set of cuda* defines between the two headers lists
one other name the gfx906 header lacks, cudaDevAttrIntegrated, and that
one is already gated upstream (#if defined(__linux__) &&
!defined(STRATA_HIP_GFX906) in src/core/expert_cache.cpp:31), so it is
deliberately not touched here.
2) Link error: undefined symbol
void mul_mat_q_case<(ggml_type)14>(ggml_backend_cuda_context&, mmq_args const&, ihipStream_t*)
(14 = GGML_TYPE_Q6_K).
src/prefill/moe_mmq.cu:219 calls it under
#if defined(__HIPCC__) || defined(STRATA_Q6K_EXPERTS), so any HIP
compilation references it; mmq.cuh declares the case as an extern
template, so the definition must come from
template-instances/mmq-instance-q6_k.cu. The CUDA branch of
CMakeLists.txt adds that file only under STRATA_Q6K_EXPERTS.
STRATA_HIP_GFX906=ON forces STRATA_ENABLE_CUDA=ON, so gfx906 takes the
CUDA branch while its .cu files are compiled as HIP.
-> widen the gate: STRATA_Q6K_EXPERTS OR STRATA_HIP_GFX906.
This adds one MMQ instance to the static library. It does not enable
STRATA_Q6K_EXPERTS (the grouped Q6_K expert kernels that allocate extra
VRAM), and STRATA_HIP_GFX906 is only ever ON in the gfx906 build, so
CUDA and wave32 builds are untouched.
Verified on a single Instinct MI50 32 GB (gfx906, wave64), ROCm/HIP 7.14,
-DSTRATA_HIP_GFX906=ON -DCMAKE_HIP_ARCHITECTURES=gfx906
-DSTRATA_MMQ_KQUANTS=ON:
- the engine builds and serves UD-IQ4_XS (Qwen3.8-Flash-Next) end to end:
/v1/models, chat completions, tool calls, MTP drafts accepted;
- nm -C shows mul_mat_q_case<(ggml_type)14> in the binary (13 instances:
7 8 12 13 14 16 17 18 20 21 22 23 42);
- change 2 is load-bearing: removing mmq-instance-q6_k.cu.o from
libstrata_mmq.a and relinking with the same link command line fails with
exactly the undefined symbol above (the archive holds one U reference
and that one W definition), while the untouched archive relinks cleanly.
Two variants of the same sweep (code-explanation prompts and Russian prose) on
one PC and one server config, engine 0.1.40.3 built from source for sm_89.
Each folder carries the 0.1.40.2 run, the 0.1.40 baseline and an
identical-binary control run so the noise band is in the report.
Result: no measurable change from 0.1.40.2 on this machine.
Same laptop, model files, configuration and scripts as the 0.1.40.1
entry (2026-10-06-community-rtx-5090-laptop, #1263); only the Strata
version changed. Raw data for 0.1.40.3, medians for 0.1.40.2 in the
README. Decode and prompt throughput unchanged within run-to-run
variation, all needles found, coding check 10/10 on 0.1.40.3. A 0/10
coding check on 0.1.40.2 is traced to a greedy reasoning loop that
occurs in 0.1.40.1 as well (re-runs documented). Results only, no
engine changes.
Coder IQ1_M, one RTX 5080 16 GB, the repository's own Dockerfile and entrypoint
on a Windows host (Docker Desktop 4.84.0 / WSL2), two context limits on the same
loaded model: 131,072 and 65,536.
- Speed: three runs at each of 4,096 / 8,192 / 16,384 / 32,768 / 65,536 /
130,560 prompt tokens (128K limit) and at 4,096 / 8,192 / 16,384 / 32,768 /
65,024 (64K limit), greedy, reasoning off, 256-token output cap, zero reused
prompt tokens in all 33 runs.
- Correctness: tools/needle_bench.py 9 of 9 (32k/64k/128k) at the 128K limit and
6 of 6 at the 64K limit, plus a vision check with the encoder on the CPU.
- Provenance: all three model files SHA-256-verified against the repository's
published LFS oids at the pinned revision; MTP manifest and inventory included.
- Environment: Docker Desktop's default WSL2 memory cap makes setup.py read ~31 GB
of a 64 GB host and silently select the low-RAM mode; the report documents the
.wslconfig change used instead. Also documents the 8-token headroom a request
needs (prompt + max_tokens + 8 <= max_context) and the engine's own VRAM warning
at a 128K context on a 16 GB card.
Results-only; no engine changes.
Nine files following docs/COMMUNITY_BENCHMARKS.md. Engine v0.1.40.3
(tag d5ea713) built from source for sm_75, IQ3_S at the model native
262,144-token window, with --vision and --reasoning-budget-tokens 12000.
Three runs each at 4,096 / 32,768 / 128,000 prompt tokens: decode
70.9 / 77.3 / 71.1 tok/s, prompt 868.9 / 1548.9 / 1626.9 tok/s, medians
in the index and full ranges in the report. needle_bench 6/6 at 32k and
128k. Records that NVLink is present but unused by the engine, so a bridge
on consumer boards is not worth buying.
The flag acts only on --expert-cache-device1..3; a plain layer split got a no-op flag that the docs listed as a reason
for the pipeline and the async tier to switch off. Built on f-141-quick's #1352 pipeline-windows rule (same hunk).
MULTI_GPU.md: the two combine (only STRATA_PIPELINE_ADAPT_ASYNC=0 turns async off); adds the 2x3090 table.
Co-authored-by: adambenhassen <[email protected]>
The function name was everything between `<function=` and `>`, untrimmed. A model that writes a space after the `=`
(seen from Qwen3.8-Flash-Next IQ2_XS in Zed's agent panel) got:
- inside the thinking: the name " capture" matched no declared tool, so the #804 rescue kept the call as reasoning
text; the reply ended with finish_reason "stop" and no tool_calls, and the agent stopped mid-task;
- after the thinking: a call named " capture", which the client does not know, its arguments typed without the
tool's schema.
The name is now stripped where the call is parsed (parse_tool_call), where the #804 rescue checks it against the
declared tools, and where a finished call looks up its schema (_finish_call). A name with surrounding whitespace is
never meaningful, so no text becomes a call that was not one: the #804/#1058 rules still decide whether a call
inside the thinking is an act.
serve/test_tool_name_space.py: both places, streamed and not, fed in 3-character pieces; fails 4/4 before, passes
after. test_reasoning_rescue, test_reasoning_tools and test_frontend unchanged and passing.
NTFS serves unbuffered reads of a mapped file one at a time (WD SN580: 2 MiB reads at QD32
2.35 GB/s mapped vs 3.57 GB/s unmapped). With the file tier unbuffered (--resident-budget-gib
or STRATA_UNBUFFERED_LOAD=1) the mapped view is only a fallback after startup, so it is closed
(STRATA_KEEP_MAPPING=1 keeps it). The prompt path's stager claims runs of consecutive jobs and
reads them with one read_direct batch (STRATA_STAGER_BATCH, default 8 for file-backed blobs).
RTX 5070 Ti, 32 GB RAM, IQ3_XXS, 47.6K prompt, --resident-budget-gib 18 --prefill auto:32768:
1260 -> 1370 tok/s (batch 8) -> 1669 tok/s (view closed).
Ported from #833 onto the rewritten main. Two fixes on the way: unmapped_ is declared on every
platform (mapped_blob reads it outside the Windows block, so Linux did not compile), and
copy_blobs asks resident_blob, which follows the exchange rotation (STRATA_EXCHANGE_ROTATE).
The fast kernels build for CUDA; sm_70 defaults to them, other CUDA cards keep
the plain ones. gr_up_fast_kernel ends with gr_q8_tail like gr_up_multi_kernel,
so STRATA_QFUSE gets its q8_1 image from it too. In the engine (staged read)
only the up projection changes: hc-read0's up 1.08 -> 0.96 ms per window on a
V100-SXM2. fused_gr_bench bitwise, gr_multi_parity OK.
Read off mmvq_il_parity --bench on a V100-SXM2-32GB with the table's own rule
(every measured shape of a class at least 3% faster, else native_mmvq). At 3
columns: the Q5_K head 1094 -> 821 us, Q6_K 12288 rows 68.9 -> 47.9 us, Q4_K
10240 rows 43.9 -> 32.5 us. Bitwise native_mmvq's output (mmvq_il_parity OK).
The AMD grouped-expert layouts (STRATA_EXP_MODE) now build for CUDA too. sm_70
defaults to mode 8 - the codebook grid and the group's q8_1 activations in
shared memory, SwiGLU + q8_1 fused into the gate/up epilogue - every other CUDA
card keeps mode 0. On CUDA, IQ2_XXS gate/up joins the shared-memory kernels,
a format without one falls back to the CUDA layout (not R2), and a strided call
(the verify window's PCIe call) keeps the CUDA layout.
V100-SXM2, setup's IQ2_XS, a verify window's VRAM call (28 groups): IQ2_S
gate/up layers 180 -> 150 us at 3 tokens, IQ2_XXS 170 -> 134 us; bitwise mode
0's output on all 48 layers (native_expert_bench), native_grouped_parity 0
failures in modes 0/2/7/8.
A staged blob (the GGUF role slices assembled into one buffer) is handed to
ExpertCache::fill_slot's cudaMemcpyAsync as-is. From a pageable buffer the
driver stages every copy through an internal bounce buffer - an extra copy at
a fraction of the transfer rate, on the calling thread - so the admit path,
the profile prefill and the lent-slot refills all pay it per expert. Allocate
the pool's buffers with cudaHostAlloc instead, falling back to pageable for a
buffer when the driver refuses (pinned memory is scarce on the small-RAM
machines this path serves).
DETAILS.md's STRATA_PREFILL_CPU_SHARE section: chunks below 3,072 tokens are staged when it is on
(STRATA_PREFILL_CPU_SHARE_MAX=1024 keeps the old limit), the experts the CPU may take (the arena, the
page-locked copy, the mapped experts.bin's pages), the layer split, and a table measured on an RTX 5070 Ti
alone and an RTX 3060 + RTX 5070 Ti split (PCIe 3.0 x8, DDR4-3200, --mmap-experts): 512-token prompts
-49% / -41%, 3,000-token -22% / -24%; a recorded 100K-token coding conversation on the two cards 137.4 ->
130.6 s.
STRATA_PREFILL_CPU_SHARE handed the CPU pool its share of a chunk below 1,024 tokens only, only experts in
page-locked RAM, and only on one GPU. With it set:
- the chunks staged after their routing (the only ones the share applies to) go up to
STRATA_PREFILL_CPU_SHARE_MAX tokens (default 3072, at least 1024) instead of 1024; Prefill::arm_cpu_share
sets the limit before any loan is sized, and init sets it too;
- an expert whose blob the CPU reads from RAM without assembling it (the arena's unpinned part, the mapped
experts.bin's pages) can go to the CPU, not only a page-locked one; the blob pointers are taken when the
experts are chosen, and one without a stable blob stays on the GPU;
- on a layer split every stage has the pool (serve, no --batch), and one stage at a time takes it for a
chunk while the other reads its chunk on the GPU alone.
Unset, nothing changes.
auto balanced the CPU's time per expert against the GPU's per streamed expert, both measured with the share on, so
nothing compared a layer with and without it. On jase100k's RX 7900 GRE + Ryzen 7 5700X3D the host's time per
streamed expert rose 107 -> 200 us with auto (0.47): the streaming side saved nothing (0.53 x 1.87 = 0.99) and the
CPU's work came on top, -4.5%. Now every eligible layer is timed with CUDA events from before its routing sync to its
combine, per non-resident expert; adjacent layers alternate no share / the measured share until three ratios are in,
then auto shares while the median of the last five is below 1, with one layer in 29 in the other arm. Layers with a
one-time cost and the first layer are left out. STRATA_DBG_CPU_GATE=1 prints each measurement.
RTX 5090 + 9950X3D, IQ2_XS, 250 / 600 / 900 tokens against off: DMA streaming -9.8/-9.9/-10.4% -> -8.8/-9.1/-10.5%;
an emulated host-copied stream -18.0/-16.7/-14.4% -> -16.1/-15.2/-13.0%; one core + one worker (where the old auto
lost up to +11.5%) +1.4..+2.4% -> +2.0..+2.3%, the cost of the first measurements. Unset: byte-identical.
The parking path has two gates in series: make_room() balances the cache's own budget, then conversation_memory_admit() checks the host's free RAM against the snapshot plus --conversation-cache-min-free-mib. The second was final: a failure dropped the snapshot even though the cache held conversations whose RAM it could have, and the next request read its whole prompt again (270k tokens, 96 s, measured by the reporter in #1347).
Evict the least recently active parked conversations until the gate admits the snapshot, bounded by the slot count; entries that hold a pinned shared prefix are not evicted. The failure line says how many were evicted and how many remain.
Reported with the fix by alanthinker in #1347.
The 16 dense GEMM geometries a Qwen3.8-Flash-Next prompt logs on a Radeon 890M (gfx1150, Strix Point) at
T = 64 .. 16384, 144 rows, calibrated with tools/hip/tune_hipblaslt.cpp against AMD's TheRock ROCm 7.14.1 for
gfx1150 (hipBLASLt 1.4.1 cd957402). Two passes back to back; each row keeps the solution with the lowest median of
all six repetitions (a single repetition stalled now and then, and the tool's mean-of-three picks differed in 26 rows
between the passes).
Every row beats plain hipBLAS, 1.55-22.6x per GEMM. End to end on GSQ-RCO IQ3_XXS, three interleaved rounds: prompts
read 1.5-1.7x faster (3.6K tokens 124 -> 216 tok/s, 7K 130 -> 226 tok/s), decode unchanged; hip_prefill_hipblaslt_gemm
passes with it and a server run logged no fallbacks. Setup does not install for gfx1150, so the table is set by hand
(STRATA_HIPBLASLT_TUNING). Method and data: bench/results/2026-10-07-community-gfx1150.
A verify window's graph waits on the GPU for flags the host raises once
cudaGraphLaunch has returned. A blocking launch returns only when the
graph has finished, so with CUDA_LAUNCH_BLOCKING=1 the first window after
a prompt never ends and the watchdog stops the engine, whatever the prompt
length, GPU count or --pipeline-windows. The engine now says so at start,
and the watchdog's line names it. Nothing changes without the variable.
A pipelined group's window spans its slots up to its last busy one, and the engine keeps no pipelined slot as a
conversation cache, so the least-recently-used pick left leading holes ([_ B]) that cost a whole row each. pick_slot
now fills such a hole first, then follows the spreading order. The engine reports INFO batch_groups so the server
knows the groups really run pipelined. serve/test_slot_alloc.py covers the allocator.
Co-authored-by: blange48 <[email protected]>
The miss cost (190 ms per unit of routed mass, fitted on x16 links) is one
constant for every stage, so --layer-split auto cannot see an asymmetric
pair: on 2x 4090 D with CUDA1 on a physical x1 link (1.6 GB/s measured) it
gives that card 26 layers (K=22) and reads a 29K prompt at ~350 tok/s.
Probe each stage's link inside the search (probe_pcie_h2d_gbps; the
stage-setup probes are skipped when --pcie-frac is given, and the search
needs the reading either way) and scale that stage's miss term by
clamp(20 GB/s / bw, 1, 64). 20 GB/s is pcie_frac_for_gbps' x16 reference;
the lower clamp keeps a symmetric machine on the exact formula the sweep's
K was fitted with (every scale 1.0 -> the objective is unchanged).
With it, auto picks the fit-in-cache boundary on that machine (K=38: the
narrow card's cache holds all of its layers' pairs, zero streaming there)
and reads the same prompt at ~1410 tok/s, ~4x.
The 190 ms mixes the decode CPU-pool cost with the prompt link cost;
scaling the whole per-stage term is the first-order fix (splitting the
constant would need a fresh fit on both a narrow and a wide link).
STRATA_PF_FUSED=1 reaches only chunks of stream_all_min() tokens or more (1,024, measured on a discrete card); a
smaller read runs its experts through MMQ. On Strix Halo every expert is resident and MMQ is ~3x slower on a
~1,000-token read, the size of an agent's turn: 441 / 855-token reads 3.1 / 4.0 s -> 0.9 / 1.2 s, a 12-turn agent
replay 3.4 -> 2.1 s per turn (C2T8, --no-prefill-borrow). It changes bits (the fused path's output, which reads of
1,024+ tokens already get), so section 5: teacher-forced KL against today's path 0.016 on reads under 1,024, the same
as fused vs MMQ on longer reads (0.019); reads of 1,024+ unchanged (KL 0).
Setup installs the ROCm 7.14.0a20260608 wheels for gfx1151 since 0.1.40.2 (#1267); their hipBLASLt is 1.4.0
(100400), which neither gfx1151 table matched, so setup's engine read the prompt through plain hipBLAS.
tune_hipblaslt --workspace-mib 32 on the 100401 table's 90 shapes, on a Ryzen AI Max+ 395 (Radeon 8060S, gfx1151,
128 GB, Fedora 44, kernel 7.2.8) with those wheels: 3.1-14.1x faster than plain hipBLAS per shape (median 6.2x).
hip_prefill_hipblaslt_gemm and hip_prefill_hcd_exact_parity pass with it (fallbacks=0); HIP ctest 61 of 62
(ple_parity needs the Q2_0 fixture). setup picks it up with no code change.
Below N of N lendable slots the uncovered experts are read from the pack during a long prompt (reporter: 18% uncovered = -26% prompt speed on gfx1100). A WARNING line now says so, and docs/AMD_HIP.md documents the headroom knob and the levers from the report. No behaviour change otherwise.
The gated delta rule in the WY form flash-linear-attention uses, all FP32: gdn_chunk_prep_kernel forms k.k / q.k and
T = (I + A)^-1 for every chunk of a 2048-token super-block at once; gdn_chunk_scan_kernel (128 blocks, a key head x
16 value columns each) walks the chunks with its state slice in registers. Every exponent <= 0 (the gates are
negative). The scratch (25.6 MB) is allocated once per device: freed and reallocated per call it cost more host time
than the kernels saved on short prompts. sm_80+, >= 128 SMs, enough shared memory, chunks of >= 128 tokens; else
gdn_rec_kh_kernel. Unset: unchanged.
gdn_rec_parity: against gdn_rec_kh_kernel <= 1.13e-6 (T = 1..4099, 8192, 32768, a non-zero starting state); against
an FP64 recurrence <= 1.05e-6 (the sequential kernel <= 3.2e-7) - FP32 rounding; 199 restarts of 7 shapes bit for
bit (NaN-filled scratch between). RTX 5090: 1.81x the recurrence at 2K-32K; IQ2_XS: the recurrence phase of a 32K
prompt 520-532 -> 306-334 ms (-210 ms, -3.7%), 8K -2.1..-3.6%; first-token KL 0.0016-0.0152, 1.15-1.83x a harmless
control arm (STRATA_PROMPT_ATTN_V1), the same top token.
The MoE input and the routing weights go to host memory on a stream beside the primary's compute stream, so its
own experts start at once instead of behind the copies over its link (mixed_h and w are rewritten only after this
layer's combine, which waits for the peer). STRATA_PF_PEER_XSTREAM=0: the compute stream. With
STRATA_PF_PEER_RING=160, prompt tok/s: 100K 1,854 -> 1,918; 8K and 3.4K unchanged.
Each group's tokens and their row lists (in the experts' order) are built on the host from the rows' pairs and
uploaded once per layer; peer_gather_add is eddoursul/Strata's moe_gather_add with the weight looked up through
the row's pair, so the sums are bitwise those of the per-expert launches. STRATA_PF_PEER_GATHER=0: per expert.
Prompt tok/s against the per-expert launches: 3.4K 825 -> 832, 8K 1,468 -> 1,500, 100K 1,821 -> 1,858.
Co-Authored-By: Eddoursul <[email protected]>
The MoE input goes out as the FFN input's existing FP16 image (mixed_h) and the per-token sums come back as
FP16, saturated at +-65504 (eddoursul/Strata f8de703's FP16 input, 2acfac4's sums_to_f16): half the bytes over
the primary's link both ways. STRATA_PF_PEER_F16=0: FP32.
Peer stream 0.6, prompt tok/s: 3.4K 805 -> 831, 8K 1,284 -> 1,495, 100K 1,578 -> 1,855 (the share off: 1,712).
The 8K and 3.4K answers stay byte-identical to the share off.
Co-Authored-By: Eddoursul <[email protected]>
The design is eddoursul/Strata's second-GPU prompt path (f8de703, 2acfac4, e8ade66): the second GPU adds its
experts' rows into each token's weighted sum and the first GPU's combine adds that to its own experts' rows.
Over the host route the peer scatter-adds its rows into per-token sums (peer_scatter_add, after the fork's
moe_scatter_add: one launch per expert in a fixed order, so the sums are repeatable) and sends T x N floats
back instead of rows_peer x N (up to K = 10 times more); moe_combine_peer adds them to the primary's rows. The
2a row buffers are not allocated (3.4 GB of pinned RAM at 32K-token chunks). STRATA_PF_PEER_SUMS=0: rows.
Same setup, peer stream 0.6, prompt tok/s against the share off: 3.4K 332 -> 805, 8K 762 -> 1,284, 100K
1,712 -> 1,578. The 8K and 3.4K answers are byte-identical to the share off (greedy, 256 / 3,221 tokens).
Co-Authored-By: Eddoursul <[email protected]>
set_peer refused a peer that cannot access the primary directly, so on GPU pairs without P2P (most GeForce
pairs) the prompt path stayed on the primary GPU. Without P2P the share now takes the route the layer split's
helper already uses: the activations and the result rows through mapped pinned host buffers read by copy
kernels, with the compact group buffers (the layer-sized path copies over P2P only). STRATA_PF_PEER_HOST=0
restores the refusal.
RTX 3090 (PCIe 4.0 x4) + RTX 3080 Ti (x16), no P2P, UD-Q4_K_XL, --prefill auto:32768, peer stream 0.35 / 1.0,
prompt tok/s against the share off: 3.4K 332 -> 492 / 551, 8K 763 -> 907 / 776, 100K 1,713 -> 1,072 / 734.
The peer's rows coming back over the primary's x4 link cost more than they save on long prompts; the next
commit sends sums instead.
- tools/check_isa_guard.py disassembles the CPU kernels (objdump + nm on the Linux kernel library, capstone + the /MAP
file of a linked binary on Windows) and fails when a function holding AVX-512 / AVX-VNNI-class instructions is
not a kernel of a wide-ISA file: a weak or COMDAT copy (the linker may pick it for every caller), a std:: /
template instantiation, a probe or helper that every CPU runs, or code in another object. MSVC's own
`__isa_available` vectorizer guard is accepted. ctest isa_guard_check runs it; exit 77 (skip) when the tools
are missing.
- -DSTRATA_SDE=<sde> adds sde_{hsw,adl}_* tests: the CPU parity tests on an emulated Haswell (AVX2 only) and Alder
Lake (AVX-VNNI, no AVX-512); SDE stops at the first instruction the chip lacks.
- tools/test_check_isa_guard.py: unit tests of the policy.
expert.cpp and iq_avx512.cpp are compiled for AVX-512, iq_avx2.cpp and kq_avx2.cpp for AVX2, and a file compiled
that way may use the wide ISA anywhere in its code. Code that every CPU runs was still defined in them:
- cpu_features, CpuFeatures::reason, cpu_require_expert_support (the function that finds out whether the CPU has
AVX-512) and expert_set_oracle_q8_0 / expert_oracle_q8_0_enabled (the first call of every generate, on any CPU)
were in expert.cpp;
- iq512_supported, iq256_supported and kq256_supported (a comparison native_gu_rows asks before it has checked
what the CPU can run) were in the kernel files; iq512_supported was called on every AVX2-only CPU for every
gate/up call.
They now live in expert_layout.cpp and native_expert.cpp, which are built for the baseline. No kernel changes;
the compiled code of the moved functions is identical, so default output is byte-identical.
RX 7900 XT 20 GB, IQ3_S, Windows HIP, auto cache, default reserve 700: 0.1.40 ended with 4,255 slots (8.12 GiB) and 2,170 MiB free, 0.1.40.2
with 5,512 slots (10.48 GiB) and 323 MiB free; WDDM paged and decode fell from 51.7 to 21-25 tok/s. With --vram-reserve-mib 2600: 4,299 slots
(8.20 GiB), 2,224 MiB free, long prompt 1,015.8 -> 1,355.9 tok/s (+33.5%), decode 24.3 -> 55.5 tok/s (+128%), back inside 0.1.40's ranges.
A reserve given on the command line is kept; a smaller card keeps its own sizing. The cache size changes no output bit.
amdhip64_7.dll imports rocm_kpack.dll, which imports the MSVC runtime. Only the two runtime DLLs were copied beside the exe, so a
full-path load of the bundled one failed with error 126, Windows fell back to System32's copy, and big prompts died with
hipErrorInvalidDeviceFunction (RX 7800 XT, gfx1101). The closure is now read from the PE import tables of the files in the zip
(tools/hip/dll_closure.py, no pefile), by setup.hip_runtime_beside_exe and tools/hip/package_windows.py. On the 0.1.40.2 HIP zip it is
amdhip64_7, amd_comgr, rocm_kpack, msvcp140, vcruntime140, vcruntime140_1. The engine, when the loaded runtime is not the one beside it,
also tries LoadLibraryEx on the bundled one and prints GetLastError. Diagnostic only. Not tested on a Windows AMD card.
The native router reads 512 logits a row; the window path two lines above already tested n_expert == 512 and K == 10, the
per-token fallback tested only native_router_enabled(), so a model with fewer experts read past each row. It now takes the
generic router_top10 like its siblings. native_multi_parity checks the generic router on 256-wide rows.
Reported-by: rwkeyes (#1357)
0.1.40.2 turned fork F4 on by default after single-GPU measurements (+4.2% on an sm_86, +1.3 to +2.2% on an sm_120). A
two-GPU Windows layer split (RTX 4080 SUPER + RTX 5070 Ti) was reported 15% slower with it on (#1373). The outputs are
bitwise the same either way, so only speed changes. STRATA_MMVQ_IL=1 turns it on; docs/DETAILS.md says why.
samp_acc_t is float by default (-DSTRATA_SYCL_SAMPLER_FP64=1 keeps the reference's double). On an Arc A750 one row of the
tail cost 451 us in double and 20 us in float; a sampled decode ran at 7.62 -> 8.10 tok/s (medians of 5 interleaved
pairs, runs 7.1-9.2). sampler_parity: 0 failures on the A750 (JIT build) and on the B70; both cards' 10-prompt gates pass
10 of 10, and the B70 gate's sampled cases pick the same tokens as before. Of the kernels that use double, only the
sampler runs on a native pack (the router, combine, gate and norm kernels there are float).
Probes: sycl/probe/fp64_cost.cpp (FP64 against float in the small reduction kernels), gemm_vary.cpp.
docs/INTEL.md: the aliased-pages bug, NO_HOST, the prompt-chunking numbers, draft-length sweep, FP64 and XMX findings;
docs/INTEL_ARC.md: a row for the B70/A750 runs.
With STRATA_PREFILL_CPU_SHARE on, the last rows_cpu rows of GU and H hold whatever the buffer held before; the check read
them and reported them as non-finite (about 1 in 256 garbage floats), with Dm and the layer output clean.
blob() bumps a plain reads_ counter and, for a file source, its mapped/staged paths; calling it from the share's
thread raced with the prefill thread's own blob_stable()/pinned() calls (the streamed experts). The blobs of the CPU's
experts are now taken before the thread starts, so the thread reads only the arena bytes, the activations and the
grouping tables, none of which the prefill thread writes until the thread is joined.
A chunk below stream_all_min() (1,024 tokens: an agent's tool result, a test's
output) streams every routed expert outside VRAM over PCIe while the decode CPU
pool sits idle and the arena holds the same experts in RAM. Opt-in,
STRATA_PREFILL_CPU_SHARE=auto|x:
- the non-resident experts routed by at most MAXT of the chunk's tokens,
fewest first, run on the pool on a thread while the GPU streams and runs the
rest; their unweighted rows go to Dm's tail (as a peer GPU's do) and the
combine weights them as any other row;
- auto: the share measured, each layer's CPU time per expert and the GPU's per
streamed expert (CUDA events around its expert work, read after the next
layer's routing sync), and the next layers hand the CPU g / (c + g), where
both end together - a small CPU takes a small share;
- Q2_0 layers take the pool's Q2_0 kernels' activations (ActQ, as decode's),
i-quant layers the native vec_dot type.
Serve without batch slots and generate, one GPU. Unset (the default): no
expert moves, byte-identical to v0.1.39 (IQ2_XS 600 and 2,100 tokens, fixed
cache, logits and 32 tokens).
RTX 5090 + 9950X3D, 600-token prompts, ms (off / auto):
- UD-Q4_K_XL: 1,548 / 1,369, 1,528 / 1,296 (share 0.65; KL between 0.006);
- Q2_0: 472 / 445, 491 / 448;
- IQ2_XS: 493 / 481, 502 / 460, 527 / 529; 250 tokens 424 / 355, 402 / 382,
379 / 367 (KL to STRATA_PREFILL_MMQ=0 0.0053 off, 0.0080 auto);
- IQ2_XS on 2 AVX2 workers (emulated): 512 / 487 (share 0.49).
Every prompt chunk streams the experts the VRAM cache does not hold, over PCIe, and a 4K prompt touches nearly all 512
experts of a layer. The short first chunk (STRATA_PREFILL_FIRST, 256 tokens, there to start the GPU while PLE rows are
read) therefore cost a whole extra pass of that stream whenever the cache holds only part of the model. Its default is
now 256 only when the cache holds every expert, else 0 (an explicit STRATA_PREFILL_FIRST still decides).
Arc Pro B70 32 GB, IQ3_S (11.5k of 24.6k experts in VRAM, the rest in the pinned mirror at 13 GB/s), a 4,095-token
prompt, MTP on, interleaved runs, medians of 3: 2048-token chunks 618 -> 730 tok/s, 4096-token chunks 784 -> 980.
sycl/setup_intel.py now writes --prefill 4096 for a card with 24 GB or more (it costs ~700 expert cache slots, about 3%
of decode on the 200-token story).
An Arc Pro B70 (xe) returned the 22 GiB expert-cache arena with two of its 2 MiB pages mapped onto the same memory:
a write to the page at +2 MiB showed at +1022 MiB, so cache slots there held other experts' bytes and the prompt
came out NaN at the first layer that routed to one of them (layer 4, expert 249). Which pages alias depends on the
allocation history (the context size, the MTP window), so it comes and goes between runs.
strata::arena_alias_check tags every 64 KiB of a fresh range and reads the tags back in a second kernel;
strata::malloc_device_guarded allocates again behind a growing spacer until the range is clean (and warns if it
never is). It guards the expert cache arena and every device allocation of 32 MiB or more (session arena, weights,
MTP, verify, head, prefill). STRATA_ARENA_ALIAS_CHECK=0 skips it. Two debug aids: STRATA_VERIFY_ALL_SLOTS=1 (with
STRATA_VERIFY_FIND=1, STRATA_VERIFY_DUMP=dir) reads every filled slot back and compares it with the GGUF, and the
probes sycl/probe/arena_rw.cpp and alias_scan.cpp.
A blocking read of the engine's READY line or of the vision encoder's pipe that
never returned held the request turn for good: the engine sat idle in its command
loop, so no watchdog could fire and every later request queued behind it (the lost
step class of #481). The READY read now has a 900 s timeout, the encoder's READY
and ENC reads 300 s each. A process that says nothing for that long is killed, the
read raises, and the next request starts a fresh one. Slow loads and slow encodes
stay far inside the limits.
Only the read timeouts of the reporter's patch are taken. The watchdog that ends
an engine after 90 s without output and CPU progress is left out for 0.1.41.
Tests: a silent engine, a silent encoder at READY and at ENC, and an in-time answer.
Reported-by: btc1000w
Co-authored-by: btc1000w <[email protected]>
A hint only: it never enables the kernel and says the output bits change. Silent when the variable is set to anything. Follows the R9700 numbers of PR #1318 (+31% prompt at 8.8K).
I calibrated a table for hipBLASLt 1.4.1 with tune_hipblaslt over the 32 geometries of
gfx1201-hipblaslt-100500.txt (R9700, system ROCm 10.1.0, hipBLASLt 1.4.1): 1.15x geometric
mean per GEMM over hipBLAS, but only 1,606 -> 1,613 tok/s median prompt over three matched
trials (a 1.008x geometric mean) and decode unchanged, so within noise - which is what the
R9700 paragraph above already reported, now from a second machine. The cold first prompt of
each start is left out of those numbers.
I dropped two rows from it: bf16 N=10240 K=320 ldy=10240 at T=4096 and at T=8192. hipBLAS
won T=8192 in five separate runs (0.609-0.619 ms against 0.648-0.677 ms for the best of 16
candidates), and dropping only the T=8192 row would send that call to the T=4096 row rather
than back to hipBLAS, because closest() takes the nearest T bucket.
Two calibration traps I ran into are recorded too: the first case of a fresh tune_hipblaslt
process times its hipBLAS baseline while the clocks are still ramping, and the first fresh
prompt after a model load is not a measurement (949 vs 1,423 tok/s in the two arms there).
On my R9700 (engine 0.1.40, IQ3_S at 431072 ctx, --kv int8, the hipBLASLt 1.4.1 table of
the previous commit, GPU idle): fresh prompts of 4,210 and 8,830 tokens, three matched
trials each and the cold first prompt of each start left out - 1,571 -> 2,064 and
1,568 -> 2,065 tok/s at 8,830 tokens (+31.4 and +31.7%) and 1,699 -> 1,990 at 4,210
(+17.1%), medians 1,571 -> 2,064, a 1.27x geometric mean over those three - with decode
unchanged (89.8 -> 86.9 tok/s). hip_prompt_attn_wmma passes on this card.
STRATA_SELECT_WMMA=1 on top of it changed nothing at 4-9K prompts (2,026 tok/s), which is
consistent with its +1.5% being a 16K/262K-ctx number.
The shipped gfx1100-hipblaslt-100401.txt calibrates "the 26 dense GEMM
geometries of the shipped gfx1100 table", all at T=4096/8192. A prompt on
Qwen3.8-Flash-Next reads a whole other geometry through the prompt path, and
it is issued once per expert group at whatever T the group has - so it never
matches a row:
STRATA_HIPBLASLT_VERBOSE=1, GSQ-RCO IQ2_XS, 192000 ctx, 3151-token prompt
hipBLASLt tuning enabled (26 rows, gfx1100, version 100401)
642 fallbacks, all f16 N=1280 K=2560 ldy=1280, over 642 distinct
T values from 1 to 1873
That is the small-T expert GEMM, and it is nearly every launch in the prompt.
TuningTable::closest() matches on (n, k, ldy) and takes the nearest t_bucket,
so the rows below are a spread of buckets over the small-T range rather than
one row per T; bf16 10240 2560 10240 (the hyper-connection projection, one
fallback) is covered the same way.
tools/hip/gfx1100-hipblaslt-100401.txt: 17 rows added with tune_hipblaslt
from this tree, against the engine's own 32 MiB workspace (gemm.cu:441), on
the same RX 7900 XTX and packaged ROCm 10.0.0 the existing rows were
calibrated on. No existing row is changed. 26 rows -> 43.
Same binary, same config, the table as the only difference, 3 interleaved
pairs at a 3151-token prompt:
691.6 -> 940.7 tok/s (+36.0%)
712.3 -> 943.2 tok/s (+32.4%)
727.7 -> 928.7 tok/s (+27.6%)
mean 710.5 -> 937.5 tok/s (+31.9%)
Decode unchanged (58.6 -> 60.6 tok/s, inside this box's 2.3% run drift), and
STRATA_HIPBLASLT_VERBOSE=1 reports fallbacks=0 with the 43 rows, which the
26-row table does not.
The pick becomes argmax_i p_i / E_i with E_i = -log(u_i) and u_i a hash of (seed, counter, token id), an exact
sample of p. The noise a token gets does not depend on which other tokens survived the draft's or the target's
cut, so a coupled draft (STRATA_SPEC_COUPLED=1) agrees with the target on the tokens they share. Off by default:
nothing changes without the variable. sampler_parity: kernel vs host reference under three chains, frequencies
vs softmax, the flag is read and greedy is unchanged.
Ported from #904 onto the 0.1.40 verify window. Every switch is OFF by default, so
default output and the default launch sequence are unchanged:
- STRATA_DF_PDL=1: programmatic dependent launch (pdl.hpp) for the window's
quantizations and dense projections (multi-column MMVQ, multi-row BF16 GEMV);
sm_90+ only, CUDA 12.3+; HIP and older cards keep ordinary launches.
- STRATA_DF_PLE=1: the window's hyper-connection write and PLE post-ops in one
pass, which also writes the commit's history snapshots (native_gr_post_multi
with strides, native_ple_postops_batch_snap).
- STRATA_DF_QB_Q4=1: a Q4_0 KV cache takes the batched QSA query path too.
- STRATA_DF_BRANCH=1: the mixer's side branches inside the window graph.
Left opt-in because the author saw a stall on Linux with NVML queries.
The q/gate split now uses copy_rows_strided (a kernel) instead of a
cudaMemcpy2DAsync so a PDL chain stays kernel to kernel; the bytes are the same.
Left out: STRATA_DF_KVAPP (kv_append_q4_step_multi). 0.1.40 already appends a
window's K/V cells in one launch (kv_append_q4_steps, #783 PR-d).
Merged by hand with the 0.1.40 forms of the same code: the exact-T BF16
multi-token kernel, the PAIR/TS MMVQ variants, QFUSE (q8_attn), the fused
norm+rope and the batched residual write.
Co-authored-by: Francesco Albano <[email protected]>
(PR #904)
The high index bits of a 32-value half are spread one per byte by a table and
punpcklbw puts them next to the low index bytes, so each grid lookup is a
16-bit load plus the table load. The sign vector is one broadcast load plus one
pshufb, and the (2s+1) scale vectors are table loads. Arithmetic is untouched
(same words in the same lanes). The decode is ported onto the 0.1.40 layout of
iq_avx2.cpp: the gathered IQ3_S/IQ2_S forms (Fmt32<121>, <122>) and the VNNI
rows are left as they are.
Co-authored-by: Stefanos Anastasiou <[email protected]>
(PR #930)
Opt-in (only with --expert-cache-per-layer). The same VRAM holds more experts
(7,008 -> 9,312 slots at equal VRAM per the author). --expert-cache N stays the
budget of N largest blobs; shrink_to keeps whole per-layer lists so a blob never
lands in another layer's range. The free-VRAM reading is device_free_bytes(), as
in the shared-cache branch. The PR's png is left out.
(PR #934)
Opt-in (default 0 keeps three tasks per participating thread, so default
output and timing are unchanged). Adds pool_tasks_test, which compares the
split against the default partition bitwise for 0..4096 tasks.
(PR #949)
The prefill Stager waited by spinning: a yield loop on `issued`, and
cudaEventSynchronize on events created without cudaEventBlockingSync
(which spins under the default scheduler). With --mmap-experts there
are 32 stagers. On Linux they inherit the host thread's single-core
pin and starve it: on a DGX Spark (GB10) prompts were read ~10x slower
(IQ3_S 8K 91 -> 1,222 tok/s, 30K 88 -> 1,388 tok/s with this change).
Unpinned instead, the 32 spinners took every core for the whole prompt
(process CPU ~1,800%); with the waits sleeping it is ~100%.
- ring wait: std::atomic::wait, with notify_all in issued_one() and
finish()
- dma_done events: cudaEventDisableTiming | cudaEventBlockingSync
- HIP shim: cudaEventBlockingSync -> hipEventBlockingSync (untested)
Decode speed and output tokens are unchanged.
(PR #1057)
ExpertCache::admit never evicts because eviction policy is a measured
question (R4.1's LFU-decay vs LRU sweep). This is the harness for that
measurement: a policy's hit rate depends only on the order (layer,
expert) pairs were routed, which --dump-routing records, so replaying a
trace through simulated slot pools needs no GPU. fill-only (the
engine's current behaviour), lru, lfu-decay and an oracle (Belady's
MIN, the upper bound); global or R4.2g per-layer scope matching
ExpertCache::layer_slot_range; any slot count; optional profile prefill;
per-window rates, so a topic shift shows up as a per-window dip.
tools/test_bench_eviction.py runs without a GPU (14 cases): each
policy's victim choice, the per-layer quota arithmetic, the windows
split, and profile prefill order.
(PR #991)
A one-int-a-column interleaved copy of the window's q8_1 blocks lets a warp decode each weight block once and apply it
to every token with the single-column dot's own expression (the same *_q8_dot_impl calls), walking the chunks of the
exact multi-column layout and adding the four virtual warps' sums in warp order, so every output is bitwise the
multi-column kernel's. IQ4_XS, Q4_K, Q5_K and Q6_K projections (and the head) of 2-4 tokens use it; the copy is made
from xq_ by one small kernel, so the fused HC-read producer works too. STRATA_MMVQ_IL=0 keeps native_mmvq's kernels.
CUDA only (HIP keeps its kernels). mmvq_il_parity checks every case bitwise for rows 1/2/4 and the table; --bench times it.
Ported by hand from Eddoursul's fork (738e166, b67a291), with the IQ3_S and Q8_0 parts and the 5-8 column widths left out.
Co-authored-by: Eddoursul <[email protected]>
kvg_trim took only the short request's cell count, so every short request in
a long conversation's session shrank the K/V to 8192 cells and gave the
expert slots back, and the conversation's next turn grew it again and moved
the experts again (a parked conversation is restored through kvg_ensure).
With STRATA_KV_GROW_HOLD=1 the trim keeps the longest parked conversation's
cells (+256) while it is parked. Opt-in: the default trims to the request
alone as in 0.1.40, so the K/V gives back its VRAM as before. Output is
unchanged; only --kv-grow is affected.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
Ported from the community PR with the author kept. Raw dumps and logs were trimmed (TRIMMED.md in the folder lists them); the README, summaries and configs are unchanged. Indexed in bench/results/COMMUNITY.md.
With --layer-split auto and STRATA_STAGE_TRIM=1 a later stage's weights load after the search, and the search priced them
as the full arena with no projections, so a small card was over-credited: P40 24 GB + RTX 3070 8 GB picked K=2 (prefill
107 / 112 tok/s at 4K / 32K) where K=16 gives 343 / 654 in the report. The carve is now priced for each candidate range:
the trimmed arena (WeightTable::pool_bytes) plus NativeDense::weight_bytes_for, a walk of load over the GGUF headers that
rounds each matrix to cudaMalloc's 2 MiB granule (+0.8% on the iq3_s pack, high on purpose). After each late stage loads, the
bytes priced are compared with the VRAM it really took and a warning says when they drift.
Only the late-weights arm changes (auto with the trim asked, opt-in); every other run has its weights loaded before the
search, carve_bytes is 0 there, and placements do not move. Ported from 572b607 of #880 onto this tree's loader (layer ranges
as arguments, STRATA_Q8_PACKED, STRATA_HC_Q8 untouched); not run on a mixed pair here.
Co-authored-by: gopinath87607 <[email protected]>
The report shows the bundled amdhip64_7.dll next to strata.exe and the System32 copy (a different build) in use, with the
prompt GEMMs failing. The import is bound before main, so the engine cannot swap it; it now prints a WARNING naming the
bundled file when the loaded one is another, so the log carries the fact. Nothing else changes. The fix itself (loading the
bundled runtime by full path, or a delay-loaded import) needs a Windows AMD machine to test and is not in this commit.
On Windows an engine with 313 MiB of VRAM free after a 15,104-token borrow exited silently inside the verify-window
capture. The cause is not reproduced here (no Windows AMD card), so this only leaves a trace and a hint, and changes
nothing: the capture prints how much VRAM is free before it starts, and a Windows start with less than 512 MiB free
and --prefill auto names --prefill auto:8192 (the reporter's workaround) and --vram-reserve-mib. The chunk choice is
untouched.
The build stopped with a FATAL_ERROR because gfx1150 was in none of the architecture lists, although the device code
(the WMMA guards, sudot4) already covers it. It is now in the unvalidated list: the build warns and goes on. The
report's machine (96 GB, TheRock 7.14.1, IQ3_XXS: decode 17-19 tok/s, prompt about 125-135 tok/s) is in docs/AMD_HIP.md.
Setup's own AMD path is unchanged.
Co-authored-by: zeriyoshi <[email protected]>
Phones save portrait photos as landscape JPEGs with an Orientation tag, and the encoder's decoder ignores it, so the
model saw them sideways. Such a JPEG now takes the Pillow route (exif_transpose, then PNG); JPEGs without the tag, PNG,
BMP and GIF still pass through byte for byte, and without Pillow nothing changes. Other formats that carry the tag
are turned too.
Co-authored-by: crazyaimachine <[email protected]>
On a layer split with --batch-groups, two concurrent requests ran at about half speed: the slot choice could put both in
one group, and a group's window carried a row for every slot, active or not. Slot choice now prefers the group with the
fewest busy slots (a held prefix of 512 tokens or more still wins), and a window covers the slots up to its last active
one. Without --batch-groups the slot choice is exactly as before.
Co-authored-by: crazyaimachine <[email protected]>
The search priced placements by decode time only, so at 512K context it could pick a split where a stage could neither
lend a 512-token prompt chunk nor afford its own buffers, and the engine ran out of VRAM at start while an explicit
--layer-split worked. A placement is now taken only if every stage can lend the last-resort chunk or afford its own
buffers (the serve path's own tests, priced from the same figures); the tie-break for balanced stages stays. A
start-time probe on the live free figures says which card is short.
Changed from the PR: nothing is refused. If no placement can start on paper, the search runs again without the gate
(the choice is then what it was before) and prints what is short and how to make room; the probe only warns.
Co-authored-by: fukc-gihtub <[email protected]>
Every layout of active slots keeps its own graph pair, and with the expert cache sized to the reserve the next window
graph can find no VRAM left; the engine then exited (#997, `parallel` 4 to 8). cudaGraphInstantiate that runs out of
memory now evicts the least recently used batch layouts, other than the one being captured, until it fits; they are
captured again when needed. The start prints how many were freed. Nothing changes while everything fits.
With the key on, a call of a declared tool is the call when the model writes the tool's name as a parameter tag,
a JSON object inside <tool_call>, or a bare <function=NAME> at the start of a line; two calls in one <tool_call>
are two calls. Anything else stays text. Without the key every reply is exactly as before. On the author's corpus of
1,462 agent turns: 92.9% read as intended before, 98.3% with the key. Re-files #966.
Codex's review request sends tools, an additional_tools item and text.format json_schema at once, and got a 400.
Now a tool call passes through as it does without the format (words written before the call go out as a message
ahead of it), and the schema is checked on the turn that ends in text, as json_schema works without tools. With
tools the system directive says the schema is for the final answer only. The chat route's rule is unchanged.
Tests in serve/test_responses.py reproduce the request shape: a call, a valid text answer, and (with jsonschema
installed) an answer that breaks the schema.
The wait for the engine's next line was 10 s, so a disconnect during a quiet prefill reached the engine only at the
next heartbeat. Both waits now poll the cancel event every 0.5 s; the HTTP heartbeat stays at 10 s, and an expired
silence deadline is enforced before another heartbeat. serve/test_quiet_cancel.py drives a fake engine that says
nothing until STOP. This is the Python half of #962; the engine half (stop between prompt chunks) is open.
--force now removes the old pack's markers (index.txt, manifest.json) before the build writes a byte. A forced build
that stops part-way leaves experts.bin without a manifest, which the next build or setup makes again, instead of the
old markers over partly new experts.
A string or list as a tool's schema passed the tools check and raised later, in parse_tool_call and the stream parser,
after the 200 and part of the reply. It is now a 400 naming the tool and the field. A missing or null schema still
passes.
A window size's cost was updated only when that size was chosen, and the re-probe covered only sizes seen fewer than
three times. A size priced high during one slow stretch (a prompt's cache refill) could stay priced out for the life
of the process. A size not measured for 300 rounds is now tried again when a confident lookup could use it, one
round per stretch. Output is unchanged (the policy only picks which drafts to verify); draft_policy_test gains a case
that fails without the change.
HIP keeps the shared expert on the draft's own stream; only CUDA branches. The stream and events were created and
never used, and the report's R9700 lost prompt speed after the first request.
STRATA_PF_FUSED=1 on gfx1100 gave prompts that answered only "!". Two causes, from the report:
- the stage loop wrote the next weight buffer with no barrier after it; one __syncthreads() now follows the write.
- the waves_per_eu(8) variant of native_w11_kernel computes wrong results with clang 22 (ROCm 7.10): there the cap is 1
(NW_LB; -DSTRATA_W_LB=N overrides it). Earlier compilers keep 8.
Default output is unchanged (the path is opt-in on gfx11).
An Arc Pro B70 (xe) page-faults when a kernel reads ordinary pageable host memory (devcoredump: ccs0, FaultType 0,
faulted address in the CPU's mmap range, then Timedout job and device lost). Without --stream-experts the experts
outside the VRAM cache sit in a pageable arena that the PCIe share hands to the GPU. The default share is 0 there
(the CPU computes those misses) with a one-line warning naming --stream-experts, as setup configures it. An explicit
--pcie-frac is kept and warned about. i915 (Arc Alchemist), CUDA and HIP are unchanged.
strata::intel_gpu_driver() reads the driver from sysfs; verify.cpp's i915 test uses it.
The mirror table (host-mirror pointers for the device plan) is filled only from pinned USM aliases under
--stream-experts, so it cannot hand out pageable pointers.
STRATA_QFUSE=1 (the gfx1151 arch default) has gdn_step_norm_multi write the
out-projection's q8_1 image itself and skips native_quantize_q8_1. The batched
recurrence (batch_rec_) calls gdn_step_norm_multi once per slot with no q8_1
destination, so with --batch the projection read stale bytes and every
concurrent answer was corrupt. The batched case now quantizes as without
QFUSE; the fused image holds the same bytes (gr_parity), so batched output
equals solo. Single-request paths are unchanged.
A stage that was 100% resident at init takes the doorbell graph while
a prompt loan/shrink/swap is in flight (ar_off_), and that graph waits
on host doorbells the skipped service never raises: one GPU spins at
100% while the host sits in cudaStreamSynchronize, until the watchdog
fires with the previous stage's stale label. Key the all-resident
service skip in run_slot_rows, batch_poll and service off ar_on(),
as run() already does.
PR #1102 (win10ogod) is the same fix for run_slot_rows and batch_poll;
this one also covers service().
Co-authored-by: win10ogod <[email protected]>
#783's batched append of a verify window's cells passed no host copy for the
hybrid K8V4 format (int8 and q4_0 pass st.host, and the per-token path passes
K8V4's two halves). With --kv-resident, a block that was not resident when its
cells were appended came back from the host copy without them, so the tokens a
window wrote were lost: free generation degenerated into repetition (#1188,
#1135) and parked conversations lost their decode-written turns (#1169). The
batched path now passes the same host halves as the per-token path.
wait_step and the chain wait polled ext_oneapi_empty() through DPCT_CHECK_ERROR, which is always 0, so the
loop left after 1024 spins and read draft tokens the GPU had not written (-1). Same migration bug as the
cudaStreamQuery fix in verify.cpp, missed in mtp.cpp. IQ3_XXS on the A750: 61 of 99 drafts accepted on the
story prompt (the 5070 gets 106 of 166); copy_'s readiness test in verify.cpp had the same flaw.
STRATA_DBG_DRAFT=1 prints each window's drafts and the target's tokens.
On an Arc A750 (i915) a verify window launched behind other GPU work in flight (the previous window's commit,
a prefill) often never started: no GPU breadcrumb, the 20 s ring timeout, the engine ends. queues_wait_and_throw
before the launch removes it (9 of 9 three-request runs pass, almost all failed before). Default on for i915,
off elsewhere (STRATA_WINDOW_SYNC=1/0/2 overrides). STRATA_RING_TIMEOUT_S raises the 20 s doorbell timeout.
Level Zero's free-memory query returns the constant total on an Arc A750 (i915); the expert cache and draft head
were sized from the whole card and the first launch failed with out-of-host-memory. device_free_bytes now lowers
the figure by what this process holds (drm-total-local0 / drm-total-vram0 in /proc/self/fdinfo) and says so once.
On the Arc A750 a kernel never saw the host's doorbell and result-row stores to ordinary host USM
(sycl/probe/doorbell.cpp: only the uncached allocation completes the handshake), so every ring wait
hit its bound and the window read stale rows: NaN logits. verify's mapped staging and the pool's
y_miss rows now use host_malloc_polled (sycl_queue.hpp; needs libze-dev's header, else sycl::malloc_host;
STRATA_HOST_UNCACHED=0 turns it off). IQ3_XXS on the A750 now answers 12*12 = 144.
Also: the expert cache and the MTP draft head check Level Zero's null return for a too-large
allocation (and an Arc A750 needs UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1 for a single allocation over
4 GiB); gather_rows no longer asks for a cooperative launch it does not use; the expert-cache
zeroing in generate.cpp is chunked too. Based on the probe in PR #423 (maxfridbe).
One memset of the 25 GiB expert-cache arena runs on the blitter engine and times out on the
Arc Pro B70 (xe: engine memory CAT error, GT reset). big_fill_zero zeroes in 256 MiB kernel
launches; used for the expert cache, the MTP and verify arenas and the KV reset.
The sycl/ copies were at 0.1.39 and did not build against the 0.1.40.x shared
headers (load_experts_gguf ready argument, fused_gr_read_multi bool, Gemm::bf16/native ldx,
13 symbols missing at link). Re-migrated the 5ffcce43 and 0.1.40.1 trees with migrate.sh,
merged with merge_upstream.py, hand-resolved the conflicts (the port's tuned expert and
MMVQ kernels kept; verify, mtp, generate adopt 0.1.40.1 with the port's queue handling).
Stubs: vmm (no driver VMM on Level Zero), --remote-expert-opt. Dropped upstream's
opt-in AMD S26/S27 grouped-expert kernels and STRATA_QFUSE/HC_Q8 reads on SYCL.
build.sh and migrate.sh source setvars.sh outside set -u (they exited silently).
The first version staged every uncached blob and made the layer wait for all of them before any expert computed, which
lost the overlap the kernels' own page faults had (-19% on a cold 16 GB lane). Now prefetch() hands the layer's uncached
blobs to the I/O threads and returns; the kernels that reach a blob before it lands wait on its pages, the others
compute. STRATA_IO_PF_STAGE=1 keeps the staging variant. Residency is probed on four sampled pages, not the whole blob.
The mapped file tier reads experts through page faults. With STRATA_IO_PREFETCH=1 (Linux, experts.bin or the GGUF
in place) a blob whose pages are not cached is read with one pread of the whole blob into a staging buffer, the
predicted experts of the next layer(s) are read ahead on I/O threads (STRATA_IO_PF_THREADS, STRATA_IO_PREFETCH_DEPTH,
STRATA_IO_PF_AHEAD=0 for demand reads only), and a cached blob is still handed out as the mapping. Same bytes, so the
same output; off by default.
Counters: each request now logs the OS's own view of the file tier (/proc/self/io read_bytes and major faults) beside
the engine's expert reads, and with STRATA_IO_STATS=1 or io prefetch a mincore split of cached and uncached bytes,
read-ahead used / unused / skipped, and the decode's wait in the fetches (--stats prints the same).
blob_stable(): a pointer that is not recycled, for the copies that run asynchronously (adaptive swaps, the GPU cache
fills, the prompt stager).
Asked with Accept: text/plain or ?format=prometheus; every other request keeps the JSON.
Latency histograms, a Grafana dashboard and scrape configs under docs/monitoring.
Only the Prometheus part of #793; the pipelined-window slot allocator needs a 2-GPU check and is left out.
Reads the routing ids first and queues the shared expert afterwards; staging
counts real transfers. STRATA_PREFILL_STREAM_AHEAD=0 restores the old schedule.
--adapt-async 1 no longer drops to the blocking tier with --pipeline-windows 2: the steps that overwrite
something a window in flight may read wait for those windows. STRATA_PIPELINE_ADAPT_ASYNC=0 keeps the
blocking tier there. Opt-in; default output unchanged.
When every placement holds the whole expert profile the predicted decode window is the same for all of them and the winner was floating-point noise: --trim-stage-weights on four GPUs chose layers 6/30/5/7 and read a 32K prompt at 1,750 tok/s, against 2,790 for 12/12/12/12 with the same decode speed and tokens (4x Radeon AI PRO R9700, IQ3_S). Among placements whose window time agrees to 1e-6 ms the smallest slowest stage is kept.
The engine's GEN line takes pin=N: the first N prompt tokens are a shared read-only prefix (a long document many
short questions follow). The prompt is read in two parts at N and the checkpoint there is pinned: retention never
evicts it, a parked conversation holding it stays parked, the session file keeps it, and a later request that resumes
from it does not park the branch it leaves, so N queries keep one prefix. One pin at a time. Without pin= nothing
changes.
The server takes "strata_prefix" in a chat, messages or responses request: {"messages": K}, {"message": I,
"chars": N} or {"tokens": N}, resolved to the longest common start of the prompt's ids. A prefix that cannot be
marked is said in the log and ignored; a malformed field is a 400.
Tests: conv_cache_test (pinned eviction), conversation_cache_test (a pinned parked conversation is not evicted),
conversation_file_test (the pinned checkpoint is the one saved), serve.test_server.SharedPrefix.
Speculative rejection sampling for the MTP drafts of a sampled request. The draft layer samples
its draft from the chain's distribution q (its own Philox stream, counter | 1<<61) and writes
q's (id, probability) list; the verify head computes the target's post-chain p per row from the
split sampler's block lists (penalties, top_k, top_p, min_p, temperature; at most 64 entries)
and accepts draft d with min(1, p(d)/q(d)), else samples the residual max(0, p - q) with its
own uniform. Row t writes the draft on an accept and the residual sample (never d) on a reject,
so the host's accept loop and the commit are unchanged. Rows past the MTP drafts (a lookup
chain's tail) and suffix windows are point masses: accept with p(d), residual p without d - no
q never means q = 0 (the llama.cpp #29924 class). Greedy requests never reach the new kernels;
the nine existing sampler kernels keep identical SASS.
- spec_prob.hpp: the rule, host reference; sample_tokens_spec / spec_draft_sample kernels
- STRATA_SPEC_PROB_GATE=pick, STRATA_SPEC_PROB_DT=x, STRATA_SPEC_MIN_TEMP=x: tunables, off
- spec_prob_test (CPU, 2M trials per case, chi-square 99.9%, controls that must fail) and
spec_verify_parity (GPU verifier vs host reference, token for token)
The per-request totals (drafts accepted / offered) hide where a window ends. With
STRATA_SPEC_DEPTH=1 the serve loop counts, for the windows drafted by the MTP layer, how many
times draft 0, 1, 2 ... was tested (every earlier draft was kept) and how many of those were
kept, and prints one stderr line per request. Off by default; nothing else changes.
Built on our RTX 5070 with nvcc 13.2.51 and 13.0.88 from the same source: with 13.2 the IQ3_S MMVQ product
(iq_multi_parity) and the IQ2_S / IQ3_S MMQ prompt path (prefill_fused_iq_test) are wrong, relative error 0.5 to 1.0; the
other formats are bitwise equal. setup now takes an installed 13.0 / 13.1 over 13.2 for sm_120 (STRATA_NVCC still wins) and
otherwise warns, saying what to install. Recommends, never refuses.
With the fused layout (STRATA_PF_FUSED=1, a chunk of 1024 tokens or more) layer 0 is treated as one the fused path does not
take, so a prompt must stop with the backstop's error instead of overflowing the shrunk buffers. The check that
the guard works at a 16,384-token chunk without needing a pack that really has such a layer.
Streams an answer, stops reading, resends or edits the prompt after 0-0.4 s (or with another request already running)
and checks every answer for real text, not a repeated token. Run the engine with STRATA_DBG_NAN=1 beside it.
--prefill auto:32768 lends 6,000 cache slots to the prompt path; on 31 GB of RAM only 3,700 of their experts stay in
RAM and the rest are read from the SSD on every chunk (prompts ~3x slower than --prefill auto). The engine now says
so at start, with the counts. A warning, never a cap: with the RAM for them the big chunk is 21-35% faster.
The LOW free-VRAM line at READY now also names the explicit --expert-cache (its slot count) and suggests a smaller
one or 'auto', which re-checks the free VRAM after the slots are written. Said, never changed.
gdn_step_commit_kernel reads n_keep from device memory. With STRATA_DBG_GDN=1 a count past the window's rows is
printed and the commit skipped, where a gfx1030 reporter sees a memory-aperture violation. A template parameter
keeps the default kernel unchanged.
fused_layout() and fused_ring() already keep the shrunk GU/H/Xq buffers and the per-layer expert path in step. If
they ever drift (a layer that takes MMQ or the FP16 path at a chunk of stream_all_min() tokens or more while the
buffers are the fused ones), the prompt now stops with 'the fused layout's MoE buffers are too small for layer N'
instead of writing past the buffer: an illegal access, or a kernel that never returns and shows up as
'prefill mmq: iota' one prompt later. Same check in the SYCL copy.
Co-authored-by: Krypto-Whitehat <[email protected]>
A deleted ROCm build named by HIP_PATH / HIP_DEVICE_LIB_PATH / LLVM_PATH, and a TEMP the comgr JIT cannot
create files in, both crash amdhip64 before the engine prints a line. Only a missing path and an unwritable
folder are touched; a healthy environment passes through unchanged.
uv's own managed Python loads pip wheels' libraries where Nix's Python cannot. Changed from the PR as the plan asked: uv is used only on NixOS or when STRATA_UV=1 is set (no silent toolchain change on other systems), and the setup.py hunk is left out. Not run on NixOS here. (PR #662)
"reasoning_close_retry": true in the config: when a reply ends on its end-of-turn token still inside <think>, with no answer and no tool call, the thinking is closed once and generation continues (the engine holds the prefix). Off by default; tested both ways and for a reply that already answered.
PleTable's Mmap+lock path now calls strata::platform::lock_resident (working-set minimum + VirtualLock, chunked per GiB) instead of refusing to start. A lock failure still falls back to touching the pages with a warning, same as the POSIX path.
Changed as the plan asked: only a full VirtualLock counts (a partial one is unlocked again and said so), a warning when the table is more than half of the RAM, the usage text indent fixed. Compiled on Windows here (MSVC, CUDA 13.0); not run with --ple-io ram yet.
(PR #813)
A reinstall put the model on the old drive again because setup keeps the last data folder in its settings file. The troubleshooting table now says where, and that --data-dir, --models-dir, --gguf-dir or deleting the data_dir entry change it.
An explicit budget bigger than the RAM the PC has free (or than its total less ~6 GiB) pushes experts to swap or the file tier; one report went from dead slow to fast by deleting the number. The server now says so at start and goes on (recommend, never force).
find_vcvars looked for versions [16.0,18.0) only, because CUDA 13.0-13.2 refuse a newer Visual Studio. The range now goes up to 19.0 when the toolkit found is 13.3 or newer (the reporter built with VS 18 and CUDA 13.3); older toolkits stay on 2019 and 2022.
Codex's compaction request has tools: [], so its prompt shared almost nothing with the cached conversation and was read again from the start. With "codex_compaction_cache": true the tools of that conversation's last prompt are kept (one entry, keyed by session and thread) and used for the compaction prompt only; the parser and the response keep the request's own tools, and a request that would not fit with them is rendered as sent. Changed from the PR as the plan asked: off by default, the mismatch documented, ported on top of the opt-in #923 and tested over HTTP with a tool call emitted in the compaction. (PR #924)
Codex sends a thread-title request beside each user turn (another session, no tools); running it on the engine replaces the one prefix cache. With "codex_thread_titles": true in the config it is answered at once with the user's line cut to 36 characters (usage 0). Changed from the PR as the plan asked: off by default (the title is not model-written), documented, tested both ways. (PR #923)
Rebased onto the 0.1.40.1 Vision (work_dir, #480 absolute paths). Added tests for the lazy encoder (nothing starts, restart starts it, a failed start is killed); the --lazy help no longer says text-only. A live smoke with a vision config and --lazy is still to run.
(PR #673)
Cut down to reporting only, as the plan asked: max_context is 0 until READY, and /v1/status, /v1/models, /props, /health, /slots and /metrics told a client the model had no context for the whole load. They now use the last known context (reported_ctx). Requests are still checked in prepare() as before.
Co-authored-by: updalla-apshir <[email protected]>
A reasoning-budget continuation (and the reasoning-loop recovery pass) is another native generation, not another API request: each pass keeps its own DONE, the prompt reuse is the first pass's (the generated reasoning is not this request's cached input), and times, tokens and draft counts (offloaded included) are summed. Rebased onto the 0.1.40.1 Service.run. (PR #615)
Cut down from the PR as the plan asked (recommend, never force): an empty --api-key or STRATA_API_KEY is already refused (#213) and a blank config key is refused with #956; an api_key that is present but empty now prints that the server has no key and starts. The setup-side refusal is left out (the final line already says no API key is set).
Co-authored-by: Guillaume PUTIER <[email protected]>
The 0.1.32 image encoder was configured without -DSTRATA_PORTABLE, so the old OFF default built it native
(MSVC + FindSIMD: /arch:AVX512 on the build machine) and it stopped with 0xC000001D on AVX2-only CPUs. A build
without flags is now the AVX2 baseline; the builds for the PC they run on (setup.py's two encoder builds, the
Dockerfile) pass -DSTRATA_PORTABLE=OFF and stay native.
- portable also forces GGML_AVX512*, GGML_AVX_VNNI and GGML_AMX_* OFF, so a -D cannot put AVX-512 back in
(with the old file, -DSTRATA_PORTABLE=ON -DGGML_AVX512=ON gave 10522 zmm instructions; now 0)
- native restores a fresh native configure's GGML_OPENMP / GGML_AVX* when a folder was portable before
- the configure log says which of the two it is
- test: build_engine's encoder build (gpu and cpu) passes -DSTRATA_PORTABLE=OFF
Rebased by hand onto the 0.1.40 setup.py (the Windows MSVC environment of #881 kept). The build script of the ready-made engine zips is not in this repository: it must pass -DSTRATA_PORTABLE=ON for the encoder (the default was OFF until now).
(PR #427)
Short forms with the required fields, a link to the troubleshooting table, and the pull request template. Taken without CODEOWNERS and without the template's checklist (the changes the plan asked for); blank issues stay allowed.
Prompt and output rates for the AMD RX 7900 XTX (gfx1100) beside the NVIDIA rows, with the measurement note. Only the DETAILS tables are taken: the PR's branch also deleted about 190 lines of current text, and the README tables are two fixed machines (RTX 5070 and RX 9070 XT).
tools/quantscope.py (from 1872183316/quantscope, MIT OR Apache-2.0) reads only the headers (a file, a folder, a URL, ms: or hf: repositories); tools/strata_inspect.py compares them with data/gguf_fingerprints.json, the files setup installs. Changed from the PR: the size menu is untouched (its bits per weight line would change the golden output; left out), and the untested verdict says that setup still refuses the GGUF names it lists as unsupported, so the two do not contradict each other. (PR #911)
Setup never switches hosts by itself: auto is Hugging Face, and a download that fails says --source modelscope exists (no probe at start). With it, each file is checked against the SHA-256 ModelScope publishes (a wrong one is deleted; a file with no published hash is warned about), the MTP tensors against the pinned checkpoint's own hashes, and setup says that ModelScope serves current files, not our pinned revision. Changed from the PR: the automatic switch and its 10 s probe are gone.
Co-authored-by: 1872183316 <[email protected]>
A new step 5 tries two tier settings (every window, 80 or 160 swaps, decay 0.97) against the engine's own, after the worker count, one restart each; a setting is kept only when it beats the default by 3%. The tier follows a text over some windows, so this step measures 512-token answers. The author measured +7.9% on a DDR3 Xeon with an RTX 4060 Ti. Applied by a 3-way merge onto the 0.1.40 calibrate.py; the check that it keeps the engine default on a fast-RAM PC is still to run on a GPU.
The generation card and board both run, the lanes now, and the copy rate that implies for a slow link. Changed from the PR: the fewer-lanes note is worded as a hint (a card or laptop can narrow the link while idle), not a verdict.
Co-authored-by: 1872183316 <[email protected]>
The model is the same base model and the same image encoder as UD-IQ4_XS; a user ran it by hand (#967, #971). Setup asks about images for it and warns that we have not run this file with images (no smoke test here: the 111 GB file is not on this PC). MCP's plan and size table follow.
Codex v0.160 sends an input item of type additional_tools, which was refused with a 400. The item is no message; a tools array in it joins the request's own tools, and one without a usable array adds nothing. The item's exact shape is not documented here: only the tools-array form is read.
Both tests upload and memset on the legacy default stream, then launch on
a cudaStreamNonBlocking stream, which does not wait for it. cudaMemset is
asynchronous to the host and a pageable cudaMemcpy may return before the
DMA lands, so the kernels can read buffers still being written. A device
sync before each launch closes it, the same fix 0.1.36 made in the fused
tests. Not reproduced on an RTX 5090: 0 failures in 210 runs of gr_parity
before and after.
Only the first commit of the PR (the stream syncs); the second one (a rewrite of ple_parity without the fixtures) is not taken. The same syncs for the HIP tests were not added (no AMD test machine). (PR #568)
With several GPUs in the config's "gpu" the server appended --layer-split auto, which the engine refuses beside
--peer-device (#531), so the peer tier could not be started through the server at all:
strata generate: --peer-device cannot be combined with --layer-split (use one or the other)
The split (and --split-skip-if-fits) is now added only when --peer-device is not among the args; the start line
names the peer tier instead of a layer split. Tests in serve/test_server.py (PeerDevice), docs/SECOND_GPU.md says
how to configure it.
Ported by hand onto the 0.1.40 engine_args (layer_split_value); the start banner now says the peer tier is in use when the split is not.
(PR #665)
--expert-cache N is N times the largest blob, compared only with the free VRAM
read before the slots are written; the check that runs once they are written is
auto-only. Under WDDM the free figure can be about a gigabyte high and an
over-committed allocation goes to system memory instead of failing, so only the
speed shows it. Measured at 1,048,576 on a 32 GB 4080 SUPER: 11,631 slots with
0 MiB free ran 13.7 tok/s, 11,178 slots (auto) with 217 MiB free ran 102-122,
and raising --vram-reserve-mib moves the free figure because it is deducted
before the cache is sized.
Also records that the GPU process Shared/Dedicated counters did not separate the
slow run from the fast ones on that machine, and adds a Troubleshooting row.
Read against generate.cpp: the second size check after the slots are written is auto-only (the loop breaks on !auto_cache), and the LOW line is printed at 256 MiB.
(PR #799)
The entry (serve/server.py with llama-swap's port, checkEndpoint /health, healthCheckTimeout), the DNS-rebinding
'Host ... is not allowed' trap and its three fixes, /upstream/<model>/ for Strata's own pages, and a note on the
Read against the 0.1.40 server: allowed_hosts and STRATA_ALLOWED_HOSTS, /health, the effort handling and the 403 text match.
(PR #829)
ggml's tiled CPU flash-attention kernel needs the head size (72 in this
encoder) to be a multiple of the vector width: 8 floats with AVX2, 16 with
AVX-512. With AVX-512, which a build from source gets on a CPU that has it,
ggml falls back to a kernel that is several times slower and accumulates in
FP16. The release builds are AVX2, where flash attention is the faster one,
so auto stays the default there.
1024x1024 picture, Ryzen 7 7700X, 8 threads, F16 mmproj: AVX2 8.3-9.9 s
with it, 14.0-15.2 s without; AVX-512 43.5 s with it (26% off the FP32
output), 13.3 s without. --gpu keeps auto, and an explicit --flash-attn
still wins.
Rebased onto the 0.1.40 strata_vision.cpp (the --min-tokens usage line). Only a source build on an AVX-512 CPU changes; the release builds (AVX2) do not. The Zen4 timing is not run here.
(PR #660)
- GGUF_DIR passes setup's --gguf-dir, so a container can use model files the
host already has instead of downloading them again (#252/#444 cover the
native setup only)
- RESIDENT_BUDGET_GIB and KV_STREAMING pass --resident-budget-gib and
--kv-streaming; unset keeps setup's own defaults
- INSTALL.md: a mounted GGUF folder must be writable on the first start
(the Unsloth SHA-256 result is kept as <shard>.done next to each shard)
- INSTALL.md: releasing tens of GB of page-locked RAM on SIGTERM takes longer
than Docker's 10 s default; without --stop-timeout the container is killed
(exit 137)
The INSTALL.md paragraph was added by hand (the PR's numbered list no longer exists); KV_STREAMING maps to setup's current --kv-streaming auto|on|off.
(PR #491)
The bare message cost a full debugging session: on Linux the failure was a
missing libcuda.so.1 (a host driver library the nix loader cannot see) in a
launch environment, not memory pressure. cudaGetErrorString names the real
cause, e.g. "CUDA driver version is insufficient for CUDA runtime version".
Ported by hand onto the 0.1.40 native_head.cpp (the PR's hunk context had moved).
(PR #476)
The PR guarded the mark of a whole shard; mark() itself now warns and goes on, so the SHA-256 result of a verified Unsloth shard and the shared-shard link on a --gguf-dir share do not stop setup either. The step is repeated on the next run.
A whole shard without its .done mark gets one written next to it. On a
read-only --gguf-dir (a network share) that write raised OSError and setup
stopped. It now warns that the mark cannot be written and goes on.
(PR #570)
A splitter may copy general.architecture into EVERY shard of a split while the
qwen4exp.* hyperparameters live only in shard 1 (seen with the Huihui
abliterated re-split of the UD-Q4_K_XL GGUF: general.architecture in shards
2-4, no qwen4exp.* there). NativeDense::load then ran check_architecture on
shard 2 and failed with 'missing qwen4exp.block_count'.
Run the full guard only when the shard is the first of the split (no split.no,
or split.no == 0); later shards take the existing split-metadata consistency
check whether or not they carry the architecture key. Verified end to end:
with this change the engine loads the re-split GGUF (previously: refused),
pack built with iq_pack.py --compat-bf16, coherent greedy output.
(PR #648)
read_expert_profile read the header's version field and then ignored it
((void) version), so a profile of a later format would have been read as a
version 1 layout: the fields after the version are only known to be
n_layers, n_expert, slots, n_ranked in a version 1 file. Refuse any other
version right after the magic, before the geometry check, with the version
found, the version this engine reads, and the file path in the message.
tests/core/expert_profile_version_test.cpp (registered like
expert_profile_save_test, CPU only): a version 1 file loads; the same bytes
with the version field at 2 or 0 are refused naming the version and the
file; a version 2 file of another geometry is refused by its version, not
its geometry; back at 1 it loads again.
(PR #632)
The inputs and the NaN sentinel go up with cudaMemcpy / cudaMemset on the legacy stream, and the kernels run on a
cudaStreamNonBlocking stream, which does not wait for it: a pageable copy or the memset can land after the kernels
started. On an RTX 4090 (sm_89, 128 SMs) that failed the screen ("unwritten or non-finite output") in 3 of 3 runs,
in a different product each time; with the sync before the kernels, 5 of 5 runs passed. The same fix as the fused
tests' (prefill_fused_moe_test.cpp, prefill_fused_iq_test.cpp).
(PR #532)
A request stopped on the solo path and continued in a slot is sent again as its prompt plus what it generated, and the engine holds all of that: the usage split counted the continuation. The first read's reuse is kept, and Anthropic's input_tokens is at least 1 (Claude Code took 0 as not given and counted the prompt twice: Prompt is too long at about half the context). Rebased onto the 0.1.40.1 server and the stop_sequence field.
strata_install refused every vision value but no with backend=hip, although setup's own --vision cpu reads images on the CPU beside the HIP engine. It now follows setup.hip_vision: cpu on Linux is planned, a GPU encoder (yes, gpu) and cpu on Windows get a message that says what works. The Windows AMD engine in MCP's hardware detection is left as it was (it still reports AMD cards on Linux only).
The configured key loses the spaces and line ends a header cannot carry; a key outside ASCII matches as UTF-8 and as Latin-1 (constant-time). Server side only: the web app part of the PR (the UTF-8 header and the key-rejected note) is left out.
reasoning_budget_tokens joins the shared Chat settings (0 means no budget, a whole number of tokens otherwise) and is read after the shared and config defaults are merged, as max_tokens is. Rebased onto the 0.1.40.1 Service.run (force argument).
req.get("defaults") is None both for an absent key and for an explicit null, and null means clients use their own settings again. A body the endpoint does not recognise cleared them all. {"defaults": null} still clears.
A context meter that divides n_prompt_tokens by n_ctx read 0 % against Strata. The key is the running request's prompt size, kept after it ends. Added to the PR: with --batch /slots lists every slot (the view /metrics has) instead of one entry.
Co-authored-by: Zerschranzer <[email protected]>
With "parallel": N the engine runs N requests at once, and /v1/status already says so. /props still said total_slots 1, so llama.cpp clients that read it refused or serialized concurrent runs. It now reports the same count, at least 1.
The caps (supports_tools, supports_tool_calls, supports_parallel_tool_calls, supports_system_role, supports_preserve_reasoning) are rendered once at template load. Changed from the PR: every probe error is swallowed (logged at debug) and turns that feature off, so a custom template that fails in an odd way cannot crash the start.
Co-authored-by: Ivan Tuzhilkin <[email protected]>
A POST behind a relay that sends Transfer-Encoding: chunked had no Content-Length and read as empty (400 No messages provided). _body(), the control-body read and the unread-body drain now decode it (it wins over a Content-Length, as RFC 9112 says). A chunked body is capped at 256 MiB (413; the announced size is checked before a chunk is read) and a malformed one is a 400. Content-Length requests are unchanged. Left out of the PR: nothing; added: the cap, the other body readers, tests.
Co-authored-by: Oscar Rojas <[email protected]>
The vision cache unlinked the oldest .sve on every insert past 64, including a file the same request had not read yet (FileNotFoundError). Vision.encode_all() protects the request's earlier files; the cache shrinks back on the next insert. A cache hit now counts as a recent use.
The solo and admission paths read the engine's ERR line, then sent STOP and drained for a DONE that never comes (it holds the control lines meanwhile). The ERR now ends the exchange.
Rows for #927 (2x RX 6900 XT), #917 (Radeon 8060S, gfx1151) and #955 (Arc Pro B65) in the new COMMUNITY.md index, with notes on what each
build was; the bullets I had added to docs/COMMUNITY_BENCHMARKS.md are removed so the index lives in one place. TRIMMED.md lists the
seven telemetry files left out of the 6900 XT folder.
The gfx1031 entry now lists the RX 6700 XT run (#1027) and the RX 6800M run on Windows (#915, #1078), and says the Windows zip
needs gfx1031 in the STRATA_HIP_ARCHS build list, which the default list has from 0.1.40.2 on.
NVIDIA_V100.md said sm_60 was not measured. It now points to the two-P40 layer split (#1028), the P100 alone and with a helper card
(#1069) and the V100 + P100 pair on 0.1.40 (#1079), each one reporter's run. OLDER_GPUS.md notes the extra host flag a GCC 12.3 +
nvcc build needed in #1074 (one report).
Three fresh engine launches, five fixed tasks at 512 and 7,000 prompt tokens, 640 output tokens: decode medians 40.4-41.3
and 39.8-40.9 tok/s, reference hashes recorded. The six local patches the report built with are kept as data; patches 3, 5 and 6
are ported as source changes in this release, the others were already in 0.1.40. Index line added to COMMUNITY_BENCHMARKS.md.
Co-authored-by: Tim Nevits <[email protected]>
Interactive Q2_0 and UD-IQ4_XS chats against llama.cpp Vulkan, then controlled A-E runs of the gfx1151 hipBLASLt tuning table
(250.9 to 528.9 prompt tok/s), dense MMQ (+0.2%) and the shared-expert decode stream (+2.5%, 8 requests). The build is
0.1.39 plus then-unmerged PRs (#895, #820, a tuning table adapted from #755), not the 0.1.40 release; the index line in
COMMUNITY_BENCHMARKS.md says so and notes that 0.1.40 turned the shared-expert stream off on HIP (#826).
One card, a layer split and the expert-helper mode, Strata 0.1.39 stock and with then-unmerged #835, #849 and #854, original
Flash-Next IQ3_S at 131,072 context; three runs at 4K, 32K and 128K prompt tokens and six recall checks per configuration.
Index line added to COMMUNITY_BENCHMARKS.md with a note that #835 is opt-in in 0.1.40, so the second set is not the
release default. The one-second telemetry.jsonl files (5 MB) are left out, and the README says so.
Co-authored-by: xjc10 <[email protected]>
The ready-made Windows HIP zip had no gfx1031 code, so an RX 6700 XT or 6800M user had to build from source with
STRATA_HIP_ARCHS=gfx1031 (#915, PR #1078; #1027 ran a 6700 XT as gfx1030). setup already knows gfx1031 (AMD_ARCHS, the
wheels index, the PCI id) and the arch gate builds it. This only adds it to the default list the zip is built from.
Prepared, not decided: building the release zip with this list makes it larger (one more arch of rocBLAS kernels and
engine code), and the owner decides whether the 0.1.40.2 zip carries it.
Two pieces of PR #895 that 0.1.40 did not have; the rest of it is in 0.1.40 (gfx1151 in the arch gate, sudot4, setup
and the Strix Halo defaults) or is not taken (the ROCm pin bump for every AMD arch).
- dq_q2_0: the compiler on gfx1151 (ROCm 7.13) folds the half path's negative scale times +0 to +0, as HIP 5.7 did on
gfx1012. The sign-preserving path now covers gfx1151 too. Changes bits on gfx1151 only (the same value the FP32 and CPU
paths give); nothing else is touched.
- Linux HIP, --expert-cache auto: an APU's "VRAM" is system RAM, and the device-free figure counts the whole GPU-addressable
pool, so a cache sized from it can ask the OOM killer for nearly all RAM. The PR capped the cache at the host's free
memory less 4 GiB; here it only prints a warning with both numbers and leaves the cache as asked
(recommend, never force). --expert-cache N sets it by hand.
Compile check only on HIP; the run needs a gfx1151 box.
Co-authored-by: mini <[email protected]>
A plain hipMemset runs on the null stream, which a non-blocking stream does not wait for; on a Radeon 8060S (gfx1151,
Windows) the memset landed after the MMQ kernel and left only the sentinel. hipMemsetAsync on the same stream orders it.
Taken from PR #919's Windows gfx1151 port; its other pieces are not taken here (the Windows carve-out accounting needs
a Windows gfx1151 run).
Co-authored-by: Stormace <[email protected]>
Merges two community reports of the same mechanism (GPU queues suspended while the kernel reclaims KFD userptr pages)
into one short section in AMD_HIP.md and one line in TROUBLESHOOTING.md: HSA_USERPTR_FOR_PAGED_MEM=0 (two R9700, ROCm
7.2) and GPU_PINNED_MIN_XFER_SIZE=1048576 with --mmap-experts (RX 6800), each labelled as one machine and not
confirmed, with the MemoryMax versus MemoryHigh note. Condensed from the two PRs.
Co-authored-by: zengyishou <[email protected]>
Co-authored-by: YamineRL <[email protected]>
setup listed the RX 7600 XT as "not supported" although the arch gate and the Windows zip already build it, and one
RX 7600 XT ran the engine (Swift IQ3_XXS, 146 requests up to 51K context, about 43 tok/s decode, 250-300 tok/s prompt,
unvalidated; docs/AMD_HIP.md has the report. The two setup tests that expected "not supported" now expect it listed.
- conversation_snapshot_test: the pinned host buffers are freed with sycl::free (they come from USM, not malloc).
- STRATA_VERIFY_NO_HOST with experts that are neither in VRAM nor mirrored now refuses to start, instead of a warning
followed by a generation with no safe host fallback.
- On QUIT the direct-access USM host mirror is released (gguf_src.close()) before the process leaves; a failure to
release is reported and exits 1.
From the Arc Pro B65 bench report (timnevits), where they are shipped as patch files; the other patches there were
already in 0.1.40. SYCL compile check only.
Co-authored-by: Tim Nevits <[email protected]>
Only subgroup row zero of the 32x8 block reaches the router's synchronization in route(); the other seven rows return
before it. A work-group barrier there is divergent, and on an Arc A770 the kernel hung. The surviving row now meets a
subgroup barrier. Also recorded in sycl/tools/fixups.py so a regenerated tree keeps it.
Salvaged from PR #797 (the rest of that draft is not taken: its build fixes were already in 0.1.40). An A770 on i915
reported decode only with this change (23.0 tok/s with MTP, 8.65 without). Needs an Arc owner to test on the current
tree; SYCL compile check only.
Co-authored-by: Txgom2 <[email protected]>
tune_hipblaslt on the gfx1201-100500 table's 32 shapes, on a Ryzen AI Max+ 395 (Radeon 8060S, gfx1151) with ROCm
10.2.0a20260930 (whl-next, hipBLASLt 1.5.0 efafc1c1). 3.9-15x faster than plain hipBLAS per shape; prefill with
STRATA_HIPBLASLT_VERBOSE=1: launches=3138 fallbacks=0.
Only the table is taken from PR #944. Its gfx11 WMMA attention and scorer duplicate the kernels already in 0.1.40, and
its "gfx11" prefix gate would enable gfx1103 and gfx1152, which 0.1.40 leaves out on purpose. To be re-measured on the
gfx1151 box.
Co-authored-by: Nathan Bradley <[email protected]>
On an Arc Pro B60 a system-scope atomic load of host USM is still served
from the GPU's cache, so a device spin never sees the host's store and
runs to kSpinMax every time. Read the flag through an annotated_ptr with
an uncached L1+L3 read_hint after an acquire fence instead. Bounded
ping-pong: 263 ms -> 2.4 us per round trip; per-layer waits on a B60
with experts in the host mirror: 8.7-10.5 -> 19.3 tok/s decode.
-DSTRATA_DOORBELL_ATOMIC_LOAD keeps the old load.
Based on konijiwa110's PR #742. Below sm_80 there is no TF32 tensor-core scorer, so the prompt path scores the QSA
blocks on the warp kernel: 5 shuffles per 4 products, ~1 TFLOPS on an RTX 2080 Ti, 24% of a 128K prompt. With
STRATA_SELECT_SIMT=1, qsa_block_scores_tc runs an FP32 tiled GEMM there (rows = query x indexer head, K = 128 through
shared memory, 2 queries x 4 heads x 4 blocks per thread, relu per head summed in registers), ~6.9 TFLOPS by the
author's measurement (128K prompt 1,016 to 1,242 tok/s, 248K 715 to 952 on one 2080 Ti).
The tiled kernel sums in another order than the warp kernel, so its scores are not bitwise equal. The PR made it the
default below sm_80; here it is opt-in, so default output is unchanged until an interleaved A/B of at least 10 pairs
supports a default. The tail block keeps the warp kernel's arithmetic. STRATA_SELECT_OLD=1 still forces the warp kernel.
qsa_select_bench is now also a CUDA test: at 32K, and at 131K with STRATA_QSA_WARP=1 and STRATA_SELECT_SIMT=1 so any
card runs the FP32 tiled scorer (FP64 gate, selections compared with the warp kernel's).
Co-authored-by: konijiwa110 <[email protected]>
On HIP the default attention kernel's score loop is bound by the LDS pipe on gfx1030: its 12 x 5 __shfl_xor_sync per
cell compile to ds_bpermute_b32 and share that pipe with the shared-memory q reads. attn_chunk_kernel_pre75 (PR #540,
bit for bit the default kernel) is now built for HIP wave32 (not the gfx906 backend) and run on gfx103x by default:
- NC_ template parameter, cells per step: CUDA 2 as before, HIP 8 (on gfx1030 every step up in NC was faster although
occupancy fell from NC 1: the q reads bound it, not occupancy)
- DPP_ template parameter: the butterfly's lane exchanges (reduce12, the softmax max / sum) through DPP row_xmask within
a row and v_permlanex16 across rows instead of ds_bpermute; same values, same pairing order, unchanged bit for bit
- pre75_attn() on HIP: gfx103x per device; STRATA_ATTN_PRE75=0 off, =1 also another wave32 card
CUDA is unchanged: the non-experimental build does not build the kernel, and the template defaults equal the old
constants with shfl_x<false> = __shfl_xor_sync.
RX 6900 XT, IQ3_S, prompt tok/s at 9.4K / 34.7K / 105.8K: 441 / 467 / 460 -> 459 / 494 / 488; with #835's FP16 GEMMs
746 / 917 / 928 -> 801 / 1,022 / 1,041. Teacher-forced log-probabilities against the default kernel: KL 0, argmax 100%.
bench, docs: the pre75 attention kernel on RX 6900 XT - measurements, exactness, decode A/B; OLDER_GPUS.md notes
bench/results/2026-10-04-rdna2-pre75-attention: the rig, the compiler output (registers, occupancy, ds_bpermute counts)
that motivated NC 8 and DPP, prompt speed on this branch (+4.2 / +5.7 / +6.1% at 9.4K / 34.7K / 105.8K tokens), the
step-by-step NC / DPP series on the FP16 prompt path, the teacher-forced exactness check (KL 0 over 1,578 positions),
a decode A/B (0.3-0.6 ms shorter windows, two rounds), the limits, and the configuration used.
docs/OLDER_GPUS.md: the RX 6800 / 6900 row says this kernel is the default on gfx1030 and names the switch; the A/B
switches line says what STRATA_ATTN_PRE75 does there.
Follow-up to #600 (fks's mma.m8n8k4 prompt attention for sm_70).
The kernel splits q (for q.k) and p (for p.v) into FP16 hi + lo halves
and runs both products into the same accumulator. Volta's tensor cores
don't keep full FP32 precision while they accumulate, so once the hi
products are in the accumulator most of the much smaller lo
contribution gets truncated away, and the split buys less than it
should. Giving the lo products their own chain and adding the two in
FP32 at the end fixes that.
qsa_prompt_attn_parity, one V100-SXM2-16GB, 2,048 queries, error vs the
FP64 reference (the FP32 kernel in brackets):
ctx 8,192 5.88e-6 -> 3.02e-6 (2.14e-6) 11.94 -> 12.20 ms
ctx 32,768 5.08e-6 -> 2.33e-6 (2.37e-6) 12.20 -> 12.38 ms
ctx 131,072 5.35e-6 -> 2.75e-6 (1.81e-6) 12.34 -> 12.51 ms
Registers unchanged (166 / 226 / 163 for modes 0 / 1 / 3), no spills.
docs/NVIDIA_V100.md updated for the new error level.
gfx1010 (Navi 10) is measured on an RX 5700 XT: 8 GB card, Coder IQ1_M, 32K
context, prompt 115 / 144 / 149 tok/s and decode 18.0 / 20.3 / 23.3 tok/s at 4K /
16K / 30K prompt tokens, 6 of 6 needle checks at 8K and 30K, 61 of 61 ctest on the
card. Nothing beyond the arch list is needed: the portable scalar dp4a that main
falls to measured 162.7 / 24.3 against 158.3 / 24.0 with the __gfx1012__ SDWA
control (#442) - inside the run-to-run spread - and 16 of 16 HIP tests pass on it.
gfx1011 (Navi 12) is untested on hardware: same RDNA1 ISA, builds with 0 errors on
ROCm 7.14's gfx101X-dgpu wheels.
Two box facts, for whoever picks this up: hipBLASLt 1.4.0 refuses all 26 of the
engine's GEMM shapes on gfx1010 (heuristic_count = 0), so there is no tuning table
and prompts run on plain hipBLAS; and in a PC that also has an AMD iGPU, ROCm's HSA
runtime segfaults in GpuAgent::InitDma while enumerating the pair - below HIP, so
HIP_VISIBLE_DEVICES cannot reach it - and the engine must start with
ROCR_VISIBLE_DEVICES naming the one KFD node (rocminfo for the index).
Co-authored-by: AcckiyGerman <[email protected]>
CMake caches CMAKE_HIP_FLAGS_<CONFIG> while it enables HIP, from the
values its platform files set once the compiler is identified. When
HIP cannot compile CMake's identification source, that configure
caches them empty and then stops; every later configure of the same
build folder keeps the cached values, so the build that finally works
compiles every HIP source without -O, i.e. at -O0, while C and C++ get
-O3 -DNDEBUG. setup configures build-hip every time, so one failed
attempt is enough.
That happened on an RX 6800 (Windows 11, HIP SDK 7.2, CMake 4.0.1):
the first attempt failed on Visual Studio 2026's <cmath>, the next one
passed with the include fix, and the engine (0.1.30 with the Windows
build of #247, --expert-cache 2048) decoded at 0.24-0.27 tok/s with
7.1-7.5 s per decode window. STRATA_VERIFY_PROFILE put the time in
the GPU's own stages (per window: q8+qkv GEMVs 1468 ms and out-proj
849 ms in the DeltaNet layers, the LM head 540 ms); the waits for flags
the CPU raises were 41 ms of its 8.4 s per window. With the flags back:
59-62 ms per window, 28.1-28.3 tok/s.
After enable_language(HIP), hip_backend.cmake gives each empty
per-configuration HIP flag the value CMake computed for the compiler
(CMAKE_HIP_FLAGS_<CONFIG>_INIT), which is what a fresh build folder
gets. Flags with a value, from CMake or the command line, are left
alone; one set empty on purpose is filled too.
Reproduced on Windows (HIP SDK 7.2, CMake 4.0.1) with a first
configure that failed on --rocm-path flags split at the space in
C:/Program Files, and on Linux (CMake 3.28, TheRock's ROCm 7.10 clang)
with one that failed on a --rocm-device-lib-path that does not exist.
After a working second configure in the same folder,
CMAKE_HIP_FLAGS_RELEASE stays empty and native_mmvq.cu compiles
without -O; with this change it gets -O3 -DNDEBUG. A fresh folder gets
-O3 -DNDEBUG either way.
setup.py's engine fingerprint now covers cmake/ as well, so a change
there alone (this one) recompiles a locally built engine on its next
start; that includes NVIDIA engines compiled here.
The batch slots' drafters share the draft-vocab head but never copied its ggml type; it stayed -1 and the first
admission of --batch-mtp exited with "unsupported native MMVQ GGML type".
cudaGetDriverEntryPointByVersion arrived in CUDA 12.5; the CUDA 12.2-12.4 engine build stopped in vmm.cpp. Same
fallback expert_cache.cpp already carries, and the sticky error of a failed lookup is cleared.
Co-authored-by: zhouxihong1 <[email protected]>
The STRATA_HIP_GFX906 build defines STRATA_HIP_GFX906, not STRATA_USE_HIP, so three
sites that gate on STRATA_USE_HIP still take the CUDA branch and fail to compile:
1. fused_gr.cu: a bare 'return;' in the bool fused_gr_read_multi -> 'return false;'.
2. mtp.cpp: the MTP stream-priority block (cudaDeviceGetStreamPriorityRange) is gated
on !defined(STRATA_HIP_GFX906) too.
3. vmm.cpp: the <cuda.h> include and the CUDA driver VMM code get the same gate.
All three are inside #if blocks that only the gfx906 build reaches.
vmm.hpp only includes <cstdint> and <vector>, so the unqualified size_t is not
declared on toolchains that do not expose ::size_t through those headers
(GCC 12.2, Debian 12). Qualify it as std::size_t, as expert_cache.hpp does.
Bench folder ported from PR #1078 by Hugua700, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #1062 by fioruccione, authorship kept.
Trimmed for 0.1.40.2: 7 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #1028 by sarge18, authorship kept.
Trimmed for 0.1.40.2: 8 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #1027 by GPF, authorship kept.
Trimmed for 0.1.40.2: 6 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #1016 by avarakin, authorship kept.
Trimmed for 0.1.40.2: 21 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #995 by lfontanez, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: bench/results/2026-09-29-speed-0126/README.md, bench/results/2026-09-29-speed-0126/matrix.json, docs/COMMUNITY_BENCHMARKS.md, tools/integrate_speed_0126_shin_blackmamba.py.
Bench folder ported from PR #832 by Zauberio, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #466 by gravitomagnetic, authorship kept.
Trimmed for 0.1.40.2: 1 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #993 by obiscr, authorship kept.
Trimmed for 0.1.40.2: 1 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #913 by 1872183316, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #902 by Mitsuasa513, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #882 by T-Crypt, authorship kept.
Trimmed for 0.1.40.2: 1 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #850 by christopherrobertbrooks-tech, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #834 by T-Crypt, authorship kept.
Trimmed for 0.1.40.2: 4 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #823 by christopherrobertbrooks-tech, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #815 by did-technomancer, authorship kept.
Trimmed for 0.1.40.2: 3 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #780 by 1314521gjy, authorship kept.
Trimmed for 0.1.40.2: 2 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #757 by 30crows, authorship kept.
Trimmed for 0.1.40.2: 4 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #745 by xyzzing, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #740 by jase100k, authorship kept.
Trimmed for 0.1.40.2: 3 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #721 by zimuhuan-code, authorship kept.
Trimmed for 0.1.40.2: 8 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #674 by pertain99, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #624 by Dmitry-B, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #618 by HarukiOtaku, authorship kept.
Trimmed for 0.1.40.2: 2 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #521 by lijieming, authorship kept.
Trimmed for 0.1.40.2: 2 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #483 by Efs-O, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #440 by Ambolio, authorship kept.
Trimmed for 0.1.40.2: 4 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #418 by qni-live, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #416 by pipeob0, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #404 by rwkeyes, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #811 by steve8697, authorship kept.
Trimmed for 0.1.40.2: 27 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #777 by steve8697, authorship kept.
Trimmed for 0.1.40.2: 21 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #758 by xxDoman, authorship kept.
Trimmed for 0.1.40.2: 21 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #698 by CYoung83, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #628 by Yasei-no-otoko, authorship kept.
Trimmed for 0.1.40.2: 30 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/AMD_HIP.md.
Bench folder ported from PR #508 by R6DJO, authorship kept.
Trimmed for 0.1.40.2: 40 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #433 by brenoperucchi, authorship kept.
Trimmed for 0.1.40.2: 2 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #417 by mlfather, authorship kept.
Trimmed for 0.1.40.2: 162 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Bench folder ported from PR #707 by noahark, authorship kept.
Trimmed for 0.1.40.2: 0 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #499 by homeofe, authorship kept.
Trimmed for 0.1.40.2: 2 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #469 by homeofe, authorship kept.
Trimmed for 0.1.40.2: 4 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
Bench folder ported from PR #389 by shrisha108, authorship kept.
Trimmed for 0.1.40.2: 1 raw dumps/logs/binaries left out (listed in TRIMMED.md of each folder).
Not ported: docs/COMMUNITY_BENCHMARKS.md.
The live smoke showed the same quoted-example problem in the answer: a model that wrote the call format in a
fenced example after </think> got a real tool call. The content parser now uses the fence / inline-code
tracking of the reasoning rescue: an opener inside a ``` or ~~~ fence (tracked across stream chunks) or inside
inline code stays text, the decision depending only on the text before it, so streamed and whole outputs agree.
Openers at the top level fire as before, including one after a sentence on the same line (none of the 37
corpus specimens needed that, but the old tests do and nothing argues against it). A closed call at the end of
an answer cut by max tokens stays a call.
Tests: quoted calls stay text and real ones fire at every chunk split; OpenAI (whole and streamed), Anthropic
and Responses over HTTP.
Found in the live smoke: an old request that learns its engine died while restart() has already taken the
process (self.proc is None) failed in death_note() with an AttributeError, and its client got no reply at
all. death_note(), alive() and _send() now read the process once and treat None as a dead engine.
Test: after close() the death note is a sentence, alive() is False and a send raises EngineDied.
Found in the live smoke of #1012: a long prompt being admitted when the engine is killed read its own
end-of-engine line (EngineDied), and its cleanup then drained the control lines for up to 300 s for an
end line that had been read already; the client got its 503 only after that. The cleanup now skips the
STOP and the drain when the engine process the request was sent to is gone (the server counts its engine
processes in gen, and a request notes the one it has the control lines of), and the drain polls once a
second and gives up as soon as that engine is gone.
Tests: a long read killed alone, and with short requests arriving meanwhile (it gives way and is admitted):
every request ends in a 503 within seconds, and the next request restarts the engine.
Versions are read as their first three numbers everywhere; the one place that was not (the version compiled
into an engine binary, `engine=0.1.40.1`) now accepts an optional fourth number. A test pins that a hotfix
tag keeps finding the engine zips: the checkout's v<project version> release first, the latest release second.
StrataEngine.restart() runs __init__ again, which made a new condition variable, control lock, wait counter
and wait list while requests were still waiting on the old ones: they hung for good, or failed with
`list.remove(x): x not in list`. The admission state is now made once and kept; restart() wakes the waiters
when it ends. A request that waits for the control lines now ends with a clean EngineDied (503, "not sent")
when the engine is gone and nobody is starting it, keeps waiting while a restart is under way, and goes on
with the new engine when it is up. The lock is checked again after it is taken, a slot that frees itself
after a restart keeps the lists of its own process, and the wait counters cannot go below zero.
Tests (fake batch engine behind the real StrataEngine and Service): waiters during a restart are served by
the new engine (2 and 4 slots, same condition variable and lock), get a clean 503 when the engine stays down,
and when the restart itself fails; the next request restarts and is answered; the solo engine is unchanged.
The #804 rescue turned any declared <tool_call> in the reasoning into a real call, so a call that was only
quoted (in a code fence, mid-sentence, in inline code, followed by more prose) ran, and so did one in a reply
cut by max tokens (47Hunter47's `rm -rf build` case). Now it must start a line (after optional whitespace),
be outside a code fence (``` or ~~~, tracked across chunks) and inline code, close with </tool_call>, and have
only whitespace or more calls after it before the end of the turn or </think>. Such calls wait until then,
and become reasoning text again when the turn ends by length, cancel or error. The decision is the same
streamed or whole. Declared calls beside an undeclared wrapper are still delivered, the wrapper stays
reasoning.
The server tells the parser how the turn ended, logs how many calls were read or kept as text, and counts
the read ones in the metrics totals (tool_calls_from_reasoning).
Tests: 37 recorded and constructed specimens (PR #525's corpus and 47Hunter47's 12 live strandings; 16 act,
21 do not) at chunk sizes 1, 2, 3, 7, random and whole; fence and opener split at every position; OpenAI
(whole and streamed), Anthropic and Responses over HTTP.
With "parallel": N, a batch window that fails prints its reason as an ERR
line on stdout and the engine exits with code 1. At that point every
request reads its own slot's lines, so nobody reads that ERR line, and the
server's note fell back to the log's last "strata" line (a stats line in
#997), or to "the usual cause is running out of RAM" without a log.
The pump now keeps an ERR line that was the engine's last stdout line, and
death_note gives it first. test_parallel's fake engine can fail its first
window over two slots: the note names the error.
(cherry picked from commit 9c5066303e)
dma_batch_parity.cu failed to compile on Linux/WSL (nvcc 13.0 with g++): 'template argument 2 is invalid' at the
std::array<T, cap> locals of copy_expert_blobs, whatever the form of the size (int, size_t, a literal). The engine
objects and the Windows build were not affected; the Linux tests build was.
vmm_test failed on the RTX 5070 / driver 610.47: the pageable host-to-device copy was still in flight when
a.unmap() pulled the chunks out from under it, an illegal memory access that stuck to every later call (the last
two checks, the 8-chunk free-VRAM gain and cudaGetLastError, failed as a result). It was not a VRAM leak:
released memory comes back at once (18 MiB gained for 9 chunks of 2 MiB, polled 10 s). generate.cpp already
synchronizes before it unmaps.
gfx1151 is now a supported AMD arch (TheRock gfx1151 index; HIP build on Linux, docs/STRIX_HALO.md). The check is exact (gfx_arch_is, as the
engine's gfx_arch.hpp): gfx1150 / gfx1152 / gfx1103 are named as Strix Point / Krackan / Phoenix-Hawk Point and stay unsupported. PCI 1002:1586
(one id for the 8060S / 8050S / 8040S) and the Windows adapter names map to gfx1151; the KFD simd_count names the 8060S / 8050S / 8040S.
Unified memory: the BIOS carve-out + the shared pool (GTT; Windows half the RAM) size the context; only the carve-out counts as extra
room in the low-RAM mode. A discrete card is picked before an APU. A Windows zip without gfx1151 code is refused with the reason.
UD-IQ4_XS is the recommended model from ~80 GB of memory; setup changes no host setting. tools/test_setup_strix_halo.py uses Aurora's real sysfs.
The default STRATA_HIP_ARCHS of tools/hip/build_windows.bat gains gfx1151. The ROCm 10.2.0a20260930 Windows SDK compiles the strata target for
gfx1151 (stream 140-j built it on this PC) and its rocm-sdk-device-gfx1151 wheel carries the rocBLAS kpack and the hipBLASLt kernels
package_windows.py copies, as for gfx1201. Unvalidated on a Windows Strix Halo.
Bitwise on the 5070 (identity gate: Q2_0 and IQ3_XXS identical 10/10 with it on against off, same exe), but the 10-pair
interleaved decode A/B (Q2_0, default config) shows no win: B/A medians 0.97 / 0.97 / 0.99 on the three greedy requests.
On the HIP build the fused kernel is off until check 6 passes there, so check 6 reports SKIPPED unless STRATA_NORM_ROPE=1 asks for it. (gfx1151 on Aurora, 0.1.40: with STRATA_NORM_ROPE=1 it does not pass - about 7% of the rotated elements differ - which is why the kernel stays off on HIP.)
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
On gfx1151 the shuffle reduction of the sum of squares inside the divergent-looking branch rounded differently for ~3% of the outputs than the single-token kernel (gdn_parity section 5, Aurora); the base kernel had the shuffles outside it, and with them back out the multi kernel is bitwise the single one for T=1..8 on gfx1151. No change for CUDA.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
Found by native_multi_parity on gfx1151 (Aurora): the float4 combine differed from the scalar kernel there, because the first pair's other product was contracted into the FMA. Written as fmaf per expert it is the pinned contract (first product rounded, FMA per expert in order) on every compiler.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
For a window of more than one token the drafter routes with one multi-column router GEMV and one multi-row top-10 and combines all rows in one launch (verify already did; each row is bitwise the single-token call, native_multi_parity). Needs the native router (512 experts, k = 10) and the native combine; otherwise the per-token calls stay.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
quantize_q8_0_scaled_kernel reduces the block's max with a warp shuffle (one warp per 32-value block) instead of one thread walking the block; fmax ignores NaN the same way in both. moe_grouped_s2 fuses SwiGLU with that quantize and writes the activation back to gate_up, so the grouped parity checks see the same buffer. STRATA_S2_SWIGLU_Q8=0 keeps the two launches; a static_assert pins FF % 32 == 0 for the fused kernel.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
STRATA_SH_FORK_LATE=0 forks at the top as before. The other two commit-4 changes of the PR are not taken: the shared-expert
Q8_1 reuse is already in f-140 as the opt-in STRATA_VERIFY_QDEDUP (and the fused q8_1 store), and the PLE native-key batch is
already in f-140 as the opt-in STRATA_PLE_BATCH=1; both would have changed output on the native packs by default.
Dropped on purpose: STRATA_VERIFY_PRELAUNCH and the dead copy_rows_from_mapped<ZERO_HITS>.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
MtpDrafter::draft now runs the round for T = a + 1: the catch-up writes the K/V of the accepted cells [0, a) and the full layer covers row a, instead of writing all T cells and rewriting cell a. No later attention reaches a cell past the one being drafted and the next round writes it first, so the draft tokens and probabilities are unchanged. draft_first (a = 0) no longer writes its repeated rows at all.
STRATA_MTP_CATCHUP_ALL=1 restores the whole-window catch-up (and the old staging sizes).
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
iq_kernels.cu: row_dot_80_sub16 (two rows per warp, all lanes active) for the n_embd == 2560 gate/up of every split format, exact-N instantiations of the multi-column MMVQ (launch_mmvq_multi_nc), the 3-entry tail in the grouped gate/up and down kernels. native_mmvq.cu: SmallTraits for Q4_0, Q5_0, Q8_0 and IQ4_NL decode the weight chunks and block scale once per (row, block) and apply them to every activation column with the arithmetic of small_q8_dot, and the multi-column launch runs two rows per block for any K.
Kill switches: STRATA_NO_SUB16_GU=1 (sub-warp gate/up/down), STRATA_OLD_IQ_MMVQ=1 (per-column kernels, existing), STRATA_NO_MMVQ_ROWS2=1 (one row per block at large K). mmvq_multi_parity, iq_multi_parity, native_grouped_parity and native_expert_parity pin the columns to the single-column results.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
gr_read_multi / gr_write_multi (and native_gr_rms_norm_weighted_multi, native_gr_pre_gated_multi, native_gr_post_multi under them) run T tokens in one launch per step; the verify window's PLE writes, last-layer write and final mixer, and the MTP drafter's write and final mixer use them. The BF16 MMVF multi kernel and the GR up / staged down kernels are instantiated per exact T (1..6, generic above), the BF16 weight tile is unpacked once for all tokens, with the same FMA order as before. STRATA_NO_MULTI_GR=1 restores the per-token calls and the generic kernels.
STRATA_GR_DOWN_MAX4 keeps its meaning from the previous commit (sm_120 default on, explicit setting decides); this commit does not change that default.
gr_multi_parity (new, memcmp): gr_read_multi (with and without injection) and gr_write_multi (in place and out of place) against the single-token calls at the real geometry, T=1..8.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
The 4-token accumulator variant of the fused GR down read keeps the same tile, block size, accumulation order and
variant as the kFusedGrMaxT one, so its bits are the same; #832 measured +2.8-3.5% decode with it on an RTX 5070.
Unset, the engine now turns it on for compute capability 12.0 only (decided once per device), and off on every
other card and on the HIP build, where it was not measured. STRATA_GR_DOWN_MAX4=0 or =1 still decides explicitly.
native_qsa_rms_norm_rope does norm<256> and the rope in one launch with the same FMA order as the pair (register barriers keep nvcc from contracting across the norm/rope boundary); rope_parity check 6 compares it with native_qsa_rms_norm_weighted + native_rope_apply by memcmp for head_dim 128/256, strided q|gate and contiguous inputs, unscaled and YaRN. The in-place rope also launches one 32-thread block per row.
Verify's norm_rope and q split, and the MTP drafter's q path (all T tokens at once; K for a one-token window) use it. native_norm_rope_usable() gates it: on the CUDA build it is on unless STRATA_NO_NORM_ROPE=1; on the HIP build it is off until rope_parity check 6 passes on AMD hardware, and STRATA_NORM_ROPE=1 turns it on there. The MTP q path's split, FWHT and gate-apply now run over all T tokens in one call each (row-independent).
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
The plan kernel grouped duplicate experts and numbered the groups with thread 0 walking all entries; every entry now finds its group in parallel and a packed (entries << 16 | groups) warp prefix scan numbers them. The residency check, the empty plan and *plan_err of #871 are kept as they are. static_asserts pin the 4-warp / packing limits.
resident_plan_parity (new, memcmp): n=1..80 entries over pools of 3..512 experts with and without a slot table, with and without a skip word, plus the non-resident case, against a host replay of the plan's contract.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
kv_append_q8_steps / kv_append_q4_steps append n cells in one 3D-grid launch per pool, and native_qsa_indexer_append_steps walks the n indexer cells in order inside one CTA (one barrier between cells). Verify record_window, the commit graph and MtpDrafter::record_forward use them; a multi-slot batch keeps the per-token loop (each row has its own K/V). STRATA_NO_BATCH_KV_STEP=1 restores the per-token calls.
The kv single-cell calls are now the n=1 case of the same kernel. kv_steps_parity (new, memcmp) covers INT8, Q4_0 and the hybrid's folded single-plane calls for n=1..8; qsa_parity gains append_steps against the sequential indexer append at strides 1 and 5 under none/linear/YaRN.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
native_router_top10_multi runs one warp per token (8 per CTA) instead of one CTA per token; native_moe_combine(_multi) takes a float4 kernel for k=10 with a shared row on aligned buffers; the shared expert's sigmoid-scale is vectorised. Same arithmetic order, so same bits.
New native_multi_parity (memcmp): the multi router against the single call for n=1..19 on random, tied and NaN logits, and the combine against a host replay of its contract for the float4 and scalar paths.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
gdn_ab_multi is instantiated per T=1..6 (generic above) with the weight unpack hoisted out of the token loop; gdn_step_norm_multi prefetches the next token's q/k while the current one runs, and when no state is written and every token is an output (the verify half) it skips the per-token bookkeeping. gdn_step_norm_kernel splits its reduction buffer so one barrier goes.
gdn_parity gains section 5: for every T = 1..8, gdn_ab_multi against fused_gdn_ab, and gdn_step_norm_multi (verify half, commit half, t_out_begin, state-only commit) against the single-token kernel chained, compared with memcmp.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
A CTA of the fixed 256-CTA grid whose first block is past the batch's largest n_bid returns before staging the queries. Same scores, same bits.
STRATA_QSA_EARLY_EXIT=0 restores the old staging. kv_stream resolve_kernel: only threads that looked something up add to the shared counter.
Ported from #783 (stuchapin909, head 4327807), split per piece.
Co-authored-by: stuchapin909 <[email protected]>
On the RTX 5070 (Windows, 4 KB pages, the whole-arena registration refused) the deferred per-layer registration made
greedy decode 4-10% slower than registering the sliced arena after the load (0.1.39's order), with identical output
(Q2_0: STRATA_DEFERRED_REGISTER=0 gave +10% on the same build; 12-pair A/B against 0.1.39 flat after).
#826 turned the fork off on HIP (it costs discrete RDNA cards). On Strix Halo's unified memory it pays, measured on the
merged 0.1.40 code (stream 140-m). The switch is read on first use, after the arch table ran.
(b & 0x08080808) * 0x1E reaches 0xF0F0F0F0 and overflowed a signed int; on sm_60 (strata_dp4a
fallback) the compiler turned that into garbage: native_expert_parity q4_0/q4_0 and q4_0/q4_1
gpu rel 1.7e4 and 6.8e2. Unsigned wrap is defined and gives the same bits.
Claude Code starts its system prompt with
`x-anthropic-billing-header: cc_version=2.1.170.bf4; cc_entrypoint=sdk-cli; cch=b145e;`.
cch changes on every request and the version's 4th part on every session. The
template renders the tool list first, so every turn's prompt differed ~22K
tokens in and only the last 16K checkpoint was reused. Both stamps are pinned to
f's, as llama.cpp does (ggml-org/llama.cpp#21793); only a header at the very
start of the system text is touched.
Port of PR #931 onto the #554 vision-tag marks: literal_tags() adds the tokenizer's
control tokens to LITERAL_TAGS, longest first; a prompt with none of them is
rendered and encoded exactly as before.
Co-authored-by: Chanyeong Lim <[email protected]>
Step 5 and step 7 decided these two settings inline. A front-end that has to say what a model would change needs the
same answer, so the rules are named and setup calls them itself: low_ram_wanted is the old `budget is None and (...)`
(budget is set only for a size that has a RAM budget), and the streaming branches keep their messages and say they
are kv_streaming_wanted.
Behaviour unchanged, and checked: test_setup_risk's new RulesAgreeWithTheWrittenConfig runs 54 mocked installs over
three sizes, two contexts, three --kv and three --kv-streaming choices, and 24 more over two PCs and four --low-ram
choices, and compares each written config's --kv-resident / --resident-experts / --mmap-experts with the function.
Co-Authored-By: Strata qwen3.8-flash-next-iq3_s <strata@localhost>
AI-developed: written by an agent (pi) running on Strata's own local model, qwen3.8-flash-next-iq3_s; tested by hand on an RTX 5060 Ti 16 GB / Ryzen 7 5700X3D / 64 GB PC with engine 0.1.39
An existing pack experts.bin (low-RAM copy or the AVX-512 Q2_0 conversion) and an
existing MTP draft layer need no new room. A re-run of setup on an installed model
used to demand ~arena_gb + 9 GB free and fail on a nearly full models volume.
The RAM line was [ok] and the verdict 'This PC can run Strata' although every size
said 'does not fit'. Now the line is [!], the verdict says no size fits, and the exit
code is 1 for scripts. Check only: --model NAME --yes still installs one anyway.
MIN_ENGINE >= RESIDENT_SPLIT_ENGINE now turns on the resident low-RAM mode on a layer split: the golden
baseline records the new 2-GPU configs (both cards, --resident-experts, --remote-expert-opt), and the
tests of the pre-0.1.40 behaviour pin MIN_ENGINE to 0.1.39. CMakeLists' project version is what
source_version() reads, so a compiled engine must say 0.1.40 to satisfy MIN_ENGINE.
- bench_tips(): a warning for a fixed --expert-cache on Windows (#780#781), --prefill auto:32768 only as a tip
with 96+ GB RAM and a warning below 64 GB, STRATA_RESIDENT_HEADROOM_GIB=6 on resident configs with <= 48 GB,
--conversation-cache-mib 8192 for agent clients when 24+ GB stay free, --vision cpu on <= 12 GB cards.
- a 2-socket tip for --pool-workers (one socket's cores - 1).
- calibrate also tries a quarter of the default, P-cores - 1 and one socket's cores - 1.
No default, config or engine argument changes; --mtp-window left out on purpose.
--host-core: the host thread's core, first (the default) or last, the workers on the others.
Windows sends a GPU's interrupts to one logical processor, usually the first, and every copy that lands raises one
(~13 us in the ISR and a DPC): a host that spins on the GPU's flags there waits for them (eddoursul's fork, F12:
the host saw a layer's ring up to 30-110 us late, p99). --host-core last (or STRATA_HOST_CORE=last) puts the host
on the last physical core and lets the workers have the first. The default is the layout the pool has always used;
a hybrid CPU keeps it. The startup log names the processors the pool workers and the host thread run on. The
placement changes no result, only where threads run. The fork's other part (a per-second re-pinning from the
processors' interrupt time) is not ported.
Ported from eddoursul's fork, commit 5137b33 (MIT).
Co-authored-by: Eddoursul <[email protected]>
Builds on midhatn's #807 (cudaMemcpyBatchAsync for the verifier's DMA group). dma_batch.hpp now holds one
helper, copy_blobs, for independent host-to-device copies on a stream, which the verifier's PCIe share
(copy_expert_blobs) and the adaptive tier's refills (both loops of generate.cpp: --serve's main device, the CLI's)
use. Changes against the pull request, from the review: mode 0 is the old loop untouched (it never stops the
engine: the PR aborted on any copy error, which 0.1.39 ignored); the Windows host alias (cudaHostGetDevicePointer)
is looked up once per blob; a batch the runtime refuses is redone as the loop with one warning; HIP, CUDA before 13
and groups over 128 keep the loop; the batch order is the stream's. eddoursul's fork batches the same copies
(F12: under WDDM each copy raises an interrupt where it lands, on the processor the GPU's interrupts go to).
STRATA_DMA_BATCH is 0 (off) unless set; 1 batches, 2 adds CUDA 13.4's overlap hint.
Co-authored-by: midhatn <[email protected]>
Co-authored-by: Eddoursul <[email protected]>
Cherry-picked unchanged from the pull request, for F12; the follow-up commit turns its helper into the single
copy_blobs helper and fixes what the review found.
The final mixer in one fused read.
The hyper-connection read before the head ran gr_read's kernels once per token of the window (lm_head_mix), and once
per draft pass in the draft layer. Both now go through fused_gr_read_multi, the layers' read (no injection weight:
the final mixer), the window's tokens in one launch. Its sums are gr_read's (fused_gr_read_multi is bitwise
fused_gr_read, which gr_parity checks against the plain read). CUDA only; HIP keeps gr_read per token
(STRATA_HEAD_MIX_MULTI=0 does too). This tree's opt-in STRATA_FUSE_HEAD_GR path is unchanged.
Ported from eddoursul's fork, commit 2ec7a80 (MIT).
Co-authored-by: Eddoursul <[email protected]>
The draft pass's kernels in the main layer's forms.
- quantize_q8_0_scaled runs a warp a 32-value block (it ran a thread a block: 80 threads for the draft layer's
input): the block's |max| over the same values, then each value's bytes as before. CUDA only.
- The draft graphs' inputs from mapped memory in one copy kernel (copy_from_mapped_multi): 7 launches in the round
graph and 2 in a step become one each (CUDA; HIP keeps its separate copies).
- The shared expert on a branch of the draft graphs beside the router and the routed experts, joined before the
combine (CUDA; HIP keeps one stream; STRATA_MTP_SHARED_BRANCH=0 turns it off).
The fork's verify_router in the draft pass and its native-format draft experts are not part of this commit (they
belong to F1 and F15). Ported from eddoursul's fork, commit d3d48b1 (MIT).
Co-authored-by: Eddoursul <[email protected]>
A one-token window commits itself.
A one-token window always keeps its token, so its graph now advances the sequence state itself: the GDN step kernel
writes the state (n_keep = 1, the arithmetic the commit's replay used), the conv kernel writes the history it read
(gdn_conv_l2_multi's commit flag), and its indexer append and PLE history were final already. Verifier::commit
launches no commit graph after it: the draft round graph then runs alone instead of beside the commit's kernels,
which read and write each GDN layer's 3 MB state.
CUDA only (HIP keeps the commit graph after every window; STRATA_ONE_TOKEN_COMMIT=0 does too); batch windows keep
theirs. Ported from eddoursul's fork, commit 235784c (MIT).
Co-authored-by: Eddoursul <[email protected]>
The draft round graph runs the draft layer's front once.
MtpDrafter::capture_round ran the draft layer's front (the embedding, the fc projections, the attention
hyper-connection read, the K/V projections and appends) over the window's T cells, then the whole layer again for
row a, repeating row a's front. record_forward is now record_front (T rows) and record_rest (one row: the attention
from the q projection, the MLP, the final mixer, the head); the round graph runs the front once, copies row a's
residual, injection and mixed input to row 0 (copy_row_to_first, the row read on the device) and runs the rest
there. The multi-row front is per row the one-row front's arithmetic (checked by the window hashes on fixed text).
The coupled-sampling and rotated-KV paths of this tree stay in the same functions.
Ported from eddoursul's fork, commit 22b46b6 (MIT).
Co-authored-by: Eddoursul <[email protected]>
The verify head's argmax and the draft's top probability over many blocks a row.
The draft head's top probability ran row_top_prob, one 1024-thread block a row over the 40,525-token draft head
(and the head's greedy pick, sampler_greedy_kernel, one block a row on sm_80 to sm_89). argmax_rows scans a row
in up to 128 blocks (4,096 logits each), the row's last block merging their picks (the larger value, on equality
the lower index: the one-block kernel's order); row_top_prob_split computes row_top_prob's 32 warps' sums in 8
blocks, the row's last block adding them in order. Both keep their counters and partials in per-row slots of one
scratch. Bitwise: the new verify_parity test checks them against sample_tokens' greedy pick and row_top_prob (ties,
NaN, -inf, 1-8 rows on one scratch).
CUDA only: argmax_rows is used on sm_80 to sm_89 (sm_90+ keeps sample_tokens' cluster kernel); row_top_prob_split on
every CUDA card; HIP and the cards below sm_80 keep the old kernels (STRATA_MULTI_BLOCK_ARGMAX=0 turns both off).
Ported from eddoursul's fork, commit fbb8bfa (MIT).
Co-authored-by: Eddoursul <[email protected]>
--spec-follow PATH makes a generate run emit a given continuation (token ids)
instead of the window's argmax, and accept the drafts that match it, so runs
with different speculation settings (or two builds) process the same text.
--window-hashes PATH writes one line per verify window: its index, position,
size, an FNV-1a hash of its final residual rows and its argmaxes; two builds
doing the same arithmetic write the same file. Both are off by default and
change nothing when unused.
Co-authored-by: Eddoursul <[email protected]>
A verify window takes the previous adapt round's swaps once they are STRATA_ADAPT_LAG windows old. The default is 1,
exactly 0.1.39; lag 2 is opt-in. pending_age resets after every pending.clear(). The batch paths, --pipeline-windows
and the peer tier are not lagged.
Co-authored-by: Benjamin Larsson <[email protected]>
- carve: with owned buffers the streamed ring is one cudaMalloc instead of up to 1,024 pieces each rounded to 2 MiB
(~0.5 GiB). A borrowed region keeps its per-slot layout and price, so the loans' slot counts do not move.
- Prefill::bytes_needed_owned prices the real allocation (2 MiB pages, the ring in one piece). The cache sizing uses it
only with STRATA_OWNED_PRICE=exact; the default is still 0.1.39's 160 + chunk * 680 / 1024 rule.
- the "device buffers do not fit" error names the free VRAM, the bytes taken and the buffer that failed.
No refusal was added: parts B (borrow_residency) and C (the advisory planner) stay out.
Co-authored-by: Jakub Luwierski <[email protected]>
Items 1-2 of #863 (gathered decode per core, AVX-VNNI rows) are bitwise and stay. Item 3 changes a lone token's IQ3_S
rounding on Intel hybrid CPUs, so it needs the switch. The #706 bit-plane path stays in front of the VNNI/plain rows.
#152's rule gives a lone token's gate/up rows to ggml's dot, because the multi-token kernels were no faster at one
token. With the gathered decode that no longer holds for IQ3_S on the Intel cores that gather it: an expert's gate/up
rows for one token on an i9-14900KF P-core take 0.34 ms with ggml's dot and 0.19 ms with the AVX-2 kernel. And in
decode most CPU experts serve one token of the window.
So on a CPU without AVX-512 where cpu_gather_fast() holds (Intel, Alder Lake or later, see the gather commit) and
STRATA_IQ256_GATHER is not 0, IQ3_S takes the multi-token kernel from one token on. This only pays together with the
gathered decode: on the same CPU's E-cores, which keep the scalar decode, it is 0.71 -> 0.68 ms, about even. That is
why it follows the gather's machine-wide rule and not the per-thread one. Measured in the engine with Swift 1.5
IQ3_XXS (whose IQ3_S layers this reaches): the CPU expert time per verify window -4%.
This changes results. A lone token's IQ3_S gate/up rows now round like the multi-token kernel's (the order of the
float additions), no longer like ggml's dot. They are the same rows the token gets in a group, so on these CPUs IQ3_S
experts no longer depend on how many drafts a window held (#152), and every core rounds them alike. Down rows and
the other formats keep the default rule. STRATA_IQ_MT_MIN set (2 is the old default) keeps its rule for every format.
The decision is native_gu_mt_min(): every probe is in expert_layout.cpp and cpu_avx2_ok() comes first. Nothing
compiled for AVX2 runs before that check, including on an older-CPU build (STRATA_ISA_FLOOR) on an AVX-only CPU.
iq_avx2_parity checks native_gu_rows wherever native_gu_mt_min is 1: each token's rows must match alone and in groups
of 2..8. docs/DETAILS.md notes the new default next to STRATA_IQ_MT_MIN.
Alder Lake, Sapphire Rapids and later have AVX-VNNI on every core: vpdpwssd does the madd and the add of an i-quant
row in one uop, and vpdpbusd the maddubs + madd(ones) of the IQ4_NL and Q2_0 down rows. The sums are integer, so the
results are bit for bit those of the AVX2 forms: vpdpwssd wraps like madd + add, and the maddubs it replaces in the
down rows cannot saturate (IQ4_NL |w| <= 127, Q2_0 codes 0..3).
The row kernels move to iq_avx2_rows.inl and q2_avx2_rows.inl, which iq_avx2.cpp and q2_avx2.cpp include twice: once
as they were (namespace plain) and once with AVX-VNNI (namespace vnni). In the second copy every function is declared
with target("avxvnni") under GCC/Clang, so the intrinsics compile there and nowhere else: neither file gets
-mavxvnni, which would let the compiler use the extension in the plain path. MSVC takes the intrinsics under
/arch:AVX2. CMake checks that the compiler has them (GCC 11, Clang 12) and otherwise builds without the copies
(STRATA_AVXVNNI=0).
cpu_avxvnni_ok() in expert_layout.cpp, which is not compiled for AVX2: cpu_avx2_ok() and CPUID 7.1:EAX[4], off under
STRATA_FORCE_ISA (a CPU that stops at AVX2), off in a STRATA_ISA_FLOOR build, and STRATA_NO_AVXVNNI=1 turns it off.
The VNNI copies are reached only through the AVX2 kernels' entry points, which the engine already calls only behind
cpu_avx2_ok().
Measured on an i9-14900KF on top of the gathered decode: another 4-10% on a P-core's i-quant gate/up rows, 15-25% on
an E-core's Q2_0 down rows.
iq_avx2_parity runs the VNNI variants as well (iq4nl256_down_rows_v, q2_0_gguf_rows_multi_avx2_v): every combination
bit for bit against the scalar one-token rows, 1..8 tokens.
0736f05 reads the IQ3 grid words with one gather under STRATA_IQ256_GATHER=1, on every core, and keeps it off by
default because AVX2 gather throughput depends on the core. This makes the gathered decode its own Fmt32 (118, 121,
122), chosen once per call instead of a branch in every decode, and takes it further:
- IQ3_XXS: the four even-sign masks by one 64-bit gather as well;
- IQ3_S: the 9th index bit by one sllv and one and; the sign bytes by one broadcast and one shuffle;
- IQ2_S: the four 10-bit grid indices built in a register and read by one 64-bit gather, the signs as IQ3_S's.
Each is its format's scalar decode bit for bit: the same words land in the same lanes. IQ2_XXS measured 4% slower
gathered and IQ2_XS has its own sign decode, so both keep the scalar one.
STRATA_IQ256_GATHER: unset = auto, 0 = the scalar decode everywhere (as before), 1 = the gathered one everywhere.
Auto gathers on a thread whose core is an Intel performance core of Alder Lake / Sapphire Rapids or newer:
- GenuineIntel with AVX-VNNI (CPUID 7.1:EAX[4]) as the generation marker. That leaves out Haswell/Broadwell (slow
gathers) and Skylake through Ice Lake / Tiger Lake (gathers slowed by the Downfall/GDS microcode);
- not the E-core-only parts, which are not hybrid: Alder Lake-N / Twin Lake, Grand Ridge, Sierra Forest and
Clearwater Forest, by model;
- on a hybrid CPU only a thread on a P-core (CPUID 1Ah core type 40h): the E-cores gather slower than they assemble.
AMD keeps the scalar decode (Zen 2/3 gather slowly; Zen 4/5 take the AVX-512 kernels). STRATA_FORCE_ISA turns auto
off. The probes are in expert_layout.cpp, which is not compiled for AVX2, and the core type is asked once per thread
(the pool pins its workers).
Measured on an i9-14900KF, gate/up rows of one expert at three tokens on one P-core, scalar -> gathered: IQ3_S 0.40
-> 0.27 ms, IQ2_S 0.31 -> 0.27, IQ3_XXS 0.26 -> 0.24. On its E-cores the gathered decode was slower.
iq_avx2.cpp has no runtime initializer again (d072203): the switch and STRATA_IQ_PREFETCH are read on first use.
iq_avx2_parity, a CPU-only CTest: random blocks (every grid index and sign pattern occurs) for IQ2_XXS, IQ2_XS, IQ2_S,
IQ3_XXS, IQ3_S and IQ4_XS gate/up rows and IQ4_NL / Q2_0 down rows; every variant, 1..8 tokens, each token's rows bit
for bit the scalar variant's rows for that token alone; the scalar rows against ggml-cpu's vec_dot (Q2_0: a double
reference). --bench: ms per expert on one thread, each variant, ggml's per-token dot and the engine's dispatch, over
more expert blobs than an L3 holds.
The PLE Q8_0 reader duplicates #651 and is dropped here. The grouped Q6_K instance is built only on request
because every instance is loaded at engine start (EAGER) and costs slots for every other model.
Co-authored-by: Thomas Weaver <[email protected]>
Gate/up is Q6_K on 47 of 48 layers and Q8_0 on layer 2; down is Q8_0 on
every layer. The page had those roles swapped. native_expert_parity
q6_K/q8_0 is now a normal test: the old one expected the pair to be
refused, and native_expert_supported accepts it.
The measured server runs are on this machine (RTX PRO 4500 Blackwell,
engine 0.1.39 after the rebase, and one cold prefill on 0.1.38). Quality
against llama.cpp is still unmeasured, and the three Q6_K parity
programs have not been run on a GPU.
PleTable accepts per_layer_token_embd.weight as Q8_0 (170-B rows): the
UD-Q6_K_XL file ships its PLE table in Q8_0, not IQ4_NL. The row reader
is already parameterized by bytes-per-row; only the type check, a
dequantize_q8_0 block loop (5 x 34 B) and the PLE_ROW_BYTES_MAX bump
(160 -> 170) were new. No GPU kernel changes: gather outputs floats.
- ple_reader_test --selftest also runs 170-B rows (90/110 were already run)
- ple_q8_parity: real-artifact Q8_0 PLE rows against ggml's reference
dequantizer, through Mmap and Direct (bit-exact on the synthetic
fixture from tools/ple_q8_fixture.py)
- docs/UNSLOTH_Q6.md: the manual workflow, verified shard inventory
(169.2 GB, 6 shards, table alone in shard 3), pending real-run numbers
A four-GPU run picked its split from a formula that reads only `layer_ms`
(SMs x clock) and shares the layers in proportion to speed; the free VRAM
that decides how many experts a stage can actually hold never entered it,
so the search could hand the most layers to the card with the least room
and never notice. All C(L-3,3) placements are now scored - 15,180 at 48
layers - and 2,068 of them cost nothing extra, because a stage's fill
depends only on its own layer range and its own free VRAM: `predict`
already hands each ranked pair to the one stage that owns its layer and
stops each stage at its first pair that does not fit, so a range is walked
once and memoised.
The cost model's mass curve changes with it. `mass[r]` was (r+1)^-1.2,
fitted to the 5080+3090 sweep - still right on that rig (99.35% predicted
at 20,000 held, 99.4% measured) and wrong on IQ3_S, where it claims the
top 5,805 pairs hold 94.9% of the routed mass and the rig measures 69.2%
over four boots and 14.9M decode lookups. No exponent reconciles them,
because the curve is a property of the model's routing and the profile
carries only a ranking, with no counts. The mass is now the increment of
H(f) = 1 - (1 - f)^b over the held fraction, whose prefixes sum to exactly
H(N/total); b = 3 (STRATA_SPLIT_COVER_B overrides it) is chosen to leave
the validated regime where it was - at f = 0.8138 it gives the old fit's
99.35% - while at f = 0.59 it gives 93.2% against 92.2% measured.
This is the placement search from #584, carried on v0.1.39 and without the
weight carve it rode in on: v0.1.39 ships its own carve (#559 / #639), and
the search only runs under `--layer-split auto`, where nothing is carved -
so `cap[i]` is already net of the dense weights and a stage's room is what
`predict` already subtracts, its session.
Measured on the 4-way IQ3_S rig (2x RTX 3060 + 2x RTX 5060), same config and
same flags in both arms, the binary the only difference: the four-way split
moves from K=10,19,34 to K=14,31,41, the prompt chunk the starved cards allow
grows 768 -> 1280 tokens, prefill goes 425 -> 597 tok/s (+40%) and decode
44 -> 51 tok/s (+16%), with the decode expert-cache hit 74.7% -> 83.3%. Two
boots per arm, and the split is deterministic. The curve and the search are
one commit, so that measures the pair, not the search alone.
The duplicate block_topk_stream_kernel is dropped: 0.1.39's block_topk_wide_kernel gives the same ids.
Only the 90-135K Turing band is new (counted bound, sm_75). Parity cases from the PR are kept.
A segment reads in ceil(tokens / max_chunk) chunks of equal size, kept only where the last chunk would stream
every expert anyway. Used only at the segment call sites; request_chunk is unchanged.
Co-authored-by: architectds <[email protected]>
Down rows are only 10 blocks, so the per-row reduction was a large share; four rows now share one hadd tree. The single-row reduction uses the same tree, so a row's bits do not depend on how rows are split into tasks. Down rows +6-7% per core.
The activation is pre-permuted once per token into the 2-bit codes' plane order (ActQ::qp/psum/pscale), so rows need no unpack interleave: four planes summed in int16, one madd, the -1 code offset subtracted exactly in the integers, one FMA per two blocks per token. 2.3-2.8x faster per core than the b256 kernel, closer to exact (rms rel 1.0e-7 vs 2.7e-7), width-invariant. STRATA_Q2_LEGACY=1 selects the previous kernel.
A conversation the engine holds (what a parked snapshot holds) goes to one file and comes back, also after a
restart, so a long prompt is not read again:
- engine: `SAVE <path>` / `RESTORE <path>` between requests, answered `SAVED|RESTORED <tokens> <bytes> <ms>` or
`ERR <reason>`; single GPU only (refused with --layer-split) and needs --prompt-cache > 0.
- format v1 (src/core/conversation_file.cpp): 64-byte header bound to a model fingerprint (name, size, first and
last MiB of each weight file) and a config fingerprint (engine version, --kv, STRATA_KV_ROT, --kv-resident,
--max-context, --mtp-window, control vectors), the payload, a payload hash and an end marker; written to
`<path>.tmp` and renamed over `path` (MoveFileExW on Windows).
- fail-closed restore: size, header, both fingerprints, every count against the bytes left, the payload hash and
the usual snapshot validation all run before any device write; a refused file leaves the session untouched.
- only the deepest checkpoint is saved (the next turn's resume point) and the K/V is streamed from the pools
straight into the file's 16 MiB blocks, without a full host capture; Linux uses O_DIRECT when the filesystem
takes it (buffered otherwise or with STRATA_SESSION_BUFFERED=1), Windows buffered Win32 I/O.
- serve/server.py: llama-server's `POST /slots/0?action=save|restore {"filename": NAME}` with --slot-save-path DIR
(or "slot_save_path" in the config); NAME must be a plain file name inside DIR.
RTX 4070 Ti, IQ3_XXS, a 63,025-token conversation: file 1.20 GB, save 0.72 s, restore 0.89 s, the next 32-token
turn after a restore 1.15 s (first token 0.55 s) with the same output tokens as without saving.
Tests: conversation_file_test (CPU, no model; also built for Windows and run under Wine),
conversation_snapshot_test, serve/test_slots.py, serve/test_security.py.
From an audit of this PR, each finding checked against the code:
- VmmRange: a run whose cuMemSetAccess failed stayed counted as mapped (a later map_range skipped it, so the chunks
were never made accessible); it is unmapped and freed again (commit_run).
- a failed growth or trim could leave host_res, d_res and the slot range half-changed while serve went on to the
next request; such a failure now stops the engine, as other CUDA failures do. The residency uploads are checked.
- the trim maps the chunks it takes back as one run (one cuMemSetAccess instead of one per 2 MiB chunk) and refills
the slots with async copies and one sync instead of a blocking cudaMemcpy per expert.
IQ2_XS: the served scenario (grow at a request and mid-decode, trim and refill, regrow, a parked conversation's
restore) runs as before; vmm_test passes.
At --max-context 262144 the K/V is allocated whole at start (3.35 GiB at int8 with the drafter's) and the expert
cache is sized around it. With --kv-grow (or STRATA_KV_GROW=1) the K/V pools and the cache arena are CUDA VMM ranges
(core/vmm.hpp; the driver calls through the runtime's entry-point query, nothing links the driver library). The K/V
maps 2 MiB chunks for the cells the requests reach (16384 at start); when a request needs more, slots just below the
prompt path's loan give theirs up, a hotter expert in them first moving into the slot of the coldest one in the loan
region. A short request after a long one hands the chunks back and the slots refill from the profile. No address
moves, so no graph is captured again, and --max-context cells always fit.
- serve: grows at a request (its prompt, and a parked conversation before its restore) and in the decode rounds;
trims once the outgoing session is parked
- generate: grows once for the prompt and max_new, before the loan
- one GPU with a profile, the whole K/V in VRAM (no --kv-resident), every expert in RAM (not the low-RAM tier);
STRATA_KV_GROW_INIT / _STEP / _FLOOR for tests. Off by default: more slots move experts between GPU and CPU.
- tests/core/vmm_test: map, a chunk moved between ranges keeps its bytes, release
IQ2_XS, RTX 5090, 262K, fixed cache: first-token logits and 64 tokens bit-identical to 0.1.31 off, mapped whole,
grown into new VRAM, and grown by giving up 266 slots. Cache auto: 15,119-15,207 -> 17,300-17,503 slots, short chat
15.06-15.33 -> 14.71-14.83 ms a round; an emulated 16 GB card: 3,396 -> 5,719 slots, short chat 100 -> 115 tok/s,
a 32K prompt 7.44 -> 6.36 s and 24.6 -> 22.2 ms a round after it.
Setup no longer excludes K8V4 from KV streaming (KV_CELL_BYTES prices 816 B/cell), and docs/DETAILS.md says
it streams. This needs the engine of the PR #711 commit: MIN_ENGINE must be raised to 0.1.40 at the release,
or an older engine refuses the pair.
Co-authored-by: Trevin <[email protected]>
K8V4 (INT8 K + rotated Q4_0 V, 816 B/cell) was refused with --kv-resident. It now streams like
the other formats:
- kv_stream: a fourth block format, kKvHybrid (= 3, the number conversation snapshots already
use for K8V4), moved as three runs per block - K codes, K scales, V q4_0 - by the same resolve,
copy, ring-restore and stage paths; kv_block_bytes knows its size.
- kv_hybrid_k_half / kv_hybrid_v_half: each half of the host copy (and of the prompt path's
staging pool) as the single-format append kernels see it, so a hybrid layer's host copy is
written by the same folded calls that write its VRAM pools. Both fields of a half point at the
same array: the kernels test the K-side pointer for presence and pick K or V per lane.
- The decode (layer.cpp), verify-window (verify.cpp) and prompt (prefill.cpp) appends pass those
halves; the prompt path's staging pool gets a K8V4 layout; the host copy is allocated in it.
- Conversation snapshots read a streamed K8V4 state's host copy, as they do for the others.
- qsa_kv_format returns kKvHybrid instead of exiting; the two refusals are gone.
setup.py still keeps K8V4 unstreamed: enabling it there needs a MIN_ENGINE that has this.
Tests: kv_stream_parity runs K8V4 too (streamed vs resident attention, bitwise): 306 batches,
70,449 misses, 0 failures, ring restore identical; a negative control (the V half's K-side
pointer left null) fails it from the first batch. kv_hybrid_parity: all pass.
The attribute-after-declarator form from #778 does not compile with nvcc and MSVC as host (the CUDA build failed in mmvq_multi_kernel). The macro uses it under __HIPCC__ only, so CUDA is as it was.
Review changes on #835: off unless STRATA_HIP_PROMPT_F16=1 (a tip on gfx103x); act16/g_act_f16 and set_act_f16 are HIP only so CUDA kernels are unchanged; Prefill::init decides the mode once per stage on the stage's own device and bf16x2 takes it as an argument; widen_rows_f16 counts non-finite outputs under STRATA_DBG_NAN; new hip_gemm_f16_io_parity test.
Co-authored-by: xjc10 <[email protected]>
A bullet in the RDNA2 section: rocBLAS on gfx1030 only has tuned FP16-in/FP16-out kernels, what #835 changes,
STRATA_HIP_PROMPT_F16=0 for the old path, the measured prompt speeds on a second community machine (2x RX 6900 XT,
ROCm 10.0, engine 0.1.39), that the output is not bit-identical (link to the distribution check), and the ctest result
with -DSTRATA_BUILD_TESTS=ON: 60 of 65 pass with and without the change, the 2 skipped and 3 failing tests and why.
Per-value rounding bounds of both paths (BF16-weight products: the BF16 activation rounding goes, an FP16 output
rounding comes; quantized-weight products: an FP16 output rounding comes), why the two are not compared by argument,
how the BF16X2 reference relates to them, and the open follow-up (an fp32 forward pass as reference).
rocBLAS on gfx103x has tuned GEMMs for FP16 in / FP16 out only; FP16 in / FP32 out (Gemm::f16, Gemm::native) and BF16 in
(Gemm::bf16) fall back to generic kernels, 5.6 / ~5.3 TFLOPS against 37.7 on gfx1030 (N=10240, T=7313, K=2560).
On gfx103x only (prompt_f16(), per device; STRATA_HIP_PROMPT_F16=0/1 overrides):
- Gemm::f16 / native run FP16 out into the start of each FP32 row of Y (ldc = 2 ldy halves); widen_rows_f16 widens each
row in place from its end - no extra buffer, no host sync
- Gemm::bf16 converts W in row slices to FP16 through the dequantization scratch and takes X as the FP16 image the
gr_* / to_bf16 kernels now write (set_act_f16; saturated, a NaN stays a NaN); BF16X2 is off there
- Gemm::set_f16_io is an explicit per-instance switch that only Prefill::init turns on, so every other caller
(gemm_bf16_parity) runs what it ran; CUDA builds are unchanged
2x RX 6900 XT, IQ3_S: one card 439 / 466 / 461 -> 744 / 915 / 926 tok/s at 9.4K / 34.7K / 105.8K tokens; layer split
444 / 707 -> 833 / 1,356 at 8.3K / 33.6K. Needle 15/15 on both paths; teacher-forced, the FP16 path is closer to a
higher-precision reference (BF16X2) than the old path. Details: bench/results/2026-10-04-rdna2-fp16-prompt/README.md.
Workaround for the gfx1030 hang that needs MMQ prompt + adaptive swaps + SDMA together. Off by default; a source outside the registered arena, or not 16-byte aligned, takes the normal cudaMemcpyAsync. Untested on the reporter's 2x RX 6900 XT.
Fixes 1 and 4 of #870, plus a test that every function setup_intel.py swaps into setup.py accepts all of the original's parameters. Fixes 2 and 3 are PR #784's session/native_dense ports.
Co-authored-by: Magh97 <[email protected]>
Reworked from #873: no 8 x 250 ms retry at every failing call site, no #if in device_info (cudaMemGetInfo already maps to this function), no retry of real failures (only hipErrorInvalidValue, once), before/after lines in the log. The strata-device 5x retry loop is dropped.
Co-authored-by: unimo-chan <[email protected]>
HIP clang for gfx1030 (ROCm 7.1) rejects 'alignas(16)' placed between
__shared__ and the declaration ('alignas attribute cannot be applied to
types'). Use __attribute__((aligned(16))) after the declarator instead —
same layout, compiles everywhere.
A function-like macro splits its arguments on the commas inside a template
argument list, so cudaFuncSetAttribute(kernel<1, 6, true>, ...) failed to
compile (fused_gr.cu:1089-1093, 'too many arguments provided to function-like
macro invocation'). Restore the template function 0.1.38's hip_compat/
cuda_runtime.h had, and add the two carveout definitions used beside it.
gfx906-only: include/strata/platform/hip_compat/ is on the include path only
inside if(STRATA_HIP_GFX906); CUDA and the wave32 RDNA backend are untouched.
Two findings from the Copilot review on #695.
The five-character prefix match (replaced by the named sibling sets in the
parent commit) also grouped gfx1010 (RDNA1) with gfx1030 (RDNA2), so the gate
would have passed a table between two architectures that share no solution
IDs. The test now pins that pair in both directions, and the third RDNA3
member (gfx1102) alongside the pair already covered. Measured against the two
implementations with MSVC, same test binary source: the prefix-match gate
fails exactly these two cases (13 cases, 2 failures), the sibling-set gate
passes all 13.
<unistd.h>/getpid() and a hard-coded /tmp broke the test on Windows/MSVC with
STRATA_BUILD_TESTS=1. It now uses std::filesystem::temp_directory_path with a
steady-clock token and a per-case file name, the shape
tests/core/gguf_split_test.cpp already uses, and it reports a scratch-file
write failure instead of letting it read as a gate rejection and pass for the
wrong reason.
Built and run with MSVC 14.4x (/std:c++20, /W4): 13 cases, 0 failures.
The header gate compared the table's architecture to the runtime's with
`arch != expected_arch`. A card that reports its RDNA4 sibling - a Radeon
AI PRO R9700 (gfx1201) under HSA_OVERRIDE_GFX_VERSION=12.0.0 reports
gfx1200 - rejected its own calibration at startup and lost the tuned
prefill route entirely:
prefill gemm: tuning architecture mismatch: file=gfx1201 runtime=gfx1200; using hipBLASEx
Solution IDs are scoped to the architecture FAMILY (gfx1200/gfx1201 share
them, as gfx1100/gfx1101/gfx1102 do), and a solution ID that does not exist
on the running card already fails the per-solution gate in prefill/gemm.cu
(getAlgosFromIndex / matmulIsAlgoSupported) and falls back to hipBLASEx per
call. The header gate can therefore be family-wide: the version gate stays
strict, cross-family pairs stay rejected, and the per-row gate decides which
rows actually run.
Adds tests/prefill/tuning_header_arch_family_test.cpp (synthetic tables, no
model, no GPU): 10 cases, RED on the strict gate (3 sibling failures),
GREEN on the family gate.
Calibrated with tools/hip/tune_hipblaslt.cpp on a Radeon PRO W7800 (gfx1100) under ROCm 7.2.4. setup.py:hipblaslt_table() resolves tools/hip/<arch>-hipblaslt-<version>.txt, and no 100202 table exists for gfx1100, so prefill silently falls back to plain hipBLAS. Requested in #505.
tools/hip/gfx1100-hipblaslt-100500.txt: the same 16 dense GEMM geometries as
the other gfx1100 tables (bf16 and f16, T=4096 and T=8192, 32 rows), calibrated
on an RX 7900 XTX with tune_hipblaslt (relative L2 <= 1e-4, max abs <= 1e-2, no
padding writes) against a ROCm 10.2.0a20261003 nightly SDK
(libamdhip64.so.7.17.26392), hipBLASLt 1.5.0 (version number 100500).
The engine refuses a table of another hipBLASLt version, so on this ROCm the
shipped gfx1100 tables (100100, 100200) are refused and prefill runs on plain
hipBLAS - the same gap the gfx1201 1.5.0 table (#481) closed for the R9700.
setup picks this one by name once the installed hipBLASLt reports 100500.
Measured on a 0.1.38-lineage nightly build of this tree, Qwen3.8-Flash-Next
IQ3_S greedy, arms differing only in STRATA_HIPBLASLT_TUNING (median of 3
clean runs each):
| prompt tokens | with table | plain hipBLAS | gain |
| 131071 | 1687 tok/s | 926 tok/s | 1.82x |
decode unchanged (62.3 vs 60.7 tok/s, overlapping ranges). Per-run data,
the calibration command and the tuner log: #745
(bench/results/2026-10-04-community-rx7900xtx-hipblaslt-100500/).
docs/AMD_HIP.md: the shipped-tables list carries the entry.
ROCm 10.0.0's hipBLASLt is 1.4.1, whose solution ids are version 100401, and no
gfx1100 table shipped for it - so on an RX 7900 XTX with a packaged ROCm 10 the
prompt path's dense GEMMs fell back to plain hipBLAS. This is the same version
mismatch PR #745 measured on ROCm 10.2 (their 100500), reproduced on 10.0.0.
tools/hip/gfx1100-hipblaslt-100401.txt: calibrated with tools/hip/tune_hipblaslt
over the 26 dense geometries of the shipped gfx1100 table, on a Ryzen 7 9800X3D.
Swift 1.5 IQ3_XXS, 262144 ctx, --kv int8, fresh prompts (a nonce per request, or
the server's prompt checkpoint reads nothing), medians of 3:
1.7K tokens 682 -> 1,038 tok/s (+52%)
6.5K tokens 857 -> 1,524 tok/s (+78%)
Decode unchanged (79.0 vs 80.3 tok/s, inside this machine's 2.3% run drift), and
hip_prefill_hipblaslt_gemm with STRATA_HIPBLASLT_VERBOSE=1 reports fallbacks=0.
Gemm::native_mmq, its call, the f16_to_f32/set_bounds kernels and the Q6_K case are under __HIPCC__; the CUDA type list loses q6_k. The K % 256 != 0 out-of-bounds read is already refused in native_mmq (kept). Still opt-in: STRATA_DENSE_MMQ=1 on a STRATA_MMQ_KQUANTS=ON HIP build.
The prompt path's dense projections (attention qkv/gate, the PLE key, the shared
experts, the router, the head) run dequantize-to-FP16 + cuBLAS: on the RDNA2
cards that is 68% of the prefill GPU time at ~25% of the card's fp32-accumulate
rate, while the same shapes through llama.cpp's int8 MMQ run ~7x faster
(measured with bench_dense_gemm). This wires Gemm::native's beta = 0 products
through the MMQ path the MoE experts already take: quantize the FP16 activations
to q8_1 (they are converted to FP32 in the dequant scratch, which the old path
only fills afterwards) and run one mul_mat_q product per 1024-row chunk. It is
opt-in (STRATA_DENSE_MMQ=1; unset, nothing changes) and falls back silently to
the old path on any shape MMQ does not take.
- STRATA_MMQ_KQUANTS also compiles the Q4_K / Q5_K / Q5_1 / Q6_K instances on
the HIP branch (the mixed-quant packs' dense tensors are K-quants: this
model's attention/PLE matrices are IQ4_XS/IQ3_S/Q4_K/Q5_K/Q6_K), adds Q6_K to
the CUDA list, and defines STRATA_MMQ_KQUANTS for the HIP target.
- Gemm::native_mmq: one MMQ context, xq/ident/bounds buffer (~9.5 MB) and a
failed-flag per Gemm; fp16->fp32 and the {0, rows} bounds helper join
moe_mmq's kernel set.
- K must be a full 256-value multiple. MMQ loads the weights in 256-value K
chunks, and past a partial row the chunk reads bytes past the row - the next
row's, or for the last weight row bytes past the tensor, which no caller
guarantees to be zeros (the MoE path is safe because its gather buffers end
in a zeroed tail). On this pack the 640-value IQ4_NL shared-expert down
picked up a NaN scale from the next tensor's blocks and zeroed the output;
the odd shapes stay on the dequantize + cuBLAS path.
Measured on the RX 6900 XT (gfx1030), swift-1.5-iq3_xxs, WORKERS=15 ADAPT=1
PREFILL=8192, bench_prefill.py fresh medians: 4K prefill 436.1 -> 757.2 tok/s,
8K fresh 447.7 -> 773.9 tok/s, 8K followup 163.9 -> 206.1 tok/s; decode within
noise (40.6 vs 40.3 tok/s at --pcie-frac 0, 52.7 vs 49.9 at the 0.55 default).
Greedy output: bit-identical for the first 200 tokens of a short prompt; a 4K
prompt's greedy text diverges at token ~26 (the q8_1 activation rounding) and
stays coherent.
HIP only (__HIPCC__); the CUDA kernels are unchanged. A volatile store to mapped host memory can stay in L2 on RDNA4 until the stream syncs; __threadfence_system() after the store flushes it. Reporter's one-line diagnostic fix, applied to doorbell_ring/publish/publish_res (publish_value already had it). For #579 and #613 to test.
The fork overlaps the shared expert with the routed experts on a second stream.
It pays off on CUDA, where a36be1d added it, but on HIP the cross-stream event
dependency costs more than the shared expert takes to run, so there the fork
starts off and STRATA_SH_STREAM=1 turns it back on. CUDA's default is unchanged.
Same shape as mtp.cpp's STRATA_MTP_PREFILL_SYNC (#382): the env var always wins,
only the default differs per backend. The profiler takes the fork off to time the
stages, so a profiled run never showed this - which is why STRATA_SH_STREAM=0 was
the only way to see it.
Radeon AI PRO R9700 (gfx1201, ROCm 6.4.3), Qwen3.8-Flash-Next IQ3_XXS, --kv int8
--kv-resident 32768 --resident-experts, temp 0, 3 runs each, same binary and
prompt, 48,067-token prompt so the KV stream path runs:
stock 0.1.39 (fork on, default) 43.40 t/s (38.90 - 44.60)
this change (fork off, default) 62.90 t/s (55.40 - 63.80)
STRATA_SH_STREAM=1 43.50 t/s (39.10 - 45.20)
STRATA_SH_STREAM=0 63.00 t/s (55.90 - 64.10)
The fork is a scheduling change only: the same four prompts answer byte for byte
the same with it on and off. #816 reports the same fall on a gfx1030 (RX 6800,
Windows, 42.5 -> 56.4 t/s), and a comment on #646 sees it on a gfx1100.
Short prompts are barely affected here (65.4 against 62.7 t/s at 256 tokens), so
this only shows once the KV stream path is in play.
- #831: an explicit --expert-cache above the profile's length was cut to it with no word on the native-pack path; a
warning now says so (the allocation is unchanged).
- #841: a layer-split stage whose expert cache fails to open (out of memory with the VRAM free) is retried with 90% of
the slots, up to 3 times, each try logged. Nothing changes when the first open works.
- #779: a large-page arena cannot be paged out, so the three working-set lock sites skip it (the note says so).
- #771: with /sys/kernel/mm/transparent_hugepage/enabled = [always] the kernel already backs the arena with huge
pages where it can; the extra MADV_HUGEPAGE only added direct reclaim and compaction on every fault (a 25 s start
took 432 s on a fragmented machine). It is not requested there, and STRATA_NO_ARENA_THP=1 skips it anywhere.
STRATA_ARENA_RELEASE already exists and means the opposite (=0 keeps the experts a GPU holds, generate.cpp), so the
opt-in is renamed in the engine, the tests, DETAILS.md and the bench folder. The working-set assert in
file_expert_source_test reports, not fails, a trimmed page that Windows maps straight back with the switch on. The
bench folder keeps the README and the two scripts (the raw export and the harness self-tests are dropped).
Co-authored-by: jordicor <[email protected]>
0.1.31 registers the whole arena with CUDA before the first byte is read. With 4 KB pages, which every account
without the Lock Pages privilege gets, that registration is ~1.5 s of the start that nothing hides (IQ2_XS:
33 GiB).
- PinnedArena(bytes, bounds, Deferred) reserves only. register_slices() registers the per-layer slices in order on
a thread and publishes how many are done.
- Every reader (load_experts_gguf buffered and unbuffered, load_experts_ranges, load_experts_direct) writes layer L
only once slices L and L+1 are registered. Writing a page while cudaHostRegister runs on it corrupted the arena
under WDDM, even on large pages. A layer boundary inside a page puts that page in both registrations, so the next
slice must be done too (page-aligned in effect, without moving the slice bounds an expert must not straddle).
- A slice CUDA refuses ends the registration, and the rest is locked resident as in the sliced fallback.
- Only where the whole arena would be registered anyway: no cap (STRATA_ARENA_PIN_GIB, multi-GPU under WDDM) and no
shared arena file. Linux keeps #253's whole-arena pin. STRATA_DEFERRED_REGISTER=0 restores the old order.
- STRATA_VERIFY_ARENA=1 prints the loaded arena's FNV-1a checksum (tests).
Checksums: the same with the old order and the deferred one, 4 KB and large pages, the GGUF in place and experts.bin
(IQ2_XS; 13 deferred loads).
IQ2_XS without experts.bin, RTX 5090, 128 GB. Each cell is the time until the arena is loaded / until a 16-token
answer is done. Cold: the files pushed out of the file cache first. Two to three runs each:
| | cold | restart |
|---|---|---|
| 4 KB pages (STRATA_NO_LARGEPAGES=1), 0.1.31 | 12.4-13.0 / 16.3-16.8 s | 7.8-8.5 / 10.9-11.7 s |
| 4 KB pages, parts 1+2 | 11.1-11.5 / 14.9-15.4 s | 5.5-5.7 / 8.5-8.9 s |
| large pages, 0.1.31 | 17.6-18.7 / 21.1-22.4 s | 4.6-5.1 / 7.0-7.6 s |
| large pages, parts 1+2 | 17.6-19.9 / 21.3-23.6 s | 4.1-4.5 / 6.7-7.1 s |
The cold spread with large pages comes from the drive: the same build gave 17.6 and 19.9 s with the deferred
registration and the cache probe switched off as well.
Review of #773 (2 x RTX 5060 Ti 16 GB, 32 GB RAM, --mmap-experts, no --resident-budget-gib): with no RAM copy the
file cache is the expert tier, and forcing unbuffered reads there re-read 163-629 GB from the drive and halved the
speed. The automatic decision never armed in that mode; now the forced one does not either, and the log says so.
through the page cache, as page faults of the mapped GGUF - small requests on the critical path, and pages that a
32 GB PC cannot keep beside a 16 GiB RAM copy anyway, so they were read again and again.
- open_direct opens the files with O_DIRECT; read_direct queues every 4 KiB-aligned window of a batch with one
io_submit (linux/aio_abi.h, no library; allowed by Docker's default seccomp profile, unlike io_uring) and waits for
all of them, each thread with its own context and aligned buffer. Windows merges windows up to 1 MiB apart for
NTFS; here only windows that touch are merged. A refused submit or a failed read falls back to the mapping, as on
Windows.
- The same decision as Windows' `cache_counts = false` arm: unbuffered when the file cache could not keep the expert
bytes outside the RAM copy (platform::file_cache_keeps, from MemAvailable and the tightest cgroup limit), re-checked
once the copy is built (#577). STRATA_UNBUFFERED_LOAD=1 / 0 forces it either way. A pack's experts.bin works too
(its Linux open now records the path).
- direct_fallbacks() counts the blobs an unbuffered read could not deliver.
- Tests: file_expert_source_test reads a synthetic pack unbuffered - one blob at a time, a batch of adjacent experts,
copy_blob - and requires the file's bytes and no fallback (a mutation that fails every read is caught). ctest: 63
of 66 pass; the three others need a Q2_0 pack, a higher `ulimit -l` and AVX-512, as before.
Measured on an RTX 3080 Ti Laptop 16 GB, i9-12900H, 30 GB RAM, KIOXIA PCIe 4 NVMe, Linux 7.0, ext4: Qwen3.8-Flash-Next
Abliterated Q4_K_M in place (--native), --resident-budget-gib 16, --expert-cache auto, 8K context, greedy; three
prompts (code, Russian prose, a 7.8K-token summary), each with and without MTP, two rounds of buffered
(STRATA_UNBUFFERED_LOAD=0) against unbuffered, the same binary. Decode 4.38 -> 7.90 tok/s on average (x1.80;
x1.49-x2.01 per prompt and mode), the 7.8K prompt read at 154/172 -> 184/290 tok/s, startup 36 -> 18-21 s. The
code prompt's answer is the same text in all 16 runs; the other two differ between runs of either arm (the
adaptive GPU cache changes which experts the CPU computes), as before.
The native output head (437 MB) and the token embedding (260 MiB) were copied
out of the mapped GGUF one page fault at a time, and strata-vision's projector
(907 MB) went through llama.cpp's sequential reads, all before the server was
ready. Each now asks for its bytes ahead (the vision encoder honours
STRATA_READ_AHEAD=0 too), and the head, the embedding, the projector load, the
encoder warm-up and the pinned cache complement log their times.
On Linux every load phase read its files one page fault (one 128 KB readahead)
at a time, so a cold start spent most of its time waiting on the drive at
queue depth ~2. A new platform::advise_willneed asks the kernel for a range in
128 KiB steps (WILLNEED reads at most one readahead window per call), and is
called before the weight load, the native dense upload, the profile fill (256
pairs ahead, STRATA_FILL_AHEAD), the peer tier's fill (--peer-device), the
resident RAM copy and the MTP draft files. STRATA_READ_AHEAD=0 turns it off.
Each phase logs its time.
Gen3 NVMe, RTX 5090, 32 GB, Q2_0 resident at 262K: ready in 70 s, was ~920 s.
- #657: check_shards refuses a one-tensor shard that is longer than its tensor (the PLE table), naming both sizes.
- #737: UD-Q4_K_XL on several GPUs below the 135 GB estimate is a warning and a question (explicit --gpus --yes is
the consent), not a refusal.
- #670: an engine update keeps the engine it replaces in engine/.previous (one generation, all-or-nothing swap);
setup.py --rollback-engine puts it back.
- #798: setup counts P/E cores from the hybrid PMU lists, else by 90% of the largest capacity (tests on a fake /sys).
- #775: a learned --expert-profile / --expert-profile-save added by hand survives a setup run.
- #779: a tip (--vram-reserve-mib 1500) when nvidia-smi says the card drives a display.
- #841: typed-ahead keys are flushed before a setup question on a terminal.
- #881: read_text() with encoding utf-8 in setup.py and package_windows.py; the CPU image encoder build finds vcvars on
Windows; PCI 73DF (RX 6700 XT / 6800M) is gfx1031 with a self-build hint.
Linux counted a core as a P-core only when cpu_capacity equalled the maximum. Turbo Boost Max 3.0 gives the "favored"
P-cores a slightly higher capacity, so a Core Ultra 7 270K Plus (8P + 16E) counted 2 P-cores and started one pool
worker. Intel's /sys/devices/cpu_core/cpus and cpu_atom/cpus decide when they exist; without them an E-core is one
under 90% of the largest capacity. A CPU whose capacities differ by a few percent is no longer hybrid.
- #480: the image encoder's directory has no space in its path (a TEMP under a user name with a space broke every
image: the ENC and GENI lines are split at spaces); the ENC line names its files relative to the encoder's cwd.
- #874: the per-request combined embeddings file goes to /dev/shm on Linux when it has room, and is opened
short-lived (FILE_ATTRIBUTE_TEMPORARY) on Windows, instead of heavy steady disk writes.
- #735: WinError 4551 / 1260 on starting the engine or the encoder says Windows blocked the (unsigned) program
(Smart App Control) and what to do. A message only; no signing.
- #779: a timestamped "engine started: exe args" line in the engine log.
- #769: only EADDRINUSE says "already in use"; any other listen error says what the OS said.
Co-authored-by: ziphao2010 <[email protected]>
A minimized console window made Windows 11 treat the server and its children as background work and move their
threads to the E-cores (about 20% slower on a hybrid CPU). contain() now sets ProcessPowerThrottling off for each
child, and the server does it for itself. Windows 10 knows only the EXECUTION_SPEED bit, so it is tried on its own
when both bits are refused.
Reported with the patch by mozophe.
NativeDense serves these formats; the packer sent them down the BF16 path, so such packs failed at start.
Co-authored-by: Wesleyvdk <[email protected]>
build did mkdir(exist_ok=True) and then wrote experts.bin, dense.bin,
embd.bin and manifest.json over whatever pack the directory held, with
nothing said. Now a manifest.json, index.txt or native_experts.txt in
--out (the files the engine reads a pack's identity from) stops build
before it opens the GGUF or writes a byte, naming the directory and the
file; it returns 1 like the other refusals. An empty or absent directory
proceeds as before, and so does experts.bin without manifest.json: build
writes the manifest last, so that is a build that did not finish, not a
pack.
tools/test_shards.py: each marker is refused with the path and the file
named and nothing in the directory touched; an empty and an absent
directory go on into the build; an unfinished build is rebuilt.
model_config_files() leaves strata-<model>.shared-settings.json out in the start menu, previous_config, the
data-folder move and strata_mcp.configs().
Co-authored-by: Christoph Wübbels <[email protected]>
#406 lets a PC below setup's 256K estimate keep 256K with a note, but the list
jumped 128K -> 256K: there was no step between the 128K rule and a size that PC
may not hold. 204800 is inside the trained 262144, so it needs no rope scaling,
and with IQ3_S it costs ~2.8 GB of 8-bit KV against ~3.6 GB at 256K.
Measured on an RTX 4080 16 GB, Ryzen 9 9950X3D, 62 GB usable, IQ3_XXS, engine
0.1.37, 8-bit KV (setup turns KV streaming off at this size, so the KV stays in
VRAM and the expert cache loses the difference):
context 128K 200K
decode, shallow 94.1 85.1 t/s (median of 10, warm)
decode, 30K deep 87.2 85.3 t/s
decode, 124K deep 94.1 79.2 t/s
prefill 3257-3284 3189-3318 t/s
experts cached 3661 2951 (-710)
a 203259-token prompt: 3093 t/s prefill, 68.6 t/s decode
tools/strata_mcp.py's fallback list mirrors CONTEXTS and moves with it, so a PC
that cannot import setup sees the same menu. tools/test_setup_risk.py picks 256K
at "6" now and a new test covers a 200K pick: kept, no rope scaling, and the RAM
note a 64 GB PC gets.
A .part holding every byte (setup stopped between the last byte and the
rename) was resumed with a range past its end; the server answers 416,
which download() retried 30 times 10 s apart before renaming the file.
get_prebuilt kept an engine archive that is not a zip (or a damaged one)
together with its .done mark. download() trusts the mark, so every later run
failed on the same file, even after the right archive was published; on the
update path the engine was never updated. The archive and its mark are now
dropped before the error propagates, as 0.1.32 does for a refused archive.
Compile check on ROCm 7.14.1 (Aurora): f-140 itself stops in fused_gr.cu (12 errors) and mtp.cpp (2) for gfx906 there; with this
commit the port adds no error of its own (fused_gr.cu's pre-existing ones remain).
Without --mtp, serve drafts with the suffix/prompt-lookup drafter only (or one
token per round), as the plain generate command already does; every token is
still verified against the model. The draft layer's VRAM goes to the expert
cache instead (1,000 -> 1,678 slots in an A/B on an RTX 2060 SUPER 8 GB). The
bind, the prompt path's draft prefill (batched and per token), the draft-KV
restore, prompt length, max drafts, draft sampling, coupled-draft history and
the draft call itself are skipped without it. The conversation cache is off
without it: a parked conversation always carries the draft layer's K/V.
With a split the two cards take turns on every verify window: stage 0 runs its
layers and hands off, then waits while stage 1 runs the rest, the head and the
draft. The batch pipeline (--batch-groups) overlaps different conversations;
one conversation's decode stayed serial.
--pipeline-windows 2 runs window K+1 on stage 0 while stage 1 verifies window
K. The guess for K+1 comes from the MTP chain, launched as soon as K's verdict
is known and teacher-forced through K+1's drafts: if K is accepted as guessed,
K+1 is already half done; if not, stage 0 restores its GDN state from a
snapshot, replays K's commit with the real count, and K+1 is rebuilt from the
real tokens. --pipeline-windows 1 overlaps only the prompt's short reads
through the windows (--short-read). Default 0: nothing changes.
What it adds:
- Verifier: a non-blocking form of run()/commit() (capture_all, pl_launch,
prestage, service, done, pl_finish, pl_commit_async), the batch pipeline's
batch_launch/batch_poll pattern for one sequence. The window, its staging,
the PLE gather at layer 0, the zero-doorbell graph's flag, the #267 release
on a timeout and the commit graph are run()'s. Two verifiers per stage (one
per window parity) share one stream of their card (set_stream) and each
pair has its own hand-off.
- set_always_publish: the pipelined verifiers copy the token rows to the host
at every layer, because the adaptive tier now marks experts evicted on the
host while windows are in flight, before the device residency table follows.
- MtpDrafter: chain_launch (the round and its steps back to back, an event
after each output, the first steps fed the in-flight window's drafts by a
one-thread forcing kernel), prefill without a host sync, 8 rows with 2, the
drafter's stream at the highest priority (CUDA only).
- the adaptive tier beside the pipeline: evictions are marked first, then the
refill copies wait (on the tier's thread) for every window in flight; the
RAM side of a swap (exchange, release, unpin) waits for the windows that may
still pull the incoming experts over PCIe.
- an agreement gate: a speculative window is launched only when the chain's
estimate says it is likely kept (STRATA_PIPELINE_THETA, default 0.2); a
window known to be wrong gets empty plans for its remaining layers; a lookup
window's continuation becomes the speculative window behind it.
- VRAM: 160 MiB more kept out of each card's cache for the second verifier,
and the two GDN snapshots on the first card.
- off, with one log line saying why, for anything it does not support: --batch
slots, --peer-device, the helper caches, three or more stages, a split on
one GPU, no drafter. Requests with repetition penalties or coupled draft
sampling decode serially. Nothing is in flight outside the two pipelined
loops, so checkpoints, parking and the serial paths are unchanged.
Measured: Swift 1.5 IQ3_XXS at 160K (q4_0 KV, the stock draft layer), RTX 4060
Ti (layers 0-19) + RTX 5080 (20-47), i9-14900KF, 32 GB of RAM, the resident
RAM mode on the split (#848); greedy, 500 tokens, two interleaved pairs of
three rounds, decode tok/s, serial -> pipelined: Python code 90.4 -> 103.1,
C code 76.6 -> 81.6, English prose 63.0 -> 71.1, Italian prose 41.6 -> 46.3,
a copy-heavy edit (a 5 KB file back with a rename) 90.5 -> 117.9; mean
72.4 -> 84.0 (+16%). With the experts read through the OS file cache
(--mmap-experts on that PC) the file reads dominate and it measured no faster.
Correctness: with STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0 the
greedy text is identical to the serial loop's when both have the same expert
caches (4/4 prompts, 500 tokens; the serial run given the pipeline's reserve
through --vram-reserve-mib). The same text with STRATA_PIPELINE_THETA=2 (no
speculative window), STRATA_PIPELINE_FORCE_MISS=1 (every speculative window
wrong, rolled back) with and without DOOM_SKIP, PRESTAGE=0 and
SNAP_OVERLAP=0. Against a run without the flag a few experts move from a card
to the CPU (the pipeline's VRAM), which rounds them differently, so a near-tie
can flip. With the adaptive tier on: 22 varied requests at temperature 0.6
(thinking on and off, five languages, code), and 11 more with
STRATA_PIPELINE_FORCE_MISS=2, gave no degenerate answer, no stall.
The watchdog prints the pipelined loop's state when it fires;
STRATA_PIPELINE_TRACE=<file> writes a timeline.
For the pipelined verify windows of a layer split (next commit). Nothing calls
it yet, and nothing changes unless set_force_capture is set before load.
- chain_launch: the catch-up round and its steps launched back to back with no
host wait, an event after each output (chain_outputs_ready reads them as they
land, chain_poll once the chain is done). Its first steps can take given
tokens as their input instead of their own picks (teacher forcing: the drafts
of a window already in flight), through a one-thread kernel at the end of the
round and step graphs that does nothing while its mapped token is -1; the
outputs stay the chain's own picks.
- prepare_chain / prepare_prefill: capture every graph and size the prefill's
input records up front, so neither a later chain nor prefill(sync = false)
waits for the device.
- set_source_R: the rows a round reads, copied into the bound buffer first (the
pipelined loop has one stage-1 verifier per window parity).
- with forcing on, the drafter's stream gets the highest priority (CUDA;
STRATA_MTP_PRIORITY=0: the default).
draft() resets the forced tokens and copies the source rows the same way, so
the serial loop's drafts are unchanged.
Extract rotation and atomic ownership publication from aa6e0e3 and 1a50d91 onto upstream a1641e9. Preserve current host-memory tests. Include one historical token-identical A/B with explicit provenance and document fresh CPU/CUDA validation.
In the resident RAM mode (--resident-experts) an adaptive round runs on a
thread the decode loop joins after the draft: choose, copy the evicted slots
back with a stream sync, copy the new experts in from the RAM copy (through
the driver's staging when it is pageable). With ~90 swaps that was ~24 ms
every 4th window (measured on our fork). --adapt-async 1 splits a round into
four steps that advance between windows on a helper thread, so no window
waits for one:
1. choose (adapt()'s rule, over snapshots of the counts and the residency)
and copy the evicted slots back (D2H), each from the cache of the card
that owns its layer
2. stage the exchanges, mark the evicted experts non-resident (the CPU reads
them from the exchange buffers), copy the new experts in: straight from
a page-locked RAM copy, else through a pinned bounce buffer
3. mark the new experts resident, move the evicted blobs into the RAM places
the new ones left
4. flip the copy's offsets
A slot is overwritten only after its old expert stopped being planned for the
GPU, and a RAM place is reused only once its expert is resident. The residency
tables on the devices are uploaded at steps 2 and 3, as apply_pending does
(the verify graph's doorbell reads them). A request starts with the round
drained, before its prompt lends any slot. --expert-profile-save counts the
routing at the start of each round, as the blocking tier does. Swaps that
need no exchange (the evicted expert is in the RAM copy already, or the new
one is not: a RAM budget) take the same steps without a copy back; a new
expert outside the RAM copy is read from the files on the helper thread.
Default off: which window first computes a swapped-in expert on the GPU
depends on when its copy lands, so the tier is not bit-exact from run to run.
Without the flag nothing changes. With it, the blocking tier still runs
outside --serve, without the resident RAM copy, and with --batch slots,
--peer-device or --remote-expert-opt (said once at startup). The exchange
buffers are twice as many with it (the bounce half): 96 more blob-sized
pinned buffers, ~130 MiB with 1.38 MB blobs.
Layer split (with the resident mode on a split, #848): each swap's copies run
on the owning card's refill stream under that card's device; the homes are
looked up the way #848's resident_stage_swaps does.
Measured: Swift 1.5 IQ3_XXS at 160K (q4_0 KV), i9-14900KF, 32 GB of RAM,
Windows 11, greedy, decode tok/s, blocking -> --adapt-async 1:
- RTX 4060 Ti + RTX 5080 layer split with the resident RAM mode (#848),
--trim-stage-weights, the stock draft layer, five prompts x two rounds,
two interleaved pairs: 79.7 / 78.8 -> 87.6 / 86.2 (+10%).
- the same split with a fine-tuned draft layer and --spec-min-p 0.65, four
prompts x three rounds:
81.0 / 81.8 -> 108.1 / 107.0 (+32%).
- RTX 5080 alone with --resident-experts: 26.7 / 26.6 -> 29.5 / 28.5 (noisy).
Without the flag the greedy output is the same as main's, byte for byte (4/4
prompts, 500 tokens, STRATA_IQ_MT_MIN=1 --pcie-frac 0 --adapt-every 0). With
it, 22 varied requests at temperature 0.6 on the split (43,825 experts
swapped in over 622 rounds) gave no degenerate answer.
Drops the dead stage_pool trim block (the search only runs for auto, where stage_trim is false), keeps the search's number
for each card unchanged (the full untrimmed arena), and fixes the 'needs an explicit --layer-split' warning, which now
fires only when there is no GPU split at all.
--trim-stage-weights (PR #559) / STRATA_STAGE_TRIM=1 (PR #639) gives a stage only its own layers' dense
weights, but it refuses to run unless --layer-split is written out: `auto` chooses the split points 590
lines after the stages' weights are loaded, so by the time the split is known there is nothing left to trim.
Those weights now load after the search. The search still prices each card's arena from the pack index
(`WeightTable::pool_bytes` reads the index and allocates nothing), and takes it off the card's measured
free VRAM, so the number it sees is the number it has always seen - the split it picks is the split it has
always picked. This buys cache, not a different placement. The expert caches are sized later, from live
free VRAM, so they collect the freed bytes.
On the 4-way IQ3_S rig, `--layer-split auto` (which picks K=10,19,34):
CUDA1 3650 -> 4608 expert slots, CUDA2 1241 -> 2485, CUDA3 533 -> 1772
prompt chunk 768 -> 6144 tokens
prefill 469.7 -> 1929.7 tok/s on the same 107,082-token prompt
CUDA0 still trims only with an explicit --layer-split. Its arena is read out to build the prompt path's PLE
tables, 590 lines before the split exists, so moving it is the larger half of this change. It is also the
cheap half to leave: measured, it costs 1% (1929.7 against 1950.8 for an explicit `10,19,34
--trim-stage-weights`, where CUDA0's cache is 4598 slots instead of 3017).
bench/results/2026-10-04-rdna2-helper-pcie-share: the rig (two cards on PCIe 4.0 x8, probe 14.1 GB/s -> pcie_frac 0.39),
the method (STRATA_DECODE_TIMING per request), and the per-window breakdown for the baseline on a1641e9 (with and
without #835 on top: 59.6-60.4 ms per window, 4 PCIe entries per layer-window, the GPU-reach wait at 43 ms), for the fix
with the probed share (two runs: 37.2 / 33.8 and 35.2 / 33.8 ms, the share down to 0.06-0.26 entries) and with
--pcie-frac 0, and for the --pcie-frac 0 workaround on the baseline; the limits and the configuration used.
docs/SECOND_GPU.md: the helper section said HIP had not been validated; it now gives the two-card HIP measurement, what
the share cost before this change and the workaround, and keeps three/four cards as not validated.
The verify window's GPU plan counted a layer's misses and moved the last pcie_num/256 of them to the primary over PCIe,
leaving out the primary's resident experts and the peer tier's but not those a helper GPU (RemoteExperts) holds. The
plan runs before RemoteExperts::begin(), which only claims rows the plan left at kind -1, so a PCIe share above 0 read
part of the helper's experts over the primary's link while the helper idled for them.
RemoteExperts::holds(layer, expert) (the helper cache's slot_of), and the plan treats such an expert as it treats the
peer tier's: not a miss, never in the PCIe share; its rows stay at kind -1 for begin(). No default changes.
2x RX 6900 XT, IQ3_S, --remote-expert-opt with the probed share 0.39: 59.6 / 58.5 -> 37.2 / 33.8 ms per window,
PCIe entries per layer-window 4.00 -> 0.26 / 0.06, decode 40.6 / 39.3 -> 65.6 / 67.7 tok/s (--pcie-frac 0: 66.9 / 67.1).
Maintainer: separate stop and recover modes (reasoning_loop_recovery: false|stop|recover), token-level splice of the effort sentence (EFFORT_TEXT), a client's top_p/top_k never touched, checks at the first clean boundary after each 512 tokens, bounded history, tests incl. a literal think tag (#728, #869)
A request with "strata_checkpoint": false sends ckpt=0 to the engine. That request reads its prompt in one part and
saves no checkpoint at its last turn boundary, nor the periodic ones of a long prompt. It still resumes from a
checkpoint it matches and still saves the system-prompt root. Absent or true: today's behavior; an engine without the
key skips it.
prepare() wrote the request's combined embeddings file (req-<id>.sve in the vision directory, ~10 MB a picture) right
after encoding, and only run()'s finally deleted it. But prepare() could still raise after writing it - EngineStarting
(the 503 while the engine starts, #344), the "no room to answer" and "exceeds the context" 400s - and a request can end
between prepare() and run(): its client gone before the first chunk was written, anything raised in between. run() is
a generator, so one that never started has no finally to run. Each of these left its file behind for good, and
clients that retry 503s added one per retry.
prepare() now writes the file only after those checks, and records its path before writing it. run() takes the path
over as it starts and deletes the file as before; do_POST's finally calls the new Service.drop_embeddings(), which
deletes a file that no run() took over.
Tests: EngineStarting, no room and exceeds-the-context leave no req-*.sve; over HTTP, on both APIs, a 503, a 400, a
client gone after prepare() and answered requests leave none.
The markers are special tokens only where the template writes them for an image item. The same strings in a
message's text (an agent reading chat_template.jinja, a tool result quoting it) became the same special ids, so text
with <|vision_start|><|image_pad|> before a picture took that picture's embeddings, every later picture moved up one
and the last real marker became text - every count still matched.
On 0.1.39 this takes #537's path: the vision markers join <think> / </think> as literal tags (frontend.VISION_TAGS),
swapped for private-use marks before the template and encoded as text through the tokenizer's plain spans. A pair
of markers cut apart across text parts is refused ("the prompt and its images do not match") instead of shifting.
With tool_result pictures reaching the encoder, a server without one (every lazy-start config is text-only) or an
encoder that refuses a picture (WebP without Pillow, an error from strata-vision) answered 400. Clients resend the
whole history every turn, so one such picture failed every later request of the conversation; before, the picture
was dropped silently. Now a tool's picture that cannot be read becomes "[image omitted: <reason>]" in its place; a
picture the user sends keeps the 400.
Maintainer: log a left-out tool picture, a tool picture's URL is downloaded once (bytes reused by prepare), tests (#529; #582 and #819 are duplicates)
Claude Code returns a picture it opens with Read as an image block inside a tool_result. The Anthropic
path turned every tool_result into its text, so the image never reached the encoder and the model
described a picture it had not seen. The tool message now carries the image item, as the OpenAI path's
tool messages already did; the chat template renders it inside <tool_response>. An image sent beside
tool results in the same user turn is kept too. A text-only tool result is still one string, so text
prompts are unchanged.
jsonschema is optional (serve/structured.py): without it a json_schema answer is checked to be one JSON object, so
the answer the test expects to fail passes, and the test failed on every install without it (0.1.39's own as well).
Codex sends its MCP tools as namespace tools (mcp__websearch with web_search
inside). The Responses adapter shows the model mcp__websearch.web_search, but
Qwen3.8-Flash-Next writes the call as mcp__websearch__web_search, the flat
form Codex used for MCP tools. That name had no entry in the reverse map, so
the call came back without a namespace and Codex rejected it as an
unsupported call. Register the double-underscore spelling too (setdefault, so
a real tool of that name still wins).
prepare_format refused every schema whose root was not literally "type":"object",
so a root anyOf/oneOf of object shapes - what apps written for llama.cpp's
json_schema send - got a 400, although every answer it allows is one JSON object.
_only_objects() accepts a root that can only be an object: "type":"object", an
anyOf/oneOf whose branches all qualify, an allOf with a qualifying member, or a
local $ref to one (a $ref cycle never qualifies). The "return exactly one JSON
object" directive and the object check without jsonschema stay true. A root that
also allows an array, string, number, boolean or null (including
"type":["object","null"] and a schema without a type) is still refused before the
engine loads.
Tests: serve/test_structured.py, unions accepted (a non-object answer is still
refused) and non-object roots refused: 10 OK with and without jsonschema
installed. test_server/security/lifecycle/detok/mcp/monitor/winjob: 214 OK.
docs/DETAILS.md: the root rule.
The server is HTTP/1.0 and closes the connection after each response, but
never said so. A client that pools connections (.NET's SocketsHttpHandler)
then sometimes put its next request on the closing socket and got
'response ended prematurely' without the server ever reading it: 25 of
~1000 requests in one night from an agent app, each 1-24 ms after the
previous answer.
A reply such as "I will use the `<tool_call>` format now." followed by the call raised
ValueError("malformed tool call") in OutputParser.feed and ended the request: everything after the
first <tool_call> was taken as the call body. A call is the tag followed (after whitespace) by
<function=, as the chat template writes it; the tag with anything else after it is now content, and
it is held while streaming until its follower arrives, so streamed and whole output agree.
Found while porting tolerant tool-call recovery from a Qwen engine (ninfer --tolerant-tool-calls):
of its three recovery cases, Strata already handled values that quote the markup (#210) and prose
after the calls; this was the one that failed. Tests: ToolCallTagInProse (fails on main with 21
subtest errors, passes here); python -m unittest serve.test_server: 120 tests OK.
Without a grammar decoder a forced tool_choice was ignored, so clients
that need a call (an agent's first step, its "done" call) got text.
The server now writes the call's opening itself - <tool_call> and
<function= (with the name for a named choice) - and the model can only
go on with a call:
- without thinking, the opening ends the prompt;
- with thinking, it is written where the thinking ends, at a clean
point, the way the budget's wrap-up is (continuing from the prefix
the engine already holds);
- when reasoning_budget_tokens wraps up the thinking, it follows the
wrap-up.
"auto" and no tool_choice are unchanged. "none" offers no tools, as on
the Responses route. An unknown name, a forced choice without tools
and other values are a 400 instead of being ignored; a forced choice
with MCP tools too. The arguments are not constrained; a reply cut by
max tokens stays "length" without tool_calls, as before.
Maintainer: unknown tool_choice values are logged and act as auto (no 400), the flat {type:function,name} shape, Anthropic any/tool/none mapped and passed to /v1/messages, tests (#790). Note: tool_choice none now offers no tools on both routes.
A body over 1 MiB was left unread, and its answer was reset on Windows. The limit is now 5 s for the whole body, read one socket read at a time: a body of any size gets its answer, and a client that sends a byte at a time is let go after the 5 s (read() waited for the whole piece, with a new timeout for each byte).
The server answers before reading the body in some cases (401, 403, /load, /unload, a method with no handler) and then closes. Closing on unread bytes makes Windows send a reset, and a client that sends its body after the headers (http.client, urllib, requests) gets WinError 10054 instead of the answer.
After an answer, the body no handler took is now read and dropped, up to 1 MiB and with a 5 s timeout. A larger or chunked body is left unread, as before.
Filter earlier assistant messages with no text, images or tool calls
before rendering the prompt. Keep the final message.
Co-authored-by: jramirez-uab <[email protected]>
Signed-off-by: Arthur031221 <[email protected]>
With tools declared, a <tool_call> in the reasoning is held whole; a declared name becomes a tool call,
anything else (undeclared name, malformed, unfinished, cut by </think>) stays reasoning text. finish() never
makes a call from it. Without tools the parser is byte-identical. Tests: parser (split tags, undeclared,
no tools, unfinished), OpenAI and Anthropic finish reasons, Anthropic block order.
Co-authored-by: kossum <[email protected]>
Maintainer: fatal lines checked before batch routing, 'an earlier window never finished' is fatal, alive() check in generate_batched, tests run on Windows (#748).
Vision.load read whatever an image source named. A network path (\\host\share\x.png, //host/share/x.png,
\\?\UNC\host\..., \??\UNC\host\..., file:////host/share/x.png) made Windows connect to that host and sign in with the
user's NTLM credentials - os.path.isfile alone does it. Such paths are now refused before anything looks at them, and
so is file://host/share/x.png (that names \\host\share; the old code read it as the relative path host/share/x.png).
An http(s) image was read whole, with no cap, inside prepare()'s `with self.fifo`: a huge one filled the memory and a
slow one held every other request for as long as urlopen waited (60 s per read). Now it is read up to 32 MiB
(IMAGE_URL_MAX; a larger Content-Length is refused unread), downloaded before the FIFO is taken, and a URL that cannot
be read is a 400 instead of a dropped connection.
Other sites' pages: with no api_key (the default) _authorized() lets every request in, and /v1/chat/completions and
/v1/messages read the body as JSON whatever its Content-Type; only /settings, /v1/load, /v1/unload and the MCP tools
check the Content-Type and Origin (_own_page). So any web page open in the user's browser could POST a chat request as
text/plain - a CORS "simple" request, no preflight - that names a file on this computer (or a network path) as an
image, and the server read it. The page gets no answer to read (no CORS headers unless cors_origins lists it or "*"),
but the read, and a network path's sign-in, happened. Reproduced over HTTP: answered 200, the file handed to the
encoder. Now a request whose Origin is neither the server's own address nor in trusted_origins may send data: and
http(s) images only; a file path or file:// URL is a 400 that says so. Clients that send no Origin (CLI tools, SDKs,
agents), Strata's own page (it sends data: URLs) and trusted origins are unchanged. As for /settings and MCP, a
DNS-rebinding page, whose Origin matches its Host header, still counts as the server's own.
Tests: each network form is refused with os.path.isfile never called, and local files, file:// and data: URLs still
load; the cap with and without a Content-Length, and a 404; the URL is downloaded with the FIFO free; a foreign-Origin
text/plain request naming a file is a 400 on both APIs, while data: images and no-Origin, own-origin and
trusted-origin requests are answered.
Row error on 4,000 random rows against the checkpoint's BF16 table (Q8_0 0.53%, FP8 2.64%, IQ4_NL 7.60%, matching the
numbers reported in #651), and teacher-forced KL on the Q2_0 model with each table.
The 0.1.39 paragraph said nobody had run the 0.1.39 port on an Arc. Two Arc Pro B60 reports (#867, #870) did, with
the two sycl/ fixes now in 0.1.40 (#784 the compile fix, #866 the ring wait). Say that, and that an unpatched 0.1.39
port did not compile against its own engine sources.
INTEL_ARC.md: "What has been run" rows for 2x and 1x Arc Pro B60, plus notes for the card: PCI id 8086:e211 (bmg-g21), setvars.sh
stopping at OCL_ICD_FILENAMES under `set -u`, running natively without Docker, and the layer-split startup warning that counts the
other card's layers.
sycl/setup_intel.py: the B60 table entry was e221, which the kernel's xe id list files under the BMG-G31 cards (0xE220-0xE223). Both
Arc Pro B60 cards measured here are 8086:e211, so setup named them "Intel GPU e211 (xe)" with a guessed VRAM size. e211 is added;
e221 stays, since there is no card here to say it is wrong. tools/test_setup_sycl.py checks the table.
COMMUNITY_BENCHMARKS.md and bench/results/2026-10-04-community-2x-arc-pro-b60/: the report, in the template's format, with the
server configs, engine logs and per-request timings.
The migration turned `q = cudaStreamQuery(cs)` into `DPCT_CHECK_ERROR(cs->ext_oneapi_empty())`. That macro yields 0 whenever
the call does not throw, whatever the queue holds. The guard `q != 1 && *seq < want` (verify.cpp, session.cpp) therefore
fired on any ring that took more than 2 ms and reported "never rang (graph finished)" while the graph was still running.
With every expert resident the host does not wait per layer (STRATA_VERIFY_NO_HOST), so the path measured on the B70 never
reached it. With host-mirrored experts the first layer whose GPU time passes 2 ms does: layer 15 of the Coder on one Arc Pro
B60, every time.
q is now 0 when the queue is empty and 1 while it is running, as the guard and the trace comment ("1 = still running") expect.
fixups.py gets the same rewrite (entry 8), so a fresh dpct run of session.cpp produces it too.
Measured on one Arc Pro B60 (4,042 of 12,288 Coder IQ1_M experts mirrored in RAM, 128 greedy tokens, AOT bmg-g21): before,
exit 1 at layer 15 in 5 of 5 tries; after, exit 0 in 7 of 7 runs, at 11.2-11.9 tok/s with coherent text.
The sycl port went in as 5ffcce4 ("sycl: port 0.1.39"). Two engine-side changes
were merged into the release branch after it (4f5e6c8, f5410ae) and nothing in
sycl/ was updated for them, so the v0.1.39 tag does not compile:
- #626 (5a8b292) made the thread affinity a value: pin_current_thread()
returns strata::kernels::cpu::ThreadAffinity and restore_thread_affinity()
takes it. sycl/include/strata/core/session.hpp still stored a `long long`
(no conversion exists, and it did not include the header that names the
type); sycl/src/core/session.cpp still assigned the returned struct to it
and passed the `long long` back.
- #559 (9461f5e) added the two layer-range arguments to NativeDense::load.
The sycl out-of-line definition was still the 4-argument one and no longer
matched the declaration in include/strata/core/native_dense.hpp.
Both are fixed by mirroring the CUDA sources: the same member type, the same
`outside()` predicate, and the same place it is applied in the tensor loop.
`pinned` now follows CUDA as well: it is pinned_core.valid, so a pin that
failed is not "restored". Nothing changes when the pin succeeds.
No caller passes layer_lo/layer_hi yet, so `outside()` is never true today:
this restores the build and does not change what the engine computes.
Measured here: Arch Linux, Intel oneAPI DPC++ 2026.0.0 + oneMKL 2026.0,
SPIR-V (JIT), cmake -S sycl -B build-sycl -G Ninja -DCMAKE_C_COMPILER=icx
-DCMAKE_CXX_COMPILER=icpx, then
`cmake --build build-sycl --target strata -- -k 0` -> 0 errors, links. The two
errors are a type and a declaration mismatch, so the compiler version does not
matter.
Not built here: the other targets (kernel parity tests, benches) and the AOT
path. Not run on an Arc in this change.
The full checkpoint stores the table as BF16 (320-byte rows, 102.4 GB), the source of record; the engine gets it as
IQ4_NL (about 8% off per row) or FP8. A bfloat16 is the top half of a float32, so widening is a shift: exact for
every value, no rounding and no scale. One entry in the PLE format list (type BF16, self-describing: no scale, no
metadata) and bf16_dequant_row; the direct reader takes the 320-byte row like any other width, and
PLE_ROW_BYTES_MAX is 320.
ple_reader_test checks the widening is exact over the patterns a lossy path breaks on (subnormals, +-inf, both NaNs);
the generic every-format test covers a synthetic BF16 GGUF through both readers.
Not in this commit: the PR's rename of tools/ple_fp8_pack.py (nothing needs it) and any setup option. Whether a
table format other than IQ4_NL is worth offering is a quality question (see the measurement in the 0.1.40 notes).
Ported by the maintainer from constantindjonkam's PR #586.
A plain llama-quantize Q4_0 file of this model (bartowski's Qwen3.8-Flash-Next-Q4_0 and
Swift-1.5 Q4_0: gate/up Q4_0, down Q4_0 on 42 layers and Q4_1 on 6, a Q4_0 PLE table and
token embedding) was refused at start: no GPU kernels for Q4_0/Q4_1 experts, and the PLE
reader took only IQ4_NL, Q5_0 and FP8. Follows #222 (Q4 to Q6 quantizations).
- Q4_0: Fmt<2> with the codes centred before dp4a (q - 8 as signed bytes), the exact
integer form ggml-cpu's q4_0 x q8_0 dot uses, rather than llama.cpp's CUDA chain
(uncentred codes minus 8 x the q8_1 float sum): on a real Q4_0 expert the CUDA chain
measured 1.4e-2 relative error against the float product, the centred form 5.3e-3.
- Q4_1: Fmt<3>, the Q5_1 chain without the fifth bit and the same min-term choice.
- dq_q4_0 / dq_q4_1 for the prompt path's dequant fallback; row bytes, is_iq and the
GU/D/MMVQ format lists.
- PLE: Q4_0 rows (90 B, as IQ4_NL) through the same reader: one entry in the PLE format list (ple_format_test and
ple_reader_test cover it).
- docs/UNSLOTH_Q4.md: a Q4_0 section with the measured run.
native_expert_parity --synthetic (now in the CMake matrix): q4_0/q4_0 CPU 1.29e-2, GPU
1.29e-2, CPU-GPU 1.1e-7; q4_0/q4_1 CPU-GPU 3.2e-4; GPU dequant bit-exact against
to_float. The existing pairs are unchanged. On bartowski's file (revision 928589f), layers
0 and 5 (Q4_1 down) and 20 and 47: CPU and GPU 1.1-1.3e-2, CPU-GPU 1.1e-7 (Q4_0 down) and
5e-4 to 1.1e-3 (Q4_1 down). Served on one RTX 3060 (all experts in RAM, 131K context):
24,461-token prompt 787 tok/s read, 24.0 tok/s out; 67,934 tokens 737 / 26.4, the hidden
fact found.
Ported onto the format table by the maintainer (the PLE part of the PR is the Q4_0 table entry).
mlock alone brings a cold file in one page at a time from one thread, ~0.47 GiB/s (about two minutes for a 54 GB
Q8_0 table); 16 threads reach ~2 GiB/s on anon761's machine. The pages are then resident and mlock only pins them.
When mlock fails, the pages stay faulted in (the message says they may be reclaimed) instead of being touched again
one by one.
ple_reader_test gets --make-table PATH FORMAT (a big synthetic table, with --rows) and --ram-time PATH (open mapped
and locked, print the seconds) to time it cold.
Ported by the maintainer from anon761's PR #651.
Co-authored-by: ksanislo <[email protected]>
PleTable::open compared rows x row_bytes against the file after the 64-bit product could already have wrapped, and
mapped the tensor before checking it fits. Now: refuse a row count that overflows rows x row_bytes and a data
section that starts past the end of the file, check offset and size without an addition that can wrap
(offset > have, need > have - offset), require a single-tensor shard to start at offset 0 and fill the file, and map
the table only after all of that. close() also resets the FP8 scale and the byte counter.
ple_reader_test --selftest writes tables that claim one row too many, one too few and a wrapping row count, for every
format, and checks both readers refuse each with a message and stay closed.
Checks taken from CC-David-CC's #865 (its opt-in Q8_0 table is #651's; the direct reader reads Q8_0, so the
experimental switch and the mapped-only limit are not needed).
Co-authored-by: CC-David-CC <[email protected]>
Ordinary GGUFs ship per_layer_token_embd.weight in other formats than IQ4_NL: Unsloth's UD-Q6_K_XL and
Swift-1.5 Q4_K_L as Q8_0 (170-byte rows, ~54 GB), a Q5_K_M finetune as Q5_1 (120-byte rows). The engine refused
both at start.
- dequantize_q5_1 in strata/artifact/dequant.hpp (ggml's dequantize_row_q5_1: unsigned 5-bit codes times d plus the
block minimum m); Q8_0 rows use the existing dequantize_q8_0
- two entries in the PLE format list (the table-driven commit before this one): Q5_1 and Q8_0, on both readers;
PLE_ROW_BYTES_MAX grows to 170
- tests: dequant_q5_1_test (synthetic blocks against ggml, no model, no GPU), dequant_bf16_test knows Q5_1,
ple_q5_parity takes Q5_0, Q5_1 or Q8_0 tables and checks both the mapped and the direct reader
Ported onto the format table by the maintainer from anon761's PR #651 (the --ple-io ram fault-in is the next
commit). Quality of the Q8_0 table: ksanislo measured 0.53% error against the BF16 table, FP8 2.66%, IQ4_NL 7.59%.
Co-authored-by: ksanislo <[email protected]>
PleTable had one bool per format (q5_0, fp8) and an if-chain in decode(), open() and format(). Now one list,
ple_formats(), holds one entry per table type: its GGUF type name, the bytes of a 160-value row, whether it
carries a scale and its dequantizer. open(), format(), the row size, the error string and the row buffers all
read it, so the next formats (Q8_0, Q5_1, Q4_0, BF16) are one entry each.
No behaviour change: IQ4_NL, Q5_0 and FP8 decode with the same functions, and the "not IQ4_NL, Q5_0 or FP8 (I8)"
error text is built from the list and reads the same.
ple_reader_test --selftest now writes a PLE-only GGUF for every format in the list and checks both readers
(direct and mapped) against the format's own dequantizer, for single rows and a 16-row token gather.
The kernel reproduces one hipBLASLt solution's summation order; the test compares both on the same inputs for T from 64
to 16384 (bitwise) and SKIPs without STRATA_HIPBLASLT_TUNING.
16 switches whose output was byte-identical to the switch off on Aurora (GDN head / PP / CONVL2 / NOY / SPLIT, QFUSE,
CVEC_FUSE, HCD_EXACT, PF_PAD, Q8 / Q6 packed, MMVF_ROWS, ATTN_LANECELL, EXPERT_V2, TSUM, LFUSE, PLE_BATCH) are set at
start-up when device 0 is gfx1151 and the user has not set them; STRATA_GFX1151_DEFAULTS=0 turns the table off. The
GDN head kernels are AMD-only now (HIP builds; CUDA keeps its key-head kernel), and verify.cpp's namespace-scope env
reads became lazy so the table reaches them.
WmmaTraits::mma's #else returned its accumulator (zeros) for a gfx11 part outside the __gfx1100__..__gfx1151__
list; it traps now, and the runtime gates of the WMMA GEMM, the prompt attention, the block scorer and the HC
up-projection take exactly gfx1100 / 1101 / 1102 / 1150 / 1151 (strata/kernels/gfx_arch.hpp) so a gfx1103 or
gfx1152 falls back to hipBLASLt instead of launching code that is not there. The gate is cached per device
(a process-wide static answered for the first GPU only; hipGetDeviceProperties on every call was the review's other
complaint).
0.1.39 already skips the host round trip when every expert is in VRAM (all_resident_, on by default, #871's
refresh_ar), so the port keeps that graph and adds only what halo built on top of it: the LFUSE gated combine.
A prompt loan, a VRAM shrink or an adaptive swap marks experts -1 in the residency table. The zero-doorbell graph
planned from the device table and returned silently on a -1, so the window ran a stale plan (!!!! at 100% residency).
refresh_ar() checks the host table before each window and runs the doorbell graph (its own captured variant) while any
expert is out; resident_plan also writes a mapped error flag the host checks after the window.
Every upload of the expert residency table (host_res -> d_res, and each stage's) was a plain cudaMemcpy: a pageable
copy on the legacy stream, which may return before its DMA has landed, while the kernels that read the table run on
streams created cudaStreamNonBlocking, which do not wait for the legacy stream. A kernel enqueued right after could
read a table half old, half new: an expert computed by both the GPU and the CPU, or by neither. The ~100 KB copy
lands in microseconds, so this rarely fires, but nothing ordered it. The uploads now go through res_put(), which
waits for the copy on the current device's legacy stream only. Same values, same output.
With an engine that keeps --resident-experts on a layer split, setup no longer
steers a low-RAM PC with two cards to one of them:
- at install, the low-RAM mode on several GPUs takes the resident variant by
the same rule as on one card (what no card holds fits the RAM with
LOW_RAM_HEADROOM_GB beside it), without the one-card question;
- at a start, a resident config on several GPUs keeps --resident-experts
(split_mmap leaves it alone);
- a resident config started on a PC with two cards is offered both, "y" by
default, as any other config.
It is gated on MIN_ENGINE >= RESIDENT_SPLIT_ENGINE (0.1.40 here, a guess at
the release that would carry the engine change), so until then setup behaves
as before and the golden configs are unchanged. Two tests run the new paths
with MIN_ENGINE patched; tools/test_setup_*.py pass (260 tests).
--resident-experts (the low-RAM mode's resident variant) ran as plain
--mmap-experts as soon as a layer split was set. It now keeps its RAM copy on
a split:
- the experts every stage's cache holds are left out of the RAM copy, not only
CUDA0's;
- when the rest does not fit whole (#467), the copy keeps the hottest by the
whole expert profile. Each stage only keeps its own layers' share of the
profile, and ranked by CUDA0's share alone the copy held layers 0..K-1's
experts and read every later layer's from the files (4.0 GiB in RAM instead
of 21.2 on the PC below);
- an adaptive swap copies the evicted expert back into RAM from the cache of
the card that owns its layer, on that card's stream. host_res holds slot
numbers of whichever cache owns the layer: reading a later stage's slot out
of CUDA0's cache put another expert's bytes in RAM, and after the first
swaps the answers turned to garbage. (Latent until now: the guard kept the
mode off a split.)
- the decode profile prints every stage's GPU profile, not only CUDA0's.
Remote expert caches (--expert-cache-remote) are still refused with it.
Swift 1.5 IQ3_XXS at 160K, q4_0 KV, RTX 4060 Ti (layers 0-19) + RTX 5080
(20-47), i9-14900KF, 32 GB of RAM, Windows 11, four greedy prompts, decode
tok/s (first round / fourth):
- 5080 alone, --resident-experts: 25 / 29
- split, --mmap-experts (what --resident-experts became): 32 / 64
- split, --resident-experts (22 GiB of experts locked in RAM): 71 / 69
With --pcie-frac 0 --adapt-every 0 the split's greedy output is the same as
--mmap-experts' (4/4 prompts). With adaptive swaps on, 33 varied requests at
temperature 0.6 (thinking on and off, five languages, code, each checked for
degenerate output) exchanged 81,783 experts with the two caches and gave no
degenerate answer.
--vram-reserve-mib is every card's reserve, but the card that drives the
monitors needs more headroom than one that drives none: one value has to fit
the display card, and the other cards then keep that headroom for nothing.
--vram-reserve-later-mib N is the reserve of every card after the first
(default: --vram-reserve-mib's value), in the split search and in each later
card's cache sizing.
RTX 4060 Ti (no display, layers 0-19) + RTX 5080 (display, 20-47), Swift 1.5
IQ3_XXS at 160K: --vram-reserve-mib 1000 gives the 4060 Ti's cache 4551 slots;
--vram-reserve-mib 300 --vram-reserve-later-mib 1800 gives it 4867, and the
5080 keeps 1.3 GB free with everything loaded.
Bitwise-identical to e58d7fc (generated ids identical on UD-IQ4_XS, UD-Q4_K_XL, IQ3_S, IQ3_XXS at 8K/4K/64K; harness fused-output hash unchanged):
- table16: IQ4_NL / IQ4_XS decode through the direct-selector v_perm form (X5; exhaustive 2^32 check). -DSTRATA_W_NO_T16 restores the old form.
- moe_combine: four columns per thread with float4 loads, same fma order (X5); falls back to the scalar kernel if Dm / shared / bo are not 16-byte aligned. -DSTRATA_W_NO_COMB.
- native_w11_kernel: the H tile aliased onto the weight buffers (gate/up LDS 46.5 -> 30.2 KB), amdgpu_waves_per_eu(8), 4 blocks per WGP (X1), only for the types that do not spill under the 192-VGPR cap (gate/up IQ2_*, IQ3_*, IQ4_XS; down Q2_0, Q8_0, IQ4_NL). -DSTRATA_W_NO_OCC.
- X3 on1 (skip the second 16-row n tile of the down kernel) NOT included: no gain alone (-0.1%), and with the occupancy cap it makes IQ4_NL down spill 62 VGPRs and the whole bundle 2.3% slower.
0.1.39 clamped two q8_1 activation quantizers (#606: an fp16 sum or scale past 65504 rounds to inf, the dot products
then read inf * 0 = NaN, and a NaN in the residual ends as one token repeated). Two more store the block unclamped:
- native_swiglu_quantize_q8_1_kernel (native_mmvq.cu), the shared expert's fused SwiGLU + q8_1, on by default
(STRATA_FUSED_SWIGLU_Q81) in decode and the multi-token pass;
- native_gu_fused_kernel (iq_kernels.cu), the routed experts' fused SwiGLU + q8_1 (gfx906 builds, STRATA_EXP_MODE=8).
Both now go through q8_1_finite / q8_1_quant / q8_1_ds as their siblings do: every finite block is stored bit for bit
as before. iq_multi_parity's #606 check covers the shared expert's one (gate = 32, up = x / 32: its activations are x
exactly). rwkeyes found the first one (#606).
gdn_rec_quad_kernel's limit was shared memory: every wave re-read the 64 k / q floats of its row group per token
and the shuffles went through the same unit (probe: no k / q reads 9.8 ms, no shuffles 16.9 ms, of 21 ms). A lane
now carries two columns (8 lanes x 2 columns x 4 row groups = 16 columns a wave, 64 a block, 96 blocks, 384 waves
in one round at 230 VGPRs): the two columns share the k / q reads, and each wave has two independent dependency
chains. Per column the same arithmetic in the same order. One 16K chunk-layer 25.4 -> 16.9 ms (gdn_probe14.hip,
bitwise vs the old kernel); 64K prompt 1232.4 -> 1267.6 tok/s (-1.47 s, interleaved, 2 runs), ids identical at 8K
and 64K. STRATA_GDN_PP=1 is the one-column kernel with the register fixes only (-0.66 s).
gdn_rec_quad_kernel needed 196 VGPRs = 7 waves a SIMD, but its 192 blocks of 4 waves need 9.6: a second round of
blocks ran after the first (half the heads took exactly half the time). Cause: the 32 state-row addresses of the
first loads were CSE'd with the final stores and kept live across the whole walk (64 VGPRs); the stores' lane offset
now goes through an empty asm, so the addresses are made again - 133 VGPRs. A full chunk's four tokens also run
with the state alternating between two register sets (the update writes the other one), which drops the 32
register copies the compiler made after each update (131 VALU a token instead of ~178). The same arithmetic in
the same order. One 16K chunk-layer 25.4 -> 20.5 ms (gdn_probe12.hip, bitwise vs the old kernel); 64K prompt
1233.7 -> 1249.0 tok/s, -0.66 s; engine output ids identical at 8K and 64K.
STRATA_GDN_CONVL2: gdn_conv_tiled_kernel with gdn_l2_kernel folded in (a block's 128 threads are one head's channels):
the tile's conv loop verbatim, then the sum of squares spelled out as the compiler built the l2 kernel (first xor step
an fma on the partner's rounded square, then adds), the four partials in the same order, the same v * rsqrtf(ss+eps).
16K chunk, per layer: 9.13 ms -> 6.95 ms. A first version keeping the values in registers differed by 1 ulp in the
silu (different code shape); this one keeps the tiled loop as is.
STRATA_GDN_NOY: the norm kernel does not store the FP32 y nothing reads (403 MB per 16K chunk and layer); the FP16
value is made with v_fma_mixlo_f16, the instruction the compiler used next to the store (otherwise a mul and a
convert round twice and 1 ulp of the FP16 results differ).
Engine ids match the unchanged binary at 8K (acd9c8328408).
native_w11_kernel: the sign bytes of the IQ2 / IQ3 grids come from a shared-memory table of byte masks
((g ^ m) + c per 4 bytes; no zero byte in any grid) instead of the compare / subtract emulation; the decoded
weight stages are one flat shared buffer addressed through a pointer, which changes the code the compiler makes
for the down kernel (7.5 -> 6.2 ms per 128-expert launch, IQ3_S / IQ4_NL, 16384 tokens). One layer of 16384
tokens, fused experts: IQ2_S 83.8 -> 76.3 ms, IQ2_XXS 74.4 -> 68.9, IQ3_S 78.5 -> 73.8, IQ3_XXS 77.6 -> 71.8.
Outputs are bit for bit the same (the test now prints a hash of the fused output in pair order).
The formats carry a minimum, so the activation blocks' spare bytes now hold the int8 code sums per 32 values (written by
every path, read by none else). Layers MMQ does not cover on HIP (Q4_K) join the fused layout through MmqPlan::fo.
Opt-in (default output bit-identical). Test pairs plus an int8-rounded model check; QA_SAME probe in the attention harness.
With the image encoder on the GPU, the 700 MiB reserve can leave ~200 MiB free on a 12 GB card (UD-IQ4_XS
end-to-end check on an RTX 5070). Setup prints a one-line recommendation; the written config is unchanged.
- "experimental" is per model now: MODELS["UD-Q4_K_XL"] keeps it, the unsloth family and UD-IQ4_XS drop it. The
first menu lists the family without [experimental], with a one-line about (UD-IQ4_XS: a 94 GB download; under
~80 GB of RAM part of its experts are read from the SSD). Its size menu lists UD-IQ4_XS first (the default) and
UD-Q4_K_XL second; an experimental size always sorts last. --model names are unchanged.
- UD-IQ4_XS: no EXPERIMENTAL warning, --check verdict without it. Images are asked (off by default), per model: the
engine's image path has no restriction for this pack, which uses the original model's image encoder; not yet run
with images. UD-Q4_K_XL keeps images off.
- MCP: the fallback tables mirror setup's (per-model experimental/vision), sizes in setup's order, per-size
"experimental" and "images", the vision check per model.
- Docs: README and its six translations, MODELS, UNSLOTH_Q4, AI_SETUP, MCP_SERVER.
- Tests: the menus, the default, --model by name, images, --check, the pack/tokenizer paths; the golden configs
are unchanged.
- a prompt that fits 0.1.39's auto chunk keeps 0.1.39's ring (Prefill::set_ring_budget's small_max): 4K prompts
were 1-17% slower with the smaller ring (Coder -15%, IQ3_S -5%, IQ3_XXS -2%; 5 interleaved pairs each);
- ring slots are given up for a bigger chunk only where 0.1.39's rule held the chunk under 6144 (ring_cap_for);
from 6144 on the ring keeps its 0.1.39 size (the Coder 6144/384 -> 7936/199 was -12% at 32K);
- the unpinned arm's cap stays 96 (the byte budget made it ~49 on IQ3_S);
- the scan sets its own ring budget instead of set_ring_override, so a layer split's 96-slot override (#340) is no
longer clobbered by the scan.
RTX 5070 vs STRATA_RING_BYTES=0: 4K -0.7..+1.1% (bit-identical); 32K IQ3_XXS +18.5% (cache 1500) / +9.4%, Coder
+3.1%, IQ3_S +0.2%.
The padded activation of ssm_out / attn_output was handed to Gemm::native, whose padded-weight path needs a
strided dequant; the GSQ-RCO IQ3_XXS model has IQ3_S ssm_out tensors (no strided dequant for the i-quants), so
the engine exited at the first 64K chunk ("a padded X needs STRATA_PF_PAD's path"). Now such a weight is
dequantized unpadded and strata_pf_gemm_f16_ld reads X at its stride: the same kernel and products as with
PF_PAD off, the same bits.
Added where a config goes to several GPUs (setup --gpus / a multi-GPU install, start --gpus, "use both" at
start). Single-GPU configs are not touched. Opt-out: setup --no-remote-expert-opt, or "remote_expert_opt": false
in the config (a user key, kept on re-runs). The engine uses it only with a helper cache (--expert-cache-device1..3),
so a layer split runs as before. tools/test_setup_remote_opt.py; the 2x16 GB golden configs gain the flag.
The hyper-connection read ran two hipBLASLt BF16 GEMMs over the same T x 10240 xn16: down (N 320, 6.78 ms per
16K-token half, 15.8 TFLOPS - the 4 KB-multiple row stride) and inject (N 4, 1.81 ms). strata_pf_hcdown_bf16 does
both in one 128 x 128 BK-64 WMMA GEMM: weight rows [w_down; w_inject] read from the two tensors, the epilogue
splitting lo (T x 320) and inj (T x 4), so xn16 is read once; with the flag, xn16 is written with token stride
10240 + 64 (gr_norm_rs / gr_write_norm_rs / gr_write_cvec_norm_rs take an ldx; unchanged and bitwise when off).
s23/hcdown_probe.hip, 16K tokens: 3.60 / 3.77 ms for both (unpadded 5.75).
Opt-in, on chunks of STRATA_PF_SWITCH_MIN_T+ tokens. FP32 accumulation in another k order than hipBLASLt:
rounding-level. Aurora, c-best3 v3.2: 64K timing hc read 7234 / 7253 -> 6439 / 6175 ms; prompt 8K 1048 -> 1075 /
1036, 64K 1053 / 1056 -> 1067 / 1068 tok/s. Teacher-forced KL (8192-token prefix, 2001 positions) vs the default,
next to the accepted MMQ-vs-fused pair: offset 0 0.162 (mmq 0.158), 30000 0.110 (0.126), 70000 0.031 (0.033);
ledger {32,17,130}, 64K retrieval 3/3.
With the speed-projection control vector on, every steered layer's FFN write-back took three passes over the
T x 10240 FP32 residual (gr_write, cvec_apply, the next half's gr_norm_rs) instead of the one gr_write_norm_rs of
the other layers. gr_write_cvec_norm_rs does the three in one: gr_write's arithmetic, cvec_kernel's dot order and
two-level reduction, gr_norm_rs's sum order and block_sum - the same bits. STRATA_CVEC_FUSE_CHECK=N runs the
three-kernel path on a copy of R on the first N fused writes and compares R, the row scales and the BF16 image
(Aurora, 8K prompt, layers 4-11: all identical). cvec_tables() hands out the current device's vector tables.
c-best3 v3.1 on Aurora, one 64K prompt with STRATA_PREFILL_TIMING (x2): hc read + combine 13.0 / 13.1 ->
10.8 / 11.2 s, GPU timeline 64.6 / 64.9 -> 60.6 / 63.0 s; benches interleaved, same ids: 8K prompt
971.5 / 1019.9 -> 998.8 / 1045.1, 64K 1025.3 / 1012.8 -> 1039.2 / 1014.4 tok/s.
`--prefill auto` walked a fixed list of chunk sizes - 32768, 16384, 8192,
6144, ... - and took the first that fitted, with the ring a fixed 384 slots
taken off the top before the chunk was ever considered. Both halves of
that were wrong on a rig like this one.
The list is coarse exactly where it matters. `bytes_needed` is a sum of
(T x positive constant) terms plus a max of such sums, so it rises
monotonically with T, and so does every test the scan applies - which makes
the largest chunk that fits a bisection on the 256-token grid the prompt
path already works on. Seven probes against the list's ten, at one
`bytes_needed` each. It matters because this rig affords ~8,700 tokens and
was handed 8192; 8704 is not on the list.
The order was wrong because chunk and ring spend the same borrowed VRAM,
and a ring slot is worth far more than a chunk token. Measured here (2x
RTX 3060 + 2x RTX 5060, CUDA3 lending at its 90% cap, 120K prompt): 8960
tokens with the 17-slot ring that leaves reads at 963 tok/s, 8192/130 at
1,008, 7168 with the ring full at its 199-slot byte budget at 1,000, 5632/199
at 915. A ring slot is worth ~0.53 tok/s and a chunk token ~0.05, so the 69
slots between a full ring and 8192's 130 are worth more than the 512 chunk
tokens they cost - and once the ring IS full, a smaller chunk buys nothing.
So: the largest chunk that still leaves the ring full, and only a rig where
no chunk can afford one falls back to the old rule with a kRingMin floor.
The room is counted in BYTES, not slots. A ring slot is max_blob, but a
cache slot holds its own layer's blob - 2.15 MiB against a 2.54 MiB max_blob
here. In slots the ring looked 18% cheaper than it is, a chunk whose ring
fitted only after that discount was refused outright, and the scan stopped a
step short. `bytes_from_slots` is the inverse of the existing
`slots_from_bytes`, and `cache_slots_for` factors the latter out for the
serve path's per-participant version. Both inverses clamp at k <= 0: the
lend budget is `min(slots - 128, pct * slots / 100)` and goes negative on a
cache under 128 slots, where `slots - k` is off_[slots] - the end of the
prefix sum, so that cache lends 0 bytes rather than reading past its offsets.
Upstream's per-participant `fits`/`fits_one` and the #448 diagnostic are kept:
the ring room is taken over the smallest participant, `alone` is the chunk
CUDA0 alone would read in, and the small-card warning is unchanged.
Verified on the rig: the scan picks 7680/199 against the old rule's 8960/17,
1018.8 and 1021.1 tok/s on two cold 120K prompts against 963.1 - +5.9%.
Q8_0 picks 5632/101, its 506 MiB budget over a 5,222,400 B blob.
0.1.39b (Niko1221): behind STRATA_RING_BYTES=1 with the byte-budget ring; the default keeps 0.1.39's chunk list
(plan_lend and the serve scan), its log line gains the ring. The INFO fields (the Monitor commit) are not taken.
With global loads removed, the BK 32 double-buffered kernel still stopped at 37-39 TFLOPS of the ~50 WMMA peak:
the K loop itself (fragment reads, one barrier per 32 k) was the limiter. pfg::kernel_bk64 stages a 64-k tile in
one LDS buffer (55 KB) from 12 named registers with 2 barriers per tile, half the loop trips. Every accumulator
takes the same WMMAs in the same k order: the same bits (s23/gemm_probe11.hip, memcmp vs the BK 32 code).
gfx1151, M 16384, both operands padded (2 runs): N 2560 K 6144 38.0 -> 41.3 TFLOPS, N 12288 K 2560 38.9 -> 40.9,
N 10240 K 2560 40.3 -> 40.4. Also probed and rejected (same bits, slower): A rows read under the previous row's
WMMAs (36.7-37.9; 8-16 B spills), half-wave 16 B reads + v_permlanex16 (31.7-32.4), both (32.8-33.5).
Used for N >= 1024 and K % 64 == 0. c-best3 v3 on Aurora, BK 32 -> BK 64, interleaved, same ids: prompt
8K 959.8 / 947.8 -> 968.0 / 1001.9, 64K 1007.9 / 1004.0 -> 1028.0 / 1025.4 tok/s.
Default: 0.1.39's 384/1024-slot ring and bytes_needed count, so the default loan and output stay as they were.
The auto chunk scan (the PR's second commit) is not taken.
On a card whose experts mostly run on the CPU the slots cost speed per request (RTX 5070, Q2_0: 11-24 %), so setup
now recommends slots only when the expert cache (every card of a split counted) still holds half of the model's
experts beside them (Q2_0: 24 GB and up, or a split); elsewhere it stays at one at a time and says "parallel N
reduces waiting for several users but costs about 10-25% speed per request on this card". --parallel N is
honoured as asked, as before. BATCHING.md: the table with the change against one at a time.
The ring was 384 slots - a number measured on Q2_0, where a slot is one
expert blob of 1,382,400 B, so 506 MiB. A slot is a whole blob, so on a
pack with bigger blobs the same 384 slots are a different amount of
memory: 1,912 MiB on Q8_0, more than half of an 8 GB card's expert cache,
and what kept that card at a 1024-token chunk when its own buffers would
have fitted 6144. Measured on the 4-way rig, one env var and nothing
else: chunk 1024 -> 6144, prefill 87 -> 402 tok/s.
So the budget is the constant and the slot count is derived from the
pack's max blob: 384 on Q2_0 (pinned) or 96 (not), exactly as before, and
the pack it was tuned on does not move. The fused rule (1024 slots, #136)
is preserved the same way, and `ring_cap()` still bounds the result. This
is also what lets the auto chunk scan treat "the ring is full" as a plain
byte comparison.
Also exports the three accessors the scan needs: bytes_needed_no_ring
(the chunk's own buffers, without the ring), ring_max_slots (the cap in
slots) and ring_slots_for (what the ring resolves to for a chunk, after
the override, the env var and the pinned-share rule - what `init` lays out
and what the INFO line reports).
And one fix on the way: `bytes_needed` counted `carve`'s buffers from
memory and got two wrong - it took `xn` unconditionally, though `carve`
takes it only under STRATA_GR_UNFUSED, and it never took `grs`. Net
T*(D-HC)*4 bytes too many: 42 MB at a 1024-token chunk. The direction is
safe - the prompt path was told it had less room than it did - but it
under-sizes every loan.
A 4 KB-multiple row stride (K 2048 / 4096 / 6144 halves) camps on the memory channels: the gfx1151 WMMA prompt
GEMM ran ssm_out (N 2560, K 6144) at 19 TFLOPS against 31-32 for K 2560. Padding the rows by 64 halves (gemm_probe8,
M 16384): K 6144 19.1 -> 35.0 TFLOPS (W alone 29.0, X alone 23.7), K 2560 31.7 -> 36.6.
With STRATA_PF_GEMM=1 on chunks of STRATA_PF_SWITCH_MIN_T+ tokens: Gemm::native dequantizes the weight into the
scratch with stride K + 64 (dequant_f16_ld, non-i-quant types) and runs strata_pf_gemm_f16_ld; the FP16
activations of the two K 6144 projections (gdn_recurrence's y16 for ssm_out, gate_attn's out16 for attn_output)
are written with stride 6144 + 64. The same products in the same order: the same bits.
c-best3 v2 + STRATA_PF_PAD=1 (Aurora, interleaved, same generated ids): prompt 8K 954.2 / 930.7 -> 983.7 / 940.8,
64K 947.0 / 946.2 -> 974.8 / 1014.4 tok/s.
Per MoE layer of a verify window (2-8 rows, resident combine path):
- the shared expert's scalar gate (a 1-row BF16 GEMV, one latency-bound
workgroup) rides as one more block of the router GEMV launch
(bf16_gemv_fp32_mmvf_multi_aux; the rows kernel's per-output code is the
multi kernel's - checked bitwise for 2-8 rows, with and without TSUM)
- its sigmoid + row scale move into the combine (combine<true, true>: the same
sigmoid expression, the product rounded on its own with contraction off in
that block only, then added as before)
- shared gate and up projections in one launch (native_mmvq_pair: blocks
n_out.. take the second matrix; Q8_0, exact multi layout, not packed)
Per attention layer: the window's int8 KV appends in one launch
(kv_append_q8_steps, blockIdx.z = token x K/V; distinct cells).
-4 launches per MoE layer, -(n-1) per attention layer; STRATA_LFUSE_GATE=0 /
STRATA_LFUSE_PAIR=0 turn the gate / pair parts off.
Engine (gfx1151, UD-IQ4_XS, c-best3 + EXPERT_V2 + Q8_PACKED + TSUM, S25's 18
prompts, same binary): 51.29 -> 50.31 ms per verify window, 54.33 -> 55.12
tok/s; 59K prompts 54.19 -> 53.17 ms, 42.97 -> 43.79 tok/s; all outputs identical.
server: a request decoding in a slot whose neighbours finished (no other slot busy, nobody waiting, at least 32
tokens allowed) is BSTOPped and continued on the solo path with its MTP drafts - the engine copies the slot's
sessions back (INFO slot_cache=1; at most twice per request; STRATA_PARALLEL_SOLO=0 keeps it in the slot).
engine: INFO batch_slots=N slot_cache=0|1.
tools/batch_interleave_test.py: the back-to-solo path, and a next turn from a slot's turn checkpoint (a client
that drops the reply's thinking).
docs: "parallel": N, what a slot costs and what setup recommends, the server's handling (interleaved prompt reads,
BYIELD, slot conversation cache, back to solo), exactness and its settings, the RTX 5070 measurements (the slots
buy waiting time, not speed, on a 12 GB card), the protocol; DETAILS / README / AI_SETUP mention the option.
INSTALL.md gets the section (the build setup compiles, what differs, the forced-path numbers from a
Ryzen 5 7600 + RTX 5070, what is untested); the CPU rows of INSTALL.md / DETAILS.md and AI_SETUP.md
point to it.
STRATA_HC_UPMIX, STRATA_PA_FAST and STRATA_PF_GEMM change the prompt path's rounding; on short prompts their time
gain is negligible but they change the generated text. With N set, chunks shorter than N keep the default
arithmetic (and outputs) while long prompts keep the gain. Unset or 0: every chunk, as before.
gfx1151, c-best3 + STRATA_PF_SWITCH_MIN_T=4096: S25's 18 prompts (1.0-1.9K tokens) give outputs identical to
c-best2 v2 on 18 of 18 (49.28 tok/s; c-best3 without the gate 48.83 / 48.67, other texts).
batch_test.py: the engine log in the temp folder, the CUDA DLLs on PATH, ASCII output, --mt-min (empty: the
engine's default kernels, for the invariance check without the exact settings). server: /metrics live.waiting
and live.outside_slots instead of a guessed count.
Windows: pool workers pin with SetThreadGroupAffinity (the processor group was dropped: a 3990X's 63 workers
collapsed to 32 group-local masks); the host thread selects a CPU Set and restores the previous selection.
Linux: the complete, dynamically sized affinity mask is kept and restored (CPU IDs past 63).
pool_affinity_test (Windows) and pool_affinity_linux_test. Squashed from the PR's 8 commits (net diff).
The slots' sessions may take a fifth of the expert cache (~VRAM less 5 GB), up to 4: 2 on 12 GB, 3 on 16 GB, 4
from 24 GB, none below. Without --parallel setup only says it once; with it the config gets "parallel": N and a
count above the recommendation is kept with a note.
- "parallel": N in the run config passes --batch N (a value the engine cannot use is said and passed on: the
engine warns and adjusts); the server uses the slot count the engine reports (INFO batch_slots)
- every running request keeps its own status, rate window and engine figures (history rows are their own);
/metrics live.slots shows each slot (idle with the tokens it holds, reading, decoding with tok/s),
/v1/status concurrency.serving says N
- a conversation's next turn goes to the slot that holds its start, a new one to an empty or the least recently
used slot; a slot left at a stop token keeps what it holds
- a long prompt read (solo or an admission) gives way (BYIELD) to a waiting request with a prompt under half as
long, at most twice; the part read waits in a reserved slot and the request goes on once the other is in
- the engine start and POST /v1/vram are serialized against parallel requests
- serve/test_parallel.py: a scripted engine behind the real StrataEngine and Service
- --slots N is the same as --batch N; a count the engine cannot run (no --serve, 1, more than 8, a split on one
GPU, sessions that do not fit, --batch-groups that do not divide) is a warning and fewer slots or none, never a
refusal; INFO says batch_slots=N when slots run
- a batch window runs over the active slots only (Verifier::run_slot_rows, graphs per row list): an idle slot is
not written, so it keeps the conversation it served
- a request whose prompt starts with what an idle slot holds (all of it, or its turn checkpoint) continues from
the slot's sessions instead of reading the history again
- with slots, a prompt is read one prompt chunk at a time (the same chunks as one run) and the slots decode
between two chunks for half as long as the chunk took (STRATA_BATCH_DECODE_SHARE)
- BYIELD <slot>: the prompt being read stops at its next chunk boundary, the part read is copied into the slot,
YIELDED <slot> <tokens> + DONE cancel; the same request sent again goes on from there with the same chunks
and checkpoints (#656's cooperative preemption, a slot as the park)
Default (no --batch) unchanged.
PR #638 was cut from 0.1.38. 0.1.39's segmented expert cache (#533) keeps its cuMem* code out of HIP builds through
STRATA_USE_HIP, which the gfx906 compat build does not define, so it included <cuda.h>. Its virtual-memory blocks are
now behind STRATA_EC_NO_VMM (STRATA_USE_HIP or STRATA_HIP_GFX906); the CUDA and RDNA preprocessor output is unchanged.
(std::max) keeps windows.h's max macro out of two Windows-only lines (found by a Windows gfx906 compile check).
s26_down_l_kernel (STRATA_EXPERT_V2): a warp's 4 rows have 40 (block, half)
items each; lane-strided per row, lanes 0-7 took 2 items of every row (8
loads) and lanes 8-31 one (4). Under STRATA_TSUM=1 the 160 items are spread
5 per lane (item p = 32 i + lane is row p / 40), all 5 loads issued before the
dots. A lane's items of a row are then exactly those of the per-row layout's
lane (l - 40 q) mod 32, in the same order, and an xor butterfly gives the same
bits under any rotation of the lanes (each level's partner differs by +-2^j,
which is +2^j modulo 2^(j+1)): every output is bitwise the same (ws3 harness,
memcmp, T 1-3: 1.09 / 1.11 / 1.12x for gate/up + down with TSUM, vs 1.03 / 1.05
/ 1.06x with the butterfly alone). The same spread for gate/up (80 items x 4
rows, 10 per lane in 1-3 load groups) was 0.6-0.8x and is not used.
A warp that ends with P independent warp sums (P tokens of a row, or rows x
tokens) used P xor butterflies of 5 shuffles each. s26_tsum.cuh does them in
one transposed butterfly: at offset 16 a lane keeps half of its values and adds
the partner's copy of that half, at 8 half again, ... then plain xor steps.
Every level adds the same pair of values the butterfly adds (fp add commutes),
so each sum is bitwise the same; P - 1 + (5 - log2 P) moves instead of 5 P, as
DPP row_xmask / permlanex16 on AMD (__shfl_xor_sync elsewhere).
Applied under STRATA_TSUM=1 to the decode kernels that end in several sums:
- HC read (STRATA_HC_Q8): gr_down_q8 / gr_up_q8 (ws4 harness, every output
equal, T 1-4: 1.00 / 1.03 / 1.03 / 1.15x per read)
- STRATA_EXPERT_V2 gate/up and down (ws3 harness, memcmp, T 1-3:
1.03 / 1.05 / 1.06x)
- native_mmvq_multi_kernel (exact layout) and the Q8_0 / Q6_K packed kernels
- bf16_f32_mmvf_rows_kernel (STRATA_MMVF_ROWS), both reduction stages
Engine (gfx1151, UD-IQ4_XS, c-best3 + EXPERT_V2 + Q8_PACKED, S25's 18
prompts): 54.21 -> 52.10 ms per verify window, 51.31 -> 53.73 tok/s, all 18
outputs identical.
- layer split: commit stays synchronous on a split (set_commit_async(!multi_gpu), the `|| next_ != nullptr`),
no pre-staging of the next stage's inputs, kDrafterMib 1000 - the split's sizing and order as in 0.1.39
(untested here: one GPU);
- the CPU expert kernels keep cpu_avx512_ok() (F/BW/VL/VNNI/VBMI): #646's AVX-512BW-only check would send a
CPU without VNNI/VBMI to kernels that use them;
- resident_plan_kernel: one thread per window entry up to 128 (64 lost a 7- or 8-token window's last entries
on the all-resident path);
- the host keeps zeroing the GPU's rows under STRATA_VERIFY_DEVICE_PLAN=1 (copy_or_zero_from_mapped copies
every row there).
dpct left the DPCT1106 note but dropped cudaMemGetInfo in generate.cpp's
stage_room, so every later stage of --layer-split sized its expert cache from
0 bytes free and the second card crashed with 'expert cache: no room'.
Reported with the fix by tmking01 in #423 (2x Arc Pro B70); also a fixups.py
entry so a re-migration keeps it.
The SYCL port of PR #423 tracked 0.1.38; 13 of its migrated copies were behind
0.1.39's sources. Re-migrated 0.1.38 and 0.1.39 with sycl/tools/migrate.sh,
normalized, merged with sycl/tools/merge_upstream.py:
- clean merges: expert_cache, expert_source, native_dense, pinned, elementwise,
kv_stream, iq_multi_parity, prefill, verify.hpp; q8_1_finite.hpp new (#606)
- verify.cpp: the #649 trace kept with the port's queue handling
(STRATA_VERIFY_TRACE; the HIP-only coherent/doorbell switches stay HIP-only)
- generate.cpp: the port's --prefill-borrow beside --vram-elastic/--vram-segment-mib
- iq_kernels / native_mmvq: the port's kernels kept, #606's finite q8_1 scale
and sum applied by hand (dpct's conflicts there were kernel-name hashes)
- expert_cache: --vram-elastic (#533) is CUDA-only on the port too (no driver
VMM on Level Zero through dpct); it reports so instead of failing to build
- version 0.1.39
Compiled here with icpx 2026.1.1 (SPIR-V, no Intel GPU on this PC): strata
builds. Not run on an Arc.
docs/OLDER_GPUS.md: Pascal / Volta through the CUDA 12 engine (#395#600#540#655), RTX 20's STRATA_BF16_TC opt-in,
AMD gfx1031 (#524), gfx1012 (#442) and gfx906 (#638, #677) with the reporters' numbers; README, AI_SETUP, AMD_HIP and
NVIDIA_V100 point to it.
Random BF16 rows of the router's shape [512 x 2560], 1..8 tokens: both within 3.3e-7 of the double
sums on a Ryzen 5 7600 (0.53 ms and 0.50 ms per layer for 4 tokens). Two ctest entries, the second with
STRATA_FORCE_ISA=avx (the AVX-only CPU's path). Test program only.
CUDA 13 dropped Pascal and Volta. A model whose GPUs include one (its oldest card decides: one engine runs per model)
now runs the experimental CUDA 12 engine in engine-cuda12/ - the ready-made strata-windows-x64-cuda12.zip with NVIDIA's
CUDA 12 pip libraries, or compiled here with a CUDA 12.x toolkit (Linux; STRATA_NVCC picks one, #601) - and setup says
why. engine/ and every model on it stay as they were.
- the cards are an opt-in: naming one (--gpu / --gpus), --cuda 12, STRATA_EXPERIMENTAL_SM60=1, or a PC with no newer
NVIDIA card and no supported AMD card; a PC with a newer card keeps recommending it
- --cuda 12|13 (or STRATA_CUDA) overrides the choice and is kept, with a warning where it cannot work (recommend,
never force); --cuda 12 also runs with a driver older than 580 (525+), which the stop now mentions
- at a start, a Pascal / Volta card added to a CUDA 13 model moves that model's config to the CUDA 12 engine
- updates look at both engine folders; a compiled CUDA 12 engine builds in build-cuda12/
- an RTX 50 card in a CUDA 12 engine is a warning (CUDA 12.8+ builds it; #220 crashed with 12.8), no longer a stop
The setup golden test (every recommended config on five PCs) is unchanged; tools/test_setup_older_gpus.py covers the rest.
- CMake: STRATA_ISA_FLOOR (empty by default) builds ggml-cpu for AVX (Sandy/Ivy Bridge) or SSE4.2
(Nehalem/Westmere) instead of the build host or AVX2; every other ggml ISA option off. Strata's own
AVX2/AVX-512 kernels stay in the binary behind their CPU checks. Excludes STRATA_PORTABLE. The ggml
block follows the Strata_Dirigo fork's (rwkeyes).
- STRATA_FORCE_ISA=avx2|avx|sse caps the engine's CPU probes, so the older-CPU paths run on any PC
(how this was tested here: no old CPU). cpu_sse42_ok(), isa_floor_build().
- The engine: a normal build still refuses a CPU without AVX2, now pointing at the experimental build;
a floor build checks its own floor (AVX, or SSE4.2 + POPCNT) and warns that it is experimental and
slow (or, on an AVX2 CPU, that the normal build is faster).
- setup: a CPU without AVX2 warns ("experimental, slow") and compiles the engine on the PC with the
floor it has, instead of failing (#595's idea); the prebuilt AVX2 engine is not used there. Recorded
in BUILD.json; a build folder for another floor is configured afresh. STRATA_ISA_FLOOR=avx|none asks
for it on any PC. Windows + AMD has no source build: a clear stop. The MCP server's recommendation
notes it instead of refusing.
- tools/test_setup_oldcpu.py.
Default build: nothing changes (no new CMake definitions, the probes answer as before without the env).
The two IQ3 gate/up formats decode each 32-value half with eight scalar loads
assembled by _mm256_set_epi32. A scratch build that substitutes a constant for
the grid vector shows that assembly is ~half the single-core kernel (IQ3_XXS
248 -> 130 us, IQ3_S 357 -> 140 us), and the two formats are ~58% of the
gate/up lookup work on the Swift 1.5 IQ3_XXS pack.
Add an opt-in AVX2 gather for IQ3_XXS (iq3xxs_grid) and IQ3_S (iq3s_grid),
the mirror of the AVX-512 kernels' STRATA_IQ_GATHER. On the i7-12850HX
(Alder Lake, no AVX-512) it is bit-identical - a direct scalar-vs-gather output
dump matches for all five gate/up formats, native_expert_parity stays green -
and 1.32-1.34x on the two formats isolated, +1.0% serve end to end (on >= off in
5 of 5 interleaved pairs). The pool is flat in the deterministic --stats (the
15-worker pool is not decode-bound), so it is a per-core win that compounds.
Default off, like the AVX-512 one: AVX2 gather throughput is microarchitecture-
dependent. IQ2_XXS / IQ2_XS / IQ2_S use 64-bit grids, where an AVX2 i64 gather
measured a wash, and are unchanged. The AVX-512 path is untouched.
Without a hugetlb pool the Linux arena fell back to a plain anonymous
mapping with 4 KB pages. The CPU expert pool streams whole experts (about
3 MB each for the 4-5 bit GGUFs) out of that arena, so every expert costs
~750 TLB misses.
Map the fallback 2 MiB-aligned and madvise(MADV_HUGEPAGE), so THP in
"madvise" mode (the common distro default) backs it. STRATA_NO_LARGEPAGES=1
keeps 4 KB pages, as on the hugetlb path; the log note says which one it got.
A prompt that fits one chunk runs a layer split's stages one after the other,
so while one card reads its layers the other idles. --peer-device already has
the mechanism for this (PeerPrefill's peer streaming: a share of the experts
the primary streams goes over the peer's own PCIe link and is computed there),
but it needs P2P and is refused with --layer-split.
This makes PeerPrefill usable as a stream-only peer between the stages:
- set_stage_helper: each stage's prompt path gets the next stage's (the last
one the first's); a stream-only PeerPrefill holds no experts (peer == null),
so every use of the peer tier is guarded
- per prompt, bind_stage_helper points it at the idle stage's own lent prompt
buffers (mixed, Xq, GU, H, Hq, Dm, ring, MMQ context): no VRAM of its own
- without P2P the activations, the row table, the group bounds and the result
rows go through mapped host memory, read by copy kernels on the device that
needs them. Copy-engine uploads queued behind the ring's expert blobs (the
grouping tables' reason for mapped memory), which cost ~20 ms per layer and
ate the whole gain; cudaMemcpyPeerAsync without P2P is staged by the driver
- only for one-chunk prompts on the MMQ path (not the fused kernels, not with
--peer-device), with a share that falls with the chunk: min(0.45,
0.5 - T/16384), off below 0.3 (~3.3K tokens) - measured, below
- STRATA_PREFILL_HELP=0 turns it off, STRATA_PREFILL_HELP_FRAC fixes the share
2x RTX 3090 (no P2P), UD-Q4_K_XL, layer split auto, 262K context,
--adapt-swaps 0, prompt tok/s without / with: 1.1K 496 / 723, 1.5K 674 / 833,
2K 878 / 1128, 2.5K 1017 / 1296, 3K 1260 / 1440, 4K 1605 / 1607, 8K 2215 /
2221; decode unchanged. The output is repeatable (same prompt, same answer)
but not bit-identical to STRATA_PREFILL_HELP=0's: the helped rows are
computed in another MMQ grouping.
0.1.39b (Niko1221): opt-in (STRATA_PREFILL_HELP=1), since the helped rows round differently from the default;
merged with #598 (run_impl / next_run_).
Add default-off cache coordination, helper-side weighted reduction and CPU activation quantization on demand. Allow helper cache capacities to be sized automatically at startup.
Port the saved v2.2 implementation onto upstream 0.1.38 without changing the separate peer tier. Document the helper-only scope and current validation limits.
The Coder's router is 256 experts, which the native 512 x 10 router does not take, so a verify window routed each
token with its own projection and top-k. moe_route_window does the window in one multi-column BF16 projection
(bf16_gemv_fp32_mmvf_cols over the existing multi-row kernel) and one n-token top-k; each token's route is bitwise
what moe_route gives it (route_window_parity, new ctest). gfx906 build only, so the CUDA and RDNA paths are
unchanged; STRATA_ROUTE_PER_TOKEN=1 restores the per-token calls.
2x MI50, Coder IQ1_M, 32K, greedy 1,500-token request: verify window 55.5 -> 52.5 ms, generated text identical.
- native_expert_grouped: the AMD layouts handle the IQ packs' gate/up and down types only; Unsloth's Q4_K / Q5_K
gate/up and Q5_1 / Q8_0 down now take the CUDA layout instead of exiting (native_grouped_parity found it).
- fused_gr_read_multi: the K-split gr_down (fixed-order finish) sums in another order than the single-token
fused_gr_read, so it broke gr_parity's multi == single contract; it is now opt-in (STRATA_GR_SPLIT=1). The
latency-hidden norm/up (same sums, same order) run in the default read (STRATA_GR_FAST=0: off).
- mmvq_multi_parity: on gfx906 the wave64 MMVQ is the same exact kernel for every column count, so the negative
control has nothing to find; a second ctest entry runs it on the CUDA layout (STRATA_MMVQ_WAVE=0), where it has
power.
- kv_hybrid_parity: the tensor-core prompt attention refuses on gfx906 as on the RDNA backend.
ctest on 2x MI50 (gfx906): 34 of 34 pass (iq_parity and ple_parity need fixture files and are not counted).
The HIP backend (STRATA_ENABLE_HIP) is wave32 RDNA only, and the engine refuses a wave64 card. This adds a
separate opt-in build for gfx906: the CUDA sources compiled as HIP through a small compat layer, with a CUDA warp
as a logical half of the 64-lane wavefront. With the option off nothing changes for CUDA or the RDNA backend;
every new code path is under STRATA_HIP_GFX906.
Build: -DSTRATA_HIP_GFX906=ON -DCMAKE_HIP_ARCHITECTURES=gfx906 (implies STRATA_ENABLE_CUDA's targets, refuses
STRATA_ENABLE_HIP). The CUDA runtime/cuBLAS names come from include/strata/platform/hip_compat/ (first on the
include path), CUDA::cudart / CUDA::cublas become aliases of hip::host / roc::hipblas, every .cu (and the ggml MMQ
sources, with GGML_USE_HIP) is compiled as HIP. The native kernels' --use_fast_math (an nvcc flag) is not passed.
Correctness fixes found on the way (each one changed results or hung, not only speed):
- __fmul_rn / __fadd_rn / __fsub_rn contracted into FMA under HIP; they no longer do (as on NVIDIA) - the
elementwise and quantize_act parity tests pass bitwise;
- the device check of the gfx906 build reads the card's gcnArchName (the wave32 check is the RDNA backend's);
- ggml MMQ got NVIDIA compute capability 9.0 for gfx906 and hung the prompt path; it gets the AMD arch code;
- __byte_perm is a scratch-memory array in HIP: the IQ4 table lookup uses llama.cpp's v_perm_b32 sequence
(agent-request decode 29.9 -> 39.2 tok/s on 2x MI50);
- the stage profiler's wall clock runs at 25 MHz on gfx906, not 100 MHz;
- a CUDA toolkit-mismatch check that reads CUDART_VERSION is skipped (no CUDART_VERSION under HIP);
- gdn_rec_parity (cp.async, CUDA occupancy API) is not built for gfx906.
wave64 kernels (gfx906 only; the CUDA layout stays one env switch away for A/B, every one bitwise-checked):
- grouped native experts, STRATA_EXP_MODE=7 (default): gfx906 has no packed byte SIMD, so __vcmpne4/__vsub4
became ~12 word ops each; signs are now dp4a(g ^ m, u) - dp4a(m, u) (exact in integers), grids and the group's
activations sit in LDS, one weight load serves every token of the group. Same k-to-lane order, so the output is
bitwise the old layout's (native_expert_bench, real GGUF rows). Verify window 60.0 -> 51.3 ms.
STRATA_EXP_MODE=2 is the previous layout;
- MMVQ: one wavefront per row with a 64-lane butterfly, no LDS/barrier, same kernel for 1..8 columns
(+2.3% decode; STRATA_MMVQ_WAVE=0: CUDA layout);
- router top-10 in one wavefront (register argmax, no block barriers), bit-identical (router_top10_parity);
- gr_down_multi split along K (8 rows x 5 slices of 2048 per block, fixed-order finish) (+2.9%;
STRATA_GR_SPLIT=0: old);
- hyper-connection read: norm with its loads in flight and xn in registers, up with 8 lanes per row
(51.3 -> 50.2 ms per window; STRATA_GR_FAST=0: old);
- the QSA prompt-attention tensor-core/WMMA kernels are not used on gfx906 (the FP32 kernel runs).
Benches (not ctest; they need a model shard): native_expert_bench (one expert layout against another, bitwise,
with timings) and fused_gr_bench (the fast hyper-connection kernels against the old ones).
The routing-aware prefetch's router dot (bf16_rows_dot_multi) lives in kq_avx2.cpp, so an AVX-only
CPU started fine and died on the first request (SIGILL in vpmovzxwd on a Xeon E5-2687W). It now
takes the AVX2 kernel where the CPU has AVX2, kq_avx1.cpp's AVX dot (mul + add, 128-bit integer
widening) where it has AVX, and the same sums in plain C++ below that. cpu_avx1_ok() probes AVX +
OSXSAVE + the YMM state. An estimate only (which experts to prefetch). A CPU with AVX2 runs exactly
what it ran before.
From rwkeyes' Strata_Dirigo fork (MIT; 0d00a66, e0754d3), with MSVC's /arch:AVX added and
__builtin_memcpy replaced by std::memcpy for MSVC.
#540's attention kernel (bit-exact, +7% prompt on a V100) uses about twice the registers and also runs decode, so it
is not taken for every card: the kernel the ready-made engine runs is restored as it was, and #540's version lives
beside it as attn_chunk_kernel_pre75, compiled only with -DSTRATA_EXPERIMENTAL_SM60=ON and launched on a card below
sm_75 (per device; STRATA_ATTN_PRE75=0 is the A/B arm). The launches pass cap/stride explicitly (the same 0, 0 as
the defaults) so either kernel can be picked through one pointer.
Conflicts with 0.1.39: --trim-stage-weights is now the same switch as STRATA_STAGE_TRIM=1 (PR #639's
implementation, already in 0.1.39); the #533 VRAM command is taken between batch windows only when no slot
is decoding.
STRATA_EXPERIMENTAL_SM60 now reaches every target (add_compile_definitions). prompt_attn_v70_kernel with its launcher,
and Gemm::bf16's fp32 path for Pascal, exist only under it: the ready-made engine (sm_75+) loads no extra kernels for
them, while the experimental build's sm_60/61/70 cards get them.
A native pack's Q2_0 layers (most IQ packs' down projections) ran on Strata's own Q2_0 kernels, which
are AVX2 at least. q2_native_kernels() keeps them there when the CPU has AVX2 and otherwise hands the
layer to ggml-cpu's Q2_0 vec_dot like every other native type (the pool, the dispatcher's activation
quantizer and the parity test). STRATA_KQ256's gate/up kernel needs AVX2 too. native_expert_parity
skips the AVX2 checks on a CPU without AVX2. No change on a CPU with AVX2.
(The engine part of #595; its setup and run-script parts are not in this commit.)
Old path against new kernel on 8.5K / 33K / 99K-token chats, 617 scored positions each, with two controls that change only the rounding
order (BF16 GEMMs without the FP16 detour, another prefill chunk), reruns of both paths (bit-identical), and the kernel's own error on the
model's data (relative 4.4e-7 mean, 4.8e-6 worst row over ~5 M rows).
The experimental sm_70 build (STRATA_EXPERIMENTAL_SM60, #236) had no page. NVIDIA_V100.md says how to build it, which kernel each
part of the prompt path uses on Volta and what the prompt attention kernel was measured at.
bench/results/2026-10-03-v100-prompt-attn: prompt speed with and without the kernel (7K / 28K / 114K tokens), the nsys share of the
attention before and after, the parity numbers, needle recall (15 of 15 with either path) and the greedy-output comparison.
On sm_70 qsa_prompt_attn_batch refused the device and the prompt path fell
back to the decode kernel (FP32 FMAs, one query at a time): 19-25% of the GPU
time of a 32K prompt on a V100.
prompt_attn_v70_kernel is the v1 kernel's structure with the two matrix
products on m8n8k4 (FP16 in, FP32 accumulate): q.k with one dim group (64 dims,
= one int8 scale group) per warp and 8 cells per quad-pair, p.v with 16 dims per
quad-pair. q and p keep their hi + lo FP16 split, the four dim groups' partials
are added in a fixed order (deterministic). Int8, FP16 and K8V4 KV are handled;
Q4_0 KV (mode 4, sm_80+ only upstream) keeps the old kernel. The fragment layout
was measured on a V100 and matches turbomind's SM70_MMA_884.
qsa_prompt_attn_parity (V100): error against FP64 ~5e-6 (output scale ~3.6),
int8 3.0x and fp16 1.3x faster than the FP32 kernel per 2048-query chunk. The
test now skips its Q4_0 cases below sm_80 instead of exiting 2 there.
End to end, 32K prompt, one V100, greedy output identical:
26 s -> 22 s on current main (STRATA_PROMPT_ATTN_OLD=1 is the old path).
PR #655's FP16 tensor-core path for Gemm::bf16, adapted so the ready-made engine stays byte-identical:
- default on for compute capability 7.0-7.4 only (Volta; the experimental STRATA_EXPERIMENTAL_SM60 build), off for
7.5 (RTX 20): STRATA_BF16_TC=1 opts in, =0 turns it off on any 7.x card
- from #540: finite values beyond FP16's range clamp to +-65504; a beta = 1 product (the BF16X2 remainder, in
FP16's subnormal band) keeps cuBLAS; the compute capability is cached per device (a layer split can mix cards)
- from #395: Pascal (6.x; cuBLAS has no BF16 GEMM there) widens both operands to fp32 by an exact shift and runs
cublasSgemm with the same fp32 accumulator, tiled over T in the instance's own buffers
sm_80 and newer, and HIP, run the same cuBLAS / hipBLAS call as before.
Co-Authored-By: Károly Sárdi <[email protected]>
Co-Authored-By: Rafal R@V Prasal <[email protected]>
native_gu_rows() tested cpu_avx512_ok() before the AVX-512 kernel but not cpu_avx2_ok() before the
AVX2 one, and native_down_rows() had the same asymmetry for its two multi-token kernels (kq256 and
the IQ4_NL one). On a CPU without AVX2 a verify window of two or more tokens took an /arch:AVX2 path
and died with 0xC000001D. With the gates it falls through to ggml-cpu's vec_dot, which covers every
native type for any nt. A CPU with AVX2 takes exactly the paths it took before.
(Cherry-picked from #394's 0083bb7 without its generate.cpp part; the startup gate is reworked for the
opt-in STRATA_ISA_FLOOR build in a later commit.)
#391 removed MSVC's BMI2 shlx from the table's constructor; GCC still vectorised the same loop into
AVX2 (vpbroadcastb) and ran it before main() on an AVX-only Xeon E5 (found by rwkeyes' Strata_Dirigo
fork, 6dd37a2, which made it constexpr). constexpr here as well, deriving ksigns_iq2xs[i] (the 7 bits
plus an even-parity bit) so the constant expression does not read a non-constexpr array. Same 128
values; the file now has no runtime initializer.
A namespace-scope static initializer shifted a byte into place with << (8 * k), which is a
variable shift. MSVC compiling this file with /arch:AVX2 emits BMI2 shlx for that - it assumes
anything with AVX2 also has BMI2, true from Haswell on and false of everything before it.
Because the table is static, the constructor runs before main(), so this faulted at PROCESS START:
strata.exe exited 0xC000001D with no output at all, not even for --help. One shlx in the whole
linked binary was enough to make the engine unstartable on any pre-Haswell CPU under MSVC.
Storing the byte directly is the same 64-bit value on the little-endian targets this file is for,
and leaves no variable shift for the compiler to widen.
Found by building the engine for sm_61 on a Sandy Bridge-E (Xeon E5-2680) with
STRATA_EXPERIMENTAL_SM60=ON, which 0.1.31 added (#236). Verified by disassembly: no shlx/shrx/sarx/
mulx/andn/bzhi/pext/pdep anywhere in strata.exe afterwards, and --help now runs.
This is independent of any AVX1 work - it is a plain startup crash on the CPU this was built on.
(cherry picked from commit 3fcaf03840)
Volta has FP16 tensor cores and no BF16 ones as well, so cuBLAS takes the same SIMT path for BF16 products there.
Suggested in review. Not measured on sm_70; STRATA_BF16_TC=0 turns it off.
Turing has no BF16 tensor cores, so cublasGemmEx on CUDA_R_16BF inputs falls back to a SIMT fp32 kernel
(magma_sgemmEx). On an RTX 2080 Ti that was the largest kernel of the prompt path, 20 % of a 4K prompt:
the hyper-connection down / up projections, the router and indexer rows, PLE key / value.
Gemm::bf16 now converts the weight and the activations (in row slices, into the instance's own scratch)
BF16 -> FP16, which is exact for every value inside FP16's range, and runs the existing FP16 GEMM (fp32
accumulate) on the tensor cores. Single-row products keep cuBLAS's GEMV. STRATA_BF16_TC=0/1 overrides;
the default is on for sm_75 only, so other cards run exactly what they ran.
New test gemm_bf16_parity: Gemm::bf16 against cuBLAS's BF16 product over the prompt path's shapes,
within fp32 rounding (worst relative difference 1.7e-5 on sm_75; 0 with the path off).
Qwen3.8-Flash-Next Q2_0, RTX 2080 Ti, i7-6700: prefill 877 / 1134 / 1153 / 1116 / 1009 t/s at
4K / 16K / 32K / 64K / 128K -> 1020 / 1337 / 1366 / 1312 / 1166.
It hung on the B70 until the stager/PLE event-wait fixes; it runs since and reads
Coder prompts faster (40K 1,106 -> 1,160, 128K 897 -> 920, 128K k8v4 1,006 ->
1,041, 256K 760 -> 780 tok/s; same output). It copies every non-resident expert,
so with a big host tier it loses (IQ2_XS, a quarter in the RAM mirror: 2,184
tokens ~560 -> 254 tok/s): generate.cpp hands the non-resident share to the
prompt path, which walks only below 10%. STRATA_PREFILL_STREAM_ALL=1/0 forces it.
Docs: the 40K prompt breakdown, and why the cluster kernels and Q4_0 XMX prompt
attention are not worth porting here; the 2K "regression" was session variance
(old and new builds A/B'd: 784-799 tok/s both).
Tested on the B70: Coder 2,184-token and 40K, IQ2_XS 19- and 2,184-token
outputs identical; IQ2_XS 562 tok/s (stream-all off by its share), Coder 40K
1,160 tok/s.
On CUDA the register kernel holds 33,792 blocks (~135K cells). Past it qsa_block_topk took the original kernel:
256 threads, each reading its own stretch of the scores on all six passes, one shared histogram updated with
atomics. The dispatch follows the capacity (--max-context), so with a context past ~135K that was every prompt
batch and every decode window on a card without the sm_75 active-bound dispatch (#512), whatever the prompt's
length, and on sm_75 every prompt batch past 33,792 blocks and every decode window.
block_topk_wide_kernel is the register kernel's layout (1,024 threads, a histogram per warp, warp scans) with the
keys read from memory:
- the four radix passes: thread t reads blocks t, t + 1024, ... - a histogram has no order, so a warp reads 32
neighbours at a time;
- the count and the output: warp w holds `per` rows of 32 consecutive blocks, and the cells are written in the
order (warp, row, lane); a row without a selected block is skipped.
The selection rule is the original kernel's (radix threshold, ties to the lowest index, cells ascending): identical
ids. One launch per call, no scratch memory, no capacity limit.
Dispatch (CUDA; HIP is unchanged and does not compile the kernel): where the blocks do not fit the registers the
new kernel, except a call that names fewer than 4,608 blocks (a prompt batch below ~18K cells), which keeps the
original kernel: its fixed cost is lower there. A decode window (no block count) takes the new kernel at any
context. The register kernel's range, the sm_75 active-bound dispatch and the sm_90+ cluster path are unchanged.
STRATA_TOPK_OLD=1 keeps the original kernel, as before.
qsa_topk_parity (new, with the kernel parity tests): the dispatched top-k against qsa_block_topk_ref on both sides
of the register kernel's reach, at short contexts under a long capacity and up to 524,288 cells, called as decode
and as the prompt path call it, with continuous scores, 64 score levels and equal scores.
RTX 3060 12 GB (sm_86), 256 queries with their block count under a 262,144-cell capacity (524,288 for the last
two), original -> dispatched:
16K cells 0.199 -> 0.199 ms (the original kernel)
32K 0.845 -> 0.328
131K 4.70 -> 0.97
262K 16.09 -> 1.71
393K 26.02 -> 2.49
524K 37.53 -> 3.24
4 queries without a count (a decode window): 0.031 -> 0.028 ms at 8K, 0.176 -> 0.083 at 131K, 0.309 -> 0.127 at
262K.
IQ3_XXS on that card (--resident-experts, --max-context 262144, --kv int8 --kv-resident 32768),
tools/needle_bench.py, STRATA_TOPK_OLD=1 against the default: tok/s of the prompt, and the prompt path's selection
phase (STRATA_PREFILL_TIMING):
16,471 tokens 1004.5, 1016.2 -> 989.0, 1013.6 195 -> 195 ms
31,970 1043.3, 1055.5 -> 1060.1, 1060.5 900 -> 731
61,333 932.9 -> 1013.8 4,061 -> 2,428
122,755 904.7, 904.2 -> 992.7, 989.6 20,019 -> 9,034
185,729 819.7 -> 954.7 50,055 -> 20,761
243,176 743.3 -> 933.7 96,326 -> 35,544
The needles are found in every run.
3-way merge from the port's "main + PRs" migration to dpct of upstream 0.1.38
(prefill.cpp re-merged with the histogram diff: upstream's peer-device and fused
restructure kept, the port's edits carried over - the stager's GGUF job, the
stream-all gate, the fp16 NaN checks; generate.cpp: the pipelined profile fill
beside upstream's prefetching serial fill; mtp/verify: the SYCL host mapping).
- the PRs' header forks dropped: upstream has them now
- stubs: --peer-device (peer_experts.cpp; open refuses) and the fused int8 prompt
kernels (moe_fused_stub.cpp; MMQ is not built here)
- off on SYCL: the sm_90 cluster greedy sampler and QSA top-k (fixups.py)
- fused_gr: dpct's undeduced kernel names filled in (fixups.py)
- STRATA_GDN_KEYHEAD stays opt-in on the port (upstream's gate is NVIDIA SM counts)
- version 0.1.38
Tested on the B70: build clean; native_grouped, iq_multi, gr, qsa, kv_hybrid,
quantize_act parity pass, qsa_prompt_attn's int8/fp16 pass (its new Q4_0
tensor-core cases are refused: not in the port); Coder short/long, IQ2_XS, 40K
outputs identical; 128K int8 KV streaming 892 / 66.8 tok/s; GR_DOWN_MAX4 neutral,
GDN_KEYHEAD 8% slower (stays off).
The trim was on by default in the HIP builds. It now waits, as on CUDA, for the PR author's re-run of the release
branch on his cards; the default flips after that. --layer-split auto and one GPU were never affected.
"saving a conversation checkpoint failed" was the only text when the checkpoint's
device sync failed - a GPU hang that Windows then resets (VIDEO_ENGINE_TIMEOUT), not a
checkpoint problem. The error now names it: "...: the GPU stopped responding before it
(<error>) - a GPU hang; on Windows the driver is then reset". Error text only.
Two private engines, the same conversation: the baseline keeps it live, the candidate
parks it (an unrelated request in between, a re-rendering client, or two long
conversations taking turns) and restores it. Each turn's greedy tokens must match and
its decode speed must be within --tolerance (15%) of the live turn's. Pure logic
tested without a GPU (tools/test_conversation_restore_speed.py).
#528 did not reproduce on the RTX 5070 with it: Q2_0 37K tokens (--kv-resident 16384)
and IQ3_XXS 50-75K tokens (--kv-resident 32768) decoded at the live turns' speed after
a restore, with the same tokens.
- The per-layer wait for the router ids names itself in a stall report ("waiting for
the GPU (attention, router) at layer"): a GPU stall reads apart from a host one.
- STRATA_PF_STEP_SYNC=1 (opt-in): the compute and copy streams are waited for after the
K/V staging, the K/V append and each layer; a step over 250 ms is logged with its
layer and chunk, and a stall names the step.
- STRATA_KV_HOST_DMA=1 (HIP only, opt-in): with KV streaming the prompt path's append
writes the staging pool and the resident slots only, and one DMA copies the chunk's
blocks to the host copy - no kernel writes host memory over PCIe. Same bytes for
every reader (the staged first block is complete; past the chunk nothing is read
before a later append writes it).
Q2_0 on the 5070: the same tokens with STRATA_PF_STEP_SYNC=1 as without (and as 0.1.38).
STRATA_VERIFY_TRACE=1 records each verify window's handshake on the host (window
start, launch, each layer's ring and service, the final sync, a timeout, the release
and how long the GPU took to drain after it) with the seq / flag / A / B words, and
GPU breadcrumbs: the profiler's stage stamps per layer and token group, written to
mapped memory so they read while the GPU hangs. A timeout prints them before and
after the release; the serve watchdog's stall report (Verifier::diag) prints them too.
HIP only, opt-in, for the reporter's A/B: STRATA_VERIFY_COHERENT=1 allocates the
handshake words and rows with hipHostMallocCoherent; STRATA_DOORBELL_STORE=1 has the
GPU store each step's ring (known at capture) instead of read-modify-writing it over
PCIe. CUDA builds and the default path are unchanged (Q2_0 on the 5070: the same 96
tokens with and without the trace as 0.1.38; a forced stall dumps the breadcrumbs up
to the withheld flag).
docs/AMD_HIP.md's gfx1030 report measured an i7-13700KF (8P + 8E) at 38-42 tok/s with
all 15 workers against 36 with 8, so a CPU with no more E-cores than P-cores keeps the
engine's own count; the E-heavy chips the fork measured (i9-14900KF: 15 of 23 faster)
get the recommendation.
--vram-elastic (the config's "vram_elastic": true) allocates the expert cache as one
reserved address range backed by physical segments (CUDA virtual memory management,
512 MiB each, --vram-segment-mib), so every slot keeps its address. Between
requests, the engine command `VRAM <reserve_mib>` gives the last segments back to
the driver until that much VRAM is free (their experts become CPU misses, the prompt
path's loan moves down and its chunk shrinks to fit), or maps them again and puts
the same experts back when more is free. `VRAM` alone: the start's reserve. The
server's POST /v1/vram {"reserve_mib": N | null} sends it after a running request,
keeps it for an engine that is unloaded, and /v1/status shows the cache's size.
Never automatic. One NVIDIA GPU only (no layer split, helper caches, --peer-device
or the resident low-RAM mode); HIP refuses the flag.
RTX 5070, Q2_0 (4.8 GiB cache): reserve 6000 -> 4.3 GiB freed in 78 ms, decode
44 -> 33 tok/s; grown back in 92 ms, the answers token for token the ones before.
Without the flag the cache is the one cudaMalloc it was (same answers with and
without the flag until a VRAM command).
IQ3_XXS on the RTX 5070, temperature 0, 256 tokens, 8 French prompts (topics outside the corpus), 2 English and 2 code
prompts, arms alternating, 2 passes each: drafts accepted in French 0.505 (en) / 0.507 (cjk) -> 0.597 (fr), higher on
all 8 prompts and in each pass; English 0.610 -> 0.629, code 0.772 -> 0.779. Decode speed was not measurable here
(builds ran beside it); the reporter measured 141 -> 158 tok/s in French on an RTX 5090.
UD-IQ4_XS sits between IQ3_S and UD-Q4_K_XL: IQ3_S gate/up experts (IQ4_XS in one layer) with IQ4_NL downs (Q8_0 in
five layers), 59.5 GB of them, and UD-Q4_K_XL's dense side. The engine's kernels already take these formats; it was
packed (iq_pack.py --compat-bf16) and run on the Strix Halo, not yet on NVIDIA.
- MODELS["UD-IQ4_XS"] in the unsloth family: three shards pinned at the same revision 38bb39e with their sizes and
SHA-256 (the Hub's LFS pointers), the RAM budget like UD-Q4_K_XL (budget, one GPU, no images, --compat-bf16 pack,
no experts.bin), engine 0.1.38 or newer (the fused-layout fix for packs not every layer of which is covered).
- A model can have its own shard pattern and count (model_file / model_shards); budget_model() tells which Unsloth
model a config runs, for the #498 multi-GPU messages and RAM need.
- The AMD question (#429) stays for UD-Q4_K_XL only (nvidia_only): UD-IQ4_XS's formats have AMD prompt kernels.
- --gguf-dir accepts UD-IQ4_XS files (#444's list says so).
- UNSLOTH_Q4.md: its files, what setup does, what is known (Strix Halo), and the BF16 rounding of the hyper-connection
projections that every --compat-bf16 pack has (not changed here). MODELS.md: a paragraph.
- test_setup_unsloth: pins, budget, a mocked install, the engine check, no AMD question, the config read back.
UD-Q4_K_XL's install, its tests and every other model's config are unchanged.
Stateless, on the chat path: a Responses request becomes the chat template's
messages, tools and kwargs, runs through Service.run, and its reasoning, text
and tool calls come back as output items and the official typed SSE events
(response.created ... response.completed / incomplete / failed).
- input as a string or items (messages, reasoning, function_call,
function_call_output, custom tool calls); instructions and leading developer
messages make one system message; tool results are put in call order
- function, namespace (Codex's MCP/agent tools) and custom tools; hosted tools
are left out with a log line
- reasoning.effort -> reasoning_effort, max_output_tokens, text.format
json_schema/json_object through the structured output path
- reasoning returned as reasoning_text (and as opaque encrypted_content when
asked) and replayed into the prompt, so Codex's next turn reuses the cache
- errors in the Responses format; same API key, Host and Origin checks as /v1
- a response.in_progress event every 15 s while a long prompt is read
(Codex's idle timeout ignores SSE comments)
- docs: DETAILS.md section with a Codex config.toml
Based on #630 by CC-David-CC (the event assembler, namespace tools, call-id
ordering of results), without its opt-in flag, replay key and crypto
dependency, strict-function and JSON-format refusals, or the second
summary generation pass. Tested with Codex CLI 0.160.0 and openai 2.15.0.
Co-authored-by: David <[email protected]>
The About tab's Model settings card reads GET /config and saves through
POST /config: the sampling defaults, reasoning_budget_tokens,
fit_max_tokens, anthropic_thinking, effort_position, aliases,
idle_unload_s, lazy_load, engine_silence_s, api_monitor, open_browser
and --vram-reserve-mib (serve/runconfig.py). Every value is checked
first (one bad value changes nothing), only the named keys change, every
other key of strata-<model>.json stays as it is (as setup run again
keeps them, #629), the earlier file is kept as .bak, and the change is
used from the next start. POST takes JSON from Strata's own page only,
like /settings, and the API key when one is set; the network, key, MCP
and program keys are not editable there.
- MODELS.md: the RAM rule is the experts + ~10 GB (setup's arena_gb), not shard 1 + 10 GB; the sizes per model.
- DETAILS.md: prompts are read in chunks of up to 8,192 tokens (32,768 opt-in), not 2,048; Q2_0's AVX-512 pack is
~40 GB on disk = 34 GB of experts plus the dense weights.
- README: the NVIDIA speed table says which engine measured which row (Q2_0 0.1.36, the others 0.1.26).
- expert_cache.hpp: the comment that the cache "does not yet compute anything" is gone.
Private reports through GitHub's private vulnerability reporting; the
server's protections as of 0.1.38 (127.0.0.1 by default, the API key,
the Host and Origin checks, CORS, MCP tools and the request monitor
opt-in), with DETAILS.md for every setting. README links it.
/metrics gains "conversation_cache": the parked conversations, their
bytes, parks/restores/evictions and the last switch, read from the
engine's own log lines ("conversation cache: parked/restored ...") from
the start of its current run on; the budget and slots from its INFO line;
and the prompt tokens reused in the last request and since the start.
The card shows the parked slots and memory when --conversation-cache-mib
is on, the reuse lines always, and how to turn parking on when it is off.
--gpu-stages replays per-layer graphs, which a native (IQ) pack does not capture, and refused with "not captured
with the split". The serve path already times a native window's stages: STRATA_DECODE_TIMING=1 prints the
per-request decode split and STRATA_VERIFY_PROFILE=1 adds the GPU stages. DETAILS.md now says how to read them, and
--gpu-stages on a native pack says to use them.
The decode hit rate is the VRAM share of the experts the pool looked up; the routed experts the GPU reads over PCIe
(--pcie-frac) or another GPU computes are in neither count, so raising --pcie-frac raised the "hit rate" while the
decode got slower. Its meaning stays; now that share is said beside it:
- engine: DriveState::offload_entries counts those entries; the serve line adds "; N more read by the GPU over
PCIe or from another GPU (X% of all M routed)", and the DONE line a 16th field (an older server ignores it).
- server: /metrics' requests get "pcie_share" (None from an older engine), the console's done line "(+X% of the
routed experts over PCIe)", and the Monitor's column is "VRAM hit rate" with "+X% PCIe" after it.
- DETAILS.md says what each means.
No number changes; tokens are unchanged.
CMake links CUDA code with cudart_static by default and the targets also link CUDA::cudart (cudart.lib); MSVC with
CUDA 12.x (the STRATA_EXPERIMENTAL_SM60 build for sm_60/sm_70) then fails with LNK1169. That build (MSVC +
STRATA_EXPERIMENTAL_SM60, no CMAKE_CUDA_RUNTIME_LIBRARY given) now sets CMAKE_CUDA_RUNTIME_LIBRARY Shared, the
reporter's verified fix. The ready-made engine and every other build are unchanged. INSTALL.md: where github.com
cannot be reached, third_party/llama.cpp at the pinned commit can be put there by hand.
The effort's instruction is the first thing the template renders, so a
request that only changes reasoning_effort (or turns thinking off) read
the whole prompt again. With "effort_position": "end" in the run config
the server renders every prompt's start as the default effort's and puts
a low/medium effort in a short system turn right before the answer
(thinking off: just the template's empty thinking block); the rendering
is template-agnostic (history + turn + the generation prompt).
Engine: --tail-role-token ID (default -1, off) moves the turn checkpoint
in front of such a trailing turn, so it holds the conversation and not the
effort text; INFO reports tail_role_token. The server passes the id of
"system" only when the engine binary knows the flag, and says so when it
does not. Without the key nothing changes: the same prompt ids, no flag.
The first mapped start writes the whole arena (25 GB for the Coder) next to the pack. A drive without that much free
space filled up before the write failed. Now the start checks the free space first (the arena plus 2 GiB to spare);
without it, it says so and keeps the arena in RAM for this start. Linux opt-in only; nothing changes without the env.
The trim was measured on 2x MI50 (gfx906) only; nobody has run it on NVIDIA multi-GPU, where CUDA-only paths could
still read another stage's dense weights on a card that no longer holds them. The HIP builds trim with explicit split
points as the PR does (STRATA_STAGE_TRIM=0 keeps the full copies); the CUDA build keeps the full copies unless
STRATA_STAGE_TRIM=1. --layer-split auto and one GPU are unchanged everywhere.
find_nvcc takes the newest toolkit it finds, so with CUDA 12.8 and 12.9 installed, 12.9 was always used - and 12.9's
math headers do not compile with glibc 2.43 (Ubuntu 26.04). STRATA_NVCC=<nvcc> is now the only one considered (a
warning when the file is missing, or when it is too new for the Pascal/Volta build). CUDA_HOME is a candidate like
CUDA_PATH. TROUBLESHOOTING.md: the two build errors and what to do (g++-14 for gcc > 14; CUDA 12.8 or 13.x for glibc
2.43). Without STRATA_NVCC nothing changes.
The pinned expert arena (the default tier) was allocated without looking at the RAM: on Linux the anonymous mapping
succeeds whatever the host has, and a container whose memory limit is below the arena was killed by the OOM killer
part-way through the load, without a message.
- detail::host_available_memory: the file tier's probe (MemAvailable lowered to the tightest cgroup v2 limit's room),
now with its files as parameters (tested on fake trees), cgroup v1's memory controller (memory.limit_in_bytes -
memory.usage_in_bytes), and no cgroup line at all meaning MemAvailable alone (it was "cannot determine", which
refused a --resident-budget-gib on a v1 host). It also returns the tightest hard limit itself.
- ArenaExpertSource::open, after the size check and before the allocation: a hard cgroup limit below the arena is
refused with both numbers (a certain OOM kill); less RAM available than the arena is a warning with the numbers
and what runs with less (--mmap-experts with --resident-budget-gib), and the load goes on. Not for a shared arena.
- file_expert_source_test: v2 limit / max / malformed / missing, v1 limit / unlimited / not mounted, no cgroup.
On a 7200 RPM disk the default --ple-io direct's random reads of the n-gram table stalled prompts past the 60 s
watchdog, and the stall report looked like any other engine stall.
- Engine (Linux): at start with --ple-io direct, the table's disk is looked up in sysfs (queue/rotational); on a
rotational one it warns and recommends --ple-io ram or an SSD (not on WSL, whose virtual disk says rotational).
- Stall report (Linux): the engine's threads in state D and their wchan ("threads waiting on the disk (state D): 16
of 35 (blk_io_schedule x16)"), with what that means.
- Setup (Linux): a model on a rotational disk gets --ple-io ram when the RAM holds the table beside the model,
otherwise a warning that recommends an SSD. Nothing changes on an SSD.
- DETAILS.md: --ple-io direct is for SSDs (it reads unbuffered, not through the OS cache), and a troubleshooting row.
The tokenizer matches <think> and </think> as their special tokens even
in plain text (type 4, as llama.cpp does), so a user quoting "</think>"
handed the model a real end-of-reasoning token and the answer came back
empty. Before the template is rendered the tags in message text (user,
tool, system, assistant history, tool call arguments, tool descriptions)
are swapped for private-use marks; after it, those spans are encoded as
ordinary text (Tokenizer.encode's new plain= spans), the template's own
markers stay special. An assistant turn that opens with its own
<think>...</think> block keeps that block. A prompt without the tags is
encoded exactly as before; count_tokens uses the same path.
hardware_key, the settings file's key for a saved calibration, read the cards with nvidia-smi's list: on an AMD PC
every card was "?|0GB" (two different cards shared one calibration), and on a PC with an NVIDIA card too a HIP index
named that NVIDIA card or stopped setup ("there is no GPU N"). A HIP config's cards now come from amd_gpus, in HIP's
numbering, with the architecture in the name (an RX 7900 XTX and an RX 7900 XT are both gfx1100; a card without a
product name is known by its arch). Setup reuses a saved calibration on Linux HIP as on NVIDIA (setup_calibration);
Windows HIP keeps the defaults for now. The tuning is still offered on NVIDIA only - a `./setup.sh --calibrate` on
HIP runs it by hand, and each control is verified on HIP before setup offers it.
The NVIDIA key and the settings file's format are unchanged. Tests: tools/test_setup_amd.py CalibrationKey (7).
French answers drafted poorly with the shipped subsets: 23.6% of the token occurrences of French text fall outside
draft_vocab_en.bin, and the draft head cannot propose those (no subset adds the accented words and word pieces).
- tools/draft_vocab.py: a coverage mode. --corpus FILE... adds to the base the most frequent tokens of the corpus that
together cover --coverage (default 0.99) of its occurrences, and prints the share outside the base and the new list.
- data/draft_vocab_fr.bin: draft_vocab_en.bin plus 5,686 tokens (46,211 ids), from 10 French Wikipedia articles
(France, Paris, Napoleon Ier, Langue francaise, Union europeenne, Litterature francaise, Cuisine francaise,
Economie de la France, Revolution francaise, Seconde Guerre mondiale; 1.24 M characters, 326,926 tokens) at 0.99:
python tools/draft_vocab.py --gguf <IQ3_XXS shard 1> --base data/draft_vocab_en.bin --corpus <the articles> \
--coverage 0.99 --out data/draft_vocab_fr.bin
Outside the list on that corpus: 0.74% (23.65% with en).
- setup.py: "fr" in DRAFT_VOCABS / --draft-vocab (~151 MiB head on IQ3_S, between en and cyrillic). The sha256 check
knows it as a shipped subset, so a later --draft-vocab cjk/en replaces it and a subset made by hand stays.
- tests: tools/test_draft_vocab.py (the coverage mode), tools/test_setup_draft_vocab.py (fr replaces a shipped subset
and back, keeps the en ids first in their order, its MiB estimate).
The default subset (cjk) is unchanged.
A broken state (or a model in a loop) can answer one token forever: #606 streamed 36,689 tokens of "!" for four
minutes, because a request without max_tokens gets the whole remaining context. The server now ends such a reply
at 256 identical tokens in a row with finish "length" and a line in its window saying what to do if it repeats on
a new request. "repeat_stop_tokens": N in strata-<model>.json sets the run (0 = off). Nothing changes for a reply
without such a run.
Both q8_1 activation quantizers (iq_kernels.cu q8_1_store, native_mmvq.cu native_quantize_q8_1_kernel) stored the
sum of a block's 32 raw activations as fp16: past 65504 (one massive-activation dimension is enough) it rounded to
inf, and d = amax / 127 did past amax = 8.3M. The dot products then read inf * 0 = NaN, and a NaN in the residual
answers one token forever (ggml-org/llama.cpp#23606 is the same defect).
Now both are clamped to the largest finite half (and the values to the int8 range they then exceed). A block that
was finite before is stored bit for bit as before - values in (65504, 65520) round to 65504 anyway, and a finite
block's |x / d| is at most 127 - so the default output only changes where a block overflowed (and was NaN). A NaN
input stays NaN, so STRATA_DBG_NAN still finds it.
iq_multi_parity checks both quantizers against the 0.1.38 formula on a host transcription: every finite block as
before, and three overflowing blocks (sum 96,000, sum -69,984, amax 2e7) stored finite.
On a hybrid CPU (Intel 12th gen+, AMD Zen 5 + Zen 5c) setup writes --pool-workers
N into the run config: the performance cores but the host loop's one, plus half
of the efficiency cores. An E-core runs the expert kernels ~2.2x slower and each
layer waits for its slowest part; Hardin22 measured 15 workers at 165 / 116 tok/s
against 106 / 84 with all 23 on an i9-14900KF (#642). A recommendation only: the
line is in the config, a saved calibration's measured count wins, and a CPU whose
cores are all alike gets the same config as before.
The engine's pool, with every core allowed (the default affinity), now orders the
P-cores first on a hybrid CPU, so a smaller pool runs on the P-cores and the first
E-cores rather than on whatever the OS numbered first (from Hardin22's fork).
Non-hybrid CPUs: unchanged.
Co-Authored-By: Hardin22 <[email protected]>
Kept in the run config's vision.max_tokens (the server already passes it
to the encoder); a setup run again keeps it for the same encoder device.
More than the default is allowed with a note on the encode time. Without
the flag the config is as before. DETAILS.md: vision.mmproj can point to
a Q8_0 mmproj of this model.
- The draft layer's K/V lives on the last stage's GPU (the drafter is loaded there); its save, ring restore and
verification now run with that stage's image under that device, instead of with stage 0's image on GPU 0, where
the ring restore kernel ran against another GPU's memory (seen in Niko1221/Strata#653, which places it the same way).
- Parking with a layer split on one GPU (--split-device 0) is refused at start: untested.
- The whole-session validate refuses an image with stage images; clearer message for stage parts in live.
- --trim-stage-weights says which layers each GPU loads, and fails when index.txt cannot be read.
- conversation_snapshot_test: the whole-session refusal; parking_test.py runs with STRATA_SNAPSHOT_VERIFY=1.
tools/make_profile.py builds the ranking as the base's order, then the
trace's pairs, then the fill. The base is only appended to, so --base can
never move a pair the base already ranks - and the shipped
data/expert-profile.bin ranks all 24,576 pairs, which makes a trace a
no-op:
wrote ...: 24576 ranked pairs (24576 from the base, 0 from the traces, ...)
So the documented way to seed a profile from --dump-routing (rank the
pairs your workload routes most) silently did nothing with the shipped
base; only --no-base changed anything, and that drops the base's ranking
entirely.
Add --reorder: rank the trace's pairs first (most frequent first), then
let the base fill the rest, then the fill. The default is unchanged. The
ranking moves into a rank_profile() so it can be tested without a model;
tools/test_make_profile.py covers the no-op, --reorder, --no-base, the
frequency/tie order and an end-to-end run over a complete base. On this
host's coding trace the same command now puts 5,808 trace pairs first.
--no-browser (on a start or at setup) saves "open_browser": false in the
model's run config; start() and run-<model>.bat/.sh then leave --open
out, and the server ignores --open when the config says so (a start
script written earlier still has it). --browser undoes it. Without the
flag the config, the command and the script are as before.
The adaptive swap loops called srcp->blob() for every swap to collect the spans pin_blobs locks, on every build -
pin_blobs is a no-op except on Linux AMD with STRATA_ARENA_MMAP=1. blob() is not free: it counts reads and the file
tier (--mmap-experts) may assemble the blob, a second time for the copy. pin_blobs_on() now guards the gathering, so
CUDA and the runs without STRATA_ARENA_MMAP do exactly what 0.1.38 did.
"layer_split" is the first layer of each later GPU ("24,36,42" for 4 GPUs), not a count of layers per card. A
value such as "24,16,12,12" failed the engine's check after the start began, and a JSON list reached the engine as
"[24, 16, 12, 12]".
- server.py layer_split_value: "auto" (default), a comma string or a JSON list of one rising number per GPU after the
first, from 2; anything else stops the start with the format and an example for that many GPUs.
- The engine's message says the same, with the value it got.
- MULTI_GPU.md and setup's --layer-split help say it.
The host expert arena is normally read into pinned (locked) RAM. On a machine whose GPUs already hold most
experts, that locks memory the CPU never reads: with the Coder (25 GB arena) on 2x 16 GB GPUs and 32 GB of RAM,
~1 GB stayed available at 128K context, the kernel swapped, and a multi-turn checkpoint took 25-80 s.
With STRATA_ARENA_MMAP=1 (opt-in, Linux; off by default, nothing changes without it):
- the first start writes the arena to the pack's experts.bin (padded by one blob so a whole-slot copy may start at
any expert); later starts map it read-only, not locked, not pinned. The page cache keeps what the CPU pool and
the prompt path read and gives the rest back under pressure; a dropped page is re-read from the file, so no
result depends on what is resident. The GPUs get no mapped alias of it: run with --pcie-frac 0;
- after the VRAM caches are filled, the pages of experts a GPU holds are handed back (MADV_DONTNEED,
STRATA_ARENA_RELEASE=0 keeps them) and the rest are prefetched (MADV_WILLNEED); an adaptive swap releases the
expert that went to VRAM and prefetches the one that came back;
- HIP: ROCclr copies a pageable source of 1 MiB or more by locking its pages in place and keeps them locked, which
undid every release. The engine raises GPU_PINNED_MIN_XFER_SIZE (an explicit setting wins) so such copies
stage through ROCclr's own buffers, and locks an adaptive swap's source pages itself only until the copy lands
(the page ranges are merged first: a copy from a range only part of which is registered fails);
- STRATA_RSS_TRACE=1 prints the process's RssFile at the startup steps (madvise reports success on locked pages
and frees nothing; RssFile is the honest check).
The existing --mmap-experts refuses native packs (the Coder); this path is for them.
Measured on 2x AMD Instinct MI50 16 GB + 32 GB DDR4 (gfx906 port, Coder IQ1_M, 128K context, layer split):
RAM available while serving ~1 GB -> 25 GB; multi-turn re-read of a checkpoint 25-80 s -> <= 3.2 s; decode
unchanged (49.06 vs 49.16 ms per verify window). Not measured on NVIDIA (pin_blobs is a no-op there).
The file tier's routing-aware prefetch (RouterLookahead) copies all layers' ffn_gate_inp.weight from CUDA0's arena.
With CUDA0 trimmed to its own layers it found them missing and turned itself off ("the routers are not BF16 in the
arena"). CUDA0 now keeps the routers (~2.5 MiB a layer) when --mmap-experts is on; the other stages and the runs
without --mmap-experts trim as before.
Without this every stage of a layer split keeps a full copy of the dense weights (~3.4 GB for the Coder),
although it only ever reads its own layers. That VRAM goes to the expert cache instead.
- the canonical arena of each GPU skips the `blk.N.*` tensors of layers outside its range (PLE tensors stay
everywhere), read from the pack's index.txt;
- NativeDense::set_layer_range limits the native projections the next `load` uploads the same way;
- only with explicit split points (`--layer-split 27`); `auto` and a single GPU are unchanged;
STRATA_STAGE_TRIM=0 keeps the full copies;
- one log line per GPU says what it loads, and a later stage logs its free VRAM before the cache is sized.
Measured on 2x AMD Instinct MI50 16 GB (gfx906 port, Coder IQ1_M, 128K context, --layer-split 27):
experts held in VRAM 8,819 -> 10,626 of 12,288, VRAM hit rate 95% -> 98%, decode on a 17K-token agent prompt
39.2 -> 41.7 tok/s. The trimmed tensors are never read by that GPU.
A re-run for an installed model (another --context, say) rewrote
strata-<model>.json from scratch and dropped the user's sampling,
mcp_servers, mcp and other blocks. Setup now carries over every key it
does not write (and env/vision entries of the user's own, such as a Q8_0
mmproj), keeps the earlier file as strata-<model>.json.bak and says what
it kept and which hand-added engine options it did not. A copy set up
like an earlier install carries them over from that install's config.
A first install writes the same config as before.
The native head's cudaMalloc ran after the expert arena had registered ~45 GiB of host pages. WDDM then refused it
on a 16 GB card with the desktop on the iGPU ("native head upload: out of memory" with GiBs free) - the drafter is
loaded before the arena for the same reason. Now the head and the logits buffer load right after the drafter, still
before the expert cache is sized, so the slots are the same. The head's, the drafter's and a split stage's head
failures print "(N MiB of M MiB VRAM free on this GPU)".
setup: a context of 64K or more whose KV streaming does not fit the RAM test now says so (it was silent, and a
regenerated config lost --kv-resident without a word).
The server replied to /load and /unload without reading the request body; on
Windows, closing a socket with unread data sends a TCP reset, so the client saw
a connection reset instead of the answer. Read the (bounded) body first.
Taken from #630 by CC-David-CC.
0.1.38 made the choice before the resident set existed, from the budget asked for, against every shard's bytes
(dense weights and the PLE table included). On a 96 GB PC with UD-Q4_K_XL (budget 72 GiB, 48.94 GiB really
resident) that read 83.0 - 72 - 4 = 7 GiB of room against 103.7 GiB of files, so every refill after a prompt read
the drive: prompts 15-40% slower than 0.1.34.
- platform::file_cache_keeps(avail, arena, read): the decision, unit-tested in platform_memory_test with the
reporter's numbers and the 32/64 GB PCs of #357/#362.
- set_unbuffered: the arena is at most every expert's bytes, and the cache has to keep only the experts outside it.
- recheck_unbuffered, after pin_cache_complement: the same decision from the RAM the copy really holds and the
expert bytes outside it; switches either way (startup, nothing reading) and logs the numbers.
- The 32 and 64 GB PCs still read unbuffered (the profile fill and the copy too, as before).
Same tokens either way: only how the same bytes are read changes.
openai_to_messages / anthropic_to_messages check that tools is a list of
named tool objects (a JSON-string list is decoded, as for messages) and
raise ValueError otherwise; the MCP name set skips non-objects too.
0.1.38's close() already ends the old engine and waits for it (QUIT, terminate, kill, 20 s each) and raises
EngineStuck when it cannot. The PR's second kill and 120 s wait, after which a new engine was started next to one
still alive, is left out; the start retry and the last known context stay.
Seen in long agent sessions (Claude Code, many turns at ~20K tokens): when the engine was ended mid-request (the
60 s watchdog), restart() started the new engine while the old process still held its VRAM. The new one ran out
of memory and exited before READY, `max_context` stayed 0, and from then on every request failed with
"400 ... exceeds the context (0)" until the server was restarted by hand.
- restart() ends the old process first (close(); kill and wait if it is stuck), so its GPU memory is free;
- a start that exits before READY is tried again, 3 times, RESTART_RETRY_S (15 s) apart; `starting` is set
meanwhile, so a request gets the 503 "starting" of #344;
- `known_ctx` keeps the last context the engine reported: after a restart that failed for good, a request is
checked against it and reaches run(), which starts the engine again, instead of a 400 about context 0.
Two tests in RestartWindow (both fail without this change): a start that fails once is retried and the engine
comes up; a request after a failed restart gets 200, not a 400. `python -m unittest serve.test_server`: 120 OK.
native_expert_grouped for the UD-IQ4_XS shapes (n_embd 2560, n_ff 640): s26_gu_l_kernel / s26_down_l_kernel keep
every row's lane-strided k order, Split<TY>::apply expression and warp_sum, so every output is bitwise the old one's;
a warp interleaves 4 rows (more loads in flight), the chunk's activation q8_1 rows are read from LDS, and for IQ3_S
the 512-entry grid table too (the same values); SwiGLU and its q8_1 run as one launch (contraction off in the
quantizer). Other formats / shapes keep native_gu_multi_kernel / native_down_multi_kernel.
gfx1151 harness (S26 ws3: real expert blobs, 32 experts x 4 layers = 298 MB, decode-style windows planned as
resident_plan does, memcmp of every output, paired HIP events): one layer's experts T=1 140.3 -> 133.3 us, T=2
225.1 -> 204.1, T=3 300.2 -> 265.9 (166 -> 175-185 GB/s). Tried and rejected: non-temporal loads (no gain), whole
rows staged in LDS (0.6-0.8x: occupancy), 8 rows per warp for down.
Engine (UD-IQ4_XS, c-best3 env + the switch, same binary, S25's 18 prompts): outputs 18/18 identical; 56.6 / 57.0
-> 55.65 / 55.27 (and 55.61 with the fused SwiGLU q8_1) ms per verify window, 48.6-48.9 -> 50.3 tok/s;
59K prompts 2/2 identical, 58.65 -> 56.77 / 56.66 ms per window, 39.70 -> 41.01 / 41.09 tok/s.
- conversation_snapshot_*(..., const QsaState* draft): nullptr saves, validates and restores an image without the
draft layer's K/V. The later stages park that way; the draft ring is saved once, with the first stage's image.
- ConversationKvReuse::stages / ConversationCache::retain(.., stage_kv): every stage's restored K/V is retained, so
its next park copies only what changed, as the first stage's already did.
- conversation_checkpoints_split / _merge: the checkpoints are moved into their stage parts for the capture and
back afterwards (and on restore), instead of copied.
- Tests: conversation_cache_test (split/merge round trip without copies, refused misalignments, per-stage reuse),
conversation_snapshot_test (draft-less images: estimate, validation both ways, A/B/A exactness, incremental reuse).
- tools/parking_test.py: a follow-up decodes the same tokens whether its conversation stayed live or came back
from the parking cache (every stage's image with a layer split). 4-GPU split, IQ3_S: 3,786 and 9,276-token
conversations restored in 42 / 51 ms, follow-ups identical; without parking the same follow-ups took 3.4 / 6.8 s.
- tools/batch_test.py: --keys to send sampling keys (sampled rows in batch windows).
- docs/BATCHING.md: parking together with batching, the Testing section; docs/DETAILS.md: parking now works with
--layer-split.
park_current snapshotted stage 0 only, so a split rejected parking. A parked conversation now holds one image per
later stage (SavedConversation::stage_images, captured from each stage's session on its own GPU); a split
checkpoint is cut into its stage-0 part and its stage parts while parking and put back together on restore; every
stage image is validated before anything is overwritten; the guard is removed.
Patch posted on #559 by crazyaimachine (written by their coding agent), tested on 2 GPUs there and on a 4-GPU
split here.
Co-authored-by: benoit lange <[email protected]>
GDN step (verify_kernels.cu, S25's STRATA_GDN_SPLIT=1 split, handed to S26):
- the commit's replay writes no outputs (t_out_begin = MT): y_dummy_ was never read; the state is unchanged;
- the commit runs gdn_state_replay_kernel: S25's column split (4 blocks of 32 columns per head) with gate, beta and the
v column of every row prefetched and the next row's k / q loaded while the current one computes;
- the window keeps the old kernel for one row (the split did not pay there) and runs the split for 2+ rows with
gdn_out_norm_kernel rewritten in the old kernel's code shape: S25's 128-thread norm differed from the old kernel by
1 ulp in ~7% of the outputs (405 of 6144 at T=1); the S x RG version is bitwise equal.
gfx1151 harness (h_v 48, 36 layers rotating, paired, every layer memcmp-equal): commit 44.1 -> 39.1 us at 1 row,
50.7 -> 41.8 at 3, 60.8 -> 46.1 at 6; window 23.8 -> 23.2 us at 2 rows, 30.1 -> 26.7 at 4.
STRATA_QFUSE=1: q8_1 images written by their producers instead of a separate native_quantize_q8_1 launch:
- the GDN output norm (both window kernels) quantizes its outputs (warp = one 32-column q8_1 block), contraction
off in the quantizer so fma(a, b, shfl) cannot round the first sum differently (that made ds.sum differ by 1 ulp);
- the hyper-connection read (gr_up_multi_kernel, gr_up_q8_kernel): the second of the two 16-column blocks of each
32-column group (a per-caller counter, reset by that block) reads the 32 `mixed` values back and quantizes them -
attention side into xq_, FFN side into nat_xq_; fused_gr_read_multi returns whether it wrote them (the v3 read
does not). gr_parity: new check, T 1..8, direct and graph replays, q8_1 bytes equal to native_quantize_q8_1's.
Harness: GDN step + quantize 23.1 -> 20.5 us (1 row), 26.4 -> 23.3 (2), 32.6 -> 27.0 (4), q8_1 bytes equal.
Engine (UD-IQ4_XS, c-best2 + the switch, S25's 18 prompts, 256 new tokens): 18/18 identical in every run;
c-best2 49.46 / 49.44 tok/s, + STRATA_GDN_SPLIT=1 50.13 / 50.23 / 49.99, + STRATA_QFUSE=1 50.10 / 49.98 / 49.95.
strata_pf_gemm_f16 (wmma_gemm.cu), tried in Gemm::f16 before hipBLASLt for T >= 64, N >= 512, K % 32 == 0, beta
0 / 1: block 128 x 256 (128 x 128 below N 1024), BK 32, 8 waves of 64 x 64, LDS double buffer with the next stage
prefetched into named registers, grouped tile order (8 row tiles sweep all column tiles, so the weights stream
from DRAM once per group). FP32 accumulation, another K order than hipBLASLt: rounding-level.
gfx1151, M 16384 (TFLOPS, tuned hipBLASLt in brackets): 12288 x 2560 33.0 (27.4), 10240 x 2560 31.9 (26.8), 6144 x
2560 31.1 (25.9), 2560 x 2560 30.1 (24.9), 640 x 2560 30.2 (25.3), 2560 x 6144 19.4 (19.1); the register-only WMMA
ceiling is ~50. Engine 8K: gdn 1122 / 1112 -> 1083 / 1071 ms, qsa proj 813 / 716 -> 692 / 676, gdn out proj
753 -> 645 / 665. Teacher-forced KL 0.099 / 0.028 on two text offsets (accepted pair: 0.117 / 0.030). Off by
default. With STRATA_HC_UPMIX + STRATA_PA_FAST, UD-IQ4_XS 64K prompt 872.8 / 880.6 -> 942.4 / 937.8 tok/s.
prompt_attn_wmma_kernel carries q and p as FP16 hi + lo pairs (two WMMAs each, FP32-level). SPLIT = false drops
the lo halves: FP16 attention as the usual flash-attention kernels, half the matrix work.
gfx1151, real 64K selections (16383 queries x 2051 cells, one QSA layer): 138.7 -> 117.8 ms; rel RMS vs FP64
4.8e-7 -> 2.7e-4. Engine 8K "qsa attn" 718 / 712 -> 589 / 590 ms. Teacher-forced KL on three text offsets 0.153 /
0.087 / 0.027 (the accepted MMQ-vs-fused pair: 0.153 / 0.117 / 0.030). Off by default.
The read's up GEMM wrote `gated` (T x 10240 FP32: 671 MB per 16K chunk) for gr_mix_r to read back. gr_upmix
computes a tile of 128 tokens x 16 columns for the 4 streams on the matrix cores (BF16 in, FP32 accumulate, the
block's 64 w_up rows staged in LDS once) and runs the mix in registers, in gr_mix_r's order. The grid is ordered
column-tile fastest: the blocks in flight share one token tile, so the R reads of the epilogue stay on few pages
(token-tile fastest touched 512 rows x 40 KB per block: 28 ms instead of 3.7 at 8K - address translation on this
box goes through the IOMMU). STRATA_HC_UPMIX_CHECK=N runs the default pair on the first N reads and prints the
difference.
gfx1151, UD-IQ4_XS: `mixed` vs the default rel RMS 1.2-3.6e-7 on every read of an 8K prompt; prompt-timing "hc
read" 1352 / 1313 -> 1080 / 1212 ms at 8K; 64K prompt 874.3 / 876.1 -> 902.7 / 903.0 tok/s. Rounding-level (the K
order of the GEMM): teacher-forced KL on three text offsets 0.170 / 0.085 / 0.032 vs the accepted MMQ-vs-fused
pair's 0.153 / 0.117 / 0.030; ledger and 8K / 64K retrieval pass. Off by default.
With --batch, a consumer that left mid-answer (a client disconnect, a tool call or a stop string ending the
response) left the engine writing the abandoned answer's tokens, and the next request read them as its own:
it answered the previous question again. Now the solo request is STOPped and read to its DONE, an admission
to its BADM, and a slot is BSTOPped and freed only at its BDONE. STRATA_BATCH_TRACE=1 logs the control lines.
tools/early_close_test.py reproduces it (solo and with a second request running).
The engine's --serve can now keep up to N conversations open (--batch N) and decode them together, one token
each per verify window; with a layer split, --batch-groups G pipelines the slot groups through the GPUs, and
--trim-stage-weights makes every GPU load only its own layers' dense weights (VRAM back to the expert cache).
The server runs concurrent requests in the slots. See docs/BATCHING.md.
Appends [n][hc*n_embd] BF16 rows (round-to-nearest-even) in position order for every batched prompt chunk: the
input the MTP draft layer reads, for training it against this trunk. Off unless the variable is set.
Layer 1's PLE block ran its key (10240 x 2560) and value (2560 x 2560) GEMVs once per window row, reading ~40 MB of
weights per row. Both depend only on the row's n-gram embedding, so with the switch the group's rows go through
the multi-row forms of the same kernels (native MMVQ over n columns / bf16_gemv_fp32_mmvf_multi, each row's
arithmetic the single-row call's) and ple_block copies its row's results instead of recomputing them. The
history-dependent rest of the block stays row by row.
The GGUF's F32 inject rows hold BF16-exact values (conversions.json: "the values are BF16's"), so quantizing them
to Q8_0 only added error: the Q8_0 read now takes the pack's BF16 inject rows (STRATA_HC_Q8_INJECT=1 keeps the
Q8_0 copy). The final mixer before the head (output_hc_down/up, also BF16-rounded in the pack) now reads the
GGUF's Q8_0 values through the same multi read.
Conflicts: mtp.cpp (the draft head: S25 converts it with --mtp-q4 first, then STRATA_Q6_PACKED packs it only
while it is still Q6_K), native_dense.cpp (includes).
gdn_step_norm_multi_kernel ran one 512-thread block per head - 32 blocks for a layer's 2 MB state, ~36 us per
launch at T=1 on gfx1151 (~60 GB/s) and +4 us per window row; it runs twice per GDN layer and window (the window
and the commit's replay). With the switch the recurrence runs on (head, 32-column) blocks with the same
(column, row group) threads and state rows, so every column's arithmetic is unchanged; the output norm, which
needs a head's 128 columns, follows in gdn_out_norm_kernel with the same warps and the same sum order.
The main output head (248320 x 2560) and the MTP draft head (its 106299-row subset) are Q6_K: 210-byte blocks, so
every int load is 2-byte aligned. STRATA_Q6_PACKED=1 builds a packed copy of each at load - the same bytes as ql /
qh / scales / d planes (+497 MiB, +213 MiB) - and native_q6_k_mmvq on a registered pointer runs
native_q6_k_packed_kernel: aligned non-temporal dword loads, 640 persistent workgroups, and the thread ownership,
q6_q8_dot_impl expression, accumulation order and reduction of native_q6_k_mmvq_kernel<false> /
native_mmvq_multi_kernel<Q6KTraits, NC, 4, 1>, so every output is bitwise equal (exact multi-column layout only).
STRATA_Q6P_SELFTEST=1 checks it at load (1-4 columns, memcmp) and times it paired; =2 also sweeps the grid.
gfx1151 load-time self-test (11 paired samples, medians): main head 1 column 2420 -> 2300 us, 2 columns 2563 ->
2455, 4 columns 2973 -> 2975 (no gain: not bandwidth-bound there); draft head 1 column 1319-1351 -> 946-961 us,
2 columns 1420-1459 -> 1064-1090, 4 columns 1629-1678 -> 1320-1365. Outputs bitwise equal. Grid 320-20480 flat.
STRATA_VERIFY_RESIDENT=1: when every expert of every layer is resident in the VRAM/UMA tier (checked once at init;
a full cache never evicts; one stage only), the window graph is recorded without the per-layer host step: no
doorbell_publish, no wait_flag_ge (3 per layer), no plan copy from mapped memory, no PCIe/CPU share (fetch_blobs,
rebase_ptrs, its empty grouped launches, copy_rows_from_mapped) and no moe_hit_add: the device plans each group
(resident_plan, the host's exact loop) and the combine reads the hit rows as 0.0f + hit (combine<true>), which is
what the zeroed part rows plus parts += hit gave. run() then skips the per-layer spin. Same tokens.
resident_plan: the plan in one block of 128 threads (entry i per thread; groups in first-occurrence order, entries
in increasing i, the same arrays) instead of one thread - 44 us -> a few us per call on gfx1151, 48 per window.
Used by STRATA_VERIFY_DEVICE_PLAN=1 too.
STRATA_VERIFY_QDEDUP=1: the experts' q8_1 image of the MoE input is made before the shared expert, whose gate/up
read it instead of quantizing the same rows again (quantize_q8_1_rows and native_quantize_q8_1 write the same
bytes); the shared expert's SwiGLU and its q8_1 run as one launch (native_swiglu_q8_1_kernel: the same expression,
then native_quantize_q8_1_kernel's block quantizer); the BF16 copy of the input is skipped when the shared expert's
native BF16 gate (which never reads it) is on. Same bytes, same tokens.
Aurora (gfx1151, UD-IQ4_XS, S23's c-u4 config, one binary, env-only arms): output ids identical in every run.
Per verify window (rocprofv3 trace, 8K prompt + 300 outputs, 114 windows): see aurora_s23.md (S26, overhead).
The router (512 x 2560) and the indexer projections run bf16_f32_mmvf_multi_kernel: one block per output row, so
the window's T activation rows were read again by every block (+3.3 us per row per launch on gfx1151, ~0.2 ms per
window row). With the switch a block computes 4 output rows from one read of the activations; per output each
thread keeps the same pairs, FMA order and reductions.
attn_chunk_kernel scored a chunk's 64 cells warp by warp: each lane 8 of the 256 dimensions, then one warp_sum
per query head - 12 shuffle reductions per cell, which is what made the kernel ~53 us per layer at T=1 and
+32 us per window row on gfx1151 (it scales linearly with T). With the switch, thread t scores cell t % 64 for
3 of the 12 heads alone: the same 8-dimension partial expression per former lane, then lane 0's butterfly order
(a[l] += a[l + o], o = 16 .. 1), so each score has the same operations in the same order. Softmax, values and the
merge are unchanged.
The read takes the Q8_0 kernels only when the inject rows have a Q8_0 copy as well, and the GGUF stores them as
F32: with only down/up loaded (0.62 GiB) every window still ran the BF16 read (Aurora: 18/18 outputs identical
to the default, verify 61.4 vs 60.9 ms). The inject rows (4 x 10240 per projection) are now quantized at load
with ggml's Q8_0 reference rounding.
NativeDense keeps the six hc_* projections per layer as stored (Q8_0, 0.65 GiB) beside the pack's BF16 copy
(1.20 GiB, --compat-bf16's rounding); the verify window's fused_gr_read_multi then reads them with split-K
kernels: gr_down_q8 (10 row groups + the inject rows x 16 chunks of 640 columns, the norm folded in as
unscaled partials + per-chunk sums of squares) and gr_up_q8. Every other path keeps BF16. Another summation
order and the GGUF's own values: output-changing, so opt-in and quality-checked separately. Also the
STRATA_GR_V3=2 knob (the two-half v3 split where only 64 KiB of LDS is available). Taken over from S23
(aurora_s23/hc_q8_grv3s2.diff) unchanged.
On Strix Halo (UD-IQ4_XS, head Q6_K) the 4-bit head costs acceptance (tokens per step 2.81 -> 2.72 with both)
while the smaller 40.5K-token subset (--mtp-draft-vocab data/draft_vocab_en.bin) costs none, so the two 4-bit
copies are now separate choices.
Measured on Strix Halo (UD-IQ4_XS, 1-2K contexts) a verify row costs ~9 ms on a ~61 ms window, so chaining
every 3+-token match (49% of chained drafts accepted) gave +6.4% tokens per step for +7% window time. The
policy now picks the number of chained tokens k with the best expected tokens per ms: the MTP window's
measured tokens + p_mtp (c + c^2 + .. + c^k) at the measured cost of the longer window, where p_mtp is the
product of the draft layer's own draft probabilities and c the chained drafts' acceptance by match length
(their own decayed counts, learned only from rounds that reached the chain). Unmeasured sizes are probed a
few times when p_mtp * c >= 0.6. STRATA_LOOKUP_CHAIN_FIXED=1 takes every proposal as before.
Both opt-in; drafts only, the verify window decides every token (18/18 prompts identical on IQ3_XXS).
- --mtp-q4: the draft layer's ten Q8_0 projections (fc_embedding/fc_hidden, q/k/v/o, the shared expert) and
the draft head's subset rows are requantized to Q4_0 at load/bind with ggml's reference quantizer
(native-experts builds). The prompt path's batched K/V pass keeps the Q8_0 originals.
- --mtp-draft-vocab FILE: the draft head's token subset from FILE instead of <mtp>/draft_vocab.bin
(data/draft_vocab_en.bin: 40,525 tokens instead of 106,299).
RTX 5070, IQ3_XXS (head Q5_K), 18 prompts: draft ms per window 2.55 -> 2.23 (q4), 2.13 (en), 1.91 (both);
tokens per step 2.64 -> 2.63 / 2.67 / 2.57.
All opt-in; with none of them given the engine runs exactly as before.
- --lookup-chain K [--lookup-chain-min M]: after the MTP's drafts, the suffix drafter matches the context
plus those drafts and appends up to K of the tokens that followed the match to the same verify window
(the window grows to at most 8). Greedy verification keeps the output. Own counts: a stderr line per
request and five more DONE fields when on.
- include/strata/spec/draft_source.hpp: a DraftSource interface (the request's own text is
PromptLookupSource; other sources register in extra_draft_sources() and the longest match wins).
- --mtp-hnorm stream: the draft layer's pre_fc_norm_hidden as one RMS per hyper-connection stream
(llama.cpp's qwen4exp MTP graph) instead of one over all four (vLLM's qwen4_exp MTP, the default).
New kernel native_qsa_rms_norm_grouped; the batched prompt K/V pass follows the switch.
- Diagnostics: STRATA_MTP_FULL_HEAD=1 drafts over the whole vocabulary; STRATA_MTP_TOP2=1 counts how often
the draft layer's runner-up is the token verify picked where a draft was rejected.
A lossless load-time repack of the native dense Q8_0 matrices with 1024 <= n_in <= 4096 and n_out >= 2048 (on
UD-IQ4_XS: attn_qkv, attn_gate, attn_q; 84 matrices, +1.87 GiB): the same int8 values in a qs plane and the same
fp16 scales in a d plane, 34 bytes per 32 values as before. native_q8_0_mmvq finds a registered matrix by its
GGUF-layout pointer (that copy stays: the prompt GEMMs read it) and runs native_q8_0_packed_kernel: 640 persistent
workgroups; each thread owns the bytes of the exact layout (block tid/4 + 32 i, ints 2 (tid%4) and +1), reads them as
one aligned 8-byte run with streaming loads, and accumulates and reduces in the same order, so every output is
bitwise equal to native_small_mmvq_kernel / native_mmvq_multi_kernel (exact layout; with the upstream multi-column
layout the GGUF kernels run).
gfx1151, S26 standalone harness (8 real UD-IQ4_XS layers per shape rotating, paired HIP events, 15 samples, every
layer x 1-5 columns memcmp-equal): attn_qkv +10-12%, attn_gate +11-12%, attn_q +10-11% per call. 6144-wide rows
(ssm_out, attn_output) and rows < 2048 are not packed (slower in the engine / standalone).
Engine (UD-IQ4_XS, S23's c-u4 config, same binary, env only): output ids identical in all 10 runs; steady output
8K 40.06 vs 39.77 tok/s (3 runs each), 64K 30.94 vs 30.34 (2 each).
HIP counts a gfx11 WGP (two CUs, which the WMMA prompt-expert kernels use as one in
the default WGP mode) as one multiprocessor, and the occupancy query answers 1 block
for these kernels where 2 run side by side - so every fused launch (native IQ and
Q2_0) ran a 20-block grid on gfx1151, 2 waves per SIMD. wgp_blocks() doubles the
query's answer (STRATA_PF_OCC=N sets it, an experiment knob).
The same work items and arithmetic: parity passes, fused-vs-MMQ rel RMS unchanged,
and the engine's outputs are the same tokens.
gfx1151, one layer, 8192 tokens x top 10, IQ3_S / IQ4_NL: blocks per WGP 1: 60.7 ms,
2: 44.7, 3: 49.9, 4: 45.2. Q2_0 (2048 tokens): 13.3 -> 11.4 ms.
Engine, UD-IQ4_XS (MTP + vision + speed projection): 64K prompt 819.8 / 823.9 ->
892.4 / 878.8 tok/s, 8K 825.5 / 820.8 -> 874.5 (the other 8K run overlapped a
download: 758); outputs identical (drafts 292/516 at 64K, 343/468 at 8K); ledger
{32,17,130}, 8K retrieval 3/3.
A rebinding page cannot send the key, so with one configured the check only
broke tunnels and proxies that pass their own name on (cloudflared does).
Keyless setups keep it, with allowed_hosts; the 403 now also says that
setting an API key turns the check off.
It keeps 256 of the 512 experts, chosen on code data, so it is weaker outside
code and in other languages; for general use, or when RAM allows, MODELS.md
now points to the full models.
/unload and /load took a plain form POST from any site: they now take JSON
from Strata's own page only, like /settings (_own_page). Without an API key, a
POST to /v1/* that carries an Origin must come from Strata's own page, a page
on localhost or an allowed host name, an origin in trusted_origins or
cors_origins, or a non-web scheme (browser extensions, desktop apps), and then
send JSON; others (and Origin: null) get 403, so a web page can no longer burn
GPU time with a text/plain POST. Requests without an Origin (curl, the SDKs)
are unchanged; with an API key, the key decides.
A web page of another site could point its own name at 127.0.0.1 and reach
the server as the same origin: without an API key it could read /status,
/metrics and /api/requests, change settings and run MCP tools. The server now
answers only Host names it knows: localhost (and *.localhost), any IP address
(rebinding needs a name), the bind address and, bound beyond this PC, the PC's
name, name.local, its LAN addresses and host.docker.internal; any port. No
Host header (HTTP/1.0) passes. "allowed_hosts" in the config and
STRATA_ALLOWED_HOSTS add names (".example.com" for a domain, "*" turns the
check off); trusted_origins' hosts count too. A refusal is a 403 naming the
setting and one log line.
The weight arena is the first large VRAM allocation and its size does not depend on the context, so when it fails
something else holds the card (another program, an engine still exiting). The error now names the CUDA error and the
card's free and total VRAM at that moment; a layer split's stage says the same for its card, with the MiB it needed.
Error paths only.
One line under the start banner: the config's engine options without the model's file paths, and the
server's fields (host:port, an API key set or not, GPUs, draft subset, fit_max_tokens, thinking budget).
The key itself is never printed.
The provider, the context and output limits, the thinking level as a default and as variants, and how
they relate to the context check, fit_max_tokens and reasoning_budget_tokens.
A CUDA 13 build that linked the system's CUDA 12 libcudart.so (a kit with a dangling libcudart.so symlink) read
cudaDeviceProp shifted: sharedMemPerBlockOptin came back as 1, #420's fits() then refused MMQ for every i-quant and
the prompts ran ~4x slower, with a log line printing "1 bytes of shared memory per block".
- ggml_cuda_info() (CUDA) reads cc, smpbo, smpb, nsm, warp size, integrated and cooperative launch with
cudaDeviceGetAttribute, which is ABI-stable. A matched build reads the same values: the same MMQ choices, the
same output.
- The engine warns at start when the loaded CUDA runtime's major version differs from the headers' it was compiled
with, and says to rebuild against one toolkit (-DCUDAToolkit_ROOT).
HIP is unchanged.
Dev::put uploads with a pageable cudaMemcpy, which can return before its DMA lands, and the graph replays and the
timing cases run on a cudaStreamNonBlocking stream that does not wait for the legacy stream: the graph could read
the previous replay's logits (one argmax token off; 5 of 6 replays on an RTX 5060 Ti, now and then on a 5090).
Dev::put now synchronizes the device after its copy, which covers run_graph_case and bench(). Test only; no engine
code.
A reply that ends with finish "length" inside the reasoning has no answer; the server log now says so
and names reasoning_budget_tokens (per request, or in the config for every request). Nothing the client
receives changes.
With the default 700 MiB reserve the expert cache fills an AMD card that also drives a KDE/Wayland desktop. When the
desktop then needs VRAM (the browser --open starts, a new app), amdgpu moves GPU memory to system RAM (GTT), which
the experts already fill: the OOM killer ends kwin_wayland / plasmashell (#560, R9700), or the compositor fails to pin
its framebuffer (#516, 7900 XTX). --vram-reserve-mib 3072 / 4000 fixed it in both reports.
A recommendation, nothing is changed: setup prints the tip on a Linux desktop session (WAYLAND_DISPLAY or DISPLAY) with
the HIP backend and no reserve given, and the server prints it after the start when the engine reports less than
2048 MiB of VRAM free (INFO vram_free_mib) and the reserve is under 3072 - before --open starts the browser.
INSTALL.md and AMD_HIP.md describe the symptom and the fix. No engine code.
installed_configs() and update_install() keep only JSON objects with "exe" and "args"; any other
strata-*.json in the folder gets a one-line warning naming it.
The verify window waits for the adaptive tier's copies since #463; STRATA_ADAPT_NOWAIT=1 starts it as 0.1.37 did (the
non-blocking query), for timing that wait. Default unchanged.
fused_layout() (STRATA_PF_FUSED=1) shrinks GU / H / Xq to the fused kernels' needs whenever
fused_ring() is true, and fused_ring() was true when ANY layer of a native pack was covered. A pack
with an uncovered layer - Unsloth UD-IQ4_XS: 5 layers with Q8_0 down projections - then ran MMQ for
those layers at the full chunk in buffers sized for stream_all_min() - 1 tokens: on gfx1151 an 8K
prompt answered "!!!!..." and a 64K prompt hung (--prefill 2048: an illegal memory access). Such a
pack now keeps MMQ's buffers; its covered layers still take the fused kernels. UD-IQ4_XS 8K
retrieval after the fix: 3/3 correct, 837.9 prompt tok/s. (The CUDA path shares this code.)
fused_layout() (STRATA_PF_FUSED=1) shrinks GU / H / Xq to the fused kernels' needs whenever
fused_ring() is true, and fused_ring() was true when ANY layer of a native pack was covered. A pack
with an uncovered layer - Unsloth UD-IQ4_XS: 5 layers with Q8_0 down projections - then ran MMQ for
those layers at the full chunk in buffers sized for stream_all_min() - 1 tokens: on gfx1151 an 8K
prompt answered "!!!!..." and a 64K prompt hung (--prefill 2048: an illegal memory access). Such a
pack now keeps MMQ's buffers; its covered layers still take the fused kernels. UD-IQ4_XS 8K
retrieval after the fix: 3/3 correct, 837.9 prompt tok/s. (The CUDA path shares this code.)
#531's peer-aware lookup indexed the residency table with the 512-expert constant NE instead of the model's own
count. The Coder has 256 experts per layer, so from its second layer a resident expert was looked up in the wrong row,
and a prompt long enough for the streamed walk stopped with "an expert is neither resident, streamed nor on the peer"
(found by the 57K real-use check). The 512-expert models index the same row as before.
30 more rows from tools/hip/tune_hipblaslt.cpp on Aurora (90 in all). IQ3_XXS, 64K prompt, the
gfx11 fused experts: --prefill 16384 with them 849.2 / 847.0 tok/s vs 8192 803.1 / 824.0 / 829.3.
Setup no longer writes a fixed 300 MiB reserve for cards under 8 GB: the engine's own rule (lower the reserve only as
far as the smallest working cache needs, down to 300) chooses better, and a fixed 300 left an emulated small card with
5 MiB free. Setup keeps the tip (an 8K context, --draft-vocab en). When the engine lowered the reserve and the card ends
below the 256 MiB LOW line, it prints one warning (requests may stall; close other programs that use the GPU, or lower
--max-context) instead of advising a larger reserve, which would leave the cache no room.
peer_experts.cpp calls cudaDeviceCanAccessPeer, cudaDeviceEnablePeerAccess and cudaMemcpyPeerAsync and tests
cudaErrorPeerAccessAlreadyEnabled; the HIP shim had none of them, so the Windows HIP build stopped there. Mapped to
their hip* names.
On an RTX 3060 Laptop (6 GB) the weights, the KV cache and the draft layer left 0.29 GiB, the default 700 MiB reserve
and the draft head took it, the expert cache got 0 slots and the verify window refused to start.
- setup: a single card under 8 GB (nvidia-smi below 7.5 GB; an 8 GB card lists 7.99) gets --vram-reserve-mib 300
unless one is given (images on the GPU keep theirs), marked in the config as setup's own choice so a later setup
decides again. It prints what frees more VRAM if the start still stops: an 8K context, --draft-vocab en, other
programs closed. The draft layer stays (the server needs --mtp and --spec). 8 GB and larger cards: the same config
(golden test).
- engine: a cache too small to lend the prompt path a 256-token chunk (its buffers plus the 128-slot floor; one slot
without --prefill) makes the prompt path allocate its own on top. When the default reserve leaves less than that,
the reserve shrinks to what leaves exactly that cache, down to 300 MiB (what comes after the cache is paid from
the reserve: at 300 an emulated small card ended with 5 MiB free, LOW by the serve check). With no slot at all the start stops with how many
MiB are short, what the session and the draft head take, and the options. A start whose cache could lend that
chunk is sized as before. Checked with a VRAM ballast on the 5070 (IQ3_XXS, 32K): 258 slots needed, 0.71 GiB free
-> "about 333 MiB more", and with --vram-reserve-mib 700 given, "733 MiB more" plus the reserve tip.
- verify's refusal points at that line.
From PR #409 (af404df), only device_free_bytes(): on an integrated GPU (cudaDevAttrIntegrated, on HIP
hipDeviceAttributeIntegrated; the Ryzen AI Max 8060S) the expert cache sizing and checks count
/proc/meminfo MemAvailable less STRATA_UMA_HEADROOM_GIB (6) when that is more than cudaMemGetInfo:
the page cache of experts read from the GGUF in place gives way to a device allocation. The PR's
aarch64 / DGX Spark CPU parts are not taken.
From PR #313 (1e42cf3): src/prefill/wmma_gemm.cu/.h, the Gemm::f16/bf16 hooks before hipBLASLt, the
build macro STRATA_WMMA_GFX11 and its parity test; not its prompt attention (STRATA_HIP_WMMA covers it).
Fix: the intrinsics guard named only gfx1100/1101/1102 while the runtime gate takes any gfx11, so on
gfx1150/gfx1151 mma() compiled to "return c" (silently wrong); it now names gfx1150/gfx1151 too.
hip_prefill_wmma_gemm_parity on gfx1151: 3520 cases pass (f16 max_abs 2.4e-7, bf16 0).
In the streamed walk every expert put a cross-stream wait (its copy) and an event record (its ring slot back) on the
compute stream around a ~3 us gather. Under WDDM each pair left ~10 us of GPU idle even when the copy had long
landed: an nsys trace of a 32K prompt showed 226 ms of idle between back-to-back gathers while the host was 3-11 ms
ahead of the GPU. Now a group of 16 is gathered by one launch (copy16_group_kernel) after one wait on its last
streamed copy - the copy stream is in order - and one event releases all of its ring slots: the issuer waits on
used[used_of[slot]]. A ring entry the routing skipped inside an open group first gathers what the group holds so
far, so no more than a group's entries are ever held back from the issuer. The same bytes in the same group slots.
STRATA_PREFILL_GROUP_GATHER=0: one at a time, as before.
IQ2_XS, RTX 5090, 262K context: 32K prompt -8.6%, 8K -10%, 2K -12.6%; first-token logits bit-identical at 2K, 8K,
16K and 32K and with a 96-slot ring.
The switch was read with getenv on every launch_multi call, which runs per layer when decode is not captured in a
graph. It is now read once, like the engine's other STRATA_* switches.
STRATA_GR_DOWN_MAX4=1: launches of at most four tokens run the GR down-projection kernels (plain, split and staged)
with per-thread storage for four tokens instead of kFusedGrMaxT, keeping the block dimensions, the tile width, the
accumulation order and the plain/split/staged choice: the same bits. Off by default. RTX PRO 6000 Blackwell, IQ3_S,
MTP 4, 64K: decode +2.1% (code) / +2.4% (prose) with identical output tokens.
From #443; its one-warp MMVQ experiment (STRATA_MMVQ_WARP1, not bitwise), the roofline benches and the benchmark
evidence files are left out.
fused_ring() turns the fused ring sizing off for any --peer-device session (core::peer_portable), but the layer's
fused choice tested the peer's prompt-path state (m.pp), which is null when set_peer declines (no P2P, the buffers do
not fit, --peer-prefill-rows 0). Both now follow the flag, so a peer session's prompt path is MMQ with the MMQ ring
and buffers it was sized for. Without --peer-device nothing changes.
A second visible CUDA GPU (N >= 1) holds an adaptive expert cache beside
CUDA0's, filled with the ranked pairs the primary does not hold. Activations
and result rows cross NVLink or another P2P path; the peer also computes its
experts' rows for prompt chunks (MMQ; the fused path is not combined yet).
Knobs: --peer-device N, --peer-reserve-mib, --peer-slots, --peer-adapt-swaps,
--peer-prefill-rows; sizing stays manual until #216's buffer lending lands.
Default output unchanged: without the flag byte-identical to v0.1.36, with it
byte-identical under the exactness gate. cudaHostAllocPortable only with a
peer. docs/SECOND_GPU.md documents the flags and refused combinations.
Measured on 2x RTX 3090 + NVLink, IQ3_S: prefill +15/+25/+25 %, decode +19/+14/+16 %.
E-9 (Prefill::draft_kv) declines when the drafter's KV is a ring, which it is
with KV streaming (--kv-resident), so every chunk falls back to the E-4 pass in
MtpDrafter::prefill. On gfx1201 / ROCm 7.2.4 that queue (a graph launch and
four device copies per group, ~2,000 groups per 8192-token chunk, one sync)
sometimes never completes: the prompt hangs in hipGraphLaunch or the final sync
until the watchdog ends the engine.
HIP builds now use the per-group loop unless STRATA_MTP_PREFILL_SYNC=0; CUDA is
unchanged. The variable now reads its value (any value, 0 included, used to
select the old loop).
R9700, full IQ3_XXS, --kv-resident 32768, 131K context, mixed load: 0.1.31
hung in round 1 on 2 of 2 tries; this change ran 30 of 30 rounds clean with no
measurable prompt cost.
Calibrated on a Radeon AI PRO R9700 with tune_hipblaslt (24 of the gfx1100
table's 26 shapes). Setup only picks it when the card is gfx1201 and the
installed hipBLASLt is exactly 100202.
With that hipBLASLt the plain hipBLAS path is far off: on the full GSQ-RCO
IQ3_XXS (KV streaming, engine 0.1.31) a fresh 32K prompt reads at 1,177 tok/s
with the table against 638 without it, and a 7K prompt at 1,164 against 656
(median of 4 warm runs per arm, arms alternated); decode is unchanged.
hip_prefill_hipblaslt_gemm passes with it. docs/AMD_HIP.md notes that the
1.4.1 result (0-3%) does not carry over to 1.2.2.
STRATA_NO_LARGEPAGES is documented as a same-run A/B switch for the large-page path, but reserve() only read it in the Windows branch; on Linux the MAP_HUGETLB path ignored it, so the switch silently did nothing. Honor it there too.
The failure message also said "no hugetlb pool configured?" even when a pool exists but is smaller than the mapping: MAP_HUGETLB is all-or-nothing, so a 34 GiB pool cannot serve a 40 GiB arena and the whole mapping falls back to 4 KiB pages. Print the needed 2 MiB page count against vm.nr_hugepages (the Windows branch already names its byte count and error codes).
setup's low-RAM packs have an experts.bin, and with it 0.1.31's tier mapped it: on an emulated 32 GB PC the mapped
reads took the RAM the budget was sized for, and available RAM fell below 1 GB (#286, measured yesterday).
- set_unbuffered also takes experts.bin (paths_ is the one file); read_direct reads one window per blob (a blob is
contiguous there) and merges near ones as for the GGUF.
- blob() hands out the stage buffers for any unbuffered source (staged()), so prefetch, begin_layer and transient
work as with the GGUF in place; the mapped experts.bin path is unchanged otherwise.
IQ2_XS pack with experts.bin, emulated 32 GB PC + 16 GB GPU, 16 GiB budget, 8K prompt + 256 tokens, two runs:
decode 100.2 / 100.6 tok/s, done at 17.2 / 15.5 s, the prompt path's host copies 2.1 s (the GGUF in place: 84-89
tok/s, 21.5-22.1 s, 4.3 s). Greedy tokens identical unbuffered, through the cache and on 0.1.31.
With --resident-budget-gib on a PC whose RAM cannot hold the expert files beside the budget, every expert outside
the budget comes from the drive anyway. Through the mapped GGUF that costs twice:
- the page faults are small requests on the critical path (the prompt path's stager, the profile fill);
- the pages they bring in take the RAM the budget was sized for. On an emulated 32 GB PC, the 16 GiB budget was
clamped to 11.65 GiB after the profile fill.
- FileExpertSource::set_unbuffered: chosen by experts_unbuffered (#357) with the budget as the RAM it keeps: whenever
the files could not be kept beside it, cached now or not - a partly cached file's mapped pages would still land in
the working set (seen on an emulated 64 GB PC: the budget shrank, then RAM ran out). Each shard is opened
FILE_FLAG_NO_BUFFERING | OVERLAPPED.
- copy_from_files, the decode prefetch and the complement copy read every role window of a batch at once, then
copy it into place.
- Windows that are less than 1 MiB apart merge into requests of up to 32 MiB. NTFS runs a file's unbuffered reads
one at a time while the file is mapped anywhere, and the engine always maps the shards (token embedding, PLE).
Measured on a PCIe 5 drive at queue depth 48: 512 KiB reads give 10.4 GB/s on an unmapped file and 3.4 GB/s on a
mapped one; 8 MiB reads give 8.6 GB/s.
- The profile fill reads its pairs in batches of 64, the next batch while this one is copied into the slots.
- warms() is false when unbuffered. A PrefetchVirtualMemory of the mapping would read the experts through the cache,
a second time.
- Otherwise nothing changes: the mapped path stays for a keepable set, for no budget, and off Windows.
STRATA_UNBUFFERED_LOAD=1/0 forces the choice.
Greedy tokens are identical with the reads unbuffered, through the cache, and on 0.1.31.
The definition gained a defaulted parameter while the header kept the four-parameter declaration. The engine's
caller is in the same file, so it built, but tests/core/expert_layout_test.cpp calls it through the header and
failed to link (LNK2019, unresolved load_experts_gguf). The default now lives in the declaration;
expert_layout_test passes.
A buffered read copies every expert through the file cache. That pays off only when the cache can keep the files
for the next start. On a PC whose RAM cannot hold them beside the arena (a 64 GB PC with IQ2_XS: 24 GiB left for
36.5 GiB of files), every start reads the drive anyway, and the copy is pure cost.
- experts_unbuffered() (pinned.cu): timed random 64 KiB reads tell whether the files are in the cache already
(a cached read takes ~10 us, the drive ~80). Unbuffered is chosen only when they are not and the RAM left beside
the arena could not keep them either. A warm restart that 0.1.31 serves from the cache stays on its FILE* path.
STRATA_UNBUFFERED_LOAD=1/0 forces the choice; the log names the reader and why.
- load_experts_gguf: each chunk's 4 KiB-aligned window is read with FILE_FLAG_NO_BUFFERING into an aligned buffer and
scattered into the blobs, with a handle per role file as expert_gguf_file() resolves it (split shards). 16 readers.
- load_experts_direct (pinned.cu): a pack's experts.bin ranges are read unbuffered straight into the arena. When the
ranges are not 4 KiB-aligned, it falls back to load_experts_ranges.
IQ2_XS without experts.bin, RTX 5090, emulated 64 GB PC (a locked-pages RAM ballast leaves 58 GiB), two rounds:
| | first start: arena / exit | restart: arena / exit |
|---|---|---|
| 0.1.31 | 22.1-25.2 s / 28.2-31.0 s | 12.5-16.0 s / 17.5-20.8 s |
| this | 18.3-20.2 s / 23.1-25.4 s | 6.0-6.1 s / 9.8-9.9 s |
With 128 GB the probe finds the files cached (15 of 16 reads) and the buffered path runs as before. Greedy tokens are
identical unbuffered, buffered and on 0.1.31 main.
--adapt-decay F sets the factor the routing counts are multiplied by after each adaptation (0.7, as before; the same
float, so the default is bit for bit 0.1.37's). #477's routing heat is added before the decay, as before.
From #407; its --adapt-tuned (opt-in re-ranking every 2 rounds, 192 swaps, x0.92 for caches holding 20-60% of the
experts) is left out of this release.
- The engine's `avx2` switch is on for AVX-512 CPUs too, so the widened gate sent an AVX-512 CPU's IQ4_XS gate/up rows
to the new AVX-2 kernel instead of ggml-cpu: other rounding on CPUs that #415 did not mean to change. A format with
only an AVX-2 kernel now takes it on CPUs without AVX-512; everything else is dispatched as before.
- native_expert_parity ran its AVX-512 check for IQ4_XS on an AVX-512 CPU: iq512_rows has no case for it, so the
rows stayed zero and the check failed. Each check runs only for the formats its kernel implements; the AVX-2 one
runs on AVX-512 CPUs too, so this PC tests the new kernel.
IQ4_XS (23) was missing from iq256_supported, so on an AVX-2-only CPU its gate/up expert rows
fell back to ggml's per-token vec_dot while every other i-quant format got the multi-token
kernel that decodes the weights once for the whole verify window.
Fmt32<23> reuses the 16-value codebook IQ4_NL already has. Each of the eight 32-value sub-blocks
carries its own 6-bit scale (two nibbles of scales_l plus two bits of scales_h), used signed as
(ls - 32): the scale folds into the int16 operand of madd_epi16 instead of becoming a float
multiply per sub-block, and the codebook sign is carried the way iq4nl_rows does it (|w| as the
unsigned maddubs operand, w's sign applied to the activation).
The gate in native_expert.cpp and in native_expert_parity cannot be iq512_supported alone:
IQ4_XS has an AVX-2 kernel and no AVX-512 one, so on an AVX-512 CPU the widened gate must still
pick each kernel only for the formats it implements - falling through an empty switch would
leave ff unwritten rather than falling back to ggml-cpu. Widening the parity gate is what makes
the format tested at all: with iq512_supported alone the tool skipped it on every CPU.
The cherry-picked change kept the old `if (wait) cudaEventSynchronize` above its own (a second, no-op wait), and its
serve-mode call site sat four columns left of its block. The same code otherwise.
A native pack leaves the per-token loop at `if (native_pack) { spec_pos = pos;
break; }`, before the site that dumps a row per position. The header was
written at open anyway, so the file claimed 64 or 66 rows and contained none -
and a reader that trusted it reported "0 differing positions" from a file with
no positions in it, which is the one answer that looks like a pass.
The header is now written with the FIRST row. A run that dumps nothing leaves
an empty file, which is what actually happened, and a reader can treat size
zero as "this mode never dumps" instead of as agreement.
Verified on the native pack: the file is now 0 bytes where it was 8. The
non-native path is unchanged apart from WHEN the header lands; it was not
exercised here because no non-native model is set up on this machine.
(cherry picked from commit b87d7dcfe9a0754c0a2b8b9776b7f2ec8980b6a1)
Greedy decode was not reproducible on this box. Repeated identical requests
diverged from about the 28th generated character, and the drafter, the expert
routing and the first verify window were all byte-stable, which left the
adaptive tier as the only thing left that could move a result.
It is. `apply_pending(false)` polls the swap's event with a NON-BLOCKING
cudaEventQuery, so whether the experts an adapt round just copied in were
resident for the next window depended on whether their PCIe copies had landed.
Resident runs the GPU kernel; non-resident is computed on the CPU, and on a
machine without AVX-512 those two kernels round differently enough to move a
top logit by 0.26 - measured at the window right after the swap was issued.
From there the trajectory is a different one, permanently.
Waiting is the fix. The copies were issued during the previous window's
drafting, so the wait is short, and in exchange the residency table is a
function of what was actually copied rather than of how fast the disk was.
With the default --adapt-every 4 left in place, five runs are now byte-identical
across all 24 verify windows and emit the identical 40-token stream; before the
fix three of five agreed and two forked at window 16. --adapt-every 0 (frozen
placement) was already deterministic, which is what identified the tier.
The STRATA_TRACE_ADAPT lines added with the probe are env-gated and off by
default: they report whether a round considered, skipped, or swapped, and
whether the previous round's copies had landed.
No cost measurement is claimed here: this says the trajectory is a function of
the inputs again, not what the wait does to decode throughput.
(cherry picked from commit 1e3888a73b69a332585056af5b86643ff8367242)
native_expert_grouped launched a block row per possible group (cap = window tokens x 10) for gate/up and down,
and SwiGLU and the q8_1 quantization as two kernels over all cap entries. The verify window's PCIe call has no
group at all at pcie_frac 0 and usually none otherwise, yet every layer paid ~17k blocks that started only to
return plus two passes over entries that are not its own.
- gate/up and down stride over the groups (rows y, y + gridDim.y, ...); the PCIe call runs 4 block rows
(kPcieGroupRows), the VRAM call keeps one per possible group. Each group's rows are computed by the same warp
code in the same order, so the results do not depend on the grid.
- SwiGLU + q8_1 in one kernel over the call's own entries [grp_start[0], grp_start[n]); the product is rounded on
its own (__fmul_rn; on HIP a plain product under contract(off)) before the block sum, as when it went through
memory, and the quantizer body is quantize_q8_1_kernel's (factored out, its PTX unchanged).
- STRATA_GROUPED_V1=1 keeps the old launches for A/B timing; native_grouped_parity checks old vs new bitwise
(every gate/up x down format, 0..cap groups, entries from 0 and past 0, grid_groups 1..4 and > cap).
The engine takes the bound on Turing only, so on other cards the test compared the capacity rule with itself.
STRATA_TOPK_ACTIVE_ANY=1 (set by the test) takes the Turing dispatch on any card; the kernels are the same on every
architecture. The CMake block moves above the qsa_select_bench comment it had split.
With --max-context 262144 the QSA score rows have a stride of 65538 blocks; a prefill batch at 131K actual context
fits the register top-k kernel's 33792-block limit, but CUDA dispatch tested the allocated capacity and chose the
reference kernel. The supplied active_blocks bound is now used on sm_75 only; other CUDA cards keep the capacity
rule, HIP its own policy, and decode (no bound) is unchanged. STRATA_TOPK_CAPACITY_GUARD restores the capacity rule.
qsa_topk_active_parity: 16 cases at stride 65538, selected ids bitwise.
Squashed from #512 (its commit and its merge of upstream 0.1.36).
Co-Authored-By: Codex <[email protected]>
PleReader's row_bytes has been a runtime parameter since the FP8 table
(160 B rows) needed it -- not 90-specific anywhere in the actual read/
cache/job-building code (RowCache, Job/Use, the straddle-length formula
in issue()). ngram.cpp's "Q5_0 PLE requires --ple-io mmap" refusal
predates that generalization (or was just never revisited): FP8's 160 B
rows already go through PleIo::Direct with no such gate.
ple_reader_test --selftest now runs its full synthetic suite (decode-
shaped tickets, page-straddling rows, dedup/edges, bulk, two in-flight
tickets, fault injection, keep-alive -- both threaded and caller-thread
modes) at row_bytes=110 as well as the production default 90, before
touching the refusal. Both: OK.
ple_reader_test --gguf against the real OrcaRouter Q4_K_M shard (the
same one from the Q5_0 kernel PR): direct vs mmap over 312 real PLE
tickets -- bit-identical, and direct is ~8.6x faster (461 vs 3959 us/
token) on this SSD, matching the ~2-2.6ms/token mmap cost the original
"sixteen serial page faults" comment in gather() measured for the
IQ4_NL table.
End to end on the running model (engine 0.1.32, RTX 3090): same text/
math/vision outputs as --ple-io mmap (deterministic sampling), with a
consistent decode speedup (~9-12% across three real generations) and
image-prompt PLE throughput up from not practical to measure to 185
tok/s -- the full per-token win is smaller than the isolated 8.6x since
PLE is one part of a token's cost, not all of it.
is_iq()/the GU/D/MMVQ format lists had Q4_K, Q5_K, Q5_1 and Q8_0 already,
but not Q5_0 -- a plain `llama-quantize` "_M"/"_S" tier GGUF (not Unsloth's
dynamic method) commonly puts Q5_0 on down-projections, sometimes gate/up
too, and native_expert_supported() refused those layers outright ("this
engine has no GPU kernels for").
vec_dot_q5_0_q8_1/dq_q5_0 port vecdotq.cuh's integer chain and the
dequantizer verbatim (symmetric format, so -- unlike Q5_1 -- there's no
min-term rounding choice to make). Fmt<6> plus the row-bytes/is_iq/
dq_dispatch entries follow the existing Q5_1 (type 7) pattern.
native_expert_parity --synthetic q4_K/q5_0 and q5_0/q8_0 (both now in the
CMake test matrix): 0 failures, GPU dequant bit-exact against to_float,
gate/up and down rows bit-exact against ggml's own vec_dot. q5_0/q5_0
(Q5_0 in both roles) is left out of the matrix -- stacking the same
lower-precision format on both roles pushes the GPU path to ~3.6% rel
error against the float reference, just over this test's 3% threshold;
that's compounded quantization noise (CPU rel error is the same ~1.25%
baseline as q5_0/q8_0), not a kernel correctness issue, and no quantizer
I've seen actually picks Q5_0 for both roles of the same expert.
Verified end to end on a real model: a community Q4_K_M GGUF (gate/up
Q4_K, down Q5_0 on several layers) that previously failed at layer 6
now loads and serves correctly on an RTX 3090 (engine 0.1.32).
gdn_rec_cols_pipe_kernel is bound by shared memory, not by arithmetic: every thread of a warp needs the same
32 q and k values per token (the k twice), and a warp receives one such broadcast value per clock. The new
kernel runs column c of the three value heads that share a key head (head % 16) in one thread, so every q/k
value it loads feeds three heads and the token's k row is loaded into registers once for both of its uses.
Its inputs come in blocks of 8 tokens, copied to shared memory by cp.async while the block before computes,
and a token needs 2 __syncthreads instead of 5. 64 blocks instead of 192. Per value head and column the same
arithmetic in the same order: gdn_rec_parity checks output, state and FP16 output bit for bit (T from 1 to
4099) and times it: 18.8 -> 13.0 ms per layer at 32768 tokens.
The output norm stored the normalized value in FP32 as well, which nothing reads (the out projection takes
the FP16 copy): 4.65 -> 3.05 ms per layer at 32768 tokens.
Together 845 -> 579 ms per 32768-token block over the 36 DeltaNet layers; in the engine +2.3 % prefill
tokens/s on 32000-token prompts and +2.2 % on 64000-token prompts, the same greedy answers. Measured on 0.1.32
on a 4080 SUPER (80 SMs); on 0.1.31 the same change gave 1.42x, 1.51x and +2.1 % / +2.0 %. CUDA sm_80+
(cp.async) on a card that holds all 64 blocks at once; otherwise, on Turing and with STRATA_GDN_KEYHEAD=0 the
kernel as before.
Squashed from #413 (5 commits: the kernel, gdn_rec_parity and its --bench SM bands; the 48-63 SM exception
added and taken back).
On Turing mode 4 would run as pairs of m16n8k8 MMAs, which no parity run has checked yet; RTX 20 cards keep the old
kernel for --kv q4_0 until one does.
The tensor-core prompt attention (D-1) refused Q4_0 pools, so --kv q4_0 read every prompt through the FP32 per-query
kernel. Mode 4 is mode 1 (int8) with a scale per 32 values: each q4_0 block's 16 code bytes become 32 exact int8
codes (code - 8) at gather, its fp16 scale goes to the per-group scales, which are applied in FP32 as int8's are
(q.k per 32-dim group; p.v folds each group's scale into p, two groups per warp). The p.v fold is bounded by the
chunk's largest |scale|: q4_0 scales are signed (ggml's d = max / -8). The caller's rotation of q and of the output
is unchanged. STRATA_PROMPT_ATTN_Q4=0: the old kernel (A/B).
qsa_prompt_attn_parity gets Q4_0 cases with signed scales spread over two decades, as a pool holds them (32K and
1.5K contexts), against FP64 and the old kernel.
With KV streaming (--kv-resident) the MTP drafter's K/V is a ring of its window (kv_mode 2: page p in slot p % n_slots
over a host copy), and draft_kv refused it, so every prompt chunk fell back to the drafter's own pass - one graph per
few rows. The batched path takes the ring with its own page table and host copy (as a streamed main layer appends);
the cells it writes are those the window can still reach, which the ring holds, and one batch never spans more than
the ring's slots, so no two cells of a batch share a slot. STRATA_MTP_BATCH_RING=0: the drafter's own pass (A/B).
STRATA_DRAFT_TIMING=1 (debug): the pass's time, synced.
Stager::work waited for a buffer's DMA only from the generation's kRing-th job on (j >= kRing); a generation's first
kRing jobs wrote their buffers straight away. The previous generation's last DMAs from those buffers are only
known to be done where something else waits for them: the routing sync before a staged layer's start(), or the
sync at the end of a chunk with an MTP drafter or stages. A stream_all chunk that ends without one (no --mtp), or a
ring entry the routing skipped (its copy is never waited for), leaves a window where the next chunk's first jobs
overwrite a pinned buffer a DMA may still read. The event is now waited for on every job; an event never recorded,
or already complete, returns at once.
IQ2_XS with --mmap-experts --resident-budget-gib 16 (12,144 of 48,668 streamed experts through the stager): the
same tokens on a 32K prompt, prefill 6,491 vs 6,546 ms.
From an audit: the upload of a chunk's rows, the event recorded after it and the wait on the other buffer's event
before the SSD thread refills it were unchecked, so a failed record or wait let the thread write a buffer an
upload might still read. A failure now ends the chunk with the CUDA error.
Layer 1 adds the per-layer embedding to every token of a chunk, so its rows must be in. A later chunk's rows were
already read on a thread during the chunk before; the first chunk's were read synchronously before layer 0, so the
GPU idled while the SSD read them (an nsys trace: ~245 ms of a single-chunk 32K prompt). Now the first chunk's
gather starts on a thread too and lands just before layer 1 (ple_land), which also starts the next chunk's.
The reads themselves went 64 at a time: a 32K prompt is 512,720 rows in 213,358 4 KiB reads, ~620K IOPS at queue
depth 64. --ple-inflight now defaults to 256, where the drive saturates (1024 is no faster). Decode reads 16 rows a
token and is unchanged. The same rows: the same results.
IQ2_XS, RTX 5090: a 32K prompt 5986 / 6013 -> 5843 / 5808 ms, 8K 1714 -> 1624 ms; first-token logits bit-identical.
In the streamed walk every expert put a cross-stream wait (its copy) and an event record (its ring slot back) on the
compute stream around a ~3 us gather. Under WDDM each pair left ~10 us of GPU idle even when the copy had long
landed: an nsys trace of a 32K prompt showed 226 ms of idle between back-to-back gathers while the host was 3-11 ms
ahead of the GPU. Now a group of 16 is gathered by one launch (copy16_group_kernel) after one wait on its last
streamed copy - the copy stream is in order - and one event releases all of its ring slots: the issuer waits on
used[used_of[slot]]. A ring entry the routing skipped inside an open group first gathers what the group holds so
far, so no more than a group's entries are ever held back from the issuer. The same bytes in the same group slots.
STRATA_PREFILL_GROUP_GATHER=0: one at a time, as before.
IQ2_XS, RTX 5090, 262K context: 32K prompt -8.6%, 8K -10%, 2K -12.6%; first-token logits bit-identical at 2K, 8K,
16K and 32K and with a 96-slot ring.
- engine (serve mode): GEN key logprobs=K (0..20) writes "LP logprob id:lp ..."
after each "T id", from the verify window's head logits (Verifier::copy_logits;
before temperature, penalties and sampling); absent key, no LP lines
- server_intel.py: pairs LP lines with their tokens (StrataEngine._pump), passes
logprobs/top_logprobs to the engine, and fills OpenAI's choices[].logprobs
(content / reasoning_content) in streamed chunks and the collected reply; the
stop token is not listed. serve/ itself is unchanged.
Tested: a fake engine on hardac (streamed, not streamed, without logprobs,
stop token excluded); on the B70 with the IQ2_XS: same answer with and without,
"Paris" -0.0019 with top-1 equal to the chosen token, no measurable time cost.
expert_w11_kernel: moe_fused.cu's design (GPU grouping, experts read in place, gate/up + SwiGLU +
int8 H, then down) on v_wmma_i32_16x16x16_iu8. A work item = 128 weight rows x a 64-row tile, 8
waves of 2x2 16x16 tiles; operands straight from global one block ahead, weight scales in LDS.
The activations take gfx11's K order (act_pos); per 32-value half the integer dot is exact and the
epilogue is the CUDA kernel's. hip_prefill_fused_moe (tests/cuda/prefill_fused_moe_test.cpp) on
gfx1151: vs FP32 reference rel RMS 1.070e-2 (= MMQ's), vs MMQ 3.0e-5; one 2048-token layer 23.2 ms
(MMQ path) -> 11.3 ms. The native packs' fused kernels stay CUDA-only (stubs on HIP).
Replaces the per-head kernel of the previous commit (VGPR-bound, spilled). Same column-split grid
as gdn_rec_cols_pipe_kernel (4 x 32-column blocks per head), but lane g of a column's four holds
exactly the old thread's row group g; kv / o partials meet through wave shuffles and are added in
the old order, so no __syncthreads per token; q/k/v/gate/beta staged 4 tokens at a time (~10 KB
LDS: all 192 blocks resident). The output norm runs the old kernel's body over a grid-stride walk
(T x 48 blocks of 128 threads before). tests/hip/gdn_rec_head.cpp: y, y16 and state bitwise equal
to the old kernels for T 1..2051 (variant 3 = new recurrence + old norm, a diagnostic); 2048 tokens
9.8 -> 4.1 ms on gfx1151 at the test's 600-880 MHz.
One block per value head; each column's 128 state rows in the registers of two lanes of one wave
(the 32-row group partials swapped within the wave and added in the old order); q/k/v/gate/beta
staged in LDS 8 tokens at a time with the next chunk prefetched; the RMS norm + gate of
gdn_out_norm_kernel fused per chunk. tests/hip/gdn_rec_head.cpp: y, y16 and the state bitwise
equal to the column-split + norm kernels for T 1..2051; 2048 tokens 9.8 -> 6.7 ms on gfx1151
(at the 600-880 MHz the GPU's DPM runs this test at). STRATA_GDN_HEAD=0 keeps the old kernels.
gfx11's bf16 WMMA layout: 16 k per lane, C rows 2i + lane/16. On gfx1151 (qsa_select_bench, 131624
capacity): ctx 32768 x 2048 queries scores 14.2 -> 3.5 ms (4.1x), selections identical 2047/2048;
ctx 65536 x 8192 queries 110.1 -> 25.9 ms (4.3x), identical 8192/8192. Still opt-in, as on gfx12.
All 15 prompt GEMM shapes of Qwen3.8-Flash-Next at chunks 2048 and 8192 (T = 2047/2048/8191/8192),
tuned on Aurora (Ryzen AI Max+ 395) with tools/hip/tune_hipblaslt.cpp.
One dpct run of upstream main + the seven PRs, 3-way merged into the files they
touch (sycl/tools/merge_upstream.py); their header changes are forked into
sycl/include until upstream merges them.
- #385 stager waits for a buffer's previous DMA on every job (port: polled seq)
- #463 decode waits for an adaptive swap before reading residency
- #453 drafter's batched K/V on the KV-streaming ring
- #374 first chunk's PLE rows read beside layer 0; port fixes: the next read
starts at c0 + T (the port's first chunk is shorter), and the upload is
marked by a polled sequence number instead of a host event wait, which
deadlocked prompts past ~4K tokens under the Level Zero v2 adapter
- #363 group stride + fused SwiGLU/q8_1 for the PCIe expert call, re-applied
to the port's lane kernels (fp contract off, as upstream's HIP form)
- #407 --adapt-tuned (opt-in), #413 key-head DeltaNet recurrence (opt-in via
STRATA_GDN_KEYHEAD=1: 8% slower on the B70; its NVIDIA gate and SM-holding
parity bench not ported)
Tested on the B70: native_grouped_parity, iq_multi, gr, qsa, kv_hybrid,
quantize_act, qsa_prompt_attn parity pass; Coder 19/2,184-token, IQ2_XS and
40K outputs identical to 0.1.33; Coder 2,184-token prompt 784 -> 825 tok/s;
128K int8 KV streaming 895 / 67.0 tok/s; IQ2_XS decode +1.7% (#363).
Co-authored-by: sergqwer, constantindjonkam, architectds, BlueKingMuch (upstream PR authors)
Port of the Aurora experiment (mmq-resident-rowcount-sorting-v1, +13% 8K prompt on gfx1151 over
0.1.30 + Lt, identical output ids there): each 16-expert MMQ group's max_rows is close to its
experts' own row counts, so less tile padding. Host-only; slot/src/off rebuilt before the upload.
gfx11's v_wmma_f32_16x16x16_f16 takes all 16 k per lane (lanes 16..31 repeat 0..15) and
returns C rows 2i + lane/16; the kernel loads its fragments in that layout. Opt-in as on gfx12.
device, expert_source, native_head, prefill gemm, prefill.cpp, generate.cpp
merged from dpct of 8554b6b and 5b734e4 with sycl/tools/merge_upstream.py (the
3-way merge restricted to files upstream changed, dpct kernel-name hashes
canonicalized to the port's). Version 0.1.35.
The fill from the GGUF (--stream-experts) read each expert's three slices,
copied the blob and waited for the copy before the next read: ~12,000-18,500
experts with neither the SSD nor PCIe kept busy. Now the slots are admitted
in profile order first (same placement), reads go in file order by up to 8
threads into page-locked batches of 64, and a batch's copies overlap the next
batch's reads. STRATA_FILL_SERIAL=1 keeps the serial fill.
Tested on the B70: cold page cache, Coder fill 23.4 GiB 76.2 s -> 18.8 s;
launch to first token Coder 82 s -> 26 s, IQ2_XS 120 s -> 41 s; outputs
identical to the serial fill.
- gguf_expert_source: ExpertLayout::gguf_file is per layer and role
(3 * layer + role) since upstream 0.1.31; the port indexed it by layer, so
experts in shard 2 were read from shard 1 at shard-2 offsets (Swift 1.5:
layers 13-47, NaN logits from layer 13). One fd per layer and role now.
- iq_kernels: vec_dot_iq2_xs and Split<17> took their 16-bit codes through
a uint16_t* to an int2 (type punning the device compiler does not honour);
by shifts now. iq_multi_parity checks IQ2_XS against a CPU decode from
ggml's tables (iq2xs_host_tables.inc) and passes.
- prefill: STRATA_DBG_NAN=1 also reports the experts' fp16 GEMM inputs.
Tested on the B70: Swift 1.5 IQ2_XS answers correctly (48 tok/s raw, short
thinking); the original IQ2_XS output is unchanged (65109a, 58 tok/s).
expert_cache.cpp, expert_source.cpp, fused_gr, qsa_prompt_attn, gr_parity and
generate.cpp merged from a fresh dpct of 2ca1b66 and 8554b6b; files that only
differed by dpct run-to-run noise left alone. Conflicts: --resident-cpu-experts
(upstream's resident_cpu_explicit kept with --stream-experts) and the prompt
attention's capability check (the port's XMX dispatch kept). Version 0.1.33.
Tested on the B70: build clean; gr_parity and qsa_prompt_attn_parity pass;
Coder short/long, IQ2_XS and the 40K prompt (borrowing) outputs identical,
78.3 / 75.7 / 58.0 / 68.9 tok/s. setup_intel.py --check and an install run, and
server_intel.py with the mock engine, against the new setup.py / server.py.
The hang was in the streamed-expert prompt path, which borrowing is the only
way to reach on the Coder: the stager threads waited on the copy queue's
events to reuse their 16 page-locked buffers, and under the Level Zero v2
adapter that left another queue's barrier on the same event waiting forever
once the ring wrapped. The copy queue now writes a sequence number into
page-locked memory after each DMA and the stager polls it.
- prefill.cpp: Stager::issued_one / buffer_free; STRATA_PREFILL_SYNC=1 debug
switch (wait and log after every phase)
- generate.cpp: borrowing is the default above a 32K context again
Tested on the B70: 40K prompt with borrowing 1,117-1,144 tok/s, decode 69
(output identical to the run without borrowing); 128K int8/q4_0/k8v4 decode
66.6/62.8/66.8 (from 59.5/53.4/54.4), 256K 55.9/51.7/52.2 (from
48.7/43.2/42.3); Coder and IQ2_XS 32K outputs unchanged.
dpct of 6ac1a2c and 2ca1b66, normalized, kernel-name hashes canonicalized to
the port's, then git merge-file into sycl/: 49 files, 10 with real conflicts.
- verify: upstream's set_commit_async/wait_commit on the port's deferred
commit (wait_commit = commit_finish); copy_logits on the queue
- fused_gr: upstream's split/staged read variants merged but unused by the
port's read; their per-card self-test segfaults on the B70, so on SYCL it
runs only with STRATA_HC_CHECK=1. The port's split-norm kernel renamed
(upstream has one of the same name); two colliding kernel names renamed
- fixups.py: fused_gr's per-block local-memory query
- hand files: mtp.cpp, verify.cpp, ple_reader.cpp (keep-alive reads + trace)
- qsa_prompt_attn_parity turns on the XMX kernel it checks
- version 0.1.32
Tested on the B70: build clean; Coder (19 / 2,184-token prompts) and IQ2_XS
outputs identical to 0.1.31 at the same speed (78.1 / 75.7, 58.5 tok/s);
40K prompt 1,201 tok/s, decode 65.1; kv_hybrid, qsa_prompt_attn, qsa, gr,
kv_q4, kv_q8, rope, router, sampler, quantize_act parity pass.
Every file outside sycl/ (and docs/INTEL.md) is now upstream's, byte for byte.
The Intel pieces that used to edit shared files move under sycl/ and wrap
upstream from outside:
- sycl/setup_intel.py: runs setup.py's main() with the GPU check, engine
step, RAM rule and config writer swapped for the SYCL port's
- sycl/serve/server_intel.py: serve/server.py plus the Arc's Monitor
readings (sycl/serve/xe_telemetry.py) and the optional model menu
(sycl/serve/web/switcher.js)
- sycl/tools/gpustat.py (was tools/intel/), sycl/.gitignore; migrate.sh
writes its compilation database outside the checkout
- removed: the llama.cpp engine path (serve/engine_llama.py), the README box
Engine sources in sycl/ still track 0.1.31; the 0.1.32 kernels are not
migrated yet.
- fixups.py: default-queue memcpy/memset in the parity tests first waits for
all device queues (cudaMemcpy's legacy-stream semantics), applied to every
src/kernels/*_parity.cpp
- generate.cpp: the post-prompt refill loops sync the queued copies every 256
entries so the ring cannot wrap over in-flight slots
- kv_hybrid_parity: enable STRATA_PROMPT_ATTN_XMX (step 5 needs the XMX path)
Tested on the B70: build clean; qsa_parity, gr_parity OK.
Re-migrated (dpct of 0.1.29 and 0.1.31, 3-way merge per file): 45 files
changed upstream, 38 conflicts resolved; verify.cpp, mtp.cpp and the new
conversation_state.cpp ported by hand; 7 new upstream files.
Fixed on the way:
- the port's grouped expert kernels stay the default: upstream's new multi
kernels (8 rows a group) ran on the port's 32-row grid and wrote a quarter
of each expert (end-of-text output); STRATA_EXPERT_SPLIT=1 runs them with
their own grid. Lanes per row now selectable (STRATA_GU/DOWN_LANES)
- dpct's unparenthesised __fadd_rn: the native QSA indexer's per-token
pooled key (decode), the native QSA scores and the s2 activation rounding
replaced instead of added - from the first migration; fixups.py fixed
- doorbell waits: fixups.py bounds upstream's new spin spelling again
- prompt-slot borrowing hangs since upstream's rework: off by default
- the Turing PTX branch in qsa_prompt_attn, dp4a.hpp's __nanosleep, the
coupled-draft capture signature, a duplicate kernel name
- gr_parity: the single-token GR read runs the multi kernels (bitwise)
- qsa_parity: a cross-queue copy race in the test itself
Decode 78.2 / 75.7 tok/s (Coder), 58.6 / 64.2 (IQ2_XS); 128K-256K complete.
- sycl/include/dpct/LICENSE.TXT: the SYCLomatic helper headers' license
(Apache-2.0 WITH LLVM-exception; their SPDX notices already name it)
- sycl/third_party/ggml/LICENSE: ggml's MIT notice beside the migrated copy
- tools/intel/gpustat.py + gpustat.service: the root sampler the Monitor
tab reads load and VRAM from on xe (finds the card, sizes VRAM from its
BAR; GPUSTAT_PCI / GPUSTAT_VRAM_MB override)
- dpct reports and .yaml sidecars removed and ignored
- docs/INTEL.md: no references to a private test rig; the sampler and the
model_switcher setting documented
Conflicts: setup.py (upstream's reworked low-RAM/paging/rope logic kept, the
Intel backends guarded out of the experts-in-RAM rules as before; meta, the
CUDA/HIP build stamp, only read where the resident variant uses it) and
serve/server.py (upstream's /load, /unload and idle-unload beside the port's
/switcher and request logging). The SYCL copies are refreshed next.
- telemetry: on an Intel Arc (xe) the GPU readings come from a root
sampler's /run/gpustat.json (load, VRAM: xe exposes them to root only)
and sysfs (temperature, power, cap, the PCIe link the card trained at,
capped by the slot). NVML stays first; nothing changes on NVIDIA.
- a run config may name a model_switcher RPC (one model at a time on a
shared GPU): GET/POST /switcher report and request a swap, and the web
app shows a Model menu with the models served on this port; it waits
for /health to name the new model, then reloads. setup.py keeps the
setting when it rewrites a config.
k8v4 (KV in VRAM, no streaming): 995 tok/s prompt / 65.8 decode at 128K,
804 / 51.2 at 256K with 994 experts served from the host mirror.
API: 48-72 tok/s on a short chat answer, prompt included.
Summary 77.9 tok/s (19-token prompt) / 74.2 after 2,184 tokens; long
contexts with setup's KV streaming: 66.1 tok/s decode after 128K, 54.1
after 256K (all experts in VRAM). The 'not re-measured' caveat is gone.
- the GR norm ran one work-group per token; one per (token, stream) visits
every element in the same order, so rs and xn are bitwise unchanged
(gr_bench checksums equal), GR read 76.7 -> 71.4 us at 6 tokens
- the gate/up expert dots read their 2-byte-aligned weights as aligned
4-byte loads (page-safe) instead of 16-bit pairs; same ints
- grids in local memory (STRATA_GRID_SLM=1) measured slower: off
Decode 77.9 tok/s (19-token prompt), output tokens unchanged.
The expert down projection (IQ4_NL in 39 of 48 layers) was the largest
expert kernel: ggml reads four uint32 of the codebook from global memory
per lookup and each weight int as two 16-bit loads. Register constants
and aligned 4-byte loads (the third only when unaligned, so page-safe)
produce the same ints: 80.8 -> 43.1 us per call, output tokens identical.
-DSTRATA_IQ4NL_FAST=0 restores ggml's path.
The summary still described llama.cpp as the Intel engine at 23-25 tok/s.
It now compares NVIDIA, the SYCL port (setup's default once built) and
llama.cpp; the stale 'not ported'/'not done' items, setup's KV choice and
the served speed are corrected, and the long-context table is marked as
measured before the decode work.
A disconnect in the log now names the client that closed it. The
strata-sycl-dev image inherits a health check on port 8080 (llama.cpp's)
that marked the serving container unhealthy forever.
Each work-group owns a 128-column slice of all 324 rows, stages that
slice of xn once (<= 4 KB SLM) and streams the rows through it, 4 lanes
per row; a reduce kernel sums the 80 slice partials in a fixed order.
GR read 108.6 -> 76.5 us at 6 tokens. STRATA_GR_DOWN_SLICED=0 restores
the direct kernel.
Q8_0 and IQ4_NL blocks (34/18 bytes) are 2-byte aligned: the wide kernels
load them through load16_a2, now page-safe (the second chunk only when the
address is unaligned). Q8_0 4-6x, IQ4_NL 640->2560 4x. Q6_K's Q8_1 activation
int4 loads were 4-byte aligned: four int loads instead.
Output identical for 146 of 256 tokens, then a near-tie flips (summation
order); STRATA_MMVQ_WIDE_32=0 restores the old kernels.
- resident_plan groups the entries with one thread per entry (thread 0
alone took 73 us per layer); the plan is the host loop's exactly
- the GR down kernel reads xn from global memory instead of a 60 KB SLM
tile per work-group (occupancy); outputs bitwise equal (gr_bench)
- 16-byte-load wide kernels for Q4_K, Q5_K and IQ4_XS (IQ4 codebook in
registers): IQ4_XS 107 -> 323 GB/s, Q4_K/Q5_K 1.3-1.6x
Output tokens identical in every engine run.
Q6_K blocks are 210 bytes, so the wide kernel's 16-byte ql/qh loads were
2-byte aligned and the B70 split them: 145 GB/s against 640 GB/s for the
same blocks at a 224-byte stride. load16_a2 does two aligned loads and a
shift; with it the wide kernel takes every Q6_K shape and up to 8 columns
(2.6-3.4x in q6k_align_bench). Output tokens identical.
Also a port copy of ple_reader.cpp with STRATA_PLE_TRACE=1, which showed
the 45/39 tok/s two-speed runs are one 226 ms PLE stall landing in either
the prompt or the first decode round (TTFT + decode is the same).
Experts without a VRAM slot are read once into pinned host memory and the
device-built verify plan points the expert kernels at them over PCIe. The
plan used to drop non-resident experts, so output with misses diverged.
Forced 1,879-expert miss: decode 2.6 -> 40.9 tok/s, output identical.
--kv-resident 32768 keeps the whole KV in pinned host memory and only the attended window in VRAM, so every
expert stays in VRAM at long contexts. B70, decode after the prompt: 128K int8 8.5 -> 38.7 tok/s, 128K q4_0
32.7 -> 35.3, 256K int8 3.7 -> 31.3, 256K q4_0 5.0 -> 31.5; the prompt pays 5-13%. setup.py now turns it on
from 64K for the SYCL engine (its RAM check counts only the KV copy and the engine, the experts being in VRAM)
and keeps INT8 KV, which is the faster of the two with streaming.
setup.py gets a third backend, sycl: on an Intel Arc with build-sycl-aot/strata and the strata-sycl-dev image
present (or --intel-engine strata; --intel-engine llama keeps llama.cpp), it builds the native pack and the
MTP draft layer (reusing an mtp-q2_0.gguf that is already there), writes the engine's arguments with the
container's paths through sycl/serve/strata-sycl.sh, --stream-experts with a reserve that keeps every expert in
VRAM (1,024 MiB up to 32K, 2,048 MiB and 4,096-token chunks above), Q4_0 KV above 64K, and keeps the sampling
block of an earlier config. start() checks the model files through the config's sycl_root; upgrade_config
leaves the SYCL config alone (the wrapper has no version, which read as 0.1.12 and downgraded --prefill).
Tested on the B70: the Coder's config regenerated by setup serves at 38.9 / 30.3 tok/s (first turn / follow-up)
with all 12,288 experts in VRAM.
The serve loop used to abort at exit (139): the Verifier's destructor waited on its queue after the Level
Zero context had begun tearing down. The destructor is now exception-safe and serve mode exits directly once
its requests are done: exit 0.
The Coder on the B70 now runs Strata's own engine behind serve/server.py --engine strata: the wrapper starts
build-sycl-aot/strata inside strata-sycl-dev with the serve pipes attached and the port's run-time switches set.
Through the OpenAI API with temperature 0.6 / top-p 0.95 / top-k 20 / repetition penalty 1.05: 36 tok/s decode
on a first turn, 33 on a follow-up (conversation cache reused), vs 24-26 for llama.cpp on the same card. The
expert reserve has to leave every expert in VRAM (--stream-experts has no host copy): 1,024 MiB at 32K context.
The selection dump measured how much neighbouring prompt positions' QSA selections overlap at 80K tokens: the
union of 8 is 3.3x one selection (16: 5.1x), 12% shared by all 8, so a grouped-gather attention kernel would buy
at most 5-8 s of an 80K prompt. XMX v2 across the 128K/256K matrix: 23-33% slower prompts than the FP32 attention.
v2: 64-cell chunks, V packed two cells at a time as 32-byte stores, K^T as 4-byte pairs with INT8 converted on
the load, hi+lo Q in one accumulator per scale group, one accumulator update per chunk. qsa_prompt_attn_parity
passes in every mode and kv_hybrid_parity passes completely with it (its last step needs a tensor-core-class
prompt kernel). 13.4 ms per chunk vs v1's 18.7 and the FP32 fallback's 5.5: the attention is gather-bound, so
it stays opt-in (STRATA_PROMPT_ATTN_XMX=1, or =32 for 32-cell chunks).
INTEL.md: 128K and 256K prompts with int8, q4_0 and k8v4 KV on the B70. All complete; q4_0 keeps every expert
in VRAM at 128K (decode 32.7 tok/s after the prompt); at 256K all three evict experts and decode drops to 4-5.
Without borrowing, the VRAM reserve for the KV state and the chunk buffers evicts ~1,900 experts, the prompt
streams them (790 tok/s at 80,000 tokens) and decode after it runs at 2 tok/s. Borrowing lends ~940 slots and
refills them in ~1 s: 1,062 tok/s and 35-40 tok/s decode. It costs ~1 s per prompt at short contexts (2,184
tokens 610 vs 792 tok/s), so the default is by context: borrow above 32K. --prefill-borrow / --no-prefill-borrow
decide explicitly; layer splits keep their own buffers as before.
- Re-migration merge of upstream 99b4097..0ea2fc3 (GDN recurrence pipelining, block scores read once per window,
the GPU-wide sampler, the draft head's VRAM reserve, correctness fixes); verify.cpp/mtp.cpp ported by hand; two
new fixups (a cudaStream_t cast in the sampler, a const sycl::free in native_head). Parity unchanged (19 of 21).
B70: 2,184-token prompt 765 -> 792 tok/s, decode at that context 43.1 -> 45.2 tok/s, outputs identical.
- The prompt path's stream plan kept GgufExpertSource::blob() pointers for ~1,900 experts per chunk against a
512-slot ring, so blobs were overwritten before they were copied (the first 80k measurement was partly wrong
experts). The stager threads now read (layer, expert) themselves via GgufExpertSource::read_into.
- The stream-all walk hangs in its first large chunk on this card (copy engine stuck on a barrier, any ring size,
either issuer; the old read path too). The port defaults to the per-layer routed-only walk (upstream e258e6e's
STRATA_PREFILL_RING=8, included); STRATA_PREFILL_STREAM_ALL=1 restores it for debugging.
- The PLE reader copy keeps its 64-deep queue. 80,000 tokens now complete: 101 s = 790 tok/s.
The rows of a 2,184-token prompt are 27k random 4 KB O_DIRECT reads (114 MB for ~7 MB of rows), all before the
first kernel: 466 ms with the reader's 16 blocking threads. 64 threads (the port's default now, a copy of
src/platform/direct_file.cpp; STRATA_IO_THREADS still overrides) give 330 ms and 128 / 256 the same - the drive
tops out near 85k IOPS. A 256-token first chunk (STRATA_PREFILL_FIRST, 0: off) lets the GPU start while the
rest of the rows are read behind it: 2,184 tokens 3,075 -> 2,860 ms, time to first token ~150 ms less; 8,000
tokens 986 tok/s. The chunk boundary moves rounding in the expert GEMMs, so a long greedy continuation can
diverge late (token 45 of 64 on the test prompt); short prompts are bit-identical.
Measured on the B70, outputs identical before and after each change (docs/INTEL.md "Speed work"):
- every window graph and the drafter's graphs captured at load (STRATA_WARM_GRAPHS=0 restores first-use capture)
- the commit graph is left running while the drafter's round runs on its own queue: decode 38.3 -> 43.1 tok/s at
the 2,184-token context, 46.5 -> 49.7 on a short prompt
- the draft layer's batched prompt pass takes prompts under 64 rows (106 -> 16 ms on 19 tokens)
- one device module per kernel: the first launch in the prompt path 245 -> 1 ms
- the expert dequant writes each thread's run as one vector store: 0.085 -> 0.030 ms per expert; the 2,184-token
prompt 566 -> 682 tok/s, 8,000 tokens 720 -> 841
- SWAR sign compare/subtract in the expert dots, a local-memory resident-plan kernel, a split-K fused down kernel
(bit-identical to the single-token kernel, gr_parity): parity-clean, no measurable decode change
XMX: oneMKL's FP16 GEMMs already run on the XMX units (30-60 TFLOP/s). Two joint_matrix kernels are in the tree,
correct and opt-in because both lose to the paths they would replace on this card: xmx_gemm_iq (fused dequant +
GEMM from the quantized rows; xmx_gemm_bench) and qsa_prompt_attn_xmx (the mma.sync prompt attention's port;
qsa_prompt_attn_parity passes at 1e-6 of scale; STRATA_PROMPT_ATTN_XMX=1).
Profiling: unitrace -q needs Intel's metrics libraries (Dockerfile.metrics) and dev.xe.observation_paranoid=0;
the per-kernel table is in INTEL.md. A trap recorded there: --prefill-until with a native pack drops the prompt
tail (the token loop is skipped), so a "windows for short prompts" routing was invalid and is not in this commit.
The merge of upstream main (99b4097) left 28 migrated copies behind. They are refreshed by re-migration:
dpct over the old and the merged tree, sycl/tools/normalize.sh (dpct's names, unchanged files, message
serials, fixups.py) on both, then a 3-way merge per file. Copies dpct never produced (verify.cpp, mtp.cpp)
take upstream's K8V4 diff by hand. The PTX gate fixup follows upstream's new __HIPCC__ guard (#elif).
The AOT build never received -cl-fp32-correctly-rounded-divide-sqrt (it went to the spir64 backend, which
an AOT build does not have): quantize_act_parity had its 303k mismatches back. The option now reaches
ocloc through -device "... -options ...". 19 of 22 parity tests pass again; kv_hybrid_parity (new) passes
all but its tensor-core step and is built, not registered.
B70, merged engine, AOT, --stream-experts --no-prefill-borrow --spec 4 --mtp: 2,184-token prompt 566.5
tok/s (was 475), decode 38.3 tok/s at that context (was 42.7; 77% draft acceptance on this text), MTP
prompt pass 38.6 ms (E-9 batched).
bw probe (incompressible data): 4-byte loads stream at 395 GB/s, 16-byte at 596. The Q6_K matvec sits at
130-160 GB/s in every variant tried (lanes per row, rows per warp, unroll, 16-byte loads, software
pipelining): latency-bound on a dependent load->unpack->dp4a chain, not ALU or bandwidth. The 16-byte
kernel is kept for n_out >= 4096 (+13% there, slower on small shapes); rel 5e-8 against the shared one.
Decode 20.80 tok/s (spec 2). The drafter is the lever now: MTP fetch + pack running.
The prompt path collects hundreds of blob pointers per layer and copies from them on other threads later;
the 8-slot ring handed back pointers that were overwritten by then, and its index was not thread-safe.
Slots are now keyed by (layer, expert), reused oldest-first among 512 (~1.4 GB), under a mutex.
Long prompts also hung in the prompt path's streamed-expert handshake (lent cache slots make experts
non-resident, the host supplies them behind a flag the GPU wait kernel does not see). --no-prefill-borrow
keeps every expert resident and removes the handshake: 2,000 tokens 693 tok/s, 2,185 tokens 560 tok/s,
300 tokens 180 tok/s (llama.cpp on the same card: 150 short, 424 at 105k).
Rig: watchdog reports and kills a hung run at the expected time (EXPECT_S/QUIET_S); --spec 2 default
(20.75 tok/s decode vs 16.95 at spec 4: the suffix drafter accepts 9%).
- exit segfault: dpct's static global_memory/constant_memory objects destructed after the default queue
(queue::get_device in the core); heap-allocated, never freed. Exit code 0; unitrace can write its table.
- Dockerfile.unitrace (pti-gpu unitrace in the dev image) + sycl/rank_kernels.py: per-kernel device time.
- Q6KTraits: the column-independent unpack (shifts, masks, saturating byte subtract) moved into load();
acts()/dot() split; a row-blocked kernel (native_mmvq_rowwarp_kernel, STRATA_MMVQ_RPW) kept but off:
with a 300 ms warm-up the shared kernel already does 2560x2560x4 in 48 us (110 GB/s) and the variants tie.
- mmvq_bench: the dense kernel timing harness (warm-up + 400 iterations; a 5 ms run measures the clock ramp).
route<NE> templated on the expert count; native_router_top10_multi_ne dispatches 256/512. Both call sites
(the verify window and the per-token moe_route) take it for the Coder. 64 tokens: 16.25 -> 16.95 tok/s,
same output. Also: Dockerfile.unitrace (per-kernel Level Zero profiling image).
A window routes the same expert for up to 6 tokens; the grouped kernels recomputed the dequantised weights
(grid lookups, sign unpacking, the byte-wise compare/xor/subtract emulations) once per token. Multi<TY>
splits each of the five formats this model uses (iq2_s, iq3_xxs, iq3_s gate/up; iq4_nl, q2_0 down) into
prep (once) and acts + finish (per token); the kernels take entries four at a time. Lanes per row is a
compile-time constant (STRATA_EXPERT_LANES, 8): 32 lanes on an 80-call row was mostly reduction.
Measured with the new NATIVE_BENCH=1 mode of native_expert_parity (10 experts x 4 entries, one layer):
iq3_xxs/iq4_nl 1.467 -> 0.649 ms, iq2_s/q2_0 ~0.68 -> 0.579 ms, iq3_s/iq4_nl 0.750 ms; parity unchanged.
strata::dp4a wraps dpct::dp4a (sycl::ext::oneapi::dot_acc is the same emulation with a header that breaks
the link across translation units).
The -Xs device flag was mis-quoted; fixed. Measured on the B70 with the same 19-token prompt, 32 tokens:
decode 15.17 tok/s, prefill 28.79 tok/s, a 4-token window 73 ms on the card. The persistent JIT cache
(SYCL_CACHE_PERSISTENT=1) is the alternative for the JIT build: 15 MB, second start 15.6 tok/s.
Measured on the B70: 64 tokens in 52.0 s, of which ~47 s is the first window (the runtime JIT-compiles
every kernel on first use: 6 tokens take 47.5 s, 64 take 52.0 s). The output is the model's real answer.
Debug tooling kept in the port: STRATA_VERIFY_EAGER=1 runs the window on the queue instead of as a graph
and turns the stage stamps into host clocks, so STRATA_VERIFY_PROFILE=1 prints ms per stage without a
device timer; STRATA_VERIFY_DEBUG=1 prints the residual after every layer and the round's phase times.
build.sh takes BUILD_DIR and AOT (an ahead-of-time bmg-g31 build removes the JIT cost).
--stream-experts (GgufExpertSource): every expert goes straight from the GGUF into the VRAM cache through a
small ring of staging buffers; no resident host arena, so the engine runs on a 23 GiB machine.
What it took to get from "loads" to "<think>\nThe user wants a Python function":
- the doorbell handshake through host-mapped memory: volatile device loads do not bypass the caches on
Intel; system-scope atomics (strata::sys_load/sys_store) do. A device-side spin is bounded (kSpinMax): an
unbounded one wedged the GT twice when its process died.
- STRATA_VERIFY_NO_HOST=1 + STRATA_VERIFY_DEVICE_PLAN=1: with every expert resident the GPU plans each
layer itself and the host waits for the whole window graph; per-layer ring visibility inside a graph is
not reliable on this platform (measured with sycl/probe).
- -fsycl-default-sub-group-size=32: the kernels are written for 32-lane warps; dpct pinned only 134 of 289.
- cudaMemcpy is synchronous; dpct's default-queue memcpy had no wait, and the streaming ring reuses its
buffers - the expert cache filled from overwritten memory (non-deterministic residuals, NaN by layer 5).
- native_expert_parity hand-ported (dpct cannot parse it without ggml-cpu.h): the GPU native expert
kernel matches the float reference on real IQ1_M rows.
- verify.cpp debug readbacks (STRATA_VERIFY_DEBUG): the per-layer residual ladder that found all of it.
Not yet: speed (0.17 tok/s - the spin budgets), the segfault at exit, the tensor-core kernels.
The port lives in sycl/ and changes no upstream file: sycl/src and sycl/include hold only the files the
migration touched, sycl/CMakeLists.txt takes everything else from the original tree.
How it was made, reproducibly (sycl/tools/): a dev image (the llama.cpp SYCL image + SYCLomatic 2025.3 +
CUDA 12.8 headers from the pip wheels), migrate.sh (compilation database + dpct over 86 translation
units), fixups.py (every hand fix with its reason, idempotent), build.sh (icpx, ninja).
What dpct got wrong and the script repairs: CUDA's null stream became a null sycl::queue* (strata::q_of
maps it to the default queue); __ldg((const float*) p) lost its cast and read one byte; __fadd_rn lost
parentheses; ggml's lookup tables were threaded through kernel parameters with the wrong table per
template; helper headers renamed since 2025.3; cudaGraphUpload, graph introspection and %globaltimer
have no equivalents. Two compiler flags are load-bearing: -fp-model=precise, and the device compiler's
-cl-fp32-correctly-rounded-divide-sqrt - the Arc's fp32 divide is not correctly rounded by default, and
quantize_act_parity went from 303k mismatches to byte-exact with it.
Not ported: the three kernels with inline PTX (mma.sync, ldmatrix, cp.async) take their existing
pre-sm_80 fallback; the launchers refuse the device. The XMX joint_matrix versions are next.
The engine itself starts on the card, loads the pack, the projections, the PLE table and the 23.4 GiB
expert arena, and is then stopped by the test rig's memory guard: Strata keeps the experts as a resident
host copy, and the rig has 23 GiB of RAM (upstream asks for 32). A streaming fill of the VRAM cache
from the GGUF is the next piece; the whole model fits in the 32 GB card.
Prompt reading scales up with prompt size (150 tok/s at 2.7k, 424 tok/s at 105k), so the full
window costs about five minutes, not fifteen. 30.3 GB in use with 1.5 GB free at 131,072.
About 20 KiB of KV per token (only 12 of 48 layers are full attention). 32k, 64k, 96k and 128k all
load with the whole model on the card, 128k with 1.6 GB to spare; 160k is under the safety margin
and 256k does not fit - asking for it spilled VRAM into host RAM and took the machine down until its
watchdog reset it, so the doc says so in bold.
Without nvidia-smi, setup now looks for an Intel discrete GPU in sysfs (vendor 0x8086 under xe or
i915, named by PCI id - lspci's database does not know an Arc Pro B70 yet) and takes a path with no
compiler and no CUDA toolkit: download the model, export the tokenizer, find a llama-server
(--llama-server PATH, one on PATH, or the B70 container image via docker), write a config with a
"llama" block and a start-llama.sh, and a run script that uses --engine llama.
The CUDA engine's rules that do not apply are stepped around rather than deleted: the compute
capability and driver-version checks, the experts-in-RAM gates (on llama.cpp the fit is VRAM,
shard 1 = the download minus the 28.8 GB lookup table), the MTP draft download (5 GB the llama
engine cannot use), and the engine's own argument list.
engine_from_config runs the config's start script when nothing answers at the url, so the run
script is one click and closing the server stops the container.
Verified on an Arc Pro B70: setup runs steps 1-7 clean against the downloaded Coder IQ1_M.
docs/INTEL.md has the whole picture and the measurements.
The Strata engine is CUDA. On a GPU without it - written for and measured on an Intel Arc Pro
B70 - the same GGUF weights run through llama.cpp's SYCL backend. serve/engine_llama.py puts
llama-server behind Engine.generate(ids, max_new, sampling, cancel), so the OpenAI and
Anthropic endpoints, streaming, tool calls, MCP and the web app are unchanged above it.
It sends the prompt as token ids (the server's own tokenizer, from the same GGUF) with
return_tokens, streams ids back with llama-server's timings mapped onto Strata's `last`,
heartbeats while a long prompt is read, and stops the server by closing the connection when a
client goes away. Attach to a running server ("llama": {"url"}) or spawn one ("llama": {"exe"}).
Not yet: images (Strata's Vision produces embeddings for the CUDA engine; llama-server takes
the image itself) - start without "vision" and image requests are refused cleanly.
Tests: a scripted llama-server (the wire format observed on b29c606), plus a live round trip
when STRATA_LLAMA_URL is set. Measured on the B70 with Coder IQ1_M: 23-25 tok/s decode,
~150 tok/s prefill, correct code, GPU-bound.
2026-09-29 11:09:39 -05:00
2064 changed files with 475670 additions and 3348 deletions
description:Something in Strata doesn't work as documented
title:"[Bug]: "
labels:["bug"]
body:
- type:markdown
value:|
Thanks for the report.
Please search the [open issues](https://github.com/Niko1221/Strata/issues?q=is%3Aissue+state%3Aopen) first — this may already be tracked.
- type:textarea
id:what
attributes:
label:What happened
description:What you did, what you got, and what you expected instead. Paste the exact error text if there is one. If there's a set of steps or commands that reproduces it, put them here too.
validations:
required:true
- type:input
id:hardware
attributes:
label:Strata version, GPU, OS
description:The engine version is on the About tab, or in `GET /v1/status` under `engine`.
placeholder:"0.1.31 · RTX 5070 12GB · Windows 11"
validations:
required:true
- type:textarea
id:log
attributes:
label:Engine log
description:Drag in `strata-<model>.log` from your Strata folder. Drag and drop works directly into this box.
validations:
required:false
- type:markdown
value:|
A couple of extras that help a lot:
- Check the [troubleshooting table](https://github.com/Niko1221/Strata/blob/main/docs/DETAILS.md#troubleshooting) first, many reports are already solved there.
description:Suggest something new or an improvement
title:"[Feature]: "
labels:["enhancement"]
body:
- type:markdown
value:|
Explain what you want in plain language.
- type:textarea
id:idea
attributes:
label:What do you want to do?
description:Describe the goal rather than the implementation. If there's a workaround you use today, mention it.
validations:
required:true
- type:markdown
value:|
All of this counts: model and architecture support, the server and API, the web app, multi-GPU and layer split, images and vision, MCP servers, setup and installation, and performance.
message(FATAL_ERROR"STRATA_ISA_FLOOR must be empty, avx or none (got '${STRATA_ISA_FLOOR}')")
endif()
if(STRATA_ISA_FLOORANDSTRATA_PORTABLE)
message(FATAL_ERROR"STRATA_ISA_FLOOR (an older-CPU build) and STRATA_PORTABLE (the AVX2 release build) exclude each other")
endif()
if(STRATA_PORTABLEANDMSVC)
set(CMAKE_MSVC_RUNTIME_LIBRARY"MultiThreaded")
endif()
@@ -33,17 +56,36 @@ if(NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
endif()
option(STRATA_ENABLE_CUDA"Build the CUDA targets (Phase 2+)"OFF)
option(STRATA_ENABLE_HIP"Build the HIP targets for wave32 AMD GPUs (gfx1100, gfx1201; gfx1030 unvalidated)"OFF)
# AMD gfx906 (MI50, wave64): the same .cu sources compiled as HIP through the gfx906 compat layer. The CUDA runtime and
# cuBLAS names are mapped by the headers in include/strata/platform/hip_compat/, which go first on the include path,
# and CUDA::cudart / CUDA::cublas become aliases of HIP's targets, so every CUDA target below stays as it is. This is
# separate from STRATA_ENABLE_HIP (the wave32 RDNA backend): with it on, STRATA_ENABLE_CUDA is on and
# STRATA_ENABLE_HIP stays off, so every "CUDA" branch of the tree is the one taken.
option(STRATA_HIP_GFX906"Build the CUDA targets for AMD gfx906 through HIP (wave64 compat layer)"OFF)
if(STRATA_HIP_GFX906)
if(STRATA_ENABLE_HIP)
message(FATAL_ERROR"STRATA_HIP_GFX906 and STRATA_ENABLE_HIP are separate backends; enable only one")
endif()
set(STRATA_ENABLE_CUDAONCACHEBOOL""FORCE)
endif()
option(STRATA_ENABLE_HIP"Build the HIP targets for wave32 AMD GPUs (gfx1100, gfx1201; gfx1012/gfx1030 unvalidated)"OFF)
option(STRATA_PREFILL_MMQ"Build the optional GGML MMQ prefill path for HIP"OFF)
# CUDA: MMQ prompt kernels for Q4_K / Q5_K / Q5_1 (Unsloth's UD-Q4_K_XL experts). Off by default: the engine loads
# every kernel when it starts (CUDA_MODULE_LOADING=EAGER), so these took ~5 MB of VRAM - 1-3 expert slots - from every
# model, and the UD-Q4_K_XL prompt is SSD-bound with or without them (docs/UNSLOTH_Q4.md)
option(STRATA_MMQ_KQUANTS"Build the CUDA MMQ prompt kernels for Q4_K / Q5_K / Q5_1 experts"OFF)
option(STRATA_MMQ_KQUANTS"Build the MMQ prompt kernels for Q4_K / Q5_K / Q5_1 / Q6_K: the CUDA experts of Unsloth's UD-Q4_K_XL, and on HIP the dense GGUF projections (STRATA_DENSE_MMQ)"OFF)
# CUDA: Q6_K gate/up in the grouped expert kernels (Unsloth's UD-Q6_K_XL, docs/UNSLOTH_Q6.md). Off by default: every
# grouped-kernel instance is loaded when the engine starts (CUDA_MODULE_LOADING=EAGER), so one more costs VRAM that
# every other model would lose as expert slots (PR #860 review)
option(STRATA_Q6K_EXPERTS"Build the Q6_K gate/up grouped expert kernels (UD-Q6_K_XL)"OFF)
if(STRATA_Q6K_EXPERTS)
add_compile_definitions(STRATA_Q6K_EXPERTS=1)
endif()
option(STRATA_ORCA_Q4KS_MMQ"Build the GGML Q5_0 MMQ instance for OrcaRouter Q4_K_S"OFF)
<p align="center"><b>Ein KI-Modell mit 125 Milliarden Parametern auf deinem eigenen Gaming-PC</b><br>
NVIDIA- oder AMD-Grafikkarte (ab 12 GB) · Windows oder Linux · kostenlos und Open Source</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Ein Voxel-Pagodengarten, den das Modell in Strata geschrieben hat, läuft im Browser"></a><br>
<sub>Ein Voxel-Pagodengarten aus einem einzigen Prompt, auf einer RTX 5070 mit Strata (IQ3_S, 128K Kontext) ·
<a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4">ganzes Video (49 s)</a></sub></p>
Strata lässt **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** auf einem normalen PC laufen.
Das ist ein großes, kluges KI-Modell, das sonst meist einen Server braucht. Es chattet, schreibt Code, versteht
Bilder und arbeitet mit deinen Apps und Coding-Agents zusammen. Nichts verlässt deinen PC.
## Wie schnell ist es?
Wir haben es auf zwei ganz normalen Gaming-PCs gemessen. Ein Token ist etwa ¾ eines Wortes.
- **Schreibt Antworten:** wie schnell die Antwort in einem kurzen Chat erscheint. 60 Tokens pro Sekunde sind
schneller, als du lesen kannst.
- **Liest deinen Prompt:** wie schnell es aufnimmt, was du schickst (hier ein Dokument, Code oder Chatverlauf mit
NVIDIA: Q2_0 mit Engine 0.1.36, die anderen Zeilen mit 0.1.26 (4K Antworten, 32K Prompts). Die vollständigen
Tabellen stehen in [DETAILS.md](docs/DETAILS.md#speed-measured). Eine Karte mit mehr VRAM ist schneller: Eine
RTX 3090 (24 GB) sollte etwa 100-140 Tokens pro Sekunde schreiben. Lange Chats und andere Karten:
[Tempo jedes Modells](docs/MODELS.md#how-fast-is-each-size), [Ergebnisse aus der Community](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata ist kostenlos. Wenn es auf deinem PC gut läuft, hält ein Kaffee die Arbeit daran am Laufen.</sub></p>
## Was du brauchst
| | |
| --- | --- |
| **Grafikkarte** | **NVIDIA** GeForce RTX 20, 30, 40 oder 50 Serie, oder **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 oder RX 6800 / 6900 Serie. Sie braucht **12 GB VRAM oder mehr**. |
| **RAM** | 32 GB oder mehr. Dein RAM entscheidet, [welches Modell](#welches-modell-soll-ich-nehmen) passt. Mit 64 GB laufen alle Größen. |
| **Festplatte** | Etwa 80 GB frei. Nimm wenn möglich eine SSD: Der erste Start geht dann viel schneller. |
| **System** | Windows 10 / 11 oder Linux und ein aktueller Grafiktreiber von NVIDIA oder AMD. |
Alles andere richtet der Installer ein. Zwei oder drei Karten können sich das Modell teilen ([Multi-GPU](docs/MULTI_GPU.md)).
Experimentell, von Community-Mitgliedern auf ihren eigenen Rechnern geschrieben und getestet:
- **Mein PC ist beim ersten Start von Strata eingefroren.** Das ist normal, während das Modell lädt. Warte und
schließ das Fenster nicht. Nach 10 Minuten immer noch eingefroren? Starte den PC neu, schließ andere Programme und
versuch es noch einmal, oder nimm eine kleinere Größe.
- **Es hat beim Herunterladen oder Installieren aufgehört.** Starte `START-HERE.bat` (oder `./setup.sh`) noch
einmal. Es macht dort weiter, wo es aufgehört hat.
- **Es ist sehr langsam und die Festplattenlampe blinkt ständig, oder es meldet „the engine stopped
unexpectedly“.** Dein PC hat nicht genug freien RAM. Schließ andere Programme (Browser brauchen viel) oder nimm
eine kleinere Größe (Q2_0 oder IQ2_XS).
- **Es meldet, dass Port 8080 schon belegt ist.** Strata läuft bereits. Such nach seinem Fenster.
Weitere Probleme und ihre Lösungen: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Kommst du nicht weiter?
Öffne ein [Issue](https://github.com/Niko1221/Strata/issues) und hänge `strata-<model>.log` aus dem Strata-Ordner
an. Du hast ein Sicherheitsproblem gefunden? Melde es vertraulich: [SECURITY.md](SECURITY.md).
## Wie funktioniert es?
Modelle wie dieses laufen normalerweise auf Servern mit Hunderten Gigabyte Grafikspeicher. Deine Grafikkarte hat
12-24 GB. Strata bringt das Modell trotzdem unter, indem es **die Arbeit auf deinen ganzen PC verteilt**. Stell dir
eine Küche vor: Was du ständig brauchst, bleibt auf der Arbeitsfläche, der Rest wartet in der Vorratskammer.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Die 24,576 Experten des Modells: die meistgenutzten auf der Grafikkarte, alle im RAM, eine Nachschlagetabelle auf der SSD"></p>
- **Das Modell ist ein Team aus 24,576 kleinen Spezialisten („Experten“).** Jedes Wort braucht nur 10 davon.
- **Deine Grafikkarte** hält die paar tausend Experten, die am häufigsten gebraucht werden. **Dein RAM** hält alle,
und **dein Prozessor** rechnet gleichzeitig am Rest. **Deine SSD** hält eine große Nachschlagetabelle.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Ein kleiner Helfer rät die nächsten Wörter; das große Modell prüft alle auf einmal und behält die richtigen"></p>
- **Raten, dann prüfen:** Ein kleiner Helfer rät die nächsten paar Wörter. Das große Modell prüft sie alle auf
einmal. Du bekommst dieselbe Antwort, nur 1.6-1.8x schneller.
- **Lange Texte werden in großen Stücken gelesen** (bis zu 8,192 Tokens auf einmal), mit über 1,000 Tokens pro
Sekunde.
Die längere Erklärung: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Jeder Teil und seine Zahlen:
[die Details](docs/DETAILS.md#how-it-works) und das [Paper](docs/paper/Strata-Paper.pdf).
## Danksagung und Lizenz
Das Modell ist [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) vom Qwen-Team. Komprimiert
wurde es von [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
und Unsloth. Strata nutzt Teile von [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Alle Danksagungen:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata ist Open Source unter der [MIT-Lizenz](LICENSE). Ein
paar Teile und alle Modelle haben eigene Lizenzen ([welche](docs/HOW_IT_WORKS.md#license)).
## Strata unterstützen
Strata ist kostenlos und Open Source. Wenn es dir nützt, kannst du die Entwicklung unterstützen:
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><b>Ejecuta un modelo de IA de 125 mil millones de parámetros en tu propio PC gaming</b><br>
Tarjeta gráfica NVIDIA o AMD (12 GB o más) · Windows o Linux · libre y de código abierto</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Un jardín con una pagoda de vóxeles que escribió el modelo de Strata, funcionando en el navegador"></a><br>
<sub>Un jardín con una pagoda de vóxeles, hecho con un solo prompt en una RTX 5070 con Strata (IQ3_S, contexto de 128K) ·
NVIDIA: Q2_0 con el motor 0.1.36, las demás filas con 0.1.26 (respuestas de 4K, prompts de 32K). Las tablas
completas están en [DETAILS.md](docs/DETAILS.md#speed-measured). Una tarjeta con más VRAM es más rápida: una
RTX 3090 (24 GB) debería escribir unos 100-140 tokens por segundo. Chats largos y otras tarjetas:
[velocidad de cada modelo](docs/MODELS.md#how-fast-is-each-size), [resultados de la comunidad](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata es gratis. Si funciona bien en tu PC, un café ayuda a seguir trabajando en él.</sub></p>
## Qué necesitas
| | |
| --- | --- |
| **Tarjeta gráfica** | **NVIDIA** GeForce serie RTX 20, 30, 40 o 50, o **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 o serie RX 6800 / 6900. Necesita **12 GB de VRAM o más**. |
| **RAM** | 32 GB o más. Tu RAM decide [qué modelo](#qué-modelo-elijo) cabe. Con 64 GB funcionan todos los tamaños. |
| **Disco** | Unos 80 GB libres. Usa un SSD si puedes: el primer arranque es mucho más rápido. |
| **Sistema** | Windows 10 / 11 o Linux, y un controlador gráfico actual de NVIDIA o AMD. |
El instalador prepara todo lo demás. Dos o tres tarjetas pueden repartirse el modelo ([multi-GPU](docs/MULTI_GPU.md)).
Experimental, escrito y probado por miembros de la comunidad en sus propios equipos:
- **Intel Arc**, compilado desde el código fuente en Linux: [Intel Arc](docs/INTEL_ARC.md).
- **AMD Ryzen AI Max (Strix Halo)**, compilado desde el código fuente en Linux: [Strix Halo](docs/STRIX_HALO.md).
- **Procesadores antiguos sin AVX2**: funcionan, pero despacio. [Older CPUs](docs/INSTALL.md#older-cpus-experimental).
La lista completa: [docs/INSTALL.md](docs/INSTALL.md#what-you-need).
## Instalación
### Deja que tu IA lo instale
¿Usas un asistente de programación con IA (Claude Code, Cursor, Codex, GitHub Copilot, ...)? Pega esto en él:
```text
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
```
Revisa tu tarjeta gráfica, tu RAM y tu disco, y elige el modelo que cabe. Luego lo instala, lo arranca y te explica
cómo conectar tus aplicaciones. Las herramientas de IA también pueden instalar, arrancar y detener Strata con su
[servidor MCP](docs/MCP_SERVER.md).
### O hazlo tú mismo
[Descarga Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) y descomprímelo (o usa `git clone`).
**Windows:** haz doble clic en **`START-HERE.bat`**. **Linux:** ejecuta **`./setup.sh`** en la carpeta de Strata.
Los pasos son los mismos para NVIDIA y AMD. El instalador detecta tu tarjeta y prepara el motor adecuado para ella.
Te hace unas pocas preguntas:
- qué modelo y qué tamaño,
- cuánto contexto (cuánto texto tiene presente el modelo),
- si debe entender imágenes.
Pulsa Enter cada vez para usar la respuesta recomendada. Después descarga el modelo (unos 70 GB) y lo arranca. Si la
descarga se detiene, vuelve a ejecutarlo: sigue donde lo dejó. Tu navegador abre la aplicación de Strata en
`http://127.0.0.1:8080`.
> **Mientras el modelo arranca, tu PC puede ir lento o dejar de responder durante 1-3 minutos** (más la primera vez).
> Strata carga 35-55 GB en tu RAM y reserva una parte para la tarjeta gráfica. Es normal. Espera y no cierres la
> ventana. La ventana muestra lo que está haciendo Strata.
**La próxima vez**, vuelve a ejecutar `START-HERE.bat` (o `./setup.sh`). Arranca enseguida y no descarga nada dos
veces. Cierra su ventana para detener el modelo. `UPDATE.bat` (`./update.sh`) actualiza Strata sin arrancarlo.
Actualizaciones, Docker, varias tarjetas, dónde se guardan los archivos y todas las opciones: [docs/INSTALL.md](docs/INSTALL.md).
## ¿Qué modelo elijo?
El instalador te recomienda uno según tu RAM. El mismo modelo viene en varios tamaños, más o menos comprimidos. Los
tamaños pequeños son más rápidos. Los grandes son algo más inteligentes.
| Tu RAM | Elige | Por qué |
| --- | --- | --- |
| **32 GB** | **Coder** | cabe en 32 GB y está hecho para programar (con una tarjeta de 24 GB también funcionan Q2_0 e IQ2_XS) |
| **48 GB** | **IQ2_XS** (o Q2_0, el más rápido) | los tamaños más grandes no caben |
| **64 GB** | **IQ2_XS** (recomendado), o IQ3_XXS / IQ3_S | caben todos los tamaños; IQ3_S es el mejor y el más lento |
| **96 GB o más** | **IQ3_S**, o el UD-IQ4_XS de Unsloth (~4 bits) | espacio para los tamaños más grandes con todo lo demás abierto |
- **[Coder](docs/MODELS.md#coder):** una versión para programar a la que se le quitó la mitad de los expertos. Alcanza
el 91% de la puntuación SWE-bench Verified del modelo completo (medido por sus autores) y cabe en 32 GB de RAM. Es
más flojo fuera del código, también con chino y otros textos CJK (#438). Para eso, elige Q2_0, IQ2_XS o IQ3_S, que
conservan todos los expertos.
- **[Swift 1.5](docs/MODELS.md#swift-15):** un ajuste fino que piensa mucho menos tiempo antes de responder. Recibes
la respuesta antes, con más o menos la misma calidad.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** la versión de ~4 bits de Unsloth, entre IQ3_S y
UD-Q4_K_XL en calidad. Una descarga de 94 GB. Con menos de ~80 GB de RAM, Strata lee una parte desde el SSD
mientras responde, así que ahí es más lento (un SSD NVMe ayuda).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (experimental): el más parecido al modelo
completo. Pero Strata lee la mayor parte desde el SSD mientras responde, así que solo escribe 7-8.5 tokens/s en un
PC con 64 GB.
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** se instala a mano. No está en
el menú del instalador.
Tamaños, descargas y qué cabe dónde: [docs/MODELS.md](docs/MODELS.md). Para añadir otro modelo más adelante, ejecuta
`SETUP.bat` (Linux: `./setup.sh --setup`).
## Cómo usarlo
<p align="center"><img src="docs/media/runpagoda.png" width="900" alt="La pestaña Monitor de la aplicación de Strata junto a un agente de programación"><br>
<sub>El <b>Monitor</b> de la aplicación de Strata (izquierda) mientras un agente de programación escribe el jardín de la pagoda del vídeo (derecha)</sub></p>
- **En el navegador:** abre `http://127.0.0.1:8080`. Tiene **Chat**, un **Monitor** en vivo del modelo y de tu
GPU/CPU/RAM, y **About** con los ajustes y las direcciones.
- **Tus aplicaciones y agentes de programación:** añade un proveedor "compatible con OpenAI" con la URL base
**`http://127.0.0.1:8080/v1`**. Sirve cualquier clave de API y cualquier nombre de modelo.
- Aplicaciones que usan la API de Anthropic: `http://127.0.0.1:8080/v1/messages` (Claude Code:
`ANTHROPIC_BASE_URL=http://127.0.0.1:8080`).
- Codex CLI y otras aplicaciones que usan la API Responses de OpenAI: `/v1/responses`
- **Razonamiento:** elige **off, low, medium o high** en el menú del chat o en el "reasoning effort" de tu aplicación.
Off es lo más rápido. High es lo mejor para preguntas difíciles.
- **Imágenes:** responde que sí a "Images?" en la instalación. Luego haz clic en **Picture** en el chat, o adjunta
imágenes en tu aplicación. Las tarjetas AMD leen imágenes en Linux a través del procesador; en Windows todavía no.
- **Desde tu móvil u otro PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Pon siempre una clave.
- **Una petición cada vez:** por defecto Strata responde una petición y las demás esperan. Para responder varias a la
vez, pon `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). En una tarjeta de 12 GB, cada respuesta va más lenta.
- **Prompts largos:** Strata lee entero el primer mensaje de un chat, más o menos 1 minuto por cada 30,000 tokens. Los
mensajes siguientes empiezan en segundos.
Más: [dónde se guardan tus chats](docs/INSTALL.md#where-things-are-stored), [la API](docs/DETAILS.md#using-it).
## ¿Algo salió mal?
- **Mi PC se congeló la primera vez que arrancó Strata.** Es normal mientras carga el modelo. Espera y no cierres la
ventana. ¿Sigue congelado después de 10 minutos? Reinicia el PC, cierra otros programas y vuelve a intentarlo, o
elige un tamaño más pequeño.
- **Se detuvo durante la descarga o la instalación.** Vuelve a ejecutar `START-HERE.bat` (o `./setup.sh`). Sigue
donde se detuvo.
- **Va muy lento y la luz del disco no deja de parpadear, o dice "the engine stopped unexpectedly".** Tu PC no tiene
suficiente RAM libre. Cierra otros programas (los navegadores usan mucha), o elige un tamaño más pequeño (Q2_0 o
IQ2_XS).
- **Dice que el puerto 8080 ya está en uso.** Strata ya está en marcha. Busca su ventana.
Más problemas y sus soluciones: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). ¿Sigues atascado? Abre un
[issue](https://github.com/Niko1221/Strata/issues) y adjunta `strata-<model>.log` de la carpeta de Strata. ¿Has
encontrado un problema de seguridad? Infórmalo en privado: [SECURITY.md](SECURITY.md).
## ¿Cómo funciona?
Los modelos como este suelen funcionar en servidores con cientos de gigabytes de memoria gráfica. Tu tarjeta gráfica
tiene 12-24 GB. Strata hace que el modelo quepa **repartiendo el trabajo por todo tu PC**. Piensa en una cocina: lo
que usas todo el rato se queda en la encimera, y el resto espera en la despensa.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Los 24,576 expertos del modelo: los más usados en la tarjeta gráfica, todos en la RAM y una tabla de consulta en el SSD"></p>
- **El modelo es un equipo de 24,576 pequeños especialistas ("expertos").** Cada palabra solo necesita 10 de ellos.
- **Tu tarjeta gráfica** guarda los pocos miles de expertos que más se usan. **Tu RAM** los tiene todos, y **tu
procesador** trabaja con el resto al mismo tiempo. **Tu SSD** guarda una gran tabla de consulta.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Un pequeño ayudante adivina las siguientes palabras; el modelo grande las comprueba todas a la vez y se queda con las correctas"></p>
- **Adivinar y comprobar:** un pequeño ayudante adivina las siguientes palabras. El modelo grande las comprueba todas
a la vez. Recibes la misma respuesta, 1.6-1.8x antes.
- **Los textos largos se leen en trozos grandes** (hasta 8,192 tokens cada vez), a más de 1,000 tokens por segundo.
La explicación larga: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Cada parte y sus números:
[los detalles](docs/DETAILS.md#how-it-works) y el [artículo](docs/paper/Strata-Paper.pdf).
## Créditos y licencia
El modelo es [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), del equipo de Qwen. Lo
y Unsloth. Strata usa partes de [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Todos los créditos:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata es de código abierto con la [licencia MIT](LICENSE).
Algunas partes y cada modelo tienen sus propias licencias ([cuáles](docs/HOW_IT_WORKS.md#license)).
## Apoya a Strata
Strata es libre y de código abierto. Si te resulta útil, puedes apoyar su desarrollo:
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><b>Faites tourner un modèle d'IA de 125 milliards de paramètres sur votre propre PC de jeu</b><br>
Carte graphique NVIDIA ou AMD (12 Go ou plus) · Windows ou Linux · gratuit et open source</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Un jardin de pagode en voxels écrit par le modèle de Strata, qui tourne dans le navigateur"></a><br>
<sub>Un jardin de pagode en voxels, généré en une seule requête sur une RTX 5070 avec Strata (IQ3_S, contexte de 128K) ·
Strata fait tourner **[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)** sur un PC normal. C'est
un grand modèle d'IA, très capable, qui demande d'habitude un serveur. Il discute, écrit du code, lit des images et
fonctionne avec vos applications et vos agents de code. Rien ne quitte votre PC.
## Quelle est sa vitesse ?
Nous l'avons mesurée sur deux PC de jeu ordinaires. Un token correspond à environ ¾ d'un mot.
- **Écrit les réponses :** la vitesse à laquelle la réponse s'affiche dans une courte discussion. 60 tokens par seconde, c'est plus rapide que votre lecture.
- **Lit votre prompt :** la vitesse à laquelle il absorbe ce que vous envoyez (ici un document, du code ou un historique de discussion de 32K tokens).
<table>
<tr><th>NVIDIA : RTX 5070 (12 Go), Ryzen 5 7600, 64 Go de RAM</th><th>AMD : RX 9070 XT (16 Go), Ryzen 9 3900X, 47 Go de RAM</th></tr>
<tr><td>
| Taille | Écrit les réponses | Lit votre prompt |
| --- | ---: | ---: |
| **Q2_0** | 94 tokens/s | 2,650 tokens/s |
| **IQ2_XS** | 79 tokens/s | 2,090 tokens/s |
| **IQ3_XXS** | 62 tokens/s | 1,750 tokens/s |
| **IQ3_S** | 53 tokens/s | 1,620 tokens/s |
| **Coder** | 55 tokens/s | 2,180 tokens/s |
</td><td>
| Taille | Écrit les réponses | Lit votre prompt |
| --- | ---: | ---: |
| **Q2_0** | 60 tokens/s | 1,160 tokens/s |
| **IQ2_XS** | 52 tokens/s | 1,110 tokens/s |
| **Coder** | 44 tokens/s | 1,420 tokens/s |
</td></tr>
</table>
NVIDIA : Q2_0 avec le moteur 0.1.36, les autres lignes avec 0.1.26 (réponses de 4K, prompts de 32K). Les tableaux
complets sont dans [DETAILS.md](docs/DETAILS.md#speed-measured). Une carte avec plus de VRAM va plus vite : une
RTX 3090 (24 Go) devrait écrire environ 100-140 tokens par seconde. Longues discussions et autres cartes :
[vitesse de chaque modèle](docs/MODELS.md#how-fast-is-each-size), [résultats de la communauté](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata est gratuit. S'il tourne bien sur votre PC, un café aide à poursuivre le travail.</sub></p>
## Ce qu'il vous faut
| | |
| --- | --- |
| **Carte graphique** | **NVIDIA** GeForce RTX série 20, 30, 40 ou 50, ou **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 ou série RX 6800 / 6900. Il faut **12 Go de VRAM ou plus**. |
| **RAM** | 32 Go ou plus. Votre RAM décide [quel modèle](#quel-modèle-choisir-) tient. Avec 64 Go, toutes les tailles tournent. |
| **Disque** | Environ 80 Go libres. Prenez un SSD si possible : le premier démarrage est bien plus rapide. |
| **Système** | Windows 10 / 11 ou Linux, avec un pilote graphique NVIDIA ou AMD à jour. |
L'installateur s'occupe de tout le reste. Deux ou trois cartes peuvent se partager le modèle ([multi-GPU](docs/MULTI_GPU.md)).
Expérimental, écrit et testé par des membres de la communauté sur leurs propres machines :
- **Réflexion :** choisissez **off, low, medium ou high** dans le menu du chat ou dans le « reasoning effort » de
votre application. Off est le plus rapide. High est le meilleur pour les questions difficiles.
- **Images :** répondez oui à « Images? » pendant l'installation. Ensuite cliquez sur **Picture** dans le chat, ou
joignez des images dans votre application. Les cartes AMD lisent les images sous Linux via le processeur ; sous
Windows, pas encore.
- **Depuis votre téléphone ou un autre PC :** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Définissez
toujours une clé.
- **Une requête à la fois :** par défaut, Strata répond à une requête et les autres attendent. Pour répondre à
plusieurs en même temps, mettez `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). Sur une carte de 12 Go, chaque
réponse devient alors plus lente.
- **Longs prompts :** Strata lit en entier le premier message d'une discussion, environ 1 minute pour 30,000 tokens.
Les messages suivants démarrent en quelques secondes.
Plus d'infos : [où sont stockées vos discussions](docs/INSTALL.md#where-things-are-stored), [l'API](docs/DETAILS.md#using-it).
## Un problème ?
- **Mon PC s'est figé au premier démarrage de Strata.** C'est normal pendant le chargement du modèle. Attendez, et ne
fermez pas la fenêtre. Toujours figé après 10 minutes ? Redémarrez le PC, fermez les autres programmes et
réessayez, ou choisissez une taille plus petite.
- **Il s'est arrêté pendant le téléchargement ou l'installation.** Relancez `START-HERE.bat` (ou `./setup.sh`). Il
reprend là où il s'était arrêté.
- **C'est très lent et le voyant du disque clignote sans arrêt, ou il affiche « the engine stopped unexpectedly ».**
Votre PC n'a pas assez de RAM libre. Fermez les autres programmes (les navigateurs en utilisent beaucoup), ou
choisissez une taille plus petite (Q2_0 ou IQ2_XS).
- **Il dit que le port 8080 est déjà utilisé.** Strata tourne déjà. Cherchez sa fenêtre.
Autres problèmes et leurs solutions : [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Toujours bloqué ? Ouvrez une
[issue](https://github.com/Niko1221/Strata/issues) et joignez `strata-<model>.log` depuis le dossier Strata. Vous
avez trouvé un problème de sécurité ? Signalez-le en privé : [SECURITY.md](SECURITY.md).
## Comment ça marche ?
Les modèles comme celui-ci tournent d'habitude sur des serveurs avec des centaines de gigaoctets de mémoire
graphique. Votre carte graphique a 12-24 Go. Strata fait tenir le modèle en **répartissant le travail sur tout votre
PC**. Pensez à une cuisine : ce que vous utilisez tout le temps reste sur le plan de travail, et le reste attend dans
le placard.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Les 24,576 experts du modèle : les plus sollicités sur la carte graphique, tous en RAM, une table de correspondance sur le SSD"></p>
- **Le modèle est une équipe de 24,576 petits spécialistes (les « experts »).** Chaque mot n'en demande que 10.
- **Votre carte graphique** garde les quelques milliers d'experts les plus utilisés. **Votre RAM** les contient tous,
et **votre processeur** travaille sur les autres en même temps. **Votre SSD** contient une grande table de
correspondance.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Un petit assistant devine les mots suivants ; le grand modèle les vérifie tous d'un coup et garde les bons"></p>
- **Deviner, puis vérifier :** un petit assistant devine les quelques mots suivants. Le grand modèle les vérifie
tous d'un coup. Vous obtenez la même réponse, 1.6-1.8x plus tôt.
- **Les longs textes sont lus par gros morceaux** (jusqu'à 8,192 tokens à la fois), à plus de 1,000 tokens par seconde.
L'explication plus longue : [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Chaque partie et ses chiffres :
[les détails](docs/DETAILS.md#how-it-works) et l'[article](docs/paper/Strata-Paper.pdf).
## Crédits et licence
Le modèle est [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), de l'équipe Qwen. Il a été
compressé par [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
et Unsloth. Strata utilise des parties de [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Tous les
crédits : [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata est open source sous [licence MIT](LICENSE).
Quelques parties et chaque modèle ont leur propre licence ([lesquelles](docs/HOW_IT_WORKS.md#license)).
## Soutenir Strata
Strata est gratuit et open source. S'il vous est utile, vous pouvez soutenir son développement :
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata は無料です。あなたの PC でうまく動いたら、コーヒー 1 杯の支援が開発を続ける力になります。</sub></p>
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>Strata is free. If it runs well on your PC, a coffee keeps the work on it going.</sub></p>
@@ -50,26 +55,34 @@ other cards: [speed of each model](docs/MODELS.md#how-fast-is-each-size), [commu
| | |
| --- | --- |
| **Graphics card** | **NVIDIA** GeForce RTX 20, 30, 40 or 50 series, or **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 or RX 6800 / 6900 series - with**12 GB of VRAM or more** |
| **RAM** | 32 GB or more - how much decides [which model](#which-model-should-i-pick) fits; 64 GB runs every size |
| **Disk** | about 80 GB free, on an SSD if you can (the first start is much faster) |
| **System** | Windows 10 / 11 or Linux, and a current graphics driver from NVIDIA or AMD |
| **Graphics card** | **NVIDIA** GeForce RTX 20, 30, 40 or 50 series, or **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 or RX 6800 / 6900 series. It needs**12 GB of VRAM or more**. |
| **RAM** | 32 GB or more. Your RAM decides [which model](#which-model-should-i-pick) fits. 64 GB runs every size. |
| **Disk** | About 80 GB free. Use an SSD if you can: the first start is much faster. |
| **System** | Windows 10 / 11 or Linux, and a current graphics driver from NVIDIA or AMD. |
The installer sets up everything else. Two or three cards can share the model ([multi-GPU](docs/MULTI_GPU.md)).
Experimental, written and tested by community members on their own machines:
- **Thinking:** choose **off, low, medium or high** in the chat menu or in your app's "reasoning effort". Off is the
fastest. High is best for hard questions.
- **Pictures:** say yes to "Images?" in setup. Then click **Picture** in the chat, or attach pictures in your app.
AMD cards read pictures on Linux through the processor; on Windows they can't yet.
- **From your phone or another PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Always set a key.
- **One request at a time:** by default Strata answers one request, and the others wait. To answer several at once,
set `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). On a 12 GB card this makes each answer slower.
- **Long prompts:** Strata reads the first message of a chat in full, about 1 minute per 30,000 tokens. Follow-up
messages start in seconds.
More: [where your chats are stored](docs/INSTALL.md#where-things-are-stored), [the API](docs/DETAILS.md#using-it).
## Something went wrong?
- **My PC froze the first time Strata started.** Normal while it loads the model: wait, don't close the window.
Still frozen after 10 minutes? Restart the PC, close other programs and try again, or pick a smaller size.
- **It stopped while downloading or installing.** Run `START-HERE.bat` (or `./setup.sh`) again: it continues where
- **My PC froze the first time Strata started.** This is normal while it loads the model. Wait, and don't close the
window. Still frozen after 10 minutes? Restart the PC, close other programs and try again, or pick a smaller size.
- **It stopped while downloading or installing.** Run `START-HERE.bat` (or `./setup.sh`) again. It continues where
it stopped.
- **It's very slow and the disk light keeps blinking, or "the engine stopped unexpectedly".** Not enough free RAM:
close other programs (browsers use a lot), or pick a smaller size (Q2_0 or IQ2_XS).
- **It says port 8080 is already in use.** Strata is already running - look for its window.
- **It's very slow and the disk light keeps blinking, or it says "the engine stopped unexpectedly".** Your PC does
not have enough free RAM. Close other programs (browsers use a lot), or pick a smaller size (Q2_0 or IQ2_XS).
- **It says port 8080 is already in use.** Strata is already running. Look for its window.
More problems and their fixes: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Still stuck? Open an
[issue](https://github.com/Niko1221/Strata/issues) and attach `strata-<model>.log` from the Strata folder.
[issue](https://github.com/Niko1221/Strata/issues) and attach `strata-<model>.log` from the Strata folder. Found a
security problem? Report it privately: [SECURITY.md](SECURITY.md).
## How does it work?
Models like this one normally run on servers with hundreds of gigabytes of graphics memory. Your graphics card has
12-24 GB. Strata makes it fit by **sharing the work across your whole PC** - like a kitchen, where the things you use
all the time stay on the counter and the rest waits in the pantry.
Models like this one usually run on servers with hundreds of gigabytes of graphics memory. Your graphics card has
12-24 GB. Strata makes the model fit by **sharing the work across your whole PC**. Think of a kitchen: the things
you use all the time stay on the counter, and the rest waits in the pantry.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="The model's 24,576 experts: the busiest on the graphics card, all of them in RAM, a lookup table on the SSD"></p>
- **The model is a team of 24,576 small specialists ("experts"),** and each word needs only 10 of them.
- **Your graphics card** keeps the few thousand experts that are asked most often;**your RAM** holds all of them,
- **The model is a team of 24,576 small specialists ("experts").** Each word needs only 10 of them.
- **Your graphics card** keeps the few thousand experts that are used most often.**Your RAM** holds all of them,
and **your processor** works on the rest at the same time. **Your SSD** holds a big lookup table.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="A small helper guesses the next words; the big model checks them all at once and keeps the right ones"></p>
- **Guess, then check:** a small helper guesses the next few words and the big model checks them all at once, so
you get the same answer, 1.6-1.8x sooner.
- **Long texts are read in big pieces** (up to 8,192 tokens at a time): over 1,000 tokens per second.
- **Guess, then check:** a small helper guesses the next few words. The big model checks them all at once. You get
the same answer, 1.6-1.8x sooner.
- **Long texts are read in big pieces** (up to 8,192 tokens at a time), at over 1,000 tokens per second.
The longer explanation: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Every part and its numbers: [the
details](docs/DETAILS.md#how-it-works) and the [paper](docs/paper/Strata-Paper.pdf).
The longer explanation: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Every part and its numbers:
[the details](docs/DETAILS.md#how-it-works) and the [paper](docs/paper/Strata-Paper.pdf).
## Credits and license
The model is [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team, compressed by
[ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5) and Unsloth;
Strata is built with parts of [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). All credits:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata is open source under the [MIT License](LICENSE); a few
parts and every model carry their own licenses ([which ones](docs/HOW_IT_WORKS.md#license)).
The model is [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) by the Qwen team. It was
compressed by [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
and Unsloth. Strata uses parts of [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). All credits:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). Strata is open source under the [MIT License](LICENSE). A few
parts and every model have their own licenses ([which ones](docs/HOW_IT_WORKS.md#license)).
<p align="center"><b>Rode um modelo de IA de 125 bilhões de parâmetros no seu próprio PC gamer</b><br>
Placa de vídeo NVIDIA ou AMD (12 GB ou mais) · Windows ou Linux · gratuito e de código aberto</p>
<p align="center"><a href="https://github.com/Niko1221/Strata/releases/download/v0.1.10/Pagoda.mp4"><img src="docs/media/pagoda-preview.webp" width="720" alt="Um jardim com pagode em voxels que o modelo do Strata escreveu, rodando no navegador"></a><br>
<sub>Um jardim com pagode em voxels, feito com um único prompt numa RTX 5070 com o Strata (IQ3_S, contexto de 128K) ·
NVIDIA: Q2_0 com o engine 0.1.36, as outras linhas com 0.1.26 (respostas de 4K, prompts de 32K). As tabelas
completas estão em [DETAILS.md](docs/DETAILS.md#speed-measured). Uma placa com mais VRAM é mais rápida: uma
RTX 3090 (24 GB) deve escrever cerca de 100-140 tokens por segundo. Conversas longas e outras placas:
[velocidade de cada modelo](docs/MODELS.md#how-fast-is-each-size), [resultados da comunidade](docs/COMMUNITY_BENCHMARKS.md).
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br>
<sub>O Strata é gratuito. Se ele roda bem no seu PC, um café ajuda a manter o trabalho nele.</sub></p>
## Do que você precisa
| | |
| --- | --- |
| **Placa de vídeo** | **NVIDIA** GeForce RTX séries 20, 30, 40 ou 50, ou **AMD** Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 ou séries RX 6800 / 6900. Ela precisa de **12 GB de VRAM ou mais**. |
| **RAM** | 32 GB ou mais. Sua RAM decide [qual modelo](#qual-modelo-devo-escolher) cabe. Com 64 GB, roda qualquer tamanho. |
| **Disco** | Cerca de 80 GB livres. Use um SSD se puder: a primeira inicialização fica bem mais rápida. |
| **Sistema** | Windows 10 / 11 ou Linux, e um driver de vídeo atualizado da NVIDIA ou da AMD. |
O instalador cuida de todo o resto. Duas ou três placas podem dividir o modelo ([multi-GPU](docs/MULTI_GPU.md)).
Experimental, escrito e testado por membros da comunidade nas próprias máquinas:
- **Placas de vídeo mais antigas** (Tesla P40 / V100, GTX 10, Radeon VII / MI50, RX 6700 XT, RX 5500 XT): [Older GPUs](docs/OLDER_GPUS.md).
- **Intel Arc**, compilado a partir do código-fonte no Linux: [Intel Arc](docs/INTEL_ARC.md).
- **AMD Ryzen AI Max (Strix Halo)**, compilado a partir do código-fonte no Linux: [Strix Halo](docs/STRIX_HALO.md).
- **Processadores mais antigos sem AVX2**: funcionam, mas devagar. [Older CPUs](docs/INSTALL.md#older-cpus-experimental).
A lista completa: [docs/INSTALL.md](docs/INSTALL.md#what-you-need).
## Instalação
### Deixe sua IA instalar
Você usa um assistente de programação com IA (Claude Code, Cursor, Codex, GitHub Copilot, ...)? Cole isto nele:
```text
Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
```
Ele verifica sua placa de vídeo, RAM e disco e escolhe o modelo que cabe. Depois instala, inicia e explica como
conectar seus apps. Ferramentas de IA também podem instalar, iniciar e parar o Strata pelo
[servidor MCP](docs/MCP_SERVER.md) dele.
### Ou faça você mesmo
[Baixe o Strata](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) e descompacte (ou use `git clone`).
**Windows:** clique duas vezes em **`START-HERE.bat`**. **Linux:** rode **`./setup.sh`** na pasta do Strata.
Os passos são os mesmos para NVIDIA e AMD. O instalador encontra sua placa e configura o engine certo para ela.
Ele faz algumas perguntas:
- qual modelo e qual tamanho,
- quanto contexto (quanto texto o modelo guarda na memória),
- se ele deve entender imagens.
Aperte Enter em cada uma para usar a resposta recomendada. Depois ele baixa o modelo (cerca de 70 GB) e o inicia.
Se o download parar, rode de novo: ele continua de onde parou. Seu navegador abre o app do Strata em
`http://127.0.0.1:8080`.
> **Enquanto o modelo inicia, seu PC pode ficar lento ou parar de responder por 1-3 minutos** (mais na primeira vez).
> O Strata carrega 35-55 GB na sua RAM e reserva parte dela para a placa de vídeo. Isso é normal. Espere e não
> feche a janela. A janela mostra o que o Strata está fazendo.
**Da próxima vez**, rode `START-HERE.bat` (ou `./setup.sh`) de novo. Ele inicia na hora e não baixa nada duas
vezes. Feche a janela para parar o modelo. `UPDATE.bat` (`./update.sh`) atualiza o Strata sem iniciá-lo.
Atualização, Docker, várias placas, onde ficam os arquivos e todas as opções: [docs/INSTALL.md](docs/INSTALL.md).
## Qual modelo devo escolher?
O instalador recomenda um de acordo com sua RAM. O mesmo modelo vem em vários tamanhos, mais ou menos comprimidos.
Os tamanhos menores são mais rápidos. Os maiores são um pouco mais inteligentes.
| Sua RAM | Escolha | Por quê |
| --- | --- | --- |
| **32 GB** | **Coder** | cabe em 32 GB e é feito para código (com uma placa de 24 GB, Q2_0 e IQ2_XS também rodam) |
| **48 GB** | **IQ2_XS** (ou Q2_0, o mais rápido) | os tamanhos maiores não cabem |
| **64 GB** | **IQ2_XS** (recomendado), ou IQ3_XXS / IQ3_S | todos os tamanhos cabem; IQ3_S é o melhor e o mais lento |
| **96 GB ou mais** | **IQ3_S**, ou o UD-IQ4_XS da Unsloth (~4-bit) | espaço para os maiores tamanhos com todo o resto aberto |
- **[Coder](docs/MODELS.md#coder):** uma versão para programação com metade dos experts removida. Ela alcança 91%
da pontuação do modelo completo no SWE-bench Verified (medido pelos autores) e cabe em 32 GB de RAM. É mais fraca
fora do código, inclusive em chinês e outros textos CJK (#438). Para isso, escolha Q2_0, IQ2_XS ou IQ3_S, que
mantêm todos os experts.
- **[Swift 1.5](docs/MODELS.md#swift-15):** um fine-tune que pensa por bem menos tempo antes de responder. Você
recebe a resposta mais cedo, com quase a mesma qualidade.
- **[Unsloth UD-IQ4_XS](docs/MODELS.md#unsloth-ud-iq4_xs):** a versão de ~4 bits da Unsloth, entre IQ3_S e
UD-Q4_K_XL em qualidade. Um download de 94 GB. Com menos de ~80 GB de RAM, o Strata lê uma parte dele do SSD
enquanto responde, então ali ele fica mais lento (um SSD NVMe ajuda).
- **[Unsloth UD-Q4_K_XL](docs/MODELS.md#unsloth-ud-q4_k_xl-experimental)** (experimental): o mais próximo do modelo
completo. Mas o Strata lê a maior parte dele do SSD enquanto responde, então ele escreve só 7-8.5 tokens/s num PC
com 64 GB.
- **[OrcaRouter's Uncensored IQ3_XXS](docs/MODELS.md#orcarouter-uncensored-iq3_xxs):** você configura à mão. Ele
não está no menu do instalador.
Tamanhos, downloads e o que cabe onde: [docs/MODELS.md](docs/MODELS.md). Para adicionar outro modelo depois, rode
`SETUP.bat` (Linux: `./setup.sh --setup`).
## Como usar
<p align="center"><img src="docs/media/runpagoda.png" width="900" alt="A aba Monitor do app Strata ao lado de um agente de programação"><br>
<sub>O <b>Monitor</b> do app Strata (à esquerda) enquanto um agente de programação escreve o jardim com pagode do vídeo (à direita)</sub></p>
- **No navegador:** abra `http://127.0.0.1:8080`. Lá tem o **Chat**, um **Monitor** ao vivo do modelo e da sua
GPU/CPU/RAM, e o **About** com as configurações e os endereços.
- **Seus apps e agentes de programação:** adicione um provedor "OpenAI-compatible" com a URL base
**`http://127.0.0.1:8080/v1`**. Qualquer chave de API e qualquer nome de modelo funcionam.
- Apps que usam a API da Anthropic: `http://127.0.0.1:8080/v1/messages` (Claude Code:
`ANTHROPIC_BASE_URL=http://127.0.0.1:8080`).
- Codex CLI e outros apps que usam a OpenAI Responses API: `/v1/responses`
- **Raciocínio:** escolha **off, low, medium ou high** no menu do chat ou no "reasoning effort" do seu app. Off é o
mais rápido. High é o melhor para perguntas difíceis.
- **Imagens:** responda sim para "Images?" na instalação. Depois clique em **Picture** no chat, ou anexe imagens no
seu app. Placas AMD entendem imagens no Linux usando o processador; no Windows, ainda não.
- **Do seu celular ou de outro PC:** `START-HERE.bat --setup --host 0.0.0.0 --api-key <secret>`. Sempre defina uma chave.
- **Uma requisição por vez:** por padrão, o Strata responde uma requisição e as outras esperam. Para responder
várias ao mesmo tempo, defina `"parallel": 2` ([BATCHING.md](docs/BATCHING.md)). Numa placa de 12 GB, isso deixa
cada resposta mais lenta.
- **Prompts longos:** o Strata lê a primeira mensagem de uma conversa inteira, cerca de 1 minuto a cada 30,000
tokens. As mensagens seguintes começam em segundos.
Mais: [onde suas conversas ficam salvas](docs/INSTALL.md#where-things-are-stored), [a API](docs/DETAILS.md#using-it).
## Algo deu errado?
- **Meu PC travou na primeira vez que o Strata iniciou.** Isso é normal enquanto ele carrega o modelo. Espere e não
feche a janela. Ainda travado depois de 10 minutos? Reinicie o PC, feche outros programas e tente de novo, ou
escolha um tamanho menor.
- **Parou durante o download ou a instalação.** Rode `START-HERE.bat` (ou `./setup.sh`) de novo. Ele continua de
onde parou.
- **Está muito lento e a luz do disco não para de piscar, ou aparece "the engine stopped unexpectedly".** Seu PC
não tem RAM livre suficiente. Feche outros programas (navegadores usam muita), ou escolha um tamanho menor (Q2_0
ou IQ2_XS).
- **Diz que a porta 8080 já está em uso.** O Strata já está rodando. Procure a janela dele.
Mais problemas e como resolver: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). Ainda sem solução? Abra uma
[issue](https://github.com/Niko1221/Strata/issues) e anexe o `strata-<model>.log` da pasta do Strata. Encontrou um
problema de segurança? Informe em particular: [SECURITY.md](SECURITY.md).
## Como funciona?
Modelos como este normalmente rodam em servidores com centenas de gigabytes de memória de vídeo. Sua placa de vídeo
tem 12-24 GB. O Strata faz o modelo caber **dividindo o trabalho pelo PC inteiro**. Pense numa cozinha: o que você
usa o tempo todo fica na bancada, e o resto espera na despensa.
<p align="center"><img src="docs/media/how-it-works.svg" width="860" alt="Os 24,576 experts do modelo: os mais usados na placa de vídeo, todos na RAM, uma tabela de consulta no SSD"></p>
- **O modelo é uma equipe de 24,576 pequenos especialistas ("experts").** Cada palavra precisa de só 10 deles.
- **Sua placa de vídeo** guarda os poucos milhares de experts usados com mais frequência. **Sua RAM** guarda todos
eles, e **seu processador** trabalha no resto ao mesmo tempo. **Seu SSD** guarda uma grande tabela de consulta.
<p align="center"><img src="docs/media/guess-and-check.svg" width="860" alt="Um pequeno ajudante adivinha as próximas palavras; o modelo grande confere todas de uma vez e fica com as certas"></p>
- **Adivinhar e conferir:** um pequeno ajudante adivinha as próximas palavras. O modelo grande confere todas de
uma vez. Você recebe a mesma resposta, 1.6-1.8x mais cedo.
- **Textos longos são lidos em grandes pedaços** (até 8,192 tokens por vez), a mais de 1,000 tokens por segundo.
A explicação mais longa: [docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md). Cada parte e seus números:
[os detalhes](docs/DETAILS.md#how-it-works) e o [artigo](docs/paper/Strata-Paper.pdf).
## Créditos e licença
O modelo é o [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), da equipe Qwen. Ele foi
comprimido por [ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF), UkisAI (Swift 1.5)
e Unsloth. O Strata usa partes do [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp). Todos os créditos:
[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md#credits). O Strata é de código aberto sob a [MIT License](LICENSE).
Algumas partes e todos os modelos têm licenças próprias ([quais](docs/HOW_IT_WORKS.md#license)).
## Apoie o Strata
O Strata é gratuito e de código aberto. Se ele é útil para você, você pode apoiar o desenvolvimento:
<p align="center"><a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a></p>
"content":"The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each of which accepts a single numeric argument `x` and applies a conditional transformation based on a specific integer constant associated with the function's name. For any given function `ruleN`, if the input `x` is strictly equal to the integer `N`, the function returns `x + (N + 1)`; otherwise, it returns `x - N`. This pattern creates a set of specialized handlers where the \"true\" branch increments the input by one more than the rule's index, while the"
}
},
{
"label":"0.1.30",
"kind":"followup",
"trial":1,
"wall_s":9.036391863017343,
"metrics":{
"prompt_tokens":4451.0,
"reused":4338.0,
"fresh":113.0,
"prefill_ms":1734.0,
"prefill_tps":65.2,
"generated":128.0,
"decode_ms":7293.0,
"decode_tps":17.6
},
"usage":{
"prompt_tokens":4451,
"completion_tokens":128,
"total_tokens":4579,
"prompt_tokens_details":{
"cached_tokens":4338
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"The most relevant boundary case is the strict integer equality check (`x === N`) within each function, which acts as a precise trigger for the \"increment\" branch while defaulting to a \"decrement\" branch for all other values. This creates a highly sensitive behavior where the output depends entirely on whether the input matches the function's specific index; for example, `rule0(0)` returns `1` (the increment path), whereas `rule0(1)` returns `1` (the decrement path), and `rule0(-1)` returns `-1` (the decrement path). The smallest useful regression test should verify this critical"
}
},
{
"label":"0.1.30",
"kind":"fresh",
"trial":2,
"wall_s":23.5992965750047,
"metrics":{
"prompt_tokens":8830.0,
"reused":0.0,
"fresh":8830.0,
"prefill_ms":16290.0,
"prefill_tps":542.0,
"generated":128.0,
"decode_ms":7296.0,
"decode_tps":17.5
},
"usage":{
"prompt_tokens":8830,
"completion_tokens":128,
"total_tokens":8958,
"prompt_tokens_details":{
"cached_tokens":0
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"The source code defines a series of 280 exported functions, named `rule0` through `rule279`, each accepting a single numeric argument `x`. Every function follows an identical logical structure: it checks if the input `x` is strictly equal to the integer value embedded in its function name (denoted as `n`); if this condition is true, the function returns `x + n + 1`, and if the condition is false, it returns `x - n`. Consequently, each function acts as a conditional modifier that either increments the input by one more than its index or decrements it by its"
}
},
{
"label":"0.1.30",
"kind":"followup",
"trial":2,
"wall_s":9.576413626025897,
"metrics":{
"prompt_tokens":9071.0,
"reused":8957.0,
"fresh":114.0,
"prefill_ms":2196.0,
"prefill_tps":51.9,
"generated":128.0,
"decode_ms":7367.0,
"decode_tps":17.4
},
"usage":{
"prompt_tokens":9071,
"completion_tokens":128,
"total_tokens":9199,
"prompt_tokens_details":{
"cached_tokens":8957
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"The most relevant boundary case is the **exact integer equality check** (`x === n`), which acts as a strict gatekeeper for the \"increment\" branch. For any function `ruleN`, the input `x` must be the specific integer `N` to trigger the `x + N + 1` return path; any other value, including `N + 1`, `N - 1`, or non-integers, falls through to the `x - N` return path. This creates a highly sensitive behavior where a single unit change in input flips the output from a large positive jump to a large negative subtraction."
}
},
{
"label":"0.1.30",
"kind":"fresh",
"trial":3,
"wall_s":11.473167807038408,
"metrics":{
"prompt_tokens":4210.0,
"reused":0.0,
"fresh":4210.0,
"prefill_ms":7721.0,
"prefill_tps":545.2,
"generated":128.0,
"decode_ms":3744.0,
"decode_tps":34.2
},
"usage":{
"prompt_tokens":4210,
"completion_tokens":128,
"total_tokens":4338,
"prompt_tokens_details":{
"cached_tokens":0
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each of which accepts a single numeric argument `x` and applies a conditional logic based on a specific integer constant associated with the function's name. For any given function `ruleN`, if the input `x` is strictly equal to the integer `N`, the function returns `x + (N + 1)`; otherwise, it returns `x - N`. This means that for every function, there is exactly one input value that causes an increment by `N + 1`, while all other inputs"
}
},
{
"label":"0.1.30",
"kind":"followup",
"trial":3,
"wall_s":6.953374623961281,
"metrics":{
"prompt_tokens":4451.0,
"reused":4338.0,
"fresh":113.0,
"prefill_ms":867.0,
"prefill_tps":130.3,
"generated":128.0,
"decode_ms":6078.0,
"decode_tps":21.1
},
"usage":{
"prompt_tokens":4451,
"completion_tokens":128,
"total_tokens":4579,
"prompt_tokens_details":{
"cached_tokens":4338
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"The most relevant boundary case is the **exact integer equality check** (`x === N`), which acts as a strict gate for the \"special\" branch (`x + N + 1`) versus the \"default\" branch (`x - N`). This behavior is highly sensitive to type coercion and precision: if `x` is a floating-point number that is mathematically equal to `N` but not strictly identical (e.g., due to floating-point errors), or if `x` is a string like `\"5\"` for `rule5`, the strict equality `===` will fail, causing the function to fall into the default"
}
},
{
"label":"0.1.30",
"kind":"fresh",
"trial":4,
"wall_s":16.981091229012236,
"metrics":{
"prompt_tokens":8830.0,
"reused":0.0,
"fresh":8830.0,
"prefill_ms":12624.0,
"prefill_tps":699.5,
"generated":128.0,
"decode_ms":4343.0,
"decode_tps":29.5
},
"usage":{
"prompt_tokens":8830,
"completion_tokens":128,
"total_tokens":8958,
"prompt_tokens_details":{
"cached_tokens":0
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"The source code defines a series of exported functions named `rule0` through `rule279`, each accepting a single numeric argument `x` and returning a numeric result based on a specific conditional logic. For any given function `ruleN`, the behavior is identical: if the input `x` is strictly equal to the integer `N` (the number embedded in the function name), the function returns `x + N + 1`; otherwise, it returns `x - N`. This pattern creates a set of 280 distinct functions where each function acts as a specialized transformation that increments the input by `N+1"
}
},
{
"label":"0.1.30",
"kind":"followup",
"trial":4,
"wall_s":6.207711412978824,
"metrics":{
"prompt_tokens":9071.0,
"reused":8823.0,
"fresh":248.0,
"prefill_ms":1062.0,
"prefill_tps":233.6,
"generated":128.0,
"decode_ms":5132.0,
"decode_tps":24.9
},
"usage":{
"prompt_tokens":9071,
"completion_tokens":128,
"total_tokens":9199,
"prompt_tokens_details":{
"cached_tokens":8823
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"The most relevant boundary case is the **exact integer equality check** (`x === N`), which acts as the primary switch between the two arithmetic branches. For any function `ruleN`, the behavior changes drastically at the single point where `x` equals `N`. Specifically, if `x === N`, the result is `2N + 1` (since `x + N + 1` = `N + N + 1`), whereas for any other value, the result is `x - N`. This creates a discontinuity in the function's output curve.\n\nThe **smallest useful regression test** should"
"content":"The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each of which accepts a single numeric argument `x` and applies a conditional transformation based on a specific integer constant associated with the function's name. For any given function `ruleN`, if the input `x` is strictly equal to the integer `N`, the function returns `x + (N + 1)`; otherwise, it returns `x - N`. This pattern creates a set of specialized arithmetic operations where the \"true\" branch adds an incrementally larger value (starting from 1 for `"
}
},
{
"label":"0.1.31",
"kind":"followup",
"trial":1,
"wall_s":8.811686859990004,
"metrics":{
"prompt_tokens":4451.0,
"reused":4203.0,
"fresh":248.0,
"prefill_ms":2051.0,
"prefill_tps":120.9,
"generated":128.0,
"decode_ms":6752.0,
"decode_tps":19.0
},
"usage":{
"prompt_tokens":4451,
"completion_tokens":128,
"total_tokens":4579,
"prompt_tokens_details":{
"cached_tokens":4203
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"The most relevant boundary case is the **exact integer equality check** (`x === N`), which acts as a strict gate for the \"special\" branch. For any function `ruleN`, the behavior changes drastically only when `x` is exactly `N`. For example, in `rule0`, if `x` is `0`, it returns `1` (`0 + 1`); if `x` is `1`, it returns `1` (`1 - 0`). This creates a collision where different inputs can produce the same output, but more importantly, it highlights that the function is **not** a simple"
}
},
{
"label":"0.1.31",
"kind":"fresh",
"trial":2,
"wall_s":15.694618315959815,
"metrics":{
"prompt_tokens":8830.0,
"reused":0.0,
"fresh":8830.0,
"prefill_ms":11265.0,
"prefill_tps":783.8,
"generated":128.0,
"decode_ms":4417.0,
"decode_tps":29.0
},
"usage":{
"prompt_tokens":8830,
"completion_tokens":128,
"total_tokens":8958,
"prompt_tokens_details":{
"cached_tokens":0
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"This source code defines a collection of 280 exported functions, named `rule0` through `rule279`, each of which accepts a single numeric argument `x` and implements a specific conditional arithmetic transformation based on a unique integer constant associated with its name. For any given function `ruleN`, the logic checks if the input `x` is strictly equal to the integer `N`; if this condition is true, the function returns `x + (N + 1)`, effectively adding the next consecutive integer to the input, whereas if the condition is false, it returns `x - N`, subtracting the function"
}
},
{
"label":"0.1.31",
"kind":"followup",
"trial":2,
"wall_s":6.075737391016446,
"metrics":{
"prompt_tokens":9071.0,
"reused":8957.0,
"fresh":114.0,
"prefill_ms":835.0,
"prefill_tps":136.5,
"generated":128.0,
"decode_ms":5225.0,
"decode_tps":24.5
},
"usage":{
"prompt_tokens":9071,
"completion_tokens":128,
"total_tokens":9199,
"prompt_tokens_details":{
"cached_tokens":8957
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"The most relevant boundary case is the **exact integer equality check** (`x === N`), which acts as a strict gatekeeper for the \"special\" branch (`x + N + 1`). This is critical because it distinguishes the single \"trigger\" value from all other inputs, which fall into the \"default\" branch (`x - N`).\n\n**Key Boundary & Edge Cases:**\n\n1. **Integer Equality (`x === N`):**\n * **Trigger:** When `x` exactly equals the rule's index `N`, the function returns `N + (N + 1) = 2N"
}
},
{
"label":"0.1.31",
"kind":"fresh",
"trial":3,
"wall_s":6.66396439098753,
"metrics":{
"prompt_tokens":4210.0,
"reused":0.0,
"fresh":4210.0,
"prefill_ms":4679.0,
"prefill_tps":899.7,
"generated":128.0,
"decode_ms":1978.0,
"decode_tps":64.7
},
"usage":{
"prompt_tokens":4210,
"completion_tokens":128,
"total_tokens":4338,
"prompt_tokens_details":{
"cached_tokens":0
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"The source code defines a series of 140 exported functions, named `rule0` through `rule139`, each of which accepts a single numeric argument `x` and applies a conditional transformation based on a specific integer constant associated with the function's name. For any given function `ruleN`, if the input `x` is strictly equal to the integer `N`, the function returns `x + (N + 1)`; otherwise, it returns `x - N`. This pattern creates a set of specialized arithmetic operations where the \"true\" branch adds an incrementally larger value (starting from 1 for `"
}
},
{
"label":"0.1.31",
"kind":"followup",
"trial":3,
"wall_s":7.167138791992329,
"metrics":{
"prompt_tokens":4451.0,
"reused":4203.0,
"fresh":248.0,
"prefill_ms":900.0,
"prefill_tps":275.6,
"generated":128.0,
"decode_ms":6259.0,
"decode_tps":20.5
},
"usage":{
"prompt_tokens":4451,
"completion_tokens":128,
"total_tokens":4579,
"prompt_tokens_details":{
"cached_tokens":4203
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"The most relevant boundary case is the strict integer equality check (`x === N`) within each function, which acts as a precise trigger for the \"true\" branch; for example, in `rule0`, the input `0` is the critical boundary where the behavior shifts from the default subtraction (`x - 0`, which is identity) to the special addition (`x + 1`), making `rule0(0)` return `1` while `rule0(1)` returns `1` (via `1 - 0`), creating a collision that highlights the fragility of this design. The smallest useful regression test would"
}
},
{
"label":"0.1.31",
"kind":"fresh",
"trial":4,
"wall_s":11.244427776022349,
"metrics":{
"prompt_tokens":8830.0,
"reused":0.0,
"fresh":8830.0,
"prefill_ms":8888.0,
"prefill_tps":993.5,
"generated":128.0,
"decode_ms":2345.0,
"decode_tps":54.6
},
"usage":{
"prompt_tokens":8830,
"completion_tokens":128,
"total_tokens":8958,
"prompt_tokens_details":{
"cached_tokens":0
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"The source code defines a series of 280 exported functions, named `rule0` through `rule279`, each of which accepts a single numeric argument `x` and implements a specific conditional arithmetic operation based on a unique integer constant associated with the function's name. For any given function `ruleN`, the logic checks if the input `x` is strictly equal to the integer `N`; if this condition is true, the function returns `x + (N + 1)`, effectively adding the successor of the rule index to the input, whereas if the condition is false, it returns `x - N`,"
}
},
{
"label":"0.1.31",
"kind":"followup",
"trial":4,
"wall_s":4.393163571017794,
"metrics":{
"prompt_tokens":9071.0,
"reused":8958.0,
"fresh":113.0,
"prefill_ms":721.0,
"prefill_tps":156.8,
"generated":128.0,
"decode_ms":3659.0,
"decode_tps":35.0
},
"usage":{
"prompt_tokens":9071,
"completion_tokens":128,
"total_tokens":9199,
"prompt_tokens_details":{
"cached_tokens":8958
}
},
"finish_reason":"length",
"message":{
"role":"assistant",
"content":"The most relevant boundary case is the **exact integer equality check** (`x === N`), which acts as a strict gatekeeper for the \"special\" branch (`x + N + 1`) versus the \"default\" branch (`x - N`). This creates a discontinuity in the function's behavior: for `ruleN`, the output jumps from `N - N = 0` (when `x = N - 1`) to `N + N + 1 = 2N + 1` (when `x = N`), and then back down to `N + 1 - N = 1"
| Storage | Model on a 3.7 TB NVMe at PCIe 4.0 x4 (`/mnt/models`). Disk only matters at startup: during generation the server read 23 MiB for 400 tokens (experts are resident in RAM/VRAM, 100% expert cache hits). Cold start was not measured. |
## Software
- Ubuntu 26.04.1 LTS, kernel 7.0.0-31-generic
- NVIDIA driver 595.84, CUDA 13.3 toolkit libraries (`/usr/local/cuda-13.3`)
- Strata engine 0.1.30 (release download, not a git checkout; no commit hash)
Review the following C++ source file from the Strata engine (v0.1.32, src/core/expert_source.cpp, first 9,000 bytes). List the three most important correctness risks you see, with line references.
// src/core/expert_source.cpp - the adapter. See the header for the three clauses of the contract.
Loaded 100 of 2064 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Niko1221