tools/idle_gpu_ab.py is the 0.2.1 split A/B harness with a per-card nvidia-smi sampler. Result on the 0.2.1 engine: the two-card
split is 47-49 % idle in decode (cards take turns), the single card 5 %; STRATA_SPLIT_OVERLAP, --pipeline-windows 1/2 (with and
without the draft layer on card 0) and --batch-groups 2 give no gain worth a default. docs/MULTI_GPU.md: the overlap flag is now
measured on NVIDIA.
Merge resolution as in f-145-perfprs-withvolta: iq_kernels.cu (IQ3_XXS / IQ3_S staged sign-mask tables over the EXPDEC grid staging), prefill/kernels.cu and prefill/prefill.cpp keep both sides; CMakeLists keeps both test blocks.
The verifier reads the hyper-connection twice per layer at 1-8 tokens: three small launches with two global
dependencies, the down projection on 41 blocks. Opt-in STRATA_HC_PERSIST=1 runs it as one kernel of 80 blocks with two
grid barriers on an atomic counter: the down projection split by column slices over all 80 blocks, a distributed
reduction, then the up projection and the same epilogue as gr_up_kernel.
gfx1030 (2x RX 6900 XT, layer split): the read 1.35-2.10x at 1-8 tokens; ms per verify window -4.2 % at 4K and
-2.8 % at 32K (one binary, the variable unset / set, 6 alternating rounds, every round the same way). Another summation order (not bitwise); against a
double-accumulation reference (STRATA_HC_PERSIST=3) as close as the default read.
Residency is decided by running every token count once at start-up with a 50 ms barrier timeout and comparing with
the default read; a timeout keeps the default read for that token count and larger ones, wave64 cards are refused.
One line per card says what was decided. Only the verifier's reads use it (one stream per card); the drafter's reads,
the prompt path, CUDA and SYCL are unchanged (CUDA and SYCL get two empty stubs). With the variable unset both entry
points return in their first line: the output is bitwise the same as main.
Measurements, the simulated smaller cards and a stand-alone timing tool:
bench/results/2026-10-12-hc-persist-gfx1030/.
The Radeon 780M (gfx1103, RDNA3, 12 CUs) builds and runs (issue #1423), but had no hipBLASLt table, so the prompt's
dense GEMMs ran on plain hipBLAS. This adds tools/hip/gfx1103-hipblaslt-100401.txt, calibrated on one 780M with
ROCm 10.1's hipBLASLt 1.4.1 (version number 100401) over the gfx1150 table's 144 GEMM cases at T = 64..16384 with
the engine's 32 MiB workspace.
Two tune_hipblaslt passes back to back; each row keeps the solution with the lowest median of all six repetitions,
and a row is left out when no candidate beats plain hipBLAS. 143 rows, 1.05-23.30x per GEMM, geometric mean 4.95x;
the one shape left out is bf16 N=1 K=2560 T=64 (best 0.039 ms against 0.024 ms on plain hipBLAS, sub-100 us
timings), where the engine falls back to hipBLASEx.
Measured on the 780M (Ryzen 7 8745HS, 60 GB, GTT 50 GiB, Q2_0 + MTP, 32K context): a 4K prompt read 90 tok/s on
plain hipBLAS against 187 with the table (medians of an interleaved A/B; with the gfx1103 exact switches and the
WMMA opt-ins, 277 at 4K and 272 at 32K). hip_prefill_hipblaslt_gemm passes with it (launches=4 fallbacks=0) and
hip_prefill_hcd_exact_parity skips, because the table picks no 1176 / 1177 for the HC down GEMM - the gfx1150 table
gets the same skip, which is why STRATA_HCD_EXACT is not in the gfx1103 switch set. Setup installs for gfx1103 only
with STRATA_EXPERIMENTAL_GFX1103=1 and then picks the table up when the installed hipBLASLt reports 1.4.1.
Docs, the community report (runs.csv, tune-summary.csv, the server config, the engine's start-up lines) and the
setup tables list carry the numbers. Refs #1423.
The scorer of 65ce329 is plain FP32 arithmetic, so it runs on any HIP card as it is; without matrix cores the
prompt path otherwise scores on the warp kernel. Its tiling becomes template parameters: every tiling keeps each
score's fmaf chain over k = 0..127 in order and the same relu sum over the heads, so every tiling gives the same
bytes (RX 6900 XT, qsa_select_bench at 32K and 128K: 14 tilings, also on 60 and 72 of its 80 CUs through
HSA_CU_MASK, all byte-identical to the original kernel; the fastest tiling the same at every width). CUDA keeps
the tiling it was written with; gfx103x runs 32/128/16/8: 256 queries at 128K 2.69 ms (warp kernel) -> 0.695 ms
(the original tiling) -> 0.587 ms. FP64 gate passed, selections identical 256/256. On the 2x RX 6900 XT layer
split a 128K-token prompt reads 7.4% faster, 32K 0.8%. A HIP ctest runs it at 131K.
Rebased onto 296b59d2: the docs/AMD_HIP.md conflict (main's SGEMM/rocBLAS-table bullets from the ports before it) is resolved by keeping all bullets.
Co-authored-by: xjc10 <[email protected]>
rocBLAS picks the kernel for each FP16 product from its own table by shape, and on gfx1030 that pick is far
from the best kernel the same library holds for some shapes: N = 10240 K = 2560 (the GDN input projection)
below T = 1152 runs at 5.2 TFLOPS where the best of rocBLAS's 237 solutions for n = 799 is 6x faster (not an
alignment effect: n = 800 is as slow as 799).
tools/hip/tune_rocblas enumerates the library's solutions per shape and token bucket, checks each against the
default kernel's output and that it also runs at smaller odd token counts, and writes the winners (and
`default` rows bounding their reach) as tools/hip/<arch>-rocblas-<version>.txt; Gemm::f16_inplace runs the
table's solution for a shape (STRATA_ROCBLAS_TUNING: a file, or the directory from which the engine takes the
file of its own rocBLAS build; the version string, tweak hash included, is checked). One tiny default product
at startup opens rocBLAS's kernel library (a solution-index call before that fails although the index is
valid); a refused solution falls back to the default kernel for that call, after three refusals for good.
setup points the engine at tools/hip when a table for the card's architecture exists. Shipped: the gfx1030
table for ROCm 10.0.0's rocBLAS 5.6.0.8d1ae90e (112 rows, 76 with a solution).
Measured (RX 6900 XT, 6f32ec0 + #835 + this, temperature 0, the same answers with and without): a 799-token
prompt's GPU time 3,329 -> 2,985 ms (-10.3%), 495 tokens 2,477 -> 2,139 (-13.6%), 8,874 tokens 11,978 ->
11,459 (-4.3%); a PCIe-bound 1,108-token prompt unchanged (its saving reappeared as `wait copy`). ctest 66 of
70 (the 4 failures are outside this change); hip_prefill_rocblas_gemm, the new smoke test, passes with 0
fallbacks. Builds on #835. bench/results/2026-10-05-rdna2-rocblas-table has the per-run lines and the tuner's
summary.
(cherry picked from commit e35e103604)
Rebased onto 296b59d2: conflicts in setup.py (SETUP_ENV gained STRATA_ROCBLAS_TUNING beside the 0.1.44 entries) and docs/AMD_HIP.md (kept both sections) resolved.
Co-authored-by: xjc10 <[email protected]>
The README's range check said "none beyond 65504" over 110K tokens without
the numbers. The engine now has the counter: STRATA_F16_RANGE=1 records,
per device, the largest |value| and the count beyond 65504 (or not finite)
where a value enters or leaves FP16 - the activation images (hf_sat, the
image writers' one funnel), the BF16 weights converted to FP16
(bf16_to_f16_rows) and the FP16 GEMM outputs as they are widened
(widen_rows_f16: an output rocBLAS wrote beyond the range is Inf there) -
and prints one line per prompt. Off, the cost is one flag read per value;
on, a 34.4K-token prompt read at 889.8 tok/s against 888.5. to_f16 (the
hc image for the FP16-weight GEMMs, the PLE key image) saturates like the
other image writers now, so every FP16 image is under the same rule. The
HIP shim gains cudaMemcpyFromSymbol.
Measured on one RX 6900 XT over 466K tokens in nine prompts (corpus
slices, two prompts built to push activations - delimiter floods, a token
repeated 4,000 times, 6,000-character runs, base64, five scripts - and
four batches of real requests): outputs peak at 409.5 (160x below 65504),
activations at 112.2, the BF16 weights at 11.56; nothing beyond the range,
nothing non-finite. The prompts built to push activations stayed below
the ordinary corpus. One model (GSQ-RCO IQ3_S); the table is in the
README's range section.
The README also gains the same-card comparison with #1006, which takes
these GEMMs as FP32 SGEMMs: 8.4K / 32.8K / 104K-token prompts 412 / 455 /
456 tok/s on the old path, 622 / 742 / 750 with #1006, 730 / 900 / 904
with this change (the card's FP32 rate against its FP16 rate), with the
phase times and what the SGEMM route covers that this change does not.
(cherry picked from commit e780c8581e)
The test ran only the FP32-out GEMMs, so the gfx103x prompt path's FP16-out
GEMM (f16_inplace, widen_rows_f16) and bf16()'s W conversion through the
scratch were not under ctest. Every shape now runs a second time with
set_f16_io(true), forced on whatever HIP card runs the test: X as the FP16
image for the BF16 products, a scratch of 64 rows so a 96-row weight converts
in two slices (the second written at Y + 64), ldy > N, an odd N (257), T = 1,
and a beta = 1 f16() call, which keeps the FP32-out GEMM. FP16-out results are
checked against the double reference at FP16's rounding (6e-4 relative per
element plus 1e-4, rel L2 < 1e-3); the row padding and the guards around Y
must be untouched, as before. The FP32-out runs keep their tolerance.
Found while adding them: the BF16 inputs were all zero. hip_bfloat16(float)
produced data 0 in this test's build (ROCm 10.0, gfx1030; not traced further),
so the two BF16 cases passed on nothing (rel L2 printed 0). The BF16 bits are
now made by hand (round to nearest even on the float's bits); with real inputs
the BF16 cases pass at rel L2 1.9e-07 (FP32 out) and 2.2e-04 (FP16 out).
On an RX 6900 XT (gfx1030, ROCm 10.0): 11 of 11 pass.
(cherry picked from commit 4c6c9fe374)
CUDA events over one gdn_recurrence call per side (S=128, 16 q/k heads, 48 v heads, C=10240,
T=1024/4096/8192), all four paths and the max abs diff between the chunked wmma output and the
pipelined recurrence. RESULTS.md records the table and the error bounds for
bench/results/2026-10-02-gdn-chunk (measured on the V100-SXM2-16GB, CUDA 12.8, sm_70 build).
Fusions (all bit-exact vs the pipelines they replace; bench/micro/fusions_parity.cu
checks every one bitwise and times the before/after at decode shapes):
- gate/up dual GEMV + SwiGLU in ONE kernel (llama.cpp's mul_mat_gated):
s_gemv_pair_silu, one read of the activation for both projections and the
SwiGLU in the epilogue - the up intermediate never lands in memory. Rows are
accumulated by the shared row functions (s_rowdev.cuh) exactly as the
standalone kernels accumulate them. shared_expert uses it whenever both
projections are S-family (bitwise checked for all three silu kinds).
- silu(gate)*up + the intermediate quantizer fused (was 2 kernels):
swilu_quantize_q8_0/_scaled/_q8_K, warp-parallel swilu then the verbatim
quantizer body. Wired into the S2 expert entry points and shared_expert.
- every activation image of x in one pass (was 2-3 kernels): quantize_act_images
produces block_q8_0 + block_q8_K + bf16 from one read of x. Wired into
gdn_layer, qsa_layer and moe_shared.
- multi-weight bf16 GEMV (llama.cpp's mul_mat_multi): bf16_gemv_pair computes
two matrices from one read of x; used for ssm_alpha/ssm_beta and for the
indexer's q/k projections (q and k now one launch).
The call sites are gated on STRATA_EXPERIMENTAL_SM60 (the Pascal/Volta build),
so the ready-made engine runs exactly the pipelines it did before; the fused
kernels are the same code either way.
Measured (V100, CUDA events, n_embd 2560 / FF 640, 200 reps):
expert intermediate (swilu+q8_0) 15.7 -> 11.6 us M=1, 18.8 -> 11.4 M=8
activation images (q8_0+q8_K+bf16) 46.2 -> 30.4 us
dual GEMV + swilu (gate,up) 30.4 -> 14.8 us
bf16 pair (alpha,beta) 24.2 -> 14.4 us M=1, 23.7 -> 14.1 M=8
Parity: fusions_parity 0 failures (bitwise), plus s_gemv/s_gemv_q8k/s2_gemv_q8/
s2_expert_grouped/shared_expert/quantize_act/bf16_gemv/elementwise/mmvq_multi/
iq_multi/native_expert parity suites all pass unchanged.
Port of oldlocal commit 4672cf9 ("kernels (sm_70): the prompt attention's
latency pipeline - 2.2x over the FP32 kernel at 32K") onto the #600
prompt_attn_v70_kernel, the m8n8k4 Volta kernel now inline in
qsa_prompt_attn.cu. Still gated on STRATA_EXPERIMENTAL_SM60, so the
ready-made engine keeps exactly the code it had.
Two of the commit's three changes port; the third (splitting the dependent
WMMA chains into f0/f1 and fh/fl) is already inherent to the mma884 kernel -
its scores run two tg[rb] chains and its p.v four tmp[rb][nt] chains - so
porting it would only change the summation order. The two ported, measured
on a V100-SXM2-16GB (CUDA 12.8):
* the staging loop issues every global load of a thread before any of its
stores (2-4 in flight), instead of paying the HBM latency per piece;
* the next chunk's ids/page-table lookups double-buffer (row[2][CH]) and run
just before the previous p.v, where their latency hides.
The original commit measured, on the #371 kernel, 2,048 queries of a 32K
selection: 34.4 -> 15.8 ms (2.18x) int8, 33.5 -> 15.4 ms (2.17x) fp16 KV,
35.6 -> 20.3 ms (1.76x) K8V4 (its q4_0 V dequantizes to FP16 at staging, the
sm_75/80 kernel's same trade).
This port, same 2,048 queries of a 32K selection (qsa_prompt_attn_sm70_bench,
which also gained the K8V4 pools): the Volta kernel goes int8 12.11 -> 10.25 ms,
fp16 KV 25.70 -> 17.39 ms, K8V4 25.71 -> 24.12 ms, so against the FP32 decode
kernel int8 rises 1.73x -> 2.05x and fp16 0.80x -> 1.18x (K8V4, 0.85x -> 0.91x,
keeps main's #600 dequant-to-FP16 trade).
qsa_prompt_attn_parity passes all four cases. An A/B against the unpatched
kernel shows the kernel's output is unchanged - every FP64 and new-vs-old
figure is identical before and after (int8 5.08e-6, fp16 5.29e-6 vs FP64,
within the 4x gate).
The experimental build (-DSTRATA_EXPERIMENTAL_SM60=ON, Pascal/Volta) gains a
kv_gather_q8_wide_kernel: one uint2 (8 B) load of codes, one uint4 (16 B) store of
the dequantized fp16 and one scale per thread instead of two. The 8 values lie in
one 64-value group and the arithmetic is unchanged ((float) code * scale ->
f16_from_f32), so the output is byte-identical. The ready-made engine keeps exactly
the 4-wide kv_gather_q8_kernel and launches it unchanged, so its gather is bit for
bit the release engine's. On a device below sm_75 the wide kernel is selected at
run time inside the experimental build (STRATA_Q8_GATHER_WIDE=0 opts out).
Measured on 2x Tesla V100-SXM2-16GB (SM 7.0), CUDA 12.8, one card: +11.4% on the
32k gather microbench (15.85 -> 14.05 us/call, 401 -> 452 GB/s). kv_q8_parity
stays OK at worst INT8-vs-FP16 error 0.590 quantization steps.
On Windows cudaHostRegister of the whole arena fails (the WDDM shared budget is about half the RAM) and the
sliced fallback registers the first ~30 GiB: with Qwen3.8-Flash-Next IQ3_S layers 0-31, so layers 32-47 never
get a PCIe share, while part of the registered prefix holds GPU-resident experts that DMA never reads.
STRATA_PIN_BY_NEED=1 (the arena on one GPU):
- every blob gets a page-aligned stride (PadMap in pinned.hpp, ArenaExpertSource::arena_off, +0.15 % RAM), so any
set of blobs can be registered exactly; the loaders place each blob at its padded offset
- the expert profile's pairs past the estimated GPU-resident count are merged into ranges by rank and registered
before any page is touched, up to STRATA_ARENA_PIN_GIB (29 GiB by default on Windows) or the first refusal
- pinned() / device_alias() / pcie_layer() work on the ranges (a blob is pinned only if wholly inside one);
lock_resident_ranges() keeps the holes between them resident
- one start-up line: "pin by need: N ranges, X GiB pinned, Y % of the non-resident experts"
STRATA_PCIE_MIN1=1: the PCIe share (nmiss * pcie_num) >> 8 rounds to 0 at 1-2 misses per layer; with two or more
misses one goes over PCIe (when its blob is pinned).
RTX 3090 + Ryzen 5 5600X: 28.96 GiB registered = 92 % of the non-resident experts, all 48 layers with a PCIe
share; prompt read +5-7 % at 56K-117K tokens, decode level alone and +2-3 % with --adapt-async 1 and
--pcie-frac 0.6 (bench/results/2026-10-11-community-rtx3090-5600x-arena-pin-by-need, docs/DETAILS.md).
Both opt-in; without them the layout, the loaders and the sliced pin are those of v0.1.42.
In the arena mode the blocking adaptive tier creates and joins a thread every verify window and the next window
waits for its copies (#1863: the time outside the decode timing breakdown). --adapt-async 1 now also runs with the
arena on one GPU, not only in the resident RAM mode. The arena holds every expert, so a round is a copy-in only:
no copy back, no RAM move, no exchange buffers.
- gates for the arena; ArenaExpertSource::blob_at() reads a blob from the job thread without the read counter
- the evicted slot is marked non-resident and uploaded before the round's copies are posted; an expert becomes
resident only once its copy's event has completed
- STRATA_ADAPT_ASYNC_WAIT=1 (arena default): the next tick waits for the round, so the window an expert turns
GPU-resident does not depend on timing; STRATA_ADAPT_PICK_INLINE=1 (arena default): one round per window;
STRATA_ADAPT_ASYNC_BATCH=N caps the copies per tick (default 0 = all); arena rounds are not capped at 96
- adapt_quiesce(): every K/V grow, kv_quiesce and the VRAM command land the round in flight (they only called
apply_pending(true)); a failed round still drains its copies. This also covers the resident mode's async tier.
- STRATA_DECODE_TIMING: "adapt wait X join Y, Z swaps/window"; --adapt-async help text
RTX 3090 + Ryzen 5 5600X, cold prompts: x1.07-1.11 decode, faster on 12 of 12 prompts; 25.9 -> 21.6 ms per window
(bench/results/2026-10-11-community-rtx3090-5600x-arena-async, docs/DETAILS.md).
Opt-in (--adapt-async 1, default 0); the default path is that of v0.1.42. With the tier on, answers are not
bit-exact run to run (which window first computes a swapped-in expert on the GPU).
bench/results/2026-10-12-p40-sm61-kernels: where a prompt and a decode window spend their time on a P40, the kernel
benches and parity runs, every request of the engine A/B (565-token prompt 205 -> 255 tok/s, decode 34-35 -> 37-38
tok/s; with STRATA_DENSE_F32=1 a 7K prompt 360-432 -> 464-474 tok/s), the byte comparison of the answers, and what
was not tested. docs/OLDER_GPUS.md: the sm_61 section and switches, and two traps when measuring on such a card.
On Linux the CUDA (and HIP) runtime creates its host-callback thread lazily, at the first cudaLaunchHostFunc, and a new
thread inherits the creator's CPU mask. The decoding host pins itself to one CPU (SessionLoopScratch::init), so a thread
created after that - which the adaptive overlap's per-layer callbacks did - ran on the host's CPU and fought the host's
spin-wait: ~1000 us instead of ~4 us per callback, decode 2-4x slower with STRATA_ADAPT_OVERLAP on (#1964, #1975).
oal's warm-up (one no-op callback + sync before ExpertPool) is now unconditional on Linux; the answers, swaps and
residency are untouched. Windows CPU Sets are per thread and not inherited, so Windows is unchanged.
Record archived-control output parity, verified candidate process, tools, CPU vision and cache restoration. No fresh 200K speed comparison or production promotion is claimed.
Preserve FP operation order and invert a bijective shared-query layout at its vector loads. Limit the path to the current gfx906 wave64 device, INT8 pools and warp scoring. Default remains off.
Add deterministic GPU parity/timing and host-guard probes, plus scoped component and model receipts. The clean upstream build passed both-card byte checks; model gains are recorded separately on the frozen experimental stack.
Niko1221