I/O path only: the same bytes are read, only HOW the experts outside the RAM copy are read (unbuffered or through the file cache) is chosen differently.
file_tier_unbuffered took its `avail` from cgroup_available_bytes, which credits the group's whole `file` counter as reclaimable. On a RAM-budget
config that credit includes pages the engine holds itself: P2003 (RTX 3060, 46 GB, 36 GiB group), UD-IQ4_XS-v31, 22 GiB RAM copy: the re-check read 18.7 GiB
available when the group had 11 GiB left (limit 36, current 24.8). With held = 0.52 >= 0.5 the timed drive probe was skipped and the file tier switched to the file
cache (direct reads 7.9 GB/s vs 2.5 GB/s cached): prompts 2.5-3x slower (pp2k 85 vs 273 tok/s, pp8k 195 vs 580), decode 10-14% slower. The one start that read
12.0 GiB kept the fast mode, which is why the first start of a session was sometimes fast.
Now the smaller of that figure and the conversation cache's sampler (only the clean inactive file cache is credited, the group's own limit, no operator
cap) decides. The --memory-limit-mib cap is not applied: it bounds the engine's own allocations, not the page cache (counting it sent IQ3_S unbuffered and cut its decode
2-3x). A flip now prints once: "strata: expert reads now go through the file cache (X GiB of RAM free for them, Y GiB limit)"; the technical line is behind STRATA_TRACE.
STRATA_UNBUFFERED_LOAD still forces either mode.
Measured on P2003 (build wip11, n >= 3 starts per arm, interleaved fresh engines, I/O-quiet before each start, memory watch on, floor 14336 MiB):
v31 unforced: main flipped in 5 of 6 starts, this change 0 of 9 (6 + 3): pp2k 85-94 -> 263-273 tok/s, pp8k 187-198 -> 580-590, code_py decode +14 to +20%;
v31 DFlash2 vs MTP in the kept state (3 rounds): DFlash2 +2 to +14% per kind (json_extract -3.5%);
IQ3_S: the file cache stays (9 of 9 starts), decode level with main (code_py 17.0 vs 15.6 tok/s, json_extract 14.2 vs 13.4, page cache evicted before every start);
low RAM (6-7 GiB ballast, budget 16 GiB): no regression, MemAvailable minimum 14911 MiB;
memory minimum with the unbuffered state 15.6-15.8 GB (floor 14.3 GB), no arm stopped.
Not P1.8: main and the P1.8 build flip alike.
(cherry picked from commit be3831e9d944fb44ac36d176f4764e30df0f9f6b)
native_w11_kernel takes a Q5_0 down block (d, qh, qs[16]; 22 bytes): the 5-bit codes are rebuilt with the qh bits and kept as the signed int8 q - 16 (no minimum
term), the scale is d. d_covered_on() says yes for T_Q5_0 only on gfx12 with the switch; the layer otherwise leaves the fused path for the gather + FP16
dequantize + hipBLASLt route as before. hip_prefill_fused_iq gets the pairs IQ3_XXS / Q5_0 and IQ4_XS / Q5_0 (reference-checked: the build has no Q5_0 MMQ).
R9700 (p620, v31, DFlash2, same executable with and without the switch, 3 rounds x 2 reps): prompt 2K +44%, 8K +40%, 32K +13%; decode unchanged.
First-token KL against the FP16 route (self-variation 0): 2K mean 0.016, max 0.068, top-1 24/24. Off by default in this commit.
(cherry picked from commit 97c3033f566d27ea1d741d7b893ac13e0f4e8526)
"strata: RDNA4 (gfx12): prompt chunks of 30,000+ tokens use the chunked GDN scan (STRATA_GDN_CHUNKED=0 turns it off)" is printed once at start with the
other gfx12 default lines (owner's wording, 11-06): only on gfx12, only while STRATA_GDN_CHUNKED is unset and STRATA_GFX12_DEFAULTS is not 0.
The gate follows the gfx12 defaults' one off switch, so the line is true whenever it is printed.
(cherry picked from commit b2d1680a864195383697559edb9b6cf8180b9a19)
STRATA_GDN_CHUNKED unset on a gfx12 card (RDNA4): the FP16 matrix-core chunked recurrence runs for prompt chunks of 30,000 tokens or more
(--prefill auto takes chunks up to 32,768 there, so a 32K prompt is one chunk of about 31,996 rows). Shorter chunks keep the kernels they
ran, so every prompt up to 20K is bit for bit what it was. STRATA_GDN_CHUNKED=0 turns it off, =1 is the old opt-in (every chunk of 64
tokens or more), STRATA_GDN_CHUNKED_MIN=N moves the line. Owner OK 11-06.
R9700 (p620, IQ3_S, DFlash2, interleaved pairs against the same tree without the change, 5 rounds x 2 reps): prompt 32K +6.1% (5/5),
64K +6.7% (5/5). First-token KL against the unchanged build (self-variation 0.00000): all 24 prompts of 2K, 8K, 20K and the 8K agent turn
identical; 32K 0.00005, 32K+ 0.00004, 64K 0.00086; top-1 equal on all 30 prompts. No log line yet (wording is with the coordinator).
(cherry picked from commit 7b579367aaa56978232cb60ef405987a199d736f)
In peer mode the adaptive overlap was refused, so every decode window first waited for the previous round's copies. With the overlap the peer
tier issues its copies in layer order, signals each layer's landing the way the primary does (a stream-ordered D2H copy of the layer number into
a pinned int, or host functions), its residency table takes the swaps at the window's start (flip_pending), and a layer waits for a tier's copies
only when that tier swapped in it (AdaptGate: peer_landed, peer_has, per-tier last). The wait before a window clears the peer's marks with the
primary's. Without the variable nothing changes.
Ported from #2720 (bulhenvaster), rebased by hand onto the current gate (the watchdog label stays). Not ported: STRATA_PEER_LAUNCH_IN_POOL
(the author measured no gain on his build), the new decode timing line and the documentation paragraph.
(cherry picked from commit e78b64a7348d685dfd11799fdb395661f2120436)
With --peer-device the prompt path never gave the CPU pool a share (cpu_maybe required !m.pp), so a short read at long context streamed every
expert neither card holds over the primary's link while the CPU sat idle. With the switch the share runs beside the peer: it takes only experts
neither card computes, its rows go between the primary's and the peer's block (the peer's stays the tail its combine reads from back_at), and the
staged chunks go up to cpu_share_max()'s 3072 tokens. Without the switch nothing changes.
Ported from #2721 (bulhenvaster), rebased by hand onto the current prompt path (the default-armed share's limit and the routed-stream gate).
(cherry picked from commit c8b6c43c1f2653636d15fe14abf53112357570aa)
Decode reads missed expert blobs from host RAM three ways; the PCIe share's
blobs cross into VRAM anyway and were dropped after the window. With the
switch set, the share takes each layer's most-routed misses and a fetched
expert routed enough more than the layer's least-routed resident expert that
the window does not route is copied straight into that expert's slot
(fetch_blobs gets an optional dst2 target per blob). =2 leaves the tier's own
swaps out of a layer where an admission ran. STRATA_ADAPT_FETCH_VERIFY=1 reads
every admitted slot back. Off, with a log line saying why, where it cannot run.
Unset changes nothing.
Ported from PR 2964 (sergqwer) onto f-0216-core; the strata generate verify
loop was merged by hand (main's speculative-q lines kept) and the DETAILS.md
note became a row of the opt-in switch table.
l620 (RTX PRO 5000, IQ3_S, 3,896-slot cache, --pcie-frac 0.25): decode =2 +10.2%,
=1 +5.0% (4 rounds); token forcing over 14,167 tokens: mismatch 2.73% vs 2.53% /
2.53% for two unset arms (+0.19 pt, 90% CI -0.02..+0.42), PASS.
(cherry picked from commit b2bfaef906acee4f86bdf9f052aee6a4d295fb4e)
Prompt-path block scores in single-pass TF32 with a per-block error bound,
3xTF32 rescoring only near each query's top-k cut-off, so qsa_block_topk
returns the same cells in the same order, ties included. Off unless the
variable is set; says once where it cannot run (HIP, Metal, SYCL, layer-major
path). Adds qsa_two_pass_test, qsa_two_pass_status_test and the GPU parity
test qsa_two_pass_parity.
Ported from PR 4119 (bitaccesscomau); the log line of the per-prompt counters
is shortened without dropping a field.
l620 (sm_120, IQ3_S), greedy ids identical off vs on in every pair:
prompt 128K +1.65% (n=12), 256K +4.10% (n=8), 32K -0.14% (noise).
(cherry picked from commit bbf7c55f1351d913233250fe7869642f7c2b98a1)
drafting.CLASS_DEFAULTS: one removable entry per set, read from the signals the confidence already reads (a CUDA card whose engine slots are under the last-but-one CONF_STEPS step).
IQ3_S: on P2003 the file cache makes the MTP faster than DFlash2 by 12-44% (RTX2 t4b). UD-IQ4_XS-v31: DFlash2 beats the MTP by about 6.5% (t8, 3 starts; DF-VAL forced-fast tuned 0.8 x1.037).
The v31 entry DEPENDS ON RTX2's FILE-TIER FIX (unbuffered reads) BEING IN THE SAME RELEASE; without it remove that entry.
Anything chosen by name wins. Resident CUDA, Strix, gfx1201, Arc and Metal entries and the conf tables are unchanged. The sm_86 support line reads the set: UD-IQ4_XS-v31 gets the existing
'faster on some kinds of text and slower on others' line, every other model keeps 'slower'.
(cherry picked from commit 42412d42ff835d6a66a78fbf6d119911908eda1b)
One more SET_DEFAULTS line for the measured cell (DF-VAL, setup's config, budget 38, 3 rounds: tuned 0.35 against stock 0.35 x1.046, 3 of 3 rounds, every kind 3 of 3).
(cherry picked from commit c2b801157e71a503da859b66baff12fff28cd299)
The SYCL 0.2.11 engine reads only bf16 selector codebooks, so Arc gets its own tuned file (qwen3.8-flash-next-dflash2.strata1.q4_0-bf16sel.pack, 522,458,112 bytes, own SHA-256 pin) through
SetDefault.pack. Only IQ3_XXS, the set measured on the A750 (tuned 0.6 x1.054, 3 of 3 rounds); other Arc sets keep the stock pack. SYCL never gets the selq4 file, CUDA / HIP / Metal never the
bf16sel one. NOTICE names the second file and no longer overclaims about the text.
(cherry picked from commit 6111c583b70e3c88a3d54039e13d76b19f15742d)
11 of 50 engines on 0.2.11 never got past the prompt on this card class. Only Auto changes: DFlash2 chosen by name, a saved choice and the person's own options still win; gfx1151, other AMD, CUDA, Metal and SYCL and the confidence tables are unchanged. The support line under the Drafting dropdown says it in plain words (app catalog entry in all nine languages).
(cherry picked from commit 8b4398743aa67368d72277310bbe8093bbe4923c)
SET_DEFAULTS entries carry a card condition (tuned_applies: the CONF_STEPS first-step slots or the Spark's shared memory for CUDA, gfx1151 for HIP); no entry names a conf, so
conf_for / CONF_STEPS / BACKEND_CONF run as before and only the pack changes. Enabled only for the cells measured at the shipped conf (CUDA-resident IQ3_S and UD-IQ4_XS-v31, Strix IQ3_S, UD-Q4_K_XL, UD-IQ4_XS).
12 GB and 24 GB cards, several cards, gfx1201, other AMD and SYCL never fetch or use it. Fetch keeps tuned, then stock, then MTP; silent offline; feed backends are asked per start's backend.
(cherry picked from commit f096dcd750b5928ace6f9eca1e2936b9e35578d4)
EXPERT_WINDOW_SLOTS (2 * MAXT * TOPK) is the slots a size pool must hold for one window's picks to have a place; ExpertCache's check, the residency
planner's floor and the arena's keep all read it. expert_residency_test now also covers the GSQ-RCO IQ3_S set's real pools (7, two of them a single
layer), a sweep of RAM targets against a copy of the old global ranking as the negative control, a refusal beyond what the floors allow, and that
the card's sliced 16-byte-padded strides partition the layers like the whole-expert bytes do. Class E (placement only).
Mixed-quant models can fail to load in the split engine after cold-expert offload: the global residency ranking can leave one size pool below the
2 * MAXT * TOPK slots Up::expert_cache() needs even while other pools can still give up experts. Group experts by both the gate/up and the
down byte strides, keep the cold-first ranking but skip eviction from a pool that has reached the window floor, count only the extra evictions
that really happen, and say so before the model is uploaded when the request cannot be placed.
Ported from #5158 (midhatn) unchanged; the tests and the engine check follow in the next commit.
With --trim-stage-weights every later stage's HC_Q8 loop over the GGUF copied the rows of all 48 layers and read its own. The loop now skips a
tensor outside the stage's layer range, the filter the upload loop above it and the bf16-pack path below it already use. The Q8_0 rows copied
and the bytes not copied are recorded in the run manifest (STRATA_MANIFEST). STRATA_HC_Q8_ALL_STAGES=1 copies every layer as before.
No value changes: a stage never reads another stage's layers; without a trim the range is unset and nothing is skipped. Class E.
Co-authored-by: havok2-htwo <[email protected]>
Niko1221