On Windows HIP, an auto expert cache with no --vram-reserve-mib kept a floor of 2560 MiB free on any card of 15 GiB or more (#1376, made for a 20 GB discrete
card that paged). On a Strix Halo that room is not free: the RAM budget's page-locked complement takes it straight back (cache + complement 71.72 GiB at 2560 and
71.73 GiB at 700 on UD-Q4_K_XL), so the prompt path's own buffers found 0 MiB and the engine stepped its chunk down 16384 ... 1024 for ever; strata launch of a
UD-Q4_K_XL config with no reserve (pictures off, or a model with no picture reader) never started.
The floor now goes through strata::core::wddm_auto_reserve_mib(integrated, total, current, floor): a discrete card of 15 GiB or more still gets 2560, an
integrated GPU (the device property the UMA sizing already reads) keeps the engine's default 700. Discrete cards, Linux and every config that passes
--vram-reserve-mib are unchanged (the block is under _WIN32 + STRATA_USE_HIP and behind !vram_reserve_given). Unit test in uma_cache_test.
CUDA sm_120 sm build (vendored ggml), base vs change from the same path: the binaries are bit-identical (sha256 2b603395...).
Strix (gfx1151, public-0.2.12-style config with no reserve) with the new engine: starts in 77 s and answers 3 chats; the old engine never started it.
Strix Halo (gfx1151): engine cap 5 -> 8 and the Terminal writes --spec 8; conf stays 0.35.
Measured (5 interleaved pairs, 5/5 faster): UD-Q4_K_XL +5.3%, UD-IQ4_XS-v31 +2.1%.
UD-Q4_K_XL greedy output differs from the old width's (deterministic run to run); v31 is identical.
Blackwell sm_120 (cc 12.0) with at least 48 GB: cap 6 -> 8, --draft-conf 0.55 (RTX PRO 5000 72 GB,
IQ3_S +2.4%, 5/5 pairs, output identical). The DGX Spark (cc 12.1) and consumer cards under 48 GB keep
conf 0.35 and cap 6. dflash2_auto_cap now takes the compute minor and the card's memory; drafting.py
writes --spec 8 only on these two classes (window_for), so no other card's effective window changes.
draft_cap_test, tests/terminal drafting tests (new test_drafting_card_window.py) and the snapshot
goldens (nv_resident and strix_halo_win only) are updated.
The warning now reads "the experts need X GiB of RAM and Y GiB is free, but free memory, file cache and reserved huge pages can hold
them, so they load anyway (same answers). STRATA_ARENA_FILE_CACHE=0 reads them from the file instead." It and the "not enough RAM for
the expert arena ... reads the experts from the file" notice go to stdout only with --serve (before READY, which the server ignores; a
service log has them). A plain generate run keeps its token stream clean: stderr only, as before the PR.
Review of #5531 on top of the contributor's two commits. The credit read Cached - Shmem - Dirty as droppable, but file pages the
process itself or an engine still exiting has locked (the PLE table's mlock is a mapping of the model file) sit in Cached and the kernel
cannot drop them; MemAvailable leaves them out. An engine with --ple-io ram and a short arena was told its own locked table could serve
the arena - the shape of #4051's stall. Now: Mlocked is subtracted; HugePages_Rsvd is subtracted from the free pool; the pool counts only
at the 2 MiB size the arena maps; STRATA_NO_LARGEPAGES (no MAP_HUGETLB try) credits no pool; and the pool must hold the padded mapping
(the pin-by-need stride padding plus the spare blob), not want + blob. Tests for each.
qsa_real_shapes() (24 / 2 / 256 / 64, indexer 4 x 128) is the shape every QSA parity program builds from; it now reads kCompiledGeometry (the same values for the default build), so the
q4_* builds' kv_stream / qsa_prompt_attn / snapshot checks run at their own head counts instead of being refused. gdn_parity's prefill cases used 48 value heads; native_grouped_parity's
compiled-shape loop no longer asks the K-quant checker for an IQ4_NL down.
GU_ROWS_K, D_ROWS_K and the 1280 / 2560 weight rows of the int8, WMMA and F16 kernels come from kMoeEmbd / kMoeFF /
kMoeGuRows, with a static_assert that a product's rows are whole work items. Flash-Next's device code (gfx1151)
is unchanged.
moe_fused.hpp gets kMoeEmbd, kMoeFF and kMoeGuRows from kCompiledGeometry (2560, 640, 1280 for Flash-Next).
moe_f16_common.cuh and moe_f16_kq.cu take GU_NFB, D_NFB, the activation and H row strides, the stage counts and the down
store stride from them, with a static_assert on the divisibility the tiles need. The default geometry's device code
(gfx1151) is unchanged.
Removes AdaptiveGate, df2_gate_on() and df2_gate_row_ms() (window_rule.hpp), their three
sites in generate.cpp and their block in window_rule_test.cpp. The opt-in gate measured
2.7-4.9% slower; with it off (the default) the window rule is unchanged. The register
marks both retired in 0.2.13.
proposed_window_mib / proposed_reserve_mib and their two constants go to
research/stage-reserve-proposal/ with a README and their own checks; the
production header keeps reserve_mib / room_bytes, and stage_reserve_test keeps
its real-arithmetic checks unchanged.
In peer mode the adaptive overlap was refused, so every decode window first waited for the previous round's copies. With the overlap the peer
tier issues its copies in layer order, signals each layer's landing the way the primary does (a stream-ordered D2H copy of the layer number into
a pinned int, or host functions), its residency table takes the swaps at the window's start (flip_pending), and a layer waits for a tier's copies
only when that tier swapped in it (AdaptGate: peer_landed, peer_has, per-tier last). The wait before a window clears the peer's marks with the
primary's. Without the variable nothing changes.
Ported from #2720 (bulhenvaster), rebased by hand onto the current gate (the watchdog label stays). Not ported: STRATA_PEER_LAUNCH_IN_POOL
(the author measured no gain on his build), the new decode timing line and the documentation paragraph.
(cherry picked from commit e78b64a7348d685dfd11799fdb395661f2120436)
Decode reads missed expert blobs from host RAM three ways; the PCIe share's
blobs cross into VRAM anyway and were dropped after the window. With the
switch set, the share takes each layer's most-routed misses and a fetched
expert routed enough more than the layer's least-routed resident expert that
the window does not route is copied straight into that expert's slot
(fetch_blobs gets an optional dst2 target per blob). =2 leaves the tier's own
swaps out of a layer where an admission ran. STRATA_ADAPT_FETCH_VERIFY=1 reads
every admitted slot back. Off, with a log line saying why, where it cannot run.
Unset changes nothing.
Ported from PR 2964 (sergqwer) onto f-0216-core; the strata generate verify
loop was merged by hand (main's speculative-q lines kept) and the DETAILS.md
note became a row of the opt-in switch table.
l620 (RTX PRO 5000, IQ3_S, 3,896-slot cache, --pcie-frac 0.25): decode =2 +10.2%,
=1 +5.0% (4 rounds); token forcing over 14,167 tokens: mismatch 2.73% vs 2.53% /
2.53% for two unset arms (+0.19 pt, 90% CI -0.02..+0.42), PASS.
(cherry picked from commit b2bfaef906acee4f86bdf9f052aee6a4d295fb4e)
Prompt-path block scores in single-pass TF32 with a per-block error bound,
3xTF32 rescoring only near each query's top-k cut-off, so qsa_block_topk
returns the same cells in the same order, ties included. Off unless the
variable is set; says once where it cannot run (HIP, Metal, SYCL, layer-major
path). Adds qsa_two_pass_test, qsa_two_pass_status_test and the GPU parity
test qsa_two_pass_parity.
Ported from PR 4119 (bitaccesscomau); the log line of the per-prompt counters
is shortened without dropping a field.
l620 (sm_120, IQ3_S), greedy ids identical off vs on in every pair:
prompt 128K +1.65% (n=12), 256K +4.10% (n=8), 32K -0.14% (noise).
(cherry picked from commit bbf7c55f1351d913233250fe7869642f7c2b98a1)
Ported from #4871 (cctan): src/prefill/moe_iq4_w4a8.cu, tried first by mmq::Context::run; unset, nothing changes.
Changes on top of the PR: the "would MMQ split this launch along K" test is now mmq::launch_splits_k, which reads the
device's real MMQ configs (tile width, stream_k flag, tile efficiency) instead of an Ampere table, so the kernel declines
exactly the launches whose sums MMQ adds in another order, on any architecture; sm_120 joins sm_80..sm_88 (bit-for-bit
against MMQ on an RTX PRO 5000 and an RTX 3060); the kernel counts what it ran and why it declined and prints that at exit;
prefill_iq4_w4a8_test takes --tokens N (tails), --wide (scale extremes) and --bench (MMQ against the kernel per product set).
Class E: 0 differing output words in every case (tokens 97 / 2049 / 4096 / 8191 / 15360 / 32768, uniform and log-normal routing,
group buffers and pointer table). Kernel time against MMQ on the down products (IQ4_NL): +9..+70% on sm_86, +13..+50% on sm_120.
cpu_vnni_ok() = AVX-512 F + DQ + BW + VL + VNNI + OS state, cpu_vbmi_ok() = F + VL + VBMI; the Q2_0 row dot and the q8 quantizer take the 512-bit form
on VNNI without VBMI (the one VBMI op, vpmultishiftqb, has a bit-for-bit substitute unpack), and the i-quant / Q5_0 rows of iq_avx512.cpp, whose object holds
no VBMI instruction, gate on cpu_vnni_ok() instead of cpu_avx512_ok(). Adjustments over the PR: DQ is asked for (the file is compiled with it), the i-quant rows
gate on VNNI (the rows use vpdpwssd/vpdpbusd), and STRATA_NO_VBMI=1 runs the substitute unpack on a CPU that has VBMI (a test knob).
IQ4_XS gate/up: ggml-cpu's dot order (the integer lanes of its AVX2 dot, f16 scale times y.d, eight float lanes, its hsum) for every token count;
before, an AVX-512 CPU ran ggml's one-token dot per token. Q5_0 down: ggml's dot order, weights decoded once per block pair for all tokens.
IQ4_NL down: iq4nl256_down_rows' order, 512-bit. Each is checked once against the code it replaces; STRATA_NO_IQ4XS512=1 / STRATA_NO_QD512=1 switch
them off. The IQ4_XS nibble-plane decode follows the layout of PR 5243.
Co-authored-by: bsorensen110 <[email protected]>
A diagnostic for testers and for the performance work, off by default. With STRATA_MANIFEST unset every call returns at once
and nothing is printed or written. With it set, the engine writes <file> (whole file, replaced by rename) when it is ready and
again after a request if something new was recorded; the launcher (strata_control/split_engine.py) writes the all-cards split
decision to <file>.launch.json. Sorted keys, no timestamps, process ids or addresses, so two matched runs give the same text.
Content:
- engine: version, engine kind (single GPU or layer split), stages, KV format, drafter, spec window, pipeline windows, PLE I/O.
- budgets: context, KV resident cells, expert slots and cache MiB (all tiers and primary), VRAM reserve, prefill chunk cap and
stages, pool workers, memory limit, resident budget, batch slots, conversation cache.
- paths: one entry per decision with selected/declined and the reason in plain words: prompt experts per format class (MMQ,
FP16 WMMA tiles, native fused kernels, FP16 GEMM, with the gate/up and down formats and the layer count), MMQ direct and group
sort, HC_Q8, the output head (VRAM or pinned host, repacked or not), route tail skip, DMA batch mode.
- tensors: dense and head formats as held (count, bytes), the packed and HC_Q8 copies made at load.
- launch: split-engine mode, engine chosen, reasons, card architecture and count, routed-expert types, and the AMD auto rule
condition by condition (architecture, card count, expert types), so four gfx1201 with the IQ3_S set is recognised and two or
three cards are not promoted by the four-card rule.
The only text it prints is "strata: STRATA_MANIFEST: cannot write <file>" on stderr, once, when the variable is set and the file
cannot be written.
Tests: tests/core/run_manifest_test.cpp (off is free and writes nothing, matched runs in any recording order give the same text,
replace-on-rerecord, escaping, dirty tracking, unwritable target), tests/split/test_split_engine.py::LaunchManifest.
Correctness (class E, not a speedup): the product values do not change; the rows past T*K are never routed to.
llama.cpp's mul_mat_q_stream_k_fixup loads ids_dst[col_low + jt*J + j] for a whole J-column tile before it compares
the tile with the expert's rows (the main kernel skips an empty tile first; the fixup only skips its writes), and its
tiles run to ceil(max_rows / J) * J for every expert of the launch. A table of exactly T*K entries is read past its
end by up to roundup(max_rows, J), and the main kernel reads up to J - 1 past its last row. Strata's `ids_identity`
was an exact-sized allocation: when it ended a 2 MiB page the over-read faulted on the next (Xid 31 FAULT_PDE,
"prefill copy_i32: an illegal memory access" on a long prompt on a CUDA layer split; issue #1468 and others).
PR #5108 added T + 256 rows by hand. Here the margin is derived: J is bounded by ggml's mul_mat_q_switch_J
(J = 8 .. 128 in steps of 8), the tile loops of moe_mmq.cu now use that one constant (kMmqTileColsMax /
kMmqTileColsStep), and the table is T*K + roundup(T, J) + J rows (an expert takes at most one row per token), with
checked 64-bit arithmetic: a size that does not fit fails the carve instead of wrapping to a small table. The same
size is used by carve, bytes_needed_impl and the three places that write the identity (helper rebind, normal prompt,
layer-major), so the estimate and the carve stay identical. The peer prefill's row table, an exact cudaMalloc used by
the same MMQ launches, gets the same margin.
Tests: tests/prefill/mmq_ids_margin_test.cpp enumerates every index the fixup and the main kernel can read for 2000
random layouts (late experts, empty tiles, every J), checks the exact-sized table of before is overrun by the late
expert layout, and the overflow refusals. Builds on CUDA sm_86 (z440).
Co-authored-by: quantforge-br <[email protected]>
Correctness (class E): kept even without a speedup.
Flag B (PR #5096): the graph waits for "flag B >= layer + 1" and flag B is one counter that only rises. A non-empty
plan raises it from a host function on the copy stream, after its uploads. An empty plan (n <= 0) raised it on the
spot, even when the copy stream still held an earlier layer's, or the other token group's, uploads: the counter
jumped past a layer whose bytes had not landed. Verifier::raise_flag_b_ordered() now raises it through the copy
stream whenever copies were queued in this window, and directly only when none were. The same ordering is used at
the four places that publish an empty plan when the pool did not (run, run_slot_rows, batch_serve, service), which
had the same hazard. A failed host-function launch used to be ignored (flag B never rose and the window hung); it
now stops the engine like a failed upload does, after releasing the GPU waits.
Adaptive-copy bookkeeping (PR #4844): a round planned on the pipelined loop whose loop drained before step 3 is
marked resident on the serial rule, which used the lo/hi/bi of an earlier round, so only that range of swaps was
marked and the others were lost (the slot held the new expert, the table still named the old one). The pipelined plan
now names the whole round. The range arithmetic moved into AdaptiveCopyRange (adaptive_range.hpp) so a host test can
walk the rounds.
Tests: tests/core/flag_order_test.cu (delayed host-function copy standing in for a slow DMA, a consumer kernel that
reads staging once flag B allows it, a negative control that shows the old direct raise releases it early, flag only
rises, nothing queued, graph replay twice, every device; passes on both A4000s of z440), tests/program/
adaptive_range_test.cpp (serial batches for 0, 1, many swaps and several caps, the stale-range loss, empty round).
Co-authored-by: AlanJager <[email protected]>
Co-authored-by: uvlad23 <[email protected]>
Reporting only (class E): no engine speedup, no change to what the engine computes.
serve/server.py
- A prompt read that gives way (BYIELD) and resumes keeps the request's external prefix boundary, and the prompt
time and tokens of every leg are summed exactly once (prefill_leg_stats; PR #5318).
- The admission's DONE times one token and BDONE counts that token but times only the slot: the slot time now
includes the admission time, and the output token is not counted twice. A figure the engine never reported
(a close at the admission token, in the slot, a carry back to the solo path) stays unknown (None) instead of
turning into 0 or into the API's output count (PR #5307).
- The two patches overlap in the request finalization; ended_on_yield is taken before the unknown-figures step so
the identity check still sees the yield's own record.
PfTimer::mark(): a mark in the phase already running is not recorded (PR #4006). The totals are the same, a
fraction of the event records; under WDDM every record is a submission of its own and the launch gaps they made
were charged to "dequant". The rule is a pure function in prefill/timer_marks.hpp with a host test.
Tests: serve/test_yield_accounting.py (new: two yields, solo and admission yields, cancel after one and two
yields), serve/test_parallel.py BatchLegAccounting (EOS and length finishes, zero slot time, close at the
admission token and in the slot, slot to solo carry), serve/test_request_accounting.py (unknown stays unknown,
explicit zero), tests/prefill/timer_marks_test.cpp (same totals with 18 of 23 records skipped).
Co-authored-by: YusenZhang0601 <[email protected]>
Co-authored-by: shoucandanghehe <[email protected]>
Co-authored-by: Vop3p <[email protected]>
The gfx12 rule in generate.cpp now serves both: nv_chunk.hpp (STRATA_NV_CHUNK=0 or STRATA_DEFAULTS=026 keep 8192, =16 keeps 16384, =1 skips the RAM guard), the fit scan still picks a smaller chunk when the cache cannot lend the buffers, the #2090 RAM guard steps 32768 -> 16384 -> 8192, and a layer split takes 8192 back (STRATA_SPLIT_CHUNK16K=1 keeps the ceiling). An explicit --prefill N, auto:N or STRATA_PREFILL_AUTO_MAX is never touched. Volta / Turing / Pascal keep 8192.
A6000 UD-Q4_K_XL, 13.1K of 24.6K experts in VRAM, MMQ_KQ on: prompt 20K +43.4%, 32K +22.5%, 64K +21.3%, 8K 0.0%; cards that hold every expert: flat.
nv_defaults_test covers the ceiling, the switches, the card rule and the step-down.
kNvKqDefaultOn is true; STRATA_MMQ_KQ=0, STRATA_NV_KQ_DEFAULT=0 and STRATA_DEFAULTS=026 turn it off, a STRATA_MMQ_KQ the user sets is kept, Volta / Turing / Pascal keep the FP16 prompt path. A pack without K-quant experts never reads the variable.
Start check (RTX PRO 6000, UD-Q4_K_XL, no STRATA variable set): the start prints the K-quant MMQ line and loads libstrata_kq; 4K prompt GPU timeline 1,047 ms (dequant 7.3%) against 1,500 ms (dequant 25.1%) with STRATA_MMQ_KQ=0, prompt 2,854 vs 2,233 tok/s (one request each).
Niko1221