Row error on 4,000 random rows against the checkpoint's BF16 table (Q8_0 0.53%, FP8 2.64%, IQ4_NL 7.60%, matching the
numbers reported in #651), and teacher-forced KL on the Q2_0 model with each table.
The 0.1.39 paragraph said nobody had run the 0.1.39 port on an Arc. Two Arc Pro B60 reports (#867, #870) did, with
the two sycl/ fixes now in 0.1.40 (#784 the compile fix, #866 the ring wait). Say that, and that an unpatched 0.1.39
port did not compile against its own engine sources.
INTEL_ARC.md: "What has been run" rows for 2x and 1x Arc Pro B60, plus notes for the card: PCI id 8086:e211 (bmg-g21), setvars.sh
stopping at OCL_ICD_FILENAMES under `set -u`, running natively without Docker, and the layer-split startup warning that counts the
other card's layers.
sycl/setup_intel.py: the B60 table entry was e221, which the kernel's xe id list files under the BMG-G31 cards (0xE220-0xE223). Both
Arc Pro B60 cards measured here are 8086:e211, so setup named them "Intel GPU e211 (xe)" with a guessed VRAM size. e211 is added;
e221 stays, since there is no card here to say it is wrong. tools/test_setup_sycl.py checks the table.
COMMUNITY_BENCHMARKS.md and bench/results/2026-10-04-community-2x-arc-pro-b60/: the report, in the template's format, with the
server configs, engine logs and per-request timings.
A plain llama-quantize Q4_0 file of this model (bartowski's Qwen3.8-Flash-Next-Q4_0 and
Swift-1.5 Q4_0: gate/up Q4_0, down Q4_0 on 42 layers and Q4_1 on 6, a Q4_0 PLE table and
token embedding) was refused at start: no GPU kernels for Q4_0/Q4_1 experts, and the PLE
reader took only IQ4_NL, Q5_0 and FP8. Follows #222 (Q4 to Q6 quantizations).
- Q4_0: Fmt<2> with the codes centred before dp4a (q - 8 as signed bytes), the exact
integer form ggml-cpu's q4_0 x q8_0 dot uses, rather than llama.cpp's CUDA chain
(uncentred codes minus 8 x the q8_1 float sum): on a real Q4_0 expert the CUDA chain
measured 1.4e-2 relative error against the float product, the centred form 5.3e-3.
- Q4_1: Fmt<3>, the Q5_1 chain without the fifth bit and the same min-term choice.
- dq_q4_0 / dq_q4_1 for the prompt path's dequant fallback; row bytes, is_iq and the
GU/D/MMVQ format lists.
- PLE: Q4_0 rows (90 B, as IQ4_NL) through the same reader: one entry in the PLE format list (ple_format_test and
ple_reader_test cover it).
- docs/UNSLOTH_Q4.md: a Q4_0 section with the measured run.
native_expert_parity --synthetic (now in the CMake matrix): q4_0/q4_0 CPU 1.29e-2, GPU
1.29e-2, CPU-GPU 1.1e-7; q4_0/q4_1 CPU-GPU 3.2e-4; GPU dequant bit-exact against
to_float. The existing pairs are unchanged. On bartowski's file (revision 928589f), layers
0 and 5 (Q4_1 down) and 20 and 47: CPU and GPU 1.1-1.3e-2, CPU-GPU 1.1e-7 (Q4_0 down) and
5e-4 to 1.1e-3 (Q4_1 down). Served on one RTX 3060 (all experts in RAM, 131K context):
24,461-token prompt 787 tok/s read, 24.0 tok/s out; 67,934 tokens 737 / 26.4, the hidden
fact found.
Ported onto the format table by the maintainer (the PLE part of the PR is the Q4_0 table entry).
With an engine that keeps --resident-experts on a layer split, setup no longer
steers a low-RAM PC with two cards to one of them:
- at install, the low-RAM mode on several GPUs takes the resident variant by
the same rule as on one card (what no card holds fits the RAM with
LOW_RAM_HEADROOM_GB beside it), without the one-card question;
- at a start, a resident config on several GPUs keeps --resident-experts
(split_mmap leaves it alone);
- a resident config started on a PC with two cards is offered both, "y" by
default, as any other config.
It is gated on MIN_ENGINE >= RESIDENT_SPLIT_ENGINE (0.1.40 here, a guess at
the release that would carry the engine change), so until then setup behaves
as before and the golden configs are unchanged. Two tests run the new paths
with MIN_ENGINE patched; tools/test_setup_*.py pass (260 tests).
--resident-experts (the low-RAM mode's resident variant) ran as plain
--mmap-experts as soon as a layer split was set. It now keeps its RAM copy on
a split:
- the experts every stage's cache holds are left out of the RAM copy, not only
CUDA0's;
- when the rest does not fit whole (#467), the copy keeps the hottest by the
whole expert profile. Each stage only keeps its own layers' share of the
profile, and ranked by CUDA0's share alone the copy held layers 0..K-1's
experts and read every later layer's from the files (4.0 GiB in RAM instead
of 21.2 on the PC below);
- an adaptive swap copies the evicted expert back into RAM from the cache of
the card that owns its layer, on that card's stream. host_res holds slot
numbers of whichever cache owns the layer: reading a later stage's slot out
of CUDA0's cache put another expert's bytes in RAM, and after the first
swaps the answers turned to garbage. (Latent until now: the guard kept the
mode off a split.)
- the decode profile prints every stage's GPU profile, not only CUDA0's.
Remote expert caches (--expert-cache-remote) are still refused with it.
Swift 1.5 IQ3_XXS at 160K, q4_0 KV, RTX 4060 Ti (layers 0-19) + RTX 5080
(20-47), i9-14900KF, 32 GB of RAM, Windows 11, four greedy prompts, decode
tok/s (first round / fourth):
- 5080 alone, --resident-experts: 25 / 29
- split, --mmap-experts (what --resident-experts became): 32 / 64
- split, --resident-experts (22 GiB of experts locked in RAM): 71 / 69
With --pcie-frac 0 --adapt-every 0 the split's greedy output is the same as
--mmap-experts' (4/4 prompts). With adaptive swaps on, 33 varied requests at
temperature 0.6 (thinking on and off, five languages, code, each checked for
degenerate output) exchanged 81,783 experts with the two caches and gave no
degenerate answer.
--vram-reserve-mib is every card's reserve, but the card that drives the
monitors needs more headroom than one that drives none: one value has to fit
the display card, and the other cards then keep that headroom for nothing.
--vram-reserve-later-mib N is the reserve of every card after the first
(default: --vram-reserve-mib's value), in the split search and in each later
card's cache sizing.
RTX 4060 Ti (no display, layers 0-19) + RTX 5080 (display, 20-47), Swift 1.5
IQ3_XXS at 160K: --vram-reserve-mib 1000 gives the 4060 Ti's cache 4551 slots;
--vram-reserve-mib 300 --vram-reserve-later-mib 1800 gives it 4867, and the
5080 keeps 1.3 GB free with everything loaded.
- "experimental" is per model now: MODELS["UD-Q4_K_XL"] keeps it, the unsloth family and UD-IQ4_XS drop it. The
first menu lists the family without [experimental], with a one-line about (UD-IQ4_XS: a 94 GB download; under
~80 GB of RAM part of its experts are read from the SSD). Its size menu lists UD-IQ4_XS first (the default) and
UD-Q4_K_XL second; an experimental size always sorts last. --model names are unchanged.
- UD-IQ4_XS: no EXPERIMENTAL warning, --check verdict without it. Images are asked (off by default), per model: the
engine's image path has no restriction for this pack, which uses the original model's image encoder; not yet run
with images. UD-Q4_K_XL keeps images off.
- MCP: the fallback tables mirror setup's (per-model experimental/vision), sizes in setup's order, per-size
"experimental" and "images", the vision check per model.
- Docs: README and its six translations, MODELS, UNSLOTH_Q4, AI_SETUP, MCP_SERVER.
- Tests: the menus, the default, --model by name, images, --check, the pack/tokenizer paths; the golden configs
are unchanged.
- a prompt that fits 0.1.39's auto chunk keeps 0.1.39's ring (Prefill::set_ring_budget's small_max): 4K prompts
were 1-17% slower with the smaller ring (Coder -15%, IQ3_S -5%, IQ3_XXS -2%; 5 interleaved pairs each);
- ring slots are given up for a bigger chunk only where 0.1.39's rule held the chunk under 6144 (ring_cap_for);
from 6144 on the ring keeps its 0.1.39 size (the Coder 6144/384 -> 7936/199 was -12% at 32K);
- the unpinned arm's cap stays 96 (the byte budget made it ~49 on IQ3_S);
- the scan sets its own ring budget instead of set_ring_override, so a layer split's 96-slot override (#340) is no
longer clobbered by the scan.
RTX 5070 vs STRATA_RING_BYTES=0: 4K -0.7..+1.1% (bit-identical); 32K IQ3_XXS +18.5% (cache 1500) / +9.4%, Coder
+3.1%, IQ3_S +0.2%.
Added where a config goes to several GPUs (setup --gpus / a multi-GPU install, start --gpus, "use both" at
start). Single-GPU configs are not touched. Opt-out: setup --no-remote-expert-opt, or "remote_expert_opt": false
in the config (a user key, kept on re-runs). The engine uses it only with a helper cache (--expert-cache-device1..3),
so a layer split runs as before. tools/test_setup_remote_opt.py; the 2x16 GB golden configs gain the flag.
On a card whose experts mostly run on the CPU the slots cost speed per request (RTX 5070, Q2_0: 11-24 %), so setup
now recommends slots only when the expert cache (every card of a split counted) still holds half of the model's
experts beside them (Q2_0: 24 GB and up, or a split); elsewhere it stays at one at a time and says "parallel N
reduces waiting for several users but costs about 10-25% speed per request on this card". --parallel N is
honoured as asked, as before. BATCHING.md: the table with the change against one at a time.
server: a request decoding in a slot whose neighbours finished (no other slot busy, nobody waiting, at least 32
tokens allowed) is BSTOPped and continued on the solo path with its MTP drafts - the engine copies the slot's
sessions back (INFO slot_cache=1; at most twice per request; STRATA_PARALLEL_SOLO=0 keeps it in the slot).
engine: INFO batch_slots=N slot_cache=0|1.
tools/batch_interleave_test.py: the back-to-solo path, and a next turn from a slot's turn checkpoint (a client
that drops the reply's thinking).
docs: "parallel": N, what a slot costs and what setup recommends, the server's handling (interleaved prompt reads,
BYIELD, slot conversation cache, back to solo), exactness and its settings, the RTX 5070 measurements (the slots
buy waiting time, not speed, on a 12 GB card), the protocol; DETAILS / README / AI_SETUP mention the option.
INSTALL.md gets the section (the build setup compiles, what differs, the forced-path numbers from a
Ryzen 5 7600 + RTX 5070, what is untested); the CPU rows of INSTALL.md / DETAILS.md and AI_SETUP.md
point to it.
docs/OLDER_GPUS.md: Pascal / Volta through the CUDA 12 engine (#395#600#540#655), RTX 20's STRATA_BF16_TC opt-in,
AMD gfx1031 (#524), gfx1012 (#442) and gfx906 (#638, #677) with the reporters' numbers; README, AI_SETUP, AMD_HIP and
NVIDIA_V100 point to it.
A prompt that fits one chunk runs a layer split's stages one after the other,
so while one card reads its layers the other idles. --peer-device already has
the mechanism for this (PeerPrefill's peer streaming: a share of the experts
the primary streams goes over the peer's own PCIe link and is computed there),
but it needs P2P and is refused with --layer-split.
This makes PeerPrefill usable as a stream-only peer between the stages:
- set_stage_helper: each stage's prompt path gets the next stage's (the last
one the first's); a stream-only PeerPrefill holds no experts (peer == null),
so every use of the peer tier is guarded
- per prompt, bind_stage_helper points it at the idle stage's own lent prompt
buffers (mixed, Xq, GU, H, Hq, Dm, ring, MMQ context): no VRAM of its own
- without P2P the activations, the row table, the group bounds and the result
rows go through mapped host memory, read by copy kernels on the device that
needs them. Copy-engine uploads queued behind the ring's expert blobs (the
grouping tables' reason for mapped memory), which cost ~20 ms per layer and
ate the whole gain; cudaMemcpyPeerAsync without P2P is staged by the driver
- only for one-chunk prompts on the MMQ path (not the fused kernels, not with
--peer-device), with a share that falls with the chunk: min(0.45,
0.5 - T/16384), off below 0.3 (~3.3K tokens) - measured, below
- STRATA_PREFILL_HELP=0 turns it off, STRATA_PREFILL_HELP_FRAC fixes the share
2x RTX 3090 (no P2P), UD-Q4_K_XL, layer split auto, 262K context,
--adapt-swaps 0, prompt tok/s without / with: 1.1K 496 / 723, 1.5K 674 / 833,
2K 878 / 1128, 2.5K 1017 / 1296, 3K 1260 / 1440, 4K 1605 / 1607, 8K 2215 /
2221; decode unchanged. The output is repeatable (same prompt, same answer)
but not bit-identical to STRATA_PREFILL_HELP=0's: the helped rows are
computed in another MMQ grouping.
0.1.39b (Niko1221): opt-in (STRATA_PREFILL_HELP=1), since the helped rows round differently from the default;
merged with #598 (run_impl / next_run_).
Add default-off cache coordination, helper-side weighted reduction and CPU activation quantization on demand. Allow helper cache capacities to be sized automatically at startup.
Port the saved v2.2 implementation onto upstream 0.1.38 without changing the separate peer tier. Document the helper-only scope and current validation limits.
- native_expert_grouped: the AMD layouts handle the IQ packs' gate/up and down types only; Unsloth's Q4_K / Q5_K
gate/up and Q5_1 / Q8_0 down now take the CUDA layout instead of exiting (native_grouped_parity found it).
- fused_gr_read_multi: the K-split gr_down (fixed-order finish) sums in another order than the single-token
fused_gr_read, so it broke gr_parity's multi == single contract; it is now opt-in (STRATA_GR_SPLIT=1). The
latency-hidden norm/up (same sums, same order) run in the default read (STRATA_GR_FAST=0: off).
- mmvq_multi_parity: on gfx906 the wave64 MMVQ is the same exact kernel for every column count, so the negative
control has nothing to find; a second ctest entry runs it on the CUDA layout (STRATA_MMVQ_WAVE=0), where it has
power.
- kv_hybrid_parity: the tensor-core prompt attention refuses on gfx906 as on the RDNA backend.
ctest on 2x MI50 (gfx906): 34 of 34 pass (iq_parity and ple_parity need fixture files and are not counted).
Conflicts with 0.1.39: --trim-stage-weights is now the same switch as STRATA_STAGE_TRIM=1 (PR #639's
implementation, already in 0.1.39); the #533 VRAM command is taken between batch windows only when no slot
is decoding.
The experimental sm_70 build (STRATA_EXPERIMENTAL_SM60, #236) had no page. NVIDIA_V100.md says how to build it, which kernel each
part of the prompt path uses on Volta and what the prompt attention kernel was measured at.
bench/results/2026-10-03-v100-prompt-attn: prompt speed with and without the kernel (7K / 28K / 114K tokens), the nsys share of the
attention before and after, the parity numbers, needle recall (15 of 15 with either path) and the greedy-output comparison.
It hung on the B70 until the stager/PLE event-wait fixes; it runs since and reads
Coder prompts faster (40K 1,106 -> 1,160, 128K 897 -> 920, 128K k8v4 1,006 ->
1,041, 256K 760 -> 780 tok/s; same output). It copies every non-resident expert,
so with a big host tier it loses (IQ2_XS, a quarter in the RAM mirror: 2,184
tokens ~560 -> 254 tok/s): generate.cpp hands the non-resident share to the
prompt path, which walks only below 10%. STRATA_PREFILL_STREAM_ALL=1/0 forces it.
Docs: the 40K prompt breakdown, and why the cluster kernels and Q4_0 XMX prompt
attention are not worth porting here; the 2K "regression" was session variance
(old and new builds A/B'd: 784-799 tok/s both).
Tested on the B70: Coder 2,184-token and 40K, IQ2_XS 19- and 2,184-token
outputs identical; IQ2_XS 562 tok/s (stream-all off by its share), Coder 40K
1,160 tok/s.
3-way merge from the port's "main + PRs" migration to dpct of upstream 0.1.38
(prefill.cpp re-merged with the histogram diff: upstream's peer-device and fused
restructure kept, the port's edits carried over - the stager's GGUF job, the
stream-all gate, the fp16 NaN checks; generate.cpp: the pipelined profile fill
beside upstream's prefetching serial fill; mtp/verify: the SYCL host mapping).
- the PRs' header forks dropped: upstream has them now
- stubs: --peer-device (peer_experts.cpp; open refuses) and the fused int8 prompt
kernels (moe_fused_stub.cpp; MMQ is not built here)
- off on SYCL: the sm_90 cluster greedy sampler and QSA top-k (fixups.py)
- fused_gr: dpct's undeduced kernel names filled in (fixups.py)
- STRATA_GDN_KEYHEAD stays opt-in on the port (upstream's gate is NVIDIA SM counts)
- version 0.1.38
Tested on the B70: build clean; native_grouped, iq_multi, gr, qsa, kv_hybrid,
quantize_act parity pass, qsa_prompt_attn's int8/fp16 pass (its new Q4_0
tensor-core cases are refused: not in the port); Coder short/long, IQ2_XS, 40K
outputs identical; 128K int8 KV streaming 892 / 66.8 tok/s; GR_DOWN_MAX4 neutral,
GDN_KEYHEAD 8% slower (stays off).
The trim was on by default in the HIP builds. It now waits, as on CUDA, for the PR author's re-run of the release
branch on his cards; the default flips after that. --layer-split auto and one GPU were never affected.
--vram-elastic (the config's "vram_elastic": true) allocates the expert cache as one
reserved address range backed by physical segments (CUDA virtual memory management,
512 MiB each, --vram-segment-mib), so every slot keeps its address. Between
requests, the engine command `VRAM <reserve_mib>` gives the last segments back to
the driver until that much VRAM is free (their experts become CPU misses, the prompt
path's loan moves down and its chunk shrinks to fit), or maps them again and puts
the same experts back when more is free. `VRAM` alone: the start's reserve. The
server's POST /v1/vram {"reserve_mib": N | null} sends it after a running request,
keeps it for an engine that is unloaded, and /v1/status shows the cache's size.
Never automatic. One NVIDIA GPU only (no layer split, helper caches, --peer-device
or the resident low-RAM mode); HIP refuses the flag.
RTX 5070, Q2_0 (4.8 GiB cache): reserve 6000 -> 4.3 GiB freed in 78 ms, decode
44 -> 33 tok/s; grown back in 92 ms, the answers token for token the ones before.
Without the flag the cache is the one cudaMalloc it was (same answers with and
without the flag until a VRAM command).
IQ3_XXS on the RTX 5070, temperature 0, 256 tokens, 8 French prompts (topics outside the corpus), 2 English and 2 code
prompts, arms alternating, 2 passes each: drafts accepted in French 0.505 (en) / 0.507 (cjk) -> 0.597 (fr), higher on
all 8 prompts and in each pass; English 0.610 -> 0.629, code 0.772 -> 0.779. Decode speed was not measurable here
(builds ran beside it); the reporter measured 141 -> 158 tok/s in French on an RTX 5090.
UD-IQ4_XS sits between IQ3_S and UD-Q4_K_XL: IQ3_S gate/up experts (IQ4_XS in one layer) with IQ4_NL downs (Q8_0 in
five layers), 59.5 GB of them, and UD-Q4_K_XL's dense side. The engine's kernels already take these formats; it was
packed (iq_pack.py --compat-bf16) and run on the Strix Halo, not yet on NVIDIA.
- MODELS["UD-IQ4_XS"] in the unsloth family: three shards pinned at the same revision 38bb39e with their sizes and
SHA-256 (the Hub's LFS pointers), the RAM budget like UD-Q4_K_XL (budget, one GPU, no images, --compat-bf16 pack,
no experts.bin), engine 0.1.38 or newer (the fused-layout fix for packs not every layer of which is covered).
- A model can have its own shard pattern and count (model_file / model_shards); budget_model() tells which Unsloth
model a config runs, for the #498 multi-GPU messages and RAM need.
- The AMD question (#429) stays for UD-Q4_K_XL only (nvidia_only): UD-IQ4_XS's formats have AMD prompt kernels.
- --gguf-dir accepts UD-IQ4_XS files (#444's list says so).
- UNSLOTH_Q4.md: its files, what setup does, what is known (Strix Halo), and the BF16 rounding of the hyper-connection
projections that every --compat-bf16 pack has (not changed here). MODELS.md: a paragraph.
- test_setup_unsloth: pins, budget, a mocked install, the engine check, no AMD question, the config read back.
UD-Q4_K_XL's install, its tests and every other model's config are unchanged.
Stateless, on the chat path: a Responses request becomes the chat template's
messages, tools and kwargs, runs through Service.run, and its reasoning, text
and tool calls come back as output items and the official typed SSE events
(response.created ... response.completed / incomplete / failed).
- input as a string or items (messages, reasoning, function_call,
function_call_output, custom tool calls); instructions and leading developer
messages make one system message; tool results are put in call order
- function, namespace (Codex's MCP/agent tools) and custom tools; hosted tools
are left out with a log line
- reasoning.effort -> reasoning_effort, max_output_tokens, text.format
json_schema/json_object through the structured output path
- reasoning returned as reasoning_text (and as opaque encrypted_content when
asked) and replayed into the prompt, so Codex's next turn reuses the cache
- errors in the Responses format; same API key, Host and Origin checks as /v1
- a response.in_progress event every 15 s while a long prompt is read
(Codex's idle timeout ignores SSE comments)
- docs: DETAILS.md section with a Codex config.toml
Based on #630 by CC-David-CC (the event assembler, namespace tools, call-id
ordering of results), without its opt-in flag, replay key and crypto
dependency, strict-function and JSON-format refusals, or the second
summary generation pass. Tested with Codex CLI 0.160.0 and openai 2.15.0.
Co-authored-by: David <[email protected]>
The About tab's Model settings card reads GET /config and saves through
POST /config: the sampling defaults, reasoning_budget_tokens,
fit_max_tokens, anthropic_thinking, effort_position, aliases,
idle_unload_s, lazy_load, engine_silence_s, api_monitor, open_browser
and --vram-reserve-mib (serve/runconfig.py). Every value is checked
first (one bad value changes nothing), only the named keys change, every
other key of strata-<model>.json stays as it is (as setup run again
keeps them, #629), the earlier file is kept as .bak, and the change is
used from the next start. POST takes JSON from Strata's own page only,
like /settings, and the API key when one is set; the network, key, MCP
and program keys are not editable there.
- MODELS.md: the RAM rule is the experts + ~10 GB (setup's arena_gb), not shard 1 + 10 GB; the sizes per model.
- DETAILS.md: prompts are read in chunks of up to 8,192 tokens (32,768 opt-in), not 2,048; Q2_0's AVX-512 pack is
~40 GB on disk = 34 GB of experts plus the dense weights.
- README: the NVIDIA speed table says which engine measured which row (Q2_0 0.1.36, the others 0.1.26).
- expert_cache.hpp: the comment that the cache "does not yet compute anything" is gone.
Zuraiz Anjum