Ported from glorysdj's draft pull request #4185 (commits 961f62963 and 0726ef5ad of that branch; the documentation commit is not taken).
A compile option STRATA_SYCL_IQ_DECODE_FAST (default OFF: packed Q2 expansion, IQ4 bit-select lookup, fast IQ3_S sign arithmetic) and fourteen runtime controls, all default off:
STRATA_IQ3S_GU_SLM, STRATA_IQ4NL_DOWN_SLM, STRATA_RESIDENT_PLAN_COMPACT, STRATA_IQ2S_GU_ROW_REUSE, STRATA_IQ3XXS_GU_ROW_REUSE, STRATA_GR_CACHED_NORM, STRATA_GR_LO_REUSE, STRATA_MMVQ_SHAPE_RPW, STRATA_GDN_QK_REG, STRATA_EXPERT_ROW_E3, STRATA_Q6_SHAPE_GRF, STRATA_IQ3S_GU_ROW_REUSE, STRATA_GR_REDUCE_ILP, STRATA_ATTN_MERGE_ACTIVE.
Conflicts resolved against this tree: the kprof launch wrappers kept (new launches in the conflicted hunks use strata::kprof::kp), the resident-plan launch keeps the kprof wrapper with the compact thread count, native_expert_bench.cpp keeps this tree's fixed bench (the PR's older version of the same hunks is dropped), qsa_prompt_attn_parity.cpp keeps the per-cell check and adds the PR's FP32HASH line and QA_FP32_GRAPH timing.
Also in the diff: docs/INTEL.md section from the PR.
docs/INTEL.md and docs/INTEL_ARC.md described an SSD expert tier that strata-sycl.sh's default (STRATA_VERIFY_NO_HOST=1) does not have: every
expert must be in VRAM or in the pinned RAM mirror, and the engine refuses to start otherwise (sycl/src/program/generate.cpp, 'REFUSED: ... are
neither in VRAM nor mirrored'). A native (IQ) pack refuses --spec 0 and 1 (sycl/src/program/generate.cpp and src/program/generate.cpp). Ported by hand
from PR #2153 by nekomario28 onto the rewritten pages, each sentence checked against those two messages. The hunk for docs/TROUBLESHOOTING.md is left
out: the entry it changed is no longer on that page.
Owner decision 2026-10-22. setup_intel no longer switches pictures off for a B-series card; it prints the line instead (known risk: the first-picture xe reset on the B70, 2026-10-19). strata launch gets the same line as a warn-level note (Computer.new_card_note) and its picture default for an Intel engine now follows the CPU-reader rule at any VRAM size (it used the GPU reader for a 32 GB B70). Gyro adoption on Arc was already allowed in this tree (adopt.FAMILIES). Tests that pinned the opt-in are updated; docs/INTEL.md, packaging/README.md and docs/GYRO.md say B-series is supported and untested by us.
MAC: the .dmg and the Terminal bring the Metal engine; pictures only with the source build. AMD_HIP: the 24 GB
picture rule and the measured R9700 GPU reader. INTEL: the grouped XMX label is engine 0.1.45.3 (not 0.1.48), plus
the A750 prompt-hang fix (fca3c5958; STRATA_PF_DMA_PAGEABLE=1 restores the old copy). INTEL_ARC: maintainers now test
on a B70 and an A750; where the Intel engine comes from. OLDER_GPUS: the P100 sm_60 defaults (STRATA_SM60_PERF).
MULTI_GPU: two models on two cards, one make per engine. MODELS and TROUBLESHOOTING: the Mac and Intel Arc have no
picture reader yet.
The multi kernel reads the dense IQ3_S matrices (attn_q, attn_gate, ssm_out, attn_k, the shared experts' gate/up)
at 76-114 GB/s on the B70. The new kernel has the shape of the Q4_K..IQ4_XS wide kernels. unitrace, one card,
IQ3_XXS, T=1: worst attn_k (512 rows) 77 -> 123 GB/s, best attn_q (12288 rows) 114 -> 307 GB/s; these matrices take
0.60 ms of kernel time per token instead of 1.36 ms.
Unset or 0: the multi kernel, output identical to main. With the option the greedy output parts from the default at
token 198 of 256 (the sub-blocks are summed in another order).
Variant 6 of gdn_recurrence_variant (sycl/include/strata/prefill/gdn_xe.hpp): the column kernel's four row groups
and FMA chains in the same order, a 16-lane sub-group holding four columns x four row groups, so the cross-row-group
sums are sub-group shuffles and a token needs one work-group barrier instead of five (256 GRFs a thread).
Arc A770 16 GB (xe, oneAPI 2026.1 JIT), same day, a0116b27 + #2190 + #2193 with and without this commit:
gdn_parity recurrence 51.0 -> 24.5 ms at 8,192 tokens, 204-236 -> 110-113 ms at 32,768 (48 heads), 0 mismatches
(state, FP32, FP16). Coder IQ1_M, 128K context, int8 KV, --resident-experts: prompt 386 -> 396 / 407 tok/s at 8K,
478 / 480 -> 511 / 518 tok/s at 32K (2 runs each), the same output ids in every run.
On by default on Alchemist only (strata::intel_gpu_gen); every other card keeps the column kernels, unchanged.
STRATA_GDN_XE=0 turns it off, =1 turns it on for any Intel GPU with 16-lane sub-groups. gdn_parity compares
variant 6 with variant 0 bit for bit (state, FP32 and FP16 outputs) on any Intel GPU and skips elsewhere;
ctest -R prefill_gdn_parity runs only that part.
On an Arc A770 every prompt answered "!!!!" (token 0). The experts in the
last 0.46 GiB of the 4.46 GiB VRAM expert cache dequantized to NaN in the
prompt path. The slots' bytes were right (a memcpy read back equal to the
GGUF, and a fresh copy dequantized clean); only kernels reading them in place
got wrong bytes. In a JIT build IGC addresses kernel pointer arguments as a
32-bit offset from the allocation's base.
The JIT (non-AOT) build now passes -ze-opt-greater-than-4GB-buffer-required
to the spir64 backend, in the same argument as the divide/sqrt option (CMake
drops a repeated -Xsycl-target-backend=spir64). Measured on the A770, Coder
IQ1_M, --stream-experts: prompts of 73-542 tokens have 0 non-finite values
under STRATA_DBG_NAN=1, and the 542-token prompt answers right. The AOT build
is unchanged. Speed cost not measured.
samp_acc_t is float by default (-DSTRATA_SYCL_SAMPLER_FP64=1 keeps the reference's double). On an Arc A750 one row of the
tail cost 451 us in double and 20 us in float; a sampled decode ran at 7.62 -> 8.10 tok/s (medians of 5 interleaved
pairs, runs 7.1-9.2). sampler_parity: 0 failures on the A750 (JIT build) and on the B70; both cards' 10-prompt gates pass
10 of 10, and the B70 gate's sampled cases pick the same tokens as before. Of the kernels that use double, only the
sampler runs on a native pack (the router, combine, gate and norm kernels there are float).
Probes: sycl/probe/fp64_cost.cpp (FP64 against float in the small reduction kernels), gemm_vary.cpp.
docs/INTEL.md: the aliased-pages bug, NO_HOST, the prompt-chunking numbers, draft-length sweep, FP64 and XMX findings;
docs/INTEL_ARC.md: a row for the B70/A750 runs.
It hung on the B70 until the stager/PLE event-wait fixes; it runs since and reads
Coder prompts faster (40K 1,106 -> 1,160, 128K 897 -> 920, 128K k8v4 1,006 ->
1,041, 256K 760 -> 780 tok/s; same output). It copies every non-resident expert,
so with a big host tier it loses (IQ2_XS, a quarter in the RAM mirror: 2,184
tokens ~560 -> 254 tok/s): generate.cpp hands the non-resident share to the
prompt path, which walks only below 10%. STRATA_PREFILL_STREAM_ALL=1/0 forces it.
Docs: the 40K prompt breakdown, and why the cluster kernels and Q4_0 XMX prompt
attention are not worth porting here; the 2K "regression" was session variance
(old and new builds A/B'd: 784-799 tok/s both).
Tested on the B70: Coder 2,184-token and 40K, IQ2_XS 19- and 2,184-token
outputs identical; IQ2_XS 562 tok/s (stream-all off by its share), Coder 40K
1,160 tok/s.
3-way merge from the port's "main + PRs" migration to dpct of upstream 0.1.38
(prefill.cpp re-merged with the histogram diff: upstream's peer-device and fused
restructure kept, the port's edits carried over - the stager's GGUF job, the
stream-all gate, the fp16 NaN checks; generate.cpp: the pipelined profile fill
beside upstream's prefetching serial fill; mtp/verify: the SYCL host mapping).
- the PRs' header forks dropped: upstream has them now
- stubs: --peer-device (peer_experts.cpp; open refuses) and the fused int8 prompt
kernels (moe_fused_stub.cpp; MMQ is not built here)
- off on SYCL: the sm_90 cluster greedy sampler and QSA top-k (fixups.py)
- fused_gr: dpct's undeduced kernel names filled in (fixups.py)
- STRATA_GDN_KEYHEAD stays opt-in on the port (upstream's gate is NVIDIA SM counts)
- version 0.1.38
Tested on the B70: build clean; native_grouped, iq_multi, gr, qsa, kv_hybrid,
quantize_act parity pass, qsa_prompt_attn's int8/fp16 pass (its new Q4_0
tensor-core cases are refused: not in the port); Coder short/long, IQ2_XS, 40K
outputs identical; 128K int8 KV streaming 892 / 66.8 tok/s; GR_DOWN_MAX4 neutral,
GDN_KEYHEAD 8% slower (stays off).
- engine (serve mode): GEN key logprobs=K (0..20) writes "LP logprob id:lp ..."
after each "T id", from the verify window's head logits (Verifier::copy_logits;
before temperature, penalties and sampling); absent key, no LP lines
- server_intel.py: pairs LP lines with their tokens (StrataEngine._pump), passes
logprobs/top_logprobs to the engine, and fills OpenAI's choices[].logprobs
(content / reasoning_content) in streamed chunks and the collected reply; the
stop token is not listed. serve/ itself is unchanged.
Tested: a fake engine on hardac (streamed, not streamed, without logprobs,
stop token excluded); on the B70 with the IQ2_XS: same answer with and without,
"Paris" -0.0019 with top-1 equal to the chosen token, no measurable time cost.
One dpct run of upstream main + the seven PRs, 3-way merged into the files they
touch (sycl/tools/merge_upstream.py); their header changes are forked into
sycl/include until upstream merges them.
- #385 stager waits for a buffer's previous DMA on every job (port: polled seq)
- #463 decode waits for an adaptive swap before reading residency
- #453 drafter's batched K/V on the KV-streaming ring
- #374 first chunk's PLE rows read beside layer 0; port fixes: the next read
starts at c0 + T (the port's first chunk is shorter), and the upload is
marked by a polled sequence number instead of a host event wait, which
deadlocked prompts past ~4K tokens under the Level Zero v2 adapter
- #363 group stride + fused SwiGLU/q8_1 for the PCIe expert call, re-applied
to the port's lane kernels (fp contract off, as upstream's HIP form)
- #407 --adapt-tuned (opt-in), #413 key-head DeltaNet recurrence (opt-in via
STRATA_GDN_KEYHEAD=1: 8% slower on the B70; its NVIDIA gate and SM-holding
parity bench not ported)
Tested on the B70: native_grouped_parity, iq_multi, gr, qsa, kv_hybrid,
quantize_act, qsa_prompt_attn parity pass; Coder 19/2,184-token, IQ2_XS and
40K outputs identical to 0.1.33; Coder 2,184-token prompt 784 -> 825 tok/s;
128K int8 KV streaming 895 / 67.0 tok/s; IQ2_XS decode +1.7% (#363).
Co-authored-by: sergqwer, constantindjonkam, architectds, BlueKingMuch (upstream PR authors)
The fill from the GGUF (--stream-experts) read each expert's three slices,
copied the blob and waited for the copy before the next read: ~12,000-18,500
experts with neither the SSD nor PCIe kept busy. Now the slots are admitted
in profile order first (same placement), reads go in file order by up to 8
threads into page-locked batches of 64, and a batch's copies overlap the next
batch's reads. STRATA_FILL_SERIAL=1 keeps the serial fill.
Tested on the B70: cold page cache, Coder fill 23.4 GiB 76.2 s -> 18.8 s;
launch to first token Coder 82 s -> 26 s, IQ2_XS 120 s -> 41 s; outputs
identical to the serial fill.
- gguf_expert_source: ExpertLayout::gguf_file is per layer and role
(3 * layer + role) since upstream 0.1.31; the port indexed it by layer, so
experts in shard 2 were read from shard 1 at shard-2 offsets (Swift 1.5:
layers 13-47, NaN logits from layer 13). One fd per layer and role now.
- iq_kernels: vec_dot_iq2_xs and Split<17> took their 16-bit codes through
a uint16_t* to an int2 (type punning the device compiler does not honour);
by shifts now. iq_multi_parity checks IQ2_XS against a CPU decode from
ggml's tables (iq2xs_host_tables.inc) and passes.
- prefill: STRATA_DBG_NAN=1 also reports the experts' fp16 GEMM inputs.
Tested on the B70: Swift 1.5 IQ2_XS answers correctly (48 tok/s raw, short
thinking); the original IQ2_XS output is unchanged (65109a, 58 tok/s).
expert_cache.cpp, expert_source.cpp, fused_gr, qsa_prompt_attn, gr_parity and
generate.cpp merged from a fresh dpct of 2ca1b66 and 8554b6b; files that only
differed by dpct run-to-run noise left alone. Conflicts: --resident-cpu-experts
(upstream's resident_cpu_explicit kept with --stream-experts) and the prompt
attention's capability check (the port's XMX dispatch kept). Version 0.1.33.
Tested on the B70: build clean; gr_parity and qsa_prompt_attn_parity pass;
Coder short/long, IQ2_XS and the 40K prompt (borrowing) outputs identical,
78.3 / 75.7 / 58.0 / 68.9 tok/s. setup_intel.py --check and an install run, and
server_intel.py with the mock engine, against the new setup.py / server.py.
The hang was in the streamed-expert prompt path, which borrowing is the only
way to reach on the Coder: the stager threads waited on the copy queue's
events to reuse their 16 page-locked buffers, and under the Level Zero v2
adapter that left another queue's barrier on the same event waiting forever
once the ring wrapped. The copy queue now writes a sequence number into
page-locked memory after each DMA and the stager polls it.
- prefill.cpp: Stager::issued_one / buffer_free; STRATA_PREFILL_SYNC=1 debug
switch (wait and log after every phase)
- generate.cpp: borrowing is the default above a 32K context again
Tested on the B70: 40K prompt with borrowing 1,117-1,144 tok/s, decode 69
(output identical to the run without borrowing); 128K int8/q4_0/k8v4 decode
66.6/62.8/66.8 (from 59.5/53.4/54.4), 256K 55.9/51.7/52.2 (from
48.7/43.2/42.3); Coder and IQ2_XS 32K outputs unchanged.
dpct of 6ac1a2c and 2ca1b66, normalized, kernel-name hashes canonicalized to
the port's, then git merge-file into sycl/: 49 files, 10 with real conflicts.
- verify: upstream's set_commit_async/wait_commit on the port's deferred
commit (wait_commit = commit_finish); copy_logits on the queue
- fused_gr: upstream's split/staged read variants merged but unused by the
port's read; their per-card self-test segfaults on the B70, so on SYCL it
runs only with STRATA_HC_CHECK=1. The port's split-norm kernel renamed
(upstream has one of the same name); two colliding kernel names renamed
- fixups.py: fused_gr's per-block local-memory query
- hand files: mtp.cpp, verify.cpp, ple_reader.cpp (keep-alive reads + trace)
- qsa_prompt_attn_parity turns on the XMX kernel it checks
- version 0.1.32
Tested on the B70: build clean; Coder (19 / 2,184-token prompts) and IQ2_XS
outputs identical to 0.1.31 at the same speed (78.1 / 75.7, 58.5 tok/s);
40K prompt 1,201 tok/s, decode 65.1; kv_hybrid, qsa_prompt_attn, qsa, gr,
kv_q4, kv_q8, rope, router, sampler, quantize_act parity pass.
Every file outside sycl/ (and docs/INTEL.md) is now upstream's, byte for byte.
The Intel pieces that used to edit shared files move under sycl/ and wrap
upstream from outside:
- sycl/setup_intel.py: runs setup.py's main() with the GPU check, engine
step, RAM rule and config writer swapped for the SYCL port's
- sycl/serve/server_intel.py: serve/server.py plus the Arc's Monitor
readings (sycl/serve/xe_telemetry.py) and the optional model menu
(sycl/serve/web/switcher.js)
- sycl/tools/gpustat.py (was tools/intel/), sycl/.gitignore; migrate.sh
writes its compilation database outside the checkout
- removed: the llama.cpp engine path (serve/engine_llama.py), the README box
Engine sources in sycl/ still track 0.1.31; the 0.1.32 kernels are not
migrated yet.
Re-migrated (dpct of 0.1.29 and 0.1.31, 3-way merge per file): 45 files
changed upstream, 38 conflicts resolved; verify.cpp, mtp.cpp and the new
conversation_state.cpp ported by hand; 7 new upstream files.
Fixed on the way:
- the port's grouped expert kernels stay the default: upstream's new multi
kernels (8 rows a group) ran on the port's 32-row grid and wrote a quarter
of each expert (end-of-text output); STRATA_EXPERT_SPLIT=1 runs them with
their own grid. Lanes per row now selectable (STRATA_GU/DOWN_LANES)
- dpct's unparenthesised __fadd_rn: the native QSA indexer's per-token
pooled key (decode), the native QSA scores and the s2 activation rounding
replaced instead of added - from the first migration; fixups.py fixed
- doorbell waits: fixups.py bounds upstream's new spin spelling again
- prompt-slot borrowing hangs since upstream's rework: off by default
- the Turing PTX branch in qsa_prompt_attn, dp4a.hpp's __nanosleep, the
coupled-draft capture signature, a duplicate kernel name
- gr_parity: the single-token GR read runs the multi kernels (bitwise)
- qsa_parity: a cross-queue copy race in the test itself
Decode 78.2 / 75.7 tok/s (Coder), 58.6 / 64.2 (IQ2_XS); 128K-256K complete.
- sycl/include/dpct/LICENSE.TXT: the SYCLomatic helper headers' license
(Apache-2.0 WITH LLVM-exception; their SPDX notices already name it)
- sycl/third_party/ggml/LICENSE: ggml's MIT notice beside the migrated copy
- tools/intel/gpustat.py + gpustat.service: the root sampler the Monitor
tab reads load and VRAM from on xe (finds the card, sizes VRAM from its
BAR; GPUSTAT_PCI / GPUSTAT_VRAM_MB override)
- dpct reports and .yaml sidecars removed and ignored
- docs/INTEL.md: no references to a private test rig; the sampler and the
model_switcher setting documented
k8v4 (KV in VRAM, no streaming): 995 tok/s prompt / 65.8 decode at 128K,
804 / 51.2 at 256K with 994 experts served from the host mirror.
API: 48-72 tok/s on a short chat answer, prompt included.
Summary 77.9 tok/s (19-token prompt) / 74.2 after 2,184 tokens; long
contexts with setup's KV streaming: 66.1 tok/s decode after 128K, 54.1
after 256K (all experts in VRAM). The 'not re-measured' caveat is gone.
The expert down projection (IQ4_NL in 39 of 48 layers) was the largest
expert kernel: ggml reads four uint32 of the codebook from global memory
per lookup and each weight int as two 16-bit loads. Register constants
and aligned 4-byte loads (the third only when unaligned, so page-safe)
produce the same ints: 80.8 -> 43.1 us per call, output tokens identical.
-DSTRATA_IQ4NL_FAST=0 restores ggml's path.
The summary still described llama.cpp as the Intel engine at 23-25 tok/s.
It now compares NVIDIA, the SYCL port (setup's default once built) and
llama.cpp; the stale 'not ported'/'not done' items, setup's KV choice and
the served speed are corrected, and the long-context table is marked as
measured before the decode work.
Each work-group owns a 128-column slice of all 324 rows, stages that
slice of xn once (<= 4 KB SLM) and streams the rows through it, 4 lanes
per row; a reduce kernel sums the 80 slice partials in a fixed order.
GR read 108.6 -> 76.5 us at 6 tokens. STRATA_GR_DOWN_SLICED=0 restores
the direct kernel.
Niko1221