setup.py run(): the app's installer also runs the bundled Mac preparation tools in metal/tools (metal_repack.py, stream_experts.py); a fresh Mac Terminal or app install of UD-Q4_K_XL or UD-IQ4_XS-v31 stopped with "only runs its bundled preparation tools" since af1ffd913. Test: they run, and -m, another interpreter and other folders are still refused

I/O path only: the same bytes are read, only HOW the experts outside the RAM copy are read (unbuffered or through the file cache) is chosen differently.

file_tier_unbuffered took its `avail` from cgroup_available_bytes, which credits the group's whole `file` counter as reclaimable. On a RAM-budget
config that credit includes pages the engine holds itself: P2003 (RTX 3060, 46 GB, 36 GiB group), UD-IQ4_XS-v31, 22 GiB RAM copy: the re-check read 18.7 GiB
available when the group had 11 GiB left (limit 36, current 24.8). With held = 0.52 >= 0.5 the timed drive probe was skipped and the file tier switched to the file
cache (direct reads 7.9 GB/s vs 2.5 GB/s cached): prompts 2.5-3x slower (pp2k 85 vs 273 tok/s, pp8k 195 vs 580), decode 10-14% slower. The one start that read
12.0 GiB kept the fast mode, which is why the first start of a session was sometimes fast.

Now the smaller of that figure and the conversation cache's sampler (only the clean inactive file cache is credited, the group's own limit, no operator
cap) decides. The --memory-limit-mib cap is not applied: it bounds the engine's own allocations, not the page cache (counting it sent IQ3_S unbuffered and cut its decode
2-3x). A flip now prints once: "strata: expert reads now go through the file cache (X GiB of RAM free for them, Y GiB limit)"; the technical line is behind STRATA_TRACE.
STRATA_UNBUFFERED_LOAD still forces either mode.

Measured on P2003 (build wip11, n >= 3 starts per arm, interleaved fresh engines, I/O-quiet before each start, memory watch on, floor 14336 MiB):
 v31 unforced: main flipped in 5 of 6 starts, this change 0 of 9 (6 + 3): pp2k 85-94 -> 263-273 tok/s, pp8k 187-198 -> 580-590, code_py decode +14 to +20%;
 v31 DFlash2 vs MTP in the kept state (3 rounds): DFlash2 +2 to +14% per kind (json_extract -3.5%);
 IQ3_S: the file cache stays (9 of 9 starts), decode level with main (code_py 17.0 vs 15.6 tok/s, json_extract 14.2 vs 13.4, page cache evicted before every start);
 low RAM (6-7 GiB ballast, budget 16 GiB): no regression, MemAvailable minimum 14911 MiB;
 memory minimum with the unbuffered state 15.6-15.8 GB (floor 14.3 GB), no arm stopped.
Not P1.8: main and the P1.8 build flip alike.

(cherry picked from commit be3831e9d944fb44ac36d176f4764e30df0f9f6b)

.github/FUNDING.yml

@@ -1 +1,2 @@
+github: Niko1221
 buy_me_a_coffee: strataengine

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -66,7 +66,7 @@ body:
     attributes:
       label: Strata version
       description: "In the app: Settings > About. In the Terminal: `strata --version`. From source: the About tab of the web page, or `engine` in `GET /v1/status`, or the git commit."
-      placeholder: "Strata 0.2.11 · engine 0.1.57"
+      placeholder: "Strata 0.2.12 · engine 0.1.58"
     validations:
       required: true
   - type: input

AGENTS.md

@@ -1,6 +1,6 @@
 # AGENTS.md
 
-Version 0.2.11, engine 0.1.57. The public model families are DeepSeekV4.1-Flash-1017,
+Version 0.2.12, engine 0.1.58. The public model families are DeepSeekV4.1-Flash-1017,
 Qwen3.8-Flash-Next and Qwen3.6-35B-A3B. DeepSeek V4.1-Flash runs in the app and in the Terminal
 (`strata launch --family deepseek41 --model ist-das-2.00bpw`; sets in
 [docs/MODELS.md](docs/MODELS.md#deepseek-v41-flash)). Read **[docs/AI_REFERENCE.md](docs/AI_REFERENCE.md)** for model availability, APIs, hardware limits and

CMakeLists.txt

@@ -8,7 +8,7 @@
 # below refuses anything older: silently building for a pre-Ampere arch would produce a binary that runs
 # and is wrong about its own performance, which is the worst of both outcomes.
 cmake_minimum_required(VERSION 3.24)
-project(strata VERSION 0.1.57 LANGUAGES CXX)
+project(strata VERSION 0.1.58 LANGUAGES CXX)
 # EXPERIMENTAL: the Intel Arc engine, the SYCL port in sycl/ (PR #423; icpx + oneMKL, docs/INTEL_ARC.md). It is its
 # own project that takes the unported sources from src/; with this on, nothing below is configured.
 option(STRATA_ENABLE_SYCL "EXPERIMENTAL: build the Intel Arc (SYCL) engine from sycl/ - docs/INTEL_ARC.md" OFF)
@@ -403,6 +403,17 @@ if(STRATA_BUILD_TESTS)
   target_include_directories(share_rules_test PRIVATE include)
   add_test(NAME share_rules_test COMMAND share_rules_test)
 
+  # STRATA_SELECT_TWO_PASS: the bound between the single-pass and 3xTF32 QSA block scores and the cut-off rule that
+  # keeps the selection (header-only, no GPU)
+  add_executable(qsa_two_pass_test tests/prefill/qsa_two_pass_test.cpp)
+  target_include_directories(qsa_two_pass_test PRIVATE include)
+  add_test(NAME qsa_two_pass_test COMMAND qsa_two_pass_test)
+  # STRATA_SELECT_TWO_PASS: the request's parse and the line it prints once where the scorer cannot run (no GPU)
+  add_executable(qsa_two_pass_status_test tests/prefill/qsa_two_pass_status_test.cpp)
+  target_include_directories(qsa_two_pass_status_test PRIVATE include)
+  add_test(NAME qsa_two_pass_status_test COMMAND qsa_two_pass_status_test
+           "${CMAKE_CURRENT_BINARY_DIR}/qsa_two_pass_status_test.log")
+           
   # STRATA_PREFILL_ROUTED_STREAM: the prompt stream's issuer walk and route gate, on a simulated link and consumer
   # (header-only, no GPU)
   add_executable(routed_stream_test tests/prefill/routed_stream_test.cpp)
@@ -410,6 +421,26 @@ if(STRATA_BUILD_TESTS)
   target_link_libraries(routed_stream_test PRIVATE Threads::Threads)
   add_test(NAME routed_stream_test COMMAND routed_stream_test)
 
+  # PR #4006: STRATA_PREFILL_TIMING does not record a mark in the phase already running (header-only, no GPU)
+  add_executable(timer_marks_test tests/prefill/timer_marks_test.cpp)
+  target_include_directories(timer_marks_test PRIVATE include)
+  add_test(NAME timer_marks_test COMMAND timer_marks_test)
+
+  # PR #5108: the MMQ row table's over-read margin follows the backend tile bound; every fixup read stays inside (header-only, no GPU)
+  add_executable(mmq_ids_margin_test tests/prefill/mmq_ids_margin_test.cpp)
+  target_include_directories(mmq_ids_margin_test PRIVATE include)
+  add_test(NAME mmq_ids_margin_test COMMAND mmq_ids_margin_test)
+
+  # PR #4844: the adaptive tier's copy-in range across serial, pipelined and drained rounds (header-only, no GPU)
+  add_executable(adaptive_range_test tests/program/adaptive_range_test.cpp)
+  target_include_directories(adaptive_range_test PRIVATE include)
+  add_test(NAME adaptive_range_test COMMAND adaptive_range_test)
+
+  # P0.1 (lite): STRATA_MANIFEST, the machine-readable record of the selected paths, tensor formats, engine and budgets (header-only, no GPU)
+  add_executable(run_manifest_test tests/core/run_manifest_test.cpp)
+  target_include_directories(run_manifest_test PRIVATE include)
+  add_test(NAME run_manifest_test COMMAND run_manifest_test)
+
   # the Windows HIP auto prefill chunk cap (#1630) and the commit-limit verdict (#1607): header-only, no GPU
   add_executable(host_limits_test tests/platform/host_limits_test.cpp)
   target_include_directories(host_limits_test PRIVATE include)
@@ -614,6 +645,13 @@ if(STRATA_ENABLE_CUDA OR STRATA_ENABLE_HIP)
     target_link_libraries(qsa_scores_parity PRIVATE strata_kernels CUDA::cudart)
     add_test(NAME qsa_scores_parity COMMAND qsa_scores_parity)
     set_tests_properties(qsa_scores_parity PROPERTIES SKIP_RETURN_CODE 77 ENVIRONMENT "STRATA_SCORES_THREAD=1")
+    # STRATA_SELECT_TWO_PASS: the two-pass prompt scores' selection against the tensor-core scorer's, ids and the
+    # scored-again blocks bitwise (sm_80+, synthetic, no model)
+    add_executable(qsa_two_pass_parity src/kernels/qsa_two_pass_parity.cpp)
+    target_include_directories(qsa_two_pass_parity PRIVATE ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES})
+    target_link_libraries(qsa_two_pass_parity PRIVATE strata_kernels CUDA::cudart)
+    add_test(NAME qsa_two_pass_parity COMMAND qsa_two_pass_parity)
+    set_tests_properties(qsa_two_pass_parity PROPERTIES SKIP_RETURN_CODE 77 ENVIRONMENT "STRATA_SELECT_TWO_PASS=1")
   endif()
   # QSAROWS (docs/QSAROWS.md): what the T rows of a verification window cost in a QSA layer's decode kernels at a long
   # context (block scores, top-k, attention over each row's selection, indexer append), row by row; GPU, synthetic, no
@@ -1169,6 +1207,14 @@ if(STRATA_ENABLE_CUDA OR STRATA_ENABLE_HIP)
     target_link_libraries(hip_gdn_rec_head PRIVATE strata_prefill strata_hip_runtime)
     add_test(NAME hip_gdn_rec_head COMMAND hip_gdn_rec_head)
     list(APPEND _strata_hip_tests hip_gdn_rec_head)
+    # PR #5096: flag B stays behind queued copies (the same test as the CUDA build's flag_order_test; the HIP compat header maps the runtime calls)
+    set_source_files_properties(tests/core/flag_order_test.cu PROPERTIES LANGUAGE HIP)
+    add_executable(hip_flag_order_test tests/core/flag_order_test.cu)
+    target_include_directories(hip_flag_order_test PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/include")
+    target_link_libraries(hip_flag_order_test PRIVATE strata_hip_runtime Threads::Threads)
+    add_test(NAME hip_flag_order_test COMMAND hip_flag_order_test)
+    set_tests_properties(hip_flag_order_test PROPERTIES TIMEOUT 60)
+    list(APPEND _strata_hip_tests hip_flag_order_test)
     add_executable(hip_gdn_chunked tests/hip/gdn_chunked.cpp)
     target_link_libraries(hip_gdn_chunked PRIVATE strata_prefill strata_hip_runtime)
     add_test(NAME hip_gdn_chunked COMMAND hip_gdn_chunked)
@@ -1372,6 +1418,12 @@ if(STRATA_ENABLE_CUDA OR STRATA_ENABLE_HIP)
     target_link_libraries(dma_batch_parity PRIVATE ${_strata_gpu_runtime_target})
     add_test(NAME dma_batch_parity COMMAND dma_batch_parity)
     set_tests_properties(dma_batch_parity PROPERTIES TIMEOUT 120)
+    # PR #5096: flag B stays behind queued copies (delayed host-function copy, a consumer kernel, graph replay, every device)
+    add_executable(flag_order_test tests/core/flag_order_test.cu)
+    target_include_directories(flag_order_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/include)
+    target_link_libraries(flag_order_test PRIVATE ${_strata_gpu_runtime_target} Threads::Threads)
+    add_test(NAME flag_order_test COMMAND flag_order_test)
+    set_tests_properties(flag_order_test PROPERTIES TIMEOUT 60)
     if(WIN32)
       add_test(NAME dma_batch_registered_parity COMMAND dma_batch_parity --registered)
       set_tests_properties(dma_batch_registered_parity PROPERTIES TIMEOUT 120)
@@ -2234,6 +2286,12 @@ if(STRATA_NATIVE_EXPERTS)
   # MODEL-UPDATE: the pool's expert rows per format on T threads against the RAM bandwidth (a benchmark, no test)
   add_executable(kq_cpu_bench src/kernels/cpu/kq_cpu_bench.cpp)
   target_link_libraries(kq_cpu_bench PRIVATE strata_kernels_cpu)
+  # round 0216: a hash per (pair, token count, row range) of the pool's native expert calls, to compare a change switched off and on
+  add_executable(cpu_dispatch_dump src/kernels/cpu/cpu_dispatch_dump.cpp)
+  target_link_libraries(cpu_dispatch_dump PRIVATE strata_kernels_cpu)
+  # round 0216: the CPU pool's part of a decode window replayed without a GPU (per-layer format pairs, calibrated expert and row counts, cold DRAM)
+  add_executable(kq_window_bench src/kernels/cpu/kq_window_bench.cpp)
+  target_link_libraries(kq_window_bench PRIVATE strata_kernels_cpu)
   if(STRATA_BUILD_TESTS)
     add_test(NAME iq_avx2_parity COMMAND iq_avx2_parity)
     add_test(NAME iq_avx2_parity_iq3s_mt1 COMMAND ${CMAKE_COMMAND} -E env STRATA_IQ3S_MT1=1 $<TARGET_FILE:iq_avx2_parity>)
@@ -2309,7 +2367,7 @@ if(STRATA_NATIVE_EXPERTS)
     endif()
     # #136: the Q2_0 pack's prompt experts on Strata's own int8 kernels (moe_fused.cu, opt-in STRATA_PF_FUSED=1; CUDA
     # only: the HIP branch below keeps MMQ, and an sm_75 device or an sm_75-only build reports them unavailable)
-    add_library(strata_mmq STATIC src/prefill/moe_mmq.cu src/prefill/moe_fused.cu src/prefill/moe_fused_iq.cu
+    add_library(strata_mmq STATIC src/prefill/moe_mmq.cu src/prefill/moe_iq4_w4a8.cu src/prefill/moe_fused.cu src/prefill/moe_fused_iq.cu
                 src/prefill/mmq_nvfp4_w4a8.cu
                 src/prefill/ggml_cuda_host.cu ${_strata_quantize_cu})
     if(STRATA_EXPERIMENTAL_SM60 AND NOT STRATA_HIP_GFX906)
@@ -2514,6 +2572,14 @@ if(STRATA_NATIVE_EXPERTS)
         set_tests_properties(prefill_ws_parity_test PROPERTIES SKIP_RETURN_CODE 77 TIMEOUT 600)
       endif()
     endif()
+    if(STRATA_BUILD_TESTS AND NOT STRATA_HIP_GFX906)
+      # STRATA_IQ4_W4A8: the IQ4 W4A8 expert kernel against MMQ, bit for bit, on the group buffers and (with
+      # STRATA_MMQ_TAB) the pointer table (exit 77 without a card or off sm_80..sm_88; gfx906: a stub, no test)
+      add_executable(prefill_iq4_w4a8_test tests/cuda/prefill_iq4_w4a8_test.cu)
+      target_link_libraries(prefill_iq4_w4a8_test PRIVATE strata_mmq ggml-base ${_strata_gpu_runtime_target})
+      add_test(NAME prefill_iq4_w4a8_test COMMAND prefill_iq4_w4a8_test)
+      set_tests_properties(prefill_iq4_w4a8_test PROPERTIES SKIP_RETURN_CODE 77 TIMEOUT 300)
+    endif()
     if(STRATA_BUILD_TESTS AND _strata_mmq_tab_on)
       # STRATA_MMQ_DIRECT: the pointer-table MMQ bit for bit against the gathered MMQ (exit 77 below sm_80)
       add_executable(prefill_mmq_tab_test tests/cuda/prefill_mmq_tab_test.cu)

README.de.md

@@ -17,6 +17,7 @@
 <p align="center">
   <a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="Strata herunterladen"></a>
   <a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="MIT-Lizenz"></a>
+  <a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="Strata über GitHub Sponsors unterstützen"></a>
   <a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Strata auf Buy Me a Coffee unterstützen"></a>
 </p>
 
@@ -175,10 +176,23 @@ Führe **`strata connect`** aus, um die Einrichtung für dein Tool zu sehen. [Ve
 
 ## Strata unterstützen
 
-**Hilf mit, lokale KI für alle offen zu halten.** Ein Kaffee hilft, Entwicklung und Tests auf mehr Hardware zu finanzieren.
+**Hilf mit, lokale KI für alle offen zu halten.** Ein GitHub-Sponsoring oder ein Kaffee hilft, Entwicklung und Tests auf mehr Hardware zu finanzieren.
 
+<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor_on_GitHub-EA4AAA?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="GitHub Sponsors: Strata unterstützen" height="48"></a>
 <a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: Strata unterstützen" height="48"></a>
 
+### Treue Unterstützer
+
+Danke an alle, die Strata unterstützen.
+
+<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
+<a href="https://github.com/ryan4yin"><img src="https://github.com/ryan4yin.png?size=96" width="48" height="48" alt="ryan4yin" title="ryan4yin"></a>
+<a href="https://github.com/themactep"><img src="https://github.com/themactep.png?size=96" width="48" height="48" alt="themactep" title="themactep"></a>
+<a href="https://github.com/macintog"><img src="https://github.com/macintog.png?size=96" width="48" height="48" alt="macintog" title="macintog"></a>
+<a href="https://github.com/bisonbet"><img src="https://github.com/bisonbet.png?size=96" width="48" height="48" alt="bisonbet" title="bisonbet"></a>
+<a href="https://github.com/vizv"><img src="https://github.com/vizv.png?size=96" width="48" height="48" alt="vizv" title="vizv"></a>
+<!-- supporters:end -->
+
 [MIT-Lizenz](LICENSE) · [Danksagung und Modelllizenzen](docs/HOW_IT_WORKS.md#credits) · [Datenschutz](docs/SAFETY.md)
 
 <a id="option-c-let-your-ai-set-it-up"></a>

README.es.md

@@ -17,6 +17,7 @@
 <p align="center">
   <a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="Descargar Strata"></a>
   <a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="Licencia MIT"></a>
+  <a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="Patrocina Strata en GitHub Sponsors"></a>
   <a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Apoya Strata en Buy Me a Coffee"></a>
 </p>
 
@@ -175,10 +176,23 @@ Ejecuta **`strata connect`** para ver las instrucciones de tu herramienta. [Guí
 
 ## Apoya Strata
 
-**Ayuda a mantener la IA local abierta a todos.** Un café ayuda a financiar el desarrollo y las pruebas en más hardware.
+**Ayuda a mantener la IA local abierta a todos.** Un patrocinio en GitHub o un café ayuda a financiar el desarrollo y las pruebas en más hardware.
 
+<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor_on_GitHub-EA4AAA?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="GitHub Sponsors: patrocina Strata" height="48"></a>
 <a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: apoya Strata" height="48"></a>
 
+### Apoyos fieles
+
+Gracias a todas las personas que apoyan Strata.
+
+<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
+<a href="https://github.com/ryan4yin"><img src="https://github.com/ryan4yin.png?size=96" width="48" height="48" alt="ryan4yin" title="ryan4yin"></a>
+<a href="https://github.com/themactep"><img src="https://github.com/themactep.png?size=96" width="48" height="48" alt="themactep" title="themactep"></a>
+<a href="https://github.com/macintog"><img src="https://github.com/macintog.png?size=96" width="48" height="48" alt="macintog" title="macintog"></a>
+<a href="https://github.com/bisonbet"><img src="https://github.com/bisonbet.png?size=96" width="48" height="48" alt="bisonbet" title="bisonbet"></a>
+<a href="https://github.com/vizv"><img src="https://github.com/vizv.png?size=96" width="48" height="48" alt="vizv" title="vizv"></a>
+<!-- supporters:end -->
+
 [Licencia MIT](LICENSE) · [Créditos y licencias de los modelos](docs/HOW_IT_WORKS.md#credits) · [Privacidad](docs/SAFETY.md)
 
 <a id="option-c-let-your-ai-set-it-up"></a>

README.fr.md

@@ -17,6 +17,7 @@
 <p align="center">
   <a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="Télécharger Strata"></a>
   <a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="Licence MIT"></a>
+  <a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="Soutenir Strata sur GitHub Sponsors"></a>
   <a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Soutenir Strata sur Buy Me a Coffee"></a>
 </p>
 
@@ -175,10 +176,23 @@ Lancez **`strata connect`** pour voir la configuration de votre outil. [Guide de
 
 ## Soutenir Strata
 
-**Aidez à garder l'IA locale ouverte à tous.** Un café aide à financer le développement et les tests sur davantage de matériel.
+**Aidez à garder l'IA locale ouverte à tous.** Un parrainage sur GitHub ou un café aide à financer le développement et les tests sur davantage de matériel.
 
+<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor_on_GitHub-EA4AAA?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="GitHub Sponsors : soutenir Strata" height="48"></a>
 <a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee : soutenir Strata" height="48"></a>
 
+### Fidèles soutiens
+
+Merci à toutes les personnes qui soutiennent Strata.
+
+<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
+<a href="https://github.com/ryan4yin"><img src="https://github.com/ryan4yin.png?size=96" width="48" height="48" alt="ryan4yin" title="ryan4yin"></a>
+<a href="https://github.com/themactep"><img src="https://github.com/themactep.png?size=96" width="48" height="48" alt="themactep" title="themactep"></a>
+<a href="https://github.com/macintog"><img src="https://github.com/macintog.png?size=96" width="48" height="48" alt="macintog" title="macintog"></a>
+<a href="https://github.com/bisonbet"><img src="https://github.com/bisonbet.png?size=96" width="48" height="48" alt="bisonbet" title="bisonbet"></a>
+<a href="https://github.com/vizv"><img src="https://github.com/vizv.png?size=96" width="48" height="48" alt="vizv" title="vizv"></a>
+<!-- supporters:end -->
+
 [Licence MIT](LICENSE) · [Crédits et licences des modèles](docs/HOW_IT_WORKS.md#credits) · [Confidentialité](docs/SAFETY.md)
 
 <a id="option-c-let-your-ai-set-it-up"></a>

README.ja.md

@@ -17,6 +17,7 @@
 <p align="center">
   <a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="Strata をダウンロード"></a>
   <a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="MIT ライセンス"></a>
+  <a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="GitHub Sponsors で Strata を支援する"></a>
   <a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Buy Me a Coffee で Strata を支援する"></a>
 </p>
 
@@ -175,10 +176,23 @@ strata launch
 
 ## Strata を支援する
 
-**ローカル AI をみんなに開かれたものに。** コーヒー1杯が、開発と、より多くのハードウェアでのテストの支えになります。
+**ローカル AI をみんなに開かれたものに。** GitHub でのスポンサーやコーヒー1杯が、開発と、より多くのハードウェアでのテストの支えになります。
 
+<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor_on_GitHub-EA4AAA?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="GitHub Sponsors: Strata を支援する" height="48"></a>
 <a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: Strata を支援する" height="48"></a>
 
+### いつも支えてくださる方々
+
+Strata を支えてくださるすべての方に感謝します。
+
+<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
+<a href="https://github.com/ryan4yin"><img src="https://github.com/ryan4yin.png?size=96" width="48" height="48" alt="ryan4yin" title="ryan4yin"></a>
+<a href="https://github.com/themactep"><img src="https://github.com/themactep.png?size=96" width="48" height="48" alt="themactep" title="themactep"></a>
+<a href="https://github.com/macintog"><img src="https://github.com/macintog.png?size=96" width="48" height="48" alt="macintog" title="macintog"></a>
+<a href="https://github.com/bisonbet"><img src="https://github.com/bisonbet.png?size=96" width="48" height="48" alt="bisonbet" title="bisonbet"></a>
+<a href="https://github.com/vizv"><img src="https://github.com/vizv.png?size=96" width="48" height="48" alt="vizv" title="vizv"></a>
+<!-- supporters:end -->
+
 [MIT ライセンス](LICENSE) · [クレジットとモデルのライセンス](docs/HOW_IT_WORKS.md#credits) · [プライバシー](docs/SAFETY.md)
 
 <a id="option-c-let-your-ai-set-it-up"></a>

README.md

@@ -17,6 +17,7 @@
 <p align="center">
   <a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="Download Strata"></a>
   <a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="MIT license"></a>
+  <a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="Sponsor Strata on GitHub Sponsors"></a>
   <a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Support Strata on Buy Me a Coffee"></a>
 </p>
 
@@ -181,10 +182,23 @@ Run **`strata connect`** to see setup instructions for your tool. [Connection gu
 
 ## Support Strata
 
-**Help keep local AI open to everyone.** A coffee helps fund development and testing on more hardware.
+**Help keep local AI open to everyone.** A GitHub sponsorship or a coffee helps fund development and testing on more hardware.
 
+<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor_on_GitHub-EA4AAA?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="GitHub Sponsors: sponsor Strata" height="48"></a>
 <a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: support Strata" height="48"></a>
 
+### Loyal supporters
+
+Thank you to everyone who supports Strata.
+
+<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
+<a href="https://github.com/ryan4yin"><img src="https://github.com/ryan4yin.png?size=96" width="48" height="48" alt="ryan4yin" title="ryan4yin"></a>
+<a href="https://github.com/themactep"><img src="https://github.com/themactep.png?size=96" width="48" height="48" alt="themactep" title="themactep"></a>
+<a href="https://github.com/macintog"><img src="https://github.com/macintog.png?size=96" width="48" height="48" alt="macintog" title="macintog"></a>
+<a href="https://github.com/bisonbet"><img src="https://github.com/bisonbet.png?size=96" width="48" height="48" alt="bisonbet" title="bisonbet"></a>
+<a href="https://github.com/vizv"><img src="https://github.com/vizv.png?size=96" width="48" height="48" alt="vizv" title="vizv"></a>
+<!-- supporters:end -->
+
 [MIT license](LICENSE) · [Credits and model licenses](docs/HOW_IT_WORKS.md#credits) · [Privacy](docs/SAFETY.md)
 
 <a id="option-c-let-your-ai-set-it-up"></a>

README.pl.md

@@ -17,6 +17,7 @@
 <p align="center">
   <a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="Pobierz Stratę"></a>
   <a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="Licencja MIT"></a>
+  <a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="Wesprzyj Stratę na GitHub Sponsors"></a>
   <a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Wesprzyj Stratę na Buy Me a Coffee"></a>
 </p>
 
@@ -179,10 +180,23 @@ Uruchom **`strata connect`**, aby zobaczyć instrukcję konfiguracji dla swojego
 
 ## Wesprzyj Stratę
 
-**Pomóż zachować lokalną sztuczną inteligencję otwartą dla wszystkich.** Kawa pomaga finansować rozwój i testy na większej liczbie sprzętu.
+**Pomóż zachować lokalną sztuczną inteligencję otwartą dla wszystkich.** Sponsoring na GitHubie lub kawa pomaga finansować rozwój i testy na większej liczbie sprzętu.
 
+<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor_on_GitHub-EA4AAA?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="GitHub Sponsors: wesprzyj Stratę" height="48"></a>
 <a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: wesprzyj Stratę" height="48"></a>
 
+### Wierni wspierający
+
+Dziękujemy wszystkim, którzy wspierają Stratę.
+
+<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
+<a href="https://github.com/ryan4yin"><img src="https://github.com/ryan4yin.png?size=96" width="48" height="48" alt="ryan4yin" title="ryan4yin"></a>
+<a href="https://github.com/themactep"><img src="https://github.com/themactep.png?size=96" width="48" height="48" alt="themactep" title="themactep"></a>
+<a href="https://github.com/macintog"><img src="https://github.com/macintog.png?size=96" width="48" height="48" alt="macintog" title="macintog"></a>
+<a href="https://github.com/bisonbet"><img src="https://github.com/bisonbet.png?size=96" width="48" height="48" alt="bisonbet" title="bisonbet"></a>
+<a href="https://github.com/vizv"><img src="https://github.com/vizv.png?size=96" width="48" height="48" alt="vizv" title="vizv"></a>
+<!-- supporters:end -->
+
 [Licencja MIT](LICENSE) · [Podziękowania i licencje modeli](docs/HOW_IT_WORKS.md#credits) · [Prywatność](docs/SAFETY.md)
 
 <a id="option-c-let-your-ai-set-it-up"></a>

README.pt-BR.md

@@ -17,6 +17,7 @@
 <p align="center">
   <a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="Baixar o Strata"></a>
   <a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="Licença MIT"></a>
+  <a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="Apoie o Strata no GitHub Sponsors"></a>
   <a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="Apoie o Strata no Buy Me a Coffee"></a>
 </p>
 
@@ -175,10 +176,23 @@ Execute **`strata connect`** para ver as instruções da sua ferramenta. [Guia d
 
 ## Apoie o Strata
 
-**Ajude a manter a IA local aberta para todos.** Um café ajuda a financiar o desenvolvimento e os testes em mais hardware.
+**Ajude a manter a IA local aberta para todos.** Um patrocínio no GitHub ou um café ajuda a financiar o desenvolvimento e os testes em mais hardware.
 
+<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor_on_GitHub-EA4AAA?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="GitHub Sponsors: apoie o Strata" height="48"></a>
 <a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee: apoie o Strata" height="48"></a>
 
+### Apoiadores fiéis
+
+Obrigado a todas as pessoas que apoiam o Strata.
+
+<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
+<a href="https://github.com/ryan4yin"><img src="https://github.com/ryan4yin.png?size=96" width="48" height="48" alt="ryan4yin" title="ryan4yin"></a>
+<a href="https://github.com/themactep"><img src="https://github.com/themactep.png?size=96" width="48" height="48" alt="themactep" title="themactep"></a>
+<a href="https://github.com/macintog"><img src="https://github.com/macintog.png?size=96" width="48" height="48" alt="macintog" title="macintog"></a>
+<a href="https://github.com/bisonbet"><img src="https://github.com/bisonbet.png?size=96" width="48" height="48" alt="bisonbet" title="bisonbet"></a>
+<a href="https://github.com/vizv"><img src="https://github.com/vizv.png?size=96" width="48" height="48" alt="vizv" title="vizv"></a>
+<!-- supporters:end -->
+
 [Licença MIT](LICENSE) · [Créditos e licenças dos modelos](docs/HOW_IT_WORKS.md#credits) · [Privacidade](docs/SAFETY.md)
 
 <a id="option-c-let-your-ai-set-it-up"></a>

README.zh-CN.md

@@ -17,6 +17,7 @@
 <p align="center">
   <a href="https://github.com/Niko1221/Strata/releases/latest"><img src="https://img.shields.io/github/v/release/Niko1221/Strata?label=download&color=0ab887" alt="下载 Strata"></a>
   <a href="LICENSE"><img src="https://img.shields.io/github/license/Niko1221/Strata" alt="MIT 许可证"></a>
+  <a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor-EA4AAA?logo=githubsponsors&logoColor=fff" alt="在 GitHub Sponsors 上支持 Strata"></a>
   <a href="https://buymeacoffee.com/strataengine"><img src="https://img.shields.io/badge/Buy_me_a_coffee-FFDD00?logo=buymeacoffee&logoColor=000" alt="在 Buy Me a Coffee 上支持 Strata"></a>
 </p>
 
@@ -175,10 +176,23 @@ strata launch
 
 ## 支持 Strata
 
-**帮助本地 AI 保持开放。** 一杯咖啡能帮助我们在更多硬件上开发和测试。
+**帮助本地 AI 保持开放。** GitHub 赞助或一杯咖啡，都能帮助我们在更多硬件上开发和测试。
 
+<a href="https://github.com/sponsors/Niko1221"><img src="https://img.shields.io/badge/Sponsor_on_GitHub-EA4AAA?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="GitHub Sponsors：支持 Strata" height="48"></a>
 <a href="https://buymeacoffee.com/strataengine"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee：支持 Strata" height="48"></a>
 
+### 忠实支持者
+
+感谢每一位支持 Strata 的人。
+
+<!-- supporters:start (one line per public supporter, newest last; copy a line and change the GitHub name in all four places) -->
+<a href="https://github.com/ryan4yin"><img src="https://github.com/ryan4yin.png?size=96" width="48" height="48" alt="ryan4yin" title="ryan4yin"></a>
+<a href="https://github.com/themactep"><img src="https://github.com/themactep.png?size=96" width="48" height="48" alt="themactep" title="themactep"></a>
+<a href="https://github.com/macintog"><img src="https://github.com/macintog.png?size=96" width="48" height="48" alt="macintog" title="macintog"></a>
+<a href="https://github.com/bisonbet"><img src="https://github.com/bisonbet.png?size=96" width="48" height="48" alt="bisonbet" title="bisonbet"></a>
+<a href="https://github.com/vizv"><img src="https://github.com/vizv.png?size=96" width="48" height="48" alt="vizv" title="vizv"></a>
+<!-- supporters:end -->
+
 [MIT 许可证](LICENSE) · [致谢和模型许可证](docs/HOW_IT_WORKS.md#credits) · [隐私](docs/SAFETY.md)
 
 <a id="option-c-let-your-ai-set-it-up"></a>

apps/desktop/package.json

@@ -1,7 +1,7 @@
 {
   "name": "strata-desktop",
   "productName": "Strata",
-  "version": "0.2.11",
+  "version": "0.2.12",
   "private": true,
   "description": "Strata Desktop - local AI, tuned to one model.",
   "scripts": {

apps/desktop/src-tauri/Cargo.lock

@@ -3625,7 +3625,7 @@ checksum = "ff7589a1859b09232522bb9edf076ca7fcf14ded96685d3fa629db73023ffbb7"
 
 [[package]]
 name = "strata-desktop"
-version = "0.2.11"
+version = "0.2.12"
 dependencies = [
  "gtk",
  "serde",

apps/desktop/src-tauri/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "strata-desktop"
-version = "0.2.11"
+version = "0.2.12"
 description = "Strata Desktop — local AI, tuned to one model."
 edition = "2021"
 rust-version = "1.77"

apps/host/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "strata",
-  "version": "0.2.11",
+  "version": "0.2.12",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "strata",
-      "version": "0.2.11",
+      "version": "0.2.12",
       "dependencies": {
         "html-to-image": "^1.11.13",
         "katex": "^0.19.0",

apps/host/package.json

@@ -1,7 +1,7 @@
 {
   "name": "strata",
   "private": true,
-  "version": "0.2.11",
+  "version": "0.2.12",
   "type": "module",
   "description": "Strata \u2014 local AI, tuned to one model. Web shell + engine host.",
   "scripts": {

apps/host/scripts/make-df2-locales.py

@@ -202,6 +202,24 @@ add("Drafting: off (the cards were short of memory)",
     "Previsão: desligada (faltou memória nas placas)",
     "초안 예측: 꺼짐(카드 메모리가 부족했습니다)",
     "Anticipation : désactivée (les cartes manquaient de mémoire)")
+add("On your Mac, DFlash2 was faster than MTP on this model in our tests.",
+    "在您的 Mac 上，我们的测试中 DFlash2 在此模型上比 MTP 更快。",
+    "お使いの Mac では、私たちのテストでこのモデルにおいて DFlash2 は MTP より速くなりました。",
+    "Auf Ihrem Mac war DFlash2 bei diesem Modell in unseren Tests schneller als MTP.",
+    "En su Mac, en nuestras pruebas DFlash2 fue más rápido que MTP con este modelo.",
+    "На вашем Mac в наших тестах DFlash2 с этой моделью был быстрее MTP.",
+    "No seu Mac, nos nossos testes o DFlash2 foi mais rápido que o MTP com este modelo.",
+    "고객님의 Mac에서 저희 테스트 결과 이 모델에서는 DFlash2가 MTP보다 빨랐습니다.",
+    "Sur votre Mac, DFlash2 a été plus rapide que MTP avec ce modèle lors de nos tests.")
+add("DFlash2 is off by default on this graphics card for now; choose it in Drafting to use it.",
+    "目前在这张显卡上 DFlash2 默认关闭；如需使用，请在“预测草稿”中选择它。",
+    "現在、このグラフィックスカードでは DFlash2 は既定でオフです。使うには「先読み」で選んでください。",
+    "DFlash2 ist auf dieser Grafikkarte vorerst standardmäßig aus; wählen Sie es unter „Vorausraten“ aus, um es zu nutzen.",
+    "DFlash2 está desactivado de forma predeterminada en esta tarjeta gráfica por ahora; elíjalo en Predicción para usarlo.",
+    "На этой видеокарте DFlash2 пока выключен по умолчанию; чтобы использовать его, выберите его в разделе «Предсказание».",
+    "O DFlash2 está desligado por predefinição nesta placa de vídeo por agora; escolha-o em Previsão para o usar.",
+    "현재 이 그래픽 카드에서는 DFlash2가 기본적으로 꺼져 있습니다. 사용하려면 ‘초안 예측’에서 선택하세요.",
+    "DFlash2 est désactivé par défaut sur cette carte graphique pour le moment ; choisissez-le dans Anticipation pour l'utiliser.")
 here = Path(__file__).resolve().parent.parent / "src" / "locales"
 NBSP = " "
 

apps/host/server/native-gateway.mjs

@@ -1,7 +1,7 @@
 /** Preserve native protocol events and tool payloads; the gateway owns identity and cancellation. */
 import { once } from 'node:events';
 export async function nativeGateway({req,res,path,body,control,serving,fetcher=fetch}){
- let instance;try{instance=control.resolveInstance(body.model);}catch(e){res.writeHead(400,{'content-type':'application/json'});res.end(JSON.stringify({error:{message:e.message,type:'invalid_request_error',param:'model',code:null}}));return;}
+ let instance;try{instance=control.resolveInstance(body.model);}catch(e){/* #5312: an Anthropic client sends a model name of its own (Claude Desktop: claude-5-5-sonnet) and cannot change it: with exactly one model loaded the route answers with it */if(path==='/v1/messages'&&body.model){try{instance=control.resolveInstance(undefined);}catch{}}if(!instance){res.writeHead(400,{'content-type':'application/json'});res.end(JSON.stringify({error:{message:e.message,type:'invalid_request_error',param:'model',code:null}}));return;}}
  const abort=new AbortController(),started=Date.now();let sawDone=false;/* #4690: a client that hangs up after `data: [DONE]` (the OpenAI SDK does) has the whole answer: that is not a cancel. */res.on('close',()=>{if(!res.writableEnded&&!sawDone)abort.abort();});
  const entry=serving.open({client:req.headers['user-agent']?.slice(0,100)??'API client',endpoint:path.slice(3),model:instance.apiModelId,instanceId:instance.id,gpuIds:instance.gpus.map(String),cancel:()=>abort.abort()});
  let tokensIn=null,tokensOut=null,tps=null,promptTps=null;

apps/host/src/components/SettingsUpdates.tsx

@@ -88,7 +88,7 @@ export function SettingsUpdates() {
   </>;
 }
 
-/** "Strata 0.2.11 · Engine 0.1.57": the app's own version first, the engine's beside it (About). */
+/** "Strata 0.2.12 · Engine 0.1.58": the app's own version first, the engine's beside it (About). */
 export function AboutVersions() {
     const native = useNative();
     const app = appVersionLabel(useStore(s => s.setup?.app.version), native?.version);

apps/host/src/locales/df2-de.json

@@ -18,5 +18,7 @@
  "df17": "Vorausraten: MTP (DFlash2 ist nicht gestartet)",
  "df18": "Automatisch nutzt die Alle-Karten-Split-Engine mit der MTP-Entwurfsschicht: etwa {0} GB, einmalig herunterzuladen.",
  "df19": "Die Alle-Karten-Split-Engine konnte nicht starten, deshalb läuft die normale Schichtaufteilung.",
- "df20": "Vorausraten: aus (den Karten fehlte Speicher)"
+ "df20": "Vorausraten: aus (den Karten fehlte Speicher)",
+ "df21": "Auf Ihrem Mac war DFlash2 bei diesem Modell in unseren Tests schneller als MTP.",
+ "df22": "DFlash2 ist auf dieser Grafikkarte vorerst standardmäßig aus; wählen Sie es unter „Vorausraten“ aus, um es zu nutzen."
 }

apps/host/src/locales/df2-en.json

@@ -18,5 +18,7 @@
  "df17": "Drafting: MTP (DFlash2 didn't start)",
  "df18": "Auto uses the all-cards split engine with the MTP draft layer: about {0} GB to download, once.",
  "df19": "The all-cards split engine could not start, so the normal layer split runs.",
- "df20": "Drafting: off (the cards were short of memory)"
+ "df20": "Drafting: off (the cards were short of memory)",
+ "df21": "On your Mac, DFlash2 was faster than MTP on this model in our tests.",
+ "df22": "DFlash2 is off by default on this graphics card for now; choose it in Drafting to use it."
 }

apps/host/src/locales/df2-es.json

@@ -18,5 +18,7 @@
  "df17": "Predicción: MTP (DFlash2 no arrancó)",
  "df18": "Automático usa el motor de reparto entre todas las tarjetas con la capa de borrador MTP: unos {0} GB, una sola vez.",
  "df19": "El motor de reparto entre todas las tarjetas no pudo arrancar, así que se usa el reparto normal por capas.",
- "df20": "Predicción: desactivada (a las tarjetas les faltaba memoria)"
+ "df20": "Predicción: desactivada (a las tarjetas les faltaba memoria)",
+ "df21": "En su Mac, en nuestras pruebas DFlash2 fue más rápido que MTP con este modelo.",
+ "df22": "DFlash2 está desactivado de forma predeterminada en esta tarjeta gráfica por ahora; elíjalo en Predicción para usarlo."
 }

apps/host/src/locales/df2-fr.json

@@ -18,5 +18,7 @@
  "df17": "Anticipation : MTP (DFlash2 n’a pas démarré)",
  "df18": "Auto utilise le moteur réparti sur toutes les cartes avec la couche de brouillon MTP : environ {0} Go à télécharger, une seule fois.",
  "df19": "Le moteur réparti sur toutes les cartes n’a pas pu démarrer, la répartition normale par couches est donc utilisée.",
- "df20": "Anticipation : désactivée (les cartes manquaient de mémoire)"
+ "df20": "Anticipation : désactivée (les cartes manquaient de mémoire)",
+ "df21": "Sur votre Mac, DFlash2 a été plus rapide que MTP avec ce modèle lors de nos tests.",
+ "df22": "DFlash2 est désactivé par défaut sur cette carte graphique pour le moment ; choisissez-le dans Anticipation pour l’utiliser."
 }

apps/host/src/locales/df2-ja.json

@@ -18,5 +18,7 @@
  "df17": "先読み：MTP（DFlash2 は起動しませんでした）",
  "df18": "自動では、全カードのスプリットエンジンと MTP ドラフト層を使います。約 {0} GB を一度だけダウンロードします。",
  "df19": "全カードのスプリットエンジンを起動できなかったため、通常のレイヤー分割で動作しています。",
- "df20": "先読み：オフ（グラフィックスカードのメモリが不足していました）"
+ "df20": "先読み：オフ（グラフィックスカードのメモリが不足していました）",
+ "df21": "お使いの Mac では、私たちのテストでこのモデルにおいて DFlash2 は MTP より速くなりました。",
+ "df22": "現在、このグラフィックスカードでは DFlash2 は既定でオフです。使うには「先読み」で選んでください。"
 }

apps/host/src/locales/df2-ko.json

@@ -18,5 +18,7 @@
  "df17": "초안 예측: MTP(DFlash2가 시작되지 않았습니다)",
  "df18": "자동은 모든 카드를 쓰는 분할 엔진과 MTP 초안 층을 사용합니다(약 {0}GB, 한 번만 내려받음).",
  "df19": "모든 카드를 쓰는 분할 엔진을 시작하지 못해 일반 레이어 분할로 실행합니다.",
- "df20": "초안 예측: 꺼짐(카드 메모리가 부족했습니다)"
+ "df20": "초안 예측: 꺼짐(카드 메모리가 부족했습니다)",
+ "df21": "고객님의 Mac에서 저희 테스트 결과 이 모델에서는 DFlash2가 MTP보다 빨랐습니다.",
+ "df22": "현재 이 그래픽 카드에서는 DFlash2가 기본적으로 꺼져 있습니다. 사용하려면 ‘초안 예측’에서 선택하세요."
 }

apps/host/src/locales/df2-pt.json

@@ -18,5 +18,7 @@
  "df17": "Previsão: MTP (o DFlash2 não iniciou)",
  "df18": "O automático usa o motor de divisão entre todas as placas com a camada de rascunho MTP: cerca de {0} GB, uma vez.",
  "df19": "O motor de divisão entre todas as placas não conseguiu iniciar, então a divisão normal por camadas está em uso.",
- "df20": "Previsão: desligada (faltou memória nas placas)"
+ "df20": "Previsão: desligada (faltou memória nas placas)",
+ "df21": "No seu Mac, nos nossos testes o DFlash2 foi mais rápido que o MTP com este modelo.",
+ "df22": "O DFlash2 está desligado por predefinição nesta placa de vídeo por agora; escolha-o em Previsão para o usar."
 }

apps/host/src/locales/df2-ru.json

@@ -18,5 +18,7 @@
  "df17": "Предсказание: MTP (DFlash2 не запустился)",
  "df18": "Авто использует движок с разделением по всем картам и черновой слой MTP: около {0} ГБ, скачивается один раз.",
  "df19": "Движок с разделением по всем картам не запустился, поэтому работает обычное послойное разделение.",
- "df20": "Предсказание: выключено (на картах не хватило памяти)"
+ "df20": "Предсказание: выключено (на картах не хватило памяти)",
+ "df21": "На вашем Mac в наших тестах DFlash2 с этой моделью был быстрее MTP.",
+ "df22": "На этой видеокарте DFlash2 пока выключен по умолчанию; чтобы использовать его, выберите его в разделе «Предсказание»."
 }

apps/host/src/locales/df2-zh.json

@@ -18,5 +18,7 @@
  "df17": "预测草稿：MTP（DFlash2 未能启动）",
  "df18": "自动模式使用全部显卡的拆分引擎和 MTP 草稿层：约 {0} GB，只下载一次。",
  "df19": "全显卡拆分引擎未能启动，因此改用常规的分层拆分。",
- "df20": "预测草稿：关闭（显卡内存不足）"
+ "df20": "预测草稿：关闭（显卡内存不足）",
+ "df21": "在您的 Mac 上，我们的测试中 DFlash2 在此模型上比 MTP 更快。",
+ "df22": "目前在这张显卡上 DFlash2 默认关闭；如需使用，请在“预测草稿”中选择它。"
 }

apps/host/tests/drafting.test.mjs

@@ -95,7 +95,7 @@ test('every sentence of the Drafting choice is in the catalog in all nine langua
     // the DFlash2 option (df2-<lang>.json): the paragraph under the dropdown, the lines about the card, the two new notes
     "Drafting: MTP (DFlash2 isn't available on this graphics card)", "There is no internet connection, so the DFlash2 drafter can't be downloaded now.",
     'On graphics cards like yours, DFlash2 was faster than MTP on some kinds of text and slower on others in our tests.',
-    'On graphics cards like yours, DFlash2 was slower than MTP in our tests.', 'DFlash2 has not been tested on your graphics card yet.', 'SSD: not needed'];
+    'On graphics cards like yours, DFlash2 was slower than MTP in our tests.', 'DFlash2 has not been tested on your graphics card yet.', 'On your Mac, DFlash2 was faster than MTP on this model in our tests.', 'DFlash2 is off by default on this graphics card for now; choose it in Drafting to use it.', 'SSD: not needed'];
   const slots = t => (t.match(/\{\d+\}/g) ?? []).sort().join();
   for (const text of sentences) {
     const id = idOf(text);

apps/host/tests/native-gateway.test.mjs

@@ -62,3 +62,20 @@ test('an Anthropic stream: the prompt count (message_start) and the output count
   assert.equal(records[0].tokensIn,29757);assert.equal(records[0].tokensOut,561);assert.equal(records[0].usageSource,'reported');
  }finally{await close(gateway);await close(engine);}
 });
+
+test('an Anthropic client that sends a model name of its own gets the one loaded model (#5312); OpenAI and Responses stay exact',{timeout:10000},async()=>{
+ const received=[];const engine=http.createServer(async(req,res)=>{let text='';for await(const chunk of req)text+=chunk;received.push(JSON.parse(text));res.writeHead(200,{'content-type':'application/json'});res.end(JSON.stringify({type:'message',content:[{type:'text',text:'Hi'}],usage:{input_tokens:1,output_tokens:1}}));});
+ const port=await listen(engine);const records=[];
+ const serving={open:value=>{const r={...value,id:String(records.length)};records.push(r);return r;},mark:(id,value)=>Object.assign(records[Number(id)],value),finish:(id,value)=>Object.assign(records[Number(id)],value)};
+ const one={id:'only',apiModelId:'iq3_s',port,key:'k',gpus:[0]},two={id:'other',apiModelId:'iq2_xs',port,key:'k',gpus:[1]};
+ const make=instances=>({resolveInstance:id=>{const found=instances.find(i=>i.apiModelId===id)??(!id&&instances.length===1?instances[0]:undefined);if(!found)throw new Error('Choose a loaded model ID.');return found;}});
+ const serve=control=>http.createServer(async(req,res)=>{let text='';for await(const chunk of req)text+=chunk;await nativeGateway({req,res,path:req.url,body:JSON.parse(text),control,serving});});
+ const single=serve(make([one])),pair=serve(make([one,two]));const p1=await listen(single),p2=await listen(pair);
+ const call=(p,path,body)=>fetch(`http://127.0.0.1:${p}${path}`,{method:'POST',headers:{'content-type':'application/json'},body:JSON.stringify(body)});
+ try{
+  let r=await call(p1,'/v1/messages',{model:'claude-5-5-sonnet',messages:[{role:'user',content:'Hi'}],max_tokens:5});assert.equal(r.status,200);await r.text();assert.equal(received.at(-1).model,'iq3_s');
+  r=await call(p2,'/v1/messages',{model:'claude-5-5-sonnet',messages:[],max_tokens:5});assert.equal(r.status,400);assert.equal((await r.json()).error.param,'model');
+  r=await call(p1,'/v1/chat/completions',{model:'claude-5-5-sonnet',messages:[]});assert.equal(r.status,400);
+  r=await call(p1,'/v1/responses',{model:'claude-5-5-sonnet',input:'Hi'});assert.equal(r.status,400);
+ }finally{await close(single);await close(pair);await close(engine);}
+});

apps/host/tests/qa1-storage.test.mjs

@@ -57,5 +57,5 @@ test('Cargo.toml, tauri.conf.json and the desktop package agree on the app versi
   const cargo = /^version\s*=\s*"([^"]+)"/m.exec(await readFile(new URL('src-tauri/Cargo.toml', root), 'utf8'))[1];
   const pkg = JSON.parse(await readFile(new URL('package.json', root), 'utf8')).version;
   assert.equal(cargo, pkg);
-  assert.equal(cargo, '0.2.11');
+  assert.equal(cargo, '0.2.12');
 });

data/dflash2-tuned/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/dflash2-tuned/NOTICE.md

@@ -0,0 +1,25 @@
+# Strata's tuned DFlash2 drafter (Apache-2.0, modified by Strata)
+
+`qwen3.8-flash-next-dflash2.strata1.q4_0-selq4.pack` is a **modified copy** of the DFlash2 draft model
+[incoai/Qwen3.8-Flash-Next-DFlash2](https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2), revision
+`a22a1744e85c17364dcc54a6a420af7135d4c7f6`, Copyright Inco AI, licensed under the Apache License, Version 2.0 (the text is in `LICENSE`
+beside this file). The upstream card names Inco AI as the author; the upstream repository carries no separate NOTICE file.
+
+**What Strata changed:** the weights. Every parameter of the published draft model (603 M) was fine-tuned by Strata on the text that
+Qwen3.8-Flash-Next quantizations (Unsloth UD-Q4_K_XL and UD-IQ4_XS-v31) wrote for Strata-made prompts (repository files and documents,
+synthetic records; no user data), then packed into the engine's runtime format (Q4_0 matrices, Q4_0 selector tables; the layout of
+`tools/dflash2_pack.py --q4_0-selector`). The pack is the same size as the unmodified one (339,694,592 bytes). The model code and the
+architecture are unchanged. The pack's SHA-256 is `4df2e5ab2b47c064afbe662e16e4f06569877125c2d0e2123a4ac755266c664c`.
+
+On Macs Strata runs it with `--draft-conf 0.7`; on CUDA (every expert resident) and on Strix Halo it runs at the confidence those platforms already use. It changes how many words are guessed right, not the model: every guess is checked by the target model (greedy output was identical to the unmodified drafter's on Metal and CUDA in the tests; on HIP the text can differ by window length, as with any confidence change).
+
+A second file, `qwen3.8-flash-next-dflash2.strata1.q4_0-bf16sel.pack` (522,458,112 bytes, SHA-256 `34ca17fee701ff088b6069acc35a26661d237953f5d114be0a3f7dae3ec5cda9`), holds the same tuned weights packed with the bf16 selector tables
+(`tools/dflash2_pack.py --q4_0`) for the engines that read no other layout (Intel Arc, SYCL). Everything above applies to it too; Strata uses it on Arc at that card's usual confidence.
+
+Training on the outputs of the Qwen3.8-Flash-Next quantizations: the owner of Strata confirmed on 2026-11-06 that the Qwen model terms allow
+training on its outputs.
+
+The idea of the Q4_0 selector tables is nazeshinjite's (Apache-2.0): see `docs/DETAILS.md`, "Credits and licenses".
+
+This file and `LICENSE` are copied next to the drafter file (`<data folder>/dflash2/`) when Strata fetches it, and are part of Strata's own
+tree. Strata's code stays MIT ([`LICENSE`](../../LICENSE)); this drafter's weights are Apache-2.0.

docker/build.sh

@@ -64,7 +64,9 @@ case "$commit" in *[!0-9a-f]*|"") commit="" ;; esac
 # tools/: every .py at any depth, because the bundle's DeepSeek scripts are nested (INSTALLER_SCRIPTS in package_terminal.py names dsv41/*:
 # the Dockerfile's terminal stage stops at "installer tool missing: tools/dsv41/tune.py" when only the top-level modules are in the context)
 cd "$repo"
-paths="packaging bin strata.py strata_control strata_cli serve requirements.txt requirements-monitor.txt docs/MANAGEMENT_PROTOCOL.md docs/TERMINAL_LAUNCH.md data bench/prompts/standard-v1 setup.py ref/load.py CMakeLists.txt"
+# metal/tools/: the two repack scripts the bundle carries (INSTALLER_METAL_TOOLS in package_terminal.py; without them the terminal stage
+# stops at "No such file or directory: '/src/metal/tools/metal_repack.py'", as the v0.2.11 image build did)
+paths="packaging bin strata.py strata_control strata_cli serve requirements.txt requirements-monitor.txt docs/MANAGEMENT_PROTOCOL.md docs/TERMINAL_LAUNCH.md data bench/prompts/standard-v1 setup.py ref/load.py CMakeLists.txt metal/tools/metal_repack.py metal/tools/stream_experts.py"
 if git rev-parse --git-dir >/dev/null 2>&1; then
   # shellcheck disable=SC2086
   { git ls-files -z -- $paths ':(glob)tools/**/*.py'; } | tar --null -T - -cf - | tar -xf - -C "$ctx/terminal-src"

docs/DETAILS.md

@@ -294,7 +294,7 @@ When experts are read from GGUF files in place, Linux runs three things by defau
 For an `experts.bin` pack, io prefetch and the gate switch on when the page cache has room for at least 55% of the experts
 (the tightest cgroup limit or `MemAvailable`, asked once at start). Below that the pack reads as before and the log says so;
 `STRATA_IO_PREFETCH=1` forces it. The container RAM check counts a cgroup's clean page cache as available. The startup log
-names what is on. Windows and macOS are unchanged. On a Radeon 780M iGPU, prefetch together with the adaptive tier under
+names what is on. When every expert is in the GPU caches (the decode reads nothing from the file tier) the defaults, the decode gate and the routing-aware prefetch are off, and they come back when a request starts with fewer experts held (a VRAM resize, an eviction); `STRATA_IO_PREFETCH=1` keeps io prefetch and the look-ahead, `STRATA_LOOKAHEAD=1` keeps the look-ahead, `STRATA_FILE_TIER_IDLE=0` turns this off. Windows and macOS are unchanged. On a Radeon 780M iGPU, prefetch together with the adaptive tier under
 memory pressure reset the GPU in most runs; with `--adapt-every 100000` there were none, and the cause is not found.
 
 The weights and the MTP draft files are asked for ahead of their reads at start (madvise and `posix_fadvise` WILLNEED, 128 KiB
@@ -492,6 +492,7 @@ applies at the next start. Network, key, MCP and program keys are not editable t
 | `effort_position` | `"start"` | `"end"` puts a `low` or `medium` effort in a short system turn right before the answer, so a request that only changes the effort reuses the conversation cache. How well the model follows a level in that place is not measured. |
 | `anthropic_thinking` | as the template | `"on_request"` renders a `/v1/messages` request that asks for no thinking without thinking, as Anthropic does. Claude Code's short helper calls need it. |
 | `tool_call_recovery` | on (since 0.2.8; off in 0.2.7 and before) | Also reads a declared tool's call written as `<parameter=NAME>`, as JSON inside `<tool_call>`, as a `<function=NAME>` block without `<tool_call>`, or as a second call inside one `<tool_call>`. On 1,462 agent turns 98.3% read as intended with it and 92.9% without; the 49 specimens of the test corpus read the same with it on or off. Only a declared tool's call is ever recovered; anything else stays text. `"tool_call_recovery": false` turns it off. |
+| `undeclared_tool_calls` | `"text"` | A well-formed call of a tool the request did not declare stays the text the model wrote, with `strata_rejected_tool_calls` beside it (the default, as since 0.2.2). `"call"` returns it as a tool call instead, for a client with its own policy for unknown names: it can answer the model with the tool names it has and ask again, which a call kept as text never lets it do. The environment variable `STRATA_UNDECLARED_TOOL_CALLS` sets the same; the config wins. Only a call the parser reads as a whole call is returned; prose and quoted examples stay text (#4622). |
 | `env_to_user` | off | Moves opencode's `<env>` block (working directory, date) to the first user message so the system turn is the same for every session and the disk prefix cache restores it. |
 | `return_progress` | off | Per request `"return_progress": true` adds llama.cpp's `prompt_progress` (`total`, `cache`, `processed`, `time_ms`) to the stream, one update per `--prefill` chunk. A prompt shorter than one chunk sends nothing. |
 | `prevent_sleep` | off | Windows only: keeps the PC awake while a request runs or waits (`SetThreadExecutionState`). On Linux and macOS one startup line says it does nothing. |
@@ -1072,11 +1073,13 @@ them on your own card.
 | --- | --- | --- | --- |
 | `STRATA_ADAPT_OVERLAP=1` | One GPU, the whole arena in RAM, no `--adapt-async`, `--pipeline-windows`, `STRATA_ADAPT_LAG`, `STRATA_ADAPT_NOWAIT`, `STRATA_DMA_BATCH`; never on a layer split | The adaptive tier's copies land while the next window runs, with the same tokens. The startup log names the state and why it cannot run. | Gains on RTX A4000 and Tesla P100 (+4 to +8%), loses on an RTX 5070 under Windows and an RTX 5090 Laptop. Opt-in on every backend. With the per-card defaults set `STRATA_ADAPT_LAG=1 STRATA_ADAPT_NOWAIT=0` beside it. |
 | `STRATA_ADAPT_LAG=2` | CUDA | The window takes the adaptive tier's swaps once they are two windows old. | Gains on a Tesla P100; on by default on sm_86 and sm_60. |
+| `STRATA_ADAPT_FETCH=1\|2` | CUDA, one GPU, serial windows, a PCIe share (`--pcie-frac`), no `--batch`, `--pipeline-windows`, `--adapt-async`, `--pcie-mode dma\|direct`, `STRATA_ADAPT_OVERLAP=1` | The PCIe share takes each layer's most-routed misses, and a fetched expert routed at least `STRATA_ADAPT_FETCH_MIN` (1) and `STRATA_ADAPT_FETCH_MARGIN` (0.5) more than the layer's least-routed resident expert that the window does not route is copied straight into that expert's slot and becomes resident. `=2` leaves the tier's own swaps out of a layer where an admission ran since the tier's last round. The startup log names the state and why it cannot run; `STRATA_ADAPT_FETCH_VERIFY=1` (`strata generate`) reads every admitted slot back. | IQ3_S on an RTX PRO 5000 with a 3,896-slot cache (16% of the experts) and `--pcie-frac 0.25`: decode `=2` +10.2%, `=1` +5.0% (4 rounds, same-arm spread -0.8%), hit rate 54-61% to 71-78%; token forcing over 14,167 tokens: mismatch 2.73% against 2.53% and 2.53% for two unset arms, +0.19 pt (90% CI -0.02 to +0.42), PASS. Opt-in; the PR's NVFP4 pack gave -5.6% a round. |
 | `STRATA_STAGE_PIN=1` | GGUF read in place | Page-locks the stage buffers the cache fill copies from. Unset, it stays off: with it on, IQ3_S decode was corrupted after a 4096-token prompt in an earlier release gate. | Keep off until that is fixed. |
 | `STRATA_PCIE_BALANCE=1` | CUDA, one GPU, no `--batch`, no `--pcie-frac` | Each verify layer reads over PCIe the number of its missed experts that balances the two sides. | Helps an RTX A4000 arena (+5.6%), hurts others slightly. |
 | `STRATA_GDN_CHUNKED=1` | CUDA, 128 or more SMs by itself (`=2` forces it); HIP gfx1100-gfx1102, gfx1150, gfx1151 | The DeltaNet recurrence of the prompt in chunks of 32 tokens (CUDA) or 16-token WMMA chunks (HIP). Changes rounding. | 1.8x on the recurrence on an RTX 5090; slower on smaller cards. See [AMD_HIP.md](AMD_HIP.md#opt-in-chunked-prompt-gdn-on-gfx11). |
 | `STRATA_FS_SLOTS=N` | CUDA | Foresight swap space: N VRAM slots per layer refilled with missed and predicted experts. | The smaller cache cost more than the slots returned in every box measured. Left off. |
 | `STRATA_MMQ_DIRECT=1` | CUDA, native packs; needs a build with `-DSTRATA_MMQ_TAB=ON` (off by default because it costs 36-38 MiB of VRAM per GPU) | The prompt's MMQ reads resident experts in their cache slots instead of copying them. Same bits. | Faster "read in" time with every expert resident. |
+| `STRATA_IQ4_W4A8=1` | CUDA sm_80..sm_88 (Ampere) and sm_120 (Blackwell, only where the prompt experts run on MMQ: the fused int8 kernels are its default and are faster), IQ4_XS / IQ4_NL experts (`=2`: the IQ4_NL down products only) | The prompt's IQ4 expert products run on a pipelined int8 tensor-core kernel instead of MMQ: the weights stay 4-bit in registers, a cp.async pipeline feeds the K steps. Same bits as MMQ (`prefill_iq4_w4a8_test`); a product it does not cover (another type or GPU, a launch MMQ would split along K) stays on MMQ. | Faster prompts on IQ4 packs; measurements in the pull request. |
 | `--prefill-head-alias` | `--serve`, one GPU | The output head's VRAM holds the prompt buffers while a prompt is read ([PREFILL_HEAD_ALIAS.md](PREFILL_HEAD_ALIAS.md)). | For 8 GB cards. |
 | `STRATA_EMB_REUSE_ACCOUNT=1` | CUDA | The planner counts the VRAM saved because the half-output buffer `bo` reuses the embedding buffer `emb`, so the auto chunk can grow. Changes prompt rounding. | Prompt about +3.9% on an RTX 3060, no KL measurement yet. |
 | `STRATA_VERIFY_ASYNC_LAUNCH=1` | CUDA, single-request verifier | Runs the CUDA graph launch of a verify window on a separate host thread so the caller keeps serving CPU expert doorbells. A diagnostic for Compute Sanitizer runs. | |
@@ -1118,6 +1121,18 @@ measured. "Exact" means the 10-prompt and 4K/20K identity gate with the switch o
 `STRATA_DEQUANT_WIDE` (wide dequantization of Q4_K, Q5_K, Q6_K and IQ4_XS for the prompt's dense weights) is on by default on
 sm_86 and sm_61, bit-exact. `STRATA_DEQUANT_WIDE=0` turns it off.
 
+`STRATA_SELECT_TWO_PASS=1` (opt-in; sm_80+, the prompt path's tensor-core block scorer): every block is scored once in
+single-pass TF32 with a per-block error bound, and again in the usual 3xTF32 only where the bound reaches the query's
+top-k cut-off (a query the bound cannot settle has its 16-query tile scored again in full), so the selected cells, their
+order and the ties are exactly the 3xTF32 scorer's (the bound and the rule: `include/strata/kernels/qsa_two_pass.hpp`).
+On an RTX 3090 (sm_86, PCIe 3.0, 250 W; the v0.2.6 release defaults, 2 pairs of 2 runs) a 256K prompt read 5.2% faster
+(1,887 to 1,984 tokens/s), a 128K prompt 2.2% faster and a 32K prompt the same within 1%; decode and the answers did not
+change. Calls whose last query has `STRATA_SELECT_TWO_PASS_MIN` blocks or more take it (default 4096, 16K cells).
+`STRATA_SELECT_TWO_PASS_CHECK=1` (a slow diagnostic) runs the 3xTF32 scorer beside it and prints, after each prompt, the
+queries whose ids differ (must be 0). Where the scorer cannot run (HIP, Metal and SYCL builds, a card below sm_80 or
+`STRATA_QSA_WARP`, the layer-major prompt path) a set switch prints one line saying it is not used there, and the
+selection is unchanged.
+
 `STRATA_PREFILL_ROUTED_STREAM=1` (opt-in; the streamed prompt walk of a chunk of 1,024 tokens or more, one GPU, the
 issuer thread): the issuer keeps at most `STRATA_PREFILL_ROUTED_STREAM_DEPTH` copies (default 16) queued ahead of the
 PCIe link, each layer's routing is handed to it as soon as the layer's router has run, and an expert not yet copied that
@@ -1390,6 +1405,7 @@ Strata itself: [MIT](../LICENSE). The model files are not part of it; their lice
   [nazeshinjite/Qwen3.8-Flash-Next-DFlash2-GGUF](https://huggingface.co/nazeshinjite/Qwen3.8-Flash-Next-DFlash2-GGUF) (Apache-2.0) stores the drafter's
   two selector tables as Q4_0; Strata converts the checked pack on your computer (`tools/dflash2_pack.py --convert-selector`) and checks the bytes
   against the same pin.
+- Strata's tuned DFlash2 drafter (`data/dflash2-tuned/`, Apache-2.0, **modified by Strata**): a fine-tuned copy of the weights of [incoai/Qwen3.8-Flash-Next-DFlash2](https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2) (revision a22a1744, Apache-2.0, Inco AI); the original notices are kept, the licence text and a notice of the change are copied next to the file when it is fetched.
 - The web app's font: [Outfit](https://github.com/Outfitio/Outfit-Fonts) (SIL Open Font License 1.1, see `serve/web/fonts/OFL.txt`). Its
   Monitor tab started from @code-martin's dashboard idea (PR #22).
   Math is drawn by [KaTeX](https://katex.org) (MIT, vendored in `serve/web/vendor/katex`). The chat page's wide-window layout, svg/html preview, math, Load/Unload, saved

docs/MAC.md

@@ -97,6 +97,10 @@ What the first run does (each step is skipped when it is already done):
 5. **Downloads the model.** The download resumes if interrupted. `--gguf-dir` uses files you already have.
 6. **Prepares the model:** the pack (seconds) and the MTP draft layer (about 5 GB of the original checkpoint). On a Mac that
    streams, the experts' copy takes the draft layer's place (about 3 minutes on the M4).
+   For UD-Q4_K_XL and UD-IQ4_XS-v31 it also downloads the DFlash2 drafter (0.34 GB, from Strata's signed update feed): the server then drafts
+   with DFlash2 instead of the MTP layer (`--spec 6`, `--draft-conf 0.7`; Apache-2.0, a copy of incoai/Qwen3.8-Flash-Next-DFlash2 that Strata
+   tuned, notice in `data/dflash2-tuned/`). With no internet the download is skipped without a word and the MTP runs as before;
+   `./setup.sh --setup` fetches it later. IQ3_S and the other sets keep the MTP.
 7. **Writes `strata-<model>.json` and `run-<model>.sh`, then starts the server.**
 
 After that, `./setup.sh` or `./run-q2_0.sh` starts the model directly. The server is the same as on a PC: the OpenAI-compatible

include/strata/core/expert_source.hpp

@@ -352,7 +352,8 @@ private:
     std::condition_variable cv_;
     bool quit_ = false, pending_ = false, busy_ = false;
     int64_t layer_ = -1, n_tok_ = 0;
-    const int32_t* host_res_ = nullptr;
+    std::vector<int32_t> res_;   ///< the residency rows of the layers predicted, copied at submit (admissions write the table)
+    bool has_res_ = false;
     uint64_t sub_gen_ = 0;
     ForesightSwap* fs_ = nullptr;                   ///< (under mu_)
     std::vector<float> x_;
@@ -386,6 +387,9 @@ struct GpuPlanSink {
     /// copy engine.  counts[0] = VRAM groups, counts[1] = all entries, counts[2] = PCIe groups.
     unsigned long long* ptr2 = nullptr;
     int32_t* start2 = nullptr;
+    /// STRATA_ADAPT_FETCH (pcie_mode 2): per PCIe group, the VRAM cache slot its blob is copied into instead of
+    /// staging (0: staging).  staging_cap entries.
+    unsigned long long* dst2 = nullptr;
     unsigned long long staging = 0;
     int64_t staging_cap = 0;
     int64_t cap = 0;
@@ -651,6 +655,24 @@ struct ExpertDispatch {
     /// whose routed ids all look resident there publishes no activation (doorbell_publish_res) - counted
     const int32_t* dres_check = nullptr;
     int64_t stale_x = 0;
+    /// Opt-in (STRATA_ADAPT_FETCH): the PCIe share admits into the VRAM tier.  Its blobs cross into VRAM anyway, so the
+    /// layer's most-routed misses (`usage`) take the share, and one routed at least `fetch_min` and `fetch_margin` more
+    /// than its layer's least-routed resident expert not routed in the window is copied into that one's slot instead
+    /// of staging - a swap with no RAM read of its own.  `res_mut` is `host_res`, written here (and listed in
+    /// `res_dirty_idx`); `in_flight`: experts the adaptive tier is copying in.
+    int fetch_admit = 0;
+    float fetch_min = 1.0f, fetch_margin = 0.5f;
+    int32_t* res_mut = nullptr;
+    std::vector<uint8_t> in_flight;
+    std::vector<uint8_t> fetch_mark;   ///< per expert of the layer: routed in the window
+    /// STRATA_ADAPT_FETCH: whether the pool can admit into `layer` now (a window still needs a PCIe share there)
+    bool fetch_can(int64_t layer) const;
+    /// per layer: the PCIe share fetched there since the tier's last round (the tier leaves those layers' fetchable
+    /// candidates to the admission under STRATA_ADAPT_FETCH=2, and swaps the rest itself)
+    std::vector<uint8_t> fetch_avail;
+    int64_t fetch_admitted = 0;
+    bool fetch_log_on = false;          ///< STRATA_ADAPT_FETCH_VERIFY: keep the admissions for a read-back
+    std::vector<std::pair<int64_t, int32_t>> fetch_log;   ///< (layer * n_expert + expert, slot)
     int64_t multi_misses = 0;      ///< distinct (layer, expert) pairs the CPU computed in verify windows
     int64_t multi_entries = 0;     ///< routed (token, expert) entries the CPU served in verify windows
     /// Multi-GPU: the second GPU's tier.  Its experts are computed there instead of on the CPU (kind 2).

include/strata/core/flag_order.hpp

@@ -0,0 +1,59 @@
+// include/strata/core/flag_order.hpp - flag B is raised behind the copies queued before it (PR #5096, @AlanJager).
+//
+// Flag B tells the graph that a layer's PCIe experts have landed in staging.  It is one counter that only rises, and the
+// graph waits for "flag B >= layer + 1".  A non-empty plan raises it from a host function on the copy stream, so it
+// rises after that plan's copies.  An empty plan used to raise it on the spot, even when the copy stream still held an
+// earlier layer's (or the other token group's) uploads: the counter jumped past a layer whose bytes had not landed, and
+// the graph read a half-written staging slot.  raise_flag_b_ordered() keeps the counter behind every copy queued on the
+// stream, and raises it directly only when nothing was queued in this window.  The plain raise is shared with Verifier.
+#pragma once
+
+#include <cuda_runtime.h>
+
+#include <cstdint>
+#if defined(_WIN32)
+#include <intrin.h>
+#endif
+
+namespace strata::core {
+
+/// Flag B only rises: a host function of an earlier layer may run after a later layer already raised it directly.
+inline void flag_raise_max(uint32_t* flag, uint32_t value) {
+#if defined(_WIN32)
+    volatile long* f = (volatile long*) flag;
+    long cur = *f;
+    while ((uint32_t) cur < value) {
+        const long prev = _InterlockedCompareExchange(f, (long) value, cur);
+        if (prev == cur) break;
+        cur = prev;
+    }
+#else
+    uint32_t cur = __atomic_load_n(flag, __ATOMIC_SEQ_CST);
+    while (cur < value && !__atomic_compare_exchange_n(flag, &cur, value, false, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)) {}
+#endif
+}
+
+/// The argument of the host function (it must outlive the call: it lives in the Verifier's table).
+struct FlagRaise {
+    uint32_t* flag;
+    uint32_t value;
+};
+
+/// Raise `flag` to `value` once every copy queued so far on `copy` has landed.  `copies_queued` is false when nothing was
+/// queued on the stream since it was last drained: then there is nothing to wait for and the flag rises at once.
+/// Returns the launch status (cudaSuccess when the flag was raised directly).  On a failure the flag has NOT been raised:
+/// the caller must not release the consumers (Verifier::fetch_dma stops the engine).
+inline cudaError_t raise_flag_b_ordered(bool copies_queued, cudaStream_t copy, FlagRaise* slot, uint32_t* flag, uint32_t value) {
+    if (!copies_queued || copy == nullptr) {
+        flag_raise_max(flag, value);
+        return cudaSuccess;
+    }
+    slot->flag = flag;
+    slot->value = value;
+    return cudaLaunchHostFunc(copy, [](void* p) {
+        FlagRaise* s = (FlagRaise*) p;
+        flag_raise_max(s->flag, s->value);
+    }, slot);
+}
+
+}  // namespace strata::core

include/strata/core/peer_experts.hpp

@@ -20,6 +20,7 @@
 
 #include <cuda_runtime.h>
 
+#include <atomic>
 #include <cstdint>
 #include <string>
 #include <utility>
@@ -72,15 +73,32 @@ public:
     /// swapped-in expert becomes resident at `apply_pending`.  Call only between verify windows.
     bool adapt(const float* usage, const int32_t* res0, int max_swaps, std::string& err);
     void apply_pending(bool wait);
+    /// STRATA_ADAPT_OVERLAP=1 with a peer: adapt() then issues its copies in layer order and raises `*landed` to each
+    /// layer as that layer's copies land (host functions on the refill stream), marking `(*has)[layer]` and `*last`
+    /// first; the verify window holds such a layer's plan back until then (AdaptGate).  Call once after open().
+    /// `copy_signal`: the landing is told by a stream-ordered D2H copy of the layer number into a pinned int instead
+    /// (no callback thread to schedule beside the pool's spinning workers); that int is returned, null when host
+    /// functions are used.
+    volatile int32_t* set_overlap_gate(std::atomic<int32_t>* landed, std::vector<uint8_t>* has, int32_t* last,
+                                       bool copy_signal);
+    /// The pending swaps enter the residency table now, their copies still landing (the gate protects every read);
+    /// apply_pending(true) after the window still waits for them and clears the list.
+    void flip_pending();
+    bool has_pending() const { return !pending_.empty(); }
 
     int64_t resident() const { return resident_; }
     /// the last launch writes its rows into the host rows itself (the caller must not touch them)
     bool launched_direct() const { return launched_direct_; }
+    /// whether a launch given host rows writes them itself (STRATA_PEER_DIRECT, default on): then the caller need not
+    /// zero a peer row at all, whether the launch has happened yet or not
+    static bool direct_mode();
     double gib() const { return cache_.gib(); }
     int64_t entries() const { return entries_; }      ///< routed entries the peer computed
     int64_t experts() const { return experts_; }      ///< distinct (layer, expert) the peer computed
     int64_t swaps() const { return swaps_; }
     double ms_wait = 0;                               ///< host time spent in finish() waiting for the peer
+    double ms_launch = 0;                             ///< host time spent in launch() (grouping, copies, kernel launches)
+    int64_t launches = 0;                             ///< layer launches that gave the peer at least one row
 
 private:
     int device_ = -1;
@@ -90,6 +108,13 @@ private:
     ExpertSource* src_ = nullptr;
     std::vector<int32_t> res_;                        ///< [n_layers * n_expert] -> peer slot or -1
     std::vector<std::pair<int32_t, int32_t>> pending_;   ///< (residency index, slot) once the copies land
+    struct Tick { std::atomic<int32_t>* counter; int32_t layer; };
+    std::atomic<int32_t>* gate_landed_ = nullptr;        ///< set_overlap_gate: the counter, per-layer marks and the last layer
+    std::vector<uint8_t>* gate_has_ = nullptr;
+    int32_t* gate_last_ = nullptr;
+    std::vector<Tick> ticks_;                            ///< one per layer (stable storage for the host functions)
+    int32_t* d_gate_vals_ = nullptr;                     ///< device [n_layers]: the layer numbers, the source of the signal copies
+    volatile int32_t* h_gate_ = nullptr;                 ///< pinned int those copies write (null: host functions)
     int64_t resident_ = 0, entries_ = 0, experts_ = 0, swaps_ = 0;
     cudaStream_t stream_ = nullptr, refill_ = nullptr;
     cudaEvent_t refill_ev_ = nullptr;

include/strata/core/run_manifest.hpp

@@ -0,0 +1,195 @@
+// include/strata/core/run_manifest.hpp - P0.1 (lite): a machine-readable record of what a run selected.
+//
+// A diagnostic for testers and for the performance work: which fast paths a run took and why the others were declined,
+// the tensor formats the engine ended up holding (after any conversion at load), the engine choice and the memory
+// budgets.  OFF by default: `STRATA_MANIFEST=<file>` turns it on and names the file; with it unset every call below
+// returns at once (one cached getenv), prints nothing and costs nothing on the normal path.  The file is a JSON document
+// with sorted keys and no timestamps, process ids or addresses, so two matched runs give files that differ only in the
+// values that really differ (a measured free-memory figure, a path chosen at run time).  It is written (replace by
+// rename) when the engine is ready and again after a request if something new was recorded.
+//
+//   { "manifest_version": 1, "stage": "...",
+//     "engine":  { key: string },                       engine choice, build, devices, modes
+//     "budgets": { key: integer },                      memory and capacity budgets (bytes or counts, the key says)
+//     "paths":   [ { "area", "name", "selected", "reason", "detail" } ],   one entry per decision, sorted by area, name
+//     "tensors": { scope: { format: { "count", "bytes" } } } }             what is held, by format
+//
+// A path entry says what was chosen (`selected` true: the fast path runs), or why it was declined (`selected` false and
+// a reason in plain words).  Recording the same (area, name) again replaces the entry.
+#pragma once
+
+#include <cstdint>
+#include <cstdio>
+#include <cstdlib>
+#include <map>
+#include <mutex>
+#include <string>
+#include <utility>
+#include <vector>
+
+namespace strata::core::manifest {
+
+struct TensorCount {
+    uint64_t count = 0, bytes = 0;
+};
+
+struct PathEntry {
+    bool selected = false;
+    std::string reason, detail;
+};
+
+class Manifest {
+public:
+    /// An explicit path turns it on (tests); the process-wide instance reads STRATA_MANIFEST once.
+    explicit Manifest(std::string file = std::string()) : file_(std::move(file)) {}
+
+    bool enabled() const { return !file_.empty(); }
+    const std::string& file() const { return file_; }
+
+    void engine(const std::string& key, const std::string& value) {
+        if (!enabled()) return;
+        std::lock_guard<std::mutex> lk(mu_);
+        const auto it = engine_.find(key);
+        mark(it == engine_.end() || it->second != value);
+        engine_[key] = value;
+    }
+    void budget(const std::string& key, int64_t value) {
+        if (!enabled()) return;
+        std::lock_guard<std::mutex> lk(mu_);
+        auto it = budgets_.find(key);
+        mark(it == budgets_.end() || it->second != value);
+        budgets_[key] = value;
+    }
+    void path(const std::string& area, const std::string& name, bool selected, const std::string& reason,
+              const std::string& detail = std::string()) {
+        if (!enabled()) return;
+        std::lock_guard<std::mutex> lk(mu_);
+        const auto ins = paths_.try_emplace({area, name});
+        PathEntry& e = ins.first->second;
+        mark(ins.second || e.reason != reason || e.detail != detail || e.selected != selected);
+        e.selected = selected;
+        e.reason = reason;
+        e.detail = detail;
+    }
+    /// Adds to the count and bytes of `format` in `scope` ("dense", "head", "experts", "ple").
+    void tensors(const std::string& scope, const std::string& format, uint64_t count, uint64_t bytes) {
+        if (!enabled()) return;
+        std::lock_guard<std::mutex> lk(mu_);
+        TensorCount& t = tensors_[scope][format];
+        t.count += count;
+        t.bytes += bytes;
+        mark(true);
+    }
+
+    /// The document as text (the same text write() stores).
+    std::string json(const std::string& stage) const {
+        std::lock_guard<std::mutex> lk(mu_);
+        std::string o = "{\n  \"manifest_version\": 1,\n  \"stage\": " + quote(stage) + ",\n  \"engine\": {";
+        bool first = true;
+        for (const auto& [k, v] : engine_) { o += (first ? "\n    " : ",\n    ") + quote(k) + ": " + quote(v); first = false; }
+        o += first ? "},\n  \"budgets\": {" : "\n  },\n  \"budgets\": {";
+        first = true;
+        for (const auto& [k, v] : budgets_) { o += (first ? "\n    " : ",\n    ") + quote(k) + ": " + std::to_string(v); first = false; }
+        o += first ? "},\n  \"paths\": [" : "\n  },\n  \"paths\": [";
+        first = true;
+        for (const auto& [k, e] : paths_) {
+            o += (first ? "\n    " : ",\n    ") + std::string("{\"area\": ") + quote(k.first) + ", \"name\": " + quote(k.second) +
+                 ", \"selected\": " + (e.selected ? "true" : "false") + ", \"reason\": " + quote(e.reason) +
+                 ", \"detail\": " + quote(e.detail) + "}";
+            first = false;
+        }
+        o += first ? "],\n  \"tensors\": {" : "\n  ],\n  \"tensors\": {";
+        first = true;
+        for (const auto& [scope, fm] : tensors_) {
+            o += (first ? "\n    " : ",\n    ") + quote(scope) + ": {";
+            bool f2 = true;
+            for (const auto& [fmt, t] : fm) {
+                o += (f2 ? "" : ", ") + quote(fmt) + ": {\"count\": " + std::to_string(t.count) + ", \"bytes\": " + std::to_string(t.bytes) + "}";
+                f2 = false;
+            }
+            o += "}";
+            first = false;
+        }
+        o += first ? "}\n}\n" : "\n  }\n}\n";
+        return o;
+    }
+
+    /// Writes the document next to the target and renames it over (a reader never sees half a file).  False: not enabled,
+    /// or the file could not be written (said once on stderr; the run goes on).
+    bool write(const std::string& stage) {
+        if (!enabled()) return false;
+        const std::string text = json(stage), tmp = file_ + ".tmp";
+        std::FILE* f = std::fopen(tmp.c_str(), "wb");
+        bool ok = f != nullptr && std::fwrite(text.data(), 1, text.size(), f) == text.size();
+        if (f != nullptr) ok = std::fclose(f) == 0 && ok;
+        ok = ok && std::rename(tmp.c_str(), file_.c_str()) == 0;
+        if (!ok) {
+            std::remove(tmp.c_str());
+#if defined(_WIN32)
+            // rename does not replace on Windows: remove the old one and try again
+            std::remove(file_.c_str());
+            if (std::FILE* g = std::fopen(file_.c_str(), "wb")) {
+                ok = std::fwrite(text.data(), 1, text.size(), g) == text.size();
+                std::fclose(g);
+            }
+#endif
+        }
+        if (!ok && !warned_) {
+            warned_ = true;
+            std::fprintf(stderr, "strata: STRATA_MANIFEST: cannot write %s\n", file_.c_str());
+        }
+        std::lock_guard<std::mutex> lk(mu_);
+        if (ok) dirty_ = false;
+        return ok;
+    }
+    /// Writes again only when something was recorded since the last write.
+    bool flush_if_dirty(const std::string& stage) {
+        if (!enabled()) return false;
+        {
+            std::lock_guard<std::mutex> lk(mu_);
+            if (!dirty_) return false;
+        }
+        return write(stage);
+    }
+
+    static std::string quote(const std::string& s) {
+        std::string o = "\"";
+        for (unsigned char c : s) {
+            switch (c) {
+                case '"': o += "\\\""; break;
+                case '\\': o += "\\\\"; break;
+                case '\n': o += "\\n"; break;
+                case '\r': o += "\\r"; break;
+                case '\t': o += "\\t"; break;
+                default:
+                    if (c < 0x20) { char b[8]; std::snprintf(b, sizeof b, "\\u%04x", c); o += b; }
+                    else o += (char) c;
+            }
+        }
+        return o + "\"";
+    }
+
+private:
+    void mark(bool changed) { if (changed) dirty_ = true; }   // callers hold mu_
+
+    std::string file_;
+    mutable std::mutex mu_;
+    std::map<std::string, std::string> engine_;
+    std::map<std::string, int64_t> budgets_;
+    std::map<std::pair<std::string, std::string>, PathEntry> paths_;
+    std::map<std::string, std::map<std::string, TensorCount>> tensors_;
+    bool dirty_ = false;
+    bool warned_ = false;
+};
+
+/// The process-wide manifest: on when STRATA_MANIFEST names a file.
+inline Manifest& instance() {
+    static Manifest m([] {
+        const char* v = std::getenv("STRATA_MANIFEST");
+        return std::string(v != nullptr ? v : "");
+    }());
+    return m;
+}
+inline bool enabled() { return instance().enabled(); }
+
+}  // namespace strata::core::manifest

include/strata/core/verify.hpp

@@ -24,6 +24,7 @@
 #include <cstdio>
 
 #include "strata/core/expert_source.hpp"
+#include "strata/core/flag_order.hpp"
 #include "strata/core/layer.hpp"
 #include "strata/core/session.hpp"
 #include "strata/kernels/sampler.hpp"
@@ -47,10 +48,11 @@ namespace strata::core {
 /// bytes arrive layer by layer: `run` holds a layer's plan (the first thing that lets the GPU read the layer's expert
 /// slots) back until `landed >= layer` for a layer with a swap (`has[layer]`).  Set `active` only around one window.
 struct AdaptGate {
-    struct Tick { AdaptGate* gate; int32_t layer; };
+    struct Tick { std::atomic<int32_t>* counter; int32_t layer; };
     std::atomic<int32_t> landed{-1};      ///< the highest layer whose swap copies have landed (n_layers: all of them)
     std::atomic<bool> active{false};
     std::vector<uint8_t> has;             ///< per layer: a swap copies into this layer's slots
+    int32_t last = -1;                    ///< the highest layer with a swap this round (-1: none)
     std::vector<Tick> ticks;              ///< host-function contexts, one per layer and the final one (stable storage)
     const int32_t* dvals = nullptr;       ///< device [0..n_layers]: the source of those copies
     volatile int32_t* hland = nullptr;   ///< pinned host int the copy stream writes (a D2H copy of the layer number), instead of a host function; null: host functions
@@ -60,6 +62,19 @@ struct AdaptGate {
         const int32_t b = *hland;
         return a > b ? a : b;
     }
+    /// The peer tier's copies (--peer-device with STRATA_ADAPT_OVERLAP=1): its own counter, raised on its refill stream
+    /// in layer order the same two ways (`peer_hland`: the pinned int its D2H copies write; null: host functions); a
+    /// layer with a peer swap waits for it as well.
+    std::atomic<int32_t> peer_landed{-1};
+    std::vector<uint8_t> peer_has;
+    int32_t peer_last = -1;
+    volatile int32_t* peer_hland = nullptr;
+    int32_t peer_reached() const {
+        const int32_t a = peer_landed.load(std::memory_order_acquire);
+        if (peer_hland == nullptr) return a;
+        const int32_t b = *peer_hland;
+        return a > b ? a : b;
+    }
 };
 
 class NativeHead;
@@ -402,7 +417,8 @@ public:
     /// the CPU is RAM-bound, Q2_0).  Set before the first `run`.
     void set_pcie_mode(int mode) { sink_.pcie_mode = mode; }
     /// Called before every window this verifier launches (run, the batch and pipelined launches): the device's
-    /// residency table catches up with the host's (the adaptive tier's evictions while its copies are in flight).
+    /// residency table catches up with the host's (the adaptive tier's evictions while its copies are in flight,
+    /// STRATA_ADAPT_FETCH's admissions).
     /// It gets this verifier's stream; false (with the message in `err`) fails the launch.
     void set_pre_launch(std::function<bool(cudaStream_t cs, std::string& err)> f) { pre_launch_ = std::move(f); }
     /// the pool never plans a PCIe share (--pcie-frac 0): the window skips that path.  Before the first run.
@@ -701,12 +717,15 @@ private:
     cudaEvent_t commit_done_ = nullptr;   // recorded after an async commit (set_commit_async); see wait_commit
     bool commit_pending_ = false;
     cudaStream_t copy_ = nullptr;                                 // the copy engine's stream (DMA of missed experts)
-    struct FlagSet { uint32_t* flag; uint32_t value; };
+    using FlagSet = strata::core::FlagRaise;
     FlagSet flag_sets_[2 * 64 * 2] = {};                          // host-function arguments, one per (layer, group)
     static void fetch_dma(void* ctx, const uint8_t* const* src, int n, size_t bytes);
     static void raise_flag(uint32_t* flag, uint32_t value);
+    /// Flag B for `want`, behind every copy queued on copy_ (an empty plan must not overtake an earlier layer's DMA).
+    void raise_flag_b_ordered(uint32_t want);
     int32_t* h_plan_ = nullptr;  int32_t* m_plan_ = nullptr;     // counts | start | dst | tok | ptr (as int32 pairs)
     int64_t plan_i32_ = 0;                                        // int32 words in the plan block
+    int64_t dst2_off_ = 0;                                        // its PCIe groups' slot targets (GpuPlanSink::dst2)
     GpuPlanSink sink_;
     uint32_t cur_layer_ = 0;
     static void publish_plan(void* ctx);

include/strata/kernels/cpu/expert_layout.hpp

@@ -44,6 +44,16 @@ struct ExpertLayout {
 /// Plan v0.3 P6: whether this CPU (and its OS) runs the AVX-512 kernels (F, BW, VL, VNNI, VBMI).  Probed in a
 /// file compiled without AVX-512, so asking is safe everywhere; STRATA_FORCE_AVX2=1 answers no (for tests).
 bool cpu_avx512_ok();
+/// The same probes with VBMI required only (the multishift unpack and other byte-permute shuffles): a CPU may run
+/// every other feature the kernels use and miss it (Cascade Lake Xeon).
+bool cpu_vbmi_ok();
+/// AVX-512F + BW + VL + VNNI, VBMI NOT required: the dpbusd row dots, the q8 quantizer and the dq/bw arithmetic
+/// need no byte-permute shuffles (the q2 unpack's op has a substitute in expert.cpp).  A CPU this answers yes on
+/// used to run every expert through the AVX-2 fallback.
+bool cpu_vnni_ok();
+/// Scattered pieces: the copy above is a declaration only (defined in expert.cpp, the AVX-512 TU, because it reads
+/// its STRATA_CPU_YMM switch): whether the 512-bit q2 row dot may run at all (VNNI and not the YMM env).
+bool q2_rows_512_ok();
 /// Whether this CPU runs AVX-512F + AVX-512BW + AVX-512VL + AVX-512DQ (e.g. Skylake-X i9-9920X) for iq_avx512.cpp.
 bool cpu_avx512bw_ok();
 /// Whether this CPU (and its OS) runs the AVX2 kernels (AVX, AVX2, FMA, F16C): the floor of every expert kernel

include/strata/kernels/cpu/iq_avx512.hpp

@@ -30,4 +30,12 @@ void iq512x_rows(int ggml_type, const uint8_t* w, size_t row_bytes, int n, const
 void iq512_reference_gu_rows(int type, const uint8_t* blob, size_t gu_row, size_t up_off, int n,
                              const void* const* act, int nt, float* const* ff, int r0, int r1);
 
+// Gate and up in one pass over the activations, ggml-cpu's dot order for every token (see iq_avx512.cpp)
+bool iq512f_supported(int ggml_type) noexcept;
+void iq512_fused_gu_rows(int type, const uint8_t* blob, size_t gu_row, size_t up_off, int n, const void* const* act, int nt,
+                         float* const* ff, int r0, int r1);
+
+// Q5_0 down rows, 512-bit, ggml-cpu's dot bit for bit (see iq_avx512.cpp)
+void q50_512_down_rows(const uint8_t* w, size_t row_bytes, int n, const void* const* hq, int nt, float* const* out, int r0, int r1);
+
 }  // namespace strata::kernels::cpu

include/strata/kernels/qsa_select.hpp

@@ -94,4 +94,33 @@ void qsa_block_scores_long(const float* pooled, const float* dead, const float*
 /// path: pass null).
 int64_t qsa_topk_multi_scratch_ints(int64_t nq);
 
+/// STRATA_SELECT_TWO_PASS=1 (the prompt path, CUDA sm_80+; the bound and the rule: qsa_two_pass.hpp).  A scores row
+/// that qsa_block_topk turns into exactly the ids of qsa_block_scores_tc's row: every block is scored once in
+/// single-pass TF32, the blocks whose error bound reaches the query's top-k cut-off are scored again with
+/// qsa_block_scores_tc's own 3xTF32 arithmetic (bit for bit), and a query whose bound cannot be settled has its
+/// 16-query tile scored again in full.  The kept blocks hold their single-pass score, so the row is NOT
+/// qsa_block_scores_tc's: it is for qsa_block_topk only.  `scratch` (qsa_two_pass_scratch_bytes(nq_cap, max_blocks),
+/// 16-byte aligned) holds the key norms qsa_two_pass_keys wrote for this layer and chunk.  False (nothing launched):
+/// the switch is off, HIP, a card or STRATA_QSA_WARP where qsa_block_scores_tc would not run its tensor-core kernel,
+/// another geometry, nq > nq_cap, too little scratch.
+bool qsa_two_pass_on();             ///< STRATA_SELECT_TWO_PASS=1, read once (HIP: false; prints once where unused)
+bool qsa_two_pass_check_on();       ///< STRATA_SELECT_TWO_PASS_CHECK=1, read once (a diagnostic, see qsa_two_pass_check)
+int64_t qsa_two_pass_min_blocks();  ///< STRATA_SELECT_TWO_PASS_MIN (blocks), read once; default two_pass::kMinBlocksDefault
+uint64_t qsa_two_pass_scratch_bytes(int64_t nq_cap, int64_t max_blocks);
+/// Once per layer and chunk, after the indexer appends and before the calls: the norms of the pooled keys [0, n_blocks).
+bool qsa_two_pass_keys(const float* pooled, int64_t n_blocks, int64_t nq_cap, int64_t max_blocks, void* scratch,
+                       uint64_t scratch_bytes, void* stream);
+bool qsa_block_scores_two_pass(const float* pooled, const float* dead, const float* q_idx, const int32_t* steps,
+                               int64_t nq, int64_t nq_cap, int64_t max_blocks, const QsaShapes& s, float* scores,
+                               void* scratch, uint64_t scratch_bytes, void* stream, int64_t active_blocks);
+/// STRATA_SELECT_TWO_PASS_CHECK: right after a two-pass call and its qsa_block_topk, against today's row and ids for
+/// the same queries (ref_scores from qsa_block_scores_tc, ref_ids from qsa_block_topk on them): counts the queries
+/// whose ids differ, the scored-again blocks whose score is not today's bit for bit, and the kept blocks whose
+/// single-pass score lies outside the bound (two_pass::Stat).
+void qsa_two_pass_check(const float* scores, const int32_t* ids, const float* ref_scores, const int32_t* ref_ids,
+                        const int32_t* steps, int64_t nq, int64_t nq_cap, int64_t max_blocks, int64_t cap,
+                        void* scratch, void* stream);
+/// The counters (two_pass::kStatCount of them, this device), read on `stream` (synchronized); reset: zeroed after.
+bool qsa_two_pass_stats(unsigned long long* out, bool reset, void* stream);
+
 }  // namespace strata::kernels

include/strata/kernels/qsa_two_pass.hpp

@@ -0,0 +1,141 @@
+// include/strata/kernels/qsa_two_pass.hpp - STRATA_SELECT_TWO_PASS: the prompt path's QSA block scores in two
+// precisions with the same selection.  Header-only and host/device: the CUDA scorer (qsa_select.cu) and the GPU-free
+// test (tests/prefill/qsa_two_pass_test.cpp) read the same constants.
+//
+// Today (qsa_block_scores_tc, sm_80+) every block score is 3xTF32: per indexer head h and pooled key k, with
+// qh = tf32(q), ql = tf32(q - qh), kh = tf32(k), kl = tf32(k - kh) (cvt.rna), the chain of 16 m16n8k8 steps adds
+// ql*kh, qh*kl, qh*kh into one FP32 accumulator (48 MMAs), and the score is relu(head 0) + ... + relu(head 3).
+// The two-pass scorer computes S1 (qh*kh only, 16 MMAs per head) for every block, bounds |S1 - S3| per block, and
+// computes S3 again, bit for bit as today, only for the blocks whose bound reaches the top-k cut-off.  Every other
+// block keeps S1, which lies strictly on the same side of the cut-off as its S3, so qsa_block_topk returns exactly
+// today's cells in today's order (the threshold key, the count above it and the tied cells are unchanged).
+//
+// THE BOUND (per query q, block b; derived for the PR, conservative on purpose):
+//   S1_h - S3_h = -(qh.kl + ql.kh) + (accumulation error of S1) - (accumulation error of S3)
+//   |qh.kl + ql.kh| <= |qh| |kl| + |ql| |kh|                      (Cauchy-Schwarz, exact TF32 operands)
+//   accumulation: each MMA's FP32 result is taken to be within kUlps * 2^-23 of the absolute sum of its 9 addends
+//     (the accumulator and 8 exact TF32 products).  A truncating alignment with no guard bits gives at most 9 such
+//     ulps (8 truncated addends plus the final rounding), so 16 is a margin, not a fit.  Over a chain of n MMAs the
+//     error is at most n * kUlps * 2^-23 * A, A = the chain's absolute sum of products:
+//       A(S1) <= |qh| |kh|,   A(S3) <= |qh| |kh| + |qh| |kl| + |ql| |kh|
+//   relu is 1-Lipschitz; the head sums ((r0 + r1) + r2) + r3 add at most 3 * 2^-24 relative each (folded into kG3).
+//   Summed over the four heads, with a = sum_h |qh_h|, c = sum_h |ql_h| (per query) and kh = |kh_b|, kl = |kl_b|:
+//     eps_b = kl * P + kh * Q + kAbs,   P = (1 + kG3) a s,   Q = ((1 + kG3) c + (kG1 + kG3) a) s,   s = kSlack
+//   kAbs (2^-100) covers flushed subnormals (inputs, products, accumulators); kSlack the second-order terms.
+//   All norms are upper bounds (double, rounded up), eps is rounded up and the interval [S1 - eps, S1 + eps] is
+//   rounded outwards (CUDA __f*_rd / __f*_ru), so the interval is an enclosure of S3 under the stated MMA model.
+//
+// THE CUT-OFF RULE (per query): T = the width-th largest S3 cell key, the threshold qsa_block_topk computes.  With
+// T1 = the width-th largest S1 and E = the largest eps over the candidate blocks, T lies in [T1 - E, T1 + E]: the
+// width cells with S1 >= T1 have S3 >= T1 - E, and every block with S3 > T1 + E has S1 > T1.  A block with
+// S1 - eps > T1 + E is above T for certain, one with S1 + eps < T1 - E below it; every other block (including every
+// block whose S3 equals T, so the ties) is scored again with today's 3xTF32 arithmetic.
+#pragma once
+
+#include <cstdint>
+#include <cstring>
+
+#if defined(__CUDACC__) || defined(__HIPCC__)
+#define STRATA_TP_HD __host__ __device__
+#else
+#define STRATA_TP_HD
+#endif
+
+namespace strata::kernels::two_pass {
+
+constexpr int kSteps = 16;                    ///< K = 128 in m16n8k8 steps, one chain per head
+constexpr double kUlps = 16.0;                ///< accumulation allowance per MMA, FP32 ulps of its absolute sum
+constexpr double kG1 = kSteps * kUlps * 0x1p-23;               ///< the single-pass chain: 16 MMAs
+constexpr double kG3 = 3 * kSteps * kUlps * 0x1p-23 + 0x1p-21;  ///< the 3xTF32 chain: 48 MMAs, plus the head sums
+constexpr double kSlack = 1.0 + 0x1p-10;
+constexpr float kAbs = 0x1p-100f;
+constexpr float kMaxEps = 1e30f;              ///< larger bounds (or non-finite norms) mean: score this query again
+
+/// Candidate blocks per query held by the band kernel (shared memory); more, and the query is scored again in full.
+constexpr int kCandCap = 5120;
+/// Blocks re-scored per query and per column residue (b % 8): eight lists of this many; more is a full re-score.
+constexpr int kResCap = 128;
+constexpr int kSampleStride = 32;             ///< the candidate threshold comes from every 32nd block, as the
+                                              ///< prompt top-k's sample kernel does
+/// The calls (a batch of up to 256 queries) that take the two-pass path by default: the batch's last query has at
+/// least this many blocks (16,384 cells).  STRATA_SELECT_TWO_PASS_MIN=<blocks> moves it (0: every call).
+constexpr int64_t kMinBlocksDefault = 4096;
+
+/// The per-query coefficients of eps_b = kl_b * P + kh_b * Q (+ kAbs), from the head norm sums a and c.
+STRATA_TP_HD inline void coefficients(double a, double c, double& P, double& Q) {
+    P = (1.0 + kG3) * a * kSlack;
+    Q = ((1.0 + kG3) * c + (kG1 + kG3) * a) * kSlack;
+}
+
+/// qsa_select.cu's order_key: the float's order as an unsigned key (NaN lowest, -0 as +0).
+STRATA_TP_HD inline uint32_t order_key_host(float s) {
+    const float v = s + 0.0f;
+    if (!(v == v)) return 0u;
+    uint32_t b;
+    std::memcpy(&b, &v, 4);
+    return (b & 0x80000000u) ? ~b : (b | 0x80000000u);
+}
+/// The float of a key order_key produced (the inverse, for keys of non-NaN values).
+STRATA_TP_HD inline float key_float_host(uint32_t k) {
+    const uint32_t b = (k & 0x80000000u) ? (k & 0x7fffffffu) : ~k;
+    float f;
+    std::memcpy(&f, &b, 4);
+    return f;
+}
+
+/// cvt.rna.tf32.f32 on the host: round to nearest, ties away from zero, 10 explicit mantissa bits (low 13 bits 0).
+inline float tf32_rna_host(float x) {
+    uint32_t b;
+    std::memcpy(&b, &x, 4);
+    if ((b & 0x7f800000u) == 0x7f800000u) return x;   // Inf and NaN unchanged
+    b = (b + 0x1000u) & 0xffffe000u;                   // magnitude rounding; a carry moves into the exponent
+    float r;
+    std::memcpy(&r, &b, 4);
+    return r;
+}
+
+/// Scratch the two-pass scorer takes from the caller (bytes, 256-aligned parts): the key norms of every block
+/// (one layer, one chunk) and, per call of at most nq_cap queries, the coefficients, modes, residue counts, re-score
+/// lists and the 16-query tile flags.
+struct Layout {
+    uint64_t kn = 0, qc = 0, mode = 0, cnt = 0, ids = 0, tiles = 0, total = 0;
+};
+inline Layout layout(int64_t nq_cap, int64_t max_blocks) {
+    auto up = [](uint64_t v) { return (v + 255u) & ~uint64_t(255u); };
+    Layout l;
+    uint64_t o = 0;
+    l.kn = o;    o += up((uint64_t) max_blocks * 8u);
+    l.qc = o;    o += up((uint64_t) nq_cap * 8u);
+    l.mode = o;  o += up((uint64_t) nq_cap * 4u);
+    l.cnt = o;   o += up((uint64_t) nq_cap * 8u * 4u);
+    l.ids = o;   o += up((uint64_t) nq_cap * 8u * (uint64_t) kResCap * 4u);
+    l.tiles = o; o += up((uint64_t) ((nq_cap + 15) / 16) * 4u);
+    l.total = o;
+    return l;
+}
+
+/// The engine's counters (qsa_two_pass_stats): what the scorer did, and with STRATA_SELECT_TWO_PASS_CHECK=1 what
+/// the comparison against today's scorer found.
+enum Stat : int {
+    kStCalls = 0,       ///< two-pass calls
+    kStQueries,         ///< their queries with a sparse selection (n_kv > width)
+    kStIdentity,        ///< their queries whose selection is every cell (nothing to score)
+    kStFull,            ///< queries flagged for a full re-score (no bound, a list full, too few cells above F)
+    kStFullTiles,       ///< 16-query tiles scored again in full (3xTF32, today's kernel; every query of the tile)
+    kStBand,            ///< blocks scored again (3xTF32) next to the cut-off
+    kStBandMax,         ///< the most blocks scored again for one query
+    kStCand,            ///< candidate blocks (the band kernel's shared list)
+    kStCandMax,
+    kStBlocks,          ///< blocks scored once (single-pass) in two-pass calls
+    kChkQueries,        ///< check: queries compared (ids)
+    kChkIdQueries,      ///< check: queries whose ids differ from today's
+    kChkIdEntries,      ///< check: id entries that differ
+    kChkRescored,       ///< check: scored-again blocks compared bitwise with today's score
+    kChkRescoredDiff,   ///< check: ... that differ (must be 0)
+    kChkKept,           ///< check: blocks that kept S1 (compared against eps)
+    kChkKeptOut,        ///< check: ... with |S1 - S3| > eps (must be 0)
+    kChkRatioMax,       ///< check: the largest |S1 - S3| / eps seen, as float bits
+    kStatCount
+};
+
+}  // namespace strata::kernels::two_pass

include/strata/kernels/qsa_two_pass_status.hpp

@@ -0,0 +1,41 @@
+#pragma once
+// STRATA_SELECT_TWO_PASS: the request, read once, and the line a set switch prints where the two-pass scorer cannot
+// run.  The scorer runs only in the CUDA prompt path's chunked walk, on a card where qsa_block_scores_tc runs its
+// tensor-core kernel (sm_80+, STRATA_QSA_WARP not set).  Elsewhere (HIP, Metal, SYCL, an older card or the warp
+// scorer, the layer-major prompt path) the selection runs as without the switch; that was silent, so the switch read
+// as active.  Header-only, so the rule and the line are tested without a GPU (qsa_two_pass_status_test).
+#include <atomic>
+#include <cstdio>
+
+namespace strata::kernels::two_pass {
+
+// The switch's own parse: exactly "1" is on (as the two-pass scorer read it before).
+inline bool requested_from(const char* v) { return v != nullptr && v[0] == '1' && v[1] == '\0'; }
+
+class RequestStatus {
+public:
+    explicit RequestStatus(const char* flag) : requested_(requested_from(flag)) {}
+
+    bool requested() const { return requested_; }
+
+    // `not_here` null: this path can run the scorer, so the result is the request (whether a call takes it still
+    // depends on the call: its size, geometry and scratch).  Otherwise the path cannot run it: false, and the first
+    // such query prints one line naming the path.  A supported query never uses up that line.
+    bool on(const char* not_here, FILE* log = stderr) {
+        if (!requested_) return false;
+        if (not_here == nullptr) return true;
+        if (!warned_.exchange(true)) {
+            std::fprintf(log, "strata: STRATA_SELECT_TWO_PASS=1 is not used on %s: the two-pass scorer runs only in "
+                              "the CUDA prompt path's tensor-core selection (sm_80+); the selection is unchanged\n",
+                         not_here);
+            std::fflush(log);
+        }
+        return false;
+    }
+
+private:
+    const bool requested_;
+    std::atomic<bool> warned_{false};
+};
+
+}  // namespace strata::kernels::two_pass

include/strata/kernels/verify_kernels.hpp

@@ -146,11 +146,18 @@ void copy_indexed(float* dst, const float* src, int64_t stride, const int32_t* i
 /// into dst + k * blob_bytes with coalesced 16-byte loads - the PCIe share of a layer's missed experts, staged
 /// into VRAM before the grouped expert kernel reads them.  Launched for a capacity of `cap` blobs.
 void fetch_blobs(const unsigned long long* src, const int32_t* n, uint8_t* dst, int64_t blob_bytes, int cap, void* stream);
+/// The same with `dst2` (device memory, may be null): blob k goes to dst2[k] instead where that is not 0
+/// (STRATA_ADAPT_FETCH: a VRAM cache slot).  A separate overload, so the SYCL copy keeps the one above.
+void fetch_blobs(const unsigned long long* src, const int32_t* n, uint8_t* dst, int64_t blob_bytes, int cap, void* stream,
+                 const unsigned long long* dst2);
 /// table[pairs[2k]] = pairs[2k + 1] for k < n, `pairs` in mapped host memory: a residency table's changed entries,
 /// stream-ordered before the window that reads it, without the copy engine (where the tier's swaps may queue).
 void res_patch(int32_t* table, const int32_t* pairs, int n, void* stream);
 /// ptr[k] = base + k * blob_bytes for k < *n (the staged copies `fetch_blobs` made).
 void rebase_ptrs(unsigned long long* ptr, const int32_t* n, uint8_t* base, int64_t blob_bytes, void* stream);
+/// The same, or dst2[k] where `dst2` (may be null) is not 0 (fetch_blobs' second target).
+void rebase_ptrs(unsigned long long* ptr, const int32_t* n, uint8_t* base, int64_t blob_bytes, void* stream,
+                 const unsigned long long* dst2);
 
 // ---- the MTP draft layer (src/core/mtp.cpp)
 /// R[t][c][:] = h[t][c][:] + e[t][:]  (the embedding branch added to every stream).

include/strata/prefill/kernels.hpp

@@ -110,6 +110,9 @@ void gather_rows16(const uint16_t* x16, const int32_t* src, uint16_t* dst16, int
 /// bo[t, :] = shared[t, :] * sigmoid(sg[t]) + sum_k w[t, k] * D[slot[t, k], :]
 void moe_combine(const float* D, const int32_t* slot, const float* w, const float* shared, const float* sg, float* bo,
                  int64_t T, void* stream, const float* row_sd = nullptr);
+/// moe_combine with the expert rows as FP16 (2,560 halves a row, fused::dm_f16_used()); shared and bo 16-byte aligned.
+void moe_combine16(const uint16_t* Dm16, const int32_t* slot, const float* w, const float* shared, const float* sg, float* bo, int64_t T,
+                   void* stream);
 /// Layer-major prefill (STRATA_PREFILL_LAYER_MAJOR=1): acc[t, :] = shared[t, :] * sigmoid(sg[t]) - the shared expert's
 /// part of moe_combine, written before the routed experts are known.
 void moe_shared_init(const float* shared, const float* sg, float* acc, int64_t T, void* stream);

include/strata/prefill/mmq_ids_margin.hpp

@@ -0,0 +1,47 @@
+// include/strata/prefill/mmq_ids_margin.hpp - the room an MMQ row table (`ids`) needs past its last row (PR #5108, @quantforge-br).
+//
+// llama.cpp's mul_mat_q_stream_k_fixup loads ids_dst[col_low + jt*J + j] for a whole J-column tile before it compares the
+// tile with the expert's rows (the main kernel skips an empty tile first; the fixup only skips its writes).  Its tiles
+// run to ceil(max_rows / J) * J for every expert of the launch, and col_low is the expert's first row, which can be the
+// table's last.  A table of exactly `rows` entries is therefore read up to roundup(max_rows, J) entries past its end, and
+// the main kernel reads up to J - 1 past the last row of its last expert.  Strata's tables were exact-sized cudaMallocs:
+// one that ended a 2 MiB page faulted on the next (Xid 31, "prefill copy_i32: an illegal memory access" on a long prompt).
+//
+// J is bounded by the backend: mul_mat_q_switch_J tries J = 8, 16, ... kMmqTileColsMax and launches one of those.  The
+// MMQ planner in moe_mmq.cu iterates the same range (it uses kMmqTileColsMax), so the margin follows the tile bound in
+// one place.  An expert takes at most one row per token, so max_rows <= the chunk's tokens.  One extra tile on top of the
+// rounded read keeps the last row's neighbour in the allocation too.  Checked arithmetic: a request that does not fit in
+// 64 bits is refused (the callers fail the carve) instead of wrapping to a small table.
+#pragma once
+
+#include <cstdint>
+
+namespace strata::prefill::mmq {
+
+/// The widest MMQ column tile: ggml-cuda's mul_mat_q_switch_J tries J = 8 .. 128 in steps of 8.
+constexpr int kMmqTileColsMax = 128;
+constexpr int kMmqTileColsStep = 8;
+
+/// Entries to add after a row table whose launch has at most `max_rows` rows for one expert: roundup(max_rows, J) + J.
+/// Returns false (and leaves *out alone) when `max_rows` is negative or the sum overflows.
+inline bool mmq_ids_margin_rows(int64_t max_rows, int tile_cols_max, int64_t* out) {
+    if (max_rows < 0 || tile_cols_max <= 0) return false;
+    const int64_t J = tile_cols_max;
+    const int64_t tiles = max_rows / J + (max_rows % J != 0 ? 1 : 0);
+    if (tiles > INT64_MAX / J) return false;
+    const int64_t rounded = tiles * J;
+    if (rounded > INT64_MAX - J) return false;
+    *out = rounded + J;
+    return true;
+}
+
+/// A table of `rows` MMQ rows (one launch, at most `max_rows` per expert): the entries to allocate and write.  0 on overflow
+/// (callers treat it as "does not fit": an empty take fails the carve).
+inline uint64_t mmq_ids_table_rows(uint64_t rows, uint64_t max_rows, int tile_cols_max = kMmqTileColsMax) {
+    int64_t margin = 0;
+    if (max_rows > (uint64_t) INT64_MAX || !mmq_ids_margin_rows((int64_t) max_rows, tile_cols_max, &margin)) return 0;
+    if (rows > UINT64_MAX - (uint64_t) margin) return 0;
+    return rows + (uint64_t) margin;
+}
+
+}  // namespace strata::prefill::mmq

include/strata/prefill/moe_fused_iq.hpp

@@ -42,6 +42,10 @@ void quantize_act_native(const float* x, int64_t rows, int64_t cols, void* xa, v
 /// through Alloc's 256-byte takes from a 256-aligned cache slot or cudaMalloc).
 void experts_native(const Batch& b, const NativeGeom& g, int n_expert, int64_t n, const void* scratch, const void* xa,
                     const int32_t* src, void* ha, float* dm, void* stream);
+/// gfx11 F16 kernels: the next experts_native calls may write the down rows as FP16 (2,560 halves a row, saturated at +-65504) instead of FP32 when the
+/// down runs on an F16 kernel; dm_f16_used() says which form the last call wrote (false on every other path), for moe_combine16.
+void want_dm_f16(bool on);
+bool dm_f16_used();
 /// The NT = 256 kernels (STRATA_PF_FUSED_NT=256 or _F16=1) or the warp-specialized kernel (STRATA_PF_FUSED_WS=1) are
 /// set up on the current device: experts_native takes a WideBatch (STRATA_PF_BATCH up to kBatchRoom experts a launch).
 bool native_wide();

include/strata/prefill/moe_iq4_w4a8.hpp

@@ -0,0 +1,20 @@
+// include/strata/prefill/moe_iq4_w4a8.hpp - the routed experts' IQ4_XS / IQ4_NL prompt products on int8 tensor
+// cores (sm_80+), a pipelined replacement for llama.cpp's MMQ tiling-mode launch with the same bits.
+#pragma once
+
+#include "strata/prefill/moe_mmq.hpp"
+
+namespace strata::prefill::mmq {
+
+/// Runs p (IQ4_XS or IQ4_NL weights, q8_1 activations as Context::run takes them, experts at w + z * expert_bytes or
+/// tab->p[z]) and returns true; returns false without launching anything when it does not cover p (another type,
+/// rows not a multiple of 128, a row table, or a launch MMQ would split along K with its stream-k fixup - those sums
+/// are added in a different order), and the caller runs Context::run.  Every output it writes is MMQ's tiling-mode
+/// sum bit for bit: per 32-value block the exact int32 dot, then sum += float(dot) * d_w * d_y in block order.
+bool run_iq4_w4a8(const Product& p, void* stream);
+
+/// STRATA_IQ4_W4A8: 0 = off (the default); 1 = Context::run tries run_iq4_w4a8 first for every product; 2 = for the
+/// IQ4_NL ones only (the down products - where MMQ wastes the most: K 640 in 256-value iterations).
+int iq4_w4a8_mode();
+
+}  // namespace strata::prefill::mmq

include/strata/prefill/moe_mmq.hpp

@@ -8,6 +8,8 @@
 #include <cstddef>
 #include <cstdint>
 
+#include "strata/prefill/mmq_ids_margin.hpp"
+
 namespace strata::prefill::mmq {
 
 /// This build has the MMQ path (the ggml sources were available to the build).
@@ -158,6 +160,11 @@ constexpr size_t kSlotGuard = 256;
 /// expert's rows get the same bits in any such launch, whatever its group-mates and max_rows (STRATA_MMQ_GROUP_SORT).
 /// 160 or 320 tiles per column tile on 82 SMs: yes; 108 or 128 SMs: no (stream-k splits the few-tile launches).
 bool whole_tiles(int ggml_type, int64_t w_rows, int n_experts);
+/// Whether `Context::run` on the current device would split a product's K across blocks with llama.cpp's stream-k
+/// decomposition (its fixup adds the partial sums in another order), for `n_experts` experts of [w_rows, *] weights of
+/// `ggml_type` and at most `max_rows` rows an expert: the tile width mul_mat_q_switch_J picks, the config's stream_k
+/// flag, and launch_mul_mat_q's tile-efficiency test.  False for a type MMQ does not cover.  (STRATA_IQ4_W4A8.)
+bool launch_splits_k(int ggml_type, int64_t w_rows, int64_t max_rows, int n_experts);
 /// STRATA_MMQ_DIRECT, read once: 0 (default) today's gather; 1 the guard layout and resident experts read in
 /// place; 2 the guard layout with today's gather (the bitwise A/B reference).  0 whenever !tab_built().
 int direct_mode();

include/strata/prefill/timer_marks.hpp

@@ -0,0 +1,20 @@
+// include/strata/prefill/timer_marks.hpp - which STRATA_PREFILL_TIMING marks are recorded (PR #4006, @Vop3p).
+//
+// PfTimer charges the gap between two consecutive marks to the phase of the earlier one.  Two marks in a row in the same
+// phase therefore add nothing to the totals: the gap between them is charged to that phase either way.  Skipping the
+// second one leaves the totals as they were with a fraction of the event records.  That matters under WDDM, where every
+// cudaEventRecord is a submission of its own (~80 us on an RX 6900 XT, Windows 11): compute() marks "dequant" once per
+// expert, which put ~16 records per MMQ group on the stream and charged the launch gaps they made to "dequant".
+// Pure functions, so a host test can check the rule and the totals without a GPU.
+#pragma once
+
+#include <cstddef>
+
+namespace strata::prefill {
+
+/// A mark for `phase` is redundant when the newest recorded mark (ph[used - 1]) is already in that phase.
+inline bool timer_mark_repeats(const int* ph, size_t used, int phase) {
+    return used > 0 && ph[used - 1] == phase;
+}
+
+}  // namespace strata::prefill

include/strata/program/adaptive_range.hpp

@@ -0,0 +1,47 @@
+// include/strata/program/adaptive_range.hpp - which swaps of an adaptive-tier round have had their copy in (PR #4844, @uvlad23).
+//
+// A round of the adaptive tier moves a list of swaps (aswaps) into the GPU expert caches.  Step 2 copies them in, step 3
+// marks the ones that landed as resident.  On the serial loop the copies go in batches (the arena may cap one), and step 3
+// marks the batch that just landed, [lo, hi).  On the pipelined loop every swap is copied at its card's next gap and
+// step 3 marks the whole list.  Which loop runs is decided per tick: the pipelined loop can drain after step 2 has been
+// planned, and the next tick then takes step 3 on the serial rule.  lo/hi/bi then still named the last batch of an earlier
+// round, so step 3 marked only that range and every other swap of the round was lost (its slot held the new expert, the
+// table still said the old one).  The pipelined plan therefore sets the range to the whole round.  Plain arithmetic, so a
+// host test can walk the rounds without a GPU.
+#pragma once
+
+#include <algorithm>
+#include <cstddef>
+
+namespace strata::program {
+
+struct AdaptiveCopyRange {
+    size_t lo = 0, hi = 0;   ///< the batch being copied in / just landed
+    size_t bi = 0;           ///< where the next batch starts
+
+    void clear() { lo = hi = bi = 0; }
+
+    /// Serial loop, step 2: the next batch is [bi, bi + batch) of `total` swaps (all the rest when batch == 0, the arena's
+    /// cap otherwise).  Never past `total`, and an overflowing sum is the end.
+    void next_batch(size_t total, size_t batch) {
+        lo = std::min(bi, total);
+        hi = total;
+        if (batch > 0 && batch < total - lo) hi = lo + batch;
+        bi = hi;
+    }
+
+    /// Pipelined loop, step 2: the whole round is one batch, whatever an earlier round left here.
+    void whole_round(size_t total) {
+        lo = 0;
+        hi = bi = total;
+    }
+
+    /// Step 3 on the serial rule: more batches of the round still to copy in.
+    bool more(size_t total) const { return bi < total; }
+
+    /// Step 3: the swaps that landed.  `pipelined` is this tick's rule; the range is the whole round on that one.
+    size_t landed_begin(bool pipelined) const { return pipelined ? 0 : lo; }
+    size_t landed_end(bool pipelined, size_t total) const { return pipelined ? total : hi; }
+};
+
+}  // namespace strata::program

include/strata/spec/window_rule.hpp

@@ -7,6 +7,7 @@
 // every backend call it.
 #pragma once
 
+#include <cmath>
 #include <cstdlib>
 #include <algorithm>
 
@@ -69,4 +70,58 @@ inline int choose_window(const WindowRuleIn& in) {
     return T;
 }
 
+/// STRATA_DF2_GATE=1 (opt-in, measured on the Mac): DFlash2's window from a calibrated survival instead of a fixed confidence.  The selector's cumulative confidence C_j of
+/// draft j over-states the real chance that drafts 1..j all hold (M5 Max UD-Q4_K_XL: C = 0.84, survival 0.70 at j = 5), and the right confidence differs by request
+/// (a story wants a narrower window than JSON).  Per request this keeps, for every position j and confidence bucket, how often the first j drafts held, shrunk toward a
+/// prior of C^1.3, and extends the window to row j+1 while  survival * (ms per committed token) >= row_ms  (what one more verify row costs).
+struct AdaptiveGate {
+    static constexpr int NB = 6;
+    double n[8][NB] = {}, h[8][NB] = {};
+    double ms_per_tok = 0.0;       // EMA of the round's wall ms per committed token
+    double row_ms = 3.0;           // one more verify row (M5 Max: 2.65-3.3 ms)
+    double prior_n = 3.0;
+    static int bucket(double c) { return c < 0.15 ? 0 : c < 0.3 ? 1 : c < 0.5 ? 2 : c < 0.7 ? 3 : c < 0.85 ? 4 : 5; }
+    double survival(int j, double c) const {
+        const int b = bucket(c);
+        const double prior = std::pow(c, 1.3);
+        return (h[j][b] + prior_n * prior) / (n[j][b] + prior_n);
+    }
+    /// rows of the next window: 1 + the drafts kept (cap = widest window)
+    int choose(const float* dprob, int cap) const {
+        int T = 1;
+        double chain = 1.0;
+        const double v = ms_per_tok > 0.0 ? ms_per_tok : 9.0;
+        while (T < cap) {
+            chain *= (double) dprob[T - 1];
+            if (survival(T, chain) * v < row_ms) break;
+            ++T;
+        }
+        return T;
+    }
+    /// after a DFlash2 round of T rows, a drafts accepted: drafts 1..T-1 were offered, draft j survived when a >= j
+    void observe(const float* dprob, int T, int a) {
+        double chain = 1.0;
+        for (int j = 1; j < T && j < 8; ++j) {
+            chain *= (double) dprob[j - 1];
+            const int b = bucket(chain);
+            n[j][b] += 1.0;
+            h[j][b] += a >= j ? 1.0 : 0.0;
+        }
+    }
+    void observe_round(double round_ms, int committed) {
+        if (committed < 1) return;
+        const double m = round_ms / committed;
+        ms_per_tok = ms_per_tok > 0.0 ? 0.9 * ms_per_tok + 0.1 * m : m;
+    }
+};
+
+inline bool df2_gate_on() {
+    static const bool v = [] { const char* e = std::getenv("STRATA_DF2_GATE"); return e != nullptr && e[0] == '1'; }();
+    return v;
+}
+inline double df2_gate_row_ms() {
+    static const double v = [] { const char* e = std::getenv("STRATA_DF2_ROW_MS"); return e != nullptr && e[0] != '\0' ? std::atof(e) : 3.0; }();
+    return v;
+}
+
 }  // namespace strata::spec

metal/CMakeLists.txt

@@ -114,7 +114,7 @@ file(GLOB STRATA_METAL_ENGINE_SOURCES CONFIGURE_DEPENDS ${CMAKE_CURRENT_SOURCE_D
 add_library(strata_metal_host STATIC ${STRATA_METAL_HOST_SOURCES})
 target_include_directories(strata_metal_host PUBLIC ${STRATA_ROOT}/include ${S}/core ${STRATA_ROOT}/third_party/ggml
                            ${STRATA_GGML_DIR}/ggml/include ${STRATA_GGML_DIR}/ggml/src ${STRATA_GGML_DIR}/ggml/src/ggml-cpu)
-target_compile_definitions(strata_metal_host PUBLIC STRATA_NATIVE_EXPERTS=1 STRATA_HAVE_DFLASH2=1 STRATA_VERSION="0.1.57-metal")
+target_compile_definitions(strata_metal_host PUBLIC STRATA_NATIVE_EXPERTS=1 STRATA_HAVE_DFLASH2=1 STRATA_VERSION="0.1.58-metal")
 target_compile_options(strata_metal_host PRIVATE -Wno-unused-parameter -Wno-deprecated-declarations)
 target_link_libraries(strata_metal_host PUBLIC strata_gpu_metal ggml-cpu ggml-base)
 
@@ -181,7 +181,7 @@ if(NOT STRATA_BUILD_ID)
   set(STRATA_BUILD_ID unknown)
 endif()
 set_source_files_properties(${DS4}/ds4_main.cpp PROPERTIES COMPILE_DEFINITIONS "DS4_BUILD_ID=\"${STRATA_BUILD_ID}\"")
-# -DSTRATA_DS4_VERSION=0.1.57-metal: the version this DeepSeek engine reports, when its package is released apart from the Qwen engine's
+# -DSTRATA_DS4_VERSION=0.1.58-metal: the version this DeepSeek engine reports, when its package is released apart from the Qwen engine's
 if(STRATA_DS4_VERSION)
   set_source_files_properties(${DS4}/ds4_main.cpp PROPERTIES COMPILE_OPTIONS "-USTRATA_VERSION;-DSTRATA_VERSION=\"${STRATA_DS4_VERSION}\"")
 endif()

metal/kernels/gr.metal

@@ -113,8 +113,41 @@ static inline void grd_acc(device const uchar* w, int nin, int ks, int KS, ushor
     }
 }
 
+// the same sums with the loads of two units in flight (units u and u + 32 decoded before either is combined): the lane order and the
+// per-token addition order are grd_acc's (A then B), so every bit is the same; only the memory latencies overlap.  A lane has 1-2 units here.
+template <int F>
+static inline void grd_acc_p(device const uchar* w, int nin, int ks, int KS, ushort lane, device const float* xn,
+                             thread float* acc, threadgroup const uint* notab) {
+    constexpr int UNIT = fmt<F>::UNIT;
+    const int U = nin / UNIT, u0 = ks * U / KS, u1 = (ks + 1) * U / KS;
+    int u = u0 + lane;
+    for (; u + 32 < u1; u += 64) {
+        float qa[UNIT], qb[UNIT];
+        float4 sa, sb;
+        Dec<F>::run(w, u, qa, sa, notab);
+        Dec<F>::run(w, u + 32, qb, sb, notab);
+        for (int t = 0; t < GR_NTOK; ++t) {
+            float xa[UNIT], xb[UNIT], xsum[2] = {0.0f, 0.0f};
+            for (int k = 0; k < UNIT; k += 16) load16(xn + t * nin + UNIT * u + k, xa + k);
+            for (int k = 0; k < UNIT; k += 16) load16(xn + t * nin + UNIT * (u + 32) + k, xb + k);
+            acc[t] += unit_combine<UNIT>(qa, sa, xa, xsum);
+            acc[t] += unit_combine<UNIT>(qb, sb, xb, xsum);
+        }
+    }
+    for (; u < u1; u += 32) {
+        float q[UNIT];
+        float4 sc;
+        Dec<F>::run(w, u, q, sc, notab);
+        for (int t = 0; t < GR_NTOK; ++t) {
+            float xs[UNIT], xsum[2] = {0.0f, 0.0f};
+            for (int k = 0; k < UNIT; k += 16) load16(xn + t * nin + UNIT * u + k, xs + k);
+            acc[t] += unit_combine<UNIT>(q, sc, xs, xsum);
+        }
+    }
+}
+
 // FI: the inject rows' format (the Unsloth files: Q8_0 down, F32 inject); FI = F is the original kernel
-template <int SG, int KS, int F, int FI>
+template <int SG, int KS, int F, int FI, int PF = 0>
 kernel void gr_down_f(device const uchar* w_down [[buffer(0)]], device const uchar* w_inject [[buffer(1)]],
                     device const float* xn [[buffer(2)]], device float* lo [[buffer(3)]],
                     device float* inject [[buffer(4)]], constant GrArgs& a [[buffer(5)]],
@@ -131,8 +164,13 @@ kernel void gr_down_f(device const uchar* w_down [[buffer(0)]], device const uch
     const uint rb = uint(nin / fmt<F>::QK * fmt<F>::BB), rbi = uint(nin / fmt<FI>::QK * fmt<FI>::BB);
     float acc[8];
     for (int t = 0; t < 8; ++t) acc[t] = 0.0f;
-    if (rr < int(a.hc_lr)) grd_acc<F>(w_down + ulong(rr) * rb, nin, ks, KS, lane, xn, acc, notab);
-    else grd_acc<FI>(w_inject + ulong(rr - int(a.hc_lr)) * rbi, nin, ks, KS, lane, xn, acc, notab);
+    if (PF) {
+        if (rr < int(a.hc_lr)) grd_acc_p<F>(w_down + ulong(rr) * rb, nin, ks, KS, lane, xn, acc, notab);
+        else grd_acc_p<FI>(w_inject + ulong(rr - int(a.hc_lr)) * rbi, nin, ks, KS, lane, xn, acc, notab);
+    } else {
+        if (rr < int(a.hc_lr)) grd_acc<F>(w_down + ulong(rr) * rb, nin, ks, KS, lane, xn, acc, notab);
+        else grd_acc<FI>(w_inject + ulong(rr - int(a.hc_lr)) * rbi, nin, ks, KS, lane, xn, acc, notab);
+    }
     for (int t = 0; t < GR_NTOK; ++t) {
         const float v = simd_sum(acc[t]);
         if (lane == 0) red[(ks * RS + rs) * 8 + t] = v;
@@ -154,7 +192,7 @@ kernel void gr_down_f(device const uchar* w_down [[buffer(0)]], device const uch
 // A threadgroup owns 16 consecutive d for all hc = 4 streams: 8 simdgroups, simdgroup s -> stream s / 2 and
 // d0 + 8 (s % 2) .. + 8; LPR = 4 lanes per row (320 = 20 units of 16, 5 per lane).
 // XP: [4 kinds][n_tok][n] then S16 [n_tok][n / 16] (units.metal's xprep layout).
-template <int F>
+template <int F, int PF = 0>
 kernel void gr_up_f(device const uchar* w_up [[buffer(0)]], device const float* lo [[buffer(1)]],
                   device const float* xn [[buffer(2)]], device float* xp [[buffer(3)]],
                   constant GrArgs& a [[buffer(4)]],
@@ -170,6 +208,21 @@ kernel void gr_up_f(device const uchar* w_up [[buffer(0)]], device const float*
     const int U = int(a.hc_lr) / UNIT;
     float acc[8];
     for (int t = 0; t < 8; ++t) acc[t] = 0.0f;
+    if (PF) {
+        // up to three units (U = 10 over 4 lanes) decoded before the first is combined; the per-token order is u = li, li + 4, li + 8
+        constexpr int NU = 3;
+        float q[NU][UNIT];
+        float4 sc[NU];
+        for (int j = 0; j < NU; ++j)
+            if (li + 4 * j < U) Dec<F>::run(w, li + 4 * j, q[j], sc[j], notab);
+        for (int t = 0; t < GR_NTOK; ++t)
+            for (int j = 0; j < NU; ++j)
+                if (li + 4 * j < U) {
+                    float xs[UNIT], xsum[2] = {0.0f, 0.0f};
+                    for (int k = 0; k < UNIT; k += 16) load16(lo + t * int(a.hc_lr) + UNIT * (li + 4 * j) + k, xs + k);
+                    acc[t] += unit_combine<UNIT>(q[j], sc[j], xs, xsum);
+                }
+    } else
     for (int u = li; u < U; u += 4) {
         float q[UNIT];
         float4 sc;
@@ -220,3 +273,16 @@ kernel void gr_up_f(device const uchar* w_up [[buffer(0)]], device const float*
 GRD(8, 8) GRD(8, 4) GRD(4, 4) GRD(4, 2) GRD(16, 16) GRD(16, 8)
 template [[host_name("gr_up")]] kernel decltype(gr_up_f<T_BF16>) gr_up_f<T_BF16>;
 template [[host_name("gr_up_q8")]] kernel decltype(gr_up_f<T_Q8_0>) gr_up_f<T_Q8_0>;
+// the prefetching variants (the engine's default; STRATA_METAL_HCLD=0 keeps the originals above)
+template [[host_name("gr_downp_q8_8_8")]] kernel decltype(gr_down_f<8, 8, T_Q8_0, T_Q8_0, 1>) gr_down_f<8, 8, T_Q8_0, T_Q8_0, 1>;
+template [[host_name("gr_downp_q8f_8_8")]] kernel decltype(gr_down_f<8, 8, T_Q8_0, T_F32, 1>) gr_down_f<8, 8, T_Q8_0, T_F32, 1>;
+template [[host_name("gr_upp_q8")]] kernel decltype(gr_up_f<T_Q8_0, 1>) gr_up_f<T_Q8_0, 1>;
+
+// wpf: a weight prefetch. Reads n16 uint4 of W (grid-strided) and writes nothing unless the xor of everything is one value in 2^32; the point is only to
+// pull the next kernel's first weights into the GPU's caches while the small hyper-connection kernels run (STRATA_METAL_WPF_MIB).
+kernel void wpf(device const uint4* W [[buffer(0)]], device uint* flag [[buffer(1)]], constant uint& n16 [[buffer(2)]],
+                uint gid [[thread_position_in_grid]], uint gsz [[threads_per_grid]]) {
+    uint4 a = 0;
+    for (uint i = gid; i < n16; i += gsz) a ^= W[i];
+    if ((a.x ^ a.y ^ a.z ^ a.w) == 0x7fffffffu) flag[0] = 1;
+}

metal/kernels/mm_i8.metal

@@ -0,0 +1,25 @@
+// metal/kernels/mm_i8.metal - matmul2d throughput probes for other operand types (tests/mm_i8_bench.cpp):
+// C[M][N] = A[M][K] . B[N][K]^T with a typed destination: int8 x int8 -> int32, fp16 x fp16 -> fp16 (and the fp32 reference).
+#include <MetalPerformancePrimitives/MetalPerformancePrimitives.h>
+using namespace mpp::tensor_ops;
+
+struct MmArgs { int M, N, K; };
+
+template <typename T, typename D, int TM, int TN, int SG>
+kernel void mpp_mm_t(device T* A [[buffer(0)]], device T* B [[buffer(1)]], device D* C [[buffer(2)]],
+                     constant MmArgs& a [[buffer(3)]], uint3 tg [[threadgroup_position_in_grid]]) {
+    tensor<device T, dextents<int32_t, 2>, tensor_inline> tA(A, dextents<int32_t, 2>(a.K, a.M));
+    tensor<device T, dextents<int32_t, 2>, tensor_inline> tB(B, dextents<int32_t, 2>(a.K, a.N));
+    tensor<device D, dextents<int32_t, 2>, tensor_inline> tC(C, dextents<int32_t, 2>(a.N, a.M));
+    constexpr auto desc = matmul2d_descriptor(TM, TN, dynamic_length_v<int>, false, true, false);
+    matmul2d<desc, execution_simdgroups<SG>> op;
+    auto mA = tA.slice(0, int(tg.y) * TM);
+    auto mB = tB.slice(0, int(tg.x) * TN);
+    auto mC = tC.slice(int(tg.x) * TN, int(tg.y) * TM);
+    op.run(mA, mB, mC);
+}
+
+#define MPPT(T, D, M, N, S) template [[host_name("mpp_" #T "_" #D "_" #M "_" #N "_" #S)]] kernel decltype(mpp_mm_t<T, D, M, N, S>) mpp_mm_t<T, D, M, N, S>;
+MPPT(char, int, 64, 64, 4) MPPT(char, int, 128, 64, 4) MPPT(char, int, 64, 128, 4)
+MPPT(half, half, 64, 64, 4) MPPT(half, half, 128, 64, 4)
+MPPT(half, float, 64, 64, 4) MPPT(half, float, 128, 64, 4)

metal/kernels/moe_mm.metal

@@ -1188,3 +1188,217 @@ kernel void mm_gu5(device const uchar* W [[buffer(0)]], device const half* XS [[
 #define MMGU5(F, N, S) MMGU5_(F, N, S)
 MMGU5(T_IQ3_S, 32, 2) MMGU5(T_IQ3_S, 32, 1) MMGU5(T_IQ3_S, 16, 4) MMGU5(T_IQ3_XXS, 32, 2) MMGU5(T_IQ3_XXS, 32, 1)
 MMGU5(T_IQ3_XXS, 16, 4) MMGU5(T_IQ2_S, 32, 2) MMGU5(T_IQ2_S, 32, 1) MMGU5(T_IQ2_S, 16, 4)
+
+// ---- v6: ragged tiles.  A tile of the plan is a GROUP of up to MS consecutive 16-row subtiles of ONE expert (the expert's
+// run is still padded only to 16 rows): the weights of a TN x TK block are decoded ONCE per k step and multiplied with every
+// subtile of the group, so the decode (which bounds mm_gu3 / mm_down3: 59% of the Q4_K gate/up time with the decode
+// removed, MM_PROBE 1) is shared by up to 16 * MS token rows without the padding of a taller plan tile.  Each subtile is an
+// independent 16-row matmul2d accumulation over the same k sequence, so every output is bitwise mm_gu3's / mm_down3's.
+//   mm_scan6   mm_scan's plan with groups: tiles[i] = (expert | nsub << 16, first padded row); eoff / rows count whole
+//              16-row subtiles; meta as mm_scan's (meta[3] / meta[6] = the number of groups)
+kernel void mm_scan6(device const uint* bcnt [[buffer(0)]], device uint* boff [[buffer(1)]],
+                     device uint* eoff [[buffer(2)]], device int2* tiles [[buffer(3)]], device uint* meta [[buffer(4)]],
+                     constant MmPlanArgs& a [[buffer(5)]],
+                     ushort tid [[thread_index_in_threadgroup]], ushort sg [[simdgroup_index_in_threadgroup]],
+                     ushort lane [[thread_index_in_simdgroup]]) {
+    threadgroup uint s_rows[32], s_tiles[32];
+    const uint ms = a.tm;   // here tm = the group size in 16-row subtiles
+    if (tid < 32) { s_rows[tid] = 0; s_tiles[tid] = 0; }
+    threadgroup_barrier(mem_flags::mem_threadgroup);
+    const uint e = tid;
+    uint tot = 0;
+    if (e < a.n_expert)
+        for (uint b = 0; b < a.n_blocks; ++b) {
+            boff[b * a.n_expert + e] = tot;
+            tot += bcnt[b * a.n_expert + e];
+        }
+    const uint nsub = (tot + 15) / 16, rows = nsub * 16, nt = (nsub + ms - 1) / ms;
+    const uint pr = simd_prefix_exclusive_sum(rows), pt = simd_prefix_exclusive_sum(nt);
+    if (lane == 31) { s_rows[sg] = pr + rows; s_tiles[sg] = pt + nt; }
+    threadgroup_barrier(mem_flags::mem_threadgroup);
+    uint orow = 0, otile = 0, all_rows = 0, all_tiles = 0;
+    for (int i = 0; i < 32; ++i) {
+        if (i < int(sg)) { orow += s_rows[i]; otile += s_tiles[i]; }
+        all_rows += s_rows[i];
+        all_tiles += s_tiles[i];
+    }
+    if (e < a.n_expert) {
+        const uint r0 = orow + pr, t0 = otile + pt;
+        eoff[e] = r0;
+        for (uint k = 0; k < nt; ++k) {
+            const uint n = min(ms, nsub - k * ms);
+            tiles[t0 + k] = int2(int(e | (n << 16)), int(r0 + k * ms * 16));
+        }
+    }
+    if (tid == 0) {
+        meta[0] = all_tiles;
+        meta[1] = all_rows;
+        meta[3] = all_tiles;
+        meta[4] = 1;
+        meta[6] = all_tiles;
+        meta[7] = 1;
+        meta[8] = all_rows;
+        meta[9] = 1;
+        meta[10] = 1;
+    }
+}
+
+template <int F, int MS, int TN, int TK, int SG>
+kernel void mm_gu6(device const uchar* W [[buffer(0)]], device const half* XS [[buffer(1)]],
+                   device half* HS [[buffer(2)]], device const int2* tiles [[buffer(3)]], constant MmArgs& a [[buffer(4)]],
+                   uint3 tg [[threadgroup_position_in_grid]], ushort sg [[simdgroup_index_in_threadgroup]],
+                   ushort lane [[thread_index_in_simdgroup]], ushort tid [[thread_index_in_threadgroup]]) {
+    constexpr int TM = 16;
+    threadgroup float ot[SG][TM * TN];
+    threadgroup uint tab[fmt<F>::TAB];
+    load_tab<F>(tab, tid, SG * 32);
+    threadgroup_barrier(mem_flags::mem_threadgroup);
+    const int2 tl = tiles[tg.y];
+    const int e = tl.x & 0xFFFF, nsub = tl.x >> 16, row0 = tl.y, n0 = (int(tg.x) * SG + sg) * TN;
+    device const uchar* E = W + ulong(e) * a.gu_stride;
+    device const uchar* g0 = E + a.gate_off + ulong(n0) * a.gu_row;
+    device const uchar* u0 = E + a.up_off + ulong(n0) * a.gu_row;
+    constexpr auto desc = matmul2d_descriptor(TM, TN, TK, false, true, false, matmul2d_descriptor::mode::multiply_accumulate);
+    matmul2d<desc, execution_simdgroup> op;
+    tensor<device half, dextents<int32_t, 2>, tensor_inline> tX((device half*)XS, dextents<int32_t, 2>(a.n_embd, 1 << 30));
+    auto rg = op.template get_right_input_cooperative_tensor<half, half, float>();
+    auto ru = op.template get_right_input_cooperative_tensor<half, half, float>();
+    auto tA = tX.template slice<TK, TM>(0, row0);
+#define MM6_DST op.template get_destination_cooperative_tensor<__tensor_ops_detail::__remove_addrspace_t<decltype(tA)>, __tensor_ops_detail::__remove_addrspace_t<decltype(rg)>, float>()
+    auto accg0 = MM6_DST;
+    auto accu0 = MM6_DST;
+    auto accg1 = MM6_DST;
+    auto accu1 = MM6_DST;
+    auto accg2 = MM6_DST;
+    auto accu2 = MM6_DST;
+    auto accg3 = MM6_DST;
+    auto accu3 = MM6_DST;
+#undef MM6_DST
+    for (uint16_t i = 0; i < accg0.get_capacity(); ++i)
+        if (accg0.is_valid_element(i)) {
+            accg0[i] = 0.0f; accu0[i] = 0.0f;
+            if (MS > 1) { accg1[i] = 0.0f; accu1[i] = 0.0f; }
+            if (MS > 2) { accg2[i] = 0.0f; accu2[i] = 0.0f; }
+            if (MS > 3) { accg3[i] = 0.0f; accu3[i] = 0.0f; }
+        }
+    for (int k0 = 0; k0 < int(a.n_embd); k0 += TK) {
+        const uint16_t cap = rg.get_capacity(), half_cap = cap / 2;
+#pragma clang loop unroll(full)
+        for (uint16_t i = 0; i < half_cap; i += 4) {
+            const auto ix = rg.get_multidimensional_index(i);
+            const int k = k0 + ix[0], n = ix[1];
+            half4 wg, wu, wg2, wu2;
+            W8<F>::run(g0 + ulong(n) * a.gu_row, k, tab, wg, wg2);
+            W8<F>::run(u0 + ulong(n) * a.gu_row, k, tab, wu, wu2);
+            rg[i] = wg.x; rg[i + 1] = wg.y; rg[i + 2] = wg.z; rg[i + 3] = wg.w;
+            ru[i] = wu.x; ru[i + 1] = wu.y; ru[i + 2] = wu.z; ru[i + 3] = wu.w;
+            const uint16_t j = i + half_cap;
+            rg[j] = wg2.x; rg[j + 1] = wg2.y; rg[j + 2] = wg2.z; rg[j + 3] = wg2.w;
+            ru[j] = wu2.x; ru[j + 1] = wu2.y; ru[j + 2] = wu2.z; ru[j + 3] = wu2.w;
+        }
+        tA = tX.template slice<TK, TM>(k0, row0);
+        op.run(tA, rg, accg0);
+        op.run(tA, ru, accu0);
+        if (MS > 1 && nsub > 1) {
+            tA = tX.template slice<TK, TM>(k0, row0 + 16);
+            op.run(tA, rg, accg1);
+            op.run(tA, ru, accu1);
+        }
+        if (MS > 2 && nsub > 2) {
+            tA = tX.template slice<TK, TM>(k0, row0 + 32);
+            op.run(tA, rg, accg2);
+            op.run(tA, ru, accu2);
+        }
+        if (MS > 3 && nsub > 3) {
+            tA = tX.template slice<TK, TM>(k0, row0 + 48);
+            op.run(tA, rg, accg3);
+            op.run(tA, ru, accu3);
+        }
+    }
+    tensor<threadgroup float, dextents<int32_t, 2>, tensor_inline> tO(ot[sg], dextents<int32_t, 2>(TN, TM));
+#define MM6_OUT(S, AG, AU) \
+    if (MS > S && nsub > S) { \
+        for (uint16_t i = 0; i < AG.get_capacity(); ++i) \
+            if (AG.is_valid_element(i)) { const float g = AG[i]; AG[i] = g / (1.0f + exp(-g)) * AU[i]; } \
+        simdgroup_barrier(mem_flags::mem_threadgroup); \
+        AG.store(tO); \
+        simdgroup_barrier(mem_flags::mem_threadgroup); \
+        for (int it = lane; it < TM * TN; it += 32) { \
+            const int m = it / TN, f = it - m * TN; \
+            HS[ulong(row0 + S * 16 + m) * a.n_ff + n0 + f] = half(ot[sg][it]); \
+        } \
+    }
+    MM6_OUT(0, accg0, accu0)
+    MM6_OUT(1, accg1, accu1)
+    MM6_OUT(2, accg2, accu2)
+    MM6_OUT(3, accg3, accu3)
+#undef MM6_OUT
+}
+
+template <int F, int MS, int TN, int TK, int SG>
+kernel void mm_down6(device const uchar* W [[buffer(0)]], device const half* HS [[buffer(1)]],
+                     device float* YE [[buffer(2)]], device const int2* tiles [[buffer(3)]], constant MmArgs& a [[buffer(4)]],
+                     uint3 tg [[threadgroup_position_in_grid]], ushort sg [[simdgroup_index_in_threadgroup]],
+                     ushort tid [[thread_index_in_threadgroup]]) {
+    constexpr int TM = 16;
+    threadgroup uint tab[fmt<F>::TAB];
+    load_tab<F>(tab, tid, SG * 32);
+    threadgroup_barrier(mem_flags::mem_threadgroup);
+    const int2 tl = tiles[tg.y];
+    const int e = tl.x & 0xFFFF, nsub = tl.x >> 16, row0 = tl.y, n0 = (int(tg.x) * SG + sg) * TN;
+    device const uchar* w0 = W + ulong(e) * a.d_stride + a.down_off + ulong(n0) * a.d_row;
+    constexpr auto desc = matmul2d_descriptor(TM, TN, TK, false, true, false, matmul2d_descriptor::mode::multiply_accumulate);
+    matmul2d<desc, execution_simdgroup> op;
+    tensor<device half, dextents<int32_t, 2>, tensor_inline> tH((device half*)HS, dextents<int32_t, 2>(a.n_ff, 1 << 30));
+    auto rw = op.template get_right_input_cooperative_tensor<half, half, float>();
+    auto tA = tH.template slice<TK, TM>(0, row0);
+#define MM6_DST op.template get_destination_cooperative_tensor<__tensor_ops_detail::__remove_addrspace_t<decltype(tA)>, __tensor_ops_detail::__remove_addrspace_t<decltype(rw)>, float>()
+    auto acc0 = MM6_DST;
+    auto acc1 = MM6_DST;
+    auto acc2 = MM6_DST;
+    auto acc3 = MM6_DST;
+#undef MM6_DST
+    for (uint16_t i = 0; i < acc0.get_capacity(); ++i)
+        if (acc0.is_valid_element(i)) {
+            acc0[i] = 0.0f;
+            if (MS > 1) acc1[i] = 0.0f;
+            if (MS > 2) acc2[i] = 0.0f;
+            if (MS > 3) acc3[i] = 0.0f;
+        }
+    for (int k0 = 0; k0 < int(a.n_ff); k0 += TK) {
+        const uint16_t half_cap = rw.get_capacity() / 2;
+#pragma clang loop unroll(full)
+        for (uint16_t i = 0; i < half_cap; i += 4) {
+            const auto ix = rw.get_multidimensional_index(i);
+            half4 w, w2;
+            W8<F>::run(w0 + ulong(ix[1]) * a.d_row, k0 + ix[0], tab, w, w2);
+            rw[i] = w.x; rw[i + 1] = w.y; rw[i + 2] = w.z; rw[i + 3] = w.w;
+            const uint16_t j = i + half_cap;
+            rw[j] = w2.x; rw[j + 1] = w2.y; rw[j + 2] = w2.z; rw[j + 3] = w2.w;
+        }
+        tA = tH.template slice<TK, TM>(k0, row0);
+        op.run(tA, rw, acc0);
+        if (MS > 1 && nsub > 1) { tA = tH.template slice<TK, TM>(k0, row0 + 16); op.run(tA, rw, acc1); }
+        if (MS > 2 && nsub > 2) { tA = tH.template slice<TK, TM>(k0, row0 + 32); op.run(tA, rw, acc2); }
+        if (MS > 3 && nsub > 3) { tA = tH.template slice<TK, TM>(k0, row0 + 48); op.run(tA, rw, acc3); }
+    }
+    tensor<device float, dextents<int32_t, 2>, tensor_inline> tY(YE, dextents<int32_t, 2>(a.n_embd, 1 << 30));
+    acc0.store(tY.slice(n0, row0));
+    if (MS > 1 && nsub > 1) acc1.store(tY.slice(n0, row0 + 16));
+    if (MS > 2 && nsub > 2) acc2.store(tY.slice(n0, row0 + 32));
+    if (MS > 3 && nsub > 3) acc3.store(tY.slice(n0, row0 + 48));
+}
+
+#define MMGU6_(F, M, N, K, S) template [[host_name("mm_gu6_" #F "_" #M "_" #N "_" #K "_" #S)]] kernel decltype(mm_gu6<F, M, N, K, S>) mm_gu6<F, M, N, K, S>;
+#define MMDN6_(F, M, N, K, S) template [[host_name("mm_down6_" #F "_" #M "_" #N "_" #K "_" #S)]] kernel decltype(mm_down6<F, M, N, K, S>) mm_down6<F, M, N, K, S>;
+#define MMGU6(F, M, N, K, S) MMGU6_(F, M, N, K, S)
+#define MMDN6(F, M, N, K, S) MMDN6_(F, M, N, K, S)
+#define MM6_GU(F) MMGU6(F, 2, 32, 32, 4) MMGU6(F, 3, 32, 32, 4) MMGU6(F, 4, 32, 32, 4) MMGU6(F, 4, 32, 32, 2) MMGU6(F, 4, 16, 32, 4)
+#define MM6_DN(F) MMDN6(F, 2, 64, 32, 2) MMDN6(F, 3, 64, 32, 2) MMDN6(F, 4, 64, 32, 2) MMDN6(F, 4, 32, 32, 4) MMDN6(F, 4, 32, 32, 2)
+MM6_GU(T_Q4_K) MM6_GU(T_Q5_K)
+MM6_DN(T_Q5_1) MM6_DN(T_Q8_0)
+// the other sets' types (UD-IQ4_XS-v31, IQ3_S): the one tile shape the engine uses (MS 4; gate/up TN 16, down TN 32, SG 4)
+#define MM6_GU1(F) MMGU6(F, 4, 16, 32, 4)
+#define MM6_DN1(F) MMDN6(F, 4, 32, 32, 4)
+MM6_GU1(T_Q6_K) MM6_GU1(T_Q8_0) MM6_GU1(T_IQ3_XXS) MM6_GU1(T_IQ2_S) MM6_GU1(T_IQ4_XS) MM6_GU1(T_IQ3_S) MM6_GU1(T_IQ2_XS) MM6_GU1(T_IQ2_XXS)
+MM6_DN1(T_Q5_0) MM6_DN1(T_IQ4_NL) MM6_DN1(T_Q2_0)

metal/kernels/qsa_prompt.metal

@@ -153,6 +153,32 @@ kernel void qsa_q_f16(device const float* q [[buffer(0)]], device half* qh [[buf
     if (gid < n) qh[gid] = half(q[gid]);
 }
 
+// qsa_merge for a prompt chunk with the fp16 output the attn_output GEMM reads: qsa_merge's y, as half, at [t][h][d]
+// (qsa_merge writes y into the first plane of the prepared activations and act_f16 rounds that plane to half; here one
+// pass, bitwise the same halves, and no other plane or sum).  grid (QH, n_tok), 256 threads.
+kernel void qsa_merge_h(device const float* part [[buffer(0)]], device const float* qfull [[buffer(1)]],
+                        device half* xh [[buffer(2)]], constant QsaArgs& a [[buffer(3)]],
+                        uint3 tg [[threadgroup_position_in_grid]], ushort tid [[thread_index_in_threadgroup]]) {
+    const int h = int(tg.x), t = int(tg.y);
+    const int nchunks = (QBUDGET + QCH - 1) / QCH;
+    const uint n_kv = a.pos0 + uint(t) + 1, width = min(n_kv, uint(QBUDGET));
+    const int nc = int((width + QCH - 1) / QCH);
+    device const float* pp = part + (uint(t) * nchunks * QH + uint(h)) * (QD + 2);
+    float M = -INFINITY;
+    for (int k = 0; k < nc; ++k) M = max(M, pp[k * QH * (QD + 2) + QD]);
+    float L = 0.0f, o = 0.0f;
+    for (int k = 0; k < nc; ++k) {
+        device const float* q = pp + k * QH * (QD + 2);
+        const float e = exp(q[QD] - M);
+        L = fma(q[QD + 1], e, L);
+        o = fma(q[tid], e, o);
+    }
+    o = o / L;
+    const float gate = qfull[(t * QH + h) * 2 * QD + QD + tid];
+    const float y = o * (1.0f / (1.0f + exp(-gate)));
+    xh[(uint(t) * QH + uint(h)) * QD + tid] = half(y);
+}
+
 // qsa_score as a GEMM (fp32 operands, so the selection sees qsa_score's values to fp32 rounding): S[(t, h)][b] =
 // qidx[t][h] . pooled[b] for 16 tokens x IH heads (64 rows) by 64 blocks a threadgroup; the epilogue sums relu over
 // the heads.  The tail block n_bid (the `dead` key, + 1e9 when it has cells) and blocks past it are left to the

metal/setup_mac.py

@@ -329,6 +329,31 @@ def start(cfg_path: Path, port: int | None, open_browser: bool = True) -> int:
     return subprocess.call(cmd)
 
 
+# ------------------------------------------------------------------------------------------------ the DFlash2 drafter
+def fetch_drafter(model_name: str, data: Path) -> Path | None:
+    """The DFlash2 drafter of a set whose Mac default is DFlash2 (strata_control/drafting.py SET_DEFAULTS: UD-Q4_K_XL and UD-IQ4_XS-v31): the tuned one (0.34 GB), else the stock one
+    (1.2 GB), through the normal download path (resumable, the pinned size and SHA-256). Offline, or an address that does not answer: nothing is said and the model keeps the MTP
+    (owner, strata-offline-users). A fault that is the person's to know (a file that failed its safety check) is one plain line. Never stops the setup. None when it is not here."""
+    from strata_control import acquisition, drafting, drafting_fetch
+    if drafting.choice_of("auto", model_name=model_name, backend="metal") != "dflash2":
+        return None
+    said = []
+
+    def show(step: str, done: int, total: int) -> None:
+        if not said:                                       # the line appears when the download starts: a computer with no internet sees nothing
+            said.append(True)
+            gb = drafting.TUNED_GB if "tuned" in step else drafting.DOWNLOAD_GB
+            S.say(f"  Downloading the DFlash2 drafter ({gb:g} GB, once) ...")
+    try:
+        got = drafting_fetch.ensure_for(data, model_name=model_name, backend="metal", progress=show)
+    except (drafting_fetch.DraftingError, acquisition.AcquisitionRefused, OSError) as exc:
+        S.warn(f"{exc} This model keeps using the MTP.")
+        return None
+    if got is not None:
+        S.ok(f"DFlash2 drafter: {got}")
+    return got
+
+
 # ------------------------------------------------------------------------------------------------ main
 def main(argv=None) -> int:
     ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
@@ -547,6 +572,8 @@ def main(argv=None) -> int:
         S.refresh_draft_vocab(rt, a.draft_vocab or "cjk")
         S.ok(f"MTP draft layer: {mtp / 'mtp-q2_0.gguf'}")
 
+    fetch_drafter(f"{fam['name']}-{model.lower()}", pack.parent.parent)       # DFlash2 is this set's default drafter (the MTP above stays as the fallback); the folder a start looks in
+
     # ---- 7. the config, the start script
     S.step(7, "writing the start script")
     port = a.port or 8080

metal/src/engine/prompt_chain.inc

@@ -62,7 +62,34 @@ bool Window::pc_setup(int C) {
     b.C = C;
     const size_t MT = (size_t) C;
     const int MB = (int) max_cells_ / 4 + 2;
-    b.maxrows = (int) MT * TOPK + nexp_ * PC_TM;
+    // the expert GEMM tiles (prompt_chain's defaults): v3 gate/up 16,32,32,4 and down 16,64,32,2 with the M5 layout, else v2.
+    // STRATA_PROMPT_GU / STRATA_PROMPT_DN = TM,TN,TK,SG override them (test tooling; TM = the plan's tile height, the same for both)
+    b.gu3[0] = 16; b.gu3[1] = 32; b.gu3[2] = 32; b.gu3[3] = 4;
+    b.dn3[0] = 16; b.dn3[1] = 64; b.dn3[2] = 32; b.dn3[3] = 2;
+    if (const char* e = std::getenv("STRATA_PROMPT_GU")) std::sscanf(e, "%d,%d,%d,%d", &b.gu3[0], &b.gu3[1], &b.gu3[2], &b.gu3[3]);
+    if (const char* e = std::getenv("STRATA_PROMPT_DN")) std::sscanf(e, "%d,%d,%d,%d", &b.dn3[0], &b.dn3[1], &b.dn3[2], &b.dn3[3]);
+    if (!b.v3e) { b.gu3[0] = b.dn3[0] = PC_TM; }
+    b.tm = b.gu3[0];
+    if (b.v3e) {   // STRATA_PROMPT_V6=MS (default 4, 0 = off): ragged tiles, groups of MS (2..4) 16-row subtiles of one expert share the weight decode
+        // the ragged-tile kernels exist for these expert types only (every layer's must be one of them)
+        auto gu_ok = [](int t) { return t == ref::Q4_K || t == ref::Q5_K || t == ref::Q6_K || t == ref::Q8_0 || t == ref::IQ3_XXS ||
+                                        t == ref::IQ2_S || t == ref::IQ4_XS || t == ref::IQ3_S || t == ref::IQ2_XS || t == ref::IQ2_XXS; };
+        auto dn_ok = [](int t) { return t == ref::Q5_1 || t == ref::Q8_0 || t == ref::Q5_0 || t == ref::IQ4_NL || t == ref::Q2_0; };
+        bool types_ok = true;
+        for (int l = 0; l < 48 && types_ok; ++l) {
+            const std::string bl = "blk." + std::to_string(l) + ".";
+            types_ok = gu_ok(T(bl + "ffn_gate_exps.weight").type) && gu_ok(T(bl + "ffn_up_exps.weight").type) && dn_ok(T(bl + "ffn_down_exps.weight").type);
+        }
+        const char* e6 = std::getenv("STRATA_PROMPT_V6");
+        const int ms = types_ok ? (e6 ? std::atoi(e6) : 4) : 0;   // on by default (STRATA_PROMPT_V6=0 off, 2..4 = the group size)
+        if (ms >= 2) {
+            b.v6 = true; b.ms = std::min(ms, 4); b.tm = 16; b.gu3[0] = b.dn3[0] = 16;
+            if (!std::getenv("STRATA_PROMPT_GU")) { b.gu3[1] = 16; b.gu3[2] = 32; b.gu3[3] = 4; }   // measured (moe_mm_parity, T 2048): the accumulators of MS subtiles x gate/up need the narrow tile
+            if (!std::getenv("STRATA_PROMPT_DN")) { b.dn3[1] = 32; b.dn3[2] = 32; b.dn3[3] = 4; }
+        }
+    }
+    if (b.dn3[0] != b.tm) { std::fprintf(stderr, "strata metal: STRATA_PROMPT_GU/DN tile heights differ: using %d for both\n", b.tm); b.dn3[0] = b.tm; }
+    b.maxrows = (int) MT * TOPK + nexp_ * b.tm;
     const int nblk_max = ((int) MT * TOPK + 1023) / 1024;
     auto F32 = [&](size_t n) { return cx_->alloc(4 * n); };
     auto F16 = [&](size_t n) { return cx_->alloc(2 * n); };
@@ -84,7 +111,7 @@ bool Window::pc_setup(int C) {
     b.xqh = F16(MT * 6144);
     b.logits = F32(MT * nexp_); b.ids = cx_->alloc(4 * MT * TOPK); b.wts = F32(MT * TOPK);
     b.bcnt = cx_->alloc(4 * (size_t) nblk_max * nexp_); b.boff = cx_->alloc(4 * (size_t) nblk_max * nexp_);
-    b.eoff = cx_->alloc(4 * nexp_); b.tiles = cx_->alloc(8 * (size_t) (b.maxrows / PC_TM + 1)); b.meta = cx_->alloc(64);
+    b.eoff = cx_->alloc(4 * nexp_); b.tiles = cx_->alloc(8 * (size_t) (b.maxrows / b.tm + 1)); b.meta = cx_->alloc(64);
     b.stok = cx_->alloc(4 * (size_t) b.maxrows); b.sw = F32(b.maxrows); b.pos = cx_->alloc(4 * MT * TOPK);
     b.XS = F16((size_t) b.maxrows * N); b.HS = F16((size_t) b.maxrows * NFF); b.YE = F32((size_t) b.maxrows * N);
     b.yr = F32(MT * N); b.gsh = F32(MT * NFF); b.ush = F32(MT * NFF); b.hh = F16(MT * NFF); b.ys = F32(MT * N); b.sgv = F32(MT);
@@ -104,10 +131,6 @@ bool Window::pc_setup(int C) {
     b.qn = F32(MT * 10240); b.gated = F32(MT * 10240); b.normW = F32(MT * 10240);
     if (had_on_) b.mixr = F32(MT * N);
     b.Wpk = F16((size_t) 10240 * N); b.Wpv = F16((size_t) N * N);
-    // the expert GEMM tiles (prompt_chain's defaults): v3 gate/up 16,32,32,4 and down 16,64,32,2 with the M5 layout,
-    // else v2
-    b.gu3[0] = 16; b.gu3[1] = 32; b.gu3[2] = 32; b.gu3[3] = 4;
-    b.dn3[0] = 16; b.dn3[1] = 64; b.dn3[2] = 32; b.dn3[3] = 2;
     b.meta.as<uint32_t>()[2] = b.v3e ? NFF / (b.gu3[1] * b.gu3[3]) : NFF / 64;
     b.meta.as<uint32_t>()[5] = b.v3e ? N / (b.dn3[1] * b.dn3[3]) : N / 64;
     std::fprintf(stderr, "strata metal: prompt kernels (mac-kern's prompt chain) for chunks of %d rows, %s, ready in %.0f ms\n",
@@ -116,32 +139,112 @@ bool Window::pc_setup(int C) {
     return true;
 }
 
-// one chunk of nT rows at cells pos0.. (the window state is exactly at pos0 before, at pos0 + nT after)
-bool Window::pc_chunk(const int64_t* tokens, int nT, int64_t pos0, std::string& err) {
-    PromptBufs& b = pb_;
-    strata::core::SessionState& ss = *ss_;
-    const double th0 = now_ms();
-    // ---- the rows: the embedding into every stream; the PLE table rows
+// STRATA_PROMPT_KPROF: the per-dispatch GPU times of one chunk (see pc_chunk).  The table (all profiled chunks summed) is
+// rewritten after every chunk to the file named by the variable's value (stderr if it is not a path): the engine may leave
+// without running atexit handlers.
+void Window::pc_kprof(const std::vector<mtl::Dispatch>& ds, int nT, int64_t pos0) {
+    static std::map<std::string, std::pair<double, int>> acc;
+    double sum = 0;
+    for (const auto& d : ds) {
+        std::string k = d.p->name;
+        if (k.rfind("gemm_f16", 0) == 0 || k.rfind("gemm_hc_up", 0) == 0) {
+            for (const auto& a : d.args)
+                if (a.bytes.size() == sizeof(PcGemmArgs)) {
+                    PcGemmArgs g;
+                    std::memcpy(&g, a.bytes.data(), sizeof g);
+                    k += " M" + std::to_string(g.M) + " N" + std::to_string(g.N) + " K" + std::to_string(g.K);
+                    break;
+                }
+        } else if (k.rfind("dq_f16", 0) == 0) {
+            for (const auto& a : d.args)
+                if (a.bytes.size() == sizeof(PcDqArgs)) {
+                    PcDqArgs g;
+                    std::memcpy(&g, a.bytes.data(), sizeof g);
+                    k += " in" + std::to_string(g.n_in) + " out" + std::to_string(g.n_out);
+                    break;
+                }
+        }
+        const double s = cx_->run({d}, 1, false, false);
+        sum += s;
+        auto& e = acc[k];
+        e.first += s;
+        e.second += 1;
+    }
+    const double g0 = now_ms();
+    const double gg = cx_->run(ds);
+    const char* path = std::getenv("STRATA_PROMPT_KPROF");
+    FILE* f = path != nullptr && path[0] == '/' ? std::fopen(path, "w") : stderr;
+    if (f == nullptr) f = stderr;
+    double tot = 0;
+    std::vector<std::pair<double, std::string>> v;
+    for (auto& [k, x] : acc) { tot += x.first; v.emplace_back(x.first, k); }
+    std::sort(v.begin(), v.end(), [](auto& a, auto& b) { return a.first > b.first; });
+    std::fprintf(f, "KPROF-TABLE total %.1f ms over all profiled chunks\n", tot * 1e3);
+    for (auto& [t, k] : v) std::fprintf(f, "KPROF %9.2f ms %5.1f%% n=%-5d %s\n", t * 1e3, 100 * t / tot, acc[k].second, k.c_str());
+    std::fprintf(f, "KPROF chunk pos %lld rows %d: %zu dispatches, sum of single-dispatch GPU times %.1f ms; the same chunk as one command buffer %.1f ms (GPU %.1f)\n",
+                 (long long) pos0, nT, ds.size(), sum * 1e3, now_ms() - g0, gg * 1e3);
+    if (f != stderr) std::fclose(f);
+}
+
+// The host rows of a chunk: the token embedding rows (dequantized from the model file; rows of an image are the caller's) and the
+// PLE table rows of its tokens.  Pure CPU work on the model file: it can run beside the GPU's previous chunk.
+bool Window::pc_host_rows(const int64_t* tokens, int nT, int64_t pos0, const int32_t prev_in[2], PcHost& out) {
+    const double t0 = now_ms();
+    out.nT = nT;
+    out.pos0 = pos0;
+    out.ok = true;
     const gguf::Tensor& te = T("token_embd.weight");
     const size_t rb = ref::row_bytes(te.type, N);
-    mrope_sync(pos0, nT);
-    for (int t = 0; t < nT; ++t) {
-        float* r = b.R.as<float>() + (size_t) t * HC * N;
-        if (const float* img = img_row(pos0 + t)) std::memcpy(r, img, (size_t) N * 4);
-        else ref::dequant(te.type, (const uint8_t*) W_.ptr + te.offset + (size_t) tokens[t] * rb, r, N);
-        for (int c = 1; c < HC; ++c) std::memcpy(r + (size_t) c * N, r, (size_t) N * 4);
-    }
+    out.emb.resize((size_t) nT * N);
+    for (int t = 0; t < nT; ++t)
+        if (img_row(pos0 + t) == nullptr)
+            ref::dequant(te.type, (const uint8_t*) W_.ptr + te.offset + (size_t) tokens[t] * rb, out.emb.data() + (size_t) t * N, N);
+    out.ms_emb = now_ms() - t0;
     if (ple_on_) {
+        strata::core::SessionState& ss = *ss_;
         std::vector<uint32_t> rows((size_t) nT * strata::kernels::PLE_N_HEADS);
-        int32_t prev[2] = {ss.ple_prev[0], ss.ple_prev[1]};
+        int32_t prev[2] = {prev_in[0], prev_in[1]};
         for (int t = 0; t < nT; ++t) {
             const int32_t tk = (int32_t) tokens[t];
             strata::kernels::ngram_rows(&tk, prev, 1, ss.ple.consts, rows.data() + (size_t) t * strata::kernels::PLE_N_HEADS);
             prev[0] = prev[1];
             prev[1] = tk;
         }
-        if (!ss.ple.table->gather_batch(rows.data(), (size_t) nT, b.pemb.as<float>(), err)) return false;
+        out.ple.resize((size_t) nT * N);
+        if (!ss.ple.table->gather_batch(rows.data(), (size_t) nT, out.ple.data(), out.err)) out.ok = false;
     }
+    out.ms = now_ms() - t0;
+    return out.ok;
+}
+
+// one chunk of nT rows at cells pos0.. (the window state is exactly at pos0 before, at pos0 + nT after).  next_tokens: the
+// following chunk of the same prompt, whose host rows are computed on a thread while this chunk runs on the GPU.
+bool Window::pc_chunk(const int64_t* tokens, int nT, int64_t pos0, std::string& err, const int64_t* next_tokens, int next_nT) {
+    PromptBufs& b = pb_;
+    strata::core::SessionState& ss = *ss_;
+    const double th0 = now_ms();
+    // ---- the rows: the embedding into every stream; the PLE table rows
+    PcHost cur;
+    bool have = false;
+    if (pc_next_.valid()) {
+        cur = pc_next_.get();
+        have = cur.ok && cur.pos0 == pos0 && cur.nT == nT;
+        if (have) b.hidden_ms += cur.ms;
+    }
+    if (have) { b.emb_ms += cur.ms_emb; b.ple_ms += cur.ms - cur.ms_emb; }
+    if (!have) {
+        const int32_t prev0[2] = {ss.ple_prev[0], ss.ple_prev[1]};
+        if (!pc_host_rows(tokens, nT, pos0, prev0, cur)) { err = cur.err; return false; }
+        b.emb_ms += cur.ms_emb; b.ple_ms += cur.ms - cur.ms_emb;
+    }
+    mrope_sync(pos0, nT);
+    for (int t = 0; t < nT; ++t) {
+        float* r = b.R.as<float>() + (size_t) t * HC * N;
+        if (const float* img = img_row(pos0 + t)) std::memcpy(r, img, (size_t) N * 4);
+        else std::memcpy(r, cur.emb.data() + (size_t) t * N, (size_t) N * 4);
+        for (int c = 1; c < HC; ++c) std::memcpy(r + (size_t) c * N, r, (size_t) N * 4);
+    }
+    if (ple_on_) std::memcpy(b.pemb.as<float>(), cur.ple.data(), (size_t) nT * N * 4);
     b.host_ms += now_ms() - th0;
     std::vector<mtl::Dispatch> ds;
     ds.reserve(2400);
@@ -296,8 +399,13 @@ bool Window::pc_chunk(const int64_t* tokens, int nT, int64_t pos0, std::string&
                                           mtl::Arg::B(b.part), mtl::Arg::V(a)},
                   PC_QCHUNKS, 2, 256, nT);
             }
-            D(P_(b.lqsa, "qsa_merge"), {mtl::Arg::B(b.part), mtl::Arg::B(b.qfull), mtl::Arg::B(b.XQ), mtl::Arg::V(a)}, 24, nT, 256);
-            act16(b.XQ, b.xqh, (size_t) nT * 6144);
+            static const bool mergeh = [] { const char* e = std::getenv("STRATA_PROMPT_MERGEH"); return !(e && std::string(e) == "0"); }();
+            if (mergeh) {   // the merge writes the fp16 operand directly (bitwise the same halves)
+                D(P_(b.lqsa, "qsa_merge_h"), {mtl::Arg::B(b.part), mtl::Arg::B(b.qfull), mtl::Arg::B(b.xqh), mtl::Arg::V(a)}, 24, nT, 256);
+            } else {
+                D(P_(b.lqsa, "qsa_merge"), {mtl::Arg::B(b.part), mtl::Arg::B(b.qfull), mtl::Arg::B(b.XQ), mtl::Arg::V(a)}, 24, nT, 256);
+                act16(b.XQ, b.xqh, (size_t) nT * 6144);
+            }
             gemm(b.xqh, b.Wout, b.bo, nT, N, 6144);
             ++qi;
         }
@@ -313,11 +421,11 @@ bool Window::pc_chunk(const int64_t* tokens, int nT, int64_t pos0, std::string&
         const uint32_t nt = (uint32_t) nT;
         D(P_(b.lib, "route_topk"), {mtl::Arg::B(b.logits), mtl::Arg::B(b.ids), mtl::Arg::B(b.wts), mtl::Arg::V(nt), mtl::Arg::V((uint32_t) nexp_)}, (nT + 7) / 8, 1, 256);
         const int nent = nT * TOPK, nblk = (nent + 1023) / 1024;
-        PcMmPlanArgs pa{(uint32_t) nent, TOPK, (uint32_t) nexp_, (uint32_t) PC_TM, (uint32_t) nblk};
+        PcMmPlanArgs pa{(uint32_t) nent, TOPK, (uint32_t) nexp_, (uint32_t) (b.v6 ? b.ms : b.tm), (uint32_t) nblk};
         const uint32_t nclr = (uint32_t) b.maxrows;
         D(P_(b.lib, "mm_clear"), {mtl::Arg::B(b.stok), mtl::Arg::V(nclr)}, (b.maxrows + 255) / 256, 1, 256);
         D(P_(b.lib, "mm_hist"), {mtl::Arg::B(b.ids), mtl::Arg::B(b.bcnt), mtl::Arg::V(pa)}, nblk, 1, 1024);
-        D(P_(b.lib, "mm_scan"), {mtl::Arg::B(b.bcnt), mtl::Arg::B(b.boff), mtl::Arg::B(b.eoff), mtl::Arg::B(b.tiles), mtl::Arg::B(b.meta),
+        D(P_(b.lib, b.v6 ? "mm_scan6" : "mm_scan"), {mtl::Arg::B(b.bcnt), mtl::Arg::B(b.boff), mtl::Arg::B(b.eoff), mtl::Arg::B(b.tiles), mtl::Arg::B(b.meta),
                                 mtl::Arg::V(pa)}, 1, 1, nexp_);
         D(P_(b.lib, "mm_place"), {mtl::Arg::B(b.ids), mtl::Arg::B(b.wts), mtl::Arg::B(b.boff), mtl::Arg::B(b.eoff), mtl::Arg::B(b.stok),
                                  mtl::Arg::B(b.sw), mtl::Arg::B(b.pos), mtl::Arg::V(pa)}, nblk, 1, 1024);
@@ -339,7 +447,9 @@ bool Window::pc_chunk(const int64_t* tokens, int nT, int64_t pos0, std::string&
         auto tl3 = [](const int* t) { return std::to_string(t[0]) + "_" + std::to_string(t[1]) + "_" + std::to_string(t[2]) + "_" + std::to_string(t[3]); };
         mtl::Dispatch gu;
         if (b.v3e) {
-            const std::string fn = "mm_gu3_" + std::to_string(eg.type) + "_" + tl3(b.gu3);
+            const std::string fn = b.v6 ? "mm_gu6_" + std::to_string(eg.type) + "_" + std::to_string(b.ms) + "_" + std::to_string(b.gu3[1]) + "_" +
+                                              std::to_string(b.gu3[2]) + "_" + std::to_string(b.gu3[3])
+                                        : "mm_gu3_" + std::to_string(eg.type) + "_" + tl3(b.gu3);
             gu.p = P_(b.lib, fn, mtl::Constants().b(7, true), fn + "#k4");
         } else {
             const std::string fn = "mm_gu2_" + std::to_string(eg.type) + gu2_tile(eg.type);
@@ -353,7 +463,9 @@ bool Window::pc_chunk(const int64_t* tokens, int nT, int64_t pos0, std::string&
         if (hd) had_rows_ind(ds, b.lib, "pc:", b.HS, b.meta);
         mtl::Dispatch dn;
         if (b.v3e) {
-            const std::string fn = "mm_down3_" + std::to_string(ed.type) + "_" + tl3(b.dn3);
+            const std::string fn = b.v6 ? "mm_down6_" + std::to_string(ed.type) + "_" + std::to_string(b.ms) + "_" + std::to_string(b.dn3[1]) + "_" +
+                                              std::to_string(b.dn3[2]) + "_" + std::to_string(b.dn3[3])
+                                        : "mm_down3_" + std::to_string(ed.type) + "_" + tl3(b.dn3);
             dn.p = P_(b.lib, fn, mtl::Constants().b(7, true), fn + "#k4");
         } else {
             dn.p = P_(b.lib, "mm_down2_" + std::to_string(ed.type) + dn2_tile(ed.type));
@@ -407,6 +519,21 @@ bool Window::pc_chunk(const int64_t* tokens, int nT, int64_t pos0, std::string&
         D(P_(b.lib, "gr_norm"), {mtl::Arg::B(b.R), mtl::Arg::B(b.bo), mtl::Arg::B(b.inj[cur_inj]),
                                 mtl::Arg::B(W_, T("output_hc_norm.weight").offset), mtl::Arg::B(b.xn), mtl::Arg::V(a)}, HC, nT, 256);
     }
+    // STRATA_PROMPT_KPROF=1 (test tooling): every dispatch alone in its own command buffer, its GPU time kept per kernel and
+    // shape, a table at exit.  The chunk then runs once more as a whole, so the GDN state is advanced twice: profiling only.
+    static const bool kprof = std::getenv("STRATA_PROMPT_KPROF") != nullptr;
+    if (kprof && nT >= 1024) pc_kprof(ds, nT, pos0);
+    static const bool hostpipe = [] { const char* e = std::getenv("STRATA_METAL_PROMPT_HOSTPIPE"); return !(e && std::string(e) == "0"); }();
+    if (hostpipe && next_tokens != nullptr && next_nT > 0 && g_embd_rows == nullptr) {
+        const int32_t pv[2] = {nT >= 2 ? (int32_t) tokens[nT - 2] : ss.ple_prev[1], (int32_t) tokens[nT - 1]};
+        const int64_t npos = pos0 + nT;
+        pc_next_ = std::async(std::launch::async, [this, next_tokens, next_nT, npos, pv0 = pv[0], pv1 = pv[1]] {
+            PcHost h;
+            const int32_t pp[2] = {pv0, pv1};
+            pc_host_rows(next_tokens, next_nT, npos, pp, h);
+            return h;
+        });
+    }
     const double g = cx_->run(ds);
     b.gpu_ms += 1000.0 * g;
     hp_ = 1 - hp_;
@@ -442,12 +569,13 @@ bool Window::prompt(const int64_t* tokens, int64_t n, int64_t pos0,
     }();
     if (!pc_setup((int) std::min<int64_t>(C, std::max<int64_t>(n, 64)))) { err = "metal prompt: setup failed"; return false; }
     const double t0 = now_ms();
-    pb_.gpu_ms = pb_.host_ms = pb_.mtp_ms = 0;
-    struct MtpJoin { Window* w; ~MtpJoin() { w->mtp_wait(); } } mtp_join{this};   // (every way out waits for the drafter's last rows)
+    pb_.gpu_ms = pb_.host_ms = pb_.mtp_ms = pb_.hidden_ms = pb_.emb_ms = pb_.ple_ms = 0;
+    struct MtpJoin { Window* w; ~MtpJoin() { w->mtp_wait(); w->pc_drain(); } } mtp_join{this};   // (every way out waits for the drafter's last rows)
     for (int64_t c = 0; c < n; c += C) {
         const int nT = (int) std::min<int64_t>(C, n - c);
         if (pos0 + c + nT + 1 >= max_cells_) { err = "metal prompt: the prompt runs past the context"; return false; }
-        if (!pc_chunk(tokens + c, nT, pos0 + c, err)) return false;
+        const int next_nT = (int) std::min<int64_t>(C, n - c - nT);
+        if (!pc_chunk(tokens + c, nT, pos0 + c, err, next_nT > 0 ? tokens + c + nT : nullptr, next_nT)) return false;
         if (on_chunk) {
             const double tm0 = now_ms();
             mpsrc_ = &pb_.R;   // the drafter's prompt rows are this chunk's final residuals
@@ -459,8 +587,8 @@ bool Window::prompt(const int64_t* tokens, int64_t n, int64_t pos0,
         }
     }
     if (std::getenv("STRATA_METAL_TRACE") || std::getenv("STRATA_METAL_PROFILE"))
-        std::fprintf(stderr, "strata metal: prompt %lld tokens in %.1f ms (GPU %.1f ms, host rows %.1f ms, drafter rows %.1f ms) "
-                             "-> %.1f tok/s\n", (long long) n, now_ms() - t0, pb_.gpu_ms, pb_.host_ms, pb_.mtp_ms,
+        std::fprintf(stderr, "strata metal: prompt %lld tokens in %.1f ms (GPU %.1f ms, host rows %.1f ms [+%.1f ms beside the GPU; embeddings %.1f, PLE %.1f], drafter rows %.1f ms) "
+                             "-> %.1f tok/s\n", (long long) n, now_ms() - t0, pb_.gpu_ms, pb_.host_ms, pb_.hidden_ms, pb_.emb_ms, pb_.ple_ms, pb_.mtp_ms,
                      1000.0 * (double) n / (now_ms() - t0));
     return true;
 }

metal/src/engine/stubs.cpp

@@ -190,7 +190,7 @@ namespace strata::kernels { void doorbell_wait(const uint32_t * p0, const uint32
 namespace strata::kernels { void embedding_gather(const uint8_t * p0, const float * p1, const float * p2, int64_t p3, int p4, int p5, int p6, float * p7, void * p8) { strata::metal_engine::missing("strata::kernels::embedding_gather"); } }
 namespace strata::kernels { void embedding_gather_dev(const uint8_t * p0, const float * p1, const float * p2, const int32_t * p3, int p4, int64_t p5, int p6, int p7, int p8, uint64_t p9, uint64_t p10, float * p11, void * p12) { strata::metal_engine::missing("strata::kernels::embedding_gather_dev"); } }
 namespace strata::kernels { void f32_to_bf16_bulk(const float * p0, uint16_t * p1, int64_t p2, void * p3) { strata::metal_engine::missing("strata::kernels::f32_to_bf16_bulk"); } }
-namespace strata::kernels { void fetch_blobs(const unsigned long long * p0, const int32_t * p1, uint8_t * p2, int64_t p3, int p4, void * p5) { strata::metal_engine::missing("strata::kernels::fetch_blobs"); } }
+namespace strata::kernels { void fetch_blobs(const unsigned long long * p0, const int32_t * p1, uint8_t * p2, int64_t p3, int p4, void * p5, const unsigned long long * p6) { strata::metal_engine::missing("strata::kernels::fetch_blobs"); } }
 namespace strata::kernels { void force_token(int32_t * p0, const int32_t * p1, int p2, void * p3) { strata::metal_engine::missing("strata::kernels::force_token"); } }
 namespace strata::kernels { void fused_gdn_ab(const float * p0, const uint16_t * p1, const uint16_t * p2, const float * p3, const float * p4, float * p5, float * p6, int p7, int p8, void * p9) { strata::metal_engine::missing("strata::kernels::fused_gdn_ab"); } }
 namespace strata::kernels { void fused_gdn_conv_l2(float * p0, const float * p1, const float * p2, float * p3, int p4, int p5, float p6, void * p7) { strata::metal_engine::missing("strata::kernels::fused_gdn_conv_l2"); } }
@@ -304,7 +304,7 @@ namespace strata::kernels { void quantize_q8_0(const float * p0, uint8_t * p1, i
 namespace strata::kernels { void quantize_q8_0_scaled(const float * p0, uint8_t * p1, float * p2, int64_t p3, void * p4) { strata::metal_engine::missing("strata::kernels::quantize_q8_0_scaled"); } }
 namespace strata::kernels { void quantize_q8_1_rows(const float * p0, int64_t p1, int64_t p2, void * p3, void * p4) { strata::metal_engine::missing("strata::kernels::quantize_q8_1_rows"); } }
 namespace strata::kernels { void quantize_q8_K(const float * p0, uint8_t * p1, int64_t p2, void * p3) { strata::metal_engine::missing("strata::kernels::quantize_q8_K"); } }
-namespace strata::kernels { void rebase_ptrs(unsigned long long * p0, const int32_t * p1, uint8_t * p2, int64_t p3, void * p4) { strata::metal_engine::missing("strata::kernels::rebase_ptrs"); } }
+namespace strata::kernels { void rebase_ptrs(unsigned long long * p0, const int32_t * p1, uint8_t * p2, int64_t p3, void * p4, const unsigned long long * p5) { strata::metal_engine::missing("strata::kernels::rebase_ptrs"); } }
 namespace strata::kernels { void resident_plan(const int32_t * p0, int p1, int p2, const int32_t * p3, int p4, const uint8_t * p5, const unsigned long long * p6, long long p7, int32_t * p8, long long p9, uint32_t * p10, uint32_t p11, void * p12, uint32_t * p13) { strata::metal_engine::missing("strata::kernels::resident_plan"); } }
 namespace strata::kernels { void rms_norm_weighted(float * p0, const float * p1, int64_t p2, int64_t p3, float p4, void * p5) { strata::metal_engine::missing("strata::kernels::rms_norm_weighted"); } }
 namespace strata::kernels { void rope_neox_apply(const float * p0, float * p1, int64_t p2, int p3, int p4, const float * p5, const float * p6, const int * p7, void * p8) { strata::metal_engine::missing("strata::kernels::rope_neox_apply"); } }

metal/src/engine/stubs_025.cpp

@@ -16,4 +16,18 @@ void iq512x_rows(int, const uint8_t*, size_t, int, const void* const*, int, floa
     strata::metal_engine::missing("strata::kernels::cpu::iq512x_rows");
 }
 
+// round 0216 (f-0216-cpu): the fused gate/up, reference gate/up, Q5_0 down and Q2_0 512-bit rows; the callers ask cpu_vnni_ok() /
+// iq512f_supported() first, and both answer no on arm64, so the stopping stubs are never reached.
+bool q2_rows_512_ok() { return false; }
+bool iq512f_supported(int) noexcept { return false; }
+void iq512_fused_gu_rows(int, const uint8_t*, size_t, size_t, int, const void* const*, int, float* const*, int, int) {
+    strata::metal_engine::missing("strata::kernels::cpu::iq512_fused_gu_rows");
+}
+void iq512_reference_gu_rows(int, const uint8_t*, size_t, size_t, int, const void* const*, int, float* const*, int, int) {
+    strata::metal_engine::missing("strata::kernels::cpu::iq512_reference_gu_rows");
+}
+void q50_512_down_rows(const uint8_t*, size_t, int, const void* const*, int, float* const*, int, int) {
+    strata::metal_engine::missing("strata::kernels::cpu::q50_512_down_rows");
+}
+
 }  // namespace strata::kernels::cpu

metal/src/engine/window.cpp

@@ -39,10 +39,13 @@ namespace strata::gpu { void* host_view(const void* device_pointer); }
 #include <cerrno>
 #include <filesystem>
 #include <functional>
+#include <future>
 #include <signal.h>
 #include <unistd.h>
 #include <map>
 #include <memory>
+#include <numeric>
+#include <set>
 #include <string>
 #include <sys/mman.h>
 #include <sys/sysctl.h>
@@ -168,9 +171,11 @@ struct PromptBufs {
     int lib = -1, lgdn = -1, lqsa = -1;
     bool k4 = false;
     bool v3e = false;
+    bool v6 = false;             // ragged expert tiles (mm_gu6 / mm_down6): groups of ms 16-row subtiles share one weight decode
+    int ms = 1;
     bool fused = false;          // the GDN input projections as one GEMM (gdn_fused_inputs)            // the M5's expert kernels (v3): not for the trellis types, which have only v2
-    int C = 0, maxrows = 0;
-    double gpu_ms = 0, host_ms = 0, mtp_ms = 0;
+    int C = 0, maxrows = 0, tm = 16;   // tm: the expert plan's tile height
+    double gpu_ms = 0, host_ms = 0, mtp_ms = 0, hidden_ms = 0, emb_ms = 0, ple_ms = 0;   // hidden_ms: the host rows computed beside the GPU (pc_host_rows)
     int gu3[4] = {}, dn3[4] = {};
     mtl::Buffer R, R2, bo, inj[2], xn, xnh, V, loh, mixed, mixh, qkv, z, al, be, yg, ygh, gqn, gkn, gvc, ggb, gkq, go, qfull,
         kcur, vcur, iqr, ikr, qcur, qcurh, qidx, scores, qids, part, XQ, xqh, logits, ids, wts, bcnt, boff, eoff, tiles, meta,
@@ -440,6 +445,16 @@ private:
     // originals wait one memory latency per loop iteration).  Bitwise the same outputs.  STRATA_METAL_LDB=0: the original kernels.
     bool ldb_ = !(std::getenv("STRATA_METAL_LDB") && std::string(std::getenv("STRATA_METAL_LDB")) == "0");
     const char* k_norm() const { return ldb_ ? "gr_norm2" : "gr_norm"; }
+    // STRATA_METAL_WPF_MIB=N (opt-in screen): while a hyper-connection read's last kernel runs, a prefetch dispatch (no barrier before it) reads the first N MiB of each
+    // weight the next kernels use (wpf_next_), so those kernels find them in the GPU's caches.  Bitwise: it changes no value.
+    std::vector<std::pair<uint64_t, uint64_t>> wpf_next_;   // (W_ offset, bytes)
+    mtl::Buffer wpf_flag_;
+    size_t wpf_bytes_ = [] { const char* e = std::getenv("STRATA_METAL_WPF_MIB"); return e ? (size_t) std::atoi(e) * 1048576 : (size_t) 0; }();
+    void wpf_add(const std::string& name) {
+        if (!wpf_bytes_) return;
+        const gguf::Tensor* t = f_.find(name);
+        if (t) wpf_next_.push_back({t->offset, std::min<uint64_t>(wpf_bytes_, t->bytes)});
+    }
     const char* k_comb() const { return ldb_ ? "moe_combine_sh2" : "moe_combine_sh"; }
     bool fuse_ = !(std::getenv("STRATA_METAL_FUSE") && std::string(std::getenv("STRATA_METAL_FUSE")) == "0");
     bool qdirty_ = false;
@@ -452,6 +467,12 @@ private:
     // STRATA_METAL_KPROF=T (test tooling): once, after the 20th cached window of T tokens with the head, the GPU time
     // of each kernel class alone (the window's own dispatches of that pipeline + weight role, run by themselves)
     void kprof(const std::vector<mtl::Dispatch>& ds, int Tn);
+    // STRATA_METAL_STEPPROF=T (measurement, round 0215): once, after the 20th cached head window of T tokens, where one decode step's GPU time
+    // goes IN SITU, by part: the window's dispatch list is run as prefixes ending at every barrier group (STRATA_METAL_STEPPROF_REPS runs each,
+    // default 5, median); a group's time is the difference of two prefixes, so the gaps between dispatches and the overlap of the groups that
+    // share a barrier are inside the numbers.  Also the weight bytes read per part (dense from the tensor table, experts from the routed ids of
+    // every layer: the distinct experts) and the GB/s each part reaches.  STRATA_METAL_STEPPROF_OUT=path: one line per group.
+    void stepprof(const std::vector<mtl::Dispatch>& ds, int Tn);
     void fold(std::vector<mtl::Dispatch>& ds, int n_keep, int hp);
     // the encoded window per (T, head, hp_): built once, then only the positions are patched (the host's encode cost)
     struct Cached {
@@ -602,7 +623,21 @@ private:
     int m5_layout_ = -1;   // the M5's matmul2d layout (coop_k4), asked once: the M4's tiles differ
     bool m5_layout() { if (m5_layout_ < 0) m5_layout_ = coop_k4() ? 1 : 0; return m5_layout_ == 1; }
     bool pc_setup(int C);
-    bool pc_chunk(const int64_t* tokens, int nT, int64_t pos0, std::string& err);
+    void pc_kprof(const std::vector<mtl::Dispatch>& ds, int nT, int64_t pos0);
+    // the chunk's host rows (the embedding rows and the PLE table rows): computed on a thread while the GPU runs the previous
+    // chunk (STRATA_METAL_PROMPT_HOSTPIPE=0 computes them inline before each chunk)
+    struct PcHost {
+        int nT = 0;
+        int64_t pos0 = -1;
+        std::vector<float> emb, ple;
+        double ms = 0, ms_emb = 0;
+        bool ok = true;
+        std::string err;
+    };
+    bool pc_host_rows(const int64_t* tokens, int nT, int64_t pos0, const int32_t prev_in[2], PcHost& out);
+    std::future<PcHost> pc_next_;
+    void pc_drain() { if (pc_next_.valid()) pc_next_.wait(); }
+    bool pc_chunk(const int64_t* tokens, int nT, int64_t pos0, std::string& err, const int64_t* next_tokens = nullptr, int next_nT = 0);
     // the draft round in one command buffer (msl/mtp_chain.metal): STRATA_METAL_MTP_CHAIN=0 runs a command buffer
     // per step with the host's pick and embedding (as before)
     int lchain_ = -1;
@@ -957,7 +992,8 @@ void Window::gr_read(std::vector<mtl::Dispatch>& ds, const std::string& pre, boo
     const bool q8 = is_q8(tdn) && is_q8(tup);
     const bool injf = q8 && !final_ && !q8off_.count(tin.name) && tin.type == ref::F32;
     mtl::Dispatch d2;
-    const char* dn = injf ? "gr_down_q8f_8_8" : q8 ? "gr_down_q8_8_8" : "gr_down_8_8";
+    static const bool hcld = std::getenv("STRATA_METAL_HCLD") && std::string(std::getenv("STRATA_METAL_HCLD")) == "1";   // OFF (measured -10.9%, 4 of 4 rounds, M5 Max UD-Q4_K_XL): the HC kernels' units loaded two or three at a time, bitwise the same; registers and occupancy, as the earlier gemvu / moeu unrolls
+    const char* dn = injf ? (hcld ? "gr_downp_q8f_8_8" : "gr_down_q8f_8_8") : q8 ? (hcld ? "gr_downp_q8_8_8" : "gr_down_q8_8_8") : "gr_down_8_8";
     d2.p = P(lib_, dn, mtl::Constants().i(1, ntok), std::string(dn) + "#" + std::to_string(ntok));
     d2.args = {mtl::Arg::B(*bd, od), mtl::Arg::B(*bi, oi), mtl::Arg::B(xn_), mtl::Arg::B(lo_),
                mtl::Arg::B(inj_[1 - cur_inj_]), mtl::Arg::V(a)};
@@ -965,7 +1001,7 @@ void Window::gr_read(std::vector<mtl::Dispatch>& ds, const std::string& pre, boo
     d2.tg[0] = 256;
     push(ds, d2);
     mtl::Dispatch d3;
-    const char* up = q8 ? "gr_up_q8" : "gr_up";
+    const char* up = q8 ? (hcld ? "gr_upp_q8" : "gr_up_q8") : "gr_up";
     d3.p = P(lib_, up, x16c(mtl::Constants().i(1, ntok)), x16k(std::string(up) + "#" + std::to_string(ntok)));
     GrArgs au = a;
     au.n_tok = MAXT;   // the plane stride of the prepared activation
@@ -973,6 +1009,18 @@ void Window::gr_read(std::vector<mtl::Dispatch>& ds, const std::string& pre, boo
     d3.grid[0] = N / 16;
     d3.tg[0] = 256;
     push(ds, d3);
+    for (const auto& [off, bytes] : wpf_next_) {
+        if (!wpf_flag_.handle) wpf_flag_ = cx_->alloc(4096);
+        mtl::Dispatch p;
+        p.p = P(lib_, "wpf", {}, "wpf");
+        const uint32_t n16 = (uint32_t) (bytes / 16);
+        p.args = {mtl::Arg::B(W_, off), mtl::Arg::B(wpf_flag_), mtl::Arg::V(n16)};
+        p.tg[0] = 256;
+        p.grid[0] = std::max<uint32_t>(1, std::min<uint32_t>(n16 / (256 * 4), 4096));
+        nb_ = true;   // beside gr_up
+        push(ds, p);
+    }
+    wpf_next_.clear();
     cur_inj_ = 1 - cur_inj_;
 }
 
@@ -1008,6 +1056,7 @@ void Window::had_rows_ind(std::vector<mtl::Dispatch>& ds, int lib, const std::st
 void Window::layer(std::vector<mtl::Dispatch>& ds, int l, int ntok, int64_t pos0) {
     const std::string b = "blk." + std::to_string(l) + ".";
     // ---- the mixer: layer 1's read after the PLE is not applied again (its pending write was applied before it)
+    if (l % 4 != 3) { wpf_add(b + "attn_qkv.weight"); wpf_add(b + "attn_gate.weight"); } else wpf_add(b + "attn_q.weight");
     gr_read(ds, b + "hc_attn", l > 0 && !(l == 1 && ple_on_), XA_, false, ntok);
     if (l % 4 != 3) {
         const int gi = l - l / 4;
@@ -1140,6 +1189,7 @@ void Window::layer(std::vector<mtl::Dispatch>& ds, int l, int ntok, int64_t pos0
         gemv(ds, T(b + "attn_output.weight"), XQ_, bo_, N, ntok);
     }
     // ---- the MoE
+    wpf_add(b + "ffn_gate_inp.weight"); wpf_add(b + "ffn_gate_shexp.weight"); wpf_add(b + "ffn_up_shexp.weight");
     gr_read(ds, b + "hc_ffn", true, XF_, false, ntok);
     const gguf::Tensor& trt = T(b + "ffn_gate_inp.weight");
     const bool rf = routef_ && (trt.type == ref::BF16 || trt.type == ref::F32);
@@ -1189,12 +1239,22 @@ void Window::layer(std::vector<mtl::Dispatch>& ds, int l, int ntok, int64_t pos0
         // the trellis gate/up: 16 lanes a row, 2 rows a lane (expert_parity --bench --sweep, M4, TQK6, 10 experts: 166.6 us a layer
         // against 174.5 for 32,2,4 and 196.8 for 32,1,4; also the fastest at 2 and 3 tokens)
         const bool tq_ = tg_.type >= 144;
-        const int gl = tq_ ? 16 : 32, gr = tq_ ? 2 : 1, gs = 4, ds_ = 4;
+        int gl = tq_ ? 16 : 32, gr = tq_ ? 2 : 1, gs = 4, ds_ = 4;
         // the down tile: Q2_0 8,1; else 4,2 - but on the M4 (no M5 layout) Q5_1 4,1 and Q8_0 8,1 (4,2 spills there:
         // 24-36 GB/s, mac-kern 513fc615)
         const bool m4 = !m5_layout();
-        const int dl = dq2 || (m4 && td.type == ref::Q8_0) ? 8 : 4,
-                  dr = dq2 || (m4 && (td.type == ref::Q8_0 || td.type == ref::Q5_1 || td.type == ref::Q5_0)) ? 1 : 2;
+        int dl = dq2 || (m4 && td.type == ref::Q8_0) ? 8 : 4,
+            dr = dq2 || (m4 && (td.type == ref::Q8_0 || td.type == ref::Q5_1 || td.type == ref::Q5_0)) ? 1 : 2;
+        {   // STRATA_METAL_MOE_GU=L,R,S / STRATA_METAL_MOE_DN=L,R,S (tile screens: lanes per row, rows per lane, simdgroups; an instantiated moe_gu_ / moe_down_ combination)
+            static const auto parse = [](const char* name, int* o) {
+                const char* e = std::getenv(name);
+                return e && std::sscanf(e, "%d,%d,%d", &o[0], &o[1], &o[2]) == 3;
+            };
+            static int gu_o[3], dn_o[3];
+            static const bool gu_e = parse("STRATA_METAL_MOE_GU", gu_o), dn_e = parse("STRATA_METAL_MOE_DN", dn_o);
+            if (gu_e && !tq_) { gl = gu_o[0]; gr = gu_o[1]; gs = gu_o[2]; }
+            if (dn_e && !dq2) { dl = dn_o[0]; dr = dn_o[1]; ds_ = dn_o[2]; }
+        }
         const bool m16 = f16_enc_ && !stream_ && tg_.type == ref::Q2_0 && dq2 && !had_gu(l) && !had_dn(l);
         a.gx_gu = NFF / (gs * (32 / gl) * gr);
         a.gx_dn = N / (ds_ * (32 / dl) * dr);
@@ -1405,6 +1465,10 @@ bool Window::run(int Tn, const int32_t* tokens, int64_t pos0, bool head, int32_t
             static const int kt = std::getenv("STRATA_METAL_KPROF") ? std::atoi(std::getenv("STRATA_METAL_KPROF")) : 0;
             static int seen = 0;
             if (kt == Tn && head && ++seen == 20 && !stream_) kprof(c.ds, Tn);
+            static const int st = std::getenv("STRATA_METAL_STEPPROF") ? std::atoi(std::getenv("STRATA_METAL_STEPPROF")) : 0;
+            static int seen2 = 0;
+            static const int st_at = std::getenv("STRATA_METAL_STEPPROF_AT") ? std::atoi(std::getenv("STRATA_METAL_STEPPROF_AT")) : 20;
+            if (st == Tn && head && ++seen2 == st_at && !stream_) stepprof(c.ds, Tn);
         }
         tp_rg_ = now_ms() - r0;
         last_n_disp_ = (int) c.ds.size();
@@ -1749,6 +1813,146 @@ void Window::kprof(const std::vector<mtl::Dispatch>& ds, int Tn) {
     for (const auto& [m, c] : rows) std::fprintf(stderr, "strata metal kprof: %8.3f ms %5.1f%%  %s\n", m, 100.0 * m / sum, c.c_str());
 }
 
+void Window::stepprof(const std::vector<mtl::Dispatch>& ds, int Tn) {
+    std::map<const mtl::Pipeline*, std::string> pname;
+    for (const auto& [k, v] : pcache_) pname[&v] = k.substr(0, k.find('#'));
+    std::map<uint64_t, const gguf::Tensor*> wt;       // W_ offset -> tensor
+    for (const auto& [k, v] : f_.tensors) wt[v.offset] = &v;
+    std::map<size_t, const gguf::Tensor*> qt;         // Q8_ offset -> tensor (the load-time Q8_0 copies)
+    for (const auto& [k, v] : q8off_) { auto it = f_.tensors.find(k); if (it != f_.tensors.end()) qt[v] = &it->second; }
+    auto role = [](const std::string& n) { return n.rfind("blk.", 0) == 0 ? n.substr(n.find('.', 4) + 1) : n; };
+    const size_t n = ds.size();
+    struct Info { std::string part; double wbytes = 0; };
+    std::vector<Info> info(n);
+    for (size_t i = 0; i < n; ++i) {
+        const auto& d = ds[i];
+        const std::string c = pname.count(d.p) ? pname[d.p] : "?";
+        std::string part = "other", tn;
+        double wb = 0;
+        auto wbuf = [&](size_t a) -> const gguf::Tensor* {
+            if (a >= d.args.size() || !d.args[a].buf) return nullptr;
+            if (d.args[a].buf == &W_) { auto it = wt.find(d.args[a].off); return it == wt.end() ? nullptr : it->second; }
+            if (d.args[a].buf == &Q8_) { auto it = qt.find(d.args[a].off); return it == qt.end() ? nullptr : it->second; }
+            return nullptr;
+        };
+        const gguf::Tensor* t0 = wbuf(0);
+        if (t0) tn = role(t0->name);
+        auto tsize = [&](size_t a) -> double {
+            const gguf::Tensor* t = wbuf(a);
+            if (!t) return 0;
+            if (d.args[a].buf == &Q8_) return (double) (t->shape[0] / 32 * 34) * (t->shape.size() > 1 ? (double) t->shape[1] : 1.0);
+            return (double) t->bytes;
+        };
+        if (c.rfind("moe_gu", 0) == 0) part = "experts gate+up";
+        else if (c.rfind("moe_down", 0) == 0) part = "experts down";
+        else if (c.rfind("moe_route", 0) == 0 || c.rfind("router_route", 0) == 0) part = "router";
+        else if (c.rfind("moe_combine", 0) == 0) part = "combine";
+        else if (c.rfind("gr_", 0) == 0) { part = "hc"; wb = c.rfind("gr_down", 0) == 0 ? tsize(0) + tsize(1) : c.rfind("gr_up", 0) == 0 ? tsize(0) : 0; }
+        else if (c.rfind("gdn_", 0) == 0) part = "gdn other";
+        else if (c.rfind("qsa_", 0) == 0) part = "qsa other";
+        else if (c.rfind("ple_", 0) == 0) part = "ple";
+        else if (c.rfind("argmax", 0) == 0) part = "lm head (argmax)";
+        else if (c.rfind("swiglu_prep", 0) == 0) part = "shared";
+        else if (c.rfind("xprep", 0) == 0) part = "xprep";
+        else if (c.rfind("gemv", 0) == 0 && t0) {
+            wb = tsize(0);
+            if (tn == "attn_qkv.weight" || tn == "attn_gate.weight" || tn == "ssm_alpha.weight" || tn == "ssm_beta.weight" || tn == "ssm_out.weight") part = "gdn dense";
+            else if (tn == "attn_q.weight" || tn == "attn_k.weight" || tn == "attn_v.weight" || tn == "attn_output.weight" || tn.rfind("indexer.", 0) == 0) part = "qsa dense";
+            else if (tn == "ffn_gate_inp.weight") part = "router";
+            else if (tn.find("shexp") != std::string::npos) part = "shared";
+            else if (tn == "output.weight" || t0->name == "output.weight") part = "lm head";
+            else if (tn.find("ple_") != std::string::npos) part = "ple";
+            else part = "gemv other " + tn;
+        }
+        info[i] = {part, wb};
+    }
+    // barrier groups
+    std::vector<size_t> gs;
+    for (size_t i = 0; i < n; ++i) if (i == 0 || ds[i].barrier) gs.push_back(i);
+    gs.push_back(n);
+    const size_t m = gs.size() - 1;
+    // the route dispatches (one per layer): the prefix that ends right after one holds that layer's routed ids
+    std::vector<size_t> routes;
+    for (size_t i = 0; i < n; ++i)
+        if (pname.count(ds[i].p) && (pname[ds[i].p].rfind("moe_route", 0) == 0 || pname[ds[i].p].rfind("router_route", 0) == 0)) routes.push_back(i);
+    const int reps = std::getenv("STRATA_METAL_STEPPROF_REPS") ? std::max(1, std::atoi(std::getenv("STRATA_METAL_STEPPROF_REPS"))) : 5;
+    std::fprintf(stderr, "strata metal stepprof: T%d, %zu dispatches in %zu barrier groups, %zu layers routed, %d reps per prefix\n", Tn, n, m, routes.size(), reps);
+    for (int i = 0; i < 40; ++i) cx_->run_graph(ds);   // clocks up
+    const double full = [&] { std::vector<double> t; for (int i = 0; i < 15; ++i) t.push_back(1000.0 * cx_->run_graph(ds)); std::sort(t.begin(), t.end()); return t[7]; }();
+    std::vector<std::vector<double>> tp(m + 1);
+    // the GPU heats up during the hundreds of runs: every pass is scaled by (the cool whole-window time / the whole window run at the start of that pass)
+    for (int r = 0; r < reps; ++r) {
+        double fr = 0;
+        { std::vector<double> q; for (int i = 0; i < 5; ++i) q.push_back(1000.0 * cx_->run_graph(ds)); std::sort(q.begin(), q.end()); fr = q[2]; }
+        const double sc = full / fr;
+        tp[0].push_back(sc * 1000.0 * cx_->run_graph(ds, 0));
+        for (size_t j = 1; j <= m; ++j) tp[j].push_back(sc * 1000.0 * cx_->run_graph(ds, gs[j]));
+        std::fprintf(stderr, "strata metal stepprof: pass %d: whole window %.3f ms (cool %.3f, scale %.3f)\n", r, fr, full, sc);
+    }
+    std::vector<double> t(m + 1);
+    for (size_t j = 0; j <= m; ++j) { std::sort(tp[j].begin(), tp[j].end()); t[j] = tp[j][tp[j].size() / 2]; }
+    // distinct experts per layer
+    std::vector<int> distinct;
+    double expert_total = 0;
+    for (size_t li = 0; li < routes.size(); ++li) {
+        cx_->run_graph(ds, routes[li] + 1);
+        std::set<int> u;
+        for (int k = 0; k < Tn * TOPK; ++k) { const int v = ids_.as<int32_t>()[k]; if (v >= 0) u.insert(v); }
+        distinct.push_back((int) u.size());
+    }
+    auto expert_bytes = [&](size_t layer, bool gu, bool dn) {
+        const std::string L = "blk." + std::to_string(layer) + ".";
+        auto eb = [&](const char* nm) { const gguf::Tensor* tt = f_.find(L + nm); return tt ? (double) tt->bytes / std::max(1, nexp_) : 0.0; };
+        double b = 0;
+        if (gu) b += eb("ffn_gate_exps.weight") + eb("ffn_up_exps.weight");
+        if (dn) b += eb("ffn_down_exps.weight");
+        return b * (layer < distinct.size() ? distinct[layer] : 0);
+    };
+    for (size_t li = 0; li < distinct.size(); ++li) expert_total += expert_bytes(li, true, true);
+    // per group: time, part (the experts if the group has any, else the dispatch with the most weight bytes, else the first), bytes
+    struct Row { double ms = 0, bytes = 0; int groups = 0, disp = 0; };
+    std::map<std::string, Row> by;
+    FILE* of = std::getenv("STRATA_METAL_STEPPROF_OUT") ? std::fopen(std::getenv("STRATA_METAL_STEPPROF_OUT"), "w") : nullptr;
+    size_t layer_gu = 0, layer_dn = 0;
+    for (size_t j = 0; j < m; ++j) {
+        const double ms = t[j + 1] - t[j];
+        size_t best = gs[j];
+        double bw = -1;
+        for (size_t i = gs[j]; i < gs[j + 1]; ++i) if (info[i].wbytes > bw) { bw = info[i].wbytes; best = i; }
+        std::string part = bw > 0 ? info[best].part : info[gs[j]].part;
+        double bytes = 0;
+        bool has_gu = false, has_dn = false;
+        for (size_t i = gs[j]; i < gs[j + 1]; ++i) {
+            bytes += info[i].wbytes;
+            if (info[i].part == "experts gate+up") has_gu = true;
+            if (info[i].part == "experts down") has_dn = true;
+        }
+        if (has_gu) { part = "experts gate+up"; bytes += expert_bytes(layer_gu++, true, false); }
+        if (has_dn) { part = has_gu ? part : "experts down"; bytes += expert_bytes(layer_dn++, false, true); }
+        Row& r = by[part];
+        r.ms += ms; r.bytes += bytes; r.groups += 1; r.disp += (int) (gs[j + 1] - gs[j]);
+        if (of) std::fprintf(of, "%zu\t%zu\t%s\t%.4f\t%.0f\t%s\n", j, gs[j + 1] - gs[j], part.c_str(), ms, bytes, pname.count(ds[gs[j]].p) ? pname[ds[gs[j]].p].c_str() : "?");
+    }
+    if (of) std::fclose(of);
+    double sum = 0, sumb = 0;
+    for (auto& [k, r] : by) { sum += r.ms; sumb += r.bytes; }
+    std::fprintf(stderr, "strata metal stepprof: window T%d: whole-window GPU %.3f ms (median of 15), prefix chain %.3f ms (empty-list base %.3f ms), weight bytes %.1f MiB (experts %.1f MiB, distinct per layer avg %.2f)\n",
+                 Tn, full, t[m] - t[0], t[0], sumb / 1048576.0, expert_total / 1048576.0,
+                 distinct.empty() ? 0.0 : std::accumulate(distinct.begin(), distinct.end(), 0.0) / (double) distinct.size());
+    std::vector<std::pair<double, std::string>> rows;
+    for (auto& [k, r] : by) rows.push_back({r.ms, k});
+    std::sort(rows.rbegin(), rows.rend());
+    for (const auto& [ms, k] : rows) {
+        const Row& r = by[k];
+        std::fprintf(stderr, "strata metal stepprof: %-22s %8.3f ms %5.1f%%  %5d disp %5d groups  %9.1f MiB  %7.1f GB/s\n", k.c_str(), ms, 100.0 * ms / sum, r.disp, r.groups,
+                     r.bytes / 1048576.0, r.bytes > 0 && ms > 0 ? r.bytes / 1e9 / (ms / 1000.0) : 0.0);
+    }
+    std::string ds_str;
+    for (size_t i = 0; i < distinct.size(); ++i) ds_str += std::to_string(distinct[i]) + (i + 1 < distinct.size() ? "," : "");
+    std::fprintf(stderr, "strata metal stepprof: distinct experts per layer: %s\n", ds_str.c_str());
+    std::fflush(stderr);
+}
+
 Window::~Window() {
     df2_free();
     if (logpos_) std::fclose(logpos_);

metal/src/mtl_rt.h

@@ -121,7 +121,7 @@ public:
     double run(const std::vector<Dispatch>& ds, int reps = 1, bool concurrent = false, bool barrier_between = false);
     // A dependency graph in ONE concurrent encoder: a memory barrier only before dispatches with `barrier` set, so
     // the dispatches between two barriers may overlap.  Returns the GPU time in seconds.
-    double run_graph(const std::vector<Dispatch>& ds);
+    double run_graph(const std::vector<Dispatch>& ds, size_t count = (size_t) -1);   // count: only the first `count` dispatches (the step profile's prefixes)
     // run_graph on a SECOND queue, committed without waiting: the GPU may overlap it with work on the main queue (the two must touch
     // different buffers).  Returns a token; wait_async(token) blocks until it has completed (returns its GPU time in seconds).
     void* run_graph_async(const std::vector<Dispatch>& ds);

metal/src/mtl_rt.mm

@@ -992,14 +992,15 @@ double Context::run(const std::vector<Dispatch>& ds, int reps, bool concurrent,
     }
 }
 
-double Context::run_graph(const std::vector<Dispatch>& ds) {
+double Context::run_graph(const std::vector<Dispatch>& ds, size_t count) {
     // The wait spins on the buffer's status for up to 20 ms before it blocks: waitUntilCompleted's thread wake costs ~0.5 ms
     // a window (M5 Max, Q2_0 decode: +5% tok/s, same tokens).  STRATA_METAL_SPIN=0: the blocking wait.
     static const bool spin = !(std::getenv("STRATA_METAL_SPIN") && std::string(std::getenv("STRATA_METAL_SPIN")) == "0");
     @autoreleasepool {
         id<MTLCommandBuffer> cb = [impl_->q commandBufferWithUnretainedReferences];
         id<MTLComputeCommandEncoder> enc = [cb computeCommandEncoderWithDispatchType:MTLDispatchTypeConcurrent];
-        for (size_t i = 0; i < ds.size(); ++i) {
+        const size_t nd = std::min(count, ds.size());
+        for (size_t i = 0; i < nd; ++i) {
             if (i > 0 && ds[i].barrier) [enc memoryBarrierWithScope:MTLBarrierScopeBuffers];
             encode(enc, ds[i]);
         }

metal/tests/mm_i8_bench.cpp

@@ -0,0 +1,50 @@
+// metal/tests/mm_i8_bench.cpp - matmul2d throughput for int8 x int8 -> int32 and fp16 -> fp16 against fp16 -> fp32 (4096^3 and a
+// prompt-sized dense shape).  Answers: does the M5's matrix unit run int8 faster than fp16?
+//   mm_i8_bench
+#include "mtl_rt.h"
+
+#include <algorithm>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <string>
+#include <vector>
+
+namespace {
+struct MmArgs { int M, N, K; };
+double median(std::vector<double> v) { std::sort(v.begin(), v.end()); return v[v.size() / 2]; }
+}  // namespace
+
+int main() {
+    mtl::Context cx;
+    std::printf("device %s\n", cx.device_name().c_str());
+    const int lib = cx.compile({"mm_i8.metal"}, "#define NO_GGML_COMMON 1\n");
+    const int MAX = 4096;
+    mtl::Buffer A = cx.alloc((size_t) MAX * MAX * 2), B = cx.alloc((size_t) MAX * MAX * 2), C = cx.alloc((size_t) MAX * MAX * 4);
+    for (size_t i = 0; i < (size_t) MAX * MAX * 2; ++i) { A.as<uint8_t>()[i] = (uint8_t) (i * 2654435761u >> 24); B.as<uint8_t>()[i] = (uint8_t) (i * 40503u >> 8); }
+    struct K { std::string name; int tm, tn, sg; };
+    const std::vector<K> ks = {{"mpp_char_int_64_64_4", 64, 64, 4}, {"mpp_char_int_128_64_4", 128, 64, 4}, {"mpp_char_int_64_128_4", 64, 128, 4},
+                               {"mpp_half_half_64_64_4", 64, 64, 4}, {"mpp_half_half_128_64_4", 128, 64, 4},
+                               {"mpp_half_float_64_64_4", 64, 64, 4}, {"mpp_half_float_128_64_4", 128, 64, 4}};
+    struct Shape { const char* name; int M, N, K; };
+    const std::vector<Shape> shapes = {{"square 4096^3", 4096, 4096, 4096}, {"dense 2048 x 4096 x 2560", 2048, 4096, 2560}};
+    for (const K& k : ks) {
+        mtl::Pipeline p = cx.pipeline(lib, k.name);
+        for (const Shape& s : shapes) {
+            MmArgs a{s.M, s.N, s.K};
+            mtl::Dispatch d;
+            d.p = &p;
+            d.args = {mtl::Arg::B(A), mtl::Arg::B(B), mtl::Arg::B(C), mtl::Arg::V(a)};
+            d.grid[0] = s.N / k.tn;
+            d.grid[1] = s.M / k.tm;
+            d.tg[0] = 32 * k.sg;
+            std::vector<mtl::Dispatch> ds(std::max(1, (int) (4e10 / (2.0 * s.M * s.N * s.K))), d);
+            cx.warm(ds, 0.2);
+            std::vector<double> t;
+            for (int i = 0; i < 5; ++i) t.push_back(cx.run(ds) / ds.size());
+            const double m = median(t);
+            std::printf("%-26s %-28s %8.1f us  %6.2f T(FL)OP/s\n", k.name.c_str(), s.name, m * 1e6, 2.0 * s.M * s.N * s.K / m / 1e12);
+        }
+    }
+    return 0;
+}

metal/tests/moe_mm_parity.cpp

@@ -32,7 +32,7 @@ float h16(float v) { return ref::f16_to_f32(ref::f32_to_f16(v)); }
 
 int main(int argc, char** argv) {
     int gu = ref::Q2_0, dn = ref::Q2_0, T = 256, TM = 16, TN = 64, TK = 64, SG = 2, DTN = -1, DTK = -1, DSG = -1;
-    bool bench = false, v2 = false, v3 = false, v4 = false, v5 = false;
+    bool bench = false, v2 = false, v3 = false, v4 = false, v5 = false, v6 = false;
     int MS = 1;
     for (int i = 1; i < argc; ++i) {
         std::string a = argv[i];
@@ -45,6 +45,7 @@ int main(int argc, char** argv) {
         else if (a == "--v2") v2 = true;
         else if (a == "--v3") v2 = v3 = true;   // v2's gather + the register-operand GEMMs (TN per simdgroup)
         else if (a == "--v4") { v2 = v3 = v4 = true; MS = std::atoi(argv[++i]); }   // mm_gu4 with MS 16-row subtiles
+        else if (a == "--v6") { v2 = v3 = true; v6 = true; MS = std::atoi(argv[++i]); }   // mm_gu6 / mm_down6 / mm_scan6: groups of MS 16-row subtiles
         else if (a == "--v5") { v2 = v3 = true; v5 = true; }   // mm_gu5 (weight blocks staged in threadgroup memory)
     }
     if (bench && T == 256) T = 2048;
@@ -52,6 +53,7 @@ int main(int argc, char** argv) {
     // It has no IQ4_XS gate/up, so that type runs the v2 path here instead of failing at the pipeline lookup.
     if (!v2 && gu == ref::IQ4_XS) { v2 = true; std::printf("note: IQ4_XS gate/up has no v1 kernel (the engine does not use v1): running v2\n"); }
     if (v4) TM = 16 * MS;   // the plan's tile height; mm_gu4's name carries MS instead
+    if (v6) TM = 16;   // ragged tiles: the padding is 16 rows, the group size is MS
     mtl::Context cx;
     const int lib = cx.compile({"quant.metal", "units.metal", "mma.metal", "moe_mm.metal"});
     const int NE = bench ? 512 : 64, LAYERS = bench ? 4 : 1;
@@ -90,26 +92,33 @@ int main(int argc, char** argv) {
         }
         for (int k = 0; k < TOPK; ++k) w.as<float>()[t * TOPK + k] /= s;
     }
-    mtl::Pipeline phist = cx.pipeline(lib, "mm_hist"), pscan = cx.pipeline(lib, "mm_scan"), pplace = cx.pipeline(lib, "mm_place"),
+    mtl::Pipeline phist = cx.pipeline(lib, "mm_hist"), pscan = cx.pipeline(lib, v6 ? "mm_scan6" : "mm_scan"), pplace = cx.pipeline(lib, "mm_place"),
                   pclear = cx.pipeline(lib, "mm_clear"), pcomb = cx.pipeline(lib, "mm_combine");
     if (DTN < 0) { DTN = TN; DTK = TK; DSG = SG; }
     if ((dn == ref::IQ4_NL || dn == ref::Q5_0) && DTK < 32) DTK = 32;   // 32-value units (the v2 / v3 down tiles of IQ4_NL / Q5_0 need TK >= 32)
     const std::string tl = std::to_string(TM) + "_" + std::to_string(TN) + "_" + std::to_string(TK) + "_" + std::to_string(SG);
     const std::string dtl = std::to_string(TM) + "_" + std::to_string(DTN) + "_" + std::to_string(DTK) + "_" + std::to_string(DSG);
+    std::string v6_gu, v6_dn;
     mtl::Constants probe;
     probe.i(5, std::getenv("MM_PROBE") ? std::atoi(std::getenv("MM_PROBE")) : 0);
     const bool k4 = coop_k4(cx);
     if (v3) std::printf("right-input layout: %s\n", k4 ? "runs of 4 k" : "other (per-element decode)");
-    mtl::Pipeline pgu = v5 ? cx.pipeline(lib, "mm_gu5_" + std::to_string(gu) + "_" + std::to_string(TN) + "_" + std::to_string(SG),
+    if (v6) {   // the ragged-tile kernels: names carry MS, TN, TK, SG (TM is the plan's 16-row padding)
+        const std::string tl6 = std::to_string(MS) + "_" + std::to_string(TN) + "_" + std::to_string(TK) + "_" + std::to_string(SG);
+        const std::string dtl6 = std::to_string(MS) + "_" + std::to_string(DTN) + "_" + std::to_string(DTK) + "_" + std::to_string(DSG);
+        v6_gu = "mm_gu6_" + std::to_string(gu) + "_" + tl6;
+        v6_dn = "mm_down6_" + std::to_string(dn) + "_" + dtl6;
+    }
+    mtl::Pipeline pgu = v6 ? cx.pipeline(lib, v6_gu, mtl::Constants().b(7, k4)) : v5 ? cx.pipeline(lib, "mm_gu5_" + std::to_string(gu) + "_" + std::to_string(TN) + "_" + std::to_string(SG),
                                          mtl::Constants().b(7, k4))
                       : v4 ? cx.pipeline(lib, "mm_gu4_" + std::to_string(gu) + "_" + std::to_string(MS) + tl.substr(tl.find('_')),
                                          mtl::Constants().b(7, k4).i(5, std::getenv("MM_PROBE") ? std::atoi(std::getenv("MM_PROBE")) : 0))
                       : v3 ? cx.pipeline(lib, "mm_gu3_" + std::to_string(gu) + "_" + tl, mtl::Constants().b(7, k4).i(5, std::getenv("MM_PROBE") ? std::atoi(std::getenv("MM_PROBE")) : 0).b(23, std::getenv("QUANT_CTAB") != nullptr))
                            : cx.pipeline(lib, (v2 ? "mm_gu2_" : "mm_gu_") + std::to_string(gu) + "_" + tl, probe),
-                  pdn = v3 ? cx.pipeline(lib, "mm_down3_" + std::to_string(dn) + "_" + dtl, mtl::Constants().b(7, k4))
+                  pdn = v6 ? cx.pipeline(lib, v6_dn, mtl::Constants().b(7, k4)) : v3 ? cx.pipeline(lib, "mm_down3_" + std::to_string(dn) + "_" + dtl, mtl::Constants().b(7, k4))
                            : cx.pipeline(lib, (v2 ? "mm_down2_" : "mm_down_") + std::to_string(dn) + "_" + (v2 ? dtl : tl)),
                   pgat = cx.pipeline(lib, "mm_gather");
-    MmPlanArgs pa{(uint32_t)nent, TOPK, (uint32_t)NE, (uint32_t)TM, (uint32_t)nblk};
+    MmPlanArgs pa{(uint32_t)nent, TOPK, (uint32_t)NE, (uint32_t)(v6 ? MS : TM), (uint32_t)nblk};
     auto encode = [&](int layer, std::vector<mtl::Dispatch>& ds, bool plan, bool gemm) {
         if (plan) {
             const uint32_t nclr = maxrows;
@@ -205,6 +214,9 @@ int main(int argc, char** argv) {
             se += (g - yr[i]) * (g - yr[i]);
             sr += yr[i] * yr[i];
         }
+        uint64_t yh = 14695981039346656037ull;   // FNV-1a over y's bits: two runs (v3 TM 16, v6 groups) must print the same hash
+        for (size_t i = 0; i < yr.size(); ++i) { uint32_t u; std::memcpy(&u, y.as<float>() + i, 4); yh = (yh ^ u) * 1099511628211ull; }
+        std::printf("yhash %016llx\n", (unsigned long long) yh);
         const double rel16 = std::sqrt(se16 / sr), rel = std::sqrt(se / sr);
         const bool ok = ok_plan && rel16 < 2e-4;
         std::printf("parity: plan %s; y vs fp16-operand reference rel L2 %.2e, vs exact float reference %.2e  %s\n",

packaging/UPDATES.md

@@ -50,6 +50,9 @@ Comments are for this document only; actual feed files must be strict JSON.
     "manifest_sha256": "...",               // of the package's engine-component.json (it holds every file's hash)
     "installed_bytes": 840893419, "notes": ""}]},
   "dsv41": {"packages": []}                 // optional: the separate DeepSeek engine packages, same entry shape as "engine"
+  "drafters": {"packages": [{               // optional: DFlash2 drafter files Strata hosts (one pinned file each, not an archive)
+    "name": "qwen3.8-flash-next-dflash2.strata1.q4_0-selq4.pack", "version": "1", "url": "qwen3.8-flash-next-dflash2.strata1.q4_0-selq4.pack",
+    "size": 339694592, "sha256": "4df2e5ab...", "backends": ["metal"], "base": "incoai/Qwen3.8-Flash-Next-DFlash2@a22a1744", "licence": "Apache-2.0"}]}
 }
 ```
 
@@ -58,6 +61,17 @@ The DeepSeek V4.1-Flash engine `strata-dsv41` reaches a PC in one of two ways. T
 `pack-engine --prefix strata-dsv41`. The separate package installs in `<data>/engine-dsv41/` beside the engine, never inside it
 (`strata_control/dsv41_engine.py`), and is fetched when DeepSeek is chosen. `STRATA_DSV41_EXE` points setup at a build of your own.
 
+The `drafters` member lists the tuned DFlash2 drafter (`strata_control/drafting.py` TUNED_NAME: Strata's fine-tuned copy of incoai/Qwen3.8-Flash-Next-DFlash2, Apache-2.0,
+modified by Strata). Control fetches it for the Mac sets that have a measured entry (`SET_DEFAULTS`: UD-Q4_K_XL and UD-IQ4_XS-v31) during setup, `strata launch` and
+`strata drafting fetch`: it reads this signed feed, takes the entry named TUNED_NAME that lists `metal`, and downloads `url` (resumable) into `<data>/dflash2/`. The size and SHA-256
+are pinned in the code too, so the feed can move the file but a feed that lists other bytes is ignored. No feed answer (offline, host down) or no such entry: nothing is said and the
+model keeps the MTP. Make the entry with `feed_tool.py pack-drafter PACK --out RELEASE/drafters --version 1 --backends metal [--url-prefix ""]` (copies the file flat and writes
+`<name>.entry.json`) and pass the folder to `build-feed --drafters RELEASE/drafters`; upload the pack beside the feed in the same release. Older Controls ignore the member.
+
+The member holds two entries (one `pack-drafter` call each, both folders' `.entry.json` files in the same `--drafters` folder): `qwen3.8-flash-next-dflash2.strata1.q4_0-selq4.pack` (339,694,592 bytes) with
+`--backends metal,cuda,hip`, and `qwen3.8-flash-next-dflash2.strata1.q4_0-bf16sel.pack` (522,458,112 bytes, sha256 34ca17fe...) with `--backends sycl` for the Intel Arc engine, which reads only bf16
+selector codebooks. Control asks the feed for the file name its table entry (`SET_DEFAULTS`) names and for its own backend, so no platform fetches the other's file.
+
 `url` is relative to the feed's own location (or an absolute `https://` address; a loopback `http://` is allowed so a local test
 feed works; a local feed folder may only name files inside itself).
 For a GitHub release the assets are flat: pack with `--url-prefix ""` and upload every file beside the feed.
@@ -161,7 +175,7 @@ All signing happens on the release PC, on the finished files.
     python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target windows-x86_64 --version 0.2.5 --archive Strata-0.2.9-windows-x64-setup.exe --sig Strata-0.2.9-windows-x64-setup.exe.sig --notes "..."
     python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target linux-x86_64   --version 0.2.5 --archive Strata-0.2.9-linux-x86_64.AppImage --sig Strata-0.2.9-linux-x86_64.AppImage.sig
     python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target darwin-aarch64 --version 0.2.5 --archive Strata.app.tar.gz --sig Strata.app.tar.gz.sig
-    python packaging/feed_tool.py build-feed --out RELEASE --sequence N --key <feed-key.json> --engine RELEASE/engine --control RELEASE/control --app-json RELEASE/tauri-latest.json [--dsv41 RELEASE/dsv41]
+    python packaging/feed_tool.py build-feed --out RELEASE --sequence N --key <feed-key.json> --engine RELEASE/engine --control RELEASE/control --app-json RELEASE/tauri-latest.json [--dsv41 RELEASE/dsv41] [--drafters RELEASE/drafters]
     python packaging/feed_tool.py verify-feed RELEASE --pub <feed-key.pub.json>
 
 `set-app-platform` checks each `.sig` against the public key in `tauri.conf.json` over the exact bytes, so a file signed with another key, or changed after

packaging/build_dsv41_hip_linux.sh

@@ -6,7 +6,7 @@
 #   VARIANT=rx7900     rx9070 (gfx1200 gfx1201) | rx7900 (gfx1100 gfx1101 gfx1102: built, NOT tested on a real RX 7000 card) | a name + ARCHS=
 #   ARCHS="gfx1100;gfx1101;gfx1102"   the code objects (default: the variant's); BUILD.json "archs" and the manifest list exactly these
 #   SRC=<this checkout>  LLAMA=<llama.cpp checkout at the pinned commit>  HP=<ROCm devel root>  CORE=<ROCm core root>  LIBW=<ROCm libraries root>
-#   OUT=<work folder>  PKGVER=0.1.57  COMMIT=<source commit for the build id>  JOBS=16
+#   OUT=<work folder>  PKGVER=0.1.58  COMMIT=<source commit for the build id>  JOBS=16
 #   NO_PACKAGE=1       stop after the stage folder
 #
 # Output: $OUT/components/dsv41/<ver>-linux-x86_64-hip-<variant>/   (sign it with packaging/feed_tool.py sign-package, then pack-engine --prefix strata-dsv41).
@@ -19,7 +19,7 @@ HP=${HP:?HP=<ROCm devel root>}
 CORE=${CORE:-$HP}
 LIBW=${LIBW:-$HP}
 OUT=${OUT:-$PWD/work-dsv41-$V}
-PKGVER=${PKGVER:-0.1.57}
+PKGVER=${PKGVER:-0.1.58}
 COMMIT=${COMMIT:-unknown}
 JOBS=${JOBS:-16}
 case "$V" in

packaging/build_hip_linux.sh

@@ -9,7 +9,7 @@
 #   SRC=<this checkout>              the Strata source (default: the checkout this script is in)
 #   LLAMA=<llama.cpp checkout>       at the pinned commit (setup.py LLAMA_CPP_COMMIT); mtmd and ggml come from it
 #   HP=<ROCm devel root>  CORE=<ROCm core root>  LIBW=<ROCm libraries root>   (TheRock: _rocm_sdk_devel, _rocm_sdk_core, _rocm_sdk_libraries_*)
-#   OUT=<work folder>  PKGVER=0.1.57  COMMIT=<source commit for the build id>  JOBS=16
+#   OUT=<work folder>  PKGVER=0.1.58  COMMIT=<source commit for the build id>  JOBS=16
 #   VISION_ONLY=1                    build only the two readers (no engine, no package)
 #   DSV41=1|0                        also build strata-dsv41 (the DeepSeek-V4.1 engine, tools/dsv41/build_hip_linux.sh's flags) and ship it beside
 #                                    strata: default 1 for the rx9070 variant (gfx1200 gfx1201; built and run on an R9700), 0 for the others
@@ -27,7 +27,7 @@ HP=${HP:?HP=<ROCm devel root>}
 CORE=${CORE:-$HP}
 LIBW=${LIBW:-$HP}
 OUT=${OUT:-$PWD/work-$V}
-PKGVER=${PKGVER:-0.1.57}
+PKGVER=${PKGVER:-0.1.58}
 COMMIT=${COMMIT:-unknown}
 JOBS=${JOBS:-16}
 case "$V" in

packaging/feed_tool.py

@@ -28,6 +28,7 @@ import datetime as dt
 import hashlib
 import json
 import os
+import shutil
 import stat
 import sys
 import tarfile
@@ -188,9 +189,23 @@ def pack_control(folder: Path, out: Path, *, version: str, platform: str, fmt: s
     return entry
 
 
+def pack_drafter(pack: Path, out: Path, *, version: str, backends: list[str], url_prefix: str = "", base: str = "", licence: str = "", notes: str = "") -> dict:
+    """Copy a DFlash2 drafter file into `out` (a flat release asset, as it is hosted) and write its feed entry (`<name>.entry.json`). The file is not an archive: the
+    client checks its size and SHA-256 against the signed feed AND against the pin in strata_control/drafting.py."""
+    pack = Path(pack)
+    Path(out).mkdir(parents=True, exist_ok=True)
+    target = Path(out) / pack.name
+    if pack.resolve() != target.resolve():
+        shutil.copy2(pack, target)
+    entry = {"name": pack.name, "version": version, "url": f"{url_prefix}{pack.name}", "size": target.stat().st_size, "sha256": _sha256(target),
+             "backends": list(backends), "base": base, "licence": licence, "notes": notes}
+    (Path(out) / (pack.name + ".entry.json")).write_text(json.dumps(entry, indent=1) + "\n", encoding="utf-8")
+    return entry
+
+
 def build_feed(out: Path, *, key: dict, sequence: int, channel: str = "stable", engine_dir: Path | None = None,
                control_dir: Path | None = None, app_json: Path | None = None, expires_days: int | None = None,
-               now: dt.datetime | None = None, dsv41_dir: Path | None = None) -> dict:
+               now: dt.datetime | None = None, dsv41_dir: Path | None = None, drafters_dir: Path | None = None) -> dict:
     now = now or dt.datetime.now(dt.timezone.utc)
     install_manifest = ""
     document: dict = {"schema_version": feedmod.SCHEMA_VERSION, "feed": "strata-updates", "channel": channel,
@@ -202,6 +217,9 @@ def build_feed(out: Path, *, key: dict, sequence: int, channel: str = "stable",
     if dsv41_dir:
         document["dsv41"] = {"packages": [json.loads(p.read_text(encoding="utf-8"))
                                           for p in sorted(Path(dsv41_dir).glob("*.entry.json"))]}
+    if drafters_dir:
+        document["drafters"] = {"packages": [json.loads(p.read_text(encoding="utf-8"))
+                                             for p in sorted(Path(drafters_dir).glob("*.entry.json"))]}
     if control_dir:
         entries = [json.loads(p.read_text(encoding="utf-8")) for p in sorted(Path(control_dir).glob("*.entry.json"))]
         versions = {e["version"] for e in entries}
@@ -311,6 +329,14 @@ def main(argv: list[str] | None = None) -> int:
     pe.add_argument("--format", choices=feedmod.FORMATS, default="tar.gz")
     pe.add_argument("--url-prefix", default="engine/", help='where the archive sits relative to the feed ("" for flat GitHub release assets)')
     pe.add_argument("--prefix", default="strata-engine", help="archive name prefix (strata-dsv41 for the DeepSeek engine packages)")
+    pd = sub.add_parser("pack-drafter", help="a DFlash2 drafter file as a feed entry (feed member drafters)")
+    pd.add_argument("pack", type=Path)
+    pd.add_argument("--out", type=Path, required=True)
+    pd.add_argument("--version", required=True)
+    pd.add_argument("--backends", required=True, help="comma-separated engine families it was measured on, e.g. metal")
+    pd.add_argument("--url-prefix", default="", help='"" for flat GitHub release assets')
+    pd.add_argument("--base", default="")
+    pd.add_argument("--licence", default="")
     pc = sub.add_parser("pack-control")
     pc.add_argument("folder", type=Path)
     pc.add_argument("--out", type=Path, required=True)
@@ -325,6 +351,7 @@ def main(argv: list[str] | None = None) -> int:
     bf.add_argument("--channel", default="stable")
     bf.add_argument("--engine", type=Path)
     bf.add_argument("--dsv41", type=Path, help="folder of the DeepSeek engine packages' .entry.json files (feed member dsv41)")
+    bf.add_argument("--drafters", type=Path, help="folder of the DFlash2 drafters' .entry.json files (feed member drafters; pack-drafter writes them)")
     bf.add_argument("--control", type=Path)
     bf.add_argument("--app-json", type=Path)
     bf.add_argument("--expires-days", type=int)
@@ -351,14 +378,17 @@ def main(argv: list[str] | None = None) -> int:
     elif a.cmd == "pack-engine":
         entry = pack_engine(a.folder, a.out, a.format, url_prefix=a.url_prefix, prefix=a.prefix)
         print(json.dumps({k: entry[k] for k in ("version", "backend", "variant", "size", "sha256", "manifest_sha256")}, indent=1))
+    elif a.cmd == "pack-drafter":
+        entry = pack_drafter(a.pack, a.out, version=a.version, backends=[b for b in a.backends.split(",") if b], url_prefix=a.url_prefix, base=a.base, licence=a.licence)
+        print(json.dumps({k: entry[k] for k in ("name", "version", "size", "sha256", "backends")}, indent=1))
     elif a.cmd == "pack-control":
         entry = pack_control(a.folder, a.out, version=a.version, platform=a.platform, fmt=a.format, url_prefix=a.url_prefix)
         print(json.dumps({k: entry[k] for k in ("version", "platform", "size", "sha256")}, indent=1))
     elif a.cmd == "build-feed":
         doc = build_feed(a.out, key=_load_key(a.key), sequence=a.sequence, channel=a.channel, engine_dir=a.engine,
-                         control_dir=a.control, app_json=a.app_json, expires_days=a.expires_days, dsv41_dir=a.dsv41)
+                         control_dir=a.control, app_json=a.app_json, expires_days=a.expires_days, dsv41_dir=a.dsv41, drafters_dir=a.drafters)
         print(f"wrote {a.out / feedmod.FEED_NAME} (+ .sig); engines {len((doc.get('engine') or {}).get('packages', []))}, "
-              f"deepseek engines {len((doc.get('dsv41') or {}).get('packages', []))}, control {'yes' if 'control' in doc else 'no'}, app {'yes' if 'app' in doc else 'no'}")
+              f"deepseek engines {len((doc.get('dsv41') or {}).get('packages', []))}, drafters {len((doc.get('drafters') or {}).get('packages', []))}, control {'yes' if 'control' in doc else 'no'}, app {'yes' if 'app' in doc else 'no'}")
     elif a.cmd == "set-app-platform":
         url = a.url or f"https://github.com/Niko1221/Strata/releases/latest/download/{a.archive.name}"
         entry = set_app_platform(a.app_json, target=a.target, archive=a.archive, signature_file=a.sig, url=url, version=a.version,

packaging/mac/build_dsv41_engine.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 # Build the DeepSeek-V4.1-Flash engine component for Apple Silicon (strata-dsv41, backend metal).
 #
-#   bash packaging/mac/build_dsv41_engine.sh [--version 0.1.57] [--build DIR] [--out DIR] [--build-id ID]
+#   bash packaging/mac/build_dsv41_engine.sh [--version 0.1.58] [--build DIR] [--out DIR] [--build-id ID]
 #
 # Run it in the source tree on the Mac. It configures and builds the `strata_ds4` target (metal/CMakeLists.txt) when the binary is
 # missing, stages it as `strata-dsv41` beside the kernel sources the engine compiles at start-up, packages them with
@@ -18,7 +18,7 @@
 #   <exe>/metal/ggml-common.h
 set -u
 ROOT=$(cd "$(dirname "$0")/../.." && pwd)
-VERSION=0.1.57
+VERSION=0.1.58
 BUILD="$ROOT/build-metal"
 OUT="$ROOT/pkg/engine"
 BUILD_ID=""

packaging/make_local_feed.py

@@ -58,6 +58,7 @@ def main(argv=None) -> int:
     ap.add_argument("--terminal", type=Path, action="append", default=[], help="a terminal archive (.tar.gz or .zip); repeat per platform")
     ap.add_argument("--engine", type=Path, action="append", default=[], help="an engine package folder (with engine-component.json)")
     ap.add_argument("--dsv41", type=Path, action="append", default=[], help="a DeepSeek engine package folder (the feed's dsv41 member)")
+    ap.add_argument("--drafter", type=Path, action="append", default=[], help="a DFlash2 drafter file (the feed's drafters member), listed for metal, hosted flat beside the feed")
     ap.add_argument("--keys", type=Path, default=None, help="folder for the test key (default: <out>/../feed-test-keys)")
     ap.add_argument("--url", default="http://127.0.0.1:8099", help="the address this feed will be served at (baked into install.sh/.ps1)")
     ap.add_argument("--sequence", type=int, default=1)
@@ -104,8 +105,12 @@ def main(argv=None) -> int:
             entry = ft.pack_engine(copy, out / "dsv41", url_prefix="dsv41/", prefix="strata-dsv41")
             print(f"deepseek engine {entry['platform']}-{entry['backend']} {entry['version']}: {entry['size'] // 1048576} MB")
             shutil.rmtree(copy.parent, ignore_errors=True)
+    for pack in a.drafter:
+        entry = ft.pack_drafter(pack, out / "drafters", version="1", backends=["metal"], url_prefix="drafters/")
+        print(f"drafter {entry['name']}: {entry['size'] // 1048576} MB")
     ft.build_feed(out, key=key, sequence=a.sequence, channel=a.channel, engine_dir=out / "engine" if a.engine else None,
-                  control_dir=out / "control" if a.terminal else None, dsv41_dir=out / "dsv41" if a.dsv41 else None)
+                  control_dir=out / "control" if a.terminal else None, dsv41_dir=out / "dsv41" if a.dsv41 else None,
+                  drafters_dir=out / "drafters" if a.drafter else None)
     if any("linux-aarch64" in p.name for p in (out / "control").glob("*.entry.json")):       # the Linux ARM64 Terminal reads its own feed file (same bytes here)
         for suffix in ("", ".sig"):
             shutil.copy2(out / ("strata-feed.json" + suffix), out / ("strata-feed-linux-aarch64.json" + suffix))

serve/frontend.py

@@ -706,13 +706,15 @@ def anthropic_to_messages(req: dict, think_unasked: bool = True) -> tuple[list[d
             elif kind == "thinking":
                 reasoning.append(block.get("thinking", ""))
             elif kind == "tool_use":
-                calls.append({"function": {"name": block.get("name"), "arguments": tool_arguments(block.get("input"))}})
+                calls.append({"function": {"name": block.get("name"), "arguments": tool_arguments(block.get("input"))},
+                              "_call_id": block.get("id")})
             elif kind == "tool_result":
                 # A tool's image (Claude Code's Read of a picture) reaches the encoder like a user's, as the OpenAI
                 # path's tool messages already do; a text-only result is one string, as before.
                 messages.append({"role": "tool", "content": _parts_of(_check_parts(
                     _tool_references(block.get("content"), deferred),
-                    f"messages[{i}].content[{j}].content", TEXT_PARTS + IMAGE_PARTS))})
+                    f"messages[{i}].content[{j}].content", TEXT_PARTS + IMAGE_PARTS)),
+                                 "_call_id": block.get("tool_use_id")})
         if text or calls or reasoning or parts:
             # an image sent beside tool results stays in this turn instead of being dropped
             out = {"role": m["role"], "content": _parts_of(parts) if _has_image(parts) else "".join(text)}
@@ -721,6 +723,8 @@ def anthropic_to_messages(req: dict, think_unasked: bool = True) -> tuple[list[d
             if calls:
                 out["tool_calls"] = calls
             messages.append(out)
+    # A client may send results in the order its tools finished (Claude Code does for parallel calls)
+    _order_chat_tool_results(messages)
     tools = [_anthropic_tool(t) for t in declared if t["name"] not in deferred] or None
     kwargs = {}
     # Anthropic: "thinking": {"type": "disabled"} or {"type": "enabled", "budget_tokens": N};
@@ -775,6 +779,15 @@ FUNC_END = "</function>"
 # #3441: the opener a model writes when it drifts by a token inside the envelope the template documents:
 # `<=NAME>` (the word `function` dropped), `<function:NAME>`, `<function =NAME>`, or the bare `<NAME>`
 _LOOSE_OPEN = re.compile(r"<\s*(?:function\s*[=:]\s*|=\s*)?([A-Za-z0-9_.\-]+)\s*>")
+_PREFIX_LOOKBACK = 256     # how far before a new delta a prefixed name or a misspelt opener can start (see OutputParser._prefixed)
+# #5409: `<function=tools actio>` / `<function=tools>`; the prefixes are the ones of #4751
+_PREFIX_SPACED = re.compile(r"(<function=\s*(?:tools|functions|default_api)[ \t]+)([^<>\s]+)\s*>")
+_PREFIX_PART = re.compile(r"<function=\s*(?:tools|functions|default_api)[ \t]+[^<>\s]{0,80}\s*$")
+_PREFIX_BARE = re.compile(r"(<tool_call>\s*)<function=\s*(tools|functions|default_api)\s*>(.*?)</function>", re.S)
+_PREFIX_BARE_OPEN = re.compile(r"<tool_call>\s*<function=\s*(tools|functions|default_api)\s*>(?!.*</function>)", re.S)
+# #5276: `<toolcall>` / `<tool-call>` as the opener (then whitespace, `<function=NAME>`), and such an opener still arriving
+_MISSPELT_OPEN = re.compile(r"<tool-?call>(\s*)<function=(\s*[^<>\s]+\s*)>")
+_MISSPELT_PART = re.compile(r"<tool-?call>\s*<function=[^<>]{0,80}$|<tool-?call>\s*(?:<(?:f(?:u(?:n(?:c(?:t(?:i(?:o(?:n)?)?)?)?)?)?)?)?)?$")
 _LOOSE_PART = re.compile(r"<\s*(?:function[ \t]*[=:]?\s*|=\s*)?[A-Za-z0-9_.\-]*\s*$")
 
 
@@ -915,8 +928,8 @@ def forced_call(tool_choice, tools: list[dict] | None) -> str | None:
         return None
     if kind == "required" and len(names) == 1:     # one tool to call: name it, the model cannot invent another
         kind, name = "named", next(iter(names))
-    if kind == "required" and names:
-        return CALL_START + "\n<function="
+    if kind == "required" and names:               # the model writes "<function=NAME>": from a written "<function=", Qwen3.8 went on with names no tool has
+        return CALL_START + "\n"
     if kind == "named" and name in names:
         return CALL_START + f"\n<function={name}>\n"
     print(f"[strata] tool_choice {json.dumps(tool_choice)[:200]} is not supported here (or names no tool of the "
@@ -1194,6 +1207,11 @@ class OutputParser:
         for name in self.schemas:
             if isinstance(name, str) and "__" in name:
                 aliases.setdefault(name.replace("__", "_"), set()).add(name)
+                # #5281: a namespaced MCP tool (`mcp__assets__get_asset_info`) written with its last segment only (`get_asset_info`),
+                # as the bare member of a dotted name is accepted above; only when exactly one declared tool ends that way
+                last = name.rsplit("__", 1)[1]
+                if last:
+                    aliases.setdefault(last, set()).add(name)
         # #4751: a programmatic-tool-calling harness (the DeepSeek Harness's PTC mode) shows the model its tools as `tools.write(...)`
         # inside code, and the model then writes the native call with that spelling (`tools.write`, also `functions.write`).  The
         # prefix is not part of the declared name; the same unambiguous-alias rule applies, an exactly declared name always wins.
@@ -1228,13 +1246,14 @@ class OutputParser:
         self.bare = False            # the call being read opened with <function= alone (no <tool_call> around it)
         self.batch = False           # the call being finished is followed by another in the same wrapper
         self._final = False          # #4701: the turn has ended: a call held for what might follow it is decided now
+        self._held = ""              # #5276/#5409: the end of a misspelt opener or a prefixed name still arriving (see _prefixed)
         self._reset_scan()
 
     @property
     def clean(self) -> bool:
         """Nothing is held back: a clean point to stop the thinking at (a parser that rewrites the model's text, as
         serve/dsml.py's does, also counts what it holds)."""
-        return not self.buf
+        return not self.buf and not self._held
 
     def _ok_at(self, p: int) -> bool:
         """self.buf[p] would open a call: at the start of a line, outside a code fence and inline code."""
@@ -1244,6 +1263,17 @@ class OutputParser:
         self._restore(snap)
         return ok
 
+    def _opens_at(self, p: int) -> bool:
+        """self.buf[p] would open a call as `<tool_call>` does: in the reasoning at the start of a line outside code, in the answer
+        anywhere outside code (a fence, inline code, an indented block)."""
+        if self.state == "reasoning":
+            return self._ok_at(p)
+        snap = self._snap()
+        self._track(self.buf[:p])
+        ok = not self._in_code()
+        self._restore(snap)
+        return ok
+
     def _callable(self, name: str) -> bool:
         """#3075: may a call for `name` be a call?  A tool the request declared (or its Responses spelling); with
         none declared, only where `declared_only` is off."""
@@ -1590,8 +1620,87 @@ class OutputParser:
                 start = text.find(tag[0], start + 1)
         return best
 
+    def _prefixed(self, fresh: int) -> str:
+        """#5409: a call written `<function=tools NAME>` (a space for the dot of `tools.NAME`) is the call NAME, and one written
+        `<function=tools>` (the bare prefix) is the one declared tool that takes the call's parameters.  Rewrites the complete
+        ones in self.buf to `<function=NAME>` and returns the tail of a call still arriving, which feed() holds back until the next
+        delta.  Only a declared name is ever the result; with none, or with two that fit, the text is left as it was.
+        `fresh`: the length of the text just added; only the end of the buffer (that and a little before it) can hold a new match,
+        so a long call is not scanned again with every token."""
+        held = self._opener(fresh)
+        start = max(0, len(self.buf) - fresh - _PREFIX_LOOKBACK)
+        buf = self.buf[start:]
+        if "<function=" in buf:
+            def spaced(m):
+                name = m.group(2).strip()
+                return f"<function={name}>" if name in self.schemas else m.group(0)
+            buf = _PREFIX_SPACED.sub(spaced, buf)
+            m = _PREFIX_PART.search(buf)
+            if m and not held:
+                held, buf = buf[m.start():], buf[:m.start()]
+            if self.recover:
+                def bare(m):
+                    body = m.group(3)
+                    names = re.findall(r"<parameter=\s*([^<>\s]+?)\s*>", body)
+                    tool = None if m.group(2) in self.schemas else self._tool_by_params(names)
+                    return f"{m.group(1)}<function={tool}>{body}</function>" if tool else m.group(0)
+                buf = _PREFIX_BARE.sub(bare, buf)
+                m = _PREFIX_BARE_OPEN.search(buf)
+                if m and m.group(1) not in self.schemas and not held and len(buf) - m.start() <= RCALL_MAX:
+                    held, buf = buf[m.start():], buf[:m.start()]
+            if buf != self.buf[start:]:
+                self.buf = self.buf[:start] + buf
+        return held
+
+    def _opener(self, fresh: int) -> str:
+        """#5276: `<toolcall>` / `<tool-call>` (the opener written as text, misspelled) followed by `<function=NAME>` of a declared
+        tool is `<tool_call>`, where a `<tool_call>` would open a call (in the reasoning the start of a line, outside code; in the
+        answer outside code).  Prose that names the misspelled tag, or any other follower, is left as it was.  Returns the tail of
+        an opener still arriving (held back by feed)."""
+        start = max(0, len(self.buf) - fresh - _PREFIX_LOOKBACK)
+        if "<tool" not in self.buf[start:]:
+            return ""
+        buf = self.buf
+        pos, out = start, []
+        for m in _MISSPELT_OPEN.finditer(buf, start):
+            name = m.group(2).strip()
+            name = self.aliases.get(name, name)
+            if name in self.schemas and self._opens_at(m.start()):
+                out.append(buf[pos:m.start()] + CALL_START + m.group(1) + FUNC_START + m.group(2) + ">")
+                pos = m.end()
+        buf = buf[:start] + "".join(out) + buf[pos:] if out else buf
+        held = ""
+        m = _MISSPELT_PART.search(buf, start)
+        tail = buf[m.start():] if m else ""
+        if not tail:                                        # the opener itself still arriving: a prefix of `<toolcall>`
+            for k in range(min(len(buf), 11), 5, -1):
+                if any(w.startswith(buf[-k:]) for w in ("<toolcall>", "<tool-call>")) and buf[-k:] != "<tool_":
+                    tail = buf[-k:]
+                    break
+        if tail:
+            held, buf = tail, buf[:len(buf) - len(tail)]
+        self.buf = buf
+        return held
+
+    def _tool_by_params(self, names: list[str]) -> str | None:
+        """The one declared tool whose properties hold every one of `names` and whose required parameters are all in them."""
+        found = []
+        for tname, schema in self.schemas.items():
+            params = (schema or {}).get("parameters") or {}
+            props = params.get("properties") or {}
+            if names and all(n in props for n in names) and all(r in names for r in params.get("required") or []):
+                found.append(tname)
+        return found[0] if len(found) == 1 else None
+
     def feed(self, delta: str) -> list[Event]:
+        if self._held:
+            delta, self._held = self._held + delta, ""
         self.buf += delta
+        if self.schemas and delta:
+            self._held = self._prefixed(len(delta))
+        return self._feed()
+
+    def _feed(self) -> list[Event]:
         out: list[Event] = []
         while True:
             if self.state == "rcall":
@@ -1885,6 +1994,8 @@ class OutputParser:
         how the turn ended: calls waiting from the reasoning become real calls only on a natural stop (None = stop);
         a turn cut by max tokens (or cancelled, or failed) keeps them as reasoning text (#1058)."""
         out = []
+        if self._held:
+            self.buf, self._held = self.buf + self._held, ""
         if self.buf and self.state in ("call", "rcall"):
             self._final = True                  # #4701: a call held after its closing tags (could a file's text go on?) is whole now
             out += self.feed("")

serve/server.py

@@ -1926,10 +1926,12 @@ class StrataEngine:
         return None
 
     def _apply_bdone(self, f: list[str]) -> None:
-        """`BDONE <slot> <produced> <finish> <ms>` into this request's figures."""
+        """BDONE counts the admission token too, but its clock starts after the admission's DONE."""
         if len(f) >= 5 and isinstance(self.last, dict):
-            self.last = {**self.last, "finish": f[3], "decode_ms": float(f[4]),
-                         "generated": int(f[2]) if f[2].isdigit() else self.last.get("generated")}
+            admission_ms = self.last.get("decode_ms")
+            self.last = {**self.last, "finish": f[3],
+                         "decode_ms": admission_ms + float(f[4]) if admission_ms is not None else None,
+                         "generated": int(f[2]) if f[2].isdigit() else None}
 
     def _raise_if_fatal_gpu(self, line: str) -> None:
         """#4350: the engine's ERR line "fatal GPU error: ..." means it is ending (its GPU context is gone).  The request in flight fails
@@ -2251,6 +2253,7 @@ class StrataEngine:
         orig_len = len(prompt)  # a leg that continues (prompt + what it generated) names where the reply began: pen_from, the penalty window's start
         reused0 = None      # the first read's reused tokens: a continued leg's count includes this request's own output
         carry = {}          # #1869: the figures of the legs that ended in a hand-back to the solo path (the final DONE adds to them)
+        yielded_legs = []    # prompt reads that gave way before producing output
         ok = yield from self._take_control(cancel, len(prompt), new=True,
                                            prompt=prompt if (COALESCE_PREFILL > 1 and not embeddings) else None)
         if not ok:
@@ -2266,6 +2269,7 @@ class StrataEngine:
             slot, cancel._strata_preslot = pre, None
         phase = "none"            # solo -> (admit -> slot) ; "done" once the engine has finished with this request
         stop_sent = False
+        admission_pending, admission_before = False, None   # the admission's DONE is partial until BADM 0 or BDONE
         yields, solo_again, refused = 0, 0, 0
         admit_wait, admit_since, admit_said = False, 0.0, False   # #1603: waiting for a slot, since when, said
         try:
@@ -2326,6 +2330,7 @@ class StrataEngine:
                     if self._done_finish() == "nonfinite":    # #879 (never promoted to a slot)
                         raise EngineNonFinite("the engine's logits became non-finite (#879)")
                     if self._yielded is not None and reserved is not None and self._yielded[0] == reserved:
+                        yielded_legs.append(self.last)
                         slot, reserved = reserved, None     # gave way: the read goes on in that slot (below)
                     elif reserved is not None:              # it did not give way: the slot is free again
                         with self.slot_cv:
@@ -2431,6 +2436,7 @@ class StrataEngine:
                     if started is not None:
                         started()
                     self.slot_held[slot] = []               # the admission overwrites what the slot held
+                    admission_pending, admission_before = True, self.last
                     phase = "admit"
                     self._ctl_mode, self._ctl_result, self._yielded = "batch", None, None
                     self._ctl_erred = False
@@ -2478,6 +2484,7 @@ class StrataEngine:
                         slot = None
                         continue
                     cont = bool(self._ctl_result and self._ctl_result[1])
+                    admission_pending = cont
                     phase = "slot" if cont else "none"
                     while pending:
                         t = pending.pop(0)
@@ -2486,6 +2493,7 @@ class StrataEngine:
                     if not cont and self._done_finish() == "nonfinite":   # #879
                         raise EngineNonFinite("the engine's logits became non-finite (#879)")
                     if not cont and self._yielded is not None and self._yielded[0] == slot and not cancel.is_set():
+                        yielded_legs.append(self.last)
                         continue                            # gave way: again once the shorter request is in
                     break
                 self.ctl.release()
@@ -2528,6 +2536,7 @@ class StrataEngine:
                         phase = "none"
                         f = line.split()
                         self._apply_bdone(f)
+                        admission_pending = False
                         # what the slot's sessions hold now: the prompt and every token fed (all but the last one)
                         self.slot_held[slot] = list(prompt) + out[gen0:-1] if len(out) > gen0 else []
                         if f[3:4] == ["nonfinite"]:              # #879: the engine dropped the slot's state
@@ -2569,6 +2578,8 @@ class StrataEngine:
                         undrained = "DONE"
                 elif phase == "admit":
                     line = self._drain_control("BADM", born=born, lines_q=lines_q)
+                    if line and line.startswith("BADM "):
+                        admission_pending = line.split()[2:3] == ["1"]
                     if line and line.startswith("BADM ") and line.split()[2:3] == ["1"]:
                         phase = "slot"
                     elif line is None and self.gen == born and self.alive():
@@ -2586,11 +2597,19 @@ class StrataEngine:
                 bd = self._bdone_after_eos(slot, q=sq[slot])    # (#1869) the slot's figures, not the admission's
                 if bd is not None:
                     self._apply_bdone(bd.split())
+                    admission_pending = False
                     bdone_in = True
+            ended_on_yield = bool(yielded_legs and self.last is yielded_legs[-1])
+            if admission_pending and isinstance(self.last, dict) and (self.last is not admission_before or carry):
+                # A background drain owns the rest. The admission's one token and time are not this request's totals.
+                self.last = {**self.last, "generated": None, "decode_ms": None}
             if reused0 is not None and isinstance(self.last, dict):
                 self.last = {**self.last, "reused": reused0}
             if carry and isinstance(self.last, dict):
                 self.last = add_carry(carry, self.last)
+            earlier_reads = yielded_legs[:-1] if ended_on_yield else yielded_legs
+            if earlier_reads and isinstance(self.last, dict):
+                self.last = prefill_leg_stats(earlier_reads, self.last, len(ids))
             if holding:
                 self.ctl.release()
             if admit_wait:
@@ -4862,6 +4881,7 @@ class Service:
         self.reasoning_budget_tokens = 0                 # #123: the config's default thinking budget (0: none)
         self.repeat_stop_tokens = REPEAT_STOP_TOKENS     # #606: one token this many times in a row ends a reply (0: off)
         self.anthropic_think_unasked = True               # #278: "anthropic_thinking": "on_request" -> False
+        self.undeclared_tool_calls = "text"               # #4622: config "undeclared_tool_calls": "call" hands a call of an undeclared name to the client
         self.tool_call_recovery = True                    # default on (0.2.8): tool calls in forms next to the template's; config "tool_call_recovery": false turns it off
         self.images = None                                # serve/images.py's ImageGen when the config has "images"
         self.restore_after_load = None                    # a session file an image job saved before it unloaded
@@ -6230,7 +6250,7 @@ class Service:
                 budget = max(1, max_new - reserve)
         parser = self.parser_cls(thinking=thinking, tools=with_history_tools(tools, sampling), stream_tools=True,
                                  recover=self.tool_call_recovery, declared_only=declared_only,
-                                 deferred=deferred)
+                                 deferred=deferred or self.undeclared_tool_calls == "call")   # #4622: opt-in, as for a deferred tool
         detok, n, finish = Detokenizer(self.tok), 0, "length"
         run_tok, run_len, repeated = None, 0, False     # #606: the current run of one repeated token
         thinking_n = 0                                  # tokens written while thinking (Responses' reasoning_tokens)
@@ -6768,8 +6788,9 @@ def prompt_tokens_seen(prompt_tokens: int, last: dict) -> int:
 def carry_leg(carry: dict, leg: dict) -> None:
     """#1869: a request that decoded in a batch slot and went back to the solo path ends with the solo path's DONE, which
     replaces the engine's figures: keep the slot leg's decode time, tokens and prompt figures to add to it."""
-    carry["decode_ms"] = carry.get("decode_ms", 0.0) + float(leg.get("decode_ms") or 0.0)
-    carry["generated"] = carry.get("generated", 0) + int(leg.get("generated") or 0)
+    for key in ("decode_ms", "generated"):
+        previous, value = carry.get(key, 0), leg.get(key)
+        carry[key] = previous + value if previous is not None and value is not None else None
     carry.setdefault("prompt_ms", float(leg.get("prompt_ms") or 0.0))
     if leg.get("prompt_read") is not None:
         carry.setdefault("prompt_read", int(leg["prompt_read"]))
@@ -6778,9 +6799,10 @@ def carry_leg(carry: dict, leg: dict) -> None:
 def add_carry(carry: dict, last: dict) -> dict:
     """The request's totals: the earlier legs' decode time and tokens plus the final one's; the prompt figures of the leg
     that read the prompt (the solo continuation reads next to nothing)."""
-    out = {**last, "decode_ms": float(last.get("decode_ms") or 0.0) + carry.get("decode_ms", 0.0),
-           "generated": int(last.get("generated") or 0) + carry.get("generated", 0),
-           "prompt_ms": max(float(last.get("prompt_ms") or 0.0), carry.get("prompt_ms", 0.0))}
+    out = {**last, "prompt_ms": max(float(last.get("prompt_ms") or 0.0), carry.get("prompt_ms", 0.0))}
+    for key in ("decode_ms", "generated"):
+        previous, value = carry.get(key, 0), last.get(key)
+        out[key] = previous + value if previous is not None and value is not None else None
     if "prompt_read" in carry:
         out["prompt_read"] = carry["prompt_read"]
     return out
@@ -6811,14 +6833,14 @@ def request_timings(prompt_tokens: int, generated: int, last: dict) -> dict | No
         # #1870: the engine says how many prompt tokens it read: the slot's own session may have held the rest
         # (not the conversation cache), so the rate is not made of tokens it never read
         cache_n = max(cache_n, max(0, prompt_tokens - int(last["prompt_read"])))
-    prompt_n, prompt_ms, decode_ms = max(0, prompt_tokens - cache_n), float(last["prompt_ms"]), float(last.get("decode_ms") or 0)
-    decoded = int(last.get("generated") or generated)          # the engine's count gives its rate, as /metrics does
+    prompt_n, prompt_ms, decode_ms = max(0, prompt_tokens - cache_n), float(last["prompt_ms"]), last.get("decode_ms")
+    decoded = last.get("generated")          # explicit zero and missing native work are not the API's output count
     return {"cache_n": cache_n, "prompt_n": prompt_n, "prompt_ms": round(prompt_ms, 1),
             "prompt_per_token_ms": round(prompt_ms / prompt_n, 3) if prompt_n else None,
             "prompt_per_second": round(prompt_n / (prompt_ms / 1000), 1) if prompt_n and prompt_ms > 0 else None,
-            "predicted_n": generated, "predicted_ms": round(decode_ms, 1),
-            "predicted_per_token_ms": round(decode_ms / decoded, 3) if decoded else None,
-            "predicted_per_second": round(decoded / (decode_ms / 1000), 1) if decoded and decode_ms > 0 else None,
+            "predicted_n": generated, "predicted_ms": round(decode_ms, 1) if decode_ms is not None else None,
+            "predicted_per_token_ms": round(decode_ms / decoded, 3) if decoded and decode_ms is not None else None,
+            "predicted_per_second": round(decoded / (decode_ms / 1000), 1) if decoded and decode_ms and decode_ms > 0 else None,
             # the speculative drafts, as llama.cpp names them (from PR #83, @mikicvi): only when the engine reported them
             **({"draft_n": int(last["drafts_offered"]), "draft_n_accepted": int(last["drafts_accepted"])}
                if last.get("drafts_offered") is not None else {})}
@@ -6836,11 +6858,23 @@ def request_stats(segments: list[dict]) -> dict:
             result.pop(key, None)
     for key in ("prompt_ms", "decode_ms", "generated", "drafts_offered", "drafts_accepted",
                 "hits", "lookups", "offloaded", "ram_blobs", "file_blobs", "file_mb", "experts_streamed"):
-        if any(segment.get(key) is not None for segment in segments):
+        if key in ("decode_ms", "generated") and any(segment.get(key) is None for segment in segments):
+            result[key] = None
+        elif any(segment.get(key) is not None for segment in segments):
             result[key] = sum(segment.get(key) or 0 for segment in segments)
     return result
 
 
+def prefill_leg_stats(earlier: list[dict], last: dict, prompt_tokens: int) -> dict:
+    """Total yielded reads once, retaining only reuse from before this API request."""
+    legs = [*earlier, last]
+    total = request_stats(legs)
+    if all(leg.get("prompt_read") is not None for leg in legs):
+        fresh = max(0, prompt_tokens - int(total.get("reused") or 0))
+        total["prompt_read"] = min(fresh, sum(int(leg["prompt_read"]) for leg in legs))
+    return total
+
+
 def _debug_req(api, req, messages, tools, max_new, thinking, prompt_tokens):
     """One compact line per request while diagnosing blank/empty turns. Set STRATA_DEBUG=1 to enable."""
     if not os.environ.get("STRATA_DEBUG"):
@@ -9816,6 +9850,18 @@ def _component_exe(reference: str, cfg: dict) -> str:
         raise SystemExit(e.exit_code) from None
 
 
+def undeclared_tool_calls_setting(cfg: dict) -> str:
+    """#4622: the config's "undeclared_tool_calls": "text" (the default, when the config does not name it) keeps a well-formed call of a tool the request
+    did not declare as the text the model wrote, with strata_rejected_tool_calls beside it; "call" returns it as a tool call, for a client that has
+    its own policy for unknown names (it answers the model with the tool names it has).  Anything else stops the start."""
+    mode = cfg.get("undeclared_tool_calls")
+    if mode is None:                                  # the environment names it too (STRATA_UNDECLARED_TOOL_CALLS); the config wins
+        mode = os.environ.get("STRATA_UNDECLARED_TOOL_CALLS") or "text"
+    if mode not in ("text", "call"):
+        raise SystemExit(f"[strata] config \"undeclared_tool_calls\" must be \"text\" or \"call\", not {mode!r}")
+    return mode
+
+
 def tool_call_recovery_setting(cfg: dict) -> bool:
     """The config's "tool_call_recovery" (see OutputParser's recover): ON when the config does not name it (since 0.2.8; 0.2.7 and before: off),
     "tool_call_recovery": false turns it off. Anything but true or false stops the start."""
@@ -10320,6 +10366,7 @@ def _main(argv=None, *, lifecycle_event=None, interrupt_exit_code: int = 0) -> i
         raise SystemExit(f"[strata] config anthropic_thinking must be \"model\" or \"on_request\", not {mode!r}")
     svc.anthropic_think_unasked = mode == "model"
     svc.tool_call_recovery = tool_call_recovery_setting(cfg)
+    svc.undeclared_tool_calls = undeclared_tool_calls_setting(cfg)
     etu = cfg.get("env_to_user", False)                # opt-in: see frontend._env_to_user
     if not isinstance(etu, bool):
         raise SystemExit(f"[strata] config \"env_to_user\" must be true or false, not {etu!r}")

serve/test_anthropic_tool_results.py

@@ -0,0 +1,155 @@
+"""Anthropic tool_result blocks must follow tool_use_id before the template drops it."""
+import copy
+import http.client
+import json
+from pathlib import Path
+import unittest
+
+from serve.frontend import ChatTemplate, anthropic_to_messages
+from serve.server import ByteTokenizer, MockEngine, Service, serve
+
+
+TEMPLATE = ChatTemplate(Path(__file__).with_name("chat_template.jinja"))
+DEEPSEEK = ChatTemplate(Path(__file__).with_name("fixtures") / "deepseek41" / "chat_template.jinja")
+
+
+def history():
+    """Two calls of one tool, answered in the order they finished: what Claude Code 2.1.316 sends for parallel Greps."""
+    return [
+        {"role": "user", "content": "Search for both."},
+        {"role": "assistant", "content": [
+            {"type": "tool_use", "id": "toolu_slow", "name": "Grep", "input": {"pattern": "slow"}},
+            {"type": "tool_use", "id": "toolu_fast", "name": "Grep", "input": {"pattern": "fast"}}]},
+        {"role": "user", "content": [
+            {"type": "tool_result", "tool_use_id": "toolu_fast", "content": "RESULT_FROM_FAST"},
+            {"type": "tool_result", "tool_use_id": "toolu_slow", "content": "RESULT_FROM_SLOW"}]},
+    ]
+
+
+def in_call_order():
+    raw = history()
+    raw[2]["content"].reverse()
+    return raw
+
+
+def normalized(messages):
+    return anthropic_to_messages({"messages": messages})[0]
+
+
+def results(messages):
+    return [m["content"] for m in normalized(messages) if m["role"] == "tool"]
+
+
+class AnthropicToolResults(unittest.TestCase):
+    def test_reversed_results_of_the_same_tool_follow_tool_use_id(self):
+        raw = history()
+        before = copy.deepcopy(raw)
+        self.assertEqual(results(raw), ["RESULT_FROM_SLOW", "RESULT_FROM_FAST"])
+        self.assertEqual(raw, before)
+        for template in (TEMPLATE, DEEPSEEK):
+            with self.subTest(template=str(template)):
+                prompt = template.render(normalized(raw), enable_thinking=False)
+                self.assertEqual(prompt, template.render(normalized(in_call_order()), enable_thinking=False))
+                self.assertLess(prompt.index("RESULT_FROM_SLOW"), prompt.index("RESULT_FROM_FAST"))
+
+    def test_already_ordered_and_legacy_prompts_are_identical(self):
+        ordered = in_call_order()
+        legacy = copy.deepcopy(ordered)
+        for block in legacy[1]["content"]:
+            block.pop("id")
+        for block in legacy[2]["content"]:
+            block.pop("tool_use_id")
+        self.assertEqual(normalized(ordered), normalized(legacy))
+        self.assertEqual([m["content"] for m in normalized(ordered)[2:]], ["RESULT_FROM_SLOW", "RESULT_FROM_FAST"])
+
+    def test_ids_never_reach_the_template(self):
+        for raw in (history(), in_call_order()):
+            messages = normalized(raw)
+            self.assertTrue(all(set(m) == {"role", "content"} for m in messages[2:]))
+            self.assertTrue(all(set(c) == {"function"} for c in messages[1]["tool_calls"]))
+
+    def test_ambiguous_or_incomplete_ids_preserve_received_order(self):
+        for case in ("missing_call", "missing_result", "duplicate_call", "duplicate_result", "unknown",
+                     "list_call", "list_result", "empty_call", "empty_result", "partial"):
+            raw = history()
+            calls, blocks = raw[1]["content"], raw[2]["content"]
+            if case == "missing_call":
+                calls[0].pop("id")
+            elif case == "missing_result":
+                blocks[0].pop("tool_use_id")
+            elif case == "duplicate_call":
+                calls[1]["id"] = "toolu_slow"
+            elif case == "duplicate_result":
+                blocks[1]["tool_use_id"] = "toolu_fast"
+            elif case == "unknown":
+                blocks[0]["tool_use_id"] = "unknown"
+            elif case == "list_call":
+                calls[0]["id"] = []
+            elif case == "list_result":
+                blocks[0]["tool_use_id"] = []
+            elif case == "empty_call":
+                calls[0]["id"] = ""
+            elif case == "empty_result":
+                blocks[0]["tool_use_id"] = ""
+            else:
+                blocks.pop()
+            with self.subTest(case=case):
+                self.assertEqual(results(raw), [b["content"] for b in blocks])
+
+    def test_each_tool_run_is_scoped_to_its_assistant_turn(self):
+        self.assertEqual(results(history() + history()), ["RESULT_FROM_SLOW", "RESULT_FROM_FAST"] * 2)
+
+    def test_results_split_across_user_messages_are_one_run(self):
+        raw = history()
+        first, second = raw[2]["content"]
+        raw[2:] = [{"role": "user", "content": [first]}, {"role": "user", "content": [second]}]
+        self.assertEqual(results(raw), ["RESULT_FROM_SLOW", "RESULT_FROM_FAST"])
+
+    def test_system_message_between_calls_and_results_keeps_received_order(self):
+        raw = history()
+        raw.insert(2, {"role": "system", "content": "late note"})
+        self.assertEqual(results(raw), ["RESULT_FROM_FAST", "RESULT_FROM_SLOW"])
+
+    def test_text_beside_the_results_stays_after_them(self):
+        raw = history()
+        raw[2]["content"].append({"type": "text", "text": "and then?"})
+        self.assertEqual([(m["role"], m["content"]) for m in normalized(raw)[2:]],
+                         [("tool", "RESULT_FROM_SLOW"), ("tool", "RESULT_FROM_FAST"), ("user", "and then?")])
+
+    def test_multimodal_result_moves_with_its_call(self):
+        raw = history()
+        image = {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "AA=="}}
+        raw[2]["content"][0]["content"] = [{"type": "text", "text": "RESULT_FROM_FAST"}, image]
+        got = results(raw)
+        self.assertEqual(got[0], "RESULT_FROM_SLOW")
+        self.assertEqual([p.get("type") for p in got[1]], ["text", "image"])
+
+    def test_http_engine_receives_the_ordered_prompt(self):
+        tok = ByteTokenizer()
+        engine = MockEngine(tok, "ok", max_context=4096)
+        service = Service(engine, tok, TEMPLATE)
+        httpd = serve(service, port=0)
+        try:
+            for stream in (False, True):
+                with self.subTest(stream=stream):
+                    engine.last_prompt = None
+                    connection = http.client.HTTPConnection("127.0.0.1", httpd.server_address[1], timeout=10)
+                    try:
+                        connection.request("POST", "/v1/messages", json.dumps({
+                            "model": "x", "messages": history(), "max_tokens": 16, "stream": stream,
+                            "thinking": {"type": "disabled"}}), {"Content-Type": "application/json"})
+                        response = connection.getresponse()
+                        body = response.read()
+                    finally:
+                        connection.close()
+                    self.assertEqual(response.status, 200, body)
+                    prompt = tok.decode(engine.last_prompt)
+                    self.assertLess(prompt.index("RESULT_FROM_SLOW"), prompt.index("RESULT_FROM_FAST"))
+        finally:
+            httpd.shutdown()
+            httpd.server_close()
+
+
+
+if __name__ == "__main__":
+    unittest.main()

serve/test_frontend.py

@@ -199,7 +199,7 @@ class ForcedCallOpening(unittest.TestCase):
         from serve.frontend import forced_call
         one, two = [{"name": "a"}], [{"name": "a"}, {"name": "b"}]
         self.assertTrue(forced_call("required", one).endswith("<function=a>\n"))
-        self.assertTrue(forced_call("required", two).endswith("<function="))
+        self.assertTrue(forced_call("required", two).endswith("<tool_call>\n"))   # the model writes "<function=NAME>" itself
         self.assertIsNone(forced_call("auto", one))
 
 

serve/test_parallel.py

@@ -32,6 +32,7 @@ fail = "--fail-window" in args
 groups = int(args[args.index("--says-groups") + 1]) if "--says-groups" in args else 0   # INFO batch_groups=G (--batch-groups auto)        # #997: the first window over two slots fails
 STEP = 0.02
 CH = 32
+BDONE_MS = " 0.0" if "--zero-bdone-ms" in args else " 1.0"
 lines, stop = queue.Queue(), threading.Event()
 def reader():
     for l in sys.stdin:
@@ -72,7 +73,7 @@ def window():        # one batch window: every active slot one token
         left, max_new, produced = active[b]
         if b in nf_slots and produced >= 2:
             nf_slots.discard(b)
-            print(f"BDONE {b} {produced} nonfinite 1.0", flush=True)
+            print(f"BDONE {b} {produced} nonfinite{BDONE_MS}", flush=True)
             del active[b]
             continue
         t = left.pop(0) if left else 257
@@ -81,7 +82,7 @@ def window():        # one batch window: every active slot one token
         if t == 257 or produced >= max_new:
             fin = 'stop' if t == 257 else 'cancel' if b in stopped else 'length'
             stopped.discard(b)
-            print(f"BDONE {b} {produced} {fin} 1.0", flush=True)
+            print(f"BDONE {b} {produced} {fin}{BDONE_MS}", flush=True)
             del active[b]
         else:
             active[b] = [left, max_new, produced]
@@ -583,6 +584,102 @@ class FakeEngineCase(unittest.TestCase):
         return res
 
 
+class BatchLegAccounting(FakeEngineCase):
+    """The admission's DONE times one token; BDONE counts that token but times only the slot."""
+
+    def queued_chats(self, texts=("first", "second"), max_tokens=10, stop=None):
+        replies, errors = {}, []
+
+        def request(text):
+            body = {"messages": [{"role": "user", "content": text}], "max_tokens": max_tokens,
+                    "reasoning_effort": "none", **({"stop": [stop]} if stop else {})}
+            req = urllib.request.Request(self.base + "/v1/chat/completions", data=json.dumps(body).encode(),
+                                         headers={"Content-Type": "application/json"})
+            try:
+                with urllib.request.urlopen(req, timeout=15) as response:
+                    replies[text] = json.loads(response.read())
+            except Exception as error:
+                errors.append(repr(error))
+
+        workers = [threading.Thread(target=request, args=(text,)) for text in texts]
+        # Both requests must be waiting before either takes control: the first cannot take the solo shortcut.
+        with self.engine.ctl:
+            for worker in workers:
+                worker.start()
+            deadline = time.monotonic() + 3
+            while self.engine.waiting < 2 and time.monotonic() < deadline:
+                time.sleep(0.01)
+            self.assertEqual(self.engine.waiting, 2)
+        for worker in workers:
+            worker.join(15)
+            self.assertFalse(worker.is_alive())
+        self.assertEqual(errors, [])
+        self.assertEqual(len(replies), 2)
+        self.assertEqual([line.split()[0] for line in self.log.read_text().splitlines()][:2], ["BGEN", "BGEN"])
+        return replies
+
+    def test_eos_adds_admission_time_without_counting_its_token_twice(self):
+        self.start(2)
+        with mock.patch.object(server, "EOS_IDS", {257}):
+            replies = self.queued_chats()
+        for reply in replies.values():
+            timings = reply["timings"]
+            self.assertEqual((timings["predicted_n"], timings["predicted_ms"]), (10, 21.0))
+            self.assertEqual(timings["predicted_per_second"], 476.2)
+        metrics = self.get("/metrics")
+        self.assertEqual([(row["engine_generated"], row["decode_ms"]) for row in metrics["requests"]],
+                         [(10, 21.0), (10, 21.0)])
+        self.assertEqual(metrics["totals"]["decode_ms"], 42.0)
+
+    def test_length_finish_also_adds_admission_time(self):
+        self.start(2)
+        for reply in self.queued_chats(max_tokens=8).values():
+            self.assertEqual((reply["timings"]["predicted_n"], reply["timings"]["predicted_ms"]), (8, 21.0))
+
+    def test_zero_slot_time_is_known_and_still_includes_admission(self):
+        self.start(2, more=("--zero-bdone-ms",))
+        with mock.patch.object(server, "EOS_IDS", {257}):
+            replies = self.queued_chats()
+        for reply in replies.values():
+            self.assertEqual(reply["timings"]["predicted_ms"], 20.0)
+
+    def assert_incomplete_after_stop(self, stop):
+        self.start(2)
+        replies = self.queued_chats(stop=stop)
+        for reply in replies.values():
+            self.assertIsNone(reply["timings"]["predicted_ms"])
+            self.assertIsNone(reply["timings"]["predicted_per_second"])
+        for row in self.get("/metrics")["requests"]:
+            self.assertIsNone(row["engine_generated"])
+            self.assertIsNone(row["decode_ms"])
+            self.assertIsNone(row["decode_tok_s"])
+        self.assertEqual(self.get("/metrics")["totals"]["decode_ms"], 0.0)
+
+    def test_close_at_admission_token_does_not_publish_partial_totals(self):
+        self.assert_incomplete_after_stop("o")
+
+    def test_close_in_slot_does_not_publish_partial_totals(self):
+        self.assert_incomplete_after_stop("k, ")
+
+    def test_slot_to_solo_carry_includes_admission_once(self):
+        self.start(2, slot_cache=True)
+        with mock.patch.object(server, "EOS_IDS", {257}):
+            replies = self.queued_chats(("short", "LONGREPLY"), max_tokens=200)
+        lines = [line.split() for line in self.log.read_text().splitlines()]
+        solos = [line for line in lines if line[0] == "GEN"]
+        self.assertEqual(len(solos), 1, lines)
+        prompt_lengths = [int(line[2]) for line in lines if line[0] == "BGEN"]
+        slot_tokens = int(solos[0][2]) - max(prompt_lengths)
+        self.assertGreaterEqual(slot_tokens, 2)
+        self.assertLess(slot_tokens, 55)
+        # "ok, " + 15 * "la " + "done." + EOS: 55 total tokens. The solo leg times its remaining tokens at 20 ms each.
+        expected_ms = 21.0 + (55 - slot_tokens) * 20.0
+        reply = replies["LONGREPLY"]
+        self.assertEqual((reply["timings"]["predicted_n"], reply["timings"]["predicted_ms"]), (55, expected_ms))
+        row = max(self.get("/metrics")["requests"], key=lambda item: item["output_tokens"])
+        self.assertEqual((row["engine_generated"], row["decode_ms"]), (55, expected_ms))
+
+
 class ParallelService(FakeEngineCase):
     """The real StrataEngine and Service over HTTP, the fake engine behind them: slots, queues, benchmarks, telemetry.
     The yielding, restart and failure tests are in serve/test_parallel_yield.py (this file alone ran past 100 s)."""

serve/test_request_accounting.py

@@ -9,7 +9,7 @@ import urllib.request
 from pathlib import Path
 
 from serve.frontend import ChatTemplate
-from serve.server import ByteTokenizer, EngineDied, Service, add_carry, carry_leg, request_timings, serve
+from serve.server import ByteTokenizer, EngineDied, Service, add_carry, carry_leg, request_stats, request_timings, serve
 from serve.test_server import ThinkingEngine
 
 
@@ -204,6 +204,33 @@ class SlotLegAccounting(unittest.TestCase):
         self.assertEqual((total["generated"], total["decode_ms"], total["prompt_ms"]), (35, 450.0, 50.0))
         self.assertNotIn("prompt_read", total)
 
+    def test_missing_slot_time_stays_unknown_after_solo_carry(self):
+        carry = {}
+        carry_leg(carry, {"generated": 10, "decode_ms": None, "prompt_ms": 50.0})
+        total = add_carry(carry, {"generated": 5, "decode_ms": 100.0, "prompt_ms": 1.0})
+        self.assertEqual(total["generated"], 15)
+        self.assertIsNone(total["decode_ms"])
+
+    def test_missing_final_figures_stay_unknown_after_carry_and_segments(self):
+        carry = {}
+        carry_leg(carry, {"generated": 10, "decode_ms": 20.0, "prompt_ms": 50.0})
+        total = add_carry(carry, {"generated": None, "decode_ms": None, "prompt_ms": 1.0})
+        combined = request_stats([{"generated": 5, "decode_ms": 100.0, "prompt_ms": 10.0}, total])
+        self.assertIsNone(combined["generated"])
+        self.assertIsNone(combined["decode_ms"])
+
+    def test_unknown_decode_time_is_not_reported_as_zero(self):
+        timings = request_timings(100, 10, {"generated": 10, "decode_ms": None, "prompt_ms": 50.0})
+        self.assertIsNone(timings["predicted_ms"])
+        self.assertIsNone(timings["predicted_per_token_ms"])
+        self.assertIsNone(timings["predicted_per_second"])
+
+    def test_explicit_zero_native_count_is_not_replaced_by_api_output_count(self):
+        timings = request_timings(100, 5, {"generated": 0, "decode_ms": 0.0, "prompt_ms": 50.0})
+        self.assertEqual((timings["predicted_n"], timings["predicted_ms"]), (5, 0.0))
+        self.assertIsNone(timings["predicted_per_token_ms"])
+        self.assertIsNone(timings["predicted_per_second"])
+
     def test_prompt_n_is_what_was_read(self):
         # the slot already held the whole prompt: 1 token read, the conversation cache off (reused 0)
         last = {"prompt_ms": 563.5, "decode_ms": 700.0, "generated": 30, "reused": 0, "prompt_read": 1, "finish": "stop"}

serve/test_server.py

@@ -4526,8 +4526,8 @@ class CallingEngine(MockEngine):
     def generate(self, ids, max_new, sampling, cancel, embeddings=None):
         self.prompts.append(list(ids))
         prompt = self.tok.decode(ids)
-        if prompt.endswith("<function="):
-            text = "search>\n" + self.ARGS
+        if prompt.endswith("<tool_call>\n"):
+            text = "<function=search>\n" + self.ARGS
         elif prompt.endswith("<function=search>\n"):
             text = self.ARGS
         elif prompt.endswith("</think>\n\n"):
@@ -4647,6 +4647,43 @@ class ForcedToolChoice(unittest.TestCase):
                 if not stream:
                     self.assertIn("<function=execute_gdscript>", b["choices"][0]["message"]["content"])   # the text is untouched
 
+    def test_undeclared_tool_calls_call_hands_the_call_to_the_client(self):
+        """#4622: "undeclared_tool_calls": "call" returns the well-formed call of an undeclared name as a tool call (the client has its
+        own policy for unknown names); the default ("text") is the test above, unchanged."""
+        self.says(self._undeclared_call_answer("execute_gdscript"))
+        self.svc.undeclared_tool_calls = "call"
+        try:
+            for stream in (False, True):
+                with self.subTest(stream=stream):
+                    code, b = self.openai(stream=stream)
+                    finish, calls, _ = self.call_of(code, b, stream)
+                    self.assertEqual((finish, calls), ("tool_calls", [("execute_gdscript", {"q": "2+2"})]), b)
+                    self.assertIsNone(self._rejected_of(b, stream))
+            # prose that names the format is still text
+            self.says("Write <tool_call> and <function=x> to call a tool.")
+            code, b = self.openai(stream=False)
+            finish, calls, _ = self.call_of(code, b, False)
+            self.assertEqual((finish, calls), ("stop", []), b)
+        finally:
+            self.svc.undeclared_tool_calls = "text"
+
+    def test_undeclared_tool_calls_setting(self):
+        from serve.server import undeclared_tool_calls_setting
+        old = os.environ.pop("STRATA_UNDECLARED_TOOL_CALLS", None)
+        try:
+            self.assertEqual(undeclared_tool_calls_setting({}), "text")
+            self.assertEqual(undeclared_tool_calls_setting({"undeclared_tool_calls": "call"}), "call")
+            os.environ["STRATA_UNDECLARED_TOOL_CALLS"] = "call"
+            self.assertEqual(undeclared_tool_calls_setting({}), "call")
+            self.assertEqual(undeclared_tool_calls_setting({"undeclared_tool_calls": "text"}), "text")   # the config wins
+            for bad in ("yes", True, 1):
+                with self.assertRaises(SystemExit):
+                    undeclared_tool_calls_setting({"undeclared_tool_calls": bad})
+        finally:
+            os.environ.pop("STRATA_UNDECLARED_TOOL_CALLS", None)
+            if old is not None:
+                os.environ["STRATA_UNDECLARED_TOOL_CALLS"] = old
+
     def test_no_marker_when_nothing_was_rejected(self):
         """An ordinary answer and a declared call carry no extra field: the API is what it was."""
         from serve.frontend import CALL_END, CALL_START
@@ -4686,6 +4723,29 @@ class ForcedToolChoice(unittest.TestCase):
         self.assertEqual(msg["strata_rejected_tool_calls"], [{"name": "bash", "reason": "undeclared"}])
         self.assertFalse([c for c in msg["content"] if c["type"] == "tool_use"])
 
+    def test_any_model_name_on_the_anthropic_route_is_the_loaded_model(self):
+        """#5312: Claude Desktop sends the model name `claude-5-5-sonnet` and cannot change it; the route serves the loaded model whatever
+        the request calls it, and answers with that model's name (the Desktop app's own gateway is a separate check in apps/host)."""
+        self.says("Hello there.")
+        for name in ("claude-5-5-sonnet", "claude-opus-4-1", "gpt-4o", "nonsense"):
+            for stream in (False, True):
+                with self.subTest(model=name, stream=stream):
+                    body = {"model": name, "max_tokens": 50, "stream": stream, "messages": [{"role": "user", "content": "hi"}],
+                            "tools": [{"name": "search", "description": "search", "input_schema": {"type": "object", "properties": {}}}]}
+                    req = urllib.request.Request(self.base + "/v1/messages", data=json.dumps(body).encode(),
+                                                 headers={"Content-Type": "application/json"})
+                    with urllib.request.urlopen(req, timeout=30) as r:
+                        raw = r.read().decode()
+                        self.assertEqual(r.status, 200)
+                    if stream:
+                        events = [json.loads(l[6:]) for l in raw.splitlines() if l.startswith("data: {")]
+                        self.assertEqual("".join(e["delta"]["text"] for e in events if e["type"] == "content_block_delta"), "Hello there.")
+                        self.assertEqual(events[0]["message"]["model"], self.svc.model)
+                    else:
+                        msg = json.loads(raw)
+                        self.assertEqual(msg["model"], self.svc.model)
+                        self.assertEqual(msg["content"][0]["text"], "Hello there.")
+
     def test_a_call_with_a_dropped_function_keyword_is_still_the_call(self):
         """#3441: `<=search>` for `<function=search>`, as a model under a large tool surface writes it."""
         from serve.frontend import CALL_END, CALL_START
@@ -4706,7 +4766,7 @@ class ForcedToolChoice(unittest.TestCase):
                     self.assertEqual((finish, calls), ("tool_calls", [("search", {"q": "2+2"})]))
                     self.assertEqual(reasoning, CallingEngine.THOUGHT)
                     first, second = self.engine.prompts
-                    opening = "<tool_call>\n<function=" + ("search>\n" if choice == self.NAMED else "")
+                    opening = "<tool_call>\n" + ("<function=search>\n" if choice == self.NAMED else "")
                     # where the thinking ended, after the blank line the template puts before a call
                     self.assertEqual(second, first + self.tok.encode(CallingEngine.THOUGHT + "</think>\n\n" + opening))
 
@@ -4715,7 +4775,7 @@ class ForcedToolChoice(unittest.TestCase):
         thought, calling = 'I will search for "2+2"', self.engine.generate
 
         def generate(ids, max_new, sampling, cancel, embeddings=None):
-            if self.tok.decode(ids).endswith(("<function=", "<function=search>\n")):
+            if self.tok.decode(ids).endswith(("<tool_call>\n", "<function=search>\n")):
                 yield from calling(ids, max_new, sampling, cancel)
                 return
             self.engine.prompts.append(list(ids))
@@ -4728,7 +4788,7 @@ class ForcedToolChoice(unittest.TestCase):
                 finish, calls, reasoning = self.call_of(code, b, stream)
                 self.assertEqual((finish, calls, reasoning), ("tool_calls", [("search", {"q": "2+2"})], thought))
                 first, second = self.engine.prompts
-                self.assertEqual(second, first + self.tok.encode(thought + "</think>\n\n<tool_call>\n<function="))
+                self.assertEqual(second, first + self.tok.encode(thought + "</think>\n\n<tool_call>\n"))
 
     def test_without_thinking_the_prompt_ends_with_the_opening(self):
         for choice in ("required", self.NAMED):
@@ -4739,7 +4799,7 @@ class ForcedToolChoice(unittest.TestCase):
                     finish, calls, reasoning = self.call_of(code, b, stream)
                     self.assertEqual((finish, calls, reasoning), ("tool_calls", [("search", {"q": "2+2"})], ""))
                     self.assertEqual(len(self.engine.prompts), 1)
-                    opening = "<tool_call>\n<function=" + ("search>\n" if choice == self.NAMED else "")
+                    opening = "<tool_call>\n" + ("<function=search>\n" if choice == self.NAMED else "")
                     self.assertTrue(self.tok.decode(self.engine.prompts[0]).endswith("</think>\n\n" + opening))
 
     def test_the_budget_wrap_up_opens_the_call(self):
@@ -4749,7 +4809,7 @@ class ForcedToolChoice(unittest.TestCase):
         self.assertEqual((finish, calls), ("tool_calls", [("search", {"q": "2+2"})]))
         self.assertEqual(reasoning, CallingEngine.THOUGHT[:20] + REASONING_WRAP_UP.split("</think>")[0])
         first, second = self.engine.prompts
-        extra = self.tok.encode(REASONING_WRAP_UP + "<tool_call>\n<function=", parse_special=True)
+        extra = self.tok.encode(REASONING_WRAP_UP + "<tool_call>\n", parse_special=True)
         self.assertEqual(second, first + self.tok.encode(CallingEngine.THOUGHT[:20]) + extra)
 
     def test_auto_absent_and_none_leave_the_tools_to_the_model(self):

serve/test_tool_name_mcp_member.py

@@ -0,0 +1,73 @@
+"""serve/test_tool_name_mcp_member.py - #5281: a namespaced MCP tool called by its last segment is that tool.
+
+Seen live (Qwen3.8-Flash-Next Q2_0 under DeepSeek Harness, stream): the request declared `mcp__assets__get_asset_info` and
+`mcp__assets__get_maintenance_plan`; the model wrote `<function=get_asset_info>` and `<function=get_maintenance_plan>`.  Neither name was
+declared, so each block went back in `content` as XML, the harness got `stop` and the lookups never ran.  The member after the last `__`
+is the tool when exactly one declared tool ends that way; a declared name always wins, and two tools with the same ending are not guessed.
+
+    python -m unittest serve.test_tool_name_mcp_member -v
+"""
+import sys
+import unittest
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from serve.frontend import OutputParser  # noqa: E402
+
+def tool(name, prop="id"):
+    return {"name": name, "parameters": {"properties": {prop: {"type": "integer"}}}}
+
+TOOLS = [tool("read_skill_resource", "path"), tool("mcp__assets__get_asset_info"), tool("mcp__assets__get_maintenance_plan")]
+
+
+def call(name, arg="id", value="101"):
+    return f"<tool_call>\n<function={name}>\n<parameter={arg}>\n{value}\n</parameter>\n</function>\n</tool_call>"
+
+
+def run(text, tools=TOOLS, stream_tools=True, step=5, recover=True):
+    p = OutputParser(thinking=False, tools=tools, stream_tools=stream_tools, recover=recover, declared_only=True)
+    evs = []
+    for i in range(0, len(text), step):
+        evs += p.feed(text[i:i + step])
+    evs += p.finish("stop")
+    return evs
+
+
+def calls(evs):
+    return [(e.call.name, e.call.arguments) for e in evs if e.kind == "tool_call"]
+
+
+class McpMember(unittest.TestCase):
+    def test_the_last_segment_is_the_tool(self):
+        for stream in (False, True):
+            for step in (1, 5, 1000):
+                evs = run(call("get_asset_info") + "\n" + call("get_maintenance_plan"), stream_tools=stream, step=step)
+                self.assertEqual(calls(evs), [("mcp__assets__get_asset_info", {"id": 101}),
+                                              ("mcp__assets__get_maintenance_plan", {"id": 101})], (stream, step))
+                self.assertFalse([e for e in evs if e.kind == "content" and e.text.strip()])
+
+    def test_the_full_name_is_unchanged(self):
+        self.assertEqual(calls(run(call("mcp__assets__get_asset_info"))), [("mcp__assets__get_asset_info", {"id": 101})])
+
+    def test_two_tools_with_the_same_ending_are_not_guessed(self):
+        tools = [tool("mcp__a__search"), tool("mcp__b__search")]
+        evs = run(call("search"), tools)
+        self.assertEqual(calls(evs), [])
+        self.assertIn("<function=search>", "".join(e.text or "" for e in evs if e.kind == "content"))
+
+    def test_a_declared_name_wins(self):
+        tools = [tool("get_asset_info"), tool("mcp__assets__get_asset_info")]
+        self.assertEqual(calls(run(call("get_asset_info"), tools)), [("get_asset_info", {"id": 101})])
+
+    def test_an_unrelated_name_stays_text(self):
+        evs = run(call("get_asset"))
+        self.assertEqual(calls(evs), [])
+
+    def test_the_dotted_member_rule_is_unchanged(self):
+        tools = [tool("mcp__fs.write_file", "path")]
+        self.assertEqual(calls(run(call("write_file", "path", "x"), tools)), [("mcp__fs.write_file", {"path": "x"})])
+
+
+if __name__ == "__main__":
+    unittest.main()

serve/test_tool_name_prefix.py

@@ -0,0 +1,132 @@
+"""serve/test_tool_name_prefix.py - #5409: `<function=tools actio>` and the bare `<function=tools>` are a declared tool's call.
+
+Seen live (Qwen3.8-Flash-Next IQ3_S through Pi, after code-mode work that spells its tools `tools.actio(...)`): the model wrote
+the prefix of #4751 with a space for the dot, or alone.  Neither named a declared tool, so the block came back as answer text and
+the turn ended.  The space form is `tools.NAME` again; the bare prefix is the one declared tool that takes the call's parameters
+(only with tool_call_recovery).  Anything else stays text as before, and an exactly declared name always wins.
+
+    python -m unittest serve.test_tool_name_prefix -v
+"""
+import sys
+import unittest
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from serve.frontend import OutputParser  # noqa: E402
+
+ACTIO = {"name": "actio", "parameters": {"properties": {"command": {"type": "string"}}, "required": ["command"]}}
+READ = {"name": "read", "parameters": {"properties": {"path": {"type": "string"}}, "required": ["path"]}}
+BASH = {"name": "bash", "parameters": {"properties": {"command": {"type": "string"}, "timeout": {"type": "integer"}}}}
+
+
+def call(name, param="command", value="ls"):
+    return f"<tool_call>\n<function={name}>\n<parameter={param}>\n{value}\n</parameter>\n</function>\n</tool_call>"
+
+
+def run(text, tools, stream_tools=False, recover=True, thinking=False, step=3):
+    p = OutputParser(thinking=thinking, tools=tools, stream_tools=stream_tools, recover=recover)
+    evs = []
+    for i in range(0, len(text), step):
+        evs += p.feed(text[i:i + step])
+    evs += p.finish("stop")
+    return evs
+
+
+def calls(evs):
+    return [(e.call.name, e.call.arguments) for e in evs if e.kind == "tool_call"]
+
+
+def text(evs):
+    return "".join(e.text or "" for e in evs if e.kind in ("content", "reasoning"))
+
+
+class SpaceForm(unittest.TestCase):
+    def test_every_prefix_and_mode(self):
+        for prefix in ("tools", "functions", "default_api"):
+            for stream in (False, True):
+                for step in (1, 3, 1000):
+                    evs = run(call(f"{prefix} actio"), [ACTIO, READ], stream_tools=stream, step=step)
+                    self.assertEqual(calls(evs), [("actio", {"command": "ls"})], (prefix, stream, step))
+                    self.assertEqual(text(evs).strip(), "")
+
+    def test_without_recovery_too(self):
+        evs = run(call("tools actio"), [ACTIO], recover=False)
+        self.assertEqual(calls(evs), [("actio", {"command": "ls"})])
+
+    def test_text_before_the_call_is_kept(self):
+        evs = run("Let me look.\n" + call("tools actio"), [ACTIO])
+        self.assertEqual(calls(evs), [("actio", {"command": "ls"})])
+        self.assertEqual(text(evs).strip(), "Let me look.")
+
+    def test_in_the_reasoning(self):
+        evs = run("thinking...\n" + call("tools actio") + "\n</think>\n", [ACTIO], thinking=True)
+        self.assertEqual(calls(evs), [("actio", {"command": "ls"})])
+
+    def test_undeclared_name_stays_text(self):
+        for nm in ("tools nothing", "tools read now"):
+            evs = run(call(nm), [ACTIO, READ])
+            self.assertEqual(calls(evs), [], nm)
+            self.assertIn(f"<function={nm}>", text(evs))
+
+    def test_cut_off_name_is_flushed_as_text(self):
+        t = "<tool_call>\n<function=tools ac"
+        evs = run(t, [ACTIO])
+        self.assertEqual(calls(evs), [])
+        self.assertEqual(text(evs), t)
+
+    def test_two_calls(self):
+        evs = run(call("tools actio") + "\n" + call("tools read", "path", "/x"), [ACTIO, READ], stream_tools=True)
+        self.assertEqual(calls(evs), [("actio", {"command": "ls"}), ("read", {"path": "/x"})])
+
+
+class BarePrefix(unittest.TestCase):
+    def test_one_tool_fits(self):
+        for prefix in ("tools", "functions"):
+            for stream in (False, True):
+                for step in (1, 3, 1000):
+                    evs = run(call(prefix, "path", "/x"), [ACTIO, READ], stream_tools=stream, step=step)
+                    self.assertEqual(calls(evs), [("read", {"path": "/x"})], (prefix, stream, step))
+
+    def test_two_tools_fit_stays_text(self):
+        both = {"name": "also", "parameters": {"properties": {"path": {"type": "string"}}}}
+        evs = run(call("tools", "path", "/x"), [READ, both])
+        self.assertEqual(calls(evs), [])
+        self.assertIn("<function=tools>", text(evs))
+
+    def test_none_fits_stays_text(self):
+        evs = run(call("tools", "nope", "1"), [ACTIO, READ])
+        self.assertEqual(calls(evs), [])
+        self.assertIn("<function=tools>", text(evs))
+
+    def test_required_parameter_missing(self):
+        evs = run(call("tools", "timeout", "5"), [ACTIO, READ])    # `command` is required by actio
+        self.assertEqual(calls(evs), [])
+
+    def test_optional_only_call_fits_one(self):
+        evs = run(call("tools", "timeout", "5"), [BASH, READ])
+        self.assertEqual(calls(evs), [("bash", {"timeout": 5})])
+
+    def test_off_without_recovery(self):
+        evs = run(call("tools", "path", "/x"), [READ], recover=False)
+        self.assertEqual(calls(evs), [])
+        self.assertIn("<function=tools>", text(evs))
+
+    def test_a_declared_tool_named_tools_wins(self):
+        mine = {"name": "tools", "parameters": {"properties": {"x": {"type": "string"}}}}
+        evs = run(call("tools", "x", "1"), [mine, READ])
+        self.assertEqual(calls(evs), [("tools", {"x": "1"})])
+
+    def test_unfinished_bare_call_is_flushed_as_text(self):
+        t = "<tool_call>\n<function=tools>\n<parameter=path>\n/x"
+        evs = run(t, [READ], stream_tools=True)
+        self.assertEqual(calls(evs), [])
+        self.assertEqual(text(evs), t)
+
+    def test_no_tools_declared_is_unchanged(self):
+        evs = run(call("tools actio"), [])
+        self.assertEqual(calls(evs), [])
+
+
+if __name__ == "__main__":
+    unittest.main()

serve/test_tool_opener_misspelt.py

@@ -0,0 +1,104 @@
+"""serve/test_tool_opener_misspelt.py - #5276: `<toolcall>` (no underscore) before `<function=NAME>` of a declared tool opens the call.
+
+Seen live (Qwen3.8-Flash-Next UD-Q4_K_XL, OpenMinis on /v1/messages): the model wrote the opener as ordinary text, misspelled
+`<toolcall>`, then a whole call of a declared tool and the real `</tool_call>`.  The block came back as answer text, the turn
+ended and the agent waited for a result that never came.  The misspelling counts only where a `<tool_call>` would open a call
+(start of a line, outside code) and only when a declared tool's `<function=NAME>` follows; prose never triggers it.
+
+    python -m unittest serve.test_tool_opener_misspelt -v
+"""
+import sys
+import unittest
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from serve.frontend import OutputParser  # noqa: E402
+
+BASH = {"name": "bash", "parameters": {"properties": {"command": {"type": "string"}, "tooltitle": {"type": "string"}}}}
+BODY = ("<function=bash>\n<parameter=command>\ncd proj && rg -o 'x' landing.html | sort | uniq -c\n</parameter>\n"
+        "<parameter=tooltitle>\nCount\n</parameter>\n</function>\n")
+ARGS = {"command": "cd proj && rg -o 'x' landing.html | sort | uniq -c", "tooltitle": "Count"}
+
+
+def run(text, stream_tools=False, recover=True, thinking=False, step=3, tools=(BASH,)):
+    p = OutputParser(thinking=thinking, tools=list(tools), stream_tools=stream_tools, recover=recover)
+    evs = []
+    for i in range(0, len(text), step):
+        evs += p.feed(text[i:i + step])
+    evs += p.finish("stop")
+    return evs
+
+
+def calls(evs):
+    return [(e.call.name, e.call.arguments) for e in evs if e.kind == "tool_call"]
+
+
+def text(evs):
+    return "".join(e.text or "" for e in evs if e.kind in ("content", "reasoning"))
+
+
+class Misspelt(unittest.TestCase):
+    def test_issue_block(self):
+        for opener in ("<toolcall>", "<tool-call>"):
+            for stream in (False, True):
+                for step in (1, 3, 1000):
+                    for recover in (False, True):
+                        evs = run(opener + "\n" + BODY + "</tool_call>", stream, recover, step=step)
+                        self.assertEqual(calls(evs), [("bash", ARGS)], (opener, stream, step, recover))
+                        self.assertEqual(text(evs).strip(), "")
+
+    def test_without_the_closing_tag_as_the_real_opener(self):
+        self.assertEqual(calls(run("<toolcall>\n" + BODY)), calls(run("<tool_call>\n" + BODY)))
+
+    def test_mid_line_as_the_real_opener(self):
+        t = "ok <toolcall>\n" + BODY + "</tool_call>"
+        self.assertEqual(calls(run(t)), calls(run(t.replace("<toolcall>", "<tool_call>"))))
+        self.assertEqual(calls(run(t)), [("bash", ARGS)])
+
+    def test_in_the_reasoning_mid_line_stays_text(self):
+        evs = run("hm <toolcall>\n" + BODY + "</tool_call>\n</think>\n", thinking=True)
+        self.assertEqual(calls(evs), [])
+
+    def test_text_before_is_kept(self):
+        evs = run("I will count.\n<toolcall>\n" + BODY + "</tool_call>")
+        self.assertEqual(calls(evs), [("bash", ARGS)])
+        self.assertEqual(text(evs).strip(), "I will count.")
+
+    def test_in_the_reasoning(self):
+        evs = run("hm\n<toolcall>\n" + BODY + "</tool_call>\n</think>\n", thinking=True)
+        self.assertEqual(calls(evs), [("bash", ARGS)])
+
+    def test_prose_naming_the_tag_stays_text(self):
+        for t in ("The model may write <toolcall> by mistake.",
+                  "Write <toolcall> and then a call.\n",
+                  "<toolcall>\nnot a call\n</tool_call>",
+                  "<toolcall>",
+                  "<toolcall>\n<function=nothing>\n<parameter=a>\n1\n</parameter>\n</function>\n</tool_call>"):
+            for stream in (False, True):
+                evs = run(t, stream)
+                self.assertEqual(calls(evs), [], t)
+                self.assertEqual(text(evs), t)
+
+    def test_code_fence_stays_text(self):
+        t = "Example:\n```\n<toolcall>\n" + BODY + "</tool_call>\n```\n"
+        evs = run(t)
+        self.assertEqual(calls(evs), [])
+        self.assertEqual(text(evs), t)
+
+    def test_indented_code_stays_text(self):
+        t = "Example:\n\n    <toolcall>\n    " + BODY.replace("\n", "\n    ") + "</tool_call>\n"
+        self.assertEqual(calls(run(t)), [])
+
+    def test_no_tools_declared(self):
+        t = "<toolcall>\n" + BODY + "</tool_call>"
+        self.assertEqual(calls(run(t, tools=())), [])
+        self.assertEqual(text(run(t, tools=())), t)
+
+    def test_the_real_opener_is_unchanged(self):
+        evs = run("<tool_call>\n" + BODY + "</tool_call>")
+        self.assertEqual(calls(evs), [("bash", ARGS)])
+
+
+if __name__ == "__main__":
+    unittest.main()

serve/test_yield_accounting.py

@@ -0,0 +1,132 @@
+"""Real serving flow over a fake native engine with accurate partial-read DONE fields."""
+
+import threading, time, unittest
+from unittest import mock
+from serve import test_parallel as tp
+
+
+def accurate_fake():
+    source = tp.FAKE_BATCH
+    source = source.replace("def reused(ids):", "partial = {}\ndef reused(ids):", 1)
+    old = "    return 0 if not REUSE else len(ids) - 1 if continued(ids)[1] else 3"
+    assert source.count(old) == 1
+    source = source.replace(
+        old,
+        """    if not REUSE: return 0
+    kept = max((len(h) for h in partial.values() if ids[:len(h)] == h), default=0)
+    return max(kept, len(ids) - 1 if continued(ids)[1] else 3)""",
+    )
+    old = "        given = None\n        for q in range(CH, len(ids) - 1, CH):"
+    assert source.count(old) == 1
+    source = source.replace(
+        old,
+        "        read_from = reused(ids)\n        given = None\n        for q in range((read_from // CH + 1) * CH, len(ids) - 1, CH):",
+    )
+    old = '            print(f"DONE 0 {len(ids)} 5.0 0.0 cancel 0 0 {reused(ids)}", flush=True)'
+    assert source.count(old) == 1
+    source = source.replace(
+        old,
+        """            partial[given[0]] = ids[:given[1]]
+            print(f"DONE 0 {len(ids)} 5.0 0.0 cancel 0 0 {read_from}" +
+                  (f" 0 0 0 0 0.0 {given[1] - read_from}" if READF else ""), flush=True)""",
+    )
+    source = source.replace("{len(ids) - reused(ids)}", "{len(ids) - read_from}")
+    source = source.replace("{reused(ids)}", "{read_from}")
+    compile(source, "accurate_fake", "exec")
+    return source
+
+
+class YieldAccounting(tp.FakeEngineCase):
+    def start_accurate(self, slots):
+        with mock.patch.object(tp, "FAKE_BATCH", accurate_fake()):
+            self.start(slots, reuse=True, more=["--readfield"])
+
+    def race_long(self):
+        replies = []
+        errors = []
+
+        def long():
+            try:
+                replies.append(self.chat("long " * 600, max_tokens=64))
+            except Exception as error:
+                errors.append(repr(error))
+
+        thread = threading.Thread(target=long)
+        thread.start()
+        time.sleep(0.15)
+        self.chat("short", max_tokens=64)
+        thread.join(20)
+        self.assertFalse(thread.is_alive())
+        self.assertEqual(errors, [])
+        self.assertEqual(len(replies), 1)
+        return replies[0]
+
+    def check_full_input(self, response):
+        n = response["usage"]["prompt_tokens"]
+        log = self.log.read_text().splitlines()
+        self.assertTrue(any(x.startswith("YIELD ") for x in log), log)
+        legs = [x for x in log if x.startswith(("GEN ", "BGEN ")) and int(x.split()[2]) == n]
+        self.assertGreaterEqual(len(legs), 2)
+        self.assertEqual(response["usage"]["prompt_tokens_details"]["cached_tokens"], 3)
+        self.assertEqual((response["timings"]["cache_n"], response["timings"]["prompt_n"]), (3, n - 3))
+        self.assertEqual(response["timings"]["prompt_ms"], 5.0 * len(legs))
+        row = max(self.svc.history, key=lambda x: x["prompt_tokens"])
+        self.assertEqual((row["reused"], row["prompt_read"]), (3, n - 3))
+
+    def test_solo_yield_keeps_original_cache_and_sums_prompt_work(self):
+        self.start_accurate(2)
+        self.check_full_input(self.race_long())
+
+    def test_admission_yield_keeps_original_cache_and_sums_prompt_work(self):
+        self.start_accurate(3)
+        errors = []
+
+        def background():
+            try:
+                self.chat("LONGREPLY background", max_tokens=60)
+            except Exception as error:
+                errors.append(repr(error))
+
+        bg = threading.Thread(target=background)
+        bg.start()
+        time.sleep(0.05)
+        response = self.race_long()
+        bg.join(20)
+        self.assertFalse(bg.is_alive())
+        self.assertEqual(errors, [])
+        self.check_full_input(response)
+
+    def cancel_after_yields(self, count):
+        self.start_accurate(2)
+        original = self.engine._take_control
+        seen = 0
+
+        def take(cancel, plen, after_epoch=None, **kwargs):
+            nonlocal seen
+            if after_epoch is not None:
+                seen += 1
+                if seen == count:
+                    cancel.set()
+                    return False
+                return (yield from original(cancel, plen, **kwargs))
+            return (yield from original(cancel, plen, **kwargs))
+
+        with mock.patch.object(self.engine, "_shorter_waiting", return_value=True), mock.patch.object(
+            self.engine, "_take_control", side_effect=take
+        ):
+            self.chat("long " * 600, max_tokens=64)
+        positions = [int(x.split()[2]) for x in self.log.read_text().splitlines() if x.startswith("YIELD ")]
+        self.assertEqual(len(positions), count)
+        self.assertEqual(positions, sorted(set(positions)))
+        row = self.svc.history[-1]
+        self.assertEqual(row["finish"], "cancel")
+        self.assertEqual(
+            (row["prompt_tokens"], row["reused"], row["prompt_read"]), (positions[-1], 3, positions[-1] - 3)
+        )
+        self.assertEqual(row["prompt_ms"], 5.0 * count)
+
+    def test_cancel_after_one_yield_counts_that_done_once(self):
+        self.cancel_after_yields(1)
+
+    def test_cancel_after_two_yields_counts_each_done_once(self):
+        self.cancel_after_yields(2)

setup.py

@@ -268,7 +268,7 @@ CUDA12_WHEELS = ["nvidia-cublas-cu12==12.9.1.4", "nvidia-cuda-runtime-cu12==12.9
 # (575.51 / 576.02), which `strata update` enforces.  cuda12_min_driver() reads that from the package or the feed (#4247 #4188).
 CUDA12_COMPAT_DRIVER = 528 if WIN else 525
 ENGINE12_DIR = "engine-cuda12"
-MIN_ENGINE = (0, 1, 46)                # versions compare all numbers; v0.1.46: the 0.2 release engine (UPDATE.bat and setup fetch it from the signed feed, #3469 #3437); v0.1.45: Qwen3.6 parallel requests and pictures, fixes #2346 #2372 #2396; v0.1.44.1: CPUs without AVX2 start again (#2324: an illegal instruction before main in the NVFP4 CPU rows), the CUDA 12.0/12.1 toolkits build (#2322), a crashed engine start is named instead of retried as a GPU runtime start, Mac M2 (#2323); v0.1.44: UD-Q4/Q5/Q6_K_XL and Q8_0 Unsloth files in setup (K-quant experts, MMQ prompt library strata_kq), NVFP4 packs, Qwen3.6-35B-A3B, Intel Arc A-series kernels, the pipelined-decode and latency work; v0.1.43.1: the adaptive overlap no longer collapses decode on pinned-host desktops (#1964 #1975), Windows PLE mmap stall (#1984), pinned prompt uploads (#1996), the I/O watchdog allowance; v0.1.43: adaptive-tier copies overlap on CUDA, P100 (sm_60) kernel tables, RDNA4 (gfx1200/gfx1201) defaults (route tail skip, measured PCIe share, 16K prompt chunks, fused IQ3 prompt experts); v0.1.42.1: the GGUF-in-place file tier on Windows unmaps every shard, dual-socket starts judge each NUMA node and STRATA_DROP_CACHE_ON_EXIT, tools/strata_bench.py; v0.1.42: STRATA_ROUTE_TAIL_SKIP=7 default on CUDA (+13-16% decode), the AMD 2-GPU checkpoint fix (H13), Phase 2 fixes and opt-ins; v0.1.41.1 (folded into 0.1.42): the #1576 card order, #1616 stage size, #1595 CPU share, #1630/#1607 Windows HIP chunk cap and commit-limit warning, #1603 parallel-request bookkeeping; v0.1.41: CPU prefill share on by default (NVIDIA, chunks < 1024), --batch-groups auto, the file tier read in batches on Windows, Pascal decode fix, per-arch tables; v0.1.40.3: the #1357 MTP router guard, the #1376 Windows HIP cache floor, #461 runtime DLLs, Intel A750 first-request fix; v0.1.40.2: F4 verify windows, the Linux file tier (#1194), the stager wait (#1057), #1264/#1201/#1139 fixes, opt-in CPU share (#1282), Intel Arc; v0.1.40: --resident-experts on a layer split with the split+resident variant (#848), --kv k8v4 with KV streaming (#711); v0.1.39: the #577 file-tier regression fixed, the OpenAI Responses API (#451, Codex), a reply stuck on one token ended (#606), the head before the arena (#620), effort_position (#458), --vram-reserve hot resize opt-in (#533), PR batch; v0.1.38: prompts faster (one gather per expert group #372, the first chunk's PLE rows beside layer 0 #374, DeltaNet three heads per thread #413), --kv q4_0 prompts on tensor cores (#452), Q5_0 experts on the GPU (#473), IQ4_XS on AVX-2 (#415), unbuffered expert loading on Windows (#357 #362), --peer-device (#531), a 6 GB card starts (#496), PR batch; v0.1.37: a silent engine is restarted (#481), Windows AMD counts the desktop's VRAM (#380 #377 #497), a steadier PCIe probe (#485), fixes #496 #495 #498 #505 #493; v0.1.36: a cancelled prompt logged as read so far (#471), the draft-head hint (#474), UPDATE.bat (#475), --expert-profile-save (#477); v0.1.35: Windows AMD uses its bundled HIP runtime (#468 #461), the low-RAM resident mode on Windows 32 GB (#467), fixes #460 #459 #446 #447 #457 #448 #444; v0.1.34: AMD on Windows (a ready-made HIP engine), an MCP server for AI assistants (tools/strata_mcp.py), a shorter README; v0.1.33: a portable image encoder again (#411 #412), setup recommends instead of forcing (#406 #403 #364 #384), fixes #352 #365 #369 #371 #375 #393 #408 #414; v0.1.32: split prompts faster (#340), AMD router +12%, Unsloth Q4 in setup, faster Q4 prompts, #326/#327/#342/#344 fixes, PR batch; v0.1.31: Unsloth UD-Q4_K_XL (experimental), GGUF-in-place low-RAM mode, Windows GGUF load 2x, server race + tokenizer fixes, AMD intrinsics; v0.1.30: short prompts faster (streaming from 1024 tokens), resident low-RAM variant, multi-GPU session carve, RDNA4; v0.1.29: sampled answers faster (split top-k), #154 correctness fixes; v0.1.28: the expert cache reserves the draft head, a cancelled request no longer fails the next; v0.1.27: RTX 20 (sm_75) in the ready-made engine, the HIP build without CUDA headers; v0.1.26: the draft layer's prompt pass in batches; v0.1.25: faster prompts (grouping off the copy engine, fused hyper-connection kernels), AMD HIP backend, --kv k8v4; v0.1.24: long prompts faster (QSA select on tensor cores); v0.1.23: image requests honor sampling, 8 GB cards start, batched verify window; v0.1.22: faster prompts (tensor-core attention), multi-GPU across images/steering/KV streaming; v0.1.21: multi-GPU layer split (--gpus); v0.1.20: system-prompt checkpoint, PCIe probe, hit rate; v0.1.19: penalties
+MIN_ENGINE = (0, 1, 46)                # versions compare all numbers; v0.1.58: engine rounds 0215-0217 (tuned drafters, the file-tier re-check, prefill and Q5_0 fixes; git log c3b3f5035..HEAD); v0.1.46: the 0.2 release engine (UPDATE.bat and setup fetch it from the signed feed, #3469 #3437); v0.1.45: Qwen3.6 parallel requests and pictures, fixes #2346 #2372 #2396; v0.1.44.1: CPUs without AVX2 start again (#2324: an illegal instruction before main in the NVFP4 CPU rows), the CUDA 12.0/12.1 toolkits build (#2322), a crashed engine start is named instead of retried as a GPU runtime start, Mac M2 (#2323); v0.1.44: UD-Q4/Q5/Q6_K_XL and Q8_0 Unsloth files in setup (K-quant experts, MMQ prompt library strata_kq), NVFP4 packs, Qwen3.6-35B-A3B, Intel Arc A-series kernels, the pipelined-decode and latency work; v0.1.43.1: the adaptive overlap no longer collapses decode on pinned-host desktops (#1964 #1975), Windows PLE mmap stall (#1984), pinned prompt uploads (#1996), the I/O watchdog allowance; v0.1.43: adaptive-tier copies overlap on CUDA, P100 (sm_60) kernel tables, RDNA4 (gfx1200/gfx1201) defaults (route tail skip, measured PCIe share, 16K prompt chunks, fused IQ3 prompt experts); v0.1.42.1: the GGUF-in-place file tier on Windows unmaps every shard, dual-socket starts judge each NUMA node and STRATA_DROP_CACHE_ON_EXIT, tools/strata_bench.py; v0.1.42: STRATA_ROUTE_TAIL_SKIP=7 default on CUDA (+13-16% decode), the AMD 2-GPU checkpoint fix (H13), Phase 2 fixes and opt-ins; v0.1.41.1 (folded into 0.1.42): the #1576 card order, #1616 stage size, #1595 CPU share, #1630/#1607 Windows HIP chunk cap and commit-limit warning, #1603 parallel-request bookkeeping; v0.1.41: CPU prefill share on by default (NVIDIA, chunks < 1024), --batch-groups auto, the file tier read in batches on Windows, Pascal decode fix, per-arch tables; v0.1.40.3: the #1357 MTP router guard, the #1376 Windows HIP cache floor, #461 runtime DLLs, Intel A750 first-request fix; v0.1.40.2: F4 verify windows, the Linux file tier (#1194), the stager wait (#1057), #1264/#1201/#1139 fixes, opt-in CPU share (#1282), Intel Arc; v0.1.40: --resident-experts on a layer split with the split+resident variant (#848), --kv k8v4 with KV streaming (#711); v0.1.39: the #577 file-tier regression fixed, the OpenAI Responses API (#451, Codex), a reply stuck on one token ended (#606), the head before the arena (#620), effort_position (#458), --vram-reserve hot resize opt-in (#533), PR batch; v0.1.38: prompts faster (one gather per expert group #372, the first chunk's PLE rows beside layer 0 #374, DeltaNet three heads per thread #413), --kv q4_0 prompts on tensor cores (#452), Q5_0 experts on the GPU (#473), IQ4_XS on AVX-2 (#415), unbuffered expert loading on Windows (#357 #362), --peer-device (#531), a 6 GB card starts (#496), PR batch; v0.1.37: a silent engine is restarted (#481), Windows AMD counts the desktop's VRAM (#380 #377 #497), a steadier PCIe probe (#485), fixes #496 #495 #498 #505 #493; v0.1.36: a cancelled prompt logged as read so far (#471), the draft-head hint (#474), UPDATE.bat (#475), --expert-profile-save (#477); v0.1.35: Windows AMD uses its bundled HIP runtime (#468 #461), the low-RAM resident mode on Windows 32 GB (#467), fixes #460 #459 #446 #447 #457 #448 #444; v0.1.34: AMD on Windows (a ready-made HIP engine), an MCP server for AI assistants (tools/strata_mcp.py), a shorter README; v0.1.33: a portable image encoder again (#411 #412), setup recommends instead of forcing (#406 #403 #364 #384), fixes #352 #365 #369 #371 #375 #393 #408 #414; v0.1.32: split prompts faster (#340), AMD router +12%, Unsloth Q4 in setup, faster Q4 prompts, #326/#327/#342/#344 fixes, PR batch; v0.1.31: Unsloth UD-Q4_K_XL (experimental), GGUF-in-place low-RAM mode, Windows GGUF load 2x, server race + tokenizer fixes, AMD intrinsics; v0.1.30: short prompts faster (streaming from 1024 tokens), resident low-RAM variant, multi-GPU session carve, RDNA4; v0.1.29: sampled answers faster (split top-k), #154 correctness fixes; v0.1.28: the expert cache reserves the draft head, a cancelled request no longer fails the next; v0.1.27: RTX 20 (sm_75) in the ready-made engine, the HIP build without CUDA headers; v0.1.26: the draft layer's prompt pass in batches; v0.1.25: faster prompts (grouping off the copy engine, fused hyper-connection kernels), AMD HIP backend, --kv k8v4; v0.1.24: long prompts faster (QSA select on tensor cores); v0.1.23: image requests honor sampling, 8 GB cards start, batched verify window; v0.1.22: faster prompts (tensor-core attention), multi-GPU across images/steering/KV streaming; v0.1.21: multi-GPU layer split (--gpus); v0.1.20: system-prompt checkpoint, PCIe probe, hit rate; v0.1.19: penalties
 # KV bytes per context token and attention layer: 8-bit 1056, rotated 4-bit 576, hybrid K8V4 (8-bit K, 4-bit V) 816
 KV_CELL_BYTES = {"q4_0": 576, "k8v4": 816}
 PY_PACKAGES = ["numpy", "jinja2", "regex", "pyyaml", "tqdm", "requests", "pillow", "psutil"]
@@ -1130,7 +1130,7 @@ def run(cmd, cwd=None, env=None, check=True, quiet=False):
     if CONTROL:                                        # the app: only this Python running a bundled tool
         argv = [str(c) for c in cmd]
         if len(argv) < 2 or os.path.normcase(argv[0]) != os.path.normcase(sys.executable) or argv[1] == "-m" or \
-                Path(argv[1]).resolve().parent not in ((ROOT / "tools").resolve(), (ROOT / "tools" / "dsv41").resolve()):
+                Path(argv[1]).resolve().parent not in ((ROOT / "tools").resolve(), (ROOT / "tools" / "dsv41").resolve(), (ROOT / "metal" / "tools").resolve()):
             raise acquisition.AcquisitionRefused("the desktop app's installer only runs its bundled preparation tools; "
                                                  + " ".join(argv[:2]))
     gguf_py = (env or {}).get("STRATA_GGUF_PY")
@@ -3284,6 +3284,14 @@ def download(url, dst: Path, what=None):
             mark(dst, f"sha256 {pin_here[1]}")
             ok(f"{what or dst.name} is already here (checked)")
             return
+    bad = dst.with_name(dst.name + ".bad")
+    if pin_here and not dst.exists() and bad.is_file():
+        # an earlier run kept this file as .bad (keep_bad): hash it again before anything is downloaded; if it matches the pin now it is the file
+        if bad.stat().st_size == pin_here[0] and _hash_file(bad, pin_here[0]) == pin_here[1]:
+            os.replace(bad, dst)
+            mark(dst, f"sha256 {pin_here[1]}")
+            ok(f"{what or dst.name} is already here (checked)")
+            return
     acquisition.require(what or dst.name)
     if not url.startswith(("http://", "https://")):
         src = Path(url[7:] if url.startswith("file://") else url)
@@ -3339,12 +3347,20 @@ def download(url, dst: Path, what=None):
         fail(f"{what or dst.name} is {total:,} bytes on {url.split('/')[2]}, not the pinned {pin[0]:,}",
              "update Strata (the file there has changed since this version pinned it)")
     label = what or dst.name
+    if bad.is_file() and not dst.exists() and total:
+        try:
+            if shutil.disk_usage(dst.parent).free < total + (1 << 30):
+                bad.unlink()                           # no room for the new copy beside the kept one
+                say(f"  Not enough free space for a new copy of {dst.name} beside the kept one ({bad.name}): the kept file was removed.")
+        except OSError:
+            pass
     conns = dl_connections()
     fixed = pin if pin else (meta if meta is not None and meta[1] else None)
     again = (lambda: repair_download(url, dst, total, label, conns)) if total >= DL_MIN_PARALLEL and conns > 1 else None
     if dst.exists() and total and dst.stat().st_size == total:    # finished by an older setup (no mark yet), or stopped before the check
         if fixed:                                      # a file that has a fixed hash is checked, not just marked
             verify_sha256(dst, *fixed, repair=again)
+            drop_bad(dst)
         else:
             mark(dst)
         ok(f"{what or dst.name} already downloaded")
@@ -3362,9 +3378,11 @@ def download(url, dst: Path, what=None):
     for side in (".chunks", ".src"):
         part.with_name(part.name + side).unlink(missing_ok=True)
     if pin:
-        verify_sha256(dst, *pin, repair=again)         # the fixed hash: a wrong file is repaired, else deleted and setup stops
+        verify_sha256(dst, *pin, repair=again)         # the fixed hash: a wrong file is repaired, else kept as .bad and setup stops
+        drop_bad(dst)
     elif meta is not None and meta[1]:
         verify_sha256(dst, meta[0], meta[1], repair=again)   # ModelScope's published hash; kept in the finish mark
+        drop_bad(dst)
     else:
         if ms is not None:
             warn(f"{what or dst.name}: ModelScope publishes no SHA-256 for it: the file is not verified")
@@ -3516,6 +3534,29 @@ def own_gguf_is_whole(path: Path) -> bool:
         return False
 
 
+def keep_bad(s: Path) -> Path | None:
+    """A downloaded file whose SHA-256 is not the pinned one is not deleted: it is renamed to <name>.bad (no finish mark), so a wrong check, a pin that
+    is out of date or a bad read costs one more hash and not another 50 GB download.  The next run of download() hashes the .bad file again and takes it
+    back when it now matches; a successful new download of the file, or too little free space for it, removes the .bad file (drop_bad).
+    Returns the .bad path, or None when the file could not be renamed (it is then deleted, as before)."""
+    bad = s.with_name(s.name + ".bad")
+    try:
+        bad.unlink(missing_ok=True)                    # at most one .bad per file
+        os.replace(s, bad)
+        return bad
+    except OSError:
+        s.unlink(missing_ok=True)
+        return None
+
+
+def drop_bad(dst: Path) -> None:
+    """The .bad copy of `dst` is not needed any more: a good file is in place."""
+    try:
+        dst.with_name(dst.name + ".bad").unlink(missing_ok=True)
+    except OSError:
+        pass
+
+
 def verify_sha256(s: Path, size: int, sha: str, own: bool = False, repair=None, also: tuple = ()) -> None:
     """A shard's size and SHA-256 against the pinned values (the Unsloth file); the result is kept in its finish mark,
     so the ~5 minutes of hashing 111 GB happen once.  A wrong file is deleted, so the next run downloads it again.
@@ -3539,9 +3580,11 @@ def verify_sha256(s: Path, size: int, sha: str, own: bool = False, repair=None,
         fail(f"{s.name} is {have:,} bytes, not {size:,}", "delete it and run setup again (the download restarts)")
     say(f"  checking {s.name} (SHA-256, {size / 1e9:.1f} GB) ...")
     digest = _hash_file(s, size)
+    server_same = False                                # the repair read the whole file from the server and found no block that differs
     if digest != sha and repair is not None and not own:
         say(f"  Some parts of {s.name} were damaged; downloading only those parts again.")
         res = repair()
+        server_same = res is not None and res[0] == 0
         if res is not None:
             say(f"  {res[0]} of {res[1]} parts were different and have been written again; checking the whole file again ...")
             digest = _hash_file(s, size)
@@ -3559,10 +3602,14 @@ def verify_sha256(s: Path, size: int, sha: str, own: bool = False, repair=None,
         if own:
             fail(f"{s.name} does not match the published file (SHA-256 {digest}, expected {sha})",
                  "it may be damaged or another version; your file was left as it is. Copy the whole published file into the folder")
-        s.unlink(missing_ok=True)
+        keep_bad(s)                                    # kept as <name>.bad, not thrown away (see keep_bad)
         m.unlink(missing_ok=True)
-        fail(f"{s.name} has the wrong SHA-256 ({digest}, expected {sha}): deleted",
-             "run setup again to download it again")
+        if server_same:
+            hint = ("the server sends exactly the bytes you have (every block was compared), so the file there is not the version this Strata pins: "
+                    "update Strata. Run setup again to check the kept file once more")
+        else:
+            hint = "run setup again: the kept file is checked once more, and if it still differs it is downloaded again and the kept one is removed"
+        fail(f"{s.name} does not match the pinned SHA-256 (it is {digest}, Strata expects {sha}); the file was kept as {s.name}.bad", hint)
     mark(s, f"sha256 {sha}")
 
 
@@ -11060,9 +11107,10 @@ def main() -> int:
                    f"so setup leaves {UNSLOTH_RAM_LEFT_GB} GB of the RAM for the OS and the desktop (#1715). "
                    "--expert-cache N in the config's args sets it")
             else:
-                warn(f"this GPU shares the RAM, and the {budget:g} GiB RAM budget leaves almost none for the expert "
-                     "cache: it stays on auto, which may take the room the OS needs (#1715). A smaller "
-                     "--resident-budget-gib leaves room")
+                # Round 0215 (owner 11-05): on a Windows Strix Halo the cache lives in the GPU's own pool (the BIOS carve-out + the shared memory), not in the RAM the OS
+                # needs, so the earlier "may take the room the OS needs" warning was misleading there; the cache stays on auto and this says where it is sized from.
+                carve = float(gpu.get("dedicated_gb") or 0.0)
+                ok(f"The expert cache is sized from the graphics memory ({f'{carve:g} GB ' if carve > 0 else ''}carve-out + shared), not from the RAM.")
     if vision != "none":
         args += ["--vision"]
         if VISION[vision]["reserve_mib"]:

src/core/arch_defaults.cpp

@@ -95,6 +95,12 @@ std::vector<std::pair<std::string, std::string>> arch_default_env(const char* gc
         for (auto it = t.begin(); it != t.end(); ++it) {
             if (it->first == "STRATA_SH_STREAM") { t.erase(it); break; }
         }
+        // round 0215 (STRIX-PK): two load-ordering switches of the decode kernels, each the same arithmetic in the same order as without it
+        // (the kernel hashes are equal; first-token logits and outputs identical in the engine).  UD-Q4_K_XL on a Strix Halo Windows PC, DFlash2
+        // serve, 4 request rounds x 2 passes x 4 prompts, interleaved: window time -0.25% (25 of 32 pairs faster, the same window and token count in
+        // every pair), alone +0.2% / +0.3%.  Small but consistent; Linux gfx1151 is unmeasured and keeps its table.
+        t.push_back({"STRATA_SMALL_UNR", "3"});
+        t.push_back({"STRATA_GR_Q8_PFS", "1"});
     }
     return t;
 }
@@ -177,6 +183,10 @@ std::vector<std::pair<std::string, std::string>> igpu_default_env(const char* gc
             // interleaved x3, same 16 greedy ids: IQ3_S 4K 1,187 -> 1,296 and 8K 1,285 -> 1,399 tok/s (x1.09), UD-IQ4_XS 4K 1,119 ->
             // 1,221 and 8K 1,180 -> 1,270; decode unchanged.  The rounding differs from the int8 kernels (KL is information only).
             {"STRATA_PF_F16", "1"},
+            // Round 0215 (STRIX-FIT): the down rows of the F16 expert kernels go to the combine as FP16 (half the bytes written and read).  Windows gfx1151,
+            // interleaved 8K pairs, same engine: IQ3_S x1.022 (4 of 4 pairs), UD-IQ4_XS x1.026 (3 of 3), UD-Q4_K_XL x1.012 (3 of 4); the first-token KL
+            // is 1e-4 at 8K and 0.02-0.04 on agent transcripts (information).  STRATA_PF_DM16=0 turns it off.
+            {"STRATA_PF_DM16", "1"},
         };
     }
     return t;
@@ -257,6 +267,11 @@ std::vector<std::string> apply_arch_defaults(const char* gcn_arch, bool integrat
             std::fprintf(stderr, "strata: gfx12 (RDNA4): the matrix-core QSA block scorer is on once a prompt is past 28,672 tokens (+5.7%% prompt speed at 98K on an R9700, shorter prompts unchanged; "
                                  "the first-token KL against the warp kernel is 0.002 on average above 30K; STRATA_SELECT_WMMA=0 or STRATA_GFX12_DEFAULTS=0 turns it off)\n");
     }
+    if (strata::kernels::gfx_arch_is_gfx12(gcn_arch) && std::getenv("STRATA_GDN_CHUNKED") == nullptr) {   // the length-gated chunked GDN scan (prefill/kernels.cu: gdn_chunked_auto)
+        const char* off12 = std::getenv("STRATA_GFX12_DEFAULTS");
+        if (!(off12 != nullptr && off12[0] == '0'))
+            std::fprintf(stderr, "strata: RDNA4 (gfx12): prompt chunks of 30,000+ tokens use the chunked GDN scan (STRATA_GDN_CHUNKED=0 turns it off)\n");
+    }
     if (!set.empty()) {
         const bool g12 = strata::kernels::gfx_arch_is_gfx12(gcn_arch);
         std::fprintf(stderr, "strata: %s: %zu exact speed switches on by default (%s=0 turns "

src/core/arch_defaults_test.cpp

@@ -90,8 +90,14 @@ int main() {
             CHECK(find(win26, "STRATA_SH_STREAM") != nullptr);     // STRATA_DEFAULTS=026: v0.2.6's table on Windows too
             CHECK(find(lin26, "STRATA_SH_STREAM") != nullptr);
             CHECK(win26 == lin26 && win26 == lin);                 // under 026 and on Linux the whole table is the same
-            CHECK(win.size() + 1 == lin.size());                   // Windows loses that one entry and nothing else
-            for (const auto& kv : win) CHECK(find(lin, kv.first.c_str()) != nullptr && *find(lin, kv.first.c_str()) == kv.second);
+            CHECK(win.size() == lin.size() + 1);                   // Windows loses SH_STREAM and gains the two round-0215 decode switches
+            CHECK(find(win, "STRATA_SMALL_UNR") != nullptr && *find(win, "STRATA_SMALL_UNR") == "3" && find(lin, "STRATA_SMALL_UNR") == nullptr);
+            CHECK(find(win, "STRATA_GR_Q8_PFS") != nullptr && *find(win, "STRATA_GR_Q8_PFS") == "1" && find(lin, "STRATA_GR_Q8_PFS") == nullptr);
+            CHECK(find(win26, "STRATA_SMALL_UNR") == nullptr && find(win26, "STRATA_GR_Q8_PFS") == nullptr);   // STRATA_DEFAULTS=026 keeps v0.2.6's table
+            for (const auto& kv : win) {
+                if (kv.first == "STRATA_SMALL_UNR" || kv.first == "STRATA_GR_Q8_PFS") continue;
+                CHECK(find(lin, kv.first.c_str()) != nullptr && *find(lin, kv.first.c_str()) == kv.second);
+            }
         }
         // other architectures: the platform changes nothing
         for (const char* a : {"gfx1100", "gfx1030", "gfx1103", "gfx1150", "gfx1152", "gfx1201", "gfx1200", ""}) {
@@ -218,7 +224,7 @@ int main() {
         // 0.2.8: Strix Halo (gfx1151, integrated) gets the matrix-core prompt set on Windows; the 780M and the 890M are
         // not measured there and stay off
         const bool win_on = m.expect_on && m.integrated && std::string(m.gcn).rfind("gfx1151", 0) == 0;
-        CHECK(igpu_default_env(m.gcn, m.integrated).size() == (win_on ? 7u : 0u));
+        CHECK(igpu_default_env(m.gcn, m.integrated).size() == (win_on ? 8u : 0u));
 #else
         CHECK(igpu_default_env(m.gcn, m.integrated).size() == (want ? 5u : 0u));
 #endif
@@ -252,10 +258,10 @@ int main() {
 #if defined(_WIN32)
             halo_win = std::string(m.gcn).rfind("gfx1151", 0) == 0;
 #endif
-            CHECK(t.size() == (halo_win ? 7u : 5u));
+            CHECK(t.size() == (halo_win ? 8u : 5u));
             std::vector<const char*> want = {"STRATA_HIP_WMMA", "STRATA_PF_GEMM", "STRATA_PA_FAST", "STRATA_HC_UPMIX"};
             want.push_back("STRATA_SELECT_WMMA");
-            if (halo_win) { want.push_back("STRATA_PF_FUSED"); want.push_back("STRATA_PF_F16"); }
+            if (halo_win) { want.push_back("STRATA_PF_FUSED"); want.push_back("STRATA_PF_F16"); want.push_back("STRATA_PF_DM16"); }
             for (const char* w : want) {
                 bool found = false;
                 for (const auto& kv : t) found = found || (kv.first == w && kv.second == "1");
@@ -321,20 +327,20 @@ int main() {
     CHECK(!igpu_prompt_set_enabled("gfx1103", true));
     {
         const auto t = igpu_default_env("gfx1151", true);
-        CHECK(t.size() == 7);
+        CHECK(t.size() == 8);
         auto has = [&t](const char* k) {
             for (const auto& kv : t) if (kv.first == k) return kv.second == "1";
             return false;
         };
         CHECK(has("STRATA_HIP_WMMA") && has("STRATA_PF_GEMM") && has("STRATA_PA_FAST") && has("STRATA_HC_UPMIX") &&
-              has("STRATA_PF_FUSED") && has("STRATA_SELECT_WMMA") && has("STRATA_PF_F16"));
+              has("STRATA_PF_FUSED") && has("STRATA_SELECT_WMMA") && has("STRATA_PF_F16") && has("STRATA_PF_DM16"));
         CHECK(igpu_default_env("gfx1151", false).empty());   // a gfx1151 reported as a discrete card
         CHECK(igpu_default_env("gfx1150", true).empty());    // 890M: unchanged
         CHECK(igpu_default_env("gfx1103", true).empty());    // 780M on Windows: unchanged
     }
     {
         const char* halo[] = {"STRATA_HIP_WMMA", "STRATA_PF_GEMM", "STRATA_PA_FAST", "STRATA_HC_UPMIX", "STRATA_PF_FUSED",
-                              "STRATA_SELECT_WMMA", "STRATA_PF_F16"};
+                              "STRATA_SELECT_WMMA", "STRATA_PF_F16", "STRATA_PF_DM16"};
         for (const char* k : halo) unset(k);
         const auto set = apply_arch_defaults("gfx1151", true);
         for (const char* k : halo) {

src/core/expert_source.cpp

@@ -1,6 +1,7 @@
 // src/core/expert_source.cpp - the adapter.  See the header for the three clauses of the contract.
 #include "strata/core/expert_source.hpp"
 #include "strata/core/expert_nibble.hpp"
+#include "strata/core/conversation_memory.hpp"
 #include "strata/core/foresight_swap.hpp"
 #include "strata/core/on_device.hpp"
 #include "strata/core/ple_residency.hpp"
@@ -923,6 +924,16 @@ bool file_tier_unbuffered(const std::vector<std::string>& paths, uint64_t arena_
         why = "the available RAM is unknown";
         return false;
     }
+    // The room the file cache can really have (on by default; STRATA_FILE_TIER_HONEST=0 turns it off; an I/O-path change only).  available_memory_bytes credits the group's whole `file`
+    // counter as reclaimable (cgroup_available_bytes), and on a RAM-budget config that credit includes pages the engine itself holds
+    // (P2003, v31, 22 GiB RAM copy: the room read 18.7 GiB while the group had limit 36 GiB, current 24.8 GiB, i.e. 11 GiB left).  The re-check
+    // after the copy then saw held >= 0.5, skipped the drive probe and switched the file tier to the cache: 2-3x slower prompts.  The
+    // conversation cache's sampler (only the clean inactive file cache is credited) is the honest figure; the smaller of the two decides.
+    // The operator cap (--memory-limit-mib) is NOT applied: it bounds the engine's own allocations, not the page cache (counting it sent IQ3_S
+    // unbuffered and cut its decode 2-3x).
+    if (const char* ho = std::getenv("STRATA_FILE_TIER_HONEST"); ho == nullptr || ho[0] != '0') {
+        if (const auto honest = strata::core::sample_host_memory(strata::core::MemoryProbePaths{}, 0); honest && honest->available < avail) avail = honest->available;
+    }
     // #1194: the experts read from the files are a skewed set: a cache holding a twentieth of them (at least 1.5 GiB)
     // serves most of the repeats, and beat the unbuffered reads at every room measured (it reads 2 to 7 times less from
     // the drive).  `avail` already has what this process and the cgroup hold taken off, so 1 GiB is left for the rest.
@@ -1159,6 +1170,7 @@ void FileExpertSource::warm_scored(int64_t layer, const int64_t* experts, const
 
 void FileExpertSource::set_io_prefetch(bool on, int threads) {
     io_stop();
+    gate_on_ = false;   // the decode gate comes and goes with io prefetch
 #if defined(__linux__)
     if (!on) return;
     if (base_ == nullptr || paths_.empty() || !direct_.empty()) return;   // unbuffered readers batch their own reads
@@ -3094,7 +3106,11 @@ void RouterLookahead::submit(int64_t layer, const float* x, int64_t n_tok, const
         n_tok_ = std::min<int64_t>(n_tok, 8);
         std::memcpy(x_.data(), x, (size_t) (n_tok_ * n_embd_) * sizeof(float));
         layer_ = layer + 1;
-        host_res_ = host_res;
+        has_res_ = host_res != nullptr;
+        if (has_res_) {   // the rows of layer + 1 .. + depth, as they are now: the table changes while this runs
+            const int64_t l1 = std::min<int64_t>(layer_ + depth_, (int64_t) routers_.size());
+            res_.assign(host_res + (size_t) (layer_ * n_expert_), host_res + (size_t) (l1 * n_expert_));
+        }
         sub_gen_ = gen_.load(std::memory_order_relaxed);
         pending_ = true;
     }
@@ -3203,7 +3219,7 @@ void RouterLookahead::run() {
     std::vector<float> wconf;   // 149-prefetch: per wanted expert, its best rank's confidence (warm_scored)
     for (;;) {
         int64_t layer, nt;
-        const int32_t* host_res;
+        const int32_t* host_res;   // res_: layer_'s row first (submit leaves it alone while busy_)
         ForesightSwap* fs;
         uint64_t sgen;
         {
@@ -3214,7 +3230,7 @@ void RouterLookahead::run() {
             busy_ = true;
             layer = layer_;
             nt = n_tok_;
-            host_res = host_res_;
+            host_res = has_res_ ? res_.data() : nullptr;
             fs = fs_;
             sgen = sub_gen_;
         }
@@ -3271,7 +3287,7 @@ void RouterLookahead::run() {
                               [&](int32_t a, int32_t b) { return lt[(size_t) a] > lt[(size_t) b]; });
             for (int j = 0; j < kmax; ++j) {
                 const int64_t e = order[(size_t) j];
-                if (host_res != nullptr && host_res[(size_t) (layer * n_expert_ + e)] >= 0) continue;   // on the GPU
+                if (host_res != nullptr && host_res[(size_t) ((layer - layer0) * n_expert_ + e)] >= 0) continue;   // on the GPU
                 if (j < k_) {
                     const auto it = std::find(want.begin(), want.end(), e);
                     // confidence: 1 for a token's top pick down to 1/k_ for its k_-th, the best over the window's tokens,
@@ -3295,7 +3311,7 @@ void RouterLookahead::run() {
                                   [&](int32_t a, int32_t b) { return lt[(size_t) a] > lt[(size_t) b]; });
                 for (int j = 0; j < k_; ++j) {
                     const int32_t e = order[(size_t) j];
-                    if (host_res != nullptr && host_res[(size_t) (layer * n_expert_ + e)] >= 0) continue;
+                    if (host_res != nullptr && host_res[(size_t) ((layer - layer0) * n_expert_ + e)] >= 0) continue;
                     auto it = std::find_if(votes.begin(), votes.end(), [&](const auto& v) { return v.second == e; });
                     if (it == votes.end()) votes.emplace_back(64 - j, e);
                     else it->first += 64 - j;
@@ -4715,6 +4731,12 @@ bool FileExpertSource::pcie_layer(int64_t layer) const {
 
 // ================================ THE ADAPTER ================================
 
+bool ExpertDispatch::fetch_can(int64_t layer) const {
+    return fetch_admit > 0 && res_mut != nullptr && !usage.empty() && plan != nullptr && plan->pcie_mode == 2 &&
+           plan->dst2 != nullptr && peer == nullptr && remote_count == 0 && fs == nullptr && pcie_num > 0 &&
+           src != nullptr && src->pcie_layer(layer);
+}
+
 void expert_pool_dispatch(void* user, const float* x_f, const int32_t* ids, const float* weights, int64_t n_embd,
                           int64_t k, float* out) {
     (void) weights;   // clause 2: `moe_combine` applies it on the device.  Not an oversight.
@@ -4961,6 +4983,21 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
                 ++d.tail_experts;
             }
         }
+        // STRATA_ROUTE_DUMP=<file> (bench only, off by default): one text line per layer-window, "layer n_tok nd nmiss | e:rows:res ..." in routing
+        // order (res 1 = resident in the GPU cache, 0 = a miss the CPU computes unless the PCIe share takes it, 2 = tail-skipped), for replaying placement
+        // policies offline on the routing the engine really produced (round 0216 P3.1).
+        static std::FILE* const route_dump = [] { const char* f = std::getenv("STRATA_ROUTE_DUMP"); return f != nullptr && f[0] != '\0' ? std::fopen(f, "a") : nullptr; }();
+        if (route_dump != nullptr) {
+            std::fprintf(route_dump, "%lld %lld %d %d |", (long long) d.layers, (long long) n_tok, nd, nmiss);
+            for (int q = 0; q < nd; ++q) {
+                const int32_t e = ids[distinct[q]];
+                int rows = 0;
+                for (int64_t i = 0; i < n; ++i) rows += first_of[i] == distinct[q];
+                const int res = skip_d[q] ? 2 : (e >= 0 && e < d.n_expert && d.host_res[(size_t) d.layers * (size_t) d.n_expert + (size_t) e] >= 0) ? 1 : 0;
+                std::fprintf(route_dump, " %d:%d:%d", (int) e, rows, res);
+            }
+            std::fputc(10, route_dump);
+        }
         // STRATA_PCIE_BALANCE (opt-in, #1548): the count comes from the measured CPU and GPU costs instead of the share
         const bool pcie_ok = (d.pcie_num > 0 || d.pcie_model.on) && d.src->pcie_layer(d.layers);
         int m = pcie_ok ? d.pcie_model.pick(nmiss, (nmiss * d.pcie_num) >> 8) : 0;
@@ -4973,6 +5010,24 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
         GpuPlanSink& P = *d.plan;
         const uint8_t* dma_src[64];
         int64_t pcie_i0[64];
+        // STRATA_ADAPT_FETCH: the share is the layer's most-routed misses, and the ones the tier would swap in are copied
+        // into a slot as they are fetched (ExpertDispatch::fetch_admit)
+        const bool fetch_on = m > 0 && d.fetch_can(d.layers);
+        const float* use = d.usage.empty() ? nullptr : d.usage.data() + (size_t) d.layers * (size_t) d.n_expert;
+        bool pick[kMaxWindowEntries];
+        if (fetch_on) {
+            int mq[kMaxWindowEntries], nm = 0;
+            for (int q = 0; q < nd; ++q) {
+                pick[q] = false;
+                const int32_t e = ids[distinct[q]];   // a tail-skipped miss is not fetched
+                // (an unpinned blob cannot be fetched: under STRATA_PIN_BY_NEED a PCIe layer is pinned in part)
+                if (!skip_d[q] && e >= 0 && e < d.n_expert && d.host_res[(size_t) d.layers * (size_t) d.n_expert + (size_t) e] < 0 &&
+                    d.src->pinned(d.layers, e))
+                    mq[nm++] = q;
+            }
+            std::stable_sort(mq, mq + nm, [&](int a, int b) { return use[ids[distinct[a]]] > use[ids[distinct[b]]]; });
+            for (int j = 0; j < m && j < nm; ++j) pick[mq[j]] = true;
+        }
         for (int q = 0; q < nd; ++q) {
             const int64_t i0 = distinct[q];
             const int32_t e = ids[i0];
@@ -4997,7 +5052,7 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
                     kd = 0;                        // Foresight: a landed swap-space slot holds it - a GPU group like a hit
                 } else {
                     if (d.fs != nullptr) d.fs->note_miss(d.src, d.layers, e);
-                    if (miss_rank >= nmiss - m && fetches < P.staging_cap && fetches < 64) {
+                    if ((fetch_on ? pick[q] : miss_rank >= nmiss - m) && fetches < P.staging_cap && fetches < 64) {
                         const uint8_t* src = d.src->pinned(d.layers, e) ? d.src->blob(d.layers, e) : nullptr;
                         if (src != nullptr) {
                             kd = 1;
@@ -5037,6 +5092,49 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32
                 }
             ++d.pcie_experts;
         }
+        if (d.fetch_admit > 0 && P.dst2 != nullptr)
+            for (int q = 0; q < fetches; ++q) P.dst2[q] = 0;   // staging
+        if (fetch_on && fetches > 0 && (size_t) d.layers < d.fetch_avail.size()) d.fetch_avail[(size_t) d.layers] = 1;
+        if (fetch_on && fetches > 0) {
+            // the fetched experts routed enough, most-routed first, each against the next least-routed resident expert
+            // of the layer that the window does not route (its slot is read by nothing until the next window)
+            const size_t lb = (size_t) d.layers * (size_t) d.n_expert;
+            if (d.fetch_mark.size() != (size_t) d.n_expert) d.fetch_mark.assign((size_t) d.n_expert, 0);
+            for (int64_t i = 0; i < n; ++i)
+                if (ids[i] >= 0 && ids[i] < d.n_expert) d.fetch_mark[(size_t) ids[i]] = 1;
+            int cq[64], nc = 0;
+            for (int q = 0; q < fetches; ++q) {
+                const int32_t e = ids[pcie_i0[q]];
+                if (use[e] >= d.fetch_min && (d.in_flight.empty() || d.in_flight[lb + (size_t) e] == 0)) cq[nc++] = q;
+            }
+            std::stable_sort(cq, cq + nc, [&](int a, int b) { return use[ids[pcie_i0[a]]] > use[ids[pcie_i0[b]]]; });
+            int32_t vic[64];
+            int nv = 0;
+            for (int32_t e = 0; e < (int32_t) d.n_expert && nc > 0; ++e) {
+                if (d.res_mut[lb + (size_t) e] < 0 || d.fetch_mark[(size_t) e] != 0) continue;
+                if (nv < nc) vic[nv++] = e;
+                else if (use[e] < use[vic[nv - 1]]) vic[nv - 1] = e;
+                else continue;
+                for (int j = nv - 1; j > 0 && use[vic[j]] < use[vic[j - 1]]; --j) std::swap(vic[j], vic[j - 1]);
+            }
+            for (int j = 0; j < nc && j < nv; ++j) {
+                const int q = cq[j];
+                const int32_t e = ids[pcie_i0[q]], v = vic[j];
+                if (use[e] < use[v] + d.fetch_margin) break;
+                const int32_t slot = d.res_mut[lb + (size_t) v];
+                P.dst2[q] = (unsigned long long) (d.cache_base + (d.cache_slot_off ? (size_t) d.cache_slot_off[slot]
+                                                                                   : (size_t) slot * (size_t) d.cache_blob));
+                d.res_mut[lb + (size_t) v] = -1;   // kNotResident
+                d.res_mut[lb + (size_t) e] = slot;
+                ++d.fetch_admitted;
+                d.res_dirty = true;
+                d.res_dirty_idx.push_back((int32_t) (lb + (size_t) v));
+                d.res_dirty_idx.push_back((int32_t) (lb + (size_t) e));
+                if (d.fetch_log_on) d.fetch_log.emplace_back((int64_t) (lb + (size_t) e), slot);
+            }
+            for (int64_t i = 0; i < n; ++i)
+                if (ids[i] >= 0 && ids[i] < d.n_expert) d.fetch_mark[(size_t) ids[i]] = 0;
+        }
         P.start2[fetches] = entries;
         P.counts[0] = groups;
         P.counts[1] = entries;

src/core/native_dense.cpp

@@ -1,4 +1,5 @@
 #include "strata/core/native_dense.hpp"
+#include "strata/core/run_manifest.hpp"
 #include <cstdlib>
 #include "strata/core/weights.hpp"
 #include "strata/artifact/gguf_reader.hpp"
@@ -82,6 +83,11 @@ bool hc_q8_env() {
     }();
     return on;
 }
+// STRATA_HC_Q8_ALL_STAGES=1: the Q8_0 hyper-connection rows of every layer are copied to every stage (the behaviour before #3559)
+bool hc_q8_stage_local() {
+    static const bool on = [] { const char* v = std::getenv("STRATA_HC_Q8_ALL_STAGES"); return !(v != nullptr && v[0] == '1'); }();
+    return on;
+}
 bool hc_q8_name(const std::string& name) {
     if (name == "output_hc_down.weight" || name == "output_hc_up.weight") return true;   // S25: the final mixer too
     if (name.rfind("blk.", 0) != 0) return false;
@@ -288,7 +294,7 @@ bool NativeDense::load(const std::vector<std::string>& shards, WeightTable& tabl
         std::vector<Pending> pending;
         std::set<std::string> seen;
         int max_in = 0;
-        uint64_t total = 0, hc_q8_bytes = 0, allocated = 0;
+        uint64_t total = 0, hc_q8_bytes = 0, allocated = 0, hc_q8_skipped_bytes = 0, hc_q8_skipped = 0;
         uint64_t split_count = 0, split_tensors = 0;
         std::set<uint64_t> split_numbers;
         bool have_architecture = false;
@@ -424,8 +430,19 @@ bool NativeDense::load(const std::vector<std::string>& shards, WeightTable& tabl
                         return false;
                     }
                 }
+                if (strata::core::manifest::enabled()) {   // STRATA_MANIFEST: the dense formats as held, and the packed copies
+                    strata::core::manifest::instance().tensors("dense", strata::ggml_type_name((uint32_t) tensor.type), 1, bytes);
+                    if (packed) strata::core::manifest::instance().tensors("dense_packed_copy", "Q8_0", 1, bytes);
+                }
                 pending.push_back(Pending{&ref, (int) tensor.type, bytes, std::move(data), std::move(packed)});
             }
+            if (strata::core::manifest::enabled()) {
+                const char* v = std::getenv("STRATA_HC_Q8");
+                strata::core::manifest::instance().path("dense", "hc_q8", hc_q8_env(),
+                    hc_q8_env() ? (v != nullptr && v[0] == '1' ? "STRATA_HC_Q8=1" : "architecture default (gfx12 / gfx1151 HIP)")
+                                : (v != nullptr && v[0] != '\0' && std::strcmp(v, "auto") != 0 ? "STRATA_HC_Q8 turned it off"
+                                                                                               : "off by default on this backend"));
+            }
             if (hc_q8_env())
                 for (const auto& tensor : gguf.tensors()) {
                     // S25: the F32 inject rows hold BF16-exact values: the read takes the pack's BF16 rows unless
@@ -433,6 +450,14 @@ bool NativeDense::load(const std::vector<std::string>& shards, WeightTable& tabl
                     static const bool q8_inject = [] { const char* v = std::getenv("STRATA_HC_Q8_INJECT"); return v && v[0] == '1'; }();
                     const bool f32_inject = q8_inject && tensor.type == 0 && tensor.name.ends_with("_inject.weight");
                     if ((tensor.type != 8 && !f32_inject) || !hc_q8_name(tensor.name) || tensor.shape.size() != 2) continue;
+                    // a trimmed layer split: another stage's layers are never read here, so their rows are not copied (the upload
+                    // loop above and the bf16 pack path below already filter this way; PR #3559, havok2-htwo).
+                    // STRATA_HC_Q8_ALL_STAGES=1 copies them as before.
+                    if (hc_q8_stage_local() && (outside(tensor.name) || !in_range(tensor.name))) {
+                        hc_q8_skipped_bytes += (uint64_t) tensor.shape[0] / 32 * 34 * (uint64_t) tensor.shape[1];
+                        ++hc_q8_skipped;
+                        continue;
+                    }
                     auto found = table.table_.find(tensor.name);
                     if (found == table.table_.end() || found->second.hc_q8 != nullptr) continue;
                     auto& ref = found->second;
@@ -451,6 +476,7 @@ bool NativeDense::load(const std::vector<std::string>& shards, WeightTable& tabl
                     ref.hc_q8 = p;
                     weights_.push_back(p);
                     hc_q8_bytes += bytes;
+                    strata::core::manifest::instance().tensors("dense_hc_q8_copy", f32_inject ? "F32->Q8_0" : "Q8_0", 1, bytes);
                 }
         }
         if (hc_q8_env()) {
@@ -538,6 +564,8 @@ bool NativeDense::load(const std::vector<std::string>& shards, WeightTable& tabl
         scratch_ = scratch.release();
         bytes_ = total + hc_q8_bytes;
         allocated_ = allocated;
+        if (hc_q8_skipped != 0 && strata::core::manifest::enabled())   // STRATA_MANIFEST: what the stage filter did not copy
+            strata::core::manifest::instance().tensors("dense_hc_q8_out_of_stage_not_copied", "Q8_0", hc_q8_skipped, hc_q8_skipped_bytes);
         if (hc_q8_env())
             std::fprintf(stderr, "strata: STRATA_HC_Q8=1: %.2f GiB of Q8_0 hyper-connection projections for the verify read\n",
                          (double) hc_q8_bytes / 1073741824.0);

src/core/native_head.cpp

@@ -1,5 +1,6 @@
 #include "strata/core/native_head.hpp"
 #include "strata/core/spill_gate.hpp"
+#include "strata/core/run_manifest.hpp"
 #include "strata/artifact/gguf_reader.hpp"
 #include "strata/kernels/iq_kernels.hpp"
 #include "strata/kernels/native_mmvq.hpp"
@@ -107,6 +108,15 @@ bool NativeHead::load(const std::vector<std::string>& shards, int64_t n_in, int6
             strata::kernels::native_q6_k_pack(weights_, n_in_, n_out_, "output head");
         if (type_ == 8)   // STRATA_Q8_SM60=1 (a no-op otherwise); the MTP draft head reads the same matrix
             strata::kernels::native_q8_0_sm60_pack(weights_, n_in_, n_out_, "output head");
+        if (strata::core::manifest::enabled()) {   // STRATA_MANIFEST: what the head holds and where
+            namespace mf = strata::core::manifest;
+            mf::instance().tensors("head", strata::ggml_type_name((uint32_t) type_), 1, bytes);
+            mf::instance().path("head", "output.weight", true,
+                                host_ != nullptr ? "held in mapped pinned host memory (short of VRAM or STRATA_HEAD_HOST)" : "held in VRAM",
+                                std::string("type ") + strata::ggml_type_name((uint32_t) type_) +
+                                    (type_ == 14 && strata::kernels::native_q6_k_packed_enabled() ? ", repacked (STRATA_Q6_PACKED)" : "") +
+                                    (type_ == 8 && std::getenv("STRATA_Q8_SM60") != nullptr ? ", repacked (STRATA_Q8_SM60)" : ""));
+        }
         return true;
     } catch (const std::exception& error) {
         err = std::string("native head: ") + error.what();

src/core/peer_experts.cpp

@@ -109,6 +109,8 @@ void PeerExperts::close() {
         if (h_x_) cudaFreeHost(h_x_);
         if (h_out_) cudaFreeHost(h_out_);
         if (h_meta_) cudaFreeHost(h_meta_);
+        if (d_gate_vals_) cudaFree(d_gate_vals_);
+        if (h_gate_) cudaFreeHost((void*) h_gate_);
         if (refill_ev_) cudaEventDestroy(refill_ev_);
         if (stream_) cudaStreamDestroy(stream_);
         if (refill_) cudaStreamDestroy(refill_);
@@ -117,6 +119,8 @@ void PeerExperts::close() {
     d_meta_ = h_meta_ = d_scratch_ = nullptr;
     d_q8_ = nullptr;
     d_sx_ = d_sh_ = nullptr;
+    d_gate_vals_ = nullptr;
+    h_gate_ = nullptr;
     stream_ = refill_ = nullptr;
     refill_ev_ = nullptr;
     device_ = -1;
@@ -242,10 +246,19 @@ bool PeerExperts::open(int device, const std::vector<std::pair<int32_t, int32_t>
     return true;
 }
 
+bool PeerExperts::direct_mode() {
+    static const bool v = [] { const char* e = std::getenv("STRATA_PEER_DIRECT"); return e == nullptr || std::atoi(e) != 0; }();
+    return v;
+}
+
 bool PeerExperts::launch(int64_t layer, const float* x, const int32_t* ids, int64_t n_tok, int64_t k,
                          const int32_t* kind, std::string& err, float* out) {
-    static const bool direct_env = [] { const char* v = std::getenv("STRATA_PEER_DIRECT"); return v == nullptr || std::atoi(v) != 0; }();
-    const bool direct = direct_env && out != nullptr;
+    const bool direct = direct_mode() && out != nullptr;
+    struct Timer {
+        double& acc;
+        std::chrono::steady_clock::time_point t0 = std::chrono::steady_clock::now();
+        ~Timer() { acc += std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - t0).count(); }
+    } timer{ms_launch};
     launched_direct_ = false;
     launched_rows_ = 0;
     row_of_.clear();
@@ -316,6 +329,7 @@ bool PeerExperts::launch(int64_t layer, const float* x, const int32_t* ids, int6
 
 bool PeerExperts::finish(float* out, std::string& err) {
     if (launched_rows_ == 0) return true;
+    ++launches;
     const auto t0 = std::chrono::steady_clock::now();
     {
         On on(device_);
@@ -362,20 +376,85 @@ bool PeerExperts::adapt(const float* usage, const int32_t* res0, int max_swaps,
     if (swaps.empty()) return true;
     std::sort(swaps.begin(), swaps.end(), [](const Swap& a, const Swap& b) { return a.gain > b.gain; });
     if ((int) swaps.size() > max_swaps) swaps.resize((size_t) max_swaps);
+    const bool gated = gate_landed_ != nullptr;
+    if (gated) {   // the copies go in layer order and each layer's landing raises the gate's counter
+        std::stable_sort(swaps.begin(), swaps.end(), [](const Swap& a, const Swap& b) { return a.layer < b.layer; });
+        gate_landed_->store(-1, std::memory_order_release);
+        if (h_gate_ != nullptr) *h_gate_ = -1;
+        gate_has_->assign((size_t) n_layers_, 0);
+        *gate_last_ = -1;
+    }
     On on(device_);
+    int32_t issued_layer = -1;   // the layer whose copies are on the stream and not yet ticked
+    auto tick = [&](int32_t layer) -> bool {
+        if ((h_gate_ != nullptr
+                 ? cudaMemcpyAsync((void*) h_gate_, d_gate_vals_ + layer, 4, cudaMemcpyDeviceToHost, refill_)
+                 : cudaLaunchHostFunc(refill_, [](void* p) {
+                auto* t = (Tick*) p;
+                int32_t cur = t->counter->load(std::memory_order_relaxed);
+                while (cur < t->layer && !t->counter->compare_exchange_weak(cur, t->layer, std::memory_order_release)) {}
+            }, &ticks_[(size_t) layer])) == cudaSuccess)
+            return true;
+        (void) cudaGetLastError();
+        cudaStreamSynchronize(refill_);   // no callback: wait here, the gate opens at once
+        gate_landed_->store((int32_t) n_layers_, std::memory_order_release);
+        return false;
+    };
     for (const Swap& s : swaps) {
         const size_t in = (size_t) (s.layer * n_expert_ + s.in), out = (size_t) (s.layer * n_expert_ + s.out);
         const int32_t slot = res_[out];
         const uint8_t* b = src_->blob(s.layer, s.in);
         if (slot < 0 || b == nullptr) continue;
+        if (gated && issued_layer >= 0 && issued_layer != s.layer) tick(issued_layer);
         if (!ck(cudaMemcpyAsync(cache_.device_slot(slot), b, (size_t) lay.blob_bytes(s.layer), cudaMemcpyHostToDevice,
                                 refill_), "refill", err))
             return false;
+        if (gated) {
+            (*gate_has_)[(size_t) s.layer] = 1;
+            *gate_last_ = std::max(*gate_last_, s.layer);
+            issued_layer = s.layer;
+        }
         res_[out] = kNotResident;              // evicted now: the CPU computes it meanwhile
         pending_.emplace_back((int32_t) in, slot);
         ++swaps_;
     }
-    return ck(cudaEventRecord(refill_ev_, refill_), "refill event", err);
+    if (gated && issued_layer >= 0) tick(issued_layer);
+    if (!ck(cudaEventRecord(refill_ev_, refill_), "refill event", err)) return false;
+    // #2188: hand the copies and their landing signals to the GPU now (WDDM batches a stream's work until a driver
+    // call flushes it, and the verify window only spins on the pinned int; Linux submits at once: a no-op)
+    if (gated && h_gate_ != nullptr && issued_layer >= 0) (void) cudaStreamQuery(refill_);
+    return true;
+}
+
+volatile int32_t* PeerExperts::set_overlap_gate(std::atomic<int32_t>* landed, std::vector<uint8_t>* has, int32_t* last,
+                                                bool copy_signal) {
+    gate_landed_ = landed;
+    gate_has_ = has;
+    gate_last_ = last;
+    ticks_.resize((size_t) n_layers_);
+    for (int64_t l = 0; l < n_layers_; ++l) ticks_[(size_t) l] = {landed, (int32_t) l};
+    if (copy_signal && h_gate_ == nullptr) {
+        On on(device_);
+        std::vector<int32_t> iota((size_t) n_layers_);
+        for (size_t i = 0; i < iota.size(); ++i) iota[i] = (int32_t) i;
+        int32_t* vals = nullptr;
+        void* hp = nullptr;
+        if (cudaMalloc((void**) &vals, iota.size() * 4) == cudaSuccess &&
+            cudaMemcpy(vals, iota.data(), iota.size() * 4, cudaMemcpyHostToDevice) == cudaSuccess &&
+            cudaHostAlloc(&hp, 64, cudaHostAllocPortable) == cudaSuccess) {
+            *(volatile int32_t*) hp = -1;
+            d_gate_vals_ = vals;
+            h_gate_ = (volatile int32_t*) hp;
+        } else {   // host functions then
+            (void) cudaGetLastError();
+            if (vals) cudaFree(vals);
+        }
+    }
+    return h_gate_;
+}
+
+void PeerExperts::flip_pending() {
+    for (const auto& [i, slot] : pending_) res_[(size_t) i] = slot;
 }
 
 void PeerExperts::apply_pending(bool wait) {

src/core/verify.cpp

@@ -1031,11 +1031,13 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState&
         bal_->on = true;
     }
     // the GPU plan: counts(4) | start(cap+1) | dst(cap) | tok(cap) | pad | ptr(cap u64) | ptr2(cap u64) | start2(cap+1)
+    // | pad | dst2(kStagingBlobs u64)
     {
         const int64_t cap = (int64_t) (T * K);
         const int64_t i32 = 4 + (cap + 1) + cap + cap;
         const int64_t ptr_off = (i32 + 1) & ~1ll;
-        plan_i32_ = ptr_off + 4 * cap + (cap + 1) + 1;
+        dst2_off_ = (ptr_off + 4 * cap + (cap + 1) + 1 + 1) & ~1ll;
+        plan_i32_ = dst2_off_ + 2 * kStagingBlobs;
         if (!mapped((size_t) plan_i32_ * 4 * 2 + 64, (void**) &h_plan_, (void**) &m_plan_)) {
             err = "verify: mapped plan allocation failed " + strata::core::pin_exhausted_hint();
             return false;
@@ -1047,6 +1049,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState&
         sink_.ptr = (unsigned long long*) (h_plan_ + ptr_off);
         sink_.ptr2 = sink_.ptr + cap;
         sink_.start2 = h_plan_ + ptr_off + 4 * cap;
+        sink_.dst2 = (unsigned long long*) (h_plan_ + dst2_off_);
         sink_.cap = cap;
         sink_.publish = &Verifier::publish_plan;
         sink_.fetch = &Verifier::fetch_dma;
@@ -1138,7 +1141,7 @@ bool Verifier::init(const WeightTable& wt, const ModelGeometry& g, SessionState&
     }
     prof_on_ = std::getenv("STRATA_VERIFY_PROFILE") != nullptr;
     if (const char* rs = std::getenv("STRATA_ROUTE_STATS"); rs != nullptr && rs[0] == '1') {
-        const size_t nrs = (size_t) 9 * (size_t) g.n_layers * 8;
+        const size_t nrs = (size_t) 17 * (size_t) g.n_layers * 16;   // [T 0..16][layer][row 0..15]
         if (cudaMalloc((void**) &rs_dev_, nrs * 8) == cudaSuccess) {
             cudaMemset(rs_dev_, 0, nrs * 8);
             rs_on_ = true;
@@ -2238,8 +2241,8 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
                                       rr_stats_ != nullptr ? rr_stats_ + (n <= 8 ? 4 : 0) : nullptr, cs);
             } catch (const std::exception& e) { err = "verify route-resident: " + std::string(e.what()); return false; }
         }
-        if (rs_on_ && G == 1 && n <= 8 && T <= 8)
-            strata::kernels::route_marginal_stats(ids_ + tb * K, n, (int) K, rs_dev_ + ((size_t) T * (size_t) g.n_layers + (size_t) l) * 8, cs);
+        if (rs_on_ && G == 1 && n <= 16 && T <= 16)
+            strata::kernels::route_marginal_stats(ids_ + tb * K, n, (int) K, rs_dev_ + ((size_t) T * (size_t) g.n_layers + (size_t) l) * 16, cs);
         if (ar_on()) {
             resident_plan(ids_ + tb * K, n * (int) K, (int) K, hits_.d_res + l * g.n_expert, (int) g.n_expert,
                           hits_.cache_base, slot_off_d_, (long long) hits_.blob,
@@ -2408,8 +2411,9 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
             if (sink_.pcie_mode == 2) {                            // stage it with a copy kernel, then point at staging
                 const int64_t per = G == 2 ? kStagingBlobs / 2 : kStagingBlobs;
                 uint8_t* stage = staging_ + (size_t) (grp * per) * lay.max_blob;
-                fetch_blobs(p_ptr2, p_counts + 2, stage, (int64_t) lay.blob_bytes(l), (int) per, cs);
-                rebase_ptrs((unsigned long long*) p_ptr2, p_counts + 2, stage, (int64_t) lay.blob_bytes(l), cs);
+                const unsigned long long* p_dst2 = (const unsigned long long*) (pl + dst2_off_);   // STRATA_ADAPT_FETCH
+                fetch_blobs(p_ptr2, p_counts + 2, stage, (int64_t) lay.blob_bytes(l), (int) per, cs, p_dst2);
+                rebase_ptrs((unsigned long long*) p_ptr2, p_counts + 2, stage, (int64_t) lay.blob_bytes(l), cs, p_dst2);
             }
             stamp(l, 21, grp);
             // the PCIe share is pcie_frac of the misses: a few groups when the cache is cold, usually none (always none at
@@ -2649,7 +2653,7 @@ std::string Verifier::profile_by_t_report() {
                                           "(gap)", "head", "hc0 norm", "hc0 down", "hc0 up", "hc1 norm", "hc1 down", "hc1 up"};
     std::string out;
     char b[96];
-    for (int T = 1; T <= 8; ++T) {
+    for (int T = 1; T <= 16; ++T) {
         if (prof_t_win_[T] == 0) continue;
         double total = 0;
         std::snprintf(b, sizeof b, "T%d (n=%lld):", T, (long long) prof_t_win_[T]);
@@ -2675,15 +2679,15 @@ std::string Verifier::profile_by_t_report() {
 std::string Verifier::route_stats_report() {
     if (!rs_on_ || rs_dev_ == nullptr || g_ == nullptr) return std::string();
     const int64_t L = g_->n_layers;
-    std::vector<unsigned long long> h((size_t) 9 * (size_t) L * 8);
+    std::vector<unsigned long long> h((size_t) 17 * (size_t) L * 16);
     cudaMemcpy(h.data(), rs_dev_, h.size() * 8, cudaMemcpyDeviceToHost);
     std::string out;
     char b[160];
-    for (int T = 1; T <= 8; ++T) {
+    for (int T = 1; T <= 16; ++T) {
         if (rs_win_[T] == 0) continue;
-        double row[8] = {};
+        double row[16] = {};
         for (int64_t l = 0; l < L; ++l)
-            for (int r = 0; r < 8; ++r) row[r] += (double) h[((size_t) T * (size_t) L + (size_t) l) * 8 + (size_t) r];
+            for (int r = 0; r < 16; ++r) row[r] += (double) h[((size_t) T * (size_t) L + (size_t) l) * 16 + (size_t) r];
         const double denom = (double) rs_win_[T] * (double) L;
         double sum = 0;
         for (int r = 0; r < T; ++r) sum += row[r] / denom;
@@ -3074,7 +3078,7 @@ void Verifier::stage_inputs(int T, const int32_t* tokens, int64_t pos0) {
     std::atomic_thread_fence(std::memory_order_seq_cst);
     strata_store_fence();
     last_t_ = T;
-    if (rs_on_ && T >= 1 && T <= 8) ++rs_win_[T];
+    if (rs_on_ && T >= 1 && T <= 16) ++rs_win_[T];
     last_pos0_ = pos0;
     for (int t = 0; t < T; ++t) last_tokens_[t] = tokens[t];
     staged_ = true;
@@ -3089,7 +3093,7 @@ void Verifier::accumulate_profile(const unsigned long long* stamps) {
     double before[2][kProfPer];
     std::memcpy(before, prof_sum_, sizeof before);
     accumulate_profile_inner(stamps);
-    if (last_t_ >= 1 && last_t_ <= 8) {
+    if (last_t_ >= 1 && last_t_ <= 16) {
         for (int k = 0; k < 2; ++k)
             for (int i = 0; i < kProfPer; ++i) prof_t_sum_[last_t_][k][i] += prof_sum_[k][i] - before[k][i];
         ++prof_t_win_[last_t_];
@@ -3144,8 +3148,13 @@ void Verifier::accumulate_profile_inner(const unsigned long long* stamps) {
 bool Verifier::adapt_gate_wait(int32_t layer, std::string& err) {
     const Clock::time_point t0 = Clock::now();
     uint32_t spins = 0;
+    // each tier's counter climbs to the last layer it swaps this round, so an all-layers wait asks for that layer
+    const AdaptGate& ag = *adapt_gate_;
+    const int32_t want = std::min(layer, ag.last), want_peer = std::min(layer, ag.peer_last);
+    const bool wait_main = ag.last >= 0 && ((size_t) layer >= ag.has.size() || ag.has[(size_t) layer] != 0);
+    const bool wait_peer = ag.peer_last >= 0 && ((size_t) layer >= ag.peer_has.size() || ag.peer_has[(size_t) layer] != 0);
     bool announced = false;   // the watchdog's label is set only once the wait really starts: no cost when the copies landed
-    while (adapt_gate_->reached() < layer) {
+    while ((wait_main && ag.reached() < want) || (wait_peer && ag.peer_reached() < want_peer)) {
         if (!announced) {
             progress_at("verify window: waiting for the adaptive tier's copies of layer", layer);
             announced = true;
@@ -3474,7 +3483,8 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool,
         if (g_trace) trace_ev("RANG", k, l, (int64_t) std::chrono::duration_cast<std::chrono::microseconds>(b - a).count());
         VDBG("layer %lld rang\n", (long long) l);
         if (adapt_gate_ != nullptr && adapt_gate_->active.load(std::memory_order_acquire) && l >= 0 &&
-            (size_t) l < adapt_gate_->has.size() && adapt_gate_->has[(size_t) l] != 0 &&
+            (((size_t) l < adapt_gate_->has.size() && adapt_gate_->has[(size_t) l] != 0) ||
+             ((size_t) l < adapt_gate_->peer_has.size() && adapt_gate_->peer_has[(size_t) l] != 0)) &&
             !adapt_gate_wait((int32_t) l, err)) return false;
         cur_layer_ = want - 1;
         set_plan_slot(grp);
@@ -3506,7 +3516,7 @@ bool Verifier::run(int T, const int32_t* tokens, int64_t pos0, PoolMultiFn pool,
             sink_.start2[0] = 0;
             std::atomic_thread_fence(std::memory_order_seq_cst);
             *(volatile uint32_t*) h_flagA_ = want;
-            raise_flag(h_flagB_, want);
+            raise_flag_b_ordered(want);   // behind any copy still queued from an earlier layer
         }
         // Layer 1's pre(1, 0) copies h_ple_ -> ple_ after Layer 0's wait_flag_ge(m_flag_, 1).
         // By collecting PLE here at k == 0 (after publishing flagA/flagB for Layer 0 so the GPU can run
@@ -3717,6 +3727,7 @@ void Verifier::set_plan_slot(int grp) {
     sink_.ptr = (unsigned long long*) (base + ptr_off);
     sink_.ptr2 = sink_.ptr + cap;
     sink_.start2 = base + ptr_off + 4 * cap;
+    sink_.dst2 = (unsigned long long*) (base + dst2_off_);
     const int G = last_batch_ ? 1 : (groups_[last_t_] > 0 ? groups_[last_t_] : 1);
     const int64_t per = G == 2 ? kStagingBlobs / 2 : kStagingBlobs;
     sink_.staging = (unsigned long long) (staging_ + (size_t) (grp * per) * strata::kernels::cpu::expert_layout().max_blob);
@@ -3724,19 +3735,21 @@ void Verifier::set_plan_slot(int grp) {
 }
 
 // Flag B only rises: a host function of an earlier layer may run after a later layer already raised it directly.
-void Verifier::raise_flag(uint32_t* flag, uint32_t value) {
-    volatile long* f = (volatile long*) flag;
-#if defined(_WIN32)
-    long cur = *f;
-    while ((uint32_t) cur < value) {
-        const long prev = _InterlockedCompareExchange(f, (long) value, cur);
-        if (prev == cur) break;
-        cur = prev;
+void Verifier::raise_flag(uint32_t* flag, uint32_t value) { strata::core::flag_raise_max(flag, value); }
+
+// Flag B for layer `want - 1`, behind every copy queued on the copy stream (PR #5096, @AlanJager).  The counter only
+// rises and the graph waits for "flag B >= layer + 1", so raising it on the spot for a layer with nothing to copy
+// releases an earlier layer (or the other token group of a split window) whose uploads are still in flight.
+void Verifier::raise_flag_b_ordered(uint32_t want) {
+    FlagSet& fs = flag_sets_[(want - 1) % (sizeof flag_sets_ / sizeof flag_sets_[0])];
+    const cudaError_t e = strata::core::raise_flag_b_ordered(copy_used_, copy_, &fs, h_flagB_, want);
+    if (e != cudaSuccess) {
+        // The flag was not raised: the consumers must not be released to staging that is still being written.
+        std::fprintf(stderr, "strata DMA: ordered flag B completion failed: %s\n", cudaGetErrorString(e));
+        std::fflush(stderr);
+        release_gpu_waits(5000);
+        std::abort();
     }
-#else
-    uint32_t cur = __atomic_load_n((uint32_t*) flag, __ATOMIC_SEQ_CST);
-    while (cur < value && !__atomic_compare_exchange_n((uint32_t*) flag, &cur, value, false, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)) {}
-#endif
 }
 
 // Plan v0.3 P6: the PCIe share by DMA.  The copy engine moves the blobs while the CPU computes its own share and the
@@ -3745,7 +3758,7 @@ void Verifier::raise_flag(uint32_t* flag, uint32_t value) {
 void Verifier::fetch_dma(void* ctx, const uint8_t* const* src, int n, size_t bytes) {
     Verifier* v = (Verifier*) ctx;
     const uint32_t want = v->cur_layer_ + 1;
-    if (n <= 0) { raise_flag(v->h_flagB_, want); return; }
+    if (n <= 0) { v->raise_flag_b_ordered(want); return; }   // an empty plan stays behind the copies already queued
     v->copy_used_ = true;
     uint8_t* stage = (uint8_t*) v->sink_.staging;                  // this group's half in a split window
     // STRATA_DMA_BATCH (midhatn's #807, F12): the group's independent uploads as one cudaMemcpyBatchAsync.  Mode 0 is
@@ -3764,10 +3777,7 @@ void Verifier::fetch_dma(void* ctx, const uint8_t* const* src, int n, size_t byt
             std::abort();
         }
     }
-    FlagSet& fs = v->flag_sets_[v->cur_layer_ % (sizeof v->flag_sets_ / sizeof v->flag_sets_[0])];
-    fs.flag = v->h_flagB_;
-    fs.value = want;
-    cudaLaunchHostFunc(v->copy_, [](void* p) { FlagSet* s = (FlagSet*) p; raise_flag(s->flag, s->value); }, &fs);
+    v->raise_flag_b_ordered(want);   // copy_used_ is set: the host function runs after this group's uploads
 }
 
 void Verifier::publish_plan(void* ctx) {
@@ -3804,6 +3814,20 @@ bool Verifier::window_logprobs(const int32_t* targets, int T, int64_t pos0, int3
         return false;
     }
 #endif
+    // STRATA_FORCE_FAULT=<m>:<bias> (test only, off by default, no output): adds <bias> to the logit of every token id divisible by <m>
+    // before the rows are scored, so the token-forcing checker (tools/force_check.py) can be shown to reject a faulty build
+    {
+        static const std::pair<int64_t, float> fault = [] {
+            const char* v = std::getenv("STRATA_FORCE_FAULT");
+            long long m = 0;
+            float b = 0.0f;
+            if (v != nullptr && std::sscanf(v, "%lld:%f", &m, &b) == 2 && m > 0) return std::make_pair((int64_t) m, b);
+            return std::make_pair((int64_t) 0, 0.0f);
+        }();
+        if (fault.first > 0)
+            for (int t = 0; t < T; ++t)
+                for (int64_t v = 0; v < n_vocab_; v += fault.first) h[(size_t) t * (size_t) n_vocab_ + (size_t) v] += fault.second;
+    }
     for (int t = 0; t < T; ++t) {
         const float* row = h.data() + (size_t) t * (size_t) n_vocab_;
         const int32_t tgt = targets[t];
@@ -4761,7 +4785,7 @@ bool Verifier::run_slot_rows(const int* rows, int S, const int32_t* tokens, cons
             sink_.start2[0] = 0;
             std::atomic_thread_fence(std::memory_order_seq_cst);
             *(volatile uint32_t*) h_flagA_ = want;
-            raise_flag(h_flagB_, want);
+            raise_flag_b_ordered(want);   // behind any copy still queued from an earlier layer
         }
         *flag = want;
         ms_wait += std::chrono::duration<double, std::milli>(b - a).count();
@@ -4976,7 +5000,7 @@ int Verifier::batch_serve(PoolMultiFn pool, void* user) {
             sink_.start2[0] = 0;
             std::atomic_thread_fence(std::memory_order_seq_cst);
             *(volatile uint32_t*) h_flagA_ = want;
-            raise_flag(h_flagB_, want);
+            raise_flag_b_ordered(want);   // behind any copy still queued from an earlier layer
         }
         *(volatile uint32_t*) h_flag_ = want;
         ms_pool += ms_since(b);
@@ -5330,7 +5354,7 @@ int Verifier::service(PoolMultiFn pool, void* user, std::string& err) {
             sink_.start2[0] = 0;
             std::atomic_thread_fence(std::memory_order_seq_cst);
             *(volatile uint32_t*) h_flagA_ = want;
-            raise_flag(h_flagB_, want);
+            raise_flag_b_ordered(want);   // behind any copy still queued from an earlier layer
         }
         if (fl_k_ == 0 && !gather_ple()) return -1;
         *(volatile uint32_t*) h_flag_ = want;

src/kernels/cpu/cpu_dispatch_dump.cpp

@@ -0,0 +1,167 @@
+// src/kernels/cpu/cpu_dispatch_dump.cpp - round 0216: what the CPU pool's expert calls (native_gu_rows + native_down_rows, both
+// arithmetics) put out, as a hash per (pair, token count, row range).  Run the same binary with a change switched off and on (or a
+// baseline build and a candidate build) and compare the lines: an E change (bit-exact against the code it replaces) gives the same text.
+//
+//     cpu_dispatch_dump [--pairs iq4_xs/q5_0,...] [--nt-max 8] [--experts 3]
+//
+// For every pair: --experts random experts (ggml-quantized random weights, valid scales), nt = 1..nt-max tokens, whole rows and
+// row ranges that start and end off any vector boundary (tails), the Default and the WindowInvariant entry points.
+#include "strata/kernels/cpu/expert_layout.hpp"
+#include "strata/kernels/cpu/native_expert.hpp"
+
+#include "ggml.h"
+#include "ggml-cpu.h"
+
+#include <cmath>
+#include <cstdint>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <random>
+#include <string>
+#include <vector>
+
+namespace cpu = strata::kernels::cpu;
+
+namespace {
+
+constexpr int64_t kH = 2560, kFF = 640;
+constexpr int kMaxT = 8;
+
+int type_of(const std::string& s) {
+    for (int t = 0; t < GGML_TYPE_COUNT; ++t) {
+        const char* n = ggml_type_name((ggml_type) t);
+        if (n != nullptr && s == n) return t;
+    }
+    return -1;
+}
+
+std::vector<std::string> split(const std::string& s) {
+    std::vector<std::string> v;
+    size_t from = 0;
+    for (;;) {
+        const size_t c = s.find(',', from);
+        v.push_back(s.substr(from, c == std::string::npos ? std::string::npos : c - from));
+        if (c == std::string::npos) break;
+        from = c + 1;
+    }
+    return v;
+}
+
+uint64_t fnv(uint64_t h, const void* p, size_t n) {
+    const uint8_t* b = (const uint8_t*) p;
+    for (size_t i = 0; i < n; ++i) h = (h ^ b[i]) * 1099511628211ull;
+    return h;
+}
+
+}  // namespace
+
+int main(int argc, char** argv) {
+    setvbuf(stdout, nullptr, _IONBF, 0);
+    std::vector<std::string> pairs = split("iq3_xxs/q5_0,iq3_xxs/iq4_nl,iq4_xs/q5_0,iq4_xs/iq4_nl,iq3_s/iq4_nl,iq2_s/iq4_nl,iq4_xs/q8_0,iq2_s/q2_0,iq3_xxs/q2_0");
+    int nt_max = 8, experts = 3;
+    for (int i = 1; i < argc; ++i) {
+        const std::string a = argv[i];
+        if (a == "--pairs" && i + 1 < argc) pairs = split(argv[++i]);
+        else if (a == "--nt-max" && i + 1 < argc) nt_max = std::atoi(argv[++i]);
+        else if (a == "--experts" && i + 1 < argc) experts = std::atoi(argv[++i]);
+        else { std::fprintf(stderr, "usage: cpu_dispatch_dump [--pairs a/b,...] [--nt-max 8] [--experts 3]\n"); return 2; }
+    }
+    ggml_cpu_init();
+    for (const std::string& pair : pairs) {
+        const size_t slash = pair.find('/');
+        const int gt = slash == std::string::npos ? -1 : type_of(pair.substr(0, slash));
+        const int dt = slash == std::string::npos ? -1 : type_of(pair.substr(slash + 1));
+        cpu::NativeFmt f;
+        std::string err;
+        if (gt < 0 || dt < 0 || !cpu::native_fmt(gt, dt, kH, kFF, f, err)) {
+            std::printf("%s: %s\n", pair.c_str(), err.empty() ? "unknown types" : err.c_str());
+            continue;
+        }
+        ggml_quantize_init((ggml_type) gt);
+        ggml_quantize_init((ggml_type) dt);
+        std::mt19937 rng(4321 + gt * 64 + dt);
+        std::normal_distribution<float> nd(0.f, 0.02f), xd(0.f, 1.f);
+        std::vector<float> row((size_t) std::max(kH, kFF));
+        const std::vector<float> imatrix((size_t) std::max(kH, kFF), 1.0f);
+        auto quant = [&](int type, uint8_t* dst, int64_t n) {
+            ggml_quantize_chunk((ggml_type) type, row.data(), dst, 0, 1, n, ggml_quantize_requires_imatrix((ggml_type) type) ? imatrix.data() : nullptr);
+        };
+        std::vector<uint8_t> blob(f.bytes + 64);
+        std::vector<std::vector<uint8_t>> xa(kMaxT, std::vector<uint8_t>(cpu::kNativeActBytes)), ha(kMaxT, std::vector<uint8_t>(cpu::kNativeHBytes));
+        std::vector<std::vector<float>> ffv(kMaxT, std::vector<float>(kFF)), outv(kMaxT, std::vector<float>(kH));
+        const void* act[kMaxT];
+        const void* hq[kMaxT];
+        float* ff[kMaxT];
+        float* out[kMaxT];
+        for (int t = 0; t < kMaxT; ++t) { act[t] = xa[(size_t) t].data(); hq[t] = ha[(size_t) t].data(); ff[t] = ffv[(size_t) t].data(); out[t] = outv[(size_t) t].data(); }
+        // a Q2_0 down (or gate/up) takes the pool's own path (pool.cpp): the activation as an ActQ and Strata's Q2_0 rows, not ggml's dot
+        const bool q2_down = cpu::q2_native_kernels(f.d_type), q2_gu = cpu::q2_native_kernels(f.gu_type);
+        std::vector<cpu::ActQ> h2(q2_down ? kMaxT : 0), x2(q2_gu ? kMaxT : 0);
+        const cpu::ActQ* a2[kMaxT] = {};
+        const cpu::ActQ* ax[kMaxT] = {};
+        std::vector<std::vector<float>> gv(q2_gu ? kMaxT : 0, std::vector<float>(kFF)), uv(q2_gu ? kMaxT : 0, std::vector<float>(kFF));
+        float* gp2[kMaxT] = {};
+        float* up2[kMaxT] = {};
+        uint64_t total = 1469598103934665603ull;
+        for (int e = 0; e < experts; ++e) {
+            for (int64_t r = 0; r < 2 * kFF; ++r) {
+                for (int64_t i = 0; i < kH; ++i) row[(size_t) i] = nd(rng);
+                quant(f.gu_type, blob.data() + (size_t) r * f.gu_row, kH);
+            }
+            for (int64_t r = 0; r < kH; ++r) {
+                for (int64_t i = 0; i < kFF; ++i) row[(size_t) i] = nd(rng);
+                quant(f.d_type, blob.data() + f.down_off + (size_t) r * f.d_row, kFF);
+            }
+            for (int t = 0; t < kMaxT; ++t) {
+                std::vector<float> x(kH), h(kFF);
+                for (auto& v : x) v = xd(rng);
+                for (auto& v : h) v = 0.3f * xd(rng);
+                if (q2_gu) { cpu::act_quant_any(x.data(), (int) kH, x2[(size_t) t]); ax[t] = &x2[(size_t) t]; gp2[t] = gv[(size_t) t].data(); up2[t] = uv[(size_t) t].data(); }
+                else cpu::native_quant_act(f, x.data(), xa[(size_t) t].data());
+                if (q2_down) { cpu::act_quant_any(h.data(), (int) kFF, h2[(size_t) t]); a2[t] = &h2[(size_t) t]; }
+                else cpu::native_quant_h(f, h.data(), ha[(size_t) t].data());
+            }
+            for (int nt = 1; nt <= nt_max && nt <= kMaxT; ++nt) {
+                // whole range, then a range starting and ending off any 8/16/64 boundary
+                const int ranges[2][4] = {{0, (int) kFF, 0, (int) kH}, {3, (int) kFF - 5, 7, (int) kH - 9}};
+                for (int inv = 0; inv < 2; ++inv) {
+                    for (int rg = 0; rg < 2; ++rg) {
+                        for (auto& v : ffv) std::fill(v.begin(), v.end(), 0.f);
+                        for (auto& v : outv) std::fill(v.begin(), v.end(), 0.f);
+                        if (q2_gu || q2_down) {
+                            if (inv) continue;   // the Q2_0 rows have one arithmetic
+                            if (q2_gu) {
+                                cpu::q2_rows_any(blob.data(), f.gu_row, (int) (kH / 64), ax, nt, gp2, ranges[rg][0], ranges[rg][1]);
+                                cpu::q2_rows_any(blob.data() + f.up_off, f.gu_row, (int) (kH / 64), ax, nt, up2, ranges[rg][0], ranges[rg][1]);
+                                for (int t = 0; t < nt; ++t)
+                                    for (int r = ranges[rg][0]; r < ranges[rg][1]; ++r)
+                                        ffv[(size_t) t][(size_t) r] = (gp2[t][r] / (1.f + std::exp(-gp2[t][r]))) * up2[t][r];
+                            } else {
+                                cpu::native_gu_rows(f, blob.data(), act, nt, ff, ranges[rg][0], ranges[rg][1]);
+                            }
+                            if (q2_down) cpu::q2_rows_any(blob.data() + f.down_off, f.d_row, (int) (f.n_ff / 64), a2, nt, out, ranges[rg][2], ranges[rg][3]);
+                            else cpu::native_down_rows(f, blob.data(), hq, nt, out, ranges[rg][2], ranges[rg][3]);
+                        } else if (inv) {
+                            cpu::native_gu_rows_invariant(f, blob.data(), act, nt, ff, ranges[rg][0], ranges[rg][1]);
+                            cpu::native_down_rows_invariant(f, blob.data(), hq, nt, out, ranges[rg][2], ranges[rg][3]);
+                        } else {
+                            cpu::native_gu_rows(f, blob.data(), act, nt, ff, ranges[rg][0], ranges[rg][1]);
+                            cpu::native_down_rows(f, blob.data(), hq, nt, out, ranges[rg][2], ranges[rg][3]);
+                        }
+                        uint64_t hg = 1469598103934665603ull, hd = 1469598103934665603ull;
+                        for (int t = 0; t < nt; ++t) {
+                            hg = fnv(hg, ffv[(size_t) t].data(), sizeof(float) * (size_t) kFF);
+                            hd = fnv(hd, outv[(size_t) t].data(), sizeof(float) * (size_t) kH);
+                        }
+                        total = fnv(fnv(total, &hg, 8), &hd, 8);
+                        std::printf("%s e%d nt%d %s range%d gu %016llx down %016llx\n", pair.c_str(), e, nt, inv ? "inv" : "def", rg,
+                                    (unsigned long long) hg, (unsigned long long) hd);
+                    }
+                }
+            }
+        }
+        std::printf("%s TOTAL %016llx\n", pair.c_str(), (unsigned long long) total);
+    }
+    return 0;
+}

src/kernels/cpu/expert.cpp

@@ -4,6 +4,7 @@
 // (rel 1.461e-06 against the ggml formula) and the three performance findings recorded in
 // include/strata/kernels/cpu/expert.hpp.  Read that header first; it says why each piece is shaped this way.
 #include "strata/kernels/cpu/expert.hpp"
+#include "strata/kernels/cpu/expert_layout.hpp"
 
 #include <immintrin.h>
 #if defined(_MSC_VER)
@@ -150,13 +151,53 @@ inline float hsum_ps(__m256 v) {
 // use this kernel, so speculative decode still reproduces plain decode exactly.
 static const bool kZmm = std::getenv("STRATA_CPU_YMM") == nullptr;
 
-inline __m512i unpack64_q2_0(const uint8_t* codes) {
+inline __m512i unpack64_q2_0_ms(const uint8_t* codes) {
     const __m128i packed = _mm_loadu_si128((const __m128i*) codes);          // 8 u16 = 64 codes
     const __m512i lanes = _mm512_cvtepu16_epi64(packed);                    // 8 qwords, 8 codes each
     const __m512i ctrl = _mm512_set1_epi64((long long) 0x0E0C0A0806040200ULL);
     return _mm512_and_si512(_mm512_multishift_epi64_epi8(ctrl, lanes), _mm512_set1_epi8(3));
 }
 
+/// 2-bit unpack WITHOUT VPMULTISHIFTQB (the one VBMI op): byte for byte the multishift's result - output byte j
+/// (code j) is (u16[j/8] >> 2*(j%8)) & 3 - for a CPU with AVX-512 VNNI (dpbusd) but no VBMI (Cascade Lake Xeon;
+/// these used to fall back to the whole AVX-2 kernel).  Four vectors of 16 codes (the u16 pairs 2R and 2R+1), each
+/// a VPERMW source-select + VPSRLVD bit-position shift + mask, truncated to bytes and concatenated in order.  ~8
+/// more instructions per 64 codes: a part of the block's work.
+template <int R>
+inline __m128i unpack_round(const __m512i words, const __m512i shift, const __m512i three) {
+    const __m256i src = _mm256_permutexvar_epi16(                        // code j/8's u16 for j = 16R..16R+15
+        _mm256_setr_epi16((int16_t) (2 * R), (int16_t) (2 * R), (int16_t) (2 * R), (int16_t) (2 * R),
+                          (int16_t) (2 * R), (int16_t) (2 * R), (int16_t) (2 * R), (int16_t) (2 * R),
+                          (int16_t) (2 * R + 1), (int16_t) (2 * R + 1), (int16_t) (2 * R + 1),
+                          (int16_t) (2 * R + 1), (int16_t) (2 * R + 1), (int16_t) (2 * R + 1),
+                          (int16_t) (2 * R + 1), (int16_t) (2 * R + 1)),
+        _mm512_castsi512_si256(words));                                  // the 16 source words
+    const __m512i codes_d = _mm512_and_si512(                            // 16 codes, shifted their bit
+        _mm512_srlv_epi32(_mm512_cvtepu16_epi32(src), shift), _mm512_set1_epi8(3));
+    return _mm512_cvtepi32_epi8(codes_d);                                // 16 dwords -> 16 bytes (VL truncate)
+}
+
+inline __m512i unpack64_q2_0_sw(const uint8_t* codes) {
+    const __m512i words = _mm512_inserti32x4(_mm512_setzero_si512(),      // 8 u16, the rest zero
+                                            _mm_loadu_si128((const __m128i*) codes), 0);
+    const __m512i shift = _mm512_setr_epi32(0, 2, 4, 6, 8, 10, 12, 14, 0, 2, 4, 6, 8, 10, 12, 14);
+    const __m512i three = _mm512_set1_epi8(3);
+    return _mm512_inserti32x4(
+        _mm512_inserti32x4(_mm512_inserti32x4(_mm512_inserti32x4(_mm512_setzero_si512(),
+                                                                 unpack_round<0>(words, shift, three), 0),
+                                              unpack_round<1>(words, shift, three), 1),
+                           unpack_round<2>(words, shift, three), 2),
+        unpack_round<3>(words, shift, three), 3);
+}
+
+inline __m512i unpack64_q2_0(const uint8_t* codes) {
+    static const bool vbmi = cpu_vbmi_ok();          // VPMULTISHIFTQB, or the substitute (bit-shifting) one here
+    return vbmi ? unpack64_q2_0_ms(codes) : unpack64_q2_0_sw(codes);
+}
+
+/// STRATA_CPU_YMM=1 keeps the 256-bit rows (whose unpack keeps the multishift): a machine with VNNI but no VBMI
+/// takes the zmm rows (the substitute unpack above), and anything older takes the AVX-2 kernel (expert_layout.cpp's
+/// dispatch).  So one value answers "can this engine's 512-bit q2 path run at all".
 // For blocks [b0, b0+nb) (nb <= 8): p[2i+h] = d_{b0+i} * xscale[2(b0+i)+h], dd[2i+h] = d_{b0+i}.
 inline void scales8(const uint8_t* wscales, const ActQ& a, int b0, int nb, __m512& p, __m512& dd) {
     const __mmask8 m8 = nb >= 8 ? (__mmask8) 0xFF : (__mmask8) ((1u << nb) - 1u);
@@ -334,6 +375,9 @@ void expert_oracle_q8_0(const uint8_t* blob, const ActQ& a1, float* out, ExpertS
                                blob + O_D_SCALES + size_t(r) * SC_D * 2, ws.a2, SC_D);
 }
 
+bool q2_rows_512_ok() {
+    return kZmm && cpu_vnni_ok();
+}
 }  // namespace
 
 // CpuFeatures::reason, cpu_features, cpu_require_expert_support and the oracle flag live in expert_layout.cpp: this
@@ -630,4 +674,8 @@ void s2_expert_scalar(const uint8_t* blob, const float* x_in, float* out, bool q
     }
 }
 
+bool q2_rows_512_ok() {
+    return kZmm && cpu_vnni_ok();
+}
+
 }  // namespace strata::kernels::cpu

src/kernels/cpu/expert_layout.cpp

@@ -48,6 +48,7 @@ bool cpu_sse42_ok() { return false; }
 bool cpu_gather_fast() { return false; }
 bool cpu_gather_fast_here() { return false; }
 bool cpu_avxvnni_ok() { return false; }
+bool cpu_vnni_ok() { return false; }
 std::string cpu_name() {
 #if defined(__APPLE__)
     return "unknown";   // as before: the Metal build's cpuid shim read all zeros
@@ -71,10 +72,14 @@ std::string cpu_name() {
 #endif  // __APPLE__
 }
 #else
+bool cpu_isa_free() {                                    // STRATA_FORCE_AVX2 and the ISA cap, shared by the probes
+    if (const char* f = std::getenv("STRATA_FORCE_AVX2"); f != nullptr && f[0] == '1') return false;
+    return cpu_isa_cap() >= 3;
+}
+
 bool cpu_avx512_ok() {
     static const bool ok = [] {
-        if (const char* f = std::getenv("STRATA_FORCE_AVX2"); f != nullptr && f[0] == '1') return false;
-        if (cpu_isa_cap() < 3) return false;
+        if (!cpu_isa_free()) return false;
         unsigned r[4] = {0, 0, 0, 0};
         auto cpuid = [&](unsigned leaf, unsigned sub) {
 #if defined(_MSC_VER)
@@ -140,6 +145,64 @@ bool cpu_avx512bw_ok() {
     return ok;
 }
 
+static void isa_probe(unsigned& ebx, unsigned& ecx) {
+    unsigned r[4] = {0, 0, 0, 0};
+    auto cpuid = [&](unsigned leaf, unsigned sub) {
+#if defined(_MSC_VER)
+        int x[4];
+        __cpuidex(x, (int) leaf, (int) sub);
+        for (int i = 0; i < 4; ++i) r[i] = (unsigned) x[i];
+#else
+        __cpuid_count(leaf, sub, r[0], r[1], r[2], r[3]);
+#endif
+    };
+    cpuid(0, 0);
+    if (r[0] < 7) return;
+    cpuid(1, 0);
+    if (!((r[2] >> 27) & 1u)) return;                     // OSXSAVE
+#if defined(_MSC_VER)
+    const unsigned long long xcr0 = _xgetbv(0);
+#else
+    unsigned lo = 0, hi = 0;
+    __asm__ volatile("xgetbv" : "=a"(lo), "=d"(hi) : "c"(0));
+    const unsigned long long xcr0 = ((unsigned long long) hi << 32) | lo;
+#endif
+    if ((xcr0 & 0xE6) != 0xE6) return;                    // the OS saves the AVX-512 state
+    cpuid(7, 0);
+    ebx = (unsigned) r[1];
+    ecx = (unsigned) r[2];
+}
+
+bool cpu_vbmi_ok() {                                     // the byte-permute shuffles (VPERMB, VPMULTISHIFTQB)
+    // STRATA_NO_VBMI=1 answers no (a test knob: it runs the substitute q2 unpack on a CPU that has VBMI)
+    static const bool ok = cpu_isa_free() && std::getenv("STRATA_NO_VBMI") == nullptr && [] {
+        unsigned ebx = 0, ecx = 0;
+        isa_probe(ebx, ecx);
+        // AVX-512F + VL + VBMI (ECX bit 1).  Cascade Lake has F/BW/VL/VNNI but NOT VBMI: that is the
+        // whole reason for the substitute unpack in expert.cpp, so this bit must not be confused with F.
+        return bool(((ebx >> 16) & 1u) && ((ebx >> 31) & 1u) && ((ecx >> 1) & 1u));
+    }();
+    return ok;
+}
+
+bool cpu_vnni_ok() {                                     // VNNI's dpbusd at all, the integer dot at half the lanes
+    static const bool ok = cpu_isa_free() && [] {
+        unsigned ebx = 0, ecx = 0;
+        isa_probe(ebx, ecx);
+        // AVX-512F + DQ + BW + VL + VNNI (ECX bit 11), VBMI not required: the Cascade Lake Xeon tier (DQ is there on every VNNI CPU; asked for
+        // because iq_avx512.cpp is compiled with it).
+        const bool vnni = bool(((ebx >> 16) & 1u) && ((ebx >> 17) & 1u) && ((ebx >> 30) & 1u) && ((ebx >> 31) & 1u) && ((ecx >> 11) & 1u));
+        if (!vnni) return false;
+        // The tier below VBMI is OPT-IN (STRATA_VNNI_TIER=1): on such a CPU the 512-bit i-quant and Q2_0 rows round differently from the
+        // AVX-2 rows it runs today (measured under SDE Cascade Lake: every multi-token gate/up line and the Q2_0 rows differ), so turning it on
+        // changes that CPU's output.  Every CPU that has VBMI as well answers as before.
+        if (cpu_avx512_ok()) return true;
+        const char* e = std::getenv("STRATA_VNNI_TIER");
+        return e != nullptr && e[0] == '1';
+    }();
+    return ok;
+}
+
 bool cpu_avx2_ok() {
     static const bool ok = [] {
         if (cpu_isa_cap() < 2) return false;
@@ -401,7 +464,9 @@ void cpu_require_expert_support() {
 
 void q2_rows_any(const uint8_t* w, size_t row_bytes, int nblocks, const ActQ* const* a, int nt, float* const* out,
                  int r0, int r1) {
-    if (cpu_avx512_ok()) q2_0_gguf_rows_multi(w, row_bytes, nblocks, a, nt, out, r0, r1);
+    // VNNI-512 without VBMI: expert.cpp's zmm row dot has a substitute for the VBMI-only 2-bit unpack there
+    // (q2_rows_512_ok also leaves the STRATA_CPU_YMM=1 choice to the AVX-2 kernel, on this CPU on the rung below).
+    if (q2_rows_512_ok()) q2_0_gguf_rows_multi(w, row_bytes, nblocks, a, nt, out, r0, r1);
     else if (cpu_avx2_ok()) q2_0_gguf_rows_multi_avx2(w, row_bytes, nblocks, a, nt, out, r0, r1);
     else q2_0_gguf_rows_multi_avx1(w, row_bytes, nblocks, a, nt, out, r0, r1);   // issue #1699: the third rung
 }
@@ -414,7 +479,8 @@ void q2_rows_any_legacy(const uint8_t* w, size_t row_bytes, int nblocks, const A
 }
 
 void act_quant_any(const float* x, int n, ActQ& a) {
-    if (cpu_avx512_ok()) act_quant_q8_1(x, n, a);
+    // the q8 quantizer is VBMI-free: a VNNI-but-no-VBMI CPU (Cascade Lake Xeon) keeps its 512-bit form
+    if (cpu_vnni_ok()) act_quant_q8_1(x, n, a);
     else if (cpu_avx2_ok()) act_quant_q8_1_avx2(x, n, a);
     else act_quant_q8_1_avx1(x, n, a);   // issue #1699
 }

src/kernels/cpu/iq_avx512.cpp

@@ -173,6 +173,32 @@ template <> struct Fmt<21> {   // IQ3_S: d, qs[64], qh[8], signs[32], scales[4]
     }
 };
 
+// IQ4_XS (23): d, scales_h (u16), scales_l[4], qs[128].  Chunk j = the 64 values of two 32-value sub-blocks (2j, 2j+1): the
+// table lookup of the nibbles is one vpshufb in 512-bit lanes (kvalues_iq4nl broadcast to every lane), laid out as ggml's AVX2 dot lays
+// out its two 256-bit halves (low nibbles of 16 bytes, then the high nibbles), so the integer lanes line up with that dot's.
+// The decode of the nibble planes follows the layout of PR #5243 (bsorensen110); only this format's part is taken.
+template <> struct Fmt<23> {
+    static constexpr int bytes = 136;
+    static constexpr float K = 1.0f;
+    static inline void decode(const uint8_t* b, int j, __m512i& g, __mmask64& m, __m512i& sc) {
+        const __m128i b0 = _mm_loadu_si128((const __m128i*) (b + 8 + 32 * j));
+        const __m128i b1 = _mm_loadu_si128((const __m128i*) (b + 24 + 32 * j));
+        const __m128i m4 = _mm_set1_epi8(0x0f);
+        const __m256i n0 = _mm256_inserti128_si256(_mm256_castsi128_si256(_mm_and_si128(b0, m4)), _mm_and_si128(_mm_srli_epi16(b0, 4), m4), 1);
+        const __m256i n1 = _mm256_inserti128_si256(_mm256_castsi128_si256(_mm_and_si128(b1, m4)), _mm_and_si128(_mm_srli_epi16(b1, 4), m4), 1);
+        const __m512i nib = _mm512_inserti64x4(_mm512_castsi256_si512(n0), n1, 1);
+        const __m512i lut = _mm512_broadcast_i32x4(_mm_loadu_si128((const __m128i*) kvalues_iq4nl));
+        const __m512i q4 = _mm512_shuffle_epi8(lut, nib);
+        g = _mm512_abs_epi8(q4);
+        m = _mm512_movepi8_mask(q4);
+        const uint8_t sl = b[4 + j];
+        const uint16_t sh = (uint16_t) (u16(b + 2) >> (4 * j));
+        const int ls0 = ((sl & 0x0f) | ((sh << 4) & 0x30)) - 32;
+        const int ls1 = ((sl >> 4) | ((sh << 2) & 0x30)) - 32;
+        sc = scales4(ls0, ls0, ls1, ls1);
+    }
+};
+
 template <bool Singleton> struct RowAccumulator;
 template <> struct RowAccumulator<false> { using Type = __m512; };
 template <> struct RowAccumulator<true> { using Type = __m256; };
@@ -205,7 +231,7 @@ inline void row_dot(const uint8_t* row, int nblocks, const block_q8_K* const* y,
             for (int t = 0; t < NT; ++t) {
                 const __m512i yv = _mm512_loadu_si512((const void*) (y[t][i].qs + 64 * j));
                 const __m512i ys = _mm512_mask_sub_epi8(yv, m, zero, yv);
-                acci[t] = _mm512_add_epi32(acci[t], _mm512_madd_epi16(_mm512_maddubs_epi16(g, ys), sc));
+                acci[t] = _mm512_dpwssd_epi32(acci[t], _mm512_maddubs_epi16(g, ys), sc);
             }
         }
         if constexpr (Singleton) {
@@ -608,6 +634,7 @@ void iq512_reference_gu_rows(int type, const uint8_t* blob, size_t gu_row, size_
         case 18: gu_rows_nt<18, true>(nt, blob, gu_row, up_off, n, act, ff, r0, r1); break;
         case 21: gu_rows_nt<21, true>(nt, blob, gu_row, up_off, n, act, ff, r0, r1); break;
         case 22: gu_rows_nt<22, true>(nt, blob, gu_row, up_off, n, act, ff, r0, r1); break;
+        case 23: gu_rows_nt<23, true>(nt, blob, gu_row, up_off, n, act, ff, r0, r1); break;
     }
 }
 
@@ -652,9 +679,180 @@ void iq512x_rows(int type, const uint8_t* w, size_t row_bytes, int n, const void
         case 18: dot_rows_x_nt<18>(nt, w, row_bytes, n, act, out, r0, r1); break;
         case 21: dot_rows_x_nt<21>(nt, w, row_bytes, n, act, out, r0, r1); break;
         case 22: dot_rows_x_nt<22>(nt, w, row_bytes, n, act, out, r0, r1); break;
+        case 23: dot_rows_x_nt<23>(nt, w, row_bytes, n, act, out, r0, r1); break;
         case 29: rows_m_nt(nt, w, row_bytes, n, act, out, r0, r1); break;
         default: break;
     }
 }
 
+// ---- gate + up in ONE pass over the activations (the idea of PR 5243's row_gu_dot, bsorensen110): the weights of the two rows are decoded side by
+// side and each token's 64 activation bytes are loaded once and applied to both.  The sums are ggml's AVX2 dot's (the integer lanes folded to eight,
+// f16 scale times y.d, eight float lanes, its hsum, K last), so a token's gate and up are ggml-cpu's vec_dot bit for bit at any token count: the
+// same bits as the one-token path of native_expert.cpp (which runs ggml's dot), and the 2+ token rows differ from the 16-lane i-quant rows by
+// rounding only.  The sign masks of IQ2_XXS / IQ2_XS / IQ3_XXS come from VPOPCNTDQ (FmtX).
+namespace {
+
+template <int TY, int NT>
+void gu_fused(const uint8_t* blob, size_t gu_row, size_t up_off, int n, const void* const* act, float* const* ff, int r0, int r1) {
+    const block_q8_K* y[NT];
+    for (int t = 0; t < NT; ++t) y[t] = (const block_q8_K*) act[t];
+    const int nb = n / QK_K;
+    const __m512i zero = _mm512_setzero_si512();
+    for (int r = r0; r < r1; ++r) {
+        const uint8_t* gr = blob + (size_t) r * gu_row;
+        const uint8_t* ur = blob + up_off + (size_t) r * gu_row;
+        __m256 ag[NT], au[NT];
+        for (int t = 0; t < NT; ++t) ag[t] = au[t] = _mm256_setzero_ps();
+        for (int i = 0; i < nb; ++i) {
+            const uint8_t* gb = gr + (size_t) i * FmtX<TY>::bytes;
+            const uint8_t* ub = ur + (size_t) i * FmtX<TY>::bytes;
+            if (prefetch_ahead > 0) {
+                _mm_prefetch((const char*) gb + prefetch_ahead, _MM_HINT_T0);
+                _mm_prefetch((const char*) gb + prefetch_ahead + 64, _MM_HINT_T0);
+                _mm_prefetch((const char*) ub + prefetch_ahead, _MM_HINT_T0);
+                _mm_prefetch((const char*) ub + prefetch_ahead + 64, _MM_HINT_T0);
+            }
+            __m512i ig[NT], iu[NT];
+            for (int t = 0; t < NT; ++t) ig[t] = iu[t] = zero;
+            for (int j = 0; j < 4; ++j) {
+                __m512i gg, gs, ug, us;
+                __mmask64 gm, um;
+                FmtX<TY>::decode(gb, j, gg, gm, gs);
+                FmtX<TY>::decode(ub, j, ug, um, us);
+                for (int t = 0; t < NT; ++t) {
+                    const __m512i yv = _mm512_loadu_si512((const void*) (y[t][i].qs + 64 * j));
+                    ig[t] = _mm512_dpwssd_epi32(ig[t], _mm512_maddubs_epi16(gg, _mm512_mask_sub_epi8(yv, gm, zero, yv)), gs);
+                    iu[t] = _mm512_dpwssd_epi32(iu[t], _mm512_maddubs_epi16(ug, _mm512_mask_sub_epi8(yv, um, zero, yv)), us);
+                }
+            }
+            const float dg = h2f(u16(gb)), du = h2f(u16(ub));
+            for (int t = 0; t < NT; ++t) {
+                const float yd = y[t][i].d;
+                ag[t] = _mm256_fmadd_ps(_mm256_set1_ps(dg * yd), _mm256_cvtepi32_ps(fold256(ig[t])), ag[t]);
+                au[t] = _mm256_fmadd_ps(_mm256_set1_ps(du * yd), _mm256_cvtepi32_ps(fold256(iu[t])), au[t]);
+            }
+        }
+        for (int t = 0; t < NT; ++t) {
+            const float g = FmtX<TY>::K * hsum8_ggml(ag[t]), u = FmtX<TY>::K * hsum8_ggml(au[t]);
+            ff[t][r] = (g / (1.f + std::exp(-g))) * u;
+        }
+    }
+}
+
+template <int TY>
+void gu_fused_nt(int nt, const uint8_t* blob, size_t gu_row, size_t up_off, int n, const void* const* act, float* const* ff, int r0, int r1) {
+    switch (nt) {
+        case 1: gu_fused<TY, 1>(blob, gu_row, up_off, n, act, ff, r0, r1); break;
+        case 2: gu_fused<TY, 2>(blob, gu_row, up_off, n, act, ff, r0, r1); break;
+        case 3: gu_fused<TY, 3>(blob, gu_row, up_off, n, act, ff, r0, r1); break;
+        case 4: gu_fused<TY, 4>(blob, gu_row, up_off, n, act, ff, r0, r1); break;
+        default:
+            for (int t0 = 0; t0 < nt; t0 += 4) {
+                const int k = nt - t0 < 4 ? nt - t0 : 4;
+                gu_fused_nt<TY>(k, blob, gu_row, up_off, n, act + t0, ff + t0, r0, r1);
+            }
+    }
+}
+
+}  // namespace
+
+bool iq512f_supported(int type) noexcept { return has_vpopcnt() && (type == 16 || type == 17 || type == 18 || type == 21 || type == 22); }
+
+void iq512_fused_gu_rows(int type, const uint8_t* blob, size_t gu_row, size_t up_off, int n, const void* const* act, int nt,
+                         float* const* ff, int r0, int r1) {
+    switch (type) {
+        case 16: gu_fused_nt<16>(nt, blob, gu_row, up_off, n, act, ff, r0, r1); break;
+        case 17: gu_fused_nt<17>(nt, blob, gu_row, up_off, n, act, ff, r0, r1); break;
+        case 18: gu_fused_nt<18>(nt, blob, gu_row, up_off, n, act, ff, r0, r1); break;
+        case 21: gu_fused_nt<21>(nt, blob, gu_row, up_off, n, act, ff, r0, r1); break;
+        case 22: gu_fused_nt<22>(nt, blob, gu_row, up_off, n, act, ff, r0, r1); break;
+        default: break;
+    }
+}
+
+// ---- Q5_0 (6) down rows in 512-bit lanes: two 32-value blocks a vector, the weights of a block pair decoded once for
+// every token.  The arithmetic is ggml-cpu's AVX2 dot, order for order: per block the 8 integer lanes (values 4l .. 4l+3 of the
+// block, one vpdpbusd lane each), converted, scaled by f16(dx) * f16(dy) and fused into ONE eight-lane float accumulator in block order,
+// hsum last: a token's row is ggml-cpu's dot bit for bit.  An odd block count ends on a 256-bit block.  (An IQ4_NL version of the same
+// shape measured 5-11% SLOWER than iq4nl256_down_rows on Zen 4 at 2-4 tokens and was dropped.)
+namespace {
+
+inline __m512i pair_q8_0(const block_q8_0* a, const block_q8_0* b) {
+    return _mm512_inserti64x4(_mm512_castsi256_si512(_mm256_loadu_si256((const __m256i*) a->qs)),
+                              _mm256_loadu_si256((const __m256i*) b->qs), 1);
+}
+
+// 32 signed weights of a Q5_0 block: the nibble (low plane, then high plane) minus 16 where the fifth bit is clear
+inline __m256i q50_weights(const block_q5_0* x) {
+    const __m128i bits = _mm_loadu_si128((const __m128i*) x->qs);
+    const __m128i m4 = _mm_set1_epi8(0x0f);
+    const __m256i nib = _mm256_inserti128_si256(_mm256_castsi128_si256(_mm_and_si128(bits, m4)), _mm_and_si128(_mm_srli_epi16(bits, 4), m4), 1);
+    uint32_t qh;
+    std::memcpy(&qh, x->qh, 4);
+    return _mm256_mask_sub_epi8(nib, (__mmask32) ~qh, nib, _mm256_set1_epi8(16));
+}
+
+template <int NT>
+void qd_rows(const uint8_t* w, size_t row_bytes, int n, const block_q8_0* const* y, float* const* out, int r0, int r1) {
+    constexpr size_t kBlk = sizeof(block_q5_0);
+    const int nb = n / 32;
+    const int pf = prefetch_ahead;
+    for (int r = r0; r < r1; ++r) {
+        const uint8_t* row = w + (size_t) r * row_bytes;
+        __m256 acc[NT];
+        for (int t = 0; t < NT; ++t) acc[t] = _mm256_setzero_ps();
+        int ib = 0;
+        for (; ib + 1 < nb; ib += 2) {
+            const uint8_t* b0 = row + (size_t) ib * kBlk;
+            if (pf > 0) { _mm_prefetch((const char*) b0 + pf, _MM_HINT_T0); _mm_prefetch((const char*) b0 + pf + 64, _MM_HINT_T0); }
+            const __m512i wq = _mm512_inserti64x4(_mm512_castsi256_si512(q50_weights((const block_q5_0*) b0)), q50_weights((const block_q5_0*) (b0 + kBlk)), 1);
+            const float d0 = h2f(((const block_q5_0*) b0)->d);
+            const float d1 = h2f(((const block_q5_0*) (b0 + kBlk))->d);
+            const __m512i aw = _mm512_abs_epi8(wq);
+            const __mmask64 neg = _mm512_movepi8_mask(wq);
+            for (int t = 0; t < NT; ++t) {
+                const __m512i yv = pair_q8_0(&y[t][ib], &y[t][ib + 1]);
+                const __m512i p = _mm512_dpbusd_epi32(_mm512_setzero_si512(), aw, _mm512_mask_sub_epi8(yv, neg, _mm512_setzero_si512(), yv));
+                const __m256 lo = _mm256_cvtepi32_ps(_mm512_castsi512_si256(p));
+                const __m256 hi = _mm256_cvtepi32_ps(_mm512_extracti64x4_epi64(p, 1));
+                acc[t] = _mm256_fmadd_ps(_mm256_set1_ps(d0 * h2f(y[t][ib].d)), lo, acc[t]);
+                acc[t] = _mm256_fmadd_ps(_mm256_set1_ps(d1 * h2f(y[t][ib + 1].d)), hi, acc[t]);
+            }
+        }
+        if (ib < nb) {
+            const uint8_t* b0 = row + (size_t) ib * kBlk;
+            const __m256i wq = q50_weights((const block_q5_0*) b0);
+            const float d0 = h2f(((const block_q5_0*) b0)->d);
+            for (int t = 0; t < NT; ++t) {
+                const __m256i yv = _mm256_loadu_si256((const __m256i*) y[t][ib].qs);
+                const __m256i p = _mm256_dpbusd_epi32(_mm256_setzero_si256(), _mm256_abs_epi8(wq), _mm256_sign_epi8(yv, wq));
+                acc[t] = _mm256_fmadd_ps(_mm256_set1_ps(d0 * h2f(y[t][ib].d)), _mm256_cvtepi32_ps(p), acc[t]);
+            }
+        }
+        for (int t = 0; t < NT; ++t) out[t][r] = hsum8_ggml(acc[t]);
+    }
+}
+
+void qd_rows_nt(int nt, const uint8_t* w, size_t row_bytes, int n, const void* const* hq, float* const* out, int r0, int r1) {
+    const block_q8_0* y[8];
+    for (int t = 0; t < nt; ++t) y[t] = (const block_q8_0*) hq[t];
+    switch (nt) {
+        case 1: qd_rows<1>(w, row_bytes, n, y, out, r0, r1); break;
+        case 2: qd_rows<2>(w, row_bytes, n, y, out, r0, r1); break;
+        case 3: qd_rows<3>(w, row_bytes, n, y, out, r0, r1); break;
+        case 4: qd_rows<4>(w, row_bytes, n, y, out, r0, r1); break;
+        default:
+            for (int t0 = 0; t0 < nt; t0 += 4) {
+                const int k = nt - t0 < 4 ? nt - t0 : 4;
+                qd_rows_nt(k, w, row_bytes, n, hq + t0, out + t0, r0, r1);
+            }
+    }
+}
+
+}  // namespace
+
+void q50_512_down_rows(const uint8_t* w, size_t row_bytes, int n, const void* const* hq, int nt, float* const* out, int r0, int r1) {
+    qd_rows_nt(nt, w, row_bytes, n, hq, out, r0, r1);
+}
+
 }  // namespace strata::kernels::cpu

src/kernels/cpu/kq_cpu_bench.cpp

@@ -1,214 +1,224 @@
-// src/kernels/cpu/kq_cpu_bench.cpp - MODEL-UPDATE: how close the CPU pool's expert rows come to the RAM bandwidth, per
-// expert format.  No GPU, no model: --mb MB of expert blobs (ggml-quantized random weights, far more than any L3) are
-// computed by T threads at once through the engine's own entry points (native_gu_rows + native_down_rows, the pool's
-// calls), each thread walking its own experts, nt tokens per expert - the decode case, where every CPU expert is a
-// fresh read from DRAM.  Prints the weight bytes per second against the --ram-gbs given (a separate read test).
-//
-//     kq_cpu_bench [--pairs q4_K/q5_1,q5_K/q8_0,q6_K/q8_0,q8_0/q8_0,iq3_s/iq4_nl] [--threads 1,8,16,32] [--nt 1,2,4]
-//                  [--mb 4096] [--reps 3] [--ram-gbs N]
-#include "strata/kernels/cpu/expert_layout.hpp"
-#include "strata/kernels/cpu/native_expert.hpp"
-
-#include "ggml.h"
-#include "ggml-cpu.h"
-
-#include <algorithm>
-#include <atomic>
-#include <chrono>
-#include <cmath>
-#include <cstdio>
-#include <cstdlib>
-#include <cstring>
-#include <memory>
-#include <random>
-#include <string>
-#include <thread>
-#include <vector>
-
-namespace cpu = strata::kernels::cpu;
-
-namespace {
-
-constexpr int64_t kH = 2560, kFF = 640;
-constexpr int kMaxT = 8;
-
-int type_of(const std::string& s) {
-    for (int t = 0; t < GGML_TYPE_COUNT; ++t) {
-        const char* n = ggml_type_name((ggml_type) t);
-        if (n != nullptr && s == n) return t;
-    }
-    return -1;
-}
-
-std::vector<std::string> split(const std::string& s) {
-    std::vector<std::string> v;
-    size_t from = 0;
-    for (;;) {
-        const size_t c = s.find(',', from);
-        v.push_back(s.substr(from, c == std::string::npos ? std::string::npos : c - from));
-        if (c == std::string::npos) break;
-        from = c + 1;
-    }
-    return v;
-}
-
-// one expert blob in the GGUF form: gate rows, up rows (n_ff x n_embd each), down rows (n_embd x n_ff), from random
-// weights quantized by ggml (valid scales, as a real file has).  ggml_quantize_chunk, not the CPU traits' from_float:
-// ggml-cpu has no from_float for the i-quants (IQ3_S called a null pointer: an access violation on Windows)
-void make_blob(const cpu::NativeFmt& f, uint8_t* blob, std::mt19937& rng) {
-    std::normal_distribution<float> nd(0.f, 0.02f);
-    std::vector<float> row((size_t) std::max(kH, kFF));
-    const std::vector<float> imatrix((size_t) std::max(kH, kFF), 1.0f);
-    auto quant = [&](int type, uint8_t* dst, int64_t n) {
-        ggml_quantize_chunk((ggml_type) type, row.data(), dst, 0, 1, n,
-                            ggml_quantize_requires_imatrix((ggml_type) type) ? imatrix.data() : nullptr);
-    };
-    for (int64_t r = 0; r < 2 * kFF; ++r) {
-        for (int64_t i = 0; i < kH; ++i) row[(size_t) i] = nd(rng);
-        quant(f.gu_type, blob + (size_t) r * f.gu_row, kH);
-    }
-    for (int64_t r = 0; r < kH; ++r) {
-        for (int64_t i = 0; i < kFF; ++i) row[(size_t) i] = nd(rng);
-        quant(f.d_type, blob + f.down_off + (size_t) r * f.d_row, kFF);
-    }
-}
-
-}  // namespace
-
-int main(int argc, char** argv) {
-    setvbuf(stdout, nullptr, _IONBF, 0);
-    std::vector<std::string> pairs = split("q4_K/q5_1,q5_K/q8_0,q6_K/q8_0,q8_0/q8_0,iq3_s/iq4_nl");
-    std::vector<int> threads = {1, 8, 16, 32}, nts = {1, 2, 4};
-    int mb = 4096, reps = 3;
-    double ram = 0;
-    for (int i = 1; i < argc; ++i) {
-        const std::string a = argv[i];
-        const bool more = i + 1 < argc;
-        auto ints = [](const std::string& s) { std::vector<int> v; for (auto& x : split(s)) v.push_back(std::atoi(x.c_str())); return v; };
-        if (a == "--pairs" && more) pairs = split(argv[++i]);
-        else if (a == "--threads" && more) threads = ints(argv[++i]);
-        else if (a == "--nt" && more) nts = ints(argv[++i]);
-        else if (a == "--mb" && more) mb = std::atoi(argv[++i]);
-        else if (a == "--reps" && more) reps = std::max(1, std::atoi(argv[++i]));
-        else if (a == "--ram-gbs" && more) ram = std::atof(argv[++i]);
-        else {
-            std::fprintf(stderr, "usage: kq_cpu_bench [--pairs a/b,...] [--threads 1,8] [--nt 1,2,4] [--mb 4096] [--reps 3] [--ram-gbs N]\n");
-            return 2;
-        }
-    }
-    ggml_cpu_init();
-    std::printf("kq_cpu_bench: %s (avx2 %d, avx512 %d), %d MB of blobs per format\n", cpu::cpu_name().c_str(),
-                (int) cpu::cpu_avx2_ok(), (int) cpu::cpu_avx512_ok(), mb);
-    for (const std::string& pair : pairs) {
-        const size_t slash = pair.find('/');
-        const int gt = slash == std::string::npos ? -1 : type_of(pair.substr(0, slash));
-        const int dt = slash == std::string::npos ? -1 : type_of(pair.substr(slash + 1));
-        cpu::NativeFmt f;
-        std::string err;
-        if (gt < 0 || dt < 0 || !cpu::native_fmt(gt, dt, kH, kFF, f, err)) {
-            std::printf("%s: %s\n", pair.c_str(), err.empty() ? "unknown types" : err.c_str());
-            continue;
-        }
-        ggml_quantize_init((ggml_type) gt);   // the i-quants' quantizers need their grids first
-        ggml_quantize_init((ggml_type) dt);
-        const size_t E = std::max<size_t>(64, ((size_t) mb << 20) / f.bytes);
-        std::unique_ptr<uint8_t[]> mem(new uint8_t[E * f.bytes + 4096]);
-        uint8_t* blobs = (uint8_t*) (((uintptr_t) mem.get() + 4095) & ~(uintptr_t) 4095);
-        {   // 16 distinct experts, copied over the whole buffer (the bytes in DRAM are what is measured)
-            std::mt19937 rng(1234 + gt * 64 + dt);
-            const size_t U = std::min<size_t>(16, E);
-            for (size_t e = 0; e < U; ++e) make_blob(f, blobs + e * f.bytes, rng);
-            for (size_t e = U; e < E; ++e) std::memcpy(blobs + e * f.bytes, blobs + (e % U) * f.bytes, f.bytes);
-        }
-        // activations: kMaxT tokens of x (n_embd) and h (n_ff), quantized the way the pool quantizes them
-        std::mt19937 rng(77);
-        std::normal_distribution<float> nd(0.f, 1.f);
-        std::vector<std::vector<uint8_t>> xa(kMaxT, std::vector<uint8_t>(cpu::kNativeActBytes)),
-            ha(kMaxT, std::vector<uint8_t>(cpu::kNativeHBytes));
-        const void* act[kMaxT];
-        const void* hq[kMaxT];
-        // a Q2_0 down takes the pool's own path (pool.cpp, run_split_multi_native and drain mode 6): the activation
-        // as an ActQ (act_quant_any) and Strata's Q2_0 rows (q2_rows_any), not ggml-cpu's Q2_0 dot
-        const bool q2_down = cpu::q2_native_kernels(f.d_type);
-        std::vector<cpu::ActQ> h2(q2_down ? kMaxT : 0);
-        const cpu::ActQ* a2[kMaxT] = {};
-        // a Q2_0 gate/up the same (pool.cpp mode 5): x as an ActQ, gate and up on q2_rows_any, then SwiGLU
-        const bool q2_gu = cpu::q2_native_kernels(f.gu_type);
-        std::vector<cpu::ActQ> x2(q2_gu ? kMaxT : 0);
-        const cpu::ActQ* ax[kMaxT] = {};
-        for (int t = 0; t < kMaxT; ++t) {
-            std::vector<float> x(kH), h(kFF);
-            for (auto& v : x) v = nd(rng);
-            for (auto& v : h) v = 0.1f * nd(rng);
-            if (q2_gu) {
-                cpu::act_quant_any(x.data(), (int) kH, x2[(size_t) t]);
-                ax[t] = &x2[(size_t) t];
-            } else {
-                cpu::native_quant_act(f, x.data(), xa[t].data());
-            }
-            if (q2_down) {
-                cpu::act_quant_any(h.data(), (int) kFF, h2[(size_t) t]);
-                a2[t] = &h2[(size_t) t];
-            } else {
-                cpu::native_quant_h(f, h.data(), ha[t].data());
-            }
-            act[t] = xa[t].data();
-            hq[t] = ha[t].data();
-        }
-        std::printf("\n%s: %zu experts of %.3f MB (%.0f MB)%s%s\n", pair.c_str(), E, f.bytes / 1e6, E * (double) f.bytes / 1e6,
-                    q2_gu ? ", Q2_0 gate/up" : "", q2_down ? ", Q2_0 down on Strata's rows (the pool's path)" : "");
-        for (int nt : nts) {
-            for (int T : threads) {
-                if (T < 1) continue;
-                std::vector<double> gbs;
-                for (int rep = 0; rep < reps; ++rep) {
-                    std::atomic<size_t> next{0};
-                    auto work = [&] {
-                        std::vector<float> ffb((size_t) kMaxT * kFF), outb((size_t) kMaxT * kH);
-                        std::vector<float> gb(q2_gu ? (size_t) kMaxT * kFF : 0), ub(q2_gu ? (size_t) kMaxT * kFF : 0);
-                        float* gp[kMaxT] = {};
-                        float* up[kMaxT] = {};
-                        if (q2_gu)
-                            for (int t = 0; t < kMaxT; ++t) { gp[t] = &gb[(size_t) t * kFF]; up[t] = &ub[(size_t) t * kFF]; }
-                        float* ff[kMaxT];
-                        float* out[kMaxT];
-                        for (int t = 0; t < kMaxT; ++t) { ff[t] = &ffb[(size_t) t * kFF]; out[t] = &outb[(size_t) t * kH]; }
-                        std::vector<std::vector<uint8_t>> hb(kMaxT, std::vector<uint8_t>(cpu::kNativeHBytes));
-                        for (;;) {
-                            const size_t e = next.fetch_add(1);
-                            if (e >= E) break;
-                            const uint8_t* blob = blobs + e * f.bytes;
-                            if (q2_gu) {
-                                cpu::q2_rows_any(blob, f.gu_row, (int) (kH / 64), ax, nt, gp, 0, (int) kFF);
-                                cpu::q2_rows_any(blob + f.up_off, f.gu_row, (int) (kH / 64), ax, nt, up, 0, (int) kFF);
-                                for (int t = 0; t < nt; ++t)
-                                    for (int64_t r = 0; r < kFF; ++r)
-                                        ff[t][r] = (gp[t][r] / (1.f + std::exp(-gp[t][r]))) * up[t][r];
-                            } else {
-                                cpu::native_gu_rows(f, blob, act, nt, ff, 0, (int) kFF);
-                            }
-                            if (q2_down)
-                                cpu::q2_rows_any(blob + f.down_off, f.d_row, (int) (f.n_ff / 64), a2, nt, out, 0, (int) kH);
-                            else
-                                cpu::native_down_rows(f, blob, hq, nt, out, 0, (int) kH);
-                        }
-                        volatile float sink = outb[0] + ffb[0];
-                        (void) sink;
-                    };
-                    const auto t0 = std::chrono::steady_clock::now();
-                    std::vector<std::thread> th;
-                    for (int k = 0; k < T; ++k) th.emplace_back(work);
-                    for (auto& x : th) x.join();
-                    const double s = std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();
-                    gbs.push_back(E * (double) f.bytes / s / 1e9);
-                }
-                std::sort(gbs.begin(), gbs.end());
-                const double med = gbs[gbs.size() / 2];
-                std::printf("  nt %d threads %2d: %6.1f GB/s of weights (%.3f ms per expert per thread)%s\n", nt, T, med,
-                            f.bytes / (med * 1e9 / T) * 1e3,
-                            ram > 0 ? (" = " + std::to_string((int) std::lround(100 * med / ram)) + "% of RAM").c_str() : "");
-            }
-        }
-    }
-    return 0;
-}
+// src/kernels/cpu/kq_cpu_bench.cpp - MODEL-UPDATE: how close the CPU pool's expert rows come to the RAM bandwidth, per
+// expert format.  No GPU, no model: --mb MB of expert blobs (ggml-quantized random weights, far more than any L3) are
+// computed by T threads at once through the engine's own entry points (native_gu_rows + native_down_rows, the pool's
+// calls), each thread walking its own experts, nt tokens per expert - the decode case, where every CPU expert is a
+// fresh read from DRAM.  Prints the weight bytes per second against the --ram-gbs given (a separate read test).
+//
+//     kq_cpu_bench [--pairs q4_K/q5_1,q5_K/q8_0,q6_K/q8_0,q8_0/q8_0,iq3_s/iq4_nl] [--threads 1,8,16,32] [--nt 1,2,4]
+//                  [--mb 4096] [--reps 3] [--ram-gbs N] [--part both|gu|down] [--hot K]
+//
+// --part times only the gate/up rows (gu) or only the down rows (down); --hot K walks only K distinct expert blobs over and
+// over (K=1: one expert, cache-resident: the arithmetic and instruction throughput; the default walks --mb of blobs: DRAM).
+#include "strata/kernels/cpu/expert_layout.hpp"
+#include "strata/kernels/cpu/native_expert.hpp"
+
+#include "ggml.h"
+#include "ggml-cpu.h"
+
+#include <algorithm>
+#include <atomic>
+#include <chrono>
+#include <cmath>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <memory>
+#include <random>
+#include <string>
+#include <thread>
+#include <vector>
+
+namespace cpu = strata::kernels::cpu;
+
+namespace {
+
+constexpr int64_t kH = 2560, kFF = 640;
+constexpr int kMaxT = 8;
+
+int type_of(const std::string& s) {
+    for (int t = 0; t < GGML_TYPE_COUNT; ++t) {
+        const char* n = ggml_type_name((ggml_type) t);
+        if (n != nullptr && s == n) return t;
+    }
+    return -1;
+}
+
+std::vector<std::string> split(const std::string& s) {
+    std::vector<std::string> v;
+    size_t from = 0;
+    for (;;) {
+        const size_t c = s.find(',', from);
+        v.push_back(s.substr(from, c == std::string::npos ? std::string::npos : c - from));
+        if (c == std::string::npos) break;
+        from = c + 1;
+    }
+    return v;
+}
+
+// one expert blob in the GGUF form: gate rows, up rows (n_ff x n_embd each), down rows (n_embd x n_ff), from random
+// weights quantized by ggml (valid scales, as a real file has).  ggml_quantize_chunk, not the CPU traits' from_float:
+// ggml-cpu has no from_float for the i-quants (IQ3_S called a null pointer: an access violation on Windows)
+void make_blob(const cpu::NativeFmt& f, uint8_t* blob, std::mt19937& rng) {
+    std::normal_distribution<float> nd(0.f, 0.02f);
+    std::vector<float> row((size_t) std::max(kH, kFF));
+    const std::vector<float> imatrix((size_t) std::max(kH, kFF), 1.0f);
+    auto quant = [&](int type, uint8_t* dst, int64_t n) {
+        ggml_quantize_chunk((ggml_type) type, row.data(), dst, 0, 1, n,
+                            ggml_quantize_requires_imatrix((ggml_type) type) ? imatrix.data() : nullptr);
+    };
+    for (int64_t r = 0; r < 2 * kFF; ++r) {
+        for (int64_t i = 0; i < kH; ++i) row[(size_t) i] = nd(rng);
+        quant(f.gu_type, blob + (size_t) r * f.gu_row, kH);
+    }
+    for (int64_t r = 0; r < kH; ++r) {
+        for (int64_t i = 0; i < kFF; ++i) row[(size_t) i] = nd(rng);
+        quant(f.d_type, blob + f.down_off + (size_t) r * f.d_row, kFF);
+    }
+}
+
+}  // namespace
+
+int main(int argc, char** argv) {
+    setvbuf(stdout, nullptr, _IONBF, 0);
+    std::vector<std::string> pairs = split("q4_K/q5_1,q5_K/q8_0,q6_K/q8_0,q8_0/q8_0,iq3_s/iq4_nl");
+    std::vector<int> threads = {1, 8, 16, 32}, nts = {1, 2, 4};
+    int mb = 4096, reps = 3, hot = 0;
+    double ram = 0;
+    std::string part = "both";
+    for (int i = 1; i < argc; ++i) {
+        const std::string a = argv[i];
+        const bool more = i + 1 < argc;
+        auto ints = [](const std::string& s) { std::vector<int> v; for (auto& x : split(s)) v.push_back(std::atoi(x.c_str())); return v; };
+        if (a == "--pairs" && more) pairs = split(argv[++i]);
+        else if (a == "--threads" && more) threads = ints(argv[++i]);
+        else if (a == "--nt" && more) nts = ints(argv[++i]);
+        else if (a == "--mb" && more) mb = std::atoi(argv[++i]);
+        else if (a == "--reps" && more) reps = std::max(1, std::atoi(argv[++i]));
+        else if (a == "--ram-gbs" && more) ram = std::atof(argv[++i]);
+        else if (a == "--part" && more) part = argv[++i];
+        else if (a == "--hot" && more) hot = std::max(0, std::atoi(argv[++i]));
+        else {
+            std::fprintf(stderr, "usage: kq_cpu_bench [--pairs a/b,...] [--threads 1,8] [--nt 1,2,4] [--mb 4096] [--reps 3] [--ram-gbs N]\n");
+            return 2;
+        }
+    }
+    const bool do_gu = part != "down", do_down = part != "gu";
+    ggml_cpu_init();
+    std::printf("kq_cpu_bench: %s (avx2 %d, avx512 %d), %d MB of blobs per format\n", cpu::cpu_name().c_str(),
+                (int) cpu::cpu_avx2_ok(), (int) cpu::cpu_avx512_ok(), mb);
+    for (const std::string& pair : pairs) {
+        const size_t slash = pair.find('/');
+        const int gt = slash == std::string::npos ? -1 : type_of(pair.substr(0, slash));
+        const int dt = slash == std::string::npos ? -1 : type_of(pair.substr(slash + 1));
+        cpu::NativeFmt f;
+        std::string err;
+        if (gt < 0 || dt < 0 || !cpu::native_fmt(gt, dt, kH, kFF, f, err)) {
+            std::printf("%s: %s\n", pair.c_str(), err.empty() ? "unknown types" : err.c_str());
+            continue;
+        }
+        ggml_quantize_init((ggml_type) gt);   // the i-quants' quantizers need their grids first
+        ggml_quantize_init((ggml_type) dt);
+        const size_t E = std::max<size_t>(64, ((size_t) mb << 20) / f.bytes);
+        std::unique_ptr<uint8_t[]> mem(new uint8_t[E * f.bytes + 4096]);
+        uint8_t* blobs = (uint8_t*) (((uintptr_t) mem.get() + 4095) & ~(uintptr_t) 4095);
+        {   // 16 distinct experts, copied over the whole buffer (the bytes in DRAM are what is measured)
+            std::mt19937 rng(1234 + gt * 64 + dt);
+            const size_t U = std::min<size_t>(16, E);
+            for (size_t e = 0; e < U; ++e) make_blob(f, blobs + e * f.bytes, rng);
+            for (size_t e = U; e < E; ++e) std::memcpy(blobs + e * f.bytes, blobs + (e % U) * f.bytes, f.bytes);
+        }
+        // activations: kMaxT tokens of x (n_embd) and h (n_ff), quantized the way the pool quantizes them
+        std::mt19937 rng(77);
+        std::normal_distribution<float> nd(0.f, 1.f);
+        std::vector<std::vector<uint8_t>> xa(kMaxT, std::vector<uint8_t>(cpu::kNativeActBytes)),
+            ha(kMaxT, std::vector<uint8_t>(cpu::kNativeHBytes));
+        const void* act[kMaxT];
+        const void* hq[kMaxT];
+        // a Q2_0 down takes the pool's own path (pool.cpp, run_split_multi_native and drain mode 6): the activation
+        // as an ActQ (act_quant_any) and Strata's Q2_0 rows (q2_rows_any), not ggml-cpu's Q2_0 dot
+        const bool q2_down = cpu::q2_native_kernels(f.d_type);
+        std::vector<cpu::ActQ> h2(q2_down ? kMaxT : 0);
+        const cpu::ActQ* a2[kMaxT] = {};
+        // a Q2_0 gate/up the same (pool.cpp mode 5): x as an ActQ, gate and up on q2_rows_any, then SwiGLU
+        const bool q2_gu = cpu::q2_native_kernels(f.gu_type);
+        std::vector<cpu::ActQ> x2(q2_gu ? kMaxT : 0);
+        const cpu::ActQ* ax[kMaxT] = {};
+        for (int t = 0; t < kMaxT; ++t) {
+            std::vector<float> x(kH), h(kFF);
+            for (auto& v : x) v = nd(rng);
+            for (auto& v : h) v = 0.1f * nd(rng);
+            if (q2_gu) {
+                cpu::act_quant_any(x.data(), (int) kH, x2[(size_t) t]);
+                ax[t] = &x2[(size_t) t];
+            } else {
+                cpu::native_quant_act(f, x.data(), xa[t].data());
+            }
+            if (q2_down) {
+                cpu::act_quant_any(h.data(), (int) kFF, h2[(size_t) t]);
+                a2[t] = &h2[(size_t) t];
+            } else {
+                cpu::native_quant_h(f, h.data(), ha[t].data());
+            }
+            act[t] = xa[t].data();
+            hq[t] = ha[t].data();
+        }
+        std::printf("\n%s: %zu experts of %.3f MB (%.0f MB)%s%s\n", pair.c_str(), E, f.bytes / 1e6, E * (double) f.bytes / 1e6,
+                    q2_gu ? ", Q2_0 gate/up" : "", q2_down ? ", Q2_0 down on Strata's rows (the pool's path)" : "");
+        const double part_bytes = do_gu && do_down ? (double) f.bytes : do_gu ? (double) f.down_off : (double) (f.bytes - f.down_off);
+        for (int nt : nts) {
+            for (int T : threads) {
+                if (T < 1) continue;
+                std::vector<double> gbs;
+                for (int rep = 0; rep < reps; ++rep) {
+                    std::atomic<size_t> next{0};
+                    auto work = [&] {
+                        std::vector<float> ffb((size_t) kMaxT * kFF), outb((size_t) kMaxT * kH);
+                        std::vector<float> gb(q2_gu ? (size_t) kMaxT * kFF : 0), ub(q2_gu ? (size_t) kMaxT * kFF : 0);
+                        float* gp[kMaxT] = {};
+                        float* up[kMaxT] = {};
+                        if (q2_gu)
+                            for (int t = 0; t < kMaxT; ++t) { gp[t] = &gb[(size_t) t * kFF]; up[t] = &ub[(size_t) t * kFF]; }
+                        float* ff[kMaxT];
+                        float* out[kMaxT];
+                        for (int t = 0; t < kMaxT; ++t) { ff[t] = &ffb[(size_t) t * kFF]; out[t] = &outb[(size_t) t * kH]; }
+                        std::vector<std::vector<uint8_t>> hb(kMaxT, std::vector<uint8_t>(cpu::kNativeHBytes));
+                        for (;;) {
+                            const size_t e = next.fetch_add(1);
+                            if (e >= E) break;
+                            const uint8_t* blob = blobs + (hot > 0 ? e % (size_t) hot : e) * f.bytes;
+                            if (!do_gu) {
+                            } else if (q2_gu) {
+                                cpu::q2_rows_any(blob, f.gu_row, (int) (kH / 64), ax, nt, gp, 0, (int) kFF);
+                                cpu::q2_rows_any(blob + f.up_off, f.gu_row, (int) (kH / 64), ax, nt, up, 0, (int) kFF);
+                                for (int t = 0; t < nt; ++t)
+                                    for (int64_t r = 0; r < kFF; ++r)
+                                        ff[t][r] = (gp[t][r] / (1.f + std::exp(-gp[t][r]))) * up[t][r];
+                            } else {
+                                cpu::native_gu_rows(f, blob, act, nt, ff, 0, (int) kFF);
+                            }
+                            if (!do_down) {
+                            } else if (q2_down)
+                                cpu::q2_rows_any(blob + f.down_off, f.d_row, (int) (f.n_ff / 64), a2, nt, out, 0, (int) kH);
+                            else
+                                cpu::native_down_rows(f, blob, hq, nt, out, 0, (int) kH);
+                        }
+                        volatile float sink = outb[0] + ffb[0];
+                        (void) sink;
+                    };
+                    const auto t0 = std::chrono::steady_clock::now();
+                    std::vector<std::thread> th;
+                    for (int k = 0; k < T; ++k) th.emplace_back(work);
+                    for (auto& x : th) x.join();
+                    const double s = std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();
+                    gbs.push_back(E * part_bytes / s / 1e9);
+                }
+                std::sort(gbs.begin(), gbs.end());
+                const double med = gbs[gbs.size() / 2];
+                std::printf("  nt %d threads %2d: %6.1f GB/s of weights (%.3f ms per expert per thread)%s\n", nt, T, med,
+                            part_bytes / (med * 1e9 / T) * 1e3,
+                            ram > 0 ? (" = " + std::to_string((int) std::lround(100 * med / ram)) + "% of RAM").c_str() : "");
+            }
+        }
+    }
+    return 0;
+}