ATIVX928 d284809fb0 V100: operator fusions for the decode GEMV / MoE paths, bitwise ...

Fusions (all bit-exact vs the pipelines they replace; bench/micro/fusions_parity.cu
checks every one bitwise and times the before/after at decode shapes):

- gate/up dual GEMV + SwiGLU in ONE kernel (llama.cpp's mul_mat_gated):
  s_gemv_pair_silu, one read of the activation for both projections and the
  SwiGLU in the epilogue - the up intermediate never lands in memory.  Rows are
  accumulated by the shared row functions (s_rowdev.cuh) exactly as the
  standalone kernels accumulate them.  shared_expert uses it whenever both
  projections are S-family (bitwise checked for all three silu kinds).
- silu(gate)*up + the intermediate quantizer fused (was 2 kernels):
  swilu_quantize_q8_0/_scaled/_q8_K, warp-parallel swilu then the verbatim
  quantizer body.  Wired into the S2 expert entry points and shared_expert.
- every activation image of x in one pass (was 2-3 kernels): quantize_act_images
  produces block_q8_0 + block_q8_K + bf16 from one read of x.  Wired into
  gdn_layer, qsa_layer and moe_shared.
- multi-weight bf16 GEMV (llama.cpp's mul_mat_multi): bf16_gemv_pair computes
  two matrices from one read of x; used for ssm_alpha/ssm_beta and for the
  indexer's q/k projections (q and k now one launch).

The call sites are gated on STRATA_EXPERIMENTAL_SM60 (the Pascal/Volta build),
so the ready-made engine runs exactly the pipelines it did before; the fused
kernels are the same code either way.

Measured (V100, CUDA events, n_embd 2560 / FF 640, 200 reps):
  expert intermediate (swilu+q8_0)   15.7 -> 11.6 us  M=1,  18.8 -> 11.4  M=8
  activation images (q8_0+q8_K+bf16) 46.2 -> 30.4 us
  dual GEMV + swilu (gate,up)        30.4 -> 14.8 us
  bf16 pair (alpha,beta)             24.2 -> 14.4 us  M=1,  23.7 -> 14.1  M=8

Parity: fusions_parity 0 failures (bitwise), plus s_gemv/s_gemv_q8k/s2_gemv_q8/
s2_expert_grouped/shared_expert/quantize_act/bf16_gemv/elementwise/mmvq_multi/
iq_multi/native_expert parity suites all pass unchanged.

2026-10-14 07:05:13 +02:00

..

cuda_callback_affinity.cpp

generate: warm CUDA callbacks before host-core pinning (opt-in)

2026-10-11 20:59:23 +02:00

fusions_parity.cu

V100: operator fusions for the decode GEMV / MoE paths, bitwise

2026-10-14 07:05:13 +02:00