native experts: gate the AVX2 multi-token kernels on cpu_avx2_ok()

native_gu_rows() tested cpu_avx512_ok() before calling the AVX-512 kernel but did NOT test
cpu_avx2_ok() before calling the AVX2 one:

    static const bool avx512 = cpu_avx512_ok() && getenv("STRATA_NO_IQ512") == nullptr;
    static const bool avx2   = getenv("STRATA_NO_IQ256") == nullptr;

So on a CPU with AVX but no AVX2, a verify window holding two or more tokens took the iq256 path.
iq_avx2.cpp and kq_avx2.cpp are compiled /arch:AVX2 and use AVX2 (vpmovzx, vbroadcastss, vpmaskmov)
and FMA3 (vfmadd231ps), none of which exist before Haswell: the engine died with 0xC000001D.

The same asymmetry guarded native_down_rows()'s two multi-token kernels, which are also /arch:AVX2
translation units.  All three now require cpu_avx2_ok().

This needs no AVX1 kernel.  Strata already computes native IQ experts "through ggml-cpu", and
ggml-cpu has a vec_dot for every one of these types (ggml_vec_dot_iq2_xxs_q8_K, _iq2_s_q8_K and the
rest), compiled for whatever baseline the build selected.  That path sits below both multi-token
kernels and loops over tokens itself, so it is correct for any nt - it was simply unreachable,
because the only way to skip the multi-token kernels was an environment variable.  On this machine
the gate/up rows now fall through to it and report 8.98 distinct CPU experts per layer.

generate.cpp's startup floor moves from cpu_avx2_ok() to cpu_avx1_ok() to match, and its message
names the new floor.

Verified end to end on a Sandy Bridge-E (Xeon E5-2680: no AVX2, FMA3, F16C, BMI2 or AVX-512) with a
GTX 1080 Ti at sm_61 under STRATA_EXPERIMENTAL_SM60, native IQ2_XS pack, 33.02 GiB loaded:

    prompt : 1 2 3
    output : 4 5
    decode 2 tokens in 770.3 ms -> 2.60 tok/s
    prefill 2 tokens in 3392.3 ms -> 0.59 tok/s (time to first token 4652.0 ms)

    ENGINE_EXIT=0

ctest -R avx1: 2/2 passed.
This commit is contained in:
Demetree Kallergis committed 2026-10-01 13:25:10 -02:30
1 parent 8854151cbe
commit 0083bb7a4c
2 files changed
+21 -6

No files matched your search

+13 -3
View File
@@ -84,7 +84,14 @@ void native_gu_rows(const NativeFmt& f, const uint8_t* blob, const void* const*
// AVX-2 kernel, STRATA_NO_IQ256 drops the AVX-2 kernel; ggml-cpu's single-token vec_dot is reached only with
// both set (and on a CPU without AVX-512, STRATA_NO_IQ512 changes nothing).
static const bool avx512 = cpu_avx512_ok() && std::getenv("STRATA_NO_IQ512") == nullptr;
static const bool avx2 = std::getenv("STRATA_NO_IQ256") == nullptr;
// STRATA_NO_IQ256 drops the AVX-2 kernel. cpu_avx2_ok() is ALSO required: iq_avx2.cpp and kq_avx2.cpp
// are compiled /arch:AVX2 and use AVX2 and FMA3, so calling them on a CPU without either is an
// illegal instruction, not a slow path. The gate sat next to `avx512` above, which does test
// cpu_avx512_ok(), and that asymmetry is what let a pre-Haswell CPU reach iq256_gu_rows.
// Without it this falls through to ggml-cpu's vec_dot below, which is compiled for whatever
// baseline the build selected (AVX1 here) and covers the same types - IQ2_XXS and IQ2_S among
// them. That path loops over tokens itself, so it is correct for any `nt`, not just one.
static const bool avx2 = cpu_avx2_ok() && std::getenv("STRATA_NO_IQ256") == nullptr;
// #152: from how many tokens the multi-token kernels run (ggml's vec_dot below that). The default 2 is the
// measured-fastest rule, but a token's expert rows then round differently alone than in a group, so greedy output
// can depend on how many drafts a verify window held. STRATA_IQ_MT_MIN=1 (opt-in, 0.1.30) uses the multi-token
@@ -129,11 +136,14 @@ void native_down_rows(const NativeFmt& f, const uint8_t* blob, const void* const
static const bool iq4nl_mt = std::getenv("STRATA_NO_IQ4NL") == nullptr;
static const int mt_min = [] { const char* e = std::getenv("STRATA_IQ_MT_MIN"); return e ? std::atoi(e) : 2; }();
static const bool kq = [] { const char* v = std::getenv("STRATA_KQ256"); return v != nullptr && std::atoi(v) != 0; }();
if (kq && nt >= 2 && (f.d_type == 7 || f.d_type == 8)) { // Q5_1 / Q8_0 down: bit-exact, any group size
// Both multi-token kernels below are /arch:AVX2 translation units (kq_avx2.cpp and iq_avx2.cpp),
// so a CPU without AVX2 has to reach ggml-cpu's vec_dot instead - same reasoning as the gate/up
// rows above, where `avx512` tested cpu_avx512_ok() and `avx2` did not.
if (cpu_avx2_ok() && kq && nt >= 2 && (f.d_type == 7 || f.d_type == 8)) { // Q5_1 / Q8_0 down: bit-exact, any group size
kq256_rows(f.d_type, blob + f.down_off, f.d_row, (int) f.n_ff, hq, nt, out, r0, r1);
return;
}
if (nt >= mt_min && f.d_type == 20 && iq4nl_mt) { // #152: the same rule as the gate/up rows
if (cpu_avx2_ok() && nt >= mt_min && f.d_type == 20 && iq4nl_mt) { // #152: the same rule as the gate/up rows
iq4nl256_down_rows(blob + f.down_off, f.d_row, (int) f.n_ff, hq, nt, out, r0, r1);
return;
}
+8 -3
View File
@@ -1550,9 +1550,14 @@ int main(int argc, char** argv) {
// ... and nothing runs on a CPU without AVX2: every CPU expert kernel is AVX2 at least (the AVX-512 ones are
// chosen above it), and so is ggml-cpu in the release build, which the native pack's layout load initializes
// next. Refused here, by name, rather than an illegal instruction in the first expert.
if (!strata::kernels::cpu::cpu_avx2_ok()) {
std::fprintf(stderr, "strata generate: this CPU (%s) does not support AVX2 with FMA and F16C, which every CPU "
"expert kernel needs; Strata runs on Intel Haswell (2013), AMD Zen (2017) or newer\n",
// A CPU with AVX but no AVX2 is also admissible. The multi-token kernels that do need AVX2 are now
// gated on cpu_avx2_ok() (see native_expert.cpp) and fall through to ggml-cpu's vec_dot, which the
// build compiles for whatever baseline it selected - so the floor is the AVX1 rung, and the message
// below names it instead of naming a floor the dispatch no longer has.
if (!strata::kernels::cpu::cpu_avx1_ok()) {
std::fprintf(stderr, "strata generate: this CPU (%s) does not support AVX (256-bit float) with SSSE3 and "
"SSE4.1, which every CPU expert kernel needs; Strata runs on Intel Sandy Bridge (2011), "
"AMD Bulldozer (2011) or newer\n",
strata::kernels::cpu::cpu_name().c_str());
return 2;
}