mirror of
https://github.com/Niko1221/Strata.git
synced 2026-11-07 18:35:04 -10:00
native experts: gate the AVX2 multi-token kernels on cpu_avx2_ok()
native_gu_rows() tested cpu_avx512_ok() before calling the AVX-512 kernel but did NOT test
cpu_avx2_ok() before calling the AVX2 one:
static const bool avx512 = cpu_avx512_ok() && getenv("STRATA_NO_IQ512") == nullptr;
static const bool avx2 = getenv("STRATA_NO_IQ256") == nullptr;
So on a CPU with AVX but no AVX2, a verify window holding two or more tokens took the iq256 path.
iq_avx2.cpp and kq_avx2.cpp are compiled /arch:AVX2 and use AVX2 (vpmovzx, vbroadcastss, vpmaskmov)
and FMA3 (vfmadd231ps), none of which exist before Haswell: the engine died with 0xC000001D.
The same asymmetry guarded native_down_rows()'s two multi-token kernels, which are also /arch:AVX2
translation units. All three now require cpu_avx2_ok().
This needs no AVX1 kernel. Strata already computes native IQ experts "through ggml-cpu", and
ggml-cpu has a vec_dot for every one of these types (ggml_vec_dot_iq2_xxs_q8_K, _iq2_s_q8_K and the
rest), compiled for whatever baseline the build selected. That path sits below both multi-token
kernels and loops over tokens itself, so it is correct for any nt - it was simply unreachable,
because the only way to skip the multi-token kernels was an environment variable. On this machine
the gate/up rows now fall through to it and report 8.98 distinct CPU experts per layer.
generate.cpp's startup floor moves from cpu_avx2_ok() to cpu_avx1_ok() to match, and its message
names the new floor.
Verified end to end on a Sandy Bridge-E (Xeon E5-2680: no AVX2, FMA3, F16C, BMI2 or AVX-512) with a
GTX 1080 Ti at sm_61 under STRATA_EXPERIMENTAL_SM60, native IQ2_XS pack, 33.02 GiB loaded:
prompt : 1 2 3
output : 4 5
decode 2 tokens in 770.3 ms -> 2.60 tok/s
prefill 2 tokens in 3392.3 ms -> 0.59 tok/s (time to first token 4652.0 ms)
ENGINE_EXIT=0
ctest -R avx1: 2/2 passed.
This commit is contained in:
1 parent
8854151cbe
commit
0083bb7a4c
2 files changed
+21
-6
No files matched your search
@@ -84,7 +84,14 @@ void native_gu_rows(const NativeFmt& f, const uint8_t* blob, const void* const*
|
||||
// AVX-2 kernel, STRATA_NO_IQ256 drops the AVX-2 kernel; ggml-cpu's single-token vec_dot is reached only with
|
||||
// both set (and on a CPU without AVX-512, STRATA_NO_IQ512 changes nothing).
|
||||
static const bool avx512 = cpu_avx512_ok() && std::getenv("STRATA_NO_IQ512") == nullptr;
|
||||
static const bool avx2 = std::getenv("STRATA_NO_IQ256") == nullptr;
|
||||
// STRATA_NO_IQ256 drops the AVX-2 kernel. cpu_avx2_ok() is ALSO required: iq_avx2.cpp and kq_avx2.cpp
|
||||
// are compiled /arch:AVX2 and use AVX2 and FMA3, so calling them on a CPU without either is an
|
||||
// illegal instruction, not a slow path. The gate sat next to `avx512` above, which does test
|
||||
// cpu_avx512_ok(), and that asymmetry is what let a pre-Haswell CPU reach iq256_gu_rows.
|
||||
// Without it this falls through to ggml-cpu's vec_dot below, which is compiled for whatever
|
||||
// baseline the build selected (AVX1 here) and covers the same types - IQ2_XXS and IQ2_S among
|
||||
// them. That path loops over tokens itself, so it is correct for any `nt`, not just one.
|
||||
static const bool avx2 = cpu_avx2_ok() && std::getenv("STRATA_NO_IQ256") == nullptr;
|
||||
// #152: from how many tokens the multi-token kernels run (ggml's vec_dot below that). The default 2 is the
|
||||
// measured-fastest rule, but a token's expert rows then round differently alone than in a group, so greedy output
|
||||
// can depend on how many drafts a verify window held. STRATA_IQ_MT_MIN=1 (opt-in, 0.1.30) uses the multi-token
|
||||
@@ -129,11 +136,14 @@ void native_down_rows(const NativeFmt& f, const uint8_t* blob, const void* const
|
||||
static const bool iq4nl_mt = std::getenv("STRATA_NO_IQ4NL") == nullptr;
|
||||
static const int mt_min = [] { const char* e = std::getenv("STRATA_IQ_MT_MIN"); return e ? std::atoi(e) : 2; }();
|
||||
static const bool kq = [] { const char* v = std::getenv("STRATA_KQ256"); return v != nullptr && std::atoi(v) != 0; }();
|
||||
if (kq && nt >= 2 && (f.d_type == 7 || f.d_type == 8)) { // Q5_1 / Q8_0 down: bit-exact, any group size
|
||||
// Both multi-token kernels below are /arch:AVX2 translation units (kq_avx2.cpp and iq_avx2.cpp),
|
||||
// so a CPU without AVX2 has to reach ggml-cpu's vec_dot instead - same reasoning as the gate/up
|
||||
// rows above, where `avx512` tested cpu_avx512_ok() and `avx2` did not.
|
||||
if (cpu_avx2_ok() && kq && nt >= 2 && (f.d_type == 7 || f.d_type == 8)) { // Q5_1 / Q8_0 down: bit-exact, any group size
|
||||
kq256_rows(f.d_type, blob + f.down_off, f.d_row, (int) f.n_ff, hq, nt, out, r0, r1);
|
||||
return;
|
||||
}
|
||||
if (nt >= mt_min && f.d_type == 20 && iq4nl_mt) { // #152: the same rule as the gate/up rows
|
||||
if (cpu_avx2_ok() && nt >= mt_min && f.d_type == 20 && iq4nl_mt) { // #152: the same rule as the gate/up rows
|
||||
iq4nl256_down_rows(blob + f.down_off, f.d_row, (int) f.n_ff, hq, nt, out, r0, r1);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1550,9 +1550,14 @@ int main(int argc, char** argv) {
|
||||
// ... and nothing runs on a CPU without AVX2: every CPU expert kernel is AVX2 at least (the AVX-512 ones are
|
||||
// chosen above it), and so is ggml-cpu in the release build, which the native pack's layout load initializes
|
||||
// next. Refused here, by name, rather than an illegal instruction in the first expert.
|
||||
if (!strata::kernels::cpu::cpu_avx2_ok()) {
|
||||
std::fprintf(stderr, "strata generate: this CPU (%s) does not support AVX2 with FMA and F16C, which every CPU "
|
||||
"expert kernel needs; Strata runs on Intel Haswell (2013), AMD Zen (2017) or newer\n",
|
||||
// A CPU with AVX but no AVX2 is also admissible. The multi-token kernels that do need AVX2 are now
|
||||
// gated on cpu_avx2_ok() (see native_expert.cpp) and fall through to ggml-cpu's vec_dot, which the
|
||||
// build compiles for whatever baseline it selected - so the floor is the AVX1 rung, and the message
|
||||
// below names it instead of naming a floor the dispatch no longer has.
|
||||
if (!strata::kernels::cpu::cpu_avx1_ok()) {
|
||||
std::fprintf(stderr, "strata generate: this CPU (%s) does not support AVX (256-bit float) with SSSE3 and "
|
||||
"SSE4.1, which every CPU expert kernel needs; Strata runs on Intel Sandy Bridge (2011), "
|
||||
"AMD Bulldozer (2011) or newer\n",
|
||||
strata::kernels::cpu::cpu_name().c_str());
|
||||
return 2;
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user