Niko1221/Strata
1
0 Fork 0
mirror of https://github.com/Niko1221/Strata.git synced 2026-11-07 14:21:06 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity

[Engine] layer-major prefill: expand the hyper-connection projections on the path's stream

Browse Source 
run_layer_major's three hyper-connection bf16_proj calls omitted the optional `stream`, so under
STRATA_HC_REQ8 bf16_proj expanded the int8 codes into its static scratch on the default stream while the
GEMMs read it on the prompt path's stream.  Pass the path's own stream at the down, up and inject calls, as
run_impl already does.

Seen on a single RTX 3070 Ti: with STRATA_PREFILL_LAYER_MAJOR=1 and STRATA_HC_REQ8=1 a needle-copy answer is
lost at every prompt length from 1,567 to 100,003 tokens, on the Q2_0 and IQ3_XXS packs, at any needle depth
and on both expert paths; exact with the switch off, and exact everywhere after this change.

Reported in #3976.
This commit is contained in:
fuoco authored and t committed 2026-10-30 10:28:37 -04:00
1 parent 46c152a131
commit bd14107015
1 file changed
+5 -3
Download Patch File Download Diff File Expand all files Collapse all files
src/prefill/prefill.cpp
+5 -3
View File
@@ -6774,14 +6774,16 @@ bool Prefill::run_layer_major(const int64_t* tokens, int64_t n, int64_t pos0, st
if (!wd_q8.data || !wu_q8.data) { err = "prefill: the Q8_0 hyper-connection projections could not be dequantized"; return false; }
wd = &wd_q8; wu = &wu_q8;
}
// the stream is not optional: under STRATA_HC_REQ8 bf16_proj expands the int8 codes into one static
// scratch per call site, on the stream it is given. Omitted, the GEMMs below read it out of order.
pt.mark(kPfHc, cs);
if (gr_unfused()) gr_norm(m.R, (const float*) wn->data, EPS, m.xn, m.xn16, T, cs, m.xn16_lo);
else if (!normed) gr_norm_rs(m.R, (const float*) wn->data, EPS, m.grs, m.xn16, T, cs, m.xn16_lo);
normed = false;
if (!bf16_proj(m.gemm, wd, m.xn16, m.lo, T, sd, err, 0, m.xn16_lo)) return false;
if (!bf16_proj(m.gemm, wd, m.xn16, m.lo, T, sd, err, 0, m.xn16_lo, 0, cs)) return false;
gr_silu(m.lo, m.lo16, T, cs, m.lo16_lo);
if (!bf16_proj(m.gemm, wu, m.lo16, m.gated, T, su, err, 0, m.lo16_lo)) return false;
if (!bf16_proj(m.gemm, wi, m.xn16, m.inj, T, si, err, 0, m.xn16_lo)) return false;
if (!bf16_proj(m.gemm, wu, m.lo16, m.gated, T, su, err, 0, m.lo16_lo, 0, cs)) return false;
if (!bf16_proj(m.gemm, wi, m.xn16, m.inj, T, si, err, 0, m.xn16_lo, 0, cs)) return false;
if (gr_unfused()) gr_mix(m.xn, m.gated, m.mixed, m.mixed_bf, T, cs, m.mixed_h, m.mixed_bf_lo);
else gr_mix_r(m.R, m.grs, (const float*) wn->data, m.gated, m.mixed, m.mixed_bf, T, cs, m.mixed_h,
m.mixed_bf_lo);