mirror of
https://github.com/Niko1221/Strata.git
synced 2026-11-07 14:21:06 -10:00
[Engine] SYCL: the prompt attention's query batch is 128 by default
Problem: `attn_batch` was 32, a value chosen for the shuffle-reduction score phase's vector-matrix kernel. The lane-cell phase is now the prompt default, and it wants more queries per launch. Result, same build, back to back, 3 warm repetitions each (2x Arc Pro B65, Qwen3.8-Flash-Next IQ3_S, 2,048-token chunks): | prompt | batch 32 | batch 128 | gain | | ---: | ---: | ---: | ---: | | 4,106 tokens | 1,227.9-1,235.1 tok/s | 1,240.9-1,247.6 | +0.9% | | 16,380 tokens | 1,582.2-1,584.2 | 1,604.9-1,605.8 | +1.3% | Batch 512 is another 0.2% on the 16K prompt but takes four times the scratch (400 MB per stage instead of 100 MB), so 128 is the trade. The expert cache keeps its slots: 10,253 either way at 4K. Approach: the default of the `STRATA_ATTN_BATCH` switch (added earlier this branch) moves from 32 to 128; the variable still overrides it in 1..1024. Validation: 64 generated tokens are identical to the serial reference (`scripts/compare_tokens.py`, first difference: none). Not run: CUDA/HIP (untouched - the switch is the SYCL port's), a card small enough for the 100 MB of extra scratch to cost expert-cache slots. Limits: the gain is attention-bound and small; it is the tuning of the phase that landed in the previous commit, not a new path.
This commit is contained in:
1 parent
949f77c0a8
commit
09524eb479
1 file changed
+3
-3
@@ -736,9 +736,9 @@ struct Prefill::Impl {
|
||||
float* attn_scratch = nullptr;
|
||||
int64_t attn_batch = [] { // SYCL port: how many queries the prompt path hands the FP32 attention kernel at once
|
||||
const char* v = std::getenv("STRATA_ATTN_BATCH"); // (the tensor-core kernel is opt-in and slower here)
|
||||
const long long n = v ? std::atoll(v) : 32; // 32 was picked for a vector-matrix kernel; more queries
|
||||
return (n >= 1 && n <= 1024) ? (int64_t) n : 32LL; // per launch give it more to overlap. Scratch grows too.
|
||||
}(), cap = 0;
|
||||
const long long n = v ? std::atoll(v) : 128; // 32 was picked for a vector-matrix kernel; with the
|
||||
return (n >= 1 && n <= 1024) ? (int64_t) n : 128LL; // lane-cell score phase 128 reads a 4,106-token prompt
|
||||
}(), cap = 0; // 1.1% faster than 32 and a 16,380-token one 1.3%
|
||||
// MoE
|
||||
float *logits = nullptr, *w = nullptr, *GU = nullptr, *Dm = nullptr, *sgate = nullptr, *sup = nullptr,
|
||||
*shared = nullptr, *sg = nullptr;
|
||||
|
||||
Reference in new issue
Block a user