[Engine] SYCL: the prompt attention's query batch is 128 by default

Problem: `attn_batch` was 32, a value chosen for the shuffle-reduction score phase's vector-matrix kernel.  The
lane-cell phase is now the prompt default, and it wants more queries per launch.

Result, same build, back to back, 3 warm repetitions each (2x Arc Pro B65, Qwen3.8-Flash-Next IQ3_S, 2,048-token
chunks):

| prompt | batch 32 | batch 128 | gain |
| ---: | ---: | ---: | ---: |
| 4,106 tokens | 1,227.9-1,235.1 tok/s | 1,240.9-1,247.6 | +0.9% |
| 16,380 tokens | 1,582.2-1,584.2 | 1,604.9-1,605.8 | +1.3% |

Batch 512 is another 0.2% on the 16K prompt but takes four times the scratch (400 MB per stage instead of 100 MB),
so 128 is the trade.  The expert cache keeps its slots: 10,253 either way at 4K.

Approach: the default of the `STRATA_ATTN_BATCH` switch (added earlier this branch) moves from 32 to 128; the
variable still overrides it in 1..1024.

Validation: 64 generated tokens are identical to the serial reference (`scripts/compare_tokens.py`, first
difference: none).  Not run: CUDA/HIP (untouched - the switch is the SYCL port's), a card small enough for the
100 MB of extra scratch to cost expert-cache slots.

Limits: the gain is attention-bound and small; it is the tuning of the phase that landed in the previous commit, not
a new path.
This commit is contained in:
Niko1221 committed 2026-10-29 12:45:35 +01:00
1 parent 949f77c0a8
commit 09524eb479
1 file changed
+3 -3
+3 -3
View File
@@ -736,9 +736,9 @@ struct Prefill::Impl {
float* attn_scratch = nullptr;
int64_t attn_batch = [] { // SYCL port: how many queries the prompt path hands the FP32 attention kernel at once
const char* v = std::getenv("STRATA_ATTN_BATCH"); // (the tensor-core kernel is opt-in and slower here)
const long long n = v ? std::atoll(v) : 32; // 32 was picked for a vector-matrix kernel; more queries
return (n >= 1 && n <= 1024) ? (int64_t) n : 32LL; // per launch give it more to overlap. Scratch grows too.
}(), cap = 0;
const long long n = v ? std::atoll(v) : 128; // 32 was picked for a vector-matrix kernel; with the
return (n >= 1 && n <= 1024) ? (int64_t) n : 128LL; // lane-cell score phase 128 reads a 4,106-token prompt
}(), cap = 0; // 1.1% faster than 32 and a 16,380-token one 1.3%
// MoE
float *logits = nullptr, *w = nullptr, *GU = nullptr, *Dm = nullptr, *sgate = nullptr, *sup = nullptr,
*shared = nullptr, *sg = nullptr;