gfx1151 Windows: keep the 16,384-token prompt chunk; do not step it down by the unified pool's "free" figure (STRATA_PF_OWN_CHECK=1 restores the check)

On a Strix Halo the prompt path keeps its own buffers at the 16,384-token chunk (prefill_own_keep_chunk) and the cache sizing already reserved them. The
step-down before the prompt path init read cudaMemGetInfo after the pinned RAM complement (which the unified pool counts twice): 860 MiB "free" in the MTP
engine (no chunk fits, the chunk stays, prompts read at 955 tok/s) but 1.7 GiB in the DFlash2-alone engine (no MTP layer, a slightly larger cache), where
1,536 fit: chunk cut to 1,536, prompts at 303 tok/s (UD-Q4_K_XL, 30K prompt 98.9 s against 31.4 s; 8K 28.3 s against 10.8 s). A real allocation failure still
steps the chunk down (the init loop). Only reached when prefill_own_keep_chunk is set: Windows HIP gfx1151 with a bare --prefill auto.
This commit is contained in:
Niko1221 committed 2026-11-02 11:16:47 +01:00
1 parent c0380157df
commit b1d9ebb4c2
1 file changed
+14
+14
View File
@@ -10526,7 +10526,20 @@ int main(int argc, char** argv) {
};
int dev = 0;
int64_t need = 0, fb = 0;
// gfx1151 (Strix Halo, Windows): `prefill_own_keep_chunk` is set above because the prompt path keeps its own buffers at the 16,384-token
// chunk and the cache sizing already reserved them. The "free VRAM" read here is the unified pool's figure after the pinned RAM complement
// (which the pool counts twice), not a limit: with 860 MiB "free" the MTP engine allocates the 6.7 GiB of buffers anyway and reads prompts at
// 955 tok/s, but this step-down found 1.7 GiB "free" in the DFlash2-alone engine (no MTP layer, a slightly larger cache), cut the chunk to
// 1,536 and read them at 303 tok/s (RESULTS-DF2STRIX.md). The step-down stays for a real allocation failure (the loop below).
// STRATA_PF_OWN_CHECK=1 (tester) restores the arithmetic check on this card.
static const bool own_check_restored = [] { const char* v = std::getenv("STRATA_PF_OWN_CHECK"); return v != nullptr && v[0] == '1'; }();
const bool keep_chunk_uma = o.prefill_own_keep_chunk && !own_check_restored;
if (!own_fits(o.prefill_chunk, dev, need, fb)) {
if (keep_chunk_uma) {
std::fprintf(stderr, "strata serve: a %lld-token chunk's prompt buffers need %lld MiB, %lld MiB are reported free: kept (this GPU's memory is the "
"unified pool; STRATA_PF_OWN_CHECK=1 steps the chunk down by that figure)\n",
(long long) o.prefill_chunk, (long long) (need >> 20), (long long) (fb >> 20));
} else {
int64_t c = 0;
for (const int64_t s : kStepChunks) {
int d2 = 0;
@@ -10539,6 +10552,7 @@ int main(int argc, char** argv) {
(long long) (need >> 20), dev, (long long) (fb >> 20), (long long) c);
o.prefill_chunk = c;
}
}
}
}
mem_mark("before the prompt path init");