mirror of
https://github.com/Niko1221/Strata.git
synced 2026-11-08 08:12:16 -10:00
gfx1151 Windows: keep the 16,384-token prompt chunk; do not step it down by the unified pool's "free" figure (STRATA_PF_OWN_CHECK=1 restores the check)
On a Strix Halo the prompt path keeps its own buffers at the 16,384-token chunk (prefill_own_keep_chunk) and the cache sizing already reserved them. The step-down before the prompt path init read cudaMemGetInfo after the pinned RAM complement (which the unified pool counts twice): 860 MiB "free" in the MTP engine (no chunk fits, the chunk stays, prompts read at 955 tok/s) but 1.7 GiB in the DFlash2-alone engine (no MTP layer, a slightly larger cache), where 1,536 fit: chunk cut to 1,536, prompts at 303 tok/s (UD-Q4_K_XL, 30K prompt 98.9 s against 31.4 s; 8K 28.3 s against 10.8 s). A real allocation failure still steps the chunk down (the init loop). Only reached when prefill_own_keep_chunk is set: Windows HIP gfx1151 with a bare --prefill auto.
This commit is contained in:
1 parent
c0380157df
commit
b1d9ebb4c2
1 file changed
+14
@@ -10526,7 +10526,20 @@ int main(int argc, char** argv) {
|
||||
};
|
||||
int dev = 0;
|
||||
int64_t need = 0, fb = 0;
|
||||
// gfx1151 (Strix Halo, Windows): `prefill_own_keep_chunk` is set above because the prompt path keeps its own buffers at the 16,384-token
|
||||
// chunk and the cache sizing already reserved them. The "free VRAM" read here is the unified pool's figure after the pinned RAM complement
|
||||
// (which the pool counts twice), not a limit: with 860 MiB "free" the MTP engine allocates the 6.7 GiB of buffers anyway and reads prompts at
|
||||
// 955 tok/s, but this step-down found 1.7 GiB "free" in the DFlash2-alone engine (no MTP layer, a slightly larger cache), cut the chunk to
|
||||
// 1,536 and read them at 303 tok/s (RESULTS-DF2STRIX.md). The step-down stays for a real allocation failure (the loop below).
|
||||
// STRATA_PF_OWN_CHECK=1 (tester) restores the arithmetic check on this card.
|
||||
static const bool own_check_restored = [] { const char* v = std::getenv("STRATA_PF_OWN_CHECK"); return v != nullptr && v[0] == '1'; }();
|
||||
const bool keep_chunk_uma = o.prefill_own_keep_chunk && !own_check_restored;
|
||||
if (!own_fits(o.prefill_chunk, dev, need, fb)) {
|
||||
if (keep_chunk_uma) {
|
||||
std::fprintf(stderr, "strata serve: a %lld-token chunk's prompt buffers need %lld MiB, %lld MiB are reported free: kept (this GPU's memory is the "
|
||||
"unified pool; STRATA_PF_OWN_CHECK=1 steps the chunk down by that figure)\n",
|
||||
(long long) o.prefill_chunk, (long long) (need >> 20), (long long) (fb >> 20));
|
||||
} else {
|
||||
int64_t c = 0;
|
||||
for (const int64_t s : kStepChunks) {
|
||||
int d2 = 0;
|
||||
@@ -10539,6 +10552,7 @@ int main(int argc, char** argv) {
|
||||
(long long) (need >> 20), dev, (long long) (fb >> 20), (long long) c);
|
||||
o.prefill_chunk = c;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
mem_mark("before the prompt path init");
|
||||
|
||||
Reference in new issue
Block a user