docs: --kv-grow on HIP with numbers, STRATA_PREFILL_HELP on two R9700

This commit is contained in:
Niko1221 committed 2026-10-18 08:32:28 +02:00
1 parent 3980a5ce70
commit d9070189c0
3 files changed
+13 -1

No files matched your search

+1 -1
View File
@@ -434,7 +434,7 @@ an RX 9070 XT 16 GB and a Radeon AI PRO R9700 32 GB (both gfx1201), a Ryzen 9 39
1,990 at 4,210 (+17.1%), medians 1,571 -> 2,064 and a 1.27x geometric mean over those three. decode
89.8 -> 86.9 tok/s, unchanged. `STRATA_SELECT_WMMA=1` on top changed nothing at those lengths (2,026 tok/s);
its +1.5% is a 16K/262K-ctx number. `hip_prompt_attn_wmma` passes on that card.
- `STRATA_HIP_WMMA=auto` (f-151-amd): the same kernel, chosen per call: on a gfx12 card with int8 KV and a prompt batch of
- `STRATA_HIP_WMMA=auto`: the same kernel, chosen per call: on a gfx12 card with int8 KV and a prompt batch of
`STRATA_HIP_WMMA_MIN_Q` (64) or more queries it runs the matrix-core kernel, everything else the FP32 one (`=1` forces it
on every call, `=0` or unset keeps the FP32 kernel; the default has not moved). A launch error switches the matrix-core
kernel off for the rest of the process and the FP32 kernel runs (before, the engine exited). Kernel timing on an R9700
+8
View File
@@ -340,6 +340,14 @@ move on a Threadripper PRO 3975WX + R9700 (14.5 ms, +0.4%, 5 pairs); no case rea
reach, and the expert cache holds the rest, giving slots back as the context grows. It needs one GPU, an expert
profile, the whole K/V in VRAM (no KV streaming) and every expert in RAM (not the resident low-RAM mode); otherwise,
and with `--batch`, `--vram-elastic` or `--peer-device`, the engine says so and stays off.
On HIP (ROCm 6+) it runs through hipMemAddressReserve / hipMemCreate / hipMemMap (2 MiB chunks; the card must report
virtual memory management, `STRATA_HIP_VMM=0` turns it off); until then every AMD card printed "--kv-grow is off".
Radeon AI PRO R9700 with `--vram-reserve-mib 17084` (a 16 GB card), Q2_0 pack, `--max-context 196608`, 5 interleaved pairs:
expert-cache slots 6,213 -> 7,968 and decode 96.3 -> 98.5 tok/s (x1.023, 5/5; the noise arm read x0.999); at
`--max-context 262144` the default engine finds no VRAM left for the cache on that budget and does not start, `--kv-grow`
starts with 7,877 slots (94.4 tok/s). Prompts of 20K and 60K tokens through the growing K/V read as fast as without it
(x1.002, x1.001) and gave identical answers. At contexts up to 128K the K/V is small and nothing changes; it pays on cards of
about 24 GB or less with contexts of 128K and more.
- `--host-core last` (or `STRATA_HOST_CORE=last`, Windows): the host thread runs on the last physical core and the
workers take the first. Windows sends a GPU's interrupts to the first core, where a host spinning on the GPU's flags
waits for them (`--host-core first` is the default; the startup log names the cores). It leaves a hybrid CPU as it
+4
View File
@@ -117,6 +117,10 @@ helped (~360 MB at 3.3K tokens). The rows it computes round like a different MMQ
bit-identical to the default's (it is repeatable: same prompt, same output), which is why it is **opt-in**:
`STRATA_PREFILL_HELP=1` turns it on. Native packs on the MMQ prompt path only (not with the fused prompt kernels,
`STRATA_PF_FUSED=1`, nor with `--peer-device`). With it on, `STRATA_PREFILL_HELP_FRAC=f` fixes the share.
On two Radeon AI PRO R9700 (each limited to 16 GB with `--vram-reserve-mib 17084`, Q2_0 pack, layer split auto, 5 interleaved pairs,
noise arm within 0.2%), `STRATA_PREFILL_HELP=1` gave prompt tok/s without -> with: 700 tokens 638 -> 672 (x1.053, 5/5), 1.5K 839 -> 809
(x0.964), 3K 1,105 -> 929 (x0.840), 6K 1,284 -> 1,211 (x0.943): the PCIe 4.0 x16 links of this box (28.4 GB/s each) make the share lose
from 1.5K tokens up, so it stays off (`STRATA_PREFILL_HELP_FRAC` was not tuned for it).
The engine flags behind it: `--layer-split K1[,K2..]|auto` and `--split-device D1[,D2..]` (the later stages'
devices; default the next visible ones). `--layer-split K --split-device 0` runs both stages on one card sharing