mirror of
https://github.com/Niko1221/Strata.git
synced 2026-11-07 22:59:00 -10:00
docs: --kv-grow on HIP with numbers, STRATA_PREFILL_HELP on two R9700
This commit is contained in:
1 parent
3980a5ce70
commit
d9070189c0
3 files changed
+13
-1
No files matched your search
+1
-1
@@ -434,7 +434,7 @@ an RX 9070 XT 16 GB and a Radeon AI PRO R9700 32 GB (both gfx1201), a Ryzen 9 39
|
||||
1,990 at 4,210 (+17.1%), medians 1,571 -> 2,064 and a 1.27x geometric mean over those three. decode
|
||||
89.8 -> 86.9 tok/s, unchanged. `STRATA_SELECT_WMMA=1` on top changed nothing at those lengths (2,026 tok/s);
|
||||
its +1.5% is a 16K/262K-ctx number. `hip_prompt_attn_wmma` passes on that card.
|
||||
- `STRATA_HIP_WMMA=auto` (f-151-amd): the same kernel, chosen per call: on a gfx12 card with int8 KV and a prompt batch of
|
||||
- `STRATA_HIP_WMMA=auto`: the same kernel, chosen per call: on a gfx12 card with int8 KV and a prompt batch of
|
||||
`STRATA_HIP_WMMA_MIN_Q` (64) or more queries it runs the matrix-core kernel, everything else the FP32 one (`=1` forces it
|
||||
on every call, `=0` or unset keeps the FP32 kernel; the default has not moved). A launch error switches the matrix-core
|
||||
kernel off for the rest of the process and the FP32 kernel runs (before, the engine exited). Kernel timing on an R9700
|
||||
|
||||
@@ -340,6 +340,14 @@ move on a Threadripper PRO 3975WX + R9700 (14.5 ms, +0.4%, 5 pairs); no case rea
|
||||
reach, and the expert cache holds the rest, giving slots back as the context grows. It needs one GPU, an expert
|
||||
profile, the whole K/V in VRAM (no KV streaming) and every expert in RAM (not the resident low-RAM mode); otherwise,
|
||||
and with `--batch`, `--vram-elastic` or `--peer-device`, the engine says so and stays off.
|
||||
On HIP (ROCm 6+) it runs through hipMemAddressReserve / hipMemCreate / hipMemMap (2 MiB chunks; the card must report
|
||||
virtual memory management, `STRATA_HIP_VMM=0` turns it off); until then every AMD card printed "--kv-grow is off".
|
||||
Radeon AI PRO R9700 with `--vram-reserve-mib 17084` (a 16 GB card), Q2_0 pack, `--max-context 196608`, 5 interleaved pairs:
|
||||
expert-cache slots 6,213 -> 7,968 and decode 96.3 -> 98.5 tok/s (x1.023, 5/5; the noise arm read x0.999); at
|
||||
`--max-context 262144` the default engine finds no VRAM left for the cache on that budget and does not start, `--kv-grow`
|
||||
starts with 7,877 slots (94.4 tok/s). Prompts of 20K and 60K tokens through the growing K/V read as fast as without it
|
||||
(x1.002, x1.001) and gave identical answers. At contexts up to 128K the K/V is small and nothing changes; it pays on cards of
|
||||
about 24 GB or less with contexts of 128K and more.
|
||||
- `--host-core last` (or `STRATA_HOST_CORE=last`, Windows): the host thread runs on the last physical core and the
|
||||
workers take the first. Windows sends a GPU's interrupts to the first core, where a host spinning on the GPU's flags
|
||||
waits for them (`--host-core first` is the default; the startup log names the cores). It leaves a hybrid CPU as it
|
||||
|
||||
@@ -117,6 +117,10 @@ helped (~360 MB at 3.3K tokens). The rows it computes round like a different MMQ
|
||||
bit-identical to the default's (it is repeatable: same prompt, same output), which is why it is **opt-in**:
|
||||
`STRATA_PREFILL_HELP=1` turns it on. Native packs on the MMQ prompt path only (not with the fused prompt kernels,
|
||||
`STRATA_PF_FUSED=1`, nor with `--peer-device`). With it on, `STRATA_PREFILL_HELP_FRAC=f` fixes the share.
|
||||
On two Radeon AI PRO R9700 (each limited to 16 GB with `--vram-reserve-mib 17084`, Q2_0 pack, layer split auto, 5 interleaved pairs,
|
||||
noise arm within 0.2%), `STRATA_PREFILL_HELP=1` gave prompt tok/s without -> with: 700 tokens 638 -> 672 (x1.053, 5/5), 1.5K 839 -> 809
|
||||
(x0.964), 3K 1,105 -> 929 (x0.840), 6K 1,284 -> 1,211 (x0.943): the PCIe 4.0 x16 links of this box (28.4 GB/s each) make the share lose
|
||||
from 1.5K tokens up, so it stays off (`STRATA_PREFILL_HELP_FRAC` was not tuned for it).
|
||||
|
||||
The engine flags behind it: `--layer-split K1[,K2..]|auto` and `--split-device D1[,D2..]` (the later stages'
|
||||
devices; default the next visible ones). `--layer-split K --split-device 0` runs both stages on one card sharing
|
||||
|
||||
Reference in new issue
Block a user