mirror of
https://github.com/Niko1221/Strata.git
synced 2026-11-07 18:35:04 -10:00
docs: the 12.0 defaults with the final A/B (decode +14-16%, prompts +3-10%), PF_FUSED quality, the ple-io rule
This commit is contained in:
1 parent
b25b309703
commit
b829a3189d
1 file changed
+21
-11
+21
-11
@@ -2217,32 +2217,42 @@ signal, early cleanup and `--errors` error-propagation checks. Its `--synchronou
|
||||
mode is the negative control under memcheck: an eight-second watchdog releases
|
||||
the original launch/service deadlock and the test fails.
|
||||
|
||||
## Decode on compute capability 12.0 cards: four bitwise switches on by default, interleaved matvec up to 8 tokens
|
||||
## Compute capability 12.0 cards: faster decode (bitwise) and fused prompt experts by default
|
||||
|
||||
Measured on an RTX PRO 5000 Blackwell 72 GB (110 SMs, sm_120, Linux, CUDA 13.2), IQ3_S with every expert in VRAM, setup's
|
||||
flags (`--spec 4 --spec-min-p 0.5`), 6 interleaved pairs of fresh engines, 300-token greedy answers, medians:
|
||||
|
||||
| | story | code | 8K document | 4K / 20K / 98K prompt |
|
||||
| | story | code | 8K document | 4K / 20K / 98K prompt (tok/s) |
|
||||
|---|---|---|---|---|
|
||||
| before (tok/s) | 168.8 | 224.5 | 186.8 | 3827 / 4607 / 4767 |
|
||||
| now | 189.0 (+11.9%, 6/6 pairs) | 256.2 (+14.0%, 6/6) | 211.2 (+13.1%, 6/6) | unchanged |
|
||||
| before (tok/s) | 168.8 | 224.6 | 186.6 | 3829 / 4613 / 4767 |
|
||||
| now | 192.0 (+13.8%, 6/6 pairs) | 260.0 (+15.8%, 6/6) | 211.9 (+13.6%, 6/6) | 4212 / 4870 / 4929 (+9.9% / +5.4% / +3.4%, 6/6 each) |
|
||||
|
||||
The answers are the same tokens (every case and pair), and the identity gate passes (10 prompts plus the 4K and 20K prompt
|
||||
dumps, with the fixed small cache and with every expert resident). What changed:
|
||||
Decode is bitwise: with `STRATA_PF_FUSED=0` the answers are the same tokens as before and the identity gate passes (10 prompts
|
||||
plus the 4K and 20K prompt dumps, with the fixed small cache and with every expert resident). The prompts change at rounding
|
||||
level because of the fused prompt experts (below). What changed:
|
||||
|
||||
- On a 12.0 card these existing switches are on unless set: `STRATA_SIDE_MIXER=2` (the mixer's input-only work on graph
|
||||
- On a 12.0 card these bitwise switches are on unless set: `STRATA_SIDE_MIXER=2` (the mixer's input-only work on graph
|
||||
branches), `STRATA_ATTN_DAILY=1` (the Daily's decode attention), `STRATA_GR_UP75=1` and `STRATA_GR_DOWN_TG=1` (the
|
||||
hyper-connection read's up projection with its loads hoisted, and its down projection with a block row per token). Each is bitwise equal
|
||||
to the kernel it replaces. A fixed 4-token verify round went 13.70 -> 12.23 ms with the four together.
|
||||
`STRATA_SM120_PERF=0` turns the set off; each switch's own variable still wins in both directions.
|
||||
hyper-connection read's up projection with its loads hoisted, and its down projection with a block row per token),
|
||||
`STRATA_GU_SWIGLU=1` (new: the routed experts' gate/up, SwiGLU and q8_1 in one launch). A fixed 4-token verify round went
|
||||
13.70 -> 11.80 ms with them. `STRATA_SM120_PERF=0` turns the set off; each switch's own variable still wins in both directions.
|
||||
- The interleaved small-batch matvec (`native_mmvq_il`, docs/MMVQ_IL_TABLE.md) also runs 5 to 8 tokens (it stopped at 4, and a
|
||||
5-token window cost 2 ms more than a 4-token one), with a 12.0 table for 5-8 tokens and, on 12.0 cards with 80+ SMs, its own
|
||||
2-4 token table. Bitwise (`mmvq_il_parity`, 1-8 tokens). `STRATA_MMVQ_IL_WIDE=0` / `STRATA_MMVQ_IL_120=0` turn them off.
|
||||
- The native packs' prompt experts run on the fused int8 kernels (`STRATA_PF_FUSED`, the rule gfx12 has) for IQ2_S, IQ3_XXS,
|
||||
IQ3_S and IQ4_XS gate/up; IQ2_XXS / IQ2_XS keep MMQ until measured. Quality on IQ3_S: teacher-forced KL against MMQ 0.0002
|
||||
(5998 positions; MMQ against itself 0.0000) and 0.0010 after ~6K-token documents (1760 positions); first-token KL against
|
||||
the FP16 prompt path 0.01181 vs MMQ's 0.01236 (48 prompts of 1.5K-24K tokens); HumanEval 156/164, reasoning 26/30, agent
|
||||
6/6, needles 6/6, CJK 8/8, tool calls 4/5, the same as MMQ. `STRATA_PF_FUSED=0` keeps MMQ.
|
||||
- With the cheaper rows, deeper speculation was measured again (3 pairs): `--spec 3` 1.145 / 1.020 / 1.123 of before,
|
||||
`--spec 4` 1.122 / 1.141 / 1.132, `--spec 5` 1.079 / 1.101 / 1.074, `--spec 6` 1.060 / 1.177 / 1.097, `--spec 8` 1.027 /
|
||||
1.105 / 1.069 (story / code / document). `--spec 4 --spec-min-p 0.5` stays the default.
|
||||
- Setup (Linux, not WSL): when the RAM covers the model's need + the n-gram table + 16 GB (Q2_0 93 GB, IQ3_S 107 GB), the
|
||||
table is kept in RAM (`--ple-io ram`), so a new prompt does not wait for its rows from the SSD: on this card 4K -18%, 20K
|
||||
-9%, 98K -2% with a fresh engine per prompt, the same bits. On an R9700 the read was already hidden; with RAM short (a 48 GB
|
||||
cap) ram mode cost +20% prompt and +14 s of start-up, which is why smaller PCs keep the SSD reads.
|
||||
|
||||
Not measured: other 12.0 cards (an RTX 5070 has not been run with these defaults), Windows, and partly-resident caches.
|
||||
Not measured: other 12.0 cards (RTX 5070), Windows, partly-resident caches, the IQ3_XXS and IQ2_XS packs.
|
||||
|
||||
## CUDA opt-in switches measured for 0.1.45 (RTX 3060 and RTX A4000, sm_86; all off unless you set them)
|
||||
|
||||
|
||||
Reference in new issue
Block a user