sycl: strata-sycl.sh keeps the engine's cache under the data root - the int8 crossover was re-measured at every start

STRATA_PREFILL_MMQ=auto (the Battlemage default) times the int8 and FP16 expert chains at the first prompt of
each expert format and caches the result in ~/.cache/strata/i8-crossover.txt. strata-sycl.sh runs the engine in
a `--rm` container, so that home - and the cache - is gone after every run: each server start paid the
measurement again inside its first prompt. Arc Pro B70, Coder IQ1_M (six format pairs), 2,185-token prompt:
761 tok/s on the first run without the cache against 1,450 once it is read back (int8 forced: 1,465-1,469); the
same choice is made either way (500 of 500 groups int8). The wrapper now sets STRATA_CACHE_DIR=/work/.strata-cache
(the mounted data root) unless the environment names another.
This commit is contained in:
maxfridbe committed 2026-10-19 13:13:15 -05:00
1 parent 8cc654f238
commit c44fb63bec
2 files changed
+7 -2

No files matched your search

+2 -1
View File
@@ -19,7 +19,8 @@ never exists and the weights are read once, compressed (2.1 MB per expert instea
At the first layer of each expert format the engine times both chains on synthetic experts (random blobs of the layer's format and
shape, uneven row counts like a real group) at 16, 32, 64, 128 and 256 mean rows per expert, interpolates where the FP16 chain
becomes the faster one, flags the result "clear" when the cost ratio falls steadily and the bracketing samples are 5% away from
1, and keeps it in `~/.cache/strata/i8-crossover.txt` (`STRATA_CACHE_DIR` moves it) keyed by device name, driver version, the two
1, and keeps it in `~/.cache/strata/i8-crossover.txt` (`STRATA_CACHE_DIR` moves it; `sycl/serve/strata-sycl.sh` sets it to `.strata-cache` in the data root it mounts, since the
container's home is gone after each run) keyed by device name, driver version, the two
ggml types and the shape. A matching key is read back instead of measured. The measurement costs ~1.2 s once, during the first
prompt. A measurement that fails (no VRAM) falls back to 64 rows.
* `STRATA_I8_FORCE=0` hides the device from the kernels, `STRATA_I8_MT=2|4|8` forces the tile height, `STRATA_I8_WG=0` the
+5 -1
View File
@@ -24,7 +24,11 @@ for a in "$@"; do args+=" $(printf '%q' "$a")"; done
# the port's run-time switches: the device-built verify plan, and no host handshakes in it. NO_HOST is for a card on
# the xe driver (the host<->GPU flag stores are not visible there); an i915 card (Arc A-series) runs without it
# (docs/INTEL.md)
declare -A setting=([STRATA_VERIFY_DEVICE_PLAN]=1 [STRATA_VERIFY_NO_HOST]=1 [STRATA_STAGER_THREADS]=12)
# STRATA_CACHE_DIR: the engine's measurements that outlive a run (the int8 prompt experts' crossover, measured at
# the first prompt of each format: ~1.4 s of a B70's first 2K prompt) - under the mounted data root, not in the
# throwaway container's home
declare -A setting=([STRATA_VERIFY_DEVICE_PLAN]=1 [STRATA_VERIFY_NO_HOST]=1 [STRATA_STAGER_THREADS]=12
[STRATA_CACHE_DIR]=/work/.strata-cache)
while IFS= read -r k; do
case "$k" in
STRATA_SYCL_*) ;; # this script's own settings