mirror of
https://github.com/Niko1221/Strata.git
synced 2026-11-07 14:21:06 -10:00
sycl: strata-sycl.sh keeps the engine's cache under the data root - the int8 crossover was re-measured at every start
STRATA_PREFILL_MMQ=auto (the Battlemage default) times the int8 and FP16 expert chains at the first prompt of each expert format and caches the result in ~/.cache/strata/i8-crossover.txt. strata-sycl.sh runs the engine in a `--rm` container, so that home - and the cache - is gone after every run: each server start paid the measurement again inside its first prompt. Arc Pro B70, Coder IQ1_M (six format pairs), 2,185-token prompt: 761 tok/s on the first run without the cache against 1,450 once it is read back (int8 forced: 1,465-1,469); the same choice is made either way (500 of 500 groups int8). The wrapper now sets STRATA_CACHE_DIR=/work/.strata-cache (the mounted data root) unless the environment names another.
This commit is contained in:
1 parent
8cc654f238
commit
c44fb63bec
2 files changed
+7
-2
No files matched your search
@@ -19,7 +19,8 @@ never exists and the weights are read once, compressed (2.1 MB per expert instea
|
||||
At the first layer of each expert format the engine times both chains on synthetic experts (random blobs of the layer's format and
|
||||
shape, uneven row counts like a real group) at 16, 32, 64, 128 and 256 mean rows per expert, interpolates where the FP16 chain
|
||||
becomes the faster one, flags the result "clear" when the cost ratio falls steadily and the bracketing samples are 5% away from
|
||||
1, and keeps it in `~/.cache/strata/i8-crossover.txt` (`STRATA_CACHE_DIR` moves it) keyed by device name, driver version, the two
|
||||
1, and keeps it in `~/.cache/strata/i8-crossover.txt` (`STRATA_CACHE_DIR` moves it; `sycl/serve/strata-sycl.sh` sets it to `.strata-cache` in the data root it mounts, since the
|
||||
container's home is gone after each run) keyed by device name, driver version, the two
|
||||
ggml types and the shape. A matching key is read back instead of measured. The measurement costs ~1.2 s once, during the first
|
||||
prompt. A measurement that fails (no VRAM) falls back to 64 rows.
|
||||
* `STRATA_I8_FORCE=0` hides the device from the kernels, `STRATA_I8_MT=2|4|8` forces the tile height, `STRATA_I8_WG=0` the
|
||||
|
||||
@@ -24,7 +24,11 @@ for a in "$@"; do args+=" $(printf '%q' "$a")"; done
|
||||
# the port's run-time switches: the device-built verify plan, and no host handshakes in it. NO_HOST is for a card on
|
||||
# the xe driver (the host<->GPU flag stores are not visible there); an i915 card (Arc A-series) runs without it
|
||||
# (docs/INTEL.md)
|
||||
declare -A setting=([STRATA_VERIFY_DEVICE_PLAN]=1 [STRATA_VERIFY_NO_HOST]=1 [STRATA_STAGER_THREADS]=12)
|
||||
# STRATA_CACHE_DIR: the engine's measurements that outlive a run (the int8 prompt experts' crossover, measured at
|
||||
# the first prompt of each format: ~1.4 s of a B70's first 2K prompt) - under the mounted data root, not in the
|
||||
# throwaway container's home
|
||||
declare -A setting=([STRATA_VERIFY_DEVICE_PLAN]=1 [STRATA_VERIFY_NO_HOST]=1 [STRATA_STAGER_THREADS]=12
|
||||
[STRATA_CACHE_DIR]=/work/.strata-cache)
|
||||
while IFS= read -r k; do
|
||||
case "$k" in
|
||||
STRATA_SYCL_*) ;; # this script's own settings
|
||||
|
||||
Reference in new issue
Block a user