sergiywith fcef1b0dbd strata-vision: --flash-attn on|off|auto; on the CPU no CUDA context, no warm-up, a thread per core ...

- --flash-attn on|off|auto (mtmd's flash_attn_type): flash attention keeps K and V in FP16. On the NVFP4 fork the
  vision tower's output was 11.2% off with it on the GPU; FP32 attention (CPU) 0.1%.
- On the CPU a CUDA build keeps the GPU unseen (CUDA_VISIBLE_DEVICES=-1 before the runtime starts): it otherwise
  opened a context there, 0.4-0.7 GB of VRAM the expert cache loses.
- On the CPU without --threads: one thread per core (mtmd's default is 4); no warm-up encode (~6 s of start).
- serve/server.py: relative exe/mmproj/model paths in the vision config are the config's cwd's, as the engine's are.

2026-10-01 08:12:47 +02:00

..

CMakeLists.txt

Strata: Qwen3.8-Flash-Next on a 12-24 GB NVIDIA GPU + 64 GB RAM, one-click install

2026-09-24 18:42:20 +02:00

strata_vision.cpp

strata-vision: --flash-attn on|off|auto; on the CPU no CUDA context, no warm-up, a thread per core

2026-10-01 08:12:47 +02:00