With an engine that keeps --resident-experts on a layer split, setup no longer
steers a low-RAM PC with two cards to one of them:
- at install, the low-RAM mode on several GPUs takes the resident variant by
the same rule as on one card (what no card holds fits the RAM with
LOW_RAM_HEADROOM_GB beside it), without the one-card question;
- at a start, a resident config on several GPUs keeps --resident-experts
(split_mmap leaves it alone);
- a resident config started on a PC with two cards is offered both, "y" by
default, as any other config.
It is gated on MIN_ENGINE >= RESIDENT_SPLIT_ENGINE (0.1.40 here, a guess at
the release that would carry the engine change), so until then setup behaves
as before and the golden configs are unchanged. Two tests run the new paths
with MIN_ENGINE patched; tools/test_setup_*.py pass (260 tests).
With the image encoder on the GPU, the 700 MiB reserve can leave ~200 MiB free on a 12 GB card (UD-IQ4_XS
end-to-end check on an RTX 5070). Setup prints a one-line recommendation; the written config is unchanged.
- "experimental" is per model now: MODELS["UD-Q4_K_XL"] keeps it, the unsloth family and UD-IQ4_XS drop it. The
first menu lists the family without [experimental], with a one-line about (UD-IQ4_XS: a 94 GB download; under
~80 GB of RAM part of its experts are read from the SSD). Its size menu lists UD-IQ4_XS first (the default) and
UD-Q4_K_XL second; an experimental size always sorts last. --model names are unchanged.
- UD-IQ4_XS: no EXPERIMENTAL warning, --check verdict without it. Images are asked (off by default), per model: the
engine's image path has no restriction for this pack, which uses the original model's image encoder; not yet run
with images. UD-Q4_K_XL keeps images off.
- MCP: the fallback tables mirror setup's (per-model experimental/vision), sizes in setup's order, per-size
"experimental" and "images", the vision check per model.
- Docs: README and its six translations, MODELS, UNSLOTH_Q4, AI_SETUP, MCP_SERVER.
- Tests: the menus, the default, --model by name, images, --check, the pack/tokenizer paths; the golden configs
are unchanged.
Added where a config goes to several GPUs (setup --gpus / a multi-GPU install, start --gpus, "use both" at
start). Single-GPU configs are not touched. Opt-out: setup --no-remote-expert-opt, or "remote_expert_opt": false
in the config (a user key, kept on re-runs). The engine uses it only with a helper cache (--expert-cache-device1..3),
so a layer split runs as before. tools/test_setup_remote_opt.py; the 2x16 GB golden configs gain the flag.
On a card whose experts mostly run on the CPU the slots cost speed per request (RTX 5070, Q2_0: 11-24 %), so setup
now recommends slots only when the expert cache (every card of a split counted) still holds half of the model's
experts beside them (Q2_0: 24 GB and up, or a split); elsewhere it stays at one at a time and says "parallel N
reduces waiting for several users but costs about 10-25% speed per request on this card". --parallel N is
honoured as asked, as before. BATCHING.md: the table with the change against one at a time.
The slots' sessions may take a fifth of the expert cache (~VRAM less 5 GB), up to 4: 2 on 12 GB, 3 on 16 GB, 4
from 24 GB, none below. Without --parallel setup only says it once; with it the config gets "parallel": N and a
count above the recommendation is kept with a note.
CUDA 13 dropped Pascal and Volta. A model whose GPUs include one (its oldest card decides: one engine runs per model)
now runs the experimental CUDA 12 engine in engine-cuda12/ - the ready-made strata-windows-x64-cuda12.zip with NVIDIA's
CUDA 12 pip libraries, or compiled here with a CUDA 12.x toolkit (Linux; STRATA_NVCC picks one, #601) - and setup says
why. engine/ and every model on it stay as they were.
- the cards are an opt-in: naming one (--gpu / --gpus), --cuda 12, STRATA_EXPERIMENTAL_SM60=1, or a PC with no newer
NVIDIA card and no supported AMD card; a PC with a newer card keeps recommending it
- --cuda 12|13 (or STRATA_CUDA) overrides the choice and is kept, with a warning where it cannot work (recommend,
never force); --cuda 12 also runs with a driver older than 580 (525+), which the stop now mentions
- at a start, a Pascal / Volta card added to a CUDA 13 model moves that model's config to the CUDA 12 engine
- updates look at both engine folders; a compiled CUDA 12 engine builds in build-cuda12/
- an RTX 50 card in a CUDA 12 engine is a warning (CUDA 12.8+ builds it; #220 crashed with 12.8), no longer a stop
The setup golden test (every recommended config on five PCs) is unchanged; tools/test_setup_older_gpus.py covers the rest.
- CMake: STRATA_ISA_FLOOR (empty by default) builds ggml-cpu for AVX (Sandy/Ivy Bridge) or SSE4.2
(Nehalem/Westmere) instead of the build host or AVX2; every other ggml ISA option off. Strata's own
AVX2/AVX-512 kernels stay in the binary behind their CPU checks. Excludes STRATA_PORTABLE. The ggml
block follows the Strata_Dirigo fork's (rwkeyes).
- STRATA_FORCE_ISA=avx2|avx|sse caps the engine's CPU probes, so the older-CPU paths run on any PC
(how this was tested here: no old CPU). cpu_sse42_ok(), isa_floor_build().
- The engine: a normal build still refuses a CPU without AVX2, now pointing at the experimental build;
a floor build checks its own floor (AVX, or SSE4.2 + POPCNT) and warns that it is experimental and
slow (or, on an AVX2 CPU, that the normal build is faster).
- setup: a CPU without AVX2 warns ("experimental, slow") and compiles the engine on the PC with the
floor it has, instead of failing (#595's idea); the prebuilt AVX2 engine is not used there. Recorded
in BUILD.json; a build folder for another floor is configured afresh. STRATA_ISA_FLOOR=avx|none asks
for it on any PC. Windows + AMD has no source build: a clear stop. The MCP server's recommendation
notes it instead of refusing.
- tools/test_setup_oldcpu.py.
Default build: nothing changes (no new CMake definitions, the probes answer as before without the env).
docs/AMD_HIP.md's gfx1030 report measured an i7-13700KF (8P + 8E) at 38-42 tok/s with
all 15 workers against 36 with 8, so a CPU with no more E-cores than P-cores keeps the
engine's own count; the E-heavy chips the fork measured (i9-14900KF: 15 of 23 faster)
get the recommendation.
IQ3_XXS on the RTX 5070, temperature 0, 256 tokens, 8 French prompts (topics outside the corpus), 2 English and 2 code
prompts, arms alternating, 2 passes each: drafts accepted in French 0.505 (en) / 0.507 (cjk) -> 0.597 (fr), higher on
all 8 prompts and in each pass; English 0.610 -> 0.629, code 0.772 -> 0.779. Decode speed was not measurable here
(builds ran beside it); the reporter measured 141 -> 158 tok/s in French on an RTX 5090.
UD-IQ4_XS sits between IQ3_S and UD-Q4_K_XL: IQ3_S gate/up experts (IQ4_XS in one layer) with IQ4_NL downs (Q8_0 in
five layers), 59.5 GB of them, and UD-Q4_K_XL's dense side. The engine's kernels already take these formats; it was
packed (iq_pack.py --compat-bf16) and run on the Strix Halo, not yet on NVIDIA.
- MODELS["UD-IQ4_XS"] in the unsloth family: three shards pinned at the same revision 38bb39e with their sizes and
SHA-256 (the Hub's LFS pointers), the RAM budget like UD-Q4_K_XL (budget, one GPU, no images, --compat-bf16 pack,
no experts.bin), engine 0.1.38 or newer (the fused-layout fix for packs not every layer of which is covered).
- A model can have its own shard pattern and count (model_file / model_shards); budget_model() tells which Unsloth
model a config runs, for the #498 multi-GPU messages and RAM need.
- The AMD question (#429) stays for UD-Q4_K_XL only (nvidia_only): UD-IQ4_XS's formats have AMD prompt kernels.
- --gguf-dir accepts UD-IQ4_XS files (#444's list says so).
- UNSLOTH_Q4.md: its files, what setup does, what is known (Strix Halo), and the BF16 rounding of the hyper-connection
projections that every --compat-bf16 pack has (not changed here). MODELS.md: a paragraph.
- test_setup_unsloth: pins, budget, a mocked install, the engine check, no AMD question, the config read back.
UD-Q4_K_XL's install, its tests and every other model's config are unchanged.
find_nvcc takes the newest toolkit it finds, so with CUDA 12.8 and 12.9 installed, 12.9 was always used - and 12.9's
math headers do not compile with glibc 2.43 (Ubuntu 26.04). STRATA_NVCC=<nvcc> is now the only one considered (a
warning when the file is missing, or when it is too new for the Pascal/Volta build). CUDA_HOME is a candidate like
CUDA_PATH. TROUBLESHOOTING.md: the two build errors and what to do (g++-14 for gcc > 14; CUDA 12.8 or 13.x for glibc
2.43). Without STRATA_NVCC nothing changes.
On a 7200 RPM disk the default --ple-io direct's random reads of the n-gram table stalled prompts past the 60 s
watchdog, and the stall report looked like any other engine stall.
- Engine (Linux): at start with --ple-io direct, the table's disk is looked up in sysfs (queue/rotational); on a
rotational one it warns and recommends --ple-io ram or an SSD (not on WSL, whose virtual disk says rotational).
- Stall report (Linux): the engine's threads in state D and their wchan ("threads waiting on the disk (state D): 16
of 35 (blk_io_schedule x16)"), with what that means.
- Setup (Linux): a model on a rotational disk gets --ple-io ram when the RAM holds the table beside the model,
otherwise a warning that recommends an SSD. Nothing changes on an SSD.
- DETAILS.md: --ple-io direct is for SSDs (it reads unbuffered, not through the OS cache), and a troubleshooting row.
hardware_key, the settings file's key for a saved calibration, read the cards with nvidia-smi's list: on an AMD PC
every card was "?|0GB" (two different cards shared one calibration), and on a PC with an NVIDIA card too a HIP index
named that NVIDIA card or stopped setup ("there is no GPU N"). A HIP config's cards now come from amd_gpus, in HIP's
numbering, with the architecture in the name (an RX 7900 XTX and an RX 7900 XT are both gfx1100; a card without a
product name is known by its arch). Setup reuses a saved calibration on Linux HIP as on NVIDIA (setup_calibration);
Windows HIP keeps the defaults for now. The tuning is still offered on NVIDIA only - a `./setup.sh --calibrate` on
HIP runs it by hand, and each control is verified on HIP before setup offers it.
The NVIDIA key and the settings file's format are unchanged. Tests: tools/test_setup_amd.py CalibrationKey (7).
French answers drafted poorly with the shipped subsets: 23.6% of the token occurrences of French text fall outside
draft_vocab_en.bin, and the draft head cannot propose those (no subset adds the accented words and word pieces).
- tools/draft_vocab.py: a coverage mode. --corpus FILE... adds to the base the most frequent tokens of the corpus that
together cover --coverage (default 0.99) of its occurrences, and prints the share outside the base and the new list.
- data/draft_vocab_fr.bin: draft_vocab_en.bin plus 5,686 tokens (46,211 ids), from 10 French Wikipedia articles
(France, Paris, Napoleon Ier, Langue francaise, Union europeenne, Litterature francaise, Cuisine francaise,
Economie de la France, Revolution francaise, Seconde Guerre mondiale; 1.24 M characters, 326,926 tokens) at 0.99:
python tools/draft_vocab.py --gguf <IQ3_XXS shard 1> --base data/draft_vocab_en.bin --corpus <the articles> \
--coverage 0.99 --out data/draft_vocab_fr.bin
Outside the list on that corpus: 0.74% (23.65% with en).
- setup.py: "fr" in DRAFT_VOCABS / --draft-vocab (~151 MiB head on IQ3_S, between en and cyrillic). The sha256 check
knows it as a shipped subset, so a later --draft-vocab cjk/en replaces it and a subset made by hand stays.
- tests: tools/test_draft_vocab.py (the coverage mode), tools/test_setup_draft_vocab.py (fr replaces a shipped subset
and back, keeps the en ids first in their order, its MiB estimate).
The default subset (cjk) is unchanged.
On a hybrid CPU (Intel 12th gen+, AMD Zen 5 + Zen 5c) setup writes --pool-workers
N into the run config: the performance cores but the host loop's one, plus half
of the efficiency cores. An E-core runs the expert kernels ~2.2x slower and each
layer waits for its slowest part; Hardin22 measured 15 workers at 165 / 116 tok/s
against 106 / 84 with all 23 on an i9-14900KF (#642). A recommendation only: the
line is in the config, a saved calibration's measured count wins, and a CPU whose
cores are all alike gets the same config as before.
The engine's pool, with every core allowed (the default affinity), now orders the
P-cores first on a hybrid CPU, so a smaller pool runs on the P-cores and the first
E-cores rather than on whatever the OS numbered first (from Hardin22's fork).
Non-hybrid CPUs: unchanged.
Co-Authored-By: Hardin22 <[email protected]>
Kept in the run config's vision.max_tokens (the server already passes it
to the encoder); a setup run again keeps it for the same encoder device.
More than the default is allowed with a note on the encode time. Without
the flag the config is as before. DETAILS.md: vision.mmproj can point to
a Q8_0 mmproj of this model.
--no-browser (on a start or at setup) saves "open_browser": false in the
model's run config; start() and run-<model>.bat/.sh then leave --open
out, and the server ignores --open when the config says so (a start
script written earlier still has it). --browser undoes it. Without the
flag the config, the command and the script are as before.
"layer_split" is the first layer of each later GPU ("24,36,42" for 4 GPUs), not a count of layers per card. A
value such as "24,16,12,12" failed the engine's check after the start began, and a JSON list reached the engine as
"[24, 16, 12, 12]".
- server.py layer_split_value: "auto" (default), a comma string or a JSON list of one rising number per GPU after the
first, from 2; anything else stops the start with the format and an example for that many GPUs.
- The engine's message says the same, with the value it got.
- MULTI_GPU.md and setup's --layer-split help say it.
A re-run for an installed model (another --context, say) rewrote
strata-<model>.json from scratch and dropped the user's sampling,
mcp_servers, mcp and other blocks. Setup now carries over every key it
does not write (and env/vision entries of the user's own, such as a Q8_0
mmproj), keeps the earlier file as strata-<model>.json.bak and says what
it kept and which hand-added engine options it did not. A copy set up
like an earlier install carries them over from that install's config.
A first install writes the same config as before.
The native head's cudaMalloc ran after the expert arena had registered ~45 GiB of host pages. WDDM then refused it
on a 16 GB card with the desktop on the iGPU ("native head upload: out of memory" with GiBs free) - the drafter is
loaded before the arena for the same reason. Now the head and the logits buffer load right after the drafter, still
before the expert cache is sized, so the slots are the same. The head's, the drafter's and a split stage's head
failures print "(N MiB of M MiB VRAM free on this GPU)".
setup: a context of 64K or more whose KV streaming does not fit the RAM test now says so (it was silent, and a
regenerated config lost --kv-resident without a word).
One line under the start banner: the config's engine options without the model's file paths, and the
server's fields (host:port, an API key set or not, GPUs, draft subset, fit_max_tokens, thinking budget).
The key itself is never printed.
With the default 700 MiB reserve the expert cache fills an AMD card that also drives a KDE/Wayland desktop. When the
desktop then needs VRAM (the browser --open starts, a new app), amdgpu moves GPU memory to system RAM (GTT), which
the experts already fill: the OOM killer ends kwin_wayland / plasmashell (#560, R9700), or the compositor fails to pin
its framebuffer (#516, 7900 XTX). --vram-reserve-mib 3072 / 4000 fixed it in both reports.
A recommendation, nothing is changed: setup prints the tip on a Linux desktop session (WAYLAND_DISPLAY or DISPLAY) with
the HIP backend and no reserve given, and the server prints it after the start when the engine reports less than
2048 MiB of VRAM free (INFO vram_free_mib) and the reserve is under 3072 - before --open starts the browser.
INSTALL.md and AMD_HIP.md describe the symptom and the fix. No engine code.
installed_configs() and update_install() keep only JSON objects with "exe" and "args"; any other
strata-*.json in the folder gets a one-line warning naming it.
Setup no longer writes a fixed 300 MiB reserve for cards under 8 GB: the engine's own rule (lower the reserve only as
far as the smallest working cache needs, down to 300) chooses better, and a fixed 300 left an emulated small card with
5 MiB free. Setup keeps the tip (an 8K context, --draft-vocab en). When the engine lowered the reserve and the card ends
below the 256 MiB LOW line, it prints one warning (requests may stall; close other programs that use the GPU, or lower
--max-context) instead of advising a larger reserve, which would leave the cache no room.
On an RTX 3060 Laptop (6 GB) the weights, the KV cache and the draft layer left 0.29 GiB, the default 700 MiB reserve
and the draft head took it, the expert cache got 0 slots and the verify window refused to start.
- setup: a single card under 8 GB (nvidia-smi below 7.5 GB; an 8 GB card lists 7.99) gets --vram-reserve-mib 300
unless one is given (images on the GPU keep theirs), marked in the config as setup's own choice so a later setup
decides again. It prints what frees more VRAM if the start still stops: an 8K context, --draft-vocab en, other
programs closed. The draft layer stays (the server needs --mtp and --spec). 8 GB and larger cards: the same config
(golden test).
- engine: a cache too small to lend the prompt path a 256-token chunk (its buffers plus the 128-slot floor; one slot
without --prefill) makes the prompt path allocate its own on top. When the default reserve leaves less than that,
the reserve shrinks to what leaves exactly that cache, down to 300 MiB (what comes after the cache is paid from
the reserve: at 300 an emulated small card ended with 5 MiB free, LOW by the serve check). With no slot at all the start stops with how many
MiB are short, what the session and the draft head take, and the options. A start whose cache could lend that
chunk is sized as before. Checked with a VRAM ballast on the 5070 (IQ3_XXS, 32K): 258 slots needed, 0.71 GiB free
-> "about 333 MiB more", and with --vram-reserve-mib 700 given, "733 MiB more" plus the reserve tip.
- verify's refusal points at that line.
The engine already leaves --vram-reserve-mib MiB of VRAM free (default 700). INSTALL.md now says what it does, its
default, and how to set it (the flag, or "--vram-reserve-mib", "2048" in the config's args). Setup's new flag writes
it into the config only when given (one value: it replaces the images' own 700, with a note when it is less than the
GPU image encoder needs); on a start of an installed model it is saved into that model's args, and a new copy set up
from an earlier install keeps it. The default config is unchanged (golden test). Tests in test_setup_choices.py.
The engine refuses --resident-budget-gib with a layer split, so setup always dropped UD-Q4_K_XL to one GPU. With RAM
>= the GGUF size + UNSLOTH_RAM_LEFT_GB (~135 GB) and 2+ usable cards, setup now asks (one GPU stays the default and
--yes keeps it; an explicit --gpus is honoured; an explicit --resident-budget-gib keeps one GPU) and the split writes
"gpu": [..], "layer_split": "auto" without the budget: all the experts loaded into RAM at start, the files through the
OS file cache (the config #498 measured: 2x RTX 3090, 31 -> 64-78 tok/s). Latent bug: START-HERE --gpus 0,1 (or the
start-time offer, or a hand-edited config) on a UD-Q4 config kept the budget and the engine exited with code 2; the
split now drops it, or setup stops before saving when the RAM is too small. docs/UNSLOTH_Q4.md says so. Tests in
tools/test_setup_unsloth.py; the golden configs are unchanged.
hf() takes the host from HF_ENDPOINT (default huggingface.co, a trailing slash dropped); the pinned revisions, the
fallback to main and the SHA-256 checks are unchanged. tools/mtp_fetch.py reads the MTP tensors from the same host.
Step 5 prints the folder setup expects the model files in and how to use files downloaded by hand (that folder with
their original names, or --gguf-dir), and the mirror when one is set. INSTALL.md says the same. Tests: the endpoint
in hf()/hf_unpinned()/mtp_fetch, and step 5's words over a mocked install. The golden configs are unchanged.
The server window said "loading the experts into RAM (about N GB)" whatever the flags. Now: --mmap-experts maps them
from the model files (the OS file cache reads them, nothing is loaded up front; its "experts via mmap" line triggers
the message when the model has no PLE line), --resident-experts copies the ones the GPU does not hold, and
--resident-budget-gib N copies up to N GiB. setup's start banner says "maps" for the plain mapped mode too. Tests:
the words per flag set, and the narrator over a mapped and an arena log.
In a git clone they run `git pull --ff-only` (a zip copy is told to download the new zip), then `setup.py --update`:
what a plain start does to an install before it starts the model - the Python packages, the ready-made engine when
MIN_ENGINE is newer (a compiled one when its source changed), each model's config upgrades and its draft subset, the
Windows AMD runtime - and nothing else: no question, no model files, no start. Both scripts run from one parsed
block, since the pull can rewrite the running script. Documented in docs/INSTALL.md (Updating) and the README.
Francesco Albano