Niko1221/Strata
1
0 Fork 0
mirror of https://github.com/Niko1221/Strata.git synced 2026-11-07 06:15:58 -10:00
Code Issues Packages Projects Releases 1 Wiki Activity

docs and packaging: the owner's 2026-10-23 guide set brought in for 0.2.5 (one installer per system, engine downloaded on first start, DeepSeek in the app and the Terminal with three sets and the Speed choice, install.sh form kept, versions 0.2.5 and 0.1.50, source setup labelled as the developer path)

Browse Source
This commit is contained in:
Niko1221 committed 2026-10-25 11:28:02 +01:00
1 parent af8c6f8ec7
commit ed4b4db405
30 files changed
+5152 -7586
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

apps/host/server/agent/README.md
+23 -9
View File
@@ -1,5 +1,8 @@
# Agent bridge (Strata chats as DeepSeek Harness sessions)
This page is for developers who work on the app host's agent code. It lists the files, the frames and routes, and the rules the
bridge enforces. Tests are listed at the end.
The Harness is a separate, pinned component (`DSH 0.2.1-alpha.3`, commit `57190202`, MIT). The host finds it through
`STRATA_HARNESS_DIR` (or `<STRATA_COMPONENTS_DIR>/harness`), starts one `dsh --profile web` process on a loopback port,
and talks to its Gateway. Nothing here downloads or builds anything.
@@ -14,12 +17,13 @@ and talks to its Gateway. Nothing here downloads or builds anything.
| `children.mjs` | pure projection of one helper agent's events into its row and its approval card |
| `preset.mjs` | the agent preset (which tools the chat has) |
| `plugins/strata-gate.mjs` | the Manual-mode approval gate (a Harness plugin, loaded by path) |
| `plugins/strata-search.mjs`, `plugins/strata-models.mjs` | the app's web search registered with the Harness; helper agents on a named model |
| `plugins/strata-context.mjs`, `context-budget.mjs`, `context-fit.mjs` | measured context pressure, summaries that fit the window, the check before a message goes out |
| `agent-bridge.mjs` | the chat-to-session map, turns, approvals, Stop, status |
## Frames sent on `/api/chat/stream` for an agent turn
`queued`, `start` (`agent:true`), `reasoning`, `delta`, `agent-tool` (new: upserts a tool card by call id with
`queued`, `start` (`agent:true`), `reasoning`, `delta`, `agent-tool` (upserts a tool card by call id with
`running | proposed | done | denied | error`), `done` (usage, `finishReason`, `cancelled`), `error`, and `notice`
(why a chat is not using the agent). The chat UI renders them with its existing components.
@@ -70,7 +74,7 @@ The chat's Mode menu is real Harness behaviour. Each turn the page sends the cha
What the person chose is saved by `tools-settings.mjs` (inside the host's settings file) and shown by `src/components/SettingsTools.tsx` (`GET/PUT /api/agent/tools`, `POST/PATCH/DELETE /api/agent/tools/custom[/id]`). A change reaches the agent from the next message: the profile carries a `toolsSignature`, and the bridge restarts the Harness (after the running turns end) when it differs.
- Tool groups (`tools-catalog.mjs`): each is one switch in plain words and is one or more preset plugins (`withoutDisabled`); `fetch` and `search` are the two halves of the Harness's `tool-web` (both off drops it). Goals off: `/goal` answers that goals are turned off.
- Web search needs no account. The default is DuckDuckGo's HTML result page (`../web-search-ddg.mjs`), with Bing's plain result page (`../web-search-bing.mjs`) as the second source. Manners, because DuckDuckGo refuses a program that asks fast (a real box: two searches in 1.2 s answered, the third got HTTP 202 and every DuckDuckGo page refused for many minutes): a browser User-Agent and the page's own request shape (Referer, Origin, Sec-Fetch-*), one request at a time per source, 2.5 to 4 s apart (random), at most 8 a minute, a 30 minute session cache per question, a cooldown after a refusal that doubles (1, 2, 4 ... 10 minutes, `Retry-After` honoured) during which that source gets no request, no lite-page follow-up after a refusal, one retry for a timeout or a 5xx, a 15 s limit. When every source refuses, the tool answers one plain sentence (`LIMITED_SENTENCE`: "Web search is limited right now. Try again in a minute, or add a search key (Exa, Perplexity or DeepSeek) in Settings -> Tools -> Web search.") plus an instruction to the model not to retry; the turn guard in `agent-bridge.mjs` (see Turn guard) stops a model that retries anyway. **The results come from DuckDuckGo or Bing**; they are not ranked or filtered here. The person may switch to Exa, Perplexity or DeepSeek with their own key (`../web-search-keyed.mjs`; DeepSeek uses the Harness's own provider): the key is saved in the settings file, reaches the agent only as the environment variable `STRATA_SEARCH_KEY`, never the profile file and never the screen (it only learns `keySet`). A keyed service without its key falls back to DuckDuckGo. The profile pins `web.searchProvider` to the chosen id so two usable backends cannot make the choice ambiguous.
- Web search needs no account. The default is DuckDuckGo's HTML result page (`../web-search-ddg.mjs`), with Bing's plain result page (`../web-search-bing.mjs`) as the second source. Manners, because DuckDuckGo refuses a program that asks fast: a browser User-Agent and the page's own request shape (Referer, Origin, Sec-Fetch-*), one request at a time per source, 2.5 to 4 s apart (random), at most 8 a minute, a 30 minute session cache per question, a cooldown after a refusal that doubles (1, 2, 4 ... 10 minutes, `Retry-After` honoured) during which that source gets no request, no lite-page follow-up after a refusal, one retry for a timeout or a 5xx, a 15 s limit. When every source refuses, the tool answers one plain sentence (`LIMITED_SENTENCE`: "Web search is limited right now. Try again in a minute, or add a search key (Exa, Perplexity or DeepSeek) in Settings -> Tools -> Web search.") plus an instruction to the model not to retry; the turn guard in `agent-bridge.mjs` (see Turn guard) stops a model that retries anyway. **The results come from DuckDuckGo or Bing**; they are not ranked or filtered here. The person may switch to Exa, Perplexity or DeepSeek with their own key (`../web-search-keyed.mjs`; DeepSeek uses the Harness's own provider): the key is saved in the settings file, reaches the agent only as the environment variable `STRATA_SEARCH_KEY`, never the profile file and never the screen (it only learns `keySet`). A keyed service without its key falls back to DuckDuckGo. The profile pins `web.searchProvider` to the chosen id so two usable backends cannot make the choice ambiguous.
- The browser: Playwright's MCP server (`@playwright/mcp`, staged next to the Harness by `apps/desktop/scripts/stage-harness.mjs`) on the person's own Chrome, Edge or Chromium (`browser-probe.mjs`), `--isolated` so it never sees their profile. Tools appear as `mcp__browser__browser_*`. In Manual, opening a page, clicking, typing, uploading, running a script in the page and closing tabs ask (plain reasons: "Open the web page ...", "Click on the page: ..."); looking (snapshot, screenshot, console, network, waiting, listing tabs) does not. Plan refuses every connected tool. The screen says plainly when no browser, or no tool server, is found.
- The person's own tools: `instructions` (the tool returns their text; look-only, never asks), `command` (runs their command with the assistant's text in the `STRATA_INPUT` environment variable, never spliced into the command line; asks in Manual, refused in Plan) and `server` (a web address or a program, added as `mcp__custom-<name>__*`; asks in Manual). Instruction and command tools are served by `custom-tools-mcp.mjs` (stdio MCP) from a JSON file the settings module rewrites on every change, so a switch in Settings is seen without a restart.
- `strata-gate.mjs` implements the Manual/Plan rules above (`planDecision`, `browserDecision`, the look-only instruction server).
@@ -81,7 +85,7 @@ What the person chose is saved by `tools-settings.mjs` (inside the host's settin
- A helper's request for approval is the same card (state `proposed`, id `<helper session>:<call>`), answered with the same `/api/chat/approval`; a chat can only answer its own helpers.
- The run (the chat's one NDJSON response) stays open until the parent is idle, every helper has finished and reported back, and no armed goal is waiting; `done` is sent then.
- Stop: pauses an active goal, interrupts every helper (`subagents/interruptByParent`; `session/cancel` on a helper session is refused by the Harness), cancels the parent, waits; a report that would wake the parent afterwards is cancelled too.
- The number of helpers at once equals the engine's request slots (`maxActiveSubagents` in the profile); the Harness refuses more with a message the model reads.
- The number of helpers at once is the engine's request slots, with a floor of 4 for a local model and 8 for an API model (`maxActiveSubagents` in the profile). The Harness refuses more with a message the model reads.
## /goal
@@ -90,7 +94,7 @@ What the person chose is saved by `tools-settings.mjs` (inside the host's settin
## Thinking and the output budget
- A local route is reasoning-capable (`reasoning: true`) with `compat.thinkingFormat: 'chat-template'`: the request carries `chat_template_kwargs {enable_thinking, reasoning_effort}`, which the Strata server reads (a top-level `enable_thinking` is ignored by it, which is why pi-ai's `qwen` format left the model thinking). The chat's Reasoning menu travels as `thinking` in the stream body; the bridge sends it with `session/selectModel` (`reasoningEffort`) whenever the model or the level changed; a level a route does not have is dropped (never an error). API routes keep the provider default (the menu is disabled for them in the UI).
- `localOutputBudget(context)` = context minus 4096 (it was context/4 capped at 8192, which cut big answers off at exactly 8192 tokens); pi-ai sends the window minus the prompt. App-loaded models get `fit_max_tokens: true` so the engine clamps instead of answering 400. Sampling is not set here: the Harness sends none, so the server's own defaults apply.
- `localOutputBudget(context)` is 30 percent of the window, at least 4096 and at most 32768 (9,830 tokens at 32K, 19,660 at 64K). The Harness's automatic compaction reads it as the room a reply reserves. pi-ai sends the window minus the prompt. App-loaded models get `fit_max_tokens: true` so the engine clamps instead of answering 400. Sampling is not set here: the Harness sends none, so the server's own defaults apply.
## Tasks
@@ -108,7 +112,7 @@ Failures the person sees are plain sentences with a next step (`AgentUnavailable
## Helper agents on a named model (`plugins/strata-models.mjs`, `model-routes.mjs`)
The Harness's `subagent` tool takes an exact `provider` route key and model id, and the app had its model choice switched off, so every helper ran on the chat's model. The profile now enables the Harness's `subagent-model-selection-settings` with every route the app can reach (local instances and saved API models), and the `strata-models` plugin wraps the two tool definitions the Harness registers for each chat (`subagent`, `list_subagent_models`) in place:
The Harness's `subagent` tool takes an exact `provider` route key and model id. The profile enables the Harness's `subagent-model-selection-settings` with every route the app can reach (local instances and saved API models), and the `strata-models` plugin wraps the two tool definitions the Harness registers for each chat (`subagent`, `list_subagent_models`) in place:
- the model is found from its id, `provider/id`, the name in the model picker, `connection name`, or "local" (any letters). A close guess is not started: the tool answers "Did you mean ...?" and nothing runs until the call is made again with that model. Several fits are listed; nothing fitting lists the models that can be used and says they can be added in Models.
- the tool's own text says when to use it ("ask a subagent", "ask model X"), not to look for the model with shell commands, and not to wait with sleep. The system prompt (`preset.mjs`) says the same, so it reaches every model, local or hosted.
@@ -126,7 +130,7 @@ Pictures are on by default for every model that can read them: an API model read
## Turn guard (`turn-guard.mjs`)
A turn that keeps failing the same way is stopped by the app, never left to the model. Seen on a search-heavy `/plan`: every web search was refused, the model searched again and rewrote its plan after each one until the context reached 100%, then the turn ended with no answer and no message.
A turn that keeps failing the same way is stopped by the app, never left to the model.
- **Counted, from what the turn shows:** web searches that failed in a row (3), searches in one turn (10 in a plan turn, 25 otherwise), a plan turn that starts its numbered list from `1.` again (4 times), and a context of 93% or more (the Harness's own compaction did not keep up).
- **Stopped the way Stop stops** (`session/cancel`, helpers first) but ended as a normal turn (`done`, not cancelled, no error card), so the plan the model had written is still proposed. The plan card carries the reason (`plan.caution`); any other turn ends with one `delta` frame that has `own: true` and `key` (the English sentence, translated by the page; `GUARD_NOTES` are in the interface catalog in all eight languages). Each sentence says what happened and gives one next step.
@@ -134,9 +138,7 @@ A turn that keeps failing the same way is stopped by the app, never left to the
- **Never silent:** a turn that ends (`completed` or `max-tokens`) with no answer text gets "I ran out of room in this chat..." when the last request used 85% of the window or more, otherwise "I stopped without writing an answer..." (not when tool cards were shown, not after Stop, not after an error, which says why by itself). Goal runs have their own ending lines and are not guarded.
- Tests: `tests/agent-turn-guard.test.mjs` (counting, always; end to end with the real Harness, the scripted model's `searchloop` and `bloat` scripts and `searchMock: 'limited'` on the bridge, which runs the real search code on a pretend web that refuses, no network).
## Context: measured pressure, summaries that fit, the check before a message goes out (2026-10-22)
Found by the owner's hands-on test of the installed 0.2.1 (Gyro-S, 64K): a 118 KB log paste was 66,500 tokens to the engine and about 29,500 to the Harness (4 characters a token; logs and code are about 2); the engine refused it with a 400, the refusal stayed in the history, and then neither the next message nor `/compact` could go through, because the summary request carried the same history and was refused for its size too. `/compact` itself was never a command of the app: it went to the model as a question and the model answered "nothing to compact".
## Context: measured pressure, summaries that fit, the check before a message goes out
- **`plugins/strata-context.mjs`** (inserted by the profile, like the gate) wraps `tokenMeter.measure` and `llm.stream` in place, once each. Pressure = the last prompt size the model server reported for an answer, plus what changed since priced at the density that measurement showed (`context-budget.mjs`: `pressureNow`, `densityOf`); never lower than the Harness's own price. Node prices are scaled the same way, so the retained tail is in real tokens. A summary request (`purpose: 'compaction'`) that would not fit the window is asked for in pieces: a message bigger than a piece is cut for the request only, each piece builds on the summary before it, a piece the server refuses is halved and tried again. `/strata-context` (a Harness command, like `/strata-gate`) answers `{tokens, window}` for a session so the host can tell the person true figures.
- **`context-fit.mjs`**: before a message is sent the model server counts it (`POST /v1/messages/count_tokens`, its own tokenizer). `decideFit`: send; summarise first and then send (the message fits an otherwise empty chat); or never fits. The last one is not sent: the error frame has `code: 'CONTEXT_MESSAGE_TOO_LARGE'` and the page offers "Attach as a file" (`oversize.mjs`, `/api/chat/oversize`: the text is saved in the chat's folder, the assistant reads it in parts) or "Send a shortened preview". An engine that cannot count lets the message through; the Harness's own compact-and-retry is the net.
@@ -144,3 +146,15 @@ Found by the owner's hands-on test of the installed 0.2.1 (Gyro-S, 64K): a 118 K
- **Visible:** `compaction/start` shows "Summarizing earlier messages..." in the status line; `compaction/end` adds one plain line to the reply. The projector also sends a `context` frame after every model request, so the meter follows the running turn.
- **A refusal that is left** (`CONTEXT_WINDOW_EXCEEDED`) is a plain sentence with the engine's text under Details and the buttons "Summarize and continue" and "Start a new chat". "Try sending the message again" is for other errors only.
- Tests: `tests/agent-context.test.mjs` (arithmetic; the real Harness against the scripted worker with `window`, `charsPerToken` and `noCount`), `tests/oversize.test.mjs`, `tests/context-ui.test.mjs`. The run against a real engine is `scripts/context-e2e.mjs`.
## History / older measurements
- **Search refusals (2026-10):** on a real box two DuckDuckGo searches in 1.2 s answered, the third got HTTP 202, and every DuckDuckGo page refused for many minutes.
The pacing, cooldown and cache rules above come from that test.
- **Search-heavy `/plan` (2026-10):** every web search was refused, the model searched again and rewrote its plan after each one until the context reached 100%.
The turn ended with no answer and no message. The turn guard comes from that run.
- **Large log paste (2026-10-22, installed 0.2.1, Gyro-S, 64K):** a 118 KB log was 66,500 tokens to the engine and about 29,500 to the Harness
(4 characters a token; logs and code are about 2). The engine refused it with a 400. The refusal stayed in the history, so neither the next message nor `/compact` could
go through, because the summary request carried the same history. `/compact` was also not a command of the app yet: it went to the model as a question. This
produced `strata-context.mjs`, `context-fit.mjs` and the host's own `/compact`.
- **Output budget:** an earlier local budget of a quarter of the window, capped at 8192, cut big answers off at exactly 8192 tokens.
docs/AMD_HIP.md
+570 -963
View File
File diff suppressed because it is too large. Load diff
docs/DETAILS.md
+996 -2210
View File
File diff suppressed because it is too large. Load diff
docs/DOCKER.md
+35 -39
View File
@@ -1,9 +1,8 @@
# Strata in Docker
Run Strata as a container: one command, nothing to install on the computer except Docker and your graphics card's driver.
The images come with Strata's engine already built for your card, so nothing is compiled on your side. The model is
downloaded on the first start into a volume (a download that stops continues where it stopped), and every later start goes
straight to serving.
Run Strata in a container with a prebuilt engine. Install Docker, the host GPU driver and the GPU-access
requirements in the table below. NVIDIA and AMD images can download and prepare a model on first start;
Intel Arc needs prepared model files. Keep the data volume to reuse downloads and settings.
> **On this page:** [Which image](#which-image-for-which-card) · [One command per card](#one-command-per-card) ·
> [Docker Compose](#docker-compose) · [Settings](#settings) · [The API key](#the-api-key) · [Connect your apps](#connect-your-coding-apps) ·
@@ -12,7 +11,7 @@ straight to serving.
Docker on a **Mac** has no access to the Mac's graphics chip (no Metal): use the Mac app or the Terminal
([MAC.md](MAC.md), [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md)). Docker Desktop on **Windows** passes NVIDIA cards through
(WSL 2) but not Radeon or Arc cards; on Windows the app is the easier way for every card.
(WSL 2) but not Radeon or Arc cards; on Windows use the native app for supported NVIDIA or AMD cards. Intel Arc has no native Windows engine; follow the Linux guide.
## Which image for which card
@@ -23,11 +22,11 @@ Docker on a **Mac** has no access to the Mac's graphics chip (no Metal): use the
| AMD **RX 9000** series and Radeon AI PRO R9700 (RDNA 4, gfx1200 / gfx1201) | `0.2.3-rocm-gfx120x` | the `amdgpu` kernel driver (Linux 6.8+) |
| AMD **RX 7000** series (RDNA 3, gfx1100 / gfx1101 / gfx1102) | `0.2.3-rocm-gfx110x` | the `amdgpu` kernel driver |
| AMD **Ryzen AI Max** ("Strix Halo", Radeon 8060S, gfx1151) | `0.2.3-rocm-gfx1151` | the `amdgpu` kernel driver |
| Intel **Arc** A-series and B-series (including Arc Pro B70) | `0.2.3-sycl` (only for a release that carries the Intel engine package; otherwise [build it](#build-it-yourself)) | the `i915` or `xe` kernel driver; **prepared model files** (see [Intel](#intel-arc)) |
| Intel **Arc** A-series and B-series (including Arc Pro B70) | `0.2.3-sycl` (if no image exists for a release, [build it](#build-it-yourself)) | the `i915` or `xe` kernel driver; **prepared model files** (see [Intel](#intel-arc)) |
Size of the images on disk (built from the engine packages of the 0.2.0-beta release): `cuda13` 2.2 GB, `cuda12` 2.9 GB, `rocm-gfx120x` 2.8 GB,
`rocm-gfx110x` 2.4 GB, `rocm-gfx1151` 1.5 GB; the download is about a third of that (measured on the 0.1.45 packages: `cuda13` 640 MB, `cuda12` 845 MB,
`rocm-gfx120x` 505 MB, `rocm-gfx110x` 330 MB, `sycl` 450 MB). The model comes on top, on the first start.
**Measured on:** images built from the 0.1.45 engine packages. Size on disk: `cuda13` 2.2 GB, `cuda12` 2.9 GB, `rocm-gfx120x` 2.8 GB,
`rocm-gfx110x` 2.4 GB, `rocm-gfx1151` 1.5 GB. The download is about a third of that: `cuda13` 640 MB, `cuda12` 845 MB,
`rocm-gfx120x` 505 MB, `rocm-gfx110x` 330 MB, `sycl` 450 MB. The model comes on top, on the first start. Current sizes may differ.
Not sure which NVIDIA image? Take `cuda13` when `nvidia-smi` shows driver 580 or newer and the card is an RTX; otherwise `cuda12`.
A wrong pick does no harm: the container starts, says in plain words that the engine does not fit this card, and stops.
@@ -40,7 +39,7 @@ tag if you want updates only when you decide. The examples below use these image
## One command per card
Each command runs Strata in the background, keeps your files in the volume `strata-data`, and listens only on this computer
(`127.0.0.1:8080`). Replace `MODEL` and the key. The first start downloads the model (tens of GB): watch it with
(`127.0.0.1:8080`). Choose `FAMILY` and `MODEL`, and generate your own key. The examples select ISTA-DASLab originals explicitly. The first start downloads the model (tens of GB): watch it with
`docker logs -f strata`. Already have the model's GGUF files? Add `-e MODEL_PATH=/models -v /your/folder:/models:ro` and nothing is
downloaded for the model itself ([Files you already have](#files-you-already-have)).
@@ -49,7 +48,7 @@ downloaded for the model itself ([Files you already have](#files-you-already-hav
```sh
docker run -d --name strata --gpus all -p 127.0.0.1:8080:8080 \
--ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \
-v strata-data:/data -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \
-v strata-data:/data -e FAMILY=qwen -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \
ghcr.io/niko1221/strata:0.2.3-cuda13
```
@@ -59,7 +58,7 @@ docker run -d --name strata --gpus all -p 127.0.0.1:8080:8080 \
docker run -d --name strata --device /dev/kfd --device /dev/dri \
--group-add video --group-add $(getent group render | cut -d: -f3) -p 127.0.0.1:8080:8080 \
--ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \
-v strata-data:/data -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \
-v strata-data:/data -e FAMILY=qwen -e MODEL=IQ3_XXS -e API_KEY=$(openssl rand -hex 24) \
ghcr.io/niko1221/strata:0.2.3-rocm-gfx120x
```
@@ -68,7 +67,7 @@ docker run -d --name strata --device /dev/kfd --device /dev/dri \
```sh
docker run -d --name strata --device /dev/dri --group-add $(getent group render | cut -d: -f3) \
-p 127.0.0.1:8080:8080 --ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \
-v strata-data:/data -e MODEL=IQ3_S -e MODEL_PATH=/models -v /srv/gguf/IQ3_S:/models:ro \
-v strata-data:/data -e FAMILY=qwen -e MODEL=IQ3_S -e MODEL_PATH=/models -v /srv/gguf/IQ3_S:/models:ro \
-e API_KEY=$(openssl rand -hex 24) ghcr.io/niko1221/strata:0.2.3-sycl
```
@@ -96,7 +95,7 @@ Ready-made files are in [docker/compose/](../docker/compose): `docker-compose.nv
```sh
cd docker/compose
cp .env.example .env # set MODEL=..., API_KEY=...
cp .env.example .env # set FAMILY=qwen, MODEL=..., API_KEY=...
docker compose -f docker-compose.nvidia.yml up -d
docker compose -f docker-compose.nvidia.yml logs -f
```
@@ -114,16 +113,17 @@ screens; leave out what you do not care about.
| Variable | What it does | Default |
| --- | --- | --- |
| `MODEL` | model size: `IQ2_XS`, `IQ3_XXS`, `IQ3_S`, `Q2_0`, `UD-Q4_K_XL`, ... ([MODELS.md](MODELS.md)) | Strata picks what suits the card, and says so in the log |
| `FAMILY` | `qwen`, `swift`, `coder`, ... when a size exists in several families | the usual one |
| `FAMILY` | `qwen`, `unsloth`, `gyro`, `coder`, `swift`, ...; use `qwen` for the original ISTA-DASLab packs | installer recommendation; may differ by hardware |
| `MODEL_PATH` | GGUF files you already have: a folder or one `.gguf`, **inside the container** (mount it with `-v`, `:ro` is fine) | download |
| `CONTEXT` | how much text it remembers at once: `32k`, `64k`, `128k`, `256k` or a number | the largest that fits |
| `PARALLEL` | requests answered at the same time, 1 to 8 | fitted to the card |
| `VISION` | `off`, `on` (the model reads pictures), `gpu`, `cpu`; the picture-reader file (about 0.9 GB) is downloaded with the model | `off` |
| `KV` | conversation memory format: `int8`, `q4_0` (half the memory), `k8v4`, `fp16` | `int8` |
| `GPU` / `GPUS` | card numbers as the monitor lists them, `0` or `0,1`; empty or `all` = the engine's choice. Changing it later makes the model's run settings again (the files stay) | all that fit |
| `GPU_ROLES` | a role for each card, `1=main,0=helper` (`main`, `helper`, `off`; cards you do not name are off; not together with `GPU` / `GPUS`). Passed to `launch --gpu-roles`; see [MULTI_GPU.md](MULTI_GPU.md) |
| `TUNE` | `quick` (up to about 20 minutes, once), `thorough`, `skip`; done once and kept in the preset (`RETUNE=1` tunes again) | `skip` |
| `GPU_ROLES` | a role for each card, `1=main,0=helper` (`main`, `helper`, `off`; cards you do not name are off; not together with `GPU` / `GPUS`). Passed to `launch --gpu-roles`; see [MULTI_GPU.md](MULTI_GPU.md) | none |
| `TUNE` | `quick` (runtime depends on hardware and workload), `thorough`, `skip`; done once and kept in the preset (`RETUNE=1` tunes again) | `skip` |
| `API_KEY` / `API_KEY_FILE` | the key clients must send ([below](#the-api-key)); the file form is for Docker secrets | made for you |
| `TUNE_ALLOW_SMALL` | `1` passes `--tune-allow-small` to `launch` (allow tuning on a small card) | `0` |
| `PORT` | the port inside the container | `8080` |
| `PRESET` | the preset name in `/data` | `docker` |
| `STRATA_SET` | advanced engine settings, space separated, like `strata launch --set` (`"prefill=65536 STRATA_PIN_BY_NEED=1"`) | none |
@@ -131,7 +131,7 @@ screens; leave out what you do not care about.
Your choices are saved as a preset in the volume. Start again with no settings and the saved preset is used; start with
settings and they replace the saved ones (tuning results are kept). To change one thing, recreate the container with the new
variable; the model and tuning stay in `/data`. Anything else: `docker exec strata strata launch --help`, or
`docker run --rm IMAGE strata launch --plan --model q2_0` to see what a choice would do without starting anything.
`docker run --rm IMAGE strata launch --plan --family qwen --model q2_0` to see what a choice would do without starting anything.
What is in the volume `/data`: `Strata-data/` (model files, the prepared pack), `state/` (presets, tuning, the key), `install/`
(run settings and logs), `home/`. The engine and the Terminal are in the image, so updating the image never touches your files.
@@ -142,14 +142,14 @@ container's user). Nothing large is written anywhere else in the container.
### Files you already have
Mount the folder with the model's GGUF files read-only and name it: `-v /srv/gguf/IQ3_XXS:/models:ro -e MODEL_PATH=/models
-e MODEL=IQ3_XXS`. Nothing is downloaded for the model itself; the prepared pack is made in the volume.
-e FAMILY=qwen -e MODEL=IQ3_XXS`. Nothing is downloaded for the model itself; the prepared pack is made in the volume.
- The folder holds the shards of one model directly (`...-00001-of-00002.gguf`, `...-00002-of-00002.gguf`), not a folder of folders.
If you mount a parent folder, the container says which sub-folder to name in `MODEL_PATH`.
- The folder has to be mounted: a `MODEL_PATH` that is not in the container, or a folder with no `.gguf` files, stops the start at
once with a message, and nothing is downloaded.
- Read-only (`:ro`) is fine. The pack (about 40 GB for IQ3_XXS) and the draft layer (about 6 GB, fetched once) are written to the
volume, so it needs that room. With pictures on (the default) the picture reader (0.9 GB) is fetched too; `VISION=off` skips it.
volume, so it needs that room. With `VISION=on` the picture reader (0.9 GB) is fetched too; `VISION=off` skips it.
- Links to the files work when the links' targets are mounted too, and `MODEL` is set (without `MODEL`, Strata identifies the files itself
and does not follow links).
- Without `MODEL_PATH`, files you put into the volume by hand (`/data/Strata-data/models/<model folder>/`) are found as well, if they have the
@@ -284,9 +284,9 @@ This is the way to get an image for a computer without registry access or after
folder (made by `packaging/package_engine.py`, or fetched by `strata update` into a components folder) and Docker on Linux:
```sh
docker/build.sh --flavor cuda --engine ~/components/engine/0.1.45.3-linux-x86_64-cuda-cuda13 --tag strata:0.2.3-cuda13
docker/build.sh --flavor hip --engine ~/components/engine/0.1.45.3-linux-x86_64-hip-rx9070 --tag strata:0.2.3-rocm-gfx120x
docker/build.sh --flavor sycl --engine ~/components/engine/<a-sycl-package-folder> --tag strata:0.2.3-sycl
docker/build.sh --flavor cuda --engine ~/components/engine/<cuda13-package-folder> --tag strata:0.2.3-cuda13
docker/build.sh --flavor hip --engine ~/components/engine/<hip-rx9070-package-folder> --tag strata:0.2.3-rocm-gfx120x
docker/build.sh --flavor sycl --engine ~/components/engine/<sycl-package-folder> --tag strata:0.2.3-sycl
```
`build.sh` assembles the build context (the Terminal's files and the package) and runs `docker build`. The build makes the
@@ -332,27 +332,12 @@ the job starts the image once to see that the Terminal inside it runs.
| `rocm-gfx120x` | `linux-x86_64`, hip, variant `rx9070` (gfx1200 / gfx1201) |
| `rocm-gfx110x` | `linux-x86_64`, hip, variant `rx7900` (gfx1100 / gfx1101 / gfx1102) |
| `rocm-gfx1151` | `linux-x86_64`, hip, variant `halo` (Strix Halo, gfx1151) |
| `sycl` | `linux-x86_64`, sycl; **only when the feed lists one** (the 0.2.0-beta feed has none, so no Intel image is built from it) |
| `sycl` | `linux-x86_64`, sycl; only when the feed lists one |
A flavor the feed has no package for is left out, with a line in the log. The same steps run from a Linux computer with Docker:
`docker/publish.sh --release-tag v0.2.3` builds everything and pushes nothing; add `--push` (after `docker login ghcr.io`) to publish.
The images run on the runner without a graphics card, so the card checks (`docker/smoke-test.sh`) stay a manual step on a computer with the card.
## Tested
2026-10-19, images built from the engine packages of the 0.2.0-beta release (engine 0.1.45.3), fetched and checked against the signed feed by
`docker/fetch-engines.py` (the packages are the ones `strata update` installs). Run on real cards with the checks below (`docker/smoke-test.sh`): `cuda13`
and `cuda12` on one RTX A4000 (driver 580), `rocm-gfx120x` on one Radeon AI PRO R9700. All passed: serving 71-161 s after `docker run` with the prepared
files, `docker stop` took 4-5 s with exit code 0, the card's memory was free again, and the second start reused the saved preset. `rocm-gfx110x` and
`rocm-gfx1151` (Strix Halo) were built and passed the package and library checks but have not run on a card. On a computer without an AMD card the
`rocm-gfx120x` container says in plain words that the engine does not match the card and exits with code 3.
2026-10-17, images built from the 0.1.45 engine packages, on real cards: NVIDIA RTX 3060 (`cuda13`
and `cuda12`), Radeon AI PRO R9700 (`rocm-gfx120x`), Intel Arc Pro B70 (`sycl`). On each: the container serves, `GET /v1/models` answers (and
refuses a request without the key), a one-word chat and a tool call work, `docker exec strata monitor --once` works,
`docker stop` ends it with exit code 0 and the graphics memory is free again, and a second start reuses the saved preset.
Not run on a card: `cuda12` on Pascal / Volta (it ran on the RTX 3060), `rocm-gfx110x` (RX 7000). They are built and checked the same way.
## Known limits
- Linux x86-64 hosts only. Docker on Windows reaches NVIDIA cards only; Docker on a Mac cannot use the GPU.
@@ -361,3 +346,14 @@ Not run on a card: `cuda12` on Pascal / Volta (it ran on the RTX 3060), `rocm-gf
- An Arc card needs prepared model files (see [Intel](#intel-arc)).
- AMD **RX 7000** (gfx110x) and RDNA 3 iGPUs: the gfx110X image is built for gfx1100 / 1101 / 1102 (RX 7900, 7800, 7700, 7600). The
Radeon 780M (gfx1103) is experimental and has its own engine package, not in these images.
## History / older measurements
**Tested on real cards.** Checks (`docker/smoke-test.sh`): the container serves, `GET /v1/models` answers (and refuses a request
without the key), a one-word chat and a tool call work, `docker exec strata monitor --once` works, `docker stop` ends it with exit
code 0 and the graphics memory is free again, and a second start reuses the saved preset.
| Date | Images built from | Card | Result |
| --- | --- | --- | --- |
| 2026-10-19 | 0.2.0-beta engine packages (engine 0.1.45.3), fetched and checked against the signed feed | `cuda13` and `cuda12` on one RTX A4000 (driver 580); `rocm-gfx120x` on one Radeon AI PRO R9700 | All passed. Serving 71-161 s after `docker run` with the prepared files; `docker stop` took 4-5 s with exit code 0. `rocm-gfx110x` and `rocm-gfx1151` passed the package and library checks but did not run on a card. On a computer without an AMD card the `rocm-gfx120x` container says the engine does not match the card and exits with code 3. |
| 2026-10-17 | 0.1.45 engine packages | RTX 3060 (`cuda13`, `cuda12`), Radeon AI PRO R9700 (`rocm-gfx120x`), Intel Arc Pro B70 (`sycl`) | All checks passed on each. Not run on a card: `cuda12` on Pascal / Volta (it ran on the RTX 3060), `rocm-gfx110x` (RX 7000). |
docs/GYRO.md
+262 -254
View File
@@ -1,178 +1,208 @@
# Gyro (agentionai rotor quants) on Strata
Gyro-S and Gyro-M are Qwen3.8-Flash-Next quantizations by agentionai
([agentionai/Qwen3.8-Flash-Next-Gyro-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF)): the routed
experts are stored as trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a Hadamard-rotated basis. Strata reads the
published files unchanged. **Setup lists them as a model family of their own** ("Qwen3.8-Flash-Next (Gyro)"); neither
is anybody's default.
This page is for people who want to run Gyro-S or Gyro-M, the smallest Qwen3.8-Flash-Next options, or who want to know
how they compare with the other sizes. They are quantizations by agentionai
([agentionai/Qwen3.8-Flash-Next-Gyro-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF)). The
routed experts are stored as trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a Hadamard-rotated basis. Strata
reads the published files unchanged.
Setup lists them as a model family of their own, "Qwen3.8-Flash-Next (Gyro)". They are not the first recommendation
([MODELS.md](MODELS.md)). Setup may still preselect Gyro for some memory tiers, so choose the provider explicitly if
you care.
> **On this page:** [Quick start](#quick-start) · [Sizes](#sizes) · [Compared with other sizes](#compared-with-other-sizes) ·
> [Platforms](#platforms) · [Intel Arc (SYCL)](#intel-arc-sycl) · [On a Mac](#on-a-mac) ·
> [Install by hand](#install-by-hand) · [Numerics](#numerics) · [Known limits](#known-limits) ·
> [Credit and licenses](#credit-and-licenses) · [History](#history--older-measurements)
## Quick start
```sh
strata launch --family gyro --model Gyro-S # or Gyro-M
```
Setup downloads the pinned revision of the file (size and SHA-256 checked) and prepares the pack
(`tools/iq_pack.py --compat-bf16`: the Hadamard table and the dense weights, 1.5 GB). It also prepares the draft head
(Strata's own MTP head, made from the original model as for every Qwen3.8-Flash-Next file) and the start script. If the
ready-made engine is too old for the pack, setup replaces it or the app offers Update engine.
## Sizes
| | Gyro-S | Gyro-M |
| --- | --- | --- |
| File | `Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf` | `Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf` |
| **Download** (the n-gram table is inside the file) | **58.5 GB** | **92.0 GB** |
| **Memory it needs** (VRAM + RAM together) | **27.6 GB** | **35 GB** |
| Download (the n-gram table is inside the file) | 58.5 GB | 92.0 GB |
| Memory it needs (VRAM + RAM together) | 27.6 GB | 35 GB |
| Experts (trellis) | TQK6 gate/up, TQK7 down | TQ2_T |
| In plain words | the smallest option, for PCs with little memory; lower quality than the 2-3-bit sizes | about IQ3_XXS quality with about 20% less GPU memory; decodes about 16% slower than Q2_0 on a 12 GB RTX 3060 (24% slower before engine 0.2.2) |
| In plain words | The smallest option, for PCs with little memory. Lower quality than the 2 to 3-bit sizes. | About IQ3_XXS quality with about 20% less GPU memory. |
About half of each file is the n-gram (PLE) table: it stays on the SSD and is read row by row for every token, so keep
About half of each file is the n-gram (PLE) table. It stays on the SSD and is read row by row for every token, so keep
the file on a fast local drive. The memory figure is what the weights take once loaded.
Not defaults, by measurement (our evaluation, 2026-10): against Q2_0 at the same GPU bytes Gyro-M halves the KL
divergence (0.159 against 0.312 on 128 chunks) and is level with IQ3_XXS (0.157), but decodes 16-21% slower on a 12 GB
card (RTX 3060, measured before engine 0.2.2: 16% since) and an R9700, and its reasoning set scored lower than Q2_0's (49/60 against 57/60). Gyro-S is 13% worse
in KL than IQ2_XS (0.268 against 0.236) with 24% fewer bytes. So: pick Gyro-S where nothing bigger fits, Gyro-M when
the 20% memory saving over IQ3_XXS matters. Setup and the MCP server mention them only where the other sizes do not
fit the PC; they warn, never refuse. Gyro-L (agentionai's 50 GiB file, 2026-10-12) is not supported: it uses another
trellis family (types 147-149) that the engine does not have.
Gyro-L (agentionai's 50 GiB file) is not supported. It uses another trellis family (types 147 to 149) that the engine
does not have.
## Install
**Pictures.** Gyro reads pictures with the original model's picture reader file, the same one every
Qwen3.8-Flash-Next model uses. Setup downloads it with the model. The reader opens only the vocabulary and the header of
the Gyro file. Pictures are on by default, with the CPU reader under 24 GB of graphics memory and the card's reader from
24 GB.
```sh
./setup.sh --setup --family gyro --model Gyro-S # or Gyro-M; Windows: START-HERE.bat --setup ...
```
## Compared with other sizes
Setup downloads the pinned revision of the file (size and SHA-256 checked), prepares the pack (`tools/iq_pack.py
--compat-bf16`: the Hadamard table and the dense weights, 1.5 GB), the draft head (Strata's own MTP head, prepared from
the original model as for every Qwen3.8-Flash-Next file) and the start script. Needs engine 0.1.45.1 or newer: the pack is native_experts.txt v5 (Hadamard-folded), which the engine reads since the
APR port, and the released 0.1.45 stops at v4. A ready-made engine that is older is replaced by one compiled from this source
(setup), or the app says the model needs a newer engine and offers Update engine.
Gyro is not a default, by our measurements. Setup and the MCP server mention it only where the other sizes do not fit
the PC. They warn and never refuse.
**Supported:** NVIDIA (CUDA, Linux and Windows) and AMD (HIP, experimental: the authors validated gfx1151; we ran
gfx1201 with identical output to rc2). **Intel Arc** (SYCL, Linux, built from source) runs both files: see [Intel Arc](#intel-arc-sycl). **Apple Silicon Macs** (Metal) run both files from
0.2.2: see [On a Mac](#on-a-mac).
- Pick Gyro-S where nothing bigger fits.
- Pick Gyro-M when the 20% memory saving over IQ3_XXS matters.
**Pictures.** Gyro reads pictures with the original model's picture reader file (the same one every Qwen3.8-Flash-Next
model uses; setup downloads it with the model). The reader opens only the vocabulary and the header of the Gyro file, so
the trellis tensors do not matter to it. Pictures are on by default, with the CPU reader under 24 GB of graphics memory and
the card's reader from 24 GB. This needs engine 0.1.45.1 or newer, like the Gyro files themselves.
> **Measured on:** our evaluation, 2026-10, Gyro files against ISTA-DASLab files at the same GPU bytes, KL divergence on
> 128 chunks (lower is better).
**RTX 50 cards (sm_120).** Gyro gives correct output on them with engine 0.1.45.3. The fused gate/up SwiGLU kernel, which
is the default on compute capability 12.0 cards, handles only rows of 80 dot products; Gyro's trellis rows have 160, and
before the fix they went through it anyway. Gyro-S then answered with loops, garbage or nothing: on an RTX PRO 5000, 0 of
8 test questions were answered right before the fix and 8 of 8 after it (greedy and with the app's sampling). Now those
rows use the full-warp kernel, and the identity gate (10 prompts, 4K and 20K prompts) is identical to the run with the fused
kernel off. If Gyro misbehaves on an RTX 50 card, update the engine.
| Comparison | Result |
| --- | --- |
| Gyro-M against Q2_0 | KL 0.159 against 0.312 |
| Gyro-M against IQ3_XXS | KL 0.157, level |
| Gyro-S against IQ2_XS | KL 0.268 against 0.236 (13% worse), with 24% fewer bytes |
| Reasoning set, Gyro-M against Q2_0 | 49/60 against 57/60 |
> **Measured on:** RTX 3060 12 GB, Core Ultra 7, Ubuntu, CUDA 13.4, default flags (MTP draft 4, auto cache), engine
> 0.2.2, the pinned Gyro files, medians of story, code and facts prompts (200 tokens each) over 5 interleaved pairs of
> whole engine starts.
| Decode | tok/s | Against Q2_0 on the same card (65.5 and 64.5 tok/s) |
| --- | ---: | --- |
| Gyro-S | 62.5 | 7% slower |
| Gyro-M | 54.8 | 16% slower |
On an R9700, Gyro-M decodes about 21% slower than Q2_0 (our evaluation, 2026-10). Gyro prompts are 7 to 9% slower than
Q2_0's. What is left of Gyro-M's gap is mostly its dense weights: its Q8_0 matrix-vector kernels take 26% of the GPU
time in a decode profile.
## Platforms
| Platform | Status |
| --- | --- |
| NVIDIA (CUDA, Linux and Windows) | Supported, including RTX 50 cards (sm_120). |
| AMD (HIP) | Experimental. The authors validated gfx1151. We ran gfx1201 with output identical to rc2's. |
| Intel Arc (SYCL, Linux, built from source) | Runs both files. See [Intel Arc (SYCL)](#intel-arc-sycl). |
| Apple Silicon (Metal) | Runs both files. See [On a Mac](#on-a-mac). |
If Gyro misbehaves on an RTX 50 card, update the engine.
**Prompt speed and the ggml build.** The trellis prompt kernels (MMQ) are agentionai's, from their llama.cpp fork (MIT).
Strata builds against upstream ggml, which does not have them, so it adds them itself at configure time:
`cmake/strata_ggml_tq_overlay.cmake` writes a copy of the pinned ggml-cuda headers into the build folder with the three
trellis types routed like IQ2_XS (the tile loader is the fork's, `third_party/ggml-tq/`); only Strata's own MMQ library sees
the copy, every other type is compiled from the untouched files (identity gate below). The ready-made engines read Gyro
prompts through it as fast as rc2 does, and bit for bit the same. If the pinned ggml changes and an anchor of the script
no longer matches, the overlay switches itself off with a message and prompts take the slower FP16 path. A build against the
fork (`-DSTRATA_GGML_DIR=<agentionai/llama.cpp>`, as rc2 does) uses the fork's own instances instead.
Strata builds against upstream ggml, which does not have them, so it adds them at configure time.
`cmake/strata_ggml_tq_overlay.cmake` writes a copy of the pinned ggml-cuda headers into the build folder with the
three trellis types routed like IQ2_XS. The tile loader is the fork's (`third_party/ggml-tq/`). Only Strata's own MMQ
library sees the copy; every other type is compiled from the untouched files. The ready-made engines read Gyro prompts
through it. If the pinned ggml changes and an anchor of the script no longer matches, the overlay switches itself off
with a message and prompts take the slower FP16 path. A build against the fork (`-DSTRATA_GGML_DIR=<agentionai/llama.cpp>`)
uses the fork's own instances instead.
**Decode kernels.** The trellis decode kernels keep the 4 KB codebook in shared memory (`STRATA_IQ_STAGE_GRID=0` reads it
from the global table again) and take four rows per warp. The sums are added in the same order as the unstaged kernels,
so answers are bit for bit the same. A Tesla P100 (compute capability 6.0) runs its own variant, `STRATA_TQ_STAGE`.
## Intel Arc (SYCL)
Gyro-S and Gyro-M run on Intel Arc cards on Linux, with the SYCL engine that setup builds from this source ([INTEL.md](INTEL.md)).
Setup lists them for an Arc like for any card and warns, never refuses, when VRAM and RAM together are short. Tested on one
card: an Arc A750 (8 GB, i915 driver) in a PC with a Ryzen 5 5600X (6 cores, AVX2) and 62 GB of RAM, on the first-release files of Gyro-S and
Gyro-M (the same size as the pinned ones, a different SHA-256). The pinned Gyro-S passes the same parity run (16 cases, 0 failures) and
answers correctly through the server; it was not compared with CUDA.
Not tested by us: a B-series card (supported since 2026-10-22; `strata launch` says it is new and how to report a problem), two Arc cards (the second-card expert helper refuses a Gyro pack and says so), an Arc on the xe driver. The Battlemage AOT build
(`bmg-g31`) compiles with the trellis kernels; it was not run.
Gyro-S and Gyro-M run on Intel Arc cards on Linux, with the SYCL engine that setup builds from source
([INTEL.md](INTEL.md)). Setup lists them for an Arc like for any card and warns, never refuses, when VRAM and RAM
together are short.
The trellis decode kernels do the same integer and float operations in the same order as the CUDA ones (int8 codebook, q8_1 activations),
the Hadamard rotation of the expert input and of the SwiGLU output runs on the card, and the prompt takes the grouped XMX path,
which now decodes the trellis types into its tiles. Experts in the card's cache run on the card; the other misses go to the CPU pool or
over the PCIe link, as with every pack.
**Tested by us:** one Arc A750 (8 GB, i915 driver) in a PC with a Ryzen 5 5600X (6 cores, AVX2) and 62 GB of RAM. It ran
the first-release files of Gyro-S and Gyro-M (same size as the pinned ones, different SHA-256). The pinned Gyro-S passes
the same parity run (16 cases, 0 failures) and answers correctly through the server. It was not compared with CUDA.
**Checks.** `tq_sycl_parity <pack> <model.gguf> [layer ...]` (also `--synthetic [--hadamard] GU/DOWN` and `--bench GU/DOWN`) takes real
experts of the file (layers 0-3, 10, 20, 30 and 47, two experts each) and compares the card with a float reference and with the CPU path:
**Not tested by us:** a B-series card (`strata launch` says it is new and how to report a problem), two Arc cards (the
second-card expert helper refuses a Gyro pack and says so) and an Arc on the xe driver. The Battlemage AOT build
(`bmg-g31`) compiles with the trellis kernels but was not run.
- the decode kernels equal a host replay of their own arithmetic **in every bit** (0 of 3,840 gate/up values and 0 of 7,680 down values
differ, Gyro-S and Gyro-M, all 16 cases each); the prompt dequantizers equal the fp16 codebook in every bit (0 of 4.9 million values);
the card's Hadamard rotation equals the CPU's bit for bit (the fp16 form within 3e-4);
- against the float reference the card is within 1.7-2.0e-2 relative L1 error (the int8 codebook; 3e-2 is the bound every expert type
gets), the CPU path within 0.9e-2. The card and the CPU path therefore agree within tolerance, not bit for bit: the CPU path uses an
int16 codebook and Q8_0 activations, the card the int8 codebook and q8_1, as on CUDA.
The trellis decode kernels do the same integer and float operations in the same order as the CUDA ones (int8 codebook,
q8_1 activations). The Hadamard rotation of the expert input and of the SwiGLU output runs on the card, and the prompt
takes the grouped XMX path, which decodes the trellis types into its tiles. Experts in the card's cache run on the card.
Misses go to the CPU pool or over the PCIe link, as with every pack.
**Against CUDA.** The same Gyro-S file on an RTX 3060 (the f-022-int engine) and on the A750, both with 64 cached experts, no PCIe share and
no CPU prompt share, ten prompts of 26-100 tokens, 48 greedy tokens with the MTP draft: 7 of the 10 prompts give the same 48 tokens. The other three (prompts 5, 7 and 8) agree for 9, 9 and 6 tokens and then differ at a near-tie. The first-token KL is 1.3e-4 to 1.1e-2 and the top token is always the same. The two SYCL prompt paths (oneMKL and XMX) also flip prompt 7 against each other, so the prompt-path rounding alone moves it. A 1,406-token prompt gives the same 16 tokens on both.
**Checks.** `tq_sycl_parity <pack> <model.gguf> [layer ...]` (also `--synthetic [--hadamard] GU/DOWN` and
`--bench GU/DOWN`) takes real experts of the file (layers 0 to 3, 10, 20, 30 and 47, two experts each) and compares the
card with a float reference and with the CPU path.
**Measured** (A750, setup's config: 32K context, 300 MiB VRAM reserve, MTP draft of 4, PCIe probe 10.4 GB/s, median of 5 interleaved pairs):
| Check | Result |
| --- | --- |
| Decode kernels against a host replay of their own arithmetic | Equal in every bit (0 of 3,840 gate/up and 0 of 7,680 down values differ; Gyro-S and Gyro-M, 16 cases each) |
| Prompt dequantizers against the fp16 codebook | Equal in every bit (0 of 4.9 million values) |
| Card's Hadamard rotation against the CPU's | Bit for bit (the fp16 form within 3e-4) |
| Card against the float reference | Within 1.7 to 2.0e-2 relative L1 error (3e-2 is the bound every expert type gets) |
| CPU path against the float reference | Within 0.9e-2 |
| | before | now |
| --- | ---: | ---: |
| Gyro-S prompt, 1,406 tokens | 40.2 tok/s | **82.0 tok/s** (5/5 pairs) |
| Gyro-S prompt, 2,377 tokens | | 101.9 tok/s |
| Gyro-S prompt, 20,000 tokens | | 195 tok/s |
| Gyro-S decode | 13.7 tok/s | **15.8 tok/s** (5/5 pairs) |
| Gyro-M decode (8K context) | | 14.2 tok/s (12.6-14.4), prompt 62 tok/s at 1,406 tokens |
The card and the CPU path agree within tolerance, not bit for bit. The CPU path uses an int16 codebook and Q8_0
activations, the card the int8 codebook and q8_1, as on CUDA.
The decode speed is the CPU's and the link's: with 8 GB the card holds 410 of the 24,576 experts (the dense weights, the draft layer,
the KV cache and the prompt buffers take the rest), so about 15 experts per layer and token come from the CPU pool
(5 workers, AVX2, 8 GB/s: the trellis decode is compute bound, 48 cycles a block) and about 5 over the link. Both move about 10 experts a
millisecond, so the link takes a bigger share than for the i-quants: for packs with trellis experts the engine scales the share with a 12 GB/s
reference instead of 20 (0.48 instead of 0.29 here; fixed 0.45 and 0.60 measured the same; 0.40 gained only 7%). The card's kernels are not the
limit: 20 microseconds an expert (53-68 GB/s of weights) against 18 for IQ3_S, 15 experts take 0.3 ms.
**Against CUDA.** Gyro-S on an RTX 3060 and on the A750, both with 64 cached experts, no PCIe share and no CPU prompt
share, ten prompts of 26 to 100 tokens, 48 greedy tokens with the MTP draft: 7 of the 10 prompts give the same 48 tokens.
The other three agree for 9, 9 and 6 tokens and then differ at a near-tie. The first-token KL is 1.3e-4 to 1.1e-2 and
the top token is always the same. A 1,406-token prompt gives the same 16 tokens on both.
Smaller prompt chunks leave room for more cached experts and cost prompt speed: `--prefill 1024` gives 1,090 slots and +13% decode on a
1,406-token prompt but -24% prompt speed, `--prefill 512` 1,429 slots, -36%. The default stays `--prefill auto`.
> **Measured on:** Arc A750 8 GB, Ryzen 5 5600X, 62 GB RAM, setup's config (32K context, 300 MiB VRAM reserve, MTP draft
> of 4, PCIe probe 10.4 GB/s), median of 5 interleaved pairs, engine 0.2.2 era.
**Gyro-M on 8 GB.** Its dense weights are bigger (2.9 GiB of native projections against 1.9): with setup's 32K context the card is 750 MiB
short of one cache slot and the engine stops with that sentence. It starts with `--max-context 8192 --vram-reserve-mib 100 --prefill 1024`
(412 slots). A card with 12 GB or more does not need that.
| | Result |
| --- | ---: |
| Gyro-S prompt, 1,406 tokens | 82.0 tok/s |
| Gyro-S prompt, 2,377 tokens | 101.9 tok/s |
| Gyro-S prompt, 20,000 tokens | 195 tok/s |
| Gyro-S decode | 15.8 tok/s |
| Gyro-M decode (8K context) | 14.2 tok/s (12.6 to 14.4); prompt 62 tok/s at 1,406 tokens |
**Memory.** Gyro-S needs about 27.6 GB of VRAM and RAM together; the n-gram table (28 GB) stays on the SSD, so on an Arc setup does not
put `--ple-io ram` in the config for Gyro (it does for the other packs on an i915 card). One run used about 24 GB of RAM for the
expert arena. Keep the file on a fast SSD.
Decode speed is the CPU's and the link's. With 8 GB the card holds 410 of the 24,576 experts, so about 15 experts per
layer and token come from the CPU pool (5 workers, AVX2, 8 GB/s; the trellis decode is compute bound) and about 5 over
the link. For packs with trellis experts the engine scales the PCIe share with a 12 GB/s reference instead of 20 (0.48
instead of 0.29 here). The card's kernels are not the limit: 20 microseconds an expert, against 18 for IQ3_S.
## Decode speed on NVIDIA (engine 0.2.2)
Smaller prompt chunks leave room for more cached experts and cost prompt speed. On the 1,406-token prompt,
`--prefill 1024` gives 1,090 slots, +13% decode and -24% prompt speed. `--prefill 512` gives 1,429 slots and -36%
prompt speed. The default stays `--prefill auto`.
The trellis decode kernels read the codebook (2048 two-byte entries, 8 random lookups for every 16 weights) from the global
table. On an RTX 3060 that made the gate/up and down kernels wait on the L1 cache, not on memory: 18 L1 sectors for every
lookup request, the card's L1 at 64% busy while the DRAM was 27% busy. The kernels now keep the 4 KB codebook in shared memory
(`STRATA_IQ_STAGE_GRID=0` reads it in place again) and take four rows per warp, 32 rows per thread block, so the copy is shared
and no lane idles in the 160-dot gate/up rows or the 40-dot down rows. The sums are added in the same order as before, so the
answers are bit for bit the old ones: the identity gate (10 prompts, 4096 and 20000 token prompts) is identical on Gyro-S,
Gyro-M, Q2_0 and IQ3_XXS, and the Q2_0 and IQ3_XXS kernels did not change. A Tesla P100 (compute capability 6.0) runs its own variant of the same idea, STRATA_TQ_STAGE (Gyro-S +30% there, 5 pairs).
**Gyro-M on 8 GB.** Its dense weights are bigger (2.9 GiB of native projections against 1.9). With setup's 32K context
the card is 750 MiB short of one cache slot and the engine stops with that message. It starts with
`--max-context 8192 --vram-reserve-mib 100 --prefill 1024` (412 slots). A card with 12 GB or more does not need that.
RTX 3060 12 GB, Core Ultra 7, Ubuntu, CUDA 13.4, default flags (MTP draft 4, auto cache), 5 interleaved pairs of whole engine
starts, medians of the story, code and facts prompts (200 tokens each), the pinned Gyro files, engine 0.2.1 against 0.2.2.
**Memory.** Gyro-S needs about 27.6 GB of VRAM and RAM together. The n-gram table (28 GB) stays on the SSD, so on an Arc
setup does not put `--ple-io ram` in the config for Gyro (it does for the other packs on an i915 card). One run used
about 24 GB of RAM for the expert arena.
| | decode before, tok/s | after | gain | pairs faster | 4K / 20K prompt |
| --- | ---: | ---: | ---: | --- | --- |
| Gyro-S | 54.3 | 62.5 | +15.2% | 5 of 5 (10 of 10 requests per prompt) | unchanged (0.996, 1.000) |
| Gyro-M | 50.2 | 54.8 | +9.1% | 5 of 5 (10 of 10) | unchanged (1.004, 0.999) |
| Q2_0 (not changed) | 66.4 | 67.1 | 1.010 (noise) | 3 of 5 | unchanged (1.003, 0.999) |
## On a Mac
Against Q2_0 on the same card, before: Gyro-S 54.2 against 64.4 tok/s (16% slower), Gyro-M 50.5 against 65.6 (23% slower).
After: Gyro-S 61.1 against 65.5 (7% slower), Gyro-M 54.1 against 64.5 (16% slower). What is left of Gyro-M's gap is mostly its dense weights: in a
decode profile its Q8_0 matrix-vector kernels take 26% of the GPU time, and they read as many bytes as the card's memory
delivers. The trellis kernels themselves are limited by the shared-memory bank conflicts of the random codebook lookups now. Prompt speed is not changed by this: Gyro prompts stay 7-8% (Gyro-S) and 8-9% (Gyro-M) slower than Q2_0's.
Apple Silicon Macs run Gyro-S and Gyro-M with Metal kernels (`metal/kernels`), the same published files and the same
Hadamard-folded pack. What a Mac needs, the measured speeds on a 32 GB Mac mini and what was checked are in
[MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts). Setup:
## Credit and licenses
```sh
./setup.sh --family gyro --model Gyro-S
```
The trellis types, the Hadamard rotation, the CPU and GPU kernels and the pack step were written by Laurent Zuijdwijk
(agentionai) in agentionai/Strata `rc2` (9ff02c9, MIT, Strata v0.1.40.3 plus Gyro) and ported here commit by commit with
his authorship kept; `third_party/ggml/ggml-common-tq.h` and the MMQ instances come from agentionai/llama.cpp (MIT, the
ggml authors). The weights are agentionai's quantization of Qwen3.8-Flash-Next under the Qwen Community License 1.0
(<https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF/blob/main/LICENSE>).
The app and `strata launch` list it under Gyro models.
## How it runs (by hand)
## Install by hand
The rest of this page is the authors' own guide for a manual install and their measurements on their machines (RTX 5090,
3090, 4090 and a Strix Halo, with Strata rc2). The numbers are theirs; ours are in the next section.
Setup does these steps for you. By hand, from a [source checkout](SOURCE_INSTALL.md):
## 1. Build
### 1. Build
Setup does these steps for you; by hand, from this checkout. For the fast prompt path Strata's GGML dependency is the agentionai llama.cpp fork, which has the trellis types:
For the fast prompt path, Strata's GGML dependency can be the agentionai llama.cpp fork, which has the trellis types:
```sh
git clone --depth 1 https://github.com/agentionai/llama.cpp ../agention-llama.cpp
# NVIDIA. RTX 50 series: use CUDA 13 (CUDA 12.8 crashes the prompt path on sm_120: issues #220, #224)
# NVIDIA. RTX 50 series: use CUDA 13 (CUDA 12.8 crashes the prompt path on sm_120)
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON \
-DCMAKE_CUDA_ARCHITECTURES=120 -DSTRATA_GGML_DIR=$PWD/../agention-llama.cpp
# AMD (experimental: kernel parity and an end-to-end run validated on gfx1151; in a non-root container add
# --group-add $(getent group render | cut -d: -f3), otherwise HIP finds no device)
# AMD (experimental; in a non-root container add --group-add $(getent group render | cut -d: -f3),
# otherwise HIP finds no device)
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_HIP=ON -DSTRATA_PREFILL_MMQ=ON \
-DCMAKE_HIP_ARCHITECTURES=gfx1201 -DSTRATA_GGML_DIR=$PWD/../agention-llama.cpp
cmake --build build -j
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
```
## 2. Download and pack
### 2. Download and pack
```sh
.venv/bin/pip install huggingface_hub hf_transfer
@@ -183,11 +213,10 @@ HF_HUB_ENABLE_HF_TRANSFER=1 .venv/bin/hf download agentionai/Qwen3.8-Flash-Next-
build/native_expert_parity --load-check packs/gyro-s models/Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf
```
The pack is 1.5 GB (dense weights, tokenizer, the Hadamard and expert tables); the experts are read from the
GGUF. Keep the GGUF on a fast local SSD: the n-gram table is read from it for every token.
`--load-check` must end with `0 failures`.
The pack is 1.5 GB (dense weights, tokenizer, the Hadamard and expert tables). The experts are read from the GGUF, so
keep it on a fast local SSD. `--load-check` must end with `0 failures`.
## 3. Draft head
### 3. Draft head
```sh
.venv/bin/python tools/mtp_fetch.py fetch --out mtp
@@ -196,7 +225,7 @@ GGUF. Keep the GGUF on a fast local SSD: the n-gram table is read from it for ev
cp data/draft_vocab.bin mtp/rt/draft_vocab.bin
```
## 4. Serve
### 4. Serve
Save as `strata-gyro-s.json` at the repository root:
@@ -223,147 +252,126 @@ Save as `strata-gyro-s.json` at the repository root:
.venv/bin/python -m serve.server --engine strata --config strata-gyro-s.json --port 8080
```
The web interface is at `http://127.0.0.1:8080`; API clients use `http://127.0.0.1:8080/v1` (OpenAI) or
`/v1/messages` (Anthropic). `--expert-cache auto` sizes the expert cache to the free VRAM: on a 32 GB card at
128k context it holds 23,464 of 24,576 experts (22.9 GiB). On smaller cards it holds fewer and streams the rest
from RAM; nothing else changes.
The web interface is at `http://127.0.0.1:8080`. API clients use `http://127.0.0.1:8080/v1` (OpenAI) or `/v1/messages`
(Anthropic). `--expert-cache auto` sizes the expert cache to the free VRAM. On a 32 GB card at 128K context it holds
23,464 of 24,576 experts (22.9 GiB). On smaller cards it holds fewer and streams the rest from RAM.
## Measured here (the port, 2026-10-15/16)
## Numerics
One AMD Radeon AI PRO R9700 (gfx1201, ROCm 7), one card of four, exclusive box lock, `--prefill auto --spec 4 --spec-min-p 0.5
--mtp --kv int8`, 22K context, interleaved pairs (the arm that goes first alternates), medians, the files of the first
Gyro release (2026-10-07). A = agentionai/Strata rc2 9ff02c9, B = this port.
- **Decode** (Strata's own expert kernels) reads the trellis weights through an int8 copy of the codebook with one
global scale (`tq_lut_i8`, relative RMS error 0.85% of the codebook's standard deviation, far below the quantization
error itself), against q8_1 activations.
- **Prompt processing** decodes them differently. The FP16 path uses the fp16 codebook (`tq_lut_f16`, what GGML's own
dequantizer reads). The MMQ path uses the llama.cpp fork's kernels, which read the int8 codebook as well.
- Decode and prompt results therefore agree within tolerance, not bit for bit. `native_expert_parity` checks one expert
on the decode kernels against GGML's float dequantizer and a float SwiGLU: relative L1 error below 3e-2, the bound
every expert type gets there. Its "bitwise equal" lines compare Strata's decode-once and per-entry decode kernels with
each other, not with a float reference.
- Strata's own CPU kernels for the trellis types (`src/kernels/cpu/tq_*.cpp`, AVX2 and AVX-512, picked at run time)
take about 45% of ggml-cpu's time per expert on AVX2 and decode a block once for all the tokens of a verify window
(`build/tq_cpu_parity --bench` measures them on your CPU). `STRATA_NO_TQ_KERNELS=1` goes back to ggml-cpu.
`STRATA_TQ_ISA=avx2` or `scalar` caps the instruction set.
| B, ggml of | file | decode tok/s A / B (B/A) | pairs B faster | prompt 4K ms A / B | prompt 20K ms A / B |
## Known limits
- The prompt helper on a second GPU does not run for a Gyro pack. Setup of two cards still runs the layer split and the
decode peer. The prompt peer says it is not used for a Hadamard-folded pack, and the prompt stays on the first card.
Prompt processing on a peer GPU over P2P is not validated end to end.
- Token parity against llama.cpp has not been measured.
- `strata --tokens-file ... --max-new N` (benchmark mode) decodes N tokens past the end of the answer. Compare speeds
only up to the answer's end.
- AMD: the trellis kernels pass parity on gfx1151 (ROCm 7.2.1) and a short end-to-end run there generated correctly
(28 to 31 tokens/s). There was no speed tuning or RDNA4 speed run. An AMD GPU other than gfx1151 runs with a warning.
- Gyro-M: 29.9 GiB of experts do not all fit a 32 GB card next to the KV cache. `auto` streams the rest.
## Credit and licenses
The trellis types, the Hadamard rotation, the CPU and GPU kernels and the pack step were written by Laurent Zuijdwijk
(agentionai) in agentionai/Strata `rc2` (MIT) and ported here commit by commit with his authorship kept.
`third_party/ggml/ggml-common-tq.h` and the MMQ instances come from agentionai/llama.cpp (MIT, the ggml authors). The
weights are agentionai's quantization of Qwen3.8-Flash-Next under the Qwen Community License 1.0
(<https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF/blob/main/LICENSE>).
## History / older measurements
### Engine 0.2.1 against 0.2.2 (decode kernels)
> **Measured on:** RTX 3060 12 GB, Core Ultra 7, Ubuntu, CUDA 13.4, default flags, 5 interleaved pairs of whole engine
> starts, the pinned Gyro files.
| | Decode before, tok/s | After | Gain | 4K / 20K prompt |
| --- | ---: | ---: | ---: | --- |
| Gyro-S | 54.3 | 62.5 | +15.2% | unchanged (0.996, 1.000) |
| Gyro-M | 50.2 | 54.8 | +9.1% | unchanged (1.004, 0.999) |
| Q2_0 (not changed) | 66.4 | 67.1 | 1.010 (noise) | unchanged (1.003, 0.999) |
Against Q2_0 before the change: Gyro-S 54.2 against 64.4 tok/s (16% slower), Gyro-M 50.5 against 65.6 (23% slower).
Gyro-S gained +30% on a Tesla P100 from the `STRATA_TQ_STAGE` variant (5 pairs).
### Arc A750 before the XMX prompt path
> **Measured on:** Arc A750 8 GB, setup's config, median of 5 interleaved pairs.
| | Before | Later |
| --- | ---: | ---: |
| Gyro-S prompt, 1,406 tokens | 40.2 tok/s | 82.0 tok/s |
| Gyro-S decode | 13.7 tok/s | 15.8 tok/s |
### The port against agentionai's rc2 (2026-10-15/16)
> **Measured on:** one AMD Radeon AI PRO R9700 (gfx1201, ROCm 7), one card of four, `--prefill auto --spec 4
> --spec-min-p 0.5 --mtp --kv int8`, 22K context, interleaved pairs, medians, the first Gyro release files (2026-10-07).
> A = agentionai/Strata rc2, B = this port (Strata 0.1.45 base).
| B's ggml | File | Decode tok/s A / B | Pairs B faster | Prompt 4K ms A / B | Prompt 20K ms A / B |
| --- | --- | ---: | --- | ---: | ---: |
| agentionai fork | Gyro-S | 85.4 / 89.7 (1.051) | 5/5 | 5418 / 5313 (0.98) | 14358 / 13859 (0.97) |
| agentionai fork | Gyro-M | 72.7 / 82.7 (1.136) | 5/5 | 5183 / 5520 (1.07) | 13421 / 14164 (1.06) |
| upstream + overlay (the ready-made engine) | Gyro-S | 85.3 / 89.6 (1.050) | 5/5 | 5413 / 5298 (0.98) | 14372 / 13854 (0.96) |
| upstream + overlay | Gyro-M | 72.6 / 82.6 (1.138) | 5/5 | 5682 / 5524 (0.97) | 15005 / 14160 (0.94) |
| upstream, no overlay (FP16 prompt path) | Gyro-S | 85.3 / 89.6 (1.050) | 5/5 | 5425 / 6939 (1.28) | 14364 / 21677 (1.51) |
| agentionai fork | Gyro-S | 85.4 / 89.7 | 5/5 | 5418 / 5313 | 14358 / 13859 |
| agentionai fork | Gyro-M | 72.7 / 82.7 | 5/5 | 5183 / 5520 | 13421 / 14164 |
| upstream + overlay (the ready-made engine) | Gyro-S | 85.3 / 89.6 | 5/5 | 5413 / 5298 | 14372 / 13854 |
| upstream + overlay | Gyro-M | 72.6 / 82.6 | 5/5 | 5682 / 5524 | 15005 / 14160 |
| upstream, no overlay (FP16 prompt path) | Gyro-S | 85.3 / 89.6 | 5/5 | 5425 / 6939 | 14364 / 21677 |
Decode is faster than rc2's because the port sits on the newer Strata (0.1.45); the same engine change makes IQ3_XXS 7% faster
(rc2 86.9, port 93.1 tok/s, 2 pairs). Prompts of the ISTA files gained 11-16% from it too (IQ3_XXS 4K 6184 to 5480 ms, 20K
16587 to 13864 ms). Gyro-M's first run against rc2 showed its prompts 5-7% slower; in the second rc2's own Gyro-M prompts
took 15.0 s instead of 13.4 s (the 92 GB file's page-cache state), the port's stayed at 14.2 s, and Gyro-M's prompts are 3-6%
faster than rc2's: that difference was rc2's baseline, not a regression. Without the overlay (plain upstream ggml) the trellis
prompt kernels are missing and prompts go through the FP16 path: 28% (4K) to 51% (20K) slower than rc2, and a different
rounding. Decode is unaffected either way.
Without the overlay, prompts are 28% (4K) to 51% (20K) slower than rc2's. Decode is unaffected. The identity gate (10
prompts, 4096 and 20000 token prompts) was identical between rc2 and the port for Gyro-S and Gyro-M, and between the
split base and the port for Q2_0 and IQ3_XXS, on AMD gfx1201 and NVIDIA sm_86. The pack step writes byte-identical files
to rc2's tools for Gyro-S.
Identity (the fixed gate flags, 10 prompts, exact_pp 4096 and 20000 tokens, same card for both arms):
### The authors' measurements (rc2)
- Q2_0 and IQ3_XXS, f-split against the port (upstream ggml): 10/10 identical, residual dumps and outputs IDENTICAL.
- Gyro-S and Gyro-M, rc2 against the port with the fork's ggml and with upstream + overlay: 10/10 identical; exact_pp 4096 /
20000 residual dumps and outputs IDENTICAL (run alone; a run beside other engines on the box once showed a different dump).
- Q2_0 and IQ3_XXS, f-split against the port with the overlay on, on AMD gfx1201 and on NVIDIA sm_86: IDENTICAL.
- The pack step (`tools/iq_pack.py --compat-bf16`) writes byte-identical files to rc2's tools for Gyro-S.
> **Measured on:** the authors' machines, agentionai/Strata rc2, 2026-10-02 to 04; greedy or the stated sampling, MTP
> draft `--spec 4 --spec-min-p 0.5`, int8 KV. llama.cpp is the agentionai fork with the CUDA trellis fixes. The numbers
> are theirs.
## Measured (the authors', rc2)
Decode tokens/s, server protocol (128K context, `auto` cache, thinking off, temperature 0.7, top-k 20, top-p 0.95,
min-p 0.05, 512 new tokens, 3 repetitions):
All runs: greedy or the stated sampling, same prompt tokens as llama.cpp, MTP draft `--spec 4 --spec-min-p 0.5`,
int8 KV. "llama.cpp" is the agentionai fork with the CUDA trellis fixes (in its `main` since 2026-10-04).
| GPU and file | Experts cached | Prose | JSON | Code | Copy | Prefill 4K | Prefill 16K |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| RTX 5090 32 GB, Gyro-S, Strata | 23,463 | 131.7 | 200.1 | 195.2 | 201.8 | 3,456 | 4,883 |
| RTX 5090, Gyro-S, llama.cpp with MTP draft | | 116.8 | 213.0 | 179.0 | 220.7 | 2,520 | 2,471 |
| RTX 5090, Gyro-S, llama.cpp no draft | | 121.8 | 123.2 | 122.5 | 121.2 | | |
| RTX 5090, Gyro-M, Strata | 18,007 | 113.8 | 170.8 | 153.0 | 175.7 | 3,369 | 4,854 |
| RTX 3090 24 GB, Gyro-S, Strata | 15,756 | 31.8 | 45.2 | 42.2 | 45.1 | 1,402 | 1,863 |
| RTX 3090, Gyro-M, Strata | 11,811 | 27.8 | 39.4 | 34.4 | 37.7 | 1,363 | 1,910 |
| RTX 3090, Gyro-S, llama.cpp `-ncmoe 10`, 32K, no draft | | 16.0 | 16.2 | 16.2 | 16.5 | 468 | 479 |
### RTX 5090 (32 GB), Gyro-S, all experts on the GPU (2026-10-03)
Single prompts of 34 to 177 tokens, 384 new tokens, `--max-context 16384`, Gyro-S, RTX 5090, all experts cached
(2026-10-03): prose 139.9, JSON 221.3, code 209.0 tok/s. Gyro-M on the same card with llama.cpp needs `-ncmoe 8` and
decodes at 70.8 tok/s without a draft. A single RTX 4090 smoke run (Gyro-S, MTP draft, greedy): code 107.5, prose 84.3,
JSON 76.3 tok/s. On 24 GB the CPU matters (the box had a Zen 2 EPYC, AVX2 only). These Strata rows predate Strata's
own trellis CPU kernels.
Decode tokens/s, single prompts of 34-177 tokens, 384 new tokens, `--max-context 16384` (all 24,576 experts cached):
Smaller expert caches, emulated on the RTX 5090 (2026-10-02, before the int8 decode kernels). A real 8 or 16 GB card is
a slower GPU, often on a slower link: read these rows as "it stays usable", not as a prediction for a specific card.
| | prose | JSON | code |
|---|---:|---:|---:|
| Strata | 139.9 | 221.3 | 209.0 |
Server, rc2's `gyro_setup.sh` config (128k context, 23,463 experts cached), thinking off, temperature 0.7,
top-k 20, top-p 0.95, min-p 0.05, 512 new tokens, 3 repetitions:
| | prose | JSON | code | copy |
|---|---:|---:|---:|---:|
| Strata | 131.7 | 200.1 | 195.2 | 201.8 |
| llama.cpp, MTP draft | 116.8 | 213.0 | 179.0 | 220.7 |
| llama.cpp, no draft | 121.8 | 123.2 | 122.5 | 121.2 |
Long real-text prompts, prefill tokens/s: 4k tokens 3,456 (llama.cpp 2,520), 16k tokens 4,883 (llama.cpp 2,471).
### RTX 5090, Gyro-M (2026-10-03)
The experts (29.9 GiB) do not all fit next to the KV cache; `auto` caches 18,007 of 24,576. Same server protocol:
prose 113.8, JSON 170.8, code 153.0, copy 175.7 tokens/s; prefill 4k 3,369, 16k 4,854 tokens/s. llama.cpp needs
`-ncmoe 8` for this file on a 32 GB card and decodes at 70.8 tokens/s without a draft.
### RTX 3090 (24 GB), 2026-10-03
Same server protocol, 128k context, `auto` cache:
| | experts cached | prose | JSON | code | copy | prefill 4k | prefill 16k |
|---|---:|---:|---:|---:|---:|---:|---:|
| Gyro-S | 15,756 | 31.8 | 45.2 | 42.2 | 45.1 | 1,402 | 1,863 |
| Gyro-M | 11,811 | 27.8 | 39.4 | 34.4 | 37.7 | 1,363 | 1,910 |
| llama.cpp, Gyro-S, `-ncmoe 10`, 32k context, no draft | (10 layers on the CPU) | 16.0 | 16.2 | 16.2 | 16.5 | 468 (`-ncmoe 11`) | 479 |
On a 24 GB card, cache misses run on the CPU, so the CPU matters: this box had a Zen 2 EPYC (AVX2, no AVX-512).
The llama.cpp row is the build before the CUDA trellis fixes; those added about 3 % at 24 GB, where the CPU layers
dominate.
The Strata rows predate Strata's own CPU kernels for the trellis types (`src/kernels/cpu/tq_*.cpp`, AVX2 and
AVX-512, picked at run time). Before them a cache miss ran ggml-cpu's `vec_dot`, one token at a time; the new kernels
take about 45 % of its time per expert on AVX2 and decode a block once for all the tokens of a verify window
(`build/tq_cpu_parity --bench` measures them on your CPU). `STRATA_NO_TQ_KERNELS=1` goes back to ggml-cpu;
`STRATA_TQ_ISA=avx2` or `scalar` caps the instruction set (for comparisons).
### Smaller caches (RTX 5090, 2026-10-02, before the int8 decode kernels)
| Expert cache | prose | JSON | code |
|---|---:|---:|---:|
| Expert cache | Prose | JSON | Code |
| --- | ---: | ---: | ---: |
| 24,576 slots (all experts) | 116.4 | 194.3 | 173.5 |
| 22,000 slots | 111.4 | 175.7 | 166.1 |
| 9,000 slots (about a 16 GB card's room) | 81.4 | 74.0 | 96.8 |
| 3,000 slots (about an 8 GB card's room) | 48.0 | 52.4 | 57.9 |
The smaller caches were emulated on the 5090 (fast GPU, PCIe 5.0 x16 at ~51 GB/s). A real 8 or 16 GB card is a
slower GPU, often on a slower link: read those rows as "it stays usable", not as a prediction for a specific card.
The decoded output was read and is fluent and correct; token-level parity against llama.cpp has not been
checked. A real chat at 128k context worked through the web interface.
### Two GPUs (2026-10-04, 2× RTX 4090, no NVLink/P2P)
- **Layer split** (Strata splits the layers across both cards automatically) works with Gyro, including the
idle-card prompt helper (`STRATA_PREFILL_HELP=1`). Repeated runs give identical output.
- **Decode with a second card** (`--peer-device 1` or `--expert-cache-device1 auto`) works and matches single-GPU output.
- **Prompt processing on the peer card** needs GPU-to-GPU access (NVLink or data-centre GPUs); without it Strata keeps
the prompt on the first GPU and logs `no P2P`.
Single RTX 4090 smoke run (Gyro-S, MTP draft, greedy): code 107.5, prose 84.3, JSON 76.3 tokens/s.
## Numerics
- **Decode** (Strata's own expert kernels) reads the trellis weights through an int8 copy of the codebook with one
global scale (`tq_lut_i8`, relative RMS error 0.85 % of the codebook's standard deviation, far below the
quantization error itself), against q8_1 activations.
- **Prompt processing** decodes them otherwise: the FP16 path through the fp16 codebook (`tq_lut_f16`, what GGML's
own dequantizer reads); the MMQ path through the llama.cpp fork's kernels, which read the int8 codebook as well
on the fork's current `main`.
- So decode and prompt results agree within tolerance, not bit for bit. `native_expert_parity` checks one expert
on the decode kernels against GGML's float dequantizer and a float SwiGLU: relative L1 error below 3e-2, the
bound every expert type gets there. Its "bitwise equal" lines compare Strata's decode-once and per-entry decode
kernels with each other, not with a float reference.
## Known limits
- **Not in this port** (rc2 has them; see the list at the end of this page): the prompt helper on a second GPU for a Gyro
pack. Setup of two cards still runs the layer split and the decode peer; the prompt peer says it is not used for a
Hadamard-folded pack and the prompt stays on the first card. Gyro-L is not supported.
- **Source builds:** the SYCL tree has the trellis kernels (Intel Arc, above), and so has the Metal engine (On a Mac, below).
- **Not measured yet:** token parity against llama.cpp.
- **Benchmark mode only:** `strata --tokens-file ... --max-new N` decodes N tokens past the end of the answer;
compare speeds only up to the answer's end.
- **AMD:** the trellis kernels pass parity on gfx1151 (ROCm 7.2.1) and a short end-to-end run there generated
correctly (~28-31 tokens/s); no speed tuning or RDNA4 run yet.
- **Gyro-M:** 29.9 GiB of experts do not all fit a 32 GB card next to the KV cache; `auto` streams the rest.
- **Not validated end to end yet** (Strata runs them and logs a warning): an AMD GPU other than gfx1151, and
prompt processing on a peer GPU over P2P.
## On a Mac
Apple Silicon Macs run Gyro-S and Gyro-M from 0.2.2 (Metal kernels in `metal/kernels`, the same published files, the same
Hadamard-folded pack). What a Mac needs, the measured speeds on a 32 GB Mac mini and what was checked are in
[MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts). Setup: `./setup.sh --family gyro --model Gyro-S` (the app and
`strata launch` list it under Gyro models).
Two RTX 4090 without NVLink or P2P (2026-10-04): the layer split works with Gyro, including the idle-card prompt helper
(`STRATA_PREFILL_HELP=1`), and repeated runs give identical output. Decode with a second card (`--peer-device 1` or
`--expert-cache-device1 auto`) matches single-GPU output. Prompt processing on the peer card needs GPU-to-GPU access
(NVLink or data-centre GPUs). Without it Strata keeps the prompt on the first GPU and logs `no P2P`.
docs/IMAGES.md
+85 -58
View File
@@ -1,88 +1,115 @@
# Image generation (Qwen-Image-2.1-Turbo)
Strata's server can also make images: `POST /v1/images/generations`, OpenAI's images API. The model is
[Qwen-Image-2.1-Turbo](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo) (8 steps, CFG 1) in
This page is for people who want Strata's server to make images through `POST /v1/images/generations`, OpenAI's
images API. It is optional, off unless you ask for it, and separate from reading pictures in chat (for that, use
`strata launch --vision on` with a supported model and reader).
It needs a [source checkout](SOURCE_INSTALL.md) and build tools. The steps below are not part of a normal app install.
The model is [Qwen-Image-2.1-Turbo](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo) (8 steps, CFG 1) in
[Unsloth's GGUF quants](https://huggingface.co/unsloth/Qwen-Image-2.1-Turbo-GGUF), run by
[stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) (`sd-cli`), which setup compiles for your GPU.
It is optional and off unless you ask for it.
The model is under the **Qwen Research License**, not Apache 2.0: read
It is under the **Qwen Research License**, not Apache 2.0. Read
[its license](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo/blob/main/LICENSE) before you use the images
commercially.
## Set it up
## Quick start
./setup.sh --setup --images qwen-image (Linux)
START-HERE.bat --setup --images qwen-image (Windows)
```sh
./setup.sh --setup --images qwen-image # Linux
START-HERE.bat --setup --images qwen-image # Windows
```
or, for an install that is already set up, without running the whole setup again:
For an install that is already set up, without running the whole setup again:
.venv/bin/python tools/image/install.py --config strata-<model>.json
```sh
.venv/bin/python tools/image/install.py --config strata-<model>.json
```
Setup compiles `sd-cli` (stable-diffusion.cpp at a pinned commit) into `engine-image/`, downloads three files into
the data folder's `models/qwen-image-2.1-turbo/` and checks each against its SHA-256, and writes an `"images"` section
into the model's run config.
Then request an image:
| piece | file | size |
|---|---|---|
| denoiser (by GPU memory) | `qwen-image-2.1-turbo-Q4_K_M.gguf` up to 8 GB, `Q6_K` up to 12 GB, `Q8_0` above | 4.2 / 6.3 / 7.6 GB |
| text encoder | `Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf` | 5.1 GB |
```sh
curl http://127.0.0.1:8080/v1/images/generations -H "Content-Type: application/json" \
-d '{"prompt": "a cartoon sloth mascot waving, flat vector illustration", "size": "1024x1024", "seed": 42}'
```
## What setup does
Setup compiles `sd-cli` (stable-diffusion.cpp at a pinned commit) into `engine-image/`, downloads three files into the
data folder's `models/qwen-image-2.1-turbo/`, checks each against its SHA-256 and writes an `"images"` section into the
model's run config.
| Piece | File | Size |
| --- | --- | --- |
| Denoiser, chosen by GPU memory | `qwen-image-2.1-turbo-Q4_K_M.gguf` up to 8 GB, `Q6_K` up to 12 GB, `Q8_0` above | 4.2 / 6.3 / 7.6 GB |
| Text encoder | `Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf` | 5.1 GB |
| VAE | `qwen_image_2.1_vae_bf16.safetensors` | 0.7 GB |
Options: `--image-quant Q4_K_M|Q6_K|Q8_0` (a bigger one than recommended is kept, with a warning),
`--image-backend cuda|hip|vulkan|cpu`, `--image-gpu-policy auto|keep|shrink|unload`, `--images no` (takes the section
out of the config).
### Setup options
Backends: CUDA on NVIDIA (needs the CUDA Toolkit), HIP on AMD cards with their own memory (Linux, the same ROCm as the
engine), Vulkan on AMD APUs/iGPUs (Radeon 780M, Strix Point/Halo), Intel Arc and AMD under Windows. A Linux PC without
the Vulkan development files gets the LunarG Vulkan SDK unpacked into `third_party/` (no sudo); on Windows install the
Vulkan SDK first. On Vulkan setup adds `--vae-tiling --vae-tile-overlap 0.25`: an untiled 1024x1024 VAE decode on a Radeon 780M is one
GPU job long enough to trip the kernel driver's job timeout (no image), and a quarter's overlap decodes in half the
time of sd.cpp's default half overlap.
| Option | Effect |
| --- | --- |
| `--image-quant Q4_K_M\|Q6_K\|Q8_0` | Picks the denoiser. A bigger one than recommended is kept, with a warning. |
| `--image-backend cuda\|hip\|vulkan\|cpu` | Picks the backend. |
| `--image-gpu-policy auto\|keep\|shrink\|unload` | Sets `images.gpu_policy` ([below](#sharing-the-gpu-with-the-text-model)). |
| `--images no` | Takes the section out of the config. |
## Use it
### Backends
curl http://127.0.0.1:8080/v1/images/generations -H "Content-Type: application/json" \
-d '{"prompt": "a cartoon sloth mascot waving, flat vector illustration", "size": "1024x1024", "seed": 42}'
| Backend | Used for |
| --- | --- |
| CUDA | NVIDIA. Needs the CUDA Toolkit. |
| HIP | AMD cards with their own memory, on Linux, with the same ROCm as the engine. |
| Vulkan | AMD APUs and iGPUs (Radeon 780M, Strix Point and Halo), Intel Arc, and AMD under Windows. |
| CPU | Fallback. |
| field | |
|---|---|
| `prompt` | the description (required) |
| `size` | `WIDTHxHEIGHT`, each side a multiple of 32 from 256 to 2048 (`max_side`); default and `auto`: `1024x1024` |
| `n` | images, 1 to 4 (`max_n`); image *i* uses seed + *i* |
| `response_format` | `b64_json` (default) or `url` (`/v1/images/files/<random name>.png`; the server keeps the newest 64) |
| `seed` | Strata's: a whole number for a repeatable image (default: random; the answer says which) |
| `steps` | Strata's: 8 (Turbo's schedule). Another count uses stable-diffusion.cpp's own schedule, which Turbo was not made for |
A Linux PC without the Vulkan development files gets the LunarG Vulkan SDK unpacked into `third_party/` (no sudo). On
Windows, install the Vulkan SDK first. On Vulkan, setup adds `--vae-tiling --vae-tile-overlap 0.25`. An untiled
1024x1024 VAE decode on a Radeon 780M is one GPU job long enough to trip the kernel driver's job timeout, and the
quarter overlap decodes in half the time of sd.cpp's default half overlap.
The answer is OpenAI's shape (`created`, `data[].b64_json` or `data[].url`, `revised_prompt` = the prompt) plus
## Request fields
| Field | Meaning |
| --- | --- |
| `prompt` | The description (required). |
| `size` | `WIDTHxHEIGHT`, each side a multiple of 32 from 256 to 2048 (`max_side`). Default and `auto`: `1024x1024`. |
| `n` | Images, 1 to 4 (`max_n`). Image *i* uses seed + *i*. |
| `response_format` | `b64_json` (default) or `url` (`/v1/images/files/<random name>.png`; the server keeps the newest 64). |
| `seed` | Strata's: a whole number for a repeatable image. Default: random; the answer says which. |
| `steps` | Strata's: 8 (Turbo's schedule). Another count uses stable-diffusion.cpp's own schedule, which Turbo was not made for. |
The answer has OpenAI's shape (`created`, `data[].b64_json` or `data[].url`, `revised_prompt` = the prompt) plus
`data[].seed` and a `strata` object with the seconds, steps and backend. The settings are the model card's for
stable-diffusion.cpp: 8 steps, CFG 1, euler, Turbo's own sigmas, `--diffusion-fa`. An image from the API is the PNG
plain `sd-cli` writes with the same arguments, byte for byte.
that plain `sd-cli` writes with the same arguments, byte for byte.
## Sharing the GPU with the text model
One job at a time: an image job takes the same turn a chat request takes, so a chat request waits for the image and
the other way round. What the text model does meanwhile is the config's `images.gpu_policy`:
One job runs at a time. An image job takes the same turn a chat request takes, so a chat request waits for the image
and the other way round. The config's `images.gpu_policy` decides what the text model does meanwhile.
| policy | the text model during an image job |
|---|---|
| `auto` (default) | `keep` when the image job fits next to it (an APU always), else `shrink` when the engine was started with `--vram-elastic`, else `unload` |
| `keep` | stays loaded and idle |
| `shrink` | gives VRAM from its expert cache (as `POST /v1/vram`) and takes it back afterwards; the conversation stays |
| `unload` | its conversation is saved to a file, the engine is stopped, and the next start restores the conversation (`"reload": "eager"` starts it again right after the image; default `lazy`: with the next chat request) |
| Policy | The text model during an image job |
| --- | --- |
| `auto` (default) | `keep` when the image job fits next to it (an APU always), else `shrink` when the engine was started with `--vram-elastic`, else `unload`. |
| `keep` | Stays loaded and idle. |
| `shrink` | Gives VRAM from its expert cache (as `POST /v1/vram`) and takes it back afterwards. The conversation stays. |
| `unload` | Its conversation is saved to a file and the engine is stopped. The next start restores the conversation. `"reload": "eager"` starts it again right after the image; the default `lazy` starts it with the next chat request. |
Each image runs its own `sd-cli` process, so no image memory stays on the GPU between images. Loading the three
files takes a few seconds per image from the page cache.
Each image runs its own `sd-cli` process, so no image memory stays on the GPU between images. Loading the three files
takes a few seconds per image from the page cache.
## Speed
## History / older measurements
Seconds per image through the API, Q8_0, 8 steps (one `sd-cli` process per image, loading included):
> **Measured on:** seconds per image through the API, Q8_0, 8 steps, one `sd-cli` process per image with loading
> included. Strata version not recorded.
| GPU | backend | 1024x1024 | 512x512 |
|---|---|---|---|
| RTX A4000 16 GB | CUDA | 27 | 8.8 |
| Radeon 780M (iGPU) | HIP | 131 | 29 |
| Radeon 780M (iGPU) | Vulkan | 179 | 43 |
| GPU | Backend | 1024x1024 | 512x512 |
| --- | --- | --- | --- |
| RTX A4000 16 GB | CUDA | 27 s | 8.8 s |
| Radeon 780M (iGPU) | HIP | 131 s | 29 s |
| Radeon 780M (iGPU) | Vulkan | 179 s | 43 s |
On the 780M HIP is faster but runs through the ROCm path whose queue evictions reset this GPU on current firmware
(a driver/firmware defect, not this feature; no reset was seen in these image runs); Vulkan is setup's default there, `--image-backend hip` picks HIP.
On the 780M, HIP is faster but runs through the ROCm path whose queue evictions reset this GPU on the firmware tested
(a driver or firmware defect, not this feature; no reset was seen in these image runs). Vulkan is setup's default
there. `--image-backend hip` picks HIP.
docs/INSTALL.md
+95 -373
View File
@@ -1,426 +1,148 @@
# Installing Strata
# Install Strata
Everything about installing, starting, updating and configuring Strata, on Windows, Linux and Mac, with an NVIDIA
or an AMD graphics card or an Apple Silicon Mac. The short version is in the [README](../README.md#install). An AI
coding assistant can do all of this for you with [AI_SETUP.md](AI_SETUP.md).
This page is for anyone installing or updating Strata (version 0.2.5). Use the **desktop app** for a graphical
interface, or the **Terminal** for `strata launch` and coding tools. Both use released engine packages. A normal
install does not need a source checkout or a startup `.bat` file.
There are three ways to install it. Take the first one unless you have a reason not to:
## Quick start
1. **[The app](#the-app):** a desktop program with the engine inside. Install it, pick a model, chat.
2. **[The Terminal](#the-terminal):** one line in a terminal, then `strata launch`. For scripts, servers and AI agents.
3. **[From source](#from-source-windows):** `START-HERE.bat` on Windows, `./setup.sh` on Linux and Mac. It builds or
fetches the engine itself.
| I want | Do this |
| --- | --- |
| A window with chat | Download the installer from the [latest release](https://github.com/Niko1221/Strata/releases/latest) ([steps](#the-app)) |
| The terminal, or my coding tool | Run the one-line install, then `strata launch` ([steps](#the-terminal)) |
| DeepSeek V4.1-Flash | Pick it in the app, or run `strata launch --family deepseek41` ([model page](MODELS.md#deepseek-v41-flash)) |
> **On this page:** [What you need](#what-you-need) · [The app](#the-app) · [The Terminal](#the-terminal) ·
> [From source: Windows](#from-source-windows) · [Linux](#from-source-linux) ([build from git](#build-from-git-on-linux-to-try-a-pull-request-or-change-the-code)) · [Mac](#from-source-mac) ·
> [AMD cards](#amd-cards) · [Several cards](#two-or-three-cards) · [Docker](#docker-linux) ·
> [Older CPUs](#older-cpus-experimental) · [Updating](#updating) · [Where things are stored](#where-things-are-stored) ·
> [Setup's questions](#setups-questions) · [Tuning](#tuning-for-your-pc) · [All options](#options-without-questions)
[Requirements](#what-you-need) · [Desktop app](#the-app) · [Terminal](#the-terminal) ·
[Update](#updating) · [Files and folders](#where-things-are-stored) · [Build from source](SOURCE_INSTALL.md)
## What you need
| | |
| Computer | Guide |
| --- | --- |
| **GPU** | **NVIDIA** RTX 20, 30, 40 or 50 series, **12 GB VRAM or more** (8 GB runs, slowly). Measured on an RTX 5070 and an RTX 3090; RTX 20 (Turing, since 0.1.27) was tested by a contributor on an RTX 2070. **AMD** Radeon RX 7900 XT / XTX, RX 9070 / 9070 XT and Radeon AI PRO R9700 (validated), RX 7800 XT / 7700 XT and RX 9060 XT (validated by their owners), RX 6800 / 6900 series (community-reported), with 12 GB of VRAM or more. See [AMD cards](#amd-cards). |
| **RAM** | Enough for the size you pick ([which model](MODELS.md#pick-by-memory)); **64 GB** runs every size. A big GPU makes up for less RAM - the [low-RAM mode](MODELS.md#a-big-graphics-card-and-little-ram). |
| **CPU** | x86-64 with AVX2 (any Intel/AMD desktop CPU from the last ~8 years). AVX-512 (Ryzen 7000/9000) is a bit faster. Older CPUs without AVX2 are experimental and slow: [Older CPUs](#older-cpus-experimental). |
| **Disk** | The app and its engine take about 1-3 GB. ~70-80 GB free for the model, ~6 GB for the MTP layer (+1 GB with images). Q2_0 on an AVX-512 CPU also writes a one-time ~40 GB copy of its experts for the fast CPU kernel. On Linux with an AMD card, ROCm takes ~10 GB more when setup installs it. An NVMe SSD is strongly recommended: it makes the first start much faster. |
| **OS** | Windows 10/11, Linux (Ubuntu 22.04/24.04 and Debian 12/13 get everything installed automatically; the app also runs on Fedora 44), or a Mac with Apple Silicon (M1 or newer) and macOS 15 or newer ([Mac](MAC.md)). |
| **Driver** | **NVIDIA:** a current driver, version 580 or newer ([nvidia.com/drivers](https://www.nvidia.com/drivers) or the NVIDIA App). **AMD:** on Linux the kernel's amdgpu driver (no ROCm install needed); on Windows a current AMD Software: Adrenalin Edition driver ([amd.com/support](https://www.amd.com/en/support)). |
| Windows 10/11 or Linux x86-64 with an NVIDIA RTX GPU | Use the app or the Terminal. The engine for your card is downloaded for you. [Older NVIDIA cards](OLDER_GPUS.md) use the CUDA 12 engine. DeepSeek V4.1-Flash runs on RTX 30, 40 and 50 cards in the app and the Terminal. |
| Windows or Linux x86-64 with a supported AMD Radeon | [AMD guide](AMD_HIP.md), including the card-to-package table. DeepSeek V4.1-Flash runs on the Radeon AI PRO R9700 under Linux, in the app and the Terminal. |
| Linux x86-64 with Intel Arc | [Intel Arc guide](INTEL_ARC.md); install the driver/compute runtime and check model preparation requirements |
| Apple Silicon Mac, macOS 15 or newer | [Mac guide](MAC.md). Intel Macs are not supported by the Mac package. |
| AMD Ryzen AI Max / Strix Halo | [Strix Halo guide](STRIX_HALO.md) |
The driver is the only thing you install yourself. With the app, the engine is inside the installer. With the
Terminal, `strata launch` fetches the engine for your card from the signed update feed, and nothing is compiled.
Installing from source sets up the rest as well, the first time: Python 3.12 if you have none (for your user
account, no admin), a private Python environment in `.venv/`, the Strata engine, the model and the MTP draft layer.
On NVIDIA it uses the ready-made engine for RTX
20/30/40/50 and NVIDIA's CUDA libraries from pip (~0.4 GB); if no ready-made engine fits your PC, it offers to install
the build tools (Visual Studio Build Tools + CUDA Toolkit on Windows, `build-essential` + CUDA on Ubuntu) and compiles
the engine for your GPU (asks first; 20-40 minutes once). More: [details](DETAILS.md#before-you-start).
Memory and disk needs depend on the **model, quantization, context and enabled features**. Check
[which model fits](MODELS.md#pick-by-memory); download size is not the same as RAM use. Unified memory counts once.
Use an SSD and leave room for helper files and model preparation. The Terminal's `--plan` prints what is missing.
The normal Python install uses `requirements.txt` and does not install CMake or Ninja. If setup has to compile an
engine or image encoder, it uses existing CMake/Ninja tools when available and installs missing pinned helpers from
`requirements-build.txt` into `.venv`. An older `.venv` keeps any build helpers it already has; setup does not remove
packages during an update.
Install a graphics driver that supports the package you choose. CUDA 13 packages require NVIDIA driver 580 or newer;
older NVIDIA cards use CUDA 12. AMD and Intel requirements are in their guides. Packaged GPU runtimes do not replace
hardware drivers. Models and updates need an internet connection to download.
## The app
The app is the easiest way. It has the engine inside, so there is nothing to build and no setup window.
1. Open the [latest release](https://github.com/Niko1221/Strata/releases/latest).
2. Download the one installer for your system. There is one per system, not one per graphics card:
- **Windows:** run `Strata-0.2.5-windows-x64-setup.exe`. It installs for your user and needs no administrator rights.
- **Linux:** make `Strata-0.2.5-linux-x86_64.AppImage` executable, then open it. If FUSE is unavailable, run the
downloaded file with `--appimage-extract-and-run`. [Linux help](TROUBLESHOOTING.md).
- **Mac:** open `Strata-0.2.5-macos-arm64.dmg`, drag Strata into Applications, then open it. [Mac help](MAC.md).
3. On Windows and Linux the installer has no engine inside. On first start the app shows the engine for your
graphics card with its size and one **Download** button, with progress and resume. If you already downloaded the
engine file, choose **I already have the file** and it installs without internet. The Mac file keeps its small
Metal engine inside.
4. Choose a model and its storage folder, let it download, then chat. DeepSeek V4.1-Flash is in the model list on
the supported cards ([model page](MODELS.md#deepseek-v41-flash)); it has three sizes, an 8K context and a **Speed**
choice, Faster or Full quality.
1. Download the installer for your system from the [releases page](https://github.com/Niko1221/Strata/releases/latest).
The page says which installer is for which graphics card (NVIDIA or AMD). Take the one that matches yours.
- **Windows 10 / 11:** run the `.exe`. It installs for your user and needs no administrator rights. Windows may
say "Windows protected your PC", because the installer is not code-signed yet. Choose **More info**, then
**Run anyway**. The installer asks for the folder (by default `%LOCALAPPDATA%\Programs\Strata`): on a PC
with little space on C:, pick a folder on another drive that you can write to (not Program Files, which needs
an administrator). Updates from inside the app go into the same folder. The engine inside the app
(`%LOCALAPPDATA%\Strata\components`) stays on C:; `STRATA_COMPONENTS_DIR` moves it, and the models go to the
folder you choose in the app.
- **Linux:** the `.AppImage`. Make it runnable (right-click, Properties, Permissions, "Allow executing file as
program", or `chmod +x`), then double-click it. It needs FUSE 2: `sudo apt install libfuse2` on Ubuntu 22.04,
`sudo apt install libfuse2t64` on Ubuntu 24.04 and newer, `sudo dnf install fuse fuse-libs` on Fedora. Without
FUSE you can still start it with `--appimage-extract-and-run` after the file name. The NVIDIA build needs
driver 580 or newer.
- **Mac:** the `.dmg`. Open it, drag Strata onto Applications, then open Strata from there. It needs Apple
Silicon and macOS 15 or newer. If macOS says the app is "damaged", the file is fine: see [MAC.md](MAC.md).
2. Start Strata. The first start is a little slower on Linux and Mac, because the engine inside the app is copied
into your home folder.
3. The first screen reads your PC and recommends a model that fits. Press the install button. The download is
big (40-70 GB). If it stops, it continues where it stopped.
4. Chat. **Closing the window stops Strata and the engine.**
The `strata-engine-…` and `strata-control-…` archives on the release page are components, not installers. The
installer is the same file for a first install and for an update. There is no Intel desktop installer yet; Intel Arc
runs on Linux only, through the [Terminal](INTEL_ARC.md).
**Update:** Settings, then Updates, then **Check for updates**. The buttons **Update Strata** and **Update engine**
appear only when something is new. Your models and saved settings are kept. Unload the model first: an engine that
is running cannot be replaced. **Roll back** in the same place brings the previous engine back.
The first launch can take time to check and prepare large model files. Watch its progress and logs; there is no fixed
startup time for every model or computer. Closing the desktop app stops its managed engine.
**Where things are:** the app's settings and chats are in the `.strata` folder in your user folder, the engine
in `%LOCALAPPDATA%\Strata\components` (Linux and Mac: `~/.local/share/strata/components`), and the models in the
`Strata` folder in your Documents (you can change the model folder in the app). Removing the app does not remove
them. **For a bug report:** Settings, Engine, Diagnostics, View logs, **Save logs**.
On a Mac pictures are on by default and read on the GPU ([MAC.md](MAC.md)). Everything else about pictures is in [Setup's questions](#setups-questions).
The provider order recommended here is **ISTA-DASLab, Unsloth, Gyro, then others**, when a compatible pack fits.
Current app recommendations can differ; choose the model explicitly when you want a particular provider.
## The Terminal
The Terminal is the `strata` command. It uses the same engine, models and saved settings as the app, and it is the
way for scripts, servers and AI agents. Linux (x86-64), Mac (Apple Silicon) and Windows (x64) are supported.
Install once, then use `strata launch`. The Terminal includes Python and downloads the engine for your computer.
Linux and Mac:
**Linux / Mac**
```sh
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
```
Windows (PowerShell):
**Windows PowerShell**
```powershell
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
```
The installer finds out which computer this is, downloads the Terminal archive, checks its SHA-256, unpacks it into
your home folder (`~/.local/share/strata/terminal`, or `%LOCALAPPDATA%\Strata\terminal` on Windows; no
administrator rights), checks the release's signed update feed with the Terminal it just unpacked (if that check
fails it removes what it unpacked), and puts `strata` on your PATH. On Windows, open a new terminal window
afterwards. Then:
On Windows, open a **new terminal** afterwards. On Linux and Mac, if `strata` is not found, use the full path printed
by the installer (normally `~/.local/bin/strata`) or add that folder to your shell's PATH.
```sh
strata launch # guided screens: model, context, pictures, who may connect, then serve
strata launch --model q2_0 --yes --no-tui # the same without questions (scripts, agents)
strata launch
```
`strata launch` downloads the engine for your graphics card (signed, and test-started before it is used) and the
model, starts serving on `http://127.0.0.1:8080` and shows the monitor. It never opens a browser unless you add
`--open`. Its flags and screens: [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md).
Follow the screens to choose a model, context, storage folder and access settings. The monitor opens after startup.
Use the address printed by Strata; the default local endpoint is `http://127.0.0.1:8080`.
The Terminal opens a browser only when you add `--open`.
**Update:** `strata update --check` says what would change, `strata update` installs the new engine (it is test-started
first and switched only if that works; the previous engine is kept), `strata components rollback` goes back. It
refuses while a model is being served. **Check your PC and the installed engine:** `strata doctor`.
For a script or AI agent, inspect a plan before downloading. This example selects ISTA-DASLab Q2_0 explicitly:
To read the install script before you run it, download it first (`curl -fsSL <address> -o install.sh`), read it,
then run `sh install.sh`.
## From source: Windows
1. [Download this project](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or
`git clone` it).
2. Double-click **`START-HERE.bat`**.
3. Answer a few questions - or just press Enter each time for the recommended choice ([the questions](#setups-questions)).
Then it downloads everything (the model is ~70 GB, so the first time takes a while - you can stop and it picks up
where it left off) and **starts the model**. Your browser opens the Strata app at `http://127.0.0.1:8080`.
> **While the model starts, your PC can be slow or stop responding for 1-3 minutes** (longest the first time): Strata
> loads 35-55 GB into your RAM and locks part of it for the graphics card. That's normal - wait, and don't close the
> window. The window tells you what it is doing.
**Next time**, just double-click `START-HERE.bat` again: it starts right away (30-90 s to load the model), nothing is
downloaded twice. Close its window to stop the model. `SETUP.bat` (the same as `START-HERE.bat --setup`) installs
another model or changes the settings. Starting Strata from Task Scheduler at logon needs two task settings, or the
start is 24x slower: [Running it at startup](DETAILS.md#running-it-at-startup-task-scheduler).
## From source: Linux
```bash
./setup.sh
```sh
strata launch --family qwen --model q2_0 --plan
strata launch --family qwen --model q2_0 --yes --no-tui
```
The same questions, the same automatic install (it uses `sudo apt` for Python and, only if it has to compile, for
the build tools), and the same start: `http://127.0.0.1:8080`. Later runs of `./setup.sh` (or `./run-<model>.sh`)
start the model directly; `./setup.sh --setup` installs another model or changes the settings. Other distributions,
WSL and compiling: [details](DETAILS.md#linux).
`--no-tui` leaves the server running in the background. Check it, connect a tool, or stop it:
On Nix/NixOS, you can use the flake:
```bash
nix develop -c ./setup.sh
```sh
strata launch --status
strata connect
strata launch --stop
```
### Build from git on Linux (to try a pull request or change the code)
You do not need the app or the AppImage for this: the source install builds the engine on your PC and keeps
everything in the checkout, so you can edit, rebuild and test.
```bash
git clone https://github.com/Niko1221/Strata.git && cd Strata
./setup.sh # first run: .venv, the engine build (build/, or build-hip/ on a Radeon), the model, then it serves
```
- **What it needs:** `git`, a C++ compiler and `cmake` (setup installs the build tools through `sudo apt` on
Debian and Ubuntu; elsewhere install them first), and the CUDA Toolkit for NVIDIA or ROCm with the HIP compiler for
a Radeon card ([AMD_HIP.md](AMD_HIP.md)). Python goes into `.venv` of the checkout.
- **After you change the code:** run `./setup.sh` again. It compiles only what changed and starts the model. To
update without starting: `git pull`, then `./update.sh`. By hand, the build folder is an ordinary CMake build:
`cmake --build build --target strata -j8` (`build-hip` for a Radeon card; the options are in
[AMD_HIP.md](AMD_HIP.md)).
- **To try a pull request:** `git fetch origin pull/<number>/head:pr-<number> && git checkout pr-<number>`, then
`./setup.sh`. `git checkout main` goes back; the next `./setup.sh` rebuilds.
- **Settings for the engine** (for example `STRATA_HIPBLASLT_TUNING=/path/to/table.txt`) go into the `"env"` object of
`strata-<model>.json`, which `./setup.sh` writes next to the checkout. `./run-<model>.sh` starts the model with it.
- **The Terminal monitor from a checkout:** the monitor needs one extra package, which the source install does not
put into `.venv`:
```bash
.venv/bin/pip install -r requirements-monitor.txt
STRATA_PYTHON="$PWD/.venv/bin/python3" bin/strata monitor --url http://127.0.0.1:8080
```
(`bin/strata` is the same `strata` command as the Terminal's: `bin/strata --help` lists the rest.)
- **Only compile, without the installer:** the engine is a normal CMake project. For NVIDIA:
`cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native &&
cmake --build build --target strata -j8`. The options, and the flags setup passes, are in `setup.py` (search for
`-DSTRATA_`).
## From source: Mac
On an Apple Silicon Mac with macOS 15 or newer, install Apple's Command Line Tools once (`xcode-select --install`),
then run `./setup.sh` in the Strata folder. It builds the Mac (Metal) engine on your Mac and asks the same
questions. The app and the Terminal do not build anything. All of it, and what fits which memory size, is in
[MAC.md](MAC.md).
## AMD cards
The app and the Terminal pick the AMD engine for a Radeon card by themselves. From source the steps are the same as with NVIDIA: `START-HERE.bat` on Windows, `./setup.sh` on Linux. Setup finds the Radeon
card and chooses the AMD (HIP) engine by itself on a PC with no NVIDIA card Strata can use; `--backend hip` chooses
it on a PC that has both. Supported cards: RX 7900 XT / XTX (gfx1100), RX 7800 XT / 7700 XT (gfx1101), RX 9060 XT
(gfx1200), RX 9070 / 9070 XT and Radeon AI PRO R9700 (gfx1201), and the RX 6800 / 6900 series (gfx1030). Integrated
Radeon GPUs are listed as not supported.
On Linux setup uses a system ROCm 7 when there is one, or installs ROCm into `.venv` from AMD's wheels (~10 GB, no
sudo), and compiles the engine on your PC for the card (10-20 minutes, once; it needs a C++ compiler and git:
`sudo apt install build-essential git`). Several AMD cards share the model with `--gpus`, as on NVIDIA.
What differs from NVIDIA for now: `--calibrate` is NVIDIA-only, and Unsloth's 4-bit model needs an NVIDIA card. Measurements per card, the
build by hand and the tuning tables: [AMD_HIP.md](AMD_HIP.md).
## Two or three cards
**Two or three NVIDIA cards?** Just run `START-HERE.bat`: it lists your cards, says which ones Strata can use, and
asks whether to share the model across them (recommended when two can). An install made on one card asks once at
its next start. Or choose yourself: `START-HERE.bat --gpus 0,2` (both, remembered), `--gpus all`, or `--gpu 0` (one
card, this start only). Each card keeps the experts of its own layers, and prompts flow through the cards in a
pipeline: on an RTX 5080 + RTX 3090 prompts were read 18-20% faster than on the 5080 alone, decoding on par.
Every card must be an RTX 20 series or newer with 8 GB or more. See [MULTI_GPU.md](MULTI_GPU.md).
**Several AMD cards:** setup takes one card (the one with the most VRAM, or `--gpu N`) unless you name more:
`--gpus 1,0` splits the model's layers across them, the faster card goes last and is the Main card (`--gpu-roles 1=main,0=helper` says it yourself; docs/MULTI_GPU.md; numbers as setup lists them). A
split pays only when no single card holds the model's experts ([AMD_HIP.md](AMD_HIP.md#rdna4-gfx1201)).
## Docker (Linux)
Ready-made images for NVIDIA (CUDA 13 and CUDA 12), AMD (RX 9000 / R9700 and RX 7000) and Intel Arc cards, with the Terminal
inside: `docker run` one command, the model downloads on the first start into a volume, nothing is compiled on your computer.
Which image for which card, the commands, Docker Compose, the API key, updating and troubleshooting: [DOCKER.md](DOCKER.md).
Building from source: `docker build -t strata:src .` (CUDA 13; [DOCKER.md](DOCKER.md#build-it-yourself)).
## Older CPUs (experimental)
Strata's ready-made engine needs AVX2 (Intel Haswell 2013, AMD Zen 2017 or newer). Since 0.1.39 an older CPU - AVX
only (Sandy Bridge / Ivy Bridge, Xeon E5 v1/v2, AMD Bulldozer) or SSE4.2 only (Nehalem / Westmere, Xeon X5600) - runs
as an **experimental** build that setup compiles on that PC. Run setup as usual: it says the CPU has no AVX2, warns
that this is experimental and slow, and compiles the engine (10-20 minutes, once; Windows needs the Visual Studio Build
Tools and the CUDA Toolkit, which setup offers to install). It does not stop. The ready-made engine still refuses such
a CPU, and says so.
What differs in that build (`STRATA_ISA_FLOOR=avx` or `none`, chosen from the CPU):
- ggml-cpu is compiled for AVX or SSE4.2 instead of AVX2, and the CPU's share of the experts runs on its kernels.
Strata's own AVX2 / AVX-512 kernels stay in the program but are used only on a CPU that has them.
- Only the i-quant models' packs run (every model setup installs on such a CPU); the AVX-512 Q2_0 pack does not.
- Speed: the GPU part is unchanged and the CPU part is slower, so a GPU with more VRAM (more experts cached there)
helps most. With the older paths forced on a Ryzen 5 7600 + RTX 5070 and the expert cache held at 1500 slots, greedy
decode went from 26 to 11-17 tok/s with the AVX build and to 3.5-3.8 tok/s with the SSE4.2 build (Coder and IQ3_XXS,
10 prompts each, one run). An old Xeon with DDR3 will be slower than these.
- Untested on a real old CPU by us. It was checked here by forcing the older paths on a Ryzen 5 7600
(`STRATA_FORCE_ISA=avx` or `sse`): the answers were sane and passed the engine's checks, but not the same token for
token as the normal build (ggml-cpu rounds the CPU experts differently, as the AVX2 kernel does from the AVX-512
one). Contributors ran earlier versions of the same approach on a Xeon E5-2680 (2.87 tok/s with a GTX 1080 Ti)
and on Xeon E5-2687W / X5690 machines.
- AMD cards: Linux only (setup compiles the AMD engine there anyway); NVIDIA on Windows and Linux.
- Two-socket Xeons: memory on the other socket is slow; keep the RAM on one CPU if you can.
By hand: `cmake ... -DSTRATA_ISA_FLOOR=avx` (or `none`) builds it; `STRATA_ISA_FLOOR=avx` set for setup builds it on
any PC (to try it). Problems and results on real hardware are welcome as GitHub issues.
If a model is already running with different settings, add `--restart` to replace that run; `--yes` alone does not
restart it. A restart interrupts current answers. See [all launch options](TERMINAL_LAUNCH.md),
[advanced Terminal commands](TERMINAL.md), and [agent setup](AI_SETUP.md).
## Updating
**The app:** Settings, Updates, **Check for updates**, then **Update Strata** or **Update engine** (see [the app](#the-app)).
**Desktop app:** open Settings and find the update controls, check for updates, and use the offered app or engine update.
Unload running models before replacing the engine. Your model files and settings are kept.
**The Terminal:** `strata update --check`, then `strata update` (see [the Terminal](#the-terminal)). Add `--app` to
download the app's installer too. Models and saved tuning presets are never downloaded, replaced or deleted by an
update. How the signed update feed works: [packaging/UPDATES.md](../packaging/UPDATES.md). What Strata downloads and
how it checks it: [SAFETY.md](SAFETY.md).
**Terminal:**
**From source: `UPDATE.bat`** (Linux: `./update.sh`) updates Strata without starting the model - for when the GPU is busy with
something else, or you just want the new version ready. In a `git clone` it runs `git pull`, then does what
`START-HERE.bat` does before a start: the engine (a new ready-made one when the new version needs it; on Linux a
compiled engine is compiled again when its source changed), the Python packages, and each installed model's settings
and draft subset. The model files are not touched (at most a new engine is downloaded) and no question is asked.
Close the model's window first (a running engine cannot be replaced); start the model later with `START-HERE.bat` as
usual. In a copy that was downloaded as a zip it says to download the new zip (below): it cannot fetch new files
itself.
```sh
strata update --check
strata update
strata --version
```
Or by hand: download the new version and unzip it anywhere (or `git pull`), then run `START-HERE.bat` (Linux:
`./setup.sh`) in it. The model files are kept in a `Strata-data` folder next to your Strata folder, so a new copy
finds them and sets itself up the same way - nothing big is downloaded again. On Linux after a `git pull`, setup
compiles the engine again when its source changed (a minute or two for the changed files); if that compile fails, it
says so and starts the engine you had.
Stop models first. The next Terminal command uses the updated Control component. The app, Control and engine have
separate version numbers, so they can differ.
Use `strata components rollback` to return to the previous engine. See [update details](../packaging/UPDATES.md).
Source checkouts have a [separate update workflow](SOURCE_INSTALL.md#updating).
## Where things are stored
- **Your chats: only in your browser.** The Chat tab keeps the conversation, its settings and the API key you typed
in the browser's local storage (`strata.*` keys) - not on the server and not in the Strata folder. Pictures are not
kept, only their names. Another browser or a private window starts empty; clearing the site's data deletes them.
- **How the model starts:** `strata-<model>.json` in the Strata folder (context, GPUs, host, API key, ...), written
by setup; next to it `run-<model>.bat` / `.sh`, the log `strata-<model>.log` and, when you use "Use for other
apps too", `strata-<model>.shared-settings.json`. Running setup again for the same model (another context, say)
rewrites the keys setup writes (`exe`, `args`, `port`, `gpu`, `host`, `api_key`, `vision`, ...) and keeps the
ones you added (`sampling`, `mcp_servers`, `mcp`, `cors_origins`, ...); the earlier file is kept as
`strata-<model>.json.bak` (0.1.39). Engine options you added to `"args"` by hand are not carried over: setup names
them, and you add them again.
- **The model files** (`models/`, `packs/`, `mtp/`, 70-120 GB): in **`Strata-data` next to the Strata folder**, or
wherever `--data-dir` put them.
- **Where that data folder is:** `%APPDATA%\Strata\settings.json` on Windows, `~/.config/strata/settings.json` on
Linux ([details](DETAILS.md#before-you-start)).
- **The program itself:** `.venv/`, `engine/` and `third_party/` in the Strata folder.
- **Where github.com cannot be reached:** a build from source downloads llama.cpp's source (ggml, gguf-py) from
GitHub at the commit `LLAMA_CPP_COMMIT` in `setup.py` names. Put that commit's tree in `third_party/llama.cpp`
yourself (from a mirror or another PC) and setup uses it instead (#585).
| Item | Default location |
| --- | --- |
| Desktop chats and host state | `~/.strata` (or `STRATA_HOME`) |
| Engine components, Windows | `%LOCALAPPDATA%\Strata\components` |
| Engine components, Linux / Mac | `$XDG_DATA_HOME/strata/components`, normally `~/.local/share/strata/components` |
| Terminal files | The `terminal` folder beside `components`; the installer prints the exact path |
| Models | The folder chosen in the app or launch screens; an existing model library is reused |
### Keeping the models on another disk
The desktop app normally proposes `Documents/Strata/Models` for a new library. The Terminal discovers saved
folders from earlier installs. Use the path shown by the app or `strata models list`, not an assumed folder name.
`--data-dir` changes where new downloads go; it does not move existing models unless you add `--move-data`.
See [model discovery](TERMINAL_LAUNCH.md#models-you-already-have).
The model files are the only big part (70 to 190 GB). Put them where there is room; the app and the Terminal use the same folder.
## More options
- **In the app:** the first screen shows the model folder with a button to change it, and Settings > Storage changes it
later. Pick a folder you can save files in; a whole drive (`D:\`) becomes `D:\Strata`, which Strata makes and says so.
Change folder then asks what to do with your models: move them (copied file by file and checked before the old copies are removed; you can stop at any time),
keep them where they are and also use the new folder, or start empty (the old files are never deleted). From the Terminal the same move is
`strata models move NEW_FOLDER` (add `--plan` to see what it would do first).
- **In the Terminal:** `strata launch --data-dir D:\Strata` once. The folder is remembered for the next `strata launch`
and `strata models list`. Until you give it one, the Terminal uses the folder the app chose; the app keeps its own setting
(Settings > Storage) and does not read the Terminal's. `--move-data` moves the models you already have into it; without it they stay where they
are and the new folder gets the new downloads. Use your own files in place with `strata launch --model-path FOLDER` (a
folder of shards, a folder above it, or one `.gguf` file); a folder that holds several sizes needs the one `.gguf` file.
- **Drives that are not NTFS** (exFAT, FAT32, a network share) work: Strata writes straight into `<folder>\models` and makes no
link there.
- **What stays on C:** the engine (`%LOCALAPPDATA%\Strata\components`, 1 to 2 GB per version; `STRATA_COMPONENTS_DIR` moves it),
the app itself (the installer asks), your settings, chat history, tuning results and logs (`%LOCALAPPDATA%\Strata`,
`%APPDATA%\Strata`; a few MB). Nothing else large: downloads, the prepared model data (`packs`) and the draft layer (`mtp`) are
written in the model folder, and so are the half-finished files of an interrupted download.
- **Docker:** `-v /mnt/big/strata:/data` (a folder on the big disk) instead of the volume `strata-data`; everything above lives in it.
## Setup's questions
These are the questions of the installer from source (`START-HERE.bat`, `./setup.sh`). The app's first screen and
`strata launch` ask the same things in their own look.
- **Which model and size?** The original, Swift 1.5, the Coder or Unsloth's 4-bit, and Q2_0, IQ2_XS, IQ3_XXS or
IQ3_S - see [which model](MODELS.md).
- **How much context?** How much text it can keep in mind at once (it suggests one for your card). 384K and
512K (experimental) extend the model past its trained 262K by rope scaling - the setup turns it on itself (yarn and a
covering factor; `--rope-scaling`/`--rope-scale` override) ([details](DETAILS.md): "Context extension past
262K").
- **Images?** Whether it should also read pictures. Pictures are on by default for models that can read them. The
image encoder (0.9 GB) runs on the CPU when your card has less than 24 GB (about 3-13 s per picture, nothing taken
from the GPU) and on the GPU from 24 GB (0.1-0.5 s per picture, ~1.4 GB of VRAM kept free for it). The rule is the
same for NVIDIA and AMD cards; `--vision gpu` and `--vision cpu` choose the reader yourself, `--vision no` turns
pictures off. On a Radeon AI PRO R9700 the GPU reader took 57 ms for a 260-token picture and the CPU reader 3.1 s.
A Mac reads on its GPU (0.45 s for a 260-token picture on an M4). [Details](DETAILS.md#images-vision).
- **Experimental speed projection?** Off unless you say yes - [read what it does](DETAILS.md#experimental-speed-projection-experimental-off-by-default)
first. It changes how the model answers, and only the original model offers it.
With `--yes` setup takes the recommended answer to every question.
## Tuning for your PC
Every PC is different: `START-HERE.bat --calibrate` (Linux: `./setup.sh --calibrate`) measures a few engine settings
on yours and keeps the fastest (about 15-30 minutes, longer on a slow card; on an RTX 5070 with a Ryzen 5 7600 it made the Coder 7% faster).
It keeps a setting only when it is more than 3% faster, and the result is remembered per PC and model, so updates
keep it. Measuring the CPU worker count needs a fresh engine, so the model is loaded more than once: the PC is
busy, and can stop responding for a minute or two, once per restart. When it finishes it **starts the model**,
so the server is already running when it returns - do not start it a second time.
NVIDIA cards for now. [What it measures](DETAILS.md#double-click-start-herebat).
## Options without questions
For the Terminal, `strata launch --help` lists every flag; the most used are in [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md). These are the options of the installer from source:
```
START-HERE.bat --setup install another model, or change context / images
SETUP.bat the same (double-click it)
START-HERE.bat --model IQ2_XS --context 32768 --vision yes --yes no questions
START-HERE.bat --gguf-dir D:\models\IQ2_XS use GGUF files you already have
START-HERE.bat --inspect D:\models\some.gguf what a GGUF really holds and whether Strata runs it (no download)
START-HERE.bat --data-dir E:\Strata-data keep the model files somewhere else
START-HERE.bat --port 8081 another port
START-HERE.bat --gpu 1 another GPU (setup picks the one with the most VRAM)
START-HERE.bat --gpus 0,2 several GPUs sharing the model
START-HERE.bat --vram-reserve-mib 2048 leave 2 GB of VRAM free for other programs (remembered)
START-HERE.bat --no-browser do not open the chat page when the model is ready (remembered;
--browser undoes it)
START-HERE.bat --setup --backend hip the AMD engine on a PC that also has an NVIDIA card
START-HERE.bat --setup --host 0.0.0.0 --api-key <secret> reachable from other devices, with a key
START-HERE.bat --calibrate tune the engine for this PC (about 15-30 minutes, longer on a slow card), then start
START-HERE.bat --check only check this PC
```
**Leaving VRAM for other programs (#493):** Strata fills the graphics card's free VRAM with experts (the expert cache)
and leaves `--vram-reserve-mib` MiB free: 700 by default. For a game, a 3D program or another model beside it, leave
more: `START-HERE.bat --vram-reserve-mib 2048` (Linux: `./setup.sh --vram-reserve-mib 2048`) writes it into the
model's `strata-<model>.json` and starts it; at setup (`--setup --vram-reserve-mib 2048`) it goes into the new config.
By hand: add `"--vram-reserve-mib", "2048"` to the config's `"args"` list and restart. The expert cache is then that
much smaller, so answers can be a little slower. The engine sizes its cache from the VRAM free when it starts: what
another program already holds then is left alone anyway; the reserve is room for what it takes later.
**An AMD card that also drives a Linux desktop (#560 #516):** with the default reserve the expert cache fills the
card, and when the desktop (the compositor, a browser, a new app) needs more VRAM, the amdgpu driver moves GPU memory
to system RAM, which the model's experts already fill: the OOM killer then ends the desktop session (KWin, plasmashell)
or the compositor fails ("Failed to pin framebuffer"). `./setup.sh --vram-reserve-mib 3072` fixed it in both reports
(about 2.3 GB fewer experts in VRAM, a few % of speed). Setup and the server window say so on such a PC.
**A card under 8 GB (#496):** when the default reserve leaves the expert cache too little room, the engine lowers the
reserve (down to 300 MiB) until it fits, and warns if the card then ends nearly full. If the start stops with "no VRAM
is left for the expert cache", that log line says how much is short; an 8K context and `--draft-vocab en` at setup
free the most.
**Model files downloaded by hand, or from a mirror (#495):** setup's step 5 prints the folder it expects them in
(`Strata-data\models\<SIZE>\`, e.g. `Strata-data\models\IQ3_XXS\`): put them there with their original names, or
point setup at them with `--gguf-dir`. To see what a file really holds before using or downloading it, `--inspect`
reads only its headers (a few MB, also over the network: a file, a folder, a URL, `ms:owner/repo` for ModelScope or
`hf:owner/repo`): every weight group's real bits per weight and storage types, and whether it is one of the files setup
installs (under any name), stored like one of them, or not one Strata runs (another architecture, or experts in a
format no Strata kernel reads). A name says little: Unsloth's `UD-IQ3_XXS` stores its routed experts at 3.22 bits per
weight, `UD-Q4_K_XL` at 5.10, the original `Q2_0` at 2.25 and `IQ3_S` at 3.33. To let setup download from a Hugging Face mirror itself, set `HF_ENDPOINT`
first (Windows: `set HF_ENDPOINT=https://hf-mirror.com`, Linux: `export HF_ENDPOINT=https://hf-mirror.com`): the same
pinned revisions and checks apply, and the MTP draft layer comes from there too.
**From ModelScope (mainland China):** every repository setup downloads from (the GSQ-RCO models, Swift 1.5, the
Coder, Unsloth's files, and the original checkpoint the MTP draft layer is taken from) is on
[ModelScope](https://www.modelscope.cn) under the same name, with the same files. Setup never switches to it by
itself (a download that fails says how): use `--source modelscope` (`STRATA_SOURCE=modelscope`). ModelScope serves a
repository's current files (no pinned revision), so each file is checked against the SHA-256 ModelScope publishes for
it (self-attested: a changed upstream repository would carry its own hash), and the MTP tensors against the
pinned checkpoint's own hashes, as from Hugging Face; setup says so when you choose it. Measured on a PC in mainland
China on 2026-10-05: 11-14 MB/s; the Q2_0 and IQ3_S files matched their published SHA-256, and Q2_0's shard 1 matched
the copy from hf-mirror.com.
On Linux the same options go to `./setup.sh`. `START-HERE.bat --help` lists them all. The server's own settings
(sharing the GPU with games, MCP tools, CORS, API keys, the API itself) are in the [details](DETAILS.md#using-it).
- [Docker](DOCKER.md)
- [Build or run from a source checkout](SOURCE_INSTALL.md)
- [Multiple GPUs](MULTI_GPU.md)
- [Troubleshooting and logs](TROUBLESHOOTING.md)
- [Model compatibility](MODELS.md)
docs/INTEL.md
+419 -932
View File
File diff suppressed because it is too large. Load diff
docs/INTEL_ARC.md
+117 -117
View File
@@ -1,97 +1,74 @@
# Intel Arc (experimental)
# Intel Arc on Linux
Strata 0.1.39 includes an **experimental Intel Arc engine**: Strata's own engine ported to SYCL (Intel oneAPI).
maxfridbe wrote it in [#423](https://github.com/Niko1221/Strata/pull/423), with fixes from the people testing it.
The code is in `sycl/`, and the port's own notes, measurements and maintenance procedure are in
[docs/INTEL.md](INTEL.md). This page covers what you need, how to build it, and what has been tested.
Strata runs on Intel Arc cards on Linux through its own engine ported to SYCL (Intel oneAPI). This page is for people with
an Arc A-series (Alchemist) or B-series (Battlemage) card. It covers the released Terminal route, what you need, how to
build the engine from source, and which cards were tested. The port's workflow and measurements are in [INTEL.md](INTEL.md).
**Experimental means:** the Intel engine is still young and Linux only. When the port arrived (0.1.39) the maintainers had no
Intel GPU and every result on Arc hardware came from the community. Since 0.1.40.2 the maintainers also test on an Arc Pro B70
(32 GB, `xe`) and an Arc A750 (8 GB, `i915`); the results from those two cards are marked in the table below. Other cards
are still community reports. The NVIDIA and AMD engines are unchanged: the Intel build is a separate CMake target, off by default.
maxfridbe wrote the port in [#423](https://github.com/Niko1221/Strata/pull/423). The code is in `sycl/`. Intel Arc is an
official platform since 0.1.40.2. The maintainers test on an Arc Pro B70 (32 GB, `xe` driver) and an Arc A750 (8 GB, `i915`
driver); results from other cards are community reports. The Mac and Windows builds have no Intel engine.
**Where the engine comes from.** On Linux, `strata update` (Terminal) and Settings > Update (app) download the Intel engine package, `linux-x86_64-sycl`, for a PC whose graphics card is made by Intel, when the release feed lists one (`strata update --check` shows what it would install). One package serves Alchemist (A-series) and Battlemage (B-series) cards: the GPU code is compiled from SPIR-V at the first start, because the A-series cards have no FP64 units and the engine's FP64 kernels cannot be built ahead of time for them. It carries oneAPI's runtime libraries; the Level Zero loader and the compute runtime stay the PC's own (see "What you need" below), and the user must be in the `render` group (`strata doctor` says so). The package is built by `packaging/build_sycl_linux.sh`. It was tested on an Arc A750 only. The release zips (the ones `START-HERE.bat` and `setup.sh` use) have no Intel engine; to build one from source, follow the steps below. The Docker route is a ready-made image, `ghcr.io/niko1221/strata:0.2.3-sycl`, which is published only for a release that carries a Linux Intel engine package; with none, [build the image yourself](DOCKER.md#build-it-yourself) from the engine you built (the image needs prepared model files; see [Docker: Intel Arc](DOCKER.md#intel-arc)). The Mac and Windows builds have no Intel engine.
## Quick start
## What has been run, and by whom
Install the [Terminal](INSTALL.md#the-terminal), then:
| | Hardware | Result | Reported in |
|---|---|---|---|
| Port author | Arc Pro B70 32 GB, Ubuntu 24.04 | Coder IQ1_M: 70-78 tok/s decode, ~790 tok/s prompt; IQ2_XS: 51-64 tok/s; 256K context measured | #423, [INTEL.md](INTEL.md) |
| Community | 2x Arc Pro B70, `--layer-split` | Flash-Next IQ3_XXS 66 tok/s decode, 394 tok/s prompt (with the `stage_room` fix that is now in 0.1.39) | #423 |
| Community | Arc Pro B50 16 GB | Coder IQ1_M ~23 tok/s, IQ2_XS ~25-27 tok/s, up to 128K | #423 |
| Community | Arc B580 12 GB, WSL2 | IQ2_XS ~21 tok/s, Coder ~15 tok/s; **device loss also seen** | #423 |
| Community | 2x Arc Pro B60 24 GB, `--layer-split` | Coder IQ1_M: 56.6 tok/s decode, 428 tok/s prompt (2,129 tokens); Flash-Next IQ2_XS: 58.6-61.3 tok/s decode, 436 tok/s prompt; all experts in VRAM; 8K context | [bench/results/2026-10-04-community-2x-arc-pro-b60](../bench/results/2026-10-04-community-2x-arc-pro-b60/README.md) |
| Community | one Arc Pro B60 24 GB | Coder IQ1_M with 4,042 of 12,288 experts mirrored in RAM: 11.9 tok/s decode (needs the ring-wait fix from the same report) | same |
| Strata maintainers (0.1.40.2 and later) | Arc Pro B70 32 GB (xe) and Arc A750 8 GB (i915), test machines | Flash-Next IQ3_S on the B70 with 11.5k of 24.6k experts resident: 10-prompt gate passes, prompt 980-1,000 tok/s on 4K tokens, decode 30 (prose) to 41 (code) tok/s; the xe aliased-pages fix and the `NO_HOST` requirement are in [INTEL.md](INTEL.md) | INTEL.md, "Arc Pro B70 with a model that does not fit" |
```sh
strata update --check # shows the Intel engine package the feed would install
strata doctor --live # checks the driver, the render group and the card
strata launch --plan # shows what would happen, changes nothing
strata launch
```
The 0.1.39 port re-migrates 0.1.38's port onto the 0.1.39 engine sources (the #606 NaN fix, the #649 verify
trace, the new prompt paths). 0.1.39's `sycl/` did not compile against 0.1.39's own engine sources (#784: the
`ThreadAffinity` type of #626 and the layer range of `NativeDense::load` from #559), and its ring waits never saw a
slow layer as still running (#866, #867). Both are fixed in 0.1.40, and the two B60 reports below were measured with
exactly those two fixes on top of 0.1.39. **No one has run an unpatched 0.1.39 port on an Arc**, and the other rows were
measured on earlier versions.
On Linux, `strata update` (Terminal) and Settings > Update (app) download the Intel engine package `linux-x86_64-sycl` for a PC
whose graphics card is made by Intel. One package serves Alchemist and Battlemage: the GPU code is compiled from SPIR-V at the
first start, because the A-series cards have no FP64 units. The package carries oneAPI's runtime libraries. The Level Zero
loader and the compute runtime stay the PC's own (see [What you need](#what-you-need)), and your user must be in the `render`
group (`strata doctor` says so). It was tested on an Arc A750 only. The package is built by `packaging/build_sycl_linux.sh`.
The two B60 rows ran `6f32ec0` plus two small `sycl/` fixes (the compile fix and the ring-wait fix), AOT `bmg-g21`, on Ubuntu 24.04 with
`xe`, Level Zero V2, NEO 26.09.37435.12 and oneAPI 2026.1.1, without Docker. Host: Ryzen 5 5600, 64 GB RAM, PCIe 3.0 x8 per card.
Full flags, per-request timings and engine logs are in the report linked in the table.
**Model preparation is separate.** `strata launch` reuses compatible prepared local files, but it does not run the Intel
source or container setup for you. When a model needs preparation, follow the [source workflow](INTEL.md#setup). Installing the
engine package alone does not prove the selected model is ready; check a loaded server with a real request.
## What was tested here (0.1.39)
- **Compiles:** Ubuntu 24.04 (WSL2), Intel oneAPI DPC++ 2026.1.1 + oneMKL 2026.1. The whole `sycl/` project
builds with 0 errors: the `strata` engine plus all 157 targets (the kernel parity tests and benches). The build is
SPIR-V (JIT). The AOT build (`STRATA_SYCL_AOT`) was not built here, because it needs `ocloc`.
- **Kernel parity tests on a CPU** (`ONEAPI_DEVICE_SELECTOR=opencl:cpu`, Intel's OpenCL CPU runtime, AMD Ryzen 5 7600):
14 of 25 pass. That is the same set that PR #423's own 0.1.38 port passes on that device: the failures are tight
float tolerances on the CPU's math (rel 3e-6 against a 1e-6 limit), model fixtures that are not present, and two
tests that time out on a CPU. This is a check that the kernels compile and compute, not a test of an Arc.
- **Setup:** `--backend sycl` warns, then hands over to `sycl/setup_intel.py` on Linux (unit tests:
`tools/test_setup_sycl.py`).
- **Unchanged:** the CUDA engine's greedy output, checked byte-identical against the gated 0.1.39 build (Q2_0 and Coder).
The HIP build is not affected (the option is off by default).
**Not tested by anyone yet:** Windows (no native build path; see below), Arc on WSL2 for the 0.1.39 port, integrated Arc
GPUs (the B390 / Panther Lake in #515; the shared-memory planning does not exist yet), and pictures on a Battlemage card
(the first picture on the Arc Pro B70 reset the GPU; they are off by default there, [INTEL.md](INTEL.md#pictures-on-an-arc-2026-10-19-branch-f-021-arcvis)).
Pictures work on the A750 (on by default, read on the CPU). The Alchemist A-series was tested later, on an A750 (rows above and
[INTEL.md](INTEL.md), "Arc A750 and the other Alchemist cards").
Docker: the ready-made image is `ghcr.io/niko1221/strata:0.2.3-sycl`, published only for a release that carries a Linux Intel
engine package. Otherwise [build the image yourself](DOCKER.md#build-it-yourself). The image needs prepared model files
([Docker: Intel Arc](DOCKER.md#intel-arc)).
## What you need
- **Linux**, e.g. Ubuntu 24.04, with Intel's GPU driver (the `xe` or `i915` kernel driver plus the compute runtime /
Level Zero; on Ubuntu, `intel-opencl-icd libze1 libze-intel-gpu1`, or Intel's
[client GPU guide](https://dgpu-docs.intel.com/driver/client/overview.html)).
- **Intel oneAPI**: the DPC++ compiler (`icpx`, 2025.3 or newer; 2026.1 is what was built here) and **oneMKL**. About 5 GB.
- `cmake` 3.24+, `ninja`, `git` (the build fetches ggml unless you point `STRATA_GGML_DIR` at a llama.cpp checkout),
Python 3.
- For `./setup.sh --backend sycl` today: **Docker**, and nothing else from the list above except the GPU driver.
`sycl/setup_intel.py` runs the engine in the `strata-sycl-dev` image built from `sycl/tools/Dockerfile`, and the
engine is built in that image too ("Build with Docker" below). On Ubuntu: `sudo apt install docker.io git`, then
`sudo usermod -aG docker,render $USER` and log in again. Note that the Dockerfile starts from a community
llama.cpp SYCL image (`ghcr.io/snailium/...`), not an Intel or Strata image.
- VRAM: the port keeps the experts on the card (`--stream-experts`). A 32 GB card holds the Coder IQ1_M or IQ2_XS.
Smaller cards mirror part of the experts in RAM and are slower.
- **Linux**, for example Ubuntu 24.04, with Intel's GPU driver: the `xe` or `i915` kernel driver plus the compute runtime and
Level Zero. On Ubuntu: `intel-opencl-icd libze1 libze-intel-gpu1`, or follow Intel's
[client GPU guide](https://dgpu-docs.intel.com/driver/client/overview.html).
- For a source build without Docker: **Intel oneAPI** with the DPC++ compiler (`icpx`, 2025.3 or newer) and **oneMKL**
(about 5 GB), `cmake` 3.24+, `ninja`, `git`, Python 3. The build fetches ggml unless you point `STRATA_GGML_DIR` at a
llama.cpp checkout.
- For `./setup.sh --backend sycl`: **Docker** and the GPU driver, nothing else from the list above. `sycl/setup_intel.py`
builds and runs the engine in the `strata-sycl-dev` image from `sycl/tools/Dockerfile`, which starts from a community
llama.cpp SYCL image (`ghcr.io/snailium/...`), not an Intel or Strata image. On Ubuntu:
`sudo apt install docker.io git`, then `sudo usermod -aG docker,render $USER` and log in again.
- VRAM: the port keeps the experts on the card (`--stream-experts`). A 32 GB card holds the Coder IQ1_M or IQ2_XS. Smaller
cards mirror part of the experts in RAM and are slower.
## Build with Docker (what `setup --backend sycl` uses)
Two commands, the same as in [INTEL.md](INTEL.md), "How to build it" (about 10 minutes the first time). From the checkout,
called `Strata` here:
About 10 minutes the first time. From the checkout, called `Strata` here:
```sh
cd Strata
docker build --pull -t strata-sycl-dev sycl/tools # once: ~6 minutes, a 13 GB image (--pull: INTEL.md says why)
docker build --pull -t strata-sycl-dev sycl/tools # once: about 6 minutes, a 13 GB image
H=$(basename "$PWD")
# Arc B-series: bmg-g31 (Pro B70), bmg-g21 (B580, B570, Pro B60). Arc A-series (Alchemist): leave out AOT (JIT) and use build-sycl
docker run --rm -u $(id -u):$(id -g) -v "$PWD/..:/work" -e AOT=bmg-g31 -e REPO=/work/$H \
-e BUILD_DIR=/work/$H/build-sycl-aot strata-sycl-dev "cd /work/$H && bash sycl/tools/build.sh strata"
```
Then `./setup.sh --backend sycl` (below). Do not run `python3 sycl/setup_intel.py` by hand: on Ubuntu it fails with
`externally-managed-environment`; `setup.sh` runs it in its own `.venv`.
`--pull` matters: [INTEL.md](INTEL.md#how-to-build-it) says why. Then run `./setup.sh --backend sycl`. Do not run
`python3 sycl/setup_intel.py` by hand: on Ubuntu it fails with `externally-managed-environment`; `setup.sh` runs it in its own
`.venv`.
## Build without Docker (Linux)
## Build without Docker
This builds the engine for running it by hand ("How to run it by hand" in [INTEL.md](INTEL.md)); `setup` itself uses the
Docker build above. Install oneAPI from Intel's apt repository (this is what was used here):
This builds the engine for running it by hand ([INTEL.md](INTEL.md#how-to-run-it-by-hand)). `setup` itself uses the Docker build.
Install oneAPI from Intel's apt repository:
```sh
wget -qO- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB \
@@ -102,8 +79,7 @@ sudo apt update
sudo apt install intel-oneapi-compiler-dpcpp-cpp intel-oneapi-mkl-devel ninja-build cmake git
```
Then build the engine. Either command works: the first goes through the top-level CMake option, the second
configures the `sycl/` project directly.
Build the engine through the top-level CMake option or by configuring the `sycl/` project directly:
```sh
source /opt/intel/oneapi/setvars.sh
@@ -112,16 +88,14 @@ cmake -S . -B build-sycl -G Ninja -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=ic
cmake --build build-sycl --target strata
```
Options:
| Option | Effect |
| --- | --- |
| `-DSTRATA_SYCL_AOT=bmg-g31` | Compile ahead of time for Arc Pro B70. Needs `ocloc` (Intel's `intel-ocloc` package). Without it the first start JIT-compiles every kernel (about 47 s). |
| `-DSTRATA_SYCL_AOT=bmg-g21` | The same for B580, B570 and Pro B60 |
| `-DSTRATA_SYCL_PARITY=OFF` | Skip the kernel tests (on by default). Run them with `ctest --test-dir build-sycl` on the card. |
- `-DSTRATA_SYCL_AOT=bmg-g31` (Arc Pro B70) or `bmg-g21` (B580 / B570 / Pro B60) compiles the GPU code ahead of
time. This needs `ocloc` (Intel's `intel-ocloc` package). Without it, the first start JIT-compiles every kernel,
which takes about 47 s.
- `-DSTRATA_SYCL_PARITY=OFF` skips the kernel tests (on by default). Run them with
`ctest --test-dir build-sycl` on the card.
`setup_intel.py` looks for `build-sycl-aot/strata` or `build-sycl/strata` in the checkout. With the
top-level option the engine is at `build-sycl/sycl/strata`, so either use `-S sycl` or set `STRATA_SYCL_BIN`.
`setup_intel.py` looks for `build-sycl-aot/strata` or `build-sycl/strata` in the checkout. With the top-level option the
engine is at `build-sycl/sycl/strata`, so use `-S sycl` or set `STRATA_SYCL_BIN`.
## Setup and running
@@ -129,50 +103,76 @@ top-level option the engine is at `build-sycl/sycl/strata`, so either use `-S sy
./setup.sh --backend sycl [setup's usual options, e.g. --model IQ2_XS --context 32768]
```
This prints the experimental warning and continues with `sycl/setup_intel.py` (in setup's own Python environment).
That script finds the Arc in sysfs, uses the SYCL engine you built, and writes the config and `run-<model>.sh`.
On a card under 12 GB or an `i915` card (the A-series) it writes the small-card config ([INTEL.md](INTEL.md), "Arc A750
and the other Alchemist cards"). It still downloads and packs the model
the usual way. To run the engine by hand (no Docker), see "How to run it by hand" in [INTEL.md](INTEL.md).
Setup continues with `sycl/setup_intel.py`, which finds the Arc in sysfs, uses the SYCL engine you built, and writes the
config and `run-<model>.sh`. On a card under 12 GB or an `i915` card (the A-series) it writes the small-card config
([INTEL.md](INTEL.md#arc-a750-and-the-other-alchemist-cards)). It downloads and packs the model the usual way. Windows has no
path: `setup --backend sycl` on Windows stops and points here.
Things that matter on an Arc (details in INTEL.md):
Things that matter on an Arc (details in [INTEL.md](INTEL.md)):
- **Do not ask for more VRAM than the card has.** On the `xe` driver, an allocation past VRAM can push buffers into
RAM until the machine runs out of memory and stalls. Leave about 1.5 GB free.
- **Do not ask for more VRAM than the card has.** On the `xe` driver an allocation past VRAM can push buffers into RAM until
the machine runs out of memory and stalls. Leave about 1.5 GB free.
- `SYCL_CACHE_PERSISTENT=0`: the persistent JIT cache crashed on Xe2 during the first compile.
- Two cards: `ONEAPI_DEVICE_SELECTOR=level_zero:*` (the image pins `level_zero:0`; `strata-sycl.sh` now passes the
variable through) and `--layer-split`.
- **`--layer-split auto` on two equal cards (0.1.44, #1440):** when several splits hold all the experts and predict the
same speed, the search now takes the one that leaves its fullest card the most VRAM to spare (within 0.2% of the best
predicted time, `STRATA_SPLIT_HEADROOM_TOL`). It used to take the first one tried, which filled the second card to its
last MiB. A full card under `xe` moves buffers to system memory instead of failing: 2x B70 with IQ3_S, auto picked
K=24 (~11 tok/s) where `--layer-split 30` ran ~54. The startup line names both splits when the choice moves.
`STRATA_SPLIT_HEADROOM=0` keeps the old pick.
- **Arc Pro B60:** the PCI id is `8086:e211` (`lspci -nn`, the kernel's `xe` id list files it with the BMG-G21 cards), `sycl-ls` prints
`Intel(R) Arc(TM) Pro B60 Graphics 20.1.0`, and `ocloc ids bmg-g21` prints 20.1.0, so `-DSTRATA_SYCL_AOT=bmg-g21` is the right target.
`setup_intel.py` knows both B60 ids, `e211` and `e221` (both seen on B60 cards). On a `--layer-split` the startup line "N experts are neither in VRAM nor mirrored"
counts the other card's layers; the lines `100% of the experts resident` that follow are the ones to read. Set `STRATA_MIRROR_MIB=0` there,
or the pinned mirror is allocated and not used.
- **`setvars.sh` and `set -u`:** `source /opt/intel/oneapi/setvars.sh` in a shell with `set -u` stops at `OCL_ICD_FILENAMES: unbound variable`
(oneAPI 2026.1.1). Source it before `set -u`, or run `set +u` around it.
- Two cards: `ONEAPI_DEVICE_SELECTOR=level_zero:*` (the image pins `level_zero:0`; `strata-sycl.sh` passes the variable
through) and `--layer-split`.
- `--layer-split auto` on two equal cards takes the split that leaves the fullest card the most VRAM to spare, within 0.2% of
the best predicted time (`STRATA_SPLIT_HEADROOM_TOL`). `STRATA_SPLIT_HEADROOM=0` turns that off. A full card under `xe` moves
buffers to system memory instead of failing. The startup line names both splits when the choice moves.
- **Arc Pro B60:** PCI id `8086:e211` (`lspci -nn`), `sycl-ls` prints `Intel(R) Arc(TM) Pro B60 Graphics 20.1.0`, so
`-DSTRATA_SYCL_AOT=bmg-g21` is the right target. `setup_intel.py` knows both B60 ids, `e211` and `e221`. On a `--layer-split`
the startup line "N experts are neither in VRAM nor mirrored" counts the other card's layers; read the following
`100% of the experts resident` lines. Set `STRATA_MIRROR_MIB=0` there, or the pinned mirror is allocated and not used.
- **`setvars.sh` and `set -u`:** `source /opt/intel/oneapi/setvars.sh` in a shell with `set -u` stops at
`OCL_ICD_FILENAMES: unbound variable` (oneAPI 2026.1.1). Source it before `set -u`.
- **Without Docker:** `sycl/serve/strata-sycl.sh` needs Docker. Natively, `source setvars.sh`, export `SYCL_CACHE_PERSISTENT=0`,
`ZES_ENABLE_SYSMAN=1` and `STRATA_VERIFY_DEVICE_PLAN=1`, then run `build-sycl-aot/strata` or `sycl/serve/server_intel.py` with an `exe` that does this.
A normal user is not in the `render` group on Ubuntu, and then `sycl-ls` shows only the CPU device.
- `STRATA_VERIFY_NO_HOST=1` (set by `strata-sycl.sh` unless the config's `"env"` says otherwise) is needed on `xe`
cards when part of the experts is not in VRAM (the host-to-GPU flag stores are not visible there, NaN logits
otherwise) and is left off on the `i915` Arc A-series: see [INTEL.md](INTEL.md). The config's `"env"` block and any
`STRATA_*`, `ONEAPI_*`, `UR_*`, `IGC_*`, `SYCL_*` or `ZES_*` variable of the server's environment reach the engine
through `strata-sycl.sh` (for example `STRATA_MIRROR_MIB`, `IGC_EnableDPEmulation`).
`ZES_ENABLE_SYSMAN=1` and `STRATA_VERIFY_DEVICE_PLAN=1`, then run `build-sycl-aot/strata` or `sycl/serve/server_intel.py`
with an `exe` that does this. A normal user is not in the `render` group on Ubuntu, and then `sycl-ls` shows only the CPU.
- `STRATA_VERIFY_NO_HOST=1` (set by `strata-sycl.sh` unless the config's `"env"` says otherwise) is needed on `xe` cards when part
of the experts is not in VRAM, and is left off on the `i915` A-series ([INTEL.md](INTEL.md)). The config's `"env"` block and
any `STRATA_*`, `ONEAPI_*`, `UR_*`, `IGC_*`, `SYCL_*` or `ZES_*` variable of the server's environment reach the engine
through `strata-sycl.sh`.
## Tested cards
> **Measured on:** the hardware in each row, Linux. Versions are the engine version of that report, not the current release.
| Who | Hardware | Result | Reported in |
| --- | --- | --- | --- |
| Port author | Arc Pro B70 32 GB, Ubuntu 24.04, 0.1.31-sycl | Coder IQ1_M: 70-78 tok/s decode, about 790 tok/s prompt; IQ2_XS: 51-64 tok/s; 256K context | #423, [INTEL.md](INTEL.md) |
| Community | 2x Arc Pro B70, `--layer-split` | Flash-Next IQ3_XXS 66 tok/s decode, 394 tok/s prompt | #423 |
| Community | Arc Pro B50 16 GB | Coder IQ1_M about 23 tok/s, IQ2_XS about 25-27 tok/s, up to 128K | #423 |
| Community | Arc B580 12 GB, WSL2 | IQ2_XS about 21 tok/s, Coder about 15 tok/s; device loss also seen | #423 |
| Community | 2x Arc Pro B60 24 GB, `--layer-split`, Ryzen 5 5600, 64 GB, PCIe 3.0 x8 per card | Coder IQ1_M: 56.6 tok/s decode, 428 tok/s prompt (2,129 tokens); Flash-Next IQ2_XS: 58.6-61.3 tok/s decode, 436 tok/s prompt; all experts in VRAM; 8K context | [report](../bench/results/2026-10-04-community-2x-arc-pro-b60/README.md) |
| Community | One Arc Pro B60 24 GB | Coder IQ1_M with 4,042 of 12,288 experts mirrored in RAM: 11.9 tok/s decode | same |
| Maintainers | Arc Pro B70 32 GB (xe), 0.1.40.2 and later | Flash-Next IQ3_S with 11.5k of 24.6k experts resident: 10-prompt gate passes, prompt 980-1,000 tok/s on 4K tokens, decode 30 (prose) to 41 (code) tok/s | [INTEL.md](INTEL.md#arc-pro-b70-with-a-model-that-does-not-fit-flash-next-iq3_s) |
| Maintainers | Arc A750 8 GB (i915) | Flash-Next IQ3_XXS about 10-19 tok/s decode | [INTEL.md](INTEL.md#arc-a750-and-the-other-alchemist-cards) |
Two cards on `xe` (2x Arc Pro B70, IQ3_S, 0.1.44): `--layer-split auto` picked a split at about 11 tok/s where
`--layer-split 30` ran about 54. The auto choice now prefers the split with the most VRAM headroom (above).
**Not tested by anyone:** Windows (no native build path), Arc on WSL2 with the current engine, integrated Arc GPUs (the B390 /
Panther Lake in #515; shared-memory planning does not exist), and pictures on a Battlemage card with a GPU reader. Pictures work
on the A750 (on by default, read on the CPU). See [INTEL.md](INTEL.md#pictures-on-an-arc).
## Windows
There is no Windows path yet. `setup --backend sycl` on Windows stops and points here. oneAPI exists for Windows,
but `sycl/CMakeLists.txt` uses GCC-style flags (`-mavx512f`, `-fp-model=precise`, `-qmkl`) and the runner is a
bash/Docker script, so a native Windows build would need work. Nobody has tried it. WSL2 with an Arc has been
used by one tester (the B580 row above), but setup cannot detect the card there, because it reads `/sys/class/drm`,
which WSL2 does not have.
There is no Windows path. `setup --backend sycl` on Windows stops and points here. oneAPI exists for Windows, but
`sycl/CMakeLists.txt` uses GCC-style flags (`-mavx512f`, `-fp-model=precise`, `-qmkl`) and the runner is a bash/Docker script.
Nobody has tried a native build. WSL2 with an Arc was used by one tester (the B580 row above), but setup cannot detect the card
there, because it reads `/sys/class/drm`, which WSL2 does not have.
## Reporting a problem
Open an issue with: the card, the driver version, `sycl-ls` output, the oneAPI version, the model and flags, and the
engine's stderr. Results from real cards are what move this from experimental to supported.
Open an issue with the card, the driver version, the `sycl-ls` output, the oneAPI version, the model and flags, and the engine's
stderr. Results from real cards are what move more cards from tested to validated.
## History / older measurements
The first port (0.1.39) compiled with 0 errors on Ubuntu 24.04 (WSL2) with oneAPI DPC++ 2026.1.1 and oneMKL 2026.1: the
`strata` engine plus 157 targets, as SPIR-V (JIT). Kernel parity tests on a CPU (`ONEAPI_DEVICE_SELECTOR=opencl:cpu`, Ryzen 5
7600) passed 14 of 25, the same set as the 0.1.38 port; the failures were tight float tolerances, missing model fixtures and two
CPU timeouts. That was a compile-and-compute check, not a test of an Arc. The CUDA engine's greedy output stayed byte-identical.
The two B60 rows ran commit `6f32ec0` plus two small `sycl/` fixes, AOT `bmg-g21`, Ubuntu 24.04, `xe`, Level Zero V2, NEO
26.09.37435.12 and oneAPI 2026.1.1, without Docker. No one ran an unpatched 0.1.39 port on an Arc; the other rows were measured on
earlier versions.
docs/LLAMA_SWAP.md
+51 -31
View File
@@ -1,55 +1,60 @@
# Running Strata behind llama-swap
[llama-swap](https://github.com/mostlygeek/llama-swap) puts several model servers behind one address and starts the one a
request names, stopping the others to free the GPU. Strata works as one of its models. This page is what we needed to
get it running; it was tested with llama-swap's default stop behaviour, Strata 0.1.39 (Coder IQ1_M on a Tesla V100) and
clients reaching the machine by its host name.
[llama-swap](https://github.com/mostlygeek/llama-swap) puts several model servers behind one address, starts the one a
request names and stops the others to free the GPU. This page is for people who already run llama-swap and want
Strata as one of its models. It needs a [source checkout](SOURCE_INSTALL.md), not the desktop installer.
## The entry
## Quick start
Set Strata up as usual first (`./setup.sh --no-start ...`), then point llama-swap at the server directly. The start script
setup writes (`run-<model>.sh`) fixes the port at 8080, so call `serve/server.py` with llama-swap's port instead:
1. Set up Strata without starting it: `./setup.sh --no-start`. This writes `strata-<model>.json` in the checkout.
2. Add an entry to llama-swap's config. The start script that setup writes (`run-<model>.sh`) fixes the port at 8080,
so call `serve/server.py` with llama-swap's port instead:
```yaml
healthCheckTimeout: 300 # Strata takes ~1.5-2 min to load; give it room
healthCheckTimeout: 300 # model loading takes minutes; give it room
models:
strata:
cmd: >-
sh -c 'cd /path/to/Strata && exec .venv/bin/python serve/server.py
--engine strata --config strata-coder-iq1_m.json --port ${PORT}'
env: ["STRATA_ALLOWED_HOSTS=my-server"] # the host name your clients use -- see below
--engine strata --config strata-<model>.json --port ${PORT}'
env: ["STRATA_ALLOWED_HOSTS=my-server"] # the host name your clients use, see below
checkEndpoint: /health
ttl: 1800
```
- `exec` makes the Python server the process llama-swap stops. On llama-swap's stop (SIGTERM) the server stops its engine:
in our test the engine was gone and the V100's memory freed within 2 s.
- `checkEndpoint: /health` -- Strata's `/health` answers only once the model is loaded. Measured load time on our machine
(model files on a SATA SSD): 80 s, 105 s while another download was using the disk.
- If other models share the GPU, put them in a llama-swap group or let swapping unload them; Strata needs the card's
memory to itself.
3. If clients reach llama-swap by a host name, keep the `STRATA_ALLOWED_HOSTS` line and set it to that name
([Host not allowed](#host-is-not-allowed)).
## The trap: "Host ... is not allowed"
## The entry
The first request through llama-swap failed at once with:
- `exec` makes the Python server the process llama-swap stops. On SIGTERM the server stops its engine and the
GPU memory is freed.
- `checkEndpoint: /health` works because Strata's `/health` answers only once the model is loaded.
- Strata needs the card's memory to itself. If other models share the GPU, put them in a llama-swap group or let
swapping unload them.
## Host is not allowed
A client that reaches the proxy by name (`http://my-server:8040`) gets:
```text
403 Host 'my-server:8040' is not allowed (DNS rebinding protection)
```
Strata only answers to the host names it knows, so a web page cannot reach it by pointing a DNS name at 127.0.0.1.
llama-swap passes the client's Host header through, so a client that reaches the proxy by name (`http://my-server:8040`)
is refused. Requests by IP address and to `localhost` always pass, which is why it can work in a quick test and fail
from another machine. Any one of these fixes it:
llama-swap passes the client's Host header through. Requests by IP address and to `localhost` always pass, so a quick
local test can work while the same setup fails from another machine. Any one of these fixes it:
- `STRATA_ALLOWED_HOSTS=my-server` in the entry's `env` (comma-separated names; ports are ignored), or
- `"allowed_hosts": ["my-server"]` in `strata-<model>.json`, or
- an `"api_key"`, which turns the check off -- then every client must send the key.
| Fix | Notes |
| --- | --- |
| `STRATA_ALLOWED_HOSTS=my-server` in the entry's `env` | Comma-separated names; ports are ignored. |
| `"allowed_hosts": ["my-server"]` in `strata-<model>.json` | Same effect, stored in the config. |
| An `"api_key"` in the config | Turns the host check off. Every client must then send the key. |
## Strata's own pages through llama-swap
llama-swap forwards `/upstream/<model>/...` unchanged, so Strata's status and health pages stay reachable, and calling one
llama-swap forwards `/upstream/<model>/...` unchanged. Strata's status and health pages stay reachable, and calling one
also loads the model without sending a chat request:
```bash
@@ -57,10 +62,25 @@ curl http://my-server:8040/upstream/strata/health
curl http://my-server:8040/upstream/strata/v1/status
```
## Claude Code and other Anthropic-API clients: the thinking level
## Claude Code and other Anthropic-API clients
Claude Code (and the Claude Agent SDK) send `"output_config": {"effort": "high"}` with every request unless told
otherwise, so Strata thinks at its highest level (seen by logging the requests). `CLAUDE_CODE_EFFORT_LEVEL=medium` (or
`low`) in the client's environment changes it. One measurement, for scale only: on one real coding task in our setup,
`medium` finished in 6.5 min against 9.7 min at `high`, both correct; `low` took 10.1 min (more trial-and-fix steps).
One run each -- not a general result.
Claude Code and the Claude Agent SDK send `"output_config": {"effort": "high"}` with every request unless told
otherwise, so Strata thinks at its highest level. Set `CLAUDE_CODE_EFFORT_LEVEL=medium` (or `low`) in the client's
environment to change it.
## History / older measurements
> **Measured on:** Tesla V100, Coder IQ1_M, Strata 0.1.39, llama-swap default stop behaviour, model files on a SATA SSD.
| Item | Result |
| --- | --- |
| Model load time | 80 s; 105 s while another download used the disk |
| Engine gone and V100 memory freed after llama-swap's stop | within 2 s |
> **Measured on:** one real coding task, Strata 0.1.39 setup, one run per level. This is not a general result.
| `CLAUDE_CODE_EFFORT_LEVEL` | Time to a correct result |
| --- | --- |
| `medium` | 6.5 min |
| `high` | 9.7 min |
| `low` | 10.1 min (more trial-and-fix steps) |
docs/MAC.md
+241 -222
View File
@@ -1,301 +1,320 @@
# Strata on a Mac (Apple Silicon, Metal)
Strata runs on Apple Silicon Macs through its own engine on Metal. This is the same engine as on a PC. The NVIDIA
kernels are replaced by Metal ones (`metal/`), and the host code and the server are the PC's. There are two ways in:
This page is for people with an Apple Silicon Mac (M1 or newer). Strata runs there on its own Metal engine: the NVIDIA kernels
are replaced by Metal ones (`metal/`), and the host code and the server are the PC's. Intel Macs cannot run it.
- **The Strata app (`.dmg`) and the Terminal (`strata launch`).** The Metal engine comes ready-made: the `.dmg` has it
inside, and the Terminal fetches it as an engine component. Nothing is built on your Mac and you need no Command Line
Tools ([below](#the-desktop-app-and-strata-launch-on-a-mac)). macOS 15 or newer, as for the source build. If macOS says the
app is "damaged", see [Troubleshooting](TROUBLESHOOTING.md#the-desktop-app-and-the-terminal); how the builds are signed:
DeepSeek V4.1-Flash does not run on a Mac yet. It runs in the Terminal on NVIDIA RTX 30 and 50 cards and on the Radeon AI PRO
R9700 ([MODELS.md](MODELS.md#deepseek-v41-flash)).
> **On this page:** [What you need](#what-you-need) · [Install](#install) · [Defaults](#defaults-and-options) ·
> [Gyro](#gyro-s-and-gyro-m-agentionais-trellis-experts) · [Measured](#measured-m5-max-128-gb) · [Limits](#limits-this-version) ·
> [Switches](#switches) · [How it works](#how-it-works) · [History](#history--older-measurements)
## Quick start
There are two ways in:
- **The Strata app (`.dmg`) and the Terminal (`strata launch`).** The Metal engine comes ready-made: the `.dmg` has it inside and
the Terminal fetches it as an engine component. Nothing is built on your Mac and you need no Command Line Tools. If macOS says
the app is "damaged", see [Troubleshooting](TROUBLESHOOTING.md#the-desktop-app-and-the-terminal). How the builds are signed:
[packaging/mac/README.md](../packaging/mac/README.md).
- **From source: `./setup.sh`.** It sets things up the way `START-HERE.bat` does on Windows and builds the engine on your Mac
(about 1-15 minutes).
(about 1 to 15 minutes).
**Pictures:** on by default in the app, the Terminal and the source build. The reader (`strata-vision`) comes in the engine package
and runs on the Mac's GPU through Metal ([Limits](#limits-this-version)). The NVIDIA, AMD and Intel engines are unchanged.
Pictures are on by default in the app, the Terminal and the source build ([Limits](#limits-this-version)). The NVIDIA, AMD and
Intel engines are unchanged.
**Status:** new. It is measured on three Macs: an M5 Max with 128 GB, where the model stays in memory; and an M4 Mac mini
with 32 GB and an M1 Max with 34 GB, where the experts stream from the SSD (the M1 Max: 7.9 tok/s decode, 186 tok/s on a
9,940-token prompt, Q2_0, the expert cache at 64 slots, as reported with the kernel fix #2348). Other chips are untested, and reports help
**Status:** new. It was measured on three Macs: an M5 Max with 128 GB, where the model stays in memory, and an M4 Mac mini with
32 GB and an M1 Max with 34 GB, where the experts stream from the SSD. Other chips are untested, and reports help
([COMMUNITY_BENCHMARKS.md](COMMUNITY_BENCHMARKS.md)).
## What you need
- **An Apple Silicon Mac (M1 or newer).** Intel Macs cannot run it.
- **An Apple Silicon Mac (M1 or newer).**
- **macOS 15 or newer.** The engine keeps its buffers resident through a residency set, which macOS 15 introduced.
- **The Command Line Tools:** `xcode-select --install`. That is the compiler. Xcode itself is not needed.
- **Enough unified memory for the pack.** The GPU maps the model's main file and keeps it in memory while the model runs. Metal lets one
program keep about 3/4 of the memory resident (107.5 GB of 128 GB, measured on the M5 Max).
- **For a source build only:** Command Line Tools (`xcode-select --install`).
- **Enough unified memory for the pack.** The GPU maps the model's main file and keeps it in memory while the model runs. Metal
lets one program keep about 3/4 of the memory resident (107.5 GB of 128 GB on the M5 Max).
| Pack | The file the GPU maps | Needs about (with the prompt buffers, KV cache, draft layer) | Recommended from |
|---|---|---|---|
| Pack | The file the GPU maps | Needs about (with prompt buffers, KV, draft layer) | Recommended from |
| --- | --- | --- | --- |
| Q2_0 | 37.6 GB | 46 GB | 64 GB |
| IQ3_XXS | 47.0 GB | 55 GB | 74 GB (where IQ3_S does not fit) |
| IQ3_S (recommended; the default from 96 GB) | 54.8 GB | 63 GB | 96 GB |
| UD-Q4_K_XL (Unsloth, recommended: higher quality) | 82.6 GB | 91 GB | 128 GB |
| UD-Q5_K_XL, UD-Q6_K_XL (Unsloth) | 103.9 / 114.8 GB | stream from the SSD on 128 GB | 128 GB, streamed |
| Gyro-S (agentionai, one 58.5 GB file) | 29.7 GB (the file's other 28.8 GB is the n-gram table) | 38 GB | 48 GB; streams below that (measured on 32 GB) |
| Gyro-M (agentionai, one 92.0 GB file) | 63.2 GB (+ the 28.8 GB table) | 71 GB | 96 GB; streams below that (not measured on a Mac) |
| UD-Q4_K_XL (Unsloth; higher quality) | 82.6 GB | 91 GB | 128 GB |
| UD-Q5_K_XL, UD-Q6_K_XL (Unsloth) | 103.9 / 114.8 GB | streams from the SSD on 128 GB | 128 GB, streamed |
| Gyro-S (agentionai, one 58.5 GB file) | 29.7 GB (the other 28.8 GB is the n-gram table) | 38 GB | 48 GB; streams below that |
| Gyro-M (agentionai, one 92.0 GB file) | 63.2 GB (+ the 28.8 GB table) | 71 GB | 96 GB; streams below that |
- **The n-gram table:** each pack also has a 28.8 GB n-gram table (the model's second file). It is read from the SSD a few rows at a time and does not count against the memory above.
- **Disk:** the download is 66-84 GB, plus about 7 GB for the pack and the draft layer. A Mac that streams also needs room for the experts' copy (below): 34-40 GB.
- **A Mac whose memory cannot hold the pack (32-48 GB):** the engine streams the experts from the SSD instead. It decides this by itself when the experts are more than 60% of Metal's share.
- It keeps the most-used experts in an arena: Metal's share, or the memory macOS reports as available less what it keeps for macOS, whichever is smaller. It keeps 7 GB, or 15% of the RAM when that is more, and it also keeps the n-gram table's share. It counts as free only what macOS can give without swapping (free and purgeable memory and the clean file cache, not other programs' idle memory). It checks again just before it takes the arena, and fills the arena in 1.5 GB steps, checking again before each one; it stops early rather than push macOS below what it keeps. On a Mac with 64 GB or more the arena, the Metal buffers and the host memory together also stay under 60% of the RAM (see "The engine's whole anonymous footprint" below).
- It reads the rest as each token needs them, and reads ahead the experts the next layer will probably use.
- Setup writes a second copy of the experts for such a Mac (`<model>.experts`, 40 GB for IQ3_XXS). In it each expert's three matrices sit together, so the SSD reads an expert with one request instead of three. On the 32 GB M4 that is about 17% faster decoding and 30% faster prompts, with the same tokens. Setup writes it only when the disk has room for it plus 10 GB; otherwise it warns, and the engine reads the model file as before.
- Setup configures such a Mac without the MTP draft layer: one token per step is faster there.
- Measured at 32 GB (below). Smaller Macs are untested.
- **Setup recommends, and never forces.** It recommends the pack by memory; a bigger pack chosen with `--model` is kept, with a warning about what it risks.
- **The n-gram table.** Each pack also has a 28.8 GB n-gram table (the model's second file). It is read from the SSD a few rows at
a time and does not count against the memory above.
- **Disk.** The original Q2_0 and IQ3 packs download roughly 66 to 84 GB, plus preparation and draft data. Unsloth packs are much
larger and Gyro uses different files. A streamed model can also need a separate experts file (34 to 40 GB for the examples
below). Check the selected pack rather than treating these sizes as a universal limit.
- **Setup recommends and never forces.** It recommends the pack by memory. A bigger pack chosen with `--model` is kept, with a
warning about what it risks.
### A Mac whose memory cannot hold the pack (32 to 48 GB)
The engine streams the experts from the SSD. It decides this itself when the experts are more than 60% of Metal's share.
- It keeps the most-used experts in an arena and reads the rest as each token needs them, with read-ahead for the next layer's
likely experts. The arena is Metal's share, or the memory macOS reports as available less what it keeps for macOS (7 GB, or 15%
of the RAM when that is more), whichever is smaller. It counts as free only what macOS can give without swapping, fills the
arena in 1.5 GB steps and stops early rather than push macOS below its reserve.
- On a Mac with 64 GB or more, the arena, the Metal buffers and the host memory together stay under 60% of the RAM, and the
filled arena is wired (`mlock`) so macOS does not compress it while the GPU is idle. `STRATA_METAL_WIRE_ARENA=0` turns the
wiring off. `STRATA_METAL_ARENA_GB` sets the arena yourself, with a warning when it is over the cap. The engine prints the
planned footprint (`planned resident footprint ... cap ...`) and the footprint after the fill.
- Setup writes a second copy of the experts (`<model>.experts`, 40 GB for IQ3_XXS) in which each expert's three matrices sit
together, so the SSD reads an expert with one request. On the 32 GB M4 that is about 17% faster decoding and 30% faster prompts
with the same tokens. Setup writes it only when the disk has room for it plus 10 GB, otherwise it warns, and it does not write
it on a Mac with 96 GB or more.
- Setup configures such a Mac without the draft layer (one token per step is faster there). A streaming Mac with 96 GB or more
keeps the draft layer and writes `--spec 3`.
- Prompts stream in chunks of 4096 on a streaming Mac with 96 GB or more, and 1024 on a smaller one. `STRATA_METAL_PROMPT_CHUNK=N`
sets it.
- The model's file must be on the internal SSD. An external disk is untested and likely much slower.
## Install
```
For everyday use follow [the app or Terminal installation](INSTALL.md). The commands below are the source installation, which
builds the Metal engine locally.
```sh
git clone https://github.com/Niko1221/Strata && cd Strata
./setup.sh # asks the size; --yes takes the recommended answers
```
What the first run does (each step is skipped when it is already done):
1. **Python.** It needs Python 3.10 or newer, but macOS ships 3.9. If that is all your Mac has, `setup.sh` downloads a standalone Python 3.12 (25 MB, pinned by SHA-256) into `.python`. No sudo, no Homebrew.
1. **Python.** It needs Python 3.10 or newer, but macOS ships 3.9. If that is all your Mac has, `setup.sh` downloads a standalone
Python 3.12 (25 MB, pinned by SHA-256) into `.python`. No sudo, no Homebrew.
2. **Checks your Mac:** the chip, the memory, the macOS version, the Command Line Tools and the free disk.
3. **Python packages** into `.venv`. CMake and Ninja come from pip too, installed just before the build in step 4.
4. **Builds the engine** (`build-metal/strata`). It uses Apple's metal-cpp headers (pinned by SHA-256) and llama.cpp's ggml at the commit the PC build pins.
3. **Python packages** into `.venv`. CMake and Ninja come from pip too, installed just before the build.
4. **Builds the engine** (`build-metal/strata`) with Apple's metal-cpp headers (pinned by SHA-256) and llama.cpp's ggml at the
commit the PC build pins.
5. **Downloads the model.** The download resumes if interrupted. `--gguf-dir` uses files you already have.
6. **Prepares the model:** the pack (seconds), and the MTP draft layer (about 5 GB of the original checkpoint). On a Mac that streams: the experts' copy instead of the draft layer (about 3 minutes on the M4, checked byte for byte against the model file).
6. **Prepares the model:** the pack (seconds) and the MTP draft layer (about 5 GB of the original checkpoint). On a Mac that
streams, the experts' copy takes the draft layer's place (about 3 minutes on the M4).
7. **Writes `strata-<model>.json` and `run-<model>.sh`, then starts the server.**
The config uses the Mac defaults: `--spec 2` and the Q8_0 hyper-connection read (below). After that, `./setup.sh` or `./run-q2_0.sh` starts the model directly. The server is the same as on a PC:
- the OpenAI-compatible API on `http://127.0.0.1:8080/v1`;
- the Anthropic API on `/v1/messages`;
- the chat page on `http://127.0.0.1:8080/`.
After that, `./setup.sh` or `./run-q2_0.sh` starts the model directly. The server is the same as on a PC: the OpenAI-compatible
API on `http://127.0.0.1:8080/v1`, the Anthropic API on `/v1/messages`, and the chat page on `http://127.0.0.1:8080/`.
Options: `--model Q2_0|IQ3_XXS|IQ3_S|UD-Q4_K_XL|UD-Q5_K_XL|UD-Q6_K_XL` (the UD files are Unsloth's), `--vision yes|no`, `--context N`, `--port`, `--host` / `--api-key`, `--draft-vocab`, `--no-browser`,
`--data-dir`, `--models-dir`, `--gguf-dir`, `--setup` (another model or new settings), `--no-start`, `--check` (only
check the Mac), `--build` (build the engine again).
| Option | Meaning |
| --- | --- |
| `--model Q2_0\|IQ3_XXS\|IQ3_S\|UD-Q4_K_XL\|UD-Q5_K_XL\|UD-Q6_K_XL\|Gyro-S\|Gyro-M` | The pack. The `UD-` files are Unsloth's. |
| `--family qwen\|unsloth\|gyro` | The model family. |
| `--vision yes\|no\|gpu\|cpu` | Picture reader. |
| `--context N`, `--port`, `--host`, `--api-key` | Server settings. |
| `--draft-vocab`, `--no-browser` | Draft vocabulary; do not open the browser. |
| `--data-dir`, `--models-dir`, `--gguf-dir` | Where files go, or where yours are. |
| `--setup`, `--no-start`, `--check`, `--build` | Choose another model or settings; install only; only check the Mac; build the engine again. |
### The desktop app and `strata launch` on a Mac
The Strata app (and the Terminal's `strata launch`) do not build anything on your Mac. The Metal engine comes ready-made, as an engine component: from the update feed, or inside the app's installer. The model install is `setup.py` in its Control mode (`STRATA_CONTROL_INSTALL=1`): it downloads (or uses with `--gguf-dir`) and prepares the model files and writes a run config whose engine is `component:strata-engine`. It does not run `metal/setup_mac.py`, needs no Command Line Tools, and writes the same Metal flags (`--mmap-experts`, `--spec 2`, no draft layer when the experts stream from the SSD).
The app and `strata launch` build nothing. The Metal engine comes ready-made as an engine component, from the update feed or
inside the app's installer. The model install is `setup.py` in its Control mode (`STRATA_CONTROL_INSTALL=1`). It downloads, or
uses with `--gguf-dir`, and prepares the model files, and writes a run config whose engine is `component:strata-engine`. It does
not run `metal/setup_mac.py`, needs no Command Line Tools, and writes the same Metal flags.
The model list and the recommendation judge a Mac by its unified memory with the rules of `strata_control/mac_fit.py`, the module `setup_mac.py` uses too: Q2_0, IQ3_XXS and IQ3_S are offered (the Unsloth many-shard files need the repack of `./setup.sh` for now); a pack that does not stay in memory is listed as "read from the SSD", and a Mac under 16 GB of unified memory is told the model does not fit.
The model list and the recommendation judge a Mac by its unified memory with the rules of `strata_control/mac_fit.py`, the module
`setup_mac.py` uses too. Q2_0, IQ3_XXS, IQ3_S and the Gyro packs are offered. Unsloth's many-shard files need the source
installer's repack for now. A pack that does not stay in memory is listed as "read from the SSD", and a Mac under 16 GB of unified
memory is told the model does not fit.
## Measured (M5 Max, 128 GB, macOS 27.2)
The ready-made Mac engine is in the app and in the Terminal's engine component, not in the release zips. The zips (for
`START-HERE.bat` and `setup.sh`) have no Mac engine: `./setup.sh` builds it.
**Decode**, in tokens per second over 200 generated tokens, with greedy answers. The table gives two prompts: a short question
("prompt1") and a story.
## Defaults and options
The config uses the Mac defaults: `--spec 2` with MTP (best on mixed text; an explicit `--spec N` in the config's `args` is kept)
and the Q8_0 hyper-connection read. `STRATA_HC_Q8=0` in the config's `env` gives the exact BF16 read. The Mac engine is
deterministic: two runs give identical numbers.
### Gyro-S and Gyro-M (agentionai's trellis experts)
The Mac engine runs [Gyro](GYRO.md). The routed experts are trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a
Hadamard-rotated basis, read from the published file unchanged. The kernels are in `metal/kernels/quant.metal`, `moe.metal` and
`moe_mm.metal`. Setup offers Gyro-S and Gyro-M next to the other sizes (`./setup.sh --family gyro --model Gyro-S`), never as the
default, and says what the choice risks on your Mac's memory. The 28.8 GB n-gram table is inside the file and is read from the SSD
like every other pack's, so Gyro-S needs 58.5 GB of disk and about 38 GB of memory to stay resident.
Gyro-M runs on the same kernels (checked tensor by tensor) but no Mac tested here could hold it. `metal/tests/gyro_parity.sh`
checks the decoder against the format definition and the expert kernels against a double-precision reference.
`STRATA_METAL_GYRO_TILES=0` turns off the dense-weight tiles.
## Measured (M5 Max, 128 GB)
**Measured on:** Apple M5 Max, 128 GB, macOS 27.2, Qwen3.8-Flash-Next packs, greedy, 200 generated tokens, Strata engine version
not recorded. Decode tok/s for a short question and a story ("prompt1 / story").
| | Q2_0 | IQ3_XXS | IQ3_S |
|---|---|---|---|
| Setup's defaults (`--spec 2` with MTP, the suffix drafter on) | **119.5 / 108.2** | **108.9 / 97.0** | - |
| --- | --- | --- | --- |
| Setup's defaults (`--spec 2` with MTP, suffix drafter on) | **119.5 / 108.2** | **108.9 / 97.0** | - |
| `--spec 2` with MTP, suffix drafter off | 114.0 / 107.9 | 103.5 / 96.9 | 98.2 / 94.1 |
| No MTP (one token per step) | 87.4 / 87.3 | 78.7 / 78.6 | 74.5 / 74.3 |
The output with MTP is the same as without it, token for token, in every run. MTP only proposes tokens, and the
full model checks every one of them.
The output with MTP is the same as without it, token for token. MTP only proposes tokens, and the full model checks each one.
**Prompt reading** (time to the first token), on the needle-test prompts below:
Prompt reading (time to first token), on the needle-test prompts:
| | 4K tokens | 32K tokens |
|---|---|---|
| Q2_0 | ~1,430 tok/s | ~1,460 tok/s |
| IQ3_XXS | ~1,260 tok/s | ~1,240 tok/s |
| IQ3_S | ~1,320 tok/s | ~1,140-1,190 tok/s |
| --- | --- | --- |
| Q2_0 | about 1,430 tok/s | about 1,460 tok/s |
| IQ3_XXS | about 1,260 tok/s | about 1,240 tok/s |
| IQ3_S | about 1,320 tok/s | about 1,140-1,190 tok/s |
The first prompt after the Mac starts is slower, while macOS reads the model file (IQ3_S: 413 tok/s on the first 4K
prompt, cold).
The first prompt after the Mac starts is slower while macOS reads the model file (IQ3_S: 413 tok/s on the first 4K prompt).
llama.cpp on the same Mac with Q2_0: 56.6 tok/s decode on the story, 887 / 809 tok/s on 4K / 32K prompts.
**For comparison**, llama.cpp on the same Mac with Q2_0: 56.6 tok/s decode on the story, and 887 / 809 tok/s on 4K / 32K prompts.
### Unsloth's UD-Q4/Q5/Q6_K_XL on the M5 Max
IQ3_S and UD-Q4_K_XL are the recommended sizes. Setup offers IQ3_S as the default and names UD-Q4_K_XL as the
higher-quality choice where it fits. Decode is tokens per second over 200 tokens (greedy, MTP); prompts are the warm rate
of the second request on (the first request after the start pays a few seconds of first-use cost).
**Unsloth packs on the same Mac.** IQ3_S and UD-Q4_K_XL are the recommended sizes. Decode is tok/s over 200 tokens (greedy, MTP);
prompts are the warm rate from the second request on.
| | IQ3_S | UD-Q4_K_XL | UD-Q5_K_XL (streamed) | UD-Q6_K_XL (streamed) |
|---|---|---|---|---|
| --- | --- | --- | --- | --- |
| Quality against Q8_0 (top-1 / KL) | 87.9% / 0.142 | 92.8% / 0.055 | not measured | not measured |
| Decode, short question, no MTP / MTP | 75.4 / 101.9 | 60.7 / 94.0 | - | - |
| Decode, story, no MTP / MTP | 74.8 / 94.7 | 59.7 / 73.7 | 28 / 46-51 (`--spec 3`) | not measured / 47-51 (`--spec 3`) |
| Prompt 4K / 32K | ~1,180 / ~1,290 | ~1,150 / ~1,290 | ~1,070 / ~1,110 | ~1,020 / ~1,020 |
| Prompt 4K / 32K | about 1,180 / 1,290 | about 1,150 / 1,290 | about 1,070 / 1,110 | about 1,020 / 1,020 |
| llama.cpp Metal, decode (story / code) | 51.1 / 44.1 | 48.4 / 47.3 | 45.1 / 43.2 | does not run (Insufficient Memory) |
| llama.cpp Metal, prompt 4K / 32K | 791 / 730 | 821 / 739 | 729 / 693 | - |
- **UD-Q5 and UD-Q6 cannot stay resident:** their repacked file (103.9 and 114.8 GB) is bigger than the largest buffer Metal gives (86.6 GB), so the experts stream from the SSD into an arena of about 65-67 GB on the 128 GB M5 Max (about 68% of Q5's experts, 62% of Q6's; it was 80-86 GB before the footprint cap below, and the decode and prompt speeds did not change: Q6 story 51 tok/s, 15K prompt 1,020 tok/s at both sizes).
- **The cost is a fixed hand-shake per layer** (the engine reads the router's choice and makes sure the experts are in the arena), not the SSD. A window of 2 tokens costs 38 ms against ~17 resident, so a longer window pays: Q5 story 28 tok/s without MTP, 41.8 with `--spec 2`, 50.6 with `--spec 3`, 46 with `--spec 4`. Setup writes `--spec 3` and keeps the MTP draft layer for a streaming Mac with 96 GB or more.
- **The engine's whole anonymous footprint is limited to 60% of the RAM on a Mac with 64 GB or more (82.4 GB of the M5 Max's 137.4 GB): the arena, every Metal buffer and the host memory the process holds, not only the Metal buffers.** macOS keeps a floor of file cache (`vm.vm_page_filecache_min`: 37.2 GB here) and compresses anonymous memory instead of dropping cache below it. The arena is anonymous, and the GPU wires it only while a command buffer runs, so while it is filled it is ordinary pageable memory. Two kernel panics on the M5 Max (the watchdog daemon starved for 92 s) had the engine at 95 and 90 GiB resident and the compressor at 2-4 million pages. Measured again with a memory watchdog every second: an old-build arena of 100 GB compressed 19 GB of itself in 7 seconds and swapped 24,000 pages; the old automatic arena (69 GB) with a 40 GB file write and a 16 GB program beside it took the compressor from 21,000 to 4.07 million pages in 10 seconds (pressure "warn"). With 30% left (footprint 94.5 GB) and the file write alone, 12,000 pages were swapped out; with 35% left, 316; with 40% left and the write and the 16 GB program, none and the compressor at 0.3 million pages. The old rule (arena + Metal buffers under 67%) counted 7.5 GB of buffers where the process held 11-15 GB, so its real total was about 100 GB. A resident pack is different (clean file pages that macOS drops, never compresses): UD-Q4_K_XL resident (wired 91.7 GB) took the same 40 GB write and 16 GB program with no swap-out and nothing compressed, so resident packs are limited to 70% of the RAM as before. The arena is also wired (`mlock`) once it is filled: unwired it is ordinary pageable memory whenever the GPU is idle (the GPU's own wiring ends a few seconds after its last command buffer; a Metal residency set did not hold it), and a 20 GB file written beside the idle engine took the arena into the compressor and froze the Mac (a third watchdog panic, in a control run of this work). Wired, the same write at full speed raised the compressor by 0.7 million pages (the engine's non-arena memory), swapped out 408 pages and changed nothing else. `STRATA_METAL_WIRE_ARENA=0` turns it off; it is on for Macs with 64 GB or more. The engine prints the planned total (`planned resident footprint ... cap ...`) and the footprint after the fill. `STRATA_METAL_ARENA_GB` still sets the arena as you say, with a warning when it is over the cap.
- **The experts' copy** (`<model>.experts`, 98 GB for Q5) did not pay on the M5 Max: decode 50.3 against 50.2 tok/s without it, 4K prompt 741 against 685 (noisy). Setup does not write it on a Mac with 96 GB or more.
- **Prompts stream in chunks of 4096** on a streaming Mac with 96 GB or more (the engine's own default there; 1024 on a smaller one): every chunk makes one pass over the experts that are not in the arena, so a longer chunk reads them fewer times. UD-Q5, warm, 2 interleaved rounds: chunks of 1024 gave 700 tok/s on the 4K prompt, 2048 gave 920 and 4096 gave 1,100 (32K: 710, 970, 1,100); the decode did not change. `STRATA_METAL_PROMPT_CHUNK=N` sets it. On a resident model 4096 instead of 2048 gained only 2-4% (Q2_0), so that default stays.
UD-Q5 and UD-Q6 cannot stay resident: their repacked file (103.9 and 114.8 GB) is bigger than the largest buffer Metal gives
(86.6 GB). The experts stream from the SSD into an arena of about 65 to 67 GB. The cost is a fixed hand-shake per layer, not the
SSD, so a longer window pays (Q5 story: 28 tok/s without MTP, 41.8 with `--spec 2`, 50.6 with `--spec 3`).
### M4 Mac mini, 32 GB, experts streamed from the SSD
The arena was 11.5-14.7 GB. Tokens per second, one token per step (no MTP), medians of 5 interleaved runs:
**Measured on:** Apple M4 Mac mini, 32 GB, experts streamed from the SSD (arena 11.5 to 14.7 GB), one token per step (no MTP),
medians of 5 interleaved runs, tok/s.
| | Q2_0 | IQ3_XXS | llama.cpp, CPU only (Q2_0 / IQ3_XXS) |
|---|---|---|---|
| --- | --- | --- | --- |
| Decode, short question (200 tokens) | **14.1** | **10.0** | 6.3 / 4.1 |
| Decode, story (200 tokens) | **13.5** | **9.2** | 6.3 / 4.1 |
| Decode after a 6K-token document | **13.1** | **7.7** | 5.1 / 2.8 |
| Prompt, 4K tokens | **93.6** | **65.8** | 21.2 / 8.7 |
- The greedy answers equal the M5 Max's (model in memory) on 10 of 10 test prompts, for both packs.
- No memory was swapped out during the runs; the least memory left to macOS was 7.1 GB.
- MTP was slower here (Q2_0 story 12.1-12.3 with it against 12.3-12.8 without). The draft layer's 1.2 GB do more as expert cache.
The greedy answers equal the M5 Max's (model in memory) on 10 of 10 test prompts. MTP was slower here, because the draft layer's
1.2 GB do more as expert cache. The arena's size matters most: with 5.7 to 9.3 GB (a Mac with less memory free) the same M4 read
about 8.3 tok/s on a story. On an M1 Max with 34 GB (Q2_0, expert cache at 64 slots, kernel fix #2348) decode was 7.9 tok/s and a 9,940-token prompt read at 186 tok/s.
**The experts' copy** (`<model>.experts`, one SSD request per expert): IQ3_XXS after the 6K-token document, the arena held at 12 GB, medians of 5 interleaved runs each:
**Measured on:** Apple M4 Mac mini, 32 GB, Gyro-S, experts streamed (arena 11 to 12 GB of the 25.8 GB of experts), no experts' copy,
one token per step, no draft layer. Decode story, 120 tokens, 5 interleaved pairs, medians: **9.14 tok/s** (8.70 with the
dense-weight tiles off). Prompt of 4,065 tokens: **109 tok/s**. The engine held 18.6 GB of the 32 GB. Against the CUDA engine on
the same Gyro-S file (5 text segments, 4,995 positions, RTX 3060) the Mac is as far from either CUDA run as the two CUDA runs are
from each other: top-1 agreement 95.4% against 95.8%, mean KL 0.0207 against 0.0186.
| | Decode | The 6K prompt | SSD wait per token |
|---|---|---|---|
| The model file (three reads per expert) | 7.37 tok/s | 64.7 tok/s | 46.6 ms |
| The experts' copy (one read per expert) | **8.62 tok/s** | **84.1 tok/s** | 31.5 ms |
## Limits (this version)
- The tokens are identical in all 10 runs.
- The 4K-token prompt (3 interleaved runs each, default arena): 72.9 tok/s with the copy against 62.5 with the model file (+17%), the same tokens.
- 32K-token prompts through the server, with the copy: 29,768 tokens at 89.2 tok/s (5.6 minutes to the first token), 33,803 tokens at 84.4 tok/s. The code word was found both times.
- A 34-minute server run with the copy (47 requests: 32K and 6K needles, short questions, a story, code; context 34,816) got every answer right and did not slow down over time. Decode was 7.2-7.7 tok/s on short answers and 9.9-10.4 on the story; the 6K prompts ran at 80-90 tok/s. macOS's memory pressure was never critical: one 2-second "warn" sample while the arena filled at the start. About 90 MB were swapped out in those same 2 seconds and nothing after; three restarts afterwards (smaller arenas, less memory free) swapped nothing.
- The SSD alone measures the same: 2.97 GB/s for one 1.75 MB read per expert against 2.30 GB/s for three 0.58 MB reads (6 threads, uncached).
- The arena's size matters most. With the model file, a 7.5 GB arena gave 5.5 tok/s, 10.5 GB gave 6.7 and 12 GB gave 7.5. The read-ahead depth, the I/O threads and the eviction order each moved decode by less than 3%.
- The Q2_0 copy is not measured yet. The numbers in the table above are with the model file.
- **Measured again on the release build, with less memory free:** the M4 had 13-17 GB available when the engine started (another user's programs held the rest), so the arena was 5.7 GB (Q2_0) and 9.3 GB (IQ3_XXS, with the experts' copy). One token per step, medians of 3: Q2_0 story 8.3 tok/s, 4K prompt 80; IQ3_XXS story 8.3, 4K prompt 89. That is the arena's size effect above, not a regression: with a 6.6 GB arena the engine used to stop at the first prompt ("no free slot"); a small arena now gives the prompt's 1.6 GB of buffers their own allocation, also when no chunk of the arena is big enough for them (on a 24 GB M2 whose arena shrank below the prompt's needs, every prompt stopped with "no arena region for the prompt buffers"), and a 26-minute server run with a 5.4 GB arena (29 requests, 6K and 31K needles found) was clean.
- With the default arena (12.1-12.2 GB here), the experts' copy gives 8.35-8.48 tok/s decode and 82.5-82.8 tok/s on the prompt. Nothing was swapped out. macOS's memory pressure stayed "normal", except for one 2-second "warn" sample while the arena fills; keeping 7 GB for macOS shows the same sample.
- Those runs and the server run below kept 5 GB for macOS. The default is now 7 GB with the stepped fill: after the 6K document, 7.3 and 6.7 tok/s decode (arenas 9.3 and 7.6 GB, as much as was free), the same tokens, the memory pressure "normal" throughout and nothing swapped out.
- **Images (vision):** on by default. The app and the Terminal use the `strata-vision` that sits beside `strata` in the engine
package; `./setup.sh` builds it and asks "Should the model also read images?" (default yes). With `--gpu` it reads on the Mac's
GPU (ggml's Metal backend), without it on the CPU. A Mac always gets the GPU reader, whatever its memory. The footprint is about
1.18 GB on the GPU and 1.05 GB on the CPU. `--vision cpu` is still allowed and prints a note. The CPU reader is slow because
ggml has no vector kernel for BF16 on ARM. If the GPU reader cannot start, the server prints one line and starts it on the CPU.
Checked on one Mac mini M4 only; the M5 Max is untested for pictures.
**Why `--spec 2`:** measured with MTP on the same two prompts.
**Measured on:** Mac mini M4, 32 GB, Qwen3.8-Flash-Next BF16 projector.
| Window | Q2_0 (prompt1 / story) | IQ3_XXS (prompt1 / story) |
|---|---|---|
| Picture | GPU reader | CPU reader |
| --- | --- | --- |
| 260 tokens (640x400) | 0.45 s | 50.7 s |
| 972 tokens (1600x1200) | 2.4 s | 194 s |
- **Context:** measured up to 32K (needles, KL). Setup recommends 32K below 96 GB and 64K from 96 GB. Up to the model's 262K can
be set, but it is not measured.
- **One request at a time.** `--parallel` (batch slots) is not ported. Several clients queue.
- **Not ported:** the control vector (`--control-vector-scaled`) and the PC tuning (`--calibrate`).
- **Streaming** is measured at 32 GB only.
- **Families:** the original Qwen3.8-Flash-Next, Unsloth's UD-Q4/Q5/Q6_K_XL and agentionai's Gyro-S are measured. Setup repacks
Unsloth's shards into one file for the GPU; a Gyro file is used as it is. The Swift and Coder releases are not set up on Mac
yet, and Qwen3.6 does not run on the Mac engine.
### Quality
- **The Q8_0 hyper-connection read** is the default. Teacher-forced over 28 prompts (code, reasoning, chat, six languages,
documents), about 6,000 positions per set, BF16 reference: mean KL 0.0007 and top-1 agreement 98.9% on the model's own text,
99.1% on another model's text, perplexity unchanged. For scale, IQ3_XXS against Q2_0 is a KL of 0.094.
- **The fast prompt path** against reading the prompt through decode windows: mean KL 0.0004, top-1 99.4 to 99.6%.
- **Needles:** a code word hidden in 8K, 16K and 32K tokens at 10/50/90% depth was found 9 of 9 with Q2_0 and with IQ3_XXS.
- **Same answers as on a PC:** against the CUDA engine on a Tesla P100 (IQ3_XXS and IQ3_S) the first 200 tokens of a reply on the
short question are identical; against the HIP engine on a Radeon AI PRO R9700 (Q2_0), 36 of 39 first tokens are identical. The
differences are near-ties between the two best tokens, from a different floating-point order.
## Switches
For testing. All are off unless set; the defaults are described above.
| Variable | Effect |
| --- | --- |
| `STRATA_HC_Q8=0` | The BF16 hyper-connection read (exact) instead of Q8_0 |
| `STRATA_METAL_PROMPT=window` | Read the prompt through decode windows (slow, exact fallback; about 190 tok/s) |
| `STRATA_METAL_PROMPT_CHUNK=N` | Prompt chunk size (default 2048 resident; see streaming above) |
| `STRATA_METAL_MTP_Q8=0` | The draft layer's matrices in BF16 instead of Q8_0 |
| `STRATA_METAL_MTP_CHAIN=0` | One GPU submission per draft step instead of one per round |
| `STRATA_METAL_FUSE=0`, `STRATA_METAL_LAZY_FOLD=0` | Turn off two decode fusions (same tokens, slower) |
| `STRATA_METAL_PARTIAL=1`, `STRATA_METAL_ROUTE=1` | Two decode variants that measured no faster. `ROUTE` is also wrong on an M1 Max (stale routing logits) |
| `STRATA_METAL_PROMPT_EXPERTS=v3` | The M5 layout for the prompt's expert GEMMs. Fails on an M1 Max, so it is off there (default v2) |
| `STRATA_METAL_STREAM=1` / `=0` | Stream the experts from the SSD even when the pack fits / never (the default decides by size) |
| `STRATA_METAL_CONTIG=0` | Streaming: read the model file even when the experts' copy is there |
| `STRATA_METAL_ARENA_GB=N`, `STRATA_METAL_KEEP_FREE_GB=N` | Streaming: the arena's size, or the memory left to macOS |
| `STRATA_METAL_LEAVE_SHARE=0.4` | Streaming, 64 GB or more: the share of the RAM left to macOS when the footprint cap is set |
| `STRATA_METAL_WIRE_ARENA=0` | Streaming, 64 GB or more: leave the filled arena pageable (default: wired) |
| `STRATA_METAL_IGNORE_REPACK=1` | Start although a model repack or setup's preparation is running (the engine refuses by default) |
| `STRATA_METAL_KEEP_HOST_WEIGHTS=1` | Streaming: keep the host code's own copies of the weights (2 to 3 GB) |
| `STRATA_METAL_TRACE=1`, `STRATA_METAL_PROFILE=1`, `STRATA_METAL_KPROF=T` | Per-window traces, a time breakdown, per-kernel GPU times |
## How it works
The code is in `metal/` (the engine's side in `metal/src/engine`, the kernels in `metal/kernels`).
- **Host code:** the PC host code compiles unchanged for macOS. `metal/include/metal_compat` maps the CUDA runtime calls it makes
onto a small Metal runtime (`metal/src/gpu`).
- **Decode and prompt work** run in Metal kernels: one command buffer per verify window, experts read in place from the mapped
model file, hyper-connections, GDN state and QSA attention on the GPU, and a whole MTP draft round in one command buffer.
- **The PC builds are byte-identical** with this code present. Every shared file's change is behind `STRATA_GPU_METAL` or `__APPLE__`.
## History / older measurements
**Why `--spec 2`.** Measured on: Apple M5 Max, 128 GB, decode tok/s with MTP on the two prompts (prompt1 / story).
| Window | Q2_0 | IQ3_XXS |
| --- | --- | --- |
| `--spec 2` | 111 / 106 | 102 / 95 |
| `--spec 3` | 116 / 92 | 108 / 95 |
| `--spec 4` | 111 / 81 | 108 / 83 |
| `--spec 5-6` | slower | slower |
`--spec 2` is the best default on mixed text. An explicit `--spec N` in the config's `args` is kept.
**The experts' copy.** Measured on: Apple M4 Mac mini, 32 GB, IQ3_XXS after a 6K-token document, arena held at 12 GB, medians of 5
interleaved runs, identical tokens in all 10 runs.
### Gyro-S and Gyro-M (agentionai's trellis experts)
| | Decode | The 6K prompt | SSD wait per token |
| --- | --- | --- | --- |
| The model file (three reads per expert) | 7.37 tok/s | 64.7 tok/s | 46.6 ms |
| The experts' copy (one read per expert) | **8.62 tok/s** | **84.1 tok/s** | 31.5 ms |
From 0.2.2 the Mac engine runs [Gyro](GYRO.md): the routed experts are trellis-coded blocks (ggml types TQ2_T, TQK6, TQK7) in a
Hadamard-rotated basis, read from the published file unchanged. The kernels are ours (`metal/kernels/quant.metal`, `moe.metal`,
`moe_mm.metal`); the codebook and the decoder are the CPU engine's, the rotation is applied to the gate/up and down inputs as on CUDA.
Setup offers Gyro-S and Gyro-M next to the other sizes (`./setup.sh --family gyro --model Gyro-S`), never as the default, and says
what the choice risks on your Mac's memory. The 28.8 GB n-gram table is inside the file and is read from the SSD like every other
pack's, so Gyro-S needs 58.5 GB of disk and about 38 GB of memory to stay resident (29.7 GB of weights).
The 4K prompt read 72.9 tok/s with the copy against 62.5 without. A 34-minute server run with the copy (47 requests, context
34,816) got every answer right and did not slow down. With the model file, an arena of 7.5 GB gave 5.5 tok/s, 10.5 GB gave 6.7 and
12 GB gave 7.5. On the 128 GB M5 Max the copy for Q5 did not pay (decode 50.3 against 50.2 tok/s). The SSD alone measures 2.97 GB/s
for one 1.75 MB read per expert against 2.30 GB/s for three 0.58 MB reads. The Q2_0 copy was not measured.
**Measured on the M4 Mac mini, 32 GB** (the experts stream from the SSD; no experts' copy, because the copy would need 26 GB more
disk than this Mac had; one token per step, no draft layer; the arena was 11-12 GB of the 25.8 GB of experts):
**Prompt chunk on a streaming Mac.** Measured on: M5 Max, UD-Q5, warm, 2 interleaved rounds. Chunks of 1024, 2048 and 4096 gave 700,
920 and 1,100 tok/s on the 4K prompt (32K: 710, 970, 1,100). Decode did not change. On a resident model 4096 instead of 2048 gained
only 2 to 4% (Q2_0), so that default stays.
| | Gyro-S |
|---|---|
| Decode, story, 120 tokens (5 interleaved pairs, medians) | **9.14 tok/s** (9.07-9.15) |
| Decode, with the dense-weight tiles turned off (`STRATA_METAL_GYRO_TILES=0`) | 8.70 tok/s (8.69-8.73); same tokens in all 10 runs |
| Prompt, 4,065 tokens (3 runs) | **109 tok/s** (108-110) |
| Memory the engine held | 18.6 GB of the 32 GB |
**Memory footprint cap.** Measured on: M5 Max, 128 GB. Before the cap, a streaming arena of 100 GB compressed 19 GB of itself in 7
seconds, and the kernel watchdog panicked three times. With 40% of the RAM left to macOS and the arena wired, a 40 GB file write
beside a 16 GB program caused no swap-out. A resident pack (UD-Q4_K_XL, wired 91.7 GB) took the same load with nothing compressed,
so resident packs are limited to 70% of the RAM.
Where the time goes: a decode window of two tokens takes 111 ms. The routed experts are about 40 ms of it, the dense weights
(Gyro keeps its attention, state-space and shared-expert matrices in Q5_K and Q6_K: 3.6 GB read a token)
about 66 ms before the tile change (a run with those kernels skipped). The kernel for one expert layer (ten experts, one token) takes
278 us on the M4, 37.5 GB/s; IQ3_XXS takes 333 us and Q2_0 166 us. The trellis decode is limited by latency and ALU, not by memory:
32-bit funnel shifts for the stream bits, fetching the next unit while decoding the current one, one lane a row with the unit index
folded away and the codebook in constant memory were all tried and none beat the plain kernel, so it is the plain kernel with the
tile that measured fastest. The prompt kernels (fp16 matrix units) are on par with IQ3_XXS: 68 ms against 62 ms a layer for
1,024 tokens.
**Gyro-S kernel profile.** Measured on: M4 Mac mini, 32 GB. A decode window of two tokens takes 111 ms: the routed experts about
40 ms, the dense weights (Q5_K and Q6_K, 3.6 GB read a token) about 66 ms before the tile change. The kernel for one expert layer
(ten experts, one token) takes 278 us at 37.5 GB/s; IQ3_XXS takes 333 us and Q2_0 166 us. The trellis decode is limited by latency
and ALU, not by memory. The prompt kernels run 68 ms a layer against 62 ms for IQ3_XXS at 1,024 tokens.
**Checked against the CPU engine's decoder, on the real file:** the decoder equals the format's definition written out bit by bit
(614,400 sampled blocks a layer, none differ); the Hadamard rotation equals the CPU's bit for bit (f32 and f16); the expert
kernels reproduce a double-precision reference to 1e-7 relative on real experts of layers 0 and 47 of Gyro-S and of layers 0, 23
and 47 of Gyro-M (the 92 GB file was read by range requests, 16 experts a layer: it does not fit this Mac's disk). Run it with
`metal/tests/gyro_parity.sh`.
**Quality against the CUDA engine, Gyro-S file.** `tools/quant_quality.py`, teacher-forced over 4,995 positions, CUDA = RTX 3060.
**Against the CUDA engine, same file** (the pinned Gyro-S; `tools/quant_quality.py`, teacher-forced over the 5 text segments of
`data/quant_quality/v1`, 4,995 positions; CUDA = an RTX 3060 with the engine of this release, run twice: experts on the GPU, experts on the CPU):
| Reference -> run | top-1 agreement | mean KL | KL p99 | perplexity of the text |
|---|---|---|---|---|
| CUDA GPU experts -> CUDA CPU experts (the noise floor: two numerics of one engine) | 95.80 % | 0.0186 | 0.195 | 4.462 / 4.477 |
| CUDA GPU experts -> Mac | 95.46 % | 0.0207 | 0.241 | 4.486 / 4.477 |
| CUDA CPU experts -> Mac | 95.38 % | 0.0187 | 0.272 | 4.486 / 4.462 |
The Mac is as far from either CUDA run as the two CUDA runs are from each other (the CUDA GPU path quantizes the activations to q8_1 and
reads an int8 copy of the codebook, the CPU path an int16 one; the Mac multiplies fp16 codebook values with the float activations).
### Quality
- **The Q8_0 hyper-connection read is the default on Mac.** It keeps a Q8_0 copy of the hyper-connection matrices instead of BF16. `STRATA_HC_Q8=0` in the config's `env` gives the exact BF16 read.
- **How the KL was measured:** teacher-forced over 28 prompts (code, reasoning, chat, six languages, documents), about 6,000 positions per set. The reference is BF16.
- On the model's own text: mean KL 0.0007, top-1 agreement 98.9%.
- On another model's text: mean KL 0.0007, top-1 agreement 99.1%.
- Perplexity of the text is unchanged (×0.9995 and ×1.0006).
- For scale, IQ3_XXS against Q2_0 is a KL of 0.094.
- The Mac engine is deterministic: two runs give identical numbers.
- **The fast prompt path** (the prompt kernels) against reading the prompt through decode windows: mean KL 0.0004, top-1 99.4-99.6%.
- **Needles:** a code word hidden in 8K, 16K and 32K tokens of text, at 10/50/90% depth. Found 9 of 9 with Q2_0 and with IQ3_XXS, both with Q8_0 and with BF16. With the fast prompt path (4K-32K) the answers are the same.
- **Same answers as on a PC:**
- against the CUDA engine on a Tesla P100 (IQ3_XXS and IQ3_S), the first 200 tokens of a reply are identical on the short question;
- on the story, the first IQ3_S tokens differ at token 1, a near-tie (margin 0.12). Following the CUDA engine's tokens, 4 of 200 picks differ, all at margins 0.06-0.21;
- against the HIP engine on a Radeon AI PRO R9700 (Q2_0), 36 of 39 first tokens are identical.
- The differences are near-ties between the two best tokens, from a different floating-point order.
## Limits (this version)
- **Images (vision):** on by default. The app and the Terminal use the `strata-vision` that sits beside `strata` in the engine package;
`./setup.sh` builds it itself and asks "Should the model also read images?" (default yes). One program does both jobs: with `--gpu` it
reads on the Mac's GPU (ggml's Metal backend, the shaders are built into the program), without it on the CPU. A Mac always gets the GPU
reader, whatever its memory. The 24 GB rule of the PC cards does not apply: the Mac has no separate card memory to keep free, and the
reader's footprint is about the same on either side (1.18 GB on the GPU, 1.05 GB on the CPU, mostly the 0.9 GB projector file).
Measured on a Mac mini M4 (32 GB), Qwen3.8-Flash-Next's BF16 projector:
| Picture | GPU reader | CPU reader |
|---|---|---|
| 260 tokens (640x400) | 0.45 s | 50.7 s |
| 972 tokens (1600x1200) | 2.4 s | 194 s |
`--vision cpu` is still allowed and prints that note. The CPU reader is slow because ggml has no vector kernel for BF16 on ARM (a build
for newer chips, with the BF16 and int8 instructions, took the same 50.7 s). If the GPU reader cannot start, the server prints one line
and starts the same program on the CPU. With Q2_0 streaming from the SSD on that M4, the STRATA 42 test picture (red circle, blue square,
green triangle, the text "STRATA 42") was described correctly with both readers. Checked on one Mac only; the M5 Max is untested for pictures.
- **Context:** measured up to 32K (needles, KL). Setup recommends 32K below 96 GB and 64K from 96 GB. Up to the model's 262K can be set, but it is not measured.
- **One request at a time.** `--parallel` (batch slots) is not ported. Several clients queue.
- **Not ported:** the control vector (`--control-vector-scaled`) and the PC tuning (`--calibrate`).
- **Streaming:** measured at 32 GB only. The model's file must be on the internal SSD; an external disk is untested and likely much slower.
- **Families:** the original Qwen3.8-Flash-Next, Unsloth's UD-Q4/Q5/Q6_K_XL and agentionai's Gyro-S are measured (above). Setup repacks Unsloth's shards into one file for the GPU; a Gyro file is used as it is. Gyro-M runs on the same kernels (checked tensor by tensor) but no Mac here could hold it. The Swift and Coder releases are not set up on Mac yet, and Qwen3.6 does not run on the Mac engine.
- **The ready-made Mac engine is in the app and in the Terminal's engine component, not in the release zips.** The zips
(for `START-HERE.bat` and `setup.sh`) have no Mac engine: `./setup.sh` builds it.
## Switches (for testing; all off unless set, the defaults are above)
| Variable | Effect |
|---|---|
| `STRATA_HC_Q8=0` | The BF16 hyper-connection read (exact) instead of Q8_0 |
| `STRATA_METAL_PROMPT=window` | Read the prompt through decode windows (the slow, exact fallback; ~190 tok/s) |
| `STRATA_METAL_PROMPT_CHUNK=N` | Prompt chunk size (default 2048) |
| `STRATA_METAL_MTP_Q8=0` | The draft layer's matrices in BF16 instead of Q8_0 |
| `STRATA_METAL_MTP_CHAIN=0` | One GPU submission per draft step instead of one per round |
| `STRATA_METAL_FUSE=0`, `STRATA_METAL_LAZY_FOLD=0` | Turn off two decode fusions (same tokens, slower) |
| `STRATA_METAL_PARTIAL=1`, `STRATA_METAL_ROUTE=1` | Two decode variants that measured no faster here (same tokens) |
| `STRATA_METAL_STREAM=1` / `=0` | Stream the experts from the SSD even when the pack fits / never (the default decides by size) |
| `STRATA_METAL_CONTIG=0` | Streaming: read the model file even when the experts' copy is there |
| `STRATA_METAL_ARENA_GB=N`, `STRATA_METAL_KEEP_FREE_GB=N` | Streaming: the arena's size, or the memory left to macOS when the arena is sized |
| `STRATA_METAL_LEAVE_SHARE=0.4` | Streaming, Macs with 64 GB or more: the share of the RAM left to macOS and its file cache when the footprint cap is set (for measuring) |
| `STRATA_METAL_WIRE_ARENA=0` | Streaming, Macs with 64 GB or more: leave the filled arena pageable (default: wired with mlock) |
| `STRATA_METAL_IGNORE_REPACK=1` | Start the engine although a model repack or setup's preparation is running beside it (the engine refuses by default: it froze this Mac) |
| `STRATA_METAL_KEEP_HOST_WEIGHTS=1` | Streaming: keep the host code's own copies of the weights (2-3 GB) instead of giving them back to macOS |
| `STRATA_METAL_TRACE=1`, `STRATA_METAL_PROFILE=1`, `STRATA_METAL_KPROF=T` | Per-window traces, a time breakdown, per-kernel GPU times |
**Two opt-ins an M1 Max cannot run (measured on an M1 Max, macOS 26.6.2, Q2_0):** `STRATA_METAL_PROMPT_EXPERTS=v3` - the
prompt's expert GEMMs with the M5's layout, `mm_gu3` - loses the threadgroup once a prompt spreads over more than 40
threadgroups of it (35 pass, 40 fail; 190 tiles and the engine's tile both fail, one tile passes), so it is off there (the
default is v2). `STRATA_METAL_ROUTE=1` - the router GEMV and `moe_route` fused into one dispatch - routes with stale
logits and gives a different top-10 per run (two identical runs differ in ids, weights and plan; the logits themselves
stay bitwise): the fusion hands the routing to the last threadgroup of the GEMV through a device counter, and this
Metal's device atomics are relaxed-only, with no acquire to pair with the writers' fence. Both are off by default. The
`no faster` measurement in the table is from the Macs above; on an M1 Max `ROUTE` is also wrong.
## How it works
The code is in `metal/` (the engine's side in `metal/src/engine`, the kernels in `metal/kernels`). In short:
- **Host code:** the PC host code compiles unchanged for macOS. `metal/include/metal_compat` maps the CUDA runtime calls it makes onto a small Metal runtime (`metal/src/gpu`).
- **The decode and prompt work** run in Metal kernels (`metal/kernels`, `metal/src/engine`):
- one command buffer per verify window;
- the experts are read in place from the mapped model file;
- the hyper-connections, the GDN state and the QSA attention stay on the GPU;
- a whole MTP draft round runs in one command buffer.
- **The PC builds are byte-identical** with this code present. Every shared file's change is behind `STRATA_GPU_METAL` or `__APPLE__`, and the CUDA and HIP objects are compared after each change.
| Reference -> run | top-1 agreement | mean KL | KL p99 | perplexity |
| --- | --- | --- | --- | --- |
| CUDA GPU experts -> CUDA CPU experts (noise floor) | 95.80% | 0.0186 | 0.195 | 4.462 / 4.477 |
| CUDA GPU experts -> Mac | 95.46% | 0.0207 | 0.241 | 4.486 / 4.477 |
| CUDA CPU experts -> Mac | 95.38% | 0.0187 | 0.272 | 4.486 / 4.462 |
docs/MCP_SERVER.md
+13 -8
View File
@@ -1,13 +1,14 @@
# Strata's MCP server: let your AI assistant install and run Strata
`tools/strata_mcp.py` is an [MCP](https://modelcontextprotocol.io) server for the AI assistant you already use
(Claude Code, Claude Desktop, Cursor, VS Code, Codex ...). Add it once, then ask in plain words:
This page is for people who use an AI assistant and want it to install, start, stop and check Strata. `tools/strata_mcp.py`
is an [MCP](https://modelcontextprotocol.io) server for the assistant you already use (Claude Code, Claude Desktop,
Cursor, VS Code, Codex ...). Add it once ([Quick start](#add-it)), then ask in plain words:
- "Install Strata for this PC."
- "Start Strata." / "Stop Strata."
- "Is Strata running? How much VRAM does it use?"
- "Which Strata model fits my PC?"
- "Strata didn't start - what does the log say?"
- "Strata didn't start. What does the log say?"
- "How fast is Strata on this PC?" (a short speed test)
- "How do I connect Open WebUI / the OpenAI SDK / Claude Code to Strata?"
@@ -21,6 +22,9 @@ direction: *your assistant* manages Strata.
## Add it
This is an optional **source-checkout tool**. It is not required to install the released app or Terminal.
For normal agent-assisted installation, use [AI_SETUP.md](AI_SETUP.md).
You need a copy of Strata: the zip from GitHub, or `git clone https://github.com/Niko1221/Strata`. Below,
`C:\Users\you\Strata` (Windows) and `/home/you/Strata` (Linux) stand for that folder. Use `python3` on Linux. On
Windows, use `py` if `python` is not on your PATH.
@@ -110,9 +114,10 @@ The command is `python <Strata folder>/tools/strata_mcp.py`. The transport is st
| Tool | What it does | Arguments (all optional) |
| --- | --- | --- |
| `strata_status` | Whether a model is running: the model, engine version, context, busy or idle, VRAM and RAM in use. What is installed: models, start scripts, engine version, data folder, downloads. An install in progress. This PC's hardware: OS, CPU, RAM, GPUs, page file, free disk. The model and context setup recommends for this PC. | `port`, `hardware` (true/false) |
| `strata_models` | Every family and size setup offers, with download size, RAM needed, whether it fits this PC, and whether it is installed. Also the context lengths and the recommendation. | - |
| `strata_install` | Runs setup without questions, in the background (see below). Without `confirm: true` it only returns the plan. A second call shows the progress. `cancel: true` stops the install. | `family` (qwen, swift, coder, unsloth), `model` (Q2_0, IQ2_XS, IQ3_XXS, IQ3_S, IQ1_M, UD-IQ4_XS, UD-Q4_K_XL), `context` (8192 ... 524288), `vision` (no, yes, gpu, cpu), `kv` (int8, q4_0, k8v4), `gpu` (a number) or `gpus` ("0,2", "all"), `backend` (auto, cuda, hip), `low_ram`, `port`, `data_dir`, `confirm`, `cancel` |
| `strata_start` | Starts an installed model in the background and waits until it answers. If it is not ready after `wait_seconds` (default 50), it returns "loading". | `model` (an id from `strata_status`, e.g. `q2_0`, `coder-iq1_m`; default: the most recently used), `port`, `gpu`, `wait_seconds` (0-600) |
| `strata_models` | Every family and size setup offers (qwen, swift, coder, unsloth, gyro, nvfp4, and Qwen3.6-35B-A3B), with download size, RAM needed, whether it fits this PC, and whether it is installed. Also the context lengths and the recommendation. | - |
| `strata_remove` | Deletes an installed model's own files (GGUFs, pack folder, run config) and nothing another installed model still uses. Without `confirm: true` it only returns the plan. Stop the model first. | `model` (required), `confirm` |
| `strata_install` | Runs setup without questions, in the background (see below). Without `confirm: true` it only returns the plan. A second call shows the progress. `cancel: true` stops the install. | `family` and `model` (the ones `strata_models` lists), `context` (1024 ... 2097152; usual sizes 8192 ... 524288), `vision` (no, yes, gpu, cpu), `kv` (int8, q4_0, k8v4), `gpu` (a number) or `gpus` ("0,2", "all"), `backend` (auto, cuda, hip), `low_ram` (auto, on, off, resident, mmap), `port`, `data_dir`, `gguf_dir` (a folder with the model's GGUF files, checked and not downloaded again), `confirm`, `cancel` |
| `strata_start` | Starts an installed model in the background and waits until it answers. If it is not ready after `wait_seconds` (default 50), it returns "loading". | `model` (an id from `strata_status`, e.g. `q2_0`, `coder-iq1_m`, `swift-iq3_xxs`; default: the most recently used), `port`, `gpu`, `wait_seconds` (0-600) |
| `strata_stop` | Unloads the model through the server's own API, then ends the server it started. | `port`, `force` |
| `strata_logs` | The last lines of the server log, the engine log or the setup log. | `source` (server, engine, setup), `model`, `lines` (1-500) |
| `strata_benchmark` | Sends a short fixed request (greedy, thinking off) and reports output and prompt tokens/s. | `port`, `max_tokens` (16-512) |
@@ -130,7 +135,7 @@ Every result is JSON with a one-line `summary`. It is sent both as text and as M
- the context setup recommends for the GPU's VRAM;
- pictures on where the model can read them (the reader runs on the CPU under 24 GB of VRAM and on the GPU from 24 GB, on NVIDIA and AMD cards; a Mac reads on its GPU).
The first call returns the plan: the download size (58-111 GB), the free disk space, the RAM check, and the exact
The first call returns the plan: the download size, the free disk space, the RAM check, and the exact
setup command. The assistant should show you the plan and call again with `confirm: true` once you agree.
It then runs the same steps `START-HERE.bat` / `setup.sh` run:
@@ -164,7 +169,7 @@ VRAM and pinned RAM itself, as when you close the window. Then the server ends:
While a request is running, the tool refuses unless you pass `force: true`.
**Status.** The tool reads `GET /health`, `/v1/status`, `/status` and, on older servers, `/metrics` on 127.0.0.1.
**Status.** The tool reads `GET /health`, `/v1/status`, `/status` and `/metrics` on 127.0.0.1.
If a config of the folder sets an `api_key`, the tool sends that key with its own requests. It never shows the key.
## Safety
docs/MODELS.md
+264 -199
View File
@@ -1,251 +1,265 @@
# Which model? Sizes, versions and what fits
Strata's main model is [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), in several **sizes**
(the same model, compressed more or less) and several **versions** (the original, a coding version, a fine-tune, other
makers' compressions). It also runs a smaller model, [Qwen3.6-35B-A3B](#qwen36-35b-a3b). The desktop app, the Terminal
(`strata launch`) and the installer recommend one for your PC; this page explains the choice. Back to the
[README](../README.md).
This page helps you choose a model and check that your computer can run it. It covers DeepSeek V4.1-Flash,
Qwen3.8-Flash-Next (and its variants) and Qwen3.6-35B-A3B. For installing, see [INSTALL.md](INSTALL.md).
> **On this page:** [Pick by memory](#pick-by-memory) · [Speed](#how-fast-is-each-size) · [The sizes](#the-sizes) ·
> [Will it fit?](#will-it-fit) · [The versions](#the-versions) · [Qwen3.6](#qwen36-35b-a3b) ·
> [Adding another model](#adding-or-switching-models)
> **On this page:** [Quick start](#quick-start) · [Pick by memory](#pick-by-memory) ·
> [DeepSeek V4.1-Flash](#deepseek-v41-flash) · [Qwen3.8-Flash-Next](#qwen38-flash-next) ·
> [Qwen3.6-35B-A3B](#qwen36-35b-a3b) · [Adding or switching models](#adding-or-switching-models) ·
> [History](#history--older-measurements)
## Quick start
1. Find your memory in [Pick by memory](#pick-by-memory) and choose the first provider that fits.
2. Print the plan. Nothing is downloaded or changed:
```sh
strata launch --plan --family qwen --model iq2_xs
```
3. Start it with `strata launch` (guided), or run the same command without `--plan` and add `--yes --no-tui`.
**Provider order:** ISTA-DASLab, then Unsloth, then Gyro, then others, using the first compatible pack that fits the
model and your hardware. ISTA-DASLab is the default recommendation. Gyro saves memory and is slower. Swift 1.5, NVFP4
and OrcaRouter are other options. An explicit choice by the user always wins. The order applies within a model family:
Qwen3.6 has Unsloth packs only, and DeepSeek has the IST-DAS sets. The order is a recommendation. It says nothing about which
provider wins a given quality or speed comparison.
Setup may preselect something else: a Swift or Gyro size at a low memory tier, or Unsloth on Strix Halo. Read the plan
and pass `--family` and `--model` when you want a particular provider.
## Pick by memory
The tiers count RAM and VRAM (the graphics card's memory) together, the same as the [README](../README.md#the-other-sizes).
`strata launch`, setup and the install tool of the MCP server pre-select by them; the desktop app does from 0.2.1. Every
other size stays a choice. Strata leaves 5 GB for Windows and the system, so the sum is RAM + VRAM minus 5 GB: a PC with a card of 12 GB and 32 GB of RAM is in the 40 GB tier (32 + 12 - 5 = 39), and a PC with a 16 GB card and 32 GB of RAM is too, not in the 48 GB tier (#3425).
Figures are the model's weights in RAM plus VRAM. Leave memory for the OS and other apps. Unified memory counts once.
They are estimates; a given context may still not fit.
| Your memory (RAM + VRAM) | Pre-selected | Notes |
| --- | --- | --- |
| **32 GB** | **Gyro-S**, or the **Coder** for code | The Coder needs 32 GB of RAM and keeps half of the experts, so it is weaker outside code ([why](#coder)). Gyro-S has lower quality than the 2-3-bit sizes, and the 2-bit Q2_0 decodes faster |
| **40 GB** | **Swift 1.5** IQ2_XS (39.2 GB) | Gyro-M (35 GB) is the other choice: close to IQ3_XXS in quality, but it writes answers 16-21% slower than Q2_0. Gyro runs on NVIDIA, AMD and Intel Arc cards and, from 0.2.2, on Macs (Gyro-S measured on a 32 GB Mac mini) |
| **48 GB** | **IQ3_XXS** (47.0 GB) | Slower, better |
| **64 GB** | **IQ3_S** (54.8 GB), the original. Swift 1.5's IQ3_S is an equal choice | The slowest, the best: it matches the full model on the published tests |
| **96 GB or more** | [Unsloth's UD-IQ4_XS to Q8_0](#unsloths-files-ud-iq4_xs-to-q8_0), 4 to 8 bit | UD-IQ4_XS keeps every expert in RAM from about 80 GB of RAM, Q8_0 from about 136 GB. Up to 192 GB, the 5-8-bit files fit in RAM. Setup still pre-selects IQ3_S here, except on a Strix Halo |
**Qwen3.8-Flash-Next, ISTA-DASLab original packs**
With less memory than a row says, Strata's [low-RAM mode](#a-big-graphics-card-and-little-ram) starts by itself and the
model runs slower, much slower with a small card. The figures above are RAM + VRAM; [Will it fit?](#will-it-fit) counts
RAM alone, for the sizes that keep their experts in RAM.
Not sure? Take what the installer pre-selects. The Coder fits a 32 GB PC, but it keeps only 256 of the 512 experts,
chosen on code data, so it is weaker outside code and in languages other than English. For general use, take a full
model (Q2_0, IQ2_XS or IQ3_S) whenever your memory allows.
## How fast is each size
These are older numbers, kept because they compare the sizes with each other. Newer engines are faster on some cards
(each change is listed with its own measurement in the [details](DETAILS.md)); the 0.2.0 engine (0.1.45.3) has not been
re-measured on the whole table. Measured on an RTX 5070 (12 GB), a Ryzen 5 7600 and 64 GB of RAM, engine 0.1.26 (the
Q2_0 row: engine 0.1.36, which made Q2_0 prompts faster):
| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt |
| --- | ---: | ---: | ---: |
| **Q2_0** | 94 tokens/s | 76 tokens/s | 2,650 tokens/s |
| **IQ2_XS** | 79 tokens/s | 63 tokens/s | 2,090 tokens/s |
| **IQ3_XXS** | 62 tokens/s | 49 tokens/s | 1,750 tokens/s |
| **IQ3_S** | 53 tokens/s | 46 tokens/s | 1,620 tokens/s |
| **Coder** (IQ1_M) | 55 tokens/s | 43 tokens/s | 2,180 tokens/s |
Measured on an AMD RX 9070 XT (16 GB), a Ryzen 9 3900X and 47 GB of RAM (Linux, setup's own install, 2026-10-02, an engine
from before 0.1.40; Q2_0 is above setup's RAM estimate for 47 GB and was installed with `--model Q2_0 --yes`):
| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt |
| --- | ---: | ---: | ---: |
| **Q2_0** | 60 tokens/s | 48 tokens/s | 1,160 tokens/s |
| **IQ2_XS** | 52 tokens/s | 36 tokens/s | 1,110 tokens/s |
| **Coder** (IQ1_M) | 44 tokens/s | 33 tokens/s | 1,420 tokens/s |
- **Writes answers** = how fast the reply appears (tokens per second; a token is about ¾ of a word).
- **Reads your prompt** = how fast it takes in what you send (long documents, code, chat history), measured on a
32K-token prompt; a 4K prompt reads at 910-1,580 tokens/s. A 32K prompt takes about 15 seconds with Q2_0.
A card with more VRAM is faster, because more of the model fits on the GPU. For an RTX 3090 (24 GB) the
[details](DETAILS.md#other-gpus-estimated) give an estimate, not a measurement. All measurements, long-context numbers
and estimates for other cards are in the [details](DETAILS.md#speed-measured). The AMD measurements per card (RX 9070 XT,
Radeon AI PRO R9700, RX 7800 XT, RX 9060 XT, RX 6900 XT) are in [AMD_HIP.md](AMD_HIP.md#rdna4-gfx1201).
Every PC is different: `START-HERE.bat --calibrate` measures a few engine settings on yours and keeps the fastest
(about 15-30 minutes, longer on a slow card; on the PC above it made the Coder 7% faster; NVIDIA cards for now).
`strata launch` and the app have their own tuning step (quick: about 20 minutes). Measured Strata on your own
PC? See [Community benchmark results](COMMUNITY_BENCHMARKS.md) for a report template and how to share your results
in a pull request.
## The sizes
| Model | RAM+VRAM Requirements | Speed | Quality |
| Size | RAM + VRAM for the model | Speed | Quality |
| --- | ---: | --- | --- |
| **Q2_0** | 37.6 GB | fastest | good |
| **IQ2_XS** | 39.2 GB | fast | better (**recommended** at 40 GB, as Swift 1.5) |
| **IQ3_XXS** | 47.0 GB | slower | great |
| **IQ3_S** | 54.8 GB | slowest | best: matches the full model on the published tests |
| Q2_0 | 37.6 GB | fastest | good |
| IQ2_XS | 39.2 GB | fast | better |
| IQ3_XXS | 47.0 GB | slower | great |
| IQ3_S | 54.8 GB | slowest | best; matches the full model on the published tests |
The download is 66-76 GB for the three smaller sizes ([details](DETAILS.md#which-model)); the first start also
fetches the MTP draft layer (~6 GB, +1 GB with images).
- **Less memory, coding only:** [Coder](#coder) runs around 32 GB of RAM with a suitable GPU.
- **Less memory, general use:** compare Unsloth, then [Gyro](#gyro-s-and-gyro-m-agentionai), which uses the least memory and is slower.
- **More memory:** [Unsloth's 4 to 8 bit files](#unsloths-files-ud-iq4_xs-to-q8_0) need 72 to 136 GB for all experts in RAM.
- **A smaller model:** [Qwen3.6-35B-A3B](#qwen36-35b-a3b), 32 to 52 GB of RAM.
- **A very large model:** [DeepSeek V4.1-Flash](#deepseek-v41-flash), 12 GB graphics card, 46 to 64 GB of RAM and 257 GB of disk.
## Will it fit?
If RAM is short, setup can read experts from the SSD instead ([below](#will-it-fit)). That works and is much slower.
Shard 1 is the part of the model that gets loaded when it starts: its experts go into your **RAM**, the rest onto
your graphics card (the second shard, a 29 GB lookup table, stays on the SSD). So it fits when your **RAM is at least
the experts + about 10 GB** for Windows and your other programs - the experts are most of shard 1: 34 GB for Q2_0,
35.5 for IQ2_XS, 43 for IQ3_XXS, 50 for IQ3_S, 23 for the Coder (the dense weights in shard 1 go to the graphics
card). With 64 GB of RAM every size fits (IQ3_S with little else open); with 48 GB, Q2_0 and IQ2_XS. A bigger graphics card makes it faster, but it doesn't lower the RAM needed
- except in the low-RAM mode below.
## DeepSeek V4.1-Flash
### A big graphics card and little RAM
DeepSeek V4.1-Flash-1017 runs in the **desktop app** (pick it in the model list) and in the **Terminal** (`strata launch`).
It uses its own engine, a separate download that the app and `strata launch` fetch for you.
On a PC whose RAM cannot hold the experts beside the system, setup maps them from the model's files instead and keeps
in RAM only what the graphics card does not hold (the [low-RAM mode](DETAILS.md#speed-measured), chosen by setup). For
example, a 32 GB PC with a 24 GB GPU runs Q2_0, IQ2_XS and the Coder this way, and a 32 GB PC with a 12-16 GB GPU the
Coder. With a small card most experts then come from the SSD and it is much slower (setup says so).
`START-HERE.bat --setup --low-ram on|off` overrides the choice.
| Set (IST-DAS) | Download | Quality against the original model |
| --- | ---: | --- |
| 1.75 bpw | 239 GB | perplexity +21%; code and agent text +8.9% and +10.2%; weaker on plain prose |
| 2.00 bpw | 257 GB | perplexity +16%; code and agent text +8.5% and +5.9% |
| 2.25 bpw | 274 GB | perplexity +7.7% to +8.3%; code and agent text +3.5% and +2.5% |
## The versions
The download includes the shared n-gram (Engram) table and the DSpark drafter. Pick 2.25 for coding and agent work if
you have the disk. Perplexity is measured on a fixed set of 4,995 test texts; lower is closer to the original.
**Pictures.** The versions in setup's menu read pictures, and pictures are on by default wherever a model can
read them. The picture reader runs on the CPU when the graphics card has less than 24 GB (a few seconds per
picture, nothing taken from the card) and on the graphics card from 24 GB (about 1-1.4 GB of VRAM; 0.1-0.5 s per
picture). The rule is the same on NVIDIA and AMD cards. `--vision off` (or "No" in the app and in `strata launch`) turns
pictures off; `--vision cpu` or `--vision gpu` picks the reader. On AMD, `strata-vision-hip` is the card's reader; if it
cannot start, Strata uses the CPU reader by itself ([AMD_HIP.md](AMD_HIP.md)). The Mac reads pictures on its GPU
([MAC.md](MAC.md)). On an Intel Arc the rule is the same, with two differences: an Arc of 24 GB or more keeps the
CPU reader unless you ask for `--vision gpu` (the Intel GPU reader has only run on an 8 GB A750), and a Battlemage card (B580, B70)
has pictures off until you pass `--vision yes` ([INTEL.md](INTEL.md#pictures-on-an-arc-2026-10-19-branch-f-021-arcvis)). More: [Images](DETAILS.md#images-vision).
**Supported hardware**
### Qwen3.8-Flash-Next
| Hardware | Systems |
| --- | --- |
| NVIDIA RTX 30, 40 and 50 series, 12 GB or more | Windows, Linux |
| AMD Radeon AI PRO R9700 | Linux |
The original, in all four sizes. With images, and with the
[experimental speed projection](DETAILS.md#experimental-speed-projection-experimental-off-by-default) as an option.
Tested on an RTX 3060 (Linux), an RTX 5070 (Windows) and four R9700 cards (Linux). The RTX 30 kernels were also checked
on an RTX A4000. Windows on RTX 30 and the RX 9070 are built but were not run. Mac and Strix Halo do not run it yet.
**What you need:** a graphics card with 12 GB or more, 46 GB of RAM at least (64 GB is the tested comfortable size),
an NVMe SSD, and the disk space above. Setup warns when disk or RAM is short and does not stop you.
```sh
strata launch --family deepseek41 --model ist-das-2.00bpw --plan
strata launch --family deepseek41 --model ist-das-2.00bpw
```
`--model-path DIR` starts from files you already have. The context is 8192 tokens (8K), the size it was checked at.
**Speed: Faster or Full quality.** The app asks, and the Terminal takes `--speed faster|full` (or `--full-quality`). Faster uses 5 of the
model's 6 experts per word, a small quality change that is usually not noticeable. It is the default on a PC with 16 GB of graphics
memory or less. Full quality uses all 6. It needs DeepSeek engine 0.1.50 or newer, which the app and `strata launch` fetch.
**Measured on:** IST-DAS 2.00 bpw, Strata 0.2.4 (before the Speed choice).
| Hardware | Output | Prompt (8K) |
| --- | ---: | ---: |
| RTX 5070 12 GB, Ryzen 5 7600, 64 GB RAM | about 5 tokens/s (code) | 231 tokens/s |
| RTX 3060 12 GB, 46 GB RAM | 6 to 9 tokens/s | about 103 tokens/s |
| 4x Radeon AI PRO R9700 | about 31 tokens/s; 55 to 69 with DSpark | about 360 tokens/s (557 at 32K) |
- The 2.25 bpw set reads 13% more data per token and is a little slower.
- A slow SSD lowers every number, because 12 GB cards spend most of each token waiting on the SSD and the CPU.
- DSpark (a small draft model) raised the RTX 3060 PC from 6.1-6.7 to 8.2-8.7 tokens/s and the R9700 box from about 31 to 55-69. It gave no gain on the RTX 5070 PC with a 6-core CPU, so `--dspark auto` leaves the drafter unloaded there.
- Answers can differ between runs with the same prompt, because the experts on the card change from moment to moment.
<a id="the-versions"></a><a id="the-sizes"></a>
## Qwen3.8-Flash-Next
Four ISTA-DASLab sizes are the original model; the versions below are variants or other providers' packs of it.
Every version in setup's menu reads pictures. [Picture rules](#pictures).
### Will it fit?
Shard 1 is loaded at start: its experts go into **RAM**, the rest onto the graphics card. The second shard (a 29 GB
lookup table) stays on the SSD. The model fits when **RAM is at least the experts plus about 10 GB** for the OS and
other programs.
| Size | Experts in shard 1 |
| --- | ---: |
| Q2_0 | 34 GB |
| IQ2_XS | 35.5 GB |
| IQ3_XXS | 43 GB |
| IQ3_S | 50 GB |
| Coder | 23 GB |
With 64 GB of RAM every size fits (IQ3_S with little else open). With 48 GB, Q2_0 and IQ2_XS fit. A bigger graphics
card makes it faster and does not lower the RAM needed, except in low-RAM mode.
<a id="a-big-graphics-card-and-little-ram"></a>
**Big graphics card, little RAM.** When RAM cannot hold the experts beside the system, setup maps them from the model
files and keeps in RAM only what the card does not hold. A 32 GB PC with a 24 GB GPU runs Q2_0, IQ2_XS and the Coder
this way, and a 32 GB PC with a 12 to 16 GB GPU runs the Coder. With a small card most experts come from the SSD and it
is much slower; setup says so. A source checkout can force the mode with `--low-ram on|off`
([SOURCE_INSTALL.md](SOURCE_INSTALL.md)). `strata launch` has no `--low-ram` flag.
The download is 66 to 76 GB for the three smaller sizes ([details](DETAILS.md#which-model)). The first start also
fetches the MTP draft layer (about 6 GB, 1 GB more with pictures).
### Pictures
Pictures are on by default wherever a model can read them. The reader runs on the CPU when the card has less than
24 GB (a few seconds per picture, no VRAM used) and on the card from 24 GB (about 1 to 1.4 GB of VRAM, 0.1 to 0.5 s per
picture). The rule is the same on NVIDIA and AMD.
| Setting | Effect |
| --- | --- |
| `--vision off` | no pictures (also "No" in the app and in `strata launch`) |
| `--vision cpu` / `--vision gpu` | choose the reader |
- **AMD:** `strata-vision-hip` is the card's reader. If it cannot start, Strata uses the CPU reader ([AMD_HIP.md](AMD_HIP.md)).
- **Mac:** pictures run on the GPU ([MAC.md](MAC.md)).
- **Intel Arc:** an Arc with 24 GB or more keeps the CPU reader unless you pass `--vision gpu`. A Battlemage card (B580, B70) has pictures off until you pass `--vision on` ([INTEL.md](INTEL.md)).
More: [Images](DETAILS.md#images-vision).
### Coder
**[Coder](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF)** - ISTA-DASLab's coding version:
it keeps 256 of each layer's 512 experts, chosen on code (with agentic and image) data, and drops the rest (91% of
the full model's SWE-bench Verified score, 99% of LiveCodeBench, by its authors). One size (IQ1_M: its experts stored
like IQ3_S): shard 1 is **29.6 GB**, so it fits a PC with **32 GB of RAM**, runs 262K context on 64 GB, and reads long
prompts the fastest of all. With half the experts gone it is weaker outside code and in languages other than
English, Chinese and other CJK text included (#438: Chinese answers came out wrong or looping where English was
fine). For general use, or whenever your RAM allows, take a size that keeps every expert: Q2_0, IQ2_XS or IQ3_S.
More: [details](DETAILS.md#or-the-coder-half-the-experts-for-code).
**[Coder](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF)** is ISTA-DASLab's coding version.
It keeps 256 of each layer's 512 experts, chosen on code (with agentic and image data), and drops the rest. Its authors
report 91% of the full model's SWE-bench Verified score and 99% of LiveCodeBench. There is one size, IQ1_M (experts
stored like IQ3_S); shard 1 is **29.6 GB**. A 32 GB RAM computer also needs enough GPU memory and room for buffers;
read the launch plan.
```
START-HERE.bat --setup --family coder
It is weaker outside code and in languages other than English, Chinese and other CJK text included. For general use,
or whenever RAM allows, take a size that keeps every expert. More: [details](DETAILS.md#or-the-coder-half-the-experts-for-code).
```sh
strata launch --family coder --model IQ1_M
```
### Swift 1.5
**[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)** - a fine-tune by UkisAI that
thinks much shorter before answering, so you get the answer sooner, with about the same quality. Same speed per
token, and about the same RAM as the same size of the original. Its own license applies (see its page).
More: [details](DETAILS.md#or-swift-15-a-fine-tune-that-thinks-shorter).
**[Swift 1.5](https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF)** is a fine-tune by UkisAI that
thinks much shorter before answering, with about the same quality. Speed per token and RAM match the same size of the
original. Its own license applies. More: [details](DETAILS.md#or-swift-15-a-fine-tune-that-thinks-shorter).
```
START-HERE.bat --setup --family swift --model IQ2_XS
```sh
strata launch --family swift --model IQ2_XS
```
<a id="unsloth-ud-iq4_xs"></a><a id="unsloth-ud-q4_k_xl-experimental"></a>
### Unsloth's files: UD-IQ4_XS to Q8_0
The fourth version in setup's menu (`--family unsloth`) is **Unsloth's quantizations** of the same model, from ~4 to 8
bits. They keep the original's experts at more bits than the 2-3-bit sizes above, so they need more RAM. Setup keeps
every expert in RAM where they and 16 GB more fit (the fastest way); with less RAM it keeps as many as fit and reads
the rest from the SSD while it answers (much slower), and for the bigger files it asks first. Pictures are on by default
(the CPU reader under 24 GB of graphics memory, the card's reader from 24 GB).
`--family unsloth` is Unsloth's 4 to 8 bit quantizations of the same model. They keep the experts at more bits than
the sizes above, so they need more RAM. Setup keeps every expert in RAM when they fit with 16 GB to spare. With less RAM
it keeps as many as fit and reads the rest from the SSD (much slower), and for the bigger files it asks first.
| Size | Download | Experts | RAM for all of them in RAM | Same top token as Q8_0 | AMD R9700 (32 GB): story / code / 32K prompt |
| --- | ---: | ---: | ---: | ---: | --- |
| **UD-IQ4_XS** (setup's first) | 94 GB | 55 GiB | ~72 GB | 88.9% | - |
| **UD-Q4_K_XL** | 111 GB | 72 GiB | ~88 GB | 93.1% | 63 / 87 tok/s / 1,110-1,320 tok/s |
| **UD-Q5_K_XL** | 158 GB | 92 GiB | ~108 GB | 94.6% | 57-60 / 82 / 810-1,260 |
| **UD-Q6_K_XL** | 169 GB | 102 GiB | ~118 GB | 94.6% | 54 / 76 / 1,020 |
| **Q8_0** | 188 GB | 120 GiB | ~136 GB | 100% (the reference) | 53-57 / 66-74 / 1,230-1,300 |
| Size | Download | Experts | RAM for all experts | Same top token as Q8_0 |
| --- | ---: | ---: | ---: | ---: |
| **UD-IQ4_XS** (setup's first) | 94 GB | 55 GiB | ~72 GB | 88.9% |
| **UD-Q4_K_XL** | 111 GB | 72 GiB | ~88 GB | 93.1% |
| **UD-Q5_K_XL** | 158 GB | 92 GiB | ~108 GB | 94.6% |
| **UD-Q6_K_XL** | 169 GB | 102 GiB | ~118 GB | 94.6% |
| **Q8_0** | 188 GB | 120 GiB | ~136 GB | 100% (the reference) |
For comparison, IQ3_S on the same card: 75 / 95 tok/s, 87.7% the same top token as Q8_0. The speed figures come from
engines older than 0.1.45.3 (the conditions are in [QUANTS.md](QUANTS.md)). The quality numbers, the
speed bound and NVIDIA/AMD/Intel details: [QUANTS.md](QUANTS.md); the files: [UD-Q4_K_XL](UNSLOTH_Q4.md),
[UD-Q6_K_XL](UNSLOTH_Q6.md).
For comparison, IQ3_S matches the top token of Q8_0 87.7% of the time. Quality numbers, the speed bound and
NVIDIA, AMD and Intel details are in [QUANTS.md](QUANTS.md); the files are described in [UD-Q4_K_XL](UNSLOTH_Q4.md) and
[UD-Q6_K_XL](UNSLOTH_Q6.md). Older AMD speeds are in [History](#history--older-measurements).
```
START-HERE.bat --setup --family unsloth --model UD-Q5_K_XL
```sh
strata launch --family unsloth --model UD-Q5_K_XL
```
<a id="gyro"></a>
### Gyro-S and Gyro-M (agentionai)
The fifth version in setup's menu (`--family gyro`) is **agentionai's Gyro quantizations** of the same model: trellis-coded
experts in a rotated basis. They are the smallest options and never setup's default. The download includes the
28 GB n-gram table; the memory figure is VRAM and RAM together.
`--family gyro` is agentionai's Gyro quantization: trellis-coded experts in a rotated basis. They are the smallest
options and never setup's default. The download includes the 28 GB n-gram table. The memory figure is VRAM and RAM together.
| Size | Download | Memory it needs (VRAM + RAM) | In plain words |
| Size | Download | Memory (VRAM + RAM) | In plain words |
| --- | ---: | ---: | --- |
| **Gyro-S** | 58.5 GB | 27.6 GB | the smallest option, for PCs with little memory; lower quality |
| **Gyro-M** | 92.0 GB | 35 GB | about IQ3_XXS quality with about 20% less GPU memory; decodes 16-21% slower than Q2_0 (RTX 3060 12 GB, R9700) |
| **Gyro-M** | 92.0 GB | 35 GB | about IQ3_XXS quality with about 20% less GPU memory |
Gyro runs on NVIDIA and AMD (experimental) cards and on Intel Arc cards on Linux (tested on an Arc A750; the Arc engine is built from this release's source, see [INTEL.md](INTEL.md)) and, from 0.2.2, on Apple Silicon Macs ([MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts): Gyro-S is measured on a 32 GB Mac mini, Gyro-M only tensor by tensor). Gyro reads pictures (the same rule as above). On NVIDIA and AMD it needs
engine 0.1.45.1 or newer; if the engine is older, the app says "This model needs a newer engine" and offers Update
engine. Gyro gives correct output on RTX 50 cards with engine 0.1.45.3. Details, measurements and credit:
[GYRO.md](GYRO.md).
Gyro runs on NVIDIA and AMD (experimental), on Intel Arc under Linux, and on Apple Silicon Macs. It reads pictures.
On an RTX 3060 12 GB and an R9700, Gyro-M decodes 16 to 21% slower than Q2_0. Details, measurements and credit:
[GYRO.md](GYRO.md). Intel Arc: [INTEL_ARC.md](INTEL_ARC.md). Mac: [MAC.md](MAC.md#gyro-s-and-gyro-m-agentionais-trellis-experts).
```
START-HERE.bat --setup --family gyro --model Gyro-S
```sh
strata launch --family gyro --model Gyro-S
```
<a id="nvfp4"></a>
<a id="nvfp4"></a><a id="nvfp4-qad"></a>
### NVFP4: three versions (NVIDIA cards)
### NVFP4 (NVIDIA cards)
The sixth version in setup's menu (`--family nvfp4`) keeps the experts in NVIDIA's 4-bit NVFP4 format. There are three
versions of it. The app shows them by who made the weights:
`--family nvfp4` keeps the experts in NVIDIA's 4-bit NVFP4 format. There are three versions:
| App name | Setup name (`--model`) | What it is | Download |
| App name | `--model` | What it is | Download |
| --- | --- | --- | ---: |
| **NVFP4 (Strata's version)** | `NVFP4` | Strata's own conversion (GPTQ; the ready-made pack is sergqwer's strata-nvfp4). The best quality of the three | 128 GB |
| **NVFP4 (NVIDIA's version)** | `NVFP4-NVIDIA` | NVIDIA's own NVFP4 checkpoint (plain rounding), converted for Strata. The same size and speed as the first | 128 GB |
| **NVFP4 (NVIDIA, trained for FP4)** | `NVFP4-QAD` | NVIDIA's retrained 4-bit model, slightly faster on short prompts ([below](#nvfp4-qad)) | 102 GB (+110 GB of disk to prepare) |
| **NVFP4 (Strata's version)** | `NVFP4` | Strata's own conversion (GPTQ; the ready pack is sergqwer's strata-nvfp4). Best quality of the three | 128 GB |
| **NVFP4 (NVIDIA's version)** | `NVFP4-NVIDIA` | NVIDIA's own checkpoint (plain rounding), converted for Strata. Same size and speed as the first | 128 GB |
| **NVFP4 (NVIDIA, trained for FP4)** | `NVFP4-QAD` | NVIDIA's retrained model, slightly faster on short prompts | 102 GB (+110 GB of disk to prepare) |
All three need about 80 GB of RAM to keep every expert in RAM, run on NVIDIA cards only, and read pictures (the
same rule as above). RTX 50 cards have FP4 tensor cores; other NVIDIA cards run NVFP4 at 8-bit precision. The first two
need engine 0.1.44 or newer, the third 0.1.45 or newer. Details and measurements: [NVFP4.md](NVFP4.md).
All three need about 80 GB of RAM to keep every expert in RAM, run on NVIDIA cards only and read pictures. RTX 50 cards
have FP4 tensor cores; other NVIDIA cards run NVFP4 at 8-bit precision. Details and measurements: [NVFP4.md](NVFP4.md).
```
START-HERE.bat --setup --family nvfp4 --model NVFP4
```
**NVFP4-QAD** is NVIDIA's [Qwen3.8-Flash-Next-FP4-Local](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local),
retrained for 4-bit experts (quantization-aware distillation). Setup downloads NVIDIA's files as they are (102 GB) and
turns them into a Strata pack on your PC once (20 to 40 minutes, about 110 GB more disk while it runs; the 102 GB can be
deleted afterwards). Strata does not redistribute NVIDIA's weights. Pictures use the original model's reader file, which
setup downloads with it. Licenses: NVIDIA Open Model Agreement and Qwen's Community License.
More: [NVFP4.md](NVFP4.md#nvidias-fp4-local-checkpoint).
<a id="nvfp4-qad"></a>
#### NVIDIA's retrained 4-bit model (NVFP4-QAD)
NVIDIA retrained Qwen3.8-Flash-Next for 4-bit experts (quantization-aware distillation) and published it as
[Qwen3.8-Flash-Next-FP4-Local](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local). In setup's NVFP4 family it is
the size **NVFP4-QAD**. It is not a ready-made pack: setup downloads NVIDIA's files as they are (102 GB) and **turns them
into a Strata pack on your PC** (once, 20-40 minutes; about 110 GB more disk while it runs, and the 102 GB can be deleted
afterwards). Nothing of NVIDIA's weights is redistributed by Strata.
| Size | Download | Memory it needs | In plain words |
| --- | ---: | ---: | --- |
| **NVFP4-QAD** (app: NVFP4 (NVIDIA, trained for FP4)) | 102 GB (+110 GB of disk to prepare) | like NVFP4: ~80 GB of RAM keeps every expert in RAM | NVIDIA's FP4, trained for FP4 - slightly faster on short prompts than the other two NVFP4 versions (Strata's version has the best quality) |
NVIDIA cards only. It reads pictures (through the original model's picture reader file, which setup downloads with it). Its licenses are the NVIDIA Open Model Agreement and Qwen's Community License.
Technical details and measurements: [NVFP4.md](NVFP4.md#nvidias-fp4-local-checkpoint).
```
START-HERE.bat --setup --family nvfp4 --model NVFP4-QAD
```sh
strata launch --family nvfp4 --model NVFP4
```
### OrcaRouter Uncensored IQ3_XXS
For **OrcaRouter's Flash-Next Uncensored IQ3_XXS**, see the [manual compatibility setup](ORCA.md). It needs an
explicit packing conversion and is not an installer menu option.
For **OrcaRouter's Flash-Next Uncensored IQ3_XXS**, see the [manual compatibility setup](ORCA.md). It needs an explicit
packing conversion and is not an installer menu option.
## Qwen3.6-35B-A3B
A smaller model from the same maker: [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (35 billion
parameters, 3 billion active per token, 256 experts, Apache 2.0 license). Strata runs [Unsloth's GGUFs with the MTP
draft layer](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) on its own engine path, chosen by the file. It is
in the app's model list and in setup (`--family qwen36`). It runs on NVIDIA and AMD cards; the Intel Arc engine and the
Mac engine do not run it. It reads pictures and serves several requests at once.
parameters, 3 billion active per token, 256 experts, Apache 2.0). Strata runs
[Unsloth's GGUFs with the MTP draft layer](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) on its own engine
path, chosen by the file. It is in the app's model list and in setup (`--family qwen36`). It runs on NVIDIA and AMD
cards; the Intel Arc engine and the Mac engine do not run it. It reads pictures and serves several requests at once.
| Size | File | RAM setup plans for | In plain words |
| --- | ---: | ---: | --- |
@@ -254,24 +268,75 @@ Mac engine do not run it. It reads pictures and serves several requests at once.
| **Q8_0** | 37.8 GB | 52 GB | 8-bit, the closest to the full model |
The picture reader file adds 0.9 GB. The engine keeps every expert it can on the graphics card and computes the rest on
the CPU. Measured with engine 0.1.45 and UD-Q4_K_XL: decode 117.8 tokens/s on a Radeon AI PRO R9700 (32 GB) and 59.7
tokens/s on an RTX 3060 (12 GB, most experts on the CPU). Pictures, requests at once, the full measurements and the
limits: [QWEN36.md](QWEN36.md).
the CPU.
```
START-HERE.bat --setup --family qwen36 --model UD-Q4_K_XL
**Measured on:** UD-Q4_K_XL, decode only.
| Hardware | Decode |
| --- | ---: |
| Radeon AI PRO R9700 (32 GB) | 117.8 tokens/s |
| RTX 3060 (12 GB), most experts on the CPU | 59.7 tokens/s |
Pictures, parallel requests, full measurements and limits: [QWEN36.md](QWEN36.md).
```sh
strata launch --family qwen36 --model UD-Q4_K_XL
```
## Adding or switching models
- **In the app:** the Models page lists every model with its size and whether it fits your computer; pick one and press
the install button.
- **In the Terminal:** `strata launch` has a Model step with the same list (and "Use model files I already have").
`strata models list` shows what is installed; `strata models import <folder or .gguf> --plan` shows what a set of files
you already have would be, without changing anything.
- **With the installer:** `SETUP.bat` (the same as `START-HERE.bat --setup`; on Linux `./setup.sh --setup`) adds another
model any time.
- **In the app:** the Models page lists every model with its size and whether it fits your computer. Pick one and press the install button.
- **In the Terminal:** `strata launch` has a Model step with the same list and "Use model files I already have".
`strata models list` shows what is installed. `strata models import <folder or .gguf> --plan` shows what a set of
files you already have would become, without changing anything.
- **From a source checkout only:** `SETUP.bat` or `./setup.sh --setup` adds another model ([SOURCE_INSTALL.md](SOURCE_INSTALL.md)).
Files another model shares are not downloaded again (the Coder uses the original's shard 2 and vision encoder). With more
than one model installed, `START-HERE.bat` asks which one to start; `run-<model>.bat` (Linux: `run-<model>.sh`) starts one
directly.
Shared files are reused. In the Terminal, choose a model in `strata launch`, pass an explicit family and model, or use a
saved `--preset`. Add `--restart` when replacing a running launch. Generated `run-<model>` scripts belong to a source
checkout, not to the installed app.
## Speed on your own PC
`--tune quick` and `--tune thorough` measure settings for the selected model, and the app offers tuning too. Results
depend on the backend and machine. All speed measurements, long-context numbers and estimates for other cards are in
[DETAILS.md](DETAILS.md#speed-measured). AMD measurements per card are in [AMD_HIP.md](AMD_HIP.md#rdna4-gfx1201).
Measured Strata on your own PC? See [Community benchmark results](COMMUNITY_BENCHMARKS.md).
<a id="how-fast-is-each-size"></a>
## History / older measurements
These numbers compare the sizes with each other. Newer engines are faster on some cards, and the tables were not
re-measured in full for the current release.
**Measured on:** RTX 5070 (12 GB), Ryzen 5 7600, 64 GB RAM; ISTA-DASLab sizes; engine 0.1.26 (Q2_0 row: engine 0.1.36).
| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt (32K) |
| --- | ---: | ---: | ---: |
| **Q2_0** | 94 tokens/s | 76 tokens/s | 2,650 tokens/s |
| **IQ2_XS** | 79 tokens/s | 63 tokens/s | 2,090 tokens/s |
| **IQ3_XXS** | 62 tokens/s | 49 tokens/s | 1,750 tokens/s |
| **IQ3_S** | 53 tokens/s | 46 tokens/s | 1,620 tokens/s |
| **Coder** (IQ1_M) | 55 tokens/s | 43 tokens/s | 2,180 tokens/s |
**Measured on:** AMD RX 9070 XT (16 GB), Ryzen 9 3900X, 47 GB RAM, Linux; engine before 0.1.40, 2026-10-02. Q2_0 is
above setup's RAM estimate for 47 GB and was installed with `--model Q2_0 --yes`.
| Size | Writes answers (short chat) | Writes answers (128K context) | Reads your prompt (32K) |
| --- | ---: | ---: | ---: |
| **Q2_0** | 60 tokens/s | 48 tokens/s | 1,160 tokens/s |
| **IQ2_XS** | 52 tokens/s | 36 tokens/s | 1,110 tokens/s |
| **Coder** (IQ1_M) | 44 tokens/s | 33 tokens/s | 1,420 tokens/s |
- A token is about three quarters of a word. A 4K prompt reads at 910 to 1,580 tokens/s on the RTX 5070; a 32K prompt takes about 15 seconds with Q2_0.
- The older NVIDIA `--calibrate` run on the RTX 5070 PC raised Coder speed by 7%.
**Measured on:** AMD Radeon AI PRO R9700 (32 GB), Unsloth files, engines before 0.1.45.3 (conditions in [QUANTS.md](QUANTS.md)).
| Size | Story | Code | 32K prompt |
| --- | ---: | ---: | ---: |
| **UD-Q4_K_XL** | 63 tokens/s | 87 tokens/s | 1,110 to 1,320 tokens/s |
| **UD-Q5_K_XL** | 57 to 60 | 82 | 810 to 1,260 |
| **UD-Q6_K_XL** | 54 | 76 | 1,020 |
| **Q8_0** | 53 to 57 | 66 to 74 | 1,230 to 1,300 |
| *IQ3_S, same card* | 75 | 95 | - |
docs/MULTI_GPU.md
+235 -466
View File
@@ -1,532 +1,301 @@
# Strata on two or more GPUs (layer split)
One model can run across several NVIDIA cards in one PC. The layers are split into contiguous ranges, one per GPU:
the first card runs layers 0 to K-1, the next card runs K onward, and so on; the last card also runs the output head
and the draft (MTP) layer. Each card keeps an expert cache for **its own layers only**, so two cards hold about twice
the experts one card holds - for the Coder model on a 16 GB + 24 GB pair, nearly all of them, which is where the
speed comes from (decode then barely touches the CPU pool).
This page is for people with two or more graphics cards of the same make in one PC. A layer split puts one model on
several cards. NVIDIA is described here; AMD limits and measurements are in [AMD_HIP.md](AMD_HIP.md).
This is pipeline (layer) parallelism, not tensor parallelism: a token crosses from one card to the next once per
verify window (a few hundred KB through pinned RAM), not twice per layer. No NVLink or peer-to-peer access is
needed; cards on x4 or x1 slots work, and the PCIe share of each card is probed on its own link. A `--pcie-frac` you give
is every card's share and skips those probes; there is no per-card setting yet.
**Two models on two cards (the app).** A layer split puts one model on several cards. The other way to use two cards is two
models, one per card, each answering on its own. In the app, turn on **Run multiple models** on the Models page: the card
list then says which card a loaded model uses ("In use by ..."), starts that card as Off for the next model, and
explains a failed start on a card that is taken. Measured on a PC with Radeon AI PRO R9700 cards (two of four, Linux): Q2_0 on
one card and IQ3_XXS on another both loaded (33 s and 59 s with the files cached, 114 s the first time), both answered chats
(about 60 tokens/s each), the Monitor listed both, and unloading one left the other answering. This was run with the AMD build only;
the same app code drives NVIDIA cards, but we have not run two models on two NVIDIA cards.
**One make of card per engine.** An engine drives one make of card (NVIDIA, AMD or Intel). The app, `strata launch` and setup
offer only the cards the engine can drive, never split a model across two makes, say in one plain line when the other make has
the bigger card, and say that two models at once on two makes is not supported.
> **On this page:** [Using it](#using-it) · [Main, Helper, Off](#main-helper-off) · [How the split works](#how-the-split-works) ·
> [Tuning options](#tuning-options) · [Pipeline windows](#pipeline-windows) · [Two models on two cards](#two-models-on-two-cards) ·
> [DeepSeek](#deepseek-v41-flash) · [Limits](#limits) · [History](#history--older-measurements)
## Using it
**Nothing to type.** `START-HERE.bat` (Linux: `./setup.sh`) lists your NVIDIA cards and says for each one whether
Strata can use it:
**Desktop app:** choose Main, Helper or Off for each card. **Terminal:** run `strata launch` and use the Cards step, or name
the cards:
```sh
strata launch --gpu 0,1
strata launch --gpu-roles 1=main,0=helper # cards you do not name are Off
```
Your NVIDIA GPUs:
GPU 0: NVIDIA GeForce RTX 5080, 16 GB VRAM - can be used
GPU 1: NVIDIA GeForce GTX 1080 Ti, 11 GB VRAM - not supported - older than the RTX 20 series (compute capability 6.1; Strata needs 7.5 or newer)
GPU 2: NVIDIA GeForce RTX 3090, 24 GB VRAM - can be used
...
1) RTX 3090 + RAM: fits: needs 44 GB of the 70 GB available (recommended)
2) RTX 3090 + RTX 5080 + RAM (layer split): fits: needs 44 GB of the 84 GB available
prompts are read about 20% faster; the cards share the layers (docs/MULTI_GPU.md)
3) RTX 5080 + RAM: fits: needs 44 GB of the 66 GB available
Which GPUs? [1]:
Choose a model that fits the combined memory. A split can add capacity without adding speed.
**Source checkout only:** `START-HERE.bat` and `./setup.sh` have their own card picker. Their flags are not `strata launch` flags:
| Source setup flag | Meaning |
| --- | --- |
| `--gpus 0,2` | These cards together, numbered as `nvidia-smi` numbers them. Strata puts the faster card last. Remembered. |
| `--gpus all` | Every card that can share the model. |
| `--gpu 0` | One card (at a start: for that start only). |
| `--layer-split auto` | The default. Or the first layer of each later card, for example `18` or `16,32`. |
| `--gpu-roles 1=main,0=helper` | Roles, saved. |
Or edit an existing run config (`strata-*.json`) and restart. Merge these fields into the config; this is not a complete config:
```json
{
"gpu": [0, 2],
"layer_split": "auto"
}
```
Docker: `GPU_ROLES=1=main,0=helper` is passed as `--gpu-roles` ([DOCKER.md](DOCKER.md)).
## Main, Helper, Off
Every card has one of three roles:
Every card has one of three roles.
| Role | What it does |
|---|---|
| **Main** | Runs the **last** stage of the split: the output head, the draft layer and the check of its guesses. A prompt chunk waits on it. Exactly one card is Main. |
| **Helper** | Holds a share of the layers and of the expert cache, so more of the model fits on graphics cards instead of being read from the drive. Prompts are read about 20% faster. |
| --- | --- |
| **Main** | Runs the last stage of the split: the output head, the draft layer and the check of its guesses. A prompt chunk waits on it. Exactly one card is Main. |
| **Helper** | Holds a share of the layers and of the expert cache, so more of the model sits on graphics cards instead of being read from the drive. The effect on speed depends on the cards and the workload. |
| **Off** | Strata does not touch the card (a game, the screen, a card that is too small). |
The Main card is the **last** card in the engine's order, not the first. The first card runs the first layers and, by default,
the picture reader. (Older versions of this page said "the first is the main one"; that has not been true since 0.1.41.1.)
The Main card is the last card in the engine's order. The first card runs the first layers and, by default, the picture reader.
**Recommended roles** (the same rule in the app, `strata launch` and setup; `strata_control/gpu_roles.py`): the faster card
is Main (speed is multiprocessors x max clock, the engine's own measure; cards within 5% are the same card, and the one with
more memory wins, then the lower number; two identical cards: the highest number is Main, so the engine order stays 0, 1 as it always was; a card with clearly more memory beats a faster one that is at least half as fast).
Every other card starts as Helper, except a card with **8 GB or less** when a bigger card is there (Off; with only 8 GB cards, say two GTX 1080, the second one is a Helper), built-in graphics beside a card with memory of
its own (Off) and a card the engine cannot run (Off). Every role is one click (the app), one key (`strata launch`: M, H, O;
R goes back to the recommendation) or one line (below).
**Recommended roles** (the same rule in the app, `strata launch` and setup; `strata_control/gpu_roles.py`):
Only a few things are warned about, in one short line each: built-in graphics as Main, a card the engine cannot run, too
little memory for the chosen setup (part of the model would be read from the drive). **Every card Off** is allowed. The CUDA,
HIP and SYCL engines cannot start without a card, so the server then says so in one line and stops; an engine that can run on
the processor (the Mac engine) says "Performance is degraded on CPU only." and goes on.
- The faster card is Main. Speed is multiprocessors x max clock. Cards within 5% count as the same card, and the one with more memory wins, then the lower number. Of two identical cards the highest number is Main. A card with clearly more memory beats a faster one that is at least half as fast.
- Every other card starts as Helper, except a card with 8 GB or less when a bigger card is present (Off; with only 8 GB cards the second one is a Helper), built-in graphics beside a card with memory of its own (Off), and a card the engine cannot run (Off).
- A role is one click (app), one key (`strata launch`: M, H, O; R restores the recommendation) or one line (above).
**Setting roles**
Only a few things are warned about, one short line each: built-in graphics as Main, a card the engine cannot run, and too
little memory for the chosen setup. All cards Off is allowed. The CUDA, HIP and SYCL engines cannot start without a card and
say so; the Mac engine says "Performance is degraded on CPU only." and goes on.
```
strata launch --gpu-roles 1=main,0=helper cards you do not name are Off
./setup.sh --gpu-roles 1=main,0=helper (START-HERE.bat --gpu-roles ... on Windows); saved
strata launch the Cards step: M Main, H Helper, O Off, R recommended, D details
```
or in the run settings (`strata-*.json`), which is what the app writes:
**In the run config** (what the app writes). Merge these fields into the existing config:
```json
"gpu_roles": {"1": "main", "0": "helper", "2": "off"},
"gpu": [0, 1],
"gpu_order": "as_given"
{
"gpu_roles": {"1": "main", "0": "helper", "2": "off"},
"gpu": [0, 1],
"gpu_order": "as_given"
}
```
`gpu` is the same choice for older engines and the Monitor: the Helpers (slowest first), then the Main card, with the Off
cards left out. The server reads `gpu_roles`; when `gpu` was edited by hand and no longer matches, `gpu` wins and the log says
so (a settings file that names its cards keeps them). Nothing is rewritten on upgrade: a file with only `"gpu": [1, 0]` shows
as Main = the last card, Helper = the first, the rest Off. The recommendation itself is not saved, so a card you add later is
recommended again; a card you add after choosing roles starts Off. Docker: `GPU_ROLES=1=main,0=helper` is passed as `--gpu-roles`.
`gpu` is the same choice for older engines and the Monitor: the Helpers (slowest first), then the Main card, Off cards left out.
The server reads `gpu_roles`. If `gpu` was edited by hand and no longer matches, `gpu` wins and the log says so. A file with only
`"gpu": [1, 0]` shows as Main = the last card, Helper = the first, the rest Off. The recommendation is not saved, so a card you
add later is recommended again; a card you add after choosing roles starts Off.
**One rule for which cards count** (`strata_control/gpu_fit.py` keeps the memory arithmetic): not an integrated chip beside
a real card, at least 4 GB of graphics memory to be listed, 8 GB or less starts Off beside a bigger card. The arithmetic is the app's `fitOn`;
each extra card of a split costs an estimated 2 GB of its own buffers.
**Which cards count** (`strata_control/gpu_fit.py` holds the memory arithmetic): not an integrated chip beside a real card; at
least 4 GB of graphics memory to be listed; 8 GB or less starts Off beside a bigger card. Each extra card of a split costs an
estimated 2 GB of its own buffers.
**Choosing the cards without roles** (at setup or at any start):
**Not supported** (setup says so and names the cards that can be used):
```
--gpus 0,2 these cards together, as nvidia-smi numbers them; Strata puts the faster card last. Remembered.
--gpus all every card that can share the model
--gpu 0 one card (at a start: for that start only)
--layer-split auto (default) or the first layer of each later card, e.g. 18 or 16,32
```
- A card older than the RTX 20 series (compute capability below 7.5). Pascal and Volta cards are admitted when you name them
with `--gpus` (or `--gpu`, `--cuda 12`) and then run the experimental CUDA 12 engine ([OLDER_GPUS.md](OLDER_GPUS.md)).
- A card with less than 8 GB of VRAM together with others, because each card holds a copy of the dense weights and its own
prompt buffers. If you name it with `--gpus`, setup states the risk and asks (`--yes` goes ahead).
- Intel GPUs in a split, and a mix of NVIDIA and AMD cards. An engine drives one make of card. The app, `strata launch` and setup
offer only the cards the engine can drive, and say so when the other make has the bigger card. AMD cards share a model among
themselves: `./setup.sh --backend hip --gpus 1,0` ([AMD_HIP.md](AMD_HIP.md)).
**Not supported** (setup says so and names the cards that can be used instead):
- a card older than the RTX 20 series (compute capability below 7.5: GTX 10 and older), unless you name it: Pascal
and Volta cards (Tesla P100 / P40, GTX 10, V100) are admitted when you choose them with `--gpus` (or `--gpu`,
`--cuda 12`) and then run the experimental CUDA 12 engine, see [OLDER_GPUS.md](OLDER_GPUS.md). Two Tesla P40s
(`"gpu": [0, 1]` in the config) ran the layer split in a community benchmark: IQ2_XS decode 19.6 tok/s on one card,
34.8 on both (#1028, experimental, one report);
- a card with less than 8 GB of VRAM, together with others (each card holds a copy of the dense weights and its
own prompt buffers) - unless you name it with `--gpus`: then setup says the risk and asks (`--yes` with the named
cards goes ahead);
- Intel GPUs, and a mix of NVIDIA and AMD cards. (AMD cards share a model among themselves: `./setup.sh --backend
hip --gpus 1,0`, see [AMD_HIP.md](AMD_HIP.md).)
## How the split works
Or edit an existing config (`strata-*.json`), then restart:
The layers are cut into contiguous ranges, one per card. The first card runs layers 0 to K-1, the next runs K onward, and the
last card also runs the output head and the draft (MTP) layer. Each card keeps an expert cache for its own layers only, so two
cards hold about twice the experts one card holds.
```json
"gpu": [0, 2],
"layer_split": "auto"
```
This is pipeline parallelism, not tensor parallelism. A token crosses from one card to the next once per verify window (a few
hundred KB through pinned RAM). No NVLink or peer-to-peer access is needed, cards on x4 or x1 slots work, and each card's PCIe
share is probed on its own link. A `--pcie-frac` you give applies to every card and skips those probes.
`"layer_split"` is `"auto"` (placed by each card's free VRAM) or the **first layer of each later card**: one rising
number per card after the first, not a count of layers per card. With 4 cards and a 48-layer model, `"24,36,42"`
(or `[24, 36, 42]`) puts layers 0-23 on the first card, 24-35 on the second, 36-41 on the third and 42-47 on the last.
The server checks it before the start and says what is wrong (0.1.39, #644).
**What each card holds**
**Card order with `"auto"`** (NVIDIA, #1352): the faster card (multiprocessors x max clock) goes **last** - the last stage runs the head, the draft layer and the verify, and a prompt chunk waits on it (a 4070 Ti SUPER + 5060 Ti read a 6K prompt in 50 s one way round and 15 s the other). Cards whose scores are within 5% of each other (two of the same model) keep your order; a manual split keeps it too. `"gpu_order": "as_given"` keeps the order you wrote under `"auto"` as well. The engine log line `layer split: card order ...` says when it changed.
- Every card: a copy of the dense weights, its own session state (the KV cache of the full context), its verify window, its
prompt-path buffers, and an expert cache for its layers filled from the expert profile.
- The last card: also the output head and the draft layer (about 0.8 GB).
- Host RAM: the expert arena once, shared by all cards. The CPU pool computes whatever no card holds.
**An unequal pair under `"auto"` (#2644).** The search prices a decode window, and it can give almost every layer to the card it
rates faster. A card with one or two layers holds a handful of experts and clamps the prompt chunk of the whole pipeline
(#1616), so the search then asks for at least 3 layers per card - which on a 4060 Ti 16 GB + 3060 12 GB (48 layers, IQ3_S)
was K=3: the 3060 ran 3 layers, 12 GB idle, 452 tok/s of prefill against 802 at 28 layers on the 3060 (the reporter's sweep of
`"layer_split"` 12, 18, 24, 28, 32: 582, 693, 762, 802, 803 tok/s; the 3060's 360 GB/s of memory bandwidth against the 4060 Ti's
288 counts for more than its SMs). On two cards the engine now first asks for a placement in which each card owns at least its
share of the layers by its room for experts, and the log line says which rule chose the split. Measured on 2x RTX A4000 made
unequal (stage 0 priced at 0.57 of the speed with 3.7 GiB less room, `STRATA_SPLIT_SPEED_SCALE=0.57` and `--vram-reserve-mib 4400`; 14.5K-token
prompt, IQ3_XXS, 5 interleaved pairs): the old rule chose K=3 (3 layers on stage 0), the new one K=20; prompt reading 945 -> 1,441 tok/s (+52%,
5/5 pairs), decode 50.2 -> 65.9 tok/s (+31%, 5/5). Equal cards and three or more GPUs are unchanged.
On x16 + x4 rigs the link-miss scaling (#2007, from a 4x gap) can also move `auto` off the balanced placement (2x RTX 3090, x16 + x4:
K=31 with miss cost x4.20, 114-119 tok/s decode against 130-136 at K=27, #2418): `STRATA_SPLIT_UNIFORM=1` (opt-in, with `"gpu_order": "as_given"`)
prices every link the same, as before 0.1.41.
Prefill that is flat in the prompt length (no gain from 4K to 16K) is the sign of an idle card; **on an unequal pair sweep
`"layer_split"` once** - a 4K-only benchmark cannot see it.
Prompts are read in chunks that flow through the cards in turn: while a later card reads chunk c, the first card reads chunk c+1.
Conversation checkpoints save and restore every card's state.
**Skip the split when the first card holds everything** (opt-in, 0.1.31): `"split_skip_if_fits": true` in the config
(engine flag `--split-skip-if-fits`, with `--layer-split auto`) runs on the first card alone when it holds every
profiled expert plus the context's KV, the draft layer and the reserve, and says so in the log; otherwise the split
stays. On an R9700 32 GB + RX 9070 XT the R9700 holds all of the Coder's experts: with the flag 4K prompts read at
1,776 tok/s instead of 1,244 (split) and decode runs at ~60 tok/s instead of ~51 (16K prompts ~5% slower than split).
**`"layer_split"`** is `"auto"` (placed by each card's free VRAM) or the first layer of each later card: one rising number per
card after the first, not a count per card. With four cards and a 48-layer model, `"24,36,42"` puts layers 0-23 on the first
card, 24-35 on the second, 36-41 on the third and 42-47 on the last. The server checks the value before the start and says what
is wrong.
**Short prompts on a split (0.1.32, #340).** 0.1.30 gave each card's prompt path a loan from its own expert cache,
refilled after every request; on cards that hold nearly all their experts that cost short prompts up to a third of
their speed. 0.1.32 refills all cards at once, uses a smaller streaming ring on a split, and lets a card with free
VRAM keep its own prompt buffers - the same output as 0.1.31, measured on an R9700 + RX 9070 XT: 2K prompts 993 ->
1,265 tok/s, 16K 1,852 -> 1,950, decode unchanged. `STRATA_SPLIT_OWN=1` (opt-in) gives every card its own buffers:
2K 1,450 and 16K 2,227 tok/s there, but a full card then keeps a different set of experts resident, so the output
differs from the default's (stable and coherent); `STRATA_SPLIT_OWN=auto` does that only where the buffers are at
most 12% of each card's VRAM.
**The idle card can help one-chunk prompts (opt-in, `STRATA_PREFILL_HELP=1`).** A prompt that fits one chunk runs the stages one after the other, so while
one card reads its layers the other idles. With it on, each stage hands a share of its streamed experts to the idle card: it
streams them over its own PCIe link into its own (lent) prompt buffers, computes their rows on the MMQ path and sends
them back - `--peer-device`'s peer streaming, without P2P (the activations and the rows go through mapped host memory,
read by copy kernels, so they do not queue behind the expert blobs on either card's copy engine). The share falls with
the prompt (0.41 of the streamed experts at 1.5K tokens, 0.32 at 3K) and is off from ~3.3K tokens, where the stages
overlap anyway and no share paid. Measured on 2x RTX 3090 (UD-Q4_K_XL, no P2P), prompt tok/s without / with: 1.1K 496 /
723, 1.5K 674 / 833, 2K 878 / 1,128, 2.5K 1,017 / 1,296, 3K 1,260 / 1,440; 4K and 8K unchanged, decode unchanged. It
costs no VRAM (the idle card's own prompt buffers) and ~110 KB of mapped host memory per token of the largest chunk it
helped (~360 MB at 3.3K tokens). The rows it computes round like a different MMQ grouping, so the output is not
bit-identical to the default's (it is repeatable: same prompt, same output), which is why it is **opt-in**:
`STRATA_PREFILL_HELP=1` turns it on. Native packs on the MMQ prompt path only (not with the fused prompt kernels,
`STRATA_PF_FUSED=1`, nor with `--peer-device`). With it on, `STRATA_PREFILL_HELP_FRAC=f` fixes the share.
On two Radeon AI PRO R9700 (each limited to 16 GB with `--vram-reserve-mib 17084`, Q2_0 pack, layer split auto, 5 interleaved pairs,
noise arm within 0.2%), `STRATA_PREFILL_HELP=1` gave prompt tok/s without -> with: 700 tokens 638 -> 672 (x1.053, 5/5), 1.5K 839 -> 809
(x0.964), 3K 1,105 -> 929 (x0.840), 6K 1,284 -> 1,211 (x0.943): the PCIe 4.0 x16 links of this box (28.4 GB/s each) make the share lose
from 1.5K tokens up, so it stays off (`STRATA_PREFILL_HELP_FRAC` was not tuned for it).
The engine flags behind it: `--layer-split K1[,K2..]|auto` and `--split-device D1[,D2..]` (the later stages'
devices; default the next visible ones). `--layer-split K --split-device 0` runs both stages on one card sharing
everything - the bit-exact check of the hand-off, not a speed mode.
**The draft layer's prompt K/V in batches on a split (opt-in, `STRATA_SPLIT_MTP_BATCH=1`).** With `--mtp` the draft
layer sits on the last card, and each prompt chunk ends by filling that layer's K/V for the chunk's rows. On one card the
prompt path does it in batches (`Prefill::draft_kv`, a few large matrix products per chunk); on a split the drafter's own
pass always did it, one graph launch and four device copies per group of 8 rows (1,024 groups per 8,192-token chunk).
With the variable set, the last card's prompt path batches it as it does on one card. Restart-only; default off, so the
default start is unchanged. It needs a `--native` pack (the GGUF-form token table) and a paged or ring (`--kv-resident`)
draft K/V (`STRATA_MTP_BATCH_RING=0` sends a ring back to the old pass); otherwise it says why once and keeps the old pass:
`strata serve: draft layer prompt K/V batched on CUDA<d> ...` or `strata serve: STRATA_SPLIT_MTP_BATCH=1 declined, the
drafter's own pass runs: <reason>`. The draft layer's K/V come out of Q8_1 x Q8_0 MMQ instead of its own mmvq, so the
drafts, and how many are accepted, can move; the target's tokens are decided by the verify window.
`tools/split_mtp_batch_parity.py` compares the greedy texts of the two arms. Measured on one machine (2x RTX 3080 20 GB
at 220 W, Xeon E5-2696 v4, UD-Q4_K_XL, `--layer-split 23`, `--batch-mtp`, two slots, `--prefill auto:16384`), two full
restarts per arm in the order off, on, off, on, the same binary in every arm: pooled medians of the prompt read were 1,996
tok/s without and 2,129 with at 25K tokens (+6.7%), 2,448 and 2,553 at 51K (+4.3%), 2,846 and 2,940 at 104K (+3.3%); all 8
on/off ratios of medians are above 1 (1.03 to 1.08), and the two off restarts differ by at most 2% at 25K and 0.3% at the
other sizes. The host's "after each chunk" time per request fell from 0.9-1.3 s to 0.1-0.25 s (one restart per arm). Greedy
output (`--adapt-every 0 --suffix-draft 0`, five prompts of 256 tokens and one 25.5K-token prompt) was identical in 6 of 6
prompts in all five comparisons, the two runs inside one process (A/A) included, and the drafts accepted were 826 of 1,210
in every run, so no difference in acceptance was visible. Decode is not shown to be unchanged: the medians with the variable
set were about 3% lower (pooled solo 82.6 -> 80.2 tok/s, two streams 93.2 -> 90.8), and the same configuration restarted
twice moved up to 4.7%, so two restarts per arm can neither confirm nor exclude it; the change touches only the prompt path.
**Each card loads only its own layers' dense weights** (0.1.39, PR #639) with explicit split points (`--layer-split
27`, not `auto`): every card used to keep a full copy (~3.4 GB for the Coder) though its stage reads only its own
layers, and the VRAM it frees goes to that card's expert cache (2x MI50 16 GB, Coder: 8,819 -> 10,626 experts in
VRAM, decode 39.2 -> 41.7 tok/s). Opt-in for now, on AMD and NVIDIA alike: `STRATA_STAGE_TRIM=1` (please report how
it goes).
A card holding more experts can change which experts run on the GPU, so the output can differ slightly from a run
without it.
**The resident RAM mode works on a split** (`--resident-experts`, the low-RAM mode). The RAM copy of the experts
leaves out the ones every card's cache holds, not only the first card's, and when the rest does not fit whole it keeps
the hottest by the expert profile over all the layers. An adaptive swap copies the evicted expert back into RAM from
the card that owns its layer. Before, `--resident-experts` with a split ran as `--mmap-experts`, and setup recommended
one card in the low-RAM mode; with an engine that has it (`RESIDENT_SPLIT_ENGINE` in setup.py), setup keeps the cards
together and the experts no card holds in RAM. The same copy, sized by a RAM budget (`--resident-budget-gib`, the
Unsloth UD-Q4_K_XL and UD-IQ4_XS configs), is kept on a split as well ([UNSLOTH_Q4.md](UNSLOTH_Q4.md)). Swift 1.5 IQ3_XXS at
160K (q4_0 KV, `--prefill 4096`, `--spec 4` with the stock draft layer), RTX 4060 Ti (layers 0-19) + RTX 5080 (20-47),
i9-14900KF, 32 GB of RAM, Windows 11, four greedy prompts at a time, decode tok/s:
| | first four prompts | after three more rounds |
|---|---|---|
| 5080 alone, `--resident-experts` | 25 | 29 |
| split, `--mmap-experts` (what `--resident-experts` became on a split) | 32 | 64 |
| split, `--resident-experts` (22 GiB of experts locked in RAM) | 71 | 69 |
The split with `--mmap-experts` catches up once the OS file cache holds the experts, on a PC with nothing else
running; the resident copy is there from the first request and stays locked when other programs need the RAM. With
`--pcie-frac 0 --adapt-every 0` the split's greedy output is the same with either mode.
**The prompt path's loan on a split and its streamed ring (`STRATA_SPLIT_RING`).** Under a split the prompt path borrows
the tail slots of each card's expert cache for its buffers (they are given back after the prompt), and the resident RAM
copy keeps those experts too, so a borrowed expert is not read back from the model file during a prompt. How many slots
are borrowed depends on the prompt path's streamed ring: a split uses 96 ring slots when at least 75% of the (layer,
expert) pairs sit in some card's cache, and `STRATA_SPLIT_RING=N` sets N slots (`0` goes back to the pinned-share rule).
The ring used to be chosen after the RAM copy's regions were sized, so the regions were sized for a different ring than
the prompt path then used: with `STRATA_SPLIT_RING=384` it borrowed 5,396 slots and the copy kept 4,662, and the
other 734 were read from the model file in every chunk. The ring is now chosen first (`Prefill::set_ring_override`
says it must be set before the buffers are counted). The start-up line `strata serve: lend sizing: the prompt path
borrows N slots (ring R slots at chunk C), K of them keep their experts in RAM too` shows both numbers, and a
`WARNING` follows when K is below N. Without a split, or without `--resident-experts`, nothing changes. With a split
the order of the two steps changed for every ring choice, the 96-slot rule included: its regions used to be sized for
the larger default ring, so there the copy probably kept more slots than were borrowed (not measured). The 5,396 /
4,662 figures were measured on the test machine (2x RTX 3080 20 GB at 220 W, Xeon E5-2696 v4, UD-Q4_K_XL,
`--layer-split 23`, resident RAM mode, `--prefill auto:16384`, 2 slots), one restart per arm, with `STRATA_SPLIT_RING=384`
and `STRATA_PREFILL_RING` unset. Before the change a 104K-token prompt made 413 blob reads and read 10.3 GB of experts
from the model file per request; after it, 0, with all 5,396 borrowed slots kept in RAM, for 2.15 GiB more pinned RAM
(13.61 -> 15.76 GiB). 25K and 51K prompts read nothing from the file in either case. With the override unset and
`STRATA_PREFILL_RING=384` the lend lines are identical before and after (5,396 / 5,396, 15.76 GiB). No speed gain is
shown: the medians of three 104K reads were 2,803 tok/s before and 2,907 after, but the spread before is 2,540-2,897
(its first read faulted 4.2 GB in from NVMe) and there is no A/A restart. Here the page cache absorbed the 10 GB per
request; with less spare page cache it would be disk traffic. Not measured: both ring variables unset.
**A separate VRAM reserve for the later cards:** `--vram-reserve-later-mib N` (default: `--vram-reserve-mib`'s value).
The card that drives the monitors needs more headroom than one that drives none; with the display on the last card,
`--vram-reserve-mib 300 --vram-reserve-later-mib 1800` gives the first card's cache that VRAM.
**auto** tries every placement (all of them for two, three or, since 0.1.40, four cards; proportional to the free VRAM beyond that) and
keeps the one whose caches would hold the most of the expert profile, hottest pairs weighted most; ties go to the
placement that leaves the fullest card the most room. The startup log prints the choice:
**`auto`** tries every placement for two, three or four cards (proportional to free VRAM beyond that) and keeps the one whose
caches hold the most of the expert profile, hottest pairs weighted most. Ties go to the placement that leaves the fullest card
the most room. The startup log prints the choice:
```
strata generate: layer split auto: K=19 - the caches hold 11767 of 12288 profiled pairs (fullest device 100%)
strata serve: layer split: layers 0-18 (CUDA0), 19-47 (CUDA1), one hand-off per window
```
## What each card holds
**Card order with `auto`:** the faster card goes last, because the last stage runs the head, the draft layer and the verify, and
a prompt chunk waits on it. Cards within 5% keep your order, and so does a manual split. `"gpu_order": "as_given"` keeps your
order under `auto` as well. The log line `layer split: card order ...` says when the order changed.
- **every card**: a copy of the dense weights (~3.4 GB for the Coder), its own session state (the KV cache of the full
context), its verify window and its prompt-path buffers, and an expert cache for its layers filled from the profile;
- **the last card**: also the output head and the draft layer (~0.8 GB);
- **host RAM**: the expert arena once, shared by all cards (the CPU pool computes whatever no card holds).
**Unequal cards under `auto`.** The search asks for at least 3 layers per card and, on two cards, first for a placement in which
each card owns at least its share of the layers by its room for experts. The log line says which rule chose the split. Prefill
that stays flat from 4K to 16K tokens is the sign of an idle card. On an unequal pair, sweep `"layer_split"` once, because a
4K-only benchmark cannot see it. On x16 plus x4 rigs the link-miss scaling can move `auto` off the balanced placement;
`STRATA_SPLIT_UNIFORM=1` (with `"gpu_order": "as_given"`) prices every link the same.
Prompts are read in chunks that flow through the cards in turn; while a later card reads chunk c, the first card
already reads chunk c+1. (0.1.44, AMD gfx12: the auto chunk on a split stays at 8192, not the single-card 16384: a 16K prompt is then two chunks
that overlap instead of one, a 32K one four instead of two. Same tokens; 2x / 3x / 4x R9700 prompt tok/s at 16K +36% / +45% / +53%,
32K +18% / +21% / +29%, 64K +4% / +12% / +17%, 8K unchanged. `STRATA_SPLIT_CHUNK16K=1` keeps 16384.) Conversation checkpoints save and restore every card's state; the adaptive expert swaps copy
into the card that owns the layer.
## Tuning options
## Overlapped hand-off (`STRATA_SPLIT_OVERLAP=1`, opt-in)
These are engine flags and environment variables. Put flags in the config's `"args"` and variables in its `"env"` object, then
restart. Defaults are off unless the table says otherwise.
By default each verify window runs the stages one after another: the host syncs stage 1, then stages and launches
stage 2's graph, and so on. `STRATA_SPLIT_OVERLAP=1` (stages on different cards only) gives the hand-off a ready
flag: the earlier stage's GPU writes the hand-off with volatile stores and raises the flag (`handoff_publish`), the
next stage's graph is staged and launched on a helper thread while the earlier stage is still running and waits for
the flag on its GPU (`wait_flag_ge`), and the host goes straight from serving one stage's layers to the next, syncing
the earlier stage only once the chain is done. It takes one stream sync and one graph launch per window off the
critical path. Batch windows (several conversations) keep the serial order. It is off beside `--pipeline-windows`, whose loop overlaps the stages its own way.
| Setting | What it does | Notes |
| --- | --- | --- |
| `"split_skip_if_fits": true` (flag `--split-skip-if-fits`) | Runs on the first card alone when it holds every profiled expert plus the context's KV, the draft layer and the reserve. Otherwise the split stays. | Opt-in. Needs `--layer-split auto`. |
| `STRATA_SPLIT_OWN=1` or `auto` | Gives every card its own prompt buffers. `auto` does it only where the buffers are at most 12% of each card's VRAM. | Output differs from the default's (stable and coherent). |
| `STRATA_PREFILL_HELP=1` | The idle card computes a share of the streamed experts of a one-chunk prompt. `STRATA_PREFILL_HELP_FRAC=f` fixes the share. | Helps prompts up to about 3.3K tokens on slow links only; not bit-identical; native packs on the MMQ prompt path only. |
| `STRATA_SPLIT_MTP_BATCH=1` | The last card fills the draft layer's prompt K/V in batches, as on one card. | Needs a `--native` pack and a paged or ring draft K/V; says why once if it declines. |
| `STRATA_STAGE_TRIM=1` | Each card loads only its own layers' dense weights (needs explicit `--layer-split K`, not `auto`). The freed VRAM goes to that card's expert cache. | Output can differ slightly. |
| `--vram-reserve-later-mib N` | A separate VRAM reserve for the later cards (default: `--vram-reserve-mib`). Use it when the display sits on the last card. | `--vram-reserve-mib 300 --vram-reserve-later-mib 1800` gives the first card that VRAM. |
| `--prefill-pipe R` (or `STRATA_PREFILL_PIPE=R`) | Reads each prompt in the chunk count with the least pipeline time for b/a = R tokens. `1` evens out today's chunk count. | Opt-in. `./setup.sh --calibrate` measures it per PC and writes it to the config. |
| `STRATA_SPLIT_RING=N` | Sets the prompt path's streamed ring to N slots (`0`: the pinned-share rule). A split uses 96 when at least 75% of the (layer, expert) pairs sit in some card's cache. | The startup line `lend sizing: ...` shows the result. |
| `STRATA_SPLIT_CHUNK16K=1` | Keeps the 16384-token auto chunk on AMD gfx12 splits (default 8192). | |
| `STRATA_SPLIT_OVERLAP=1` | Overlapped hand-off between stages on different cards. | Helps AMD; no gain on NVIDIA. See [below](#overlapped-hand-off). |
| `--split-device D1[,D2..]` | The later stages' devices (default: the next visible ones). `--layer-split K --split-device 0` runs both stages on one card; a bit-exact hand-off check, not a speed mode. | |
Measured on 2x RX 7900 XTX (gfx1100, ROCm, Linux), Ryzen 9 9900X, 52 GB RAM, Swift 1.5 IQ3_XXS, 128K context, 8-bit
KV, layers 0-25 / 26-47; 512-token greedy answers, median of 8 runs:
**The resident RAM mode works on a split.** `--resident-experts` (the low-RAM mode) leaves out of the RAM copy the experts any
card's cache holds. An adaptive swap copies the evicted expert back into RAM from the card that owns its layer. A RAM budget
(`--resident-budget-gib`) works on a split as well ([UNSLOTH_Q4.md](UNSLOTH_Q4.md)). With `--pcie-frac 0 --adapt-every 0` the
split's greedy output is the same in either mode.
| | serial (default) | `STRATA_SPLIT_OVERLAP=1` | one card |
|---|---|---|---|
| decode, code prompt | 53 tok/s | 70.1 tok/s | ~70 tok/s |
| decode, prose prompt | 45 tok/s | 55.9 tok/s | ~68 tok/s |
**Under WDDM (Windows and WSL2)** only 8 GiB of the expert arena is pinned on a multi-GPU run; the rest streams through the
pinned staging ring. Linux pins the whole arena. `STRATA_ARENA_PIN_GIB=N` pins at most N GiB, `0` the whole arena, on any OS.
Greedy output is byte-identical with the flag at 0 and 1; the quiz set scored the same (47/51). On NVIDIA it
gives nothing: 2x RTX A4000 (IQ3_XXS, 5 interleaved pairs, 0.2.1 engine), decode story +0.3 %, code +2.1 %, facts -0.3 % against
+2.8 % of noise, prompts +0.1 %, with the same tokens. A sync and a graph launch cost less there. The second card then reads about 94 %
busy in `nvidia-smi` (a kernel waiting for the flag), against 58 % without the flag, with no more tokens per second.
### Overlapped hand-off
The reading stage's wait for the flag is bounded on the GPU: `STRATA_SPLIT_WAIT_MS` (default 30000, 0 = no bound).
If the flag does not arrive in time, the wait marks an error word and returns, the window's graph completes, and the
host fails that window with an error naming the setting (the server then restarts the engine) instead of using a
stale hand-off. Normally the host's own per-layer wait (20 s, #267) ends a stalled window first and releases every
GPU wait, the hand-off's included. Test hook: `STRATA_TEST_HANDOFF_DROP=N` withholds the flag in the first stage's
N-th window, as if the publish were lost.
## Prompt chunks sized for the pipeline (`--prefill-pipe`, opt-in; setup's `--calibrate` sets it)
By default each verify window runs the stages one after another. `STRATA_SPLIT_OVERLAP=1` (stages on different cards only) gives
the hand-off a ready flag, so the next stage's graph is launched while the earlier stage still runs and waits on the flag on
its GPU. It takes one stream sync and one graph launch per window off the critical path. Batch windows keep the serial order, and
it is off beside `--pipeline-windows`. The wait is bounded by `STRATA_SPLIT_WAIT_MS` (default 30000, `0` = no bound); on timeout
the window fails with an error naming the setting and the server restarts the engine.
`--prefill auto` takes the largest chunk the buffers allow (up to 8192), which is what one card wants: a chunk costs a
fixed pass over its layers' experts (b: dequantized, or streamed over PCIe where they are not in VRAM) plus a part per
token (a), so few large chunks are cheapest. A layer split reads a prompt as a pipeline - stage s reads chunk c while
stage s+1 reads chunk c-1 - and then takes about sum_i (b + a C_i) + (S - 1) max_i (b + a C_i) for chunks C_i over S
stages: large or uneven chunks keep the later cards waiting (a 10.7K-token prompt in a 7680 and a 3011 chunk over
four stages: the first stage carried 24 of the 36 s), small ones pay b again and again.
## Pipeline windows
`--prefill-pipe R` reads each prompt in the chunk count with the least pipeline time for b/a = R tokens, when that
beats today's split by 3% or more: every count from the fewest the buffers allow is tried, each chunk evened out on the
256-token grid (at least 512, at most the buffers' chunk; keep `--prefill auto` so long prompts may use large chunks).
b/a differs by an order of magnitude between rigs - about 700 tokens on four GP100 dies (every expert in VRAM, slow
arithmetic), about 5000 on two RTX 3090 Ti (part of the experts streamed per chunk, fast arithmetic) - and a wrong one
costs: a fixed 1024 read 19-24% slower on the two RTX 3090 Ti and 22-36% slower on two RTX 4090 (reported in the PR by
MistyMoonR). So it is neither guessed nor learned from the prompts the engine serves (that made a prompt's chunks, and
so its output bits, depend on how busy the PC happened to be): like the PCIe share and the draft floor, it is a
setting of the PC that setup's `--calibrate` measures once (`tools/calibrate.py`: on a layer split, the prompt read
speed of a 2K- and an 8K-token prompt - the same text every time, after a header of its own so nothing is reused -
with the chunks off, evened out and b/a 768, 1536, 3072 and 6144; the best is kept only when it reads faster than off
in each of three paired rounds and by more than 3% in the median) and writes into the config. Measured so: four GP100
dies keep b/a 768 (355-356 against 219-220 tok/s in every round, +62%); two RTX 3090 Ti keep it off (nothing read
faster than off, b/a 768 20% slower). A first version that read a different stretch of text per value and confirmed in
two rounds picked 1536 on the RTX 3090 Ti pair from one fast read - where the experts are streamed, the read speed
depends on the text. `--prefill-pipe 1` needs no
b/a: it is `STRATA_PREFILL_EQUAL=1`'s rule (#693) - today's chunk count, evened out on the 256-token grid, and only
when today's last chunk has 1024 tokens or more (3072 with `STRATA_PREFILL_CPU_SHARE` set). A chunk that size streams
every expert its card does not hold, a shorter one only the experts it routes to, so a prompt just past the chunk size
reads as one full chunk and a nearly free tail, and evening that out would add a second full pass over the experts:
the first version evened every split out, and on two RTX 4090 MistyMoonR's calibration read 1,601 tok/s with it
against 2,129 off. 0 (the default) is today's split;
`STRATA_PREFILL_PIPE=<R>` is the same as the flag where the config cannot carry it. The chunk geometry changes the
rounding, which is why it is opt-in.
`--pipeline-windows 2` lets the first card start the next verify window while the last card still verifies this one. The next
window is a guess: that this window is accepted whole and that its bonus token is the one the draft layer predicts. If the guess
holds, half of the next window is already done. If not, the first card restores its state and builds the window from the real
tokens. The tokens are the serial loop's, up to CPU rounding.
A prompt segment (the prompt between two of its cuts: the system-prompt root, the last turn, a `pin=`) is planned once,
on its whole length and R, and every run of it reads in that chunk. So its chunks, and the loan its buffers are laid
out for, are the same whether it is read in one run, in pieces between the windows of decoding `--batch` slots, or goes
on after a `BYIELD` or from a periodic checkpoint taken inside it (the slot and the checkpoint keep the plan; the read
goes on in it when it continues the same segment, else it plans again and says so in the log). The output bits then
match as far as the experts the cards hold do (`--adapt-swaps 0` for byte-identical repeats, as without the pipe).
Before this, a read beside decoding slots re-planned every piece, and one that went on after a `BYIELD` re-planned what
was left. A chunk the pipe planned takes no `STRATA_PREFILL_CPU_SHARE`: the stages read
such chunks at the same time, and which stage got the CPU pool first would decide which experts the CPU computed. The
DONE line ends with `chunk=` (the first chunk of the request's longest batched segment) and `experts_streamed=` (the
expert blobs its prompt copied to the cards), and the INFO line carries `prefill_cap`, `prefill_stages`,
`stream_all_min` and `prefill_pipe`. One line in the log says what a plan chose:
- **Default:** on for AMD RDNA4 (gfx1200 / gfx1201) with `--serve`, the draft layer, and a split of two or more stages when the
drafter can sit on the first card, otherwise three or more. The log says "is on by default for a layer split of N stages".
`--pipeline-windows 0` or `STRATA_PIPELINE_AUTO=0` turns it off. NVIDIA builds keep it opt-in (`--pipeline-windows 2`).
- **Draft layer placement:** with the pipeline on, the draft layer loads on the first card, whose window has finished when the
chain starts. `STRATA_MTP_STAGE0=0` keeps the old place.
- **Off, with one log line saying why:** `--batch` slots, `--peer-device`, the helper caches (`--expert-cache-device1..3`, which
`--remote-expert-opt` builds on), a split onto one GPU, no draft layer, and on three or more stages `--pipeline-windows 1` and
`--adapt-async 1`. A request with repetition penalties or coupled draft sampling decodes serially.
- **Cost:** a second verify window on each card, 160 MiB less expert cache per card, plus two copies of the first card's
recurrent state (about 3 MiB per GDN layer it runs).
- **Config:** `"args": [ ..., "--pipeline-windows", "2" ]` with `"layer_split": "20"`.
strata prefill: 10058 tokens in 1280-token chunks over 4 stages (--prefill-pipe 706: b/a)
`--remote-expert-opt` does something only with a helper cache; on a plain layer split it is inert.
The `STRATA_PIPELINE_*` tuning and test variables are read only with `STRATA_PIPELINE_DEBUG=1`.
Four GP100 dies (PH402 SKU 200, `--layer-split 12,24,36`, Flash-Next IQ3_XXS, 64K context), prompt tok/s, fresh
prompts, two interleaved rounds (each within 0.2%), with b/a 1024 (the same chunks as 700-890, measured on this rig):
## Two models on two cards
| Prompt tokens | `--prefill auto` | `--prefill 2048` | `--prefill auto` + `--prefill-pipe 1024` (chunk) |
The other way to use two cards is two models, one per card, each answering on its own. In the app, turn on **Run multiple
models** on the Models page. The card list then says which card a loaded model uses ("In use by ..."), starts that card as Off
for the next model, and explains a failed start on a card that is taken. Two models at once on two makes is not supported.
**Measured on:** a PC with two of four Radeon AI PRO R9700 cards, Linux, AMD build, Q2_0 on one card and IQ3_XXS on the other.
Both loaded (33 s and 59 s with the files cached, 114 s the first time) and answered chats at about 60 tokens/s each. Unloading
one left the other answering. The same app code drives NVIDIA cards, but two models on two NVIDIA cards were not run.
## DeepSeek V4.1-Flash
DeepSeek V4.1-Flash runs in the app and the Terminal and spreads its experts and its dense weights over several cards. It does not use
the Qwen layer split described above. On four Radeon AI PRO R9700 cards it reaches about 31 tokens/s. Hardware, sets and speeds:
[MODELS.md](MODELS.md#deepseek-v41-flash).
## Limits
- Every card needs compute capability 7.5 (RTX 20 or newer), unless you opt into the CUDA 12 engine for older cards.
- Works across cards: images (`--vision`), control vectors, KV streaming (`--kv-resident`) and mid-prompt checkpoints
(`--prompt-cache-every`). Each card keeps its own part.
- The older helper-GPU caches (`--expert-cache-device1..3`, [SECOND_GPU.md](SECOND_GPU.md)) take the visible GPUs no stage runs on.
A weak helper can make decoding slower, because its per-layer round trip costs more than the CPU pool needs for those experts.
- `--mmap-experts` needs a canonical pack (`experts.bin`); a native (IQ) pack says so at start.
- The prompt path has its own buffers on every card (1.5 GB each at the default 2048-token chunk; `--prefill 1024` halves that).
An explicit `--expert-cache` on the first card is capped to leave room for them.
- More cards pay off when the model's routed experts do not fit the faster ones. An extra slow card costs its own round per
window and can make a good pair slower.
- For several conversations at once, `--batch N --batch-groups G --trim-stage-weights` pipelines them through the cards
([BATCHING.md](BATCHING.md)).
## History / older measurements
Dated numbers behind the settings above. Each row is one machine; none is a claim about yours.
<a id="measured"></a>
**Layer split, Coder model.** Measured on: RTX 5080 + RTX 3090, Ryzen 9 9950X3D, Coder, 32K context, engine 0.1.20-era
(`bench/results/2026-09-29-layer-split/`).
| Setup | Prompt 16K / 28K tok/s | Decode story / code tok/s |
| --- | --- | --- |
| 5080 alone | 1,726-2,017 / 1,970 | 83-87 / 88-105 |
| 5080 + 3090, best split (K=26) | 2,039 / 2,357 | 84 / 110 |
| 5080 + 3090, auto (K=22) | 2,037 / 2,073 | 80 / 109 |
A third card (RTX 2080 Ti) made that pair slower: 68 / 90 tok/s decode.
**Option measurements**
| Setting | Measured on | Result |
| --- | --- | --- |
| Unequal pair, `auto` rule | 2x RTX A4000 made unequal, IQ3_XXS, 14.5K prompt, 5 interleaved pairs | prompt 945 -> 1,441 tok/s (+52%), decode 50.2 -> 65.9 (+31%) |
| Unequal pair, hand sweep of `"layer_split"` | 4060 Ti 16 GB + 3060 12 GB, IQ3_S | K=12/18/24/28/32: 582 / 693 / 762 / 802 / 803 prompt tok/s; the old rule chose K=3: 452 |
| x16 + x4 link scaling | 2x RTX 3090, x16 + x4 | K=31: 114-119 tok/s decode, K=27: 130-136 |
| `split_skip_if_fits` | R9700 32 GB + RX 9070 XT, Coder | 4K prompts 1,244 -> 1,776 tok/s, decode ~51 -> ~60; 16K prompts ~5% slower |
| `STRATA_SPLIT_OWN=1` | R9700 + RX 9070 XT | 2K prompts 1,450 and 16K 2,227 tok/s (default 1,265 and 1,950) |
| `STRATA_PREFILL_HELP=1` | 2x RTX 3090, UD-Q4_K_XL, no P2P | prompt tok/s without / with: 1.1K 496 / 723, 2K 878 / 1,128, 3K 1,260 / 1,440; 4K and 8K unchanged |
| `STRATA_PREFILL_HELP=1` | 2x R9700 (16 GB each), Q2_0, PCIe 4.0 x16 | 700 tokens 638 -> 672; 1.5K 839 -> 809; 3K 1,105 -> 929; slower from 1.5K, so it stays off |
| `STRATA_SPLIT_MTP_BATCH=1` | 2x RTX 3080 20 GB, Xeon E5-2696 v4, UD-Q4_K_XL, `--layer-split 23` | prompt read +6.7% at 25K, +4.3% at 51K, +3.3% at 104K; decode not shown unchanged (about 3% lower, within restart noise) |
| `STRATA_STAGE_TRIM=1` | 2x MI50 16 GB, Coder | experts in VRAM 8,819 -> 10,626; decode 39.2 -> 41.7 tok/s |
| Prompt chunk on a split, AMD gfx12 | 2x / 3x / 4x R9700 | 16K prompt +36% / +45% / +53%; 32K +18% / +21% / +29%; 64K +4% / +12% / +17% |
| `STRATA_SPLIT_OVERLAP=1` | 2x RX 7900 XTX, Swift 1.5 IQ3_XXS, 128K, median of 8 | code decode 53 -> 70.1 tok/s (one card ~70); prose 45 -> 55.9 (one card ~68) |
| `STRATA_SPLIT_OVERLAP=1` | 2x RTX A4000, IQ3_XXS | +0.3% story, +2.1% code, -0.3% facts, inside the noise |
| Two Tesla P40, layer split | community report, IQ2_XS (#1028), experimental | decode 19.6 tok/s on one card, 34.8 on both |
**Resident RAM mode on a split.** Measured on: RTX 4060 Ti (layers 0-19) + RTX 5080 (20-47), i9-14900KF, 32 GB RAM, Windows 11,
Swift 1.5 IQ3_XXS, 160K context, four greedy prompts at a time, decode tok/s.
| Setup | First four prompts | After three more rounds |
| --- | ---: | ---: |
| 5080 alone, `--resident-experts` | 25 | 29 |
| split, `--mmap-experts` | 32 | 64 |
| split, `--resident-experts` (22 GiB locked in RAM) | 71 | 69 |
**Prompt chunks for the pipeline (`--prefill-pipe`).** Prompt tok/s, fresh prompts, two interleaved rounds per row.
| Prompt tokens | `--prefill auto` | `--prefill 2048` | `--prefill-pipe 1024` (chunk) |
| ---: | ---: | ---: | ---: |
| 2,013 | 180.9 | 181.4 | 228.0 (1024) |
| 3,865 | 207.6 | 276.5 | 312.4 (1024) |
| 10,041 | 261.1 | 433.0 | 432.6 (2048) |
| 28,998 | 475.7 | 554.1 | 559.7 (3328) |
Two RTX 3090 Ti (CUDA 12.9 build for sm_86, two stages, Swift 1.5 IQ3_XXS, 1M context, 13,371 of 24,576 experts in
VRAM), prompt tok/s, fresh prompts, two starts per arm, each reading the four prompts twice (the cold first pass after
a card was freed is left out); b/a 5836 and 6432 (timed on the rig), against 1024:
Measured on: four GP100 dies (PH402 SKU 200), `--layer-split 12,24,36`, Flash-Next IQ3_XXS, 64K context. On 2x RTX 3090 Ti
(Swift 1.5 IQ3_XXS, 1M context) calibration kept it off, and a fixed 1024 chunk read 19-24% slower there and 22-36% slower on
2x RTX 4090 (community report).
| Prompt tokens | `--prefill auto` | + b/a 5836 / 6432 (chunk) | for comparison: b/a 1024 |
| ---: | ---: | ---: | ---: |
| 2,035 | 694-815 | 698-816 (today's) | 560-562 |
| 3,887 | 1,127-1,207 | 1,115-1,227 (today's) | 880-885 |
| 10,063 | 1,437-1,553 | 1,654-1,681 (2 x 5120) | 1,109-1,113 |
| 29,019 | 2,532-2,603 | 2,566-2,620 (today's) | 2,064-2,089 |
**Pipeline windows.** Decode tok/s.
## Limits (for now)
| Measured on | Result |
| --- | --- |
| 2 cards, R9700, code, serial loop -> draft layer on the first card | 103 -> 125 (+21%); 3 cards +19%, 4 cards +22% |
| RTX 4060 Ti + RTX 5080, i9-14900KF, 32 GB, Swift 1.5 IQ3_XXS, 160K, greedy 500 tokens | mean 72.4 -> 84.0 (+16%): Python 90.4 -> 103.1, English prose 63.0 -> 71.1, copy-heavy edit 90.5 -> 117.9 |
| 4x RX 7900 XT, Flash-Next GSQ-RCO IQ3_S, 262K, `--layer-split 12,24,36` | 4K 59.4 -> 67.6; 32K 51.8 -> 65.6 |
- **Works across cards** (bench/results/2026-09-29-layer-split-limits):
- images (`--vision`): each card keeps its own image-position table;
- control vectors and the experimental speed projection: each card holds the vector's tables, switched on and
off per request on all of them;
- KV streaming (`--kv-resident`): each card streams the KV of its own session;
- mid-prompt checkpoints (`--prompt-cache-every`): each card saves its part of a checkpoint when it has read that
chunk;
- the older helper-GPU caches (`--expert-cache-remote`, docs/SECOND_GPU.md): they take the visible GPUs no stage
runs on, and hold only experts no stage's cache holds. On the test rig, a 2080 Ti helper made decoding slower,
as it did without a split: its per-layer round trip costs more than the CPU pool needs for those experts.
- `--mmap-experts` needs a canonical pack (`experts.bin`), with or without a split; a native (IQ) pack says so at
start.
- The prompt path has its own buffers on every card (1.5 GB each at the default 2048-token chunk; `--prefill 1024`
halves that) instead of borrowing cache slots as one card does. An explicit `--expert-cache` on the first card is
capped to leave room for them.
- Under WDDM (Windows, and WSL2) only 8 GiB of the expert arena is pinned (more, mapped into two GPU contexts,
leaves WDDM refusing allocations); the rest streams through the pinned staging ring. A Linux driver has no such
limit, so there the whole arena is pinned (since 0.1.31; the cap cost a 4090 + 3060 split two thirds of its
prompt speed, #253). `STRATA_ARENA_PIN_GIB=N` pins at most N GiB, `0` the whole arena, on any OS.
- Every card needs compute capability 7.5 (RTX 20 or newer). The pre-sm_80 QSA scorer path is fp32 FMAs, so a
Turing card runs the same kernels instead of the tensor-core prompt attention.
## Measured
The Coder on an RTX 5080 + RTX 3090 (Ryzen 9 9950X3D), 32K context; details in
`bench/results/2026-09-29-layer-split/`:
| | Prompt 16K / 28K tok/s | Decode story / code tok/s |
|---|---|---|
| 5080 alone | 1,726-2,017 / 1,970 | 83-87 / 88-105 |
| 5080 + 3090, best split (K=26) | 2,039 / 2,357 | 84 / 110 |
| 5080 + 3090, auto (K=22) | 2,037 / 2,073 | 80 / 109 |
- **Prompts gain the most** (+18-20%): each card reads its own layers of the chunk while the other reads the next.
- **Decode is on par with the faster card alone**, and ahead on code. Once both caches hold nearly every routed
expert, the per-layer GPU time decides.
- **Correctness:** one GPU is byte-identical to 0.1.20, and the hand-off itself is bit-exact.
**Which cards and in what order:**
- Put the fastest card first; auto gives it as many layers as its cache allows.
- Leave out a much slower card when two already hold the model. An RTX 2080 Ti as a third card made the 5080 +
3090 pair slower (68 / 90 tok/s decode): every extra card costs its own round per window.
- More cards pay off when the model's routed experts do not fit the faster ones.
## One conversation with both cards busy (`--pipeline-windows`, default on two or more AMD gfx12 stages)
**Default on AMD RDNA4 (gfx1200 / gfx1201).** With `--serve`, the draft layer and a layer split of **two or more stages**,
`--pipeline-windows 2` is on unless you say otherwise (the log says "is on by default for a layer split of N stages"; two
stages since 0.1.44, three or more since 0.1.43). It is no quality trade: the tokens are the serial loop's up to CPU rounding
(below). `--pipeline-windows 0` or `STRATA_PIPELINE_AUTO=0` turns it off; `--pipeline-windows 2` asks for it on any split.
The same options that switch it off when you ask for it (`--batch`, `--peer-device`, the helper caches, no draft layer,
`--adapt-async 1`) leave it off without a line when it is only the default. NVIDIA builds keep it opt-in.
**Where the draft layer sits (0.1.44).** The draft layer's chain used to run on the last card, beside the last stage's window.
On 2x / 4x R9700 the two streams then slowed each other, in most engine starts and for the whole life of that engine: the
last stage's window took 27 ms instead of 13 ms while a chain ran (14.9 ms in the lucky starts, which also decoded 25% faster;
the 0.1.43 notes' "128.9 does not reproduce" was this). With the pipeline on, the engine now loads the draft layer on the
**first** card, whose window has finished when the chain starts: the chain's source rows are copied across once per chain and the
draft head's rows once at load. Same tokens; `STRATA_MTP_STAGE0=0` keeps the old place. Exact-flags A/B, 5 pairs, decode tok/s
(serial loop -> now): 2 cards code 103 -> 125 (+21%), 4K prompt +8%, prose equal; 3 cards code +19%, 4K +12%, prose +2%; 4 cards
code +22%, prose +4%, 4K +15%.
With a split the cards take turns on a verify window: the first card runs its layers and hands off, then waits while
the last card runs the rest, the head and the draft. `--pipeline-windows 2` lets the first card start the **next**
window while the last card still verifies this one. The next window is a guess: that this window is accepted whole
and that its bonus token is the one the draft layer predicts (the draft layer is run on through the drafts of the
window in flight). When the guess holds, half of the next window is already done; when it does not, the first card
puts its state back (a copy of its recurrent state taken while the window ran) and the next window is built from the
real tokens. A guessed window is only started when the draft layer's estimate says it is likely to be kept. Windows
copied from earlier context (`--suffix-draft`) are guessed past as well, which is where edits that copy text gain
most. `--pipeline-windows 1` overlaps only the short prompt reads that go through the verify windows
(`--short-read`).
Two or more cards, `--serve`. With three or more stages every stage but the last is a front stage: the guessed
window follows the verified one through them one card behind (each front stage keeps its own pair of recurrent-state
copies and puts its state back on a wrong guess), and only the last stage waits for the verdict. `--pipeline-windows 1`
and the asynchronous tier stay two-stage. In the config:
```
"args": [ ..., "--pipeline-windows", "2" ],
"layer_split": "20"
```
- **Cost**: a second verify window on each card, 160 MiB more kept out of each card's expert cache, plus two copies
of the first card's recurrent state (about 3 MiB per GDN layer it runs) on the first card with `2` (on every front
card with three or more stages; only the first card's room is kept out of its cache, the others allocate theirs after).
- **Same text**: the last card only ever runs windows that are verified, and every window row computes what it
would in any other window, so the tokens are the serial loop's. With `STRATA_IQ_MT_MIN=1 --pcie-frac 0
--adapt-every 0` the greedy output is identical bit for bit to the serial loop's with the same expert caches. The pipeline keeps
its VRAM out of the caches, so against a run without the flag a few experts move from a card to the CPU, which
rounds them differently, and a near-tie can flip (a serial run given the same caches through `--vram-reserve-mib`
matches it exactly).
- **Off, with one line in the log saying why**, with `--batch` slots, `--peer-device`, the helper caches
(`--expert-cache-device1..3`, which `--remote-expert-opt` builds on), a split onto one GPU (`--split-device 0`), or no
draft layer; on three or more stages also `--pipeline-windows 1` and `--adapt-async 1`. A request with repetition penalties (`penalty_last_n`) or coupled
draft sampling decodes serially.
- **With the resident RAM mode's asynchronous swaps** (`--adapt-async 1`, [DETAILS.md](DETAILS.md)) a round's steps
advance between the verified windows. Each card's copies are queued by the decode loop itself while that card has
no window in flight, after every window that may still read what they overwrite has finished; the moves into RAM
wait the same way.
- **Measured** (Swift 1.5 IQ3_XXS, 160K context, q4_0 KV, the stock draft layer, RTX 4060 Ti (layers 0-19) +
RTX 5080 (20-47), i9-14900KF, 32 GB of RAM with the resident RAM mode on the split (#848); greedy, 500 tokens, two
interleaved pairs of three rounds, decode tok/s): Python code 90.4 -> 103.1, C code 76.6 -> 81.6, English prose
63.0 -> 71.1, Italian prose 41.6 -> 46.3, a copy-heavy edit (a 5 KB file back with a rename) 90.5 -> 117.9; mean
72.4 -> 84.0 (+16%). It pays when the windows are GPU-bound: with the experts read through the OS file cache
(`--mmap-experts` on that 32 GB PC) the file reads dominate and it measured no faster.
- **Measured on four stages** (Flash-Next GSQ-RCO IQ3_S, 262K context, int8 KV, layer split 12,24,36 on 4x RX 7900 XT,
every expert in VRAM; greedy, reasoning off, 256 tokens, repository-text prompts): decode 59.4 -> 67.6 tok/s at 4K
(median of 3) and 51.8 -> 65.6 at 32K. A guessed window that holds costs ~15 ms against ~43 ms for a fresh one; about
a third of the guesses held. The greedy text was identical to the serial loop's on all four prompts (`STRATA_IQ_MT_MIN=1
--pcie-frac 0 --adapt-every 0`), and with every guess forced wrong (`STRATA_PIPELINE_FORCE_MISS=1`).
- **HIP**: the draft layer's per-row `gr_read` used the last stage's session scratch (`ss.block.gr`), which that stage's
verifier also uses; a chain launched at a verdict and the next window's last stage then ran on the card at once and
the window's head read a clobbered mix (wrong tokens after a guess that held). The draft layer now has its own scratch.
The `STRATA_PIPELINE_*` tuning and test variables (THETA, FORCE_MISS, SWITCH, LOG, TRACE, SPEC_DEPTH and the like) are read only with
`STRATA_PIPELINE_DEBUG=1`. `--pipeline-windows` and `--adapt-async 1` combine: the engine turns the asynchronous tier
off beside `--pipeline-windows 2` only when `STRATA_PIPELINE_ADAPT_ASYNC=0` is set. What switches either one off is
printed once at start ("is off: ..."). `--remote-expert-opt` does something only with a helper cache
(`--expert-cache-device1..3`); on a plain layer split it is inert, and setup no longer writes it there (#1447).
Measured on 2x RTX 3090 (sm_86, 250 W limits; GPU0 PCIe 4.0 x16, GPU1 x4, no NVLink; Ryzen 7 9800X3D, 32 GB RAM),
Qwen3.8-Flash-Next GSQ-RCO IQ3_S, `--resident-experts`, KV int8, `--spec 4 --mtp`, `"layer_split": "29"` for the
`--pipeline-windows` rows (without `--remote-expert-opt`); the others are setup's config (`auto`). Decode is the mean of
runs 2-6 of 1,500-token coding replies at temperature 0.6, prefill one cold 19.9K-token prompt; one run per arm unless
noted (reported by adambenhassen, #1447; not repeated on our boxes):
**Several settings on one rig.** Measured on: 2x RTX 3090 (250 W, x16 + x4, no NVLink), Ryzen 7 9800X3D, 32 GB RAM, Flash-Next
GSQ-RCO IQ3_S, `--resident-experts`, int8 KV, `--spec 4 --mtp`; decode is the mean of 1,500-token coding replies, prefill one
cold 19.9K prompt; one run per arm (community report #1447, not repeated here).
| Arm | Decode tok/s | Prefill tok/s |
|---|---:|---:|
| setup defaults (two runs) | 137.0 / 136.9 | 1798 |
| `--adapt-async 1` (two runs) | 145.2 / 143.7 | 1800 / 1797 |
| `--adapt-async 1`, no `--remote-expert-opt` | 145.0 | 1807 |
| `STRATA_ADAPT_LAG=2` | 142.8 | 1800 |
| `STRATA_EXCHANGE_ROTATE=1` | 139.6 | 1800 |
| `STRATA_PF_FUSED=1` | 137.2 | 1913 |
| `--pipeline-windows 2` | 136.7 | 2097 |
| `STRATA_SPEC_COUPLED=1` | 137.6 | 1804 |
| `STRATA_SPEC_PROB=1` | 137.2 | 1788 |
| `STRATA_SPEC_COUPLED=1` + `STRATA_SPEC_GUMBEL=1` | 136.5 | 1805 |
| async + lag 2 + rotate + pf_fused | 145.0 | 1829 |
| pw2 + lag 2 + rotate + pf_fused | 135.9 | 2296 |
| pw2 + async + lag 2 + rotate + pf_fused | 142.0 | 2302 |
The last row keeps most of the asynchronous tier's decode gain and the pipeline's +28% prefill; no stalls in any arm.
These are opt-in settings on one rig, not defaults.
## Several conversations at once
With a layer split, `--batch N --batch-groups G --trim-stage-weights` decodes several conversations together and
pipelines them through the cards: see [BATCHING.md](BATCHING.md).
| --- | ---: | ---: |
| setup defaults | 137.0 | 1,798 |
| `--adapt-async 1` | 144.5 | 1,799 |
| `--pipeline-windows 2` | 136.7 | 2,097 |
| `STRATA_PF_FUSED=1` | 137.2 | 1,913 |
| pipeline-windows 2 + async + lag 2 + rotate + pf_fused | 142.0 | 2,302 |
docs/NVFP4.md
+198 -154
View File
@@ -1,201 +1,155 @@
# NVFP4 routed experts
This page is for people who run, convert or test Qwen3.8-Flash-Next packs whose routed experts are NVFP4. They need an
NVIDIA card. For the choice between the three versions, see [MODELS.md](MODELS.md#nvfp4).
The engine can serve a Qwen3.8-Flash-Next whose routed experts are NVFP4, as NVIDIA's ModelOpt stores them: 64-value
blocks of E2M1 codes with four UE4M3 sub-block scales, plus one FP32 `weight_scale_2` per expert and projection.
Everything else (attention, GDN, shared experts, routers, the head) stays in the formats the native path already
reads. Measured on Windows 11, RTX 5090 (32 GB), Ryzen 9 9950X3D, 128 GB DDR5-5600.
Everything else (attention, GDN, shared experts, routers, the head) stays in the formats the native path already reads.
**The three NVFP4 versions in setup and the app.** `NVFP4` is "NVFP4 (Strata's version)", `NVFP4-NVIDIA` is "NVFP4
(NVIDIA's version)", and `NVFP4-QAD` is "NVFP4 (NVIDIA, trained for FP4)" ([the short guide](MODELS.md#nvfp4)). All three
read pictures, through the original model's picture reader file that setup downloads with them (see "Images" below and
the FP4-Local section). The other sections of this page apply to all three unless they say otherwise.
## Quick start
## Getting a model
Choose a ready-made pack with the Terminal. Manual conversion below needs a source checkout.
```sh
strata launch --family nvfp4 --model NVFP4
```
| Size in setup and the app | Name | Source |
| --- | --- | --- |
| `NVFP4` | NVFP4 (Strata's version) | GPTQ-quantized pack, `Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata` |
| `NVFP4-NVIDIA` | NVFP4 (NVIDIA's version) | `nvidia/Qwen3.8-Flash-Next-NVFP4` quantizes the routed experts the same way (NVFP4, group 16; other layers BF16, PLE table FP8, MTP experts FP8). Setup offers it as the ready-made pack `Maximilian228/Qwen3.8-Flash-Next-NVIDIA-NVFP4-Strata`. |
| `NVFP4-QAD` | NVFP4 (NVIDIA, trained for FP4) | `nvidia/Qwen3.8-Flash-Next-FP4-Local`, converted on your PC ([below](#nvidias-fp4-local-checkpoint)) |
All three read pictures through the original model's picture reader file that setup downloads with them
([Images](#images)). The other sections of this page apply to all three unless they say otherwise. Setup runs a pack in
the RAM-budget mode (`--resident-budget-gib`, the RAM less 24 GB, as for Unsloth's files) when the RAM is below the 68 GB
of experts plus room.
## Getting a model by hand
```sh
python tools/nvfp4_convert.py --model <ModelOpt NVFP4 checkpoint dir> --outfile <model>.gguf
python tools/iq_pack.py --gguf <model>.gguf --out <pack dir>
```
- `nvfp4_convert.py` runs llama.cpp's own converter (`third_party/llama.cpp`, the commit setup pins) with the type
policy the engine needs. The experts are repacked without loss: `tools/nvfp4_verify.py` compares them with the
policy the engine needs. The experts are repacked without loss. `tools/nvfp4_verify.py` compares them with the
checkpoint bit for bit, and `tools/nvfp4_verify_gguf.py` checks the GGUF.
- The small projections the BF16 kernels read (routers, SSM gates, indexer, PLE and hyper-connection projections)
are written as BF16.
- Everything served natively goes to Q8_0: attention, `ssm_out`, the shared experts, the output head, and the
token embedding.
- The small projections the BF16 kernels read (routers, SSM gates, indexer, PLE and hyper-connection projections) are
written as BF16.
- Everything served natively goes to Q8_0: attention, `ssm_out`, the shared experts, the output head and the token
embedding.
- The 51.2e9-value PLE table is left out.
- `iq_pack.py` writes `experts.bin` for an NVFP4 GGUF by itself. Each blob is `[gate | up | down]` plus a 16-byte
tail `{s_gate, s_up, s_down, 0}`. The GGUF has no room for these scales, so the engine reads NVFP4 experts from
- `iq_pack.py` writes `experts.bin` for an NVFP4 GGUF by itself. Each blob is `[gate | up | down]` plus a 16-byte tail
`{s_gate, s_up, s_down, 0}`. The GGUF has no room for these scales, so the engine reads NVFP4 experts from
`experts.bin` only.
Run it like an IQ pack, with the GGUF as its own dense-weight source and the PLE table from an IQ GGUF's PLE shard:
```
```sh
strata --pack <pack dir> --native <model>.gguf --native-dense-gguf <model>.gguf \
--ple-gguf <an IQ model's shard with per_layer_token_embd> --mtp <mtp/rt> \
--expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 ...
```
The NVFP4 checkpoints change only the routed experts. The PLE table, the MTP layer and the expert profile of the
original model therefore work as they are.
The NVFP4 checkpoints change only the routed experts. The PLE table, the MTP layer and the expert profile of the original
model therefore work as they are. The conversion was tested end to end with
`jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4`. The NVIDIA checkpoint takes the same path but was
not run here as downloaded.
Tested end to end with `jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4`. By its
`hf_quant_config.json`, NVIDIA's `nvidia/Qwen3.8-Flash-Next-NVFP4` quantizes the routed experts the same way
(NVFP4, group 16; the other layers BF16, the PLE table FP8, the MTP experts FP8), so it takes the same path. It has
not been run here as downloaded. Setup offers it as a ready-made pack, the size `NVFP4-NVIDIA`.
## Expert size and the PCIe share
## Measured
An NVFP4 expert blob is 2.76 MB, against 1.51 MB for IQ2_XS. The cache holds half as many experts, and a miss costs twice
the bytes over PCIe. The PCIe share of the missed experts is therefore 0.25 for an NVFP4 pack, not the usual 0.55. At
0.55 the copy kernel was 40% of all GPU kernel time (nsys, 300-token decode). Measurements are in
[History](#history--older-measurements).
RTX 5090, 64K context, int8 K/V, `--vram-reserve-mib 1500`:
| | NVFP4 pack | IQ2_XS, same engine |
| --- | ---: | ---: |
| experts in RAM | 63.3 GiB | 33.0 GiB |
| expert cache slots | ~8,350 | ~17,300 |
| decode, 400-token answer | 114-116 tokens/s | 132-139 tokens/s |
| a 32K prompt | 3,900 tokens/s | 5,600 tokens/s |
An NVFP4 expert blob is 2.76 MB, against 1.51 MB for IQ2_XS: the cache holds half as many, and a miss costs twice
the bytes over PCIe. The PCIe share of the missed experts is therefore 0.25 for an NVFP4 pack, not the usual 0.55.
At 0.55 the copy kernel was 40% of all GPU kernel time (nsys, 300-token decode). A sweep at 262K context gave the
following decode rates:
| PCIe share | short prompt | 32K prompt |
| --- | ---: | ---: |
| 0.55 | 103 tokens/s | 105 |
| 0.40 | 110 | 117 |
| 0.25 | 117 | 124 |
| 0.10 | 112 | 118 |
## Measured in Strata (2026-10-13)
The numbers above are the fork author's, on his RTX 5090 and 128 GB. These were measured here, on the ready-made GPTQ
pack (`--family nvfp4 --model NVFP4`).
Quality, the quant yardstick (`tools/quant_quality.py`, 4,995 teacher-forced positions of code, docs, English, other
languages and an agent transcript, against Unsloth's Q8_0; RTX 3060 box):
| | same top token as Q8_0 | mean KL | KL p99 |
| --- | ---: | ---: | ---: |
| NVFP4 GPTQ, experts on the CPU | 92.09% | 0.070 | 1.12 |
| NVFP4 GPTQ, 1,500 experts on the GPU | 91.79% | 0.069 | 1.07 |
| the same with `STRATA_ROUTE_TAIL_SKIP=7` | 88.59% | 0.118 | 1.76 |
| NVFP4 GPTQ with the GSQ-RCO IQ4_NL n-gram table instead of the FP8 one | 92.15% | 0.069 | 1.07 |
| for comparison, the same tool: Unsloth UD-Q5_K_XL | 94.55% | 0.038 | 0.64 |
| Unsloth UD-Q4_K_XL | 93.09% | 0.056 | - |
| Unsloth UD-IQ4_XS | 88.91% | 0.101 | 1.60 |
| GSQ-RCO IQ3_S | 87.69% | 0.144 | 1.98 |
The route tail skip moves the answers by more than the quantization itself, so it is off by default for NVFP4
experts. `STRATA_ROUTE_TAIL_SKIP=7` turns it on: +7-25% decode on the RTX 3060, +47-55% on the RTX 5070 (64 GB).
Decode experts (`nvfp4_decode_bench`, 64 distinct experts per call, the share of the card's DRAM bandwidth):
| tokens per expert | RTX 5070 | RTX 3060 | Tesla P100 |
| --- | ---: | ---: | ---: |
| 1 | 0.294 ms (89%) | 0.554 ms (89%) | 0.910 ms |
| 2 | 0.298 ms (88%) | 0.596 ms | 1.247 ms |
| 4 | 0.335 ms (79%) | 0.766 ms | 1.954 ms |
Each weight is decoded once for all the tokens of a verify window (before the port's kernel work: 2.26 ms on the 3060
at 4 tokens), bit for bit the same results.
Prompt experts on the RTX 5070 (`mmq_expert_bench`, one launch of 16 experts with 625 rows each, gate/up + down): w4a8
1.52 ms, w4a4x2 (`STRATA_PREFILL_NVFP4=w4a4x2`) 1.13 ms, w4a4x2 with down too (`STRATA_PREFILL_X2_MIN_K=640`)
0.995 ms, UD-IQ4_XS's experts 1.20 ms. First-token KL against the exact FP16 path (24 prompts, 1.5-8K tokens): w4a8
0.0068, w4a4x2 0.0056, w4a4x2 with down 0.0044. On a 64 GB PC the prompt waits for the experts read from the SSD,
so the modes give the same prompt speed there; w4a8 stays the default until a 96 GB+ RTX 50 PC measures the others.
On RTX 20-40 the NVFP4 prompt products are 1.35-1.6x slower than IQ3_S / IQ4_NL ones at 32K: NVFP4's 16-value scales
are applied in FP32 after every int8 step, which only Blackwell's FP4 tensor cores avoid.
Below its RAM need (68 GB of experts plus room) setup runs the pack in the RAM-budget mode (`--resident-budget-gib`,
the RAM less 24 GB), as for Unsloth's files. Mapped experts (`--mmap-experts`) on a 64 GB Windows PC filled the RAM.
Images: the packs keep Qwen's vision tower unquantized, so the original's image encoder reads pictures for them
(`--vision`; strata-vision opens the dense GGUF for the vocabulary). Checked on an RTX 5070 and an RTX 3060 against
IQ3_XXS's answers on the same picture (a number in the picture read, two shapes with their colours and sides). The
encoder on the GPU takes ~1.1 GiB of VRAM: on the 3060 at 8K context the cache held 1,293 experts with images against
1,734 without. Pictures are on by default; setup and the app pick the CPU reader under 24 GB of graphics memory and the
GPU reader from 24 GB (`--vision cpu` / `--vision gpu` change it).
`STRATA_ROUTE_TAIL_SKIP` is off by default for NVFP4 experts, because it moves the answers by more than the quantization
itself. `STRATA_ROUTE_TAIL_SKIP=7` turns it on.
## Where the global scales go
Each scale multiplies its own projection's FP32 output, as llama.cpp applies `.scale`:
`h = silu(s_gate * (G x)) * (s_up * (U x))`, `y = s_down * (D h)`.
The scales are ~1e-4 each. Folding `s_down` into up is algebraically the same, but it leaves the hidden near 1e-5.
The hidden is then rounded to Q8_0 (CPU) or q8_1 (GPU) blocks with an FP16 scale, and `amax / 127` lands deep in
FP16's subnormals: one expert's output was off by 2-12% instead of 1.1% (`nvfp4_avx512_parity --expert`). The FP16
prompt path had the same problem in its dequantized up weights (~1e-8). Any future fold must be checked against
FP16's range.
The scales are about 1e-4 each. Folding `s_down` into up is algebraically the same, but it leaves the hidden near 1e-5.
The hidden is then rounded to Q8_0 (CPU) or q8_1 (GPU) blocks with an FP16 scale, and `amax / 127` lands deep in FP16's
subnormals. One expert's output was off by 2 to 12% instead of 1.1% (`nvfp4_avx512_parity --expert`). The FP16 prompt
path had the same problem in its dequantized up weights (about 1e-8). Any future fold must be checked against FP16's
range.
## Prompt path precision
`STRATA_PREFILL_NVFP4` picks how the batched prompt path multiplies the experts:
| mode | kernel | activations | one product vs FP64 (`mmq_nvfp4_parity`) |
|---|---|---|---|
| Mode | Kernel | Activations | One product against FP64 (`mmq_nvfp4_parity`) |
| --- | --- | --- | --- |
| `w4a8` (default) | llama.cpp MMQ, int8 tensor cores | q8_1, float block scales | 0.53% |
| `w4a4` | llama.cpp MMQ, Blackwell FP4 MMA (sm_120a only) | NVFP4 | 8.6% |
| `w4a4x2` | Blackwell FP4 MMA | NVFP4 | not in this table; first-token KL in [History](#history--older-measurements) |
| `fp16` | dequantize + cuBLAS FP16 GEMM | FP16 | reference |
`src/prefill/mmq_nvfp4_w4a8.cu` compiles mmq.cuh's int8 NVFP4 path with Blackwell's FP4 MMA hidden from both the
device code and the host's tile config, so it is the same on every architecture; decode already multiplies at this
precision. The FP4 MMA exists only in the arch-specific target, so `src/prefill/mmq_nvfp4_w4a4.cu` (the FP4 x FP4
kernel and its FP4 activation quantizer) is the one unit CMake builds with `12x` -> `12xa` (`strata_mmq_w4a4`);
the rest of the engine keeps the architectures as given and runs on sm_121 too. A card without that unit's image
(another architecture, or sm_121 against a 120a build) falls back to `w4a8` with a note.
`src/prefill/mmq_nvfp4_w4a8.cu` compiles mmq.cuh's int8 NVFP4 path with Blackwell's FP4 MMA hidden from both the device
code and the host's tile config, so it is the same on every architecture. Decode already multiplies at this precision.
The FP4 MMA exists only in the arch-specific target, so `src/prefill/mmq_nvfp4_w4a4.cu` (the FP4 x FP4 kernel and its FP4
activation quantizer) is the one unit CMake builds with `12x` to `12xa` (`strata_mmq_w4a4`). The rest of the engine keeps
the architectures as given and runs on sm_121 too. A card without that unit's image (another architecture, or sm_121
against a 120a build) falls back to `w4a8` with a note.
First-token KL divergence against `fp16`, 8 prompts (code and prose, 1K-8K tokens). The noise floor is the same
`fp16` path cut into 4096-token chunks instead of 8192: another summation order, same arithmetic.
> **Measured on:** first-token KL divergence against `fp16`, 8 prompts (code and prose, 1K to 8K tokens). Hardware and
> Strata version not recorded. The noise floor is the same `fp16` path cut into 4096-token chunks instead of 8192.
| mode | KL mean | KL median | KL max | top-1 agreement | prefill, 4-8K prompts |
|---|---|---|---|---|---|
| Mode | KL mean | KL median | KL max | Top-1 agreement | Prefill, 4-8K prompts |
| --- | --- | --- | --- | --- | --- |
| noise floor | 0.00023 | 0.00003 | 0.0016 | 8/8 | - |
| `w4a8` | 0.0018 | 0.0010 | 0.0096 | 8/8 | 2,503 tok/s |
| `w4a4` | 0.0080 | 0.0027 | 0.040 | 7/8 | 2,904 tok/s |
| `fp16` | reference | | | | 1,905 tok/s |
`w4a8` is the default. It is 4.5x closer to the reference than `w4a4`, at 86% of its speed, and it is the precision
every generated token already runs at. `fp16` is the choice when the prompt must be read exactly.
`w4a8` is the default. It is 4.5x closer to the reference than `w4a4`, at 86% of its speed, and it is the precision every
generated token already runs at. `fp16` is the choice when the prompt must be read exactly.
`STRATA_DUMP_MOE_INPUT` + `mmq_nvfp4_parity --real` test one product on a real prompt's rows: layer 20, its 12
busiest experts, max|x|/rms 4 median and 9 at most. The errors:
`STRATA_DUMP_MOE_INPUT` with `mmq_nvfp4_parity --real` tests one product on a real prompt's rows: layer 20, its 12
busiest experts, max|x|/rms 4 median and 9 at most.
| mode | gate/up | down (heavier SwiGLU tails) |
| Mode | Gate/up error | Down error (heavier SwiGLU tails) |
| --- | ---: | ---: |
| `w4a8` | 0.46% | 0.90% |
| `w4a4` | 7.2% | 8.4% |
| `fp16` | 0.017% | 0.021% |
On RTX 20 to 40 cards the NVFP4 prompt products are 1.35 to 1.6x slower than IQ3_S / IQ4_NL ones at 32K, because
NVFP4's 16-value scales are applied in FP32 after every int8 step, which only Blackwell's FP4 tensor cores avoid.
## CPU experts
`src/kernels/cpu/nvfp4_avx512.cpp` computes the CPU pool's NVFP4 rows in 512-bit lanes. Each 64-value block is
decoded once for every token of a verify window: `vpdpbusd` on |E2M1| codes, with the sign moved onto the
activation. The arithmetic is ggml-cpu's `ggml_vec_dot_nvfp4_q8_0`, with the float additions ordered differently
(worst 4.5e-7 relative, `nvfp4_avx512_parity`). `STRATA_NO_NVFP4_512=1` falls back to ggml-cpu.
`src/kernels/cpu/nvfp4_avx512.cpp` computes the CPU pool's NVFP4 rows in 512-bit lanes. Each 64-value block is decoded
once for every token of a verify window: `vpdpbusd` on |E2M1| codes, with the sign moved onto the activation. The
arithmetic is ggml-cpu's `ggml_vec_dot_nvfp4_q8_0`, with the float additions ordered differently (worst 4.5e-7 relative,
`nvfp4_avx512_parity`). `STRATA_NO_NVFP4_512=1` falls back to ggml-cpu.
A CPU without AVX-512 (Zen 1-3, Intel Core 4th-10th and 12th-14th gen, Core Ultra, Xeon E5 v3/v4) runs the same rows in 256-bit lanes
(`src/kernels/cpu/nvfp4_avx2.cpp`): ggml's AVX2 arithmetic step for step, so every row is bit-equal to ggml-cpu's
(`nvfp4_cpu_parity`, 0 of 1280x9 / 2560x9 rows differ on a Ryzen 5 5600X, a Ryzen 7 255 and a Core Ultra 7 265).
`STRATA_NO_NVFP4_256=1` falls back to ggml-cpu. An NVFP4 pack is a native pack (its folder holds `native_experts.txt`),
so the engine starts on such a CPU; the AVX-512 requirement belongs to the canonical Q2_0 pack only (#2347: a `--pack`
folder without `native_experts.txt` is read as that pack, and the engine now says so before it refuses).
A CPU without AVX-512 (Zen 1 to 3, Intel Core 4th to 10th and 12th to 14th gen, Core Ultra, Xeon E5 v3/v4) runs the same
rows in 256-bit lanes (`src/kernels/cpu/nvfp4_avx2.cpp`): ggml's AVX2 arithmetic step for step, so every row is bit-equal
to ggml-cpu's (`nvfp4_cpu_parity`, 0 of 1280x9 and 2560x9 rows differ on a Ryzen 5 5600X, a Ryzen 7 255 and a Core Ultra
7 265). `STRATA_NO_NVFP4_256=1` falls back to ggml-cpu. An NVFP4 pack is a native pack (its folder holds
`native_experts.txt`), so the engine starts on such a CPU. The AVX-512 requirement belongs to the canonical Q2_0 pack
only; a `--pack` folder without `native_experts.txt` is read as that pack, and the engine says so before it refuses.
| tokens | ggml-cpu | AVX-512 | speedup |
|---|---|---|---|
> **Measured on:** one expert's gate+up (1280 rows of 2560, cache-resident), AVX-512 CPU, Strata version not recorded.
> The pool itself is bound by DRAM.
| Tokens | ggml-cpu | AVX-512 | Speedup |
| --- | --- | --- | --- |
| 1 | 0.102 ms | 0.058 ms | 1.76x |
| 4 | 0.410 ms | 0.125 ms | 3.29x |
| 7 | 0.719 ms | 0.196 ms | 3.66x |
These are one expert's gate+up (1280 rows of 2560, cache-resident). The pool itself is bound by DRAM.
> **Measured on:** the pool in DRAM terms (`nvfp4_avx512_parity --bw THREADS TOKENS`: 1536 experts, 4.2 GB), GB/s of
> weights read. Tokens is the tokens an expert serves in one verify window (1 to 2 in decode).
The pool in DRAM terms (`nvfp4_avx512_parity --bw THREADS TOKENS`: 1536 experts, 4.2 GB, GB/s of weights read; tokens = tokens an
expert serves in one verify window, 1-2 in decode):
| CPU (threads) | kernel | 1 thread, 1 tok | all cores, 1 tok | 2 tok | 4 tok |
|---|---|---|---|---|---|
| CPU (threads) | Kernel | 1 thread, 1 tok | All cores, 1 tok | 2 tok | 4 tok |
| --- | --- | --- | --- | --- | --- |
| Ryzen 7 255, Zen 4 (8) | AVX-512 | 15.4 | 58.6 | 57.6 | 47.7 |
| | AVX2 rows | 13.9 | 58.9 | 58.1 | 42.1 |
| | ggml-cpu | 12.6 | 58.7 | 44.8 | 22.4 |
@@ -204,18 +158,25 @@ expert serves in one verify window, 1-2 in decode):
| Core Ultra 7 265, DDR5-5600 (20) | AVX2 rows | 16.0 | 74.2 | 73.0 | 60.6 |
| | ggml-cpu | 10.3 | 74.1 | 62.7 | 37.1 |
Without AVX-512 a core reads ~10% slower (13.9 vs 15.4 GB/s), but a pool that reads DRAM runs at the same rate up to two tokens a
window and ~12% below at four. The CPU's memory decides, not its vector width.
Without AVX-512 a core reads about 10% slower (13.9 against 15.4 GB/s). A pool that reads DRAM runs at the same rate up
to two tokens a window and about 12% below at four. The CPU's memory decides, not its vector width.
## Images
The packs keep Qwen's vision tower unquantized, so the original's image encoder reads pictures for them (`--vision`;
strata-vision opens the dense GGUF for the vocabulary). Pictures are on by default. Setup and the app pick the CPU reader
under 24 GB of graphics memory and the GPU reader from 24 GB (`--vision cpu` and `--vision gpu` change it). The encoder on
the GPU takes about 1.1 GiB of VRAM.
## Other GPUs
Only the optional `w4a4` needs Blackwell. The default `w4a8` is ggml's int8 MMQ (sm_75 and up), and decode uses the
engine's own q8_1 kernels, which decode UE4M3/E2M1 in software. The engine was built for sm_75/86/89 as PTX and run
on the RTX 5090 with the host answering as that generation. On those builds `mmq_nvfp4_parity` and
`nvfp4_expert_gpu_parity` gave results identical to sm_120.
engine's own q8_1 kernels, which decode UE4M3/E2M1 in software. The engine was built for sm_75/86/89 as PTX and run on an
RTX 5090 with the host answering as that generation. On those builds `mmq_nvfp4_parity` and `nvfp4_expert_gpu_parity` gave
results identical to sm_120.
HIP: the decode kernels use no CUDA-only intrinsics. The NVFP4 MMQ instance and the W4A8 unit are CUDA-only and
guarded (`GGML_USE_HIP`). Neither was built for HIP here.
HIP: the decode kernels use no CUDA-only intrinsics. The NVFP4 MMQ instance and the W4A8 unit are CUDA-only and guarded
(`GGML_USE_HIP`). Neither was built for HIP. Setup does not offer the NVFP4 packs on AMD and Intel.
## Tests
@@ -227,26 +188,109 @@ guarded (`GGML_USE_HIP`). Neither was built for HIP here.
## NVIDIA's FP4-Local checkpoint
`nvidia/Qwen3.8-Flash-Next-FP4-Local` (setup size **NVFP4-QAD**) is NVIDIA's own retrained 4-bit model: ModelOpt 0.47
safetensors, experts retrained by quantization-aware distillation. Strata reads it by converting it on the user's PC;
nothing of NVIDIA's weights is redistributed (licenses: NVIDIA Open Model Agreement and Qwen Community License 1.0).
safetensors, experts retrained by quantization-aware distillation. Strata reads it by converting it on the user's PC.
Nothing of NVIDIA's weights is redistributed (licenses: NVIDIA Open Model Agreement and Qwen Community License 1.0).
What the checkpoint holds, and what `tools/nvfp4_local_pack.py` does with it:
| In the checkpoint | Becomes |
| --- | --- |
| routed experts: NVFP4 (E2M1, E4M3 scale per 16, F32 scale per tensor); the scales stored in NVIDIA's "CSF" form (`.nvfp4_csf_fixed` U8 + `.nvfp4_csf_exceptions` U32) | the scales are decoded to the plain E4M3 matrix (byte-exact; the format is the one vLLM's loader documents), then repacked exactly like every other NVFP4 pack: `experts.bin` with the same 36-byte super-blocks and the 16-byte scale tail. Checked: 304 complete experts and the whole `experts.bin` (SHA-256) equal the route through a full GGUF |
| attention, GDN projections, shared experts, indexer: MXFP8 (E4M3 values, E8M0 scale per 32) | dequantized exactly, then Q8_0 (same 32-value blocks; +0.6% RMS on top of MXFP8's own ~3%), or BF16 for the small projections the BF16 kernels read |
| n-gram (PLE) table, 26.8 GiB: NVFP4 rows of 160 values | a one-tensor GGUF of 90-byte rows (80 B codes + 10 E4M3 scales, one F32 for the table), read by the new PLE format `NVFP4` (`strata.ple.format = nvfp4x16`) in `ngram.cpp`; same row size as IQ4_NL |
| MTP head: NVFP4 experts, BF16 rest | a Q2_0 draft layer built with `mtp_pack.py` / `mtp_rt.py` (88.7% of drafts accepted over 225 test requests) |
| vision tower | not converted. Pictures are read by the original model's picture reader (the shared `mmproj` file that setup downloads, as for the other NVFP4 versions); the dense GGUF supplies the vocabulary |
| Routed experts: NVFP4 (E2M1, E4M3 scale per 16, F32 scale per tensor); the scales stored in NVIDIA's "CSF" form (`.nvfp4_csf_fixed` U8 + `.nvfp4_csf_exceptions` U32) | The scales are decoded to the plain E4M3 matrix (byte-exact; the format is the one vLLM's loader documents), then repacked exactly like every other NVFP4 pack: `experts.bin` with the same 36-byte super-blocks and the 16-byte scale tail. Checked: 304 complete experts and the whole `experts.bin` (SHA-256) equal the route through a full GGUF. |
| Attention, GDN projections, shared experts, indexer: MXFP8 (E4M3 values, E8M0 scale per 32) | Dequantized exactly, then Q8_0 (same 32-value blocks; +0.6% RMS on top of MXFP8's own ~3%), or BF16 for the small projections the BF16 kernels read. |
| N-gram (PLE) table, 26.8 GiB: NVFP4 rows of 160 values | A one-tensor GGUF of 90-byte rows (80 B codes + 10 E4M3 scales, one F32 for the table), read by the PLE format `NVFP4` (`strata.ple.format = nvfp4x16`) in `ngram.cpp`; same row size as IQ4_NL. |
| MTP head: NVFP4 experts, BF16 rest | A Q2_0 draft layer built with `mtp_pack.py` / `mtp_rt.py` (88.7% of drafts accepted over 225 test requests). |
| Vision tower | Not converted. Pictures are read by the original model's picture reader (the shared `mmproj` file that setup downloads); the dense GGUF supplies the vocabulary. |
`python tools/nvfp4_local_pack.py prepare --model <checkpoint> --out <dir>` runs every step (setup does this): the n-gram
table, the dense GGUF (no experts), the pack (experts written straight from the checkpoint) and the draft layer. It needs
about 110 GB of free disk beside the 102 GB download, which can be deleted afterwards. On a Threadripper PRO with the files
on NVMe it took about 12 minutes.
about 110 GB of free disk beside the 102 GB download, which can be deleted afterwards. On a Threadripper PRO with the
files on NVMe it took about 12 minutes.
Measured on an RTX PRO 5000 72 GB (Blackwell), 128K context, int8 KV, every expert in VRAM (`tools/strata_bench.py`):
decode 159 tokens/s (median of chat answers, 137-216), 4K prompt 3,981 tokens/s, 16K 4,813, 32K 4,916 (first token 6.6 s),
agent session 151 tokens/s; the engine holds 69 GB of RAM beside 72 GB of VRAM. Quality is in the table below
(`tools/quant_quality.py`, set v1, against Unsloth's Q8_0): top-1 90.5%, mean KL 0.085, perplexity 3.95 (Q8_0: 4.05).
This is a retrained model, so its distance to the original's Q8_0 includes real weight changes, not only rounding.
> **Measured on:** RTX PRO 5000 72 GB (Blackwell), 128K context, int8 KV, every expert in VRAM
> (`tools/strata_bench.py`). Strata version not recorded.
| Measure | Result |
| --- | --- |
| Decode (median of chat answers) | 159 tok/s (137 to 216) |
| Prompt, 4K / 16K / 32K | 3,981 / 4,813 / 4,916 tok/s (first token at 32K: 6.6 s) |
| Agent session | 151 tok/s |
| Memory | 69 GB of RAM beside 72 GB of VRAM |
Quality (`tools/quant_quality.py`, set v1, against Unsloth's Q8_0): top-1 90.5%, mean KL 0.085, perplexity 3.95 (Q8_0:
4.05). This is a retrained model, so its distance to the original's Q8_0 includes real weight changes, not only rounding.
## History / older measurements
### The fork author's measurements
> **Measured on:** Windows 11, RTX 5090 (32 GB), Ryzen 9 9950X3D, 128 GB DDR5-5600, 64K context, int8 K/V,
> `--vram-reserve-mib 1500`, the fork author's engine.
| | NVFP4 pack | IQ2_XS, same engine |
| --- | ---: | ---: |
| Experts in RAM | 63.3 GiB | 33.0 GiB |
| Expert cache slots | about 8,350 | about 17,300 |
| Decode, 400-token answer | 114 to 116 tok/s | 132 to 139 tok/s |
| A 32K prompt | 3,900 tok/s | 5,600 tok/s |
PCIe share sweep at 262K context (decode tok/s):
| PCIe share | Short prompt | 32K prompt |
| --- | ---: | ---: |
| 0.55 | 103 | 105 |
| 0.40 | 110 | 117 |
| 0.25 | 117 | 124 |
| 0.10 | 112 | 118 |
### Measured in Strata (2026-10-13)
> **Measured on:** the ready-made GPTQ pack (`--family nvfp4 --model NVFP4`), Strata 0.1.44 era. Quality on an RTX 3060
> box with `tools/quant_quality.py`: 4,995 teacher-forced positions of code, docs, English, other languages and an agent
> transcript, against Unsloth's Q8_0.
| | Same top token as Q8_0 | Mean KL | KL p99 |
| --- | ---: | ---: | ---: |
| NVFP4 GPTQ, experts on the CPU | 92.09% | 0.070 | 1.12 |
| NVFP4 GPTQ, 1,500 experts on the GPU | 91.79% | 0.069 | 1.07 |
| The same with `STRATA_ROUTE_TAIL_SKIP=7` | 88.59% | 0.118 | 1.76 |
| NVFP4 GPTQ with the GSQ-RCO IQ4_NL n-gram table instead of the FP8 one | 92.15% | 0.069 | 1.07 |
| For comparison, Unsloth UD-Q5_K_XL | 94.55% | 0.038 | 0.64 |
| Unsloth UD-Q4_K_XL | 93.09% | 0.056 | - |
| Unsloth UD-IQ4_XS | 88.91% | 0.101 | 1.60 |
| GSQ-RCO IQ3_S | 87.69% | 0.144 | 1.98 |
`STRATA_ROUTE_TAIL_SKIP=7` gave +7 to 25% decode on the RTX 3060 and +47 to 55% on the RTX 5070 (64 GB).
> **Measured on:** decode experts (`nvfp4_decode_bench`, 64 distinct experts per call), share of the card's DRAM bandwidth
> in brackets.
| Tokens per expert | RTX 5070 | RTX 3060 | Tesla P100 |
| --- | ---: | ---: | ---: |
| 1 | 0.294 ms (89%) | 0.554 ms (89%) | 0.910 ms |
| 2 | 0.298 ms (88%) | 0.596 ms | 1.247 ms |
| 4 | 0.335 ms (79%) | 0.766 ms | 1.954 ms |
Each weight is decoded once for all the tokens of a verify window, bit for bit the same results. Before the kernel work,
the RTX 3060 took 2.26 ms at 4 tokens.
> **Measured on:** prompt experts on an RTX 5070 (`mmq_expert_bench`, one launch of 16 experts with 625 rows each,
> gate/up + down); first-token KL against the exact FP16 path over 24 prompts of 1.5K to 8K tokens.
| Mode | Time | First-token KL |
| --- | ---: | ---: |
| `w4a8` (default) | 1.52 ms | 0.0068 |
| `w4a4x2` (`STRATA_PREFILL_NVFP4=w4a4x2`) | 1.13 ms | 0.0056 |
| `w4a4x2` with down too (`STRATA_PREFILL_X2_MIN_K=640`) | 0.995 ms | 0.0044 |
| UD-IQ4_XS's experts | 1.20 ms | - |
On a 64 GB PC the prompt waits for the experts read from the SSD, so the modes give the same prompt speed there. `w4a8`
stays the default until a 96 GB or larger RTX 50 PC measures the others.
> **Measured on:** RTX 5070 and RTX 3060, images with an NVFP4 pack against IQ3_XXS's answers on the same picture (a
> number in the picture read, two shapes with their colours and sides).
The answers matched. On the 3060 at 8K context the cache held 1,293 experts with images (GPU encoder) against 1,734
without.
Mapped experts (`--mmap-experts`) on a 64 GB Windows PC filled the RAM, which is why setup uses the RAM-budget mode below
the RAM need.
docs/NVIDIA_V100.md
+73 -63
View File
@@ -1,12 +1,22 @@
# NVIDIA Tesla V100 / Titan V (Volta, sm_70): the community build
# NVIDIA Tesla V100 / Titan V (Volta, sm_70)
Strata's ready-made engine and the support matrix start at compute capability 7.5 (RTX 20). A V100 (7.0) runs the same
engine compiled for it, through the experimental build that issue #236 added. It is **community-tested, not part of the
supported cards**: no ready-made engine is published for it and the numbers below come from one machine.
This page is for people running Strata on a V100 or Titan V. Volta uses the experimental CUDA 12 engine, which current releases
provide as an engine component through the app installer or the Terminal ([INSTALL.md](INSTALL.md)). CUDA 13 cannot run Volta.
How setup selects the CUDA 12 engine and the driver requirements are in [OLDER_GPUS.md](OLDER_GPUS.md). Speeds below were
reported on the named machines. They are not a guarantee for another V100 setup.
## Build
> **On this page:** [Quick start](#quick-start) · [What runs on Volta](#what-runs-on-volta) · [Opt-in switches](#opt-in-switches) ·
> [Checking the kernels](#checking-and-benchmarking-the-volta-kernels) · [Limits](#limits) · [History](#history--older-measurements)
CUDA 13 dropped Volta, so the build needs a **CUDA 12.x** toolkit (12.8 was used) and a host GCC that toolkit accepts.
## Quick start
1. Install the app or Terminal for your system ([INSTALL.md](INSTALL.md)). Setup picks the CUDA 12 engine when the model runs on
a Volta card.
2. To choose it by hand from a source checkout: `START-HERE.bat --setup --gpu 0 --cuda 12` (Linux: `./setup.sh`).
### Build from source
CUDA 13 dropped Volta, so the build needs a CUDA 12.x toolkit (12.8 was used) and a host GCC that toolkit accepts.
```sh
cmake -S . -B build -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF \
@@ -14,53 +24,49 @@ cmake -S . -B build -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF \
cmake --build build --target strata -j
```
Setup does the same by itself for a model on a V100 (the experimental CUDA 12 engine, in `engine-cuda12/`: ready-made
on Windows, compiled with a CUDA 12.x toolkit on Linux; [OLDER_GPUS.md](OLDER_GPUS.md)). For a PC with a V100 and a newer card, a build for both architectures (`-DCMAKE_CUDA_ARCHITECTURES="70;86"`) is the
natural choice (a community report in issue #509 ran an RTX 3080 beside a V100 that way; it was not repeated here). On the
measuring PC a Quadro RTX 4000 (sm_75) sits beside the V100s and was hidden with `CUDA_VISIBLE_DEVICES` because the sm_70-only
build has no code for it.
A source checkout can fetch or build its CUDA 12 engine; `--build` requests compilation with the toolkit (see
[OLDER_GPUS.md](OLDER_GPUS.md)). For a PC with a V100 and a newer card, build for both architectures
(`-DCMAKE_CUDA_ARCHITECTURES="70;86"`). A sm_70-only build has no code for a card such as a Quadro RTX 4000 (sm_75); hide it with
`CUDA_VISIBLE_DEVICES`.
## What runs on Volta
The flag only lowers the floor (CMake and the runtime device check) and replaces two intrinsics Volta lacks. It does not
select "old" kernels everywhere: which kernel runs is decided per architecture when the engine is built.
The flag lowers the floor (CMake and the runtime device check) and replaces two intrinsics Volta lacks. Which kernel runs is
decided per architecture when the engine is built.
| Part of the prompt path | On sm_70 |
| --- | --- |
| MoE experts | opt-in (`STRATA_PF_WMMA=1`, only when every visible card is compute capability 7+; MMQ otherwise): `gemm_iq_f16_grouped`: a group of experts per launch on the FP16 tensor cores, the weights dequantized in shared memory (+10% prompt speed against MMQ's `dp4a` kernels, [bench](../bench/results/2026-10-07-v100-prompt-experts/README.md); FP16 activations instead of MMQ's q8_1, so the answers differ) |
| MoE experts | MMQ by default. Opt-in `STRATA_PF_WMMA=1` (only when every visible card is compute capability 7 or newer): `gemm_iq_f16_grouped`, a group of experts per launch on the FP16 tensor cores. FP16 activations instead of MMQ's q8_1, so answers differ. |
| Dense projections (dequantized weights) | FP16 tensor-core GEMMs (cuBLAS / CUTLASS `s884`) |
| BF16 projections (hyper-connection, router, indexer, ...) | converted to FP16 and run on the FP16 tensor cores (#655, #540; `STRATA_BF16_TC=0`: cuBLAS BF16, an FP32 SIMT kernel on Volta) |
| QSA attention for decode and verify windows (and prompts with `STRATA_PROMPT_ATTN_OLD=1`) | #540's kernel (fewer shuffles, bit-exact; `STRATA_ATTN_PRE75=0`: the one other cards run) |
| BF16 projections (hyper-connection, router, indexer) | converted to FP16 and run on the FP16 tensor cores (`STRATA_BF16_TC=0`: cuBLAS BF16, an FP32 SIMT kernel on Volta) |
| QSA attention for decode and verify windows | the #540 kernel, fewer shuffles, bit-exact (`STRATA_ATTN_PRE75=0`: the one other cards run) |
| QSA prompt attention, int8 / FP16 / K8V4 KV | `prompt_attn_v70_kernel` on `mma.m8n8k4` (`STRATA_PROMPT_ATTN_OLD=1`: the decode kernel, one query at a time) |
| QSA prompt attention, Q4_0 KV | the decode kernel |
| QSA block scores | the warp kernel (the tensor-core scorer needs sm_80) |
| Part of a decode / verify window | On sm_70 |
| Part of a decode or verify window | On sm_70 |
| --- | --- |
| Routed experts in VRAM | gfx906's expert mode 8: the codebook grid and the group's activations in shared memory, SwiGLU fused (`STRATA_EXP_MODE=0`: the CUDA layout other cards run) |
| Dense 2-4 column GEMVs and the head | the interleaved `native_mmvq_il` with a Volta rows table (`STRATA_MMVQ_IL=0`: `native_mmvq`) |
| Hyper-connection read, up projection | gfx906's latency-hidden up kernel (`STRATA_GR_FAST=0`: the plain one) |
| Routed experts in VRAM | expert mode 8 (codebook grid and the group's activations in shared memory, SwiGLU fused). Opt-in, see below. |
| Dense 2-4 column GEMVs and the head | the interleaved `native_mmvq_il` with a Volta rows table. Opt-in, see below. |
| Hyper-connection read, up projection | the latency-hidden up kernel. Opt-in, see below. |
All three are bitwise the kernels they replace; on a V100-SXM2 they take a verify window's GPU work from 22.8 to 21.2 ms
([bench/results/2026-10-07-v100-decode-kernels](../bench/results/2026-10-07-v100-decode-kernels/README.md)).
## Opt-in switches
## Measured
None of these is on by default. Each was measured by its contributor on a V100, not by the maintainers.
One V100-PCIE-32GB, i9-7960X, Unsloth `UD-IQ4_XS` (not a setup model), int8 KV, MTP `--spec 2`; prompt read speed of random-word prompts, tokens/s,
with and without the Volta attention kernel (`STRATA_PROMPT_ATTN_OLD=1`):
| Switch | What it does |
| --- | --- |
| `STRATA_SM70_TABLE=1` | Turns on the three decode kernels in the second table above together. Bitwise the default kernels' output. Individual switches `STRATA_EXP_MODE`, `STRATA_MMVQ_IL` and `STRATA_GR_FAST` still apply on top. |
| `STRATA_FUSE_DECODE=1` | The decode GEMV / MoE fusions of #1097. Bitwise the unfused kernels. |
| `STRATA_PF_WMMA=1` | The FP16 tensor-core prompt experts; also takes IQ4_NL down projections. |
| `STRATA_PF_WMMA2=1` | A second prompt-expert kernel for IQ2_S gate/up and IQ4_NL down (#2356). |
| `STRATA_ARGMAX_ROWS_SM70=1` | The multi-block greedy pick on sm_7x (#2357), bitwise the one-block pick. |
| `STRATA_GR_NORM_FAST=1` | The register-split hyper-connection norm (#2358). |
| `STRATA_Q8_GATHER_WIDE=0` | Turns off the 8-wide int8 KV gather (#1095) that the experimental build uses below compute capability 7.5. |
| `STRATA_GDN_REC_WARP=2` | The chain-split DeltaNet recurrence of the prompt (#1098). Changes the FP32 summation order, so the default stays the pipelined kernel. `1` is the bit-exact sync-free variant; `STRATA_GDN_CHUNK=1` the chunked path. |
| Prompt tokens | old attention path | `prompt_attn_v70_kernel` |
| --- | ---: | ---: |
| 7,194 | 1,010 | 1,164 |
| 28,650 | 1,063 | 1,251 |
| 114,338 | 973 | 1,123 |
Decode was 38-51 tok/s in every run and is not affected. The profile, the parity numbers, the needle test and the limits are in
[bench/results/2026-10-03-v100-prompt-attn](../bench/results/2026-10-03-v100-prompt-attn/README.md).
What the kernel does and does not guarantee: against an FP64 reference its error is about 2-3e-6 (output scale 3.6; the FP32 kernel it replaces: 2e-6; 5e-6 before the hi and lo halves got separate accumulator chains); 15 of 15
needles were found; greedy output was identical on the 28,650- and 114,338-token prompts and differed late in the answer on the 7,194-token one (another
summation order).
Volta's tensor cores take FP16 only. A model's BF16 weights are converted, and a weight outside FP16's range would saturate (none
did in the check in the history section).
## Checking and benchmarking the Volta kernels
@@ -69,37 +75,41 @@ cmake -S . -B build -DSTRATA_PARITY_PROMPT_ATTN=ON ... && cmake --build build --
CUDA_VISIBLE_DEVICES=0 ./build/qsa_prompt_attn_parity 32768 2048 5 # synthetic, no model; exit 0 = PASS
```
It compares the tensor-core attention with the FP32 kernel and an FP64 host reference and times both. The Q4_0 cases are skipped below
sm_80.
It compares the tensor-core attention with the FP32 kernel and an FP64 host reference and times both. The Q4_0 cases are skipped
below sm_80.
## Limits
- A layer split over two V100 was not measured with this kernel. The cards work in turn on a single request, so a split adds expert-cache room, not prompt speed.
- sm_60 (Pascal) is covered by the same flag, and community runs exist: two Tesla P40 with a layer split (#1028, bench
[2026-10-05-community-2x-p40](../bench/results/2026-10-05-community-2x-p40/README.md)), a Tesla P100 alone and with an RTX 2070 SUPER as
the expert-helper card (#1069: 30.5 tok/s alone, 35.8-38.4 with the helper), and a V100 + P100 pair (#1079: +6-11% decode from 0.1.39 to
0.1.40 with the P100 as helper cache). One reporter's numbers each; this page's own kernel measurements are V100 only.
- Volta's tensor cores take FP16 only: a model's BF16 weights are converted, and a weight outside FP16's range would saturate (none did in
the check above).
- A layer split over two V100 was not measured with this kernel. The cards work in turn on a single request, so a split adds
expert-cache room, not prompt speed.
- Pascal (sm_60) is covered by the same flag. Community runs exist for it; this page's own kernel measurements are V100 only.
## The decode kernels measured on a V100 (opt-in: `STRATA_SM70_TABLE=1`)
## History / older measurements
The interleaved 2-4 column matvec with a Volta rows table, expert mode 8 and the latency-hidden norm/up (bitwise the default
kernels' output, GPU work per verify window 22.8 -> 21.2 ms on a V100-SXM2 in the contributor's measurement) are used on
sm_70 only when `STRATA_SM70_TABLE=1` is set. They become the default once the contributor has confirmed them on a V100 with
the 0.1.41 code. Without the variable sm_70 runs what 0.1.40.3 ran. The single switches (`STRATA_EXP_MODE`, `STRATA_MMVQ_IL`,
`STRATA_GR_FAST`) still apply on top.
**Prompt attention kernel.** Measured on: one V100-PCIE-32GB, i9-7960X, Unsloth UD-IQ4_XS, int8 KV, MTP `--spec 2`, random-word
prompts, prompt read speed in tok/s with and without the Volta kernel (`STRATA_PROMPT_ATTN_OLD=1`). Decode was 38-51 tok/s in
every run and is not affected. Report: [bench/results/2026-10-03-v100-prompt-attn](../bench/results/2026-10-03-v100-prompt-attn/README.md).
## More opt-in switches (CUDA 12 experimental build)
| Prompt tokens | old attention path | `prompt_attn_v70_kernel` |
| --- | ---: | ---: |
| 7,194 | 1,010 | 1,164 |
| 28,650 | 1,063 | 1,251 |
| 114,338 | 973 | 1,123 |
None of these is on by default. Each was measured by its contributor on a V100, not by the maintainers.
Against an FP64 reference the kernel's error is about 2-3e-6 (the FP32 kernel it replaces: 2e-6). 15 of 15 needles were found.
Greedy output was identical on the 28,650- and 114,338-token prompts and differed late in the answer on the 7,194-token one.
| Switch | What it does | Contributor's number |
| --- | --- | --- |
| `STRATA_FUSE_DECODE=1` | the decode GEMV / MoE operator fusions of #1097 (gate/up + SwiGLU in one kernel, SwiGLU + quantizer in one kernel, all activation images of a vector in one pass, two BF16 projections in one kernel); bitwise the unfused kernels | 4-16 us per token saved, not visible end to end |
| `STRATA_PF_WMMA=1` | the FP16 tensor-core prompt experts (above); also takes IQ4_NL down projections (K=640, #1771) | +10% prompt speed (#1401) |
| `STRATA_PF_WMMA2=1` | a second prompt-expert kernel for IQ2_S gate/up and IQ4_NL down (#2356) | +2.5-4.3% prompt speed |
| `STRATA_ARGMAX_ROWS_SM70=1` | the multi-block greedy pick on sm_7x (#2357), bitwise the one-block pick | +0.7-0.8% decode |
| `STRATA_GR_NORM_FAST=1` | the register-split hyper-connection norm on CUDA (#2358) | +0.5-0.6% decode |
| `STRATA_Q8_GATHER_WIDE=0` | turns off the 8-wide int8 KV gather (#1095) that the experimental build uses below compute capability 7.5 | +11% on the gather kernel |
| `STRATA_GDN_REC_WARP=2` | the chain-split DeltaNet recurrence of the prompt on a V100 (#1098); changes the summation order (FP32 level), so the default stays the pipelined kernel; `1` is the bit-exact sync-free variant, `STRATA_GDN_CHUNK=1` the chunked path | +6-9% on the recurrence |
**Other measurements**
| Measured on | Result |
| --- | --- |
| V100-SXM2, contributor's measurement | decode kernels (`STRATA_SM70_TABLE=1`): GPU work per verify window 22.8 -> 21.2 ms ([report](../bench/results/2026-10-07-v100-decode-kernels/README.md)) |
| V100, `STRATA_PF_WMMA=1` | prompt +10% against MMQ ([report](../bench/results/2026-10-07-v100-prompt-experts/README.md)) |
| V100, `STRATA_PF_WMMA2=1` | prompt +2.5-4.3% |
| V100, `STRATA_ARGMAX_ROWS_SM70=1` / `STRATA_GR_NORM_FAST=1` | decode +0.7-0.8% / +0.5-0.6% |
| V100, `STRATA_Q8_GATHER_WIDE=0` | +11% on the gather kernel |
| V100, `STRATA_GDN_REC_WARP=2` | +6-9% on the recurrence |
| V100, `STRATA_FUSE_DECODE=1` | 4-16 us per token saved, not visible end to end |
| Tesla P100 alone and with an RTX 2070 SUPER as expert helper (#1069) | 30.5 tok/s alone, 35.8-38.4 with the helper |
| Two Tesla P40, layer split (#1028) | [report](../bench/results/2026-10-05-community-2x-p40/README.md) |
| V100 + P100 pair (#1079) | +6-11% decode from engine 0.1.39 to 0.1.40 with the P100 as helper cache |
docs/OLDER_GPUS.md
+224 -224
View File
@@ -1,51 +1,61 @@
# Older GPUs (experimental)
Strata's supported cards are NVIDIA RTX 20 / 30 / 40 / 50 and the AMD cards in [AMD_HIP.md](AMD_HIP.md). The cards
below run through **opt-in, experimental** paths that community members wrote and measured on their own machines. The
maintainers have one of these cards, a Tesla P100, and test on it; they have none of the others: each path is
compile-checked and unit-tested here, and the ready-made engines and their output stay exactly as they were. Numbers are
the reporters' own, on one machine each, unless the text says the maintainers measured them.
This page is for people with an NVIDIA card older than the RTX 20 series, an older AMD card, or a Turing card they want to tune.
For a normal install use the app installer for your hardware or the [Terminal](INSTALL.md#the-terminal). Pascal and Volta use the
CUDA 12 engine package. The `--cuda`, `--gpus`, `START-HERE.bat` and `UPDATE.bat` examples below are
[source-setup](SOURCE_INSTALL.md) options, not Terminal launch flags.
Strata's supported cards are NVIDIA RTX 20 / 30 / 40 / 50 and the AMD cards in [AMD_HIP.md](AMD_HIP.md). The cards below run
through opt-in, experimental paths that community members wrote and measured. The maintainers own one of these cards, a Tesla
P100, and test on it. Every other path is compile-checked and unit-tested, and the ready-made engines and their output are
unchanged. Numbers are the reporters' own, on one machine each, unless the text says the maintainers measured them.
> **On this page:** [Support matrix](#support-matrix) · [CUDA 12 engine](#nvidia-the-cuda-12-engine) ·
> [Tesla P100](#tesla-p100-sm_60) · [GP100 Q8_0 kernel](#pascal-gp100-a-q8_0-decode-kernel-opt-in) ·
> [Tesla P40](#tesla-p40-sm_61-what-was-measured-and-changed) · [Turing switches](#quadro-rtx-8000-and-other-turing-cards) ·
> [AMD](#amd-building-gfx906-and-gfx1012) · [History](#history--older-measurements)
## Support matrix
### NVIDIA
| Cards | Compute capability | How it runs | What is different on it | Reported |
| --- | --- | --- | --- | --- |
| Tesla P100 | 6.0 | the CUDA 12 engine | `__dp4a` emulated (bit-exact); BF16 projections through fp32; opt-in: `STRATA_Q8_SM60=1` runs the Q8_0 dense projections and head on a GP100 kernel ([below](#pascal-gp100-a-q8_0-decode-kernel-opt-in)) | 2x P100, IQ3_S, engine 0.1.39: prompt 526-532 tok/s, decode 28-32 tok/s at 128K prompt tokens, 3 of 3 needle checks at 122K ([report](../bench/results/2026-10-06-community-2x-p100/README.md), #1157); PH402 (4 GP100 dies), IQ3_XXS: decode 28.3 -> 38.9 tok/s with `STRATA_Q8_SM60=1` and a Q8_0 dense shard |
| Tesla P40 / P4, GTX 10 series | 6.1 | the CUDA 12 engine | BF16 projections through fp32 (cuBLAS has no BF16 GEMM there, #395); the dense weights dequantized by a wide kernel, expert mode 8 and its own interleaved-matvec table; opt-in `STRATA_DENSE_F32=1`: the large dense GEMMs as `cublasSgemm` ([below](#tesla-p40-sm_61-what-was-measured-and-changed)) | P40, IQ3_S, engine 0.1.30: prompt 217-374 tok/s, decode 30-33 tok/s (#395); 0.1.43.1 with the sm_61 changes: prompt 255 tok/s at 565 tokens (was 205), decode 37-38 tok/s (was 34-35); with `STRATA_DENSE_F32=1` 464-474 tok/s at 7K tokens (was 360-432) |
| Tesla V100, Titan V | 7.0 | the CUDA 12 engine | BF16 projections on the FP16 tensor cores (#655, #540); the prompt attention on `mma.m8n8k4` (#600); a leaner attention kernel (#540) | V100-PCIE-32GB, UD-IQ4_XS: prompt 1,123-1,251 tok/s (#600); V100 32GB, IQ2_XS: prompt +22% from #540 |
| RTX 20 (Turing) | 7.5 | **supported**, the ready-made engine | opt-in: `STRATA_BF16_TC=1` runs the BF16 projections on the FP16 tensor cores | RTX 2080 Ti, Q2_0: prompt +15-18% (#655) |
| Cards | Compute capability | How it runs | What is different on it |
| --- | --- | --- | --- |
| Tesla P100 | 6.0 | the CUDA 12 engine | `__dp4a` emulated (bit-exact); BF16 projections through fp32; seven scheduling switches on by default ([below](#tesla-p100-sm_60)); opt-in `STRATA_Q8_SM60=1` ([below](#pascal-gp100-a-q8_0-decode-kernel-opt-in)) |
| Tesla P40 / P4, GTX 10 series | 6.1 | the CUDA 12 engine | BF16 projections through fp32 (cuBLAS has no BF16 GEMM there); dense weights dequantized by a wide kernel; expert mode 8 and its own interleaved-matvec table; opt-in `STRATA_DENSE_F32=1` ([below](#tesla-p40-sm_61-what-was-measured-and-changed)) |
| Tesla V100, Titan V | 7.0 | the CUDA 12 engine | BF16 projections on the FP16 tensor cores; prompt attention on `mma.m8n8k4`; a leaner attention kernel. See [NVIDIA_V100.md](NVIDIA_V100.md) |
| RTX 20 (Turing) | 7.5 | **supported**, the ready-made engine | opt-in `STRATA_BF16_TC=1` runs the BF16 projections on the FP16 tensor cores |
A card needs enough VRAM to be useful: 12 GB or more is recommended, as for every card (a 2 GB GT 1030 is compute
capability 6.1 too, and cannot hold any of the model).
A card needs enough VRAM to be useful. 12 GB or more is recommended, as for every card. A 2 GB GT 1030 is compute capability 6.1
too and cannot hold any of the model.
### AMD
| Cards | Architecture | How it runs | Reported |
| --- | --- | --- | --- |
| RX 6800 / 6900 series | gfx1030 | setup (`--backend hip`), unvalidated; #540's attention kernel is the default there, with 8 cells per step and DPP lane exchanges (bit-exact; `STRATA_ATTN_PRE75=0` runs the standard kernel) | [AMD_HIP.md](AMD_HIP.md#rdna2-gfx1030); RX 6900 XT, IQ3_S: prompts +4-6% alone, +7-12% with #835 ([bench/results/2026-10-04-rdna2-pre75-attention](../bench/results/2026-10-04-rdna2-pre75-attention/README.md)) |
| RX 6700 XT | gfx1031 | setup (`--backend hip`), unvalidated (#524) | used daily by its reporter, one card |
| RX 5500 XT (RDNA1) | gfx1012 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1012` (HIP 5.7 or 7) | 8 GB card, IQ3_S, 8K prompt: 15.3 tok/s decode (#442) |
| RX 5700 XT (RDNA1) | gfx1010 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1010`, on ROCm 7.14's `gfx101X-dgpu` wheels; needs `ROCR_VISIBLE_DEVICES=0` in a PC that also has an AMD iGPU | 8 GB card, Coder IQ1_M, 32K context: prompt 115 / 144 / 149 tok/s and decode 18.0 / 20.3 / 23.3 tok/s at 4K / 16K / 30K prompt tokens, 6 of 6 needle checks at 8K and 30K, 61 of 61 ctest on the card, 607 of 12,288 experts in VRAM |
| RX 5700 and the 6 GB RX 5600 (RDNA1) | gfx1010 | as above | the same Navi 10 silicon as the RX 5700 XT, so the same build; a 6 GB card leaves ~50 expert slots, expect the RX 5500 XT's range, not this one's |
| Radeon PRO V520 / Pro 5600M (RDNA1) | gfx1011 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1011` | untested on hardware: it is the same RDNA1 ISA as gfx1010 (Navi 12), and the engine builds for it with 0 errors on the same wheels |
| Instinct MI50 / MI60, Radeon VII | gfx906 (wave64) | built by hand: `-DSTRATA_HIP_GFX906=ON` | 2x MI50, Coder IQ1_M, 128K context: decode 50.1 / 47.8 / 45.7 tok/s at 4K / 32K / 128K prompt tokens, prompt ~520 tok/s (#677) |
| Cards | Architecture | How it runs |
| --- | --- | --- |
| RX 6800 / 6900 series | gfx1030 | setup (`--backend hip`), unvalidated. The #540 attention kernel is the default there (`STRATA_ATTN_PRE75=0` runs the standard kernel). See [AMD_HIP.md](AMD_HIP.md#rdna2-gfx1030) |
| RX 6700 XT | gfx1031 | setup (`--backend hip`), unvalidated; used daily by its reporter on one card |
| RX 5500 XT (RDNA1) | gfx1012 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1012` (HIP 5.7 or 7) |
| RX 5700 XT (RDNA1) | gfx1010 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1010` on ROCm 7.14's `gfx101X-dgpu` wheels; needs `ROCR_VISIBLE_DEVICES=0` on a PC that also has an AMD iGPU |
| RX 5700, 6 GB RX 5600 (RDNA1) | gfx1010 | as above; a 6 GB card leaves about 50 expert slots |
| Radeon PRO V520 / Pro 5600M (RDNA1) | gfx1011 | built by hand: `-DCMAKE_HIP_ARCHITECTURES=gfx1011`; builds with 0 errors, untested on hardware |
| Instinct MI50 / MI60, Radeon VII | gfx906 (wave64) | built by hand: `-DSTRATA_HIP_GFX906=ON` |
## NVIDIA: the CUDA 12 engine
CUDA 13 dropped Pascal and Volta: it cannot compile for them. Setup therefore keeps a **second engine**, built with
CUDA 12.9 and `-DSTRATA_EXPERIMENTAL_SM60=ON`, in its own folder (`engine-cuda12\`, beside `engine\`). One engine runs
per model, so the choice is made per model, by the oldest card that model runs on:
CUDA 13 dropped Pascal and Volta and cannot compile for them. Setup therefore keeps a second engine, built with CUDA 12.9 and
`-DSTRATA_EXPERIMENTAL_SM60=ON`, in its own folder (`engine-cuda12\`, beside `engine\`). One engine runs per model, so the
choice is made per model, by the oldest card that model runs on:
- **Every card the model uses is RTX 20 or newer:** the ready-made CUDA 13 engine, as always.
- **A card is Pascal or Volta:** the CUDA 12 engine. Setup says so (`CUDA 12: sm_70 is older than CUDA 13 supports
...`). On Windows it downloads `strata-windows-x64-cuda12.zip` with NVIDIA's CUDA 12 libraries (from pip, like the
CUDA 13 ones); on Linux, or with `--build`, it compiles the engine with a CUDA 12.x toolkit.
- Every card the model uses is RTX 20 or newer: the ready-made CUDA 13 engine.
- A card is Pascal or Volta: the CUDA 12 engine. Setup says so (`CUDA 12: sm_70 is older than CUDA 13 supports ...`).
Release builds deliver engine components through the signed feed (`strata update`). For a source checkout, `--build` compiles with
a CUDA 12.x toolkit ([source installation](SOURCE_INSTALL.md)).
### Opting in
An older card is used only when you choose it; a PC with a newer card keeps recommending the newer one.
An older card is used only when you choose it. A PC with a newer card keeps recommending the newer one.
| You | Setup |
| --- | --- |
@@ -53,22 +63,21 @@ An older card is used only when you choose it; a PC with a newer card keeps reco
| name the card: `START-HERE.bat --setup --gpu 1`, or `--gpus 0,1` with a newer card | uses it; the model gets the CUDA 12 engine |
| `--cuda 12` (or `STRATA_CUDA=12`) | the CUDA 12 engine for this model, on any card |
| `--cuda 13` | the CUDA 13 engine even with an older card (a warning: it has no code for that card) |
| `STRATA_EXPERIMENTAL_SM60=1` | the older cards are listed as usable (the setting from #295 still works) |
| `STRATA_EXPERIMENTAL_SM60=1` | the older cards are listed as usable |
The choice is kept in the model's config (`"cuda": 12`): its starts and `UPDATE.bat` keep it, and other models keep
their own engine. Setting the model up again chooses again (by its cards; add `--cuda 12` to keep a forced choice). A Pascal / Volta card added to a model at a start (`--gpus`) moves that model to the
CUDA 12 engine.
The choice is kept in the model's config (`"cuda": 12`). Starts and `UPDATE.bat` keep it, and other models keep their own engine.
Setting the model up again chooses again (add `--cuda 12` to keep a forced choice). A Pascal or Volta card added to a model at a
start (`--gpus`) moves that model to the CUDA 12 engine.
`--cuda 12` is also the way to run Strata with an NVIDIA driver older than 580: CUDA 12 needs 528 or newer on Windows
(527.41, NVIDIA's minor-version compatibility) and 525 on Linux; setup's "driver too old" stop says so. Such old
drivers were not tested here.
`--cuda 12` is also the way to run Strata with an NVIDIA driver older than 580. The CUDA 12 engine needs driver 528 or newer on
Windows and 525 on Linux (`CUDA12_MIN_DRIVER` in `setup.py`); setup's "driver too old" stop says so. The Docker `cuda12` image
asks for 575 ([DOCKER.md](DOCKER.md)). Drivers that old were not tested here.
### Mixed cards
A model that shares a V100 with an RTX 30 / 40 card runs both on the CUDA 12 engine (it has code for sm_60 to sm_89,
plus PTX). An RTX 50 card (sm_120) in a CUDA 12 engine is a warning, not a stop: CUDA 12.8 and newer compile for it,
but engines built with 12.8 crashed on long prompts there (#220, #224). Keep the RTX 50 card on its own model (`--gpu
N`), where it runs the CUDA 13 engine.
A model that shares a V100 with an RTX 30 / 40 card runs both on the CUDA 12 engine (code for sm_60 to sm_89, plus PTX). An RTX
50 card (sm_120) in a CUDA 12 engine is a warning, not a stop: engines built with 12.8 crashed on long prompts there (#220,
#224). Keep the RTX 50 card on its own model (`--gpu N`), where it runs the CUDA 13 engine.
### Building it yourself
@@ -79,71 +88,167 @@ cmake -S . -B build-cuda12 -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF -DST
cmake --build build-cuda12 --target strata -j
```
Setup does the same when it compiles: it looks for the newest CUDA 12.x toolkit (`STRATA_NVCC=<path to nvcc>` picks
one, #601; on glibc 2.43 use 12.8, see [TROUBLESHOOTING.md](TROUBLESHOOTING.md)).
Setup does the same when it compiles: it looks for the newest CUDA 12.x toolkit. `STRATA_NVCC=<path to nvcc>` picks one. On
glibc 2.43 use 12.8 ([TROUBLESHOOTING.md](TROUBLESHOOTING.md)). On GCC 12.3 with nvcc (openEuler 24.03, CUDA 12.8) one report
needed `-D_BITS_OPT_RANDOM_H` added to the host flags (#1074, not reproduced here).
On GCC 12.3 with nvcc (openEuler 24.03, CUDA 12.8, 2x V100) the build needed `-D_BITS_OPT_RANDOM_H` added to the host flags (#1074; one
report, not reproduced here). The `size_t` error in `vmm.hpp` that the same report hit is fixed in 0.1.40.2.
### What the flag changes
### What the flag changes, and what it does not
`-DSTRATA_EXPERIMENTAL_SM60=ON` lowers the runtime floor to compute capability 6.0 and compiles the older cards' code paths into
that build only: the Volta prompt attention, the #540 attention kernel (used below sm_75), and the Pascal BF16 path. The
ready-made CUDA 13 engine has none of them, so its kernels and output are unchanged.
`-DSTRATA_EXPERIMENTAL_SM60=ON` lowers the runtime floor to compute capability 6.0 and compiles the older cards' code
paths into that build only: the Volta prompt attention (#600), #540's attention kernel (used below sm_75), and the
Pascal BF16 path. The ready-made CUDA 13 engine has none of them, so its kernels and its output are unchanged. The
FP16 path for BF16 projections is in every build but runs by default only below sm_75; RTX 20 owners can try it with
`STRATA_BF16_TC=1` (`=0` turns it off on a V100). It is not bitwise the same as cuBLAS's BF16 kernel (FP16 tensor-core
sums round differently): #540 measured a mean KL of 8.4e-3 on the next-token distribution of 24 code prompts on a V100
(the same top-1 in 23), the size of other summation-order changes; #655 a worst relative difference of 3.5e-5 per
product on an RTX 2080 Ti.
The FP16 path for BF16 projections is in every build but runs by default only below sm_75. RTX 20 owners can try it with
`STRATA_BF16_TC=1` (`=0` turns it off on a V100). It is not bitwise the same as cuBLAS's BF16 kernel, because FP16 tensor-core
sums round differently.
### Tesla P100 (sm_60): seven decode switches on by default (engine 0.1.45.3)
### Switches by card
On a Tesla P100 the engine now turns on a set of seven switches that change only how the work is scheduled, so the answers
stay bit for bit the same: `STRATA_ADAPT_NOWAIT=1`, `STRATA_ADAPT_LAG=2`, `STRATA_SIDE_MIXER=2`, `STRATA_FUSE_WAIT=2`,
`STRATA_B6_MMVQ_ROWS=1`, `STRATA_FUSE_DECODE=1` and `STRATA_GR_UP75=1`. `STRATA_SM60_PERF=0` turns the whole set off, and a
switch you set yourself, to any value, is kept. The set is used when every visible card has compute capability 6.0.
Measured by the maintainers on a Tesla P100 16 GB, IQ3_XXS, default flags, 8 interleaved pairs of fresh engines: decode 36.56
-> 41.76 tokens/s (+14.2%, faster in 8 of 8 pairs; a run against itself gave 0.975), with the same number of expert-cache
slots; the identity gate (10 prompts and 4K/20K prompt dumps) is identical with the set on. The full list, the RTX 30 set
and the per-switch results: [DETAILS.md](DETAILS.md#rtx-30-series-sm_86-and-tesla-p100-sm_60-bitwise-decode-switches-on-by-default).
| Variable | Card | Default | What it does |
| --- | --- | --- | --- |
| `STRATA_SM60_PERF=0` | P100 | set of 7 on | Turns off the seven scheduling switches below. |
| `STRATA_Q8_SM60=1` | GP100 | off | Q8_0 dense projections and head on a GP100 kernel. |
| `STRATA_IQ_STAGE_TABLES=1` | Pascal | off | IQ3_XXS / IQ3_S tables staged in shared memory (bitwise the same output). |
| `STRATA_DEQUANT_WIDE=0` | sm_61 | on | Back to the old dequantization. |
| `STRATA_SM61_TABLE=0` | sm_61 | on | Back to the CUDA expert layout and `native_mmvq`. |
| `STRATA_DENSE_F32=1` | Pascal | off | Large dense GEMMs as `cublasSgemm`. |
| `STRATA_BF16_TC=0\|1` | sm_7x | on below sm_75 | BF16 projections on the FP16 tensor cores. |
| `STRATA_PROMPT_ATTN_OLD=1` | Volta | off | The decode kernel for prompts instead of the Volta kernel. |
| `STRATA_ATTN_PRE75=0` | sm_70, gfx103x | on | Turns the #540 attention kernel off. `=1` turns it on for another wave32 AMD card. |
### Pascal GP100: a Q8_0 decode kernel (opt-in)
[NVIDIA_V100.md](NVIDIA_V100.md) has the V100 build, its switches and the parity test.
GP100 (P100, PH402) is the one Pascal chip without `__dp4a`, and the decode kernels are written for the cards that have
it: on a GP100 die the dense projections of a verify window read their weights at 40-80 GB/s of its ~730, decoding K-
and i-quant blocks (Q6_K, IQ4_XS, Q4_K, IQ3_S, ...) one int at a time. Two changes, both measured on a PH402 only:
## Tesla P100 (sm_60)
- **`STRATA_Q8_SM60=1`** packs every Q8_0 dense matrix and the head into the planes of `STRATA_Q8_PACKED` (any shape)
and runs them on `src/kernels/cuda/q8_sm60.cuh`: a warp (half or quarter warp for narrow rows) per row, 16-byte
weight loads, the window's q8_1 activations copied once per workgroup into shared memory as an int8 plane, the dot
on `STRATA_DP4A` (vmad), a shuffle sum - no barrier per row. Shapes it declines (a 6144-wide row with 7-8 columns)
keep the usual kernel. Not bitwise the exact kernels (the per-32 products are added in another order). The packed
copies cost their size in VRAM beside the GGUF layout the prompt path reads (Flash-Next: 2.9 GiB of dense
projections over the stages and 0.6 GiB for the head).
- The GSQ-RCO packs have no Q8_0 dense matrices, so the kernel needs a copy of the model whose dense projections and head are
Q8_0: `python tools/q8_dense_gguf.py <model>-00001-of-00002.gguf <new folder>` writes one: every shard that holds
such matrices is converted (Swift 1.5 keeps them in both of its shards) and the rest are hard-linked beside them
(+3.5 GiB; the experts keep their offsets, so the native pack is reused). Point `--native` and
`--ple-gguf` at the new folder. Requantizing costs at most 0.4% of a tensor's |w|max (Q8_0 is finer than the
sources).
- **Opt-in `STRATA_IQ_STAGE_TABLES=1`**: IQ3_XXS's and IQ3_S's codebooks and sign-mask tables are staged in shared memory by
the grouped expert kernels (IQ3_XXS's ported from shinbunbun/llama-cpp-p100-patches 29 and 30, IQ3_S's the same for its
sign bytes; bitwise the same output). IQ3_S was the one gate/up format whose codebook those kernels still read from
global memory: the contributor measured a verify window's layer of IQ3_S experts (20 experts, 24 entries) 24% faster on a
GP100 die (517 -> 393 us with a Q2_0 down projection, 492 -> 369 us with IQ4_NL), and Swift 1.5 IQ3_XXS (13 of its 48
layers are IQ3_S) 3.2% faster at 1M context (39.3 -> 40.5 tok/s); IQ3_XXS's tables alone were within noise (+0-4%). The
maintainers' interleaved A/Bs on a Tesla P100 (PCIe, IQ3_XXS pack and IQ3_S pack, 5-6 pairs each) showed no end-to-end
change (decode 0.97-1.03x, inside the noise of that box), so it is off by default (the contributor's version had it on
for compute capability 6.0). On an RTX 3090 Ti, forced on, the same IQ3_S layers took 3-13% longer (those kernels already
read at 490-665 GB/s there).
- Neither change is measured on compute capability 6.1 (P40, P4, GTX 10 series), which has `__dp4a`. Both build for
sm_61, and on a card with `__dp4a` the Q8_0 kernel's dot is the hardware instruction (`tools/q8_sm60_check.cu`
passes on an RTX 3090 Ti); numbers from a 6.1 card with `STRATA_Q8_SM60=1` or `STRATA_IQ_STAGE_TABLES=1` would
show whether either helps there.
On a Tesla P100 the engine turns on seven switches that change only how the work is scheduled, so answers stay bit for bit the
same: `STRATA_ADAPT_NOWAIT=1`, `STRATA_ADAPT_LAG=2`, `STRATA_SIDE_MIXER=2`, `STRATA_FUSE_WAIT=2`, `STRATA_B6_MMVQ_ROWS=1`,
`STRATA_FUSE_DECODE=1` and `STRATA_GR_UP75=1`. `STRATA_SM60_PERF=0` turns the whole set off, and a switch you set yourself, to
any value, is kept. The set is used when every visible card has compute capability 6.0. The full list and per-switch results are
in [DETAILS.md](DETAILS.md#rtx-30-series-sm_86-and-tesla-p100-sm_60-bitwise-decode-switches-on-by-default).
Measured on a PH402 SKU 200 (two boards, four GP100 dies of 48 SMs and 32 GB HBM2), application clocks locked at
1050 MHz, Windows 11, driver 581.80 (TCC), CUDA 12.9, engine 0.1.40.2; Flash-Next IQ3_XXS, 32K, `--layer-split 12,24,36
--prefill 2048`; greedy, 256 tokens, medians of 2-3 interleaved starts, decode tok/s:
## Pascal GP100: a Q8_0 decode kernel (opt-in)
GP100 (P100, PH402) is the one Pascal chip without `__dp4a`. Its dense projections read their weights at 40-80 GB/s of about
730, decoding K- and i-quant blocks one int at a time. Two opt-in changes, both measured on a PH402 only:
- **`STRATA_Q8_SM60=1`** packs every Q8_0 dense matrix and the head into the planes of `STRATA_Q8_PACKED` and runs them on
`src/kernels/cuda/q8_sm60.cuh`: a warp per row, 16-byte weight loads, activations copied once per workgroup into shared memory,
the dot on `STRATA_DP4A`, no barrier per row. Shapes it declines keep the usual kernel. It is not bitwise the exact kernels.
The packed copies cost their size in VRAM (Flash-Next: 2.9 GiB of dense projections over the stages and 0.6 GiB for the head).
- The GSQ-RCO packs have no Q8_0 dense matrices, so the kernel needs a copy whose dense projections and head are Q8_0:
`python tools/q8_dense_gguf.py <model>-00001-of-00002.gguf <new folder>` writes one (+3.5 GiB; experts keep their offsets,
so the native pack is reused). Point `--native` and `--ple-gguf` at the new folder.
- **`STRATA_IQ_STAGE_TABLES=1`** stages IQ3_XXS and IQ3_S codebooks in shared memory in the grouped expert kernels. Bitwise the
same output. Interleaved A/Bs on a Tesla P100 showed no end-to-end change, so it is off by default. On an RTX 3090 Ti, forced on,
the same layers took 3-13% longer.
Neither change is measured on compute capability 6.1, which has `__dp4a`. Two practical notes for GP100 cards: the default
application clock is 759 MHz and a layer split leaves each die idle most of a window, so lock the clocks
(`nvidia-smi -i <dies> -ac 715,1050` as administrator). And a second engine pinned to the same logical CPUs halves this one's
decode; keep two engines on different logical CPUs.
`tools/q8_sm60_check.cu` checks the kernel against a CPU reference on these shapes (1-8 columns).
## Tesla P40 (sm_61): what was measured and changed
Three things run differently on compute capability 6.1 by default and a fourth is opt-in. Each gives the same bits as what it
replaces.
| Change | What it replaces | Switch |
| --- | --- | --- |
| Wide dequantization (a thread per 8 values, 16-bit loads, 16-byte stores) for Q4_K / Q5_K / Q6_K / IQ4_XS | a thread per 32 values with byte loads | on; `STRATA_DEQUANT_WIDE=0` |
| Expert mode 8 for the experts in VRAM (decode and verify windows) | the CUDA layout | on; `STRATA_SM61_TABLE=0` (or `STRATA_EXP_MODE=0`) |
| The interleaved small-batch matvec with a P40 table (`kIlRows_61`) | `native_mmvq` | on; `STRATA_SM61_TABLE=0` (or `STRATA_MMVQ_IL=0`) |
| The large dense GEMMs as `cublasSgemm` on FP32 copies of the FP16 operands | cuBLAS's FP16-operand kernels (4.3 TFLOPS instead of 7.6 for most chunk sizes) | **opt-in**, `STRATA_DENSE_F32=1` |
The GEMM route is opt-in because its equality is a property of the cuBLAS build, not of the code. It was measured with cuBLAS
12.4.5.8 only. Run `gemm_f32_parity` (a build with `-DSTRATA_BUILD_TESTS=ON`; exit 0 = every size bitwise) with your toolkit's
library before you set it.
`STRATA_GR_FAST=1` is slower on the P40 below 5 tokens, and `STRATA_PREFILL_CPU_SHARE=auto` and `STRATA_ADAPT_OVERLAP=1` gave
nothing there.
Two things to know when measuring on such a card:
- **Temperature.** A P40 has no fan of its own. At 89 C the driver reports `SW Thermal Slowdown: Active` and the clock falls from
1,531 to about 1,300 MHz (`nvidia-smi -q -d PERFORMANCE`). Start every arm of an A/B at the same temperature, or the later arm loses.
- **Greedy answers are not identical run to run on a CPU without AVX-512**, even with `--prompt-cache 0 --adapt-swaps 0
--pcie-frac 0`. Add `STRATA_IQ_MT_MIN=1` (and `STRATA_PREFILL_CPU_SHARE=0`, `STRATA_ROUTE_TAIL_SKIP=0`) to compare two builds'
answers byte for byte.
Profiling switches: `STRATA_PREFILL_TIMING=1`, `STRATA_DECODE_TIMING=1`, `STRATA_VERIFY_PROFILE=1`.
## Quadro RTX 8000 and other Turing cards
Turing (sm_75) has FP16 tensor cores but no native BF16 or TF32 tensor cores. Two existing, opt-in switches are worth comparing
on your own workload:
- `STRATA_BF16_TC=1` converts BF16 projection operands to FP16 and uses FP16 tensor cores with FP32 accumulation. Default on sm_75
is off; `=0` restores the cuBLAS BF16 path. Values outside FP16 range clamp to +-65504, and differently ordered sums can round
differently.
- `STRATA_SELECT_SIMT=1` uses the tiled FP32 QSA block scorer on the pre-sm_80 CUDA path instead of the warp scorer. `=0` keeps
the warp scorer. It does not enable TF32 and its summation order differs.
Set the variables before starting the engine, or in the model config's `env` object, then restart. Merge these fields into the
existing config; this is not a complete run config:
```json
{
"env": {
"STRATA_BF16_TC": "1",
"STRATA_SELECT_SIMT": "1"
}
}
```
`STRATA_BF16_TC=1` improved prompt throughput by 20-34% on one Quadro RTX 8000 and changed replies in 3 of 9 paired requests, so
use it only if different rounding is acceptable. It did not improve decode. Numbers are in the history section. These switches are
existing code ([gemm.cu](../src/prefill/gemm.cu), [qsa_select.cu](../src/kernels/cuda/qsa_select.cu)), not new kernels.
## AMD: building gfx906 and gfx1012
Setup does not build these. Build by hand and run `serve/server.py` with a config, as on any other card.
- **gfx906** (MI50 / MI60 / Radeon VII, wave64): a separate opt-in build, `-DSTRATA_HIP_GFX906=ON
-DCMAKE_HIP_ARCHITECTURES=gfx906` (not `STRATA_ENABLE_HIP`). Current ROCm no longer ships gfx906 libraries; the reporter used a
community ROCm 7.14 image. Recipe, kernels and measurements:
[AMD_HIP.md](AMD_HIP.md#gfx906-instinct-mi50--mi60-radeon-vii-wave64-built-from-source).
- **gfx1012** (RX 5500 XT): the wave32 backend, `-DSTRATA_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1012`. HIP 5.7 (Ubuntu's
packages) works: older hipBLAS is used through rocBLAS, legacy HIP names and the missing `__syncwarp` are version-gated, and
RDNA1's missing signed dot4 uses llama.cpp's SDWA sequence (`-DSTRATA_GFX1012_PORTABLE_DOT=ON`: the portable one).
## Reports welcome
These paths stay experimental until more people run them. A report with the card, the driver or ROCm version, the model and the
engine log (`strata-<model>.log`) in an issue helps. [COMMUNITY_BENCHMARKS.md](COMMUNITY_BENCHMARKS.md) has the format.
## History / older measurements
Dated numbers. Each row is one machine and one engine version, as labelled. They are not a promise for your card.
**Community reports by card**
| Card | Measured on | Result |
| --- | --- | --- |
| 2x Tesla P100 | IQ3_S, engine 0.1.39 ([report](../bench/results/2026-10-06-community-2x-p100/README.md), #1157) | prompt 526-532 tok/s; decode 28-32 tok/s at 128K prompt tokens; 3 of 3 needle checks at 122K |
| PH402 (4 GP100 dies) | IQ3_XXS, engine 0.1.40.2 | decode 28.3 -> 38.9 tok/s with `STRATA_Q8_SM60=1` and a Q8_0 dense shard |
| Tesla P40 | IQ3_S, engine 0.1.30 (#395) | prompt 217-374 tok/s; decode 30-33 tok/s |
| Tesla P40 | IQ3_S, engine 0.1.43.1 with the sm_61 changes | prompt 255 tok/s at 565 tokens (was 205); decode 37-38 tok/s (was 34-35); with `STRATA_DENSE_F32=1` 464-474 tok/s at 7K tokens (was 360-432) |
| V100-PCIE-32GB | UD-IQ4_XS (#600) | prompt 1,123-1,251 tok/s |
| V100 32GB | IQ2_XS (#540) | prompt +22% |
| RTX 2080 Ti | Q2_0 (#655) | prompt +15-18% with `STRATA_BF16_TC=1` |
| RX 6900 XT | IQ3_S ([report](../bench/results/2026-10-04-rdna2-pre75-attention/README.md)) | prompts +4-6% alone, +7-12% with #835 |
| RX 5500 XT, 8 GB | IQ3_S, 8K prompt (#442) | decode 15.3 tok/s |
| RX 5700 XT, 8 GB | Coder IQ1_M, 32K context | prompt 115 / 144 / 149 tok/s and decode 18.0 / 20.3 / 23.3 tok/s at 4K / 16K / 30K; 6 of 6 needle checks; 607 of 12,288 experts in VRAM |
| 2x MI50 | Coder IQ1_M, 128K context (#677) | decode 50.1 / 47.8 / 45.7 tok/s at 4K / 32K / 128K; prompt about 520 tok/s |
**Tesla P100 default switch set.** Measured on: Tesla P100 16 GB, IQ3_XXS, default flags, engine 0.1.45.3, 8 interleaved pairs of
fresh engines: decode 36.56 -> 41.76 tok/s (+14.2%, faster in 8 of 8 pairs), same number of expert-cache slots. The identity gate
(10 prompts and 4K / 20K prompt dumps) is identical with the set on.
**GP100 Q8_0 kernel.** Measured on: PH402 SKU 200 (two boards, four GP100 dies, 48 SMs and 32 GB HBM2 each), application clocks
locked at 1050 MHz, Windows 11, driver 581.80 (TCC), CUDA 12.9, engine 0.1.40.2, Flash-Next IQ3_XXS, 32K, `--layer-split
12,24,36 --prefill 2048`, greedy, 256 tokens, medians of 2-3 interleaved starts, decode tok/s.
| | ~10.7K-token document | the same, cached | short reasoning | code | mean |
| --- | ---: | ---: | ---: | ---: | ---: |
@@ -151,140 +256,35 @@ Measured on a PH402 SKU 200 (two boards, four GP100 dies of 48 SMs and 32 GB HBM
| Q8_0 dense shard, `STRATA_Q8_PACKED=1` | 27.5 | 31.0 | 34.4 | 31.8 | 31.1 |
| Q8_0 dense shard, `STRATA_Q8_SM60=1` | 37.5 | 38.4 | 41.2 | 38.5 | 38.9 |
GPU time per verify window (`STRATA_VERIFY_PROFILE`): dense projections 19.2 -> 11.2 ms, head 4.25 -> 2.1 ms; the
draft policy then picks longer windows (2.10 -> 2.31 tokens per window). Prompt reads are unchanged (437-438 tok/s at
10.7K). `tools/q8_sm60_check.cu` checks the kernel against a CPU reference on these shapes (1-8 columns) and times it.
GPU time per verify window: dense projections 19.2 -> 11.2 ms, head 4.25 -> 2.1 ms. Prompt reads are unchanged (437-438 tok/s at
10.7K). Locking clocks at 1050 MHz took one case from 25.0 to 29.1 tok/s at 10.7K context.
Two things that mattered as much on this card: its default application clock is 759 MHz, and a layer split leaves
each die idle most of a window, so the dies decode below 1050 MHz unless the clocks are locked
(`nvidia-smi -i <dies> -ac 715,1050` as administrator; 25.0 -> 29.1 tok/s at 10.7K context). And a second engine on
the same PC that pins its threads to the same logical CPUs halves this one's decode while it runs (the host thread
is time-sliced); keeping the two on different logical CPUs (SMT siblings) avoided it.
**Tesla P40 sm_61 kernels.** Measured on: one Tesla P40 24 GB in a 20-vCPU VM (Xeon E5-2695 v4, 98 GB RAM), CUDA 12.4 (cuBLAS
12.4.5.8), IQ3_S with MTP (`--spec 4`), int8 KV, `--max-context 262144`, engine 0.1.43.1 ([report](../bench/results/2026-10-12-p40-sm61-kernels/README.md)).
A/B switches: `STRATA_DEQUANT_WIDE=0`, `STRATA_SM61_TABLE=0` (sm_61: the kernels 0.1.43.1 ran), `STRATA_DENSE_F32=1` (Pascal, opt-in),
`STRATA_Q8_SM60=0|1` (Pascal GP100, opt-in), `STRATA_BF16_TC=0|1`, `STRATA_PROMPT_ATTN_OLD=1` (the decode kernel for prompts), `STRATA_ATTN_PRE75=0`
(#540's kernel off; on gfx103x with HIP that kernel is the default, see the AMD table above, and `=1` turns it on for
another wave32 AMD card). [NVIDIA_V100.md](NVIDIA_V100.md) has the V100 build, its measurements and the parity test.
| Change | Result on the P40 |
| --- | --- |
| Wide dequantization | Q6_K 10240 x 2560: 7.2 -> 0.33 ms (10 -> 227 GB/s); Q4_K 5.2x, Q5_K 3.8x, IQ4_XS 3.6x |
| Expert mode 8 | an 18-expert call at 2 tokens: IQ3_S 390 -> 255 us, IQ3_XXS 345 -> 252, IQ2_S 290 -> 238, IQ4_XS 328 -> 261 |
| Interleaved matvec table | IQ4_XS 2560 x 10240 at 2 columns 112 -> 65 us; Q6_K 12288 x 2560 at 4 columns 189 -> 127 us |
| `STRATA_DENSE_F32=1` | 8.4-9.8 TFLOPS at every size from 512 tokens; qkv at 7,041 tokens 85 -> 45 ms |
### Tesla P40 (sm_61): what was measured and changed
Through the server, medians of 3 requests, two runs per build alternated, card at 65 C. By default: a 565-token prompt 203 / 205
-> 256 / 255 tok/s, decode 35.0 / 34.2 -> 38.2 / 37.2 tok/s. With `STRATA_DENSE_F32=1`: a 7,047-token prompt 430 / 364 -> 474 / 473
tok/s. Of a 7K-token prompt, 36% is the dense projections' GEMMs, 28% the experts, 13% the hyper-connection read and 12%
attention.
One Tesla P40 24 GB in a 20-vCPU VM (Xeon E5-2695 v4, 98 GB RAM), CUDA 12.4 (cuBLAS 12.4.5.8), IQ3_S with MTP
(`--spec 4`), int8 KV, `--max-context 262144`, engine 0.1.43.1; the numbers and every run are in
[bench/results/2026-10-12-p40-sm61-kernels](../bench/results/2026-10-12-p40-sm61-kernels/README.md).
`STRATA_PREFILL_TIMING=1`, `STRATA_DECODE_TIMING=1` and `STRATA_VERIFY_PROFILE=1` show a card that is the bottleneck
everywhere: the prompt's GPU timeline equals its wall time (waiting for expert copies 0.2%, host grouping 0.2% of a
7K-token prompt) and a decode window is 49 ms of GPU stages with 1-3 ms of CPU experts. Of a 7K-token prompt, 36% is
the dense projections' GEMMs, 28% the experts (ggml's MMQ, which has its own Pascal dp4a configuration), 13% the
hyper-connection read and 12% the attention.
**Quadro RTX 8000 (sm_75).** Measured on: one 48 GiB Quadro RTX 8000, 260 W limit, Xeon W-2295, CUDA 12.4, GCC 13, 2026-10-07,
source build based on v0.1.40.3 with other local changes; IQ3_S with MTP (`--spec 4`), `--max-context 262144`, INT8 KV with
`--kv-resident 32768`, `--pcie-frac 0.20`, `--expert-cache 19000`, `--prefill auto`, prompt cache and adaptive swaps off. Three
seeded synthetic prompts per size, 128 output tokens, median engine-reported rates.
Three things run differently on compute capability 6.1 by default and a fourth is opt-in. Each gives the same bits as
what it replaces:
| Change | What it replaces | Measured on the P40 | Switch |
| --- | --- | --- | --- |
| Wide dequantization (a thread per 8 values, 16-bit loads, 16-byte stores) for Q4_K / Q5_K / Q6_K / IQ4_XS | a thread per 32 values with byte loads: 0.5 s of every prompt chunk, whatever its length | Q6_K 10240 x 2560: 7.2 -> 0.33 ms (10 -> 227 GB/s); Q4_K 5.2x, Q5_K 3.8x, IQ4_XS 3.6x; `dequant_wide_parity` 360 cases bitwise | on; `STRATA_DEQUANT_WIDE=0` |
| Expert mode 8 for the experts in VRAM (decode and verify windows) | the CUDA layout | an 18-expert call at 2 tokens: IQ3_S 390 -> 255 us, IQ3_XXS 345 -> 252, IQ2_S 290 -> 238, IQ4_XS 328 -> 261; `native_expert_bench` bitwise | on; `STRATA_SM61_TABLE=0` (or `STRATA_EXP_MODE=0`) |
| The interleaved small-batch matvec with a table measured on the P40 (`kIlRows_61`) | `native_mmvq` | IQ4_XS 2560 x 10240 at 2 columns 112 -> 65 us, Q6_K 12288 x 2560 at 4 columns 189 -> 127 us, the Q5_K head at 3 columns 2,545 -> 1,766 us; `mmvq_il_parity` bitwise | on; `STRATA_SM61_TABLE=0` (or `STRATA_MMVQ_IL=0`) |
| The large dense GEMMs (qkv, query, gate, output projections) as `cublasSgemm` on FP32 copies of the FP16 operands | cuBLAS's FP16-operand kernels, which run at 4.3 TFLOPS instead of 7.6 for most chunk sizes (a 7K-token prompt reads in 16.3 or 19.5 s by its exact token count) | 8.4-9.8 TFLOPS at every size from 512 tokens; qkv at 7,041 tokens 85 -> 45 ms; `gemm_f32_parity` 5,201 chunk sizes bitwise | **opt-in**, `STRATA_DENSE_F32=1` |
The GEMM route is opt-in because its equality is a property of the cuBLAS build, not of the code: it was measured with
cuBLAS 12.4.5.8 only. Run `gemm_f32_parity` (a build with `-DSTRATA_BUILD_TESTS=ON`; exit 0 = every size bitwise) with
your toolkit's library before you set it.
Through the server, medians of 3 requests per run, two runs per build alternated, each run started with the card at
65 C (see below). By default: a 565-token prompt 203 / 205 -> 256 / 255 tok/s, decode after it 35.0 / 34.2 -> 38.2 /
37.2 tok/s, a 7,047-token prompt 1-3% faster at the same token count. With `STRATA_DENSE_F32=1` as well: the
7,047-token prompt 430 / 364 -> 474 / 473 tok/s (the old build reads it at ~361 or ~431 tok/s depending on whether the
chunk has more than 7,040 tokens; with the route every request read at 464-474), decode after it 32.1 / 33.4 -> 35.9 /
34.0. With `STRATA_DEQUANT_WIDE=0 STRATA_SM61_TABLE=0` and the route off the new build measures as the old one (210 /
365 tok/s, 33.3 tok/s).
Measured and **not** changed: the latency-hidden hyper-connection read (`STRATA_GR_FAST=1`) is slower on the P40 below
5 tokens (82 -> 88 us at one); `STRATA_PREFILL_CPU_SHARE=auto` and `STRATA_ADAPT_OVERLAP=1` gave nothing.
Two things to know when measuring on such a card:
- **Temperature.** A P40 has no fan of its own. This one reaches 87-89 C in a few minutes of work; at 89 C the driver
reports `SW Thermal Slowdown: Active` and the clock falls from 1,531 to ~1,300 MHz (`nvidia-smi -q -d PERFORMANCE`,
`nvidia-smi --query-gpu=temperature.gpu,clocks.gr --format=csv -l 2`). Start every arm of an A/B at the same
temperature, or the later arm loses.
- **Greedy answers are not identical run to run on a CPU without AVX-512**, even with `--prompt-cache 0
--adapt-swaps 0 --pcie-frac 0`: an expert computed on the CPU rounds differently for a lone token than in a window
of several (`native_gu_mt_min`), and how many drafts a window holds depends on timing. Add `STRATA_IQ_MT_MIN=1`
(and `STRATA_PREFILL_CPU_SHARE=0`, `STRATA_ROUTE_TAIL_SKIP=0`) to compare two builds' answers byte for byte.
### Quadro RTX 8000: measurements of existing switches
The Quadro RTX 8000 is Turing (sm_75): it has FP16 tensor cores, but no native BF16 or TF32 tensor cores. Two
**existing, opt-in** CUDA switches are worth comparing on your own workload:
- `STRATA_BF16_TC=1` converts BF16 projection operands to FP16 and uses FP16 tensor cores with FP32 accumulation.
The default on sm_75 is off; `=0` restores the cuBLAS BF16 path. The switch applies to 7.x cards, not the native
BF16 path on sm_80 and newer. Conversion clamps finite values outside FP16's range to ±65504; small values and
differently ordered sums can round differently.
- `STRATA_SELECT_SIMT=1` uses the tiled FP32 QSA block scorer on the pre-sm_80 CUDA path instead of the default
warp scorer. `=0` keeps the warp scorer. It does not enable TF32 on Turing, and its summation order differs.
The existing paths are in [gemm.cu](../src/prefill/gemm.cu) and [qsa_select.cu](../src/kernels/cuda/qsa_select.cu).
Set the environment before starting the engine, or set the switches in the model config's `env` object, then
restart. For example, this config fragment opts into both (keep the config's other settings):
```json
"env": {
"STRATA_BF16_TC": "1",
"STRATA_SELECT_SIMT": "1"
}
```
This section records an investigation, **not a new kernel or a change to defaults**. Measurements on 2026-10-07
used one 48 GiB Quadro RTX 8000 (physical GPU 3), a 260 W power limit, a Xeon W-2295, CUDA 12.4 and GCC 13.
The source-built engine was based on v0.1.40.3 with other local changes integrated; unrelated opt-in changes
were disabled in these arms. These are not measurements of an otherwise clean documentation-only checkout,
not results at the launcher's 200 W setting, and not a speed promise for every sm_75 card.
The model was IQ3_S with MTP (`--spec 4`), `--max-context 262144`, INT8 KV with `--kv-resident 32768`,
`--pcie-frac 0.20`, a fixed `--expert-cache 19000` request and `--prefill auto` (8,192-token ceiling).
The engine reported 21,189 cache slots including the prompt loan. Prompt caching and adaptive swaps were off
(`--prompt-cache 0 --adapt-swaps 0`). Each size used three fresh, seeded synthetic prompts and 128 output tokens
per request; the table gives median engine-reported rates, not end-to-end server latency.
| Existing switch settings | 537 tokens: prompt / decode tok/s | 4,057 tokens: prompt / decode tok/s | 32,057 tokens: prompt / decode tok/s |
| Switches | 537 tokens: prompt / decode tok/s | 4,057 tokens | 32,057 tokens |
| --- | --- | --- | --- |
| BF16_TC=0, SELECT_SIMT=0 | 493 / 86.8 | 1,055 / 79.9 | 1,135 / 70.6 |
| BF16_TC=1, SELECT_SIMT=0 | 661 / 83.7 | 1,262 / 75.4 | 1,416 / 68.4 |
| BF16_TC=1, SELECT_SIMT=1 | 679 / 87.1 | 1,290 / 78.2 | 1,501 / 71.6 |
BF16_TC improved prompt throughput by about 20–34%, but did not improve decode in these comparisons. Adding
SELECT_SIMT improved the 32K prompt rate by about 6% over BF16_TC alone; this is not a universal decode win.
BF16_TC changed replies in 3 of the 9 paired requests, so use it only if different rounding is acceptable.
All five basic arithmetic/JSON checks passed in each arm, which is a smoke check, **not a model-quality study**.
A separate synthetic QSA scorer check at 131,072 context tokens matched selected IDs for all 256 queries and
passed its FP64 accuracy gate. That check does not prove identical selections on arbitrary prompts or model quality.
Two other existing tuning options did not justify a recommendation on this machine:
- `--prefill auto:16384`, with BF16_TC on and SELECT_SIMT off, increased the prompt's cache loan from 4.38 to
7.38 GiB for only about 0.6% more 32K prompt throughput (1,425 versus 1,416 tok/s). Just 24 MiB of VRAM remained
at the last graph capture. Keep the 8,192 ceiling rather than spend that memory margin for this small gain.
- `STRATA_PREFILL_CPU_SHARE=auto` gave no gain at the two tested sizes (537 and 4,057 tokens): prompt rates were
647 and 1,253 tok/s versus 661 and 1,262 with BF16_TC alone; decode was also lower. Larger prompts were not
tested for this setting.
## AMD: building gfx906 and gfx1012
Setup does not build these; build by hand and run `serve/server.py` with a config, as on any other card.
- **gfx906** (MI50 / MI60 / Radeon VII, wave64): a separate opt-in build, `-DSTRATA_HIP_GFX906=ON
-DCMAKE_HIP_ARCHITECTURES=gfx906` (not `STRATA_ENABLE_HIP`). Current ROCm no longer ships gfx906 libraries; the
reporter used a community ROCm 7.14 image. Recipe, kernels and measurements: [AMD_HIP.md](AMD_HIP.md#gfx906-instinct-mi50--mi60-radeon-vii-wave64-built-from-source).
- **gfx1012** (RX 5500 XT): the wave32 backend, `-DSTRATA_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1012`. HIP 5.7
(Ubuntu's packages) works: older hipBLAS (0.x) is used through rocBLAS, the legacy HIP names and the missing
`__syncwarp` are version-gated, and RDNA1's missing signed dot4 uses llama.cpp's SDWA sequence
(`-DSTRATA_GFX1012_PORTABLE_DOT=ON`: the portable one).
## Reports welcome
These paths stay experimental until more people run them. A report with the card, the driver / ROCm version, the
model and the engine log (`strata-<model>.log`) in an issue helps; [COMMUNITY_BENCHMARKS.md](COMMUNITY_BENCHMARKS.md)
has the format for measurements.
`SELECT_SIMT` added about 6% to the 32K prompt rate over BF16_TC alone. `--prefill auto:16384` gained about 0.6% at 32K for 3 GiB
more prompt-cache loan, and `STRATA_PREFILL_CPU_SHARE=auto` gave no gain, so neither is recommended. All five basic arithmetic and
JSON checks passed in each arm; that is a smoke check, not a quality study.
docs/ORCA.md
+50 -41
View File
@@ -1,25 +1,30 @@
# OrcaRouter IQ3_XXS compatibility
Validated on Linux with an RTX 5090 (32 GB), Ryzen 9 9950X3D and 128 GB RAM on 2026-09-27.
This page is for people who want to run `orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF` (**IQ3_XXS**, two shards,
85.20 GB) with Strata. It is a manual conversion workflow from a [source checkout](SOURCE_INSTALL.md). OrcaRouter is
not an installer menu entry, and there is no `--family orca` command.
The target is `orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, **IQ3_XXS** (two shards, 85.20 GB total).
It has Strata's Qwen4Exp geometry, but ordinary quantization also compresses the hyperconnection,
small attention and PLE projections which Strata's kernels read as BF16.
## Quick start
`tools/iq_pack.py --compat-bf16` expands only those projections, rounding to BF16 using nearest-even.
This adds rounding relative to the GGUF's dequantized weights; it does not recover the original BF16
checkpoint or its quality. Expert weights, native attention weights, embeddings and the 28.8 GB PLE
lookup table remain unchanged. The latter stays on disk. The pack records its conversions in
`compat-bf16.json`.
1. Build Strata normally, then pack the model ([Preparation](#preparation)).
2. Prepare the optional draft head (MTP) with the same section's commands.
3. Save the config from [Local server](#local-server) and start the server.
The native loader retains this pack's converted PLE key. Its existing native Q2_0 key path remains
available to the original model. Hugging Face snapshot symlinks are supported; all split shards must
finish downloading before packing. Pass the snapshot filename, not its hash-named blob target.
## How it works
The model has Strata's Qwen4Exp geometry. Ordinary quantization also compresses the hyperconnection, small attention
and PLE projections, which Strata's kernels read as BF16. `tools/iq_pack.py --compat-bf16` expands only those
projections, rounding to BF16 with nearest-even. This adds rounding relative to the GGUF's dequantized weights. It does
not recover the original BF16 checkpoint or its quality.
Expert weights, native attention weights, embeddings and the 28.8 GB PLE lookup table stay unchanged, and the lookup
table stays on disk. The pack records its conversions in `compat-bf16.json`. Hugging Face snapshot symlinks work. All
split shards must finish downloading before packing. Pass the snapshot filename, not its hash-named blob target.
## Preparation
Build Strata normally using the pinned llama.cpp dependency. Python needs numpy, regex and gguf-py
from that dependency (`STRATA_GGUF_PY` can point to its `gguf-py` directory).
Python needs numpy, regex and gguf-py from the pinned llama.cpp dependency. `STRATA_GGUF_PY` can point to its
`gguf-py` directory.
```sh
.venv/bin/python tools/iq_pack.py \
@@ -27,11 +32,10 @@ from that dependency (`STRATA_GGUF_PY` can point to its `gguf-py` directory).
--out packs/orca-iq3_xxs --compat-bf16
```
Use this model's own tokenizer exported into the pack. Do not share another model's `dense.bin`,
and do not rename the Orca files to impersonate one of setup's GSQ-RCO models.
Use this model's own tokenizer, exported into the pack. Do not share another model's `dense.bin`, and do not rename
the Orca files to pass as one of setup's GSQ-RCO models.
Strata's persistent server also uses the MTP runtime (optional: without `--mtp` it drafts by lookup only), prepared by
the existing tools:
The persistent server can use the MTP runtime. It is optional: without `--mtp` the server drafts by lookup only.
```sh
.venv/bin/python tools/mtp_fetch.py fetch --out mtp
@@ -40,15 +44,15 @@ the existing tools:
cp data/draft_vocab.bin mtp/rt/draft_vocab.bin
```
This uses the original model's draft head, as setup does for Swift. The target model verifies its
proposals; draft acceptance and performance must be measured for this fine-tune.
This uses the original model's draft head, as setup does for Swift. The target model verifies every proposal. Measure
draft acceptance and speed for this fine-tune; the original model's numbers do not carry over.
## Local server
Save the following as `strata-orca-iq3_xxs.json` at the repository root, replacing both `/path/to/`
entries with the **same first shard**. Orca's PLE table is in shard 1. Start with a 32K context and
512-token prefill chunks; the expert cache sizes itself to available VRAM. The expert arena alone
needs about 49.8 GiB of available system RAM, plus draft/runtime buffers and other applications.
Save the following as `strata-orca-iq3_xxs.json` at the repository root. Replace both `/path/to/` entries with the
**same first shard**, because Orca's PLE table is in shard 1. The config uses a 32K context and 512-token prefill
chunks. The expert cache sizes itself to the available VRAM. The expert arena alone needs about 49.8 GiB of free system
RAM, plus draft and runtime buffers and your other applications.
```json
{
@@ -70,27 +74,32 @@ needs about 49.8 GiB of available system RAM, plus draft/runtime buffers and oth
}
```
Run from the repository root:
Run it from the repository root:
```sh
.venv/bin/python -m serve.server --engine strata --config strata-orca-iq3_xxs.json --port 8080
```
The web interface is at `http://127.0.0.1:8080`; API clients use `http://127.0.0.1:8080/v1`.
Vision is not configured in this text-only example. Performance of the original GSQ-RCO model
does not establish this fine-tune's speed or accuracy.
The web interface is at `http://127.0.0.1:8080`. API clients use `http://127.0.0.1:8080/v1`. This example is text-only;
vision is not configured.
## Scope and validation
## Scope
- IQ3_XXS is the target of this change; other Orca quantizations are not validated.
- IQ3_M additionally uses Q5_0 expert down matrices, which the native GPU expert path does not support.
- The installer model menu is unchanged. This is an explicit local packing workflow.
- Focused conversion and split-file tests: `.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`.
- All eight packing tests, 17 server tests and the GPU gated-residual parity check passed.
- The complete model packed successfully: 460 tensors converted, 1.39 GiB of converted BF16 weights;
1.43 GiB total packed dense weights. The 48 expert layers remain IQ3_XXS gate/up and IQ4_NL down.
- Real server checks passed: arithmetic, Python code, translation, a longer explanation, multi-turn context
reuse (25 prompt tokens reused), streaming cancellation and a subsequent correct response. The engine
PID stayed unchanged. These are smoke tests, not a quality benchmark or a full-context stress test.
- With the configuration above, a 111-token explanation generated at 77.7 tokens/s (engine decode timing);
total request time was 1.77 seconds. This single short measurement is not a general throughput claim.
- IQ3_XXS is the only validated Orca quantization.
- IQ3_M uses Q5_0 expert down matrices, which the native GPU expert path does not support.
- The installer model menu does not list this model.
- Packing tests: `.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`.
## History / older measurements
> **Measured on:** Linux, RTX 5090 (32 GB), Ryzen 9 9950X3D, 128 GB RAM, 2026-09-27, the configuration above.
| Check | Result |
| --- | --- |
| Packing | 460 tensors converted, 1.39 GiB of converted BF16 weights, 1.43 GiB total packed dense weights; the 48 expert layers stay IQ3_XXS gate/up and IQ4_NL down |
| Tests | all eight packing tests, 17 server tests and the GPU gated-residual parity check passed |
| Server smoke tests | arithmetic, Python code, translation, a longer explanation, multi-turn reuse (25 prompt tokens reused), streaming cancellation and a correct answer afterwards; the engine PID stayed unchanged |
| Decode speed | 111-token explanation at 77.7 tokens/s (engine decode timing), 1.77 s total request time |
These are smoke tests and one short measurement, not a quality benchmark, a full-context stress test or a general
throughput claim.
docs/QWEN36.md
+207 -158
View File
@@ -1,23 +1,51 @@
# Qwen3.6-35B-A3B: pictures, several requests at once and speed
Strata runs [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (35B mixture of experts, 3B active, 256 experts, top-8) from
[Unsloth's GGUFs with the MTP layer](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) (UD-Q4_K_XL, UD-Q6_K_XL, Q8_0) on its own
engine path (`src/qwen35`; `strata` switches to it from the file's architecture). Pick it in the app's model list, in `strata launch` or in setup (`--family qwen36`). Besides text it
reads **pictures** and serves **several requests at once**, the same two options the Flash-Next models have.
This page is for people who run [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (35B mixture of
experts, 3B active, 256 experts, top-8) with Strata and want to know how pictures, parallel requests and the prompt
chunk behave. For sizes and memory, see [MODELS.md](MODELS.md).
Measured on an AMD Radeon AI PRO R9700 32 GB (HIP, Linux, UD-Q4_K_XL, every expert in VRAM) and on an RTX 3060 12 GB (CUDA, 20 threads,
UD-Q4_K_XL in the hybrid mode: 22-37 of 256 experts per layer in VRAM, the rest on the CPU), llama.cpp 972b5fc on the same files.
Strata runs [Unsloth's GGUFs with the MTP layer](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF)
(UD-Q4_K_XL, UD-Q6_K_XL, Q8_0) on its own engine path (`src/qwen35`; `strata` switches to it from the file's
architecture). It runs on NVIDIA and AMD cards. The Intel Arc and Mac engines do not run it.
## Quick start
Pick it in the app's model list, in `strata launch` or in setup:
```sh
strata launch --family qwen36 --model ud-q4_k_xl
strata launch --family qwen36 --model ud-q4_k_xl --vision on --parallel 2
```
| Option | Effect |
| --- | --- |
| `--vision on\|off\|gpu\|cpu` | Terminal: pictures and which reader. Source setup uses `--vision yes\|no\|gpu\|cpu` and `--vision-tokens N`. |
| `--parallel N` | How many requests are answered at once, 1 to 8. Source setup has the same option. |
| `--prefill 512` | Engine flag in the config's `args`, not a `strata launch` option: the prompt chunk size ([Prompt chunk](#prompt-chunk)). |
## Pictures
The vision encoder is Unsloth's `mmproj-F16.gguf` (0.9 GB, pinned revision, SHA-256 checked by setup): a 27-layer ViT with a 2x2 merger,
the same projector family as the Flash-Next models' (`qwen3vl_merger`; only its output width differs: 2048 here, 2560 there). Strata runs
it with the same helper (`strata-vision`, llama.cpp's `mtmd`), so the encoder choices are the Flash-Next ones
([Images](DETAILS.md#images-vision)). Pictures are on by default, the same rule as for every model on NVIDIA and AMD cards:
the CPU reader under 24 GB of graphics memory (a few seconds per picture, no VRAM), the card's reader from 24 GB (GPU encoder,
~1.2 GB of VRAM at up to 1,024 image tokens; on AMD it is `strata-vision-hip`, with the CPU reader as its fallback).
`--vision yes|no|gpu|cpu` and `--vision-tokens N` work as there; what you ask for is kept, with a note. The Mac engine does
not run this model.
The vision encoder is Unsloth's `mmproj-F16.gguf` (0.9 GB, pinned revision, SHA-256 checked by setup): a 27-layer ViT
with a 2x2 merger, the same projector family as the Flash-Next models' (`qwen3vl_merger`; only its output width
differs: 2048 here, 2560 there). Strata runs it with the same helper (`strata-vision`, llama.cpp's `mtmd`), so the
encoder choices are the Flash-Next ones ([Images](DETAILS.md#images-vision)).
Pictures are on by default, by the same rule as for every model on NVIDIA and AMD cards:
- The CPU reader runs under 24 GB of graphics memory. It takes a few seconds per picture and no VRAM.
- The card's reader runs from 24 GB (GPU encoder, about 1.2 GB of VRAM at up to 1,024 image tokens). On AMD it is
`strata-vision-hip`, with the CPU reader as its fallback.
Your explicit choice is kept, with a note. A picture becomes up to 1,024 tokens of the context (the CPU default is 300).
The engine puts the encoder's rows where the prompt has `<|image_pad|>` tokens and gives every row its 2-D position,
interleaved M-RoPE as llama.cpp does for this model (rotary sections 11 / 11 / 10).
Pictures work with several requests at once, with the MTP drafts and with the conversation cache. The same conversation
sent again continues from the held state when its pictures are the same (matched by the server's key for each
picture). A different picture starts that conversation again.
> **Measured on:** RTX 3060 12 GB (CUDA) and an R9700 box (32 threads) and a 3060 box (10 threads), UD-Q4_K_XL, Strata
> 0.1.45.
| Encoder | Time per picture (640x400 = 260 tokens / 800x450 = 350 tokens) | VRAM |
| --- | --- | --- |
@@ -25,179 +53,200 @@ not run this model.
| CPU, 32 threads (R9700 box) | 0.9 s / 1.1 s | none |
| CPU, 10 threads (3060 box) | 1.6 s / 2.1 s | none |
A picture becomes up to 1,024 tokens of the context (the CPU default is 300). The engine puts the encoder's rows where the prompt has
`<|image_pad|>` tokens and gives every row its 2-D position, interleaved M-RoPE exactly as llama.cpp does for this model (rotary sections
11 / 11 / 10, a picture's rows at (t, h, w) = (p, p + row, p + column), the text after it continuing at p + max(columns, rows)).
## Several requests at once
**Checked against llama.cpp** (its `mtmd` and its own graph, the encoder on the CPU on both sides, the same prompt ids, the answer
llama.cpp gives fed to both so the logits are compared step by step, 24 steps; the "floor" is llama.cpp's CPU backend against its own
HIP run, the noise two correct engines have on this model):
`"parallel": N` in the model's config (`strata-qwen36-<size>.json`) or `--parallel N` makes the engine keep N
conversations open (`--batch N`, up to 8) and decode them together. Every window carries one token of each
conversation, so the dense weights, the head and every routed expert that conversations share are read once for all of
them. The server speaks the same engine protocol as for Flash-Next ([BATCHING.md](BATCHING.md)).
| Picture and question | Prompt tokens | First token | Mean KL, llama.cpp HIP vs Strata | Floor (llama.cpp CPU vs HIP) |
A new request waits only for the read of its own prompt, which runs between the decode windows of the others. A
conversation continues in the slot that holds its start, or the engine copies the longer held start from another slot.
A request that runs alone (the solo path, with its MTP drafts) moves into a batch slot when another arrives, and back
when it is alone again, and gives the same text either way.
`STRATA_Q36_READ_WINDOWS=4` lets the decoding slots produce more tokens during a long prompt read, at the cost of a
slower read. The default is one decode window per chunk.
**MTP drafts.** A request that has the engine to itself keeps its MTP drafts. With several requests the batch window has
no drafts unless `--batch-mtp` is in the config's `args` (opt-in). Each slot's window then also carries the proposal its
MTP block made, one per slot, up to the rows to spare in the 8-row window.
**What a slot costs.** Each slot keeps its own cache in VRAM: 1.5 GB at a context of 32K (fp32 keys and values of the 10
attention layers and the MTP block, 45 KB per token) and 0.07 GB of recurrent state. Setup writes the context for one
conversation, so N slots take N times that from the room the experts would have. On a card that does not hold every
expert (12 and 16 GB, the hybrid mode) the expert slots lose that room, and a batch reads more distinct experts from RAM.
Setup offers `--parallel` where the sessions do not eat the experts' room, says the price in the same breath and asks
"How many requests at once?" Enter keeps one at a time, and any number you give is kept.
| Card type | Slots setup offers |
| --- | --- |
| Every expert still fits | Up to 3 to 4 (3 on a 32 GB card at 32K) |
| Hybrid mode | The count whose sessions take at most 35% of the room the experts would have. 12 GB: 4 slots at 8K, none at 32K. 16 GB: 2 slots at 32K. |
The engine says what the sessions take before it allocates them and refuses, with the numbers, a count that cannot fit.
In the hybrid mode the text of a request can differ a little between running alone and running beside others. The
experts that sit on the GPU or on the CPU change with the routing of all the requests, and the two compute with
different rounding. With the slot tables fixed (`STRATA_Q36_FIXED_SLOTS=1`, a test setting) every slot equals its solo
run there too.
## Prompt chunk
The MoE prompt kernels read every expert once per chunk, so a bigger chunk feeds them more rows per expert. With every
expert in VRAM the engine takes the largest of 4096 / 2048 / 1024 that fits beside the experts, the 0.77 GiB reserve
and the sessions' caches.
| Situation | Chunk |
| --- | --- |
| Every expert in VRAM, 8K to 64K context | 4096 |
| Every expert in VRAM, 100K context | 2048 |
| Every expert in VRAM, 131K context | 512 |
| `--batch 2..4` | 1024 at most, so the other slots are not held for a whole chunk |
| `--batch 8` | 512 |
| Hybrid mode | 2048; 1024 or 512 on a card with 7.5 or 5.5 GiB free |
In the hybrid mode a bigger chunk takes expert slots (an RTX 3060 at chunk 4096 read prompts 5% faster and decoded 13%
slower). `--prefill 512` (or `STRATA_Q36_CHUNK=512`) gives the 512 size, and then the text is the base engine's, bit
for bit. The chunk size changes the GEMM tiles the BLAS picks and with them the last bits of the prompt state. This is
the size of the llama.cpp CPU-against-HIP difference on the same file, not a quality loss. A prompt shorter than the
chunk (every chat turn after the first) is one chunk either way.
## Speed and exactness
Every change on the decode path gives the old kernels' bits. `STRATA_Q36_VERIFY=1 strata ... --no-graph` runs the old
kernel beside each new one on the real activations and compares them bit for bit. `STRATA_Q36_OLD_KERNELS=1` switches
the new kernels off. Gate/up for Q4_K experts uses the V2K kernels (`STRATA_EXPERT_V2K`, the default on gfx12).
> **Measured on:** AMD Radeon AI PRO R9700 32 GB (HIP gfx1201), UD-Q4_K_XL, Strata 0.1.45 against the engine before it,
> 5 interleaved pairs, medians; llama.cpp 972b5fc HIP on the same file (`-fa 1`).
| | Before | 0.1.45 | llama.cpp |
| --- | ---: | ---: | ---: |
| Decode tg128 | 83.2 tok/s | 117.8 | 81.9 to 82.1 |
| Prompt pp4096 | 3,453 | 6,220 (chunk 4096) | 3,794 (`-ub 2048`) / 3,042 (`-ub 512`) |
More numbers are in [History](#history--older-measurements).
## Limits
- Eight slots at most (the window holds 8 rows). `--batch-mtp` shares those rows between slots and proposals.
- A sampling request in a batch decodes one token per window without proposals.
- `logit_bias` is not available on this engine path. The server refuses a request that has it (as it does with
`--batch` for Flash-Next).
- A slot that stopped being alone goes back to the solo path only through the server's own hand-over. The MTP block's
cache has holes for the tokens a slot decoded in batch windows, so the drafts of a later solo stretch are a little
worse. The text is not affected.
- Intel Arc (SYCL) and Mac do not run this family.
## Testing
The gates are in `tools/qwen35/` and talk to the engine through its serve protocol (no server needed). Run them under
the box's GPU lock.
| Script | Checks |
| --- | --- |
| `q36_solo_gate.py` + `q36_cmp_gate.py` | The 10 parity prompts x 128 greedy tokens through `GEN`, compared with another build's. |
| `q36_batch_gate.py --slots N --waves --long 2500` | Every slot's tokens equal its solo tokens, slots reused, a long prompt read beside decoding slots. |
| `q36_proto_edge.py` | `max_new` 1, `BSTOP`, a busy slot, `STOP` during a prompt read, held-state reuse, sampled requests, the context's end. |
| `q36_interleave_probe.py` | What the decoding slots see while a long prompt is read. |
| `q36_vision_cmp.py` (with `q36_mtmd_dump.cpp` built against llama.cpp) | The picture comparison against llama.cpp. |
| `q36_vision_proto.py` | Pictures through `GENI` / `BGENI` (solo, in a batch, the conversation cache). |
| `q36_vision_serve.py`, `q36_serve_par.py`, `q36_serve_handoff.py` | The same through the server. |
| `q36_par_bench.py` | The aggregate-throughput A/B against `llama-server -np N`. |
## History / older measurements
### Pictures checked against llama.cpp
> **Measured on:** R9700 32 GB (HIP), UD-Q4_K_XL, Strata 0.1.45. llama.cpp's `mtmd` and its own graph, the encoder on the
> CPU on both sides, the same prompt ids, llama.cpp's answer fed to both, logits compared step by step over 24 steps.
> The floor is llama.cpp's CPU backend against its own HIP run.
| Picture and question | Prompt tokens | First token | Mean KL, llama.cpp HIP vs Strata | Floor |
| --- | ---: | --- | ---: | ---: |
| 640x400, text in the image | 288 | same | 0.029 | 0.056 |
| 640x400, list the shapes | 294 | same | 0.035 | 0.039 |
| 800x450, text in the image | 378 | same | 0.021 | 0.031 |
| 800x450, describe the shapes | 381 | same | 0.0005 | 0.0019 |
| 1024x1024, the title (1,024 image tokens, read in 3 chunks) | 1,042 | same | 0.0022 | 0.0026 |
| two pictures in one request, the text in each | 655 | differs (a near tie: the floor's KL at this step is 0.16 too) | 0.014 | 0.022 |
| 1024x1024, the title (1,024 image tokens, 3 chunks) | 1,042 | same | 0.0022 | 0.0026 |
| two pictures in one request | 655 | differs (a near tie; the floor's KL at this step is 0.16) | 0.014 | 0.022 |
The answers are the same text as llama.cpp's: `STRATA 42` (with llama.cpp's own spelling ` STRATA42`), `CODE 4217`, `Red circle / Blue square
/ Green triangle`, `A red square is positioned on the left ... A blue circle is positioned on the right ...`. With the picture options on,
a request without pictures gives the same tokens as without them (10 prompts x 128 greedy tokens, identical).
The answers are the same text as llama.cpp's. With the picture options on, a request without pictures gives the same
tokens as without them (10 prompts x 128 greedy tokens).
Pictures work with several requests at once, with the MTP drafts and with the conversation cache: the same conversation sent again (a chat
app resends its pictures) continues from the held state when its pictures are the same (they are matched by the server's key for each
picture); a different picture starts that conversation again.
### Aggregate decode speed, several requests at once
## Several requests at once
> **Measured on:** same prompts, greedy, 128 new tokens each, wall time from first request to last answer, 5
> interleaved pairs, medians, Strata 0.1.45 with `--batch 8 --mtp 0` against `llama-server -np N`.
`"parallel": N` in the model's config (`strata-qwen36-<size>.json`, or `--parallel N` at setup) makes the engine keep N conversations open
(`--batch N`, up to 8) and decode them together: every window carries one token of each conversation, so the dense weights, the head and
every routed expert two conversations share are read once for all of them. The server needs no change: it speaks the same engine protocol
as for Flash-Next ([BATCHING.md](BATCHING.md)). A request that arrives while another is read waits only for that read: the prompt of a
new request is read **between** the decode windows of the others, and a conversation continues in the slot that holds its start (or the
engine copies the longer held start from another slot). A 3,008-token prompt read in 0.95 s (3,200 tok/s, 6 chunks) beside three decoding
slots: each of them paused at most 160 ms (one chunk) instead of the whole read, and produced 20 tokens meanwhile (one window per chunk,
the default); `STRATA_Q36_READ_WINDOWS=4` gives them 65 tokens during a 1.14 s read (more of the time, a slower read).
A request that runs alone (the solo path, with its MTP drafts) and is moved into a batch slot when another arrives, and back when it is
alone again, gives the text it gives running alone (the server's hand-over reuses the held state).
| Card | Requests | Strata tok/s | llama.cpp | Ratio |
| --- | ---: | ---: | ---: | ---: |
| R9700 32 GB, UD-Q4_K_XL | 1 | 83.9 | 73.8 | 1.14 |
| | 2 | 173.5 | 107.4 | 1.61 |
| | 4 | 252.0 | 165.0 | 1.53 |
| | 8 | 325.4 | 208.1 | 1.56 |
| RTX 3060 12 GB, hybrid (llama.cpp `-ncmoe 28 -t 8`, the best split of a sweep) | 1 | 64.0 | 41.1 | 1.56 |
| | 2 | 85.1 | 52.5 | 1.62 |
| | 4 | 99.3 | 63.0 | 1.58 |
Aggregate decode speed, N requests at once, same prompts, greedy, 128 new tokens each, wall time from the first request to the last
answer, **R9700 32 GB, UD-Q4_K_XL, llama.cpp `llama-server -np N`** (5 interleaved pairs, median; Strata `--batch 8 --mtp 0`):
All pairs were faster for Strata. Every slot's tokens equal the tokens of the same request alone.
| Requests at once | Strata | llama.cpp | Ratio | Pairs faster |
| ---: | ---: | ---: | ---: | ---: |
| 1 | 83.9 tok/s | 73.8 | 1.14 | 5/5 |
| 2 | 173.5 | 107.4 | 1.61 | 5/5 |
| 4 | 252.0 | 165.0 | 1.53 | 5/5 |
| 8 | 325.4 | 208.1 | 1.56 | 5/5 |
### Several requests with the base engine, and with `--batch-mtp`
**RTX 3060 12 GB, UD-Q4_K_XL in the hybrid mode** (each engine with the slots it needs: `--batch N` takes room from the expert slots; llama.cpp
`llama-server -np N -ncmoe 28 -t 8`, the best split of a sweep on this card; 5 interleaved pairs, medians; the box was shared with other
jobs between runs, never during one):
> **Measured on:** R9700 32 GB, 5 interleaved pairs, Strata 0.1.45 (before the speed work below).
| Requests at once | Strata | llama.cpp | Ratio | Pairs faster |
| ---: | ---: | ---: | ---: | ---: |
| 1 | 64.0 tok/s | 41.1 | 1.56 | 5/5 |
| 2 | 85.1 | 52.5 | 1.62 | 5/5 |
| 4 | 99.3 | 63.0 | 1.58 | 5/5 |
| Requests | `--mtp 0` | `--mtp 2 --batch-mtp` | Ratio |
| ---: | ---: | ---: | ---: |
| 1 | 83.7 tok/s | 134.0 | 1.60 |
| 2 | 173.1 | 207.0 | 1.20 |
| 4 | 251.0 | 275.0 | 1.10 |
| 8 | 324.0 | 320.5 | 0.99 (no row to spare: no proposals) |
Every slot's tokens equal the tokens of the same request alone (the base engine's), at 2, 4 and 8 slots, with a 2.5K-token prompt read beside
decoding slots, with slots reused for later requests, with pictures and with sampling (same seed). Through the server: four concurrent chat
requests give the texts of the same four one after the other.
> **Measured on:** hybrid cards, UD-Q4_K_XL, medians of 5 interleaved pairs, `--mtp 2`; "together" is the total tokens
> per second of N simultaneous requests against one request at a time with the draft layer.
**MTP drafts.** A request that has the engine to itself keeps its MTP drafts (the server sends it alone while nothing else runs). With
several requests the batch window has no drafts unless `--batch-mtp` is in the config's `args` (opt-in): every slot's window then also
carries the proposal its MTP block made (one per slot, up to the rows to spare in the 8-row window) and a confirmed proposal moves two
tokens in the window. Same prompts, 5 interleaved pairs, R9700:
| Requests at once | `--mtp 0` | `--mtp 2 --batch-mtp` | Ratio | Pairs faster |
| ---: | ---: | ---: | ---: | ---: |
| 1 | 83.7 tok/s | 134.0 | 1.60 | 5/5 |
| 2 | 173.1 | 207.0 | 1.20 | 5/5 |
| 4 | 251.0 | 275.0 | 1.10 | 5/5 |
| 8 | 324.0 | 320.5 | 0.99 | 0/5 (no row to spare: no proposals) |
The text is the same with and without it (every slot equal to its solo tokens).
**What a slot costs.** Each slot keeps its own cache in VRAM: 1.5 GB at a context of 32K (fp32 keys and values of the 10 attention layers
and the MTP block, 45 KB per token) and 0.07 GB of recurrent state. Setup writes the context for one conversation, so N slots take N times
that from the room the experts would have: on a card that does not hold every expert (12 and 16 GB, the hybrid mode) that is room the
expert slots lose, and a batch reads more distinct experts from RAM. Setup **offers `--parallel`** where the sessions do not eat the experts' room: up to 3-4 where every expert still fits (3 on a 32 GB card at
32K), and on a card that does not (the hybrid mode) the count whose sessions take at most 35% of the room the experts would have, so it
depends on the context you chose (12 GB: 4 slots at 8K, none at 32K; 16 GB: 2 slots at 32K). It says the price in the same breath, asks
"How many requests at once?" (Enter keeps one at a time), and any number you give is kept. Measured on the hybrid cards (UD-Q4_K_XL, medians of 5
interleaved pairs, `--mtp 2` as setup writes it; "together" = total tokens per second of N simultaneous requests, against ONE request at a time
with the draft layer):
| card, context | slots | one request alone, vs 1 slot | N together, vs one at a time |
| Card, context | Slots | One request alone, against 1 slot | N together, against one at a time |
| --- | ---: | ---: | ---: |
| RTX 3060 12 GB, 8K | 2 / 4 / 8 | -3.5% / -4.3% / -12% | 0.99x / 1.15x / 1.10x |
| RTX A4000 16 GB, 32K | 2 / 4 | -10.6% / -29.9% | 1.03x / 0.76x |
| Tesla P100 16 GB, 32K | 2 / 4 | -7% / -23% | 0.98x / 0.81x |
Without the draft layer (`--mtp 0`) the same slots give 1.26-1.64x together (3060 at 8K: 4 slots 1.64x for -5.5% alone), but a single request
with drafts is already about 1.6x as fast as one without, so on a small card the main gain is that a second user starts at once; the
speed gain is small. On a card that holds every expert (R9700 32 GB, 8K prompts) 4 slots run 1.45x one-at-a-time-with-drafts (1.59x with
`--batch-mtp`). The engine says what the sessions take before it allocates them and refuses (with the numbers) a count that cannot fit.
Without the draft layer (`--mtp 0`) the same slots give 1.26x to 1.64x together (3060 at 8K: 4 slots 1.64x for -5.5%
alone). A single request with drafts is already about 1.6x as fast as one without, so on a small card the main gain is
that a second user starts at once. On a card that holds every expert (R9700 32 GB, 8K prompts), 4 slots run 1.45x
one-at-a-time-with-drafts (1.59x with `--batch-mtp`).
In the hybrid mode the text of a request can differ a little between running alone and running beside others: the experts that sit on the
GPU or on the CPU change with the routing of all the requests, and the two compute with different rounding. With the slot tables fixed (`STRATA_Q36_FIXED_SLOTS=1`, a test setting) every slot equals its
solo run there too, with and without `--batch-mtp`.
### Prompt read beside decoding slots
## Speed (0.1.45)
> **Measured on:** R9700 32 GB, Strata 0.1.45, three decoding slots.
Decode and prompt kernels of this engine path were rewritten for speed; **every change on the decode path gives the old kernels' bits**
(`STRATA_Q36_VERIFY=1 strata ... --no-graph` runs the old kernel beside each new one on the real activations and compares them bit for bit:
0 mismatches in 18,360 / 8,187 / 6,399 comparisons on the R9700 and the RTX 3060; `STRATA_Q36_OLD_KERNELS=1` switches them all off). The
10 parity prompts (greedy 128) give the base engine's tokens, plain and with `--mtp 2`. Where the prompt's chunk size changes, the numbers
change a little: see "Prompt chunk".
A 3,008-token prompt was read in 0.95 s (3,200 tok/s, 6 chunks). Each decoding slot paused at most 160 ms (one chunk)
and produced 20 tokens meanwhile. With `STRATA_Q36_READ_WINDOWS=4` they produced 65 tokens during a 1.14 s read.
**What was slow** (rocprofv3, R9700, UD-Q4_K_XL, plain decode, 11.25 ms of kernel time per token): the big Q8_0 products (attention, GDN
projections, head: 1.9 GB per token) run at 530 GB/s, the memory bound. The rest did not: the shared expert's and the attention k/v products
(140 per token) at 18 us, the Q4_K gate/up kernel at 250 GB/s, `k_moe_out` (one block) 17 us, `k_group` (one thread walking 256 experts) 9 us.
Now: those products one warp per row (a launch of 2048 warps took 21 us on gfx1201 where 1024 warps took 5 us, so the grid is capped), shared
gate+up+SwiGLU+q8_1 in one launch, the MoE end with 1024 threads and the next layer's q8_1 image, add+norm+q8_1 in one launch, router top-k and
grouping in one launch, the router and the shared gate in one product, Q4_K gate/up on the V2K kernels (`STRATA_EXPERT_V2K`, gfx12 default).
The prompt's GDN conv runs in tiles of 16 tokens side by side and the delta rule loads one token ahead with two barriers per token (both exact).
### Speed work in 0.1.45
R9700 32 GB (p620, HIP gfx1201), UD-Q4_K_XL, base = the engine before this work, 5 interleaved pairs, 3 repetitions each, medians; llama.cpp
972b5fc HIP on the same file (`-fa 1`, best of `-ub` 512 / 2048):
> **Measured on:** R9700 32 GB (HIP gfx1201), UD-Q4_K_XL, 5 interleaved pairs, 3 repetitions each, medians; base = the
> engine before 0.1.45.
| | base | 0.1.45 | ratio | pairs faster | llama.cpp |
| --- | ---: | ---: | ---: | ---: | ---: |
| decode tg128 | 83.2 tok/s | 117.8 | 1.42 | 5/5 | 81.9-82.1 |
| prompt pp4096 | 3,453 | 6,220 (chunk 4096) | 1.80 | 5/5 | 3,794 (-ub 2048) / 3,042 (-ub 512) |
| `--mtp 2`, 10 prompts greedy 128 | 167.7 | 183.7 | 1.10 | 3/3 | 129 (draft-mtp, n-max 2, earlier run) |
| 1 / 2 / 4 / 8 requests at once (`--batch 8 --mtp 0`) | 85.2 / 182.4 / 271.9 / 361.1 | 122.4 / 205.1 / 303.3 / 386.9 | 1.44 / 1.12 / 1.12 / 1.07 | 5/5 | |
| | Base | 0.1.45 | Ratio |
| --- | ---: | ---: | ---: |
| `--mtp 2`, 10 prompts greedy 128 | 167.7 tok/s | 183.7 | 1.10 |
| 1 / 2 / 4 / 8 requests (`--batch 8 --mtp 0`) | 85.2 / 182.4 / 271.9 / 361.1 | 122.4 / 205.1 / 303.3 / 386.9 | 1.44 / 1.12 / 1.12 / 1.07 |
| Prompt pp4096 at chunk 512 / 1024 / 2048 / 4096 | | 3,453 / 4,786 / 5,766 / 6,223 | |
UD-Q6_K_XL on the same card (every main layer's experts in VRAM, so the prompt chunk stays 512): decode 80.4 -> 106.9 tok/s (1.33), pp4096
2,160 -> 2,200 (llama.cpp 3,486 at -ub 2048, 2,738 at -ub 512; decode 81.1), `--mtp 2` 154.7 -> 167.3. `--chunk 2048` there moves the experts
to the hybrid placement (205 of 256 per layer in VRAM): prompt 3,500 tok/s, decode 94 - an option, not the default.
UD-Q6_K_XL on the same card (every main layer's experts in VRAM, chunk stays 512): decode 80.4 to 106.9 tok/s, pp4096
2,160 to 2,200 (llama.cpp 3,486 at `-ub 2048`, 2,738 at `-ub 512`), `--mtp 2` 154.7 to 167.3. `--chunk 2048` there moves
the experts to the hybrid placement (205 of 256 per layer in VRAM): prompt 3,500 tok/s, decode 94. That is an option,
not the default.
RTX 3060 12 GB (P2003, CUDA sm_86, 20 threads), UD-Q4_K_XL in the hybrid mode (37 of 256 experts per layer in VRAM): the GPU part of a
token is the dense Q8_0 at the card's bandwidth (6 ms of the 17) and the rest waits for the CPU pool (9 ms of experts), so only the small
kernels' share is gained: decode tg128 58.5 -> 59.7 tok/s (+2%), pp2048 1,958 -> 2,022 (+3.3%), 10/10 identical, plain and `--mtp 2`
(84.3 -> 84.7). llama.cpp 972b5fc CUDA (`-ncmoe 26 -t 8 -ub 2048`): pp2048 1,328, tg128 48-52.
> **Measured on:** RTX 3060 12 GB (CUDA sm_86, 20 threads), UD-Q4_K_XL in the hybrid mode (37 of 256 experts per layer
> in VRAM), Strata 0.1.45.
**Prompt chunk (the default changed).** The MoE prompt kernels read every expert once per chunk, so a bigger chunk feeds them more rows per
expert: R9700 pp4096 at chunk 512 / 1024 / 2048 / 4096 = 3,453 / 4,786 / 5,766 / 6,223 tok/s. With every expert in VRAM the engine now takes the
largest of 4096 / 2048 / 1024 that fits beside the experts, the 0.77 GiB reserve and the sessions' caches without giving the packed decode
copies back (8K-64K context: 4096, 100K: 2048, 131K: 512; `--batch 2..4`: 1024 at most so the other slots are not held for a whole chunk,
`--batch 8`: 512). The hybrid mode keeps 2048 (a bigger chunk takes expert slots: RTX 3060 chunk 4096 reads prompts 5% faster and decodes 13%
slower) and goes down to 1024 / 512 on a card with 7.5 / 5.5 GiB free. **`--prefill 512` (or `STRATA_Q36_CHUNK=512`) gives the old size, and
then the text is the base engine's, bit for bit.** The chunk size changes the GEMM tiles the BLAS picks and with them the last bits of the
prompt state: KL against the 512 chunk on six real 1.5-3.2K-token documents (48 teacher-forced rows each): chunk 1024 4.0e-3, 2048 4.2e-3,
4096 3.6e-3 (per prompt 2.3e-3 .. 5.9e-3), top-1 agreement 44-48 of 48 rows - the size of the llama.cpp CPU-vs-HIP difference on the same
file (4.2e-3), not a quality loss. Greedy 64 tokens after those six documents: the default chunk gives the base engine's text for 2 of 6,
`--prefill 512` for 6 of 6. A prompt shorter than the chunk (every chat turn after the first) is one chunk either way: unchanged.
The maintainers keep the new default's reference outputs (six documents, greedy 64) and check later changes against them.
Decode tg128 58.5 to 59.7 tok/s (+2%), pp2048 1,958 to 2,022 (+3.3%), 10/10 prompts identical, plain and with `--mtp 2`.
llama.cpp 972b5fc CUDA (`-ncmoe 26 -t 8 -ub 2048`): pp2048 1,328, tg128 48 to 52. The GPU part of a token is the dense
Q8_0 at the card's bandwidth and the rest waits for the CPU pool, so only the small kernels' share is gained.
**Tried and dropped.** The MTP drafts' head over a subset of the vocabulary (`--draft-vocab N`, kept as an opt-in; the answer never
changes): the first 16K-131K rows gain 5-8% on English and code but lose 10-15% on Chinese / Russian / Arabic / Hindi (70% of a Chinese
prompt's token ids are above 65,536), and a candidate list built from the tokens seen so far loses on both. The shared expert on a second
stream (captured into the decode graph) decodes at 36 tok/s instead of 117 with this ROCm. Chunked (WY) GDN and the PCIe-assisted hybrid decode
were not attempted.
## Not done / limits
- Eight slots at most (the window holds 8 rows); `--batch-mtp` shares those rows between slots and proposals.
- A sampling request in a batch decodes one token per window without proposals. `logit_bias` is not available on this engine path: the
server refuses a request that has it (as it does with `--batch` for Flash-Next).
- A slot that stopped being alone goes back to the solo path only through the server's own hand-over; the MTP block's cache has holes for
the tokens a slot decoded in batch windows (the drafts of a later solo stretch are a little worse; the text is not affected).
- Intel Arc (SYCL) does not run this family.
## Testing
The gates are in `tools/qwen35/` and talk to the engine through its serve protocol (no server needed); run them under the box's GPU lock:
- `q36_solo_gate.py` + `q36_cmp_gate.py`: the 10 parity prompts x 128 greedy tokens through `GEN`, compared with another build's.
- `q36_batch_gate.py --slots N --waves --long 2500`: every slot's tokens equal its solo tokens, slots reused, a long prompt read beside
decoding slots. `q36_proto_edge.py`: `max_new` 1, `BSTOP`, a busy slot, `STOP` during a prompt read, held-state reuse, sampled requests,
the context's end. `q36_interleave_probe.py`: what the decoding slots see while a long prompt is read.
- `q36_vision_cmp.py` (with `q36_mtmd_dump.cpp` built against llama.cpp): the picture comparison above; `q36_vision_proto.py`: pictures through
`GENI` / `BGENI` (solo, in a batch, the conversation cache); `q36_vision_serve.py` and `q36_serve_par.py` / `q36_serve_handoff.py`:
the same through the server.
- `q36_par_bench.py`: the aggregate-throughput A/B against `llama-server -np N`.
KL against the 512 chunk on six real 1.5K to 3.2K-token documents (48 teacher-forced rows each): chunk 1024 4.0e-3,
2048 4.2e-3, 4096 3.6e-3, top-1 agreement 44 to 48 of 48 rows. Greedy 64 tokens after those documents: the default chunk
gives the base engine's text for 2 of 6, `--prefill 512` for 6 of 6.
docs/SECOND_GPU.md
+97 -194
View File
@@ -1,227 +1,130 @@
# CUDA1–3 expert experiment
# CUDA1-3 expert caches and the peer tier (experimental)
CUDA0 keeps the dense layers, KV/state, MTP and its existing expert cache. Up to
three independent expert caches fill CUDA1, CUDA2 and CUDA3 in that order. Each
expert has one owner: CUDA0, one secondary GPU, or the CPU pool. Each secondary
GPU computes its rows while the CPU pool works; the results return through
private pinned host buffers before CUDA0 continues the layer. The GPU launches
can overlap, but each layer waits for all three.
This page is for people who tune the engine by hand and want a second, third or fourth card to hold extra experts. These are
experimental engine settings. For ordinary multi-GPU use in the app or `strata launch`, see [MULTI_GPU.md](MULTI_GPU.md). The
settings go into the run config (`strata-*.json`) as `"args"` entries; they are engine flags, not `strata launch` flags.
The shipped profile ranks 8,000 `(layer, expert)` pairs. CUDA0 takes its usual
`--expert-cache auto` allocation. With only CUDA1, it takes the first remaining
pairs. When CUDA2 or CUDA3 is enabled, the remaining profiled pairs are assigned
round robin to the secondary GPUs. The ranked list is then extended with all
other pairs in expert-then-layer order and distributed in the same way. This
fills the additional VRAM and lets all secondary GPUs see frequent experts, but
unranked experts may hardly ever be routed. The startup
log reports slots and GiB per card; each request reports the number of expert
entries actually computed on each secondary GPU. Remote outputs are packed
before returning over PCIe/USB4: the request log compares MiB transferred
with the previous full-row transfer. The input and routing metadata use pinned
host staging. This reduces transfer traffic and launch overhead, but does not
release the expert arena in system RAM or significantly change VRAM use.
> **On this page:** [How it works](#how-it-works) · [Helper caches](#helper-caches) · [Faster helper decode](#faster-helper-decode---remote-expert-opt) ·
> [Peer tier](#peer-tier---peer-device) · [Resident RAM mode](#the-resident-ram-mode-with-the-helper-caches) ·
> [Choosing the primary device](#choosing-the-primary-device-on-windows-strata_primary_device) · [History](#history--older-measurements)
The optional `--gpu-placement layer` gives each layer to exactly one secondary
GPU (layer number modulo the number of enabled cards). It fills that card with
the most frequently routed experts of its layers first, then the unranked ones.
This can reduce USB4 device switches and synchronization when several secondary
cards serve the same layer in the default `stripe` mode. All three cards still
keep their requested expert slots; the primary CUDA0 cache and CPU fallback
continue to work. The log counts active layer launches on each card, so the
tradeoff can be measured alongside tokens per second. This placement is
experimental and may be slower if a given layer needs more than one card's
compute capacity.
## How it works
The experiment requires the corresponding visible CUDA devices and a CUDA
enabled build. Each card must keep at least 512 MiB free. Tiers must be enabled
in order. Under WDDM (Windows, WSL2), CUDA registration of the host expert arena is capped at
8 GiB to leave room for the contexts and MTP on CUDA0 (`STRATA_ARENA_PIN_GIB` overrides it; on Linux the whole
arena is registered). The rest remains
available to the CPU pool; the PCIe expert path is available only for the
registered layers. Without secondary GPUs, the original uncapped registration
behavior applies.
CUDA0 keeps the dense layers, the KV and state, the draft layer (MTP) and its own expert cache. Up to three independent expert
caches fill CUDA1, CUDA2 and CUDA3 in that order. Each expert has one owner: CUDA0, one helper card, or the CPU pool. Each
helper computes its rows while the CPU pool works, and the results return through private pinned host buffers before CUDA0
continues the layer. Each layer waits for all helpers.
On an existing Linux installation, apply the patch to its source tree, then run:
- No peer-to-peer access is needed. Prompt prefill still uses CUDA0; helpers serve decode, including draft verification.
- Each helper card must keep at least 512 MiB free. Enable the tiers in order (CUDA1 before CUDA2).
- The startup log reports slots and GiB per card, and each request reports how many expert entries each helper computed.
Compare identical requests at one, two and four GPUs. VRAM use alone does not show useful offload.
- Longer contexts reserve more KV on CUDA0 and shrink its automatic expert cache.
- Under WDDM (Windows, WSL2) only 8 GiB of the host expert arena is registered with CUDA when helpers are on
(`STRATA_ARENA_PIN_GIB` overrides it). On Linux the whole arena is registered.
- The CUDA0 expert-cache GPU hit path gives outputs that diverge from cache-off runs. Treat performance as experimental until the
generated tokens are validated.
```sh
./setup.sh --setup --gpu1-experts 5000 --gpu2-experts 5000 --gpu3-experts 5000
```
## Helper caches
On Windows, from PowerShell in the installation folder:
| Engine flag | Meaning |
| --- | --- |
| `--expert-cache-device1 N\|auto` | Fill CUDA1 with N experts, or as many as fit. |
| `--expert-cache-device2 N\|auto` | Fill CUDA2 with more experts. |
| `--expert-cache-device3 N\|auto` | Fill CUDA3 with more experts. |
| `--expert-cache-remote-placement stripe\|layer` | `stripe` (default) spreads the ranked expert list over the helpers. `layer` gives each layer to one helper (layer number modulo the number of cards), which can reduce device switches on USB4 but may be slower. |
```powershell
.\START-HERE.bat --setup --gpu1-experts 5000 --gpu2-experts 5000 --gpu3-experts 5000
```
The shipped profile ranks 8,000 `(layer, expert)` pairs. CUDA0 takes its usual `--expert-cache auto` share. With only CUDA1 it
takes the first remaining pairs. With CUDA2 or CUDA3 the remaining ranked pairs go round robin to the helpers, then all other
pairs in expert-then-layer order, so the extra VRAM fills even though unranked experts may rarely be routed.
To test the layer placement instead, add `--gpu-placement layer` to the setup
command. To go back, run the same command with `--gpu-placement stripe`. After
the new engine is built, the mode can also be changed without a rebuild: edit
`--expert-cache-remote-placement` in `strata-iq3_xxs.json` from `layer` to
`stripe` or vice versa and restart the server. If the argument is absent, the
default is `stripe`.
`auto` fills each helper from its assigned ranking using actual aligned expert bytes and free VRAM, keeping the 512 MiB allowance.
It sizes capacity at startup, not throughput, and a card's truncated tail is not moved to another card. Explicit numbers still work.
Choose the same model, size, context and image settings as your current
installation when prompted. Setup recompiles the engine, updates its server
configuration, and starts it. Later starts keep those settings. To tune the
slots without recompiling, edit the values after `--expert-cache-device1`,
`--expert-cache-device2` and `--expert-cache-device3` in the stored
`strata-iq3_xxs.json`, then restart with `.\run-iq3_xxs.bat`.
Compare identical requests at one, two and four GPUs, preferably with several
repeats. VRAM use alone does not show useful offload: compare the per-request
CUDA1–3 counts and tokens per second. More GPU contexts and synchronization
may lower the speed. Prompt prefill still uses CUDA0; secondary GPUs serve
decode, including MTP verification. No peer-to-peer access is required.
Longer contexts reserve more KV/state memory on CUDA0, which reduces its
automatic expert cache. On a 64 GB PC `setup.py` limits IQ3_XXS to 128K even
if 262K was selected; the secondary caches still require the host expert arena.
To try 262K on 64 GB, choose Q2_0 or IQ2_XS instead and check that CUDA0 still
has enough free VRAM after loading. Benchmark long prompts separately from
short decode requests.
The existing warning about the CUDA0 expert-cache GPU hit path still applies:
its outputs diverge from cache-off runs. Treat performance as experimental
until the generated tokens have been validated.
## Optional helper decode optimization
`--remote-expert-opt` (`--serve` only) optimizes the CUDA1-3 helper caches
above. The engine's default is off; since 0.1.39b setup adds it to a config on
two or more GPUs (`--gpus`, or "use both" at start). It acts only when a helper
cache is configured; a layer split runs exactly as before. To leave it out:
setup's `--no-remote-expert-opt`, or `"remote_expert_opt": false` in the
model's `strata-*.json` (kept when setup runs again). Measured by the PR's
author: dual RTX 4090 +63% mixed / +132% code decode over the plain helper
path; RTX 5090 + 4090 +28% / +63%. The primary cache avoids admitting experts already held
by a helper, and helpers replace cold experts with frequently routed CPU
misses using their existing same-layer slots. Each helper reduces its expert
outputs to a weighted partial sum on its GPU before returning one vector per
token. Tokens with no CPU expert work skip CPU activation quantization.
For an existing server configuration with CUDA0 as the primary and CUDA1 as a
helper, use these engine arguments alongside the model and profile arguments:
Example engine arguments beside the model and profile arguments:
```text
--expert-cache auto --expert-cache-device1 auto --remote-expert-opt
```
`--expert-cache-device1`, `--expert-cache-device2` and
`--expert-cache-device3` now also accept `auto`: fill each helper from its
assigned ranking using actual aligned expert bytes and free VRAM, retaining
the existing 512 MiB allowance. Explicit numeric budgets still work. This is
startup capacity sizing, not throughput balancing; with several helpers a
card's truncated candidate tail is not redistributed to another card.
To change slot counts without recompiling, edit the values in the stored config and restart.
The optimization uses the existing host scheduling and pinned-host transport,
not P2P or tensor parallelism. Each layer still waits for its participating
helpers. It changes floating-point summation order, so enabled output is not
claimed to be bitwise identical. Only two-card CUDA operation has been measured
by its author; a community machine measured two cards on HIP (2x RX 6900 XT,
PCIe 4.0 x8 each, IQ3_S): there the expert plan's PCIe share had been taking
experts the helper already held, which made the primary card wait 43 ms per
verify window instead of 19 (decode 40 instead of 66 tok/s). #854 keeps a
helper's experts out of that share; before it, `--pcie-frac 0` avoided the
cost ([bench/results/2026-10-04-rdna2-helper-pcie-share](../bench/results/2026-10-04-rdna2-helper-pcie-share/README.md)).
Three/four cards have not been validated. Without the switch, the
existing decode path remains in use. This does not optimize the separate
`--peer-device` path below or change its existing incompatibility with helper
caches.
**Size a helper against the card's real free VRAM.** On Windows `auto` reads the process's WDDM budget. It cannot see the
desktop, the vision encoder (a separate process), a streaming host or other apps. A helper auto-sized on a card that also drives
the display over-commits, and WDDM backs the excess with system RAM. Use explicit counts on a card that shares its VRAM.
## Faster helper decode (`--remote-expert-opt`)
`--remote-expert-opt` (serve mode only) optimizes the helper caches. The primary cache avoids admitting experts a helper already
holds, helpers replace cold experts with frequently routed CPU misses, each helper reduces its expert outputs to a weighted
partial sum on its GPU before returning one vector per token, and tokens with no CPU expert work skip CPU activation
quantization. It does nothing without a helper cache, and a plain layer split runs as before.
- Setup adds it to a config with two or more GPUs and a helper cache. To leave it out, use setup's `--no-remote-expert-opt` or
`"remote_expert_opt": false` in the config (kept when setup runs again).
- It changes the floating-point summation order, so output is not claimed to be bitwise identical.
- It switches `--pipeline-windows` off ([MULTI_GPU.md](MULTI_GPU.md#pipeline-windows)). Keep one of the two.
- Only two-card CUDA operation has been measured by its author. Three and four cards are not validated.
- It does not apply to the `--peer-device` path below.
## Peer tier (`--peer-device`)
`--peer-device N` puts a second adaptive expert cache on CUDA device N. It
takes the ranked pairs CUDA0's cache does not hold, as many as fit. The peer
computes the rows of its own experts, for decode windows and for prompt
chunks; the activations and the results cross NVLink or another P2P path. The
tier adapts while the server runs, like the primary cache. It is an
alternative to the CUDA1-3 caches above, not a third tier beside them.
`--peer-device N` puts a second adaptive expert cache on CUDA device N. It takes the ranked pairs CUDA0's cache does not hold, as
many as fit. The peer computes the rows of its own experts for decode windows and for prompt chunks. Activations and results cross
NVLink or another peer-to-peer path. It is an alternative to the helper caches above, not a third tier beside them.
- `--peer-device N` (default off): enable the tier on CUDA device N (N >= 1).
- `--peer-reserve-mib M` (default 600): leave M MiB free on the peer card; the
cache takes what remains. The prompt-path buffers need this headroom.
Keep a few hundred MiB free on both cards beyond this: the prompt path loads its kernels and cuBLAS on first use
(a start with 5 MiB left on the peer and 485 MiB on the primary stopped at the first prompt with "cuBLAS status 14").
- Whether the two cards can really exchange data is tested once at start (#2666): two RTX A4000 behind a host
bridge reported peer access, enabled it, and `cudaMemcpyPeerAsync` then copied nothing, so the peer computed on
zeros and the first prompt came back NaN. A test copy each way now decides; if it arrives wrong the log says so
and the prompt path uses the host route (the one without P2P: activations and rows through mapped host memory).
`STRATA_PEER_NO_P2P=1` asks for that route by hand.
- `--peer-slots N` (default 0): cap the tier at N experts; 0 = as many as fit.
- `--peer-adapt-swaps N` (default -1): swaps per adaptive round on the peer;
-1 uses the primary's `--adapt-swaps`.
- `--peer-prefill-rows N` (default -1): the share of each prompt chunk's rows
the peer computes; -1 is half of chunk x top-k, 0 keeps prompt rows on the
primary.
| Flag | Default | Meaning |
| --- | --- | --- |
| `--peer-device N` | off | Enable the tier on CUDA device N (N >= 1). |
| `--peer-reserve-mib M` | 600 | VRAM left free on the peer card. Keep a few hundred MiB free on both cards beyond this, because the prompt path loads its kernels and cuBLAS on first use. |
| `--peer-slots N` | 0 | Cap the tier at N experts; 0 means as many as fit. |
| `--peer-adapt-swaps N` | -1 | Swaps per adaptive round on the peer; -1 uses the primary's `--adapt-swaps`. |
| `--peer-prefill-rows N` | -1 | Share of each prompt chunk's rows the peer computes; -1 is half of chunk x top-k, 0 keeps prompt rows on the primary. |
Through the server, list both cards in the config's `"gpu"` (e.g. `[0, 1]`, numbered as nvidia-smi numbers them)
and add `--peer-device 1` to its `"args"`; the number is the card's position in that list (with `"gpu": [2, 0]`,
`--peer-device 1` is nvidia-smi's card 0). Several GPUs in `"gpu"` are otherwise a layer split: the server adds
`--layer-split` only when `--peer-device` is not among the args.
Whether the two cards can really exchange data is tested once at start with a copy each way. If the copy arrives wrong, the log
says so and the prompt path uses the host route (activations and rows through mapped host memory). `STRATA_PEER_NO_P2P=1` asks
for that route by hand.
`--peer-device` requires `--expert-profile` and an enabled expert cache, and
the device must be visible; it refuses otherwise. It also refuses
`--layer-split` (a different second-GPU mode: use one or the other) and
`--expert-cache-device1..3` (the peer tier already caches experts on that card):
Through the server, list both cards in the config's `"gpu"` (for example `[0, 1]`, numbered as `nvidia-smi` numbers them) and add
`--peer-device 1` to `"args"`. The number is the card's position in that list (with `"gpu": [2, 0]`, `--peer-device 1` is
`nvidia-smi`'s card 0). Several GPUs in `"gpu"` are otherwise a layer split; the server adds `--layer-split` only when
`--peer-device` is not among the args.
strata generate: --peer-device cannot be combined with --layer-split (use one or the other)
strata generate: --peer-device cannot be combined with --expert-cache-device1..3 (the peer tier already caches experts there)
`--peer-device` requires `--expert-profile` and an enabled expert cache, and the device must be visible. It refuses
`--layer-split` and `--expert-cache-device1..3`:
Without `--peer-device` the binary is unchanged; its output is byte-identical
to the release. With `--peer-device` and the same expert set split across the
two cards, the generated tokens are byte-identical to the single-GPU run under
the exactness gate.
```text
strata generate: --peer-device cannot be combined with --layer-split (use one or the other)
strata generate: --peer-device cannot be combined with --expert-cache-device1..3 (the peer tier already caches experts there)
```
With a peer the prompt path keeps the MMQ path; the fused int8 prompt path is
not yet combined with the peer's rows. Mapped host buffers gain
`cudaHostAllocPortable` only with a peer, since only then does a second
context write them. The tier size is manual for now (`--peer-reserve-mib`,
`--peer-slots`); automatic sizing on small cards wants the buffer lending of
#216 and is a follow-up.
With a peer, the prompt path keeps the MMQ path (the fused int8 prompt path is not combined with the peer's rows). The tier size
is manual (`--peer-reserve-mib`, `--peer-slots`). With the same expert set split across the two cards, generated tokens are
byte-identical to the single-GPU run under the exactness gate.
## The resident RAM mode with the helper caches
`--resident-experts` (the low-RAM mode) works with the helper caches: the
experts a helper holds are left out of the RAM copy as a split stage's are, and
the helpers' refills read through the source - the RAM copy when it holds the
bytes, the files otherwise. Measured here on an RX 6800 (gfx1030, 16 GB) + RX
6600 (gfx1032, 8 GB), Windows 11, an i5-11400F and 32 GB of RAM (the 6800 alone:
~37.5 decode tok/s, the same worker configuration): the 6600's 3.4-3.8k-expert
cache lifted decode to ~40.6 / 44.2 tok/s on 8K-token requests at 3800 slots,
with the CPU pool's share of the experts down from ~2.9 to ~0.4 entries per
layer-window. The helper's rows cost about 0.3-0.6 ms per layer on the pool's
critical path - usually less than the CPU rows and page faults they replace,
which is why the 6600 (about 2-3x the 6800's cost per row) still pays off as a
cache tier.
**Size a helper against the card's *real* free VRAM.** `--expert-cache-device1
auto` reads `cudaMemGetInfo`, which on Windows reports the *process's* WDDM
budget: it cannot see the desktop, the vision encoder (a separate process), a
streaming host or other apps on that card. A helper auto-sized on a card that
also drives the display or the encoder over-commits, WDDM backs the excess with
system RAM ("shared GPU memory"), and that memory leaves the machine's RAM - on
a 32 GB PC it pushed the resident complement into the pagefile and made the
helper read its own cache over PCIe. An explicit count
(`--expert-cache-device1 3800` here, of ~4100 that fit an empty card) kept the
cache in real VRAM; freeing another app's sticky allocation on the same card
was worth about the same again. Prefer explicit counts on a card that shares
its VRAM with anything else.
`--resident-experts` (the low-RAM mode) works with the helper caches. The experts a helper holds are left out of the RAM copy,
and the helpers' refills read through the source: the RAM copy when it holds the bytes, the model files otherwise.
## Choosing the primary device on Windows (`STRATA_PRIMARY_DEVICE`)
The engine runs the model on the device the runtime numbers 0. Windows' HIP
runtime enumerates the cards in a fixed order that `HIP_VISIBLE_DEVICES` only
filters - it cannot reorder - so on a PC whose faster card comes second (an RX
6800 after an RX 6600, say) a run with both visible could only put the model on
the slow one. `STRATA_PRIMARY_DEVICE=N` picks another visible ordinal as the
primary; unset or 0 is upstream's behavior.
The engine runs the model on the device the runtime numbers 0. Windows' HIP runtime enumerates cards in a fixed order that
`HIP_VISIBLE_DEVICES` only filters, so when the faster card comes second, the model can only land on the slower one.
`STRATA_PRIMARY_DEVICE=N` picks another visible ordinal as primary. Unset or 0 keeps the default.
- the helper caches (`--expert-cache-device1..3`) take the visible devices
other than the primary, in order;
- the peer tier's cross-device checks use the same ordinal;
- a layer split is refused with this set (its later stages must be > 0, and
stage 0 is the primary);
- NVIDIA cards do not need it: `CUDA_VISIBLE_DEVICES` with
`CUDA_DEVICE_ORDER=PCI_BUS_ID` already reorders.
- The helper caches (`--expert-cache-device1..3`) take the visible devices other than the primary, in order.
- The peer tier uses the same ordinal.
- A layer split is refused when this is set.
- NVIDIA cards do not need it: `CUDA_VISIBLE_DEVICES` with `CUDA_DEVICE_ORDER=PCI_BUS_ID` already reorders.
Measured on RX 6800 + RX 6600, Windows 11; the report accompanies this change.
## History / older measurements
Each row is one machine, reported by the author of the change or a community member.
| Setting | Measured on | Result |
| --- | --- | --- |
| `--remote-expert-opt` | dual RTX 4090 (PR author) | +63% mixed, +132% code decode over the plain helper path |
| `--remote-expert-opt` | RTX 5090 + RTX 4090 (PR author) | +28% mixed, +63% code |
| Helper PCIe share on HIP | 2x RX 6900 XT, PCIe 4.0 x8 each, IQ3_S (community) | before the fix, decode 40 tok/s with the primary waiting 43 ms per verify window; after, 66 tok/s and 19 ms. `--pcie-frac 0` avoided the cost ([report](../bench/results/2026-10-04-rdna2-helper-pcie-share/README.md)) |
| Resident RAM mode with a helper | RX 6800 (16 GB) + RX 6600 (8 GB), Windows 11, i5-11400F, 32 GB RAM | RX 6800 alone ~37.5 tok/s; with a 3,800-slot helper ~40.6 / 44.2 tok/s on 8K-token requests; CPU pool share of experts down from ~2.9 to ~0.4 entries per layer-window |
docs/STRIX_HALO.md
+138 -112
View File
@@ -1,26 +1,47 @@
# Strix Halo (Ryzen AI Max, gfx1151)
Strata runs on the Ryzen AI Max "Strix Halo" APUs (Radeon 8060S / 8050S, RDNA3.5, `gfx1151`, one pool of LPDDR5X shared by the
CPU and the GPU) through the same HIP backend as the Radeon cards in [AMD_HIP.md](AMD_HIP.md). **Status: experimental.**
Everything here was built and measured on one machine, the maintainers' Strix Halo box (Ryzen AI Max+ 395, 128 GB,
Ubuntu 26.04, kernel 7.0, ROCm 7.14.1). Setup (`setup.sh`, `START-HERE.bat`) recognizes it from 0.1.40, by its PCI id (`1002:1586`, one id for the 8060S, 8050S and 8040S) and by
`gfx1151` exactly: a Strix Point (890M / 880M, gfx1150), Krackan (860M / 840M, gfx1152) or Phoenix / Hawk Point (780M / 760M, gfx1103) Radeon is
named as what it is and not supported. On Linux setup compiles the engine for gfx1151 itself (the steps below are the same, by hand). The 0.2.0-beta release also has ready-made packages: `Strata-0.2.0-beta-linux-amd-strixhalo.AppImage` (engine `halo`, gfx1151, ROCm 7.14.1) and, on Windows, `Strata-0.2.0-beta-windows-hip-rdna3.exe` (engine `rdna3`: gfx1100, gfx1101, gfx1102 and gfx1151, ROCm 10.2.0a20260930); any app installer plus **Update engine** (or `strata update`) fetches the same package. What was run: both packages were built, linked and scanned, and the Linux package refuses to start on a card of another family (tried on an RX 9000 card); the maintainers have no Strix Halo and ran neither on one. The measurements on this page come from source builds on Linux (the maintainers' earlier box and community machines). Nobody has run a Strix Halo on Windows. The packaged engines do not carry the hipBLASLt table of section 3 yet (planned for 0.2.1), so their prompt reading is slower than the numbers below until then. A Windows zip of the 0.1.x START-HERE route without gfx1151 code is refused with a message, never used. Setup counts the GPU's memory as the BIOS carve-out plus the shared memory
(the GTT pool; on Windows half the RAM) and recommends UD-IQ4_XS from about 80 GB of memory; it changes no BIOS or kernel setting.
This page is for people with a Ryzen AI Max "Strix Halo" PC (Radeon 8060S / 8050S, RDNA 3.5, `gfx1151`, one pool of LPDDR5X
shared by the CPU and the GPU). Strata runs there through the same HIP backend as the Radeon cards in [AMD_HIP.md](AMD_HIP.md).
**Status: experimental.**
**Windows start-up (0.2.1.2).** On a Strix Halo under Windows the engine no longer registers the expert arena with the GPU driver: registering it took all free VRAM (91 GiB before, 0 after, on a 64 GB carve-out + 32 GB shared PC), the expert cache got no slots and the engine stopped with "the embedding buffer failed" (0.2.1 and earlier). The arena stays in RAM, held by the working-set lock, and the cache gets its normal slots. `STRATA_ARENA_PIN_GIB=N` (or `0`) registers it as before, `STRATA_UMA_REGISTER=1` too. Linux is unchanged.
DeepSeek V4.1-Flash does not run on Strix Halo yet. It runs in the app and the Terminal on NVIDIA RTX 30, 40 and 50 cards and on the Radeon AI
PRO R9700 ([MODELS.md](MODELS.md#deepseek-v41-flash)).
What the engine does differently on this chip:
> **On this page:** [Quick start](#quick-start) · [What the engine does differently](#what-the-engine-does-differently) ·
> [1. Toolchain](#1-the-toolchain-no-root-needed) · [2. Engine](#2-the-engine) · [3. Tuning table](#3-the-prompt-table-and-running) ·
> [4. Defaults](#4-what-is-on-by-default-on-gfx1151) · [5. Opt-in](#5-what-is-not-on-by-default-it-changes-bits) ·
> [5b. Desktop and RAM](#5b-the-desktop-the-ram-and-a-discrete-card-beside-the-strix-halo-linux) · [6. Measured](#6-measured) ·
> [History](#history--older-measurements)
- **Unified memory.** The GPU's "VRAM" is the GTT pool of the same RAM. `device_free_bytes()` (the figure `--expert-cache auto` and
the memory plan use) counts what Linux can give back (`MemAvailable` less 6 GiB for the system, `STRATA_UMA_HEADROOM_GIB`
changes the 6) instead of only `MemFree`, per device, for any integrated GPU on Linux (this applies to a DGX Spark too).
## Quick start
Use the [current release](https://github.com/Niko1221/Strata/releases/latest). You do not need to build ROCm for a packaged install.
| System | What to install | Engine variant |
| --- | --- | --- |
| Linux | `Strata-0.2.5-linux-x86_64.AppImage` (it downloads the engine on first start), or `strata launch` | `linux-x86_64-hip-halo` |
| Windows | `Strata-0.2.5-windows-x64-setup.exe` (it downloads the engine on first start), or `strata launch` | `windows-x86_64-hip-rdna3` |
Packaged engines carry the matching hipBLASLt tuning table for gfx1151 and select it on their own. Setup recognizes `gfx1151` and
PCI ID `1002:1586`; other integrated Radeon chips have separate support limits. Unified memory is counted once. Setup does not
change BIOS or kernel settings. The manual Linux toolchain below is for reproduction and builds, not a prerequisite for the app.
## What the engine does differently
- **Unified memory.** The GPU's "VRAM" is the GTT pool of the same RAM. `device_free_bytes()` (the figure `--expert-cache auto`
and the memory plan use) counts what Linux can give back (`MemAvailable` less 6 GiB for the system; `STRATA_UMA_HEADROOM_GIB`
changes the 6) for any integrated GPU on Linux, a DGX Spark included.
- **gfx11 matrix cores.** The prompt attention, the block scorer, the prompt GEMMs and the prompt experts have WMMA kernels for
gfx1100 / 1101 / 1102 / 1103 / 1150 / 1151. A gfx11 part outside that list (gfx1152) is not given them: it takes the portable path.
- **The hipBLASLt table** `tools/hip/gfx1151-hipblaslt-100401.txt` (ROCm 7.14.1's hipBLASLt 1.x `100401`), including the prompt shapes
at `--prefill 16384` (T 16383 / 16384). Setup's own ROCm for gfx1151 (the 7.14.0a20260608 wheels, hipBLASLt 1.4.0) takes
`tools/hip/gfx1151-hipblaslt-100400.txt`: the same shapes, calibrated against that library (docs/AMD_HIP.md, Tuning table).
- **The expert cache and the RAM budget share the RAM.** For a model with a RAM budget (UD-IQ4_XS, UD-Q4_K_XL) setup writes a number for `--expert-cache` (the RAM less 24 GB less the budget), not `auto`: `auto` counts all the memory the OS can give back at start, and with a 55 GiB budget and a desktop open it took the room the OS needed (#1715, a global OOM on a 128 GB box). The number is in the config's `args`; `--expert-cache N` there sets it. An engine run by hand should pass one too.
gfx1100 / 1101 / 1102 / 1103 / 1150 / 1151. A gfx11 part outside that list (gfx1152) takes the portable path.
- **The hipBLASLt table.** `tools/hip/gfx1151-hipblaslt-100401.txt` is for ROCm 7.14.1's hipBLASLt (`100401`), including the prompt
shapes at `--prefill 16384`. Setup's own ROCm for gfx1151 (the 7.14.0a20260608 wheels, hipBLASLt 1.4.0) takes
`tools/hip/gfx1151-hipblaslt-100400.txt`, calibrated against that library.
- **The expert cache and the RAM budget share the RAM.** For a model with a RAM budget (UD-IQ4_XS, UD-Q4_K_XL) setup writes a number
for `--expert-cache` (the RAM less 24 GB less the budget), not `auto`, because `auto` counts all memory the OS can give back at
start and can take the room the OS needs. The number is in the config's `args`. An engine run by hand should pass one too.
- **Windows.** The engine does not register the expert arena with the GPU driver. The arena stays in RAM, held by the
working-set lock, and the expert cache gets its normal slots. `STRATA_ARENA_PIN_GIB=N` (or `0`) or `STRATA_UMA_REGISTER=1`
registers it. Linux registers the whole arena by default.
## 1. The toolchain (no root needed)
@@ -37,18 +58,17 @@ export PATH=$SDK/bin:$SDK/lib/llvm/bin:$PATH
export LD_LIBRARY_PATH=$SDK/lib:$SDK/lib/rocm_sysdeps/lib:$SDK/lib/llvm/lib:${LD_LIBRARY_PATH:-}
```
You also need CMake 3.24+, Ninja, git and a C++ host toolchain with libstdc++ headers (`sudo apt install build-essential cmake ninja-build git`).
The maintainers' box has no system compiler: they point ROCm's clang at a user-space copy of gcc 15 with
`--gcc-toolchain=<dir>/usr` (added to `CMAKE_HIP_FLAGS` below; leave it out when the system has one).
You also need CMake 3.24+, Ninja, git and a C++ host toolchain with libstdc++ headers
(`sudo apt install build-essential cmake ninja-build git`). If the system has no compiler, point ROCm's clang at a user-space gcc
with `--gcc-toolchain=<dir>/usr` in `CMAKE_HIP_FLAGS`.
The kernel's amdgpu driver and `/dev/kfd` access (your user in the `render` and `video` groups) are all the GPU side needs.
To let the GPU use most of a 128 GB machine the maintainers' box boots with `ttm.pages_limit=29360128 ttm.page_pool_size=29360128`
(a 112 GiB GTT; the default is about half of the RAM) and `amd_iommu=off`; both are host settings the engine does not need to run,
only to have the room.
The GPU side needs the kernel's amdgpu driver and `/dev/kfd` access (your user in the `render` and `video` groups). To let the GPU
use most of a 128 GB machine, the maintainers' box boots with `ttm.pages_limit=29360128 ttm.page_pool_size=29360128` (a 112 GiB GTT;
the default is about half of the RAM) and `amd_iommu=off`. Both are host settings the engine does not need to run, only to have the room.
## 2. The engine
llama.cpp's `ggml` sources are pinned by this repository: CMake fetches the revision it needs. For an offline build, point
llama.cpp's `ggml` sources are pinned by this repository, and CMake fetches the revision it needs. For an offline build, point
`STRATA_GGML_DIR` at a checkout of exactly that revision.
```sh
@@ -62,50 +82,52 @@ cmake -S . -B build-halo -G Ninja -DCMAKE_BUILD_TYPE=Release \
cmake --build build-halo --target strata -j 8 # about 25 minutes the first time
```
`gfx1151` is "unvalidated" in the build's own wording (it warns). `STRATA_PREFILL_MMQ=ON` builds ggml's MMQ kernels the prompt
experts need. The tests: `cmake --build build-halo -j 8` builds them all, `ctest --test-dir build-halo` runs them;
`hip_prefill_hcd_exact_parity` and `hip_prefill_hipblaslt_gemm` need the table (next section) and skip without it.
The build warns that `gfx1151` is "unvalidated". `STRATA_PREFILL_MMQ=ON` builds ggml's MMQ kernels the prompt experts need. To test:
`cmake --build build-halo -j 8` builds the tests and `ctest --test-dir build-halo` runs them. `hip_prefill_hcd_exact_parity` and
`hip_prefill_hipblaslt_gemm` need the table (next section) and skip without it.
## 3. The prompt table, and running
Hand the engine the table (`setup` does it for the supported cards; for gfx1151 you do it): without it the prompt's dense matrix
products use plain hipBLAS and are slower.
A packaged engine finds its table itself. For a hand-built engine, hand it the table, because without it the prompt's dense matrix
products use plain hipBLAS and are slower:
```sh
export STRATA_HIPBLASLT_TUNING=$PWD/tools/hip/gfx1151-hipblaslt-100401.txt
```
The engine accepts it only for the architecture and the hipBLASLt version named on the table's second line
The engine accepts a table only for the architecture and hipBLASLt version named on its second line
(`STRATA_HIPBLASLT_TUNING_V1 gfx1151 100401`: ROCm 7.14.1) and says so at start-up (`prefill gemm: hipBLASLt tuning enabled (90 rows,
gfx1151, version 100401)`). Then run `strata` with the same arguments as on any other card (the model packs, `--serve`, the
settings of [DETAILS.md](DETAILS.md)); a pack of the Unsloth UD-IQ4_XS or UD-Q4_K_XL GGUFs is what the numbers below used.
gfx1151, version 100401)`). Then run `strata` with the same arguments as on any other card (the model packs, `--serve`, the settings
of [DETAILS.md](DETAILS.md)). The numbers below used packs of the Unsloth UD-IQ4_XS or UD-Q4_K_XL GGUFs.
## 4. What is on by default on gfx1151
At start the engine looks at GPU 0; on a gfx1151 it sets the switches below unless you set them yourself
(`STRATA_GFX1151_DEFAULTS=0` turns the whole table off, and the start-up line names what was set). **Every one of them gave
byte-identical output on the maintainers' box** (the same token ids, 4K / 32K / 64K and four models, and a bitwise harness per kernel):
they change speed, not answers. Nothing here applies to CUDA or to another AMD architecture.
At start the engine looks at GPU 0. On a gfx1151 it sets the switches below unless you set them yourself.
`STRATA_GFX1151_DEFAULTS=0` turns the whole table off, and the start-up line names what was set. Every one of them gave
byte-identical output on the maintainers' box (same token ids at 4K / 32K / 64K and on four models, plus a bitwise harness per
kernel): they change speed, not answers. Nothing here applies to CUDA or to another AMD architecture.
| Switch | What it does |
|---|---|
| `STRATA_GDN_HEAD`, `STRATA_GDN_PP=2`, `STRATA_GDN_CONVL2`, `STRATA_GDN_NOY` | the prompt's gated-delta-net recurrence: four lanes per column, two columns per lane, the convolution and the norms fused |
| `STRATA_HCD_EXACT` | the hyper-connection down projection (N 320, K 10240) in a WMMA kernel that reproduces hipBLASLt solution 1176 / 1177's summation order (`hip_prefill_hcd_exact_parity` checks it bit for bit); it runs only where the table makes hipBLASLt pick one of those two, and says once if it cannot |
| `STRATA_CVEC_FUSE`, `STRATA_PF_PAD` | a steered layer's write, control vector and next norm in one pass; padded GEMM row strides |
| `STRATA_TOPK_REG_MIN_BLOCKS=4096` | the prompt's QSA top-k uses the register kernel from 16K cells of context instead of 28K (the original kernel is slower there; the selected cells are identical, `qsa_topk_parity` checks it) |
| `STRATA_Q8_PACKED`, `STRATA_Q6_PACKED`, `STRATA_MMVF_ROWS`, `STRATA_ATTN_LANECELL` | decode layouts and kernels (packed Q8_0 / Q6_K weights, 4-row BF16 GEMV, one-cell-per-thread attention scores) |
| `STRATA_EXPERT_V2`, `STRATA_TSUM`, `STRATA_LFUSE` | the grouped decode experts (IQ3_S / IQ4_NL), single-butterfly warp sums, fewer launches around the shared expert and the KV append |
| `STRATA_GDN_SPLIT`, `STRATA_QFUSE`, `STRATA_PLE_BATCH` | the GDN step over four blocks per head, activation images written by their producers, the PLE key / value projections of a verify window at once |
| `STRATA_SH_STREAM=1` | the shared expert on a second stream beside the routed experts. It is off by default on other AMD cards (#816), but it pays on gfx1151: decode +1.8% (UD-IQ4_XS) and +6.7% (UD-Q4_K_XL) at 8K |
| --- | --- |
| `STRATA_GDN_HEAD`, `STRATA_GDN_PP=2`, `STRATA_GDN_CONVL2`, `STRATA_GDN_NOY` | The prompt's gated-delta-net recurrence: four lanes per column, two columns per lane, convolution and norms fused. |
| `STRATA_HCD_EXACT` | The hyper-connection down projection (N 320, K 10240) in a WMMA kernel that reproduces hipBLASLt solution 1176 / 1177's summation order (`hip_prefill_hcd_exact_parity` checks it bit for bit). It runs only where the table makes hipBLASLt pick one of those two, and says once if it cannot. |
| `STRATA_CVEC_FUSE`, `STRATA_PF_PAD` | A steered layer's write, control vector and next norm in one pass; padded GEMM row strides. |
| `STRATA_TOPK_REG_MIN_BLOCKS=4096` | The prompt's QSA top-k uses the register kernel from 16K cells of context instead of 28K. The selected cells are identical (`qsa_topk_parity`). |
| `STRATA_Q8_PACKED`, `STRATA_Q6_PACKED`, `STRATA_MMVF_ROWS`, `STRATA_ATTN_LANECELL` | Decode layouts and kernels: packed Q8_0 / Q6_K weights, 4-row BF16 GEMV, one-cell-per-thread attention scores. |
| `STRATA_EXPERT_V2`, `STRATA_TSUM`, `STRATA_LFUSE` | The grouped decode experts (IQ3_S / IQ4_NL), single-butterfly warp sums, fewer launches around the shared expert and the KV append. |
| `STRATA_GDN_SPLIT`, `STRATA_QFUSE`, `STRATA_PLE_BATCH` | The GDN step over four blocks per head, activation images written by their producers, the PLE key / value projections of a verify window at once. |
| `STRATA_SH_STREAM=1` | The shared expert on a second stream beside the routed experts. Off by default on other AMD cards, but it pays on gfx1151 (decode +1.8% UD-IQ4_XS, +6.7% UD-Q4_K_XL at 8K). |
Two further defaults change bits. `STRATA_HC_Q8` reads the hyper-connection weights as 8-bit (`STRATA_HC_Q8=0` restores the BF16
read), and on Windows the prompt is read in chunks of 8192 tokens (`STRATA_GFX1151_CHUNK=0` keeps 6144). About `STRATA_HC_Q8`: it
halves the hyper-connection weight bytes (13.29 -> 7.15 MB per read) and gave +8.4% decode on `json_extract`, at the cost of answers
that are not bit-identical to the BF16 read. How far they move: [AMD_HIP.md](AMD_HIP.md) ("Strix Halo: 8-bit hyper-connection weights").
## 5. What is not on by default (it changes bits)
**Five of the switches below (`STRATA_PF_GEMM`, `STRATA_HC_UPMIX`, `STRATA_PA_FAST`, `STRATA_HIP_WMMA`, `STRATA_SELECT_WMMA`) are one opt-in on an integrated Radeon: `STRATA_IGPU_PROMPT_SET=1` ([AMD_HIP.md](AMD_HIP.md), "Integrated GPUs"). It was measured on a Radeon 780M only, where it is the default.** `STRATA_PF_FUSED`, `STRATA_PF_SWITCH_MIN_T` and `STRATA_PREFILL_STREAM_MIN` stay opt-in as well. `STRATA_HC_Q8` is on by default on gfx1151 (below); `STRATA_HC_Q8=0` turns it off.
These are faster on gfx1151 and **round differently** from the default path (a different summation order or a half-precision
intermediate), so they are opt-in and the maintainers gate them with KL against the default (`STRATA_PF_SWITCH_MIN_T=4096` keeps
the short prompts on the default numerics, since these pay on long ones). The maintainers' fast configuration sets all of
them:
These are faster on gfx1151 and round differently from the default path (a different summation order or a half-precision
intermediate), so they are opt-in. The maintainers gate them with KL against the default. `STRATA_PF_SWITCH_MIN_T=4096` keeps
short prompts on the default numerics, since these pay on long ones. The maintainers' fast configuration sets all of them:
```sh
export STRATA_PF_FUSED=1 # the prompt experts on the matrix cores (IQ2 / IQ3 / IQ4_XS gate/up, Q2_0 / IQ4_NL / Q8_0 down)
@@ -115,64 +137,64 @@ export STRATA_PA_FAST=1 # the prompt attention with single FP16 q and p
export STRATA_HIP_WMMA=1 # the prompt attention on matrix cores
export STRATA_SELECT_WMMA=1 # the block scorer on matrix cores
export STRATA_PF_SWITCH_MIN_T=4096
export STRATA_PREFILL_STREAM_MIN=128 # prompt reads of 128-1,023 tokens on the fused experts too (below)
export STRATA_PREFILL_STREAM_MIN=128 # prompt reads of 128-1,023 tokens on the fused experts too
```
`STRATA_HC_Q8` (on by default on gfx1151 since 0.2.2; `STRATA_HC_Q8=0` restores the bf16 read) measured on the Strix Halo (IQ3_S, gfx1151, 2026-10-21, MTP 4 drafts / min-p 0.5): **+8.4%**
on `json_extract` at fixed windows (53.55 -> 58.03 tok/s), at the cost of **94 of 250 output tokens differing** from
the BF16 read (first at 60). It halves the hyper-connection weight bytes (13.29 -> 7.15 MB per read; cold read
93.8 -> 20.3 us, 142 -> 352 GB/s). The answers are not bit-identical to the bf16 read; how far they move is in docs/AMD_HIP.md ("Strix Halo: 8-bit hyper-connection weights").
`STRATA_PF_GEMM`, `STRATA_HC_UPMIX`, `STRATA_PA_FAST`, `STRATA_HIP_WMMA` and `STRATA_SELECT_WMMA` together are one opt-in on an
integrated Radeon: `STRATA_IGPU_PROMPT_SET=1` ([AMD_HIP.md](AMD_HIP.md), "Integrated GPUs"), measured on a Radeon 780M only, where
it is the default. `STRATA_PF_FUSED`, `STRATA_PF_SWITCH_MIN_T` and `STRATA_PREFILL_STREAM_MIN` stay opt-in as well.
`STRATA_PF_FUSED=1` reaches only chunks of `STRATA_PREFILL_STREAM_MIN` tokens or more (1,024 by default, the floor measured on
a discrete card, where a smaller chunk does not pay for streaming every expert); a smaller read runs its experts through MMQ.
With every expert in the unified memory nothing streams, and MMQ is the slow part: an agent's turn - a tool result of ~1,000
tokens on a cached conversation - read in 3.1-6.0 s at 1,024 and 0.9-2.5 s at 128 (C2T8, `--no-prefill-borrow`; 12-turn
replay 3.4 -> 2.1 s per turn). Its output is the fused path's, which reads of 1,024 tokens or more already take; it is not
under `STRATA_PF_SWITCH_MIN_T`, which gates only the hyper-connection and padding switches. A UD-Q4_K_XL or other K-quant
pack takes the fused experts only with `STRATA_PF_FUSED_KQ=1` as well.
`STRATA_PF_F16=1` (gfx11 only; add it to the list above) runs the covered prompt experts on the F16 matrix cores instead of
the int8 ones: K-quant gate/up and Q5_1 down (with `STRATA_PF_FUSED_KQ=1`), IQ2 / IQ3 / IQ4_XS gate/up, Q2_0 / IQ4_NL / Q8_0
down. One layer, 16,384 tokens: Q4_K / Q5_1 85.3 -> 53.5 ms, IQ3_S / IQ4_NL gate/up 11.65 -> 9.87 and down 6.11 -> 4.71 ms.
It rounds differently from the int8 kernels. First-token KL against a run with no switches, 12 natural-text prompts of
1.5K-24K tokens (Strata's own docs and sources), top-1 the same in 12 of 12: UD-Q4_K_XL mean 0.0068 (median 0.0021) against
0.0031 (0.0024) for `STRATA_PF_FUSED=1 STRATA_PF_FUSED_KQ=1` without it; UD-IQ4_XS 0.0085 (0.0036) against 0.0068 (0.0032). One
1.5K prompt carries most of the Q4_K_XL mean (0.049).
`STRATA_REFILL_PARTS=1` (opt-in, exact; HIP and CUDA, a GGUF read in place) refills the expert-cache slots a prompt borrowed straight from the GGUF mapping (an expert's gate, up and down tensors) instead of through an assembled 4 MB copy per expert: the same bytes in the slot (checked against the assembled blob on 162 slots), 0.1 to 0.3 s less at the end of a prompt; gfx1151, UD-Q4_K_XL and UD-IQ4_XS: a 4K-8K prompt +2.3% to +3.5%, 32K +1.1%. It is not in the table above because only gfx1151 has measured it.
Two engine flags help on long contexts and are not Strix-specific: `--mtp-window 8192` (the draft layer attends to the last 8,192
cells: 64K output +3.5%, 32K -0.4%) and `--spec`, `--lookup-chain`, `--mtp-q4` (see [DETAILS.md](DETAILS.md)).
| Switch | What it does |
| --- | --- |
| `STRATA_PREFILL_STREAM_MIN=N` | `STRATA_PF_FUSED=1` reaches only chunks of N tokens or more (1,024 by default, the floor measured on a discrete card). With every expert in unified memory nothing streams and MMQ is the slow part, so a smaller N (128) speeds up an agent's turn (a tool result of about 1,000 tokens on a cached conversation: 3.1 to 6.0 s at 1,024 against 0.9 to 2.5 s at 128). |
| `STRATA_PF_FUSED_KQ=1` | A UD-Q4_K_XL or other K-quant pack takes the fused experts only with this as well. |
| `STRATA_PF_F16=1` (gfx11 only) | Runs the covered prompt experts on the F16 matrix cores instead of the int8 ones. It rounds differently from the int8 kernels. |
| `STRATA_REFILL_PARTS=1` (exact; HIP and CUDA, a GGUF read in place) | Refills the expert-cache slots a prompt borrowed straight from the GGUF mapping instead of through an assembled 4 MB copy per expert. Only gfx1151 has measured it. |
| `--mtp-window 8192` | Not Strix-specific: the draft layer attends to the last 8,192 cells. |
| `--spec`, `--lookup-chain`, `--mtp-q4` | See [DETAILS.md](DETAILS.md). |
## 5b. The desktop, the RAM, and a discrete card beside the Strix Halo (Linux)
The Strix Halo chip usually draws your desktop, and its graphics memory is your RAM. Two things follow, and the engine now takes both into account.
The Strix Halo chip usually draws your desktop, and its graphics memory is your RAM. The engine takes both into account.
- **One memory, counted once.** The expert cache of an integrated GPU lives in the same RAM as the expert arena. The automatic cache size is the smaller of
what the GPU reports free and what the OS reports available, less a reserve for the desktop (the larger of 8 GiB and 8% of the RAM; the start-up line says so
when it changes the size). `STRATA_UMA_RESERVE_GIB=N` sets the reserve (`0` = the old sizing, nothing kept back); `--expert-cache N` is never changed.
- **A card of its own beside the chip** (for example a Radeon AI PRO R9700). A card of its own is the recommended Main card and the chip stays Off in the card roles.
If you name the chip in a layer split anyway (`--gpus 0,1`, or the config's `"gpu": [0, 1]`), the split runs as asked, the chip is placed on the first layers and
never on the last stage (the output head, the draft layer and the check of its guesses), the picture reader is started on the card of its own, and the
start prints one plain line about it. `"gpu_order": "as_given"` and a manual `layer_split` keep your order. `setup --gpus all` leaves the chip out
when two cards of their own are present; name it by number to use it.
- **The CPU.** On Linux with built-in Radeon graphics the engine starts at nice +5, so the desktop keeps its share of the CPU while the engine's workers spin.
- **One memory, counted once.** The automatic expert cache size is the smaller of what the GPU reports free and what the OS reports
available, less a reserve for the desktop (the larger of 8 GiB and 8% of the RAM; the start-up line says so when it changes the
size). `STRATA_UMA_RESERVE_GIB=N` sets the reserve (`0` = nothing kept back). `--expert-cache N` is never changed.
- **A card of its own beside the chip** (for example a Radeon AI PRO R9700). That card is the recommended Main card and the chip
stays Off in the card roles. If you name the chip in a layer split anyway (`--gpus 0,1`, or the config's `"gpu": [0, 1]`), the
split runs as asked, the chip takes the first layers and never the last stage, the picture reader starts on the card of its own,
and the start prints one plain line about it. `"gpu_order": "as_given"` and a manual `layer_split` keep your order.
`setup --gpus all` leaves the chip out when two cards of their own are present; name it by number to use it.
- **The CPU.** On Linux with built-in Radeon graphics the engine starts at nice +5, so the desktop keeps its share of the CPU.
`STRATA_ENGINE_NICE=0` turns it off, `STRATA_ENGINE_NICE=N` sets another value.
What is not known yet: how much of the screen freeze a long compute job on the same chip causes, and whether registering the whole expert arena with the driver
(the Linux default on gfx1151; Windows uses the unregistered arena since 0.2.1.2) hurts when the GTT is small. `STRATA_ARENA_PIN_GIB=none` runs the unregistered
arena on any OS; reports with `amdgpu_top` and `dmesg` are welcome.
Not known yet: how much of the screen freeze a long compute job on the same chip causes, and whether registering the whole expert
arena with the driver (the Linux default on gfx1151) hurts when the GTT is small. `STRATA_ARENA_PIN_GIB=none` runs the unregistered
arena on any OS. Reports with `amdgpu_top` and `dmesg` are welcome.
## 6. Measured
Ryzen AI Max+ 395 (128 GB, gfx1151), the maintainers' fast configuration of section 5 on top of the defaults of section 4, `--spec 4`,
`--mtp`, `--lookup-chain 3`, `--mtp-q4 all`, `--prefill 16384`, int8 KV, all experts in the unified memory. Prompt tok/s is
(prompt tokens - 1) / prompt time, output tok/s is the tokens / decode time; medians of 3 runs, one fresh process per run, the two
engines interleaved. "Before" is the maintainers' earlier engine branch (the same switches set by hand), "now" is this release.
Every pair printed the same token ids.
**Measured on:** a Strix Halo PC (gfx1151, Windows), IQ3_S model, Strata 0.2.3 (engine 0.1.48), 3 pairs against 0.2.2, from
[release-notes-0.2.3.md](release-notes-0.2.3.md). Answer speed +6% to +19% (tool calls +19%, code +18%, 64K prompt +13%, 8K prompt
+12%, 4K prompt +8%, JSON +6%, agent chat +9%). Prompt speed +5% to +14% (8K +14%, 64K +5%, 4K unchanged because it is one chunk).
Prompt speed at 4K, 8K and 64K is 685, 783 and 749 tokens per second. This is specific evidence, not validation of every APU or
quantization.
The Linux numbers from the maintainers' Ryzen AI Max+ 395 (128 GB, Ubuntu 26.04, kernel 7.0, ROCm 7.14.1) are in the history
section below.
## History / older measurements
<a id="measured"></a>
**Measured on:** Ryzen AI Max+ 395, 128 GB, gfx1151, Linux, Strata 0.1.40 code, the maintainers' fast configuration of section 5 on
top of the defaults of section 4, `--spec 4`, `--mtp`, `--lookup-chain 3`, `--mtp-q4 all`, `--prefill 16384`, int8 KV, all experts
in unified memory. Prompt tok/s is (prompt tokens - 1) / prompt time. Medians of 3 runs, one fresh process per run, two engines
interleaved, every pair printed the same token ids. "Before" is the maintainers' earlier engine branch with the same switches set
by hand.
| Model | Context | Prompt before -> now | Output before -> now |
|---|---|---|---|
| --- | --- | --- | --- |
| UD-IQ4_XS | 8K | 1,212 -> 1,293 (+6.7%) | 51.7 -> 53.8 (+4.0%) |
| UD-IQ4_XS | 128K | 1,241 -> 1,320 (+6.4%) | 49.3 -> 51.4 (+4.2%) |
| UD-Q4_K_XL | 8K | 1,136 -> 1,169 (+2.9%) | 50.6 -> 52.6 (+3.9%) |
@@ -183,29 +205,33 @@ Every pair printed the same token ids.
| IQ3_XXS | 128K | 1,259 -> 1,252 (-0.6%) | 48.4 -> 50.9 (+5.1%) |
UD-IQ4_XS at 4K / 32K / 64K (5-6 interleaved pairs): prompt +9.2% / +6.0% / +5.3%, output +5.0% / +3.9% / +4.6%.
IQ3_XXS at 8K was measured again with 6 interleaved pairs (an earlier 3-run set had read 5% low while other jobs shared the GPU): prompt equal, output +5.6%. With the defaults off the same build's prompt is 8% lower (1,137), so the table helps there too.
### Re-measured on the merged 0.1.40 code
The same box and method, no environment switches at all (the arch defaults only), on the final merged tree. 4 interleaved pairs per cell,
medians; "j" is the build of the table above (the Aurora port before the other 0.1.40 work was merged), "merged" is the release
code. Every run printed the same token ids, and the ids also match `strata-int29` (the v4.2 config) on UD-IQ4_XS and UD-Q4_K_XL at
a8 / 4K / 64K.
**Same box, merged 0.1.40 code, no environment switches (arch defaults only).** 4 interleaved pairs per cell, medians. "j" is the
build of the table above, "merged" is the 0.1.40 release code. Every run printed the same token ids. Prompt numbers at 8K swing by
more than the differences because other jobs shared the box, so read them as unchanged.
| Model | Context | Prompt j -> merged | Output j -> merged |
|---|---|---|---|
| UD-IQ4_XS | 8K | 1,260 -> 1,159 (-8.0%, spread 1,070-1,320) | 53.94 -> 53.14 (-1.5%) |
| --- | --- | --- | --- |
| UD-IQ4_XS | 8K | 1,260 -> 1,159 | 53.94 -> 53.14 (-1.5%) |
| UD-IQ4_XS | 128K | 1,317 -> 1,299 (-1.4%) | 51.38 -> 50.72 (-1.3%) |
| UD-Q4_K_XL | 8K | 1,163 -> 1,104 (-5.0%, spread 890-1,240) | 52.66 -> 50.47 (-4.2%) |
| UD-Q4_K_XL | 8K | 1,163 -> 1,104 | 52.66 -> 50.47 (-4.2%) |
| UD-Q4_K_XL | 128K | 1,252 -> 1,211 (-3.3%) | 43.28 -> 42.64 (-1.5%) |
| IQ3_S | 8K | 1,283 -> 1,265 (-1.4%) | 59.81 -> 59.10 (-1.2%) |
| IQ3_S | 128K | 1,311 -> 1,310 (-0.1%) | 42.74 -> 42.09 (-1.5%) |
| IQ3_XXS | 8K | 1,127 -> 1,212 (+7.6%) | 51.64 -> 50.95 (-1.3%) |
| IQ3_XXS | 128K | 1,252 -> 1,250 (-0.1%) | 50.88 -> 50.22 (-1.3%) |
The prompt numbers at 8K swing by more than the differences (other jobs share this box), so read them as unchanged. The output
column moved by a steady -1.2% to -1.5% (-4% on UD-Q4_K_XL at 8K, where the default arm also had slow runs): since 0.1.39 the
shared-expert stream fork is off by default on HIP (#816). It overlaps the shared expert with the routed experts on a second
stream and does help on gfx1151, so 0.1.40 turns it on there by default (section 4). It changes no bits (same ids). 4 interleaved pairs at 8K, merged
code with the fork off -> on: UD-IQ4_XS output 53.1 -> 54.05 (+1.8%), UD-Q4_K_XL 49.5 -> 52.8 (+6.7%, the default arm ranged 46.0-52.1),
prompt unchanged. With the fork on, the merged code's output is at or above the j build's (IQ4_XS 54.05 vs 53.94).
With the shared-expert stream (`STRATA_SH_STREAM=1`, now a default) at 8K, 4 pairs: UD-IQ4_XS output 53.1 -> 54.05 (+1.8%),
UD-Q4_K_XL 49.5 -> 52.8 (+6.7%), prompt unchanged.
**Other switch measurements** (Measured on: the same Ryzen AI Max+ 395)
| Switch | Result |
| --- | --- |
| `STRATA_HC_Q8` (IQ3_S, 2026-10-21, MTP 4 drafts, min-p 0.5) | `json_extract` 53.55 -> 58.03 tok/s (+8.4%); 94 of 250 output tokens differ from the BF16 read (first at 60); cold read 93.8 -> 20.3 us (142 -> 352 GB/s) |
| `STRATA_PF_F16=1`, one layer, 16,384 tokens | Q4_K / Q5_1 85.3 -> 53.5 ms; IQ3_S / IQ4_NL gate/up 11.65 -> 9.87 ms and down 6.11 -> 4.71 ms |
| `STRATA_PF_F16=1`, first-token KL against no switches, 12 prompts of 1.5K-24K tokens, top-1 same in 12 of 12 | UD-Q4_K_XL mean 0.0068 (median 0.0021) against 0.0031 (0.0024) without it; UD-IQ4_XS 0.0085 (0.0036) against 0.0068 (0.0032) |
| `STRATA_REFILL_PARTS=1`, UD-Q4_K_XL and UD-IQ4_XS | 4K-8K prompt +2.3% to +3.5%, 32K +1.1% |
| `--mtp-window 8192` | 64K output +3.5%, 32K -0.4% |
| 12-turn agent replay, `STRATA_PREFILL_STREAM_MIN=128`, C2T8, `--no-prefill-borrow` | 3.4 -> 2.1 s per turn |
| Defaults off, IQ3_XXS 8K | prompt 8% lower (1,137) |
docs/TERMINAL_LAUNCH.md
+195 -80
View File
@@ -1,116 +1,231 @@
# `strata launch`: set up, serve and watch, in the terminal
# `strata launch`: set up, serve and watch
One command for the whole path: choose a model, tune it, start serving, and watch it. It is the same questions as the one-click setup (`START-HERE.bat` / `setup.sh`),
in the Terminal's look, with presets, an Advanced table, an access choice and a Connect view for your coding apps. Text previews of every screen are in
This page is for anyone who uses the installed Terminal. One command chooses a model, tunes it, starts serving and
opens a monitor. It also connects your coding apps. Text previews of the screens are in
[docs/terminal-previews/](terminal-previews/) (`launch-*.txt`).
## Quick start
1. Install the Terminal with the one-line command in [INSTALL.md](INSTALL.md#the-terminal).
2. Run `strata launch` and follow the screens, or run a command without screens:
```sh
strata launch --plan --family qwen --model q2_0 # what would happen; nothing changes
strata launch --family qwen --model q2_0 --yes --no-tui # download, serve, print the address, return
strata launch --status # what is serving now
strata launch --stop
```
`strata launch --help` lists every flag with examples. An unknown model or flag gets a plain message with the closest match.
## Install (one line)
Linux and Mac:
```sh
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh # Linux, macOS
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex # Windows PowerShell
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
```
It finds out which computer this is, downloads the Terminal archive named in the release folder's `install-manifest.txt`, checks its SHA-256, unpacks it into your
home (`~/.local/share/strata/terminal`, `%LOCALAPPDATA%\Strata\terminal`; no administrator rights), checks the release's signed feed with the Terminal it just unpacked
(`strata update --check`; if that fails it removes what it unpacked), puts `strata` on your PATH and prints `Installed. Run: strata launch`. The engine (the part that talks
to your graphics card) comes with the first model: `strata launch` fetches the one for your card from the same release folder, signed, and test-starts it before using it.
The release folder is the latest GitHub release of Niko1221/Strata (`https://github.com/Niko1221/Strata/releases/latest/download`). The script only fetches and checks; read it first if you like (it is the same file as `install.sh` / `install.ps1` in the repository's `packaging` folder and in the release). Set `STRATA_FEED` to another folder to try a local one; `packaging/make_local_feed.py` builds a test-signed one. How far to trust a piped installer is in [SECURITY.md](../SECURITY.md).
Windows PowerShell (then open a new terminal):
## The screens (`strata launch`)
```powershell
irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
```
1. **Start**: saved presets (name, model, context, parallel requests, tuned or not, last used) or "Set up from the beginning" or "Use model files I already have".
2. **Model**: the catalog of the app and `setup.py` (one source), with download size, what it needs (graphics memory + RAM) and "fits / tight / does not fit this computer".
A warning, never a block. The n-gram table stays on your SSD and is not counted. `o` = your own files (a folder or a .gguf; checked with the fingerprint table).
*(Cards: only with two or more usable graphics cards. Each card gets a role: M Main, H Helper, O Off, R back to the recommendation, D details. `--gpu-roles 1=main,0=helper` does the same from the command line; see [MULTI_GPU.md](MULTI_GPU.md).)*
3. **Folder**: where model files are stored, free space shown. For a model that is already installed (also one the desktop app installed) it starts where the model is, says
"Already downloaded" and lists only what is really missing with its size (the draft layer, about 5.2 GB; the picture reader, 0.9 GB). It never says "download the model" when only
a helper file is missing.
More in [INSTALL.md](INSTALL.md#the-terminal). The installer needs no administrator rights and puts `strata` on your PATH.
How far to trust a piped installer is in [SECURITY.md](../SECURITY.md). The engine (the part that talks to your graphics
card) comes with the first model: `strata launch` fetches the one for your card from the same release, checks its signature
and test-starts it before using it.
## The screens
1. **Start:** saved presets (name, model, context, parallel requests, tuned or not, last used), "Set up from the beginning"
or "Use model files I already have".
2. **Model:** the catalog the app and `setup.py` share, with download size, what it needs (graphics memory and RAM) and
"fits / tight / does not fit this computer". This is a warning, never a block. The n-gram table stays on your SSD and is
not counted. `o` picks your own files (a folder or a `.gguf`).
3. **Folder:** where model files are stored, with free space. For a model that is already installed, including one the
desktop app installed, it starts where the model is and lists only what is missing (the draft layer, about 5.2 GB; the
picture reader, 0.9 GB).
4. **Context** (8K to 512K, with the extra memory each costs) and the memory format (`m`: 8-bit or 4-bit).
5. **Pictures**: only for models that can read them. They are **on by default** (Yes is preselected; `--vision off` or "No" turns them off). It says what it adds (about 0.9 GB:
the picture-reader file, downloaded with the model through setup's Control mode, the same path the app's "Read pictures" uses). The reader runs on the CPU
for a card under 24 GB (a few seconds per picture, nothing taken from the card) and on the graphics card from 24 GB (about 1-1.4 GB kept free), on NVIDIA and AMD cards; the
rule lives in one place, `strata_control/vision_reader.py`, for setup, `strata launch` and the app's catalog. `--vision gpu` / `--vision cpu` (or the Pictures step) switch it per model. It needs a picture reader beside the engine in the engine package; with an
engine that has none, pictures default to off and the screen says so in plain words. A model that cannot read pictures stays off with no question.
6. **Parallel requests**: "fit to your card" (12 GB: 1, 20: 2, 24: 3, 32+: 4) or 1 to 4; a warning if it will not fit.
7. **Tuning**: quick (about 20 minutes) / thorough (up to 2 hours) / skip (`strata tune auto`; the "answers may change very slightly" tick is off). The screens suggest quick the first
time (it is done once and kept); the command line skips tuning unless you add `--tune`.
8. **Advanced** (optional): every engine setting with default, tuned value and your value; edit, reset, add a setting by name (checked against the engine's own list; a warning, not a refusal);
the exact command (it includes the context, memory format and picture choice you made above; it has its own scroll area: Tab, then the arrows) and the matching `strata launch` flags.
9. **Access**: this computer only (127.0.0.1), your network (LAN address, key), Cloudflare quick tunnel (needs `cloudflared`), Tailscale (needs it installed and connected); port; API key
(optional, recommended beyond this computer: Strata makes one for you, and one keypress (`t`) turns it off. Without a key the server still starts and warns that anyone who can reach the port can use the model and its tools; `--no-api-key` does the same without the screens, and on a Cloudflare tunnel the warning is stronger. This computer only never needs a key).
10. **Save as preset** and launch. Then downloads (with progress), engine start, and the monitor. In a small window the summary shrinks so the three choices stay on screen,
and the key line says what Enter will do. No web page opens by itself; only `--open` opens one.
5. **Pictures:** only for models that can read them. They are on by default (`--vision off` or "No" turns them off). The
reader adds about 0.9 GB. It runs on the CPU for a card under 24 GB and on the graphics card from 24 GB (about 1 to 1.4 GB kept
free). The rule is in `strata_control/vision_reader.py`. `--vision gpu` or `--vision cpu` overrides it per model. If the
engine package has no picture reader, pictures default to off and the screen says so.
6. **Parallel requests:** "fit to your card" (12 GB: 1, 20: 2, 24: 3, 32 and more: 4) or a number from 1 to 8, with a warning
if it will not fit. Check the effective slots after startup.
7. **Tuning:** quick (about 20 minutes), thorough (up to 2 hours) or skip. The screens suggest quick the first time; the
command line skips tuning unless you add `--tune`.
8. **Advanced** (optional): every engine setting with its default, tuned value and your value. You can edit, reset, or add a
setting by name (checked against the engine's own list, with a warning instead of a refusal). It also shows the exact command and
the matching `strata launch` flags.
9. **Access:** this computer only (127.0.0.1), your network (LAN address and key), a Cloudflare quick tunnel (needs `cloudflared`)
or Tailscale (needs it installed and connected). Also the port and an API key. Beyond this computer a key is recommended:
Strata makes one, and `t` turns it off. Without a key the server still starts and warns that anyone who can reach the port
can use the model and its tools; `--no-api-key` does the same without screens, and the warning is stronger on a Cloudflare tunnel.
10. **Save as preset** and launch. Downloads show progress, then the engine starts and the monitor opens. No web page opens
unless you pass `--open`.
Keys: arrows choose, Enter goes on, Esc goes back, `?` explains the screen, `q` leaves. Every screen says in plain words what it means and what to do next.
Keys: arrows choose, Enter goes on, Esc goes back, `?` explains the screen, `q` leaves.
With two or more usable graphics cards the Model step also asks for a role per card (M main, H helper, O off). On the command
line, `--gpu-roles 1=main,0=helper` does the same ([MULTI_GPU.md](MULTI_GPU.md)).
## Flags
```sh
strata launch --preset "IQ3_XXS 128K" # a saved preset (opens the monitor)
strata launch --family qwen --model iq3_xxs --context 128k --parallel 4 --vision off --tune quick \
--access lan --port 8080 --save-preset "Office" --no-tui
strata launch --model-path /data/models/IQ3_XXS --no-tui # files you already have
strata launch --preset Office --set prefill=65536 --set STRATA_PIN_BY_NEED=1 --no-tui
strata launch --family qwen --model q2_0 --context 64k --restart --no-tui
strata launch --list-presets
strata launch --show-key
```
| Flag | What it does |
| --- | --- |
| `--preset NAME` | Start a saved preset. Other flags change that start only; add `--save-preset` to keep them. |
| `--family`, `--model` | The model family and size, for example `qwen`, `q2_0`. Any case. |
| `--model-path PATH` | Use model files you already have: a folder of shards or one `.gguf`. |
| `--data-dir DIR`, `--move-data` | Where new downloads go. `--move-data` also moves the models you already have. |
| `--context SIZE` | `32k`, `64k`, `128k`, `256k` or a number of tokens. |
| `--kv FORMAT` | Conversation memory format: `int8` (default), `q4_0` (half the memory), `k8v4`, `fp16`. |
| `--vision off\|on\|gpu\|cpu` | Picture reading. Default: on when the model and the engine can. |
| `--parallel N` | Requests answered at the same time, 1 to 8. Default: fit to your card. |
| `--mtp on\|off` | The draft layer. `off` saves about 1 GB of graphics memory and is slower. Default: on. |
| `--drafting auto\|mtp\|dflash2\|off` | Which drafter guesses ahead. `dflash2` is for Qwen3.8-Flash-Next only and downloads a 1.2 GB drafter once. If it cannot run, MTP is used and the start says why. |
| `--gpu LIST`, `--gpu-roles LIST` | Which cards to use, and a role per card. |
| `--tune quick\|thorough\|skip`, `--tune-allow-small`, `--retune` | Tuning. `--retune` measures again; a model tuned before for the same choices on this computer reuses the stored result. |
| `--set KEY=VALUE` | An advanced engine setting or `STRATA_*` variable (repeatable). Unknown names get a warning. |
| `--pool-cores LIST` | Experimental: pin CPU workers to these logical processors, for example `2-14:2` ([DETAILS.md](DETAILS.md)). |
| `--access local\|lan\|cloudflare\|tailscale`, `--port N` | Who can reach the server. |
| `--api-key KEY`, `--api-key-env NAME`, `--new-key`, `--no-api-key` | The key clients must send. |
| `--save-preset NAME`, `--list-presets`, `--delete-preset NAME` | Presets. |
| `--no-tui` | No screens: serve, print the address and return. |
| `--yes` | No questions. It never restarts a running model. |
| `--restart` | Stop a model that is serving other choices (or holds the port), then start these. |
| `--open` | Open the server's web page once it serves. |
| `--plan` | Show what would happen and exit. It lists the model, context, picture setting and what would download (0 when everything is here). |
| `--feed URL\|DIR` | The signed update feed. Default: `STRATA_UPDATE_FEED` or the project's. |
| `--output human\|json` | Output format. |
| `--status`, `--stop`, `--show-key` | Show, stop, or print the key of the running launch (add `--preset NAME` to pick one). |
Presets are small files in `launch-presets/` beside the profiles folder. A run uses the shared profile `launch-<name>` and the shared
tuning presets (`strata presets`), so the app sees them too.
**Launching while a model is serving.** The same choices restart nothing and print the running addresses. Different choices
(another context, pictures, access, or another model on the same port) make the screens ask once, "Restart the model with these
settings? Answers in progress stop." With `--no-tui` the command refuses with one line naming the running model, its port and settings. Add
`--restart` or run `strata launch --stop` first. A different `--port` runs beside the first server; two big models may not fit in graphics memory together, and the launch says so.
## DeepSeek V4.1-Flash
DeepSeek V4.1-Flash runs in the Terminal and in the app, on NVIDIA RTX 30, 40 and 50 (Windows and Linux) and on the Radeon AI PRO R9700 (Linux). Hardware, sizes and speeds are in [MODELS.md](MODELS.md#deepseek-v41-flash).
```sh
strata launch --plan --family deepseek41 --model ist-das-2.00bpw
strata launch --family deepseek41 --model ist-das-2.00bpw
```
| Model name | Download |
| --- | ---: |
| `ist-das-1.75bpw` | 239 GB |
| `ist-das-2.00bpw` | 257 GB |
| `ist-das-2.25bpw` | 274 GB |
| `ud-q8_k_xl` | 508 GB |
- The context is 8192 tokens (8K), the size the engine was checked at.
- `--speed faster|full` picks how many of the model's 6 experts it uses per word. Faster uses 5 (a small quality change, usually not noticeable) and is the default on a PC with 16 GB of graphics memory or less and on PCs whose graphics share the computer's memory. Full uses all 6. `--full-quality` is the same as `--speed full`. Both need DeepSeek engine 0.1.50 or newer, which `strata launch` fetches. The engine's own flag is `--experts N`.
- It answers one request at a time. `--parallel` above 1 is ignored with a note.
- It cannot read pictures, so pictures are off and `--vision on` is ignored with a note.
- `--tune` runs DeepSeek's own short tuner (drafter on or off, CPU threads), not the Qwen tuner.
- `--model-path DIR` starts from the shards you already have.
- `--plan` shows where the model goes on your cards and how much is held in RAM.
## After the start: the monitor
The Strata Monitor with the new **Overview** (key 1: model state bar, speed, GPU load, VRAM, temperature, power, PCIe, CPU, disk, context fill, experts in VRAM, system RAM, recent requests; read from the
server's `/metrics`, the same data as the browser Monitor), the other five views as before, and **7 Connect** (the seventh view: the arrow keys reach it like the others, and every view's hint says 1-7): the address, the key (hidden until `k`), and the setup for each app
(`c` copy, `w` write it for you after a backup, `h` check it: one prompt and one tool call sent the way that app sends them). The list keeps two things apart: **Vendor tested** (the app
version its maker tested the setup with: history) and **This server** (Not checked / Passed / Failed for your server, after `h`). `q` asks: keep serving in the background, or stop
(stopping unloads the model and frees the graphics memory).
The monitor has seven views. Press `1` to `7` or use the arrow keys.
In the classic Windows console (the Windows 10 default window) the monitor first switches that window's font to Cascadia Mono when it is installed (the console font API: no admin, nothing saved, gone with the window) and draws the full look; otherwise it draws a solid-block look: █ bars with a thin ▄ track, ▄ / █ sparklines, square boxes and the 16 palette colours (`•` instead of `●` when the font is not Consolas). It always shows one line: "For the full look, use Windows Terminal (built into Windows 11)". Windows Terminal, Linux and macOS are never touched. `STRATA_GLYPHS=unicode|ascii|solid` forces a look. With two or more graphics cards the Overview tiles show the busiest card (load), all cards (memory), the hottest card and the total power, a "Cards" strip lists each card (up to 4, then "press 5" for the Hardware view), and the CPU tile says "2 CPUs · 64 cores" from the server's new `sockets` figure.
| Key | View |
| --- | --- |
| 1 | Overview: model state, speed, GPU load, VRAM, temperature, power, PCIe, CPU, disk, context fill, experts in VRAM, system RAM, recent requests. It reads the server's `/metrics`, like the browser Monitor. |
| 2 to 6 | Serving, Requests, Engine, Hardware, Logs |
| 7 | Connect: the address, the key (hidden until `k`) and the setup for each app |
## Without the screens (flags)
In Connect, `c` copies a setup, `w` writes it after a backup and `h` checks it by sending one prompt and one tool call the way that
app sends them. The list keeps two columns apart. **Vendor tested** is the app version its maker tested the setup with.
**This server** is Not checked, Passed or Failed for your server after `h`.
```sh
strata launch --preset "IQ3_XXS 128K" # a saved preset (opens the monitor)
strata launch --model iq3_xxs --context 128k --parallel 4 --vision off --tune quick \
--access lan --port 8080 --save-preset "Office" --no-tui # serve and print the address
strata launch --model-path /data/models/IQ3_XXS --no-tui # files you already have
strata launch --preset Office --set prefill=65536 --set STRATA_PIN_BY_NEED=1 --no-tui
strata launch --model q2_0 --pool-cores 2-14:2 --no-tui # experimental: pin the CPU workers to these logical processors (see DETAILS.md)
strata launch --plan --model q2_0 # what would happen; nothing changes
strata launch --model q2_0 --yes --no-tui # no questions (for one: download on a computer with no engine yet)
strata launch --model q2_0 --open # also open the server's page in your browser once it serves
strata launch --model q2_0 --context 64k --restart --no-tui # a model is already serving other choices (or holds the port): stop it and start these
strata launch --list-presets | --delete-preset NAME | --status | --stop [--preset NAME] | --show-key
```
`q` asks whether to keep serving in the background or stop. Stopping unloads the model and frees graphics memory.
`strata launch --help` lists everything with examples. An unknown model or flag gets a plain message with the closest match. Presets are small files in `launch-presets/` beside the profiles
folder; the run uses the shared profile `launch-<name>` and the shared tuning presets (`strata presets`), so the app sees them too. Flags next to `--preset` change that start only
(add `--save-preset` to keep them). `--plan` shows the model (also for your own files, which it recognises), the context, the picture setting and what would be downloaded and how big (0 when everything is already here). A model that was tuned before for the same choices on the same computer is not measured again by `--tune quick`; the stored result is used and the run says so. Add `--retune` to measure again (the model of the same launch is stopped for the measuring and starts again with the result).
With two or more cards the Overview shows the busiest card's load, all cards' memory, the hottest card and the total power, and a
"Cards" strip lists up to 4 cards (`5` opens Hardware). The CPU tile shows sockets and cores.
**A launch while a model is already serving.** The same choices: nothing restarts, the running addresses are printed. Different choices (another context, pictures, access, or another model on the same port): the screens ask once, "Restart the model with these settings? Answers in progress stop.", with Restart and Cancel; `--no-tui` refuses with one line naming the running model, its port and settings, and says to add `--restart` or run `strata launch --stop` first. `--yes` alone never restarts. With `--restart` the old server is stopped first and the new one starts on the same port. A different port given with `--port` runs beside the first one; two big models may not fit in the graphics memory together, and the launch says so.
**Windows console.** In the classic Windows 10 console the monitor switches that window's font to Cascadia Mono when it is installed
and draws the full look. Otherwise it draws a solid-block look and shows one line suggesting Windows Terminal. Nothing is saved and
Windows Terminal, Linux and macOS are never touched. `STRATA_GLYPHS=unicode|ascii|solid` forces a look.
## Connect your apps: `strata connect`
`strata connect` lists OpenCode, Codex CLI, Claude Code, oh-my-pi (omp), pi, Hermes Agent, OpenClaw and the DeepSeek Harness (built into the app). `strata connect opencode` prints the file and
environment for the running server (address, key, model name and context filled in); `--write` writes it after asking and keeps a backup of anything it replaces; `--check` sends one prompt and one tool call
the way that app does and says whether each worked. With `--output json` (`strata connect opencode --check --output json`) it prints one JSON object with the same file and environment fields plus `checks` (one entry per check: `name`, `ok`, `detail`, `seconds`) and `checked_server`; `ok` is true only when both checks passed. The exit code is 0 when both worked and 1 when one failed or the server could not be reached; in that case the object also carries an `error` that says which of the two it was. Each configuration was tested for real against a Strata server at 32K context (the app versions are on screen); the known limits are printed in plain
words (for example: Claude Code needs `CLAUDE_CODE_MAX_CONTEXT_TOKENS`; OpenCode needs `limit.input`; Hermes refuses windows under 64K unless set up as the config does; OpenClaw needs Node 24.16+).
`strata connect` lists OpenCode, Codex CLI, Claude Code, oh-my-pi (`omp`), pi, Hermes Agent, OpenClaw and the DeepSeek Harness
(built into the app).
## Boxes the installer cannot set up
```sh
strata connect opencode # print the file and environment, with address, key, model and context filled in
strata connect opencode --write # write it after asking; anything replaced is backed up
strata connect opencode --check # send one prompt and one tool call the way that app does
strata connect opencode --check --output json
```
`setup.py` sets up NVIDIA (CUDA) and AMD (HIP) cards. For an Intel Arc card its Linux path is a container build that the Terminal does not run: if the model's files and their prepared pack are
already on the disk, `strata launch` writes the run settings itself and says so; otherwise it says which file is missing. Nothing is downloaded or compiled in that case.
| Flag | What it does |
| --- | --- |
| `--url URL`, `--key KEY`, `--key-env NAME` | The server address and key. Default: the running launch, else `http://127.0.0.1:8080`. |
| `--model NAME`, `--context SIZE` | The model name the server reports and the context it started with. Default: the running launch's, else 32k. |
| `--preset NAME` | Take address, key and model from a saved preset. |
| `--yes`, `--show-key` | Do not ask before `--write`; print the real key instead of a masked one. |
With `--output json` the object has the file and environment fields, `checks` (one entry per check with `name`, `ok`, `detail`,
`seconds`) and `checked_server`. `ok` is true only when both checks passed. The exit code is 0 when both worked and 1 otherwise;
then `error` says which check failed or that the server could not be reached.
Each configuration was tested against a Strata server at 32K context. Known limits are printed in plain words, for example:
Claude Code needs `CLAUDE_CODE_MAX_CONTEXT_TOKENS`; OpenCode needs `limit.input`; Hermes refuses windows under 64K unless set up
as the config does; OpenClaw needs Node 24.16 or newer.
## Models you already have
`strata models list` looks in the same places the screens and `strata launch` look (one function, `strata_control/discovery.py`): the folder you name with `--install-dir` or
STRATA_INSTALL_DIR (then only that one), the Terminal's own folder, the folders earlier installs remembered in settings.json, and the desktop app's install folder, which the Terminal
shares (Windows `%LOCALAPPDATA%\Strata\install`; Linux and macOS `$XDG_DATA_HOME/strata/install`, by default `~/.local/share/strata/install`). The data folder is the one you set, else the
app's (`app-settings.json` beside its install folder, else a `Documents/Strata` that holds models), else Strata's default. `strata models import PATH --plan` knows the Gyro-S and Gyro-M files
(TQK6/TQK7 and TQ2_T experts) and every file set setup installs, also under another name when the header matches, and says which run config, prepared pack, draft layer and picture reader
already exist for them. Nothing is copied or moved.
`strata models list` looks in these places (one function, `strata_control/discovery.py`):
- the folder you name with `--install-dir` or `STRATA_INSTALL_DIR` (then only that one)
- the Terminal's own folder
- folders earlier installs remembered in `settings.json`
- the desktop app's install folder, which the Terminal shares: `%LOCALAPPDATA%\Strata\install` on Windows,
`$XDG_DATA_HOME/strata/install` (normally `~/.local/share/strata/install`) on Linux and macOS
The data folder is the one you set, else the app's (`app-settings.json` beside its install folder, else a `Documents/Strata` that holds
models), else Strata's default.
`strata models import PATH --plan` recognises the Gyro-S and Gyro-M files (TQK6/TQK7 and TQ2_T experts) and every file set setup
installs, also under another name when the header matches. It says which run config, prepared pack, draft layer and picture reader
already exist. Nothing is copied or moved.
## When there is no engine for this computer
`strata launch` does not refuse. It says, in plain words, "Strata couldn't find an engine for this computer (the reason). The model will download, but it may not run until you install the right
engine (next step: ...)", asks whether to go on (Enter means yes; `--yes` skips the question; nothing to download means no question), downloads the model, and ends with a message that names the
engine problem and the next step. The exit code is 0 and nothing is started.
`strata launch` does not refuse. It says "Strata couldn't find an engine for this computer" with the reason, asks whether to go on
(Enter means yes; `--yes` skips the question), downloads the model, and ends with a message that names the problem and the next step.
The exit code is 0 and nothing is started.
## Cards of different makes
## Intel Arc and cards of different makes
One engine drives the cards of one make (CUDA: NVIDIA, HIP: AMD). Only those cards are offered for running and splitting; a model is never split across makes, and two models at the same
time, one on each make, is not supported yet. When the other make has the bigger card you get one line: "This computer also has an AMD R9700 (32 GB). The AMD build of Strata can use it - it fits bigger models."
- **Intel Arc:** the Terminal downloads the released Linux SYCL engine, but its model preparation for Arc is a container build that it does not
run. If the model's files and prepared pack are already on disk, `strata launch` writes the run settings itself. Otherwise it reports the preparation problem. Use
[Intel setup](INTEL_ARC.md) for the preparation route.
- **Mixed makes:** one engine drives the cards of one make (CUDA: NVIDIA, HIP: AMD). Only those cards are offered, and a model is never
split across makes. Two models at once, one per make, is not supported. If the other make has the bigger card, you get one line saying so.
## Limits
* The model download uses `setup.py` in its control mode (no pip, no engine fetch, no compiler), as the desktop app does; it needs internet and room for the files.
* `strata update`'s test start of an engine runs `<engine> --version`; the SYCL (Intel) package's engine does not answer that flag (exit 2), so on an Intel box the engine package has to be copied into the
components folder until that is fixed.
* The Windows console has no way for the test scripts to drive the screens; they were checked by pilot tests and by hand.
- The model download uses `setup.py` in its control mode (no pip, no engine fetch, no compiler), as the desktop app does. It needs internet and room for the files.
- The Linux SYCL package updates through `strata update`. Do not copy it into the components folder by hand.
docs/TERMINAL_QUICKSTART.md
+6 -3
View File
@@ -1,8 +1,11 @@
# Terminal quick start
New here? `strata launch` sets up a model, serves it and opens the monitor in one guided flow: [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md).
This page is for people who already have a model and want to read a running server, rerun it from a saved profile, or measure it.
To set up and serve a new model, use [`strata launch`](TERMINAL_LAUNCH.md).
These commands use the `strata` command that the one-line Terminal install puts on your PATH ([how to install](TERMINAL_LAUNCH.md#install-one-line)). The Terminal carries its own Python, so there is nothing else to install. In a source checkout, type `python strata.py` (or `python3 strata.py`) in place of `strata`, from the repository root. The commands below read, prepare and rerun a model you already have; they do not download one. The complete options, permissions and limits are in [TERMINAL.md](TERMINAL.md).
The commands use the `strata` command from the [Terminal install](INSTALL.md#the-terminal), which carries its own Python.
In a source checkout, type `python strata.py` in place of `strata`, from the repository root. These commands read, prepare
and rerun a model you already have; they do not download one. All options, permissions and limits are in [TERMINAL.md](TERMINAL.md).
## Read a running server
@@ -18,7 +21,7 @@ The interactive monitor comes with the installed Terminal. Open the one-screen O
strata monitor --url http://127.0.0.1:8080
```
The Overview puts serving state, prompt and answer speed, batch/queue pressure, four selected GPU rows, requests and activity on one screen. Press `1`–`6` for detail (`1`–`7` in the monitor that `strata launch` opens, where `7` is the Connect view), `Space` to freeze the display, `r` to refresh, and `q` to detach. Detach leaves the server running. On a machine without a terminal, use `status --json` or `monitor --once --json`.
The Overview puts serving state, prompt and answer speed, batch/queue pressure, four selected GPU rows, requests and activity on one screen. Press `1`–`7` for the views (`7` is Connect), `Space` to freeze the display, `r` to refresh, and `q` to detach. Detach leaves the server running. On a machine without a terminal, use `status --json` or `monitor --once --json`.
For a remote server, use HTTPS or a local SSH tunnel. A saved connection stores only the credential environment variable's **name**:
docs/TROUBLESHOOTING.md
+109 -246
View File
@@ -1,279 +1,142 @@
# Something went wrong?
# Troubleshooting
The common problems and what to do, for the desktop app, the Terminal (`strata launch`) and the installer. Back to the [README](../README.md#something-went-wrong). The full table of
error messages, with the older engine fixes, is in the [details](DETAILS.md#troubleshooting).
This page is for anyone whose install or startup fails. Find your symptom below. Each entry gives one fix.
Start from the route you installed: [desktop app or Terminal](INSTALL.md), or [source checkout](SOURCE_INSTALL.md).
The normal release uses the app or `strata launch` and needs no `.bat` file. Detailed engine errors are in the
[engine reference](DETAILS.md#troubleshooting).
**Quick check.** Run these and keep the output for a bug report:
```sh
strata --version
strata doctor --live
strata launch --status
```
## The desktop app and the Terminal
**A warning says Strata "couldn't find an engine for this computer".**
The engine is the part that talks to your graphics card. Strata only warns and still downloads the
model, and the model files stay on your computer, but the model may not run until the right engine is installed. The usual
reasons are that no engine is installed yet and the update source has none for your card, that the engine found is for
another make of card, or that the graphics driver is too old. In the app, open Settings, then Engine. In the Terminal, the
message names the next step: run `strata update` and start again. If your computer has no NVIDIA or AMD graphics card
(only an Intel UHD or Iris chip, an AMD processor graphics chip, a virtual machine, or Windows on ARM), Strata says so and
there is nothing to install: it needs a computer that has one. The model files stay where they are.
**I cannot find `START-HERE.bat` or `SETUP.bat`.**
Open the installed Strata app, or run `strata launch`. Those wrappers are in the source repository and are not the
startup path for the app. For a fresh desktop install, download the one installer for your system
(`Strata-0.2.5-windows-x64-setup.exe`, `Strata-0.2.5-linux-x86_64.AppImage` or `Strata-0.2.5-macos-arm64.dmg`), not a
`strata-engine-…` archive.
**"This model needs a newer engine." (the app), or a button called Update engine.**
Some models need an engine newer than the one installed: Gyro needs 0.1.45.1 or newer, NVFP4 needs 0.1.44, and NVFP4 (NVIDIA,
trained for FP4) needs 0.1.45. In the app, open Settings, then Engine, and press Update engine. It asks first, keeps your
model files and settings, and wants running models unloaded. In the Terminal: `strata update --check` says what would
change, `strata update` installs it (it switches to the new engine only after a test start). If the engine you
installed is the newest and the message stays, the update source has no newer engine for your card yet.
**`strata` is not found.**
On Windows, open a new PowerShell window after installing. On Linux or Mac, use the full path the installer printed,
normally `~/.local/bin/strata`, and add `~/.local/bin` to PATH if needed. Check `strata --version`.
**On a Mac: "Strata is damaged and can't be opened", "cannot be opened", or "unidentified developer".**
Nothing is wrong with the file. macOS marks everything a browser downloads, and only lets a marked program run if Apple
has checked it. Strata's release builds are checked by Apple ("notarised"): drag Strata onto Applications and open it; the
first time, macOS asks "Are you sure you want to open it?" and you press Open. A test build that is not notarised says
"damaged". Do not move it to the Trash. Open the Terminal app and remove the mark, with the line that matches what you have
(drag the file from Finder into the Terminal window to type its path):
**Strata could not find an engine for this computer.**
Run `strata doctor --live` and `strata update --check`. Check the selected GPU, driver and
[backend guide](INSTALL.md#what-you-need). Supported paths include Apple Silicon and Intel Arc on Linux; do not
assume every machine without an NVIDIA/AMD discrete card is unsupported. A model download can finish even when
no engine starts. Verify `strata launch --status` and `/health` before calling the installation successful.
xattr -dr com.apple.quarantine ~/Downloads/<the .dmg file> # the .dmg, before opening it
xattr -dr com.apple.quarantine /Applications/Strata.app # the app, after dragging it over
xattr -dr com.apple.quarantine <the folder you unzipped> # an unzipped Terminal folder
**This model needs a newer engine.**
Unload running models, then use the app's engine update or `strata update`. Minimum engine versions vary by model and
backend. Check the model guide and the installed engine, not only the app version. If the feed has no compatible engine,
report the missing combination.
Then open it again. If macOS says only "the developer cannot be verified", you can instead try to open it once, then open
System Settings > Privacy & Security and press "Open Anyway" next to the message about Strata. Installs made with
`curl ... | sh` carry no mark and need none of this. To see which kind you have: `spctl -a -vv /Applications/Strata.app`
answers "accepted" and "Notarized Developer ID" for a notarised build. (How the builds are signed: `packaging/mac/README.md`.)
**An update says it succeeded but `strata --version` is still old.**
The Terminal switches to the updated Control on the next command, so run `strata --version` again. If it is still old,
run the one-line installer once more; it keeps models and settings. The app, Control and engine have separate version
numbers; see [updating](INSTALL.md#updating) and the [release notes](https://github.com/Niko1221/Strata/releases).
**Checking what is installed: `strata doctor`.**
It is a read-only check of the installed engine parts (it changes nothing): which engine and helper files a saved model
needs, and whether they are there. `strata doctor --live` also asks the computer which graphics cards it sees and compares
them with the cards the model is set to use; `--output json` prints the same as data. Put its output into a bug report.
**A new engine misbehaves after an update.**
Stop the model, then use the app's rollback control or `strata components rollback`. Source-checkout engines use a
separate rollback workflow, `python setup.py --rollback-engine`, described in [source setup](SOURCE_INSTALL.md).
**Sending logs for a bug report.**
In the app: Settings, Engine, Diagnostics, View logs, then Save logs. It writes one text file,
`strata-diagnostics-<date>.txt`, into your Downloads folder and shows where. It holds the engine's messages, the failed
steps and the end of each model's server log. In the Terminal, send the output of `strata doctor`. If the Linux app window
crashes or will not open, run the AppImage with `--save-logs` in a terminal instead; it writes the crash log and the other
logs to one file in Downloads. If the Windows app closes right after you start it, it shows a message with the reason and writes
`desktop-crash-<date>.log` in `%LOCALAPPDATA%\Strata\logs`; Save logs includes it, and so can you, by hand. Look through a log before you send it, and never send an API key.
**The Mac app cannot be opened.**
Check that the download is for Apple Silicon and that macOS is 15 or newer. Follow
[Mac installation](MAC.md#the-desktop-app-and-strata-launch-on-a-mac). A warning alone does not establish that a file
is safe or intact: verify the download source and release before changing quarantine settings. Release signing and
test-build handling are described in [Mac packaging](../packaging/mac/README.md).
**The Linux AppImage does not open.**
Make the downloaded file executable. It may need FUSE 2 (`libfuse2` on Ubuntu 22.04, `libfuse2t64` on newer Ubuntu;
package names vary by distribution). Without FUSE, run the file with `--appimage-extract-and-run`. For a crash log,
run it with `--save-logs`; see the logs section below.
## While installing or starting
**My PC froze, or got very slow, the first time Strata started.**
That's normal while it starts, most of all the first time. Strata loads 35-55 GB into your RAM, locks part of it for
the graphics card, and works out how much of the model fits on your GPU. The mouse can freeze for a few minutes.
**Wait, and don't close the window.** The next starts are much faster. Still frozen after 10 minutes? Restart the
PC, close other programs (browsers use a lot of RAM) and try again. If it keeps happening, pick a smaller size (Q2_0
or IQ2_XS).
**A download stopped.**
Retry it in the app, or rerun the same `strata launch` command. Completed model-download steps are reused. Check
free space and the error message. For an interrupted Terminal installer, rerun the install command. Only a source
checkout uses `START-HERE.bat` / `./setup.sh` to resume source setup.
**The engine stops while it starts, on a small graphics card or a PC with little RAM. Can I turn the draft layer (MTP) off?**
Yes. The draft layer is what makes answers faster, and it takes about 1 GB of graphics memory (plus some RAM and a memory
copy that grows with the context). Without it the expert cache gets that memory back and answers are slower. Turn it off
for one model: in the app, Models, Expert settings, "Draft layer (MTP)": Off; in the Terminal,
`strata launch --model q2_0 --mtp off` (older Terminals: `--set mtp=off --set spec-min-p=off`); by hand, add `"mtp": false` to the
model's `strata-<model>.json`. The choice is kept per model and applied on every start.
**Startup takes a long time or the computer becomes slow.**
Large models need time to check files, prepare packs and fill memory. A first launch can take many minutes; there is
no fixed startup time. Watch progress and the engine log. If memory pressure is high, close memory-heavy programs and try a
smaller model or context. DeepSeek V4.1-Flash needs 46 GB of RAM at least ([model page](MODELS.md#deepseek-v41-flash));
`strata launch --plan --family deepseek41 --model ist-das-2.00bpw` shows what your computer is missing.
**I changed `strata-<model>.json` by hand and the change came back.**
Engine options you add to `args` by hand (control vectors, `--spec`, `--vram-reserve-mib 0`, anything the app does not offer)
are kept. The file has two helper keys for this. `user_args` lists your options; every start and every setup run puts them
over the ones the app generates, and for the same option your value wins. `args_written` is a copy of `args` as the app last
wrote it, so an option you added or changed in `args` is noticed and moved into `user_args` the next time the app or setup
touches the file. To undo one, delete it from both `args` and `user_args`. A value you save in the app's Expert settings
("GPU memory for other apps") is put in `user_args` for you.
**The NVIDIA driver is too old.**
Use the driver required by the engine package. CUDA 13 needs driver 580 or newer; older cards use the
[CUDA 12 route](OLDER_GPUS.md). After updating the driver and restarting if needed, open the app or rerun
`strata launch`.
Two things the app still chooses itself: the context size, KV type, picture reading and draft layer. Set them in the Models
screen, or with `"mtp": false` (draft layer). The app keeps the earlier file as `strata-<model>.json.bak`.
**Port 8080 is already in use, or I changed a running model's settings.**
Check `strata launch --status`. To replace your existing launch, rerun the intended command with `--restart`;
it stops current answers. To run beside it, choose another port, for example `--port 8081`, if memory permits.
Do not stop an unrelated program just because it holds the port. `--yes` alone does not restart a running model.
**The app keeps more graphics memory free than I want (decode is slower than in 0.1.45).**
Without a value of yours the app keeps 1536 MiB free (2048 with picture reading) on a card that drives a display, and 512 MiB
(700 with pictures) on a card that has no display attached, such as a second GPU with the screen on the integrated one. It
reads that from the driver (nvidia-smi `display_active`, or the Windows adapter that has the desktop); when it cannot tell, it
assumes a display. Your own value is never raised or lowered: set "GPU memory for other apps" in Expert settings, or
`--vram-reserve-mib 0` in `args`. The engine's own warning about low free memory stays; a card that also draws the desktop
can freeze or slow to a few tokens a second with a very small reserve.
**The engine stops on a small GPU or low-memory computer.**
Try a smaller pack or context. The draft layer also uses memory. In the app, set the model's Draft layer to Off;
in the Terminal, an explicit example is:
**It stopped while downloading or installing.**
Run `START-HERE.bat` (Linux: `./setup.sh`) again. It continues where it stopped.
```sh
strata launch --family qwen --model q2_0 --context 32k --mtp off
```
**It says the NVIDIA driver is too old.**
Update it (NVIDIA App or [nvidia.com/drivers](https://www.nvidia.com/drivers)), restart the PC, and run
`START-HERE.bat` again.
Use `--restart` if replacing an active run. Turning MTP off can slow decoding. Keep some VRAM free for the display;
setting the reserve to zero is not a general speed fix.
**It says port 8080 is already in use.**
Strata is already running. Look for its window. Or another program uses the port: `START-HERE.bat --port 8081`.
Only an address that is really taken says "already in use". Any other reason (Windows keeps the port reserved, or
the config's `"host"` is not an address of this PC) is printed as what the OS said, with what to change (#769).
**Pictures do not work.**
The model, backend and installed engine package must include a compatible picture reader. Inspect the startup
error and the model guide. `--vision off` disables reading pictures; it is different from optional
[image generation](IMAGES.md). In the Terminal, use `on|off|gpu|cpu`, not source setup's `yes|no` spelling.
**"Windows blocked the Strata engine (or the image encoder) ... Smart App Control".**
On a clean Windows 11, Smart App Control refuses programs that are not signed, and Strata's `strata.exe` and
`strata-vision.exe` are not (#735). Turn it off (Windows Security > App & browser control > Smart App Control
settings; it cannot be turned back on without a reset of Windows), or, if only the image encoder is blocked, run
`START-HERE.bat --setup --vision no`.
**A new engine misbehaves after an update.**
An update keeps the engine it replaced in `engine\.previous` (one generation, about 210 MiB).
`python setup.py --rollback-engine` puts it back (and keeps the newer one there: run it again to go forward) (#670).
**UPDATE.bat / update.sh say "git pull did not succeed", or "has no commit in common" (#1276).**
The repository's history was cleaned up on 2026-10-06, so a clone made before that cannot be updated with `git pull`.
The 0.1.40.2 scripts handle it: when no tracked file is edited they keep your old commits in the branch
`pre-cleanup-backup` and move the clone to the new history; otherwise they print the two commands for you. Your models,
settings and engine are untracked files and are never touched. An older UPDATE.bat / update.sh (before 0.1.40.2) cannot
do this itself: run these once in the Strata folder, then use UPDATE.bat as usual (add `git stash` first if `git status`
lists edited files):
git fetch origin
git branch pre-cleanup-backup
git checkout -B main origin/main
**Python or the build tools could not be installed.**
Install what it names (links are printed), then run it again. Everything already done is kept.
**Linux: the engine does not compile (`unsupported GNU version`, or `exception specification is incompatible` for
`cospi`/`sinpi`/`rsqrt`).** Two known mismatches between the CUDA toolkit and a new Linux (#601):
- gcc newer than 14 (Ubuntu 26.04's default 15): CUDA 12.x and 13.0 refuse it. Install g++-14 and run
`CXX=g++-14 ./setup.sh` (CC, CXX and CUDAHOSTCXX name the build's compilers; setup passes them to CMake itself,
so an already-configured build folder picks them up too, #1645).
- glibc 2.43 with CUDA 12.9: the toolkit's math headers clash with glibc's, already in CMake's first test compile.
Use CUDA 12.8 or 13.x instead. Setup takes the newest toolkit it finds; `STRATA_NVCC=/usr/local/cuda-12.8/bin/nvcc
./setup.sh` makes it use that one (only that one).
**The first start takes minutes.**
It is reading 34-55 GB into RAM; the second start is faster while the files are in the OS cache. Started from Task
Scheduler, it can be 24x slower: see [Running it at startup](DETAILS.md#running-it-at-startup-task-scheduler).
On Linux the engine now asks the kernel to read the model files ahead (a cold start went from ~920 s to 70 s on one
PC); `STRATA_READ_AHEAD=0` turns that off. On Linux with transparent huge pages on `always`, the arena no longer asks
for `MADV_HUGEPAGE` on top (a fragmented machine spent minutes compacting memory, #771); `STRATA_NO_ARENA_THP=1` skips
that request on any setting.
**Setup says the engine does not match GitHub's checksum.**
Setup checks the downloaded ready-made engine against the SHA-256 GitHub publishes. On a mismatch the file is corrupt or was changed on the way, so setup deletes it and stops; run setup again to download it afresh. If GitHub gives no checksum (offline, rate limited, your own mirror), setup warns and installs anyway. `STRATA_SKIP_SHA256=1` skips the check, only if you insist.
**Windows blocks an executable.**
Verify the official release and the component named in the message. Feed signatures and Windows code-signing are
different checks. If only the picture reader is blocked, `--vision off` may let text requests run. Do not disable
system-wide protection as the routine installation fix; include the blocked file and exact message in a bug report.
## While it answers
**It's very slow and the disk light keeps blinking.**
Your PC is out of free RAM. Close other programs, or pick a smaller size (Q2_0 or IQ2_XS).
**It is slow and the disk stays busy.**
The chosen model may exceed available RAM/VRAM and be using the file tier. Close other programs, check memory in
the monitor, and choose a smaller model or context. Compare performance only with the same hardware and workload.
**An answer stopped with "the engine stopped unexpectedly".**
Usually not enough RAM (on Linux the system then stops the engine). Just send your message again: Strata starts the
engine by itself. If it keeps happening, close other programs or pick a smaller size.
**The prompt exceeds the context.**
Start a new chat, shorten the input, or use the app's Summarize and continue action. A larger context needs more
memory. In the Terminal, restart your chosen model with an explicit `--context`, such as `64k`; preserve its family,
model and other choices. Do not look for `SETUP.bat` in an app install.
**On Linux the engine stops with "exit code -9", but nothing ran out of RAM.**
Check `ulimit -R`. If it prints 0 (or any number), the desktop session sets a real-time CPU limit: the audio system (PipeWire) does that when rtkit is not installed, and every program started from the desktop inherits it. The kernel can then end a program with signal 9 and no message. Install `rtkit`, or start Strata outside the session's limits, for example `systemd-run --user --collect -p LimitRTTIME=infinity -- strata serve ...`. Strata prints a warning at start when it sees this limit.
**On Linux the engine exits with code -9.**
Check system logs for an out-of-memory kill. Also check `ulimit -R`: a desktop-session real-time limit can cause a
kill even when memory is available. The engine prints a warning for that condition; see the
[detailed troubleshooting table](DETAILS.md#troubleshooting) before changing system limits.
**It says the prompt exceeds the context.**
The conversation is longer than the context you chose. Start a new chat, or run `SETUP.bat` and pick more
context.
**A hand-edited engine option disappears.**
The run config tracks explicit options in `user_args` and the generated baseline in `args_written`. Prefer the
app's Expert settings or Terminal `--set` for supported overrides. Context, KV, picture reading and drafting are
launch choices; set them through the app or the corresponding `strata launch` flags. Inspect the effective config
rather than assuming a saved preference was applied.
**It is slower than the tables.**
The monitor plugged into the graphics card and other GPU programs take VRAM from the expert cache; RAM running below
its rated speed (enable EXPO/XMP in the BIOS) slows the CPU half.
## Source-checkout problems
**An earlier reply that came back empty is gone from the chat history (0.1.40, #843).**
The server leaves out finished assistant turns that have no text, so the next reply does not copy the empty ones.
`STRATA_KEEP_EMPTY_TURNS=1` in the server's environment renders them as before.
Python/venv, CUDA compiler versions, ROCm builds, `UPDATE.bat`, `update.sh` and generated `run-<model>` scripts
belong to [SOURCE_INSTALL.md](SOURCE_INSTALL.md). Use the backend guide for build prerequisites. The packaged
Terminal carries Python; a missing system Python is not a reason to install a compiler for the normal release.
**A Turing card (RTX 20) reads prompts above ~90K tokens differently (0.1.40, #743).**
The prompt's top-k selection takes a wider kernel there, with the same ids. `STRATA_TOPK_STREAM=0` restores the old one.
If `git pull` fails in an old clone, back up your local work and follow the recovery steps the source update script prints.
Do not overwrite local branches or edits just to make `git pull` succeed.
**Decode is several times slower with `--expert-cache N` than with `auto` (#781, #831).**
An explicit N is a byte budget that is not checked against the free VRAM once the slots are written. On a card it
fills (the log says `0 MiB of VRAM free with everything loaded - LOW`), Windows pages the GPU's memory and decode drops
from 100 to 14 tok/s with only a few hundred slots too many. Use `--expert-cache auto`, or a smaller N.
## Logs and bug reports
**A low-RAM PC (about 32 GB) stalls with `--resident-budget-gib N` (#649).**
The budget is pinned (locked) in RAM, which leaves the OS and the CPU workers little room. `STRATA_RESIDENT_PIN=0` keeps
it pageable; a smaller budget works too.
- **App:** Settings → Engine → Diagnostics → View logs → Save logs. The diagnostic file goes to Downloads.
- **Terminal:** include `strata --version`, `strata doctor --live`, `strata launch --status`, and the relevant log excerpt.
- **Windows app crash:** inspect `%LOCALAPPDATA%\Strata\logs` for `desktop-crash-<date>.log`.
- **Linux app crash:** start the AppImage with `--save-logs` to collect diagnostics.
**Decode got slower in 0.1.40 on a 32 GB PC with `--resident-budget-gib N`, and the drive is read hard (#1194, #1116, #1085).**
The start line "the file tier reads unbuffered" means the experts outside the RAM copy are read straight from the drive.
0.1.40 chose that whenever the free RAM could not hold all of them; but the same few come back token after token, and
a cache a twentieth of their size serves most of the repeats. 0.1.40.2 asks for that much (at least 1.5 GiB beside the
4 GiB headroom) and reads through the cache otherwise, which measured 7% to 118% faster decode with a third to a sixth of
the drive traffic on a 20 to 32 GB box. On 0.1.40 / 0.1.40.1: `STRATA_UNBUFFERED_LOAD=0` does the same, `=1` forces the
unbuffered reads.
**Linux: the start is OOM-killed while the engine allocates the page-locked expert copy (#1250).**
Page-locked pages come from the driver in one go and cannot wait for the kernel to give back file cache, so with the
model files' cache in the way (little really free RAM, much "available") the host could run out. 0.1.40.2 gives back the
cached pages of the model files it mapped before it page-locks the copy; when the really free RAM still does not cover it,
it takes the pages in 1 GiB steps while the RAM available (cgroup limits included) stays above the headroom, page-locks
what it has, and keeps the rest resident but pageable with a warning ("only N of M GiB could be page-locked"): the answers
are the same, only the copies of the pageable part to the GPU go through the CPU. `STRATA_PIN_GUARD=0` turns this off,
`STRATA_PIN_RESERVE_GIB=N` sets the RAM that must stay free (default: `STRATA_RESIDENT_HEADROOM_GIB`, 4 GiB).
**The Windows display driver resets, then the PC blue-screens (0x141, then 0x116 in `nvlddmkm`) during an answer (#961).**
This is the NVIDIA driver, not Strata: the report shows the same wedge with other CUDA programs on that driver (610.88,
RTX 4080 SUPER). Try another driver (a Studio one, or an older one) and a lower power limit; if it only happens with
`--spec 4`, `--spec 0` avoids the verify window while you wait for a driver fix.
**An RTX 50 card (a source build with CUDA 13.2) answers with nonsense, or reads prompts wrongly (#892, #968).**
CUDA 13.2.0 and 13.2.1's compiler (nvcc 13.2.51) miscompile some of the engine's kernels for sm_120: on our RTX 5070 the IQ2_S and IQ3_S
products are wrong (relative error 0.5 to 1.0 in the tests) with 13.2 and right with 13.0. The ready-made engine is built
with 13.0. If you compile it yourself, use CUDA 13.0, 13.1 or 13.2.2 (13.2.2, nvcc build 13.2.86, fixes it; an older one can sit next to a newer: `STRATA_NVCC=<path to its nvcc>`);
setup warns when it finds 13.2.0 or 13.2.1 for such a card, and takes an older 13.x when one is installed.
**Pictures are refused, or slow.**
"this server was started without the vision encoder": the model was set up for text only (someone chose "No", or the
engine has no picture reader) - run setup again with `--vision yes` (or `--vision gpu` / `--vision cpu`), or switch pictures
on in the app or on the Pictures step of `strata launch`. Pictures are on by default wherever a model can read them. The
reader runs on the CPU under 24 GB of graphics memory and on the card from 24 GB, on NVIDIA and AMD cards. Pictures that take
several seconds (about 3 s at 300 image tokens on 8 cores, more with more tokens) are read by the encoder on the CPU;
`--vision gpu` (NVIDIA and AMD, ~1.4 GB of VRAM) makes it 0.1-0.5 s. On AMD the card's reader is `strata-vision-hip`; if it
cannot start, the server prints "the GPU picture reader did not start (...); pictures are read on the CPU" and goes on with
the CPU reader. On a Mac the reader runs on the GPU; `--vision cpu` takes about 50 s per picture there. On Intel Arc, pictures are off by default for now (`--vision yes` turns them on).
## AMD cards
**"No AMD GPU found (the amdgpu driver's KFD topology is empty)" on Linux.**
The kernel's amdgpu driver is not loaded for the card. Integrated Radeon GPUs are listed as not supported; setup
lists every card it found and whether Strata can use it.
**The engine stops at start with the card's name, its architecture and the build's list.**
The engine was compiled for another card (for example after moving the Strata folder to another PC). Run
`./setup.sh --setup --backend hip`: it compiles the engine for this card's architecture.
**On Windows the engine exits with `0xC0000005` in `amdhip64_7.dll` before it prints anything (#654).**
Two things in your environment can cause it, and the server now repairs both before it starts the engine (it says so in
the log): `HIP_PATH`, `HIP_DEVICE_LIB_PATH` or `LLVM_PATH` naming a ROCm folder that no longer exists (a deleted build),
and a `TEMP`/`TMP` folder the engine cannot create files in (AMD's runtime compiles its first kernels through temporary
files). If you start `strata.exe` by hand, fix them yourself: `set HIP_PATH=`, and point `TEMP` and `TMP` at a normal
folder such as `C:\Temp`.
**Windows AMD: the driver resets (VIDEO_ENGINE_TIMEOUT_DETECTED / screen flicker / the engine dies mid-answer).**
Windows gives a GPU about 2 seconds to answer; when a graphics card does not, it resets the driver (a TDR): the screen
flickers, the engine dies, and the Event Viewer shows `VIDEO_ENGINE_TIMEOUT_DETECTED` (#613, with #579 and #541 on
Linux). Strata 0.1.39 and newer say "the GPU stopped responding" in the log. It was seen with KV streaming on
(`--kv-resident 32768`, which setup turns on from 64K context) while reading a long prompt on a gfx1201 card. Try these
in order, one at a time, and tell us what changed:
1. **Send us the evidence.** The last ~80 lines of `strata-<model>.log` in the Strata folder (the server window shows
the same lines), your AMD driver version (AMD Software > System), your card and Windows version, the Strata
version, and the exact steps (context size, how long the prompt was, which app sent it). If Windows wrote a
dump, the newest `.dmp` in `C:\Windows\LiveKernelReports\WATCHDOG` (or `C:\Windows\Minidump`) helps; it holds no
chats, only the driver state.
2. **Find the step that hangs:** add `"STRATA_PF_STEP_SYNC": "1"` to the `"env"` block of `strata-<model>.json`
and restart. The prompt path then waits for the GPU after each step and logs any step over 250 ms, so the log names
the step. It is slower; use it to diagnose, then take it out.
3. **`"STRATA_KV_HOST_DMA": "1"`** in the same `"env"` block: no GPU kernel writes the RAM copy of the K/V during a
prompt, it is copied by DMA instead. Same answers.
4. **No KV streaming:** run setup again and pick 32K context (or remove `--kv-resident` and its number from
`"args"` in the JSON, with a context that fits the VRAM).
5. **A smaller prompt chunk:** change `--prefill` in `"args"` (for example `--prefill 512`): each GPU launch then does
less work between two checks by Windows. Prompts are read more slowly.
6. **Optional, your decision: give Windows more time (`TdrDelay`).** The default is 2 seconds. In the registry,
`HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers`, a DWORD `TdrDelay` of `10` (seconds)
lets a long GPU job finish before Windows resets the driver; restart the PC. It hides the symptom and does not fix
a GPU that has really hung, and a hung GPU then freezes the screen for those 10 seconds. Strata changes no Windows
setting itself; to undo it, delete the value.
7. **Driver:** use the newest AMD Software release for your card (Adrenalin, not a Windows Update driver) and restart
after installing. If it started after a driver update, tell us both versions; going back one version is a fair
test.
**Large pinned host allocations fail on ROCm although RAM is free.**
See [AMD_HIP.md](AMD_HIP.md#model-and-serving-configuration): the mapped expert mode avoids the full pinned arena.
**"verify: timed out at layer N" or "no progress for 60 s" on AMD (Linux), mostly with `--mmap-experts`.**
Two reports fixed it with `"GPU_PINNED_MIN_XFER_SIZE": "1048576"` (RX 6800) or `"HSA_USERPTR_FOR_PAGED_MEM": "0"` (two
R9700) in the `env` block of the server JSON, then a restart. Experimental, one machine each; on a systemd unit use
`MemoryMax`, not `MemoryHigh`. Details: [AMD_HIP.md](AMD_HIP.md#linux-verify-timeouts-while-the-kernel-reclaims-host-memory-experimental-workarounds).
## Still stuck?
Look in the [full troubleshooting table](DETAILS.md#troubleshooting), or open an
[issue](https://github.com/Niko1221/Strata/issues) and attach the log: in the app, the file Save logs writes (see
[above](#the-desktop-app-and-the-terminal)); in the Terminal, the output of `strata doctor`; with the installer,
`strata-<model>.log` from the Strata folder. Say which one you use (app, Terminal or the installer), your Strata and
engine versions and your card.
Include model and quantization, OS, GPU/VRAM, RAM, driver, install route, exact command, expected result and actual
error. Remove API keys and private data before sharing. Follow [CONTRIBUTING.md](../CONTRIBUTING.md).
docs/UNSLOTH_Q4.md
+184 -178
View File
@@ -1,60 +1,52 @@
# Unsloth UD-Q4_K_XL
A regular size since MODEL-UPDATE Phase 2 (it was experimental): setup offers it from engine 0.1.32
([below](#setup)); the manual workflow after that section works with engine 0.1.31 or newer. First validated on
Windows 10, an RTX 5070 (12 GB), 64 GB of RAM and an AVX-512 CPU (2026-10-01); since then on an R9700 (AMD, every
expert in RAM) and an RTX A4000 (62 GB, the RAM budget) - quality, speed and images in [QUANTS.md](QUANTS.md).
This page is for people who want to run Unsloth's 4-bit Qwen3.8-Flash-Next file (UD-Q4_K_XL) with Strata, and for
people who pack it by hand. It also covers UD-IQ4_XS and plain Q4_0 files. For UD-Q5_K_XL, UD-Q6_K_XL and Q8_0, see
[UNSLOTH_Q6.md](UNSLOTH_Q6.md). Quality, speed and images across the Unsloth sizes are in [QUANTS.md](QUANTS.md).
The target is Unsloth's 4-bit quantization of the same model the other packs use:
[unsloth/Qwen3.8-Flash-Next-GGUF, `UD-Q4_K_XL`, revision `38bb39e`](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q4_K_XL).
"UD-Q4_K_XL" is a mix of formats, not Q4_K everywhere: the routed experts are Q4_K (Q5_K in layer 2) for gate/up and
Q5_1 (Q8_0 in layers 2, 4, 30, 46, 47) for down; the token embedding, the output head, the attention and shared-expert
projections and the PLE key are Q8_0; the 28.8 GB PLE table is IQ4_NL. The routed experts are 71.7 GiB, about twice
the 3-bit models'.
"UD-Q4_K_XL" is a mix of formats, not Q4_K everywhere:
## Setup
- Routed experts: Q4_K (Q5_K in layer 2) for gate/up, Q5_1 (Q8_0 in layers 2, 4, 30, 46, 47) for down. They are
71.7 GiB, about twice the 3-bit models'.
- Token embedding, output head, attention and shared-expert projections, and the PLE key: Q8_0.
- The 28.8 GB PLE table: IQ4_NL.
`START-HERE.bat --setup` (Linux: `./setup.sh --setup`) and choose **Qwen3.8-Flash-Next (Unsloth)**; or directly:
## Quick start
Run `strata launch` and choose **Qwen3.8-Flash-Next (Unsloth)**, or:
```sh
START-HERE.bat --setup --family unsloth --model UD-Q4_K_XL
strata launch --family unsloth --model UD-Q4_K_XL
```
What setup does differently for this model:
## What setup does
- Every expert in RAM where its 72 GiB of experts and 16 GiB more fit the RAM (~88 GB: no `--resident-budget-gib`;
the rule is sergqwer's fork's, MIT), else the RAM budget below. It needs 48 GB of RAM or more (with less it asks,
default no; `--model UD-Q4_K_XL --yes` installs it anyway) and engine 0.1.32 or newer (checked before anything is
downloaded). NVIDIA or AMD: it ran on an R9700 too (#429's question is gone). One GPU by default in the budget mode
(the tested setup); on several, see the next point. No experimental speed projection (not tested with it). Images
(#967) are on by default on an NVIDIA card of 16 GB or more and on AMD (CPU encoder) where every expert fits in RAM;
run end to end on CUDA and HIP (QUANTS.md).
- Several GPUs, with an engine that keeps the resident RAM copy on a layer split (0.1.40 and newer, #642): the RAM
budget is kept on the split. Each card caches the experts of its own layers, and the budget holds the hottest of
the rest by the whole expert profile (the experts no card holds), so any RAM works as on one card. Setup asks (one
GPU stays the default; `--gpus 0,1`, "2" at setup or "y" when a start offers both cards takes the split), and a
start on several GPUs keeps `--resident-budget-gib`. Measured on an RTX 3090 (x16) + RTX 3080 Ti (x4), 91 GB of RAM,
budget 64 GiB, engine 0.1.41: decode 45 tok/s against 38 on the 3090 alone (the auto split gives the x4 card two
layers); 57.6 GiB of experts in RAM, no file reads per request.
- Several GPUs with an older engine (#498): when the RAM holds the GGUF files and 24 GB more (~135 GB of RAM) and two or more cards can
share it, setup asks (one GPU stays the default; `--gpus 0,1` takes the split). The split runs **without** the RAM budget: all 77 GB of experts are loaded into RAM from the GGUFs at
start, the files pass through the OS file cache while they load, and the config gets `"gpu": [0, 1]` and
`"layer_split": "auto"`. Measured on 2x RTX 3090 with 165 GiB (#498): decode 31 tok/s on one card with the budget,
64-78 tok/s split (55 tok/s at a 128K prompt), with `MemAvailable` never under 68 GiB. With less RAM, `--gpus`
keeps one GPU and says so, and `START-HERE.bat --gpus 0,1` on an installed UD-Q4_K_XL stops with the reason
(it used to keep the budget, and the engine exited with code 2).
- It downloads the four shards below from the pinned revision `38bb39e` (resumable, like the other models), then
checks each one's size and SHA-256 against the table below; the check takes a few minutes once and is remembered
in the file's finish mark. A file with the wrong hash is deleted, so the next run downloads it again.
- It packs with `--compat-bf16` and never writes `experts.bin` (`--low-ram` does not apply).
- The RAM budget (below ~88 GB of RAM) is the PC's RAM less 24 GB: `--resident-budget-gib 40` on 64 GB, at most all 71 GiB of experts on
a PC that asks for a budget. From a 64K context up, where the KV cache moves to RAM, its size comes out of the budget. Setup's
`--resident-budget-gib N` sets another one (a bigger one is kept, with a note on what it risks).
- It recommends an 8K context on a GPU under 14 GB (every GB of KV cache is a GB less of cached experts).
- **Where the experts live.** Every expert goes into RAM when its 72 GiB plus 16 GiB more fit (about 88 GB of RAM). On a
PC with less RAM, or on a unified-memory APU, setup keeps a RAM budget: the hottest experts in RAM, the rest read
from the GGUF on the SSD. It needs 48 GB of RAM or more. With less it asks (default no), and
`--model UD-Q4_K_XL --yes` installs it anyway.
- **The RAM budget** is the PC's RAM less 24 GB (`--resident-budget-gib 40` on 64 GB), at most all 71 GiB of experts. From
a 64K context up, where the KV cache moves to RAM, its size comes out of the budget. `--resident-budget-gib N` at
setup sets another one; a bigger one is kept, with a note on what it risks.
- **Download.** The four shards below from the pinned revision `38bb39e`, resumable. Setup checks each one's size and
SHA-256 against the table below once and remembers it in the file's finish mark. A file with the wrong hash is
deleted, so the next run downloads it again.
- **Packing.** `--compat-bf16`, and never an `experts.bin` (`--low-ram` does not apply).
- **Defaults it writes** into the run config's `env`: `STRATA_HC_Q8=1` on CUDA (the GGUF's own Q8_0 hyper-connection
projections, closer to the file and faster), and `STRATA_ROUTE_TAIL_SKIP=0` where every expert is in RAM. UD-Q4_K_XL
keeps `STRATA_MMQ_KQ` opt-in.
- **Context.** It recommends 8K on a GPU under 14 GB, because every GB of KV cache is a GB less of cached experts.
- **GPUs.** NVIDIA or AMD. One GPU is the default in the budget mode. With several GPUs, each card caches the experts of
its own layers and the budget holds the hottest of the rest by the whole expert profile. Setup asks; `--gpus 0,1`
takes the split, and a start on several GPUs keeps `--resident-budget-gib`.
- **Pictures.** The reader is the CPU one below 24 GB of VRAM and the GPU one from 24 GB, when the package has a reader.
No experimental speed projection is used with this file.
## The files
Four shards, 111,334,654,784 bytes (111.3 GB) together. Shard 1 holds only the metadata; layer 11's down projection is
Four shards, 111,334,654,784 bytes (111.3 GB) together. Shard 1 holds only the metadata. Layer 11's down projection is
in shard 2 and its gate/up in shard 3, which this engine handles (`native_experts.txt` v4).
| File | Bytes | SHA-256 |
@@ -64,23 +56,22 @@ in shard 2 and its gate/up in shard 3, which this engine handles (`native_expert
| `Qwen3.8-Flash-Next-UD-Q4_K_XL-00003-of-00004.gguf` | 49,376,141,504 | `56758f40269cad5cd9b0d3d6fbae0f40f6d5be6de49e4ab392dbe83157d9cbd3` |
| `Qwen3.8-Flash-Next-UD-Q4_K_XL-00004-of-00004.gguf` | 12,087,983,520 | `753bda48b98ba4f1636134a90a967de1b2d3908a236c026e464777342e53510a` |
Keep the four files together in one folder under these names (the engine finds shards 2-4 from shard 1's name; a
missing shard is an error that names it). Setup checks them itself; by hand, check them before packing
(`sha256sum -c`, or `Get-FileHash` on Windows).
Hugging Face snapshot symlinks work if you pass the snapshot's file name, not the hash-named blob it points to.
Keep the four files together in one folder under these names. The engine finds shards 2 to 4 from shard 1's name, and a
missing shard is an error that names it. By hand, check the files before packing (`sha256sum -c`, or `Get-FileHash` on
Windows). Hugging Face snapshot symlinks work if you pass the snapshot's file name, not the hash-named blob it points to.
## What it needs
| | |
| --- | --- |
| Disk | 111.3 GB for the four files, 1.4 GB for the pack, ~6 GB for the MTP draft layer if you have none yet. **No `experts.bin`**: the engine reads the experts from the GGUF files in place. An NVMe SSD matters: the experts that fit neither VRAM nor the RAM budget are read from it for every token. |
| RAM | 64 GB measured. The RAM budget (below) holds the most-used experts; the rest come from the SSD through the OS file cache. |
| GPU | 12 GB measured (RTX 5070): after the weights, the draft layer and the buffers, the expert cache held 1,280 of the 24,576 experts (3.7 GiB) at 4K context. |
| Disk | 111.3 GB for the four files, 1.4 GB for the pack, about 6 GB for the MTP draft layer if you have none yet. There is **no `experts.bin`**: the engine reads the experts from the GGUF files in place. An NVMe SSD matters, because experts that fit neither VRAM nor the RAM budget are read from it for every token. |
| RAM | 48 GB minimum; about 88 GB for every expert in RAM. The RAM budget holds the most-used experts and the rest come from the SSD through the OS file cache. |
| GPU | 12 GB works: after the weights, the draft layer and the buffers, the expert cache held 1,280 of the 24,576 experts (3.7 GiB) at 4K context on an RTX 5070. |
## The pack (by hand)
Build Strata (or install 0.1.31+), then, from the repository root, with gguf-py from the pinned llama.cpp
(setup installs it; `STRATA_GGUF_PY` can point to its `gguf-py` folder):
Build Strata, then from the repository root, with gguf-py from the pinned llama.cpp (setup installs it;
`STRATA_GGUF_PY` can point to its `gguf-py` folder):
```sh
.venv/bin/python tools/iq_pack.py \
@@ -88,36 +79,40 @@ Build Strata (or install 0.1.31+), then, from the repository root, with gguf-py
--out packs/ud-q4_k_xl --compat-bf16
```
About 30 seconds. It writes `index.txt`, `dense.bin` (1.4 GB), `native_experts.txt`, the tokenizer, `conversions.json`
and `compat-bf16.json`. **`--compat-bf16` is required**: this file stores 195 small projections (the hyper-connection
up/down matrices, `output_hc_*` and the PLE value) as Q8_0, while the engine reads them as BF16. Without the flag the
packer refuses and writes nothing; with it, they are dequantized and rounded to BF16 (nearest-even). Do not add
`--experts-bin`: it would write a 77 GB copy of the experts that this mode does not need.
It takes about 30 seconds and writes `index.txt`, `dense.bin` (1.4 GB), `native_experts.txt`, the tokenizer,
`conversions.json` and `compat-bf16.json`. **`--compat-bf16` is required.** This file stores 195 small projections (the
hyper-connection up/down matrices, `output_hc_*` and the PLE value) as Q8_0, while the engine reads them as BF16.
Without the flag the packer refuses and writes nothing. With it, they are dequantized and rounded to BF16
(nearest-even). Do not add `--experts-bin`: it would write a 77 GB copy of experts that this mode does not need.
## The numbers: what is native and what is converted
The routed experts, the token embedding, the output head, the attention and shared-expert projections, the PLE key and
the PLE table are used as the GGUF stores them (the engine has GPU kernels and CPU (ggml) paths for Q4_K, Q5_K, Q5_1 and
Q8_0). The small tensors the engine reads as floats are converted, and `conversions.json` lists every one with its
source/destination type, method, whether it is exact, the largest absolute error and the source bytes' SHA-256:
264 F32 tensors (routers, injections, SSM gates) whose values are already BF16's are stored as BF16 exactly; the 195
Q8_0 projections above are rounded to BF16 (largest absolute error 0.0144, in `hc_ffn_up`); the PLE convolution (F32)
is narrowed to F16 when the engine loads it (largest error 3e-8). That is the compatibility Unsloth's file needs; it is
not the original BF16 checkpoint. The output matches llama.cpp's on the same file:
[Quality](#quality-against-llamacpp-on-the-same-file).
the PLE table are used as the GGUF stores them. The engine has GPU kernels and CPU (ggml) paths for Q4_K, Q5_K, Q5_1 and
Q8_0.
The small tensors the engine reads as floats are converted. `conversions.json` lists every one with its source and
destination type, method, whether it is exact, the largest absolute error and the source bytes' SHA-256:
- 264 F32 tensors (routers, injections, SSM gates) whose values are already BF16's are stored as BF16 exactly.
- The 195 Q8_0 projections above are rounded to BF16 (largest absolute error 0.0144, in `hc_ffn_up`).
- The PLE convolution (F32) is narrowed to F16 when the engine loads it (largest error 3e-8).
That is the compatibility Unsloth's file needs. It is not the original BF16 checkpoint. The output matches llama.cpp's on
the same file ([Quality](#quality-against-llamacpp-on-the-same-file)).
## The MTP draft layer
The draft layer is the base model's, the same one the other packs use. If setup installed Strata, you have it
(`mtp/rt` in `Strata-data`). Otherwise build it as [docs/ORCA.md](ORCA.md#preparation) shows (`tools/mtp_fetch.py`,
The draft layer is the base model's, the same one the other packs use. If setup installed Strata, you have it (`mtp/rt`
in `Strata-data`). Otherwise build it as [ORCA.md](ORCA.md#preparation) shows (`tools/mtp_fetch.py`,
`tools/mtp_pack.py`, `tools/mtp_rt.py`, then copy `data/draft_vocab.bin` to `mtp/rt/`).
## The server
Save as `strata-ud-q4_k_xl.json` at the repository root (or next to setup's other `strata-*.json`), with `/path/to/`
pointing at **shard 1**. `--ple-gguf` is not needed: the engine finds the shard that holds the PLE table
(`per_layer_token_embd.weight`, shard 2) by name. `--resident-budget-gib` turns on the mapped mode with a RAM budget
(it implies `--mmap-experts`).
(`per_layer_token_embd.weight`, shard 2) by name. `--resident-budget-gib` turns on the mapped mode with a RAM budget (it
implies `--mmap-experts`).
```json
{
@@ -143,61 +138,35 @@ pointing at **shard 1**. `--ple-gguf` is not needed: the engine finds the shard
.venv/bin/python -m serve.server --engine strata --config strata-ud-q4_k_xl.json --port 8080
```
(`engine/strata.exe` is setup's engine on Windows; a self-built one is `build/strata`.) The expert profile is the base
model's (`data/expert-profile.bin`); it ranks the same 48 x 512 experts and decides which ones the GPU cache and the RAM
`engine/strata.exe` is setup's engine on Windows; a self-built one is `build/strata`. The expert profile is the base
model's (`data/expert-profile.bin`). It ranks the same 48 x 512 experts and decides which ones the GPU cache and the RAM
budget take first.
**Choosing the RAM budget N:** your RAM minus 20-24 GB (the OS, the engine itself, the 126 MB router copy, and room for
the OS file cache that serves the rest). On 64 GB, 40. The engine checks available RAM and, on Windows, available
commit capacity. If N exceeds the smaller value minus 4 GiB headroom, it reduces N to that amount minus a 256 MiB
margin and reports which limit applies. Before #403's fix a clamped budget could then fail the safety check.
Everything above N comes from the SSD for every token, so N is the setting that matters
most; a bigger budget was faster in every measurement (24 / 32 / 40 GiB). When the driver page-locks the whole budget
(24 GiB did on this PC, 32 and 40 did not), the GPU also computes a share of the misses over PCIe, as in the resident
low-RAM mode; `STRATA_RESIDENT_PIN=0` uses ordinary pageable memory (the CPU then computes every miss).
**Choosing the RAM budget N.** Use your RAM minus 20 to 24 GB (the OS, the engine itself, the 126 MB router copy, and
room for the OS file cache that serves the rest). On 64 GB that is 40. The engine checks available RAM and, on Windows,
available commit capacity. If N exceeds the smaller value minus 4 GiB headroom, it reduces N to that amount minus a
256 MiB margin and reports which limit applies. Everything above N comes from the SSD for every token, so N is the
setting that matters most; a bigger budget was faster in every measurement (24 / 32 / 40 GiB).
On Windows, the allocation log reports available physical RAM and commit capacity before allocation and after a
failure. Free RAM alone does not guarantee that Windows can commit the requested memory. If the pageable fallback
also fails, the warning preserves both the GPU allocation or mapping error and the `malloc` error.
When the driver page-locks the whole budget, the GPU also computes a share of the misses over PCIe, as in the resident
low-RAM mode. `STRATA_RESIDENT_PIN=0` uses ordinary pageable memory instead, and the CPU then computes every miss. On
Windows the allocation log reports available physical RAM and commit capacity before allocation and after a failure,
because free RAM alone does not guarantee that Windows can commit the requested memory.
**Context:** measured at 4K. The KV cache takes VRAM from the expert cache, so a longer context makes decoding slower;
8K is a reasonable start on 12 GB. `--kv int8` halves the KV cache's VRAM.
## What to expect (RTX 5070 12 GB, 64 GB RAM, Windows, one short greedy prompt)
| RAM budget | Greedy output | Notes |
| --- | --- | --- |
| 24 GiB | 5.1-5.2 tok/s (2 runs, before the routing prefetch below) | 2.2 GB read from the SSD per verify round (~3.5 tokens) |
| 40 GiB | **7-8.5 tok/s** (6 runs: 6.7-8.6, mean 7.9) | 1.1 GB read from the SSD per verify round, ~0.33 GB per token |
- The first answer comes about 60 s after the engine starts (mostly loading: the RAM budget is copied from the GGUF
files at start). A 30-token prompt then took 5-6 s.
- Long prompts (engine 0.1.32): a 16K prompt is read at **160 tokens/s** (15,873 tokens in 99 s; 0.1.31: 57 tokens/s,
273-282 s). Reading a prompt streams every expert once per 4K chunk, and the third of them outside the RAM budget
come from the SSD (~35 GB per chunk), so the prompt path now reads them with 32 threads and up to 128 blobs in
flight when the experts are read from the GGUF in place (`STRATA_STAGER_THREADS` / `STRATA_STAGER_RING` override
it; 4 and 16 before, still the default for every other model). Between the chunks, the 28.8 GB PLE table's rows
take ~5 s per chunk.
- With `STRATA_MMQ_KQ=1` the engine multiplies the Q4_K / Q5_K / Q5_1 experts of a prompt with llama.cpp's MMQ
kernels (as the other models' formats always are) instead of dequantizing them to FP16: 4x less GPU time for those
products, but on this PC the prompt waits for the SSD either way (29 s per 4K chunk with or without). Opt-in: it
rounds differently from the FP16 path. On CUDA these kernels are a library next to the engine (`libstrata_kq.so` /
`strata_kq.dll`), opened only then, so they take no VRAM from other models ([QUANTS.md](QUANTS.md)); before, they were
the build option `-DSTRATA_MMQ_KQUANTS=ON` and took 1-3 expert slots from every model.
- The speed varies from run to run (6.7-8.6 tok/s at 40 GiB for the same prompt), with what the OS file cache holds.
- For comparison, IQ3_S (all its experts in RAM) writes ~53 tok/s on the same PC ([the speed tables](DETAILS.md#speed-measured)).
- Where the time goes (`--stats` on the command line): at 40 GiB about 550-750 ms of each verify round (3.5 tokens) is
reading experts from the SSD, ~75 ms the CPU's expert kernels, ~13 ms the GPU.
**Context.** The KV cache takes VRAM from the expert cache, so a longer context makes decoding slower. 8K is a reasonable
start on 12 GB. `--kv int8` halves the KV cache's VRAM.
## Quality: against llama.cpp on the same file
Strata and llama.cpp (the pinned commit `3cf0325`, a CPU build reading the GGUF memory-mapped) were given the same
token sequences, and at every position each wrote its 20 most likely next tokens with their log-probabilities: Strata
through its verify windows (`STRATA_LOGPOS` with `STRATA_LOGPOS_TOPK=20`), llama.cpp from `llama_get_logits_ith` over
one batch. Strata ran with the recommended settings (RAM budget 40 GiB, the MTP draft layer on, greedy,
`--adapt-every 100000` so the expert cache does not move during the run). Per position: whether the most likely token is the same (argmax
agreement), how many of the 5 / 10 most likely tokens both have, the KL divergence (llama.cpp's distribution against
Strata's, over llama.cpp's top 20 plus one bucket for the rest), and the perplexity of the sequence under each.
> **Measured on:** Strata 0.1.32 era engine against llama.cpp pinned commit `3cf0325` (CPU build, GGUF memory-mapped).
> Strata ran with RAM budget 40 GiB, the MTP draft layer on, greedy and `--adapt-every 100000`. Hardware is not recorded
> on this page.
Both were given the same token sequences, and at every position each wrote its 20 most likely next tokens with their
log-probabilities. Strata used its verify windows (`STRATA_LOGPOS` with `STRATA_LOGPOS_TOPK=20`), llama.cpp used
`llama_get_logits_ith` over one batch. Per position: whether the most likely token is the same (argmax agreement), how
many of the 5 and 10 most likely tokens both have, the KL divergence (llama.cpp's distribution against Strata's, over
llama.cpp's top 20 plus one bucket for the rest), and the perplexity of the sequence under each.
| Token set | Positions | Argmax agreement | Top-5 / top-10 overlap | KL | Perplexity Strata / llama.cpp |
| --- | ---: | ---: | ---: | ---: | ---: |
@@ -207,42 +176,44 @@ Strata's, over llama.cpp's top 20 plus one bucket for the rest), and the perplex
| Real text after a 16K prompt: the last 510 tokens of 16,384 tokens of this repository's docs and code | 510 | 91.2% | 88.3% / 89.9% | 0.035 | 4.27 / 4.29 |
| The same 511 positions with only 512 tokens before them | 511 | 89.2% | 86.0% / 86.3% | 0.061 | 8.99 / 8.87 |
For comparison, [eddoursul/Strata](https://github.com/eddoursul/Strata) reported 98.2-99.3% (code), 95.6-95.8%
For comparison, [eddoursul/Strata](https://github.com/eddoursul/Strata) reported 98.2 to 99.3% (code), 95.6 to 95.8%
(thinking) and 97.7% (after a 16K prompt) for the same file against llama.cpp on an RTX 3090.
**Where they differ:** at near-ties. Grouped by llama.cpp's own gap between its two most likely tokens, Strata picks
the same token at every position of the code and thinking answers where that gap is 0.2 or more (695 of 695) and at
every position after the 16K prompt where it is 0.5 or more (144 of 144 in the continuation, 254 of 254 in the real
text; 50 of 54 and 117 of 121 between 0.2 and 0.5). Text after a long document has many more near-ties than a short
answer (58 of the continuation's 256 positions and 135 of the real text's 510 have a gap under 0.2, against 13 of the
code answer's 400), which is why its agreement is lower. It is not the long context itself: the same real-text
positions after only 512 tokens agree a little less (89.2%), and the perplexities match within 3% in every set. For
scale, Strata's own greedy run and its teacher-forced rerun of the continuation pick different tokens at 6% of the
positions (93.8% the same: other window sizes, other experts in VRAM), the same kind of near-tie flips. The 16K
prompts were read by Strata's batched prompt path (15,872 tokens), the rest through the verify windows. These numbers
are the FP16 prompt path's (the released engine's); with the MMQ prompt path (`-DSTRATA_MMQ_KQUANTS=ON`, above) the
The two differ at near-ties. Where llama.cpp's own gap between its two most likely tokens is 0.2 or more, Strata picks
the same token at every position of the code and thinking answers (695 of 695). After the 16K prompt it does so at every
position where the gap is 0.5 or more (144 of 144 in the continuation, 254 of 254 in the real text). Text after a long
document has many more near-ties than a short answer (58 of the continuation's 256 positions and 135 of the real text's
510 have a gap under 0.2, against 13 of the code answer's 400), which is why its agreement is lower. The perplexities
match within 3% in every set. The 16K prompts were read by Strata's batched prompt path (15,872 tokens), the rest
through the verify windows. These numbers are the FP16 prompt path's. With the MMQ prompt path (`STRATA_MMQ_KQ=1`) the
continuation after the 16K prompt agrees at 92.2% (KL 0.029, three runs, identical).
To repeat it: Strata's side is `STRATA_LOGPOS=<file>` with `STRATA_LOGPOS_TOPK=20` (engine 0.1.32) on a serve
engine with `--short-read` covering the positions to compare; llama.cpp's side was a short program over
`llama_decode` / `llama_get_logits_ith` writing the same top 20 per position.
To repeat it, set `STRATA_LOGPOS=<file>` with `STRATA_LOGPOS_TOPK=20` on a serve engine with `--short-read` covering the
positions to compare. llama.cpp's side was a short program over `llama_decode` and `llama_get_logits_ith` writing the
same top 20 per position.
## Opt-ins and switches (environment variables)
| | |
| Variable | Effect |
| --- | --- |
| `STRATA_LOOKAHEAD=0` | Turns off the routing-aware prefetch (on by default in this mode): while the CPU works on a layer, a thread applies the next layer's router to this layer's input and asks the OS to read the predicted experts' pages. Pages only, the answers are the same. About half of the SSD reads were predicted; mean +14% (6.9 -> 7.9 tok/s). `STRATA_LOOKAHEAD_K` sets the experts per token (default 10). |
| `STRATA_LOOKAHEAD=0` | Turns off the routing-aware prefetch (on by default in this mode). While the CPU works on a layer, a thread applies the next layer's router to this layer's input and asks the OS to read the predicted experts' pages. Pages only; the answers are the same. `STRATA_LOOKAHEAD_K` sets the experts per token (default 10). |
| `STRATA_MMQ_KQ=1` | Multiplies the Q4_K / Q5_K / Q5_1 experts of a prompt with llama.cpp's MMQ kernels instead of dequantizing them to FP16. It rounds differently from the FP16 path, so it is opt-in for this file. On CUDA the kernels are a library next to the engine (`libstrata_kq.so` / `strata_kq.dll`), opened only then, so they take no VRAM from other models ([QUANTS.md](QUANTS.md)). |
| `STRATA_KQ256=1` | Multi-token AVX2 kernels for the Q4_K / Q5_1 / Q8_0 experts. Bit-exact with ggml's, but measured no faster, so off. |
| `STRATA_PARTIAL_PIN=1` | Registers the hottest part of the RAM budget (up to `STRATA_PARTIAL_PIN_GIB`, default 24) with the GPU driver, so the GPU computes a share of the misses over PCIe (`--pcie-frac`). Measured no faster on this PC, and it changes the numerics of those experts (GPU kernels instead of the CPU's), so off. |
| `STRATA_FETCH_THREADS=N` | Threads that read the experts from the GGUF while it answers (default 8; 16 was no faster). The prompt path has its own: `STRATA_STAGER_THREADS` (default 32 here) and `STRATA_STAGER_RING` (128). |
| `STRATA_PARTIAL_PIN=1` | Registers the hottest part of the RAM budget (up to `STRATA_PARTIAL_PIN_GIB`, default 24) with the GPU driver, so the GPU computes a share of the misses over PCIe (`--pcie-frac`). Measured no faster, and it changes the numerics of those experts, so off. |
| `STRATA_FETCH_THREADS=N` | Threads that read the experts from the GGUF while it answers (default 8). The prompt path has its own: `STRATA_STAGER_THREADS` (default 32 here) and `STRATA_STAGER_RING` (128). |
| `STRATA_RESIDENT_PIN=0` | Ordinary pageable memory for the RAM budget (see above). |
## UD-IQ4_XS (setup from 0.1.39, #621)
## UD-IQ4_XS
Unsloth's `UD-IQ4_XS` at the same revision is in setup too, as a regular choice (not experimental; the Unsloth
family's first size and its default): `START-HERE.bat --setup --family unsloth --model UD-IQ4_XS` (engine 0.1.38 or
newer). It sits between IQ3_S and UD-Q4_K_XL: its routed experts are IQ3_S gate/up (IQ4_XS
in one layer) with IQ4_NL downs (Q8_0 in five layers), 59.5 GB of them; the dense side (Q8_0 projections, the IQ4_NL
PLE table, a Q6_K head) is UD-Q4_K_XL's. Three shards, 93.7 GB:
Unsloth's `UD-IQ4_XS` at the same revision is in setup as a regular choice, the Unsloth family's first size:
```sh
strata launch --family unsloth --model UD-IQ4_XS
```
It sits between IQ3_S and UD-Q4_K_XL. Its routed experts are IQ3_S gate/up (IQ4_XS in one layer) with IQ4_NL downs (Q8_0
in five layers), 59.5 GB of them. The dense side (Q8_0 projections, the IQ4_NL PLE table, a Q6_K head) is UD-Q4_K_XL's.
Three shards, 93.7 GB:
| File | Bytes | SHA-256 |
| --- | ---: | --- |
@@ -250,37 +221,38 @@ PLE table, a Q6_K head) is UD-Q4_K_XL's. Three shards, 93.7 GB:
| `Qwen3.8-Flash-Next-UD-IQ4_XS-00002-of-00003.gguf` | 49,835,229,856 | `577a38a2392b40ca2193cea502e1d92f60b8cd370675d308e0ec21885d9daaa7` |
| `Qwen3.8-Flash-Next-UD-IQ4_XS-00003-of-00003.gguf` | 43,836,407,744 | `d4634e6d84f0ebb0940be15c90d3790bf6464e3dea3a1cddc567dc0e83ad8833` |
Setup treats it like UD-Q4_K_XL (the list above): the same RAM budget (your RAM less 24 GB; at most all 55 GiB of its
experts, so a PC with ~80 GB of RAM or more holds all of them), the pack with `--compat-bf16`, no `experts.bin`, one
GPU by default (several keep the budget, as above). It does not ask on AMD: its experts' formats have prompt kernels there too. Images are an option (asked,
off by default; NVIDIA): the image path has no restriction for this pack, which uses the original model's image
encoder; not yet run with images.
Setup treats it like UD-Q4_K_XL: the same RAM budget rule (your RAM less 24 GB, at most all 55 GiB of its experts, so a
PC with about 80 GB of RAM or more holds all of them), the pack with `--compat-bf16`, no `experts.bin`, and one GPU by
default. It uses the original model's image encoder.
What is known so far: it packs and runs on a Strix Halo (AMD gfx1151, 128 GB unified memory), where prompts read at
~820 tokens/s at 8K and 64K and the answers passed the retrieval checks at 8K and 64K. It has **not** been measured on
NVIDIA yet; please report what you see. One fidelity note that applies to every `--compat-bf16` pack (UD-Q4_K_XL too):
the pack rounds the file's Q8_0 hyper-connection projections to BF16, and on the Strix Halo that rounding put
perplexity 6-9% above llama.cpp's on the same file (teacher-forced, short context). Reading the Q8_0 values instead
closes most of that gap; it is measured on AMD only and not in this release.
It packs and runs on a Strix Halo (AMD gfx1151, 128 GB unified memory), where prompts read at about 820 tokens/s at 8K
and 64K and the answers passed the retrieval checks at 8K and 64K. It has not been measured on NVIDIA; please report
what you see.
One fidelity note applies to every `--compat-bf16` pack, UD-Q4_K_XL included. The pack rounds the file's Q8_0
hyper-connection projections to BF16, and on the Strix Halo that put perplexity 6 to 9% above llama.cpp's on the same
file (teacher-forced, short context). `STRATA_HC_Q8=1`, which setup writes on CUDA for the UD files, reads the file's Q8_0
values instead of the rounded BF16. The 6 to 9% gap was measured on AMD, where setup does not write it.
## Q4_0 files
A plain `llama-quantize` Q4_0 file of this model runs the same way: pack shard 1 with `--compat-bf16` as above and
point `--native` at it. Tested:
A plain `llama-quantize` Q4_0 file of this model runs the same way: pack shard 1 with `--compat-bf16` as above and point
`--native` at it. Tested:
[bartowski/Qwen3.8-Flash-Next-GGUF, `Qwen3.8-Flash-Next-Q4_0`, revision `928589f`](https://huggingface.co/bartowski/Qwen3.8-Flash-Next-GGUF/tree/928589fdb66c6ff07f22ac561e3fbce76553548f/Qwen3.8-Flash-Next-Q4_0)
(3 shards, 100.6 GB). Its routed experts are Q4_0 for gate/up and for down, except down in layers 0-5, which is Q4_1;
the PLE table and the token embedding are Q4_0. The pack serves 302 tensors from the GGUF as stored and converts 362
(3 shards, 100.6 GB). Its routed experts are Q4_0 for gate/up and for down, except down in layers 0 to 5, which is Q4_1.
The PLE table and the token embedding are Q4_0. The pack serves 302 tensors from the GGUF as stored and converts 362
(168 exact, 194 rounded to BF16, largest absolute error 0.0156; 1.23 GiB).
The Q4_0 expert kernels centre the codes (q - 8) before the integer dot product, as ggml-cpu's Q4_0 x Q8_0 dot does,
instead of llama.cpp's CUDA form (uncentred codes, minus 8 x the Q8_1 block sum in float); Q4_1 follows the Q5_1
kernel without the fifth bit. `native_expert_parity` on this file (layers 0 and 5: Q4_0/Q4_1, layers 20 and 47:
Q4_0/Q4_0): relative error against the float product 1.1-1.3e-2 on the CPU and the GPU alike, CPU vs GPU 1.1e-7 for
Q4_0/Q4_0 and 5e-4 to 1.1e-3 with Q4_1 down.
instead of llama.cpp's CUDA form (uncentred codes, minus 8 x the Q8_1 block sum in float). Q4_1 follows the Q5_1 kernel
without the fifth bit. `native_expert_parity` on this file (layers 0 and 5: Q4_0/Q4_1, layers 20 and 47: Q4_0/Q4_0)
gives a relative error against the float product of 1.1 to 1.3e-2 on the CPU and the GPU alike, and CPU against GPU of
1.1e-7 for Q4_0/Q4_0 and 5e-4 to 1.1e-3 with Q4_1 down.
Measured on one RTX 3060 12 GB (PCIe 3.0 x16), Xeon E5-2696 v4, 121 GiB DDR4-2400, Linux, CUDA 12.4; every expert in
RAM, `--max-context 131072 --kv int8 --kv-resident 20480`, expert cache `auto` (1,960 slots, 5.06 GiB), prompt chunk
`auto` (6,144 tokens); greedy, `reasoning_effort` medium, one request at a time:
> **Measured on:** one RTX 3060 12 GB (PCIe 3.0 x16), Xeon E5-2696 v4, 121 GiB DDR4-2400, Linux, CUDA 12.4. Every
> expert in RAM, `--max-context 131072 --kv int8 --kv-resident 20480`, expert cache `auto` (1,960 slots, 5.06 GiB),
> prompt chunk `auto` (6,144 tokens), greedy, `reasoning_effort` medium, one request at a time. Strata version not
> recorded.
| Prompt | Prompt read | Output |
| --- | --- | --- |
@@ -289,28 +261,62 @@ RAM, `--max-context 131072 --kv int8 --kv-resident 20480`, expert cache `auto` (
| 24,461 tokens | 787 tok/s | 24.0 tok/s |
| 67,934 tokens (a fact hidden in the middle: found) | 737 tok/s | 26.4 tok/s |
A coding prompt was answered correctly. Not measured: agreement with llama.cpp on this file (as
[above](#quality-against-llamacpp-on-the-same-file) for UD-Q4_K_XL), AMD, Windows, and bartowski's Swift-1.5 Q4_0,
whose header shows the same formats.
A coding prompt was answered correctly. Not measured: agreement with llama.cpp on this file, AMD, Windows, and
bartowski's Swift-1.5 Q4_0, whose header shows the same formats.
## Scope and validation
- UD-Q4_K_XL and UD-IQ4_XS at revision `38bb39e` are targeted, and plain Q4_0 files ([above](#q4_0-files)). Other
quantizations use formats this engine may not have kernels for; the engine checks every layer's formats at start
and refuses an unsupported one by name.
- Tests: the packer's synthetic 4-shard and conversion tests (`.venv/bin/python -m unittest discover -s tools -p
test_iq_pack.py`); CTests `gguf_split_test`, `expert_layout_test`, `native_expert_parity_*` (the three real expert
format pairs against ggml-cpu, the Q5_1 min term, Q8_0 rows, Q4_0/Q4_0 and Q4_0/Q4_1), `prefill_mmq_kquant_test` (with `-DSTRATA_MMQ_KQUANTS=ON`: the
prompt path's MMQ products for Q4_K / Q5_K / Q5_1 / Q8_0 against ggml's dequantized weights); the in-place mode against `experts.bin` on the Coder
(identical tokens and logits).
- Real runs: greedy answers to a coding prompt (correct) at every budget and setting above, identical across them;
the same tokens as llama.cpp at 97.5-99% of the positions of short greedy answers and 90-91% after a 16K prompt,
differing mostly at near-ties (above). Not yet: sampled decoding, long conversations. Please report what you see.
- UD-Q4_K_XL and UD-IQ4_XS at revision `38bb39e` are targeted, and plain Q4_0 files. Other quantizations use formats this
engine may not have kernels for. The engine checks every layer's formats at start and refuses an unsupported one by
name.
- Tests: the packer's synthetic 4-shard and conversion tests
(`.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`) and the CTests `gguf_split_test`,
`expert_layout_test`, `native_expert_parity_*` (the three real expert format pairs against ggml-cpu, the Q5_1 min term,
Q8_0 rows, Q4_0/Q4_0 and Q4_0/Q4_1). With `-DSTRATA_MMQ_KQUANTS=ON`, `prefill_mmq_kquant_test` checks the prompt path's
MMQ products for Q4_K / Q5_K / Q5_1 / Q8_0 against ggml's dequantized weights.
- Not validated: sampled decoding and long conversations. Please report what you see.
## Credits
The split-artifact loader, the expert formats' GPU kernels and the packing of Unsloth's files follow
[eddoursul/Strata](https://github.com/eddoursul/Strata) (PR #245), which ran these files first. The Q8_0 kernels, the
per-role shard column and the PLE convolution's F32 -> F16 fix come from @gopinath87607's PR #255; per-role shard names
per-role shard column and the PLE convolution's F32 to F16 fix come from @gopinath87607's PR #255. Per-role shard names
were also proposed by @jagsan-cyber in PR #247. The quantization is [Unsloth](https://huggingface.co/unsloth)'s; the
model is Qwen's ([license](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)).
## History / older measurements
### RAM budget, speed on a 12 GB card
> **Measured on:** RTX 5070 12 GB, 64 GB RAM, Windows 10, AVX-512 CPU, UD-Q4_K_XL, one short greedy prompt, first
> validation (2026-10-01, engine 0.1.31/0.1.32).
| RAM budget | Greedy output | Notes |
| --- | --- | --- |
| 24 GiB | 5.1 to 5.2 tok/s (2 runs, before the routing prefetch) | 2.2 GB read from the SSD per verify round (about 3.5 tokens) |
| 40 GiB | 7 to 8.5 tok/s (6 runs: 6.7 to 8.6, mean 7.9) | 1.1 GB read from the SSD per verify round, about 0.33 GB per token |
- The first answer comes about 60 s after the engine starts (the RAM budget is copied from the GGUF files at start). A
30-token prompt then took 5 to 6 s.
- A 16K prompt was read at 160 tokens/s (15,873 tokens in 99 s) on engine 0.1.32, up from 57 tokens/s (273 to 282 s) on
0.1.31, after the prompt path began reading experts with 32 threads and up to 128 blobs in flight.
- The routing-aware prefetch added a mean of +14% (6.9 to 7.9 tok/s); about half of the SSD reads were predicted.
- With `STRATA_MMQ_KQ=1` the products took 4x less GPU time, but on this PC the prompt waited for the SSD either way (29 s
per 4K chunk).
- Where the time goes at 40 GiB (`--stats`): about 550 to 750 ms of each verify round (3.5 tokens) reading experts from
the SSD, about 75 ms the CPU's expert kernels, about 13 ms the GPU.
- For comparison, IQ3_S (all its experts in RAM) wrote about 53 tok/s on the same PC
([the speed tables](DETAILS.md#speed-measured)).
### Two GPUs with the RAM budget
> **Measured on:** RTX 3090 (x16) + RTX 3080 Ti (x4), 91 GB RAM, budget 64 GiB, engine 0.1.41.
Decode 45 tok/s against 38 on the 3090 alone (the auto split gives the x4 card two layers); 57.6 GiB of experts in RAM,
no file reads per request.
> **Measured on:** 2x RTX 3090, 165 GiB RAM, an older engine that ran the split without the RAM budget (all 77 GB of
> experts loaded into RAM at start).
Decode 31 tok/s on one card with the budget, 64 to 78 tok/s split (55 tok/s at a 128K prompt), with `MemAvailable` never
under 68 GiB.
docs/UNSLOTH_Q6.md
+62 -87
View File
@@ -1,24 +1,27 @@
# Unsloth UD-Q6_K_XL
**In setup since MODEL-UPDATE Phase 2** (`START-HERE.bat --setup --family unsloth --model UD-Q6_K_XL`, engine 0.1.44 or
newer: every expert in RAM from ~118 GB, else it asks and keeps a RAM budget; quality, speed and images in
[QUANTS.md](QUANTS.md); UD-Q5_K_XL and Q8_0 likewise). The manual workflow below still works. This is Unsloth's 6-bit
file of the same model the other packs use:
[unsloth/Qwen3.8-Flash-Next-GGUF, `UD-Q6_K_XL`, revision `38bb39e`](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q6_K_XL).
Build it for engine 0.1.38 or newer (earlier engines read `per_layer_token_embd.weight` in IQ4_NL, Q5_0 or FP8 only,
below). UD-Q4_K_XL's manual workflow asks nothing of setup, and neither does this one.
This page is for people who want to run Unsloth's 6-bit Qwen3.8-Flash-Next file with Strata, and for people who pack it
by hand. UD-Q5_K_XL and Q8_0 are set up the same way. Quality, speed and images across the Unsloth sizes are in
[QUANTS.md](QUANTS.md). UD-Q4_K_XL, with the full setup details, is in [UNSLOTH_Q4.md](UNSLOTH_Q4.md).
> **Status (MODEL-UPDATE Phase 1).** The Q6_K gate/up kernels are part of every build now (they were the opt-in
> `-DSTRATA_Q6K_EXPERTS=ON`): measured to leave other models' expert-cache slots unchanged ([QUANTS.md](QUANTS.md)). The
> MMQ prompt kernels for Q6_K come with the K-quant library and `STRATA_MMQ_KQ=1`; without it the prompt takes the FP16
> path. The file runs on the standard engine (AMD R9700: the quality yardstick in [QUANTS.md](QUANTS.md)).
The file is [unsloth/Qwen3.8-Flash-Next-GGUF, `UD-Q6_K_XL`, revision `38bb39e`](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q6_K_XL),
the same model the other packs use.
## Quick start
```sh
strata launch --family unsloth --model UD-Q6_K_XL
```
Setup keeps every expert in RAM from about 118 GB of RAM. Below that it asks and keeps a RAM budget, as for UD-Q4_K_XL
([What setup does](UNSLOTH_Q4.md#what-setup-does)). For UD-Q5_K_XL and UD-Q6_K_XL it writes `STRATA_MMQ_KQ=1`, the
K-quant MMQ prompt kernels, into the run config.
## What is in the file
Six shards, 169,165,382,688 bytes (169.2 GB) together ([sizes and SHA-256 from the HF tree API](https://huggingface.co/api/models/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q6_K_XL);
shards 1, 2 and 6 were also checked bit for bit against their Hugging Face LFS hashes after download). Shard 1 holds
only metadata; shard 2 only the output head (`output.weight`, `output_hc_*`, all Q8_0 but `output_hc_norm`, F32);
shard 3 is exactly the PLE table, nothing else.
Six shards, 169,165,382,688 bytes (169.2 GB) together ([sizes and SHA-256 from the HF tree API](https://huggingface.co/api/models/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q6_K_XL)).
Shard 1 holds only metadata. Shard 2 holds only the output head (`output.weight`, `output_hc_*`, all Q8_0 but
`output_hc_norm`, F32). Shard 3 is exactly the PLE table.
| File | Bytes | SHA-256 |
| --- | --- | --- |
@@ -29,30 +32,24 @@ shard 3 is exactly the PLE table, nothing else.
| `Qwen3.8-Flash-Next-UD-Q6_K_XL-00005-of-00006.gguf` | 49,419,317,344 | `9948e81ae8368144b135a7403d8c0f2cf9a6c2f9f692313865855b2832e696c9` |
| `Qwen3.8-Flash-Next-UD-Q6_K_XL-00006-of-00006.gguf` | 14,838,313,984 | `e4ae255234f42b18012f6d6eec5bf615b57479b7e0a0233c47436a321ffeec3f` |
Formats, read from the shard headers with `tools/gguf_reader.py` (all six shards verified: shard 4 = blocks 0-20,
520 tensors, shard 5 = blocks 20-41, 530 tensors, and per layer the types above):
Formats, read from the shard headers with `tools/gguf_reader.py`:
- **Routed experts:** `ffn_gate_exps` and `ffn_up_exps` are Q6_K (seen in every layer of shard 6: blocks 41-47);
`ffn_down_exps` is Q8_0 (per layer 891.3 MB at 640×2560×512, against Q6_K's 688.1 MB at 2560×640×512 for gate or
up). Gate and up are Q6_K in 47 of 48 layers and Q8_0 in layer 2; down is Q8_0 in every layer (the pack's
`native_experts.txt`: 47 rows of ggml types 14/8, and layer 2 is 8/8). Q6_K gate/up is native: the
engine multiplies them directly on the GPU in every path check below, and the CPU has the ggml block layout (210
bytes per 256 elements); the per-role layout in `native_experts.txt` v4 handles shards whose boundaries cut
through a layer like UD-Q4_K_XL's layer 11 case does.
- **Everything else:** attention and shared-expert projections, SSM, hyper-connections, the PLE key/value, the token
embedding and the output head are Q8_0, or F32/BF16 where the engine reads floats — the same pattern
- **Routed experts.** `ffn_gate_exps` and `ffn_up_exps` are Q6_K in 47 of 48 layers and Q8_0 in layer 2. `ffn_down_exps`
is Q8_0 in every layer (891.3 MB per layer at 640x2560x512, against Q6_K's 688.1 MB for gate or up). The pack's
`native_experts.txt` has 47 rows of ggml types 14/8, and layer 2 is 8/8. The engine multiplies Q6_K gate/up natively on
the GPU, and the CPU has the ggml block layout (210 bytes per 256 elements). The per-role layout in
`native_experts.txt` v4 handles shards whose boundaries cut through a layer.
- **Everything else.** Attention and shared-expert projections, SSM, hyper-connections, the PLE key/value, the token
embedding and the output head are Q8_0, or F32/BF16 where the engine reads floats. This is the same pattern
[UD-Q4_K_XL](UNSLOTH_Q4.md) handles with `tools/iq_pack.py --compat-bf16`.
- **The PLE table** (`per_layer_token_embd.weight`, [160, 320001536]): **Q8_0**, its own shard 3. A row is
5 blocks of 32 elements at 34 bytes = 170 bytes, and the whole table is 54.4 GB, against the 28.8 GB IQ4_NL of
UD-Q4_K_XL. Engine 0.1.38 reads Q8_0 rows natively: `PleTable` accepts the type, the row reader is already
parameterized by bytes-per-row (its selftest runs 90, 110 and 170 byte rows), and no GPU kernel changes —
`gather` outputs floats. (UD-Q4_K_XL's table is IQ4_NL; the FP8 table of [tools/ple_fp8_pack.py](../tools/ple_fp8_pack.py)
needs the original checkpoint, which this file is not a substitute for. Converting this Q8_0 table to FP8 would
trade table precision away for 6% smaller rows; Q8_0 read directly has none that conversion's error.)
- **The PLE table** (`per_layer_token_embd.weight`, [160, 320001536]) is **Q8_0**, in its own shard 3. A row is 5 blocks
of 32 elements at 34 bytes, 170 bytes in all, and the whole table is 54.4 GB (UD-Q4_K_XL's IQ4_NL table is 28.8 GB).
The engine reads Q8_0 rows natively. The FP8 table of [tools/ple_fp8_pack.py](../tools/ple_fp8_pack.py) needs the
original checkpoint, which this file is not a substitute for.
Q6_K gate/up costs 210 bytes per 256 elements, against Q4_K's 144, so the routed experts' share of the file grows
about 45% per element read; expected decode speed accordingly lower than UD-Q4_K_XL's on the same hardware. The
table row gather grows 170/90 = 1.89x over IQ4_NL.
Q6_K gate/up costs 210 bytes per 256 elements, against Q4_K's 144, so the routed experts' share of the file grows about
45% per element read. Decode is accordingly slower than UD-Q4_K_XL's on the same hardware. The table row gather grows
170/90 = 1.89x over IQ4_NL.
## The pack (by hand)
@@ -64,22 +61,18 @@ Same workflow as [UD-Q4_K_XL](UNSLOTH_Q4.md#the-pack-by-hand), from the reposito
--out packs/ud-q6_k_xl --compat-bf16
```
**`--compat-bf16` is required** for the same reason as there (the 195 small Q8_0 projections the engine reads as
BF16). Do not add `--experts-bin`; the experts stay in the GGUF files. Built here end to end on this file
(`packs/ud-q6_k_xl`, the packer's own tests cover the shard and conversion logic via
`.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`): 1079 tensors indexed, 303 served natively, 460
tensors converted (264 exact, 196 rounded, max abs error 0.0144 - the same profile as UD-Q4_K_XL's), 195
compat-BF16 tensors (1.20 GiB), arena 1.38 GiB, and `native_experts.txt` v4 with the per-role shard column on
layers 20 and 41.
**`--compat-bf16` is required**, for the same reason as there (the 195 small Q8_0 projections the engine reads as BF16).
Do not add `--experts-bin`; the experts stay in the GGUF files. The pack on the finished file has 1079 tensors indexed,
303 served natively, 460 converted (264 exact, 196 rounded, max abs error 0.0144), 195 compat-BF16 tensors (1.20 GiB),
an arena of 1.38 GiB, and `native_experts.txt` v4 with the per-role shard column on layers 20 and 41. The packer's tests:
`.venv/bin/python -m unittest discover -s tools -p test_iq_pack.py`.
## The server
Like UD-Q4_K_XL's [config](UNSLOTH_Q4.md#the-server), with the Q6 paths and names. `--ple-gguf` is not needed: the
engine finds shard 3 by the table's name. On a machine whose RAM holds the ~110 GB of routed experts plus the rest
(499 GB total, ~325 GB free, measured on the machine this was written on), there is no RAM budget and no
`--expert-cache` gymnastics: drop `--resident-budget-gib` and `--expert-cache-device1` from the Q4 config. The
54.4 GB table is mmap'd (the default) — do not switch it to `--ple-io ram` unless the RAM really is there to lock
it. `mtp/rt` is the base model's draft layer as always.
Like UD-Q4_K_XL's [config](UNSLOTH_Q4.md#the-server), with the Q6 paths and names. `--ple-gguf` is not needed: the engine
finds shard 3 by the table's name. On a machine whose RAM holds the roughly 110 GB of routed experts plus the rest, drop
`--resident-budget-gib` and `--expert-cache-device1` from the Q4 config. The 54.4 GB table is mapped (the default). Do
not switch it to `--ple-io ram` unless the RAM really is there to lock it. `mtp/rt` is the base model's draft layer.
```json
{
@@ -99,46 +92,28 @@ it. `mtp/rt` is the base model's draft layer as always.
}
```
On 64 GB machines treat the Q4 numbers below as the starting point: the RAM budget matters the same way, and
everything it leaves out comes from the SSD, faster now per expert (Q6_K) and per table row (170 bytes).
On a 64 GB machine, start from UD-Q4_K_XL's RAM-budget advice. The budget matters the same way, and everything it leaves
out comes from the SSD.
## What the engine did not have, and has now
## Kernels and tests
UD-Q4_K_XL's expert formats (gate/up Q4_K, down Q5_1) were in every path; Q6_K gate/up was not. This branch adds
it (the PLE Q8_0 reader is the other half):
- Decode windows: the grouped expert kernels (`native_expert_grouped`) take Q6_K gate/up (a pinned copy of llama.cpp's
`vec_dot_q6_K_q8_1` behind `Fmt<14>`). Per-token MMVQ (`native_mmvq`) has Q6_K too.
- Prompt path: llama.cpp's MMQ prompt kernels instantiate Q6_K in the K-quant library (`STRATA_MMQ_KQ=1`). Without it
the prompt takes the FP16 path.
- Tests: `native_grouped_parity` loops Q6_K/Q8_0 pairs, `prefill_mmq_kquant_test` runs Q6_K gate/up products at
1280x2560 against a double-precision reference, and `native_expert_parity --synthetic q6_K/q8_0` checks GPU and CPU
against ggml's dequantized weights. `ple_reader_test --selftest` runs 90, 110 and 170-byte PLE rows, and
`ple_q8_parity` matches ggml's reference dequantizer bit for bit through both Mmap and Direct.
- Not measured on this page: quality against llama.cpp on this file ([QUANTS.md](QUANTS.md) has the Unsloth quality
comparison).
- **Decode windows:** the grouped expert kernels (`native_expert_grouped`) take gate/up Q6_K — a pinned copy of
llama.cpp's `vec_dot_q6_K_q8_1` behind `Fmt<14>`, and the dequantizer (`dequantize_q6_K` folded onto the kernel's
32 threads). Per-token MMVQ (`native_mmvq`) already had Q6_K.
- **The prompt path:** llama.cpp's MMQ prompt kernels instantiate Q6_K (the K-quant library, `STRATA_MMQ_KQ=1`; it builds
`template-instances/mmq-instance-q6_k.cu` alongside q4_k/q5_k/q5_1; `moe_mmq` accepts and dispatches it).
- **The gates:** `native_expert_supported` took Q6_K/Q8_0 pairs (gate/up Q8_0 as in layer 2 was already in the
lists) — the engine previously refused the whole pack with "no GPU kernels for".
## History / older measurements
Tests: `native_grouped_parity` loops Q6_K/Q8_0 pairs; `prefill_mmq_kquant_test` runs Q6_K gate/up products at
1280×2560 against a double-precision reference; `native_expert_parity --synthetic q6_K/q8_0` checks GPU and CPU
against ggml's dequantized weights. All three are built (CUDA, `86;120`). They have not been run on a GPU: the
card was serving the file below.
> **Measured on:** Linux, RTX PRO 4500 Blackwell (compute capability 12.0) plus a second GPU, Threadripper PRO 5975WX
> (AVX2, no AVX-512), 30 expert-pool workers, 499 GB RAM, 2026-10-04. One code-review session.
## Status: what is verified and what is not
Verified here (test clone, Linux, NVIDIA GPU, source build):
- The six files' sizes and SHA-256 (HF tree API; shards 1, 2, 6 additionally against the downloaded blobs).
- The tensor inventory of shards 1, 2 and 6, read from the headers (`tools/gguf_reader.py`); shards 4 and 5's
tensor counts and per-layer expert types, from the pack's `conversions.json` and `native_experts.txt` v4.
- Q8_0 PLE rows: `ple_reader_test --selftest` runs 90, 110 and 170-byte rows (all OK); `ple_q8_parity` on a
synthetic Q8_0 table (`tools/ple_q8_fixture.py`) matches ggml's reference dequantizer bit for bit through both
Mmap and Direct (`max_abs 0.000e+00`). Both still pass on this branch with the Q6_K expert work in.
- The pack on the finished file (numbers above), and the engine binary builds (CUDA `86;120`, MMQ K-quants on).
- Q6_K expert source support: decode-side MMVQ was already there; the grouped verify-window kernels, the MMQ
prompt kernels and the `native_expert_supported` gate were not, and are in this branch (section above).
The server has run this file. On 2026-10-04, Linux, an RTX PRO 4500 Blackwell (compute capability 12.0) plus a
second GPU, a Threadripper PRO 5975WX (AVX2, no AVX-512), 30 expert-pool workers and 499 GB of RAM, engine 0.1.39
loaded 101.75 GiB of experts (8.43 GiB/s) and served one review. The cold read was 14,915 tokens at 3,176 tok/s.
Fourteen turns generated 5,799 tokens in 74.1 s (78.3 tok/s) and finished at 44,160 tokens of context; drafts were
accepted 3,485 of 4,406 (79.1%). The same machine on engine 0.1.38, before this branch was rebased onto 0.1.39,
read 249,193 tokens cold at 2,802 tok/s and then generated 1,079 tokens at 69.0 tok/s. Quality against llama.cpp
on this file has not been measured, so the page stays experimental. The three Q6_K parity programs above are the
remaining GPU check.
| Engine | Experts loaded | Prompt | Decode | Drafts accepted |
| --- | --- | --- | --- | --- |
| 0.1.39 | 101.75 GiB at 8.43 GiB/s | 14,915 tokens cold at 3,176 tok/s | 5,799 tokens in 74.1 s (78.3 tok/s) over 14 turns, ending at 44,160 tokens of context | 3,485 of 4,406 (79.1%) |
| 0.1.38 | | 249,193 tokens cold at 2,802 tok/s | 1,079 tokens at 69.0 tok/s | |
packaging/ENGINE-COMPONENT.md
+28 -17
View File
@@ -1,7 +1,7 @@
# Engine component manifest (schema 2)
One format for every engine package: Windows CUDA 13 and 12, Windows HIP, Linux CUDA, Linux HIP, Linux SYCL and macOS
Metal. Control (`strata_control/engine_component.py`) reads it, checks it, and only then starts the engine.
This page is for developers who build or check engine packages. It defines the manifest that every engine package carries:
Windows CUDA 13 and 12, Windows HIP, Linux CUDA, Linux HIP, Linux SYCL and macOS Metal. Control (`strata_control/engine_component.py`) reads it, checks it, and only then starts the engine.
`packaging/package_engine.py` writes it. A package you make locally is unsigned and loads. A package in the release feed is signed with the
feed key (an Ed25519 key the owner keeps offline), and `strata update` / "Update engine" install it only when that signature verifies
([UPDATES.md](UPDATES.md)). The installers carry an engine of their own, so a new install needs no download to start a model.
@@ -24,7 +24,7 @@ A package folder is immutable: a new build is a new folder.
|---|---|---|
| `schema_version` | 2 | schema 1 packages (no `archs`, `min_driver`, `build_id`, `signature`; `cuda_archs` instead of `archs`) still load |
| `id`, `name` | `"strata-engine"` | the component name; anything else is refused |
| `version` | string | the engine release, e.g. `0.1.45`; the newest compatible version wins |
| `version` | string | the engine release, e.g. `0.1.50`; the newest compatible version wins |
| `platform` | string | `windows-x86_64`, `linux-x86_64`, `macos-arm64` (must match this PC) |
| `backend` | string | `cuda`, `hip`, `sycl`, `metal`, `cpu`; must be a combination `BACKEND_MATRIX` commits to, and the one this PC's GPU needs |
| `archs` | list of strings | what the build contains. CUDA: `sm_86` (machine code for that card), `compute_89` (PTX, runs on that generation or newer). HIP: `gfx1201`. SYCL and Metal: a descriptive tag (`intel-arc`, `apple-gpu`). Required for every non-`cpu` backend. Enforced for CUDA against `nvidia-smi` compute capability; recorded for the others |
@@ -40,7 +40,7 @@ A package folder is immutable: a new build is a new folder.
Example (shortened):
```json
{"schema_version": 2, "id": "strata-engine", "name": "strata-engine", "version": "0.1.45",
{"schema_version": 2, "id": "strata-engine", "name": "strata-engine", "version": "0.1.50",
"platform": "linux-x86_64", "backend": "cuda", "archs": ["sm_86"], "min_driver": {"nvidia": "580.65"},
"build_id": "be885026f96a-cuda13.0", "entrypoint": "strata",
"files": [{"path": "strata", "sha256": "<64 hex>", "bytes": 191000000}], "signature": null}
@@ -59,7 +59,7 @@ In this order, each refusal names the cause; none of them starts anything and no
6. on POSIX the entrypoint is executable.
If a package fails (5) or (4) and no other package fits this PC, the start ends with `EngineIntegrityError`
(exit code 5): "The engine package failed its integrity check, so it was not started: 0.1.45 linux-x86_64-cuda:
(exit code 5): "The engine package failed its integrity check, so it was not started: 0.1.50 linux-x86_64-cuda:
libcudart.so.13 does not match its recorded SHA-256. Repair or reinstall the engine component with Update engine in the app's
Settings or `strata update` in the Terminal". A package
for another GPU or an older driver is `EngineMissing` with its reason. A tampered newer package never falls back silently
@@ -83,29 +83,35 @@ UTF-8). It contains every file hash, so one signature covers the whole package.
## Building a package
python packaging/package_engine.py --src <build folder> --backend cuda --version 0.1.45 \
python packaging/package_engine.py --src <build folder> --backend cuda --version 0.1.50 \
--out <components>/engine --arch sm_86 --arch compute_89 --min-driver nvidia=580.65 \
[--build-id <id>] [--variant cuda13] [--platform linux-x86_64] [--include 'rocm/**' --include strata-vision.exe | --tree]
[--build-id <id>] [--variant cuda13] [--platform linux-x86_64] [--include 'rocm/**' --include strata-vision.exe | --tree] [--require-dsv41]
python packaging/package_engine.py --verify <package folder> # full hash check
Without `--include`/`--tree` it takes the entrypoint, the libraries beside it, `BUILD.json` and `LICENSE*`. The build id
Without `--include`/`--tree` it takes the entrypoint, the libraries beside it, `BUILD.json`, `LICENSE*`, the picture reader
`strata-vision` when the build has one, and the DeepSeek engine `strata-dsv41[.exe]` when it sits beside `strata`.
`--require-dsv41` refuses a package without `strata-dsv41`. The build id
defaults to the checkout's commit (`-dirty` when the tree has changes). It prints the size, file count and the SHA-256 of
the manifest file (which holds every file's hash), used as the package's digest in notes.
## What each target's package holds
Variants in the release: Windows `cuda13`, `cuda12`, `rx9070`, `rdna3`, `rdna2`; Linux `cuda13`, `cuda12`, `rx9070`, `rx7900`, `halo`, `rx6000`, plus
the SYCL and Metal packages. The packages that also carry `strata-dsv41` (DeepSeek V4.1-Flash, engine 0.1.50) are Windows `cuda13`, Linux `cuda13`
and Linux `rx9070`. Build `strata-dsv41` with `packaging/build_dsv41_cuda.sh` or `packaging/build_dsv41_windows.bat` and copy it beside `strata` before packaging.
| Package | Built on | Contents | Notes |
|---|---|---|---|
| `windows-x86_64` cuda, variant `cuda13` | the owner PC, the same cmake line as the release zips (CUDA 13.0, `75;86;89;120` real + `120` PTX, `STRATA_PORTABLE=ON`) | `strata.exe`, `strata_kq.dll`, `strata-vision.exe` (built for this variant's archs by `packaging/build_vision_windows.bat cuda13 <stage>`, which writes `vision_archs` into BUILD.json), `strata-dsv41.exe` (DeepSeek-V4.1, `packaging/build_dsv41_windows.bat`, sm_86 + sm_120; `--require-dsv41`), `BUILD.json` | `min_driver` nvidia 580.65 |
| `windows-x86_64` cuda, variant `cuda12` | the owner PC (CUDA 12.9, `60;61;70;75;80;86;89` real + `89` PTX, `STRATA_EXPERIMENTAL_SM60=ON`) | the same four files, with its own `strata-vision.exe` (`packaging/build_vision_windows.bat cuda12 <stage>`: sm_60 .. sm_89; the CUDA 13 reader has no Pascal code, #3669) | experimental, `min_driver` nvidia 575.51; the CUDA 13 build wins when both fit |
| `windows-x86_64` hip | the owner PC, `tools/hip/build_windows.bat` (ROCm 10.2 nightly wheels) then `tools/hip/package_windows.py` | `strata.exe`, `strata-device.exe`, the ROCm DLLs and the rocBLAS/hipBLASLt kernels of the zip (`--tree`) | no Windows AMD test machine yet; with `STRATA_NO_VISION` unset it also builds the two picture readers (`strata-vision.exe` on the CPU, `strata-vision-hip.exe` on the GPU) |
| `windows-x86_64` cuda, variant `cuda13` | the owner PC, the same cmake line as the release zips (CUDA 13.0, `75;86;89;120` real + `120` PTX, `STRATA_PORTABLE=ON`) | `strata.exe`, `strata_kq.dll`, `strata-vision.exe` (built for this variant's archs by `packaging/build_vision_windows.bat cuda13 <stage>`, which writes `vision_archs` into BUILD.json), `BUILD.json` | `min_driver` nvidia 580.65 |
| `windows-x86_64` cuda, variant `cuda12` | the owner PC (CUDA 12.9, `60;61;70;75;80;86;89` real + `89` PTX, `STRATA_EXPERIMENTAL_SM60=ON`) | the same four files, with its own `strata-vision.exe` (`packaging/build_vision_windows.bat cuda12 <stage>`: sm_60 .. sm_89; the CUDA 13 reader has no Pascal code) | experimental, `min_driver` nvidia 575.51; the CUDA 13 build wins when both fit |
| `windows-x86_64` hip | the owner PC, `tools/hip/build_windows.bat` (ROCm 10.2 nightly wheels) then `tools/hip/package_windows.py` | `strata.exe`, `strata-device.exe`, the ROCm DLLs and the rocBLAS/hipBLASLt kernels of the zip (`--tree`) | with `STRATA_NO_VISION` unset it also builds the two picture readers (`strata-vision.exe` on the CPU, `strata-vision-hip.exe` on the GPU) |
| `linux-x86_64` cuda, variant `cuda13` | a Linux box with the CUDA 13 toolkit, `VARIANT=cuda13 packaging/build_cuda_linux.sh` | `strata`, `libstrata_kq.so`, `strata-vision` (built by the same script for the same archs, `BUILD_VISION=0` for a text-only package), `libcudart/libcublas/libcublasLt` (13.x) | the GPU driver's `libcuda` stays the PC's |
| `linux-x86_64` cuda, variant `cuda12` | a Linux box with CUDA 12.9, `VARIANT=cuda12 packaging/build_cuda_linux.sh` (`60;61;70;75;80;86;89` real + `89` PTX, `STRATA_EXPERIMENTAL_SM60=ON`) | the same files with the 12.x libraries (the reader has sm_60 / 61 / 70 too) | experimental, `min_driver` nvidia 575.51; the Pascal and Volta package (the 0.2.1 one lacked sm_60 / 61 / 70, #3503); `package_engine.py` refuses a cuda12 package without them |
| `linux-x86_64` cuda, variant `cuda12` | a Linux box with CUDA 12.9, `VARIANT=cuda12 packaging/build_cuda_linux.sh` (`60;61;70;75;80;86;89` real + `89` PTX, `STRATA_EXPERIMENTAL_SM60=ON`) | the same files with the 12.x libraries (the reader has sm_60 / 61 / 70 too) | experimental, `min_driver` nvidia 575.51; the Pascal and Volta package; `package_engine.py` refuses a cuda12 package without them |
| `linux-x86_64` hip | a box with ROCm (the Python wheels are enough) | `strata`, the HIP runtime, hipBLAS/hipBLASLt/rocBLAS, comgr + libLLVM, `rocblas/library` and `hipblaslt/library` for the listed gfx archs; `packaging/build_hip_linux.sh` builds it with both picture readers |
| `linux-x86_64` hip, variant `rx6000` (RX 6000, gfx1030 / gfx1031) | `VARIANT=rx6000 packaging/build_hip_linux.sh` in the jammy chroot, ROCm 7.13.0a20260515 `gfx103X-all` wheels | the same set as the other HIP packages, with `rocblas/library` for gfx1030 / gfx1031; no `hipblaslt/library` kernels exist for them and no `tuning/` table | built, not tested on a real RX 6000 card (the manifest notes say so). `windows-x86_64` hip, variant `rdna2`: the same archs from `tools/hip/build_windows.bat` with `STRATA_HIP_ARCHS=gfx1030;gfx1031`, packaged with `--variant rdna2`. A PC with an RX 7000 / 9000 card beside an RX 6000 one never takes it (`engine_component.hip_preference`) |
| `linux-x86_64` hip, gfx1103 (Radeon 780M / Phoenix iGPU) | `packaging/build_hip_gfx1103.sh` on the mini PC (TheRock gfx110X-all SDK) | the same set as the gfx1200/1201 package, with `rocblas/library` and `hipblaslt/library` for gfx1103 (600 MiB) | experimental, unvalidated: the iGPU resets now and then (MES REMOVE_QUEUE); never kill the engine mid-request; run configs set `STRATA_NO_ARENA_THP=1`. `engine_component.resolve` picks a HIP package only when its `archs` list a gfx target the Linux kfd topology reports |
| `linux-x86_64` sycl, `archs` `intel-arc` | a Linux box with oneAPI DPC++ 2026.1 and `patchelf`, `packaging/build_sycl_linux.sh` (JIT from SPIR-V; ocloc cannot build the Alchemist FP64 kernels ahead of time, so one JIT package serves A-series and B-series) | `strata`, libsycl, the Unified Runtime adapters, the oneMKL SYCL libraries, libimf / libsvml / libirng / libintlc (about 650 MiB); every ELF file has RUNPATH `$ORIGIN` (Intel's libraries ship without it), so `env -i ./strata --version` runs | the Level Zero loader and compute runtime are the PC's driver; the user must be in the `render` group (`strata doctor` warns). No Windows package: `BACKEND_MATRIX` allows sycl on `linux-x86_64` only |
| `macos-arm64` metal | an Apple Silicon Mac | `strata`, `strata-vision` (`packaging/build_vision_mac.sh`; `package_engine.py` refuses a macOS package without it, #3697), `metal/kernels/*.metal`, `metal/engine-msl/*.metal`, `metal/ggml-common.h`, `metal/data/*.bin` | the Metal engine compiles its kernel sources when it starts; since S1 it finds them beside the executable (`metal/src/resource_path.h`: environment variable, then `<exe dir>/metal/`, then the compiled-in source path) |
| `macos-arm64` metal | an Apple Silicon Mac | `strata`, `strata-vision` (`packaging/build_vision_mac.sh`; `package_engine.py` refuses a macOS package without it), `metal/kernels/*.metal`, `metal/engine-msl/*.metal`, `metal/ggml-common.h`, `metal/data/*.bin` | the Metal engine compiles its kernel sources when it starts; it finds them beside the executable (`metal/src/resource_path.h`: environment variable, then `<exe dir>/metal/`, then the compiled-in source path) |
`packaging/collect_libs.py` gathers the Linux libraries (an `ldd` closure with the dlopen'd extras named, kernel data filtered by
GPU architecture) into the folder `package_engine.py` packages; libraries the distribution provides are listed in
@@ -119,16 +125,14 @@ with the package). `packaging/stage_tuning.py` does it for the Linux build scrip
`package_engine.py` refuses an AMD stage without `hipblaslt_version` or with a table for another version or architecture. The server's start
(`serve/server.py with_component_engine`, rule in `strata_control/hipblaslt_tuning.py`) picks the table of the card and sets
`STRATA_HIPBLASLT_TUNING` for the engine; a version mismatch, an unreadable card or a missing version are one log line and plain hipBLAS.
`STRATA_HIPBLASLT_TABLE=on|off` (or `"hipblaslt_table"` in the run config) overrides; unset it follows `AUTO_USES_TABLE`, which is off until the
owner decides, because a table changes the rounding of the prompt's GEMMs (docs/AMD_HIP.md, Tuning table). Tests: `tests/terminal/test_hipblaslt_tuning.py`.
`STRATA_HIPBLASLT_TABLE=on|off` (or `"hipblaslt_table"` in the run config) overrides; unset it follows `AUTO_USES_TABLE`, which is off by default, because a table changes the rounding of the prompt's GEMMs (docs/AMD_HIP.md, Tuning table). Tests: `tests/terminal/test_hipblaslt_tuning.py`.
## The picture reader must match the package
`package_engine.py` (the command line, not the library call) checks the reader before it packages:
* a CUDA package with `strata-vision` needs `vision_archs` in BUILD.json, and every `sm_NN` of the package must be in it. A reader reused from an
older build was the cause of #3669 (the cuda12 package on a Pascal card: the engine ran, the reader stopped with "device kernel image is
invalid"). `--allow-stale-vision` packages it anyway and prints which archs are missing;
older build fails on cards it was not built for. `--allow-stale-vision` packages it anyway and prints which archs are missing;
* a macOS package without `strata-vision` is refused (`--no-picture-reader` ships a text-only Mac package on purpose);
* a CUDA package with no reader is allowed: it answers text only, the app says "the engine installed here does not include the picture reader
yet" and keeps Read pictures off.
@@ -146,3 +150,10 @@ and the CPU reader as `vision.cpu_exe`, and the GPU reader does not start (the c
busy) the server starts the CPU reader by itself and prints one line. Both programs have RUNPATH `$ORIGIN` on Linux; on Windows the
ROCm DLLs they import sit beside the programs or in `rocm/bin` (the engine's PATH entry, `lib_dirs`). `strata_control/vision_reader.py`
holds the choice; `tools/test_vision_reader.py` the tests.
## History / older measurements
- **Stale picture reader (#3669):** a cuda12 package reused a reader built for newer cards. On a Pascal card the engine ran and the reader stopped with
"device kernel image is invalid". `package_engine.py` now checks `vision_archs`.
- **Missing sm_60 / 61 / 70 (#3503):** the first cuda12 package lacked Pascal and Volta code. `package_engine.py` now refuses a cuda12 package without them.
- **Mac without a reader (#3697):** a macOS package once shipped without `strata-vision`. It is refused unless `--no-picture-reader` is given.
packaging/README.md
+112 -114
View File
@@ -1,42 +1,51 @@
# Strata packaging: targets, recipes and locks
Revision: Strata 0.2.0. This directory holds the per-target build recipes and the machine-readable
manifests. **It describes what can be built where, and it is explicit about what has *not* been
built or tested.** Nothing here is a support claim.
This page is for developers who build the desktop app, the Terminal archive and the engine packages. To install Strata, use
[docs/INSTALL.md](../docs/INSTALL.md). It describes the current release (app and Terminal 0.2.5, Control and engines 0.1.50).
How packages are signed and published is in [UPDATES.md](UPDATES.md), and the package format is in [ENGINE-COMPONENT.md](ENGINE-COMPONENT.md).
The installers carry the engine: the desktop installers hold an engine package for their platform and graphics card, and the first
start copies it into the per-user component folder after checking every file against its manifest. A new install therefore starts a
model without a download; later engines come from the signed feed ([UPDATES.md](UPDATES.md)). The Terminal install is the other
route: one line, and the first model brings the engine for that PC from the same release.
Release assets show what was published; the platform guides and retained reports show what was tested. A published package does not
qualify every GPU, model or feature on its platform.
## Quick start
| I want to | See |
| --- | --- |
| Build the desktop app | [Desktop app](#desktop-app) |
| Build an engine package | [ENGINE-COMPONENT.md](ENGINE-COMPONENT.md#building-a-package) |
| Build the Terminal archive | [The Terminal archive](#the-terminal-archive) |
| Sign and publish a release | [UPDATES.md](UPDATES.md#releasing) |
| Test an update against a local feed | [UPDATES.md](UPDATES.md#testing-locally) |
## The target matrix
A backend name alone is not a target. Each row is an OS x CPU ABI x backend tuple, and a row is
`qualified` only after the acceptance checklist has actually run on it.
Each row is an OS, CPU ABI and backend. A row is `qualified` only after the acceptance checklist
([below](#what-a-platform-must-produce-before-it-is-qualified)) has run on it.
| Target tuple | App artifact | Engine package | Status |
| Target | App artifact | Engine package recipe | Status |
| --- | --- | --- | --- |
| `macos-arm64-metal` | `.app` + `.dmg` (Tauri) | Metal `strata`, bundled | **built.** The app's self-test passed from inside the mounted DMG and the engine's 17 kernel-parity suites pass on an Apple M4 GPU. A large model does not fit the 16 GiB test Mac. Carries `strata-vision` (Metal), built by `packaging/build_vision_mac.sh`. Signing and notarising: `packaging/mac/sign-and-notarize.sh` (see [mac/README.md](mac/README.md)); it has been run with stand-ins for Apple's service, not yet against the real one. |
| Terminal `darwin-arm64` | self-contained `.tar.gz` | any | **built and verified.** Runs from a moved directory in a clean environment with no usable system Python. |
| `windows-x86_64-cuda` | NSIS `.exe` (Tauri) | `cuda13` and `cuda12` packages | **built and tested** on a Windows PC with an NVIDIA card: silent install with a PATH of System32 only, a real model, an agent chat. |
| `windows-x86_64-hip` | NSIS `.exe` (Tauri) | `tools/hip/build_windows.bat` | **built; never run on a real AMD card.** |
| `linux-x86_64-cuda` | AppImage (Tauri) | `setup.py`; built in a glibc 2.35 chroot | **cuda13 built and tested** (RTX 3060 and RTX A4000: app in the browser, a real model, a picture). cuda12 built. |
| `linux-x86_64-hip` | AppImage (Tauri) | `packaging/build_hip_linux.sh` | **RX 9000 (gfx1200 / gfx1201) built and tested** on a Radeon AI PRO R9700. **RX 7000 (gfx1100 / 1101 / 1102) built; not run on an RX 7000 card.** **RX 6000 (gfx1030 / 1031, 0.2.2, engine package `rx6000` only) built; not tested on a real RX 6000 card.** |
| `linux-x86_64-sycl` | | `packaging/build_sycl_linux.sh` (one JIT package for Alchemist and Battlemage) | **engine package built; tested on an Arc A750** (Intel Arc is an official platform since 0.1.40.2, docs/INTEL_ARC.md). Arc B-series (Battlemage) cards are supported and untested by us: no B-series card was available for the package (the B70 tests of 2026-10 were with the source build), and the Terminal says so once and tells the person how to report a problem. `sycl/setup_intel.py` (Docker) remains the source-build route. |
| `windows-x86_64-sycl` | none | none | **gated.** No native Windows SYCL path exists upstream; it needs a separate feasibility gate, not a recipe. |
| `macos-x86_64-*` | none | none | **unsupported.** The macOS engine is Apple-Silicon only (`metal/CMakeLists.txt` requires Apple Silicon for the Metal runtime). |
| `macos-arm64-metal` | `.app` and `.dmg` (Tauri) | Metal `strata`, bundled with `strata-vision` (`packaging/build_vision_mac.sh`) | Built. Self-test passes from the mounted DMG. Inference measurements are in [MAC.md](../docs/MAC.md). Signing: `packaging/mac/sign-and-notarize.sh` ([mac/README.md](mac/README.md)). |
| `darwin-arm64` Terminal | self-contained `.tar.gz` | any | Built and verified. Runs from a moved directory with no system Python. |
| `windows-x86_64-cuda` | NSIS `.exe` (Tauri) | `cuda13` and `cuda12` | Built and tested on a Windows PC with an NVIDIA card: silent install with a System32-only PATH, a real model, an agent chat. |
| `windows-x86_64-hip` | NSIS `.exe` (Tauri) | `tools/hip/build_windows.bat` | Published. Strix Halo inference is recorded in [STRIX_HALO.md](../docs/STRIX_HALO.md). This does not qualify every discrete Radeon card ([AMD_HIP.md](../docs/AMD_HIP.md)). |
| `linux-x86_64-cuda` | AppImage (Tauri) | `packaging/build_cuda_linux.sh`, built in a glibc 2.35 chroot | `cuda13` built and tested on RTX 3060 and RTX A4000 (app in the browser, a real model, a picture). `cuda12` built. |
| `linux-x86_64-hip` | AppImage (Tauri) | `packaging/build_hip_linux.sh` | `rx9070` (gfx1200 / gfx1201) built and tested on a Radeon AI PRO R9700. `rx7900` (gfx1100 / 1101 / 1102) built, not run on an RX 7000 card. `rx6000` (gfx1030 / 1031) built, not tested on a real RX 6000 card. |
| `linux-x86_64-sycl` | none | `packaging/build_sycl_linux.sh` (one JIT package for Alchemist and Battlemage) | Engine package built and tested on an Arc A750. Battlemage is supported and untested by us. |
| `windows-x86_64-sycl` | none | none | Gated. No native Windows SYCL path exists upstream. |
| `macos-x86_64-*` | none | none | Unsupported. The macOS engine is Apple Silicon only. |
**Cross-compilation is not attempted for the desktop app.** Tauri bundles are per-OS and the
Windows installer must be produced on Windows. What *is* portable is
everything below the carrier: the Python contracts, the owner transport, the checks, and the engine
build recipes.
**DeepSeek V4.1-Flash.** Three packages carry engine 0.1.50 with the DeepSeek engine `strata-dsv41`: Windows NVIDIA `cuda13`,
Linux NVIDIA `cuda13` and Linux AMD `rx9070`. All other packages carry engine 0.1.50. Build `strata-dsv41` with
`packaging/build_dsv41_cuda.sh` (Linux) or `packaging/build_dsv41_windows.bat` (Windows), then package it with
`package_engine.py --require-dsv41`. The DeepSeek engine can also be a separate download (feed member `dsv41`, see [UPDATES.md](UPDATES.md#the-feed)).
Cross-compilation is not attempted for the desktop app. Tauri bundles are per OS, and the Windows installer is built on Windows.
The Python contracts, the owner transport, the checks and the engine build recipes are portable.
## The private Node runtime
The desktop bundle ships a pinned standalone Node beside the App Host, because the packaging
requirement is that an end user supplies no runtime. It is **not committed**: it is a ~117 MB
platform binary, it cannot be reviewed in a diff, and it is an input rather than source. Fetch and
verify it instead:
The desktop bundle ships a pinned standalone Node beside the App Host, so an end user supplies no runtime. It is not committed
(about 117 MB, an input rather than source). Fetch and verify it:
```sh
apps/desktop/scripts/fetch-node.sh # host platform
@@ -44,11 +53,9 @@ apps/desktop/scripts/fetch-node.sh --target linux-x64
apps/desktop/scripts/fetch-node.sh --check # verify what is installed
```
The script refuses an archive whose SHA-256 does not match the digest recorded for that target. The
digests are taken from nodejs.org's own `SHASUMS256.txt` for the pinned version, so they are not
derived from the same response as the payload. **To bump the Node version: change
`STRATA_NODE_VERSION`, add the new digests, and re-run `--check`** — the script will refuse an
unknown target's archive rather than installing it unverified.
The script refuses an archive whose SHA-256 differs from the digest recorded for that target. The digests come from nodejs.org's
`SHASUMS256.txt` for the pinned version. To bump the version, change `STRATA_NODE_VERSION`, add the new digests and run `--check`.
`fetch-node.ps1` is the Windows equivalent.
## Desktop app
@@ -56,104 +63,95 @@ unknown target's archive rather than installing it unverified.
cd apps/desktop
./scripts/fetch-node.sh # the sidecar (once)
npm install --prefix ../host && npm --prefix ../host run build # the UI
cargo tauri build --bundles app # -> Strata.app
# macOS DMG (headless: Tauri's default step needs a Finder session)
./scripts/make-dmg.sh
# Windows (on Windows): produces an NSIS per-user installer
cargo tauri build --bundles nsis
# Linux (on Linux): AppImage plus a deb
cargo tauri build --bundles appimage deb
cargo tauri build --bundles app # macOS: Strata.app
./scripts/make-dmg.sh # macOS DMG (headless; Tauri's default step needs a Finder session)
cargo tauri build --bundles nsis # Windows, on Windows: per-user NSIS installer
cargo tauri build --bundles appimage deb # Linux, on Linux
```
Windows installer policy is set in `tauri.conf.json`: `installMode: currentUser`, so the installer
is **per-user and needs no elevation**. Uninstall retention (checklist P10) is a property of *where
the app keeps its data*, not of an installer flag: models, profiles and sessions live outside the
install directory (`~/.strata` and the platform state directories), so removing the application
cannot remove them. The packaging config was validated against this Tauri version's schema — an
earlier draft used `perMachine`/`oneClick`/`deleteAppDataOnUninstall`, which `tauri-utils` 2.10 does
not declare and which are rejected outright rather than ignored.
The Windows installer is per-user and needs no elevation (`installMode: currentUser` in `tauri.conf.json`). Models, profiles and
sessions live outside the install directory (`~/.strata` and the platform state directories), so uninstalling the app does not remove them.
**The `dmg`/`app` targets were rebuilt and the packaged self-test re-run (8/8) after the Windows and
Linux targets were added**, so the added targets do not affect the macOS artifact.
Build boxes build without a signing key: the default `tauri.conf.json` has `createUpdaterArtifacts: false`. Signing happens on the
release PC ([UPDATES.md](UPDATES.md#releasing)).
## Strata Control and the engine in the installed app
The installer carries two things that start a model, and neither fetches anything when it runs.
The installer carries Control, and on a Mac the Metal engine too. Neither fetches anything when it runs.
**Control** (`<install>/control`, built by `apps/desktop/scripts/stage-control.py`): the same bundle as the Terminal
archive, the pinned CPython 3.12.15, the hash-locked wheels, `strata_control`, `strata_cli` and `serve`, plus a
`BUNDLE.json` with the interpreter and lock digests. The host finds it beside itself (or through `STRATA_CONTROL_DIR`),
runs it on its own interpreter (`PYTHONNOUSERSITE=1`, no `PYTHONHOME`/`PYTHONPATH`) and starts a model with
`strata profile create|use-config` + `strata runtime start`, so the engine runs under the owner runtime (loopback TCP and a
256-bit token on Windows, a private socket elsewhere; a Job Object / process group; leases) and a second client (the
Terminal) attaches to the same instance.
**Control** (`<install>/control`, built by `apps/desktop/scripts/stage-control.py`) is the same bundle as the Terminal archive:
the pinned CPython 3.12.15, the hash-locked wheels, `strata_control`, `strata_cli` and `serve`, plus a `BUNDLE.json` with the
interpreter and lock digests. The host finds it beside itself or through `STRATA_CONTROL_DIR`, runs it on its own interpreter
(`PYTHONNOUSERSITE=1`, no `PYTHONHOME` or `PYTHONPATH`) and starts a model with `strata profile create|use-config` and
`strata runtime start`. The engine then runs under the owner runtime (loopback TCP and a 256-bit token on Windows, a private
socket elsewhere; a Job Object or process group; leases), and a second client such as the Terminal can attach to the same instance.
The app starts each engine with `runtime start --die-with-pid <host>`, so the engine ends with the host, and the host ends
with the shell (a kill-on-close Job Object around the app on Windows; the host watching its parent everywhere). The app is a
single instance: a second launch focuses the open window. One app can run several engines (profiles `app-1`, `app-2`, ...).
docs/CLOSING.md has every close path and how each is tested.
The app starts each engine with `runtime start --die-with-pid <host>`, so the engine ends with the host. The host ends with the shell
(a kill-on-close Job Object on Windows; the host watches its parent everywhere). The app is a single instance: a second launch focuses
the open window. One app can run several engines (profiles `app-1`, `app-2`, ...). Every close path is in [docs/CLOSING.md](../docs/CLOSING.md).
**The engine** comes with the installer (`<install>/engine-bundle/<package folder>`, the same layout as below); the first start checks it against its own manifest and copies it into the per-user component folder, where Control finds it like any engine that `strata update` installed. It is a component in the per-user component folder
(`%LOCALAPPDATA%\Strata\components`, `$XDG_DATA_HOME/strata/components`, or `STRATA_COMPONENTS_DIR`):
**The engine.** The Windows and Linux installers carry no engine. On the first start the app shows the engine for the card with its size
and one Download button (with progress and resume), or installs an engine file the person already has ("I already have the file").
The Mac dmg keeps its small Metal engine in `<install>/engine-bundle/<package folder>`. Either way the engine ends up in the per-user
component folder, after the package is checked against its manifest (`%LOCALAPPDATA%\Strata\components`, `$XDG_DATA_HOME/strata/components`, or `STRATA_COMPONENTS_DIR`),
where Control finds it like any engine that `strata update` installed:
components/engine/<version>-<platform>-<backend>/
engine-component.json schema 2: id/name "strata-engine", version, platform, backend, archs, min_driver,
build_id, min_runtime (optional), entrypoint, files [{path, sha256, bytes}],
signature (null for a local package, set by the release feed); full table and the checks in ENGINE-COMPONENT.md
components/engine/<version>-<platform>-<backend>[-<variant>]/
engine-component.json the manifest (see ENGINE-COMPONENT.md)
strata[.exe] *.dll | *.so the binary and the GPU runtime libraries beside it
A run config names it as `"exe": "component:strata-engine"`; at start `strata_control/engine_component.py` picks the package
for this PC (platform, then the backend its GPU needs: nvidia-smi / rocminfo / sycl-ls, or `STRATA_ENGINE_BACKEND`; CUDA
architectures, `min_driver` and `min_runtime` are checked; the newest version wins), checks every file's size and
SHA-256 (a start hashes the whole package; a mismatch is `EngineIntegrityError` and nothing starts) and puts the package
folder first on the library path.
Nothing matching is `EngineMissing`: `strata serve` ends with exit 5 and "No engine for this PC was found in ... The app never downloads an engine
while it starts or serves", and no fetch. That is the start itself. Everything before it is a warning, not a refusal: `strata launch` says in plain
words that it could not find an engine for this computer, asks whether to go on (Enter means yes), downloads the model and ends with the next step
(exit code 0, nothing started), and the app shows the same warning with an "Update engine" button in Settings. `packaging/package_engine.py` makes a local package from a build folder.
A run config names it as `"exe": "component:strata-engine"`. At start, `strata_control/engine_component.py` picks the package for this PC
(platform, then the backend its GPU needs, found with nvidia-smi, rocminfo or sycl-ls, or `STRATA_ENGINE_BACKEND`), checks every
file's size and SHA-256 and puts the package folder first on the library path. A mismatch is `EngineIntegrityError` and nothing starts.
**No acquisition at run time.** `STRATA_NO_ACQUIRE=1` (set by the Tauri shell for the host, and for Control by the host)
makes `strata_control/acquisition.py` refuse pip, downloads and compiles; on the host side `server/acquisition.mjs`
refuses the llama.cpp and cloudflared fetches (a connector placed in `components/cloudflared/` is used as it is). Models the
user chooses to download are a separate, explicit action. `apps/host/tests/no-acquire.test.mjs` runs the host under
`tests/helpers/acquire-spy.mjs` and fails on any outbound request or installer process;
`apps/desktop/scripts/installed-flow.mjs` drives an installed app end to end (stub engine anywhere, real engine with `--components`).
When no package matches, `strata serve` ends with exit 5 and "No engine for this PC was found". It never fetches. Before that point
everything is a warning: `strata launch` says it could not find an engine, asks whether to go on, downloads the model and ends with
the next step (exit 0, nothing started). The app shows the same warning with an "Update engine" button in Settings.
`packaging/package_engine.py` makes a local package from a build folder.
## What every platform must produce before it can be called qualified
**No acquisition at run time.** `STRATA_NO_ACQUIRE=1` (set by the Tauri shell for the host, and by the host for Control) makes
`strata_control/acquisition.py` refuse pip, downloads and compiles. On the host side `server/acquisition.mjs` refuses the llama.cpp
and cloudflared fetches (a connector placed in `components/cloudflared/` is used as it is). Models the user chooses to download are a
separate, explicit action. `apps/host/tests/no-acquire.test.mjs` fails on any outbound request or installer process.
`apps/desktop/scripts/installed-flow.mjs` drives an installed app end to end.
## The Terminal archive
The Terminal bundles a private Python and its dependencies for `windows-x86_64`, `linux-x86_64` and `darwin-arm64`
(`package_terminal.py --target ...`, built on the target OS).
- The interpreter is python-build-standalone CPython 3.12.15, digest-checked at build time.
- Packages come from `packaging/locks/runtime-<target>-cp312.txt` and are installed with
`pip install --require-hashes --no-deps --only-binary=:all:` at build time only. `lock_wheels.py` makes the lock; review its diff when a pin changes.
- Wrappers: `bin/strata.cmd` and `bin/strata.ps1` on Windows, `bin/strata` on POSIX. `START-HERE.bat`, `SETUP.bat`, `UPDATE.bat`, `setup.sh`
and `update.sh` use a `python/` folder beside them when there is one.
- `smoke_terminal.py <archive>` extracts an archive into a clean environment (a path with a space, a PATH with no Python) and runs
`--version`, `doctor`, `profile list`, `install` and `serve --engine mock`.
- Check each archive's `BUNDLE.json` and lock digests. An old archive is not upgraded when the source lock changes.
## What a platform must produce before it is qualified
1. The app launches and the window serves the supplied UI from the host's origin.
2. The owner transport works: one owner per installation, a second client attaches, a stale owner
can be recovered, and no client can stop a process it does not own (R03/R04).
2. The owner transport works: one owner per installation, a second client attaches, a stale owner can be recovered, and no client can stop a process it does not own.
3. Real inference on a supported pack, with the engine build hash and the model hashes recorded.
4. `docs/work/QUALIFICATION-KIT.md` run end to end, with raw evidence retained.
4. `docs/work/QUALIFICATION-KIT.md` run end to end, with raw evidence kept.
## Honest gaps in this directory
## Release checks and gaps
* There is no CI. Every recipe here is written to be run by hand; making them automated is separate
work.
* Signing needs the owner's credentials, which are never in this tree. The update feed and every engine package are signed with the feed key
(`feed_tool.py`, [UPDATES.md](UPDATES.md)); the app installers with the Tauri updater key. The Mac app and Terminal archive are signed and
notarised on a Mac that holds the Developer ID certificate, with `packaging/mac/sign-and-notarize.sh` ([mac/README.md](mac/README.md)); for a release, `--updater-archive` also packs the notarised app as `Strata.app.tar.gz` for the in-app Mac update (the one `tauri build` makes holds the unsigned app), signed with the Tauri key on the PC and added with `feed_tool.py set-app-platform`. No
Windows code-signing certificate is configured in this tree; Windows may warn about an installer that is not signed.
* No font or asset redistribution evidence has been gathered (D05). The app currently ships the
brand assets from the supplied UI ZIP, which is fine for an internal candidate and not sufficient
for redistribution.
* The terminal distribution bundles a private Python runtime and its dependencies for `windows-x86_64`,
`linux-x86_64` and `darwin-arm64` (`package_terminal.py --target ...`, built on the target OS). The interpreter is
the pinned python-build-standalone CPython 3.12.15 (digest-checked at build time); the packages come from
`packaging/locks/runtime-<target>-cp312.txt`, installed with `pip install --require-hashes --no-deps
--only-binary=:all:` at BUILD time only (`lock_wheels.py` makes the lock; review its diff when a pin changes).
Wrappers: `bin/strata.cmd` / `bin/strata.ps1` on Windows, `bin/strata` on POSIX; START-HERE.bat, SETUP.bat,
UPDATE.bat, setup.sh and update.sh use a `python/` folder beside them when there is one.
`smoke_terminal.py <archive>` extracts an archive into a clean environment (a path with a space, a PATH with no
Python) and runs `--version`, `doctor`, `profile list`, `install` and `serve --engine mock`. The macOS archive
predates the lock: build it again on a Mac to get the hash-pinned install (the lock is already here).
* `.gitignore` coverage for build output was added late. `apps/desktop/src-tauri/target` and the
Node sidecar were briefly committed, which left ~324 MB in the git object store. The files are
untracked now and the ignore rules are in place, but the history is not rewritten: that is a
deliberate choice, since rewriting published history is worse than a large local repository. A
developer cloning this branch gets the current tree, and the sidecar arrives via
`fetch-node.sh`.
- CI covers scripts, Terminal checks, Metal builds and Docker images ([.github/workflows](../.github/workflows)). It does not replace native desktop
packaging, signing or real-GPU acceptance on every target. Some recipes still run by hand.
- Signing needs the owner's credentials, which are never in this tree. The feed and every engine package are signed with the feed key
(`feed_tool.py`); the app installers with the Tauri updater key ([UPDATES.md](UPDATES.md)). The Mac app and Terminal archive are signed and
notarised on a Mac that holds the Developer ID certificate. No Windows code-signing certificate is configured, so Windows may warn about the installer.
- Check the licenses and provenance of bundled fonts and assets when packaging. The early D05 checklist lacked redistribution evidence;
treat that as an open checklist finding, not an audit of every release asset.
## History / older measurements
- **Tauri config:** an early draft used `perMachine`, `oneClick` and `deleteAppDataOnUninstall`. The Tauri schema rejects them; the
current config uses `installMode: currentUser`.
- **Repository size:** `apps/desktop/src-tauri/target` and the Node sidecar were briefly committed, which left about 324 MB in the git object
store. They are untracked and ignored now. History was not rewritten.
- **Mac packaging tests:** the first DMG self-test (8/8) and the engine's 17 kernel-parity suites ran on an Apple M4 GPU in a 16 GiB Mac, with service stand-ins for signing.
Later inference measurements on larger Macs are in [MAC.md](../docs/MAC.md).
- **Intel Arc:** official since 0.1.40.2. The 2026-10 B70 tests used the source build, not the engine package. `sycl/setup_intel.py` (Docker) remains the source-build route.
packaging/UPDATES.md
+38 -19
View File
@@ -1,10 +1,14 @@
# Updates: the signed feed, the component updater, the app updater
Everything that downloads software on a user's PC lives behind one explicit step: `strata update` in the Terminal, or
"Update engine" / "Update Strata" in the app's Settings. Starting a model and serving never download anything
This page is for developers who build, sign and publish releases. Users update with [INSTALL.md](../docs/INSTALL.md#updating).
It covers the three updated parts, the feed format, what a PC checks, and the release steps.
Everything that downloads software on a user's PC sits behind one explicit step: `strata update` in the Terminal, or
"Update engine" or "Update Strata" in the app's Settings. Starting a model and serving never download anything
(`STRATA_NO_ACQUIRE` stays on for them; the update step is a separate Control process that is allowed to). Engines are
separate from the app and ship more often: an app-only update downloads 0 engine bytes (the update installers are built without the engine), an engine update downloads only the
package for that PC's GPU. **Models and saved tuning presets are never downloaded, replaced or deleted by any update.**
separate from the app and ship more often. An app-only update downloads no engine bytes, because the Windows and Linux installers are built
without the engine. An engine update downloads only the package for that PC's GPU.
**Models and saved tuning presets are never downloaded, replaced or deleted by any update.**
## The three things that are updated
@@ -19,33 +23,41 @@ Two separate private keys, both kept by the owner, neither in git, neither the s
* **feed key**: Ed25519, signs `strata-feed.json` and every engine package manifest.
* **Tauri updater key**: minisign (what `tauri signer generate` makes), signs the app installers.
<a id="the-feed"></a>
## The feed (`strata-feed.json` + `strata-feed.json.sig`)
`strata-feed.json.sig` is `{"alg": "ed25519", "key_id": "<16 hex>", "value": "<base64>"}` over the **exact bytes** of
`strata-feed.json` (no canonicalisation). The feed:
`strata-feed.json` (no canonicalisation). The annotated example below illustrates the format; it is not a current signed feed.
Comments are for this document only; actual feed files must be strict JSON.
```json
```jsonc
{
"schema_version": 1, "feed": "strata-updates", "channel": "stable",
"sequence": 12, // goes up with every feed; an older one than the last accepted is refused
"generated_at": "2026-10-20T10:00:00Z", "expires_at": "2027-10-20T10:00:00Z", // expires_at may be null
"app": { // exactly Tauri's static updater JSON, plus sha256/size per platform
"version": "0.2.0", "notes": "...", "pub_date": "2026-10-20T10:00:00Z",
"platforms": {"windows-x86_64": {"url": "app/Strata_0.2.0_x64-setup.exe", "signature": "<minisign, base64>", "sha256": "...", "size": 71000000}}
"version": "0.2.5", "notes": "...", "pub_date": "2026-10-20T10:00:00Z",
"platforms": {"windows-x86_64": {"url": "app/Strata-0.2.5-windows-x64-setup.exe", "signature": "<minisign, base64>", "sha256": "...", "size": 71000000}}
},
"control": {"version": "0.1.48", "packages": [{"platform": "linux-x86_64", "format": "tar.gz", "url": "control/...tar.gz",
"control": {"version": "0.1.50", "packages": [{"platform": "linux-x86_64", "format": "tar.gz", "url": "control/...tar.gz",
"size": 0, "sha256": "...", "installed_bytes": 0, "entry": "strata.py"}]},
"engine": {"packages": [{
"version": "0.1.48", "platform": "linux-x86_64", "backend": "cuda", "variant": "cuda13",
"version": "0.1.50", "platform": "linux-x86_64", "backend": "cuda", "variant": "cuda13",
"archs": ["sm_75", "sm_86", "sm_89", "sm_120", "compute_120"], "min_driver": {"nvidia": "580.65"}, "min_runtime": null,
"build_id": "be885026f96a-cuda13.0", "format": "tar.gz", "url": "engine/strata-engine-0.1.48-linux-x86_64-cuda-cuda13.tar.gz",
"build_id": "be885026f96a-cuda13.0", "format": "tar.gz", "url": "engine/strata-engine-0.1.50-linux-x86_64-cuda-cuda13.tar.gz",
"size": 505568970, // bytes of the archive
"sha256": "...", // of the archive
"manifest_sha256": "...", // of the package's engine-component.json (it holds every file's hash)
"installed_bytes": 840893419, "notes": ""}]}
"installed_bytes": 840893419, "notes": ""}]},
"dsv41": {"packages": []} // optional: the separate DeepSeek engine packages, same entry shape as "engine"
}
```
The DeepSeek V4.1-Flash engine `strata-dsv41` reaches a PC in one of two ways. The engine package carries it beside `strata`
(`package_engine.py --require-dsv41`), or it is a separate package in the feed's `dsv41` member, packed with
`pack-engine --prefix strata-dsv41`. The separate package installs in `<data>/engine-dsv41/` beside the engine, never inside it
(`strata_control/dsv41_engine.py`), and is fetched when DeepSeek is chosen. `STRATA_DSV41_EXE` points setup at a build of your own.
`url` is relative to the feed's own location (or an absolute `https://` address; a loopback `http://` is allowed so a local test
feed works; a local feed folder may only name files inside itself).
For a GitHub release the assets are flat: pack with `--url-prefix ""` and upload every file beside the feed.
@@ -98,7 +110,7 @@ On an offline machine, outside any git checkout:
npx tauri signer generate -w D:\strata-keys\tauri-updater.key # asks for a password; prints the public key
Back both up (a lost feed key means every installed PC needs a build with a new key; a lost Tauri key means the same for the app).
Then, in git, once, in the commit that ships the first signed release (done for 0.2.0, key id 5d1bc23abbeffd0d):
Then, in git, once, in the commit that ships the first signed release (the release key id is `5d1bc23abbeffd0d`):
* the `keys` entry from `feed-key.pub.json` into `strata_control/update_keys.json`;
* the Tauri public key into `apps/desktop/src-tauri/tauri.conf.json` (`plugins.updater.pubkey`) and into `update_keys.json` as `app_minisign_public`
@@ -110,7 +122,7 @@ without their key being part of a release build. A build whose `update_keys.json
feed with "this build trusts no update key".
A key rotation is a release that carries both public keys; the old private key signs that release, the new one signs the next.
## Releasing (when the owner says go)
## Releasing
Keys live only on the release PC (`feed-key.json`, `tauri-updater.key`; the Tauri key has no password). Every build box builds WITHOUT a key:
the default `tauri.conf.json` has `createUpdaterArtifacts: false`, so `tauri build` needs no signing key and makes the same installer bytes.
@@ -129,9 +141,9 @@ All signing happens on the release PC, on the finished files.
python packaging/feed_tool.py sign-package <engine package folder> --key <feed-key.json> # every engine package (the Mac app bundles its UNSIGNED copy)
python packaging/feed_tool.py pack-engine <engine package folder> --out RELEASE/engine --url-prefix=
python packaging/feed_tool.py pack-control <control folder> --version 0.1.48 --platform P --out RELEASE/control --url-prefix= # --format zip for windows-x86_64
python packaging/feed_tool.py pack-control <control folder> --version 0.1.50 --platform P --out RELEASE/control --url-prefix= # --format zip for windows-x86_64
# --url-prefix= (not --url-prefix "") because PowerShell 5.1 drops an empty argument. The Control version is the VERSION file
# (0.1.48), not the app version: a Control package newer than the installed one is installed by `strata update`.
# (0.1.50), not the app version: a Control package newer than the installed one is installed by `strata update`.
# Pack Linux and Mac packages on Linux / the Mac or on Windows with this tool: it sets the execute bit from the file header either way.
cd apps/desktop && npx tauri build --bundles nsis --config src-tauri/tauri.update.conf.json # Windows installer (no engine inside)
# Linux: tauri build --bundles appimage (no engine inside) is the AppImage
@@ -140,11 +152,11 @@ All signing happens on the release PC, on the finished files.
python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target windows-x86_64 --version 0.2.5 --archive Strata-0.2.5-windows-x64-setup.exe --sig Strata-0.2.5-windows-x64-setup.exe.sig --notes "..."
python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target linux-x86_64 --version 0.2.5 --archive Strata-0.2.5-linux-x86_64.AppImage --sig Strata-0.2.5-linux-x86_64.AppImage.sig
python packaging/feed_tool.py set-app-platform --app-json RELEASE/tauri-latest.json --target darwin-aarch64 --version 0.2.5 --archive Strata.app.tar.gz --sig Strata.app.tar.gz.sig
python packaging/feed_tool.py build-feed --out RELEASE --sequence N --key <feed-key.json> --engine RELEASE/engine --control RELEASE/control --app-json RELEASE/tauri-latest.json
python packaging/feed_tool.py build-feed --out RELEASE --sequence N --key <feed-key.json> --engine RELEASE/engine --control RELEASE/control --app-json RELEASE/tauri-latest.json [--dsv41 RELEASE/dsv41]
python packaging/feed_tool.py verify-feed RELEASE --pub <feed-key.pub.json>
`set-app-platform` checks each `.sig` against the public key in `tauri.conf.json` over the exact bytes, so a file signed with another key, or changed after
signing, stops the release. The full installers are not in `latest.json` and need no Tauri signature (they are downloaded by hand).
signing, stops the release. The Mac dmg is not in `latest.json` (the updater uses `Strata.app.tar.gz`) and needs no Tauri signature; it is downloaded by hand.
### The Mac app
@@ -163,7 +175,7 @@ Mach-O in the app with the Developer ID and notarises and staples it, which chan
The Windows installer and the Linux AppImage need no extra step: their `.sig` files from `tauri build` are the ones to pass to `set-app-platform`. Re-signing the Mac app after the archive
was made (a second `sign-and-notarize.sh` run) makes a new archive: sign and add it again.
`build-feed` also writes `RELEASE/install-manifest.txt`, the file the one-line installers read (`install.sh` L76, `install.ps1` L45): one line per Control package,
`build-feed` also writes `RELEASE/install-manifest.txt`, the file the one-line installers read: one line per Control package,
`platform version url sha256 size`, taken from the `*.entry.json` files in `RELEASE/control`. The Control package is the Terminal that `install.sh` unpacks (it must hold `bin/strata`).
`install.sh` unpacks with `tar -xzf` and `install.ps1` with `Expand-Archive`, so `windows-x86_64` must be packed with `--format zip` and the other platforms as `tar.gz`;
`build-feed` leaves a pack in the wrong format out of the manifest and says so. Pack Control for every platform you want the one-line install to cover (`install.sh` knows `linux-x86_64` and `macos-arm64`, `install.ps1` `windows-x86_64`);
@@ -185,3 +197,10 @@ Everything above runs against a folder or a loopback server: `tests/terminal/tes
`tests/terminal/test_updater.py` (end to end against a loopback HTTP feed with stub engines: update, test-start, switch; a failing test start
keeps the old engine; an interrupted download resumes; rollback; Control; the app installer's minisign check), `apps/host/tests/component-updates.test.mjs`
(the Settings buttons). For a real engine on a box, build a package with `packaging/package_engine.py`, serve it from a local feed made by `packaging/make_local_feed.py`, and run `strata update --feed` against it.
## History / older measurements
- **Keys:** the feed key and the Tauri updater key were created once and the feed key's public half was committed in `strata_control/update_keys.json`
with the first signed release (0.2.0-beta). Terminals from before that build trust no key and refuse every feed.
- **Engine-less installers:** an app update once carried an engine. The Windows and Linux installers now drop `engine-bundle`
(`tauri.update.conf.json`), so one `latest.json` entry per OS serves every card variant.
tools/dflash2/README.md
+25 -15
View File
@@ -1,20 +1,30 @@
# DFlash2 acceptance experiment (research tooling, not shipped)
# DFlash2 acceptance experiment
Measures how many tokens per round the incoai DFlash2 drafter for Qwen3.8-Flash-Next would get on OUR quantized target, offline, from the hidden states
the engine dumps. Results and method: phaseA-tests/notes-dflash2.md. All scripts keep the paths of the run (C:\dflash2-work on the owner PC, C:\strata-test\dflash2
on the test box): edit them before reuse.
Research tooling, not shipped with the app. It measures how many tokens per round the incoai DFlash2 drafter for
Qwen3.8-Flash-Next would accept on Strata's quantized target. The measurement is offline, from hidden states the
engine dumps. It is for developers who want to repeat or extend the experiment.
Engine side (both off unless the variable is set):
The scripts keep the paths of the original run (`C:\dflash2-work` on the development PC, `C:\strata-test\dflash2` on
the test box). Edit the paths before you reuse them. The method and results are in the project's internal notes
and are not part of this tree.
- `STRATA_TAP_FILE=<file>` (+ `STRATA_TAP_LAYERS`, default `5,15,25,35,45`): `Verifier::record_window` copies each verified row's completed layer output (the mean of
the 4 hyper-connection streams, the quantity the drafter was trained on) and `Verifier::run` appends `pos, token, taps` records.
- `STRATA_SPEC_T_CYCLE=1,2,...,8` (CLI bench only): the verify window size cycles; the report line `window cost (ms/round)` gives the cost of each width.
## Engine variables
Scripts:
Both are off unless set.
- `mk_prompts.py` builds chat-template prompt ids with the server's own tokenizer and template.
- `orch.py <prompt> <max_new> [gen,genf,genfm,tap]` drives `run_one.ps1` on the test box: free generation (ids, MTP statistics), the same text with MTP only, and the
teacher-forced tap dump with oracle drafts (`--spec 8 --spec-oracle --spec-follow`, a 2-token prompt: a native pack never starts the speculative loop from a 1-token prompt).
- `dflash2_ref.py` is a plain-torch port of z-lab's `DFlash2DraftModel` (checked against it by `test_port.py`), with the target's embedding and head dequantized from the GGUF.
- `run_eval.py` (walk, a few block sizes), `run_all2.py` (every position, resumable: per-anchor correctness and the selector's confidences), `analyze2.py` and
`final_tables.py` (policies, speed estimates from the measured window costs).
| Variable | Effect |
| --- | --- |
| `STRATA_TAP_FILE=<file>` | Writes `pos, token, taps` records. `Verifier::record_window` copies each verified row's completed layer output (the mean of the 4 hyper-connection streams, the quantity the drafter was trained on) and `Verifier::run` appends it. |
| `STRATA_TAP_LAYERS` | Layers to tap, default `5,15,25,35,45`. Used with `STRATA_TAP_FILE`. |
| `STRATA_SPEC_T_CYCLE=1,2,...,8` | CLI bench only. The verify window size cycles; the report line `window cost (ms/round)` gives the cost of each width. |
## Scripts
| Script | Purpose |
| --- | --- |
| `mk_prompts.py` | Builds chat-template prompt ids with the server's own tokenizer and template. |
| `orch.py <prompt> <max_new> [gen,genf,genfm,tap]` | Drives `run_one.ps1` on the test box: free generation (ids and MTP statistics), the same text with MTP only, and the teacher-forced tap dump with oracle drafts (`--spec 8 --spec-oracle --spec-follow`, a 2-token prompt, because a native pack never starts the speculative loop from a 1-token prompt). |
| `dflash2_ref.py` | Plain-torch port of z-lab's `DFlash2DraftModel`, with the target's embedding and head dequantized from the GGUF. `test_port.py` checks it against the original. |
| `run_eval.py` | Walks the tap file with a few block sizes. |
| `run_all2.py` | Evaluates every position, resumable: per-anchor correctness and the selector's confidences. |
| `analyze2.py`, `final_tables.py` | Policies, and speed estimates from the measured window costs. |